Repository navigation
Add opt-in deadline-triggered durable cleanup with an immutable total budget #863
Description
User problem and product decision
Applications using sagas reasonably need a business deadline to trigger durable compensation. Today an engine run/execution deadline closes the run without executing workflow code. Correct documentation is necessary, but it does not provide automatic deadline-driven cleanup.
Accept an additive, opt-in timeout cleanup feature for design and implementation after current correctness fixes. Keep today's hard timeout as the default. Documentation correction durable-workflow/durable-workflow.github.io#287 proceeds independently and must not advertise this proposal as shipped.
Existing mechanisms to reuse
- Workflow #483 delivered durable cooperative cancellation and bounded shielded cleanup.
- Shared #136 delivered published Server/PHP/Python/Rust cooperation, recovery, lineage and observable cleanup outcomes.
- Workflow #595 enforces the original cleanup deadline at task completion and renewal.
Those mechanisms respond to cancellation requests. They do not automatically turn a run/execution deadline into a cleanup turn. At released Core 64efa176e4c37e5d6ae80e767ba3b41464024e53, WorkflowExecutor::timeoutIfDeadlineExpired() invokes the terminal timeoutRun() transition before workflow execution. Merely putting compensation in finally cannot change that.
Proposed contract
- Configure an ordinary-work deadline and a finite cleanup allowance at admission. Both are durable, immutable and inspectable. Define the total wall-clock cap explicitly. For example, 60 seconds of work plus at most 10 seconds of cleanup means an original hard cap of 70 seconds. Never silently extend the existing execution deadline to accommodate cleanup.
- At the work deadline, atomically enter timeout cleanup, fence ordinary progress and deliver a typed timeout reason through the existing durable interruption/shield machinery. Workflow-authored cleanup may explicitly invoke saga compensation or idempotent cleanup activities. The engine does not guess how to undo external effects.
- Preserve the timeout cause and final timed-out outcome even if compensation succeeds. Expose cleanup as completed, failed, expired or preempted, independently of the business outcome. Reusing cancellation machinery must not accidentally report the workflow as a successful completion or an ordinary user cancellation.
- Anchor the cleanup deadline to the original work deadline, not when a watchdog or replacement worker notices it. Late detection consumes the allowance. Duplicate delivery, retries, continue-as-new and replacement must not refresh either clock. Run and whole-execution budgets require explicit precedence and inheritance.
- Hard timeout and forced termination can preempt cleanup. An unavailable worker cannot guarantee cleanup. If the original cap is already exhausted, close and fence rather than start an unbounded recovery period.
- Retain existing child/activity cancellation policies, original request lineage and stale-attempt fencing. Define races with signals, updates, completion, cancellation and continue-as-new before freezing a wire change. Cleanup cannot reopen expired business-work authority.
- Document at-least-once external effects, idempotent compensations and independent leases/reconciliation. Cleanup is durable execution, not exactly-once undo.
Exact API names and history representation are implementation decisions after compatibility review; the two-deadline and outcome contract above is the accepted direction.
Acceptance
- Reproduce the current hard-timeout behavior and keep it unchanged for workflows without the opt-in policy.
- Test genuine deadline expiry while waiting on activity, timer, signal, child and selection; verify the cleanup transition once, original identities and typed timeout cause.
- Prove compensation success, retry/exhausted failure, delayed watchdog, duplicate delivery, worker SIGKILL/fresh-process replay and expiry under the original total cap.
- Prove terminate/timeout/completion races and late activity/workflow results cannot commit after their authority ends. Show the workflow remains timed out after successful cleanup.
- Specify execution/run/continue-as-new/child budget relationships and diagnostics. Test both embedded execution and service-mode PHP/Python/Rust before advertising portable support; native Server adoption remains separately coordinated.
- Qualify exact published packages/images and document the resulting user journey before closure. A source prototype or passing documentation build is not feature delivery.
Priority P2: a useful additive reliability feature. No implementation is shipped by this issue, no release date is promised, and the current update-idempotency P1 remains ahead of it.
Activity
rmcdaniel commentedon Nov 4, 2026
Owner: Workflow for the initial opt-in deadline-cleanup source contract; General/Security for compatibility/outcome decisions; Server/SDK owners for portability. Existing cancellation shields do not cause a hard deadline to execute cleanup.
Next is the two immutable admission deadlines and typed timeout-cleanup state/outcome table, including late detection, original total cap, continue-as-new inheritance and terminate/completion races. I asked General to prioritize that concrete contract after correctness work. Preserve hard timeout as the default and do not refresh an allowance on watchdog/replacement/retry. No new history/API or producer is implemented as part of #1030's unrelated guard floor, and a docs finally example cannot complete the feature.
rmcdaniel commentedon Nov 4, 2026
General concrete open-item decision, November4.
The issue's two-clock contract is affirmed for opt-in new executions: work deadline and absolute total hard cap are fixed at original admission; cleanup deadline is min(work deadline + admitted finite allowance, original total cap). Late detection, renewal, retry, replacement and continue-as-new never refresh either clock. At work expiry, fence normal work and enter typed timeout cleanup once; final business result remains timed-out with independent completed/failed/expired/preempted cleanup outcome. Hard cap and termination can preempt it. Parent/child cleanup cannot create authority beyond the inherited original cap. Workflow owns the exact state/outcome/vector draft after correctness work, with Server/SDK review; existing hard timeout stays default, and no new producer/runtime is admitted yet.
This assigns the next bounded source action; normal exact-head CI/Security/integration remain required. Existing operator stops and production/release/resource gates are unchanged.
rmcdaniel commentedon Nov 4, 2026
Workflow source-plan follow-through to General’s decision.
Owner: Workflow. Prepare a state/outcome/vector contract for opt-in new executions: fixed original work deadline and total cap, cleanup deadline min(work + admitted finite allowance, original cap), one timeout-cleanup transition and independent cleanup outcome. Include late detection, renewal, retry, replacement, continue-as-new, parent/child cap inheritance and hard-cap/termination preemption. Existing hard timeout remains default; no producer or runtime begins.
The active #1030 frozen quality checkpoint remains first. This records the concrete next source action and its required review, without claiming implementation or starting another qualification workload. General/Security retain compatibility/integration disposition and named peer reviews remain as recorded on this issue.
rmcdaniel commentedon Nov 11, 2026
#1030 was closed as superseded by #1076, so this issue no longer waits on it. The next source action recorded above is unchanged. Owner: Workflow. It's queued behind the open defect PRs #1077–#1083.
rmcdaniel commentedon Nov 13, 2026
The contract draft is up for review in #1122 (
docs/architecture/timeout-cleanup.md). It covers admission (the cleanup allowance reserves the end of the existing hard timeout, so the cap is never extended), inheritance, the one-timeWorkflowTimeoutCleanupStartedtransition reusing cooperative-cancellation delivery, the always-WorkflowTimedOutoutcome with an independenttimeout_cleanup.outcome, the service-mode capability, 12 vectors and three open questions.Next action: Server, the SDK owners and Waterline review the shape on #1122. Once the open questions are decided, the embedded implementation follows. Owner: Workflow (the workflow agent).
Record server's review decisions in the timeout cleanup draft (#863)
Record the SDK owner's decisions in the timeout cleanup draft (#863)
Record the failure-category decision; the timeout cleanup contract is…
Merge pull request #1122 from durable-workflow/design/863-timeout-cle…
rmcdaniel commentedon Nov 13, 2026
The contract is agreed and merged: #1122 (
docs/architecture/timeout-cleanup.md), with server's, General's and Security's decisions recorded. Next action: the embedded implementation, after 2.28.0. Owner: Workflow.rmcdaniel commentedon Nov 13, 2026
Implementation plan (embedded first), following the merged contract
docs/architecture/timeout-cleanup.md. Branch:feature/863-timeout-cleanup-admission.StartOptionsgetstimeoutCleanupSecondsandwithTimeoutCleanup(), validated before any write: 1–86,400 s, at least one hard timeout, and smaller than each one.WorkflowStartedrecordstimeout_cleanup {allowance_seconds, work_deadline_at per kind}, and the payload contract and api-stability are updated. Continue-as-new and redrive carry the allowance; the execution clocks are inherited and the run clocks re-derived.WorkflowExecutor::timeoutIfDeadlineExpired()runs beforetimeoutRun(). If the run opted in, the work deadline has passed, the hard deadline hasn't, and no cooperative request exists, it enters cleanup once, under the run lock. Entering cleanup does three things in one step:RequestCancellationcommand, using the same shape asWorkflowStub::attemptRequestCancellation, with aCancellationContextofcause: timeout, sourcesystem, andcleanup_deadline_atset to the contract's cleanup deadline;cancellation_request_command_id,cancellation_requested_atandcancellation_deadline_at, and recordsCooperativeCancellationRequestedplus the newWorkflowTimeoutCleanupStarted;The watchdog finds the work deadline the same way.
WorkflowTimeoutCleanupRequested(a subclass ofWorkflowCancellationRequestedException) when the context's cause istimeout.finishCooperativeCancellation()and the cleanup-deadline-expiry path detect a timeout-cause request. They close the run withWorkflowTimedOut(closed_reason: timed_out,failure_category: timeout) plustimeout_cleanup {outcome: completed|failed|expired, started_at, cleanup_deadline_at, finished_at, failure_id}, instead ofWorkflowCancelled. A preemption by terminate, a hard cancel or the hard deadline recordsoutcome: preemptedon the preempting terminal event. Run detail, describe, the export and the failure list show 'cleanup failed' plainly, and redrive eligibility follows the timeout category.WorkflowTimeoutCleanupStarted.timeout_cleanupcapability and admission refusal, then the server parity issue.Owner: Workflow (the workflow agent).
Store the derived timeout cleanup work deadline on the run (#863, ste…
Amend the timeout cleanup contract: embedded stores the work deadline (…
Carry the timeout cleanup allowance across continue-as-new (#863, ste…
Carry the timeout cleanup allowance across redrive (#863, step 2c)
Deliver the timeout cleanup interruption with cause timeout (#863, st…
Enter timeout cleanup at the work deadline and end it timed out (#863,…
Record a preempted timeout cleanup; changelog and api-stability (#863)
Add timeout cleanup corpus fixtures and consumers (#863)
Say how timeout cleanup ended wherever a timeout is shown (#863)
Write the work-deadline column only for opted-in runs (#863)
Settle the SDK cause mismatches (#863)
Keep only `cause` in the cancellation context
rmcdaniel commentedon Nov 13, 2026
Workflow is driving the remaining Security findings on this PR. The current head
14a4b944is green in CI and fixes the scoped-descendant cause leak, but the other findings from the review at5207d232remain open.Next actions, owned by Workflow:
timeout_cleanup.outcome: expiredat every hard-deadline boundary.The embedded implementation remains on this PR. Service-mode publication and consumer follow-through will stay explicit on #863.
rmcdaniel commentedon Nov 13, 2026
Workflow pushed the review corrections to #1129 at
396b78b1. MySQL 8.4.8 focused checks pass (28 tests / 135 assertions), ECS and PHPStan pass, and four new expiry/replay-blocking regression controls fail on the preceding head then pass with the fix. Full unit/coverage/feature checks, CI and exact-head Security re-review are in progress; the PR stays draft meanwhile.Native/SDK implementation and published acceptance are now tracked in server#1201, including the language-neutral timeout interruption marker. This embedded increment explicitly refuses unsupported poll/service admission and bridge claims. Embedded merge or publication alone will not close the cross-product requirement here.