Repository navigation
Account every completion write path before mutation and bound legacy deadline closure (#930 phase B) #1116
Description
Split from #930. Phase A shipped the reported defect's fix. Phase A (287bfe7, merged in #934) added:
- timing caps on
record_local_activity, each with its own 422; - a
HistoryTransactionMeteron every completion that charges events, bytes (repeated heartbeat snapshot bytes included) and rows, and rolls back with the lease kept; - an early
completion_budget_exceededcheck; - metered staging of external payloads.
This issue holds the hardening that's left over. It is beyond the reported amplifier, so it's P3.
Remaining work
- Exact pre-write accounting for every write path. Phase A's early check counts local activities exactly and every other command at a lower bound of one event. State-driven cascades (projection, child/parent fan-out, cancellation cascades, checkpoints, bulk/pivot writes) are caught only by the meter when the write happens, not before. Inventory each mutation family's logical writes and repeated encoded and raw bytes. Then admit with queries that are scoped and bounded by remaining+1, before the first write.
- Bounded closure of oversized legacy deadlines. A selected run's deadline close is unmetered today. Ordinary expired runs with an existing Ready/Leased workflow task can also be excluded from repair selection, leaving an unclaimed Ready run pending beyond its deadline (confirmed in Workflow's 2026-11-13 source-prototype follow-up below). Propose a bounded fence or cleanup-cursor design instead of an unmetered table-wide cascade. A budget refusal must never prevent a deadline close indefinitely.
Acceptance
docs/architecture/completion-budget.mdlists each write family and how it is charged.- Every family has exact-boundary and one-over tests, each showing that a refusal leaves executions, attempts, history, projections and receipts unchanged and keeps the task lease.
- The deadline-closure design is reviewed by General and Security before closure semantics change.
- Native parity is tracked in Rust #325: record local activities like Server 2.11.1 server#766.
Owner: Workflow. Decision record: General's 2026-11-04 comment on #930.
Activity
rmcdaniel commentedon Nov 13, 2026
Workflow owns phase B. The active timeout-cleanup PR #1129 still uses runtime-owned deadline closure rather than a budget refusal; changing that closure design remains this issue's separately reviewed work. Its new engine-owned timeout-cleanup entry and close-settlement writes must be included in the mutation-family inventory alongside checkpoints, projections and child/cancellation cascades.
Next action after the current #863 and #996 implementations: draft exact pre-write charges and a bounded legacy-close design in docs/architecture/completion-budget.md, with exact-boundary/one-over rollback controls. General and Security review the closure design before source behavior changes. Native parity remains server#766. No coverage-reservation work from #426 is being taken over.
rmcdaniel commentedon Nov 13, 2026
The installed-source prototype for863 found a concrete legacy deadline-closure gap to include in this design. On source
b930facd(and the same predicate exists in released2.288ac207da), a no-opt-in run can remain Pending with its original workflow task Ready/attempt0 after its admitted execution deadline. RepeatedTaskWatchdog::runPass(runIds: [...])does not select it:deadlineExpiredRunQuery()excludes ordinary runs with an existing Ready/Leased workflow task. The opt-in work-deadline/cancellation exceptions added for863 do not apply to an ordinary run.This was an exported-source Composer-path prototype, not published-artifact acceptance. The database readback shows the fixed expired execution deadline, no timeout cleanup work deadline, and the original unclaimed Ready task. It is not caused by missing workers executing PHP or by a renewed budget. Current1129's new opt-in no-worker and late-detection cases pass; changing ordinary deadline closure would broaden its agreed scope.
Workflow owns the follow-up here: the bounded closure/fence/cursor design must cover repair selection and progress for these existing Ready/Leased tasks as well as mutation cost. General/Security review remains required before that closure change. The existing sentence saying every expired run always closes needs this qualification: the close is unmetered once selected, but availability selection can currently defer it indefinitely. For1130, the legacy compatibility control will start the ordinary task and verify its existing hard-timeout/no-cleanup behavior; it will not claim that the separate unclaimed legacy case is fixed.
rmcdaniel commentedon Nov 13, 2026
The new prototype's ordinary Pending/Ready deadline exclusion is a confirmed correctness defect to repair under this existing issue. It is not completed by describing the limitation. Preserve #1129/#1130's current opt-in release scope and do not claim an unclaimed legacy-run repair from its started-task compatibility control.
Workflow owns the smallest bounded proposal for the Ready/Leased selection and fenced closure after the current release, with focused deadline/lease/replacement-worker and budget-refusal controls. General and Security retain this issue's existing review gate before changing legacy closure semantics. The proposal can address that concrete defect without waiting for every other Phase B accounting family. Native parity remains linked to Server #766.