Skip to content

Fail the run on a typed oversized-completion 413 (.github#228) - #308

Merged
rmcdaniel merged 5 commits into
mainfrom
agent/general/228-completion-413
Nov 13, 2026
Merged

rmcdaniel merged 5 commits into
mainfrom
agent/general/228-completion-413

Conversation

@rmcdaniel

@rmcdaniel rmcdaniel commented Nov 13, 2026 •

Copy link
Copy Markdown
Member

SDK-python slice of durable-workflow/.github#228.

Problem. Server 3.6.0 (server#1152) accepts a workflow-task completion up to 16 payload limits (32 MiB by default). A larger one gets 413 payload_too_large with limit_kind: workflow_task_completion_bytes. The worker treated this as a definite completion rejection and called POST .../fail with workflow task completion failed after commands were produced: .... The Server's blocks_replay matches that text, so the run was replay-blocked. A plain task failure is no better: the Server retries the task right away, replay produces the same commands, and the worker gets the same 413 again, with no end.

Change.

  • WorkflowTaskCompletionTooLarge, a non-retryable ServerError exported from durable_workflow, is raised only for the typed 413 (status 413, reason: payload_too_large, limit_kind: workflow_task_completion_bytes). It carries limit from the Server and size, the encoded request body length the client measured (the Server body does not include the actual size).
  • The worker never resends the refused completion. It completes the same leased task once with one fail_workflow command: non_retryable: true, exception_type: WorkflowTaskCompletionTooLarge, and a message starting Structural limit exceeded: workflow_task_completion_bytes: that gives the size and the limit and says to split the fan-out, use child workflows or continue-as-new. The Server records that prefix under the structural_limit failure category, the documented structural-limit outcome. It does not report a workflow-task failure. If that small completion also fails, the existing completion-error handling applies to its error, with no loop.
  • Other 413 bodies and 5xx/429 retry behaviour are unchanged.
  • CHANGELOG (Unreleased) and a new "Workflow-task completion size" section in docs/sdk-reference.md.

The SDK does not use the activity-task continuation completion route, so that route has nothing to change here.

Tests. tests/test_completion_size_limit.py uses a real Client over httpx.MockTransport. It checks:

  • a typed 413 maps to the typed error with limit and size
  • the oversized completion is sent exactly once, then one fail_workflow completion follows
  • if the fallback completion fails, the existing handling runs once
  • an untyped 413 keeps the old task failure
  • a 503 still retries the identical completion

A replay-regression corpus fixture, completion-size-limit-fails-run-once, records the same request sequence. CI's corpus policy requires one because the worker changed.

Local, run in Docker the same way CI does: ruff, mypy, regression-corpus validation, benchmark, full unit suite (5928 passed), package build, metadata and smoke, strict docs build.

🤖 Generated with Claude Code

Server 3.6.0 refuses a workflow-task completion over 16 payload limits
(32 MiB by default) with 413 payload_too_large and limit_kind
workflow_task_completion_bytes. The worker treated that as a definite
completion rejection and reported a workflow-task failure whose message
("workflow task completion failed after commands were produced") makes
the Server block replay; an ordinary task failure would instead retry the
same oversized completion.

Map the typed 413 to WorkflowTaskCompletionTooLarge (a non-retryable
ServerError with the Server's limit and the encoded request size). The
worker never resends the refused completion: it completes the same task
once with a non-retryable fail_workflow command whose message starts with
"Structural limit exceeded: workflow_task_completion_bytes:", which the
Server records as a structural_limit failure, with guidance to split the
fan-out, use child workflows or continue-as-new. If that completion also
fails, the existing completion-error handling applies to it. Other 413s
and 5xx retries are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rmcdaniel and others added 2 commits November 13, 2026 04:41
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@rmcdaniel

Copy link
Copy Markdown
Member Author

Security: CHANGES REQUESTED at 35235ff. The fallback can reach /fail.
_fail_run_for_oversized_completion (worker.py:3076-3114) re-raises the follow-up's own error (raise fail_error from error). The outer handler then sends it through _should_fail_workflow_task_after_completion_error (:3046-3049), which returns True for a 409 invalid_commands, a 409 with no reason, a 422, InvalidArgument and Unauthorized. So _report_workflow_task_after_completion_error calls fail_workflow_task, which is /fail. The test test_failed_structural_limit_completion_falls_back_without_looping (tests/test_completion_size_limit.py:136) asserts this /fail call. That contradicts the agreed design: on this path /fail must never be called, because it blocks replay on Server.
Fix: when the follow-up fails, raise the original typed WorkflowTaskCompletionTooLarge (with the follow-up error as __cause__), or mark it so the outer handler returns without reporting. Flip that test to assert /fail is never called, and add a 422 case.
Everything else passes: strict detection, a single oversized send (the retry predicates only cover network errors, 5xx and 429), the follow-up uses the same task, worker id and attempt, the message is bounded with server_message kept out of it, and nothing from the payload is logged.

When the follow-up fail_workflow completion after a typed 413 is itself
rejected (409 invalid_commands, 422, 401, ...), re-raise the typed
WorkflowTaskCompletionTooLarge with the follow-up error as its cause, so
the worker returns without calling /fail. A workflow-task failure would
retry into the same 413 or block replay on the Server.

Refs durable-workflow/.github#228

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@rmcdaniel

Copy link
Copy Markdown
Member Author

Security PASS at 855055a, on green CI. This resolves my change request. When the follow-up fail_workflow fails, _fail_run_for_oversized_completion now re-raises the typed WorkflowTaskCompletionTooLarge (raise error from fail_error). _should_fail_workflow_task_after_completion_error returns False for that type first (worker.py:383), ahead of the generic 4xx rule, so /fail can't be reached on this path. The flipped test asserts that.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

# Conflicts:
#	CHANGELOG.md
#	src/durable_workflow/client.py
@sonarqubecloud

Copy link
Copy Markdown

@rmcdaniel

Copy link
Copy Markdown
Member Author

Security PASS at 9b6efca, on green CI. I checked it with git show --remerge-diff: only CHANGELOG.md (both entries kept) and the one client.py hunk. The resolution passes request_bytes only when raise_for_status is _raise_for_status, so the typed 413 still carries the sent size. #309's injected update-outcome hook is called exactly as main calls it, without the keyword it doesn't accept. If the default call ever returns without raising, the second call is the same default, so it's harmless. No other changes.

@rmcdaniel
rmcdaniel merged commit d179d75 into main Nov 13, 2026
21 checks passed
rmcdaniel added a commit that referenced this pull request Nov 13, 2026
Brings in #308 (typed completion-size 413) and #309 (update outcomes).
Only CHANGELOG.md conflicted; every Unreleased entry is kept.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant