Large fan-outs within documented limits block their run with a 413

Problem

A fan-out that stays within every documented limit can block its run permanently. The limits allow up to 1,000 operations in one all() and 2 MiB per argument. But a workflow task's commands are sent in one completion request, and the Server rejects any request body over 2 MiB.

Reproduced on published Server 2.9.9 with Python SDK 2.6.1:

A workflow that schedules 50 activities with 10 KB arguments in one parallel group completed.
One that schedules 300 such activities (about 3 MB in total) got 413 payload_too_large on workflow-task completion.
The worker recorded "workflow task completion failed after commands were produced". The run became workflow_replay_blocked with a failed task, no retry, no activities and only StartAccepted and WorkflowStarted in history.

Repair re-runs the same code into the same limit, so the run can't progress without a code change. The diagnosis says "replay blocked" rather than naming the size limit that was hit.

Acceptance criteria

A workflow task whose commands are each within the documented limits completes, in the PHP, Python and Rust SDKs and embedded mode. For example, large arguments are offloaded through the runtime payload transport, or commands are split across requests.
Anything that still can't be accepted fails with a typed structural-limit failure that names the limit and the measured size. It is not recorded as a replay block.
The structural limits page states any per-task aggregate limit beside command_batch_size and payload_size_bytes.
Tests against a published Server cover a 300 × 10 KB fan-out and the over-limit case in each SDK.

Accepted as a cross-repository product defect. The published 300 × 10 KB reproduction and permanent replay block are the baseline; the documented per-command limits must compose into a task that can actually be delivered.

The next implementation step is to trace the aggregate command serialization and runtime external-payload path in all three SDKs and the Server, then add a failing whole-task acceptance control. Preserve one atomic command batch and its deterministic ordering: splitting requests cannot introduce partial command acceptance or replay differences. An unsupported aggregate must produce a typed size-limit outcome with measured size, rather than a determinism/replay-block diagnosis. This is separate from oversized failure diagnostics in #225, though transport-sizing code may be shared after review.

Keep the 300 × 10 KB and actual over-limit published-artifact controls, embedded coverage and explicit aggregate-limit documentation in acceptance. Documentation alone does not close this issue. Current P1 activity-lease delivery remains ahead of this P2; no implementation or released fix is claimed here.

Scope note: the 2 MiB request cap also undercuts the documented per-payload limit for a single value, and for client requests as well as worker completions.

The cap applies to every request, because the middleware is prepended globally. That covers starts, signals and updates.
Payloads travel as base64 around the Avro bytes, which adds about a third. An argument under the 2 MiB per-payload limit, at roughly 1.5 MiB or more, therefore gets a 413 unless external payload storage offloads it.
The same setting is described two ways: the config reference calls DW_MAX_PAYLOAD_BYTES the maximum for one payload, while structural limits calls it the total request body.

Source-verified, not reproduced. The acceptance should state which size the limit measures (encoded or raw Avro bytes) and keep the request cap from rejecting a value that the documented per-payload limit allows.

General open-item pass, November 4. Owner: Server for request/command boundary; Workflow and SDKs for producer behavior; General for shared size contract.

Next action: Define encoded-envelope versus per-value limits and a bounded completion strategy with countercontrols for valid large fan-outs, avoiding silent drops or endless413 retries.

Action taken in this pass: Asked Server and Workflow for exact existing issue/source ownership and the narrow contract decision; documented1000-operation/per-value limits cannot be dismissed by an unrelated request cap.

General retains the cross-repository follow-through. Existing operator stops and exact source/release/production gates remain binding; the item stays open until its actual acceptance is evidenced.

Workflow maps this to existing #865 (per-site payload/pending guards) and #930 Phase B (whole-transition events/bytes/rows/payload staging). Both remain active; neither is an excuse for the published valid-300x10KB failure. Current bridge pending guards from #933 and start-input guarding already exist and must not be reimplemented from the old report.

Named next action: Server returns a raw Avro value versus base64/JSON envelope/request measurement table and a typed over-limit admission outcome; Workflow audits every normalized field/externalization site and transaction-local staging against it. Keep one atomic ordered completion, zero partial commands/puts on refusal, original deadlines and no endless413/replay-block loop. General owns any effective aggregate cap/capability change; SDKs must not split accepted semantics into independent partial requests. No new published cohort is started from this mapping, and #225 oversized diagnostics remains separate.

Workflow: in progress. workflow#865: service-mode argument/result size guards merged (#1083, ships in 2.23.1). Next: the remaining payload sites (side-effect, local-activity and update results, search attributes), then the fan-out audit against Server's size table. workflow#930 Phase B is P3 hardening.

Server: remaining, tracked in durable-workflow/server#1136 (native caps the whole completion body). Next: per-command limits with a typed structural_limit_exceeded. Owner: server.

PHP: offloads oversized activity arguments when the namespace has external payload storage. The typed over-limit outcome waits on server#1136, followed by a 300×10 KB test against a published Server. Owner: server, then sdk-php.

Server: durable-workflow/server#1152 lets workflow-task completions carry up to 16 payload limits (32 MiB by default) while each command's arguments still meet the 2 MiB limit. A 300 × 10 KB fan-out now completes on native. A larger completion gets 413 payload_too_large with limit_kind: workflow_task_completion_bytes and a message naming both limits. Remaining: the docs page states the aggregate, and SDKs surface that typed 413 as a structural-limit failure instead of a replay block. Owner: server (merged code), SDK owners and docs for the rest.

Add replay-corpus evidence for the oversized-completion outcome

Never report /fail when the structural-limit follow-up is rejected

Keep the Server's message out of the structural-limit failure

Cap the Server's message on the typed completion error

Installed acceptance on 2026-11-13, 09:08–09:15 UTC. Owner: sdk-python. Published artifacts only: PyPI durable-workflow==2.17.0 (installed outside the source checkout; release commit 29f69bfb74b03053c70d7e769196ded0c8668cbc) and native Server 3.8.0, Docker Hub index sha256:7988a23525d9e50e997a9f6f02e801300ed4f626de3cf6977033657ae7e9f58e. Disposable SQLite runtime, amd64, 2 CPUs / 2 GiB. No source overlay or production credentials. Publication workflow: https://github.com/durable-workflow/sdk-python/actions/runs/43398192253 .

Results:

Update outcome reads PASS: an accepted update without a worker returns typed UpdateStillPending; a replacement worker completes it with None; the later read returns None and a completed receipt. Exactly one UpdateAccepted is recorded. Reads do not resubmit the update.
300 × 10 KB fan-out PASS: a real Worker completed all 300 activities and the workflow, returning [300, 3000000].
Aggregate over-limit PRODUCT FAIL: 300 × 120 KB arguments produce an over-limit completion. The transport gets an empty httpx.ReadError, so it never receives the typed 413. The worker retries the completion and the run remains pending/repaired rather than failing structurally. Server confirms its Content-Length middleware rejects before reading this body. Server owns the response-delivery correction; Python will rerun installed acceptance after publication. This is not a complete Large fan-outs within documented limits block their run with a 413 #228 pass.

Minimal transport reproduction against that published Server (the token below is synthetic and local):

import asyncio
import httpx

async def main():
    async with httpx.AsyncClient(timeout=10) as client:
        response = await client.post(
            "http://127.0.0.1:8080/api/worker/workflow-tasks/fixture-task/complete",
            headers={"Authorization": "Bearer fixture-token", "X-Namespace": "default"},
            json={"commands": [{"arguments": "x" * 120000} for _ in range(300)]},
        )
        print(response.status_code)

asyncio.run(main())

Observed: httpx.ReadError('') in 0.11 seconds instead of HTTP 413. The middleware should reject before task lookup, so the synthetic task identity suffices. SDK mock controls pass (69 installed-package update/size tests), but those did not exercise this real HTTP transport failure. The first local fan-out attempt used the sequential example helper and exhausted its harness budget; the qualifying 300 × 10 KB result used the normal concurrent Worker.

Rust preparatory transport control, 2026-11-13: exact pre-publication SDK 4.15.0 package bytes (Cargo 1.86.0, source 01d72911078276f7b00cd5efbd145f1c0a5f3df3, SHA-256 4d6504e24845a37afaf325ae576570c9446a261430ecfc9ec795db46c8c1b328), against published native Server 3.8.0 index sha256:7988a23525d9e50e997a9f6f02e801300ed4f626de3cf6977033657ae7e9f58e, amd64. A direct Client::complete_workflow_task with 300 opaque 120KB entries produced a roughly 36MB completion body. Result: PRODUCT FAIL for the transport control, Error::Transport while sending the request; no typed HTTP 413 reaches the SDK. The synthetic task is intentionally nonexistent because aggregate middleware refusal precedes task lookup. This corroborates Python transport evidence, not full fan-out conformance. Runtime container cleanup passed. A first preparatory attempt was runner-blocked by offline Cargo configuration and was corrected before this product result.

The Rust mock controls and ordinary installed-native lifecycle/deadline acceptance remain passing, but they do not close this over-limit requirement. SDK is not yet published, so this is not published-SDK acceptance. Next action: Server owns response-delivery correction and its published tuple; sdk-rust owns rerunning actual installed fan-out/over-limit controls on that tuple and keeping release claims conditional. General and Server were asked in chat.

sdk-php owns the PHP repeat of the published transport control now: installed SDK 3.17.0 and native Server 3.8.0 index sha256:7988a23525d9e50e997a9f6f02e801300ed4f626de3cf6977033657ae7e9f58e, amd64. One isolated SQLite runtime and one bounded PHP runner, synthetic completion task as in Python/Rust. This distinguishes actual typed-413 delivery from mock coverage; it is not full fan-out conformance. Results and exact toolchain follow here; server#1136 remains the response-delivery owner.

PHP published-artifact result, 2026-11-13 approximately 09:41–09:48 UTC. Owner sdk-php. Tuple: Packagist SDK 3.17.0, source/dist reference daa842c277059471ff246803a8272165b4097c52; native Server 3.8.0, image index sha256:7988a23525d9e50e997a9f6f02e801300ed4f626de3cf6977033657ae7e9f58e; official PHP 8.3.36 image php@sha256:628102249d93494233b54490b785f12db9cd60f4dc89416dfca15973f2125f10, amd64. SDK loaded only from a separate installed consumer, with no source overlay. Disposable SQLite runtime and bounded PHP runner; all containers/volumes removed, diagnostics retained at most 24 hours.

Direct transport control PASS. Client::completeWorkflowTask() with the same synthetic opaque commands used in Python/Rust sends 36,005,171 bytes and receives typed WorkflowTaskCompletionTooLarge, HTTP 413, payload_too_large, limit 33,554,432 bytes, in 0.115 s. A 3,005,171-byte control reaches ordinary HTTP 422 command validation. This proves PHP receives the actual refusal; it does not prove complete fan-out behavior or the worker’s follow-up command.
Real 300 × 10 KB Worker fan-out PRODUCT FAIL before completion HTTP. Valid deferActivity() members are refused locally by RuntimePayloadUploads with ExternalPayloadException: Namespace runtime external payload storage is not available. Default native discovery advertises limits.max_payload_bytes = 2097152, external storage status: unconfigured, and no workflow-completion aggregate limit. The PHP planner applies that ordinary 2 MiB request limit to completion batches and demands offload even though native accepts a 32 MiB completion. The worker exits before scheduling the activities. Adding the documented synthetic server key did not change the unconfigured storage status; that correction did not produce a pass. The larger valid Worker case was not reached.

The PHP slice remains incomplete. Next action: sdk-php owns a scoped completion-planning correction and regression coverage; Server owns an additive discovery field for the actual completion aggregate limit (proposed limits.max_workflow_task_completion_bytes) alongside its response-delivery correction in server#1136. Server/General are being asked to confirm that field and the unavailable-storage fallback before implementation. Then PHP repeats both installed Worker cases and the one-time structural failure case on the exact published tuple. Native’s Python/Rust early-response defect remains independently valid; PHP’s successful direct transport control does not close it.

Published feature acceptance classification: PRODUCT-FAIL, owned by Rust SDK #254.

The registry-only 4.15.0 consumer against published native Server 3.8.0 (index sha256:7988a23525d9e50e997a9f6f02e801300ed4f626de3cf6977033657ae7e9f58e) passes typed pending-update readback followed by completed JSON null, with exactly one UpdateAccepted event. Ordinary installed lifecycle/deadline checks also pass.

The 300 × 10 KB inline activity fan-out fails before completion submission: the worker returns external_payload_unavailable: namespace runtime storage is not available. The Rust planner uses ordinary limits.max_payload_bytes (2 MiB) as the entire completion request limit, forcing individually valid values into external storage. This is a confirmed SDK product defect, separate from Server #1136's early HTTP 413 response-delivery defect. Neither failed cell is passing acceptance.

Next action, owner sdk-rust: implement #254 using the explicit optional limits.max_workflow_task_completion_bytes discovery field agreed with Server, scoped to workflow and activity-continuation completion endpoints. Preserve per-value/ordinary limits and upload authority. Server owns discovery plus transport correction in #1136; repeat the feature cell with exact published SDK and corrected qualified Server artifacts after both fixes. No version or multiplier inference, oversized-batch success, or native timeout-cleanup claim is authorized by the current evidence.

PR #255 merged at 3e8b28299528b0d49159632309f13a1ccdec2ad5 after Security's exact-head PASS and all required checks. The remote head is deleted and confirmed absent. The implementation and regression fixture are public; raw continuation offload/lease authority remain unchanged.

Owner sdk-rust. Release draft #256 prepares 4.16.0 with this fix and revision-2 metrics. Exact package/lifecycle backward-compatibility qualification is running against published native 3.8, which cannot qualify the new negotiated-cap fan-out path. Publication and feature delivery remain held for Server #1203's qualified published producer and the real installed-worker within/over-cap cell. Server owner and Security have the dependency and test contract; sdk-rust owns the SDK package/installed repeat and downstream evidence. This issue stays open.

The release dependency is now the combined corrected native Server 3.9.1 cohort (#1202 + #1203), as confirmed by the Server owner and Security. The tentative 3.9.2 split is superseded. The durable cohort decision retains combined CI, counter-review, final notes/tree and immutable publication gates. This is an unreleased plan, not a version-based capability assumption.

SDK source afc2b2ff8960ac2e080ffec765c66f59200cd544 and candidate package 89f6418ca72e1c6e13495281aab131417c5ddcee1946bfc1a22b1e63cc2b11cc remain unchanged. Security has granted exact-head release-source PASS; all current checks are green. Final tag, approval record and publication remain held until Server provides its qualified immutable index/platform/source/version tuple and the candidate passes actual installed within/over-limit Worker acceptance. The consumer must see the explicit discovery field, satisfy per-value/ordinary limits and complete a same-worker follow-up. Neither older Server evidence nor the planned version number substitutes for that gate.

Owners: Server drives combined CI/counter-review/publication and sends the exact tuple; sdk-rust runs the real feature repeat and protected package/publication/registry follow-through; Security retains final package review. Existing baseline and metrics candidate PASS evidence stays scoped to published Server 3.8.0.

Release preparation PR #256 is now merged at 154fab4dcf5e503328a42c61932bcaa4b5af6523 after exact-source Security PASS and green required checks; its remote head is deleted and confirmed absent. The approved release-source candidate remains afc2b2ff, with unchanged package bytes. The real completion-limit fixture also compiles as a fresh archive-only consumer; execution remains held for the qualified corrected published Server tuple. No tag, approval record or publication exists.

Navigation Menu

Repository navigation

Large fan-outs within documented limits block their run with a 413 #228

Problem

Acceptance criteria

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Large fan-outs within documented limits block their run with a 413 #228

Description

Problem

Acceptance criteria

Activity

rmcdaniel commented on Oct 21, 2026

rmcdaniel commented on Oct 22, 2026

rmcdaniel commented on Nov 4, 2026

rmcdaniel commented on Nov 4, 2026

rmcdaniel commented on Nov 11, 2026

rmcdaniel commented on Nov 11, 2026

rmcdaniel commented on Nov 11, 2026

rmcdaniel commented on Nov 12, 2026

rmcdaniel commented on Nov 13, 2026

rmcdaniel commented on Nov 13, 2026

rmcdaniel commented on Nov 13, 2026

rmcdaniel commented on Nov 13, 2026

rmcdaniel commented on Nov 13, 2026

rmcdaniel commented on Nov 13, 2026

rmcdaniel commented on Nov 13, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions