From e74bb154929692016af10def969bdfaaf942f5f7 Mon Sep 17 00:00:00 2001 From: Richard McDaniel <1130888+rmcdaniel@users.noreply.github.com> Date: Fri, 13 Nov 2026 10:55:13 +0000 Subject: [PATCH 1/3] Prepare native 3.9.1 correction notes and Compose default --- docker-compose.published.yml | 2 +- docs/releases/3.9.1.md | 230 +++++++++++++++++++++++++++++++++++ 2 files changed, 231 insertions(+), 1 deletion(-) create mode 100644 docs/releases/3.9.1.md diff --git a/docker-compose.published.yml b/docker-compose.published.yml index 3282f6e50..3e1481089 100644 --- a/docker-compose.published.yml +++ b/docker-compose.published.yml @@ -16,7 +16,7 @@ name: durable-workflow-server # release whose default names another image). Set DW_SERVER_IMAGE to run # any other native image, by tag or digest; pin a digest for production # change control. -x-server-image: &server-image ${DW_SERVER_IMAGE:-durableworkflow/server:3.9.0} +x-server-image: &server-image ${DW_SERVER_IMAGE:-durableworkflow/server:3.9.1} services: server: diff --git a/docs/releases/3.9.1.md b/docs/releases/3.9.1.md new file mode 100644 index 000000000..f6294e253 --- /dev/null +++ b/docs/releases/3.9.1.md @@ -0,0 +1,230 @@ +# Durable Workflow Server 3.9.1 (native) + +Server 3.9.1 corrects legacy signal/update request-key readback and ambiguous +identity handling. Upgrades from 3.8.x also gain durable command keys, run-close +settlement of open activities/timers and storage admission. Two inherited changes +affect behavior a client can observe; see +[Behavior changes](#behavior-changes). Its exact images are +`durableworkflow/server:3.9.1` and `ghcr.io/durable-workflow/server:3.9.1`; +the published Compose default selects 3.9.1. Rolling aliases move separately +after published-image qualification. + +This patch corrects legacy request-key readback and ambiguous identity handling +from the immutable 3.9.0 image. The 3.9.0 rolling aliases and final GitHub +Release were held for this correction. Use 3.9.1 for the corrected contract; +3.9.0 tags are unchanged. The changes inherited from 3.8.x are included below. + +## Upgrade + +**From 3.9.0, no additional schema step is needed.** Stop the Server replicas, +back up as usual, select the 3.9.1 image, restart and verify readiness and the +reported version. The physical schema and `dw_command_intents` contents are +unchanged. The schema transition below applies when coming from 3.8.x or earlier. + +**Schema upgrade required on every backend.** This release adds one +native-only table, `dw_command_intents`, which is created empty: + +| Backend | Schema | +|------------|---------| +| SQLite | 12 → 13 | +| PostgreSQL | 13 → 14 | +| MariaDB | 13 → 14 | +| MySQL | 14 → 15 | + +The step only creates the table; it rewrites no rows and takes seconds. A +server never upgrades its schema at startup. Started on the earlier schema, +3.9.1 refuses with `native_schema_upgrade_required`. + +For the single-node Compose deployment: + +1. Stop the Server and leave MySQL running: + + ```bash + docker compose --env-file durable-workflow.prod.env \ + -f docker-compose.published.yml stop server + ``` +2. Back up MySQL and `server_storage`, and record the current image + reference. This backup is the only way back to 3.8.x. +3. Change `DW_SERVER_IMAGE` to the 3.9.1 image, pull it and run + `schema-upgrade` once: + + ```bash + docker compose --env-file durable-workflow.prod.env \ + -f docker-compose.published.yml pull server + docker compose --env-file durable-workflow.prod.env \ + -f docker-compose.published.yml run --rm --no-deps server schema-upgrade + ``` + + From 3.8.x on MySQL it reports + `"from_schema_version":14,"schema_version":15,"status":"upgraded"`. Run + again, it reports `"status":"current"`. +4. Start the Server with + `docker compose --env-file durable-workflow.prod.env -f docker-compose.published.yml up -d --wait`, + then confirm `/api/ready`, `/api/cluster/info` (`version` is `3.9.1`) and + worker registration. + +With several Servers on one database, stop every one of them before step 3 +and start them all on 3.9.1 afterwards. On MySQL and MariaDB, an interrupted +upgrade resumes when you run `schema-upgrade` again; SQLite and PostgreSQL +upgrade in one transaction. + +**Rolling back to 3.8.x requires restoring the backup.** 3.8.0 refuses the +upgraded database with `native_schema_newer_than_binary`. + +**From earlier releases,** the 3.9.1 image's `schema-upgrade` applies every +earlier step in the same run. From 3.7.x that includes the +[3.8.0 step](https://github.com/durable-workflow/server/releases/tag/3.8.0). On MySQL from 3.6.x or earlier (schema 12 or +below), it includes the [3.7.0 JSON rebuild](https://github.com/durable-workflow/server/releases/tag/3.7.0), with its +downtime of about 2 minutes per million history rows. + +On Kubernetes, scale the Server to zero, back up, run `helm upgrade` with +`native.schemaUpgrade.enabled=true` so the 3.9.1 image runs `schema-upgrade` +once, then scale back up. Chart 1.7.0 is tracked in +[PR #1199](https://github.com/durable-workflow/server/pull/1199) and will default +to Server 3.9.1 after image qualification. Its release-specific instructions +will appear in the chart's [upgrade notes](../../k8s/helm/durable-workflow/docs/UPGRADING.md). + +The worker protocol stays at `1.20` and the control plane at version `2`. + +## Behavior changes + +### A signal or update `request_id` is a durable command key + +A signal or update `request_id` now names one command on its workflow +instance ([#934](https://github.com/durable-workflow/server/issues/934)). +Cluster info advertises `capabilities.command_request_keys: true`. + +- A retry with the same command intent returns the original receipt, + including its `command_id`, `run_id` and, for an update, `update_id`, and + writes nothing. The receipt is found before any run gate, so a retry after + the run closed or continued as new still gets it. The intent is Core's + CommandIntent v1 digest over the command type, name, typed arguments and + explicit run target, so `1`, `1.0` and `"1"` differ and map key order does + not. +- The same key with a different name, arguments or run target is refused with + `409`, `reason: "request_id_conflict"` and + `outcome: "rejected_request_id_conflict"`. The response names the differing + part in `conflict` and the original command in `retained_command_id`. + Nothing is written. **Do not retry this 409 unchanged:** it means the key + was reused for different input. Send the original input to recover the + receipt, or use a new key. +- A receipt retained from before intent recording is found by its exact raw + request key as well as by the modern hashed identity. A compatible retry + returns its original command/run/update identity with `retained: true` and + `intent_verified: false`; it writes nothing and does not claim a retry's + external-payload reference. Changed arguments cannot be proved equal for + that legacy receipt. A changed operation name or explicit run target is + refused with `request_id_conflict`, rather than suggesting that a new target + was honored. Modern retained update receipts report `intent_verified: true`. +- A raw/token collision or a match that only exists because of a database's + case-insensitive collation is refused with `409 request_id_ambiguous`, + `retryable: false`, `retained: true` and `intent_verified: false`. No receipt + is selected and nothing is written. Inspect the original command before + deciding whether another key is appropriate. +- New command keys are stored hashed and their intents as digests; the + intent table stores no argument values. Existing raw-key rows are not + rewritten. These records live as long as their command; retention removes + them with the run's details. + +**This differs from PHP 2.x,** which recorded a second signal for a repeated +signal `request_id` and returned the original update receipt for a retry with +different arguments. Clients that reuse one `request_id` for different input +get the 409 on native. sdk-php 3.17.0 surfaces it as a typed conflict +(`SignalFailed::conflict()` / `UpdateFailed::conflict()`), and +sdk-rust 4.15.0 reports the key it sent; none of the SDKs resend on 409. See +[Behavior that differs from PHP 2.x](../rust-server-upgrade-takeover.md#behavior-that-differs-from-php-2x). + +### A run's own close settles its open activities and timers + +When a run's own code closes it with `complete_workflow`, `fail_workflow` or +`continue_as_new`, the close now cancels the activities and timers it still +has open, in the same transaction and before the terminal event, as Workflow +2.28.0 does ([#1193](https://github.com/durable-workflow/server/issues/1193), +Core workflow#1117). + +- Each open activity, such as a `select()` loser the workflow never awaited, + gets `ActivityCancelled` with `cancellation_reason: "run_closed"`. Each + pending timer, a wait's timeout timer included, gets `TimerCancelled`. + Nothing is appended after the terminal event. +- A late heartbeat or lease renewal for such an activity answers + `reason: "run_closed"` with `can_continue: false`. A late `complete` or + `fail` gets `409` with `outcome: "ignored"`, `recorded: false`, + `reason: "run_closed"` and `activity_status: "cancelled"`. Workers treat it + as final. +- Replay ignores an `ActivityCancelled` whose `cancellation_reason` is + `run_closed`; the workflow never observed it. sdk-php 3.17.0 does this, as do + sdk-python 2.17.0 and sdk-rust 4.15.0. +- Native refuses the `abandon` cancellation policy, so it never keeps an + activity running past its run's close. Children are unchanged: they follow + their parent-close policy. + +PHP 2.x left this work open, so the activity still ran and its history was +appended after the terminal event. **A workflow that needs a loser's effect +must await it before returning.** + +## Storage admission + +Native now honors the operator storage-pressure observation that PHP Server +documents ([#951](https://github.com/durable-workflow/server/issues/951)). +It is off unless `DW_STORAGE_ADMISSION_FILE`, `DW_STORAGE_ADMISSION_SOURCE` and +`DW_STORAGE_ADMISSION_MAX_AGE_SECONDS` (1 to 300) are set. + +- `normal` admits everything. `draining` refuses starts, signals, updates, + queries, registrations, uploads, polls and other new work with `503`, + `Retry-After: 5` and `storage_pressure`, while existing leases still + complete, fail, heartbeat and renew. `fenced`, and any missing, malformed, + stale, wrong-source or untrusted observation, keeps only reads. +- A long poll that is already waiting answers the poll refusal at its next + probe. The maintenance loop starts no pass under pressure. +- `/api/ready` reports pressure as not ready; `/api/health` keeps its + database meaning. Refusals never expose the file path or source. +- The file is opened without following symlinks and checked on the same + descriptor. Native does not yet serve the draining completion-upload + allowance, so an upload is refused like other new work; a worker keeps its + lease and uploads after pressure clears. + +See [Storage Admission](../storage-admission.md#native-server). + +## Other changes + +- Worker command names longer than their database columns are refused with + the typed `422 validation_failed` naming `commands.N.`, before any + write ([#607](https://github.com/durable-workflow/server/issues/607)): + `activity_type`, `signal_name`, `condition_key` and + `condition_definition_fingerprint` at 191 characters; `queue`, + `connection` and `workflow_type` at 255; a child's `workflow_type` and + `queue` at 191. Before, MySQL and PostgreSQL failed such a completion with a + database error and the task retried, while **SQLite stored the over-long + name**; on SQLite such a completion is now refused. +- `DW_MAX_HISTORY_EVENTS` has no effect, and the reference now says so + ([#504](https://github.com/durable-workflow/server/issues/504)). The Server + does not send `invocable_http` activity requests, and the reference now says + so too ([#527](https://github.com/durable-workflow/server/issues/527)). +- Native GitHub Releases carry the tagged release notes after the identity + table ([#1111](https://github.com/durable-workflow/server/issues/1111)). +- Tests now cover numeric-string names, selected-build routing and bounded + observation for declared worker command contracts + ([#436](https://github.com/durable-workflow/server/issues/436)). +- The two unproved APT paths in conformance and performance images run with + APT retries off (durable-workflow/.github#295). + +## SDKs + +- sdk-php 3.17.0 is out: typed `request_id` conflicts, replay that ignores + `run_closed` cancellations, and a one-time structural-limit failure for an + oversized completion. +- sdk-python 2.17.0 and sdk-rust 4.15.0 add the same handling: + `run_closed` replay, the structural-limit failure, and, in sdk-rust, the + key it sent on a `request_id` conflict. + +## Remaining completion-transport work + +The oversized-completion HTTP response correction and accurate completion-budget +discovery are tracked separately in [#1136](https://github.com/durable-workflow/server/issues/1136) +and [shared #228](https://github.com/durable-workflow/.github/issues/228). +They are not included in 3.9.1. Published Python/Rust clients can still observe +a transport error instead of a typed 413 while an oversized body is being sent; +PHP/Rust completion planners can apply the ordinary limit to a valid aggregate +and require unavailable external storage. Source changes or ordinary small +fan-out successes do not establish installed over-limit acceptance. Use the +qualified corrected producer/SDK tuple recorded on those issues when available. From 87d826752a0e74f1f9026c650d4d3f8d27214671 Mon Sep 17 00:00:00 2001 From: Richard McDaniel <1130888+rmcdaniel@users.noreply.github.com> Date: Fri, 13 Nov 2026 11:16:18 +0000 Subject: [PATCH 2/3] Describe the combined 3.9.1 completion correction --- docs/releases/3.9.1.md | 59 ++++++++++++++++++++++++++++++++---------- 1 file changed, 45 insertions(+), 14 deletions(-) diff --git a/docs/releases/3.9.1.md b/docs/releases/3.9.1.md index f6294e253..8f24afb6d 100644 --- a/docs/releases/3.9.1.md +++ b/docs/releases/3.9.1.md @@ -1,7 +1,8 @@ # Durable Workflow Server 3.9.1 (native) Server 3.9.1 corrects legacy signal/update request-key readback and ambiguous -identity handling. Upgrades from 3.8.x also gain durable command keys, run-close +identity handling, preserves typed oversized-completion refusals, and advertises +the actual completion wire budget. Upgrades from 3.8.x also gain durable command keys, run-close settlement of open activities/timers and storage admission. Two inherited changes affect behavior a client can observe; see [Behavior changes](#behavior-changes). Its exact images are @@ -9,8 +10,8 @@ affect behavior a client can observe; see the published Compose default selects 3.9.1. Rolling aliases move separately after published-image qualification. -This patch corrects legacy request-key readback and ambiguous identity handling -from the immutable 3.9.0 image. The 3.9.0 rolling aliases and final GitHub +This patch corrects legacy request-key readback, ambiguous identity handling +and completion transport/discovery from the immutable 3.9.0 image. The 3.9.0 rolling aliases and final GitHub Release were held for this correction. Use 3.9.1 for the corrected contract; 3.9.0 tags are unchanged. The changes inherited from 3.8.x are included below. @@ -217,14 +218,44 @@ See [Storage Admission](../storage-admission.md#native-server). `run_closed` replay, the structural-limit failure, and, in sdk-rust, the key it sent on a `request_id` conflict. -## Remaining completion-transport work - -The oversized-completion HTTP response correction and accurate completion-budget -discovery are tracked separately in [#1136](https://github.com/durable-workflow/server/issues/1136) -and [shared #228](https://github.com/durable-workflow/.github/issues/228). -They are not included in 3.9.1. Published Python/Rust clients can still observe -a transport error instead of a typed 413 while an oversized body is being sent; -PHP/Rust completion planners can apply the ordinary limit to a valid aggregate -and require unavailable external storage. Source changes or ordinary small -fan-out successes do not establish installed over-limit acceptance. Use the -qualified corrected producer/SDK tuple recorded on those issues when available. +## Completion request limits and typed refusals + +This release also corrects [#1136](https://github.com/durable-workflow/server/issues/1136): +HTTP/1 callers sending an oversized completion could get a transport error +instead of the server's typed 413 when the request reader closed during upload. +The server now keeps that reader alive for a bounded discard before returning +the original refusal and closing the connection. It discards at most 64 MiB of +remaining data for at most two seconds; uploads exceeding either bound may +still encounter a transport failure. + +`GET /api/cluster/info` advertises the optional positive integer +`limits.max_workflow_task_completion_bytes` from the same function used by HTTP +admission. It covers both workflow-task completion and activity continuation +completion. Its default is 33,554,432 bytes, measured across the complete +encoded JSON request body, including envelopes, metadata and commands. +`limits.max_payload_bytes` remains the per-value and ordinary API limit; adding +decoded argument sizes does not measure the completion wire budget. + +A completion over the cap returns HTTP 413 with `reason: payload_too_large`, +`limit_kind: workflow_task_completion_bytes` and the actual `limit`, before any +workflow state change. The shared wire vectors cover exact/over-cap requests +on both completion routes, and source controls cover fixed-length and chunked +TCP delivery, bounded discard, a valid 300×10 KiB fan-out and zero-write refusal. + +SDK planners must use the explicit positive integer for the two completion +routes, including a positive cap smaller than the ordinary limit. Absence +preserves the existing conservative fallback. Explicit malformed/null/zero/ +negative/non-integer values refuse completion planning before upload/submission; +ordinary routes ignore this unused field. The cap grants no new external-payload +upload or reference-placement authority. Activity/result-bound allowances cannot +be reused for continuation command arguments, even when bytes are identical; +continuation command arguments remain inline. + +[Shared #228](https://github.com/durable-workflow/.github/issues/228) stays open +until the PHP, Python and Rust owners qualify actual installed workers on the +exact published Server/SDK tuple, including valid within-cap batches and the +one-time structural failure/follow-up behavior for oversized batches. Server +source tests and ordinary small fan-out successes do not establish that +installed acceptance. PHP/Rust planner package updates and their published +qualification are tracked there; an older planner can still apply the ordinary +limit to an aggregate and require unavailable external storage. From df0503ae03880de1add82d40f8526a17f30bc727 Mon Sep 17 00:00:00 2001 From: Richard McDaniel <1130888+rmcdaniel@users.noreply.github.com> Date: Fri, 13 Nov 2026 11:50:29 +0000 Subject: [PATCH 3/3] Keep separate reliability and qualification limits explicit in 3.9.1 notes --- docs/releases/3.9.1.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/docs/releases/3.9.1.md b/docs/releases/3.9.1.md index 8f24afb6d..5f2d831fc 100644 --- a/docs/releases/3.9.1.md +++ b/docs/releases/3.9.1.md @@ -259,3 +259,12 @@ source tests and ordinary small fan-out successes do not establish that installed acceptance. PHP/Rust planner package updates and their published qualification are tracked there; an older planner can still apply the ordinary limit to an aggregate and require unavailable external storage. + +## Separate reliability and qualification records + +Source and native-image checks do not by themselves close the PHP reference's +[20-run corpus gate (#1194)](https://github.com/durable-workflow/server/issues/1194) +or the separate [intermittent MariaDB registration pool-timeout investigation (#1204)](https://github.com/durable-workflow/server/issues/1204). +This patch claims no underlying pool-timeout fix. Their current evidence and +remaining actions stay on those issues; neither incomplete nor +infrastructure-blocked controls count as passing qualification.