Prepare native 3.9.1 for legacy keys and completion transport

docker-compose.published.yml

@@ -16,7 +16,7 @@ name: durable-workflow-server
 # release whose default names another image). Set DW_SERVER_IMAGE to run
 # any other native image, by tag or digest; pin a digest for production
 # change control.
-x-server-image: &server-image ${DW_SERVER_IMAGE:-durableworkflow/server:3.9.0}
+x-server-image: &server-image ${DW_SERVER_IMAGE:-durableworkflow/server:3.9.1}
 
 services:
   server:

docs/releases/3.9.1.md

@@ -0,0 +1,270 @@
+# Durable Workflow Server 3.9.1 (native)
+
+Server 3.9.1 corrects legacy signal/update request-key readback and ambiguous
+identity handling, preserves typed oversized-completion refusals, and advertises
+the actual completion wire budget. Upgrades from 3.8.x also gain durable command keys, run-close
+settlement of open activities/timers and storage admission. Two inherited changes
+affect behavior a client can observe; see
+[Behavior changes](#behavior-changes). Its exact images are
+`durableworkflow/server:3.9.1` and `ghcr.io/durable-workflow/server:3.9.1`;
+the published Compose default selects 3.9.1. Rolling aliases move separately
+after published-image qualification.
+
+This patch corrects legacy request-key readback, ambiguous identity handling
+and completion transport/discovery from the immutable 3.9.0 image. The 3.9.0 rolling aliases and final GitHub
+Release were held for this correction. Use 3.9.1 for the corrected contract;
+3.9.0 tags are unchanged. The changes inherited from 3.8.x are included below.
+
+## Upgrade
+
+**From 3.9.0, no additional schema step is needed.** Stop the Server replicas,
+back up as usual, select the 3.9.1 image, restart and verify readiness and the
+reported version. The physical schema and `dw_command_intents` contents are
+unchanged. The schema transition below applies when coming from 3.8.x or earlier.
+
+**Schema upgrade required on every backend.** This release adds one
+native-only table, `dw_command_intents`, which is created empty:
+
+| Backend    | Schema  |
+|------------|---------|
+| SQLite     | 12 → 13 |
+| PostgreSQL | 13 → 14 |
+| MariaDB    | 13 → 14 |
+| MySQL      | 14 → 15 |
+
+The step only creates the table; it rewrites no rows and takes seconds. A
+server never upgrades its schema at startup. Started on the earlier schema,
+3.9.1 refuses with `native_schema_upgrade_required`.
+
+For the single-node Compose deployment:
+
+1. Stop the Server and leave MySQL running:
+
+   ```bash
+   docker compose --env-file durable-workflow.prod.env \
+     -f docker-compose.published.yml stop server
+   ```
+2. Back up MySQL and `server_storage`, and record the current image
+   reference. This backup is the only way back to 3.8.x.
+3. Change `DW_SERVER_IMAGE` to the 3.9.1 image, pull it and run
+   `schema-upgrade` once:
+
+   ```bash
+   docker compose --env-file durable-workflow.prod.env \
+     -f docker-compose.published.yml pull server
+   docker compose --env-file durable-workflow.prod.env \
+     -f docker-compose.published.yml run --rm --no-deps server schema-upgrade
+   ```
+
+   From 3.8.x on MySQL it reports
+   `"from_schema_version":14,"schema_version":15,"status":"upgraded"`. Run
+   again, it reports `"status":"current"`.
+4. Start the Server with
+   `docker compose --env-file durable-workflow.prod.env -f docker-compose.published.yml up -d --wait`,
+   then confirm `/api/ready`, `/api/cluster/info` (`version` is `3.9.1`) and
+   worker registration.
+
+With several Servers on one database, stop every one of them before step 3
+and start them all on 3.9.1 afterwards. On MySQL and MariaDB, an interrupted
+upgrade resumes when you run `schema-upgrade` again; SQLite and PostgreSQL
+upgrade in one transaction.
+
+**Rolling back to 3.8.x requires restoring the backup.** 3.8.0 refuses the
+upgraded database with `native_schema_newer_than_binary`.
+
+**From earlier releases,** the 3.9.1 image's `schema-upgrade` applies every
+earlier step in the same run. From 3.7.x that includes the
+[3.8.0 step](https://github.com/durable-workflow/server/releases/tag/3.8.0). On MySQL from 3.6.x or earlier (schema 12 or
+below), it includes the [3.7.0 JSON rebuild](https://github.com/durable-workflow/server/releases/tag/3.7.0), with its
+downtime of about 2 minutes per million history rows.
+
+On Kubernetes, scale the Server to zero, back up, run `helm upgrade` with
+`native.schemaUpgrade.enabled=true` so the 3.9.1 image runs `schema-upgrade`
+once, then scale back up. Chart 1.7.0 is tracked in
+[PR #1199](https://github.com/durable-workflow/server/pull/1199) and will default
+to Server 3.9.1 after image qualification. Its release-specific instructions
+will appear in the chart's [upgrade notes](../../k8s/helm/durable-workflow/docs/UPGRADING.md).
+
+The worker protocol stays at `1.20` and the control plane at version `2`.
+
+## Behavior changes
+
+### A signal or update `request_id` is a durable command key
+
+A signal or update `request_id` now names one command on its workflow
+instance ([#934](https://github.com/durable-workflow/server/issues/934)).
+Cluster info advertises `capabilities.command_request_keys: true`.
+
+- A retry with the same command intent returns the original receipt,
+  including its `command_id`, `run_id` and, for an update, `update_id`, and
+  writes nothing. The receipt is found before any run gate, so a retry after
+  the run closed or continued as new still gets it. The intent is Core's
+  CommandIntent v1 digest over the command type, name, typed arguments and
+  explicit run target, so `1`, `1.0` and `"1"` differ and map key order does
+  not.
+- The same key with a different name, arguments or run target is refused with
+  `409`, `reason: "request_id_conflict"` and
+  `outcome: "rejected_request_id_conflict"`. The response names the differing
+  part in `conflict` and the original command in `retained_command_id`.
+  Nothing is written. **Do not retry this 409 unchanged:** it means the key
+  was reused for different input. Send the original input to recover the
+  receipt, or use a new key.
+- A receipt retained from before intent recording is found by its exact raw
+  request key as well as by the modern hashed identity. A compatible retry
+  returns its original command/run/update identity with `retained: true` and
+  `intent_verified: false`; it writes nothing and does not claim a retry's
+  external-payload reference. Changed arguments cannot be proved equal for
+  that legacy receipt. A changed operation name or explicit run target is
+  refused with `request_id_conflict`, rather than suggesting that a new target
+  was honored. Modern retained update receipts report `intent_verified: true`.
+- A raw/token collision or a match that only exists because of a database's
+  case-insensitive collation is refused with `409 request_id_ambiguous`,
+  `retryable: false`, `retained: true` and `intent_verified: false`. No receipt
+  is selected and nothing is written. Inspect the original command before
+  deciding whether another key is appropriate.
+- New command keys are stored hashed and their intents as digests; the
+  intent table stores no argument values. Existing raw-key rows are not
+  rewritten. These records live as long as their command; retention removes
+  them with the run's details.
+
+**This differs from PHP 2.x,** which recorded a second signal for a repeated
+signal `request_id` and returned the original update receipt for a retry with
+different arguments. Clients that reuse one `request_id` for different input
+get the 409 on native. sdk-php 3.17.0 surfaces it as a typed conflict
+(`SignalFailed::conflict()` / `UpdateFailed::conflict()`), and
+sdk-rust 4.15.0 reports the key it sent; none of the SDKs resend on 409. See
+[Behavior that differs from PHP 2.x](../rust-server-upgrade-takeover.md#behavior-that-differs-from-php-2x).
+
+### A run's own close settles its open activities and timers
+
+When a run's own code closes it with `complete_workflow`, `fail_workflow` or
+`continue_as_new`, the close now cancels the activities and timers it still
+has open, in the same transaction and before the terminal event, as Workflow
+2.28.0 does ([#1193](https://github.com/durable-workflow/server/issues/1193),
+Core workflow#1117).
+
+- Each open activity, such as a `select()` loser the workflow never awaited,
+  gets `ActivityCancelled` with `cancellation_reason: "run_closed"`. Each
+  pending timer, a wait's timeout timer included, gets `TimerCancelled`.
+  Nothing is appended after the terminal event.
+- A late heartbeat or lease renewal for such an activity answers
+  `reason: "run_closed"` with `can_continue: false`. A late `complete` or
+  `fail` gets `409` with `outcome: "ignored"`, `recorded: false`,
+  `reason: "run_closed"` and `activity_status: "cancelled"`. Workers treat it
+  as final.
+- Replay ignores an `ActivityCancelled` whose `cancellation_reason` is
+  `run_closed`; the workflow never observed it. sdk-php 3.17.0 does this, as do
+  sdk-python 2.17.0 and sdk-rust 4.15.0.
+- Native refuses the `abandon` cancellation policy, so it never keeps an
+  activity running past its run's close. Children are unchanged: they follow
+  their parent-close policy.
+
+PHP 2.x left this work open, so the activity still ran and its history was
+appended after the terminal event. **A workflow that needs a loser's effect
+must await it before returning.**
+
+## Storage admission
+
+Native now honors the operator storage-pressure observation that PHP Server
+documents ([#951](https://github.com/durable-workflow/server/issues/951)).
+It is off unless `DW_STORAGE_ADMISSION_FILE`, `DW_STORAGE_ADMISSION_SOURCE` and
+`DW_STORAGE_ADMISSION_MAX_AGE_SECONDS` (1 to 300) are set.
+
+- `normal` admits everything. `draining` refuses starts, signals, updates,
+  queries, registrations, uploads, polls and other new work with `503`,
+  `Retry-After: 5` and `storage_pressure`, while existing leases still
+  complete, fail, heartbeat and renew. `fenced`, and any missing, malformed,
+  stale, wrong-source or untrusted observation, keeps only reads.
+- A long poll that is already waiting answers the poll refusal at its next
+  probe. The maintenance loop starts no pass under pressure.
+- `/api/ready` reports pressure as not ready; `/api/health` keeps its
+  database meaning. Refusals never expose the file path or source.
+- The file is opened without following symlinks and checked on the same
+  descriptor. Native does not yet serve the draining completion-upload
+  allowance, so an upload is refused like other new work; a worker keeps its
+  lease and uploads after pressure clears.
+
+See [Storage Admission](../storage-admission.md#native-server).
+
+## Other changes
+
+- Worker command names longer than their database columns are refused with
+  the typed `422 validation_failed` naming `commands.N.<field>`, before any
+  write ([#607](https://github.com/durable-workflow/server/issues/607)):
+  `activity_type`, `signal_name`, `condition_key` and
+  `condition_definition_fingerprint` at 191 characters; `queue`,
+  `connection` and `workflow_type` at 255; a child's `workflow_type` and
+  `queue` at 191. Before, MySQL and PostgreSQL failed such a completion with a
+  database error and the task retried, while **SQLite stored the over-long
+  name**; on SQLite such a completion is now refused.
+- `DW_MAX_HISTORY_EVENTS` has no effect, and the reference now says so
+  ([#504](https://github.com/durable-workflow/server/issues/504)). The Server
+  does not send `invocable_http` activity requests, and the reference now says
+  so too ([#527](https://github.com/durable-workflow/server/issues/527)).
+- Native GitHub Releases carry the tagged release notes after the identity
+  table ([#1111](https://github.com/durable-workflow/server/issues/1111)).
+- Tests now cover numeric-string names, selected-build routing and bounded
+  observation for declared worker command contracts
+  ([#436](https://github.com/durable-workflow/server/issues/436)).
+- The two unproved APT paths in conformance and performance images run with
+  APT retries off (durable-workflow/.github#295).
+
+## SDKs
+
+- sdk-php 3.17.0 is out: typed `request_id` conflicts, replay that ignores
+  `run_closed` cancellations, and a one-time structural-limit failure for an
+  oversized completion.
+- sdk-python 2.17.0 and sdk-rust 4.15.0 add the same handling:
+  `run_closed` replay, the structural-limit failure, and, in sdk-rust, the
+  key it sent on a `request_id` conflict.
+
+## Completion request limits and typed refusals
+
+This release also corrects [#1136](https://github.com/durable-workflow/server/issues/1136):
+HTTP/1 callers sending an oversized completion could get a transport error
+instead of the server's typed 413 when the request reader closed during upload.
+The server now keeps that reader alive for a bounded discard before returning
+the original refusal and closing the connection. It discards at most 64 MiB of
+remaining data for at most two seconds; uploads exceeding either bound may
+still encounter a transport failure.
+
+`GET /api/cluster/info` advertises the optional positive integer
+`limits.max_workflow_task_completion_bytes` from the same function used by HTTP
+admission. It covers both workflow-task completion and activity continuation
+completion. Its default is 33,554,432 bytes, measured across the complete
+encoded JSON request body, including envelopes, metadata and commands.
+`limits.max_payload_bytes` remains the per-value and ordinary API limit; adding
+decoded argument sizes does not measure the completion wire budget.
+
+A completion over the cap returns HTTP 413 with `reason: payload_too_large`,
+`limit_kind: workflow_task_completion_bytes` and the actual `limit`, before any
+workflow state change. The shared wire vectors cover exact/over-cap requests
+on both completion routes, and source controls cover fixed-length and chunked
+TCP delivery, bounded discard, a valid 300×10 KiB fan-out and zero-write refusal.
+
+SDK planners must use the explicit positive integer for the two completion
+routes, including a positive cap smaller than the ordinary limit. Absence
+preserves the existing conservative fallback. Explicit malformed/null/zero/
+negative/non-integer values refuse completion planning before upload/submission;
+ordinary routes ignore this unused field. The cap grants no new external-payload
+upload or reference-placement authority. Activity/result-bound allowances cannot
+be reused for continuation command arguments, even when bytes are identical;
+continuation command arguments remain inline.
+
+[Shared #228](https://github.com/durable-workflow/.github/issues/228) stays open
+until the PHP, Python and Rust owners qualify actual installed workers on the
+exact published Server/SDK tuple, including valid within-cap batches and the
+one-time structural failure/follow-up behavior for oversized batches. Server
+source tests and ordinary small fan-out successes do not establish that
+installed acceptance. PHP/Rust planner package updates and their published
+qualification are tracked there; an older planner can still apply the ordinary
+limit to an aggregate and require unavailable external storage.
+
+## Separate reliability and qualification records
+
+Source and native-image checks do not by themselves close the PHP reference's
+[20-run corpus gate (#1194)](https://github.com/durable-workflow/server/issues/1194)
+or the separate [intermittent MariaDB registration pool-timeout investigation (#1204)](https://github.com/durable-workflow/server/issues/1204).
+This patch claims no underlying pool-timeout fix. Their current evidence and
+remaining actions stay on those issues; neither incomplete nor
+infrastructure-blocked controls count as passing qualification.