From 29249ff9166e5be8b937c5fdcf63248edef1b241 Mon Sep 17 00:00:00 2001 From: James Ross Date: Tue, 22 Sep 2026 09:21:57 -0700 Subject: [PATCH 1/4] docs: add runnable cooperating-worker integration --- CHANGELOG.md | 4 + Makefile | 2 +- README.md | 4 + examples/cooperating-workers/README.md | 103 ++++++++++++++++++ examples/cooperating-workers/demo.sh | 142 +++++++++++++++++++++++++ examples/cooperating-workers/worker.sh | 31 ++++++ test/cooperating-workers.py | 91 ++++++++++++++++ test/test.sh | 3 + 8 files changed, 379 insertions(+), 1 deletion(-) create mode 100644 examples/cooperating-workers/README.md create mode 100755 examples/cooperating-workers/demo.sh create mode 100755 examples/cooperating-workers/worker.sh create mode 100644 test/cooperating-workers.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 1a0a60c..95cd716 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,10 @@ All notable changes to this project are recorded here. The format follows Keep a ## [Unreleased] +### Added + +- A runnable cooperating-worker example (#40) acquires a path set in the mutation launcher, shows holder/note contention, allows unrelated work, and demonstrates renewal, superseded cleanup, failure cleanup and non-renewing TTL expiry. JSON receipts and behavior tests cover the golden path, an existing-output edge, and two concurrent isolated runs. The runbook distinguishes these controlled flows from unresolved #45 coherence work and defines external adoption validation as an unrun experiment. + ## [0.7.0] - 2026-09-16 ### Added diff --git a/Makefile b/Makefile index fede374..5085028 100644 --- a/Makefile +++ b/Makefile @@ -1,6 +1,6 @@ SHELL := /usr/bin/env bash # lib/*.sh are fragments of one script and only lint as the whole they build into (bin/git-locks). -SCRIPTS := bin/git-locks test/test.sh scripts/hooks/pre-commit scripts/hooks/pre-push scripts/build.sh +SCRIPTS := bin/git-locks test/test.sh examples/cooperating-workers/demo.sh examples/cooperating-workers/worker.sh scripts/hooks/pre-commit scripts/hooks/pre-push scripts/build.sh PREFIX ?= $(HOME)/.local .PHONY: build lint test test-docker install uninstall diff --git a/README.md b/README.md index 64cdd64..55d356f 100644 --- a/README.md +++ b/README.md @@ -216,6 +216,10 @@ When a claim finds an expired lock on a path it wants, it evicts the whole expir In summary, expiry is a field, not a process. A dead holder's lock is free the moment its time is up, and the next writer removes it as part of taking the path. +## A runnable cooperating-worker example + +The [two-worker example](examples/cooperating-workers/README.md) reserves a path set before launching mutation, shows a competing worker who holds it and why, and lets unrelated work finish. It also demonstrates renewal, acquisition-aware cleanup, worker failure and the TTL boundary in an isolated store. The runbook defines an external adoption experiment as unrun. Its controlled flows do not resolve the observation-coherence failures tracked by [#45](https://github.com/git-stunts/locks/issues/45). + ## Families and batches: all or nothing across locks One transaction per claim already makes a multi-path claim atomic; this section extends that to several locks at once, in two forms that share one mechanism. A child lock is tied to a parent so that the family lives and dies together, and a batch claims several independent locks in one stanza. diff --git a/examples/cooperating-workers/README.md b/examples/cooperating-workers/README.md new file mode 100644 index 0000000..b511b98 --- /dev/null +++ b/examples/cooperating-workers/README.md @@ -0,0 +1,103 @@ +# Two cooperating workers + +This example puts reservation acquisition in the code that launches a mutation. Alice reserves two generated files before her worker starts. Bob receives a refusal naming Alice and her note, then completes unrelated work while Alice remains active. The same run demonstrates renewal, acquisition-aware cleanup, a failed worker, and the TTL boundary. + +The example exercises controlled local flows. It does not resolve the mixed-observation failures in [#45](https://github.com/git-stunts/locks/issues/45). Evaluate an external runner integration after the hardening work and that correctness gate; the adoption experiment below has not been run. + +## Run it + +From this checkout, with Bash 4+ and Git available: + +```bash +./examples/cooperating-workers/demo.sh /tmp/locks-workers-review +``` + +Choose a fresh output path. An existing directory is refused before any file is overwritten. With no argument, the launcher creates a fresh temporary directory and prints its location. It uses this checkout's `bin/git-locks`; no installation, service, or package download is required. + +The launcher selects an explicit bare store at `/store.git`, uses `/work` as the workers' common artifact directory, and writes command receipts under `/receipts`. It retains those files when it finishes. The demonstration does not modify this checkout's source files or project refs. + +The expected transcript is: + +```text +Path set acquired before mutation; overlap refused; unrelated work completed; renewed acquisition released. +Superseded cleanup preserved the replacement acquisition. +Worker exit 17 propagated; its reservation was released. +TTL expired while the command remained active; automatic renewal is not provided. +Artifacts and JSONL receipts: +``` + +Inspect the evidence directly: + +```bash +cat /tmp/locks-workers-review/receipts/worker-b-refusal.jsonl +cat /tmp/locks-workers-review/receipts/before-renewal.jsonl +cat /tmp/locks-workers-review/receipts/after-renewal.jsonl +cat /tmp/locks-workers-review/receipts/replacement-survives.jsonl +cat /tmp/locks-workers-review/receipts/expired-while-running.jsonl +cat /tmp/locks-workers-review/receipts/final-doctor.jsonl +``` + +Each run records CLI `version`, selected store and checkout revision. The [recorded example](recorded-run.json) retains one observed run's CLI records and results against its named source revision. Object IDs and acquisition IDs change on later runs; compare their relationships and the actual outcomes. + +## What the launcher does + +The admission boundary is the `with` invocation in [demo.sh](demo.sh): + +```bash +"${DEMO_BIN}" with --job build --holder alice \ + --note 'regenerating API and types' --ttl 60 \ + generated/api.txt generated/types.txt -- \ + bash "${HERE}/worker.sh" worker-a build "${gate}" build +``` + +`with` obtains the whole path set before it invokes the mutation command. The worker records its admitted acquisition, writes both artifacts, and signals readiness through a gate file. Bob's launcher also uses `with`; a failed acquisition prevents Bob's mutation command from running. A prior `check` is not the admission mechanism. + +The gates control order without guessing how long a worker will take. Alice remains inside her command while Bob's conflicting launch is refused and Bob's independent launch writes `independent.txt`. The independent worker records Alice's live acquisition during its own execution. It then exits and releases its separate reservation. + +The launcher renews Alice's reservation with `extend`. The before/after records have different `record` object IDs and the same `acquisition` ID. When Alice's gate opens, her wrapper releases that original acquisition despite the renewal. Both generated paths become free. + +A separate case starts a wrapper with job name `reused`, then creates a replacement acquisition under that name. The old wrapper's cleanup reports `nothing` with `reason: superseded`; the replacement stays live and is released explicitly by its own acquisition ID. + +The failed-worker case returns status 17 after writing partial output. `with` propagates 17 and releases the reservation. Cleanup does not roll back the worker's file changes: `failed.txt` intentionally remains as partial output. + +Finally, a worker with TTL 1 waits at a gate. An observation at a simulated later clock reports the reservation expired while the command is still active. The launcher then opens the gate and lets cleanup complete. The example fixes `GIT_LOCKS_NOW=1000000` and explicitly advances one observation to `1000002`; this is a deterministic TTL demonstration, not a two-second benchmark. Real integrations should use the normal clock. `with` neither renews automatically nor terminates a command when its reservation expires. Choose a TTL appropriate to the workload and put any renewal policy in the runner. + +## Store and worktree meaning + +This example coordinates two workers accessing the same physical artifact directory. Its explicit store isolates the exercise and makes the sharing policy visible. + +In a project integration, the default separate store keeps coordination refs out of the project and is shared by linked worktrees. Reserving the same relative path across linked worktrees coordinates logical ownership; the files may be physically different. A runner must choose whether that shared logical ownership is the intended policy. Using distinct explicit stores intentionally creates independent coordination domains. Workers that should coordinate must select the same store and agree on relative path meaning. + +The reservations are cooperative. Other programs can write the files without using the launcher. These controlled runs do not prove the reader coherence or arbitrary interleaving properties tracked by #45. + +## Repeatable verification + +```bash +python3 test/cooperating-workers.py +make lint +``` + +The Python test requires the same `jsonschema` dependency as the existing suite; the example itself uses Bash and Git. `make test` runs the example test as part of the normal suite. + +The oracle parses the real command receipts and validates lifecycle JSON against the public schema. It checks acquisition before mutation, both reserved paths, refusal holder/note, absence of Bob's blocked mutation marker, unrelated progress during Alice's acquisition, renewal identity, release after renewal, superseded cleanup, status-17 cleanup, simulated expiry, and final store health. + +The golden run uses an output path containing spaces. An existing-directory case checks preservation of a sentinel file. Two complete demonstrations then run concurrently with separate stores and both must satisfy the same behavioral assertions. That is bounded stress of this example and its isolation, not arbitrary-schedule fuzzing or evidence of external adoption. + +## Adoption experiment, not yet run + +After the hardening gates, recruit one actual runner maintainer or integration user and agree on one existing generator or coding-worker task that mutates a known path set. Run it through the launch boundary above in a shared checkout or deliberately chosen artifact directory. Keep the experiment to that task and its existing workflow. + +Record the following before deciding whether to add features: + +| Question | Evidence to retain | +| --- | --- | +| Can the maintainer install and wire the launcher? | Setup minutes, commands changed, platform/runtime versions, and each obstacle. | +| Is it useful beyond the first demonstration? | Number of runs on at least three workdays and whether the maintainer chose to keep using it. | +| Is contention understandable? | The refusal shown to the user and their explanation of who held the paths, why, and what they did next. | +| Does unrelated work keep moving? | A concrete blocked path set and an unrelated task that completed during it. | +| Is lifecycle handling dependable for the task? | Renewal/expiry decisions, worker failures, interrupted runs, cleanup receipts and any unexpected artifacts. | +| Does the sharing policy fit? | Shared checkout or linked-worktree layout, selected store, and any mismatch between logical paths and physical files. | + +Ask the maintainer: "What did this refusal tell you?", "Where would you place acquisition in your launcher?", "What happened when the worker exceeded its TTL?", and "Would you keep this in the workflow next week, and why?" Record their words rather than substituting an inferred adoption score. + +A useful result would be a maintainer who completes integration, uses it repeatedly, explains contention correctly, and wants to keep it. A passing local demo alone does not supply that evidence. No external maintainer has been recruited or contacted as part of this change, and no standalone-business conclusion follows from it. diff --git a/examples/cooperating-workers/demo.sh b/examples/cooperating-workers/demo.sh new file mode 100755 index 0000000..84b5193 --- /dev/null +++ b/examples/cooperating-workers/demo.sh @@ -0,0 +1,142 @@ +#!/usr/bin/env bash +# Local integration example. It retains all artifacts in one fresh directory. +# The fixed clock makes expiry an explicit demonstration, not a timing guess. +set -euo pipefail +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +ROOT="$(cd "${HERE}/../.." && pwd)" +export DEMO_BIN="${ROOT}/bin/git-locks" +if (($# > 1)); then + printf 'usage: bash examples/cooperating-workers/demo.sh [fresh-output-directory]\n' >&2 + exit 2 +fi +if (($#)); then + output="$1" + if [[ -e "${output}" ]]; then + printf 'output directory already exists: %s\n' "${output}" >&2 + exit 2 + fi + mkdir -p "${output}" +else + output="$(mktemp -d "${TMPDIR:-/tmp}/git-locks-workers.XXXXXX")" +fi +output="$(cd "${output}" && pwd)" +mkdir -p "${output}/work/generated" "${output}/receipts" "${output}/gates" "${output}/tmp" +export DEMO_RECEIPTS="${output}/receipts" +export GIT_LOCKS_STORE="${output}/store.git" GIT_LOCKS_NOW=1000000 +export TMPDIR="${output}/tmp" +unset GIT_DIR GIT_WORK_TREE GIT_INDEX_FILE GIT_COMMON_DIR GIT_PREFIX GIT_OBJECT_DIRECTORY GIT_NAMESPACE +children=() gates=() +cleanup() { + local gate pid + for gate in "${gates[@]}"; do : >"${gate}"; done + for pid in "${children[@]}"; do wait "${pid}" 2>/dev/null || true; done +} +trap cleanup EXIT +trap 'exit 130' INT +trap 'exit 143' TERM +cd "${output}/work" +"${DEMO_BIN}" version >"${DEMO_RECEIPTS}/version.jsonl" +git -C "${ROOT}" rev-parse HEAD >"${DEMO_RECEIPTS}/source-revision.txt" +"${DEMO_BIN}" store >"${DEMO_RECEIPTS}/store.jsonl" + +await_ready() { # gate pid: wait for the admitted worker's explicit ready signal + local gate="$1" pid="$2" attempt + for ((attempt = 0; attempt < 1500; attempt++)); do + [[ ! -f "${gate}.ready" ]] || return 0 + if ! kill -0 "${pid}" 2>/dev/null; then + printf 'worker exited before signalling readiness\n' >&2 + return 1 + fi + sleep 0.02 + done + printf 'launcher timed out waiting for worker readiness\n' >&2 + return 1 +} + +expect_status() { # expected command... + local expected="$1" actual=0 + shift + "$@" || actual=$? + if ((actual != expected)); then + printf 'expected exit %s, got %s: %s\n' "${expected}" "${actual}" "$*" >&2 + return 1 + fi +} + +acquisition_from() { # VAR file: IDs emitted by this example are plain strings + local line + read -r line <"$2" + [[ "${line}" =~ \"acquisition\":\"([^\"]+)\" ]] || return 1 + printf -v "$1" '%s' "${BASH_REMATCH[1]}" +} + +# A owns both outputs before its mutation command starts. The command remains +# gated while B attempts overlap and then completes unrelated work. +gate="${output}/gates/build" +gates+=("${gate}") +"${DEMO_BIN}" with --job build --holder alice --note 'regenerating API and types' --ttl 60 \ + generated/api.txt generated/types.txt -- bash "${HERE}/worker.sh" worker-a build "${gate}" build \ + >"${DEMO_RECEIPTS}/worker-a.stdout.txt" 2>"${DEMO_RECEIPTS}/worker-a.jsonl" & +worker=$! +children+=("${worker}") +await_ready "${gate}" "${worker}" +expect_status 1 "${DEMO_BIN}" with --job competing --holder bob --note 'updating API' generated/api.txt \ + -- bash -c 'printf "blocked mutation ran\n" >blocked-ran' \ + >"${DEMO_RECEIPTS}/worker-b.stdout.txt" 2>"${DEMO_RECEIPTS}/worker-b-refusal.jsonl" +"${DEMO_BIN}" with --job independent --holder bob --note 'writing unrelated notes' independent.txt \ + -- bash "${HERE}/worker.sh" independent independent unused independent \ + >"${DEMO_RECEIPTS}/independent.stdout.txt" 2>"${DEMO_RECEIPTS}/independent.jsonl" +"${DEMO_BIN}" show --job build >"${DEMO_RECEIPTS}/before-renewal.jsonl" +"${DEMO_BIN}" extend --job build --ttl 120 >"${DEMO_RECEIPTS}/renewal.jsonl" +"${DEMO_BIN}" show --job build >"${DEMO_RECEIPTS}/after-renewal.jsonl" +: >"${gate}" +wait "${worker}" +"${DEMO_BIN}" check generated/api.txt generated/types.txt >"${DEMO_RECEIPTS}/after-worker-a.jsonl" +printf 'Path set acquired before mutation; overlap refused; unrelated work completed; renewed acquisition released.\n' + +# A later acquisition can reuse the name; the old wrapper releases only its +# own acquisition, so the replacement survives that wrapper's cleanup. +gate="${output}/gates/reused" +gates+=("${gate}") +"${DEMO_BIN}" with --job reused --holder alice reused.txt -- bash "${HERE}/worker.sh" old reused "${gate}" superseded \ + >"${DEMO_RECEIPTS}/superseded-worker.stdout.txt" 2>"${DEMO_RECEIPTS}/superseded-worker.jsonl" & +worker=$! +children+=("${worker}") +await_ready "${gate}" "${worker}" +"${DEMO_BIN}" claim --job reused --holder bob reused.txt >"${DEMO_RECEIPTS}/replacement.jsonl" +replacement_acquisition='' +acquisition_from replacement_acquisition "${DEMO_RECEIPTS}/replacement.jsonl" +: >"${gate}" +wait "${worker}" +"${DEMO_BIN}" show --job reused >"${DEMO_RECEIPTS}/replacement-survives.jsonl" +"${DEMO_BIN}" release --job reused --acquisition "${replacement_acquisition}" >"${DEMO_RECEIPTS}/replacement-release.jsonl" +printf 'Superseded cleanup preserved the replacement acquisition.\n' + +# A nonzero mutation status propagates through with while it releases the +# reservation. Reservation cleanup does not roll back partial file writes. +status=0 +"${DEMO_BIN}" with --job failing --holder alice failed.txt -- bash "${HERE}/worker.sh" failing failing unused failure \ + >"${DEMO_RECEIPTS}/failing-worker.stdout.txt" 2>"${DEMO_RECEIPTS}/failing-worker.jsonl" || status=$? +printf '%s\n' "${status}" >"${DEMO_RECEIPTS}/failing-worker.status" +[[ "${status}" == 17 ]] +"${DEMO_BIN}" check failed.txt >"${DEMO_RECEIPTS}/after-failure.jsonl" +printf 'Worker exit 17 propagated; its reservation was released.\n' + +# Advancing only the observation's test clock shows that with neither renews +# automatically nor terminates a still-running command when the TTL expires. +gate="${output}/gates/ttl" +gates+=("${gate}") +"${DEMO_BIN}" with --job short --holder alice --ttl 1 short.txt -- bash "${HERE}/worker.sh" ttl short "${gate}" ttl \ + >"${DEMO_RECEIPTS}/ttl-worker.stdout.txt" 2>"${DEMO_RECEIPTS}/ttl-worker.jsonl" & +worker=$! +children+=("${worker}") +await_ready "${gate}" "${worker}" +GIT_LOCKS_NOW=1000002 "${DEMO_BIN}" check short.txt >"${DEMO_RECEIPTS}/expired-while-running.jsonl" +kill -0 "${worker}" +printf 'yes\n' >"${DEMO_RECEIPTS}/worker-active-after-expiry.txt" +: >"${gate}" +wait "${worker}" +printf 'TTL expired while the command remained active; automatic renewal is not provided.\n' +"${DEMO_BIN}" list >"${DEMO_RECEIPTS}/final-list.jsonl" +"${DEMO_BIN}" doctor >"${DEMO_RECEIPTS}/final-doctor.jsonl" +printf 'Artifacts and JSONL receipts: %s\n' "${output}" diff --git a/examples/cooperating-workers/worker.sh b/examples/cooperating-workers/worker.sh new file mode 100755 index 0000000..3e400a2 --- /dev/null +++ b/examples/cooperating-workers/worker.sh @@ -0,0 +1,31 @@ +#!/usr/bin/env bash +# Invoked by the demo launcher through git-locks with, after admission. +set -euo pipefail +DEMO_BIN="${DEMO_BIN:?the launcher sets the executable}" +DEMO_RECEIPTS="${DEMO_RECEIPTS:?the launcher sets the receipt directory}" +label="$1" job="$2" gate="$3" mode="$4" +"${DEMO_BIN}" show --job "${job}" >"${DEMO_RECEIPTS}/${label}-entered.jsonl" +case "${mode}" in + build) + printf 'generated by alice\n' >generated/api.txt + printf 'generated by alice\n' >generated/types.txt + ;; + independent) + "${DEMO_BIN}" show --job build >"${DEMO_RECEIPTS}/independent-during.jsonl" + printf 'unrelated work completed\n' >independent.txt + exit 0 + ;; + failure) + printf 'partial output before failure\n' >failed.txt + exit 17 + ;; + superseded | ttl) ;; + *) exit 2 ;; +esac +: >"${gate}.ready" +for ((attempt = 0; attempt < 1500; attempt++)); do + [[ ! -e "${gate}" ]] || exit 0 + sleep 0.02 +done +printf 'worker timed out waiting for %s\n' "${gate}" >&2 +exit 1 diff --git a/test/cooperating-workers.py b/test/cooperating-workers.py new file mode 100644 index 0000000..ada2182 --- /dev/null +++ b/test/cooperating-workers.py @@ -0,0 +1,91 @@ +#!/usr/bin/env python3 +"""Behavior tests for the runnable demo, using its real launchers and Git store.""" + +import json +from pathlib import Path +import subprocess +import tempfile + +import jsonschema + +ROOT = Path(__file__).resolve().parents[1] +DEMO = ROOT / "examples/cooperating-workers/demo.sh" +SCHEMA = jsonschema.Draft202012Validator(json.loads((ROOT / "schema/git-locks.schema.json").read_text())) + + +def lines(root, name): + return [json.loads(line) for line in (root / "receipts" / name).read_text().splitlines() if line.strip()] + + +def verify(root): + receipts = root / "receipts" + for path in receipts.glob("*.jsonl"): + for record in lines(root, path.name): + SCHEMA.validate(record) + lifecycle = lines(root, "worker-a.jsonl") + acquired, released = lifecycle[0], lifecycle[-1] + assert acquired["event"] == "claimed" and acquired["holder"] == "alice" + assert acquired["paths"] == ["generated/api.txt", "generated/types.txt"] + assert released["event"] == "released" and released["job"] == "build" + entered = lines(root, "worker-a-entered.jsonl")[0] + assert entered["acquisition"] == acquired["acquisition"] and entered["state"] == "live" + refusals = lines(root, "worker-b-refusal.jsonl") + assert refusals and all(r["holder"] == "alice" and r["note"] == "regenerating API and types" for r in refusals) + assert not (root / "work" / "blocked-ran").exists(), "refused mutation ran" + during = lines(root, "independent-during.jsonl")[0] + assert during["state"] == "live" and during["acquisition"] == acquired["acquisition"] + assert (root / "work" / "independent.txt").read_text() == "unrelated work completed\n" + before = lines(root, "before-renewal.jsonl")[0] + after = lines(root, "after-renewal.jsonl")[0] + assert before["record"] != after["record"] + assert before["acquisition"] == after["acquisition"] == acquired["acquisition"] + assert after["expires"] > before["expires"] + assert all(r["state"] == "free" for r in lines(root, "after-worker-a.jsonl")) + for name in ("api.txt", "types.txt"): + assert (root / "work/generated" / name).read_text() == "generated by alice\n" + old = lines(root, "superseded-worker.jsonl") + replacement = lines(root, "replacement.jsonl")[0] + survivor = lines(root, "replacement-survives.jsonl")[0] + assert old[-1] == {"event": "nothing", "job": "reused", "reason": "superseded"} + assert old[0]["acquisition"] != replacement["acquisition"] == survivor["acquisition"] + assert survivor["holder"] == "bob" and survivor["state"] == "live" + assert (receipts / "failing-worker.status").read_text().strip() == "17" + assert lines(root, "failing-worker.jsonl")[-1]["event"] == "released" + assert lines(root, "after-failure.jsonl")[0]["state"] == "free" + expired = lines(root, "expired-while-running.jsonl")[0] + assert expired["state"] == "expired" and expired["remaining"] == 0 + assert (receipts / "worker-active-after-expiry.txt").read_text().strip() == "yes" + assert lines(root, "ttl-worker.jsonl")[-1]["event"] == "released" + assert lines(root, "final-list.jsonl") == [] + assert lines(root, "final-doctor.jsonl")[-1]["healthy"] is True + assert (receipts / "version.jsonl").exists() + assert (receipts / "source-revision.txt").read_text().strip() + + +def run(output): + result = subprocess.run(["bash", str(DEMO), str(output)], cwd=ROOT, text=True, capture_output=True, timeout=45) + assert result.returncode == 0, (result.returncode, result.stdout, result.stderr) + verify(output) + + +with tempfile.TemporaryDirectory(prefix="git-locks-demo-tests-") as tmp: + scratch = Path(tmp) + run(scratch / "golden path with spaces") + # Existing caller-owned output is rejected before anything is overwritten. + occupied = scratch / "occupied" + occupied.mkdir() + (occupied / "sentinel").write_text("keep me") + result = subprocess.run(["bash", str(DEMO), str(occupied)], text=True, capture_output=True, timeout=10) + assert result.returncode == 2 and (occupied / "sentinel").read_text() == "keep me" + assert sorted(p.name for p in occupied.iterdir()) == ["sentinel"] + # Two complete demonstrations overlap in time but use separate stores. + runs = [] + for index in range(2): + output = scratch / f"parallel-{index}" + proc = subprocess.Popen(["bash", str(DEMO), str(output)], cwd=ROOT, text=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE) + runs.append((proc, output)) + for proc, output in runs: + stdout, stderr = proc.communicate(timeout=60) + assert proc.returncode == 0, (proc.returncode, stdout, stderr) + verify(output) + print("cooperating-worker demo: golden path, lifecycle edges, failure/expiry/supersession and 2 concurrent isolated runs passed") diff --git a/test/test.sh b/test/test.sh index 22657a5..6d27e9e 100755 --- a/test/test.sh +++ b/test/test.sh @@ -1571,6 +1571,9 @@ check "sibling prefixes in one batch are not an overlap" "$?" "0" lines n "${out}" check "and both records claimed" "${n}" "2" +python3 "${HERE}/cooperating-workers.py" +check "runnable cooperating-worker demo and isolated stress cases" "$?" 0 + printf '\n%d passed, %d failed\n' "${PASS}" "${FAIL}" if ((FAIL > 0)); then printf 'failed: %s\n' "${FAILED[@]}" From 834733721bf67973540e956d1ae83b3dce1acc2a Mon Sep 17 00:00:00 2001 From: James Ross Date: Tue, 22 Sep 2026 09:22:41 -0700 Subject: [PATCH 2/4] docs: retain a verified cooperating-worker run --- .../cooperating-workers/recorded-run.json | 381 ++++++++++++++++++ 1 file changed, 381 insertions(+) create mode 100644 examples/cooperating-workers/recorded-run.json diff --git a/examples/cooperating-workers/recorded-run.json b/examples/cooperating-workers/recorded-run.json new file mode 100644 index 0000000..e5412d3 --- /dev/null +++ b/examples/cooperating-workers/recorded-run.json @@ -0,0 +1,381 @@ +{ + "artifacts": { + "failed.txt": "partial output before failure\n", + "generated/api.txt": "generated by alice\n", + "generated/types.txt": "generated by alice\n", + "independent.txt": "unrelated work completed\n" + }, + "blocked_mutation_ran": false, + "cli_records": { + "after-failure.jsonl": [ + { + "path": "failed.txt", + "state": "free" + } + ], + "after-renewal.jsonl": [ + { + "acquisition": "1000000-14871-1557608271", + "claimed": 1000000, + "expires": 1000120, + "holder": "alice", + "job": "build", + "note": "regenerating API and types", + "paths": [ + "generated/api.txt", + "generated/types.txt" + ], + "record": "42f460f8ca99ed329796ae3ad764a7f7de281cb1", + "remaining": 120, + "state": "live" + } + ], + "after-worker-a.jsonl": [ + { + "path": "generated/api.txt", + "state": "free" + }, + { + "path": "generated/types.txt", + "state": "free" + } + ], + "before-renewal.jsonl": [ + { + "acquisition": "1000000-14871-1557608271", + "claimed": 1000000, + "expires": 1000060, + "holder": "alice", + "job": "build", + "note": "regenerating API and types", + "paths": [ + "generated/api.txt", + "generated/types.txt" + ], + "record": "bdcd0624e1b766f4daf98c4df3853e109545d7dd", + "remaining": 60, + "state": "live" + } + ], + "expired-while-running.jsonl": [ + { + "expires": 1000001, + "holder": "alice", + "job": "short", + "path": "short.txt", + "remaining": 0, + "state": "expired" + } + ], + "failing-entered.jsonl": [ + { + "acquisition": "1000000-15967-0795531467", + "claimed": 1000000, + "expires": 1014400, + "holder": "alice", + "job": "failing", + "paths": [ + "failed.txt" + ], + "record": "e7886c03f16b440aa078757a56d9926b810cb28e", + "remaining": 14400, + "state": "live" + } + ], + "failing-worker.jsonl": [ + { + "acquisition": "1000000-15967-0795531467", + "claimed": 1000000, + "event": "claimed", + "expires": 1014400, + "holder": "alice", + "job": "failing", + "paths": [ + "failed.txt" + ], + "record": "e7886c03f16b440aa078757a56d9926b810cb28e" + }, + { + "event": "released", + "job": "failing", + "paths": 1 + } + ], + "final-doctor.jsonl": [ + { + "basis": { + "now": 1000000, + "records": 1, + "refs": 1 + }, + "checks": [ + "record-decodes", + "job-ref-name", + "path-ref-missing", + "path-ref-elsewhere", + "path-ref-orphan", + "path-ref-stray", + "parent-missing", + "parent-expired", + "parent-holder", + "family-cycle", + "sem-meta", + "sem-gen", + "sem-record", + "sem-capacity", + "unknown-ref" + ], + "event": "doctor", + "findings": 0, + "healthy": true, + "store": "/tmp/locks-workers-recorded/store.git" + } + ], + "final-list.jsonl": [], + "independent-during.jsonl": [ + { + "acquisition": "1000000-14871-1557608271", + "claimed": 1000000, + "expires": 1000060, + "holder": "alice", + "job": "build", + "note": "regenerating API and types", + "paths": [ + "generated/api.txt", + "generated/types.txt" + ], + "record": "bdcd0624e1b766f4daf98c4df3853e109545d7dd", + "remaining": 60, + "state": "live" + } + ], + "independent-entered.jsonl": [ + { + "acquisition": "1000000-15120-0805317732", + "claimed": 1000000, + "expires": 1014400, + "holder": "bob", + "job": "independent", + "note": "writing unrelated notes", + "paths": [ + "independent.txt" + ], + "record": "0aec71f60e3cca5072d22d3273ef15a02b0cff1f", + "remaining": 14400, + "state": "live" + } + ], + "independent.jsonl": [ + { + "acquisition": "1000000-15120-0805317732", + "claimed": 1000000, + "event": "claimed", + "expires": 1014400, + "holder": "bob", + "job": "independent", + "note": "writing unrelated notes", + "paths": [ + "independent.txt" + ], + "record": "0aec71f60e3cca5072d22d3273ef15a02b0cff1f" + }, + { + "event": "released", + "job": "independent", + "paths": 1 + } + ], + "old-entered.jsonl": [ + { + "acquisition": "1000000-15593-1116817115", + "claimed": 1000000, + "expires": 1014400, + "holder": "alice", + "job": "reused", + "paths": [ + "reused.txt" + ], + "record": "2d62d41f0d6a81ad8f373df01671d2484efa96aa", + "remaining": 14400, + "state": "live" + } + ], + "renewal.jsonl": [ + { + "event": "extended", + "expires": 1000120, + "job": "build" + } + ], + "replacement-release.jsonl": [ + { + "event": "released", + "job": "reused", + "paths": 1 + } + ], + "replacement-survives.jsonl": [ + { + "acquisition": "1000000-15770-0659222577", + "claimed": 1000000, + "expires": 1014400, + "holder": "bob", + "job": "reused", + "paths": [ + "reused.txt" + ], + "record": "42d4a425e282e80a90a2c37f7b5ef85a303c2ff6", + "remaining": 14400, + "state": "live" + } + ], + "replacement.jsonl": [ + { + "acquisition": "1000000-15770-0659222577", + "claimed": 1000000, + "event": "claimed", + "expires": 1014400, + "holder": "bob", + "job": "reused", + "paths": [ + "reused.txt" + ], + "record": "42d4a425e282e80a90a2c37f7b5ef85a303c2ff6" + } + ], + "store.jsonl": [ + { + "store": "/tmp/locks-workers-recorded/store.git" + } + ], + "superseded-worker.jsonl": [ + { + "acquisition": "1000000-15593-1116817115", + "claimed": 1000000, + "event": "claimed", + "expires": 1014400, + "holder": "alice", + "job": "reused", + "paths": [ + "reused.txt" + ], + "record": "2d62d41f0d6a81ad8f373df01671d2484efa96aa" + }, + { + "event": "nothing", + "job": "reused", + "reason": "superseded" + } + ], + "ttl-entered.jsonl": [ + { + "acquisition": "1000000-16154-1586421457", + "claimed": 1000000, + "expires": 1000001, + "holder": "alice", + "job": "short", + "paths": [ + "short.txt" + ], + "record": "10574b228557f532fe36fe787e6954f7c6b1b2ec", + "remaining": 1, + "state": "live" + } + ], + "ttl-worker.jsonl": [ + { + "acquisition": "1000000-16154-1586421457", + "claimed": 1000000, + "event": "claimed", + "expires": 1000001, + "holder": "alice", + "job": "short", + "paths": [ + "short.txt" + ], + "record": "10574b228557f532fe36fe787e6954f7c6b1b2ec" + }, + { + "event": "released", + "job": "short", + "paths": 1 + } + ], + "version.jsonl": [ + { + "name": "git-locks", + "version": "0.7.0" + } + ], + "worker-a-entered.jsonl": [ + { + "acquisition": "1000000-14871-1557608271", + "claimed": 1000000, + "expires": 1000060, + "holder": "alice", + "job": "build", + "note": "regenerating API and types", + "paths": [ + "generated/api.txt", + "generated/types.txt" + ], + "record": "bdcd0624e1b766f4daf98c4df3853e109545d7dd", + "remaining": 60, + "state": "live" + } + ], + "worker-a.jsonl": [ + { + "acquisition": "1000000-14871-1557608271", + "claimed": 1000000, + "event": "claimed", + "expires": 1000060, + "holder": "alice", + "job": "build", + "note": "regenerating API and types", + "paths": [ + "generated/api.txt", + "generated/types.txt" + ], + "record": "bdcd0624e1b766f4daf98c4df3853e109545d7dd" + }, + { + "event": "released", + "job": "build", + "paths": 2 + } + ], + "worker-b-refusal.jsonl": [ + { + "event": "refused", + "expires": 1000060, + "holder": "alice", + "job": "build", + "note": "regenerating API and types", + "path": "generated/api.txt" + } + ] + }, + "clock": { + "default": 1000000, + "expiry_observation": 1000002, + "kind": "simulated" + }, + "external_adoption": "not run", + "failing_worker_status": 17, + "schema_validated_records": 30, + "source_revision": "29249ff9166e5be8b937c5fdcf63248edef1b241", + "source_sha256": { + "bin/git-locks": "fc686ce320cdebba1420f2bb6e7d56f2cd79c1e40ecdd7b31178a9d2cf27aae3", + "examples/cooperating-workers/demo.sh": "0175c1ce33d97e0b7c84b0331d62d522e6a03731ed8a3227bcbc19e8d23e77f7", + "examples/cooperating-workers/worker.sh": "fdaedec546078ea505960544553372826712b816647cca4a24684bc379c9c8b0" + }, + "transcript": [ + "Path set acquired before mutation; overlap refused; unrelated work completed; renewed acquisition released.", + "Superseded cleanup preserved the replacement acquisition.", + "Worker exit 17 propagated; its reservation was released.", + "TTL expired while the command remained active; automatic renewal is not provided.", + "Artifacts and JSONL receipts: /tmp/locks-workers-recorded" + ], + "worker_active_after_expiry": true +} From f488ad76acb80341b6c42db196e0dce09fa97a89 Mon Sep 17 00:00:00 2001 From: James Ross Date: Fri, 2 Oct 2026 02:42:57 -0700 Subject: [PATCH 3/4] test: stop the whole demo session and guard the recorded run against drift A timed-out or failed demo run killed only the launching bash, orphaning gated launchers and workers that kept writing into a deleted temp dir. The demo now runs in its own session and every exit path kills the group. The retained recorded-run.json is now checked against a fresh run's receipt set and transcript, with its records schema-validated. Refs #40 --- test/cooperating-workers.py | 95 +++++++++++++++++++++++++++++++------ 1 file changed, 81 insertions(+), 14 deletions(-) diff --git a/test/cooperating-workers.py b/test/cooperating-workers.py index ada2182..3cedad3 100644 --- a/test/cooperating-workers.py +++ b/test/cooperating-workers.py @@ -2,14 +2,18 @@ """Behavior tests for the runnable demo, using its real launchers and Git store.""" import json +import os from pathlib import Path +import signal import subprocess import tempfile +import time import jsonschema ROOT = Path(__file__).resolve().parents[1] DEMO = ROOT / "examples/cooperating-workers/demo.sh" +RECORDED = ROOT / "examples/cooperating-workers/recorded-run.json" SCHEMA = jsonschema.Draft202012Validator(json.loads((ROOT / "schema/git-locks.schema.json").read_text())) @@ -17,6 +21,36 @@ def lines(root, name): return [json.loads(line) for line in (root / "receipts" / name).read_text().splitlines() if line.strip()] +def start_demo(output): + # Its own session, so a stop reaches the background launchers and gated workers too. + return subprocess.Popen(["bash", str(DEMO), str(output)], cwd=ROOT, text=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, start_new_session=True) + + +def group_alive(pgid): + try: + os.killpg(pgid, 0) + except ProcessLookupError: + return False + except PermissionError: # macOS: only exited members awaiting their reaper remain + return False + return True + + +def stop_demo(proc): + # Kill the whole session: the launcher's background workers outlive a kill of bash alone. + if group_alive(proc.pid): + os.killpg(proc.pid, signal.SIGKILL) + proc.wait() + + +def finish_demo(proc, timeout): + try: + stdout, stderr = proc.communicate(timeout=timeout) + finally: + stop_demo(proc) + return stdout, stderr + + def verify(root): receipts = root / "receipts" for path in receipts.glob("*.jsonl"): @@ -62,30 +96,63 @@ def verify(root): assert (receipts / "source-revision.txt").read_text().strip() +def verify_recorded(root, stdout): + # The retained run must still describe what the demo emits today. + recorded = json.loads(RECORDED.read_text()) + records = recorded["cli_records"] + assert sorted(records) == sorted(p.name for p in (root / "receipts").glob("*.jsonl")), "recorded receipt set drifted" + for batch in records.values(): + for record in batch: + SCHEMA.validate(record) + assert sum(len(batch) for batch in records.values()) == recorded["schema_validated_records"] + assert recorded["transcript"][:-1] == stdout.splitlines()[:-1], "recorded transcript drifted" + + def run(output): - result = subprocess.run(["bash", str(DEMO), str(output)], cwd=ROOT, text=True, capture_output=True, timeout=45) - assert result.returncode == 0, (result.returncode, result.stdout, result.stderr) + proc = start_demo(output) + stdout, stderr = finish_demo(proc, 45) + assert proc.returncode == 0, (proc.returncode, stdout, stderr) verify(output) + return stdout + + +def stopped_demo_leaves_no_workers(output): + # A stop while Alice is gated must not orphan her launcher and worker. + proc = start_demo(output) + try: + ready = output / "gates" / "build.ready" + deadline = time.monotonic() + 30 + while not ready.exists(): + assert proc.poll() is None and time.monotonic() < deadline, "demo never gated its first worker" + time.sleep(0.02) + finally: + stop_demo(proc) + deadline = time.monotonic() + 5 + while group_alive(proc.pid) and time.monotonic() < deadline: + time.sleep(0.02) + assert not group_alive(proc.pid), "stopped demo left worker processes running" with tempfile.TemporaryDirectory(prefix="git-locks-demo-tests-") as tmp: scratch = Path(tmp) - run(scratch / "golden path with spaces") + golden = scratch / "golden path with spaces" + verify_recorded(golden, run(golden)) # Existing caller-owned output is rejected before anything is overwritten. occupied = scratch / "occupied" occupied.mkdir() (occupied / "sentinel").write_text("keep me") - result = subprocess.run(["bash", str(DEMO), str(occupied)], text=True, capture_output=True, timeout=10) + result = subprocess.run(["bash", str(DEMO), str(occupied)], text=True, capture_output=True, timeout=10, check=False) assert result.returncode == 2 and (occupied / "sentinel").read_text() == "keep me" assert sorted(p.name for p in occupied.iterdir()) == ["sentinel"] + stopped_demo_leaves_no_workers(scratch / "stopped") # Two complete demonstrations overlap in time but use separate stores. - runs = [] - for index in range(2): - output = scratch / f"parallel-{index}" - proc = subprocess.Popen(["bash", str(DEMO), str(output)], cwd=ROOT, text=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE) - runs.append((proc, output)) - for proc, output in runs: - stdout, stderr = proc.communicate(timeout=60) - assert proc.returncode == 0, (proc.returncode, stdout, stderr) - verify(output) - print("cooperating-worker demo: golden path, lifecycle edges, failure/expiry/supersession and 2 concurrent isolated runs passed") + runs = [(start_demo(scratch / f"parallel-{index}"), scratch / f"parallel-{index}") for index in range(2)] + try: + for proc, output in runs: + stdout, stderr = proc.communicate(timeout=60) + assert proc.returncode == 0, (proc.returncode, stdout, stderr) + verify(output) + finally: + for proc, _ in runs: + stop_demo(proc) + print("cooperating-worker demo: golden path, recorded run, lifecycle edges, failure/expiry/supersession, stop cleanup and 2 concurrent isolated runs passed") From ebe14da54627a2103513dbad4e749cbdd23f4555 Mon Sep 17 00:00:00 2001 From: James Ross Date: Fri, 2 Oct 2026 02:43:00 -0700 Subject: [PATCH 4/4] docs: place the worker example after with, and list every with option Refs #40 --- CHANGELOG.md | 6 +++++- README.md | 12 ++++++------ examples/cooperating-workers/README.md | 2 +- 3 files changed, 12 insertions(+), 8 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 95cd716..70d7402 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,7 +6,11 @@ All notable changes to this project are recorded here. The format follows Keep a ### Added -- A runnable cooperating-worker example (#40) acquires a path set in the mutation launcher, shows holder/note contention, allows unrelated work, and demonstrates renewal, superseded cleanup, failure cleanup and non-renewing TTL expiry. JSON receipts and behavior tests cover the golden path, an existing-output edge, and two concurrent isolated runs. The runbook distinguishes these controlled flows from unresolved #45 coherence work and defines external adoption validation as an unrun experiment. +- A runnable cooperating-worker example (#40) acquires a path set in the mutation launcher, shows holder/note contention, allows unrelated work, and demonstrates renewal, superseded cleanup, failure cleanup and non-renewing TTL expiry. JSON receipts and behavior tests cover the golden path, agreement with the retained recorded run, an existing-output edge, a stopped run leaving no worker processes, and two concurrent isolated runs. The runbook distinguishes these controlled flows from unresolved #45 coherence work and defines external adoption validation as an unrun experiment. + +### Fixed + +- The README's `with` synopses now list every option the command accepts, including `--note` and `--ttl`, which the example uses. ## [0.7.0] - 2026-09-16 diff --git a/README.md b/README.md index 55d356f..635e50f 100644 --- a/README.md +++ b/README.md @@ -216,10 +216,6 @@ When a claim finds an expired lock on a path it wants, it evicts the whole expir In summary, expiry is a field, not a process. A dead holder's lock is free the moment its time is up, and the next writer removes it as part of taking the path. -## A runnable cooperating-worker example - -The [two-worker example](examples/cooperating-workers/README.md) reserves a path set before launching mutation, shows a competing worker who holds it and why, and lets unrelated work finish. It also demonstrates renewal, acquisition-aware cleanup, worker failure and the TTL boundary in an isolated store. The runbook defines an external adoption experiment as unrun. Its controlled flows do not resolve the observation-coherence failures tracked by [#45](https://github.com/git-stunts/locks/issues/45). - ## Families and batches: all or nothing across locks One transaction per claim already makes a multi-path claim atomic; this section extends that to several locks at once, in two forms that share one mechanism. A child lock is tied to a parent so that the family lives and dies together, and a batch claims several independent locks in one stanza. @@ -320,7 +316,7 @@ In summary, a semaphore is a set of slot refs plus one ref that every writer mus Most callers want the lock only for the duration of one command, and forgetting the release is the common failure. This section shows `with`, which does the three steps and cannot forget the third. -`git locks with --job --holder [--wait ] [--sem ] ... -- ...` claims the paths (and a semaphore slot if asked), runs the command, and releases on exit, on failure, and on Ctrl-C or a termination signal, then exits with the command's own status. The command owns stdout; git-locks reports its claim and release on stderr, so a pipeline reading the command's output sees only that output: +`git locks with --job --holder [--ttl ] [--wait ] [--sem ] [--note ] ... -- ...` claims the paths (and a semaphore slot if asked), runs the command, and releases on exit, on failure, and on Ctrl-C or a termination signal, then exits with the command's own status. The command owns stdout; git-locks reports its claim and release on stderr, so a pipeline reading the command's output sees only that output: ```text $ git locks with --job build --holder alice dist/bundle.js -- sh -c 'echo building' @@ -333,6 +329,10 @@ building In summary, `with` is the shape most scripts should use: the lock's lifetime is the command's lifetime, by construction. +## A runnable cooperating-worker example + +The [two-worker example](examples/cooperating-workers/README.md) reserves a path set before launching mutation, shows a competing worker who holds it and why, and lets unrelated work finish. It also demonstrates renewal, acquisition-aware cleanup, worker failure and the TTL boundary in an isolated store. The runbook defines an external adoption experiment as unrun. Its controlled flows do not resolve the observation-coherence failures tracked by [#45](https://github.com/git-stunts/locks/issues/45). + ## Output: JSON Lines, always Every example above showed one JSON object per line, and this section states the contract behind that so a consumer can rely on it. There is no plain-text mode. Stdout carries one object per result, written as each result is known; stderr carries refusals and errors as objects; `git locks help` is a `usage` object; `git locks schema` prints the schema as one line. The single exception is a command wrapped by `with`, which owns stdout while git-locks reports around it on stderr. @@ -401,7 +401,7 @@ Output is JSON Lines on every command; there is no text mode. | `claim … --parent ` | make the lock a child: the parent must be live and held by the same holder (verified inside the transaction); the child is released or swept with it | as `claim`, with `parent` | 0, 1 if refused | | `batch < records` | claim several locks in one transaction, or none; records are blank-line separated `job:`, `holder:`, `ttl:`, `parent:`, then `paths:` with one path per line | one `claimed` object per record | 0, 1 if any is refused, 2 on a malformed record | | `release --job [--record ] [--job ...]` | release the jobs and all their descendants in one transaction; `--record` releases only that acquisition | one object per job, `cascaded` lists descendants, `nothing` with `reason: superseded` when the record no longer matches | 0 | -| `with --job --holder [--ttl ] [--wait ] [--parent ] ... -- ...` | claim, run the command, release; `--wait` retries once a second until the paths are free or the wait runs out | the command's own stdout; git-locks' `claimed`, `released` and refusals go to **stderr** | the command's exit status; 1 if never acquired; 130/143 on INT/TERM after releasing | +| `with --job --holder [--ttl ] [--wait ] [--sem ] [--parent ] [--note ] ... -- ...` | claim, run the command, release; `--wait` retries once a second until the paths are free or the wait runs out | the command's own stdout; git-locks' `claimed`, `released` and refusals go to **stderr** | the command's exit status; 1 if never acquired; 130/143 on INT/TERM after releasing | | `version` | tool name and version | one object | 0 | | `schema` | the JSON Schema every line above conforms to | the schema document | 0 | | `doctor` | read-only invariant check of the store; nothing is repaired | one `finding` object per broken invariant as it is found, then one `doctor` object with the basis (refs, records, clock), the checks run and the verdict | 0 healthy, 1 with findings, 2 if the store cannot be read | diff --git a/examples/cooperating-workers/README.md b/examples/cooperating-workers/README.md index b511b98..2c06bab 100644 --- a/examples/cooperating-workers/README.md +++ b/examples/cooperating-workers/README.md @@ -81,7 +81,7 @@ The Python test requires the same `jsonschema` dependency as the existing suite; The oracle parses the real command receipts and validates lifecycle JSON against the public schema. It checks acquisition before mutation, both reserved paths, refusal holder/note, absence of Bob's blocked mutation marker, unrelated progress during Alice's acquisition, renewal identity, release after renewal, superseded cleanup, status-17 cleanup, simulated expiry, and final store health. -The golden run uses an output path containing spaces. An existing-directory case checks preservation of a sentinel file. Two complete demonstrations then run concurrently with separate stores and both must satisfy the same behavioral assertions. That is bounded stress of this example and its isolation, not arbitrary-schedule fuzzing or evidence of external adoption. +The golden run uses an output path containing spaces, and its receipt set and transcript must still match [recorded-run.json](recorded-run.json), whose records must remain schema-valid. An existing-directory case checks preservation of a sentinel file. A demonstration stopped while Alice's worker is gated must leave no launcher or worker process behind. Two complete demonstrations then run concurrently with separate stores and both must satisfy the same behavioral assertions. That is bounded stress of this example and its isolation, not arbitrary-schedule fuzzing or evidence of external adoption. ## Adoption experiment, not yet run