Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 26 additions & 20 deletions demo-rate-limiter/evidence.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,21 +56,21 @@ The writeup below, in brief:
Earlier revisions (2026-07-25, 2026-07-27) were autonomous and are still
unapproved; treat them as the weaker part of the spec.
- Independent verification: **not performed against the final source state
`3bf9a2e`.** Six earlier rounds were performed; the last verified state
`ec89de1`.** Six earlier rounds were performed; the last verified state
`d0b506c` returned `failed`, and the fixes made since — one of them
behavioural — are disclosed below as unverified. This report is finalized as
a **declared downgrade**, not on the strength of a passing verdict. A
verdict attaches to the state a verifier actually saw, and no verifier has
seen this one.
- Source state: source commit `3bf9a2e`; sha256 tree hash
`0f6d9e8aa98315ec` — reproduce both with `./tools/source_state.sh` from any
- Source state: source commit `ec89de1`; sha256 tree hash
`ffda7020f2c961fc` — reproduce both with `./tools/source_state.sh` from any
directory, or read them from `gauntlet-stamp.txt`, which the entry point
writes on every exit path. When a binding is produced the tree hash is the required content
identity; the source commit is provenance and is supplied only where
complete history is available, so a shallow checkout reports
`(unavailable: shallow history)` and a no-Git archive reports `(no git)`,
both alongside this same tree hash. No error path emits a binding at all.
The script separately reports current HEAD; commits after `3bf9a2e` that
The script separately reports current HEAD; commits after `ec89de1` that
touch only this report or other out-of-scope paths preserve the source
commit and tree hash. The manifest includes `.github/workflows`, which
decides whether the gauntlet runs in CI at all.
Expand All @@ -80,29 +80,35 @@ The writeup below, in brief:
SHAs, so none of them can be checked against the current state. The
`evidence-binding` layer therefore holds this report below a bare `PASSED`
mechanically, which is the verdict it already declared for its own reasons.
- Toolchain: pinned in `requirements-dev.txt`. Two Pythons, both run against
this state: locally on 3.14.7, and on 3.12.14 in CI via
`.github/workflows/gauntlet.yml`. **The CI run derived this report's exact
binding**, tree `0f6d9e8aa98315ec`, and reported all 21 layers green:
<https://github.com/drmikecrowe/old-coder/actions/runs/34606856991>. Its HEAD
was `ad2b4a0`.

That run is credited only because it was checked, on this repository, against
this tree. The run before it went red, and the failure was real rather than
environmental: CI's `shellcheck` flags a pattern the author's 0.11.0 does not,
so `shell-lint` was passing locally on code CI refused. The local gate was the
weaker of the two. Nothing here was proven on one machine and assumed on the
other.
- Toolchain: pinned in `requirements-dev.txt`. Locally on 3.14.7 at the current
binding, and on 3.12.14 in CI via `.github/workflows/gauntlet.yml`. **CI
verified an earlier binding**, tree `0f6d9e8aa98315ec` at HEAD `ad2b4a0`, and
reported all 21 layers green:
<https://github.com/drmikecrowe/old-coder/actions/runs/34606856991>. The
hooks-tier tooling landed since (A2 object H3, in `demo-rate-limiter/tools`)
moved the tree to `ffda7020f2c961fc`. The local gauntlet is green at that
tree on 3.14.7; **no CI run has verified it yet.** This report is bound to the
current tree and its verification for that tree is local-only, which the
declared downgrade above already accounts for.

That earlier CI run is credited only because it was checked, on this
repository, against that tree. The run before it went red, and the failure was
real rather than environmental: CI's `shellcheck` flags a pattern the author's
0.11.0 does not, so `shell-lint` was passing locally on code CI refused. The
local gate was the weaker of the two. For that binding, nothing was proven on
one machine and assumed on the other; the current binding is the stated
exception, green on 3.14.7 alone until CI reruns.

- Entry point: `./tools/gauntlet.sh` reruns every layer below and writes
`gauntlet-stamp.txt`: the result, the layer sets, a UTC timestamp, and the
source binding — on the failure path too, with the exit status
distinguishing a layer verdict (2) from an orchestration failure (3) and a
crash (passed through).

All numbers are from one final fresh run of the entry point, executed
2026-09-11 at source commit `3bf9a2e` after the last code edit; the stamp
from that run reads `result: green` over the binding above.
All numbers are from a fresh run of the entry point, executed 2026-09-13 at
source commit `ec89de1` (tree `ffda7020f2c961fc`) after the hooks-tier tooling
landed; the stamp from that run reads `result: green` over the binding above,
on 3.14.7. The 2026-09-11 run at `3bf9a2e` was the last one CI also verified.

The branch carrying that state was merged to fork `main` as `8e2b2c2` on
2026-09-09. The entry point was rerun at that HEAD: every layer green, and the
Expand Down
3 changes: 2 additions & 1 deletion demo-rate-limiter/spec.md
Original file line number Diff line number Diff line change
Expand Up @@ -303,12 +303,13 @@ can tell a layer that failed from a layer that was never wired up.
| `mutation` | tests that assert nothing | every mutant in the committed table is killed |
| `real-execution` | a suite that passes against a fake clock only | `examples/demo.py` runs against the real clock and exits 0 |
| `audit-sweep` | an audit row crediting a bound its agent's tools cannot hold | 0 unsupported claims |
| `audit-sweep-controls` | a sweep whose hooks-tier lift credits any agent that mentions hooks | 12 cases pass, including a wildcard matcher, a shell tool, and an exact hook with no recorded probe |
| `audit-sweep-controls` | a sweep whose hooks-tier lift credits any agent that mentions hooks | 24 cases pass, including a wildcard matcher, an exact hook with no recorded probe, and one per condition of the shell lift |
| `ceiling-ids` | a published ceiling that has drifted from the audit | 0 disagreements in either direction |
| `contract-ids` | this contract drifting from the harness it describes | 0 disagreements in either direction |
| `hooks-registered` | a frontmatter hook whose handler was renamed, deleted, or left non-executable, which fails open silently | 0 unusable hooks; this is the CI half and is not the proof that the host calls them |
| `hooks-registered-controls` | a registration check that reddens because a host declined an opt-in tier, or that misses a deleted handler | 9 cases pass, including a bare environment and a command interpolating a variable the runtime does not substitute |
| `hook-controls` | a hook handler that allows what it claims to deny | 12 cases pass, symlink escape and empty payload included |
| `bash-grammar-controls` | a reviewer that can rewrite what it reviews, and a grammar that denies the reads it was measured against | 124 cases pass, one per excluded class and the whole 57-row harvest; the suite is then run against a permissive stub and must fail |
| `source-state` | a report bound to a state nobody can return to | a binding is produced, or a named reason why it is not |
| `evidence-binding` | a report whose numbers came from a different tree | the report's tree hash equals the derived one, and a stale review round does not sit under a bare `PASSED` |

Expand Down
11 changes: 11 additions & 0 deletions demo-rate-limiter/tools/gauntlet.sh
Original file line number Diff line number Diff line change
Expand Up @@ -96,6 +96,17 @@ run_layer hooks-registered "$PY/python" ../tools/hooks_registered.py
run_layer hooks-registered-controls env PYTHON="$(pwd)/$PY/python" sh ../tools/test_hooks_registered.sh
run_layer hook-controls sh ../hooks/test_spec_intent_scope.sh

# Two invocations, and the second is the layer's whole worth. The suite's cases
# all pass against a handler that allows everything, except the ones that do
# not, and `--stub` is what proves the difference. A grammar suite that stayed
# green against a permissive handler would report a bound while measuring
# nothing.
layer_bash_grammar_controls() {
sh ../hooks/test_adversary_bash_grammar.sh || return $?
sh ../hooks/test_adversary_bash_grammar.sh --stub || return $?
}
run_layer bash-grammar-controls layer_bash_grammar_controls

run_layer source-state tools/source_state.sh

# Last, and after source-state on purpose: it grades evidence.md against the
Expand Down
2 changes: 1 addition & 1 deletion demo-rate-limiter/tools/gauntlet_layers.sh
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
#!/bin/sh
# Fail-closed execution and completion accounting for the gauntlet entry point.
GAUNTLET_EXPECTED_LAYERS="orchestration-self-test checker-self-test source-state-self-test tests-coverage types lint-format shell-lint supply-chain must-not-scans mutation-control mutation real-execution audit-sweep audit-sweep-controls ceiling-ids contract-ids hooks-registered hooks-registered-controls hook-controls source-state evidence-binding"
GAUNTLET_EXPECTED_LAYERS="orchestration-self-test checker-self-test source-state-self-test tests-coverage types lint-format shell-lint supply-chain must-not-scans mutation-control mutation real-execution audit-sweep audit-sweep-controls ceiling-ids contract-ids hooks-registered hooks-registered-controls hook-controls bash-grammar-controls source-state evidence-binding"
GAUNTLET_COMPLETED_LAYERS=""

# State the exit trap classifies from. run_layer and finish_gauntlet keep
Expand Down
Loading
Loading