Repository navigation
ci(mutants): per-PR mutation testing mutates only the change, on rust-cpu, and fails when it selects nothing (#400) - #522
Merged
Conversation
…-cpu, and fails when it selects nothing over changed source (#400) The required `Mutation Testing` job ran the full spar-analysis crate (1755–1763 mutants) on the 4-runner lean-mem pool for every PR and every rebase of a strict merge queue. Measured 2026-10-20…22: 1h34–3h13 per success, one run cancelled at the 240-minute timeout. On 2026-10-22 spar held all four lean-mem runners at once (smithy). One job, two modes by event. Never two same-named jobs, because a SKIPPED `Mutation Testing` satisfies the required context. - pull_request: `cargo mutants --in-diff pr.diff --test-workspace=true` on rust-cpu, 60-minute ceiling. pr.diff is `git diff HEAD^1 HEAD` on the checked-out merge commit (fetch-depth 2), because --in-diff matches hunks against the tree it mutates: a diff of a branch that is not checked out selected 0/0/0 on three real diffs. The workspace test scope is needed because many kills come from CLI tests in another package. Measured on real queued diffs: 3–27 mutants (against 1755), and the 27-mutant worst case took 343 s locally at --jobs 4. - push to main / workflow_dispatch: the full crate with the exact 292 ratchet, on lean-mem, 240 minutes. It is the only gate that sees a change which WEAKENS a test without touching source, since that diff selects nothing. The new `workflow_dispatch:` trigger lets smithy's downtime dispatcher run it off-peak. - cargo-mutants is pinned to 27.1.0: the gate reads its report layout and its --in-diff selection, both measured against that version. check_mutants_report.py --in-diff DIFF (the per-PR gate) reads the SAME diff cargo-mutants received. With 27.1.0 a zero selection prints "No mutants to filter", exits 0 and writes no report (measured), so the report alone cannot tell "nothing to mutate" from "selected nothing over real source". The rules: - empty diff: FAIL (the diff was taken against the wrong tree); - added mutable source + no report or 0 selected: FAIL; - no added mutable source + no report: labelled "nothing to mutate", not scored; - any survivor in the diff: FAIL (zero tolerance, DEC-MUTANTS-IN-DIFF-001); - selected but nothing caught: FAIL. A deletion inside an existing function was measured to select that function's mutants. Only a pure removal selects nothing. --assert-workflow-threshold now requires BOTH declarations: exactly one in-diff gate guarded `== 'pull_request'`, and exactly one bounded ratchet guarded `!= 'pull_request'`, neither neutered. check_job_timeouts.py accepts `cond && A || B` timeouts whose A and B are integer literals, checking each bound; fromJSON and other expressions stay rejected. Oracles: - check_mutants_report self-test: 34 → 47 cases (8 in-diff, 5 workflow-shape); - check_job_timeouts self-test: 11 → 14; - check_self_test_potency: three new mutants (zero-selected-ok, one-survivor-ok, in-diff-gate-unguarded) are all CAUGHT, MIN_COVERED goes 6 → 7, and the zero-selected case is shown load-bearing (the mutant SURVIVES with that case removed). check_job_timeouts cannot be enrolled: under the harness it cannot import its sibling check_required_contexts; - the real ci.yml passes --assert-workflow-threshold, check_job_timeouts, check_required_contexts (18 = gateable set) and the other workflow guards run with CI's arguments. rivet: REQ-CI-MUTANTS-IN-DIFF-001 (v0.47.0, implemented; verified once this PR's own in-diff run is observed on a runner), and DEC-MUTANTS-IN-DIFF-001 (accepted). TEST-GUARD-GATE-EVIDENCE is updated and verifies it. The new gate is strict on lines that already carry a survivor. #512's diff touches seven untested `cmd_*` functions and gives 7 missed locally; that PR is fixed separately. Refs #400, #382 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BnBWxgVEUPdsBTLNKtrA6Z
Rivet verification gate✅ 20/20 passed
Filter: Failed artifacts(none) Updated automatically by |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The required
Mutation Testingjob ran the full spar-analysis crate(1755–1763 mutants) on the 4-runner lean-mem pool for every PR and every
rebase of a strict merge queue. Measured 2026-10-20…22: 1h34–3h13 per
success, one run cancelled at the 240-minute timeout. On 2026-10-22 spar
held all four lean-mem runners at once (smithy).
One job, two modes by event. Never two same-named jobs, because a SKIPPED
Mutation Testingsatisfies the required context.cargo mutants --in-diff pr.diff --test-workspace=trueon rust-cpu, 60-minute ceiling. pr.diff is
git diff HEAD^1 HEADonthe checked-out merge commit (fetch-depth 2), because --in-diff matches
hunks against the tree it mutates: a diff of a branch that is not
checked out selected 0/0/0 on three real diffs. The workspace test
scope is needed because many kills come from CLI tests in another
package. Measured on real queued diffs: 3–27 mutants (against 1755),
and the 27-mutant worst case took 343 s locally at --jobs 4.
ratchet, on lean-mem, 240 minutes. It is the only gate that sees a
change which WEAKENS a test without touching source, since that diff
selects nothing. The new
workflow_dispatch:trigger lets smithy'sdowntime dispatcher run it off-peak.
and its --in-diff selection, both measured against that version.
check_mutants_report.py --in-diff DIFF (the per-PR gate) reads the SAME
diff cargo-mutants received. With 27.1.0 a zero selection prints "No
mutants to filter", exits 0 and writes no report (measured), so the
report alone cannot tell "nothing to mutate" from "selected nothing over
real source". The rules:
scored;
A deletion inside an existing function was measured to select that
function's mutants. Only a pure removal selects nothing.
--assert-workflow-threshold now requires BOTH declarations: exactly one
in-diff gate guarded
== 'pull_request', and exactly one boundedratchet guarded
!= 'pull_request', neither neutered.check_job_timeouts.py accepts
cond && A || Btimeouts whose A and Bare integer literals, checking each bound; fromJSON and other
expressions stay rejected.
Oracles:
workflow-shape);
one-survivor-ok, in-diff-gate-unguarded) are all CAUGHT, MIN_COVERED
goes 6 → 7, and the zero-selected case is shown load-bearing (the
mutant SURVIVES with that case removed). check_job_timeouts cannot be
enrolled: under the harness it cannot import its sibling
check_required_contexts;
check_required_contexts (18 = gateable set) and the other workflow
guards run with CI's arguments.
rivet: REQ-CI-MUTANTS-IN-DIFF-001 (v0.47.0, implemented; verified once
this PR's own in-diff run is observed on a runner), and
DEC-MUTANTS-IN-DIFF-001 (accepted). TEST-GUARD-GATE-EVIDENCE is updated
and verifies it.
The new gate is strict on lines that already carry a survivor. #512's
diff touches seven untested
cmd_*functions and gives 7 missedlocally; that PR is fixed separately.
Refs #400, #382
Merge timing: smithy asked that this land Fri–Tue (mid-week, the PR's own CI adds to the queue it fixes). Its own
Mutation Testingrun uses this branch's workflow, so it is the cheap in-diff path on rust-cpu, and observing that run is the evidence REQ-CI-MUTANTS-IN-DIFF-001 needs before it is marked verified.🤖 Generated with Claude Code
https://claude.ai/code/session_01BnBWxgVEUPdsBTLNKtrA6Z