Skip to content

ci(mutants): per-PR mutation testing mutates only the change, on rust-cpu, and fails when it selects nothing (#400) - #522

Merged
avrabe merged 1 commit into
mainfrom
ci/mutants-in-diff-400
Oct 23, 2026
Merged

avrabe merged 1 commit into
mainfrom
ci/mutants-in-diff-400

Conversation

@avrabe

@avrabe avrabe commented Oct 22, 2026

Copy link
Copy Markdown
Contributor

The required Mutation Testing job ran the full spar-analysis crate
(1755–1763 mutants) on the 4-runner lean-mem pool for every PR and every
rebase of a strict merge queue. Measured 2026-10-20…22: 1h34–3h13 per
success, one run cancelled at the 240-minute timeout. On 2026-10-22 spar
held all four lean-mem runners at once (smithy).

One job, two modes by event. Never two same-named jobs, because a SKIPPED
Mutation Testing satisfies the required context.

  • pull_request: cargo mutants --in-diff pr.diff --test-workspace=true
    on rust-cpu, 60-minute ceiling. pr.diff is git diff HEAD^1 HEAD on
    the checked-out merge commit (fetch-depth 2), because --in-diff matches
    hunks against the tree it mutates: a diff of a branch that is not
    checked out selected 0/0/0 on three real diffs. The workspace test
    scope is needed because many kills come from CLI tests in another
    package. Measured on real queued diffs: 3–27 mutants (against 1755),
    and the 27-mutant worst case took 343 s locally at --jobs 4.
  • push to main / workflow_dispatch: the full crate with the exact 292
    ratchet, on lean-mem, 240 minutes. It is the only gate that sees a
    change which WEAKENS a test without touching source, since that diff
    selects nothing. The new workflow_dispatch: trigger lets smithy's
    downtime dispatcher run it off-peak.
  • cargo-mutants is pinned to 27.1.0: the gate reads its report layout
    and its --in-diff selection, both measured against that version.

check_mutants_report.py --in-diff DIFF (the per-PR gate) reads the SAME
diff cargo-mutants received. With 27.1.0 a zero selection prints "No
mutants to filter", exits 0 and writes no report (measured), so the
report alone cannot tell "nothing to mutate" from "selected nothing over
real source". The rules:

  • empty diff: FAIL (the diff was taken against the wrong tree);
  • added mutable source + no report or 0 selected: FAIL;
  • no added mutable source + no report: labelled "nothing to mutate", not
    scored;
  • any survivor in the diff: FAIL (zero tolerance, DEC-MUTANTS-IN-DIFF-001);
  • selected but nothing caught: FAIL.

A deletion inside an existing function was measured to select that
function's mutants. Only a pure removal selects nothing.

--assert-workflow-threshold now requires BOTH declarations: exactly one
in-diff gate guarded == 'pull_request', and exactly one bounded
ratchet guarded != 'pull_request', neither neutered.
check_job_timeouts.py accepts cond && A || B timeouts whose A and B
are integer literals, checking each bound; fromJSON and other
expressions stay rejected.

Oracles:

  • check_mutants_report self-test: 34 → 47 cases (8 in-diff, 5
    workflow-shape);
  • check_job_timeouts self-test: 11 → 14;
  • check_self_test_potency: three new mutants (zero-selected-ok,
    one-survivor-ok, in-diff-gate-unguarded) are all CAUGHT, MIN_COVERED
    goes 6 → 7, and the zero-selected case is shown load-bearing (the
    mutant SURVIVES with that case removed). check_job_timeouts cannot be
    enrolled: under the harness it cannot import its sibling
    check_required_contexts;
  • the real ci.yml passes --assert-workflow-threshold, check_job_timeouts,
    check_required_contexts (18 = gateable set) and the other workflow
    guards run with CI's arguments.

rivet: REQ-CI-MUTANTS-IN-DIFF-001 (v0.47.0, implemented; verified once
this PR's own in-diff run is observed on a runner), and
DEC-MUTANTS-IN-DIFF-001 (accepted). TEST-GUARD-GATE-EVIDENCE is updated
and verifies it.

The new gate is strict on lines that already carry a survivor. #512's
diff touches seven untested cmd_* functions and gives 7 missed
locally; that PR is fixed separately.

Refs #400, #382

Merge timing: smithy asked that this land Fri–Tue (mid-week, the PR's own CI adds to the queue it fixes). Its own Mutation Testing run uses this branch's workflow, so it is the cheap in-diff path on rust-cpu, and observing that run is the evidence REQ-CI-MUTANTS-IN-DIFF-001 needs before it is marked verified.

🤖 Generated with Claude Code

https://claude.ai/code/session_01BnBWxgVEUPdsBTLNKtrA6Z

…-cpu, and fails when it selects nothing over changed source (#400)

The required `Mutation Testing` job ran the full spar-analysis crate
(1755–1763 mutants) on the 4-runner lean-mem pool for every PR and every
rebase of a strict merge queue. Measured 2026-10-20…22: 1h34–3h13 per
success, one run cancelled at the 240-minute timeout. On 2026-10-22 spar
held all four lean-mem runners at once (smithy).

One job, two modes by event. Never two same-named jobs, because a SKIPPED
`Mutation Testing` satisfies the required context.

- pull_request: `cargo mutants --in-diff pr.diff --test-workspace=true`
  on rust-cpu, 60-minute ceiling. pr.diff is `git diff HEAD^1 HEAD` on
  the checked-out merge commit (fetch-depth 2), because --in-diff matches
  hunks against the tree it mutates: a diff of a branch that is not
  checked out selected 0/0/0 on three real diffs. The workspace test
  scope is needed because many kills come from CLI tests in another
  package. Measured on real queued diffs: 3–27 mutants (against 1755),
  and the 27-mutant worst case took 343 s locally at --jobs 4.
- push to main / workflow_dispatch: the full crate with the exact 292
  ratchet, on lean-mem, 240 minutes. It is the only gate that sees a
  change which WEAKENS a test without touching source, since that diff
  selects nothing. The new `workflow_dispatch:` trigger lets smithy's
  downtime dispatcher run it off-peak.
- cargo-mutants is pinned to 27.1.0: the gate reads its report layout
  and its --in-diff selection, both measured against that version.

check_mutants_report.py --in-diff DIFF (the per-PR gate) reads the SAME
diff cargo-mutants received. With 27.1.0 a zero selection prints "No
mutants to filter", exits 0 and writes no report (measured), so the
report alone cannot tell "nothing to mutate" from "selected nothing over
real source". The rules:
- empty diff: FAIL (the diff was taken against the wrong tree);
- added mutable source + no report or 0 selected: FAIL;
- no added mutable source + no report: labelled "nothing to mutate", not
  scored;
- any survivor in the diff: FAIL (zero tolerance, DEC-MUTANTS-IN-DIFF-001);
- selected but nothing caught: FAIL.

A deletion inside an existing function was measured to select that
function's mutants. Only a pure removal selects nothing.

--assert-workflow-threshold now requires BOTH declarations: exactly one
in-diff gate guarded `== 'pull_request'`, and exactly one bounded
ratchet guarded `!= 'pull_request'`, neither neutered.
check_job_timeouts.py accepts `cond && A || B` timeouts whose A and B
are integer literals, checking each bound; fromJSON and other
expressions stay rejected.

Oracles:
- check_mutants_report self-test: 34 → 47 cases (8 in-diff, 5
  workflow-shape);
- check_job_timeouts self-test: 11 → 14;
- check_self_test_potency: three new mutants (zero-selected-ok,
  one-survivor-ok, in-diff-gate-unguarded) are all CAUGHT, MIN_COVERED
  goes 6 → 7, and the zero-selected case is shown load-bearing (the
  mutant SURVIVES with that case removed). check_job_timeouts cannot be
  enrolled: under the harness it cannot import its sibling
  check_required_contexts;
- the real ci.yml passes --assert-workflow-threshold, check_job_timeouts,
  check_required_contexts (18 = gateable set) and the other workflow
  guards run with CI's arguments.

rivet: REQ-CI-MUTANTS-IN-DIFF-001 (v0.47.0, implemented; verified once
this PR's own in-diff run is observed on a runner), and
DEC-MUTANTS-IN-DIFF-001 (accepted). TEST-GUARD-GATE-EVIDENCE is updated
and verifies it.

The new gate is strict on lines that already carry a survivor. #512's
diff touches seven untested `cmd_*` functions and gives 7 missed
locally; that PR is fixed separately.

Refs #400, #382

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BnBWxgVEUPdsBTLNKtrA6Z
@github-actions

Copy link
Copy Markdown

Rivet verification gate

✅ 20/20 passed

count
Passed 20
Failed 0
Skipped (no steps) 0

Filter: (and (= type "feature") (or (has-tag "v093") (has-tag "v0100")))

Failed artifacts

(none)

Updated automatically by tools/post_verification_comment.py. Source of truth: artifacts/verification.yaml.

@codecov

codecov Bot commented Oct 22, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@avrabe
avrabe merged commit d274e9e into main Oct 23, 2026
21 checks passed
@avrabe
avrabe deleted the ci/mutants-in-diff-400 branch October 23, 2026 18:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant