ci: touched packages runs in parallel legs and re-checks a failure before blaming the change

[AbirAbbas]

Why

A month of touched packages history (2 Sep – 1 Oct, 357 red runs, each classified against the commits that followed) found:

• about a third of reds were a real bug in the PR;
• about 40% were not caused by the PR at all: a test already failing on dev (12%), a flaky test (16%), or the selector itself (8%, fixed in ci: a changed directory in a nested module is not a package of the root module #1332);
• the rest were tests pinning behaviour the PR changed on purpose.

Every one of those not-your-fault reds cost a person or an agent a full investigate-and-rerun cycle. Separately, the job ran touched packages one after another, and its Go build cache had not been refreshed since 2026-09-10.

What changes

A failure is re-checked before it blames the change. Each named failing test is run once more on the head. If it passes, it is reported as flaky. If it fails again, it is run at the base commit (PR base SHA, previous push tip, or HEAD~1 on a dispatch); if it fails there too, it is reported as already failing on the base. Both warn, stay green, appear in the step summary, and get one comment on a standing issue titled touched packages: flaky or already-failing tests, so a flaky test still has an owner. Red, as before:

• a failure that passes at the base, or a test/package absent at the base;
• build and setup failures, package timeouts, panics outside a test, killed processes and shard errors (never re-run);
• more than five failed tests in one leg (never re-run).

Nothing is skipped by name and every test still runs. The log keeps the text Go would have printed (the attribution reads Go's JSON events behind it).

Parallel legs. Touched packages run as separate legs (tui3, session, codeaf, rest) on separate runners, so a change spanning the heavy packages waits for the slowest leg instead of the sum. rest uses -p 2 on the 16 GB public runner; each leg prints nproc/free -g. cmd/codeaf stays unsharded: sharding it was tried and its sharded runs failed. A job still named touched packages aggregates the legs, and check is unchanged.

A build cache that refreshes. setup-go keyed its cache on go.sum, so every run restored the snapshot from the last dependency change and logged "not saving cache". Caches are now keyed by UTC day, saved by the first dev push of the day that misses, restored by PRs from the newest one, and left to GitHub's eviction (about 2 GB a day at most, against the 10 GB limit).

One selector. scripts/touched-packages.sh is now the single selector for CI and make test-touched, which also runs the same classifier locally (no issue calls).

Rule change

docs/rules/ci.md said a failure is never retried. This replaces that with one re-run of the failing tests plus a base comparison, because a flaky or inherited failure is a real bug that belongs on the standing issue, not on the PR that happened to touch the package.

Validation

• Contract tests: scripts/touched-packages_test.sh (no-Go change, each named leg, mixed, go.mod/go.sum → whole tree, nested modules, emptied dirs) and scripts/touched-verdict_test.sh (flaky, inherited, own regression, absent test/package at base, unbuildable/skipped base, build failure, timeout, panic outside a test, over the cap, subtests, readable log, issue find-or-create/fork/API error). Both are in make test-tooling.
• make pr-ready green on this branch; YAML parses; go vet, gofmt, laws.
• Live proof on throwaway dispatch branches (flaky, already failing at base, own regression, a wide tui3+session+codeaf change) follows in a comment.
• Not verifiable before merge: cache saves/restores (only a dev push saves).

🤖 Generated with Claude Code

[AbirAbbas]

Live proof on throwaway branches (workflow_dispatch, this branch's workflow at c3f3e98)

Branch	What it does	Result
ci-proof/flaky	a test that fails on its first run in the job and passes on the re-run	green, ::warning "flaky", one issue comment
ci-proof/inherited	base commit has an always-failing test; head only touches the package	green, ::warning "already failing on the base", one issue comment
ci-proof/own	test passes at base; head breaks the function	red: fails twice on head, passes at base, ::error … introduced by this change
ci-proof/wide	comment-only change in tui3, session and cmd/codeaf	green; legs in parallel: tui3 2m06s, codeaf 3m27s, session 3m55s
ci-proof/module	comment in go.mod, so the whole tree runs	green; tui3 2m20s, codeaf 3m46s, session 3m54s, rest 12m10s; no flaky or already-failing findings

Runs: 36940090816, 36940093594, 36940096225, 36940099149, 36940101663.

All of these ran with no cache (only a dev push saves the new keys), so they are the worst case. The light gate ran 3m36s–4m36s cold; it should drop once the first dev push saves its daily cache, and I'll check that on the first PR after merge.

What the first round of live runs caught and this branch fixed before the round above:

• go test -json changes test behaviour (Go 1.26 swaps os.Stderr; helper processes inherit -test.v=test2json). TestARenameNoticeNeverReachesTheJSONOnStdout (fixed to check the descriptor) and four internal/seniordev/util process tests failed only under -json, and a -json-only failure would have been "already failing on the base" forever. The classifier now reads the plain output of the same run make test gives everyone.
• cmd/codeaf-notices runs go list with GOPROXY=off; a leg with a cold module cache lacked modules it never compiled. Each leg now runs go mod download first.

The standing issue the proofs created (#1735) is closed: its comments are proof noise, and after merge the job opens a fresh one on the first real finding.