ci(security): retry the semgrep install fail-closed (transient ResolutionImpossible reded the queue)

[wshallwshall]

What broke

Merge-group run 36832387843 (job 110271740736) failed the blocking semgrep (project SAST rules) gate at install time. pip reported ResolutionImpossible, with "no matching distributions available for your environment: pydantic-core" at every pydantic-core version from 2.33.0 to 2.46.5. The log shows a 37-second stall just before.

Root cause: a transient empty index read, not a new release

• The same semgrep==1.172.0 pin resolved pydantic-core==2.46.5 six minutes later, on the same runner image (ubuntu24/20260927.320), in job 110273573584.
• It also resolved an hour earlier on the same queue head (run 36826480823).
• It installs cleanly in a fresh local Python 3.14 venv: semgrep 1.172.0, mcp 1.23.3, pydantic-core 2.46.5.
• PyPI lists cp314 manylinux wheels for pydantic-core 2.46.5, and none are yanked.

So this was one bad read, and a re-run alone would have passed.

The fix

Both semgrep install sites (the standalone job and the repo-scan composite, still byte-identical) now retry up to three times, with a 15s and then 30s backoff, and annotate each retry with ::warning::. It stays fail-closed: after the third failure the step exits 1 and the scan never runs. semgrep==1.172.0 is unchanged, and so is the scan command.

tests/test_semgrep_install_retry.py runs the committed loop under bash -e with python and sleep stubbed. A recovered install reaches the scan. Three failures exit 1 without reaching it. Swapping exit 1 for break turns the fail-closed test red.

Why not a hash lock (the brief's first choice)

• semgrep is kept out of [dependency-groups] on purpose. ADR 0034 section 3 records the [otel] conflict, and tests/test_ci_venv_pinning.py says the same. A ci/locks/ lock would mean reopening that decision.
• A lock would not have stopped this failure. Even with --require-hashes, pip still reads the index page for every package, and that read is what came back empty.

Checks run

• pytest over every test file that mentions security.yml, semgrep or ci/locks, plus the new file: 562 passed, 1 skipped.
• Mutation arm (fall through instead of exit 1): 1 failed, 4 passed.
• ruff check (All checks passed), ruff format --check, and mypy on the new test (no issues).
• The pre-commit workflow lint ("Lint GitHub Actions workflow files"): Passed. A standalone actionlint is not installed here.
• semgrep 1.172.0 over the repo, with the job's exact args and excludes: Ran 5 rules on 435 files: 0 findings.
• code-review subagent at xhigh: three low findings, no blockers. Two are fixed in the second commit (the comment wording and the ::warning:: annotation). The third (no test pinned fail-closed) is fixed by the new test.

Legs to read on the PR: semgrep (project SAST rules) and repo-scan, which run the edited steps.

[wshallwshall]

QA: code-review subagent xhigh, head 170651c, verdict 3 low findings, no blockers. All three are addressed in 67d8b2d (the current head), which was not re-reviewed: one round, as briefed.

[wshallwshall]

QA addendum: head 2dbe7a6 adds tests/test_semgrep_install_retry.py to tests/tooling_manifest.txt (manifest-only, not re-reviewed); test_tooling_partition.py plus the new test: 43 passed.