[nullifyshot]

tests/test_release_scripts.py's test_nightly_two_runs_at_once_end_at_the_newer_commit fails now and then on the ubuntu part 2 check (klimovich008 saw it on #2708 and #2732 today), and the same race can fail a real nightly when the Linux and Windows runs finish together: release_files.sh finish gives up with "SHA256SUMS still differs from the files after 5 uploads".

Cause

From the failing runs' logs (e.g. run 43237869157): after nightly: moved forward ... and its uploads, the run printed gh: Not Found (HTTP 404) three times, then the five-uploads error. uploaded() lists the release's files, then reads SHA256SUMS by its id; the other platform's run replaces SHA256SUMS (a new id) between those two calls, the read 404s, and || true turned that into "no SHA256SUMS". So the run uploads it again, which replaces it under the other run's read, and the two runs ping-pong until one has used its five uploads (in the tests sleep is a no-op, so the runs retry in step).

Change

• uploaded() lists and reads again (up to three times) when the read fails, so a file replaced meanwhile is read under its new id instead of counting as missing. A read that keeps failing still answers nothing, as before.
• The stand-in gh can replay that race (race: {"SHA256SUMS": n}: the next n reads of it find it replaced under a new id), and test_sums_replaced_between_listing_and_read_are_read_again checks that check passes and finish uploads nothing through two such replacements. On main's script check fails (the empty read) and finish uploads SHA256SUMS again.

Overlap: #2779 (Nitnek, draft) adds one more check after the fifth upload in finish(); this PR does not touch finish() and the two merge in either order. Both together: this one stops the ping-pong, #2779 catches a last upload not yet visible. #2626 changes other lines of finish().

Tested

• python -I -B tests/run_all.py on Windows 11 (419 passed; 5 failed, the same 5 as main on this PC: compiler headers and Python modules here). test_release_scripts.py is skipped on Windows (it needs fcntl and jq), so its run is this PR's Linux CI.
• bash -n scripts/release_files.sh.
• Not tested: the flake's own rate. It is a race; the new test pins the mechanism deterministically, and the ubuntu part 2 check on later PRs is where it would show again.

🤖 Generated with Claude Code

[nullifyshot]

The one red check here (test_linux_saves_move, ubuntu part 1/2) is not this PR: it is an order-dependent bug on main in host/hos_main.c's copy_link (an absolute link listed after the entry it names keeps the old place). Fix and a stronger test: #2832. A rerun of the failed job (needs a maintainer) should pass.

[hckrman101]

Thanks, this is a good catch and a clean fix for the flaky nightly test. Read from the diff only (I did not run anything); CI is the test evidence.

The race. uploaded() now re-lists and re-reads when the read itself fails, so a SHA256SUMS replaced under a new id between the listing and the read is picked up instead of turning into "nothing there". The retry is bounded (3 tries) and only triggers on a failed read, never on a content mismatch: a real difference between the sums and the files still comes out of cmp in finish()/check() exactly as before, so it cannot be hidden. The retry is also the only thing that changed; finish() and its 5-upload limit with the clear error are untouched.

Shell. Fine under set -euo pipefail: the read goes to a temp file, so a half-written body is never printed before a failed gh, and the [[ -n $asset ]] || break is safe. Nit, not blocking: a persistent read failure after 3 tries still answers nothing silently (as before, and finish() then reports it); a one-line stderr note there would make the logs clearer. Another nit: if assets itself fails under set -e the temp file is left behind (the other helpers use a trap or have the same shape, so I would not bother).

Test. The fake gh's race counter makes the replacement deterministic: each read of SHA256SUMS bumps the id and 404s, race: 2 needs exactly the third try, and the test checks both check and finish and that nothing uploaded SHA256SUMS. On main's script the first read 404s, || true makes it empty, so check fails and finish uploads again, as the description says. That matches the cause from the failing runs' logs (404 x3 then the five-uploads error), so it explains main's red runs too.

One residual window (non-blocking, a separate follow-up if you want it). upload() deletes the old SHA256SUMS and then POSTs the new one, so there is a short moment when no SHA256SUMS is listed at all. A run that lists in that moment still gets "none" and uploads again. This PR closes the replaced-under-a-new-id case that the CI logs show; the delete/POST gap is much narrower and is what #2779's extra check after the fifth upload would catch.

CI. ubuntu part 2/2 (where test_release_scripts runs) passes. The one red check, ubuntu part 1/2, is test_linux_saves_move, unrelated to these files; the author points at #2832 for it. Nothing here for rule 3c beyond the test failing before and passing after, which holds.

[nullifyshot]

Thanks @hckrman101 for the review and the merge. Agreed on the delete/POST window: it is narrower than the replaced-under-a-new-id case the CI logs showed, and #2779's check after the fifth upload covers it. The two nits (a stderr note after the third failed read, the temp file left when assets fails) are fair; I'll keep them for a small follow-up if the nightly test flakes again rather than open one for logs alone.