Repository navigation
Corpus-b flake: large-command-batches overflow polls exhaust PHP reference lock-pressure retries #1194
Description
What fails
Shared Server fixtures → execution-and-codec (corpus-b) intermittently fails before comparison:
RuntimeException: Large command batches: no workflow task for select_member_overflow {"http_status":503,…,"poll_retries":15,"poll_status":"backend_lock_pressure","reason":"backend_lock_pressure",…}
The poll that large-command-batches-probe.php makes against the published PHP SQLite reference keeps getting 503 backend_lock_pressure until lockPressurePollRetryDelay (scripts/conformance/server-parity/lock-pressure-retry.php, 15 retries / 30 s) gives up. The PHP reference shares its SQLite file with its maintenance loop, and the large batches (up to 1,000 commands) appear to hold the file long enough that the budget runs out.
Evidence (merge queue, 2026-11-13; 3 of the last ~10 groups)
| Group | Run | Case |
|---|---|---|
| pr-1187 on 4c5f226 | 43345755536 job 131626581193 | select_member_overflow |
| pr-1189 on 4c5f226 | 43349737218 job 131639640905 | select_member_overflow |
| pr-1186 | 43349740361 job 131639561736 | select_base_overflow |
The PRs in those groups don't touch the corpus, the probe or the server code it exercises (#1187 is a Python benchmark script, #1189 adds a Rust test, #1186 bounds command names). Every earlier group and every main push in the same window passed. All three were re-queued unchanged.
Acceptance
- Find what holds the reference's SQLite file during the overflow cases (maintenance pass vs. the probe's own batch writes) and remove the contention at its source. For example, finish or quiesce the reference maintenance loop around the large-batch cases, or poll each case's task after its namespace's writes settle. Don't just raise the retry budget blindly.
- If the budget does change, keep it bounded and reported (
lock_pressure_poll_retrieson stderr), and keep recorders seeing only the final reply. - Native recordings are unaffected and must stay unchanged.
- Evidence: 20 consecutive green corpus-b runs, or a deterministic reproduction that the fix closes.
Activity
rmcdaniel commentedon Nov 13, 2026
Evidence on #1195 (head 5b3ff71, which terminates each large-command-batches run as soon as it's recorded): run 43353355060's
execution-and-codec (corpus-b)passed on attempts 1–5 (jobs 131651317384, 131658917257, 131662725363, 131666891832, 131670037451). Attempt 6 is running. Five greens against the observed ~3-in-10 group failure rate is suggestive, not proof (~17% chance unfixed). After merge, every merge-queue group runs corpus-b, so I'll keep counting there and close this at 20 consecutive green corpus-b runs, or reopen with the quarantine fallback on the first lock-pressure failure.rmcdaniel commentedon Nov 13, 2026
Attempt 6 of run 43353355060 also passed
execution-and-codec (corpus-b)(job 131675499385): 6 of 6 green on #1195's head 5b3ff71. That's about 12% likely by chance at the unfixed ~3-in-10 rate. I'll keep counting in merge-queue groups after #1195 merges.rmcdaniel commentedon Nov 13, 2026
#1195 merged 2026-11-13 04:57Z (f294214). From here I count
execution-and-codec (corpus-b)in merge-queue groups and close this at 20 consecutive greens, counting the 6 on #1195's head. The first lock-pressure failure reopens this with the quarantine fallback.rmcdaniel commentedon Nov 13, 2026
I audited actual corpus-b jobs after #1195. The verified gate is 8/20, not 20:
execution-and-codec; their green result summaries are not corpus-b executions and do not count.Owner: Server. Next action: continue the existing exact-head #1195 corpus-b qualification, one job at a time, until the remaining 12 qualifying successes are recorded. Any original lock-pressure recurrence stops the count and reopens the source investigation. I am keeping unrelated #1202 retained-receipt comparator failures separate; those failed jobs are not counted as green evidence. This issue stays open until its gate is met.
rmcdaniel commentedon Nov 13, 2026
The existing exact-head qualification has advanced to10/20: attempt7 of43353355060 passed job131831498524, and attempt8 passed job131836811910. These add to the six original attempts and two audited actual merge-group corpus-b jobs. Attempt9 is starting; one job at a time, same head5b3ff717, no changed retry budget. The20-success gate remains open, and any original lock-pressure recurrence stops qualification.
rmcdaniel commentedon Nov 13, 2026
The existing exact-head qualification is now12/20. Attempt9 passed corpus-b job131841336506 and attempt10 passed job131846481354; these add to the prior10 recorded successes. Same5b3ff717 head, one job at a time, unchanged retry budget. Attempt11 starts next. No skipped merge-group summary, unrelated failed comparator or incomplete local control is counted. The20-success gate remains open.