Repository navigation
test: bound every test with pytest-timeout so a hang fails fast (#543) - #563
Conversation
#341 moved the concurrency tests from clock-based sleeps to events. That removed the flake but moved the failure mode: an Event.wait() whose signal never fires no longer trips a timing assertion, it burns the runner's global timeout and reports nothing attributable. #341's own risk note anticipated it. Adds pytest-timeout to the dev extra and a 60s per-test ceiling with timeout_method = "thread", chosen from measurement rather than taste: slowest healthy test, measured at multiplier 1 4.34s tests/helpers.py BARRIER_TIMEOUT_S give-up bound 5.00s scaled() offsets sub-second 60s leaves >12x headroom over the slowest healthy test and clears BARRIER_TIMEOUT_S by an order of magnitude, so it cannot become a flake source itself. thread rather than signal because a test blocked in a C-level call is not interruptible by SIGALRM, and those are exactly the hangs this catches. The reasoning is recorded next to the setting, not just here. One correction to the issue: it asks that the ceiling clear "BARRIER_TIMEOUT_S multiplied by the largest CHAINWEAVER_TEST_TIMING_MULTIPLIER". BARRIER_TIMEOUT_S is a fixed 5.0 and is deliberately NOT scaled -- helpers.py says so explicitly. The real relationship is BARRIER_TIMEOUT_S plus scaled work, so the ceiling is set against that instead. The multiplier is also not set anywhere under .github/, so CI runs at 1 today. No per-lane override added: at a 4.34s worst case nothing comes close to 60s, so overrides would be unused machinery. Raise the single value if that changes. Verified the ceiling actually fires rather than assuming it -- a throwaway test calling threading.Event().wait() produced a named Timeout with a traceback at the blocking line, not a hang. Probe removed before commit. Same test count with and without this change, identical flags: main 2372 passed, 1 skipped this branch 2372 passed, 1 skipped ruff check All checks passed! ruff format 263 files already formatted mypy Success: no issues found in 256 source files coverage 93.57% (gate 80) Closes #543 Co-Authored-By: Claude <noreply@anthropic.com>
CI result — every gating lane green; one advisory lane red for an unrelated upstream reason21 of 23 checks green, 1 skipped, 1 failed. Breaking that down rather than leaving it as a red tick:
The failure is not this changeThe job dies during dependency installation:
Not merging
I would also have held off regardless: the rule I run under says not to merge with CI red including when the red is pre-existing and demonstrably not mine. The So: ready to merge on your click. Nothing here needs another commit from me. Generated by Claude Code |
dgenio
left a comment
There was a problem hiding this comment.
No code changes requested. All gating lanes are green; the Python 3.15 pre-release failure is advisory (continue-on-error) and fails during installation of transitive cachebox, unrelated to this PR. The only remaining merge blocker is the required external approval.
dgenio
left a comment
There was a problem hiding this comment.
No code changes requested. All gating lanes are green; the Python 3.15 pre-release failure is advisory (continue-on-error) and fails during installation of transitive cachebox, unrelated to this PR. The only remaining merge blocker is the required external approval.
dgenio
left a comment
There was a problem hiding this comment.
No code changes requested. All gating lanes are green; the Python 3.15 pre-release failure is advisory (continue-on-error) and fails during installation of transitive cachebox, unrelated to this PR. The only remaining merge blocker is the required external approval.
What
pytest-timeoutin thedevextra, and a 60s per-test ceiling withtimeout_method = "thread"in[tool.pytest.ini_options].Closes #543.
Why
#341 replaced clock-based waiting with events. That removed the flake and moved the failure mode: an
Event.wait()whose signal never fires no longer trips a timing assertion — it burns the runner's global timeout and reports nothing you can attribute. #341's own risk note called this out. A per-test ceiling turns a silent hang into a named failure with a traceback.Choosing 60s
From measurement, not taste:
test_openai_agents_tool_recipe_runs)tests/helpers.pyBARRIER_TIMEOUT_Sgive-up boundscaled()offsets60s leaves >12× headroom over the slowest healthy test and clears
BARRIER_TIMEOUT_Sby an order of magnitude, so it cannot become a flake source itself.threadrather thansignalbecause a test blocked in a C-level call is not interruptible bySIGALRM, and those are precisely the hangs this exists to catch.The reasoning lives next to the setting in
pyproject.toml, not only here — acceptance criterion 3.One correction to the issue
The issue asks that the ceiling clear "
BARRIER_TIMEOUT_S(5.0s) multiplied by the largestCHAINWEAVER_TEST_TIMING_MULTIPLIER".BARRIER_TIMEOUT_Sis a fixed5.0and is deliberately not scaled —helpers.pysays so in its own docstring:So the real relationship is
BARRIER_TIMEOUT_Splus scaled work, and the ceiling is set against that. Flagging it rather than quietly implementing something different from what was asked. Also worth knowing:CHAINWEAVER_TEST_TIMING_MULTIPLIERis not set anywhere under.github/, so CI runs at1today.No per-lane override added — at a 4.34s worst case nothing approaches 60s, so overrides would be unused machinery. Raise the single value if that ever changes.
Acceptance criteria
A test that blocks forever fails with an attributable timeout rather than hanging. Verified, not assumed — a throwaway test calling
threading.Event().wait():A named
Timeoutwith a traceback at the exact blocking line. The probe file was removed before committing — it is not in this diff.The default ceiling does not fire for any healthy test. Full suite green at multiplier 1; slowest test 4.34s against a 60s ceiling.
The relationship to
BARRIER_TIMEOUT_Sis documented next to the setting.All four gates pass (below).
Validation
This change drops no tests. Controlled A/B with identical flags:
mainOne honest note: an earlier exploratory run reported 2385 passed. That run was taken moments after
pip install -e '.[dev]', which hit a resolver conflict, and this repository has tests gated on optional dependencies. Collection is now deterministic at 2373 across repeated runs (2372 + 1 skipped). I have not fully explained the earlier 13-test delta and am not claiming it is understood — the load-bearing evidence is the A/B above, which isolates this change and shows it accounts for none of it.Generated by Claude Code