Skip to content

test: bound every test with pytest-timeout so a hang fails fast (#543) - #563

Merged
dgenio merged 1 commit into
mainfrom
claude/daily-critical-triage-7tip88
Aug 28, 2026
Merged

dgenio merged 1 commit into
mainfrom
claude/daily-critical-triage-7tip88

Conversation

@dgenio

@dgenio dgenio commented Aug 28, 2026

Copy link
Copy Markdown
Owner

What

pytest-timeout in the dev extra, and a 60s per-test ceiling with timeout_method = "thread" in [tool.pytest.ini_options].

Closes #543.

Why

#341 replaced clock-based waiting with events. That removed the flake and moved the failure mode: an Event.wait() whose signal never fires no longer trips a timing assertion — it burns the runner's global timeout and reports nothing you can attribute. #341's own risk note called this out. A per-test ceiling turns a silent hang into a named failure with a traceback.

Choosing 60s

From measurement, not taste:

slowest healthy test, measured at multiplier 1 4.34s (test_openai_agents_tool_recipe_runs)
tests/helpers.py BARRIER_TIMEOUT_S give-up bound 5.00s
scaled() offsets sub-second

60s leaves >12× headroom over the slowest healthy test and clears BARRIER_TIMEOUT_S by an order of magnitude, so it cannot become a flake source itself. thread rather than signal because a test blocked in a C-level call is not interruptible by SIGALRM, and those are precisely the hangs this exists to catch.

The reasoning lives next to the setting in pyproject.toml, not only here — acceptance criterion 3.

One correction to the issue

The issue asks that the ceiling clear "BARRIER_TIMEOUT_S (5.0s) multiplied by the largest CHAINWEAVER_TEST_TIMING_MULTIPLIER". BARRIER_TIMEOUT_S is a fixed 5.0 and is deliberately not scaled — helpers.py says so in its own docstring:

Deliberately not scaled by :data:TIMING_MULTIPLIER for that reason.

So the real relationship is BARRIER_TIMEOUT_S plus scaled work, and the ceiling is set against that. Flagging it rather than quietly implementing something different from what was asked. Also worth knowing: CHAINWEAVER_TEST_TIMING_MULTIPLIER is not set anywhere under .github/, so CI runs at 1 today.

No per-lane override added — at a 4.34s worst case nothing approaches 60s, so overrides would be unused machinery. Raise the single value if that ever changes.

Acceptance criteria

  • A test that blocks forever fails with an attributable timeout rather than hanging. Verified, not assumed — a throwaway test calling threading.Event().wait():

      File "/home/user/ChainWeaver/tests/test_zz_hang_probe.py", line 5, in test_blocks_forever
        threading.Event().wait()
      File "/usr/lib/python3.11/threading.py", line 629, in wait
        signaled = self._cond.wait(timeout)
    +++++++++++++++++++++++++++++++++++ Timeout ++++++++++++++++++++++++++++++++++++
    

    A named Timeout with a traceback at the exact blocking line. The probe file was removed before committing — it is not in this diff.

  • The default ceiling does not fire for any healthy test. Full suite green at multiplier 1; slowest test 4.34s against a 60s ceiling.

  • The relationship to BARRIER_TIMEOUT_S is documented next to the setting.

  • All four gates pass (below).

Validation

ruff check chainweaver/ tests/          →  All checks passed!
ruff format --check chainweaver/ tests/ →  263 files already formatted
mypy chainweaver/ tests/                →  Success: no issues found in 256 source files
pytest tests/                           →  2372 passed, 1 skipped in 65.25s
                                           Total coverage: 93.57% (gate 80)

This change drops no tests. Controlled A/B with identical flags:

tree result
main 2372 passed, 1 skipped
this branch 2372 passed, 1 skipped

One honest note: an earlier exploratory run reported 2385 passed. That run was taken moments after pip install -e '.[dev]', which hit a resolver conflict, and this repository has tests gated on optional dependencies. Collection is now deterministic at 2373 across repeated runs (2372 + 1 skipped). I have not fully explained the earlier 13-test delta and am not claiming it is understood — the load-bearing evidence is the A/B above, which isolates this change and shows it accounts for none of it.


Generated by Claude Code

#341 moved the concurrency tests from clock-based sleeps to events. That
removed the flake but moved the failure mode: an Event.wait() whose signal
never fires no longer trips a timing assertion, it burns the runner's global
timeout and reports nothing attributable. #341's own risk note anticipated it.

Adds pytest-timeout to the dev extra and a 60s per-test ceiling with
timeout_method = "thread", chosen from measurement rather than taste:

  slowest healthy test, measured at multiplier 1        4.34s
  tests/helpers.py BARRIER_TIMEOUT_S give-up bound      5.00s
  scaled() offsets                                      sub-second

60s leaves >12x headroom over the slowest healthy test and clears
BARRIER_TIMEOUT_S by an order of magnitude, so it cannot become a flake source
itself. thread rather than signal because a test blocked in a C-level call is
not interruptible by SIGALRM, and those are exactly the hangs this catches.
The reasoning is recorded next to the setting, not just here.

One correction to the issue: it asks that the ceiling clear "BARRIER_TIMEOUT_S
multiplied by the largest CHAINWEAVER_TEST_TIMING_MULTIPLIER". BARRIER_TIMEOUT_S
is a fixed 5.0 and is deliberately NOT scaled -- helpers.py says so explicitly.
The real relationship is BARRIER_TIMEOUT_S plus scaled work, so the ceiling is
set against that instead. The multiplier is also not set anywhere under
.github/, so CI runs at 1 today.

No per-lane override added: at a 4.34s worst case nothing comes close to 60s,
so overrides would be unused machinery. Raise the single value if that changes.

Verified the ceiling actually fires rather than assuming it -- a throwaway test
calling threading.Event().wait() produced a named Timeout with a traceback at
the blocking line, not a hang. Probe removed before commit.

Same test count with and without this change, identical flags:
  main            2372 passed, 1 skipped
  this branch     2372 passed, 1 skipped
  ruff check      All checks passed!
  ruff format     263 files already formatted
  mypy            Success: no issues found in 256 source files
  coverage        93.57% (gate 80)

Closes #543

Co-Authored-By: Claude <noreply@anthropic.com>

dgenio commented Aug 28, 2026

Copy link
Copy Markdown
Owner Author

CI result — every gating lane green; one advisory lane red for an unrelated upstream reason

21 of 23 checks green, 1 skipped, 1 failed. Breaking that down rather than leaving it as a red tick:

lane result
test × {ubuntu, windows, macos} × {3.10, 3.11, 3.12, 3.13, 3.14} — 15 jobs ✅ all success
floor deps (min versions, Python 3.10) ✅ success
free-threaded smoke (Python 3.14t) ✅ success
weaver-spec conformance (#91) ✅ success
lint, format, types, workflows ✅ success
bench (ubuntu-22.04, 3.10), build docs ✅ success
latest/pre-release deps (Python 3.14) ⊘ skipped
Python 3.15 pre-release ❌ failure

floor deps (min versions, Python 3.10) passing matters here specifically — it resolves each direct dependency to its declared minimum, so it is the job that exercises the new pytest-timeout>=2.3 floor rather than whatever pip felt like installing.

The failure is not this change

The job dies during dependency installation:

╰─> cachebox
##[error]Process completed with exit code 1.

cachebox is a Rust extension with no wheel for the 3.15 pre-release yet. Three checks, so this is not an assumption:

  • cachebox is not a ChainWeaver dependency — it does not appear in pyproject.toml at all.
  • It arrives transitively: deepdiff -> cachebox>=5.2,<6.
  • pytest-timeout requires only pytest — no native code, nothing that could pull it.

.github/workflows/python-next.yml also declares the job continue-on-error: true, so the repository already treats this lane as advisory rather than gating. That is the right call for a pre-release Python lane whose failures are usually upstream wheel availability, exactly as here.

Not merging

mergeable_state is blocked with zero reviews on the PR, which means a required approval is outstanding, and GitHub will not let me approve a PR authored on your behalf.

I would also have held off regardless: the rule I run under says not to merge with CI red including when the red is pre-existing and demonstrably not mine. The continue-on-error flag is a reasonable argument that this particular lane is not a gate — but that is a judgement about your repository's intent, and it is yours to make rather than mine to assume at 3am.

So: ready to merge on your click. Nothing here needs another commit from me.


Generated by Claude Code

@dgenio
dgenio merged commit 67cd745 into main Aug 28, 2026
22 of 23 checks passed

@dgenio dgenio left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No code changes requested. All gating lanes are green; the Python 3.15 pre-release failure is advisory (continue-on-error) and fails during installation of transitive cachebox, unrelated to this PR. The only remaining merge blocker is the required external approval.

@dgenio dgenio left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No code changes requested. All gating lanes are green; the Python 3.15 pre-release failure is advisory (continue-on-error) and fails during installation of transitive cachebox, unrelated to this PR. The only remaining merge blocker is the required external approval.

@dgenio dgenio left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No code changes requested. All gating lanes are green; the Python 3.15 pre-release failure is advisory (continue-on-error) and fails during installation of transitive cachebox, unrelated to this PR. The only remaining merge blocker is the required external approval.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add pytest-timeout so a never-firing event wait fails fast instead of hanging the suite

2 participants