Repository navigation
ci: make the Python 3.15 canary able to report a failure - #565
Conversation
#341 moved the concurrency tests from clock-based sleeps to events. That removed the flake but moved the failure mode: an Event.wait() whose signal never fires no longer trips a timing assertion, it burns the runner's global timeout and reports nothing attributable. #341's own risk note anticipated it. Adds pytest-timeout to the dev extra and a 60s per-test ceiling with timeout_method = "thread", chosen from measurement rather than taste: slowest healthy test, measured at multiplier 1 4.34s tests/helpers.py BARRIER_TIMEOUT_S give-up bound 5.00s scaled() offsets sub-second 60s leaves >12x headroom over the slowest healthy test and clears BARRIER_TIMEOUT_S by an order of magnitude, so it cannot become a flake source itself. thread rather than signal because a test blocked in a C-level call is not interruptible by SIGALRM, and those are exactly the hangs this catches. The reasoning is recorded next to the setting, not just here. One correction to the issue: it asks that the ceiling clear "BARRIER_TIMEOUT_S multiplied by the largest CHAINWEAVER_TEST_TIMING_MULTIPLIER". BARRIER_TIMEOUT_S is a fixed 5.0 and is deliberately NOT scaled -- helpers.py says so explicitly. The real relationship is BARRIER_TIMEOUT_S plus scaled work, so the ceiling is set against that instead. The multiplier is also not set anywhere under .github/, so CI runs at 1 today. No per-lane override added: at a 4.34s worst case nothing comes close to 60s, so overrides would be unused machinery. Raise the single value if that changes. Verified the ceiling actually fires rather than assuming it -- a throwaway test calling threading.Event().wait() produced a named Timeout with a traceback at the blocking line, not a hang. Probe removed before commit. Same test count with and without this change, identical flags: main 2372 passed, 1 skipped this branch 2372 passed, 1 skipped ruff check All checks passed! ruff format 263 files already formatted mypy Success: no issues found in 256 source files coverage 93.57% (gate 80) Closes #543 Co-Authored-By: Claude <noreply@anthropic.com>
Closes #550. The `dev` extra mixed two different things: tools that exist to run this repository's own gate (ruff, mypy, nbmake, a Jupyter kernel) and the integration packages the suite needs to exercise the published extras. Only the second kind belongs in wheel metadata, and `pip install chainweaver[dev]` offered the first kind to anyone who wanted the library. - `dev` extra removed. Maintainer tooling lives in `[dependency-groups]`, which is not published: the built wheel's `Provides-Extra` no longer lists `dev`, and its METADATA no longer mentions ruff, nbmake or opentelemetry-sdk (verified by building the wheel and reading it). - New published extra `integrations`: the composition of every user-facing integration extra, as a single self-reference. Naming it once is the point — the old list existed in eight workflow steps, CONTRIBUTING, the README, the docs site and the floor job. It excludes `llm-anthropic`/`llm-openai` deliberately; evals.yml adds the provider it needs. - Second group `test-runners` holds the four pytest packages. The free-threaded 3.14t lane and the Python-next canary install exactly that group instead of hand-copying the list, and that copy had already drifted: neither gained `pytest-timeout` when #543 added it, so the two lanes most likely to hang were the ones running unbounded. - Every `--group` install upgrades pip first. PEP 735 support landed in pip 25.1, and a gate that silently depends on whatever the runner image ships is a gate that reports a fact about its environment. Resolution is unchanged, measured rather than argued: pip install --dry-run --ignore-installed -e ".[dev]" # old pip install --dry-run --ignore-installed -e ".[integrations]" --group dev resolve the same 193 distributions at the same versions — set difference empty in both directions, no version differences. uv 0.8.17 honours `--group` too (checked: an unknown group name is rejected, and a real one pulls ruff/mypy/nbmake), so the `lowest-direct` floor job keeps working. New `tests/test_dependency_groups.py` (8 tests) fails when a maintainer tool reappears in a published extra, when a workflow installs `.[dev]` or pins a pytest runner inline, when `dev` restates the runners instead of including the group, or when the `integrations` composition and the integration extras disagree. Both halves were proven to fail against the defect they name before being kept: reinstating a `dev` extra failed two of them, and adding an inline `"pytest-timeout>=2.3"` to fuzz.yml failed the drift guard; reverting each restored green. The TOML-reading tests skip on 3.10, which has no stdlib tomllib and where the repo declines to add tomli; the workflow scans run everywhere. Validation: ruff check chainweaver/ tests/ examples/ All checks passed! ruff format --check chainweaver/ tests/ examples/ 308 files already formatted mypy chainweaver/ Success: no issues found in 105 source files pytest tests/ -q --no-cov 2393 passed, 1 skipped Docs updated atomically: CONTRIBUTING, README extras table and quickstart, docs/getting-started/installation.md, examples/README.md, CHANGELOG. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhJp1Rnr6mF4vdSLkj14jU
mypy runs at python_version = "3.10", where tomllib does not exist. Guard the import behind the same sys.version_info check test_metadata_consistency uses so the branch is unreachable under the configured version, and give the helper signatures their type arguments for --strict. mypy chainweaver/ tests/ Success: no issues found in 261 source files ruff check / format clean pytest tests/test_dependency_groups.py 8 passed Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhJp1Rnr6mF4vdSLkj14jU
The Python Next Canary has never once run the test suite. Every run since the lane was added on 2026-08-10 failed at "Install compatibility surface" and skipped "Run core suite" — checked at job level on the runs of 10 Aug (the commit that added it), 12 Aug, 19 Aug, 26 Aug, 28 Aug and 30 Aug. Job-level continue-on-error reported every one of those *workflow runs* as success, so five consecutive scheduled canaries looked green and proved nothing. The blocker is real and upstream. deepdiff 9.1.0 added `cachebox<6,>=5.2`; the newest cachebox in that range is 5.2.3 (2026-04-10), whose wheels stop at cp314, and no cachebox 5.x ships a cp315 wheel. So pip builds it from source on 3.15 and its vendored PyO3 refuses the interpreter. cachebox 6.1.2+ does ship cp315 wheels, but deepdiff's `<6` cap excludes them, so no resolution of deepdiff 9.1.0 installs on 3.15 today. That is one of the two things this lane exists to report. The other — whether ChainWeaver's own code works on the next Python — was being swallowed by it. So the install now tries the real thing first, records which outcome happened, and on failure falls back to constraints/python-next.txt so the suite still runs. A summary step with `if: always()` states which of the two it proved, including the case where neither install worked. The constraint is deepdiff<9.1 and nothing else. deepdiff 9.0.0 carries no cachebox dependency and satisfies this project's declared `deepdiff>=9.0` floor. Measured rather than assumed — resolving the canary's install set with and without the constraint file: unconstrained: deepdiff 9.1.0 | cachebox 5.2.3 constrained: deepdiff 9.0.0 | cachebox None dropped: ['cachebox'] added: [] Exactly one package leaves the graph. The file says why it exists and when to delete it; nothing else reads it, and it must never be used for the supported matrix. tests/test_python_next_canary.py (6 tests) pins what keeps the lane honest: the suite step cannot become conditional, the install must record its mode and apply the fallback, the summary must be `if: always()` and must say "proved nothing" when neither attempt worked, the constraint may pin down but never below the declared floor, and the constraints file must parse as one `pip -c` would actually honour. Three of them were confirmed to fail against the defect they name before being kept: adding `if: steps.install.outputs.mode == 'full'` to the suite step, removing `if: always()` from the summary, and lowering the cap to `deepdiff<9.0` each failed exactly one test; reverting restored green. What is NOT verified here: that the constrained install succeeds on a real 3.15 interpreter. There is no 3.15 in this environment, the resolution above was measured on 3.11, and this lane is non-gating — CI on this PR is the first real evidence. If the fallback also fails, the new summary step now says so out loud instead of reporting success. ruff check chainweaver/ tests/ examples/ All checks passed! ruff format --check 309 files already formatted mypy chainweaver/ tests/ no issues in 262 source files pytest tests/ -q --no-cov 2399 passed, 1 skipped Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhJp1Rnr6mF4vdSLkj14jU
First evidence from the fixed lane, and it changes the picture. The fallback install worked — `Install compatibility surface` succeeded for the first time since the lane was added — and `Run core suite` ran instead of being skipped. It then died at collection in 7.5s: ImportError: cannot import name 'no_type_check_decorator' from 'typing' ... ERROR tests/test_mcp_adapter.py ERROR tests/test_mcp_adapter_trust.py ERROR tests/test_mcp_server.py ERROR tests/test_mcp_server_security.py Interrupted: 4 errors during collection 8 skipped, 4 errors in 7.52s That is a second 3.15 blocker, independent of the cachebox one and not caused by the constraint: fastmcp -> key_value.aio -> beartype, and beartype imports `typing.no_type_check_decorator`, which 3.15 removed. It surfaces as "FastMCP server support is not installed" because fastmcp catches the underlying error, and the four tests/test_mcp_*.py modules import that surface unguarded at module scope — the same pattern the free-threaded lane already documents in ci.yml. So the lane got one step further and still reported nothing about the other ~2400 tests. Adding --continue-on-collection-errors: the step still fails, every error is still printed, but one unimportable dependency no longer erases the whole signal. Not pinning fastmcp: it is precisely the surface this canary exists to test on the next Python. The blocker is recorded in constraints/python-next.txt next to the cachebox one, with why it is deliberately not constrained around. ruff check chainweaver/ tests/ examples/ All checks passed! ruff format --check 309 files already formatted pytest tests/test_python_next_canary.py 6 passed Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XhJp1Rnr6mF4vdSLkj14jU
First evidence from the fixed lane — it works, and it immediately found a second blockerThe
Interpreter confirmed And the lane immediately earned its keepIt got to collection and died there in 7.5s — on something that has nothing to do with the cachebox constraint: A second, independent 3.15 blocker: This is exactly the class of thing the canary exists to find, and it could not have reported it in the three weeks it was silently dead. Follow-up pushed (
|
|
| 10 Aug – 30 Aug (6 runs) | fbac841 |
f0c1e1a |
|
|---|---|---|---|
| Install | failure | ✅ success | ✅ success |
| Suite | skipped | ran, aborted at collection | ✅ ran |
| Tests executed | 0 | 0 | 2183 |
The --continue-on-collection-errors change did what it was meant to: one unimportable dependency no longer erases the whole signal.
What it found, all one root cause
ERROR tests/test_mcp_adapter.py · test_mcp_adapter_trust.py · test_mcp_server.py · test_mcp_server_security.py
FAILED tests/test_cli_serve.py::TestServeCommandRegistration::test_transport_enum_matches_library_literal
- ImportError: chainweaver.mcp.server requires the 'fastmcp' package and the 'mcp' SDK.
FAILED tests/test_cli_serve.py::TestBuildFlowServer::test_exposes_one_tool_per_flow - typer.exceptions.Exit
FAILED tests/test_cli_serve.py::TestBuildFlowServer::test_prefix_namespaces_tool_names - typer.exceptions.Exit
FAILED tests/test_cli_serve.py::TestBuildFlowServer::test_directory_exposes_active_withholds_draft - typer.exceptions.Exit
FAILED tests/test_cli_serve.py::TestBuildFlowServer::test_missing_flow_file_exits_2 - assert 1 == 2
All nine trace back to the single beartype/3.15 blocker: fastmcp cannot import, so chainweaver.mcp.server raises, so the CLI's serve path exits early. The assert 1 == 2 is the same thing wearing a different hat — the command exits 1 on the missing extra before it can reach its own exit-2 path.
Nothing here is a ChainWeaver bug on 3.15, which is itself the useful result: outside the MCP surface, 2183 tests pass on the next Python. The lane could not have told you that before.
Status
The check stays failure, honestly, and stays continue-on-error, so it warns rather than blocks. Both blockers are upstream (deepdiff→cachebox has no cp315 wheel; beartype imports typing.no_type_check_decorator, removed in 3.15) and neither is fixable here. No further pushes planned — this is the lane reporting the truth, which is what the PR set out to make possible.
The PR body's one unverified claim ("that the constrained install actually succeeds on a real 3.15 interpreter") is now confirmed.
Generated by Claude Code
…t-canary # Conflicts: # .github/workflows/python-next.yml
Summary
The Python Next Canary has never once run the test suite. Not "went red recently" — never worked, from the commit that added it on 2026-08-10 to today.
I found this while looking into why #564's canary check was red, assumed it was a fresh upstream break, and said so. That was wrong: I had read run conclusions, which
continue-on-errorreports assuccess. At job level, every run tells the same story.Run core suitemainmainmainFive consecutive scheduled canaries looked green and proved nothing.
The blocker is real, and upstream
cacheboxis not something the lane asks for. It arrives throughdeepdiff, a base runtime dependency:deepdiff9.1.0 (2026-05-15) addedcachebox<6,>=5.2— 9.0.0 and earlier have no cachebox dependency at all (confirmed, read from each release's metadata);<6cap excludes them.So there is no resolution of
deepdiff 9.1.0that installs on CPython 3.15 today, and pip is forced to build 5.2.3 from source.What this changes
That upstream gap is one of the two things this lane exists to report. The other — does ChainWeaver's own code work on the next Python — was being swallowed by it. So:
mode=full/mode=constrained). The first attempt failing is a signal, not an error to hide.constraints/python-next.txtso the suite still runs.if: always()writes to$GITHUB_STEP_SUMMARYwhich of the two it proved — including the case where neither install worked, where it says "the suite did not run. This lane proved nothing on this commit."The lane stays
continue-on-error: true. It warns; it does not block. It just no longer warns into a void.The constraint is one line, and measured
deepdiff<9.1, nothing else. deepdiff 9.0.0 carries no cachebox dependency and satisfies the declareddeepdiff>=9.0floor. Resolving the canary's exact install set with and without the file:Exactly one package leaves the graph. The file documents the mechanism and its own exit condition ("Remove this file once deepdiff relaxes the cap or cachebox 5.x ships cp315"), and states that nothing else may read it.
Testing
ruff check chainweaver/ tests/ examples/—All checks passed!ruff format --check chainweaver/ tests/ examples/—309 files already formattedpython -m mypy chainweaver/ tests/—Success: no issues found in 262 source filespython -m pytest tests/ -q --no-cov—2399 passed, 1 skipped in 60.38stests/test_python_next_canary.py, 6 testsThree of the six were confirmed to fail against the defect they name before being kept:
if: steps.install.outputs.mode == 'full'on the suite steptest_the_suite_step_is_not_conditional_on_the_install_succeedingif: always()from the summarytest_the_lane_always_says_what_it_proveddeepdiff<9.0test_the_canary_constraint_stays_within_the_declared_floorEach failed exactly one test; reverting restored green.
What is not verified
That the constrained install actually succeeds on a real 3.15 interpreter. There is no 3.15 in my environment; the resolution above was measured on 3.11. CI on this PR is the first real evidence — the lane triggers on
pyproject.tomland on its own workflow file, so it will run here. If the fallback also fails, the new summary step now says so out loud rather than reporting success.unverifieduntil that job posts.Issues closed by this PR
None yet — I have not opened one. The lane's three-week silence and the upstream
deepdiff/cacheboxgap are both worth issues; say the word and I will file them, or fold this under an existing number.Related Issues
Stacked on #564. Refs #236 (the dependency-policy work this canary belongs to).
Checklist
AGENTS.mdanddocs/agent-context/)Generated by Claude Code