Skip to content

ci: make the Python 3.15 canary able to report a failure - #565

Merged
dgenio merged 7 commits into
mainfrom
claude/fix-python-next-canary
Aug 30, 2026
Merged

dgenio merged 7 commits into
mainfrom
claude/fix-python-next-canary

Conversation

@dgenio

@dgenio dgenio commented Aug 30, 2026

Copy link
Copy Markdown
Owner

Summary

The Python Next Canary has never once run the test suite. Not "went red recently" — never worked, from the commit that added it on 2026-08-10 to today.

I found this while looking into why #564's canary check was red, assumed it was a fresh upstream break, and said so. That was wrong: I had read run conclusions, which continue-on-error reports as success. At job level, every run tells the same story.

Run Date Trigger Run conclusion Job conclusion Run core suite
1 10 Aug the PR that added the lane success failure skipped
15 12 Aug schedule, main success failure skipped
16 19 Aug schedule, main success failure skipped
17 26 Aug schedule, main success failure skipped
18 28 Aug PR success failure skipped
19 30 Aug PR (#564) success failure skipped

Five consecutive scheduled canaries looked green and proved nothing.

⚠️ This PR is stacked on #564 — it needs that PR's --group test-runners change to the same file. Review/merge #564 first; the diff here is the three files listed below.

The blocker is real, and upstream

error: the configured Python interpreter version (3.15) is newer than PyO3's maximum supported version
💥 maturin failed
ERROR: Failed building wheel for cachebox

cachebox is not something the lane asks for. It arrives through deepdiff, a base runtime dependency:

  • deepdiff 9.1.0 (2026-05-15) added cachebox<6,>=5.2 — 9.0.0 and earlier have no cachebox dependency at all (confirmed, read from each release's metadata);
  • the newest cachebox in that range is 5.2.3 (2026-04-10), whose wheels stop at cp314. No cachebox 5.x ships a cp315 wheel;
  • cachebox 6.1.2+ does ship cp315 wheels — but deepdiff's <6 cap excludes them.

So there is no resolution of deepdiff 9.1.0 that installs on CPython 3.15 today, and pip is forced to build 5.2.3 from source.

What this changes

That upstream gap is one of the two things this lane exists to report. The other — does ChainWeaver's own code work on the next Python — was being swallowed by it. So:

  1. The install tries the real thing first and records the outcome (mode=full / mode=constrained). The first attempt failing is a signal, not an error to hide.
  2. On failure it falls back to constraints/python-next.txt so the suite still runs.
  3. A summary step with if: always() writes to $GITHUB_STEP_SUMMARY which of the two it proved — including the case where neither install worked, where it says "the suite did not run. This lane proved nothing on this commit."

The lane stays continue-on-error: true. It warns; it does not block. It just no longer warns into a void.

The constraint is one line, and measured

deepdiff<9.1, nothing else. deepdiff 9.0.0 carries no cachebox dependency and satisfies the declared deepdiff>=9.0 floor. Resolving the canary's exact install set with and without the file:

unconstrained: deepdiff 9.1.0 | cachebox 5.2.3
constrained:   deepdiff 9.0.0 | cachebox None
packages dropped by the constraint: ['cachebox']
packages added   by the constraint: []

Exactly one package leaves the graph. The file documents the mechanism and its own exit condition ("Remove this file once deepdiff relaxes the cap or cachebox 5.x ships cp315"), and states that nothing else may read it.

Testing

  • ruff check chainweaver/ tests/ examples/ — All checks passed!
  • ruff format --check chainweaver/ tests/ examples/ — 309 files already formatted
  • python -m mypy chainweaver/ tests/ — Success: no issues found in 262 source files
  • python -m pytest tests/ -q --no-cov — 2399 passed, 1 skipped in 60.38s
  • New tests — tests/test_python_next_canary.py, 6 tests

Three of the six were confirmed to fail against the defect they name before being kept:

Injected defect Test that failed
if: steps.install.outputs.mode == 'full' on the suite step test_the_suite_step_is_not_conditional_on_the_install_succeeding
removed if: always() from the summary test_the_lane_always_says_what_it_proved
lowered the cap to deepdiff<9.0 test_the_canary_constraint_stays_within_the_declared_floor

Each failed exactly one test; reverting restored green.

What is not verified

That the constrained install actually succeeds on a real 3.15 interpreter. There is no 3.15 in my environment; the resolution above was measured on 3.11. CI on this PR is the first real evidence — the lane triggers on pyproject.toml and on its own workflow file, so it will run here. If the fallback also fails, the new summary step now says so out loud rather than reporting success. unverified until that job posts.

Issues closed by this PR

None yet — I have not opened one. The lane's three-week silence and the upstream deepdiff/cachebox gap are both worth issues; say the word and I will file them, or fold this under an existing number.

Related Issues

Stacked on #564. Refs #236 (the dependency-policy work this canary belongs to).

Checklist

  • Code follows project conventions (see AGENTS.md and docs/agent-context/)
  • Public API changes are documented — none; no runtime code changed
  • No secrets or credentials included

Generated by Claude Code

claude added 6 commits August 28, 2026 02:53
#341 moved the concurrency tests from clock-based sleeps to events. That
removed the flake but moved the failure mode: an Event.wait() whose signal
never fires no longer trips a timing assertion, it burns the runner's global
timeout and reports nothing attributable. #341's own risk note anticipated it.

Adds pytest-timeout to the dev extra and a 60s per-test ceiling with
timeout_method = "thread", chosen from measurement rather than taste:

  slowest healthy test, measured at multiplier 1        4.34s
  tests/helpers.py BARRIER_TIMEOUT_S give-up bound      5.00s
  scaled() offsets                                      sub-second

60s leaves >12x headroom over the slowest healthy test and clears
BARRIER_TIMEOUT_S by an order of magnitude, so it cannot become a flake source
itself. thread rather than signal because a test blocked in a C-level call is
not interruptible by SIGALRM, and those are exactly the hangs this catches.
The reasoning is recorded next to the setting, not just here.

One correction to the issue: it asks that the ceiling clear "BARRIER_TIMEOUT_S
multiplied by the largest CHAINWEAVER_TEST_TIMING_MULTIPLIER". BARRIER_TIMEOUT_S
is a fixed 5.0 and is deliberately NOT scaled -- helpers.py says so explicitly.
The real relationship is BARRIER_TIMEOUT_S plus scaled work, so the ceiling is
set against that instead. The multiplier is also not set anywhere under
.github/, so CI runs at 1 today.

No per-lane override added: at a 4.34s worst case nothing comes close to 60s,
so overrides would be unused machinery. Raise the single value if that changes.

Verified the ceiling actually fires rather than assuming it -- a throwaway test
calling threading.Event().wait() produced a named Timeout with a traceback at
the blocking line, not a hang. Probe removed before commit.

Same test count with and without this change, identical flags:
  main            2372 passed, 1 skipped
  this branch     2372 passed, 1 skipped
  ruff check      All checks passed!
  ruff format     263 files already formatted
  mypy            Success: no issues found in 256 source files
  coverage        93.57% (gate 80)

Closes #543

Co-Authored-By: Claude <noreply@anthropic.com>
Closes #550.

The `dev` extra mixed two different things: tools that exist to run this
repository's own gate (ruff, mypy, nbmake, a Jupyter kernel) and the
integration packages the suite needs to exercise the published extras.
Only the second kind belongs in wheel metadata, and `pip install
chainweaver[dev]` offered the first kind to anyone who wanted the
library.

- `dev` extra removed. Maintainer tooling lives in `[dependency-groups]`,
  which is not published: the built wheel's `Provides-Extra` no longer
  lists `dev`, and its METADATA no longer mentions ruff, nbmake or
  opentelemetry-sdk (verified by building the wheel and reading it).
- New published extra `integrations`: the composition of every
  user-facing integration extra, as a single self-reference. Naming it
  once is the point — the old list existed in eight workflow steps,
  CONTRIBUTING, the README, the docs site and the floor job. It excludes
  `llm-anthropic`/`llm-openai` deliberately; evals.yml adds the provider
  it needs.
- Second group `test-runners` holds the four pytest packages. The
  free-threaded 3.14t lane and the Python-next canary install exactly
  that group instead of hand-copying the list, and that copy had already
  drifted: neither gained `pytest-timeout` when #543 added it, so the two
  lanes most likely to hang were the ones running unbounded.
- Every `--group` install upgrades pip first. PEP 735 support landed in
  pip 25.1, and a gate that silently depends on whatever the runner image
  ships is a gate that reports a fact about its environment.

Resolution is unchanged, measured rather than argued:

  pip install --dry-run --ignore-installed -e ".[dev]"                  # old
  pip install --dry-run --ignore-installed -e ".[integrations]" --group dev

resolve the same 193 distributions at the same versions — set difference
empty in both directions, no version differences. uv 0.8.17 honours
`--group` too (checked: an unknown group name is rejected, and a real one
pulls ruff/mypy/nbmake), so the `lowest-direct` floor job keeps working.

New `tests/test_dependency_groups.py` (8 tests) fails when a maintainer
tool reappears in a published extra, when a workflow installs `.[dev]` or
pins a pytest runner inline, when `dev` restates the runners instead of
including the group, or when the `integrations` composition and the
integration extras disagree. Both halves were proven to fail against the
defect they name before being kept: reinstating a `dev` extra failed two
of them, and adding an inline `"pytest-timeout>=2.3"` to fuzz.yml failed
the drift guard; reverting each restored green. The TOML-reading tests
skip on 3.10, which has no stdlib tomllib and where the repo declines to
add tomli; the workflow scans run everywhere.

Validation:
  ruff check chainweaver/ tests/ examples/    All checks passed!
  ruff format --check chainweaver/ tests/ examples/   308 files already formatted
  mypy chainweaver/                           Success: no issues found in 105 source files
  pytest tests/ -q --no-cov                   2393 passed, 1 skipped

Docs updated atomically: CONTRIBUTING, README extras table and quickstart,
docs/getting-started/installation.md, examples/README.md, CHANGELOG.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XhJp1Rnr6mF4vdSLkj14jU
mypy runs at python_version = "3.10", where tomllib does not exist. Guard
the import behind the same sys.version_info check test_metadata_consistency
uses so the branch is unreachable under the configured version, and give
the helper signatures their type arguments for --strict.

mypy chainweaver/ tests/  Success: no issues found in 261 source files
ruff check / format       clean
pytest tests/test_dependency_groups.py  8 passed

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XhJp1Rnr6mF4vdSLkj14jU
The Python Next Canary has never once run the test suite. Every run
since the lane was added on 2026-08-10 failed at "Install compatibility
surface" and skipped "Run core suite" — checked at job level on the runs
of 10 Aug (the commit that added it), 12 Aug, 19 Aug, 26 Aug, 28 Aug and
30 Aug. Job-level continue-on-error reported every one of those *workflow
runs* as success, so five consecutive scheduled canaries looked green and
proved nothing.

The blocker is real and upstream. deepdiff 9.1.0 added
`cachebox<6,>=5.2`; the newest cachebox in that range is 5.2.3
(2026-04-10), whose wheels stop at cp314, and no cachebox 5.x ships a
cp315 wheel. So pip builds it from source on 3.15 and its vendored PyO3
refuses the interpreter. cachebox 6.1.2+ does ship cp315 wheels, but
deepdiff's `<6` cap excludes them, so no resolution of deepdiff 9.1.0
installs on 3.15 today.

That is one of the two things this lane exists to report. The other —
whether ChainWeaver's own code works on the next Python — was being
swallowed by it. So the install now tries the real thing first, records
which outcome happened, and on failure falls back to
constraints/python-next.txt so the suite still runs. A summary step with
`if: always()` states which of the two it proved, including the case
where neither install worked.

The constraint is deepdiff<9.1 and nothing else. deepdiff 9.0.0 carries
no cachebox dependency and satisfies this project's declared
`deepdiff>=9.0` floor. Measured rather than assumed — resolving the
canary's install set with and without the constraint file:

  unconstrained: deepdiff 9.1.0 | cachebox 5.2.3
  constrained:   deepdiff 9.0.0 | cachebox None
  dropped: ['cachebox']   added: []

Exactly one package leaves the graph. The file says why it exists and
when to delete it; nothing else reads it, and it must never be used for
the supported matrix.

tests/test_python_next_canary.py (6 tests) pins what keeps the lane
honest: the suite step cannot become conditional, the install must record
its mode and apply the fallback, the summary must be `if: always()` and
must say "proved nothing" when neither attempt worked, the constraint may
pin down but never below the declared floor, and the constraints file must
parse as one `pip -c` would actually honour.

Three of them were confirmed to fail against the defect they name before
being kept: adding `if: steps.install.outputs.mode == 'full'` to the suite
step, removing `if: always()` from the summary, and lowering the cap to
`deepdiff<9.0` each failed exactly one test; reverting restored green.

What is NOT verified here: that the constrained install succeeds on a real
3.15 interpreter. There is no 3.15 in this environment, the resolution
above was measured on 3.11, and this lane is non-gating — CI on this PR is
the first real evidence. If the fallback also fails, the new summary step
now says so out loud instead of reporting success.

  ruff check chainweaver/ tests/ examples/    All checks passed!
  ruff format --check                          309 files already formatted
  mypy chainweaver/ tests/                     no issues in 262 source files
  pytest tests/ -q --no-cov                    2399 passed, 1 skipped

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XhJp1Rnr6mF4vdSLkj14jU
First evidence from the fixed lane, and it changes the picture. The
fallback install worked — `Install compatibility surface` succeeded for
the first time since the lane was added — and `Run core suite` ran
instead of being skipped. It then died at collection in 7.5s:

  ImportError: cannot import name 'no_type_check_decorator' from 'typing'
  ...
  ERROR tests/test_mcp_adapter.py
  ERROR tests/test_mcp_adapter_trust.py
  ERROR tests/test_mcp_server.py
  ERROR tests/test_mcp_server_security.py
  Interrupted: 4 errors during collection
  8 skipped, 4 errors in 7.52s

That is a second 3.15 blocker, independent of the cachebox one and not
caused by the constraint: fastmcp -> key_value.aio -> beartype, and
beartype imports `typing.no_type_check_decorator`, which 3.15 removed. It
surfaces as "FastMCP server support is not installed" because fastmcp
catches the underlying error, and the four tests/test_mcp_*.py modules
import that surface unguarded at module scope — the same pattern the
free-threaded lane already documents in ci.yml.

So the lane got one step further and still reported nothing about the
other ~2400 tests. Adding --continue-on-collection-errors: the step still
fails, every error is still printed, but one unimportable dependency no
longer erases the whole signal. Not pinning fastmcp: it is precisely the
surface this canary exists to test on the next Python.

The blocker is recorded in constraints/python-next.txt next to the
cachebox one, with why it is deliberately not constrained around.

  ruff check chainweaver/ tests/ examples/    All checks passed!
  ruff format --check                          309 files already formatted
  pytest tests/test_python_next_canary.py      6 passed

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XhJp1Rnr6mF4vdSLkj14jU

dgenio commented Aug 30, 2026

Copy link
Copy Markdown
Owner Author

First evidence from the fixed lane — it works, and it immediately found a second blocker

The unverified item in the PR body is now measured. Run on fbac841, job-level:

Step Before (all 6 prior runs) This run
Install compatibility surface failure ✅ success
Run core suite skipped ran
Summarise what this lane proved did not exist ✅ success, printed the constrained branch

Interpreter confirmed 3.15.0-rc.1. The fallback install did what it was meant to do.

And the lane immediately earned its keep

It got to collection and died there in 7.5s — on something that has nothing to do with the cachebox constraint:

from beartype.typing import Tuple
E   ImportError: cannot import name 'no_type_check_decorator' from 'typing'
    (/opt/hostedtoolcache/Python/3.15.0-rc.1/x64/lib/python3.15/typing.py)
...
E   ImportError: FastMCP server support is not installed.
ERROR tests/test_mcp_adapter.py · test_mcp_adapter_trust.py · test_mcp_server.py · test_mcp_server_security.py
Interrupted: 4 errors during collection
8 skipped, 4 errors in 7.52s

A second, independent 3.15 blocker: fastmcp → key_value.aio → beartype, and beartype does from typing import no_type_check_decorator, which 3.15 removed. fastmcp catches the underlying error and re-raises it as "server support is not installed", which is why the message looks like a missing extra rather than an incompatibility. The four tests/test_mcp_*.py modules import that surface unguarded at module scope — the same pattern ci.yml already documents for the free-threaded lane.

This is exactly the class of thing the canary exists to find, and it could not have reported it in the three weeks it was silently dead.

Follow-up pushed (f0c1e1a)

The lane got one step further and still said nothing about the other ~2400 tests, so: --continue-on-collection-errors. The step still fails and every error is still printed; one unimportable dependency just no longer erases the whole signal.

Not pinning fastmcp around it — that is precisely the surface this canary is supposed to test on the next Python. The blocker is recorded in constraints/python-next.txt alongside the cachebox one, with why it is deliberately left unconstrained.

ruff check clean · ruff format --check 309 files · pytest tests/test_python_next_canary.py 6 passed.

What the lane will report from here

failure, honestly — ChainWeaver's dependency tree genuinely does not work on 3.15 yet, for two separate upstream reasons. It stays continue-on-error, so it warns rather than blocks; the difference is that it now warns about something, with a step summary saying which of the two things it proved. Both blockers are upstream and neither is yours to fix.


Generated by Claude Code

dgenio commented Aug 30, 2026

Copy link
Copy Markdown
Owner Author

f0c1e1a: 2183 tests just ran on CPython 3.15.0-rc.1

5 failed, 2183 passed, 11 skipped, 4 errors in 35.32s

Three weeks of this lane, summarised:

10 Aug – 30 Aug (6 runs) fbac841 f0c1e1a
Install failure ✅ success ✅ success
Suite skipped ran, aborted at collection ✅ ran
Tests executed 0 0 2183

The --continue-on-collection-errors change did what it was meant to: one unimportable dependency no longer erases the whole signal.

What it found, all one root cause

ERROR tests/test_mcp_adapter.py · test_mcp_adapter_trust.py · test_mcp_server.py · test_mcp_server_security.py
FAILED tests/test_cli_serve.py::TestServeCommandRegistration::test_transport_enum_matches_library_literal
       - ImportError: chainweaver.mcp.server requires the 'fastmcp' package and the 'mcp' SDK.
FAILED tests/test_cli_serve.py::TestBuildFlowServer::test_exposes_one_tool_per_flow      - typer.exceptions.Exit
FAILED tests/test_cli_serve.py::TestBuildFlowServer::test_prefix_namespaces_tool_names   - typer.exceptions.Exit
FAILED tests/test_cli_serve.py::TestBuildFlowServer::test_directory_exposes_active_withholds_draft - typer.exceptions.Exit
FAILED tests/test_cli_serve.py::TestBuildFlowServer::test_missing_flow_file_exits_2      - assert 1 == 2

All nine trace back to the single beartype/3.15 blocker: fastmcp cannot import, so chainweaver.mcp.server raises, so the CLI's serve path exits early. The assert 1 == 2 is the same thing wearing a different hat — the command exits 1 on the missing extra before it can reach its own exit-2 path.

Nothing here is a ChainWeaver bug on 3.15, which is itself the useful result: outside the MCP surface, 2183 tests pass on the next Python. The lane could not have told you that before.

Status

The check stays failure, honestly, and stays continue-on-error, so it warns rather than blocks. Both blockers are upstream (deepdiff→cachebox has no cp315 wheel; beartype imports typing.no_type_check_decorator, removed in 3.15) and neither is fixable here. No further pushes planned — this is the lane reporting the truth, which is what the PR set out to make possible.

The PR body's one unverified claim ("that the constrained install actually succeeds on a real 3.15 interpreter") is now confirmed.


Generated by Claude Code

…t-canary

# Conflicts:
#	.github/workflows/python-next.yml
@dgenio
dgenio merged commit fd8ddf4 into main Aug 30, 2026
22 of 23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants