Repository navigation
docs: make ChainWeaver validation-first before v1 - #557
Conversation
Centralize point-in-time snapshot semantics at the StepRecord boundary so later mutation of nested live-context data cannot rewrite historical trace records. Includes nested mutation and serde regressions, data-integrity documentation, and benchmark verification. Closes #398.
Adds opt-in redaction_policy= to load_agent_trace() and parse_agent_trace(), and --redact recommended|strict|none to traces mine / draft-flows / backtest. Redaction applies only to data-bearing args and vendor metadata, leaving structural mining fields untouched, so candidate IDs and ordering, support/session counts, scores, draft mappings, rendered reports and backtest verdicts are unchanged. A fresh load-scoped placeholder map keeps repeated sensitive values equal within a single load without persisting a cross-load mapping. Default behaviour is unchanged: without redaction_policy= / --redact, imported values stay raw. Negative controls cover SHA-256-like values, UUIDs and unlabelled base64; raw-vs-redacted A/B tests assert downstream equality; serialized report/draft/sidecar/backtest shapes are read back to assert the canary secret is absent. chainweaver.trace_redaction is added to BANNED_INREPO alongside its siblings in the trace family, so the module-map annotation and the enforced set stay one-to-one. Its only importer is chainweaver/traces.py, itself already banned, so the executor path was never able to reach it. Scope boundary: does not change executor-side ExecutionResult / checkpoint persistence, which remains the separate #292 storage privacy boundary. Closes #376 Refs #292 #448
#563) #341 moved the concurrency tests from clock-based sleeps to events. That removed the flake but moved the failure mode: an Event.wait() whose signal never fires no longer trips a timing assertion, it burns the runner's global timeout and reports nothing attributable. #341's own risk note anticipated it. Adds pytest-timeout to the dev extra and a 60s per-test ceiling with timeout_method = "thread", chosen from measurement rather than taste: slowest healthy test, measured at multiplier 1 4.34s tests/helpers.py BARRIER_TIMEOUT_S give-up bound 5.00s scaled() offsets sub-second 60s leaves >12x headroom over the slowest healthy test and clears BARRIER_TIMEOUT_S by an order of magnitude, so it cannot become a flake source itself. thread rather than signal because a test blocked in a C-level call is not interruptible by SIGALRM, and those are exactly the hangs this catches. The reasoning is recorded next to the setting, not just here. One correction to the issue: it asks that the ceiling clear "BARRIER_TIMEOUT_S multiplied by the largest CHAINWEAVER_TEST_TIMING_MULTIPLIER". BARRIER_TIMEOUT_S is a fixed 5.0 and is deliberately NOT scaled -- helpers.py says so explicitly. The real relationship is BARRIER_TIMEOUT_S plus scaled work, so the ceiling is set against that instead. The multiplier is also not set anywhere under .github/, so CI runs at 1 today. No per-lane override added: at a 4.34s worst case nothing comes close to 60s, so overrides would be unused machinery. Raise the single value if that changes. Verified the ceiling actually fires rather than assuming it -- a throwaway test calling threading.Event().wait() produced a named Timeout with a traceback at the blocking line, not a hang. Probe removed before commit. Same test count with and without this change, identical flags: main 2372 passed, 1 skipped this branch 2372 passed, 1 skipped ruff check All checks passed! ruff format 263 files already formatted mypy Success: no issues found in 256 source files coverage 93.57% (gate 80) Closes #543 Co-authored-by: Claude <noreply@anthropic.com>
The dev extra mixed tools that run this repository's own gate (ruff, mypy, nbmake, a Jupyter kernel) with the integration packages the suite needs. Only the second kind belongs in wheel metadata. - dev extra removed; maintainer tooling moves to [dependency-groups], which is never published. Verified by building the wheel: Provides-Extra no longer lists dev, and the metadata no longer mentions ruff, nbmake or opentelemetry-sdk. - New published extra `integrations`: every user-facing integration extra at once, named once instead of in eight workflow steps plus four documents. Excludes llm-anthropic/llm-openai; evals.yml adds the provider it needs. - Second group `test-runners` holds the four pytest packages. The free-threaded 3.14t lane and the Python-next canary had each hand-copied that list, and neither copy gained pytest-timeout when #543 added it — the two lanes most likely to hang were the only two running unbounded. - Every --group install upgrades pip first (PEP 735 needs pip >= 25.1). Resolution unchanged, measured: both compositions resolve the same 193 distributions at the same versions, empty set difference both ways. tests/test_dependency_groups.py (8 tests) fails if maintainer tooling reappears in a published extra, if a workflow installs .[dev] or pins a pytest runner inline, or if the integrations composition and the integration extras disagree. Closes #550
The Python Next Canary had never once run the test suite. From the commit that added it on 2026-08-10 through 2026-08-30, every run failed at "Install compatibility surface" and skipped "Run core suite"; job-level continue-on-error reported each workflow run as success, so five consecutive scheduled canaries looked green and proved nothing. The blocker is upstream: deepdiff 9.1.0 requires cachebox<6,>=5.2, the newest cachebox 5.x is 5.2.3 whose wheels stop at cp314, and no cachebox 5.x ships a cp315 wheel, so pip builds it from source on 3.15 and PyO3 refuses the interpreter. cachebox 6.1.2+ has cp315 wheels but the <6 cap excludes them. That is one of the two things this lane exists to report; the other — whether ChainWeaver's own code works on the next Python — was being swallowed by it. The install now tries the real thing first and records the outcome, falls back to constraints/python-next.txt so the suite still runs, and an always-run summary states which of the two it proved, including the case where neither install worked. The constraint is deepdiff<9.1 and nothing else; measured, it removes exactly one package (cachebox) from the graph and satisfies the declared deepdiff>=9.0 floor. Result on 3.15.0-rc.1: install succeeded for the first time and 2183 tests ran, surfacing a second independent blocker the lane could never report before — fastmcp -> key_value.aio -> beartype imports typing.no_type_check_decorator, removed in 3.15. --continue-on-collection-errors keeps one unimportable module from erasing the other ~2400 tests. The lane stays continue-on-error and will keep reporting failure honestly while both upstream blockers stand. tests/test_python_next_canary.py (6 tests); three were confirmed to fail against the defect they name before being kept.
|
Reviewed as part of a sweep of every open PR. Leaving this one to you deliberately — and it is worth saying why, because "stale draft" is the wrong reading. Not superseded. All five added files ( Two blocking failures, and they are current — not stale CI. Reproduced by merging this head into today's main locally:
Why I am not just fixing those and merging. The failures are the mechanical half. The other half is not mine to decide:
Your own PR body says the draft status is intentional "so the strategy/security claims can be reviewed before becoming the public contract." That is exactly right, and an automated reviewer approving it would be overstepping. If you want the mechanical half cleared while you decide the positioning, say so and I will fix the vocabulary and README-guard failures and leave it in draft, green. Not verified, flagged rather than assumed: the body's claims about re-prioritising #555, #556, #497, #334, #498, #231, #325, #407 and #380 (I checked only #553 and #558), and whether a docs PR here wants a CHANGELOG entry. Generated by Claude Code |
Both failures reproduced against current main before fixing. **Vocabulary gate.** Three flagged uses of "pipeline", handled differently because they are not the same case. The allowlist's own rule is to add entries only for genuine non-flow uses and "never to paper over a flow that should just be called a flow", so: - `docs/comparisons.md` "recurring pipelines" — allowlisted. It describes what Prefect and Dagster are for, in the passage recommending them instead. Their vocabulary, not ours. - `docs/v1-release-criteria.md` "The release pipeline" (#519) — allowlisted. The CI/release process, not a ChainWeaver flow. - `docs/v1-release-criteria.md` "disappearing from the pipeline" — reworded to "disappearing from the recommendation". This one *was* using "pipeline" as a synonym for ChainWeaver's own processing, which is exactly the misuse the rule bans. Allowlisting it would have papered over the thing the gate exists to catch. **README asset guard (#227).** The rewrite dropped both the headline numbers and the regeneration command, while still calling the report "reproducible". Split rather than deleted: - Retired the quantified-headline half, with the reason recorded in the test's docstring: a synthetic executor benchmark is evidence about the executor, not about product-market fit, and leading on it overstated what it proves. That is this PR's deliberate repositioning. - Kept the reproducibility half enforced, and restored `python benchmarks/report.py` to the README callout so the word "reproducible" is backed by instructions. Without it the README asserts reproducibility while telling the reader nothing about how — precisely the unbacked claim #227 exists to prevent. Verified with CI's own invocations (`ruff check chainweaver/ tests/ examples/`, `ruff format --check` the same, `mypy chainweaver/ tests/`, `scripts/check_vocabulary.py`): all clean. Full suite 2399 passed, 1 skipped. Note: `ruff format --check .` reports 13 files needing reformatting, but that is pre-existing on main and outside CI's checked paths — not introduced here and not silently fixed here either. Co-Authored-By: Claude <noreply@anthropic.com>
|
Both blocking failures fixed in Vocabulary gate. Three flagged uses, handled differently — they are not the same case, and the allowlist's own rule is to add entries only for genuine non-flow uses, "never to paper over a flow that should just be called a flow":
The third was genuinely using "pipeline" as a synonym for ChainWeaver's own processing — the misuse the rule exists to catch. Allowlisting it would have hollowed out the gate. README asset guard (#227). Split rather than deleted, because only half of it was actually superseded:
Verified with CI's own invocations, not my own approximations: One thing I did not touch: Merging once CI is green. Generated by Claude Code |
What changed
This PR turns the red-team findings into repository policy instead of leaving them as advice.
Product thesis becomes falsifiable
docs/product-validation.mddocs/validation-participant-guide.mdso external participants can run the experiment locally without sharing sensitive raw tracesIndependent validation resultissue form that requires negative evidence, the pre-analysis baseline, manual comparison, security result, and keep/uninstall verdictSecurity becomes a product invariant
docs/macro-capability-security.md, including a current-state audit: FlowServer has macro-level caller authentication/authorization and FlowExecutor has child safety/approval gates, but caller identity is not yet threaded into per-child executor authorizationv1 becomes evidence-gated
Rewrites
docs/v1-release-criteria.mdsov1.0.0requires more than feature completion:Stars are explicitly not a v1 criterion.
API compatibility stops growing accidentally
docs/api-tiers.md__all__before P0: falsify the product thesis on independent real agent traces before more lifecycle architecture #553, because independent validation may reveal that the durable product is more analyzer/compiler-centric than runtime-centricREADME/docs positioning
The moat — observe → compile → replaceframingWhy not just write a Python function?answer: often, you shouldcurrentroadmap with the actual validation-first gatesRoadmap gating already reflected in issues
Related issue changes made separately:
flow_to_callable/framework adapters solve adapter portability, not true execution portabilityWhy
The repository's engineering maturity is substantially ahead of its external adoption evidence. The red-team review found the most dangerous failure mode is continuing to build a sophisticated lifecycle because it is architecturally coherent, without first proving that independent teams value the discovery/evidence/governance job beyond
def macro(...): ...or their existing agent framework.This PR makes that uncertainty explicit and turns it into measurable release and roadmap gates.
Scope
Documentation, community workflow, and package/docs metadata only. No runtime behavior changes.
The caller-specific child-authorization gap discovered during the audit is deliberately documented, not papered over with a guessed public API. #558 should challenge the model before #554 freezes core principal/policy contracts.
Validation
The PR remains intentionally draft so the strategy/security claims can be reviewed before becoming the public contract.