Skip to content

docs: make ChainWeaver validation-first before v1 - #557

Merged
dgenio merged 24 commits into
mainfrom
agent/product-validation-gates
Aug 30, 2026
Merged

dgenio merged 24 commits into
mainfrom
agent/product-validation-gates

Conversation

@dgenio

@dgenio dgenio commented Aug 10, 2026 •

Copy link
Copy Markdown
Owner

What changed

This PR turns the red-team findings into repository policy instead of leaving them as advice.

Product thesis becomes falsifiable

  • adds docs/product-validation.md
  • makes P0: falsify the product thesis on independent real agent traces before more lifecycle architecture #553 the primary independent-trace product gate
  • requires a human/manual-function baseline rather than comparing only against a deliberately naive agent loop
  • records explicit kill/pivot outcomes if users value analysis but not the runtime, or the runtime but not discovery
  • adds docs/validation-participant-guide.md so external participants can run the experiment locally without sharing sensitive raw traces
  • adds an Independent validation result issue form that requires negative evidence, the pre-analysis baseline, manual comparison, security result, and keep/uninstall verdict

Security becomes a product invariant

v1 becomes evidence-gated

Rewrites docs/v1-release-criteria.md so v1.0.0 requires more than feature completion:

Stars are explicitly not a v1 criterion.

API compatibility stops growing accidentally

README/docs positioning

  • removes the previous The moat — observe → compile → replace framing
  • says directly that deterministic execution itself is not a defensible moat
  • adds a prominent Why not just write a Python function? answer: often, you should
  • reframes ChainWeaver around trace evidence → useful rejection → review → governed deterministic capability
  • labels executor benchmarks as executor evidence, not product-market-fit evidence
  • makes MCP an interoperability surface rather than the product category
  • replaces the stale v0.9 current roadmap with the actual validation-first gates
  • aligns PyPI/MkDocs package descriptions with the same thesis

Roadmap gating already reflected in issues

Related issue changes made separately:

Why

The repository's engineering maturity is substantially ahead of its external adoption evidence. The red-team review found the most dangerous failure mode is continuing to build a sophisticated lifecycle because it is architecturally coherent, without first proving that independent teams value the discovery/evidence/governance job beyond def macro(...): ... or their existing agent framework.

This PR makes that uncertainty explicit and turns it into measurable release and roadmap gates.

Scope

Documentation, community workflow, and package/docs metadata only. No runtime behavior changes.

The caller-specific child-authorization gap discovered during the audit is deliberately documented, not papered over with a guessed public API. #558 should challenge the model before #554 freezes core principal/policy contracts.

Validation

  • MkDocs strict build should include the new validation, API-tier, participant, and security pages
  • existing README/docs link checks should cover the new links and navigation
  • package metadata checks should validate the new description
  • no public runtime API or serialization behavior changes in this PR

The PR remains intentionally draft so the strategy/security claims can be reviewed before becoming the public contract.

dgenio and others added 8 commits August 10, 2026 23:51
Centralize point-in-time snapshot semantics at the StepRecord boundary so later mutation of nested live-context data cannot rewrite historical trace records. Includes nested mutation and serde regressions, data-integrity documentation, and benchmark verification.

Closes #398.
Adds opt-in redaction_policy= to load_agent_trace() and
parse_agent_trace(), and --redact recommended|strict|none to traces
mine / draft-flows / backtest.

Redaction applies only to data-bearing args and vendor metadata, leaving
structural mining fields untouched, so candidate IDs and ordering,
support/session counts, scores, draft mappings, rendered reports and
backtest verdicts are unchanged. A fresh load-scoped placeholder map
keeps repeated sensitive values equal within a single load without
persisting a cross-load mapping.

Default behaviour is unchanged: without redaction_policy= / --redact,
imported values stay raw.

Negative controls cover SHA-256-like values, UUIDs and unlabelled
base64; raw-vs-redacted A/B tests assert downstream equality; serialized
report/draft/sidecar/backtest shapes are read back to assert the canary
secret is absent.

chainweaver.trace_redaction is added to BANNED_INREPO alongside its
siblings in the trace family, so the module-map annotation and the
enforced set stay one-to-one. Its only importer is chainweaver/traces.py,
itself already banned, so the executor path was never able to reach it.

Scope boundary: does not change executor-side ExecutionResult /
checkpoint persistence, which remains the separate #292 storage privacy
boundary.

Closes #376
Refs #292 #448
#563)

#341 moved the concurrency tests from clock-based sleeps to events. That
removed the flake but moved the failure mode: an Event.wait() whose signal
never fires no longer trips a timing assertion, it burns the runner's global
timeout and reports nothing attributable. #341's own risk note anticipated it.

Adds pytest-timeout to the dev extra and a 60s per-test ceiling with
timeout_method = "thread", chosen from measurement rather than taste:

  slowest healthy test, measured at multiplier 1        4.34s
  tests/helpers.py BARRIER_TIMEOUT_S give-up bound      5.00s
  scaled() offsets                                      sub-second

60s leaves >12x headroom over the slowest healthy test and clears
BARRIER_TIMEOUT_S by an order of magnitude, so it cannot become a flake source
itself. thread rather than signal because a test blocked in a C-level call is
not interruptible by SIGALRM, and those are exactly the hangs this catches.
The reasoning is recorded next to the setting, not just here.

One correction to the issue: it asks that the ceiling clear "BARRIER_TIMEOUT_S
multiplied by the largest CHAINWEAVER_TEST_TIMING_MULTIPLIER". BARRIER_TIMEOUT_S
is a fixed 5.0 and is deliberately NOT scaled -- helpers.py says so explicitly.
The real relationship is BARRIER_TIMEOUT_S plus scaled work, so the ceiling is
set against that instead. The multiplier is also not set anywhere under
.github/, so CI runs at 1 today.

No per-lane override added: at a 4.34s worst case nothing comes close to 60s,
so overrides would be unused machinery. Raise the single value if that changes.

Verified the ceiling actually fires rather than assuming it -- a throwaway test
calling threading.Event().wait() produced a named Timeout with a traceback at
the blocking line, not a hang. Probe removed before commit.

Same test count with and without this change, identical flags:
  main            2372 passed, 1 skipped
  this branch     2372 passed, 1 skipped
  ruff check      All checks passed!
  ruff format     263 files already formatted
  mypy            Success: no issues found in 256 source files
  coverage        93.57% (gate 80)

Closes #543

Co-authored-by: Claude <noreply@anthropic.com>
The dev extra mixed tools that run this repository's own gate (ruff, mypy, nbmake, a Jupyter kernel) with the integration packages the suite needs. Only the second kind belongs in wheel metadata.

- dev extra removed; maintainer tooling moves to [dependency-groups], which is never published. Verified by building the wheel: Provides-Extra no longer lists dev, and the metadata no longer mentions ruff, nbmake or opentelemetry-sdk.
- New published extra `integrations`: every user-facing integration extra at once, named once instead of in eight workflow steps plus four documents. Excludes llm-anthropic/llm-openai; evals.yml adds the provider it needs.
- Second group `test-runners` holds the four pytest packages. The free-threaded 3.14t lane and the Python-next canary had each hand-copied that list, and neither copy gained pytest-timeout when #543 added it — the two lanes most likely to hang were the only two running unbounded.
- Every --group install upgrades pip first (PEP 735 needs pip >= 25.1).

Resolution unchanged, measured: both compositions resolve the same 193 distributions at the same versions, empty set difference both ways.

tests/test_dependency_groups.py (8 tests) fails if maintainer tooling reappears in a published extra, if a workflow installs .[dev] or pins a pytest runner inline, or if the integrations composition and the integration extras disagree.

Closes #550
The Python Next Canary had never once run the test suite. From the commit that added it on 2026-08-10 through 2026-08-30, every run failed at "Install compatibility surface" and skipped "Run core suite"; job-level continue-on-error reported each workflow run as success, so five consecutive scheduled canaries looked green and proved nothing.

The blocker is upstream: deepdiff 9.1.0 requires cachebox<6,>=5.2, the newest cachebox 5.x is 5.2.3 whose wheels stop at cp314, and no cachebox 5.x ships a cp315 wheel, so pip builds it from source on 3.15 and PyO3 refuses the interpreter. cachebox 6.1.2+ has cp315 wheels but the <6 cap excludes them.

That is one of the two things this lane exists to report; the other — whether ChainWeaver's own code works on the next Python — was being swallowed by it. The install now tries the real thing first and records the outcome, falls back to constraints/python-next.txt so the suite still runs, and an always-run summary states which of the two it proved, including the case where neither install worked.

The constraint is deepdiff<9.1 and nothing else; measured, it removes exactly one package (cachebox) from the graph and satisfies the declared deepdiff>=9.0 floor.

Result on 3.15.0-rc.1: install succeeded for the first time and 2183 tests ran, surfacing a second independent blocker the lane could never report before — fastmcp -> key_value.aio -> beartype imports typing.no_type_check_decorator, removed in 3.15. --continue-on-collection-errors keeps one unimportable module from erasing the other ~2400 tests.

The lane stays continue-on-error and will keep reporting failure honestly while both upstream blockers stand.

tests/test_python_next_canary.py (6 tests); three were confirmed to fail against the defect they name before being kept.

dgenio commented Aug 30, 2026

Copy link
Copy Markdown
Owner Author

Reviewed as part of a sweep of every open PR. Leaving this one to you deliberately — and it is worth saying why, because "stale draft" is the wrong reading.

Not superseded. All five added files (docs/product-validation.md, docs/api-tiers.md, docs/macro-capability-security.md, docs/validation-participant-guide.md, .github/ISSUE_TEMPLATE/independent_validation.yml) are absent from origin/main. docs/v1-release-criteria.md on main is byte-identical to this PR's merge base, and README has moved only 9/−3, all from #564. It still merges cleanly.

Two blocking failures, and they are current — not stale CI. Reproduced by merging this head into today's main locally:

  1. Vocabulary gate — scripts/check_vocabulary.py flags docs/comparisons.md:152 ("recurring pipelines"), docs/v1-release-criteria.md:155 ("disappearing from the pipeline") and :296 ("The release pipeline"). Reading .vocabulary-allowlist.txt, the first and third look exactly like the generic-software-pipeline case the allowlist exists to permit.
  2. README asset guard — the rewrite drops the quantified reproducible-benchmark claim that tests/test_readme_assets.py:80-82 mandates (the guard for benchmark: publish one headline cost/latency/correctness number with a reproducible harness (README hero) #227). Either the guard is amended with a recorded decision, or the reproduction pointer comes back. Merging as-is leaves a red guard encoding an abandoned policy.

ruff, ruff format and mypy all pass; the rest of the suite is 2382 passed, 1 skipped, and plain main passes those two files. So the failures are this PR's, and nothing else is broken.

Why I am not just fixing those and merging. The failures are the mechanical half. The other half is not mine to decide:

Your own PR body says the draft status is intentional "so the strategy/security claims can be reviewed before becoming the public contract." That is exactly right, and an automated reviewer approving it would be overstepping.

If you want the mechanical half cleared while you decide the positioning, say so and I will fix the vocabulary and README-guard failures and leave it in draft, green.

Not verified, flagged rather than assumed: the body's claims about re-prioritising #555, #556, #497, #334, #498, #231, #325, #407 and #380 (I checked only #553 and #558), and whether a docs PR here wants a CHANGELOG entry.


Generated by Claude Code

claude added 2 commits August 30, 2026 11:20
Both failures reproduced against current main before fixing.

**Vocabulary gate.** Three flagged uses of "pipeline", handled differently
because they are not the same case. The allowlist's own rule is to add entries
only for genuine non-flow uses and "never to paper over a flow that should just
be called a flow", so:

- `docs/comparisons.md` "recurring pipelines" — allowlisted. It describes what
  Prefect and Dagster are for, in the passage recommending them instead. Their
  vocabulary, not ours.
- `docs/v1-release-criteria.md` "The release pipeline" (#519) — allowlisted.
  The CI/release process, not a ChainWeaver flow.
- `docs/v1-release-criteria.md` "disappearing from the pipeline" — reworded to
  "disappearing from the recommendation". This one *was* using "pipeline" as a
  synonym for ChainWeaver's own processing, which is exactly the misuse the
  rule bans. Allowlisting it would have papered over the thing the gate exists
  to catch.

**README asset guard (#227).** The rewrite dropped both the headline numbers
and the regeneration command, while still calling the report "reproducible".
Split rather than deleted:

- Retired the quantified-headline half, with the reason recorded in the test's
  docstring: a synthetic executor benchmark is evidence about the executor, not
  about product-market fit, and leading on it overstated what it proves. That
  is this PR's deliberate repositioning.
- Kept the reproducibility half enforced, and restored `python
  benchmarks/report.py` to the README callout so the word "reproducible" is
  backed by instructions. Without it the README asserts reproducibility while
  telling the reader nothing about how — precisely the unbacked claim #227
  exists to prevent.

Verified with CI's own invocations (`ruff check chainweaver/ tests/ examples/`,
`ruff format --check` the same, `mypy chainweaver/ tests/`,
`scripts/check_vocabulary.py`): all clean. Full suite 2399 passed, 1 skipped.

Note: `ruff format --check .` reports 13 files needing reformatting, but that
is pre-existing on main and outside CI's checked paths — not introduced here
and not silently fixed here either.

Co-Authored-By: Claude <noreply@anthropic.com>
@dgenio
dgenio marked this pull request as ready for review August 30, 2026 11:25

dgenio commented Aug 30, 2026

Copy link
Copy Markdown
Owner Author

Both blocking failures fixed in 7512070, and marked ready for review per your decision to merge the repositioning.

Vocabulary gate. Three flagged uses, handled differently — they are not the same case, and the allowlist's own rule is to add entries only for genuine non-flow uses, "never to paper over a flow that should just be called a flow":

Use Action
comparisons.md "recurring pipelines" allowlisted — describes what Prefect/Dagster are for, in the passage recommending them instead
v1-release-criteria.md "The release pipeline" (#519) allowlisted — the CI/release process
v1-release-criteria.md "disappearing from the pipeline" reworded to "disappearing from the recommendation"

The third was genuinely using "pipeline" as a synonym for ChainWeaver's own processing — the misuse the rule exists to catch. Allowlisting it would have hollowed out the gate.

README asset guard (#227). Split rather than deleted, because only half of it was actually superseded:

  • Retired the quantified-headline requirement, with the reason recorded in the test docstring. A synthetic executor benchmark is evidence about the executor, not about product-market fit — which is this PR's own argument.
  • Kept the reproducibility requirement enforced, and restored python benchmarks/report.py to the README callout. The rewrite still calls the report "reproducible" while removing any indication of how to reproduce it; that is an unbacked claim, and unbacked claims are what benchmark: publish one headline cost/latency/correctness number with a reproducible harness (README hero) #227 exists to prevent. The headline numbers stay gone.

Verified with CI's own invocations, not my own approximations: ruff check chainweaver/ tests/ examples/ → all passed; ruff format --check the same paths → 309 files formatted; mypy chainweaver/ tests/ → 262 source files clean; scripts/check_vocabulary.py → exit 0. Full suite 2399 passed, 1 skipped.

One thing I did not touch: ruff format --check . reports 13 files needing reformatting, but that reproduces identically on pristine main and sits outside the paths CI checks. Pre-existing, and not something to fix silently inside this PR.

Merging once CI is green.


Generated by Claude Code

@dgenio
dgenio merged commit bb62666 into main Aug 30, 2026
22 of 23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants