Skip to content

playwright: the GenClass fixture (seeded chaos, one race file per test) - #72

Merged
MeharPro merged 1 commit into
mainfrom
claude/wizardly-hypatia-713pwa
Oct 25, 2026
Merged

MeharPro merged 1 commit into
mainfrom
claude/wizardly-hypatia-713pwa

Conversation

@MeharPro

@MeharPro MeharPro commented Oct 24, 2026 •

Copy link
Copy Markdown

What

@genclass/playwright/fixture: Playwright's test (test.extend) and expect with a genclass option, so the reporter that already shipped in this package has something to read. On every page the fixture puts @genclass/runtime's script-tag build before the app's first script (observe mode, telemetry off, visual checks and devtools off), served from the installed package's dist/ on the page's own origin at /__genclass__/ with page.route (never a CDN). The model is skipped by default (no 13 MB download): the file then carries the runtime's witnesses (overwrote-newer, identical-in-flight), computed without the model through the agent entry connected with every hold off; model: "load" adds the model's detections with their confidence. Seeded chaos (chaos: { seed, delayMs, reorder, match }) holds each matched fetch/XHR request for a latency derived from the seed and the request's key, then passes it on with route.fallback(), so it still reaches the test's own routes and the browser's network stack and a run repeats exactly; reorder: false sends matched requests in order, at least 30 ms apart. At the end of each test it writes one genclass-races/1 file with the seed, the chaos settings and a fixture record of what it could and could not do (the report notes runs it could not check). enabled: false is inert.

Observe changes nothing in the app under test. The runtime runs in observe mode: it never holds, delays or changes anything the app does; the agent entry is connected with hold: false and a stand-in fetch for its own address, so it sends nothing (no console error, failed request or CSP report). A URL or localStorage switch that raised the mode is set back to observe. Chaos changes only when a request leaves, never what it says, only for fetch/XHR requests inside the filter, and is off unless given. No finding changes a test's outcome.

The summariser gives witness signatures (trigger witness) their own group, shown only when present, no confidence on the line and a footer sentence; files without a fixture record are reported exactly as before (the README block and test/package.test.ts still match).

Checks

  • npm run typecheck -w @genclass/runtime and npm run build -w @genclass/runtime pass (no runtime source is touched; the build was run because the e2e serves packages/runtime/dist/)
  • unit tests pass: cd packages/playwright && npx vitest run, 114 tests in 9 files, with new tests for the chaos scheduler (deterministic by seed, nothing outside the filter, FIFO with reorder: false), the race-file builder (validates against the reporter's schema), the installer on a fake page, the fixture entry (inert when disabled) and the witness rendering. The runtime's own suite was not run (unchanged)
  • no packages/runtime/test/review-*.test.ts was edited
  • no model-visible text changed (nothing under packages/runtime/src/)
  • docs/runtime/API.md, the runtime's src/types.ts and packages/runtime/CHANGELOG.md: not applicable (runtime untouched). The package's own README.md, CHANGELOG.md and src/types.ts JSDoc are updated; the root README.md package table and AGENTS.md repo map describe the fixture
  • a correct app is not made worse: observe only; the holds are the test harness's own (CONTRACT.md section 16 exception, hold: false on the agent link), bounded by delayMs (at most 10 s) and released when the page closes
  • sim/ and demos/ are untouched
  • the root package-lock.json is updated in the same commit (npm install output: the workspace entry gains the peer and dev dependencies, and npm added "peer": true to the 26 @esbuild/* platform entries, which is its own bookkeeping for packages reachable through a peer edge)

Notes for the reviewer

Look first at packages/playwright/src/install.ts (the chaos route: plan when the request is sent, hold, route.fallback(); the runtime files route), src/page.ts (what runs in the page before the app's first script, serialised with toString: the agent link's stand-in fetch, the mode hold, detections from decide), and the fixture record and notes in src/schema.ts and src/summarize.ts.

Review. The first commit went through an adversarial review (6 reviewers, then 3 skeptics per finding): 54 of 55 findings upheld, none blockers, 16 major. The second commit, 70c8aed, applies them; its message lists each. The largest: chaos now holds requests instead of fetching responses from Node (user mocks, streams, redirects, cookies and CORS now untouched; keys in request order, so seeds reproduce), the fixture no longer logs a console error per page load, detection confidence is the diagnosis probability, and the CI e2e step now runs with pipefail (before, it could not fail).

Measured (packages/playwright/test/e2e/run.mjs, the second job of .github/workflows/playwright-reporter.yml; headless Chromium 153, Playwright 1.63.0, 2026-10-25, model: "skip", six runs of five seeds, so 30 seeds per row), on the MCP e2e search page (packages/mcp/test/e2e/app/app.js, its store on the fixture's runtime through the server's script-tag mapping), typing "react" at 80 ms per key against a backend that answers in 40 ms in order:

page chaos seeds with witness:overwrote-newer on GET /api/search any other signature console errors or failed requests
buggy latency 0 to 600 ms, reorder 30 of 30 0 0
buggy latency 0 to 600 ms, latency only (reorder: false) 0 of 30 0 0
fixed (window.__FIXED__) latency 0 to 600 ms, reorder 0 of 30 0 0

The latency-only row is the check that the witness comes from the order, not from the latency; the fixed row is the false-report count on the correct variant. With model: "load" (one run by hand on the first commit, two seeds, not in the workflow because it downloads the model): the model loaded through the fixture's path (genclass-runtime-r17 2.0.0-rc4t, WASM, in a worker), the witness was recorded in 2 of 2 seeds, and the model gave no likely detection on that page (stale at 0.64 in seed 1, below the 0.90 report gate; no decision in seed 2), which matches the compatibility matrix and is why witnesses are the default.

Things to know:

  • The e2e serves the MCP app's app.js untouched but with an import map that resolves @genclass/runtime to the page's window.GenClass (the script-tag build the fixture injected, as an app integrated through the script tag would call it) and stubs @genclass/runtime/agent (the MCP bridge link, not run here). An app that starts its own ESM runtime runs a second runtime next to the fixture's, and stores it registers with rt.atom or rt.guard are not visible to the fixture's runtime; the README's limits section says so.
  • Peer dependencies only: @playwright/test and @genclass/runtime, both optional (the reporter needs neither). Nothing new at run time; nothing under packages/runtime/src/ changed.
  • The task asked to update docs/runtime/README.md and llms.txt. Neither is on main; both are added by the open docs PR, Docs: answer-first READMEs, FAQ, docs index, llms.txt, trimmed npm page; factual fixes #69. This PR updates what main has (the root README.md package table and AGENTS.md). Both files in that PR still say the fixture "has not shipped", so whichever of the two merges second should replace those lines with these:
    • llms.txt line 52: - [Playwright fixture and reporter](https://raw.githubusercontent.com/genclass-dev/GenClass-lib/main/packages/playwright/README.md): `@genclass/playwright` 0.1.0, not on npm yet: a fixture that puts the runtime on every page under test (observe mode, seeded network chaos, the model skipped by default) and writes one race file per test, and a reporter that turns those files into one markdown report per pull request
    • docs/runtime/README.md line 43: | [Playwright fixture and reporter](../../packages/playwright/README.md) | How do I get one pull-request report of what GenClass flagged in my Playwright tests? | reference (`@genclass/playwright` 0.1.0, not on npm yet) | |
  • .github/workflows/mcp.yml pipes its e2e into tee without pipefail, the same bug fixed here; it is outside this PR.
  • Not run: the runtime's own vitest suite and Playwright suites (unchanged), test/smoke/smoke.sh, the install suite, anything on Azure. npm publish was not run.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Fjo9SusiHz3LvqdFnr5wYP

@genclass/playwright/fixture: Playwright's `test` (test.extend) and `expect`
with a `genclass` option. On every `page` it puts @genclass/runtime's
script-tag build before the app's first script (observe mode, telemetry off,
no visual checks or devtools), served from the installed package's dist/ on
the page's origin at /__genclass__/ with page.route, never from a CDN. The
model is skipped by default: the file carries the runtime's witnesses
(overwrote-newer, identical-in-flight), computed without the model through
the agent entry connected with every hold off; model: "load" adds the
model's detections with their confidence. Seeded chaos (page.route: the
real response is fetched at once and its delivery held for a latency derived
from the seed and the request's key; reorder: false keeps the order) applies
to fetch/XHR requests inside the filter only, never to the runtime's files.
One genclass-races/1 file per test in test-results/genclass/, with the seed
and the chaos settings. enabled: false is inert.

The summariser gives witness signatures (trigger "witness") their own group,
shown only when present, no confidence on the line and a footer sentence;
state "off" (no model by choice) gets no "model not ready" note. The
model-only report is byte for byte unchanged.

Unit tests: the seeded scheduler (deterministic, nothing outside the filter,
FIFO with reorder off), the race-file builder (validates against the
reporter's schema), the installer on a fake page, the fixture entry (inert
when disabled), the witness rendering. End to end (test/e2e, now in the
workflow with Chromium): on the MCP e2e search page with latency 0 to 600 ms
and reordering, the buggy page gets witness:overwrote-newer on
GET /api/search in 5 of 5 seeds, the same page with latency only in 0 of 5,
the fixed page in 0 of 5, no other signature anywhere.

Peer dependencies only (@playwright/test, @genclass/runtime, both optional);
the root lockfile updated in the same commit. README rewritten with the
fixture first; CHANGELOG, the root README table and AGENTS.md updated.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Fjo9SusiHz3LvqdFnr5wYP
@MeharPro
MeharPro merged commit 6e729fa into main Oct 25, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants