Repository navigation
playwright: the GenClass fixture (seeded chaos, one race file per test) - #72
Merged
Merged
Conversation
@genclass/playwright/fixture: Playwright's `test` (test.extend) and `expect` with a `genclass` option. On every `page` it puts @genclass/runtime's script-tag build before the app's first script (observe mode, telemetry off, no visual checks or devtools), served from the installed package's dist/ on the page's origin at /__genclass__/ with page.route, never from a CDN. The model is skipped by default: the file carries the runtime's witnesses (overwrote-newer, identical-in-flight), computed without the model through the agent entry connected with every hold off; model: "load" adds the model's detections with their confidence. Seeded chaos (page.route: the real response is fetched at once and its delivery held for a latency derived from the seed and the request's key; reorder: false keeps the order) applies to fetch/XHR requests inside the filter only, never to the runtime's files. One genclass-races/1 file per test in test-results/genclass/, with the seed and the chaos settings. enabled: false is inert. The summariser gives witness signatures (trigger "witness") their own group, shown only when present, no confidence on the line and a footer sentence; state "off" (no model by choice) gets no "model not ready" note. The model-only report is byte for byte unchanged. Unit tests: the seeded scheduler (deterministic, nothing outside the filter, FIFO with reorder off), the race-file builder (validates against the reporter's schema), the installer on a fake page, the fixture entry (inert when disabled), the witness rendering. End to end (test/e2e, now in the workflow with Chromium): on the MCP e2e search page with latency 0 to 600 ms and reordering, the buggy page gets witness:overwrote-newer on GET /api/search in 5 of 5 seeds, the same page with latency only in 0 of 5, the fixed page in 0 of 5, no other signature anywhere. Peer dependencies only (@playwright/test, @genclass/runtime, both optional); the root lockfile updated in the same commit. README rewritten with the fixture first; CHANGELOG, the root README table and AGENTS.md updated. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Fjo9SusiHz3LvqdFnr5wYP
8 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
@genclass/playwright/fixture: Playwright'stest(test.extend) andexpectwith agenclassoption, so the reporter that already shipped in this package has something to read. On everypagethe fixture puts@genclass/runtime's script-tag build before the app's first script (observe mode, telemetry off, visual checks and devtools off), served from the installed package'sdist/on the page's own origin at/__genclass__/withpage.route(never a CDN). The model is skipped by default (no 13 MB download): the file then carries the runtime's witnesses (overwrote-newer,identical-in-flight), computed without the model through the agent entry connected with every hold off;model: "load"adds the model's detections with their confidence. Seeded chaos (chaos: { seed, delayMs, reorder, match }) holds each matched fetch/XHR request for a latency derived from the seed and the request's key, then passes it on withroute.fallback(), so it still reaches the test's own routes and the browser's network stack and a run repeats exactly;reorder: falsesends matched requests in order, at least 30 ms apart. At the end of each test it writes onegenclass-races/1file with the seed, the chaos settings and afixturerecord of what it could and could not do (the report notes runs it could not check).enabled: falseis inert.Observe changes nothing in the app under test. The runtime runs in observe mode: it never holds, delays or changes anything the app does; the agent entry is connected with
hold: falseand a stand-in fetch for its own address, so it sends nothing (no console error, failed request or CSP report). A URL or localStorage switch that raised the mode is set back to observe. Chaos changes only when a request leaves, never what it says, only for fetch/XHR requests inside the filter, and is off unless given. No finding changes a test's outcome.The summariser gives witness signatures (trigger
witness) their own group, shown only when present, no confidence on the line and a footer sentence; files without afixturerecord are reported exactly as before (the README block andtest/package.test.tsstill match).Checks
npm run typecheck -w @genclass/runtimeandnpm run build -w @genclass/runtimepass (no runtime source is touched; the build was run because the e2e servespackages/runtime/dist/)cd packages/playwright && npx vitest run, 114 tests in 9 files, with new tests for the chaos scheduler (deterministic by seed, nothing outside the filter, FIFO withreorder: false), the race-file builder (validates against the reporter's schema), the installer on a fake page, the fixture entry (inert when disabled) and the witness rendering. The runtime's own suite was not run (unchanged)packages/runtime/test/review-*.test.tswas editedpackages/runtime/src/)docs/runtime/API.md, the runtime'ssrc/types.tsandpackages/runtime/CHANGELOG.md: not applicable (runtime untouched). The package's ownREADME.md,CHANGELOG.mdandsrc/types.tsJSDoc are updated; the rootREADME.mdpackage table andAGENTS.mdrepo map describe the fixturehold: falseon the agent link), bounded bydelayMs(at most 10 s) and released when the page closessim/anddemos/are untouchedpackage-lock.jsonis updated in the same commit (npm installoutput: the workspace entry gains the peer and dev dependencies, and npm added"peer": trueto the 26@esbuild/*platform entries, which is its own bookkeeping for packages reachable through a peer edge)Notes for the reviewer
Look first at
packages/playwright/src/install.ts(the chaos route: plan when the request is sent, hold,route.fallback(); the runtime files route),src/page.ts(what runs in the page before the app's first script, serialised withtoString: the agent link's stand-in fetch, the mode hold, detections fromdecide), and thefixturerecord and notes insrc/schema.tsandsrc/summarize.ts.Review. The first commit went through an adversarial review (6 reviewers, then 3 skeptics per finding): 54 of 55 findings upheld, none blockers, 16 major. The second commit,
70c8aed, applies them; its message lists each. The largest: chaos now holds requests instead of fetching responses from Node (user mocks, streams, redirects, cookies and CORS now untouched; keys in request order, so seeds reproduce), the fixture no longer logs a console error per page load, detection confidence is the diagnosis probability, and the CI e2e step now runs with pipefail (before, it could not fail).Measured (
packages/playwright/test/e2e/run.mjs, the second job of.github/workflows/playwright-reporter.yml; headless Chromium 153, Playwright 1.63.0, 2026-10-25,model: "skip", six runs of five seeds, so 30 seeds per row), on the MCP e2e search page (packages/mcp/test/e2e/app/app.js, its store on the fixture's runtime through the server's script-tag mapping), typing "react" at 80 ms per key against a backend that answers in 40 ms in order:witness:overwrote-neweronGET /api/searchreorder: false)window.__FIXED__)The latency-only row is the check that the witness comes from the order, not from the latency; the fixed row is the false-report count on the correct variant. With
model: "load"(one run by hand on the first commit, two seeds, not in the workflow because it downloads the model): the model loaded through the fixture's path (genclass-runtime-r172.0.0-rc4t, WASM, in a worker), the witness was recorded in 2 of 2 seeds, and the model gave no likely detection on that page (staleat 0.64 in seed 1, below the 0.90 report gate; no decision in seed 2), which matches the compatibility matrix and is why witnesses are the default.Things to know:
app.jsuntouched but with an import map that resolves@genclass/runtimeto the page'swindow.GenClass(the script-tag build the fixture injected, as an app integrated through the script tag would call it) and stubs@genclass/runtime/agent(the MCP bridge link, not run here). An app that starts its own ESM runtime runs a second runtime next to the fixture's, and stores it registers withrt.atomorrt.guardare not visible to the fixture's runtime; the README's limits section says so.@playwright/testand@genclass/runtime, both optional (the reporter needs neither). Nothing new at run time; nothing underpackages/runtime/src/changed.docs/runtime/README.mdandllms.txt. Neither is onmain; both are added by the open docs PR, Docs: answer-first READMEs, FAQ, docs index, llms.txt, trimmed npm page; factual fixes #69. This PR updates whatmainhas (the rootREADME.mdpackage table andAGENTS.md). Both files in that PR still say the fixture "has not shipped", so whichever of the two merges second should replace those lines with these:llms.txtline 52:- [Playwright fixture and reporter](https://raw.githubusercontent.com/genclass-dev/GenClass-lib/main/packages/playwright/README.md): `@genclass/playwright` 0.1.0, not on npm yet: a fixture that puts the runtime on every page under test (observe mode, seeded network chaos, the model skipped by default) and writes one race file per test, and a reporter that turns those files into one markdown report per pull requestdocs/runtime/README.mdline 43:| [Playwright fixture and reporter](../../packages/playwright/README.md) | How do I get one pull-request report of what GenClass flagged in my Playwright tests? | reference (`@genclass/playwright` 0.1.0, not on npm yet) | |.github/workflows/mcp.ymlpipes its e2e intoteewithout pipefail, the same bug fixed here; it is outside this PR.test/smoke/smoke.sh, the install suite, anything on Azure.npm publishwas not run.🤖 Generated with Claude Code
https://claude.ai/code/session_01Fjo9SusiHz3LvqdFnr5wYP