playwright: the GenClass fixture (seeded chaos, one race file per test)

.github/workflows/playwright-reporter.yml

@@ -1,20 +1,29 @@
 name: Playwright reporter
 
-# @genclass/playwright (packages/playwright): typecheck, build and the unit tests of the reporter, the summariser and
-# the command, then one real Playwright run that loads the reporter by its module name. No browser: the package only
-# reads the files the GenClass Playwright fixture writes, and the fixture lives elsewhere. A separate workflow from
-# ci.yml so it never blocks the main CI job. It renders the unit-test fixture's report into the job summary to show
-# what a pull-request comment would look like; it posts nothing.
+# @genclass/playwright (packages/playwright): typecheck, build and the unit tests of the fixture (on a fake page),
+# the reporter, the summariser and the command, then one real Playwright run (no browser) that loads the reporter by
+# its module name, and in a second job the fixture's end-to-end check in Chromium (packages/playwright/test/e2e):
+# the MCP e2e search page with the fixture's seeded chaos, five seeds per test, the buggy page must get the
+# overwrote-newer witness, the fixed page and the latency-only run must get nothing, and the reporter must render
+# it. A separate workflow from ci.yml so it never blocks the main CI job. It renders the unit-test fixture's report
+# into the job summary to show what a pull-request comment would look like; it posts nothing and sends nothing
+# anywhere (the app server listens on 127.0.0.1 inside the runner; the page's model is skipped).
 
 on:
   push:
     paths:
       - "packages/playwright/**"
+      - "packages/runtime/src/agent/**"
+      - "packages/runtime/src/cdn/**"
+      - "packages/mcp/test/e2e/app/**"
       - "package-lock.json"
       - ".github/workflows/playwright-reporter.yml"
   pull_request:
     paths:
       - "packages/playwright/**"
+      - "packages/runtime/src/agent/**"
+      - "packages/runtime/src/cdn/**"
+      - "packages/mcp/test/e2e/app/**"
       - "package-lock.json"
       - ".github/workflows/playwright-reporter.yml"
   workflow_dispatch:
@@ -75,3 +84,46 @@ jobs:
           node ../../bin/genclass-playwright-report.mjs --out - > "$RUNNER_TEMP/expected.md"
           diff "$RUNNER_TEMP/expected.md" genclass-report.md
           grep -qF '**1 reproducible**' genclass-report.md
+
+  e2e:
+    name: "@genclass/playwright: the fixture end to end in Chromium"
+    runs-on: ubuntu-latest
+    timeout-minutes: 20
+    env:
+      ONNXRUNTIME_NODE_INSTALL: skip
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-node@v4
+        with:
+          node-version: 22
+          cache: npm
+
+      - name: Install
+        run: npm ci --no-audit --no-fund
+
+      - name: Build (the runtime's dist/ is what the fixture serves to the page)
+        run: |
+          npm run build -w @genclass/runtime
+          npm run build -w @genclass/playwright
+
+      - name: Chromium
+        run: npx playwright install --with-deps chromium
+
+      - name: The fixture end to end (buggy page flagged, fixed page and latency-only run clean, report rendered)
+        run: node packages/playwright/test/e2e/run.mjs | tee "$RUNNER_TEMP/e2e.log"
+
+      - name: Summary
+        if: always()
+        run: |
+          {
+            echo '### GenClass fixture, end to end'
+            echo '```json'
+            sed -n '/^{/,$p' "$RUNNER_TEMP/e2e.log" | tail -60
+            echo '```'
+            if [ -f packages/playwright/test/e2e/genclass-report.md ]; then
+              echo
+              echo '### The report the reporter wrote'
+              cat packages/playwright/test/e2e/genclass-report.md
+            fi
+          } >> "$GITHUB_STEP_SUMMARY"

AGENTS.md

@@ -21,7 +21,7 @@ Default mode is `observe` (report only); `guard` is opt-in; `heal` is experiment
 | `training/` | vocabulary pruning, curriculum (`curriculum/rt.py` mirrors the runtime's situation renderer), Azure training, ONNX export, `EVAL.md` |
 | `demos/` | six demo apps with a chaos backend and Playwright trials; `bench/heal/` runs them per mode |
 | `packages/mcp/` | `@genclass/mcp`: the MCP server for coding agents (bridge on 127.0.0.1:47357, store, explore verdicts, replays, regression-test generation, 8 tools); no runtime deps; `test/e2e/race.e2e.mjs` is the real-browser check (not in CI's main job) |
-| `packages/playwright/` | `@genclass/playwright`: Playwright reporter + pure summariser that read the GenClass fixture's `test-results/genclass/*.json` (`genclass-races/1`) and write `genclass-report.md` for the PR; injects nothing, no runtime deps |
+| `packages/playwright/` | `@genclass/playwright`: the fixture (`src/fixture.ts`, `install.ts`, `page.ts`, `chaos.ts`, `races.ts`: puts the runtime's script-tag build on every `page` in observe mode with telemetry off, seeded chaos on fetch/XHR responses with `page.route`, witnesses through the agent entry with every hold off, one `genclass-races/1` file per test in `test-results/genclass/`) and the reporter + pure summariser that read those files and write `genclass-report.md` for the PR; peer deps only (`@playwright/test`, `@genclass/runtime`); `test/e2e/run.mjs` is the real-browser check (its own workflow) |
 | `bench/visual/` | the visual checks' bench (GitHub Actions only, `.github/workflows/visual-measure.yml`): false reports on the demos, templates, compat apps and about 40 public sites, recall on faults planted from the harness, cost; public sites are counted per check kind only, never named in the repo, its run logs or plain artifacts (per-site results only encrypted with the `VISUAL_BENCH_KEY` secret) |
 | `compat/`, `templates/` | framework compatibility matrix (`RESULTS.md`) and starter templates; evaluation only |
 | `telemetry-worker/` | the diagnostics collector with the dashboards and accounts (Cloudflare Worker). genclass.dev itself (the site, its admin and the pitch deck) and the page check (`scan-worker/`) live in the private repo genclass-dev/genclass-site, which has this repo as its `lib/` submodule |
@@ -46,9 +46,10 @@ NODE_OPTIONS=--expose-gc npx vitest run test/review-perf.test.ts --retry=2   # t
 `npm run typecheck -w @genclass/mcp && npm run build -w @genclass/mcp && (cd packages/mcp && npx vitest run)`; the e2e
 needs both packages built: `node packages/mcp/test/e2e/race.e2e.mjs`.
 
-`packages/playwright/` (`@genclass/playwright`, the race-file reporter) has its own workflow,
-`playwright-reporter.yml` (typecheck, build, vitest; no browser; never blocks `ci.yml`):
-`npm run typecheck -w @genclass/playwright && npm run build -w @genclass/playwright && (cd packages/playwright && npx vitest run)`.
+`packages/playwright/` (`@genclass/playwright`, the fixture and the race-file reporter) has its own workflow,
+`playwright-reporter.yml` (typecheck, build, vitest; then the fixture's e2e in Chromium; never blocks `ci.yml`):
+`npm run typecheck -w @genclass/playwright && npm run build -w @genclass/playwright && (cd packages/playwright && npx vitest run)`;
+the e2e needs the runtime built too: `node packages/playwright/test/e2e/run.mjs`.
 
 One test: `npx vitest run test/delivery.test.ts -t "<name>"`. The 14 skipped tests are model-parity tests that need
 `GENCLASS_MODEL_DIR`. CI also runs the `visual-browser` job: the first-load size against the release line (`FIRST_LOAD_BASE`, release/0.3.0 until 0.3.0 is on main; `test/first-load-size.mjs`, fails at 0.6 KB

README.md

@@ -254,7 +254,7 @@ All numbers are on held-out data, with each recall reported next to its false-in
 |---|---|
 | [`packages/runtime`](packages/runtime) | `@genclass/runtime`, the library. Observers, causality, stores and adapters, learned baselines/relations/profiles, facts, triage, policy gate, actions, model host (Web Worker, ONNX Runtime Web on WebGPU/WASM), devtools overlay. [README](packages/runtime/README.md) · [CHANGELOG](packages/runtime/CHANGELOG.md) |
 | [`packages/mcp`](packages/mcp) | `@genclass/mcp` (unreleased), the MCP server for coding agents: pages connect through `@genclass/runtime/agent`; the agent gets findings in causal order, replays, explore verdicts and generated regression tests ([AGENT.md](docs/runtime/AGENT.md)). [README](packages/mcp/README.md) |
-| [`packages/playwright`](packages/playwright) | `@genclass/playwright` (unreleased), a Playwright reporter: reads what the GenClass fixture flagged per test (`test-results/genclass/*.json`) and writes one markdown report for the pull request, with the evidence line and what reproduced across seeds. [README](packages/playwright/README.md) |
+| [`packages/playwright`](packages/playwright) | `@genclass/playwright` (unreleased), GenClass for Playwright: a fixture that puts the runtime on every page under test (observe mode, seeded network chaos with `page.route`, the model skipped by default) and writes one race file per test, and a reporter that turns those files into one markdown report for the pull request, with the evidence line and what reproduced across seeds. [README](packages/playwright/README.md) |
 | [`packages/runtime-model`](packages/runtime-model) | `@genclass/runtime-model`, the default model package (data only; `files/` is gitignored and filled at release time) and its [model card](packages/runtime-model/MODEL_CARD.md) |
 | [`sim`](sim) | Training-data simulator: random apps run on the real runtime in a deterministic virtual world, labelled by counterfactual outcomes |
 | [`realapps`](realapps) | 128 real apps (written for the corpus, plus open-source ones) run with the real runtime in headless Chromium, labelled the same way; the never-worse sweep covered the first 66. [README](realapps/README.md) |
@@ -290,9 +290,9 @@ provenance.
 Not covered by CI:
 
 - Playwright browser tests: `npm run test:browser` in `packages/runtime`.
-- `@genclass/playwright`: its typecheck, build and unit tests run in their own workflow,
-  [.github/workflows/playwright-reporter.yml](.github/workflows/playwright-reporter.yml), on pushes and pull
-  requests that touch the package; it never blocks `ci.yml`.
+- `@genclass/playwright`: its typecheck, build, unit tests and the fixture's end-to-end check in Chromium run in
+  their own workflow, [.github/workflows/playwright-reporter.yml](.github/workflows/playwright-reporter.yml), on
+  pushes and pull requests that touch the package; it never blocks `ci.yml`.
 - The npm tarball smoke test: `test/smoke/smoke.sh`.
 - The sim, the real-app corpus, the demos' evaluation, and training.
 

packages/playwright/.gitignore

@@ -1 +1,2 @@
 test/playwright-smoke/genclass-report.md
+test/e2e/genclass-report.md