diff --git a/.github/ISSUE_TEMPLATE/config.yml b/.github/ISSUE_TEMPLATE/config.yml new file mode 100644 index 00000000..9e7acbb6 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/config.yml @@ -0,0 +1,14 @@ +blank_issues_enabled: true +contact_links: + - name: Questions and answers (FAQ) + url: https://github.com/genclass-dev/GenClass-lib/blob/main/FAQ.md + about: Short answers to common questions (race conditions in React, stale responses, double submits, what it sends, what it costs), with links to the details. + - name: Ask a question in Discussions + url: https://github.com/genclass-dev/GenClass-lib/discussions/categories/q-a + about: Questions about installing, configuring or reading GenClass Runtime's reports. + - name: GenClass Discord + url: https://discord.gg/kN9kRnFtZm + about: Talk with the maintainers and other users. + - name: Report a security vulnerability + url: https://github.com/genclass-dev/GenClass-lib/blob/main/SECURITY.md + about: Please report security issues privately, not in a public issue; SECURITY.md says how. diff --git a/AGENTS.md b/AGENTS.md index fda7dd28..0244204c 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -26,7 +26,7 @@ Default mode is `observe` (report only); `guard` is opt-in; `heal` is experiment | `packages/nuxt/`, `packages/astro/` | `@genclass/nuxt` (a Nuxt module, `@nuxt/kit`, built with `@nuxt/module-builder`) and `@genclass/astro` (an Astro integration, `injectScript("page", ...)`): from options `{ mode, token?, telemetry?, devtools? }` they generate the runtime's own one line (`/auto` or `/auto/guard`) plus a `window.GENCLASS_CONFIG` module imported before it; browser only, nothing on the server; `@genclass/runtime` is a peer dependency. The generated code is pure (`src/source.ts`, `src/index.ts`) and the tests check it exactly, then build one fixture app. Workflow: `integrations.yml` (Node 24) | | `docs/adoption/listings/` | how the framework packages and templates get listed (nuxt.com/modules entry and steps, astro.build/integrations, sveltesociety.dev, awesome-vue and Angular lists): research and draft entries; nothing is submitted from the repo | | `bench/visual/` | the visual checks' bench (GitHub Actions only, `.github/workflows/visual-measure.yml`): false reports on the demos, templates, compat apps and about 40 public sites, recall on faults planted from the harness, cost; public sites are counted per check kind only, never named in the repo, its run logs or plain artifacts (per-site results only encrypted with the `VISUAL_BENCH_KEY` secret) | -| `bench/stall/` | `request-stalled` on the demos (`run.ts`, local, headless Chromium): false reports on clean and chaos trials, recall on planted 60 s hangs; numbers in RESULTS.md section 21 | +| `bench/stall/` | `request-stalled` on the demos (`run.ts`, local, headless Chromium): false reports on clean and chaos trials, recall on planted 60 s hangs; numbers in RESULTS.md section 23 | | `compat/`, `templates/` | framework compatibility matrix (`RESULTS.md`) and starter templates (react-vite, nextjs, vue-vite, sveltekit, astro, nuxt, react-router, angular; `compat/templates-check.mjs` builds and boots them in Chromium against the packed runtime, module and integration); evaluation only | | `telemetry-worker/` | the diagnostics collector with the dashboards and accounts (Cloudflare Worker). genclass.dev itself (the site, its admin and the pitch deck) and the page check (`scan-worker/`) live in the private repo genclass-dev/genclass-site, which has this repo as its `lib/` submodule | | `research/` | earlier Jev research: extension, `jev_local` (Python encoder the training scripts import), benchmarks | diff --git a/CITATION.cff b/CITATION.cff new file mode 100644 index 00000000..ad3e2853 --- /dev/null +++ b/CITATION.cff @@ -0,0 +1,32 @@ +cff-version: 1.2.0 +message: "If you use GenClass Runtime in your work, please cite it with these metadata and name the version of @genclass/runtime you used." +type: software +title: "GenClass Runtime" +abstract: >- + GenClass Runtime (@genclass/runtime) is an open-source (Apache-2.0) browser library that reports race + conditions in web apps while they happen: stale responses that land over newer data, duplicate requests such as + double submits, inconsistent state and failure storms. A small model runs on the user's device, in a Web Worker on + WebGPU or WASM, and each report comes with its evidence. By default it only reports (observe): it never holds, + delays or changes a request, a response or a state write. The opt-in guard mode may act on a few of them, and + heal is experimental. +authors: + - name: "Daybot Solutions Inc." + website: "https://genclass.dev" +repository-code: "https://github.com/genclass-dev/GenClass-lib" +url: "https://genclass.dev" +license: Apache-2.0 +# The newest published release of @genclass/runtime (npm `beta`; `latest` is 0.2.2, dated 2026-10-13 in the +# CHANGELOG and published to npm on 2026-10-15, before this one). +# Update both fields at each release. +version: "0.3.0-beta.0" +date-released: "2026-10-15" +keywords: + - race condition + - stale response + - double submit + - web app + - browser + - runtime monitoring + - on-device model + - WebGPU + - WebAssembly diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index dffb6e1b..6e848e98 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,26 +1,52 @@ -# Contributing +# Contributing to GenClass -Thanks for looking. This repository holds `@genclass/runtime` (the library), its model package, and everything -used to train and evaluate the model. Issues and pull requests are welcome; the rules below keep the measurements -honest. +How to change GenClass-lib, the repository of `@genclass/runtime` (GenClass Runtime, an open-source browser library +that reports race conditions in web apps), its model package, and everything used to train and evaluate the model: +where things are, what CI runs, the rules that keep the measurements honest, and how a release is made. To use +GenClass in an app, read the [package README](packages/runtime/README.md) instead. + +Thanks for looking. Issues and pull requests are welcome. ## Where things are - `packages/runtime/` is the library: `src/` (observers, causality, stores, facts, triage, the policy gate, actions, - the model host, devtools), `test/` (vitest), `bin/` (the `init` / `remove` / `fetch-model` CLI). Read - `docs/runtime/CONTRACT.md` (binding design rules), `docs/runtime/API.md` and `packages/runtime/INTERCEPTION.md` - before changing behaviour. + the model host, devtools, the measured visual checks in `src/visual/`, interaction analytics in `src/analytics/`), + `test/` (vitest; `test/browser/` holds the Playwright suites), `bin/` (the CLI: `init`, `remove`, `login`, + `connect`, `whoami`, `logout`, `fetch-model`, `info`). Read `docs/runtime/CONTRACT.md` (binding design rules), + `docs/runtime/API.md` and `packages/runtime/INTERCEPTION.md` before changing behaviour. - `packages/runtime-model/` is the data-only model package and its model card. +- `packages/playwright/` is `@genclass/playwright` (0.1.0, not on npm yet), a Playwright reporter that turns the race + files the GenClass Playwright fixture writes into one markdown report for a pull request (the fixture is built + separately and has not shipped). `packages/genclass-runtime/` is `genclass-runtime`, the former short name for the + CLI: deprecated in this repository and not published by the release workflow, but not yet marked with + `npm deprecate` on the registry, so `npx genclass-runtime init` still installs its newest npm version, 0.1.0-beta.3, + with an unsupported runtime and no deprecation warning from npm. Marking it is a maintainer step: + `npm deprecate genclass-runtime "Use @genclass/runtime: npx @genclass/runtime init"`. Use + `npx @genclass/runtime@beta init` (or `npx @genclass/runtime init` for 0.2.x). - `sim/`, `realapps/`, `training/` produce and train on the model's data; `demos/`, `compat/`, `bench/heal/` - evaluate the result. `docs/runtime/RESULTS.md` is where every measured number lives. + evaluate the result; `bench/visual/` measures the visual checks' false reports, recall and cost (GitHub Actions + only). `docs/runtime/RESULTS.md` is where every measured number lives. +- `telemetry-worker/` is the diagnostics collector with the dashboards and accounts on genclass.dev (a Cloudflare + Worker). `billing/plans.json` is the one place the plans are defined. The website itself is built in a separate, + private repository (see [Releasing](#releasing-maintainers)). +- `templates/` (starter apps with the runtime installed), `research/` (earlier research; the training scripts + import its `jev_local` encoder), `scripts/` (Azure VM helpers for training, and the Stripe setup script). - `AGENTS.md` is the short guide for coding agents; it applies to people too. ## Build and test -Node ≥ 20. CI (`.github/workflows/ci.yml`) runs exactly this on every pull request: +Node ≥ 20. CI (`.github/workflows/ci.yml`, Node 22) runs three jobs on every pull request: + +- **`runtime`**: typecheck, build and the unit tests of `@genclass/runtime`, exactly as below. +- **`visual-browser`**: the first-load size of `/auto` and the main entry against the merge base with `main` (fails + at 0.6 KB gzip growth), and the measured visual checks in Chromium with Playwright + (`test/browser/visual/playwright.config.ts`). Chromium is installed in CI only. +- **`telemetry-worker`**: the collector's unit tests (`cd telemetry-worker && npx vitest run`). + +The `runtime` job: ```bash -ONNXRUNTIME_NODE_INSTALL=skip npm ci +ONNXRUNTIME_NODE_INSTALL=skip npm ci --no-audit --no-fund npm run typecheck -w @genclass/runtime npm run build -w @genclass/runtime cd packages/runtime @@ -28,9 +54,15 @@ NODE_OPTIONS=--expose-gc npx vitest run --exclude "test/browser/**" --exclude te NODE_OPTIONS=--expose-gc npx vitest run test/review-perf.test.ts --retry=2 # timing budgets; reported, not blocking ``` -One test: `npx vitest run test/delivery.test.ts -t ""`. Playwright tests (`npm run test:browser`), the -tarball smoke test (`test/smoke/smoke.sh`), the sim, the real-app corpus, the demos' evaluation and training are -not run in CI; say in the PR which of them you could not run. +One test: `npx vitest run test/delivery.test.ts -t ""`. `@genclass/playwright` has its own workflow, +`playwright-reporter.yml` (typecheck, build, vitest; it runs when `packages/playwright/` or the lockfile changes and +never blocks `ci.yml`). The visual bench (`visual-measure.yml`) runs on manual dispatch, and on pushes that change +the bench on the `wave2/visual` branch; it loads public sites, so it never runs on pull requests. `release.yml` +publishes from a tag (see [Releasing](#releasing-maintainers)). + +Not run in CI: the model Playwright suite (`npm run test:browser`), the tarball smoke test (`test/smoke/smoke.sh`), +the install suite (`test/install/run-all.sh`), the sim and real-app generators, the demos' evaluation and training. +Say in the PR which of them you could not run. ## The rules @@ -40,7 +72,12 @@ not run in CI; say in the PR which of them you could not run. that adds a hold, a delay or an action on clean traffic needs a measurement, not an argument. 2. **No hardcoded bug patterns.** The runtime computes generic facts and decides what is salient; it never maps a fact pattern to a diagnosis or an action with an if/then. Actions are chosen only by the policy gate from the - model's answers. + model's answers. The one exception is the measured visual checks in `src/visual/`: deterministic measurements + with documented thresholds, reported as kind `"check"` and never as a model finding. They never decide network + actions and never feed situations, facts, the event log, the gate or the model; their fixes are opt-in, + reversible and never run in `observe` (`docs/runtime/VISUAL.md`). One indirect path remains: a check step that + makes the browser lay out a very heavy page can take 50 ms or more, and the `perf` observer records that long task + as a `perf` event the model can read (VISUAL.md section 1). 3. **Model-visible text is frozen.** Anything the model reads (`packages/runtime/src/situation/*`, facts, question, action and diagnosis wording, redaction output, op and event names) is tagged (`situation-v2.3`). Changing it means a new tag, regenerated data, an updated `training/curriculum/rt.py` mirror and retraining. Open an issue @@ -54,7 +91,11 @@ not run in CI; say in the PR which of them you could not run. 6. **`packages/runtime/test/review-*.test.ts` are a contract.** Fix `src/`, never the test. 7. **Dependencies.** `onnxruntime-web` is the only runtime dependency; ask before adding another. Keep the committed root `package-lock.json` in sync. -8. **Releases** are published only by `.github/workflows/release.yml` from a `v*` tag, with npm provenance. Do not +8. **Telemetry stays disclosed** (the console notice, `packages/runtime/TELEMETRY.md`, both READMEs, the CHANGELOG), + sends no page text unless the app opts in, keeps every opt-out (`telemetry: false`, `?genclass=no-telemetry`, + the `localStorage` switch, Global Privacy Control) and never feeds a decision. Widening what is sent needs the + owner's OK. +9. **Releases** are published only by `.github/workflows/release.yml` from a `v*` tag, with npm provenance. Do not publish from a laptop. ## Pull requests @@ -68,21 +109,36 @@ not run in CI; say in the PR which of them you could not run. ## Releasing (maintainers) -In this order, so nothing on `main` points at a version npm does not have yet: - -1. On the release branch (`release/`): `packages/runtime/package.json` has the version, the CHANGELOG - section is final, CI is green, and the first-load limits in `test/bundle.test.ts` match the README and - `docs/runtime/RESULTS.md`. -2. Tag `v` at that commit and push the tag; approve the `npm-publish` environment run of - `.github/workflows/release.yml`. -3. Check that the version is live: `npm view @genclass/runtime@ version` and - `curl -sI https://cdn.jsdelivr.net/npm/@genclass/runtime@` (HTTP 200). -4. Only then merge the release branch into `main`. Both READMEs' quick starts pin the version in a CDN script tag; - merged before the publish finishes, the snippet on GitHub loads a 404. -5. Site: after the merge, regenerate `site/public` with `site/build.sh` (move `RT` in `site/build.py` to this version - when the site should show it; `build.sh` refuses a version the CDN does not serve), so `/docs/telemetry` and - `/privacy` match the released `TELEMETRY.md` and `PRIVACY.md`. Deploying the site is a separate, owner-approved - step. +`.github/workflows/release.yml` publishes `@genclass/runtime` to npm from a tag `v` whose version equals +`packages/runtime/package.json`; its header comment is the authority on the order. In this order: + +1. On the release branch (`release/`): `packages/runtime/package.json` and `packages/runtime/src/version.ts` + (`RUNTIME_VERSION`) have the version (`release.yml` checks both), the CHANGELOG section is final and dated, CI is + green, and the first-load limits in `test/bundle.test.ts` match the README and `docs/runtime/RESULTS.md`. Every place + a reader sees the version says the same: the version notes at the top of `docs/runtime/API.md`, + `docs/runtime/OPTIONS-SPEC.md`, `packages/runtime/TELEMETRY.md` and `packages/runtime/INTERCEPTION.md` (and any other + page that opens with one), the CDN pins and the `@beta` commands in both READMEs, and the Versions lines in both + READMEs (`README.md` and `packages/runtime/README.md`). +2. The previous release is on npm as `latest` (`npm view @genclass/runtime dist-tags`) and + `https://cdn.jsdelivr.net/npm/@genclass/runtime@` answers 200. +3. Merge the release branch into `main` by pull request, with CI green: the READMEs link `TELEMETRY.md`, + `PRIVACY.md` and `RESULTS.md` at `blob/main`. When the READMEs pin the new version, their CDN snippet on GitHub + answers 404 until the publish finishes, so tag as soon as step 4 is done. +4. Deploy `telemetry-worker/` from that commit (`d1 migrations apply genclass-dashboard --remote`, then + `wrangler deploy`; `telemetry-worker/README.md`), together with the site's admin aggregates (in + genclass-dev/genclass-site: `site/src/admin.js` and `site/migrations-admin/`; ask its maintainer), before the + tag: a worker older than 0.3.0 counts possible detections as detections. +5. Tag `v` on that `main` commit and push the tag; approve the `npm-publish` environment run of + `release.yml`. +6. Check the release: the dist-tags (a prerelease lands on the tag named after it and `latest` stays), the "Verify + the registry signature and provenance" step, `npm view @genclass/runtime@ version`, and that jsDelivr + serves `@`, `/dist/cdn/worker.js` and `/dist/sinks/webhook.js` (HTTP 200). Then set `RUNTIME_ON_NPM` to + the version in `telemetry-worker/wrangler.toml` and in `telemetry-worker/wrangler.selfhost.toml` (the config + self-hosters deploy; its Connect panel names the runtime that sends tokens), and deploy the worker again. +7. Site: genclass.dev is built in the private repository genclass-dev/genclass-site, which has this repository as + its `lib/` submodule. After the merge, ask its maintainer to move that submodule, and the runtime version the + site's pages name, to this release, so `/docs/telemetry` and `/privacy` match the released `TELEMETRY.md` and + `PRIVACY.md`. Deploying the site is a separate, owner-approved step. ## Reporting a false flag diff --git a/FAQ.md b/FAQ.md new file mode 100644 index 00000000..d27cda35 --- /dev/null +++ b/FAQ.md @@ -0,0 +1,216 @@ +# GenClass Runtime: questions and answers + +Short answers to the questions developers ask about GenClass Runtime, for `@genclass/runtime` 0.3.0-beta.0 (npm +`beta`) and 0.2.2 (npm `latest`; the dist-tags on 2026-10-22), both with the model +`@genclass/runtime-model@0.2.0`. Each answer comes first and the link after it has the details. Every measured number +is from +[RESULTS.md](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/RESULTS.md) (the compatibility +details from [compat/RESULTS.md](https://github.com/genclass-dev/GenClass-lib/blob/main/compat/RESULTS.md)), which +say how and on what each was measured; defaults and limits come from the code. The headline numbers are under +[current numbers in one place](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/RESULTS.md#current-numbers-in-one-place). + +## What is GenClass Runtime? + +GenClass Runtime (`@genclass/runtime`) is an open-source (Apache-2.0) browser library that reports race conditions +in web apps while they happen: stale responses that land over newer data, duplicate requests such as double submits, +inconsistent state and failure storms. A small model runs on the user's device, in a Web Worker on WebGPU or WASM, +and each report comes with its evidence. By default it only reports (`observe`): it never holds, delays or changes a +request, a response or a state write. The opt-in `guard` mode may act on a few of them +([how often](#can-it-prevent-these-bugs-or-only-report-them)), and `heal` is experimental. + +Details: [README](https://github.com/genclass-dev/GenClass-lib#readme), +[package README](https://github.com/genclass-dev/GenClass-lib/blob/main/packages/runtime/README.md). + +## How do I find race conditions in a React app? + +Fix the ones you know in code: cancel the previous request in the effect's cleanup (an `AbortController` or an +ignore flag), or compare a request id. To find the rest while the app runs, put `import "@genclass/runtime/auto"` +first in your entry file. React component state is found automatically, and in `observe` each detection prints one +console line with its evidence. + +Details: [guide: race conditions in React](https://genclass.dev/guides/race-conditions-react), +[what a detection looks like](https://github.com/genclass-dev/GenClass-lib/blob/main/packages/runtime/README.md#what-it-looks-like). + +## Why does my search box show results for an older query? + +The response for an earlier keystroke arrived after the latest one's and overwrote it. Debouncing makes this rarer, +not impossible: cancel the older request, or ignore a response whose query is no longer current. GenClass can report +this as a stale response when it sees the state the response writes: with `/auto` or the script tag, React and Redux +state and Zustand stores with the `devtools` middleware are found automatically; register other state with `rt.atom` +or `rt.guard`. In `guard`, for state it can write, it can drop the stale write when its gate clears. + +Details: [guide: stale responses out of order](https://genclass.dev/guides/stale-responses-out-of-order), +[state it can protect](https://github.com/genclass-dev/GenClass-lib/blob/main/packages/runtime/README.md#state-it-can-protect). + +## Is AbortController enough to stop stale responses? + +For the requests you wire it to, yes: an aborted request's late response never reaches your state. It does not +cover requests you did not wire, or WebSocket and EventSource messages. GenClass watches fetch, XHR, WebSocket and +EventSource for writes over newer data, but it misses most stale overwrites, so it does not replace the fix. + +Details: [How accurate is it?](#how-accurate-is-it) + +## How do I stop a double click from submitting a form twice? + +Disable the submit button while the request is in flight, and send an idempotency key that the server checks, so a +repeat creates nothing. GenClass reports a repeated request as `duplicate`: at the default profile it detected 75% of +duplicate submits on held-out real apps with the right diagnosis, under a false-detection limit of 2% per clean +decision that held on held-out test. The 0.3.0 changes to observe were not measured end to end (§11). + +Details: [guide: prevent duplicate submissions](https://genclass.dev/guides/prevent-duplicate-submissions), +[RESULTS.md §11](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/RESULTS.md#observe-detection). + +## Does GenClass change what my app does? + +Not in `observe`, the default: it never holds, delays or changes a request, a response or a state write. Its own +work can still shift timing. In the compatibility matrix (an unpublished 0.1.0-beta.4 build with model 0.2.0, +2026-10-10; not re-run on 0.2.2 or 0.3.0) it introduced 0 bugs in 3,510 runs. In 4 runs an app's own bug did not +show with GenClass although it took no action (1 in observe, 3 in guard or heal: its own work shifted the timing of +the app's race), and in 38 more guard and heal runs where the bug showed without GenClass, it still showed but the +page or server ended in a different state. `guard` and `heal` act only when their gate clears. + +Details: [RESULTS.md §16](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/RESULTS.md#16-compatibility-matrix-summary), +[INTERCEPTION.md](https://github.com/genclass-dev/GenClass-lib/blob/main/packages/runtime/INTERCEPTION.md). + +## How accurate is it? + +At `balanced` (the default), on held-out real apps, it detected 75% of duplicate submits, 30% of stale overwrites +(on the `situation-v2.3` slice the current runtime renders; 21% on `situation-v2`) and 6% of broken state with the +right diagnosis. The report gate was fitted on development data to at most 2% false detections per clean decision +(3% on simulated apps) and met that limit on held-out test; the exact rate is not published. In 0.3.0, three observe +fixes (superseded writes, the response-body wait, the idle unload) can add likely reports, false ones included, at a +rate not measured. Guard acted on 7.5% of real-app cases where acting would help, with 0.00% false interventions +there, and on 7.8% of clear simulated cases with 0.13%. + +In 0.3.0-beta.0, observe also records *possible* detections by default, at the `eager` gate (0.70), to +`console.debug`, the devtools and telemetry. Likely and possible together are expected to match the `eager` +profile's offline rates (77% of duplicate submits, 45% of stale overwrites, 40% on `situation-v2`, and 13% of broken +state; false-detection limit 5% on real apps and 6% on simulated, met on held-out test). This was not measured end to +end on 0.3.0; how many false detections the possible tier adds, and what share of possible detections is false, are +not measured. + +Details: [Measured results](https://github.com/genclass-dev/GenClass-lib#measured-results), +[current numbers in one place](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/RESULTS.md#current-numbers-in-one-place) (RESULTS.md §1, §11). + +## Can it prevent these bugs, or only report them? + +Mostly report. `guard` (opt-in) can discard a stale write, defer a response, or coalesce or delay a request, only +when the model's gate clears, which with the shipped model is rare. In the compatibility matrix (an unpublished +0.1.0-beta.4 build with model 0.2.0, 2026-10-10; not re-run on 0.2.2 or 0.3.0), of 250 runs where an app's own bug +showed, guard left 249 buggy and heal 240 (its 10 fixes were all `block` on one data layer); guard and heal took 0 +actions on correct apps in 1,760 runs. In the six demos under network chaos no mode lowered the bug rate (72% without +GenClass, 70% observe, 72% guard, 73% heal; differences under about 5 points are noise; runtime 0.1.0-beta.4, model +0.2.0), with 0 false interventions in 30 clean trials each in guard and heal. + +Details: [Modes](https://github.com/genclass-dev/GenClass-lib/blob/main/packages/runtime/README.md#modes), +[RESULTS.md §16](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/RESULTS.md#16-compatibility-matrix-summary). + +## Which frameworks and data libraries does it work with? + +It wraps the browser's network APIs, DOM events and timers, so it needs no framework. The compatibility matrix ran +React 19, Next.js 16, Vue 3, SvelteKit 3, Angular 22, Solid 1.9 and plain HTML with 15 data layers in every mode, +in Chromium only: 0 bugs introduced in 3,510 runs. It tested an unpublished 0.1.0-beta.4 build (commit `d04eff3`, +which does not contain 0.2.0's audit trail, protect presets, hold-budget ceilings and invariant fixes, committed on +another branch, and predates 0.2.1's first-request hold and telemetry defaults) with model 0.2.0 on 2026-10-10, and +has not been re-run on 0.2.2 or 0.3.0. With `/auto` or the script tag, React and Redux state is found +automatically, and so are Zustand stores with the `devtools` middleware (in production builds only with +`devtools(..., { enabled: true })`: found in 240 of 240 runs with it, 0 of 240 without). With `GenClass.init()`, +import `@genclass/runtime/discover` first and pass `autoState: true`. Vue / Pinia, Svelte stores, signals, MobX, +Jotai and Recoil are not discovered: register such state with `rt.atom` / `rt.guard`. + +Details: [compat/RESULTS.md](https://github.com/genclass-dev/GenClass-lib/blob/main/compat/RESULTS.md), +[automatic state discovery](https://github.com/genclass-dev/GenClass-lib/blob/main/packages/runtime/README.md#automatic-state-discovery). + +## How much does it add to my page? + +About 108 KB of JavaScript (minified, gzip) on every page for `@genclass/runtime/auto` in 0.3.0-beta.0 (99 KB in +0.2.2), 100 KB for the main entry (92 KB) and 129 KB for the script tag (113 KB). The model loads at idle: about +13 MB downloaded once on WASM, then cached, and +190 to 280 MB of renderer memory while loaded (one pilot app, +runtime 0.1.0-beta.3). From 0.3.0, devices reporting 2 GB of memory or less, and Save-Data connections, skip it. + +Details: [RESULTS.md §10](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/RESULTS.md#first-load) +and [§17](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/RESULTS.md#17-browser-cost-on-a-pilot-app-2026-10-09-and-2026-10-10), +[Costs](https://github.com/genclass-dev/GenClass-lib/blob/main/packages/runtime/README.md#costs). + +## Does it work without the model, or on phones? + +Without a model it keeps observing (console, `rt.status`, devtools), but the model's detections and actions stop; +the measured visual checks and the opt-in flow expectations, which use no model, still run where they are on +(0.3.0). From 0.3.0 the model is not loaded on devices reporting 2 GB of memory or less, nor on Save-Data +connections, and one console line says why. `model.loadIf` changes that rule; `model: false` loads no model at all. +If no Web Worker can start, the model runs on the main thread unless `model: { inlineFallback: false }` (0.3.0; 0.2.x +ignores it and always falls back to the main thread); each model pass then blocks it (about 180 to 600 ms at 500 to +1,170 tokens on one WASM thread, estimated from runs in Node, not in a browser). + +Details: [Costs](https://github.com/genclass-dev/GenClass-lib/blob/main/packages/runtime/README.md#costs), +[Content-Security-Policy and self-hosting](https://github.com/genclass-dev/GenClass-lib/blob/main/packages/runtime/README.md#content-security-policy-and-self-hosting). + +## Does it send my data anywhere? + +Decisions are made in the browser; nothing is sent to make one. Anonymous diagnostics are on by default on public +hosts, sent to the GenClass maintainers to improve the model, and off on local hosts from 0.2.2. They carry +GenClass's decisions and counts (in 0.3.0 also its possible detections, as `tier: "possible"`), a random per-page +session id, versions, mode, device class, and the app's hostname and id-normalised path; no page text unless the app +opts in. The collector adds a coarse country and stores no IP address or user agent. With an app token (0.3.0) they +also carry the token and the names of `protect()`ed functions. Opt out with `GenClass.init({ telemetry: false })` +(script tag: `data-telemetry="off"`; meta tag: `telemetry=off`; or `window.GENCLASS_CONFIG = { telemetry: false }`), +`npx @genclass/runtime@beta init --no-telemetry`, `?genclass=no-telemetry` or +`localStorage.setItem("genclass.telemetry", "off")`; Global Privacy Control is honoured. + +Details: [TELEMETRY.md](https://github.com/genclass-dev/GenClass-lib/blob/main/packages/runtime/TELEMETRY.md), +[PRIVACY.md](https://github.com/genclass-dev/GenClass-lib/blob/main/PRIVACY.md). + +## What does it cost? Does it need an API key? + +The runtime and model are free, open source (Apache-2.0) and need no API key or server. The plans of the optional +genclass.dev service are defined in `billing/plans.json`, which the collector and the pricing page read, and it +defines two. Free includes the dashboard, data export, forwarding and your own classifier, which GenClass trains on +request. Enterprise, agreed by email, adds GenClass's models in its cloud, a year of history and contract terms. +Nothing is sold on the site. The pricing page shows the plans in force. + +Details: [genclass.dev/pricing](https://genclass.dev/pricing), +[billing/plans.json](https://github.com/genclass-dev/GenClass-lib/blob/main/billing/plans.json). + +## How is it different from Sentry, Datadog or my tests? + +Error trackers and APM collect exceptions, traces and timings; a stale response or a double submit usually throws +nothing they could collect. Tests usually run on fast, ordered networks, where these bugs rarely show. +GenClass looks for them inside the page while people use it, and its reports can go to your own logging. +Use it next to both. + +Details: [How GenClass compares](https://github.com/genclass-dev/GenClass-lib#how-genclass-compares). + +## Can it explain flaky Playwright tests? + +Not as a packaged tool yet. `@genclass/playwright`, a reporter that turns what GenClass flagged in each test into +one markdown report for the pull request, is in this repository but not on npm, and the fixture it reads has not +shipped. Today you can load the runtime in the app under test and read its `[GenClass]` console lines. + +Details: [packages/playwright](https://github.com/genclass-dev/GenClass-lib/blob/main/packages/playwright/README.md). + +## How do I turn it off or remove it? + +`?genclass=off` in the URL installs nothing on that page load; `localStorage.genclass = "off"` does the same until +you remove it. `enabled: false` installs nothing. `rt.disable()` releases anything held and restores what GenClass +wrapped. `npx @genclass/runtime@beta remove` takes out the lines `init` added to your code and uninstalls the package +when nothing else imports it (if you edited a marked line, it changes nothing and lists where). It keeps +`.genclass.local` and its `.gitignore` line, and a dashboard `init` created stays on genclass.dev. +`?genclass=no-telemetry` turns off only the diagnostics. + +Details: [INTERCEPTION.md, turning it off](https://github.com/genclass-dev/GenClass-lib/blob/main/packages/runtime/INTERCEPTION.md#turning-it-off-and-restoring-the-page). + +## How was it measured, and does that apply to my app? + +On held-out data never used in training: simulated apps, real apps run in headless Chromium, a compatibility matrix +of seven apps (six frameworks and plain HTML) and six demo apps under network chaos. Each recall is reported next +to its false rate. It was not evaluated on your app, so watch `observe` before turning on `guard`. + +Details: [RESULTS.md](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/RESULTS.md), +[model card](https://github.com/genclass-dev/GenClass-lib/blob/main/packages/runtime-model/MODEL_CARD.md). + +## Where can I ask something else? + +In [GitHub Discussions](https://github.com/genclass-dev/GenClass-lib/discussions/categories/q-a) or on +[Discord](https://discord.gg/kN9kRnFtZm). Report bugs and false detections in +[Issues](https://github.com/genclass-dev/GenClass-lib/issues); the false-detection template asks for the console +group and `rt.explain(id)`. diff --git a/PRIVACY.md b/PRIVACY.md index 449b239c..8f0eda91 100644 --- a/PRIVACY.md +++ b/PRIVACY.md @@ -1,6 +1,6 @@ # GenClass Runtime: privacy policy -_Last updated: 2026-10-25._ +_Last updated: 2026-10-27._ This policy covers the diagnostics that `@genclass/runtime` (from `0.1.0-beta.3`) sends to its maintainers. It does not cover the website or app that uses GenClass; that site has its own privacy policy. @@ -79,8 +79,10 @@ came from, devices, countries, new or returning, entry and exit pages and funnel (time, route or function name, diagnosis, action). Dashboards do not show the situation text. Data from browsers that opted out or send Global Privacy Control never reaches a dashboard. -When a developer creates a token with `npx @genclass/runtime@beta init`, after approving its changes, the tool sends us -the name in their project's `package.json`. We keep one record per token: the token, that name, when it was created, +When a developer creates a token with `npx @genclass/runtime@beta init` (runtime 0.3.0-beta.0 and later; +`npx @genclass/runtime init` once 0.3.0 is `latest`), after approving its changes, the tool sends us the name in +their project's `package.json`. When they create one at https://genclass.dev/start, we receive the app name they +type, if any. We keep one record per token: the token, that name, when it was created, when data first and last arrived, a one-way hash of the dashboard link's secret, and the account it belongs to, if any. These records are kept until the owner asks us to delete them (contact below, with the token); the dashboard data itself is deleted after 90 days. @@ -88,8 +90,8 @@ asks us to delete them (contact below, with the token); the dashboard data itsel ## Accounts on genclass.dev An account is optional: GenClass, its diagnostics and the private dashboard links work without one. A developer who -creates one (at genclass.dev/signup, or with `npx @genclass/runtime@beta login`, runtime 0.3.0-beta.0 and later) can keep -their apps in one place. An account stores: +creates one (at genclass.dev/signup, or with `npx @genclass/runtime@beta login`, runtime 0.3.0-beta.0 and later; +`npx @genclass/runtime login` once 0.3.0 is `latest`) can keep their apps in one place. An account stores: - **An email address:** as Google returned it when you sign in with Google, or as you typed it (trimmed and lowercased) when you sign up with an email and a password or add a password to your account, together with whether @@ -137,11 +139,12 @@ and your plan. Never card numbers, bank details or billing addresses. For a purc record of the purchase (account id, email, subscription id, time) so that a person from GenClass can write to the buyer to start the work. -**Forwarding** is off unless an account turns it on for one of its apps. Then, for each new detection -from that app, GenClass sends a short record to the services that account chose (Sentry, PostHog, Datadog, Slack or -its own webhook URL): the kind of problem, the model's confidence, what triggered the check, the page path, the -runtime and model versions, the time and a link to the app's dashboard. Never text from the page. Those services -handle the records under their own terms with the account owner. The keys and URLs for them are stored encrypted. +**Forwarding** is off unless an account turns it on for one of its apps. Then, for each new likely detection +from that app (never a possible one), GenClass sends a short record to the services that account chose (Sentry, +PostHog, Datadog, Slack or its own webhook URL): the kind of problem, the model's confidence, what triggered the +check, the page path, the runtime and model versions, the time and a link to the app's dashboard. Never text from +the page. Those services handle the records under their own terms with the account owner. The keys and URLs for them +are stored encrypted. **Export** lets an account download, or read with an export key, the events GenClass received for its own apps, as stored. Export keys are stored as one-way hashes. @@ -177,7 +180,7 @@ can also contact your local data protection authority. The data comes from your users' browsers. Depending on where you operate, you may need to mention GenClass in your own privacy policy, or keep `telemetry: false` until your users consent to analytics. Full technical details are in -[TELEMETRY.md](packages/runtime/TELEMETRY.md). +[TELEMETRY.md](https://github.com/genclass-dev/GenClass-lib/blob/main/packages/runtime/TELEMETRY.md). ## Changes diff --git a/README.md b/README.md index 1dc0bb57..d935c3c8 100644 --- a/README.md +++ b/README.md @@ -1,91 +1,152 @@ # GenClass Runtime -[![npm](https://img.shields.io/npm/v/@genclass/runtime/latest?label=%40genclass%2Fruntime)](https://www.npmjs.com/package/@genclass/runtime) -[![CI](https://github.com/genclass-dev/GenClass-lib/actions/workflows/ci.yml/badge.svg)](.github/workflows/ci.yml) -[![license](https://img.shields.io/badge/license-Apache--2.0-blue)](LICENSE) -[![website](https://img.shields.io/badge/web-genclass.dev-14B86A)](https://genclass.dev) +[![npm](https://img.shields.io/npm/v/@genclass/runtime/latest?label=npm)](https://www.npmjs.com/package/@genclass/runtime) +[![npm beta](https://img.shields.io/npm/v/@genclass/runtime/beta?label=beta)](https://www.npmjs.com/package/@genclass/runtime?activeTab=versions) +[![CI](https://github.com/genclass-dev/GenClass-lib/actions/workflows/ci.yml/badge.svg)](https://github.com/genclass-dev/GenClass-lib/actions/workflows/ci.yml) +[![license](https://img.shields.io/badge/license-Apache--2.0-blue)](https://github.com/genclass-dev/GenClass-lib/blob/main/LICENSE) +[![model](https://img.shields.io/badge/model-runs%20in%20the%20browser-brightgreen)](https://github.com/genclass-dev/GenClass-lib/blob/main/packages/runtime-model/MODEL_CARD.md) + +**Website and docs: [genclass.dev](https://genclass.dev)** · [Docs](https://genclass.dev/docs) · [Guides](https://genclass.dev/guides) · [Benchmarks](https://genclass.dev/benchmarks) · [Pricing](https://genclass.dev/pricing) · [FAQ](FAQ.md) · [Discord](https://discord.gg/kN9kRnFtZm) + +**GenClass Runtime (`@genclass/runtime`) is an open-source (Apache-2.0) browser library that reports race conditions +in web apps while they happen: stale responses that land over newer data, duplicate requests such as double submits, +inconsistent state and failure storms.** A small model runs on the user's device, in a Web Worker on WebGPU or WASM, +and each report comes with its evidence. By default it only reports (`observe`): it never holds, delays or changes a +request, a response or a state write. The opt-in `guard` mode may act on a few of them +([how often](#what-it-finds-measured)), and `heal` is experimental. It runs in React, Next.js, Vue, SvelteKit, +Angular, Solid and plain-HTML apps (measured in Chromium with an unpublished 0.1.0-beta.4 build and model 0.2.0 on +2026-10-10, not re-run on 0.2.2 or 0.3.0: +[compatibility matrix](https://github.com/genclass-dev/GenClass-lib/blob/main/compat/RESULTS.md)). + +**Status: `@genclass/runtime@0.3.0-beta.4`, the newest line (npm `beta` dist-tag: `npm install @genclass/runtime@beta`; +the stable line is `0.2.x`), with model `@genclass/runtime-model@0.2.0`.** npm dist-tags on 2026-10-27: `latest` is +`0.2.2` and `beta` is `0.3.0-beta.4` (`0.3.0-beta.1` to `0.3.0-beta.4` were released between 2026-10-22 and +2026-10-27). This README describes the `0.3.0` line; `0.2.x`, npm's `latest`, is the exception for an app that cannot +take a prerelease. Both lines load the same model. AI coding agents working on this repository: start at +[AGENTS.md](AGENTS.md). + + + +## Quick start -**Website and docs: [genclass.dev](https://genclass.dev)** · [Discord](https://discord.gg/kN9kRnFtZm) · [Docs](https://genclass.dev/docs) · [Benchmarks](https://genclass.dev/benchmarks) · [Pricing](https://genclass.dev/pricing) - -**A runtime for web apps. It watches the app from the inside and uses a small local model to flag, and optionally -prevent, stale responses, races, duplicate requests, inconsistent state and failure storms.** +```bash +npx @genclass/runtime@beta init # finds your framework and shows the diff before it writes anything +``` -One line, first in `` or first in your entry file: +Once you approve, `init` installs the package, adds one import first in your entry file (plus a line that loads the +devtools overlay in development only) and creates your app's dashboard token (`--no-token` skips it; every flag: +[INSTALL.md](docs/runtime/INSTALL.md)). -```html - -``` +Or by hand: `npm install @genclass/runtime@beta`, then first in your entry file (any bundler; Next.js 15.3 or later: +`instrumentation-client.ts`, earlier versions: a client component in the root layout or `pages/_app`): ```ts -import "@genclass/runtime/auto"; // after npm install @genclass/runtime@beta +import "@genclass/runtime/auto"; ``` -Use the newest line, `0.3.0-beta.1` (npm dist-tag `beta`): every npm and npx command here names `@beta`, and the -script tag pins the version (the unversioned URL serves npm's `latest`, possibly an older release from the CDN cache). -A bare `npm install @genclass/runtime` or `npx @genclass/runtime init` gets npm's `latest`, the older `0.2.x` line, -which has no token, dashboard, interaction analytics, visual checks or agent link, whose CLI has no `login`, -`connect`, `dashboard` or `--sink`, and whose runtime ignores `token` and `analytics: true` without a word. Keep -`0.2.x` only for an app that cannot take a prerelease. - -That covers the whole app: network, user actions, errors, timing, and the app's state (React component state, -Redux / Redux Toolkit and Zustand `devtools` stores are discovered automatically; discovered React and Zustand state -is observed only). Observe mode by default: it reports only. `npx @genclass/runtime@beta init` writes the line for you; -`GenClass.init()` is the explicit setup ([package README](packages/runtime/README.md#automatic-state-discovery)). - -GenClass Runtime records what the app does: user actions, async operations and their causal chains, network -traffic (fetch, XHR, WebSocket, EventSource), store writes with per-field versions, errors and timing. It computes -generic facts about each write, request and response, and asks a small GenClass model only about salient situations. -The model runs in the browser (WebGPU or WASM, in a Web Worker). It answers two questions: what is happening, and -which available action is best. There is no list of known bugs in the code. - -**Status: `@genclass/runtime@0.3.0-beta.1`, the newest line (npm `beta` dist-tag: `npm install @genclass/runtime@beta`; -npm's `latest` is still the older `0.2.x`, the exception for apps that cannot take a prerelease) with model -`@genclass/runtime-model@0.2.0`.** Observe by default; `guard` is -opt-in; `heal` is experimental. Measured false-intervention and recall rates are under -[Measured behaviour](#measured-behaviour). AI coding agents: start at [AGENTS.md](AGENTS.md). +Or with no build step, first in ``: -## Install - -Pick one; all of them start the same runtime in observe mode unless you choose otherwise. Full install guide, with -`init` flags, page configuration and the script-tag attributes: [package README](packages/runtime/README.md#install). - -```bash -npx @genclass/runtime@beta init # finds the framework, adds the import, shows the diff first -npm install @genclass/runtime@beta # then: import "@genclass/runtime/auto" (or GenClass.init()) +```html + ``` +- **That covers network, user actions, errors and timing in the whole app, and its state where it can be found.** + React component state, Redux / Redux Toolkit stores and Zustand stores with `devtools` (in production builds only + with `devtools(..., { enabled: true })`) are discovered automatically; discovered React and Zustand state is + observed only. Vue / Pinia, Svelte stores, signals, MobX, Jotai and Recoil are not discovered: register such state + with `rt.atom` / `rt.guard` + ([state it can protect](https://github.com/genclass-dev/GenClass-lib/blob/main/packages/runtime/README.md#state-it-can-protect)). + `GenClass.init()` is the explicit setup; it discovers state only if you import `@genclass/runtime/discover` first + and pass `autoState: true`. +- **Pin the version in the script tag**, as above: the unversioned URL serves `latest`, and the CDN cache may serve an + older release. +- **Keep `@beta` while `0.3.0` is in beta.** A bare `npm install @genclass/runtime` or `npx @genclass/runtime init` + gets npm's `latest`, the older `0.2.x` line, which has no token, dashboard, interaction analytics, visual checks or + agent link, whose CLI has no `login`, `connect`, `whoami`, `logout`, `dashboard` or `--sink`, and whose runtime + ignores `token` and `analytics: true` without a word. Keep `0.2.x` only for an app that cannot take a prerelease. +- **ESM only** (no `require`). Jest users need a `transformIgnorePatterns` entry that lets `@genclass/runtime` be + transformed. + +Full install guide, with every `init` and `remove` flag, the app token and the tested project setups: +[docs/runtime/INSTALL.md](docs/runtime/INSTALL.md). The short version, with page configuration and the script-tag +attributes, is in the +[package README](https://github.com/genclass-dev/GenClass-lib/blob/main/packages/runtime/README.md#install). + To audit any page for visual bugs without adding anything to it: `npx @genclass/runtime@beta scan ` (alias -`audit`; a score and each problem with its element and a suggested CSS fix; [Audit a page](packages/runtime/README.md#audit-a-page)). +`audit`; a score and each problem with its element and a suggested CSS fix; +[Audit a page](https://github.com/genclass-dev/GenClass-lib/blob/main/packages/runtime/README.md#audit-a-page)). Coding agents get the same as `genclass_scan` in [`@genclass/mcp`](packages/mcp). -The package is **ESM only** (no `require`). Jest users need `transformIgnorePatterns` that lets -`@genclass/runtime` be transformed. +## What it finds (measured) + +Model `@genclass/runtime-model@0.2.0` at the default profile, `balanced`, on held-out data never used in training +([RESULTS.md](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/RESULTS.md) §1, §11; every headline number is under +[current numbers in one place](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/RESULTS.md#current-numbers-in-one-place)): -### What you will see in the first two minutes +| problem | reported with the right diagnosis (observe, held-out real apps; false detections limited to 2% per clean decision) | +|---|---| +| duplicate submits | 75% | +| stale responses written over newer data | 30% on the `situation-v2.3` slice the current runtime renders (21% on `situation-v2`) | +| broken state (a relation between fields stops holding) | 6% | + +- **False reports.** The report gate was fitted on development data to at most 2% false detections per clean + decision on real apps (3% on simulated apps) and met that limit on held-out test; the exact rate is not published. + In `0.3.0`, three observe fixes (superseded writes, the response-body wait, the idle unload) can add likely reports, + false ones included, at a rate not measured (§11). In the compatibility matrix (an unpublished 0.1.0-beta.4 build, + commit `d04eff3`, 2026-10-10), observe made 49 false findings in 880 runs of correct apps, all in one polling and + fan-out scenario + ([RESULTS.md §16](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/RESULTS.md#16-compatibility-matrix-summary)). +- **Possible detections (0.3.0).** Observe in `0.3.0-beta.0` also records detections below the report gate, at the + `eager` profile's gate (0.70), to `console.debug`, the devtools and telemetry. Likely and possible together are + expected to match the `eager` profile's offline rates (77% of duplicate submits, 45% of stale overwrites, 40% on + `situation-v2`, and 13% of broken state; false-detection limit 5% on real apps and 6% on simulated, met on held-out + test). This was not measured end to end on `0.3.0`; how many false detections the possible tier adds, and what + share of possible detections is false, are not measured (§11). +- **Effect on correct apps.** In the same matrix GenClass introduced 0 bugs in 3,510 runs. Observe is still not + byte-identical to running without GenClass: in 1 of its 117 observe cells (Next.js + SWR, scenario d, on seed 7) + its own work changed which side of the app's own race won, towards the correct result, without any action; the + same happened in 3 guard and heal runs. +- **Acting (`guard`, opt-in).** It acts on 7.8% of clear simulated cases and 7.5% of real-app cases where acting + would help. It fired on 0.13% of the simulated cases where doing nothing was best, and on 0.00% on held-out real + apps. It reports far more than it fixes. +- **Visual and CSS checks** (new in 0.3.0; measured checks with fixed thresholds, not the model): 35 of 35 planted + faults found on fixture pages written by the checks' author. On the bench, recall 1.00 for 20 of 23 planted fault + kinds (19 of 23 after the review; `click-blocked` and `scroll-lock-stuck` 0.67) with 0 false reports in 128 page + views of unmodified demo, template and compatibility apps. On public sites the false-report rate after the review + is not known, and three of the proposed cost limits are not met (§13, §14). + +The full tables, with every false-intervention rate next to its recall, are under [Measured results](#measured-results). + +## What you will see in the first two minutes - One console line saying anonymous diagnostics are on (`[GenClass] Sends anonymous diagnostics (decisions and - counts, no page text) to improve the model. Opt out: …`); since `0.2.2` it does not appear on `localhost` - or other local hosts (`0.2.1` sends and shows it there too, by mistake). + counts, no page text) to improve the model. Opt out: …`). From `0.2.2` it does not appear on `localhost` or other + local hosts; earlier versions (`0.1.0-beta.3` to `0.2.1`) send from them too. - `[GenClass] Model ready (genclass-runtime-r17, wasm, q8, 0.4s). Mode: observe.` once the model has loaded at idle. - Then possibly nothing for a while. Observe mode reports only salient situations (a response landing over newer data, a repeat, a failure streak, a broken relation, an error), and the first response of each kind after a page load is never a delivery decision: the runtime first has to see what that request writes. - The devtools overlay: `Alt+Shift+G` on localhost (the `init` command loads it in development only). -- The first-visit cost: about 13 MB downloaded from jsDelivr on WASM (model 10 MB plus ONNX Runtime's wasm) or - 18 MB on WebGPU, cached afterwards, and 190–280 MB of memory in a Web Worker while the model is loaded. Devices - that report 2 GB of memory or less, and Save-Data connections, do not load the model by default (they observe - without it and say so in one console line; new in `0.3.0`: `0.2.x` loads everywhere and only lazily on - Save-Data). For phones and strict Content-Security-Policies see - [self-hosting](packages/runtime/README.md#content-security-policy-and-self-hosting) and - [`loadIf`](packages/runtime/README.md#costs) in the package README. - -### New: your coding agent can see the race, replay it, and prove the fix - -`@genclass/mcp` (unreleased) is an MCP server for Claude Code and other coding agents. Your app's development build -connects to it with one line, and the agent gets every click, request, the order the responses arrived and what they -wrote, uncaught errors with the click and request behind them, and the on-device model's detections. It can run your -steps again with the network answering in different orders (and slower) to find what depends on timing, replay the -exact order that broke, and write a Playwright test that fails until the fix is in. Everything stays on your machine. +- The first-visit cost, measured on one pilot app with runtime 0.1.0-beta.3 and model 0.2.0 + ([RESULTS.md](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/RESULTS.md) §17): about 13 MB + downloaded from jsDelivr on WASM (12.7 MB measured, with the q8 model at 9.45 MB as recorded, ONNX Runtime's wasm + at 3.07 MB brotli and the tokenizer; the q8 file itself is 10.2 MB); on WebGPU with `shader-f16`, the 13.6 MB fp16 + model plus a larger ONNX Runtime build. Both are cached afterwards. The renderer process uses 190 to 280 MB more + memory while the model is loaded. Devices that report 2 GB of memory or less, and Save-Data connections, do not + load the model by default (they observe without it and say so in one console line; new in `0.3.0`: `0.2.x` loads + everywhere and only lazily on Save-Data). For phones and strict Content-Security-Policies see + [self-hosting](https://github.com/genclass-dev/GenClass-lib/blob/main/packages/runtime/README.md#content-security-policy-and-self-hosting) + and [`loadIf`](https://github.com/genclass-dev/GenClass-lib/blob/main/packages/runtime/README.md#costs). + +## New: your coding agent can see the race, replay it, and prove the fix + +`@genclass/mcp` (`0.2.0`, on npm since 2026-10-25) is an MCP server for Claude Code and other coding agents. Your +app's development build (`@genclass/runtime` `0.3.0-beta.1` or later) connects to it with one line, and the agent +gets every click, request, the order the responses arrived and what they wrote, uncaught errors with the click and +request behind them, and the on-device model's detections. It can run your steps again with the network answering in +different orders (and slower) to find what depends on timing, replay the exact order that broke, and write a +Playwright test that fails until the fix is in. Its `genclass_scan` tool audits any page for visual bugs, like +`scan ` above. Everything stays on your machine. ```bash claude mcp add genclass -- npx -y @genclass/mcp@0.2.1 @@ -101,12 +162,13 @@ if (import.meta.env.DEV) connectAgent(rt); // development builds only On a real page in headless Chromium, it found a stale-results race in 5 of 6 reordered runs of a page that never showed it on the developer's fast backend, with no false report on the fixed page ([AGENT.md](docs/runtime/AGENT.md), -[RESULTS.md](docs/runtime/RESULTS.md) section 16). +[RESULTS.md](docs/runtime/RESULTS.md) section 18). -### New in 0.3.0: app dashboards and `protect()` +## New in 0.3.0: app dashboards and `protect()` -In `0.3.0-beta.0` (`@beta` on npm): `0.2.x` ignores `token` and has no `protect()` export (an import of it fails), -and its `init` creates no token. Details: [CHANGELOG](packages/runtime/CHANGELOG.md), 0.3.0-beta.0. +Since `0.3.0-beta.0` (the `beta` line on npm). `0.2.x` ignores `token`, has no `protect()` export (an import of it +fails), and its `init` creates no token. Details: +[CHANGELOG](https://github.com/genclass-dev/GenClass-lib/blob/main/packages/runtime/CHANGELOG.md), 0.3.0-beta.0. ```ts import { GenClass, protect } from "@genclass/runtime"; // 0.3.0 @@ -117,114 +179,174 @@ export const submitOrder = protect("checkout submit", (cart) => fetch("/api/orde ``` - **Token and dashboard.** A token (`gc_` plus 22 letters and digits, public) groups your app's diagnostics under a - private dashboard link (`https://genclass.dev/dashboard/...`). The dashboard is free for every app, with or - without an account. - Get one at [genclass.dev/start](https://genclass.dev/start), or let `init` create it after you approve its diff. Or - keep the dashboards in an optional genclass.dev account: `npx @genclass/runtime@beta login` signs in from the - terminal (device flow) and connects the app in the current folder; `connect`, `whoami` and `logout` do the rest. The - dashboard fills from deployed pages with telemetry on. Local and private hosts (`localhost`, `*.local`, `*.test`, - private networks) do not send by default; `telemetry: true` or any telemetry options object (`telemetry: { ... }`) - is an explicit opt-in that sends from them too, with the token. Data is kept for 90 days. + private dashboard link (`https://genclass.dev/dashboard/...`). The dashboard is free for every app, with or without + an account. Get a token at [genclass.dev/start](https://genclass.dev/start), or let `init` create it after you + approve its diff. To keep dashboards in an optional genclass.dev account, `npx @genclass/runtime@beta login` signs in + from the terminal (device flow) and connects the app in the current folder; `connect`, `whoami` and `logout` do the + rest, and `npx @genclass/runtime@beta dashboard` prints an app's link again (from `0.3.0-beta.2`). The dashboard + fills from deployed pages with telemetry on. Local and private hosts (`localhost`, `*.local`, `*.test`, private + networks) do not send by default; `telemetry: true` or any telemetry options object is an + explicit opt-in that sends from them too, with the token. Data is kept for 90 days (a year on Enterprise). Details: + [docs/runtime/DASHBOARD.md](docs/runtime/DASHBOARD.md). - **`protect(name, fn)`** returns a function with the same signature (same `this`, arguments, return value and errors); decisions about what it causes carry `fn: name`. Without a runtime it just calls `fn`. - [Package README](packages/runtime/README.md#function-specific-protect). -- **More sensitive observe mode.** Observe also reports *possible* detections (below the report gate, at the - `eager` profile's value; one `console.debug` line each, ending in `(possible)`; your `rt.on("detect")` and report - function get them only with `detect: "possible"`; `detect: "likely"` turns them off), and decides what happens - while the model loads or the tab is hidden once it can, instead of skipping it, reporting those at the possible - tier at most until their false-report rate is measured. Their telemetry `detect` events carry `tier: "possible"`; - nothing new about the page is sent. - [Package README](packages/runtime/README.md#modes). + [Package README](https://github.com/genclass-dev/GenClass-lib/blob/main/packages/runtime/README.md#function-specific-protect). +- **More sensitive observe mode.** Observe also reports *possible* detections, below the report gate at the `eager` + profile's value: one `console.debug` line each, ending in `(possible)`. Your `rt.on("detect")` and report function + get them only with `detect: "possible"`; `detect: "likely"` turns them off. Situations raised while the model loads + or the tab is hidden are decided once it can, and reported at the possible tier at most until their false-report + rate is measured. + [Package README](https://github.com/genclass-dev/GenClass-lib/blob/main/packages/runtime/README.md#modes). ## How it works +GenClass Runtime records what the app does: user actions, async operations and their causal chains, network traffic +(fetch, XHR, WebSocket, EventSource), store writes with per-field versions, errors and timing. It computes generic +facts about each write, request and response, and asks the model only about salient situations and the questions you +ask it yourself (step 3). The model answers two questions: what is happening, and which available action is best. +The model path has no list of known bugs in the code; the separate visual and CSS checks are measured checks with +documented thresholds and do not feed the model. One indirect path remains: a check that makes the browser lay out a +very heavy page can cause a long task, which the runtime records as a `perf` event the model can read +([VISUAL.md](docs/runtime/VISUAL.md), section 1). + 1. **Observe.** Fetch, XHR, WebSocket, EventSource, DOM events, errors, navigation, storage, long tasks and timers are instrumented. Async work is linked into causal chains. Stores created through GenClass, or wrapped with its Redux, Zustand, React or generic adapters, get per-field versions and write histories. 2. **Learn.** The runtime learns per-operation baselines (latency, failure rate, request rate) and which fields each operation usually writes. It also learns cadences (polling, debounced saves) and relations that keep holding (`total == sum(items[*].price × qty)`, `selectedId ∈ items[*].id`). -3. **Triage.** Cheap facts are computed for every write and request. Only salient situations reach the model: a - response landing over newer data, a repeated request, a failure streak, a broken relation, an error. -4. **Decide at the network boundary.** Responses and push messages are judged *before the app sees them*: +3. **Triage.** Cheap facts are computed for every write and request. By default only salient situations reach the + model: a response landing over newer data, a repeated request, a failure streak, a broken relation, an error. + Your own questions (`rt.ask()`, `rt.decide()`) and plugin questions marked `always` are asked regardless. +4. **Decide at the network boundary.** In `guard` and `heal`, responses and push messages are judged *before the app + sees them*; in `observe` they reach the app at once and are judged in the background: - `deliver` (passive); - `discard` (deliver, but drop the writes it makes over newer data); - `defer` (wait for related in-flight work). Store writes are never held or reordered by default. Salient writes that no delivery decision covered are - decided in the background, and can be reverted within 2 s under strict rules. + decided in the background; in `guard` and `heal` one can be reverted only if the decision comes within 800 ms of + the write applying (`LATE_REVERT_MS`; 2 s before `0.2.0`) and nothing has overwritten it. 5. **Act by mode.** | mode | behaviour | |---|---| - | `observe` (default) | reports only; never holds, delays or changes anything | + | `observe` (default) | reports only; never holds, delays or changes a request, a response or a state write | | `guard` (opt-in) | `discard`, `defer`, `coalesce`, `delay`, only when the model's gate for that trigger says acting beats doing nothing by the margin of the chosen `aggressiveness` profile | - | `heal` (experimental) | also `retry`, `serve_cached`, `block`, `hedge`, `rollback`, `resync` and custom actions, at the heal margin of the profile | + | `heal` (experimental) | also `retry`, `serve_cached`, `block`, `hedge`, `rollback`, `resync` and plugin actions declared with `tier: "heal"` (the default; one declared `tier: "guard"` also runs in guard), at the heal margin of the profile | Detections are reported when the top diagnosis is not `expected` with probability at or above the profile's - report threshold (0.95 / 0.90 / 0.70 for cautious / balanced / eager). `rt.gates()` shows what is in force. + report threshold (0.95 / 0.90 / 0.70 for cautious / balanced / eager with model 0.2.0). `rt.gates()` shows what + is in force. 6. **Explain.** - Every detection and intervention is logged in plain English, with the exact situation text the model read. - Reversible actions can be undone. - `?genclass=off` installs nothing. -## Measured behaviour - -> **`@genclass/runtime@0.3.0-beta.1` with the model `@genclass/runtime-model@0.2.0`.** The model and the situation -> text it reads (`situation-v2.3`) are the same as in 0.2.x; the numbers below were measured with them. -> -> - **Runtime:** works and is unit-tested (situation format tag `situation-v2.3`). Install with -> `npx @genclass/runtime@beta init`, one import (`@genclass/runtime/auto`) or one script tag. -> - **Default mode** is `observe`: the model diagnoses salient situations and the runtime reports likely problems, -> without changing what the app does (it never holds or delays a response). `guard` is opt-in; `heal` is -> experimental. -> - **How eager it acts** once you opt in: `aggressiveness: "cautious" | "balanced" (default) | "eager"`. -> - **Model:** `genclass-runtime-r17` 2.0.0-rc4t (`r17-v2dT`, gain gate, 10 MB (10.2 MB q8), about 180 ms per -> decision on one WASM thread; WASM or WebGPU), loaded from jsDelivr at idle and cached. On held-out data -> ([RESULTS.md](docs/runtime/RESULTS.md) §1): -> -> | profile | guard FIR | guard recall (clear / real) | heal FIR | heal recall (clear / real) | report threshold | -> |---|---|---|---|---|---| -> | `cautious` | 0.005% | 2.4% / 1.4% | 0.26% | 3.7% / 3.8% | 0.95 | -> | `balanced` (default) | 0.13% | 7.8% / 7.5% | 0.59% | 7.1% / 10.4% | 0.90 | -> | `eager` | 0.54% | 14.2% / 21.0% | 1.84% | 17.1% / 24.2% | 0.70 | -> -> Our targets are guard FIR ≤ 0.1% and heal FIR ≤ 0.5%. `cautious` stays under both; `balanced` is slightly over -> both (0.13% and 0.59% on simulated apps); `eager` is well over. FIR on held-out real apps was 0.00% for every -> profile. Recall is still low: at `balanced`, guard acts on under 8% of the cases where acting would help. -> Installed from the registry into a fresh app, guard mode fixed an out-of-order typeahead in 6 of 6 trials, and -> clean typing made 0 model calls. See the [model card](packages/runtime-model/MODEL_CARD.md). -> - **Options** (activation, route scopes, protected endpoints, breaker, shadow, veto, action limits, sinks): -> [OPTIONS-SPEC.md](docs/runtime/OPTIONS-SPEC.md). -> - **Privacy notice:** anonymous diagnostics are on by default, without page text, and off on localhost (since -> `0.2.2`; `0.2.1` sends from localhost too, by mistake); opt out -> with `GenClass.init({ telemetry: false })` or `?genclass=no-telemetry`. See -> [Privacy and telemetry](#privacy-and-telemetry). +## How GenClass compares + +- **Error trackers and APM (Sentry, Datadog).** They collect exceptions, traces and timings. A stale response, a + double submit or a write race usually throws nothing, so there is no error to collect. GenClass looks for those + situations inside the page, decides on the user's device and reports what it saw. It does not replace an error + tracker: from `0.3.0` its reports can go to your own logging through `sinks` ([SINKS.md](docs/runtime/SINKS.md)), + and for apps in a genclass.dev account (any plan, Free included) the collector can forward likely detections to + Sentry, PostHog, Datadog, Slack or a signed webhook. +- **Product analytics.** Analytics count what users do; GenClass reports what the app got wrong while they did it. +- **Tests and linters.** These bugs depend on response order and user timing. A test on a fast, ordered network can + pass while production shows the bug. Use GenClass next to your tests, in development (the devtools overlay) and in + production. +- **AbortController, request ids, TanStack Query or SWR deduplication, idempotency keys.** These are the fixes, and + they work where they are used. GenClass reports places where one is missing; in `guard` it can discard a stale + write or coalesce a repeated request when its gate clears. + + + +## Measured results + +Every recall is reported next to its false-intervention rate (FIR): the share of cases where doing nothing was best +but the gate fired anyway. Source and history: +[RESULTS.md](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/RESULTS.md). + +**Shipped model:** `genclass-runtime-r17` 2.0.0-rc4t (`@genclass/runtime-model@0.2.0`, run `r17-v2dT`, gain gate), +10.2 MB q8, about 180 to 600 ms per model pass at 500 to 1,170 tokens on one WASM thread (estimated from runs in Node, +not in a browser), loaded from jsDelivr at idle and cached. The model and the +situation text it reads (`situation-v2.3`) are the same in `0.2.2` and in every `0.3.0` release so far +(`0.3.0-beta.0` to `0.3.0-beta.4`; each CHANGELOG entry says no model-visible text changed). On held-out data (§1): + +| profile | guard FIR | guard recall (clear / real) | heal FIR | heal recall (clear / real) | report threshold | +|---|---|---|---|---|---| +| `cautious` | 0.005% | 2.4% / 1.4% | 0.26% | 3.7% / 3.8% | 0.95 | +| `balanced` (default) | 0.13% | 7.8% / 7.5% | 0.59% | 7.1% / 10.4% | 0.90 | +| `eager` | 0.54% | 14.2% / 21.0% | 1.84% | 17.1% / 24.2% | 0.70 | + +The FIR columns are on simulated apps; on held-out real apps FIR was 0.00% for every profile. Our targets are guard +FIR ≤ 0.1% and heal FIR ≤ 0.5%: `cautious` stays under both, `balanced` is slightly over both, `eager` is well over. +Recall is still low: at `balanced`, guard acts on under 8% of the cases where acting would help. Calibration error +is not yet measured for this model (r17-v2b: 0.021 to 0.023). See the +[model card](https://github.com/genclass-dev/GenClass-lib/blob/main/packages/runtime-model/MODEL_CARD.md). + +| what | result | +|---|---| +| Compatibility matrix (7 apps: 6 frameworks and plain HTML; 15 data layers, 8 scenarios, 10 seeds; an unpublished 0.1.0-beta.4 build, commit `d04eff3`, which does not contain 0.2.0's audit trail, protect presets, hold-budget ceilings and invariant fixes (committed on another branch) and predates 0.2.1's first-request hold and telemetry defaults; model 0.2.0, `balanced`, 2026-10-10; not re-run on 0.2.2 or 0.3.0) | 0 bugs introduced in 3,510 runs; 0 actions on correct apps in 1,760 guard and heal runs; observe: 49 false findings in 880 correct-app runs; of 250 runs where the app's own bug showed, guard left 249 buggy and heal 240. 350 of 351 cells left the app at least as correct as without GenClass on every seed (observe: the same final page and server state); the other is observe on Next.js + SWR, scenario d, where on seed 7 GenClass's timing tipped the app's own race to the correct result without any action ([compat/RESULTS.md](https://github.com/genclass-dev/GenClass-lib/blob/main/compat/RESULTS.md), §16). | +| Never make a correct app worse (always-passive model, heal vs observe, 66 real apps × 6 seeds; runtime as of 0.1.0-alpha.1, not re-run) | 0 of 396 clean runs changed (final page text, inputs and alerts excluded, and server state). Request timing and store contents are not compared, nor is observe against no runtime. With chaos: 3 of 198. In the demos (guard, same runtime and model), a clean typeahead made 0 model calls and held for 0 ms (§4). | +| Six demo apps under network chaos (model 0.2.0, `balanced`, runtime 0.1.0-beta.4, an unpublished build of 2026-10-09; 10 chaos + 5 clean seeds per demo and mode) | Bug rate 72% off, 70% observe, 72% guard, 73% heal (differences under 5 points are noise): nothing is healed at the shipped gates, because candidate gains rarely clear the margins; 0 false interventions on clean runs (§5). | + +**Training.** A 150M teacher was trained and scored below r17-v2b on every held-out set, so distillation was skipped +(§1). Data so far: 10.4M simulated gold rows plus a 2.1M `situation-v2.3` top-up, 2.3M on-policy rows, 51M unlabeled +rows and 837,623 real-browser rows (§6). Recall is the open problem. + +
+Earlier models (not shipped today) + +| what | result | +|---|---| +| Round 1 model, R17 (9.6 MB q8; format `situation-v1`, not compatible with this runtime), simulated apps | Diagnosis 90.5%, action 81.9%, guard FIR 0.05%, heal FIR 0.24%; recall on clear stale and duplicate cases only 7.7% (precise but timid) (§1). | +| Why round 1 was timid ([sim/SEPARABILITY.md](sim/SEPARABILITY.md), §3) | 62 to 82% of clear actionable rows had a benign twin with identical visible facts; 24% of clear rows were mislabelled `expected`; labels assumed knowledge a runtime cannot have. The v2 format adds measured facts, with relabelled data. | +| r17-v2b (shipped as `@genclass/runtime-model@0.1.0`) | Simulated apps: diagnosis 84.2%, action 77.8%. Real apps: diagnosis 83.6%, action 80.0%. Guard FIR 0.01% with guard recall 0.6% of clear simulated cases; heal FIR 0.07% with heal recall 5.7% of actionable real-app cases (§1; an earlier, stricter fit, not exactly like-for-like with 0.2.0). | +| The same never-worse check on the v1 runtime (store-write holds) | 4 of 198 clean runs changed (2 apps); the React/Redux RealWorld app never rendered its home feed (§4). | + +
## Privacy and telemetry -- **Decisions are local.** Situations are built and decided in the browser by the local model; nothing is sent anywhere to make a decision. -- **Anonymous diagnostics are on by default** with `GenClass.init()` in a browser, except on local and private hosts (`localhost`, `*.localhost`, `*.local`, `*.test`, `127.0.0.1`, `::1`, private IP ranges) (`0.2.1` sends from local hosts too, by mistake; fixed in `0.2.2`, see the CHANGELOG), in Node/SSR, and in `createRuntime()`. They go to the GenClass maintainers' collector ([`telemetry-worker/`](telemetry-worker), a Cloudflare Worker writing to a private R2 bucket) and are used to improve the model. -- **By default no text from your page is sent.** Sent: a random per-page session id, versions, mode, device class, the app's hostname and id-normalised path (from `0.3.0` a segment with an `@`, such as an e-mail address, is `:id` too), and per decision (every detection and every decision whose diagnosis is not `expected`, every action and veto; of the routine ones, one in four and at most 200 per page load; counts are exact; `0.2.2` and earlier send every decision) the trigger, the diagnosis and its confidence, the action and gate, the outcome and latency. The collector stores no IP address or user agent. One console notice per page says it is on. -- **The measured visual checks send nothing** (new in `0.3.0`): their findings stay in the page (the console, `rt.checks()`, your own sinks). -- **The situation text the model read is sent only if you opt in** with `GenClass.init({ telemetry: { include: { situation: true } } })`. It is redacted by field meaning, but can still contain route names with query strings, short error messages and short field values; read [TELEMETRY.md](packages/runtime/TELEMETRY.md) before enabling it. -- **Opt out:** `GenClass.init({ telemetry: false })`, `?genclass=no-telemetry`, `localStorage.setItem("genclass.telemetry", "off")`; browsers sending Global Privacy Control are never collected. -- **Interaction analytics are off unless the app turns them on** (`analytics: true` with the app's token, new in `0.3.0-beta.1`): per page view the route, engaged time, scroll depth, and clicks, rage clicks and dead clicks per element, the element named from markup, never its text; and, within one page load, how each page view was reached, the route before it and the time between, the first element used on it, and where the page load ended with its routes in order. No visitor id; one `localStorage` entry (the visit flag, which links the page loads of a visit and says whether the browser visited before; `runtime.analytics.stop()` removes it), every opt-out above applies, and they never feed a decision. Where analytics need consent (EU, EEA, UK), start them after consent with `runtime.analytics.start()`. See [TELEMETRY.md](packages/runtime/TELEMETRY.md#interaction-analytics). -- **With a token** (`0.3.0`: `token` option, `data-token`, `token=` in the meta tag), each batch also carries - your app's public token, so the same diagnostics also show on your app's private dashboard (whoever holds its link - can open it), and decisions about `protect()`ed functions carry the function's name (`fn`, the label you chose). - Without a token neither is sent; without telemetry, a token does nothing. -- **Possible detections** (`0.3.0`, observe): their `detect` events carry `tier: "possible"` (a likely detection - has no `tier`, as before), and the page-load summary counts them apart from detections (`possibleDetections`). They - say which tier a detection was in; nothing new about the page is sent. -- **The CLI** contacts genclass.dev only for the command you run: `init` creates a dashboard only after you approve its - diff (it sends the `name` from `package.json`); `login` asks for a sign-in code with - `client_name: "genclass-runtime "` (shown on the approval page and in your account's session list) and - polls until you approve it; `whoami` and `connect` read your account's apps; `connect` sends a new app's name, or an - existing dashboard's link from `.genclass.local` to add it to your account; `logout` ends the CLI session. The - account requests carry `user-agent: genclass-runtime-cli/`. What an account stores: [PRIVACY.md](PRIVACY.md). -- Apps that ship GenClass may need to disclose this to their users (GDPR/CCPA). Full schema, storage and guidance: [TELEMETRY.md](packages/runtime/TELEMETRY.md). +- **Decisions are local.** The model runs in the browser; nothing is sent to make a decision. +- **Anonymous diagnostics are on by default** with `GenClass.init()`, `/auto` and the script tag on public hosts, + sent to the GenClass maintainers' collector ([`telemetry-worker/`](telemetry-worker), a Cloudflare Worker writing + to a private R2 bucket) to improve the model. One console line per page says it is on. Off on local and private + hosts (`localhost`, `*.local`, `*.test`, private networks) from `0.2.2` (`0.1.0-beta.3` to `0.2.1` send from them + too), in Node/SSR and in `createRuntime()`. +- **By default no text from your page is sent.** Sent: a random per-page session id, versions, mode, device class, + the page's hostname and id-normalised path, and per decision the trigger, the diagnosis and its confidence, the + action and gate, the outcome and latency (every detection, every decision whose diagnosis is not `expected`, every + action and veto; in `0.3.0` possible detections too, marked `tier: "possible"`, and routine decisions only one in + four, at most 200 per page load, with exact counts; `0.2.2` and earlier send every decision). The collector adds a + coarse country (two letters) and stores no IP address or user agent. +- **Opt out:** `GenClass.init({ telemetry: false })` (script tag: `data-telemetry="off"`; meta tag: + `telemetry=off`; or `window.GENCLASS_CONFIG = { telemetry: false }`), `?genclass=no-telemetry`, + `localStorage.setItem("genclass.telemetry", "off")` or `npx @genclass/runtime@beta init --no-telemetry`; browsers + sending Global Privacy Control are never collected. +- **The situation text the model read is sent only if you opt in** (`telemetry: { include: { situation: true } }`; + off by default since `0.2.1`, while `0.1.0-beta.3` to `0.2.0` send it by default). It is redacted by field + meaning, but can still contain route names with query strings, short error messages and short field values; read + TELEMETRY.md first. +- **With an app token** (`0.3.0`), batches also carry the token and the names of your `protect()`ed functions, so + they show on your app's dashboard. Without telemetry, a token does nothing. +- **Interaction analytics are off unless the app turns them on** (`analytics: true` with the app's token; new in + `0.3.0-beta.1`): per page view the route, engaged time, scroll depth, and clicks, rage clicks and dead clicks per + element (named from markup, never its text); and, within one page load, how each page view was reached, the route + before it and the time between, the first element used on it, and where the page load ended with its routes in + order. No visitor id; one `localStorage` entry (the visit flag, which links the page loads of a visit and says + whether the browser visited before; `runtime.analytics.stop()` removes it); never used for a decision, and every + opt-out above turns them off. Where consent is needed (the EU, EEA and UK), leave `analytics` unset and call + `runtime.analytics.start()` once the visitor consents + ([TELEMETRY.md](https://github.com/genclass-dev/GenClass-lib/blob/main/packages/runtime/TELEMETRY.md#interaction-analytics)). +- **The visual checks send nothing.** +- **The CLI** contacts genclass.dev only for the command you run (`init` creates a dashboard only after you approve + its diff). +- Apps that ship GenClass may need to disclose this to their users (GDPR/CCPA). Full schema, storage and guidance: + [TELEMETRY.md](https://github.com/genclass-dev/GenClass-lib/blob/main/packages/runtime/TELEMETRY.md); what an + account stores: [PRIVACY.md](https://github.com/genclass-dev/GenClass-lib/blob/main/PRIVACY.md). ## Security @@ -237,43 +359,28 @@ export const submitOrder = protect("checkout submit", (cart) => fetch("/api/orde - `rt.audit()` exports a structured record of every decision and action; payment, checkout and sign-in endpoints should stay observe-only (`requests: { protect: protectPreset("payments", "auth") }`). -Full guide (modes, triggers, adapters, observability, performance, privacy, known limitations): -**[packages/runtime/README.md](packages/runtime/README.md)**. API: [docs/runtime/API.md](docs/runtime/API.md). - -## Results so far - -All numbers are on held-out data, with each recall reported next to its false-intervention rate (FIR). Source: -[docs/runtime/RESULTS.md](docs/runtime/RESULTS.md). - -| what | result | -|---|---| -| Round 1 model, R17 (9.6 MB int8; **previous format** `situation-v1`), simulated apps | diagnosis 90.5%, action 81.9%, guard FIR 0.05%, heal FIR 0.24%, calibration error 0.009. Recall on clear stale/duplicate cases is only 7.7% (precise but timid). | -| Why round 1 was timid ([sim/SEPARABILITY.md](sim/SEPARABILITY.md)) | Many clear cases had benign twins with identical visible facts; 24% of clear rows were mislabelled `expected`; labels assumed knowledge a runtime cannot have. v2 adds measured facts and fixes the labels. | -| **r17-v2dT** (`situation-v2`; shipped as `@genclass/runtime-model@0.2.0`) | Gain gate with three profiles (table above). `balanced`: guard FIR 0.13%, heal FIR 0.59% on simulated apps (slightly over the 0.1% / 0.5% targets); `cautious`: 0.005% / 0.26% (under). 0.00% on held-out real apps for every profile. Calibration error not yet measured for this model (r17-v2b: 0.021–0.023). | -| r17-v2b (shipped as `@genclass/runtime-model@0.1.0`) | Simulated apps: diagnosis 84.2%, action 77.8%. Real apps: diagnosis 83.6%, action 80.0%. Guard FIR 0.01%, heal FIR 0.07%; heal acts on 5.9% of clear simulated cases and 5.7% of actionable real-app cases; guard on 0.6%. | -| Training continues | teacher model, distillation, DAgger rounds; ~10M simulated gold rows, ~50M unlabeled rows, ~0.6M real-app rows | -| Never make a correct app worse (always-passive model, heal vs observe, 66 real apps × 6 seeds) | 0/396 clean runs changed: final page text (inputs and alerts excluded) and server state. Request timing and store contents are not compared, nor is observe mode against no runtime. With chaos: 3/198 changed. | -| Same check on the v1 runtime (store-write holds) | 4/198 clean runs changed; the React/Redux RealWorld app never rendered its home feed | -| Demos (model 0.2.0, `balanced`, 10 chaos + 5 clean seeds per demo and mode) | The chaos bug rate over the six demos was 72% off, 70% observe, 72% guard and 73% heal (differences under 5 points are noise): nothing is healed at the shipped gates, because candidate gains rarely clear the margins; 0 false interventions on clean runs. | - -## What's in this repo - -| path | what | -|---|---| -| [`packages/runtime`](packages/runtime) | `@genclass/runtime`, the library. Observers, causality, stores and adapters, learned baselines/relations/profiles, facts, triage, policy gate, actions, model host (Web Worker, ONNX Runtime Web on WebGPU/WASM), devtools overlay. [README](packages/runtime/README.md) · [CHANGELOG](packages/runtime/CHANGELOG.md) | -| [`packages/mcp`](packages/mcp) | `@genclass/mcp` (unreleased), the MCP server for coding agents: pages connect through `@genclass/runtime/agent`; the agent gets findings in causal order, replays, explore verdicts and generated regression tests ([AGENT.md](docs/runtime/AGENT.md)). [README](packages/mcp/README.md) | -| [`packages/playwright`](packages/playwright) | `@genclass/playwright` (unreleased), GenClass for Playwright: a fixture that puts the runtime on every page under test (observe mode, seeded network chaos with `page.route`, the model skipped by default) and writes one race file per test, and a reporter that turns those files into one markdown report for the pull request, with the evidence line and what reproduced across seeds. [README](packages/playwright/README.md) | -| [`packages/nuxt`](packages/nuxt), [`packages/astro`](packages/astro) | `@genclass/nuxt` and `@genclass/astro` (unreleased): the runtime as a Nuxt module and as an Astro integration, one line in the framework's config, with options for the mode, the token, telemetry and the devtools overlay; browser only. [Nuxt README](packages/nuxt/README.md) · [Astro README](packages/astro/README.md) | -| [`packages/runtime-model`](packages/runtime-model) | `@genclass/runtime-model`, the default model package (data only; `files/` is gitignored and filled at release time) and its [model card](packages/runtime-model/MODEL_CARD.md) | -| [`sim`](sim) | Training-data simulator: random apps run on the real runtime in a deterministic virtual world, labelled by counterfactual outcomes | -| [`realapps`](realapps) | 128 real apps (written for the corpus, plus open-source ones) run with the real runtime in headless Chromium, labelled the same way; the never-worse sweep covered the first 66. [README](realapps/README.md) | -| [`training`](training) | Vocabulary pruning, curriculum (`curriculum/rt.py` mirrors the runtime's renderer), multi-node CPU training on Azure, int8 ONNX export, evaluation ([EVAL.md](training/EVAL.md)) | -| [`demos`](demos) | Six demo apps (typeahead, autosave, checkout, flaky dashboard, live kanban, real-time decisions) with a Service Worker chaos backend and Playwright trials; [`bench/heal`](bench/heal) runs them per mode, [`bench/visual`](bench/visual) measures the visual checks on them | -| [`compat`](compat), [`templates`](templates) | Framework compatibility matrix (7 apps, 15 data layers, every mode; [RESULTS.md](compat/RESULTS.md)) and eight starter templates with the one line (React + Vite, Next.js, Vue + Vite, SvelteKit, Astro, Nuxt, React Router, Angular; [templates/README.md](templates/README.md)) | -| [`docs/runtime`](docs/runtime) | [Results](docs/runtime/RESULTS.md) · [Coding agents](docs/runtime/AGENT.md) · [Architecture](docs/runtime/ARCHITECTURE.md) · [Build contract](docs/runtime/CONTRACT.md) · [API](docs/runtime/API.md) · [Threat model](docs/runtime/THREAT-MODEL.md) | -| [`telemetry-worker`](telemetry-worker), [`site`](site) | The diagnostics collector (Cloudflare Worker) and the genclass.dev site | -| [`research`](research) | Earlier GenClass/Jev research (voice-control Chrome extension, the `jev_local` encoder and server, classifier benchmarks); the training scripts still use `jev_local` ([README](research/README.md)) | -| [`AGENTS.md`](AGENTS.md), [`CONTRIBUTING.md`](CONTRIBUTING.md) | How to work in this repository | +## Documentation + +- [Docs index and glossary](docs/runtime/README.md): every document, what it answers and its status. +- [Package README](https://github.com/genclass-dev/GenClass-lib/blob/main/packages/runtime/README.md): the full guide + (modes, triggers, adapters, observability, performance, privacy, known limitations). +- [API reference](docs/runtime/API.md), [interception surface](packages/runtime/INTERCEPTION.md), + [CHANGELOG](packages/runtime/CHANGELOG.md). +- Measurements: [RESULTS.md](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/RESULTS.md) (every + measured number), [compat/RESULTS.md](https://github.com/genclass-dev/GenClass-lib/blob/main/compat/RESULTS.md) (the + compatibility matrix), the + [model card](https://github.com/genclass-dev/GenClass-lib/blob/main/packages/runtime-model/MODEL_CARD.md). +- Guides on genclass.dev: [race conditions in React](https://genclass.dev/guides/race-conditions-react), + [stale responses out of order](https://genclass.dev/guides/stale-responses-out-of-order), + [preventing duplicate submissions](https://genclass.dev/guides/prevent-duplicate-submissions). + +## Questions + +Short answers to the common questions (race conditions in React, stale responses, double submits, whether it changes +your app, what it sends, how big it is, what it costs, how it compares) are in [FAQ.md](FAQ.md). Ask in +[GitHub Discussions](https://github.com/genclass-dev/GenClass-lib/discussions/categories/q-a) or on +[Discord](https://discord.gg/kN9kRnFtZm); report bugs and false detections in +[Issues](https://github.com/genclass-dev/GenClass-lib/issues). ## Development @@ -288,30 +395,68 @@ NODE_OPTIONS=--expose-gc npx vitest run --exclude "test/browser/**" --exclude te NODE_OPTIONS=--expose-gc npx vitest run test/review-perf.test.ts --retry=2 # timing budgets, run alone ``` -The same steps run in CI ([.github/workflows/ci.yml](.github/workflows/ci.yml), Node 22) on pushes to `main`, -`runtime`, `mvp` and `mvp-v2`, on pull requests, and on manual dispatch. 667 unit tests pass in CI (14 skipped: the -model-parity tests, which need `GENCLASS_MODEL_DIR`), plus the 4 perf tests, which time a 5,000-item store and run -in their own step because they can fail under parallel load. +CI ([.github/workflows/ci.yml](.github/workflows/ci.yml), Node 22) runs on pushes to `main`, `runtime`, `mvp` and +`mvp-v2`, on pull requests, and on manual dispatch, in three jobs: + +- `runtime`: the typecheck, build and unit tests above (14 tests skip: the model-parity tests, which need + `GENCLASS_MODEL_DIR`), then the perf budgets in their own step, reported but not blocking, because they time a + 5,000-item store and can fail under parallel load; +- `visual-browser`: the first-load size against the release line, and the measured visual checks in Chromium with + Playwright; +- `telemetry-worker`: the collector's unit tests. + +`@genclass/playwright` has its own workflow, +[.github/workflows/playwright-reporter.yml](.github/workflows/playwright-reporter.yml) (typecheck, build, unit tests +and the fixture's end-to-end check in Chromium); it never blocks `ci.yml`. The +visual bench, [.github/workflows/visual-measure.yml](.github/workflows/visual-measure.yml), runs on manual dispatch +and on pushes that change `bench/visual/` on the `wave2/visual` branch, never on pull requests, because it loads +public sites. + +Not in CI, because they are heavier or need Azure (read [AGENTS.md](AGENTS.md) before running them): the model +Playwright suite (`npm run test:browser` in `packages/runtime`), the npm tarball smoke test (`test/smoke/smoke.sh`), +the install suite (`test/install/run-all.sh`), the sim, the real-app corpus, the demos' evaluation, and training. Releases are published by [.github/workflows/release.yml](.github/workflows/release.yml) from a `v*` tag, with npm provenance. -Not covered by CI: +**The training format is tagged.** The current tag is `situation-v2.3`, which runtime `0.1.0-beta.0` and later +render; the shipped model was trained on `situation-v2` data plus the `situation-v2.3` top-ups (simulated and real). +Any change to `packages/runtime/src/situation/*` (or other text the model reads) changes the model's input and means +a new tag and regenerated data (`training/curriculum/rt.py` mirrors the renderer and must change with it). -- Playwright browser tests: `npm run test:browser` in `packages/runtime`. -- `@genclass/playwright`: its typecheck, build, unit tests and the fixture's end-to-end check in Chromium run in - their own workflow, [.github/workflows/playwright-reporter.yml](.github/workflows/playwright-reporter.yml), on - pushes and pull requests that touch the package; it never blocks `ci.yml`. -- The npm tarball smoke test: `test/smoke/smoke.sh`. -- The sim, the real-app corpus, the demos' evaluation, and training. + -These are heavier or need Azure; see [AGENTS.md](AGENTS.md) before running them. +## Repository layout + +| path | what | +|---|---| +| [`packages/runtime`](packages/runtime) | `@genclass/runtime`, the library: observers, causality, stores and adapters, learned baselines, relations and profiles, facts, triage, policy gate, actions, model host (Web Worker, ONNX Runtime Web on WebGPU or WASM), measured visual checks, devtools overlay. [README](packages/runtime/README.md) · [CHANGELOG](packages/runtime/CHANGELOG.md) | +| [`packages/runtime-model`](packages/runtime-model) | `@genclass/runtime-model`, the default model package (data only; `files/` is gitignored and filled at release time) and its [model card](packages/runtime-model/MODEL_CARD.md) | +| [`packages/mcp`](packages/mcp) | `@genclass/mcp` (`0.2.0` on npm), the MCP server for coding agents: pages connect through `@genclass/runtime/agent`; the agent gets findings in causal order, replays, explore verdicts, generated regression tests and, with `genclass_scan`, the page audit ([AGENT.md](docs/runtime/AGENT.md)). [README](packages/mcp/README.md) · [CHANGELOG](packages/mcp/CHANGELOG.md) | +| [`packages/playwright`](packages/playwright) | `@genclass/playwright` (unreleased), GenClass for Playwright: a fixture that puts the runtime on every page under test (observe mode, seeded network chaos with `page.route`, the model skipped by default) and writes one race file per test, and a reporter that turns those files into one markdown report for the pull request, with the evidence line and what reproduced across seeds. [README](packages/playwright/README.md) | +| [`packages/nuxt`](packages/nuxt), [`packages/astro`](packages/astro) | `@genclass/nuxt` and `@genclass/astro` (unreleased): the runtime as a Nuxt module and as an Astro integration, one line in the framework's config, with options for the mode, the token, telemetry and the devtools overlay; browser only. [Nuxt README](packages/nuxt/README.md) · [Astro README](packages/astro/README.md) | +| [`packages/sv`](packages/sv) | `@genclass/sv` (unreleased), a community add-on for the Svelte CLI: `npx sv add @genclass` sets up GenClass in a SvelteKit app (the dependency, its import first in `src/hooks.client`, the devtools overlay in development). [README](packages/sv/README.md) | +| [`docs/runtime`](docs/runtime) | [Docs index](docs/runtime/README.md) · [Results](docs/runtime/RESULTS.md) · [Coding agents](docs/runtime/AGENT.md) · [API](docs/runtime/API.md) · [Architecture](docs/runtime/ARCHITECTURE.md) · [Build contract](docs/runtime/CONTRACT.md) · [Threat model](docs/runtime/THREAT-MODEL.md) | +| [`compat`](compat), [`templates`](templates) | Framework compatibility matrix (7 apps, 15 data layers, every mode; [RESULTS.md](compat/RESULTS.md)) and eight starter templates with the one line (React + Vite, Next.js, Vue + Vite, SvelteKit, Astro, Nuxt, React Router, Angular; [templates/README.md](templates/README.md)) | +| [`demos`](demos), [`bench`](bench) | Six demo apps (typeahead, autosave, checkout, flaky dashboard, live kanban, real-time decisions) with a Service Worker chaos backend and Playwright trials; [`bench/heal`](bench/heal) runs them per mode, [`bench/visual`](bench/visual) measures the visual checks | + +
+The rest of the repository + +| path | what | +|---|---| +| [`packages/genclass-runtime`](packages/genclass-runtime) | `genclass-runtime`, the former short name for the CLI: no longer published, and its newest npm version installs an unsupported runtime; use `@genclass/runtime` | +| [`sim`](sim) | Training-data simulator: random apps run on the real runtime in a deterministic virtual world, labelled by counterfactual outcomes | +| [`realapps`](realapps) | 158 real apps (144 written for the corpus, 14 open-source RealWorld front-ends) run with the real runtime in headless Chromium, labelled the same way; the never-worse sweep covered the first 66. [README](realapps/README.md) | +| [`training`](training) | Vocabulary pruning, curriculum (`curriculum/rt.py` mirrors the runtime's renderer), multi-node CPU training on Azure, ONNX export, evaluation ([EVAL.md](training/EVAL.md)) | +| [`telemetry-worker`](telemetry-worker) | The diagnostics collector, the dashboards and the accounts (Cloudflare Worker). The genclass.dev site and the page check are in a separate, private repository. | +| [`billing`](billing) | `plans.json`, the one place the genclass.dev plans are defined; the collector and the pricing page read it | +| [`research`](research) | Earlier GenClass/Jev research (voice-control Chrome extension, the `jev_local` encoder and server, classifier benchmarks); the training scripts still use `jev_local` ([README](research/README.md)) | +| [`scripts`](scripts) | Azure VM helpers for training runs | +| [`AGENTS.md`](AGENTS.md), [`CONTRIBUTING.md`](CONTRIBUTING.md) | How to work in this repository | -**The training format is tagged.** The current tag is `situation-v2.3`; the shipped model was trained on data from -`situation-v2`. Any change to `packages/runtime/src/situation/*` (or other text the model reads) changes the model's -input and means a new tag and regenerated data (`training/curriculum/rt.py` mirrors the renderer and must change -with it). +
## License -Apache-2.0. See [NOTICE](NOTICE). +Apache-2.0. See [LICENSE](LICENSE) and [NOTICE](NOTICE). diff --git a/SECURITY.md b/SECURITY.md index c51fb1f5..80d7a45a 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -9,8 +9,8 @@ state writes proceed. We take reports about it seriously. Please report security issues privately, not in public GitHub issues, discussions or pull requests. - **Email:** [mehar@daybot.ca](mailto:mehar@daybot.ca) -- **GitHub:** a private vulnerability report (Security → Report a vulnerability) on the GenClass-lib repository, if - private reporting is enabled there. +- **GitHub:** a private vulnerability report on the GenClass-lib repository + ([Security → Report a vulnerability](https://github.com/genclass-dev/GenClass-lib/security/advisories/new)). Include what you can of: the affected package and version (`GenClass.runtime.status`, or the version in `package.json`), how GenClass was installed (`@genclass/runtime/auto`, `GenClass.init`, the script tag), the mode @@ -38,13 +38,20 @@ GenClass is pre-1.0. Security fixes go into the newest published version only; u | `@genclass/runtime`, the newest published `0.2.x` release (npm dist-tag `latest`) | yes | | `@genclass/runtime`, the newest `0.3.0` prerelease (npm dist-tag `beta`, from `0.3.0-beta.0`) | yes, fixed in the next prerelease; older `0.3.0` prereleases no | | `@genclass/runtime`, all `0.1.0` betas and alphas | no | +| `genclass-runtime` (the former short name for the CLI, deprecated in this repository but not yet marked with `npm deprecate`; newest on npm 0.1.0-beta.3, which installs `@genclass/runtime` 0.1.0-beta.3 without a warning) | no: use `@genclass/runtime` | | `@genclass/runtime-model`, the version the supported runtime loads by default (`DEFAULT_MODEL_BASE_URL`) | yes | | `research/` and the other directories of this repository that are not published packages | no | ## In scope -- the runtime library (`packages/runtime/src`), its zero-code entries (`/auto`, the script-tag build), the `init` / - `remove` CLI (`packages/runtime/bin`), the devtools overlay; +- the runtime library (`packages/runtime/src`), its zero-code entries (`/auto`, the script-tag build), the devtools + overlay; +- the CLI (`packages/runtime/bin`: `init`, `remove`, `login`, `connect`, `whoami`, `logout`, `fetch-model`, `info`), + including the session credentials `login` stores in `~/.config/genclass/credentials.json` + (`$XDG_CONFIG_HOME/genclass/` when that is set to an absolute path; `%APPDATA%\genclass\` on Windows; on macOS and + Linux, file mode 600 in a directory with mode 700), and the app token and private dashboard link `init` saves in + `.genclass.local` in the project (it makes sure git ignores that file, adding it to `.gitignore` when needed; the + link is a secret: whoever holds it sees the app's dashboard); - the model loader and host (integrity checks, Cache Storage, the worker); - the telemetry client and the collector (`telemetry-worker/`), including anything that sends more than [TELEMETRY.md](packages/runtime/TELEMETRY.md) says; diff --git a/bench/explore/README.md b/bench/explore/README.md index 04153c8c..fc6e87f5 100644 --- a/bench/explore/README.md +++ b/bench/explore/README.md @@ -2,7 +2,7 @@ Measures `genclass_explore` (docs/runtime/AGENT.md section 5) the way an agent uses it: two fifo baselines and seeded reorderings of the same user steps, compared by the MCP server. Every number goes to -[RESULTS.md](../../docs/runtime/RESULTS.md) section 20 with its false-report rate next to it. +[RESULTS.md](../../docs/runtime/RESULTS.md) section 22 with its false-report rate next to it. ## What is here diff --git a/bench/hunt/README.md b/bench/hunt/README.md index f2e1f434..9ee3a730 100644 --- a/bench/hunt/README.md +++ b/bench/hunt/README.md @@ -69,7 +69,7 @@ node bench/hunt/scenarios/e2e-search-server.mjs --port 4778 --fixed & GENCLASS_APP_URL=http://127.0.0.1:4778 npx playwright test --config bench/hunt/results/e2e-buggy/tests/playwright.config.mjs ``` -The committed reports are in `results/e2e-*/`; RESULTS.md section 18 has the counts. +The committed reports are in `results/e2e-*/`; RESULTS.md section 20 has the counts. ## A real app: Strapi 5.59.0 (`results/sample/`, `results/strapi-search/`) diff --git a/bench/visual/README.md b/bench/visual/README.md index 569c8a38..813dda74 100644 --- a/bench/visual/README.md +++ b/bench/visual/README.md @@ -89,6 +89,6 @@ gh run download --repo genclass-dev/GenClass-lib --name visual-summary On the visual branch, a push that changes `bench/visual/` starts it (`[smoke]` in the commit message: two pages per group). Locally, `summarize.mjs` runs on downloaded artifacts: `node bench/visual/summarize.mjs --labels bench/visual/labels.json`. The demos and templates groups also run on a machine with Chromium and 16 GB -(2026-10-25, RESULTS.md section 22): `node bench/visual/build.mjs`, `node bench/visual/corpus.mjs prepare demos` +(2026-10-25, RESULTS.md section 24): `node bench/visual/build.mjs`, `node bench/visual/corpus.mjs prepare demos` (templates: with `VISUAL_BENCH_WORK=`, which `npm install`s each starter), then `node bench/visual/run.mjs --group demos --parts clean,cost,faults`. The public sites run only in the workflow. diff --git a/docs/runtime/AGENT.md b/docs/runtime/AGENT.md index 7608cc61..54698c63 100644 --- a/docs/runtime/AGENT.md +++ b/docs/runtime/AGENT.md @@ -35,7 +35,7 @@ Enterprise offer ([genclass.dev/enterprise](https://genclass.dev/enterprise)). ``` Status: new in `@genclass/runtime` 0.3.0-beta.1 (unreleased) and `@genclass/mcp` 0.1.0 (unreleased). Measured -results are in section 7 and in [RESULTS.md](RESULTS.md) section 16. +results are in section 7 and in [RESULTS.md](RESULTS.md) section 18. ## 1. Setup @@ -102,7 +102,7 @@ lists. It never writes to the event log and never feeds a situation, a decision a write that only reordered the field's elements or entries (the same multiset of elements, the same keys and values: a cache's list of keys re-sorted by arrival puts nothing older in place of anything newer), and a path the hub's layout re-cut in the same write (an object past 32 keys or a store past 200 fields becomes one field, or the - reverse: the path's "change" is GenClass's layout, not data; RESULTS.md section 20 counts the witnesses and + reverse: the path's "change" is GenClass's layout, not data; RESULTS.md section 22 counts the witnesses and stale-write findings before and after). - `identical-in-flight`: two identical requests (same method, URL and body) in flight together, started by two separate user actions. Not counted: a request the app aborted, GenClass's own retries and hedges, and keystrokes @@ -306,7 +306,7 @@ What a seeded run ends with beyond that is sorted by where it shows, from the ev The internal tier exists because a data-fetching cache keeps bookkeeping whose final value depends on arrival order by construction, with nothing visible attached: on the Strapi admin (Redux Toolkit Query) the request ids, the fulfilled timestamps, the lists of cache keys a tag provides and the subscriptions differed between orders in every session -while the page did not (RESULTS.md section 20). The rule is generic and measured on the run itself, not a list of +while the page did not (RESULTS.md section 22). The rule is generic and measured on the run itself, not a list of library fields: a store value that matters to the user reaches the page, and when it does the run is in the visible tier, with the store fields listed beside the page lines. Its limit: a difference that shows only in a control's state (a button left disabled, a spinner without text) is not in the page text, so it lands in the internal tier @@ -413,7 +413,7 @@ well, fixed in `src/observe/xhr.ts`. Tests: 36 in `packages/runtime/test/agent.test.ts`, 104 in `packages/mcp/test/`; the runtime's whole suite passes, and the first load stays within its limits. -False reports on real apps (RESULTS.md section 20, the Strapi admin with the hunt harness and two small apps on Redux +False reports on real apps (RESULTS.md section 22, the Strapi admin with the hunt harness and two small apps on Redux Toolkit Query and TanStack Query): before the rules of section 5.1, 16 of 21 interaction sessions on Strapi got a race-like verdict (62 reordered runs, all false by a DOM oracle) and a correct Redux Toolkit Query page got "RACE" in 18 of 18 reordered runs; after, 0 of 21 and 0 of 18, with the real 403 race still found in 6 of 6 reordered runs at @@ -435,7 +435,7 @@ This is one page and one bug class, a proof of the mechanism, not a recall figur budget (1,000 fields) are not compared; the report says when that happened. - An overwrite that a later response repaired (a flicker) is reported along the way, not as a race; the stale-write findings each witness makes stay as they were, so on an app whose cache writes its own bookkeeping (Redux Toolkit - Query's statuses, subscriptions) `genclass_findings` still lists those (RESULTS.md section 20 counts them). + Query's statuses, subscriptions) `genclass_findings` still lists those (RESULTS.md section 22 counts them). - WebSocket and EventSource messages are recorded, not reordered; failure responses are not held. Both are next. - The script-tag build has no agent entry yet (ESM only). - Production: today the agent reads production through the dashboard (counts by diagnosis and page) and reproduces @@ -444,7 +444,7 @@ This is one page and one bug class, a proof of the mechanism, not a recall figur failing session, without page text) is the natural next step and widens what telemetry sends, so it needs the owner's OK first (AGENTS.md rule 8). - Next measurement: explore on the six demos and the real-apps sweep, reporting races found next to false reports on - their correct variants (the Strapi admin and two small apps are measured, RESULTS.md section 20). + their correct variants (the Strapi admin and two small apps are measured, RESULTS.md section 22). ## 9. Relation to the rules diff --git a/docs/runtime/ANALYTICS.md b/docs/runtime/ANALYTICS.md index 7f64ada6..0ddea03e 100644 --- a/docs/runtime/ANALYTICS.md +++ b/docs/runtime/ANALYTICS.md @@ -1,14 +1,28 @@ -# Interaction analytics: design +# GenClass interaction analytics: design -Status: design written 2026-10-17 and **approved by the owner the same day** with every recommendation in +GenClass interaction analytics count page views, engaged time, scroll depth and clicks (including rage and dead +clicks) per element, for an app's own dashboard on genclass.dev. They are off by default, need telemetry on and the +app's token (or a custom telemetry endpoint), are turned off by every telemetry opt-out and Global Privacy Control, +never record element text, and never reach the model or a decision. Status: merged into `main` on 2026-10-18 and +shipped in `@genclass/runtime` 0.3.0-beta.1 (2026-10-22, npm's `beta` dist-tag); visits, funnels, navigation paths +and the version check (sections 1.2, 3.1, 11 and 4.1) followed in 0.3.0-beta.2 (2026-10-25); the newest release is +0.3.0-beta.4 (2026-10-27). npm's `latest`, 0.2.2, has no analytics (section 4.1). + +Design written 2026-10-17 and **approved by the owner the same day** with every recommendation in [section 9](#9-decisions-for-the-owner) (AGENTS.md rule 8: widening what telemetry sends needs that OK). **Built** 2026-10-18 on branch `feat/interaction-analytics`: the runtime (`packages/runtime/src/analytics/`), the collector and the dashboard (`telemetry-worker/`), and the admin filter for genclass.dev (`site/src/admin.js`); the differences -from this note are in [section 10](#10-implementation-notes). Not released: genclass.dev needs a runtime release -first (section 7). +from this note are in [section 10](#10-implementation-notes). First-load cost: +[RESULTS.md section 15](RESULTS.md#analytics). Adopting them on genclass.dev itself is section 7: the site pinned +0.2.2 when this was written, and its pages were unchanged as of 2026-10-25 (sections 10 and 11.3). + +**Where the genclass.dev paths are now.** Since 2026-10-18 the website and its admin (`site/`) live in the private +repository `genclass-dev/genclass-site`, which includes this repository as its `lib/` submodule. The paths `site/...` +in sections 6, 7 and 10 (`site/src/admin.js`, `site/src/worker.js`, `site/src/pages/privacy.html`) are in that +repository; `telemetry-worker/` and `billing/plans.json` stay here. **Product analytics (2026-10-25).** The owner asked for analytics "like PostHog in the dashboard ... choose the most -easy and reliable way". Section [1.2](#12-visits-sources-devices-unreleased) adds visits, where they came from, +easy and reliable way". Section [1.2](#12-visits-sources-devices-2026-10-25) adds visits, where they came from, devices, countries, new or returning, entry and exit pages and funnels, inside the same `analytics: true` opt-in and the same no-text rule; section [3.1](#31-visits-and-funnels) says how the collector counts them. @@ -92,7 +106,7 @@ is queued when a view ends, when the tab is hidden (before the existing hidden f `summary`, and only if something changed. At most 50 element keys per event for `clicks` and 20 each for `rage` and `dead`; the rest are added up under `(other)`. A view of a few minutes sends one or two events of well under 2 KB. -### 1.2 Visits, sources, devices (Unreleased) +### 1.2 Visits, sources, devices (2026-10-25) Built 2026-10-25 (branch `dash-analytics`) on the owner's request above, inside `analytics: true`; nothing changes without it. @@ -219,9 +233,9 @@ GenClass.init({ token: "gc_…", analytics: true }); ### 4.1 Which runtime: loud, never silent Interaction analytics first shipped in `@genclass/runtime` **0.3.0-beta.1**, on npm's `beta` dist-tag (install it with -`npm install @genclass/runtime@beta`, `npx @genclass/runtime@beta init`, or a script tag pinned to -`@0.3.0-beta.1`). npm's `latest` is still `0.2.2` until 0.3.0 is stable, and `0.2.x` and `0.3.0-beta.0` have no -analytics: they ignore `analytics: true` without a word (reproduced on 0.2.2 in jsdom: no warning, no +`npm install @genclass/runtime@beta`, `npx @genclass/runtime@beta init`, or a script tag pinned to the exact version, +`@0.3.0-beta.4` on 2026-10-27). npm's `latest` is still `0.2.2` until 0.3.0 is stable, and `0.2.x` and `0.3.0-beta.0` +have no analytics: they ignore `analytics: true` without a word (reproduced on 0.2.2 in jsdom: no warning, no `runtime.analytics`, nothing sent; a TypeScript app gets a type error for the unknown option). Those releases cannot change, so every path that the newest line controls says it instead: @@ -315,8 +329,9 @@ a consent banner (section 7 step 4). default-on clickstream from every app using GenClass would change what the library is. 3. **Element key sources**: data attributes, then ids, then link targets, then stable class names, then a structural path. Recommended. The stricter alternative drops ids and class names (more `nth-of-type` paths, less readable). -4. **Storage**: `interactions` events stored in R2 with the batch (90 days, and in the paid export), but never used - for model training. The alternative strips them from R2 and keeps only the dashboard counters. +4. **Storage**: `interactions` events stored in R2 with the batch (90 days, 1 year on the Enterprise plan, and in + the data export, which was a paid feature when this was written and is on every plan since 2026-10-19), but never + used for model training. The alternative strips them from R2 and keeps only the dashboard counters. 5. **genclass.dev**: which of the two options in section 7 step 3, and how to handle consent for EU, EEA and UK visitors (section 7 step 4). diff --git a/docs/runtime/API.md b/docs/runtime/API.md index 3822759e..759c3a38 100644 --- a/docs/runtime/API.md +++ b/docs/runtime/API.md @@ -1,8 +1,19 @@ -# @genclass/runtime API reference - -GenClass Runtime watches a web app from the inside (user actions, async operations, network, state changes, -errors, timing and causality) and asks a small local model, running in the browser, about situations that look -risky. Depending on the mode it reports what it saw, or prevents the failure with a minimal, reversible action. +# GenClass Runtime API reference (@genclass/runtime) + +This page describes every export, option, event, action and type of `@genclass/runtime` on `main`. +`npm i @genclass/runtime` installs 0.2.2 (`latest`); `npm i @genclass/runtime@beta` installs 0.3.0-beta.0. Items +marked 0.3.0 are not in 0.2.x; items marked 0.3.0-beta.1 (interaction analytics, the devtools redesign) are not on +npm yet. + +GenClass Runtime (`@genclass/runtime`) is an open-source (Apache-2.0) browser library that reports race conditions in +web apps while they happen: stale responses that land over newer data, duplicate requests such as double submits, +inconsistent state and failure storms. A small model runs on the user's device, in a Web Worker on WebGPU or WASM, +and each report comes with its evidence. By default it only reports (`observe`): it never holds, delays or changes a +request, a response or a state write. The opt-in `guard` mode may act on a few of them: on held-out data it acted on +7.5% of actionable real-app cases at a 0.00% false-intervention rate (7.8% of clear simulated cases at 0.13%), but in +the six demos no mode lowered the bug rate ([RESULTS.md §1](RESULTS.md#1-model-stages-compared-held-out-simulated-apps), +[§5](RESULTS.md#5-demos)). The `heal` mode is experimental. Terms are defined in the [glossary](README.md#glossary); +measured numbers are in [RESULTS.md](RESULTS.md). ```ts import { GenClass } from "@genclass/runtime"; @@ -40,10 +51,14 @@ isValidToken(v: unknown): v is string Kill switch, for "is my app broken or did GenClass change something?": - `?genclass=off` in the page URL, or `localStorage.genclass = "off"`: `init` installs nothing and logs one line. -- `?genclass=observe|guard|heal`: overrides the mode, within limits (`src/index.ts` -> `initUnsafe`; a missing - `mode` counts as `guard` there): `observe` always; `guard` when no mode, `guard` or `heal` is configured (the - default `/auto` and script tag included, not `/auto/observe`); `heal` only when `heal` is configured, or with - `debug: true`. +- `?genclass=observe|guard|heal` (or the same value in `localStorage.genclass`): sets the mode for that visitor, but + only to demote it from the app's mode (from `observe` when the app sets none), so a crafted link cannot raise the + mode of a site on a public host (`src/index.ts` -> `initUnsafe`). It may raise the mode with `debug: true`, or on + a local or private host (`isLocalHost` in `src/telemetry/config.ts`: `localhost` and `*.localhost`, `*.local`, + `*.test`, `*.internal`, `*.home.arpa`, single-label intranet names, an empty host name such as a `file:` page's, + and loopback, private, link-local, CGNAT (100.64/10) and 0/8 addresses). That includes a production app served from such a host, + such as an intranet app; `routes: [{ match: "*", mode: "observe" }]` keeps it at observe whatever the URL says. + Before 0.2.2 it could also raise the mode to `guard` when the app set no mode. Outside a browser (no `window`/`document`, e.g. SSR), `GenClass.init()` returns an inert runtime: no observers, no model. Use [`createRuntime`](#headless-use-tests-ssr-simulation) for headless work. @@ -170,8 +185,10 @@ interface InitOptions { report?: "console" | "interventions" | "silent" | ((r: Report) => void); // "interventions": console prints interventions, undos, breaker only learn?: { persist?: boolean | "local" | "session"; key?: string; version?: string; maxBytes?: number; // persist [unset: profiles in memory; flows in localStorage when expectations are on; true/"local"/"session": both; false: nothing]; key ["genclass:learn"]; version [session.tags.release]: mismatch discards; maxBytes [200000] expectations?: boolean; minSupport?: number; minShare?: number; minSessions?: number }; // [false: opt-in until its false-report rate is measured], [8 in one page load, 20 with stored counts], [0.95], [3 with localStorage, else 1] (docs/runtime/EXPECTATIONS.md) - // model: also loadIf [{ minDeviceMemoryGB: 4, saveData: "skip" }: no model on devices reporting ≤ 2 GB or on Save-Data; () => true forces], - // inlineFallback [true; false: a failed worker means no model, never the main thread], threads ["auto"], timeoutMs [10000], + // model: also loadIf [0.3.0: { minDeviceMemoryGB: 4, saveData: "skip" }: no model on devices reporting ≤ 2 GB or on Save-Data; () => true forces; + // 0.2.x: { saveData: "lazy" }], + // inlineFallback [true; false (0.3.0): a failed worker means no model, never the main thread; 0.2.x ignored false], + // threads ["auto"], timeoutMs [10000], // maxDecisionsPerMinute [30], unloadAfterIdleMs [false] } ``` @@ -197,10 +214,11 @@ mode/allow/deny/requireDiagnosis (at the effective mode) → thresholds → `pol (`limit:perMinute|perSubject|perSession`; defaults 60/min global, 10/min per subject, 200 per session) → `onBeforeAction` (`vetoed`, `would-veto`, or `limit:hold` past the budget) → execute. effectiveMode = min(mode, sample cap, breaker cap, route rule); URL overrides (`?genclass`, `?genclass-mode`, `?genclass-aggr`, `?genclass-sample`, `localStorage.genclass`) only demote, from the -app's mode or from `observe` when it sets none; they may raise only with `debug: true` or on a development host -(`localhost`, `*.local`, `*.test`, private networks). Before 0.2.2, `?genclass=guard` opted a visitor into guard when -the app set no mode. `policy.holdBudgetMs` is a hard ceiling that includes defers, the -wait for a response or request body, a held write's deferred re-decisions and the veto hook. +app's mode or from `observe` when it sets none; `?genclass=` / `localStorage.genclass` may also raise on a development +host (http(s) on a loopback name with an explicit port, or `*.test`; `src/devhost.ts`), and since 0.2.3 and 0.3.0-beta.2 +`debug: true` raises nothing ([Init and modes](#init-and-modes)). Before 0.2.2, +`?genclass=guard` opted a visitor into guard when the app set no mode. `policy.holdBudgetMs` is a hard ceiling that +includes defers, the wait for a response or request body, a held write's deferred re-decisions and the veto hook. Protected requests (`requests.protect`) are never held, never wait for their body to be read, and neither they nor anything their response callbacks cause (writes, timers, follow-up requests: the ops inherit the protection) are @@ -280,7 +298,9 @@ still on its way), put back the value of an unconfirmed local change (a user act request is still in flight), or replace text the user typed meanwhile. The model may let it through (`deliver`), let it through but drop the writes it makes over newer data (`discard`: each later `set()` of that response's chain applies its other fields and skips those), or hold it until the related requests finish and decide again (`defer`). Holding -only adds latency; messages of one socket or stream keep their order. Before asking, it reads the response (a +only adds latency; messages of one socket or stream keep their order. Only guard and heal hold a delivery, and only +when an action is permitted and the model can answer within the hold budget; in observe, and whenever no hold is +possible, the delivery is released at once, exactly as without GenClass, and decided in the background. Before asking, it reads the response (a clone, ≤ 256 KB of JSON) and compares it with the store: a response that equals the current values changes nothing and is let through without a model call. The situation then says what the response would do: put back a value a newer operation replaced (per field, or per item cell of a list joined by id), replace text the user typed since the request started (with a @@ -288,9 +308,9 @@ preview centred on the difference), or reload a list without the item a recent c `set()` always applies at once (`set(x); get()` returns x) and notifies subscribers; writes to one store apply in the order they were made. A write that looks wrong on its own (not covered by a delivery decision) is decided in the -background: if the model answers `discard` within 2 s after the write applied, nothing has overwritten it and the -same operation made no other write since, GenClass reverts exactly that write (a "late revert", reported as such and -undoable). +background: if the model answers `discard` within 800 ms after the write applied (`LATE_REVERT_MS`), nothing has +overwritten it and the same operation made no other write since, GenClass reverts exactly that write (a "late +revert", reported as such and undoable). Before 0.2.0 the window was 2 s. `policy.holdWrites: true` (opt-in) holds salient writes until the model answers (at most the hold budget): a later write to the same store first applies the earlier held ones in order, and `get()` inside the writing operation @@ -507,7 +527,7 @@ every observer and restores the globals it wrapped. | action | tier | effect | |---|---|---| -| `discard` | guard | delivery: deliver it, but drop the state changes its operation makes over newer data (`ActionRecord.dropped` lists them; undo: apply them now). mutation: revert exactly that write if decided ≤ 2 s after it applied, nothing overwrote it and its operation wrote nothing else since (undo: re-apply); with `holdWrites`, drop the held write | +| `discard` | guard | delivery: deliver it, but drop the state changes its operation makes over newer data (`ActionRecord.dropped` lists them; undo: apply them now). mutation: revert exactly that write if decided ≤ 800 ms after it applied, nothing overwrote it and its operation wrote nothing else since (undo: re-apply); with `holdWrites`, drop the held write | | `defer` | guard | hold the response/message (or held write) until related requests finish, then decide again (twice at most) | | `coalesce` | guard | do not send; reuse the response of the identical request in flight or just finished (`x-genclass: coalesced`) | | `delay` | guard | wait min(250 ms · 2^failure streak, 8 s), then send | @@ -546,21 +566,25 @@ match anything. ```ts interface PolicyOptions { - thresholds?: { report?: number; guard?: number; heal?: number }; // overrides; else the model's own gate, else 0.6 / 0.9 / 0.8 + thresholds?: { report?: number; guard?: number; heal?: number }; // overrides, read in the model's gate kind (mass: probabilities; + // gain: margins); else the model's own gate, else 0.6 / 0.9 / 0.8 (gain: 0.6 / 2 / 2) allow?: string[]; // only these non-passive actions may run deny?: string[]; // these never run holdBudgetMs?: number | "auto"; // default "auto": clamp(1.5 × median recent model latency, 150, 800) ms - holdUserWrites?: boolean; // default false + holdUserWrites?: boolean; // default false: with holdWrites on, writes made synchronously by a user handler still apply at once idempotencyHeaders?: string[]; // default ["Idempotency-Key", "X-Idempotency-Key"]: a POST/PATCH carrying one may be retried idempotencyBodyFields?: string[]; // opt-in, default []: top-level JSON body fields (e.g. "request_id") the server dedupes on holdWrites?: boolean; // default false: state writes apply at once (decided at the network boundary) - maxActionsPerMinute?: number; // default 60 + actionLimits?: { perMinute?: number; perSubject?: number; perSession?: number }; // default 60 / 10 / 200 + maxActionsPerMinute?: number; // deprecated alias of actionLimits.perMinute (ignored when that is set) requireDiagnosis?: boolean; // default true } ``` Thresholds: the model's action probabilities are calibrated against the gate it ships with (`gate` in its -meta.json, visible as `runtime.status.gate`). Two gate kinds exist: +meta.json, visible as `runtime.status.gate`). Model 0.2.0, the default, ships a gain gate with three profiles (its +margins and report values are in the [model card](../../packages/runtime-model/MODEL_CARD.md)). Two gate kinds +exist: - `kind: "mass"` (the default, also when meta.json has no kind): the most probable permitted action runs when the summed probability of the permitted actions reaches its tier's threshold (`{ report?, guard: { default, byTrigger? }, heal: {...} }`, probabilities; defaults 0.9 / 0.8). @@ -582,10 +606,13 @@ each comes from (`policy`, `model`, `default`); every decision records `gateKind and `margin` (gain), with `thresholdSource`; `explain(id).gates` has the full set. The permitted actions are the applicable non-passive actions the mode allows (observe: none; guard: guard tier; -heal: both), minus denied ones (only allowed ones when `allow` is set). GenClass runs the most probable permitted -action only if all of these hold, otherwise the passive action runs and the decision's `reason` says why: the -summed probability of the permitted actions reaches that action's tier threshold; the model's top diagnosis is not -`expected` (unless `requireDiagnosis: false`); fewer than `maxActionsPerMinute` actions ran in the last minute; the +heal: both), minus denied ones (only allowed ones when `allow` is set); a protected or cross-origin subject, or one +whose operation started under an `off` or `observe` route rule, has none. GenClass runs the most probable permitted +action only if all of these hold, otherwise the passive action runs and the decision's `reason` says why: the gate +clears (mass: the summed probability of the permitted actions reaches that action's tier threshold; gain: ĝ of that +action is above its tier's margin); the model's top diagnosis is not +`expected` (unless `requireDiagnosis: false`); the action limits allow it (`policy.actionLimits`: by default 60 a +minute, 10 per subject a minute, 200 per session); the `onBeforeAction` hook, if any, does not veto it; the decision arrived within the hold budget (a late `discard` may still revert the write, see State). While the model is loading, everything proceeds unchanged: guard and heal build no situation then, as before 0.3.0; in observe (0.3.0, where the possible tier is in force) a salient situation is built at its trigger and decided once the model is ready, @@ -617,8 +644,10 @@ in `rt.explain(id)`; it reaches the devtools as a dimmed card left out of the co is reported at the possible tier at most. `explain()`, the console, the devtools and telemetry all read the tier recorded with the decision. The tiers are reporting only: no action gate reads them. +An intervention line in guard mode (captured with model 0.1.0): + ``` -[GenClass] Prevented a stale response: search.results was written twice by other operations since its operation (#6) started (version 1 → 3), last 0.69s ago by GET /api/search?q=reac (#8), which started 0.09s after #6, from a later user action (#7). Delivered the response to GET /api/search?q=rea (#6) and dropped the state changes it makes over newer data (search.results). (stale, 0.97; discard 0.97) +[GenClass] Prevented a stale response: search.results was written twice by other operations since its operation (#6) started (version 1 → 3), last 0.69s ago by GET /api/search?q=reac (#8), which started 0.09s after #6, from a later user action (#7). Delivered the response to GET /api/search?q=rea (#6) and dropped the state changes it makes over newer data (search.results). (stale, 0.97; discard 0.98) ``` ```ts @@ -642,7 +671,9 @@ overlay uses it), so debugging tools never become causes in situations. 0.3.0 (CHANGELOG): the adapters, the `@genclass/runtime/sinks` exports, sinks by name and `init --sink` are not in `0.2.x`. The runtime has no Sentry, PostHog or Datadog adapter. On every genclass.dev plan, Free included, the GenClass -collector forwards detections to those tools once you turn it on (https://genclass.dev/pricing). +collector forwards likely detections to those tools once you turn it on for an app in your genclass.dev account that +sends telemetry with its token; with telemetry off nothing is forwarded, and a forwarded record never carries page +text (https://genclass.dev/pricing; OPTIONS-SPEC §4.11). `sinks` receive a `SinkRecord` per report (OPTIONS-SPEC §4.11): `kind`, `id`, `ts`, `sessionId`, `tags`, `mode`, `sampled`, `diagnosis`, `p`, `detection` (kind `detection`: `"likely"`, or `"possible"` for a sink with @@ -968,9 +999,11 @@ Subpath exports: `@genclass/runtime/react` (`useGenClassState(name, initial, opt `@genclass/runtime/sinks/{console-json,webhook}` (each also its default export; see "See it where you already look"; there is no Sentry, PostHog or Datadog adapter in the free runtime), `@genclass/runtime/visual` (`checkPage(options?)`, the measured visual checks without a runtime; 0.3.0). They are documented with their source. -The devtools overlay is in genclass.dev's black design whatever the page's theme (`theme: "dark"`, the default since -0.3.0; `"light"` is the same design on white, `"auto"` follows `prefers-color-scheme`). To integrate another state -library use: +From 0.3.0-beta.1 (not on npm yet) the devtools overlay is in genclass.dev's black design whatever the page's theme +(`theme: "dark"`, the new default; `"light"` is the same design on white, `"auto"` follows `prefers-color-scheme`). +0.3.0-beta.0 and 0.2.x use their earlier design and default to `"auto"`. + +To integrate another state library use: ```ts rt.adapter(name: string, io: { get(): T; set?(v: T): void; subscribe?(fn: () => void): () => void }, opts?): { @@ -1073,5 +1106,6 @@ credential-like strings (JWTs, API keys). Query parameters and body keys follow inputs with `autocomplete` cc-* / one-time-code are never recorded. Element text such as a kanban "card" is not a secret. Response bodies are read only from a clone the runtime already keeps (≤ 256 KB) and only to compare them with the store. Decisions are made in the browser by the local model; the only data GenClass sends is its anonymous -telemetry (on by default with `GenClass.init()` in a browser; [TELEMETRY.md](../../packages/runtime/TELEMETRY.md)), -which includes the redacted situation text. \ No newline at end of file +telemetry (on by default with `GenClass.init()` in a browser on a public host; +[TELEMETRY.md](../../packages/runtime/TELEMETRY.md)), which includes the redacted situation text only when the app +sets `telemetry: { include: { situation: true } }`, and what the app's own sinks send. \ No newline at end of file diff --git a/docs/runtime/ARCHITECTURE.md b/docs/runtime/ARCHITECTURE.md index 042d67a4..0c3fd445 100644 --- a/docs/runtime/ARCHITECTURE.md +++ b/docs/runtime/ARCHITECTURE.md @@ -1,75 +1,116 @@ # GenClass Runtime: architecture -GenClass Runtime is a client-side intelligence layer. It runs inside the page, watches what the application does, -and uses a small local model to decide, in real time, whether something is going wrong and whether to step in. -This document explains how the parts fit together and why they are built this way. The binding spec is -[CONTRACT.md](CONTRACT.md) and the API is in [API.md](API.md). +GenClass Runtime (`@genclass/runtime`) is a browser library. It watches what a web app does (requests, pushed +messages, user actions, timers, state writes, errors) and, for the few situations that look risky, asks a small +on-device model whether something is going wrong and what to do. This page explains how the parts fit together and +why they are built this way, for `@genclass/runtime` on `main` (version string 0.3.0-beta.0, plus the changes the +CHANGELOG lists under 0.3.0-beta.1, unreleased; npm `latest` is 0.2.2, `beta` is 0.3.0-beta.0) with model +`@genclass/runtime-model` 0.2.0. Items marked 0.3.0 are in the 0.3.0 line (0.3.0-beta.0 and later) and not in +0.2.x. How to use it: [API.md](API.md). Binding rules: [CONTRACT.md](CONTRACT.md). Every measured number: +[RESULTS.md](RESULTS.md). Terms: the [glossary](README.md#glossary). ## Design principles 1. **No hardcoded bugs.** The runtime contains no rule like "if a stale response arrives, drop it". It computes - generic facts and asks the model. All runtimes compute the same facts: happens-before order, versions, - repetition, failure streaks, latency against learned baselines, learned invariants and transition profiles. - The diagnosis and the action come from the model. + generic facts and asks the model. Every app gets the same kinds of facts: happens-before order, versions, + repetition, failure streaks, latency against learned baselines, learned relations between fields and learned + transition profiles. The diagnosis and the action come from the model. The measured visual checks are the one + documented exception, and they stay outside the model ([What runs outside the + model?](#what-runs-outside-the-model)). 2. **Precision first.** A false intervention costs more than a missed bug. The default `observe` mode only - reports. The opt-in `guard` mode takes only minimal actions, and only when the model is very confident and its - own diagnosis says something is wrong. `heal` is experimental. + reports. The opt-in `guard` mode takes only minimal actions, and only when the model's gate clears and its own + diagnosis says something is wrong. `heal` is experimental. 3. **Observable.** Every detection and intervention is reported in plain English with its evidence, can be - explained after the fact, and can be undone where possible. A kill switch rules GenClass out in one step. + explained after the fact, and can be undone where possible. A kill switch (`?genclass=off`) rules GenClass out + in one step. 4. **Cheap by default.** Facts are cheap and always on. The model runs only for salient situations, off the main - thread, sized to the device. + thread, with a situation sized to the device. 5. **Train what you run.** The training data comes from the real runtime code, so the model sees exactly the text it will see in production. -## Data flow + + +## How does data flow through the runtime? ``` - app code ──► observers ──► trace (events, ops, causality) ──► stores (versions, holds) + app code ──► observers ──► trace (events, ops, causality) ──► stores (field versions) │ │ ▼ ▼ - learned baselines, invariants, transition profiles + learned baselines, cadence, relations between fields, transition profiles │ - trigger (write about to apply, request about to be sent, failure, stall, - broken invariant, unusual transition, error, developer question) + trigger (a response or pushed message about to reach the app, a state write, a request about + to be sent, a failure, a stall, a broken relation, an unusual transition, an error, + a developer question) │ ▼ - facts ──► triage ──(salient)──► situation + questions + facts ──► triage ──(salient)──► situation + standing questions │ │ (not salient) ▼ - │ model host (Web Worker, ONNX on WebGPU/WASM) + │ model host (Web Worker, ONNX on WebGPU or WASM) ▼ │ passive action ▼ answers: diagnosis + action probabilities │ ▼ - policy gate (mode, tier, confidence, diagnosis, rate) + policy gate (protection, mode, allow/deny, diagnosis, model gate, limits, veto) │ ▼ - action (or passive) ──► report / explain / undo / events + action (or passive) ──► report / explain / undo / audit / events ``` ### Observers and the trace (`src/observe`, `src/trace`) -- **Instrumented sources:** `fetch`, `XMLHttpRequest`, `WebSocket`, DOM user events (capture phase, trusted - events only), errors and unhandled rejections, history navigation, storage, and long tasks. +- **Instrumented sources:** `fetch`, `XMLHttpRequest`, `WebSocket`, `EventSource`, the timers, DOM user events + (capture phase, trusted events only unless `observe.untrustedEvents` is set), errors and unhandled rejections, + history navigation, storage (key names only) and long tasks. - **Operations:** every asynchronous operation becomes an *op* with a signature (`GET /api/items/:id`), a cause and a root. User actions are instantaneous ops, so they can be causes. - **Causality across `await`:** an "ambient op" is set during user handlers and `rt.op`, re-established when a - wrapped request or its body settles, and cleared after the task. State writes record the ambient op as their - cause. This is best effort, and it is enough to say "this write comes from the response to the request the user - started by typing *rea*". - -### Stores and the mutation pipeline (`src/state`) - -- **Field versions:** state registered through atoms, `guard` or the Redux/Zustand/React adapters gets per-field - versions with writer history. -- **Proposals and holds:** each write is first a proposal. A salient proposal from asynchronous code is *held* - until the model answers or the hold budget runs out, then applied, dropped or deferred. Writes made - synchronously by a user handler are never held. -- **Invariant miner:** at settled points it learns relations that keep holding, such as `total == sum(price·qty)`, - `count == len(items)` and membership. A learned relation that breaks raises an `inconsistency`. -- **Transition profiles:** these learn which fields each operation usually writes. A deviation such as "wrote only - `cart.items`, unlike the last 317 times" raises a `transition`. + wrapped request or its body settles and inside timers scheduled during an operation, and cleared after the task. + State writes record the ambient op as their cause. This is best effort, and it is enough to say "this write + comes from the response to the request the user started by typing *rea*". + + + +### Where are decisions made? The network boundary and the stores (`src/observe`, `src/state`) + +- **Writes apply at once by default** (`policy.holdWrites: false`). `set(x); get()` returns `x`, and writes to one + store apply in the order they were made. +- **Decisions happen at the network boundary** (the `delivery` trigger). GenClass predicts which fields a response + (fetch, XHR) or a WebSocket or EventSource message will write, from what the same operation wrote before, and + decides only when it would overwrite newer data, put back the value of an unconfirmed local change, or replace + text the user typed meanwhile. In guard and heal, when an action is permitted and the model can answer within the + hold budget, the delivery waits for the model, which may deliver it, drop the writes it makes over newer data + (`discard`), or hold it until related requests finish and decide again (`defer`, at most twice). Holding only adds + latency, at most the hold budget, and messages of one socket or stream keep their order. In observe, and whenever + no hold is possible (no permitted action, the model not ready or not expected to answer within the hold budget), + the delivery is released at once, exactly as without GenClass, and decided in the background (`runDelivery` in + `runtime.ts`). +- **Late revert.** A write that looks wrong on its own (the `mutation` trigger, not covered by a delivery decision) + is decided in the background. If the model answers `discard` and the gate clears (guard or heal), the write is + reverted only within 800 ms of applying (`LATE_REVERT_MS` in `runtime.ts`), if nothing has overwritten it and the + same operation chain has written nothing else since. A late revert is reported as such and can be undone. +- **Why not hold store writes?** The first design (v1) held salient writes from asynchronous code until the model + answered. With a model that always chose the passive action, it still changed 4 of 198 clean real-app runs, the + React/Redux RealWorld app never rendered its home feed, and the demos showed card jump-backs and overwritten + keystrokes. Deciding at the network boundary (v2) changed 0 of 396 clean runs (66 real apps × 6 seeds, runtime + 0.1.0-alpha.1, compared with observe mode and not re-run since; with network chaos 3 of 198 runs changed; + [RESULTS.md section 4](RESULTS.md#4-never-make-a-correct-app-worse-runtime-design-comparison)). The later + compatibility matrix, which compares every mode with no GenClass, found 0 bugs introduced in 3,510 runs + ([section 16](RESULTS.md#16-compatibility-matrix-summary)). Holding writes is still available as an opt-in, + `policy.holdWrites: true`; even then, writes made synchronously by a user handler apply at once unless + `policy.holdUserWrites: true` (`src/state/hub.ts`). +- **Field versions:** state registered through `atom`, `guard` or the Redux, Zustand and React adapters, or found + by automatic state discovery, gets per-field versions with writer history. Discovered React and Zustand state is + observed only: GenClass never holds, drops, reverts or rolls back its writes. +- **Relations between fields** (`src/state/invariants.ts`): at settled points the runtime learns relations that + keep holding, such as `total == sum(price·qty)`, `count == len(items)` and membership. A learned relation that + breaks raises an `inconsistency`. +- **Transition profiles** (`src/learn/profiles.ts`): these learn which fields each operation usually writes. A + deviation such as "wrote only `cart.items`, unlike the last 317 times" raises a `transition`. +- **Baselines and cadence** (`src/learn/baselines.ts`, `src/learn/cadence.ts`): latency, error rate and failure + streaks per request signature, and whether a signature runs on a schedule (polling, debounced saves), so the + model can tell whether a stale value will be replaced by itself soon. ### Situations and triage (`src/situation`) @@ -77,13 +118,16 @@ This document explains how the parts fit together and why they are built this wa `stats`, plus two standing questions: - `diagnosis`: expected, stale, conflict, duplicate, inconsistent, failing, transient, slow, overload or unusual; - `action`: one option per applicable generic action, with its description. + The format is frozen under the tag `situation-v2.3`: any change to it means new training data and a new model. - **Facts:** short sentences with explicit relations and numbers, for example "written once by other operations since this write's cause started (v0 → v1), last 0.69 s ago by GET …?q=reac, which started 0.09 s after it, from a later user action". - **Triage:** the model is asked only when some fact is non-neutral (concurrency, repetition, failure, an anomaly - against a baseline, a broken relation, an error). Plain traffic never reaches it. -- **Device-sized budget:** 3,200 characters on WebGPU, 2,000 with WASM threads, 1,000 on single-thread WASM. Small - budgets also use compact questions (bare labels). + against a baseline, a broken relation, an error). Plain traffic never reaches it. Facts only decide whether to + ask; no fact selects a diagnosis or an action. +- **Device-sized budget:** 2,400 characters (about 1,000 tokens) on WebGPU; on WASM from 1,000 characters with one + thread to 2,000 with four (crossOriginIsolated pages). At 1,400 characters or less the questions are compact + (bare labels and action names). ### Decisions and actions (`src/decide`, `src/runtime.ts`) @@ -91,56 +135,133 @@ This document explains how the parts fit together and why they are built this wa - Guard tier: `discard`, `defer`, `coalesce`, `delay`. - Heal tier: `block`, `serve_cached`, `retry`, `hedge`, `rollback`, `resync`. - Passive: `apply`, `send`, `deliver`, `wait`, `ignore`. - - Plugins can add their own; the model reads each action's description. -- **Gate:** let *A* be the permitted non-passive actions. GenClass acts only if three things hold: - - the model's summed probability over *A* reaches the tier threshold (guard 0.9, heal 0.8); - - its top diagnosis is not `expected`; - - the decision is within the rate limit and the hold budget. + - Plugins can add their own; the model reads each action's description. An action is offered only when its + preconditions hold (for example, `retry` only when repeating the request is safe by HTTP semantics or an + idempotency key). +- **Gate.** Let *A* be the permitted non-passive actions: the applicable ones the mode allows (observe: none; + guard: guard tier; heal: both), minus `policy.deny` (only `policy.allow` when it is set). A protected or + cross-origin subject, or one whose operation started under an `off` or `observe` route rule, has none + (`blockOf` in `runtime.ts`). GenClass runs the most probable action in *A* only when all of these hold: + - the model's gate clears. Model 0.2.0 ships a gain gate: the estimated gain of the best permitted action over + doing nothing, in cost units, must be above a margin per tier and trigger, taken from the `cautious`, + `balanced` (default) or `eager` profile in the model's `meta.json`. A model without a gate of its own uses a + summed probability over *A* of 0.9 (guard) or 0.8 (heal) at `balanced`, 0.05 more at `cautious` and 0.05 less + at `eager`. The app's `policy.thresholds` override either, read in the gate kind the model ships; + - the model's top diagnosis is not `expected`; + - the decision is within the action limits (by default 60 a minute, 10 per subject a minute, 200 per session) + and, for a held subject, within the hold budget; + - the app's `onBeforeAction` hook, if any, does not veto it. - The chosen action is the most likely action in *A*. -- **Late revert:** a decision that arrives after a held write already applied can revert exactly that write. That - is allowed only within 2 s, if nothing has touched the write since, and if no other write from the same chain - happened. + Otherwise the passive action runs and the decision records why. The hold budget is 1.5 times the model's recent + median latency, between 150 and 800 ms (`policy.holdBudgetMs: "auto"`). +- **Breaker.** After 2 undos, or 3 errors that follow actions, the session drops to `observe`. +- **Detections.** A decision whose top diagnosis is not `expected` is reported when its probability reaches the + profile's report gate (likely), or, in observe from 0.3.0, the `eager` profile's lower report value (possible). + The detection tiers are reporting only: no action gate reads them. ### Model host (`src/model`) - **Engine:** a TypeScript port of the GenClass engine (byte-level BPE tokenizer, packer, calibration). It is bit-exact with the Python packer. -- **Inference:** runs in a module Web Worker with an inline fallback. WebGPU (fp16 when the GPU has `shader-f16`, - int8 otherwise) is preferred; WASM is the fallback. The WASM-only ONNX Runtime build is used when WebGPU is - unavailable. -- **Loading:** the model loads at idle and is verified with sha256 and cached in Cache Storage. While it loads, - every decision fails open at once. +- **Inference:** runs in a module Web Worker, with an inline fallback on the main thread unless + `model.inlineFallback: false` (0.3.0; 0.2.x ignores `false`). The load order is WebGPU with the fp16 file when + the GPU has `shader-f16`, then WebGPU with the q8 file, then WASM with the q8 file. The WASM-only ONNX Runtime + build is used when WebGPU is unavailable. +- **Loading:** by default the model loads when the browser is idle after the page's `load` event, from a pinned + npm version on jsDelivr (`@genclass/runtime-model@0.2.0`), is verified with sha256 and is cached in Cache Storage. + It is not loaded on devices reporting 2 GB of memory or less, nor on Save-Data connections (`model.loadIf`; + 0.3.0; 0.2.x loads lazily on Save-Data and has no memory floor). While it loads, guard and heal fail open at once + (the passive action runs). In observe (0.3.0), where the possible tier is in force (by default; not with `detect: "likely"`, nor with `policy.thresholds.report` unless + `detect: "possible"` is set), up to 32 salient situations wait and are decided once the model is ready, reported + at the possible tier at most. Otherwise observe skips them, as in 0.2.x. + +## What runs outside the model? + +Three layers are deterministic or counting layers, separate from the model. They never feed situations, facts, the +gate or the model (the visual checks have one indirect path, below), and their code is in lazy chunks that load +only when the layer is on. All three keep some code in the first load of `/auto`: the visual checks a 550 B stub and +analytics a 288 B stub (the script-tag build loads their chunks by URL), and flow expectations 3.22 KB when they were +added for 0.3.0-beta.0 (102.94 to 106.16 KB at `fcefedb`: their hooks, the console line, learned-state persistence +and the overhead of splitting the util code they share with their lazy chunk into a 3.05 KB chunk of its own; the +console text later moved into the lazy chunk). The script-tag build carries the flow layer inline +([RESULTS.md section 10](RESULTS.md#10-release-030-beta0-first-load-and-telemetry-volume-ci), +[section 11](RESULTS.md#11-observe-mode-detection-in-030-the-possible-tier-and-deferred-situations) and +[section 15](RESULTS.md#15-interaction-analytics-first-load-cost-train-vm-2026-10-18-branch-featinteraction-analytics)). + +- **Measured visual checks** (`src/visual/`, [VISUAL.md](VISUAL.md)): measurements of visual and CSS bugs that + never throw (unreadable text, broken images, a stuck scroll lock, and others) with documented thresholds, + reported as kind `"check"`, never as a model finding. Report-only by default, and on by default only on local + development URLs (http(s) on `localhost`, `127.0.0.1`, `[::1]` or `*.localhost` with a port, and `*.test` names; + `devHost` in `src/visual/start.ts`) and wherever the devtools overlay mounts (`visual: "local"`). Their three + runtime fixes are opt-in, run only in guard or heal, are re-measured and undone if they do not clear the + threshold. One indirect path remains: a page read that makes the browser lay out a very heavy page can take 50 ms + or more, and the runtime records such a long task as a `perf` event the model can read (VISUAL.md section 1). +- **Flow expectations** (opt-in, `learn: { expectations: true }`, [EXPECTATIONS.md](EXPECTATIONS.md)): learned + counts of what each user action usually causes (a request, a write, a navigation), and a report when a step did + not happen. The model is not consulted, and nothing is held, retried or changed in any mode. +- **Interaction analytics** (opt-in, `analytics: true` with the app's token, 0.3.0-beta.1, not on npm yet; + [ANALYTICS.md](ANALYTICS.md)): page views, engaged time, scroll depth and clicks per element key, counted for the + app's dashboard with their own passive listeners, outside the event log. + +Reports leave the runtime through the console, `rt.on(...)` events, the audit trail, sinks the app configures and +the anonymous telemetry ([TELEMETRY.md](../../packages/runtime/TELEMETRY.md)). None of these outputs feeds a +decision. -## The model and how it is trained + -GenClass is a Jev-style typed-decision encoder: one forward pass reads the state and answers several typed +## What is the model, and how is it trained? + +The model is a Jev-style typed-decision encoder: one forward pass reads the state and answers several typed questions (choice, yes/no, score). Each option attends only to the state and its own question, so option order -cannot change the answer. The runtime model is a specialist fine-tuned from the GenClass base. There are two -sizes, R17 (17M parameters) and R32 (32M), both with a pruned 16k vocabulary and int8 weights. - -Training data (`sim/`, `training/`): - -1. **Simulator.** Thousands of random apps across 55 domains are built from 15 feature combinators: search, - lists, forms, carts, boards, chat, autosave, polling, uploads and more. They mix correct guards with realistic - defects, and run real `async` code against the **real runtime** in a deterministic virtual world (clock, - network with latency, failures and outages, push channels, users who type, double-click and change their - minds). -2. **Labels from outcomes, not rules.** At a sampled decision point, the sim re-runs the scenario with each - applicable action forced there, scored over several sampled futures. The cost of each run is how far the app - drifts from what the user intended, which is the same session in an ideal serial, zero-latency, exactly-once - world. It also counts persistent server damage, user-visible errors, wasted requests and added latency. The - action label is a soft distribution over those costs: ties go to doing nothing, and labels are sharp only when - outcomes agree across futures. The diagnosis label comes from the simulator's knowledge of user intents. -3. **Curriculum.** Programmatic error-detection-and-fixing exercises: ordering, causality, versions, duplicates, - streaks, baselines, invariants, and HTTP/JS error semantics. -4. **Held out for testing.** Domains and program families that never appear in training. The demos are written - independently of the simulator and act as an external test. - -## Evaluation - -- **Simulator test split:** action and diagnosis accuracy, plus the false-intervention rate and precision of the - guard and heal gates, per trigger and per device budget. -- **Demos:** six real small apps covering typeahead, autosave, checkout, a flaky dashboard, a live kanban board, - and real-time decisions. Each is driven by Playwright under chaos and on clean runs with GenClass off, in guard - and in heal. Reports cover bug rate, false interventions on clean runs, and added latency. +cannot change the answer. + +One model ships: R17 (`genclass-runtime-r17`, in `@genclass/runtime-model` 0.2.0). It is initialised from +`jhu-clsp/ettin-encoder-17m` and, after vocabulary pruning to 16,364 tokens, has 8.8M parameters (8.1M encoder and +0.7M heads). The q8 file is 10.16 MB and the fp16 file 13.57 MB +([model card](../../packages/runtime-model/MODEL_CARD.md)). R32 was compared and not shipped: R17 matched its +accuracy at about a third of the latency and under half the size (q8 9.58 MB against 22.5 MB, round-1 exports; +[RESULTS.md section 2](RESULTS.md#2-r17-vs-r32-choosing-the-shipping-model)). + +Training data (`sim/`, `realapps/`, `training/`): + +1. **Simulator** (`sim/`). Random apps across 115 domains are built from 46 feature combinators: search, lists, + forms, carts, boards, chat, autosave, polling, uploads, offline queues, websocket reconnects, payments, + multi-tab sync and more ([sim/README.md](../../sim/README.md)). They mix correct guards with realistic defects, + and run real `async` code against the **real runtime** in a deterministic virtual world (clock, network with + latency, failures and outages, push channels, users who type, double-click and change their minds). +2. **Real apps** (`realapps/`). 158 apps (144 written for the corpus and 14 open-source RealWorld front-ends) run + in headless Chromium with the real runtime ([realapps/README.md](../../realapps/README.md)). They give training + rows, a held-out evaluation set and the never-worse sweep. +3. **Labels from outcomes, not rules.** At a sampled decision point, the scenario is re-run with each applicable + action forced there, scored over several sampled futures. The cost of each run is how far the app drifts from + what the user intended, which is the same session in an ideal serial, zero-latency, exactly-once world. It also + counts persistent server damage, user-visible errors, wasted requests and added latency. For model 0.2.0 the + action label is `softmax(gain / τ)` with τ = 1, where gain is the expected cost each action saves over doing + nothing. The diagnosis label comes from the simulator's knowledge of user intents. +4. **On-policy rows.** 1,248,131 situations from an earlier model (`r17-v2a`) acting inside the simulator, + including 448k at its shipping gate and 672k at an exploratory gate of 0.5, labelled the same way. Round b's + rows (from `r17-v2b`) were not trained on; the shipping-gate rows of both rounds were used to fit and test the + gates ([model card](../../packages/runtime-model/MODEL_CARD.md), "Training data and licence"). +5. **Curriculum.** Programmatic error-detection-and-fixing exercises: ordering, causality, versions, duplicates, + streaks, baselines, invariants, and HTTP and JavaScript error semantics. +6. **Held out for testing.** Domains, feature combinators, program families and real apps that never appear in + training. The demos are written independently of the simulator and act as an external test. + + + +## How is it evaluated? + +- **Simulator test split:** action and diagnosis accuracy, plus the false-intervention rate (FIR), harm and recall + of the guard and heal gates, per trigger and per profile. +- **Real-app evaluation set:** detections and actions on held-out real apps, with the FIR on clean and benign + traffic. For model 0.2.0 the gate fit held each profile's false detections under a limit, and every profile met + its limits on held-out test; the exact 0.2.0 false-detection rates are not published + ([model card](../../packages/runtime-model/MODEL_CARD.md), "Observe-mode false detections"). +- **Never worse:** real apps run with an always-passive model in each runtime design, counting clean runs whose + outcome changed (RESULTS.md section 4). +- **Demos:** six small apps covering typeahead, autosave, checkout, a flaky dashboard, a live kanban board and + real-time decisions. Each is driven by Playwright under chaos and on clean runs with GenClass off and in each + mode. Reports cover bug rate, false interventions on clean runs and added latency. +- **Compatibility:** the framework matrix in [compat/RESULTS.md](../../compat/RESULTS.md). + +Every result, with its FIR next to its recall, is in [RESULTS.md](RESULTS.md). diff --git a/docs/runtime/DASHBOARD.md b/docs/runtime/DASHBOARD.md new file mode 100644 index 00000000..9914c508 --- /dev/null +++ b/docs/runtime/DASHBOARD.md @@ -0,0 +1,152 @@ +# Your GenClass dashboard and the account commands + +How an app's diagnostics reach a dashboard at genclass.dev: the app token (project key), the private dashboard link, +GenClass accounts, the CLI's `login`, `connect`, `whoami`, `logout` and `dashboard`, and the `.genclass.local` file. For +`@genclass/runtime` 0.3.0 (`0.3.0-beta.0` on the npm `beta` dist-tag); `0.2.x` ignores a token and has none of these +commands. What is sent is in [TELEMETRY.md](../../packages/runtime/TELEMETRY.md); the privacy policy for the +diagnostics and for accounts (what is collected and stored, how long it is kept, your rights) is +[PRIVACY.md](../../PRIVACY.md). + +## What is the dashboard? + +Each web app gets a **project key** (token: `gc_` + 22 letters and digits) that groups its diagnostics on a +dashboard at genclass.dev. The key is public: it ships in your page and can only add telemetry to its app. Keep the +dashboards in a GenClass account, or use one private link per app without an account. The dashboard is free for every +app, with 90 days of history (1 year on Enterprise), and so is your own classifier, which a person from GenClass trains +on your app's traffic, bug reports or labelled examples once you ask ([pricing](https://genclass.dev/pricing)). + +The dashboard reads the same anonymous diagnostics the runtime sends by default; it gets nothing while telemetry is +off ([Why does my dashboard show no data?](#why-does-my-dashboard-show-no-data)). + +## What is the difference between the token and the dashboard link? + +| | what | who may see it | +|---|---|---| +| token | `gc_` + 22 letters and digits, e.g. `gc_4fQ9...` | public: it ships in your page. It only lets your app's diagnostics be grouped together. | +| dashboard link | `https://genclass.dev/dashboard/` | **private**: anyone with the link sees your app's stats, and it is the only way to open them | +| CLI session | `gcs_...`, saved in `~/.config/genclass/credentials.json` | **secret**: `logout` ends it at genclass.dev and deletes the file; you can also revoke it from your account page | + +## How do I get a dashboard without an account? + +Run `npx @genclass/runtime@beta init` while not signed in. After you approve its diff it creates both the token and +the link, writes the token into what it adds, prints the link and saves it in `.genclass.local` +([INSTALL.md](INSTALL.md#how-does-init-create-a-token-and-a-dashboard)). Or click "Get a token" at +[genclass.dev/start](https://genclass.dev/start). A link can be added to an account later (`connect`, or "Add an +existing dashboard" on your account page). + +## What is `.genclass.local`? + +The file where `init` saves the token and the private dashboard link, before it writes anything else, so a failed +write cannot lose it. init asks git whether the file is ignored; if not, it adds it to the project's `.gitignore` +(inside a git repository without one there, such as a monorepo package, it creates one with that line). `remove` +leaves the file and that line alone. Never paste the link from it into code, docs, commits or pull requests: anyone +with the link sees the app's stats. `connect` reads it to add that dashboard to your account, and sends the link only +to the host it belongs to. + +## How do I print my dashboard link again? + +```bash +npx @genclass/runtime@beta dashboard # print the link to this app's dashboard (--open: open it in your browser) +``` + +New in `0.3.0-beta.2`. `dashboard` reads the private link in `.genclass.local` without contacting anything; signed +in, it finds the app in your account (one `GET /api/me`), or lists your apps' links. `init`'s notice, `whoami` and +`login` point to the dashboards too. + +## How do I keep my dashboards in an account? + +```bash +npx @genclass/runtime@beta login # sign in in the browser, then connect the app in this folder +npx @genclass/runtime@beta connect # connect an app folder later (or another one) +npx @genclass/runtime@beta whoami # your account and its apps +npx @genclass/runtime@beta logout # end the CLI session +``` + +`login` prints a code and opens `https://genclass.dev/login/device`, where you sign in (Google, your email and +password, or GitHub, plus your passkey if your account has one) and approve that code. It waits until you do (Ctrl-C +stops it and saves nothing), then saves the CLI session in `~/.config/genclass/credentials.json` +(`$XDG_CONFIG_HOME/genclass/`; Windows `%APPDATA%\genclass\`), readable only by you. Inside an app's folder, `login` +then asks whether to connect that app; `connect` does that step alone: + +- pick one of your account's apps, or a new one named after `package.json`; a new app is created only once you + approve the changes; +- it writes the key the way `init` does (`genclass.config.*`, `data-token` on the script tag, or the Astro meta + line), shows the diff and asks first, and sets GenClass up first if `init` has not; +- an app `init` already created without an account (its private link in `.genclass.local`) can be added to your + account instead, keeping its key and its data. + +``` + ✔ Account you@example.com + ✔ App shop-web (a new app in your account, created once you apply the changes) + ... + Your dashboard https://genclass.dev/account/apps/... +``` + +An app's dashboard in your account opens for you when you are signed in at genclass.dev. Until the app's first event +arrives it shows the key and these same two ways to connect it; you can also sign up at genclass.dev, create an app +there, and pass its key yourself ([below](#how-do-i-pass-a-token-myself)). Signed in, `init` first asks whether a +new app goes into your account instead of getting a private link (`--yes`: it does). + +### Account command flags + +| flag | command | what it does | +|---|---|---| +| `--yes`, `-y` | login, connect | no questions: `login` connects this folder's app; `connect` uses your app named like `package.json` (or creates it) and applies the changes | +| `--no-browser` | login | print the link only, do not open a browser (it works from any device) | +| `--no-connect` | login | do not offer to connect the app in this folder | +| `--app ` | login, connect | use this app of your account | +| `--name ` | login, connect | create a new app with this name (default: `package.json`'s name) | +| `--dry-run` | connect | show what would change; it reads your app list, and creates and writes nothing | +| `--no-install` | login, connect | do not run the package manager (when connect sets GenClass up) | +| `--cwd ` | all four | the project directory (default: the current directory) | + +`npx @genclass/runtime@beta login --help` prints the same list. + +## What does the CLI send to genclass.dev? + +Nothing until you run one of these commands (or approve `init`'s changes). `init` creates a dashboard only after you +approve its diff (it sends the `name` from `package.json`). `login` asks genclass.dev for a sign-in code with +`client_name: "genclass-runtime "` (the approval page shows that name and your account page lists the +session under it), and polls until you approve it, then reads your account's apps. `whoami`, `connect` and (signed +in, without a private link here) `dashboard` read your account's apps; creating an app sends its name, and adding an existing dashboard sends its link from +`.genclass.local`. `logout` ends the CLI session. The account requests carry +`user-agent: genclass-runtime-cli/`. + +A CLI session can read your account's apps, create apps, add an app from its private link and end itself; it cannot +read stats, end other sessions or change how you sign in. + +## How do I pass a token myself? + +`GenClass.init({ token })`, `data-token="..."` on the script tag, `` or +`window.GENCLASS_CONFIG = { token }`. A malformed token logs one console warning and is ignored. With `/auto` or the +script tag, GenClass is already running before your code, so set the token where GenClass starts (the +`genclass.config.ts` / `.js` that `init` writes, the meta tag or the script tag), not in a later `GenClass.init()`. + +## What does the dashboard show? + +Decisions, detections by diagnosis, actions taken, top routes and protected functions, model load, backend (WebGPU or +WASM) and latency, runtime and model versions, and recent detections. With `analytics: true` (new in `0.3.0-beta.1`; +`0.3.0-beta.0` does not have it; off by default), also how visitors use the app: page +views, time on page and scroll depth per page, the most clicked elements, and rage and dead clicks +([TELEMETRY.md](../../packages/runtime/TELEMETRY.md#interaction-analytics)). Put `data-genclass-name="pricing-cta"` on +an element to give it a readable name there. Dashboard data is kept for 90 days (1 year on Enterprise). + +## Why does my dashboard show no data? + +The dashboard needs telemetry on (the default with `GenClass.init()` in a browser) on a deployed host. Local hosts +(`localhost`, `*.local`, `*.test`, private networks) do not send by default, so a page there does not feed the +dashboard (the console says so once when a token is set). `telemetry: true` or any telemetry options object (for +example `telemetry: { decisionSample: 1 }`) is an explicit opt-in that sends from local hosts too, token included. +With telemetry off for any other reason, a token does nothing (with `debug: true` the console says so once). Visitors +who opt out (`?genclass=no-telemetry`, `localStorage["genclass.telemetry"] = "off"`) or whose browser sends Global +Privacy Control are not counted. + +With a token, each telemetry batch carries it and decisions about `protect()`ed functions carry their `fn` name; +nothing else changes about what is sent. Without a token neither is sent. + +## Which version do I need? + +The token option, `scope`, `protect()` and the `login`, `connect`, `whoami` and `logout` commands are in `0.3.0` +(first as `0.3.0-beta.0`, see the [CHANGELOG](../../packages/runtime/CHANGELOG.md)); `dashboard` from +`0.3.0-beta.2`. A key needs `@genclass/runtime` `0.3.0-beta.0` or later in your app. A dashboard opens on every plan, +whichever runtime version sends to it. diff --git a/docs/runtime/EXPECTATIONS.md b/docs/runtime/EXPECTATIONS.md index c6806a71..5c75e6e0 100644 --- a/docs/runtime/EXPECTATIONS.md +++ b/docs/runtime/EXPECTATIONS.md @@ -1,10 +1,17 @@ -# Flow expectations (v1: learn and report; no actions) +# GenClass flow expectations (v1: learn and report; no actions) -Status: in `@genclass/runtime` 0.3.0 as a deterministic, reporting-only layer, **opt-in** +GenClass flow expectations learn, per route and user action, what usually follows (a request, a follow-up request, a +state write, a navigation, a response in time) and report when it does not happen, such as an "Add to cart" click +that usually sends a request and this time sent nothing. They only report, use no model, and are opt-in with +`learn: { expectations: true }` in `@genclass/runtime` 0.3.0-beta.0 and later; their false-report rate is not measured +yet. + +Status: in 0.3.0-beta.0 (npm `beta`) as a deterministic, reporting-only layer, **opt-in** (`GenClass.init({ learn: { expectations: true } })`, default false) until its false-report rate on correct apps is -measured (the acceptance gate in docs/runtime/RESULTS.md, "Flow expectations"). Off, nothing is learned, stored or -reported and `src/learn/flows.ts` is never loaded; on, the runtime imports it with a dynamic import after the first -idle (a lazy chunk: no first-load bytes; inlined in the script-tag file), and activity before it arrives is not learned. +measured (the acceptance gate in [RESULTS.md section 5](RESULTS.md#5-demos), "Flow expectations"). Off, nothing is +learned, stored or reported and `src/learn/flows.ts` is never loaded; on, the runtime imports it with a dynamic import +on the next task after init (a 0 ms timer on the injected clock, not an idle callback; a lazy chunk: no first-load +bytes; inlined in the script-tag file), and activity before it arrives is not learned. Code: `packages/runtime/src/learn/flows.ts`, wired in `src/runtime.ts`; tests in `test/expectations.test.ts`. Nothing here is model-visible (CONTRACT §0 rule 1 and AGENTS.md rule 3 are untouched: no new fact, trigger, question or situation text), and nothing here acts in any mode. The one learned state the model does read, the diff --git a/docs/runtime/INSTALL.md b/docs/runtime/INSTALL.md new file mode 100644 index 00000000..c71c1fa0 --- /dev/null +++ b/docs/runtime/INSTALL.md @@ -0,0 +1,258 @@ +# Installing GenClass Runtime + +How to add `@genclass/runtime` to a web app and take it out again: the `init` and `remove` commands with every +flag, the app token and dashboard that `init` creates, the zero-code options, the import, the script tag and its +attributes, `GenClass.init()`, and the projects `init` was tested on. For `@genclass/runtime` 0.3.0 (`0.3.0-beta.0` on the npm +`beta` dist-tag). The short version is in the [package README](../../packages/runtime/README.md#install). + +## Which version should I install? + +While `0.3.0` is in beta, every npm and npx command here names the `@beta` dist-tag. Without it you get the stable +`0.2.x` (`0.2.2` was npm `latest` on 2026-10-22; `npm view @genclass/runtime dist-tags` shows them now), whose +runtime ignores a token and whose CLI has no `login`, `connect`, `whoami`, `logout`, `dashboard`, `--token` or `--sink`. Both +lines load model `@genclass/runtime-model@0.2.0` by default. + +Pin the script tag's version (`https://cdn.jsdelivr.net/npm/@genclass/runtime@0.3.0-beta.0`): the unversioned URL +serves the `latest` dist-tag (`0.2.2` on 2026-10-22) until 0.3.0 is released. + +**Older versions.** `0.1.0-beta.3` has these problems, fixed in `0.2.0`: app builds carried 41 MB of unused ONNX +Runtime wasm, a CSP-blocked model failed silently, `init` could not turn telemetry off, and `status.scope.mode` read +"heal" in observe mode. `0.1.0-beta.2` is `0.1.0-beta.3` without telemetry. `0.1.0-beta.1` has model 0.2.0 and the +new options but defaults to `guard`, and lacks the observe-delivery, redaction and install fixes of `0.1.0-beta.0` +(all back in `0.1.0-beta.2`). `0.1.0-beta.0` ships model 0.1.0. `0.1.0-alpha.0` (v1 runtime: guard by default, holds +store writes, `NaN` crash) and `0.1.0-alpha.1` (observe by default, but no model was published for it) predate the +model; do not use them. Anonymous diagnostics are on by default from `0.1.0-beta.3`; `0.1.0-beta.3` to `0.2.1` also +send them from `localhost` and other local hosts, and `0.1.0-beta.3` to `0.2.0` send the redacted situation text by +default ([CHANGELOG](../../packages/runtime/CHANGELOG.md)). + +## How do I install it with one command? + +`init` finds your framework and package manager, installs the package, adds one import as the first line of your +entry file (plus a line that loads the devtools overlay in development only), and shows you the diff before writing +anything. Running it again changes nothing; running it with another `--mode` switches the import it added. All of +the ways below start the same runtime, in observe mode unless you choose otherwise. + +```bash +npx @genclass/runtime@beta init # observe; shows the diff, asks, then writes +npx @genclass/runtime@beta init --mode guard # let GenClass act (also: --mode heal); run again later to switch +npx @genclass/runtime@beta init --yes # no questions +npx @genclass/runtime@beta init --dry-run # show the diff, write nothing +npx @genclass/runtime@beta remove # undo exactly what init added +``` + +`init` also says that telemetry is on by default and how to turn it off, and when your app sets a +Content-Security-Policy (`next.config` headers, a proxy or middleware, a ``, helmet, `vercel.json`, +`_headers`, ...) it prints the steps to self-host the model +([Content-Security-Policy and self-hosting](../../packages/runtime/README.md#content-security-policy-and-self-hosting)). + +### Every `init` and `remove` flag + +| flag | command | what it does | +|---|---|---| +| `--mode observe\|guard\|heal` | init | the mode of the import init adds (default `observe`: reports only; never holds, delays or changes a request, response or state write) | +| `--token ` | init | use this app token; genclass.dev is not contacted | +| `--no-token` | init | no token and no dashboard; genclass.dev is not contacted (the package manager still runs unless `--no-install`; add a token later at [genclass.dev/start](https://genclass.dev/start)) | +| `--no-telemetry` | init | turn off GenClass's anonymous diagnostics; implies `--no-token` (a dashboard only gets data while telemetry is on) | +| `--telemetry` | init | undo `--no-telemetry`: back to the default (on, but never from local hosts); init takes the opt-out out and writes no telemetry option of its own | +| `--model-url ` | init | use a self-hosted model directory (made with `fetch-model`), with ONNX Runtime from `ort/` | +| `--sink ` | init | also send findings to your own side: `console-json`, `console-json:noevidence` or `webhook:`; repeat the flag or comma-separate ([SINKS.md](SINKS.md)) | +| `--yes`, `-y` | both | apply without asking | +| `--dry-run` | both | show what would change; write nothing, send nothing | +| `--no-install` | init | do not run the package manager | +| `--no-devtools` | init | do not add the dev-only devtools overlay | +| `--cwd ` | both | the project directory (default: the current directory) | +| `--from ` | init | install `@genclass/runtime` from this spec (a version, tag or tarball path) | +| `--cdn ` | init | plain HTML: the script URL (default: jsDelivr, this version, with an SRI hash; this version's unpkg URL gets one too; other URLs get no hash) | +| `--no-sri` | init | plain HTML: no integrity attribute | +| `--strategy instrumentation\|layout\|pages` | init | Next.js only: where the import goes. By default `instrumentation-client` on Next.js 15.3 and later, else `layout` (App Router) or `pages` (Pages Router) | +| `--keep-package` | remove | leave `@genclass/runtime` installed (by default `remove` uninstalls it if nothing else imports it) | + +`npx @genclass/runtime@beta init --help` prints the same list except `--strategy` (Next.js only). The account +commands (`login`, `connect`, `whoami`, `logout`, `dashboard`) and their flags are in [DASHBOARD.md](DASHBOARD.md); `fetch-model` +and `info` are under [Content-Security-Policy and self-hosting](../../packages/runtime/README.md#content-security-policy-and-self-hosting). + +### What does `init` edit, and what does `remove` delete? + +- **What `init` edits.** It only edits browser apps. It stops and says why for a Node server, a library (a UI + framework as a peer dependency, or an `exports`, `module`, `types` or `bin` field), or an entry file that imports + Node-only modules. +- **What `remove` deletes.** Only what `init` wrote (the lines and blocks it marked), including lines a formatter + rewrapped. If you edited a marked line or block, it changes nothing and lists where. +- **Read the diff.** Always read the diff `init` and `remove` show before you confirm. + +### Options without editing code + +These also work on a project `init` already set up; `remove` takes them out too: + +```bash +npx @genclass/runtime@beta init --no-telemetry # anonymous diagnostics off (--telemetry: explicitly on) +npx @genclass/runtime@beta init --model-url /genclass-model/ # a self-hosted model +npx @genclass/runtime@beta init --sink webhook:/genclass # findings to your own endpoint too; also console-json +``` + +They go into a marked `genclass.config.ts` (or `.js`) next to your entry, which sets `window.GENCLASS_CONFIG` and is +imported right before the `/auto` import (an ES module's imports run before its own code, so options cannot be set +above the import in the same file). Any `GenClass.init()` option works in that file. Plain HTML gets +`data-telemetry="off"` (and `data-model`, `data-ort`, `data-sink`) on the script tag, Astro a `` +line. + +## How does `init` create a token and a dashboard? + +New in `0.3.0`; `init` in `0.2.x` creates no token. On a new setup, once you approve the diff (or pass `--yes`), +`init` also creates a project for your app at genclass.dev: it sends `package.json`'s `name` to +`https://genclass.dev/api/projects`, writes the token it gets back into what it adds (the diff shows +`gc_new-token-created-when-you-apply` in its place until then), and prints your private dashboard link: + +``` + Your dashboard https://genclass.dev/dashboard/... + Keep this link private; it is the only way to open your dashboard. + Saved in .genclass.local (added to .gitignore). +``` + +From `0.3.0-beta.1`, `init` also says after the second line that the dashboard is free; from `0.3.0-beta.2` that +reads: + +``` + Free for every app: what GenClass found, on which pages, and what it did, and (with analytics: true) + how visitors use each page. Print this link again any time: npx @genclass/runtime dashboard +``` + +The link is saved in `.genclass.local` (with the token) before init writes anything else, so a failed write cannot +lose it. init asks git whether that file is ignored; if not, it adds it to the project's `.gitignore` (inside a git +repository without one there, such as a monorepo package, it creates one with that line). `remove` leaves the file +and that line alone. Declining, running without a terminal and without `--yes`, and `--dry-run` send nothing and +create nothing. Running `init` again reuses the token (no new project); on a setup made without one it never creates +a project as a side effect: pass `--token`, or sign in and run `connect`. Signed in +(`npx @genclass/runtime@beta login`), `init` first asks whether the new app goes into your GenClass account instead +of getting a private link ([DASHBOARD.md](DASHBOARD.md)). + +```bash +npx @genclass/runtime@beta init --token gc_... # use a token you already have (genclass.dev is not contacted) +npx @genclass/runtime@beta init --no-token # no token; genclass.dev is not contacted +``` + +`--no-telemetry` implies `--no-token` (a dashboard only gets data while telemetry is on). If genclass.dev cannot be +reached, `init` says so, sets GenClass up without a token, and points you to +[genclass.dev/start](https://genclass.dev/start); run `init --token gc_...` later. `--dry-run` sends nothing and +installs nothing. + +## How do I install it by hand? + +**One import** (any bundler; the package is ESM only, there is no `require` build), first in your entry file, so it +is installed before your framework and stores created at import time see the runtime (automatic state discovery is +on): + +```ts +import "@genclass/runtime/auto"; // observe (the default) +import "@genclass/runtime/auto/guard"; // or guard +import "@genclass/runtime/auto/heal"; // or heal (experimental) +import "@genclass/runtime/auto/observe"; // observe, explicitly + +import rt from "@genclass/runtime/auto"; // the same, and the runtime it started +``` + +Next.js 15.3 or later: put it in `instrumentation-client.ts`; on earlier versions `init` imports it from a +`"use client"` component rendered in the root layout (App Router) or from `pages/_app` (Pages Router). Optional page +configuration, read once: +`` inside `` or +`window.GENCLASS_CONFIG = { mode: "guard", devtools: true }` (any `InitOptions`; since `0.3.0` also `token=gc_...` / +`token: "gc_..."`; every key: [below](#which-attributes-does-the-script-tag-take)), set before the import runs. +During SSR or in Node, `/auto` installs nothing and returns an inert runtime. + +**One script tag** (no build step), first in ``: + +```html + +``` + +It exposes `window.GenClass` (since `0.3.0` also `window.GenClass.protect`), discovers the app's state like the +import does, and loads the model worker, ONNX Runtime Web and the overlay on demand from the same version on the CDN. +Its `data-*` attributes configure it ([every attribute](#which-attributes-does-the-script-tag-take)). Keep the +version pinned in production; the plain-HTML path of `init` writes a pinned jsDelivr URL with SRI. + +**`GenClass.init()`**, the explicit setup: + +```bash +npm install @genclass/runtime@beta +``` + +```ts +// first thing in your entry file +import { GenClass } from "@genclass/runtime"; + +const rt = GenClass.init(); // observe; GenClass.init({ mode: "guard" }) to let it act +``` + +`GenClass.init()` does not discover state on its own: register stores +([State it can protect](../../packages/runtime/README.md#state-it-can-protect)), or import +`@genclass/runtime/discover` first and pass `autoState: true`. + +The unscoped `genclass-runtime` alias is deprecated; use `@genclass/runtime`. + +## Which attributes does the script tag take? + +The script tag reads these `data-*` attributes; `` in `` takes +the same keys (with `/auto` too). Later sources win: the meta tag, then the script tag, then +`window.GENCLASS_CONFIG`, then options passed to a manual `GenClass.init()`; the URL switch (`?genclass=...`) and +`localStorage.genclass` still beat all of them. Unknown keys and values are ignored. + +| attribute | meta key | what it sets | +|---|---|---| +| `data-mode` | `mode` | `observe` (the default), `guard` or `heal` | +| `data-devtools` | `devtools` | the overlay: bare or `true` always, `local` only on local hosts (`localhost`, `127.0.0.1`, `[::1]`, `*.localhost`, `*.local`, `*.test`, `file:`), `off`, or a corner (`bottom-right`, `bottom-left`, `top-right`, `top-left`) | +| `data-telemetry="off"` | `telemetry=off` | no anonymous diagnostics (an opt-out only: on is the default) | +| `data-token` | `token` | the app's dashboard token, `gc_...` (`0.3.0`) | +| `data-scope` | `scope` | `app` (the default) or `functions` (`0.3.0`) | +| `data-sink` | `sink` | sinks by name: `console-json`, `console-json:noevidence`, `webhook:`; commas between names on the tag, `+` in the meta tag ([SINKS.md](SINKS.md), `0.3.0`) | +| `data-detect` | `detect` | `likely` or `possible` (`0.3.0`) | +| `data-autostate="off"` | `autostate=off` | no automatic state discovery (an opt-out only) | +| `data-visual` | `visual` | measured visual checks: `on`, `off`, `local`, or `fix` (on, with runtime fixes in guard or heal mode) (`0.3.0`) | +| `data-analytics` | `analytics` | interaction analytics `on` (off by default; they need telemetry and the app's token; `0.3.0-beta.1`) | +| `data-model` | `model` | a self-hosted model directory URL, or `off` for no model | +| `data-ort` | `ort` | the directory of ONNX Runtime Web's wasm files | +| `data-device` | `device` | `auto`, `webgpu` or `wasm` | +| `data-preload` | `preload` | when the model downloads: `eager`, `idle` or `lazy` | +| `data-worker="off"` | `worker=off` | run the model on the main thread instead of a Web Worker | +| `data-report` | `report` | `console` or `silent` | +| `data-debug` | `debug` | console logging of every decision | +| `data-triage` | `triage` | `salient` (the default) or `always` | +| `data-manual` | none | expose `window.GenClass` without starting it; call `GenClass.init(options)` yourself (the page configuration still applies) | +| `data-base` | none | the URL of this package's `dist/` directory, when you self-host the script file | + +A meta tag only takes same-origin URLs for `model`, `ort` and a `webhook:` sink, and only inside ``; put a +cross-origin URL in the script tag's attributes or `window.GENCLASS_CONFIG` instead. The source of this list is +[`src/cdn/config.ts`](../../packages/runtime/src/cdn/config.ts). + +## ESM only, and Jest + +The package ships ES modules (no `require`). Jest users need a `transformIgnorePatterns` entry that lets +`@genclass/runtime` be transformed, for example `["/node_modules/(?!@genclass/runtime)"]`. + +## Which projects was `init` tested on? + +Tested end to end on fresh projects from each framework's own generator (a locally packed pre-release build whose +version string was `0.1.0-alpha.0`, 2026-10-08; the published `0.1.0-alpha.0` has no `init`; model files served +locally; not re-run since): Vite 8 (React with npm and pnpm, Vue, Svelte), Next.js 16.4 (App and Pages Router, plus +the paths for Next < 15.3), Create React App 5, SvelteKit, Astro, Nuxt 4.6, React Router 8 (framework mode), Angular +20 and plain HTML. In all 15 projects: + +- the app built and ran in Chromium after `init` (production build where the project has one); +- the model loaded in a worker and a warm load had no console errors; the overlay appeared only in development + (plain HTML: wherever `data-devtools="local"` sees a local host, so also in its production run on `127.0.0.1`; the + dev server of the Pages Router `--strategy pages` variant was not run); +- `remove` left every file byte-identical to the scaffold (node_modules, lockfiles and build output excluded). + +Remix, Solid, Preact and Next.js before 15.3 are detected but were not scaffolded with their own generators. Those +scaffold runs predate the `init` / `remove` changes since (`--mode`, formatter handling, server and library +detection, app tokens), which are covered by unit tests only. The compatibility matrix below adds the one line by +hand, so it is end-to-end evidence for the runtime, not for `init`. Details: +[test/install/RESULTS.md](../../packages/runtime/test/install/RESULTS.md). + +The [compatibility matrix](../../compat/RESULTS.md) runs the one line in React + Vite, Next.js 16, Vue + Pinia, +SvelteKit 3, Angular 22, Solid and plain HTML, with 15 data layers (TanStack Query, SWR, Redux Toolkit, Zustand, +Apollo, WebSocket and EventSource among them), in every mode, and checks that GenClass never makes those apps worse +(runtime commit `d04eff3`, the 0.1.0-beta.4 code plus the "Model ready" log fix; the matrix has not run 0.2.0's +other changes or anything since; 2026-10-10). Starter templates are in [templates/](../../templates). + +**React Router dev server.** On the very first dev start after `init`, Vite discovers the new imports late, +re-optimizes and reloads the page, logging a few "Outdated Optimize Dep" errors once. Later loads are clean. diff --git a/docs/runtime/OPTIONS-SPEC.md b/docs/runtime/OPTIONS-SPEC.md index 850eac4f..8806de9f 100644 --- a/docs/runtime/OPTIONS-SPEC.md +++ b/docs/runtime/OPTIONS-SPEC.md @@ -1,6 +1,18 @@ -# @genclass/runtime: configuration extensions spec (v0.3 options), final - -Status: ready for implementation. Branch `runtime`, package `packages/runtime`. I read only `types.ts`, `API.md`, README and the CONTRACT sections. Nothing was run. `aggressiveness` is already in flight, so it is not proposed here. Today's documented default is `"balanced"`, and this spec does not change it. +# GenClass Runtime options: design spec (@genclass/runtime) + +This page is the design spec for the configuration options of `@genclass/runtime`: why each option exists and how +the options combine. The reference for each option is [API.md](API.md) and the JSDoc in +`packages/runtime/src/types.ts`; where they differ from this spec, they describe the code. `npm i @genclass/runtime@beta` +installs 0.3.0-beta.4 (npm's `beta` dist-tag, 2026-10-27); plain `npm i @genclass/runtime` installs the stable line, +0.2.2 (`latest`). Items marked 0.3.0 are not in 0.2.x. + +Status: built. Most options here shipped in 0.1.0-beta.1 (the CHANGELOG's "Batch 12, 19 options") and are in every +later release. New in 0.3.0, not in 0.2.x: `detect` (§4.19), `visual` (§4.20), `learn.expectations` (§4.14), +observe's deferred situations (§5.1, §5.3), the `model.loadIf` device-floor default and `inlineFallback: false` +skipping the model (§4.15; 0.2.x defaulted to `{ saveData: "lazy" }` and ignored `inlineFallback: false`), the +sink adapters, sinks by name and the sink `possible` flag (§4.11), and, from 0.3.0-beta.2 (2026-10-25, issue 94), +`rt.summary().requests` (§4.11) and the `request-stalled` check with `visual.checks.stallMs` (§4.20). +`aggressiveness` (0.1.0-beta.1, runtime batch 11) is specified in API.md, not here; its default is `"balanced"`. ## 0. Design rules @@ -9,7 +21,7 @@ Status: ready for implementation. Branch `runtime`, package `packages/runtime`. 3. **One effective-mode function, and every term can only demote:** ``` effectiveMode(op) = state ∈ {disabled} ? off - : min( globalMode // InitOptions.mode / rt.setMode / URL param (URL may only demote unless debug:true) + : min( globalMode // InitOptions.mode / rt.setMode / URL param (only demotes; ?genclass= / localStorage.genclass in GenClass.init may also raise on a development host, src/devhost.ts; debug:true raises nothing) , sample cap // 'observe' if sampled out , breaker cap // breaker.downgradeTo once tripped , route rule.mode ) // first matching routes[] rule @@ -45,14 +57,14 @@ order: off < observe < guard < heal | 11 | `sinks` + `rt.summary()` | telemetry | `[]` | new | | 12 | `session` + `rt.setSession()` | telemetry | random id, `{}` | new | | 13 | `report` | telemetry | `"console"` | extended (`"interventions"`) | -| 14 | `learn` (+ `rt.learn.clear()`, `rt.learned()`) | persistence | `{persist:unset, expectations:false}`: nothing stored; with `expectations: true`, flows stored, profiles in memory (EXPECTATIONS.md §4) | extended | -| 15 | `model.loadIf`, `model.inlineFallback` | load | `{minDeviceMemoryGB:4, saveData:"skip"}`, `true` | new | +| 14 | `learn` (+ `rt.learn.clear()`; `rt.learned()` and `expectations` 0.3.0) | persistence | `{persist:unset, expectations:false}`: nothing stored; with `expectations: true`, flows stored, profiles in memory (EXPECTATIONS.md §4) | extended | +| 15 | `model.loadIf`, `model.inlineFallback` | load | `{minDeviceMemoryGB:4, saveData:"skip"}` (0.3.0; 0.2.x: `{saveData:"lazy"}`), `true` | new | | 16 | `model.threads`, `model.timeoutMs` | load | `"auto"`, `10000` | promoted from internal | | 17 | `model.maxDecisionsPerMinute` | load | `30` | new | | 18 | `model.unloadAfterIdleMs` | load | `false` | new | -| 19 | `rt.on()` typed events: `shadow`, `breaker`, `limit`, `modelBudget` | API | — | extended | +| 19 | `rt.on()` typed events: `shadow`, `breaker`, `limit`, `modelBudget` | API | none | extended | | 20 | `detect` (0.3.0) | reporting | `"possible"` where the effective mode is observe and `policy.thresholds.report` is unset, else `"likely"` | new | -| 21 | `visual` (+ `rt.checks()`, `rt.checkNow()`, the `check` event) | measured visual checks | `"local"` in `GenClass.init()` in a browser, `false` in `createRuntime()` | new (§4.20); `checks.stallMs` (issue 94) | +| 21 | `visual` (0.3.0; + `rt.checks()`, `rt.checkNow()`, the `check` event) | measured visual checks | `"local"` in `GenClass.init()` in a browser, `false` in `createRuntime()` | new (§4.20); `checks.stallMs` (0.3.0-beta.2, issue 94) | | 22 | `requestTimeoutMs` | heal | off | **proposed only, not implemented** (§4.21) | Built in, with no option: hidden tabs (§5.1): guard and heal skip background situations; observe defers them until @@ -434,11 +446,11 @@ Only `rt.breaker.reset()` clears a trip. `setMode` does not. - Dispatch is queued (a microtask batch, then `requestIdleCallback`) and never runs on the decision or hold path. - Errors from a sink are swallowed, with at most one status event per sink per minute. - On `pagehide`, or once on `visibilitychange→hidden` as a fallback, one `summary` record is emitted and each sink's `flush()` is called. -- `rt.summary().requests` (issue 94): request counts in total and per host (API.md), present once the measured - checks' chunk has loaded. In a sink's `summary` record `requests.hosts` is always empty (host names are request URLs - in part). Telemetry does not send `requests` (§4.21 proposes it). -- `rt.summary()` is always available. Two vendor-neutral adapters ship as entries of the package, outside the first-load bundle: `@genclass/runtime/sinks/{console-json,webhook}` (`src/sinks/`, each a function returning a `SinkObject` that never throws and sends only the record: one JSON line per record on the console, or batched POSTs to the URL the developer configures). The zero-code entries take them by name (`src/cdn/sinks.ts`: `window.GENCLASS_CONFIG.sinks: ["webhook:/genclass"]`, ``, `data-sink`, `init --sink`), as one proxy sink that queues records and loads the adapters on the first record, never before (`src/cdn/sinks-fanout.ts`, a lazy chunk of `/auto`; `dist/sinks/.js` from the CDN for the script tag), so only the proxy is in the first load; a `webhook:` is made absolute at init (against `document.baseURI`) and its endpoint (origin and path) joins `requests.ignore` as an exact match, and from a `` tag it must be same-origin as the browser resolves it (against the page and `document.baseURI`). Each adapter takes `evidence` (default true; false sends the structured fields only; `console-json:noevidence` in the no-code forms); the webhook caps repeats of one bug at `maxPerSignature` (10) per page load. A string in `GenClass.init({ sinks })` is ignored with one warning. -- No Sentry, PostHog or Datadog adapter (owner decision, 2026-10-13): forwarding to those tools is not part of the free runtime. The owner intends it as a paid, server-side feature; it is not built, and no plan on https://genclass.dev/pricing offers it yet. A page config that names `sentry`, `posthog` or `datadog` (any case, with or without an option) gets one console warning for them all that says so, with https://genclass.dev/pricing, and nothing is loaded or sent for those names (`applySinks`); `init --sink sentry` is a usage error that says the same. +- `rt.summary().requests` (0.3.0-beta.2, issue 94): request counts in total and per host (API.md), present once the + measured checks' chunk has loaded. In a sink's `summary` record `requests.hosts` is always empty (host names are + request URLs in part). Telemetry does not send `requests` (§4.21 proposes it). +- `rt.summary()` is always available. Two vendor-neutral adapters (0.3.0, with sinks by name) ship as entries of the package, outside the first-load bundle: `@genclass/runtime/sinks/{console-json,webhook}` (`src/sinks/`, each a function returning a `SinkObject` that never throws and sends only the record: one JSON line per record on the console, or batched POSTs to the URL the developer configures). The zero-code entries take them by name (`src/cdn/sinks.ts`: `window.GENCLASS_CONFIG.sinks: ["webhook:/genclass"]`, ``, `data-sink`, `init --sink`), as one proxy sink that queues records and loads the adapters on the first record, never before (`src/cdn/sinks-fanout.ts`, a lazy chunk of `/auto`; `dist/sinks/.js` from the CDN for the script tag), so only the proxy is in the first load; a `webhook:` is made absolute at init (against `document.baseURI`) and its endpoint (origin and path) joins `requests.ignore` as an exact match, and from a `` tag it must be same-origin as the browser resolves it (against the page and `document.baseURI`). Each adapter takes `evidence` (default true; false sends the structured fields only; `console-json:noevidence` in the no-code forms); the webhook caps repeats of one bug at `maxPerSignature` (10) per page load. A string in `GenClass.init({ sinks })` is ignored with one warning. +- No Sentry, PostHog or Datadog adapter (owner decision, 2026-10-13): forwarding to those tools is not part of the runtime. The genclass.dev collector forwards likely detections to Sentry, PostHog, Datadog, Slack or a signed webhook on every plan, Free included (https://genclass.dev/pricing, `billing/plans.json`), once the app's owner turns it on for an app in their genclass.dev account that sends telemetry with its token: nothing is forwarded for an app without an account or with telemetry off (as it is by default on local hosts), and possible detections are not forwarded (`telemetry-worker/src/forward.ts`). A forwarded record carries the diagnosis, confidence, trigger, page route, runtime and model versions, the time and a dashboard link, never page text. It is sent from the collector, so nothing changes in the app. A page config that names `sentry`, `posthog` or `datadog` (any case, with or without an option) gets one console warning for them all (`Ignored sink : Sentry, PostHog and Datadog forwarding is not in the free runtime: https://genclass.dev/pricing`), and nothing is loaded or sent for those names (`applySinks`); `init --sink sentry` is a usage error that says the same. **Hooks.** `report.ts` (`toSinkRecord`, `SinkDispatcher`), the new `decide/summary.ts`, and a pagehide listener in `runtime.ts`. @@ -503,7 +515,7 @@ Only `rt.breaker.reset()` clears a trip. `setMode` does not. - The default is a device floor: the model (10 MB on the wire, a worker of 190 to 280 MB) is not loaded on devices whose `navigator.deviceMemory` is under 4 GB, nor on Save-Data connections. `deviceMemory` is quantised to powers of two (0.25 … 8), so the floor skips exactly the devices reporting 2 GB or less; browsers that do not report it (Safari, Firefox) load. `model: { loadIf: () => true }` forces the load. - `false` or `"skip"` → no download, `status.state="skipped"` with a reason (`device-memory`, `save-data`, `loadIf`), the runtime behaves as if `model:false` (observes without a model) and prints one console status line saying so (`[GenClass] Model not loaded: …`; silent with `report: "silent"`). - `"lazy"` → `preload` is forced to `"lazy"`. -- `inlineFallback:false` → if the worker cannot be created or fails before it is up, ORT does not run on the main thread: the state is `skipped` with reason `worker-unavailable` and one console line names the worker error. `worker:false` still runs inline (explicit). +- `inlineFallback:false` (0.3.0; 0.2.x ignored it and ran the model inline) → if the worker cannot be created or fails before it is up, ORT does not run on the main thread: the state is `skipped` with reason `worker-unavailable` and one console line names the worker error. `worker:false` still runs inline (explicit). - `loadIf` never affects decisions. **Tests.** @@ -586,7 +598,7 @@ Design, every check and threshold: [VISUAL.md](VISUAL.md). Types: `VisualOptions - `enabled` false or pending, a hidden tab or a prerendered page: no chunk download. A route whose mode is `"off"`: no checks there. `disable()`, `destroy()` and `setMode("observe")` undo every fix at once, `pause()` within 2 s. -`checks.stallMs` (after 0.3.0-beta.1, issue 94): how long a `fetch` or XHR request may stay pending before the +`checks.stallMs` (0.3.0-beta.2, 2026-10-25; issue 94): how long a `fetch` or XHR request may stay pending before the `request-stalled` check reports it; default 15,000, clamped to 1,000..600,000 (VISUAL.md 3.24). **Precedence.** `?genclass-visual=off` > an explicit `visual: false` > `?genclass-visual=on` > the `"local"` default. @@ -697,9 +709,9 @@ status.mode = requested; status.effectiveMode / Report.mode / SinkRecord.mode = > URL overrides (`?genclass-mode`, `?genclass-aggr`, `?genclass-sample`) can only lower settings, unless `debug: true` is set. > > ```ts -> import { init } from "@genclass/runtime"; +> import { GenClass, protectPreset } from "@genclass/runtime"; > -> const rt = init({ +> const rt = GenClass.init({ > mode: "guard", > aggressiveness: "cautious", > enabled: { // live feature flag; flipping off disables mid-session @@ -713,7 +725,7 @@ status.mode = requested; status.effectiveMode / Report.mode / SinkRecord.mode = > ], > requests: { > ignore: ["https://www.google-analytics.com/", /sentry\.io/, "*/rum/*"], -> protect: ["/api/auth/", "/api/payments/", (r) => r.method !== "GET" && r.url.includes("/billing")], +> protect: [...protectPreset("payments", "auth"), (r) => r.method !== "GET" && r.url.includes("/account")], > crossOrigin: "observe", > labels: [{ match: "/api/orders", label: "place order" }], // reports only by default > correlate: (r) => r.headers["x-request-id"], @@ -747,8 +759,9 @@ status.mode = requested; status.effectiveMode / Report.mode / SinkRecord.mode = > onIncident(() => rt.disable({ undo: true })); > ``` > -> **Recommended `requests.protect` starter** (it is not built in; adapt it to your endpoints): -> `[/\/(auth|login|logout|oauth|token|session)\b/, /\/(payment|checkout|billing)\b/]` +> **Recommended `requests.protect` starter:** `protectPreset("payments", "auth")` is built in (or +> `"preset:payments"` and `"preset:auth"` in JSON configs such as `window.GENCLASS_CONFIG`); extend it with your own +> endpoints: `[...protectPreset("payments", "auth"), "/api/cart/"]`. > > **Rollout recipe:** start with `mode: "observe", shadow: "guard"`, and compare the shadow records with your undo and complaint rates. Then switch to `mode: "guard", sample: 0.05`. Watch `rt.summary().undos` and `breaker` events, and widen `sample` as they stay quiet. For QA, `?genclass-sample=1` together with `debug: true` forces a session into the acting group. @@ -759,10 +772,10 @@ status.mode = requested; status.effectiveMode / Report.mode / SinkRecord.mode = 1. `breaker` is on by default. Opt out with `breaker:false`. 2. New `perSubject:10` (per rolling minute) and `perSession:200` limits. Opt out with `Infinity`. This may suppress frequent retries on polling endpoints. 3. `holdBudgetMs` is now a hard ceiling that includes defers and `onBeforeAction` time. - 4. `model.loadIf` defaults to a device floor: no model on devices reporting 2 GB or less or on Save-Data connections (`{ minDeviceMemoryGB: 4, saveData: "skip" }`; the earlier draft said `saveData:"lazy"`). + 4. `model.loadIf` defaults to a device floor: no model on devices reporting 2 GB or less or on Save-Data connections (`{ minDeviceMemoryGB: 4, saveData: "skip" }`, from 0.3.0; 0.1.0-beta.1 to 0.2.x shipped `{ saveData: "lazy" }`). 5. Hidden tabs skip background evaluation and release held items immediately (counted in `summary.model.hiddenSkipped`). 0.3.0: in observe they are deferred and decided when the tab is visible again (§5.1, §5.3). 6. Query and fragment values in recorded URLs are now `…` in situation text. **Model input changes, so add this to the eval plan and re-measure accuracy before release.** Sent URLs are unchanged. - 7. URL overrides can no longer raise mode or aggressiveness without `debug:true`. + 7. URL overrides can no longer raise mode or aggressiveness without `debug:true` (and, since 0.2.3 and 0.3.0-beta.2, not with it either: §1 URL overrides). 8. Ops created under `off`/`observe` route scope can no longer be action targets. - **Renames from the earlier draft:** @@ -781,11 +794,11 @@ status.mode = requested; status.effectiveMode / Report.mode / SinkRecord.mode = - **Hook points:** `types.ts`, `runtime.ts`, `auto.ts`, `decide/{policy,exec,report,decider}.ts`, `observe/{fetch,xhr,websocket,eventsource,nav,errors,dom-user}.ts`, `model/{host,backend}.ts`, `learn/*`. - **New files:** `util/match.ts`, `decide/breaker.ts`, `decide/summary.ts`, `learn/store.ts`. -Nothing was run. This spec comes from reading `packages/runtime/src/types.ts`, `docs/runtime/API.md`, the README and the CONTRACT sections. `types.ts` gives `"balanced"` as the default aggressiveness, so the README text now says "balanced". `API.md` already has `rt.on`, and the new events extend it. +This spec was written before the options were built, from `packages/runtime/src/types.ts`, `docs/runtime/API.md`, the README and the CONTRACT sections. What was built, and how it differs, is in `packages/runtime/CHANGELOG.md` (0.1.0-beta.1 and later) and the JSDoc in `types.ts`. `types.ts` gives `"balanced"` as the default aggressiveness, so the README text says "balanced". `API.md` already had `rt.on`, and the new events extend it. ## Dropped proposals -- [dx] env / preset: Auto-detecting the environment from hostname or NODE_ENV is magic that will silently go wrong (tunnels, staging), and production behaviour would change on upgrade. The defaults are already the safe production profile (guard, cautious), and dev is just mode:'observe', debug:true. A docs snippet works better than another layer of option resolution. A ?genclass-mode URL override is a production safety risk. +- [dx] env / preset: Auto-detecting the environment from hostname or NODE_ENV is magic that will silently go wrong (tunnels, staging), and production behaviour would change on upgrade. The defaults are already the safe production profile (observe, balanced), and dev is just mode:'observe', debug:true. A docs snippet works better than another layer of option resolution. A ?genclass-mode URL override is a production safety risk. - [safety] policy.rules: This is a second policy language with ordering semantics, and a misordered rule can quietly widen permissions. protect already covers the main safety need, and per-scope mode/allow is rarely needed. If scoping is ever needed it should be narrow-only (deny-only), but leave it out for now to keep the library simple. - [safety] rollout: Developers can hand-roll this in two lines around init (mode: inCohort ? 'guard' : 'observe'), and a key() hook adds a PII risk. It is better as docs, or a plugin that tags reports with the cohort. - [integrations] devtools: mountDevtools already exists; an auto-mount with URL/localStorage flags exposes internals to end users in production and adds dev-build detection heuristics. A one-line conditional import is not burdensome. Better kept as the explicit opt-in subpath export (could document an `if (import.meta.env.DEV)` snippet). diff --git a/docs/runtime/QUERY-CACHE-ADAPTERS.md b/docs/runtime/QUERY-CACHE-ADAPTERS.md index cfe77ef7..bfb0b8f4 100644 --- a/docs/runtime/QUERY-CACHE-ADAPTERS.md +++ b/docs/runtime/QUERY-CACHE-ADAPTERS.md @@ -1,7 +1,13 @@ -# Design note: TanStack Query / SWR adapters (not built) +# GenClass design note: TanStack Query and SWR adapters (not built) -Status: proposal, 2026-10-10 (COMPAT workstream). Nothing here is implemented. Evidence: `compat/RESULTS.md` -(run of 2026-10-10). +This note asks whether `@genclass/runtime` should ship adapters that expose the TanStack Query and SWR caches as +stores, and recommends not building them yet: in the compatibility matrix of 2026-10-10 those libraries' keyed caches +already prevented the stale typeahead without GenClass, and the double submits that still happen are visible without +an adapter. If one is built, the SWR cache provider comes first. + +Status: proposal, 2026-10-10 (COMPAT workstream). Nothing here is implemented in any version (0.2.2, 0.3.0-beta.0 or +`main`). Evidence: [`compat/RESULTS.md`](../../compat/RESULTS.md) (run of 2026-10-10; summary in +[RESULTS.md section 16](RESULTS.md#compatibility)). ## Question @@ -51,8 +57,12 @@ like the Redux and Zustand adapters. Apollo's `InMemoryCache` could follow varia ## Open problems (why it is not built now) - **Query keys in the situation text.** A field path built from a query key carries what the user typed - (`search.san francisco`), and situation text is sent with telemetry. Keys would have to be normalised to a shape - (`search.:q`) or hashed. Choosing that is a privacy and telemetry decision for the owner (AGENTS.md rule 9). + (`search.san francisco`), and situation text reaches telemetry when an app turns that on + (`telemetry: { include: { situation: true } }`, off by default since 0.2.1). A webhook sink (0.3.0) carries the + same field paths without that opt-in: in the subject and timeline of the redacted evidence it sends by default + (`evidence: false` leaves those out) and in the changed paths of an intervention record (sent either way); the + telemetry opt-outs and Global Privacy Control do not stop sinks (`packages/runtime/TELEMETRY.md`, webhook sink; + `packages/runtime/src/sinks/webhook.ts`). Keys would have to be normalised to a shape (`search.:q`) or hashed. Choosing that is a privacy and telemetry decision for the owner (AGENTS.md rule 8). - **Distribution shift.** No SIM or REAL row has key-shaped cache paths; the model has not seen this kind of store. Before enabling it by default, generate REAL rows from the TanStack and SWR apps already in `realapps/` (not from `compat/`, which stays evaluation-only) and check false interventions on them. diff --git a/docs/runtime/README.md b/docs/runtime/README.md new file mode 100644 index 00000000..8547780d --- /dev/null +++ b/docs/runtime/README.md @@ -0,0 +1,125 @@ +# GenClass documentation + +This is the index of the GenClass Runtime documentation: which document answers which question, how current each one +is, and where genclass.dev shows it. It covers `@genclass/runtime` on `main` (version string 0.3.0-beta.0, plus the +changes the CHANGELOG lists under 0.3.0-beta.1, unreleased; `npm i @genclass/runtime@beta` installs 0.3.0-beta.0, +`npm i @genclass/runtime` installs 0.2.2) and the model `@genclass/runtime-model` 0.2.0. Items marked 0.3.0 are in +the 0.3.0 line (0.3.0-beta.0 and later) and not in 0.2.x. The [glossary](#glossary) at the end defines the terms the +other documents use. + +## What is GenClass Runtime? + +GenClass Runtime (`@genclass/runtime`) is an open-source (Apache-2.0) browser library that reports race conditions in +web apps while they happen: stale responses that land over newer data, duplicate requests such as double submits, +inconsistent state and failure storms. A small model runs on the user's device, in a Web Worker on WebGPU or WASM, +and each report comes with its evidence. By default it only reports (`observe`): it never holds, delays or changes a +request, a response or a state write. The opt-in `guard` mode may act on a few of them: on held-out data it acted on +7.5% of actionable real-app cases at a 0.00% false-intervention rate (7.8% of clear simulated cases at 0.13%), but in +the six demos no mode lowered the bug rate ([RESULTS.md §1](RESULTS.md#1-model-stages-compared-held-out-simulated-apps), +[§5](RESULTS.md#5-demos)). The `heal` mode is experimental. + +Every measured number (recall, false-intervention rate, cost, size) is in [RESULTS.md](RESULTS.md), with the +false-intervention rate next to each recall figure. + +## Which document answers my question? + +| document | the question it answers | status | on genclass.dev | +|---|---|---|---| +| [Package README](../../packages/runtime/README.md) | What does it do, and how do I install and use it? | current (0.3.0) | [/docs](https://genclass.dev/docs), [npm](https://www.npmjs.com/package/@genclass/runtime) | +| [FAQ](../../FAQ.md) | Short answers to the common questions, with links to the details | current | | +| [INSTALL.md](INSTALL.md) | Every `init` and `remove` flag, the app token, the tested project setups | reference (0.3.0) | | +| [DASHBOARD.md](DASHBOARD.md) | How do I see my app's reports on genclass.dev, and what do `login`, `connect`, `whoami`, `logout` and `dashboard` do? | reference (0.3.0) | [/docs/dashboard](https://genclass.dev/docs/dashboard) | +| [SINKS.md](SINKS.md) | How do I send reports to my own logging (the sink record format and the two adapters)? | reference (0.3.0) | | +| [API.md](API.md) | Every export, option, event, action and type | reference | [/docs/api](https://genclass.dev/docs/api) | +| [OPTIONS-SPEC.md](OPTIONS-SPEC.md) | Why each configuration option exists and how the options combine | design spec, built | [/docs/options](https://genclass.dev/docs/options) | +| [INTERCEPTION.md](../../packages/runtime/INTERCEPTION.md) | Which browser APIs it wraps, what each action does, and how to undo it | reference, checked by a test | [/docs/interception](https://genclass.dev/docs/interception) | +| [TELEMETRY.md](../../packages/runtime/TELEMETRY.md) | What anonymous diagnostics are sent, and how to turn them off | reference | [/docs/telemetry](https://genclass.dev/docs/telemetry) | +| [PRIVACY.md](../../PRIVACY.md) | What genclass.dev collects and keeps | policy | [/privacy](https://genclass.dev/privacy) | +| [SECURITY.md](../../SECURITY.md) | How do I report a vulnerability, and how do I check a published tarball? | policy | [/security](https://genclass.dev/security) | +| [THREAT-MODEL.md](THREAT-MODEL.md) | What could go wrong, or be made to go wrong, and what the code does about it | reference | [/docs/threat-model](https://genclass.dev/docs/threat-model) | +| [RESULTS.md](RESULTS.md) | How well does it work, and what does it cost? Every measured number | source of truth | [/benchmarks](https://genclass.dev/benchmarks) | +| [compat/RESULTS.md](../../compat/RESULTS.md) | Which frameworks and state libraries does it run in? | generated by the compat runs | [/docs/compatibility](https://genclass.dev/docs/compatibility) | +| [AGENT.md](AGENT.md) | How does my coding agent see a race, replay it and prove the fix? `@genclass/runtime/agent` and the `@genclass/mcp` server, its tools and its threat model | reference (`0.3.0-beta.1` and `@genclass/mcp` 0.1.0, not on npm yet) | | +| [GenClass for Playwright](../../packages/playwright/README.md) | How do I put GenClass on every page my Playwright tests open, with seeded network chaos, and get one pull-request report of what it flagged? | reference (`@genclass/playwright` 0.1.0, the fixture and the reporter, not on npm yet) | | +| [Model card](../../packages/runtime-model/MODEL_CARD.md) | What is the model: size, training data, gate profiles, accuracy | reference (model 0.2.0) | | +| [ARCHITECTURE.md](ARCHITECTURE.md) | How do the parts fit together, and why are they built this way? | design | [/docs/architecture](https://genclass.dev/docs/architecture) | +| [VISUAL.md](VISUAL.md) | How the measured visual checks work: every check, threshold and exclusion | design; v1 built in 0.3.0 | [/visual](https://genclass.dev/visual) | +| [EXPECTATIONS.md](EXPECTATIONS.md) | How flow expectations learn what each user action usually does | design; built in 0.3.0, opt-in, false-report rate not measured | | +| [ANALYTICS.md](ANALYTICS.md) | How interaction analytics count page views, time on page, scrolls and clicks | design; built in 0.3.0-beta.1, not on npm yet | | +| [QUERY-CACHE-ADAPTERS.md](QUERY-CACHE-ADAPTERS.md) | Should GenClass ship TanStack Query and SWR adapters? | proposal, not built | | +| [CONTRACT.md](CONTRACT.md) | The rules for people changing the runtime and its training pipeline | binding, for contributors | | +| [CHANGELOG](../../packages/runtime/CHANGELOG.md) | What changed in each release | current | | + +The repository's [README](../../README.md) gives the overview, and [AGENTS.md](../../AGENTS.md) the repository map +and the rules for coding agents working on GenClass itself. + +## Glossary + +- **observe**: the default mode. GenClass watches and reports what the model finds; it never holds, delays or + changes a request, a response or a state write, and no action runs. +- **guard**: an opt-in mode. GenClass may also run guard-tier actions when the gate clears: `discard`, `defer`, + `coalesce` and `delay`. They withhold, deduplicate or slow something down; they never invent data or fail a + request. +- **heal**: the experimental mode. It adds the heal-tier actions `block`, `serve_cached`, `retry`, `hedge`, + `rollback`, `resync` and plugin actions. +- **trigger**: the kind of moment at which the runtime may ask the model: `delivery` (a response or pushed message + is about to reach the app), `mutation` (a state write), `request` (a request is about to be sent), `failure`, + `stall`, `inconsistency` (a learned relation between fields broke), `transition` (an operation changed state + unlike it usually does), `error`, and `ask` (a question the app asks itself with `rt.ask()` or `rt.decide()`, + which offers no actions). Each of the other eight has its own passive action and its own set of actions + ([API.md, Triggers and actions](API.md#triggers-and-actions)). +- **situation**: the short text the model reads at a trigger (sections `app`, `trigger`, `facts`, `in_flight`, + `timeline`, `state`, `stats`), with two standing questions: the diagnosis and the action. Its format is frozen + under the tag `situation-v2.3`. By default (`triage: "salient"`) only salient situations (at least one + non-neutral fact) reach the model; the app's own `rt.ask()` and `rt.decide()` questions, and plugin questions + marked `always`, are asked regardless. +- **diagnosis**: the model's label for a situation: `expected`, `stale`, `conflict`, `duplicate`, `inconsistent`, + `failing`, `transient`, `slow`, `overload` or `unusual`. +- **detection (likely, possible)**: a report that something is wrong. Likely: the top diagnosis is not `expected` + and its probability reaches the report gate of the active profile. Possible (0.3.0, observe by default): it + reaches the `eager` profile's lower report value but not the report gate. It prints with `console.debug`, shows + dimmed in the devtools, is counted in `summary().possibleDetections` and is sent in telemetry as a `detect` event + with `tier: "possible"`; `detect` and `report` listeners and a `report` function get it only when the app sets + `detect: "possible"`, and sinks only with `possible: true`. +- **gate**: the policy check before any non-passive action. It applies protected and cross-origin requests, the + route rule an operation started under (`off` or `observe` rules block it), the mode, `allow` and `deny`, the + diagnosis (never `expected`), the model's gate for the active profile, the action limits and the app's + `onBeforeAction` veto ([API.md, Policy](API.md#policy)). Model 0.2.0 ships a gain gate: the estimated gain of the + best permitted action over doing nothing must clear a margin per tier and trigger. A model without a gate of its + own uses a summed probability of 0.9 (guard) or 0.8 (heal) at `balanced`; `cautious` adds 0.05 to both and + `eager` subtracts 0.05. +- **profile, aggressiveness**: `cautious`, `balanced` (default) or `eager`, or a number from 0 to 1 between them. + It selects the model's gate profile: its margins for actions and its report value for detections. The values + are in the [model card](../../packages/runtime-model/MODEL_CARD.md). +- **FIR (false-intervention rate)**: the share of cases where doing nothing was best but the gate fired anyway. + RESULTS.md reports it next to every recall figure. +- **false-detection rate**: the share of decisions on correct traffic that were reported as a problem. +- **recall**: for detections, the share of real problems reported with the right diagnosis. For actions, on + simulated apps, the share of clear cases (the gold label puts at least 0.9 on one permitted action other than + doing nothing) on which the gate acted, and on real apps the share of actionable cases on which it acted + ([RESULTS.md §1](RESULTS.md#1-model-stages-compared-held-out-simulated-apps)). +- **hold budget**: the longest GenClass may hold a response, message or request in guard and heal while the model + decides (`policy.holdBudgetMs`, by default 1.5 times the model's recent median latency, between 150 and 800 ms). + Observe holds nothing. +- **late revert**: in guard and heal, a state write that already applied and that the model then answers `discard` + for is reverted, but only within 800 ms of applying (`LATE_REVERT_MS`), if nothing has overwritten it and its + operation has written nothing else since. It is reported as such and can be undone. +- **local or private host**: a page host where the default telemetry stays off: `localhost` and `*.localhost`, + `*.local`, `*.test`, `*.internal`, `*.home.arpa`, single-label intranet names, loopback, private, link-local, CGNAT + and 0/8 addresses, and an empty host name such as a `file:` page's (`isLocalHost` in `src/telemetry/config.ts`). An + intranet or VPN-only production app is on one. Not the same as a **development host** (`src/devhost.ts`: http(s) on + `localhost`, `127.0.0.1`, `[::1]` or `*.localhost` with an explicit port, or `*.test`), the narrower set where + `?genclass=` or `localStorage.genclass` may raise the mode and where the measured visual checks are on by default + ([THREAT-MODEL.md T8](THREAT-MODEL.md#t8-configuration-by-url-storage-and-meta-tags)). +- **protected request**: a request matched by `requests.protect` (for example `protectPreset("payments", "auth")`). + It is observed and reported but never acted on, nor is anything its response causes. +- **measured visual check**: a deterministic measurement of a visual or CSS bug (unreadable text, a broken image, a + stuck scroll lock, and others) with a documented threshold, reported as kind `"check"`. It is never a model + finding and never feeds a situation, the gate or the model ([VISUAL.md](VISUAL.md)). One indirect path remains: a + check's read that makes the browser lay out a very heavy page can cause a long task, which the runtime's `perf` + observer records as an event the model can read (VISUAL.md section 1). +- **flow expectation**: an opt-in, deterministic report that a step a user action usually causes (a request, a + write, a navigation) did not happen this time ([EXPECTATIONS.md](EXPECTATIONS.md)). The model is not consulted. +- **Jev**: the name of TypeSafe AI's typed-decision API, which GenClass is not affiliated with. The GenClass model + reads questions in the same request shape (`noul`, `choice` and `score` questions over a state), from the + project's earlier research in `research/`. diff --git a/docs/runtime/RESULTS.md b/docs/runtime/RESULTS.md index 1bb537f0..b1b9c9e8 100644 --- a/docs/runtime/RESULTS.md +++ b/docs/runtime/RESULTS.md @@ -1,24 +1,73 @@ # GenClass Runtime: results, comparisons and training log -Updated 2026-10-08. All numbers are measured on held-out data. Every recall or fix rate is reported next to its +This page collects the measured numbers for GenClass Runtime (`@genclass/runtime`) and its model +(`@genclass/runtime-model`): model quality, false interventions, detection rates, demo and real-app runs, the +compatibility matrix, first-load size, browser cost and the measured visual checks. The READMEs and the model card +quote these numbers; the raw reports they come from are listed below. The current releases are runtime 0.2.2 (npm +`latest`) and 0.3.0-beta.0 (`beta`) with model 0.2.0, but the measurements span runtimes from before 0.1.0-alpha.1 +to 0.3.0-beta.0, release branches and unreleased work on `main`. Measured on 0.2.2 or on the branches that became +0.3.0-beta.0: the CI first-load and telemetry-volume numbers (§8 to §11), the dashboard counts (§12) and the visual +checks (§13, §14). The demo, real-app, compatibility-matrix and browser-cost numbers come from earlier runtimes and +were not re-run; the model numbers are held-out evaluations of each model, with model 0.2.0 the current one. Each +section says what it measured and on which model, runtime version or branch. + +Updated 2026-10-22 (the summary below and sections 16 and 17 added, and the numbers that were recorded only in the +READMEs and the model card added to sections 1, 2 and 8; the newest measurement is section 15, 2026-10-18). Model +numbers are measured on held-out data. Every recall or fix rate is reported next to its false-intervention rate (FIR): the share of cases where doing nothing was best but the gate fired anyway. The raw -reports are in `training/EVAL.md`, `sim/SEPARABILITY.md`, `realapps/README.md` and -`demos/results.md`. +reports are in `training/EVAL.md`, `sim/SEPARABILITY.md`, `realapps/README.md`, `demos/results.md` and +[`compat/RESULTS.md`](../../compat/RESULTS.md). + +## Current numbers in one place + +The headline numbers for the current releases, each checked against the section it names. On npm (dist-tags on +2026-10-22): `@genclass/runtime` 0.2.2 (`latest`) and 0.3.0-beta.0 (`beta`), with the model +`@genclass/runtime-model@0.2.0` (training run `r17-v2dT`) at its default profile, `balanced`. Rows further down for +round 1, `r17-v2a` to `r17-v2c` and model 0.1.0 (`r17-v2b`) are superseded and kept as the record. + +| measure | number | section | +|---|---|---| +| Observe (the default): problems detected with the right diagnosis, held-out real apps | likely tier (report gate 0.90 at `balanced`; what `rt.on("detect")`, `report` listeners and the console's default level get, and all that 0.2.2 reports): duplicate submits 75%, stale overwrites 30% (`situation-v2.3` slice; 21% on `situation-v2`), genuine broken state 6%. 0.3.0-beta.0's observe default (`detect: "possible"`) also records possible detections, at the `eager` report gate 0.70, to `console.debug`, the devtools and telemetry (`tier: "possible"`): likely and possible together 77% / 45% (40% on `situation-v2`) / 13%, the `eager` profile's measured rates, not measured end to end on 0.3.0 | §1, §11 | +| Observe: false detections | likely tier: the 0.2.0 fit held them to at most 2% (real apps) and 3% (simulated) of clean decisions, met on held-out test; the exact rate is not published. Possible tier (the 0.3.0 observe default): at most 5% / 6% for likely and possible together; the share of possible detections that are false is not measured. Compatibility matrix (likely tier, `balanced`): 49 false findings in 880 runs on correct apps. Bench demos (observe, model 0.2.0): 2 of 30 clean trials had a detection at `balanced`, 3 of 29 at `eager` | §11, §16 | +| Guard (opt-in): false-intervention rate (FIR) | 0.13% of held-out simulated cases where doing nothing was best; 0.00% on held-out real apps. `cautious` 0.005%, `eager` 0.54% | §1 | +| Guard: recall, at the FIR above | acts on 7.8% of clear simulated cases and 7.5% of actionable real-app cases | §1 | +| Heal (experimental): FIR and recall | FIR 0.59% simulated, 0.00% real apps; acts on 7.1% of clear simulated and 10.4% of actionable real-app cases | §1 | +| Project targets | guard FIR ≤ 0.1% and heal FIR ≤ 0.5%: `cautious` meets both, `balanced` is slightly over both (it was fitted to guard ≤ 0.3% and heal ≤ 1%, and meets those), `eager` is well over | §1 | +| Six demos under network chaos (model 0.2.0, runtime 0.1.0-beta.4, `balanced`) | bug rate 72% without GenClass, 70% observe, 72% guard, 73% heal: no mode lowered it (differences under about 5 points are noise); 0 false interventions in 30 clean trials each in guard and heal | §5 | +| Correct apps left unchanged (always-passive model in heal mode, compared with observe mode, not with running without GenClass; runtime as of 0.1.0-alpha.1, not re-run since) | 0 of 396 clean runs changed (66 real apps × 6 seeds); with network chaos, 3 of 198 | §4 | +| Compatibility matrix (7 apps: 6 frameworks and plain HTML; 15 data layers, 8 scenarios, 10 seeds, every mode; 2026-10-10, runtime commit `d04eff3`: the 0.1.0-beta.4 code plus the "Model ready" log fix; 0.2.0's audit trail, protect presets and hold-budget ceilings and 0.2.1's first-request hold were not run through the matrix; it was not re-run on 0.2.2 or 0.3.0-beta.0) | 0 bugs introduced in 3,510 runs; 0 actions on correct apps in 1,760 guard and heal runs; 10 double submits prevented, all by heal `block` on one data layer | §16 | +| Browsers measured | Chromium only (the demos, the compatibility matrix, the visual checks); Firefox and Safari are not measured | §5, §13, §14, §16 | +| First-load JavaScript, minified and gzipped | 0.3.0-beta.0 (the published build, tag `v0.3.0-beta.0`): 108.25 KB for `/auto`, 100.13 KB for the main entry, 128.56 KB for the script tag. 0.2.2: 99.17 / 91.70 KB. With the interaction-analytics stub (branch `feat/interaction-analytics`, train VM, 2026-10-18, since merged into `main` and unreleased; current `main` not re-measured): 108.529 / 100.395 KB | §9, §10, §15 | +| Model files and speed | q8 10.16 MB (WASM), fp16 13.57 MB (WebGPU with `shader-f16`); 8.8M parameters; about 182 / 330 / 599 ms per pass at 500 / 780 / 1,170 tokens on one WASM thread (a least-squares fit over runs of onnxruntime-web 1.30 in Node on the training VM, not in a browser; `training/ortweb/validate.mjs`) | §2 | +| Browser cost on a pilot app (0.1.0-beta.3, model 0.2.0, 4× CPU throttling) | +190 to 280 MB renderer memory while the model is loaded; 12.7 MB downloaded on a first visit on WASM (the READMEs say about 13 MB); no measurable change to the load event, LCP or long tasks | §17 | +| Telemetry volume, one tab at the model's maximum decision rate for an hour (scripted decider, observe mode, situation text off as by default; release review commit `9d567a0`, not the published build) | 251.9 KB in 165 requests when every decision is routine; 2,068.5 KB in 361 requests when every decision is a detection | §10 | +| Measured visual checks (deterministic, not the model) | fixture pages: 35 of 35 planted faults found, 0 false reports. Bench: 0 false reports and 0 fixes in 128 page views of the demos, templates and compat apps; recall 1.00 on 20 of 23 planted fault kinds (run 4), 19 of 23 (run 5); public sites: 8 findings in 96 loads in run 4 (5 confirmed real, 0 check errors, 3 unconfirmed), 6 findings in 97 loads in run 5 (after the review; not looked at) | §13, §14 | +| Flow expectations (opt-in) | false-report rate not measured yet | §5 | +| Coding agents: `genclass_explore` and `genclass_replay` on one real page (train VM, headless Chromium, 2026-10-20; one page, one bug class, not a recall figure) | explore found the stale-results race in 5 of 6 reordered runs of the buggy page and reported nothing on the fixed page (0 of 6 differ: no false report); replay reproduced it before the fix and not after; the generated Playwright test failed before and passed after | §18 | ## 1. Model stages compared (held-out simulated apps) -| Stage | Data | Diagnosis | Action | Guard FIR | Heal FIR | Recall on clear stale/duplicate cases | +| Stage | Data | Diagnosis | Action | Guard FIR | Heal FIR | Recall on clear cases (see the note below) | |---|---|---|---|---|---|---| | GenClass 0.1 (no runtime training) | none | 8.0% | 37.6% | ~0% | 0.0–0.33% | ~0% | | Stage 1: curriculum only, R17 (tested on simulated apps) | 1.2M curriculum rows | 44% | 48% | 9.7% | 18.4% | n/a (over-intervenes) | -| Stage 2 pilot, R17 | 224k pre-freeze sim rows, mushy labels | 91.1% | 78.3% | 0.00% | 0.08% | ~1% | -| **Final round 1, R17** (format v1) | 448k sim rows (phase A) | **90.5%** | **81.9%** | **0.05%** | **0.24%** | **7.7%** | -| Final round 1, R32 (format v1) | same | 89.7% | 81.8% | 0.05% | 0.22% | 5.9% | -| **r17-v2a** (format v2, first round) | 2B tokens of v2 sim gold + v2 curriculum | 84.4% | 77.9% | **0.00%** | 0.46% | 3.1% (heal clear-case 11.3%) | -| **r17-v2b** (+1B tokens with real-app gold) | sim 84.2% / real 83.6% | sim 77.8% / real 80.0% | 0.02% sim, 0.00% real (derived gates) | 0.23% sim, 0.00% real | heal 6.0% clear; real duplicate 8.2% (14% held-out apps) | -| 150M teacher (t150-v2a, 1B tokens, v2 + real gold) | sim 81.8% / real 82.0% | sim 76.5% / real 78.1% | – | – | lower than r17-v2b everywhere, so distillation was skipped | -| **r17-v2c** (+ on-policy round a, v2.2 retry labels) | sim 84.4% / real 83.7% | sim 77.8% / real 79.9% | 0.02% sim, 0.00% real | 0.22% sim, 0.00% real | guard recall on clear 2.1% (v2b 0.6%); heal gain 5.6% (v2b 3.8%); real duplicate 11.2% | -| Next: r17-v2d (mass gate) vs r17-v2dT (gain gate), winner ships as model 0.2.0 | 10.4M sim gold + 51M unlabeled + 616k real | in progress | | | | | +| Stage 2 pilot, R17 | 224k pre-freeze sim rows, mushy labels | 91.1% | 78.3% | 0.00% | 0.08% | ~1% (heal recall on every row where acting was best, not clear cases only: 1.4%) | +| Final round 1, R17 (format v1) | 448k sim rows (phase A) | 90.5% | 81.9% | 0.05% | 0.24% | 7.7% | +| Final round 1, R32 (format v1) | same | 89.7% | 81.8% | 0.05% | 0.22% | 6.8% | +| r17-v2a (format v2, first round) | 2B tokens of v2 sim gold + v2 curriculum | 84.4% | 77.9% | 0.00% | 0.46% | heal 11.3% (all clear cases, fixed gates; `training/EVAL.md`, `sim2e`) | +| r17-v2b (its gates were refit at 10:30 and that refit shipped as model 0.1.0: guard FIR 0.01%, heal 0.07%, "Model 0.2.0" table) | from r17-v2a, +1B tokens: sim v2 gold 70%, real-app gold 18% (eval-set rows left out), curriculum replay 12% | sim 84.2% / real 83.6% | sim 77.8% / real 80.0% | 0.02% sim, 0.00% real (first derived gates) | 0.23% sim, 0.00% real (first derived gates) | heal 6.0% clear, real-app action cases 9.5% (derived gates; `training/EVAL.md`); at the fixed heal gate 0.8 (heal FIR 0.50% `sim2e`, 0.31% `real2e`, 0.00% on the eval set's clean rows): real duplicate 8.2% (14% on held-out apps) | +| 150M teacher (t150-v2a) | 1B tokens of v2 sim gold and real-app gold | sim 81.8% / real 82.0% | sim 76.5% / real 78.1% | – | – | lower than r17-v2b everywhere, so distillation was skipped | +| r17-v2c (its q8 export, in the head-to-head for model 0.2.0) | + on-policy round a, v2.2 retry labels (`training/configs/mix_v2c.json`) | sim 85.2% / real 84.0% | sim 77.8% / real 79.9% | 0.016% sim, 0.00% real | 0.21% sim, 0.00% real | guard 1.8% clear; heal 6.0% clear, real-app action cases 9.8% | +| **r17-v2dT (gain gate), shipped as `@genclass/runtime-model@0.2.0`**, profile `balanced` | from r17-v2c, 1.5B tokens in all (the same data and schedule as v2d): sim v2 gold (`sim2r`), the `situation-v2.3` sim gold, on-policy round a, real-app gold (v2c plus the v2d and v2.3 top-ups), curriculum replay (8% of the mix, `training/configs/mix_v2dT.json`), both eval sets left out; action labels are gain distributions (softmax(gain/τ = 1)) | sim 85.4% / real 85.5% | sim 73.9% / real 77.8% (gain labels: not comparable with the rows above) | **0.13% sim, 0.00% real** | **0.59% sim, 0.00% real** | all clear cases: guard 7.8% (real-app action cases 7.5%); heal 7.1% (real 10.4%); all three profiles in "Model 0.2.0" below | + +Recall column: the share of clear cases (the gold label puts at least 0.9 on one permitted action other than doing +nothing) on which the gate acted. The two final round 1 rows count the clear stale and duplicate cases only (R17 7.7% +of 220 rows; on all clear cases R17 acted on 4.2% in guard and 6.1% in heal, R32 on 3.5% and 5.9%). From r17-v2b on, +the cells count all clear cases in the mode named, with real-app recall where given (`training/EVAL.md`). r17-v2a's +cell (heal, 11.3%) counts all clear cases at the fixed gates of its row. The r17-v2c row is the q8 +export measured in the head-to-head (`training/EVAL.md`, "Head-to-head for `@genclass/runtime-model@0.2.0`"), with +gates fitted on dev at 0.8 × each limit; an earlier PyTorch evaluation of r17-v2c, recorded only in commit `f9509b8` +without its report, is not used here. ### r17-v2a on held-out data (details) @@ -26,14 +75,17 @@ reports are in `training/EVAL.md`, `sim/SEPARABILITY.md`, `realapps/README.md` a |---|---|---|---|---|---|---| | Sim test (`sim2e`) | 84.4% | 77.9% | 0.00% | 0.46% | heal 6.8% | heal 45% on clear cases (harm 1.8% of rows) | | Held-out app features (`sim2f`) | 81.0% | 77.4% | 0.00% | 0.75% | heal 6.8% | – | -| **Real apps, eval set (16,600)** | – | – | **0.00%** | **0.00%** | duplicate 0.6%, stale 1.0% | **duplicate 35% (64% on held-out apps)**, broken state 10%, stale 9%, with FIR 0.05–0.35% | +| Real apps, eval set (16,600) | – | – | 0.00% | 0.00% | duplicate 0.6%, stale 1.0% | duplicate 35% (64% on held-out apps), broken state 10%, stale 9%, with FIR 0.05–0.35% | **Reading.** v2 kept v1's safety and made the model much better at ranking, but the fixed 0.9/0.8 gates were -chosen before labels became expected costs and now block nearly every action. Next: +chosen before labels became expected costs and now block nearly every action. The next steps planned then: - **Thresholds:** derive per-tier, per-trigger gate thresholds from held-out data (FIR ≤ 0.1% guard / 0.5% heal on clean real-app and sim traffic; fit on dev, verify on test) and ship them in the model's meta.json. - **Training:** finish the 150M teacher (with real-app gold), distil, and run DAgger rounds. +What happened: the derived gates below; the teacher came out weaker than r17-v2b, so nothing was distilled; on-policy +(DAgger) rounds a and b; then r17-v2c, r17-v2d and r17-v2dT (§7). + ### Data-derived gate thresholds (r17-v2a) The thresholds are fitted per tier and trigger on held-out dev data, under these constraints: FIR ≤ 0.1% (guard) @@ -52,17 +104,28 @@ limited by the model's discrimination, which is what the 150M teacher, distillat ### Observe mode: detection quality (r17-v2b) +Model 0.1.0, superseded: model 0.2.0's detection rates are in the "Model 0.2.0" table below and in §11. + A *detection* is GenClass reporting a problem, such as "this response overwrote newer state", without changing anything. This is the phase-1 product. Detections fire when the top diagnosis is not `expected` and its probability is ≥ `gate.report`. | Held-out test set | Report threshold | Detected | Precision | False detections on clean real-app traffic | |---|---|---|---|---| -| Real apps never trained on (eval set) | 0.70 | duplicate submits **76%**, stale overwrites **43%**, broken state 27% | 0.96 | 0.62% | -| Real apps never trained on (eval set) | **0.85 (shipped)** | duplicate ~76%, stale 28% | higher | **0.36%** | +| Real apps never trained on (eval set) | 0.70 | duplicate submits 76%, stale overwrites 43%, broken state 27% | 0.96 | 0.62% | +| Real apps never trained on (eval set) | 0.85 (shipped in model 0.1.0) | duplicate ~76%, stale 28% | higher | 0.36% | | Simulated apps (sim2e) | 0.70 | 50.9% of anomalies | 0.91 | gold-expected rows: 2.59% (over the 2% limit) | | Simulated apps (sim2e) | 0.85 | – | – | 1.11% | +Model 0.1.0's card ([`MODEL_CARD.md` at `1f0f617`](https://github.com/genclass-dev/GenClass-lib/blob/1f0f617/packages/runtime-model/MODEL_CARD.md), +tag `runtime-model-v0.1.0`, "Observe mode: detections at report 0.85") also counts false detections on every held-out +test decision whose correct diagnosis was `expected`: 1.41% simulated (149 / 10,582: `sim2e` 1.11%, `sim2f` 1.72%) +and 3.78% real apps (305 / 8,079). On the `real2e` test slice, 240 of its 303 false detections came from one held-out +app, flagged as `stale` on its `article.inProgress` field; without that app the slice is 1.05% (63 / 6,006). The same +card: 65.0% (simulated) and 60.7% (real) of problem decisions were flagged, and 94.1% and 89.5% of flags had the right +diagnosis; that one app's 240 false detections were 25% of its 955 clean decisions; at report 0.9, 0.89% (simulated) +and 3.44% (real) of clean decisions were flagged, with 58% and 52% of problems flagged. + Per-class precision / recall at 0.70 on sim2e: | Class | Precision | Recall | @@ -75,8 +138,12 @@ Per-class precision / recall at 0.70 on sim2e: | inconsistent | 0.97 | 0.53 | | conflict | 0.89 | 0.54 | -The shipped threshold, 0.85, was chosen after the dev-fitted 0.70 failed the simulated false-detection limit on -test, so that one choice is test-informed. Future exports fit with a stricter dev margin. +In r17-v2b's first fit, 0.85 was chosen after the dev-fitted 0.70 failed the simulated false-detection limit on +test, so that choice was test-informed. In the refit published as model 0.1.0, 0.85 is the dev fit itself; that +refit's one test-informed choice is the heal failure gate, 0.95 (`training/EVAL.md`). On the shipped model's own +traffic (on-policy round b test) the refit's 0.85 gave 3.1% false detections on simulated gold-`expected` rows, over +the 2% limit. The refit and the 0.2.0 head-to-head fitted at 0.8 × each limit on dev; model 0.2.0's final fit +used the limits themselves, with the real-app certification set added to dev ("Model 0.2.0" below). ### On-policy round a (the model acting inside the simulator) @@ -106,8 +173,8 @@ Compared with round a: - **delivery** now acts: 99 acts, 18% false. Remaining issues: -- the **heal transition gate (0.55) is too loose on-policy**: 41% of its acts are false. Gates are being refitted on - on-policy dev data, falling back to the tier default where evidence is thin; +- the **heal transition gate (0.55) is too loose on-policy**: 41% of its acts are false. Model 0.2.0's gates were + refitted with on-policy dev data, falling back to the tier default where evidence is thin ("Model 0.2.0" below); - **request** acts are 19% false, mostly coalescing deliberate repeated clicks. ### Head-to-head for model 0.2.0 (all on the shipped q8 outputs) @@ -133,9 +200,9 @@ At equal safety (one global threshold swept on the same test records), v2dT capt v2dT also leads guard-mode real-app recall at FIR ≤ 0.5%: 14.1%, against 3.1–3.4%. **Why v2dT ships conservatively.** There are too few clean real-app dev rows to certify per-trigger margins, so the -fit falls back to strict defaults. The same thin evidence pushes `gate.report` up to 0.97–0.99. Next: a large -real-app certification dev set (dev apps only), then refit all three, then pick 0.2.0 by its shipped operating -point. +fit falls back to strict defaults. The same thin evidence pushes `gate.report` up to 0.97–0.99. The next step, a +large real-app certification dev set (dev apps only), a refit of all three and a choice of 0.2.0 by its shipped +operating point, is "Model 0.2.0" below. **Quantization.** The q8 export was fixed by moving to MatMulNBits block 16: 223/223 decisions match PyTorch, the file is 10.16 MB, and it is fp16-free. @@ -156,10 +223,16 @@ The 0.1.0 row comes from the earlier, stricter fit without the certification set like-for-like. FIR on real apps is 0.00% for every profile. Why v2dT: v2d is better in heal mode (gain 8.1% vs 5.7%, real recall 21.8% vs 10.4%), and v2dT is better in guard -mode (6.7% vs 4.2%). Guard is the default mode, so v2dT ships. Choosing the model per mode is an open item. +mode (6.7% vs 4.2%). v2dT ships because guard is the first mode that acts (observe only reports). When 0.2.0 was +chosen, guard was the default of the branch it shipped from (runtime 0.1.0-beta.1); observe has been the default +again since 0.1.0-beta.2 (CHANGELOG). Choosing the model per mode is an open item. -Targets: guard FIR ≤ 0.1% (met), heal FIR ≤ 0.5% (met), calibration error ≤ 0.02 (not yet measured for 0.2.0; r17-v2b: 0.021–0.023), -diagnosis ≥ 95% (not yet), clear-case recall ≥ 80% (not yet). +Targets: guard FIR ≤ 0.1% and heal FIR ≤ 0.5%. `cautious` meets both (0.005% / 0.26%); `balanced`, the default, is +slightly over both (0.13% / 0.59%); `eager` is well over (0.54% / 1.84%). FIR on held-out real apps is 0.00% for every +profile. Each profile was fitted to limits of its own, and all three meet them on held-out test: `cautious` guard +FIR ≤ 0.1%, heal ≤ 0.5%, report false ≤ 1% real / 2% simulated; `balanced` guard ≤ 0.3%, heal ≤ 1%, report ≤ 2% / 3%; +`eager` guard ≤ 1%, heal ≤ 3%, report ≤ 5% / 6% (`training/EVAL.md`). Calibration error ≤ 0.02 (not yet measured +for 0.2.0; r17-v2b: 0.021–0.023), diagnosis ≥ 95% (not yet), clear-case recall ≥ 80% (not yet). The stage-1 row comes from the 200 sim sample rows (`training/EVAL.md`). On its own curriculum test set, stage 1 scores 98% / 98% with 0% FIR. @@ -172,9 +245,30 @@ scores 98% / 98% with 0% FIR. | R32 (GenClass 0.1 / ettin-32m, pruned) | 22.5 MB | 569 / 954 / 1,580 | 89.7% / 81.8% / 0.05% | | GenClass 0.1 (original) | 57 MB | ≈ 900 at 780 | not trained for this | -**Decision:** R17 is the default on every device. It matches R32 at about a third of the size and latency. Both -q8 exports are fp16-free, so they run on WebGPU without `shader-f16`, and agree with PyTorch on 223/223 decisions -in onnxruntime-web. +**Decision:** R17 is the default on every device. It matches R32 at under half the size and about a third of the +latency. Both q8 exports are fp16-free, so they run on WebGPU without `shader-f16`, and agree with PyTorch on 223/223 +decisions in onnxruntime-web. + +Sources: the R17 row is the delivered round-1 export in `training/EVAL.md`. The R32 latencies are the delivered +round-1 R32 export, measured the same way and recorded in the training log (`training/LOG.md` at commit `fcd1e68`, +taken out of the public tree in `66c0f03`); `training/EVAL.md` has only the earlier stage-1c R32 export, at +499 / 879 / 1,539 ms. Against either, R17 takes 31% to 38% of R32's time. + +**The shipped export** (`@genclass/runtime-model@0.2.0`, `r17-v2dT`; `packages/runtime-model/MODEL_CARD.md`, +`training/EVAL.md`): q8 10,160,186 bytes (MatMulNBits 8-bit, block 16, which restored parity with PyTorch on 223/223 +decisions) and fp16 13,569,879 bytes; 8.8M parameters (8.1M encoder + 0.7M heads after vocabulary pruning); about +182 / 330 / 599 ms at 500 / 780 / 1,170 tokens on one WASM thread (a least-squares fit over runs of onnxruntime-web +1.30 in Node on the training VM, not in a browser; `training/ortweb/validate.mjs`). The 9.58 MB size and the +latencies in the R17 row above are the round-1 export, obtained the same way, at MatMulNBits block 32, the q8 +format `training/README.md` describes and still the default of `training/export_runtime.py` (`--block 32`); the 0.2.0 +exports were made with block 16 (`training/EVAL.md`). + +**Other latencies.** On WebGPU, on one Apple GPU with model 0.2.0, a warm-up pass on a situation of about 430 tokens +took 49–74 ms and the first real decision 137 ms (`packages/runtime/CHANGELOG.md`, 0.2.2; the model card); there is +no systematic WebGPU benchmark. Four WASM threads on a crossOriginIsolated page were about 3× faster than one with +GenClass 0.1's 32M-parameter q8 model, in headless Chromium on the shared train VM: 271–360 ms against 837–981 ms at +780 sequence tokens, and 464–556 ms against 1,452–1,791 ms at 1,174 (`packages/runtime/src/model/README.md`); threads +have not been re-measured with model 0.2.0. ## 3. Why round 1 was timid (format v1 vs v2) @@ -201,15 +295,20 @@ separability (100k rows, v1 → v2): ## 4. "Never make a correct app worse" (runtime design comparison) -GenClass with an always-passive model, compared against observe mode: +GenClass in heal mode with an always-passive model, compared against observe mode (v1: `realapps/README.md`, Pilot, +"Interference: 4 of 198 clean runs changed"; v2: `packages/runtime/CHANGELOG.md`, 0.1.0-alpha.1): -| Runtime | Real apps, clean runs changed | Conduit (React/Redux RealWorld) | Demos (guard, zero actions) | +| Runtime | Real apps (66), clean runs changed | Conduit (React/Redux RealWorld) | Demos (guard, zero actions) | |---|---|---|---| -| v1 (store-write holds) | 4/198 (2 apps) | **never renders its home feed** | board 11 card jump-backs, editor 37 overwritten keystrokes, checkout 24 reset quantities, search clean latency 6 → 389 ms | -| **v2 (network-boundary decisions)** | **0/396** | identical, 0/30 | holds only at delivery; clean typeahead makes 0 model calls and holds for 0 ms | +| v1 (store-write holds) | 4/198 (2 apps; 66 apps × 3 seeds) | **never renders its home feed** | board 11 card jump-backs, editor 37 overwritten keystrokes, checkout 24 reset quantities, search clean latency 6 → 389 ms | +| **v2 (network-boundary decisions)** | **0/396** (66 apps × 6 seeds) | identical, 0/30 | holds only at delivery; clean typeahead makes 0 model calls and holds for 0 ms | Chaos runs under v2: 3/198 changed. A request held about 25 ms drew a different simulated network outcome (open -item). +item). The v2 runs used the runtime as of 0.1.0-alpha.1 (2026-10-08) and were not re-run on later versions. They +compare final page text (inputs and alerts excluded) and server state, not request timing or store contents; the +baseline is observe mode, not running without GenClass. realapps' own sweeps on the same `situation-v2` runtime (tag +`6e5e86e`), before its production run, found 0 of 132 and 0 of 256 clean runs changed (`realapps/README.md`, +"Production on `situation-v2`"). §16 has the later compatibility matrix, which compares every mode with no GenClass. ## 5. Demos @@ -252,8 +351,8 @@ production names are minified) and `er` (minified `MenuBrowser`). The nine exist (12 per mode over 27 trials: failure 9, request 3; detections 9), since each of its page loads sees at most one response per request kind that changes state. New scenario `order-poll-reorder-2` (another device removes a line, the guest removes the other while the next ticket poll is answered 3 s late): **3 / 3 delivery decisions with discovery, 0 / 3 -with `autoState: false`**, all `expected` (correct: the app drops stale polls; no bug in any mode). Overhead: see the -package README "Costs" (+7 KB first-load JS, walk p95 under 2 ms at 4× throttling, +0.5 MB heap, no new long tasks). +with `autoState: false`**, all `expected` (correct: the app drops stale polls; no bug in any mode). Overhead: §17 +(+7 KB first-load JS, walk p95 under 2 ms at 4× throttling, +0.5 MB heap, no new long tasks). ### Flow expectations (0.3.0, opt-in: `learn.expectations`): not yet measured @@ -281,8 +380,8 @@ run 38455195053 on branch `wave1/expectations`, 2026-10-13, after the review fix | board | 57% | 63% | 77% | 0 | | decisions | 83% | 97% | 97% | 0 | -Guard executed 0 actions. Heal took 156 actions, which fixed nothing. These numbers are re-measured with the -trained v2 model once it exists. +Guard executed 0 actions. Heal took 156 actions, which fixed nothing. The trained model, 0.2.0, was measured on the +same six demos on 2026-10-09 (the first table in this section). ## 6. Data volume @@ -295,7 +394,7 @@ trained v2 model once it exists. | Sim on-policy round a (model r17-v2a acting in the sim; DAgger) | **1,248,131** (448k at shipping gate, 672k at explore gate 0.5) | done | | Sim on-policy round b (r17-v2b with its own gates, v2.3) | **1,094,953** | done | | Sim v2 unlabeled (for teacher labelling) | **51,272,078** | done (~21k rows/s per node) | -| Real-browser v2 gold (158 apps, 40+ stacks) | **616,437** (+523k unlabeled) | done; determinism 3,030/3,030, interference 0/256 | +| Real-browser v2 gold (128 apps: 96 in `v2c1..4`, plus 32 in the `v2d` top-up, wave 4) | **616,437** (+523k unlabeled) | done; determinism 3,030/3,030 (96 apps × 30 seeds) and 640/640 (128 apps), interference 0/256 | | Real-browser v2.3 top-up (relation-learner fixes) | **221,186** (real total 837,623) | done; real inconsistency triggers 18.3% → 2.6% of decisions, genuine share 10% → 40% | | Real-app eval set (unambiguous) | **16,600** (clean-benign 4,000, benign-salient 4,000, duplicate 4,000, genuine break 3,085, stale 1,515) | done | @@ -311,7 +410,7 @@ trained v2 model once it exists. | 10-08 00:35 | Auto-shutdown disabled with the owner's OK. Quota raised to 2,048 vCPU; nodes c12–c23 added. | | 10-08 02:00–04:00 | Separability analysis led to the S1/S2 label fixes and the F1–F9 facts. Runtime batches 4–5 (network-boundary decisions). | | 10-08 ~04:00 | `situation-v2` frozen. v2 generation launched: ~10M gold + ~50M unlabeled in the sim, ~495k real-browser rows. | -| next | 150M teacher on v2 gold, teacher labels on unlabeled rows, distil R17/R32, DAgger rounds (SIM `--on-policy`), EVAL. | +| 10-08 06:10–18:12 | v2 rounds (§1, `training/EVAL.md`): r17-v2a at 06:10; r17-v2b at 07:10, its gates refit on on-policy dev data at 10:30 and published as `@genclass/runtime-model@0.1.0` at 13:38 (npm); the 150M teacher came out weaker than r17-v2b, so nothing was distilled; on-policy (DAgger) rounds a and b in the sim; r17-v2c; the head-to-head of v2c, v2d and v2dT at 11:45; at 16:30 the refit with the real-app certification set chose r17-v2dT, published as `@genclass/runtime-model@0.2.0` at 18:12 (npm). | Compute: Azure CPU nodes (F80 and D64 sizes); the day-by-day record is kept internally. @@ -346,6 +445,14 @@ if either default case reaches 300 KB. A real page sends far fewer decisions to so these are upper bounds for one tab, not typical volumes. **Superseded for 0.3.0 by §10:** detections are no longer sampled or capped, so the "every one a detection, defaults" case now sends every record. +**Hot-path timings** (`packages/runtime/test/review-perf.test.ts`, run in CI, reported, not blocking; budgets: a +keystroke write under 1 ms, an async write under 2 ms, a Redux-style dispatch under 1 ms (2 ms from async work), a +settled point under 16 ms). The last recorded run is from a shared VM on 2026-10-07, on the runtime just before +0.1.0-alpha.1 (commit `bac4409`, `packages/runtime/STATUS.md` at that commit), and has not been re-recorded since: a +keystroke write to a store holding a 5,000-item array 0.22 ms, an async write 0.14 ms, a Redux-style dispatch on +5,000 entities 0.71 ms (0.70 ms from async work), a settled point 0.3 ms (+2.3 ms with an unchanged 5,000-item +adapter store). + ## 9. Install footprint (first-load JavaScript, CI) `packages/runtime/test/bundle.test.ts` bundles the built `dist/` the way an app's bundler would (esbuild, minified, @@ -367,6 +474,8 @@ adapters, their name table and the fan-out are absent from both first loads (ass needs a smaller first load or a new limit. The 0.3.0-beta.0 sizes and limits are in §10. The Sentry, PostHog and Datadog adapters measured here were lazy chunks, outside both first loads; they are not shipped (§10, last rows). + + ## 10. Release 0.3.0-beta.0: first load and telemetry volume (CI) Branch `release/0.3.0` (wave2/accounts-base = main 88df703 + wave1/projects, then wave1/defaults-footprint, @@ -390,7 +499,7 @@ walk, which missed the two shared chunks (2.74 + 0.19 KB). | + sink adapters, sinks by name (`53b04f2`, run 38462380432; the Sentry, PostHog and Datadog adapters in it were removed before release and never shipped, last rows) | 102.94 KB in 3 files | 94.79 KB in 3 files | 117.96 KB | | + flow expectations, `src/learn/flows.ts` lazy (`fcefedb`, run 38463736575) | 106.16 KB in 4 files | 98.13 KB in 4 files | 126.03 KB | | + even sampling draw (`c2f0b54`, run 38464240236) | 106.23 KB | 98.20 KB | 126.11 KB | -| **0.3.0-beta.0** (`bff4a71`, run 38465242176) | **106.24 KB** (100.26 + 3.05 + 2.74 + 0.19) | **98.21 KB** (92.24 + 3.05 + 2.74 + 0.19) | **126.12 KB** (369.4 KB minified) | +| version bump to 0.3.0-beta.0 (`bff4a71`, run 38465242176; not the published build, which is the visual-checks row below) | 106.24 KB (100.26 + 3.05 + 2.74 + 0.19) | 98.21 KB (92.24 + 3.05 + 2.74 + 0.19) | 126.12 KB (369.4 KB minified) | | + telemetry 429 back-off, no sampling settings in `session` (`7c7c6d6`, run 38477010836) | **106.59 KB** (100.62 + 3.05 + 2.74 + 0.19) | **98.57 KB** (92.59 + 3.05 + 2.74 + 0.19) | **126.45 KB** (370.5 KB minified) | | + detection tiers, deferred decisions, review fixes (wave2/detect-observe `dd20233`, run 38482297879; §11) | 107.56 KB | 99.59 KB | 127.89 KB | | **release candidate**: + main (accounts and the CLI commands: no browser code) and wave2/dash-detect (the worker only) (`18e89f0`, run 38485063587) | **107.56 KB** (101.59 + 3.05 + 2.74 + 0.19) | **99.59 KB** (93.62 + 3.05 + 2.74 + 0.19) | **127.89 KB** (374.9 KB minified) | @@ -398,7 +507,7 @@ walk, which missed the two shared chunks (2.74 + 0.19 KB). | + no vendor sink adapters: the Sentry, PostHog and Datadog adapters removed (never shipped), one console warning for those names in a page config (wave2/no-vendor-sinks `31ddc98`, run 38491077597) | **107.73 KB** (101.75 + 3.05 + 2.74 + 0.19) | **99.61 KB** (93.64 + 3.05 + 2.74 + 0.19) | **128.01 KB** (375.2 KB minified) | | + that warning reworded to claim no paid feature: "Sentry, PostHog and Datadog forwarding is not in the free runtime" (`eea042b`, run 38494904699) | **107.71 KB** (101.74 + 3.05 + 2.74 + 0.19) | **99.61 KB** (93.64 + 3.05 + 2.74 + 0.19) | **128.00 KB** (375.1 KB minified) | | limit in `bundle.test.ts` (measured at `bff4a71` + 1.5 KB, owner-approved for 0.3.0; not re-set: the release candidate was 0.13 / 0.10 KB under it, and with the vendor-name warning `/auto` is 0.03 KB under it) | 107.74 KB | 99.71 KB | none | -| + measured visual checks' first-load stub: wave2/visual `6ae209f` merged (`release/0.3.0-visual`, merge `2c7014f`, PR head `be8cb7c`, run 38676978903; §13) | **108.25 KB** (102.28 + 3.05 + 2.74 + 0.19) | **100.13 KB** (94.15 + 3.05 + 2.74 + 0.19) | **128.56 KB** (376.5 KB minified) | +| + measured visual checks' first-load stub: wave2/visual `6ae209f` merged (`release/0.3.0-visual`, merge `2c7014f`, PR head `be8cb7c`, run 38676978903; §13) = the published 0.3.0-beta.0 (tag `v0.3.0-beta.0`, `a86c928`; the only later `src/` change there is a comment) | **108.25 KB** (102.28 + 3.05 + 2.74 + 0.19) | **100.13 KB** (94.15 + 3.05 + 2.74 + 0.19) | **128.56 KB** (376.5 KB minified) | | limit in `bundle.test.ts` for 0.3.0-beta.0, **owner-approved 2026-10-15** (the measured sizes + 0.05 / 0.07 KB) | 108.30 KB | 100.20 KB | none | | + interaction analytics' first-load stub (§15; limits owner-approved 2026-10-18) | 108.53 KB | 100.40 KB | not measured | | + review of the mode-raise fix: `normalizeRoute`, the `mode` validation and its warning, the sample bucket's `act@` form, `readWindowConfig`'s mode check (branch `security/advisory-mode-raise`, 0.3.0-beta.5; local, 2026-10-27, `test/first-load-size.mjs` and gzip -9 against main `49c8c03`'s dist: 108.36 / 100.28 / 128.55 KB) | **108.63 KB** (+0.27) | **100.46 KB** (+0.17) | **128.83 KB** (387.4 KB minified, +0.28) | @@ -417,7 +526,8 @@ to the main entry, measured by `test/first-load-size.mjs` against `release/0.3.0 `visual-browser` job (107.711 to 108.248 KB, 99.614 to 100.126 KB; the job fails at 0.6 KB of growth). The checks themselves are one lazy file (`dist/visual/index.js`, about 35 KB gzip), outside every first load; the script tag loads it by URL, and the script-tag file grows by 0.56 KB. Both entries are over the 107.74 / 99.71 KB limits, so -`bundle.test.ts` carries 108.30 / 100.20 KB on `release/0.3.0-visual` until the owner decides. +`bundle.test.ts` carried 108.30 / 100.20 KB for 0.3.0-beta.0; the owner approved these limits on 2026-10-15 (the +table above). The interaction-analytics stub later raised them to 108.6 / 100.48 KB (§15). The flow expectations were +7.3 KB on `/auto` when `src/learn/flows.ts` was in the first load (106.5 KB vs 0.2.2's 99.17 KB on wave1/expectations, run 38455195053). Imported lazily, and only with `learn.expectations: true`, they @@ -436,7 +546,7 @@ routine records per page load): |---|---|---|---| | `738a05b` (38461782967) | 242.4 KB, 106 requests | 2,064.7 KB, 361 requests | 2,064.7 KB, 361 requests | | `c2f0b54` (38464240236), the even draw below | 249.0 KB, 162 requests | 2,064.7 KB, 361 requests | 2,064.7 KB, 361 requests | -| `bff4a71` (38465242176), 0.3.0-beta.0 (a longer version string in every envelope) | 250.3 KB, 163 requests | 2,067.1 KB, 361 requests | 2,067.1 KB, 361 requests | +| `bff4a71` (38465242176), the version bump to 0.3.0-beta.0 (a longer version string in every envelope) | 250.3 KB, 163 requests | 2,067.1 KB, 361 requests | 2,067.1 KB, 361 requests | | **`7c7c6d6` (38477010836)**: 429 back-off; `session` without `decisionSample` / `maxDecisions` (about 40 bytes once per page load, below the 0.1 KB rounding) | **250.3 KB, 163 requests** | **2,067.1 KB, 361 requests** | 2,067.1 KB, 361 requests | | **release candidate `18e89f0` (38485063587)**: with wave2/detect-observe (§11) | **251.9 KB, 165 requests** | **2,068.5 KB, 361 requests** | 2,068.5 KB, 361 requests | | release candidate `f363e92` (38485289326), docs only | 250.9 KB, 157 requests | 2,068.5 KB, 361 requests | 2,068.5 KB, 361 requests | @@ -463,6 +573,8 @@ limit in production is not measured. **Not measured here:** browser timings, the install suite (`test/install/run-all.sh`, VM only), the demos' clean-trial gate for the flow expectations (§5), and any number with the model. + + ## 11. Observe-mode detection in 0.3.0: the "possible" tier and deferred situations Branch `wave2/detect-observe` (from `release/0.3.0`), 2026-10-13. Two kinds of change: a lower reporting tier in @@ -531,7 +643,7 @@ likely reports (they are capped at the possible tier, below). **What CI measured on this branch** (GitHub Actions `ci.yml`, run 38471973596, commit `8501528` on `wave2/detect-observe`; no model involved, so no recall or false-detection rate applies to these): -| | 0.3.0-beta.0 (§10, run 38465242176) | with this change (run 38471973596) | limit (unchanged) | +| | 0.3.0-beta.0 version bump (§10, run 38465242176) | with this change (run 38471973596) | limit (unchanged) | |---|---|---|---| | first load, `/auto` | 106.24 KB | 107.66 KB (101.69 + 3.05 + 2.74 + 0.19) | 107.74 KB | | first load, main entry `GenClass.init({})` | 98.21 KB | 99.69 KB (93.72 + 3.05 + 2.74 + 0.19) | 99.71 KB | @@ -565,7 +677,8 @@ no recall or false-detection rate applies to these numbers. **Deploy order for the aggregates** (no deploy from this branch): the telemetry-worker dashboard (`telemetry-worker/src/dashboard.ts`, `possible_detections`) and the maintainers' admin aggregates (`site/src/admin.js`, `possible_detections` and recent rows of kind `possible`; the live demo counter in -`site/public/live.js` skips them) both count `tier: "possible"` apart from detections. Both must be deployed before a +`site/public/live.js` skips them; since 2026-10-18 `site/` is in the private repository `genclass-dev/genclass-site`) +both count `tier: "possible"` apart from detections. Both must be deployed before a 0.3.0 runtime reports to them; an older aggregate counts every possible detection, deferred ones included, as a detection, and the tiers cannot be separated afterwards (the admin `daily` metric has no tier). @@ -576,6 +689,7 @@ preload, plus hidden-tab trials with background pollers) that would show how man how many false ones come with them, which is what lifting the possible-tier cap on deferred decisions needs; the likely reports the three fixes add on correct apps (compat, observe, slow WASM and slow-body scenarios); and guard/heal FIR with `model.unloadAfterIdleMs`. + ## 12. Dashboard counts and the possible-issues tier (telemetry-worker, 2026-10-13) Branch `wave2/dash-detect` (merged with `release/0.3.0`). The worker takes each page load's totals (page loads, @@ -613,8 +727,10 @@ records that arrived. Routine records are sampled (1 in 4, at most 200 per page non-routine decisions; the session event no longer carries the sampling settings (§10), so the worker cannot reweight, and the page now labels them as records ("average of N decision records received"). -Not measured: anything on production data. No released runtime sends a token yet (`0.3.0` will), so no real page -load reaches a dashboard, and no released runtime sends `tier`. +Not measured: anything on production data. When this was measured no released runtime sent a token or `tier`, so no +real page load reached a dashboard; 0.3.0-beta.0 (npm `beta`) is the first release that sends both. + + ## 13. Measured visual checks (v1; CI on GitHub runners, Chromium, 2026-10-13, branch `wave2/visual`) @@ -627,13 +743,15 @@ checks work as designed. Section 14 has the bench on the demos, templates, compa | planted faults found on the fixture pages | 35 of 35 (every planted fault, on its element, 23 kinds) | `visual.spec.ts`, 24 tests passed | | findings on clean fixture pages (false reports) | 0 on the clean page at 1280 x 800 and at 390 x 844 (phone), 0 on the light variant of the dark-mode page, 0 on a 10,000-element page, 0 besides the planted fault on every other page | same | | runtime fixes on fixture pages | text colour, `pointer-events` and scroll lock applied, verified, undone; 0 CSP violations from the owned rule under `style-src 'self'` | same | -| first-load growth against main (minified + gzip) | first build, against an older main: `/auto` +486 B (99.649 KB), main entry +431 B (92.118 KB); limit 512 B. After the review, against main of 2026-10-13 (app tokens, `protect()`; CI run 38497342999): `/auto` +582 B (101.244 KB, over main's 101 KB ceiling), main entry +524 B (93.699 KB of 94 KB). The narrow development-host rule and the hidden-tab gate account for about 100 B. Ceilings raised on the branch to 101.5 KB and 0.6 KB growth; the owner approved the 0.6 KB growth gate with the 0.3.0-beta.0 limits (108.30 / 100.20 KB) on 2026-10-15. Merged into 0.3.0-beta.0 (`release/0.3.0-visual`, CI run 38676978903), against `release/0.3.0`: `/auto` +550 B (108.248 KB), main entry +524 B (100.126 KB); limits 108.30 / 100.20 KB in `bundle.test.ts`, also pending (§10) | `test/first-load-size.mjs` in the `visual-browser` job, `test/bundle.test.ts` | +| first-load growth against main (minified + gzip) | first build, against an older main: `/auto` +486 B (99.649 KB), main entry +431 B (92.118 KB); limit 512 B. After the review, against main of 2026-10-13 (app tokens, `protect()`; CI run 38497342999): `/auto` +582 B (101.244 KB, over main's 101 KB ceiling), main entry +524 B (93.699 KB of 94 KB). The narrow development-host rule and the hidden-tab gate account for about 100 B. Ceilings raised on the branch to 101.5 KB and 0.6 KB growth; the owner approved the 0.6 KB growth gate with the 0.3.0-beta.0 limits (108.30 / 100.20 KB) on 2026-10-15. Merged into 0.3.0-beta.0 (`release/0.3.0-visual`, CI run 38676978903), against `release/0.3.0`: `/auto` +550 B (108.248 KB), main entry +524 B (100.126 KB); limits 108.30 / 100.20 KB for 0.3.0-beta.0, approved by the owner on 2026-10-15 (§10); raised to 108.6 / 100.48 KB on 2026-10-18 for interaction analytics (§15) | `test/first-load-size.mjs` in the `visual-browser` job, `test/bundle.test.ts` | | lazy chunk `dist/visual/index.js` | about 35 KB gzip (97 KB minified) | esbuild, minified | | a 10,000-element page | 2 passes, 72.9 to 127.8 ms of check time in all (two runs), longest slice 8.5 ms (budget 8 ms per slice), no partial pass; after the review: 106.7 ms, longest slice 8.4 ms | `big.html` | | the same page with hover, clicks and scrolling (after the review) | 137.1 ms of check time in all, longest slice 8.7 ms, no long task (50 ms or more) on the page once the checks ran | `big.html`, CI run 38497342999 | | 5,000 stylesheet rules with `var()` (after the review) | 49 ms of check time, longest slice 10.9 ms, no long task | `big-css.html`, same run | | correct pages a review found reported as broken (after the review) | 0 findings on each: dark mode from the `color-scheme` meta tag and a dark-only page (both schemes), white text over `pointer-events: none` overlay, canvas, video and a full-screen particles canvas, a dialog that scrolls inside, a web-component app, a full-page section scroller, a busy shield during a 6 s request; a fix undone when its background changed; a scroll lock held since load reported and not removed | `visual.spec.ts`, 44 tests passed | + + ## 14. Visual bench: false reports, recall and cost (GitHub runners, Chromium 153, 2026-10-13, branch `wave2/visual`) Measured checks, not model results. [`bench/visual`](../../bench/visual/README.md) runs the checks' engine @@ -793,21 +911,99 @@ longest slice at most budget + 4 ms (12 ms): p95 8.9 ms on local pages, 57.1 ms code change before run 4 (popup triggers for two interaction checks) did not touch cost, so run 4 is the outlier, not an improvement. + + ## 15. Interaction analytics: first-load cost (train VM, 2026-10-18, branch `feat/interaction-analytics`) `test/first-load-size.mjs` against the merge base with main (`e22fc24`), minified + gzip as an app's bundler ships them, measured on the train VM (not CI): -| entry | main | with the analytics stub | growth | +| entry | merge base `e22fc24` | with the analytics stub | growth | |---|---|---|---| | `/auto` | 108.248 KB | 108.529 KB | 288 B | | main entry (`GenClass.init({})`) | 100.126 KB | 100.395 KB | 275 B | -`test/bundle.test.ts` measures the same build at 108.53 / 100.39 KB (script tag 128.83 KB). The -recording itself is `dist/analytics/index.js` (12.5 KB minified, not gzipped), loaded only when an app turns +`test/bundle.test.ts` measures the same build at 108.53 / 100.39 KB (script tag 128.83 KB); the comment in that test +rounds the main entry's 100.395 KB to 100.40 KB. The merge base predates the WASM warm-up fix (`042c813`, which +changes `src/runtime.ts` and `src/model/backend.ts`) that reached `main` before this branch did, so current `main` is +not measured here. The recording itself is `dist/analytics/index.js` (12.5 KB minified, not gzipped), loaded only when an app turns analytics on. No detection numbers: analytics never feed a decision, so recall and FIR do not change. -## 16. Coding agents: explore and replay on a real page (train VM, 2026-10-20, branch `feat/agent-loop`) +The branch raised the `bundle.test.ts` limits to 108.6 / 100.48 KB, approved by the owner on 2026-10-18, and was +merged into `main` the same day (pull request #53). Not released yet: it is in the CHANGELOG's unreleased +0.3.0-beta.1. Design: [ANALYTICS.md](ANALYTICS.md). + + + +## 16. Compatibility matrix (summary) + +Does adding GenClass break an app? This is a summary of [`compat/RESULTS.md`](../../compat/RESULTS.md), which a script +generates from a run on 2026-10-10 (5,850 trials, 0 incomplete) and which is not edited by hand; that page has every +cell, every finding and the method. No model accuracy is reported there: it counts bugs, actions and findings on +idiomatic apps. + +Setup: `@genclass/runtime` packed from commit `d04eff3` (version string 0.1.0-beta.4, never published: the +0.1.0-beta.4 code plus the "Model ready" log fix; it does not contain 0.2.0's audit trail, protect presets, hold-budget +ceilings and invariant fixes (`e84ddc7`, committed on another branch) and predates 0.2.1's first-request hold and +telemetry defaults (`5a90f15`), which were not run through the matrix, nor was any later release (0.2.2, +0.3.0-beta.0); the version note in `compat/RESULTS.md` calls it the code released as 0.2.0, which the +git history does not support: `e84ddc7` is not an ancestor of `d04eff3`), model 0.2.0 (`genclass-runtime-r17` 2.0.0-rc4t) on +WASM at `balanced`, headless Chromium 153, telemetry off. 7 apps (React 19 + Vite 8, Next.js 16 App Router, Vue 3 + +Vite 8, SvelteKit 3, Angular 22, Solid 1.9 + Vite 8, plain HTML with the CDN script tag), 15 data layers, 8 scenarios +(a and b carry a latent app bug; c to h are correct apps), 10 seeds per cell, in observe, guard and heal, each run +paired with a run without GenClass on the same seed. + +| measured | result | +|---|---| +| cells (app × data layer × scenario × mode) left as correct as without GenClass on every seed | 350 of 351 | +| bugs introduced | 0 in 3,510 mode runs | +| non-passive actions on correct apps (scenarios c to h, guard and heal) | 0 in 1,760 runs | +| observe findings on correct apps (scenarios c to h) | 319 in 880 runs: 270 report a failure the scenario injects on purpose (an HTTP 500, going offline), which the app handles; 49 are false (`request:overload` ×45 and `request:duplicate` ×4, all in scenario g, polling and fan-out) | +| latent app bugs (a: stale typeahead; b: double submit) | shown in 250 of 290 runs without GenClass; guard left 249 and heal 240 of the same runs buggy; the 10 fixed by an action were all heal `block` on one data layer (Angular with XMLHttpRequest, double submit) | +| runs where an app's bug did not show with GenClass although it took no action | 4 (GenClass's own work shifted the timing of the app's race) | +| determinism control (two runs without GenClass per seed) | 1,170 of 1,170 pairs identical | +| automatic state discovery | React state, the Redux store and Zustand with devtools `enabled: true` found in 240 of 240 runs each; Zustand's devtools middleware in a production build: 0 of 240 (it turns itself off there) | + +The one ✗ is observe on Next.js 16 with SWR 2, scenario d (two overlapping optimistic updates, one rolled back), seed +7: without GenClass the screen ended wrong in both runs, with GenClass it ended right, and GenClass took no action. So +observe mode is not byte-identical to running without GenClass on an app with its own race: it can change which side +of the race wins. Limits (from that page): small apps, one machine, headless Chromium only, WASM inference, production +builds, few seeds per cell. + +## 17. Browser cost on a pilot app (2026-10-09 and 2026-10-10) + +What GenClass costs a real page: memory, download, main-thread time and the state-discovery walk, in one production +app (the pilot app of §5, a Next.js 16 ordering site) with Playwright Chromium at a 390×844 phone viewport. These +tables were first recorded in the package README's "Costs" section and are kept here as the record; no model accuracy +is involved, so no recall or FIR applies. Not measured: other apps, real phones, Firefox and Safari. + +**Runtime 0.1.0-beta.3 with model 0.2.0 (2026-10-09).** Production build, 4× CPU throttling, medians of 5 runs, +observe mode; the same build with and without `?genclass=off`: + +| | with GenClass | +|---|---| +| JavaScript on every page (first load, gzip) | +97 KB with 0.1.0-beta.3 (today's first load: §10) | +| Main thread: load event, LCP, long tasks | no measurable change; main JS heap +0.4 MB | +| Memory: renderer process (holds the model worker) | +190 to 280 MB RSS while the model is loaded (cold 448 vs 169 MB, warm 361 vs 171 MB) | +| First visit download, WASM | 12.7 MB after the load event, as recorded then: q8 model 9.45 MB, ONNX Runtime wasm 3.07 MB brotli, tokenizer 207 KB. The READMEs round this to about 13 MB; with the 0.2.0 q8 file at its full size, 10.16 MB (§2), the same sum is about 13.4 MB. WebGPU devices: the 13.6 MB fp16 model plus the larger ONNX Runtime build | +| Later visits | about 1 KB (`model.json` revalidated); the rest comes from Cache Storage, checked with sha256 | +| Model load | WASM (1 thread): 0.9 to 1.3 s cold, 0.4 s warm. WebGPU (Apple M-series): 12 s cold, 2 s warm | +| App build output | 0.1.0-beta.3: +41 MB of unused ONNX Runtime files. 0.2.0: none | + +**Automatic state discovery, runtime 0.1.0-beta.4 (never published; its changes shipped in 0.2.0), 2026-10-10.** The pilot app with only the one +line, against the same build with `autoState: false`; Playwright Chromium, 390×844, observe mode, telemetry off: + +| | cost | +|---|---| +| JavaScript (first load, bytes on the wire) | +7 KB on `/menu` (discovery's share of the `/auto` entry; `GenClass.init()` setups do not load it) | +| React commit walk, 4× CPU throttling | p50 0.2 ms, p95 1.8 ms, max 2.1 ms per commit including recording the change; total about 5 ms over the `/order` visit | +| React commit walk, no throttling, 54 bench trials (about 2,000 commits) | p50 below 0.1 ms (the timer's resolution), p95 0.8 to 0.9 ms, max 3.8 ms | +| Main-thread JS heap (after GC, 4× throttling) | 5.7 MB vs 5.2 MB without discovery | +| Long tasks (> 50 ms) | unchanged (1 per run, 65 to 70 ms, with and without) | + + + +## 18. Coding agents: explore and replay on a real page (train VM, 2026-10-20, branch `feat/agent-loop`) [AGENT.md](AGENT.md) section 7. `packages/mcp/test/e2e/race.e2e.mjs` on the train VM: Node 22, headless Chromium through Playwright 1.63.0, `@genclass/runtime` and `@genclass/mcp` built from the branch, `GenClass.init({ model: false @@ -836,7 +1032,7 @@ gzip; limits 108.6 / 100.48 KB): the agent link is its own entry (`dist/agent.js load; the XHR fix adds about 10 bytes. Tests: 36 in `test/agent.test.ts`, 57 in `packages/mcp`; the runtime suite passes (1,101 tests, 14 model-parity skips); `review-perf` 4 of 4. -## 17. Coding agents on a real library issue: redux-toolkit#5139 (local, 2026-10-25, Chromium through Playwright 1.63.0) +## 19. Coding agents on a real library issue: redux-toolkit#5139 (local, 2026-10-25, Chromium through Playwright 1.63.0) [docs/adoption/hunts/rtk-5139.md](../adoption/hunts/rtk-5139.md). The agent link and `@genclass/mcp` (both built from the repository) recording a standalone Redux Toolkit 2.13.0 reproduction: an optimistic `updateQueryData` patch @@ -857,7 +1053,9 @@ none from the control. The Playwright test `genclass_explain` generated for the (the witness depends on the click's timing, which a delivery-order replay does not recreate). One case, not a recall figure. -## 18. The hunt harness: explore and replay against apps it does not own (cloud session, 2026-10-25, branch `claude/optimistic-feynman-8lbnyg`) + + +## 20. The hunt harness: explore and replay against apps it does not own (cloud session, 2026-10-25, branch `claude/optimistic-feynman-8lbnyg`) [bench/hunt](../../bench/hunt/README.md) runs the same loop against an app it does not own: an init script runs the runtime's script-tag build inline before the page's first script (and removes the element), imports the agent link @@ -879,7 +1077,7 @@ orders that delivered at least one response after the answer to a later request. | the harness's own DOM oracle | differs in 5 of 6 | same in 6 of 6 | same | same | same | | wall time (8 or 9 page loads, the test run included) | 22 s | 17 s | 15 s | 15 s | 17 s | -The same counts as section 16, through the harness. One page and three planted bugs: a check that the harness and the +The same counts as section 18, through the harness. One page and three planted bugs: a check that the harness and the injection do not change what GenClass finds, not a recall figure. False reports on the fixed variants: 0 of 6 (search); the fixed form gives no verdict rather than a false "no race". @@ -910,7 +1108,7 @@ or nothing was reordered, the document rewrite (redirect targets never rewritten mangled), a generated-test precondition that failed on about 7 of 30 loads, findings lost on steps that navigate, and session files outside `.gitignore`. -## 19. Coding agents on a real app: Gitea's markdown preview (cloud container, 2026-10-25, adoption hunt) +## 21. Coding agents on a real app: Gitea's markdown preview (cloud container, 2026-10-25, adoption hunt) [adoption/hunts/gitea.md](../adoption/hunts/gitea.md). Gitea `main` at `ff68b39` (same code as v28.2.0), built from source with SQLite; GenClass from `main` at `d77ffd8` (runtime 0.3.0-beta.1, mcp 0.1.0), injected from outside through @@ -933,7 +1131,7 @@ seeded priorities of its two request keys favour the older one for seeds 1 to 8) change, a dropped update rather than a reordering, was reported as a race in 5 of 6 at 300 ms but not at 600 or 900 ms, where the baselines lost the update too. One app, three surfaces of one bug class: not a recall figure. -## 20. Coding agents: explore's false reports before and after (cloud session, 2026-10-25, branch `claude/great-dijkstra-oa9emj`) +## 22. Coding agents: explore's false reports before and after (cloud session, 2026-10-25, branch `claude/great-dijkstra-oa9emj`) [AGENT.md](AGENT.md) sections 2 and 5.1. The Strapi race hunt (`docs/adoption/hunts/strapi.md` on the `claude/dazzling-cray-8tqzkn` branch) found a real race on the Strapi admin and also showed `genclass_explore` @@ -1060,7 +1258,7 @@ reads; for double-save it leaves out the notifications and blanks the digits of The two small apps were written by the author of the rules they measure (a mechanism check on the libraries' documented patterns, not an independent recall figure). One Strapi, one seed, one machine; seeds 1 to 6 only. -## 21. Issue 94: `request-stalled` on the demos, and the first-load cost of `summary().requests` (local, 2026-10-25, branch `issue-94`) +## 23. Issue 94: `request-stalled` on the demos, and the first-load cost of `summary().requests` (local, 2026-10-25, branch `issue-94`) **`request-stalled` (VISUAL.md 3.24), default threshold 15 s.** `bench/stall/run.ts`, headless Chromium through Playwright 1.63.0 on a Mac, the demos built against this branch's runtime (`DEMOS_OUT_DIR=... node @@ -1100,7 +1298,7 @@ sets when the app reads a non-JSON answer with `response.json()`. The request co snapshot and `request-stalled` are in the measured checks' chunk. A first version that counted in the first load cost +583 / +556 B and needed the owner-approved limits raised (`test/bundle.test.ts`), so it was moved; CI's build measures about 0.2 KB more than this machine for both branches. -## 22. Measured visual checks `ad-heavy` and `text-warped` (development Mac, Chromium 153 through Playwright 1.63.0, 2026-10-25, branch `strange-sites`) +## 24. Measured visual checks `ad-heavy` and `text-warped` (development Mac, Chromium 153 through Playwright 1.63.0, 2026-10-25, branch `strange-sites`) Measured checks, not model results (VISUAL.md sections 3.25 and 3.26). Run on a 16 GB development Mac, not on GitHub runners, so the cost figures are this machine's and only the comparison with main's engine on the same machine, diff --git a/docs/runtime/SINKS.md b/docs/runtime/SINKS.md new file mode 100644 index 00000000..a0bcae6f --- /dev/null +++ b/docs/runtime/SINKS.md @@ -0,0 +1,130 @@ +# Sinks: GenClass findings in your own logs and endpoints + +How to send GenClass's detections, interventions and other records to your own side: the `console-json` and +`webhook` adapters, sinks by name in the zero-code setups, a sink of your own, what a record holds and what the webhook +sends. For `@genclass/runtime` 0.3.0 (`0.3.0-beta.0` on the npm `beta` dist-tag). The option reference is in +[API.md](API.md#see-it-where-you-already-look) and [OPTIONS-SPEC.md](OPTIONS-SPEC.md) (§4.11). + +## Which versions have sinks? + +The `sinks` option, with a function or a `{ send }` object of your own, exists since `0.1.0-beta.1`. The ready-made +`console-json` and `webhook` adapters, sinks by name (`data-sink`, `sink=`, `init --sink`, names in +`GENCLASS_CONFIG.sinks`), the `possible` option and `check` records are `0.3.0` and later. `0.2.x` ignores `data-sink` +and `sink=`, stops at `init --sink` with "unknown option", has no `@genclass/runtime/sinks` exports, and prints "a sink +failed" for a sink name in `GENCLASS_CONFIG.sinks`. Pin `0.3.0-beta.0` or later in the script tag to use the adapters. + +## Why use a sink? + +In production the `[GenClass]` console lines print in your users' browsers, not yours. Sinks send every detection, +intervention, undo, breaker trip and runtime error to your own side as well: one JSON line per record on the console +for a log shipper (`console-json`), batched POSTs to an endpoint you own (`webhook`), or a function or `{ send }` +object of yours. The records go only where you point them. The console default does not change. Sinks are separate +from GenClass's anonymous diagnostics ([TELEMETRY.md](../../packages/runtime/TELEMETRY.md)): turning those off does +not turn a sink off, and a sink sends nothing to GenClass. + +## How do I add a sink without code? + +With `/auto` or the script tag you write no adapter code yourself: + +```bash +npx @genclass/runtime@beta init --sink webhook:/genclass # or console-json; repeat it or comma-separate +``` + +```html + + + +``` + +In a project with an entry file `init` writes the names to its option file; on a plain HTML page it sets `data-sink`, +in an Astro layout `sink=` in the meta line. A name loads its adapter on the first record, never before. A `` +tag only takes a same-origin webhook URL, checked as the browser resolves it (against the page and +`document.baseURI`); a cross-origin one is ignored with a console warning, so put a collector on another origin in +`data-sink` or `window.GENCLASS_CONFIG`. A webhook URL is made absolute once, at init, and only that endpoint (origin +and path) is excluded from what GenClass observes. Add `:noevidence` to `console-json` +(`--sink console-json:noevidence`) to print the structured fields without the evidence text. + +## How do I add a sink in code? + +Pass the adapters to `GenClass.init`: + +```ts +import { GenClass } from "@genclass/runtime"; +import { consoleJson } from "@genclass/runtime/sinks/console-json"; +import { webhook } from "@genclass/runtime/sinks/webhook"; + +GenClass.init({ + sinks: [ + consoleJson(), // console.log('{"source":"genclass","schema":1,"kind":"detection",...}') for log shippers + webhook({ url: "/genclass" }), // batched POSTs of { schema: "genclass-sink/1", version, records } to your endpoint + ], +}); +``` + +`@genclass/runtime/sinks` exports both. Neither adds a dependency or ever throws into your app. Both forward every +record kind except the visual checks' `check` records (below), the per-session `summary` record included, with the +redacted evidence unless you pass `evidence: false`; the webhook caps repeats of one bug at 10 per page load +(`maxPerSignature`, `0` for no cap). Both take `kinds`, `sampleRate`, `possible` and `evidence`. A sink's +`sampleRate` below 1 can drop detections. + +A sink of your own is a function or a `{ send }` object. A plain function gets likely detections and every other +record kind except `check`, without evidence. A `{ send }` object takes `kinds`, `sampleRate`, `possible` and +`evidence` (default false, unlike the adapters): + +```ts +GenClass.init({ + sinks: [ + { send: (r) => navigator.sendBeacon("/genclass", JSON.stringify(r)), + kinds: ["intervention", "undo", "breaker", "summary"] }, + { send: (r) => myLogger.info("genclass", r), sampleRate: 0.2 }, + ], +}); +``` + +Observe mode's *possible* detections reach a sink only with `possible: true`; the measured visual checks only when +its `kinds` lists `"check"`. + +## What does a sink receive? + +The record the runtime built and nothing else: `kind`, `route` (the page path with id-like segments and segments +holding an `@`, such as an e-mail address, as `:id`, as telemetry sends it from 0.3.0), `diagnosis` and its +probability `p`, `action`, `tier`, `trigger`, `mode`, `label`, `reason`, `changed` (state paths), `sessionId` and your +session `tags`, plus the evidence: the console line, the subject and the timeline, with URL query and fragment values +replaced by `…`. Timeline lines can hold short fragments of app data, as the console does (a store value summary, an +error message, text typed into an ordinary input; secret-named fields are `[redacted]`). `evidence: false` on an +adapter sends the structured fields only. A sink never gets the full situation text the model read, request or +response bodies, headers or cookies. + +## What does the webhook send? + +The webhook POSTs exactly those records as `{ schema: "genclass-sink/1", version, records }`, by +`navigator.sendBeacon` (which GenClass never wraps, and which survives page exit), else `fetch` with `keepalive`; the +body is JSON sent as `text/plain` (no preflight), so parse it as JSON. One POST carries the records GenClass hands +over in one idle-time round, at most 50 records and 32,000 bytes of them, and each round sends at most one: the +browser allows 64 KiB of beacons in flight per page, shared with yours (your analytics on page exit, say), so GenClass +never takes more than about half. What does not fit waits for the next round, and on page hide one last batch goes +(the session summary first) and the rest is dropped. Pass a `clock` and `flushMs` to batch over a time window +instead. `headers` switches it to `fetch`; then list the URL in `requests.ignore` (the zero-code forms do that for +you). + +## Can GenClass send findings to Sentry, PostHog or Datadog? + +Not from the runtime: it has no adapter for them. For an app in a genclass.dev account, on any plan, Free included +(the `forwarding` entitlement in [`billing/plans.json`](../../billing/plans.json)), the GenClass collector forwards +each likely detection to them (or to Slack, or a signed webhook) once you turn it on in the dashboard +([genclass.dev/pricing](https://genclass.dev/pricing), [DASHBOARD.md](DASHBOARD.md)). Possible detections are not +forwarded. That path uses the anonymous diagnostics, so it needs telemetry on, a token and the app in a GenClass +account: nothing is forwarded for apps without an account. A forwarded record holds the problem type, confidence, +trigger, page route, versions and time, never page text ([TELEMETRY.md](../../packages/runtime/TELEMETRY.md)). The +same diagnosis on the same route goes at most once per UTC hour, and an app sends at most 60 records an hour +(`HOURLY_CAP` in the collector's `telemetry-worker/src/forward.ts`). + +A page config that still names `sentry`, `posthog` or `datadog` gets one console warning, and nothing is loaded or +sent for those names; `init --sink sentry` stops with a usage error that says the same. + +## What does a sink need from a Content-Security-Policy? + +A webhook on another origin needs that origin in `connect-src` (a same-origin path such as `webhook:/genclass` needs +nothing). With the script tag, `data-sink` (or `sink=` / `GENCLASS_CONFIG.sinks`) loads `dist/sinks/.js` and +the chunk the adapters share from the tag's own CDN path, so allow `https://cdn.jsdelivr.net` in `script-src`, or +self-host this package's `dist/` and point `data-base` at it. With `/auto` the adapters are chunks of your own bundle. diff --git a/docs/runtime/THREAT-MODEL.md b/docs/runtime/THREAT-MODEL.md index b035adbf..05537aaa 100644 --- a/docs/runtime/THREAT-MODEL.md +++ b/docs/runtime/THREAT-MODEL.md @@ -1,11 +1,20 @@ -# Threat model +# GenClass threat model What could go wrong, or be made to go wrong, when `@genclass/runtime` runs in a web app; what the code does about it; -and what remains. It covers the runtime as published (`packages/runtime`), its default model package and the telemetry -collector. Companion documents: [INTERCEPTION.md](../../packages/runtime/INTERCEPTION.md) (the full list of what -GenClass wraps and what each action does), [SECURITY.md](../../SECURITY.md) (reporting), +and what remains. It covers the runtime on `main` (version string 0.3.0-beta.1; on 2026-10-24 npm `latest` was 0.2.2 +and `beta` was 0.3.0-beta.0), its default model package +(`@genclass/runtime-model` 0.2.0), the telemetry collector and the CLI's account commands. Items marked 0.3.0 are in +the 0.3.0 line (0.3.0-beta.0 and later) and not in 0.2.x. Companion documents: +[INTERCEPTION.md](../../packages/runtime/INTERCEPTION.md) (the full list of what GenClass wraps and what each action +does), [SECURITY.md](../../SECURITY.md) (reporting), [TELEMETRY.md](../../packages/runtime/TELEMETRY.md) (what leaves the page). Code references are to -`packages/runtime/src/`. Reviewed against commit 2f89fb5 plus the SAFETY changes of 2026-10-10. +`packages/runtime/src/` unless stated otherwise. T1 to T9 were first reviewed against commit 2f89fb5 plus the SAFETY +changes of 2026-10-10; T4, T5 and T8 were later updated for 0.2.2 and 0.3.0, and T5 also covers app tokens (0.3.0). +The gate guidance (T2, T3, residual 2, the recommended setup), the telemetry and URL wording (the trust table's +telemetry row, T5's forwarding paragraph, T6's residual, T8's local and private hosts, residuals 5, 8 and 10) and +T10 to T12 were written on 2026-10-22 against the code on `main` at commit 902f529: the measured visual checks +(0.3.0, T10), interaction analytics (0.3.0-beta.1, T11) and the CLI account commands (0.3.0, T12). T13, the +coding-agent bridge (0.3.0-beta.1), was written on 2026-10-22 with that feature (commit d20d64e). ## Contents @@ -20,7 +29,10 @@ GenClass wraps and what each action does), [SECURITY.md](../../SECURITY.md) (rep - [T7. A wrong action on a correct app](#t7-a-wrong-action-on-a-correct-app) - [T8. Configuration by URL, storage and meta tags](#t8-configuration-by-url-storage-and-meta-tags) - [T9. Cached and shared responses](#t9-cached-and-shared-responses) -- [T10. The coding-agent bridge](#t10-the-coding-agent-bridge) +- [T10. Measured visual checks and their fixes](#t10-measured-visual-checks-and-their-fixes) +- [T11. Interaction analytics](#t11-interaction-analytics) +- [T12. CLI account credentials](#t12-cli-account-credentials) +- [T13. The coding-agent bridge](#t13-the-coding-agent-bridge) - [Residual risks, in one list](#residual-risks-in-one-list) - [Recommended production setup](#recommended-production-setup) @@ -43,7 +55,7 @@ response contents), the app's availability and latency, and the integrity of the | the page's own scripts | same trust as GenClass | any script in the origin can already read and change everything GenClass can | | responses from the app's servers and third parties | data, untrusted as instructions | they become situation text the model reads (T3) | | the model files (`model.baseUrl`) and onnxruntime-web (`model.ortWasmPaths`) | integrity-checked, not trusted blindly | default: jsDelivr, from the npm packages (T2) | -| the telemetry collector | receives redacted diagnostics only | default on for `GenClass.init` in a browser (T5) | +| the telemetry collector | receives redacted diagnostics only | default on for `GenClass.init` in a browser on a public host; off on localhost and private networks; forwards likely detections, never page text, to third-party tools only where an account owner turned forwarding on for the app (T5) | | the developer's configuration | trusted | but parts of it can come from the URL, `localStorage` or meta tags (T8) | | learned state in storage (`genclass:learn`; the learned flows when `learn.expectations` is on (opt-in), transition profiles only with `learn.persist`, EXPECTATIONS.md §4) | data, checked field by field and against the learner's bounds on load, discarded on a format or runtime-version change | holds request signatures, routes and field names (id-like segments and keys normalised; other path segments and map keys, such as `/u/jane.doe@example.com` or a store keyed by email, as they are), counts and hashes of action labels, no element text or values; a script that rewrites the flows can make expectation reports wrong or silent, and, when profiles are stored, change which `transition` situations reach the model, which any same-origin script could already do (T1) | @@ -96,8 +108,13 @@ gate, or run hostile code. **Residual.** - `model.json` itself is not pinned by a hash the app supplies: whoever controls the model URL can publish a new card - with matching hashes. The model's meta gate (`meta.json` → `gate`) sets the thresholds unless the app sets - `policy.thresholds`, so a hostile model could lower them. Pin `policy.thresholds` and self-host if that matters. + with matching hashes. The model's meta gate (`meta.json` → `gate`) sets the thresholds or margins unless the app + sets `policy.thresholds`, and it always sets the gate kind and, for a gain gate, `tauGain`, the scale of the + estimated gain, which `policy.thresholds` does not pin ([decide/policy.ts](../../packages/runtime/src/decide/policy.ts) + `effectiveGates`, `gainOf`). Whoever controls the model files also controls its probabilities, so a hostile model + can make GenClass act more often whatever the app pins. Self-host the model (`model.baseUrl`) if that matters; a + pin of `policy.thresholds` in the model's gate kind ([Recommended production setup](#recommended-production-setup)) + only keeps a new model release from changing the margins. - onnxruntime-web's wasm is verified by its version tag only, not a sha256; an `integrity` attribute on the script tag covers that one file, not the modules, worker and wasm it loads later (dynamic `import()` and module workers take no SRI). @@ -118,9 +135,12 @@ submitted) contains text meant to push the model towards an action, the way prom itself ([situation/build.ts](../../packages/runtime/src/situation/build.ts) `builtinUnavailable`): no retry of a non-idempotent request without an idempotency key (`repeatUnsafe`), cache answers only for GET, nothing for a cross-origin or protected subject. -- Precision-first gate: the summed calibrated probability must reach 0.9 (guard) / 0.8 (heal) by default and the - diagnosis must not be `expected`; action limits (60 per minute, 10 per subject per minute, 200 per session); the - breaker drops the session to `observe` after 2 undos or 3 errors that follow actions. +- Precision-first gate: the model's own gate must clear (with model 0.2.0, a gain margin per tier and trigger from + the active profile; only for a model that ships no gate, a summed probability of 0.9 for guard and 0.8 for heal + at `balanced`, 0.05 higher at `cautious`), and the top diagnosis must not be `expected` + ([decide/policy.ts](../../packages/runtime/src/decide/policy.ts) `gate`); action limits (60 per minute, 10 per + subject per minute, 200 per session); the breaker drops the session to `observe` after 2 undos or 3 errors that + follow actions. - Every action is recorded with what it changed and, where possible, can be undone. **Residual.** An adversarial input can, in principle, flip a classifier. The impact is bounded by the mode and the @@ -180,6 +200,18 @@ rows with values of their choosing. The dashboard inserts every such string as t never as markup, and a token grants no read access (the dashboard needs the private link or the owning account). A dashboard is therefore a diagnostic view, not an audit record. +Forwarding (on every genclass.dev plan, Free included; `billing/plans.json`): once the owner of an app in a +genclass.dev account turns it on, the collector sends each new likely detection that app reports (never a possible +one) to the destinations the owner configured: Sentry, PostHog, Datadog, Slack or an HTTPS webhook signed with +HMAC-SHA256. A forwarded record carries the diagnosis, confidence, trigger, route, runtime and model versions, the +time and a dashboard link, never page text, even when the app opted in to situation text. The same diagnosis on the +same route is forwarded at most once per UTC hour, and at most 60 records per app per hour. Destination settings +(keys, webhook URLs) are stored only as AES-GCM ciphertext under the collector's `INTEGRATIONS_KEY` secret, bound to +the app and the destination kind, and shown masked +([telemetry-worker/src/forward.ts](../../telemetry-worker/src/forward.ts)). Nothing is forwarded for an app without +an account or with telemetry off, and nothing changes in the app. The route in a forwarded record is the telemetry +route above, so personal data in an ordinary path segment reaches the destination too. + ## T6. State discovery reading application state **Threat.** `autoState` hooks React's DevTools interface and the Redux DevTools globals and so sees component and store @@ -193,8 +225,9 @@ bounded (1 ms, 20,000 fibers, 48 component stores). A real React DevTools or Red (theirs run first). `autoState: false` (or `{ react: false }`, `{ redux: false }`, `{ zustand: false }`) turns it off; `?genclass=off` installs nothing. -**Residual.** Discovered state appears in situation text, and so in telemetry (marked `autoState`), unless redacted or -turned off. Redux stores created through the DevTools compose or enhancer are full adapters: in guard/heal their writes +**Residual.** Discovered state appears in situation text, and so in telemetry only when the app opts in to situation +text (`telemetry: { include: { situation: true } }`), unless redacted or turned off; by default a decision event +carries only the flag `autoState: true`. Redux stores created through the DevTools compose or enhancer are full adapters: in guard/heal their writes can be acted on like a registered store's (INTERCEPTION.md). The DevTools hook GenClass installs stays (inert) after `destroy()`, since React keeps a reference to it. @@ -287,7 +320,81 @@ requests ([observe/cache.ts](../../packages/runtime/src/observe/cache.ts), [obse without a page load, a heal-mode `serve_cached` could answer a GET with the previous user's cached response. Protect account and sign-in endpoints (`protectPreset("auth")`), `policy.deny: ["serve_cached"]`, or reload on sign-out. -## T10. The coding-agent bridge +## T10. Measured visual checks and their fixes + +**Threat.** The measured visual checks (0.3.0, [VISUAL.md](VISUAL.md)) read the page's layout, computed styles and +stylesheets, and their opt-in fixes write to the page. A wrong fix could restyle correct content, a check could slow +a heavy page, and findings could carry page text out of the page. + +**Mitigations.** + +- On by default only on local development URLs (http(s) on `localhost`, `127.0.0.1`, `[::1]` or `*.localhost` with + a port, and `*.test` names; `devHost` in [visual/start.ts](../../packages/runtime/src/visual/start.ts), narrower + than the local or private hosts of T8) and when the devtools overlay mounts (`visual: "local"`), and report-only. + Fixes need `visual: { on: true, fix: true }` and an effective mode of guard or heal; a URL never turns them on, and in + `observe` the checks write nothing to the page (VISUAL.md section 1). +- A fix is either one rule in an adopted stylesheet GenClass owns, keyed by a `data-genclass-fix` attribute, whose + values are colours the chunk generates, never page text, so the rule cannot inject CSS; or the removal of one + leaked inline `pointer-events: none` or `overflow: hidden` on `html` or `body`. Each fix is re-measured 50 ms later + and undone if it does not clear its threshold; it is undone on `setMode("observe")`, the breaker, `pause()`, + `disable()` and `destroy()`, dropped when the app changes the element, and capped at 50 per page. A `style-src` + Content-Security-Policy violation during a fix removes all fixes for the session (VISUAL.md sections 11 and 12). +- The checks never feed situations, facts, the event log, the gate or the model, and never fire `detect` + (AGENTS.md rule 2 exception). Page reads run in idle slices of 8 ms, and nothing is measured in a hidden tab. +- Nothing about the checks is sent in telemetry (VISUAL.md section 10). A sink that lists `"check"` in its `kinds` + receives the check kind, its severity, whether it was fixed and the normalised route; with `evidence: true` also + `message`, `target` and `selector` ([visual/bridge.ts](../../packages/runtime/src/visual/bridge.ts)). Otherwise + selectors, element text, colours and URLs stay in the page. + +**Residual.** + +- `?genclass-visual=on` lets a link turn on report-only checks on any host for one page load (idle-time CPU and + console lines), unless the app sets `visual: false`. +- A read that makes the browser lay out a very heavy page can take 50 ms or more, and the runtime's `perf` observer + records that long task as an event the model can read (VISUAL.md section 1). +- Findings name elements by short visible text (`target`, such as `paragraph "Ship faster"`), shown in the console, + the devtools overlay and `rt.checks()` (T1). + +## T11. Interaction analytics + +**Threat.** Interaction analytics (0.3.0-beta.1, not on npm yet; [ANALYTICS.md](ANALYTICS.md)) send counts of page +views, engaged time, scroll depth and clicks per element key to the collector. The keys could identify a person or +carry personal data, and in some jurisdictions the collection needs the visitor's consent. + +**Mitigations.** Off by default: it needs `analytics: true` and the app's token (or a custom `telemetry.endpoint`), +and without them records nothing. Every telemetry opt-out and Global Privacy Control turn it off; +`rt.analytics.stop()` turns it off and drops the counts not sent yet. It sends counts per page view, never element +text, input values, mouse coordinates, the order of clicks, the query string, the referrer or a visitor id, and +writes nothing to cookies or storage. Element keys come from markup only, and each part read from an attribute is +dropped when it looks like data (an e-mail address, a URL, 4 or more digits in a row, 12 or more hex digits; +[analytics/index.ts](../../packages/runtime/src/analytics/index.ts) `keyOf`). Its listeners are passive and outside +the event log: nothing reaches a situation, a decision or the model. + +**Residual.** Element keys are built from markup (`data-genclass-name`, `data-testid`, ids, a link's same-site path +or another site's origin, class names). Values that look like data are skipped, but a readable name in an id, a test +id or a link path (`/u/jane-doe`) is sent as it is, and routes are sent as telemetry sends them (T5). Sites with +visitors in the EU, EEA or UK should start with analytics off and call `rt.analytics.start()` after the visitor +consents (ANALYTICS.md section 5). + +## T12. CLI account credentials + +**Threat.** `npx @genclass/runtime@beta login` (0.3.0; `npx @genclass/runtime login` once 0.3.0 is `latest`) keeps a +genclass.dev session on the developer's machine; whoever reads it can act as that session. + +**Mitigations.** The session token (`gcs_...`) is stored in `$XDG_CONFIG_HOME/genclass/credentials.json`, else +`~/.config/genclass/credentials.json` (Windows: `%APPDATA%\genclass\credentials.json`), with mode 600 in a directory +with mode 700, and is used only for the host it was made for +([bin/lib/account.mjs](../../packages/runtime/bin/lib/account.mjs)). It can read the account's apps, create apps in +it and add a private-link app to it. It expires after 90 days (`CLI_SESSION_MS` in `telemetry-worker/src/auth.ts`); +`logout` revokes it on genclass.dev and deletes the file, and the account page lists CLI sessions and revokes them. +The runtime in the browser never reads this file. The app token (`gc_...`) that `init` writes into the app is public +by design (T5). + +**Residual.** Anyone who can read the file (the same user, a backup, a copied home directory or container image) can +use the session until it expires or is revoked. On Windows the CLI sets no file mode, so the file has the user +profile's permissions. + +## T13. The coding-agent bridge `@genclass/mcp` runs a bridge on the developer's machine (127.0.0.1:47357) that pages with `connectAgent` (`@genclass/runtime/agent`, development builds) send their recording to, and that a coding agent reads through MCP @@ -324,36 +431,77 @@ tools ([AGENT.md](AGENT.md) section 6). ## Residual risks, in one list 1. Same-origin scripts have full access to GenClass and its data (as to the rest of the page). -2. The model card is not pinned by an app-supplied hash; the model's meta gate can lower thresholds unless - `policy.thresholds` is set; onnxruntime-web's wasm has no sha256 check; SRI covers only the script tag's own file. +2. The model card is not pinned by an app-supplied hash; the model's meta gate can lower thresholds or margins + (and, for a gain gate, raise `tauGain`, which `policy.thresholds` does not pin), and a hostile model controls its + own probabilities; onnxruntime-web's wasm has no sha256 check; SRI covers only the script tag's own file. 3. Adversarial data may flip the classifier within the offered actions of one subject (guard/heal only). 4. Up to 30 model evaluations per minute and one hold budget of latency per held subject can be forced (guard/heal). -5. Name-based redaction misses personal data in ordinarily named fields; with default telemetry, such data in situation - text leaves the page. +5. Name-based redaction misses personal data in ordinarily named fields; with situation text opted in + (`telemetry: { include: { situation: true } }`), such data leaves the page. 6. Discovered state is read and appears in situation text; DevTools-composed Redux stores are actionable. 7. In guard/heal, identical POSTs within 2 s may be coalesced; heal may block requests or roll back whole stores; a late revert may make a loop resend (bounded). -8. Meta tags in `` configure the zero-code entries (mode, same-origin model URLs); `debug: true` lets the URL raise the mode. +8. Meta tags in `` configure the zero-code entries (mode, same-origin model, onnxruntime-web and webhook sink + URLs); on a development host (http(s) on a loopback name with an explicit port, or `*.test`) `?genclass=` and + `localStorage.genclass` raise the mode, so a production app served that way (an Electron app with an embedded + server, a self-hosted local tool) can have its mode raised by a link (T8). 9. `serve_cached` ignores cookies (cross-user responses in a long-lived SPA session, heal only). 10. Telemetry and sink routes keep path segments that are neither id-like nor hold an `@` (usernames, names, search - terms in the path); app tokens are public, so anyone can post forged batches to an app's dashboard (T5). -11. The coding-agent bridge shows the agent text recorded from development pages; quoting and the server's - instructions reduce, but cannot remove, the chance that a model follows instructions inside that text (T10). + terms in the path), and forwarded records carry the same routes to the destinations an account owner turned on; + app tokens are public, so anyone can post forged batches to an app's dashboard (T5). +11. A link with `?genclass-visual=on` turns on report-only visual checks for one page load; a check on a very heavy + page can cause a long task the model can read; visual findings show short element text locally (T10). +12. With interaction analytics on, readable names in ids, test ids and link paths are sent as element keys; sites + with EU, EEA or UK visitors need consent first (T11). +13. The CLI's session file grants access to the genclass.dev account's apps to whoever can read it, until it + expires (90 days) or is revoked (T12). +14. The coding-agent bridge shows the agent text recorded from development pages; quoting and the server's + instructions reduce, but cannot remove, the chance that a model follows instructions inside that text (T13). ## Recommended production setup ```js import { GenClass, protectPreset } from "@genclass/runtime"; GenClass.init({ - mode: "observe", // explicit: URL parameters can then only lower it + mode: "observe", // explicit; on public hosts URL parameters only lower the mode (T8) requests: { protect: protectPreset("payments", "auth") }, // money and identity flows: never acted on - policy: { thresholds: { guard: 0.9, heal: 0.8 } }, // gate thresholds you control, not the model's meta + aggressiveness: "cautious", // guard/heal: the strictest action gate (see below) audit: { sink: (e) => myLogger.info("genclass", e) }, // keep the audit trail telemetry: false, // if your privacy review requires it model: { baseUrl: "/genclass-model/", ortWasmPaths: "/genclass-model/ort/" }, // self-hosted, with a CSP }); ``` +A production app served from a loopback name with an explicit port (an Electron app with an embedded server, a +self-hosted local tool; T8) should also set `routes: [{ match: "*", mode: "observe" }]` while it observes: there +`?genclass=` may raise the mode, and the route rule keeps every route at observe (route rules only lower the mode). An +intranet or VPN-only app no longer needs this: since `0.3.0-beta.2` and `0.2.3` a link cannot raise the mode there. + +In `observe` no action runs, so `aggressiveness: "cautious"` only raises the report gate from 0.90 to 0.95. With model +0.2.0 that lowers the problems reported as likely on held-out real apps from 75% / 30% / 6% to 72% / 28% / 0% +(duplicate / stale / broken; [model card](../../packages/runtime-model/MODEL_CARD.md), "Shipped profiles"). The +0.2.0 gate fit held false detections to at most 1% of clean real-app decisions and 2% of simulated `expected` rows at +`cautious`, against 2% / 3% at `balanced`; every profile met those limits on held-out test, and the exact rates are +not published (model card, "Observe-mode false detections"). Findings from 0.90 up to 0.95, likely at `balanced`, +become possible detections (that tier starts at 0.70 in both profiles): printed with `console.debug`, shown dimmed +in the devtools, counted in `summary().possibleDetections`, sent in telemetry as a `detect` event with +`tier: "possible"` while telemetry is on, and sent to a sink only when it sets `possible: true` +([API.md, Reports](API.md#reports-explain-and-undo); [TELEMETRY.md](../../packages/runtime/TELEMETRY.md)). Leave the line out while you observe if you want the default +reports, and add it when you move to guard. + +Pinning `policy.thresholds` fixes the margins only. The gate kind, `tauGain` and the probabilities still come from the +model files, so a changed or hostile model can still make GenClass act more often (T2). Self-host the model +(`model.baseUrl`) so its sha256-checked files change only when you change them. A pin then protects against a new +model release changing its profile's margins, not against a hostile model. Set it in the kind of gate the model +ships (`rt.gates().kind`). Model 0.2.0 ships a gain gate, so the values are margins in cost units. Its `cautious` +profile uses guard 6.0 and heal 5.0 by default, with other margins for some triggers +([model card](../../packages/runtime-model/MODEL_CARD.md), "Gain margins"). A pinned value replaces the model's margin for +every trigger of that tier: `policy: { thresholds: { guard: 6, heal: 8 } }` is at least as strict as `cautious` on +every trigger except `mutation`, which `cautious` never acts on. If a later model ships a probability gate, values +above 1 never pass, so such a pin fails safe. Do not set the probability values 0.9 and 0.8 with model 0.2.0: read +as margins, they are below even the `eager` profile's default margins of 3.0 and 2.5, so on most triggers the gate +would act more often than at `eager`. + Move to guard route by route once `rt.decisions()` and the audit trail show what it would do (`shadow: "guard"` records that without acting): `mode: "guard"` with `routes: [{ match: "/search*" }, { match: "*", mode: "observe" }]` (the first matching rule wins and rules can only lower the mode). diff --git a/docs/runtime/VISUAL.md b/docs/runtime/VISUAL.md index e7caa429..79f52c83 100644 --- a/docs/runtime/VISUAL.md +++ b/docs/runtime/VISUAL.md @@ -1,10 +1,17 @@ # GenClass visual checks: design (v1, with v2 and v3 outlined) -Status: design written 2026-10-13; **v1 implemented** the same day on branch `wave2/visual` -(`packages/runtime/src/visual/`), with the differences listed in [section 19](#19-v1-implementation-notes); **measured** -the same day by the bench of section 15 ([RESULTS.md section 14](RESULTS.md)), which tightened four checks; then a review -of the checks (section 19, "Review of 2026-10-13") found correct pages that looked broken and costs outside the slices, -and the code changed again (measured as run 5). The open owner decisions are in +GenClass measured visual checks find visual and CSS bugs that throw no error, such as unreadable text, broken images, +fonts and stylesheets, sideways scrolling on phones, and invisible layers that swallow clicks (23 kinds, section 3). +They are deterministic measurements with documented thresholds; the model is not involved and does not look at +pixels. In `@genclass/runtime` 0.3.0-beta.0 they are on by default on development hosts (`visual: "local"`), three +narrow runtime fixes run only when the app opts in and the mode is guard or heal, and the measurements are in +RESULTS.md, [section 13](RESULTS.md#visual-checks) and [section 14](RESULTS.md#visual-bench). + +Status: v1 shipped in 0.3.0-beta.0 (npm `beta`). Design written 2026-10-13 and implemented the same day on branch +`wave2/visual` (`packages/runtime/src/visual/`), with the differences listed in +[section 19](#19-v1-implementation-notes); measured the same day by the bench of section 15, which tightened four +checks; then a review of the checks (section 19, "Review of 2026-10-13") found correct pages that looked broken and +costs outside the slices, and the code changed again (measured as run 5). The open owner decisions are in [section 17](#17-owner-decisions-and-open-questions). [CONTRACT.md](CONTRACT.md) section 14, [API.md](API.md), [OPTIONS-SPEC.md](OPTIONS-SPEC.md) section 4.20 and the CHANGELOG have the matching entries. Two checks were added on 2026-10-25 at the owner's request for "strange sites": `ad-heavy` (section 3.25) and @@ -197,8 +204,12 @@ labelled "likely source rule", because rule order only approximates the cascade. ## 3. v1 checks -Only checks expected to be high precision are in v1. "Expected" is the honest word: no precision figure below is -measured yet. Each check must pass the gates in section 15 before it is on by default anywhere. +Only checks expected to be high precision are in v1. False reports on clean local pages and recall on planted faults +were measured after this was written (RESULTS.md sections 13 and 14). Precision on public sites was measured for runs 1 +to 4 (run 4: 8 findings in 96 loads, 5 confirmed real, 0 check errors, 3 unconfirmed) but not for the code after the +review (run 5's 6 findings were not looked at), and the planted faults are the checks' author's own forms, so +"expected" below still means expected. Each check must pass the gates in section 15 before it is on by default +anywhere. Severity: **high** means content or controls the visitor needs cannot be read, seen or used; **medium** means visibly wrong but usable; **low** means a defect with a small visible effect. @@ -735,7 +746,7 @@ button, sends a request with no timeout, and re-enables the button in `finally`; layer knows as a background schedule (long polling); a signature whose learned p95 latency is at least `stallMs` (an endpoint that usually takes that long); a host or path prefix in `visual.ignore`; elements inside `[data-genclass-ignore]` as the control. A streamed fetch is not pending once its headers arrive. The first request - of a long poll that has no history yet is not recognised (measured on the demos: RESULTS.md section 21). + of a long poll that has no history yet is not recognised (measured on the demos: RESULTS.md section 23). - Evidence: method, host, normalised path (no query), `pendingMs`, the control's state and description, when it became busy relative to the request. - Severity: high with a control still busy, medium without. @@ -1319,7 +1330,8 @@ packages/runtime/src/visual/ narrow development-host rule and the hidden-tab gate account for about 100 B of it. The owner decides the ceiling. Merged into 0.3.0-beta.0 (`release/0.3.0-visual`, CI run 38676978903, RESULTS.md sections 10 and 13): the stub adds 550 B to `/auto` (108.25 KB) and 524 B to the main entry (100.13 KB) against `release/0.3.0`, over its 107.74 / - 99.71 KB limits; the owner approved 108.30 / 100.20 KB for 0.3.0-beta.0 (2026-10-15), which `bundle.test.ts` carries. + 99.71 KB limits; the owner approved 108.30 / 100.20 KB for 0.3.0-beta.0 (2026-10-15), and `bundle.test.ts` carried + them until interaction analytics raised them to 108.6 / 100.48 KB on 2026-10-18 (RESULTS.md section 15). ## 14. Test plan @@ -1383,9 +1395,9 @@ Browser job (new, `visual-browser` in `.github/workflows/ci.yml`, in parallel wi ## 15. Measurement plan in CI **Status: built** (2026-10-13): [`bench/visual/`](../../bench/visual/README.md) and -`.github/workflows/visual-measure.yml`. The numbers are in [RESULTS.md section 14](RESULTS.md); what the bench changed -in the checks is in section 19. Where the bench differs from the plan below: the workflow is `visual-measure.yml`, -started by hand (and by pushes to `bench/visual/` on the visual branch), not weekly; the public sites' run logs and plain +`.github/workflows/visual-measure.yml`. The numbers are in [RESULTS.md section 14](RESULTS.md#visual-bench); what +the bench changed in the checks is in section 19. Where the bench differs from the plan below: the workflow is +`visual-measure.yml`, started by hand (and by pushes to `bench/visual/` on the visual branch), not weekly; the public sites' run logs and plain artifacts hold counts only (no site is named next to a finding or a kind: the repository and its runs are public), and their per-load results and crops are uploaded only encrypted with the `VISUAL_BENCH_KEY` secret (not at all without it, so their findings then count as unconfirmed); the public sites run without `bypassCSP` (Playwright's @@ -1408,7 +1420,7 @@ A workflow `visual-bench.yml` (manual dispatch and weekly, not on pull requests) Every finding on an unmodified page is a candidate false report. A committed `bench/visual/labels.json` (page, kind, selector) records a person's verdict, real or false; unlabelled findings are listed separately and never counted as correct. If a check finds a real issue in a demo, it is labelled and the demo is left alone. -- **Seeded faults for recall**: `bench/visual/faults.mjs` (a seeded RNG; `Math.random` is fine in harness code) +- **Seeded faults for recall**: `bench/visual/faults.ts` (a seeded RNG; `Math.random` is fine in harness code) picks visible candidates and applies each fault through a harness stylesheet plus a `data-bench-fault` marker: text colour equal to its resolved background, contrast 1.3:1, `pointer-events: none` left on `body`, `overflow: hidden` left on `body`, a transparent overlay over a button, a block with `min-width: 130vw`, an image routed to a 404, an @@ -1437,7 +1449,7 @@ A workflow `visual-bench.yml` (manual dispatch and weekly, not on pull requests) ## 16. Product wording -What to say once v1 ships (README and genclass.dev), in this voice: +What to say about v1, which shipped in 0.3.0-beta.0 (README and genclass.dev), in this voice: > GenClass also measures visual bugs that never throw an error: text you cannot read because it is the same colour as > its background, images, fonts and stylesheets that failed to load, pages that scroll sideways on a phone, invisible @@ -1476,14 +1488,15 @@ Owner decisions: 3. **Telemetry**: the fields in section 10 (counts per kind and normalised route), under rule 8. 4. **Sinks**: `"check"` records only for sinks that list it in `kinds`. 5. **Gates**: the numbers in section 15. Measured on 2026-10-13 (RESULTS.md section 14): the false-report and fix gates - are met; the 4x-throttling gate is not reliably met (pass 1 inside 100 ms on 78.1% of clean local pages in runs 1 - to 3 and 100% in run 4, against at least 90%, with no change to cost between runs 3 and 4); recall is below 0.7 for + are met in run 4; in run 5 the public-site false-report gate is not known (6 findings, none looked at); the + 4x-throttling gate is not reliably met (pass 1 inside 100 ms on 78.1% of clean local pages in runs 1 to 3 and 100% + in run 4, against at least 90%, with no change to cost between runs 3 and 4); recall is below 0.7 for `click-blocked` and `scroll-lock-stuck` (0.67, pages whose DOM changes all the time); the cost gates were not met on public sites (p95 pass time 97.8 ms and p95 longest slice 57.1 ms in run 4; 95.2 ms and 18.9 ms in run 5, after the stylesheet walk became a sliced step; 78.1% at 4x throttling in run 5). Whether kinds that miss a gate stay on by default on development hosts is open. -6. **CONTRACT.md**: the same rule 2 exception added to CONTRACT section 0 rule 1, and a new section for visual checks, - when implementation starts. +6. **CONTRACT.md**: done. CONTRACT section 0 rule 1 carries the same rule 2 exception and section 14 describes the + visual checks; both still say the owner confirms the wording. 7. **Public-site bench**: the list of about 40 sites and loading them weekly from GitHub runners. The list is `bench/visual/sites.json`; the workflow runs on manual dispatch and on pushes that change `bench/visual/` on the visual branch (runs 1 to 3 were push-triggered), with no schedule, until this is decided. Runs on 2026-10-13 @@ -1551,8 +1564,8 @@ What the code does where it differs from the design above (2026-10-13, branch `w - **Not built yet:** the explain step (likely source rule, 2.6), the "inserted content" cause tag of `layout-shift`, `genclass-runtime init --no-visual`, Firefox and WebKit in the browser job (it runs Chromium only), v2 and v3. -What the bench (section 15, [RESULTS.md section 14](RESULTS.md)) changed, 2026-10-13. Every finding on a public site was -looked at from its element crop; each check error led to an exclusion with a unit test in +What the bench (section 15, [RESULTS.md section 14](RESULTS.md#visual-bench)) changed, 2026-10-13. Every finding on +a public site was looked at from its element crop; each check error led to an exclusion with a unit test in `test/visual-checks.test.ts` ("bench regressions") and a fixture in `test/browser/visual/fixtures/regress-*.html` that reproduces the pattern, asserts that it happened, and asserts that nothing but the planted faults is reported: @@ -1643,6 +1656,6 @@ What the build does where sections 3.25 and 3.26 were not specific, and what the the first), which also describes three lines at `line-height: 0`. Self-overlap is now skipped under an ellipsis (section 3.26); a `line-height: 0` paragraph is still found. CI's Chromium on Linux then did the same for the last line of a `-webkit-line-clamp` box (the development Mac's did not), so a line clamp is skipped the same way. -- **Recall and false reports** are in RESULTS.md section 22 (the Chromium fixtures, and the demos and templates bench +- **Recall and false reports** are in RESULTS.md section 24 (the Chromium fixtures, and the demos and templates bench run on a development machine). The compat apps and the public sites were not run for these two kinds yet: they need the bench workflow (section 15), which loads public sites and is the owner's to start. diff --git a/llms.txt b/llms.txt new file mode 100644 index 00000000..c4aaa2b3 --- /dev/null +++ b/llms.txt @@ -0,0 +1,54 @@ +# GenClass Runtime + +> GenClass Runtime (`@genclass/runtime`) is an open-source (Apache-2.0) browser library that reports race conditions in web apps while they happen: stale responses that land over newer data, duplicate requests such as double submits, inconsistent state and failure storms. A small model runs on the user's device, in a Web Worker on WebGPU or WASM, and each report comes with its evidence. By default it only reports (`observe`): it never holds, delays or changes a request, a response or a state write. The opt-in `guard` mode may act on a few of them, and `heal` is experimental. + +This file indexes the Markdown sources in the GenClass-lib repository, for `@genclass/runtime` 0.2.2 (npm `latest`) and 0.3.0-beta.0 (`beta`) as of 2026-10-22, both with the model `@genclass/runtime-model@0.2.0`, and the unreleased work on `main`. It holds no numbers on purpose: every measured number, each recall next to its false rate, is in docs/runtime/RESULTS.md, so quote numbers from there. On public hosts it sends anonymous diagnostics about its own decisions to its maintainers by default (with the page's hostname and id-normalised path, and no page text unless the app opts in); TELEMETRY.md lists every field and every opt-out. The website keeps its own index at https://genclass.dev/llms.txt. + +## Start here + +- [README](https://raw.githubusercontent.com/genclass-dev/GenClass-lib/main/README.md): what GenClass Runtime is, the quick start, what it finds, how it compares with error trackers and tests, and the measured results +- [FAQ](https://raw.githubusercontent.com/genclass-dev/GenClass-lib/main/FAQ.md): short answers to common questions (race conditions in React, stale responses, double submits, what it sends, what it costs), each with a link to the details +- [Package README](https://raw.githubusercontent.com/genclass-dev/GenClass-lib/main/packages/runtime/README.md): the guide published on npm: install, modes, configuration, model quality, costs, privacy and known limitations +- [Docs index and glossary](https://raw.githubusercontent.com/genclass-dev/GenClass-lib/main/docs/runtime/README.md): which document answers which question, how current each one is, and the terms they use + +## Install and use + +- [INSTALL.md](https://raw.githubusercontent.com/genclass-dev/GenClass-lib/main/docs/runtime/INSTALL.md): every `init` and `remove` flag, the app token, the import, the script tag and the tested project setups +- [DASHBOARD.md](https://raw.githubusercontent.com/genclass-dev/GenClass-lib/main/docs/runtime/DASHBOARD.md): app dashboards on genclass.dev and the `login`, `connect`, `whoami`, `logout` and `dashboard` commands +- [SINKS.md](https://raw.githubusercontent.com/genclass-dev/GenClass-lib/main/docs/runtime/SINKS.md): sending reports to your own logging or endpoint, and the record format +- [API.md](https://raw.githubusercontent.com/genclass-dev/GenClass-lib/main/docs/runtime/API.md): every export, option, event, action and type +- [OPTIONS-SPEC.md](https://raw.githubusercontent.com/genclass-dev/GenClass-lib/main/docs/runtime/OPTIONS-SPEC.md): why each configuration option exists and how the options combine + +## Behaviour, privacy and security + +- [INTERCEPTION.md](https://raw.githubusercontent.com/genclass-dev/GenClass-lib/main/packages/runtime/INTERCEPTION.md): every browser API it wraps or listens to, what each mode may change, what it never does, and how to turn it off +- [TELEMETRY.md](https://raw.githubusercontent.com/genclass-dev/GenClass-lib/main/packages/runtime/TELEMETRY.md): what the default-on anonymous diagnostics contain, where they go, and every opt-out +- [PRIVACY.md](https://raw.githubusercontent.com/genclass-dev/GenClass-lib/main/PRIVACY.md): what genclass.dev collects and keeps +- [SECURITY.md](https://raw.githubusercontent.com/genclass-dev/GenClass-lib/main/SECURITY.md): how to report a vulnerability, supported versions, and how to check a published tarball +- [THREAT-MODEL.md](https://raw.githubusercontent.com/genclass-dev/GenClass-lib/main/docs/runtime/THREAT-MODEL.md): what could go wrong or be made to go wrong, with mitigations and residual risks + +## Measurements + +- [RESULTS.md](https://raw.githubusercontent.com/genclass-dev/GenClass-lib/main/docs/runtime/RESULTS.md): every measured number (detection, false interventions, demos, compatibility, first-load size, browser cost, visual checks), each section naming its model, runtime version and setup +- [compat/RESULTS.md](https://raw.githubusercontent.com/genclass-dev/GenClass-lib/main/compat/RESULTS.md): the framework compatibility matrix, generated by a script from a recorded run +- [Model card](https://raw.githubusercontent.com/genclass-dev/GenClass-lib/main/packages/runtime-model/MODEL_CARD.md): the default model, its training data, gate profiles, accuracy and limits + +## Guides + +- [How to fix race conditions in React data fetching](https://genclass.dev/guides/race-conditions-react): why `useEffect` fetches show stale data, and four fixes: an ignore flag, `AbortController`, request ids and the runtime +- [Out-of-order responses and stale search results](https://genclass.dev/guides/stale-responses-out-of-order): why debounce is not enough, cancelling or comparing before writing, and the `switchMap` pattern +- [How to prevent duplicate form submissions](https://genclass.dev/guides/prevent-duplicate-submissions): locking the UI, idempotency keys, deduplicating on the server and coalescing repeats at runtime + +## Optional + +- [ARCHITECTURE.md](https://raw.githubusercontent.com/genclass-dev/GenClass-lib/main/docs/runtime/ARCHITECTURE.md): how the parts fit together and why they are built this way +- [VISUAL.md](https://raw.githubusercontent.com/genclass-dev/GenClass-lib/main/docs/runtime/VISUAL.md): design of the measured visual and CSS checks, which are deterministic checks, not the model +- [EXPECTATIONS.md](https://raw.githubusercontent.com/genclass-dev/GenClass-lib/main/docs/runtime/EXPECTATIONS.md): design of the opt-in flow expectations, whose false-report rate is not measured yet +- [ANALYTICS.md](https://raw.githubusercontent.com/genclass-dev/GenClass-lib/main/docs/runtime/ANALYTICS.md): design of the opt-in interaction analytics, not yet in a published version +- [AGENT.md](https://raw.githubusercontent.com/genclass-dev/GenClass-lib/main/docs/runtime/AGENT.md): the coding-agent link (`@genclass/runtime/agent`, 0.3.0-beta.1) and the `@genclass/mcp` server, which record a page's run, replay one delivery order, explore others to find races, and generate a Playwright test; neither on npm yet +- [QUERY-CACHE-ADAPTERS.md](https://raw.githubusercontent.com/genclass-dev/GenClass-lib/main/docs/runtime/QUERY-CACHE-ADAPTERS.md): a proposal for TanStack Query and SWR adapters, not built +- [CONTRACT.md](https://raw.githubusercontent.com/genclass-dev/GenClass-lib/main/docs/runtime/CONTRACT.md): the binding rules for people changing the runtime and its training pipeline +- [CHANGELOG](https://raw.githubusercontent.com/genclass-dev/GenClass-lib/main/packages/runtime/CHANGELOG.md): what changed in each release of `@genclass/runtime` +- [GenClass for Playwright](https://raw.githubusercontent.com/genclass-dev/GenClass-lib/main/packages/playwright/README.md): `@genclass/playwright` 0.1.0, not on npm yet: a fixture that puts the runtime on every page under test with seeded network chaos and writes one race file per test, and a reporter that turns those files into one markdown report per pull request +- [EVAL.md](https://raw.githubusercontent.com/genclass-dev/GenClass-lib/main/training/EVAL.md): the raw model evaluation reports that RESULTS.md summarises +- [AGENTS.md](https://raw.githubusercontent.com/genclass-dev/GenClass-lib/main/AGENTS.md): the repository map and rules for coding agents working on GenClass itself diff --git a/packages/genclass-runtime/README.md b/packages/genclass-runtime/README.md index 8d17b535..937a6549 100644 --- a/packages/genclass-runtime/README.md +++ b/packages/genclass-runtime/README.md @@ -1,28 +1,66 @@ -# genclass-runtime +# genclass-runtime (deprecated) -Short alias for the [`@genclass/runtime`](https://www.npmjs.com/package/@genclass/runtime) CLI. +`genclass-runtime` is the deprecated, unscoped short name for the command line of +[`@genclass/runtime`](https://www.npmjs.com/package/@genclass/runtime). Use `@genclass/runtime` instead: this package +is no longer published (the release workflow publishes only `@genclass/runtime`), and its newest version on npm +installs an unsupported runtime. -```bash -npx @genclass/runtime@beta init -``` +GenClass Runtime (`@genclass/runtime`) is an open-source (Apache-2.0) browser library that reports race conditions +in web apps while they happen: stale responses that land over newer data, duplicate requests such as double +submits, inconsistent state and failure storms. A small model runs on the user's device, in a Web Worker on WebGPU +or WASM, and each report comes with its evidence. By default it only reports (`observe`): since `0.2.0` it never +holds, delays or changes a request, a response or a state write. The opt-in `guard` mode can prevent a few of these +failures, and `heal` is experimental. -Use that command, the newest line, for now: npm's `genclass-runtime` is still at `0.1.0-beta.3` (this alias is not -published by the release workflow yet), so `npx genclass-runtime init` runs that old CLI. Once this alias is published -at the newest version, `npx genclass-runtime init` does the same as `npx @genclass/runtime@beta init`: it finds your -framework, installs `@genclass/runtime`, adds one line to your entry file and shows you the diff first. From `0.3.0`, -once you approve the diff, it also creates your app's token and private dashboard link at genclass.dev (sending your `package.json` name), writes -the token into what it adds, and saves the link in `.genclass.local` (keep that link private; it is the only way to -open your dashboard). +## What should I run instead? ```bash -npx @genclass/runtime@beta init --token gc_... # use a token you already have (no network) -npx @genclass/runtime@beta init --no-token # no token, no network +npx @genclass/runtime@beta init # the newest line: 0.3.0-beta.4 on 2026-10-27 +npx @genclass/runtime@beta init --token gc_... # use a token you already have (no call to genclass.dev) +npx @genclass/runtime@beta init --no-token # no token, no call to genclass.dev ``` -Docs: [@genclass/runtime](https://www.npmjs.com/package/@genclass/runtime) (one-line setup, function-specific -`protect()`, your dashboard). +`@beta` is the newest line of `@genclass/runtime` (`0.3.0-beta.4` on 2026-10-27); the stable line, `0.2.2` (npm's +`latest`, `npx @genclass/runtime init`), is the exception. npm's `genclass-runtime` is still at `0.1.0-beta.3`, so +`npx genclass-runtime init` runs that old CLI unless the project already has `@genclass/runtime` installed (see +below). + +`init` finds your framework, shows you the diff first, then installs `@genclass/runtime` and adds one line to your +entry file. From `0.3.0`, once you approve the diff, it also creates your app's token and private dashboard link at +genclass.dev (sending your `package.json` name), writes the token into what it adds and saves the link in +`.genclass.local` (keep that link private: anyone with it sees your app's dashboard, and it is the only way to open +it). `--token gc_...` uses a token you already have and `--no-token` skips the token; neither contacts genclass.dev, +though the package manager still installs from its registry. Docs: +[`@genclass/runtime`](https://www.npmjs.com/package/@genclass/runtime). + +## What does this package install? + +The newest published version, `genclass-runtime@0.1.0-beta.3`, depends on `@genclass/runtime@0.1.0-beta.3`, which is +no longer supported ([SECURITY.md](https://github.com/genclass-dev/GenClass-lib/blob/main/SECURITY.md)). With its +defaults that version sends anonymous diagnostics from local hosts too, and they include the redacted situation text +the model read. Its `observe` mode can also delay a `fetch` whose `Request` or `Blob` body GenClass reads to identify +it, by a few microtasks to 100 ms (fixed in `0.2.0`). + +The command name itself is not going away: `@genclass/runtime` installs a command called `genclass-runtime`, so in a +project that has `@genclass/runtime` installed, `npx genclass-runtime ...` runs that installed version. Outside such +a project, `npx genclass-runtime` downloads this deprecated package. + +## Privacy + +`@genclass/runtime` sends **anonymous diagnostics by default** on public hosts: GenClass's decisions and counts, no +text from your page unless the app opts in, to the GenClass maintainers to improve the model; with an app token the +same diagnostics also feed your app's private dashboard. They are off on `localhost` and other local hosts from +`0.2.2`; earlier versions (`0.1.0-beta.3` to `0.2.1`) send from them too, and `0.1.0-beta.3` to `0.2.0` include the +redacted situation text by default. Opt out with `GenClass.init({ telemetry: false })`, `?genclass=no-telemetry`, +`localStorage.setItem("genclass.telemetry", "off")` or `init --no-telemetry` (which also skips the token); browsers +sending Global Privacy Control are never collected. Details: +[TELEMETRY.md](https://github.com/genclass-dev/GenClass-lib/blob/main/packages/runtime/TELEMETRY.md). -Privacy: since `0.1.0-beta.3` the installed runtime sends anonymous diagnostics (GenClass's decisions and -counts; no page text unless you opt in) to the GenClass maintainers by default. Opt out with `GenClass.init({ telemetry: false })` -or `?genclass=no-telemetry` (`init --no-telemetry` also skips the token); with a token the same diagnostics also feed -your app's private dashboard. Details in [TELEMETRY.md](https://github.com/genclass-dev/GenClass-lib/blob/main/packages/runtime/TELEMETRY.md). + diff --git a/packages/genclass-runtime/package.json b/packages/genclass-runtime/package.json index f6002048..0f995ec3 100644 --- a/packages/genclass-runtime/package.json +++ b/packages/genclass-runtime/package.json @@ -1,7 +1,7 @@ { "name": "genclass-runtime", "version": "0.3.0-beta.1", - "description": "Short alias for the @genclass/runtime CLI: npx genclass-runtime init sets up GenClass Runtime in your app.", + "description": "Deprecated: use @genclass/runtime (npx @genclass/runtime init). Former short name for the CLI of GenClass Runtime, a browser library that reports race conditions, stale responses and duplicate requests in web apps.", "type": "module", "license": "Apache-2.0", "repository": { @@ -10,6 +10,17 @@ "directory": "packages/genclass-runtime" }, "homepage": "https://genclass.dev", + "bugs": { + "url": "https://github.com/genclass-dev/GenClass-lib/issues" + }, + "keywords": [ + "genclass", + "cli", + "deprecated", + "race-condition", + "stale-response", + "duplicate-request" + ], "bin": { "genclass-runtime": "./cli.mjs" }, diff --git a/packages/mcp/CHANGELOG.md b/packages/mcp/CHANGELOG.md index 23b5cf72..59076910 100644 --- a/packages/mcp/CHANGELOG.md +++ b/packages/mcp/CHANGELOG.md @@ -69,7 +69,7 @@ Everything below ships first in `0.2.0`. proved fixed. Off unless set; nothing sent feeds a decision, a finding or a tool result. - **Explore stops reporting order-dependent bookkeeping as races** ([docs/runtime/AGENT.md](../../docs/runtime/AGENT.md) - section 5.1, [RESULTS.md](../../docs/runtime/RESULTS.md) section 20). A reordered run is a race only when what the + section 5.1, [RESULTS.md](../../docs/runtime/RESULTS.md) section 22). A reordered run is a race only when what the page shows, its form values, the requests it sent or its errors differ from the request-order baseline; store values that differ while the page is the same are listed as "Internal state only", and an overwrite a later response repaired as "Along the way only", neither with a finding. A `[redacted]` value equals any value. Scenarios are diff --git a/packages/playwright/CHANGELOG.md b/packages/playwright/CHANGELOG.md index daa16710..ec9a0b19 100644 --- a/packages/playwright/CHANGELOG.md +++ b/packages/playwright/CHANGELOG.md @@ -1,5 +1,9 @@ # Changelog +Changes to `@genclass/playwright`, the Playwright reporter for +[`@genclass/runtime`](https://www.npmjs.com/package/@genclass/runtime) race files. Version 0.1.0 is not on npm yet; +everything below is unreleased. + ## Unreleased - **The fixture**, `@genclass/playwright/fixture`: Playwright's `test` (`test.extend`) and `expect`, with a @@ -49,3 +53,8 @@ then whole tests, with a note saying what was left out; the counts include everything. - The "model was not ready" note is only for files that flagged nothing, and no longer says those runs could not flag anything (a file can carry detections with state `unloaded`). +- Docs: the README opens with what the package is and links `@genclass/runtime` on npm; the config example imports + `defineConfig`; the measured double-submit list names all 14 data layers and the runtime build the matrix ran; + the privacy note gives the versions that send diagnostics from local hosts (`0.1.0-beta.3` to `0.2.1`); links + outside the package are absolute, so they work on npm. `homepage` points at the README, and the package + description says the race files come from the GenClass Playwright fixture, which is not released yet. diff --git a/packages/playwright/README.md b/packages/playwright/README.md index 6961d77b..6cc24245 100644 --- a/packages/playwright/README.md +++ b/packages/playwright/README.md @@ -2,7 +2,11 @@ ### Explain your flaky e2e test: which response landed over newer data, which click submitted twice. -GenClass for Playwright, in two parts that share one file format: +`@genclass/playwright` is GenClass for Playwright: it puts +[`@genclass/runtime`](https://www.npmjs.com/package/@genclass/runtime) ([genclass.dev](https://genclass.dev)), the +open-source (Apache-2.0) browser library that reports race conditions in web apps while they happen, on every page +your tests open, and turns what it flagged into one pull-request report. This page covers `@genclass/playwright` +0.1.0, which is not on npm yet, and the `genclass-races/1` file format. Two parts that share that file format: - **The fixture** (`@genclass/playwright/fixture`) puts [`@genclass/runtime`](../runtime) on every page your tests open, before the app's first script, in observe mode with telemetry off. With `chaos` it slows and reorders the @@ -289,7 +293,7 @@ What it does so the comment is safe to post: ### What the comment looks like -This is the exact output for the unit-test fixture in [`test/fixtures/races`](test/fixtures/races): two tests, two +This is the exact output for the unit-test fixture in [`test/fixtures/races`](https://github.com/genclass-dev/GenClass-lib/tree/main/packages/playwright/test/fixtures/races): two tests, two seeds each, one signature flagged in both seeds and one in a single seed, both model detections (a run with `model: "load"`). The fixture is synthetic (the evidence lines are written for the test, not taken from a real run); `test/package.test.ts` fails if this block and the summariser drift apart. Each `#` of a request id below is @@ -419,13 +423,19 @@ The same with the GitHub CLI only (it updates the bot's comment that starts with ## What to expect from the model With `model: "load"`, the report passes on what the runtime's model flagged. That was measured on the framework -compatibility matrix ([`compat/RESULTS.md`](../../compat/RESULTS.md), run of 2026-10-10: headless Chromium, -10 seeds per cell, model `genclass-runtime-r17` 2.0.0-rc4t, the default of `@genclass/runtime-model` 0.2.0, WASM, -balanced profile, whose report threshold is 0.90 per [`docs/runtime/RESULTS.md`](../../docs/runtime/RESULTS.md)): +compatibility matrix +([`compat/RESULTS.md`](https://github.com/genclass-dev/GenClass-lib/blob/main/compat/RESULTS.md), run of +2026-10-10; headless Chromium, 10 seeds per cell, model `genclass-runtime-r17` 2.0.0-rc4t, the default of +`@genclass/runtime-model` 0.2.0, WASM, balanced profile, whose report threshold is 0.90). The runtime was +`@genclass/runtime` packed at commit `d04eff3` (version string 0.1.0-beta.4, never published): the 0.1.0-beta.4 code +plus the "Model ready" log fix. 0.2.0's audit trail, protect presets, hold-budget ceilings and invariant fixes, 0.2.1's +first-request hold and telemetry defaults and every later release were not run through the matrix +([`docs/runtime/RESULTS.md`](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/RESULTS.md) §16): - **Double submit** (scenario b, a create form clicked twice): 10 `request:duplicate` detections over 10 seeds in - each of the 14 data layers where the scenario applies (React state, TanStack Query, Zustand, Redux, Apollo, - Next.js, SWR, Pinia, Svelte stores, Angular fetch and XHR, Solid, plain fetch). + each of the 14 data layers where the scenario applies (React state, TanStack Query, Zustand with its devtools + middleware at the default and with `enabled: true`, Redux, Apollo, Next.js state, SWR, Pinia, Svelte stores, + Angular fetch, Angular XHR, Solid, plain fetch). - **Stale typeahead** (scenario a; the app showed stale results on 10 of 10 seeds without GenClass in 11 data layers): `delivery:stale` detections over 10 seeds were 2 (React state), 5 (Zustand with its devtools middleware set to `enabled: true`), 1 (Redux) and 2 (Next.js state), and 0 in the other 7 layers with the bug (Zustand with @@ -451,6 +461,17 @@ store field's name and the runtime's short summary of its value before and after fields whose name says they hold a secret are redacted by the runtime, as elsewhere). The comment you post contains the same, so read a report before posting it from a repository whose test data is private. +Without the fixture (the reporter alone, with `@genclass/runtime` installed in your app), the runtime under +test sends **anonymous diagnostics by default** on public hosts: GenClass's decisions and counts, +no text from your page unless the app opts in, to the GenClass maintainers to improve the model. They are off on +`localhost` and other local hosts (`*.localhost`, `*.local`, `*.test`, `127.0.0.1`, private networks) from +`@genclass/runtime` 0.2.2, which is where e2e suites usually run. Earlier versions (`0.1.0-beta.3` to `0.2.1`) send +from local hosts too, and `0.1.0-beta.3` to `0.2.0` include the redacted situation text by default. With those +versions, or with a test server on a non-local host name, pass `telemetry: false` to the runtime under test; the +other opt-outs are `?genclass=no-telemetry`, `localStorage.setItem("genclass.telemetry", "off")` and Global Privacy +Control. Details: +[`TELEMETRY.md`](https://github.com/genclass-dev/GenClass-lib/blob/main/packages/runtime/TELEMETRY.md). + ## Development From the repository root (CI runs these in `.github/workflows/playwright-reporter.yml`): diff --git a/packages/playwright/package.json b/packages/playwright/package.json index f3bbb4dc..ceb072c9 100644 --- a/packages/playwright/package.json +++ b/packages/playwright/package.json @@ -9,7 +9,7 @@ "url": "git+https://github.com/genclass-dev/GenClass-lib.git", "directory": "packages/playwright" }, - "homepage": "https://genclass.dev", + "homepage": "https://github.com/genclass-dev/GenClass-lib/tree/main/packages/playwright#readme", "bugs": { "url": "https://github.com/genclass-dev/GenClass-lib/issues" }, "keywords": [ "playwright", @@ -19,7 +19,12 @@ "stale-response", "duplicate-request", "genclass", - "e2e" + "e2e", + "reporter", + "test-reporter", + "github-actions", + "pull-request", + "ci" ], "main": "./dist/index.js", "types": "./dist/index.d.ts", diff --git a/packages/runtime-model/MODEL_CARD.md b/packages/runtime-model/MODEL_CARD.md index e84ab47f..36b8c4ec 100644 --- a/packages/runtime-model/MODEL_CARD.md +++ b/packages/runtime-model/MODEL_CARD.md @@ -1,17 +1,35 @@ # GenClass runtime model: model card -**`@genclass/runtime-model@0.2.0` ships `genclass-runtime-r17` 2.0.0-rc4t (training run `r17-v2dT`), with a gain -gate and three aggressiveness profiles in `meta.json`.** It reads the runtime's frozen situation format -`situation-v2`, which `@genclass/runtime` 0.1.0-alpha.1 and later render. The numbers below are copied from -`training/EVAL.md` and `docs/runtime/RESULTS.md` in the GenClass-lib repository; those files have the definitions -and full tables. +This card describes `@genclass/runtime-model@0.2.0`, the on-device model that +[`@genclass/runtime`](https://www.npmjs.com/package/@genclass/runtime) 0.1.0-beta.1 and later load by default to +diagnose stale responses, duplicate requests, inconsistent state and failure storms in web apps. **It ships +`genclass-runtime-r17` 2.0.0-rc4t (training run `r17-v2dT`), with a gain gate and three aggressiveness profiles in +`meta.json`.** It reads the runtime's frozen situation format: it was trained on `situation-v2` data plus the +`situation-v2.3` top-ups (simulated and real), and `@genclass/runtime` 0.1.0-beta.0 and later render +`situation-v2.3`. + +The numbers below are copied from +[`training/EVAL.md`](https://github.com/genclass-dev/GenClass-lib/blob/main/training/EVAL.md) and +[`docs/runtime/RESULTS.md`](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/RESULTS.md) in the +GenClass-lib repository; those files have the definitions and full tables. Two latency measurements come from +elsewhere and are linked where they are used: the runtime's changelog (WebGPU) and +[`packages/runtime/src/model/README.md`](https://github.com/genclass-dev/GenClass-lib/blob/main/packages/runtime/src/model/README.md) +(WASM threads); the app counts come from +[`realapps/README.md`](https://github.com/genclass-dev/GenClass-lib/blob/main/realapps/README.md). The quality +numbers (accuracy, false-intervention rates, recall, detections, calibration) are on held-out data. ## What it is -A Jev-style typed-decision encoder for `@genclass/runtime`. One forward pass reads a *situation* (the runtime's -`app / trigger / facts / in_flight / timeline / state / stats` text) together with typed questions (choice / noul / -score). Each question gets calibrated answers, and options are attention-isolated, so option order cannot change an -answer. For every salient trigger the runtime asks two standing questions: +A typed-decision encoder for `@genclass/runtime`, built like the project's earlier research encoder, GenClass 0.1 +([`research/`](https://github.com/genclass-dev/GenClass-lib/tree/main/research)). Older docs call this design +"Jev-style" after Jev, a TypeSafe AI product whose System One request format of typed questions GenClass 0.1 +reimplemented; GenClass is not affiliated with or endorsed by TypeSafe AI +([`research/GENCLASS.md`](https://github.com/genclass-dev/GenClass-lib/blob/main/research/GENCLASS.md)). + +One forward pass reads a *situation* (the runtime's `app / trigger / facts / in_flight / timeline / state / stats` +text) together with typed questions (choice, noul for yes/no, score). Each question gets calibrated answers, and +options are attention-isolated, so option order cannot change an answer. For every salient trigger the runtime asks +two standing questions: - `action`: which of the applicable actions to take, each with its description; - `diagnosis`: what is going on (`expected` / `stale` / `conflict` / `duplicate` / `inconsistent` / `failing` / @@ -27,7 +45,7 @@ of the best permitted non-passive action clears the margin for that tier and tri | | | |---|---| | backbone | ettin-encoder-17m (d 256, 7 layers: global attention in layers 1, 4, 7, 64-token sliding windows in the others) | -| parameters | 8.1M encoder + 0.7M heads after vocabulary pruning | +| parameters | 8.8M: 8.1M encoder + 0.7M heads after vocabulary pruning | | vocabulary | 16,364 tokens (the first 16,000 merges of the ettin tokenizer, plus markers `[Q] [O] [L] [T] [F]` = ids 16359–16363) | | positions | `max_len` 2,048 (state plus the longest question branch), `max_total` 8,192 | | `genclass-runtime-r17-q8.onnx` | 10.16 MB (10,160,186 bytes): MatMulNBits 8-bit (block 16) and int8 row-wise token embeddings, fp16-free. For WASM, and WebGPU without `shader-f16` | @@ -35,9 +53,18 @@ of the best permitted non-passive action clears the margin for that tier and tri | graph | inputs `input_ids, position_ids, q_group, i_group, choice_q, choice_items, score_q, score_items, noul_q, noul_t, noul_f`; outputs `choice_logits, score_logits, noul_logits` | | parity | onnxruntime-web q8 vs PyTorch: 223/223 decisions (block 32 lost parity; block 16 restored it) | -**Latency** (onnxruntime-web 1.30, WASM, 1 thread, q8): about 182, 330 and 599 ms at 500, 780 and 1,170 sequence -tokens, so about 180 ms per decision at the single-thread situation budget. WebGPU is much faster, and so is -4-thread WASM on crossOriginIsolated pages. Normal traffic makes no model calls: the runtime asks only about salient +**Latency** (onnxruntime-web 1.30, WASM, 1 thread, q8; a least-squares fit over the validation requests, run in +Node on the training VM, not in a browser, so these are estimates at those lengths, not single measurements; +[`training/ortweb/validate.mjs`](https://github.com/genclass-dev/GenClass-lib/blob/main/training/ortweb/validate.mjs), +[RESULTS.md §2](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/RESULTS.md#2-r17-vs-r32-choosing-the-shipping-model)): +about 182, 330 and 599 ms per forward pass at 500, 780 and 1,170 sequence tokens. On WebGPU, on one Apple GPU with +this model, a ~430-token warm-up pass took 49–74 ms and the first real decision 137 ms +([runtime 0.2.2 changelog](https://github.com/genclass-dev/GenClass-lib/blob/main/packages/runtime/CHANGELOG.md#022-2026-10-13)); +no systematic WebGPU latency benchmark has been published. Four WASM threads on a crossOriginIsolated page were +about 3× faster than one in headless Chromium in an earlier measurement with GenClass 0.1's 32M-parameter model +(271–360 ms against 837–981 ms at 780 sequence tokens; +[`packages/runtime/src/model/README.md`](https://github.com/genclass-dev/GenClass-lib/blob/main/packages/runtime/src/model/README.md)); +this has not been re-measured for 0.2.0. Normal traffic makes no model calls: the runtime asks only about salient situations. ## Results on held-out data @@ -47,9 +74,11 @@ situations. Gates were fitted on the outputs of the shipped q8 file, on dev data (simulated apps, on-policy rows from the shipping gate, and a real-app certification set counted per trajectory), as the lowest margin per tier and trigger whose cluster-robust 95% Wilson upper bound meets the profile's limits, then verified on held-out test. FIR is the -false-intervention rate (the gate fired where doing nothing was best); harm is the share of decisions whose action -made the outcome worse; recall is on clear cases (simulated) and on actionable real-app cases; gain is the share of -the available counterfactual improvement captured. +false-intervention rate: the share of cases where doing nothing was best but the gate fired anyway. Harm is the +share of decisions whose action made the outcome worse; recall is on clear cases (simulated) and on actionable +real-app cases; gain is the share of the available counterfactual improvement captured; detected is the share of +real problems on held-out real apps that were reported with the right diagnosis (stale on the `situation-v2.3` +slice; on `situation-v2`: 14% / 21% / 40%). | profile | limits (guard FIR / harm; heal FIR / harm) | guard FIR / harm | guard recall (clear / real) | guard gain | heal FIR / harm | heal recall (clear / real) | heal gain | report | detected (duplicate / stale / broken) | |---|---|---|---|---|---|---|---|---|---| @@ -87,35 +116,59 @@ The false-intervention rates here use the fixed 0.9 / 0.8 gates. | sim2f | 77.0% | 80.6% | 0.00% | 0.70% | 0.022 | | real2e | 80.0% | 83.6% | 0.00% | 0.31% | 0.021 | -**Observe-mode detections** (0.1.0, `gate.report` = 0.85; not re-measured for 0.2.0, whose profiles report at -0.95 / 0.90 / 0.70): -- On real apps never trained on, it detected about 76% of duplicate submits and 28% of stale overwrites, with 0.36% - false detections on clean or benign traffic. -- On the simulated test sets, 1.1% and 1.7% of rows whose correct diagnosis is `expected` were detected. - -**Reading.** The model is safe and rarely wrong when it acts, but it acts rarely. In guard mode at `balanced` it -intervenes in under 8% of the clear cases where acting would help. Its main value today is observe mode: -plain-English detections with the evidence. Later training rounds (a 150M teacher, distillation, on-policy rounds) -target recall. +**Observe-mode false detections.** For 0.2.0 the gate fit held them to at most 1% / 2% / 5% of clean real-app +decisions and 2% / 3% / 6% of simulated `expected` rows for `cautious` / `balanced` / `eager`, and every profile met +those limits on held-out test; the exact rates are not published. 0.2.0's detection rates are in the profile table +above. From `@genclass/runtime` 0.3.0, observe by default also records *possible* detections, whose probability +reaches the `eager` report value (0.70) but not the profile's report gate, on `console.debug`, in the devtools and in +telemetry; at `balanced`, the *likely* detections (those at the report gate) and the possible ones together are then +the `eager` row's detections (duplicate / stale / broken 77% / 45% / 13%, false detections held to at most 5% / 6%). Neither tier has been measured end to end on 0.3.0, and the share of possible +detections that are false is not measured +([RESULTS.md §11](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/RESULTS.md#11-observe-mode-detection-in-030-the-possible-tier-and-deferred-situations)). +For 0.1.0 (`gate.report` 0.85): 0.36% on clean or benign real-app traffic; 3.78% of held-out real-app +decisions whose correct diagnosis was `expected` (305 / 8,079; on the `real2e` slice, 240 of its 303 false +detections came from one held-out app, flagged as `stale` on one field, and without that app the slice is 1.05%); +1.1% / 1.7% of simulated `expected` rows (`sim2e` / `sim2f`), but 3.1% on the on-policy round b test set, over the +2% limit ([RESULTS.md §1](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/RESULTS.md#1-model-stages-compared-held-out-simulated-apps)). +It detected about 76% of duplicate submits and 28% of stale overwrites on real apps never trained on. + +**Reading.** The model rarely acts, and misses most problems. In guard mode at `balanced` it intervenes in 7.8% of +the clear simulated cases where acting would help (FIR 0.13%). Precision among its actions was not measured for +0.2.0 (on-policy round b, `r17-v2b` with its data-derived gates: 11% of its actions were false). Its main value +today is observe mode: plain-English detections with the evidence. Recall is the open problem: a 150M teacher was +trained and scored below `r17-v2b` on every held-out set, so distillation was skipped +([RESULTS.md §1](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/RESULTS.md#1-model-stages-compared-held-out-simulated-apps)). ## Training data and licence Apache-2.0. Initialised from `jhu-clsp/ettin-encoder-17m` (MIT). It was trained only on data generated for this project: -- **Curriculum:** about 1.2M programmatic rows of runtime situations with exact labels. They cover +- **Curriculum:** about 1.9M programmatic rows (`cur1` to `cur4`) of runtime situations with exact labels; stage 1 + trained on 1.2M of them, and later rounds replay a share, with a `situation-v2` curriculum (`cur5`). They cover timeline/causality/version/identity/streak/baseline/invariant/transition/HTTP/JS-error primitives and described-option decisions. -- **SIM:** the project's deterministic simulator of combinator-built apps. Labels come from counterfactual outcomes: - each candidate action is replayed and its cost measured. -- **REAL:** traces from about 150 web apps run in headless Chromium with the real runtime, labelled the same - counterfactual way. 144 apps were written for the corpus. 14 are MIT-licensed open-source RealWorld front-ends, - listed in `realapps/corpus/LICENSES.md`; their code is not part of this model, only traces of running them. Every - row of the real-app evaluation set was excluded from training. -- **On-policy rows:** situations the previous model's own gate produced in the simulator (DAgger-style), labelled - the same way. +- **SIM:** the project's deterministic simulator of combinator-built apps (46 features, 115 domains): 10.4M + `situation-v2` gold rows, of which 7.56M are the training split (318k dev and 2.55M test held out), plus a 2.1M + `situation-v2.3` top-up. Labels come from counterfactual outcomes: each candidate action is replayed and its cost + measured. +- **REAL:** traces from 158 web apps run in headless Chromium with the real runtime (128 apps for the + `situation-v2` rows, all 158 for the `situation-v2.3` top-up; `realapps/README.md`), 837,623 rows: 616,437 + `situation-v2` and 221,186 `situation-v2.3`, labelled the same counterfactual way. 144 apps were written for the + corpus. 14 are MIT-licensed open-source RealWorld front-ends, listed in + [`realapps/corpus/LICENSES.md`](https://github.com/genclass-dev/GenClass-lib/blob/main/realapps/corpus/LICENSES.md); + their code is not part of this model, only traces of running them. Every row of the real-app evaluation sets was + excluded from training. +- **On-policy rows:** 1,248,131 situations from `r17-v2a` acting in the simulator (round a, DAgger-style; of them + 448k at the shipping gate and 672k at an exploratory gate of 0.5, the rest not broken down in RESULTS.md §6), + labelled the same way. Round b (1,094,953 rows, from `r17-v2b`) was not trained on. The shipping-gate rows of + rounds a and b were used to fit and test the gates. - **Replay:** GenClass 0.1's synthetic generic data. +Row counts are from [RESULTS.md §6](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/RESULTS.md#6-data-volume) +and count every row generated, held-out dev and test splits included (splits are per trajectory). The 51M unlabeled +simulated rows listed there were generated for teacher labelling and not used, since distillation was skipped. + No benchmark, scraped or third-party dataset was used, and no weights from other GenClass experiments. ## Intended use and limits @@ -128,9 +181,14 @@ No benchmark, scraped or third-party dataset was used, and no weights from other - ignore / rollback / resync It never sees your source code. -- **Precision first.** About half of the training decisions have "do nothing" as the best answer. That includes +- **Precision first.** "Do nothing" is the best answer for most decisions in the project's data, including salient-looking benign cases: older writers in between, deliberate repeats, polling, typing bursts and explained - relation changes. Please report false interventions and false detections from your app. + relation changes. It was about 55% of curriculum rows and 70–96% of simulated rows in the stage-1 measurements + (`training/EVAL.md`), 50% to 100% per trigger in the earlier `situation-v1` real-app pilots + (`realapps/README.md`), and 75% of decision rows in `sim2e` (9,464 of 12,696) and 74% in `sim2f` (8,703 of + 11,756), the held-out `situation-v2` simulated test sets (the guard FIR denominators in `training/EVAL.md`). The + share in the training split has not been published. Please report false interventions and false detections from + your app ([GitHub Issues](https://github.com/genclass-dev/GenClass-lib/issues)). - Recall on actionable cases is low (see above). The runtime fails open: while the model is loading, after an error, or when it answers too late, the app runs unchanged. - Developer `ask` / `decide` questions work for short factual questions about the current situation. It is a small diff --git a/packages/runtime-model/README.md b/packages/runtime-model/README.md index 2898221f..5a28a1e8 100644 --- a/packages/runtime-model/README.md +++ b/packages/runtime-model/README.md @@ -1,17 +1,47 @@ # @genclass/runtime-model -The local decision model of [`@genclass/runtime`](https://www.npmjs.com/package/@genclass/runtime): a small -GenClass encoder (8.8M parameters, pruned from ettin-encoder-17m) that reads what the runtime observed in your app (a *situation*) and answers typed questions about -it: what is going on (`expected`, `stale`, `duplicate`, `failing`, ...) and which generic action fits. It runs in -the browser with onnxruntime-web, in a Web Worker, on WebGPU or WASM. Nothing leaves the page. - -**You don't need to install this package.** The runtime loads it from jsDelivr when the browser is idle after page -load, and caches it in Cache Storage. This is the default model URL: +The on-device model of [`@genclass/runtime`](https://www.npmjs.com/package/@genclass/runtime), the open-source +browser library that reports race conditions in web apps while they happen: stale responses, duplicate requests, +inconsistent state and failure storms. The runtime runs this model with onnxruntime-web in a Web Worker, on WebGPU +or WASM. Inference happens on the user's device and the model itself sends nothing; the runtime's anonymous +diagnostics are separate (see [Privacy](#privacy)). + +This page covers `@genclass/runtime-model@0.2.0` (model `genclass-runtime-r17` 2.0.0-rc4t, training run +`r17-v2dT`), the default model of `@genclass/runtime` 0.1.0-beta.1 and later, including `0.2.2` (npm `latest`) and +`0.3.0-beta.4` (`beta`, 2026-10-27). Training data, held-out results, thresholds and limits are in the +[model card](https://github.com/genclass-dev/GenClass-lib/blob/main/packages/runtime-model/MODEL_CARD.md). + +## At a glance + +| | `@genclass/runtime-model@0.2.0` | +|---|---| +| what it does | reads what the runtime observed in your app (a *situation*) and answers typed questions about it: what is going on (`expected`, `stale`, `duplicate`, `failing`, ...) and which generic action fits | +| used by | `@genclass/runtime` 0.1.0-beta.1 and later (its default model URL) | +| size | 8.8M parameters: an ettin-encoder-17m backbone pruned to a 16,364-token vocabulary (8.1M encoder + 0.7M heads) | +| download | about 13 MB on a first visit on WASM: the 10.2 MB q8 model file, ONNX Runtime's wasm (3.07 MB brotli) and the tokenizer. On WebGPU, a 5.5 MB brotli ONNX Runtime build instead, and with `shader-f16` the 13.6 MB fp16 file instead of the q8. Once, then cached | +| memory | +190 to 280 MB renderer memory while the model is loaded, on one pilot app (runtime 0.1.0-beta.3, Playwright Chromium, 4× CPU throttling). From runtime `0.3.0` the model is not loaded on devices that report 2 GB of memory or less, nor on Save-Data connections | +| speed | 182 / 330 / 599 ms per forward pass at 500 / 780 / 1,170 tokens on one WASM thread (fitted from runs in Node, not in a browser); normal traffic makes no model calls | +| false interventions at `balanced` (guard / heal) | 0.13% / 0.59% on held-out simulated apps; 0.00% on held-out real apps | +| guard recall at `balanced` (same gate) | 7.8% of clear simulated cases, 7.5% of actionable real-app cases | +| observe detections at `balanced`, held-out real apps, right diagnosis | duplicate submits 75%, stale overwrites 30% (`situation-v2.3` slice; 21% on `situation-v2`), genuine breaks 6%. False detections were held to at most 2% (real) and 3% (simulated) of clean decisions, met on held-out test; the exact rates are not published. Runtime `0.3.0`'s observe default also records *possible* detections, down to the `eager` report gate (0.70): likely and possible together are expected at the `eager` profile's 77% / 45% / 13%, with false detections held to at most 5% (real) and 6% (simulated); the share of possible detections that are false is not measured | +| licence | Apache-2.0 (initialised from `jhu-clsp/ettin-encoder-17m`, MIT) | + +Numbers: [model card](https://github.com/genclass-dev/GenClass-lib/blob/main/packages/runtime-model/MODEL_CARD.md) +and [RESULTS.md](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/RESULTS.md) §1, §2, §11 and +§17. The false-intervention rate (FIR) is the share of cases where doing nothing was best but the gate fired anyway. + +## Do I need to install it? + +No. `@genclass/runtime` loads the model from jsDelivr when the browser is idle after page load, caches it in Cache +Storage and checks every file's sha256. From `0.3.0` it skips the model on devices that report 2 GB of memory or +less and on Save-Data connections. This is the default model URL: ``` https://cdn.jsdelivr.net/npm/@genclass/runtime-model@0.2.0/files/ ``` +Install this package only to serve the model yourself ([Self-hosting](#self-hosting)). + ## Files `files/model.json` lists every file with its size and sha256. The runtime checks both before using a file. @@ -20,49 +50,107 @@ https://cdn.jsdelivr.net/npm/@genclass/runtime-model@0.2.0/files/ |---|---|---| | `files/model.json` | 1 KB | the model card the runtime reads first (`genclass-runtime-model/1`) | | `files/genclass-runtime-r17-q8.onnx` | 10.2 MB (10,160,186 bytes) | WASM, and WebGPU without `shader-f16`. 8-bit weights (block 16), int8 embeddings | -| `files/genclass-runtime-r17-fp16.onnx` | 13.6 MB | WebGPU with `shader-f16` | +| `files/genclass-runtime-r17-fp16.onnx` | 13.6 MB (13,569,879 bytes) | WebGPU with `shader-f16` | | `files/tokenizer.json` | 1.1 MB | byte-level BPE, 16,364 tokens | | `files/calibration.json` | 1 KB | per-question-kind temperatures | | `files/meta.json` | 5 KB | graph inputs and outputs, the gain-gate margins and detection thresholds per aggressiveness profile | -A browser downloads `model.json`, the tokenizer, the calibration, the meta file and **one** of the two `.onnx` files: -about 11.5 MB on WASM. One decision takes about 180 ms on one WASM thread (182 / 330 / 599 ms at 500 / 780 / 1,170 tokens). onnxruntime-web's wasm (3.1 MB brotli) comes from jsDelivr's `onnxruntime-web` package. - -Model: `genclass-runtime-r17` 2.0.0-rc4t (training run `r17-v2dT`). `model.json` sha256 `3f79289280dc11e4284b04d63a505e3cac9e77b3dc06a981cd4686c880169daf`. Its `meta.json` gate is a gain gate with three aggressiveness profiles (`cautious`, `balanced`, `eager`), chosen with `GenClass.init({ aggressiveness })` in `@genclass/runtime` 0.1.0-beta.1 and later. +A browser downloads `model.json`, the tokenizer, the calibration, the meta file and **one** of the two `.onnx` files, +plus onnxruntime-web's wasm from jsDelivr's `onnxruntime-web` package. On WASM a first visit downloads about 13 MB. +A pilot app with `@genclass/runtime` 0.1.0-beta.3 and this model recorded 12.7 MB after the load event: the q8 model +at 9.45 MB as recorded, ONNX Runtime's wasm at 3.07 MB brotli and the tokenizer at 207 KB; with the q8 file at its +full 10.2 MB the same sum is about 13.4 MB +([RESULTS.md §17](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/RESULTS.md#17-browser-cost-on-a-pilot-app-2026-10-09-and-2026-10-10)). +On WebGPU the ONNX Runtime build is larger (5.5 MB brotli), and with `shader-f16` the 13.6 MB fp16 file replaces the +q8. Later visits revalidate `model.json` (about 1 KB) and read the rest from Cache Storage. A forward pass on one WASM +thread is estimated at about 182 / 330 / 599 ms at 500 / 780 / 1,170 tokens (a least-squares fit over runs of +onnxruntime-web 1.30 in Node on the training VM, not in a browser; +[RESULTS.md §2](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/RESULTS.md#2-r17-vs-r32-choosing-the-shipping-model)). + +Model: `genclass-runtime-r17` 2.0.0-rc4t (training run `r17-v2dT`). `model.json` sha256 +`3f79289280dc11e4284b04d63a505e3cac9e77b3dc06a981cd4686c880169daf`. Its `meta.json` gate is a gain gate with three +aggressiveness profiles (`cautious`, `balanced`, `eager`), chosen with `GenClass.init({ aggressiveness })` in +`@genclass/runtime` 0.1.0-beta.1 and later. ## Self-hosting -To serve the model yourself (offline apps, strict CSP, no third-party CDN): +To serve the model yourself (offline apps, strict CSP, no third-party CDN), download it with the runtime's CLI, +which verifies every hash: ```bash -npm install @genclass/runtime-model -cp -r node_modules/@genclass/runtime-model/files public/genclass-model +npx @genclass/runtime@beta fetch-model public/genclass-model # model + ONNX Runtime files (into public/genclass-model/ort/) ``` ```ts -GenClass.init({ model: { baseUrl: "/genclass-model/" } }); +GenClass.init({ model: { baseUrl: "/genclass-model/", ortWasmPaths: "/genclass-model/ort/" } }); ``` -Or download it with the runtime's CLI, which verifies every hash: +`fetch-model` downloads from `https://cdn.jsdelivr.net/npm/@genclass/runtime-model@0.2.0/files/` by default +(`--from ` for another copy). `--ort all`, the default, copies both ONNX Runtime builds: the WASM one (14 MB) and +the WebGPU one (27 MB), which is over some static hosts' per-file limit (Cloudflare: 25 MiB); there, use `--ort wasm` +and add `device: "wasm"` to the `model` options above. `--ort none` keeps loading ONNX Runtime from jsDelivr. + +Or copy the model files from this package: ```bash -npx @genclass/runtime@beta fetch-model public/genclass-model --from https://cdn.jsdelivr.net/npm/@genclass/runtime-model@0.2.0/files/ +npm install @genclass/runtime-model +cp -r node_modules/@genclass/runtime-model/files public/genclass-model +``` + +```ts +GenClass.init({ model: { baseUrl: "/genclass-model/" } }); ``` -With the script tag, use `data-model="/genclass-model/"`. +This serves the model only: ONNX Runtime's wasm still loads from jsDelivr. For no third-party CDN, use `fetch-model` +above and `ortWasmPaths`. -## What it can and can't do +With the script tag, use `data-model="/genclass-model/"` (and `data-ort` for a self-hosted ONNX Runtime). The full +Content-Security-Policy guidance is in the +[`@genclass/runtime` README](https://github.com/genclass-dev/GenClass-lib/blob/main/packages/runtime/README.md#content-security-policy-and-self-hosting). -See [MODEL_CARD.md](MODEL_CARD.md) for the training data, held-out results, thresholds and limits. In short: +## What it can and can't do -- **It acts rarely, by design.** It acts only when its expected gain over doing nothing clears the profile's margin. - At the default `balanced` profile the false-intervention rate on held-out simulated apps is 0.13% (guard) and - 0.59% (heal), and 0.00% on held-out real apps for every profile; `cautious` is 0.005% / 0.26%. At `balanced`, - guard acts on under 8% of the clear cases where acting would help. -- **In observe mode it reports.** At `balanced` it detects about 75% of duplicate submits, 30% of stale overwrites - and 6% of genuine breaks on held-out real apps. False-report rates were last measured on the previous model - (0.1.0: 0.36% on clean real-app traffic at report 0.85); ECE has not yet been measured for 0.2.0 (0.1.0: 0.021–0.023). +In short (details in the +[model card](https://github.com/genclass-dev/GenClass-lib/blob/main/packages/runtime-model/MODEL_CARD.md)): + +- **It rarely acts, and misses most problems.** In `guard` or `heal` mode it acts only when its expected gain over + doing nothing clears the profile's margin and the top diagnosis is not `expected`. At the default `balanced` + profile, guard acts on 7.8% of the clear simulated cases where acting would help, with a false-intervention rate + of 0.13% (guard) and 0.59% (heal) on held-out simulated apps, and 0.00% on held-out real apps for every profile. + `cautious` is 0.005% / 0.26%, with guard recall 2.4%. +- **In observe mode (the runtime's default; only 0.1.0-beta.1 defaulted to `guard`) it reports.** At `balanced` it + detects, with the right diagnosis, about 75% of duplicate submits, 30% of stale overwrites and 6% of genuine breaks + on held-out real apps. The 0.2.0 fit held false detections to at most 2% of clean real-app decisions and 3% of + simulated ones; the exact rates were not published. From runtime `0.3.0`, observe by default also records + *possible* detections (probability from the `eager` report value, 0.70, up to the report gate) on `console.debug`, + in the devtools and in telemetry (on the app's own listeners only with `detect: "possible"`). Likely and possible + together are expected to match the `eager` profile: 77% / 45% / 13% detected, false detections held to at most 5% + (real) and 6% (simulated); the share of possible detections that are false is not measured, and neither tier has + been measured end to end on 0.3.0 ([RESULTS.md §11](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/RESULTS.md#11-observe-mode-detection-in-030-the-possible-tier-and-deferred-situations)). + The previous model (0.1.0, report threshold 0.85) had 0.36% on clean or benign real-app traffic and + 3.8% of held-out real-app decisions whose correct diagnosis was `expected` (most of them from one app). +- **Calibration** (ECE) has not been measured for 0.2.0 (0.1.0: 0.021 to 0.023). +- It never sees your source code, only the situation text the runtime builds. It is a small encoder, not a general + reasoner. + +## Privacy + +The model runs inside the page and sends nothing. `@genclass/runtime`, which loads it, sends **anonymous diagnostics +by default** on public hosts: GenClass's decisions and counts, no text from your page unless the app opts in, to the +GenClass maintainers to improve the model. They are off on `localhost` and other local hosts from `0.2.2`. Earlier +versions differ: `0.1.0-beta.1` and `0.1.0-beta.2` have no diagnostics, `0.1.0-beta.3` to `0.2.1` send from local +hosts too, and `0.1.0-beta.3` to `0.2.0` include the redacted situation text the model read by default. Opt out with +`GenClass.init({ telemetry: false })`, `?genclass=no-telemetry`, `localStorage.setItem("genclass.telemetry", "off")` +or `init --no-telemetry`; browsers sending Global Privacy Control are never collected. +Details: [TELEMETRY.md](https://github.com/genclass-dev/GenClass-lib/blob/main/packages/runtime/TELEMETRY.md). ## License -Apache-2.0. Initialised from `jhu-clsp/ettin-encoder-17m` (MIT); trained only on data generated for this project (see the model card). +Apache-2.0. Initialised from `jhu-clsp/ettin-encoder-17m` (MIT); trained only on data generated for this project +(see the [model card](https://github.com/genclass-dev/GenClass-lib/blob/main/packages/runtime-model/MODEL_CARD.md)). + + diff --git a/packages/runtime-model/package.json b/packages/runtime-model/package.json index 53b0e830..87636f85 100644 --- a/packages/runtime-model/package.json +++ b/packages/runtime-model/package.json @@ -1,7 +1,7 @@ { "name": "@genclass/runtime-model", "version": "0.2.0", - "description": "The GenClass runtime model (8.8M parameters, ONNX) for @genclass/runtime.", + "description": "The on-device model of @genclass/runtime: an 8.8M-parameter ONNX encoder (10.2 MB q8 for WASM, 13.6 MB fp16 for WebGPU with shader-f16) that diagnoses stale responses, duplicate requests and race conditions in the browser.", "license": "Apache-2.0", "repository": { "type": "git", @@ -15,11 +15,17 @@ "keywords": [ "genclass", "onnx", - "model", + "onnxruntime-web", "webgpu", "wasm", - "onnxruntime-web", - "runtime" + "web-worker", + "on-device-ai", + "browser-ai", + "encoder", + "race-condition", + "stale-response", + "duplicate-request", + "model-card" ], "files": [ "files/", diff --git a/packages/runtime/CHANGELOG.md b/packages/runtime/CHANGELOG.md index c3372da4..0dd690e8 100644 --- a/packages/runtime/CHANGELOG.md +++ b/packages/runtime/CHANGELOG.md @@ -142,7 +142,7 @@ opt-outs are unchanged, and the default model is unchanged. that long, EventSource and WebSocket are not checked. Same rules as the other visual checks: on by default on development hosts, `visual: true` elsewhere, report-only, never a situation or model input, nothing sent. It lives in the lazily loaded checks chunk: no first-load bytes. Demos: 0 false reports in 120 clean and chaos runs, 11 of 18 - planted 60 s hangs found (polls on a learned schedule are skipped by design; RESULTS.md §21). + planted 60 s hangs found (polls on a learned schedule are skipped by design; RESULTS.md §23). - **`init` writes `defer` on the plain-HTML script tag**, so a slow or blocked CDN no longer holds up rendering. GenClass then starts after classic scripts without `defer` or `async` (their requests during page load are not seen), still before deferred and module scripts; model loading is unchanged (idle after `load`). `--sync` writes the @@ -220,8 +220,8 @@ model-visible text changed (`situation-v2.3`); the default model is still `@genc first load stays within its limits (108.54 / 100.40 KB). It wraps the runtime's delivery gate and request ignore list while connected and restores both on `disconnect()`; nothing it records reaches a situation, a decision, the event log or the model (no model-visible text changed, - `situation-v2.3`), and telemetry is unchanged. INTERCEPTION.md, THREAT-MODEL.md (T10), API.md, CONTRACT.md §16, - RESULTS.md section 16. + `situation-v2.3`), and telemetry is unchanged. INTERCEPTION.md, THREAT-MODEL.md (T13), API.md, CONTRACT.md §16, + RESULTS.md section 18. - **Fix: a reused XHR object could get the next request's response early.** When the app aborted an XHR whose completion listeners the runtime held for a delivery decision (guard or heal) and sent a new request on the same object, releasing the old, held delivery ran the listeners queued for the new request before its own decision. The @@ -236,7 +236,7 @@ model-visible text changed (`situation-v2.3`); the default model is still `@genc no longer names Inter or JetBrains Mono, so a page that declares either as a web font no longer downloads it for the overlay. The tab row scrolls sideways instead of hiding Expectations and Now past the panel's edge, and the header no longer clips the name. Every view, control and string is the same; no model-visible text changed. -- **Interaction analytics, off by default (`analytics: true`; [docs/runtime/ANALYTICS.md](../../docs/runtime/ANALYTICS.md)).** +- **Interaction analytics, off by default (`analytics: true`; [docs/runtime/ANALYTICS.md](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/ANALYTICS.md)).** An app that sets `analytics: true` with its token also sends `interactions` telemetry events for its dashboard: per page view, the route, engaged time (the tab visible, at most 30 s after the last input), scroll depth milestones (25 / 50 / 75 / 100 %), and clicks, rage clicks (3 within 1 s on one element) and dead clicks (a link or button @@ -268,14 +268,35 @@ model-visible text changed (`situation-v2.3`); the default model is still `@genc and heal the first duplicate submit of every page load went through on a CPU about 15% slower than the VM's. The warm-up now times a second pass on every device: `status.latency` comes from it, `status.warmupMs` is still the first, cold pass. A load on WASM takes one more warm-up pass (a ~430-token situation, warm). No model-visible text + changed. - **Release notes: the npm registry's empty `readme` field.** Since 0.3.0-beta.0 went to `beta`, the registry's top-level `readme` for the package (`npm view @genclass/runtime readme`) is empty: the registry empties it on every publish to a tag other than `latest` and fills it again on the next `latest` publish, as it does for react, typescript, next and vitest. The npm and jsDelivr package pages are not affected; they show each version's full README. `.github/workflows/release.yml` now says so. No package contents, runtime code or model-visible text changed. - -## 0.3.0-beta.0 (unreleased) +- **Docs and package metadata.** The npm description, keywords and author are new, and `CHANGELOG.md` now ships in + the package. The README is shorter: the full install guide, the dashboard and the sinks moved to + [INSTALL.md](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/INSTALL.md), + [DASHBOARD.md](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/DASHBOARD.md) and + [SINKS.md](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/SINKS.md), next to a new + [FAQ](https://github.com/genclass-dev/GenClass-lib/blob/main/FAQ.md) and a + [docs index](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/README.md) with a glossary. + TELEMETRY.md gives the six larger events (`session`, `model`, `decision`, `detect`, `summary`, `interactions`) a + section each, adds `init --no-telemetry` (with its one exception: in an Astro layout where GenClass is inline in + ``, add the `telemetry=off` meta tag yourself) and the `genclass` kill switch to its opt-out table and opens + with a version note. It now also lists two things that were already sent: a decision that would have acted in + shadow mode is always sent (never sampled), and `summary.runtime` carries the model state and the longest hold + (`heldMs.max`). INTERCEPTION.md also opens with a version note, lists the webhook sink's requests (and the chunks + the zero-code entries load for a named sink) and corrects four rows: learned state is stored only with + `learn.expectations` or `learn.persist`, only routine decision records are sampled and capped (0.3.0), the devtools + overlay listens for theme changes only with `theme: "auto"`, and a `?genclass=` value in the URL takes precedence + over the `genclass` kill switch in localStorage. It also marks the model's default device floor and + `inlineFallback: false` as 0.3.0 only. Links that leave the package point at GitHub. + The unscoped `genclass-runtime` alias is documented as deprecated in this repository (it is not yet marked + deprecated on the npm registry). Nothing that is sent changed; no runtime code and no model-visible text changed. + +## 0.3.0-beta.0 (2026-10-15) The first 0.3.0 prerelease, for the `beta` dist-tag (release.yml publishes a prerelease version to the dist-tag named after it; `latest` stays where it is until the owner moves it). No model-visible text changed: the situation @@ -718,7 +739,7 @@ observer does not record the runtime's own writes. ### Measured visual checks -- **New: measured visual checks** ([docs/runtime/VISUAL.md](../../docs/runtime/VISUAL.md)). GenClass now finds visual +- **New: measured visual checks** ([docs/runtime/VISUAL.md](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/VISUAL.md)). GenClass now finds visual and CSS bugs that never throw an error: text you cannot read because it is nearly the colour of its background (below 1.5:1, including on hover, on focus and in dark mode), images, CSS background images, stylesheets, web fonts, icon fonts and media that failed to load, styles and images a Content-Security-Policy blocked, `[object Object]`, @@ -766,7 +787,7 @@ observer does not record the runtime's own writes. the visitor is not using, layout shifts and failed CSS images outside Chromium, low contrast that is readable but below WCAG AA. - **Measured** by a new bench (`bench/visual`, GitHub Actions only; - [RESULTS.md section 14](../../docs/runtime/RESULTS.md)): 0 false reports in 128 views of the demos, templates and + [RESULTS.md section 14](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/RESULTS.md#visual-bench)): 0 false reports in 128 views of the demos, templates and compat apps (desktop and phone, light and dark), and recall 1.00 on planted faults for 19 of the 23 kinds (`click-blocked` and `scroll-lock-stuck` 0.67, `pointer-events-leak` 0.94, `css-var-undefined` 0.96; run 5). On about 40 public home pages the checks diff --git a/packages/runtime/INTERCEPTION.md b/packages/runtime/INTERCEPTION.md index d6e597c8..2b3abac9 100644 --- a/packages/runtime/INTERCEPTION.md +++ b/packages/runtime/INTERCEPTION.md @@ -1,4 +1,4 @@ -# Interception surface +# GenClass interception surface: every browser API @genclass/runtime wraps Everything `@genclass/runtime` wraps, replaces or listens to in a page, what each mode may change, what it never does, how every action can be undone, and how everything is put back. The source is the code; each row names the file @@ -7,9 +7,24 @@ browser global, compares what changed with the inventory at the end of this page and counts the patch sites in `src/` (a new `addEventListener`, `defineProperty` or global assignment fails the test until this page is updated). -Related: [THREAT-MODEL.md](../../docs/runtime/THREAT-MODEL.md) (what an attacker could do with this surface), -[SECURITY.md](../../SECURITY.md), [TELEMETRY.md](TELEMETRY.md) (what leaves the page), -[OPTIONS-SPEC.md](../../docs/runtime/OPTIONS-SPEC.md) (every option), [API.md](../../docs/runtime/API.md). +> **Version.** Written for `@genclass/runtime` 0.3.0-beta.4 as it is on GitHub `main` (released 2026-10-27). The +> copy in an npm package describes that package's version. As of 2026-10-27, `npm i @genclass/runtime@beta` installs +> 0.3.0-beta.4 (the 0.3.0 line: 0.3.0-beta.1 to 0.3.0-beta.4, released 2026-10-22 to 2026-10-27), and +> `npm i @genclass/runtime` installs 0.2.2 (`latest`, 2026-10-13), which has no measured visual checks, no `protect()`, +> no webhook sink and no interaction analytics. Items marked 0.3.0 are not in 0.2.x; the interaction analytics rows +> (`src/analytics/`) and the devtools overlay's `theme: "dark"` default are 0.3.0-beta.1, the `genclass.analytics.` +> entry 0.3.0-beta.2. + +The `src/...` links are relative to `packages/runtime/` in the repository +([GenClass-lib on GitHub](https://github.com/genclass-dev/GenClass-lib/tree/main/packages/runtime)): the npm package +ships the built `dist/`, not `src/`. + +Related: [THREAT-MODEL.md](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/THREAT-MODEL.md) (what +an attacker could do with this surface), +[SECURITY.md](https://github.com/genclass-dev/GenClass-lib/blob/main/SECURITY.md), +[TELEMETRY.md](TELEMETRY.md) (what leaves the page), +[OPTIONS-SPEC.md](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/OPTIONS-SPEC.md) (every option), +[API.md](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/API.md). ## Contents @@ -94,10 +109,10 @@ devtools overlay's Alt+Shift+G hotkey (only when you mount the overlay). | `document` | `DOMContentLoaded` (once) | [src/cdn/config.ts](src/cdn/config.ts) `whenBody` | mount the devtools overlay when `` exists | fires once | | the model `Worker` | `message`, `error`, `messageerror` | [src/model/host.ts](src/model/host.ts), [src/cdn/global.ts](src/cdn/global.ts) `blobModuleWorker` | the worker GenClass started | terminated | | inside the worker | `message`, `securitypolicyviolation` | [src/model/worker.ts](src/model/worker.ts) | the worker's own scope | terminated | -| the devtools overlay (only when mounted) | `keydown` on `document` (capture: Alt+Shift+G toggles the overlay and calls `preventDefault` on that combination only), clicks and keys inside its own shadow root, `matchMedia` theme changes | [src/devtools/index.ts](src/devtools/index.ts), [src/devtools/ui.ts](src/devtools/ui.ts) | the overlay | on unmount | -| `document` (capture), `window` | `securitypolicyviolation` (at most 50 kept), `load` (once) | [src/visual/start.ts](src/visual/start.ts) `startVisual` | measured visual checks ([VISUAL.md](../../docs/runtime/VISUAL.md)), only when `InitOptions.visual` is on for this page: violations during page load cannot be queried later; the checks' chunk loads 1.5 s after `load` | `securitypolicyviolation` yes; `load` fires once and is inert after `destroy()` | +| the devtools overlay (only when mounted) | `keydown` on `document` (capture: Alt+Shift+G toggles the overlay and calls `preventDefault` on that combination only), clicks and keys inside its own shadow root, `matchMedia` theme changes (only with `theme: "auto"`, the default before 0.3.0-beta.1) | [src/devtools/index.ts](src/devtools/index.ts), [src/devtools/ui.ts](src/devtools/ui.ts) | the overlay | on unmount | +| `document` (capture), `window` | `securitypolicyviolation` (at most 50 kept), `load` (once) | [src/visual/start.ts](src/visual/start.ts) `startVisual` | measured visual checks ([VISUAL.md](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/VISUAL.md)), only when `InitOptions.visual` is on for this page: violations during page load cannot be queried later; the checks' chunk loads 1.5 s after `load` | `securitypolicyviolation` yes; `load` fires once and is inert after `destroy()` | | `document`, `window`, `document.fonts` (capture, passive; only once the visual chunk runs) | `document`: `error`, `load`, `securitypolicyviolation`, `click`, `pointerover`, `focusin`, `pointerdown`, `pointerup`, `pointercancel`, `keydown`, `visibilitychange`, `prerenderingchange`; `window`: `wheel`, `touchmove`, `scroll`, `hashchange`, `popstate`, `resize`, `pageshow`; `document.fonts`: `loadingdone` | [src/visual/engine.ts](src/visual/engine.ts) `Visual.install` | measured visual checks: failed assets, clicks swallowed by an invisible layer, hover and focus contrast, focus under a sticky header, scroll attempts on a locked page. Read only: never `preventDefault`, `stopPropagation` or a write | yes | -| `window` (capture, passive), `document` (only with `InitOptions.analytics` or `rt.analytics.start()`, once the analytics chunk runs) | `window`: `click`, `pointerdown`, `pointermove`, `keydown`, `wheel`, `touchstart` (capture), `scroll`, `pagehide`, `pageshow`; `document`: `visibilitychange`, `prerenderingchange` (once, on a prerendered page) | [src/analytics/index.ts](src/analytics/index.ts) `startAnalytics` | interaction analytics ([ANALYTICS.md](../../docs/runtime/ANALYTICS.md)): page views, engaged time, scroll depth, and clicks, rage clicks and dead clicks per element key, sent as `interactions` telemetry events. Trusted events only; never `preventDefault`, `stopPropagation` or a write; nothing reaches the event log or the model | yes, and on `rt.analytics.stop()` | +| `window` (capture, passive), `document` (only with `InitOptions.analytics` or `rt.analytics.start()`, once the analytics chunk runs) | `window`: `click`, `pointerdown`, `pointermove`, `keydown`, `wheel`, `touchstart` (capture), `scroll`, `pagehide`, `pageshow`; `document`: `visibilitychange`, `prerenderingchange` (once, on a prerendered page) | [src/analytics/index.ts](src/analytics/index.ts) `startAnalytics` | interaction analytics ([ANALYTICS.md](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/ANALYTICS.md)): page views, engaged time, scroll depth, and clicks, rage clicks and dead clicks per element key, sent as `interactions` telemetry events. Trusted events only; never `preventDefault`, `stopPropagation` or a write; nothing reaches the event log or the model | yes, and on `rt.analytics.stop()` | ## Objects it instruments after handing them to the app @@ -118,7 +133,7 @@ Installed with `autoState` (on by default for `@genclass/runtime/auto` and the s | Source | Hook point | What GenClass does there | Store kind | May GenClass change it? | |---|---|---|---|---| | React ≥ 16.8 (all renderers, dev and prod builds) | `__REACT_DEVTOOLS_GLOBAL_HOOK__` (above) | on each commit, walks only the fibers that re-rendered (≤ 1 ms, ≤ 20,000 fibers per commit) and records `useState`, `useReducer`, `useSyncExternalStore` and class-component state of named components (≤ 3 instances per component, ≤ 48 component stores, ≤ 16 hooks each). Functions, promises, elements, DOM nodes and class instances are skipped; a value equal to what a password, card or one-time-code input holds is redacted from then on | `observed` | **No.** Never held, dropped, reverted, rolled back or resynced (`state/hub.ts` `StoreHub.observe` only records; mutation `discard`/`defer` are not offered: `situation/build.ts` `builtinUnavailable`; rollback needs a writable store) | -| React renderers that inject after GenClass | the renderer's hooks dispatcher (`ReactSharedInternals.H` in React 19, `ReactCurrentDispatcher.current` before) becomes an accessor that returns an object inheriting from React's dispatcher with own `useState`/`useReducer` whose setters are wrapped; class instances' `updater` is wrapped | the wrapper calls the real setter unchanged and first notes which operation (request, timer, user action) called it | — | No. After `destroy()` the wrappers stay (React keeps the setters' identity) and pass through | +| React renderers that inject after GenClass | the renderer's hooks dispatcher (`ReactSharedInternals.H` in React 19, `ReactCurrentDispatcher.current` before) becomes an accessor that returns an object inheriting from React's dispatcher with own `useState`/`useReducer` whose setters are wrapped; class instances' `updater` is wrapped | the wrapper calls the real setter unchanged and first notes which operation (request, timer, user action) called it | none | No. After `destroy()` the wrappers stay (React keeps the setters' identity) and pass through | | Redux and Redux Toolkit stores created with the Redux DevTools compose or enhancer (RTK's default) | `__REDUX_DEVTOOLS_EXTENSION_COMPOSE__`, `__REDUX_DEVTOOLS_EXTENSION__()` | adds GenClass's Redux enhancer innermost (as the manual `genclassEnhancer`, [src/adapters/redux.ts](src/adapters/redux.ts)); the real extension stays outermost | `adapter` (full) | **Yes, in guard/heal**: like a registered store (a salient async dispatch may be held with `policy.holdWrites`, dropped, late-reverted within 800 ms, rolled back). Redux stores created before a runtime attached are observed only | | Zustand `devtools()` middleware and other Redux DevTools `connect()` clients | `__REDUX_DEVTOOLS_EXTENSION__.connect` | records every state they send (functions left out), attributed to the operation that made the change | `observed` | **No** | @@ -138,14 +153,15 @@ from ever being held. | What | Where | When | File | |---|---|---|---| -| Network: the model card, tokenizer, calibration, meta and the model variant (about 10 MB) from `model.baseUrl` (default jsDelivr `@genclass/runtime-model`), and onnxruntime-web's wasm from `model.ortWasmPaths` | with the page's fetch as it was when the module loaded (never through the wrapper) | at idle after load, `GenClass.init` in a browser with a model; not on devices reporting 2 GB or less nor on Save-Data (`model.loadIf` default) | [src/model/loader.ts](src/model/loader.ts), [src/model/backend.ts](src/model/backend.ts) | -| Network: anonymous diagnostics to the GenClass collector (`DEFAULT_TELEMETRY_ENDPOINT`, a Cloudflare Worker): a summary a minute while active, records of at most 200 decisions per page load (routine ones 1 in 4) | native `fetch`/`sendBeacon` captured at module load, `text/plain`, `credentials: "omit"`, `referrerPolicy: "no-referrer"` | `GenClass.init` in a browser unless turned off ([TELEMETRY.md](TELEMETRY.md)); never in `createRuntime()` | [src/telemetry/](src/telemetry/) | +| Network: the model card, tokenizer, calibration, meta and the model variant (about 10 MB) from `model.baseUrl` (default jsDelivr `@genclass/runtime-model`), and onnxruntime-web's wasm from `model.ortWasmPaths` | with the page's fetch as it was when the module loaded (never through the wrapper) | at idle after load, `GenClass.init` in a browser with a model; not on devices reporting 2 GB or less nor on Save-Data (`model.loadIf` default, 0.3.0; 0.2.x loads on every device, and on Save-Data only when first needed) | [src/model/loader.ts](src/model/loader.ts), [src/model/backend.ts](src/model/backend.ts) | +| Network: anonymous diagnostics to the GenClass collector (`DEFAULT_TELEMETRY_ENDPOINT`, a Cloudflare Worker): a summary about once a minute while decisions continue; every detection with its decision record, and every other non-routine decision record; routine decision records (top diagnosis `expected`, nothing acted) sampled 1 in 4 and capped at 200 per page load (0.3.0; 0.2.x sends every decision record) | native `fetch`/`sendBeacon` captured at module load, `text/plain`, `credentials: "omit"`, `referrerPolicy: "no-referrer"` | `GenClass.init` in a browser unless turned off ([TELEMETRY.md](TELEMETRY.md)); never in `createRuntime()` | [src/telemetry/](src/telemetry/) | +| Network (0.3.0): a webhook sink's batched POSTs of `{ schema: "genclass-sink/1", version, records }` to the URL the app configured, only when the app adds `webhook()` from `@genclass/runtime/sinks/webhook` or names `webhook:` in its config (`init --sink webhook:`) | with `/auto` and a named sink, first a dynamic `import()` of the sink resolver and then of the adapter, both lazy chunks of the app's own bundle; with the script tag, a dynamic `import()` of the adapter, `dist/sinks/webhook.js`, from where the script was loaded (the CDN, or `data-base` when self-hosted); a named `console-json` sink loads its adapter the same way and sends nothing. The POSTs use `navigator.sendBeacon`, else `fetch` with `keepalive`, both looked up at send time; `headers` force the fetch path; when the fetch path is used, list the URL in `requests.ignore` (the zero-code entries do this for `webhook:`) | with the zero-code entries, the imports when the first sink record arrives; then each round of sink records, and on `pagehide` or when the tab is hidden | [src/sinks/webhook.ts](src/sinks/webhook.ts), [src/cdn/sinks.ts](src/cdn/sinks.ts) `namedSinks`, [src/cdn/sinks-load.ts](src/cdn/sinks-load.ts) | | Cache Storage | `genclass-model` cache (`model.cacheName`): model files, each checked against the card's size and sha256 | model load | [src/model/loader.ts](src/model/loader.ts) `fetchFile` | | `sessionStorage` | `genclass:bucket` (`sample`: `act@` or `observe@`, reused only under the same `sample`; never read or written at `sample` 0 or 1), the breaker's trip state | `sample` < 1; breaker trips | [src/runtime.ts](src/runtime.ts) `sampleBucket`, [src/decide/breaker.ts](src/decide/breaker.ts) | -| `localStorage` / `sessionStorage` | `genclass:learn` (learned flows by default; operation shapes too with `learn.persist`: signatures, routes and field names with id-like parts normalised, counts, hashed action labels; ≤ `learn.maxBytes`), `genclass:learn:probe` (set and removed once at init to check storage works), `genclass:learn:session` in sessionStorage (the tab's session ordinal) | flows by default, not in jsdom / happy-dom (`learn.persist: false` turns it all off); the runtime's own writes are not recorded as storage events | [src/runtime.ts](src/runtime.ts) `saveProfilesSoon`, `startSession`, [docs/runtime/EXPECTATIONS.md](../../docs/runtime/EXPECTATIONS.md) | +| `localStorage` / `sessionStorage` | `genclass:learn` (only with `learn.expectations: true` (0.3.0) or `learn.persist`: learned flows, and operation shapes too with `learn.persist`: signatures, routes and field names with id-like parts normalised, counts, hashed action labels; ≤ `learn.maxBytes`), `genclass:learn:probe` (set and removed once at init when a store is wanted, to check storage works), `genclass:learn:session` in sessionStorage (the tab's session ordinal, with a localStorage store) | only with `learn.expectations: true` or `learn.persist` (by default nothing is stored); with `learn.expectations` alone, not in jsdom / happy-dom; `learn.persist: false` turns it all off; the runtime's own writes are not recorded as storage events | [src/runtime.ts](src/runtime.ts) `saveProfilesSoon`, `startSession`, [docs/runtime/EXPECTATIONS.md](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/EXPECTATIONS.md) | | `localStorage` | `genclass.analytics.`: the visit id (16 random hex characters), the time of the visit's last page view or event, and a visited-before flag; written with the storage observer muted (`rt.selfStorage`), so it never becomes a `storage` event | interaction analytics on (`InitOptions.analytics`, `rt.analytics.start()`): each page view and each `interactions` event; removed by `rt.analytics.stop()` | [src/analytics/index.ts](src/analytics/index.ts) `startAnalytics` | -| `localStorage` (read only) | `genclass` (kill switch), `genclass.telemetry` (opt-out) | init | [src/index.ts](src/index.ts) `killSwitch`, [src/telemetry/config.ts](src/telemetry/config.ts) | -| A module `Worker` (`genclass-model`) running onnxruntime-web; inline on the main thread only if the worker cannot start (`model.inlineFallback`, default true; `false`: no model then, status `skipped` / `worker-unavailable`) | | model load | [src/model/host.ts](src/model/host.ts) | +| `localStorage` (read only) | `genclass` (kill switch; read only when the URL has no `?genclass=` value, which takes precedence), `genclass.telemetry` (opt-out) | init | [src/index.ts](src/index.ts) `killSwitch`, [src/telemetry/config.ts](src/telemetry/config.ts) | +| A module `Worker` (`genclass-model`) running onnxruntime-web; inline on the main thread only if the worker cannot start (`model.inlineFallback`, default true; `false` (0.3.0; 0.2.x ignores it): no model then, status `skipped` / `worker-unavailable`) | | model load | [src/model/host.ts](src/model/host.ts) | | Memory | ≤ 64 response bodies of ≤ 256 KB (last good GET per identity), ≤ 4 MB of recent responses for coalescing, a 500-event history, 200 decisions, 1,000 audit entries | always | [src/observe/cache.ts](src/observe/cache.ts) | | Console | one line per detection or intervention (`report`), the telemetry notice, warnings | `report: "console"` (default) | [src/decide/report.ts](src/decide/report.ts) | | Network: the measured visual checks' chunk (`dist/visual/index.js`, from the same place as the runtime) | a dynamic `import()` | 1.5 s after `load` when `InitOptions.visual` is on for the page (development hosts by default), or when the devtools overlay mounts or `rt.checkNow()` runs | [src/visual/start.ts](src/visual/start.ts) | @@ -190,7 +206,7 @@ including the time spent waiting for a body, deferred re-decisions and `onBefore (`test/invariants/network.test.ts`, `test/invariants/state.test.ts` "the hold budget caps ..."). When the model is not ready, slow, erroring or over its rate limit, the subject proceeds unchanged (fail open). -**Measured visual checks** ([VISUAL.md](../../docs/runtime/VISUAL.md)) only read in every mode. Their runtime fixes need +**Measured visual checks** ([VISUAL.md](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/VISUAL.md)) only read in every mode. Their runtime fixes need `visual: { on: true, fix: true }` and guard or heal mode, and only ever do two things: add an owned stylesheet rule that sets a readable colour on one element (text measured below 1.5:1), or remove one leaked inline `pointer-events: none` or `overflow: hidden` declaration on `html` or `body` after recording it. Each is re-measured @@ -202,7 +218,7 @@ Every term can only lower the mode: `routes` rules, `sample`, the breaker (after the session drops to `observe`), a hidden tab (nothing new is held; background decisions are skipped in guard and heal, and deferred until the tab is visible in observe, where they can only detect) and `?genclass-mode`. Since 0.2.2 no URL or `localStorage` switch can raise it on a production host: `?genclass=guard` -or `heal` raises only with `debug: true` or on a development host (see [THREAT-MODEL.md](../../docs/runtime/THREAT-MODEL.md)). +or `heal` raises only with `debug: true` or on a development host (see [THREAT-MODEL.md](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/THREAT-MODEL.md)). ## What it never does @@ -230,13 +246,13 @@ proceeds or the chosen action runs. Every action is recorded (`rt.interventions( | Trigger | Action | Tier | What it does | Preconditions (besides the gate) | Undo | |---|---|---|---|---|---| | mutation | `discard` | guard | drops a held store write; decided after the write applied (hold budget expired), reverts exactly that write | registered store (not observed-only); held: `policy.holdWrites`; late revert ≤ 800 ms after it applied and nothing written since depends on it (`StoreHub.revertable`) | yes: the write is committed / re-applied | -| mutation | `defer` | guard | holds the write until related in-flight operations finish, then decides again | held write, at most 2 defers, within what is left of the hold budget | — (the write applies or is decided again) | +| mutation | `defer` | guard | holds the write until related in-flight operations finish, then decides again | held write, at most 2 defers, within what is left of the hold budget | not needed: the write applies or is decided again | | request | `coalesce` | guard | does not send; answers with the response of the identical request in flight or finished ≤ 2 s ago (`x-genclass: coalesced`) | fetch; same identity (method, URL, headers, body); a shareable buffered response (≤ 256 KB) | no: the request was not sent. Any method, POST included (double submits): protect or `deny` it where repeats are deliberate | | request | `delay` | guard | sends after 250 ms × 2^(recent failure streak), at most 8 s | fetch, XHR | no (latency only) | | request | `block` | heal | does not send; answers 503 with `x-genclass: blocked` | fetch, XHR | no | | request | `serve_cached` | heal | does not send; answers with the last good response for this identity (`x-genclass: cached`) | GET; a cached body (≤ 256 KB, this page's memory) | no | | delivery | `discard` | guard | delivers the response, then for 10 s drops the writes its causal chain makes over newer data | the response would overwrite newer data; at least one target field in a registered store | yes: the dropped values are restored | -| delivery | `defer` | guard | holds the response until related in-flight operations finish, then decides again | related work in flight; at most 2 defers; within the hold budget | — | +| delivery | `defer` | guard | holds the response until related in-flight operations finish, then decides again | related work in flight; at most 2 defers; within the hold budget | not needed: the response is delivered or decided again | | failure | `retry` | heal | sends the request again after 200 ms × 2^(attempt − 1) (≤ 5 s); the app gets that attempt's result | fetch; replayable body; fewer than 4 attempts; idempotent method or idempotency key | no | | failure | `serve_cached` | heal | answers the failed request with the last good response | fetch; GET; cached body | no | | stall | `hedge` | heal | sends a second identical request; the first good answer reaches the app | fetch; GET; replayable | no | @@ -292,7 +308,7 @@ source, the model's probabilities, the action proposed and the one that ran with (name, version, variant, device and the sha256 its file was verified against). `InitOptions.audit` sets the in-memory size (default 1,000) and a `sink` for your own logging. Query values in free text are replaced with "…". The trail is built from values the runtime already has: it never feeds a decision, never changes what the model reads and is never -sent anywhere by GenClass ([src/decide/audit.ts](src/decide/audit.ts); API: [API.md](../../docs/runtime/API.md)). +sent anywhere by GenClass ([src/decide/audit.ts](src/decide/audit.ts); API: [API.md](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/API.md)). ## Machine-checked inventory diff --git a/packages/runtime/README.md b/packages/runtime/README.md index f79b341e..72e9b007 100644 --- a/packages/runtime/README.md +++ b/packages/runtime/README.md @@ -1,107 +1,97 @@ # @genclass/runtime [![npm](https://img.shields.io/npm/v/@genclass/runtime/latest?label=npm)](https://www.npmjs.com/package/@genclass/runtime) -[![license](https://img.shields.io/badge/license-Apache--2.0-blue)](LICENSE) -![model](https://img.shields.io/badge/model-runs%20in%20the%20browser-brightgreen) +[![npm beta](https://img.shields.io/npm/v/@genclass/runtime/beta?label=beta)](https://www.npmjs.com/package/@genclass/runtime?activeTab=versions) +[![CI](https://github.com/genclass-dev/GenClass-lib/actions/workflows/ci.yml/badge.svg)](https://github.com/genclass-dev/GenClass-lib/actions/workflows/ci.yml) +[![license](https://img.shields.io/badge/license-Apache--2.0-blue)](https://github.com/genclass-dev/GenClass-lib/blob/main/LICENSE) -### Your app's race conditions, stale responses and double submits, caught while they happen. +### Reports race conditions, stale responses and double submits in your web app while they happen. -The bugs your tests never catch are the ones that depend on timing: a slow response landing after a fast one, a -button clicked twice, a save racing an edit, an endpoint that starts failing at 2 a.m. +GenClass Runtime (`@genclass/runtime`) is an open-source (Apache-2.0) browser library that reports race conditions in +web apps while they happen: stale responses that land over newer data, duplicate requests such as double submits, +inconsistent state and failure storms. A small model runs on the user's device, in a Web Worker on WebGPU or WASM, +and each report comes with its evidence. By default it only reports (`observe`): it never holds, delays or changes a +request, a response or a state write. The opt-in `guard` mode can prevent a few of these failures, and `heal` is +experimental. -GenClass Runtime records what your app does: user actions, async operations and their causes, fetch/XHR/WebSocket/ -EventSource traffic, store writes with per-field versions, errors and timing. It computes generic facts about each -write, request and response. When a situation looks risky, a small GenClass model running in the browser (WebGPU or -WASM, in a Web Worker) answers two questions: what is happening, and which of the available actions is best. The -runtime has no list of known bugs: triage picks the situations, the model decides. By default it only reports; in -`guard` mode it can also stop the failure before your users see it. +It is for front-end apps in React, Next.js, Vue, SvelteKit, Angular, Solid or plain HTML, and for the timing bugs +tests rarely catch: a slow response landing after a fast one, a button clicked twice, a save racing an edit, an +endpoint that starts failing at 2 a.m. Stale responses are reported for state it can see +([which](#automatic-state-discovery)). It runs in the browser only: during SSR or in Node, `/auto` installs nothing. -**One line covers the whole app.** First in ``: +[Docs](https://genclass.dev/docs) · [Questions](https://github.com/genclass-dev/GenClass-lib/blob/main/FAQ.md) · +[Measured results](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/RESULTS.md) · +[API](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/API.md) · +[Changelog](https://github.com/genclass-dev/GenClass-lib/blob/main/packages/runtime/CHANGELOG.md) · +[Discord](https://discord.gg/kN9kRnFtZm) -```html - +## Quick start + +```bash +npx @genclass/runtime@beta init # finds your framework, shows the diff, then adds one import ``` -(pin the version: the unversioned URL may serve an older release from the CDN cache) or first in your entry file (any bundler; Next.js: `instrumentation-client.ts`): +Once you approve the diff, `init` also creates your app's free dashboard at genclass.dev and sends it your +`package.json` `name`; `--no-token`, `--token`, `--no-telemetry` and `--dry-run` create no dashboard and never contact +genclass.dev ([Your dashboard](#your-dashboard)). + +Or by hand: + +```bash +npm install @genclass/runtime@beta +``` ```ts -import "@genclass/runtime/auto"; // after npm install @genclass/runtime@beta +import "@genclass/runtime/auto"; // first line of your entry file (Next.js 15.3+: instrumentation-client.ts) ``` -Use the newest line, `0.3.0-beta.1` (npm dist-tag `beta`): every command in this README names `@beta`. Without it, -npm installs its `latest`, the older `0.2.x`, which has no token, dashboard, interaction analytics, visual checks or -agent link and ignores those options without a word (a TypeScript app gets a type error for `analytics` and -`token`). `0.2.x` is the exception, for an app that cannot take a prerelease. +Or with no build step, first in ``: + +```html + +``` That is the whole setup. Network traffic, user actions, errors and timing are observed, and so is your app's state: -React component state (`useState`, `useReducer`, `useSyncExternalStore`, class state), Redux and Redux Toolkit -stores, and Zustand stores with `devtools` are found automatically, with no store to register -([Automatic state discovery](#automatic-state-discovery)). It starts in observe mode: it reports, and never changes -what your app does (see Known limitations). `npx @genclass/runtime@beta init` writes the line for you. - -**New in `0.3.0`:** a per-app dashboard on genclass.dev fed by an app token, in a GenClass account -(`npx @genclass/runtime@beta login`) or behind a private link, and `protect(name, fn)` with `scope: "functions"` to point -GenClass at a few functions instead of the whole app. See [Your dashboard](#your-dashboard) and -[Function-specific: protect()](#function-specific-protect); `0.2.x` ignores a token and has no `login` command or -`protect` export. - -> **Privacy notice: anonymous diagnostics are on by default.** `GenClass.init()` in a browser sends GenClass's -> decisions and counts, without any text from your page, to the GenClass maintainers to improve the model; it is -> off on `localhost` and other local hosts (`0.2.1` sends from them too, by mistake; fixed in `0.2.2`), and the situation text is sent only if you opt in. Opt out with -> `npx @genclass/runtime@beta init --no-telemetry`, `GenClass.init({ telemetry: false })`, `?genclass=no-telemetry`, or -> `localStorage["genclass.telemetry"] = "off"`; -> browsers sending Global Privacy Control are never collected. Details: [Privacy and telemetry](#privacy-and-telemetry) -> and [TELEMETRY.md](https://github.com/genclass-dev/GenClass-lib/blob/main/packages/runtime/TELEMETRY.md). - -> **Status: `0.3.0-beta.1`, the newest line (npm `beta` dist-tag: `npm install @genclass/runtime@beta`; npm's -> `latest` is still the older `0.2.x`, the exception for apps that cannot take a prerelease), with model -> `@genclass/runtime-model@0.2.0`.** -> -> - **The model loads by default.** A default `GenClass.init()` loads `@genclass/runtime-model@0.2.0` (10.2 MB q8, -> `genclass-runtime-r17` 2.0.0-rc4t, run `r17-v2dT`, with a gain gate and three aggressiveness profiles) from -> jsDelivr at idle, in a Web Worker, and caches it. What that costs (memory, download) is under [Costs](#costs); -> self-hosting (for a Content-Security-Policy) under -> [Content-Security-Policy and self-hosting](#content-security-policy-and-self-hosting). -> - **Observe is the default.** The model diagnoses salient situations and the runtime reports likely problems; it -> never changes what your app does and never holds or delays a response. `guard` is opt-in and `heal` is -> experimental. Once you opt in, `aggressiveness` (`"cautious"`, `"balanced"` (default), `"eager"`) sets how eager -> it acts. -> - **The model is precise when it acts, but misses most problems.** On held-out simulated apps, at `balanced`: -> guard intervened wrongly on 0.13% of cases and heal on 0.59%, slightly above our 0.1% / 0.5% targets; at -> `cautious`, 0.005% and 0.26%, under them. No wrong interventions were seen on held-out real apps. At `balanced`, -> guard acts on under 8% of the cases where acting would help. Details: [Model quality](#model-quality) and the -> [model card](https://www.npmjs.com/package/@genclass/runtime-model). -> - **Older versions.** `0.1.0-beta.3` has the same runtime with these problems (fixed in `0.2.0`): app builds -> carried 41 MB of unused ONNX Runtime wasm, a CSP-blocked model failed silently, `init` could not turn telemetry -> off, and `status.scope.mode` read "heal" in observe mode. `0.1.0-beta.2` is `0.1.0-beta.3` without telemetry. `0.1.0-beta.1` has model 0.2.0 and the new options but defaults to `guard`, and lacks the -> observe-delivery, redaction and install fixes of `0.1.0-beta.0` (all back in `0.1.0-beta.2`). `0.1.0-beta.0` -> ships model 0.1.0. `0.1.0-alpha.1` and `0.1.0-alpha.0` (v1 runtime: guard by default, holds store writes, `NaN` -> crash) predate the model; do not use them. -> -> Measured results: [RESULTS.md](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/RESULTS.md). - -Questions, bug reports and false detections: [GitHub Issues](https://github.com/genclass-dev/GenClass-lib/issues) or the [GenClass Discord](https://discord.gg/kN9kRnFtZm). +React component state, Redux and Redux Toolkit stores, and Zustand stores with `devtools` are found automatically +([Automatic state discovery](#automatic-state-discovery)). Pinia, Svelte stores, signals and a plain HTML page's DOM +are not: there, stale responses are reported only for stores you register (double submits need no state). + +**Versions:** this README describes `0.3.0` with model `@genclass/runtime-model@0.2.0`. The newest line is +`0.3.0-beta.4`, npm's `beta` dist-tag (`0.3.0-beta.1` to `beta.4` were released between 2026-10-22 and 2026-10-27); +npm's `latest` is still the older `0.2.2` (`npm view @genclass/runtime dist-tags` shows them now). Every npm and npx +command here names `@beta`: without it npm installs `0.2.x`, which has no token, dashboard, interaction analytics, +visual checks or agent link and ignores those options without a word (a TypeScript app gets a type error for +`analytics` and `token`), and whose CLI has no `login`, `dashboard`, `scan` or `--sink`. `0.2.x` is the exception, +for an app that cannot take a prerelease. Pin the script tag: the unversioned URL serves `latest`. A feature marked +with a beta number (`0.3.0-beta.2`, say) is not in the betas before it. + +> **Anonymous diagnostics are on by default** on public hosts: GenClass's decisions and counts, no text from your +> page, sent to the GenClass maintainers to improve the model. Off on `localhost` and other local hosts from `0.2.2` +> (`0.1.0-beta.3` to `0.2.1` send from them too). Opt out with `GenClass.init({ telemetry: false })`, +> `?genclass=no-telemetry`, `localStorage["genclass.telemetry"] = "off"` or +> `npx @genclass/runtime@beta init --no-telemetry`; browsers sending Global Privacy Control are never collected. +> Details: [Privacy and telemetry](#privacy-and-telemetry) and [TELEMETRY.md](https://github.com/genclass-dev/GenClass-lib/blob/main/packages/runtime/TELEMETRY.md). ## Contents -[Install](#install) · [Audit a page](#audit-a-page) · [Your coding agent](#your-coding-agent-replay-and-explore) · [Function-specific: protect()](#function-specific-protect) · [Your dashboard](#your-dashboard) · -[Automatic state discovery](#automatic-state-discovery) · -[Why it's an easy yes](#why-its-an-easy-yes-measured) · [Modes](#modes) · -[What it looks for](#what-it-looks-for) · [Visual and CSS checks](#visual-and-css-checks-new-in-030) · -[State it can protect](#state-it-can-protect) · [Ask it questions](#ask-it-questions) · [Observability](#observability) · -[See it where you already look](#see-it-where-you-already-look) · [Extend it](#extend-it) · -[Model quality](#model-quality) · [Performance](#performance) · [Costs](#costs) · -[Content-Security-Policy and self-hosting](#content-security-policy-and-self-hosting) · [Privacy and telemetry](#privacy-and-telemetry) · -[Known limitations](#known-limitations) · [API reference](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/API.md) · -[Interception surface](INTERCEPTION.md) · [Security](https://github.com/genclass-dev/GenClass-lib/blob/main/SECURITY.md) +[At a glance](#at-a-glance) · [Modes](#modes) · +[What it looks for](#what-it-looks-for) · [Safety and cost](#safety-and-cost-measured) · +[Model quality](#model-quality) · [Costs](#costs) · [Privacy and telemetry](#privacy-and-telemetry) · +[Known limitations](#known-limitations) · [Install](#install) · +[For AI coding agents](#for-ai-coding-agents-installing-genclass) · [Audit a page](#audit-a-page) · +[Your coding agent](#your-coding-agent-replay-and-explore) · +[protect()](#function-specific-protect) · [Your dashboard](#your-dashboard) · +[Automatic state discovery](#automatic-state-discovery) · [State it can protect](#state-it-can-protect) · +[Visual and CSS checks](#visual-and-css-checks-new-in-030) · [Observability](#observability) · +[Configuration](#configuration) · [Performance](#performance) · +[Content-Security-Policy and self-hosting](#content-security-policy-and-self-hosting) ## What it looks like -Run your app as usual. When the model flags something, the console gets one plain-English line per detection or -intervention, followed by a collapsed group with the evidence. Here a typeahead's slow response for "rea" lands -after the response for "reac". This is real output (format unchanged since `0.1.0-beta.0`; q8 on WASM, in Node), -in the default observe mode. The response reaches the app at once, exactly as without GenClass, and the decision is -made in the background, for the report only: +When the model flags something, the console gets one plain-English line per detection or intervention, followed by a +collapsed group with the evidence. Here a typeahead's slow response for "rea" lands after the response for "reac". +This is real output (format unchanged since `0.1.0-beta.0`; q8 on WASM, in Node) in the default observe mode. The +response reaches the app at once, exactly as without GenClass, and the decision is made in the background: ``` [GenClass] Model ready (genclass-runtime-r17, wasm, q8, 0.43s). Mode: observe. @@ -110,8 +100,8 @@ started (version 1 → 3), last 0.69s ago by GET /api/search?q=reac (#8), which user action (#7). (stale, 0.96) ``` -The same run in guard mode. The model's `discard` (0.98) passed model 0.1.0's delivery gate (0.80), so the results -for "reac" stayed: +The same run in guard mode (captured with model 0.1.0; how often model 0.2.0 acts: [Model quality](#model-quality)). +The results for "reac" stayed: ``` [GenClass] Prevented a stale response: search.results was written twice by other operations since its operation (#6) @@ -120,171 +110,368 @@ user action (#7). Delivered the response to GET /api/search?q=rea (#6) and dropp newer data (search.results). (stale, 0.97; discard 0.98) ``` -In production these lines print in your users' browsers. From `0.3.0` you can send them to your own endpoint or your -log shipper as well: `npx @genclass/runtime@beta init --sink webhook:/genclass`, or `data-sink="webhook:/genclass"` on -the script tag ([See it where you already look](#see-it-where-you-already-look)). Forwarding to Sentry, PostHog or -Datadog is not part of the runtime; on every genclass.dev plan, Free included, the GenClass collector forwards detections -to them once you turn it on ([genclass.dev/pricing](https://genclass.dev/pricing)). +## At a glance -## Install +| | | +|---|---| +| Default | `observe`: reports only; never holds, delays or changes a request, response or state write | +| Opt-in | `guard`: drop a stale response's writes, reuse a duplicate's result, defer, back off. `heal` (experimental): also retry, serve cached, block, hedge, roll back, resync | +| Detects (observe, `balanced`, held-out real apps, scored offline) | 75% of duplicate submits, 30% of stale overwrites (`situation-v2.3` slice; 21% on `situation-v2`), 6% of broken state, with the right diagnosis; report gate fitted to keep false detections at or below 2% (real apps) / 3% (simulated) of clean decisions, met on held-out test (exact rate not published) | +| Acts (guard, `balanced`) | on 7.8% of clear simulated and 7.5% of actionable real-app cases where acting would help; false interventions 0.13% (held-out simulated), 0.00% (held-out real apps) | +| First-load JavaScript | 108.25 KB gzip with `/auto`, 100.13 KB with `GenClass.init()` (`0.3.0-beta.0`; [Performance](#performance)) | +| Model | 10.2 MB (q8: WASM, and WebGPU without `shader-f16`) or 13.6 MB (fp16: WebGPU with `shader-f16`), downloaded once at idle and cached; about 180 ms per decision on one WASM thread (estimated in Node, not in a browser); not loaded on devices reporting 2 GB of memory or less, or on Save-Data | +| Memory | +190 to 280 MB in the renderer while the model is loaded (one pilot app, `0.1.0-beta.3`) | +| Browsers | measured in Chromium only | +| Data sent | anonymous diagnostics without page text, on by default on public hosts, off on local hosts; `telemetry: false` turns them off | +| License | Apache-2.0, the model package too | + +Sources: [RESULTS.md](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/RESULTS.md#current-numbers-in-one-place) +§1, §2, §10, §11 and §17, and the +[model card](https://github.com/genclass-dev/GenClass-lib/blob/main/packages/runtime-model/MODEL_CARD.md). -Pick one. All of them start the same runtime, in observe mode unless you choose otherwise. While `0.3.0` is in beta, -every npm and npx command here names the `@beta` dist-tag: without it they get the stable `0.2.x`, whose runtime -ignores a token and whose CLI has no `login`, `connect`, `whoami`, `logout`, `--token` or `--sink`. +## Modes -**1. One command.** `init` finds your framework and package manager, installs the package, adds one import as the -first line of your entry file (plus a line that loads the devtools overlay in development only), and shows you the -diff before writing anything. Running it again changes nothing; running it with another `--mode` switches the -import it added. +| mode | what it does | non-passive actions | +|---|---|---| +| `observe` (**default**) | Finds and reports what it sees and what it would have done. Never holds a request or a response, never runs an action. | none | +| `guard` (opt-in) | Also prevents failures with minimal guard-tier actions: `discard`, `defer`, `coalesce`, `delay` (drop a stale response's writes, reuse a duplicate's result, back off). Only when the model's gain gate says acting beats doing nothing by the profile's margin and the top diagnosis is not `expected` ([model card](https://github.com/genclass-dev/GenClass-lib/blob/main/packages/runtime-model/MODEL_CARD.md)). | guard tier | +| `heal` (**experimental**) | Also recovers: `retry`, `serve_cached`, `block`, `hedge`, `rollback`, `resync`, plus your own actions. | guard + heal tier | -```bash -npx @genclass/runtime@beta init # observe; shows the diff, asks, then writes -npx @genclass/runtime@beta init --mode guard # let GenClass act (also: --mode heal); run again later to switch -npx @genclass/runtime@beta init --yes # no questions -npx @genclass/runtime@beta init --dry-run # show the diff, write nothing -npx @genclass/runtime@beta remove # undo exactly what init added -``` +`GenClass.init()` with no `mode`, `/auto` and the script tag without `data-mode` observe only (`0.1.0-alpha.0` and +`0.1.0-beta.1` defaulted to `guard`). Pass `mode: "guard"` or `init --mode guard` to let it act. -Other flags: `--no-install`, `--no-devtools`, `--no-network`, `--cwd `, `--cdn `, `--no-sri` and `--sync` -(plain HTML), `remove --keep-package`. +**Detections in observe (`0.3.0`).** A detection is *likely* when the model's top diagnosis is not `expected` and its +probability reaches the report gate (0.90 at `balanced` with model 0.2.0): one console line with its evidence. Observe +also reports *possible* detections, from the `eager` report value (0.70) up to the gate: on `console.debug`, counted +apart (`summary().possibleDetections`) and sent in the anonymous diagnostics as `detect` events with `tier: "possible"`. +`rt.on("detect")` and `report` get them only when you set `detect: "possible"` explicitly, sinks only with +`possible: true`; `detect: "likely"` turns them off. Situations raised while the model loads or the tab is hidden are +decided once it can (at most 32 wait), at the possible tier at most +([RESULTS.md §11](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/RESULTS.md#11-observe-mode-detection-in-030-the-possible-tier-and-deferred-situations)). -**Which version.** `init` (and `login`, `connect`) first prints its own version and the npm dist-tag it came from, -asks npm's dist-tags once (`npm view @genclass/runtime dist-tags`, at most 3 s; never with `--dry-run`, -`--no-network`, `--no-token` or `--no-telemetry`, nothing when offline) and says when a newer release is on npm, with -the command that runs it: +### How eager should it be? -``` - ✔ Version @genclass/runtime 0.3.0-beta.1 (npm dist-tag beta) - ▲ Newer @genclass/runtime 0.3.0-beta.2 is on npm (dist-tag beta): run npx @genclass/runtime@beta init instead +Once you opt into `guard` or `heal`, GenClass only acts when the model is confident enough. You choose how confident: + +```ts +GenClass.init({ aggressiveness: "cautious" }); // "cautious" | "balanced" (default) | "eager", or a number 0 to 1 +GenClass.runtime.setAggressiveness("eager"); // change it later (the devtools have a selector too) ``` -It installs its own version (`@genclass/runtime@^0.3.0-beta.1`), and a project that already has an older one (a -`0.2.x` from `npx @genclass/runtime init` without `@beta`) is upgraded, after the same diff and question. Every -command it prints names `@beta` while it is a prerelease. +`cautious` acts less often; `eager` fixes more at a higher risk of acting when it did not need to (numbers under +[Model quality](#model-quality)). In observe, aggressiveness only moves the report gate: 0.95 / 0.90 / 0.70 for +cautious / balanced / eager. The thresholds ship in the model's `meta.json`, fitted on held-out dev data. +`policy.thresholds` overrides them (with model 0.2.0's gain gate, `guard` and `heal` are margins in cost units, not +probabilities); `rt.gates(trigger)` shows what is in force. -On a plain HTML site `init` writes the script tag with `defer` (after `0.3.0-beta.1`), so a slow or blocked CDN never -holds up the page. GenClass then starts once the page is parsed: before your `defer` and `type="module"` scripts and -before `DOMContentLoaded`, but after classic scripts that have neither `defer` nor `async`. Requests and errors those -scripts cause while the page is still loading are not seen, and a library loaded that way before GenClass (a UMD React -from a CDN) is not discovered; a click handler's `fetch` is seen either way, and the model still loads at idle after -`load`, as before. `--sync` writes the blocking tag of earlier versions, which runs before every script. `remove` takes -the tag out whatever attributes you add to it (`defer`, `async`, `nonce`). When a project looks mostly static (no -`fetch`, XHR, WebSocket or EventSource call in its sources, or a plain HTML page with one script), `init` says so in one -sentence and points to the visual checks and the page scan (`npx @genclass/runtime@beta scan `). +**Limits and switches.** Actions are limited to 60 per minute, 10 per minute on the same subject and 200 per session +(`policy.actionLimits`). `?genclass=off` (or `localStorage.genclass = "off"`) installs nothing. `GenClass.init()` +never throws, and outside a browser it returns a runtime with no observers and no model; for tests, use +`createRuntime(options)`. + +## What it looks for -**Token and dashboard** (`0.3.0`; `init` in `0.2.x` creates no token). On a new setup, once you approve the -diff (or pass `--yes`), `init` also creates a project for your app at genclass.dev: it sends `package.json`'s `name` -to `https://genclass.dev/api/projects`, writes the token it gets back into what it adds (the diff shows -`gc_new-token-created-when-you-apply` in its place until then), and prints your private dashboard link: +Actions other than flagging need `guard` or `heal`: + +| situation | example | what GenClass can do | +|---|---|---| +| Stale response | an old search response lands after a newer one | deliver it but drop its writes over newer data, or defer it (guard) | +| Race / conflict | a server echo would overwrite what the user just typed | drop the stale part, defer (guard) | +| Duplicate | a double click sends the same order twice | reuse the first response (`coalesce`, guard) | +| Inconsistent state | the cart total no longer equals the sum of the lines | roll back or resync (heal) | +| Failure pattern | an endpoint fails 5 times in a row | serve the last good response (heal) | +| Transient failure | a one-off 503 | retry (heal) | +| Slow or flooding | 8× slower than usual; a render loop hammering an API | hedge (heal), delay (guard) | +| Unusual behaviour | an operation writes different fields than it usually does | flag; roll back (heal) | +| Your own question | "is now a good moment to start the upload?" | `rt.ask()` / `rt.decide()` | + +The runtime decides at nine **triggers**: `delivery` (a response or WebSocket/EventSource message is about to reach +the app), `request`, `failure`, `stall`, `mutation` (a salient store write no delivery decision covered), +`inconsistency` (a learned relation such as `total == sum(items[*].price × qty)` broke), `transition` (an operation +wrote different fields than usual), `error` and `ask` (your own question, always asked). By default the others reach +the model only when salient; the rest is a cheap fact computation. The model gives a diagnosis (`expected`, `stale`, +`conflict`, `duplicate`, `inconsistent`, `failing`, `slow`, `overload`, `unusual` or `transient`) and picks an +action. The runtime has no list of known bugs: triage picks the situations, the model decides. Every trigger's actions: +[API.md](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/API.md#triggers-and-actions). + +**Delivery decisions.** In guard and heal, GenClass decides about responses *at the network boundary*, before the app +sees them (observe decides in the background), and never holds or reorders the app's store writes. A delivery is +salient only when a field it is predicted to write already holds **newer applied data**, or it would **revert a +pending local change** or **replace text the user typed**. The model then reads facts such as these: ``` - Your dashboard https://genclass.dev/dashboard/... - Keep this link private; it is the only way to open your dashboard. - Free for every app: what GenClass found, on which pages, and what it did, and (with analytics: true) - how visitors use each page. Print this link again any time: npx @genclass/runtime@beta dashboard - Saved in .genclass.local (added to .gitignore). +The response has search.results = 1 item ["rea-1"]: neither the current value 1 item ["react-1"], nor the value when #4 started. +The response to GET /api/search?q=rea (#4) arrived after 1.80s (200); the app has not seen it yet. +This request comes from user typed "rea" into input "Search" (#3), started 1.80s ago. ``` -The link is saved in `.genclass.local` (with the token) before init writes anything else, so a failed write cannot -lose it. init asks git whether that file is ignored; if not, it adds it to the project's `.gitignore` (inside a git -repository without one there, such as a monorepo package, it creates one with that line). `remove` leaves the file -and that line alone. Declining, running without a terminal and without `--yes`, and `--dry-run` send nothing and -create nothing. Running `init` again reuses the token (no new project); on a setup made -without one it never creates a project as a side effect: pass `--token`, or sign in and run `connect`. Signed in -(`npx @genclass/runtime@beta login`), `init` first asks whether the new app goes into your GenClass account instead of -getting a private link ([Your dashboard](#your-dashboard)). +Clean in-order typeahead and autosave make no model calls and hold nothing. The model does not catch logic that is +consistently wrong, CSS or security bugs, and it never rewrites your code; some visual and CSS bugs are covered by +[measured checks](#visual-and-css-checks-new-in-030). Opt-in flow expectations (`learn: { expectations: true }`) +report a usual step that did not happen; off by default until their false-report rate is measured +([EXPECTATIONS.md](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/EXPECTATIONS.md)). + + + +## Safety and cost (measured) + +These hold for the runtime; whether the model's decisions are good is under [Model quality](#model-quality). + +- **It is not expected to make a working app worse.** Decisions about responses happen at the network boundary, which + looks like ordinary latency to your app, and the default never holds or reorders your app's own store writes. With a + model that never intervenes (heal mode, compared against observe mode), GenClass changed the outcome in **0 of 396** + clean runs across **66 real apps in 23 frameworks** (including 14 open-source RealWorld front-ends), 6 seeds each + (runtime as of `0.1.0-alpha.1`; not re-run since). The check compares final page text (inputs and alerts excluded) and + server state, not request timing, store contents or observe against no GenClass. With network chaos, 3 of 198 runs + differed: a request held about 25 ms changed the simulated network's draws + ([RESULTS.md §4](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/RESULTS.md#4-never-make-a-correct-app-worse-runtime-design-comparison)). +- **Every mode, on seven idiomatic apps** (runtime commit `d04eff3`, the 0.1.0-beta.4 code: the matrix has not run + 0.2.0's other changes or anything since; model 0.2.0 at `balanced`, 2026-10-10). The + [compatibility matrix](https://github.com/genclass-dev/GenClass-lib/blob/main/compat/RESULTS.md) (six frameworks, + React, Next.js, Vue, SvelteKit, Angular and Solid, and plain HTML; 15 data layers; 10 seeds per cell) found 0 bugs + introduced in 3,510 runs with GenClass and 0 actions in 1,760 guard and heal runs on correct apps. Observe left page + and server state identical to the run without GenClass in every cell but one, an app's own timing race (SWR's + optimistic rollback) that came out right with GenClass, with no action taken: observe holds nothing, but its + main-thread work is not zero. Observe reported 49 false findings in 880 correct-app runs, all in one polling and + fan-out scenario. +- **Normal traffic costs little.** The model is consulted only for salient situations ([Performance](#performance)). +- **The model runs in the browser**, cached after the first load: no inference server, no API key, and decisions + never wait on the network. +- **You can see what it did, and undo it.** Every detection and action gets one console line with its evidence; + `rt.explain(id)` shows what the model read; discards and rollbacks can be undone. + +## Model quality + +The default model is `genclass-runtime-r17` 2.0.0-rc4t (`@genclass/runtime-model@0.2.0`, training run `r17-v2dT`): an +encoder of 8.1M parameters plus 0.7M of heads, trained on simulated apps and on real apps driven in headless +Chromium. It ships a gain gate with one fitted profile per aggressiveness level. FIR, the false-intervention rate, is +the share of cases where doing nothing was best but the gate fired anyway. Held-out test, +[RESULTS.md §1](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/RESULTS.md#1-model-stages-compared-held-out-simulated-apps): + +| profile | guard FIR | guard recall (clear / real) | heal FIR | heal recall (clear / real) | report threshold | detected (duplicate / stale / broken) | +|---|---|---|---|---|---|---| +| `cautious` | 0.005% | 2.4% / 1.4% | 0.26% | 3.7% / 3.8% | 0.95 | 72% / 28% / 0% | +| **`balanced` (default)** | 0.13% | 7.8% / 7.5% | 0.59% | 7.1% / 10.4% | 0.90 | 75% / 30% / 6% | +| `eager` | 0.54% | 14.2% / 21.0% | 1.84% | 17.1% / 24.2% | 0.70 | 77% / 45% / 13% | + +FIR is on simulated apps; on held-out real apps it was 0.00% for every profile. The targets are 0.1% (guard) and 0.5% +(heal): `cautious` meets both, `balanced` is slightly over both, `eager` trades well over them for about twice +`balanced`'s recall. Pick `cautious` if a wrong intervention costs more than a missed one. + +- **Observe (default).** At `balanced`, scored offline per situation on held-out real apps, it reports 75% of duplicate + submits, 30% of stale overwrites (`situation-v2.3`; 21% on `situation-v2`) and 6% of broken state with the right + diagnosis. End to end in the compatibility matrix (commit `d04eff3`), it made 10 duplicate-submit detections + over 10 seeds in each of the 14 data layers with that bug, and 10 stale-response detections in all over the 110 + stale-typeahead runs (none in 7 of those 11 layers), with 49 false findings in 880 correct-app runs. The 0.2.0 + report gate was fitted on dev data to keep false detections at or below 2% (real apps) / 3% (simulated) of clean + decisions, and met that on held-out test; the exact rate is not published (RESULTS.md §11; model 0.1.0's exact rates: + §1). One recurring pattern in your app can produce many more; `policy.thresholds.report` raises the bar. With the + possible tier, the fit held false detections to at most 5% / 6%; the share of possible ones that are false is not + measured. Three 0.3.0 runtime fixes (overwritten writes decided in observe on slow devices, the body wait, the idle + unload) can add likely reports, false ones included, at a rate not measured yet; `detect: "likely"` does not turn them + off. +- **Guard (opt-in).** It rarely acts and misses most problems (the `balanced` row: it acts on 7.8% of clear + simulated cases at a FIR of 0.13%). Precision among its actions was not measured for 0.2.0. +- **Heal (experimental).** Its FIR at `balanced`, 0.59%, is over the 0.5% target, and it acts on 7.1% of clear + simulated cases. +- **Not yet met:** diagnosis ≥ 95% and clear-case recall ≥ 80%; recall is the open problem (training data: + RESULTS.md §6). +- **Format.** Trained on `situation-v2` and `situation-v2.3` data; this runtime renders `situation-v2.3`. + +On the six demo apps under network chaos, the shipped gates did not lower the bug rate (72% with GenClass off, 70% +observe, 72% guard, 73% heal), with 0 false interventions on clean trials +([RESULTS.md §5](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/RESULTS.md#5-demos)). Its main +value today is observe mode: plain-English detections with the evidence. + +## Costs + +Measured on a pilot app (2026-10-09: a Next.js 16 ordering site, production build, Playwright Chromium at 390×844, +4× CPU throttling, medians of 5 runs, observe, `0.1.0-beta.3` with model 0.2.0) against the same build with +`?genclass=off` ([RESULTS.md §17](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/RESULTS.md#17-browser-cost-on-a-pilot-app-2026-10-09-and-2026-10-10)): + +| | with GenClass | +|---|---| +| JavaScript on every page (first load, gzip) | +97 KB with `0.1.0-beta.3` (today: [Performance](#performance)) | +| Main thread: load event, LCP, long tasks | no measurable change; main JS heap +0.4 MB | +| **Memory: renderer process (holds the model worker)** | **+190 to 280 MB RSS** while the model is loaded (cold and warm) | +| **First visit download** | **about 13 MB** on WASM after the load event (12.7 MB measured: q8 model, ONNX Runtime wasm, tokenizer); WebGPU: a 5.5 MB brotli ONNX Runtime build, and with `shader-f16` the 13.6 MB fp16 model instead of the q8 | +| Later visits | about 1 KB (`model.json` revalidated); the rest comes from Cache Storage, checked with sha256 | +| Model load | WASM (1 thread): 0.9 to 1.3 s cold, 0.4 s warm. WebGPU (Apple M-series): 12 s cold, 2 s warm | + +**On phones** the memory and the download are what matter. **By default the model is not loaded on devices that +report 2 GB of memory or less, nor on Save-Data connections** (since `0.3.0`; `model.loadIf` defaults to +`{ minDeviceMemoryGB: 4, saveData: "skip" }`; Safari and Firefox do not report device memory and load). The runtime +then keeps observing without a model (nothing is detected or prevented on that device) and says why in one console +line. `model.loadIf` also takes a function, such as `(env) => !env.mobile` for no model on phones; +`model.preload: "lazy"` downloads on the first salient situation, and `model.unloadAfterIdleMs` frees the worker's +memory (guard and heal actions after a reload: FIR not measured, RESULTS.md §11). `model: false` keeps the runtime +observing with no download and no worker. + +## Privacy and telemetry + +- **Decisions are local.** Situations are built and decided in the browser; nothing is sent to make a decision. +- **Anonymous diagnostics are on by default** with `GenClass.init()` in a browser (and so with `/auto`, the script + tag and `init` setups), except on local and private hosts (`localhost`, `*.localhost`, `*.local`, `*.test`, + `127.0.0.1`, `::1`, private IP ranges), in Node/SSR, and in `createRuntime()`. `0.1.0-beta.3` to `0.2.1` send from + local hosts too (fixed in `0.2.2`), and `0.1.0-beta.3` to `0.2.0` send the redacted situation text by default. They + go to the GenClass maintainers' collector + ([`telemetry-worker/`](https://github.com/genclass-dev/GenClass-lib/tree/main/telemetry-worker), a Cloudflare + Worker writing to a private R2 bucket) and are used to improve the model. One console notice per page says it is on. +- **By default no text from your page is sent.** Sent: a random per-page session id, versions, mode, device class, + the app's hostname and id-normalised path, and per decision (every detection and every decision whose diagnosis is + not `expected`, every action and veto; of the routine ones, one in four and at most 200 per page load; counts are + exact) the trigger, the diagnosis and its confidence, the action and gate, the outcome and latency. The collector + adds the visitor's coarse country (two letters) and stores no IP address or user agent. +- **Opt out:** `GenClass.init({ telemetry: false })`, `?genclass=no-telemetry`, + `localStorage.setItem("genclass.telemetry", "off")`, or `npx @genclass/runtime@beta init --no-telemetry`, which + writes the opt-out into your project (`telemetry=off` in the meta tag or `data-telemetry="off"` on the script tag + do the same). Browsers sending Global Privacy Control are never collected. `runtime.telemetry` says whether it is + on and why not. +- **The situation text the model read is sent only if you opt in** with + `GenClass.init({ telemetry: { include: { situation: true } } })`. It is redacted by field meaning, but can still + contain route names with query strings, short error messages and short field values; read TELEMETRY.md first. +- **With a token** (`0.3.0`), each batch also carries your app's public token, so the same diagnostics show on your + app's private dashboard, and decisions about `protect()`ed functions carry the function's name. Without a token + neither is sent; without telemetry, a token does nothing. +- **Interaction analytics are off unless you turn them on** (new in `0.3.0-beta.1`; `0.3.0-beta.0` does not have + them). `GenClass.init({ token, analytics: true })` also sends, per page view, the route, engaged time, scroll depth, + and clicks, rage clicks and dead clicks per element (the element named from markup, never its text), and from + `0.3.0-beta.2`, within one page load, how each view was reached, the route before it with the time between, the + first element used on it, and where the page load ended with its routes in order (at most 10). They need + telemetry on and the app's token (or your own `telemetry.endpoint`); no visitor id exists, and the one thing stored + on the device is the visit flag in `localStorage` (from `0.3.0-beta.2`: it links the page loads of a visit and says + whether the browser visited before; `runtime.analytics.stop()` removes it); they never feed a decision or the + model. Every opt-out above turns them off. Where consent is needed (the EU, EEA and UK), leave `analytics` unset and + call `runtime.analytics.start()` once the visitor consents. Details: + [TELEMETRY.md](https://github.com/genclass-dev/GenClass-lib/blob/main/packages/runtime/TELEMETRY.md#interaction-analytics). +- **The measured visual checks send nothing**, and sinks send only where you point them. +- **The CLI** contacts genclass.dev only for the command you run: `init` creates a dashboard only after you approve + its diff (it sends `package.json`'s `name`); `login`, `connect`, `whoami`, `logout` and, signed in without a private + link in the folder, `dashboard` talk to your account + ([DASHBOARD.md](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/DASHBOARD.md#what-does-the-cli-send-to-genclassdev)). +- **Privacy policy** for the diagnostics and for genclass.dev accounts (what is collected, how long it is kept, your + rights): [PRIVACY.md](https://github.com/genclass-dev/GenClass-lib/blob/main/PRIVACY.md). +- Apps that ship GenClass may need to disclose this to their users (GDPR/CCPA). `telemetry: { endpoint, ... }` sends + to your own collector instead. Full schema, storage, every opt-out and guidance: + [TELEMETRY.md](https://github.com/genclass-dev/GenClass-lib/blob/main/packages/runtime/TELEMETRY.md). +- **Inputs and redaction.** Typed values of password, `cc-*` and `one-time-code` fields, and of fields whose name or + label names a secret, are never recorded. The default redactor works by the leaf field's meaning (`auth.token`, + `payment.card.number`; `auth.loading` stays visible); booleans and null are never redacted. Under a container named + for a secret, strings, numbers and arrays are all redacted, so a word that is a secret in another sense hides numbers + too (`map.pin.lat`). The "would replace text the user typed" fact shows a character diff only when neither value is + redacted, else `[redacted] → [redacted]`. Pass your own `redact(path, value)` for app-specific secrets; the default + has gaps (below). + +## Known limitations + +- **The model is new and imperfect.** It misses most problems, it was not evaluated on your app, and in observe mode + it makes some false flags ([Model quality](#model-quality)). It is not a substitute for tests. Watch observe mode + before turning on `guard` or `heal`. +- **Browsers.** Every browser measurement here ran in Chromium (headless or Playwright); model latency, the perf + tests and the simulated apps ran in Node. Firefox and Safari are not tested. Layout shifts and failed CSS + background images are checked in Chromium only. +- **Redaction gaps.** A container named by a two-word secret (`cardNumber`, `apiKey`, `creditCard`) counts as broad, + like `auth`, so `payment.cardNumber.value = "4111 1111 1111 1111"` is shown in the situation text, `explain()`, the + console evidence and devtools, and the telemetry situation text if you opted in (a leaf with such a name, + `settings.apiKey`, is redacted). "X changed since #N started and is back to V" compares rendered text, so two + different redacted values read as "back to [redacted]". Pass a custom `redact`, and keep secrets out of observed + stores. +- **The discard mark lasts 10 s.** After a `discard`, writes by operations chained from the discarded one (a + `setTimeout`-driven poll, a saga) to the protected fields are also dropped for 10 s, even when they carry fresh + data. The marks also outlive `rt.pause()` and `setMode("observe")`. +- **`defer` can wait long.** A delivery may be deferred twice, each time until the related operations finish or 10 s + pass. A deferred WebSocket/EventSource message holds back the messages queued behind it, and is still dispatched if + the app closed the socket meanwhile. EventSource `open` events are not kept in order behind held messages. XHR + listeners of a held response run after the original dispatch, so `e.currentTarget` is `null`: read the `xhr` + object itself. +- **`retry` (heal) trusts HTTP semantics.** It is offered for idempotent methods, and for POST / PATCH only with an + idempotency key (`policy.idempotencyHeaders`). An endpoint that is not idempotent despite its method (a `PUT` that + appends) can still be retried. A 502 is described to the model as a status usually returned without processing the + request, which is not always true behind proxies. `coalesce`, `retry`, `hedge` and failure/stall `serve_cached` are + fetch-only. +- **`policy.holdWrites: true` (opt-in).** A held write that a later write flushed early can be recorded as dropped + while it stays applied. +- **Synthetic events** (`isTrusted === false`: `el.click()`, `dispatchEvent`, in-page test drivers) are not user + actions unless you pass `observe: { untrustedEvents: true }`. +- **Causality** across `await` is tracked by instrumenting fetch, XHR, timers, message events and Response bodies. + This is best effort; wrap important work in `rt.op(name, fn)` for exact attribution. +- **Store state.** Discovered React and Zustand state is observed only + ([Automatic state discovery](#automatic-state-discovery)). The first response of each kind after a page load is + never a delivery decision: GenClass has not yet seen what that request writes. +- **Install and CDN.** Read the diff `init` and `remove` show before you confirm. SRI covers only the script-tag file; + what it loads is not integrity-checked by the browser (the runtime checks the model files' sha256), so pin a + version, or self-host. A page that lets untrusted HTML into `` should not use ``. + +## Install + +Pick one; all start the same runtime, in observe mode unless you choose otherwise. Every `init` and `remove` flag, the +token and dashboard `init` creates, page configuration and the tested setups: +[INSTALL.md](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/INSTALL.md). + +**1. One command.** `init` finds your framework and package manager, installs the package, adds one import as the +first line of your entry file (plus a dev-only devtools line), and shows you the diff before writing anything. ```bash -npx @genclass/runtime@beta init --token gc_... # use a token you already have (no network) -npx @genclass/runtime@beta init --no-token # no token, no network +npx @genclass/runtime@beta init # observe; shows the diff, asks, then writes +npx @genclass/runtime@beta init --mode guard # let GenClass act (also: --mode heal); run again later to switch +npx @genclass/runtime@beta init --dry-run # show the diff, write nothing, send nothing +npx @genclass/runtime@beta remove # undo exactly what init added ``` -`--no-telemetry` implies `--no-token` (a dashboard only gets data while telemetry is on). If genclass.dev cannot be -reached, `init` says so, sets GenClass up without a token, and points you to -[genclass.dev/start](https://genclass.dev/start); run `init --token gc_...` later. `--dry-run` never calls the -network. +Other flags: `--yes`, `--no-telemetry`, `--no-token`, `--token gc_...`, `--sink`, `--model-url`, `--no-install`, +`--no-devtools`, and from `0.3.0-beta.2` `--no-network` and `--sync` (plain HTML). -Options without editing code (also on a project `init` already set up; `remove` takes them out too): +**Which version** (from `0.3.0-beta.2`). `init` (and `login`, `connect`) first prints its own version and the npm +dist-tag it came from, asks npm's dist-tags once (`npm view @genclass/runtime dist-tags`, at most 3 s; never with +`--dry-run`, `--no-network`, `--no-token` or `--no-telemetry`, nothing when offline) and says when a newer release is +on npm, with the command that runs it: -```bash -npx @genclass/runtime@beta init --no-telemetry # anonymous diagnostics off (--telemetry: explicitly on) -npx @genclass/runtime@beta init --model-url /genclass-model/ # a self-hosted model (see Content-Security-Policy below) -npx @genclass/runtime@beta init --sink webhook:/genclass # findings to your own endpoint too; also console-json - # (0.3.0; "See it where you already look") ``` + ✔ Version @genclass/runtime 0.3.0-beta.3 (npm dist-tag beta) + ▲ Newer @genclass/runtime 0.3.0-beta.4 is on npm (dist-tag beta): run npx @genclass/runtime@beta init instead +``` + +It installs its own version (`@genclass/runtime@^0.3.0-beta.4` today), and a project that already has an older one +(a `0.2.x` from `npx @genclass/runtime init` without `@beta`) is upgraded, after the same diff and question. Every +command it prints names `@beta` while it is a prerelease. + +**Plain HTML** (from `0.3.0-beta.2`): `init` writes the script tag with `defer`, so a slow or blocked CDN never holds +up the page. GenClass then starts once the page is parsed: before your `defer` and `type="module"` scripts and before +`DOMContentLoaded`, but after classic scripts that have neither `defer` nor `async`, so requests and errors those +scripts cause while the page is still loading are not seen, and a library loaded that way before GenClass (a UMD +React from a CDN) is not discovered; a click handler's `fetch` is seen either way, and the model still loads at idle +after `load`. `--sync` writes the blocking tag of earlier versions, which runs before every script. `remove` takes +the tag out whatever attributes you add to it (`defer`, `async`, `nonce`). When a project looks mostly static (no +`fetch`, XHR, WebSocket or EventSource call in its sources, or a plain HTML page with one script), `init` says so in +one sentence and points to the visual checks and the page scan (`npx @genclass/runtime@beta scan `, +[Audit a page](#audit-a-page)). -They go into a marked `genclass.config.ts` (or `.js`) next to your entry, which sets `window.GENCLASS_CONFIG` and is -imported right before the `/auto` import (an ES module's imports run before its own code, so options cannot be set -above the import in the same file). Any `GenClass.init()` option works in that file. Plain HTML gets -`data-telemetry="off"` (and `data-model`, `data-ort`, `data-sink`) on the script tag, Astro a `` line. - -`init` also says that telemetry is on by default and how to turn it off, and when your app sets a -Content-Security-Policy (`next.config` headers, a proxy or middleware, a ``, helmet, `vercel.json`, -`_headers`, ...) it prints the steps to self-host the model. - -- **What `init` edits.** It only edits browser apps. It stops and says why for a Node server, a library (a UI - framework as a peer dependency, or an `exports`, `module`, `types` or `bin` field), or an entry file that imports - Node-only modules. -- **What `remove` deletes.** Only what `init` wrote, including lines a formatter rewrapped. If you edited a marked - line or block, it changes nothing and lists where. - -Tested end to end on fresh projects from each framework's own generator: Vite 8 (React with npm and pnpm, Vue, -Svelte), Next.js 16.4 (App and Pages Router, plus the paths for Next < 15.3), Create React App 5, SvelteKit, Astro, -Nuxt 4.6, React Router 8 (framework mode), Angular 20 and plain HTML. In all 15 projects: - -- the app built and ran in Chromium after `init` (production build where the project has one); -- the model loaded in a worker, the overlay appeared only in development, and a warm load had no console errors; -- `remove` left every file byte-identical to the scaffold (node_modules, lockfiles and build output excluded). - -Remix, Solid, Preact and Next.js before 15.3 are detected but were not scaffolded with their own generators. -The [compatibility matrix](https://github.com/genclass-dev/GenClass-lib/blob/main/compat/RESULTS.md) runs the -one line in React + Vite, Next.js 16, Vue + Pinia, SvelteKit 3, Angular 22, Solid and plain HTML, with 15 data layers -(TanStack Query, SWR, Redux Toolkit, Zustand, Apollo, WebSocket and EventSource among them), in every mode, and checks -that GenClass never makes those apps worse; starter templates are in -[templates/](https://github.com/genclass-dev/GenClass-lib/tree/main/templates). -Details: [test/install/RESULTS.md](https://github.com/genclass-dev/GenClass-lib/blob/main/packages/runtime/test/install/RESULTS.md). -Those scaffold runs predate the `init` / `remove` fixes in this version (`--mode`, formatter handling, server and -library detection), which are covered by unit tests; the current end-to-end evidence is the compatibility matrix above. - -**2. One import** (any bundler; the package is ESM only, there is no `require` build), first in your entry file, so it is installed before your framework and stores -created at import time see the runtime (automatic state discovery is on): +**2. One import** (any bundler), first in your entry file, so it is installed before your framework: ```ts -import "@genclass/runtime/auto"; // observe (the default) -import "@genclass/runtime/auto/guard"; // or guard -import "@genclass/runtime/auto/heal"; // or heal (experimental) -import "@genclass/runtime/auto/observe"; // observe, explicitly - -import rt from "@genclass/runtime/auto"; // the same, and the runtime it started +import "@genclass/runtime/auto"; // observe (the default); also /auto/guard, /auto/heal ``` -Optional page configuration, read once: `` inside -`` or `window.GENCLASS_CONFIG = { mode: "guard", devtools: true }` (any `InitOptions`; since `0.3.0` also -`token=gc_...` / `token: "gc_..."`), set before the import runs. During SSR or in Node, `/auto` installs nothing and -returns an inert runtime. +Options go in `window.GENCLASS_CONFIG = { ... }` (any `InitOptions`, set before the import runs) or +`` in ``. **3. One script tag** (no build step), first in ``: ```html - + ``` -It exposes `window.GenClass`, discovers the app's state like the import does, and loads the model worker, ONNX -Runtime Web and the overlay on demand from the same version on the CDN. `data-mode` takes `observe` (the default), `guard` or `heal`; `data-devtools="local"` shows the -overlay only on local hosts (loopback, `*.localhost`, `*.local`, `*.test` and `file:`, so also in a Capacitor, Tauri or -Electron build that ships it); `data-manual` skips the automatic `GenClass.init()`. Since `0.3.0`: `data-token="gc_..."` -(your app's token), `data-scope="functions"` and `window.GenClass.protect`. Keep the version pinned, as above: the -unversioned URL serves npm's `latest` (the older `0.2.x` until `0.3.0` is stable); the plain-HTML path of `init` -writes a pinned jsDelivr URL with SRI. +It exposes `window.GenClass` and loads the model worker and the overlay on demand from the same version; `data-mode`, +`data-devtools`, `data-telemetry`, `data-token`, `data-scope`, `data-sink` and more configure it +([every attribute](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/INSTALL.md#which-attributes-does-the-script-tag-take)). +Keep the version pinned: the unversioned URL serves npm's `latest` (the older `0.2.x` until `0.3.0` is stable); on a +plain HTML page `init` writes a pinned jsDelivr URL with SRI and, from `0.3.0-beta.2`, `defer`. +`data-devtools="local"` shows the overlay on local hosts (loopback with or without a port, `*.localhost`, `*.local`, +`*.test` and `file:`), so also in a Capacitor, Tauri or Electron build that ships it: set it per environment. -**4. By hand:** - -```bash -npm install @genclass/runtime@beta -``` +**4. By hand:** `npm install @genclass/runtime@beta`, then first in your entry file: ```ts -// first thing in your entry file import { GenClass } from "@genclass/runtime"; const rt = GenClass.init(); // observe; GenClass.init({ mode: "guard" }) to let it act @@ -293,16 +480,31 @@ const rt = GenClass.init(); // observe; GenClass.init({ mode: "guard" }) to let `GenClass.init()` does not discover state on its own: register stores ([State it can protect](#state-it-can-protect)), or import `@genclass/runtime/discover` first and pass `autoState: true`. -The unscoped `genclass-runtime` alias is deprecated; use `@genclass/runtime`. +**ESM only.** No `require` build. Jest needs a `transformIgnorePatterns` entry such as +`["/node_modules/(?!@genclass/runtime)"]`. The unscoped `genclass-runtime` alias is deprecated. -**ESM only.** The package ships ES modules (no `require`). Jest users need a `transformIgnorePatterns` entry that lets -`@genclass/runtime` be transformed, for example `["/node_modules/(?!@genclass/runtime)"]`. +## For AI coding agents installing GenClass + +1. Run `npx @genclass/runtime@beta init --dry-run` (it shows the diff and sends nothing). Tell the user that + anonymous diagnostics are on by default on public hosts, and that applying `init` creates the app's dashboard at + genclass.dev from `package.json`'s `name` unless they pass `--no-token`. +2. Apply it with `--yes` only once they agree. Telemetry is the user's decision: add `--no-telemetry` or + `telemetry: false` only if they ask. +3. Keep the default observe mode. Use `--mode guard` only on request, and then keep payment and sign-in endpoints + observe-only: `requests: { protect: ["preset:payments", "preset:auth"] }` in `window.GENCLASS_CONFIG` (the + `genclass.config.ts` that `init` writes), or `protect: protectPreset("payments", "auth")` with `GenClass.init()`. +4. `.genclass.local` holds the private dashboard link and is gitignored: never paste it into code, docs, commits or + pull requests. The token (`gc_...`) is public and belongs in the code. +5. Check: the console shows `[GenClass] Model ready (...). Mode: observe.`; undo with + `npx @genclass/runtime@beta remove`. + +To change GenClass itself, read [AGENTS.md](https://github.com/genclass-dev/GenClass-lib/blob/main/AGENTS.md). ## Audit a page -One command checks any page, yours or not, with GenClass's 23 measured visual checks (below) and prints a score and -each problem with the element, its selector and a suggested CSS fix, worst first. Nothing needs to be installed in -the page: +One command checks any page, yours or not, with GenClass's 26 measured visual checks +([below](#visual-and-css-checks-new-in-030)) and prints a score and each problem with the element, its selector and a +suggested CSS fix, worst first. Nothing needs to be installed in the page: ```bash npx @genclass/runtime@beta scan https://example.com # also: audit; a dev server works too (localhost:5173) @@ -372,487 +574,104 @@ wait for the bridge's hello, at most 0.6 s). Design and measurements: [AGENT.md] ## Function-specific: protect() -**New in `0.3.0`** (`0.2.x` has no `protect` export: importing it fails). Use this when you want GenClass -on a few functions that matter (a checkout submit, a search, a save) rather than on the whole app. +**New in `0.3.0`** (`0.2.x` has no `protect` export). Use it when you want GenClass on a few functions that matter (a +checkout submit, a search, a save) rather than on the whole app. ```ts import { GenClass, protect } from "@genclass/runtime"; GenClass.init({ token: "gc_...", scope: "functions" }); // early in your entry file (with /auto or the script tag: see below) -// Wrap at module level; calls made before GenClass.init() simply run fn. +// Wrap at module level; calls made before GenClass.init() run fn directly. export const submitOrder = protect("checkout submit", async (cart: Cart) => { const res = await fetch("/api/orders", { method: "POST", body: JSON.stringify(cart) }); if (!res.ok) throw new Error(`order failed: ${res.status}`); return res.json(); }); - -export const search = protect("search", async (q: string) => { - const res = await fetch(`/api/search?q=${encodeURIComponent(q)}`); - return (await res.json()) as Result[]; -}); - -class CartStore { - save = protect("cart save", function (this: CartStore, items: Item[]) { - return api.put("/api/cart", items); // `this` is the store, as without protect - }); -} -``` - -What `protect(name, fn)` does: - -- **Same function.** It returns a function with the same signature: `this`, arguments, return value and thrown - errors pass through, and a sync function stays sync. For an `async` function you get a promise that settles - exactly like `fn`'s (it is a new promise object, as with `runtime.op`). Other thenables are returned untouched. - `protect` itself never throws. -- **Each call is tracked.** The call runs as an operation named `name`: for an `async` function the same operation, - and the same situation, as `runtime.op(name, fn)` (a sync function's operation ends as soon as it returns). The - requests, state writes, timers and errors it starts are linked to it, directly or through the causal chain GenClass - already follows (a request started after an `await` on one of the function's own requests counts). Work started - from somewhere else, such as a WebSocket message or another click handler, does not. -- **Decisions name the function.** A decision about something a protected call caused carries `fn: name` - (`rt.on("decide", (d) => d.fn)`, and, when your app sets a token, the telemetry `decision` event and your - dashboard; without a token the name is never sent). With nested protected functions, the outermost one is named. - Use a fixed name: in telemetry, a label that looks like data (an e-mail address, a URL, 4 or more digits in a row, - 12 or more hex digits) is sent as `[redacted]`; `d.fn` in your app keeps it as written. -- **No runtime, no cost.** A protected function looks up `GenClass.runtime` on every call. Before `GenClass.init()` - (or after `GenClass.destroy()`) it calls `fn` directly. -- **Also** as `GenClass.protect`, and on the script tag's `window.GenClass.protect`. With the script tag, use - `window.GenClass.protect`: a copy imported from npm would not see the script tag's runtime. - -**`scope: "functions"`** (`InitOptions.scope`; also `data-scope="functions"` / `scope=functions` in the meta tag): -GenClass still observes the whole app and keeps it as context in the situations it builds, but it raises decisions -only for activity inside protected functions; it never holds or acts on anything else. Standing questions from -plugins (`rt.question()`, a plugin's `questions`) are only asked about protected activity too. Explicit questions -(`rt.ask()`, `rt.decide()`) are still answered everywhere. The default, -`scope: "app"`, decides about the whole app as before; `protect()` then only adds the function names. The mode still -applies: in `observe` (the default) GenClass reports, in `guard` it may act on what protected functions cause. - -**With `/auto` or the script tag** (what `npx @genclass/runtime@beta init` sets up), GenClass is already running before -your code, so a later `GenClass.init({ token, scope })` is ignored (the console says so once). Set them where GenClass -starts: `window.GENCLASS_CONFIG = { token: "gc_...", scope: "functions" }` in the `genclass.config.ts`/`.js` that -`init` imports right before `/auto`, `` in ``, or -`data-scope="functions"` on the script tag. Then wrap your functions with `protect()` as above. - -A per-function mode (`protect(name, fn, { mode })`) does not exist yet: use `routes` or `requests.protect` to narrow -where GenClass acts. - -## Your dashboard - -**New in `0.3.0`** (`0.2.x` ignores a token and has no `login`, `connect`, `whoami`, `logout` or `dashboard`). Each -web app gets a **project key** (token: `gc_` + 22 letters and digits) that groups its diagnostics on a dashboard at -genclass.dev. The key is public: it ships in your page and can only add telemetry to its app. Keep the dashboards in -a GenClass account, or use one private link per app without an account. The dashboard is free for every app, with -90 days of history, and so are your own classifier, which GenClass trains on your app, and the `@genclass/mcp` server -for coding agents ([pricing](https://genclass.dev/pricing)). - -```bash -npx @genclass/runtime@beta dashboard # print the link to this app's dashboard (--open: open it in your browser) -``` - -`dashboard` reads the private link in `.genclass.local` without contacting anything; signed in, it finds the app in -your account (one `GET /api/me`), or lists your apps' links. - -**With an account:** - -```bash -npx @genclass/runtime@beta login # sign in in the browser, then connect the app in this folder -npx @genclass/runtime@beta connect # connect an app folder later (or another one) -npx @genclass/runtime@beta whoami # your account and its apps -npx @genclass/runtime@beta logout # end the CLI session ``` -`login` prints a code and opens `https://genclass.dev/login/device`, where you sign in (Google, your email and password, or GitHub, -plus your passkey if your account has one) and approve that code. It waits until you do (Ctrl-C stops it and saves nothing), then saves the CLI -session in `~/.config/genclass/credentials.json` (`$XDG_CONFIG_HOME/genclass/`; Windows `%APPDATA%\genclass\`), -readable only by you. `--no-browser` prints the link without opening a browser (it works from any device). -Inside an app's folder, `login` then asks whether to connect that app; `connect` does that step alone: +- **Same function.** Same signature: `this`, arguments, return value and thrown errors pass through, and a sync + function stays sync. `protect` itself never throws. +- **Each call is tracked** as an operation named `name`, the same as `runtime.op(name, fn)`: the requests, state + writes, timers and errors it starts are linked to it; work started elsewhere (a WebSocket message) is not. +- **Decisions name the function**: `fn: name` on `rt.on("decide")`, and, with a token, in telemetry and on your + dashboard (without a token the name is never sent). Use a fixed name: a label that looks like data (such as an + e-mail address, a URL or 4 or more digits in a row) is sent as `[redacted]`. With the script tag, use + `window.GenClass.protect`. -- pick one of your account's apps, or a new one named after `package.json` (`--app `, `--name `); - a new app is created only once you approve the changes; -- it writes the key the way `init` does (`genclass.config.*`, `data-token` on the script tag, or the Astro meta - line), shows the diff and asks first (`--yes`: no questions; `--dry-run`: it reads your app list, and creates and - writes nothing), and sets GenClass up first if `init` has not; -- an app `init` already created without an account (its private link in `.genclass.local`) can be added to your - account instead, keeping its key and its data. The link is only sent to the host it belongs to. +**`scope: "functions"`** (also `data-scope="functions"` or `scope=functions` in the meta tag): GenClass still observes +the whole app as context, but raises decisions only for activity inside protected functions and never holds or acts +on anything else; `rt.ask()` and `rt.decide()` are still answered everywhere. The default, `scope: "app"`, decides +about the whole app. There is no per-function mode: narrow where GenClass acts with `routes` or `requests.protect`. +With `/auto` or the script tag, GenClass starts before your code, so a later `GenClass.init({ token, scope })` is +ignored (the console says so once): set them in `window.GENCLASS_CONFIG` (the `genclass.config.ts` that `init` +writes), the meta tag or the script tag. -``` - ✔ Account you@example.com - ✔ App shop-web (a new app in your account, created once you apply the changes) - ... - Your dashboard https://genclass.dev/account/apps/... -``` +## Your dashboard -An app's dashboard in your account opens for you when you are signed in at genclass.dev. Until the app's first event -arrives it shows the key and these same two ways to connect it; you can also sign up at genclass.dev, create an app -there, and pass its key yourself (below). What the CLI sends: `login` asks genclass.dev for a sign-in code, naming -itself `genclass-runtime ` (the approval page shows that name and your account page lists the session -under it), and polls until you approve it, then reads your account's apps; creating an app sends its name, and -adding an existing dashboard sends its link from `.genclass.local`. Nothing is sent until you run -one of these commands (or approve `init`'s changes). The CLI session (`gcs_...`) is a secret: `logout` ends it at -genclass.dev and deletes the file, and you can revoke it from your account page. A CLI session can read your -account's apps, create apps, add an app from its private link and end itself; it cannot read stats, end other -sessions or change how you sign in. What an account stores: -[PRIVACY.md](https://github.com/genclass-dev/GenClass-lib/blob/main/PRIVACY.md). - -**Without an account:** - -| | what | who may see it | -|---|---|---| -| token | `gc_` + 22 letters and digits, e.g. `gc_4fQ9...` | public: it ships in your page. It only lets your app's diagnostics be grouped together. | -| dashboard link | `https://genclass.dev/dashboard/` | **private**: anyone with the link sees your app's stats, and it is the only way to open them | - -**Get one:** `npx @genclass/runtime@beta init` (not signed in, after you approve its diff: creates both, writes the -token, prints the link and saves it in `.genclass.local`), or click "Get a token" at -[genclass.dev/start](https://genclass.dev/start). A link can be added to an account later (`connect`, or "Add an -existing dashboard" on your account page). - -**Pass a key yourself:** `GenClass.init({ token })`, `data-token="..."` on the script tag, -`` or `window.GENCLASS_CONFIG = { token }`. A malformed token logs one -console warning and is ignored. - -**What it shows:** decisions, detections by diagnosis, actions taken, top routes and protected functions, -model load, backend (WebGPU or WASM) and latency, runtime and model versions, and recent detections. With -`analytics: true` (new in `0.3.0-beta.1`, off by default), also how visitors use the app: page views, time on page and -scroll depth per page, the most clicked elements, rage and dead clicks, and how visitors move: the most taken paths, -page-to-page steps with the time before each, page load starts and ends, back and forward use and the first element -used per page ([Privacy and telemetry](#privacy-and-telemetry)). -Put `data-genclass-name="pricing-cta"` on an element to give it a readable name there. Analytics need -`0.3.0-beta.1` or later (`npm install @genclass/runtime@beta`): `0.2.x`, npm's `latest` until 0.3.0, ignores -`analytics: true` without a word, so the dashboard's Visitors view says which runtime versions your app sends from. -With the script tag, when the analytics code itself does not load (a self-hosted `dist/` missing `analytics/`), the -console says so and `runtime.analytics.reason` is `"error"`; files of two releases in one `dist/` get one warning -naming both -([ANALYTICS.md](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/ANALYTICS.md) section 4.1). - -**What it needs:** telemetry on (the default with `GenClass.init()` in a browser) on a deployed host. Local hosts -(`localhost`, `*.local`, `*.test`, private networks) do not send by default, so a page there does not feed the -dashboard (the console says so once when a token is set). `telemetry: true` or any telemetry options object (for -example `telemetry: { decisionSample: 1 }`) is an explicit opt-in that sends from local hosts too, token included. With telemetry off for any other reason, a token does nothing (with -`debug: true` the console says so once). Visitors who opt out (`?genclass=no-telemetry`, -`localStorage["genclass.telemetry"] = "off"`) or whose browser sends Global Privacy Control are not counted. With a -token, each telemetry batch carries it and decisions about protected functions carry their `fn` name; nothing else -changes about what is sent ([Privacy and telemetry](#privacy-and-telemetry)). Dashboard data is kept for 90 days. - -**Which version:** the token option, `scope`, `protect()` and the `login`, `connect`, `whoami` and `logout` commands -are in `0.3.0` (first as `0.3.0-beta.0`, see the [CHANGELOG](CHANGELOG.md)); `dashboard` from `0.3.0-beta.2`. A key -needs `@genclass/runtime` `0.3.0-beta.0` or later in your app. A dashboard opens on every plan, whichever runtime -version sends to it. +**New in `0.3.0`** (`0.2.x` ignores a token). Each web app gets a public **project key** (token: `gc_` + 22 letters and +digits) that groups its diagnostics on a dashboard at genclass.dev, kept in a GenClass account (the CLI's `login`, +then `connect`, `whoami`, `logout`) or behind one private link per app (anyone with the link can open it), which +`init` saves in `.genclass.local` and `npx @genclass/runtime@beta dashboard` prints again (from `0.3.0-beta.2`; +`--open` opens it). The dashboard is free for every app, with 90 days of history (1 year on Enterprise), and so are +your own classifier, which a person from GenClass trains on your app's traffic, bug reports or labelled examples once +you ask, and the `@genclass/mcp` server for coding agents ([pricing](https://genclass.dev/pricing)). It needs +telemetry on and a deployed host: local hosts do not send by default. Pass a key with `GenClass.init({ token })`, +`data-token`, `token=` in the meta tag or `window.GENCLASS_CONFIG = { token }`. Accounts, the private link and what +the CLI sends: [DASHBOARD.md](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/DASHBOARD.md). + +**What it shows:** what GenClass found, on which pages, and what it did (decisions, detections by diagnosis, actions, +top routes and protected functions, model load and latency, runtime and model versions). With `analytics: true` +(`0.3.0-beta.1` or later, off by default) also how visitors use each page (page views, time on page, scroll depth, the +most clicked elements, rage and dead clicks; `data-genclass-name="pricing-cta"` gives an element a readable name) and, +from `0.3.0-beta.2`, how they move: the most taken paths, page-to-page steps with the time before each, where page +loads start and end, back and forward use, and the first element used per page. `0.2.x` ignores `analytics: true` +without a word, so the Visitors view says which runtime versions your app sends from. With the script tag, when the +analytics code itself does not load (a self-hosted `dist/` missing `analytics/`), the console says so (from +`0.3.0-beta.2`) and `runtime.analytics.reason` is `"error"`; files of two releases in one `dist/` get one warning +naming both ([ANALYTICS.md](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/ANALYTICS.md) +section 4.1). ## Automatic state discovery With the one line (`@genclass/runtime/auto*` or the script tag) GenClass finds the app's state on its own. It -installs itself before your framework loads (that is why the line goes first), the way React DevTools and Redux -DevTools do, and records every state change with the operation that caused it (the response, timer or user action -whose callback called `setState` or `dispatch`), exactly as for a registered store. Delivery decisions ("this -response would overwrite newer data") need those writes: without them, GenClass could only see the network. +installs itself before your framework loads, the way React DevTools and Redux DevTools do, and records every state +change with the operation that caused it (the response, timer or user action whose callback called `setState` or +`dispatch`). Delivery decisions need those writes: without them, GenClass could only see the network. | state | how it is found | what GenClass can do with it | |---|---|---| -| React ≥ 16.8 (development and production builds): `useState`, `useReducer`, `useSyncExternalStore`, class component state | the React DevTools global hook (installed, or chained onto the real extension's, which keeps working); each commit, only the components that re-rendered are compared, within 1 ms | **observed only**: detections, facts, triage and delivery decisions (`deliver` / `defer`); never held, dropped, reverted or rolled back | -| Redux / Redux Toolkit (`configureStore` with `devTools` on, its default; `createStore` with the Redux DevTools compose or enhancer) | the Redux DevTools globals (forwarded to the real extension when installed) | **controllable**: GenClass's Redux enhancer, as if you had added it: delivery `discard`, late revert, `rollback`, `resync` | -| Zustand with the `devtools` middleware (and other libraries reporting to Redux DevTools' `connect`) | the Redux DevTools `connect` API. **Production builds:** Zustand turns `devtools` off when `import.meta.env.MODE` is `production` unless you pass `enabled: true`, so a production build is not discovered without it (or wrap the store with `genclass` from `@genclass/runtime/zustand`, which also makes it controllable) | **observed only** | - -- **Names.** A component's store is named after the component: `SearchPage.state0`, `SearchPage.state1` (hooks in - order), `SearchPage.external0` (`useSyncExternalStore`), or the class state's keys (`Wizard.step`). Production - builds minify component names, so a store is then named after the element the component renders (`id`, - `data-testid`, its first `data-*` attribute or `aria-label`: `orderChip`, `addToOrder`), else the minified name. - At most 3 instances per component (`Item`, `Item_2`, `Item_3`) and 48 component stores. A store appears on its - first state change; mounting alone records nothing. Redux stores are named after the devtools `name` option, else - `redux`; Zustand stores after their `devtools` `name`, else `store`. -- **Skipped.** Anonymous components, framework internals (Next.js and React Router components, error boundaries), - and values that are not app data: functions, promises, React elements, DOM nodes, class instances and objects - holding them (router caches, query clients). -- **Secrets.** Field names are redacted as usual (`form.password`). A hook has no name, so a `useState` whose - string equals what a password, card-number or one-time-code input holds is redacted from then on. Discovered - state reaches telemetry only if you opt into sending situation text (`telemetry: { include: { situation: true } }`). -- **Turn it off:** ``, `data-autostate="off"` on the script tag, - `window.GENCLASS_CONFIG = { autoState: false }`, or one source only: `autoState: { react: false }` (also `redux`, - `zustand`). `?genclass=off` installs nothing at all. -- **With `GenClass.init()`** discovery is off by default, so explicit setups do not change. To turn it on, import - `@genclass/runtime/discover` first (before React, Redux or Zustand) and pass `autoState: true`; the runtime may start - later, before the first render. Redux stores created before it starts are observed only. -- **Not covered yet:** Vue / Pinia, MobX, Jotai, Recoil, signals, Svelte stores, and state kept outside these - libraries (module variables, refs). A React renderer that loaded before GenClass is observed without exact causes; - one that rendered before GenClass is not observed at all. Register such state with `rt.atom` / `rt.guard` - ([State it can protect](#state-it-can-protect)), which also makes it controllable. -- **Cost:** about 7 KB gzip more JavaScript in `/auto`; a commit walk of p50 below 0.2 ms and p95 under 2 ms on a - 4× throttled phone profile, about +0.5 MB heap ([Costs](#costs)). - -## Why it's an easy yes (measured) - -These hold for the runtime; whether the model's decisions are good is a separate question (see -[Model quality](#model-quality)). - -- **It is not expected to make a working app worse.** Decisions about responses happen at the network boundary, - which looks like ordinary latency to your app, and the default never holds or reorders your app's own store - writes. With a model that never intervenes (heal mode, compared against observe mode), GenClass changed the - outcome in **0 of 396** clean runs across **66 real apps in 23 frameworks** (React, Vue, Svelte, Solid, Angular, - Ember, Elm, Lit, Redux, Zustand, MobX, TanStack Query and more, including 14 unmodified open-source RealWorld - front-ends), 6 seeds each. What that check does and does not compare is under [Model quality](#model-quality). -- **Every mode, on idiomatic apps in seven frameworks.** The - [compatibility matrix](https://github.com/genclass-dev/GenClass-lib/blob/main/compat/RESULTS.md) (React, - Next.js, Vue, SvelteKit, Angular, Solid, plain HTML; 15 data layers; 10 seeds per cell) found 0 bugs introduced in - 3,510 runs with GenClass and 0 actions on correct apps. Observe left page and server state identical to the run - without GenClass in every cell but one: an app's own timing race (SWR's optimistic rollback) that came out right - with GenClass and wrong without it, with no action taken. Observe holds nothing, but its main-thread work is not - zero. -- **Normal traffic costs little.** Facts are computed for every write and request; the model is consulted only for - salient situations. Clean in-order typeahead makes no model calls and holds nothing. A keystroke write to a store - holding a 5,000-item array takes about 0.22 ms. -- **The model runs in the browser.** It runs locally in a Web Worker on WebGPU or WASM and is cached after the - first load: no inference server, no API key, and decisions never wait on the network. Anonymous diagnostics about - its decisions (without page text unless you opt in) are sent to the GenClass maintainers by default, except on - local hosts; one option turns that off (see [Privacy and telemetry](#privacy-and-telemetry)). Typed values of - password and payment fields are never recorded (other redaction has gaps). -- **You can see what it did.** Every detection and action gets one plain-English console line with the evidence - behind it, and `rt.explain(id)` shows exactly what the model read. Discards and rollbacks can be undone; responses - it changed carry an `x-genclass` header. -- **Off in one step.** `?genclass=off` in the URL installs nothing, and observe (the default) never changes - execution. -- **Small on the main thread.** The main entry is about 100 KB gzip in `0.3.0` (minified, without the optional - devtools; 92 KB in `0.2.2`; see Performance). The - default model is 10.2 MB on WASM (13.6 MB fp16 on WebGPU with `shader-f16`), downloaded once at idle and cached. - It runs in a worker that holds 190 to 280 MB while loaded: see [Costs](#costs). - -## Modes - -| mode | what it does | non-passive actions | -|---|---|---| -| `observe` (**default**) | Finds and reports what it sees and what it would have done. Never holds a request or a response, never runs an action. | none | -| `guard` (opt-in) | Also prevents failures with minimal, reversible guard-tier actions: `discard`, `defer`, `coalesce`, `delay` (drop a stale response, reuse a duplicate's result, back off). Only when the model's gate says acting beats doing nothing (with model 0.2.0 at `balanced`: about 90% sure). | guard tier | -| `heal` (**experimental**) | Also recovers: `retry`, `serve_cached`, `block`, `hedge`, `rollback`, `resync`, plus your own actions. | guard + heal tier | - -**Detections in observe (`0.3.0`).** A detection is *likely* when the model's top diagnosis is not `expected` and -its probability reaches the report gate (0.90 at `balanced` with model 0.2.0), and is printed as one console line -with its evidence, as before. In observe GenClass also reports *possible* detections: at least the `eager` -profile's report value (0.70) but below the report gate. They print with `console.debug` only, ending in -`(possible)` (Chrome hides them unless the console level includes "Verbose"), show as dimmed cards in the devtools, -and are counted on their own (`summary().possibleDetections`, telemetry). Your own wiring gets them only if you ask: -`rt.on("detect")`, `rt.on("report")` and `report: (r) => …` receive the likely ones unless you set -`detect: "possible"` (every decision, with `detection` set, is on `rt.on("decide")`), and sinks need -`possible: true`. `detect: "likely"` turns them off; `detect: "possible"` turns them on in guard and heal too (reporting -only: what guard and heal do is unchanged). Setting `policy.thresholds.report` turns them off unless `detect` asks. -Measured effect and false-detection bounds: -[RESULTS.md §11](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/RESULTS.md). - -Observe no longer goes blind while the model loads or the tab is hidden (`0.3.0`): situations raised then are built -at once and decided when the model is ready and the tab visible (at most 32 wait; a hidden tab never loads the -model, the tab becoming visible does). Until their false-report rate is measured, what they find is reported at the -possible tier at most. `detect: "likely"` turns this off too. In guard and heal, nothing waits: they skip, as before. -Three `0.3.0` fixes can add likely reports in observe that `detect: "likely"` does not turn off, at a rate not -measured yet (overwritten writes decided on slow devices, the body-wait and idle-unload fixes; CHANGELOG). - -## How eager should it be? - -Once you opt into `guard` or `heal`, GenClass only acts when the model is confident enough. You choose how confident: - -```ts -GenClass.init({ aggressiveness: "cautious" }); // "cautious" | "balanced" (default) | "eager", or a number 0–1 -GenClass.runtime.setAggressiveness("eager"); // change it later (the devtools have a selector too) -``` - -`cautious` acts less often (fewer interventions, almost never a wrong one); `eager` fixes more problems at a -somewhat higher risk of acting when it did not need to. A number in between interpolates (0 = cautious, -0.5 = balanced, 1 = eager). Try a level without redeploying with `?genclass-aggr=eager` in the URL. The model ships -tuned thresholds for each level; explicit `policy.thresholds` still override them. `runtime.gates()` shows what is -in force. In observe, aggressiveness only moves the report gate (what counts as a likely detection); the possible -tier below it stays at the `eager` value (see [Modes](#modes)). - -## Connect your state (optional) - -Network, user actions, errors and timing are observed with zero code. To let GenClass also protect your state -(roll back an inconsistent cart, revert a stale write), create it through GenClass or wrap the store you already -have. Each option is one line: - -```ts -GenClass.init({ mode: "guard" }); -``` - -- **Where the thresholds come from.** They were fitted on held-out dev data, including the model's own on-policy - traffic, and ship in the model's `meta.json` - ([model card](https://www.npmjs.com/package/@genclass/runtime-model)). A model without a `gate` gets the - runtime's defaults: report 0.6, guard 0.9, heal 0.8. `policy.thresholds: { report, guard, heal }` overrides both. - `rt.gates(trigger)` shows the values in force and where each came from (`"policy"`, `"model"` or `"default"`). -- **Detections.** In every mode, a decision is reported as a detection when its top diagnosis is not `expected` and - its probability is at least the report threshold of the active profile: 0.95 / 0.90 / 0.70 for cautious / - balanced / eager with the default model. -- **Limits.** Actions are limited to 60 per minute overall, 10 per minute on the same subject and 200 per session - (`policy.actionLimits`). A held decision that misses the - hold budget runs the passive action; a background write decision can still revert the write late. -- **Control.** The `allow` / `deny` lists are in `policy`. Switch at runtime with `rt.setMode(mode)`, or stop - consulting the model with `rt.pause()` / `rt.resume()`. - -**The default changed.** `0.1.0-alpha.0` and `0.1.0-beta.1` defaulted to `guard`. Now `GenClass.init()` with no `mode` (and -`@genclass/runtime/auto`, and the script tag without `data-mode`) observes only; pass `mode: "guard"` to let it act. - -**Kill switch.** Append `?genclass=off` to the URL, or set `localStorage.genclass = "off"`, and nothing is installed. -`?genclass=observe|guard|heal` (or the same localStorage value) overrides the mode, including the mode of the `/auto` -entries and the script tag, within limits: `observe` always works; `guard` works when no mode, `guard` or `heal` is -configured (so on the default `/auto` and script tag, but not on `/auto/observe`); `heal` works only when `heal` is -configured, or with `debug: true` (for QA). - -`GenClass.init()` never throws, and a second call returns the first runtime (its options are ignored). Outside a -browser (SSR, Node) it returns a runtime with no observers and no model. For tests and headless use, call -`createRuntime(options)` instead. - -## What it looks for - -At a glance (actions other than flagging need `guard` or `heal`): - -| situation | example | what GenClass can do | -|---|---|---| -| Stale response | an old search response lands after a newer one | deliver it but drop its writes over newer data, or defer it (guard) | -| Race / conflict | a server echo would overwrite what the user just typed | drop the stale part, defer (guard) | -| Duplicate | a double click sends the same order twice | reuse the first response (`coalesce`, guard) | -| Inconsistent state | the cart total no longer equals the sum of the lines | roll back or resync (heal) | -| Failure pattern | an endpoint fails 5 times in a row | serve the last good response (heal) | -| Transient failure | a one-off 503 | retry (heal) | -| Slow or flooding | 8× slower than usual; a render loop hammering an API | hedge (heal), delay (guard) | -| Unusual behaviour | an operation writes different fields than it usually does | flag; roll back (heal) | -| Your own question | "is now a good moment to start the upload?" | `rt.ask()` / `rt.decide()` | - -The runtime decides at nine **triggers**. Only salient ones (a conflict, a repeat, a failure, an anomaly, a broken -relation, an error) reach the model; everything else is a cheap fact computation. - -| trigger | when | passive action | other actions (tier) | -|---|---|---|---| -| `delivery` | a fetch/XHR response or a WebSocket/EventSource message is about to reach the app | `deliver` | `discard` (guard): deliver, but drop the writes it makes over newer data · `defer` (guard): wait for related in-flight work, then decide again | -| `request` | a request is about to be sent | `send` | `coalesce`, `delay` (guard) · `block`, `serve_cached` (heal) | -| `failure` | a request failed (network error, timeout, 5xx/429/408) | `deliver` | `retry`, `serve_cached` (heal) | -| `stall` | a request is far slower than usual | `wait` | `hedge`, `serve_cached` (heal) | -| `mutation` | a salient store write that no delivery decision covered | `apply` | `discard`, `defer` (guard), decided in the background | -| `inconsistency` | a learned relation (`total == sum(items[*].price × qty)`) broke at a settled point | `ignore` | `rollback`, `resync` (heal) | -| `transition` | an operation wrote different fields than it usually does | `ignore` | `rollback`, `resync` (heal) | -| `error` | an uncaught error or rejection | `ignore` | `rollback` (heal) | -| `ask` | your own question (`rt.ask`, `rt.decide`) | | | - -The model also gives a diagnosis: `expected`, `stale`, `conflict`, `duplicate`, `inconsistent`, `failing`, `slow`, -`overload`, `unusual` or `transient`. - -**Delivery decisions (the main change in this version).** GenClass decides about responses and messages *at the -network boundary*, before the app sees them, and never holds or reorders the app's store writes. A delivery is -salient only when: - -- a field the response is predicted to write already holds **newer applied data** (an operation that started later - wrote it since) and no newer request of the same kind is still in flight; -- or it would **revert a pending local change** (an optimistic update whose request is still in flight) to the value - the user's change replaced; -- or it would **replace text the user typed** after the request started. - -The runtime predicts the write set from what the same operation wrote before. For salient candidates it reads the -response body (a clone, at most 256 KB of JSON, waiting at most 100 ms). A response equal to the current values is not -salient. The model then reads facts such as these (real output from the example above): - -``` -The response has search.results = 1 item ["rea-1"]: neither the current value 1 item ["react-1"], nor the value when #4 started. -In 1 earlier completions of GET /api/search its chain wrote search.results (1 of 1 wrote state). -The response to GET /api/search?q=rea (#4) arrived after 1.80s (200); the app has not seen it yet. -This request comes from user typed "rea" into input "Search" (#3), started 1.80s ago. -``` - -Clean in-order typeahead and autosave make no model calls and hold nothing (covered by the unit tests). Other facts -cover: - -- repeats and multi-clicks; -- failure streaks and failure scope (other endpoints, `navigator.onLine`); -- whether a failed POST may have been applied; -- learned cadences (polling, debounced saves); -- latency against learned baselines; -- values known to be stale; -- read-your-writes. - -It catches runtime failures that leave evidence: ordering, staleness, duplicates, broken relations, failure patterns. -The model does not catch logic that is consistently wrong, CSS or security bugs, and it never rewrites your code. A -set of visual and CSS bugs is covered by measured checks instead (next section). - -**Steps that did not happen (opt-in, `0.3.0`).** `GenClass.init({ learn: { expectations: true } })` also learns what -each user action on a route usually does (the requests it sends, what their answers write, where it navigates) and -reports a usual step that did not happen this time: `[GenClass] Expected a request on /cart: clicking button "Add to -cart" usually sends POST /api/cart (...); this time nothing was sent in 2.0 s.` Deterministic counts, no model, no -action in any mode. It is off by default until its false-report rate on correct apps is measured -([RESULTS.md](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/RESULTS.md), "Flow expectations"); -off, its code is never loaded (a lazy chunk, no first-load cost). Design and controls: -[EXPECTATIONS.md](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/EXPECTATIONS.md). - -## Visual and CSS checks (new in 0.3.0) - -GenClass also measures visual bugs that never throw an error: text you cannot read because it is nearly the colour of -its background, images, fonts, stylesheets and media that failed to load, pages that scroll sideways on a phone, -invisible layers that swallow clicks, dialogs hidden behind other elements, and leaks such as "[object Object]" or -"$NaN". Each report names the element, what was measured and the threshold, and suggests a CSS change for that -selector: - -``` -[GenClass] Check (measured): text contrast 1.04:1 on p.hero__subtitle "Ship faster with fewer bugs", below 1.5:1 (WCAG 2.2 contrast formula; unreadable-text threshold): #fafafa on #ffffff from section.hero. Suggested CSS: p.hero__subtitle { color: #737373; } -[GenClass] Check (measured): the page scrolls sideways on a 390 px viewport: div.pricing-table ends at 512 px (122 px too wide, a width wider than the viewport). Suggested CSS: div.pricing-table { max-width: 100%; } -``` - -These are measured checks with fixed thresholds. The model is not involved, does not look at pixels, and never sees -them; they never feed a decision. They abstain when they cannot know: text over photos, video, canvases and gradients -(including layers that ignore the pointer), filters, blend modes, partial opacity, forced colours and page translation -are skipped, not guessed. - -| kind | what it finds | -|---|---| -| `text-invisible` | text below 1.5:1 against a fully known solid background, at rest, on hover, on focus, in light or dark mode | -| `image-broken`, `image-css-broken`, `media-failed` | an `img`, a CSS background image (Chromium) or a video or audio player that failed to load | -| `stylesheet-failed`, `font-failed`, `icon-font-failed` | a first-party stylesheet, a web font in use, or an icon font (boxes or ligature words shown) that failed | -| `csp-blocked` | a style, font, image or media blocked by the page's Content-Security-Policy | -| `text-template-leak`, `text-placeholder`, `text-encoding` | `[object Object]`, `$NaN`, `{{ x }}`, `Invalid Date`; lorem ipsum and placeholder-image hosts; mojibake and double-escaped entities | -| `css-var-undefined` | `var(--x)` without a fallback where `--x` is defined nowhere and the fallback leaves text below 3:1 | -| `image-distorted`, `page-overflow-x`, `viewport-meta-missing`, `layout-shift` | a stretched image; sideways scrolling and its cause; a phone showing the 980 px layout; a layout shift above 0.25 that names the element that moved and does not follow a scroll (Chromium) | -| `page-blank` | the main region shows no text and no media after the page settled | -| `click-blocked`, `pointer-events-leak`, `scroll-lock-stuck` | an invisible layer that took two clicks; `pointer-events: none` or a scroll lock left on `body` | -| `control-unreachable`, `focus-obscured`, `dialog-covered` | a control under a fixed bar with no controls of its own; a focused element or anchor target under a sticky header or banner that paints over it; a dialog under other elements | -| `request-stalled` | a `fetch` or XHR request still pending after 15 s (`visual: { checks: { stallMs } }`), and the button the page disabled or marked `aria-busy` when it started, if it still is; long polls, streams, sockets and endpoints that usually take that long are not checked | -| `ad-heavy` | ad slots (named so by their id, class or ad attributes, or cross-site frames at ad sizes) cover 30% of the page or half the first screen, a fixed ad 30% of the screen, or a full-screen ad sits on top of the page | -| `text-warped` | text that cannot be read as text: lines on top of each other, a line cut through by its box, half an emoji or another character no font can draw, mirrored or skewed text, letters spaced wider than words, text drawn below 8 px | - -**Where they run.** On by default on development hosts (a dev server's port on `localhost`, `127.0.0.1`, `[::1]` or a -`*.localhost` name, and `*.test` names) and when the devtools overlay mounts, where the developer reads the console; -off everywhere else, including hybrid apps served from `https://localhost` (Capacitor), Tauri, Electron files and -intranet hosts. Turn them on in production -with `GenClass.init({ visual: true })` (or `data-visual="on"`, ``) and read -them with a sink or `rt.on("check", f => ...)`; `?genclass-visual=on` turns on report-only checks for the person who -opened the link; `visual: false` or `?genclass-visual=off` turns them off. - -**Cost.** About 0.5 KB gzip of first load (550 B on `/auto`, 524 B on the main entry; see Performance). The checks are a separate file loaded 1.5 s after the page's `load` event -in a visible tab, and all their page reads (passes, the 2 s read of `html` and `body`, the click, hover, focus and -scroll checks) run in idle callbacks: slices of 8 ms, at most 100 ms per pass and 1 s per 10 minutes. One step can -still take longer than a slice when the browser has to lay out a heavy page first (measured in RESULTS.md section 14). -Low-end devices run only the near-free checks (failed assets, CSP, layout shifts, the interaction checks). Nothing -about the checks is sent anywhere. - -**Runtime fixes (opt-in).** With `GenClass.init({ mode: "guard", visual: { on: true, fix: true } })`, GenClass can -apply three narrow fixes: a readable text colour (one rule in its own stylesheet), and removing a leaked inline -`pointer-events: none` or `overflow: hidden` on `html` or `body`. Each is re-measured and undone if it did not help; -a colour fix is measured again every 2 s and after class changes, and undone as soon as the background it was made for -changes. A scroll lock held since the page loaded, with no dialog ever open, is reported but never removed. Every fix -can be undone (`rt.checks().find(c => c.id === "c3").fix.undo()` or the devtools Undo button), and is undone when the -mode drops to observe. Observe mode never changes the page. - -**In your own tests.** `import { checkPage } from "@genclass/runtime/visual"` runs one report-only pass in the page and -resolves with the findings, for a Playwright or Cypress test. For any URL without writing a test: -`npx @genclass/runtime@beta scan ` ([Audit a page](#audit-a-page)). - -Not covered yet: text over photos, video and gradients; canvas and WebGL content; the colour scheme, viewport or hover -state the visitor is not using; layout shifts and failed CSS images outside Chromium; low contrast that is readable but -below WCAG AA. Measured in CI on the demos, templates, compat apps and 40 public home pages -([RESULTS.md section 14](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/RESULTS.md)): 0 false -reports in 128 views of the local pages, and recall 1.00 on planted faults for 19 of the 23 kinds (0.67 to 0.96 for -the other four). On the public pages -the checks were tuned on the same sites (the first run had 44 check errors in 100 loads); there is no held-out site -set yet, so those figures are not a false-report rate for other sites. Design and thresholds: -[VISUAL.md](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/VISUAL.md). +| React ≥ 16.8 (development and production builds): `useState`, `useReducer`, `useSyncExternalStore`, class state | the React DevTools global hook (installed, or chained onto the real extension's); each commit, only re-rendered components are compared | **observed only**: detections and delivery decisions (`deliver` / `defer`); never dropped or rolled back | +| Redux / Redux Toolkit (`configureStore` with `devTools` on, its default; `createStore` with the Redux DevTools enhancer) | the Redux DevTools globals (forwarded to the real extension) | **controllable**: as with GenClass's Redux enhancer: `discard`, late revert, `rollback`, `resync` | +| Zustand with the `devtools` middleware | the Redux DevTools `connect` API; off in production builds unless you pass `enabled: true` (or wrap the store with `genclass` from `@genclass/runtime/zustand`) | **observed only** | + +- **Names.** A component's store is named after the component (`SearchPage.state0`); in production builds, after the + element it renders (`id`, `data-testid`, `aria-label`). At most 3 instances per component and 48 component stores; + a store appears on its first state change. Framework internals and values that are not app data (functions, + promises, DOM nodes, class instances) are skipped. +- **Secrets.** Field names are redacted as usual, and a `useState` whose string equals what a password, card-number + or one-time-code input holds is redacted from then on. Discovered state reaches telemetry only if you opt into + sending situation text. +- **Turn it off:** ``, `data-autostate="off"`, + `window.GENCLASS_CONFIG = { autoState: false }`, or one source: `autoState: { react: false }` (also `redux`, + `zustand`). With `GenClass.init()` it is off unless you import `@genclass/runtime/discover` first and pass + `autoState: true`. +- **Not covered yet:** Vue / Pinia, MobX, Jotai, Recoil, signals, Svelte stores, and state outside these libraries. + A React renderer that loaded before GenClass is observed without exact causes, and one that rendered before it is + not observed at all. With `GenClass.init()` and `@genclass/runtime/discover`, Redux stores created before the + runtime starts are observed only. Register such state with `rt.atom` / `rt.guard` + ([State it can protect](#state-it-can-protect)). +- **Cost** (a pilot app): about 7 KB gzip in `/auto`; a React commit walk of p95 1.8 ms at 4× CPU throttling + ([RESULTS.md §17](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/RESULTS.md#17-browser-cost-on-a-pilot-app-2026-10-09-and-2026-10-10)). + + ## State it can protect Fetch, XHR, WebSocket, EventSource, DOM user events, errors, navigation, storage, long tasks and timers are observed -automatically, with no code. With the one line, React, Redux and Zustand state is found too -([Automatic state discovery](#automatic-state-discovery)); discovered Redux stores are controllable, React and Zustand -state is observed only. A write can be dropped or reverted only when the store goes through GenClass. Each option is -one line: +with no code, and with the one line React, Redux and Zustand state is found too. To let GenClass also protect your +state (roll back an inconsistent cart, revert a stale write), create it through GenClass or wrap the store you +already have; a write can be dropped or reverted only when the store goes through GenClass. Each option is one line: ```ts const rt = GenClass.init({ mode: "guard" }); @@ -877,73 +696,79 @@ const useBoard = create(genclass(rt, "board")((set) => ({ cards: [], move: () => const prefs = rt.guard("prefs", { get: () => store.prefs, set: (v) => store.setPrefs(v) }); ``` -What each kind of protection needs: - | protection | how | works with | |---|---|---| -| delivery `discard` | the response is delivered; its chain's writes to the protected fields are dropped synchronously inside each write, and its other fields (loading flags, counts) apply | atoms, `rt.guard`, `useGenClassState`, Redux and Zustand adapters (also discovered Redux stores). Never offered when every field the response would write is observed only (discovered React or Zustand state): `defer` still is | -| late revert | a background `mutation` decision to `discard` reverts the write if it is at most 2 s old, its fields are unchanged since, and no later write of the same chain followed | any GenClass-aware store | -| `rollback` (heal) | restores the last settled snapshot where every learned relation held | stores GenClass can write (atoms, `rt.guard`, Redux and Zustand adapters) | +| delivery `discard` | the response is delivered; its chain's writes to the protected fields are dropped synchronously inside each write, and its other fields (loading flags, counts) apply | atoms, `rt.guard`, `useGenClassState`, Redux and Zustand adapters, discovered Redux stores. Not offered when every field the response would write is observed only: `defer` still is | +| late revert | a background `mutation` decision to `discard` reverts the write if it was decided at most 800 ms after the write applied (`LATE_REVERT_MS`), its fields are unchanged since, and no later write of the same chain followed | any GenClass-aware store | +| `rollback` (heal) | restores the last settled snapshot where every learned relation held | stores GenClass can write | | `resync` (heal) | calls your `resync` handler | stores registered with `{ resync }` | -`{ hold: false }` on a store keeps its writes out of the opt-in write holds (`policy.holdWrites: true`, off by -default). The default never holds a store write: `set(x); get()` always returns `x`. +The default never holds a store write: `set(x); get()` always returns `x` (write holds are opt-in, +`policy.holdWrites: true`). -## Ask it questions - -The model answers typed questions about what is happening now (the request shape of Jev's System One API): +## Visual and CSS checks (new in 0.3.0) -```ts -const busy = await rt.ask({ type: "noul", instructions: "Is a save in flight or failing?" }); -// { type: "noul", noul: } +GenClass also measures visual bugs that never throw an error, in 26 kinds (23 before `0.3.0-beta.2`): text nearly the +colour of its background, images, fonts, stylesheets and media that failed to load, pages that scroll sideways on a +phone, invisible layers that swallow clicks, dialogs hidden behind other elements, and leaks such as "[object Object]" +or "$NaN"; from `0.3.0-beta.2` also a `fetch` or XHR request still pending after 15 s (`request-stalled`, with the +button the page disabled or marked `aria-busy` for it; `visual: { checks: { stallMs } }` sets the wait; long polls, +streams, sockets and endpoints that usually take that long are not checked), ad slots (named so by their id, class or +ad attributes, or cross-site frames at ad sizes) covering 30% of the page, half the first screen or, fixed, 30% of +the screen, or a full-screen ad on top of the page (`ad-heavy`), and text that cannot be read as text (`text-warped`: +lines on top of each other or cut through by their box, half an emoji or another character no font can draw, +mirrored or skewed text, letters spaced wider than words, text drawn below 8 px). Each report names the element, +what was measured and the threshold, and suggests a CSS change: -const when = await rt.decide("Which upload strategy fits what is happening now?", { - now: "start the upload immediately", - later: "wait until the network is calm and the user is idle", -}); -// "now" | "later" +``` +[GenClass] Check (measured): text contrast 1.04:1 on p.hero__subtitle "Ship faster with fewer bugs", below 1.5:1 (WCAG 2.2 contrast formula; unreadable-text threshold): #fafafa on #ffffff from section.hero. Suggested CSS: p.hero__subtitle { color: #737373; } ``` -Without a model (`model: false`, or when it cannot load) both reject with `GenClassUnavailableError` (`reason`: -`off`, `error`, `timeout` or `destroyed`). `{ timeoutMs }` bounds the wait. With the default model, short factual -questions about the current situation work best; it is not a general-purpose reasoner. - -Standing questions ride along with built-in decisions: `rt.question({ id, on: ["failure"], question, onAnswer })`. +These are measured checks with fixed thresholds, reported as kind `"check"`; the model never sees them and they never +feed a decision. They skip what they cannot know (text over photos, video, canvases, gradients). They run by default +on development hosts (a dev server's port on `localhost` and similar, `*.test` names) and when the devtools overlay +mounts; elsewhere turn them on with `visual: true`. First-load cost is about 0.5 KB gzip; the checks load 1.5 s after +`load`, read the page in idle slices of 8 ms and send nothing. Opt-in fixes (`visual: { on: true, fix: true }` in +guard or heal mode) are re-measured and undoable. `checkPage()` from `@genclass/runtime/visual` runs one pass in a +test; `npx @genclass/runtime@beta scan ` checks any URL without one ([Audit a page](#audit-a-page)). + +On the demos, templates and compat apps: 0 false reports in 128 page views, and in the bench's run 5 recall 1.00 on +planted faults for 19 of the 23 kinds (0.67 to 0.96 for the other four; the planted forms are the checks' author's). +The checks were tuned on 40 public home pages, where run 5's 6 findings in 97 loads were not reviewed, so there is no +false-report rate for other sites yet +([RESULTS.md §14](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/RESULTS.md#14-visual-bench-false-reports-recall-and-cost-github-runners-chromium-153-2026-10-13-branch-wave2visual)). +`request-stalled` on the demos: 0 false reports in 120 clean and chaos runs, and 11 of 18 planted 60 s hangs found +(polls on a learned schedule are skipped by design; +[RESULTS.md §23](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/RESULTS.md#23-issue-94-request-stalled-on-the-demos-and-the-first-load-cost-of-summaryrequests-local-2026-10-25-branch-issue-94)). +Every kind: [API.md](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/API.md#measured-visual-checks); +thresholds: [VISUAL.md](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/VISUAL.md). ## Observability -- **Console:** one line per detection and per intervention, with the evidence collapsed below it. Identical repeats - within a minute are summarised. `report: "silent"` or `report: (r) => …` to route them yourself. Possible - detections (observe, `0.3.0`) are one `console.debug` line each, ending in `(possible)`; a report function gets - them only with `detect: "possible"`. -- **Events:** `rt.on("detect" | "decide" | "act" | "event" | "status" | "report", fn)` returns an unsubscribe function. -- **What it saw:** `rt.summary().requests` counts the `fetch` and XHR requests of this page load, in total and per host - (`"same-origin"` or the API's host): `requests`, answers by status class (`2xx` to `5xx`), `network` (no answer), - `aborted`, `notJson` (read with `response.json()` while the Content-Type is not JSON), `byGenClass`, `pending`, and - `latency` (`p50`, `p95` of the answers that did not fail). A quiet summary with `requests: 0` means GenClass did not - see your requests (it loaded after them, or the app keeps its own `fetch` reference); requests counted with nothing - in `detections` means it saw them and nothing was salient: a 404 or a non-JSON answer your own `catch` handles is - counted here, not reported as a detection. It is counted by the measured checks' chunk, so it is there wherever the - checks run (development hosts by default; `visual: true` or `data-visual="on"` in production, the devtools overlay, - `rt.checkNow()`), and it costs no first-load bytes. Kept in the page: telemetry does not send it. -- **`rt.explain(id)`:** the exact situation text the model read, its facts and timeline, every answer with - probabilities, and what changed. -- **`rt.decisions()`, `rt.interventions()`, `rt.history()`, `rt.inflight()`, `rt.situation()`** for introspection. -- **Undo, where it exists:** `ActionRecord.undo()` is set for `discard` (delivery and write), late reverts, - `rollback` and chain reverts, plus custom actions that register `onUndo`. `defer`, `coalesce`, `delay`, `block`, - `serve_cached`, `retry`, `hedge` and `resync` cannot be undone; their record says what changed. -- **Altered responses** carry an `x-genclass` header: `coalesced`, `cached` or `blocked`. -- **Audit trail:** `rt.audit()` returns a JSON-serialisable record of every decision (acted on or not), action, undo, - breaker trip and mode change, each with the mode, aggressiveness profile, gate thresholds, the model's - probabilities and the model's name, version and sha256. `audit: { sink }` ships it to your logging. -- **What it touches:** [INTERCEPTION.md](INTERCEPTION.md) lists every API GenClass wraps or listens to, what each mode - may change, what it never does, every action's preconditions and undo, and how `disable()` restores the page (a - unit test keeps it in sync with the code). Threat model: - [docs/runtime/THREAT-MODEL.md](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/THREAT-MODEL.md). -- **Devtools overlay** with four views: **Interventions** (what GenClass did, with Undo where it exists), - **Detections** (what it noticed but did not act on), **Activity** (a live log of requests, writes, user actions and - errors with their causal links) and **Now** (what the model would see at this moment). About 52 KB minified / - 17 KB gzip, so load it in development only (`init` does this for you): +- **Console:** one line per detection and per intervention, with the evidence collapsed below it; identical repeats + within a minute are summarised. `report: "silent"` or `report: (r) => …` to route them yourself. +- **Events and introspection:** `rt.on("detect" | "decide" | "act" | "report", fn)`; `rt.explain(id)` gives the exact + situation text the model read, every answer with probabilities, and what changed; `rt.decisions()`, + `rt.interventions()`. +- **What it saw** (from `0.3.0-beta.2`): `rt.summary().requests` counts this page load's `fetch` and XHR requests, in + total and per host (`"same-origin"` or the API's host): `requests`, answers by status class (`2xx` to `5xx`), + `network` (no answer), `aborted`, `notJson` (read with `response.json()` while the Content-Type is not JSON), + `byGenClass`, `pending`, and `latency` (`p50`, `p95` of the answers that did not fail). `requests: 0` means GenClass + did not see your requests (it loaded after them, or the app keeps its own `fetch` reference); requests counted with + nothing in `detections` means it saw them and nothing was salient (a 404 or a non-JSON answer your own `catch` + handles is counted, not reported). It is counted by the measured checks' chunk, so it is there wherever the checks + run (development hosts by default; `visual: true` or `data-visual="on"` in production, the devtools overlay, + `rt.checkNow()`), costs no first-load bytes, and stays in the page: telemetry does not send it. +- **Undo and audit:** `ActionRecord.undo()` for `discard`, late reverts, `rollback` and custom actions with `onUndo`; + altered responses carry an `x-genclass` header. `rt.audit()` records every decision, action, undo and mode change + with the gate, the probabilities and the model's sha256. +- **What it touches:** + [INTERCEPTION.md](https://github.com/genclass-dev/GenClass-lib/blob/main/packages/runtime/INTERCEPTION.md) lists + every API GenClass wraps or listens to, what each mode may change, and every action's undo (a unit test keeps it in + sync with the code). +- **Devtools overlay** with six views: Interventions (with Undo), Detections, Checks, Activity (requests, writes, + user actions and errors with their causal links), Expectations and Now. It is a separate chunk, so load it in + development only (`init` does this for you): ```ts if (import.meta.env.DEV) { @@ -954,358 +779,121 @@ if (import.meta.env.DEV) { ### See it where you already look -> **New in `0.3.0`.** `0.2.x` ignores `data-sink` and `sink=`, stops at `init --sink` with "unknown option", has no -> `@genclass/runtime/sinks` exports, and prints "a sink failed" for a sink name in `GENCLASS_CONFIG.sinks`. Pin -> `0.3.0-beta.0` or later in the script tag below. - -In production the `[GenClass]` console lines print in your users' browsers, not yours. Sinks send every detection, -intervention, undo, breaker trip and runtime error to your own side as well: one JSON line per record on the console -for a log shipper (`console-json`), batched POSTs to an endpoint you own (`webhook`), or a function or `{ send }` object -of yours. The records go only where you point them. The console default does not change. - -**Sentry, PostHog and Datadog.** Forwarding findings to those tools is not part of the runtime. On every -genclass.dev plan, Free included, the GenClass collector forwards detections to them (or Slack, or a signed webhook) once you turn it on in the dashboard -([genclass.dev/pricing](https://genclass.dev/pricing)). The runtime itself has no adapter for them. A page config that still names `sentry`, `posthog` or `datadog` gets one console warning that says -so, and nothing is loaded or sent for those names; `init --sink sentry` stops with a usage error that says the same. - -With `/auto` or the script tag you write no adapter code yourself: - -```bash -npx @genclass/runtime@beta init --sink webhook:/genclass # or console-json; repeat it or comma-separate -``` - -```html - - - -``` - -In a project with an entry file `init` writes the names to its option file; on a plain HTML page it sets `data-sink`, -in an Astro layout `sink=` in the meta line. A name loads its adapter on the first record, never before. A `` -tag only takes a same-origin webhook URL, checked as the browser resolves it (against the page and -`document.baseURI`); a cross-origin one is ignored with a console warning, so put a collector on another origin in -`data-sink` or `window.GENCLASS_CONFIG`. A webhook URL is made absolute once, at init, and only that endpoint (origin -and path) is excluded from what GenClass observes. Add `:noevidence` to `console-json` (`--sink console-json:noevidence`) -to print the structured fields without the evidence text. In code, pass the adapters to `GenClass.init`: - -```ts -import { GenClass } from "@genclass/runtime"; -import { consoleJson } from "@genclass/runtime/sinks/console-json"; -import { webhook } from "@genclass/runtime/sinks/webhook"; - -GenClass.init({ - sinks: [ - consoleJson(), // console.log('{"source":"genclass","schema":1,"kind":"detection",...}') for log shippers - webhook({ url: "/genclass" }), // batched POSTs of { schema: "genclass-sink/1", version, records } to your endpoint - ], -}); -``` - -`@genclass/runtime/sinks` exports both. Neither adds a dependency or ever throws into your app. Both forward every -record kind, the per-session `summary` record included; the webhook caps repeats of one bug at 10 per page load -(`maxPerSignature`). Both take `kinds`, `sampleRate` and `evidence`. - -**What a sink receives** is the record the runtime built and nothing else: `kind`, `route` (the page path with -id-like segments and segments holding an `@`, such as an e-mail address, as `:id`, as telemetry sends it from 0.3.0), `diagnosis` and its -probability `p`, `action`, `tier`, `trigger`, `mode`, `label`, `reason`, `changed` (state paths), `sessionId` and your -session `tags`, plus the evidence: the console line, the subject and the timeline, with URL query and fragment values -replaced by `…`. Timeline lines can hold short fragments of app data, as the console does (a store value summary, an -error message, text typed into an ordinary input; secret-named fields are `[redacted]`). `evidence: false` on an -adapter sends the structured fields only. A sink never gets the full situation text the model read, request or -response bodies, headers or cookies. The webhook POSTs exactly those records as -`{ schema: "genclass-sink/1", version, records }`, by `navigator.sendBeacon` (which GenClass never wraps, and which -survives page exit), else `fetch` with `keepalive`; the body is JSON sent as `text/plain` (no preflight), so parse it -as JSON. One POST carries the records GenClass hands over in one idle-time round, at most 50 records and 32,000 -bytes of them, and each round sends at most one: the browser allows 64 KiB of beacons in flight per page, shared with -yours (your analytics on page exit, say), so GenClass never takes more than about half. What does not fit waits for the -next round, and on page hide one last batch goes (the session summary first) and the rest is dropped. Pass a `clock` -and `flushMs` to batch over a time window instead. `headers` switches it to `fetch`; then list the URL in -`requests.ignore` (the zero-code forms do that for you). - -## Extend it - -Plugins can add their own observers, facts, actions and questions: - -```ts -rt.use({ - name: "sync", - actions: [{ - name: "pause_sync", - description: "pause background sync until the page is visible again", - on: ["failure", "stall"], - tier: "heal", - run: (ctx) => { sync.pause(); ctx.describe("Paused background sync."); ctx.onUndo(() => sync.resume()); }, - }], -}); -``` - -Plugins can add observers (`setup`), facts, actions, standing questions and diagnosis labels. The model reads each -action's description. The goal is that clearly described actions work without retraining; this has not been -measured. +In production the console lines print in your users' browsers. Sinks also send every detection, intervention, undo +and error to your side: JSON console lines for a log shipper (`console-json`), batched POSTs to your endpoint +(`webhook`), or a function or `{ send }` object of yours. `npx @genclass/runtime@beta init --sink webhook:/genclass`, +`data-sink="webhook:/genclass"`, or `sinks: [consoleJson(), webhook({ url: "/genclass" })]` from +`@genclass/runtime/sinks` (adapters and names: new in `0.3.0`). A sink never gets the situation text, bodies, headers +or cookies. Forwarding to Sentry, PostHog or Datadog is not part of the runtime: for an app in a GenClass account +with telemetry on and a token, the genclass.dev collector forwards likely detections once you turn it on in the +dashboard (every plan, Free included). Details: +[SINKS.md](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/SINKS.md). ## Configuration -GenClass runs with safe defaults (`mode: "observe"`, a `"balanced"` gate that applies once you opt into guard or heal, circuit breaker on). The options below narrow where it acts, cap how much it does, and send findings to your tools. None of them tell the model what a bug looks like. They only scope and limit it. - -- **Your app** (`0.3.0`): `token` (your dashboard, see [Your dashboard](#your-dashboard)), `scope` (`"app"` or `"functions"`, see [protect()](#function-specific-protect)). -- **Activation**: `enabled` (a boolean, a predicate, or a subscribable feature flag; while it is false the model is never downloaded), `rt.disable({ undo: true })` (remote kill that also rolls back recent actions), `sample` (the fraction of sessions allowed to act; the rest only observe). -- **Scope**: `routes` (per-route mode and aggressiveness, which can only be lowered), `requests.ignore` (analytics traffic), `requests.protect` (endpoints that are never held, retried, cached or discarded), `requests.labels` (endpoint names for reports), `requests.correlate` (attach your trace id to records). -- **Safety**: `breaker` (automatic downgrade after undos, or after errors that follow an action), `shadow` (records what a higher mode would have done), `onBeforeAction` + `vetoMode` (a synchronous veto, or a report-only trial of one), `policy.actionLimits` (per-minute, per-subject and per-session caps), `policy.holdBudgetMs` (a hard ceiling on added latency). -- **Your own telemetry**: `sinks` (structured, redacted records; ready-made adapters in `@genclass/runtime/sinks` for JSON console lines and a webhook to your own endpoint, see "See it where you already look"; forwarding to Sentry, PostHog or Datadog is not in the free runtime), `session` (id and tags, never shown to the model), `report: "interventions"` (a quiet production console), `rt.summary()`, `rt.on("shadow" | "breaker" | "limit" | "modelBudget", cb)`. -- **Loading and cost**: `model.loadIf`, `model.threads`, `model.timeoutMs`, `model.maxDecisionsPerMinute`, `model.unloadAfterIdleMs`. - -Action limits default to 60 per minute overall, 10 per minute on the same subject (store field or endpoint), and 200 per session. - -URL overrides (`?genclass`, `?genclass-mode`, `?genclass-aggr`, `?genclass-sample`, and `localStorage.genclass`) can only lower settings, unless `debug: true` is set or the page is on a development host (`localhost`, `*.local`, `*.test`, private networks). Since 0.2.2 a link such as `?genclass=guard` can no longer raise a production site that set no `mode` (it stays in `observe`). +GenClass runs with safe defaults (`mode: "observe"`, a `"balanced"` gate once you opt into guard or heal, circuit +breaker on). The options narrow where it acts, cap how much it does, and send findings to your tools; none of them +tell the model what a bug looks like. + +- **Your app** (`0.3.0`): `token` ([Your dashboard](#your-dashboard)), `scope` ([protect()](#function-specific-protect)). +- **Activation**: `enabled` (a boolean, a predicate or a feature flag; while false the model is never downloaded), + `rt.disable({ undo: true })` (remote kill that also rolls back recent actions), `sample` (the share of sessions + allowed to act). +- **Scope**: `routes` (per-route mode, which can only be lowered), `requests.ignore`, `requests.protect` (endpoints + never held, retried, cached or discarded), `requests.labels`. +- **Safety**: `breaker` (automatic downgrade after undos or errors), `shadow` (records what a higher mode would have + done), `onBeforeAction` (a synchronous veto), `policy.actionLimits`, `policy.holdBudgetMs`. +- **Reporting and loading**: `sinks`, `session` (tags, never shown to the model), `report: "interventions"`, + `rt.summary()`; `model.loadIf`, `model.preload`, `model.maxDecisionsPerMinute`, `model.unloadAfterIdleMs`. + +URL overrides can only lower settings. `?genclass=observe|guard|heal` and `localStorage.genclass` may also raise the +mode on a development host (http(s) on `localhost`, `127.0.0.1`, `[::1]` or `*.localhost` with an explicit port, or +`*.test`); since `0.3.0-beta.2` and `0.2.3`, `debug: true` no longer lets a link raise anything (it is console logging +only), and `?genclass-mode`, `?genclass-aggr` and `?genclass-sample` only demote. `rt.status.effectiveMode` is the mode +in force now. ```ts -import { GenClass } from "@genclass/runtime"; +import { GenClass, protectPreset } from "@genclass/runtime"; const rt = GenClass.init({ mode: "guard", aggressiveness: "cautious", - enabled: { // live feature flag; flipping off disables mid-session - get: () => flags.isEnabled("genclass") && consent.analytics, - subscribe: (cb) => flags.onChange("genclass", cb), - }, sample: 0.1, // 10% of sessions act, 90% observe - routes: [ - { match: "/checkout/*", mode: "observe" }, - { match: /^\/admin/, mode: "off" }, - ], - requests: { - ignore: ["https://www.google-analytics.com/", /sentry\.io/, "*/rum/*"], - protect: ["/api/auth/", "/api/payments/", (r) => r.method !== "GET" && r.url.includes("/billing")], - crossOrigin: "observe", - labels: [{ match: "/api/orders", label: "place order" }], // reports only by default - correlate: (r) => r.headers["x-request-id"], - }, - breaker: { undos: 2, errorsAfterAction: 3, attributionMs: 5000, downgradeTo: "observe" }, + routes: [{ match: "/checkout/*", mode: "observe" }, { match: /^\/admin/, mode: "off" }], + requests: { ignore: [/sentry\.io/], protect: [...protectPreset("payments", "auth"), "/api/cart"] }, + breaker: { undos: 2, errorsAfterAction: 3, downgradeTo: "observe" }, shadow: "heal", - onBeforeAction: (a) => !(cart.isSubmitting && a.tier === "heal"), - vetoMode: "enforce", - policy: { actionLimits: { perMinute: 30, perSubject: 3, perSession: 100 }, holdBudgetMs: "auto" }, report: import.meta.env.PROD ? "interventions" : "console", - sinks: [ - { send: (r) => navigator.sendBeacon("/genclass", JSON.stringify(r)), - kinds: ["intervention", "undo", "breaker", "summary"] }, - { send: (r) => myLogger.info("genclass", r), sampleRate: 0.2 }, - ], - session: { tags: { release: __RELEASE__, tenant: tenantTier } }, - learn: { persist: "local", key: "genclass:shop" }, // version = session.tags.release at init - model: { - preload: "idle", - loadIf: { minDeviceMemoryGB: 8, saveData: "skip" }, // default { minDeviceMemoryGB: 4, saveData: "skip" } - threads: 2, - timeoutMs: 10_000, - maxDecisionsPerMinute: 30, - unloadAfterIdleMs: 300_000, - }, }); - -rt.on("breaker", (e) => log.warn("genclass downgraded", e)); -onLogin((u) => rt.setSession({ tags: { plan: u.plan } })); -onLogout(() => rt.learn.clear()); -onIncident(() => rt.disable({ undo: true })); ``` -**What is in force:** `rt.status.effectiveMode` is the mode the runtime acts in now (the requested mode, demoted by -`sample`, the breaker and the matching `routes` rule). `rt.status.scope` is the same for the current route: -`{ route, mode, aggressiveness }`, plus `rule` (index into `routes`) and `ceiling` (that rule's mode) when a rule -matches. Route rules only ever lower the mode. `rt.gates()` carries the same `mode` next to the thresholds. - **Payments, checkout and sign-in: keep them observe-only.** Automatic intervention in money and identity flows is not -recommended. `requests.protect` is checked before everything else, in every mode, and covers the request, its -response and everything its response callbacks cause. Opt-in presets (adapt or extend them to your endpoints): - -```ts -import { GenClass, protectPreset } from "@genclass/runtime"; -GenClass.init({ mode: "guard", requests: { protect: [...protectPreset("payments", "auth"), "/api/cart"] } }); -// JSON configs (window.GENCLASS_CONFIG, the file `init` writes): requests: { protect: ["preset:payments", "preset:auth"] } -``` +recommended. `requests.protect` is checked first, in every mode, and covers the request, its response and everything +its response callbacks cause. `protectPreset("payments", "auth")` (in JSON configs, `"preset:payments"`, +`"preset:auth"`) matches checkout, payment, billing, order, Stripe-style and sign-in URLs; +`npx @genclass/runtime@beta init` suggests them when your project uses a payment SDK +([INTERCEPTION.md](https://github.com/genclass-dev/GenClass-lib/blob/main/packages/runtime/INTERCEPTION.md#money-and-identity-flows)). -`payments` matches URL words such as checkout, payment(s), payment intents, billing, invoices, charges, refunds, -subscriptions, orders, transactions, transfers, wallets and Stripe/PayPal/Braintree/Adyen/Klarna/Square; `auth` -matches login, logout, sign-in/up, oauth, tokens, sessions, SSO, MFA, OTP, passwords and verify. `npx -@genclass/runtime init` suggests them when your project uses a payment SDK. Details: -[INTERCEPTION.md](INTERCEPTION.md#money-and-identity-flows). +**Rollout recipe:** start with `mode: "observe", shadow: "guard"`, then `mode: "guard", sample: 0.05`, and widen +`sample` while `rt.summary().undos` and `breaker` events stay quiet. Every option: +[API.md](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/API.md#options); why each exists: +[OPTIONS-SPEC.md](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/OPTIONS-SPEC.md). -**Rollout recipe:** start with `mode: "observe", shadow: "guard"`, and compare the shadow records with your undo and complaint rates. Then switch to `mode: "guard", sample: 0.05`. Watch `rt.summary().undos` and `breaker` events, and widen `sample` as they stay quiet. For QA, `?genclass-sample=1` together with `debug: true` forces a session into the acting group. +## Ask it questions -## Model quality +The model answers typed questions about what is happening now: -The default model is `genclass-runtime-r17` 2.0.0-rc4t (`@genclass/runtime-model@0.2.0`, run `r17-v2dT`): a -GenClass encoder trained on simulated apps and on real apps driven in headless Chromium. It ships a gain gate (act -only when the model's expected gain over doing nothing clears a margin) with one fitted profile per aggressiveness -level. Held-out test, from -[RESULTS.md §1](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/RESULTS.md): +```ts +const busy = await rt.ask({ type: "noul", instructions: "Is a save in flight or failing?" }); +// { type: "noul", noul: } -| profile | guard false interventions | guard recall (clear / real) | heal false interventions | heal recall (clear / real) | report threshold | -|---|---|---|---|---|---| -| `cautious` | 0.005% | 2.4% / 1.4% | 0.26% | 3.7% / 3.8% | 0.95 | -| **`balanced` (default)** | 0.13% | 7.8% / 7.5% | 0.59% | 7.1% / 10.4% | 0.90 | -| `eager` | 0.54% | 14.2% / 21.0% | 1.84% | 17.1% / 24.2% | 0.70 | +const when = await rt.decide("Which upload strategy fits what is happening now?", { + now: "start the upload immediately", + later: "wait until the network is calm and the user is idle", +}); +// "now" | "later" +``` -False interventions are on simulated apps; on held-out real apps they were 0.00% for every profile. Our targets are -0.1% (guard) and 0.5% (heal): `cautious` meets both, `balanced` is slightly over both, `eager` trades well over -them for about twice `balanced`'s recall. Pick `cautious` if a wrong intervention costs more than a missed one. +Without a model both reject with `GenClassUnavailableError`. Short factual questions about the current situation work +best; it is not a general-purpose reasoner. -The observe-mode numbers below were measured on the previous model, 0.1.0 (`r17-v2b`, report threshold 0.85), and -have not been re-measured for 0.2.0: +## Extend it -| model 0.1.0 | Simulated apps | Real apps | -|---|---|---| -| Diagnosis accuracy | 84.2% | 83.6% | -| Action accuracy | 77.8% | 80.0% | -| **Observe:** decisions flagged where nothing was wrong (report 0.85) | 1.41% (149 / 10,582) | 3.78% (305 / 8,079); 1.05% without one app | -| Observe: problem decisions flagged | 65% | 61% | -| Observe: flags with the right diagnosis | 94% | 89% | - -What this means in practice: - -- **Observe (default).** Expect roughly 1 to 4 false flags per 100 decisions where nothing was wrong (model 0.1.0; - `cautious` raises the report threshold to 0.95, `eager` lowers it to 0.70). `0.3.0` adds *possible* detections at - the `eager` value (0.70) in observe, on `console.debug` only. The 0.2.0 fit held eager-profile detections (likely - and possible together) to at most 5% (real apps) / 6% (simulated) false per clean decision on held-out test, - against 2% / 3% at `balanced`; the share of possible detections that are false is not measured and is expected to - be well above the likely tier's - ([RESULTS.md §11](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/RESULTS.md)). - - In one held-out real app, `oss-rtk-conduit`, the model flagged 25% of clean decisions as `stale`, all on its - `article.inProgress` field. A single recurring pattern in your app can do the same. - - Each flag names its evidence, and identical lines within a minute are summarised. - - Raise `policy.thresholds.report` to see fewer flags. At 0.9: 0.9% (simulated) and 3.4% (real) false, with 58% and - 52% of problems flagged. -- **Guard (opt-in).** When it acts, it is almost always right, but it rarely acts. Expect it to miss most problems. - `balanced` is slightly above the 0.1% false-intervention target; `cautious` is under it. -- **Heal (experimental).** At `balanced`, 0.59% of heal interventions were wrong on simulated apps (target 0.5%); - with model 0.1.0, heal actions on failures (`retry`, `serve_cached`) were its least precise actions. -- **Not yet met:** diagnosis ≥ 95%, clear-case recall ≥ 80%, and false flags ≤ 1% on held-out data. Training - continues (a larger teacher model, distillation, more real-app data). -- **Format.** The model was trained on situations rendered by the runtime at `situation-v2`. This version renders - `situation-v2.3`: - - an extra stall fact; - - `retry` offered only when safe; - - fewer learned relations; - - redacted values printed differently. - - The retry change is accounted for in the numbers above; the others were not measured. - -**Round 1** (previous format `situation-v1`, not compatible with this runtime): R17 reached 90.5% diagnosis and 81.9% -action accuracy on held-out simulated apps, with 0.05% / 0.24% guard / heal false interventions, but acted on only -7.7% of clear cases. The analysis found the limit was in the situation text and the labels, not the model size. The -v2 format adds measured facts for those cases, with relabelled data. - -The "never make a correct app worse" check uses an always-passive model in heal mode, compared against observe mode, -on 66 real apps (6 seeds each), with the v2 runtime as of `0.1.0-alpha.1` (not re-run on this version). -**0 of 396 clean runs** changed, measured on final page text (inputs and alerts -excluded) and server state. The check does not compare request timing or store contents, and it does not compare -observe mode against running without GenClass. With network chaos, 3 of 198 runs differed: a request held about -25 ms changed the simulated network's draws. Details: -[RESULTS.md §4](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/RESULTS.md). +Plugins (`rt.use({ name, setup, actions, ... })`) can add observers, facts, actions, standing questions and diagnosis +labels. The model reads each action's description; the goal is that clearly described actions work without +retraining, which has not been measured +([API.md](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/API.md#plugins)). ## Performance -- **Most work never reaches the model.** Facts are computed for every write and request. Measured by the perf tests - on a shared VM: - - | operation | time | - |---|---| - | keystroke write into a store holding a 5,000-item array | 0.22 ms | - | Redux-style dispatch on 5,000 entities | about 0.7 ms | - | settled-point check | 0.3 ms | - -- **Bundle:** what every page loads is about 108 KB gzip for `@genclass/runtime/auto` with automatic state - discovery and 100 KB for the main entry with `GenClass.init()`, which does not carry discovery: 108.25 KB and - 100.13 KB, minified and gzipped by esbuild in `test/bundle.test.ts` (CI run 38676978903), which keeps them under - 108.30 KB and 100.20 KB (raised for the measured visual checks from 107.74 / 99.71 KB, the `0.3.0-beta.0` release - build's 106.24 / 98.21 KB plus 1.5 KB). `0.2.2` measured - 99.17 KB and 91.70 KB; `0.3.0` adds the app token and `protect()`, the device floor, telemetry sampling, the sink - names' proxy and the learned-state persistence (docs/runtime/RESULTS.md §10), then observe's detection tiers and - deferred situations (+1.3 / +1.4 KB, §11), and on `/auto` the one warning for the sink names `sentry`, `posthog` and - `datadog` (+0.10 KB, §10), then the measured visual checks' stub (+0.54 / +0.51 KB, §10 and §13). The model worker, ONNX Runtime Web's JavaScript (about - 70 KB gzip for WASM, 115 KB with WebGPU, minified), the devtools, the sink adapters, the measured visual checks - and the opt-in flow expectations are separate chunks loaded on demand. - ONNX Runtime's own `.wasm` (14 MB, 3.1 MB brotli; 27 MB, 5.5 MB brotli with WebGPU) is never part of your build: - the runtime fetches it when the model loads (from jsDelivr, or `model.ortWasmPaths`). The script-tag file is about - 129 KB gzip (128.56 KB in the same CI run, with discovery, the sink names' proxy and the flow expectations inline); - its model worker, ONNX Runtime glue, sink adapters and visual checks load on demand. -- **Model:** the default model is 10.2 MB (q8, pruned 16k vocabulary) on WASM and 13.6 MB (fp16) on WebGPU with - `shader-f16`. With onnxruntime-web 1.30 on single-thread WASM, measured in Node on the training VM, a forward pass - took about 182 / 330 / 599 ms at 500 / 780 / 1,170 tokens: about 180 ms per decision at the single-thread budget - (1,000 characters, about 500 tokens with the questions), which is the budget single-thread WASM uses. -- **Holds are bounded.** A held response or request waits at most the hold budget, then proceeds unchanged. The - default `"auto"` budget is 1.5 × the median of the last 20 model latencies, clamped to 150–800 ms; it is 300 ms - before any latency is known. - - A hold happens only if the model is expected to answer within the budget. - - Reading a response body for salience adds at most 100 ms. - - Background (non-held) decisions have a 5 s deadline. -- **Situation size by device:** 2,400 characters on WebGPU; on WASM, 1,000 (1 thread) to 2,000 (4 threads). - Override with `situation: { budget }`. -- **Threads:** serving the page with `Cross-Origin-Opener-Policy: same-origin` and - `Cross-Origin-Embedder-Policy: require-corp` enables WASM threads (about 3× faster with 4 threads in Chromium, - measured with the round-1 model of the same size). -- **Loading:** the model loads at idle after page load (`model.preload: "idle"`) from - `https://cdn.jsdelivr.net/npm/@genclass/runtime-model@0.2.0/files/`, except on devices that report 2 GB of - memory or less and on Save-Data connections (the default `model.loadIf` since `0.3.0`, see [Costs](#costs)). It is cached in - Cache Storage and checked with sha256. To self-host, see - [Content-Security-Policy and self-hosting](#content-security-policy-and-self-hosting). `model: false` loads no - model (nothing is detected or prevented then). - -## Costs - -Measured in a real app (a pilot app, 2026-10-09: a Next.js 16 ordering site, production build, Playwright Chromium, -390×844 mobile viewport, 4× CPU throttling, medians of 5 runs, observe mode, `0.1.0-beta.3`), the same build with and -without `?genclass=off`: - -| | with GenClass | -|---|---| -| JavaScript on every page (first load, gzip) | about +108 KB (`/auto` with discovery; 100 KB for the main entry, see Performance) | -| Main thread: load event, LCP, long tasks | no measurable change; main JS heap +0.4 MB | -| **Memory: renderer process (holds the model worker)** | **+190 to 280 MB RSS** while the model is loaded (cold 448 vs 169 MB, warm 361 vs 171 MB) | -| **First visit download** | **about 13 MB** after the load event (q8 model 10.2 MB, ONNX Runtime wasm 3.07 MB brotli, tokenizer 207 KB); WebGPU devices: 13.6 MB fp16 model plus the larger ONNX Runtime build | -| Later visits | about 1 KB (`model.json` revalidated); the rest comes from Cache Storage, checked with sha256 | -| Model load | WASM (1 thread): 0.9 to 1.3 s cold, 0.4 s warm. WebGPU (Apple M-series): 12 s cold, 2 s warm | -| App build output | `0.1.0-beta.3`: +41 MB of unused ONNX Runtime files. `0.2.0`: none | - -**Automatic state discovery** (`0.2.0`, the pilot app with only the one line, 2026-10-10; same build with -`autoState: false` as the control; Playwright Chromium, 390×844, observe mode, telemetry off): - -| | cost | -|---|---| -| JavaScript (first load, bytes on the wire) | +7 KB on `/menu` (discovery's share of the `/auto` entry; `GenClass.init()` setups do not load it) | -| React commit walk, 4× CPU throttling | p50 0.2 ms, p95 1.8 ms, max 2.1 ms per commit including recording the change; total about 5 ms over the `/order` visit | -| React commit walk, no throttling, 54 bench trials (about 2,000 commits) | p50 below 0.1 ms (the timer's resolution), p95 0.8 to 0.9 ms, max 3.8 ms | -| Main-thread JS heap (after GC, 4× throttling) | 5.7 MB vs 5.2 MB without discovery | -| Long tasks (> 50 ms) | unchanged (1 per run, 65 to 70 ms, with and without) | - -**On phones** the memory and the download are what matter (the page may also run a call, the camera or a -recorder). **By default the model is not loaded on devices that report 2 GB of memory or less, nor on Save-Data -connections** (since `0.3.0`; `0.2.x` loads everywhere and only lazily on Save-Data; `model.loadIf` defaults to `{ minDeviceMemoryGB: 4, saveData: "skip" }`; `navigator.deviceMemory` is -quantised to powers of two, so 4 is exactly that floor; Safari and Firefox do not report it and load). The runtime -then keeps observing without a model (nothing is detected or prevented on that device) and prints one line, -`[GenClass] Model not loaded: this device reports 2 GB of memory and model.loadIf asks for at least 4 GB. …`. -Options on top of that: - -```ts -GenClass.init({ - model: { - // no model on phones at all (Chromium reports userAgentData.mobile; Safari does not), or a higher floor - loadIf: (env) => !env.mobile && (env.deviceMemoryGB ?? 8) >= 8, - // or everywhere, floor and Save-Data ignored: loadIf: () => true - preload: "lazy", // download on the first salient situation, not at idle on every page - unloadAfterIdleMs: 60_000, // free the worker's memory after a minute without a decision - }, -}); -``` - -`model: false` keeps the runtime observing (console reports, `rt.status`, devtools) with no download and no worker. +- **Most work never reaches the model.** `test/review-perf.test.ts` (run in CI, reported, not blocking) holds a + keystroke write into a store with a 5,000-item array under 1 ms, a Redux-style dispatch on 5,000 entities under + 1 ms (2 ms for a write from async work) and a settled-point check under 16 ms. Its last recorded run (a shared VM, + 2026-10-07, the runtime just before `0.1.0-alpha.1`, commit `bac4409`; not re-recorded since) measured 0.22 ms, + about 0.7 ms and 0.3 ms. +- **Bundle:** the published `0.3.0-beta.0` measured 108.25 KB gzip for `/auto` and 100.13 KB for the main entry + ([RESULTS.md §10](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/RESULTS.md#10-release-030-beta0-first-load-and-telemetry-volume-ci)). + The interaction-analytics stub measured about 108.5 / 100.4 KB on its branch (§15); `0.3.0-beta.1` to `beta.4` are + not re-measured here (`summary().requests` added 34 B in `0.3.0-beta.2`, §21), and `test/bundle.test.ts` caps it + at 108.6 / 100.48 KB. The model worker, ONNX + Runtime, the devtools, the sinks and the visual checks load on demand; ONNX Runtime's `.wasm` (3.1 MB brotli; 5.5 MB + with WebGPU, per the install tests) is never part of your build. The script-tag file is about 129 KB gzip. +- **Model:** with onnxruntime-web 1.30 on single-thread WASM, a forward pass takes an estimated 182 / 330 / 599 ms at + 500 / 780 / 1,170 tokens (fitted to runs in Node on the training VM, not in a browser), so about 180 ms per decision + at the single-thread situation budget (1,000 characters). Serving the page with + `Cross-Origin-Opener-Policy: same-origin` and `Cross-Origin-Embedder-Policy: require-corp` enables WASM threads + (about 3× faster with 4 threads in headless Chromium, measured with GenClass 0.1's 32M-parameter model; not + re-measured for model 0.2.0; model card). +- **Holds are bounded.** A held response or request waits at most the hold budget (by default 1.5 × the median of + the last 20 model latencies, clamped to 150 to 800 ms), then proceeds unchanged. Background decisions have a 5 s + deadline. ## Content-Security-Policy and self-hosting -By default the model files and ONNX Runtime's wasm come from `https://cdn.jsdelivr.net`, fetched by a worker. A -policy whose `connect-src` does not allow that origin blocks them: GenClass then keeps observing without a model and -prints **one** console warning naming the blocked origin and the fix (`status.blocked` has the details). Either allow -`https://cdn.jsdelivr.net` in `connect-src`, or self-host (no third-party origin): +By default the model files and ONNX Runtime's wasm come from `https://cdn.jsdelivr.net`, fetched by a worker. A policy +whose `connect-src` does not allow that origin blocks them: GenClass then keeps observing without a model and prints +**one** console warning naming the blocked origin and the fix. Either allow `https://cdn.jsdelivr.net` in +`connect-src`, or self-host (no third-party origin): ```bash npx @genclass/runtime@beta fetch-model public/genclass-model # model + ONNX Runtime files (into public/genclass-model/ort/) @@ -1316,137 +904,42 @@ npx @genclass/runtime@beta init --model-url /genclass-model/ # or pass the opt GenClass.init({ model: { baseUrl: "/genclass-model/", ortWasmPaths: "/genclass-model/ort/" } }); ``` -- `fetch-model` downloads every model variant and both ONNX Runtime builds (about 64 MB). `--variant q8 --ort wasm` - is about 24 MB; then also set `model.device: "wasm"` (`init --model-url` does it when it finds only the WASM - build there). The WebGPU build's wasm is 27 MB, over the 25 MiB per-file limit of some static hosts (Cloudflare). -- The policy also needs `'wasm-unsafe-eval'` in `script-src` (WebAssembly) and `worker-src 'self'` (or `blob:` - for the script tag). When the worker cannot start (a policy without it, `file:`), the default - `model.inlineFallback: true` runs the model on the main thread instead (the `Model ready` line then says - `inline on the main thread`; each decision then blocks the page for about 180 ms on single-thread WASM, see - Model above). `model: { inlineFallback: false }` - skips the model in that case (`status.state` `skipped`, reason `worker-unavailable`, one warning naming the - worker error) and the runtime observes without it. +- `fetch-model` downloads every model variant and both ONNX Runtime builds (about 64 MB); `--variant q8 --ort wasm` is + about 24 MB (then set `model.device: "wasm"`; `init --model-url` does this when it finds only the WASM build). The + WebGPU build's wasm is 27 MB, over the 25 MiB per-file limit of some static hosts such as Cloudflare: use + `--ort wasm` there. `info ` verifies a downloaded directory's files. +- The policy also needs `'wasm-unsafe-eval'` in `script-src` and `worker-src 'self'` (or `blob:` for the script tag). + When the worker cannot start, the model runs on the main thread (about 180 ms of blocking per decision on + single-thread WASM) unless you pass `model: { inlineFallback: false }`. - Anonymous diagnostics (on by default on public hosts) go to `https://genclass.dev/api/v1/events`: add - `https://genclass.dev` to `connect-src`, or pass `telemetry: false`. A blocked page stops trying after 3 failed - requests, so the policy logs at most 3 violations. -- Sinks ([See it where you already look](#see-it-where-you-already-look), `0.3.0`): a webhook on another origin - needs that origin in `connect-src` (a same-origin path such as `webhook:/genclass` needs nothing). With the script - tag, `data-sink` (or `sink=` / `GENCLASS_CONFIG.sinks`) loads `dist/sinks/.js` and the chunk the adapters - share from the tag's own CDN path, so allow `https://cdn.jsdelivr.net` in `script-src`, or self-host this package's - `dist/` and point `data-base` at it. With `/auto` the adapters are chunks of your own bundle. + `https://genclass.dev` to `connect-src`, or pass `telemetry: false`. A blocked page stops after 3 failed requests. +- A webhook sink on another origin needs that origin in `connect-src`. -## Privacy and telemetry +CLI: `init`, `remove`, `login`, `connect`, `whoami`, `logout`, `dashboard`, `scan ` (alias `audit`, from +`0.3.0-beta.2`: [Audit a page](#audit-a-page)), `fetch-model ` (self-hosting) and `info `; +`npx @genclass/runtime@beta --help` lists them, and `init --help` and `login --help` give every flag (Next.js-only +`--strategy`: INSTALL.md). -- **Decisions are local.** Situations are built and decided in the browser by the local model; nothing is sent anywhere to make a decision. -- **Anonymous diagnostics are on by default** with `GenClass.init()` in a browser, except on local and private hosts (`localhost`, `*.localhost`, `*.local`, `*.test`, `127.0.0.1`, `::1`, private IP ranges) (`0.2.1` sends from local hosts too, by mistake; fixed in `0.2.2`, see the CHANGELOG), in Node/SSR, and in `createRuntime()`. They go to the GenClass maintainers' collector ([`telemetry-worker/`](https://github.com/genclass-dev/GenClass-lib/tree/main/telemetry-worker), a Cloudflare Worker writing to a private R2 bucket) and are used to improve the model. -- **By default no text from your page is sent.** Sent: a random per-page session id, versions, mode, device class, the app's hostname and id-normalised path (from `0.3.0` a segment with an `@`, such as an e-mail address, is `:id` too), and per decision (every detection and every decision whose diagnosis is not `expected`, every action and veto; of the routine ones, one in four and at most 200 per page load; counts are exact; `0.2.2` and earlier send every decision) the trigger, the diagnosis and its confidence, the action and gate, the outcome and latency. The collector stores no IP address or user agent. One console notice per page says it is on. -- **The measured visual checks send nothing** (new in `0.3.0`): their findings stay in the page (the console, `rt.checks()`, your own sinks). -- **Interaction analytics are off unless you turn them on** (new in `0.3.0-beta.1`): `GenClass.init({ token, analytics: true })` also sends, per page view, the route, engaged time, scroll depth and clicks, rage clicks and dead clicks per element, and within one page load how each view was reached, the route before it with the time between, the first element used on it, and where the page load ended with its routes in order (at most 10), for your dashboard. Elements are named from markup (`data-genclass-name`, `data-testid`, an id, a link's target, class names, a short path), never their text; the one thing stored on the device is the visit flag in `localStorage` (it links the page loads of a visit and says whether the browser visited before; `runtime.analytics.stop()` removes it), and no visitor id exists. They need the token and telemetry on, every opt-out above turns them off, and they never feed a decision or the model. Where analytics need consent (the EU, EEA and UK), leave them off and call `runtime.analytics.start()` after consent. Details: [TELEMETRY.md](https://github.com/genclass-dev/GenClass-lib/blob/main/packages/runtime/TELEMETRY.md#interaction-analytics). -- **The situation text the model read is sent only if you opt in** with `GenClass.init({ telemetry: { include: { situation: true } } })`. It is redacted by field meaning, but can still contain route names with query strings, short error messages and short field values; read [TELEMETRY.md](https://github.com/genclass-dev/GenClass-lib/blob/main/packages/runtime/TELEMETRY.md) before enabling it. -- **Opt out:** `GenClass.init({ telemetry: false })`, `?genclass=no-telemetry`, `localStorage.setItem("genclass.telemetry", "off")`; browsers sending Global Privacy Control are never collected. -- **With a token** (`0.3.0`: `token` option, `data-token`, `token=` in the meta tag), each batch also carries - your app's public token, so the same diagnostics also show on your app's private dashboard (whoever holds its link - can open it), and decisions about `protect()`ed functions carry the function's name (`fn`, the label you chose). - Without a token neither is sent; without telemetry, a token does nothing. -- **Possible detections** (`0.3.0`, observe): their `detect` events carry `tier: "possible"` (a likely detection - has no `tier`, as before), and the page-load summary counts them apart from detections (`possibleDetections`). They - say which tier a detection was in; nothing new about the page is sent. -- **The CLI** contacts genclass.dev only for the command you run: `init` creates a dashboard only after you approve its - diff (it sends the `name` from `package.json`); `login` asks for a sign-in code with - `client_name: "genclass-runtime "` (shown on the approval page and in your account's session list) and - polls until you approve it; `whoami`, `connect` and (signed in, without a private link here) `dashboard` read your - account's apps; `connect` sends a new app's name, or an - existing dashboard's link from `.genclass.local` to add it to your account; `logout` ends the CLI session. The - account requests carry `user-agent: genclass-runtime-cli/`. What an account stores: [PRIVACY.md](https://github.com/genclass-dev/GenClass-lib/blob/main/PRIVACY.md). -- Apps that ship GenClass may need to disclose this to their users (GDPR/CCPA). Full schema, storage and guidance: [TELEMETRY.md](https://github.com/genclass-dev/GenClass-lib/blob/main/packages/runtime/TELEMETRY.md). -- Other telemetry options: `npx @genclass/runtime@beta init --no-telemetry` writes the opt-out into your project (`telemetry=off` in the meta tag or `data-telemetry="off"` on the script tag do the same); `telemetry: { endpoint, sample, decisionSample, maxDecisions, flushMs, maxBatch, include }` sends to your own collector, samples page loads or decision records, caps them per page load, or tunes batching; `runtime.telemetry` tells whether it is on and why not. Privacy policy: [PRIVACY.md](https://github.com/genclass-dev/GenClass-lib/blob/main/PRIVACY.md). -- **Inputs:** typed values of password fields, `cc-*` / `one-time-code` / password autocomplete fields, and fields - whose name or label names a secret are never recorded. -- **Redaction:** the default redactor (`redact` option) works by the leaf field's meaning, not by substring. - - Redacted: `auth.token`, `form.password`, `users.3.password`, `payment.card.number`, `settings.apiKey`, and - opaque credential-like strings under `auth` / `session` / `cookie`. - - Under a container named for a secret, strings, numbers and arrays are all redacted: `payment.cvv.value = 123`, - `login.otp.code`, `account.password.history = [...]`. A side effect: numbers under a container whose name is - a secret word in another sense are hidden too (`map.pin.lat`). - - Visible: `auth.loading`, `auth.user.name`, and a kanban `card`. - - Booleans and null are never redacted. - - Query parameters are redacted by the same rule. - - The "would replace text the user typed" fact shows a character diff only when the redactor leaves both values - unchanged; otherwise it prints `[redacted] → [redacted]` (or your redactor's replacement). - - Pass your own `redact(path, value)` for app-specific secrets or PII. The default has gaps (see below). - - The container rule for numbers and arrays and the typed-text rule are new in `0.1.0-beta.0` (missing from - `0.1.0-beta.1`, back in `0.1.0-beta.2`). +## Questions -## Known limitations +Short answers to the common questions (how to find race conditions in a React app, whether it changes what the app +does, how accurate it is, what it costs, what it sends) are in the +[FAQ](https://github.com/genclass-dev/GenClass-lib/blob/main/FAQ.md). Ask anything else, or report a bug or a false +detection, on [GitHub Issues](https://github.com/genclass-dev/GenClass-lib/issues) or +[Discord](https://discord.gg/kN9kRnFtZm). Security reports: +[SECURITY.md](https://github.com/genclass-dev/GenClass-lib/blob/main/SECURITY.md). -- **The model is new and imperfect.** The numbers are under [Model quality](#model-quality). - - In observe mode, expect some false flags: 1–4 per 100 decisions where nothing was wrong, more in some apps. - - It misses most problems. - - It was not evaluated on your app. - - Heal-mode actions on failures (`retry`, `serve_cached`) are its least precise actions. - - Watch observe mode before turning on `guard` or `heal`. -- **Redaction gaps (all modes with a model).** - - A container named by a two-word secret (`cardNumber`, `apiKey`, `creditCard`) counts as broad, like `auth`. - So `payment.cardNumber.value = "4111 1111 1111 1111"`, or the same number, is shown. It reaches the situation - text, `explain()`, the console evidence and devtools, and the telemetry situation text if you opted in. A leaf - with such a name - (`settings.apiKey`) is redacted. - - The fact "X changed since #N started and is back to V" compares the rendered text, so two different redacted - values read as "back to [redacted]". - - Pass a custom `redact`, and avoid keeping secrets in observed stores. -- **The discard mark lasts 10 s.** After a `discard`, writes by operations chained from the discarded one (a - `setTimeout`-driven poll, a saga) to the protected fields are also dropped for 10 s, even when they carry fresh - data. The marks also outlive `rt.pause()` and `setMode("observe")`. -- **`defer` can wait long.** A delivery may be deferred twice, each time until the related operations finish or - 10 s pass. A deferred WebSocket/EventSource message holds back the messages queued behind it. -- **Held messages after `close()`.** A WebSocket/EventSource message held for a decision is still dispatched if - the app closed the socket meanwhile. EventSource `open` events are not kept in order behind held messages. -- **XHR listeners of a held response** run after the original dispatch, so `e.currentTarget` is `null`. Use the - `xhr` object itself. -- **`retry` (heal) trusts HTTP semantics.** It is offered for idempotent methods, and for POST / PATCH only when - the request carries an idempotency key (`policy.idempotencyHeaders`, default `Idempotency-Key`, - `X-Idempotency-Key`). An endpoint that is not idempotent despite its method (a `PUT` that appends) can still be - retried. HTTP 502 is described as usually not processed, which is not always true behind proxies. -- **Transport coverage.** `coalesce`, `retry`, `hedge` and failure/stall `serve_cached` are fetch-only. XHR - failures and stalls are detection-only. -- **Synthetic events.** DOM events with `isTrusted === false` (`el.click()`, `dispatchEvent`, in-page test - drivers) are not user actions unless you pass `observe: { untrustedEvents: true }`. -- **`policy.holdWrites: true` (opt-in).** A held write that a later write flushed early can be recorded as dropped - while it stays applied. -- **Causality** across `await` is tracked by instrumenting fetch, XHR, timers, message events and Response bodies. - This is best effort; wrap important work in `rt.op(name, fn)` for exact attribution. -- **Store state.** With the one line, React, Redux and Zustand (`devtools`) state is discovered; discovered React - and Zustand state is observed only (detections and delivery `deliver` / `defer`, no write actions). Zustand's - `devtools` middleware is off in production builds unless you pass `enabled: true`, and then nothing is discovered - from it (measured in the [compatibility matrix](https://github.com/genclass-dev/GenClass-lib/blob/main/compat/RESULTS.md)). Other state - (Vue / Pinia, MobX, Jotai, signals, module variables) is seen only through its effects unless you register it. - Discovered React state has no variable names (`Comp.state0`), production builds minify component names (the store - is then named after the element the component renders), and a component's store appears on its first state change. - A response is checked against newer data only once GenClass has seen what that request writes, so the first - response of each kind after a page load is never a delivery decision. See - [Automatic state discovery](#automatic-state-discovery). -- **Install and CDN.** - - **Read the diff.** Always read the diff `init` and `remove` show before you confirm. - - **SRI covers only the script-tag file.** The model worker, ONNX Runtime glue and overlay it loads from the CDN, - and the model files, are not integrity-checked by the browser. The runtime checks the model files against the - sha256 values in `model.json`. Pin a version, or self-host. - - **Page configuration is read from `` tags in `` only** (since 0.2.2), so markup - injected into the body cannot configure GenClass. A meta tag may set the mode; the model and ONNX Runtime URLs it - names must be on the page's own origin (relative or same origin), otherwise they are ignored with one warning. - Point at a CDN of your own with the script tag's `data-model` / `data-ort`, `window.GENCLASS_CONFIG` (it - overrides meta tags) or `GenClass.init(options)`. A page that lets untrusted HTML into `` should not use - meta configuration at all. - - **React Router dev server.** On the very first dev start after `init`, Vite discovers the new imports late, - re-optimizes and reloads the page, logging a few "Outdated Optimize Dep" errors once. Later loads are clean. - - **Scaffolds not re-run.** The framework scaffold runs predate this version's `init` / `remove` changes, which - are covered by unit tests; the compatibility matrix is the current end-to-end evidence. -- **The model can be wrong.** It is trained on simulated apps and real apps driven in a headless browser. It is not - a substitute for tests. That is why the default only observes, guard acts only above thresholds fitted to keep - false interventions at or below 0.1% on data the model never trained on, and every action is logged. +## Documentation + +- [Install](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/INSTALL.md): every `init` flag, tested setups +- [Dashboard and CLI](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/DASHBOARD.md): accounts, `login`, `.genclass.local` +- [Configuration](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/OPTIONS-SPEC.md) and [API](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/API.md): every option and export +- [Sinks](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/SINKS.md) · [Visual checks](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/VISUAL.md) · [Threat model](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/THREAT-MODEL.md) +- [What it intercepts](https://github.com/genclass-dev/GenClass-lib/blob/main/packages/runtime/INTERCEPTION.md) · [Telemetry](https://github.com/genclass-dev/GenClass-lib/blob/main/packages/runtime/TELEMETRY.md) +- [Measured results](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/RESULTS.md) · [Compatibility matrix](https://github.com/genclass-dev/GenClass-lib/blob/main/compat/RESULTS.md) · [Model card](https://github.com/genclass-dev/GenClass-lib/blob/main/packages/runtime-model/MODEL_CARD.md) +- [Changelog](https://github.com/genclass-dev/GenClass-lib/blob/main/packages/runtime/CHANGELOG.md) · [All documents and the glossary](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/README.md#glossary) ## License Apache-2.0. The model package, `@genclass/runtime-model` (Apache-2.0), comes with its own -[model card](https://www.npmjs.com/package/@genclass/runtime-model). +[model card](https://github.com/genclass-dev/GenClass-lib/blob/main/packages/runtime-model/MODEL_CARD.md). diff --git a/packages/runtime/TELEMETRY.md b/packages/runtime/TELEMETRY.md index 392d9843..ed8763b9 100644 --- a/packages/runtime/TELEMETRY.md +++ b/packages/runtime/TELEMETRY.md @@ -1,11 +1,22 @@ # GenClass Runtime telemetry +GenClass Runtime (`@genclass/runtime`), an open-source browser library that reports race conditions in web apps, +sends anonymous diagnostics about its own decisions to its maintainers, by default on public websites; this page lists +every event and field it sends, where they are stored and for how long, and every way to turn it off. + +> **Version.** Written for `@genclass/runtime` 0.3.0-beta.4 (2026-10-27), the version on GitHub `main`, in +> `package.json` and in the `runtime` field of every batch. The copy in an npm package describes that package's version. +> As of 2026-10-27, `npm i @genclass/runtime@beta` installs 0.3.0-beta.4 (the newest line; 0.3.0-beta.1 to beta.4 were +> released 2026-10-22 to 2026-10-27) and `npm i @genclass/runtime` installs the stable line, 0.2.2 (`latest`). Items +> marked 0.3.0 are not in 0.2.x; interaction analytics are marked 0.3.0-beta.1, visits and navigation paths +> 0.3.0-beta.2. + Since **`0.1.0-beta.3`**, `GenClass.init()` in a browser sends **anonymous diagnostics** about GenClass's own decisions to the GenClass maintainers (Daybot Solutions Inc.). We use them to measure and improve the GenClass -model (accuracy, false interventions, latency, which situations it sees in real apps). This page lists exactly what -is sent, where it goes, and how to turn it off. +model (accuracy, false interventions, latency, which situations it sees in real apps). -**Turn it off:** `GenClass.init({ telemetry: false })`, `?genclass=no-telemetry` in the URL, or +**Turn it off:** `GenClass.init({ telemetry: false })` (or `npx @genclass/runtime@beta init --no-telemetry` while +0.3.0 is in beta; on 0.2.x, the same without `@beta`), `?genclass=no-telemetry` in the URL, or `localStorage.setItem("genclass.telemetry", "off")`. Browsers that send Global Privacy Control are never collected. Details under [Opting out](#opting-out). @@ -14,13 +25,13 @@ Details under [Opting out](#opting-out). | how GenClass starts | default | |---|---| | `GenClass.init()`, `import "@genclass/runtime/auto"` (and `/auto/*`), the CDN script tag, `npx @genclass/runtime@beta init` setups | **on** in a browser on a public host | -| the same, on a local or private host (`localhost`, `*.localhost`, `*.local`, `*.test`, `*.internal`, `127.0.0.1`, `::1`, `10/8`, `172.16/12`, `192.168/16`, link-local, `file:`) | off unless `telemetry: true` / `{ ... }` (`0.2.1`: on, by mistake; fixed in `0.2.2`, see the CHANGELOG) | +| the same, on a local or private host (`localhost`, `*.localhost`, `*.local`, `*.test`, `*.internal`, `127.0.0.1`, `::1`, `10/8`, `172.16/12`, `192.168/16`, link-local, `file:`) | off unless `telemetry: true` / `{ ... }` (`0.1.0-beta.3` to `0.2.0`: on, with no local-host rule yet; `0.2.1`: on, by mistake; fixed in `0.2.2`, see the CHANGELOG) | | `GenClass.init()` outside a browser (Node, SSR, workers) | off unless `telemetry: true` / `{ ... }` | | `createRuntime()` (headless, tests, the simulator) | off unless `telemetry: true` / `{ ... }` | In `0.2.1` the local-host row did not apply to `GenClass.init()` (and so to `/auto` and the script tag): it passed an explicit `telemetry: true` internally, so local and private hosts sent too. Fixed in `0.2.2` -([CHANGELOG](CHANGELOG.md)). +([CHANGELOG](https://github.com/genclass-dev/GenClass-lib/blob/main/packages/runtime/CHANGELOG.md)). When it is on, the console shows one notice per page: @@ -36,7 +47,8 @@ When it is on, the console shows one notice per page: (The coding-agent link, `connectAgent` from `@genclass/runtime/agent`, is not telemetry: it runs in development builds on development hosts and sends its recording only to a bridge on the same machine, `http://127.0.0.1:47357`. It changes -nothing below. [docs/runtime/AGENT.md](../../docs/runtime/AGENT.md) section 6.) +nothing below. [docs/runtime/AGENT.md](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/AGENT.md) +section 6.) Only data the runtime already computes for its own decisions. **By default, no text from the page is sent:** decision events carry the trigger kind, the diagnosis and its confidence, the chosen action, the gate, the outcome, @@ -46,7 +58,7 @@ segment holding an `@`, such as an e-mail address, is `:id` too). - **The situation text the model read is sent only with `include: { situation: true }`** (default off since `0.2.1`). It is the text exactly as built for the model, after the redactor (`InitOptions.redact`, default: secret-named fields such as passwords, tokens, card numbers, API keys are replaced by `[redacted]`; see the - README's [Privacy](README.md#privacy-and-telemetry) section for the rules and their known gaps). It describes + README's [Privacy](https://github.com/genclass-dev/GenClass-lib/blob/main/packages/runtime/README.md#privacy-and-telemetry) section for the rules and their known gaps). It describes operations (e.g. `GET /api/items/:id`), store fields and short value summaries, timing and the facts GenClass computed. **Before enabling it, know what it can contain:** request names with their query strings (parameter values are redacted only when the parameter name looks secret), route changes with query string and fragment, @@ -55,15 +67,15 @@ segment holding an `@`, such as an e-mail address, is `:id` too). (up to 60 characters) and store leaf values up to 48 characters, including state the runtime discovered on its own (`autoState`: React, Redux or Zustand state the app never registered; those decisions carry `autoState: true`). Add a `redact` function for fields with ordinary names and personal values. -- **Interaction analytics** (new in `0.3.0-beta.1`; [design](../../docs/runtime/ANALYTICS.md)): **off unless the app +- **Interaction analytics** (new in `0.3.0-beta.1`; [design](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/ANALYTICS.md)): **off unless the app turns them on** with `analytics: true`, and only with the app's token (or your own `telemetry.endpoint`). Then the page also sends `interactions` events: per page view, the route, engaged time, scroll depth milestones and clicks, rage clicks and dead clicks per element, the element named from markup (a `data-genclass-name`, a `data-testid`, an - id, a link's target, stable class names or a short structural path), never its text; and (Unreleased) per visit a - random visit id kept 30 minutes, where the visit came from (the referring host, utm values), the device class, + id, a link's target, stable class names or a short structural path), never its text; and (`0.3.0-beta.2`) per visit + a random visit id kept 30 minutes, where the visit came from (the referring host, utm values), the device class, browser and OS families and whether the browser visited before (a flag in `localStorage`). See [Interaction analytics](#interaction-analytics). -- **Measured visual checks** ([VISUAL.md](../../docs/runtime/VISUAL.md); new in `0.3.0`): nothing about +- **Measured visual checks** ([VISUAL.md](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/VISUAL.md); new in `0.3.0`): nothing about them is sent. Their findings stay in the page (the console, `rt.checks()`, your own sinks). Counts per check kind are a proposal that needs the maintainers' approval of the exact fields first; this file will list them if that happens. @@ -83,7 +95,7 @@ segment holding an `@`, such as an e-mail address, is `:id` too). "token": "gc_…", // only when the app configured a valid token (InitOptions.token); see "App tokens" "sid": "3f9c…", // 24 hex chars from crypto.getRandomValues, new on every page load, never stored "sent": 12034, // ms since this page's GenClass started (no wall-clock time is sent) - "runtime": "0.3.0-beta.0", // @genclass/runtime version + "runtime": "0.3.0-beta.4", // @genclass/runtime version "model": "2.0.0-rc4t", // model card version, or null without a model "events": [ /* ≤ 100 events, ≤ 60 KB per request */ ] } @@ -93,22 +105,195 @@ Every event has `t` (type), `seq` (0, 1, 2, … per page) and `at` (ms since sta | `t` | when | fields | |---|---|---| -| `session` | once, at init | `runtime`, `host` (the page's hostname, which identifies the app using GenClass), `route` (the page path with id-like segments replaced by `:id`, and from 0.3.0 any segment holding an `@` or `%40`, such as an e-mail address; no query or fragment), `mode`, `effectiveMode`, `aggressiveness`, `sampled`, `model` (`local` / `custom` / `off`), `modelState`, `modelVersion`, `triage`, `shadow`, `holdWrites`, `device` (`webgpu` available, `cores`, `memoryGB` (the browser's coarse deviceMemory), `crossOriginIsolated`, `effectiveType`, `saveData`), `sample`, `situation` (whether situation text is included). Not the page's `decisionSample` or `maxDecisions` setting: `summary.telemetry` says how many records were sent and withheld ([Volume](#volume)) | -| `model` | model state changes, and once right after `session` when the model was skipped or failed before telemetry started (the device floor: `state` `skipped`, `reason` `device-memory` / `save-data` / `loadIf`; with `model.inlineFallback: false` and no worker, `reason` `worker-unavailable`) | `state`, `version`, `model`, `variant`, `device` (webgpu/wasm), `threads`, `worker`, `workerError` (why the model worker was not used, URLs replaced by ``), `gpu` (WebGPU probe summary), `ort`, `loadMs`, `warmupMs`, `fromCache`, `bytes`, `phase`, `reason`, `error` (GenClass model-host error text, ≤ 200 chars, URLs replaced by ``), `attempts` (variant, device and error per load attempt, URLs replaced by ``) | +| `session` | once, at init | see [`session` event](#session-event) | +| `model` | model state changes, and (0.3.0) once right after `session` when the model was skipped or failed before telemetry started | see [`model` event](#model-event) | | `status` | mode, aggressiveness or breaker changes | `mode`, `effectiveMode`, `aggressiveness`, `breaker` | -| `decision` | a model decision: every one whose top diagnosis is not `expected` (detections and near-misses), that acted or was vetoed, always; a `decisionSample` share (default 1 in 4) of the routine ones (top diagnosis `expected`, nothing acted), at most `maxDecisions` (default 200) routine ones per page load ([Volume](#volume)); all are counted in `summary` | `id`, `trigger`, `route` (the page path, normalised; for a decision deferred until the model loaded or the tab was visible again, 0.3.0, the path at its trigger), `model`, `latencyMs`, `held` (the subject waited for the answer; false = decided in the background), `situation` (text, see above), `autoState` (true when discovered state was recorded on the page), `budget`, `compact`, `questions` (question ids), `answers` (per question: calibrated `probabilities` per label, `choice`, `confidence`; or `p` / `score`), `diagnosis`, `diagnosisConfidence`, `action` (chosen), `confidence`, `tier`, `candidate`, `ran`, `executed`, `acted`, `reason` (why the passive action ran), `mass`, `gateKind`, `threshold` / `gain` / `margin`, `thresholdSource`, `gates` (report/guard/heal thresholds, aggressiveness, level and sources), `effectiveMode`, `shadow`, `fn` (only on batches with a token: the name the app gave the `protect()`ed function the decision's subject ran inside, the outermost one; omitted otherwise) | -| `detect` | a decision reported as a detection: every one, never sampled or capped, with its `decision` record; since 0.3.0 also observe's "possible" ones (`InitOptions.detect`: below the report gate, at or above the eager profile's report value) | `decision`, `trigger`, `diagnosis`, `p`, `tier` (0.3.0: `"possible"` on a possible detection; absent on a likely one, as from earlier runtimes. It cannot be derived from the decision record: a possible detection is one whose `p` is below `gates.report` and at least the eager profile's report value (`gates().possible`, 0.70 with model 0.2.0), which the record does not carry, so `tier` says which side of that threshold a near-miss fell on. A decision deferred until the model loaded or the tab was visible again is `"possible"` whatever its `p`, until the false-report rate of those late decisions is measured. Nothing about the page) | +| `decision` | a model decision: always when its top diagnosis is not `expected`, it acted, it would have acted in shadow mode or it was vetoed; a sample of the routine ones (0.3.0; [Volume](#volume)) | see [`decision` event](#decision-event) | +| `detect` | a decision reported as a detection (from 0.3.0 also a "possible" one): every one, never sampled or capped | see [`detect` event](#detect-event) | | `action` | an action ran, failed or was undone | `id`, `decision`, `action`, `tier`, `trigger`, `outcome` (`applied` / `failed` / `undone`), `late` (a late revert), `reversible`, `droppedFields` (count) | | `veto` | `onBeforeAction` vetoed an action (every one) | `decision`, `action`, `enforced` | | `breaker` | the breaker tripped or reset | `tripped`, `reason`, `undos`, `errors`, `decisions` (ids) | | `model-error` | model errors (first 20 per page; all are counted) | `code`, `message` (≤ 200 chars, URLs replaced by ``: a model host's message can quote the browser's worker error, which names the script and the page origin) | -| `summary` | about once a minute (≥ 60 s apart) while decisions continue, never capped; when the tab is hidden, on pagehide and on destroy | `reason`, `counts` (decisions, acted, detections (likely ones), `possibleDetections` (0.3.0: the possible ones, counted apart), actions applied/failed, late reverts, undos, vetoes, breaker trips, per trigger / diagnosis / action ran, detections per diagnosis, action limits hit, model errors per code, fail-opens per reason and trigger), `runtime` (model p50/p95 latency, decisions, dropped, hidden-tab skips, held time, denied and shadow counts, error count), `telemetry` (queued, dropped, send failures, events sent, `decisionRecords` (decisions whose records were sent), `decisionsSampledOut`, `decisionsCapped`) | -| `interactions` | only with `analytics: true` and a token (or a custom endpoint): when a page view ends (a route change, page exit), when the tab is hidden, and before a `summary`, if anything changed ([Interaction analytics](#interaction-analytics)) | `view` (page view number in this page load), `route` (normalised as on `session`), `start` (the view's first event: a page view), `engagedMs`, `scroll` (milestones 25 / 50 / 75 / 100 first reached since the previous event), `clicks`, `rage`, `dead` (counts per element key, at most 50 / 20 / 20 keys, the rest under `(other)`), `end` (the view ended). Counts are deltas since the view's previous event. Unreleased, on a view's first event only: `visit` (16 hex characters, random, see [Visits](#visits)); and when that view opened the visit, `entry: true`, `ref` (the referring site's host, `www.` dropped, when it is another site; `(other)` when the host looks like data), `utm` (`source`, `medium`, `campaign`: lower case, letters, digits, `. _ + -` and spaces, words that look like ids dropped, at most 40 characters each), `device` (`desktop` / `mobile` / `tablet`), `browser` (`Chrome`, `Safari`, `Firefox`, `Edge`, `Opera`, `Samsung Internet`, `Other`), `os` (`Windows`, `macOS`, `iOS`, `Android`, `Linux`, `ChromeOS`, `Other`), `returning` (whether this browser had a visit before; absent when storage is blocked). Navigation paths (Unreleased): on the view's first event `nav` (how it was reached: `load`, `reload`, `back_forward`, `push`, `back`, `forward`, `pop`) and, from a page load's second view on, `from` (the previous view's route) and `fromMs`; once per view `first` (the element key of the first click, or of the first focus on a form field after a key press) and `firstMs`; on the page load's last event (pagehide) `exit: true` and `path` (its routes in order, at most 10, then `"(more)"`) | +| `summary` | about once a minute (≥ 60 s apart) while decisions continue, never capped; when the tab is hidden, on pagehide and on destroy | see [`summary` event](#summary-event) | +| `interactions` | only with `analytics: true` and a token (or a custom endpoint; 0.3.0-beta.1): when a page view ends, when the tab is hidden and before a `summary`, if anything changed | see [`interactions` event](#interactions-event) | -Not sent: `expectation` reports (a learned step that did not happen, `docs/runtime/EXPECTATIONS.md`) and anything +Not sent: `expectation` reports (a learned step that did not happen, +[EXPECTATIONS.md](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/EXPECTATIONS.md)) and anything else learned about the page (flows, profiles, relations); those stay in the browser (`localStorage`, `rt.learned()`). -Numbers are rounded to 4 decimals. The source of truth is `packages/runtime/src/telemetry/client.ts`. +Numbers are rounded to 4 decimals. The source of truth is +[`packages/runtime/src/telemetry/client.ts`](https://github.com/genclass-dev/GenClass-lib/blob/main/packages/runtime/src/telemetry/client.ts). + +#### `session` event + +**When:** once, at init. + +**Fields:** + +- `runtime` +- `host` (the page's hostname, which identifies the app using GenClass) +- `route` (the page path with id-like segments replaced by `:id`, and from 0.3.0 any segment holding an `@` or `%40`, + such as an e-mail address; no query or fragment) +- `mode` +- `effectiveMode` +- `aggressiveness` +- `sampled` +- `model` (`local` / `custom` / `off`) +- `modelState` +- `modelVersion` +- `triage` +- `shadow` +- `holdWrites` +- `device` (`webgpu` available, `cores`, `memoryGB` (the browser's coarse deviceMemory), `crossOriginIsolated`, + `effectiveType`, `saveData`) +- `sample` +- `situation` (whether situation text is included) + +Not the page's `decisionSample` or `maxDecisions` setting: `summary.telemetry` says how many records were sent and +withheld ([Volume](#volume)). + +#### `model` event + +**When:** model state changes, and (0.3.0) once right after `session` when the model was skipped or failed before +telemetry started (the device floor, 0.3.0: `state` `skipped`, `reason` `device-memory` / `save-data` / `loadIf`; +with `model.inlineFallback: false` and no worker, also 0.3.0, `reason` `worker-unavailable`). On 0.2.x the model is +skipped only when the app sets `model.loadIf` to skip, and a skip before telemetry started shows only as +`modelState` on the `session` event, with no `model` event. + +**Fields:** + +- `state` +- `version` +- `model` +- `variant` +- `device` (webgpu/wasm) +- `threads` +- `worker` +- `workerError` (why the model worker was not used, URLs replaced by ``) +- `gpu` (WebGPU probe summary) +- `ort` +- `loadMs` +- `warmupMs` +- `fromCache` +- `bytes` +- `phase` +- `reason` +- `error` (GenClass model-host error text, ≤ 200 chars, URLs replaced by ``) +- `attempts` (variant, device and error per load attempt, URLs replaced by ``) + +#### `decision` event + +**When:** a model decision: every one whose top diagnosis is not `expected` (detections and near-misses), that acted, +that would have acted in shadow mode, or that was vetoed, always; a `decisionSample` share (default 1 in 4) of the +routine ones (top diagnosis `expected`, nothing acted), at most `maxDecisions` (default 200) routine ones per page +load (0.3.0; 0.2.x sends every decision record; [Volume](#volume)); all are counted in `summary`. + +**Fields:** + +- `id` +- `trigger` +- `route` (the page path, normalised; for a decision deferred until the model loaded or the tab was visible again, + 0.3.0, the path at its trigger) +- `model` +- `latencyMs` +- `held` (the subject waited for the answer; false = decided in the background) +- `situation` (text, see above) +- `autoState` (true when discovered state was recorded on the page) +- `budget` +- `compact` +- `questions` (question ids) +- `answers` (per question: calibrated `probabilities` per label, `choice`, `confidence`; or `p` / `score`) +- `diagnosis` +- `diagnosisConfidence` +- `action` (chosen) +- `confidence` +- `tier` +- `candidate` +- `ran` +- `executed` +- `acted` +- `reason` (why the passive action ran) +- `mass` +- `gateKind` +- `threshold` / `gain` / `margin` +- `thresholdSource` +- `gates` (report/guard/heal thresholds, aggressiveness, level and sources) +- `effectiveMode` +- `shadow` +- `fn` (only on batches with a token: the name the app gave the `protect()`ed function the decision's subject ran + inside, the outermost one; omitted otherwise) + +#### `detect` event + +**When:** a decision reported as a detection: every one, never sampled or capped, with its `decision` record; since +0.3.0 also observe's "possible" ones (`InitOptions.detect`: below the report gate, at or above the eager profile's +report value). + +**Fields:** + +- `decision` +- `trigger` +- `diagnosis` +- `p` +- `tier` (0.3.0: `"possible"` on a possible detection; absent on a likely one, as from earlier runtimes. It cannot be + derived from the decision record: a possible detection is one whose `p` is below `gates.report` and at least the eager + profile's report value (`gates().possible`, 0.70 with model 0.2.0), which the record does not carry, so `tier` says + which side of that threshold a near-miss fell on. A decision deferred until the model loaded or the tab was visible + again is `"possible"` whatever its `p`, until the false-report rate of those late decisions is measured. Nothing about + the page) + +#### `summary` event + +**When:** about once a minute (≥ 60 s apart) while decisions continue, never capped; when the tab is hidden, on pagehide +and on destroy. + +**Fields:** + +- `reason` +- `counts` (decisions, acted, detections (likely ones), `possibleDetections` (0.3.0: the possible ones, counted apart), + actions applied/failed, late reverts, undos, vetoes, breaker trips, per trigger / diagnosis / action ran, detections + per diagnosis, action limits hit, model errors per code, fail-opens per reason and trigger) +- `runtime` (model state, p50/p95 latency, decisions, dropped, hidden-tab skips; held time total and max; denied and + shadow counts; error count) +- `telemetry` (queued, dropped, send failures, events sent; and, 0.3.0, `decisionRecords` (decisions whose records + were sent), `decisionsSampledOut` and `decisionsCapped` (routine ones sampled out and capped)) + +#### `interactions` event + +**When:** only with `analytics: true` and a token (or a custom endpoint; 0.3.0-beta.1): when a page view ends (a route +change, page exit), when the tab is hidden, and before a `summary`, if anything changed ([Interaction +analytics](#interaction-analytics)). + +**Fields:** + +- `view` (page view number in this page load) +- `route` (normalised as on `session`) +- `start` (the view's first event: a page view) +- `engagedMs` +- `scroll` (milestones 25 / 50 / 75 / 100 first reached since the previous event) +- `clicks`, `rage`, `dead` (counts per element key, at most 50 / 20 / 20 keys, the rest under `(other)`) +- `end` (the view ended) + +Counts are deltas since the view's previous event. + +**Visits** (0.3.0-beta.2; [Visits](#visits)), on a view's first event only: + +- `visit` (16 hex characters, random) +- and when that view opened the visit: `entry: true`; `ref` (the referring site's host, `www.` dropped, when it is + another site; `(other)` when the host looks like data); `utm` (`source`, `medium`, `campaign`: lower case, letters, + digits, `. _ + -` and spaces, words that look like ids dropped, at most 40 characters each); `device` (`desktop` / + `mobile` / `tablet`); `browser` (`Chrome`, `Safari`, `Firefox`, `Edge`, `Opera`, `Samsung Internet`, `Other`); `os` + (`Windows`, `macOS`, `iOS`, `Android`, `Linux`, `ChromeOS`, `Other`); `returning` (whether this browser had a visit + before; absent when storage is blocked) + +**Navigation paths** (0.3.0-beta.2; +[design](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/ANALYTICS.md#11-navigation-paths-2026-10-25)): + +- on the view's first event, `nav` (how it was reached: `load`, `reload`, `back_forward`, `push`, `back`, `forward`, + `pop`) and, from a page load's second view on, `from` (the previous view's route) and `fromMs` +- once per view, `first` (the element key of the first click, or of the first focus on a form field after a key press) + and `firstMs` +- on the page load's last event (pagehide), `exit: true` and `path` (its routes in order, at most 10, then `"(more)"`) ### Volume @@ -204,7 +389,8 @@ What is recorded, per page view (a page load, and each route change to a new pat on your site, the other site's origin only), the tag with up to two stable class names, or an `nth-of-type` path of up to 4 levels under a named ancestor or a landmark. At most 120 characters. Any attribute value that looks like data (an e-mail address, a URL, 4 or more digits, 12 or more hex digits) is not used; -- **navigation within the page load** (added after `0.3.0-beta.1`, [design](../../docs/runtime/ANALYTICS.md#11-navigation-paths-2026-10-25)): +- **navigation within the page load** (`0.3.0-beta.2`, + [design](https://github.com/genclass-dev/GenClass-lib/blob/main/docs/runtime/ANALYTICS.md#11-navigation-paths-2026-10-25)): how each view was reached (a page load, a reload, the browser's back or forward button, an in-app route change, an in-app back or forward), the previous view's route and the time from its start to this view's, the element key of the view's first interaction (a click, or a focus on a form field right after a key press) and the time to it, and @@ -214,7 +400,7 @@ What is recorded, per page view (a page load, and each route change to a new pat #### Visits -Unreleased (the next `@beta` after 0.3.0-beta.1). A **visit** is a run of page views with no gap over 30 minutes: +New in `0.3.0-beta.2` (2026-10-25). A **visit** is a run of page views with no gap over 30 minutes: - **The visit flag, in `localStorage`.** The runtime keeps one entry, `genclass.analytics.` (or `genclass.analytics.endpoint` with your own endpoint and no token), holding `{ "v": , "t":