Measured visual checks: counts in telemetry, their own kind on the dashboard (0.3.0-beta.5)

PRIVACY.md

@@ -30,6 +30,11 @@ object; version 0.2.1 sent from those hosts by default, by mistake, fixed in 0.2
   their users.
 - **Technical details:** GenClass and model versions, settings, device type (WebGPU or WASM), load times and error
   counts, and the website's hostname.
+- **Counts of the measured visual checks, from runtime 0.3.0-beta.5, only where they run** (off on public sites
+  unless the website's developer turned them on): whether the checks run on the page and whether their fixes are on,
+  how many problems each check found (such as text in its background's colour, or a page that stopped scrolling), the
+  same per page path, and how many fixes GenClass applied on the page and undid. Never which element, its text, its
+  colours or any address.
 - **The page's path**, with id-like segments (and, from runtime 0.3.0, segments containing an `@`, such as an e-mail
   address) replaced by `:id` and no query string or fragment (for example `/orders/:id`), on each page load and each
   decision. Other words in the path, such as a username, are sent as they are.
@@ -73,7 +78,7 @@ works on their own site. We do not sell the data, use it for advertising, or try
 ## Dashboards for website owners
 
 A website owner can get a private dashboard link for their site's token. Whoever holds that link can see aggregated diagnostics for that site (counts of decisions, detections, possible detections and actions, the site's hostnames,
-routes, the names the developer gave to protected functions, model speed, versions; with interaction analytics on,
+routes, the names the developer gave to protected functions, model speed, versions, the measured visual checks' counts per check and page and their fixes; with interaction analytics on,
 page views, the paths between pages, time on page, scroll depth and clicks per page and element, visits per day, pages per visit, where visits
 came from, devices, countries, new or returning, entry and exit pages and funnels) and a list of recent detections
 (time, route or function name, diagnosis, action). Dashboards do not show the situation text. Data from browsers that opted out or

README.md

@@ -207,7 +207,7 @@ export const submitOrder = protect("checkout submit", (cart) => fetch("/api/orde
 - **Decisions are local.** Situations are built and decided in the browser by the local model; nothing is sent anywhere to make a decision.
 - **Anonymous diagnostics are on by default** with `GenClass.init()` in a browser, except on local and private hosts (`localhost`, `*.localhost`, `*.local`, `*.test`, `127.0.0.1`, `::1`, private IP ranges) (`0.2.1` sends from local hosts too, by mistake; fixed in `0.2.2`, see the CHANGELOG), in Node/SSR, and in `createRuntime()`. They go to the GenClass maintainers' collector ([`telemetry-worker/`](telemetry-worker), a Cloudflare Worker writing to a private R2 bucket) and are used to improve the model.
 - **By default no text from your page is sent.** Sent: a random per-page session id, versions, mode, device class, the app's hostname and id-normalised path (from `0.3.0` a segment with an `@`, such as an e-mail address, is `:id` too), and per decision (every detection and every decision whose diagnosis is not `expected`, every action and veto; of the routine ones, one in four and at most 200 per page load; counts are exact; `0.2.2` and earlier send every decision) the trigger, the diagnosis and its confidence, the action and gate, the outcome and latency. The collector stores no IP address or user agent. One console notice per page says it is on.
-- **The measured visual checks send nothing** (new in `0.3.0`): their findings stay in the page (the console, `rt.checks()`, your own sinks).
+- **The measured visual checks send counts only** (from `0.3.0-beta.5`): whether they run on the page and whether their runtime fixes are on (`session`), and in each `summary` the findings per check, per check and page path (at most 20 paths), the fixes applied and undone, and their cost. Never a selector, element text, colour, URL or one event per finding; the findings themselves stay in the page (the console, `rt.checks()`, your own sinks). They are off on public hosts unless you turn them on.
 - **The situation text the model read is sent only if you opt in** with `GenClass.init({ telemetry: { include: { situation: true } } })`. It is redacted by field meaning, but can still contain route names with query strings, short error messages and short field values; read [TELEMETRY.md](packages/runtime/TELEMETRY.md) before enabling it.
 - **Opt out:** `GenClass.init({ telemetry: false })`, `?genclass=no-telemetry`, `localStorage.setItem("genclass.telemetry", "off")`; browsers sending Global Privacy Control are never collected.
 - **Interaction analytics are off unless the app turns them on** (`analytics: true` with the app's token, new in `0.3.0-beta.1`): per page view the route, engaged time, scroll depth, and clicks, rage clicks and dead clicks per element, the element named from markup, never its text; and, within one page load, how each page view was reached, the route before it and the time between, the first element used on it, and where the page load ended with its routes in order. No visitor id; one `localStorage` entry (the visit flag, which links the page loads of a visit and says whether the browser visited before; `runtime.analytics.stop()` removes it), every opt-out above applies, and they never feed a decision. Where analytics need consent (EU, EEA, UK), start them after consent with `runtime.analytics.start()`. See [TELEMETRY.md](packages/runtime/TELEMETRY.md#interaction-analytics).

docs/runtime/VISUAL.md

@@ -82,7 +82,8 @@ What follows from that:
   sampling hashes the session id; no `Math.random`.
 - **Rule 7.** No new dependency. axe-core is not added; the contrast pipeline below reimplements the parts of its
   approach that matter (stack at text rects, walk to the first opaque layer, abstain on anything complex).
-- **Rule 8.** No visual telemetry ships until the owner approves the exact fields (section 10).
+- **Rule 8.** No visual telemetry shipped until the owner approved the exact fields (section 10): counts only,
+  approved on 2026-10-27 and sent from `0.3.0-beta.5`.
 
 ## 2. Shared machinery
 
@@ -996,8 +997,9 @@ Why off on public hosts by default:
 1. **The visitor pays the CPU.** A layout pass costs main-thread time on every page view, and on a low-end phone that
    time is 4 to 6 times larger (estimate, to be measured). The visitor gets nothing back unless a developer reads the
    reports.
-2. **Nobody reads production consoles.** Until telemetry for checks is approved (section 10), production reports
-   reach only an app's own `report` function or sinks. An app that wants them sets `visual: true` and wires a sink.
+2. **Nobody reads production consoles.** Production findings reach only an app's own `report` function or sinks;
+   since `0.3.0-beta.5` their counts also reach the app's dashboard through telemetry (section 10). An app that wants
+   them sets `visual: true`.
 3. **Visitors' environments cause false reports.** Dark-mode extensions, page translation, zoom, forced colours and
    user font settings change what is measured. The abstain list covers the known ones; the bench (section 15) has to
    show the rest is rare before a production default is reconsidered.
@@ -1182,26 +1184,54 @@ receives every report.
 
 ## 10. Telemetry
 
-Nothing about visual checks is sent until the owner approves the exact fields. Rule 8 says widening what is sent
-needs the owner's OK, and the request of 2026-10-13 did not mention telemetry. Until then the telemetry client has no
-visual code at all (which also keeps its first-load bytes unchanged). (The first implementation added a `visual`
-field to the session and summary events; the review of 2026-10-13 removed both, section 19.)
-
-Proposed fields, counts only:
+**Approved by the owner on 2026-10-27; sent from `0.3.0-beta.5`.** Rule 8 says widening what is sent needs the
+owner's OK, and the request of 2026-10-13 did not mention telemetry, so until then nothing about the checks was sent.
+(The first implementation added a `visual` field to the session and summary events; the review of 2026-10-13 removed
+both, section 19.) The owner approved these fields, counts only, on 2026-10-27, after a production site full of stuck
+scroll locks and frozen clicks (each measured and fixed in its visitors' consoles) looked clean on its dashboard:
 
 | field | where | content |
 |---|---|---|
 | `counts.checks` | summary event | `{ [kind]: n }` |
-| `counts.checksByRoute` | summary event | `{ [normalisedRoute]: { [kind]: n } }`, at most 20 routes; the route is the same `normalizePath(pathname)` already sent with decisions |
+| `counts.checksByRoute` | summary event | `{ [normalisedRoute]: { [kind]: n } }`, at most 20 routes; the route is the same `normalizePath(pathname)` already sent with decisions (a segment holding an `@` as `:id`) |
 | `counts.fixes` | summary event | `{ applied, undone }` |
 | `runtime.visual` | summary event | `{ passes, ms, partial, tier }` |
 | `visual`, `fix` | session event | `"off" \| "local" \| "on"` and a boolean, so counts can be normalised by coverage |
 
-Never sent: selectors, element text, colours, URLs, sizes, per-finding events. Route keys would be cut as telemetry's
-`route()` cuts other routes. When approved: update `TELEMETRY.md`, both READMEs, the CHANGELOG and the console notice
-check. Coverage note: telemetry is off on local hosts and visual is off on public hosts by default, so counts would
-come from apps that set `visual: true` in production, from visitors who open a link with `?genclass-visual=on`, and
-from pages where the devtools overlay mounted or `rt.checkNow()` ran.
+Never sent: selectors, element text, colours, URLs, sizes, per-finding events, and the rest of
+`SessionSummary.visual` (`elements`, `maxSliceMs`, `abstained`, `unsupported`, `stopped`).
+
+As built:
+
+- **Session event** (`src/telemetry/client.ts`): `visual` is the first-load stub's setting (`rt.vis.setting`, `"off"`
+  without a stub) and `fix` is the stub's `fix`: `visual.on === true` and `visual.fix` is `true` or a non-empty list,
+  the condition under which the engine's `resolveCfg` allows any fix kind. Whether a fix can run also needs guard or
+  heal, which the same event's `effectiveMode` says. These two fields are the only first-load bytes.
+- **Summary event**: the chunk adds the counts (`src/visual/bridge.ts`, `withChecks`), so first load carries no code
+  for them. `follow()` replaces the telemetry client's `push` with one that adds them to every `summary` event before
+  it is queued; the client queues its own events through `this.push`, so the own property sees periodic, hidden,
+  pagehide and destroy summaries alike (teardowns run in order, so the destroy summary still reads the engine). A page
+  where the chunk never loaded (`"local"` on a public host) sends summaries exactly as before. The counts are
+  cumulative for the page load, like every other summary count; the collector counts them exactly through its
+  per-page-load high-water marks.
+- **Console notice**: when the chunk starts on a page with telemetry on, it prints once per page `[GenClass] The
+  diagnostics also carry the measured visual checks' counts: findings per check and page, runtime fixes applied and
+  undone, never selectors, page text, colours or URLs.` (`VISUAL_TELEMETRY_NOTICE`). The telemetry notice itself is
+  unchanged ("decisions and counts, no page text").
+- **Collector and dashboard** (`telemetry-worker/src/dashboard.ts`, `pages.ts`): counters `checks`, `check_kind`,
+  `check_issue` (kind TAB route), `check_fix` (applied, undone), `check_loads`, `check_tier`, `check_partial` and
+  `check_ms` from summaries, `visual_setting` and `visual_fix` from session events; no migration (the counters tables
+  hold any metric). The dashboard shows checks as their own kind (a "Measured check" badge, a hatched slot in the
+  problems chart, cards with what was measured, where, the usual fix and a Copy prompt that uses `genclass_scan`), counts
+  them in the app's status and in Fix them all, and lists their fixes in What GenClass did. They are never counted as
+  model detections.
+- Tests: `packages/runtime/test/telemetry-visual.test.ts`, `telemetry-worker/test/dashboard.test.ts` and
+  `pages.test.ts` ("measured visual checks").
+
+Coverage: telemetry is off on local hosts and visual is off on public hosts by default, so counts come from apps
+that set `visual: true` in production (`data-visual="on"`, or `data-visual="fix"` with the runtime fixes), from
+visitors who open a link with `?genclass-visual=on`, and from pages where the devtools overlay mounted or
+`rt.checkNow()` ran. The dashboard's Health view says on how many page loads the checks ran.
 
 ## 11. Runtime fixes and undo
 
@@ -1473,7 +1503,8 @@ Owner decisions:
 1. **Defaults**: `visual: "local"` (on for development hosts and with devtools mounted, off on public hosts unless
    `visual: true`), and whether `?genclass-visual=on` may enable report-only checks on public hosts.
 2. **Fixes**: explicit `visual.fix` plus `guard` or `heal`, and the three v1 fix kinds.
-3. **Telemetry**: the fields in section 10 (counts per kind and normalised route), under rule 8.
+3. **Telemetry**: the fields in section 10 (counts per kind and normalised route), under rule 8. Approved on
+   2026-10-27, sent from `0.3.0-beta.5`.
 4. **Sinks**: `"check"` records only for sinks that list it in `kinds`.
 5. **Gates**: the numbers in section 15. Measured on 2026-10-13 (RESULTS.md section 14): the false-report and fix gates
    are met; the 4x-throttling gate is not reliably met (pass 1 inside 100 ms on 78.1% of clean local pages in runs 1

packages/runtime/CHANGELOG.md

@@ -1,5 +1,27 @@
 # Changelog
 
+## Unreleased (0.3.0-beta.5)
+
+- **The measured visual checks' counts reach the dashboard** (owner approval 2026-10-27; VISUAL.md section 10,
+  TELEMETRY.md). A site full of stuck scroll locks and frozen clicks, each measured and fixed in its visitors'
+  consoles, looked clean on genclass.dev, because telemetry sent nothing about the checks. Now, counts only: the
+  `session` event says whether the checks run on the page (`visual`: `off`, `local` or `on`) and whether the app
+  turned their runtime fixes on (`fix`); once the checks have run, every `summary` event carries `counts.checks` (per
+  kind), `counts.checksByRoute` (per normalised path, at most 20), `counts.fixes` (`applied`, `undone`) and
+  `runtime.visual` (`passes`, `ms`, `partial`, `tier`). Never a selector, element text, colour, URL, size or one event
+  per finding. The summary counts are added by the checks' own file (`src/visual/bridge.ts`), so first load grows only
+  by the two session fields. With telemetry on, the console says once when the checks start what of them is sent.
+  The checks stay off on public hosts unless the app turns them on (`visual: true`, `data-visual="on"`, or
+  `data-visual="fix"` in guard mode for the fixes). Tests: `telemetry-visual.test.ts`. No model-visible text changed
+  (`situation-v2.3`).
+- **The dashboard shows them** (telemetry-worker): checks are their own kind ("Measured check", never a model
+  detection), with a card per check (what was measured, the pages, the usual fix, a Copy prompt that measures the page
+  again with `genclass_scan`), a hatched slot in the problems chart, their pages in the Pages table, their runtime
+  fixes in What GenClass did, and their coverage in Health. They count in the app's status (Critical, Needs attention,
+  Healthy) and in Fix them all; a check fix still in place counts as fixed live. The sample dashboard shows simulated
+  checks. The Snow Day Predictor's demo estimate (`src/demo.ts`) yields to the measured counts as soon as that project
+  receives its first page load with measured checks.
+
 ## 0.3.0-beta.3 (2026-10-26)
 
 - **A colour fix stays while its element is out of view** (`src/visual/engine.ts`, VISUAL.md 11.2). The periodic