Repository navigation
test(webui): let the browser acceptance matrix survive a CI run - #228
Merged
Merged
Conversation
modacker
changed the title
added 2 commits
October 23, 2026 05:33
A record file is tracked: the plan's `complete/` directory is where the
accepted evidence lives, and the file in the checkout is the record of
the run that was accepted. A run rewrites it — every record carries the
time and the machine it ran on — so the working tree is dirty once the
suite is over.
That dirt is the run's own timestamps, not a source change, and the
release packaging refuses to run with a dirty tree ("Packaging requires
a clean committed working tree"). So on CI the suite puts the file back
when it is done: the record helper remembers what the file held before
the first write and restores it at the end of the suite.
Locally the records stay — they are what the run just produced, and the
published matrix is written from them.
The acceptance matrix drives a real browser: the first case launches it, opens a tab and waits for the strip to gain it. Vitest's default budget is 5s, which is smaller than a cold browser start on a loaded runner — and a case that runs out of time leaves the browser half launched, so the next case finds a browser that exited during startup and fails too. The sibling suite (`browser-panel-live-navigation.test.ts`) has always given its cases 30s; the acceptance cases get the same budget.
modacker
force-pushed
the
fix/webui-acceptance-record-restore
branch
from
October 22, 2026 21:33
2f6b5b0 to
342d468
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two things keep the browser acceptance matrix from surviving a CI run. Both are in the test tree; no product code changes.
1. The release packaging refuses to run with a dirty working tree
The tree is dirty because the suites rewrite tracked acceptance records.
.plan/<plan>/complete/<suite>.acceptance.jsonis where the accepted evidence lives, and a run replaces it — every record carries the time and the machine it ran on — so aftertest:webui(the vitest acceptance matrix) andtest:webui-browser(the three real-browser suites) the checkout no longer matches HEAD.That dirt is the run's own timestamps, not a source change, so the suites put the files back when they are done on CI. The record helper remembers what a file held before the first write and restores it at the end of the suite.
Locally the records stay. They are what the run just produced, and the published matrix (
packages/webui/docs/browser-use/09-acceptance-record.md) is written from them — restoring them locally would throw away the evidence a run was run for.2. The cases ran on Vitest's default 5s budget
The matrix drives a real browser: the first case launches it, opens a tab and waits for the strip to gain it. On a loaded runner that start takes longer than 5s, so the case times out — and a case that runs out of time leaves the browser half launched, which is why the cases after it fail with
The browser exited during startup with code 21rather than with a timeout of their own.The sibling suite (
browser-panel-live-navigation.test.ts) has always given its cases 30s; the acceptance cases get the same budget.How it was verified
CI=truerunning the acceptance matrix leaves.plan/untouched;CIleaves the record file updated, as it is today;CI=trueonbrowser-bridge-real-browser.spec.tsleaves.plan/browser-use/complete/real-browser.acceptance.jsonuntouched, and withoutCIthe file is updated;CI=true(211 files) leaves the tree clean;pnpm --filter @mavis/webui typecheck:testis clean.The change is confined to the test tree: the record helper, the four suites that write records, and the cases' budgets. No product code, and no registry entries are needed because no file was added.
This is what keeps #224's
CLI releasejob red: that job runs both suites and then packages, so it hits the dirty tree on the record files. With this merged, #224's own remaining red goes away.