Skip to content

test(webui): refresh the Linux Chromium visual baselines and settle the settings dialog - #236

Merged
modacker merged 4 commits into
webuifrom
fix/webui-linux-baseline-regen
Oct 23, 2026
Merged

modacker merged 4 commits into
webuifrom
fix/webui-linux-baseline-regen

Conversation

@modacker

Copy link
Copy Markdown
Collaborator

Change

The test:webui-browser gate on webui was red with 14 failures. None of them is a product regression. The red had three causes, all in the test fixtures rather than in the WebUI client, and this PR fixes all three.

1. Twelve visual baselines were generated in the wrong environment

The twelve failing toHaveScreenshot cases reported pixel differences of 0.01 to 0.03 of the image pixels, and every one of the fourteen baseline images kept its exact pixel dimensions — shell-and-rail-linux.png is 1434x900 in both the committed and the freshly rendered version. A layout change would have moved the dimensions; nothing moved.

Rendering the committed baseline next to a freshly rendered one shows what the pixels are. The committed images draw the Chinese interface text with real CJK glyphs (新建任务, 导入会话, 插件, 输入消息…(输入 / 唤起命令)). GitHub's ubuntu-latest runner image ships no CJK font, so Chromium draws the very same labels as tofu boxes. The baselines were produced on a Linux box that had a CJK font installed, not on the runner the gate uses.

The baselines are therefore regenerated on the Linux Chromium the gate runs on, following the procedure the repository already documents in packages/webui/docs/verification/browser-suite.md:

To deliberately regenerate only these baselines, use pnpm exec playwright test packages/webui/test/browser/visual-baselines.spec.mjs --update-snapshots on Linux CI Chromium, inspect every resulting image, then run the same command without --update-snapshots to compare.

Nothing else changes: the file names, the file count and the pixel dimensions are all identical, so the repository still keeps the fourteen representative screenshots it documents. The job that produced them was temporary and is not part of this PR.

2. Two rail-pin expectations were left behind by a fixture rename

de314e05 changed the shared browser fixture's two synthetic session titles to English — { id: "A", title: "Synthetic session A" } and { id: "B", title: "Synthetic session B" } — for cross-environment portability. Two expectations in rail-pin.spec.mjs still asserted the old Chinese titles:

Expected: "取消置顶:绿川椒 Demo 订货小程序"
Received: "取消置顶:Synthetic session A"

Expected: "置顶:灵动岛卡住问题"
Received: "置顶:Synthetic session B"

The rendered label is correct; only the expectation is stale, so the two expectations now read the title the fixture actually publishes. The glyph assertions next to them are untouched, because they check the drawing rather than the title. The two old Chinese titles still in settings-account-tab.spec.mjs are deliberately left alone: those cases inject their own session array and do not read the shared fixture.

3. The settings dialog baseline recorded a frame of an entrance animation

After the regeneration, thirteen of the fourteen images came back byte for byte identical across independent runs on different runner instances, and one did not: settings-dialog-linux.png. It moved by a few hundred pixels, the text ink ran one pixel wider, and the theme preview images were re-rasterised a fraction of a pixel apart.

The cause is that the settings overlay fades in over the app and the panel behind it slides in (webui-overlay-enter, webui-modal-enter, webui-settings-content-in in packages/webui/src/client/features/settings/ui/settings.css). The screenshot was taken while those were still running, so the baseline held a frame of the entrance — and the entrance advances on the clock, so the next run caught a different frame. The browser panel case in the same file already waits for its animations to finish before it screenshots; the settings dialog case did not, and now does, over the whole subtree.

Validation

  • Checks run and results: the Source verification workflow on this branch, verify (ubuntu-latest, 24), node 24, pnpm verify --profile full — green, including the test:webui-browser gate that was red on webui at 97e33fd8.
  • Reproducibility check: two regenerations on the Linux runner with the settle wait in place produce all fourteen images byte for byte identical. Before the wait, the same comparison across three independent runs (twice with the plain job shape, once after the full verify pipeline) returned thirteen identical images and a differing settings dialog, so the job shape and the environment are not the cause and the animation frame was.
  • Local run: packages/webui/test/browser/rail-pin.spec.mjs — 11 passed, including the pin button offers the unpin glyph on a pinned row and the pin button offers the pin glyph on a row that is not pinned. packages/webui/test/browser/settings-account-tab.spec.mjs — 9 passed, on a file this PR does not touch.
  • NOT RUN: a local run of the two screenshot specs. A macOS run cannot compare against -linux.png baselines, so it proves nothing here; the Linux runner is the only meaningful check, which is why the gate was verified there.

Two trade-offs worth a maintainer decision

The baselines are now full of tofu boxes. They record what ubuntu-latest renders, which is the Chinese interface text as empty squares. The baselines therefore no longer show — and so no longer prove — that the Chinese text renders. The alternative is to install a CJK font in the browser job before the gate runs and regenerate the baselines with it, which would give baselines that show the real text, at the cost of a permanent CI change and an apt-get step on every Linux run. This PR takes the minimal route and leaves that decision open.

The baselines sit on a shared runner image. If that image ever gains a CJK font — or drops one — the gate goes red again for the same reason as before. The repository's own documentation already treats the runner as the pinned reference, so this is the status quo rather than something this PR introduces.

Publication and contribution checks

  • I have permission to contribute these changes under the existing licenses applicable to the changed files/packages; imported material and its provenance are identified and existing notices are preserved.
  • No credentials, account data, real user content, internal source history or private review material is included.
  • Added/removed source files were reviewed before regenerating release/public-source.json; new tests are declared in test/vitest-suites.json where applicable. — No source file is added or removed: the diff is 14 modified baseline PNGs and two modified specs, so both manifests are unchanged.
  • Shared English/Chinese documentation and capability/verification records are updated where applicable. — The procedure this PR follows is already documented in packages/webui/docs/verification/browser-suite.md; no document describes the old baselines.

Maintainer handoff

Publication scope or license changes (if any): none.

Shared-source port: not needed.

probe added 4 commits October 23, 2026 11:36
…itles

What broke: the Linux browser gate failed on two rail-pin cases because the
assertions still expected the old Chinese session titles. The shared browser
fixture now names its two synthetic sessions "Synthetic session A" and
"Synthetic session B", and de314e0 changed the titles to English for
cross-environment portability without updating these two expectations. The
rendered labels are correct; only the expectations are stale.

Why this fix: the pin button label is built from the session title, so the two
expectations now read the title the fixture actually publishes. The glyph
assertions next to them are untouched, because those check the drawing rather
than the title.

How this was verified: the two rail-pin cases pass in a local Playwright run
against the built client, and the browser gate is green on the Linux runner
for this branch.
What broke: the Linux browser gate failed twelve toHaveScreenshot cases.
Every diff was tiny -- 0.01 to 0.03 of the image pixels -- and every image kept
its exact pixel dimensions, so no layout had moved. Comparing the committed
baseline against a freshly rendered one shows why: the committed images draw
the Chinese UI text with real CJK glyphs, while GitHub's ubuntu-latest runner
ships no CJK font, so Chromium renders the same labels as tofu boxes. The
baselines had been produced on a Linux box that had a CJK font installed, not
on the runner the gate actually uses.

Why this fix: the baselines are regenerated on the Linux Chromium the browser
gate runs on, following the procedure in
packages/webui/docs/verification/browser-suite.md, which asks for
--update-snapshots on Linux CI Chromium. Nothing else changes: the file names,
the count and the pixel dimensions are all the same, so the repository still
keeps the fourteen representative screenshots it documented.

How this was verified: the regeneration ran as a one-off job on ubuntu-latest
with node 24, where pnpm exec playwright test --update-snapshots over
visual-baselines.spec.mjs and plugin-catalogue-layout.spec.mjs produced all
fourteen files and thirteen cases passed. The browser gate on this branch is
green on the Linux runner afterwards.
What broke: the settings dialog baseline was the one of the fourteen that two
runs of the same build did not agree on. Comparing two independent
regenerations on the Linux runner, thirteen of the fourteen files came back
byte for byte identical and this one did not: it moved by a few hundred
pixels, the text ink ran one pixel wider and the theme preview images were
re-rasterised a fraction of a pixel apart. The overlay fades in over the app
and the panel behind it slides in, so the frame a screenshot caught was a
frame of that entrance, and the entrance advances on the clock.

Why this fix: the browser panel case in this same file already waits for the
element's animations to finish before it screenshots, and the settings dialog
case did not. It now waits for the same thing over the whole subtree, so the
recorded frame is the settled one that every later run records again. The
infinite animations are left alone, so the wait cannot hang on one.

How this was verified: two regenerations on the Linux runner with this wait in
place produce the same settings dialog image byte for byte, and the browser
gate on this branch is green afterwards.
What broke: the settings dialog baseline was regenerated with the entrance
animation still running, so it held a frame of the overlay fading in rather
than the frame the app settles on. Two runs disagreed on it while the other
thirteen came back byte for byte identical.

Why this fix: the previous commit waits for the overlay to settle before the
screenshot, so the baseline records that settled frame. Only this one image
changes; the other thirteen are byte for byte what they already were, which
is the check that the wait did not move anything else.

How this was verified: two regenerations on the Linux runner with the wait in
place produce all fourteen images byte for byte identical, and the browser
gate on this branch is green afterwards.
@modacker
modacker merged commit eb11f47 into webui Oct 23, 2026
9 checks passed
@modacker
modacker deleted the fix/webui-linux-baseline-regen branch October 23, 2026 04:44
modacker pushed a commit that referenced this pull request Oct 23, 2026
The baselines merged in #236 were rendered on a runner that had no CJK
font at all, so every Chinese glyph in them is a tofu box. This commit
replaces all 14 Linux screenshots with the same surfaces rendered with
Playwright's Linux system dependencies installed, which include
fonts-wqy-zenhei. The Chinese in the new baselines is readable text.

Regenerated on ubuntu-latest with `playwright install --with-deps
chromium` and `--update-snapshots`, three times. Under Playwright's own
comparison (pixelmatch, threshold 0.1) the three runs are identical:
zero differing pixels in all 14 images, in every pair of runs. At the
byte level two images differ between runs, by one and by nine raw pixels
respectively; both are far below the comparator threshold, so the gate
result does not depend on which run supplied the file.

No assertion, expectation or pixel threshold changed, and the file names
and count are unchanged.
modacker pushed a commit that referenced this pull request Oct 23, 2026
The baselines merged in #236 were rendered on a runner that had no CJK
font at all, so every Chinese glyph in them is a tofu box. This commit
replaces all 14 Linux screenshots with the same surfaces rendered with
Playwright's Linux system dependencies installed, which include
fonts-wqy-zenhei. The Chinese in the new baselines is readable text.

Regenerated on ubuntu-latest with `playwright install --with-deps
chromium` and `--update-snapshots`, three times. Under Playwright's own
comparison (pixelmatch, threshold 0.1) the three runs are identical:
zero differing pixels in all 14 images, in every pair of runs. At the
byte level two images differ between runs, by one and by nine raw pixels
respectively; both are far below the comparator threshold, so the gate
result does not depend on which run supplied the file.

No assertion, expectation or pixel threshold changed, and the file names
and count are unchanged.
modacker added a commit that referenced this pull request Oct 23, 2026
… baselines (#239)

* Install Chromium system dependencies in the browser gate

The Linux browser gate installed the pinned Chromium revision but none of
its system dependencies, so the runner had no CJK font available. The
WebUI is Chinese, and every Chinese glyph in the committed Linux
baselines rendered as a tofu box.

`playwright install --with-deps chromium` installs Playwright's own
dependency set for the platform, which for ubuntu24.04-x64 includes
fonts-wqy-zenhei. The command lives in the npm script rather than in a
workflow file on purpose: both `Source verification` and
`WebUI npm preview` reach the gate through `pnpm verify --profile full`,
so the script is the single place that decides what the gate installs.
Putting it in one workflow would let the other path drift.

On macOS and Windows `registry.installDeps` has no matching branch and
returns without installing anything, so local development is unaffected.

* Regenerate the Linux visual baselines with real Chinese glyphs

The baselines merged in #236 were rendered on a runner that had no CJK
font at all, so every Chinese glyph in them is a tofu box. This commit
replaces all 14 Linux screenshots with the same surfaces rendered with
Playwright's Linux system dependencies installed, which include
fonts-wqy-zenhei. The Chinese in the new baselines is readable text.

Regenerated on ubuntu-latest with `playwright install --with-deps
chromium` and `--update-snapshots`, three times. Under Playwright's own
comparison (pixelmatch, threshold 0.1) the three runs are identical:
zero differing pixels in all 14 images, in every pair of runs. At the
byte level two images differ between runs, by one and by nine raw pixels
respectively; both are far below the comparator threshold, so the gate
result does not depend on which run supplied the file.

No assertion, expectation or pixel threshold changed, and the file names
and count are unchanged.

---------

Co-authored-by: probe <probe@local>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant