Skip to content

fix(flow): halt when another shown row has the same identity - #1504

Merged
abrichr merged 4 commits into
mainfrom
claude/affectionate-cerf-161abd
Oct 25, 2026
Merged

abrichr merged 4 commits into
mainfrom
claude/affectionate-cerf-161abd

Conversation

@abrichr

@abrichr abrichr commented Oct 24, 2026 •

Copy link
Copy Markdown
Member

Fixes the homonym-row fail-open the realism audit reproduced in headless Chromium (REALISM_AUDIT identity-8, audit PR #1467).

Before

A worklist shows two rows with the same text, "John Smith Follow-up Dr Lee" (data-id 1001 and 1002). The recording opens 1001 and clicks "Sign visit note". After the rows swap order, replay clicks the row where 1001 used to be, which is now 1002. The structured identity check returns verified, because the text matches exactly, and the run reports success=True with "Signed chart 1002" on screen. Every rung behaves this way: once the point lands on a row whose text matches, nothing checks whether another shown row reads the same.

After

Once the identity ladder verifies, the runtime counts the shown rows that carry the same identity:

Shown rows with that identity Record id recorded at teach time Result
1 none, or the same id verified, as before
1 a different id identity mismatch, halt
2 or more yes, and one shown row has it acts on that row; a template or text resolution moves onto it
2 or more none, or several rows share it halts before acting, whatever the step's risk
too many rows to count any halts
the target is not a row (a form field, a button) any proceeds as before
the surface can't count rows (no census on the backend, or a pixel crop with no OCR band) any halts if this step or a later one is a consequential write (identity_uncounted), under every profile; otherwise proceeds and records census_status="cannot_count"
  • The browser backend gets identity_candidates_at(x, y). It lists each shown row (tr, li, [role=row], [role=listitem]) whose structured identity equals the identity under the point, with the row's stable record id: app-owned data-* id attributes (data-id, data-record-id, data-mrn, ...) and the path of the row's link. Attribute names that look like an index, and values equal to the row's position, are ignored.
  • The recorder captures that id on click events. The compiler stores it in the identity template as a salted HMAC, so the bundle never holds the plaintext. It isn't stored when the row's identity contains a run parameter, since the demonstrated id would belong to another record.
  • A structural resolution is never moved. If it points at the wrong record, the identity check refuses it.
  • On pixel-only screens the same count runs over the frame's OCR rows, using the same band verifier as the OCR tier.
  • The recorder also records whether the identity came from a row (structured_row). That lets a replay surface with no census still run form fields and buttons.
  • "Sign" now counts as a write verb ("Sign visit note", "Co-sign", "e-sign"), never "sign in", "sign on" or "sign out". In the audit repro, the compiled Sign step wasn't treated as consequential before this change.
  • Qualification checks retained identity evidence against the ladder verdict alone. The census reads the whole screen, which that evidence doesn't hold.
  • IdentityCheck gains candidate_count, ambiguity_halt and census_status. Both are left out of the JSON when unset, and so is the template's record_key, so existing bundle and report bytes don't change.

With the fix, the audit's replay signs chart 1001 and reports success. Without data ids, the run stops at the first click and nothing is opened.

Tests

  • tests/test_realism_browser.py (real headless Chromium): this PR carries the file from test: realism audit and real-browser tests for gaps the fakes hide #1467 with the strict xfail removed. The audit test passes. New cases cover rows in the recorded order (no false stop), a swap with ids (signs 1001), two and three identical rows without ids in several orders (halt, nothing opened), and a unique row after a swap (passes).
  • tests/test_identity_census.py: the decision rules for two and three identical candidates, identical text with distinct ids (picked by id), a unique row, a unique row carrying another id, a recorded id on two rows, frames, incomplete censuses, hashing of the record id, and the OCR row count.
  • On a backend with no census (a Playwright backend with the census removed, standing in for native and remote surfaces), the swapped worklist halts before opening a chart, and the note form still saves.
  • The first CI run failed on two shards, for real reasons. The OCR census's extra screen read used up the scripted OCR results in fake-vision tests, and qualification's exact-equality check saw the new candidate_count. The scripts now include the census read. Fakes that model a single record row now expose a one-row census.
  • I ran ruff, mypy (including the strict safety path) and 79 targeted test files (2,971 passed) locally. CI runs the full suite.
  • Behavior change on native and remote surfaces: a structured or pixel identity that leads into a write now halts there until those backends get a census.

Not covered here

  • The native accessibility backends (Windows, macOS, Linux) and pixel crops with no OCR band have no candidate census yet. They now halt before writes (above) instead of guessing.
  • The in-page interactive recorder doesn't capture the record id. Its bundles still get the census, so identical rows halt there instead of being picked by id.
  • tests/e2e/test_chaos.py::TestLayoutSwapMidRun::test_swapped_type_buttons_still_select_the_right_type fails on origin/main as well. Swapped type buttons replay as success and select the wrong type. That's a separate fail-open and isn't touched here.

🤖 Generated with Claude Code

abrichr and others added 2 commits October 24, 2026 12:24
Replay could open and sign the wrong patient's chart and report success.
Two worklist rows read "John Smith Follow-up Dr Lee" (data-id 1001 and
1002). After they swapped order, the template rung landed on 1002 and the
structured identity check verified it, because the text matched exactly.
Nothing checked whether the identity picked one record.

After the identity ladder verifies, the runtime now counts the shown rows
with the same identity (a DOM census on the browser backend, an OCR pass
over the frame on pixel-only screens):

- one row: the verdict stands;
- several rows and the bundle recorded the row's stable record id (an
  app-owned data-* id or the row link's path, captured at teach time and
  stored as a salted hash): act only on the row with that id, moving a
  visual resolution onto it when needed;
- several rows and no recorded id: the step halts before it acts,
  whatever its risk (IdentityCheck.ambiguity_halt);
- a recorded id that no shown row carries: identity mismatch.

The realism audit's real-browser test passes and loses its xfail. New
browser cases cover the recorded order, the swap with ids, two and three
identical rows without ids, and a unique row after a swap.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
abrichr added a commit that referenced this pull request Oct 24, 2026
Each pull request is made of move commits, proven by AST equality, and
seam commits read in full. Coordination table refreshed against the open
pull requests (#1500, #1501 and #1504 added).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The census treated "this backend cannot count rows" like "the target is
not a row" and let the run continue. On native, RDP and Citrix surfaces
the homonym-row fail-open stayed open.

The census now tells the two apart:

- not_a_row: the target has no row identity (the browser reports it, or
  teach time recorded the identity as a form field or control). The
  verdict stands.
- cannot_count: the target reads as a row (a structured row identity, or
  a pixel match with no OCR band to read), but the surface has no census.
  A step that leads into a consequential write halts with
  ambiguity_halt="identity_uncounted" under every profile. Elsewhere it
  runs and records census_status="cannot_count".

"Sign" now counts as a write verb ("Sign visit note", "Co-sign",
"e-sign"), never as "sign in", "sign on" or "sign out". Before this, the
compiled Sign step in the audit repro was not consequential.

Qualification compares retained identity evidence with the ladder verdict
only; the census reads the whole screen, which that evidence doesn't hold.

CI shards 1/4 and 4/4 failed on the first push for real reasons: the OCR
census's extra screen read used up scripted fake OCR results, and the
exact-equality check in qualification saw the new candidate_count. Test
scripts now include the census read. Fakes that model one record row
expose a one-row census.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@abrichr
abrichr enabled auto-merge October 24, 2026 18:40
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@abrichr
abrichr added this pull request to the merge queue Oct 25, 2026
abrichr added a commit that referenced this pull request Oct 25, 2026
…e only)

Qualification fault injection, consequential delivery, effect checks and
read-back, the API tier, acting by kind, settling and confirming move
verbatim into mixins (107 units; verify_moved_code finds no edits).
replayer.py goes from 18,300 to 10,926 lines.

Left in place because open pull requests change them: identity checks
(#1518, #1504), _resolve_step (#1504), postconditions and step evidence
(#1516, #1514), the starting-state settle and qualification boundary
(#1449), and the run and step drivers. Two methods that name Replayer
stay too.

Two tests patched module globals the moved code now reads from its new
module, and patch them there as well: resolve (scroll readiness is in
replay.acting, interstitials in replay.settling) and _DIALOG_WAIT_S
(replay.acting).

Checked: strict mypy on runtime/core, runtime/replay and replayer.py;
package-wide mypy; both import contracts; 3,149 tests across the 146
files that import the replayer plus the real-browser e2e files.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@abrichr
abrichr enabled auto-merge October 25, 2026 05:40
@abrichr
abrichr merged commit 6d04711 into main Oct 25, 2026
39 checks passed
@abrichr
abrichr deleted the claude/affectionate-cerf-161abd branch October 25, 2026 05:40
abrichr added a commit that referenced this pull request Oct 25, 2026
Brings in #1504, #1557 and the reviewer's control-role fix. In
_apply_candidate_census, the pixel census still runs when a pixel or VLM
identity has no OCR band; a teach-time control role then replaces only
a cannot_count. A census that can count, pixel included, always decides.

The #1557 tests for the no-census branch now make the pixel census
unavailable (crop-only steps can otherwise be counted by pixels). A new
test checks that the pixel census decides for a control: the crop found
once verifies, found twice halts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant