Skip to content

fix(flow): the census follows the target's teach-time role on every rung - #1557

Merged
abrichr merged 3 commits into
mainfrom
fix/census-teach-time-role
Oct 25, 2026
Merged

abrichr merged 3 commits into
mainfrom
fix/census-teach-time-role

Conversation

@abrichr

@abrichr abrichr commented Oct 25, 2026

Copy link
Copy Markdown
Member

Fixes the admission block in the production lifecycle run (packages/flow/benchmark/lifecycle/results/production-2026-10-25-final-blocked/README.md, commit 84ce8bd on measure/production-lifecycle-final).

Before

#1504 skips the candidate census for a target that teach time recorded as a form field or control, but only when the structured tier verified the step. On OpenEMR, step_016 is the "Contact" section button (structural role button), and its pixel crop verifies it. The OCR census had no band to count, and a save follows, so the census turned the verified match into identity_uncounted. The closed-loop scroll before it (step_015) read that as "anchor not in view" and spent its whole budget. The representative case halted in 2 of 2 attempts, so the production program couldn't be admitted. It admitted at 38fc21a.

After

  • identity_census.teach_time_role(step) says what teach time recorded:
    • row when the recorder saw a record row (structured_row true);
    • control when it saw none (structured_row false), or when the structural locator names a form-field or control role (textbox, searchbox, combobox, checkbox, radio, spinbutton, switch, slider, button, tab, menuitem);
    • unknown otherwise.
  • The census treats control as not a row on every rung (structured, template, OCR, pixel), so a verified identity stands. row and unknown are still counted, so a pixel-resolved link, cell, list item or row, or a target with no recorded role, still halts when it can't be counted. A teach-time row wins over a control role (a row's own button is still counted).
  • If the next target is in view but the census refuses it, the closed-loop scroll stops with a safety halt that names the census reason, and sends no input. It no longer scrolls until the budget runs out.

Tests

tests/test_census_teach_role.py:

  • A pixel-resolved button, text box, combo box or check box before a save verifies (census_status="not_a_row").
  • A pixel-resolved target with no role, or a row, list item, link or grid-cell role, still halts with identity_uncounted.
  • A row recorded at teach time with a button role is still counted.
  • Two OCR rows that read the recorded band halt with identical_candidates.
  • A scroll whose in-view target is refused stops without wheel input, and its error names the census reason.

Locally (1 worker): ruff, mypy (including strict), and 13 test files, 376 passed.

I haven't rerun OpenEMR here; that's the lifecycle run's job.

🤖 Generated with Claude Code

#1504 exempted a teach-time form field or control from the candidate
census only when the structured tier verified it. On OpenEMR, step_016
(the "Contact" section button, recorded with structural role "button")
is verified by its pixel crop. The OCR census found no band to count,
a save follows, and the census turned the verified match into
identity_uncounted. The closed-loop scroll before it read that as
"anchor not in view" and spent its whole budget, so the production
program could not be admitted (2 of 2 attempts).

- identity_census.teach_time_role: "row" when the recorder saw a record
  row (structured_row True), "control" when it saw none
  (structured_row False) or the structural locator names a form-field
  or control role (textbox, combobox, checkbox, radio, spinbutton,
  switch, slider, button, tab, menuitem), otherwise "unknown".
- The census treats a "control" as not_a_row on every rung. "row" and
  "unknown" are still counted, so a link, cell or list item, or a target
  with no recorded role, still halts when it can't be counted.
- Closed-loop scroll: when the next target is in view but the census
  refuses it, the scroll stops and says why (safety halt, no input),
  instead of scrolling until the budget runs out.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@abrichr
abrichr enabled auto-merge October 25, 2026 08:55
@abrichr

abrichr commented Oct 25, 2026

Copy link
Copy Markdown
Member Author

Real-app check, run by the production-lifecycle session: the OpenEMR representative admission case is VERIFIED with the census on (exact_record read-back, 0 model calls, 108 s engine). It was tested at 76ad93efa: measure/production-lifecycle-final (origin/main 6dc0be4 + #1516 + harness) merged with this branch's 39daa43. Before the fix, the same case halted at step_015 in 2 of 2 attempts. Only the representative case ran, not the 5 fault cases.

@abrichr
abrichr disabled auto-merge October 25, 2026 09:05
Review fix for #1557. The teach-time control role now only replaces a
cannot_count verdict; an OCR census that can count still decides. A
recorded structured identity with no teach-time census (a backend
without one, a legacy bundle) is unknown, not a control.

Without this, a per-row button (role button) with OCR look-alike rows
was clicked, and a table-row Edit button taught and replayed on a
backend without a census signed the swapped look-alike chart.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@abrichr

abrichr commented Oct 25, 2026

Copy link
Copy Markdown
Member Author

AI review (coordinator)

Verdict: this reopens the wrong-row bug on two paths. I turned off auto-merge. A fix with regression tests is on branch review/1557-census-control-fix (commit f911e85, based on this PR's head).

How teach time decides structured_row

The recorder sets structured_row only when the target has a structured identity and the backend returned a census (recorder.py:695). The browser census finds a row only through closest('tr, [role="row"], li, [role="listitem"]'). Otherwise structured_row stays None. That happens with:

  • div, CSS-grid or card rows (no structured identity is recorded, so the census never runs)
  • every native backend (Windows, macOS, Linux). They record role="button" but have no identity_candidates_at.
  • legacy bundles, and a teach-time census that throws

With None, teach_time_role falls back to the structural role. For any per-row button (button, tab, menuitem, and also checkbox for row-select boxes) that fallback returns "control".

Hole 1: the role skips an OCR census that could count

When the check is not structured, this PR returns not_a_row for a "control" role without running _ocr_census. Before this PR, that census found look-alike rows and halted. I tested a per-row "Open" button with a recorded OCR band and two OCR rows with the same text, at irreversible risk:

role base (merge-base) this PR
none halts, identical_candidates halts
button / menuitem / tab halts, identical_candidates clicks, census_status=not_a_row

Hole 2: a structured row identity with no teach-time census becomes a control

Real headless Chromium. The page has two identical <tr> rows, each with <button id="edit-N">Edit</button>, where the id follows the row's position. I taught "Edit on chart 1001, then Sign" and replayed with the rows swapped, using a backend without a census (_CannotCountBackend, which stands in for native and remote surfaces) for both teach and replay.

  • Teach recorded structured_identity='John Smith Follow-up Dr Lee' (that is row text), structured_row=None and role button, so teach_time_role returned control.
  • Base: the step halts with identity_uncounted.
  • This PR: success=True, status Opened chart 1002, Signed chart 1002, census_status=not_a_row. That is the wrong record.

What still holds

  • On Playwright, a <tr> with a button inside gets structured_row=True, so the role is row and the step halts (identical_candidates). This PR doesn't change that.
  • The role+name structural locator refuses div rows whose buttons all have the same name ("Open" x3, candidate_count=3). The census isn't involved there.

Not caused by this PR, but open

Div rows whose buttons have positional ids (#open-0) sign the wrong chart on base and on this PR. No identity is armed for that step, and the DOM census can't see div rows, so #1504 never covered them. This needs its own issue: count repeated sibling structures, not only tr/li/role=row.

Fix (on the branch)

--- identity_census.py  teach_time_role
     if recorded_row is False:
         return "control"
+    if getattr(template, "structured", None):
+        # A structured identity was recorded but nothing counted its rows (a
+        # backend without a census, a legacy bundle): it may be a row's text.
+        return "unknown"
--- replayer.py  _apply_candidate_census
         if check.mode == "structured":
             decision = self._structured_census(step, resolution.point)
-            if decision.verdict == "cannot_count" and teach_role == "control":
-                decision = census_mod.CensusDecision("not_a_row")
-        elif teach_role == "control":
-            decision = census_mod.CensusDecision("not_a_row")
         else:
             decision = self._ocr_census(step, resolution, before_png, params, workflow)
+        if decision.verdict == "cannot_count" and teach_role == "control":
+            # A census that can count always decides.
+            decision = census_mod.CensusDecision("not_a_row")

The fix follows one rule: a control role can only replace cannot_count, and a census that can count always decides. The OpenEMR step_016 case still passes. Its pixel-only _ocr_census returns cannot_count (no band), and the role then turns that into not_a_row. All 13 of this PR's tests pass with the fix.

Tests on the branch

New tests:

  • test_a_control_role_never_skips_an_ocr_census_that_can_count (button, menuitem, tab, checkbox)
  • test_a_recorded_structured_identity_with_no_teach_census_is_not_a_control
  • test_realism_browser.py::test_a_row_button_taught_without_a_census_halts_after_a_swap (real Chromium)

Results:

  • On this PR's head, all 6 new tests fail.
  • With the fix, test_census_teach_role.py, test_realism_browser.py and test_identity_census.py pass, and ruff is clean.

If you merge the branch into this PR, re-enable auto-merge.

…nsus that can count

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@abrichr
abrichr enabled auto-merge October 25, 2026 09:11
@abrichr
abrichr merged commit 78ff5ad into main Oct 25, 2026
57 of 59 checks passed
@abrichr
abrichr deleted the fix/census-teach-time-role branch October 25, 2026 10:22
abrichr added a commit that referenced this pull request Oct 25, 2026
Brings in #1504, #1557 and the reviewer's control-role fix. In
_apply_candidate_census, the pixel census still runs when a pixel or VLM
identity has no OCR band; a teach-time control role then replaces only
a cannot_count. A census that can count, pixel included, always decides.

The #1557 tests for the no-census branch now make the pixel census
unavailable (crop-only steps can otherwise be counted by pixels). A new
test checks that the pixel census decides for a control: the crop found
once verifies, found twice halts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant