Skip to content

benchmark(auto_repair): classify every measured stop by root cause - #1433

Merged
abrichr merged 2 commits into
mainfrom
repair/stop-causes
Oct 21, 2026
Merged

abrichr merged 2 commits into
mainfrom
repair/stop-causes

Conversation

@abrichr

@abrichr abrichr commented Oct 20, 2026

Copy link
Copy Markdown
Member

Measurement only; no engine change.

stop_causes.py reads every halted run's report.json and, when self-repair ran, its evidence, and writes one row per stop: what the halted step's error says (stop), what kept repair from resuming it (blocker) and, for RECONCILIATION_REQUIRED runs, the steps whose effect could not be settled. STOP_CAUSES.md groups the 71 stops behind README.md's numbers by root cause and marks false stops.

Root cause Stops False stops Resumed
Control renamed or moved 24 0 19
Interstitial dismissed by "Continue" 13 0 0
Save behind a closed menu 3 0 0
Pixel or mined-text postcondition on a correct screen 9 9 0
Identity check read a label behind a modal (#1347) 3 3 0
Repaired target cropped in a transient state (openIMIS) 3 0 0
Correct stop (data fault, new business choice) 16 0 n/a

A finding that corrects README.md: the OpenEMR stops were not blocked by step_039 alone. The transaction classifier could not settle the 12 unreviewed Control+a / Enter steps (step_002...step_037), which #1366 classifies on a recording made with network capture; self-repair's own check then also refuses on step_039 ("Create New Patient"), whose label reads as a save although the click only opens the duplicate check.

The fixes follow in separate PRs.

🤖 Generated with Claude Code

stop_causes.py reads each halted run's report and self-repair evidence and
names the stop and what kept repair from resuming it. STOP_CAUSES.md groups
the 71 measured stops (MockMed, worklist, OpenEMR, openIMIS, silent-saves)
by root cause and marks the false stops.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@abrichr
abrichr added this pull request to the merge queue Oct 21, 2026
Merged via the queue into main with commit 1d5ab8f Oct 21, 2026
38 of 39 checks passed
@abrichr
abrichr deleted the repair/stop-causes branch October 21, 2026 05:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant