diff --git a/docs/audits/REALISM_AUDIT.md b/docs/audits/REALISM_AUDIT.md new file mode 100644 index 0000000000..4dfbe05546 --- /dev/null +++ b/docs/audits/REALISM_AUDIT.md @@ -0,0 +1,113 @@ +# Realism audit: assumptions the tests do not check + +Date: 2026-10-20. Code: `origin/main` at `0669c4b1f`. + +## Why this audit + +A sign-in spike found that window-scoped recording re-finds its target window +by owner and title every frame. Chrome changes its title on almost every +navigation, so the recorder lost the window. No test caught it, because the +capture tests use a fake platform whose window title never changes. That fix +is in progress separately and is not repeated here. + +This audit asks the same question of every critical component: what does the +code assume about real apps, and do the tests use fakes that hide those +assumptions? The safety rule is that the runtime fails closed: it stops +rather than guessing. A gap that **fails open**, doing the wrong thing and +reporting success, is the worst kind and is ranked first. + +## How to read the table + +- **Rank** orders the gaps by fail-open first, then by likelihood in real use + times impact. +- **L × I** is likelihood and impact: H (high), M (medium) or L (low). +- **Fails** is OPEN (wrong action or wrong verdict, reported as success) or + CLOSED (the run stops; this costs availability, not correctness). +- **Test** names the realistic test that now pins the gap. An *xfail* test is + marked `xfail(strict=True)`. It runs in CI and is expected to fail until the + gap is fixed; when a fix lands, the test passes, strict mode fails the + build, and the marker must be removed. + +Paths are relative to `packages/flow/openadapt_flow/` unless they start with +`packages/`. + +## Ranked gaps + +| Rank | Component | Assumption in the code | Real case that breaks it | Current test (and the fake that hides it) | Fails | L × I | Realistic test | +|---|---|---|---|---|---|---|---| +| 1 | Replay: record identity | A row whose visible text matches the recording is the recorded row. Identity is verified per row and never checked for uniqueness on screen (`runtime/identity.py`). | Two worklist rows read "John Smith Follow-up Dr Lee" and swap order between recording and replay. Replay opens and signs the other patient's chart. | Homonyms in `validation/dense_surface.py` always differ in date of birth or record number. | **OPEN, reproduced** | M × H | [`test_realism_browser.py`](../../packages/flow/tests/test_realism_browser.py) `::test_identical_rows_that_swap_order_never_sign_the_other_chart` (real Chromium, xfail). Issue [#1469](https://github.com/OpenAdaptAI/OpenAdapt/issues/1469) | +| 2 | Durable resume | A save that was delivered cleanly but then failed its record check wrote nothing. The pause carries `delivery_uncertainty` only when the delivery itself raised (`runtime/durable/controller.py:621-641`). The resume guard checks only that field (`runtime/durable/resume.py:1050-1067`). | The system of record is slow and the read-back times out after Save was clicked. The operator approves the default "resume", and Save is clicked again, creating a duplicate record. | `tests/test_durable_runtime.py::test_resume_continues_from_last_checkpoint` asserts the re-delivery as correct. `FakeBackend` writes nothing. | **OPEN, reproduced** | H × H | `test_realism_durable_resume.py` (the backend writes a row per press), fix PR [#1484](https://github.com/OpenAdaptAI/OpenAdapt/pull/1484). Issue [#1470](https://github.com/OpenAdaptAI/OpenAdapt/issues/1470) | +| 3 | Capture: input attribution | Every key and click during a window-scoped recording belongs to the target window. `_geometry_for_action` checks identity, visibility and bounds, never focus (`packages/capture/openadapt_capture/window_capture.py:597-628`). The active-window poller is off in window mode (`recorder.py:3149`). | Mid-recording, the user switches to a chat app or a password prompt and types. Those keys are stored as target actions against the target's frame, including the other app's text. | `FakePlatform` (`packages/capture/tests/test_window_capture.py`) has no notion of focus. | OPEN | H × H | A focus probe on the capture scope; a fake platform that reports another focused process must refuse the key. Owned by the window-identity session (see [Coordination](#coordination)). Issue [#1471](https://github.com/OpenAdaptAI/OpenAdapt/issues/1471) | +| 4 | Read-back values | A non-exact read-back match means the expected value is present. `compare_value(..., exact=False)` is a substring test (`runtime/effects/readback_values.py:214`), used for every on-screen check (`runtime/effects/onscreen.py:603-610`). | The expected amount is 250 and the screen shows "$1,250.00". The expected quantity is 5 and the screen shows "Qty 15". A credit "$100.00 CR" reads as 100.00. Each confirms the write. | `test_onscreen_verifier.py` confirms embedded labels and never tests a neighbouring number. | **OPEN, reproduced** | H × H | [`test_realism_readback_values.py`](../../packages/flow/tests/test_realism_readback_values.py) (Hypothesis over integers and formats; xfail). Issue [#1472](https://github.com/OpenAdaptAI/OpenAdapt/issues/1472) | +| 5 | Teach: parameter inference | A value typed whole in one step is the parameter, even if the same value also appears inside longer text in another step (`jit/lift.py`). | The agent types the patient's name into its field and into a note. The note keeps the demonstrated name on every later run. The record check passes because the literal is constant. | `tests/test_jit.py` asserted the old behaviour. | OPEN → **fixed** | H × H | Hypothesis property in [`test_jit.py`](../../packages/flow/tests/test_jit.py) (fix PR [#1465](https://github.com/OpenAdaptAI/OpenAdapt/pull/1465)) | +| 6 | Effect checks: baselines | A record's identity is its raw `id`, or `str()` of each field (`runtime/effects/effect.py` `stable_id`). The baseline is stored with `model_dump(mode="json")` (`verify_run.py:709`). | Postgres, MSSQL and Oracle return UUID, Decimal and datetime values. After the JSON round trip they no longer compare equal, so every existing row counts as new. `count_new_only` then confirms a write that never happened. | `test_verify_run*.py` and `test_effect_sql.py` use SQLite ints and strings. | OPEN → **fixed** | M × H | Hypothesis round trip in `tests/test_stable_id_json_baseline.py` (fix PR, see below) | +| 7 | Teach: canaries | Canary values are different from run 1's. Nothing checks it (`compile_agent/pipeline.py:1010-1041`). | A canary run with run 1's values passes even when the program types a constant, which makes canaries blind to ranks 5 and 8. | Every test happens to use different values. | OPEN | H × H | Property test: admitting with canary values equal to run 1's must refuse. Issue [#1473](https://github.com/OpenAdaptAI/OpenAdapt/issues/1473). | +| 8 | Teach: parameter inference | A value typed whole exactly once belongs to the field it was typed in, with no record-field check. Record binding matches any field whose value equals the input (`jit/lift.py:164-171`, `record_sor.py:272-291`). | `quantity="1"` is set with a stepper, and "1" is typed into refills. Refills becomes the quantity parameter. | Synthetic workflows with long, unique values. | OPEN after the write | M × H | Hypothesis workflows with short values that collide. Issue [#1474](https://github.com/OpenAdaptAI/OpenAdapt/issues/1474). | +| 9 | Read-back values | A fuzzy text match whose longest common run covers 90% of the value confirms it (`runtime/effects/onscreen.py:262-277`). | "follow-up scheduled for 2026-10-29" confirms "…2026-10-21". "john smithson" confirms "John Smith". | No test with a difference at the end. | **OPEN, reproduced** | M × M-H | [`test_realism_readback_values.py`](../../packages/flow/tests/test_realism_readback_values.py) (xfail). Issue [#1472](https://github.com/OpenAdaptAI/OpenAdapt/issues/1472) | +| 10 | UI read-back | A "not found" page proves a record is absent. The browser read-back returns before its identity check (`runtime/effects/ui_readback.py:1089-1104`). The desktop read-back already guards this (`desktop_readback.py:1310`). | A void or delete workflow is verified by absence, but a wrong URL, an access-denied page or a filtered list also shows "not found". | `test_ui_readback_verifier.py` has no such case. | OPEN | M × H | Port the desktop test. Issue [#1475](https://github.com/OpenAdaptAI/OpenAdapt/issues/1475). | +| 11 | Effect checks: paging | The first page of a FHIR or REST search holds every match (`runtime/effects/fhir.py:245`, `rest.py:100-150`). | Servers page at 20 to 50 results. A duplicate on page 2 still confirms `expected_count=1`. | `test_effect_fhir.py` has no paging. | OPEN | M × H | Fake FHIR server with `_count=2`. Issue [#1476](https://github.com/OpenAdaptAI/OpenAdapt/issues/1476). | +| 12 | Capture: several windows | When several windows match, the largest is the target (`packages/capture/openadapt_capture/window_capture.py:2050`, `window_capture_linux.py:553`). macOS raises `WindowCaptureAmbiguousError` instead. | Two browser or EMR windows are open on different patients. The bigger one is recorded. | `test_window_capture_linux.py::test_resolver_binds_process_and_root_pixel_coordinates` asserted that the largest wins. | OPEN → **fixed** | M × H | [`test_realism_capture.py`](../../packages/capture/tests/test_realism_capture.py) (fix PR [#1468](https://github.com/OpenAdaptAI/OpenAdapt/pull/1468)). Issue [#1477](https://github.com/OpenAdaptAI/OpenAdapt/issues/1477) | +| 13 | Replay: typed-input check | Typing into a field produces the field's old value plus the typed text, and that counts as landed (`runtime/replayer.py:16545-16555`). Nothing clears the field first. | A search box keeps its last query, or a restored draft fills the field. The step saves "SmithJones". | No test with a prefilled field. In a real browser, the click's region check halts this case today. | CLOSED in the tested shape; OPEN when the fill arrives after the click check | M × H | [`test_realism_browser.py`](../../packages/flow/tests/test_realism_browser.py) `::test_a_field_autofilled_on_focus_never_saves_a_concatenation` (passes; pins the halt) | +| 14 | Replay: browser dialogs | A dialog the recording never saw is auto-dismissed by Playwright and not reported (`backends/playwright_backend.py:1137-1140, 3200-3232`). | A `confirm()` before Save returns false, so the save silently doesn't happen. | `test_multi_surface_dialog.py` covers recorded dialogs only. | CLOSED when the step has a visible postcondition (tested); OPEN when it has none | H × M | [`test_realism_browser.py`](../../packages/flow/tests/test_realism_browser.py) `::test_an_unrecorded_confirm_dialog_never_ends_in_success` (passes; pins the halt) | +| 15 | Replay: pixel typed-input check | OCR read-back of a typed value passes at 80% of the value, and any pixel change passes for values under 4 characters (`runtime/replayer.py:16598-16604`, `runtime/identity.py:631-638`). | Keystrokes are dropped over RDP or Citrix, or sent to a guest with a different keyboard layout. MRN `12345678901` arrives as `123456789xy`. | Synthetic OCR fakes. | OPEN | M × H | Fake OCR read-back that is wrong in the last 2 characters must halt. Issue [#1478](https://github.com/OpenAdaptAI/OpenAdapt/issues/1478). | +| 16 | Replay: retry after no change | When a field shows no change, the replayer re-clicks, selects all and retypes. The first delivery may have gone to another window (`runtime/replayer.py:16670-16760`). Windows-agent typing has no foreground check (`backends/win_agent/server.py:419-458`). | A notification steals focus. The patient name lands in a chat box, the retry fills the right field, and the step passes. | Fake backends never change focus. | OPEN | M × H | Fake desktop backend that moves focus on the first keystroke must halt. Issue [#1478](https://github.com/OpenAdaptAI/OpenAdapt/issues/1478). | +| 17 | Replay: occlusion | A structurally present but covered or off-viewport element returns the same value as a missing element, so resolution falls through to the pixel rungs (`backends/playwright_backend.py:2027-2048`, `runtime/resolver.py:767`). | A chat bubble or sticky banner covers the button's centre, and the template click lands on the overlay. | `test_structural_rung.py` has no partial occlusion. | OPEN | M × M-H | Real page with a fixed overlay on the button's centre. Issue [#1479](https://github.com/OpenAdaptAI/OpenAdapt/issues/1479). | +| 18 | Durable: deliveries outside a fence | A first, customer-local leg writes its checkpoint after delivery. A crash in between leaves no pause, so the only path is a fresh run from step 0. The local CLI builds no idempotency ledger. | The process is killed after Save. The re-run repeats every save. | Crashes are simulated with monkeypatch; no test kills a real process. | OPEN | M × H | Subprocess killed with SIGKILL at a random write, then re-run. Issue [#1480](https://github.com/OpenAdaptAI/OpenAdapt/issues/1480). | +| 19 | Capture: clocks | Input events use `time.time()`, and frames use a `perf_counter` offset (`input_observer/*.py`, `recorder.py:2030`). | An NTP step, or sleep and wake on macOS, makes action timestamps drift from frame timestamps. | The observer tests inject a fake clock. | OPEN (alignment) | M × M | Monkeypatch `time.time` to jump mid-recording. Not filed yet. | +| 20 | Capture: Windows | `IsWindowVisible` means the window is on screen. The region grab is what the window shows (`window_capture.py:2004, 2075-2096`). | A minimized window, or one on another virtual desktop. A toast or overlay over the window is recorded as the target's pixels. | No unit tests for the Windows resolver or grab. | OPEN | M × H | A resolver over injected Win32 attributes. Issue [#1477](https://github.com/OpenAdaptAI/OpenAdapt/issues/1477). | +| 21 | Capture: title binding | The title selector re-matches every frame. Linux, macOS and remote-display replay also re-find by exact title. | Chrome, LibreOffice and EMR windows retitle per record. | Static-title fakes. | CLOSED (stops) | H × L | Owned by the window-identity session. | +| 22 | Replay: settle timeout | Outside production profiles, a page that never settles proceeds with the latest frame (`vision/settle.py:304-313`). | Long-poll or server-sent-event requests keep the page "busy" forever. | `test_page_settle.py` covers a blocked settle, not the proceed path. | OPEN (demo profile only) | H × M | Real page with an open EventSource. Not filed yet. | + +### Lower-ranked gaps worth knowing + +- **Scrolling:** a scroll step wheels at the last pointer position. Over a + number input, that can change its value (`runtime/replayer.py:17193`). +- **macOS click mapping:** y is mapped with the x scale (`backends/macos_backend.py:811`). + This is up to about 9 points off at the bottom of a window with slightly + different x and y scales. +- **Windows non-ASCII typing:** text goes through the clipboard and the + clipboard is never restored, so accented names stay on it. +- **Single-surface workflows:** an unexpected new tab or popup is not checked. +- **Late duplicate writes:** record polling stops at the first CONFIRMED, so + a second row written a moment later is missed. +- **List walks:** identical rows on the same page are merged, and a row + inserted between page reads is not detected. +- **macOS:** trackpad scroll magnitudes are lost because only line deltas + are read. Windows low-level hooks can be removed without notice, and a + macOS permission revoked mid-recording is not detected. +- **Durable runs on macOS:** `fsync` without `F_FULLFSYNC` is not durable + across power loss. + +## What already fails closed, and was checked + +- **Structural ambiguity:** several DOM matches are refused at the structural, + template and OCR rungs. +- **Stale or re-rendered elements:** refused by the actuation token. +- **Frames:** the iframe chain is re-proven before delivery. +- **Browser input:** Playwright input is scoped to the page, so OS focus + changes, password-manager bubbles and permission prompts cannot redirect + keystrokes. +- **Window identity in capture:** bound to window id, process id and process + start time. Bounds are compared exactly before and after each grab. +- **Approvals:** bound to the exact pause digest, run, run directory and + bundle version. A rejected pause is terminal. +- **Checkpoint writes:** atomic (temporary file, fsync, replace). +- **OCR glyph confusion and ambiguous dates:** become INDETERMINATE. + +## Coordination + +The window-identity fix (rank 21) belongs to the session fixing the recorder +title bug. Ranks 3 and 12 touch the same file, +`packages/capture/openadapt_capture/window_capture.py`, and have been raised +with that session rather than changed here. + +## Test infrastructure notes + +- CI installs Chromium for the flow shards (`.github/workflows/packages.yml`), + so the browser tests here run on every flow change. +- Several browser test modules call `pytest.skip` when Chromium fails to + launch. In CI that skip should be a failure, or a broken browser install + silently drops those tests. +- The macOS, Windows, Linux, RDP and Citrix replay backends are tested only + against fake clients. None of the fakes change focus, titles, keyboard + layout or codec noise. A realistic test for those needs a real OS runner. diff --git a/packages/capture/tests/test_realism_capture.py b/packages/capture/tests/test_realism_capture.py new file mode 100644 index 0000000000..5a0477bc5c --- /dev/null +++ b/packages/capture/tests/test_realism_capture.py @@ -0,0 +1,45 @@ +"""Capture cases the single-window fakes do not model (REALISM_AUDIT.md). + +Real desktops have several windows of one app: two browser windows, two EMR +charts. When more than one window matches the target, binding the largest +one records whichever chart happens to be bigger. Every resolver raises +``WindowCaptureAmbiguousError`` here (#1468); these tests pin the rule on X11. +""" + +from __future__ import annotations + +import pytest + +from openadapt_capture.window_capture import WindowCaptureError, WindowTarget +from openadapt_capture.window_capture_linux import ( + LinuxWindowCaptureError, + X11WindowRecord, + resolve_window_linux, +) +from tests.test_window_capture_linux import FakeX11Client + + +@pytest.fixture +def x11_environment(monkeypatch): + monkeypatch.setenv("DISPLAY", ":99") + monkeypatch.setenv("XDG_SESSION_TYPE", "x11") + monkeypatch.delenv("WAYLAND_DISPLAY", raising=False) + + +def _window(window_id: int, title: str, bounds) -> X11WindowRecord: + return X11WindowRecord(window_id=window_id, title=title, pid=1000, bounds=bounds, viewable=True) + + +def test_two_matching_browser_windows_are_ambiguous(x11_environment) -> None: + client = FakeX11Client( + [ + _window(41, "Chart: Patient A - Chromium", (0, 0, 900, 700)), + _window(42, "Chart: Patient B - Chromium", (40, 40, 1200, 800)), + ] + ) + with pytest.raises((WindowCaptureError, LinuxWindowCaptureError)): + resolve_window_linux( + WindowTarget(owner="chromium", title="Chart"), + client_factory=lambda: client, + process_identity=lambda _pid: ("chromium", 100.0), + ) diff --git a/packages/flow/tests/test_realism_browser.py b/packages/flow/tests/test_realism_browser.py new file mode 100644 index 0000000000..9c9623c838 --- /dev/null +++ b/packages/flow/tests/test_realism_browser.py @@ -0,0 +1,318 @@ +"""Real-browser cases the fakes do not model (docs/audits/REALISM_AUDIT.md). + +Each test records a short task on a local web app in headless Chromium, +compiles it, then replays it after the page changes in a way real apps do: + +* the field the task types into already holds text (form restore, autofill, + a search box that keeps its last query); +* the app raises a ``confirm()`` the recording never saw; +* two worklist rows show the same visible text and swap order; +* the page retitles on every navigation. + +Every replay must end in one of two ways: the run halts, or it did exactly +what was recorded. A run that reports success after doing something else is +the failure these tests exist to catch. +""" + +from __future__ import annotations + +import threading +from collections.abc import Iterator +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer +from pathlib import Path +from urllib.parse import parse_qs, urlparse + +import pytest + +pytest.importorskip("playwright.sync_api") + +from openadapt_flow.backends.playwright_backend import PlaywrightBackend # noqa: E402 +from openadapt_flow.compiler import compile_recording # noqa: E402 +from openadapt_flow.ir import Workflow # noqa: E402 +from openadapt_flow.recorder import Recorder # noqa: E402 +from openadapt_flow.runtime.replayer import Replayer # noqa: E402 + +pytestmark = pytest.mark.timeout(600) + +VIEWPORT = {"width": 900, "height": 600} + +_STYLE = """ +body { font-family: sans-serif; margin: 24px; font-size: 18px; } +input, button { font-size: 18px; padding: 6px; margin: 4px; } +li { padding: 10px; border: 1px solid #888; margin: 6px; list-style: none; + cursor: pointer; width: 420px; } +#status { margin-top: 16px; font-weight: bold; } +""" + + +def _form(query: dict[str, list[str]]) -> str: + prefill = query.get("prefill", [""])[0] + ask = "confirm" in query + return f"""New note + +

New note

+

+

Clinic intake form, draft copy +

+
Not saved
+""" + + +def _worklist(query: dict[str, list[str]]) -> str: + order = query.get("order", ["ab"])[0] + rows = { + "a": ("1001", "John Smith Follow-up Dr Lee"), + "b": ("1002", "John Smith Follow-up Dr Lee"), + "c": ("1003", "Maria Garcia Intake Dr Lee"), + } + items = "".join( + f'
  • {rows[k][1]}
    • ' + for k in order + "c" + ) + return f"""Worklist + +

    Worklist

    No chart open
    + +
    Nothing signed
    +""" + + +class _Handler(BaseHTTPRequestHandler): + def do_GET(self) -> None: # noqa: N802 + parsed = urlparse(self.path) + query = parse_qs(parsed.query) + page = {"/form.html": _form, "/worklist.html": _worklist}.get(parsed.path) + if page is None: + self.send_error(404) + return + body = page(query).encode() + self.send_response(200) + self.send_header("Content-Type", "text/html; charset=utf-8") + self.send_header("Content-Length", str(len(body))) + self.end_headers() + self.wfile.write(body) + + def log_message(self, *_args: object) -> None: + pass + + +@pytest.fixture(scope="module") +def app_url() -> Iterator[str]: + server = ThreadingHTTPServer(("127.0.0.1", 0), _Handler) + threading.Thread(target=server.serve_forever, daemon=True).start() + yield f"http://127.0.0.1:{server.server_address[1]}" + server.shutdown() + + +@pytest.fixture(scope="module") +def browser() -> Iterator[object]: + from openadapt_flow._browser_setup import ensure_chromium_installed + + ensure_chromium_installed() + from playwright.sync_api import sync_playwright + + with sync_playwright() as pw: + chromium = pw.chromium.launch(headless=True) + yield chromium + chromium.close() + + +def _page(browser: object, url: str): + page = browser.new_page(viewport=VIEWPORT, device_scale_factor=1) # type: ignore[attr-defined] + page.goto(url) + page.wait_for_load_state("load") + page.wait_for_timeout(150) + return page + + +def _center(page, selector: str) -> tuple[int, int]: + box = page.locator(selector).first.bounding_box() + assert box is not None, selector + return int(box["x"] + box["width"] / 2), int(box["y"] + box["height"] / 2) + + +def _recorder(page, root: Path, url: str) -> Recorder: + return Recorder( + PlaywrightBackend(page), + root / "recording", + app_url=url, + settle_interval_s=0.05, + settle_timeout_s=2.0, + ) + + +def _replay(browser, url, bundle_dir: Path, run_dir: Path, params=None): + page = _page(browser, url) + report = Replayer(PlaywrightBackend(page), poll_interval_s=0.05).run( + Workflow.load(bundle_dir), + params=params or {}, + bundle_dir=bundle_dir, + run_dir=run_dir, + ) + return report, page + + +def _summary(report) -> str: + return "\n".join(f"{r.step_id} ok={r.ok} err={r.error}" for r in report.results) + + +@pytest.fixture(scope="module") +def note_bundle(app_url, browser, tmp_path_factory) -> Path: + root = tmp_path_factory.mktemp("note") + url = app_url + "/form.html" + page = _page(browser, url) + try: + recorder = _recorder(page, root, url) + recorder.click(*_center(page, "#name")) + recorder.type_text("Jane Doe", param="patient") + recorder.click(*_center(page, "#save")) + assert page.locator("#status").inner_text() == "Saved Jane Doe" + recording = recorder.finish() + finally: + page.close() + compile_recording(recording, root / "bundle", name="note") + return root / "bundle" + + +def test_note_replays_on_the_page_it_was_recorded_on( + note_bundle, app_url, browser, tmp_path +) -> None: + """Control: an unchanged page with a new parameter saves the new value.""" + + report, page = _replay( + browser, + app_url + "/form.html", + note_bundle, + tmp_path / "run", + params={"patient": "Ada"}, + ) + try: + assert report.success, _summary(report) + assert page.locator("#status").inner_text() == "Saved Ada" + finally: + page.close() + + +def test_a_field_autofilled_on_focus_never_saves_a_concatenation( + note_bundle, app_url, browser, tmp_path +) -> None: + """The field fills itself when focused (autofill, a restored draft). + + Typing appends. A run that reports success with "John SmithAda" saved + has written a wrong value silently. Today the click's region check halts + the run before typing; this pins that. + """ + + report, page = _replay( + browser, + app_url + "/form.html?prefill=John%20Smith", + note_bundle, + tmp_path / "run", + params={"patient": "Ada"}, + ) + try: + status = page.locator("#status").inner_text() + if report.success: + assert status == "Saved Ada", _summary(report) + else: + assert status in {"Not saved", "Saved Ada"}, status + finally: + page.close() + + +def test_an_unrecorded_confirm_dialog_never_ends_in_success( + note_bundle, app_url, browser, tmp_path +) -> None: + """Save raises a ``confirm()`` the recording never saw. + + The browser dismisses it, so nothing is saved. The run must not report + success on a save that did not happen. + """ + + report, page = _replay( + browser, + app_url + "/form.html?confirm=1", + note_bundle, + tmp_path / "run", + params={"patient": "Ada"}, + ) + try: + status = page.locator("#status").inner_text() + assert not (report.success and status == "Not saved"), _summary(report) + finally: + page.close() + + +@pytest.fixture(scope="module") +def worklist_bundle(app_url, browser, tmp_path_factory) -> Path: + root = tmp_path_factory.mktemp("worklist") + url = app_url + "/worklist.html" + page = _page(browser, url) + try: + recorder = _recorder(page, root, url) + recorder.click(*_center(page, "li[data-id='1001']")) + assert page.locator("#status").inner_text() == "Opened chart 1001" + recorder.click(*_center(page, "#sign")) + assert page.locator("#signed").inner_text() == "Signed chart 1001" + recording = recorder.finish() + finally: + page.close() + compile_recording(recording, root / "bundle", name="worklist") + return root / "bundle" + + +@pytest.mark.xfail( + strict=True, + reason=( + "REALISM_AUDIT identity-8: identity is verified per row, never for " + "uniqueness, so a swapped homonym row is opened and signed" + ), +) +def test_identical_rows_that_swap_order_never_sign_the_other_chart( + worklist_bundle, app_url, browser, tmp_path +) -> None: + """Two rows show the same text; at replay time they have swapped. + + The task opens chart 1001 and signs its note. Signing chart 1002 and + reporting success acts on the wrong record. The run must halt (the + target is ambiguous) or sign 1001. + """ + + report, page = _replay( + browser, + app_url + "/worklist.html?order=ba", + worklist_bundle, + tmp_path / "run", + ) + try: + status = page.locator("#signed").inner_text() + assert not (report.success and status == "Signed chart 1002"), _summary(report) + finally: + page.close() diff --git a/packages/flow/tests/test_realism_readback_values.py b/packages/flow/tests/test_realism_readback_values.py new file mode 100644 index 0000000000..9f9e27c739 --- /dev/null +++ b/packages/flow/tests/test_realism_readback_values.py @@ -0,0 +1,68 @@ +"""Read-back value comparison against neighbouring values (REALISM_AUDIT.md). + +The on-screen read-back compares the expected value with OCR text from the +field's region using ``exact=False``. The tests next to it check that a +value inside a label ("Amount: 250") still confirms. They never check a +neighbouring value that contains the expected one as a substring: "$1,250.00" +when 250 was expected, "Qty 15" when 5 was. Those must not confirm a write. +""" + +from __future__ import annotations + +import pytest +from hypothesis import assume, given, settings +from hypothesis import strategies as st + +from openadapt_flow.runtime.effects.onscreen import value_matches +from openadapt_flow.runtime.effects.readback_values import compare_value + +_AUDIT = "REALISM_AUDIT identity-3: non-exact read-back matches a substring" + + +def test_a_value_inside_its_label_still_confirms() -> None: + """Control: the case the substring match exists for.""" + + assert compare_value("Amount: 250", "250", exact=False) == "match" + + +@pytest.mark.xfail(strict=True, reason=_AUDIT) +@settings(max_examples=200, deadline=None) +@given(expected=st.integers(1, 99_999), observed=st.integers(1, 9_999_999)) +def test_a_different_number_never_confirms(expected: int, observed: int) -> None: + assume(expected != observed and str(expected) in str(observed)) + for shown in (f"{observed}", f"${observed:,}.00", f"Qty {observed}"): + assert compare_value(shown, str(expected), exact=False) != "match", shown + + +@pytest.mark.xfail(strict=True, reason=_AUDIT) +@pytest.mark.parametrize( + ("shown", "expected", "kind"), + [ + ("$1,250.00", "250", "text"), + ("mrn 123456", "12345", "code"), + ("Qty 15", "5", "text"), + ("100.00-", "100.00", "currency"), + ("$100.00 CR", "100.00", "currency"), + ], +) +def test_neighbouring_values_do_not_confirm(shown: str, expected: str, kind) -> None: + assert compare_value(shown, expected, kind=kind, exact=False) != "match" + + +@pytest.mark.xfail( + strict=True, reason="REALISM_AUDIT identity-4: fuzzy text match ignores the end" +) +@pytest.mark.parametrize( + ("shown", "expected"), + [ + ( + "follow-up scheduled for 2026-10-29", + "Follow-up scheduled for 2026-10-21", + ), + ("patient: john smithson", "John Smith"), + ], +) +def test_fuzzy_text_does_not_confirm_a_different_value( + shown: str, expected: str +) -> None: + assert not value_matches(shown, expected)