fix(flow): halt when another shown row has the same identity

CAPABILITIES.md

@@ -186,6 +186,7 @@ improvising or retrying when something is off.
 | Chain programs across surfaces, passing confirmed values | Released | Synthetic | `openadapt-flow compose`, `process`; [PROCESS_CONTRACT.md](packages/flow/docs/PROCESS_CONTRACT.md) |
 | One program that switches surfaces, tabs and popups mid-run | On main ([#1197](https://github.com/OpenAdaptAI/OpenAdapt/pull/1197)) | Real Chromium; Windows switch on a mock | [#1197](https://github.com/OpenAdaptAI/OpenAdapt/pull/1197) |
 | Halt when a workflow qualified with page structure runs on a pixels-only surface | Released | Synthetic | `openadapt-flow qualify` |
+| Stop before acting when another shown row has the same identity; when the rows carry an app-owned record id (a `data-*` id, the row link), act on the row with the recorded id; on a surface that cannot count the rows, stop before any step that leads into a write | On main ([#1504](https://github.com/OpenAdaptAI/OpenAdapt/pull/1504)) | Real browser (headless Chromium worklist with identical rows) | `runtime/identity_census.py`; `PlaywrightBackend.identity_candidates_at` |
 | Lift look-alike identity stops with a complete known-ID list | Released | Synthetic (rendered record list) | `openadapt-flow known-ids` |
 | Take extra looks before stopping on a look-alike record ID: fresh frames, the known-ID list, and a select-and-copy of the ID where the surface allows it | On main ([#1333](https://github.com/OpenAdaptAI/OpenAdapt/pull/1333)) | Synthetic (dense record list in a browser) | `runtime.identity_copy_look` in deployment.yaml; [IDENTITY_LOOKS.md](packages/flow/benchmark/identity_looks/IDENTITY_LOOKS.md) |
 | Check an icon-only click target's hover response (highlight, tooltip) before clicking on a pixel surface, and re-check the target and identity regions on a fresh frame right before every pixel click | On main ([#1334](https://github.com/OpenAdaptAI/OpenAdapt/pull/1334)) | Synthetic (in-process kiosk) | `Recorder(capture_hover=True)`; [hover_identity](packages/flow/benchmark/hover_identity/README.md) |

llms-full.txt

@@ -583,6 +583,7 @@ improvising or retrying when something is off.
 | Chain programs across surfaces, passing confirmed values | Released | Synthetic | `openadapt-flow compose`, `process`; [PROCESS_CONTRACT.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/PROCESS_CONTRACT.md) |
 | One program that switches surfaces, tabs and popups mid-run | On main ([#1197](https://github.com/OpenAdaptAI/OpenAdapt/pull/1197)) | Real Chromium; Windows switch on a mock | [#1197](https://github.com/OpenAdaptAI/OpenAdapt/pull/1197) |
 | Halt when a workflow qualified with page structure runs on a pixels-only surface | Released | Synthetic | `openadapt-flow qualify` |
+| Stop before acting when another shown row has the same identity; when the rows carry an app-owned record id (a `data-*` id, the row link), act on the row with the recorded id; on a surface that cannot count the rows, stop before any step that leads into a write | On main ([#1504](https://github.com/OpenAdaptAI/OpenAdapt/pull/1504)) | Real browser (headless Chromium worklist with identical rows) | `runtime/identity_census.py`; `PlaywrightBackend.identity_candidates_at` |
 | Lift look-alike identity stops with a complete known-ID list | Released | Synthetic (rendered record list) | `openadapt-flow known-ids` |
 | Take extra looks before stopping on a look-alike record ID: fresh frames, the known-ID list, and a select-and-copy of the ID where the surface allows it | On main ([#1333](https://github.com/OpenAdaptAI/OpenAdapt/pull/1333)) | Synthetic (dense record list in a browser) | `runtime.identity_copy_look` in deployment.yaml; [IDENTITY_LOOKS.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/identity_looks/IDENTITY_LOOKS.md) |
 | Check an icon-only click target's hover response (highlight, tooltip) before clicking on a pixel surface, and re-check the target and identity regions on a fresh frame right before every pixel click | On main ([#1334](https://github.com/OpenAdaptAI/OpenAdapt/pull/1334)) | Synthetic (in-process kiosk) | `Recorder(capture_hover=True)`; [hover_identity](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/hover_identity/README.md) |

packages/flow/openadapt_flow/backend.py

@@ -218,6 +218,35 @@ def structured_text_at(self, x: int, y: int) -> Optional[str]:
         ...
 
 
+@runtime_checkable
+class IdentityCandidatesBackend(Protocol):
+    """Optional census of rows that share the target's record identity.
+
+    ``structured_text_at`` says what the row under a point is. It cannot say
+    whether that text picks one record: two worklist rows that both read
+    "John Smith Follow-up Dr Lee" are each an exact match for the recorded
+    identity, and after the rows swap order the resolver lands on the other
+    patient. ``identity_candidates_at`` counts every shown row whose identity
+    equals the identity under the point and reads each row's stable record id
+    (app-owned ``data-*`` ids, the row link's path). The runtime acts only
+    when the target is the one shown row with that identity, or when the
+    recorded record id picks exactly one of them.
+
+    Returns a dict ``{"record_key": str | None, "incomplete": bool,
+    "candidates": [{"self": bool, "record_key": str | None, "dx": float,
+    "dy": float}, ...], "top_level": bool}``, where ``dx``/``dy`` are each
+    row's offset from the row under the point in click coordinates;
+    ``{"row": False}`` when the point has no row identity (a standalone
+    control); or None when the census failed (never raises). A backend without
+    this method cannot count rows, and the runtime halts a step that leads
+    into a consequential write rather than assume the row is unique.
+    """
+
+    def identity_candidates_at(self, x: int, y: int) -> Optional[dict[str, Any]]:
+        """Census of the shown rows with the same identity as (x, y)."""
+        ...
+
+
 @runtime_checkable
 class FieldLabelBackend(Protocol):
     """Optional RECORD-TIME field-label capability a backend MAY expose.

packages/flow/openadapt_flow/backends/playwright_backend.py

@@ -700,6 +700,138 @@ class _FramePoint:
 }"""
 )
 
+# The stable record id a row carries in the DOM: app-owned ``data-*`` id
+# attributes on the target, its ancestors up to the row, and the row, plus the
+# path and id-like query parameters of the row's link. Index-like attribute
+# names, and values that are only the row's position, are left out: a key that
+# moves with the row would point at whichever record now sits there. Null when
+# the row exposes no such id.
+_RECORD_KEY_JS = r"""(row, node) => {
+    const ID_NAME = /^data-(?:[a-z0-9_-]*-)?(?:id|key|uid|uuid|guid|mrn|pid|record|recordid)$/i;
+    const POSITIONAL = /(?:index|idx|position|pos|order|rank|seq)/i;
+    const parent = row.parentElement;
+    const index = parent ? Array.prototype.indexOf.call(parent.children, row) : -1;
+    const positional = (value) => {
+        const m = /^\D*(\d+)\D*$/.exec(value);
+        if (!m) return false;
+        const n = Number(m[1]);
+        return n === index || n === index + 1;
+    };
+    const parts = [];
+    let cursor = node;
+    for (let depth = 0; cursor && depth < 12; depth += 1) {
+        for (const attr of Array.from(cursor.attributes || [])) {
+            const name = attr.name.toLowerCase();
+            if (name.startsWith('data-openadapt-actuation-')) continue;
+            if (name !== 'data-openadapt-record' &&
+                    (!ID_NAME.test(name) || POSITIONAL.test(name))) continue;
+            const value = (attr.value || '').trim();
+            if (!value || value.length > 256 || positional(value)) continue;
+            parts.push(name + '=' + value);
+        }
+        if (cursor === row) break;
+        cursor = cursor.parentElement;
+    }
+    let link = node.closest('a[href]');
+    if (!link || !row.contains(link)) {
+        const links = row.querySelectorAll('a[href]');
+        link = row.matches('a[href]') ? row : (links.length === 1 ? links[0] : null);
+    }
+    if (link) {
+        try {
+            const url = new URL(link.getAttribute('href'), location.href);
+            if (url.protocol === 'http:' || url.protocol === 'https:') {
+                const ids = [];
+                for (const [k, v] of url.searchParams) {
+                    if (/(?:^|[_-])(?:id|key|uid|uuid|guid|mrn|pid)$/i.test(k) ||
+                            /^(?:id|pid|mrn)$/i.test(k)) ids.push(k + '=' + v);
+                }
+                ids.sort();
+                const here = url.pathname === location.pathname && !ids.length;
+                const value = url.pathname + (ids.length ? '?' + ids.join('&') : '') + url.hash;
+                if (!(here && !url.hash) && value.length <= 512 && !positional(value)) {
+                    parts.push('href=' + value);
+                }
+            }
+        } catch (e) {}
+    }
+    if (!parts.length) return null;
+    return Array.from(new Set(parts)).sort().join('\n');
+}"""
+
+# Every shown row whose record identity equals the identity of the row under
+# the point, with each row's stable record id and position. The identity is
+# computed exactly as ``structured_text_at`` does, for the cell at the same
+# place in each row. ``{row: false}`` when the point has no row identity (a
+# form field group, a standalone control): there is no row to compare.
+_IDENTITY_CANDIDATES_AT_JS = (
+    r"""([px, py, cap]) => {
+    const el = document.elementFromPoint(px, py);
+    if (!el) return null;
+    const describe = """
+    + _DESCRIBE_TARGET_JS
+    + r""";
+    const recordKey = """
+    + _RECORD_KEY_JS
+    + r""";
+    const target = describe(el);
+    if (!target.rowIdentity || !target.row) return {row: false};
+    const row = target.row;
+    const own = el.closest('td, th, [role="cell"], [role="gridcell"]') || el;
+    const path = [];
+    let cursor = own;
+    while (cursor && cursor !== row) {
+        const parent = cursor.parentElement;
+        if (!parent) break;
+        path.unshift(Array.prototype.indexOf.call(parent.children, cursor));
+        cursor = parent;
+    }
+    const ownInRow = cursor === row;
+    const shown = (node) => {
+        if (!node.isConnected) return false;
+        if (typeof node.checkVisibility === 'function') {
+            return node.checkVisibility({visibilityProperty: true, opacityProperty: true});
+        }
+        if (!node.getClientRects().length) return false;
+        const style = window.getComputedStyle(node);
+        return style.visibility !== 'hidden' && style.display !== 'none';
+    };
+    const rows = document.querySelectorAll('tr, [role="row"], li, [role="listitem"]');
+    const rowRect = row.getBoundingClientRect();
+    const result = {
+        record_key: recordKey(row, el),
+        incomplete: rows.length > cap,
+        candidates: [],
+    };
+    if (result.incomplete) return result;
+    for (const candidate of rows) {
+        if (candidate !== row && !shown(candidate)) continue;
+        let node = candidate;
+        if (ownInRow) {
+            for (const index of path) node = node && node.children[index];
+        }
+        if (!node) continue;
+        const observed = candidate === row ? target : describe(node);
+        if (observed.row !== candidate || observed.rowIdentity !== target.rowIdentity) {
+            continue;
+        }
+        const rect = candidate.getBoundingClientRect();
+        result.candidates.push({
+            self: candidate === row,
+            record_key: candidate === row ? result.record_key : recordKey(candidate, node),
+            dx: rect.x - rowRect.x,
+            dy: rect.y - rowRect.y,
+        });
+    }
+    return result;
+}"""
+)
+
+# Rows scanned by the candidate census before it gives up. A document with
+# more rows than this reports ``incomplete`` and the runtime cannot prove the
+# target unique.
+IDENTITY_CANDIDATE_ROW_CAP = 5000
+
 _TEXT_VALUE_AT_JS = r"""([px, py]) => {
     const hit = document.elementFromPoint(px, py);
     if (!hit) return null;
@@ -1640,6 +1772,42 @@ def structured_text_at(self, x: int, y: int) -> Optional[str]:
             return None
         return result or None
 
+    def identity_candidates_at(self, x: int, y: int) -> Optional[dict[str, Any]]:
+        """Census of the rows that show the same record identity as (x, y).
+
+        ``structured_text_at`` says what the row under the point is; this says
+        whether any OTHER shown row says the same thing. Two worklist rows
+        reading "John Smith Follow-up Dr Lee" are both a verified match for
+        the recorded text, so the text alone cannot pick the recorded record
+        (see :class:`openadapt_flow.backend.IdentityCandidatesBackend`).
+
+        Returns ``{"record_key", "incomplete", "candidates"}``: the stable
+        record id of the row under the point (app-owned ``data-*`` ids and the
+        row link's path, or None), whether the document had too many rows to
+        scan, and one entry per shown row with the same identity (the row
+        under the point included, ``self`` true) carrying its record id and
+        its offset ``(dx, dy)`` from the row under the point, in viewport CSS
+        pixels. ``{"row": False}`` when the point has no row identity (a
+        standalone control, a form field); None when the census failed.
+        """
+        try:
+            point = self._frame_point(int(x), int(y))
+            if point is None:
+                return None
+            result = point.scope.evaluate(
+                _IDENTITY_CANDIDATES_AT_JS,
+                [point.x, point.y, IDENTITY_CANDIDATE_ROW_CAP],
+            )
+        except Exception:
+            return None
+        if not isinstance(result, dict):
+            return None
+        # Offsets are in the hit frame's own pixels. Outside the top-level
+        # document they cannot be mapped back with certainty, so a retarget
+        # is not offered there; the count still holds.
+        result["top_level"] = not point.frame_path
+        return result
+
     def text_value_at(self, x: int, y: int) -> Optional[str]:
         """Return the exact value of the editable control under ``(x, y)``.
 

packages/flow/openadapt_flow/compiler/compile.py

@@ -2274,6 +2274,15 @@ def cached_lines(i: int, suffix: str, png: bytes) -> list[OcrLine]:
             # OCR context band; replay prefers it (no OCR glyph ambiguity) and
             # falls back to the band on pixel-only substrates.
             structured_identity = event.get("structured_identity") or None
+            # The row's stable record id (a DOM data-* id, the row link's
+            # path), captured with the structured identity. Hashed into the
+            # identity template so replay acts only on the row with this id.
+            record_key = (
+                event.get("record_key") or None if structured_identity else None
+            )
+            structured_row = event.get("structured_row")
+            if not structured_identity or not isinstance(structured_row, bool):
+                structured_row = None
             # Structural locator (DOM selector / role+name, or UIA identifiers)
             # of the clicked element, captured by the recorder when the
             # recording backend exposed it (StructuralActionBackend). Drives the
@@ -2381,6 +2390,8 @@ def cached_lines(i: int, suffix: str, png: bytes) -> list[OcrLine]:
                 ocr_text=anchor_ocr_text,
                 context_text=context_text,
                 structured_identity=structured_identity,
+                record_key=record_key,
+                structured_row=structured_row,
                 structural=structural,
                 landmarks=anchor_landmarks,
                 identifier_crop=identifier_crop_rel,
@@ -3590,9 +3601,13 @@ def _phi_free_identity(workflow: Workflow) -> None:
             structured_identity=anchor.structured_identity,
             param_examples=workflow.params,
             salt_hex=salt,
+            record_key=anchor.record_key,
+            structured_row=anchor.structured_row,
         )
         if template is None:
             continue
         anchor.identity_template = template
         anchor.context_text = None
         anchor.structured_identity = None
+        anchor.record_key = None
+        anchor.structured_row = None