Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
8021adf
feat(flow): sign a person's answer over a pipe (--approver-signer stdio)
abrichr Oct 24, 2026
01bc599
feat(desktop): set up the approver key in the app, behind Touch ID
abrichr Oct 24, 2026
3e6e98b
docs: list the Touch ID approver key in CAPABILITIES
abrichr Oct 24, 2026
77fa76a
Merge remote-tracking branch 'origin/main' into feat/stop-review-queue
abrichr Oct 24, 2026
d5d650e
feat(desktop): review stops as one-tap cards with evidence side by side
abrichr Oct 24, 2026
28fe7ef
style: format the stop queue tests
abrichr Oct 24, 2026
5bc04d1
feat(desktop): answer the stop-review queue from a paired phone
abrichr Oct 25, 2026
8d7dc9d
docs(desktop): describe the phone stop queue and remote access
abrichr Oct 25, 2026
760d6b6
feat(desktop): compare each possible match field by field; measure stops
abrichr Oct 25, 2026
0279299
Merge remote-tracking branch 'origin/feat/stop-queue-phone' into feat…
abrichr Oct 25, 2026
45e2b94
feat(desktop): phone shows matches field by field; close stops while …
abrichr Oct 25, 2026
e109de5
docs: link the stop queue's capability row to its PR
abrichr Oct 25, 2026
440f702
feat(desktop): re-arm "only ask about problems" from a stop card
abrichr Oct 25, 2026
99e9981
fix(desktop): a locked Mac isn't a Mac without a Secure Enclave
abrichr Oct 25, 2026
4c8c514
Merge remote-tracking branch 'origin/fix/signed-human-approval' into …
abrichr Oct 25, 2026
21fa218
fix(desktop): presence before re-registering; no file-key fallback on…
abrichr Oct 25, 2026
343d2d3
Merge remote-tracking branch 'origin/feat/desktop-keychain-approver' …
abrichr Oct 25, 2026
4512c91
fix(desktop): re-check stop cards after Touch ID; no file-key fallback
abrichr Oct 25, 2026
a12ab81
Merge the signed approver roster (#1486 fix) into the stop queue
abrichr Oct 25, 2026
b675522
wip: desktop pin precedence and roster head (squashed into the next c…
abrichr Oct 25, 2026
dc03478
Merge remote-tracking branch 'origin/fix/signed-human-approval' into …
abrichr Oct 25, 2026
95b3e93
fix(desktop): the Keychain pin wins; each run's config carries it; no…
abrichr Oct 25, 2026
719c0c7
fix(desktop): pass the Keychain head to Flow; stage configs outside r…
abrichr Oct 25, 2026
db37048
Merge remote-tracking branch 'origin/fix/signed-human-approval' into …
abrichr Oct 25, 2026
bb671b1
test(flow): stdio signer tests pass their pin explicitly
abrichr Oct 25, 2026
eced3b1
fix(desktop): config digest, head on answers, no env root, no symlink…
abrichr Oct 25, 2026
0d188fb
Merge remote-tracking branch 'origin/fix/signed-human-approval' into …
abrichr Oct 25, 2026
70184d3
fix(desktop): only the Keychain names this Mac's approver root
abrichr Oct 25, 2026
beb1d2f
fix(desktop): no Keychain pin means no root reaches Flow
abrichr Oct 25, 2026
79cfa54
fix(desktop): with no Keychain pin, hand Flow no approvers section
abrichr Oct 25, 2026
7dbbc98
Merge remote-tracking branch 'origin/main' into feat/stop-review-queue
abrichr Oct 25, 2026
c0a8307
Merge remote-tracking branch 'origin/main' into feat/stop-review-queue
abrichr Oct 25, 2026
83d1fb7
test(desktop): the citrix governed-run bridge takes the config digest
abrichr Oct 25, 2026
c7a9a1e
Merge remote-tracking branch 'origin/main' into feat/stop-review-queue
abrichr Oct 25, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CAPABILITIES.md
Original file line number Diff line number Diff line change
Expand Up @@ -139,6 +139,8 @@ improvising or retrying when something is off.
| Admit a record check only after a fault suite written to a store copy shows it catches every fault it claims, and sign its coverage into the receipt | On main | Synthetic (in-memory REST store, SQLite copy) | `openadapt_flow/fault_gate.py`, `scaffold-verifier --fault-gate`, `verify-run` `admission:` |
| Check that the saved record matches its source document, not only the parameters: intake keeps each value's line, a second reading must agree before a guarded save, and a round trip compares the read-back with the document | On main | Synthetic (MockMed, text referral) | `openadapt_flow/provenance.py`, `intake`, `verify-run --provenance`, guarded save |
| Guarded save: a safety layer any agent wraps around one write (check the record, hold for approval, read back) | On main ([#1258](https://github.com/OpenAdaptAI/OpenAdapt/pull/1258)) | Synthetic | `GUARDED_SAVE.md` |
| A person's approval in Desktop is signed with a key kept behind Touch ID on a Mac: one-prompt setup in the app, a prompt for every answer, and Flow never sees the key | On main ([#1500](https://github.com/OpenAdaptAI/OpenAdapt/pull/1500)) | Unit tests; opt-in test on a real Secure Enclave | `packages/desktop/openadapt_desktop/approver_keychain.py`, [APPROVAL_MODES.md](packages/flow/docs/APPROVAL_MODES.md#the-approver-key) |
| Review stops: every stop as one card with the evidence side by side (the item next to each possible match, field by field), answered with one key or tap on the computer or a paired phone, signed with the approver key, with batches for similar stops. The queue presents and collects; a changed or expired card changes nothing | On main ([#1519](https://github.com/OpenAdaptAI/OpenAdapt/pull/1519)) | Unit tests; a keystroke-level estimate on 20 soak stops (26.7 s to 8.2 s to decide, a model, not a person) | `packages/desktop/openadapt_desktop/stop_queue.py`, [benchmark/stop_queue](packages/desktop/benchmark/stop_queue/README.md) |
| Draft a record check from the demonstration's write steps (`scaffold-verifier`) | Released | Synthetic | `openadapt-flow scaffold-verifier` (a person fills in the draft) |
| Build a working record check from the reads the app made while the person viewed the saved record, and test it against injected bad writes before anyone reviews it | On main ([#1284](https://github.com/OpenAdaptAI/OpenAdapt/pull/1284)) | Synthetic (MockMed, OpenEMR fixture) | `openadapt-flow scaffold-verifier` |
| Check a save on desktop, RDP or Citrix by looking the record up in a separate read-only session | On main ([#1280](https://github.com/OpenAdaptAI/OpenAdapt/pull/1280)) | Synthetic (WinForms, RDP and Citrix stand-ins) | `runtime/effects/` |
Expand Down
2 changes: 2 additions & 0 deletions llms-full.txt
Original file line number Diff line number Diff line change
Expand Up @@ -536,6 +536,8 @@ improvising or retrying when something is off.
| Admit a record check only after a fault suite written to a store copy shows it catches every fault it claims, and sign its coverage into the receipt | On main | Synthetic (in-memory REST store, SQLite copy) | `openadapt_flow/fault_gate.py`, `scaffold-verifier --fault-gate`, `verify-run` `admission:` |
| Check that the saved record matches its source document, not only the parameters: intake keeps each value's line, a second reading must agree before a guarded save, and a round trip compares the read-back with the document | On main | Synthetic (MockMed, text referral) | `openadapt_flow/provenance.py`, `intake`, `verify-run --provenance`, guarded save |
| Guarded save: a safety layer any agent wraps around one write (check the record, hold for approval, read back) | On main ([#1258](https://github.com/OpenAdaptAI/OpenAdapt/pull/1258)) | Synthetic | `GUARDED_SAVE.md` |
| A person's approval in Desktop is signed with a key kept behind Touch ID on a Mac: one-prompt setup in the app, a prompt for every answer, and Flow never sees the key | On main ([#1500](https://github.com/OpenAdaptAI/OpenAdapt/pull/1500)) | Unit tests; opt-in test on a real Secure Enclave | `packages/desktop/openadapt_desktop/approver_keychain.py`, [APPROVAL_MODES.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/APPROVAL_MODES.md#the-approver-key) |
| Review stops: every stop as one card with the evidence side by side (the item next to each possible match, field by field), answered with one key or tap on the computer or a paired phone, signed with the approver key, with batches for similar stops. The queue presents and collects; a changed or expired card changes nothing | On main ([#1519](https://github.com/OpenAdaptAI/OpenAdapt/pull/1519)) | Unit tests; a keystroke-level estimate on 20 soak stops (26.7 s to 8.2 s to decide, a model, not a person) | `packages/desktop/openadapt_desktop/stop_queue.py`, [benchmark/stop_queue](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/desktop/benchmark/stop_queue/README.md) |
| Draft a record check from the demonstration's write steps (`scaffold-verifier`) | Released | Synthetic | `openadapt-flow scaffold-verifier` (a person fills in the draft) |
| Build a working record check from the reads the app made while the person viewed the saved record, and test it against injected bad writes before anyone reviews it | On main ([#1284](https://github.com/OpenAdaptAI/OpenAdapt/pull/1284)) | Synthetic (MockMed, OpenEMR fixture) | `openadapt-flow scaffold-verifier` |
| Check a save on desktop, RDP or Citrix by looking the record up in a separate read-only session | On main ([#1280](https://github.com/OpenAdaptAI/OpenAdapt/pull/1280)) | Synthetic (WinForms, RDP and Citrix stand-ins) | `runtime/effects/` |
Expand Down
25 changes: 25 additions & 0 deletions packages/desktop/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,31 @@
# CHANGELOG


## v0.24.0 (2026-10-24)

### Features

- **desktop**: set up the approver key in the app, behind Touch ID
- **desktop**: review stops as one-tap cards with the evidence side by side,
on the computer (keys 1-9, batches of similar stops) or a paired phone; every
answer is signed with the approver key
- **desktop**: re-arm "only ask about problems" from a stop card with one
Touch ID

### Bug Fixes

- **desktop**: a locked Mac isn't a Mac without a Secure Enclave
- **desktop**: the Touch ID setup signs the approver roster's root behind its one prompt and pins that root for Desktop's runs; a key rotation is signed by the old key, and a roster this Mac didn't start needs an approver to add the key
- **desktop**: re-register a stored approver key only after it opens behind Touch ID
- **desktop**: Flow checks each run's config against the digest Desktop computed (--config-sha256); Run it answers carry the Keychain's head; staged-config folders that are symlinks are refused; only the Keychain names Desktop's approver root (not OPENADAPT_APPROVER_ROOT, not a user-writable file); with no Keychain pin, no person's answer counts
- **desktop**: each run's deployment config also carries the Keychain's roster head and is staged outside the run directory; Run it answers name this Mac's pin with --approver-root
- **desktop**: the Keychain's approver pin wins over OPENADAPT_APPROVER_ROOT (a different value is refused); each run's deployment config carries the pin; the roster head is kept in the Keychain and an older roster put back is refused
- **desktop**: a Mac that can keep the key behind Touch ID never falls back to a terminal-made key
- **desktop**: a stop card that changes during the Touch ID prompt changes nothing
- **desktop**: a stop where nothing was written also offers "I did it by hand"



## v0.23.0 (2026-10-24)

### Bug Fixes
Expand Down
119 changes: 119 additions & 0 deletions packages/desktop/benchmark/stop_queue/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,119 @@
# Seconds per stop: the stop queue on 20 recorded soak stops

How long does a person take to decide one stop, before and after the stop
queue (Desktop's **Review stops**)?

## The stops

The 20 stops are the first 20 that the scripted operator handled in the
`after` arm of the production soak of 2026-10-24. That soak ran the
OpenEMR referral fixture with synthetic patients, and the files are in
`packages/flow/benchmark/lifecycle/results/production-2026-10-24/after/`.
The stops break down as follows:

- 15 "Same person?" stops, where the duplicate check found a possible match:
- 8 matches are a different person, so the right answer is to add the patient;
- 7 are the same patient (a resend, a chart that already exists, or a chart
another clerk made a moment earlier), so the right answer is not to add it.
- 3 "Did it save?" stops, where the record check couldn't confirm the save:
- 2 saves went in wrong;
- 1 save is right.
- 2 stops on the item's inputs: one has no date of birth, and one is missing
an optional phone number.

`build_drill.py` turns each stop into the card Desktop's `stop_queue` builds
for it, so the drill uses the same code as the app. The "possible match" rows
are reconstructed from the soak, because the soak did not keep the app's list.
A chart is listed when it existed at that moment and shares the item's last
name or date of birth, which is the rule the duplicate check applies. The
charts that existed are:

- the starting charts;
- charts that earlier items made;
- a chart another clerk made just before the item.

The ground truth gives the right answer for each card. A card is **hard**
when a quick glance would get it wrong: the possible match is a different
person, or the save went in wrong. Ten of the 20 are hard.

```bash
cd packages/desktop
uv run python benchmark/stop_queue/build_drill.py \
--arm ../flow/benchmark/lifecycle/results/production-2026-10-24/after \
--out benchmark/stop_queue/drill-2026-10-24.json
```

## Estimate: keystroke-level model

`klm.py` applies the keystroke-level model to the steps each stop takes. It
uses the operator times from Card, Moran and Newell (1983), as given by
Kieras (2001):

| Operator | Seconds |
| --- | --- |
| Keystroke | 0.28 |
| Point | 1.1 |
| Click | 0.2 |
| Move hand between keyboard and mouse | 0.4 |
| Mental act (read, compare one value, decide) | 1.2 |

It also assumes 1.5 s for the app's patient search and 1.0 s for a Touch ID
prompt. It counts deciding only, not entering an item by hand afterwards.
The steps in each arm are:

- **Before** (Needs you today):
1. Open the item.
2. Read why it stopped.
3. Remember the key values.
4. Switch to the app, search the last name, and open and compare each
listed record.
5. Switch back, tick "I looked at the record", and pick the action.
- **After** (the card):
1. Read the question.
2. Scan the marked comparison for each possible match.
3. Decide, press one key, and confirm Touch ID.

The results are in `klm-2026-10-24.json`:

| Stops | Before, mean | After, mean | Before, median | After, median |
| --- | --- | --- | --- | --- |
| All 20 | 26.7 s | 8.2 s | 27.7 s | 5.3 s |
| 15 "Same person?" | 29.6 s | 5.5 s | 27.9 s | 5.3 s |
| 3 "Did it save?" | 25.6 s | 23.5 s | 25.5 s | 23.4 s |

**This is a model, not a measurement of a person.** The model is what puts
"Same person?" cards in seconds: the card shows the item next to each
possible match, field by field, and marks each value Same or Different, so
no app lookup is needed.

"Did it save?" cards barely change. Their cards don't carry the values the
record check read back, so the person still looks the record up in the app.
The next step is to add those read-back values to the halt.

For scale, the soak's cost model charged 90 s for a stop plus a 30 s
interrupt; it is an assumption too.

## Measuring a person

The practice page runs the real **Review stops** screen on the 20 cards.
Nothing is signed or saved. It records each answer and how long its card was
on screen.

```bash
cd packages/desktop
npx vite --port 1420
# open http://localhost:1420/benchmark/stop_queue/practice.html
```

The score is accuracy only. It says how many stops were right and how many
hard ones the person caught. Time goes into the downloaded results for this
measurement; it is never scored or ranked. Answering blindly scores
accordingly: pressing 1 on every card gets 8 of 20 and catches none of the 10
hard stops.

For "before", time the same 20 stops through Needs you and the app, item by
item. In the product, every answer from **Review stops** is also logged with
its time on screen (`stop_decisions.jsonl`, `shown_ms`). Real operator times
can therefore replace the cost model's 90 s assumption.

No person has run either arm yet.
167 changes: 167 additions & 0 deletions packages/desktop/benchmark/stop_queue/build_drill.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,167 @@
"""Turn recorded soak stops into the stop-queue cards Desktop would show.

Input: a lifecycle soak arm directory (``ledger.jsonl`` and ``workload.jsonl``
from ``packages/flow/benchmark/lifecycle/results/<run>/<arm>/``). Output: a
drill file with the first N stops the soak's operator handled, each as:

* the card Desktop's ``stop_queue`` builds for it (the same code path the app
uses), from the item's values and the stop Flow reported;
* the possible matches the app's duplicate list held at that moment,
reconstructed from the soak: the charts that existed before the run
(``existing_chart``), the charts earlier items created, and a chart another
clerk created just before the item. A row is listed when it shares the
item's last name or date of birth, the rule the duplicate check applies;
* the answer the ground truth calls right, and what the soak's scripted
operator did and was charged.

The patients are the soak's synthetic ones. Run from ``packages/desktop``::

uv run python benchmark/stop_queue/build_drill.py \\
--arm PATH/TO/after --out benchmark/stop_queue/drill-2026-10-24.json
"""

from __future__ import annotations

import argparse
import json
from pathlib import Path
from typing import Any

from openadapt_desktop import stop_queue
from openadapt_desktop.work_items import allowed_actions

#: Columns the app's possible-duplicate list shows.
LIST_FIELDS = ("lname", "fname", "DOB", "sex", "street")


def _rows(path: Path) -> list[dict]:
return [json.loads(line) for line in path.read_text().splitlines() if line.strip()]


def _transaction(reason: str) -> str:
return reason.split(":", 1)[0].strip()


def _expected(kind: str, *, chart_wanted: bool, final_right: bool) -> set[str]:
"""The answers the ground truth calls right for one card."""

if kind == "same_record":
return {"different_by_hand"} if chart_wanted else {"same", "decline"}
if kind == "confirm_save":
return {"saved"} if final_right else {"fixed_by_hand"}
# A stop before saving that isn't a duplicate question: the inputs were
# refused (missing or malformed values).
return {"by_hand"} if chart_wanted else {"skip", "decline"}


def build(arm: Path, *, count: int = 20) -> dict[str, Any]:
ledger = _rows(arm / "ledger.jsonl")
workload = _rows(arm / "workload.jsonl")
items = {r["item_id"]: r for r in workload if r.get("type") == "item"}
existing = [r for r in workload if r.get("type") == "existing_chart"]
results = {r["item_id"]: r for r in ledger if r.get("type") == "item_result"}
stops = [
r
for r in ledger
if r.get("type") == "operator" and r.get("kind") in {"stop", "reconcile"}
][:count]

out = []
for stop in stops:
item = items[stop["item_id"]]
truth = item["truth"]
result = results[stop["item_id"]]
reason = str(stop.get("reason") or "")
transaction = _transaction(reason)
nothing_written = transaction in {"HALTED_BEFORE_EFFECT", "REJECTED_POLICY"}
# Charts in the app when this item ran: the starting set, every
# earlier item that made one, and another clerk's just before it.
charts = [dict(c) for c in existing]
for earlier in sorted(results.values(), key=lambda r: r["seq"]):
if earlier["seq"] >= item["seq"]:
break
made = (earlier.get("final") or {}).get("new_charts") or 0
if made:
charts.append(items[earlier["item_id"]]["truth"]["person"])
for event in truth.get("inject") or []:
if event.get("event") == "other_clerk_creates" and event.get("person"):
charts.append(event["person"])
params = item["params"]
lname = str(params.get("lname") or "").strip().casefold()
dob = str(params.get("DOB") or "").strip()
candidates = [
{f: str(c.get(f) or "") for f in LIST_FIELDS}
for c in charts
if (lname and str(c.get("lname") or "").strip().casefold() == lname)
or (dob and c.get("DOB") == dob)
]
desktop_item = {
"item_id": stop["item_id"],
"source": "item",
"workflow_id": "openemr-referral",
"task_name": "New patient referral",
"label": f"Referral {stop['item_id']}",
# No retained frames from the soak: the card shows no screenshot.
"run_id": None,
"status": "open",
"nothing_written": nothing_written,
"result": "stopped_before_saving" if nothing_written else "check_record",
"approval": None,
"halt_kind": "screen_changed" if nothing_written else "save_not_confirmed",
"halt_reason": reason[:500],
"halt_step": "",
"expected": "",
"observed": "",
"field_diffs": [],
"inputs": {k: str(v) for k, v in params.items() if v is not None},
"candidates": candidates if "no_duplicate_candidate" in reason else [],
"created_at": None,
}
desktop_item["actions"] = allowed_actions(desktop_item)
card = stop_queue.card_for_item(desktop_item)
if card is None:
continue
chart_wanted = bool(result.get("chart_wanted"))
final_right = (result.get("final") or {}).get("verdict") == "correct"
out.append(
{
"item_id": stop["item_id"],
"exception": truth.get("exception"),
"rationale": truth.get("rationale"),
"card": card,
"expected": sorted(
_expected(card["kind"], chart_wanted=chart_wanted, final_right=final_right)
),
# A card a quick glance would get wrong: a possible match that
# is a different person, or a save that went in wrong.
"hard": card["kind"] == "same_record" and chart_wanted
or card["kind"] == "confirm_save" and not final_right,
"soak_operator": {
"action": stop.get("action"),
"rule": stop.get("rule"),
"charged_s": stop.get("charged_s"),
"cost_model": stop.get("cost_model"),
},
}
)
return {
"schema": "openadapt.stop-queue-drill/v1",
"source": str(arm),
"selection": f"the first {count} stops the soak's operator handled, in order",
"stops": out,
}


def main() -> None:
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
parser.add_argument("--arm", required=True, type=Path)
parser.add_argument("--out", required=True, type=Path)
parser.add_argument("--count", type=int, default=20)
args = parser.parse_args()
drill = build(args.arm, count=args.count)
args.out.write_text(json.dumps(drill, indent=2, sort_keys=True) + "\n")
print(f"wrote {len(drill['stops'])} stops to {args.out}")


if __name__ == "__main__":
main()
Loading
Loading