diff --git a/packages/flow/benchmark/economic_comparison/.gitignore b/packages/flow/benchmark/economic_comparison/.gitignore new file mode 100644 index 000000000..2312e255c --- /dev/null +++ b/packages/flow/benchmark/economic_comparison/.gitignore @@ -0,0 +1 @@ +results/totals.json diff --git a/packages/flow/benchmark/economic_comparison/README.md b/packages/flow/benchmark/economic_comparison/README.md new file mode 100644 index 000000000..043dbc4b0 --- /dev/null +++ b/packages/flow/benchmark/economic_comparison/README.md @@ -0,0 +1,337 @@ +# Economic comparison: OpenAdapt against competent alternatives + +This directory compares OpenAdapt with three alternatives a competent team +could build today. It counts every person's minute and the total cost over +10, 100 and 1,000 items, not only the per-run speed. Every number in the +tables comes from the raw rows listed under [Raw results](#raw-results). +People's time is priced from stated assumptions +([`assumptions.json`](assumptions.json)), never measured. + +Measured on 2026-10-24. Synthetic data only. + +## Short answer + +On these two tasks OpenAdapt does not win the economics. + +- **OpenEMR referral intake (55 items, one app change).** The agent with + structured tools and REST read-back beat OpenAdapt's production + configuration on most rows: + - automated items: 49 vs 41; + - false stops: 5 vs 9 of 35; + - median time: 24 s vs 119 s an item; + - operator time: 0.91 vs 2.15 minutes an item; + - projected total for 1,000 items: $606 vs $1,158. + + OpenAdapt was better only on model cost ($0 vs $0.05 an item). Neither arm + made a silent wrong save (0 of 55 each). +- **MockMed fault suite (100 items, 10 UI changes, 5 server faults).** Both + competent alternatives verified 90 of 90 items with 0 silent wrong saves. + The agent-written script cost $0.26 in model calls for all 100 items, + including four repairs of its own. OpenAdapt's program verified 71 of 90 + with 0 silent wrong saves and 19 false stops. The 1,000-item totals are + within about 30% of each other ($104 to $135). OpenAdapt is lowest in + dollars and highest in people minutes. +- **The silent-wrong advantage holds only against an agent that trusts the + screen.** The screenshot-only computer-use agent reported all 5 server-side + bad saves as saved. Every arm that reads the record back reported none of + them as saved, whether it was OpenAdapt or not. + +## The arms + +Every arm gets the same task text, the same app and the same read access to +the system of record that OpenAdapt's record check uses: REST on OpenEMR, +`GET /api/db` on MockMed. No arm gets the fault, and no arm judges itself. On +OpenEMR, the oracle is a direct SQL read; on MockMed, it is the store's rows +before and after each item. + +| # | Arm | What it is | Model | +|---|---|---|---| +| 1 | Agent with structured tools | Reads the page's controls (DOM and accessibility view, not screenshots), acts on them by reference, and reads the record before and after the save. It is told to claim success only when exactly one correct record landed. On OpenEMR: [`lifecycle/arms/agent_tools.py`](../lifecycle/arms/agent_tools.py). On MockMed: `AgentTools` in [`silent_saves/agent_arms.py`](../silent_saves/agent_arms.py). | Claude Sonnet 5.5 | +| 2 | Agent-written script | A coding agent writes a Playwright script with the same checks: a duplicate guard before the save and a read-back after it. It dry-runs the script with the save blocked at the network layer, then submits it. Items then run with no model call. When the script breaks, the agent fixes it against the live page and the item runs again. On OpenEMR: [`lifecycle/arms/agent_script.py`](../lifecycle/arms/agent_script.py). On MockMed: `AgentScript` in [`silent_saves/agent_arms.py`](../silent_saves/agent_arms.py). | Claude Sonnet 5.5 | +| 3 | OpenAdapt, as a customer would run it | On OpenEMR: the Regulated profile, `problems_only` approval, durable runs, pipelined read-back, self-repair for a relabeled save button, and the governed run gate on the sealed bundle. This is the production-lifecycle session's arm, `lifecycle/production_arm.py`, run on its branch's engine. The API step is refused on this form (see below), so the GUI save runs. On MockMed: the admitted program with read-back and no model calls. The MockMed program does not run the Regulated profile. | none | +| 4 | Computer-use agent, screenshots only | The scoreboard's reference agent. Measured on MockMed only (committed runs of 2026-10-19). | Claude Sonnet 5.5 | + +A fifth MockMed row runs arm 4's own runs with each claim gated on +OpenAdapt's read-back (`verify-run`). That is the cheapest way to add the +read-back to an existing agent. + +### The API step + +The 3.33 s API-path figure was measured without approval gates or durable +runs. On current main the compiler refuses the API step for this form: +OpenEMR validates the date of birth with a script rule the API step can't +evaluate, and another write happens during the form span. So OpenAdapt's +gated OpenEMR numbers below are the GUI path's +([`../lifecycle/results/api_tier/`](../lifecycle/results/api_tier/)). + +## OpenEMR referral intake + +Seed 7, 55 referrals over 10 simulated days, workload +`lifecycle-workload/2026-10-24.1`. About a third of the referrals carry an +exception: duplicates, resends, shared surnames and dates of birth, a missing +date of birth, malformed fields, other clerks, a forced logout, slow and +failed saves. One app change: the save button "Confirm Create New Patient" +becomes "Save new patient" from item 26 on. Local OpenEMR 8.0.0.3. Host +1-minute load: median 5.8 (max 79.4) for the agent with tools, 22.1 (max +44.2) for the agent script, and 8.0 (max 102.5) for OpenAdapt's run, read +from each item row of the ledgers. + +| Arm | Items | Right in the end | Automated (no hand entry) | Silent wrong | False stops (of clean items) | Median s an item | Model $ an item | Operator min an item | After the app change | Model $ in total | +|---|---|---|---|---|---|---|---|---|---|---| +| 3. OpenAdapt, production configuration | 55 | 53 | 41 | 0 | 9 of 35 | 118.5 | 0 | 2.15 | 7 of 30 later items false-stopped; 7.5 min of maintenance; first verified item 2 items after the change | 0 | +| 3 on current main | 27 (stopped) | 27 | 4 | 0 | 19 of 20 | 226.9 | 0 | 4.56 | not reached | 0 | +| 1. Agent with tools | 55 | 55 | 49 | 0 | 5 of 35 | 24.3 | 0.050 | 0.91 | no stop in items 26-42; 6 stops in items 43-54 (a stuck confirm popup) | 2.77 | +| 2. Agent-written script | 16 (hung) | 16 | 7 | 0 | 7 of 12 | 5.3 | 0 | 2.44 | not reached | 0.86 | +| By hand (reference) | | | 0 | | | 210 | 0 | 3.50 | | 0 | + +"Right in the end" counts items the oracle judges right, whoever did them. +Items an arm stopped on were entered by hand by the scripted operator, and +that time is in "Operator min an item". The silent-wrong 95% CI for 0 of 55 +is 0 to 6.5%. + +What happened to each arm: + +- **OpenAdapt.** It stopped on every possible-duplicate listing. That is + right for real duplicates, and a false stop for new patients who share a + surname or a date of birth with an existing chart (9 false stops). On + item 26, the first after the relabel, it stopped after the save had + landed (`RECONCILIATION_REQUIRED`), so a person reconciled it. A person + approved the proposed relabel repair once, and later items ran on the + repaired program. `problems_only` still asked a person to approve the save + click on 30 items (20 s each in the cost model). +- **OpenAdapt on current main** created no chart by itself in 27 items. Its + duplicate check stopped on every listing. +- **Agent with tools.** It checked the record before saving and judged the + duplicate listings itself. It stopped on all 10 items where stopping was + right: existing charts, resends, another clerk's registration, an + ambiguous match and missing dates of birth. It handled the relabel with + no stop. From item 43 on it stopped 6 + times: the confirm popup did not close after a click, and the agent + reported "not saved" rather than saving twice. None of those items left a + chart, so they are false stops, not wrong saves. +- **Agent-written script.** Its first script stopped on every + possible-duplicate listing (5 items). It also timed out on the title field + on 4 items; the agent fixed it 4 times. After item 16 the run made no + progress for about 2 hours and was stopped to free the shared fixture + (ledger row `stopped: hung`). No stack was captured, so the cause is not + known; every wait in the agent's script had a timeout, which points at a + harness call with none (such as `page.evaluate` in OpenEMR's tab shell). + Its numbers are 16 items before the app change, and are not comparable to + the full runs. The hand-written script and the screenshot agent were not + run on OpenEMR. `lifecycle.arms.compare` now bounds every item + (`LIFECYCLE_ITEM_TIMEOUT_S`, default 900 s). On a timeout it writes every + thread's stack to `stacks.txt`, kills the arm's browser driver, records + the item as an error and gives the arm a fresh browser. If the arm still + does not return, the process exits after 3 more minutes. The remaining + OpenEMR arms are queued for the fixture as `baselines-rest`. + +### Totals for OpenEMR + +Setup + items + app changes (4 per 1,000 items) + cleanup of silent wrongs. +People time is in the brackets at x0.5 and x2 of the assumed rates. + +| Arm | 10 items | 100 items | 1,000 items | People minutes for 1,000 | +|---|---|---|---|---| +| 3. OpenAdapt, production configuration | $72 [36-144] | $171 [86-340] | $1,158 [584-2,307] | 2,228 | +| 3 on current main | $84 | $290 | $2,351 | 4,608 | +| 1. Agent with tools | $105 [53-210] | $151 [78-296] | $606 [329-1,161] | 969 | +| 2. Agent-written script (16 items only) | $130 | $239 | $1,337 | 2,508 | +| By hand | $18 | $175 | $1,750 | 3,500 | + +At 10 items, entering every referral by hand costs least. From 100 items on, +the agent with tools costs least; that holds at x0.5 and x2. OpenAdapt's +production configuration removes about 36% of the people time of doing +everything by hand, and the agent with tools about 72%. + +## MockMed fault suite + +[`silent_saves`](../silent_saves/README.md) fault set 1.1.0: 100 runs of +"open Jane Sample's referral, start a Triage encounter, save this note". Ten +blocks each switch one UI change on: renamed controls, display scaling, moved +buttons, a What's New screen, latency, a reordered queue, a moved New +Encounter control, a look-alike referral, a relabeled type and a missing +referral. Five items get a server-side bad save. The store is read before and +after every item. + +| Arm | Verified (of 90) | Silent wrong (of 100) | False stops | Server faults reported saved (of 5) | Stopped when no save existed (of 5) | Model $ for 100 items | Seconds an item, control block | +|---|---|---|---|---|---|---|---| +| 1. Agent with tools | 90 | 0 | 0 | 0 | 5 | 1.45 | 13.7 | +| 2. Agent-written script | 90 | 0 | 0 | 0 | 5 | 0.26 | 0.8 | +| 3. OpenAdapt program (current main) | 71 | 0 | 19 | 0 | 5 | 0 | 7.3 | +| 3r. OpenAdapt program with self-repair (engine of 2026-10-19) | 62 | 0 | 28 | 0 | 5 | 0.48 | 7.4 | +| 4. Computer-use agent, screenshots only | 90 | 5 | 0 | 5 | 5 | 3.90 | 22.5 | +| 4 with OpenAdapt read-back | 90 | 0 | 0 | 0 | 5 | 3.90 | 22.5 + 1.2 | + +The silent-wrong 95% CI for 0 of 100 is 0 to 3.7%. Seconds come from a +separate 15-item control-block pass at normal load +(`/2026-10-24-timing/`). The full runs ran at host loads of 77 to 374 +and their times are not used. Rows 3r and 4 are committed runs of +2026-10-19. + +Where each arm spent people time, per UI change, in minutes over a steady +item (+fix: a person fixed the program, script or prompt because the arm had +not recovered by the end of the block): + +| Change | Agent with tools | Agent script | OpenAdapt program | Screenshot agent | +|---|---|---|---|---| +| What's New screen | 0 | 10 (one fix reviewed) | 52 +fix (re-teach) | 0 | +| New Encounter moved and renamed | 0 | 10 (one fix reviewed) | 49 +fix (re-teach) | 0 | +| Look-alike referral above the target | 0 | 10 (one fix reviewed) | 0 | 0 | +| Target referral gone (right answer: stop) | 10 | 40 (3 fix attempts reviewed; it kept stopping) | 10 | 10 | +| The other six changes | 0 | 0 | 0 | 0 | + +The OpenAdapt program stopped on every item under two changes: the What's +New screen and the moved New Encounter control. Its self-repair did not +propose an admissible fix for either. + +### Totals for MockMed + +| Arm | 10 items | 100 items | 1,000 items | People minutes for 1,000 | +|---|---|---|---|---| +| 1. Agent with tools | $100 | $104 | $135 | 102 | +| 2. Agent-written script | $51 | $56 | $111 | 96 | +| 3. OpenAdapt program | $61 | $65 | $104 | 134 | +| 3r. with self-repair (2026-10-19) | $61 | $66 | $110 | 148 | +| 4. Screenshot agent | $51 | $56 | $110 | 69, plus 10 silent wrongs | +| 4 with OpenAdapt read-back | $51 | $56 | $111 | 72 | + +On MockMed the totals are dominated by setup (an engineer's or operator's +hour) and by assumed server faults (10 per 1,000 items). Setup and faults +cost about the same in every arm. The screenshot agent's 10 silent wrongs in +1,000 items are charged only the lower-bound cleanup: a reconcile plus a +re-entry each. Their harm is not priced. + +## Where OpenAdapt loses and where it wins + +OpenAdapt loses on: + +- Verified and automated items, on both apps: 41 vs 49 of 55 on OpenEMR, + and 71 vs 90 of 90 on MockMed. +- False stops: 9 vs 5 on OpenEMR, and 19 vs 0 on MockMed. +- Seconds an item on OpenEMR: 119 vs 24. +- Operator minutes and total cost from 100 items on, on OpenEMR. +- People time per UI change on MockMed, under the two changes it could not + repair. + +OpenAdapt wins on: + +- Model cost an item: $0 vs $0.013 to $0.05 for the agent arms. At 1,000 + items, model cost is $13 to $50 and so a small share of every total. +- Silent wrong saves compared with an agent that trusts the screen: 0 vs 5 + of 5 server faults. This is not a win over agents that read the record + back: they also had 0. +- Stopping when no correct save exists: 5 of 5, the same as every + read-back arm. + +## Assumptions + +All in [`assumptions.json`](assumptions.json): + +- **Labor:** operator $30 an hour and engineer $100 an hour. +- **Setup people time:** + - OpenAdapt: 22 operator minutes (teach, review, admit), plus 30 engineer + minutes to install it and write the record check. + - Agent with tools: 60 engineer minutes. + - Agent script: 30 engineer minutes. + - Screenshot agent: 30 engineer minutes. +- **Per-touch operator time (cost model V1):** + - stop: 90 s; + - reconcile: 120 s; + - repair review: 120 s; + - manual entry: 180 s on OpenEMR and 60 s on MockMed; + - interrupt: 30 s per request. +- **Maintenance people time:** + - an engineer reviews each agent-made script fix: 10 min; + - an engineer fixes a script by hand: 30 min; + - an operator re-teaches OpenAdapt after a change it could not handle: + 22 min; + - an engineer adjusts an agent's prompt: 15 min. +- **Volume:** 4 UI changes and 10 server-side bad saves per 1,000 items. + +No operator timing here is from a real person. Change any number in +`assumptions.json` and run `python -m economic_comparison.totals`. + +## Limits + +- One run per arm, one task per app, a scripted operator and synthetic + data. Small samples: read the confidence intervals. +- The OpenEMR runs for arm 3 and arms 1 and 2 used the same seeded stream on + two copies of the same pinned fixture, a few hours apart. +- Arm 3 on OpenEMR runs the engine of the `measure/production-lifecycle` + branch (duplicate-list check, relabel repair). Current main is the + "3 on main" row. +- The agent-written script on OpenEMR hung after 16 items. The hand-written + script and the screenshot agent were not run on OpenEMR. A stronger + authoring model or more engineer time might give a better script. That + was not measured. +- The MockMed OpenAdapt program is not the Regulated configuration. It has + no Regulated MockMed admission. +- The MockMed program's first pass broke on items 16-18 and 83-100. Item 83 + hit the 300 s watchdog under host load 374 and left the browser driver + unusable for the items after it; items 16-18 hit a busy durable authority. + Those 21 items were rerun in a fresh process. The first pass is kept as + `items.first-pass.csv`. +- Model spend for this comparison: $6.05 of the $30 cap, ledger + [`../lifecycle/results/spend.json`](../lifecycle/results/spend.json). + +## Raw results + +In git: every row the tables are computed from. + +| What | Where | +|---|---| +| OpenEMR, arms 1 and 2 | [`../lifecycle/results/economics-2026-10-24/`](../lifecycle/results/economics-2026-10-24/) (`/ledger.jsonl`, `workload.jsonl`, `plan.json`) | +| OpenEMR, arm 3 | [`../lifecycle/results/production-2026-10-24/`](../lifecycle/results/production-2026-10-24/) (`after/`, `before/`) | +| MockMed, new runs | `../silent_saves/results/1.1.0/{agent-tools,agent-script,openadapt-program}/2026-10-24*/` (the agent's scripts are in `agent-script/2026-10-24/scripts/`) | +| MockMed, earlier runs | `../silent_saves/results/1.1.0/{claude-computer-use+readback,self-repair}/2026-10-19/` | +| Tables | [`results/TABLE.md`](results/TABLE.md); `python -m economic_comparison.totals` also writes `results/totals.json` | +| Model spend | [`../lifecycle/results/spend.json`](../lifecycle/results/spend.json), totals per label | + +Outside git: the agents' per-item tool traces (OpenEMR agent with tools and +agent script, MockMed agent with tools) and the per-call spend ledger. +They are the release asset `economic-comparison-2026-10-24-traces.tar.gz` +(63 KB compressed, 220 files, sha256 +`906a9dafaba28a98bdd4a427f519c57adf9f66a3b23ec91ebddef833c8146f57`) of the +private repository `OpenAdaptAI/openadapt-workspace`, tag +`evidence-economic-comparison-2026-10-24`. Unpack it in +`packages/flow/benchmark`. + +## Next runs + +- **The rest of OpenEMR.** The hand-written script, the screenshot agent, + and the agent script from the start, each bounded by the per-item + watchdog. +- **A hybrid arm: give your agent a program.** The agent with tools also + gets OpenAdapt-taught programs as tools (`taught_`, through + `packages/agent`). It calls the program for the referral and keeps the + record read-back. This tests whether OpenAdapt is cheaper as a tool + inside an agent than as the whole runtime. +- **Timed people instead of the cost model.** Operator minutes here are + charged from cost model V1. The planned rerun uses timed people (pilot + users or volunteers), with this method: + - every stop, approval, reconcile and repair review is served through + the product's own surfaces (Desktop and the console) to a person who + did not build the arm; + - each request is timed from when it is shown to when it is answered, + plus the person's own count of switching time; + - by-hand entries are timed on the same referrals; + - the same seeded stream runs on every arm, in a different order per + person, so learning does not favour one arm; + - medians and spread are reported per touch kind beside cost model V1, + and the totals are recomputed from the timings. + +## Reproduce + +```bash +cd packages/flow/benchmark +python -m economic_comparison.totals # recompute every table from the raw rows + +# MockMed (no fixture) +python -m silent_saves --fault-set 1.1.0 --agent python:silent_saves.agent_arms:AgentTools --out +python -m silent_saves --fault-set 1.1.0 --agent python:silent_saves.agent_arms:AgentScript --out + +# OpenEMR (one Docker fixture per machine at a time) +python -m lifecycle.apps up +python -m lifecycle.arms.compare --out --seed 7 --items 55 --days 10 \ + --drift relabel_save@26 --arms-json economic_comparison/openemr_arms.json +``` diff --git a/packages/flow/benchmark/economic_comparison/__init__.py b/packages/flow/benchmark/economic_comparison/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/packages/flow/benchmark/economic_comparison/assumptions.json b/packages/flow/benchmark/economic_comparison/assumptions.json new file mode 100644 index 000000000..ac5f3f9a3 --- /dev/null +++ b/packages/flow/benchmark/economic_comparison/assumptions.json @@ -0,0 +1,100 @@ +{ + "version": "economic-comparison/2026-10-24.1", + "note": "Every number here is an assumption, not a measurement. totals.py takes measured rates from the raw results and these assumptions, and shows the totals at half and double the people-time assumptions. Change a number here, never in code.", + "labor_usd_per_hour": { + "operator": 30.0, + "engineer": 100.0, + "source": "lifecycle/arms/prices.json (labor): round figures for a fully loaded front-desk clerk hour and automation engineer hour" + }, + "operator_s": { + "source": "lifecycle/cost_model.py V1 (lifecycle-cost/2026-10-20.1); no real operator timings exist yet", + "stop": 90.0, + "reconcile": 120.0, + "repair_review": 120.0, + "manual_entry_mockmed": 60.0, + "manual_entry_openemr": 180.0, + "interrupt": 30.0, + "manual_entry_mockmed_why": "a MockMed triage note is one click path and one note field, about a third of the 12-field OpenEMR registration the cost model's 180 s is for" + }, + "per_outcome": { + "note": "What a person does after each outcome. One interrupt per item that reaches a person.", + "correct": [], + "claimed_failure_save_possible": [ + "stop", + "manual_entry" + ], + "claimed_failure_no_save_possible": [ + "stop" + ], + "flagged_wrong": [ + "stop", + "reconcile", + "manual_entry" + ], + "false_alarm": [ + "stop", + "reconcile" + ], + "silent_wrong": [ + "reconcile", + "manual_entry", + "interrupt" + ], + "silent_wrong_why": "lower bound: someone later finds the wrong record, fixes it and enters the item again (two interruptions: when found and when fixed). Clinical or business harm is not priced." + }, + "setup_people_min": { + "note": "One-time people time per workflow, by role. The agent's own authoring time and model cost are measured and reported separately.", + "openadapt": { + "operator": 22.0, + "engineer": 30.0, + "why": "cost model V1: teach preparation 5 min + a 2 min demonstration + compile review 10 min + admission 5 min; plus an engineer's 30 min to install it and write the record check (verify.yaml), the same read-back work the agent arms' engineer does" + }, + "openadapt_regulated_openemr": { + "operator": 22.0, + "engineer": 30.0, + "why": "as above, plus an engineer's 30 min for the Regulated deployment (keys, sealed bundle, durable store, problems-only switch); the qualification campaign runs unattended" + }, + "agent_tools": { + "operator": 0.0, + "engineer": 60.0, + "why": "an engineer wires a browser-tools agent (an off-the-shelf framework gives read_screen/click/fill), writes the record read-back tool and the check rule into the prompt, and tries it" + }, + "agent_script": { + "operator": 0.0, + "engineer": 30.0, + "why": "an engineer prompts the coding agent with the demonstration and the API description, reads and tests the script it submits" + }, + "cua_alone": { + "operator": 0.0, + "engineer": 30.0, + "why": "an engineer writes the task prompt and deploys the computer-use loop" + }, + "human_script": { + "operator": 0.0, + "engineer": 120.0, + "why": "lifecycle/arms/prices.json engineer_script_author_s (2 h)" + } + }, + "maintenance_people_min": { + "note": "People time per maintenance event, by kind.", + "script_fix_review_engineer": 10.0, + "script_fix_review_why": "an engineer reads the coding agent's diff and its dry run, and deploys it", + "script_fix_by_hand_engineer": 30.0, + "script_fix_by_hand_why": "lifecycle/arms/prices.json engineer_script_fix_s (30 min), when the coding agent's own fix did not hold", + "openadapt_reteach_operator": 22.0, + "openadapt_reteach_why": "a change OpenAdapt neither handled nor self-repaired: the person teaches the workflow again (same as setup)", + "agent_prompt_fix_engineer": 15.0, + "agent_prompt_fix_why": "a change an agent kept failing on: an engineer adjusts the prompt" + }, + "detection": { + "items_until_a_silent_wrong_is_noticed": null, + "note": "Silent wrongs are charged one cleanup each; how long they go unnoticed is not modelled." + }, + "volume": { + "ui_changes_per_1000_items": 4, + "why": "lifecycle/arms/prices.json drift rate: about one vendor release every few weeks at a clinic's referral volume", + "server_faults_per_1000_items": 10, + "server_faults_why": "assumed: one save in 100 goes wrong on the server side. The MockMed suite has 5 in 100 to measure detection; the excess cost per fault item is measured there and applied at this rate." + }, + "compute_usd_per_hour": 0.2 +} diff --git a/packages/flow/benchmark/economic_comparison/openemr_arms.json b/packages/flow/benchmark/economic_comparison/openemr_arms.json new file mode 100644 index 000000000..665213d39 --- /dev/null +++ b/packages/flow/benchmark/economic_comparison/openemr_arms.json @@ -0,0 +1,6 @@ +[ + ["agent_tools", "lifecycle.arms.agent_tools:factory", {"model": "claude-sonnet-5-5", "effort": "medium", "item_budget_usd": 0.50}], + ["agent_script", "lifecycle.arms.agent_script:factory", {"model": "claude-sonnet-5-5", "effort": "high", "author_budget_usd": 3.0, "fix_budget_usd": 1.5}], + ["human_script", "lifecycle.arms.human_script:factory", {}], + ["cua_alone", "lifecycle.arms.cua:factory", {"model": "claude-sonnet-5-5", "effort": "medium", "item_budget_usd": 0.80, "stop_after": 12}] +] diff --git a/packages/flow/benchmark/economic_comparison/results/TABLE.md b/packages/flow/benchmark/economic_comparison/results/TABLE.md new file mode 100644 index 000000000..f70752c09 --- /dev/null +++ b/packages/flow/benchmark/economic_comparison/results/TABLE.md @@ -0,0 +1,56 @@ +## OpenEMR referral intake (seed 7, 55 items, one app change: the save button relabeled at item 26) + +| Arm | Items | Verified in the end | Automated (no hand entry) | Silent wrong | False stops | Median s/item | Model $/item | Operator min/item | Maintenance for the change (people min / model $) | Setup people min | Setup agent min / $ | +|---|---|---|---|---|---|---|---|---|---|---|---| +| 3. OpenAdapt, Regulated + problems_only, durable, pipelined read-back, relabel repair (engine of measure/production-lifecycle; the API step is refused on this form, so the GUI save runs) | 55 | 53 | 41 | 0 | 9 of 35 | 118.5 | 0.000 | 2.15 | 7.5 / 0.00 | 52 | 13.3 / 0.00 | +| 3 on main. Same configuration, current main's engine [short run: 27 of 55 planned items; the ledger ends before the planned items (the run exited or was killed)] | 27 | 27 | 4 | 0 | 19 of 20 | 226.9 | 0.000 | 4.56 | 0.0 / 0.00 | 52 | 13.4 / 0.00 | +| 1. Agent with structured tools and REST read-back (Claude Sonnet 5.5) | 55 | 55 | 49 | 0 | 5 of 35 | 24.3 | 0.050 | 0.91 | 0.0 / 0.00 | 60 | 0.0 / 0.00 | +| 2. Agent-written Playwright + REST script with read-back and duplicate check, agent-maintained (Claude Sonnet 5.5) [short run: 16 of 55 planned items; hung: no progress for about 2 h after item 16 (process at 0% CPU, one Playwright driver idle); stopped by hand to release the shared fixture lease] | 16 | 16 | 7 | 0 | 7 of 12 | 5.3 | 0.000 | 2.44 | not reached | 70 | 2.6 / 0.86 | +| Reference: a person enters every item (cost model V1) | None | None | 0 | 0 | None of None | 210.0 | 0.000 | 3.50 | 0.0 / 0.00 | 0 | 0.0 / 0.00 | + +### Totals (people time priced at the assumed rates; x0.5 and x2 in brackets) + +| Arm | 10 items: total $ | people min | 100 items: total $ | people min | 1000 items: total $ | people min | +|---|---|---|---|---|---|---| +| openadapt-production-relabel | 72 [36-144] | 74 | 171 [86-340] | 270 | 1,158 [584-2,307] | 2,228 | +| openadapt-production | 84 [42-168] | 98 | 290 [146-579] | 508 | 2,351 [1,181-4,689] | 4,608 | +| agent_tools | 105 [53-210] | 69 | 151 [78-296] | 151 | 606 [329-1,161] | 969 | +| agent_script | 130 [65-259] | 94 | 239 [120-478] | 314 | 1,337 [669-2,672] | 2,508 | +| all_manual | 18 [9-35] | 35 | 175 [88-350] | 350 | 1,750 [875-3,500] | 3,500 | + +## MockMed, fault set 1.1.0 (100 items, 10 UI changes, 5 server faults) + +| Arm | Verified (of 90) | Silent wrong | False stops | Faults claimed saved (of 5) | Stopped when no save possible (of 5) | Control s/item (load) | Model $/item | Setup people min | Setup agent min / $ | +|---|---|---|---|---|---|---|---|---|---| +| 4. Computer-use agent, screenshots only (Claude Sonnet 5.5) | 90 | 5 | 0 | 5 | 5 | 22.5 (17) | 0.0386 | 30 | 0.0 / 0.000 | +| 1. Agent with structured tools and record read-back (Claude Sonnet 5.5) | 90 | 0 | 0 | 0 | 5 | 13.7 (15) | 0.0132 | 60 | 0.0 / 0.000 | +| 2. Agent-written Playwright script with read-back and duplicate guard, agent-maintained | 90 | 0 | 0 | 0 | 5 | 0.8 (6) | 0.0000 | 30 | 0.4 / 0.026 | +| 3. OpenAdapt admitted program, read-back on, no model calls | 71 | 0 | 19 | 0 | 5 | 7.3 (8) | 0.0000 | 52 | 0.0 / 0.000 | +| 3r. OpenAdapt program with self-repair (engine of 2026-10-19) | 62 | 0 | 28 | 0 | 5 | 7.4 (10) | 0.0000 | 52 | 0.0 / 0.000 | +| 4 + OpenAdapt read-back: the same agent runs, each claim gated on verify-run | 90 | 0 | 0 | 0 | 5 | 22.5 (17) | 0.0386 | 30 | 0.0 / 0.000 | + +### Each UI change: excess people minutes (silent wrongs in the block) + +| Change | cua_alone | agent_tools | agent_script | openadapt_program | openadapt_self_repair | cua_plus_openadapt_readback | +|---|---|---|---|---|---|---| +| renamed controls | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | +| display scaling | 0.0 | 0.0 | 0.0 | 0.0 | 49.0 +fix | 0.0 | +| moved controls | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | +| new interstitial | 0.0 | 0.0 | 10.0 | 52.0 +fix | 52.0 +fix | 0.0 | +| latency | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | +| reordered queue | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | +| renamed and moved New Encounter | 0.0 | 0.0 | 10.0 | 49.0 +fix | 49.0 +fix | 0.0 | +| look-alike referral | 0.0 | 0.0 | 10.0 | 0.0 | 0.0 | 0.0 | +| relabeled and reordered type | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | +| target referral gone | 10.0 | 10.0 | 40.0 | 10.0 | 10.0 | 10.0 | + +### Totals (people time priced at the assumed rates; x0.5 and x2 in brackets) + +| Arm | 10 items: total $ | people min | silent wrongs | 100 items: total $ | people min | silent wrongs | 1000 items: total $ | people min | silent wrongs | +|---|---|---|---|---|---|---|---|---|---| +| cua_alone | 51 [26-101] | 30 | 0.1 | 56 [30-108] | 34 | 1.0 | 110 [75-179] | 69 | 10.0 | +| agent_tools | 100 [50-201] | 60 | 0.0 | 104 [52-206] | 64 | 0.0 | 135 [74-256] | 102 | 0.0 | +| agent_script | 51 [25-101] | 31 | 0.0 | 56 [28-112] | 37 | 0.0 | 111 [56-222] | 96 | 0.0 | +| openadapt_program | 61 [31-123] | 53 | 0.0 | 65 [33-130] | 60 | 0.0 | 104 [53-206] | 134 | 0.0 | +| openadapt_self_repair | 61 [31-123] | 53 | 0.0 | 66 [33-132] | 62 | 0.0 | 110 [55-219] | 148 | 0.0 | +| cua_plus_openadapt_readback | 51 [26-101] | 30 | 0.0 | 56 [30-108] | 34 | 0.0 | 111 [76-182] | 72 | 0.0 | diff --git a/packages/flow/benchmark/economic_comparison/totals.py b/packages/flow/benchmark/economic_comparison/totals.py new file mode 100644 index 000000000..3eee4f3af --- /dev/null +++ b/packages/flow/benchmark/economic_comparison/totals.py @@ -0,0 +1,811 @@ +"""Lifecycle totals over 10, 100 and 1,000 items, per arm, from raw results. + + cd packages/flow/benchmark + python -m economic_comparison.totals # writes results/totals.json and results/TABLE.md + +Measured, per arm, from the raw rows (nothing here is re-run): + +- per-item time, model cost and outcome on the control block (no UI change, + no server fault); +- the excess people time and model cost of each server-fault item; +- the excess people time and model cost of each UI change (one block of + items with that change on), including any repair or re-authoring the arm + did, any silent wrongs it made under the change, and a person's fix when + the arm had not recovered by the end of the block; +- setup: the agent's measured authoring time and cost, plus the people time + in ``assumptions.json``. + +People time is priced from ``assumptions.json`` (cost model V1 and the labor +rates), never measured; every total is also given at half and double it. + +Totals for N items = setup + N x (control per-item cost) ++ N x (faults per item) x (mean excess per fault item) ++ N x (changes per item) x (mean excess per UI change). +""" + +from __future__ import annotations + +import csv +import json +import statistics +import sys +from pathlib import Path +from typing import Any, Optional + +HERE = Path(__file__).resolve().parent +BENCH = HERE.parent +SS = BENCH / "silent_saves" / "results" / "1.1.0" +ASSUME = json.loads((HERE / "assumptions.json").read_text()) +SIZES = (10, 100, 1000) + +#: MockMed arms: (name, kind, items.csv, outcome column, setup key, label) +MOCKMED_ARMS: list[dict[str, Any]] = [ + { + "arm": "cua_alone", + "kind": "agent", + "dir": "claude-computer-use+readback/2026-10-19", + "outcome": "outcome_alone", + "setup": "cua_alone", + "label": "4. Computer-use agent, screenshots only (Claude Sonnet 5.5)", + }, + { + "arm": "agent_tools", + "timing_dir": "agent-tools/2026-10-24-timing", + "kind": "agent", + "dir": "agent-tools/2026-10-24", + "outcome": "outcome", + "setup": "agent_tools", + "label": "1. Agent with structured tools and record read-back (Claude Sonnet 5.5)", + }, + { + "arm": "agent_script", + "timing_dir": "agent-script/2026-10-24-timing", + "kind": "script", + "dir": "agent-script/2026-10-24", + "outcome": "outcome", + "setup": "agent_script", + "label": "2. Agent-written Playwright script with read-back and duplicate guard, agent-maintained", + }, + { + "arm": "openadapt_program", + "timing_dir": "openadapt-program/2026-10-24-timing", + "kind": "program", + "dir": "openadapt-program/2026-10-24", + "outcome": "outcome", + "setup": "openadapt", + "label": "3. OpenAdapt admitted program, read-back on, no model calls", + }, + { + "arm": "openadapt_self_repair", + "kind": "program", + "dir": "self-repair/2026-10-19", + "outcome": "outcome", + "setup": "openadapt", + "label": "3r. OpenAdapt program with self-repair (engine of 2026-10-19)", + }, + { + "arm": "cua_plus_openadapt_readback", + "kind": "agent", + "dir": "claude-computer-use+readback/2026-10-19", + "outcome": "outcome", + "setup": "cua_alone", + "label": "4 + OpenAdapt read-back: the same agent runs, each claim gated on verify-run", + }, +] + + +def _people_s(steps: list[str], manual_key: str) -> float: + op = ASSUME["operator_s"] + total = 0.0 + for s in steps: + total += op[manual_key] if s == "manual_entry" else op[s] + if steps and "interrupt" not in steps: + total += op["interrupt"] + return total + + +def operator_s(row: dict[str, Any], outcome: str, manual_key: str) -> float: + """People seconds after one item, from what the arm reported and the store.""" + + per = ASSUME["per_outcome"] + save_possible = row["expect"] == "save" and row["fault"] == "ok" + if outcome == "correct": + return 0.0 + if outcome == "safe_stop": + key = ( + "claimed_failure_save_possible" + if save_possible + else "claimed_failure_no_save_possible" + ) + return _people_s(per[key], manual_key) + return _people_s(per[outcome], manual_key) + + +def _rows(arm: dict[str, Any]) -> Optional[list[dict[str, Any]]]: + p = SS / arm["dir"] / "items.csv" + if not p.is_file(): + return None + rows = sorted(csv.DictReader(p.open()), key=lambda r: int(r["item"])) + return rows if len(rows) == 100 else None + + +def _maint_events(row: dict[str, Any]) -> tuple[float, float]: + """(engineer s, operator s) for the repair events an item logged.""" + + eng = op = 0.0 + m = ASSUME["maintenance_people_min"] + for e in json.loads(row["human_events"] or "[]"): + if e.get("kind") == "script_review" and e.get("phase") == "maintenance": + eng += m["script_fix_review_engineer"] * 60 + elif e.get("kind") == "repair_approval": + op += ( + ASSUME["operator_s"]["repair_review"] + + ASSUME["operator_s"]["interrupt"] + ) + return eng, op + + +def mockmed_arm(arm: dict[str, Any]) -> Optional[dict[str, Any]]: + rows = _rows(arm) + if rows is None: + return None + oc = arm["outcome"] + rate = ASSUME["labor_usd_per_hour"] + cpu = ASSUME["compute_usd_per_hour"] / 3600.0 + + def cost(r: dict[str, Any], *, setup_item: bool = False) -> dict[str, float]: + model = float(r["cost_usd"] or 0) + events = json.loads(r["human_events"] or "[]") + if setup_item or any(e.get("phase") == "setup" for e in events): + model = 0.0 # authoring cost is setup, counted there + eng, op_maint = _maint_events(r) + op = operator_s(r, r[oc], "manual_entry_mockmed") + op_maint + return { + "model_usd": model, + "operator_s": op, + "engineer_s": eng, + "compute_usd": float(r["item_s"] or 0) * cpu, + } + + def usd(c: dict[str, float]) -> float: + return ( + c["model_usd"] + + c["compute_usd"] + + c["operator_s"] / 3600 * rate["operator"] + + c["engineer_s"] / 3600 * rate["engineer"] + ) + + control = [r for r in rows if r["release"] == "baseline" and r["fault"] == "ok"] + steady = { + k: statistics.mean(cost(r)[k] for r in control) + for k in ("model_usd", "operator_s", "engineer_s", "compute_usd") + } + steady_usd = usd(steady) + st_machine = steady["model_usd"] + steady["compute_usd"] + + setup_rows = [ + r + for r in rows + if any(e.get("phase") == "setup" for e in json.loads(r["human_events"] or "[]")) + ] + setup_model = sum(float(r["cost_usd"] or 0) for r in setup_rows) + setup_agent_min = sum(float(r["item_s"] or 0) for r in setup_rows) / 60 + sp = ASSUME["setup_people_min"][arm["setup"]] + setup = { + "model_usd": setup_model, + "agent_min": setup_agent_min, + "operator_min": sp["operator"], + "engineer_min": sp["engineer"], + } + setup["usd"] = ( + setup_model + + sp["operator"] / 60 * rate["operator"] + + sp["engineer"] / 60 * rate["engineer"] + ) + + faults = [r for r in rows if r["fault"] != "ok"] + fault_rows = [] + for r in faults: + c = cost(r) + fault_rows.append( + { + "item": int(r["item"]), + "fault": r["fault"], + "outcome": r[oc], + "excess_usd": usd(c) - steady_usd, + "excess_machine_usd": c["model_usd"] + c["compute_usd"] - st_machine, + "excess_people_usd": _split( + (c["operator_s"] - steady["operator_s"]) / 60, + (c["engineer_s"] - steady["engineer_s"]) / 60, + ), + "excess_people_min": ( + c["operator_s"] + + c["engineer_s"] + - steady["operator_s"] + - steady["engineer_s"] + ) + / 60, + } + ) + + m = ASSUME["maintenance_people_min"] + changes = [] + for rel in dict.fromkeys(r["release"] for r in rows if r["release"] != "baseline"): + block = [r for r in rows if r["release"] == rel and r["fault"] == "ok"] + outs = [r[oc] for r in block] + cs = [cost(r) for r in block] + op_s = sum(c["operator_s"] for c in cs) + eng_s = sum(c["engineer_s"] for c in cs) + model = sum(c["model_usd"] for c in cs) + comp = sum(c["compute_usd"] for c in cs) + expect_save = all(r["expect"] == "save" for r in block) + last_ok = outs[-1] == "correct" if expect_save else outs[-1] in ("safe_stop",) + fix = "" + if not last_ok: + # Still failing when the block ended: a person fixes it once. + if arm["kind"] == "program": + op_s += m["openadapt_reteach_operator"] * 60 + fix = "re-teach" + elif arm["kind"] == "script": + eng_s += m["script_fix_by_hand_engineer"] * 60 + fix = "engineer fixes the script" + else: + eng_s += m["agent_prompt_fix_engineer"] * 60 + fix = "engineer adjusts the prompt" + n = len(block) + excess_usd = ( + model + + comp + + op_s / 3600 * rate["operator"] + + eng_s / 3600 * rate["engineer"] + ) - n * steady_usd + changes.append( + { + "change": rel, + "drift": block[0]["drift"], + "items": n, + "outcomes": {o: outs.count(o) for o in sorted(set(outs))}, + "silent_wrong": outs.count("silent_wrong"), + "recovered_by_block_end": last_ok, + "people_fix_added": fix, + "operator_min": round(op_s / 60, 2), + "engineer_min": round(eng_s / 60, 2), + "model_usd": round(model, 4), + "excess_people_min": round( + (op_s + eng_s) / 60 + - n * (steady["operator_s"] + steady["engineer_s"]) / 60, + 2, + ), + "excess_usd": round(excess_usd, 3), + "excess_machine_usd": model + comp - n * st_machine, + "excess_people_usd": _split( + op_s / 60 - n * steady["operator_s"] / 60, + eng_s / 60 - n * steady["engineer_s"] / 60, + ), + } + ) + + outcomes = [r[oc] for r in rows] + timing = control + if arm.get("timing_dir") and (SS / arm["timing_dir"] / "items.csv").is_file(): + # A separate control-block pass at normal host load; the full run's times are not used. + timing = [ + r + for r in csv.DictReader((SS / arm["timing_dir"] / "items.csv").open()) + if r["release"] == "baseline" and r["fault"] == "ok" + ] + walls = sorted(float(r["item_s"]) for r in timing) + loads = [float(r["host_load_1m"] or 0) for r in timing] + measured = { + "items": len(rows), + "verified_of_saveable": sum( + 1 + for r in rows + if r[oc] == "correct" and r["expect"] == "save" and r["fault"] == "ok" + ), + "saveable": sum( + 1 for r in rows if r["expect"] == "save" and r["fault"] == "ok" + ), + "silent_wrong": outcomes.count("silent_wrong"), + "flagged_wrong": outcomes.count("flagged_wrong"), + "false_stops": sum( + 1 + for r in rows + if r[oc] in ("safe_stop", "false_alarm") + and r["expect"] == "save" + and r["fault"] == "ok" + ), + "faults_reported_as_success": sum(1 for r in faults if r[oc] == "silent_wrong"), + "stopped_when_no_save_possible": sum( + 1 for r in rows if r["expect"] == "stop" and r[oc] == "safe_stop" + ), + "model_usd_total": round(sum(float(r["cost_usd"] or 0) for r in rows), 4), + "model_calls_total": sum(int(r["model_calls"] or 0) for r in rows), + "control_median_s": round(statistics.median(walls), 2), + "control_median_host_load": round(statistics.median(loads), 1), + "control_model_usd_per_item": round(steady["model_usd"], 5), + } + return { + "arm": arm["arm"], + "label": arm["label"], + "kind": arm["kind"], + "source": f"silent_saves/results/1.1.0/{arm['dir']}", + "measured": measured, + "setup": setup, + "steady": steady, + "steady_usd": steady_usd, + "faults": fault_rows, + "changes": changes, + } + + +def _split(c_people_min_op: float, c_people_min_eng: float) -> float: + rate = ASSUME["labor_usd_per_hour"] + return ( + c_people_min_op / 60 * rate["operator"] + + c_people_min_eng / 60 * rate["engineer"] + ) + + +def project(a: dict[str, Any], n: int, people_factor: float = 1.0) -> dict[str, float]: + """Totals for n items. people_factor scales every people-time assumption.""" + + vol = ASSUME["volume"] + ch_rate = vol["ui_changes_per_1000_items"] / 1000 + f_rate = vol["server_faults_per_1000_items"] / 1000 + st = a["steady"] + s = a["setup"] + # (machine $, people $, people min) for each part + setup = ( + s["model_usd"], + _split(s["operator_min"], s["engineer_min"]), + s["operator_min"] + s["engineer_min"], + ) + per_item = ( + st["model_usd"] + st["compute_usd"], + _split(st["operator_s"] / 60, st["engineer_s"] / 60), + (st["operator_s"] + st["engineer_s"]) / 60, + ) + + def mean(xs: list[float]) -> float: + return statistics.mean(xs) if xs else 0.0 + + fault = ( + mean([f["excess_machine_usd"] for f in a["faults"]]), + mean([f["excess_people_usd"] for f in a["faults"]]), + mean([f["excess_people_min"] for f in a["faults"]]), + ) + change = ( + mean([c["excess_machine_usd"] for c in a["changes"]]), + mean([c["excess_people_usd"] for c in a["changes"]]), + mean([c["excess_people_min"] for c in a["changes"]]), + ) + parts = { + "setup": (1.0, setup), + "per_item": (n, per_item), + "faults": (n * f_rate, fault), + "ui_changes": (n * ch_rate, change), + } + out: dict[str, Any] = {} + total = people_min = 0.0 + for k, (mult, (mach, ppl, mins)) in parts.items(): + usd = mult * (mach + people_factor * ppl) + out[f"{k}_usd"] = round(usd, 2) + total += usd + people_min += mult * mins * people_factor + out.update( + total_usd=round(total, 2), + people_min=round(people_min, 1), + model_usd=round(s["model_usd"] + n * st["model_usd"], 2), + silent_wrong_expected=round( + n + * f_rate + * mean( + [1.0 if f["outcome"] == "silent_wrong" else 0.0 for f in a["faults"]] + ) + + n * ch_rate * mean([c["silent_wrong"] for c in a["changes"]]), + 2, + ), + ) + return out + + +# -- OpenEMR: the lifecycle soak's ledgers ------------------------------------------ + +LIFE = BENCH / "lifecycle" / "results" +#: (arm in the ledger, ledger dir, setup key, label) +OPENEMR_ARMS: list[dict[str, Any]] = [ + { + "arm": "openadapt-production-relabel", + "dir": "production-2026-10-24/after", + "setup": "openadapt_regulated_openemr", + "kind": "program", + "label": "3. OpenAdapt, Regulated + problems_only, durable, pipelined read-back, relabel repair " + "(engine of measure/production-lifecycle; the API step is refused on this form, so the GUI save runs)", + }, + { + "arm": "openadapt-production", + "dir": "production-2026-10-24/before", + "setup": "openadapt_regulated_openemr", + "kind": "program", + "label": "3 on main. Same configuration, current main's engine", + }, + { + "arm": "agent_tools", + "dir": "economics-2026-10-24/agent_tools", + "setup": "agent_tools", + "kind": "agent", + "label": "1. Agent with structured tools and REST read-back (Claude Sonnet 5.5)", + }, + { + "arm": "agent_script", + "dir": "economics-2026-10-24/agent_script", + "setup": "agent_script", + "kind": "script", + "label": "2. Agent-written Playwright + REST script with read-back and duplicate check, agent-maintained (Claude Sonnet 5.5)", + }, + { + "arm": "human_script", + "dir": "economics-2026-10-24/human_script", + "setup": "human_script", + "kind": "script", + "label": "2h. Hand-written script, same checks, not maintained during the run", + }, + { + "arm": "cua_alone", + "dir": "economics-2026-10-24/cua_alone", + "setup": "cua_alone", + "kind": "agent", + "label": "4. Computer-use agent, screenshots only (Claude Sonnet 5.5), first 12 items", + }, +] + + +def openemr_arm(spec: dict[str, Any]) -> Optional[dict[str, Any]]: + sys.path.insert(0, str(BENCH)) + from lifecycle.arms import economics as econ + + ledger = LIFE / spec["dir"] / "ledger.jsonl" + if not ledger.is_file(): + return None + rows = econ.read_rows([ledger]) + res = econ.compute(rows) + a = next((x for x in res["arms"] if x["arm"] == spec["arm"]), None) + if a is None: + return None + # A run that ended short of its planned items is labelled, never silently + # shrunk: its rates come from fewer items than the stream it was started on. + partial = bool(a.get("short_run")) + label = spec["label"] + if partial: + label += f" [short run: {a['items']} of {a['planned_items']} planned items; {a['short_why']}]" + m = a["measured"] + n = a["items"] + op = ASSUME["operator_s"] + mine = [r for r in rows if r.get("arm") == spec["arm"]] + fixes = [ + r + for r in mine + if r.get("type") == "phase" + and r.get("stage") == "maintenance" + and r.get("phase") in ("fix_script",) + ] + reviews = [ + r + for r in mine + if r.get("type") == "operator" and r.get("kind") == "repair_review" + ] + setup_walls = [ + float(r.get("wall_s", 0) or 0) + for r in mine + if r.get("type") == "phase" and r.get("stage") == "setup" + ] + sp = ASSUME["setup_people_min"][spec["setup"]] + # Script fixes are reviewed by an engineer (assumption), not charged as an operator repair review. + eng_maint_min = ( + len(fixes) * ASSUME["maintenance_people_min"]["script_fix_review_engineer"] + if spec["kind"] == "script" + else 0.0 + ) + op_maint_min = 0.0 if spec["kind"] == "script" else m["maintenance_human_min"] + if spec["kind"] == "script" and spec["arm"] == "human_script" and a["drift_events"]: + # Not maintained during the run: an engineer fixes it once per change (prices.json, 30 min). + broke = any( + r.get("type") == "item_result" and (r.get("detail") or {}).get("broke") + for r in mine + ) + eng_maint_min = ( + ASSUME["maintenance_people_min"]["script_fix_by_hand_engineer"] + if broke + else 0.0 + ) + d = max(a["drift_events"], 1) + sw = a["silent_wrong"] + cleanup_s = sw * ( + op["reconcile"] + op["manual_entry_openemr"] + 2 * op["interrupt"] + ) + item_op_min = m["operator_min"] - ( + len(reviews) * (op["repair_review"] + op["interrupt"]) / 60 + if spec["kind"] == "script" + else 0.0 + ) + out = { + "arm": spec["arm"], + "label": label, + "kind": spec["kind"], + "source": f"lifecycle/results/{spec['dir']}", + "partial": partial, + "items": n, + "verified": a["verified"], + "automated": a["automated"], + "by_hand": a["by_hand"], + "outcomes": a["outcomes"], + "silent_wrong": sw, + "false_stop": a["false_stop"], + "false_stop_of": a["false_stop_denominator"], + "drift_events": a["drift_events"], + "median_s": m["median_wall_s_automated"], + "p90_s": m["p90_wall_s_automated"], + "model_usd_total": round( + m["item_model_usd"] + m["setup_model_usd"] + m["maintenance_model_usd"], 4 + ), + "setup": { + "model_usd": m["setup_model_usd"], + "agent_min": round(sum(setup_walls) / 60, 1), + "operator_min": sp["operator"], + "engineer_min": sp["engineer"], + }, + "per_item": { + "model_usd": m["item_model_usd"] / n, + "compute_usd": m["compute_usd"] / n, + "operator_min": item_op_min / n, + }, + "per_change": { + "model_usd": m["maintenance_model_usd"] / d, + "operator_min": op_maint_min / d, + "engineer_min": eng_maint_min / d, + "fixes": len(fixes), + }, + "per_item_cleanup_min": cleanup_s / 60 / n, + } + if not a["drift_events"]: + # The run never reached the app change: its fixes were first-use debugging, so they are + # setup, and maintenance after a change was not measured. + out["setup"]["model_usd"] += m["maintenance_model_usd"] + out["setup"]["engineer_min"] += eng_maint_min + out["setup"]["debug_fixes"] = len(fixes) + out["per_change"] = { + "model_usd": 0.0, + "operator_min": 0.0, + "engineer_min": 0.0, + "fixes": 0, + "not_measured": True, + } + return out + + +def project_openemr(a: dict[str, Any], n: int, f: float = 1.0) -> dict[str, float]: + rate = ASSUME["labor_usd_per_hour"] + ch = ASSUME["volume"]["ui_changes_per_1000_items"] / 1000 * n + s, pi, pc = a["setup"], a["per_item"], a["per_change"] + op_usd = rate["operator"] / 60 + eng_usd = rate["engineer"] / 60 + setup_usd = s["model_usd"] + f * ( + s["operator_min"] * op_usd + s["engineer_min"] * eng_usd + ) + items_usd = n * ( + pi["model_usd"] + pi["compute_usd"] + f * pi["operator_min"] * op_usd + ) + change_usd = ch * ( + pc["model_usd"] + + f * (pc["operator_min"] * op_usd + pc["engineer_min"] * eng_usd) + ) + cleanup_usd = n * f * a["per_item_cleanup_min"] * op_usd + people = f * ( + s["operator_min"] + + s["engineer_min"] + + n * pi["operator_min"] + + ch * (pc["operator_min"] + pc["engineer_min"]) + + n * a["per_item_cleanup_min"] + ) + return { + "setup_usd": round(setup_usd, 2), + "per_item_usd": round(items_usd, 2), + "ui_changes_usd": round(change_usd, 2), + "silent_wrong_cleanup_usd": round(cleanup_usd, 2), + "total_usd": round(setup_usd + items_usd + change_usd + cleanup_usd, 2), + "people_min": round(people, 1), + "model_usd": round( + s["model_usd"] + n * pi["model_usd"] + ch * pc["model_usd"], 2 + ), + "silent_wrong_expected": round(n * a["silent_wrong"] / a["items"], 2) + if a["items"] + else 0.0, + } + + +def render_openemr(arms: list[dict[str, Any]]) -> str: + L = [ + "## OpenEMR referral intake (seed 7, 55 items, one app change: the save button relabeled at item 26)", + "", + "| Arm | Items | Verified in the end | Automated (no hand entry) | Silent wrong | False stops | Median s/item | Model $/item | Operator min/item | Maintenance for the change (people min / model $) | Setup people min | Setup agent min / $ |", + "|---|---|---|---|---|---|---|---|---|---|---|---|", + ] + for a in arms: + pc, s = a["per_change"], a["setup"] + L.append( + f"| {a['label']} | {a['items']} | {a['verified']} | {a['automated']} | {a['silent_wrong']} | " + f"{a['false_stop']} of {a['false_stop_of']} | {_f(a['median_s'], 1)} | {_f(a['per_item']['model_usd'], 3)} | " + f"{_f(a['per_item']['operator_min'], 2)} | " + + ( + "not reached | " + if pc.get("not_measured") and a["kind"] != "manual" + else f"{_f(pc['operator_min'] + pc['engineer_min'], 1)} / {_f(pc['model_usd'], 2)} | " + ) + + f"{_f(s['operator_min'] + s['engineer_min'], 0)} | {_f(s['agent_min'], 1)} / {_f(s['model_usd'], 2)} |" + ) + L += [ + "", + "### Totals (people time priced at the assumed rates; x0.5 and x2 in brackets)", + "", + "| Arm | " + + " | ".join(f"{n} items: total $ | people min" for n in SIZES) + + " |", + "|---|" + "---|---|" * len(SIZES), + ] + for a in arms: + cells = [] + for n in SIZES: + t, lo, hi = ( + a["totals"][str(n)], + a["totals_people_x0.5"][str(n)], + a["totals_people_x2"][str(n)], + ) + cells += [ + f"{_f(t['total_usd'], 0)} [{_f(lo['total_usd'], 0)}-{_f(hi['total_usd'], 0)}]", + _f(t["people_min"], 0), + ] + L.append(f"| {a['arm']} | " + " | ".join(cells) + " |") + return "\n".join(L) + "\n" + + +def main(argv: Optional[list[str]] = None) -> int: + out = HERE / "results" + out.mkdir(exist_ok=True) + arms = [x for x in (mockmed_arm(a) for a in MOCKMED_ARMS) if x] + missing = [ + a["arm"] for a in MOCKMED_ARMS if not any(x["arm"] == a["arm"] for x in arms) + ] + for a in arms: + a["totals"] = {str(n): project(a, n) for n in SIZES} + a["totals_people_x0.5"] = {str(n): project(a, n, 0.5) for n in SIZES} + a["totals_people_x2"] = {str(n): project(a, n, 2.0) for n in SIZES} + oe = [x for x in (openemr_arm(a) for a in OPENEMR_ARMS) if x] + op = ASSUME["operator_s"] + manual_min = (op["manual_entry_openemr"] + op["interrupt"]) / 60 + oe.append( + { + "arm": "all_manual", + "label": "Reference: a person enters every item (cost model V1)", + "kind": "manual", + "source": "cost model", + "partial": False, + "items": None, + "verified": None, + "automated": 0, + "by_hand": None, + "outcomes": {}, + "silent_wrong": 0, + "false_stop": None, + "false_stop_of": None, + "drift_events": 0, + "median_s": manual_min * 60, + "p90_s": None, + "model_usd_total": 0.0, + "setup": { + "model_usd": 0.0, + "agent_min": 0.0, + "operator_min": 0.0, + "engineer_min": 0.0, + }, + "per_item": { + "model_usd": 0.0, + "compute_usd": 0.0, + "operator_min": manual_min, + }, + "per_change": { + "model_usd": 0.0, + "operator_min": 0.0, + "engineer_min": 0.0, + "fixes": 0, + }, + "per_item_cleanup_min": 0.0, + } + ) + for a in oe: + a["totals"] = {str(n): project_openemr(a, n) for n in SIZES} + a["totals_people_x0.5"] = {str(n): project_openemr(a, n, 0.5) for n in SIZES} + a["totals_people_x2"] = {str(n): project_openemr(a, n, 2.0) for n in SIZES} + result = { + "assumptions": ASSUME["version"], + "openemr": oe, + "mockmed": arms, + "mockmed_missing": missing, + "openemr_missing": [ + a["arm"] for a in OPENEMR_ARMS if not any(x["arm"] == a["arm"] for x in oe) + ], + } + (out / "totals.json").write_text(json.dumps(result, indent=1, default=str) + "\n") + text = render_openemr(oe) + "\n" + render(result) + (out / "TABLE.md").write_text(text) + print(text) + return 0 + + +def _f(x: Any, nd: int = 2) -> str: + return f"{x:,.{nd}f}" if isinstance(x, (int, float)) else str(x) + + +def render(result: dict[str, Any]) -> str: + arms = result["mockmed"] + L = [ + "## MockMed, fault set 1.1.0 (100 items, 10 UI changes, 5 server faults)", + "", + "| Arm | Verified (of 90) | Silent wrong | False stops | Faults claimed saved (of 5) | Stopped when no save possible (of 5) | Control s/item (load) | Model $/item | Setup people min | Setup agent min / $ |", + "|---|---|---|---|---|---|---|---|---|---|", + ] + for a in arms: + m, s = a["measured"], a["setup"] + L.append( + f"| {a['label']} | {m['verified_of_saveable']} | {m['silent_wrong']} | {m['false_stops']} | " + f"{m['faults_reported_as_success']} | {m['stopped_when_no_save_possible']} | " + f"{_f(m['control_median_s'], 1)} ({_f(m['control_median_host_load'], 0)}) | {_f(m['control_model_usd_per_item'], 4)} | " + f"{_f(s['operator_min'] + s['engineer_min'], 0)} | {_f(s['agent_min'], 1)} / {_f(s['model_usd'], 3)} |" + ) + L += [ + "", + "### Each UI change: excess people minutes (silent wrongs in the block)", + "", + "| Change | " + " | ".join(a["arm"] for a in arms) + " |", + "|---|" + "---|" * len(arms), + ] + for i, c in enumerate(arms[0]["changes"]): + cells = [] + for a in arms: + ch = a["changes"][i] + cells.append( + f"{_f(ch['excess_people_min'], 1)}" + + (f" ({ch['silent_wrong']} SW)" if ch["silent_wrong"] else "") + + (" +fix" if ch["people_fix_added"] else "") + ) + L.append(f"| {c['change']} | " + " | ".join(cells) + " |") + L += [ + "", + "### Totals (people time priced at the assumed rates; x0.5 and x2 in brackets)", + "", + "| Arm | " + + " | ".join(f"{n} items: total $ | people min | silent wrongs" for n in SIZES) + + " |", + "|---|" + "---|---|---|" * len(SIZES), + ] + for a in arms: + cells = [] + for n in SIZES: + t, lo, hi = ( + a["totals"][str(n)], + a["totals_people_x0.5"][str(n)], + a["totals_people_x2"][str(n)], + ) + cells += [ + f"{_f(t['total_usd'], 0)} [{_f(lo['total_usd'], 0)}-{_f(hi['total_usd'], 0)}]", + _f(t["people_min"], 0), + _f(t["silent_wrong_expected"], 1), + ] + L.append(f"| {a['arm']} | " + " | ".join(cells) + " |") + if result.get("mockmed_missing"): + L += ["", f"Not yet run: {', '.join(result['mockmed_missing'])}."] + return "\n".join(L) + "\n" + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:])) diff --git a/packages/flow/benchmark/lifecycle/.gitignore b/packages/flow/benchmark/lifecycle/.gitignore new file mode 100644 index 000000000..6b79aefcf --- /dev/null +++ b/packages/flow/benchmark/lifecycle/.gitignore @@ -0,0 +1,2 @@ +results/spend.lock +results/economics-*/*/arms/ diff --git a/packages/flow/benchmark/lifecycle/arms/agent_script.py b/packages/flow/benchmark/lifecycle/arms/agent_script.py new file mode 100644 index 000000000..3d6d934ed --- /dev/null +++ b/packages/flow/benchmark/lifecycle/arms/agent_script.py @@ -0,0 +1,373 @@ +"""Arm ``agent_script``: an agent writes a Playwright + REST script, then maintains it. + +Setup: the agent gets the same demonstration and documents as OpenAdapt (the +recorded steps, the procedure, the REST read) and the live form to inspect. It +writes a script with an equivalent check (a REST read-back after the save, and +a duplicate check before it), tries it in dry runs (the save request is +blocked at the network layer), and submits it. A person then reviews the script +once before it goes live: the setup phase is named ``compile`` so the cost +model charges it exactly what it charges for reviewing a compiled program. + +Per item: the script runs with no model calls. A business stop (duplicate, +missing date of birth) goes to the operator. When the script breaks (an +exception, or a reason starting "UNEXPECTED"), the agent is called again with +the script, the failure and the live form, writes a fix, dry-runs it on this +item, and the operator reviews the fix (``repair_review``) before the item is +run again. That maintenance is charged as maintenance, with its tokens and +the review, not as per-item cost. +""" + +from __future__ import annotations + +import json +import time +from pathlib import Path +from typing import Any, Optional + +from ..arm import Env, ItemResult, Operator, OperatorRequest, WorkItem +from . import budget, scripts +from .agent_tools import READ_JS +from .openemr_ui import SOP, OpenEMRSession, RestReader + +CONTRACT = '''\ +Write one Python module that defines + + def register(ctx, referral: dict) -> dict + +returning {"result": "SAVED" | "NOT SAVED", "reason": ""}. +referral has the keys title, fname, lname, DOB, sex, street, city, +state_label, postal_code, phone_home, email, country_code, exactly as the +referral states them (values may be missing, blank or malformed). +ctx.page is a Playwright sync Page (signed in, on a blank New Patient form); +ctx.form is the form's Frame; ctx.find_patients(fname=, lname=, DOB=) is the +REST search; ctx.commit is False in a dry run; ctx.log(msg) records a line. +Only re, time, datetime and json may be imported; no file, OS or eval access. +Required behaviour: +- Apply the procedure's rules (normalise names and dates; stop on a missing + date of birth; check for an existing chart before saving; never create a + duplicate). +- After the confirmed save, read the chart back with ctx.find_patients and + return SAVED only if exactly one chart matches the referral's values. +- In a dry run (ctx.commit is False), do everything including the confirm + click (the save request is blocked for you), skip the read-back and return + {"result": "NOT SAVED", "reason": "DRY RUN reached confirm"}. +- When the screen is not what the script expects (a control or dialog is + missing, a wait times out), return NOT SAVED with a reason starting + "UNEXPECTED:" rather than guessing.''' + +SYSTEM = """\ +You are a careful automation engineer. You write and maintain a Playwright \ +script that registers referred patients in a clinic's OpenEMR, against a local \ +synthetic test copy. Use the tools to inspect the live form, write the script, \ +and dry-run it. Submit only a script whose dry runs pass. Be economical: \ +a few dry runs, then submit.""" + +TOOLS: list[dict[str, Any]] = [ + { + "name": "read_screen", + "description": "Open a blank New Patient form and list the visible controls, buttons and dialog text of every frame.", + "input_schema": {"type": "object", "properties": {}, "additionalProperties": False}, + }, + { + "name": "find_patients", + "description": "REST search of existing charts by fname, lname, DOB.", + "input_schema": { + "type": "object", + "properties": {"fname": {"type": "string"}, "lname": {"type": "string"}, "DOB": {"type": "string"}}, + "additionalProperties": False, + }, + }, + { + "name": "dry_run", + "description": "Run the given script on a fresh blank form with the save request blocked. Returns its result, reason, log, and the controls on screen at the end.", + "input_schema": { + "type": "object", + "properties": { + "source": {"type": "string", "description": "the whole module"}, + "referral": {"type": "object", "description": "a synthetic referral dict to try"}, + }, + "required": ["source", "referral"], + "additionalProperties": False, + }, + }, + { + "name": "submit", + "description": "Submit the final script (the whole module).", + "input_schema": { + "type": "object", + "properties": {"source": {"type": "string"}, "notes": {"type": "string"}}, + "required": ["source"], + "additionalProperties": False, + }, + }, +] + + +class AgentScriptArm: + name = "agent_script" + + def __init__(self, options: dict[str, Any]) -> None: + self.name = str(options.get("name", self.name)) + self.model = options.get("model", "claude-opus-5-5") + self.effort = options.get("effort", "high") + self.author_budget_usd = float(options.get("author_budget_usd", 4.0)) + self.fix_budget_usd = float(options.get("fix_budget_usd", 2.0)) + self.max_turns = int(options.get("max_turns", 30)) + self.headless = bool(options.get("headless", True)) + self.session: Optional[OpenEMRSession] = None + self.rest: Optional[RestReader] = None + self.source = "" + self.version = 0 + self.dir: Optional[Path] = None + + def describe(self) -> dict[str, Any]: + return { + "arm": self.name, + "kind": "agent-written Playwright + REST script with read-back; agent maintains it on failure", + "model": self.model, + "effort": self.effort, + "author_budget_usd": self.author_budget_usd, + "fix_budget_usd": self.fix_budget_usd, + "prices_as_of": budget.prices()["as_of"], + "per_item_model_calls": 0, + "sandbox": "static import/name check; dry runs block the save request", + } + + # -- authoring and maintenance --------------------------------------------------- + + def _read_screen(self) -> dict[str, Any]: + self.session.blank_form() + frames = [] + n = 1 + for frame in self.session.page.frames: + try: + got = frame.evaluate(READ_JS, n) + except Exception: # noqa: BLE001 + continue + n = got.pop("next") + if got["controls"] or got["buttons"] or got.get("notices"): + frames.append({"frame": (frame.url or "").split("?")[0].rsplit("/", 1)[-1], **got}) + return {"frames": frames} + + def _end_screen(self) -> list[dict[str, Any]]: + out = [] + for frame in self.session.page.frames: + try: + got = frame.evaluate(READ_JS, 1) + except Exception: # noqa: BLE001 + continue + got.pop("next", None) + if got["buttons"] or got.get("notices"): + out.append({"frame": (frame.url or "").split("?")[0].rsplit("/", 1)[-1], + "buttons": got["buttons"][:30], "notices": got.get("notices", [])}) + return out + + def _session_loop(self, opening: str, meter: budget.Meter, trace: list[dict[str, Any]]) -> tuple[Optional[str], list[dict[str, Any]]]: + """Run the authoring agent until it submits. Returns (source, dry runs).""" + + client = budget.client(self.model) + messages: list[dict[str, Any]] = [{"role": "user", "content": opening}] + dry_runs: list[dict[str, Any]] = [] + for turn in range(1, self.max_turns + 1): + approx_in = 8000 + 3000 * turn + meter.before_call(self.model, budget.max_call_usd(self.model, approx_in, 8000)) + with client.messages.stream( + model=self.model, + max_tokens=32000, + system=SYSTEM, + tools=TOOLS, + output_config={"effort": self.effort}, + cache_control={"type": "ephemeral"}, + messages=messages, + ) as stream: + response = stream.get_final_message() + meter.record(self.model, response.usage) + messages.append({"role": "assistant", "content": response.content}) + uses = [b for b in response.content if b.type == "tool_use"] + trace.append({"turn": turn, "stop": response.stop_reason, "tools": [b.name for b in uses]}) + if response.stop_reason == "refusal": + return None, dry_runs + if not uses: + messages.append({"role": "user", "content": "Use the tools; finish with submit."}) + continue + results = [] + for b in uses: + args = dict(b.input) + if b.name == "submit": + try: + scripts.check_source(args["source"]) + except Exception as exc: # noqa: BLE001 + results.append({"type": "tool_result", "tool_use_id": b.id, "is_error": True, + "content": f"rejected: {exc}"}) + continue + return args["source"], dry_runs + try: + if b.name == "read_screen": + out: Any = self._read_screen() + elif b.name == "find_patients": + out = self.rest.find(**args) + elif b.name == "dry_run": + r = scripts.run(args["source"], self.session, self.rest, args.get("referral") or {}, commit=False) + out = {"result": r.result, "reason": r.reason, "log": r.log[-30:], + "save_requests_blocked": r.blocked_saves, "seconds": round(r.seconds, 1), + "screen_at_end": self._end_screen()} + dry_runs.append({k: out[k] for k in ("result", "reason", "save_requests_blocked")}) + else: + raise KeyError(b.name) + content = json.dumps(out, default=str)[:30_000] + results.append({"type": "tool_result", "tool_use_id": b.id, "content": content}) + except Exception as exc: # noqa: BLE001 + results.append({"type": "tool_result", "tool_use_id": b.id, "is_error": True, + "content": f"Error: {exc}"[:600]}) + messages.append({"role": "user", "content": results}) + return None, dry_runs + + def _save_version(self, source: str, why: str) -> None: + self.version += 1 + self.source = source + assert self.dir is not None + (self.dir / f"script_v{self.version}.py").write_text(f"# {why}\n" + source) + + def setup(self, env: Env, operator: Operator) -> list[dict[str, Any]]: + self.session = OpenEMRSession(env, headless=self.headless) + self.rest = RestReader(env) + self.dir = Path(env.workdir) / self.name + self.dir.mkdir(parents=True, exist_ok=True) + self.session.start() + self.session.sign_in() + meter = budget.Meter(f"{self.name} author", self.author_budget_usd) + trace: list[dict[str, Any]] = [] + t0 = time.time() + opening = "\n\n".join([ + "Write the registration script.", SOP, scripts.DEMONSTRATION, scripts.API_DOC, CONTRACT, + ]) + source, dry = None, [] + error = "" + try: + source, dry = self._session_loop(opening, meter, trace) + except budget.SpendCapReached: + raise + except Exception as exc: # noqa: BLE001 + error = f"{type(exc).__name__}: {exc}"[:500] + wall = time.time() - t0 + (self.dir / "author_trace.json").write_text(json.dumps({"trace": trace, "dry_runs": dry, "error": error}, indent=1)) + if source is None: + raise RuntimeError(f"the agent did not submit a script: {error or 'no submit'}") + self._save_version(source, "authored by the agent at setup") + return [ + { + "phase": "author_script", + "wall_s": round(wall, 2), + "attended_s": 0.0, + **budget.summarize(meter), + "detail": {"dry_runs": dry, "version": self.version}, + }, + { + # The person reads the script once before it goes live; charged + # like the review of a compiled program. + "phase": "compile", + "wall_s": 0.0, + "attended_s": 0.0, + "model_calls": 0, + "cost_usd": 0.0, + "detail": {"what": "a person reviews the generated script before it goes live"}, + }, + ] + + def _maintain(self, item: WorkItem, failure: scripts.RunOutcome, operator: Operator) -> tuple[bool, dict[str, Any]]: + """Ask the agent for a fix; the operator reviews it. Returns (accepted, phase).""" + + meter = budget.Meter(f"{self.name} fix {item.item_id}", self.fix_budget_usd) + trace: list[dict[str, Any]] = [] + t0 = time.time() + opening = "\n\n".join([ + "The live script failed on a referral. Fix it, dry-run the fix on this referral, and submit the whole module.", + "Failure:\n" + failure.reason + "\nScript log:\n" + "\n".join(failure.log[-30:]), + "The referral:\n" + json.dumps(item.params), + "The current script:\n```python\n" + self.source + "\n```", + CONTRACT, SOP, + ]) + source, dry, error = None, [], "" + try: + source, dry = self._session_loop(opening, meter, trace) + except budget.SpendCapReached: + raise + except Exception as exc: # noqa: BLE001 + error = f"{type(exc).__name__}: {exc}"[:500] + probes_ok = bool(source) and bool(dry) and dry[-1]["result"] != "ERROR" and not dry[-1]["reason"].upper().startswith("UNEXPECTED") + decision = operator.ask( + OperatorRequest( + kind="repair_review", + item_id=item.item_id, + reason=f"script fix after: {failure.reason.splitlines()[-1][:200] if failure.reason else ''}", + surface="agent_script maintenance", + detail={"probes_passed": probes_ok, "dry_runs": dry[-3:], "error": error}, + ) + ) + accepted = source is not None and decision.action == "accept_repair" + if accepted: + self._save_version(source, f"fix after {item.item_id}: {failure.reason.splitlines()[-1][:120]}") + phase = { + "phase": "fix_script", + "item_id": item.item_id, + "wall_s": round(time.time() - t0, 2), + "attended_s": 0.0, + **budget.summarize(meter), + "detail": {"accepted": accepted, "probes_passed": probes_ok, "dry_runs": dry, + "version": self.version, "error": error, "turns": len(trace)}, + } + return accepted, phase + + # -- per item ------------------------------------------------------------------- + + def run_item(self, item: WorkItem, env: Env, operator: Operator) -> ItemResult: + assert self.session is not None and self.rest is not None + t0 = time.time() + phases: list[dict[str, Any]] = [] + attempts: list[dict[str, Any]] = [] + out = scripts.run(self.source, self.session, self.rest, item.params, commit=True) + attempts.append({"version": self.version, "result": out.result, "reason": out.reason[-300:], "s": round(out.seconds, 2)}) + if out.broke: + accepted, phase = self._maintain(item, out, operator) + phases.append(phase) + if accepted: + out = scripts.run(self.source, self.session, self.rest, item.params, commit=True) + attempts.append({"version": self.version, "result": out.result, "reason": out.reason[-300:], "s": round(out.seconds, 2)}) + wall = time.time() - t0 + if out.result == "SAVED": + claimed, status = "done", f"SAVED: {out.reason}" + elif out.result == "NOT SAVED": + claimed, status = "stopped", f"NOT SAVED: {out.reason}" + else: + claimed, status = "error", out.reason[-300:] + if claimed != "done": + operator.ask(OperatorRequest(kind="stop", item_id=item.item_id, reason=status[:300], surface="script result")) + maint_cost = sum(p.get("cost_usd", 0.0) for p in phases) + return ItemResult( + item_id=item.item_id, + claimed=claimed, + product_status=status[:500], + wall_s=round(wall, 2), + model_calls=0, + cost_usd=0.0, + evidence=f"{self.name}/", + phases=phases, + detail={ + "attempts": attempts, + "script_version": self.version, + "maintenance_cost_usd": round(maint_cost, 6), + "usage": [], + "compute_usd": round(budget.compute_usd(sum(a["s"] for a in attempts)), 6), + }, + ) + + def after_drift(self, event: dict[str, Any], env: Env, operator: Operator) -> list[dict[str, Any]]: + return [] + + def close(self) -> None: + if self.session is not None: + self.session.close() + + +def factory(options: dict[str, Any]) -> AgentScriptArm: + return AgentScriptArm(options) diff --git a/packages/flow/benchmark/lifecycle/arms/agent_tools.py b/packages/flow/benchmark/lifecycle/arms/agent_tools.py new file mode 100644 index 000000000..06252c16e --- /dev/null +++ b/packages/flow/benchmark/lifecycle/arms/agent_tools.py @@ -0,0 +1,369 @@ +"""Arm ``agent_tools``: the same frontier model with structured tools and read-back. + +This is the strong agent, not a strawman. Instead of screenshots it gets what +OpenAdapt's runtime gets: + +- ``read_screen``: the visible controls of every frame (the DOM and + accessibility view): id, label, type, value, required, options; the visible + buttons and links; the text of any dialog or alert, including OpenEMR's + possible-duplicate popup. +- ``set_field`` / ``click``: act on a control by the reference ``read_screen`` + gave it. +- ``find_patients``: a REST search of the record (the clerk's own API access), + so it can check for an existing chart before it saves and confirm the chart + after it saves. +- ``finish``: its claim, which the oracle checks independently. + +Same procedure, model, effort and per-item budget style as ``cua_alone``. +""" + +from __future__ import annotations + +import json +import time +from pathlib import Path +from typing import Any + +from ..arm import Env, ItemResult, Operator, OperatorRequest, WorkItem +from . import budget +from .openemr_ui import SOP, OpenEMRSession, RestReader + +SYSTEM = """\ +You register patients in a clinic's OpenEMR through tools. You cannot see the \ +screen; read_screen gives you its controls and any dialog text. References from \ +read_screen are valid until the next read_screen. After a click that saves or \ +opens a dialog, call read_screen again. Use find_patients to look the person up \ +in the record before you save and to confirm the chart after you save. Call \ +finish exactly once at the end.""" + +TOOLS: list[dict[str, Any]] = [ + { + "name": "read_screen", + "description": "List the visible form controls, buttons and dialog text on the OpenEMR page, with a ref for each control.", + "input_schema": {"type": "object", "properties": {}, "additionalProperties": False}, + }, + { + "name": "set_field", + "description": "Set a text field, date field or select list (by its visible option text) to a value.", + "input_schema": { + "type": "object", + "properties": {"ref": {"type": "string"}, "value": {"type": "string"}}, + "required": ["ref", "value"], + "additionalProperties": False, + }, + "strict": True, + }, + { + "name": "click", + "description": "Click a button, link or checkbox by its ref.", + "input_schema": { + "type": "object", + "properties": {"ref": {"type": "string"}}, + "required": ["ref"], + "additionalProperties": False, + }, + "strict": True, + }, + { + "name": "click_text", + "description": "Click the first visible element whose text is exactly this (any frame), e.g. a section header.", + "input_schema": { + "type": "object", + "properties": {"text": {"type": "string"}}, + "required": ["text"], + "additionalProperties": False, + }, + "strict": True, + }, + { + "name": "find_patients", + "description": "Search existing patient charts through OpenEMR's REST API. Any of fname, lname, DOB (YYYY-MM-DD). Returns matching charts.", + "input_schema": { + "type": "object", + "properties": { + "fname": {"type": "string"}, + "lname": {"type": "string"}, + "DOB": {"type": "string"}, + }, + "additionalProperties": False, + }, + }, + { + "name": "finish", + "description": "Report the outcome. result is SAVED only if exactly one new chart for this referral exists now.", + "input_schema": { + "type": "object", + "properties": { + "result": {"type": "string", "enum": ["SAVED", "NOT SAVED"]}, + "reason": {"type": "string"}, + }, + "required": ["result", "reason"], + "additionalProperties": False, + }, + "strict": True, + }, +] + +READ_JS = r""" +(start) => { + let n = start; + const vis = e => { const r = e.getBoundingClientRect(); const s = getComputedStyle(e); + return r.width > 0 && r.height > 0 && s.visibility !== 'hidden' && s.display !== 'none'; }; + const labelOf = e => { + if (e.id) { const l = document.querySelector(`label[for="${e.id}"]`); if (l && l.innerText.trim()) return l.innerText.trim(); } + if (e.getAttribute('aria-label')) return e.getAttribute('aria-label'); + if (e.title) return e.title; + const td = e.closest('td'); if (td && td.previousElementSibling) { const t = td.previousElementSibling.innerText.trim(); if (t) return t.slice(0,60); } + const div = e.closest('.form-group, .col, div'); if (div) { const l = div.querySelector('label'); if (l && l.innerText.trim()) return l.innerText.trim().slice(0,60); } + return e.placeholder || ''; + }; + const out = {controls: [], buttons: []}; + document.querySelectorAll('[data-ar]').forEach(e => e.removeAttribute('data-ar')); + document.querySelectorAll('input, select, textarea').forEach(e => { + if (!vis(e) || e.type === 'hidden') return; + if (e.type === 'button' || e.type === 'submit') return; + const ref = String(n++); e.setAttribute('data-ar', ref); + const c = {ref, id: e.id || e.name, label: labelOf(e), type: e.tagName === 'SELECT' ? 'select' : (e.type || e.tagName.toLowerCase()), value: e.tagName === 'SELECT' ? (e.selectedOptions[0] ? e.selectedOptions[0].text : '') : (e.type === 'checkbox' ? String(e.checked) : e.value)}; + if (e.required) c.required = true; + if (e.tagName === 'SELECT') { const o = [...e.options].map(x => x.text.trim()).filter(Boolean); c.options = o.length <= 15 ? o : `${o.length} options; set by visible text`; } + out.controls.push(c); + }); + document.querySelectorAll('button, input[type=button], input[type=submit], a.btn, a[onclick], a[href^="#"], [role=button], [role=tab], [data-toggle], [data-bs-toggle], [aria-expanded], summary').forEach(e => { + if (!vis(e)) return; + const t = (e.innerText || e.value || e.title || '').trim(); if (!t) return; + const ref = String(n++); e.setAttribute('data-ar', ref); + out.buttons.push({ref, text: t.slice(0,60), id: e.id || undefined}); + }); + const alerts = [...document.querySelectorAll('.alert, .modal.show, .error, .text-danger, .invalid-feedback, [role=dialog], [role=alert]')] + .filter(vis).map(e => e.innerText.trim()).filter(Boolean).map(t => t.slice(0,400)); + if (alerts.length) out.notices = alerts; + out.next = n; + return out; +} +""" + + +class AgentToolsArm: + name = "agent_tools" + + def __init__(self, options: dict[str, Any]) -> None: + self.name = str(options.get("name", self.name)) + self.model = options.get("model", "claude-fable-5-1") + self.effort = options.get("effort", "medium") + self.max_turns = int(options.get("max_turns", 40)) + self.item_budget_usd = float(options.get("item_budget_usd", 1.00)) + self.headless = bool(options.get("headless", True)) + self.session: OpenEMRSession | None = None + self.rest: RestReader | None = None + self._dialogs: list[str] = [] + + def describe(self) -> dict[str, Any]: + return { + "arm": self.name, + "kind": "agent with structured tools and record read-back", + "model": self.model, + "effort": self.effort, + "tools": [t["name"] for t in TOOLS], + "max_turns_per_item": self.max_turns, + "item_budget_usd": self.item_budget_usd, + "prices_as_of": budget.prices()["as_of"], + "sees": "DOM and accessibility view of every frame; dialog text", + "reads_record": "OpenEMR REST patient search (clerk's API access)", + } + + def setup(self, env: Env, operator: Operator) -> list[dict[str, Any]]: + self.session = OpenEMRSession(env, headless=self.headless) + self.rest = RestReader(env) + t0 = time.time() + self.session.start() + self.session.page.on("dialog", self._on_dialog) + self.session.sign_in() + return [ + { + "phase": "write_procedure_and_tools", + "wall_s": round(time.time() - t0, 2), + "attended_s": 0.0, + "model_calls": 0, + "cost_usd": 0.0, + "detail": {"note": "the shared SOP plus five tool wrappers (this file); engineering time to build the tools is reported in the report's setup table, not here"}, + } + ] + + def _on_dialog(self, dialog: Any) -> None: + self._dialogs.append(f"{dialog.type}: {dialog.message}"[:300]) + try: + dialog.accept() + except Exception: # noqa: BLE001 + pass + + def after_drift(self, event: dict[str, Any], env: Env, operator: Operator) -> list[dict[str, Any]]: + return [] + + def close(self) -> None: + if self.session is not None: + self.session.close() + + # -- tools ------------------------------------------------------------------- + + def _read_screen(self) -> dict[str, Any]: + page = self.session.page + page.wait_for_timeout(300) + n = 1 + frames: list[dict[str, Any]] = [] + for i, frame in enumerate(page.frames): + try: + got = frame.evaluate(READ_JS, n) + except Exception: # noqa: BLE001 - detached frames are skipped + continue + n = got.pop("next") + if got["controls"] or got["buttons"] or got.get("notices"): + url = (frame.url or "").split("?")[0] + frames.append({"frame": url.rsplit("/", 2)[-1] or f"frame{i}", **got}) + out: dict[str, Any] = {"frames": frames} + if self._dialogs: + out["browser_dialogs"] = self._dialogs[:] + self._dialogs.clear() + return out + + def _locate(self, ref: str) -> Any: + for frame in self.session.page.frames: + try: + loc = frame.locator(f'[data-ar="{ref}"]') + if loc.count(): + return loc.first + except Exception: # noqa: BLE001 + continue + raise KeyError(f"no control with ref {ref}; call read_screen again") + + def _settle(self) -> None: + page = self.session.page + page.wait_for_timeout(400) + try: + page.wait_for_load_state("networkidle", timeout=3000) + except Exception: # noqa: BLE001 + pass + + def _tool(self, name: str, args: dict[str, Any]) -> Any: + if name == "read_screen": + return self._read_screen() + if name == "set_field": + loc = self._locate(args["ref"]) + tag = loc.evaluate("e => e.tagName.toLowerCase()") + if tag == "select": + try: + loc.select_option(label=args["value"]) + except Exception: # noqa: BLE001 + loc.select_option(value=args["value"]) + else: + loc.fill(args["value"]) + loc.dispatch_event("change") + return "ok" + if name == "click": + self._locate(args["ref"]).click(timeout=15_000) + self._settle() + return "clicked; call read_screen to see the result" + if name == "click_text": + for frame in self.session.page.frames: + try: + loc = frame.get_by_text(args["text"], exact=True).locator("visible=true") + if loc.count(): + loc.first.click(timeout=15_000) + self._settle() + return "clicked; call read_screen to see the result" + except Exception: # noqa: BLE001 + continue + raise KeyError(f"no visible element with the text {args['text']!r}") + if name == "find_patients": + return self.rest.find(**args) + raise KeyError(name) + + # -- one item ---------------------------------------------------------------- + + def run_item(self, item: WorkItem, env: Env, operator: Operator) -> ItemResult: + assert self.session is not None + prep = self.session.blank_form() + meter = budget.Meter(f"{self.name} {item.item_id}", self.item_budget_usd) + evidence = Path(env.workdir) / self.name / item.item_id + evidence.mkdir(parents=True, exist_ok=True) + t0 = time.time() + claim: dict[str, Any] | None = None + error = "" + turns = 0 + trace: list[dict[str, Any]] = [] + try: + claim, turns = self._loop(item, meter, trace) + except budget.SpendCapReached: + raise + except budget.ItemBudgetExceeded as exc: + error = f"budget: {exc}" + except Exception as exc: # noqa: BLE001 + error = f"{type(exc).__name__}: {exc}"[:500] + finally: + (evidence / "trace.json").write_text(json.dumps(trace, indent=1, default=str)[:400_000]) + wall = time.time() - t0 + if claim is not None and claim.get("result") == "SAVED": + claimed, status = "done", f"RESULT: SAVED: {claim.get('reason', '')}" + elif claim is not None: + claimed, status = "stopped", f"RESULT: NOT SAVED: {claim.get('reason', '')}" + else: + claimed, status = "error", error or "no finish" + if claimed != "done": + operator.ask(OperatorRequest(kind="stop", item_id=item.item_id, reason=status, surface="agent finish")) + return ItemResult( + item_id=item.item_id, + claimed=claimed, + product_status=status, + wall_s=round(wall, 2), + model_calls=meter.calls, + cost_usd=round(meter.usd, 6), + evidence=str(evidence.relative_to(env.workdir)), + detail={**budget.summarize(meter), **prep, "turns": turns, "compute_usd": round(budget.compute_usd(wall), 6)}, + ) + + def _loop(self, item: WorkItem, meter: budget.Meter, trace: list[dict[str, Any]]) -> tuple[dict[str, Any] | None, int]: + client = budget.client(self.model) + messages: list[dict[str, Any]] = [ + {"role": "user", "content": SOP.replace( + "End your final reply with exactly one line, either\nRESULT: SAVED\nor\nRESULT: NOT SAVED: ", + "Finish by calling the finish tool.", + ) + "\n\nThe referral:\n" + item.source_text} + ] + for turn in range(1, self.max_turns + 1): + approx_in = 3000 + 2500 * turn + meter.before_call(self.model, budget.max_call_usd(self.model, approx_in, 4000)) + response = client.messages.create( + model=self.model, + max_tokens=16000, + system=SYSTEM, + tools=TOOLS, + output_config={"effort": self.effort}, + cache_control={"type": "ephemeral"}, + messages=messages, + ) + meter.record(self.model, response.usage) + messages.append({"role": "assistant", "content": response.content}) + uses = [b for b in response.content if b.type == "tool_use"] + trace.append({"turn": turn, "stop": response.stop_reason, "calls": [{"name": b.name, "input": b.input} for b in uses]}) + if response.stop_reason == "refusal": + return {"result": "NOT SAVED", "reason": "the model declined the request"}, turn + if not uses: + text = " ".join(b.text for b in response.content if b.type == "text") + messages.append({"role": "user", "content": "Call the finish tool to report the outcome." + (" " if text else "")}) + continue + results = [] + for b in uses: + if b.name == "finish": + return dict(b.input), turn + try: + out = self._tool(b.name, dict(b.input)) + content = out if isinstance(out, str) else json.dumps(out, default=str) + results.append({"type": "tool_result", "tool_use_id": b.id, "content": content[:20_000]}) + except Exception as exc: # noqa: BLE001 + results.append({"type": "tool_result", "tool_use_id": b.id, "content": f"Error: {exc}"[:400], "is_error": True}) + trace[-1].setdefault("results", []).append(results[-1]["content"][:2000]) + messages.append({"role": "user", "content": results}) + return None, self.max_turns + + +def factory(options: dict[str, Any]) -> AgentToolsArm: + return AgentToolsArm(options) diff --git a/packages/flow/benchmark/lifecycle/arms/budget.py b/packages/flow/benchmark/lifecycle/arms/budget.py new file mode 100644 index 000000000..ecde3f780 --- /dev/null +++ b/packages/flow/benchmark/lifecycle/arms/budget.py @@ -0,0 +1,232 @@ +"""Model prices, usage accounting and the shared spend cap for the comparison arms. + +- Prices come only from ``prices.json`` (dated). Nothing here hard-codes a price. +- Every model call made by any arm is added to one spend ledger + (``benchmark/lifecycle/results/spend.json``) under a file lock, so the cap + holds across arms and across resumed runs. +- Before a call, an arm asks :func:`reserve` for the most that call could cost. + If that could pass the cap, :class:`SpendCapReached` is raised and nothing is + sent. The cap is $30 unless ``LIFECYCLE_SPEND_CAP_USD`` raises it. +- The API key is read from the environment (``ANTHROPIC_API_KEY``) or, when + ``LIFECYCLE_KEYCHAIN_SERVICE`` names one, from the macOS keychain. It is held + in this process only and never printed, logged or written. +""" + +from __future__ import annotations + +import fcntl +import json +import os +import subprocess +import time +from contextlib import contextmanager +from dataclasses import asdict, dataclass, field +from pathlib import Path +from typing import Any, Iterator, Optional + +HERE = Path(__file__).resolve().parent +PRICES_PATH = HERE / "prices.json" +RESULTS = HERE.parent / "results" +SPEND_PATH = Path(os.environ.get("LIFECYCLE_SPEND_LEDGER", str(RESULTS / "spend.json"))) +DEFAULT_CAP_USD = 30.0 + + +class SpendCapReached(BaseException): + """The next model call could take total spend past the cap. + + A BaseException on purpose: no `except Exception` in an arm or a harness + can swallow it, so reaching the cap always stops the run (resume later). + """ + + +class ItemBudgetExceeded(RuntimeError): + """One item (or one phase) used up its own fixed budget.""" + + +def prices() -> dict[str, Any]: + return json.loads(PRICES_PATH.read_text()) + + +def price(model: str) -> dict[str, float]: + table = prices()["models"] + if model not in table: + raise KeyError(f"{model} is not in {PRICES_PATH.name} (as of {prices()['as_of']})") + return table[model] + + +def cap_usd() -> float: + return float(os.environ.get("LIFECYCLE_SPEND_CAP_USD", DEFAULT_CAP_USD)) + + +@dataclass +class Usage: + """Token usage of one model on one item or phase, priced from the table.""" + + model: str + input: int = 0 + output: int = 0 + cache_read: int = 0 + cache_write: int = 0 + calls: int = 0 + + @property + def usd(self) -> float: + p = price(self.model) + return ( + self.input * p["input"] + + self.output * p["output"] + + self.cache_read * p["cache_read"] + + self.cache_write * p["cache_write"] + ) / 1_000_000 + + def add_response(self, usage: Any) -> float: + """Add one API response's ``usage``; returns that call's cost.""" + + def n(name: str) -> int: + return int(getattr(usage, name, 0) or 0) + + one = Usage( + self.model, + input=n("input_tokens"), + output=n("output_tokens"), + cache_read=n("cache_read_input_tokens"), + cache_write=n("cache_creation_input_tokens"), + calls=1, + ) + self.input += one.input + self.output += one.output + self.cache_read += one.cache_read + self.cache_write += one.cache_write + self.calls += 1 + return one.usd + + def to_dict(self) -> dict[str, Any]: + return {**asdict(self), "usd": round(self.usd, 6)} + + +@dataclass +class Meter: + """Usage for one item or phase, with that unit's own fixed budget.""" + + label: str + budget_usd: float + usage: dict[str, Usage] = field(default_factory=dict) + + def of(self, model: str) -> Usage: + return self.usage.setdefault(model, Usage(model)) + + @property + def usd(self) -> float: + return sum(u.usd for u in self.usage.values()) + + @property + def calls(self) -> int: + return sum(u.calls for u in self.usage.values()) + + def before_call(self, model: str, max_call_usd: float) -> None: + """Refuse a call that could pass this unit's budget or the global cap.""" + + if self.usd + max_call_usd > self.budget_usd: + raise ItemBudgetExceeded( + f"{self.label}: ${self.usd:.3f} used of ${self.budget_usd:.2f}; " + f"the next call could cost ${max_call_usd:.3f}" + ) + check_cap(max_call_usd) + + def record(self, model: str, response_usage: Any) -> float: + usd = self.of(model).add_response(response_usage) + add_spend(self.label, model, usd) + return usd + + def to_list(self) -> list[dict[str, Any]]: + return [u.to_dict() for u in self.usage.values()] + + +def max_call_usd(model: str, input_tokens: int, max_tokens: int) -> float: + """An upper bound for one call: every input token uncached, full output.""" + + p = price(model) + return (input_tokens * p["cache_write"] + max_tokens * p["output"]) / 1_000_000 + + +@contextmanager +def _locked() -> Iterator[dict[str, Any]]: + SPEND_PATH.parent.mkdir(parents=True, exist_ok=True) + lock = SPEND_PATH.with_suffix(".lock") + with open(lock, "w") as fh: + fcntl.flock(fh, fcntl.LOCK_EX) + data = ( + json.loads(SPEND_PATH.read_text()) + if SPEND_PATH.is_file() + else {"total_usd": 0.0, "entries": []} + ) + yield data + fcntl.flock(fh, fcntl.LOCK_UN) + + +def spent() -> float: + if not SPEND_PATH.is_file(): + return 0.0 + return float(json.loads(SPEND_PATH.read_text()).get("total_usd", 0.0)) + + +def check_cap(next_call_usd: float) -> None: + total, cap = spent(), cap_usd() + if total + next_call_usd > cap: + raise SpendCapReached( + f"${total:.2f} spent; a call of up to ${next_call_usd:.2f} could pass the " + f"${cap:.2f} cap (raise LIFECYCLE_SPEND_CAP_USD only with the coordinator's go)" + ) + + +def add_spend(label: str, model: str, usd: float) -> float: + with _locked() as data: + data["entries"].append( + {"label": label, "model": model, "usd": round(usd, 6), "at": round(time.time(), 3)} + ) + data["total_usd"] = round(sum(e["usd"] for e in data["entries"]), 6) + data["cap_usd"] = cap_usd() + data["prices_as_of"] = prices()["as_of"] + SPEND_PATH.write_text(json.dumps(data, indent=1) + "\n") + return data["total_usd"] + + +def ensure_key() -> None: + """Make the key available to the SDK without ever printing it.""" + + if os.environ.get("ANTHROPIC_API_KEY"): + return + service = os.environ.get("LIFECYCLE_KEYCHAIN_SERVICE") + if service: + out = subprocess.run( + ["security", "find-generic-password", "-s", service, "-w"], + capture_output=True, + text=True, + ) + if out.returncode == 0 and out.stdout.strip(): + os.environ["ANTHROPIC_API_KEY"] = out.stdout.strip() + return + raise SystemExit(f"keychain service {service!r} has no readable password") + raise SystemExit( + "no Anthropic key: set ANTHROPIC_API_KEY, or LIFECYCLE_KEYCHAIN_SERVICE to a " + "keychain item that holds it" + ) + + +def client(model: str = "") -> Any: + ensure_key() + import anthropic + + return anthropic.Anthropic(max_retries=4) + + +def compute_usd(wall_s: float) -> float: + """Nominal local compute for ``wall_s`` seconds of one worker.""" + + return prices()["compute"]["host_usd_per_hour"] * wall_s / 3600.0 + + +def summarize(meter: Optional[Meter]) -> dict[str, Any]: + if meter is None: + return {"model_calls": 0, "cost_usd": 0.0, "usage": []} + return {"model_calls": meter.calls, "cost_usd": round(meter.usd, 6), "usage": meter.to_list()} diff --git a/packages/flow/benchmark/lifecycle/arms/compare.py b/packages/flow/benchmark/lifecycle/arms/compare.py new file mode 100644 index 000000000..962922a9b --- /dev/null +++ b/packages/flow/benchmark/lifecycle/arms/compare.py @@ -0,0 +1,393 @@ +"""One command for the whole economic comparison: every arm, one seeded stream, resumable. + + cd packages/flow/benchmark + python -m lifecycle.arms.compare --out lifecycle/results/economics- [--seed 7 --items 40 --days 10] + +- Every arm runs the same seeded worklist and event ledger through the soak + harness's ``run_arm`` (same reset, seeded charts, drift days, injected + faults, oracle, scripted operator and cost model). +- Each arm writes its own ``//ledger.jsonl``. Rerunning the same + command skips arms whose ledger ends with ``arm_end``. An arm that stopped + part-way (the spend cap, a crash, Ctrl-C) is moved to + ``.partial-`` and run again from its start, because the app is reset + per arm; the partial ledger is kept and its model spend stays counted. +- Model spend is capped across all arms at $30 (``LIFECYCLE_SPEND_CAP_USD`` + raises it only with the coordinator's go). The cap is checked before every + model call; reaching it stops the run with a message to resume. +- The API key comes from ``ANTHROPIC_API_KEY`` or the keychain item named in + ``LIFECYCLE_KEYCHAIN_SERVICE``; it is never printed. +- At the end, ``economics.json`` and ``REPORT.md`` are written from the + ledgers alone (``python -m lifecycle.arms.economics `` recomputes them). +""" + +from __future__ import annotations + +import argparse +import json +import shutil +import sys +import time +from pathlib import Path +from typing import Any, Optional + +from . import budget, economics, report + +DEFAULT_ARMS = [ + # (name in the ledger, module:factory, options) + # The soak's OpenAdapt arm: the certified referral program under the production + # configuration. Its API tier is refused on this form (results/api_tier/). + ("openadapt", "lifecycle.openadapt_arm:make", {}), + ( + "cua_alone", + "lifecycle.arms.cua:factory", + {"model": "claude-opus-5-5", "item_budget_usd": 1.00}, + ), + ( + "agent_tools", + "lifecycle.arms.agent_tools:factory", + {"model": "claude-opus-5-5", "item_budget_usd": 0.60}, + ), + ( + "agent_script", + "lifecycle.arms.agent_script:factory", + {"model": "claude-opus-5-5"}, + ), + ("human_script", "lifecycle.arms.human_script:factory", {}), +] + + +MIN_FREE_GIB = float(__import__("os").environ.get("LIFECYCLE_MIN_FREE_GIB", "10")) + + +class DiskLow(BaseException): + """Free disk fell under the shared machine's floor; stop between items.""" + + +def free_gib() -> float: + return shutil.disk_usage("/System/Volumes/Data").free / 2**30 + + +ITEM_TIMEOUT_S = float(__import__("os").environ.get("LIFECYCLE_ITEM_TIMEOUT_S", "900")) +#: After the soft timeout, how long to wait for the arm to return before the +#: whole process exits (so a hang can never hold the shared fixture for hours). +HARD_GRACE_S = 180.0 + + +def _playwright_children() -> list[int]: + import os + import subprocess + + out = subprocess.run( + ["pgrep", "-P", str(os.getpid())], capture_output=True, text=True + ).stdout.split() + pids = [] + for pid in out: + cmd = subprocess.run( + ["ps", "-o", "command=", "-p", pid], capture_output=True, text=True + ).stdout + if "playwright" in cmd: + pids.append(int(pid)) + return pids + + +class ItemWatchdog: + """Bounds one item. + + At ``timeout_s`` it writes every thread's Python stack to ``stack_path`` + (to find what hung) and kills this process's Playwright drivers, so a + blocked browser call raises and the runner records the item as an error. + If the arm still has not returned ``HARD_GRACE_S`` later (a hang outside + the browser), it exits the process with code 5; the comparison resumes + from the arm's start and keeps the partial ledger. + """ + + def __init__(self, timeout_s: float, stack_path: Path) -> None: + import threading + + self.fired = False + self.stack_path = stack_path + self._soft = threading.Timer(timeout_s, self._fire) + self._hard = threading.Timer(timeout_s + HARD_GRACE_S, self._exit) + for t in (self._soft, self._hard): + t.daemon = True + t.start() + + def _dump(self, why: str) -> None: + import faulthandler + + with open(self.stack_path, "a") as fh: + fh.write(f"--- {why} at {time.strftime('%Y-%m-%dT%H:%M:%S')}\n") + fh.flush() + faulthandler.dump_traceback(file=fh, all_threads=True) + + def _fire(self) -> None: + import os + import signal + + self.fired = True + self._dump("item timeout") + for pid in _playwright_children(): + try: + os.kill(pid, signal.SIGKILL) + except OSError: + pass + + def _exit(self) -> None: + import os + + self._dump("hard timeout; exiting") + os._exit(5) + + def cancel(self) -> None: + self._soft.cancel() + self._hard.cancel() + + +class Guarded: + """Any arm, checked for free disk before every item and bounded in time. + + The check runs between items, never inside the harness's app reset, so a + stop leaves the fixture whole. DiskLow is a BaseException so the runner's + ``except Exception`` cannot turn it into an item result. + """ + + def __init__(self, options: dict[str, Any]) -> None: + from ..arm import load_arm + + inner_opts = dict(options) + spec = inner_opts.pop("_inner") + self.inner = load_arm(spec, inner_opts) + self.name = self.inner.name + + def describe(self) -> dict[str, Any]: + return self.inner.describe() + + def setup(self, env: Any, operator: Any) -> list[dict[str, Any]]: + return self.inner.setup(env, operator) + + def run_item(self, item: Any, env: Any, operator: Any) -> Any: + free = free_gib() + if free < MIN_FREE_GIB: + raise DiskLow( + f"{free:.1f} GiB free, under {MIN_FREE_GIB:.0f} GiB; stopped before {item.item_id}" + ) + dog = ItemWatchdog(ITEM_TIMEOUT_S, Path(env.workdir) / "stacks.txt") + try: + result = self.inner.run_item(item, env, operator) + except Exception: + if dog.fired: + self._fresh_browser() + raise + finally: + dog.cancel() + if dog.fired: + self._fresh_browser() + result.claimed = "error" + result.product_status = ( + f"item timed out after {ITEM_TIMEOUT_S:.0f} s; browser driver stopped. " + + str(result.product_status) + )[:500] + return result + + def _fresh_browser(self) -> None: + """The browser driver was killed: the arm starts a new browser on its next item.""" + + session = getattr(self.inner, "session", None) + if session is not None and hasattr(session, "close"): + session.close() + + def after_drift( + self, event: dict[str, Any], env: Any, operator: Any + ) -> list[dict[str, Any]]: + return self.inner.after_drift(event, env, operator) + + def close(self) -> None: + self.inner.close() + + +def guarded(options: dict[str, Any]) -> Guarded: + return Guarded(options) + + +def _complete(ledger: Path) -> bool: + if not ledger.is_file(): + return False + lines = [ln for ln in ledger.read_text().splitlines() if ln.strip()] + return ( + bool(lines) + and json.loads(lines[-1]).get("type") in ("arm_end", "run_end") + and any(json.loads(ln).get("type") == "arm_end" for ln in lines[-3:]) + ) + + +def _move_partial(d: Path) -> Optional[Path]: + if not d.exists(): + return None + n = 1 + while (d.parent / f"{d.name}.partial-{n}").exists(): + n += 1 + dest = d.parent / f"{d.name}.partial-{n}" + d.rename(dest) + return dest + + +def main(argv: Optional[list[str]] = None) -> int: + p = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + p.add_argument("--out", type=Path, required=True) + p.add_argument("--seed", type=int, default=7) + p.add_argument("--items", type=int, default=40) + p.add_argument("--days", type=int, default=10) + p.add_argument("--existing", type=int, default=12) + p.add_argument("--only", action="append", help="run only these arms (by name)") + p.add_argument( + "--arms-json", type=Path, help="a JSON list of [name, module:factory, options]" + ) + p.add_argument("--app", default="openemr") + p.add_argument("--report-only", action="store_true") + p.add_argument( + "--drift", + action="append", + default=None, + metavar="KIND@SEQ", + help="replace the drift schedule, as lifecycle.run --drift (e.g. relabel_save@26)", + ) + args = p.parse_args(argv) + + from ..cost_model import V1 + from ..run import PROTOCOL_VERSION, Ledger, engine_commit, run_arm + from ..workload import WORKLOAD_VERSION, generate + + out: Path = args.out + out.mkdir(parents=True, exist_ok=True) + arms = json.loads(args.arms_json.read_text()) if args.arms_json else DEFAULT_ARMS + if args.only: + arms = [a for a in arms if a[0] in args.only] + + plan = { + "seed": args.seed, + "items": args.items, + "days": args.days, + "existing": args.existing, + "app": args.app, + "drift": args.drift, + "arms": arms, + } + plan_path = out / "plan.json" + if plan_path.is_file(): + old = json.loads(plan_path.read_text()) + same = { + k: old.get(k) for k in ("seed", "items", "days", "existing", "app", "drift") + } + if same != {k: plan[k] for k in same}: + raise SystemExit( + f"{out} was started with a different stream {same}; use a new --out" + ) + else: + plan_path.write_text(json.dumps(plan, indent=1) + "\n") + + if not args.report_only: + workload = generate(args.seed, args.items, args.days, args.existing) + if args.drift is not None: + by_seq = {i.seq: i for i in workload.items} + workload.drift = [ + {"day": by_seq[int(seq)].day, "kind": kind, "first_seq": int(seq)} + for kind, _, seq in (spec.partition("@") for spec in args.drift) + ] + wl = out / "workload.jsonl" + text = "".join( + json.dumps(r, sort_keys=True) + "\n" for r in workload.ledger_rows() + ) + if wl.is_file() and wl.read_text() != text: + raise SystemExit( + "the seeded workload differs from the one this run started with" + ) + wl.write_text(text) + from ..apps import make_app + + app = None + for name, spec, options in arms: + d = out / name + if _complete(d / "ledger.jsonl"): + print(f"[{name}] already complete; skipping") + continue + moved = _move_partial(d) + if moved: + print( + f"[{name}] earlier attempt stopped part-way; kept as {moved.name}" + ) + d.mkdir(parents=True) + if app is None: + app = make_app(args.app) + ledger = Ledger(d / "ledger.jsonl") + ledger.write( + { + "type": "run_start", + "protocol": PROTOCOL_VERSION, + "workload": WORKLOAD_VERSION, + "seed": args.seed, + "items": args.items, + "days": args.days, + "cost_model": V1.to_dict(), + "engine_commit": engine_commit(), + "prices_as_of": budget.prices()["as_of"], + "spend_before_usd": budget.spent(), + "spend_cap_usd": budget.cap_usd(), + "app": app.describe(), + } + ) + t0 = time.time() + try: + if free_gib() < MIN_FREE_GIB: + raise DiskLow(f"{free_gib():.1f} GiB free before {name}") + run_arm( + arm_spec="lifecycle.arms.compare:guarded", + arm_options={**options, "_inner": spec}, + app=app, + workload=workload, + ledger=ledger, + out=d, + cost_name=V1.version, + stop_after=options.get("stop_after"), + ) + ledger.write( + { + "type": "run_end", + "wall_s": round(time.time() - t0, 1), + "spend_after_usd": budget.spent(), + } + ) + except budget.SpendCapReached as exc: + ledger.write( + {"type": "stopped", "why": "spend_cap", "detail": str(exc)} + ) + ledger.close() + print( + f"[{name}] stopped: {exc}\nResume with the same command once the cap is raised." + ) + return 3 + except BaseException as exc: + # Matched by name: the guarded arm is loaded through the + # module path, so its DiskLow is not this __main__'s class. + if type(exc).__name__ != "DiskLow": + raise + ledger.write({"type": "stopped", "why": "disk_low", "detail": str(exc)}) + ledger.close() + print( + f"[{name}] stopped: {exc}\nResume with the same command when there is room." + ) + return 4 + finally: + ledger.close() + + rows = economics.read_rows(sorted(out.glob("*/ledger.jsonl"))) + result = economics.compute(rows) + (out / "economics.json").write_text( + json.dumps(result, indent=1, default=str) + "\n" + ) + (out / "REPORT.md").write_text(report.render(result, out, plan)) + print(economics.render(result)) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/packages/flow/benchmark/lifecycle/arms/cua.py b/packages/flow/benchmark/lifecycle/arms/cua.py new file mode 100644 index 000000000..e84971fe9 --- /dev/null +++ b/packages/flow/benchmark/lifecycle/arms/cua.py @@ -0,0 +1,238 @@ +"""Arm ``cua_alone``: a screenshot-driven computer-use agent, alone. + +The current frontier model drives the browser through the computer use toolset +(``computer_toolset_20260801``). It sees screenshots only: no DOM, no API, no +record read-back. Nothing records or checks it. Each item has a fixed budget +(model calls and dollars); an item that runs out is a stop the operator sees. + +Actions go to the page through the scoreboard's :class:`PlainComputer`, the +same translator the earlier scoreboard agent arm used. +""" + +from __future__ import annotations + +import base64 +import json +import time +from pathlib import Path +from typing import Any + +from ..arm import Env, ItemResult, Operator, OperatorRequest, WorkItem +from . import budget +from .openemr_ui import SOP, OpenEMRSession, claimed_from + +DISABLED_MEMBERS = ( + "zoom", + "triple_click", + "middle_click", + "left_mouse_down", + "left_mouse_up", + "hold_key", +) + +SYSTEM = """\ +You operate a web browser that shows one page of a clinic's electronic medical \ +record (OpenEMR). There is no address bar and no other application. The screen \ +is 1280x800 pixels. Work only on the page you are shown. Do not navigate away, \ +open menus you do not need, or change any field the task does not mention. + +Click a field before you type into it. To scroll, scroll at the point you last \ +clicked. When the task is done, stop and reply; do not take further actions.""" + + +class CuaArm: + name = "cua_alone" + + def __init__(self, options: dict[str, Any]) -> None: + self.name = str(options.get("name", self.name)) + self.model = options.get("model", "claude-fable-5-1") + self.effort = options.get("effort", "medium") + self.max_turns = int(options.get("max_turns", 40)) + self.item_budget_usd = float(options.get("item_budget_usd", 1.50)) + self.headless = bool(options.get("headless", True)) + self.session: OpenEMRSession | None = None + + def describe(self) -> dict[str, Any]: + return { + "arm": self.name, + "kind": "computer-use agent alone (screenshots only)", + "model": self.model, + "effort": self.effort, + "tool": "computer_toolset_20260801", + "disabled_members": list(DISABLED_MEMBERS), + "max_turns_per_item": self.max_turns, + "item_budget_usd": self.item_budget_usd, + "prices_as_of": budget.prices()["as_of"], + "sees": "screenshots of the OpenEMR page", + "reads_record": False, + } + + def setup(self, env: Env, operator: Operator) -> list[dict[str, Any]]: + # Writing the procedure (the SOP above) is the only setup; it is shared + # by every agent arm and charged the same to each. + self.session = OpenEMRSession(env, headless=self.headless) + t0 = time.time() + self.session.start() + self.session.sign_in() + return [ + { + "phase": "write_procedure", + "wall_s": round(time.time() - t0, 2), + "attended_s": 0.0, + "model_calls": 0, + "cost_usd": 0.0, + "detail": {"note": "the shared SOP; its writing time is charged by the cost model's procedure rule if any"}, + } + ] + + def after_drift(self, event: dict[str, Any], env: Env, operator: Operator) -> list[dict[str, Any]]: + return [] + + def close(self) -> None: + if self.session is not None: + self.session.close() + + def run_item(self, item: WorkItem, env: Env, operator: Operator) -> ItemResult: + try: + from benchmark.scoreboard.agent import PlainComputer + except ImportError: # run from packages/flow/benchmark + from scoreboard.agent import PlainComputer + + assert self.session is not None + prep = self.session.blank_form() + meter = budget.Meter(f"{self.name} {item.item_id}", self.item_budget_usd) + evidence = Path(env.workdir) / self.name / item.item_id + evidence.mkdir(parents=True, exist_ok=True) + computer = PlainComputer(self.session.page) + t0 = time.time() + final_text, error = "", "" + turns = 0 + try: + final_text, turns = self._loop(item, computer, meter, evidence) + except (budget.ItemBudgetExceeded, budget.SpendCapReached) as exc: + if isinstance(exc, budget.SpendCapReached): + raise + error = f"budget: {exc}" + except Exception as exc: # noqa: BLE001 - an agent crash is a stop + error = f"{type(exc).__name__}: {exc}"[:500] + wall = time.time() - t0 + claimed, status = claimed_from(final_text) if not error else ("error", error) + if claimed != "done": + operator.ask( + OperatorRequest( + kind="stop", + item_id=item.item_id, + reason=status, + surface="agent final reply", + ) + ) + (evidence / "reply.json").write_text( + json.dumps({"final_text": final_text, "error": error, "turns": turns}, indent=1) + ) + return ItemResult( + item_id=item.item_id, + claimed=claimed if claimed != "error" else "error", + product_status=status, + wall_s=round(wall, 2), + model_calls=meter.calls, + cost_usd=round(meter.usd, 6), + evidence=str(evidence.relative_to(env.workdir)), + detail={ + **budget.summarize(meter), + **prep, + "turns": turns, + "compute_usd": round(budget.compute_usd(wall), 6), + }, + ) + + def _loop(self, item: WorkItem, computer: Any, meter: budget.Meter, evidence: Path) -> tuple[str, int]: + client = budget.client(self.model) + tools = [ + { + "type": "computer_toolset_20260801", + "configs": {m: {"enabled": False} for m in DISABLED_MEMBERS}, + } + ] + first = computer.execute({"action": "screenshot"}) + instruction = SOP + "\n\nThe referral:\n" + item.source_text + messages: list[dict[str, Any]] = [ + { + "role": "user", + "content": [ + {"type": "text", "text": instruction}, + _image(first.png), + ], + } + ] + ledger: list[dict[str, Any]] = [] + for turn in range(1, self.max_turns + 1): + # An upper bound for this call: history so far, uncached, plus 16k out. + approx_in = 4000 + 1600 * turn + meter.before_call(self.model, budget.max_call_usd(self.model, approx_in, 4000)) + t0 = time.time() + response = client.messages.create( + model=self.model, + max_tokens=16000, + system=SYSTEM, + tools=tools, + output_config={"effort": self.effort}, + cache_control={"type": "ephemeral"}, + messages=messages, + ) + usd = meter.record(self.model, response.usage) + calls = [b for b in response.content if b.type == "tool_use"] + ledger.append( + { + "turn": turn, + "s": round(time.time() - t0, 2), + "usd": round(usd, 6), + "stop_reason": response.stop_reason, + "actions": [{"name": b.name, "input": b.input} for b in calls], + } + ) + (evidence / "calls.json").write_text(json.dumps(ledger, indent=1)) + messages.append({"role": "assistant", "content": response.content}) + if response.stop_reason == "refusal": + return "RESULT: NOT SAVED: the model declined the request", turn + if not calls: + return " ".join(b.text for b in response.content if b.type == "text"), turn + results: list[dict[str, Any]] = [] + for block in calls: + native = {"action": block.name, **dict(block.input)} + if block.name == "key" and block.input.get("repeat"): + n = max(1, min(int(block.input["repeat"]), 100)) + native = {"action": "key", "text": " ".join([str(block.input.get("text", ""))] * n)} + result: dict[str, Any] = { + "type": "tool_result", + "tool_use_id": block.id, + "toolset_name": "computer", + } + try: + obs = computer.execute(native) + except Exception as exc: # noqa: BLE001 - tell the model, let it choose + result["content"] = f"Error: {exc}"[:300] + result["is_error"] = True + results.append(result) + continue + if block.name == "screenshot" or block is calls[-1]: + content: list[dict[str, Any]] = [] + if obs.notes: + content.append({"type": "text", "text": "; ".join(obs.notes)}) + content.append(_image(obs.png)) + result["content"] = content + else: + result["content"] = [{"type": "text", "text": "OK"}] + results.append(result) + messages.append({"role": "user", "content": results}) + return f"RESULT: NOT SAVED: no finish within {self.max_turns} model calls", self.max_turns + + +def _image(png: bytes) -> dict[str, Any]: + return { + "type": "image", + "source": {"type": "base64", "media_type": "image/png", "data": base64.b64encode(png).decode()}, + } + + +def factory(options: dict[str, Any]) -> CuaArm: + return CuaArm(options) diff --git a/packages/flow/benchmark/lifecycle/arms/economics.py b/packages/flow/benchmark/lifecycle/arms/economics.py new file mode 100644 index 000000000..19ead8cbe --- /dev/null +++ b/packages/flow/benchmark/lifecycle/arms/economics.py @@ -0,0 +1,318 @@ +"""Total lifecycle cost per 1,000 verified items, per arm, from the results ledgers alone. + +Every number comes from ledger rows the harness wrote (items, operator +requests, setup and maintenance phases, drift events) priced with the soak's +versioned operator cost model and this directory's dated ``prices.json``. +Nothing is re-measured here, so ``python -m lifecycle.arms.economics `` +recomputes the report from committed data. + +Definitions: + +- *Verified item*: the oracle's final read says the item is right (the + intended chart exactly once, or no new chart when none is wanted), whoever + did it. Items a person did by hand count, and so does their time: this is + the cost of getting the work done right, not of automating it. +- *Automated*: verified and not done by hand. +- *Setup*: one-time, per workflow. People's minutes (attended phase time plus + the cost model's fixed charges for teach, compile review and admission) and + model dollars. +- *Per item*: model dollars, nominal compute, and operator minutes (approvals, + stops, questions, reconciliations, items done by hand). +- *Maintenance*: after a UI change. Model dollars spent repairing or + re-authoring, and people's minutes reviewing the repair. +- *Silent-wrong cleanup*: each silent wrong is charged, as a lower bound, + the people time to find and fix it later (one reconcile plus one manual + entry, plus the interrupt charge each). Its clinical or business harm is + not priced; the silent-wrong rate is reported beside every total. +- Projection to 1,000 verified items: setup once, per-item costs scaled by + 1,000 / verified, maintenance per UI change times the assumed changes per + 1,000 items (``prices.json`` drift rate). +""" + +from __future__ import annotations + +import json +import statistics +import sys +from collections import Counter +from pathlib import Path +from typing import Any, Iterable, Optional + +from . import budget + +try: + from ..cost_model import MODELS + from ..metrics import wilson +except ImportError: # pragma: no cover - the harness is not checked in yet + MODELS = {} + + def wilson(k: int, n: int, z: float = 1.96) -> Optional[tuple[float, float]]: # type: ignore[misc] + import math + + if n == 0: + return None + p = k / n + d = 1 + z * z / n + c = p + z * z / (2 * n) + r = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) + return (round((c - r) / d, 4), round((c + r) / d, 4)) + + +FIXED_PHASES = {"teach": "teach_prep_s", "compile": "compile_review_s", "admit": "admit_s"} + + +def read_rows(paths: Iterable[Path]) -> list[dict[str, Any]]: + rows: list[dict[str, Any]] = [] + for p in paths: + for line in p.read_text().splitlines(): + if line.strip(): + row = json.loads(line) + row["_ledger"] = str(p) + rows.append(row) + return rows + + +def planned_items(rows: list[dict[str, Any]], arm: str) -> Optional[int]: + """How many items the arm's run was started for. + + From the ``run_start`` row of the arm's own ledger, lowered by a + ``stop_after`` option in the comparison's ``plan.json``. None when the + ledger does not say. + """ + + start = next((r for r in rows if r.get("type") == "arm_start" and r.get("arm") == arm), None) + if start is None or "_ledger" not in start: + return None + ledger = start["_ledger"] + planned = next((r.get("items") for r in rows if r.get("type") == "run_start" and r.get("_ledger") == ledger), None) + if planned is None: + return None + plan = Path(ledger).parent.parent / "plan.json" + if plan.is_file(): + for entry in json.loads(plan.read_text()).get("arms", []): + if entry[0] == arm and isinstance(entry[2], dict) and entry[2].get("stop_after"): + planned = min(int(planned), int(entry[2]["stop_after"])) + return int(planned) + + +def _cost_model(rows: list[dict[str, Any]]) -> dict[str, Any]: + for r in rows: + if r.get("type") == "run_start" and r.get("cost_model"): + return r["cost_model"] + if MODELS: + return next(iter(MODELS.values())).to_dict() + raise ValueError("no cost model in the ledger and none importable") + + +def _phase_human_s(cm: dict[str, Any], phase: dict[str, Any]) -> float: + fixed = cm.get(FIXED_PHASES.get(phase.get("phase", ""), ""), 0.0) + return float(phase.get("attended_s", 0.0) or 0.0) + float(fixed or 0.0) + + +def arm_economics(rows: list[dict[str, Any]], arm: str, cm: dict[str, Any], labor: dict[str, Any], + drift_per_1000: float) -> dict[str, Any]: + mine = [r for r in rows if r.get("arm") == arm] + items = [r for r in mine if r["type"] == "item_result"] + ops = [r for r in mine if r["type"] == "operator"] + setup = [r for r in mine if r["type"] == "phase" and r.get("stage") == "setup"] + maint = [r for r in mine if r["type"] == "phase" and r.get("stage") == "maintenance"] + drifts = [r for r in mine if r["type"] == "drift_applied"] + describe = next((r.get("describe") for r in mine if r["type"] == "arm_start"), {}) or {} + + n = len(items) + planned = planned_items(rows, arm) + short = planned is not None and n < planned + stopped = next((r for r in reversed(mine) if r.get("type") == "stopped"), None) + if short: + short_why = (f"{stopped.get('why')}: {stopped.get('detail', '')}" if stopped + else "the ledger ends before the planned items (the run exited or was killed)") + else: + short_why = None + by_hand = {r["item_id"] for r in ops if r.get("action") == "do_by_hand" and r.get("item_id")} + verified = [r for r in items if r.get("final_oracle") == "correct"] + automated = [r for r in verified if r["item_id"] not in by_hand] + outcomes = Counter(r["outcome"] for r in items) + wanted = [r for r in items if r.get("chart_wanted")] + clean = [r for r in wanted if r.get("exception") in ("clean", "missing_optional", "malformed_dob", + "malformed_case", "ambiguous_new_person")] + + setup_human_s = sum(_phase_human_s(cm, p) for p in setup) + setup_model_usd = sum(float(p.get("cost_usd", 0.0) or 0.0) for p in setup) + item_model_usd = sum(float(r.get("cost_usd", 0.0) or 0.0) for r in items) + compute_usd = sum(float((r.get("detail") or {}).get("compute_usd") or budget.compute_usd(r.get("wall_s", 0.0))) + for r in items) + item_op_s = sum(float(r["charged_s"]) for r in ops if r["kind"] != "repair_review") + review_s = sum(float(r["charged_s"]) for r in ops if r["kind"] == "repair_review") + maint_human_s = review_s + sum(_phase_human_s(cm, p) for p in maint) + maint_model_usd = sum(float(p.get("cost_usd", 0.0) or 0.0) for p in maint) + model_calls = sum(int(r.get("model_calls", 0) or 0) for r in items) + + op_rate = labor["operator_usd_per_hour"] / 3600.0 + sw = outcomes.get("silent_wrong", 0) + cleanup_s = sw * (float(cm.get("reconcile_s", 0.0)) + float(cm.get("manual_entry_s", 0.0)) + + 2 * float(cm.get("interrupt_s", 0.0))) + v = len(verified) + per_v = (1000.0 / v) if v else float("nan") + items_per_1000v = n * per_v + d = len(drifts) + maint_per_event_usd = (maint_model_usd + maint_human_s * op_rate) / d if d else 0.0 + maint_per_event_min = maint_human_s / 60.0 / d if d else 0.0 + proj_events = drift_per_1000 * items_per_1000v / 1000.0 if v else float("nan") + + proj = { + "setup_usd": setup_model_usd + setup_human_s * op_rate, + "setup_model_usd": setup_model_usd, + "setup_human_min": setup_human_s / 60.0, + "item_model_usd": item_model_usd * per_v, + "compute_usd": compute_usd * per_v, + "operator_min": item_op_s / 60.0 * per_v, + "operator_usd": item_op_s * op_rate * per_v, + "maintenance_events": proj_events, + "maintenance_usd": maint_per_event_usd * proj_events, + "maintenance_min": maint_per_event_min * proj_events, + "silent_wrong": sw * per_v, + "cleanup_min": cleanup_s / 60.0 * per_v, + "cleanup_usd": cleanup_s * op_rate * per_v, + } + proj["total_usd"] = (proj["setup_usd"] + proj["item_model_usd"] + proj["compute_usd"] + + proj["operator_usd"] + proj["maintenance_usd"] + proj["cleanup_usd"]) + proj["total_human_min"] = (proj["setup_human_min"] + proj["operator_min"] + proj["maintenance_min"] + + proj["cleanup_min"]) + walls = sorted(float(r["wall_s"]) for r in automated) + return { + "arm": arm, + "describe": describe, + "items": n, + "planned_items": planned, + "short_run": short, + "short_why": short_why, + "verified": v, + "automated": len(automated), + "by_hand": len(by_hand), + "outcomes": dict(outcomes), + "silent_wrong": outcomes.get("silent_wrong", 0), + "silent_wrong_rate": round(outcomes.get("silent_wrong", 0) / n, 4) if n else None, + "silent_wrong_ci95": wilson(outcomes.get("silent_wrong", 0), n), + "false_stop": outcomes.get("false_stop", 0), + "false_stop_rate": round(outcomes.get("false_stop", 0) / len(clean), 4) if clean else None, + "false_stop_denominator": len(clean), + "false_stop_ci95": wilson(outcomes.get("false_stop", 0), len(clean)), + "drift_events": d, + "measured": { + "setup_human_min": round(setup_human_s / 60.0, 2), + "setup_model_usd": round(setup_model_usd, 4), + "item_model_usd": round(item_model_usd, 4), + "model_calls": model_calls, + "compute_usd": round(compute_usd, 4), + "operator_min": round(item_op_s / 60.0, 2), + "maintenance_human_min": round(maint_human_s / 60.0, 2), + "maintenance_model_usd": round(maint_model_usd, 4), + "maintenance_per_event_usd": round(maint_per_event_usd, 4), + "maintenance_per_event_min": round(maint_per_event_min, 2), + "median_wall_s_automated": round(statistics.median(walls), 2) if walls else None, + "p90_wall_s_automated": round(walls[int(0.9 * (len(walls) - 1))], 2) if walls else None, + "per_item_model_usd": round(item_model_usd / n, 4) if n else None, + }, + "per_1000_verified": {k: (round(x, 2) if isinstance(x, float) else x) for k, x in proj.items()}, + } + + +AXES = [ + # (key path, label, lower is better) + (("per_1000_verified", "total_usd"), "Total $ per 1,000 verified", True), + (("per_1000_verified", "setup_usd"), "Setup $ (people + model)", True), + (("measured", "per_item_model_usd"), "Model $ per item", True), + (("per_1000_verified", "operator_min"), "Operator minutes per 1,000 verified", True), + (("measured", "maintenance_per_event_usd"), "Maintenance $ per UI change", True), + (("silent_wrong_rate",), "Silent-wrong rate", True), + (("false_stop_rate",), "False-stop rate", True), + (("measured", "median_wall_s_automated"), "Median seconds per automated item", True), +] + + +def _get(d: dict[str, Any], path: tuple[str, ...]) -> Any: + for k in path: + d = d.get(k) if isinstance(d, dict) else None + return d + + +def winners(arms: list[dict[str, Any]]) -> list[dict[str, Any]]: + out = [] + for path, label, lower in AXES: + vals = [(a["arm"], _get(a, path)) for a in arms if isinstance(_get(a, path), (int, float))] + vals = [(k, v) for k, v in vals if v == v] # drop NaN + if not vals: + continue + best = min(v for _, v in vals) if lower else max(v for _, v in vals) + best_arms = [k for k, v in vals if v == best] + sw = {a["arm"]: a["silent_wrong"] for a in arms} + caveat = [k for k in best_arms if sw.get(k)] if path[-1] != "silent_wrong_rate" else [] + out.append({"axis": label, "best": best_arms, "value": best, "all": dict(vals), + "best_has_silent_wrong": caveat}) + return out + + +def compute(rows: list[dict[str, Any]]) -> dict[str, Any]: + cm = _cost_model(rows) + table = budget.prices() + labor = table["labor"] + drift_rate = float(table["drift"]["events_per_1000_items"]) + arms = [] + for name in dict.fromkeys(r["arm"] for r in rows if r.get("type") == "arm_start"): + arms.append(arm_economics(rows, name, cm, labor, drift_rate)) + bands = {} + for f in (0.5, 2.0): + scaled = {**labor, "operator_usd_per_hour": labor["operator_usd_per_hour"] * f} + bands[f"labor_x{f:g}"] = {a["arm"]: arm_economics(rows, a["arm"], cm, scaled, drift_rate)["per_1000_verified"]["total_usd"] + for a in arms} + return { + "prices_as_of": table["as_of"], + "labor": labor, + "drift_events_per_1000_items": drift_rate, + "cost_model": cm.get("version"), + "arms": arms, + "winners": winners(arms), + "bands": bands, + } + + +def _fmt(x: Any, nd: int = 2) -> str: + if x is None or (isinstance(x, float) and x != x): + return "n/a" + if isinstance(x, float): + return f"{x:,.{nd}f}" + return str(x) + + +def render(result: dict[str, Any]) -> str: + arms = result["arms"] + lines = ["| Arm | Items | Verified | Automated | Silent wrong | False stops | Setup $ | Model $/item | Operator min /1k | Maint. $/change | Total $ /1k verified |", + "|---|---|---|---|---|---|---|---|---|---|---|"] + for a in arms: + m, p = a["measured"], a["per_1000_verified"] + lines.append( + f"| {a['arm']}{' (SHORT RUN: ' + str(a['items']) + ' of ' + str(a['planned_items']) + ' planned)' if a.get('short_run') else ''} | {a['items']} | {a['verified']} | {a['automated']} | " + f"{a['silent_wrong']} ({_fmt(a['silent_wrong_rate'], 3)}) | " + f"{a['false_stop']} of {a['false_stop_denominator']} | {_fmt(p['setup_usd'])} | " + f"{_fmt(m['per_item_model_usd'], 3)} | {_fmt(p['operator_min'], 0)} | " + f"{_fmt(m['maintenance_per_event_usd'])} | {_fmt(p['total_usd'], 0)} |" + ) + lines += ["", "| Axis | Best | Value |", "|---|---|---|"] + for w in result["winners"]: + flag = " (has silent wrongs)" if w.get("best_has_silent_wrong") else "" + lines.append(f"| {w['axis']} | {', '.join(w['best'])}{flag} | {_fmt(w['value'], 3)} |") + return "\n".join(lines) + "\n" + + +def main(argv: Optional[list[str]] = None) -> int: + args = argv if argv is not None else sys.argv[1:] + run_dir = Path(args[0]) + ledgers = sorted(run_dir.glob("**/ledger.jsonl")) + result = compute(read_rows(ledgers)) + (run_dir / "economics.json").write_text(json.dumps(result, indent=1, default=str) + "\n") + print(render(result)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/packages/flow/benchmark/lifecycle/arms/human_script.py b/packages/flow/benchmark/lifecycle/arms/human_script.py new file mode 100644 index 000000000..b8f4116b1 --- /dev/null +++ b/packages/flow/benchmark/lifecycle/arms/human_script.py @@ -0,0 +1,114 @@ +"""Arm ``human_script`` (optional): a Playwright + REST script written once by hand. + +The script (``human_script_v1.py``) was written by this benchmark's session the +way a skilled engineer would write it, from the same demonstration, procedure +and REST read the other arms get, and tested on the offline mock before it ran +against the fixture. Its authoring time is logged below. That time is a coding +agent's wall time, so it understates what a person would spend; the report +shows the setup charge with this measured time and with the cost model's +engineer assumption. + +No one maintains it during a run: when it breaks (an exception or an +"UNEXPECTED" stop), the item goes to the operator, and so does every later item +it breaks on. The fix an engineer writes afterwards is timed and reported as +the maintenance cost per drift event. +""" + +from __future__ import annotations + +import time +from pathlib import Path +from typing import Any, Optional + +from ..arm import Env, ItemResult, Operator, OperatorRequest, WorkItem +from . import budget, scripts +from .openemr_ui import OpenEMRSession, RestReader + +HERE = Path(__file__).resolve().parent + +#: Wall-clock authoring log (UTC), appended as the work happens. +AUTHORING = [ + {"what": "write v1 from the demonstration and SOP", "start": "2026-10-20T16:39:05Z", "end": "2026-10-20T16:42:39Z", + "note": "includes the first offline mock run (dry run, save, duplicate, typo date, missing date, failed save, interstitial)"}, +] + + +def authoring_seconds() -> float: + from datetime import datetime + + def t(s: str) -> float: + return datetime.strptime(s, "%Y-%m-%dT%H:%M:%SZ").timestamp() + + return sum(t(a["end"]) - t(a["start"]) for a in AUTHORING) + + +class HumanScriptArm: + name = "human_script" + + def __init__(self, options: dict[str, Any]) -> None: + self.name = str(options.get("name", self.name)) + self.version = str(options.get("version", "v1")) + self.headless = bool(options.get("headless", True)) + self.session: Optional[OpenEMRSession] = None + self.rest: Optional[RestReader] = None + self.source = (HERE / f"human_script_{self.version}.py").read_text() + self.breaks: list[dict[str, Any]] = [] + + def describe(self) -> dict[str, Any]: + return { + "arm": self.name, + "kind": "hand-written Playwright + REST script with read-back; not maintained during the run", + "version": self.version, + "authoring_log": AUTHORING, + "authoring_s": authoring_seconds(), + "per_item_model_calls": 0, + } + + def setup(self, env: Env, operator: Operator) -> list[dict[str, Any]]: + self.session = OpenEMRSession(env, headless=self.headless) + self.rest = RestReader(env) + self.session.start() + self.session.sign_in() + return [ + { + "phase": "author_script_by_hand", + "wall_s": authoring_seconds(), + # The engineer attends all of their own authoring time. + "attended_s": authoring_seconds(), + "model_calls": 0, + "cost_usd": 0.0, + "detail": {"authoring_log": AUTHORING, "note": "a coding agent's wall time; a person's would be longer"}, + } + ] + + def run_item(self, item: WorkItem, env: Env, operator: Operator) -> ItemResult: + assert self.session is not None and self.rest is not None + t0 = time.time() + out = scripts.run(self.source, self.session, self.rest, item.params, commit=True) + if out.broke: + self.breaks.append({"item_id": item.item_id, "reason": out.reason[-300:]}) + claimed = {"SAVED": "done", "NOT SAVED": "stopped"}.get(out.result, "error") + status = f"{out.result}: {out.reason}"[:500] + if claimed != "done": + operator.ask(OperatorRequest(kind="stop", item_id=item.item_id, reason=status[:300], surface="script result")) + wall = time.time() - t0 + return ItemResult( + item_id=item.item_id, + claimed=claimed, + product_status=status, + wall_s=round(wall, 2), + evidence=f"{self.name}/", + detail={"version": self.version, "broke": out.broke, "log": out.log[-10:], "usage": [], + "compute_usd": round(budget.compute_usd(wall), 6)}, + ) + + def after_drift(self, event: dict[str, Any], env: Env, operator: Operator) -> list[dict[str, Any]]: + return [] + + def close(self) -> None: + if self.session is not None: + self.session.close() + + +def factory(options: dict[str, Any]) -> HumanScriptArm: + return HumanScriptArm(options) diff --git a/packages/flow/benchmark/lifecycle/arms/human_script_v1.py b/packages/flow/benchmark/lifecycle/arms/human_script_v1.py new file mode 100644 index 000000000..9d76cd6bd --- /dev/null +++ b/packages/flow/benchmark/lifecycle/arms/human_script_v1.py @@ -0,0 +1,141 @@ +"""Register one referral in OpenEMR: the hand-written baseline script (v1). + +Written once by the session as a skilled engineer would, against the same +demonstration, procedure and REST read the other arms get. Authoring time is +in ``human_script.py`` (AUTHORING). Runs under the script contract in +``scripts.py``. +""" + +import datetime +import re +import time + + +def _name(v): + v = re.sub(r"\s+", " ", (v or "").strip()) + return v.title() if v.isupper() or v.islower() else v + + +def _dob(v): + v = (v or "").strip() + if not v: + return None + for fmt in ("%Y-%m-%d", "%m/%d/%Y", "%Y/%m/%d"): + try: + return datetime.datetime.strptime(v, fmt).strftime("%Y-%m-%d") + except ValueError: + continue + return "BAD" + + +def _one_char_apart(a, b): + return len(a) == len(b) and sum(x != y for x, y in zip(a, b)) == 1 + + +def _visible(form, sel): + return form.locator(sel + " >> visible=true").first + + +def _confirm_or_matches(page, timeout_s=30.0): + """('confirm', locator) when OpenEMR found no match, ('matches', text) otherwise.""" + + end = time.time() + timeout_s + while time.time() < end: + for fr in page.frames: + try: + btn = fr.locator("#confirmCreate") + if btn.count() and btn.first.is_visible(): + body = fr.locator("body").inner_text() + if "No matches were found" in body: + ready = btn.first.evaluate( + "e => e.ownerDocument.readyState === 'complete' && " + "typeof e.ownerDocument.defaultView.dlgclose === 'function'" + ) + if ready: + return "confirm", btn.first + else: + return "matches", body[:300] + except Exception: + continue + page.wait_for_timeout(250) + return None, None + + +def _fill(ctx, r): + form = ctx.form + form.locator("#form_title").select_option(label=r["title"]) if r.get("title") else None + for key in ("fname", "lname", "DOB"): + loc = _visible(form, f"#form_{key}") + loc.fill(r[key]) + loc.dispatch_event("change") + form.locator("#form_sex").select_option(label=r["sex"]) + contact = form.get_by_text("Contact", exact=True).first + contact.scroll_into_view_if_needed() + contact.click() + form.wait_for_timeout(300) + for key in ("street", "city", "postal_code", "phone_home", "email"): + loc = _visible(form, f"#form_{key}") + loc.fill(r.get(key, "") or "") + loc.dispatch_event("change") + _visible(form, "#form_state").select_option(label=r["state_label"]) + _visible(form, "#form_country_code").select_option(label=r.get("country_code") or "USA") + + +def register(ctx, referral): + r = dict(referral) + r["fname"], r["lname"] = _name(r.get("fname")), _name(r.get("lname")) + dob = _dob(r.get("DOB")) + if dob is None: + return {"result": "NOT SAVED", "reason": "no date of birth; return the referral to the sender"} + if dob == "BAD": + return {"result": "NOT SAVED", "reason": f"unreadable date of birth {r.get('DOB')!r}"} + r["DOB"] = dob + + same_name = [p for p in ctx.find_patients(fname=r["fname"], lname=r["lname"]) if "error" not in p] + for p in same_name: + if p.get("DOB") == dob: + return {"result": "NOT SAVED", "reason": f"patient already has a chart (pid {p.get('pid')})"} + if _one_char_apart(p.get("DOB") or "", dob) and (p.get("street") or "").upper() == r.get("street", "").upper(): + return {"result": "NOT SAVED", "reason": f"same person as pid {p.get('pid')}; the date of birth looks like a typo"} + + for attempt in (1, 2): + try: + _fill(ctx, r) + ctx.form.locator("#create").click(timeout=15_000) + except Exception as exc: + return {"result": "NOT SAVED", "reason": f"UNEXPECTED: form step failed: {str(exc).splitlines()[0][:200]}"} + kind, what = _confirm_or_matches(ctx.page) + if kind == "matches": + return {"result": "NOT SAVED", "reason": "OpenEMR's duplicate check lists a matching patient"} + if kind is None: + return {"result": "NOT SAVED", "reason": "UNEXPECTED: no duplicate-check dialog after Create"} + what.click() + if not ctx.commit: + return {"result": "NOT SAVED", "reason": "DRY RUN reached confirm"} + found = [] + for _ in range(20): + ctx.page.wait_for_timeout(500) + found = [p for p in ctx.find_patients(fname=r["fname"], lname=r["lname"], DOB=dob) if "error" not in p] + if found: + break + if len(found) == 1: + p = found[0] + if (p.get("street") or "").upper() == r["street"].upper() and (p.get("email") or "") == r["email"]: + return {"result": "SAVED", "reason": f"read back pid {p.get('pid')}"} + return {"result": "NOT SAVED", "reason": f"read-back differs for pid {p.get('pid')}"} + if len(found) > 1: + return {"result": "NOT SAVED", "reason": f"read-back found {len(found)} charts"} + ctx.log(f"attempt {attempt}: no chart after confirm") + if attempt == 1: + ctx.page.reload() + ctx.page.wait_for_timeout(1000) + # The runner's blank form is gone after a reload; reopen it. + ctx.page.evaluate( + "() => { window.navigateTab('/interface/new/new_comprehensive.php', " + "'openadapt-new-patient'); window.activateTabByName('openadapt-new-patient', true); }" + ) + ctx.page.wait_for_timeout(2000) + for fr in ctx.page.frames: + if "new_comprehensive.php" in (fr.url or ""): + ctx.form = fr + return {"result": "NOT SAVED", "reason": "the save did not land after two attempts"} diff --git a/packages/flow/benchmark/lifecycle/arms/mock_openemr.py b/packages/flow/benchmark/lifecycle/arms/mock_openemr.py new file mode 100644 index 000000000..39d73b54e --- /dev/null +++ b/packages/flow/benchmark/lifecycle/arms/mock_openemr.py @@ -0,0 +1,195 @@ +"""A small in-browser stand-in for OpenEMR's referral flow, for offline tests of the arms. + +It is not OpenEMR. It copies only what the arms touch: the sign-in page, the tab +shell's ``navigateTab``/``activateTabByName``, the New Patient form's control +ids, the collapsed Contact section, the duplicate-check popup with +``#confirmCreate`` and "No matches were found.", the save POST to +``new_comprehensive_save.php``, and a REST patient search. Served through +Playwright routes on ``http://mock-openemr.test``; nothing listens on a port. + +Drift kinds mirror the soak's: ``relabel`` renames the save button, +``interstitial`` puts a notice over the form that must be dismissed first. +""" + +from __future__ import annotations + +import json +import urllib.parse +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any, Optional + +BASE = "http://mock-openemr.test" + +LOGIN = """

OpenEMR

+ + +""" + +MAIN = """
+""" + +FORM = """ +%(interstitial)s + + + + + + +
Title
First Name
Last Name
DOB
Sex
+
+Contact + + +""" + +INTERSTITIAL = """
+

Release notes: the registration form has moved fields.

+
""" + + +@dataclass +class MockOpenEMR: + """The store and the routes. ``patients`` is the system of record.""" + + patients: list[dict[str, str]] = field(default_factory=list) + drift: set[str] = field(default_factory=set) + fail_next_save: int = 0 + logged_in: bool = False + + def seed(self, person: dict[str, str]) -> None: + self.add(person) + + def add(self, p: dict[str, str]) -> dict[str, str]: + row = {k: p.get(k, "") for k in ("title", "fname", "lname", "DOB", "sex", "street", "city", + "postal_code", "phone_home", "email")} + row["state"] = p.get("state") or ("MA" if p.get("state_label") == "Massachusetts" else p.get("state_label", "")) + row["country_code"] = p.get("country_code", "USA") + row["pid"] = str(len(self.patients) + 1) + row["uuid"] = f"mock-{row['pid']}" + self.patients.append(row) + return row + + def search(self, **q: str) -> list[dict[str, str]]: + def ok(p: dict[str, str]) -> bool: + return all(not v or p.get(k, "").lower() == str(v).lower() for k, v in q.items()) + + return [dict(p) for p in self.patients if ok(p)] + + # -- Playwright routes --------------------------------------------------------- + + def install(self, context: Any) -> None: + context.route(BASE + "/**", self._route) + + def _route(self, route: Any) -> None: + req = route.request + url = urllib.parse.urlparse(req.url) + path = url.path + html = None + if path.startswith("/interface/login"): + html = LOGIN + elif path == "/interface/main/tabs/main.php": + self.logged_in = True + html = MAIN + elif path == "/interface/new/new_comprehensive.php": + html = FORM % { + "save_label": "Register patient" if "relabel" in self.drift else "Create New Patient", + "interstitial": INTERSTITIAL if "interstitial" in self.drift else "", + } + elif path == "/interface/new/dup_check.php": + q = dict(urllib.parse.parse_qsl(url.query)) + matches = self.search(fname=q.get("fname", ""), lname=q.get("lname", "")) + if matches: + rows = "".join(f"
  • {m['fname']} {m['lname']} {m['DOB']}
    • " for m in matches) + html = f"

    Possible matches

      {rows}
    " + else: + html = ("

    No matches were found.

    " + "" + "") + elif path == "/interface/new/new_comprehensive_save.php": + if self.fail_next_save > 0: + self.fail_next_save -= 1 + route.fulfill(status=500, body="error") + return + self.add(json.loads(req.post_data or "{}")) + route.fulfill(status=200, body="ok") + return + if html is None: + route.fulfill(status=404, body="not found") + else: + route.fulfill(status=200, content_type="text/html", body=html) + + +class _Resp: + def __init__(self, data: Any) -> None: + self.status_code = 200 + self._data = data + + def json(self) -> Any: + return self._data + + +class MockRestSession: + def __init__(self, app: MockOpenEMR) -> None: + self.app = app + + def get(self, url: str, params: Optional[dict[str, str]] = None) -> _Resp: + return _Resp({"data": self.app.search(**(params or {}))}) + + +def mock_env(app: MockOpenEMR, workdir: Path) -> Any: + """An ``Env`` for the arms, pointed at the mock.""" + + from ..arm import Env + + return Env( + app="openemr", + ui_url=BASE, + api_url=BASE + "/rest", + accounts=(("clerk", "synthetic"),), + workdir=workdir, + seed=0, + extra={"context_hook": app.install, "rest_session": lambda: MockRestSession(app)}, + ) diff --git a/packages/flow/benchmark/lifecycle/arms/openemr_ui.py b/packages/flow/benchmark/lifecycle/arms/openemr_ui.py new file mode 100644 index 000000000..9c7501860 --- /dev/null +++ b/packages/flow/benchmark/lifecycle/arms/openemr_ui.py @@ -0,0 +1,266 @@ +"""The OpenEMR plumbing every non-OpenAdapt arm shares. + +All arms get the same start for every item: a browser signed in as the clerk, +looking at a blank New Patient form opened through OpenEMR's own tab shell +(the form needs ``top.restoreSession`` from that shell to save). When the +session has ended (the soak's ``session_expiry`` event), the same runner code +signs in again before handing the page over. This is the runner's job in a +deployment of any of these arms, so it is the same for all of them and is not +charged to the arm's model budget. It is recorded per item (``signed_in_again``). + +The REST read is read-only (the harness's ``rest_read_session``, scope +``user/patient.rs``), the same REST read-back OpenAdapt's record check uses. No arm ever sees the +harness's oracle. +""" + +from __future__ import annotations + +import time +from dataclasses import dataclass, field +from typing import Any, Optional + +FORM_PATH = "/interface/new/new_comprehensive.php" +FORM_FRAME_NAME = "openadapt-new-patient" +VIEWPORT = {"width": 1280, "height": 800} + +#: The fields a referral sets, with the New Patient form's control ids. +FORM_IDS = { + "title": "form_title", + "fname": "form_fname", + "lname": "form_lname", + "DOB": "form_DOB", + "sex": "form_sex", + "street": "form_street", + "city": "form_city", + "state_label": "form_state", + "postal_code": "form_postal_code", + "phone_home": "form_phone_home", + "email": "form_email", + "country_code": "form_country_code", +} + +SOP = """\ +Register one referred patient in OpenEMR from the referral below. +Rules (the clinic's standard procedure): +- Create exactly one new patient chart with the referral's values. Dates of \ +birth go in as YYYY-MM-DD; names as the person's name (trim stray spaces, \ +normal capitalisation). +- If OpenEMR shows that a matching patient may already exist, or you otherwise \ +find this person already has a chart (same name and date of birth, or the same \ +name and a date of birth that differs only by an obvious typo with the same \ +address), do not create another chart: stop and say so. +- If the referral has no date of birth, do not create a chart: stop and say \ +the referral goes back to the sender. A missing home phone is fine. +- If the save fails, you may try once more; never leave two charts. +- Work only in the OpenEMR page. All data is synthetic test data. +End your final reply with exactly one line, either +RESULT: SAVED +or +RESULT: NOT SAVED: """ + + +def claimed_from(text: str) -> tuple[str, str]: + """(``done`` | ``stopped``, last line) from an agent's final reply.""" + + lines = [ln.strip() for ln in (text or "").strip().splitlines() if ln.strip()] + last = lines[-1] if lines else "" + for ln in reversed(lines): + if ln.upper().startswith("RESULT:"): + last = ln + break + up = last.upper() + if up.startswith("RESULT: SAVED"): + return "done", last + return "stopped", last or "(no final reply)" + + +@dataclass +class OpenEMRSession: + """One Playwright browser, signed in, reused across items.""" + + env: Any + headless: bool = True + _pw: Any = None + _browser: Any = None + context: Any = None + page: Any = None + signins: int = 0 + log: list[dict[str, Any]] = field(default_factory=list) + + @property + def ui_url(self) -> str: + return self.env.ui_url.rstrip("/") + + def start(self) -> None: + if self.page is not None: + return + from playwright.sync_api import sync_playwright + + self._pw = sync_playwright().start() + self._browser = self._pw.chromium.launch(headless=self.headless) + self.context = self._browser.new_context(viewport=VIEWPORT, ignore_https_errors=True) + hook = getattr(self.env, "extra", {}).get("context_hook") + if hook is not None: # the offline mock serves the app through routes + hook(self.context) + self.page = self.context.new_page() + + def close(self) -> None: + for obj, how in ((self._browser, "close"), (self._pw, "stop")): + try: + if obj is not None: + getattr(obj, how)() + except Exception: # noqa: BLE001 - closing is best effort + pass + self._pw = self._browser = self.context = self.page = None + + def _account(self) -> tuple[str, str]: + if self.env.accounts: + return self.env.accounts[0] + fx = self.env.extra["fixture"] + v = fx._runtime_values() + return v["OPENEMR_ACTOR_USER"], v["OPENEMR_ACTOR_PASSWORD"] + + def sign_in(self) -> None: + page = self.page + user, password = self._account() + page.goto(self.ui_url + "/interface/login/login.php?site=default") + page.locator("#authUser").fill(user) + page.locator("#clearPass").fill(password) + page.locator("#login-button").click() + page.wait_for_url("**/interface/main/tabs/main.php**", timeout=90_000) + self.signins += 1 + self._dismiss_registration() + + def _shell_ready(self) -> bool: + try: + return bool( + self.page.evaluate( + "() => typeof window.navigateTab === 'function' " + "&& typeof window.activateTabByName === 'function'" + ) + ) + except Exception: # noqa: BLE001 + return False + + def _dismiss_registration(self) -> None: + reg = self.page.locator(".product-registration-modal") + try: + if reg.count() and reg.is_visible(): + tel = reg.locator("#allowTelemetry") + if tel.count() and tel.is_checked(): + tel.uncheck() + reg.get_by_role("button", name="Ask again later", exact=True).click() + reg.wait_for(state="hidden", timeout=30_000) + except Exception: # noqa: BLE001 - the dialog only shows on a fresh profile + pass + + def form_frame(self, timeout_s: float = 60.0) -> Any: + deadline = time.monotonic() + timeout_s + while time.monotonic() < deadline: + frame = self.page.frame(name=FORM_FRAME_NAME) + if frame is None: + try: + h = self.page.locator(f'iframe[name="{FORM_FRAME_NAME}"]').element_handle() + frame = None if h is None else h.content_frame() + except Exception: # noqa: BLE001 + frame = None + if frame is not None and FORM_PATH in (frame.url or ""): + return frame + self.page.wait_for_timeout(150) + raise RuntimeError("the New Patient form frame did not open") + + def blank_form(self) -> dict[str, Any]: + """Leave the page on a blank New Patient form; sign in again if needed.""" + + self.start() + t0 = time.time() + again = False + if not self._shell_ready() or "login" in (self.page.url or ""): + self.sign_in() + again = self.signins > 1 + # Close any earlier form tab, then open a fresh one. + try: + self.page.evaluate( + "({path, name}) => { window.navigateTab(path, name); " + "window.activateTabByName(name, true); }", + {"path": FORM_PATH + "?_t=" + str(int(time.time() * 1000)), "name": FORM_FRAME_NAME}, + ) + frame = self.form_frame() + frame.locator("#form_fname").wait_for(state="visible", timeout=60_000) + except Exception: + # The shell may have been logged out under us (expired session). + self.sign_in() + again = True + self.page.evaluate( + "({path, name}) => { window.navigateTab(path, name); " + "window.activateTabByName(name, true); }", + {"path": FORM_PATH, "name": FORM_FRAME_NAME}, + ) + frame = self.form_frame() + frame.locator("#form_fname").wait_for(state="visible", timeout=60_000) + self._dismiss_registration() + return {"setup_s": round(time.time() - t0, 2), "signed_in_again": again} + + +class RestReader: + """Read-only use of OpenEMR's patient API, as the clerk's integration.""" + + def __init__(self, env: Any) -> None: + self.env = env + self._session: Any = None + self._issued = 0.0 + + def _s(self) -> Any: + # Tokens are short-lived; take a fresh one every 5 minutes. + if self._session is None or time.time() - self._issued > 300: + getter = self.env.extra.get("rest_read_session") or self.env.extra.get("rest_session") + if getter is not None: + self._session = getter() + else: + self._session = self.env.extra["fixture"].token_session("actor") + self._issued = time.time() + return self._session + + @property + def base(self) -> str: + api = self.env.api_url or self.env.extra["fixture"].api_base_url + return api.rstrip("/") + "/apis/default/api" + + def find(self, **query: str) -> list[dict[str, Any]]: + q = {k: v for k, v in query.items() if v} + r = self._s().get(self.base + "/patient", params=q) + if r.status_code != 200: + return [{"error": f"HTTP {r.status_code}"}] + data = r.json() + rows = data.get("data") if isinstance(data, dict) else data + out = [] + for p in rows or []: + out.append( + { + k: p.get(k) + for k in ( + "uuid", "pid", "title", "fname", "lname", "DOB", "sex", + "street", "city", "state", "postal_code", "phone_home", "email", + ) + } + ) + return out + + +def fill_value(frame: Any, field_id: str, value: str) -> None: + """Set one New Patient control the way a person would.""" + + loc = frame.locator(f"#{field_id}") + tag = loc.evaluate("e => e.tagName.toLowerCase()") + if tag == "select": + try: + loc.select_option(label=value) + except Exception: # noqa: BLE001 - some lists key by value + loc.select_option(value=value) + else: + loc.fill(value) + loc.dispatch_event("change") + + +def deadline_ok(started: float, limit_s: Optional[float]) -> bool: + return limit_s is None or time.time() - started < limit_s diff --git a/packages/flow/benchmark/lifecycle/arms/prices.json b/packages/flow/benchmark/lifecycle/arms/prices.json new file mode 100644 index 000000000..19d8f4f8d --- /dev/null +++ b/packages/flow/benchmark/lifecycle/arms/prices.json @@ -0,0 +1,56 @@ +{ + "as_of": "2026-10-20", + "currency": "USD", + "unit": "per million tokens", + "source": "Anthropic first-party API list prices, as listed in the claude-api reference cached 2026-10-06 and checked 2026-10-20. Cache writes are the 5-minute TTL (1.25x input). Haiku 5.5 cache prices are derived from the same rule (0.1x read, 1.25x write) and are marked derived.", + "models": { + "claude-fable-5-1": { + "input": 10.0, + "output": 50.0, + "cache_read": 0.25, + "cache_write": 12.5 + }, + "claude-opus-5-5": { + "input": 4.0, + "output": 20.0, + "cache_read": 0.2, + "cache_write": 5.0 + }, + "claude-sonnet-5-5": { + "input": 2.0, + "output": 10.0, + "cache_read": 0.2, + "cache_write": 2.5 + }, + "claude-haiku-5-5": { + "input": 0.1, + "output": 0.5, + "cache_read": 0.01, + "cache_write": 0.125, + "derived": [ + "cache_read", + "cache_write" + ], + "note": "prompts up to 100K tokens; $0.50/$2.50 beyond" + } + }, + "compute": { + "note": "Local compute per item is charged at a nominal rate so arms that hold a browser longer cost more. One worker = one browser plus its share of one 4 vCPU / 16 GB host.", + "host_usd_per_hour": 0.2, + "host_source": "a nominal 4 vCPU / 16 GB on-demand cloud VM, round figure; change here, never in code" + }, + "labor": { + "note": "Assumptions, not measurements. People's time is priced only to put every component in one currency; the report also gives every component in minutes, and shows the total at half and double these rates.", + "operator_usd_per_hour": 30.0, + "operator_source": "round figure for a fully loaded US medical front-desk / registration clerk hour (wage plus overhead); assumption", + "engineer_usd_per_hour": 100.0, + "engineer_source": "round figure for a fully loaded automation engineer hour; assumption", + "engineer_script_author_s": 7200.0, + "engineer_script_fix_s": 1800.0, + "engineer_source_times": "assumed time for a skilled engineer to write and test this script (2 h) and to diagnose and fix it after one UI change (30 min); used only for the human_script arm's assumption column" + }, + "drift": { + "events_per_1000_items": 4, + "note": "assumed for the per-1,000 projection: the soak's four UI-change kinds, once each per 1,000 items (about one vendor release per few weeks at a clinic's referral volume). The measured stream has 4 changes in 40 items; the report gives maintenance per change so any rate can be applied." + } +} diff --git a/packages/flow/benchmark/lifecycle/arms/report.py b/packages/flow/benchmark/lifecycle/arms/report.py new file mode 100644 index 000000000..d55f045ff --- /dev/null +++ b/packages/flow/benchmark/lifecycle/arms/report.py @@ -0,0 +1,141 @@ +"""REPORT.md for one comparison run: tables from ``economics.json``, findings from ``findings.md``. + +The tables are generated. The findings (which arm wins on which axis, where +OpenAdapt loses, what the numbers do not show) are written by a person after +reading the tables, kept in ``findings.md`` beside the ledgers, and included +as is. +""" + +from __future__ import annotations + +import json +from pathlib import Path +from typing import Any + +from .economics import _fmt + + +def _components(result: dict[str, Any]) -> list[str]: + rows = [ + "| Arm | Setup: people min | Setup: model $ | Model $ per item | Compute $ per item | Operator min per item | Maint. people min per change | Maint. model $ per change | Median s per automated item |", + "|---|---|---|---|---|---|---|---|---|", + ] + for a in result["arms"]: + m = a["measured"] + n = a["items"] or 1 + rows.append( + f"| {a['arm']} | {_fmt(m['setup_human_min'], 1)} | {_fmt(m['setup_model_usd'])} | " + f"{_fmt(m['per_item_model_usd'], 3)} | {_fmt(m['compute_usd'] / n, 4)} | " + f"{_fmt(m['operator_min'] / n, 2)} | {_fmt(m['maintenance_per_event_min'], 1)} | " + f"{_fmt(m['maintenance_model_usd'] / a['drift_events'] if a['drift_events'] else 0.0)} | " + f"{_fmt(m['median_wall_s_automated'], 1)} |" + ) + return rows + + +def _projection(result: dict[str, Any]) -> list[str]: + rows = [ + "| Arm | Setup $ | Model $ | Compute $ | Operator min | Operator $ | Maint. changes | Maint. $ | Silent wrongs | Cleanup $ (lower bound) | Total $ | Total people min | Total $ at labor x0.5 / x2 |", + "|---|---|---|---|---|---|---|---|---|---|---|---|---|", + ] + for a in result["arms"]: + p = a["per_1000_verified"] + lo = result["bands"]["labor_x0.5"].get(a["arm"]) + hi = result["bands"]["labor_x2"].get(a["arm"]) + rows.append( + f"| {a['arm']} | {_fmt(p['setup_usd'])} | {_fmt(p['item_model_usd'])} | {_fmt(p['compute_usd'])} | " + f"{_fmt(p['operator_min'], 0)} | {_fmt(p['operator_usd'])} | {_fmt(p['maintenance_events'], 1)} | " + f"{_fmt(p['maintenance_usd'])} | {_fmt(p['silent_wrong'], 0)} | {_fmt(p['cleanup_usd'])} | **{_fmt(p['total_usd'], 0)}** | {_fmt(p['total_human_min'], 0)} | " + f"{_fmt(lo, 0)} / {_fmt(hi, 0)} |" + ) + return rows + + +def _quality(result: dict[str, Any]) -> list[str]: + rows = [ + "| Arm | Items | Verified (final) | Automated | Done by hand | Silent wrong (95% CI) | False stops (95% CI) | Outcomes |", + "|---|---|---|---|---|---|---|---|", + ] + def ci(c: Any) -> str: + return f" ({c[0]:.2f}–{c[1]:.2f})" if c else "" + + for a in result["arms"]: + outcomes = ", ".join(f"{k} {v}" for k, v in sorted(a["outcomes"].items())) + rows.append( + f"| {a['arm']} | {a['items']} | {a['verified']} | {a['automated']} | {a['by_hand']} | " + f"{a['silent_wrong']}{ci(a['silent_wrong_ci95'])} | " + f"{a['false_stop']} of {a['false_stop_denominator']}{ci(a['false_stop_ci95'])} | {outcomes} |" + ) + return rows + + +def _config(result: dict[str, Any]) -> list[str]: + out = [] + for a in result["arms"]: + d = a.get("describe") or {} + keep = {k: d[k] for k in ("kind", "model", "effort", "tool", "tools", "item_budget_usd", + "max_turns_per_item", "profile", "api_tier", "sees", "reads_record") + if k in d} + out.append(f"- **{a['arm']}**: `{json.dumps(keep, default=str)}`") + return out + + +def render(result: dict[str, Any], out: Path, plan: dict[str, Any]) -> str: + labor = result["labor"] + parts = [ + "# Lifecycle economics: OpenAdapt against the alternatives", + "", + f"Seed {plan['seed']}, {plan['items']} items over {plan['days']} days on the local OpenEMR test " + f"fixture (synthetic patients only). Every arm ran the same worklist and the same event ledger " + f"(UI changes, duplicates, expiry, concurrent edits, failed saves) and was judged by the same " + f"independent oracle and the same scripted operator. Model prices as of {result['prices_as_of']} " + f"(`arms/prices.json`); operator cost model `{result['cost_model']}`; people's time at " + f"${labor['operator_usd_per_hour']:.0f}/h (an assumption; bands below).", + "", + "## Total cost per 1,000 verified items", + "", + "Setup once, per-item costs scaled to 1,000 verified items, maintenance at " + f"{result['drift_events_per_1000_items']:g} UI changes per 1,000 items (assumed; per-change cost " + "is measured and shown below).", + "", + *_projection(result), + "", + "## Which arm wins on which axis", + "", + "| Axis | Best | Value | All arms |", + "|---|---|---|---|", + *[ + f"| {w['axis']} | {', '.join(w['best'])}{' (has silent wrongs)' if w.get('best_has_silent_wrong') else ''} | {_fmt(w['value'], 3)} | " + + ", ".join(f"{k} {_fmt(v, 3)}" for k, v in w["all"].items()) + + " |" + for w in result["winners"] + ], + "", + "## Correctness", + "", + *_quality(result), + "", + "## Measured components", + "", + *_components(result), + "", + "## Arm configurations", + "", + *_config(result), + "", + ] + findings = out / "findings.md" + if findings.is_file(): + parts += [findings.read_text().strip(), ""] + parts += [ + "## Reproduce", + "", + "```bash", + "cd packages/flow/benchmark", + f"python -m lifecycle.arms.compare --out {out.as_posix().split('benchmark/')[-1]} " + f"--seed {plan['seed']} --items {plan['items']} --days {plan['days']}", + f"python -m lifecycle.arms.economics {out.as_posix().split('benchmark/')[-1]} # recompute from the ledgers", + "```", + "", + ] + return "\n".join(parts) diff --git a/packages/flow/benchmark/lifecycle/arms/scripts.py b/packages/flow/benchmark/lifecycle/arms/scripts.py new file mode 100644 index 000000000..db8739b22 --- /dev/null +++ b/packages/flow/benchmark/lifecycle/arms/scripts.py @@ -0,0 +1,176 @@ +"""What both script arms share: the script contract, the sandbox, and the runner. + +A script is a Python module that defines:: + + def register(ctx, referral: dict) -> dict: + # returns {"result": "SAVED" | "NOT SAVED", "reason": str} + +``ctx`` gives the script exactly what the other arms get: + +- ``ctx.page``: the Playwright page, already signed in, on a blank New Patient + form; ``ctx.form``: that form's frame. +- ``ctx.find_patients(fname=, lname=, DOB=)``: the clerk's REST search. +- ``ctx.commit``: False in a dry run. In a dry run the save request + (``new_comprehensive_save.php``) is blocked at the network layer, so a dry + run cannot write, whatever the script does. +- ``ctx.log(msg)``: a line in the run's evidence. + +The sandbox is a static check, not isolation: a script may import only +``re``, ``time``, ``datetime`` and ``json``, and may not name ``open``, +``exec``, ``eval``, ``compile``, ``__import__``, ``globals``, ``os``, +``sys`` or ``subprocess``. Scripts touch only the local synthetic fixture. +""" + +from __future__ import annotations + +import ast +import time +import traceback +from dataclasses import dataclass, field +from typing import Any, Callable + +ALLOWED_IMPORTS = {"re", "time", "datetime", "json"} +FORBIDDEN_NAMES = { + "open", "exec", "eval", "compile", "__import__", "globals", "locals", "os", "sys", + "subprocess", "importlib", "builtins", "__builtins__", "breakpoint", "input", +} +SAVE_URL_PART = "new_comprehensive_save.php" + + +class ScriptRejected(ValueError): + """The script failed the static check.""" + + +def check_source(source: str) -> None: + tree = ast.parse(source) + for node in ast.walk(tree): + if isinstance(node, ast.Import): + for a in node.names: + if a.name.split(".")[0] not in ALLOWED_IMPORTS: + raise ScriptRejected(f"import {a.name} is not allowed") + elif isinstance(node, ast.ImportFrom): + if (node.module or "").split(".")[0] not in ALLOWED_IMPORTS: + raise ScriptRejected(f"from {node.module} import is not allowed") + elif isinstance(node, ast.Name) and node.id in FORBIDDEN_NAMES: + raise ScriptRejected(f"name {node.id} is not allowed") + elif isinstance(node, ast.Attribute) and node.attr.startswith("__"): + raise ScriptRejected(f"attribute {node.attr} is not allowed") + if not any(isinstance(n, ast.FunctionDef) and n.name == "register" for n in tree.body): + raise ScriptRejected("the script must define register(ctx, referral)") + + +def load(source: str) -> Callable[..., dict[str, Any]]: + check_source(source) + ns: dict[str, Any] = {"__name__": "lifecycle_script"} + exec(compile(source, "