diff --git a/packages/flow/benchmark/economic_comparison/README.md b/packages/flow/benchmark/economic_comparison/README.md new file mode 100644 index 000000000..6052b3626 --- /dev/null +++ b/packages/flow/benchmark/economic_comparison/README.md @@ -0,0 +1,299 @@ +# Economic comparison: OpenAdapt against competent alternatives + +This directory compares OpenAdapt with three alternatives a competent team +could build today. It counts every person's minute and the total cost over +10, 100 and 1,000 items, not only the per-run speed. Every number in the +tables comes from the raw rows listed under [Raw results](#raw-results). +People's time is priced from stated assumptions +([`assumptions.json`](assumptions.json)), never measured. + +Measured on 2026-10-24. Synthetic data only. + +## Short answer + +On these two tasks OpenAdapt does not win the economics. + +- **OpenEMR referral intake (55 items, one app change).** The agent with + structured tools and REST read-back beat OpenAdapt's production + configuration on most rows: + - automated items: 49 vs 41; + - false stops: 5 vs 9 of 35; + - median time: 24 s vs 119 s an item; + - operator time: 0.91 vs 2.15 minutes an item; + - projected total for 1,000 items: $606 vs $1,158. + + OpenAdapt was better only on model cost ($0 vs $0.05 an item). Neither arm + made a silent wrong save (0 of 55 each). +- **MockMed fault suite (100 items, 10 UI changes, 5 server faults).** Both + competent alternatives verified 90 of 90 items with 0 silent wrong saves. + The agent-written script cost $0.26 in model calls for all 100 items, + including four repairs of its own. OpenAdapt's program verified 71 of 90 + with 0 silent wrong saves and 19 false stops. The 1,000-item totals are + within about 30% of each other ($104 to $135). OpenAdapt is lowest in + dollars and highest in people minutes. +- **The silent-wrong advantage holds only against an agent that trusts the + screen.** The screenshot-only computer-use agent reported all 5 server-side + bad saves as saved. Every arm that reads the record back reported none of + them as saved, whether it was OpenAdapt or not. + +## The arms + +Every arm gets the same task text, the same app and the same read access to +the system of record that OpenAdapt's record check uses: REST on OpenEMR, +`GET /api/db` on MockMed. No arm gets the fault, and no arm judges itself. On +OpenEMR, the oracle is a direct SQL read; on MockMed, it is the store's rows +before and after each item. + +| # | Arm | What it is | Model | +|---|---|---|---| +| 1 | Agent with structured tools | Reads the page's controls (DOM and accessibility view, not screenshots), acts on them by reference, and reads the record before and after the save. It is told to claim success only when exactly one correct record landed. On OpenEMR: [`lifecycle/arms/agent_tools.py`](../lifecycle/arms/agent_tools.py). On MockMed: `AgentTools` in [`silent_saves/agent_arms.py`](../silent_saves/agent_arms.py). | Claude Sonnet 5.5 | +| 2 | Agent-written script | A coding agent writes a Playwright script with the same checks: a duplicate guard before the save and a read-back after it. It dry-runs the script with the save blocked at the network layer, then submits it. Items then run with no model call. When the script breaks, the agent fixes it against the live page and the item runs again. On OpenEMR: [`lifecycle/arms/agent_script.py`](../lifecycle/arms/agent_script.py). On MockMed: `AgentScript` in [`silent_saves/agent_arms.py`](../silent_saves/agent_arms.py). | Claude Sonnet 5.5 | +| 3 | OpenAdapt, as a customer would run it | On OpenEMR: the Regulated profile, `problems_only` approval, durable runs, pipelined read-back, self-repair for a relabeled save button, and the governed run gate on the sealed bundle. This is the production-lifecycle session's arm, `lifecycle/production_arm.py`, run on its branch's engine. The API step is refused on this form (see below), so the GUI save runs. On MockMed: the admitted program with read-back and no model calls. The MockMed program does not run the Regulated profile. | none | +| 4 | Computer-use agent, screenshots only | The scoreboard's reference agent. Measured on MockMed only (committed runs of 2026-10-19). | Claude Sonnet 5.5 | + +A fifth MockMed row runs arm 4's own runs with each claim gated on +OpenAdapt's read-back (`verify-run`). That is the cheapest way to add the +read-back to an existing agent. + +### The API step + +The 3.33 s API-path figure was measured without approval gates or durable +runs. On current main the compiler refuses the API step for this form: +OpenEMR validates the date of birth with a script rule the API step can't +evaluate, and another write happens during the form span. So OpenAdapt's +gated OpenEMR numbers below are the GUI path's +([`../lifecycle/results/api_tier/`](../lifecycle/results/api_tier/)). + +## OpenEMR referral intake + +Seed 7, 55 referrals over 10 simulated days, workload +`lifecycle-workload/2026-10-24.1`. About a third of the referrals carry an +exception: duplicates, resends, shared surnames and dates of birth, a missing +date of birth, malformed fields, other clerks, a forced logout, slow and +failed saves. One app change: the save button "Confirm Create New Patient" +becomes "Save new patient" from item 26 on. Local OpenEMR 8.0.0.3. Host +1-minute load was about 5 to 20 during these runs. + +| Arm | Items | Right in the end | Automated (no hand entry) | Silent wrong | False stops (of clean items) | Median s an item | Model $ an item | Operator min an item | After the app change | Model $ in total | +|---|---|---|---|---|---|---|---|---|---|---| +| 3. OpenAdapt, production configuration | 55 | 53 | 41 | 0 | 9 of 35 | 118.5 | 0 | 2.15 | 7 of 30 later items false-stopped; 7.5 min of maintenance; first verified item 2 items after the change | 0 | +| 3 on current main | 27 (stopped) | 27 | 4 | 0 | 19 of 20 | 226.9 | 0 | 4.56 | not reached | 0 | +| 1. Agent with tools | 55 | 55 | 49 | 0 | 5 of 35 | 24.3 | 0.050 | 0.91 | no stop in items 26-42; 6 stops in items 43-54 (a stuck confirm popup) | 2.77 | +| 2. Agent-written script | 16 (hung) | 16 | 7 | 0 | 7 of 12 | 5.3 | 0 | 2.44 | not reached | 0.86 | +| By hand (reference) | | | 0 | | | 210 | 0 | 3.50 | | 0 | + +"Right in the end" counts items the oracle judges right, whoever did them. +Items an arm stopped on were entered by hand by the scripted operator, and +that time is in "Operator min an item". The silent-wrong 95% CI for 0 of 55 +is 0 to 6.5%. + +What happened to each arm: + +- **OpenAdapt.** It stopped on every possible-duplicate listing. That is + right for real duplicates, and a false stop for new patients who share a + surname or a date of birth with an existing chart (9 false stops). On + item 26, the first after the relabel, it stopped after the save had + landed (`RECONCILIATION_REQUIRED`), so a person reconciled it. A person + approved the proposed relabel repair once, and later items ran on the + repaired program. `problems_only` still asked a person to approve the save + click on 30 items (20 s each in the cost model). +- **OpenAdapt on current main** created no chart by itself in 27 items. Its + duplicate check stopped on every listing. +- **Agent with tools.** It checked the record before saving and judged the + duplicate listings itself. It stopped on all 10 items where stopping was + right: existing charts, resends, another clerk's registration, an + ambiguous match and missing dates of birth. It handled the relabel with + no stop. From item 43 on it stopped 6 + times: the confirm popup did not close after a click, and the agent + reported "not saved" rather than saving twice. None of those items left a + chart, so they are false stops, not wrong saves. +- **Agent-written script.** Its first script stopped on every + possible-duplicate listing (5 items). It also timed out on the title field + on 4 items; the agent fixed it 4 times. After item 16 the run made no + progress for about 2 hours and was stopped to free the shared fixture + (ledger row `stopped: hung`). No stack was captured, so the cause is not + known; every wait in the agent's script had a timeout, which points at a + harness call with none (such as `page.evaluate` in OpenEMR's tab shell). + Its numbers are 16 items before the app change, and are not comparable to + the full runs. The hand-written script and the screenshot agent were not + run on OpenEMR. `lifecycle.arms.compare` now bounds every item + (`LIFECYCLE_ITEM_TIMEOUT_S`, default 900 s). On a timeout it writes every + thread's stack to `stacks.txt`, kills the arm's browser driver, records + the item as an error and gives the arm a fresh browser. If the arm still + does not return, the process exits after 3 more minutes. The remaining + OpenEMR arms are queued for the fixture as `baselines-rest`. + +### Totals for OpenEMR + +Setup + items + app changes (4 per 1,000 items) + cleanup of silent wrongs. +People time is in the brackets at x0.5 and x2 of the assumed rates. + +| Arm | 10 items | 100 items | 1,000 items | People minutes for 1,000 | +|---|---|---|---|---| +| 3. OpenAdapt, production configuration | $72 [36-144] | $171 [86-340] | $1,158 [584-2,307] | 2,228 | +| 3 on current main | $84 | $290 | $2,351 | 4,608 | +| 1. Agent with tools | $105 [53-210] | $151 [78-296] | $606 [329-1,161] | 969 | +| 2. Agent-written script (16 items only) | $130 | $239 | $1,337 | 2,508 | +| By hand | $18 | $175 | $1,750 | 3,500 | + +At 10 items, entering every referral by hand costs least. From 100 items on, +the agent with tools costs least; that holds at x0.5 and x2. OpenAdapt's +production configuration removes about 36% of the people time of doing +everything by hand, and the agent with tools about 72%. + +## MockMed fault suite + +[`silent_saves`](../silent_saves/README.md) fault set 1.1.0: 100 runs of +"open Jane Sample's referral, start a Triage encounter, save this note". Ten +blocks each switch one UI change on: renamed controls, display scaling, moved +buttons, a What's New screen, latency, a reordered queue, a moved New +Encounter control, a look-alike referral, a relabeled type and a missing +referral. Five items get a server-side bad save. The store is read before and +after every item. + +| Arm | Verified (of 90) | Silent wrong (of 100) | False stops | Server faults reported saved (of 5) | Stopped when no save existed (of 5) | Model $ for 100 items | Seconds an item, control block | +|---|---|---|---|---|---|---|---| +| 1. Agent with tools | 90 | 0 | 0 | 0 | 5 | 1.45 | 13.7 | +| 2. Agent-written script | 90 | 0 | 0 | 0 | 5 | 0.26 | 0.8 | +| 3. OpenAdapt program (current main) | 71 | 0 | 19 | 0 | 5 | 0 | 7.3 | +| 3r. OpenAdapt program with self-repair (engine of 2026-10-19) | 62 | 0 | 28 | 0 | 5 | 0.48 | 7.4 | +| 4. Computer-use agent, screenshots only | 90 | 5 | 0 | 5 | 5 | 3.90 | 22.5 | +| 4 with OpenAdapt read-back | 90 | 0 | 0 | 0 | 5 | 3.90 | 22.5 + 1.2 | + +The silent-wrong 95% CI for 0 of 100 is 0 to 3.7%. Seconds come from a +separate 15-item control-block pass at normal load +(`/2026-10-24-timing/`). The full runs ran at host loads of 77 to 374 +and their times are not used. Rows 3r and 4 are committed runs of +2026-10-19. + +Where each arm spent people time, per UI change, in minutes over a steady +item (+fix: a person fixed the program, script or prompt because the arm had +not recovered by the end of the block): + +| Change | Agent with tools | Agent script | OpenAdapt program | Screenshot agent | +|---|---|---|---|---| +| What's New screen | 0 | 10 (one fix reviewed) | 52 +fix (re-teach) | 0 | +| New Encounter moved and renamed | 0 | 10 (one fix reviewed) | 49 +fix (re-teach) | 0 | +| Look-alike referral above the target | 0 | 10 (one fix reviewed) | 0 | 0 | +| Target referral gone (right answer: stop) | 10 | 40 (3 fix attempts reviewed; it kept stopping) | 10 | 10 | +| The other six changes | 0 | 0 | 0 | 0 | + +The OpenAdapt program stopped on every item under two changes: the What's +New screen and the moved New Encounter control. Its self-repair did not +propose an admissible fix for either. + +### Totals for MockMed + +| Arm | 10 items | 100 items | 1,000 items | People minutes for 1,000 | +|---|---|---|---|---| +| 1. Agent with tools | $100 | $104 | $135 | 102 | +| 2. Agent-written script | $51 | $56 | $111 | 96 | +| 3. OpenAdapt program | $61 | $65 | $104 | 134 | +| 3r. with self-repair (2026-10-19) | $61 | $66 | $110 | 148 | +| 4. Screenshot agent | $51 | $56 | $110 | 69, plus 10 silent wrongs | +| 4 with OpenAdapt read-back | $51 | $56 | $111 | 72 | + +On MockMed the totals are dominated by setup (an engineer's or operator's +hour) and by assumed server faults (10 per 1,000 items). Setup and faults +cost about the same in every arm. The screenshot agent's 10 silent wrongs in +1,000 items are charged only the lower-bound cleanup: a reconcile plus a +re-entry each. Their harm is not priced. + +## Where OpenAdapt loses and where it wins + +OpenAdapt loses on: + +- Verified and automated items, on both apps: 41 vs 49 of 55 on OpenEMR, + and 71 vs 90 of 90 on MockMed. +- False stops: 9 vs 5 on OpenEMR, and 19 vs 0 on MockMed. +- Seconds an item on OpenEMR: 119 vs 24. +- Operator minutes and total cost from 100 items on, on OpenEMR. +- People time per UI change on MockMed, under the two changes it could not + repair. + +OpenAdapt wins on: + +- Model cost an item: $0 vs $0.013 to $0.05 for the agent arms. At 1,000 + items, model cost is $13 to $50 and so a small share of every total. +- Silent wrong saves compared with an agent that trusts the screen: 0 vs 5 + of 5 server faults. This is not a win over agents that read the record + back: they also had 0. +- Stopping when no correct save exists: 5 of 5, the same as every + read-back arm. + +## Assumptions + +All in [`assumptions.json`](assumptions.json): + +- **Labor:** operator $30 an hour and engineer $100 an hour. +- **Setup people time:** + - OpenAdapt: 22 operator minutes (teach, review, admit), plus 30 engineer + minutes to install it and write the record check. + - Agent with tools: 60 engineer minutes. + - Agent script: 30 engineer minutes. + - Screenshot agent: 30 engineer minutes. +- **Per-touch operator time (cost model V1):** + - stop: 90 s; + - reconcile: 120 s; + - repair review: 120 s; + - manual entry: 180 s on OpenEMR and 60 s on MockMed; + - interrupt: 30 s per request. +- **Maintenance people time:** + - an engineer reviews each agent-made script fix: 10 min; + - an engineer fixes a script by hand: 30 min; + - an operator re-teaches OpenAdapt after a change it could not handle: + 22 min; + - an engineer adjusts an agent's prompt: 15 min. +- **Volume:** 4 UI changes and 10 server-side bad saves per 1,000 items. + +No operator timing here is from a real person. Change any number in +`assumptions.json` and run `python -m economic_comparison.totals`. + +## Limits + +- One run per arm, one task per app, a scripted operator and synthetic + data. Small samples: read the confidence intervals. +- The OpenEMR runs for arm 3 and arms 1 and 2 used the same seeded stream on + two copies of the same pinned fixture, a few hours apart. +- Arm 3 on OpenEMR runs the engine of the `measure/production-lifecycle` + branch (duplicate-list check, relabel repair). Current main is the + "3 on main" row. +- The agent-written script on OpenEMR hung after 16 items. The hand-written + script and the screenshot agent were not run on OpenEMR. A stronger + authoring model or more engineer time might give a better script. That + was not measured. +- The MockMed OpenAdapt program is not the Regulated configuration. It has + no Regulated MockMed admission. +- The MockMed program's first pass broke on items 16-18 and 83-100. Item 83 + hit the 300 s watchdog under host load 374 and left the browser driver + unusable for the items after it; items 16-18 hit a busy durable authority. + Those 21 items were rerun in a fresh process. The first pass is kept as + `items.first-pass.csv`. +- Model spend for this comparison: $6.05 of the $30 cap, ledger + [`../lifecycle/results/spend.json`](../lifecycle/results/spend.json). + +## Raw results + +| What | Where | +|---|---| +| OpenEMR, arms 1 and 2 | [`../lifecycle/results/economics-2026-10-24/`](../lifecycle/results/economics-2026-10-24/) (`/ledger.jsonl`, `workload.jsonl`, `plan.json`) | +| OpenEMR, arm 3 | [`../lifecycle/results/production-2026-10-24/`](../lifecycle/results/production-2026-10-24/) (`after/`, `before/`) | +| MockMed, new runs | `../silent_saves/results/1.1.0/{agent-tools,agent-script,openadapt-program}/2026-10-24*/` (the agent's scripts are in `agent-script/2026-10-24/scripts/`, and the agent's tool traces in `agent-tools/2026-10-24/traces/`) | +| MockMed, earlier runs | `../silent_saves/results/1.1.0/{claude-computer-use+readback,self-repair}/2026-10-19/` | +| Totals | [`results/totals.json`](results/totals.json), [`results/TABLE.md`](results/TABLE.md) | + +## Reproduce + +```bash +cd packages/flow/benchmark +python -m economic_comparison.totals # recompute every table from the raw rows + +# MockMed (no fixture) +python -m silent_saves --fault-set 1.1.0 --agent python:silent_saves.agent_arms:AgentTools --out +python -m silent_saves --fault-set 1.1.0 --agent python:silent_saves.agent_arms:AgentScript --out + +# OpenEMR (needs the fixture lease; see ~/oa/scratch/fixture-lease/README.md) +python -m lifecycle.apps up +python -m lifecycle.arms.compare --out --seed 7 --items 55 --days 10 \ + --drift relabel_save@26 --arms-json economic_comparison/openemr_arms.json +``` diff --git a/packages/flow/benchmark/economic_comparison/__init__.py b/packages/flow/benchmark/economic_comparison/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/packages/flow/benchmark/economic_comparison/assumptions.json b/packages/flow/benchmark/economic_comparison/assumptions.json new file mode 100644 index 000000000..ac5f3f9a3 --- /dev/null +++ b/packages/flow/benchmark/economic_comparison/assumptions.json @@ -0,0 +1,100 @@ +{ + "version": "economic-comparison/2026-10-24.1", + "note": "Every number here is an assumption, not a measurement. totals.py takes measured rates from the raw results and these assumptions, and shows the totals at half and double the people-time assumptions. Change a number here, never in code.", + "labor_usd_per_hour": { + "operator": 30.0, + "engineer": 100.0, + "source": "lifecycle/arms/prices.json (labor): round figures for a fully loaded front-desk clerk hour and automation engineer hour" + }, + "operator_s": { + "source": "lifecycle/cost_model.py V1 (lifecycle-cost/2026-10-20.1); no real operator timings exist yet", + "stop": 90.0, + "reconcile": 120.0, + "repair_review": 120.0, + "manual_entry_mockmed": 60.0, + "manual_entry_openemr": 180.0, + "interrupt": 30.0, + "manual_entry_mockmed_why": "a MockMed triage note is one click path and one note field, about a third of the 12-field OpenEMR registration the cost model's 180 s is for" + }, + "per_outcome": { + "note": "What a person does after each outcome. One interrupt per item that reaches a person.", + "correct": [], + "claimed_failure_save_possible": [ + "stop", + "manual_entry" + ], + "claimed_failure_no_save_possible": [ + "stop" + ], + "flagged_wrong": [ + "stop", + "reconcile", + "manual_entry" + ], + "false_alarm": [ + "stop", + "reconcile" + ], + "silent_wrong": [ + "reconcile", + "manual_entry", + "interrupt" + ], + "silent_wrong_why": "lower bound: someone later finds the wrong record, fixes it and enters the item again (two interruptions: when found and when fixed). Clinical or business harm is not priced." + }, + "setup_people_min": { + "note": "One-time people time per workflow, by role. The agent's own authoring time and model cost are measured and reported separately.", + "openadapt": { + "operator": 22.0, + "engineer": 30.0, + "why": "cost model V1: teach preparation 5 min + a 2 min demonstration + compile review 10 min + admission 5 min; plus an engineer's 30 min to install it and write the record check (verify.yaml), the same read-back work the agent arms' engineer does" + }, + "openadapt_regulated_openemr": { + "operator": 22.0, + "engineer": 30.0, + "why": "as above, plus an engineer's 30 min for the Regulated deployment (keys, sealed bundle, durable store, problems-only switch); the qualification campaign runs unattended" + }, + "agent_tools": { + "operator": 0.0, + "engineer": 60.0, + "why": "an engineer wires a browser-tools agent (an off-the-shelf framework gives read_screen/click/fill), writes the record read-back tool and the check rule into the prompt, and tries it" + }, + "agent_script": { + "operator": 0.0, + "engineer": 30.0, + "why": "an engineer prompts the coding agent with the demonstration and the API description, reads and tests the script it submits" + }, + "cua_alone": { + "operator": 0.0, + "engineer": 30.0, + "why": "an engineer writes the task prompt and deploys the computer-use loop" + }, + "human_script": { + "operator": 0.0, + "engineer": 120.0, + "why": "lifecycle/arms/prices.json engineer_script_author_s (2 h)" + } + }, + "maintenance_people_min": { + "note": "People time per maintenance event, by kind.", + "script_fix_review_engineer": 10.0, + "script_fix_review_why": "an engineer reads the coding agent's diff and its dry run, and deploys it", + "script_fix_by_hand_engineer": 30.0, + "script_fix_by_hand_why": "lifecycle/arms/prices.json engineer_script_fix_s (30 min), when the coding agent's own fix did not hold", + "openadapt_reteach_operator": 22.0, + "openadapt_reteach_why": "a change OpenAdapt neither handled nor self-repaired: the person teaches the workflow again (same as setup)", + "agent_prompt_fix_engineer": 15.0, + "agent_prompt_fix_why": "a change an agent kept failing on: an engineer adjusts the prompt" + }, + "detection": { + "items_until_a_silent_wrong_is_noticed": null, + "note": "Silent wrongs are charged one cleanup each; how long they go unnoticed is not modelled." + }, + "volume": { + "ui_changes_per_1000_items": 4, + "why": "lifecycle/arms/prices.json drift rate: about one vendor release every few weeks at a clinic's referral volume", + "server_faults_per_1000_items": 10, + "server_faults_why": "assumed: one save in 100 goes wrong on the server side. The MockMed suite has 5 in 100 to measure detection; the excess cost per fault item is measured there and applied at this rate." + }, + "compute_usd_per_hour": 0.2 +} diff --git a/packages/flow/benchmark/economic_comparison/openemr_arms.json b/packages/flow/benchmark/economic_comparison/openemr_arms.json new file mode 100644 index 000000000..665213d39 --- /dev/null +++ b/packages/flow/benchmark/economic_comparison/openemr_arms.json @@ -0,0 +1,6 @@ +[ + ["agent_tools", "lifecycle.arms.agent_tools:factory", {"model": "claude-sonnet-5-5", "effort": "medium", "item_budget_usd": 0.50}], + ["agent_script", "lifecycle.arms.agent_script:factory", {"model": "claude-sonnet-5-5", "effort": "high", "author_budget_usd": 3.0, "fix_budget_usd": 1.5}], + ["human_script", "lifecycle.arms.human_script:factory", {}], + ["cua_alone", "lifecycle.arms.cua:factory", {"model": "claude-sonnet-5-5", "effort": "medium", "item_budget_usd": 0.80, "stop_after": 12}] +] diff --git a/packages/flow/benchmark/economic_comparison/results/TABLE.md b/packages/flow/benchmark/economic_comparison/results/TABLE.md new file mode 100644 index 000000000..a27f8cdfa --- /dev/null +++ b/packages/flow/benchmark/economic_comparison/results/TABLE.md @@ -0,0 +1,56 @@ +## OpenEMR referral intake (seed 7, 55 items, one app change: the save button relabeled at item 26) + +| Arm | Items | Verified in the end | Automated (no hand entry) | Silent wrong | False stops | Median s/item | Model $/item | Operator min/item | Maintenance for the change (people min / model $) | Setup people min | Setup agent min / $ | +|---|---|---|---|---|---|---|---|---|---|---|---| +| 3. OpenAdapt, Regulated + problems_only, durable, pipelined read-back, relabel repair (engine of measure/production-lifecycle; the API step is refused on this form, so the GUI save runs) | 55 | 53 | 41 | 0 | 9 of 35 | 118.5 | 0.000 | 2.15 | 7.5 / 0.00 | 52 | 13.3 / 0.00 | +| 3 on main. Same configuration, current main's engine (the run was stopped at 27 items) | 27 | 27 | 4 | 0 | 19 of 20 | 226.9 | 0.000 | 4.56 | 0.0 / 0.00 | 52 | 13.4 / 0.00 | +| 1. Agent with structured tools and REST read-back (Claude Sonnet 5.5) | 55 | 55 | 49 | 0 | 5 of 35 | 24.3 | 0.050 | 0.91 | 0.0 / 0.00 | 60 | 0.0 / 0.00 | +| 2. (hung after 16 items) Agent-written Playwright + REST script with read-back and duplicate check, agent-maintained (Claude Sonnet 5.5) | 16 | 16 | 7 | 0 | 7 of 12 | 5.3 | 0.000 | 2.44 | not reached | 70 | 2.6 / 0.86 | +| Reference: a person enters every item (cost model V1) | None | None | 0 | 0 | None of None | 210.0 | 0.000 | 3.50 | 0.0 / 0.00 | 0 | 0.0 / 0.00 | + +### Totals (people time priced at the assumed rates; x0.5 and x2 in brackets) + +| Arm | 10 items: total $ | people min | 100 items: total $ | people min | 1000 items: total $ | people min | +|---|---|---|---|---|---|---| +| openadapt-production-relabel | 72 [36-144] | 74 | 171 [86-340] | 270 | 1,158 [584-2,307] | 2,228 | +| openadapt-production | 84 [42-168] | 98 | 290 [146-579] | 508 | 2,351 [1,181-4,689] | 4,608 | +| agent_tools | 105 [53-210] | 69 | 151 [78-296] | 151 | 606 [329-1,161] | 969 | +| agent_script | 130 [65-259] | 94 | 239 [120-478] | 314 | 1,337 [669-2,672] | 2,508 | +| all_manual | 18 [9-35] | 35 | 175 [88-350] | 350 | 1,750 [875-3,500] | 3,500 | + +## MockMed, fault set 1.1.0 (100 items, 10 UI changes, 5 server faults) + +| Arm | Verified (of 90) | Silent wrong | False stops | Faults claimed saved (of 5) | Stopped when no save possible (of 5) | Control s/item (load) | Model $/item | Setup people min | Setup agent min / $ | +|---|---|---|---|---|---|---|---|---|---| +| 4. Computer-use agent, screenshots only (Claude Sonnet 5.5) | 90 | 5 | 0 | 5 | 5 | 22.5 (17) | 0.0386 | 30 | 0.0 / 0.000 | +| 1. Agent with structured tools and record read-back (Claude Sonnet 5.5) | 90 | 0 | 0 | 0 | 5 | 13.7 (15) | 0.0132 | 60 | 0.0 / 0.000 | +| 2. Agent-written Playwright script with read-back and duplicate guard, agent-maintained | 90 | 0 | 0 | 0 | 5 | 0.8 (6) | 0.0000 | 30 | 0.4 / 0.026 | +| 3. OpenAdapt admitted program, read-back on, no model calls | 71 | 0 | 19 | 0 | 5 | 7.3 (8) | 0.0000 | 52 | 0.0 / 0.000 | +| 3r. OpenAdapt program with self-repair (engine of 2026-10-19) | 62 | 0 | 28 | 0 | 5 | 7.4 (10) | 0.0000 | 52 | 0.0 / 0.000 | +| 4 + OpenAdapt read-back: the same agent runs, each claim gated on verify-run | 90 | 0 | 0 | 0 | 5 | 22.5 (17) | 0.0386 | 30 | 0.0 / 0.000 | + +### Each UI change: excess people minutes (silent wrongs in the block) + +| Change | cua_alone | agent_tools | agent_script | openadapt_program | openadapt_self_repair | cua_plus_openadapt_readback | +|---|---|---|---|---|---|---| +| renamed controls | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | +| display scaling | 0.0 | 0.0 | 0.0 | 0.0 | 49.0 +fix | 0.0 | +| moved controls | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | +| new interstitial | 0.0 | 0.0 | 10.0 | 52.0 +fix | 52.0 +fix | 0.0 | +| latency | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | +| reordered queue | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | +| renamed and moved New Encounter | 0.0 | 0.0 | 10.0 | 49.0 +fix | 49.0 +fix | 0.0 | +| look-alike referral | 0.0 | 0.0 | 10.0 | 0.0 | 0.0 | 0.0 | +| relabeled and reordered type | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | +| target referral gone | 10.0 | 10.0 | 40.0 | 10.0 | 10.0 | 10.0 | + +### Totals (people time priced at the assumed rates; x0.5 and x2 in brackets) + +| Arm | 10 items: total $ | people min | silent wrongs | 100 items: total $ | people min | silent wrongs | 1000 items: total $ | people min | silent wrongs | +|---|---|---|---|---|---|---|---|---|---| +| cua_alone | 51 [26-101] | 30 | 0.1 | 56 [30-108] | 34 | 1.0 | 110 [75-179] | 69 | 10.0 | +| agent_tools | 100 [50-201] | 60 | 0.0 | 104 [52-206] | 64 | 0.0 | 135 [74-256] | 102 | 0.0 | +| agent_script | 51 [25-101] | 31 | 0.0 | 56 [28-112] | 37 | 0.0 | 111 [56-222] | 96 | 0.0 | +| openadapt_program | 61 [31-123] | 53 | 0.0 | 65 [33-130] | 60 | 0.0 | 104 [53-206] | 134 | 0.0 | +| openadapt_self_repair | 61 [31-123] | 53 | 0.0 | 66 [33-132] | 62 | 0.0 | 110 [55-219] | 148 | 0.0 | +| cua_plus_openadapt_readback | 51 [26-101] | 30 | 0.0 | 56 [30-108] | 34 | 0.0 | 111 [76-182] | 72 | 0.0 | diff --git a/packages/flow/benchmark/economic_comparison/results/totals.json b/packages/flow/benchmark/economic_comparison/results/totals.json new file mode 100644 index 000000000..bbf5685a8 --- /dev/null +++ b/packages/flow/benchmark/economic_comparison/results/totals.json @@ -0,0 +1,2863 @@ +{ + "assumptions": "economic-comparison/2026-10-24.1", + "openemr": [ + { + "arm": "openadapt-production-relabel", + "label": "3. OpenAdapt, Regulated + problems_only, durable, pipelined read-back, relabel repair (engine of measure/production-lifecycle; the API step is refused on this form, so the GUI save runs)", + "kind": "program", + "source": "lifecycle/results/production-2026-10-24/after", + "partial": false, + "items": 55, + "verified": 53, + "automated": 41, + "by_hand": 14, + "outcomes": { + "correct": 30, + "false_stop": 9, + "flagged_wrong": 2, + "right_stop": 10, + "stopped_saved": 1, + "data_stop": 2, + "fault_stop": 1 + }, + "silent_wrong": 0, + "false_stop": 9, + "false_stop_of": 35, + "drift_events": 1, + "median_s": 118.53, + "p90_s": 236.89, + "model_usd_total": 0.0, + "setup": { + "model_usd": 0.0, + "agent_min": 13.3, + "operator_min": 22.0, + "engineer_min": 30.0 + }, + "per_item": { + "model_usd": 0.0, + "compute_usd": 0.009216363636363637, + "operator_min": 2.1454545454545455 + }, + "per_change": { + "model_usd": 0.0, + "operator_min": 7.5, + "engineer_min": 0.0, + "fixes": 0 + }, + "per_item_cleanup_min": 0.0, + "totals": { + "10": { + "setup_usd": 61.0, + "per_item_usd": 10.82, + "ui_changes_usd": 0.15, + "silent_wrong_cleanup_usd": 0.0, + "total_usd": 71.97, + "people_min": 73.8, + "model_usd": 0.0, + "silent_wrong_expected": 0.0 + }, + "100": { + "setup_usd": 61.0, + "per_item_usd": 108.19, + "ui_changes_usd": 1.5, + "silent_wrong_cleanup_usd": 0.0, + "total_usd": 170.69, + "people_min": 269.5, + "model_usd": 0.0, + "silent_wrong_expected": 0.0 + }, + "1000": { + "setup_usd": 61.0, + "per_item_usd": 1081.94, + "ui_changes_usd": 15.0, + "silent_wrong_cleanup_usd": 0.0, + "total_usd": 1157.94, + "people_min": 2227.5, + "model_usd": 0.0, + "silent_wrong_expected": 0.0 + } + }, + "totals_people_x0.5": { + "10": { + "setup_usd": 30.5, + "per_item_usd": 5.46, + "ui_changes_usd": 0.07, + "silent_wrong_cleanup_usd": 0.0, + "total_usd": 36.03, + "people_min": 36.9, + "model_usd": 0.0, + "silent_wrong_expected": 0.0 + }, + "100": { + "setup_usd": 30.5, + "per_item_usd": 54.56, + "ui_changes_usd": 0.75, + "silent_wrong_cleanup_usd": 0.0, + "total_usd": 85.81, + "people_min": 134.8, + "model_usd": 0.0, + "silent_wrong_expected": 0.0 + }, + "1000": { + "setup_usd": 30.5, + "per_item_usd": 545.58, + "ui_changes_usd": 7.5, + "silent_wrong_cleanup_usd": 0.0, + "total_usd": 583.58, + "people_min": 1113.7, + "model_usd": 0.0, + "silent_wrong_expected": 0.0 + } + }, + "totals_people_x2": { + "10": { + "setup_usd": 122.0, + "per_item_usd": 21.55, + "ui_changes_usd": 0.3, + "silent_wrong_cleanup_usd": 0.0, + "total_usd": 143.85, + "people_min": 147.5, + "model_usd": 0.0, + "silent_wrong_expected": 0.0 + }, + "100": { + "setup_usd": 122.0, + "per_item_usd": 215.47, + "ui_changes_usd": 3.0, + "silent_wrong_cleanup_usd": 0.0, + "total_usd": 340.47, + "people_min": 539.1, + "model_usd": 0.0, + "silent_wrong_expected": 0.0 + }, + "1000": { + "setup_usd": 122.0, + "per_item_usd": 2154.67, + "ui_changes_usd": 30.0, + "silent_wrong_cleanup_usd": 0.0, + "total_usd": 2306.67, + "people_min": 4454.9, + "model_usd": 0.0, + "silent_wrong_expected": 0.0 + } + } + }, + { + "arm": "openadapt-production", + "label": "3 on main. Same configuration, current main's engine (the run was stopped at 27 items)", + "kind": "program", + "source": "lifecycle/results/production-2026-10-24/before", + "partial": true, + "items": 27, + "verified": 27, + "automated": 4, + "by_hand": 23, + "outcomes": { + "false_stop": 19, + "data_stop": 3, + "right_stop": 4, + "fault_stop": 1 + }, + "silent_wrong": 0, + "false_stop": 19, + "false_stop_of": 20, + "drift_events": 1, + "median_s": 226.86, + "p90_s": 226.94, + "model_usd_total": 0.0, + "setup": { + "model_usd": 0.0, + "agent_min": 13.4, + "operator_min": 22.0, + "engineer_min": 30.0 + }, + "per_item": { + "model_usd": 0.0, + "compute_usd": 0.011818518518518518, + "operator_min": 4.555555555555555 + }, + "per_change": { + "model_usd": 0.0, + "operator_min": 0.0, + "engineer_min": 0.0, + "fixes": 0 + }, + "per_item_cleanup_min": 0.0, + "totals": { + "10": { + "setup_usd": 61.0, + "per_item_usd": 22.9, + "ui_changes_usd": 0.0, + "silent_wrong_cleanup_usd": 0.0, + "total_usd": 83.9, + "people_min": 97.6, + "model_usd": 0.0, + "silent_wrong_expected": 0.0 + }, + "100": { + "setup_usd": 61.0, + "per_item_usd": 228.96, + "ui_changes_usd": 0.0, + "silent_wrong_cleanup_usd": 0.0, + "total_usd": 289.96, + "people_min": 507.6, + "model_usd": 0.0, + "silent_wrong_expected": 0.0 + }, + "1000": { + "setup_usd": 61.0, + "per_item_usd": 2289.6, + "ui_changes_usd": 0.0, + "silent_wrong_cleanup_usd": 0.0, + "total_usd": 2350.6, + "people_min": 4607.6, + "model_usd": 0.0, + "silent_wrong_expected": 0.0 + } + }, + "totals_people_x0.5": { + "10": { + "setup_usd": 30.5, + "per_item_usd": 11.51, + "ui_changes_usd": 0.0, + "silent_wrong_cleanup_usd": 0.0, + "total_usd": 42.01, + "people_min": 48.8, + "model_usd": 0.0, + "silent_wrong_expected": 0.0 + }, + "100": { + "setup_usd": 30.5, + "per_item_usd": 115.07, + "ui_changes_usd": 0.0, + "silent_wrong_cleanup_usd": 0.0, + "total_usd": 145.57, + "people_min": 253.8, + "model_usd": 0.0, + "silent_wrong_expected": 0.0 + }, + "1000": { + "setup_usd": 30.5, + "per_item_usd": 1150.71, + "ui_changes_usd": 0.0, + "silent_wrong_cleanup_usd": 0.0, + "total_usd": 1181.21, + "people_min": 2303.8, + "model_usd": 0.0, + "silent_wrong_expected": 0.0 + } + }, + "totals_people_x2": { + "10": { + "setup_usd": 122.0, + "per_item_usd": 45.67, + "ui_changes_usd": 0.0, + "silent_wrong_cleanup_usd": 0.0, + "total_usd": 167.67, + "people_min": 195.1, + "model_usd": 0.0, + "silent_wrong_expected": 0.0 + }, + "100": { + "setup_usd": 122.0, + "per_item_usd": 456.74, + "ui_changes_usd": 0.0, + "silent_wrong_cleanup_usd": 0.0, + "total_usd": 578.74, + "people_min": 1015.1, + "model_usd": 0.0, + "silent_wrong_expected": 0.0 + }, + "1000": { + "setup_usd": 122.0, + "per_item_usd": 4567.37, + "ui_changes_usd": 0.0, + "silent_wrong_cleanup_usd": 0.0, + "total_usd": 4689.37, + "people_min": 9215.1, + "model_usd": 0.0, + "silent_wrong_expected": 0.0 + } + } + }, + { + "arm": "agent_tools", + "label": "1. Agent with structured tools and REST read-back (Claude Sonnet 5.5)", + "kind": "agent", + "source": "lifecycle/results/economics-2026-10-24/agent_tools", + "partial": false, + "items": 55, + "verified": 55, + "automated": 49, + "by_hand": 6, + "outcomes": { + "correct": 39, + "right_stop": 10, + "false_stop": 5, + "data_stop": 1 + }, + "silent_wrong": 0, + "false_stop": 5, + "false_stop_of": 35, + "drift_events": 1, + "median_s": 24.27, + "p90_s": 27.8, + "model_usd_total": 2.7678, + "setup": { + "model_usd": 0.0, + "agent_min": 0.0, + "operator_min": 0.0, + "engineer_min": 60.0 + }, + "per_item": { + "model_usd": 0.05032363636363636, + "compute_usd": 0.0015454545454545456, + "operator_min": 0.9090909090909091 + }, + "per_change": { + "model_usd": 0.0, + "operator_min": 0.0, + "engineer_min": 0.0, + "fixes": 0 + }, + "per_item_cleanup_min": 0.0, + "totals": { + "10": { + "setup_usd": 100.0, + "per_item_usd": 5.06, + "ui_changes_usd": 0.0, + "silent_wrong_cleanup_usd": 0.0, + "total_usd": 105.06, + "people_min": 69.1, + "model_usd": 0.5, + "silent_wrong_expected": 0.0 + }, + "100": { + "setup_usd": 100.0, + "per_item_usd": 50.64, + "ui_changes_usd": 0.0, + "silent_wrong_cleanup_usd": 0.0, + "total_usd": 150.64, + "people_min": 150.9, + "model_usd": 5.03, + "silent_wrong_expected": 0.0 + }, + "1000": { + "setup_usd": 100.0, + "per_item_usd": 506.41, + "ui_changes_usd": 0.0, + "silent_wrong_cleanup_usd": 0.0, + "total_usd": 606.41, + "people_min": 969.1, + "model_usd": 50.32, + "silent_wrong_expected": 0.0 + } + }, + "totals_people_x0.5": { + "10": { + "setup_usd": 50.0, + "per_item_usd": 2.79, + "ui_changes_usd": 0.0, + "silent_wrong_cleanup_usd": 0.0, + "total_usd": 52.79, + "people_min": 34.5, + "model_usd": 0.5, + "silent_wrong_expected": 0.0 + }, + "100": { + "setup_usd": 50.0, + "per_item_usd": 27.91, + "ui_changes_usd": 0.0, + "silent_wrong_cleanup_usd": 0.0, + "total_usd": 77.91, + "people_min": 75.5, + "model_usd": 5.03, + "silent_wrong_expected": 0.0 + }, + "1000": { + "setup_usd": 50.0, + "per_item_usd": 279.14, + "ui_changes_usd": 0.0, + "silent_wrong_cleanup_usd": 0.0, + "total_usd": 329.14, + "people_min": 484.5, + "model_usd": 50.32, + "silent_wrong_expected": 0.0 + } + }, + "totals_people_x2": { + "10": { + "setup_usd": 200.0, + "per_item_usd": 9.61, + "ui_changes_usd": 0.0, + "silent_wrong_cleanup_usd": 0.0, + "total_usd": 209.61, + "people_min": 138.2, + "model_usd": 0.5, + "silent_wrong_expected": 0.0 + }, + "100": { + "setup_usd": 200.0, + "per_item_usd": 96.1, + "ui_changes_usd": 0.0, + "silent_wrong_cleanup_usd": 0.0, + "total_usd": 296.1, + "people_min": 301.8, + "model_usd": 5.03, + "silent_wrong_expected": 0.0 + }, + "1000": { + "setup_usd": 200.0, + "per_item_usd": 960.96, + "ui_changes_usd": 0.0, + "silent_wrong_cleanup_usd": 0.0, + "total_usd": 1160.96, + "people_min": 1938.2, + "model_usd": 50.32, + "silent_wrong_expected": 0.0 + } + } + }, + { + "arm": "agent_script", + "label": "2. (hung after 16 items) Agent-written Playwright + REST script with read-back and duplicate check, agent-maintained (Claude Sonnet 5.5)", + "kind": "script", + "source": "lifecycle/results/economics-2026-10-24/agent_script", + "partial": true, + "items": 16, + "verified": 16, + "automated": 7, + "by_hand": 9, + "outcomes": { + "false_stop": 7, + "correct": 5, + "data_stop": 1, + "right_stop": 2, + "fault_stop": 1 + }, + "silent_wrong": 0, + "false_stop": 7, + "false_stop_of": 12, + "drift_events": 0, + "median_s": 5.32, + "p90_s": 6.56, + "model_usd_total": 0.8569, + "setup": { + "model_usd": 0.8569, + "agent_min": 2.6, + "operator_min": 0.0, + "engineer_min": 70.0, + "debug_fixes": 4 + }, + "per_item": { + "model_usd": 0.0, + "compute_usd": 0.000525, + "operator_min": 2.4375 + }, + "per_change": { + "model_usd": 0.0, + "operator_min": 0.0, + "engineer_min": 0.0, + "fixes": 0, + "not_measured": true + }, + "per_item_cleanup_min": 0.0, + "totals": { + "10": { + "setup_usd": 117.52, + "per_item_usd": 12.19, + "ui_changes_usd": 0.0, + "silent_wrong_cleanup_usd": 0.0, + "total_usd": 129.72, + "people_min": 94.4, + "model_usd": 0.86, + "silent_wrong_expected": 0.0 + }, + "100": { + "setup_usd": 117.52, + "per_item_usd": 121.93, + "ui_changes_usd": 0.0, + "silent_wrong_cleanup_usd": 0.0, + "total_usd": 239.45, + "people_min": 313.8, + "model_usd": 0.86, + "silent_wrong_expected": 0.0 + }, + "1000": { + "setup_usd": 117.52, + "per_item_usd": 1219.28, + "ui_changes_usd": 0.0, + "silent_wrong_cleanup_usd": 0.0, + "total_usd": 1336.8, + "people_min": 2507.5, + "model_usd": 0.86, + "silent_wrong_expected": 0.0 + } + }, + "totals_people_x0.5": { + "10": { + "setup_usd": 59.19, + "per_item_usd": 6.1, + "ui_changes_usd": 0.0, + "silent_wrong_cleanup_usd": 0.0, + "total_usd": 65.29, + "people_min": 47.2, + "model_usd": 0.86, + "silent_wrong_expected": 0.0 + }, + "100": { + "setup_usd": 59.19, + "per_item_usd": 60.99, + "ui_changes_usd": 0.0, + "silent_wrong_cleanup_usd": 0.0, + "total_usd": 120.18, + "people_min": 156.9, + "model_usd": 0.86, + "silent_wrong_expected": 0.0 + }, + "1000": { + "setup_usd": 59.19, + "per_item_usd": 609.9, + "ui_changes_usd": 0.0, + "silent_wrong_cleanup_usd": 0.0, + "total_usd": 669.09, + "people_min": 1253.8, + "model_usd": 0.86, + "silent_wrong_expected": 0.0 + } + }, + "totals_people_x2": { + "10": { + "setup_usd": 234.19, + "per_item_usd": 24.38, + "ui_changes_usd": 0.0, + "silent_wrong_cleanup_usd": 0.0, + "total_usd": 258.57, + "people_min": 188.8, + "model_usd": 0.86, + "silent_wrong_expected": 0.0 + }, + "100": { + "setup_usd": 234.19, + "per_item_usd": 243.8, + "ui_changes_usd": 0.0, + "silent_wrong_cleanup_usd": 0.0, + "total_usd": 477.99, + "people_min": 627.5, + "model_usd": 0.86, + "silent_wrong_expected": 0.0 + }, + "1000": { + "setup_usd": 234.19, + "per_item_usd": 2438.03, + "ui_changes_usd": 0.0, + "silent_wrong_cleanup_usd": 0.0, + "total_usd": 2672.22, + "people_min": 5015.0, + "model_usd": 0.86, + "silent_wrong_expected": 0.0 + } + } + }, + { + "arm": "all_manual", + "label": "Reference: a person enters every item (cost model V1)", + "kind": "manual", + "source": "cost model", + "partial": false, + "items": null, + "verified": null, + "automated": 0, + "by_hand": null, + "outcomes": {}, + "silent_wrong": 0, + "false_stop": null, + "false_stop_of": null, + "drift_events": 0, + "median_s": 210.0, + "p90_s": null, + "model_usd_total": 0.0, + "setup": { + "model_usd": 0.0, + "agent_min": 0.0, + "operator_min": 0.0, + "engineer_min": 0.0 + }, + "per_item": { + "model_usd": 0.0, + "compute_usd": 0.0, + "operator_min": 3.5 + }, + "per_change": { + "model_usd": 0.0, + "operator_min": 0.0, + "engineer_min": 0.0, + "fixes": 0 + }, + "per_item_cleanup_min": 0.0, + "totals": { + "10": { + "setup_usd": 0.0, + "per_item_usd": 17.5, + "ui_changes_usd": 0.0, + "silent_wrong_cleanup_usd": 0.0, + "total_usd": 17.5, + "people_min": 35.0, + "model_usd": 0.0, + "silent_wrong_expected": 0.0 + }, + "100": { + "setup_usd": 0.0, + "per_item_usd": 175.0, + "ui_changes_usd": 0.0, + "silent_wrong_cleanup_usd": 0.0, + "total_usd": 175.0, + "people_min": 350.0, + "model_usd": 0.0, + "silent_wrong_expected": 0.0 + }, + "1000": { + "setup_usd": 0.0, + "per_item_usd": 1750.0, + "ui_changes_usd": 0.0, + "silent_wrong_cleanup_usd": 0.0, + "total_usd": 1750.0, + "people_min": 3500.0, + "model_usd": 0.0, + "silent_wrong_expected": 0.0 + } + }, + "totals_people_x0.5": { + "10": { + "setup_usd": 0.0, + "per_item_usd": 8.75, + "ui_changes_usd": 0.0, + "silent_wrong_cleanup_usd": 0.0, + "total_usd": 8.75, + "people_min": 17.5, + "model_usd": 0.0, + "silent_wrong_expected": 0.0 + }, + "100": { + "setup_usd": 0.0, + "per_item_usd": 87.5, + "ui_changes_usd": 0.0, + "silent_wrong_cleanup_usd": 0.0, + "total_usd": 87.5, + "people_min": 175.0, + "model_usd": 0.0, + "silent_wrong_expected": 0.0 + }, + "1000": { + "setup_usd": 0.0, + "per_item_usd": 875.0, + "ui_changes_usd": 0.0, + "silent_wrong_cleanup_usd": 0.0, + "total_usd": 875.0, + "people_min": 1750.0, + "model_usd": 0.0, + "silent_wrong_expected": 0.0 + } + }, + "totals_people_x2": { + "10": { + "setup_usd": 0.0, + "per_item_usd": 35.0, + "ui_changes_usd": 0.0, + "silent_wrong_cleanup_usd": 0.0, + "total_usd": 35.0, + "people_min": 70.0, + "model_usd": 0.0, + "silent_wrong_expected": 0.0 + }, + "100": { + "setup_usd": 0.0, + "per_item_usd": 350.0, + "ui_changes_usd": 0.0, + "silent_wrong_cleanup_usd": 0.0, + "total_usd": 350.0, + "people_min": 700.0, + "model_usd": 0.0, + "silent_wrong_expected": 0.0 + }, + "1000": { + "setup_usd": 0.0, + "per_item_usd": 3500.0, + "ui_changes_usd": 0.0, + "silent_wrong_cleanup_usd": 0.0, + "total_usd": 3500.0, + "people_min": 7000.0, + "model_usd": 0.0, + "silent_wrong_expected": 0.0 + } + } + } + ], + "mockmed": [ + { + "arm": "cua_alone", + "label": "4. Computer-use agent, screenshots only (Claude Sonnet 5.5)", + "kind": "agent", + "source": "silent_saves/results/1.1.0/claude-computer-use+readback/2026-10-19", + "measured": { + "items": 100, + "verified_of_saveable": 90, + "saveable": 90, + "silent_wrong": 5, + "flagged_wrong": 0, + "false_stops": 0, + "faults_reported_as_success": 5, + "stopped_when_no_save_possible": 5, + "model_usd_total": 3.8962, + "model_calls_total": 511, + "control_median_s": 22.51, + "control_median_host_load": 17.0, + "control_model_usd_per_item": 0.03863 + }, + "setup": { + "model_usd": 0, + "agent_min": 0.0, + "operator_min": 0.0, + "engineer_min": 30.0, + "usd": 50.0 + }, + "steady": { + "model_usd": 0.03862666666666666, + "operator_s": 0.0, + "engineer_s": 0.0, + "compute_usd": 0.0014030222222222223 + }, + "steady_usd": 0.04002968888888889, + "faults": [ + { + "item": 20, + "fault": "no_persist", + "outcome": "silent_wrong", + "excess_usd": 1.7496672555555555, + "excess_machine_usd": -0.0003327444444444494, + "excess_people_usd": 1.75, + "excess_people_min": 3.5 + }, + { + "item": 30, + "fault": "wrong_record", + "outcome": "silent_wrong", + "excess_usd": 1.7494242, + "excess_machine_usd": -0.0005757999999999944, + "excess_people_usd": 1.75, + "excess_people_min": 3.5 + }, + { + "item": 40, + "fault": "partial", + "outcome": "silent_wrong", + "excess_usd": 1.7495784222222222, + "excess_machine_usd": -0.00042157777777777905, + "excess_people_usd": 1.75, + "excess_people_min": 3.5 + }, + { + "item": 60, + "fault": "duplicate", + "outcome": "silent_wrong", + "excess_usd": 1.7429637, + "excess_machine_usd": -0.007036299999999995, + "excess_people_usd": 1.75, + "excess_people_min": 3.5 + }, + { + "item": 80, + "fault": "optimistic", + "outcome": "silent_wrong", + "excess_usd": 1.7498304777777778, + "excess_machine_usd": -0.00016952222222221813, + "excess_people_usd": 1.75, + "excess_people_min": 3.5 + } + ], + "changes": [ + { + "change": "renamed controls", + "drift": "rename", + "items": 9, + "outcomes": { + "correct": 9 + }, + "silent_wrong": 0, + "recovered_by_block_end": true, + "people_fix_added": "", + "operator_min": 0.0, + "engineer_min": 0.0, + "model_usd": 0.3474, + "excess_people_min": 0.0, + "excess_usd": -0.002, + "excess_machine_usd": -0.0024891444444444244, + "excess_people_usd": 0.0 + }, + { + "change": "display scaling", + "drift": "zoom", + "items": 9, + "outcomes": { + "correct": 9 + }, + "silent_wrong": 0, + "recovered_by_block_end": true, + "people_fix_added": "", + "operator_min": 0.0, + "engineer_min": 0.0, + "model_usd": 0.322, + "excess_people_min": 0.0, + "excess_usd": -0.028, + "excess_machine_usd": -0.02787225555555556, + "excess_people_usd": 0.0 + }, + { + "change": "moved controls", + "drift": "move", + "items": 9, + "outcomes": { + "correct": 9 + }, + "silent_wrong": 0, + "recovered_by_block_end": true, + "people_fix_added": "", + "operator_min": 0.0, + "engineer_min": 0.0, + "model_usd": 0.347, + "excess_people_min": 0.0, + "excess_usd": -0.001, + "excess_machine_usd": -0.0013365888888888788, + "excess_people_usd": 0.0 + }, + { + "change": "new interstitial", + "drift": "notice", + "items": 10, + "outcomes": { + "correct": 10 + }, + "silent_wrong": 0, + "recovered_by_block_end": true, + "people_fix_added": "", + "operator_min": 0.0, + "engineer_min": 0.0, + "model_usd": 0.4552, + "excess_people_min": 0.0, + "excess_usd": 0.073, + "excess_machine_usd": 0.0731377222222222, + "excess_people_usd": 0.0 + }, + { + "change": "latency", + "drift": "slow", + "items": 9, + "outcomes": { + "correct": 9 + }, + "silent_wrong": 0, + "recovered_by_block_end": true, + "people_fix_added": "", + "operator_min": 0.0, + "engineer_min": 0.0, + "model_usd": 0.4469, + "excess_people_min": 0.0, + "excess_usd": 0.102, + "excess_machine_usd": 0.10218618888888886, + "excess_people_usd": 0.0 + }, + { + "change": "reordered queue", + "drift": "sort", + "items": 10, + "outcomes": { + "correct": 10 + }, + "silent_wrong": 0, + "recovered_by_block_end": true, + "people_fix_added": "", + "operator_min": 0.0, + "engineer_min": 0.0, + "model_usd": 0.386, + "excess_people_min": 0.0, + "excess_usd": -0.001, + "excess_machine_usd": -0.0010644444444444234, + "excess_people_usd": 0.0 + }, + { + "change": "renamed and moved New Encounter", + "drift": "relink", + "items": 9, + "outcomes": { + "correct": 9 + }, + "silent_wrong": 0, + "recovered_by_block_end": true, + "people_fix_added": "", + "operator_min": 0.0, + "engineer_min": 0.0, + "model_usd": 0.3477, + "excess_people_min": 0.0, + "excess_usd": -0.002, + "excess_machine_usd": -0.001974088888888892, + "excess_people_usd": 0.0 + }, + { + "change": "look-alike referral", + "drift": "lookalike", + "items": 5, + "outcomes": { + "correct": 5 + }, + "silent_wrong": 0, + "recovered_by_block_end": true, + "people_fix_added": "", + "operator_min": 0.0, + "engineer_min": 0.0, + "model_usd": 0.1931, + "excess_people_min": 0.0, + "excess_usd": -0.001, + "excess_machine_usd": -0.0009323333333333406, + "excess_people_usd": 0.0 + }, + { + "change": "relabeled and reordered type", + "drift": "typelabel", + "items": 5, + "outcomes": { + "correct": 5 + }, + "silent_wrong": 0, + "recovered_by_block_end": true, + "people_fix_added": "", + "operator_min": 0.0, + "engineer_min": 0.0, + "model_usd": 0.1929, + "excess_people_min": 0.0, + "excess_usd": -0.001, + "excess_machine_usd": -0.001077722222222216, + "excess_people_usd": 0.0 + }, + { + "change": "target referral gone", + "drift": "missing", + "items": 5, + "outcomes": { + "safe_stop": 5 + }, + "silent_wrong": 0, + "recovered_by_block_end": true, + "people_fix_added": "", + "operator_min": 10.0, + "engineer_min": 0.0, + "model_usd": 0.0933, + "excess_people_min": 10.0, + "excess_usd": 4.895, + "excess_machine_usd": -0.10457955555555554, + "excess_people_usd": 5.0 + } + ], + "totals": { + "10": { + "setup_usd": 50.0, + "per_item_usd": 0.4, + "faults_usd": 0.17, + "ui_changes_usd": 0.02, + "total_usd": 50.6, + "people_min": 30.4, + "model_usd": 0.39, + "silent_wrong_expected": 0.1 + }, + "100": { + "setup_usd": 50.0, + "per_item_usd": 4.0, + "faults_usd": 1.75, + "ui_changes_usd": 0.2, + "total_usd": 55.95, + "people_min": 33.9, + "model_usd": 3.86, + "silent_wrong_expected": 1.0 + }, + "1000": { + "setup_usd": 50.0, + "per_item_usd": 40.03, + "faults_usd": 17.48, + "ui_changes_usd": 2.01, + "total_usd": 109.53, + "people_min": 69.0, + "model_usd": 38.63, + "silent_wrong_expected": 10.0 + } + }, + "totals_people_x0.5": { + "10": { + "setup_usd": 25.0, + "per_item_usd": 0.4, + "faults_usd": 0.09, + "ui_changes_usd": 0.01, + "total_usd": 25.5, + "people_min": 15.2, + "model_usd": 0.39, + "silent_wrong_expected": 0.1 + }, + "100": { + "setup_usd": 25.0, + "per_item_usd": 4.0, + "faults_usd": 0.87, + "ui_changes_usd": 0.1, + "total_usd": 29.98, + "people_min": 16.9, + "model_usd": 3.86, + "silent_wrong_expected": 1.0 + }, + "1000": { + "setup_usd": 25.0, + "per_item_usd": 40.03, + "faults_usd": 8.73, + "ui_changes_usd": 1.01, + "total_usd": 74.78, + "people_min": 34.5, + "model_usd": 38.63, + "silent_wrong_expected": 10.0 + } + }, + "totals_people_x2": { + "10": { + "setup_usd": 100.0, + "per_item_usd": 0.4, + "faults_usd": 0.35, + "ui_changes_usd": 0.04, + "total_usd": 100.79, + "people_min": 60.8, + "model_usd": 0.39, + "silent_wrong_expected": 0.1 + }, + "100": { + "setup_usd": 100.0, + "per_item_usd": 4.0, + "faults_usd": 3.5, + "ui_changes_usd": 0.4, + "total_usd": 107.9, + "people_min": 67.8, + "model_usd": 3.86, + "silent_wrong_expected": 1.0 + }, + "1000": { + "setup_usd": 100.0, + "per_item_usd": 40.03, + "faults_usd": 34.98, + "ui_changes_usd": 4.01, + "total_usd": 179.03, + "people_min": 138.0, + "model_usd": 38.63, + "silent_wrong_expected": 10.0 + } + } + }, + { + "arm": "agent_tools", + "label": "1. Agent with structured tools and record read-back (Claude Sonnet 5.5)", + "kind": "agent", + "source": "silent_saves/results/1.1.0/agent-tools/2026-10-24", + "measured": { + "items": 100, + "verified_of_saveable": 90, + "saveable": 90, + "silent_wrong": 0, + "flagged_wrong": 3, + "false_stops": 0, + "faults_reported_as_success": 0, + "stopped_when_no_save_possible": 5, + "model_usd_total": 1.4546, + "model_calls_total": 756, + "control_median_s": 13.66, + "control_median_host_load": 15.4, + "control_model_usd_per_item": 0.01318 + }, + "setup": { + "model_usd": 0, + "agent_min": 0.0, + "operator_min": 0.0, + "engineer_min": 60.0, + "usd": 100.0 + }, + "steady": { + "model_usd": 0.0131802, + "operator_s": 0.0, + "engineer_s": 0.0, + "compute_usd": 0.000720762962962963 + }, + "steady_usd": 0.013900962962962962, + "faults": [ + { + "item": 20, + "fault": "no_persist", + "outcome": "safe_stop", + "excess_usd": 1.006023537037037, + "excess_machine_usd": 0.006023537037037039, + "excess_people_usd": 1.0, + "excess_people_min": 2.0 + }, + { + "item": 30, + "fault": "wrong_record", + "outcome": "flagged_wrong", + "excess_usd": 2.5019235370370367, + "excess_machine_usd": 0.0019235370370370396, + "excess_people_usd": 2.5, + "excess_people_min": 5.0 + }, + { + "item": 40, + "fault": "partial", + "outcome": "flagged_wrong", + "excess_usd": 2.501775481481481, + "excess_machine_usd": 0.001775481481481482, + "excess_people_usd": 2.5, + "excess_people_min": 5.0 + }, + { + "item": 60, + "fault": "duplicate", + "outcome": "flagged_wrong", + "excess_usd": 2.506804259259259, + "excess_machine_usd": 0.00680425925925926, + "excess_people_usd": 2.5, + "excess_people_min": 5.0 + }, + { + "item": 80, + "fault": "optimistic", + "outcome": "safe_stop", + "excess_usd": 1.0038038148148147, + "excess_machine_usd": 0.0038038148148148154, + "excess_people_usd": 1.0, + "excess_people_min": 2.0 + } + ], + "changes": [ + { + "change": "renamed controls", + "drift": "rename", + "items": 9, + "outcomes": { + "correct": 9 + }, + "silent_wrong": 0, + "recovered_by_block_end": true, + "people_fix_added": "", + "operator_min": 0.0, + "engineer_min": 0.0, + "model_usd": 0.1289, + "excess_people_min": 0.0, + "excess_usd": 0.011, + "excess_machine_usd": 0.01095416666666671, + "excess_people_usd": 0.0 + }, + { + "change": "display scaling", + "drift": "zoom", + "items": 9, + "outcomes": { + "correct": 9 + }, + "silent_wrong": 0, + "recovered_by_block_end": true, + "people_fix_added": "", + "operator_min": 0.0, + "engineer_min": 0.0, + "model_usd": 0.1281, + "excess_people_min": 0.0, + "excess_usd": 0.009, + "excess_machine_usd": 0.009194111111111125, + "excess_people_usd": 0.0 + }, + { + "change": "moved controls", + "drift": "move", + "items": 9, + "outcomes": { + "correct": 9 + }, + "silent_wrong": 0, + "recovered_by_block_end": true, + "people_fix_added": "", + "operator_min": 0.0, + "engineer_min": 0.0, + "model_usd": 0.1278, + "excess_people_min": 0.0, + "excess_usd": 0.009, + "excess_machine_usd": 0.009121666666666667, + "excess_people_usd": 0.0 + }, + { + "change": "new interstitial", + "drift": "notice", + "items": 10, + "outcomes": { + "correct": 10 + }, + "silent_wrong": 0, + "recovered_by_block_end": true, + "people_fix_added": "", + "operator_min": 0.0, + "engineer_min": 0.0, + "model_usd": 0.163, + "excess_people_min": 0.0, + "excess_usd": 0.033, + "excess_machine_usd": 0.03343253703703705, + "excess_people_usd": 0.0 + }, + { + "change": "latency", + "drift": "slow", + "items": 9, + "outcomes": { + "correct": 9 + }, + "silent_wrong": 0, + "recovered_by_block_end": true, + "people_fix_added": "", + "operator_min": 0.0, + "engineer_min": 0.0, + "model_usd": 0.1779, + "excess_people_min": 0.0, + "excess_usd": 0.062, + "excess_machine_usd": 0.06249383333333336, + "excess_people_usd": 0.0 + }, + { + "change": "reordered queue", + "drift": "sort", + "items": 10, + "outcomes": { + "correct": 10 + }, + "silent_wrong": 0, + "recovered_by_block_end": true, + "people_fix_added": "", + "operator_min": 0.0, + "engineer_min": 0.0, + "model_usd": 0.1427, + "excess_people_min": 0.0, + "excess_usd": 0.011, + "excess_machine_usd": 0.011379814814814815, + "excess_people_usd": 0.0 + }, + { + "change": "renamed and moved New Encounter", + "drift": "relink", + "items": 9, + "outcomes": { + "correct": 9 + }, + "silent_wrong": 0, + "recovered_by_block_end": true, + "people_fix_added": "", + "operator_min": 0.0, + "engineer_min": 0.0, + "model_usd": 0.129, + "excess_people_min": 0.0, + "excess_usd": 0.011, + "excess_machine_usd": 0.01070122222222225, + "excess_people_usd": 0.0 + }, + { + "change": "look-alike referral", + "drift": "lookalike", + "items": 5, + "outcomes": { + "correct": 5 + }, + "silent_wrong": 0, + "recovered_by_block_end": true, + "people_fix_added": "", + "operator_min": 0.0, + "engineer_min": 0.0, + "model_usd": 0.0725, + "excess_people_min": 0.0, + "excess_usd": 0.007, + "excess_machine_usd": 0.006629685185185183, + "excess_people_usd": 0.0 + }, + { + "change": "relabeled and reordered type", + "drift": "typelabel", + "items": 5, + "outcomes": { + "correct": 5 + }, + "silent_wrong": 0, + "recovered_by_block_end": true, + "people_fix_added": "", + "operator_min": 0.0, + "engineer_min": 0.0, + "model_usd": 0.0723, + "excess_people_min": 0.0, + "excess_usd": 0.007, + "excess_machine_usd": 0.006753018518518533, + "excess_people_usd": 0.0 + }, + { + "change": "target referral gone", + "drift": "missing", + "items": 5, + "outcomes": { + "safe_stop": 5 + }, + "silent_wrong": 0, + "recovered_by_block_end": true, + "people_fix_added": "", + "operator_min": 10.0, + "engineer_min": 0.0, + "model_usd": 0.0295, + "excess_people_min": 10.0, + "excess_usd": 4.961, + "excess_machine_usd": -0.03893248148148148, + "excess_people_usd": 5.0 + } + ], + "totals": { + "10": { + "setup_usd": 100.0, + "per_item_usd": 0.14, + "faults_usd": 0.19, + "ui_changes_usd": 0.02, + "total_usd": 100.35, + "people_min": 60.4, + "model_usd": 0.13, + "silent_wrong_expected": 0.0 + }, + "100": { + "setup_usd": 100.0, + "per_item_usd": 1.39, + "faults_usd": 1.9, + "ui_changes_usd": 0.2, + "total_usd": 103.5, + "people_min": 64.2, + "model_usd": 1.32, + "silent_wrong_expected": 0.0 + }, + "1000": { + "setup_usd": 100.0, + "per_item_usd": 13.9, + "faults_usd": 19.04, + "ui_changes_usd": 2.05, + "total_usd": 134.99, + "people_min": 102.0, + "model_usd": 13.18, + "silent_wrong_expected": 0.0 + } + }, + "totals_people_x0.5": { + "10": { + "setup_usd": 50.0, + "per_item_usd": 0.14, + "faults_usd": 0.1, + "ui_changes_usd": 0.01, + "total_usd": 50.24, + "people_min": 30.2, + "model_usd": 0.13, + "silent_wrong_expected": 0.0 + }, + "100": { + "setup_usd": 50.0, + "per_item_usd": 1.39, + "faults_usd": 0.95, + "ui_changes_usd": 0.1, + "total_usd": 52.45, + "people_min": 32.1, + "model_usd": 1.32, + "silent_wrong_expected": 0.0 + }, + "1000": { + "setup_usd": 50.0, + "per_item_usd": 13.9, + "faults_usd": 9.54, + "ui_changes_usd": 1.05, + "total_usd": 74.49, + "people_min": 51.0, + "model_usd": 13.18, + "silent_wrong_expected": 0.0 + } + }, + "totals_people_x2": { + "10": { + "setup_usd": 200.0, + "per_item_usd": 0.14, + "faults_usd": 0.38, + "ui_changes_usd": 0.04, + "total_usd": 200.56, + "people_min": 120.8, + "model_usd": 0.13, + "silent_wrong_expected": 0.0 + }, + "100": { + "setup_usd": 200.0, + "per_item_usd": 1.39, + "faults_usd": 3.8, + "ui_changes_usd": 0.4, + "total_usd": 205.6, + "people_min": 128.4, + "model_usd": 1.32, + "silent_wrong_expected": 0.0 + }, + "1000": { + "setup_usd": 200.0, + "per_item_usd": 13.9, + "faults_usd": 38.04, + "ui_changes_usd": 4.05, + "total_usd": 255.99, + "people_min": 204.0, + "model_usd": 13.18, + "silent_wrong_expected": 0.0 + } + } + }, + { + "arm": "agent_script", + "label": "2. Agent-written Playwright script with read-back and duplicate guard, agent-maintained", + "kind": "script", + "source": "silent_saves/results/1.1.0/agent-script/2026-10-24", + "measured": { + "items": 100, + "verified_of_saveable": 90, + "saveable": 90, + "silent_wrong": 0, + "flagged_wrong": 3, + "false_stops": 0, + "faults_reported_as_success": 0, + "stopped_when_no_save_possible": 5, + "model_usd_total": 0.264, + "model_calls_total": 36, + "control_median_s": 0.8, + "control_median_host_load": 5.7, + "control_model_usd_per_item": 0.0 + }, + "setup": { + "model_usd": 0.026108, + "agent_min": 0.3522666666666667, + "operator_min": 0.0, + "engineer_min": 30.0, + "usd": 50.026108 + }, + "steady": { + "model_usd": 0.0, + "operator_s": 0.0, + "engineer_s": 0.0, + "compute_usd": 0.00012392962962962963 + }, + "steady_usd": 0.00012392962962962963, + "faults": [ + { + "item": 20, + "fault": "no_persist", + "outcome": "safe_stop", + "excess_usd": 0.9999222925925926, + "excess_machine_usd": -7.77074074074074e-05, + "excess_people_usd": 1.0, + "excess_people_min": 2.0 + }, + { + "item": 30, + "fault": "wrong_record", + "outcome": "flagged_wrong", + "excess_usd": 2.4999299592592594, + "excess_machine_usd": -7.004074074074075e-05, + "excess_people_usd": 2.5, + "excess_people_min": 5.0 + }, + { + "item": 40, + "fault": "partial", + "outcome": "flagged_wrong", + "excess_usd": 2.499924625925926, + "excess_machine_usd": -7.537407407407406e-05, + "excess_people_usd": 2.5, + "excess_people_min": 5.0 + }, + { + "item": 60, + "fault": "duplicate", + "outcome": "flagged_wrong", + "excess_usd": 2.5004645703703705, + "excess_machine_usd": 0.0004645703703703704, + "excess_people_usd": 2.5, + "excess_people_min": 5.0 + }, + { + "item": 80, + "fault": "optimistic", + "outcome": "safe_stop", + "excess_usd": 1.0000090703703703, + "excess_machine_usd": 9.070370370370379e-06, + "excess_people_usd": 1.0, + "excess_people_min": 2.0 + } + ], + "changes": [ + { + "change": "renamed controls", + "drift": "rename", + "items": 9, + "outcomes": { + "correct": 9 + }, + "silent_wrong": 0, + "recovered_by_block_end": true, + "people_fix_added": "", + "operator_min": 0.0, + "engineer_min": 0.0, + "model_usd": 0.0, + "excess_people_min": 0.0, + "excess_usd": -0.001, + "excess_machine_usd": -0.0006858666666666668, + "excess_people_usd": 0.0 + }, + { + "change": "display scaling", + "drift": "zoom", + "items": 9, + "outcomes": { + "correct": 9 + }, + "silent_wrong": 0, + "recovered_by_block_end": true, + "people_fix_added": "", + "operator_min": 0.0, + "engineer_min": 0.0, + "model_usd": 0.0, + "excess_people_min": 0.0, + "excess_usd": -0.001, + "excess_machine_usd": -0.0006713666666666668, + "excess_people_usd": 0.0 + }, + { + "change": "moved controls", + "drift": "move", + "items": 9, + "outcomes": { + "correct": 9 + }, + "silent_wrong": 0, + "recovered_by_block_end": true, + "people_fix_added": "", + "operator_min": 0.0, + "engineer_min": 0.0, + "model_usd": 0.0, + "excess_people_min": 0.0, + "excess_usd": -0.001, + "excess_machine_usd": -0.0006850333333333334, + "excess_people_usd": 0.0 + }, + { + "change": "new interstitial", + "drift": "notice", + "items": 10, + "outcomes": { + "correct": 10 + }, + "silent_wrong": 0, + "recovered_by_block_end": true, + "people_fix_added": "", + "operator_min": 0.0, + "engineer_min": 10.0, + "model_usd": 0.0385, + "excess_people_min": 10.0, + "excess_usd": 16.706, + "excess_machine_usd": 0.03937581481481482, + "excess_people_usd": 16.666666666666664 + }, + { + "change": "latency", + "drift": "slow", + "items": 9, + "outcomes": { + "correct": 9 + }, + "silent_wrong": 0, + "recovered_by_block_end": true, + "people_fix_added": "", + "operator_min": 0.0, + "engineer_min": 0.0, + "model_usd": 0.0, + "excess_people_min": 0.0, + "excess_usd": 0.004, + "excess_machine_usd": 0.004206244444444444, + "excess_people_usd": 0.0 + }, + { + "change": "reordered queue", + "drift": "sort", + "items": 10, + "outcomes": { + "correct": 10 + }, + "silent_wrong": 0, + "recovered_by_block_end": true, + "people_fix_added": "", + "operator_min": 0.0, + "engineer_min": 0.0, + "model_usd": 0.0, + "excess_people_min": 0.0, + "excess_usd": 0.001, + "excess_machine_usd": 0.0008796481481481483, + "excess_people_usd": 0.0 + }, + { + "change": "renamed and moved New Encounter", + "drift": "relink", + "items": 9, + "outcomes": { + "correct": 9 + }, + "silent_wrong": 0, + "recovered_by_block_end": true, + "people_fix_added": "", + "operator_min": 0.0, + "engineer_min": 10.0, + "model_usd": 0.0389, + "excess_people_min": 10.0, + "excess_usd": 16.707, + "excess_machine_usd": 0.04070241111111111, + "excess_people_usd": 16.666666666666664 + }, + { + "change": "look-alike referral", + "drift": "lookalike", + "items": 5, + "outcomes": { + "correct": 5 + }, + "silent_wrong": 0, + "recovered_by_block_end": true, + "people_fix_added": "", + "operator_min": 0.0, + "engineer_min": 10.0, + "model_usd": 0.0391, + "excess_people_min": 10.0, + "excess_usd": 16.708, + "excess_machine_usd": 0.04089062962962963, + "excess_people_usd": 16.666666666666664 + }, + { + "change": "relabeled and reordered type", + "drift": "typelabel", + "items": 5, + "outcomes": { + "correct": 5 + }, + "silent_wrong": 0, + "recovered_by_block_end": true, + "people_fix_added": "", + "operator_min": 0.0, + "engineer_min": 0.0, + "model_usd": 0.0, + "excess_people_min": 0.0, + "excess_usd": 0.0, + "excess_machine_usd": 4.207407407407415e-05, + "excess_people_usd": 0.0 + }, + { + "change": "target referral gone", + "drift": "missing", + "items": 5, + "outcomes": { + "safe_stop": 5 + }, + "silent_wrong": 0, + "recovered_by_block_end": true, + "people_fix_added": "", + "operator_min": 10.0, + "engineer_min": 30.0, + "model_usd": 0.1214, + "excess_people_min": 40.0, + "excess_usd": 55.128, + "excess_machine_usd": 0.12781646296296295, + "excess_people_usd": 55.0 + } + ], + "totals": { + "10": { + "setup_usd": 50.03, + "per_item_usd": 0.0, + "faults_usd": 0.19, + "ui_changes_usd": 0.42, + "total_usd": 50.64, + "people_min": 30.7, + "model_usd": 0.03, + "silent_wrong_expected": 0.0 + }, + "100": { + "setup_usd": 50.03, + "per_item_usd": 0.01, + "faults_usd": 1.9, + "ui_changes_usd": 4.21, + "total_usd": 56.15, + "people_min": 36.6, + "model_usd": 0.03, + "silent_wrong_expected": 0.0 + }, + "1000": { + "setup_usd": 50.03, + "per_item_usd": 0.12, + "faults_usd": 19.0, + "ui_changes_usd": 42.1, + "total_usd": 111.25, + "people_min": 96.0, + "model_usd": 0.03, + "silent_wrong_expected": 0.0 + } + }, + "totals_people_x0.5": { + "10": { + "setup_usd": 25.03, + "per_item_usd": 0.0, + "faults_usd": 0.1, + "ui_changes_usd": 0.21, + "total_usd": 25.33, + "people_min": 15.3, + "model_usd": 0.03, + "silent_wrong_expected": 0.0 + }, + "100": { + "setup_usd": 25.03, + "per_item_usd": 0.01, + "faults_usd": 0.95, + "ui_changes_usd": 2.11, + "total_usd": 28.1, + "people_min": 18.3, + "model_usd": 0.03, + "silent_wrong_expected": 0.0 + }, + "1000": { + "setup_usd": 25.03, + "per_item_usd": 0.12, + "faults_usd": 9.5, + "ui_changes_usd": 21.1, + "total_usd": 55.75, + "people_min": 48.0, + "model_usd": 0.03, + "silent_wrong_expected": 0.0 + } + }, + "totals_people_x2": { + "10": { + "setup_usd": 100.03, + "per_item_usd": 0.0, + "faults_usd": 0.38, + "ui_changes_usd": 0.84, + "total_usd": 101.25, + "people_min": 61.3, + "model_usd": 0.03, + "silent_wrong_expected": 0.0 + }, + "100": { + "setup_usd": 100.03, + "per_item_usd": 0.01, + "faults_usd": 3.8, + "ui_changes_usd": 8.41, + "total_usd": 112.25, + "people_min": 73.2, + "model_usd": 0.03, + "silent_wrong_expected": 0.0 + }, + "1000": { + "setup_usd": 100.03, + "per_item_usd": 0.12, + "faults_usd": 38.0, + "ui_changes_usd": 84.1, + "total_usd": 222.25, + "people_min": 192.0, + "model_usd": 0.03, + "silent_wrong_expected": 0.0 + } + } + }, + { + "arm": "openadapt_program", + "label": "3. OpenAdapt admitted program, read-back on, no model calls", + "kind": "program", + "source": "silent_saves/results/1.1.0/openadapt-program/2026-10-24", + "measured": { + "items": 100, + "verified_of_saveable": 71, + "saveable": 90, + "silent_wrong": 0, + "flagged_wrong": 3, + "false_stops": 19, + "faults_reported_as_success": 0, + "stopped_when_no_save_possible": 5, + "model_usd_total": 0.0, + "model_calls_total": 0, + "control_median_s": 7.31, + "control_median_host_load": 7.5, + "control_model_usd_per_item": 0.0 + }, + "setup": { + "model_usd": 0, + "agent_min": 0.0, + "operator_min": 22.0, + "engineer_min": 30.0, + "usd": 61.0 + }, + "steady": { + "model_usd": 0.0, + "operator_s": 0.0, + "engineer_s": 0.0, + "compute_usd": 0.0014513296296296298 + }, + "steady_usd": 0.0014513296296296298, + "faults": [ + { + "item": 20, + "fault": "no_persist", + "outcome": "safe_stop", + "excess_usd": 1.0026920037037037, + "excess_machine_usd": 0.002692003703703703, + "excess_people_usd": 1.0, + "excess_people_min": 2.0 + }, + { + "item": 30, + "fault": "wrong_record", + "outcome": "flagged_wrong", + "excess_usd": 2.501130448148148, + "excess_machine_usd": 0.001130448148148148, + "excess_people_usd": 2.5, + "excess_people_min": 5.0 + }, + { + "item": 40, + "fault": "partial", + "outcome": "flagged_wrong", + "excess_usd": 2.500343114814815, + "excess_machine_usd": 0.00034311481481481463, + "excess_people_usd": 2.5, + "excess_people_min": 5.0 + }, + { + "item": 60, + "fault": "duplicate", + "outcome": "flagged_wrong", + "excess_usd": 2.501498337037037, + "excess_machine_usd": 0.001498337037037037, + "excess_people_usd": 2.5, + "excess_people_min": 5.0 + }, + { + "item": 80, + "fault": "optimistic", + "outcome": "safe_stop", + "excess_usd": 0.9999588925925926, + "excess_machine_usd": -4.1107407407407585e-05, + "excess_people_usd": 1.0, + "excess_people_min": 2.0 + } + ], + "changes": [ + { + "change": "renamed controls", + "drift": "rename", + "items": 9, + "outcomes": { + "correct": 9 + }, + "silent_wrong": 0, + "recovered_by_block_end": true, + "people_fix_added": "", + "operator_min": 0.0, + "engineer_min": 0.0, + "model_usd": 0.0, + "excess_people_min": 0.0, + "excess_usd": 0.0, + "excess_machine_usd": 0.00014881111111111033, + "excess_people_usd": 0.0 + }, + { + "change": "display scaling", + "drift": "zoom", + "items": 9, + "outcomes": { + "correct": 9 + }, + "silent_wrong": 0, + "recovered_by_block_end": true, + "people_fix_added": "", + "operator_min": 0.0, + "engineer_min": 0.0, + "model_usd": 0.0, + "excess_people_min": 0.0, + "excess_usd": 0.012, + "excess_machine_usd": 0.011711699999999999, + "excess_people_usd": 0.0 + }, + { + "change": "moved controls", + "drift": "move", + "items": 9, + "outcomes": { + "correct": 9 + }, + "silent_wrong": 0, + "recovered_by_block_end": true, + "people_fix_added": "", + "operator_min": 0.0, + "engineer_min": 0.0, + "model_usd": 0.0, + "excess_people_min": 0.0, + "excess_usd": 0.005, + "excess_machine_usd": 0.004591700000000001, + "excess_people_usd": 0.0 + }, + { + "change": "new interstitial", + "drift": "notice", + "items": 10, + "outcomes": { + "safe_stop": 10 + }, + "silent_wrong": 0, + "recovered_by_block_end": false, + "people_fix_added": "re-teach", + "operator_min": 52.0, + "engineer_min": 0.0, + "model_usd": 0.0, + "excess_people_min": 52.0, + "excess_usd": 26.003, + "excess_machine_usd": 0.0029962037037037023, + "excess_people_usd": 26.0 + }, + { + "change": "latency", + "drift": "slow", + "items": 9, + "outcomes": { + "correct": 9 + }, + "silent_wrong": 0, + "recovered_by_block_end": true, + "people_fix_added": "", + "operator_min": 0.0, + "engineer_min": 0.0, + "model_usd": 0.0, + "excess_people_min": 0.0, + "excess_usd": 0.008, + "excess_machine_usd": 0.007667477777777779, + "excess_people_usd": 0.0 + }, + { + "change": "reordered queue", + "drift": "sort", + "items": 10, + "outcomes": { + "correct": 10 + }, + "silent_wrong": 0, + "recovered_by_block_end": true, + "people_fix_added": "", + "operator_min": 0.0, + "engineer_min": 0.0, + "model_usd": 0.0, + "excess_people_min": 0.0, + "excess_usd": 0.007, + "excess_machine_usd": 0.007410592592592594, + "excess_people_usd": 0.0 + }, + { + "change": "renamed and moved New Encounter", + "drift": "relink", + "items": 9, + "outcomes": { + "safe_stop": 9 + }, + "silent_wrong": 0, + "recovered_by_block_end": false, + "people_fix_added": "re-teach", + "operator_min": 49.0, + "engineer_min": 0.0, + "model_usd": 0.0, + "excess_people_min": 49.0, + "excess_usd": 24.496, + "excess_machine_usd": -0.003620911111111111, + "excess_people_usd": 24.5 + }, + { + "change": "look-alike referral", + "drift": "lookalike", + "items": 5, + "outcomes": { + "correct": 5 + }, + "silent_wrong": 0, + "recovered_by_block_end": true, + "people_fix_added": "", + "operator_min": 0.0, + "engineer_min": 0.0, + "model_usd": 0.0, + "excess_people_min": 0.0, + "excess_usd": -0.003, + "excess_machine_usd": -0.0033223703703703714, + "excess_people_usd": 0.0 + }, + { + "change": "relabeled and reordered type", + "drift": "typelabel", + "items": 5, + "outcomes": { + "correct": 5 + }, + "silent_wrong": 0, + "recovered_by_block_end": true, + "people_fix_added": "", + "operator_min": 0.0, + "engineer_min": 0.0, + "model_usd": 0.0, + "excess_people_min": 0.0, + "excess_usd": -0.005, + "excess_machine_usd": -0.004878425925925927, + "excess_people_usd": 0.0 + }, + { + "change": "target referral gone", + "drift": "missing", + "items": 5, + "outcomes": { + "safe_stop": 5 + }, + "silent_wrong": 0, + "recovered_by_block_end": true, + "people_fix_added": "", + "operator_min": 10.0, + "engineer_min": 0.0, + "model_usd": 0.0, + "excess_people_min": 10.0, + "excess_usd": 4.993, + "excess_machine_usd": -0.006515537037037038, + "excess_people_usd": 5.0 + } + ], + "totals": { + "10": { + "setup_usd": 61.0, + "per_item_usd": 0.01, + "faults_usd": 0.19, + "ui_changes_usd": 0.22, + "total_usd": 61.43, + "people_min": 52.8, + "model_usd": 0.0, + "silent_wrong_expected": 0.0 + }, + "100": { + "setup_usd": 61.0, + "per_item_usd": 0.15, + "faults_usd": 1.9, + "ui_changes_usd": 2.22, + "total_usd": 65.27, + "people_min": 60.2, + "model_usd": 0.0, + "silent_wrong_expected": 0.0 + }, + "1000": { + "setup_usd": 61.0, + "per_item_usd": 1.45, + "faults_usd": 19.01, + "ui_changes_usd": 22.21, + "total_usd": 103.67, + "people_min": 134.4, + "model_usd": 0.0, + "silent_wrong_expected": 0.0 + } + }, + "totals_people_x0.5": { + "10": { + "setup_usd": 30.5, + "per_item_usd": 0.01, + "faults_usd": 0.1, + "ui_changes_usd": 0.11, + "total_usd": 30.72, + "people_min": 26.4, + "model_usd": 0.0, + "silent_wrong_expected": 0.0 + }, + "100": { + "setup_usd": 30.5, + "per_item_usd": 0.15, + "faults_usd": 0.95, + "ui_changes_usd": 1.11, + "total_usd": 32.71, + "people_min": 30.1, + "model_usd": 0.0, + "silent_wrong_expected": 0.0 + }, + "1000": { + "setup_usd": 30.5, + "per_item_usd": 1.45, + "faults_usd": 9.51, + "ui_changes_usd": 11.11, + "total_usd": 52.57, + "people_min": 67.2, + "model_usd": 0.0, + "silent_wrong_expected": 0.0 + } + }, + "totals_people_x2": { + "10": { + "setup_usd": 122.0, + "per_item_usd": 0.01, + "faults_usd": 0.38, + "ui_changes_usd": 0.44, + "total_usd": 122.84, + "people_min": 105.6, + "model_usd": 0.0, + "silent_wrong_expected": 0.0 + }, + "100": { + "setup_usd": 122.0, + "per_item_usd": 0.15, + "faults_usd": 3.8, + "ui_changes_usd": 4.44, + "total_usd": 130.39, + "people_min": 120.5, + "model_usd": 0.0, + "silent_wrong_expected": 0.0 + }, + "1000": { + "setup_usd": 122.0, + "per_item_usd": 1.45, + "faults_usd": 38.01, + "ui_changes_usd": 44.41, + "total_usd": 205.87, + "people_min": 268.8, + "model_usd": 0.0, + "silent_wrong_expected": 0.0 + } + } + }, + { + "arm": "openadapt_self_repair", + "label": "3r. OpenAdapt program with self-repair (engine of 2026-10-19)", + "kind": "program", + "source": "silent_saves/results/1.1.0/self-repair/2026-10-19", + "measured": { + "items": 100, + "verified_of_saveable": 62, + "saveable": 90, + "silent_wrong": 0, + "flagged_wrong": 2, + "false_stops": 28, + "faults_reported_as_success": 0, + "stopped_when_no_save_possible": 5, + "model_usd_total": 0.4767, + "model_calls_total": 30, + "control_median_s": 7.43, + "control_median_host_load": 10.5, + "control_model_usd_per_item": 0.0 + }, + "setup": { + "model_usd": 0, + "agent_min": 0.0, + "operator_min": 22.0, + "engineer_min": 30.0, + "usd": 61.0 + }, + "steady": { + "model_usd": 0.0, + "operator_s": 0.0, + "engineer_s": 0.0, + "compute_usd": 0.0004141740740740741 + }, + "steady_usd": 0.0004141740740740741, + "faults": [ + { + "item": 20, + "fault": "no_persist", + "outcome": "safe_stop", + "excess_usd": 1.000312548148148, + "excess_machine_usd": 0.00031254814814814816, + "excess_people_usd": 1.0, + "excess_people_min": 2.0 + }, + { + "item": 30, + "fault": "wrong_record", + "outcome": "safe_stop", + "excess_usd": 1.0000156037037036, + "excess_machine_usd": 1.56037037037037e-05, + "excess_people_usd": 1.0, + "excess_people_min": 2.0 + }, + { + "item": 40, + "fault": "partial", + "outcome": "flagged_wrong", + "excess_usd": 2.5002997703703707, + "excess_machine_usd": 0.0002997703703703705, + "excess_people_usd": 2.5, + "excess_people_min": 5.0 + }, + { + "item": 60, + "fault": "duplicate", + "outcome": "flagged_wrong", + "excess_usd": 2.500575714814815, + "excess_machine_usd": 0.000575714814814815, + "excess_people_usd": 2.5, + "excess_people_min": 5.0 + }, + { + "item": 80, + "fault": "optimistic", + "outcome": "safe_stop", + "excess_usd": 1.0361346037037036, + "excess_machine_usd": 0.0361346037037037, + "excess_people_usd": 1.0, + "excess_people_min": 2.0 + } + ], + "changes": [ + { + "change": "renamed controls", + "drift": "rename", + "items": 9, + "outcomes": { + "correct": 9 + }, + "silent_wrong": 0, + "recovered_by_block_end": true, + "people_fix_added": "", + "operator_min": 0.0, + "engineer_min": 0.0, + "model_usd": 0.0, + "excess_people_min": 0.0, + "excess_usd": -0.0, + "excess_machine_usd": -4.39999999999989e-06, + "excess_people_usd": 0.0 + }, + { + "change": "display scaling", + "drift": "zoom", + "items": 9, + "outcomes": { + "safe_stop": 9 + }, + "silent_wrong": 0, + "recovered_by_block_end": false, + "people_fix_added": "re-teach", + "operator_min": 49.0, + "engineer_min": 0.0, + "model_usd": 0.0, + "excess_people_min": 49.0, + "excess_usd": 24.5, + "excess_machine_usd": 0.0002111000000000001, + "excess_people_usd": 24.5 + }, + { + "change": "moved controls", + "drift": "move", + "items": 9, + "outcomes": { + "correct": 9 + }, + "silent_wrong": 0, + "recovered_by_block_end": true, + "people_fix_added": "", + "operator_min": 0.0, + "engineer_min": 0.0, + "model_usd": 0.0, + "excess_people_min": 0.0, + "excess_usd": 0.0, + "excess_machine_usd": 1.9266666666666737e-05, + "excess_people_usd": 0.0 + }, + { + "change": "new interstitial", + "drift": "notice", + "items": 10, + "outcomes": { + "safe_stop": 10 + }, + "silent_wrong": 0, + "recovered_by_block_end": false, + "people_fix_added": "re-teach", + "operator_min": 52.0, + "engineer_min": 0.0, + "model_usd": 0.1134, + "excess_people_min": 52.0, + "excess_usd": 26.118, + "excess_machine_usd": 0.11763375925925927, + "excess_people_usd": 26.0 + }, + { + "change": "latency", + "drift": "slow", + "items": 9, + "outcomes": { + "correct": 9 + }, + "silent_wrong": 0, + "recovered_by_block_end": true, + "people_fix_added": "", + "operator_min": 0.0, + "engineer_min": 0.0, + "model_usd": 0.0, + "excess_people_min": 0.0, + "excess_usd": 0.006, + "excess_machine_usd": 0.006329155555555558, + "excess_people_usd": 0.0 + }, + { + "change": "reordered queue", + "drift": "sort", + "items": 10, + "outcomes": { + "correct": 10 + }, + "silent_wrong": 0, + "recovered_by_block_end": true, + "people_fix_added": "", + "operator_min": 0.0, + "engineer_min": 0.0, + "model_usd": 0.0, + "excess_people_min": 0.0, + "excess_usd": 0.0, + "excess_machine_usd": 6.42592592592587e-05, + "excess_people_usd": 0.0 + }, + { + "change": "renamed and moved New Encounter", + "drift": "relink", + "items": 9, + "outcomes": { + "safe_stop": 9 + }, + "silent_wrong": 0, + "recovered_by_block_end": false, + "people_fix_added": "re-teach", + "operator_min": 49.0, + "engineer_min": 0.0, + "model_usd": 0.3274, + "excess_people_min": 49.0, + "excess_usd": 24.829, + "excess_machine_usd": 0.32941971111111107, + "excess_people_usd": 24.5 + }, + { + "change": "look-alike referral", + "drift": "lookalike", + "items": 5, + "outcomes": { + "correct": 5 + }, + "silent_wrong": 0, + "recovered_by_block_end": true, + "people_fix_added": "", + "operator_min": 0.0, + "engineer_min": 0.0, + "model_usd": 0.0, + "excess_people_min": 0.0, + "excess_usd": -0.0, + "excess_machine_usd": -3.964814814814803e-05, + "excess_people_usd": 0.0 + }, + { + "change": "relabeled and reordered type", + "drift": "typelabel", + "items": 5, + "outcomes": { + "correct": 5 + }, + "silent_wrong": 0, + "recovered_by_block_end": true, + "people_fix_added": "", + "operator_min": 0.0, + "engineer_min": 0.0, + "model_usd": 0.0, + "excess_people_min": 0.0, + "excess_usd": -0.0, + "excess_machine_usd": -1.720370370370374e-05, + "excess_people_usd": 0.0 + }, + { + "change": "target referral gone", + "drift": "missing", + "items": 5, + "outcomes": { + "safe_stop": 5 + }, + "silent_wrong": 0, + "recovered_by_block_end": true, + "people_fix_added": "", + "operator_min": 10.0, + "engineer_min": 0.0, + "model_usd": 0.0, + "excess_people_min": 10.0, + "excess_usd": 4.999, + "excess_machine_usd": -0.0013910925925925926, + "excess_people_usd": 5.0 + } + ], + "totals": { + "10": { + "setup_usd": 61.0, + "per_item_usd": 0.0, + "faults_usd": 0.16, + "ui_changes_usd": 0.32, + "total_usd": 61.49, + "people_min": 53.0, + "model_usd": 0.0, + "silent_wrong_expected": 0.0 + }, + "100": { + "setup_usd": 61.0, + "per_item_usd": 0.04, + "faults_usd": 1.61, + "ui_changes_usd": 3.22, + "total_usd": 65.87, + "people_min": 61.6, + "model_usd": 0.0, + "silent_wrong_expected": 0.0 + }, + "1000": { + "setup_usd": 61.0, + "per_item_usd": 0.41, + "faults_usd": 16.07, + "ui_changes_usd": 32.18, + "total_usd": 109.67, + "people_min": 148.0, + "model_usd": 0.0, + "silent_wrong_expected": 0.0 + } + }, + "totals_people_x0.5": { + "10": { + "setup_usd": 30.5, + "per_item_usd": 0.0, + "faults_usd": 0.08, + "ui_changes_usd": 0.16, + "total_usd": 30.75, + "people_min": 26.5, + "model_usd": 0.0, + "silent_wrong_expected": 0.0 + }, + "100": { + "setup_usd": 30.5, + "per_item_usd": 0.04, + "faults_usd": 0.81, + "ui_changes_usd": 1.62, + "total_usd": 32.97, + "people_min": 30.8, + "model_usd": 0.0, + "silent_wrong_expected": 0.0 + }, + "1000": { + "setup_usd": 30.5, + "per_item_usd": 0.41, + "faults_usd": 8.07, + "ui_changes_usd": 16.18, + "total_usd": 55.17, + "people_min": 74.0, + "model_usd": 0.0, + "silent_wrong_expected": 0.0 + } + }, + "totals_people_x2": { + "10": { + "setup_usd": 122.0, + "per_item_usd": 0.0, + "faults_usd": 0.32, + "ui_changes_usd": 0.64, + "total_usd": 122.97, + "people_min": 105.9, + "model_usd": 0.0, + "silent_wrong_expected": 0.0 + }, + "100": { + "setup_usd": 122.0, + "per_item_usd": 0.04, + "faults_usd": 3.21, + "ui_changes_usd": 6.42, + "total_usd": 131.67, + "people_min": 123.2, + "model_usd": 0.0, + "silent_wrong_expected": 0.0 + }, + "1000": { + "setup_usd": 122.0, + "per_item_usd": 0.41, + "faults_usd": 32.07, + "ui_changes_usd": 64.18, + "total_usd": 218.67, + "people_min": 296.0, + "model_usd": 0.0, + "silent_wrong_expected": 0.0 + } + } + }, + { + "arm": "cua_plus_openadapt_readback", + "label": "4 + OpenAdapt read-back: the same agent runs, each claim gated on verify-run", + "kind": "agent", + "source": "silent_saves/results/1.1.0/claude-computer-use+readback/2026-10-19", + "measured": { + "items": 100, + "verified_of_saveable": 90, + "saveable": 90, + "silent_wrong": 0, + "flagged_wrong": 3, + "false_stops": 0, + "faults_reported_as_success": 0, + "stopped_when_no_save_possible": 5, + "model_usd_total": 3.8962, + "model_calls_total": 511, + "control_median_s": 22.51, + "control_median_host_load": 17.0, + "control_model_usd_per_item": 0.03863 + }, + "setup": { + "model_usd": 0, + "agent_min": 0.0, + "operator_min": 0.0, + "engineer_min": 30.0, + "usd": 50.0 + }, + "steady": { + "model_usd": 0.03862666666666666, + "operator_s": 0.0, + "engineer_s": 0.0, + "compute_usd": 0.0014030222222222223 + }, + "steady_usd": 0.04002968888888889, + "faults": [ + { + "item": 20, + "fault": "no_persist", + "outcome": "safe_stop", + "excess_usd": 0.9996672555555555, + "excess_machine_usd": -0.0003327444444444494, + "excess_people_usd": 1.0, + "excess_people_min": 2.0 + }, + { + "item": 30, + "fault": "wrong_record", + "outcome": "flagged_wrong", + "excess_usd": 2.4994242, + "excess_machine_usd": -0.0005757999999999944, + "excess_people_usd": 2.5, + "excess_people_min": 5.0 + }, + { + "item": 40, + "fault": "partial", + "outcome": "flagged_wrong", + "excess_usd": 2.4995784222222226, + "excess_machine_usd": -0.00042157777777777905, + "excess_people_usd": 2.5, + "excess_people_min": 5.0 + }, + { + "item": 60, + "fault": "duplicate", + "outcome": "flagged_wrong", + "excess_usd": 2.4929637000000002, + "excess_machine_usd": -0.007036299999999995, + "excess_people_usd": 2.5, + "excess_people_min": 5.0 + }, + { + "item": 80, + "fault": "optimistic", + "outcome": "safe_stop", + "excess_usd": 0.9998304777777778, + "excess_machine_usd": -0.00016952222222221813, + "excess_people_usd": 1.0, + "excess_people_min": 2.0 + } + ], + "changes": [ + { + "change": "renamed controls", + "drift": "rename", + "items": 9, + "outcomes": { + "correct": 9 + }, + "silent_wrong": 0, + "recovered_by_block_end": true, + "people_fix_added": "", + "operator_min": 0.0, + "engineer_min": 0.0, + "model_usd": 0.3474, + "excess_people_min": 0.0, + "excess_usd": -0.002, + "excess_machine_usd": -0.0024891444444444244, + "excess_people_usd": 0.0 + }, + { + "change": "display scaling", + "drift": "zoom", + "items": 9, + "outcomes": { + "correct": 9 + }, + "silent_wrong": 0, + "recovered_by_block_end": true, + "people_fix_added": "", + "operator_min": 0.0, + "engineer_min": 0.0, + "model_usd": 0.322, + "excess_people_min": 0.0, + "excess_usd": -0.028, + "excess_machine_usd": -0.02787225555555556, + "excess_people_usd": 0.0 + }, + { + "change": "moved controls", + "drift": "move", + "items": 9, + "outcomes": { + "correct": 9 + }, + "silent_wrong": 0, + "recovered_by_block_end": true, + "people_fix_added": "", + "operator_min": 0.0, + "engineer_min": 0.0, + "model_usd": 0.347, + "excess_people_min": 0.0, + "excess_usd": -0.001, + "excess_machine_usd": -0.0013365888888888788, + "excess_people_usd": 0.0 + }, + { + "change": "new interstitial", + "drift": "notice", + "items": 10, + "outcomes": { + "correct": 10 + }, + "silent_wrong": 0, + "recovered_by_block_end": true, + "people_fix_added": "", + "operator_min": 0.0, + "engineer_min": 0.0, + "model_usd": 0.4552, + "excess_people_min": 0.0, + "excess_usd": 0.073, + "excess_machine_usd": 0.0731377222222222, + "excess_people_usd": 0.0 + }, + { + "change": "latency", + "drift": "slow", + "items": 9, + "outcomes": { + "correct": 9 + }, + "silent_wrong": 0, + "recovered_by_block_end": true, + "people_fix_added": "", + "operator_min": 0.0, + "engineer_min": 0.0, + "model_usd": 0.4469, + "excess_people_min": 0.0, + "excess_usd": 0.102, + "excess_machine_usd": 0.10218618888888886, + "excess_people_usd": 0.0 + }, + { + "change": "reordered queue", + "drift": "sort", + "items": 10, + "outcomes": { + "correct": 10 + }, + "silent_wrong": 0, + "recovered_by_block_end": true, + "people_fix_added": "", + "operator_min": 0.0, + "engineer_min": 0.0, + "model_usd": 0.386, + "excess_people_min": 0.0, + "excess_usd": -0.001, + "excess_machine_usd": -0.0010644444444444234, + "excess_people_usd": 0.0 + }, + { + "change": "renamed and moved New Encounter", + "drift": "relink", + "items": 9, + "outcomes": { + "correct": 9 + }, + "silent_wrong": 0, + "recovered_by_block_end": true, + "people_fix_added": "", + "operator_min": 0.0, + "engineer_min": 0.0, + "model_usd": 0.3477, + "excess_people_min": 0.0, + "excess_usd": -0.002, + "excess_machine_usd": -0.001974088888888892, + "excess_people_usd": 0.0 + }, + { + "change": "look-alike referral", + "drift": "lookalike", + "items": 5, + "outcomes": { + "correct": 5 + }, + "silent_wrong": 0, + "recovered_by_block_end": true, + "people_fix_added": "", + "operator_min": 0.0, + "engineer_min": 0.0, + "model_usd": 0.1931, + "excess_people_min": 0.0, + "excess_usd": -0.001, + "excess_machine_usd": -0.0009323333333333406, + "excess_people_usd": 0.0 + }, + { + "change": "relabeled and reordered type", + "drift": "typelabel", + "items": 5, + "outcomes": { + "correct": 5 + }, + "silent_wrong": 0, + "recovered_by_block_end": true, + "people_fix_added": "", + "operator_min": 0.0, + "engineer_min": 0.0, + "model_usd": 0.1929, + "excess_people_min": 0.0, + "excess_usd": -0.001, + "excess_machine_usd": -0.001077722222222216, + "excess_people_usd": 0.0 + }, + { + "change": "target referral gone", + "drift": "missing", + "items": 5, + "outcomes": { + "safe_stop": 5 + }, + "silent_wrong": 0, + "recovered_by_block_end": true, + "people_fix_added": "", + "operator_min": 10.0, + "engineer_min": 0.0, + "model_usd": 0.0933, + "excess_people_min": 10.0, + "excess_usd": 4.895, + "excess_machine_usd": -0.10457955555555554, + "excess_people_usd": 5.0 + } + ], + "totals": { + "10": { + "setup_usd": 50.0, + "per_item_usd": 0.4, + "faults_usd": 0.19, + "ui_changes_usd": 0.02, + "total_usd": 50.61, + "people_min": 30.4, + "model_usd": 0.39, + "silent_wrong_expected": 0.0 + }, + "100": { + "setup_usd": 50.0, + "per_item_usd": 4.0, + "faults_usd": 1.9, + "ui_changes_usd": 0.2, + "total_usd": 56.1, + "people_min": 34.2, + "model_usd": 3.86, + "silent_wrong_expected": 0.0 + }, + "1000": { + "setup_usd": 50.0, + "per_item_usd": 40.03, + "faults_usd": 18.98, + "ui_changes_usd": 2.01, + "total_usd": 111.03, + "people_min": 72.0, + "model_usd": 38.63, + "silent_wrong_expected": 0.0 + } + }, + "totals_people_x0.5": { + "10": { + "setup_usd": 25.0, + "per_item_usd": 0.4, + "faults_usd": 0.09, + "ui_changes_usd": 0.01, + "total_usd": 25.51, + "people_min": 15.2, + "model_usd": 0.39, + "silent_wrong_expected": 0.0 + }, + "100": { + "setup_usd": 25.0, + "per_item_usd": 4.0, + "faults_usd": 0.95, + "ui_changes_usd": 0.1, + "total_usd": 30.05, + "people_min": 17.1, + "model_usd": 3.86, + "silent_wrong_expected": 0.0 + }, + "1000": { + "setup_usd": 25.0, + "per_item_usd": 40.03, + "faults_usd": 9.48, + "ui_changes_usd": 1.01, + "total_usd": 75.53, + "people_min": 36.0, + "model_usd": 38.63, + "silent_wrong_expected": 0.0 + } + }, + "totals_people_x2": { + "10": { + "setup_usd": 100.0, + "per_item_usd": 0.4, + "faults_usd": 0.38, + "ui_changes_usd": 0.04, + "total_usd": 100.82, + "people_min": 60.8, + "model_usd": 0.39, + "silent_wrong_expected": 0.0 + }, + "100": { + "setup_usd": 100.0, + "per_item_usd": 4.0, + "faults_usd": 3.8, + "ui_changes_usd": 0.4, + "total_usd": 108.2, + "people_min": 68.4, + "model_usd": 3.86, + "silent_wrong_expected": 0.0 + }, + "1000": { + "setup_usd": 100.0, + "per_item_usd": 40.03, + "faults_usd": 37.98, + "ui_changes_usd": 4.01, + "total_usd": 182.03, + "people_min": 144.0, + "model_usd": 38.63, + "silent_wrong_expected": 0.0 + } + } + } + ], + "mockmed_missing": [], + "openemr_missing": [ + "human_script", + "cua_alone" + ] +} diff --git a/packages/flow/benchmark/economic_comparison/totals.py b/packages/flow/benchmark/economic_comparison/totals.py new file mode 100644 index 000000000..6708ee2e0 --- /dev/null +++ b/packages/flow/benchmark/economic_comparison/totals.py @@ -0,0 +1,810 @@ +"""Lifecycle totals over 10, 100 and 1,000 items, per arm, from raw results. + + cd packages/flow/benchmark + python -m economic_comparison.totals # writes results/totals.json and results/TABLE.md + +Measured, per arm, from the raw rows (nothing here is re-run): + +- per-item time, model cost and outcome on the control block (no UI change, + no server fault); +- the excess people time and model cost of each server-fault item; +- the excess people time and model cost of each UI change (one block of + items with that change on), including any repair or re-authoring the arm + did, any silent wrongs it made under the change, and a person's fix when + the arm had not recovered by the end of the block; +- setup: the agent's measured authoring time and cost, plus the people time + in ``assumptions.json``. + +People time is priced from ``assumptions.json`` (cost model V1 and the labor +rates), never measured; every total is also given at half and double it. + +Totals for N items = setup + N x (control per-item cost) ++ N x (faults per item) x (mean excess per fault item) ++ N x (changes per item) x (mean excess per UI change). +""" + +from __future__ import annotations + +import csv +import json +import statistics +import sys +from pathlib import Path +from typing import Any, Optional + +HERE = Path(__file__).resolve().parent +BENCH = HERE.parent +SS = BENCH / "silent_saves" / "results" / "1.1.0" +ASSUME = json.loads((HERE / "assumptions.json").read_text()) +SIZES = (10, 100, 1000) + +#: MockMed arms: (name, kind, items.csv, outcome column, setup key, label) +MOCKMED_ARMS: list[dict[str, Any]] = [ + { + "arm": "cua_alone", + "kind": "agent", + "dir": "claude-computer-use+readback/2026-10-19", + "outcome": "outcome_alone", + "setup": "cua_alone", + "label": "4. Computer-use agent, screenshots only (Claude Sonnet 5.5)", + }, + { + "arm": "agent_tools", + "timing_dir": "agent-tools/2026-10-24-timing", + "kind": "agent", + "dir": "agent-tools/2026-10-24", + "outcome": "outcome", + "setup": "agent_tools", + "label": "1. Agent with structured tools and record read-back (Claude Sonnet 5.5)", + }, + { + "arm": "agent_script", + "timing_dir": "agent-script/2026-10-24-timing", + "kind": "script", + "dir": "agent-script/2026-10-24", + "outcome": "outcome", + "setup": "agent_script", + "label": "2. Agent-written Playwright script with read-back and duplicate guard, agent-maintained", + }, + { + "arm": "openadapt_program", + "timing_dir": "openadapt-program/2026-10-24-timing", + "kind": "program", + "dir": "openadapt-program/2026-10-24", + "outcome": "outcome", + "setup": "openadapt", + "label": "3. OpenAdapt admitted program, read-back on, no model calls", + }, + { + "arm": "openadapt_self_repair", + "kind": "program", + "dir": "self-repair/2026-10-19", + "outcome": "outcome", + "setup": "openadapt", + "label": "3r. OpenAdapt program with self-repair (engine of 2026-10-19)", + }, + { + "arm": "cua_plus_openadapt_readback", + "kind": "agent", + "dir": "claude-computer-use+readback/2026-10-19", + "outcome": "outcome", + "setup": "cua_alone", + "label": "4 + OpenAdapt read-back: the same agent runs, each claim gated on verify-run", + }, +] + + +def _people_s(steps: list[str], manual_key: str) -> float: + op = ASSUME["operator_s"] + total = 0.0 + for s in steps: + total += op[manual_key] if s == "manual_entry" else op[s] + if steps and "interrupt" not in steps: + total += op["interrupt"] + return total + + +def operator_s(row: dict[str, Any], outcome: str, manual_key: str) -> float: + """People seconds after one item, from what the arm reported and the store.""" + + per = ASSUME["per_outcome"] + save_possible = row["expect"] == "save" and row["fault"] == "ok" + if outcome == "correct": + return 0.0 + if outcome == "safe_stop": + key = ( + "claimed_failure_save_possible" + if save_possible + else "claimed_failure_no_save_possible" + ) + return _people_s(per[key], manual_key) + return _people_s(per[outcome], manual_key) + + +def _rows(arm: dict[str, Any]) -> Optional[list[dict[str, Any]]]: + p = SS / arm["dir"] / "items.csv" + if not p.is_file(): + return None + rows = sorted(csv.DictReader(p.open()), key=lambda r: int(r["item"])) + return rows if len(rows) == 100 else None + + +def _maint_events(row: dict[str, Any]) -> tuple[float, float]: + """(engineer s, operator s) for the repair events an item logged.""" + + eng = op = 0.0 + m = ASSUME["maintenance_people_min"] + for e in json.loads(row["human_events"] or "[]"): + if e.get("kind") == "script_review" and e.get("phase") == "maintenance": + eng += m["script_fix_review_engineer"] * 60 + elif e.get("kind") == "repair_approval": + op += ( + ASSUME["operator_s"]["repair_review"] + + ASSUME["operator_s"]["interrupt"] + ) + return eng, op + + +def mockmed_arm(arm: dict[str, Any]) -> Optional[dict[str, Any]]: + rows = _rows(arm) + if rows is None: + return None + oc = arm["outcome"] + rate = ASSUME["labor_usd_per_hour"] + cpu = ASSUME["compute_usd_per_hour"] / 3600.0 + + def cost(r: dict[str, Any], *, setup_item: bool = False) -> dict[str, float]: + model = float(r["cost_usd"] or 0) + events = json.loads(r["human_events"] or "[]") + if setup_item or any(e.get("phase") == "setup" for e in events): + model = 0.0 # authoring cost is setup, counted there + eng, op_maint = _maint_events(r) + op = operator_s(r, r[oc], "manual_entry_mockmed") + op_maint + return { + "model_usd": model, + "operator_s": op, + "engineer_s": eng, + "compute_usd": float(r["item_s"] or 0) * cpu, + } + + def usd(c: dict[str, float]) -> float: + return ( + c["model_usd"] + + c["compute_usd"] + + c["operator_s"] / 3600 * rate["operator"] + + c["engineer_s"] / 3600 * rate["engineer"] + ) + + control = [r for r in rows if r["release"] == "baseline" and r["fault"] == "ok"] + steady = { + k: statistics.mean(cost(r)[k] for r in control) + for k in ("model_usd", "operator_s", "engineer_s", "compute_usd") + } + steady_usd = usd(steady) + st_machine = steady["model_usd"] + steady["compute_usd"] + + setup_rows = [ + r + for r in rows + if any(e.get("phase") == "setup" for e in json.loads(r["human_events"] or "[]")) + ] + setup_model = sum(float(r["cost_usd"] or 0) for r in setup_rows) + setup_agent_min = sum(float(r["item_s"] or 0) for r in setup_rows) / 60 + sp = ASSUME["setup_people_min"][arm["setup"]] + setup = { + "model_usd": setup_model, + "agent_min": setup_agent_min, + "operator_min": sp["operator"], + "engineer_min": sp["engineer"], + } + setup["usd"] = ( + setup_model + + sp["operator"] / 60 * rate["operator"] + + sp["engineer"] / 60 * rate["engineer"] + ) + + faults = [r for r in rows if r["fault"] != "ok"] + fault_rows = [] + for r in faults: + c = cost(r) + fault_rows.append( + { + "item": int(r["item"]), + "fault": r["fault"], + "outcome": r[oc], + "excess_usd": usd(c) - steady_usd, + "excess_machine_usd": c["model_usd"] + c["compute_usd"] - st_machine, + "excess_people_usd": _split( + (c["operator_s"] - steady["operator_s"]) / 60, + (c["engineer_s"] - steady["engineer_s"]) / 60, + ), + "excess_people_min": ( + c["operator_s"] + + c["engineer_s"] + - steady["operator_s"] + - steady["engineer_s"] + ) + / 60, + } + ) + + m = ASSUME["maintenance_people_min"] + changes = [] + for rel in dict.fromkeys(r["release"] for r in rows if r["release"] != "baseline"): + block = [r for r in rows if r["release"] == rel and r["fault"] == "ok"] + outs = [r[oc] for r in block] + cs = [cost(r) for r in block] + op_s = sum(c["operator_s"] for c in cs) + eng_s = sum(c["engineer_s"] for c in cs) + model = sum(c["model_usd"] for c in cs) + comp = sum(c["compute_usd"] for c in cs) + expect_save = all(r["expect"] == "save" for r in block) + last_ok = outs[-1] == "correct" if expect_save else outs[-1] in ("safe_stop",) + fix = "" + if not last_ok: + # Still failing when the block ended: a person fixes it once. + if arm["kind"] == "program": + op_s += m["openadapt_reteach_operator"] * 60 + fix = "re-teach" + elif arm["kind"] == "script": + eng_s += m["script_fix_by_hand_engineer"] * 60 + fix = "engineer fixes the script" + else: + eng_s += m["agent_prompt_fix_engineer"] * 60 + fix = "engineer adjusts the prompt" + n = len(block) + excess_usd = ( + model + + comp + + op_s / 3600 * rate["operator"] + + eng_s / 3600 * rate["engineer"] + ) - n * steady_usd + changes.append( + { + "change": rel, + "drift": block[0]["drift"], + "items": n, + "outcomes": {o: outs.count(o) for o in sorted(set(outs))}, + "silent_wrong": outs.count("silent_wrong"), + "recovered_by_block_end": last_ok, + "people_fix_added": fix, + "operator_min": round(op_s / 60, 2), + "engineer_min": round(eng_s / 60, 2), + "model_usd": round(model, 4), + "excess_people_min": round( + (op_s + eng_s) / 60 + - n * (steady["operator_s"] + steady["engineer_s"]) / 60, + 2, + ), + "excess_usd": round(excess_usd, 3), + "excess_machine_usd": model + comp - n * st_machine, + "excess_people_usd": _split( + op_s / 60 - n * steady["operator_s"] / 60, + eng_s / 60 - n * steady["engineer_s"] / 60, + ), + } + ) + + outcomes = [r[oc] for r in rows] + timing = control + if arm.get("timing_dir") and (SS / arm["timing_dir"] / "items.csv").is_file(): + # A separate control-block pass at normal host load; the full run's times are not used. + timing = [ + r + for r in csv.DictReader((SS / arm["timing_dir"] / "items.csv").open()) + if r["release"] == "baseline" and r["fault"] == "ok" + ] + walls = sorted(float(r["item_s"]) for r in timing) + loads = [float(r["host_load_1m"] or 0) for r in timing] + measured = { + "items": len(rows), + "verified_of_saveable": sum( + 1 + for r in rows + if r[oc] == "correct" and r["expect"] == "save" and r["fault"] == "ok" + ), + "saveable": sum( + 1 for r in rows if r["expect"] == "save" and r["fault"] == "ok" + ), + "silent_wrong": outcomes.count("silent_wrong"), + "flagged_wrong": outcomes.count("flagged_wrong"), + "false_stops": sum( + 1 + for r in rows + if r[oc] in ("safe_stop", "false_alarm") + and r["expect"] == "save" + and r["fault"] == "ok" + ), + "faults_reported_as_success": sum(1 for r in faults if r[oc] == "silent_wrong"), + "stopped_when_no_save_possible": sum( + 1 for r in rows if r["expect"] == "stop" and r[oc] == "safe_stop" + ), + "model_usd_total": round(sum(float(r["cost_usd"] or 0) for r in rows), 4), + "model_calls_total": sum(int(r["model_calls"] or 0) for r in rows), + "control_median_s": round(statistics.median(walls), 2), + "control_median_host_load": round(statistics.median(loads), 1), + "control_model_usd_per_item": round(steady["model_usd"], 5), + } + return { + "arm": arm["arm"], + "label": arm["label"], + "kind": arm["kind"], + "source": f"silent_saves/results/1.1.0/{arm['dir']}", + "measured": measured, + "setup": setup, + "steady": steady, + "steady_usd": steady_usd, + "faults": fault_rows, + "changes": changes, + } + + +def _split(c_people_min_op: float, c_people_min_eng: float) -> float: + rate = ASSUME["labor_usd_per_hour"] + return ( + c_people_min_op / 60 * rate["operator"] + + c_people_min_eng / 60 * rate["engineer"] + ) + + +def project(a: dict[str, Any], n: int, people_factor: float = 1.0) -> dict[str, float]: + """Totals for n items. people_factor scales every people-time assumption.""" + + vol = ASSUME["volume"] + ch_rate = vol["ui_changes_per_1000_items"] / 1000 + f_rate = vol["server_faults_per_1000_items"] / 1000 + st = a["steady"] + s = a["setup"] + # (machine $, people $, people min) for each part + setup = ( + s["model_usd"], + _split(s["operator_min"], s["engineer_min"]), + s["operator_min"] + s["engineer_min"], + ) + per_item = ( + st["model_usd"] + st["compute_usd"], + _split(st["operator_s"] / 60, st["engineer_s"] / 60), + (st["operator_s"] + st["engineer_s"]) / 60, + ) + + def mean(xs: list[float]) -> float: + return statistics.mean(xs) if xs else 0.0 + + fault = ( + mean([f["excess_machine_usd"] for f in a["faults"]]), + mean([f["excess_people_usd"] for f in a["faults"]]), + mean([f["excess_people_min"] for f in a["faults"]]), + ) + change = ( + mean([c["excess_machine_usd"] for c in a["changes"]]), + mean([c["excess_people_usd"] for c in a["changes"]]), + mean([c["excess_people_min"] for c in a["changes"]]), + ) + parts = { + "setup": (1.0, setup), + "per_item": (n, per_item), + "faults": (n * f_rate, fault), + "ui_changes": (n * ch_rate, change), + } + out: dict[str, Any] = {} + total = people_min = 0.0 + for k, (mult, (mach, ppl, mins)) in parts.items(): + usd = mult * (mach + people_factor * ppl) + out[f"{k}_usd"] = round(usd, 2) + total += usd + people_min += mult * mins * people_factor + out.update( + total_usd=round(total, 2), + people_min=round(people_min, 1), + model_usd=round(s["model_usd"] + n * st["model_usd"], 2), + silent_wrong_expected=round( + n + * f_rate + * mean( + [1.0 if f["outcome"] == "silent_wrong" else 0.0 for f in a["faults"]] + ) + + n * ch_rate * mean([c["silent_wrong"] for c in a["changes"]]), + 2, + ), + ) + return out + + +# -- OpenEMR: the lifecycle soak's ledgers ------------------------------------------ + +LIFE = BENCH / "lifecycle" / "results" +#: (arm in the ledger, ledger dir, setup key, label) +OPENEMR_ARMS: list[dict[str, Any]] = [ + { + "arm": "openadapt-production-relabel", + "dir": "production-2026-10-24/after", + "setup": "openadapt_regulated_openemr", + "kind": "program", + "label": "3. OpenAdapt, Regulated + problems_only, durable, pipelined read-back, relabel repair " + "(engine of measure/production-lifecycle; the API step is refused on this form, so the GUI save runs)", + }, + { + "arm": "openadapt-production", + "dir": "production-2026-10-24/before", + "setup": "openadapt_regulated_openemr", + "kind": "program", + "allow_partial": True, + "label": "3 on main. Same configuration, current main's engine (the run was stopped at 27 items)", + }, + { + "arm": "agent_tools", + "dir": "economics-2026-10-24/agent_tools", + "setup": "agent_tools", + "kind": "agent", + "label": "1. Agent with structured tools and REST read-back (Claude Sonnet 5.5)", + }, + { + "arm": "agent_script", + "dir": "economics-2026-10-24/agent_script", + "setup": "agent_script", + "kind": "script", + "allow_partial": True, + "label": "2. (hung after 16 items) Agent-written Playwright + REST script with read-back and duplicate check, agent-maintained (Claude Sonnet 5.5)", + }, + { + "arm": "human_script", + "dir": "economics-2026-10-24/human_script", + "setup": "human_script", + "kind": "script", + "label": "2h. Hand-written script, same checks, not maintained during the run", + }, + { + "arm": "cua_alone", + "dir": "economics-2026-10-24/cua_alone", + "setup": "cua_alone", + "kind": "agent", + "label": "4. Computer-use agent, screenshots only (Claude Sonnet 5.5), first 12 items", + }, +] + + +def openemr_arm(spec: dict[str, Any]) -> Optional[dict[str, Any]]: + sys.path.insert(0, str(BENCH)) + from lifecycle.arms import economics as econ + + ledger = LIFE / spec["dir"] / "ledger.jsonl" + if not ledger.is_file(): + return None + rows = econ.read_rows([ledger]) + partial = not any(r.get("type") == "arm_end" for r in rows) + if partial and not spec.get("allow_partial"): + return None + res = econ.compute(rows) + a = next((x for x in res["arms"] if x["arm"] == spec["arm"]), None) + if a is None: + return None + m = a["measured"] + n = a["items"] + op = ASSUME["operator_s"] + mine = [r for r in rows if r.get("arm") == spec["arm"]] + fixes = [ + r + for r in mine + if r.get("type") == "phase" + and r.get("stage") == "maintenance" + and r.get("phase") in ("fix_script",) + ] + reviews = [ + r + for r in mine + if r.get("type") == "operator" and r.get("kind") == "repair_review" + ] + setup_walls = [ + float(r.get("wall_s", 0) or 0) + for r in mine + if r.get("type") == "phase" and r.get("stage") == "setup" + ] + sp = ASSUME["setup_people_min"][spec["setup"]] + # Script fixes are reviewed by an engineer (assumption), not charged as an operator repair review. + eng_maint_min = ( + len(fixes) * ASSUME["maintenance_people_min"]["script_fix_review_engineer"] + if spec["kind"] == "script" + else 0.0 + ) + op_maint_min = 0.0 if spec["kind"] == "script" else m["maintenance_human_min"] + if spec["kind"] == "script" and spec["arm"] == "human_script" and a["drift_events"]: + # Not maintained during the run: an engineer fixes it once per change (prices.json, 30 min). + broke = any( + r.get("type") == "item_result" and (r.get("detail") or {}).get("broke") + for r in mine + ) + eng_maint_min = ( + ASSUME["maintenance_people_min"]["script_fix_by_hand_engineer"] + if broke + else 0.0 + ) + d = max(a["drift_events"], 1) + sw = a["silent_wrong"] + cleanup_s = sw * ( + op["reconcile"] + op["manual_entry_openemr"] + 2 * op["interrupt"] + ) + item_op_min = m["operator_min"] - ( + len(reviews) * (op["repair_review"] + op["interrupt"]) / 60 + if spec["kind"] == "script" + else 0.0 + ) + out = { + "arm": spec["arm"], + "label": spec["label"], + "kind": spec["kind"], + "source": f"lifecycle/results/{spec['dir']}", + "partial": partial, + "items": n, + "verified": a["verified"], + "automated": a["automated"], + "by_hand": a["by_hand"], + "outcomes": a["outcomes"], + "silent_wrong": sw, + "false_stop": a["false_stop"], + "false_stop_of": a["false_stop_denominator"], + "drift_events": a["drift_events"], + "median_s": m["median_wall_s_automated"], + "p90_s": m["p90_wall_s_automated"], + "model_usd_total": round( + m["item_model_usd"] + m["setup_model_usd"] + m["maintenance_model_usd"], 4 + ), + "setup": { + "model_usd": m["setup_model_usd"], + "agent_min": round(sum(setup_walls) / 60, 1), + "operator_min": sp["operator"], + "engineer_min": sp["engineer"], + }, + "per_item": { + "model_usd": m["item_model_usd"] / n, + "compute_usd": m["compute_usd"] / n, + "operator_min": item_op_min / n, + }, + "per_change": { + "model_usd": m["maintenance_model_usd"] / d, + "operator_min": op_maint_min / d, + "engineer_min": eng_maint_min / d, + "fixes": len(fixes), + }, + "per_item_cleanup_min": cleanup_s / 60 / n, + } + if not a["drift_events"]: + # The run never reached the app change: its fixes were first-use debugging, so they are + # setup, and maintenance after a change was not measured. + out["setup"]["model_usd"] += m["maintenance_model_usd"] + out["setup"]["engineer_min"] += eng_maint_min + out["setup"]["debug_fixes"] = len(fixes) + out["per_change"] = { + "model_usd": 0.0, + "operator_min": 0.0, + "engineer_min": 0.0, + "fixes": 0, + "not_measured": True, + } + return out + + +def project_openemr(a: dict[str, Any], n: int, f: float = 1.0) -> dict[str, float]: + rate = ASSUME["labor_usd_per_hour"] + ch = ASSUME["volume"]["ui_changes_per_1000_items"] / 1000 * n + s, pi, pc = a["setup"], a["per_item"], a["per_change"] + op_usd = rate["operator"] / 60 + eng_usd = rate["engineer"] / 60 + setup_usd = s["model_usd"] + f * ( + s["operator_min"] * op_usd + s["engineer_min"] * eng_usd + ) + items_usd = n * ( + pi["model_usd"] + pi["compute_usd"] + f * pi["operator_min"] * op_usd + ) + change_usd = ch * ( + pc["model_usd"] + + f * (pc["operator_min"] * op_usd + pc["engineer_min"] * eng_usd) + ) + cleanup_usd = n * f * a["per_item_cleanup_min"] * op_usd + people = f * ( + s["operator_min"] + + s["engineer_min"] + + n * pi["operator_min"] + + ch * (pc["operator_min"] + pc["engineer_min"]) + + n * a["per_item_cleanup_min"] + ) + return { + "setup_usd": round(setup_usd, 2), + "per_item_usd": round(items_usd, 2), + "ui_changes_usd": round(change_usd, 2), + "silent_wrong_cleanup_usd": round(cleanup_usd, 2), + "total_usd": round(setup_usd + items_usd + change_usd + cleanup_usd, 2), + "people_min": round(people, 1), + "model_usd": round( + s["model_usd"] + n * pi["model_usd"] + ch * pc["model_usd"], 2 + ), + "silent_wrong_expected": round(n * a["silent_wrong"] / a["items"], 2) + if a["items"] + else 0.0, + } + + +def render_openemr(arms: list[dict[str, Any]]) -> str: + L = [ + "## OpenEMR referral intake (seed 7, 55 items, one app change: the save button relabeled at item 26)", + "", + "| Arm | Items | Verified in the end | Automated (no hand entry) | Silent wrong | False stops | Median s/item | Model $/item | Operator min/item | Maintenance for the change (people min / model $) | Setup people min | Setup agent min / $ |", + "|---|---|---|---|---|---|---|---|---|---|---|---|", + ] + for a in arms: + pc, s = a["per_change"], a["setup"] + L.append( + f"| {a['label']} | {a['items']} | {a['verified']} | {a['automated']} | {a['silent_wrong']} | " + f"{a['false_stop']} of {a['false_stop_of']} | {_f(a['median_s'], 1)} | {_f(a['per_item']['model_usd'], 3)} | " + f"{_f(a['per_item']['operator_min'], 2)} | " + + ( + "not reached | " + if pc.get("not_measured") and a["kind"] != "manual" + else f"{_f(pc['operator_min'] + pc['engineer_min'], 1)} / {_f(pc['model_usd'], 2)} | " + ) + + f"{_f(s['operator_min'] + s['engineer_min'], 0)} | {_f(s['agent_min'], 1)} / {_f(s['model_usd'], 2)} |" + ) + L += [ + "", + "### Totals (people time priced at the assumed rates; x0.5 and x2 in brackets)", + "", + "| Arm | " + + " | ".join(f"{n} items: total $ | people min" for n in SIZES) + + " |", + "|---|" + "---|---|" * len(SIZES), + ] + for a in arms: + cells = [] + for n in SIZES: + t, lo, hi = ( + a["totals"][str(n)], + a["totals_people_x0.5"][str(n)], + a["totals_people_x2"][str(n)], + ) + cells += [ + f"{_f(t['total_usd'], 0)} [{_f(lo['total_usd'], 0)}-{_f(hi['total_usd'], 0)}]", + _f(t["people_min"], 0), + ] + L.append(f"| {a['arm']} | " + " | ".join(cells) + " |") + return "\n".join(L) + "\n" + + +def main(argv: Optional[list[str]] = None) -> int: + out = HERE / "results" + out.mkdir(exist_ok=True) + arms = [x for x in (mockmed_arm(a) for a in MOCKMED_ARMS) if x] + missing = [ + a["arm"] for a in MOCKMED_ARMS if not any(x["arm"] == a["arm"] for x in arms) + ] + for a in arms: + a["totals"] = {str(n): project(a, n) for n in SIZES} + a["totals_people_x0.5"] = {str(n): project(a, n, 0.5) for n in SIZES} + a["totals_people_x2"] = {str(n): project(a, n, 2.0) for n in SIZES} + oe = [x for x in (openemr_arm(a) for a in OPENEMR_ARMS) if x] + op = ASSUME["operator_s"] + manual_min = (op["manual_entry_openemr"] + op["interrupt"]) / 60 + oe.append( + { + "arm": "all_manual", + "label": "Reference: a person enters every item (cost model V1)", + "kind": "manual", + "source": "cost model", + "partial": False, + "items": None, + "verified": None, + "automated": 0, + "by_hand": None, + "outcomes": {}, + "silent_wrong": 0, + "false_stop": None, + "false_stop_of": None, + "drift_events": 0, + "median_s": manual_min * 60, + "p90_s": None, + "model_usd_total": 0.0, + "setup": { + "model_usd": 0.0, + "agent_min": 0.0, + "operator_min": 0.0, + "engineer_min": 0.0, + }, + "per_item": { + "model_usd": 0.0, + "compute_usd": 0.0, + "operator_min": manual_min, + }, + "per_change": { + "model_usd": 0.0, + "operator_min": 0.0, + "engineer_min": 0.0, + "fixes": 0, + }, + "per_item_cleanup_min": 0.0, + } + ) + for a in oe: + a["totals"] = {str(n): project_openemr(a, n) for n in SIZES} + a["totals_people_x0.5"] = {str(n): project_openemr(a, n, 0.5) for n in SIZES} + a["totals_people_x2"] = {str(n): project_openemr(a, n, 2.0) for n in SIZES} + result = { + "assumptions": ASSUME["version"], + "openemr": oe, + "mockmed": arms, + "mockmed_missing": missing, + "openemr_missing": [ + a["arm"] for a in OPENEMR_ARMS if not any(x["arm"] == a["arm"] for x in oe) + ], + } + (out / "totals.json").write_text(json.dumps(result, indent=1, default=str) + "\n") + text = render_openemr(oe) + "\n" + render(result) + (out / "TABLE.md").write_text(text) + print(text) + return 0 + + +def _f(x: Any, nd: int = 2) -> str: + return f"{x:,.{nd}f}" if isinstance(x, (int, float)) else str(x) + + +def render(result: dict[str, Any]) -> str: + arms = result["mockmed"] + L = [ + "## MockMed, fault set 1.1.0 (100 items, 10 UI changes, 5 server faults)", + "", + "| Arm | Verified (of 90) | Silent wrong | False stops | Faults claimed saved (of 5) | Stopped when no save possible (of 5) | Control s/item (load) | Model $/item | Setup people min | Setup agent min / $ |", + "|---|---|---|---|---|---|---|---|---|---|", + ] + for a in arms: + m, s = a["measured"], a["setup"] + L.append( + f"| {a['label']} | {m['verified_of_saveable']} | {m['silent_wrong']} | {m['false_stops']} | " + f"{m['faults_reported_as_success']} | {m['stopped_when_no_save_possible']} | " + f"{_f(m['control_median_s'], 1)} ({_f(m['control_median_host_load'], 0)}) | {_f(m['control_model_usd_per_item'], 4)} | " + f"{_f(s['operator_min'] + s['engineer_min'], 0)} | {_f(s['agent_min'], 1)} / {_f(s['model_usd'], 3)} |" + ) + L += [ + "", + "### Each UI change: excess people minutes (silent wrongs in the block)", + "", + "| Change | " + " | ".join(a["arm"] for a in arms) + " |", + "|---|" + "---|" * len(arms), + ] + for i, c in enumerate(arms[0]["changes"]): + cells = [] + for a in arms: + ch = a["changes"][i] + cells.append( + f"{_f(ch['excess_people_min'], 1)}" + + (f" ({ch['silent_wrong']} SW)" if ch["silent_wrong"] else "") + + (" +fix" if ch["people_fix_added"] else "") + ) + L.append(f"| {c['change']} | " + " | ".join(cells) + " |") + L += [ + "", + "### Totals (people time priced at the assumed rates; x0.5 and x2 in brackets)", + "", + "| Arm | " + + " | ".join(f"{n} items: total $ | people min | silent wrongs" for n in SIZES) + + " |", + "|---|" + "---|---|---|" * len(SIZES), + ] + for a in arms: + cells = [] + for n in SIZES: + t, lo, hi = ( + a["totals"][str(n)], + a["totals_people_x0.5"][str(n)], + a["totals_people_x2"][str(n)], + ) + cells += [ + f"{_f(t['total_usd'], 0)} [{_f(lo['total_usd'], 0)}-{_f(hi['total_usd'], 0)}]", + _f(t["people_min"], 0), + _f(t["silent_wrong_expected"], 1), + ] + L.append(f"| {a['arm']} | " + " | ".join(cells) + " |") + if result.get("mockmed_missing"): + L += ["", f"Not yet run: {', '.join(result['mockmed_missing'])}."] + return "\n".join(L) + "\n" + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:])) diff --git a/packages/flow/benchmark/lifecycle/.gitignore b/packages/flow/benchmark/lifecycle/.gitignore new file mode 100644 index 000000000..9ebec926b --- /dev/null +++ b/packages/flow/benchmark/lifecycle/.gitignore @@ -0,0 +1 @@ +results/spend.lock diff --git a/packages/flow/benchmark/lifecycle/README.md b/packages/flow/benchmark/lifecycle/README.md new file mode 100644 index 000000000..328021895 --- /dev/null +++ b/packages/flow/benchmark/lifecycle/README.md @@ -0,0 +1,125 @@ +# Lifecycle soak: does OpenAdapt remove work under ordinary business conditions? + +A "synthetic customer": a seeded stream of referrals is worked through on a +real local OpenEMR 8.0.0.3, with duplicates, bad source documents, other +clerks, forced logouts, slow and failed saves and four UI changes, while a +scripted operator answers every stop. The result is one number with its +parts, computed only from the run's ledger. + +**Verified completed items per operator-hour**: items whose final state the +independent oracle judges right and that the operator did not do by hand, +divided by every operator hour the run charged (setup, per-item requests, +maintenance after UI changes). + +Reported beside it, never folded in: + +- setup minutes (teach, compile, qualify, admit); +- per-item operator minutes (approvals, stops, reconciliations, repair reviews); +- maintenance minutes after each UI change; +- false-stop rate: stops with nothing written on items whose data was right + and that carried no injected fault, over all such items (Wilson 95% CI), and + how many of them happened with a UI change in force; +- silent-wrong rate: items the arm claimed done that the oracle judged wrong, + over all items it claimed done; the oracle is a direct SQL read of + `patient_data` from inside the database container, never the product's + verdict; +- wall time, host load, model calls and list-price cost; +- the same headline if every silent wrong is later found and fixed + (reconcile + re-entry per silent wrong), and the all-manual baseline + (every item entered by hand). + +## The headline + +`results.json` (`arms..headline`) and the top of `REPORT.md` give, per arm: + +- **verified items per operator-minute**: items the oracle judges right that + nobody entered by hand, over every charged operator minute (approvals, + stops, reconciliations, resumes, repair reviews, items done by hand and + maintenance). Shown with and without setup, and at x0.5, x1 and x2 of the + cost model; +- false stops per 100 items, and all stops per 100 items; +- operator minutes per stopped item; +- after each app change: the items it false-stopped, the operator minutes of + maintenance, and the wall time until the next verified item; +- stops by cause, and for duplicate-check stops the identity field the item + shared with an existing chart. + +`--drift KIND@SEQ` replaces the four-change schedule with your own, for +example `--drift relabel_save@26` (proof-15's change: only the save button is +relabeled). + +## Files + +| File | What it is | +|---|---| +| `arm.py` | The interface every arm implements (`lifecycle-arm/1`) | +| `workload.py` | The seeded referral stream, its exception mix and ground truth | +| `apps.py` | The OpenEMR adapter: reset, SQL oracle channel, every injected event | +| `oracle.py` | The independent judge (per item, and the item's final state) | +| `operator.py` | The scripted operator and its decision rules | +| `cost_model.py` | Seconds charged per operator touch, versioned, with its calibration note | +| `metrics.py` | The metric and its parts, from the ledger | +| `report.py` | `results.json`, `REPORT.md` and `stops.jsonl` from the ledger | +| `run.py` | The runner | +| `openadapt_arm.py` | The OpenAdapt arm (CLI and Standard governed modes) | +| `production_arm.py` | The production configuration of `benchmark/lifecycle_proof` (Regulated, `problems_only`, durable, pipelined read-back, repair on) as an arm; `recovery: "relabel"` adds the relabel repair, `duplicate_check: true` admits the program with the match-list duplicate check | +| `fake.py` | An in-memory app and two toy arms, for tests and for checking a new arm | +| `results//` | Committed runs: `ledger.jsonl`, `workload.jsonl`, `results.json`, `REPORT.md`, `stops.jsonl` | + +## Run it + +```bash +cd packages/flow/benchmark +python -m lifecycle.apps up # the lifecycle OpenEMR copy, ports 9420/9421 +python -m lifecycle.run --app openemr --seed 7 --items 300 --days 10 \ + --arm lifecycle.openadapt_arm:make --out lifecycle/results/ +python -m lifecycle.report lifecycle/results/ # recompute from the ledger +``` + +`--app fake --arm lifecycle.fake:careful` runs the whole pipeline in a second +with no fixture. Two runs with the same seed see the same items, events and +UI changes, in the same order. + +## The workload + +Each item is one new-patient referral (the 12 fields of OpenEMR's New +Patient form). About a quarter carry an exception (`workload.py` has the +exact mix and the right outcome of each): duplicates of existing charts, +resent referrals, same-name patients who are and are not the same person, a +missing date of birth, a missing home phone, a US-format date of birth, names +in capitals with stray spaces, another clerk registering the same patient +first, another clerk editing an unrelated chart mid-item, a forced logout, a +slow save and a save that fails once. The mix is stress-weighted to exercise +every kind several times, not a measured intake distribution. + +UI changes are applied in the app server, so every client sees them, and +stay on to the end of the run: a theme (day 2), a relabel of both create +buttons (day 4), a reordered demographics layout (day 6) and a notice that +must be dismissed before the form takes input (day 8). A version bump is not +run: the fixture pins one OpenEMR image. + +Every injected event, with exactly what was changed, is a row in the ledger. + +## The operator + +`operator.py` answers every request from the ledger's truth and from a +look at the app, never from the arm's verdict, and records the rule that +decided. Requests and their charges come from `cost_model.py`. No real +operator timings exist yet, so every charge is an assumption, stated in the +model and shown at x0.5 and x2. + +## Adding an arm + +Implement `arm.Arm` and pass `--arm module:factory`. Route every human touch +through `operator.ask`; never charge yourself and never judge yourself. +`Env.extra` carries the fixture's ports and read-only and actor REST +sessions; arms never get the oracle's SQL. Raise `arm.SpendCapReached` to +pause an arm at a spend cap. + +## Limits + +- One machine, shared with other work; every item row records the 1-minute + load. Wall times are not comparable across runs at different loads. +- Synthetic data, a synthetic workload mix, and assumed operator costs. +- The OpenAdapt arm's configuration, and what the product refused, is in + each run's `results.json` (`arms..describe`) and its report. diff --git a/packages/flow/benchmark/lifecycle/__init__.py b/packages/flow/benchmark/lifecycle/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/packages/flow/benchmark/lifecycle/apps.py b/packages/flow/benchmark/lifecycle/apps.py new file mode 100644 index 000000000..8c11978c8 --- /dev/null +++ b/packages/flow/benchmark/lifecycle/apps.py @@ -0,0 +1,589 @@ +"""The apps a lifecycle run drives, and every event the harness injects into them. + +OpenEMR 8.0.0.3 runs as its own Compose project (``openadapt-openemr-lifecycle``, +loopback ports 9421 for the browser and 9420 for the API) from the pinned +fixture in ``benchmark/openemr_local``, with synthetic data only. Nothing here +touches another session's copy. + +Injected events, each logged with exactly what was changed: + +- other clerks: a chart created through the REST API as the actor account, + or an unrelated chart's home phone changed through SQL; +- forced logout: every PHP session file in the app container is deleted; +- slow save: a ``BEFORE INSERT`` trigger on ``patient_data`` that sleeps for + the item's patient only; +- failed save: the same trigger raising an error, a set number of times (the + counter lives in a non-transactional table, so it survives the rollback); +- UI changes, served to every client as a vendor release would be: + - ``theme``: a stylesheet that changes colours, fonts and spacing on the + New Patient form and its duplicate-check popup; + - ``relabel``: the save button "Confirm Create New Patient" becomes + "Save new patient", and "Create New Patient" becomes "Check and create"; + - ``relabel_save``: only the save button is relabeled (proof-15's change); + - ``reorder``: the demographics layout (``layout_options``) puts Birth Sex + before DOB, and Home Phone and Contact Email before Address; + - ``interstitial``: a notice over the New Patient form that must be + dismissed ("Continue") before the form takes input. + A version bump is not run: the lock pins one OpenEMR image and a second + pinned image would need its own baseline (see the report's limits). +""" + +from __future__ import annotations + +import json +import os +import re +import subprocess +import threading +import time +from pathlib import Path +from typing import Any, Optional + +HERE = Path(__file__).resolve().parent +BENCH = HERE.parent +#: Another session can run its own copy: set these three (and +#: LIFECYCLE_OPENEMR_STATE) to its own project, ports and state folder. +PROJECT = os.environ.get("LIFECYCLE_OPENEMR_PROJECT", "openadapt-openemr-lifecycle") +UI_PORT = os.environ.get("LIFECYCLE_UI_PORT", "9421") +API_PORT = os.environ.get("LIFECYCLE_API_PORT", "9420") +WEBROOT = "/var/www/localhost/htdocs/openemr" +FORM = f"{WEBROOT}/interface/new/new_comprehensive.php" +POPUP = f"{WEBROOT}/interface/new/new_search_popup.php" +DRIFT_JS = f"{WEBROOT}/public/lifecycle-drift.js" +SQL_FIELDS = ( + "id", + "pid", + "title", + "fname", + "lname", + "DOB", + "sex", + "street", + "city", + "state", + "postal_code", + "phone_home", + "email", + "country_code", +) + +THEME_CSS = ( + "body{background:#fbf6ee !important;font-family:Georgia,serif !important;}" + ".card,.card-header,.nav-tabs .nav-link{background:#efe4d2 !important;}" + ".btn-primary{background:#6b3d8f !important;border-color:#6b3d8f !important;}" + "label,.label_custom{color:#5a3218 !important;}" + ".form-control{border-radius:0 !important;border-color:#a08060 !important;}" +) +INTERSTITIAL_JS = r""" +(function(){ + if (!/new_comprehensive\.php/.test(location.pathname)) return; + function show(){ + if (document.getElementById('oa-lifecycle-notice')) return; + var d=document.createElement('div'); + d.id='oa-lifecycle-notice'; + d.setAttribute('role','dialog'); + d.style.cssText='position:fixed;inset:0;background:rgba(0,0,0,.45);z-index:99999;display:flex;align-items:center;justify-content:center'; + d.innerHTML='
'+ + '

Registration form updated

'+ + '

Contact details are now checked when you save. Review the new layout before you continue.

'+ + '
'; + document.body.appendChild(d); + document.getElementById('oa-lifecycle-continue').onclick=function(){d.remove();}; + } + if (document.readyState==='loading') document.addEventListener('DOMContentLoaded',show); else show(); +})(); +""" + + +def podman_env() -> dict[str, str]: + """Docker Compose pointed at the benchmark Podman machine.""" + + env = dict(os.environ) + if not env.get("DOCKER_HOST"): + sock = subprocess.run( + [ + "podman", + "machine", + "inspect", + "openadapt-benchmark", + "--format", + "{{.ConnectionInfo.PodmanSocket.Path}}", + ], + capture_output=True, + text=True, + check=False, + ).stdout.strip() + if sock: + env["DOCKER_HOST"] = f"unix://{sock}" + return env + + +def make_fixture(state_dir: Path) -> Any: + import benchmark.openemr_local.fixture as fx + + class LifecycleFixture(fx.OpenEMRFixture): + def __init__(self, state: Path) -> None: + super().__init__(state) + self.lock = {**self.lock, "project": PROJECT} + + @property + def ui_base_url(self) -> str: + return f"http://127.0.0.1:{UI_PORT}" + + @property + def api_base_url(self) -> str: + return f"https://127.0.0.1:{API_PORT}" + + def _expected(self) -> dict[str, str]: + return { + "OPENEMR_IMAGE": self.lock["services"]["openemr"], + "MARIADB_IMAGE": self.lock["services"]["mariadb"], + "OPENEMR_ACTOR_USER": "openadapt_actor", + "OPENEMR_HTTP_PORT": UI_PORT, + "OPENEMR_HTTPS_PORT": API_PORT, + } + + def _validate_runtime_env(self) -> None: + values = self._runtime_values() + for k, v in self._expected().items(): + if values.get(k) != v: + raise fx.FixtureError(f"runtime.env {k} differs") + self.runtime_env.chmod(0o600) + + def runtime_preflight(self) -> None: + # The shared machine's rules (~/oa/scratch/fixture-lease/README.md): + # at least 15 GiB free and a 1-minute load under 40 before any + # container starts. The pinned images are already in the Podman + # machine. + import shutil + + free = shutil.disk_usage("/System/Volumes/Data").free / 2**30 + if free < 15: + raise fx.FixtureError(f"only {free:.1f} GiB free on the host") + load = os.getloadavg()[0] + if load >= 40: + raise fx.FixtureError(f"1-minute load is {load:.0f}; wait for < 40") + + def prepare(self) -> None: + import secrets + + self._protect_state_dir() + if not self.runtime_env.exists(): + values = { + **self._expected(), + "MARIADB_ROOT_PASSWORD": secrets.token_urlsafe(32), + "OPENEMR_DB_PASSWORD": secrets.token_urlsafe(32), + "OPENEMR_ACTOR_PASSWORD": secrets.token_urlsafe(32), + } + self._write_private_exclusive( + self.runtime_env, + ( + "# generated local synthetic fixture; never commit\n" + + "".join(f"{k}={v}\n" for k, v in values.items()) + ).encode(), + ) + super().prepare() + + def _run(self, argv, *, input_bytes=None, timeout_s=300.0): # type: ignore[override] + # Same as the base, with Compose pointed at the Podman machine. + if argv[:2] == ["docker", "compose"] or argv[:1] == ["docker"]: + proc = subprocess.run( + argv, + input=input_bytes, + capture_output=True, + timeout=timeout_s, + env=podman_env(), + check=False, + ) + if proc.returncode != 0: + raise fx.FixtureError( + f"{' '.join(argv[:4])} failed: {proc.stderr.decode()[-800:]}" + ) + return proc + return super()._run(argv, input_bytes=input_bytes, timeout_s=timeout_s) + + return LifecycleFixture(state_dir) + + +class OpenEMRApp: + """The app adapter the harness drives: reset, read, inject, change.""" + + def __init__(self, state_dir: Optional[Path] = None) -> None: + self.state_dir = Path( + state_dir + or os.environ.get("LIFECYCLE_OPENEMR_STATE") + or Path.home() / "oa" / "scratch" / "lifecycle" / "openemr-state" + ) + self.fx = make_fixture(self.state_dir) + self.db_container = f"{PROJECT}-db-1" + self.app_container = f"{PROJECT}-openemr-1" + self.ui_url = self.fx.ui_base_url + self.api_url = self.fx.api_base_url + self.drift_on: list[str] = [] + self._fault_lock = threading.Lock() + from benchmark.openemr_throughput.harness import CLERKS + + #: Added before the baseline snapshot, so every reset keeps them. + self._clerks: list[str] = list(CLERKS[:3]) + + # -- bring-up ---------------------------------------------------------- + + def bootstrapped(self) -> bool: + return all( + p.is_file() + for p in ( + self.fx.runtime_env, + self.fx.clients_path, + self.fx.snapshot_path, + self.fx.snapshot_hash_path, + ) + ) + + def bring_up(self, say: Any = print) -> str: + if self.bootstrapped(): + say("starting the existing copy") + self.fx._compose("up", "-d", timeout_s=900.0) + self.fx._wait_ready(timeout_s=900.0) + return "restarted" + self.fx.runtime_preflight() + say("checking the pinned OpenEMR release") + self.fx.prepare() + self.fx.image_identity() + say("starting; OpenEMR installs itself on first start") + self.fx._compose("up", "-d", timeout_s=900.0) + self.fx._wait_ready(timeout_s=1800.0) + self.fx._verify_running_source() + say("adding synthetic API clients") + self.fx.bootstrap() + self.add_clerks(3) + self.fx.snapshot() + return "created" + + def describe(self) -> dict[str, Any]: + return { + "app": "openemr", + "version": "8.0.0.3", + "project": PROJECT, + "ui_url": self.ui_url, + "api_url": self.api_url, + "images": self.fx.lock["services"], + "baseline_sha256": self.fx.baseline_hash() if self.bootstrapped() else None, + } + + def extra(self) -> dict[str, Any]: + """What arms may use. ``rest_read_session()`` returns an httpx client + with a read-only bearer token (``user/patient.rs``), the same REST + read-back OpenAdapt's record check uses; ``rest_write_session()`` one + with the actor's create/update scope. Arms never get the oracle's SQL.""" + + return { + "fixture_state": str(self.state_dir), + "fixture_lock": str(self.state_dir / "environment.lock.json"), + "project": PROJECT, + "ui_port": int(UI_PORT), + "api_port": int(API_PORT), + "rest_read_session": lambda: self.fx.token_session("oracle"), + "rest_write_session": lambda: self.fx.token_session("actor"), + "fixture": self.fx, + } + + def point_hooks_here(self) -> None: + """verify.yaml's hooks (sign-in, REST read-back) read these variables.""" + + lock = self.state_dir / "environment.lock.json" + lock.write_text(json.dumps(self.fx.lock, indent=2) + "\n") + os.environ["OPENEMR_FIXTURE_STATE"] = str(self.state_dir) + os.environ["OPENEMR_FIXTURE_LOCK"] = str(lock) + os.environ["OPENEMR_UI_URL"] = self.ui_url + os.environ["OPENEMR_API_URL"] = self.api_url + os.environ.update({k: v for k, v in podman_env().items() if k == "DOCKER_HOST"}) + + # -- accounts ------------------------------------------------------------ + + def add_clerks(self, n: int) -> list[str]: + from benchmark.openemr_throughput.harness import add_clerks + + self._clerks = add_clerks(self.fx, n) + return self._clerks + + def accounts(self) -> list[tuple[str, str]]: + pw = self.fx._runtime_values()["OPENEMR_ACTOR_PASSWORD"] + users = ["openadapt_actor"] + [ + c for c in self._clerks if c != "openadapt_actor" + ] + return [(u, pw) for u in users] + + # -- direct database channel (the oracle's) ----------------------------- + + def sql(self, statement: str, *, timeout_s: float = 120.0) -> str: + proc = subprocess.run( + [ + "podman", + "exec", + "-i", + self.db_container, + "sh", + "-c", + 'MYSQL_PWD="$MARIADB_ROOT_PASSWORD" exec mariadb -uroot --batch --raw ' + "--skip-column-names", + ], + input=statement.encode(), + capture_output=True, + timeout=timeout_s, + check=False, + ) + if proc.returncode != 0: + raise RuntimeError(f"sql failed: {proc.stderr.decode()[-500:]}") + return proc.stdout.decode() + + def rows(self) -> list[dict[str, str]]: + out = self.sql( + "SELECT " + ",".join(SQL_FIELDS) + " FROM openemr.patient_data ORDER BY id;" + ) + return [ + { + k: ("" if v == "NULL" else v) + for k, v in zip(SQL_FIELDS, line.split("\t")) + } + for line in out.splitlines() + if line + ] + + def app_exec(self, script: str, *, timeout_s: float = 60.0) -> str: + proc = subprocess.run( + ["podman", "exec", self.app_container, "sh", "-c", script], + capture_output=True, + timeout=timeout_s, + check=False, + ) + if proc.returncode != 0: + raise RuntimeError(f"app exec failed: {proc.stderr.decode()[-500:]}") + return proc.stdout.decode() + + # -- state --------------------------------------------------------------- + + def reset(self) -> str: + self.clear_all_events() + return self.fx.reset() + + def seed_existing(self, people: list[dict[str, str]]) -> list[str]: + return [self.create_chart(p, who="seed") for p in people] + + def create_chart(self, fields: dict[str, str], *, who: str) -> str: + """A chart created through the REST API (another clerk, the operator + doing an item by hand, or the seed). Returns its patient_data id.""" + + body = { + "title": fields["title"], + "fname": fields["fname"].strip(), + "lname": fields["lname"].strip(), + "DOB": fields["DOB"], + "sex": fields["sex"], + "street": fields["street"], + "city": fields["city"], + "state": "MA", + "postal_code": fields["postal_code"], + "phone_home": fields.get("phone_home", ""), + "email": fields["email"], + "country_code": fields.get("country_code", "USA"), + } + before = {r["id"] for r in self.rows()} + session = self.fx.token_session("actor") + try: + resp = session.post(f"{self.api_url}/apis/default/api/patient", json=body) + finally: + session.close() + if resp.status_code not in (200, 201): + raise RuntimeError(f"{who} create returned HTTP {resp.status_code}") + new = [r["id"] for r in self.rows() if r["id"] not in before] + if len(new) != 1: + raise RuntimeError(f"{who} create made {len(new)} rows") + return new[0] + + # -- item-level events ------------------------------------------------------ + + def inject( + self, + inj: dict[str, Any], + truth: Any, + *, + harness: set[str], + stop: Optional[threading.Event] = None, + ) -> dict[str, Any]: + event = inj["event"] + if event == "other_clerk_creates": + cid = self.create_chart(inj["person"], who="other clerk") + return {"event": event, "chart_id": cid} + if event == "force_logout": + n = self.app_exec( + "n=$(ls /tmp/sess_* 2>/dev/null | wc -l); rm -f /tmp/sess_*; echo $n" + ) + return {"event": event, "sessions_ended": int(n.strip() or 0)} + if event == "other_clerk_edits_unrelated": + # wait a moment so the edit lands while the item is under way + if stop is not None: + stop.wait(5.0) + rows = [r for r in self.rows() if r["id"] not in harness] + target = rows[truth.seq % len(rows)] if rows else None + if target is None: + return {"event": event, "touched_ids": []} + phone = f"555-01{truth.seq % 100:02d}" + self.sql( + f"UPDATE openemr.patient_data SET phone_home='{phone}' " + f"WHERE id={int(target['id'])};" + ) + return {"event": event, "touched_ids": [target["id"]], "phone_home": phone} + if event in ("slow_save", "failed_save"): + email = truth.person["email"] + assert re.fullmatch(r"[a-z0-9.@-]+", email) + if event == "slow_save": + body = f"IF NEW.email = '{email}' THEN SET @oa_slept = SLEEP({int(inj['seconds'])}); END IF;" + else: + body = ( + f"IF NEW.email = '{email}' AND (SELECT n FROM lifecycle.fault_budget " + f"WHERE k='{email}') > 0 THEN " + f"UPDATE lifecycle.fault_budget SET n = n - 1 WHERE k='{email}'; " + "SIGNAL SQLSTATE '45000' SET MESSAGE_TEXT = 'lifecycle: injected save failure'; " + "END IF;" + ) + self.sql( + "CREATE DATABASE IF NOT EXISTS lifecycle; " + "CREATE TABLE IF NOT EXISTS lifecycle.fault_budget " + "(k VARCHAR(190) PRIMARY KEY, n INT) ENGINE=Aria TRANSACTIONAL=0; " + f"REPLACE INTO lifecycle.fault_budget VALUES ('{email}', {int(inj.get('times', 0))}); " + "DROP TRIGGER IF EXISTS openemr.oa_lifecycle_fault;\n" + "DELIMITER //\n" + "CREATE TRIGGER openemr.oa_lifecycle_fault BEFORE INSERT ON openemr.patient_data " + f"FOR EACH ROW BEGIN {body} END//\n" + "DELIMITER ;\n" + ) + return {"event": event, "trigger": body} + raise ValueError(f"unknown injection {event}") + + def clear_item_events(self) -> list[str]: + out = self.sql( + "SELECT TRIGGER_NAME FROM information_schema.TRIGGERS " + "WHERE TRIGGER_NAME='oa_lifecycle_fault';" + ) + if out.strip(): + self.sql("DROP TRIGGER IF EXISTS openemr.oa_lifecycle_fault;") + return ["oa_lifecycle_fault"] + return [] + + # -- UI changes ---------------------------------------------------------------- + + def _ensure_originals(self) -> None: + self.app_exec( + f"for f in {FORM} {POPUP}; do [ -f $f.oa-orig ] || cp -p $f $f.oa-orig; done" + ) + + def _write_drift_js(self) -> None: + parts = [] + if "theme" in self.drift_on: + parts.append( + "(function(){var s=document.createElement('style');s.textContent=" + + json.dumps(THEME_CSS) + + ";(document.head||document.documentElement).appendChild(s);})();" + ) + if "interstitial" in self.drift_on: + parts.append(INTERSTITIAL_JS) + js = "\n".join(parts) + self.app_exec(f"cat > {DRIFT_JS} <<'OAEOF'\n{js}\nOAEOF\nchmod 644 {DRIFT_JS}") + + def apply_drift(self, kind: str) -> dict[str, Any]: + self._ensure_originals() + if kind not in self.drift_on: + self.drift_on.append(kind) + changed: dict[str, Any] = {"kind": kind} + if kind in ("theme", "interstitial"): + self._write_drift_js() + tag = '' + for f in (FORM, POPUP): + self.app_exec( + f"grep -q lifecycle-drift.js {f} || " + f"sed -i 's##{tag}#' {f}" + ) + changed["files"] = [FORM, POPUP, DRIFT_JS] + elif kind == "relabel": + self.app_exec( + f"sed -i \"s/xla('Confirm Create New Patient')/xla('Save new patient')/\" {POPUP}; " + f"sed -i \"s/xlt('Create New Patient')/xlt('Check and create')/; " + f"s/xla('Create New Patient')/xla('Check and create')/; " + f"s/xlj('Create New Patient')/xlj('Check and create')/\" {FORM}" + ) + changed["labels"] = { + "Confirm Create New Patient": "Save new patient", + "Create New Patient": "Check and create", + } + elif kind == "relabel_save": + # proof-15's app change: only the save button gets a new label. + self.app_exec( + f"sed -i \"s/xla('Confirm Create New Patient')/xla('Save new patient')/\" {POPUP}" + ) + changed["labels"] = {"Confirm Create New Patient": "Save new patient"} + elif kind == "reorder": + sql = ( + "UPDATE openemr.layout_options SET seq=85 WHERE form_id='DEM' AND field_id='sex';" + "UPDATE openemr.layout_options SET seq=0 WHERE form_id='DEM' AND field_id='phone_home';" + "UPDATE openemr.layout_options SET seq=0 WHERE form_id='DEM' AND field_id='email';" + ) + self.sql(sql) + changed["sql"] = sql + else: + raise ValueError(f"unknown UI change {kind}") + return changed + + def clear_all_events(self) -> None: + self.drift_on = [] + try: + self.app_exec( + f"for f in {FORM} {POPUP}; do [ -f $f.oa-orig ] && cp -p $f.oa-orig $f; done; " + f"rm -f {DRIFT_JS}; true" + ) + except (RuntimeError, subprocess.SubprocessError): + pass + try: + self.clear_item_events() + self.sql( + "UPDATE openemr.layout_options SET seq=100 WHERE form_id='DEM' AND field_id='sex';" + "UPDATE openemr.layout_options SET seq=11 WHERE form_id='DEM' AND field_id='phone_home';" + "UPDATE openemr.layout_options SET seq=14 WHERE form_id='DEM' AND field_id='email';" + ) + except (RuntimeError, subprocess.SubprocessError): + pass + + +def make_app(name: str) -> Any: + if name == "fake": + from .fake import FakeApp + + return FakeApp() + if name == "openemr": + app = OpenEMRApp() + app.point_hooks_here() + return app + raise ValueError(f"unknown app {name}") + + +def main() -> int: + """Bring the lifecycle OpenEMR copy up (and set it up on first use).""" + + import argparse + + p = argparse.ArgumentParser(description=main.__doc__) + p.add_argument("command", choices=("up", "status", "down")) + args = p.parse_args() + app = OpenEMRApp() + if args.command == "up": + t = time.monotonic() + how = app.bring_up() + print(json.dumps({"how": how, "seconds": round(time.monotonic() - t, 1)})) + elif args.command == "status": + print(json.dumps({"bootstrapped": app.bootstrapped(), "ui": app.ui_url})) + else: + app.fx._compose("stop", timeout_s=300.0) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/packages/flow/benchmark/lifecycle/arm.py b/packages/flow/benchmark/lifecycle/arm.py new file mode 100644 index 000000000..3a1e1fecb --- /dev/null +++ b/packages/flow/benchmark/lifecycle/arm.py @@ -0,0 +1,146 @@ +"""The interface every comparison arm implements. + +The harness owns the worklist and its ground truth, the app and every injected +event, the independent oracle, the simulated operator and the cost model. An +arm owns only how it sets itself up and how it does one item. + +Two rules keep arms comparable: + +- An arm never charges itself operator time. Every human touch (an approval, + a stop, a question, a reconciliation, a repair review, an item done by hand) + goes through ``operator.ask``, which returns the decision and the seconds the + versioned cost model charges for it. +- An arm never judges itself. ``ItemResult.claimed`` is what the arm says; the + oracle reads the system of record after every item and decides. + +Arms are loaded from ``module:factory`` strings; ``factory(options) -> Arm``. +""" + +from __future__ import annotations + +import importlib +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any, Literal, Optional, Protocol + +INTERFACE_VERSION = "lifecycle-arm/1" + +RequestKind = Literal[ + "approval", "stop", "reconcile", "repair_review", "question", "manual_entry" +] + + +@dataclass(frozen=True) +class WorkItem: + """One unit of work as the source document states it. + + ``params`` may be missing fields or hold malformed values: they are what + the referral says, not what is true. The harness keeps the truth. + """ + + seq: int + item_id: str + day: int + app: str + params: dict[str, str] + source_text: str + + +@dataclass(frozen=True) +class Env: + app: str + ui_url: str + api_url: Optional[str] + #: Synthetic (user, password) pairs the arm may sign in with. + accounts: tuple[tuple[str, str], ...] + #: The arm's private scratch directory for this run. + workdir: Path + seed: int + #: App-specific extras (for example the fixture's container names). + extra: dict[str, Any] = field(default_factory=dict) + + +@dataclass(frozen=True) +class OperatorRequest: + kind: RequestKind + item_id: Optional[str] + #: The product's own reason code, or the agent's question, verbatim. + reason: str + #: Where the request surfaced, e.g. "cli:openadapt-flow durable resume". + surface: str + detail: dict[str, Any] = field(default_factory=dict) + + +@dataclass(frozen=True) +class OperatorDecision: + #: approve | deny | resume | skip | do_by_hand | accept_repair | + #: reject_repair | answer | confirm_saved | confirm_not_saved + action: str + charged_s: float + #: Which scripted-policy rule decided it. + rule: str + answer: Optional[str] = None + + +class Operator(Protocol): + def ask(self, request: OperatorRequest) -> OperatorDecision: ... + + +@dataclass +class ItemResult: + item_id: str + claimed: Literal["done", "stopped", "error"] + #: The product's final status or the agent's last line, verbatim. + product_status: str + #: The arm's own wall time for the item (first action to return). + wall_s: float + model_calls: int = 0 + cost_usd: float = 0.0 + #: Evidence path relative to the run directory. + evidence: str = "" + #: Maintenance the arm did inside this item (for example re-authoring a + #: script after it failed on a changed screen). Same shape as a setup + #: phase; charged as maintenance, not as per-item work. + phases: list[dict[str, Any]] = field(default_factory=list) + #: Free-form, kept in the ledger. ``detail["usage"]`` may hold a token + #: breakdown: [{"model", "input", "output", "cache_read", "cache_write", + #: "calls", "usd"}]. + detail: dict[str, Any] = field(default_factory=dict) + + +class Arm(Protocol): + name: str + + def describe(self) -> dict[str, Any]: + """The exact configuration, recorded in the ledger as is.""" + + def setup(self, env: Env, operator: Operator) -> list[dict[str, Any]]: + """Teach, compile, qualify, admit. One dict per phase: + ``{"phase", "wall_s", "attended_s", "model_calls", "cost_usd"}``. + ``attended_s`` is the part of the wall time a person must watch.""" + + def run_item(self, item: WorkItem, env: Env, operator: Operator) -> ItemResult: ... + + def after_drift( + self, event: dict[str, Any], env: Env, operator: Operator + ) -> list[dict[str, Any]]: + """Maintenance an arm does on its own initiative after a UI change it + was told about (most arms are not told, and return []).""" + + def close(self) -> None: ... + + +class SpendCapReached(RuntimeError): + """Raised by an arm before a model call that could cross a spend cap. The + runner stops that arm, records the rest of its items as paused, and goes + on with the next arm.""" + + +def load_arm(spec: str, options: Optional[dict[str, Any]] = None) -> Arm: + """Build an arm from ``module:factory``.""" + + module, _, name = spec.partition(":") + if not module or not name: + raise ValueError(f"arm must look like module:factory, got {spec!r}") + factory = getattr(importlib.import_module(module), name) + return factory(dict(options or {})) diff --git a/packages/flow/benchmark/lifecycle/arms/agent_script.py b/packages/flow/benchmark/lifecycle/arms/agent_script.py new file mode 100644 index 000000000..3d6d934ed --- /dev/null +++ b/packages/flow/benchmark/lifecycle/arms/agent_script.py @@ -0,0 +1,373 @@ +"""Arm ``agent_script``: an agent writes a Playwright + REST script, then maintains it. + +Setup: the agent gets the same demonstration and documents as OpenAdapt (the +recorded steps, the procedure, the REST read) and the live form to inspect. It +writes a script with an equivalent check (a REST read-back after the save, and +a duplicate check before it), tries it in dry runs (the save request is +blocked at the network layer), and submits it. A person then reviews the script +once before it goes live: the setup phase is named ``compile`` so the cost +model charges it exactly what it charges for reviewing a compiled program. + +Per item: the script runs with no model calls. A business stop (duplicate, +missing date of birth) goes to the operator. When the script breaks (an +exception, or a reason starting "UNEXPECTED"), the agent is called again with +the script, the failure and the live form, writes a fix, dry-runs it on this +item, and the operator reviews the fix (``repair_review``) before the item is +run again. That maintenance is charged as maintenance, with its tokens and +the review, not as per-item cost. +""" + +from __future__ import annotations + +import json +import time +from pathlib import Path +from typing import Any, Optional + +from ..arm import Env, ItemResult, Operator, OperatorRequest, WorkItem +from . import budget, scripts +from .agent_tools import READ_JS +from .openemr_ui import SOP, OpenEMRSession, RestReader + +CONTRACT = '''\ +Write one Python module that defines + + def register(ctx, referral: dict) -> dict + +returning {"result": "SAVED" | "NOT SAVED", "reason": ""}. +referral has the keys title, fname, lname, DOB, sex, street, city, +state_label, postal_code, phone_home, email, country_code, exactly as the +referral states them (values may be missing, blank or malformed). +ctx.page is a Playwright sync Page (signed in, on a blank New Patient form); +ctx.form is the form's Frame; ctx.find_patients(fname=, lname=, DOB=) is the +REST search; ctx.commit is False in a dry run; ctx.log(msg) records a line. +Only re, time, datetime and json may be imported; no file, OS or eval access. +Required behaviour: +- Apply the procedure's rules (normalise names and dates; stop on a missing + date of birth; check for an existing chart before saving; never create a + duplicate). +- After the confirmed save, read the chart back with ctx.find_patients and + return SAVED only if exactly one chart matches the referral's values. +- In a dry run (ctx.commit is False), do everything including the confirm + click (the save request is blocked for you), skip the read-back and return + {"result": "NOT SAVED", "reason": "DRY RUN reached confirm"}. +- When the screen is not what the script expects (a control or dialog is + missing, a wait times out), return NOT SAVED with a reason starting + "UNEXPECTED:" rather than guessing.''' + +SYSTEM = """\ +You are a careful automation engineer. You write and maintain a Playwright \ +script that registers referred patients in a clinic's OpenEMR, against a local \ +synthetic test copy. Use the tools to inspect the live form, write the script, \ +and dry-run it. Submit only a script whose dry runs pass. Be economical: \ +a few dry runs, then submit.""" + +TOOLS: list[dict[str, Any]] = [ + { + "name": "read_screen", + "description": "Open a blank New Patient form and list the visible controls, buttons and dialog text of every frame.", + "input_schema": {"type": "object", "properties": {}, "additionalProperties": False}, + }, + { + "name": "find_patients", + "description": "REST search of existing charts by fname, lname, DOB.", + "input_schema": { + "type": "object", + "properties": {"fname": {"type": "string"}, "lname": {"type": "string"}, "DOB": {"type": "string"}}, + "additionalProperties": False, + }, + }, + { + "name": "dry_run", + "description": "Run the given script on a fresh blank form with the save request blocked. Returns its result, reason, log, and the controls on screen at the end.", + "input_schema": { + "type": "object", + "properties": { + "source": {"type": "string", "description": "the whole module"}, + "referral": {"type": "object", "description": "a synthetic referral dict to try"}, + }, + "required": ["source", "referral"], + "additionalProperties": False, + }, + }, + { + "name": "submit", + "description": "Submit the final script (the whole module).", + "input_schema": { + "type": "object", + "properties": {"source": {"type": "string"}, "notes": {"type": "string"}}, + "required": ["source"], + "additionalProperties": False, + }, + }, +] + + +class AgentScriptArm: + name = "agent_script" + + def __init__(self, options: dict[str, Any]) -> None: + self.name = str(options.get("name", self.name)) + self.model = options.get("model", "claude-opus-5-5") + self.effort = options.get("effort", "high") + self.author_budget_usd = float(options.get("author_budget_usd", 4.0)) + self.fix_budget_usd = float(options.get("fix_budget_usd", 2.0)) + self.max_turns = int(options.get("max_turns", 30)) + self.headless = bool(options.get("headless", True)) + self.session: Optional[OpenEMRSession] = None + self.rest: Optional[RestReader] = None + self.source = "" + self.version = 0 + self.dir: Optional[Path] = None + + def describe(self) -> dict[str, Any]: + return { + "arm": self.name, + "kind": "agent-written Playwright + REST script with read-back; agent maintains it on failure", + "model": self.model, + "effort": self.effort, + "author_budget_usd": self.author_budget_usd, + "fix_budget_usd": self.fix_budget_usd, + "prices_as_of": budget.prices()["as_of"], + "per_item_model_calls": 0, + "sandbox": "static import/name check; dry runs block the save request", + } + + # -- authoring and maintenance --------------------------------------------------- + + def _read_screen(self) -> dict[str, Any]: + self.session.blank_form() + frames = [] + n = 1 + for frame in self.session.page.frames: + try: + got = frame.evaluate(READ_JS, n) + except Exception: # noqa: BLE001 + continue + n = got.pop("next") + if got["controls"] or got["buttons"] or got.get("notices"): + frames.append({"frame": (frame.url or "").split("?")[0].rsplit("/", 1)[-1], **got}) + return {"frames": frames} + + def _end_screen(self) -> list[dict[str, Any]]: + out = [] + for frame in self.session.page.frames: + try: + got = frame.evaluate(READ_JS, 1) + except Exception: # noqa: BLE001 + continue + got.pop("next", None) + if got["buttons"] or got.get("notices"): + out.append({"frame": (frame.url or "").split("?")[0].rsplit("/", 1)[-1], + "buttons": got["buttons"][:30], "notices": got.get("notices", [])}) + return out + + def _session_loop(self, opening: str, meter: budget.Meter, trace: list[dict[str, Any]]) -> tuple[Optional[str], list[dict[str, Any]]]: + """Run the authoring agent until it submits. Returns (source, dry runs).""" + + client = budget.client(self.model) + messages: list[dict[str, Any]] = [{"role": "user", "content": opening}] + dry_runs: list[dict[str, Any]] = [] + for turn in range(1, self.max_turns + 1): + approx_in = 8000 + 3000 * turn + meter.before_call(self.model, budget.max_call_usd(self.model, approx_in, 8000)) + with client.messages.stream( + model=self.model, + max_tokens=32000, + system=SYSTEM, + tools=TOOLS, + output_config={"effort": self.effort}, + cache_control={"type": "ephemeral"}, + messages=messages, + ) as stream: + response = stream.get_final_message() + meter.record(self.model, response.usage) + messages.append({"role": "assistant", "content": response.content}) + uses = [b for b in response.content if b.type == "tool_use"] + trace.append({"turn": turn, "stop": response.stop_reason, "tools": [b.name for b in uses]}) + if response.stop_reason == "refusal": + return None, dry_runs + if not uses: + messages.append({"role": "user", "content": "Use the tools; finish with submit."}) + continue + results = [] + for b in uses: + args = dict(b.input) + if b.name == "submit": + try: + scripts.check_source(args["source"]) + except Exception as exc: # noqa: BLE001 + results.append({"type": "tool_result", "tool_use_id": b.id, "is_error": True, + "content": f"rejected: {exc}"}) + continue + return args["source"], dry_runs + try: + if b.name == "read_screen": + out: Any = self._read_screen() + elif b.name == "find_patients": + out = self.rest.find(**args) + elif b.name == "dry_run": + r = scripts.run(args["source"], self.session, self.rest, args.get("referral") or {}, commit=False) + out = {"result": r.result, "reason": r.reason, "log": r.log[-30:], + "save_requests_blocked": r.blocked_saves, "seconds": round(r.seconds, 1), + "screen_at_end": self._end_screen()} + dry_runs.append({k: out[k] for k in ("result", "reason", "save_requests_blocked")}) + else: + raise KeyError(b.name) + content = json.dumps(out, default=str)[:30_000] + results.append({"type": "tool_result", "tool_use_id": b.id, "content": content}) + except Exception as exc: # noqa: BLE001 + results.append({"type": "tool_result", "tool_use_id": b.id, "is_error": True, + "content": f"Error: {exc}"[:600]}) + messages.append({"role": "user", "content": results}) + return None, dry_runs + + def _save_version(self, source: str, why: str) -> None: + self.version += 1 + self.source = source + assert self.dir is not None + (self.dir / f"script_v{self.version}.py").write_text(f"# {why}\n" + source) + + def setup(self, env: Env, operator: Operator) -> list[dict[str, Any]]: + self.session = OpenEMRSession(env, headless=self.headless) + self.rest = RestReader(env) + self.dir = Path(env.workdir) / self.name + self.dir.mkdir(parents=True, exist_ok=True) + self.session.start() + self.session.sign_in() + meter = budget.Meter(f"{self.name} author", self.author_budget_usd) + trace: list[dict[str, Any]] = [] + t0 = time.time() + opening = "\n\n".join([ + "Write the registration script.", SOP, scripts.DEMONSTRATION, scripts.API_DOC, CONTRACT, + ]) + source, dry = None, [] + error = "" + try: + source, dry = self._session_loop(opening, meter, trace) + except budget.SpendCapReached: + raise + except Exception as exc: # noqa: BLE001 + error = f"{type(exc).__name__}: {exc}"[:500] + wall = time.time() - t0 + (self.dir / "author_trace.json").write_text(json.dumps({"trace": trace, "dry_runs": dry, "error": error}, indent=1)) + if source is None: + raise RuntimeError(f"the agent did not submit a script: {error or 'no submit'}") + self._save_version(source, "authored by the agent at setup") + return [ + { + "phase": "author_script", + "wall_s": round(wall, 2), + "attended_s": 0.0, + **budget.summarize(meter), + "detail": {"dry_runs": dry, "version": self.version}, + }, + { + # The person reads the script once before it goes live; charged + # like the review of a compiled program. + "phase": "compile", + "wall_s": 0.0, + "attended_s": 0.0, + "model_calls": 0, + "cost_usd": 0.0, + "detail": {"what": "a person reviews the generated script before it goes live"}, + }, + ] + + def _maintain(self, item: WorkItem, failure: scripts.RunOutcome, operator: Operator) -> tuple[bool, dict[str, Any]]: + """Ask the agent for a fix; the operator reviews it. Returns (accepted, phase).""" + + meter = budget.Meter(f"{self.name} fix {item.item_id}", self.fix_budget_usd) + trace: list[dict[str, Any]] = [] + t0 = time.time() + opening = "\n\n".join([ + "The live script failed on a referral. Fix it, dry-run the fix on this referral, and submit the whole module.", + "Failure:\n" + failure.reason + "\nScript log:\n" + "\n".join(failure.log[-30:]), + "The referral:\n" + json.dumps(item.params), + "The current script:\n```python\n" + self.source + "\n```", + CONTRACT, SOP, + ]) + source, dry, error = None, [], "" + try: + source, dry = self._session_loop(opening, meter, trace) + except budget.SpendCapReached: + raise + except Exception as exc: # noqa: BLE001 + error = f"{type(exc).__name__}: {exc}"[:500] + probes_ok = bool(source) and bool(dry) and dry[-1]["result"] != "ERROR" and not dry[-1]["reason"].upper().startswith("UNEXPECTED") + decision = operator.ask( + OperatorRequest( + kind="repair_review", + item_id=item.item_id, + reason=f"script fix after: {failure.reason.splitlines()[-1][:200] if failure.reason else ''}", + surface="agent_script maintenance", + detail={"probes_passed": probes_ok, "dry_runs": dry[-3:], "error": error}, + ) + ) + accepted = source is not None and decision.action == "accept_repair" + if accepted: + self._save_version(source, f"fix after {item.item_id}: {failure.reason.splitlines()[-1][:120]}") + phase = { + "phase": "fix_script", + "item_id": item.item_id, + "wall_s": round(time.time() - t0, 2), + "attended_s": 0.0, + **budget.summarize(meter), + "detail": {"accepted": accepted, "probes_passed": probes_ok, "dry_runs": dry, + "version": self.version, "error": error, "turns": len(trace)}, + } + return accepted, phase + + # -- per item ------------------------------------------------------------------- + + def run_item(self, item: WorkItem, env: Env, operator: Operator) -> ItemResult: + assert self.session is not None and self.rest is not None + t0 = time.time() + phases: list[dict[str, Any]] = [] + attempts: list[dict[str, Any]] = [] + out = scripts.run(self.source, self.session, self.rest, item.params, commit=True) + attempts.append({"version": self.version, "result": out.result, "reason": out.reason[-300:], "s": round(out.seconds, 2)}) + if out.broke: + accepted, phase = self._maintain(item, out, operator) + phases.append(phase) + if accepted: + out = scripts.run(self.source, self.session, self.rest, item.params, commit=True) + attempts.append({"version": self.version, "result": out.result, "reason": out.reason[-300:], "s": round(out.seconds, 2)}) + wall = time.time() - t0 + if out.result == "SAVED": + claimed, status = "done", f"SAVED: {out.reason}" + elif out.result == "NOT SAVED": + claimed, status = "stopped", f"NOT SAVED: {out.reason}" + else: + claimed, status = "error", out.reason[-300:] + if claimed != "done": + operator.ask(OperatorRequest(kind="stop", item_id=item.item_id, reason=status[:300], surface="script result")) + maint_cost = sum(p.get("cost_usd", 0.0) for p in phases) + return ItemResult( + item_id=item.item_id, + claimed=claimed, + product_status=status[:500], + wall_s=round(wall, 2), + model_calls=0, + cost_usd=0.0, + evidence=f"{self.name}/", + phases=phases, + detail={ + "attempts": attempts, + "script_version": self.version, + "maintenance_cost_usd": round(maint_cost, 6), + "usage": [], + "compute_usd": round(budget.compute_usd(sum(a["s"] for a in attempts)), 6), + }, + ) + + def after_drift(self, event: dict[str, Any], env: Env, operator: Operator) -> list[dict[str, Any]]: + return [] + + def close(self) -> None: + if self.session is not None: + self.session.close() + + +def factory(options: dict[str, Any]) -> AgentScriptArm: + return AgentScriptArm(options) diff --git a/packages/flow/benchmark/lifecycle/arms/agent_tools.py b/packages/flow/benchmark/lifecycle/arms/agent_tools.py new file mode 100644 index 000000000..06252c16e --- /dev/null +++ b/packages/flow/benchmark/lifecycle/arms/agent_tools.py @@ -0,0 +1,369 @@ +"""Arm ``agent_tools``: the same frontier model with structured tools and read-back. + +This is the strong agent, not a strawman. Instead of screenshots it gets what +OpenAdapt's runtime gets: + +- ``read_screen``: the visible controls of every frame (the DOM and + accessibility view): id, label, type, value, required, options; the visible + buttons and links; the text of any dialog or alert, including OpenEMR's + possible-duplicate popup. +- ``set_field`` / ``click``: act on a control by the reference ``read_screen`` + gave it. +- ``find_patients``: a REST search of the record (the clerk's own API access), + so it can check for an existing chart before it saves and confirm the chart + after it saves. +- ``finish``: its claim, which the oracle checks independently. + +Same procedure, model, effort and per-item budget style as ``cua_alone``. +""" + +from __future__ import annotations + +import json +import time +from pathlib import Path +from typing import Any + +from ..arm import Env, ItemResult, Operator, OperatorRequest, WorkItem +from . import budget +from .openemr_ui import SOP, OpenEMRSession, RestReader + +SYSTEM = """\ +You register patients in a clinic's OpenEMR through tools. You cannot see the \ +screen; read_screen gives you its controls and any dialog text. References from \ +read_screen are valid until the next read_screen. After a click that saves or \ +opens a dialog, call read_screen again. Use find_patients to look the person up \ +in the record before you save and to confirm the chart after you save. Call \ +finish exactly once at the end.""" + +TOOLS: list[dict[str, Any]] = [ + { + "name": "read_screen", + "description": "List the visible form controls, buttons and dialog text on the OpenEMR page, with a ref for each control.", + "input_schema": {"type": "object", "properties": {}, "additionalProperties": False}, + }, + { + "name": "set_field", + "description": "Set a text field, date field or select list (by its visible option text) to a value.", + "input_schema": { + "type": "object", + "properties": {"ref": {"type": "string"}, "value": {"type": "string"}}, + "required": ["ref", "value"], + "additionalProperties": False, + }, + "strict": True, + }, + { + "name": "click", + "description": "Click a button, link or checkbox by its ref.", + "input_schema": { + "type": "object", + "properties": {"ref": {"type": "string"}}, + "required": ["ref"], + "additionalProperties": False, + }, + "strict": True, + }, + { + "name": "click_text", + "description": "Click the first visible element whose text is exactly this (any frame), e.g. a section header.", + "input_schema": { + "type": "object", + "properties": {"text": {"type": "string"}}, + "required": ["text"], + "additionalProperties": False, + }, + "strict": True, + }, + { + "name": "find_patients", + "description": "Search existing patient charts through OpenEMR's REST API. Any of fname, lname, DOB (YYYY-MM-DD). Returns matching charts.", + "input_schema": { + "type": "object", + "properties": { + "fname": {"type": "string"}, + "lname": {"type": "string"}, + "DOB": {"type": "string"}, + }, + "additionalProperties": False, + }, + }, + { + "name": "finish", + "description": "Report the outcome. result is SAVED only if exactly one new chart for this referral exists now.", + "input_schema": { + "type": "object", + "properties": { + "result": {"type": "string", "enum": ["SAVED", "NOT SAVED"]}, + "reason": {"type": "string"}, + }, + "required": ["result", "reason"], + "additionalProperties": False, + }, + "strict": True, + }, +] + +READ_JS = r""" +(start) => { + let n = start; + const vis = e => { const r = e.getBoundingClientRect(); const s = getComputedStyle(e); + return r.width > 0 && r.height > 0 && s.visibility !== 'hidden' && s.display !== 'none'; }; + const labelOf = e => { + if (e.id) { const l = document.querySelector(`label[for="${e.id}"]`); if (l && l.innerText.trim()) return l.innerText.trim(); } + if (e.getAttribute('aria-label')) return e.getAttribute('aria-label'); + if (e.title) return e.title; + const td = e.closest('td'); if (td && td.previousElementSibling) { const t = td.previousElementSibling.innerText.trim(); if (t) return t.slice(0,60); } + const div = e.closest('.form-group, .col, div'); if (div) { const l = div.querySelector('label'); if (l && l.innerText.trim()) return l.innerText.trim().slice(0,60); } + return e.placeholder || ''; + }; + const out = {controls: [], buttons: []}; + document.querySelectorAll('[data-ar]').forEach(e => e.removeAttribute('data-ar')); + document.querySelectorAll('input, select, textarea').forEach(e => { + if (!vis(e) || e.type === 'hidden') return; + if (e.type === 'button' || e.type === 'submit') return; + const ref = String(n++); e.setAttribute('data-ar', ref); + const c = {ref, id: e.id || e.name, label: labelOf(e), type: e.tagName === 'SELECT' ? 'select' : (e.type || e.tagName.toLowerCase()), value: e.tagName === 'SELECT' ? (e.selectedOptions[0] ? e.selectedOptions[0].text : '') : (e.type === 'checkbox' ? String(e.checked) : e.value)}; + if (e.required) c.required = true; + if (e.tagName === 'SELECT') { const o = [...e.options].map(x => x.text.trim()).filter(Boolean); c.options = o.length <= 15 ? o : `${o.length} options; set by visible text`; } + out.controls.push(c); + }); + document.querySelectorAll('button, input[type=button], input[type=submit], a.btn, a[onclick], a[href^="#"], [role=button], [role=tab], [data-toggle], [data-bs-toggle], [aria-expanded], summary').forEach(e => { + if (!vis(e)) return; + const t = (e.innerText || e.value || e.title || '').trim(); if (!t) return; + const ref = String(n++); e.setAttribute('data-ar', ref); + out.buttons.push({ref, text: t.slice(0,60), id: e.id || undefined}); + }); + const alerts = [...document.querySelectorAll('.alert, .modal.show, .error, .text-danger, .invalid-feedback, [role=dialog], [role=alert]')] + .filter(vis).map(e => e.innerText.trim()).filter(Boolean).map(t => t.slice(0,400)); + if (alerts.length) out.notices = alerts; + out.next = n; + return out; +} +""" + + +class AgentToolsArm: + name = "agent_tools" + + def __init__(self, options: dict[str, Any]) -> None: + self.name = str(options.get("name", self.name)) + self.model = options.get("model", "claude-fable-5-1") + self.effort = options.get("effort", "medium") + self.max_turns = int(options.get("max_turns", 40)) + self.item_budget_usd = float(options.get("item_budget_usd", 1.00)) + self.headless = bool(options.get("headless", True)) + self.session: OpenEMRSession | None = None + self.rest: RestReader | None = None + self._dialogs: list[str] = [] + + def describe(self) -> dict[str, Any]: + return { + "arm": self.name, + "kind": "agent with structured tools and record read-back", + "model": self.model, + "effort": self.effort, + "tools": [t["name"] for t in TOOLS], + "max_turns_per_item": self.max_turns, + "item_budget_usd": self.item_budget_usd, + "prices_as_of": budget.prices()["as_of"], + "sees": "DOM and accessibility view of every frame; dialog text", + "reads_record": "OpenEMR REST patient search (clerk's API access)", + } + + def setup(self, env: Env, operator: Operator) -> list[dict[str, Any]]: + self.session = OpenEMRSession(env, headless=self.headless) + self.rest = RestReader(env) + t0 = time.time() + self.session.start() + self.session.page.on("dialog", self._on_dialog) + self.session.sign_in() + return [ + { + "phase": "write_procedure_and_tools", + "wall_s": round(time.time() - t0, 2), + "attended_s": 0.0, + "model_calls": 0, + "cost_usd": 0.0, + "detail": {"note": "the shared SOP plus five tool wrappers (this file); engineering time to build the tools is reported in the report's setup table, not here"}, + } + ] + + def _on_dialog(self, dialog: Any) -> None: + self._dialogs.append(f"{dialog.type}: {dialog.message}"[:300]) + try: + dialog.accept() + except Exception: # noqa: BLE001 + pass + + def after_drift(self, event: dict[str, Any], env: Env, operator: Operator) -> list[dict[str, Any]]: + return [] + + def close(self) -> None: + if self.session is not None: + self.session.close() + + # -- tools ------------------------------------------------------------------- + + def _read_screen(self) -> dict[str, Any]: + page = self.session.page + page.wait_for_timeout(300) + n = 1 + frames: list[dict[str, Any]] = [] + for i, frame in enumerate(page.frames): + try: + got = frame.evaluate(READ_JS, n) + except Exception: # noqa: BLE001 - detached frames are skipped + continue + n = got.pop("next") + if got["controls"] or got["buttons"] or got.get("notices"): + url = (frame.url or "").split("?")[0] + frames.append({"frame": url.rsplit("/", 2)[-1] or f"frame{i}", **got}) + out: dict[str, Any] = {"frames": frames} + if self._dialogs: + out["browser_dialogs"] = self._dialogs[:] + self._dialogs.clear() + return out + + def _locate(self, ref: str) -> Any: + for frame in self.session.page.frames: + try: + loc = frame.locator(f'[data-ar="{ref}"]') + if loc.count(): + return loc.first + except Exception: # noqa: BLE001 + continue + raise KeyError(f"no control with ref {ref}; call read_screen again") + + def _settle(self) -> None: + page = self.session.page + page.wait_for_timeout(400) + try: + page.wait_for_load_state("networkidle", timeout=3000) + except Exception: # noqa: BLE001 + pass + + def _tool(self, name: str, args: dict[str, Any]) -> Any: + if name == "read_screen": + return self._read_screen() + if name == "set_field": + loc = self._locate(args["ref"]) + tag = loc.evaluate("e => e.tagName.toLowerCase()") + if tag == "select": + try: + loc.select_option(label=args["value"]) + except Exception: # noqa: BLE001 + loc.select_option(value=args["value"]) + else: + loc.fill(args["value"]) + loc.dispatch_event("change") + return "ok" + if name == "click": + self._locate(args["ref"]).click(timeout=15_000) + self._settle() + return "clicked; call read_screen to see the result" + if name == "click_text": + for frame in self.session.page.frames: + try: + loc = frame.get_by_text(args["text"], exact=True).locator("visible=true") + if loc.count(): + loc.first.click(timeout=15_000) + self._settle() + return "clicked; call read_screen to see the result" + except Exception: # noqa: BLE001 + continue + raise KeyError(f"no visible element with the text {args['text']!r}") + if name == "find_patients": + return self.rest.find(**args) + raise KeyError(name) + + # -- one item ---------------------------------------------------------------- + + def run_item(self, item: WorkItem, env: Env, operator: Operator) -> ItemResult: + assert self.session is not None + prep = self.session.blank_form() + meter = budget.Meter(f"{self.name} {item.item_id}", self.item_budget_usd) + evidence = Path(env.workdir) / self.name / item.item_id + evidence.mkdir(parents=True, exist_ok=True) + t0 = time.time() + claim: dict[str, Any] | None = None + error = "" + turns = 0 + trace: list[dict[str, Any]] = [] + try: + claim, turns = self._loop(item, meter, trace) + except budget.SpendCapReached: + raise + except budget.ItemBudgetExceeded as exc: + error = f"budget: {exc}" + except Exception as exc: # noqa: BLE001 + error = f"{type(exc).__name__}: {exc}"[:500] + finally: + (evidence / "trace.json").write_text(json.dumps(trace, indent=1, default=str)[:400_000]) + wall = time.time() - t0 + if claim is not None and claim.get("result") == "SAVED": + claimed, status = "done", f"RESULT: SAVED: {claim.get('reason', '')}" + elif claim is not None: + claimed, status = "stopped", f"RESULT: NOT SAVED: {claim.get('reason', '')}" + else: + claimed, status = "error", error or "no finish" + if claimed != "done": + operator.ask(OperatorRequest(kind="stop", item_id=item.item_id, reason=status, surface="agent finish")) + return ItemResult( + item_id=item.item_id, + claimed=claimed, + product_status=status, + wall_s=round(wall, 2), + model_calls=meter.calls, + cost_usd=round(meter.usd, 6), + evidence=str(evidence.relative_to(env.workdir)), + detail={**budget.summarize(meter), **prep, "turns": turns, "compute_usd": round(budget.compute_usd(wall), 6)}, + ) + + def _loop(self, item: WorkItem, meter: budget.Meter, trace: list[dict[str, Any]]) -> tuple[dict[str, Any] | None, int]: + client = budget.client(self.model) + messages: list[dict[str, Any]] = [ + {"role": "user", "content": SOP.replace( + "End your final reply with exactly one line, either\nRESULT: SAVED\nor\nRESULT: NOT SAVED: ", + "Finish by calling the finish tool.", + ) + "\n\nThe referral:\n" + item.source_text} + ] + for turn in range(1, self.max_turns + 1): + approx_in = 3000 + 2500 * turn + meter.before_call(self.model, budget.max_call_usd(self.model, approx_in, 4000)) + response = client.messages.create( + model=self.model, + max_tokens=16000, + system=SYSTEM, + tools=TOOLS, + output_config={"effort": self.effort}, + cache_control={"type": "ephemeral"}, + messages=messages, + ) + meter.record(self.model, response.usage) + messages.append({"role": "assistant", "content": response.content}) + uses = [b for b in response.content if b.type == "tool_use"] + trace.append({"turn": turn, "stop": response.stop_reason, "calls": [{"name": b.name, "input": b.input} for b in uses]}) + if response.stop_reason == "refusal": + return {"result": "NOT SAVED", "reason": "the model declined the request"}, turn + if not uses: + text = " ".join(b.text for b in response.content if b.type == "text") + messages.append({"role": "user", "content": "Call the finish tool to report the outcome." + (" " if text else "")}) + continue + results = [] + for b in uses: + if b.name == "finish": + return dict(b.input), turn + try: + out = self._tool(b.name, dict(b.input)) + content = out if isinstance(out, str) else json.dumps(out, default=str) + results.append({"type": "tool_result", "tool_use_id": b.id, "content": content[:20_000]}) + except Exception as exc: # noqa: BLE001 + results.append({"type": "tool_result", "tool_use_id": b.id, "content": f"Error: {exc}"[:400], "is_error": True}) + trace[-1].setdefault("results", []).append(results[-1]["content"][:2000]) + messages.append({"role": "user", "content": results}) + return None, self.max_turns + + +def factory(options: dict[str, Any]) -> AgentToolsArm: + return AgentToolsArm(options) diff --git a/packages/flow/benchmark/lifecycle/arms/budget.py b/packages/flow/benchmark/lifecycle/arms/budget.py new file mode 100644 index 000000000..ecde3f780 --- /dev/null +++ b/packages/flow/benchmark/lifecycle/arms/budget.py @@ -0,0 +1,232 @@ +"""Model prices, usage accounting and the shared spend cap for the comparison arms. + +- Prices come only from ``prices.json`` (dated). Nothing here hard-codes a price. +- Every model call made by any arm is added to one spend ledger + (``benchmark/lifecycle/results/spend.json``) under a file lock, so the cap + holds across arms and across resumed runs. +- Before a call, an arm asks :func:`reserve` for the most that call could cost. + If that could pass the cap, :class:`SpendCapReached` is raised and nothing is + sent. The cap is $30 unless ``LIFECYCLE_SPEND_CAP_USD`` raises it. +- The API key is read from the environment (``ANTHROPIC_API_KEY``) or, when + ``LIFECYCLE_KEYCHAIN_SERVICE`` names one, from the macOS keychain. It is held + in this process only and never printed, logged or written. +""" + +from __future__ import annotations + +import fcntl +import json +import os +import subprocess +import time +from contextlib import contextmanager +from dataclasses import asdict, dataclass, field +from pathlib import Path +from typing import Any, Iterator, Optional + +HERE = Path(__file__).resolve().parent +PRICES_PATH = HERE / "prices.json" +RESULTS = HERE.parent / "results" +SPEND_PATH = Path(os.environ.get("LIFECYCLE_SPEND_LEDGER", str(RESULTS / "spend.json"))) +DEFAULT_CAP_USD = 30.0 + + +class SpendCapReached(BaseException): + """The next model call could take total spend past the cap. + + A BaseException on purpose: no `except Exception` in an arm or a harness + can swallow it, so reaching the cap always stops the run (resume later). + """ + + +class ItemBudgetExceeded(RuntimeError): + """One item (or one phase) used up its own fixed budget.""" + + +def prices() -> dict[str, Any]: + return json.loads(PRICES_PATH.read_text()) + + +def price(model: str) -> dict[str, float]: + table = prices()["models"] + if model not in table: + raise KeyError(f"{model} is not in {PRICES_PATH.name} (as of {prices()['as_of']})") + return table[model] + + +def cap_usd() -> float: + return float(os.environ.get("LIFECYCLE_SPEND_CAP_USD", DEFAULT_CAP_USD)) + + +@dataclass +class Usage: + """Token usage of one model on one item or phase, priced from the table.""" + + model: str + input: int = 0 + output: int = 0 + cache_read: int = 0 + cache_write: int = 0 + calls: int = 0 + + @property + def usd(self) -> float: + p = price(self.model) + return ( + self.input * p["input"] + + self.output * p["output"] + + self.cache_read * p["cache_read"] + + self.cache_write * p["cache_write"] + ) / 1_000_000 + + def add_response(self, usage: Any) -> float: + """Add one API response's ``usage``; returns that call's cost.""" + + def n(name: str) -> int: + return int(getattr(usage, name, 0) or 0) + + one = Usage( + self.model, + input=n("input_tokens"), + output=n("output_tokens"), + cache_read=n("cache_read_input_tokens"), + cache_write=n("cache_creation_input_tokens"), + calls=1, + ) + self.input += one.input + self.output += one.output + self.cache_read += one.cache_read + self.cache_write += one.cache_write + self.calls += 1 + return one.usd + + def to_dict(self) -> dict[str, Any]: + return {**asdict(self), "usd": round(self.usd, 6)} + + +@dataclass +class Meter: + """Usage for one item or phase, with that unit's own fixed budget.""" + + label: str + budget_usd: float + usage: dict[str, Usage] = field(default_factory=dict) + + def of(self, model: str) -> Usage: + return self.usage.setdefault(model, Usage(model)) + + @property + def usd(self) -> float: + return sum(u.usd for u in self.usage.values()) + + @property + def calls(self) -> int: + return sum(u.calls for u in self.usage.values()) + + def before_call(self, model: str, max_call_usd: float) -> None: + """Refuse a call that could pass this unit's budget or the global cap.""" + + if self.usd + max_call_usd > self.budget_usd: + raise ItemBudgetExceeded( + f"{self.label}: ${self.usd:.3f} used of ${self.budget_usd:.2f}; " + f"the next call could cost ${max_call_usd:.3f}" + ) + check_cap(max_call_usd) + + def record(self, model: str, response_usage: Any) -> float: + usd = self.of(model).add_response(response_usage) + add_spend(self.label, model, usd) + return usd + + def to_list(self) -> list[dict[str, Any]]: + return [u.to_dict() for u in self.usage.values()] + + +def max_call_usd(model: str, input_tokens: int, max_tokens: int) -> float: + """An upper bound for one call: every input token uncached, full output.""" + + p = price(model) + return (input_tokens * p["cache_write"] + max_tokens * p["output"]) / 1_000_000 + + +@contextmanager +def _locked() -> Iterator[dict[str, Any]]: + SPEND_PATH.parent.mkdir(parents=True, exist_ok=True) + lock = SPEND_PATH.with_suffix(".lock") + with open(lock, "w") as fh: + fcntl.flock(fh, fcntl.LOCK_EX) + data = ( + json.loads(SPEND_PATH.read_text()) + if SPEND_PATH.is_file() + else {"total_usd": 0.0, "entries": []} + ) + yield data + fcntl.flock(fh, fcntl.LOCK_UN) + + +def spent() -> float: + if not SPEND_PATH.is_file(): + return 0.0 + return float(json.loads(SPEND_PATH.read_text()).get("total_usd", 0.0)) + + +def check_cap(next_call_usd: float) -> None: + total, cap = spent(), cap_usd() + if total + next_call_usd > cap: + raise SpendCapReached( + f"${total:.2f} spent; a call of up to ${next_call_usd:.2f} could pass the " + f"${cap:.2f} cap (raise LIFECYCLE_SPEND_CAP_USD only with the coordinator's go)" + ) + + +def add_spend(label: str, model: str, usd: float) -> float: + with _locked() as data: + data["entries"].append( + {"label": label, "model": model, "usd": round(usd, 6), "at": round(time.time(), 3)} + ) + data["total_usd"] = round(sum(e["usd"] for e in data["entries"]), 6) + data["cap_usd"] = cap_usd() + data["prices_as_of"] = prices()["as_of"] + SPEND_PATH.write_text(json.dumps(data, indent=1) + "\n") + return data["total_usd"] + + +def ensure_key() -> None: + """Make the key available to the SDK without ever printing it.""" + + if os.environ.get("ANTHROPIC_API_KEY"): + return + service = os.environ.get("LIFECYCLE_KEYCHAIN_SERVICE") + if service: + out = subprocess.run( + ["security", "find-generic-password", "-s", service, "-w"], + capture_output=True, + text=True, + ) + if out.returncode == 0 and out.stdout.strip(): + os.environ["ANTHROPIC_API_KEY"] = out.stdout.strip() + return + raise SystemExit(f"keychain service {service!r} has no readable password") + raise SystemExit( + "no Anthropic key: set ANTHROPIC_API_KEY, or LIFECYCLE_KEYCHAIN_SERVICE to a " + "keychain item that holds it" + ) + + +def client(model: str = "") -> Any: + ensure_key() + import anthropic + + return anthropic.Anthropic(max_retries=4) + + +def compute_usd(wall_s: float) -> float: + """Nominal local compute for ``wall_s`` seconds of one worker.""" + + return prices()["compute"]["host_usd_per_hour"] * wall_s / 3600.0 + + +def summarize(meter: Optional[Meter]) -> dict[str, Any]: + if meter is None: + return {"model_calls": 0, "cost_usd": 0.0, "usage": []} + return {"model_calls": meter.calls, "cost_usd": round(meter.usd, 6), "usage": meter.to_list()} diff --git a/packages/flow/benchmark/lifecycle/arms/compare.py b/packages/flow/benchmark/lifecycle/arms/compare.py new file mode 100644 index 000000000..962922a9b --- /dev/null +++ b/packages/flow/benchmark/lifecycle/arms/compare.py @@ -0,0 +1,393 @@ +"""One command for the whole economic comparison: every arm, one seeded stream, resumable. + + cd packages/flow/benchmark + python -m lifecycle.arms.compare --out lifecycle/results/economics- [--seed 7 --items 40 --days 10] + +- Every arm runs the same seeded worklist and event ledger through the soak + harness's ``run_arm`` (same reset, seeded charts, drift days, injected + faults, oracle, scripted operator and cost model). +- Each arm writes its own ``//ledger.jsonl``. Rerunning the same + command skips arms whose ledger ends with ``arm_end``. An arm that stopped + part-way (the spend cap, a crash, Ctrl-C) is moved to + ``.partial-`` and run again from its start, because the app is reset + per arm; the partial ledger is kept and its model spend stays counted. +- Model spend is capped across all arms at $30 (``LIFECYCLE_SPEND_CAP_USD`` + raises it only with the coordinator's go). The cap is checked before every + model call; reaching it stops the run with a message to resume. +- The API key comes from ``ANTHROPIC_API_KEY`` or the keychain item named in + ``LIFECYCLE_KEYCHAIN_SERVICE``; it is never printed. +- At the end, ``economics.json`` and ``REPORT.md`` are written from the + ledgers alone (``python -m lifecycle.arms.economics `` recomputes them). +""" + +from __future__ import annotations + +import argparse +import json +import shutil +import sys +import time +from pathlib import Path +from typing import Any, Optional + +from . import budget, economics, report + +DEFAULT_ARMS = [ + # (name in the ledger, module:factory, options) + # The soak's OpenAdapt arm: the certified referral program under the production + # configuration. Its API tier is refused on this form (results/api_tier/). + ("openadapt", "lifecycle.openadapt_arm:make", {}), + ( + "cua_alone", + "lifecycle.arms.cua:factory", + {"model": "claude-opus-5-5", "item_budget_usd": 1.00}, + ), + ( + "agent_tools", + "lifecycle.arms.agent_tools:factory", + {"model": "claude-opus-5-5", "item_budget_usd": 0.60}, + ), + ( + "agent_script", + "lifecycle.arms.agent_script:factory", + {"model": "claude-opus-5-5"}, + ), + ("human_script", "lifecycle.arms.human_script:factory", {}), +] + + +MIN_FREE_GIB = float(__import__("os").environ.get("LIFECYCLE_MIN_FREE_GIB", "10")) + + +class DiskLow(BaseException): + """Free disk fell under the shared machine's floor; stop between items.""" + + +def free_gib() -> float: + return shutil.disk_usage("/System/Volumes/Data").free / 2**30 + + +ITEM_TIMEOUT_S = float(__import__("os").environ.get("LIFECYCLE_ITEM_TIMEOUT_S", "900")) +#: After the soft timeout, how long to wait for the arm to return before the +#: whole process exits (so a hang can never hold the shared fixture for hours). +HARD_GRACE_S = 180.0 + + +def _playwright_children() -> list[int]: + import os + import subprocess + + out = subprocess.run( + ["pgrep", "-P", str(os.getpid())], capture_output=True, text=True + ).stdout.split() + pids = [] + for pid in out: + cmd = subprocess.run( + ["ps", "-o", "command=", "-p", pid], capture_output=True, text=True + ).stdout + if "playwright" in cmd: + pids.append(int(pid)) + return pids + + +class ItemWatchdog: + """Bounds one item. + + At ``timeout_s`` it writes every thread's Python stack to ``stack_path`` + (to find what hung) and kills this process's Playwright drivers, so a + blocked browser call raises and the runner records the item as an error. + If the arm still has not returned ``HARD_GRACE_S`` later (a hang outside + the browser), it exits the process with code 5; the comparison resumes + from the arm's start and keeps the partial ledger. + """ + + def __init__(self, timeout_s: float, stack_path: Path) -> None: + import threading + + self.fired = False + self.stack_path = stack_path + self._soft = threading.Timer(timeout_s, self._fire) + self._hard = threading.Timer(timeout_s + HARD_GRACE_S, self._exit) + for t in (self._soft, self._hard): + t.daemon = True + t.start() + + def _dump(self, why: str) -> None: + import faulthandler + + with open(self.stack_path, "a") as fh: + fh.write(f"--- {why} at {time.strftime('%Y-%m-%dT%H:%M:%S')}\n") + fh.flush() + faulthandler.dump_traceback(file=fh, all_threads=True) + + def _fire(self) -> None: + import os + import signal + + self.fired = True + self._dump("item timeout") + for pid in _playwright_children(): + try: + os.kill(pid, signal.SIGKILL) + except OSError: + pass + + def _exit(self) -> None: + import os + + self._dump("hard timeout; exiting") + os._exit(5) + + def cancel(self) -> None: + self._soft.cancel() + self._hard.cancel() + + +class Guarded: + """Any arm, checked for free disk before every item and bounded in time. + + The check runs between items, never inside the harness's app reset, so a + stop leaves the fixture whole. DiskLow is a BaseException so the runner's + ``except Exception`` cannot turn it into an item result. + """ + + def __init__(self, options: dict[str, Any]) -> None: + from ..arm import load_arm + + inner_opts = dict(options) + spec = inner_opts.pop("_inner") + self.inner = load_arm(spec, inner_opts) + self.name = self.inner.name + + def describe(self) -> dict[str, Any]: + return self.inner.describe() + + def setup(self, env: Any, operator: Any) -> list[dict[str, Any]]: + return self.inner.setup(env, operator) + + def run_item(self, item: Any, env: Any, operator: Any) -> Any: + free = free_gib() + if free < MIN_FREE_GIB: + raise DiskLow( + f"{free:.1f} GiB free, under {MIN_FREE_GIB:.0f} GiB; stopped before {item.item_id}" + ) + dog = ItemWatchdog(ITEM_TIMEOUT_S, Path(env.workdir) / "stacks.txt") + try: + result = self.inner.run_item(item, env, operator) + except Exception: + if dog.fired: + self._fresh_browser() + raise + finally: + dog.cancel() + if dog.fired: + self._fresh_browser() + result.claimed = "error" + result.product_status = ( + f"item timed out after {ITEM_TIMEOUT_S:.0f} s; browser driver stopped. " + + str(result.product_status) + )[:500] + return result + + def _fresh_browser(self) -> None: + """The browser driver was killed: the arm starts a new browser on its next item.""" + + session = getattr(self.inner, "session", None) + if session is not None and hasattr(session, "close"): + session.close() + + def after_drift( + self, event: dict[str, Any], env: Any, operator: Any + ) -> list[dict[str, Any]]: + return self.inner.after_drift(event, env, operator) + + def close(self) -> None: + self.inner.close() + + +def guarded(options: dict[str, Any]) -> Guarded: + return Guarded(options) + + +def _complete(ledger: Path) -> bool: + if not ledger.is_file(): + return False + lines = [ln for ln in ledger.read_text().splitlines() if ln.strip()] + return ( + bool(lines) + and json.loads(lines[-1]).get("type") in ("arm_end", "run_end") + and any(json.loads(ln).get("type") == "arm_end" for ln in lines[-3:]) + ) + + +def _move_partial(d: Path) -> Optional[Path]: + if not d.exists(): + return None + n = 1 + while (d.parent / f"{d.name}.partial-{n}").exists(): + n += 1 + dest = d.parent / f"{d.name}.partial-{n}" + d.rename(dest) + return dest + + +def main(argv: Optional[list[str]] = None) -> int: + p = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + p.add_argument("--out", type=Path, required=True) + p.add_argument("--seed", type=int, default=7) + p.add_argument("--items", type=int, default=40) + p.add_argument("--days", type=int, default=10) + p.add_argument("--existing", type=int, default=12) + p.add_argument("--only", action="append", help="run only these arms (by name)") + p.add_argument( + "--arms-json", type=Path, help="a JSON list of [name, module:factory, options]" + ) + p.add_argument("--app", default="openemr") + p.add_argument("--report-only", action="store_true") + p.add_argument( + "--drift", + action="append", + default=None, + metavar="KIND@SEQ", + help="replace the drift schedule, as lifecycle.run --drift (e.g. relabel_save@26)", + ) + args = p.parse_args(argv) + + from ..cost_model import V1 + from ..run import PROTOCOL_VERSION, Ledger, engine_commit, run_arm + from ..workload import WORKLOAD_VERSION, generate + + out: Path = args.out + out.mkdir(parents=True, exist_ok=True) + arms = json.loads(args.arms_json.read_text()) if args.arms_json else DEFAULT_ARMS + if args.only: + arms = [a for a in arms if a[0] in args.only] + + plan = { + "seed": args.seed, + "items": args.items, + "days": args.days, + "existing": args.existing, + "app": args.app, + "drift": args.drift, + "arms": arms, + } + plan_path = out / "plan.json" + if plan_path.is_file(): + old = json.loads(plan_path.read_text()) + same = { + k: old.get(k) for k in ("seed", "items", "days", "existing", "app", "drift") + } + if same != {k: plan[k] for k in same}: + raise SystemExit( + f"{out} was started with a different stream {same}; use a new --out" + ) + else: + plan_path.write_text(json.dumps(plan, indent=1) + "\n") + + if not args.report_only: + workload = generate(args.seed, args.items, args.days, args.existing) + if args.drift is not None: + by_seq = {i.seq: i for i in workload.items} + workload.drift = [ + {"day": by_seq[int(seq)].day, "kind": kind, "first_seq": int(seq)} + for kind, _, seq in (spec.partition("@") for spec in args.drift) + ] + wl = out / "workload.jsonl" + text = "".join( + json.dumps(r, sort_keys=True) + "\n" for r in workload.ledger_rows() + ) + if wl.is_file() and wl.read_text() != text: + raise SystemExit( + "the seeded workload differs from the one this run started with" + ) + wl.write_text(text) + from ..apps import make_app + + app = None + for name, spec, options in arms: + d = out / name + if _complete(d / "ledger.jsonl"): + print(f"[{name}] already complete; skipping") + continue + moved = _move_partial(d) + if moved: + print( + f"[{name}] earlier attempt stopped part-way; kept as {moved.name}" + ) + d.mkdir(parents=True) + if app is None: + app = make_app(args.app) + ledger = Ledger(d / "ledger.jsonl") + ledger.write( + { + "type": "run_start", + "protocol": PROTOCOL_VERSION, + "workload": WORKLOAD_VERSION, + "seed": args.seed, + "items": args.items, + "days": args.days, + "cost_model": V1.to_dict(), + "engine_commit": engine_commit(), + "prices_as_of": budget.prices()["as_of"], + "spend_before_usd": budget.spent(), + "spend_cap_usd": budget.cap_usd(), + "app": app.describe(), + } + ) + t0 = time.time() + try: + if free_gib() < MIN_FREE_GIB: + raise DiskLow(f"{free_gib():.1f} GiB free before {name}") + run_arm( + arm_spec="lifecycle.arms.compare:guarded", + arm_options={**options, "_inner": spec}, + app=app, + workload=workload, + ledger=ledger, + out=d, + cost_name=V1.version, + stop_after=options.get("stop_after"), + ) + ledger.write( + { + "type": "run_end", + "wall_s": round(time.time() - t0, 1), + "spend_after_usd": budget.spent(), + } + ) + except budget.SpendCapReached as exc: + ledger.write( + {"type": "stopped", "why": "spend_cap", "detail": str(exc)} + ) + ledger.close() + print( + f"[{name}] stopped: {exc}\nResume with the same command once the cap is raised." + ) + return 3 + except BaseException as exc: + # Matched by name: the guarded arm is loaded through the + # module path, so its DiskLow is not this __main__'s class. + if type(exc).__name__ != "DiskLow": + raise + ledger.write({"type": "stopped", "why": "disk_low", "detail": str(exc)}) + ledger.close() + print( + f"[{name}] stopped: {exc}\nResume with the same command when there is room." + ) + return 4 + finally: + ledger.close() + + rows = economics.read_rows(sorted(out.glob("*/ledger.jsonl"))) + result = economics.compute(rows) + (out / "economics.json").write_text( + json.dumps(result, indent=1, default=str) + "\n" + ) + (out / "REPORT.md").write_text(report.render(result, out, plan)) + print(economics.render(result)) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/packages/flow/benchmark/lifecycle/arms/cua.py b/packages/flow/benchmark/lifecycle/arms/cua.py new file mode 100644 index 000000000..e84971fe9 --- /dev/null +++ b/packages/flow/benchmark/lifecycle/arms/cua.py @@ -0,0 +1,238 @@ +"""Arm ``cua_alone``: a screenshot-driven computer-use agent, alone. + +The current frontier model drives the browser through the computer use toolset +(``computer_toolset_20260801``). It sees screenshots only: no DOM, no API, no +record read-back. Nothing records or checks it. Each item has a fixed budget +(model calls and dollars); an item that runs out is a stop the operator sees. + +Actions go to the page through the scoreboard's :class:`PlainComputer`, the +same translator the earlier scoreboard agent arm used. +""" + +from __future__ import annotations + +import base64 +import json +import time +from pathlib import Path +from typing import Any + +from ..arm import Env, ItemResult, Operator, OperatorRequest, WorkItem +from . import budget +from .openemr_ui import SOP, OpenEMRSession, claimed_from + +DISABLED_MEMBERS = ( + "zoom", + "triple_click", + "middle_click", + "left_mouse_down", + "left_mouse_up", + "hold_key", +) + +SYSTEM = """\ +You operate a web browser that shows one page of a clinic's electronic medical \ +record (OpenEMR). There is no address bar and no other application. The screen \ +is 1280x800 pixels. Work only on the page you are shown. Do not navigate away, \ +open menus you do not need, or change any field the task does not mention. + +Click a field before you type into it. To scroll, scroll at the point you last \ +clicked. When the task is done, stop and reply; do not take further actions.""" + + +class CuaArm: + name = "cua_alone" + + def __init__(self, options: dict[str, Any]) -> None: + self.name = str(options.get("name", self.name)) + self.model = options.get("model", "claude-fable-5-1") + self.effort = options.get("effort", "medium") + self.max_turns = int(options.get("max_turns", 40)) + self.item_budget_usd = float(options.get("item_budget_usd", 1.50)) + self.headless = bool(options.get("headless", True)) + self.session: OpenEMRSession | None = None + + def describe(self) -> dict[str, Any]: + return { + "arm": self.name, + "kind": "computer-use agent alone (screenshots only)", + "model": self.model, + "effort": self.effort, + "tool": "computer_toolset_20260801", + "disabled_members": list(DISABLED_MEMBERS), + "max_turns_per_item": self.max_turns, + "item_budget_usd": self.item_budget_usd, + "prices_as_of": budget.prices()["as_of"], + "sees": "screenshots of the OpenEMR page", + "reads_record": False, + } + + def setup(self, env: Env, operator: Operator) -> list[dict[str, Any]]: + # Writing the procedure (the SOP above) is the only setup; it is shared + # by every agent arm and charged the same to each. + self.session = OpenEMRSession(env, headless=self.headless) + t0 = time.time() + self.session.start() + self.session.sign_in() + return [ + { + "phase": "write_procedure", + "wall_s": round(time.time() - t0, 2), + "attended_s": 0.0, + "model_calls": 0, + "cost_usd": 0.0, + "detail": {"note": "the shared SOP; its writing time is charged by the cost model's procedure rule if any"}, + } + ] + + def after_drift(self, event: dict[str, Any], env: Env, operator: Operator) -> list[dict[str, Any]]: + return [] + + def close(self) -> None: + if self.session is not None: + self.session.close() + + def run_item(self, item: WorkItem, env: Env, operator: Operator) -> ItemResult: + try: + from benchmark.scoreboard.agent import PlainComputer + except ImportError: # run from packages/flow/benchmark + from scoreboard.agent import PlainComputer + + assert self.session is not None + prep = self.session.blank_form() + meter = budget.Meter(f"{self.name} {item.item_id}", self.item_budget_usd) + evidence = Path(env.workdir) / self.name / item.item_id + evidence.mkdir(parents=True, exist_ok=True) + computer = PlainComputer(self.session.page) + t0 = time.time() + final_text, error = "", "" + turns = 0 + try: + final_text, turns = self._loop(item, computer, meter, evidence) + except (budget.ItemBudgetExceeded, budget.SpendCapReached) as exc: + if isinstance(exc, budget.SpendCapReached): + raise + error = f"budget: {exc}" + except Exception as exc: # noqa: BLE001 - an agent crash is a stop + error = f"{type(exc).__name__}: {exc}"[:500] + wall = time.time() - t0 + claimed, status = claimed_from(final_text) if not error else ("error", error) + if claimed != "done": + operator.ask( + OperatorRequest( + kind="stop", + item_id=item.item_id, + reason=status, + surface="agent final reply", + ) + ) + (evidence / "reply.json").write_text( + json.dumps({"final_text": final_text, "error": error, "turns": turns}, indent=1) + ) + return ItemResult( + item_id=item.item_id, + claimed=claimed if claimed != "error" else "error", + product_status=status, + wall_s=round(wall, 2), + model_calls=meter.calls, + cost_usd=round(meter.usd, 6), + evidence=str(evidence.relative_to(env.workdir)), + detail={ + **budget.summarize(meter), + **prep, + "turns": turns, + "compute_usd": round(budget.compute_usd(wall), 6), + }, + ) + + def _loop(self, item: WorkItem, computer: Any, meter: budget.Meter, evidence: Path) -> tuple[str, int]: + client = budget.client(self.model) + tools = [ + { + "type": "computer_toolset_20260801", + "configs": {m: {"enabled": False} for m in DISABLED_MEMBERS}, + } + ] + first = computer.execute({"action": "screenshot"}) + instruction = SOP + "\n\nThe referral:\n" + item.source_text + messages: list[dict[str, Any]] = [ + { + "role": "user", + "content": [ + {"type": "text", "text": instruction}, + _image(first.png), + ], + } + ] + ledger: list[dict[str, Any]] = [] + for turn in range(1, self.max_turns + 1): + # An upper bound for this call: history so far, uncached, plus 16k out. + approx_in = 4000 + 1600 * turn + meter.before_call(self.model, budget.max_call_usd(self.model, approx_in, 4000)) + t0 = time.time() + response = client.messages.create( + model=self.model, + max_tokens=16000, + system=SYSTEM, + tools=tools, + output_config={"effort": self.effort}, + cache_control={"type": "ephemeral"}, + messages=messages, + ) + usd = meter.record(self.model, response.usage) + calls = [b for b in response.content if b.type == "tool_use"] + ledger.append( + { + "turn": turn, + "s": round(time.time() - t0, 2), + "usd": round(usd, 6), + "stop_reason": response.stop_reason, + "actions": [{"name": b.name, "input": b.input} for b in calls], + } + ) + (evidence / "calls.json").write_text(json.dumps(ledger, indent=1)) + messages.append({"role": "assistant", "content": response.content}) + if response.stop_reason == "refusal": + return "RESULT: NOT SAVED: the model declined the request", turn + if not calls: + return " ".join(b.text for b in response.content if b.type == "text"), turn + results: list[dict[str, Any]] = [] + for block in calls: + native = {"action": block.name, **dict(block.input)} + if block.name == "key" and block.input.get("repeat"): + n = max(1, min(int(block.input["repeat"]), 100)) + native = {"action": "key", "text": " ".join([str(block.input.get("text", ""))] * n)} + result: dict[str, Any] = { + "type": "tool_result", + "tool_use_id": block.id, + "toolset_name": "computer", + } + try: + obs = computer.execute(native) + except Exception as exc: # noqa: BLE001 - tell the model, let it choose + result["content"] = f"Error: {exc}"[:300] + result["is_error"] = True + results.append(result) + continue + if block.name == "screenshot" or block is calls[-1]: + content: list[dict[str, Any]] = [] + if obs.notes: + content.append({"type": "text", "text": "; ".join(obs.notes)}) + content.append(_image(obs.png)) + result["content"] = content + else: + result["content"] = [{"type": "text", "text": "OK"}] + results.append(result) + messages.append({"role": "user", "content": results}) + return f"RESULT: NOT SAVED: no finish within {self.max_turns} model calls", self.max_turns + + +def _image(png: bytes) -> dict[str, Any]: + return { + "type": "image", + "source": {"type": "base64", "media_type": "image/png", "data": base64.b64encode(png).decode()}, + } + + +def factory(options: dict[str, Any]) -> CuaArm: + return CuaArm(options) diff --git a/packages/flow/benchmark/lifecycle/arms/economics.py b/packages/flow/benchmark/lifecycle/arms/economics.py new file mode 100644 index 000000000..15f1c9ec3 --- /dev/null +++ b/packages/flow/benchmark/lifecycle/arms/economics.py @@ -0,0 +1,282 @@ +"""Total lifecycle cost per 1,000 verified items, per arm, from the results ledgers alone. + +Every number comes from ledger rows the harness wrote (items, operator +requests, setup and maintenance phases, drift events) priced with the soak's +versioned operator cost model and this directory's dated ``prices.json``. +Nothing is re-measured here, so ``python -m lifecycle.arms.economics `` +recomputes the report from committed data. + +Definitions: + +- *Verified item*: the oracle's final read says the item is right (the + intended chart exactly once, or no new chart when none is wanted), whoever + did it. Items a person did by hand count, and so does their time: this is + the cost of getting the work done right, not of automating it. +- *Automated*: verified and not done by hand. +- *Setup*: one-time, per workflow. People's minutes (attended phase time plus + the cost model's fixed charges for teach, compile review and admission) and + model dollars. +- *Per item*: model dollars, nominal compute, and operator minutes (approvals, + stops, questions, reconciliations, items done by hand). +- *Maintenance*: after a UI change. Model dollars spent repairing or + re-authoring, and people's minutes reviewing the repair. +- *Silent-wrong cleanup*: each silent wrong is charged, as a lower bound, + the people time to find and fix it later (one reconcile plus one manual + entry, plus the interrupt charge each). Its clinical or business harm is + not priced; the silent-wrong rate is reported beside every total. +- Projection to 1,000 verified items: setup once, per-item costs scaled by + 1,000 / verified, maintenance per UI change times the assumed changes per + 1,000 items (``prices.json`` drift rate). +""" + +from __future__ import annotations + +import json +import statistics +import sys +from collections import Counter +from pathlib import Path +from typing import Any, Iterable, Optional + +from . import budget + +try: + from ..cost_model import MODELS + from ..metrics import wilson +except ImportError: # pragma: no cover - the harness is not checked in yet + MODELS = {} + + def wilson(k: int, n: int, z: float = 1.96) -> Optional[tuple[float, float]]: # type: ignore[misc] + import math + + if n == 0: + return None + p = k / n + d = 1 + z * z / n + c = p + z * z / (2 * n) + r = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) + return (round((c - r) / d, 4), round((c + r) / d, 4)) + + +FIXED_PHASES = {"teach": "teach_prep_s", "compile": "compile_review_s", "admit": "admit_s"} + + +def read_rows(paths: Iterable[Path]) -> list[dict[str, Any]]: + rows: list[dict[str, Any]] = [] + for p in paths: + for line in p.read_text().splitlines(): + if line.strip(): + rows.append(json.loads(line)) + return rows + + +def _cost_model(rows: list[dict[str, Any]]) -> dict[str, Any]: + for r in rows: + if r.get("type") == "run_start" and r.get("cost_model"): + return r["cost_model"] + if MODELS: + return next(iter(MODELS.values())).to_dict() + raise ValueError("no cost model in the ledger and none importable") + + +def _phase_human_s(cm: dict[str, Any], phase: dict[str, Any]) -> float: + fixed = cm.get(FIXED_PHASES.get(phase.get("phase", ""), ""), 0.0) + return float(phase.get("attended_s", 0.0) or 0.0) + float(fixed or 0.0) + + +def arm_economics(rows: list[dict[str, Any]], arm: str, cm: dict[str, Any], labor: dict[str, Any], + drift_per_1000: float) -> dict[str, Any]: + mine = [r for r in rows if r.get("arm") == arm] + items = [r for r in mine if r["type"] == "item_result"] + ops = [r for r in mine if r["type"] == "operator"] + setup = [r for r in mine if r["type"] == "phase" and r.get("stage") == "setup"] + maint = [r for r in mine if r["type"] == "phase" and r.get("stage") == "maintenance"] + drifts = [r for r in mine if r["type"] == "drift_applied"] + describe = next((r.get("describe") for r in mine if r["type"] == "arm_start"), {}) or {} + + n = len(items) + by_hand = {r["item_id"] for r in ops if r.get("action") == "do_by_hand" and r.get("item_id")} + verified = [r for r in items if r.get("final_oracle") == "correct"] + automated = [r for r in verified if r["item_id"] not in by_hand] + outcomes = Counter(r["outcome"] for r in items) + wanted = [r for r in items if r.get("chart_wanted")] + clean = [r for r in wanted if r.get("exception") in ("clean", "missing_optional", "malformed_dob", + "malformed_case", "ambiguous_new_person")] + + setup_human_s = sum(_phase_human_s(cm, p) for p in setup) + setup_model_usd = sum(float(p.get("cost_usd", 0.0) or 0.0) for p in setup) + item_model_usd = sum(float(r.get("cost_usd", 0.0) or 0.0) for r in items) + compute_usd = sum(float((r.get("detail") or {}).get("compute_usd") or budget.compute_usd(r.get("wall_s", 0.0))) + for r in items) + item_op_s = sum(float(r["charged_s"]) for r in ops if r["kind"] != "repair_review") + review_s = sum(float(r["charged_s"]) for r in ops if r["kind"] == "repair_review") + maint_human_s = review_s + sum(_phase_human_s(cm, p) for p in maint) + maint_model_usd = sum(float(p.get("cost_usd", 0.0) or 0.0) for p in maint) + model_calls = sum(int(r.get("model_calls", 0) or 0) for r in items) + + op_rate = labor["operator_usd_per_hour"] / 3600.0 + sw = outcomes.get("silent_wrong", 0) + cleanup_s = sw * (float(cm.get("reconcile_s", 0.0)) + float(cm.get("manual_entry_s", 0.0)) + + 2 * float(cm.get("interrupt_s", 0.0))) + v = len(verified) + per_v = (1000.0 / v) if v else float("nan") + items_per_1000v = n * per_v + d = len(drifts) + maint_per_event_usd = (maint_model_usd + maint_human_s * op_rate) / d if d else 0.0 + maint_per_event_min = maint_human_s / 60.0 / d if d else 0.0 + proj_events = drift_per_1000 * items_per_1000v / 1000.0 if v else float("nan") + + proj = { + "setup_usd": setup_model_usd + setup_human_s * op_rate, + "setup_model_usd": setup_model_usd, + "setup_human_min": setup_human_s / 60.0, + "item_model_usd": item_model_usd * per_v, + "compute_usd": compute_usd * per_v, + "operator_min": item_op_s / 60.0 * per_v, + "operator_usd": item_op_s * op_rate * per_v, + "maintenance_events": proj_events, + "maintenance_usd": maint_per_event_usd * proj_events, + "maintenance_min": maint_per_event_min * proj_events, + "silent_wrong": sw * per_v, + "cleanup_min": cleanup_s / 60.0 * per_v, + "cleanup_usd": cleanup_s * op_rate * per_v, + } + proj["total_usd"] = (proj["setup_usd"] + proj["item_model_usd"] + proj["compute_usd"] + + proj["operator_usd"] + proj["maintenance_usd"] + proj["cleanup_usd"]) + proj["total_human_min"] = (proj["setup_human_min"] + proj["operator_min"] + proj["maintenance_min"] + + proj["cleanup_min"]) + walls = sorted(float(r["wall_s"]) for r in automated) + return { + "arm": arm, + "describe": describe, + "items": n, + "verified": v, + "automated": len(automated), + "by_hand": len(by_hand), + "outcomes": dict(outcomes), + "silent_wrong": outcomes.get("silent_wrong", 0), + "silent_wrong_rate": round(outcomes.get("silent_wrong", 0) / n, 4) if n else None, + "silent_wrong_ci95": wilson(outcomes.get("silent_wrong", 0), n), + "false_stop": outcomes.get("false_stop", 0), + "false_stop_rate": round(outcomes.get("false_stop", 0) / len(clean), 4) if clean else None, + "false_stop_denominator": len(clean), + "false_stop_ci95": wilson(outcomes.get("false_stop", 0), len(clean)), + "drift_events": d, + "measured": { + "setup_human_min": round(setup_human_s / 60.0, 2), + "setup_model_usd": round(setup_model_usd, 4), + "item_model_usd": round(item_model_usd, 4), + "model_calls": model_calls, + "compute_usd": round(compute_usd, 4), + "operator_min": round(item_op_s / 60.0, 2), + "maintenance_human_min": round(maint_human_s / 60.0, 2), + "maintenance_model_usd": round(maint_model_usd, 4), + "maintenance_per_event_usd": round(maint_per_event_usd, 4), + "maintenance_per_event_min": round(maint_per_event_min, 2), + "median_wall_s_automated": round(statistics.median(walls), 2) if walls else None, + "p90_wall_s_automated": round(walls[int(0.9 * (len(walls) - 1))], 2) if walls else None, + "per_item_model_usd": round(item_model_usd / n, 4) if n else None, + }, + "per_1000_verified": {k: (round(x, 2) if isinstance(x, float) else x) for k, x in proj.items()}, + } + + +AXES = [ + # (key path, label, lower is better) + (("per_1000_verified", "total_usd"), "Total $ per 1,000 verified", True), + (("per_1000_verified", "setup_usd"), "Setup $ (people + model)", True), + (("measured", "per_item_model_usd"), "Model $ per item", True), + (("per_1000_verified", "operator_min"), "Operator minutes per 1,000 verified", True), + (("measured", "maintenance_per_event_usd"), "Maintenance $ per UI change", True), + (("silent_wrong_rate",), "Silent-wrong rate", True), + (("false_stop_rate",), "False-stop rate", True), + (("measured", "median_wall_s_automated"), "Median seconds per automated item", True), +] + + +def _get(d: dict[str, Any], path: tuple[str, ...]) -> Any: + for k in path: + d = d.get(k) if isinstance(d, dict) else None + return d + + +def winners(arms: list[dict[str, Any]]) -> list[dict[str, Any]]: + out = [] + for path, label, lower in AXES: + vals = [(a["arm"], _get(a, path)) for a in arms if isinstance(_get(a, path), (int, float))] + vals = [(k, v) for k, v in vals if v == v] # drop NaN + if not vals: + continue + best = min(v for _, v in vals) if lower else max(v for _, v in vals) + best_arms = [k for k, v in vals if v == best] + sw = {a["arm"]: a["silent_wrong"] for a in arms} + caveat = [k for k in best_arms if sw.get(k)] if path[-1] != "silent_wrong_rate" else [] + out.append({"axis": label, "best": best_arms, "value": best, "all": dict(vals), + "best_has_silent_wrong": caveat}) + return out + + +def compute(rows: list[dict[str, Any]]) -> dict[str, Any]: + cm = _cost_model(rows) + table = budget.prices() + labor = table["labor"] + drift_rate = float(table["drift"]["events_per_1000_items"]) + arms = [] + for name in dict.fromkeys(r["arm"] for r in rows if r.get("type") == "arm_start"): + arms.append(arm_economics(rows, name, cm, labor, drift_rate)) + bands = {} + for f in (0.5, 2.0): + scaled = {**labor, "operator_usd_per_hour": labor["operator_usd_per_hour"] * f} + bands[f"labor_x{f:g}"] = {a["arm"]: arm_economics(rows, a["arm"], cm, scaled, drift_rate)["per_1000_verified"]["total_usd"] + for a in arms} + return { + "prices_as_of": table["as_of"], + "labor": labor, + "drift_events_per_1000_items": drift_rate, + "cost_model": cm.get("version"), + "arms": arms, + "winners": winners(arms), + "bands": bands, + } + + +def _fmt(x: Any, nd: int = 2) -> str: + if x is None or (isinstance(x, float) and x != x): + return "n/a" + if isinstance(x, float): + return f"{x:,.{nd}f}" + return str(x) + + +def render(result: dict[str, Any]) -> str: + arms = result["arms"] + lines = ["| Arm | Items | Verified | Automated | Silent wrong | False stops | Setup $ | Model $/item | Operator min /1k | Maint. $/change | Total $ /1k verified |", + "|---|---|---|---|---|---|---|---|---|---|---|"] + for a in arms: + m, p = a["measured"], a["per_1000_verified"] + lines.append( + f"| {a['arm']} | {a['items']} | {a['verified']} | {a['automated']} | " + f"{a['silent_wrong']} ({_fmt(a['silent_wrong_rate'], 3)}) | " + f"{a['false_stop']} of {a['false_stop_denominator']} | {_fmt(p['setup_usd'])} | " + f"{_fmt(m['per_item_model_usd'], 3)} | {_fmt(p['operator_min'], 0)} | " + f"{_fmt(m['maintenance_per_event_usd'])} | {_fmt(p['total_usd'], 0)} |" + ) + lines += ["", "| Axis | Best | Value |", "|---|---|---|"] + for w in result["winners"]: + flag = " (has silent wrongs)" if w.get("best_has_silent_wrong") else "" + lines.append(f"| {w['axis']} | {', '.join(w['best'])}{flag} | {_fmt(w['value'], 3)} |") + return "\n".join(lines) + "\n" + + +def main(argv: Optional[list[str]] = None) -> int: + args = argv if argv is not None else sys.argv[1:] + run_dir = Path(args[0]) + ledgers = sorted(run_dir.glob("**/ledger.jsonl")) + result = compute(read_rows(ledgers)) + (run_dir / "economics.json").write_text(json.dumps(result, indent=1, default=str) + "\n") + print(render(result)) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/packages/flow/benchmark/lifecycle/arms/human_script.py b/packages/flow/benchmark/lifecycle/arms/human_script.py new file mode 100644 index 000000000..b8f4116b1 --- /dev/null +++ b/packages/flow/benchmark/lifecycle/arms/human_script.py @@ -0,0 +1,114 @@ +"""Arm ``human_script`` (optional): a Playwright + REST script written once by hand. + +The script (``human_script_v1.py``) was written by this benchmark's session the +way a skilled engineer would write it, from the same demonstration, procedure +and REST read the other arms get, and tested on the offline mock before it ran +against the fixture. Its authoring time is logged below. That time is a coding +agent's wall time, so it understates what a person would spend; the report +shows the setup charge with this measured time and with the cost model's +engineer assumption. + +No one maintains it during a run: when it breaks (an exception or an +"UNEXPECTED" stop), the item goes to the operator, and so does every later item +it breaks on. The fix an engineer writes afterwards is timed and reported as +the maintenance cost per drift event. +""" + +from __future__ import annotations + +import time +from pathlib import Path +from typing import Any, Optional + +from ..arm import Env, ItemResult, Operator, OperatorRequest, WorkItem +from . import budget, scripts +from .openemr_ui import OpenEMRSession, RestReader + +HERE = Path(__file__).resolve().parent + +#: Wall-clock authoring log (UTC), appended as the work happens. +AUTHORING = [ + {"what": "write v1 from the demonstration and SOP", "start": "2026-10-20T16:39:05Z", "end": "2026-10-20T16:42:39Z", + "note": "includes the first offline mock run (dry run, save, duplicate, typo date, missing date, failed save, interstitial)"}, +] + + +def authoring_seconds() -> float: + from datetime import datetime + + def t(s: str) -> float: + return datetime.strptime(s, "%Y-%m-%dT%H:%M:%SZ").timestamp() + + return sum(t(a["end"]) - t(a["start"]) for a in AUTHORING) + + +class HumanScriptArm: + name = "human_script" + + def __init__(self, options: dict[str, Any]) -> None: + self.name = str(options.get("name", self.name)) + self.version = str(options.get("version", "v1")) + self.headless = bool(options.get("headless", True)) + self.session: Optional[OpenEMRSession] = None + self.rest: Optional[RestReader] = None + self.source = (HERE / f"human_script_{self.version}.py").read_text() + self.breaks: list[dict[str, Any]] = [] + + def describe(self) -> dict[str, Any]: + return { + "arm": self.name, + "kind": "hand-written Playwright + REST script with read-back; not maintained during the run", + "version": self.version, + "authoring_log": AUTHORING, + "authoring_s": authoring_seconds(), + "per_item_model_calls": 0, + } + + def setup(self, env: Env, operator: Operator) -> list[dict[str, Any]]: + self.session = OpenEMRSession(env, headless=self.headless) + self.rest = RestReader(env) + self.session.start() + self.session.sign_in() + return [ + { + "phase": "author_script_by_hand", + "wall_s": authoring_seconds(), + # The engineer attends all of their own authoring time. + "attended_s": authoring_seconds(), + "model_calls": 0, + "cost_usd": 0.0, + "detail": {"authoring_log": AUTHORING, "note": "a coding agent's wall time; a person's would be longer"}, + } + ] + + def run_item(self, item: WorkItem, env: Env, operator: Operator) -> ItemResult: + assert self.session is not None and self.rest is not None + t0 = time.time() + out = scripts.run(self.source, self.session, self.rest, item.params, commit=True) + if out.broke: + self.breaks.append({"item_id": item.item_id, "reason": out.reason[-300:]}) + claimed = {"SAVED": "done", "NOT SAVED": "stopped"}.get(out.result, "error") + status = f"{out.result}: {out.reason}"[:500] + if claimed != "done": + operator.ask(OperatorRequest(kind="stop", item_id=item.item_id, reason=status[:300], surface="script result")) + wall = time.time() - t0 + return ItemResult( + item_id=item.item_id, + claimed=claimed, + product_status=status, + wall_s=round(wall, 2), + evidence=f"{self.name}/", + detail={"version": self.version, "broke": out.broke, "log": out.log[-10:], "usage": [], + "compute_usd": round(budget.compute_usd(wall), 6)}, + ) + + def after_drift(self, event: dict[str, Any], env: Env, operator: Operator) -> list[dict[str, Any]]: + return [] + + def close(self) -> None: + if self.session is not None: + self.session.close() + + +def factory(options: dict[str, Any]) -> HumanScriptArm: + return HumanScriptArm(options) diff --git a/packages/flow/benchmark/lifecycle/arms/human_script_v1.py b/packages/flow/benchmark/lifecycle/arms/human_script_v1.py new file mode 100644 index 000000000..9d76cd6bd --- /dev/null +++ b/packages/flow/benchmark/lifecycle/arms/human_script_v1.py @@ -0,0 +1,141 @@ +"""Register one referral in OpenEMR: the hand-written baseline script (v1). + +Written once by the session as a skilled engineer would, against the same +demonstration, procedure and REST read the other arms get. Authoring time is +in ``human_script.py`` (AUTHORING). Runs under the script contract in +``scripts.py``. +""" + +import datetime +import re +import time + + +def _name(v): + v = re.sub(r"\s+", " ", (v or "").strip()) + return v.title() if v.isupper() or v.islower() else v + + +def _dob(v): + v = (v or "").strip() + if not v: + return None + for fmt in ("%Y-%m-%d", "%m/%d/%Y", "%Y/%m/%d"): + try: + return datetime.datetime.strptime(v, fmt).strftime("%Y-%m-%d") + except ValueError: + continue + return "BAD" + + +def _one_char_apart(a, b): + return len(a) == len(b) and sum(x != y for x, y in zip(a, b)) == 1 + + +def _visible(form, sel): + return form.locator(sel + " >> visible=true").first + + +def _confirm_or_matches(page, timeout_s=30.0): + """('confirm', locator) when OpenEMR found no match, ('matches', text) otherwise.""" + + end = time.time() + timeout_s + while time.time() < end: + for fr in page.frames: + try: + btn = fr.locator("#confirmCreate") + if btn.count() and btn.first.is_visible(): + body = fr.locator("body").inner_text() + if "No matches were found" in body: + ready = btn.first.evaluate( + "e => e.ownerDocument.readyState === 'complete' && " + "typeof e.ownerDocument.defaultView.dlgclose === 'function'" + ) + if ready: + return "confirm", btn.first + else: + return "matches", body[:300] + except Exception: + continue + page.wait_for_timeout(250) + return None, None + + +def _fill(ctx, r): + form = ctx.form + form.locator("#form_title").select_option(label=r["title"]) if r.get("title") else None + for key in ("fname", "lname", "DOB"): + loc = _visible(form, f"#form_{key}") + loc.fill(r[key]) + loc.dispatch_event("change") + form.locator("#form_sex").select_option(label=r["sex"]) + contact = form.get_by_text("Contact", exact=True).first + contact.scroll_into_view_if_needed() + contact.click() + form.wait_for_timeout(300) + for key in ("street", "city", "postal_code", "phone_home", "email"): + loc = _visible(form, f"#form_{key}") + loc.fill(r.get(key, "") or "") + loc.dispatch_event("change") + _visible(form, "#form_state").select_option(label=r["state_label"]) + _visible(form, "#form_country_code").select_option(label=r.get("country_code") or "USA") + + +def register(ctx, referral): + r = dict(referral) + r["fname"], r["lname"] = _name(r.get("fname")), _name(r.get("lname")) + dob = _dob(r.get("DOB")) + if dob is None: + return {"result": "NOT SAVED", "reason": "no date of birth; return the referral to the sender"} + if dob == "BAD": + return {"result": "NOT SAVED", "reason": f"unreadable date of birth {r.get('DOB')!r}"} + r["DOB"] = dob + + same_name = [p for p in ctx.find_patients(fname=r["fname"], lname=r["lname"]) if "error" not in p] + for p in same_name: + if p.get("DOB") == dob: + return {"result": "NOT SAVED", "reason": f"patient already has a chart (pid {p.get('pid')})"} + if _one_char_apart(p.get("DOB") or "", dob) and (p.get("street") or "").upper() == r.get("street", "").upper(): + return {"result": "NOT SAVED", "reason": f"same person as pid {p.get('pid')}; the date of birth looks like a typo"} + + for attempt in (1, 2): + try: + _fill(ctx, r) + ctx.form.locator("#create").click(timeout=15_000) + except Exception as exc: + return {"result": "NOT SAVED", "reason": f"UNEXPECTED: form step failed: {str(exc).splitlines()[0][:200]}"} + kind, what = _confirm_or_matches(ctx.page) + if kind == "matches": + return {"result": "NOT SAVED", "reason": "OpenEMR's duplicate check lists a matching patient"} + if kind is None: + return {"result": "NOT SAVED", "reason": "UNEXPECTED: no duplicate-check dialog after Create"} + what.click() + if not ctx.commit: + return {"result": "NOT SAVED", "reason": "DRY RUN reached confirm"} + found = [] + for _ in range(20): + ctx.page.wait_for_timeout(500) + found = [p for p in ctx.find_patients(fname=r["fname"], lname=r["lname"], DOB=dob) if "error" not in p] + if found: + break + if len(found) == 1: + p = found[0] + if (p.get("street") or "").upper() == r["street"].upper() and (p.get("email") or "") == r["email"]: + return {"result": "SAVED", "reason": f"read back pid {p.get('pid')}"} + return {"result": "NOT SAVED", "reason": f"read-back differs for pid {p.get('pid')}"} + if len(found) > 1: + return {"result": "NOT SAVED", "reason": f"read-back found {len(found)} charts"} + ctx.log(f"attempt {attempt}: no chart after confirm") + if attempt == 1: + ctx.page.reload() + ctx.page.wait_for_timeout(1000) + # The runner's blank form is gone after a reload; reopen it. + ctx.page.evaluate( + "() => { window.navigateTab('/interface/new/new_comprehensive.php', " + "'openadapt-new-patient'); window.activateTabByName('openadapt-new-patient', true); }" + ) + ctx.page.wait_for_timeout(2000) + for fr in ctx.page.frames: + if "new_comprehensive.php" in (fr.url or ""): + ctx.form = fr + return {"result": "NOT SAVED", "reason": "the save did not land after two attempts"} diff --git a/packages/flow/benchmark/lifecycle/arms/mock_openemr.py b/packages/flow/benchmark/lifecycle/arms/mock_openemr.py new file mode 100644 index 000000000..39d73b54e --- /dev/null +++ b/packages/flow/benchmark/lifecycle/arms/mock_openemr.py @@ -0,0 +1,195 @@ +"""A small in-browser stand-in for OpenEMR's referral flow, for offline tests of the arms. + +It is not OpenEMR. It copies only what the arms touch: the sign-in page, the tab +shell's ``navigateTab``/``activateTabByName``, the New Patient form's control +ids, the collapsed Contact section, the duplicate-check popup with +``#confirmCreate`` and "No matches were found.", the save POST to +``new_comprehensive_save.php``, and a REST patient search. Served through +Playwright routes on ``http://mock-openemr.test``; nothing listens on a port. + +Drift kinds mirror the soak's: ``relabel`` renames the save button, +``interstitial`` puts a notice over the form that must be dismissed first. +""" + +from __future__ import annotations + +import json +import urllib.parse +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any, Optional + +BASE = "http://mock-openemr.test" + +LOGIN = """

OpenEMR

+ + +""" + +MAIN = """
+""" + +FORM = """ +%(interstitial)s + + + + + + +
Title
First Name
Last Name
DOB
Sex
+
+Contact + + +""" + +INTERSTITIAL = """
+

Release notes: the registration form has moved fields.

+
""" + + +@dataclass +class MockOpenEMR: + """The store and the routes. ``patients`` is the system of record.""" + + patients: list[dict[str, str]] = field(default_factory=list) + drift: set[str] = field(default_factory=set) + fail_next_save: int = 0 + logged_in: bool = False + + def seed(self, person: dict[str, str]) -> None: + self.add(person) + + def add(self, p: dict[str, str]) -> dict[str, str]: + row = {k: p.get(k, "") for k in ("title", "fname", "lname", "DOB", "sex", "street", "city", + "postal_code", "phone_home", "email")} + row["state"] = p.get("state") or ("MA" if p.get("state_label") == "Massachusetts" else p.get("state_label", "")) + row["country_code"] = p.get("country_code", "USA") + row["pid"] = str(len(self.patients) + 1) + row["uuid"] = f"mock-{row['pid']}" + self.patients.append(row) + return row + + def search(self, **q: str) -> list[dict[str, str]]: + def ok(p: dict[str, str]) -> bool: + return all(not v or p.get(k, "").lower() == str(v).lower() for k, v in q.items()) + + return [dict(p) for p in self.patients if ok(p)] + + # -- Playwright routes --------------------------------------------------------- + + def install(self, context: Any) -> None: + context.route(BASE + "/**", self._route) + + def _route(self, route: Any) -> None: + req = route.request + url = urllib.parse.urlparse(req.url) + path = url.path + html = None + if path.startswith("/interface/login"): + html = LOGIN + elif path == "/interface/main/tabs/main.php": + self.logged_in = True + html = MAIN + elif path == "/interface/new/new_comprehensive.php": + html = FORM % { + "save_label": "Register patient" if "relabel" in self.drift else "Create New Patient", + "interstitial": INTERSTITIAL if "interstitial" in self.drift else "", + } + elif path == "/interface/new/dup_check.php": + q = dict(urllib.parse.parse_qsl(url.query)) + matches = self.search(fname=q.get("fname", ""), lname=q.get("lname", "")) + if matches: + rows = "".join(f"
  • {m['fname']} {m['lname']} {m['DOB']}
  • " for m in matches) + html = f"

    Possible matches

      {rows}
    " + else: + html = ("

    No matches were found.

    " + "" + "") + elif path == "/interface/new/new_comprehensive_save.php": + if self.fail_next_save > 0: + self.fail_next_save -= 1 + route.fulfill(status=500, body="error") + return + self.add(json.loads(req.post_data or "{}")) + route.fulfill(status=200, body="ok") + return + if html is None: + route.fulfill(status=404, body="not found") + else: + route.fulfill(status=200, content_type="text/html", body=html) + + +class _Resp: + def __init__(self, data: Any) -> None: + self.status_code = 200 + self._data = data + + def json(self) -> Any: + return self._data + + +class MockRestSession: + def __init__(self, app: MockOpenEMR) -> None: + self.app = app + + def get(self, url: str, params: Optional[dict[str, str]] = None) -> _Resp: + return _Resp({"data": self.app.search(**(params or {}))}) + + +def mock_env(app: MockOpenEMR, workdir: Path) -> Any: + """An ``Env`` for the arms, pointed at the mock.""" + + from ..arm import Env + + return Env( + app="openemr", + ui_url=BASE, + api_url=BASE + "/rest", + accounts=(("clerk", "synthetic"),), + workdir=workdir, + seed=0, + extra={"context_hook": app.install, "rest_session": lambda: MockRestSession(app)}, + ) diff --git a/packages/flow/benchmark/lifecycle/arms/openemr_ui.py b/packages/flow/benchmark/lifecycle/arms/openemr_ui.py new file mode 100644 index 000000000..9c7501860 --- /dev/null +++ b/packages/flow/benchmark/lifecycle/arms/openemr_ui.py @@ -0,0 +1,266 @@ +"""The OpenEMR plumbing every non-OpenAdapt arm shares. + +All arms get the same start for every item: a browser signed in as the clerk, +looking at a blank New Patient form opened through OpenEMR's own tab shell +(the form needs ``top.restoreSession`` from that shell to save). When the +session has ended (the soak's ``session_expiry`` event), the same runner code +signs in again before handing the page over. This is the runner's job in a +deployment of any of these arms, so it is the same for all of them and is not +charged to the arm's model budget. It is recorded per item (``signed_in_again``). + +The REST read is read-only (the harness's ``rest_read_session``, scope +``user/patient.rs``), the same REST read-back OpenAdapt's record check uses. No arm ever sees the +harness's oracle. +""" + +from __future__ import annotations + +import time +from dataclasses import dataclass, field +from typing import Any, Optional + +FORM_PATH = "/interface/new/new_comprehensive.php" +FORM_FRAME_NAME = "openadapt-new-patient" +VIEWPORT = {"width": 1280, "height": 800} + +#: The fields a referral sets, with the New Patient form's control ids. +FORM_IDS = { + "title": "form_title", + "fname": "form_fname", + "lname": "form_lname", + "DOB": "form_DOB", + "sex": "form_sex", + "street": "form_street", + "city": "form_city", + "state_label": "form_state", + "postal_code": "form_postal_code", + "phone_home": "form_phone_home", + "email": "form_email", + "country_code": "form_country_code", +} + +SOP = """\ +Register one referred patient in OpenEMR from the referral below. +Rules (the clinic's standard procedure): +- Create exactly one new patient chart with the referral's values. Dates of \ +birth go in as YYYY-MM-DD; names as the person's name (trim stray spaces, \ +normal capitalisation). +- If OpenEMR shows that a matching patient may already exist, or you otherwise \ +find this person already has a chart (same name and date of birth, or the same \ +name and a date of birth that differs only by an obvious typo with the same \ +address), do not create another chart: stop and say so. +- If the referral has no date of birth, do not create a chart: stop and say \ +the referral goes back to the sender. A missing home phone is fine. +- If the save fails, you may try once more; never leave two charts. +- Work only in the OpenEMR page. All data is synthetic test data. +End your final reply with exactly one line, either +RESULT: SAVED +or +RESULT: NOT SAVED: """ + + +def claimed_from(text: str) -> tuple[str, str]: + """(``done`` | ``stopped``, last line) from an agent's final reply.""" + + lines = [ln.strip() for ln in (text or "").strip().splitlines() if ln.strip()] + last = lines[-1] if lines else "" + for ln in reversed(lines): + if ln.upper().startswith("RESULT:"): + last = ln + break + up = last.upper() + if up.startswith("RESULT: SAVED"): + return "done", last + return "stopped", last or "(no final reply)" + + +@dataclass +class OpenEMRSession: + """One Playwright browser, signed in, reused across items.""" + + env: Any + headless: bool = True + _pw: Any = None + _browser: Any = None + context: Any = None + page: Any = None + signins: int = 0 + log: list[dict[str, Any]] = field(default_factory=list) + + @property + def ui_url(self) -> str: + return self.env.ui_url.rstrip("/") + + def start(self) -> None: + if self.page is not None: + return + from playwright.sync_api import sync_playwright + + self._pw = sync_playwright().start() + self._browser = self._pw.chromium.launch(headless=self.headless) + self.context = self._browser.new_context(viewport=VIEWPORT, ignore_https_errors=True) + hook = getattr(self.env, "extra", {}).get("context_hook") + if hook is not None: # the offline mock serves the app through routes + hook(self.context) + self.page = self.context.new_page() + + def close(self) -> None: + for obj, how in ((self._browser, "close"), (self._pw, "stop")): + try: + if obj is not None: + getattr(obj, how)() + except Exception: # noqa: BLE001 - closing is best effort + pass + self._pw = self._browser = self.context = self.page = None + + def _account(self) -> tuple[str, str]: + if self.env.accounts: + return self.env.accounts[0] + fx = self.env.extra["fixture"] + v = fx._runtime_values() + return v["OPENEMR_ACTOR_USER"], v["OPENEMR_ACTOR_PASSWORD"] + + def sign_in(self) -> None: + page = self.page + user, password = self._account() + page.goto(self.ui_url + "/interface/login/login.php?site=default") + page.locator("#authUser").fill(user) + page.locator("#clearPass").fill(password) + page.locator("#login-button").click() + page.wait_for_url("**/interface/main/tabs/main.php**", timeout=90_000) + self.signins += 1 + self._dismiss_registration() + + def _shell_ready(self) -> bool: + try: + return bool( + self.page.evaluate( + "() => typeof window.navigateTab === 'function' " + "&& typeof window.activateTabByName === 'function'" + ) + ) + except Exception: # noqa: BLE001 + return False + + def _dismiss_registration(self) -> None: + reg = self.page.locator(".product-registration-modal") + try: + if reg.count() and reg.is_visible(): + tel = reg.locator("#allowTelemetry") + if tel.count() and tel.is_checked(): + tel.uncheck() + reg.get_by_role("button", name="Ask again later", exact=True).click() + reg.wait_for(state="hidden", timeout=30_000) + except Exception: # noqa: BLE001 - the dialog only shows on a fresh profile + pass + + def form_frame(self, timeout_s: float = 60.0) -> Any: + deadline = time.monotonic() + timeout_s + while time.monotonic() < deadline: + frame = self.page.frame(name=FORM_FRAME_NAME) + if frame is None: + try: + h = self.page.locator(f'iframe[name="{FORM_FRAME_NAME}"]').element_handle() + frame = None if h is None else h.content_frame() + except Exception: # noqa: BLE001 + frame = None + if frame is not None and FORM_PATH in (frame.url or ""): + return frame + self.page.wait_for_timeout(150) + raise RuntimeError("the New Patient form frame did not open") + + def blank_form(self) -> dict[str, Any]: + """Leave the page on a blank New Patient form; sign in again if needed.""" + + self.start() + t0 = time.time() + again = False + if not self._shell_ready() or "login" in (self.page.url or ""): + self.sign_in() + again = self.signins > 1 + # Close any earlier form tab, then open a fresh one. + try: + self.page.evaluate( + "({path, name}) => { window.navigateTab(path, name); " + "window.activateTabByName(name, true); }", + {"path": FORM_PATH + "?_t=" + str(int(time.time() * 1000)), "name": FORM_FRAME_NAME}, + ) + frame = self.form_frame() + frame.locator("#form_fname").wait_for(state="visible", timeout=60_000) + except Exception: + # The shell may have been logged out under us (expired session). + self.sign_in() + again = True + self.page.evaluate( + "({path, name}) => { window.navigateTab(path, name); " + "window.activateTabByName(name, true); }", + {"path": FORM_PATH, "name": FORM_FRAME_NAME}, + ) + frame = self.form_frame() + frame.locator("#form_fname").wait_for(state="visible", timeout=60_000) + self._dismiss_registration() + return {"setup_s": round(time.time() - t0, 2), "signed_in_again": again} + + +class RestReader: + """Read-only use of OpenEMR's patient API, as the clerk's integration.""" + + def __init__(self, env: Any) -> None: + self.env = env + self._session: Any = None + self._issued = 0.0 + + def _s(self) -> Any: + # Tokens are short-lived; take a fresh one every 5 minutes. + if self._session is None or time.time() - self._issued > 300: + getter = self.env.extra.get("rest_read_session") or self.env.extra.get("rest_session") + if getter is not None: + self._session = getter() + else: + self._session = self.env.extra["fixture"].token_session("actor") + self._issued = time.time() + return self._session + + @property + def base(self) -> str: + api = self.env.api_url or self.env.extra["fixture"].api_base_url + return api.rstrip("/") + "/apis/default/api" + + def find(self, **query: str) -> list[dict[str, Any]]: + q = {k: v for k, v in query.items() if v} + r = self._s().get(self.base + "/patient", params=q) + if r.status_code != 200: + return [{"error": f"HTTP {r.status_code}"}] + data = r.json() + rows = data.get("data") if isinstance(data, dict) else data + out = [] + for p in rows or []: + out.append( + { + k: p.get(k) + for k in ( + "uuid", "pid", "title", "fname", "lname", "DOB", "sex", + "street", "city", "state", "postal_code", "phone_home", "email", + ) + } + ) + return out + + +def fill_value(frame: Any, field_id: str, value: str) -> None: + """Set one New Patient control the way a person would.""" + + loc = frame.locator(f"#{field_id}") + tag = loc.evaluate("e => e.tagName.toLowerCase()") + if tag == "select": + try: + loc.select_option(label=value) + except Exception: # noqa: BLE001 - some lists key by value + loc.select_option(value=value) + else: + loc.fill(value) + loc.dispatch_event("change") + + +def deadline_ok(started: float, limit_s: Optional[float]) -> bool: + return limit_s is None or time.time() - started < limit_s diff --git a/packages/flow/benchmark/lifecycle/arms/prices.json b/packages/flow/benchmark/lifecycle/arms/prices.json new file mode 100644 index 000000000..19d8f4f8d --- /dev/null +++ b/packages/flow/benchmark/lifecycle/arms/prices.json @@ -0,0 +1,56 @@ +{ + "as_of": "2026-10-20", + "currency": "USD", + "unit": "per million tokens", + "source": "Anthropic first-party API list prices, as listed in the claude-api reference cached 2026-10-06 and checked 2026-10-20. Cache writes are the 5-minute TTL (1.25x input). Haiku 5.5 cache prices are derived from the same rule (0.1x read, 1.25x write) and are marked derived.", + "models": { + "claude-fable-5-1": { + "input": 10.0, + "output": 50.0, + "cache_read": 0.25, + "cache_write": 12.5 + }, + "claude-opus-5-5": { + "input": 4.0, + "output": 20.0, + "cache_read": 0.2, + "cache_write": 5.0 + }, + "claude-sonnet-5-5": { + "input": 2.0, + "output": 10.0, + "cache_read": 0.2, + "cache_write": 2.5 + }, + "claude-haiku-5-5": { + "input": 0.1, + "output": 0.5, + "cache_read": 0.01, + "cache_write": 0.125, + "derived": [ + "cache_read", + "cache_write" + ], + "note": "prompts up to 100K tokens; $0.50/$2.50 beyond" + } + }, + "compute": { + "note": "Local compute per item is charged at a nominal rate so arms that hold a browser longer cost more. One worker = one browser plus its share of one 4 vCPU / 16 GB host.", + "host_usd_per_hour": 0.2, + "host_source": "a nominal 4 vCPU / 16 GB on-demand cloud VM, round figure; change here, never in code" + }, + "labor": { + "note": "Assumptions, not measurements. People's time is priced only to put every component in one currency; the report also gives every component in minutes, and shows the total at half and double these rates.", + "operator_usd_per_hour": 30.0, + "operator_source": "round figure for a fully loaded US medical front-desk / registration clerk hour (wage plus overhead); assumption", + "engineer_usd_per_hour": 100.0, + "engineer_source": "round figure for a fully loaded automation engineer hour; assumption", + "engineer_script_author_s": 7200.0, + "engineer_script_fix_s": 1800.0, + "engineer_source_times": "assumed time for a skilled engineer to write and test this script (2 h) and to diagnose and fix it after one UI change (30 min); used only for the human_script arm's assumption column" + }, + "drift": { + "events_per_1000_items": 4, + "note": "assumed for the per-1,000 projection: the soak's four UI-change kinds, once each per 1,000 items (about one vendor release per few weeks at a clinic's referral volume). The measured stream has 4 changes in 40 items; the report gives maintenance per change so any rate can be applied." + } +} diff --git a/packages/flow/benchmark/lifecycle/arms/report.py b/packages/flow/benchmark/lifecycle/arms/report.py new file mode 100644 index 000000000..d55f045ff --- /dev/null +++ b/packages/flow/benchmark/lifecycle/arms/report.py @@ -0,0 +1,141 @@ +"""REPORT.md for one comparison run: tables from ``economics.json``, findings from ``findings.md``. + +The tables are generated. The findings (which arm wins on which axis, where +OpenAdapt loses, what the numbers do not show) are written by a person after +reading the tables, kept in ``findings.md`` beside the ledgers, and included +as is. +""" + +from __future__ import annotations + +import json +from pathlib import Path +from typing import Any + +from .economics import _fmt + + +def _components(result: dict[str, Any]) -> list[str]: + rows = [ + "| Arm | Setup: people min | Setup: model $ | Model $ per item | Compute $ per item | Operator min per item | Maint. people min per change | Maint. model $ per change | Median s per automated item |", + "|---|---|---|---|---|---|---|---|---|", + ] + for a in result["arms"]: + m = a["measured"] + n = a["items"] or 1 + rows.append( + f"| {a['arm']} | {_fmt(m['setup_human_min'], 1)} | {_fmt(m['setup_model_usd'])} | " + f"{_fmt(m['per_item_model_usd'], 3)} | {_fmt(m['compute_usd'] / n, 4)} | " + f"{_fmt(m['operator_min'] / n, 2)} | {_fmt(m['maintenance_per_event_min'], 1)} | " + f"{_fmt(m['maintenance_model_usd'] / a['drift_events'] if a['drift_events'] else 0.0)} | " + f"{_fmt(m['median_wall_s_automated'], 1)} |" + ) + return rows + + +def _projection(result: dict[str, Any]) -> list[str]: + rows = [ + "| Arm | Setup $ | Model $ | Compute $ | Operator min | Operator $ | Maint. changes | Maint. $ | Silent wrongs | Cleanup $ (lower bound) | Total $ | Total people min | Total $ at labor x0.5 / x2 |", + "|---|---|---|---|---|---|---|---|---|---|---|---|---|", + ] + for a in result["arms"]: + p = a["per_1000_verified"] + lo = result["bands"]["labor_x0.5"].get(a["arm"]) + hi = result["bands"]["labor_x2"].get(a["arm"]) + rows.append( + f"| {a['arm']} | {_fmt(p['setup_usd'])} | {_fmt(p['item_model_usd'])} | {_fmt(p['compute_usd'])} | " + f"{_fmt(p['operator_min'], 0)} | {_fmt(p['operator_usd'])} | {_fmt(p['maintenance_events'], 1)} | " + f"{_fmt(p['maintenance_usd'])} | {_fmt(p['silent_wrong'], 0)} | {_fmt(p['cleanup_usd'])} | **{_fmt(p['total_usd'], 0)}** | {_fmt(p['total_human_min'], 0)} | " + f"{_fmt(lo, 0)} / {_fmt(hi, 0)} |" + ) + return rows + + +def _quality(result: dict[str, Any]) -> list[str]: + rows = [ + "| Arm | Items | Verified (final) | Automated | Done by hand | Silent wrong (95% CI) | False stops (95% CI) | Outcomes |", + "|---|---|---|---|---|---|---|---|", + ] + def ci(c: Any) -> str: + return f" ({c[0]:.2f}–{c[1]:.2f})" if c else "" + + for a in result["arms"]: + outcomes = ", ".join(f"{k} {v}" for k, v in sorted(a["outcomes"].items())) + rows.append( + f"| {a['arm']} | {a['items']} | {a['verified']} | {a['automated']} | {a['by_hand']} | " + f"{a['silent_wrong']}{ci(a['silent_wrong_ci95'])} | " + f"{a['false_stop']} of {a['false_stop_denominator']}{ci(a['false_stop_ci95'])} | {outcomes} |" + ) + return rows + + +def _config(result: dict[str, Any]) -> list[str]: + out = [] + for a in result["arms"]: + d = a.get("describe") or {} + keep = {k: d[k] for k in ("kind", "model", "effort", "tool", "tools", "item_budget_usd", + "max_turns_per_item", "profile", "api_tier", "sees", "reads_record") + if k in d} + out.append(f"- **{a['arm']}**: `{json.dumps(keep, default=str)}`") + return out + + +def render(result: dict[str, Any], out: Path, plan: dict[str, Any]) -> str: + labor = result["labor"] + parts = [ + "# Lifecycle economics: OpenAdapt against the alternatives", + "", + f"Seed {plan['seed']}, {plan['items']} items over {plan['days']} days on the local OpenEMR test " + f"fixture (synthetic patients only). Every arm ran the same worklist and the same event ledger " + f"(UI changes, duplicates, expiry, concurrent edits, failed saves) and was judged by the same " + f"independent oracle and the same scripted operator. Model prices as of {result['prices_as_of']} " + f"(`arms/prices.json`); operator cost model `{result['cost_model']}`; people's time at " + f"${labor['operator_usd_per_hour']:.0f}/h (an assumption; bands below).", + "", + "## Total cost per 1,000 verified items", + "", + "Setup once, per-item costs scaled to 1,000 verified items, maintenance at " + f"{result['drift_events_per_1000_items']:g} UI changes per 1,000 items (assumed; per-change cost " + "is measured and shown below).", + "", + *_projection(result), + "", + "## Which arm wins on which axis", + "", + "| Axis | Best | Value | All arms |", + "|---|---|---|---|", + *[ + f"| {w['axis']} | {', '.join(w['best'])}{' (has silent wrongs)' if w.get('best_has_silent_wrong') else ''} | {_fmt(w['value'], 3)} | " + + ", ".join(f"{k} {_fmt(v, 3)}" for k, v in w["all"].items()) + + " |" + for w in result["winners"] + ], + "", + "## Correctness", + "", + *_quality(result), + "", + "## Measured components", + "", + *_components(result), + "", + "## Arm configurations", + "", + *_config(result), + "", + ] + findings = out / "findings.md" + if findings.is_file(): + parts += [findings.read_text().strip(), ""] + parts += [ + "## Reproduce", + "", + "```bash", + "cd packages/flow/benchmark", + f"python -m lifecycle.arms.compare --out {out.as_posix().split('benchmark/')[-1]} " + f"--seed {plan['seed']} --items {plan['items']} --days {plan['days']}", + f"python -m lifecycle.arms.economics {out.as_posix().split('benchmark/')[-1]} # recompute from the ledgers", + "```", + "", + ] + return "\n".join(parts) diff --git a/packages/flow/benchmark/lifecycle/arms/scripts.py b/packages/flow/benchmark/lifecycle/arms/scripts.py new file mode 100644 index 000000000..db8739b22 --- /dev/null +++ b/packages/flow/benchmark/lifecycle/arms/scripts.py @@ -0,0 +1,176 @@ +"""What both script arms share: the script contract, the sandbox, and the runner. + +A script is a Python module that defines:: + + def register(ctx, referral: dict) -> dict: + # returns {"result": "SAVED" | "NOT SAVED", "reason": str} + +``ctx`` gives the script exactly what the other arms get: + +- ``ctx.page``: the Playwright page, already signed in, on a blank New Patient + form; ``ctx.form``: that form's frame. +- ``ctx.find_patients(fname=, lname=, DOB=)``: the clerk's REST search. +- ``ctx.commit``: False in a dry run. In a dry run the save request + (``new_comprehensive_save.php``) is blocked at the network layer, so a dry + run cannot write, whatever the script does. +- ``ctx.log(msg)``: a line in the run's evidence. + +The sandbox is a static check, not isolation: a script may import only +``re``, ``time``, ``datetime`` and ``json``, and may not name ``open``, +``exec``, ``eval``, ``compile``, ``__import__``, ``globals``, ``os``, +``sys`` or ``subprocess``. Scripts touch only the local synthetic fixture. +""" + +from __future__ import annotations + +import ast +import time +import traceback +from dataclasses import dataclass, field +from typing import Any, Callable + +ALLOWED_IMPORTS = {"re", "time", "datetime", "json"} +FORBIDDEN_NAMES = { + "open", "exec", "eval", "compile", "__import__", "globals", "locals", "os", "sys", + "subprocess", "importlib", "builtins", "__builtins__", "breakpoint", "input", +} +SAVE_URL_PART = "new_comprehensive_save.php" + + +class ScriptRejected(ValueError): + """The script failed the static check.""" + + +def check_source(source: str) -> None: + tree = ast.parse(source) + for node in ast.walk(tree): + if isinstance(node, ast.Import): + for a in node.names: + if a.name.split(".")[0] not in ALLOWED_IMPORTS: + raise ScriptRejected(f"import {a.name} is not allowed") + elif isinstance(node, ast.ImportFrom): + if (node.module or "").split(".")[0] not in ALLOWED_IMPORTS: + raise ScriptRejected(f"from {node.module} import is not allowed") + elif isinstance(node, ast.Name) and node.id in FORBIDDEN_NAMES: + raise ScriptRejected(f"name {node.id} is not allowed") + elif isinstance(node, ast.Attribute) and node.attr.startswith("__"): + raise ScriptRejected(f"attribute {node.attr} is not allowed") + if not any(isinstance(n, ast.FunctionDef) and n.name == "register" for n in tree.body): + raise ScriptRejected("the script must define register(ctx, referral)") + + +def load(source: str) -> Callable[..., dict[str, Any]]: + check_source(source) + ns: dict[str, Any] = {"__name__": "lifecycle_script"} + exec(compile(source, "