Skip to content

Economic comparison: OpenAdapt vs agent with tools, agent-written script and computer-use, lifecycle totals (evidence) #1528

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
abrichr wants to merge 2 commits into
base: main
Choose a base branch
from
+7,951 −1
Open

Economic comparison: OpenAdapt vs agent with tools, agent-written script and computer-use, lifecycle totals (evidence) #1528

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
40a19d2
Economic comparison: competing arms, MockMed and OpenEMR results, lif…
abrichr Oct 25, 2026
b5be96f
Economic comparison: review fixes
abrichr Oct 25, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions packages/flow/benchmark/economic_comparison/.gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
results/totals.json
337 changes: 337 additions & 0 deletions packages/flow/benchmark/economic_comparison/README.md
View file
Open in desktop

Large diffs are not rendered by default.

Empty file added packages/flow/benchmark/economic_comparison/__init__.py
View file
Open in desktop
Empty file.
100 changes: 100 additions & 0 deletions packages/flow/benchmark/economic_comparison/assumptions.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
{
"version": "economic-comparison/2026-10-24.1",
"note": "Every number here is an assumption, not a measurement. totals.py takes measured rates from the raw results and these assumptions, and shows the totals at half and double the people-time assumptions. Change a number here, never in code.",
"labor_usd_per_hour": {
"operator": 30.0,
"engineer": 100.0,
"source": "lifecycle/arms/prices.json (labor): round figures for a fully loaded front-desk clerk hour and automation engineer hour"
},
"operator_s": {
"source": "lifecycle/cost_model.py V1 (lifecycle-cost/2026-10-20.1); no real operator timings exist yet",
"stop": 90.0,
"reconcile": 120.0,
"repair_review": 120.0,
"manual_entry_mockmed": 60.0,
"manual_entry_openemr": 180.0,
"interrupt": 30.0,
"manual_entry_mockmed_why": "a MockMed triage note is one click path and one note field, about a third of the 12-field OpenEMR registration the cost model's 180 s is for"
},
"per_outcome": {
"note": "What a person does after each outcome. One interrupt per item that reaches a person.",
"correct": [],
"claimed_failure_save_possible": [
"stop",
"manual_entry"
],
"claimed_failure_no_save_possible": [
"stop"
],
"flagged_wrong": [
"stop",
"reconcile",
"manual_entry"
],
"false_alarm": [
"stop",
"reconcile"
],
"silent_wrong": [
"reconcile",
"manual_entry",
"interrupt"
],
"silent_wrong_why": "lower bound: someone later finds the wrong record, fixes it and enters the item again (two interruptions: when found and when fixed). Clinical or business harm is not priced."
},
"setup_people_min": {
"note": "One-time people time per workflow, by role. The agent's own authoring time and model cost are measured and reported separately.",
"openadapt": {
"operator": 22.0,
"engineer": 30.0,
"why": "cost model V1: teach preparation 5 min + a 2 min demonstration + compile review 10 min + admission 5 min; plus an engineer's 30 min to install it and write the record check (verify.yaml), the same read-back work the agent arms' engineer does"
},
"openadapt_regulated_openemr": {
"operator": 22.0,
"engineer": 30.0,
"why": "as above, plus an engineer's 30 min for the Regulated deployment (keys, sealed bundle, durable store, problems-only switch); the qualification campaign runs unattended"
},
"agent_tools": {
"operator": 0.0,
"engineer": 60.0,
"why": "an engineer wires a browser-tools agent (an off-the-shelf framework gives read_screen/click/fill), writes the record read-back tool and the check rule into the prompt, and tries it"
},
"agent_script": {
"operator": 0.0,
"engineer": 30.0,
"why": "an engineer prompts the coding agent with the demonstration and the API description, reads and tests the script it submits"
},
"cua_alone": {
"operator": 0.0,
"engineer": 30.0,
"why": "an engineer writes the task prompt and deploys the computer-use loop"
},
"human_script": {
"operator": 0.0,
"engineer": 120.0,
"why": "lifecycle/arms/prices.json engineer_script_author_s (2 h)"
}
},
"maintenance_people_min": {
"note": "People time per maintenance event, by kind.",
"script_fix_review_engineer": 10.0,
"script_fix_review_why": "an engineer reads the coding agent's diff and its dry run, and deploys it",
"script_fix_by_hand_engineer": 30.0,
"script_fix_by_hand_why": "lifecycle/arms/prices.json engineer_script_fix_s (30 min), when the coding agent's own fix did not hold",
"openadapt_reteach_operator": 22.0,
"openadapt_reteach_why": "a change OpenAdapt neither handled nor self-repaired: the person teaches the workflow again (same as setup)",
"agent_prompt_fix_engineer": 15.0,
"agent_prompt_fix_why": "a change an agent kept failing on: an engineer adjusts the prompt"
},
"detection": {
"items_until_a_silent_wrong_is_noticed": null,
"note": "Silent wrongs are charged one cleanup each; how long they go unnoticed is not modelled."
},
"volume": {
"ui_changes_per_1000_items": 4,
"why": "lifecycle/arms/prices.json drift rate: about one vendor release every few weeks at a clinic's referral volume",
"server_faults_per_1000_items": 10,
"server_faults_why": "assumed: one save in 100 goes wrong on the server side. The MockMed suite has 5 in 100 to measure detection; the excess cost per fault item is measured there and applied at this rate."
},
"compute_usd_per_hour": 0.2
}
6 changes: 6 additions & 0 deletions packages/flow/benchmark/economic_comparison/openemr_arms.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
[
["agent_tools", "lifecycle.arms.agent_tools:factory", {"model": "claude-sonnet-5-5", "effort": "medium", "item_budget_usd": 0.50}],
["agent_script", "lifecycle.arms.agent_script:factory", {"model": "claude-sonnet-5-5", "effort": "high", "author_budget_usd": 3.0, "fix_budget_usd": 1.5}],
["human_script", "lifecycle.arms.human_script:factory", {}],
["cua_alone", "lifecycle.arms.cua:factory", {"model": "claude-sonnet-5-5", "effort": "medium", "item_budget_usd": 0.80, "stop_after": 12}]
]
56 changes: 56 additions & 0 deletions packages/flow/benchmark/economic_comparison/results/TABLE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
## OpenEMR referral intake (seed 7, 55 items, one app change: the save button relabeled at item 26)

| Arm | Items | Verified in the end | Automated (no hand entry) | Silent wrong | False stops | Median s/item | Model $/item | Operator min/item | Maintenance for the change (people min / model $) | Setup people min | Setup agent min / $ |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 3. OpenAdapt, Regulated + problems_only, durable, pipelined read-back, relabel repair (engine of measure/production-lifecycle; the API step is refused on this form, so the GUI save runs) | 55 | 53 | 41 | 0 | 9 of 35 | 118.5 | 0.000 | 2.15 | 7.5 / 0.00 | 52 | 13.3 / 0.00 |
| 3 on main. Same configuration, current main's engine [short run: 27 of 55 planned items; the ledger ends before the planned items (the run exited or was killed)] | 27 | 27 | 4 | 0 | 19 of 20 | 226.9 | 0.000 | 4.56 | 0.0 / 0.00 | 52 | 13.4 / 0.00 |
| 1. Agent with structured tools and REST read-back (Claude Sonnet 5.5) | 55 | 55 | 49 | 0 | 5 of 35 | 24.3 | 0.050 | 0.91 | 0.0 / 0.00 | 60 | 0.0 / 0.00 |
| 2. Agent-written Playwright + REST script with read-back and duplicate check, agent-maintained (Claude Sonnet 5.5) [short run: 16 of 55 planned items; hung: no progress for about 2 h after item 16 (process at 0% CPU, one Playwright driver idle); stopped by hand to release the shared fixture lease] | 16 | 16 | 7 | 0 | 7 of 12 | 5.3 | 0.000 | 2.44 | not reached | 70 | 2.6 / 0.86 |
| Reference: a person enters every item (cost model V1) | None | None | 0 | 0 | None of None | 210.0 | 0.000 | 3.50 | 0.0 / 0.00 | 0 | 0.0 / 0.00 |

### Totals (people time priced at the assumed rates; x0.5 and x2 in brackets)

| Arm | 10 items: total $ | people min | 100 items: total $ | people min | 1000 items: total $ | people min |
|---|---|---|---|---|---|---|
| openadapt-production-relabel | 72 [36-144] | 74 | 171 [86-340] | 270 | 1,158 [584-2,307] | 2,228 |
| openadapt-production | 84 [42-168] | 98 | 290 [146-579] | 508 | 2,351 [1,181-4,689] | 4,608 |
| agent_tools | 105 [53-210] | 69 | 151 [78-296] | 151 | 606 [329-1,161] | 969 |
| agent_script | 130 [65-259] | 94 | 239 [120-478] | 314 | 1,337 [669-2,672] | 2,508 |
| all_manual | 18 [9-35] | 35 | 175 [88-350] | 350 | 1,750 [875-3,500] | 3,500 |

## MockMed, fault set 1.1.0 (100 items, 10 UI changes, 5 server faults)

| Arm | Verified (of 90) | Silent wrong | False stops | Faults claimed saved (of 5) | Stopped when no save possible (of 5) | Control s/item (load) | Model $/item | Setup people min | Setup agent min / $ |
|---|---|---|---|---|---|---|---|---|---|
| 4. Computer-use agent, screenshots only (Claude Sonnet 5.5) | 90 | 5 | 0 | 5 | 5 | 22.5 (17) | 0.0386 | 30 | 0.0 / 0.000 |
| 1. Agent with structured tools and record read-back (Claude Sonnet 5.5) | 90 | 0 | 0 | 0 | 5 | 13.7 (15) | 0.0132 | 60 | 0.0 / 0.000 |
| 2. Agent-written Playwright script with read-back and duplicate guard, agent-maintained | 90 | 0 | 0 | 0 | 5 | 0.8 (6) | 0.0000 | 30 | 0.4 / 0.026 |
| 3. OpenAdapt admitted program, read-back on, no model calls | 71 | 0 | 19 | 0 | 5 | 7.3 (8) | 0.0000 | 52 | 0.0 / 0.000 |
| 3r. OpenAdapt program with self-repair (engine of 2026-10-19) | 62 | 0 | 28 | 0 | 5 | 7.4 (10) | 0.0000 | 52 | 0.0 / 0.000 |
| 4 + OpenAdapt read-back: the same agent runs, each claim gated on verify-run | 90 | 0 | 0 | 0 | 5 | 22.5 (17) | 0.0386 | 30 | 0.0 / 0.000 |

### Each UI change: excess people minutes (silent wrongs in the block)

| Change | cua_alone | agent_tools | agent_script | openadapt_program | openadapt_self_repair | cua_plus_openadapt_readback |
|---|---|---|---|---|---|---|
| renamed controls | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| display scaling | 0.0 | 0.0 | 0.0 | 0.0 | 49.0 +fix | 0.0 |
| moved controls | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| new interstitial | 0.0 | 0.0 | 10.0 | 52.0 +fix | 52.0 +fix | 0.0 |
| latency | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| reordered queue | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| renamed and moved New Encounter | 0.0 | 0.0 | 10.0 | 49.0 +fix | 49.0 +fix | 0.0 |
| look-alike referral | 0.0 | 0.0 | 10.0 | 0.0 | 0.0 | 0.0 |
| relabeled and reordered type | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
| target referral gone | 10.0 | 10.0 | 40.0 | 10.0 | 10.0 | 10.0 |

### Totals (people time priced at the assumed rates; x0.5 and x2 in brackets)

| Arm | 10 items: total $ | people min | silent wrongs | 100 items: total $ | people min | silent wrongs | 1000 items: total $ | people min | silent wrongs |
|---|---|---|---|---|---|---|---|---|---|
| cua_alone | 51 [26-101] | 30 | 0.1 | 56 [30-108] | 34 | 1.0 | 110 [75-179] | 69 | 10.0 |
| agent_tools | 100 [50-201] | 60 | 0.0 | 104 [52-206] | 64 | 0.0 | 135 [74-256] | 102 | 0.0 |
| agent_script | 51 [25-101] | 31 | 0.0 | 56 [28-112] | 37 | 0.0 | 111 [56-222] | 96 | 0.0 |
| openadapt_program | 61 [31-123] | 53 | 0.0 | 65 [33-130] | 60 | 0.0 | 104 [53-206] | 134 | 0.0 |
| openadapt_self_repair | 61 [31-123] | 53 | 0.0 | 66 [33-132] | 62 | 0.0 | 110 [55-219] | 148 | 0.0 |
| cua_plus_openadapt_readback | 51 [26-101] | 30 | 0.0 | 56 [30-108] | 34 | 0.0 | 111 [76-182] | 72 | 0.0 |
Loading
Loading