[abrichr]

Evidence: OpenAdapt against competent alternatives on the same task, counting every person's minute and total cost over 10, 100 and 1,000 items. Write-up and tables: packages/flow/benchmark/economic_comparison/README.md.

Stacked on #1513 (production-lifecycle soak, which sits on #1514). Split from #1441: this PR carries only code, the raw rows the tables come from, and the README. The agents' per-item traces and the per-call spend ledger are a release asset in the private openadapt-workspace repo (tag evidence-economic-comparison-2026-10-24, sha256 906a9daf…).

What was measured (2026-10-24, synthetic data)

OpenEMR referral intake. Seed 7, 55 items, the save button relabeled at item 26.

Arm	Automated of 55	Silent wrong	False stops	Median s/item	Operator min/item	Model $/item	Total $ for 1,000 items
OpenAdapt, Regulated + problems_only, durable, pipelined read-back (#1513's engine; the API step is refused on this form)	41	0	9 of 35	118.5	2.15	0	1,158
OpenAdapt, same configuration, current main (27 items)	4	0	19 of 20	226.9	4.56	0	2,351
Agent with structured tools + REST read-back (Sonnet 5.5)	49	0	5 of 35	24.3	0.91	0.050	606
Agent-written script, agent-maintained (hung after 16 items; partial)	7 of 16	0	7 of 12	5.3	2.44	0	not comparable
By hand	0	-	-	210	3.50	0	1,750

MockMed silent-saves 1.1.0. 100 items, 10 UI changes, 5 server-side bad saves.

Arm	Verified of 90	Silent wrong	False stops	Bad saves reported saved (of 5)	Model $ for 100	s/item	Total $ for 1,000
Agent with tools + read-back	90	0	0	0	1.45	13.7	135
Agent-written script, agent-maintained	90	0	0	0	0.26	0.8	111
OpenAdapt program (current main)	71	0	19	0	0	7.3	104
Screenshot computer-use agent (2026-10-19 runs)	90	5	0	5	3.90	22.5	110

Where a baseline beats OpenAdapt. On OpenEMR, the agent with tools wins every row except model cost and setup. On MockMed, both competent alternatives verify more items and have fewer false stops. OpenAdapt's silent-wrong advantage holds only against the agent that trusts the screen; every arm that reads the record back had 0.

People time is priced from economic_comparison/assumptions.json (cost model V1, $30/h operator, $100/h engineer; shown at x0.5 and x2). It is not measured.

Not done

• OpenEMR: the hand-written script and the screenshot agent were not run, and the agent script stopped at item 16. They are queued as baselines-rest in the fixture queue. The agent script will rerun in full under the new watchdog. That run also adds a hybrid arm: the agent with tools calling OpenAdapt-taught programs (taught_<name>) with read-back.
• Operator minutes come from cost model V1. The README describes the planned rerun with timed people.
• The MockMed OpenAdapt program is not the Regulated configuration.

Code

• silent_saves/agent_arms.py: the two new MockMed executors.
• economic_comparison/totals.py and assumptions.json: the totals.
• lifecycle.arms.compare: --drift, and a per-item watchdog. On a timeout it dumps every thread's stack to stacks.txt, kills the browser driver, records an error row and gives the arm a fresh browser; it hard-exits if the arm still does not return (tests/test_lifecycle_watchdog.py).

Model spend: $6.05 of the $30 cap (lifecycle/results/spend.json). Tests: test_lifecycle_watchdog.py, test_lifecycle_arms.py, test_lifecycle_arms_mock.py and test_lifecycle_benchmark.py, 34 passing locally. No public copy changed.

🤖 Generated with Claude Code

[abrichr]

Coordinator review (AI-assisted). Approved, with small fixes before merge.

The numbers reproduce: economic_comparison.totals regenerates results/TABLE.md identically, and spot recomputations match. Fairness is acceptable and disclosed.

Fixes:

1. Remove the absolute local paths in silent_saves/results/1.1.0/openadapt-program/2026-10-24*/meta.json:7 and the ~/oa/scratch/... reference in economic_comparison/README.md:331.
2. lifecycle/arms/economics.py:90-97 takes n from the item_result rows. Compare it with the planned item count and fail or label any run that came up short, so a hard exit can't shrink the denominator without notice.
3. Reword the load note: the ledgers peak at 79 and 102, not "about 5 to 20".