[abrichr]

This PR measures, on one production configuration, whether OpenAdapt removes operator work. It covers the whole lifecycle, including an app change. It is the evidence for OpenAdaptAI/OpenAdapt#1514 and is stacked on it. It resumes and supersedes the harness on draft #1448.

Configuration: proof-15's, on local OpenEMR with synthetic patients:

• Regulated, problems_only;
• durable, pipelined read-back;
• governed run gate, pinned digest;
• repair on.

Results

At x1 of the cost model. The full table, bands and causes are in packages/flow/benchmark/lifecycle/results/production-2026-10-24/README.md.

	before (main)	after
Verified items per operator-minute	0.033	0.327
Entering every item by hand	0.333	0.333
Charts saved by the program and confirmed by the oracle	0 of 27	30 of 55
False stops per 100 items	70.4	16.4
Silent wrong	0	0 of 30
Wrong writes caught by the record check	0	2

• before was stopped early at 27 items, at the coordinator's request. Every item had stopped.
• Headline: about break-even on this stream. 0.327 verified items per operator-minute against 0.333 by hand, 0.98 times in every cost-model band. The README defines the numerator and denominator exactly. As a secondary view, the stream took 125.5 operator minutes against 165 by hand. The operator time went mostly to two places:
  • stops at the duplicate check for new patients who share a last name or date of birth with an existing chart: a person decides by design;
  • 30 approvals. The problems_only switch turned off after a malformed date of birth was saved wrong and caught.
    25.0 of the 125.5 minutes (20%) were these approvals. Whether the switch should re-arm on its own is a policy decision and isn't changed here. The date fix (ISO conversion at intake, and a typed date refused before any step) is in Relabel repair, a duplicate check that reads the match list, ISO dates at intake, and two re-admission and approval fixes #1514 but not in this measured run.
• Recovery after the relabel took 8.3 operator minutes, measured live:
  • a proof that only the label changed;
  • re-qualification on the new release (763 s of machine time);
  • one review and a fresh admission;
  • the stopped item re-run.
    It took 896 s from the stop to the item done. Before this change, every item after the app change stopped until a person re-taught the step.

Re-run with the fixes (2026-10-25)

The same 55-item stream and app change, after #1514, the switch policy (#1517) and the record-channel duplicate check over FHIR (#1516, still a draft). Measured at 38fc21a. Evidence: packages/flow/benchmark/lifecycle/results/production-2026-10-25-rerun/.

	before	first after	re-run
Verified items per operator-minute (3 min/item by hand)	0.033	0.327	0.658
Ratio to by hand at 3 / 5 / 8 / 10 min per item		0.98 / 1.34 / 1.68 / 1.83	1.97 / 2.98 / 4.18 / 4.83
False stops per 100 items	70.4	16.4	1.8
Silent wrongs	0	0 of 30	0 of 40
Wrong writes caught	0	2	0
Items right at the end	27 of 27	53 of 55	55 of 55

• Phase 0: throughput criterion not met (1.97×, just under 2×). The silent-wrong criterion is met (0). The fault-coverage criterion is measured separately (fault gate: admit a check only when it catches every class it claims (#1506) #1539 / independent fault suite).
• Deviation. The run used problems_only switch policy: input problems leave it on, a person re-arms with one signed tap #1517 at 6b52dd3, which didn't count the record check's "exact" and "close" matches as duplicate candidates. So the first real duplicate (item 12) turned the switch off, and every later save needed an approval: 27.5 of 77.5 operator minutes. No run of 10 verified items completed, so no re-arm was offered. problems_only switch policy: input problems leave it on, a person re-arms with one signed tap #1517 as merged classes these correctly, so this run understates the policy.
• Next. Superseded by the final run below.
• Where the harness code is. The re-run's harness changes (per-item timeout, re-arm offered to the operator, FHIR fixture patches) are on measure/production-lifecycle-rerun. They land after Duplicate check asks the system of record (FHIR/REST, SQL fallback) instead of the screen list #1516.

Final run on main (2026-10-25)

This is the same stream and app change on origin/main 78ff5ad, which includes #1514, #1517 as merged and #1557, plus #1516 (4b635f2) and the harness. Measured at 8bfd7f6. Evidence: packages/flow/benchmark/lifecycle/results/production-2026-10-25-final/.

	re-run	final
Verified items per operator-minute (3 min/item by hand)	0.658	1.055
Ratio to by hand at 3 / 5 / 8 / 10 min per item	1.97 / 2.98 / 4.18 / 4.83	3.17 / 4.53 / 5.97 / 6.68
False stops per 100 items	1.8	0
Silent wrongs	0 of 40	0 of 41
Approvals	33	1
Re-arm minutes	0	0
Operator minutes	77.5	48.3
Items right at the end	55 of 55	55 of 55

• Phase 0: throughput and silent-wrong criteria met (3.17 times hand entry, 0 silent wrongs). The fault-coverage criterion is measured separately (fault gate: admit a check only when it catches every class it claims (#1506) #1539 / independent fault suite), so this run doesn't show that Phase 0 is met overall. The per-touch charges are the cost model's assumptions, not timed people.
• Items 7 to 10 ran with host load above 40 (up to 46.5). They are counted in the numbers above. Without them the rate is 1.036 per operator-minute, or 3.11 times hand entry.
• Item 26 was stopped by a harness defect. The scripted approval after the relabel repair was written unsigned. Main has refused unsigned approvals since ea96cc4, so the gate stopped before saving and nothing was written. The item was entered by hand, so the headline under-states the program by one item.
• An earlier attempt on main was blocked at admission by a census regression from fix(flow): halt when another shown row has the same identity #1504. It was fixed in fix(flow): the census follows the target's teach-time role on every rung #1557. Evidence: results/production-2026-10-25-final-blocked/.

What's in it

• benchmark/lifecycle. This is Lifecycle soak harness: work removed per operator-hour (draft) #1448's soak harness, resumed. It adds:
  • the headline (verified items per operator-minute, false stops per 100 items, operator minutes per stop, maintenance minutes and time to running again after an app change);
  • stops by cause, and the field each duplicate-check stop shared;
  • --drift KIND@SEQ with relabel_save;
  • production_arm.py (proof-15's configuration as an arm);
  • workload kinds common_surname and shared_dob.
• benchmark/lifecycle_proof. Soak items bring their own values. The DB dump is binary-safe and leaves OpenEMR's OAuth keys table alone.
• docs/PILOT_MEASUREMENT.md. The internal protocol for measuring a pilot at a customer, and how the harness records the same numbers.
• Results under benchmark/lifecycle/results/production-2026-10-24/: before/, after/ and recovery-smoke/, each with its ledger, workload, results, report, stops and switch logs.

Limits

• Operator times come from the versioned cost model, not timed people.
• One seed.
• Some items ran at a 1-minute load above 40 (max 102).
• The re-admitted version's switch record is opened locally, as proof-15 does.

I scanned the committed files for the fixture's credentials and keys and found none. Paths in the committed results are relative to the run root (runs/<name>/...), and the harness now writes them that way (benchmark/lifecycle/paths.py, with a test). The reports recompute to the same results.json. No prospect names.

🤖 Generated with Claude Code