Skip to content

generate-parts: the app-specific parts of a workflow, generated and verified - #1520

Draft
abrichr wants to merge 4 commits into
mainfrom
claude/generate-parts
Draft

abrichr wants to merge 4 commits into
mainfrom
claude/generate-parts

Conversation

@abrichr

@abrichr abrichr commented Oct 25, 2026 •

Copy link
Copy Markdown
Member

Draft, parked. Before review it needs two things: the live OpenEMR back-test (queued for the shared fixture slot), and approve signing with a registered approver key on the #1486 path instead of a typed name. The pagination engine fix moved to #1538; this branch drops it when #1538 merges.

openadapt-flow generate-parts builds the four parts of a workflow that are specific to one application at setup time. Each is admitted only by a check that does not depend on the generator. Healthy runs still make no model calls, and nothing in this PR calls a model. Design: packages/flow/docs/GENERATED_PARTS.md.

Generated, and how each is admitted

Part Generated from Admitted by
Read channel (generate/channel.py) OpenAPI document, FHIR CapabilityStatement, or teach-time traffic Channel test. It shows each demonstration's record exactly once and reads every page. Through a fault proxy it must read as unreadable, never as an empty or partial set: no auth, 401, 500, truncated body, not JSON, a failed next page
Record contract (generate/contract.py) Before/after difference across 2-3 varied demonstrations: bound fields, constants, volatile fields, match keys, frame The fault gate (#1445) on a test copy; each surviving fault class adds the check that kills it, until all are killed or reported
Identity (generate/identity_template.py) A reviewed template (patient-identity/1: #1516's record check, never generated) plus a generated mapping of this system's fields and the task's parameters onto it The demonstrations (each saved record reads back as typed) and a perturbation suite from the site's own records, with one case per slot only that slot can catch; adopted only if it does no worse than the site's hand-written adapter
Input types (generate/inputs.py) Sample source documents: labels, dates and the source's date order, code tables (Massachusetts -> MA) Round trip of every sample; an ambiguous date or an unknown code halts
  • The floor (generate/baseline.py): every typed parameter is checked, the frame is on, nothing ignores or drops a written field, no collateral rule is off, and there is no attribution. Generation can add checks, never remove one.
  • Never generated: the safety policy (profile, approvals, switch, consequential classification, gate order, attribution).
  • Review: parts.review.md states every check in plain words, including what it can't catch. generate-parts approve needs a person at a terminal. It refuses an agent's name and any part that changed after generation, because the approval is bound to the manifest digest.
  • Hostile content is data, with a test for each:
    • an "instruction" in an OpenAPI description;
    • untouched records filled with instructions and changed every round, which still can't hide the email check;
    • an instruction line in a referral document.
  • Verifier fixes found by the channel test:
    • The FHIR read used to read only the first page. It now follows next links on the same origin, and a failed page, another origin, more than 50 pages, or a total larger than what was read all read as unreadable.
    • The REST read can follow records_next_path (new, optional; digests unchanged when unset).

Credit

FHIR/REST pagination also proposed by @yunusarda-sudo in #1511 (superseded by this PR).

Changelog entry (for the next openadapt-flow release)

  • openadapt-flow generate-parts generates the app-specific parts of a workflow (read channel, record contract, identity mapping onto a reviewed template, input types) from two or three demonstrations, admits each by a check that doesn't depend on the generator, and records a person's approval of the plain-language review (#1520). FHIR reads follow every next page on the same origin, REST reads follow records_next_path, and a broken, cross-origin or short page reads as unreadable. FHIR/REST pagination also proposed by @yunusarda-sudo in #1511 (superseded by this PR).

Measured

Identity: a generated mapping onto the reviewed template, against #1516's hand-written adapters. #1516's people and its 676-item stream (223 true duplicates, 440 new patients) are rendered in each site shape. The mapping is generated from the site's rows and three demonstrations (benchmark/generated_parts/identity_mapping_backtest.py).

Site shape Suite misses, of 580 (reference) Adopted Stream misses, of 223 (reference) Stream false stops (reference)
#1516's rows 5 (5) generated 24 (24) 2 (2)
OpenEMR REST rows 5 (5) generated 24 (24) 2 (2)
FHIR Patient 5 (5) generated 24 (24) 2 (2)
FHIR read by first list item only 30 (5) reference 30 (24) 2 (2)

The generated mapping matches the hand-written adapters exactly on all three shapes. The fourth row loses the email and the prior name. The suite catches that where the demonstrations can't, and the adoption rule keeps the reference.

A rule calibrated on one site's records (the approach the template replaces): at every false-stop budget it missed 52-54 of 223 against the hand-built 24, with 0 false stops against 2. With 30 records, calibration can't learn which near-matches are one person. The frontier, and both adoption rules (--reference-identity, --reference-duplicate-check), are in the README.

Not in this PR

The live OpenEMR back-test is not in this PR; it waits for the fixture lease (autogen-backtest is in the queue). It will generate all four parts from three demonstrations on the fixture and run the fault gate on the generated and the hand-built contract with the same copy. It will report coverage per class, false stops, cost and time, review effort, and setup effort.

Tests

  • New: test_generate_{contract,channel,inputs,identity,identity_mapping,parts_pipeline}.py (37 tests).
  • 283 passed locally across those plus the verifier, FHIR, REST, deployment, verify-run, fault-gate, frame, provenance, intake and command-map suites.
  • ruff and mypy clean on the changed modules; source-boundary check OK.

🤖 Generated with Claude Code

abrichr and others added 4 commits October 24, 2026 20:52
…erified

openadapt-flow generate-parts builds the four parts of a workflow that are
specific to one application at setup time, and admits each only by a check
that does not depend on the generator. Healthy runs still make no model
calls; nothing here calls a model.

- Read channel (generate/channel.py): candidates from an OpenAPI document,
  a FHIR CapabilityStatement or teach-time traffic. The channel test reads
  each demonstration's record, every page, and must read as unreadable
  through a fault proxy (no auth, 401, 500, truncated, not JSON, a failed
  next page).
- Record contract (generate/contract.py): from the before/after difference
  across 2-3 varied demonstrations (bound fields, constants, volatile
  fields, match keys, frame), refined on a test copy by the fault gate until
  every fault class is killed or reported.
- Duplicate rule (generate/identity.py): roles from the schema, clauses
  calibrated on the system's own records with a generic error model and
  per-comparison false-match rates; a person labels the pairs that block a
  clause; unknown goes to a person.
- Input types (generate/inputs.py): labels, dates and the source's date
  order, code tables, admitted by a round trip; ambiguity halts.
- A floor generated parts can't go below (generate/baseline.py); a plain-
  language review, a digest manifest and a person's approval bound to it
  (generate/build.py, generate/cli.py).
- Verifiers: the FHIR read follows next links on the same origin and
  refuses a short read; the REST read follows records_next_path.
- benchmark/generated_parts: the generated duplicate rule on #1516's stream
  (676 items, 223 duplicates): 54 missed, 0 false stops, against the
  hand-built check's 24 and 2.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A missed duplicate costs far more than a false stop, so the duplicate
rule is now chosen for the fewest calibration misses within an estimated
false-stop budget (a knapsack: gain-first and gain-per-cost greedy, the
better kept). New signals: a generic, model-made table of given-name
variants (common names only; no record is sent), the medical record
number as an exact match, and typos in both names in the error model.

Adoption rule: a generated rule replaces a system's hand-built duplicate
check only if it misses no more on the same calibration set (identity.adopt;
generate-parts build --reference-duplicate-check). On #1516's stream it
does not (it misses 45 or more of 604 calibration duplicates where the
hand-built check misses 3), so the hand-built check is kept. The frontier,
the reasons and every arm are in benchmark/generated_parts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Calibrating a matching rule on one site's few dozen records can't learn
which near-matches are one person, so identity is now a reviewed template
(patient-identity/1: #1516's record check, never generated) fed by a
generated per-site mapping: which record fields feed each slot (given,
family, prior last name, date of birth and its format, MRN, phone, email,
street) and which task parameter feeds each input. Fields are chosen by
grounding on the demonstrations, the slot's name and its value shape; FHIR
coded lists are read by code (telecom[system=email].value).

The mapping is verified against the demonstrations and a perturbation
suite built from the site's own records, with one case per slot that only
that slot can catch, and adopted only if it does no worse than the site's
hand-written adapter (adopt_mapping; build --reference-identity).

Back-test on #1516's people and stream, in three site shapes (#1516's rows,
OpenEMR REST, FHIR): the generated mapping matched the hand-written
adapters on the suite and the stream (24 of 223 missed, 2 false stops). A
FHIR mapping read by first list item only (no email, no prior name) misses
30 suite duplicates against 5 and is refused.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- baseline: a contract without a frame, or a frame without an identity
  field, is below the floor.
- adoption: a generated rule or mapping must catch every duplicate the
  reference catches (a superset), not just miss fewer; the README says
  the yardsticks are the generator's own synthetic sets and lean toward
  generation.
- a REST next link to another origin reads as unreadable (test); the
  FHIR one had a test.
- CI: import order in rest.py; generate-parts channels' URL flags in
  BACKEND_PARITY.md; the approve hint uses the CLI spelling helper.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant