diff --git a/CAPABILITIES.md b/CAPABILITIES.md index e541d413f..3ba3930da 100644 --- a/CAPABILITIES.md +++ b/CAPABILITIES.md @@ -137,6 +137,7 @@ improvising or retrying when something is off. | Judge any executor's write, including an agent's or a script's, from the system of record | Released | Synthetic | `openadapt-flow verify-run`, [VERIFY_RUN.md](packages/flow/docs/VERIFY_RUN.md) | | Prove that nothing outside the declared change moved (frame conditions), and attribute concurrent edits by other users through the app's audit log or a modified-by field; the receipt signs which parts were checked | On main | Synthetic (MockMed, SQLite audit log) | `verify-run`, [VERIFICATION_COVERAGE.md](packages/flow/docs/VERIFICATION_COVERAGE.md) | | Admit a record check only after a fault suite written to a store copy shows it catches every fault it claims, and sign its coverage into the receipt | On main | Synthetic (in-memory REST store, SQLite copy) | `openadapt_flow/fault_gate.py`, `scaffold-verifier --fault-gate`, `verify-run` `admission:` | +| Generate the app-specific parts of a workflow (read channel, record contract, duplicate rule, input types) from two or three demonstrations, each admitted only by a check that doesn't depend on the generator, and approved by a person in plain words | On main | Synthetic (a fake REST and FHIR app); identity: a generated field mapping onto #1516's reviewed check matched its hand-written REST and FHIR adapters on #1516's stream (24 of 223 missed, 2 false stops) | `openadapt-flow generate-parts`, [GENERATED_PARTS.md](packages/flow/docs/GENERATED_PARTS.md) | | Check that the saved record matches its source document, not only the parameters: intake keeps each value's line, a second reading must agree before a guarded save, and a round trip compares the read-back with the document | On main | Synthetic (MockMed, text referral) | `openadapt_flow/provenance.py`, `intake`, `verify-run --provenance`, guarded save | | Guarded save: a safety layer any agent wraps around one write (check the record, hold for approval, read back) | On main ([#1258](https://github.com/OpenAdaptAI/OpenAdapt/pull/1258)) | Synthetic | `GUARDED_SAVE.md` | | Draft a record check from the demonstration's write steps (`scaffold-verifier`) | Released | Synthetic | `openadapt-flow scaffold-verifier` (a person fills in the draft) | @@ -302,6 +303,7 @@ A test fails when a command has no file or this list is out of date. - `bind-record-checks` - `certify` - `check` +- `generate-parts` - `jit` - `known-ids` - `lint` diff --git a/llms-full.txt b/llms-full.txt index 02ba1a00d..c104c6e17 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -534,6 +534,7 @@ improvising or retrying when something is off. | Judge any executor's write, including an agent's or a script's, from the system of record | Released | Synthetic | `openadapt-flow verify-run`, [VERIFY_RUN.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/VERIFY_RUN.md) | | Prove that nothing outside the declared change moved (frame conditions), and attribute concurrent edits by other users through the app's audit log or a modified-by field; the receipt signs which parts were checked | On main | Synthetic (MockMed, SQLite audit log) | `verify-run`, [VERIFICATION_COVERAGE.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/VERIFICATION_COVERAGE.md) | | Admit a record check only after a fault suite written to a store copy shows it catches every fault it claims, and sign its coverage into the receipt | On main | Synthetic (in-memory REST store, SQLite copy) | `openadapt_flow/fault_gate.py`, `scaffold-verifier --fault-gate`, `verify-run` `admission:` | +| Generate the app-specific parts of a workflow (read channel, record contract, duplicate rule, input types) from two or three demonstrations, each admitted only by a check that doesn't depend on the generator, and approved by a person in plain words | On main | Synthetic (a fake REST and FHIR app); identity: a generated field mapping onto #1516's reviewed check matched its hand-written REST and FHIR adapters on #1516's stream (24 of 223 missed, 2 false stops) | `openadapt-flow generate-parts`, [GENERATED_PARTS.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/GENERATED_PARTS.md) | | Check that the saved record matches its source document, not only the parameters: intake keeps each value's line, a second reading must agree before a guarded save, and a round trip compares the read-back with the document | On main | Synthetic (MockMed, text referral) | `openadapt_flow/provenance.py`, `intake`, `verify-run --provenance`, guarded save | | Guarded save: a safety layer any agent wraps around one write (check the record, hold for approval, read back) | On main ([#1258](https://github.com/OpenAdaptAI/OpenAdapt/pull/1258)) | Synthetic | `GUARDED_SAVE.md` | | Draft a record check from the demonstration's write steps (`scaffold-verifier`) | Released | Synthetic | `openadapt-flow scaffold-verifier` (a person fills in the draft) | @@ -563,6 +564,7 @@ improvising or retrying when something is off. | Learn one program with branches from two or more recordings, or say which recordings are missing | Released | Synthetic | `openadapt-flow induce`, `disambiguate`; [CAPABILITIES.md (flow)](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/CAPABILITIES.md) | | Learn a guarded branch from several halts fixed the same way | Released | Synthetic | `openadapt-flow teach-branch`, `learning/branch_teach.py` | | Run a program over a worklist | Released | Real app | `openadapt-flow for-each` | +| Learn a worklist loop from recordings that search a record, open it, work it and go back to the list, even when the last row skips the return or rows differ (a field some rows lack, a write some rows get behind a screen check a person accepts). Each pass selects the row's own record by its values, verifies it, and checks that row's write. Certified and governed runs refuse it until qualification covers it | On main ([#1435](https://github.com/OpenAdaptAI/OpenAdapt/pull/1435)) | Synthetic (held-out rows run through the interpreter against an in-memory app) | `openadapt-flow induce`; `Step.target_param` | | Typed business decisions inside a run | Released | Synthetic | `business_decision` states, `openadapt-flow business-decisions` | | Watch many real runs in the background and mine candidate branches (shadow mode) | On main ([#1212](https://github.com/OpenAdaptAI/OpenAdapt/pull/1212)) | Synthetic | [SHADOW_MINING.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/capture/docs/SHADOW_MINING.md), [LEARN_FROM_MANY_RUNS.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/LEARN_FROM_MANY_RUNS.md) | | Fill a workflow's inputs from a referral PDF, fax image or email | Released | Synthetic | `openadapt-flow intake`, [INTAKE.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/docs/INTAKE.md) | @@ -698,6 +700,7 @@ A test fails when a command has no file or this list is out of date. - `bind-record-checks` - `certify` - `check` +- `generate-parts` - `jit` - `known-ids` - `lint` diff --git a/packages/flow/benchmark/generated_parts/README.md b/packages/flow/benchmark/generated_parts/README.md new file mode 100644 index 000000000..2bf85e928 --- /dev/null +++ b/packages/flow/benchmark/generated_parts/README.md @@ -0,0 +1,174 @@ +# Generated parts against hand-built parts + +Measures `openadapt-flow generate-parts` +([`docs/GENERATED_PARTS.md`](../../docs/GENERATED_PARTS.md)) against the parts +written by hand for the OpenEMR referral workflow. + +## Identity: a vetted template, and a generated per-site mapping + +Calibrating a matching rule on one site's records can't learn which +near-matches are the same person: on #1516's stream a rule calibrated on the +30 existing charts misses 52 to 54 of 223 duplicates at every false-stop +budget, against the hand-built check's 24 (measured below). So identity is +built from two parts: + +1. **A template, never generated.** `patient-identity/1` is #1516's record + check (`openadapt_flow.runtime.checks.duplicate_record`): exact, close, + prior-name and nickname clauses with fixed thresholds. It is reviewed, + versioned code, owned like the safety policy, and named in the parts + (`openadapt_flow/generate/identity_template.py`, `TEMPLATES`). +2. **A mapping, generated per site.** Which of this system's record fields + feed each template slot (given and family name, a prior last name, date of + birth and its stored format, MRN, phone, email, street), and which task + parameter feeds each input. A field is chosen because its value is what + the person typed in every demonstration, it is named like the slot, and + its values look like the slot's (an email has an @); FHIR's coded lists + are read by code (`telecom[system=email].value`, `name[use=old].family`). + +The mapping is verified, never calibrated on the live table: + +- **against the demonstrations**: each saved record, read through the + mapping, must give back what was typed, input by input; +- **on a perturbation suite** built from the site's own records: generated + duplicates (a typo in one or both names, a nickname, day and month + swapped, a digit wrong in the date, a changed or hyphenated last name, an + initial, swapped names), each with the contact details kept and changed; + one case per slot that only that slot can catch (a referral under the + prior last name with a mistyped day and new contact details; a new last + name with a mistyped day and only the email, or only the phone, kept); and + every existing record posed as a new patient. + +**Adoption rule** (`identity_template.adopt_mapping`, enforced by +`generate-parts build --reference-identity`): a generated mapping is used +only if template plus mapping catches every duplicate on the suite that +template plus the reference mapping (the site's hand-written source adapter) +catches, and stops no more existing records. Fewer misses is not enough if +they are different ones. Otherwise the reference stays and the review says +why. The same rule governs a generated duplicate rule (`identity.adopt`, +`--reference-duplicate-check`). + +Both yardsticks are built by the generator from synthetic duplicates, so +they lean toward generation: on its calibration set #1516's check missed 0 +to 3 duplicates, and on the real stream 24. Passing the adoption rule is +necessary, not sufficient; the stream rows below are the evidence. + +### Measured: the generated mapping against #1516's hand-written adapters + +`identity_mapping_backtest.py` renders #1516's people (the existing charts +and every new chart the stream creates) in each site shape, generates the +mapping from the site's rows and three demonstrations, and runs the +template, from a checkout of #1516, with each mapping. + +| Site shape | Demonstrations read back | Suite misses, of 580 | Suite stops of 30 existing | Adopted | Stream misses, of 223 | Stream false stops, of 440 | +|---|---|---:|---:|---|---:|---:| +| #1516's rows | 6 of 6 inputs | 5 (reference 5) | 12 (12) | generated | 24 (reference 24) | 2 (2) | +| OpenEMR REST rows | 6 of 6 | 5 (5) | 12 (12) | generated | 24 (24) | 2 (2) | +| FHIR Patient | 6 of 6 | 5 (5) | 12 (12) | generated | 24 (24) | 2 (2) | +| FHIR, read by first list item only | 5 of 5 (no email) | **30** (5) | 12 (12) | **reference** | 30 (24) | 2 (2) | + +The generated mapping matches the hand-written adapter on all three +shapes, on the suite and on the 676-item stream. The fourth row is the +reason for the suite: read by the first list item only, a FHIR mapping loses +the email and the prior name. The demonstrations can't show that, the suite +does (30 misses against 5), and the adoption rule keeps the reference; on +the stream that mapping would have missed 30 of 223, against 24. + +The suite's misses that the reference shares (5 of 580: nicknames the +template's own table doesn't list) and its 12 stops (the six pairs of +existing charts that share a first name and a date of birth, which the +template stops by design) are the template's, not the mapping's. + +Two mapping bugs were found by tests, not by this table: grounded matching +once let the email slot take the street (fixed: the values must look like +the slot's), and the last name once took an old name's given name (fixed: +the field or the parameter must be named like the slot). + +## A rule calibrated on one site's records (the approach this replaces) + +`dup_backtest.py` runs both checks on #1516's own streams +(`benchmark/dup_check/measure.py` at `09d0298dc`, read from a checkout of that +branch). The long stream has 676 items: 440 new patients and 223 true +duplicates, 176 of them planted variants of the existing charts. The +generated rule is calibrated only on the 30 charts that exist before the +stream starts, with the generic error model in +`openadapt_flow/generate/identity.py`. It never sees the stream or its labels. +No app; generation takes about 0.25 s. The one model call is the generic +name-variant table (below), $0.24, made once. + +```bash +python -m generated_parts.dup_backtest --dup-check +``` + +### The objective, and the adoption rule + +A miss (a second chart for one patient) is much worse than a false stop (a +person looks for a few seconds), so the rule is chosen to **miss as few +calibration duplicates as possible, subject to an estimated false-stop +budget** per new record. False stops are estimated per comparison from the +system's own records (Fellegi-Sunter u-probabilities), multiplied over a +clause's terms and by the table size. + +**Adoption rule** (`identity.adopt`, enforced by `generate-parts build +--reference-duplicate-check`): a generated rule replaces the system's +hand-built check only if it misses no more duplicates on the same +calibration set. Otherwise the hand-built check stays, and the review says +why. + +### The frontier (long stream) + +| Budget (false stops per new record) | Calibration misses, generated | Test misses, of 223 | Test false stops, of 440 | +|---:|---:|---:|---:| +| 0.005 | 142 of 604 | 54 | 0 | +| 0.01 | 114 | 54 | 0 | +| 0.02 | 100 | 54 | 0 | +| 0.05 | 73 | 54 | 0 | +| 0.1 | 45 | 53 | 0 | +| #1516 record check (hand-built) | 3 of 604 | 24 | 2 | + +With the name-variant table. Without it the rows are within a few misses of +these; with assumed labels (two existing charts sharing a first name and a +date of birth judged one person, not a person's labels) the 0.1 row gives +41 calibration misses and 52 test misses. + +**On this system the adoption rule keeps the hand-built check at every +budget**: it misses 3 of 604 calibration duplicates where the generated rule +misses 45 or more. Generation is worse here. + +### Why it is worse here + +- **The calibration doesn't transfer.** Raising the budget cuts calibration + misses from 142 to 45 and test misses not at all. Of the generated rule's + test misses, 24 are the two kinds #1516 also can't catch, and the rest + (married name and new contact details; typos in both names; a prior name + with new contact details and a mistyped day) are the error kinds it still + misses in calibration, at every budget. +- **This system's own records say the key clause is risky.** Catching a + married name with new contact details needs "same first name and same date + of birth". Six pairs of the 30 existing charts share both, so on this + system's data that clause would stop 12 of 30 existing patients posed as + new; #1516's hand-built rule accepts that (it has the same 12 calibration + false stops) and is right on the stream (2 false stops in 440). Thirty + records are too few to tell an artifact from a real risk; the generated + rule takes the conservative reading. +- **Signals.** The generated rule now has what #1516 has: exact name and date, + the medical record number, nicknames (a generic, model-made table of + given-name variants, `generate/name_variants.json`, from common names only: + no record leaves the system), initials, phonetic and one-letter matches, + a date one part off or with day and month swapped, shared or hyphenated + surname parts, a prior last name, swapped names, and contact tie-breakers. + What it lacks is the hand-written judgment of which combinations to accept. + +### How the rule got here + +The rule changed four times on this stream, after seeing results, so this +stream is no longer a clean held-out test for the generator: + +| Version | Change | Test missed | Test false stops | +|---|---|---:|---:| +| 1 | one error at a time | 78 | 0 | +| 2 | each error with contact details kept and changed; swapped names, an initial, a second surname, a prior name | 42 | 12 | +| 3 | estimated false stops per record from per-comparison rates | 54 | 0 | +| 4 | misses first within a budget; nicknames, MRN, typos in both names; adoption rule | 52-54 | 0 | + +The committed result is `results/duplicates.json`: every arm, each rule in +words, and both checks' calibration scores. diff --git a/packages/flow/benchmark/generated_parts/__init__.py b/packages/flow/benchmark/generated_parts/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/packages/flow/benchmark/generated_parts/dup_backtest.py b/packages/flow/benchmark/generated_parts/dup_backtest.py new file mode 100644 index 000000000..1c5b64d23 --- /dev/null +++ b/packages/flow/benchmark/generated_parts/dup_backtest.py @@ -0,0 +1,313 @@ +"""Back-test: the generated duplicate rule against #1516's record check. + + python -m generated_parts.dup_backtest --dup-check + +Uses #1516's own streams (``benchmark/dup_check/measure.py``): the soak's 55 +items and the 500-item seed-7 stream, each with the planted true duplicates, +676 items with 223 true duplicates on the long stream. The generated rule is +calibrated only on the charts that exist before the stream starts (the +system's own data) with the generic error model in +``openadapt_flow/generate/identity.py``; it never sees the stream, the +planted kinds or their labels. Both checks then meet the same items in the +same order, with each earlier item's right outcome added to the table. + +No app, no model calls. +""" + +from __future__ import annotations + +import argparse +import json +import sys +import time +from pathlib import Path +from typing import Any + +HERE = Path(__file__).resolve().parent + + +_DUMP = r""" +import dataclasses, json, sys +sys.path = [p for p in sys.path if "openadapt-evals" not in p] +from benchmark.dup_check import measure +long_items = int(sys.argv[1]) +out = {} +for name, (existing, items) in ( + ("soak-seed7-55", measure.from_ledger(measure.SOAK / "workload.jsonl")), + (f"seed7-{long_items}", measure.from_generator(7, long_items, 10)), +): + extra, extra_items = measure.extra_existing(existing) + profile = existing + extra + stream = measure.plant(existing, items + extra_items, 7) + hand = measure.measure(profile, stream) + out[name] = { + "profile": profile, + "items": [dataclasses.asdict(i) for i in stream], + "duplicate_kinds": sorted(measure.DUPLICATE_KINDS), + "hand_rows": hand["rows"], + } +json.dump(out, sys.stdout) +""" + + +def dump_streams(checkout: Path, long_items: int) -> dict[str, Any]: + """#1516's streams and its record check's decisions, from its own checkout.""" + + import os + import subprocess + + flow = checkout / "packages" / "flow" + env = { + **os.environ, + "PYTHONPATH": os.pathsep.join( + [str(flow), str(checkout / "packages" / "types")] + ), + } + out = subprocess.run( + [sys.executable, "-c", _DUMP, str(long_items)], + capture_output=True, + text=True, + env=env, + cwd=str(flow), + check=True, + ).stdout + return json.loads(out) + + +def _iso(value: str) -> str: + import re + + m = re.fullmatch(r"(\d{2})/(\d{2})/(\d{4})", value or "") + return f"{m.group(3)}-{m.group(1)}-{m.group(2)}" if m else (value or "") + + +_REFERENCE = r""" +import json, sys +sys.path = [p for p in sys.path if "openadapt-evals" not in p] +from benchmark.dup_check import measure +from openadapt_flow.runtime.checks.duplicate_record import TableSource, check_duplicate +doc = json.load(sys.stdin) +profile = doc["profile"] +out = [] +for case in doc["cases"]: + table = profile if case["skip"] is None else profile[: case["skip"]] + profile[case["skip"] + 1 :] + records = [measure.record_of(i, p) for i, p in enumerate(table)] + decision = check_duplicate(TableSource(records), case["new"]).decision + out.append(decision != "none") +json.dump(out, sys.stdout) +""" + + +def reference_on(checkout: Path, calset: Any) -> Any: + """#1516's record check on ``calset``, as ``decide(new, table) -> bool``.""" + + import os + import subprocess + + cases = [{"new": new, "skip": None} for _k, new, _s in calset.positives] + cases += [{"new": calset.profile[i], "skip": i} for i in calset.negatives] + flow = checkout / "packages" / "flow" + env = { + **os.environ, + "PYTHONPATH": os.pathsep.join( + [str(flow), str(checkout / "packages" / "types")] + ), + } + decisions = json.loads( + subprocess.run( + [sys.executable, "-c", _REFERENCE], + input=json.dumps({"profile": calset.profile, "cases": cases}), + capture_output=True, + text=True, + env=env, + cwd=str(flow), + check=True, + ).stdout + ) + lookup: dict[tuple[str, int], bool] = {} + for case, stop in zip(cases, decisions): + key = ( + json.dumps(case["new"], sort_keys=True, default=str), + len(calset.profile) - (case["skip"] is not None), + ) + lookup[key] = stop + + def decide(new: dict, table: list) -> bool: + return lookup[(json.dumps(new, sort_keys=True, default=str), len(table))] + + return decide + + +def _same_first_and_dob(a: dict, b: dict) -> bool: + return str(a.get("fname", "")).casefold() == str( + b.get("fname", "") + ).casefold() and a.get("DOB") == b.get("DOB") + + +def run( + stream: dict[str, Any], + *, + seed: int, + budget: float, + name_variants: bool, + checkout: Path, + assumed_labels: bool = False, +) -> dict[str, Any]: + from openadapt_flow.generate.identity import adopt, generate_rule + + profile = [dict(p, DOB=_iso(p.get("DOB", ""))) for p in stream["profile"]] + # The table a new record meets grows to the stream's size: the known + # operational volume, not any outcome. + expected = len(profile) + sum( + 1 for i in stream["items"] if i["intended"] is not None + ) + started = time.perf_counter() + rule = generate_rule( + profile, + seed=seed, + expected_records=expected, + false_stop_budget=budget, + name_variants=name_variants, + # ASSUMED labels, not a person's: two existing charts with the same + # first name and date of birth are judged one person. A sensitivity + # arm only. + labeller=( + (lambda item: _same_first_and_dob(*item["records"])) + if assumed_labels + else None + ), + ) + gen_s = time.perf_counter() - started + adoption = adopt(rule, reference_on(checkout, rule.calibration_set)) + table: list[dict] = [dict(p) for p in profile] + kinds = set(stream["duplicate_kinds"]) + hand = {r["item"]: r for r in stream["hand_rows"]} + rows = [] + for item in stream["items"]: + table += [dict(p, DOB=_iso(p.get("DOB", ""))) for p in item["before"]] + params = { + k: (v.strip() if isinstance(v, str) else v) + for k, v in item["params"].items() + } + params["DOB"] = _iso(params.get("DOB", "")) + decision, _why = rule.decide(params, table) + rows.append( + { + "item": item["item_id"], + "exception": item["exception"], + "right": "new chart" + if item["intended"] is not None + else ( + "no chart: duplicate" if item["exception"] in kinds else "no chart" + ), + "generated": decision, + "record": hand[item["item_id"]]["record"], + } + ) + if item["intended"] is not None: + table.append( + dict(item["intended"], DOB=_iso(item["intended"].get("DOB", ""))) + ) + return { + "rule": rule, + "rows": rows, + "generation_s": round(gen_s, 3), + "adoption": adoption, + } + + +def summarize(rows: list[dict], key: str, stops: Any) -> dict: + new = [r for r in rows if r["right"] == "new chart"] + dups = [r for r in rows if r["right"] == "no chart: duplicate"] + false = [r for r in new if stops(r)] + missed = [r for r in dups if not stops(r)] + + def count(rs: list[dict]) -> dict: + out: dict[str, int] = {} + for r in rs: + out[r["exception"]] = out.get(r["exception"], 0) + 1 + return dict(sorted(out.items())) + + return { + "items": len(rows), + "new_chart_items": len(new), + "true_duplicates": len(dups), + "false_stops": len(false), + "false_stops_per_100_items": round(100 * len(false) / max(1, len(rows)), 1), + "missed_duplicates": len(missed), + "missed_by_exception": count(missed), + "false_stops_by_exception": count(false), + } + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + parser.add_argument("--dup-check", type=Path, required=True) + parser.add_argument("--long-items", type=int, default=500) + parser.add_argument("--seed", type=int, default=0) + parser.add_argument( + "--budgets", + default="0.005,0.01,0.02,0.05,0.1", + help="estimated false stops per new record the rule may spend (a sweep)", + ) + parser.add_argument("--out", type=Path, default=HERE / "results") + args = parser.parse_args() + streams = dump_streams(args.dup_check, args.long_items) + args.out.mkdir(parents=True, exist_ok=True) + report: dict[str, Any] = {} + budgets = [float(b) for b in args.budgets.split(",")] + for name, stream in streams.items(): + for variants, labels in ((False, False), (True, False), (True, True)): + for budget in budgets: + result = run( + stream, + seed=args.seed, + budget=budget, + name_variants=variants, + checkout=args.dup_check, + assumed_labels=labels, + ) + rows = result["rows"] + key = f"{name}|variants={variants}|labels={labels}|budget={budget}" + adoption = result["adoption"] + report[key] = { + "stream": name, + "name_variants": variants, + "assumed_labels": labels, + "budget": budget, + "profile_records": len(stream["profile"]), + "generation_s": result["generation_s"], + "rule": result["rule"].as_dict(), + "rule_text": result["rule"].describe(), + "adoption": { + "use": adoption.use, + "why": adoption.why, + "calibration_generated": adoption.generated, + "calibration_reference": adoption.reference, + }, + "generated": summarize( + rows, "generated", lambda r: r["generated"] != "none" + ), + "hand_built_record_check": summarize( + rows, "record", lambda r: r["record"] != "none" + ), + } + g = report[key]["generated"] + h = report[key]["hand_built_record_check"] + print( + f"{key}: test missed {g['missed_duplicates']}/{g['true_duplicates']} " + f"false stops {g['false_stops']} ({g['false_stops_per_100_items']}/100) " + f"| hand missed {h['missed_duplicates']} fs {h['false_stops']} " + f"| calib gen missed {adoption.generated['missed']} ref " + f"{adoption.reference['missed'] if adoption.reference else '-'} " + f"of {adoption.generated['duplicates']} -> use {adoption.use}", + flush=True, + ) + (args.out / "duplicates.json").write_text( + json.dumps(report, indent=2, sort_keys=True, default=str) + "\n" + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/packages/flow/benchmark/generated_parts/identity_mapping_backtest.py b/packages/flow/benchmark/generated_parts/identity_mapping_backtest.py new file mode 100644 index 000000000..858ded07c --- /dev/null +++ b/packages/flow/benchmark/generated_parts/identity_mapping_backtest.py @@ -0,0 +1,243 @@ +"""Back-test: a generated per-site mapping onto the vetted identity template. + + python -m generated_parts.identity_mapping_backtest --dup-check + +The template is #1516's record check (``patient-identity/1``), run from that +checkout. The people are #1516's own (the stream's existing charts and every +new chart it creates), rendered in three site shapes: + +- ``rows``: #1516's own patient rows (``fname``, ``lname``, ``DOB``, ...); +- ``openemr_rest``: OpenEMR's REST patient rows (``pubpid``, ``birth_lname``); +- ``fhir``: FHIR R4 ``Patient`` resources, flattened the way a generated FHIR + read channel flattens them (``generate/channel.py``: first list item). + +For each shape the mapping is generated from the site's rows and three +demonstrations (three of the charts, with the values typed for them), then +compared with the reference mapping (#1516's own adapter for that shape) on: + +- the demonstrations: each saved record must read back as typed; +- a perturbation suite built from the site's records (generated duplicates, + and every record posed as a new patient); +- #1516's 676-item stream (223 true duplicates), rendered in that shape. + +No app, no model calls. Runs in a subprocess against the #1516 checkout. +""" + +from __future__ import annotations + +import argparse +import json +import os +import subprocess +import sys +from pathlib import Path + +HERE = Path(__file__).resolve().parent +FLOW = HERE.parents[1] + +_RUN = r""" +import dataclasses, importlib.util, json, sys +sys.path = [p for p in sys.path if "openadapt-evals" not in p] +spec = importlib.util.spec_from_file_location("gp_identity_template", sys.argv[1]) +it = importlib.util.module_from_spec(spec); sys.modules[spec.name] = it; spec.loader.exec_module(it) +from benchmark.dup_check import measure +from openadapt_flow.runtime.checks import duplicate_record as dr + +variants = json.load(open(sys.argv[2]))["variants"] +nicknames = {} +for name, vs in variants.items(): + for v in vs: + nicknames.setdefault(name.casefold(), set()).add(v) + nicknames.setdefault(v.casefold(), set()).add(name) +nicknames = {k: sorted(v) for k, v in nicknames.items()} + +def iso(v): + return measure.iso(v or "") + +def person(p): + return { + "fname": p.get("fname", ""), "lname": p.get("lname", ""), "DOB": iso(p.get("DOB", "")), + "phone_home": p.get("phone_home", ""), "email": p.get("email", ""), + "street": p.get("street", ""), "prior_lname": p.get("prior_lname") or "", + } + +def rows_shape(i, p): + return dict(p, id=str(i), DOB=iso(p.get("DOB", ""))) + +def rest_shape(i, p): + row = {"id": str(i), "pid": str(i), "uuid": f"u-{i}", "fname": p.get("fname", ""), + "lname": p.get("lname", ""), "DOB": iso(p.get("DOB", "")), "pubpid": f"MRN{i:05d}", + "phone_home": p.get("phone_home", ""), "street": p.get("street", ""), + "email": p.get("email", ""), "sex": p.get("sex", "")} + if p.get("prior_lname"): + row["birth_lname"] = p["prior_lname"] + return row + +def fhir_resource(i, p): + names = [{"use": "official", "family": p.get("lname", ""), "given": [p.get("fname", "")]}] + if p.get("prior_lname"): + names.append({"use": "old", "family": p["prior_lname"], "given": [p.get("fname", "")]}) + return { + "resourceType": "Patient", "id": str(i), "name": names, "birthDate": iso(p.get("DOB", "")), + "identifier": [{"type": {"coding": [{"code": "MR"}]}, "value": f"MRN{i:05d}"}], + "telecom": [{"system": "phone", "use": "home", "value": p.get("phone_home", "")}, + {"system": "email", "value": p.get("email", "")}], + "address": [{"line": [p.get("street", "")]}], + } + +def flatten(resource): + out = {} + def walk(node, path): + if isinstance(node, list): + if node: walk(node[0], path) + return + if isinstance(node, dict): + for k in sorted(node): walk(node[k], f"{path}.{k}" if path else k) + return + if path: out[path] = node + walk(resource, "") + return out + +class FakeResp: + def __init__(self, body, url): self.status_code, self._body, self.url = 200, body, url + def json(self): return self._body +class FakeSession: + def __init__(self, rows): self.rows = rows + def get(self, url, **kw): return FakeResp({"data": self.rows, "links": {}}, url) + +def ref_rows(rows): + return [measure.record_of(i, r) for i, r in enumerate(rows)] + +def ref_rest(rows): + src = dr.RestPatientSource("http://ref.invalid", session=FakeSession(rows), page_size=10**6) + return src.search({}) + +def ref_fhir(resources): + src = dr.FhirPatientSource("http://ref.invalid") + return [src._record(r) for r in resources] + +stream = json.load(open(sys.argv[3]))["seed7-500"] +profile = stream["profile"] +items = stream["items"] +kinds = set(stream["duplicate_kinds"]) +template = it.Template("patient-identity/1") +report = {} +shapes = { + "rows": (rows_shape, None, ref_rows), + "openemr_rest": (rest_shape, None, ref_rest), + "fhir": (fhir_resource, None, ref_fhir), + # The same resources read the way a generic flattening reads them (first + # list item only): no email, no prior name. The adoption rule must refuse. + "fhir_first_item": (fhir_resource, flatten, ref_fhir), +} +for name, (shape, flat, reference) in shapes.items(): + raw = [shape(i, p) for i, p in enumerate(profile)] + site = [flat(r) if flat else r for r in raw] + # Three demonstrations: three charts, with the values a person typed. + demos = [(person(p), site[i]) for i, p in list(enumerate(profile))[:3]] + mapping = it.generate_mapping(site, demos) + agree = it.check_demos(mapping, demos) + # The suite knows who each record is: the people the rows were made from. + cases = it.perturbation_suite([person(p) for p in profile], lambda who: who, seed=0, nicknames=nicknames) + gen_table = [template.record(mapping.record_slots(r)) for r in site] + ref_table = reference(raw) + decide = lambda want, rows: template.decide_records({k: v for k, v in want.items() if k != "prior_lname"}, rows) + gen_suite = it.run_suite(cases, gen_table, decide) + ref_suite = it.run_suite(cases, ref_table, decide) + adoption = it.adopt_mapping(gen_suite, ref_suite) + # The stream, rendered in this shape. + def stream_run(to_records): + table_people = [dict(p) for p in profile] + stops = {"missed": 0, "false_stops": 0, "dups": 0, "new": 0} + for item in items: + table_people += item["before"] + want = person(item["params"]) + want.pop("prior_lname") + records = to_records(table_people) + stop = template.decide_records(want, records) != "none" + if item["intended"] is not None: + stops["new"] += 1 + stops["false_stops"] += int(stop) + table_people.append(item["intended"]) + elif item["exception"] in kinds: + stops["dups"] += 1 + stops["missed"] += int(not stop) + return stops + def gen_records(people_rows): + raw_rows = [shape(i, p) for i, p in enumerate(people_rows)] + site_rows = [flat(r) if flat else r for r in raw_rows] + return [template.record(mapping.record_slots(r)) for r in site_rows] + def ref_records(people_rows): + return reference([shape(i, p) for i, p in enumerate(people_rows)]) + report[name] = { + "mapping": mapping.as_dict(), + "demos_agree": agree, + "demos": len(demos), + "suite_generated": gen_suite, + "suite_reference": ref_suite, + "adoption": {"use": adoption.use, "why": adoption.why}, + "stream_generated": stream_run(gen_records), + "stream_reference": stream_run(ref_records), + } +json.dump(report, sys.stdout, indent=1, default=str) +""" + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + parser.add_argument("--dup-check", type=Path, required=True) + parser.add_argument("--out", type=Path, default=HERE / "results") + args = parser.parse_args() + sys.path.insert(0, str(HERE.parent)) + from generated_parts.dup_backtest import dump_streams + + streams = dump_streams(args.dup_check, 500) + args.out.mkdir(parents=True, exist_ok=True) + stream_file = args.out / ".streams.json" + stream_file.write_text(json.dumps(streams)) + flow = args.dup_check / "packages" / "flow" + env = { + **os.environ, + "PYTHONPATH": os.pathsep.join( + [str(flow), str(args.dup_check / "packages" / "types")] + ), + } + try: + out = subprocess.run( + [ + sys.executable, + "-c", + _RUN, + str(FLOW / "openadapt_flow" / "generate" / "identity_template.py"), + str(FLOW / "openadapt_flow" / "generate" / "name_variants.json"), + str(stream_file), + ], + capture_output=True, + text=True, + env=env, + cwd=str(flow), + ) + finally: + stream_file.unlink(missing_ok=True) + if out.returncode != 0: + print(out.stderr[-3000:], file=sys.stderr) + return out.returncode + report = json.loads(out.stdout) + (args.out / "identity_mapping.json").write_text( + json.dumps(report, indent=2, sort_keys=True) + "\n" + ) + for name, r in report.items(): + g, f = r["stream_generated"], r["stream_reference"] + print( + f"{name}: demos agree {r['demos_agree']} | suite gen missed " + f"{r['suite_generated']['missed']}/{r['suite_generated']['duplicates']} fs " + f"{r['suite_generated']['false_stops']} vs ref " + f"{r['suite_reference']['missed']} fs {r['suite_reference']['false_stops']} " + f"-> {r['adoption']['use']} | stream gen missed {g['missed']}/{g['dups']} fs " + f"{g['false_stops']}/{g['new']} vs ref {f['missed']} fs {f['false_stops']}" + ) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/packages/flow/benchmark/generated_parts/results/duplicates.json b/packages/flow/benchmark/generated_parts/results/duplicates.json new file mode 100644 index 000000000..bc9acae52 --- /dev/null +++ b/packages/flow/benchmark/generated_parts/results/duplicates.json @@ -0,0 +1,6867 @@ +{ + "seed7-500|variants=False|labels=False|budget=0.005": { + "adoption": { + "calibration_generated": { + "distinct_queries": 30, + "duplicates": 586, + "false_stops": 0, + "missed": 139, + "missed_by_error": { + "both_names_typo+contact_changed": 29, + "first_typo+contact_changed": 26, + "last_changed+contact_changed": 30, + "last_typo+contact_changed": 24, + "names_swapped+contact_changed": 30 + } + }, + "calibration_reference": { + "distinct_queries": 30, + "duplicates": 586, + "false_stops": 12, + "missed": 0, + "missed_by_error": {} + }, + "use": "reference", + "why": "the generated rule misses 139 of 586 calibration duplicates and the reference 0: the reference stays" + }, + "assumed_labels": false, + "budget": 0.005, + "generated": { + "false_stops": 0, + "false_stops_by_exception": {}, + "false_stops_per_100_items": 0.0, + "items": 676, + "missed_by_exception": { + "limit_married_moved_dob_typo": 12, + "limit_new_name_moved_same_dob": 12, + "pl_both_typos_same_contact": 12, + "pl_married_moved": 12, + "r2_prior_name_moved_dob_typo": 6 + }, + "missed_duplicates": 54, + "new_chart_items": 440, + "true_duplicates": 223 + }, + "generation_s": 0.199, + "hand_built_record_check": { + "false_stops": 2, + "false_stops_by_exception": { + "shared_dob": 2 + }, + "false_stops_per_100_items": 0.3, + "items": 676, + "missed_by_exception": { + "limit_married_moved_dob_typo": 12, + "limit_new_name_moved_same_dob": 12 + }, + "missed_duplicates": 24, + "new_chart_items": 440, + "true_duplicates": 223 + }, + "name_variants": false, + "profile_records": 30, + "rule": { + "calibration": { + "calibration_duplicates": 586, + "distinct_queries": 870, + "false_match_rate": 0.0, + "miss_by_error": { + "both_names_typo+contact_changed": 0.967, + "both_names_typo+contact_kept": 0.0, + "contact_changed+contact_changed": 0.0, + "day_month_swapped+contact_changed": 0.0, + "day_month_swapped+contact_kept": 0.0, + "dob_digit+contact_changed": 0.0, + "dob_digit+contact_kept": 0.0, + "first_initial+contact_changed": 0.0, + "first_initial+contact_kept": 0.0, + "first_typo+contact_changed": 0.867, + "first_typo+contact_kept": 0.0, + "last_changed+contact_changed": 1.0, + "last_changed+contact_kept": 0.0, + "last_hyphenated+contact_changed": 0.0, + "last_hyphenated+contact_kept": 0.0, + "last_second_surname+contact_changed": 0.0, + "last_second_surname+contact_kept": 0.0, + "last_typo+contact_changed": 0.8, + "last_typo+contact_kept": 0.0, + "names_swapped+contact_changed": 1.0, + "names_swapped+contact_kept": 0.0 + }, + "miss_rate": 0.2372 + }, + "close": [ + [ + [ + "email", + "eq" + ], + [ + "phone", + "eq" + ], + [ + "street", + "eq" + ] + ], + [ + [ + "first", + "eq" + ], + [ + "last", + "share" + ], + [ + "dob", + "near" + ] + ], + [ + [ + "first", + "initial" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "first", + "edit1" + ], + [ + "last", + "share" + ], + [ + "dob", + "swap" + ] + ] + ], + "estimated_false_stops_per_record": 0.00455, + "exact": [ + [ + "first", + "eq" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ], + "name_variants": null, + "notes": [], + "roles": { + "dob": "DOB", + "email": "email", + "first": "fname", + "last": "lname", + "phone": "phone_home", + "prior_last": "prior_lname", + "street": "street" + }, + "schema_version": "openadapt.duplicate-rule/v1" + }, + "rule_text": [ + "Stop: the record exists when first the same and last the same and dob the same.", + "Stop for a person when email the same and phone the same and street the same.", + "Stop for a person when first the same and last sharing a name part and dob one part off.", + "Stop for a person when first matching by initial and last the same and dob the same.", + "Stop for a person when first one letter apart and last sharing a name part and dob with day and month swapped.", + "Stop for a person when the new record lacks a name or a date of birth." + ], + "stream": "seed7-500" + }, + "seed7-500|variants=False|labels=False|budget=0.01": { + "adoption": { + "calibration_generated": { + "distinct_queries": 30, + "duplicates": 586, + "false_stops": 0, + "missed": 109, + "missed_by_error": { + "both_names_typo+contact_changed": 25, + "last_changed+contact_changed": 30, + "last_typo+contact_changed": 24, + "names_swapped+contact_changed": 30 + } + }, + "calibration_reference": { + "distinct_queries": 30, + "duplicates": 586, + "false_stops": 12, + "missed": 0, + "missed_by_error": {} + }, + "use": "reference", + "why": "the generated rule misses 109 of 586 calibration duplicates and the reference 0: the reference stays" + }, + "assumed_labels": false, + "budget": 0.01, + "generated": { + "false_stops": 0, + "false_stops_by_exception": {}, + "false_stops_per_100_items": 0.0, + "items": 676, + "missed_by_exception": { + "limit_married_moved_dob_typo": 12, + "limit_new_name_moved_same_dob": 12, + "pl_both_typos_same_contact": 12, + "pl_married_moved": 12, + "r2_prior_name_moved_dob_typo": 6 + }, + "missed_duplicates": 54, + "new_chart_items": 440, + "true_duplicates": 223 + }, + "generation_s": 0.201, + "hand_built_record_check": { + "false_stops": 2, + "false_stops_by_exception": { + "shared_dob": 2 + }, + "false_stops_per_100_items": 0.3, + "items": 676, + "missed_by_exception": { + "limit_married_moved_dob_typo": 12, + "limit_new_name_moved_same_dob": 12 + }, + "missed_duplicates": 24, + "new_chart_items": 440, + "true_duplicates": 223 + }, + "name_variants": false, + "profile_records": 30, + "rule": { + "calibration": { + "calibration_duplicates": 586, + "distinct_queries": 870, + "false_match_rate": 0.0, + "miss_by_error": { + "both_names_typo+contact_changed": 0.833, + "both_names_typo+contact_kept": 0.0, + "contact_changed+contact_changed": 0.0, + "day_month_swapped+contact_changed": 0.0, + "day_month_swapped+contact_kept": 0.0, + "dob_digit+contact_changed": 0.0, + "dob_digit+contact_kept": 0.0, + "first_initial+contact_changed": 0.0, + "first_initial+contact_kept": 0.0, + "first_typo+contact_changed": 0.0, + "first_typo+contact_kept": 0.0, + "last_changed+contact_changed": 1.0, + "last_changed+contact_kept": 0.0, + "last_hyphenated+contact_changed": 0.0, + "last_hyphenated+contact_kept": 0.0, + "last_second_surname+contact_changed": 0.0, + "last_second_surname+contact_kept": 0.0, + "last_typo+contact_changed": 0.8, + "last_typo+contact_kept": 0.0, + "names_swapped+contact_changed": 1.0, + "names_swapped+contact_kept": 0.0 + }, + "miss_rate": 0.186 + }, + "close": [ + [ + [ + "email", + "eq" + ], + [ + "phone", + "eq" + ], + [ + "street", + "eq" + ] + ], + [ + [ + "first", + "edit1" + ], + [ + "last", + "share" + ], + [ + "dob", + "near" + ] + ], + [ + [ + "first", + "initial" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "first", + "eq" + ], + [ + "last", + "eq" + ], + [ + "dob", + "swap" + ] + ] + ], + "estimated_false_stops_per_record": 0.00944, + "exact": [ + [ + "first", + "eq" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ], + "name_variants": null, + "notes": [], + "roles": { + "dob": "DOB", + "email": "email", + "first": "fname", + "last": "lname", + "phone": "phone_home", + "prior_last": "prior_lname", + "street": "street" + }, + "schema_version": "openadapt.duplicate-rule/v1" + }, + "rule_text": [ + "Stop: the record exists when first the same and last the same and dob the same.", + "Stop for a person when email the same and phone the same and street the same.", + "Stop for a person when first one letter apart and last sharing a name part and dob one part off.", + "Stop for a person when first matching by initial and last the same and dob the same.", + "Stop for a person when first the same and last the same and dob with day and month swapped.", + "Stop for a person when the new record lacks a name or a date of birth." + ], + "stream": "seed7-500" + }, + "seed7-500|variants=False|labels=False|budget=0.02": { + "adoption": { + "calibration_generated": { + "distinct_queries": 30, + "duplicates": 586, + "false_stops": 0, + "missed": 104, + "missed_by_error": { + "both_names_typo+contact_changed": 25, + "last_changed+contact_changed": 30, + "last_typo+contact_changed": 22, + "names_swapped+contact_changed": 27 + } + }, + "calibration_reference": { + "distinct_queries": 30, + "duplicates": 586, + "false_stops": 12, + "missed": 0, + "missed_by_error": {} + }, + "use": "reference", + "why": "the generated rule misses 104 of 586 calibration duplicates and the reference 0: the reference stays" + }, + "assumed_labels": false, + "budget": 0.02, + "generated": { + "false_stops": 0, + "false_stops_by_exception": {}, + "false_stops_per_100_items": 0.0, + "items": 676, + "missed_by_exception": { + "limit_married_moved_dob_typo": 12, + "limit_new_name_moved_same_dob": 12, + "pl_both_typos_same_contact": 12, + "pl_married_moved": 12, + "r2_prior_name_moved_dob_typo": 6 + }, + "missed_duplicates": 54, + "new_chart_items": 440, + "true_duplicates": 223 + }, + "generation_s": 0.2, + "hand_built_record_check": { + "false_stops": 2, + "false_stops_by_exception": { + "shared_dob": 2 + }, + "false_stops_per_100_items": 0.3, + "items": 676, + "missed_by_exception": { + "limit_married_moved_dob_typo": 12, + "limit_new_name_moved_same_dob": 12 + }, + "missed_duplicates": 24, + "new_chart_items": 440, + "true_duplicates": 223 + }, + "name_variants": false, + "profile_records": 30, + "rule": { + "calibration": { + "calibration_duplicates": 586, + "distinct_queries": 870, + "false_match_rate": 0.0, + "miss_by_error": { + "both_names_typo+contact_changed": 0.833, + "both_names_typo+contact_kept": 0.0, + "contact_changed+contact_changed": 0.0, + "day_month_swapped+contact_changed": 0.0, + "day_month_swapped+contact_kept": 0.0, + "dob_digit+contact_changed": 0.0, + "dob_digit+contact_kept": 0.0, + "first_initial+contact_changed": 0.0, + "first_initial+contact_kept": 0.0, + "first_typo+contact_changed": 0.0, + "first_typo+contact_kept": 0.0, + "last_changed+contact_changed": 1.0, + "last_changed+contact_kept": 0.0, + "last_hyphenated+contact_changed": 0.0, + "last_hyphenated+contact_kept": 0.0, + "last_second_surname+contact_changed": 0.0, + "last_second_surname+contact_kept": 0.0, + "last_typo+contact_changed": 0.733, + "last_typo+contact_kept": 0.0, + "names_swapped+contact_changed": 0.9, + "names_swapped+contact_kept": 0.0 + }, + "miss_rate": 0.1775 + }, + "close": [ + [ + [ + "email", + "eq" + ], + [ + "phone", + "eq" + ], + [ + "street", + "eq" + ] + ], + [ + [ + "first", + "edit1" + ], + [ + "last", + "share" + ], + [ + "dob", + "near" + ] + ], + [ + [ + "first", + "initial" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "first", + "eq" + ], + [ + "last", + "edit1" + ], + [ + "dob", + "swap" + ] + ], + [ + [ + "dob", + "swap" + ], + [ + "names_swapped", + "eq" + ] + ] + ], + "estimated_false_stops_per_record": 0.01883, + "exact": [ + [ + "first", + "eq" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ], + "name_variants": null, + "notes": [], + "roles": { + "dob": "DOB", + "email": "email", + "first": "fname", + "last": "lname", + "phone": "phone_home", + "prior_last": "prior_lname", + "street": "street" + }, + "schema_version": "openadapt.duplicate-rule/v1" + }, + "rule_text": [ + "Stop: the record exists when first the same and last the same and dob the same.", + "Stop for a person when email the same and phone the same and street the same.", + "Stop for a person when first one letter apart and last sharing a name part and dob one part off.", + "Stop for a person when first matching by initial and last the same and dob the same.", + "Stop for a person when first the same and last one letter apart and dob with day and month swapped.", + "Stop for a person when dob with day and month swapped and first and last names swapped.", + "Stop for a person when the new record lacks a name or a date of birth." + ], + "stream": "seed7-500" + }, + "seed7-500|variants=False|labels=False|budget=0.05": { + "adoption": { + "calibration_generated": { + "distinct_queries": 30, + "duplicates": 586, + "false_stops": 0, + "missed": 77, + "missed_by_error": { + "both_names_typo+contact_changed": 25, + "last_changed+contact_changed": 30, + "last_typo+contact_changed": 22 + } + }, + "calibration_reference": { + "distinct_queries": 30, + "duplicates": 586, + "false_stops": 12, + "missed": 0, + "missed_by_error": {} + }, + "use": "reference", + "why": "the generated rule misses 77 of 586 calibration duplicates and the reference 0: the reference stays" + }, + "assumed_labels": false, + "budget": 0.05, + "generated": { + "false_stops": 0, + "false_stops_by_exception": {}, + "false_stops_per_100_items": 0.0, + "items": 676, + "missed_by_exception": { + "limit_married_moved_dob_typo": 12, + "limit_new_name_moved_same_dob": 12, + "pl_both_typos_same_contact": 12, + "pl_married_moved": 12, + "r2_prior_name_moved_dob_typo": 6 + }, + "missed_duplicates": 54, + "new_chart_items": 440, + "true_duplicates": 223 + }, + "generation_s": 0.202, + "hand_built_record_check": { + "false_stops": 2, + "false_stops_by_exception": { + "shared_dob": 2 + }, + "false_stops_per_100_items": 0.3, + "items": 676, + "missed_by_exception": { + "limit_married_moved_dob_typo": 12, + "limit_new_name_moved_same_dob": 12 + }, + "missed_duplicates": 24, + "new_chart_items": 440, + "true_duplicates": 223 + }, + "name_variants": false, + "profile_records": 30, + "rule": { + "calibration": { + "calibration_duplicates": 586, + "distinct_queries": 870, + "false_match_rate": 0.0, + "miss_by_error": { + "both_names_typo+contact_changed": 0.833, + "both_names_typo+contact_kept": 0.0, + "contact_changed+contact_changed": 0.0, + "day_month_swapped+contact_changed": 0.0, + "day_month_swapped+contact_kept": 0.0, + "dob_digit+contact_changed": 0.0, + "dob_digit+contact_kept": 0.0, + "first_initial+contact_changed": 0.0, + "first_initial+contact_kept": 0.0, + "first_typo+contact_changed": 0.0, + "first_typo+contact_kept": 0.0, + "last_changed+contact_changed": 1.0, + "last_changed+contact_kept": 0.0, + "last_hyphenated+contact_changed": 0.0, + "last_hyphenated+contact_kept": 0.0, + "last_second_surname+contact_changed": 0.0, + "last_second_surname+contact_kept": 0.0, + "last_typo+contact_changed": 0.733, + "last_typo+contact_kept": 0.0, + "names_swapped+contact_changed": 0.0, + "names_swapped+contact_kept": 0.0 + }, + "miss_rate": 0.1314 + }, + "close": [ + [ + [ + "email", + "eq" + ], + [ + "phone", + "eq" + ], + [ + "street", + "eq" + ] + ], + [ + [ + "first", + "edit1" + ], + [ + "last", + "share" + ], + [ + "dob", + "near" + ] + ], + [ + [ + "first", + "initial" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "dob", + "eq" + ], + [ + "names_swapped", + "eq" + ] + ], + [ + [ + "first", + "eq" + ], + [ + "last", + "edit1" + ], + [ + "dob", + "swap" + ] + ] + ], + "estimated_false_stops_per_record": 0.04726, + "exact": [ + [ + "first", + "eq" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ], + "name_variants": null, + "notes": [], + "roles": { + "dob": "DOB", + "email": "email", + "first": "fname", + "last": "lname", + "phone": "phone_home", + "prior_last": "prior_lname", + "street": "street" + }, + "schema_version": "openadapt.duplicate-rule/v1" + }, + "rule_text": [ + "Stop: the record exists when first the same and last the same and dob the same.", + "Stop for a person when email the same and phone the same and street the same.", + "Stop for a person when first one letter apart and last sharing a name part and dob one part off.", + "Stop for a person when first matching by initial and last the same and dob the same.", + "Stop for a person when dob the same and first and last names swapped.", + "Stop for a person when first the same and last one letter apart and dob with day and month swapped.", + "Stop for a person when the new record lacks a name or a date of birth." + ], + "stream": "seed7-500" + }, + "seed7-500|variants=False|labels=False|budget=0.1": { + "adoption": { + "calibration_generated": { + "distinct_queries": 30, + "duplicates": 586, + "false_stops": 0, + "missed": 50, + "missed_by_error": { + "both_names_typo+contact_changed": 23, + "last_changed+contact_changed": 27 + } + }, + "calibration_reference": { + "distinct_queries": 30, + "duplicates": 586, + "false_stops": 12, + "missed": 0, + "missed_by_error": {} + }, + "use": "reference", + "why": "the generated rule misses 50 of 586 calibration duplicates and the reference 0: the reference stays" + }, + "assumed_labels": false, + "budget": 0.1, + "generated": { + "false_stops": 0, + "false_stops_by_exception": {}, + "false_stops_per_100_items": 0.0, + "items": 676, + "missed_by_exception": { + "limit_married_moved_dob_typo": 12, + "limit_new_name_moved_same_dob": 12, + "pl_both_typos_same_contact": 11, + "pl_married_moved": 12, + "r2_prior_name_moved_dob_typo": 6 + }, + "missed_duplicates": 53, + "new_chart_items": 440, + "true_duplicates": 223 + }, + "generation_s": 0.205, + "hand_built_record_check": { + "false_stops": 2, + "false_stops_by_exception": { + "shared_dob": 2 + }, + "false_stops_per_100_items": 0.3, + "items": 676, + "missed_by_exception": { + "limit_married_moved_dob_typo": 12, + "limit_new_name_moved_same_dob": 12 + }, + "missed_duplicates": 24, + "new_chart_items": 440, + "true_duplicates": 223 + }, + "name_variants": false, + "profile_records": 30, + "rule": { + "calibration": { + "calibration_duplicates": 586, + "distinct_queries": 870, + "false_match_rate": 0.0, + "miss_by_error": { + "both_names_typo+contact_changed": 0.767, + "both_names_typo+contact_kept": 0.0, + "contact_changed+contact_changed": 0.0, + "day_month_swapped+contact_changed": 0.0, + "day_month_swapped+contact_kept": 0.0, + "dob_digit+contact_changed": 0.0, + "dob_digit+contact_kept": 0.0, + "first_initial+contact_changed": 0.0, + "first_initial+contact_kept": 0.0, + "first_typo+contact_changed": 0.0, + "first_typo+contact_kept": 0.0, + "last_changed+contact_changed": 0.967, + "last_changed+contact_kept": 0.0, + "last_hyphenated+contact_changed": 0.0, + "last_hyphenated+contact_kept": 0.0, + "last_second_surname+contact_changed": 0.0, + "last_second_surname+contact_kept": 0.0, + "last_typo+contact_changed": 0.0, + "last_typo+contact_kept": 0.0, + "names_swapped+contact_changed": 0.0, + "names_swapped+contact_kept": 0.0 + }, + "miss_rate": 0.0887 + }, + "close": [ + [ + [ + "email", + "eq" + ], + [ + "phone", + "eq" + ], + [ + "street", + "eq" + ] + ], + [ + [ + "first", + "edit1" + ], + [ + "last", + "share" + ], + [ + "dob", + "near" + ] + ], + [ + [ + "first", + "initial" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "dob", + "eq" + ], + [ + "names_swapped", + "eq" + ] + ], + [ + [ + "first", + "eq" + ], + [ + "last", + "edit1" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "first", + "edit1" + ], + [ + "last", + "edit1" + ], + [ + "dob", + "swap" + ] + ] + ], + "estimated_false_stops_per_record": 0.09845, + "exact": [ + [ + "first", + "eq" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ], + "name_variants": null, + "notes": [], + "roles": { + "dob": "DOB", + "email": "email", + "first": "fname", + "last": "lname", + "phone": "phone_home", + "prior_last": "prior_lname", + "street": "street" + }, + "schema_version": "openadapt.duplicate-rule/v1" + }, + "rule_text": [ + "Stop: the record exists when first the same and last the same and dob the same.", + "Stop for a person when email the same and phone the same and street the same.", + "Stop for a person when first one letter apart and last sharing a name part and dob one part off.", + "Stop for a person when first matching by initial and last the same and dob the same.", + "Stop for a person when dob the same and first and last names swapped.", + "Stop for a person when first the same and last one letter apart and dob the same.", + "Stop for a person when first one letter apart and last one letter apart and dob with day and month swapped.", + "Stop for a person when the new record lacks a name or a date of birth." + ], + "stream": "seed7-500" + }, + "seed7-500|variants=True|labels=False|budget=0.005": { + "adoption": { + "calibration_generated": { + "distinct_queries": 30, + "duplicates": 604, + "false_stops": 0, + "missed": 142, + "missed_by_error": { + "both_names_typo+contact_changed": 30, + "first_typo+contact_changed": 30, + "last_changed+contact_changed": 28, + "last_typo+contact_changed": 24, + "names_swapped+contact_changed": 30 + } + }, + "calibration_reference": { + "distinct_queries": 30, + "duplicates": 604, + "false_stops": 12, + "missed": 3, + "missed_by_error": { + "first_nickname+contact_changed": 3 + } + }, + "use": "reference", + "why": "the generated rule misses 142 of 604 calibration duplicates and the reference 3: the reference stays" + }, + "assumed_labels": false, + "budget": 0.005, + "generated": { + "false_stops": 0, + "false_stops_by_exception": {}, + "false_stops_per_100_items": 0.0, + "items": 676, + "missed_by_exception": { + "limit_married_moved_dob_typo": 12, + "limit_new_name_moved_same_dob": 12, + "pl_both_typos_same_contact": 12, + "pl_married_moved": 12, + "r2_prior_name_moved_dob_typo": 6 + }, + "missed_duplicates": 54, + "new_chart_items": 440, + "true_duplicates": 223 + }, + "generation_s": 0.224, + "hand_built_record_check": { + "false_stops": 2, + "false_stops_by_exception": { + "shared_dob": 2 + }, + "false_stops_per_100_items": 0.3, + "items": 676, + "missed_by_exception": { + "limit_married_moved_dob_typo": 12, + "limit_new_name_moved_same_dob": 12 + }, + "missed_duplicates": 24, + "new_chart_items": 440, + "true_duplicates": 223 + }, + "name_variants": true, + "profile_records": 30, + "rule": { + "calibration": { + "calibration_duplicates": 604, + "distinct_queries": 870, + "false_match_rate": 0.0, + "miss_by_error": { + "both_names_typo+contact_changed": 1.0, + "both_names_typo+contact_kept": 0.0, + "contact_changed+contact_changed": 0.0, + "day_month_swapped+contact_changed": 0.0, + "day_month_swapped+contact_kept": 0.0, + "dob_digit+contact_changed": 0.0, + "dob_digit+contact_kept": 0.0, + "first_initial+contact_changed": 0.0, + "first_initial+contact_kept": 0.0, + "first_nickname+contact_changed": 0.0, + "first_nickname+contact_kept": 0.0, + "first_typo+contact_changed": 1.0, + "first_typo+contact_kept": 0.0, + "last_changed+contact_changed": 1.0, + "last_changed+contact_kept": 0.0, + "last_hyphenated+contact_changed": 0.0, + "last_hyphenated+contact_kept": 0.0, + "last_second_surname+contact_changed": 0.0, + "last_second_surname+contact_kept": 0.0, + "last_typo+contact_changed": 0.8, + "last_typo+contact_kept": 0.0, + "names_swapped+contact_changed": 1.0, + "names_swapped+contact_kept": 0.0 + }, + "miss_rate": 0.2384 + }, + "close": [ + [ + [ + "email", + "eq" + ], + [ + "phone", + "eq" + ], + [ + "street", + "eq" + ] + ], + [ + [ + "first", + "initial" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "first", + "eq" + ], + [ + "last", + "share" + ], + [ + "dob", + "swap" + ] + ], + [ + [ + "first", + "eq" + ], + [ + "last", + "share" + ], + [ + "dob", + "near" + ] + ], + [ + [ + "first", + "variant" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ] + ], + "estimated_false_stops_per_record": 0.00437, + "exact": [ + [ + "first", + "eq" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ], + "name_variants": "a generic table of given-name variants (name_variants.json, made by a model from common names only)", + "notes": [], + "roles": { + "dob": "DOB", + "email": "email", + "first": "fname", + "last": "lname", + "phone": "phone_home", + "prior_last": "prior_lname", + "street": "street" + }, + "schema_version": "openadapt.duplicate-rule/v1" + }, + "rule_text": [ + "Stop: the record exists when first the same and last the same and dob the same.", + "Stop for a person when email the same and phone the same and street the same.", + "Stop for a person when first matching by initial and last the same and dob the same.", + "Stop for a person when first the same and last sharing a name part and dob with day and month swapped.", + "Stop for a person when first the same and last sharing a name part and dob one part off.", + "Stop for a person when first a known variant of the other (a nickname) and last the same and dob the same.", + "Stop for a person when the new record lacks a name or a date of birth.", + "Nicknames come from a generic table of given-name variants (name_variants.json, made by a model from common names only)." + ], + "stream": "seed7-500" + }, + "seed7-500|variants=True|labels=False|budget=0.01": { + "adoption": { + "calibration_generated": { + "distinct_queries": 30, + "duplicates": 604, + "false_stops": 0, + "missed": 114, + "missed_by_error": { + "both_names_typo+contact_changed": 24, + "day_month_swapped+contact_changed": 8, + "last_changed+contact_changed": 28, + "last_typo+contact_changed": 24, + "names_swapped+contact_changed": 30 + } + }, + "calibration_reference": { + "distinct_queries": 30, + "duplicates": 604, + "false_stops": 12, + "missed": 3, + "missed_by_error": { + "first_nickname+contact_changed": 3 + } + }, + "use": "reference", + "why": "the generated rule misses 114 of 604 calibration duplicates and the reference 3: the reference stays" + }, + "assumed_labels": false, + "budget": 0.01, + "generated": { + "false_stops": 0, + "false_stops_by_exception": {}, + "false_stops_per_100_items": 0.0, + "items": 676, + "missed_by_exception": { + "limit_married_moved_dob_typo": 12, + "limit_new_name_moved_same_dob": 12, + "pl_both_typos_same_contact": 12, + "pl_married_moved": 12, + "r2_prior_name_moved_dob_typo": 6 + }, + "missed_duplicates": 54, + "new_chart_items": 440, + "true_duplicates": 223 + }, + "generation_s": 0.221, + "hand_built_record_check": { + "false_stops": 2, + "false_stops_by_exception": { + "shared_dob": 2 + }, + "false_stops_per_100_items": 0.3, + "items": 676, + "missed_by_exception": { + "limit_married_moved_dob_typo": 12, + "limit_new_name_moved_same_dob": 12 + }, + "missed_duplicates": 24, + "new_chart_items": 440, + "true_duplicates": 223 + }, + "name_variants": true, + "profile_records": 30, + "rule": { + "calibration": { + "calibration_duplicates": 604, + "distinct_queries": 870, + "false_match_rate": 0.0, + "miss_by_error": { + "both_names_typo+contact_changed": 0.8, + "both_names_typo+contact_kept": 0.0, + "contact_changed+contact_changed": 0.0, + "day_month_swapped+contact_changed": 1.0, + "day_month_swapped+contact_kept": 0.0, + "dob_digit+contact_changed": 0.0, + "dob_digit+contact_kept": 0.0, + "first_initial+contact_changed": 0.0, + "first_initial+contact_kept": 0.0, + "first_nickname+contact_changed": 0.0, + "first_nickname+contact_kept": 0.0, + "first_typo+contact_changed": 0.0, + "first_typo+contact_kept": 0.0, + "last_changed+contact_changed": 1.0, + "last_changed+contact_kept": 0.0, + "last_hyphenated+contact_changed": 0.0, + "last_hyphenated+contact_kept": 0.0, + "last_second_surname+contact_changed": 0.0, + "last_second_surname+contact_kept": 0.0, + "last_typo+contact_changed": 0.8, + "last_typo+contact_kept": 0.0, + "names_swapped+contact_changed": 1.0, + "names_swapped+contact_kept": 0.0 + }, + "miss_rate": 0.1921 + }, + "close": [ + [ + [ + "email", + "eq" + ], + [ + "phone", + "eq" + ], + [ + "street", + "eq" + ] + ], + [ + [ + "first", + "edit1" + ], + [ + "last", + "share" + ], + [ + "dob", + "near" + ] + ], + [ + [ + "first", + "initial" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "first", + "variant" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ] + ], + "estimated_false_stops_per_record": 0.00958, + "exact": [ + [ + "first", + "eq" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ], + "name_variants": "a generic table of given-name variants (name_variants.json, made by a model from common names only)", + "notes": [], + "roles": { + "dob": "DOB", + "email": "email", + "first": "fname", + "last": "lname", + "phone": "phone_home", + "prior_last": "prior_lname", + "street": "street" + }, + "schema_version": "openadapt.duplicate-rule/v1" + }, + "rule_text": [ + "Stop: the record exists when first the same and last the same and dob the same.", + "Stop for a person when email the same and phone the same and street the same.", + "Stop for a person when first one letter apart and last sharing a name part and dob one part off.", + "Stop for a person when first matching by initial and last the same and dob the same.", + "Stop for a person when first a known variant of the other (a nickname) and last the same and dob the same.", + "Stop for a person when the new record lacks a name or a date of birth.", + "Nicknames come from a generic table of given-name variants (name_variants.json, made by a model from common names only)." + ], + "stream": "seed7-500" + }, + "seed7-500|variants=True|labels=False|budget=0.02": { + "adoption": { + "calibration_generated": { + "distinct_queries": 30, + "duplicates": 604, + "false_stops": 0, + "missed": 100, + "missed_by_error": { + "both_names_typo+contact_changed": 24, + "last_changed+contact_changed": 27, + "last_typo+contact_changed": 22, + "names_swapped+contact_changed": 27 + } + }, + "calibration_reference": { + "distinct_queries": 30, + "duplicates": 604, + "false_stops": 12, + "missed": 3, + "missed_by_error": { + "first_nickname+contact_changed": 3 + } + }, + "use": "reference", + "why": "the generated rule misses 100 of 604 calibration duplicates and the reference 3: the reference stays" + }, + "assumed_labels": false, + "budget": 0.02, + "generated": { + "false_stops": 0, + "false_stops_by_exception": {}, + "false_stops_per_100_items": 0.0, + "items": 676, + "missed_by_exception": { + "limit_married_moved_dob_typo": 12, + "limit_new_name_moved_same_dob": 12, + "pl_both_typos_same_contact": 12, + "pl_married_moved": 12, + "r2_prior_name_moved_dob_typo": 6 + }, + "missed_duplicates": 54, + "new_chart_items": 440, + "true_duplicates": 223 + }, + "generation_s": 0.236, + "hand_built_record_check": { + "false_stops": 2, + "false_stops_by_exception": { + "shared_dob": 2 + }, + "false_stops_per_100_items": 0.3, + "items": 676, + "missed_by_exception": { + "limit_married_moved_dob_typo": 12, + "limit_new_name_moved_same_dob": 12 + }, + "missed_duplicates": 24, + "new_chart_items": 440, + "true_duplicates": 223 + }, + "name_variants": true, + "profile_records": 30, + "rule": { + "calibration": { + "calibration_duplicates": 604, + "distinct_queries": 870, + "false_match_rate": 0.0, + "miss_by_error": { + "both_names_typo+contact_changed": 0.8, + "both_names_typo+contact_kept": 0.0, + "contact_changed+contact_changed": 0.0, + "day_month_swapped+contact_changed": 0.0, + "day_month_swapped+contact_kept": 0.0, + "dob_digit+contact_changed": 0.0, + "dob_digit+contact_kept": 0.0, + "first_initial+contact_changed": 0.0, + "first_initial+contact_kept": 0.0, + "first_nickname+contact_changed": 0.0, + "first_nickname+contact_kept": 0.0, + "first_typo+contact_changed": 0.0, + "first_typo+contact_kept": 0.0, + "last_changed+contact_changed": 0.967, + "last_changed+contact_kept": 0.0, + "last_hyphenated+contact_changed": 0.0, + "last_hyphenated+contact_kept": 0.0, + "last_second_surname+contact_changed": 0.0, + "last_second_surname+contact_kept": 0.0, + "last_typo+contact_changed": 0.733, + "last_typo+contact_kept": 0.0, + "names_swapped+contact_changed": 0.9, + "names_swapped+contact_kept": 0.0 + }, + "miss_rate": 0.1689 + }, + "close": [ + [ + [ + "email", + "eq" + ], + [ + "phone", + "eq" + ], + [ + "street", + "eq" + ] + ], + [ + [ + "first", + "edit1" + ], + [ + "last", + "share" + ], + [ + "dob", + "near" + ] + ], + [ + [ + "first", + "initial" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "first", + "eq" + ], + [ + "last", + "edit1" + ], + [ + "dob", + "swap" + ] + ], + [ + [ + "first", + "variant" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "dob", + "swap" + ], + [ + "names_swapped", + "eq" + ] + ] + ], + "estimated_false_stops_per_record": 0.01939, + "exact": [ + [ + "first", + "eq" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ], + "name_variants": "a generic table of given-name variants (name_variants.json, made by a model from common names only)", + "notes": [], + "roles": { + "dob": "DOB", + "email": "email", + "first": "fname", + "last": "lname", + "phone": "phone_home", + "prior_last": "prior_lname", + "street": "street" + }, + "schema_version": "openadapt.duplicate-rule/v1" + }, + "rule_text": [ + "Stop: the record exists when first the same and last the same and dob the same.", + "Stop for a person when email the same and phone the same and street the same.", + "Stop for a person when first one letter apart and last sharing a name part and dob one part off.", + "Stop for a person when first matching by initial and last the same and dob the same.", + "Stop for a person when first the same and last one letter apart and dob with day and month swapped.", + "Stop for a person when first a known variant of the other (a nickname) and last the same and dob the same.", + "Stop for a person when dob with day and month swapped and first and last names swapped.", + "Stop for a person when the new record lacks a name or a date of birth.", + "Nicknames come from a generic table of given-name variants (name_variants.json, made by a model from common names only)." + ], + "stream": "seed7-500" + }, + "seed7-500|variants=True|labels=False|budget=0.05": { + "adoption": { + "calibration_generated": { + "distinct_queries": 30, + "duplicates": 604, + "false_stops": 0, + "missed": 73, + "missed_by_error": { + "both_names_typo+contact_changed": 24, + "last_changed+contact_changed": 27, + "last_typo+contact_changed": 22 + } + }, + "calibration_reference": { + "distinct_queries": 30, + "duplicates": 604, + "false_stops": 12, + "missed": 3, + "missed_by_error": { + "first_nickname+contact_changed": 3 + } + }, + "use": "reference", + "why": "the generated rule misses 73 of 604 calibration duplicates and the reference 3: the reference stays" + }, + "assumed_labels": false, + "budget": 0.05, + "generated": { + "false_stops": 0, + "false_stops_by_exception": {}, + "false_stops_per_100_items": 0.0, + "items": 676, + "missed_by_exception": { + "limit_married_moved_dob_typo": 12, + "limit_new_name_moved_same_dob": 12, + "pl_both_typos_same_contact": 12, + "pl_married_moved": 12, + "r2_prior_name_moved_dob_typo": 6 + }, + "missed_duplicates": 54, + "new_chart_items": 440, + "true_duplicates": 223 + }, + "generation_s": 0.225, + "hand_built_record_check": { + "false_stops": 2, + "false_stops_by_exception": { + "shared_dob": 2 + }, + "false_stops_per_100_items": 0.3, + "items": 676, + "missed_by_exception": { + "limit_married_moved_dob_typo": 12, + "limit_new_name_moved_same_dob": 12 + }, + "missed_duplicates": 24, + "new_chart_items": 440, + "true_duplicates": 223 + }, + "name_variants": true, + "profile_records": 30, + "rule": { + "calibration": { + "calibration_duplicates": 604, + "distinct_queries": 870, + "false_match_rate": 0.0, + "miss_by_error": { + "both_names_typo+contact_changed": 0.8, + "both_names_typo+contact_kept": 0.0, + "contact_changed+contact_changed": 0.0, + "day_month_swapped+contact_changed": 0.0, + "day_month_swapped+contact_kept": 0.0, + "dob_digit+contact_changed": 0.0, + "dob_digit+contact_kept": 0.0, + "first_initial+contact_changed": 0.0, + "first_initial+contact_kept": 0.0, + "first_nickname+contact_changed": 0.0, + "first_nickname+contact_kept": 0.0, + "first_typo+contact_changed": 0.0, + "first_typo+contact_kept": 0.0, + "last_changed+contact_changed": 0.967, + "last_changed+contact_kept": 0.0, + "last_hyphenated+contact_changed": 0.0, + "last_hyphenated+contact_kept": 0.0, + "last_second_surname+contact_changed": 0.0, + "last_second_surname+contact_kept": 0.0, + "last_typo+contact_changed": 0.733, + "last_typo+contact_kept": 0.0, + "names_swapped+contact_changed": 0.0, + "names_swapped+contact_kept": 0.0 + }, + "miss_rate": 0.1242 + }, + "close": [ + [ + [ + "email", + "eq" + ], + [ + "phone", + "eq" + ], + [ + "street", + "eq" + ] + ], + [ + [ + "first", + "edit1" + ], + [ + "last", + "share" + ], + [ + "dob", + "near" + ] + ], + [ + [ + "first", + "initial" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "dob", + "eq" + ], + [ + "names_swapped", + "eq" + ] + ], + [ + [ + "first", + "eq" + ], + [ + "last", + "edit1" + ], + [ + "dob", + "swap" + ] + ], + [ + [ + "first", + "variant" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ] + ], + "estimated_false_stops_per_record": 0.04782, + "exact": [ + [ + "first", + "eq" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ], + "name_variants": "a generic table of given-name variants (name_variants.json, made by a model from common names only)", + "notes": [], + "roles": { + "dob": "DOB", + "email": "email", + "first": "fname", + "last": "lname", + "phone": "phone_home", + "prior_last": "prior_lname", + "street": "street" + }, + "schema_version": "openadapt.duplicate-rule/v1" + }, + "rule_text": [ + "Stop: the record exists when first the same and last the same and dob the same.", + "Stop for a person when email the same and phone the same and street the same.", + "Stop for a person when first one letter apart and last sharing a name part and dob one part off.", + "Stop for a person when first matching by initial and last the same and dob the same.", + "Stop for a person when dob the same and first and last names swapped.", + "Stop for a person when first the same and last one letter apart and dob with day and month swapped.", + "Stop for a person when first a known variant of the other (a nickname) and last the same and dob the same.", + "Stop for a person when the new record lacks a name or a date of birth.", + "Nicknames come from a generic table of given-name variants (name_variants.json, made by a model from common names only)." + ], + "stream": "seed7-500" + }, + "seed7-500|variants=True|labels=False|budget=0.1": { + "adoption": { + "calibration_generated": { + "distinct_queries": 30, + "duplicates": 604, + "false_stops": 0, + "missed": 45, + "missed_by_error": { + "both_names_typo+contact_changed": 22, + "last_changed+contact_changed": 23 + } + }, + "calibration_reference": { + "distinct_queries": 30, + "duplicates": 604, + "false_stops": 12, + "missed": 3, + "missed_by_error": { + "first_nickname+contact_changed": 3 + } + }, + "use": "reference", + "why": "the generated rule misses 45 of 604 calibration duplicates and the reference 3: the reference stays" + }, + "assumed_labels": false, + "budget": 0.1, + "generated": { + "false_stops": 0, + "false_stops_by_exception": {}, + "false_stops_per_100_items": 0.0, + "items": 676, + "missed_by_exception": { + "limit_married_moved_dob_typo": 12, + "limit_new_name_moved_same_dob": 12, + "pl_both_typos_same_contact": 11, + "pl_married_moved": 12, + "r2_prior_name_moved_dob_typo": 6 + }, + "missed_duplicates": 53, + "new_chart_items": 440, + "true_duplicates": 223 + }, + "generation_s": 0.228, + "hand_built_record_check": { + "false_stops": 2, + "false_stops_by_exception": { + "shared_dob": 2 + }, + "false_stops_per_100_items": 0.3, + "items": 676, + "missed_by_exception": { + "limit_married_moved_dob_typo": 12, + "limit_new_name_moved_same_dob": 12 + }, + "missed_duplicates": 24, + "new_chart_items": 440, + "true_duplicates": 223 + }, + "name_variants": true, + "profile_records": 30, + "rule": { + "calibration": { + "calibration_duplicates": 604, + "distinct_queries": 870, + "false_match_rate": 0.0, + "miss_by_error": { + "both_names_typo+contact_changed": 0.733, + "both_names_typo+contact_kept": 0.0, + "contact_changed+contact_changed": 0.0, + "day_month_swapped+contact_changed": 0.0, + "day_month_swapped+contact_kept": 0.0, + "dob_digit+contact_changed": 0.0, + "dob_digit+contact_kept": 0.0, + "first_initial+contact_changed": 0.0, + "first_initial+contact_kept": 0.0, + "first_nickname+contact_changed": 0.0, + "first_nickname+contact_kept": 0.0, + "first_typo+contact_changed": 0.0, + "first_typo+contact_kept": 0.0, + "last_changed+contact_changed": 0.867, + "last_changed+contact_kept": 0.0, + "last_hyphenated+contact_changed": 0.0, + "last_hyphenated+contact_kept": 0.0, + "last_second_surname+contact_changed": 0.0, + "last_second_surname+contact_kept": 0.0, + "last_typo+contact_changed": 0.0, + "last_typo+contact_kept": 0.0, + "names_swapped+contact_changed": 0.0, + "names_swapped+contact_kept": 0.0 + }, + "miss_rate": 0.0795 + }, + "close": [ + [ + [ + "email", + "eq" + ], + [ + "phone", + "eq" + ], + [ + "street", + "eq" + ] + ], + [ + [ + "first", + "edit1" + ], + [ + "last", + "share" + ], + [ + "dob", + "near" + ] + ], + [ + [ + "first", + "initial" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "dob", + "eq" + ], + [ + "names_swapped", + "eq" + ] + ], + [ + [ + "first", + "eq" + ], + [ + "last", + "edit1" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "first", + "edit1" + ], + [ + "last", + "edit1" + ], + [ + "dob", + "swap" + ] + ], + [ + [ + "first", + "variant" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ] + ], + "estimated_false_stops_per_record": 0.09901, + "exact": [ + [ + "first", + "eq" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ], + "name_variants": "a generic table of given-name variants (name_variants.json, made by a model from common names only)", + "notes": [], + "roles": { + "dob": "DOB", + "email": "email", + "first": "fname", + "last": "lname", + "phone": "phone_home", + "prior_last": "prior_lname", + "street": "street" + }, + "schema_version": "openadapt.duplicate-rule/v1" + }, + "rule_text": [ + "Stop: the record exists when first the same and last the same and dob the same.", + "Stop for a person when email the same and phone the same and street the same.", + "Stop for a person when first one letter apart and last sharing a name part and dob one part off.", + "Stop for a person when first matching by initial and last the same and dob the same.", + "Stop for a person when dob the same and first and last names swapped.", + "Stop for a person when first the same and last one letter apart and dob the same.", + "Stop for a person when first one letter apart and last one letter apart and dob with day and month swapped.", + "Stop for a person when first a known variant of the other (a nickname) and last the same and dob the same.", + "Stop for a person when the new record lacks a name or a date of birth.", + "Nicknames come from a generic table of given-name variants (name_variants.json, made by a model from common names only)." + ], + "stream": "seed7-500" + }, + "seed7-500|variants=True|labels=True|budget=0.005": { + "adoption": { + "calibration_generated": { + "distinct_queries": 30, + "duplicates": 604, + "false_stops": 0, + "missed": 135, + "missed_by_error": { + "both_names_typo+contact_changed": 29, + "first_typo+contact_changed": 27, + "last_changed+contact_changed": 27, + "last_typo+contact_changed": 22, + "names_swapped+contact_changed": 30 + } + }, + "calibration_reference": { + "distinct_queries": 30, + "duplicates": 604, + "false_stops": 12, + "missed": 3, + "missed_by_error": { + "first_nickname+contact_changed": 3 + } + }, + "use": "reference", + "why": "the generated rule misses 135 of 604 calibration duplicates and the reference 3: the reference stays" + }, + "assumed_labels": true, + "budget": 0.005, + "generated": { + "false_stops": 0, + "false_stops_by_exception": {}, + "false_stops_per_100_items": 0.0, + "items": 676, + "missed_by_exception": { + "limit_married_moved_dob_typo": 12, + "limit_new_name_moved_same_dob": 12, + "pl_both_typos_same_contact": 12, + "pl_married_moved": 12, + "r2_prior_name_moved_dob_typo": 6 + }, + "missed_duplicates": 54, + "new_chart_items": 440, + "true_duplicates": 223 + }, + "generation_s": 0.25, + "hand_built_record_check": { + "false_stops": 2, + "false_stops_by_exception": { + "shared_dob": 2 + }, + "false_stops_per_100_items": 0.3, + "items": 676, + "missed_by_exception": { + "limit_married_moved_dob_typo": 12, + "limit_new_name_moved_same_dob": 12 + }, + "missed_duplicates": 24, + "new_chart_items": 440, + "true_duplicates": 223 + }, + "name_variants": true, + "profile_records": 30, + "rule": { + "calibration": { + "calibration_duplicates": 604, + "distinct_queries": 858, + "false_match_rate": 0.0, + "miss_by_error": { + "both_names_typo+contact_changed": 0.967, + "both_names_typo+contact_kept": 0.0, + "contact_changed+contact_changed": 0.0, + "day_month_swapped+contact_changed": 0.0, + "day_month_swapped+contact_kept": 0.0, + "dob_digit+contact_changed": 0.0, + "dob_digit+contact_kept": 0.0, + "first_initial+contact_changed": 0.0, + "first_initial+contact_kept": 0.0, + "first_nickname+contact_changed": 0.0, + "first_nickname+contact_kept": 0.0, + "first_typo+contact_changed": 0.9, + "first_typo+contact_kept": 0.0, + "last_changed+contact_changed": 0.967, + "last_changed+contact_kept": 0.0, + "last_hyphenated+contact_changed": 0.0, + "last_hyphenated+contact_kept": 0.0, + "last_second_surname+contact_changed": 0.0, + "last_second_surname+contact_kept": 0.0, + "last_typo+contact_changed": 0.733, + "last_typo+contact_kept": 0.0, + "names_swapped+contact_changed": 1.0, + "names_swapped+contact_kept": 0.0 + }, + "miss_rate": 0.2268 + }, + "close": [ + [ + [ + "email", + "eq" + ], + [ + "phone", + "eq" + ], + [ + "street", + "eq" + ] + ], + [ + [ + "first", + "eq" + ], + [ + "last", + "share" + ], + [ + "dob", + "near" + ] + ], + [ + [ + "first", + "initial" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "first", + "edit1" + ], + [ + "last", + "share" + ], + [ + "dob", + "swap" + ] + ], + [ + [ + "first", + "variant" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "first", + "eq" + ], + [ + "last", + "edit1" + ], + [ + "dob", + "swap" + ] + ] + ], + "estimated_false_stops_per_record": 0.00321, + "exact": [ + [ + "first", + "eq" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ], + "name_variants": "a generic table of given-name variants (name_variants.json, made by a model from common names only)", + "notes": [ + "a person labelled 6 pair(s) of existing records as one person; they no longer count as two" + ], + "roles": { + "dob": "DOB", + "email": "email", + "first": "fname", + "last": "lname", + "phone": "phone_home", + "prior_last": "prior_lname", + "street": "street" + }, + "schema_version": "openadapt.duplicate-rule/v1" + }, + "rule_text": [ + "Stop: the record exists when first the same and last the same and dob the same.", + "Stop for a person when email the same and phone the same and street the same.", + "Stop for a person when first the same and last sharing a name part and dob one part off.", + "Stop for a person when first matching by initial and last the same and dob the same.", + "Stop for a person when first one letter apart and last sharing a name part and dob with day and month swapped.", + "Stop for a person when first a known variant of the other (a nickname) and last the same and dob the same.", + "Stop for a person when first the same and last one letter apart and dob with day and month swapped.", + "Stop for a person when the new record lacks a name or a date of birth.", + "Nicknames come from a generic table of given-name variants (name_variants.json, made by a model from common names only)." + ], + "stream": "seed7-500" + }, + "seed7-500|variants=True|labels=True|budget=0.01": { + "adoption": { + "calibration_generated": { + "distinct_queries": 30, + "duplicates": 604, + "false_stops": 0, + "missed": 103, + "missed_by_error": { + "both_names_typo+contact_changed": 24, + "last_changed+contact_changed": 27, + "last_typo+contact_changed": 22, + "names_swapped+contact_changed": 30 + } + }, + "calibration_reference": { + "distinct_queries": 30, + "duplicates": 604, + "false_stops": 12, + "missed": 3, + "missed_by_error": { + "first_nickname+contact_changed": 3 + } + }, + "use": "reference", + "why": "the generated rule misses 103 of 604 calibration duplicates and the reference 3: the reference stays" + }, + "assumed_labels": true, + "budget": 0.01, + "generated": { + "false_stops": 0, + "false_stops_by_exception": {}, + "false_stops_per_100_items": 0.0, + "items": 676, + "missed_by_exception": { + "limit_married_moved_dob_typo": 12, + "limit_new_name_moved_same_dob": 12, + "pl_both_typos_same_contact": 12, + "pl_married_moved": 12, + "r2_prior_name_moved_dob_typo": 6 + }, + "missed_duplicates": 54, + "new_chart_items": 440, + "true_duplicates": 223 + }, + "generation_s": 0.232, + "hand_built_record_check": { + "false_stops": 2, + "false_stops_by_exception": { + "shared_dob": 2 + }, + "false_stops_per_100_items": 0.3, + "items": 676, + "missed_by_exception": { + "limit_married_moved_dob_typo": 12, + "limit_new_name_moved_same_dob": 12 + }, + "missed_duplicates": 24, + "new_chart_items": 440, + "true_duplicates": 223 + }, + "name_variants": true, + "profile_records": 30, + "rule": { + "calibration": { + "calibration_duplicates": 604, + "distinct_queries": 858, + "false_match_rate": 0.0, + "miss_by_error": { + "both_names_typo+contact_changed": 0.8, + "both_names_typo+contact_kept": 0.0, + "contact_changed+contact_changed": 0.0, + "day_month_swapped+contact_changed": 0.0, + "day_month_swapped+contact_kept": 0.0, + "dob_digit+contact_changed": 0.0, + "dob_digit+contact_kept": 0.0, + "first_initial+contact_changed": 0.0, + "first_initial+contact_kept": 0.0, + "first_nickname+contact_changed": 0.0, + "first_nickname+contact_kept": 0.0, + "first_typo+contact_changed": 0.0, + "first_typo+contact_kept": 0.0, + "last_changed+contact_changed": 0.967, + "last_changed+contact_kept": 0.0, + "last_hyphenated+contact_changed": 0.0, + "last_hyphenated+contact_kept": 0.0, + "last_second_surname+contact_changed": 0.0, + "last_second_surname+contact_kept": 0.0, + "last_typo+contact_changed": 0.733, + "last_typo+contact_kept": 0.0, + "names_swapped+contact_changed": 1.0, + "names_swapped+contact_kept": 0.0 + }, + "miss_rate": 0.1738 + }, + "close": [ + [ + [ + "email", + "eq" + ], + [ + "phone", + "eq" + ], + [ + "street", + "eq" + ] + ], + [ + [ + "first", + "edit1" + ], + [ + "last", + "share" + ], + [ + "dob", + "near" + ] + ], + [ + [ + "first", + "initial" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "first", + "eq" + ], + [ + "last", + "edit1" + ], + [ + "dob", + "swap" + ] + ], + [ + [ + "first", + "variant" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ] + ], + "estimated_false_stops_per_record": 0.00711, + "exact": [ + [ + "first", + "eq" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ], + "name_variants": "a generic table of given-name variants (name_variants.json, made by a model from common names only)", + "notes": [ + "a person labelled 6 pair(s) of existing records as one person; they no longer count as two" + ], + "roles": { + "dob": "DOB", + "email": "email", + "first": "fname", + "last": "lname", + "phone": "phone_home", + "prior_last": "prior_lname", + "street": "street" + }, + "schema_version": "openadapt.duplicate-rule/v1" + }, + "rule_text": [ + "Stop: the record exists when first the same and last the same and dob the same.", + "Stop for a person when email the same and phone the same and street the same.", + "Stop for a person when first one letter apart and last sharing a name part and dob one part off.", + "Stop for a person when first matching by initial and last the same and dob the same.", + "Stop for a person when first the same and last one letter apart and dob with day and month swapped.", + "Stop for a person when first a known variant of the other (a nickname) and last the same and dob the same.", + "Stop for a person when the new record lacks a name or a date of birth.", + "Nicknames come from a generic table of given-name variants (name_variants.json, made by a model from common names only)." + ], + "stream": "seed7-500" + }, + "seed7-500|variants=True|labels=True|budget=0.02": { + "adoption": { + "calibration_generated": { + "distinct_queries": 30, + "duplicates": 604, + "false_stops": 0, + "missed": 74, + "missed_by_error": { + "both_names_typo+contact_changed": 24, + "last_changed+contact_changed": 23, + "names_swapped+contact_changed": 27 + } + }, + "calibration_reference": { + "distinct_queries": 30, + "duplicates": 604, + "false_stops": 12, + "missed": 3, + "missed_by_error": { + "first_nickname+contact_changed": 3 + } + }, + "use": "reference", + "why": "the generated rule misses 74 of 604 calibration duplicates and the reference 3: the reference stays" + }, + "assumed_labels": true, + "budget": 0.02, + "generated": { + "false_stops": 0, + "false_stops_by_exception": {}, + "false_stops_per_100_items": 0.0, + "items": 676, + "missed_by_exception": { + "limit_married_moved_dob_typo": 12, + "limit_new_name_moved_same_dob": 12, + "pl_both_typos_same_contact": 12, + "pl_married_moved": 12, + "r2_prior_name_moved_dob_typo": 6 + }, + "missed_duplicates": 54, + "new_chart_items": 440, + "true_duplicates": 223 + }, + "generation_s": 0.252, + "hand_built_record_check": { + "false_stops": 2, + "false_stops_by_exception": { + "shared_dob": 2 + }, + "false_stops_per_100_items": 0.3, + "items": 676, + "missed_by_exception": { + "limit_married_moved_dob_typo": 12, + "limit_new_name_moved_same_dob": 12 + }, + "missed_duplicates": 24, + "new_chart_items": 440, + "true_duplicates": 223 + }, + "name_variants": true, + "profile_records": 30, + "rule": { + "calibration": { + "calibration_duplicates": 604, + "distinct_queries": 858, + "false_match_rate": 0.0, + "miss_by_error": { + "both_names_typo+contact_changed": 0.8, + "both_names_typo+contact_kept": 0.0, + "contact_changed+contact_changed": 0.0, + "day_month_swapped+contact_changed": 0.0, + "day_month_swapped+contact_kept": 0.0, + "dob_digit+contact_changed": 0.0, + "dob_digit+contact_kept": 0.0, + "first_initial+contact_changed": 0.0, + "first_initial+contact_kept": 0.0, + "first_nickname+contact_changed": 0.0, + "first_nickname+contact_kept": 0.0, + "first_typo+contact_changed": 0.0, + "first_typo+contact_kept": 0.0, + "last_changed+contact_changed": 0.867, + "last_changed+contact_kept": 0.0, + "last_hyphenated+contact_changed": 0.0, + "last_hyphenated+contact_kept": 0.0, + "last_second_surname+contact_changed": 0.0, + "last_second_surname+contact_kept": 0.0, + "last_typo+contact_changed": 0.0, + "last_typo+contact_kept": 0.0, + "names_swapped+contact_changed": 0.9, + "names_swapped+contact_kept": 0.0 + }, + "miss_rate": 0.1275 + }, + "close": [ + [ + [ + "email", + "eq" + ], + [ + "phone", + "eq" + ], + [ + "street", + "eq" + ] + ], + [ + [ + "first", + "edit1" + ], + [ + "last", + "share" + ], + [ + "dob", + "near" + ] + ], + [ + [ + "first", + "initial" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "first", + "eq" + ], + [ + "last", + "edit1" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "first", + "variant" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "first", + "eq" + ], + [ + "last", + "eq" + ], + [ + "dob", + "swap" + ] + ], + [ + [ + "dob", + "swap" + ], + [ + "names_swapped", + "eq" + ] + ] + ], + "estimated_false_stops_per_record": 0.01902, + "exact": [ + [ + "first", + "eq" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ], + "name_variants": "a generic table of given-name variants (name_variants.json, made by a model from common names only)", + "notes": [ + "a person labelled 6 pair(s) of existing records as one person; they no longer count as two" + ], + "roles": { + "dob": "DOB", + "email": "email", + "first": "fname", + "last": "lname", + "phone": "phone_home", + "prior_last": "prior_lname", + "street": "street" + }, + "schema_version": "openadapt.duplicate-rule/v1" + }, + "rule_text": [ + "Stop: the record exists when first the same and last the same and dob the same.", + "Stop for a person when email the same and phone the same and street the same.", + "Stop for a person when first one letter apart and last sharing a name part and dob one part off.", + "Stop for a person when first matching by initial and last the same and dob the same.", + "Stop for a person when first the same and last one letter apart and dob the same.", + "Stop for a person when first a known variant of the other (a nickname) and last the same and dob the same.", + "Stop for a person when first the same and last the same and dob with day and month swapped.", + "Stop for a person when dob with day and month swapped and first and last names swapped.", + "Stop for a person when the new record lacks a name or a date of birth.", + "Nicknames come from a generic table of given-name variants (name_variants.json, made by a model from common names only)." + ], + "stream": "seed7-500" + }, + "seed7-500|variants=True|labels=True|budget=0.05": { + "adoption": { + "calibration_generated": { + "distinct_queries": 30, + "duplicates": 604, + "false_stops": 0, + "missed": 47, + "missed_by_error": { + "both_names_typo+contact_changed": 24, + "last_changed+contact_changed": 23 + } + }, + "calibration_reference": { + "distinct_queries": 30, + "duplicates": 604, + "false_stops": 12, + "missed": 3, + "missed_by_error": { + "first_nickname+contact_changed": 3 + } + }, + "use": "reference", + "why": "the generated rule misses 47 of 604 calibration duplicates and the reference 3: the reference stays" + }, + "assumed_labels": true, + "budget": 0.05, + "generated": { + "false_stops": 0, + "false_stops_by_exception": {}, + "false_stops_per_100_items": 0.0, + "items": 676, + "missed_by_exception": { + "limit_married_moved_dob_typo": 12, + "limit_new_name_moved_same_dob": 12, + "pl_both_typos_same_contact": 12, + "pl_married_moved": 12, + "r2_prior_name_moved_dob_typo": 6 + }, + "missed_duplicates": 54, + "new_chart_items": 440, + "true_duplicates": 223 + }, + "generation_s": 0.236, + "hand_built_record_check": { + "false_stops": 2, + "false_stops_by_exception": { + "shared_dob": 2 + }, + "false_stops_per_100_items": 0.3, + "items": 676, + "missed_by_exception": { + "limit_married_moved_dob_typo": 12, + "limit_new_name_moved_same_dob": 12 + }, + "missed_duplicates": 24, + "new_chart_items": 440, + "true_duplicates": 223 + }, + "name_variants": true, + "profile_records": 30, + "rule": { + "calibration": { + "calibration_duplicates": 604, + "distinct_queries": 858, + "false_match_rate": 0.0, + "miss_by_error": { + "both_names_typo+contact_changed": 0.8, + "both_names_typo+contact_kept": 0.0, + "contact_changed+contact_changed": 0.0, + "day_month_swapped+contact_changed": 0.0, + "day_month_swapped+contact_kept": 0.0, + "dob_digit+contact_changed": 0.0, + "dob_digit+contact_kept": 0.0, + "first_initial+contact_changed": 0.0, + "first_initial+contact_kept": 0.0, + "first_nickname+contact_changed": 0.0, + "first_nickname+contact_kept": 0.0, + "first_typo+contact_changed": 0.0, + "first_typo+contact_kept": 0.0, + "last_changed+contact_changed": 0.867, + "last_changed+contact_kept": 0.0, + "last_hyphenated+contact_changed": 0.0, + "last_hyphenated+contact_kept": 0.0, + "last_second_surname+contact_changed": 0.0, + "last_second_surname+contact_kept": 0.0, + "last_typo+contact_changed": 0.0, + "last_typo+contact_kept": 0.0, + "names_swapped+contact_changed": 0.0, + "names_swapped+contact_kept": 0.0 + }, + "miss_rate": 0.0828 + }, + "close": [ + [ + [ + "email", + "eq" + ], + [ + "phone", + "eq" + ], + [ + "street", + "eq" + ] + ], + [ + [ + "first", + "initial" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "first", + "eq" + ], + [ + "last", + "share" + ], + [ + "dob", + "swap" + ] + ], + [ + [ + "first", + "eq" + ], + [ + "last", + "share" + ], + [ + "dob", + "near" + ] + ], + [ + [ + "first", + "variant" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "first", + "edit1" + ], + [ + "last", + "share" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "first", + "eq" + ], + [ + "last", + "edit1" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "dob", + "eq" + ], + [ + "names_swapped", + "eq" + ] + ] + ], + "estimated_false_stops_per_record": 0.04266, + "exact": [ + [ + "first", + "eq" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ], + "name_variants": "a generic table of given-name variants (name_variants.json, made by a model from common names only)", + "notes": [ + "a person labelled 6 pair(s) of existing records as one person; they no longer count as two" + ], + "roles": { + "dob": "DOB", + "email": "email", + "first": "fname", + "last": "lname", + "phone": "phone_home", + "prior_last": "prior_lname", + "street": "street" + }, + "schema_version": "openadapt.duplicate-rule/v1" + }, + "rule_text": [ + "Stop: the record exists when first the same and last the same and dob the same.", + "Stop for a person when email the same and phone the same and street the same.", + "Stop for a person when first matching by initial and last the same and dob the same.", + "Stop for a person when first the same and last sharing a name part and dob with day and month swapped.", + "Stop for a person when first the same and last sharing a name part and dob one part off.", + "Stop for a person when first a known variant of the other (a nickname) and last the same and dob the same.", + "Stop for a person when first one letter apart and last sharing a name part and dob the same.", + "Stop for a person when first the same and last one letter apart and dob the same.", + "Stop for a person when dob the same and first and last names swapped.", + "Stop for a person when the new record lacks a name or a date of birth.", + "Nicknames come from a generic table of given-name variants (name_variants.json, made by a model from common names only)." + ], + "stream": "seed7-500" + }, + "seed7-500|variants=True|labels=True|budget=0.1": { + "adoption": { + "calibration_generated": { + "distinct_queries": 30, + "duplicates": 604, + "false_stops": 2, + "missed": 41, + "missed_by_error": { + "both_names_typo+contact_changed": 22, + "last_changed+contact_changed": 19 + } + }, + "calibration_reference": { + "distinct_queries": 30, + "duplicates": 604, + "false_stops": 12, + "missed": 3, + "missed_by_error": { + "first_nickname+contact_changed": 3 + } + }, + "use": "reference", + "why": "the generated rule misses 41 of 604 calibration duplicates and the reference 3: the reference stays" + }, + "assumed_labels": true, + "budget": 0.1, + "generated": { + "false_stops": 0, + "false_stops_by_exception": {}, + "false_stops_per_100_items": 0.0, + "items": 676, + "missed_by_exception": { + "limit_married_moved_dob_typo": 12, + "limit_new_name_moved_same_dob": 12, + "pl_both_typos_same_contact": 11, + "pl_married_moved": 11, + "r2_prior_name_moved_dob_typo": 6 + }, + "missed_duplicates": 52, + "new_chart_items": 440, + "true_duplicates": 223 + }, + "generation_s": 0.239, + "hand_built_record_check": { + "false_stops": 2, + "false_stops_by_exception": { + "shared_dob": 2 + }, + "false_stops_per_100_items": 0.3, + "items": 676, + "missed_by_exception": { + "limit_married_moved_dob_typo": 12, + "limit_new_name_moved_same_dob": 12 + }, + "missed_duplicates": 24, + "new_chart_items": 440, + "true_duplicates": 223 + }, + "name_variants": true, + "profile_records": 30, + "rule": { + "calibration": { + "calibration_duplicates": 604, + "distinct_queries": 858, + "false_match_rate": 0.0, + "miss_by_error": { + "both_names_typo+contact_changed": 0.733, + "both_names_typo+contact_kept": 0.0, + "contact_changed+contact_changed": 0.0, + "day_month_swapped+contact_changed": 0.0, + "day_month_swapped+contact_kept": 0.0, + "dob_digit+contact_changed": 0.0, + "dob_digit+contact_kept": 0.0, + "first_initial+contact_changed": 0.0, + "first_initial+contact_kept": 0.0, + "first_nickname+contact_changed": 0.0, + "first_nickname+contact_kept": 0.0, + "first_typo+contact_changed": 0.0, + "first_typo+contact_kept": 0.0, + "last_changed+contact_changed": 0.733, + "last_changed+contact_kept": 0.0, + "last_hyphenated+contact_changed": 0.0, + "last_hyphenated+contact_kept": 0.0, + "last_second_surname+contact_changed": 0.0, + "last_second_surname+contact_kept": 0.0, + "last_typo+contact_changed": 0.0, + "last_typo+contact_kept": 0.0, + "names_swapped+contact_changed": 0.0, + "names_swapped+contact_kept": 0.0 + }, + "miss_rate": 0.0728 + }, + "close": [ + [ + [ + "email", + "eq" + ], + [ + "phone", + "eq" + ], + [ + "street", + "eq" + ] + ], + [ + [ + "first", + "initial" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "first", + "eq" + ], + [ + "last", + "share" + ], + [ + "dob", + "swap" + ] + ], + [ + [ + "first", + "eq" + ], + [ + "last", + "share" + ], + [ + "dob", + "near" + ] + ], + [ + [ + "first", + "variant" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "first", + "edit1" + ], + [ + "last", + "share" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "first", + "eq" + ], + [ + "last", + "edit1" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "dob", + "eq" + ], + [ + "names_swapped", + "eq" + ] + ], + [ + [ + "first", + "edit1" + ], + [ + "last", + "edit1" + ], + [ + "dob", + "swap" + ] + ], + [ + [ + "first", + "eq" + ], + [ + "last", + "sound" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "first", + "eq" + ], + [ + "dob", + "swap" + ] + ] + ], + "estimated_false_stops_per_record": 0.08134, + "exact": [ + [ + "first", + "eq" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ], + "name_variants": "a generic table of given-name variants (name_variants.json, made by a model from common names only)", + "notes": [ + "a person labelled 6 pair(s) of existing records as one person; they no longer count as two" + ], + "roles": { + "dob": "DOB", + "email": "email", + "first": "fname", + "last": "lname", + "phone": "phone_home", + "prior_last": "prior_lname", + "street": "street" + }, + "schema_version": "openadapt.duplicate-rule/v1" + }, + "rule_text": [ + "Stop: the record exists when first the same and last the same and dob the same.", + "Stop for a person when email the same and phone the same and street the same.", + "Stop for a person when first matching by initial and last the same and dob the same.", + "Stop for a person when first the same and last sharing a name part and dob with day and month swapped.", + "Stop for a person when first the same and last sharing a name part and dob one part off.", + "Stop for a person when first a known variant of the other (a nickname) and last the same and dob the same.", + "Stop for a person when first one letter apart and last sharing a name part and dob the same.", + "Stop for a person when first the same and last one letter apart and dob the same.", + "Stop for a person when dob the same and first and last names swapped.", + "Stop for a person when first one letter apart and last one letter apart and dob with day and month swapped.", + "Stop for a person when first the same and last sounding alike and dob the same.", + "Stop for a person when first the same and dob with day and month swapped.", + "Stop for a person when the new record lacks a name or a date of birth.", + "Nicknames come from a generic table of given-name variants (name_variants.json, made by a model from common names only)." + ], + "stream": "seed7-500" + }, + "soak-seed7-55|variants=False|labels=False|budget=0.005": { + "adoption": { + "calibration_generated": { + "distinct_queries": 30, + "duplicates": 586, + "false_stops": 0, + "missed": 102, + "missed_by_error": { + "both_names_typo+contact_changed": 23, + "last_changed+contact_changed": 30, + "last_typo+contact_changed": 22, + "names_swapped+contact_changed": 27 + } + }, + "calibration_reference": { + "distinct_queries": 30, + "duplicates": 586, + "false_stops": 12, + "missed": 0, + "missed_by_error": {} + }, + "use": "reference", + "why": "the generated rule misses 102 of 586 calibration duplicates and the reference 0: the reference stays" + }, + "assumed_labels": false, + "budget": 0.005, + "generated": { + "false_stops": 0, + "false_stops_by_exception": {}, + "false_stops_per_100_items": 0.0, + "items": 231, + "missed_by_exception": { + "limit_married_moved_dob_typo": 12, + "limit_new_name_moved_same_dob": 12, + "pl_both_typos_same_contact": 11, + "pl_married_moved": 12, + "r2_prior_name_moved_dob_typo": 6 + }, + "missed_duplicates": 53, + "new_chart_items": 45, + "true_duplicates": 184 + }, + "generation_s": 0.215, + "hand_built_record_check": { + "false_stops": 0, + "false_stops_by_exception": {}, + "false_stops_per_100_items": 0.0, + "items": 231, + "missed_by_exception": { + "limit_married_moved_dob_typo": 12, + "limit_new_name_moved_same_dob": 12 + }, + "missed_duplicates": 24, + "new_chart_items": 45, + "true_duplicates": 184 + }, + "name_variants": false, + "profile_records": 30, + "rule": { + "calibration": { + "calibration_duplicates": 586, + "distinct_queries": 870, + "false_match_rate": 0.0, + "miss_by_error": { + "both_names_typo+contact_changed": 0.767, + "both_names_typo+contact_kept": 0.0, + "contact_changed+contact_changed": 0.0, + "day_month_swapped+contact_changed": 0.0, + "day_month_swapped+contact_kept": 0.0, + "dob_digit+contact_changed": 0.0, + "dob_digit+contact_kept": 0.0, + "first_initial+contact_changed": 0.0, + "first_initial+contact_kept": 0.0, + "first_typo+contact_changed": 0.0, + "first_typo+contact_kept": 0.0, + "last_changed+contact_changed": 1.0, + "last_changed+contact_kept": 0.0, + "last_hyphenated+contact_changed": 0.0, + "last_hyphenated+contact_kept": 0.0, + "last_second_surname+contact_changed": 0.0, + "last_second_surname+contact_kept": 0.0, + "last_typo+contact_changed": 0.733, + "last_typo+contact_kept": 0.0, + "names_swapped+contact_changed": 0.9, + "names_swapped+contact_kept": 0.0 + }, + "miss_rate": 0.1741 + }, + "close": [ + [ + [ + "email", + "eq" + ], + [ + "phone", + "eq" + ], + [ + "street", + "eq" + ] + ], + [ + [ + "first", + "edit1" + ], + [ + "last", + "share" + ], + [ + "dob", + "near" + ] + ], + [ + [ + "first", + "initial" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "first", + "edit1" + ], + [ + "last", + "edit1" + ], + [ + "dob", + "swap" + ] + ], + [ + [ + "dob", + "swap" + ], + [ + "names_swapped", + "eq" + ] + ] + ], + "estimated_false_stops_per_record": 0.00455, + "exact": [ + [ + "first", + "eq" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ], + "name_variants": null, + "notes": [], + "roles": { + "dob": "DOB", + "email": "email", + "first": "fname", + "last": "lname", + "phone": "phone_home", + "prior_last": "prior_lname", + "street": "street" + }, + "schema_version": "openadapt.duplicate-rule/v1" + }, + "rule_text": [ + "Stop: the record exists when first the same and last the same and dob the same.", + "Stop for a person when email the same and phone the same and street the same.", + "Stop for a person when first one letter apart and last sharing a name part and dob one part off.", + "Stop for a person when first matching by initial and last the same and dob the same.", + "Stop for a person when first one letter apart and last one letter apart and dob with day and month swapped.", + "Stop for a person when dob with day and month swapped and first and last names swapped.", + "Stop for a person when the new record lacks a name or a date of birth." + ], + "stream": "soak-seed7-55" + }, + "soak-seed7-55|variants=False|labels=False|budget=0.01": { + "adoption": { + "calibration_generated": { + "distinct_queries": 30, + "duplicates": 586, + "false_stops": 0, + "missed": 75, + "missed_by_error": { + "both_names_typo+contact_changed": 23, + "last_changed+contact_changed": 30, + "last_typo+contact_changed": 22 + } + }, + "calibration_reference": { + "distinct_queries": 30, + "duplicates": 586, + "false_stops": 12, + "missed": 0, + "missed_by_error": {} + }, + "use": "reference", + "why": "the generated rule misses 75 of 586 calibration duplicates and the reference 0: the reference stays" + }, + "assumed_labels": false, + "budget": 0.01, + "generated": { + "false_stops": 0, + "false_stops_by_exception": {}, + "false_stops_per_100_items": 0.0, + "items": 231, + "missed_by_exception": { + "limit_married_moved_dob_typo": 12, + "limit_new_name_moved_same_dob": 12, + "pl_both_typos_same_contact": 11, + "pl_married_moved": 12, + "r2_prior_name_moved_dob_typo": 6 + }, + "missed_duplicates": 53, + "new_chart_items": 45, + "true_duplicates": 184 + }, + "generation_s": 0.204, + "hand_built_record_check": { + "false_stops": 0, + "false_stops_by_exception": {}, + "false_stops_per_100_items": 0.0, + "items": 231, + "missed_by_exception": { + "limit_married_moved_dob_typo": 12, + "limit_new_name_moved_same_dob": 12 + }, + "missed_duplicates": 24, + "new_chart_items": 45, + "true_duplicates": 184 + }, + "name_variants": false, + "profile_records": 30, + "rule": { + "calibration": { + "calibration_duplicates": 586, + "distinct_queries": 870, + "false_match_rate": 0.0, + "miss_by_error": { + "both_names_typo+contact_changed": 0.767, + "both_names_typo+contact_kept": 0.0, + "contact_changed+contact_changed": 0.0, + "day_month_swapped+contact_changed": 0.0, + "day_month_swapped+contact_kept": 0.0, + "dob_digit+contact_changed": 0.0, + "dob_digit+contact_kept": 0.0, + "first_initial+contact_changed": 0.0, + "first_initial+contact_kept": 0.0, + "first_typo+contact_changed": 0.0, + "first_typo+contact_kept": 0.0, + "last_changed+contact_changed": 1.0, + "last_changed+contact_kept": 0.0, + "last_hyphenated+contact_changed": 0.0, + "last_hyphenated+contact_kept": 0.0, + "last_second_surname+contact_changed": 0.0, + "last_second_surname+contact_kept": 0.0, + "last_typo+contact_changed": 0.733, + "last_typo+contact_kept": 0.0, + "names_swapped+contact_changed": 0.0, + "names_swapped+contact_kept": 0.0 + }, + "miss_rate": 0.128 + }, + "close": [ + [ + [ + "email", + "eq" + ], + [ + "phone", + "eq" + ], + [ + "street", + "eq" + ] + ], + [ + [ + "first", + "edit1" + ], + [ + "last", + "share" + ], + [ + "dob", + "near" + ] + ], + [ + [ + "first", + "initial" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "dob", + "eq" + ], + [ + "names_swapped", + "eq" + ] + ], + [ + [ + "first", + "edit1" + ], + [ + "last", + "edit1" + ], + [ + "dob", + "swap" + ] + ] + ], + "estimated_false_stops_per_record": 0.00909, + "exact": [ + [ + "first", + "eq" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ], + "name_variants": null, + "notes": [], + "roles": { + "dob": "DOB", + "email": "email", + "first": "fname", + "last": "lname", + "phone": "phone_home", + "prior_last": "prior_lname", + "street": "street" + }, + "schema_version": "openadapt.duplicate-rule/v1" + }, + "rule_text": [ + "Stop: the record exists when first the same and last the same and dob the same.", + "Stop for a person when email the same and phone the same and street the same.", + "Stop for a person when first one letter apart and last sharing a name part and dob one part off.", + "Stop for a person when first matching by initial and last the same and dob the same.", + "Stop for a person when dob the same and first and last names swapped.", + "Stop for a person when first one letter apart and last one letter apart and dob with day and month swapped.", + "Stop for a person when the new record lacks a name or a date of birth." + ], + "stream": "soak-seed7-55" + }, + "soak-seed7-55|variants=False|labels=False|budget=0.02": { + "adoption": { + "calibration_generated": { + "distinct_queries": 30, + "duplicates": 586, + "false_stops": 0, + "missed": 50, + "missed_by_error": { + "both_names_typo+contact_changed": 23, + "last_changed+contact_changed": 27 + } + }, + "calibration_reference": { + "distinct_queries": 30, + "duplicates": 586, + "false_stops": 12, + "missed": 0, + "missed_by_error": {} + }, + "use": "reference", + "why": "the generated rule misses 50 of 586 calibration duplicates and the reference 0: the reference stays" + }, + "assumed_labels": false, + "budget": 0.02, + "generated": { + "false_stops": 0, + "false_stops_by_exception": {}, + "false_stops_per_100_items": 0.0, + "items": 231, + "missed_by_exception": { + "limit_married_moved_dob_typo": 12, + "limit_new_name_moved_same_dob": 12, + "pl_both_typos_same_contact": 11, + "pl_married_moved": 12, + "r2_prior_name_moved_dob_typo": 6 + }, + "missed_duplicates": 53, + "new_chart_items": 45, + "true_duplicates": 184 + }, + "generation_s": 0.207, + "hand_built_record_check": { + "false_stops": 0, + "false_stops_by_exception": {}, + "false_stops_per_100_items": 0.0, + "items": 231, + "missed_by_exception": { + "limit_married_moved_dob_typo": 12, + "limit_new_name_moved_same_dob": 12 + }, + "missed_duplicates": 24, + "new_chart_items": 45, + "true_duplicates": 184 + }, + "name_variants": false, + "profile_records": 30, + "rule": { + "calibration": { + "calibration_duplicates": 586, + "distinct_queries": 870, + "false_match_rate": 0.0, + "miss_by_error": { + "both_names_typo+contact_changed": 0.767, + "both_names_typo+contact_kept": 0.0, + "contact_changed+contact_changed": 0.0, + "day_month_swapped+contact_changed": 0.0, + "day_month_swapped+contact_kept": 0.0, + "dob_digit+contact_changed": 0.0, + "dob_digit+contact_kept": 0.0, + "first_initial+contact_changed": 0.0, + "first_initial+contact_kept": 0.0, + "first_typo+contact_changed": 0.0, + "first_typo+contact_kept": 0.0, + "last_changed+contact_changed": 0.967, + "last_changed+contact_kept": 0.0, + "last_hyphenated+contact_changed": 0.0, + "last_hyphenated+contact_kept": 0.0, + "last_second_surname+contact_changed": 0.0, + "last_second_surname+contact_kept": 0.0, + "last_typo+contact_changed": 0.0, + "last_typo+contact_kept": 0.0, + "names_swapped+contact_changed": 0.0, + "names_swapped+contact_kept": 0.0 + }, + "miss_rate": 0.0887 + }, + "close": [ + [ + [ + "email", + "eq" + ], + [ + "phone", + "eq" + ], + [ + "street", + "eq" + ] + ], + [ + [ + "first", + "initial" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "first", + "eq" + ], + [ + "last", + "share" + ], + [ + "dob", + "swap" + ] + ], + [ + [ + "first", + "eq" + ], + [ + "last", + "share" + ], + [ + "dob", + "near" + ] + ], + [ + [ + "first", + "edit1" + ], + [ + "last", + "share" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "dob", + "eq" + ], + [ + "names_swapped", + "eq" + ] + ], + [ + [ + "first", + "eq" + ], + [ + "last", + "edit1" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "first", + "edit1" + ], + [ + "last", + "edit1" + ], + [ + "dob", + "swap" + ] + ] + ], + "estimated_false_stops_per_record": 0.01629, + "exact": [ + [ + "first", + "eq" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ], + "name_variants": null, + "notes": [], + "roles": { + "dob": "DOB", + "email": "email", + "first": "fname", + "last": "lname", + "phone": "phone_home", + "prior_last": "prior_lname", + "street": "street" + }, + "schema_version": "openadapt.duplicate-rule/v1" + }, + "rule_text": [ + "Stop: the record exists when first the same and last the same and dob the same.", + "Stop for a person when email the same and phone the same and street the same.", + "Stop for a person when first matching by initial and last the same and dob the same.", + "Stop for a person when first the same and last sharing a name part and dob with day and month swapped.", + "Stop for a person when first the same and last sharing a name part and dob one part off.", + "Stop for a person when first one letter apart and last sharing a name part and dob the same.", + "Stop for a person when dob the same and first and last names swapped.", + "Stop for a person when first the same and last one letter apart and dob the same.", + "Stop for a person when first one letter apart and last one letter apart and dob with day and month swapped.", + "Stop for a person when the new record lacks a name or a date of birth." + ], + "stream": "soak-seed7-55" + }, + "soak-seed7-55|variants=False|labels=False|budget=0.05": { + "adoption": { + "calibration_generated": { + "distinct_queries": 30, + "duplicates": 586, + "false_stops": 2, + "missed": 24, + "missed_by_error": { + "last_changed+contact_changed": 24 + } + }, + "calibration_reference": { + "distinct_queries": 30, + "duplicates": 586, + "false_stops": 12, + "missed": 0, + "missed_by_error": {} + }, + "use": "reference", + "why": "the generated rule misses 24 of 586 calibration duplicates and the reference 0: the reference stays" + }, + "assumed_labels": false, + "budget": 0.05, + "generated": { + "false_stops": 0, + "false_stops_by_exception": {}, + "false_stops_per_100_items": 0.0, + "items": 231, + "missed_by_exception": { + "limit_married_moved_dob_typo": 12, + "limit_new_name_moved_same_dob": 12, + "pl_married_moved": 11, + "r2_prior_name_moved_dob_typo": 6 + }, + "missed_duplicates": 41, + "new_chart_items": 45, + "true_duplicates": 184 + }, + "generation_s": 0.205, + "hand_built_record_check": { + "false_stops": 0, + "false_stops_by_exception": {}, + "false_stops_per_100_items": 0.0, + "items": 231, + "missed_by_exception": { + "limit_married_moved_dob_typo": 12, + "limit_new_name_moved_same_dob": 12 + }, + "missed_duplicates": 24, + "new_chart_items": 45, + "true_duplicates": 184 + }, + "name_variants": false, + "profile_records": 30, + "rule": { + "calibration": { + "calibration_duplicates": 586, + "distinct_queries": 870, + "false_match_rate": 0.0023, + "miss_by_error": { + "both_names_typo+contact_changed": 0.0, + "both_names_typo+contact_kept": 0.0, + "contact_changed+contact_changed": 0.0, + "day_month_swapped+contact_changed": 0.0, + "day_month_swapped+contact_kept": 0.0, + "dob_digit+contact_changed": 0.0, + "dob_digit+contact_kept": 0.0, + "first_initial+contact_changed": 0.0, + "first_initial+contact_kept": 0.0, + "first_typo+contact_changed": 0.0, + "first_typo+contact_kept": 0.0, + "last_changed+contact_changed": 0.867, + "last_changed+contact_kept": 0.0, + "last_hyphenated+contact_changed": 0.0, + "last_hyphenated+contact_kept": 0.0, + "last_second_surname+contact_changed": 0.0, + "last_second_surname+contact_kept": 0.0, + "last_typo+contact_changed": 0.0, + "last_typo+contact_kept": 0.0, + "names_swapped+contact_changed": 0.0, + "names_swapped+contact_kept": 0.0 + }, + "miss_rate": 0.0444 + }, + "close": [ + [ + [ + "email", + "eq" + ], + [ + "phone", + "eq" + ], + [ + "street", + "eq" + ] + ], + [ + [ + "first", + "initial" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "first", + "eq" + ], + [ + "last", + "share" + ], + [ + "dob", + "swap" + ] + ], + [ + [ + "first", + "eq" + ], + [ + "last", + "share" + ], + [ + "dob", + "near" + ] + ], + [ + [ + "first", + "edit1" + ], + [ + "last", + "share" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "dob", + "eq" + ], + [ + "names_swapped", + "eq" + ] + ], + [ + [ + "first", + "eq" + ], + [ + "last", + "edit1" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "first", + "edit1" + ], + [ + "last", + "edit1" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "first", + "eq" + ], + [ + "dob", + "swap" + ] + ] + ], + "estimated_false_stops_per_record": 0.04392, + "exact": [ + [ + "first", + "eq" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ], + "name_variants": null, + "notes": [], + "roles": { + "dob": "DOB", + "email": "email", + "first": "fname", + "last": "lname", + "phone": "phone_home", + "prior_last": "prior_lname", + "street": "street" + }, + "schema_version": "openadapt.duplicate-rule/v1" + }, + "rule_text": [ + "Stop: the record exists when first the same and last the same and dob the same.", + "Stop for a person when email the same and phone the same and street the same.", + "Stop for a person when first matching by initial and last the same and dob the same.", + "Stop for a person when first the same and last sharing a name part and dob with day and month swapped.", + "Stop for a person when first the same and last sharing a name part and dob one part off.", + "Stop for a person when first one letter apart and last sharing a name part and dob the same.", + "Stop for a person when dob the same and first and last names swapped.", + "Stop for a person when first the same and last one letter apart and dob the same.", + "Stop for a person when first one letter apart and last one letter apart and dob the same.", + "Stop for a person when first the same and dob with day and month swapped.", + "Stop for a person when the new record lacks a name or a date of birth." + ], + "stream": "soak-seed7-55" + }, + "soak-seed7-55|variants=False|labels=False|budget=0.1": { + "adoption": { + "calibration_generated": { + "distinct_queries": 30, + "duplicates": 586, + "false_stops": 2, + "missed": 22, + "missed_by_error": { + "last_changed+contact_changed": 22 + } + }, + "calibration_reference": { + "distinct_queries": 30, + "duplicates": 586, + "false_stops": 12, + "missed": 0, + "missed_by_error": {} + }, + "use": "reference", + "why": "the generated rule misses 22 of 586 calibration duplicates and the reference 0: the reference stays" + }, + "assumed_labels": false, + "budget": 0.1, + "generated": { + "false_stops": 0, + "false_stops_by_exception": {}, + "false_stops_per_100_items": 0.0, + "items": 231, + "missed_by_exception": { + "limit_married_moved_dob_typo": 12, + "limit_new_name_moved_same_dob": 12, + "pl_married_moved": 11, + "r2_prior_name_moved_dob_typo": 6 + }, + "missed_duplicates": 41, + "new_chart_items": 45, + "true_duplicates": 184 + }, + "generation_s": 0.209, + "hand_built_record_check": { + "false_stops": 0, + "false_stops_by_exception": {}, + "false_stops_per_100_items": 0.0, + "items": 231, + "missed_by_exception": { + "limit_married_moved_dob_typo": 12, + "limit_new_name_moved_same_dob": 12 + }, + "missed_duplicates": 24, + "new_chart_items": 45, + "true_duplicates": 184 + }, + "name_variants": false, + "profile_records": 30, + "rule": { + "calibration": { + "calibration_duplicates": 586, + "distinct_queries": 870, + "false_match_rate": 0.0023, + "miss_by_error": { + "both_names_typo+contact_changed": 0.0, + "both_names_typo+contact_kept": 0.0, + "contact_changed+contact_changed": 0.0, + "day_month_swapped+contact_changed": 0.0, + "day_month_swapped+contact_kept": 0.0, + "dob_digit+contact_changed": 0.0, + "dob_digit+contact_kept": 0.0, + "first_initial+contact_changed": 0.0, + "first_initial+contact_kept": 0.0, + "first_typo+contact_changed": 0.0, + "first_typo+contact_kept": 0.0, + "last_changed+contact_changed": 0.8, + "last_changed+contact_kept": 0.0, + "last_hyphenated+contact_changed": 0.0, + "last_hyphenated+contact_kept": 0.0, + "last_second_surname+contact_changed": 0.0, + "last_second_surname+contact_kept": 0.0, + "last_typo+contact_changed": 0.0, + "last_typo+contact_kept": 0.0, + "names_swapped+contact_changed": 0.0, + "names_swapped+contact_kept": 0.0 + }, + "miss_rate": 0.041 + }, + "close": [ + [ + [ + "email", + "eq" + ], + [ + "phone", + "eq" + ], + [ + "street", + "eq" + ] + ], + [ + [ + "first", + "initial" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "first", + "eq" + ], + [ + "last", + "share" + ], + [ + "dob", + "swap" + ] + ], + [ + [ + "first", + "eq" + ], + [ + "last", + "share" + ], + [ + "dob", + "near" + ] + ], + [ + [ + "first", + "edit1" + ], + [ + "last", + "share" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "dob", + "eq" + ], + [ + "names_swapped", + "eq" + ] + ], + [ + [ + "first", + "eq" + ], + [ + "last", + "edit1" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "first", + "edit1" + ], + [ + "last", + "edit1" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "first", + "eq" + ], + [ + "dob", + "swap" + ] + ], + [ + [ + "first", + "eq" + ], + [ + "last", + "sound" + ], + [ + "dob", + "eq" + ] + ] + ], + "estimated_false_stops_per_record": 0.0536, + "exact": [ + [ + "first", + "eq" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ], + "name_variants": null, + "notes": [], + "roles": { + "dob": "DOB", + "email": "email", + "first": "fname", + "last": "lname", + "phone": "phone_home", + "prior_last": "prior_lname", + "street": "street" + }, + "schema_version": "openadapt.duplicate-rule/v1" + }, + "rule_text": [ + "Stop: the record exists when first the same and last the same and dob the same.", + "Stop for a person when email the same and phone the same and street the same.", + "Stop for a person when first matching by initial and last the same and dob the same.", + "Stop for a person when first the same and last sharing a name part and dob with day and month swapped.", + "Stop for a person when first the same and last sharing a name part and dob one part off.", + "Stop for a person when first one letter apart and last sharing a name part and dob the same.", + "Stop for a person when dob the same and first and last names swapped.", + "Stop for a person when first the same and last one letter apart and dob the same.", + "Stop for a person when first one letter apart and last one letter apart and dob the same.", + "Stop for a person when first the same and dob with day and month swapped.", + "Stop for a person when first the same and last sounding alike and dob the same.", + "Stop for a person when the new record lacks a name or a date of birth." + ], + "stream": "soak-seed7-55" + }, + "soak-seed7-55|variants=True|labels=False|budget=0.005": { + "adoption": { + "calibration_generated": { + "distinct_queries": 30, + "duplicates": 604, + "false_stops": 0, + "missed": 98, + "missed_by_error": { + "both_names_typo+contact_changed": 22, + "last_changed+contact_changed": 27, + "last_typo+contact_changed": 22, + "names_swapped+contact_changed": 27 + } + }, + "calibration_reference": { + "distinct_queries": 30, + "duplicates": 604, + "false_stops": 12, + "missed": 3, + "missed_by_error": { + "first_nickname+contact_changed": 3 + } + }, + "use": "reference", + "why": "the generated rule misses 98 of 604 calibration duplicates and the reference 3: the reference stays" + }, + "assumed_labels": false, + "budget": 0.005, + "generated": { + "false_stops": 0, + "false_stops_by_exception": {}, + "false_stops_per_100_items": 0.0, + "items": 231, + "missed_by_exception": { + "limit_married_moved_dob_typo": 12, + "limit_new_name_moved_same_dob": 12, + "pl_both_typos_same_contact": 11, + "pl_married_moved": 12, + "r2_prior_name_moved_dob_typo": 6 + }, + "missed_duplicates": 53, + "new_chart_items": 45, + "true_duplicates": 184 + }, + "generation_s": 0.239, + "hand_built_record_check": { + "false_stops": 0, + "false_stops_by_exception": {}, + "false_stops_per_100_items": 0.0, + "items": 231, + "missed_by_exception": { + "limit_married_moved_dob_typo": 12, + "limit_new_name_moved_same_dob": 12 + }, + "missed_duplicates": 24, + "new_chart_items": 45, + "true_duplicates": 184 + }, + "name_variants": true, + "profile_records": 30, + "rule": { + "calibration": { + "calibration_duplicates": 604, + "distinct_queries": 870, + "false_match_rate": 0.0, + "miss_by_error": { + "both_names_typo+contact_changed": 0.733, + "both_names_typo+contact_kept": 0.0, + "contact_changed+contact_changed": 0.0, + "day_month_swapped+contact_changed": 0.0, + "day_month_swapped+contact_kept": 0.0, + "dob_digit+contact_changed": 0.0, + "dob_digit+contact_kept": 0.0, + "first_initial+contact_changed": 0.0, + "first_initial+contact_kept": 0.0, + "first_nickname+contact_changed": 0.0, + "first_nickname+contact_kept": 0.0, + "first_typo+contact_changed": 0.0, + "first_typo+contact_kept": 0.0, + "last_changed+contact_changed": 0.967, + "last_changed+contact_kept": 0.0, + "last_hyphenated+contact_changed": 0.0, + "last_hyphenated+contact_kept": 0.0, + "last_second_surname+contact_changed": 0.0, + "last_second_surname+contact_kept": 0.0, + "last_typo+contact_changed": 0.733, + "last_typo+contact_kept": 0.0, + "names_swapped+contact_changed": 0.9, + "names_swapped+contact_kept": 0.0 + }, + "miss_rate": 0.1656 + }, + "close": [ + [ + [ + "email", + "eq" + ], + [ + "phone", + "eq" + ], + [ + "street", + "eq" + ] + ], + [ + [ + "first", + "edit1" + ], + [ + "last", + "share" + ], + [ + "dob", + "near" + ] + ], + [ + [ + "first", + "initial" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "first", + "edit1" + ], + [ + "last", + "edit1" + ], + [ + "dob", + "swap" + ] + ], + [ + [ + "first", + "variant" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "dob", + "swap" + ], + [ + "names_swapped", + "eq" + ] + ] + ], + "estimated_false_stops_per_record": 0.00464, + "exact": [ + [ + "first", + "eq" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ], + "name_variants": "a generic table of given-name variants (name_variants.json, made by a model from common names only)", + "notes": [], + "roles": { + "dob": "DOB", + "email": "email", + "first": "fname", + "last": "lname", + "phone": "phone_home", + "prior_last": "prior_lname", + "street": "street" + }, + "schema_version": "openadapt.duplicate-rule/v1" + }, + "rule_text": [ + "Stop: the record exists when first the same and last the same and dob the same.", + "Stop for a person when email the same and phone the same and street the same.", + "Stop for a person when first one letter apart and last sharing a name part and dob one part off.", + "Stop for a person when first matching by initial and last the same and dob the same.", + "Stop for a person when first one letter apart and last one letter apart and dob with day and month swapped.", + "Stop for a person when first a known variant of the other (a nickname) and last the same and dob the same.", + "Stop for a person when dob with day and month swapped and first and last names swapped.", + "Stop for a person when the new record lacks a name or a date of birth.", + "Nicknames come from a generic table of given-name variants (name_variants.json, made by a model from common names only)." + ], + "stream": "soak-seed7-55" + }, + "soak-seed7-55|variants=True|labels=False|budget=0.01": { + "adoption": { + "calibration_generated": { + "distinct_queries": 30, + "duplicates": 604, + "false_stops": 0, + "missed": 71, + "missed_by_error": { + "both_names_typo+contact_changed": 22, + "last_changed+contact_changed": 27, + "last_typo+contact_changed": 22 + } + }, + "calibration_reference": { + "distinct_queries": 30, + "duplicates": 604, + "false_stops": 12, + "missed": 3, + "missed_by_error": { + "first_nickname+contact_changed": 3 + } + }, + "use": "reference", + "why": "the generated rule misses 71 of 604 calibration duplicates and the reference 3: the reference stays" + }, + "assumed_labels": false, + "budget": 0.01, + "generated": { + "false_stops": 0, + "false_stops_by_exception": {}, + "false_stops_per_100_items": 0.0, + "items": 231, + "missed_by_exception": { + "limit_married_moved_dob_typo": 12, + "limit_new_name_moved_same_dob": 12, + "pl_both_typos_same_contact": 11, + "pl_married_moved": 12, + "r2_prior_name_moved_dob_typo": 6 + }, + "missed_duplicates": 53, + "new_chart_items": 45, + "true_duplicates": 184 + }, + "generation_s": 0.229, + "hand_built_record_check": { + "false_stops": 0, + "false_stops_by_exception": {}, + "false_stops_per_100_items": 0.0, + "items": 231, + "missed_by_exception": { + "limit_married_moved_dob_typo": 12, + "limit_new_name_moved_same_dob": 12 + }, + "missed_duplicates": 24, + "new_chart_items": 45, + "true_duplicates": 184 + }, + "name_variants": true, + "profile_records": 30, + "rule": { + "calibration": { + "calibration_duplicates": 604, + "distinct_queries": 870, + "false_match_rate": 0.0, + "miss_by_error": { + "both_names_typo+contact_changed": 0.733, + "both_names_typo+contact_kept": 0.0, + "contact_changed+contact_changed": 0.0, + "day_month_swapped+contact_changed": 0.0, + "day_month_swapped+contact_kept": 0.0, + "dob_digit+contact_changed": 0.0, + "dob_digit+contact_kept": 0.0, + "first_initial+contact_changed": 0.0, + "first_initial+contact_kept": 0.0, + "first_nickname+contact_changed": 0.0, + "first_nickname+contact_kept": 0.0, + "first_typo+contact_changed": 0.0, + "first_typo+contact_kept": 0.0, + "last_changed+contact_changed": 0.967, + "last_changed+contact_kept": 0.0, + "last_hyphenated+contact_changed": 0.0, + "last_hyphenated+contact_kept": 0.0, + "last_second_surname+contact_changed": 0.0, + "last_second_surname+contact_kept": 0.0, + "last_typo+contact_changed": 0.733, + "last_typo+contact_kept": 0.0, + "names_swapped+contact_changed": 0.0, + "names_swapped+contact_kept": 0.0 + }, + "miss_rate": 0.1209 + }, + "close": [ + [ + [ + "email", + "eq" + ], + [ + "phone", + "eq" + ], + [ + "street", + "eq" + ] + ], + [ + [ + "first", + "edit1" + ], + [ + "last", + "share" + ], + [ + "dob", + "near" + ] + ], + [ + [ + "first", + "initial" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "dob", + "eq" + ], + [ + "names_swapped", + "eq" + ] + ], + [ + [ + "first", + "edit1" + ], + [ + "last", + "edit1" + ], + [ + "dob", + "swap" + ] + ], + [ + [ + "first", + "variant" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ] + ], + "estimated_false_stops_per_record": 0.00918, + "exact": [ + [ + "first", + "eq" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ], + "name_variants": "a generic table of given-name variants (name_variants.json, made by a model from common names only)", + "notes": [], + "roles": { + "dob": "DOB", + "email": "email", + "first": "fname", + "last": "lname", + "phone": "phone_home", + "prior_last": "prior_lname", + "street": "street" + }, + "schema_version": "openadapt.duplicate-rule/v1" + }, + "rule_text": [ + "Stop: the record exists when first the same and last the same and dob the same.", + "Stop for a person when email the same and phone the same and street the same.", + "Stop for a person when first one letter apart and last sharing a name part and dob one part off.", + "Stop for a person when first matching by initial and last the same and dob the same.", + "Stop for a person when dob the same and first and last names swapped.", + "Stop for a person when first one letter apart and last one letter apart and dob with day and month swapped.", + "Stop for a person when first a known variant of the other (a nickname) and last the same and dob the same.", + "Stop for a person when the new record lacks a name or a date of birth.", + "Nicknames come from a generic table of given-name variants (name_variants.json, made by a model from common names only)." + ], + "stream": "soak-seed7-55" + }, + "soak-seed7-55|variants=True|labels=False|budget=0.02": { + "adoption": { + "calibration_generated": { + "distinct_queries": 30, + "duplicates": 604, + "false_stops": 0, + "missed": 45, + "missed_by_error": { + "both_names_typo+contact_changed": 22, + "last_changed+contact_changed": 23 + } + }, + "calibration_reference": { + "distinct_queries": 30, + "duplicates": 604, + "false_stops": 12, + "missed": 3, + "missed_by_error": { + "first_nickname+contact_changed": 3 + } + }, + "use": "reference", + "why": "the generated rule misses 45 of 604 calibration duplicates and the reference 3: the reference stays" + }, + "assumed_labels": false, + "budget": 0.02, + "generated": { + "false_stops": 0, + "false_stops_by_exception": {}, + "false_stops_per_100_items": 0.0, + "items": 231, + "missed_by_exception": { + "limit_married_moved_dob_typo": 12, + "limit_new_name_moved_same_dob": 12, + "pl_both_typos_same_contact": 11, + "pl_married_moved": 12, + "r2_prior_name_moved_dob_typo": 6 + }, + "missed_duplicates": 53, + "new_chart_items": 45, + "true_duplicates": 184 + }, + "generation_s": 0.228, + "hand_built_record_check": { + "false_stops": 0, + "false_stops_by_exception": {}, + "false_stops_per_100_items": 0.0, + "items": 231, + "missed_by_exception": { + "limit_married_moved_dob_typo": 12, + "limit_new_name_moved_same_dob": 12 + }, + "missed_duplicates": 24, + "new_chart_items": 45, + "true_duplicates": 184 + }, + "name_variants": true, + "profile_records": 30, + "rule": { + "calibration": { + "calibration_duplicates": 604, + "distinct_queries": 870, + "false_match_rate": 0.0, + "miss_by_error": { + "both_names_typo+contact_changed": 0.733, + "both_names_typo+contact_kept": 0.0, + "contact_changed+contact_changed": 0.0, + "day_month_swapped+contact_changed": 0.0, + "day_month_swapped+contact_kept": 0.0, + "dob_digit+contact_changed": 0.0, + "dob_digit+contact_kept": 0.0, + "first_initial+contact_changed": 0.0, + "first_initial+contact_kept": 0.0, + "first_nickname+contact_changed": 0.0, + "first_nickname+contact_kept": 0.0, + "first_typo+contact_changed": 0.0, + "first_typo+contact_kept": 0.0, + "last_changed+contact_changed": 0.867, + "last_changed+contact_kept": 0.0, + "last_hyphenated+contact_changed": 0.0, + "last_hyphenated+contact_kept": 0.0, + "last_second_surname+contact_changed": 0.0, + "last_second_surname+contact_kept": 0.0, + "last_typo+contact_changed": 0.0, + "last_typo+contact_kept": 0.0, + "names_swapped+contact_changed": 0.0, + "names_swapped+contact_kept": 0.0 + }, + "miss_rate": 0.0795 + }, + "close": [ + [ + [ + "email", + "eq" + ], + [ + "phone", + "eq" + ], + [ + "street", + "eq" + ] + ], + [ + [ + "first", + "initial" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "first", + "eq" + ], + [ + "last", + "share" + ], + [ + "dob", + "swap" + ] + ], + [ + [ + "first", + "eq" + ], + [ + "last", + "share" + ], + [ + "dob", + "near" + ] + ], + [ + [ + "first", + "variant" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "first", + "edit1" + ], + [ + "last", + "share" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "dob", + "eq" + ], + [ + "names_swapped", + "eq" + ] + ], + [ + [ + "first", + "eq" + ], + [ + "last", + "edit1" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "first", + "edit1" + ], + [ + "last", + "edit1" + ], + [ + "dob", + "swap" + ] + ] + ], + "estimated_false_stops_per_record": 0.01638, + "exact": [ + [ + "first", + "eq" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ], + "name_variants": "a generic table of given-name variants (name_variants.json, made by a model from common names only)", + "notes": [], + "roles": { + "dob": "DOB", + "email": "email", + "first": "fname", + "last": "lname", + "phone": "phone_home", + "prior_last": "prior_lname", + "street": "street" + }, + "schema_version": "openadapt.duplicate-rule/v1" + }, + "rule_text": [ + "Stop: the record exists when first the same and last the same and dob the same.", + "Stop for a person when email the same and phone the same and street the same.", + "Stop for a person when first matching by initial and last the same and dob the same.", + "Stop for a person when first the same and last sharing a name part and dob with day and month swapped.", + "Stop for a person when first the same and last sharing a name part and dob one part off.", + "Stop for a person when first a known variant of the other (a nickname) and last the same and dob the same.", + "Stop for a person when first one letter apart and last sharing a name part and dob the same.", + "Stop for a person when dob the same and first and last names swapped.", + "Stop for a person when first the same and last one letter apart and dob the same.", + "Stop for a person when first one letter apart and last one letter apart and dob with day and month swapped.", + "Stop for a person when the new record lacks a name or a date of birth.", + "Nicknames come from a generic table of given-name variants (name_variants.json, made by a model from common names only)." + ], + "stream": "soak-seed7-55" + }, + "soak-seed7-55|variants=True|labels=False|budget=0.05": { + "adoption": { + "calibration_generated": { + "distinct_queries": 30, + "duplicates": 604, + "false_stops": 0, + "missed": 21, + "missed_by_error": { + "last_changed+contact_changed": 21 + } + }, + "calibration_reference": { + "distinct_queries": 30, + "duplicates": 604, + "false_stops": 12, + "missed": 3, + "missed_by_error": { + "first_nickname+contact_changed": 3 + } + }, + "use": "reference", + "why": "the generated rule misses 21 of 604 calibration duplicates and the reference 3: the reference stays" + }, + "assumed_labels": false, + "budget": 0.05, + "generated": { + "false_stops": 0, + "false_stops_by_exception": {}, + "false_stops_per_100_items": 0.0, + "items": 231, + "missed_by_exception": { + "limit_married_moved_dob_typo": 12, + "limit_new_name_moved_same_dob": 12, + "pl_married_moved": 12, + "r2_prior_name_moved_dob_typo": 6 + }, + "missed_duplicates": 42, + "new_chart_items": 45, + "true_duplicates": 184 + }, + "generation_s": 0.229, + "hand_built_record_check": { + "false_stops": 0, + "false_stops_by_exception": {}, + "false_stops_per_100_items": 0.0, + "items": 231, + "missed_by_exception": { + "limit_married_moved_dob_typo": 12, + "limit_new_name_moved_same_dob": 12 + }, + "missed_duplicates": 24, + "new_chart_items": 45, + "true_duplicates": 184 + }, + "name_variants": true, + "profile_records": 30, + "rule": { + "calibration": { + "calibration_duplicates": 604, + "distinct_queries": 870, + "false_match_rate": 0.0, + "miss_by_error": { + "both_names_typo+contact_changed": 0.0, + "both_names_typo+contact_kept": 0.0, + "contact_changed+contact_changed": 0.0, + "day_month_swapped+contact_changed": 0.0, + "day_month_swapped+contact_kept": 0.0, + "dob_digit+contact_changed": 0.0, + "dob_digit+contact_kept": 0.0, + "first_initial+contact_changed": 0.0, + "first_initial+contact_kept": 0.0, + "first_nickname+contact_changed": 0.0, + "first_nickname+contact_kept": 0.0, + "first_typo+contact_changed": 0.0, + "first_typo+contact_kept": 0.0, + "last_changed+contact_changed": 0.8, + "last_changed+contact_kept": 0.0, + "last_hyphenated+contact_changed": 0.0, + "last_hyphenated+contact_kept": 0.0, + "last_second_surname+contact_changed": 0.0, + "last_second_surname+contact_kept": 0.0, + "last_typo+contact_changed": 0.0, + "last_typo+contact_kept": 0.0, + "names_swapped+contact_changed": 0.0, + "names_swapped+contact_kept": 0.0 + }, + "miss_rate": 0.0397 + }, + "close": [ + [ + [ + "email", + "eq" + ], + [ + "phone", + "eq" + ], + [ + "street", + "eq" + ] + ], + [ + [ + "first", + "initial" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "first", + "eq" + ], + [ + "last", + "share" + ], + [ + "dob", + "swap" + ] + ], + [ + [ + "first", + "eq" + ], + [ + "last", + "share" + ], + [ + "dob", + "near" + ] + ], + [ + [ + "first", + "variant" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "first", + "edit1" + ], + [ + "last", + "share" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "dob", + "eq" + ], + [ + "names_swapped", + "eq" + ] + ], + [ + [ + "first", + "eq" + ], + [ + "last", + "edit1" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "first", + "edit1" + ], + [ + "last", + "edit1" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "first", + "eq" + ], + [ + "last", + "sound" + ], + [ + "dob", + "eq" + ] + ] + ], + "estimated_false_stops_per_record": 0.04195, + "exact": [ + [ + "first", + "eq" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ], + "name_variants": "a generic table of given-name variants (name_variants.json, made by a model from common names only)", + "notes": [], + "roles": { + "dob": "DOB", + "email": "email", + "first": "fname", + "last": "lname", + "phone": "phone_home", + "prior_last": "prior_lname", + "street": "street" + }, + "schema_version": "openadapt.duplicate-rule/v1" + }, + "rule_text": [ + "Stop: the record exists when first the same and last the same and dob the same.", + "Stop for a person when email the same and phone the same and street the same.", + "Stop for a person when first matching by initial and last the same and dob the same.", + "Stop for a person when first the same and last sharing a name part and dob with day and month swapped.", + "Stop for a person when first the same and last sharing a name part and dob one part off.", + "Stop for a person when first a known variant of the other (a nickname) and last the same and dob the same.", + "Stop for a person when first one letter apart and last sharing a name part and dob the same.", + "Stop for a person when dob the same and first and last names swapped.", + "Stop for a person when first the same and last one letter apart and dob the same.", + "Stop for a person when first one letter apart and last one letter apart and dob the same.", + "Stop for a person when first the same and last sounding alike and dob the same.", + "Stop for a person when the new record lacks a name or a date of birth.", + "Nicknames come from a generic table of given-name variants (name_variants.json, made by a model from common names only)." + ], + "stream": "soak-seed7-55" + }, + "soak-seed7-55|variants=True|labels=False|budget=0.1": { + "adoption": { + "calibration_generated": { + "distinct_queries": 30, + "duplicates": 604, + "false_stops": 2, + "missed": 19, + "missed_by_error": { + "last_changed+contact_changed": 19 + } + }, + "calibration_reference": { + "distinct_queries": 30, + "duplicates": 604, + "false_stops": 12, + "missed": 3, + "missed_by_error": { + "first_nickname+contact_changed": 3 + } + }, + "use": "reference", + "why": "the generated rule misses 19 of 604 calibration duplicates and the reference 3: the reference stays" + }, + "assumed_labels": false, + "budget": 0.1, + "generated": { + "false_stops": 0, + "false_stops_by_exception": {}, + "false_stops_per_100_items": 0.0, + "items": 231, + "missed_by_exception": { + "limit_married_moved_dob_typo": 12, + "limit_new_name_moved_same_dob": 12, + "pl_married_moved": 11, + "r2_prior_name_moved_dob_typo": 6 + }, + "missed_duplicates": 41, + "new_chart_items": 45, + "true_duplicates": 184 + }, + "generation_s": 0.229, + "hand_built_record_check": { + "false_stops": 0, + "false_stops_by_exception": {}, + "false_stops_per_100_items": 0.0, + "items": 231, + "missed_by_exception": { + "limit_married_moved_dob_typo": 12, + "limit_new_name_moved_same_dob": 12 + }, + "missed_duplicates": 24, + "new_chart_items": 45, + "true_duplicates": 184 + }, + "name_variants": true, + "profile_records": 30, + "rule": { + "calibration": { + "calibration_duplicates": 604, + "distinct_queries": 870, + "false_match_rate": 0.0023, + "miss_by_error": { + "both_names_typo+contact_changed": 0.0, + "both_names_typo+contact_kept": 0.0, + "contact_changed+contact_changed": 0.0, + "day_month_swapped+contact_changed": 0.0, + "day_month_swapped+contact_kept": 0.0, + "dob_digit+contact_changed": 0.0, + "dob_digit+contact_kept": 0.0, + "first_initial+contact_changed": 0.0, + "first_initial+contact_kept": 0.0, + "first_nickname+contact_changed": 0.0, + "first_nickname+contact_kept": 0.0, + "first_typo+contact_changed": 0.0, + "first_typo+contact_kept": 0.0, + "last_changed+contact_changed": 0.733, + "last_changed+contact_kept": 0.0, + "last_hyphenated+contact_changed": 0.0, + "last_hyphenated+contact_kept": 0.0, + "last_second_surname+contact_changed": 0.0, + "last_second_surname+contact_kept": 0.0, + "last_typo+contact_changed": 0.0, + "last_typo+contact_kept": 0.0, + "names_swapped+contact_changed": 0.0, + "names_swapped+contact_kept": 0.0 + }, + "miss_rate": 0.0364 + }, + "close": [ + [ + [ + "email", + "eq" + ], + [ + "phone", + "eq" + ], + [ + "street", + "eq" + ] + ], + [ + [ + "first", + "initial" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "first", + "eq" + ], + [ + "last", + "share" + ], + [ + "dob", + "swap" + ] + ], + [ + [ + "first", + "eq" + ], + [ + "last", + "share" + ], + [ + "dob", + "near" + ] + ], + [ + [ + "first", + "variant" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "first", + "edit1" + ], + [ + "last", + "share" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "dob", + "eq" + ], + [ + "names_swapped", + "eq" + ] + ], + [ + [ + "first", + "eq" + ], + [ + "last", + "edit1" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "first", + "edit1" + ], + [ + "last", + "edit1" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "first", + "eq" + ], + [ + "last", + "sound" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "first", + "eq" + ], + [ + "dob", + "swap" + ] + ] + ], + "estimated_false_stops_per_record": 0.05369, + "exact": [ + [ + "first", + "eq" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ], + "name_variants": "a generic table of given-name variants (name_variants.json, made by a model from common names only)", + "notes": [], + "roles": { + "dob": "DOB", + "email": "email", + "first": "fname", + "last": "lname", + "phone": "phone_home", + "prior_last": "prior_lname", + "street": "street" + }, + "schema_version": "openadapt.duplicate-rule/v1" + }, + "rule_text": [ + "Stop: the record exists when first the same and last the same and dob the same.", + "Stop for a person when email the same and phone the same and street the same.", + "Stop for a person when first matching by initial and last the same and dob the same.", + "Stop for a person when first the same and last sharing a name part and dob with day and month swapped.", + "Stop for a person when first the same and last sharing a name part and dob one part off.", + "Stop for a person when first a known variant of the other (a nickname) and last the same and dob the same.", + "Stop for a person when first one letter apart and last sharing a name part and dob the same.", + "Stop for a person when dob the same and first and last names swapped.", + "Stop for a person when first the same and last one letter apart and dob the same.", + "Stop for a person when first one letter apart and last one letter apart and dob the same.", + "Stop for a person when first the same and last sounding alike and dob the same.", + "Stop for a person when first the same and dob with day and month swapped.", + "Stop for a person when the new record lacks a name or a date of birth.", + "Nicknames come from a generic table of given-name variants (name_variants.json, made by a model from common names only)." + ], + "stream": "soak-seed7-55" + }, + "soak-seed7-55|variants=True|labels=True|budget=0.005": { + "adoption": { + "calibration_generated": { + "distinct_queries": 30, + "duplicates": 604, + "false_stops": 0, + "missed": 72, + "missed_by_error": { + "both_names_typo+contact_changed": 22, + "last_changed+contact_changed": 23, + "names_swapped+contact_changed": 27 + } + }, + "calibration_reference": { + "distinct_queries": 30, + "duplicates": 604, + "false_stops": 12, + "missed": 3, + "missed_by_error": { + "first_nickname+contact_changed": 3 + } + }, + "use": "reference", + "why": "the generated rule misses 72 of 604 calibration duplicates and the reference 3: the reference stays" + }, + "assumed_labels": true, + "budget": 0.005, + "generated": { + "false_stops": 0, + "false_stops_by_exception": {}, + "false_stops_per_100_items": 0.0, + "items": 231, + "missed_by_exception": { + "limit_married_moved_dob_typo": 12, + "limit_new_name_moved_same_dob": 12, + "pl_both_typos_same_contact": 11, + "pl_married_moved": 12, + "r2_prior_name_moved_dob_typo": 6 + }, + "missed_duplicates": 53, + "new_chart_items": 45, + "true_duplicates": 184 + }, + "generation_s": 0.237, + "hand_built_record_check": { + "false_stops": 0, + "false_stops_by_exception": {}, + "false_stops_per_100_items": 0.0, + "items": 231, + "missed_by_exception": { + "limit_married_moved_dob_typo": 12, + "limit_new_name_moved_same_dob": 12 + }, + "missed_duplicates": 24, + "new_chart_items": 45, + "true_duplicates": 184 + }, + "name_variants": true, + "profile_records": 30, + "rule": { + "calibration": { + "calibration_duplicates": 604, + "distinct_queries": 858, + "false_match_rate": 0.0, + "miss_by_error": { + "both_names_typo+contact_changed": 0.733, + "both_names_typo+contact_kept": 0.0, + "contact_changed+contact_changed": 0.0, + "day_month_swapped+contact_changed": 0.0, + "day_month_swapped+contact_kept": 0.0, + "dob_digit+contact_changed": 0.0, + "dob_digit+contact_kept": 0.0, + "first_initial+contact_changed": 0.0, + "first_initial+contact_kept": 0.0, + "first_nickname+contact_changed": 0.0, + "first_nickname+contact_kept": 0.0, + "first_typo+contact_changed": 0.0, + "first_typo+contact_kept": 0.0, + "last_changed+contact_changed": 0.867, + "last_changed+contact_kept": 0.0, + "last_hyphenated+contact_changed": 0.0, + "last_hyphenated+contact_kept": 0.0, + "last_second_surname+contact_changed": 0.0, + "last_second_surname+contact_kept": 0.0, + "last_typo+contact_changed": 0.0, + "last_typo+contact_kept": 0.0, + "names_swapped+contact_changed": 0.9, + "names_swapped+contact_kept": 0.0 + }, + "miss_rate": 0.1242 + }, + "close": [ + [ + [ + "email", + "eq" + ], + [ + "phone", + "eq" + ], + [ + "street", + "eq" + ] + ], + [ + [ + "first", + "edit1" + ], + [ + "last", + "share" + ], + [ + "dob", + "near" + ] + ], + [ + [ + "first", + "initial" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "first", + "eq" + ], + [ + "last", + "edit1" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "first", + "edit1" + ], + [ + "last", + "edit1" + ], + [ + "dob", + "swap" + ] + ], + [ + [ + "first", + "variant" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "dob", + "swap" + ], + [ + "names_swapped", + "eq" + ] + ] + ], + "estimated_false_stops_per_record": 0.00436, + "exact": [ + [ + "first", + "eq" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ], + "name_variants": "a generic table of given-name variants (name_variants.json, made by a model from common names only)", + "notes": [ + "a person labelled 6 pair(s) of existing records as one person; they no longer count as two" + ], + "roles": { + "dob": "DOB", + "email": "email", + "first": "fname", + "last": "lname", + "phone": "phone_home", + "prior_last": "prior_lname", + "street": "street" + }, + "schema_version": "openadapt.duplicate-rule/v1" + }, + "rule_text": [ + "Stop: the record exists when first the same and last the same and dob the same.", + "Stop for a person when email the same and phone the same and street the same.", + "Stop for a person when first one letter apart and last sharing a name part and dob one part off.", + "Stop for a person when first matching by initial and last the same and dob the same.", + "Stop for a person when first the same and last one letter apart and dob the same.", + "Stop for a person when first one letter apart and last one letter apart and dob with day and month swapped.", + "Stop for a person when first a known variant of the other (a nickname) and last the same and dob the same.", + "Stop for a person when dob with day and month swapped and first and last names swapped.", + "Stop for a person when the new record lacks a name or a date of birth.", + "Nicknames come from a generic table of given-name variants (name_variants.json, made by a model from common names only)." + ], + "stream": "soak-seed7-55" + }, + "soak-seed7-55|variants=True|labels=True|budget=0.01": { + "adoption": { + "calibration_generated": { + "distinct_queries": 30, + "duplicates": 604, + "false_stops": 0, + "missed": 45, + "missed_by_error": { + "both_names_typo+contact_changed": 22, + "last_changed+contact_changed": 23 + } + }, + "calibration_reference": { + "distinct_queries": 30, + "duplicates": 604, + "false_stops": 12, + "missed": 3, + "missed_by_error": { + "first_nickname+contact_changed": 3 + } + }, + "use": "reference", + "why": "the generated rule misses 45 of 604 calibration duplicates and the reference 3: the reference stays" + }, + "assumed_labels": true, + "budget": 0.01, + "generated": { + "false_stops": 0, + "false_stops_by_exception": {}, + "false_stops_per_100_items": 0.0, + "items": 231, + "missed_by_exception": { + "limit_married_moved_dob_typo": 12, + "limit_new_name_moved_same_dob": 12, + "pl_both_typos_same_contact": 11, + "pl_married_moved": 12, + "r2_prior_name_moved_dob_typo": 6 + }, + "missed_duplicates": 53, + "new_chart_items": 45, + "true_duplicates": 184 + }, + "generation_s": 0.252, + "hand_built_record_check": { + "false_stops": 0, + "false_stops_by_exception": {}, + "false_stops_per_100_items": 0.0, + "items": 231, + "missed_by_exception": { + "limit_married_moved_dob_typo": 12, + "limit_new_name_moved_same_dob": 12 + }, + "missed_duplicates": 24, + "new_chart_items": 45, + "true_duplicates": 184 + }, + "name_variants": true, + "profile_records": 30, + "rule": { + "calibration": { + "calibration_duplicates": 604, + "distinct_queries": 858, + "false_match_rate": 0.0, + "miss_by_error": { + "both_names_typo+contact_changed": 0.733, + "both_names_typo+contact_kept": 0.0, + "contact_changed+contact_changed": 0.0, + "day_month_swapped+contact_changed": 0.0, + "day_month_swapped+contact_kept": 0.0, + "dob_digit+contact_changed": 0.0, + "dob_digit+contact_kept": 0.0, + "first_initial+contact_changed": 0.0, + "first_initial+contact_kept": 0.0, + "first_nickname+contact_changed": 0.0, + "first_nickname+contact_kept": 0.0, + "first_typo+contact_changed": 0.0, + "first_typo+contact_kept": 0.0, + "last_changed+contact_changed": 0.867, + "last_changed+contact_kept": 0.0, + "last_hyphenated+contact_changed": 0.0, + "last_hyphenated+contact_kept": 0.0, + "last_second_surname+contact_changed": 0.0, + "last_second_surname+contact_kept": 0.0, + "last_typo+contact_changed": 0.0, + "last_typo+contact_kept": 0.0, + "names_swapped+contact_changed": 0.0, + "names_swapped+contact_kept": 0.0 + }, + "miss_rate": 0.0795 + }, + "close": [ + [ + [ + "email", + "eq" + ], + [ + "phone", + "eq" + ], + [ + "street", + "eq" + ] + ], + [ + [ + "first", + "edit1" + ], + [ + "last", + "share" + ], + [ + "dob", + "near" + ] + ], + [ + [ + "first", + "initial" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "dob", + "eq" + ], + [ + "names_swapped", + "eq" + ] + ], + [ + [ + "first", + "eq" + ], + [ + "last", + "edit1" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "first", + "edit1" + ], + [ + "last", + "edit1" + ], + [ + "dob", + "swap" + ] + ], + [ + [ + "first", + "variant" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ] + ], + "estimated_false_stops_per_record": 0.00801, + "exact": [ + [ + "first", + "eq" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ], + "name_variants": "a generic table of given-name variants (name_variants.json, made by a model from common names only)", + "notes": [ + "a person labelled 6 pair(s) of existing records as one person; they no longer count as two" + ], + "roles": { + "dob": "DOB", + "email": "email", + "first": "fname", + "last": "lname", + "phone": "phone_home", + "prior_last": "prior_lname", + "street": "street" + }, + "schema_version": "openadapt.duplicate-rule/v1" + }, + "rule_text": [ + "Stop: the record exists when first the same and last the same and dob the same.", + "Stop for a person when email the same and phone the same and street the same.", + "Stop for a person when first one letter apart and last sharing a name part and dob one part off.", + "Stop for a person when first matching by initial and last the same and dob the same.", + "Stop for a person when dob the same and first and last names swapped.", + "Stop for a person when first the same and last one letter apart and dob the same.", + "Stop for a person when first one letter apart and last one letter apart and dob with day and month swapped.", + "Stop for a person when first a known variant of the other (a nickname) and last the same and dob the same.", + "Stop for a person when the new record lacks a name or a date of birth.", + "Nicknames come from a generic table of given-name variants (name_variants.json, made by a model from common names only)." + ], + "stream": "soak-seed7-55" + }, + "soak-seed7-55|variants=True|labels=True|budget=0.02": { + "adoption": { + "calibration_generated": { + "distinct_queries": 30, + "duplicates": 604, + "false_stops": 0, + "missed": 23, + "missed_by_error": { + "last_changed+contact_changed": 23 + } + }, + "calibration_reference": { + "distinct_queries": 30, + "duplicates": 604, + "false_stops": 12, + "missed": 3, + "missed_by_error": { + "first_nickname+contact_changed": 3 + } + }, + "use": "reference", + "why": "the generated rule misses 23 of 604 calibration duplicates and the reference 3: the reference stays" + }, + "assumed_labels": true, + "budget": 0.02, + "generated": { + "false_stops": 0, + "false_stops_by_exception": {}, + "false_stops_per_100_items": 0.0, + "items": 231, + "missed_by_exception": { + "limit_married_moved_dob_typo": 12, + "limit_new_name_moved_same_dob": 12, + "pl_married_moved": 12, + "r2_prior_name_moved_dob_typo": 6 + }, + "missed_duplicates": 42, + "new_chart_items": 45, + "true_duplicates": 184 + }, + "generation_s": 0.236, + "hand_built_record_check": { + "false_stops": 0, + "false_stops_by_exception": {}, + "false_stops_per_100_items": 0.0, + "items": 231, + "missed_by_exception": { + "limit_married_moved_dob_typo": 12, + "limit_new_name_moved_same_dob": 12 + }, + "missed_duplicates": 24, + "new_chart_items": 45, + "true_duplicates": 184 + }, + "name_variants": true, + "profile_records": 30, + "rule": { + "calibration": { + "calibration_duplicates": 604, + "distinct_queries": 858, + "false_match_rate": 0.0, + "miss_by_error": { + "both_names_typo+contact_changed": 0.0, + "both_names_typo+contact_kept": 0.0, + "contact_changed+contact_changed": 0.0, + "day_month_swapped+contact_changed": 0.0, + "day_month_swapped+contact_kept": 0.0, + "dob_digit+contact_changed": 0.0, + "dob_digit+contact_kept": 0.0, + "first_initial+contact_changed": 0.0, + "first_initial+contact_kept": 0.0, + "first_nickname+contact_changed": 0.0, + "first_nickname+contact_kept": 0.0, + "first_typo+contact_changed": 0.0, + "first_typo+contact_kept": 0.0, + "last_changed+contact_changed": 0.867, + "last_changed+contact_kept": 0.0, + "last_hyphenated+contact_changed": 0.0, + "last_hyphenated+contact_kept": 0.0, + "last_second_surname+contact_changed": 0.0, + "last_second_surname+contact_kept": 0.0, + "last_typo+contact_changed": 0.0, + "last_typo+contact_kept": 0.0, + "names_swapped+contact_changed": 0.0, + "names_swapped+contact_kept": 0.0 + }, + "miss_rate": 0.043 + }, + "close": [ + [ + [ + "email", + "eq" + ], + [ + "phone", + "eq" + ], + [ + "street", + "eq" + ] + ], + [ + [ + "first", + "initial" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "first", + "eq" + ], + [ + "last", + "share" + ], + [ + "dob", + "swap" + ] + ], + [ + [ + "first", + "eq" + ], + [ + "last", + "share" + ], + [ + "dob", + "near" + ] + ], + [ + [ + "first", + "variant" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "first", + "edit1" + ], + [ + "last", + "share" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "first", + "eq" + ], + [ + "last", + "edit1" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "dob", + "eq" + ], + [ + "names_swapped", + "eq" + ] + ], + [ + [ + "first", + "edit1" + ], + [ + "last", + "edit1" + ], + [ + "dob", + "eq" + ] + ] + ], + "estimated_false_stops_per_record": 0.01781, + "exact": [ + [ + "first", + "eq" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ], + "name_variants": "a generic table of given-name variants (name_variants.json, made by a model from common names only)", + "notes": [ + "a person labelled 6 pair(s) of existing records as one person; they no longer count as two" + ], + "roles": { + "dob": "DOB", + "email": "email", + "first": "fname", + "last": "lname", + "phone": "phone_home", + "prior_last": "prior_lname", + "street": "street" + }, + "schema_version": "openadapt.duplicate-rule/v1" + }, + "rule_text": [ + "Stop: the record exists when first the same and last the same and dob the same.", + "Stop for a person when email the same and phone the same and street the same.", + "Stop for a person when first matching by initial and last the same and dob the same.", + "Stop for a person when first the same and last sharing a name part and dob with day and month swapped.", + "Stop for a person when first the same and last sharing a name part and dob one part off.", + "Stop for a person when first a known variant of the other (a nickname) and last the same and dob the same.", + "Stop for a person when first one letter apart and last sharing a name part and dob the same.", + "Stop for a person when first the same and last one letter apart and dob the same.", + "Stop for a person when dob the same and first and last names swapped.", + "Stop for a person when first one letter apart and last one letter apart and dob the same.", + "Stop for a person when the new record lacks a name or a date of birth.", + "Nicknames come from a generic table of given-name variants (name_variants.json, made by a model from common names only)." + ], + "stream": "soak-seed7-55" + }, + "soak-seed7-55|variants=True|labels=True|budget=0.05": { + "adoption": { + "calibration_generated": { + "distinct_queries": 30, + "duplicates": 604, + "false_stops": 12, + "missed": 0, + "missed_by_error": {} + }, + "calibration_reference": { + "distinct_queries": 30, + "duplicates": 604, + "false_stops": 12, + "missed": 3, + "missed_by_error": { + "first_nickname+contact_changed": 3 + } + }, + "use": "generated", + "why": "it misses 0 of 604 calibration duplicates, the reference 3" + }, + "assumed_labels": true, + "budget": 0.05, + "generated": { + "false_stops": 0, + "false_stops_by_exception": {}, + "false_stops_per_100_items": 0.0, + "items": 231, + "missed_by_exception": { + "limit_married_moved_dob_typo": 12, + "limit_new_name_moved_same_dob": 12, + "r2_prior_name_moved_dob_typo": 6 + }, + "missed_duplicates": 30, + "new_chart_items": 45, + "true_duplicates": 184 + }, + "generation_s": 0.24, + "hand_built_record_check": { + "false_stops": 0, + "false_stops_by_exception": {}, + "false_stops_per_100_items": 0.0, + "items": 231, + "missed_by_exception": { + "limit_married_moved_dob_typo": 12, + "limit_new_name_moved_same_dob": 12 + }, + "missed_duplicates": 24, + "new_chart_items": 45, + "true_duplicates": 184 + }, + "name_variants": true, + "profile_records": 30, + "rule": { + "calibration": { + "calibration_duplicates": 604, + "distinct_queries": 858, + "false_match_rate": 0.0, + "miss_by_error": { + "both_names_typo+contact_changed": 0.0, + "both_names_typo+contact_kept": 0.0, + "contact_changed+contact_changed": 0.0, + "day_month_swapped+contact_changed": 0.0, + "day_month_swapped+contact_kept": 0.0, + "dob_digit+contact_changed": 0.0, + "dob_digit+contact_kept": 0.0, + "first_initial+contact_changed": 0.0, + "first_initial+contact_kept": 0.0, + "first_nickname+contact_changed": 0.0, + "first_nickname+contact_kept": 0.0, + "first_typo+contact_changed": 0.0, + "first_typo+contact_kept": 0.0, + "last_changed+contact_changed": 0.0, + "last_changed+contact_kept": 0.0, + "last_hyphenated+contact_changed": 0.0, + "last_hyphenated+contact_kept": 0.0, + "last_second_surname+contact_changed": 0.0, + "last_second_surname+contact_kept": 0.0, + "last_typo+contact_changed": 0.0, + "last_typo+contact_kept": 0.0, + "names_swapped+contact_changed": 0.0, + "names_swapped+contact_kept": 0.0 + }, + "miss_rate": 0.0 + }, + "close": [ + [ + [ + "email", + "eq" + ], + [ + "phone", + "eq" + ], + [ + "street", + "eq" + ] + ], + [ + [ + "first", + "initial" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "first", + "eq" + ], + [ + "last", + "share" + ], + [ + "dob", + "swap" + ] + ], + [ + [ + "first", + "eq" + ], + [ + "last", + "share" + ], + [ + "dob", + "near" + ] + ], + [ + [ + "first", + "variant" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "first", + "edit1" + ], + [ + "last", + "share" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "first", + "eq" + ], + [ + "last", + "edit1" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "dob", + "eq" + ], + [ + "names_swapped", + "eq" + ] + ], + [ + [ + "first", + "edit1" + ], + [ + "last", + "edit1" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "first", + "eq" + ], + [ + "dob", + "eq" + ] + ] + ], + "estimated_false_stops_per_record": 0.03859, + "exact": [ + [ + "first", + "eq" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ], + "name_variants": "a generic table of given-name variants (name_variants.json, made by a model from common names only)", + "notes": [ + "a person labelled 6 pair(s) of existing records as one person; they no longer count as two" + ], + "roles": { + "dob": "DOB", + "email": "email", + "first": "fname", + "last": "lname", + "phone": "phone_home", + "prior_last": "prior_lname", + "street": "street" + }, + "schema_version": "openadapt.duplicate-rule/v1" + }, + "rule_text": [ + "Stop: the record exists when first the same and last the same and dob the same.", + "Stop for a person when email the same and phone the same and street the same.", + "Stop for a person when first matching by initial and last the same and dob the same.", + "Stop for a person when first the same and last sharing a name part and dob with day and month swapped.", + "Stop for a person when first the same and last sharing a name part and dob one part off.", + "Stop for a person when first a known variant of the other (a nickname) and last the same and dob the same.", + "Stop for a person when first one letter apart and last sharing a name part and dob the same.", + "Stop for a person when first the same and last one letter apart and dob the same.", + "Stop for a person when dob the same and first and last names swapped.", + "Stop for a person when first one letter apart and last one letter apart and dob the same.", + "Stop for a person when first the same and dob the same.", + "Stop for a person when the new record lacks a name or a date of birth.", + "Nicknames come from a generic table of given-name variants (name_variants.json, made by a model from common names only)." + ], + "stream": "soak-seed7-55" + }, + "soak-seed7-55|variants=True|labels=True|budget=0.1": { + "adoption": { + "calibration_generated": { + "distinct_queries": 30, + "duplicates": 604, + "false_stops": 12, + "missed": 0, + "missed_by_error": {} + }, + "calibration_reference": { + "distinct_queries": 30, + "duplicates": 604, + "false_stops": 12, + "missed": 3, + "missed_by_error": { + "first_nickname+contact_changed": 3 + } + }, + "use": "generated", + "why": "it misses 0 of 604 calibration duplicates, the reference 3" + }, + "assumed_labels": true, + "budget": 0.1, + "generated": { + "false_stops": 0, + "false_stops_by_exception": {}, + "false_stops_per_100_items": 0.0, + "items": 231, + "missed_by_exception": { + "limit_married_moved_dob_typo": 12, + "limit_new_name_moved_same_dob": 12, + "r2_prior_name_moved_dob_typo": 6 + }, + "missed_duplicates": 30, + "new_chart_items": 45, + "true_duplicates": 184 + }, + "generation_s": 0.242, + "hand_built_record_check": { + "false_stops": 0, + "false_stops_by_exception": {}, + "false_stops_per_100_items": 0.0, + "items": 231, + "missed_by_exception": { + "limit_married_moved_dob_typo": 12, + "limit_new_name_moved_same_dob": 12 + }, + "missed_duplicates": 24, + "new_chart_items": 45, + "true_duplicates": 184 + }, + "name_variants": true, + "profile_records": 30, + "rule": { + "calibration": { + "calibration_duplicates": 604, + "distinct_queries": 858, + "false_match_rate": 0.0, + "miss_by_error": { + "both_names_typo+contact_changed": 0.0, + "both_names_typo+contact_kept": 0.0, + "contact_changed+contact_changed": 0.0, + "day_month_swapped+contact_changed": 0.0, + "day_month_swapped+contact_kept": 0.0, + "dob_digit+contact_changed": 0.0, + "dob_digit+contact_kept": 0.0, + "first_initial+contact_changed": 0.0, + "first_initial+contact_kept": 0.0, + "first_nickname+contact_changed": 0.0, + "first_nickname+contact_kept": 0.0, + "first_typo+contact_changed": 0.0, + "first_typo+contact_kept": 0.0, + "last_changed+contact_changed": 0.0, + "last_changed+contact_kept": 0.0, + "last_hyphenated+contact_changed": 0.0, + "last_hyphenated+contact_kept": 0.0, + "last_second_surname+contact_changed": 0.0, + "last_second_surname+contact_kept": 0.0, + "last_typo+contact_changed": 0.0, + "last_typo+contact_kept": 0.0, + "names_swapped+contact_changed": 0.0, + "names_swapped+contact_kept": 0.0 + }, + "miss_rate": 0.0 + }, + "close": [ + [ + [ + "email", + "eq" + ], + [ + "phone", + "eq" + ], + [ + "street", + "eq" + ] + ], + [ + [ + "first", + "initial" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "first", + "eq" + ], + [ + "last", + "share" + ], + [ + "dob", + "swap" + ] + ], + [ + [ + "first", + "eq" + ], + [ + "last", + "share" + ], + [ + "dob", + "near" + ] + ], + [ + [ + "first", + "variant" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "first", + "edit1" + ], + [ + "last", + "share" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "first", + "eq" + ], + [ + "last", + "edit1" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "dob", + "eq" + ], + [ + "names_swapped", + "eq" + ] + ], + [ + [ + "first", + "edit1" + ], + [ + "last", + "edit1" + ], + [ + "dob", + "eq" + ] + ], + [ + [ + "first", + "eq" + ], + [ + "dob", + "eq" + ] + ] + ], + "estimated_false_stops_per_record": 0.03859, + "exact": [ + [ + "first", + "eq" + ], + [ + "last", + "eq" + ], + [ + "dob", + "eq" + ] + ], + "name_variants": "a generic table of given-name variants (name_variants.json, made by a model from common names only)", + "notes": [ + "a person labelled 6 pair(s) of existing records as one person; they no longer count as two" + ], + "roles": { + "dob": "DOB", + "email": "email", + "first": "fname", + "last": "lname", + "phone": "phone_home", + "prior_last": "prior_lname", + "street": "street" + }, + "schema_version": "openadapt.duplicate-rule/v1" + }, + "rule_text": [ + "Stop: the record exists when first the same and last the same and dob the same.", + "Stop for a person when email the same and phone the same and street the same.", + "Stop for a person when first matching by initial and last the same and dob the same.", + "Stop for a person when first the same and last sharing a name part and dob with day and month swapped.", + "Stop for a person when first the same and last sharing a name part and dob one part off.", + "Stop for a person when first a known variant of the other (a nickname) and last the same and dob the same.", + "Stop for a person when first one letter apart and last sharing a name part and dob the same.", + "Stop for a person when first the same and last one letter apart and dob the same.", + "Stop for a person when dob the same and first and last names swapped.", + "Stop for a person when first one letter apart and last one letter apart and dob the same.", + "Stop for a person when first the same and dob the same.", + "Stop for a person when the new record lacks a name or a date of birth.", + "Nicknames come from a generic table of given-name variants (name_variants.json, made by a model from common names only)." + ], + "stream": "soak-seed7-55" + } +} diff --git a/packages/flow/benchmark/generated_parts/results/identity_mapping.json b/packages/flow/benchmark/generated_parts/results/identity_mapping.json new file mode 100644 index 000000000..02751aa0d --- /dev/null +++ b/packages/flow/benchmark/generated_parts/results/identity_mapping.json @@ -0,0 +1,359 @@ +{ + "fhir": { + "adoption": { + "use": "generated", + "why": "on the perturbation suite it misses 5 and stops 12 distinct records, the reference 5 and 12" + }, + "demos": 3, + "demos_agree": { + "DOB": 3, + "email": 3, + "fname": 3, + "lname": 3, + "phone_home": 3, + "street": 3 + }, + "mapping": { + "dob_format": null, + "notes": [], + "record": { + "dob": "birthDate", + "email": "telecom[system=email].value", + "fname": "name[use=official].given", + "lname": "name[use=official].family", + "mrn": "identifier.value", + "phone": "telecom[system=phone].value", + "prior_lname": "name[use=old].family", + "record_id": "id", + "street": "address.line" + }, + "schema_version": "openadapt.identity-mapping/v1", + "template": "patient-identity/1", + "want": { + "DOB": "DOB", + "email": "email", + "fname": "fname", + "lname": "lname", + "phone_home": "phone_home", + "street": "street" + } + }, + "stream_generated": { + "dups": 223, + "false_stops": 2, + "missed": 24, + "new": 440 + }, + "stream_reference": { + "dups": 223, + "false_stops": 2, + "missed": 24, + "new": 440 + }, + "suite_generated": { + "distinct": 30, + "duplicates": 580, + "false_stops": 12, + "missed": 5, + "missed_by_kind": { + "first_nickname+contact_changed": 5 + }, + "missed_cases": [ + 35, + 56, + 296, + 319, + 363 + ] + }, + "suite_reference": { + "distinct": 30, + "duplicates": 580, + "false_stops": 12, + "missed": 5, + "missed_by_kind": { + "first_nickname+contact_changed": 5 + }, + "missed_cases": [ + 35, + 56, + 296, + 319, + 363 + ] + } + }, + "fhir_first_item": { + "adoption": { + "use": "reference", + "why": "on the perturbation suite the generated mapping misses 30 and stops 12 distinct records, the reference 5 and 12: the reference stays" + }, + "demos": 3, + "demos_agree": { + "DOB": 3, + "fname": 3, + "lname": 3, + "phone_home": 3, + "street": 3 + }, + "mapping": { + "dob_format": null, + "notes": [], + "record": { + "dob": "birthDate", + "fname": "name.given", + "lname": "name.family", + "mrn": "identifier.value", + "phone": "telecom.value", + "record_id": "id", + "street": "address.line" + }, + "schema_version": "openadapt.identity-mapping/v1", + "template": "patient-identity/1", + "want": { + "DOB": "DOB", + "fname": "fname", + "lname": "lname", + "phone_home": "phone_home", + "street": "street" + } + }, + "stream_generated": { + "dups": 223, + "false_stops": 2, + "missed": 30, + "new": 440 + }, + "stream_reference": { + "dups": 223, + "false_stops": 2, + "missed": 24, + "new": 440 + }, + "suite_generated": { + "distinct": 30, + "duplicates": 580, + "false_stops": 12, + "missed": 30, + "missed_by_kind": { + "first_nickname+contact_changed": 5, + "only_email_kept+contact_kept": 25 + }, + "missed_cases": [ + 19, + 35, + 40, + 56, + 61, + 82, + 141, + 160, + 179, + 198, + 219, + 238, + 259, + 280, + 296, + 301, + 319, + 324, + 347, + 363, + 368, + 387, + 406, + 425, + 444, + 484, + 506, + 528, + 548, + 608 + ] + }, + "suite_reference": { + "distinct": 30, + "duplicates": 580, + "false_stops": 12, + "missed": 5, + "missed_by_kind": { + "first_nickname+contact_changed": 5 + }, + "missed_cases": [ + 35, + 56, + 296, + 319, + 363 + ] + } + }, + "openemr_rest": { + "adoption": { + "use": "generated", + "why": "on the perturbation suite it misses 5 and stops 12 distinct records, the reference 5 and 12" + }, + "demos": 3, + "demos_agree": { + "DOB": 3, + "email": 3, + "fname": 3, + "lname": 3, + "phone_home": 3, + "street": 3 + }, + "mapping": { + "dob_format": null, + "notes": [], + "record": { + "dob": "DOB", + "email": "email", + "fname": "fname", + "lname": "lname", + "mrn": "pubpid", + "phone": "phone_home", + "prior_lname": "birth_lname", + "record_id": "id", + "street": "street" + }, + "schema_version": "openadapt.identity-mapping/v1", + "template": "patient-identity/1", + "want": { + "DOB": "DOB", + "email": "email", + "fname": "fname", + "lname": "lname", + "phone_home": "phone_home", + "street": "street" + } + }, + "stream_generated": { + "dups": 223, + "false_stops": 2, + "missed": 24, + "new": 440 + }, + "stream_reference": { + "dups": 223, + "false_stops": 2, + "missed": 24, + "new": 440 + }, + "suite_generated": { + "distinct": 30, + "duplicates": 580, + "false_stops": 12, + "missed": 5, + "missed_by_kind": { + "first_nickname+contact_changed": 5 + }, + "missed_cases": [ + 35, + 56, + 296, + 319, + 363 + ] + }, + "suite_reference": { + "distinct": 30, + "duplicates": 580, + "false_stops": 12, + "missed": 5, + "missed_by_kind": { + "first_nickname+contact_changed": 5 + }, + "missed_cases": [ + 35, + 56, + 296, + 319, + 363 + ] + } + }, + "rows": { + "adoption": { + "use": "generated", + "why": "on the perturbation suite it misses 5 and stops 12 distinct records, the reference 5 and 12" + }, + "demos": 3, + "demos_agree": { + "DOB": 3, + "email": 3, + "fname": 3, + "lname": 3, + "phone_home": 3, + "street": 3 + }, + "mapping": { + "dob_format": null, + "notes": [], + "record": { + "dob": "DOB", + "email": "email", + "fname": "fname", + "lname": "lname", + "phone": "phone_home", + "prior_lname": "prior_lname", + "record_id": "id", + "street": "street" + }, + "schema_version": "openadapt.identity-mapping/v1", + "template": "patient-identity/1", + "want": { + "DOB": "DOB", + "email": "email", + "fname": "fname", + "lname": "lname", + "phone_home": "phone_home", + "street": "street" + } + }, + "stream_generated": { + "dups": 223, + "false_stops": 2, + "missed": 24, + "new": 440 + }, + "stream_reference": { + "dups": 223, + "false_stops": 2, + "missed": 24, + "new": 440 + }, + "suite_generated": { + "distinct": 30, + "duplicates": 580, + "false_stops": 12, + "missed": 5, + "missed_by_kind": { + "first_nickname+contact_changed": 5 + }, + "missed_cases": [ + 35, + 56, + 296, + 319, + 363 + ] + }, + "suite_reference": { + "distinct": 30, + "duplicates": 580, + "false_stops": 12, + "missed": 5, + "missed_by_kind": { + "first_nickname+contact_changed": 5 + }, + "missed_cases": [ + 35, + 56, + 296, + 319, + 363 + ] + } + } +} diff --git a/packages/flow/docs/BACKEND_PARITY.md b/packages/flow/docs/BACKEND_PARITY.md index b3b06caea..7f699432e 100644 --- a/packages/flow/docs/BACKEND_PARITY.md +++ b/packages/flow/docs/BACKEND_PARITY.md @@ -159,6 +159,8 @@ and `--log-url`, the transparency log) are not app URLs. | `openadapt-flow validate-hosted` | `--target-url` | The surface comes from the signed run report. A non-web report refuses a target URL ("Target URL is web-only and must be omitted"). | | `openadapt-agent authoring connect` | `--url` | `--url` opens a fresh browser. Without it the command pins the local app window (one macOS or Linux window). Windows, RDP and Citrix are coach-only on the mailbox: the person drives and Capture records (`COACH_ONLY_BACKENDS` in `agent/mailbox.py`). | | `openadapt-agent seal keys pin` | `--url` | The seal key server's address, not an app. | +| `openadapt-flow generate-parts channels` | `--base-url` | The system of record's API, read once to list candidate read channels; no surface is driven. | +| `openadapt-flow generate-parts channels` | `--fhir-base-url` | The system of record's FHIR API, read once to list candidate read channels; no surface is driven. | ## Open gaps diff --git a/packages/flow/docs/GENERATED_PARTS.md b/packages/flow/docs/GENERATED_PARTS.md new file mode 100644 index 000000000..93c9a2941 --- /dev/null +++ b/packages/flow/docs/GENERATED_PARTS.md @@ -0,0 +1,92 @@ +# Generated application parts + +`openadapt-flow generate-parts` builds the parts of a +workflow that are specific to one application at setup time, and admits each +one only after a check that does not depend on the generator. Healthy runs +still make no model calls: everything here happens once, before a person +approves the workflow. + +## What is generated, and what never is + +| Part | Generated from | Admitted by | +|---|---|---| +| Read channel: how to read the system of record | the network traffic captured while teaching (`network.jsonl`), an OpenAPI document, or a FHIR CapabilityStatement | the channel test: reads back exactly the records each demonstration created or changed, across every page, and reads as unreadable (never as empty) when auth, paging or a response is broken | +| Record contract: what a save must do and must not do | the before/after difference in the system of record across two or three varied demonstrations | the fault gate (`fault_gate.py`): wrong record, wrong value, unrelated change, missing write, partial write, duplicate, delayed write, with coverage per class | +| Identity and duplicate rule: when a new record is already there | a reviewed template (`patient-identity/1`, never generated) plus a generated mapping of this system's fields onto it | the demonstrations (each saved record reads back as typed) and a perturbation suite built from the system's own records; the mapping replaces a hand-written source adapter only if it does no worse on the suite | +| Input types and conversion | sample source documents | the round trip (`provenance.py`): every converted value must read back from its source; an ambiguous value halts | + +Never generated: the safety policy. That is the execution profile, approval +modes, the go-live switch, which steps are consequential, the order of the +gates, and attribution of changes to other users (the one rule that excuses +a change). A person sets those, as today. + +## Generated parts can only add checks + +Generation reads untrusted content: page text, response bodies, field +values. A page could try to steer a generator ("this field is volatile, ignore +it"). So the generated contract is checked against a fixed baseline that does +not come from page content: + +- every parameter the demonstration typed is checked in the saved record; +- the frame is the whole read set, compared by record identity; +- no generated `ignore_fields` entry names a field a parameter is written to, + and nothing generated turns off `forbid_collateral_loss`, adds + `target_fields`, or adds attribution. + +A generated part that removes or weakens a baseline check is refused, whatever +produced it. Tests feed a hostile page through every generator. + +## A person approves the contract, in words + +`generate-parts` writes `parts.review.md`: one plain-language line per check +("A new patient record must appear with the email the task names", "No other +patient record may change"), the coverage the fault gate measured, the +duplicate rule's estimated rates, and what the parts can't catch. The person +approves that text. The approval is bound to the digest of the parts it +describes; a changed part needs a new approval. + +## Use + +```bash +openadapt-flow generate-parts channels --openapi api.json --base-url URL \ + --auth-env READ_TOKEN --out channels.json +openadapt-flow generate-parts snapshot demo1 --channels channels.json --label before +# demonstrate the save once, with the values in demo1/params.json +openadapt-flow generate-parts snapshot demo1 --channels channels.json --label after +# repeat for demo2 and demo3, with different values +openadapt-flow generate-parts build --demo demo1 --demo demo2 --demo demo3 \ + --channels channels.json --out parts/ +openadapt-flow generate-parts approve parts/ --approved-by "Dana Reyes" +``` + +`--fhir-capability URL` takes a FHIR server's CapabilityStatement instead of, +or as well as, an OpenAPI document. Each demonstration folder holds +`params.json` (the values typed) and, optionally, `source.*` (the document +they came from). + +`build` writes `channel.json`, `verify.yaml`, `check_coverage.json`, +`duplicate_rule.json`, `inputs.json`, `parts.review.md` and +`parts.manifest.json`. It exits 0 only when every part was admitted. `approve` +shows the review, needs a person at a terminal who types `approve`, and +refuses an agent's name or a part that changed after it was generated. + +The fault gate runs on a test copy of the application when the caller gives +one (`build(..., test_copy=...)` in Python). Without one it runs on a +replica: an in-memory store shaped like the channel's records. A replica +tests the contract's judgment, not the application's read, and the coverage +record says `test_copy: replica`, which an admission policy can refuse. + +Generation needs two or three demonstrations with different values. A value +typed the same every time is checked against what was typed, but can't be +told from a constant, and the review says so. + +## Measurements + +The back-test against the hand-built OpenEMR referral parts, including where +generation is worse, is in `benchmark/generated_parts/`. On #1516's +duplicate stream (676 items, 223 true duplicates), the generated duplicate +rule missed 52 to 54 duplicates at every false-stop budget, against the +hand-built check's 24, so identity is a vetted template plus a generated +mapping. On the same stream, rendered as #1516's rows, OpenEMR REST rows and +FHIR resources, the generated mapping matched the hand-written adapters +exactly (24 missed, 2 false stops). diff --git a/packages/flow/docs/commands/02-check-and-admit/generate-parts.md b/packages/flow/docs/commands/02-check-and-admit/generate-parts.md new file mode 100644 index 000000000..e69de29bb diff --git a/packages/flow/openadapt_flow/__main__.py b/packages/flow/openadapt_flow/__main__.py index b0c6e9a66..cf25df1cd 100644 --- a/packages/flow/openadapt_flow/__main__.py +++ b/packages/flow/openadapt_flow/__main__.py @@ -9393,6 +9393,10 @@ def build_parser() -> argparse.ArgumentParser: _add_intake_parser(sub) + from openadapt_flow.generate.cli import add_parser as _add_generate_parser + + _add_generate_parser(sub) + from openadapt_flow.decision.cli import add_parser as _add_triage_parser _add_triage_parser(sub) diff --git a/packages/flow/openadapt_flow/deployment.py b/packages/flow/openadapt_flow/deployment.py index 3fa342ef1..ed165dcbe 100644 --- a/packages/flow/openadapt_flow/deployment.py +++ b/packages/flow/openadapt_flow/deployment.py @@ -387,6 +387,11 @@ class EffectsConfig(BaseModel): base_url: Optional[str] = None records_path: str = "/api/db" records_key: Optional[str] = "records" + #: ``rest`` only: dotted path in the response body to the next page's URL + #: (``links.next``). Set, the read follows every page on the same origin + #: and reads as unreadable when a page fails or the chain is too long. + #: Unset: one response is the whole read, as before. + records_next_path: Optional[str] = None #: Secret-isolated auth headers for the records read (bearer_env / #: header+value_env / basic_env). Never a credential literal. auth: Optional[AuthRef] = None @@ -612,7 +617,7 @@ def _serialize_compatible(self, handler: Any) -> dict[str, Any]: """Keep configs without a desktop read-back byte-identical.""" data: dict[str, Any] = handler(self) - for added in ("desktop_readback", "deferred_report"): + for added in ("desktop_readback", "deferred_report", "records_next_path"): if data.get(added, "") is None: data.pop(added, None) return data @@ -1887,6 +1892,7 @@ def sanitized(value: Any, *, key: str = "") -> Any: "fresh_session", "desktop_readback", "deferred_report", + "records_next_path", ): if config.get(added) is None: config.pop(added, None) @@ -2039,6 +2045,7 @@ def _build_effect_verifier_unredacted( timeout_s=cfg.timeout_s, poll_interval_s=cfg.poll_interval_s, drop_fields=tuple(cfg.records_drop_fields), + next_path=cfg.records_next_path, ) if kind == "fhir": diff --git a/packages/flow/openadapt_flow/generate/__init__.py b/packages/flow/openadapt_flow/generate/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/packages/flow/openadapt_flow/generate/baseline.py b/packages/flow/openadapt_flow/generate/baseline.py new file mode 100644 index 000000000..bce3acc7a --- /dev/null +++ b/packages/flow/openadapt_flow/generate/baseline.py @@ -0,0 +1,109 @@ +"""The floor a generated record contract may never go below. + +Generation reads untrusted content: page text, response bodies, field values. +Whatever a generator proposes, the contract it hands to a person must keep a +fixed set of checks that does not come from that content: + +- every parameter the demonstration typed is checked in the saved record; +- the frame (nothing else in the read set changes) is on, compared by record + identity, with nothing excused by attribution; +- no ignored field is a field a parameter is written to, and no collateral + rule is turned off. + +:func:`baseline_problems` lists every way a contract falls short. A contract +with any problem is refused, so a generated part can add checks but never +remove or weaken one. Attribution, the one rule that excuses a change, is +safety policy: a person adds it, never a generator. +""" + +from __future__ import annotations + +from collections.abc import Iterable, Sequence +from typing import Any + +from openadapt_flow.runtime.effects.effect import Effect, EffectKind + +#: Verifier kinds that read through the acting screen: never a generated channel. +_SCREEN_KINDS = frozenset({"none", "onscreen"}) + + +def _param_fields(effects: Sequence[Effect]) -> dict[str, set[str]]: + """Parameter -> the record fields effects check it against.""" + + out: dict[str, set[str]] = {} + for effect in effects: + selectors = ( + list(effect.new_records) + if effect.kind is EffectKind.EXACT_NEW_SET + else [effect.match] + ) + for selector in selectors: + for field, expr in selector.items(): + if expr.param is not None: + out.setdefault(expr.param, set()).add(field) + if effect.field and effect.value is not None and effect.value.param: + out.setdefault(effect.value.param, set()).add(effect.field) + return out + + +def baseline_problems(cfg: Any, typed_params: Iterable[str]) -> list[str]: + """Every way ``cfg`` (a parsed ``verify.yaml``) is weaker than the floor. + + ``typed_params`` are the parameters the demonstration typed into the app: + each must be checked against a record field. + """ + + problems: list[str] = [] + effects: list[Effect] = list(cfg.effects) + if cfg.verifier.kind in _SCREEN_KINDS: + problems.append(f"read channel {cfg.verifier.kind!r} reads the acting screen") + if not any( + e.kind in (EffectKind.RECORD_WRITTEN, EffectKind.EXACT_NEW_SET) + and (e.expected_count >= 1 or e.kind is EffectKind.EXACT_NEW_SET) + for e in effects + ): + problems.append("no effect requires the saved record to exist") + checked = _param_fields(effects) + for name in sorted(set(typed_params)): + if name not in checked: + problems.append(f"parameter {name!r} is not checked in the saved record") + for effect in effects: + if "forbid_collateral_loss" in effect.model_fields_set and not ( + effect.forbid_collateral_loss + ): + problems.append("an effect turns off the collateral-loss rule") + if "forbid_collateral_change" in effect.model_fields_set and not ( + effect.forbid_collateral_change + ): + problems.append("an effect turns off the collateral-change rule") + if effect.needs_operator_confirmation: + problems.append("an effect is still a placeholder") + frame = getattr(cfg, "frame", None) + if frame is None: + problems.append("the contract declares no frame (nothing else may change)") + elif not frame.identity_field: + problems.append("the frame names no identity field to compare records by") + if frame is not None: + if frame.attribution is not None: + problems.append( + "the frame excuses changes by attribution, which only a person sets" + ) + if frame.target_fields is not None: + problems.append("the frame widens what a target record may change") + written = {f for fields in checked.values() for f in fields} + hidden = sorted(set(frame.ignore_fields) & written) + if hidden: + problems.append( + "the frame ignores field(s) a parameter is written to: " + + ", ".join(hidden) + ) + drop = set(cfg.verifier.records_drop_fields) + for sub in cfg.verifier.surfaces.values(): + drop |= set(sub.records_drop_fields) + written = {f for fields in checked.values() for f in fields} + dropped = sorted(drop & written) + if dropped: + problems.append( + "the read drops field(s) a parameter is written to: " + ", ".join(dropped) + ) + return problems diff --git a/packages/flow/openadapt_flow/generate/build.py b/packages/flow/openadapt_flow/generate/build.py new file mode 100644 index 000000000..a01b3d725 --- /dev/null +++ b/packages/flow/openadapt_flow/generate/build.py @@ -0,0 +1,650 @@ +"""Generate all four application parts from demonstrations, and verify each. + +``openadapt-flow generate-parts build`` reads a folder per demonstration: + +- ``params.json``: the values the person typed; +- ``reads/.before.json`` and ``reads/.after.json``: each + candidate channel read just before and just after the save + (``generate-parts snapshot`` takes them); +- optionally ``source.*``: the document the values came from. + +and writes, in the output folder: + +========================== ================================================== +``channel.json`` the admitted read channel and its channel test +``verify.yaml`` the record contract (``openadapt.verify/v1``) +``check_coverage.json`` the fault gate's coverage of that contract +``duplicate_rule.json`` the identity and duplicate rule, with calibration +``inputs.json`` input types, labels and conversions, round trip +``parts.review.md`` every check in plain words, for a person +``parts.manifest.json`` the digest of each part; what an approval binds +========================== ================================================== + +Nothing here is approved. ``generate-parts approve`` records a person's +approval of ``parts.review.md``, bound to the manifest digest. The safety +policy (profiles, approval modes, the switch, which steps are consequential, +the gate order, attribution) is never generated. +""" + +from __future__ import annotations + +import hashlib +import json +import time +from collections.abc import Mapping, Sequence +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any, Optional + +from openadapt_flow.generate.channel import ( + ChannelCandidate, + ChannelReport, + DemoReads, + check_channel, +) +from openadapt_flow.generate.contract import ( + ContractError, + Demonstration, + GatedContract, + generate_and_gate, +) + +MANIFEST_SCHEMA = "openadapt.generated-parts/v1" +APPROVAL_SCHEMA = "openadapt.generated-parts-approval/v1" +PART_FILES = ( + "channel.json", + "verify.yaml", + "check_coverage.json", + "identity_mapping.json", + "duplicate_rule.json", + "inputs.json", + "parts.review.md", +) + + +class BuildError(ValueError): + """The demonstrations can't produce admitted parts. Nothing is written.""" + + +@dataclass +class DemoFolder: + path: Path + params: dict[str, str] + reads: dict[str, tuple[Optional[list[dict]], Optional[list[dict]]]] + source: Optional[Path] = None + + +def channel_id(candidate: ChannelCandidate) -> str: + raw = json.dumps(candidate.section, sort_keys=True, default=str) + return "ch-" + hashlib.sha256(raw.encode()).hexdigest()[:12] + + +def _load_json(path: Path) -> Any: + return json.loads(path.read_text()) if path.is_file() else None + + +def load_demo(path: Path) -> DemoFolder: + params = _load_json(path / "params.json") + if not isinstance(params, dict): + raise BuildError(f"{path}: params.json is missing or not an object") + reads: dict[str, tuple[Optional[list[dict]], Optional[list[dict]]]] = {} + for before in sorted((path / "reads").glob("*.before.json")): + cid = before.name[: -len(".before.json")] + after = path / "reads" / f"{cid}.after.json" + reads[cid] = (_load_json(before), _load_json(after)) + sources = sorted(p for p in path.glob("source.*") if p.is_file()) + return DemoFolder( + path, + {k: str(v) for k, v in params.items()}, + reads, + sources[0] if sources else None, + ) + + +def snapshot( + candidates: Sequence[ChannelCandidate], + params: Mapping[str, object], + demo: Path, + label: str, +) -> dict[str, int]: + """Read every candidate once; write ``reads/.