CRF ablations and learning curve

README.md

@@ -156,7 +156,7 @@ item can be highlighted in the original advisory.
 | 1 · Feasibility | Pinned ingestion, corpus statistics, label schema, offset-preserving preprocessing, pilot sample, annotation tooling | ✅ built |
 | 1 · Pilot | 100-document pilot (`pilot-2`: 22 hand-labelled, 78 from model drafts), [decision gate](docs/pilot_gate.md), guidelines v0.2 frozen, [acceptance targets](docs/targets.md) declared | ✅ done |
 | 2 · Baselines | Rules baseline, CRF + logistic-regression attributes and groups, evaluator with bootstrap CIs, evidence viewer | ✅ built · preliminary pilot dev scores |
-| 3 · Neural models | Frozen leakage-safe splits, annotation batches (01–03, 06: model-drafted), DistilBERT and DeBERTa-v3 with span / role / group heads, experiment runner | 🟨 trained on model drafts (run 3) · DeBERTa ≈ CRF on held-out agreement, no clear winner |
+| 3 · Neural models | Frozen leakage-safe splits, annotation batches (01–03, 06: model-drafted), DistilBERT and DeBERTa-v3 with span / role / group heads, experiment runner, CRF ablations and learning curve | 🟨 trained on model drafts (run 3) · DeBERTa ≈ CRF on held-out agreement, no clear winner |
 | 4 · Reliability | Calibration and review policy, formatting robustness (rules: 100%), challenge-set and error-catalogue tooling | 🟨 tooling ready · thresholds await labels |
 | 5 · Application | FastAPI API, review UI, SQLite (WAL) with append-only history, exports, Docker Compose stack, release manifests, tested rollback, CI image smoke test | ✅ built |
 | 6 · Production | Benchmark harness, ONNX/INT8 export (FP32 parity exact), local proxy benchmarks and load tests, monitoring; target-host numbers and hosted release pending | 🟨 harness and proxy results |
@@ -217,6 +217,12 @@ to three times as many agreeing conditions. Its held-out typed F1 is clearly hig
 clearly different, and its false-prerequisite rate is higher at every target (not clearly).
 The CRF therefore stays the default model; see [`reports/calibration/`](reports/calibration/README.md).
 
+**Ablations and learning curve (CRF).** On the 172 agreement records, removing the heading/list
+context features (−0.006 typed F1 [−0.027, +0.020]) or the part-of-speech features (+0.002
+[−0.025, +0.032]) makes no clear difference. Typed agreement rises from 0.054 with 30 training
+documents to 0.143 with 120, then stays flat (0.145 with all 239); span agreement keeps rising
+slowly. See [`reports/crf-ablations/`](reports/crf-ablations/README.md).
+
 95% project-group bootstrap intervals (1,000 resamples). Full reports, including scores against
 the 28 model-made dev records (which measure agreement with the drafting model, not accuracy),
 are in [`reports/pilot/`](reports/pilot/README.md). The paired difference from adding `batch-01` (+0.214 typed F1,

configs/experiments/crf-ablations.toml

@@ -0,0 +1,82 @@
+# CRF ablations and learning curve, one factor at a time. The CRF is deterministic for a
+# fixed training set, so the full-data variants run once; each budget below the full train
+# split draws three seeded random subsets.
+#   bash scripts/experiments/crf_ablations.sh
+# Dev records only measure; nothing here is fitted or selected on them.
+
+[train]
+family = "crf"
+model_id = "crf-abl"
+use_pos = true
+use_context = true
+
+[[runs]]
+name = "full"
+
+[[runs]]
+name = "no-context"       # heading, list and lead-in features removed
+use_context = false
+
+[[runs]]
+name = "no-pos"           # spaCy part-of-speech features removed
+use_pos = false
+
+[[runs]]
+name = "docs30-seed0"
+max_train_documents = 30
+seed = 0
+
+[[runs]]
+name = "docs30-seed1"
+max_train_documents = 30
+seed = 1
+
+[[runs]]
+name = "docs30-seed2"
+max_train_documents = 30
+seed = 2
+
+[[runs]]
+name = "docs60-seed0"
+max_train_documents = 60
+seed = 0
+
+[[runs]]
+name = "docs60-seed1"
+max_train_documents = 60
+seed = 1
+
+[[runs]]
+name = "docs60-seed2"
+max_train_documents = 60
+seed = 2
+
+[[runs]]
+name = "docs120-seed0"
+max_train_documents = 120
+seed = 0
+
+[[runs]]
+name = "docs120-seed1"
+max_train_documents = 120
+seed = 1
+
+[[runs]]
+name = "docs120-seed2"
+max_train_documents = 120
+seed = 2
+
+[[runs]]
+name = "docs180-seed0"
+max_train_documents = 180
+seed = 0
+
+[[runs]]
+name = "docs180-seed1"
+max_train_documents = 180
+seed = 1
+
+[[runs]]
+name = "docs180-seed2"
+max_train_documents = 180
+seed = 2

docs/model_card.md

@@ -134,6 +134,17 @@ The default model is therefore still the CRF. The live review policy stays
 `review-policy-0` (uncalibrated: every prediction is flagged), because thresholds fitted on
 model drafts measure agreement, not accuracy.
 
+## Ablations and learning curve
+
+Measured on the CRF, as agreement with the 172 model-drafted dev records
+([`reports/crf-ablations/`](../reports/crf-ablations/README.md)):
+- **Context and part-of-speech features:** removing either makes no clear difference (typed F1
+  −0.006 [−0.027, +0.020] and +0.002 [−0.025, +0.032]).
+- **Learning curve:** typed agreement rises from 0.054 with 30 training documents to 0.143 with
+  120, then stays flat at about 0.14. Span agreement keeps rising slowly.
+
+The group-objective ablation and transformer learning curves need a GPU and have not been run.
+
 ## Release process
 
 - **Thresholds.** Review thresholds and confidence cut-offs are chosen on the dev calibration

reports/crf-ablations/README.md

@@ -0,0 +1,91 @@
+# CRF ablations and learning curve (2026-11-07)
+
+How much each feature group and each extra training document contributes to the CRF behind
+the v0.1.0 preview (`crf-batches-01-02`). The runs are in
+[`configs/experiments/crf-ablations.toml`](../../configs/experiments/crf-ablations.toml) and are
+run by [`scripts/experiments/crf_ablations.sh`](../../scripts/experiments/crf_ablations.sh).
+
+- **Training data:** the train split of `pilot-2`, `batch-01` and `batch-02` (239 documents: 7
+  `manual`, 232 model drafts), the released model's training set.
+- **Scored on:** the 172 model-drafted dev records (`batch-03` + `batch-06`). These scores are
+  **agreement with the drafting model, not accuracy**. The CRF is not tuned or selected on
+  them; every run here is a measurement.
+- **Also scored on:** the 10 hand-labelled dev documents of `pilot-2`, for the ablations.
+- **Untouched:** the test split.
+- **Reproducible:** the full-data run reproduces the released model exactly. All 272 of its
+  predictions equal those of `crf-batches-01-02` in
+  [`reports/agreement/predictions/`](../agreement/predictions/), apart from the model id.
+
+95% project-group bootstrap intervals, 1,000 resamples.
+
+## Ablations
+
+One factor removed at a time, on all 239 training documents:
+- **`no-context`** drops the heading, list-level and lead-in features from the tagger and the
+  attribute classifiers, and the list and lead-in features from the group classifier.
+- **`no-pos`** drops the spaCy part-of-speech features.
+
+| Run | Typed F1 | Exact span F1 | Role macro-F1 | False-prerequisite rate | Training seconds |
+|---|---:|---:|---:|---:|---:|
+| full | 0.145 [0.104, 0.186] | 0.221 [0.169, 0.268] | 0.115 [0.066, 0.161] | 0.353 [0.268, 0.417] | 22 |
+| no-context | 0.139 [0.101, 0.178] | 0.203 [0.160, 0.242] | 0.114 [0.061, 0.172] | 0.377 [0.298, 0.436] | 19 |
+| no-pos | 0.147 [0.110, 0.188] | 0.217 [0.168, 0.259] | 0.121 [0.069, 0.177] | 0.331 [0.228, 0.410] | 8 |
+
+Paired differences against the full model (B − A):
+
+| B | Slice | Typed F1 | Exact span F1 | False-prerequisite rate (lower is better) |
+|---|---|---:|---:|---:|
+| no-context | agreement (172) | −0.006 [−0.027, +0.020] | −0.018 [−0.044, +0.011] | +0.024 [−0.030, +0.073] |
+| no-pos | agreement (172) | +0.002 [−0.025, +0.032] | −0.004 [−0.035, +0.031] | −0.022 [−0.089, +0.024] |
+| no-context | hand-labelled (10) | +0.030 [+0.000, +0.095] | +0.030 [+0.000, +0.095] | −0.048 [−0.500, +0.068] |
+| no-pos | hand-labelled (10) | +0.000 [+0.000, +0.000] | +0.000 [+0.000, +0.000] | +0.000 [+0.000, +0.000] |
+
+None of these differences is clear. Full reports: `compare_agreement_*.md` and
+`compare_manual_*.md`.
+
+## Learning curve
+
+Seeded random subsets of the 239 training documents, three subsets per budget. Mean ± SD
+across the three subsets. The CRF is deterministic for a fixed training set, so the
+239-document point is a single run.
+
+| Training documents | Typed F1 | Exact span F1 | Role macro-F1 | False-prerequisite rate |
+|---:|---:|---:|---:|---:|
+| 30 | 0.054 ± 0.006 | 0.099 ± 0.007 | 0.027 ± 0.009 | 0.465 ± 0.084 |
+| 60 | 0.094 ± 0.017 | 0.143 ± 0.007 | 0.061 ± 0.028 | 0.386 ± 0.055 |
+| 120 | 0.143 ± 0.008 | 0.194 ± 0.006 | 0.088 ± 0.023 | 0.375 ± 0.042 |
+| 180 | 0.140 ± 0.010 | 0.209 ± 0.008 | 0.108 ± 0.006 | 0.364 ± 0.034 |
+| 239 | 0.145 | 0.221 | 0.115 | 0.353 |
+
+Every run, with intervals, is in [`experiment.md`](experiment.md).
+
+## Reading
+
+- **Heading and list context adds nothing measurable to the CRF.** Removing it lowers span
+  agreement by 0.018, with an interval across zero. The group classifier, where list context
+  should matter most, matches no model-drafted group in any run (group F1 0.000).
+- **Part-of-speech features add nothing measurable either,** and they cost the most. Without
+  them training is about three times faster and needs no spaCy model. On the 10 hand-labelled
+  documents the two models' predictions score identically. This makes dropping spaCy from the
+  CRF a candidate simplification. It is not applied here: the released model stays as it is.
+- **Typed agreement stops rising at about 120 documents.** It climbs from 0.054 (30 documents)
+  to 0.143 (120), then stays flat (0.140 at 180, 0.145 at 239). Span agreement and role
+  macro-F1 keep rising slowly, and the false-prerequisite rate keeps falling slowly. More
+  model-drafted documents of the same kind are therefore unlikely to move the CRF's typed
+  agreement much. The limit is the attribute classifiers (category and role), not the number of
+  spans.
+- **The group-objective ablation does not apply to the CRF.** Its group classifier is separate
+  and predicts no matching group. That ablation is planned for the transformer
+  (`configs/experiments/ablations.toml`), which needs a GPU.
+
+Every number here is agreement with model drafts or a 10-document slice; neither is a release
+result.
+
+## Reproduce
+
+```bash
+bash scripts/experiments/crf_ablations.sh   # 15 trainings, a few minutes on a laptop
+```
+
+Bundles go to `artifacts/experiments/crf-ablations/` (not committed). The predictions of the
+three full-data runs are committed in `predictions/`.

reports/crf-ablations/compare_agreement_no-context.md

@@ -0,0 +1,25 @@
+# Agreement (172 model-drafted dev records): full -> no-context
+
+- Gold: `annotations/batches/batch-03.jsonl`, `annotations/batches/batch-06.jsonl` (dev split of `splits/pool-v1.json`; origin: all; 172 documents)
+- Baseline (A): `reports/crf-ablations/predictions/crf-abl-full.jsonl` (crf-abl-full)
+- Candidate (B): `reports/crf-ablations/predictions/crf-abl-no-context.jsonl` (crf-abl-no-context)
+- Bootstrap: 1000 paired resamples, project clusters, seed 0
+- Documents: 172
+
+Differences are B minus A with 95% paired bootstrap intervals. *Clear* means the whole interval lies on one side of zero.
+
+| Metric | A (crf-abl-full) | B (crf-abl-no-context) | B - A | 95% CI | Verdict |
+|---|---:|---:|---:|:---:|---|
+| Typed condition micro-F1 (span + category + role) | 0.145 | 0.139 | -0.006 | [-0.027, +0.020] | no clear difference |
+| Exact span F1 | 0.221 | 0.203 | -0.018 | [-0.044, +0.011] | no clear difference |
+| Exact span precision | 0.332 | 0.326 | -0.006 | [-0.048, +0.050] | no clear difference |
+| Exact span recall | 0.166 | 0.148 | -0.018 | [-0.040, +0.003] | no clear difference |
+| Role macro-F1 (roles with gold support) | 0.115 | 0.114 | -0.000 | [-0.011, +0.017] | no clear difference |
+| Category macro-F1 | 0.137 | 0.122 | -0.015 | [-0.038, +0.007] | no clear difference |
+| Group exact-match F1 | 0.000 | 0.000 | +0.000 | [+0.000, +0.000] | no clear difference |
+| Complete-document exact match, documents with conditions | 0.065 | 0.045 | -0.019 | [-0.043, +0.000] | no clear difference |
+| Complete-document exact match, documents without conditions | 0.882 | 1.000 | +0.118 | [+0.000, +0.286] | no clear difference |
+| False prerequisite rate (lower is better) | 0.353 | 0.377 | +0.024 | [-0.030, +0.073] | no clear difference |
+| Reversed meaning rate (lower is better) | 0.074 | 0.064 | -0.010 | [-0.039, +0.009] | no clear difference |
+| Category accuracy on exactly matched spans | 0.797 | 0.807 | +0.010 | [-0.055, +0.098] | no clear difference |
+| Role accuracy on exactly matched spans | 0.859 | 0.860 | +0.000 | [-0.051, +0.044] | no clear difference |

reports/crf-ablations/compare_agreement_no-pos.md

@@ -0,0 +1,25 @@
+# Agreement (172 model-drafted dev records): full -> no-pos
+
+- Gold: `annotations/batches/batch-03.jsonl`, `annotations/batches/batch-06.jsonl` (dev split of `splits/pool-v1.json`; origin: all; 172 documents)
+- Baseline (A): `reports/crf-ablations/predictions/crf-abl-full.jsonl` (crf-abl-full)
+- Candidate (B): `reports/crf-ablations/predictions/crf-abl-no-pos.jsonl` (crf-abl-no-pos)
+- Bootstrap: 1000 paired resamples, project clusters, seed 0
+- Documents: 172
+
+Differences are B minus A with 95% paired bootstrap intervals. *Clear* means the whole interval lies on one side of zero.
+
+| Metric | A (crf-abl-full) | B (crf-abl-no-pos) | B - A | 95% CI | Verdict |
+|---|---:|---:|---:|:---:|---|
+| Typed condition micro-F1 (span + category + role) | 0.145 | 0.147 | +0.002 | [-0.025, +0.032] | no clear difference |
+| Exact span F1 | 0.221 | 0.217 | -0.004 | [-0.035, +0.031] | no clear difference |
+| Exact span precision | 0.332 | 0.333 | +0.002 | [-0.046, +0.070] | no clear difference |
+| Exact span recall | 0.166 | 0.161 | -0.005 | [-0.030, +0.022] | no clear difference |
+| Role macro-F1 (roles with gold support) | 0.115 | 0.121 | +0.006 | [-0.015, +0.035] | no clear difference |
+| Category macro-F1 | 0.137 | 0.140 | +0.003 | [-0.021, +0.031] | no clear difference |
+| Group exact-match F1 | 0.000 | 0.000 | +0.000 | [+0.000, +0.000] | no clear difference |
+| Complete-document exact match, documents with conditions | 0.065 | 0.045 | -0.019 | [-0.041, +0.000] | no clear difference |
+| Complete-document exact match, documents without conditions | 0.882 | 1.000 | +0.118 | [+0.000, +0.286] | no clear difference |
+| False prerequisite rate (lower is better) | 0.353 | 0.331 | -0.022 | [-0.089, +0.024] | no clear difference |
+| Reversed meaning rate (lower is better) | 0.074 | 0.085 | +0.011 | [-0.037, +0.054] | no clear difference |
+| Category accuracy on exactly matched spans | 0.797 | 0.806 | +0.010 | [-0.058, +0.097] | no clear difference |
+| Role accuracy on exactly matched spans | 0.859 | 0.871 | +0.012 | [-0.009, +0.036] | no clear difference |

reports/crf-ablations/compare_manual_no-context.md

@@ -0,0 +1,25 @@
+# Hand-labelled dev (10): full -> no-context
+
+- Gold: `annotations/pilot/pilot-2.jsonl` (dev split of `splits/pool-v1.json`; origin: manual; 10 documents)
+- Baseline (A): `reports/crf-ablations/predictions/crf-abl-full.jsonl` (crf-abl-full)
+- Candidate (B): `reports/crf-ablations/predictions/crf-abl-no-context.jsonl` (crf-abl-no-context)
+- Bootstrap: 1000 paired resamples, project clusters, seed 0
+- Documents: 10
+
+Differences are B minus A with 95% paired bootstrap intervals. *Clear* means the whole interval lies on one side of zero.
+
+| Metric | A (crf-abl-full) | B (crf-abl-no-context) | B - A | 95% CI | Verdict |
+|---|---:|---:|---:|:---:|---|
+| Typed condition micro-F1 (span + category + role) | 0.370 | 0.400 | +0.030 | [+0.000, +0.095] | no clear difference |
+| Exact span F1 | 0.370 | 0.400 | +0.030 | [+0.000, +0.095] | no clear difference |
+| Exact span precision | 0.500 | 0.625 | +0.125 | [+0.000, +0.379] | no clear difference |
+| Exact span recall | 0.294 | 0.294 | +0.000 | [+0.000, +0.000] | no clear difference |
+| Role macro-F1 (roles with gold support) | 0.200 | 0.217 | +0.017 | [+0.000, +0.066] | no clear difference |
+| Category macro-F1 | 0.237 | 0.232 | -0.004 | [-0.083, +0.047] | no clear difference |
+| Group exact-match F1 | 0.500 | 0.500 | +0.000 | [+0.000, +0.000] | no clear difference |
+| Complete-document exact match, documents with conditions | 0.222 | 0.222 | +0.000 | [+0.000, +0.000] | no clear difference |
+| Complete-document exact match, documents without conditions | 1.000 | 1.000 | +0.000 | [+0.000, +0.000] | no clear difference |
+| False prerequisite rate (lower is better) | 0.333 | 0.286 | -0.048 | [-0.500, +0.068] | no clear difference |
+| Reversed meaning rate (lower is better) | 0.000 | 0.000 | +0.000 | [+0.000, +0.000] | no clear difference |
+| Category accuracy on exactly matched spans | 1.000 | 1.000 | +0.000 | [+0.000, +0.000] | no clear difference |
+| Role accuracy on exactly matched spans | 1.000 | 1.000 | +0.000 | [+0.000, +0.000] | no clear difference |

reports/crf-ablations/compare_manual_no-pos.md

@@ -0,0 +1,25 @@
+# Hand-labelled dev (10): full -> no-pos
+
+- Gold: `annotations/pilot/pilot-2.jsonl` (dev split of `splits/pool-v1.json`; origin: manual; 10 documents)
+- Baseline (A): `reports/crf-ablations/predictions/crf-abl-full.jsonl` (crf-abl-full)
+- Candidate (B): `reports/crf-ablations/predictions/crf-abl-no-pos.jsonl` (crf-abl-no-pos)
+- Bootstrap: 1000 paired resamples, project clusters, seed 0
+- Documents: 10
+
+Differences are B minus A with 95% paired bootstrap intervals. *Clear* means the whole interval lies on one side of zero.
+
+| Metric | A (crf-abl-full) | B (crf-abl-no-pos) | B - A | 95% CI | Verdict |
+|---|---:|---:|---:|:---:|---|
+| Typed condition micro-F1 (span + category + role) | 0.370 | 0.370 | +0.000 | [+0.000, +0.000] | no clear difference |
+| Exact span F1 | 0.370 | 0.370 | +0.000 | [+0.000, +0.000] | no clear difference |
+| Exact span precision | 0.500 | 0.500 | +0.000 | [+0.000, +0.000] | no clear difference |
+| Exact span recall | 0.294 | 0.294 | +0.000 | [+0.000, +0.000] | no clear difference |
+| Role macro-F1 (roles with gold support) | 0.200 | 0.200 | +0.000 | [+0.000, +0.000] | no clear difference |
+| Category macro-F1 | 0.237 | 0.237 | +0.000 | [+0.000, +0.000] | no clear difference |
+| Group exact-match F1 | 0.500 | 0.500 | +0.000 | [+0.000, +0.000] | no clear difference |
+| Complete-document exact match, documents with conditions | 0.222 | 0.222 | +0.000 | [+0.000, +0.000] | no clear difference |
+| Complete-document exact match, documents without conditions | 1.000 | 1.000 | +0.000 | [+0.000, +0.000] | no clear difference |
+| False prerequisite rate (lower is better) | 0.333 | 0.333 | +0.000 | [+0.000, +0.000] | no clear difference |
+| Reversed meaning rate (lower is better) | 0.000 | 0.000 | +0.000 | [+0.000, +0.000] | no clear difference |
+| Category accuracy on exactly matched spans | 1.000 | 1.000 | +0.000 | [+0.000, +0.000] | no clear difference |
+| Role accuracy on exactly matched spans | 1.000 | 1.000 | +0.000 | [+0.000, +0.000] | no clear difference |