Record CRF v0.1.0 benchmarks on the target host

README.md

@@ -159,7 +159,7 @@ item can be highlighted in the original advisory.
 | 3 · Neural models | Frozen leakage-safe splits, annotation batches (01–03, 06: model-drafted), DistilBERT and DeBERTa-v3 with span / role / group heads, experiment runner, CRF ablations and learning curve | 🟨 trained on model drafts (run 3) · DeBERTa ≈ CRF on held-out agreement, no clear winner |
 | 4 · Reliability | Calibration and review policy, formatting robustness (rules: 100%), challenge-set and error-catalogue tooling | 🟨 tooling ready · thresholds await labels |
 | 5 · Application | FastAPI API, review UI, SQLite (WAL) with append-only history, exports, Docker Compose stack, release manifests, tested rollback, CI image smoke test | ✅ built |
-| 6 · Production | Benchmark harness, ONNX/INT8 export (FP32 parity exact), local proxy benchmarks and load tests, monitoring; target-host numbers and hosted release pending | 🟨 harness and proxy results |
+| 6 · Production | Benchmark harness, ONNX/INT8 export (FP32 parity exact), local proxy benchmarks and load tests, monitoring; CRF v0.1.0 measured on the 2 vCPU / 4 GB target host | 🟨 CRF meets every serving target on the target host · transformers not yet measured there |
 
 ### Preliminary pilot results (dev split only)
 
@@ -373,9 +373,23 @@ acceptance gate has been evaluated, because labels after the pilot are model dra
 [release notes](docs/release_notes/v0.1.0.md). Its rollback drill (rules `r1` → CRF `v0.1.0`
 → `r1`) passed on local Docker.
 
-## Serving cost (local proxy)
+## Serving cost
 
-Measured on an 8 GB Apple M1 with inference limited to 2 threads. This is **not the target
+**On the target host (2 vCPU / 4 GB x86-64, AWS EC2, 2026-11-09)**, the released CRF
+(`crf-batches-01-02`, v0.1.0) meets every serving target:
+
+| Measure | CRF v0.1.0 | Target |
+|---|---:|---:|
+| Warm p95 latency, ~512-token advisory | 0.06 s | ≤ 2.5 s |
+| Warm p95 latency, ~2,000-token advisory | 0.14 s | ≤ 8 s |
+| Throughput, natural workload (in process) | 23.6 docs/s | ≥ 0.25 docs/s |
+| Peak memory (in process) | 276 MB | ≤ 3 GB |
+| HTTP load test, 1,000 requests at concurrency 4 (Compose stack) | 100% succeeded, p95 0.61 s, 18.6 req/s | ≥ 99.5% |
+
+Full report: [`reports/benchmarks/target-host-crf.md`](reports/benchmarks/target-host-crf.md).
+The transformer models have not been measured on this host.
+
+**Local proxy for the transformers.** Measured on an 8 GB Apple M1 with inference limited to 2 threads. This is **not the target
 host**, and the transformer bundles have untrained heads, so latency and memory are real but
 accuracy is not measured. Details and caveats: [`docs/benchmarks.md`](docs/benchmarks.md).
 

docs/benchmarks.md

@@ -71,6 +71,29 @@ FP32 ONNX is numerically exact. INT8 moves the logits, and DeBERTa much more tha
 DistilBERT. With untrained heads the INT8 *predictions* are not meaningful, so whether INT8
 is usable is decided only by the ≤ 1-point typed-F1 gate on a trained model.
 
+## Results on the target host: CRF v0.1.0
+
+Full tables: [`reports/benchmarks/target-host-crf.md`](../reports/benchmarks/target-host-crf.md)
+(raw JSON and checkpoint next to it).
+- **Host:** AWS EC2, 2 vCPU / 4 GiB x86-64 (Intel Xeon Platinum 8488C, 3.72 GB visible), Ubuntu
+  24.04. The harness recognised it as the documented host.
+- **Model:** the released v0.1.0 bundle `crf-batches-01-02`, measured on 2026-11-09.
+
+| Measure | Result | Target |
+|---|---:|---:|
+| Warm p95, ~512 tokens | 0.062 s | ≤ 2.5 s ✅ |
+| Warm p95, ~2,000 tokens | 0.145 s | ≤ 8 s ✅ |
+| Natural throughput, in process | 23.6 docs/s | ≥ 0.25 ✅ |
+| Peak RSS | 276 MB | ≤ 3,072 MB ✅ |
+| Cold start (imports, load, first extraction) | 4.1 s | not a target |
+| Load test: 1,000 requests, concurrency 4 | 100% succeeded; p50 0.14 s, p95 0.61 s, p99 1.02 s; 18.6 req/s | ≥ 99.5% ✅ |
+
+- **How the load test ran:** against the Docker Compose stack of release v0.1.0, from a client
+  on the same host, so the client shared the 2 vCPUs. The service's latency is therefore, if
+  anything, overstated.
+- **Not measured on this host:** the transformers. Measuring them needs a trained bundle, and
+  the INT8 ≤ 1-point gate still applies.
+
 ## Results so far: local proxy (Apple M1, 2 inference threads)
 
 Full tables: [`reports/benchmarks/local-m1-2threads.md`](../reports/benchmarks/local-m1-2threads.md)

reports/benchmarks/load-target-host-crf.json

@@ -0,0 +1,19 @@
+{
+  "requests": 1000,
+  "concurrency": 4,
+  "distinct_texts": 100,
+  "cache": "bypassed (unique raw text per request)",
+  "succeeded": 1000,
+  "success_rate": 1.0,
+  "status_counts": {
+    "200": 1000
+  },
+  "latency_s": {
+    "p50": 0.1403,
+    "p95": 0.6144,
+    "p99": 1.0167
+  },
+  "throughput_rps": 18.6,
+  "elapsed_s": 53.78,
+  "label": "crf"
+}

reports/benchmarks/target-host-crf.json

@@ -0,0 +1,80 @@
+{
+  "environment": {
+    "measured_at": "2026-11-09T03:34:45+00:00",
+    "system": "Linux",
+    "machine": "x86_64",
+    "cpu_model": "Intel(R) Xeon(R) Platinum 8488C",
+    "logical_cpus": 2,
+    "cpu_limit": null,
+    "memory_gb": 3.72,
+    "in_container": false,
+    "python": "3.12.16",
+    "git_commit": "55e1778",
+    "matches_target_host": true
+  },
+  "results": [
+    {
+      "config": {
+        "kind": "crf",
+        "model_dir": "crf-batches-01-02",
+        "threads": null,
+        "batch_size": null,
+        "label": "crf",
+        "name": "crf"
+      },
+      "model_id": "crf-batches-01-02",
+      "cold_start": {
+        "imports_s": 0.22,
+        "load_s": 2.501,
+        "first_extraction_s": 1.4,
+        "total_s": 4.121
+      },
+      "buckets": {
+        "natural": {
+          "n": 100,
+          "p50": 0.0231,
+          "p95": 0.1314,
+          "max": 0.3479,
+          "mean": 0.0423,
+          "docs_per_s": 23.619
+        },
+        "~2000": {
+          "n": 60,
+          "p50": 0.1227,
+          "p95": 0.1446,
+          "max": 0.1764,
+          "mean": 0.1195
+        },
+        "~512": {
+          "n": 60,
+          "p50": 0.0395,
+          "p95": 0.0617,
+          "max": 0.0892,
+          "mean": 0.0425
+        }
+      },
+      "peak_rss_mb": 276.2
+    }
+  ],
+  "load": [
+    {
+      "requests": 1000,
+      "concurrency": 4,
+      "distinct_texts": 100,
+      "cache": "bypassed (unique raw text per request)",
+      "succeeded": 1000,
+      "success_rate": 1.0,
+      "status_counts": {
+        "200": 1000
+      },
+      "latency_s": {
+        "p50": 0.1403,
+        "p95": 0.6144,
+        "p99": 1.0167
+      },
+      "throughput_rps": 18.6,
+      "elapsed_s": 53.78,
+      "label": "crf"
+    }
+  ]
+}

reports/benchmarks/target-host-crf.md

@@ -0,0 +1,32 @@
+# CPU inference benchmark: target host, CRF v0.1.0
+
+- Measured 2026-11-09T03:34:45+00:00 at commit `55e1778`
+- Hardware: Intel(R) Xeon(R) Platinum 8488C (x86_64), 2 CPUs, 3.72 GB, Linux, Python 3.12.16
+- **This is the documented target host**: targets are judged below.
+- Workload `workload-v1`: train and dev pools of splits/pool-v1.json; lengths in `distilbert/distilbert-base-cased` subword tokens. Natural-sample distribution: max 5772, p10 59, p50 234, p90 1775, p99 3112
+- AWS EC2, 2 vCPU / 4 GiB x86-64 (Intel Xeon Platinum 8488C), Ubuntu 24.04, measured 2026-11-09.
+- Model: released v0.1.0 bundle crf-batches-01-02 (SHA-256 65677a74…d29a), in process.
+- Load test: the Docker Compose stack of release v0.1.0 (image sha256:0b9e2f30…adac), client on the same host against the app's loopback port, so the client shares the 2 vCPUs.
+
+## Warm latency, memory and throughput (in process, one request at a time)
+
+| Configuration | Model | p50 ~512 | p95 ~512 | p50 ~2000 | p95 ~2000 | Natural docs/s | Peak RSS (MB) |
+|---|---|---:|---:|---:|---:|---:|---:|
+| crf | `crf-batches-01-02` | 0.0395 | 0.0617 ✅ | 0.1227 | 0.1446 ✅ | 23.619 ✅ | 276.2 ✅ |
+| *Target (documented host)* | | | ≤ 2.5 | | ≤ 8.0 | ≥ 0.25 | ≤ 3072 |
+
+Latencies are seconds per document, including preprocessing, windowing, inference and decoding; p95 over all documents and repeats in the bucket.
+
+## Cold start (fresh process)
+
+| Configuration | Imports (s) | Model load (s) | First extraction (s) | Total (s) |
+|---|---:|---:|---:|---:|
+| crf | 0.22 | 2.501 | 1.4 | 4.121 |
+
+## HTTP load test
+
+| Service | Requests | Concurrency | Success | p50 (s) | p95 (s) | p99 (s) | Throughput (req/s) |
+|---|---:|---:|---:|---:|---:|---:|---:|
+| crf | 1000 | 4 | 100.00% | 0.1403 | 0.6144 | 1.0167 | 18.6 |
+
+Target: ≥ 99.5% successful requests at the supported load.