diff --git a/README.md b/README.md index 827272e..f10a5ec 100644 --- a/README.md +++ b/README.md @@ -159,7 +159,7 @@ item can be highlighted in the original advisory. | 3 · Neural models | Frozen leakage-safe splits, annotation batches (01–03, 06: model-drafted), DistilBERT and DeBERTa-v3 with span / role / group heads, experiment runner, CRF ablations and learning curve | 🟨 trained on model drafts (run 3) · DeBERTa ≈ CRF on held-out agreement, no clear winner | | 4 · Reliability | Calibration and review policy, formatting robustness (rules: 100%), challenge-set and error-catalogue tooling | 🟨 tooling ready · thresholds await labels | | 5 · Application | FastAPI API, review UI, SQLite (WAL) with append-only history, exports, Docker Compose stack, release manifests, tested rollback, CI image smoke test | ✅ built | -| 6 · Production | Benchmark harness, ONNX/INT8 export (FP32 parity exact), local proxy benchmarks and load tests, monitoring; target-host numbers and hosted release pending | 🟨 harness and proxy results | +| 6 · Production | Benchmark harness, ONNX/INT8 export (FP32 parity exact), local proxy benchmarks and load tests, monitoring; CRF v0.1.0 measured on the 2 vCPU / 4 GB target host | 🟨 CRF meets every serving target on the target host · transformers not yet measured there | ### Preliminary pilot results (dev split only) @@ -373,9 +373,23 @@ acceptance gate has been evaluated, because labels after the pilot are model dra [release notes](docs/release_notes/v0.1.0.md). Its rollback drill (rules `r1` → CRF `v0.1.0` → `r1`) passed on local Docker. -## Serving cost (local proxy) +## Serving cost -Measured on an 8 GB Apple M1 with inference limited to 2 threads. This is **not the target +**On the target host (2 vCPU / 4 GB x86-64, AWS EC2, 2026-11-09)**, the released CRF +(`crf-batches-01-02`, v0.1.0) meets every serving target: + +| Measure | CRF v0.1.0 | Target | +|---|---:|---:| +| Warm p95 latency, ~512-token advisory | 0.06 s | ≤ 2.5 s | +| Warm p95 latency, ~2,000-token advisory | 0.14 s | ≤ 8 s | +| Throughput, natural workload (in process) | 23.6 docs/s | ≥ 0.25 docs/s | +| Peak memory (in process) | 276 MB | ≤ 3 GB | +| HTTP load test, 1,000 requests at concurrency 4 (Compose stack) | 100% succeeded, p95 0.61 s, 18.6 req/s | ≥ 99.5% | + +Full report: [`reports/benchmarks/target-host-crf.md`](reports/benchmarks/target-host-crf.md). +The transformer models have not been measured on this host. + +**Local proxy for the transformers.** Measured on an 8 GB Apple M1 with inference limited to 2 threads. This is **not the target host**, and the transformer bundles have untrained heads, so latency and memory are real but accuracy is not measured. Details and caveats: [`docs/benchmarks.md`](docs/benchmarks.md). diff --git a/docs/benchmarks.md b/docs/benchmarks.md index af1550f..a7759b0 100644 --- a/docs/benchmarks.md +++ b/docs/benchmarks.md @@ -71,6 +71,29 @@ FP32 ONNX is numerically exact. INT8 moves the logits, and DeBERTa much more tha DistilBERT. With untrained heads the INT8 *predictions* are not meaningful, so whether INT8 is usable is decided only by the ≤ 1-point typed-F1 gate on a trained model. +## Results on the target host: CRF v0.1.0 + +Full tables: [`reports/benchmarks/target-host-crf.md`](../reports/benchmarks/target-host-crf.md) +(raw JSON and checkpoint next to it). +- **Host:** AWS EC2, 2 vCPU / 4 GiB x86-64 (Intel Xeon Platinum 8488C, 3.72 GB visible), Ubuntu + 24.04. The harness recognised it as the documented host. +- **Model:** the released v0.1.0 bundle `crf-batches-01-02`, measured on 2026-11-09. + +| Measure | Result | Target | +|---|---:|---:| +| Warm p95, ~512 tokens | 0.062 s | ≤ 2.5 s ✅ | +| Warm p95, ~2,000 tokens | 0.145 s | ≤ 8 s ✅ | +| Natural throughput, in process | 23.6 docs/s | ≥ 0.25 ✅ | +| Peak RSS | 276 MB | ≤ 3,072 MB ✅ | +| Cold start (imports, load, first extraction) | 4.1 s | not a target | +| Load test: 1,000 requests, concurrency 4 | 100% succeeded; p50 0.14 s, p95 0.61 s, p99 1.02 s; 18.6 req/s | ≥ 99.5% ✅ | + +- **How the load test ran:** against the Docker Compose stack of release v0.1.0, from a client + on the same host, so the client shared the 2 vCPUs. The service's latency is therefore, if + anything, overstated. +- **Not measured on this host:** the transformers. Measuring them needs a trained bundle, and + the INT8 ≤ 1-point gate still applies. + ## Results so far: local proxy (Apple M1, 2 inference threads) Full tables: [`reports/benchmarks/local-m1-2threads.md`](../reports/benchmarks/local-m1-2threads.md) diff --git a/reports/benchmarks/load-target-host-crf.json b/reports/benchmarks/load-target-host-crf.json new file mode 100644 index 0000000..6f02c5e --- /dev/null +++ b/reports/benchmarks/load-target-host-crf.json @@ -0,0 +1,19 @@ +{ + "requests": 1000, + "concurrency": 4, + "distinct_texts": 100, + "cache": "bypassed (unique raw text per request)", + "succeeded": 1000, + "success_rate": 1.0, + "status_counts": { + "200": 1000 + }, + "latency_s": { + "p50": 0.1403, + "p95": 0.6144, + "p99": 1.0167 + }, + "throughput_rps": 18.6, + "elapsed_s": 53.78, + "label": "crf" +} \ No newline at end of file diff --git a/reports/benchmarks/target-host-crf.json b/reports/benchmarks/target-host-crf.json new file mode 100644 index 0000000..c45613f --- /dev/null +++ b/reports/benchmarks/target-host-crf.json @@ -0,0 +1,80 @@ +{ + "environment": { + "measured_at": "2026-11-09T03:34:45+00:00", + "system": "Linux", + "machine": "x86_64", + "cpu_model": "Intel(R) Xeon(R) Platinum 8488C", + "logical_cpus": 2, + "cpu_limit": null, + "memory_gb": 3.72, + "in_container": false, + "python": "3.12.16", + "git_commit": "55e1778", + "matches_target_host": true + }, + "results": [ + { + "config": { + "kind": "crf", + "model_dir": "crf-batches-01-02", + "threads": null, + "batch_size": null, + "label": "crf", + "name": "crf" + }, + "model_id": "crf-batches-01-02", + "cold_start": { + "imports_s": 0.22, + "load_s": 2.501, + "first_extraction_s": 1.4, + "total_s": 4.121 + }, + "buckets": { + "natural": { + "n": 100, + "p50": 0.0231, + "p95": 0.1314, + "max": 0.3479, + "mean": 0.0423, + "docs_per_s": 23.619 + }, + "~2000": { + "n": 60, + "p50": 0.1227, + "p95": 0.1446, + "max": 0.1764, + "mean": 0.1195 + }, + "~512": { + "n": 60, + "p50": 0.0395, + "p95": 0.0617, + "max": 0.0892, + "mean": 0.0425 + } + }, + "peak_rss_mb": 276.2 + } + ], + "load": [ + { + "requests": 1000, + "concurrency": 4, + "distinct_texts": 100, + "cache": "bypassed (unique raw text per request)", + "succeeded": 1000, + "success_rate": 1.0, + "status_counts": { + "200": 1000 + }, + "latency_s": { + "p50": 0.1403, + "p95": 0.6144, + "p99": 1.0167 + }, + "throughput_rps": 18.6, + "elapsed_s": 53.78, + "label": "crf" + } + ] +} \ No newline at end of file diff --git a/reports/benchmarks/target-host-crf.md b/reports/benchmarks/target-host-crf.md new file mode 100644 index 0000000..13ef367 --- /dev/null +++ b/reports/benchmarks/target-host-crf.md @@ -0,0 +1,32 @@ +# CPU inference benchmark: target host, CRF v0.1.0 + +- Measured 2026-11-09T03:34:45+00:00 at commit `55e1778` +- Hardware: Intel(R) Xeon(R) Platinum 8488C (x86_64), 2 CPUs, 3.72 GB, Linux, Python 3.12.16 +- **This is the documented target host**: targets are judged below. +- Workload `workload-v1`: train and dev pools of splits/pool-v1.json; lengths in `distilbert/distilbert-base-cased` subword tokens. Natural-sample distribution: max 5772, p10 59, p50 234, p90 1775, p99 3112 +- AWS EC2, 2 vCPU / 4 GiB x86-64 (Intel Xeon Platinum 8488C), Ubuntu 24.04, measured 2026-11-09. +- Model: released v0.1.0 bundle crf-batches-01-02 (SHA-256 65677a74…d29a), in process. +- Load test: the Docker Compose stack of release v0.1.0 (image sha256:0b9e2f30…adac), client on the same host against the app's loopback port, so the client shares the 2 vCPUs. + +## Warm latency, memory and throughput (in process, one request at a time) + +| Configuration | Model | p50 ~512 | p95 ~512 | p50 ~2000 | p95 ~2000 | Natural docs/s | Peak RSS (MB) | +|---|---|---:|---:|---:|---:|---:|---:| +| crf | `crf-batches-01-02` | 0.0395 | 0.0617 ✅ | 0.1227 | 0.1446 ✅ | 23.619 ✅ | 276.2 ✅ | +| *Target (documented host)* | | | ≤ 2.5 | | ≤ 8.0 | ≥ 0.25 | ≤ 3072 | + +Latencies are seconds per document, including preprocessing, windowing, inference and decoding; p95 over all documents and repeats in the bucket. + +## Cold start (fresh process) + +| Configuration | Imports (s) | Model load (s) | First extraction (s) | Total (s) | +|---|---:|---:|---:|---:| +| crf | 0.22 | 2.501 | 1.4 | 4.121 | + +## HTTP load test + +| Service | Requests | Concurrency | Success | p50 (s) | p95 (s) | p99 (s) | Throughput (req/s) | +|---|---:|---:|---:|---:|---:|---:|---:| +| crf | 1000 | 4 | 100.00% | 0.1403 | 0.6144 | 1.0167 | 18.6 | + +Target: ≥ 99.5% successful requests at the supported load.