[forhappy]

This PR advances MR-13 operations and MR-14 performance for native multi-Raft cells. It adds production startup and bounded operator observations, improves multi-group scheduling and archive work, and supplies executable workload and recovery evidence. Both milestones remain in progress. Startup routes remain private until all local groups and the archive owner are ready. Current performance work addresses intermittent observation stalls at 1,500-cell density; it does not claim sustained capacity or a speedup.

Runtime and operations

Production CLI and embedded startup share validation, require persistent Raft storage, and bind the compiled application before ingress. Shutdown seals admission, attempts bounded leadership handoff and joins physical owners. Durable management receipts report actual closure, unresolved commands and known archive lag. Resource views expose role, readiness, lag, SQL, readers, images, connections and controller reservations. Bounded receipt maintenance preserves retry deadlines, latest receipts and non-reusable client floors.

Shared persistence batches retain per-group Ready ordering and exact sync receipts. A large FIFO waiter cannot be bypassed indefinitely; groups waiting for persistence leave the runnable queue until the matching completion. Background SQL admission and shutdown handoff retain already-budgeted work within their original bounds.

The control sender pipelines at most eight already-admitted FIFO frames on its existing authenticated connection. Reads and writes share the original deadline; partial failures retain verified earlier admissions and never replay unknown frames. Route revocation and physical permits survive cancellation. Data windows remain one frame.

Archive work coalesces at most two same-namespace cuts into an immutable 8-MiB-bounded bundle, with independent per-cell verification and conditional publication. Recovery validates every intermediate LTX state in one private image and syncs the final image once. Committed images and accepted archive coverage wake waiting work through the existing authority and proof checks. The marker-1 /bundles-1 cutover requires fresh compatible experimental roots.

Performance and recovery tooling

Native Raft, Fleet and bucket adapters share a noncommutative SQL workload and fixed open-loop arrivals. Ledgers retain every offered operation, rejection, timeout and ambiguous outcome. Independent analyzers check histories and every assigned replica/archive or owner route. Comparisons refuse ratios when durability, read, workload, hardware or executable contracts differ.

Three-to-five-node Raft runs verify placement and executable identity, 1-vCPU/3-GiB/no-swap limits, separate RustFS resources, cgroup/PID observations and fixed-cardinality metrics. New control/data timers separate sender queue residence, exchange windows and receiver runtime admission. TLS tests distinguish held replies from queued work and preserve failure/cancellation outcomes. The strict analyzer requires the new vocabulary; older evidence uses its pinned analyzer. These timings enable diagnosis; they do not establish the density slowdown's cause or a speedup.

Read-only fault probes now retain bounded client connect/send/response timings and server status/replication timings. Missing records and failed HTTP reads remain failures; diagnostics cannot establish quorum or acknowledged reads.

Bounded slow-follower and object-store pauses retain exact container ownership and inspect ambiguous control responses. Quorum, S3, Fleet and bucket keep their own acknowledgement guarantees. The explicit long-outage recovery scenario quarantines uncertain commands, then retries and confirms their original identities and deadlines after measurement. Renewed observation tokens cannot disguise changes to durable receipt fields.

The bucket-only all-owner restart drill requires joined expired-lease fences before one start operation targets the same immutable containers and volumes. Fresh lease generations, executable identity and health are verified. Pre-retry reads must preserve all acknowledged state; recovery cannot repair acknowledged-prefix loss. New writes and final reads prove resumed service. Missing old-process counters are explicit and prevent performance comparison.

Measured limits and evidence

• Latest client-timed healthy attempt, source d0eb310: the unchanged five-node/1,500-cell run fails cycle 1's cell-20 probe. Connect/send takes 0.127 ms; header waiting times out after 102.226 ms. The matching server status stage takes 111.883 ms and replication inspection 0.550 ms. All eleven early probes correlate, but later the sampler stops with macOS EADDRNOTAVAIL. Raw traffic retains 235 completions, 15 ambiguous commands, 11 failures and 939 rejected arrivals. Final replica/archive checks never run. The extracted capsule remains independently rejected as unfinished. The status delay and client socket refusal need separate regressions; this is not capacity or performance qualification.

• Startup fix and fresh density run, source f96a94b: five 1-vCPU/3-GiB/no-swap nodes start all 1,500 cells and verify all 4,500 replicas and archive coverage. All 1,200 offered operations complete without errors or rejections; overall p99 is 123.939 ms, command p99 149.962 ms and query p99 38.711 ms. Traffic and memory objectives pass for this one short run with thirty active cells and ten offered operations/second. This does not establish saturation, steady state or a causal cross-revision speedup.

• Repeated density controls: two healthy runs verify 4,500 replicas/archive coverage at 95.426/79.254 ms p99. The intervening paused-node attempt times out during probe cycle 91 and remains unqualified.

• Probe-stage evidence, source 17fd0b1: the same timeout occurs without a node pause. One request has no handler record while the other 413 correlate. No later probes run, so eviction does not explain it. A separate pause run verifies all replicas/archive coverage and 116 cycles, but misses traffic targets at 304.107 ms p99 with 24 rejected arrivals. These findings narrow the investigation to before recorded handler entry, without proving the cause.

• Fenced-owner restart: two runs at aaa83f8, with a twelve-second RustFS pause and unchanged ten-second lease. Each preserves 36 pre-retry owner reads, resolves eight original identities, commits twelve new writes and verifies 36 final reads. Recovery bounds are 8.22 and 7.69 seconds. Each original measurement retains 32/200 completions, eight ambiguous commands and ten failed queries. This is retained-volume recovery, not cold/disaster restore or performance qualification.

• Store-outage matrix: all eight histories reconcile. Healthy controls and the Quorum outage meet short-run traffic targets; S3, Fleet and bucket outages miss them. Cross-adapter guarantees remain different.

• Five-node density: 150/500/1,500-cell histories reconcile, but 500/1,500-cell and repeated 150-cell controls miss latency/throughput targets. Queues approach 4,000 frames per node. Shared-host pressure prevents causal attribution.

• Ambiguous recovery retains successful Raft S3/bucket receipt checks and rejected attempts, including mixed live/fenced bucket owners. The later all-owner scenario does not retroactively qualify that failure.

Raw histories, source/image pins, hashes and failed attempts remain in the evidence. Exact experiment containers are stopped, only inspected empty owned networks are removed, and volumes remain. Only the separate cellar-multi-raft profile was controlled; it is stopped. No Amazon S3, independent-zone or sustained capacity claim is made.

Validation

• Read-accounting fixture correction, source 4dcaf1d: fresh free-threaded CI exposed a timing assumption in the read-timeout test. A delayed real fsync deterministically reproduces one failed read instead of three, because the next arrivals correctly see the cell busy. Controlled clock/executor completion now isolates accounting while retaining three timeout assertions, zero ambiguity and unchanged state. Separate real-thread admission tests remain. The reproducer, 201 performance tests (one opt-in Docker skip), documentation/plan checks and all 17 plan tests pass. No production runner, deadline or SLO change. Fresh-head CI is still required.

• Merge and diagnostic validation: merged origin/main at 64a4562. All 127 native host, membership, multigroup, operations and snapshot cases pass at merge 1209e0d; four directly ignored helpers execute through their parent crash tests. Upstream per-cell callback accounting deliberately exercises a recovering node leading a separate healthy cell. After combining the shared Blob/observer helpers, all six focused snapshot cases and five-target all-feature Clippy pass at b0bc2ff. Format/layout, dependency/docs, workflow parsing, 201 performance Python tests (one opt-in skip), Python qualification guards and 17 plan tests pass. Both Clippy merge failures and their correction remain in the capsule. This supersedes the narrower local snapshot fixture placement while preserving its original source-pinned evidence.

• Previous head 70c5695 failed the free-threaded snapshot fixture job; the ordinary SDK job exceeded its 90-minute workflow limit. Fresh CI must validate this resolved head. Upstream main's expanded workflow uses 120 minutes; individual test deadlines and performance SLOs remain unchanged.

• Startup correction at f96a94b: 97 Raft unit cases and 125 native host, membership, multigroup, operations and snapshot cases pass. Four ignored entries are child-process fixtures invoked by passing parent crash tests. A real-TLS regression admits three frames then overloads thirteen before the fix; afterward all sixteen receive UnknownGroup, both groups attach and a subsequent heartbeat is accepted. All-target/all-feature Cellar Clippy, formatting/layout, dependency/docs checks, plan validation and all 17 plan tests pass. The physical capsule is independently extracted and re-analyzed; original compiler diagnostics and the failing-before output remain retained.

• Transport timing evidence, source 87ed5d6: all 96 Raft unit cases, 20 native read/replication cases and 20 native operations cases pass. Real TLS tests cover held/lost/reordered replies, revocation, deadlines, cancellation and joined resource release; actual native traffic exercises both lanes. All 192 Python cases pass with one existing opt-in Docker skip. All-target/all-feature Cellar Clippy, formatting/layout, dependency/documentation checks and all 17 plan tests pass. The diagnostic change itself did not claim a physical speedup; the subsequent source-pinned density attempts are listed above.

• Fenced-owner recovery at aaa83f8: 192 Python cases pass with one existing opt-in Docker skip, including eight restart cases, plus all five native baseline cases and focused Clippy. Two physical three-node restart drills pass independent analysis and retain their original failed traffic objectives. This validation predates the transport instrumentation and keeps its separate source pin.

• Publication documentation and all 17 plan tests pass at 3ecd5bc.

Earlier native and physical checks retain their individual source pins and are not pooled into current counts. Full current-head CI must pass separately; previously retained concurrent qualification findings remain open.

Remaining priorities

1. Resolve the healthy observation stall and establish repeatable density/tail latency using the unchanged five-node workload. Then complete sustained S3, equivalent-contract comparisons, instrumentation overhead and device-I/O accounting. No single passing run closes MR-14.
2. Complete MR-13 authenticated management, receipt qualification, platform-enforced fencing and disaster restore; close retained intermittent findings.
3. Complete MR-15 concurrent histories, failure/rollback drills, soak and the supported deployment envelope. Real S3 and independent failure domains require separate evidence.
4. Defer MR-16 scale and MR-17 federation qualification until those gates pass.

Q1 remains six processes and 10,000 three-voter cells: 5,000 average replicas per process versus the unchanged 4,096 bound. It exceeds the agreed five-node local scope. No raised limit, relaxed deadline, weakened durability proof or reduced SLO substitutes for qualification.