[forhappy]

Cellar’s multi-Raft write path builds and publishes independent SQLite images for each cell and replica. This PR reduces avoidable image I/O and scheduling work while preserving quorum acknowledgement, committed-image proofs, bounded resources and RustFS recovery coverage. Main is integrated through 1455fb0; conflicts are resolved.

Sustained performance qualification remains incomplete. The overlapping bundle-read experiment is removed after two paired comparisons fail to demonstrate a qualified gain. Active runtime source returns to 5138559 and matches all 118 frozen inputs of control 0b07410. The experimental commit 3d3ba9e, image, correctness tests and raw measurements remain preserved. Rejection reflects insufficient performance evidence, not a claim that shared-host measurements prove a causal regression.

Retained changes

• Create immutable images exclusively, removing a placeholder and rename while retaining replay/digest checks, file and directory sync, capacity charges, marker ordering and quarantine.
• Seek past the archive round-robin cursor before evaluating eligibility. The 1,500-cell regression reduces next-cell eligibility checks from 1,202 to one while preserving cyclic fairness.
• Refill the bounded eight-frame control window after authenticated replies and physical write completion, preserving FIFO, deadlines, permits and joined cancellation.
• Reuse exact sealed leader previews and bounded encoded LTX cuts; preserve a joined Ready private workspace after known pre-mutation refusal. Mutation failures still require rebuild.
• Share obsolete-image deletion barriers, rotate expensive archive tails by size, prioritize committed SQL work fairly, reclaim checkpoints before blocked appends and cache admission accounting by revision.
• Integrate main’s two-cell archive bundles and bounded publishers. Keep the 16-frame / 1-MiB data window, two-cell archive batches, four retained archive cuts in the benchmark and 256-MiB admission budget.
• Record transport, publication, coverage and admission measurements; verify actual benchmark resource settings.

Latest comparison: 20 writers, control first

Two 180-second windows per build, each starting with a cold benchmark VM and fresh node volumes. Three pinned 1-vCPU / 3-GiB nodes, 500 leaders and 1,000 followers each, and 1,500 three-voter groups. Frozen source/image/binary identities, load AST and recorded RustFS/Python image identities match. There are no timed mutation retries or changes to durability/admission limits.

Build	Successful TPS, windows 1 / 2	Successful-request p99 ms	Timed overload refusals
Control	265.86 / 235.77	175.69 / 220.76	22,919 / 944
Rejected shared bundle reads	256.86 / 168.99	206.78 / 279.41	11,749 / 0

Mean successful TPS is 250.81 versus 212.92, a descriptive 15.11% decrease. Both complete repeat sets fail the zero-error gate at 200 and 500 ms. The candidate’s error-free second window alone does not qualify sustained performance.

Sharing is exercised: 7,097 / 4,180 joined reads, or 6.16% / 5.65% of bundle-read admissions. These counters include later failure/cancellation and do not count successful GETs or saved bytes. Retained-entry counters explain all control refusals and 11,748 of the candidate’s 11,749 first-window refusals; the remaining refusal is not attributed to that counter. Publication failures and rejected coverage updates remain zero.

Both builds verify 4,500 exact replica values and 1,500 archive positions, without reconciliation or mutation retries, then stop cleanly with no OOM. Peak node memory is 1.74 GB control / 1.48 GB candidate. Final archive checks take 5.94 / 0.64 seconds; these post-load checks do not establish steady-state drain capacity.

Other builds overlap 37/37 and 38/40 control samples versus 32/37 and 15/36 candidate samples. Host swap ranges from 7.53 to 7.86 GiB. These uncontrolled shared-host conditions prevent causal attribution. This task performs no builds, profiling or preservation during the timed cohorts.

The earlier 24-writer, candidate-first pair was also unqualified: mean successful TPS was descriptively +7.10%, but both first windows had refusals, and the error-free second window had 4.64% lower candidate TPS and 16.81% higher p99. The two pairs use different offered concurrency and are not identical-load reversed confirmation.

20-writer evidence (tikv-bundle-read-c20.json, retained locally) · 24-writer evidence (tikv-bundle-read.json, retained locally)

Environment and validation

The user authorized recreation of cellar-raft-perf after a separate Colima reset. The original 352-GiB data disk is reattached with its existing filesystem. All 414 pre-run volumes and 58 images, both frozen binary hashes, and the original mounts were verified. After this pair, all 422 volumes and 58 images remain, with no benchmark containers running. The eight-CPU/16-GiB VM retains the three measured containers’ one-CPU/3-GiB limits. The default profile and selected Docker context are unchanged. Inventory verification is not a full content audit of every historical volume.

Restoration validation: 518 library, 53 archive, 19 execution, 29 host and nine multi-group tests pass serially. Three ignored entries are crash-test child helpers exercised by their parents. Strict Cellar all-target/all-feature Clippy, formatting, layout, native dependency policy and docs checks pass. All 118 frozen control source inputs match. Local correctness does not change the failed performance gates; no complete CI claim is made for the new head.

Profile restoration (tikv-benchmark-profile-recreation.json, retained locally) · Preserved candidate local tests (tikv-bundle-read-local.json, retained locally)

TiKV comparison and remaining work

The comparison uses TiKV revision b3995dab28308f09cbb7c125bf32f789dbc203fe. Its async_io/write.rs::write_to_db shares storage-engine I/O and independently notifies each region Ready after persistence. Cellar’s shared Raft log does not remove independent SQLite preparation, replay and image publication. TiKV’s shared KV apply batch is a structural advantage, not a drop-in replacement for Cellar’s per-cell SQLite contract.

1. Find sustainable capacity with a fixed offered-rate sweep of the restored control. Require repeated zero-error windows, bounded archive lag and p99 below the selected bound. Reducing writer count alone has not prevented refusal bursts.
2. Reduce repeated archive downloads/replays and duplicate publication across replicas while preserving per-cell recovery proofs, publisher failover and memory bounds. Larger admission limits do not increase archive service capacity.
3. Reduce independent SQLite image preparation and SQL queueing. Delayed checkpoints require distinct applied/checkpointed progress, retained replay history, snapshot/GC rules and crash qualification.
4. Measure scheduling before changing it. TiKV region hibernation requires quorum agreement, wakeup and election handling; skipping Cellar ticks alone would be unsafe.

Earlier rejected experiments and their measurements remain documented, including admission waiting, independent archive completion, gathered control frames, singleton archive batches and ephemeral capture. No repeatable error-free throughput gain is claimed.

Detailed comparison and complete investigation

Raw benchmark JSON files under plans/multi-raft/evidence/ are excluded from this PR and retained locally. Existing evidence files on main are unchanged. New local evidence is ignored by Git; the performance and validation summaries remain in this description and the investigation document.