CAS: opt-in content-defined chunking for large part files - #2370
DimensionWieldr wants to merge 3 commits into
Conversation
A CAS blob is one whole part file, so two files deduplicate only when they are byte-for-byte identical. A merge or mutation that re-emits most of its input bytes therefore publishes entirely new blobs, which is the storage cost measured in Altinity#2314. Add an opt-in `Chunked` manifest-entry placement: a large part file is split at boundaries chosen by a rolling hash over its content, and each chunk becomes an ordinary blob. Because the boundaries follow the content, an unchanged run lands in the same chunk regardless of what moved around it, so a rewrite pays only for the chunks that actually changed. A chunk is an ordinary blob in every respect -- same key space, envelope, freshness sidecar and GC treatment -- so this adds a manifest placement, not an object kind. No `Tree` object kind is revived and no pack files are introduced. Measured on the 16 x 42 MB CODEC(LZ4) merge from Altinity#2314, chunking off vs on: bytes written by OPTIMIZE FINAL 1,181,953,697 -> 170,808,185 (-85.5%), pool size 1,856,801,807 -> 845,740,839 (-54.5%), CAS/restic ratio 2.61x -> 1.18x. fsck reports 475 blob references resolving to 210 distinct blobs. The cost is request amplification: blob PUTs per insert 4 -> 20, object count 197 -> 489. Off by default. It only pays off when a rewrite re-emits byte-identical compressed frames (parts that concatenate rather than interleave); an interleaving merge that re-sorts rows shares close to nothing. Details: - `Primitives/CasContentChunker`: Gear rolling hash (FastCDC shape), gear table generated by constexpr splitmix64 from a pinned seed so every build cuts identically. `feed` reports the boundary as an explicit flag rather than implying it from the consumed count, so cut positions never depend on the caller's slice size -- otherwise two writers of identical bytes would disagree on boundaries and deduplicate nothing. - Manifest format: `EntryPlacement::Chunked` plus a per-entry `ChunkRef` list. Chunk records ride their own lines rather than a nested array, because `line_cap` is 64 KiB and a multi-GB column file would breach it as one line. The chunk count uses the `!nchunks` critical-key spelling, so a build without this placement reports UNKNOWN_FORMAT_VERSION instead of corruption -- no pool-wide `G_BUILD` bump, which would have restamped every object class. - Read path: `ReadBufferFromRemoteFSGather` takes one uniform `object_payload_offset` (the envelope length is pool-wide) and hands the creator the physical object size, so a wrapped cache layer still sees the real object length. `StoredObject::bytes_size` keeps meaning the logical contribution, leaving all existing offset arithmetic untouched. - Write path: `CaContentWriteBuffer` gains a chunked local-staging mode; the two existing constructors are unchanged. S3 staging stays unchunked, since its promote is a verbatim server-side copy of one [header][payload] object. A stream yielding fewer than two chunks is published as an ordinary whole-file blob, so small, empty and sub-floor files keep today's exact bytes and keys. - GC, fsck and relink adoption iterate an entry's referenced blobs through the new `forEachEntryBlobRef`, so a chunked entry emits one source edge per chunk. The edge key is already `(BlobRef, source_id)`, so no GC model change. Tests: 11 chunker tests (determinism, slice-size independence, boundary re-sync after a byte shift, max_bytes ceiling on repetitive input, pinned cut positions), 10 manifest tests (round trip, chunk-list digest coverage, inconsistent-total and chunk-count-cap rejection), and 04344_cas_chunking.sh, which asserts a file really was split and then compares scans, point reads, bounded range reads, merge, mutation and post-GC reads against a plain MergeTree oracle. Co-authored-by: Cursor <cursoragent@cursor.com>
|
Summarizing the discussion on #2314: Append-only, time-ordered tables are a common ClickHouse shape: logs, metrics, events; That is where this shines, and many use cases match it. On that workload (16 × ~42 MB Wide,
The close-comment caveat still holds for interleaved merges (overlapping keys, upserts): those re-pack frames throughout, CDC shares close to nothing, and you still pay extra requests. Whole-file hashing already covers byte-identical files. Chunking is for the file that changed but whose interior compressed runs did not — which is exactly the concatenative merge above. Off by default; disk setting |
|
I am skeptical, and I do not expect this to be merged. It buys a small byte saving with more objects, more requests, a second pass over every written byte, and a third manifest placement. CAS needs the opposite right now: fewer objects, fewer requests, less code, and the existing parts working reliably. Better to say this now than after more of your time goes in. Why I think the benchmark does not transfer:
If you want to check it yourself, two runs are enough:
I would be glad to be wrong. Either way, please write the result up here or in #2314. A clear negative result with data is a real outcome and will save the next person the same experiment. |
|
Review follow-up: correctness coverage, operator docs, and an operational envelope. Raw tables: https://gist.github.com/DimensionWieldr/03f8e29af8b07bc46de4463b0d47434a (one-shot, not CI). The 16 × 42 MB headline remains the issue #2314 harness. Where the review notes landed
Not in this round: soak / alter-stress, mixed-package N vs N−1, a
16 × 42 MB concatenative LZ4 (unchanged headline)
Codec / chunk-size / larger parts (4 concatenative Wide parts unless noted)Production floors 1 / 4 / 16 MiB.
Codec is not the discriminator — LZ4 / ZSTD / NONE are the same shape. Coarser chunks: fewer objects, fewer avoided PUTs, still below the unchunked write. Interleave, ReplacingMergeTree, mutations
Cold/warm cache, concurrent readers, high object count
Cold then warm is GET count, not an SLA (wall clock was 821 ms then 1560 ms). Four concurrent readers during Alignment check (re: @filimonov)Ran (1):
(2) A related data point: the 4 × 4096-row Operator facts from this round are in |
|
I largely agree that the initial tested use case is too specific. I narrowed down the criteria for chunking to actually be useful:
Miss any one of those and you pay extra objects for almost no reuse. Most tables use at least one of: variable-length strings, irregular insert sizes, and overlapping keys. That's the case where this feature costs requests and saves almost nothing. The 85% figure is a lab merge I constructed to keep compressed frames stable. I have not shown a real dataset that behaves that way, which is going to be my next test. |
IRL those are never true. You're taking wrong assumptions as your foundation. Real data never follow the rules. Insert come in parallel, out of order, different batch sizes, every row have different size etc. Also - you need at least 5-10 gigabytes (better few dozens) of data to compare something in clickhouse. Before that size everything is 'almost instant'. |
|
ClickBench
The lab merge was 6.9× fewer write bytes and 2.2× smaller pool. That saving does not appear here. Looks like with a real-world data set, there is next to no benefit, and actually more detriment because of the increased PUTs. @filimonov I agree that with the current state of how chunking is handled, it is not really useful. Thanks for helping review this experiment! |
|
Tried another dataset just to see. Public Bitcoin transactions from AWS https://registry.opendata.aws/aws-public-blockchain/ (one week, 4.3 million rows). Same test as hits: uneven insert sizes, merges left running. Rows are stored in block order, which is the case where chunking was supposed to help. After leftover files were cleaned up:
The final merge reused a small number of chunks (about 1,500 skipped uploads vs none without chunking), but the amount of data written did not drop. Large files were split into ~5 MB pieces, so we paid for more objects and more uploads to store the same 3.48 GB. Chunking looks at ClickHouse’s own compressed files, not the original parquet on S3. Loading the dump rewrites the data, so similar-looking files in the public bucket do not get reused. Bitcoin scripts are a different length on almost every row. A merge compresses that data again, and the result is new bytes. Chunking can only skip an upload when those bytes come out identical. Same conclusion as hits: no saving on data written, and more uploads. The lab merge that wrote 6.9× less and stored 2.2× less only happens in the small case where a merge reproduces the same compressed bytes. |
Post-compression CDC never shared across recompress. MergeTree now cuts uncompressed column bytes into ClickHouse compressed blocks so concatenative background merges can re-reference the same CAS blobs. Co-authored-by: Cursor <cursoragent@cursor.com>
|
Grok summary of commit 2af4bc1: Chunking used to run FastCDC on already-compressed It now cuts uncompressed column bytes. Each window is one ClickHouse compressed block, and CAS stores that block. A merge that re-emits the same uncompressed run can reuse those blobs. Results (chunking off vs on)Bitcoin
Why we cannot improve both storage and PUTsChunking only helps when two live parts contain the same uncompressed runs. That is the −22% after background merges: insert parts and merge outputs still overlap, so some blobs are shared.
So a unique-storage win and fewer PUTs together are not feasible. The 22% only exists while leftover parts are still around, so it makes it hard to justify the PUT costs. |
|
What is the problem you trying to solve here? By any cost show that we can reuse some data blocks between parts "before and after the merge" with significant complexity explosion, data fragmentation etc? Do you know that original parts get removed 8 minutes after the merge? So where is the win? It would be good to discuss such work & ideas before burning tokens on that :) |
Just experimenting. Not actually pushing for any big changes. The win at this point is really just confirming that the original full-file/blob implementation was the correct decision. It has also helped me to learn a lot about CAS as a junior engineer, so there's that too. |
2 participants
Addresses #2314: whole-file CAS republishes an entire part file on merge even when most of the compressed bytes already live in the pool.
Summary
Chunkedmanifest placement. Large part files are split by a rolling hash (FastCDC / Gear) into ordinary CAS blobs — same key space, envelope,.meta, and GC. NoTreekind, no pack files.cas_chunking_enabled). Disk setting, captured at mount; changing it needs a restart and only affects new writes. Existing chunked parts stay readable with the flag off.Where it shines (the common ClickHouse shape)
Append-only, time-ordered tables — logs, metrics, events;
PARTITION BYday/month;ORDER BYstarting with time; batches landing roughly in time order.Part ranges barely overlap, so a merge is close to concatenation. The uncompressed granules going into LZ4 are the same as in the inputs; LZ4 of the same bytes is the same bytes; CDC reuses those runs.
That is the measured case, and many production use cases match it.
CDC still does not help when a merge weaves overlapping keys (
ORDER BY user_idwith concurrent writers,ReplacingMergeTreeupserts). Those re-pack compression frames throughout; chunking then costs extraPUTs for no byte win, so in this case the setting should not be turned on.Gains on that use case
Same protocol as #2314, finding 5, at the larger size: 16 Wide parts of ~42 MB,
CODEC(LZ4), uniquesipHashpayload, restic after every insert and afterOPTIMIZE FINAL, then CAS GC +restic forget --keep-last 1 --prune.INSERTsOPTIMIZEOPTIMIZE FINALS3 write bytesWith chunking, CAS is ~2.2× more storage-efficient than whole-file addressing on this merge (1.86 GB → 846 MB) and closes the 2.61× restic gap to 1.18×.
SYSTEM CAS FSCK: 475 blob refs → 210 distinct blobs, 0 dangling.Object storage bills the write.
OPTIMIZE FINALwas putting 1.18 GB of objects that were already in the bucket as the 16 input parts. Chunking puts 171 MB instead. Those PUT bytes are charged whether or not GC would later drop the duplicate whole files. Background merges keep parent and child parts live together forold_parts_lifetime, so that write amplification is the steady cost, not onlyOPTIMIZE FINAL.The insert phase is a wash on unique data (nothing to share) and pays more requests (
d_cas_blob_putper insert 4 → 20; objects 197 → 489). Raisecas_chunk_min_bytes/cas_chunk_avg_bytesto trade granularity for fewer requests.Test plan
04344_cas_chunking— split is real; scans, point reads, range reads, merge, mutation, post-GC vs MergeTree oraclecas_chunking_enabledis set and the server restartsworkload-merge-lz4-largewith the flag on vs off if reviewing storage numbers