[core] Add standalone manifest sidecar format and utilities by leaves12138 · Pull Request #9845 · apache/paimon

leaves12138 · 2026-09-15T10:03:27Z

Purpose

Introduce a standalone manifest sidecar format and utilities for selecting physical Avro blocks using independent partition, row-ID and bucket coverage. This provides the block-selection building block; automatic integration with table writers, scans and cleanup remains separate work.

Define a four-byte magic, version, original Avro header, partition dictionary, physical block directory, independent coverage payloads and SHA-256 checksum. Counts, lengths and offsets use canonical unsigned varints. Manifest length and entry count are validated against the supplied ManifestFileMeta instead of duplicated in the sidecar.
Share the count-prefixed intsDeltaPayload through the streaming DeltaVarintCodec. Row-ID coverage keeps fixed-width minRowId and maxRowId followed by sorted interior endpoints. Bucket pairs are sorted and stored as paired bucket/total-bucket sequences; totals use ZigZag deltas to preserve decreases after rescaling.
Always generate partition coverage. Row-ID and bucket generation are controlled by method arguments; callers decide whether to build or read a sidecar.
Include validation, conservative fallback, sequential selected-block reads and optional bounded caching. Sidecar reads and selected-block reads use separate buffers.

Tests

Java 8: 55 targeted tests passed, including Checkstyle and Spotless checks.
Coverage includes golden format fixtures, 64-bit row-ID boundaries and gaps, decreasing bucket totals and rescaling, malformed/truncated varints, directory validation, unavailable dimensions, interruption behavior, real manifest rewrites, cache behavior and large block reads.
Independent review probes: 1,000 generated sidecars with 16,000 filter combinations matched an entry-level oracle. All tested truncations were rejected; 10,000 checksum-recomputed framing/payload mutations produced no unchecked parser failures.

mvn -B -ntp -nsu -pl paimon-core -am -DwildcardSuites=none -DfailIfNoTests=false -Dflink.forkCount=2 -Dtest=DeltaVarintCodecTest,ManifestSidecarTest,ManifestBlockIndexTest test

XiaoHongbo-Hope · 2026-09-15T11:25:32Z

Suggestion：Could we use .avro.index instead of .avro.sidecar ? Or do you want to distinguish it from data file indexes?

JingsongLi

We should probably remove maxBytes; it is always effective once the file is opened.
The row IDs, partition data, or bucket data within a block should be designed using a lightweight, compressed storage format, such as delta encoding combined with RLE. (Should be sorted too)
Four bytes are sufficient for the magic number.
manifestNameHash is too tightly coupled; I don't think it's necessary.
Partition data should always be enabled.

leaves12138 · 2026-09-15T12:10:01Z

Suggestion：Could we use .avro.index instead of .avro.sidecar ? Or do you want to distinguish it from data file indexes?

Yes, I want to name it sidecar, a special format for manifest file

leaves12138 · 2026-09-15T13:56:04Z

RLE increased the sidecar size in the tested sample because nearly all row-ID endpoint deltas formed runs of length one. We therefore removed RLE and use delta encoding with varints, which reduced the sidecar size by over 50% compared with the original fixed-width encoding.

JingsongLi · 2026-09-16T03:11:10Z

+    }
+
+    /** Read/write switches and optional payloads. Partition coverage is always enabled. */
+    public static final class Settings {


Remove this class, just use arguement.

JingsongLi · 2026-09-16T03:11:15Z

+
+    /** Read/write switches and optional payloads. Partition coverage is always enabled. */
+    public static final class Settings {
+        public final boolean write;


Remove this.

JingsongLi · 2026-09-16T03:11:23Z

+    /** Read/write switches and optional payloads. Partition coverage is always enabled. */
+    public static final class Settings {
+        public final boolean write;
+        public final boolean read;


Remove this.

JingsongLi · 2026-09-16T03:13:06Z

+```text
+magic : 4 bytes                         // ASCII PMSC
+formatVersion : int                    // 1
+manifestLength : long


Remove this.

JingsongLi · 2026-09-16T03:13:15Z

+magic : 4 bytes                         // ASCII PMSC
+formatVersion : int                    // 1
+manifestLength : long
+manifestEntryCount : long               // ADD + DELETE


Remove this.

JingsongLi · 2026-09-16T03:13:32Z

+formatVersion : int                    // 1
+manifestLength : long
+manifestEntryCount : long               // ADD + DELETE
+avroHeaderLength : int


JingsongLi · 2026-09-16T03:13:54Z

+manifestEntryCount : long               // ADD + DELETE
+avroHeaderLength : int
+avroHeader : bytes                      // original schema, codec and sync marker
+partitionCount : int


var int. All need var int use var int, except too big.

JingsongLi · 2026-09-16T03:15:11Z

+| --- | --- | --- |
+| Any | `0` | Unavailable; only the encoding byte is present. |
+| Partition | `1` | Count and delta/varint-compressed sorted unique dictionary IDs. |
+| Row ID | `1` | Interval count, minimum, span, and delta/varint-compressed interior endpoints. |


JingsongLi · 2026-09-16T03:18:39Z

+
+```text
+partitionPayload
+  partitionIdCount : int               // N > 0


You can introduce a intsDeltaPayload:

count

deltas[]

Partition and row Id and bucket all refer to it.

JingsongLi · 2026-09-16T03:19:03Z

+
+```text
+rowIdPayload
+  rangeCount : int                     // N > 0


minRowId

maxRowId

intsDeltaPayload

JingsongLi · 2026-09-16T03:20:48Z

+
+```text
+bucketPayload
+  pairCount : int                      // N > 0


sort by bucket

bucket intsDeltaPayload

totalBucket intsDeltaPayload

leaves12138 · 2026-09-16T06:02:26Z

We should probably remove maxBytes; it is always effective once the file is opened.

The row IDs, partition data, or bucket data within a block should be designed using a lightweight, compressed storage format, such as delta encoding combined with RLE. (Should be sorted too)

Four bytes are sufficient for the magic number.

manifestNameHash is too tightly coupled; I don't think it's necessary.

Partition data should always be enabled.

Done

JingsongLi · 2026-09-16T06:34:53Z

+  if bucketEncoding != 0:
+    bucketPayloadLength : varint
+    bucketPayload : bytes
+checksum : 32 bytes                     // SHA-256 of all preceding bytes


Use crc32c 4 bytes

I will use CRC32 to verify

JingsongLi · 2026-09-16T06:39:51Z

+
+```text
+partitionPayload
+  intsDeltaPayload                    // N > 0 dictionary IDs, base = 0


longsDeltaPayload

partitionPayload are ints, but row id payload are longs, I will comment it in manifest.md

JingsongLi · 2026-09-16T06:44:10Z

+    }
+
+    private static int readInt(ByteBuffer in) throws IOException {
+        long value = VarLengthIntUtils.decodeLong(in);


add decodeInt to VarLengthIntUtils

JingsongLi · 2026-09-16T08:24:37Z

+1

[core] Add standalone manifest sidecar format and utilities

38b932b

JingsongLi requested changes Sep 15, 2026

View reviewed changes

leaves12138 added 3 commits September 15, 2026 20:32

[core] Compress sidecar payloads and simplify the format

eecc11e

[docs] Remove redundant manifest sidecar hash explanation

2ac84f3

[core] Restore manifest sidecar read and write settings

e5cb4b8

leaves12138 marked this pull request as draft September 15, 2026 13:24

leaves12138 added 2 commits September 15, 2026 21:33

[core] Use fixed-width manifest sidecar payload prefixes

aedfb48

[core] Share delta-varint codec and remove sidecar RLE

3080860

leaves12138 marked this pull request as ready for review September 15, 2026 14:07

leaves12138 added 4 commits September 15, 2026 22:24

[core] Skip sidecar builds when writing is disabled

f46e60a

[core] Honor sidecar read settings before accessing metadata

84d94d3

[core] Separate sidecar and manifest block read buffers

f0ce997

[core] Test nine MiB of consecutive manifest blocks

5b8ffb5

leaves12138 requested a review from JingsongLi September 16, 2026 02:03

Fix typo

fd998cc

JingsongLi reviewed Sep 16, 2026

View reviewed changes

[core] Simplify manifest sidecar framing and delta payloads

36ea6b6

leaves12138 requested a review from JingsongLi September 16, 2026 06:02

JingsongLi reviewed Sep 16, 2026

View reviewed changes

Fix minus

e016a05

leaves12138 force-pushed the codex/manifest-sidecar-format branch from fdd01c5 to e016a05 Compare September 16, 2026 08:02

Fix minus

1cf280c

leaves12138 requested a review from JingsongLi September 16, 2026 08:08

JingsongLi merged commit c4bb875 into apache:master Sep 16, 2026
17 checks passed

This was referenced Sep 16, 2026

[core] Add manifest sidecars for partition, row-id and bucket pruning #9743

Draft

[core] Manage manifest extra file lifecycle #9886

Merged

[core] Generate sidecars for newly written manifests #9889

Merged

Uh oh!

Conversation

leaves12138 commented Sep 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Tests

Uh oh!

XiaoHongbo-Hope commented Sep 15, 2026

Uh oh!

JingsongLi left a comment

Choose a reason for hiding this comment

Uh oh!

leaves12138 commented Sep 15, 2026

Uh oh!

leaves12138 commented Sep 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

leaves12138 commented Sep 16, 2026

Uh oh!

JingsongLi Sep 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JingsongLi commented Sep 16, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

leaves12138 commented Sep 15, 2026 •

edited

Loading

leaves12138 commented Sep 15, 2026 •

edited

Loading

JingsongLi Sep 16, 2026 •

edited

Loading