[linick-kang]

Description

This PR adds t4pack, a container that stores all .pcd.bin frames of a channel in one zstd-compressed file, data/<CHANNEL>.pack. It also adds a t4pack CLI that packs, verifies and (optionally) deletes the loose frames.

On real T4 LiDAR data at zstd level 19, the pack is ~2.45x smaller than the loose .pcd.bin files (2.34x-2.54x per DB). Measured on a 100-scene pilot; about 4,500 scenes have been packed with the same format so far.

Loading needs no change in user code: if data/LIDAR_CONCAT/00000.pcd.bin is missing but data/LIDAR_CONCAT.pack exists, LidarPointCloud.from_file() and SegmentationPointCloud.from_file() read the frame from the pack. sample_data.filename stays the same.

Format (t4pack v1)

magic b'T4PACK\x00\x01'
blob_0 .. blob_{N-1}            zstd(column_0 bytes | column_1 bytes | ...)
zstd(JSON index)                name, offset, size, n_points, dtypes per frame
<QQ index_offset index_size>    little-endian
magic

• Each frame is compressed independently, so reading one frame decodes only that frame.
• Columns are stored column-major. Float32 columns are byte-shuffled (f4s), so zstd can match the sign/exponent bytes; intensity/ring are stored as u1/i1.
• If the source values don't fit u1/i1 (e.g. non-integer intensity or -0.0), the channel falls back to all-f4s columns. Either way, every frame is verified bit-exact against its source before the pack is put in place.
• The format is byte-compatible with existing t4pack v1 readers and with packs already produced by them.
• Frames are indexed by their real file name, so non-contiguous names like 0.pcd.bin, 10.pcd.bin work correctly.

Changes

• t4_devkit/common/t4pack.py (new)
  • Format: encode_frame, decode_frame, write_pack, and PackReader, which supports read_frame and read_frame_by_name.
  • pack_directory: pack, verify, then atomically rename into place.
  • verify_pack: compares frames that still have a .pcd.bin bit-exact, and checks that frames whose .pcd.bin was deleted can be decoded.
  • delete_loose_frames: deletes .pcd.bin only after verifying them against the pack, and deletes only the frames it verified.
• t4_devkit/dataclass/pointcloud.py: .pcd.bin loading falls back to the pack when the file does not exist. Existing behavior is unchanged when the file exists.
• t4_devkit/sanity/reference/ref201.py: REF201 now passes if a .pcd.bin frame exists as a file or in its pack. Other file types still have to exist on disk.
• t4_devkit/cli/pack.py (new): the t4pack command.
  • t4pack pack <DATA_ROOT> [-rv REV] [-l LEVEL] [--overwrite] [--delete-loose]
  • t4pack verify <DATA_ROOT> [-rv REV]
• pyproject.toml: adds the zstandard>=0.23.0 dependency and the t4pack script.
• Refactor: the default point width 5 (x, y, z, intensity, ring) is now a named constant, DEFAULT_NUM_PTS_FEATS in t4_devkit/common/t4pack.py. It replaces the bare literal in the existing PointCloudMetainfo default and both from_file loaders in pointcloud.py, and is used by the new pack functions and CLI. No behavior change.
• Docs: docs/cli/t4pack.md and a new "Packed Pointcloud" section in docs/schema/data.md. The REF201 description is updated in docs/schema/requirement.md.

Safety

The loose .pcd.bin files are never deleted unless --delete-loose is passed.

• Verified before deletion. Loose frames are deleted only after every one of them matches the pack bit-exact. If anything differs, nothing in that channel is deleted.
• No frames lost on overwrite. --overwrite is refused if the existing pack contains frames whose .pcd.bin has already been deleted, e.g. after an interrupted deletion.
• Safe with concurrent workers.
  • Each process writes its own uniquely named temp file and renames it into place atomically.
  • Concurrent deletes of the same channel tolerate each other.
  • Stress-tested with 3 packers (--overwrite) and 3 deleters on the same channel at the same time: 0 frames lost in 30 trials, and no temp files left behind.
• Readers are unaffected by a pack being replaced. An open reader keeps reading the old file.
• Packs keep normal file permissions (umask), so they stay readable by other users on shared storage.

How to review / test

uv run pytest tests
t4pack pack <DATA_ROOT> --level 19      # keeps .pcd.bin
t4pack verify <DATA_ROOT>
t4pack pack <DATA_ROOT> --delete-loose  # verify, then delete .pcd.bin
t4sanity <DATA_ROOT>                     # REF201 passes with packed frames

• New tests cover:
  • format round-trip, including empty frames, NaN and -0.0
  • the lossless fallback
  • deletion and its refusals
  • concurrent pack/delete with 4 processes
  • the loader fallback, REF201 and the CLI
• Checked against a real pack (178 frames): frames are read bit-exact, and re-encoding at level 19 reproduces the stored blobs byte for byte.

Notes / known limitations

• Old packs with non-padded file names. Packs written by the original script stored synthetic names (00000.pcd.bin, ...). For scenes whose files are not named 00000..N-1 (e.g. 0.pcd.bin, -1-01.pcd.bin), the old packs don't match the real names. In that case --delete-loose refuses to delete and verify reports mismatches, instead of returning wrong frames. Repack these scenes with t4pack pack --overwrite before deleting.
• Where num_pts_feats comes from. It is read from data/<CHANNEL>_INFO/*.json (default 5).
• Leftover temp files. A process killed with SIGKILL can leave a *.<pid>.<random>.tmp file next to the pack; it has to be removed manually. Normal exits and Ctrl-C clean up.
• ruff format --check reports 3 test files that are already unformatted on main and are not touched by this PR.