feat(dataclass): add t4pack for pointcloud compression

docs/cli/index.md

@@ -4,3 +4,4 @@ Following command line tools are supported:
 
 - [`t4viz`: Visualize T4 dataset features](./t4viz.md)
 - [`t4sanity`: Sanity checker of T4 dataset](./t4sanity.md)
+- [`t4pack`: Pack pointcloud frames into a single compressed file](./t4pack.md)

docs/cli/t4pack.md

@@ -0,0 +1,93 @@
+`t4pack` packs the `.pcd.bin` pointcloud frames of each sensor channel into a single compressed file, `data/<CHANNEL>.pack`.
+See [Packed Pointcloud](../schema/data.md#packed-pointcloud) for the format.
+
+```shell
+$ t4pack -h
+
+ Usage: t4pack [OPTIONS] COMMAND [ARGS]...
+
+╭─ Options ────────────────────────────────────────────────────────────────────────────────────────────────────────╮
+│ --version             -v        Show the application version and exit.                                           │
+│ --install-completion            Install completion for the current shell.                                        │
+│ --show-completion               Show completion for the current shell, to copy it or customize the installation. │
+│ --help                -h        Show this message and exit.                                                      │
+╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
+╭─ Commands ───────────────────────────────────────────────────────────────────────────────────────────────────────╮
+│ pack     Pack .pcd.bin frames of each channel into '<CHANNEL>.pack'.                                             │
+│ verify   Verify that each '<CHANNEL>.pack' matches its .pcd.bin frames if they exist.                            │
+╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
+```
+
+## Shell Completion
+
+Run the following command to install completion, and reload shell.
+
+```{ .shell .copy }
+t4pack --install-completion
+```
+
+## Usages
+
+### Pack Pointclouds
+
+```shell
+$ t4pack pack -h
+
+ Usage: t4pack pack [OPTIONS] DATA_ROOT
+
+ Pack .pcd.bin frames of each channel into '<CHANNEL>.pack'.
+
+╭─ Arguments ──────────────────────────────────────────────────────────────────────────────────────────────────────╮
+│ *    data_root      TEXT  Root directory path to the dataset. [required]                                         │
+╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
+╭─ Options ────────────────────────────────────────────────────────────────────────────────────────────────────────╮
+│ --revision      -rv      TEXT     Specify if you want to pack the specific version.                              │
+│ --level         -l       INTEGER  zstd compression level. [default: 19]                                          │
+│ --overwrite                       Overwrite existing packs.                                                      │
+│ --delete-loose                    Delete .pcd.bin frames after verifying them against the pack.                  │
+│ --help          -h                Show this message and exit.                                                    │
+╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
+```
+
+For example, the following command packs `data/LIDAR_CONCAT/*.pcd.bin` into `data/LIDAR_CONCAT.pack`:
+
+```shell
+t4pack pack <DATA_ROOT>
+```
+
+Every frame is verified bit-exact against its source before the pack is put in place, so an existing pack is always complete.
+If the source values do not fit the compact `uint8`/`int8` columns, e.g. non-integer intensity, every column is stored as `float32` instead.
+A channel that is already packed is skipped unless `--overwrite` is specified.
+If a channel fails, it is reported as `FAILED` and the other channels are still processed, then the command exits with `1`.
+
+The source `.pcd.bin` files are kept by default. With `--delete-loose`, they are deleted after every frame is verified against the pack again, which also applies to a channel that is already packed:
+
+```shell
+t4pack pack <DATA_ROOT> --delete-loose
+```
+
+Nothing is deleted from a channel if any of its frames is missing from the pack or differs from it, and files other than `.pcd.bin` are kept.
+
+`t4pack pack` can be run by multiple workers at the same time, even on the same dataset: each worker writes and verifies its own temporary file, and a verified pack is put in place atomically.
+`--overwrite` is refused if the existing pack contains frames whose `.pcd.bin` has already been deleted, so those frames are never lost.
+
+### Verify Packs
+
+```shell
+$ t4pack verify -h
+
+ Usage: t4pack verify [OPTIONS] DATA_ROOT
+
+ Verify that each '<CHANNEL>.pack' matches its .pcd.bin frames if they exist.
+
+╭─ Arguments ──────────────────────────────────────────────────────────────────────────────────────────────────────╮
+│ *    data_root      TEXT  Root directory path to the dataset. [required]                                         │
+╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
+╭─ Options ────────────────────────────────────────────────────────────────────────────────────────────────────────╮
+│ --revision  -rv      TEXT  Specify if you want to verify the specific version.                                   │
+│ --help      -h             Show this message and exit.                                                           │
+╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
+```
+
+Each frame whose `.pcd.bin` still exists is compared bit-exact with the pack, and each frame whose `.pcd.bin` has been deleted is checked to be decodable.
+The command exits with `1` if any frame is missing from the pack, differs from its source or is broken.

docs/schema/data.md

@@ -77,6 +77,43 @@ Example pointcloud metainfo JSON:
 }
 ```
 
+### Packed Pointcloud
+
+The `.pcd.bin` frames of a channel can be packed into a single file next to the directory, `data/<CHANNEL>.pack`, with [`t4pack`](../cli/t4pack.md):
+
+```shell
+data/
+├── LIDAR_CONCAT
+│   ├── <FRAME_ID>.pcd.bin
+│   ...
+└── LIDAR_CONCAT.pack
+```
+
+When `<FRAME_ID>.pcd.bin` does not exist but `<CHANNEL>.pack` does, `LidarPointCloud.from_file()` and `SegmentationPointCloud.from_file()` read the frame from the pack by its file name, so `sample_data.filename` does not need to change.
+Therefore, the `.pcd.bin` files can be deleted after packing (`t4pack pack --delete-loose`), and the sanity check `REF201` accepts a `.pcd.bin` frame stored in its pack.
+
+The pack (t4pack v1) stores each frame as an independently compressed zstd blob, followed by a JSON index of the frames:
+
+```shell
+magic b'T4PACK\x00\x01'
+blob_0 .. blob_{N-1}            zstd(column_0 bytes | column_1 bytes | ...)
+zstd(JSON index)                name, offset, size, n_points, dtypes per frame
+<QQ index_offset index_size>    little-endian
+magic
+```
+
+Points are stored column-major. `x, y, z` (and other float features) are byte-shuffled `float32`, so they round-trip bit-exact, while `intensity` and `ring` are stored as `uint8` and `int8`.
+The column layout is chosen by `num_pts_feats` in the pointcloud metainfo JSON.
+
+Each frame can also be loaded directly as follows:
+
+```python
+from t4_devkit.common.t4pack import PackReader
+
+with PackReader("data/LIDAR_CONCAT.pack") as reader:
+    points = reader.read_frame_by_name("00000.pcd.bin")  # (N, num_pts_feats) float32
+```
+
 ## Camera Image
 
 Camera directory contains raw images as the name of `<CAM_XXX>`:

docs/schema/requirement.md

@@ -62,7 +62,7 @@
 
 | ID       | Name                                 | Severity | Fixable | Description                                            |
 | -------- | ------------------------------------ | -------- | ------- | ------------------------------------------------------ |
-| `REF201` | `sample-data-filename-presence`      | `ERROR`  | `N/A`   | `SampleData.filename` exists.                          |
+| `REF201` | `sample-data-filename-presence`      | `ERROR`  | `N/A`   | `SampleData.filename` exists as a file or in its pack. |
 | `REF202` | `sample-data-info-filename-presence` | `ERROR`  | `N/A`   | `SampleData.info_filename` exists if it is not `None`. |
 | `REF203` | `lidarseg-filename-presence`         | `ERROR`  | `N/A`   | `LidarSeg.filename` exists if `lidarseg.json` exists.  |
 

mkdocs.yaml

@@ -20,6 +20,7 @@ nav:
       - Home: cli/index.md
       - t4viz: cli/t4viz.md
       - t4sanity: cli/t4sanity.md
+      - t4pack: cli/t4pack.md
   - API References:
       - t4_devkit.tier4: apis/tier4.md
       - t4_devkit.helper: apis/helper.md

pyproject.toml

@@ -22,6 +22,7 @@ dependencies = [
     "returns>=0.26.0",
     "pyarrow<24.0.0", # NOTE: pyarrow==24.0.0 is only supported on macOS
     "pypcd4>=1.4.3,<2",
+    "zstandard>=0.23.0",
 ]
 
 [dependency-groups]
@@ -42,6 +43,7 @@ dev = [
 [project.scripts]
 t4viz = "t4_devkit.cli.visualize:cli"
 t4sanity = "t4_devkit.cli.sanity:cli"
+t4pack = "t4_devkit.cli.pack:cli"
 
 [tool.ruff]
 line-length = 100

t4_devkit/cli/pack.py

@@ -0,0 +1,169 @@
+from __future__ import annotations
+
+import os
+import os.path as osp
+import sys
+from glob import glob
+from typing import Annotated
+
+import typer
+from rich.console import Console
+
+from t4_devkit.common.t4pack import (
+    DEFAULT_NUM_PTS_FEATS,
+    FRAME_SUFFIX,
+    PACK_SUFFIX,
+    delete_loose_frames,
+    pack_directory,
+    verify_pack,
+)
+from t4_devkit.dataclass.pointcloud import PointCloudMetainfo
+from t4_devkit.tier4 import load_metadata
+
+from .version import version_callback
+
+console = Console()
+
+cli = typer.Typer(
+    name="t4pack",
+    no_args_is_help=True,
+    context_settings={"help_option_names": ["-h", "--help"]},
+    pretty_exceptions_enable=False,
+)
+
+
+@cli.callback()
+def main(
+    version: Annotated[
+        bool,
+        typer.Option(
+            "--version",
+            "-v",
+            help="Show the application version and exit.",
+            callback=version_callback,
+            is_eager=True,
+        ),
+    ] = False,
+) -> None:
+    pass
+
+
+@cli.command("pack", help="Pack .pcd.bin frames of each channel into '<CHANNEL>.pack'.")
+def pack(
+    data_root: Annotated[str, typer.Argument(help="Root directory path to the dataset.")],
+    revision: Annotated[
+        str | None,
+        typer.Option(
+            ..., "-rv", "--revision", help="Specify if you want to pack the specific version."
+        ),
+    ] = None,
+    level: Annotated[int, typer.Option(..., "-l", "--level", help="zstd compression level.")] = 19,
+    overwrite: Annotated[
+        bool, typer.Option(..., "--overwrite", help="Overwrite existing packs.")
+    ] = False,
+    delete_loose: Annotated[
+        bool,
+        typer.Option(
+            ...,
+            "--delete-loose",
+            help="Delete .pcd.bin frames after verifying them against the pack.",
+        ),
+    ] = False,
+) -> None:
+    channel_dirs = _find_channel_dirs(data_root, revision)
+
+    is_passed = True
+    for channel_dir in channel_dirs:
+        num_pts_feats = _num_pts_feats(channel_dir)
+        try:
+            stats = pack_directory(
+                channel_dir,
+                num_pts_feats=num_pts_feats,
+                level=level,
+                overwrite=overwrite,
+            )
+            if stats is None:
+                console.print(f"[yellow]SKIP[/yellow] {channel_dir}: already packed")
+            else:
+                console.print(
+                    f"[green]PACKED[/green] {channel_dir}: {stats.num_frames} frames, "
+                    f"{stats.raw_bytes} -> {stats.packed_bytes} bytes ({stats.ratio:.2f}x)"
+                )
+
+            if delete_loose:
+                num_deleted = delete_loose_frames(channel_dir, num_pts_feats=num_pts_feats)
+                console.print(f"[green]DELETED[/green] {channel_dir}: {num_deleted} frames")
+        except (OSError, ValueError) as e:
+            is_passed = False
+            console.print(f"[red]FAILED[/red] {channel_dir}: {e}")
+
+    sys.exit(0 if is_passed else 1)
+
+
+@cli.command(
+    "verify", help="Verify that each '<CHANNEL>.pack' matches its .pcd.bin frames if they exist."
+)
+def verify(
+    data_root: Annotated[str, typer.Argument(help="Root directory path to the dataset.")],
+    revision: Annotated[
+        str | None,
+        typer.Option(
+            ..., "-rv", "--revision", help="Specify if you want to verify the specific version."
+        ),
+    ] = None,
+) -> None:
+    channel_dirs = _find_packed_channel_dirs(data_root, revision)
+    if not channel_dirs:
+        console.print("[red]No packs to verify[/red]")
+        sys.exit(1)
+
+    is_passed = True
+    for channel_dir in channel_dirs:
+        mismatches = verify_pack(channel_dir, num_pts_feats=_num_pts_feats(channel_dir))
+        if mismatches:
+            is_passed = False
+            console.print(
+                f"[red]FAILED[/red] {channel_dir}: {len(mismatches)} frames differ, "
+                f"e.g. {mismatches[:3]}"
+            )
+        else:
+            console.print(f"[green]VERIFIED[/green] {channel_dir}")
+
+    sys.exit(0 if is_passed else 1)
+
+
+def _find_data_dir(data_root: str, revision: str | None) -> str:
+    """Return the path to `data/` of the dataset."""
+    data_dir = osp.join(load_metadata(data_root, revision).data_root, "data")
+    if not osp.isdir(data_dir):
+        raise FileNotFoundError(f"Data directory is not found: {data_dir}")
+    return data_dir
+
+
+def _find_channel_dirs(data_root: str, revision: str | None) -> list[str]:
+    """Return directories under `data/` that contain .pcd.bin frames."""
+    data_dir = _find_data_dir(data_root, revision)
+    return sorted(
+        osp.join(data_dir, name)
+        for name in os.listdir(data_dir)
+        if osp.isdir(osp.join(data_dir, name))
+        and glob(osp.join(data_dir, name, f"*{FRAME_SUFFIX}"))
+    )
+
+
+def _find_packed_channel_dirs(data_root: str, revision: str | None) -> list[str]:
+    """Return channel directories under `data/` that have a pack, even if they are deleted."""
+    data_dir = _find_data_dir(data_root, revision)
+    return sorted(
+        osp.join(data_dir, name.removesuffix(PACK_SUFFIX))
+        for name in os.listdir(data_dir)
+        if name.endswith(PACK_SUFFIX)
+    )
+
+
+def _num_pts_feats(channel_dir: str) -> int:
+    """Return `num_pts_feats` from the metainfo in `<CHANNEL>_INFO/`, or the default if none."""
+    info_filepaths = sorted(glob(osp.join(f"{channel_dir}_INFO", "*.json")))
+    if not info_filepaths:
+        return DEFAULT_NUM_PTS_FEATS
+    return PointCloudMetainfo.from_file(info_filepaths[0]).num_pts_feats