feat(v1.0): T4Dataset v1.0 Release

.pre-commit-config.yaml

@@ -32,6 +32,8 @@ repos:
     rev: v4.0.0-alpha.8
     hooks:
       - id: prettier
+        # written by DatasetWriter: manifest.json holds the hashes of these files
+        exclude: ^tests/sample/t4dataset_v1/
 
   - repo: https://github.com/astral-sh/ruff-pre-commit
     rev: v0.6.8

docs/cli/index.md

@@ -4,3 +4,4 @@ Following command line tools are supported:
 
 - [`t4viz`: Visualize T4 dataset features](./t4viz.md)
 - [`t4sanity`: Sanity checker of T4 dataset](./t4sanity.md)
+- [`t4migrate`: Migrate a 0.x dataset to 1.0](./t4migrate.md)

docs/cli/t4migrate.md

@@ -0,0 +1,43 @@
+`t4migrate` migrates a 0.x dataset to a new [1.0](../schema/v1/index.md) dataset. The 0.x dataset is only read.
+
+```shell
+$ t4migrate -h
+
+ Usage: t4migrate [OPTIONS] {data_root} {output_dir}
+
+╭─ Arguments ──────────────────────────────────────────────────────────────────────────────────────────────────────────╮
+│ *    data_root       <str>  0.x dataset: the <DATASET_ID>/<VERSION> directory with annotation/. [required]           │
+│ *    output_dir      <str>  Parent directory: the dataset goes to <OUTPUT_DIR>/<DATASET_ID>/<DATASET_VERSION>.       │
+│                             [required]                                                                               │
+╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
+╭─ Options ────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
+│    --version             -v             Show the application version and exit.                                       │
+│ *  --dataset-version     -d      <int>  Version of the 1.0 dataset (given by Web.Auto). [required]                   │
+│    --dataset-id                  <str>  Dataset ID, if not the 0.x dataset's.                                        │
+│    --keep-images                        Copy camera images as they are instead of encoding one AV1 video per         │
+│                                         channel.                                                                     │
+│    --help                -h             Show this message and exit.                                                  │
+╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
+```
+
+## Usage
+
+```shell
+t4migrate <0.x DATASET_ID>/<VERSION> <OUTPUT_DIR> --dataset-version <DATASET_VERSION>
+```
+
+This writes `<OUTPUT_DIR>/<DATASET_ID>/<DATASET_VERSION>`, prints the migration notes, and runs [`t4sanity`](./t4sanity.md) on the result. It exits with `1` if a sanity rule fails.
+
+The dataset version is given by the caller: Web.Auto assigns it when the new version is uploaded. An existing output folder is never overwritten.
+
+## What Changes
+
+- Tables are converted as described in [Loading 0.x Datasets](../schema/v1/index.md#loading-0x-datasets).
+- Each LiDAR channel becomes one Parquet file, one row group per frame:
+  - fields get their 1.0 types (e.g. `intensity` uint8) when every value fits; otherwise they keep their 0.x type, with a warning and a migration note;
+  - a ring (`channel`) of `-1` on every point (not recorded) is dropped, with a note;
+  - `.info` files become the `source_index` / `source_timestamp` columns;
+  - lidarseg becomes the `label` / `instance` columns (`instance` is `-1`: 0.x labels are semantic only). Category `0` must be `unpainted`.
+- Each camera channel becomes one AV1 video, frames in timestamp order (or the images are copied with `--keep-images`). Frames with `is_valid: false` are encoded as they are.
+- `map/` and `input_bag/` are copied as they are.
+- `manifest.json` has `migrated_from_schema_version: "0.x"` and the migration notes. Provenance that 0.x does not store (Web.Auto IDs, topics) is empty, for backfilling.

docs/cli/t4sanity.md

@@ -1,4 +1,4 @@
-`t4sanity` performs sanity checks on T4 datasets, reporting any issues regarding the [dataset requirements](../schema/requirement.md).
+`t4sanity` performs sanity checks on T4 datasets, reporting any issues regarding the dataset requirements: [0.x](../schema/requirement.md) or [1.0](../schema/v1/requirement.md), chosen by the dataset (1.0 if it has a `manifest.json`).
 
 ```shell
 $ t4sanity -h
@@ -15,6 +15,7 @@ $ t4sanity -h
 │ --exclude             -e       TEXT  Exclude specific rules or rule groups. [default: None]                           │
 │ --strict              -s             Indicates whether warnings are treated as failures.                              │
 │ --fix                 -f             Attempt to fix the issues reported by the sanity check.                          │
+│ --checksum            -c             Verify the SHA-256 of every file listed in manifest.json (1.0 datasets).          │
 │ --install-completion                 Install completion for the current shell.                                        │
 │ --show-completion                    Show completion for the current shell, to copy it or customize the installation. │
 │ --help                -h             Show this message and exit.                                                      │

docs/cli/t4viz.md

@@ -58,7 +58,7 @@ t4viz instance <DATA_ROOT> <INSTANCE_TOKEN1> <INSTANCE_TOKEN2> ... [OPTIONS]
 
 This command performs the same behavior with [`T4Devkit.render_pointcloud(...)`](../tutorials/render.md#rendering-pointcloud).
 
-If a pointcloud `sample_data` record has `info_filename`, `t4viz pointcloud` loads the metainfo JSON automatically and uses `num_pts_feats` to parse extended point layouts. Without that file, it falls back to the standard 5-feature layout.
+The point layout comes from the file (Parquet, BIN, PCD) or from `sensor.lidar.fields` (`.bin`; for 0.x datasets it is taken from the `.info` files when loading).
 
 For options, run `t4viz pointcloud -h`.
 

docs/schema/requirement.md

@@ -1,5 +1,7 @@
 # Dataset Requirements
 
+The rules `t4sanity` checks on 0.x datasets. For T4Dataset 1.0, see [Dataset Requirements 1.0](./v1/requirement.md).
+
 ## Structure (`STR`)
 
 | ID       | Name                          | Severity  | Fixable | Description                                                          |

docs/schema/v1/data.md

@@ -0,0 +1,79 @@
+# Sensor Data 1.0
+
+Each sensor channel is stored as a single file, `data/<CHANNEL>/<CHANNEL>.<FORMAT>`. A frame is addressed by `sample_data.filename` (the file) and `sample_data.frame_index` (the frame inside it), starting at 0.
+
+How the file is written is described on the channel's `sensor` row (`camera.video` or `lidar.container`, see [Schema Tables](./table.md#sensor)).
+
+## Camera
+
+One intra-only AV1 video per channel:
+
+```shell
+data/
+└── CAM_FRONT/
+    └── CAM_FRONT.mp4
+```
+
+| Setting         | Value                                                              |
+| --------------- | ------------------------------------------------------------------ |
+| Container       | MP4                                                                |
+| Codec           | AV1, encoded with SVT-AV1 (preset 6, CRF 26)                       |
+| Frame types     | Intra-only: every frame is a keyframe (`-g 1`)                     |
+| Pixel format    | `yuv420p`, full range, BT.601                                      |
+| Track timescale | Equal to the frame rate, so a frame's pts equals its `frame_index` |
+
+Because every frame is a keyframe, any frame can be decoded without decoding its neighbours. Readers seek by `frame_index`.
+
+Datasets migrated from 0.x without re-encoding keep one image file per frame (`.jpg` or `.png`). In that case `sensor.camera.video` is `null` and `sample_data.frame_index` is `null`.
+
+## LiDAR
+
+One Parquet file per channel, with one row group per frame:
+
+```shell
+data/
+└── LIDAR_CONCAT/
+    └── LIDAR_CONCAT.parquet
+```
+
+- Columns are the point fields listed in `sensor.lidar.fields`, in that order, e.g. `x, y, z, intensity, ring`.
+- Each row is one point. Row group `frame_index` holds all points of that frame.
+- Compression is lossless (zstd), so values are bit-exact.
+- Frame of reference is given in `sensor.lidar.frame_id` (`base_link` by default).
+
+### Point-wise Labels
+
+When `manifest.json` lists `segmentation3d` in `capabilities.tasks`, the LiDAR file has two extra columns for panoptic labels, listed in `sensor.lidar.fields` like any other field:
+
+| Column     | Type    | Value                                                                                         |
+| ---------- | ------- | --------------------------------------------------------------------------------------------- |
+| `label`    | `uint8` | [`category.index`](./table.md#category) of the point's category; `0` for unpainted points     |
+| `instance` | `int16` | [`instance.index`](./table.md#instance) of the point's object; `-1` for points with no object |
+
+Each column reserves one value:
+
+- `label` `0` is the `unpainted` category: points the annotator did not label. When point-wise labels are present, `category.json` has a category named `unpainted` with `index` `0`.
+- `instance` `-1` means no object: points of categories without `is_thing` (road, vegetation) and unpainted points. `instance.index` runs from `0` to `32767`.
+
+Labeled frames have a value for every point. Frames without labels (for example, non-key frames) have `null` in both columns for every point, because a Parquet file has the same columns in every frame. Readers leave out a column that is `null` for the whole frame.
+
+This replaces the `lidarseg` table and the `lidarseg/` folder of 0.x.
+
+### Merged Sources
+
+When `sensor.lidar.source_channels` is not empty, the channel merges several LiDARs and the file has two extra columns, listed in `sensor.lidar.fields` like any other field:
+
+| Column             | Type    | Value                                                                |
+| ------------------ | ------- | -------------------------------------------------------------------- |
+| `source_index`     | `uint8` | [`sensor.index`](./table.md#sensor) of the LiDAR the point came from |
+| `source_timestamp` | `int64` | Timestamp of that LiDAR's scan, in microseconds                      |
+
+Each source LiDAR has its own `sensor` row (with its calibration) and no `sample_data`. Points are grouped by source, so both columns are runs of equal values and cost almost nothing to store.
+
+This replaces the per-frame `.info` metainfo files of 0.x.
+
+0.x datasets loaded without migration keep one `.pcd.bin` or `.pcd` file per frame, as described in [Sensor Data](../data.md). In that case `lidar.container.format` is `bin` or `pcd` and `sample_data.frame_index` is `null`.
+
+## Radar
+
+Radar is not defined in 1.0.