Skip to content

feat(schema): add T4Dataset 1.0 dataset loader with lookup shortcut fields - #304

Merged
jacoblambert merged 2 commits into
v1-devfrom
v1/loader
Oct 22, 2026
Merged

jacoblambert merged 2 commits into
v1-devfrom
v1/loader

Conversation

@jacoblambert

@jacoblambert jacoblambert commented Oct 13, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Adds load_dataset(data_root) (t4_devkit/schema/v1/loader.py), a minimal loader for 1.0 datasets:

  1. Reads manifest.json.
  2. Reads the table files it lists, as raw dicts. A required table that isn't listed raises FileNotFoundError; a missing optional table is empty.
  3. Builds and validates the records once, through the 1.0 registry (feat(schema): add T4Dataset 1.0 table names and registry #302).
  4. Returns them as T4Tables: the manifest, the records per table, and get(table, token) lookup.

0.x datasets (no manifest.json) raise NotImplementedError for now; their conversion plugs in at step 2 in a later PR.

Shortcut fields

T4Tables fills the same convenience fields (init=False, not written to JSON) that 0.x T4Devkit fills when loading:

  • sample.data (channel → key frame token), sample.ann_3ds, sample.ann_2ds
  • sample_data.channel, sample_data.modality
  • sensor.first_sd_token
  • category_name on sample_annotation and object_ann

Differences from 0.x:

Why outside T4Devkit

A first attempt added a second load path inside T4Devkit, with "not supported for 1.0" guards in 11 helper methods. That puts scaffolding in the class users call. Instead, the load path is built and tested on its own (this PR, LiDAR and camera reading, 0.x conversion), and T4Devkit switches to it once, in a later PR.

Why raw dicts first, then records

That split is where the 0.x conversion plugs in: 0.x tables are converted to 1.0 dicts, then go through the same build and validation as 1.0 datasets.

No change to existing code; 0.x loading in T4Devkit is untouched until we finish the implementation and merge everything.

Tests

tests/schema/v1/test_loader.py:

  • Every table of the 1.0 sample loads with its 1.0 class, and lookup by token works.
  • Shortcut fields are filled from the links.
  • A 0.x dataset raises NotImplementedError.
  • A manifest without a required table (vehicle) raises FileNotFoundError; without an optional one (caption), that table is empty.

ktro2828
ktro2828 previously approved these changes Oct 16, 2026

@ktro2828 ktro2828 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@jacoblambert
jacoblambert removed this pull request from stack #315 October 22, 2026 08:55
Base automatically changed from v1/index-fields to v1-dev October 22, 2026 09:43
@jacoblambert
jacoblambert merged commit 4e88e45 into v1-dev Oct 22, 2026
0 of 5 checks passed
@jacoblambert
jacoblambert deleted the v1/loader branch October 22, 2026 09:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants