[hello-peter-tang]

Summary

Adds daft_lance.validate_fragments, a read-only operation that validates selected Lance fragments without scanning table rows, closing the integration gap in #63.

Planning happens on the driver: it opens the requested snapshot (or version=/asof=), enumerates fragment metadata from the pinned manifest, and validates the requested subset of fragment IDs. Execution happens on Daft workers, which reopen that exact pinned snapshot through the existing DatasetOpenContext and call Lance's native LanceFragment.validate() with bounded concurrency. This reuses Lance's validator rather than implementing a second one, so fragment file/metadata and deletion-vector consistency checks stay upstream.

API

report = daft_lance.validate_fragments(
    "s3://bucket/table.lance",   # or namespace_impl/table_id
    fragment_ids=[3, 7],         # None validates every fragment
    version=12,
    concurrency=8,
)

The returned DataFrame has one row per selected fragment:

column	type	meaning
version	Int64	pinned dataset version that was validated
fragment_id	Int64	fragment in that version
valid	Boolean	whether validation succeeded
error	String	null on success; the exception otherwise

Behaviour:

• Read-only: no commits, repairs, or version changes during validation (asserted by tests).
• Pinned snapshot: a concurrent append after planning does not change which version or fragments are validated.
• Failures are attributed, not skipped: worker dataset-open/authentication failures carry a failed to open the dataset at version ... prefix, distinct from detected fragment corruption.
• Explicit selection semantics: unknown fragment IDs and empty selections raise ValueError; duplicates are ignored; fragment_ids=None validates every fragment in the pinned manifest.
• Local and distributed execution both work; CI runs the file under the native and Ray runners.

PyLance requirement: LanceFragment.validate() needs PyLance >= 11. The package already requires >= 12, so the floor is unchanged; this is documented on the public function.

Tests

New tests/io/lance/test_fragment_validation.py (13 tests) covers:

• healthy multi-fragment datasets; report schema, version, and read-only behaviour;
• subset selection, duplicate IDs, unknown IDs, empty selection, invalid concurrency;
• fragments with multiple column files (add_columns enrichment);
• deletion-vector fragments, and fully deleted fragments removed from the manifest;
• a missing data file attributed to the correct fragment while neighbours stay valid;
• pinning against a concurrent append, and explicit version= validation;
• a dir namespace-addressed table reopening on workers;
• worker open failure distinguished from corruption.

Closes #63

[hello-peter-tang]

Heads-up: the PR Test Suite / PR Checks run for this branch is sitting in action_required, so the PR shows no CI results at all. GitHub is holding fork runs until a maintainer approves them ("Approve and run"). @maintainers, could someone approve the pending run so this gets tested?

[hello-peter-tang]

CI for this branch is still parked in action_required, so I ran the same checks locally on the PR head ca09b40 to make the review less blind:

ruff check daft_lance tests            -> All checks passed!
ruff format --check daft_lance tests   -> 51 files already formatted
mypy --strict --ignore-missing-imports daft_lance
                                       -> Success: no issues found in 16 source files
pytest tests/ -q                       -> 565 passed, 10 skipped, 2 xfailed, 2 xpassed in 32.70s

mypy is scoped to daft_lance because that is what the style job effectively checks; running it over tests/ reports 582 pre-existing errors on main too, so it is not a signal about this PR.

Environment: Linux x86_64, Python 3.11.6, deps installed from the committed uv.lock, DAFT_ANALYTICS_ENABLED=0. For reference, main on the same setup is 552 passed, 10 skipped, 2 xfailed, 2 xpassed, so this branch adds 13 passing tests. Not covered locally: the 3.10/3.12/3.13 matrix and the DAFT_RUNNER=ray job — those still need the real CI run.