perf(file): reduce metadata read scheduling rounds

[Ali2Arslan]

Status

Draft and stacked on #9168. After the format contract merges, rebase away its duplicated contract changes and mark this PR ready.

What this does

• Adds intent-specific, private-field options around one reusable tail-read planner.
• Full metadata opens use the exact persisted suffix size to schedule every required metadata byte immediately.
• Projected data-file opens derive the initial range from existing manifest column mappings: baseline_tail + 16 * estimated_physical_columns.
• Footer offsets and counts remain authoritative. Absent, stale, oversized, underestimated, or inconsistent hints retain footer-directed decoding.
• Hint-driven overread is admitted only when the complete planned tail fits the scheduler's configured unread-I/O byte budget. This bounds corrupt in-range hints without conflating the policy with max_iop_size; admitted logical ranges may still split concurrently. A zero/unbounded byte-backpressure setting conservatively uses the baseline footer path.
• Fallbacks extend one accumulated suffix with only the missing prefix. CMO and GBO bytes are coalesced instead of issuing overlapping reads.
• Large logical ranges remain eligible for FileScheduler splitting into concurrent physical requests; dependency depth and physical request count remain separate concerns.
• Speculative prefetching retains global-buffer bytes only from the original baseline tail window.
• Current-format writers return an opaque, evolution-safe write result and propagate the exact suffix through data files, replacements, concat/staged parts, binary copy, scalar indices, and modern vector indices.
• Python and Java models preserve the advisory value across transaction and fragment round trips while keeping existing constructors compatible; zero remains the unavailable sentinel.
• Scalar index stores use one typed per-file hint map instead of parallel maps.
• Modern IVF cold opens dispatch the root reader once and prefetch exact root and auxiliary metadata concurrently; both decoded readers are reused.
• Legacy writers and readers are unchanged.

I/O behavior covered by tests

• Exact full-metadata hint: 2 dependent reads -> 1 scheduling round.
• Correct projected column estimate: tail + CMO dependency -> 1 scheduling round.
• Underestimates extend only the missing prefix, with no duplicate bytes.
• Multi-step full-metadata fallback maintains one accumulated suffix.
• Overestimates and stale/oversized values preserve correct footer-directed decoding.
• A corrupt in-range exact hint and a pathological projected-column estimate cannot trigger a logical read larger than the configured scheduler byte budget.
• FileScheduler request splitting remains unconstrained by the hint planner.
• Stale over-prefetch does not increase retained global-buffer cache residency.
• Modern vector root and auxiliary readers issue no duplicate request signatures; legacy vector opens still succeed.
• Missing hints in older manifests and staged-part checkpoints remain supported.

Validation

• cargo test -p lance-file
• cargo test -p lance-index metadata_hint_removes_scalar_index_footer_dependency
• cargo test -p lance concatenates_parts_in_caller_order_without_reusing_staging_files
• cargo test -p lance test_hnsw_index_records_file_sizes
• cargo test -p lance test_legacy_non_divisible_pq_search
• uv run pytest python/tests/test_fragment.py::test_fragment_meta python/tests/test_commit_index.py::test_commit_index_with_files python/tests/test_table_ops.py::test_data_file_create_basic python/tests/test_table_ops.py::test_data_file_create_end_to_end
• uv run make lint
• cargo clippy --tests --manifest-path java/lance-jni/Cargo.toml -- -D warnings
• ./mvnw -Dtest=FragmentTest,TransactionTest test
• cargo clippy --all --tests --benches -- -D warnings

Made with Cursor