Skip to content

[core] Skip BTree index reads for all-matching predicates - #9834

Open
jerry-024 wants to merge 1 commit into
apache:masterfrom
jerry-024:improve-btree-all-match-pruning
Open

jerry-024 wants to merge 1 commit into
apache:masterfrom
jerry-024:improve-btree-all-match-pruning

Conversation

@jerry-024

Copy link
Copy Markdown
Contributor

Purpose

Avoid opening BTree global index files when their manifest metadata already proves that every indexed row matches an equality or range predicate. For example, an index whose values are all dog should not read and decode its posting list for column = 'dog'.

  • Pass the existing index-manifest row count through GlobalIndexIOMeta; retain the existing constructor for callers without row-count metadata.
  • Check the whole reader's local row-ID domain, not an individual index file. Short-circuit only when the scalar row counts cover that domain and every file has non-null min/max with no null values.
  • For =, <, <=, >, >=, and BETWEEN, return the covered row range directly when both aggregate bounds match. Partial matches, unknown counts, sparse coverage, and nullable files keep the existing lookup path.
  • Preserve row-ID offsets, selective sibling predicates, and fast/full/detail coverage behavior. No on-disk format changes or index rebuilds are required.

All-match range queries need no index I/O, so they do not consume the fallback scan budget; partially matching ranges remain subject to it.

Tests

  • Added parameterized reader tests covering zero index-file opens, row-ID offsets, strict/inclusive range boundaries, multiple files, null values, sparse/unknown coverage, and fallback scan budgets.
  • Added a DE table regression that deletes the all-matching test index files before scanning and verifies equality, AND/OR combinations, and all three search modes.
  • Confirmed the zero-open regression fails before the optimization.
  • Passed 363 tests across paimon-common and paimon-core, with Checkstyle and Spotless enabled:
mvn -pl paimon-core -am \
  -DfailIfNoTests=false -DwildcardSuites=none \
  -Dtest=LazyFilteredBTreeIndexReaderTest,SortedFileMetaSelectorTest,BTreeIndexReaderTest,BTreeIndexReaderCloseTest,BTreeThreadSafetyTest,GlobalIndexEvaluatorTest,BtreeGlobalIndexTableTest,DataEvolutionBatchScanTest,BTreeTopNIndexFileSelectorTest \
  test

@jerry-024
jerry-024 marked this pull request as draft September 15, 2026 06:14
@jerry-024
jerry-024 marked this pull request as ready for review September 15, 2026 07:17
@jerry-024
jerry-024 marked this pull request as draft September 15, 2026 10:23
@jerry-024
jerry-024 marked this pull request as ready for review September 15, 2026 10:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant