Skip to content

[common] Stop answering value predicates from a truncated bitmap index - #9866

Merged
JingsongLi merged 1 commit into
apache:masterfrom
thswlsqls:fix/common-bitmap-index-truncated-timestamp
Sep 17, 2026
Merged

JingsongLi merged 1 commit into
apache:masterfrom
thswlsqls:fix/common-bitmap-index-truncated-timestamp

Conversation

@thswlsqls

Copy link
Copy Markdown
Contributor

Purpose

fix #9863

  • BitmapFileIndex keys TIMESTAMP / TIMESTAMP_LTZ through Timestamp.toMicros(), so on precision 7..9 values in the same microsecond share one key; visitNotIn flips that key and the exact BitmapIndexResult drops the whole bucket from <> / NOT IN.
  • Wrap the reader for those types so value predicates return REMAIN while IS NULL / IS NOT NULL still come from the index.
  • Mirrors [common] Stop answering value predicates from a truncated BSI index #9654 (BSI); range-bitmap already rejects precision > 6 at build time.

Tests

  • Added BitmapFileIndexTest#testSubMicrosecondTimestampIndexAnswersNoValuePredicate and #testMicrosecondTimestampIndexStillAnswersValuePredicates.
  • Added DataEvolutionFileIndexTest#testSubMicrosecondTimestampBitmapMatchesUnindexed (Parquet, ORC): indexed executeFilter() read vs. unindexed scan for =, <>, IN, NOT IN, IS [NOT] NULL, AND, OR.
  • Without the fix the unit test gets BitmapIndexResult instead of REMAIN and the table-level test fails on <> for both formats.
  • mvn -pl paimon-common clean install (JDK 11): BUILD SUCCESS, 0 failures; DataEvolutionFileIndexTest 32/32 in the reactor.

BitmapFileIndex keys TIMESTAMP / TIMESTAMP_LTZ through toMicros(), so on
precision 7..9 columns two values in the same microsecond share one bitmap
key and visitNotIn's flip drops every row of that bucket from <> / NOT IN.
Wrap the reader for those types so value predicates return REMAIN and only
IS NULL / IS NOT NULL are still answered from the index, mirroring apache#9654.

Generated-by: Claude Code
@JingsongLi

Copy link
Copy Markdown
Contributor

Solid correctness fix. Verified the gate matches the mapper's lossiness exactly: the only lossy value mappers are the TIMESTAMP ones going through Timestamp.toMicros(), and precision > 6 is precisely where that truncates. Precision 6 keeps answering value predicates (the second test pins that), and inheriting REMAIN for everything except isNull/isNotNull is the right conservative choice — <> and NOT IN were the dangerous ones because they flip the matched rows over the whole row count.

The DE integration test comparing the indexed read with a full scan across the predicate shapes is a good safety net, including the guard that sub-microsecond values survive the write/read round trip.

@JingsongLi
JingsongLi merged commit 5422b26 into apache:master Sep 17, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Bitmap file index drops rows for <> / NOT IN on TIMESTAMP(7..9) columns

2 participants