[common] Stop answering value predicates from a truncated bitmap index - #9866
Merged
JingsongLi merged 1 commit intoSep 17, 2026
Merged
JingsongLi merged 1 commit into
JingsongLi merged 1 commit into
Conversation
BitmapFileIndex keys TIMESTAMP / TIMESTAMP_LTZ through toMicros(), so on precision 7..9 columns two values in the same microsecond share one bitmap key and visitNotIn's flip drops every row of that bucket from <> / NOT IN. Wrap the reader for those types so value predicates return REMAIN and only IS NULL / IS NOT NULL are still answered from the index, mirroring apache#9654. Generated-by: Claude Code
Contributor
|
Solid correctness fix. Verified the gate matches the mapper's lossiness exactly: the only lossy value mappers are the TIMESTAMP ones going through The DE integration test comparing the indexed read with a full scan across the predicate shapes is a good safety net, including the guard that sub-microsecond values survive the write/read round trip. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
fix #9863
BitmapFileIndexkeys TIMESTAMP / TIMESTAMP_LTZ throughTimestamp.toMicros(), so on precision 7..9 values in the same microsecond share one key;visitNotInflips that key and the exactBitmapIndexResultdrops the whole bucket from<>/NOT IN.REMAINwhileIS NULL/IS NOT NULLstill come from the index.Tests
BitmapFileIndexTest#testSubMicrosecondTimestampIndexAnswersNoValuePredicateand#testMicrosecondTimestampIndexStillAnswersValuePredicates.DataEvolutionFileIndexTest#testSubMicrosecondTimestampBitmapMatchesUnindexed(Parquet, ORC): indexedexecuteFilter()read vs. unindexed scan for=,<>,IN,NOT IN,IS [NOT] NULL,AND,OR.BitmapIndexResultinstead ofREMAINand the table-level test fails on<>for both formats.mvn -pl paimon-common clean install(JDK 11): BUILD SUCCESS, 0 failures;DataEvolutionFileIndexTest32/32 in the reactor.