[core] Use full-range anchor as row representative in DE read-type pruning - #9899
Open
LuciferYang wants to merge 1 commit into
Open
LuciferYang wants to merge 1 commit into
LuciferYang wants to merge 1 commit into
Conversation
…uning When every file in a data-evolution row-id-range group lacks the columns of the query projection (e.g. a freshly added column), pruneByReadType keeps one file as a row-count representative so the reader can emit the right number of NULL-filled rows. The representative was group.get(0), which can be a blob or vector-store file: such files cover only a sub-range of the group's row ids, so the split silently emitted fewer rows than the group contains. Use retrieveAnchorFile instead, which selects the oldest full-range normal file and never a blob/vector-store file. The pruning core is extracted into a static method (mirroring evolutionStats) so the representative choice can be pinned by a unit test with a blob file ordered first in the group. Assisted-by: GLM-5.3
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
close #9898
When a data-evolution row-id-range group has no file that writes the query's projected columns, read-type pruning keeps one file as a row-count representative so the reader emits the right number of NULL-filled rows. The representative was
group.get(0), which can be a blob or vector-store file. Those files cover only a sub-range of the group's row ids, so the split silently emitted fewer rows than the group actually contains. This usesretrieveAnchorFileinstead, which returns a full-range normal data file and never a blob or vector-store file (the same primitive the deletion-vector anchor path already relies on).The pruning core is extracted into a
@VisibleForTestingstatic method so the representative choice can be pinned by a unit test with a blob file ordered first in the group. The filter-field logic added by #9872 is preserved: the instance method still folds in the fields referenced only by the filter and passes them to the static method, so the set of kept files is unchanged.Tests
DataEvolutionFileStoreScanTest#testReadTypePruningKeepsAnchorAsRowRepresentativebuilds a group of a blob file covering rows 0-1 (ordered first) plus two full-range normal files covering rows 0-9, with areadTypereferencing a column no file writes. It asserts the representative is the full-range normal file. Against the oldgroup.get(0)fallback the representative is the blob file, so the assertion fails.