[fix](mv) Compensate complete invalid roll-up partition buckets - #67882
Open
morrySnow wants to merge 1 commit into
Open
[fix](mv) Compensate complete invalid roll-up partition buckets#67882morrySnow wants to merge 1 commit into
morrySnow wants to merge 1 commit into
Conversation
morrySnow
requested review from
924060929,
englefly and
starocean999
as code owners
September 11, 2026 17:24
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: A roll-up materialized-view partition can cover several base-table partitions. When that MV partition is invalid, compensation previously used only the intersection with the raw query partition set. Removing the whole MV bucket while reading only part of its mapped base partitions leaves a data gap and can return incorrect aggregates. Use the intersection only as a relevance test; once an invalid MV bucket intersects the query, compensate with its complete mapped base-partition set. Disjoint invalid buckets remain excluded.
### Release note
Materialized-view rewrite now compensates every base partition represented by an invalid roll-up partition.
### Check List (For Author)
- Test: Unit tests, regression test, and FE build
- `PartitionCompensatorTest` (13 tests passed)
- `mtmv_range_date_part_up_rewrite` regression suite
- `DISABLE_BUILD_UI=ON ./build.sh --fe`
- Behavior changed: Yes. Invalid roll-up partitions are compensated atomically instead of partially.
- Does this need documentation: No
morrySnow
force-pushed
the
fix/atomic-mv-partition-compensation
branch
from
September 11, 2026 19:21
372afe5 to
4deeef3
Compare
Contributor
Author
|
run buildall |
8 tasks
hello-stephen
pushed a commit
that referenced
this pull request
Sep 12, 2026
…67897) Since today every `Doris_DorisCloudRegression_VaultP0` run dies in the `run` step before executing a single test, e.g. #67883 (TeamCity build 39010) and #67881 / #67882 / #67885 / #67886 / #67892 / #67893: ``` doris-external--minio Pulling doris-external--minio Error Error response from daemon: pull access denied for minio/minio, repository does not exist or may require 'docker login': denied: requested access to the resource is denied ERROR: start minio docker twice failed ``` MinIO stopped publishing container images in October 2025 (the project is a source-only distribution now, see minio/minio#21647) and the `minio/minio` and `minio/mc` repositories have since been removed from Docker Hub altogether (`https://hub.docker.com/v2/repositories/minio/minio/` answers 404, same for `minio/mc`). The few VaultP0 runs that still pass do so only on agents that have the image cached locally (their logs have no `Pulling` line). The iceberg, hudi and polaris third-party fixtures, `test_file_cache_warmup_read_metrics_docker` (which runs a `docker run minio/minio` itself), the all-in-one `cloud.yml` and the datalake samples reference the same images and are one cache eviction away from the same failure. `quay.io/minio/minio` and `quay.io/minio/mc` still serve every tag we use -- `RELEASE.2024-11-07T00-52-20Z`, `RELEASE.2025-01-20T14-49-07Z`, mc `RELEASE.2025-01-17T23-25-50Z`, the two 2022 tags of the samples and `latest` -- and MinIO keeps pushing hotfix tags there (latest one dated 2026-04). `docker manifest inspect` resolves all of them (amd64 / arm64 / ppc64le). So every reference gets the `quay.io/` prefix and the tags stay exactly as they were: same builds, different registry. The CI agents already pull from quay.io for the OceanBase fixture. A longer-term option is to mirror these three tags into the project's own `doristhirdpartydocker` namespace, which already hosts hive / zookeeper / kafka / trinodb; that needs someone with push access to that Docker Hub organization and can follow separately.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
During partition-union rewrite, a roll-up materialized-view partition can cover multiple base-table partitions. When that MV partition is invalid and removed as a whole, compensating only the base partitions that overlap the raw query partition set can leave the replacement scope smaller than the removed MV bucket and produce incorrect aggregate results.
Root cause
The compensation calculation used the intersection between an invalid MV partition's complete base-partition mapping and the raw query partitions as both the relevance test and the compensation set. This lost the atomic relationship between one MV partition and every base partition mapped to it.
Reproduction
Use one valid MV partition mapped to
p3, one invalid roll-up partition mapped to{p1, p2}, and another invalid partition mapped to{p4, p5}. For a raw query partition set{p1, p3}, the old calculation compensated onlyp1, even though the complete intersecting roll-up bucket was removed. The disjoint{p4, p5}bucket must still remain outside compensation.Fix
Use the intersection only to decide whether an invalid MV partition is relevant to the query. Once relevant, add its complete base-partition mapping to compensation. Invalid MV partitions with no intersection remain excluded, preserving the existing query-range guard. The existing multi-PCT-table merge behavior is unchanged.
Tests
./run-fe-ut.sh --run org.apache.doris.nereids.rules.exploration.mv.PartitionCompensatorTest(13 tests passed)DISABLE_BUILD_UI=ON ./build.sh --fe(passed; the host's Node.js 16 cannot build the current UI, so the supported no-UI FE build path was used)./run-regression-test.sh --run -s mtmv_range_date_part_up_rewrite ...(1 suite passed)