[fix](nereids) Unify null-input safety checks for join pushdown - #67889
Open
morrySnow wants to merge 1 commit into
Open
[fix](nereids) Unify null-input safety checks for join pushdown#67889morrySnow wants to merge 1 commit into
morrySnow wants to merge 1 commit into
Conversation
morrySnow
requested review from
924060929,
englefly and
starocean999
as code owners
September 11, 2026 19:24
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
morrySnow
changed the title
morrySnow
marked this pull request as draft
September 12, 2026 03:29
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: Expressions evaluated around outer joins can observe NULL-extended rows, but project pushdown, runtime-filter pushdown, and eager aggregation used different approximations of whether an expression remains NULL. Project and filter expressions could be moved below null extension and return an incorrect value, while shallow structural checks and global marker scans could also reject safe runtime filters and aggregate pushdown. Introduce one fail-closed typed-NULL evaluator, add an FE-only folding path for optimizer safety checks, and reuse the exact complete-expression result across logical and physical pushdown decisions. Runtime-filter contexts now carry the rewrite context through all creation and probe-rewrite paths, eager aggregation proves every argument for each candidate nullable side, and the obsolete null-to-non-null marker interface is removed.
### Release note
Fix incorrect results from pushing expressions that turn outer-join NULL inputs into non-NULL values below null extension, and use precise NULL-input proofs for safe runtime-filter and eager-aggregation pushdown.
### Check List (For Author)
- Test: Unit tests, regression tests, and full FE build
- Five related FE unit-test classes passed 88 tests, including typed-NULL classification, project/filter guards, runtime-filter targets, eager-aggregation safety, and existing expression utilities
- Sandbox regressions passed for outer-join project null extension, outer-join runtime-filter safety, and eager aggregation after loading its standard fixture data
- Full FE build passed across 80 modules with zero Checkstyle violations using DISABLE_BUILD_UI=ON
- Behavior changed: Yes. Unsafe pushdown across null-generating join sides is rejected, while complete expressions proven to evaluate to SQL NULL remain eligible.
- Does this need documentation: No
morrySnow
force-pushed
the
fix/guard-null-side-project-pushdown
branch
from
September 12, 2026 10:01
ca02b36 to
f966529
Compare
morrySnow
changed the title
Contributor
Author
|
run buildall |
morrySnow
marked this pull request as ready for review
September 12, 2026 10:02
Contributor
TPC-H: Total hot run time: 16983 ms |
Contributor
TPC-DS: Total hot run time: 82676 ms |
Contributor
ClickBench: Total hot run time: 14.95 s |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Expressions evaluated around outer joins can observe NULL-extended rows. Several optimizer components independently approximated whether such expressions remain NULL: project pushdown had no nullable-side guard, runtime-filter pushdown used a shallow Slot/Cast/PropagateNullable check, and eager aggregation relied on a global marker that both over-blocked safe nested expressions and could not reason about each candidate side. These inconsistent checks could cause incorrect project/filter placement or unnecessarily reject safe runtime-filter and aggregate pushdown.
Root cause
The optimizer lacked one fail-closed, typed SQL-NULL evaluation primitive. The existing general constant-fold entry point may invoke BE folding when enabled, which is unsuitable for physical post-processing and repeated optimizer safety checks. Runtime filters therefore duplicated structural logic, while eager aggregation scanned for null-to-non-null markers instead of evaluating the complete aggregate argument after project substitution and for the actual null-generating side.
Reproduction
element_at(coalesce(right_array, [99]), 1)must return 99 for an unmatched row. The old project pushdown could evaluate it below the join and return NULL instead.c_custkey + cast(c_custkey is null as int)evaluates to NULL when the nullable-side key is NULL, but the old structural classifier could not prove it and stopped safe pushdown;coalesce(c_custkey, 0)must remain blocked.max(t2.id2 + cast(t2.id2 is null as int))remains NULL on a NULL-extended right side and is safe to pre-aggregate, but the old marker scan rejected it merely because it contained IS NULL.Fix
NULL,FALSE,TRUE,OTHER_NON_NULL,UNKNOWN) and its fail-closed consumers in not-null inference, hypergraph conflict rules, and project/filter pushdown.FoldConstantRuleentry point with identical debug-skip and exception behavior that never invokes BE-folding RPC, and expose it throughNullInputEvaluatorwithout duplicating replacement or classification.ExpressionRewriteContextthrough every runtime-filter pushdown factory, including CTE and min/max paths, preserve it when rewriting probe expressions, and allow traversal of a non-builder outer join's nullable child only when the complete typed-NULL probe evaluates exactly to SQL NULL.count(*), non-NULL literals, incomplete folds, and decomposed aggregate IF remain blocked; ordinary nullable columns, NULL literals, and complete nested expressions that evaluate to NULL remain eligible.Tests
NullInputEvaluatorTest,PushDownProjectTest,EagerAggRewriterTest,RuntimeFilterTest, andExpressionUtilsTest. They cover five-state classification, debug-skip fail-closed behavior, FE-only evaluation with BE folding enabled, safe and unsafe runtime-filter expressions and the concrete customer-scan target, count/literal/project-hidden eager-aggregation hazards, and the formerly over-blocked nested expression.DISABLE_BUILD_UI=ON ./build.sh --fe.git diff --checkcompleted with no output.