[format] Drop filters on unprojected columns before Parquet conversion

[wangzhigang1999]

Purpose

Fixes #9857.

Java table reads can lose matching rows when a filter references a column outside the read projection. With four rows (id, status) = (0, A), (1, B), (2, null), (3, A), newReadBuilder().withFilter(statusEqualsA).withReadType(idOnly) returns no rows on the baseline, with or without a bitmap file index.

Filter predicates in ParquetReaderFactory before converting them to Parquet filters. Split top-level AND conditions and retain only conjuncts whose referenced fields are projected. Keep each OR conjunct intact and discard it if any referenced field is unprojected. Field matching follows the reader's case-sensitivity setting.

This follows the projection-filtering direction suggested in #5385. It preserves the read projection, physical type resolution, and Parquet page-index implementation. Non-projected conditions are omitted even when the physical file lacks those columns, so the reader may return extra candidates under the existing best-effort filtering contract. No public API or storage format changes.

Tests

Added two parameterized methods in existing test classes (four cases total). The public Java table-read regression covers bitmap file indexing enabled and disabled; both cases fail on the baseline. The format-level method checks that an AND retains projected-column pruning and an OR preserves matches from an unprojected column; the OR case fails on the baseline.

On the fixed 43e9ad7 baseline, all 52 Parquet read/write tests and the two public API cases pass per Surefire phase, with no failures, errors, or skips. Checkstyle, Spotless, Enforcer, and root RAT pass. All builds and tests ran on JDK 17 with Java 8 source compatibility and an isolated Maven repository, without fast-build.

mvn -B -pl paimon-core -am \
  -Dtest=ParquetReadWriteTest,AppendOnlySimpleTableTest#testParquetFilterOnUnprojectedColumn \
  -DfailIfNoTests=false -Dsurefire.failIfNoSpecifiedTests=false \
  -DwildcardSuites=none verify
mvn -B -N apache-rat:check

After merging upstream master at 3a858842a498c61f1c385291873c81002bb90856, the fork CI run passed the Java matrix, including Core, Flink, Spark, E2E, and Iceberg, as well as artifact licensing. The Python 3.6.15 job was cancelled and the aggregate CI check failed, so the full run is not green. The earlier dependency download failure and Iceberg test error did not recur in this run.

The reproduction and validation cover Java table and file-format reads. They do not establish a Spark/Flink SQL failure or exact row-level filtering.

[Copilot]

🟢 Approval recommended

No unresolved blocking issues were identified.

Pull request overview

Fixes Parquet reads dropping rows when filters reference unprojected columns.

Changes:

• Filters predicates before Parquet conversion based on projected fields.
• Preserves safe AND pruning and handles OR predicates safely.
• Adds format-level and public table-read regression tests.

File summaries

File	Summary
paimon-format/src/test/java/org/apache/paimon/format/parquet/ParquetReadWriteTest.java	Tests AND/OR projection behavior.
paimon-format/src/main/java/org/apache/paimon/format/parquet/ParquetReaderFactory.java	Filters predicates based on projected fields and case sensitivity.
paimon-core/src/test/java/org/apache/paimon/table/AppendOnlySimpleTableTest.java	Tests public table reads with bitmap indexing.

Review details

• Files reviewed: 3/3 changed files
• Comments generated: 0
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.