Skip to content

datalake_fdw: read path — planned fragments, CustomScan, projection and pruning #2019

Description

@MisterRaindrop

Part of #2008. Letters (A, B0–B7, C, D, E) are the PRs listed there; this is D.

Scope

  • Fragments from the engine's plan (files, row-group ranges) assigned to segments; one large file may be split by row group.
  • A CustomScan (planner hook, design decision D10) that opens a reader per fragment and decodes batches into slots.
  • Projection by field id from the table's Iceberg schema (ProjectionSet), so files written before a column was added or renamed read correctly; only needed columns are read.
  • Row-group pruning from the query's quals against Parquet statistics; EXPLAIN ANALYZE reports row groups skipped.
  • With it: datalake_fdw: Iceberg name mapping for data files without field ids #1989 name mapping for files without field ids.

Out of scope

Delete files (E), time travel (#1683 §2.3), ANALYZE beyond the current zero-sample no-op.

Depends on

A, B2, B4. Skeleton, projection and pruning are testable against the stub and local files first.

Acceptance

  • Same rows as Spark on the same table, including one Spark evolved (column added, renamed, dropped, int promoted to long).
  • A file split into three fragments reads the same as whole; every segment reads only its fragments.
  • EXPLAIN ANALYZE shows pruning for a selective predicate and none for WHERE true.
  • Unsupported or dictionary-encoded columns are refused naming the column; cancel mid-scan releases the reader.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    datalakecontrib/datalake_fdw and contrib/datalake_agent: Iceberg lake tables

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions