You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Everything between the DDL skeleton (#1842) and a lake table that can be written and read from SQL. Design: #1683. This issue is the execution view.
Landed: #1842, the skeleton (AM, catalog/volume FDWs, stub engine, DDL guards); #1951, the Parquet format layer (Arrow, field ids, tracked memory pool, type rules).
Twelve PRs, three waves
wave 1 wave 2 wave 3
A S3 storage ────────────────────────────────────────┐
B0 proto ─┬► B1 agent framework ─┬► B3 Iceberg ops ─► B4 Builtin ─┼► C write ─┬► E merge-on-read + DML
└► B2 engine client ───┘ ├► B5 Polaris ├► D read ──┘
├► B6 Hive Metastore
└► B7 Hadoop
C and D need A, B2 and B4; with the Builtin catalog they run in CI against MinIO and the agent alone. B5–B7 need B3 and run alongside C and D.
Attached to the PR they belong with: #1988 NUMERIC (C), #1990 timestamp units (C), #1989 name mapping (D). Later, blocking nothing: the optional datalake_proxy bgworker (#1683 §5.4).
Every PR leaves both components shippable: an unfinished path refuses with a clear error. The agent answers UNIMPLEMENTED for RPCs it does not carry yet, as the stub engine does.
The proto files and the shared headers (format/format.h, meta/iceberg_meta_engine.h, common/*.h) change only in their own small PR.
Nothing generated is committed; protoc runs at build time on both sides.
At most two PRs in review at once; the next stays a draft.
Each PR brings its tests: a smoke category on the C side, mvn -B verify on the Java side, a CI job for any service it needs.
Everything between the DDL skeleton (#1842) and a lake table that can be written and read from SQL. Design: #1683. This issue is the execution view.
Landed: #1842, the skeleton (AM, catalog/volume FDWs, stub engine, DDL guards); #1951, the Parquet format layer (Arrow, field ids, tracked memory pool, type rules).
Twelve PRs, three waves
C and D need A, B2 and B4; with the Builtin catalog they run in CI against MinIO and the agent alone. B5–B7 need B3 and run alongside C and D.
Attached to the PR they belong with: #1988 NUMERIC (C), #1990 timestamp units (C), #1989 name mapping (D). Later, blocking nothing: the optional
datalake_proxybgworker (#1683 §5.4).Rules
mainand are squash-merged; the description is the commit message (datalake_fdw: read and write Parquet through Arrow #1951 layout).format/format.h,meta/iceberg_meta_engine.h,common/*.h) change only in their own small PR.mvn -B verifyon the Java side, a CI job for any service it needs.datalake_fdw: Iceberg read/write MVP, labeldatalake.Out of scope: the FDW raw-file path (#1683 §2.3), the CI build image (devops repo), HDFS storage, execution-engine work.