Skip to content

datalake_fdw: storage I/O over object storage (S3) #2009

Description

@MisterRaindrop

Part of #2008. Letters (A, B0–B7, C, D, E) are the PRs listed there; this is A.

Scope

  • An S3 backend (AWS SDK for C++) in backend_registry, configured from the volume server and user-mapping options the validator already accepts (endpoint, region, path style, credentials, session token).
  • Arrow RandomAccessFile / OutputStream adapters over common/file_system_wrapper.h; parquet_read.cpp and parquet_write.cpp are the only format files that change.
  • The writer's rule carried over: an object is created once, never overwritten, and a failed writer removes exactly that object.
  • A storage_s3 smoke category against MinIO, and the CI job that starts MinIO (the CATEGORY_SERVICES hook in run_smoke_tests.sh).

Out of scope

HDFS (libhdfs3 is unmaintained; the wrapper leaves room for a second backend). Iceberg path layout (C).

Depends on

Nothing. Starts first.

Acceptance

  • The datalake_fdw: read and write Parquet through Arrow #1951 type set round-trips through s3:// against MinIO in CI.
  • A write aborted after the first row group leaves no object; an existing key is refused.
  • Credentials never appear in logs, errors or EXPLAIN.
  • nm -D exports only the PostgreSQL entry points; Arrow 9 and 17 build.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

datalakecontrib/datalake_fdw and contrib/datalake_agent: Iceberg lake tables

Type

No type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions