From 5a4a0a67796d21059484de9f6e7abddbf32af1db Mon Sep 17 00:00:00 2001 From: Joel Shepherd Date: Mon, 19 Oct 2026 21:06:57 +0000 Subject: [PATCH 1/2] docs: document the experimental Cassandra storage backend --- AGENTS.md | 48 +++-- README.md | 63 ++++-- docs/cassandra-setup.md | 285 ++++++++++++++++++++++++++ docs/getting-started.md | 63 +++++- docs/manuals/01-architecture-guide.md | 79 +++++-- docs/manuals/02-design-guide.md | 12 +- docs/manuals/11-deployment-guide.md | 145 ++++++++----- 7 files changed, 571 insertions(+), 124 deletions(-) create mode 100644 docs/cassandra-setup.md diff --git a/AGENTS.md b/AGENTS.md index 56ec61178..9b0d32e52 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -4,13 +4,13 @@ This document provides AI agents with essential context for working on the Exten ## Project Overview -**ExtendDB** is an open source DynamoDB-compatible runtime backed by PostgreSQL storage, released under the Apache 2.0 +**ExtendDB** is an open source DynamoDB-compatible runtime with pluggable storage, released under the Apache 2.0 License. ExtendDB is a clean-room implementation of the DynamoDB wire protocol, written in Rust and initially developed by AWS engineers. It is not a fork of DynamoDB and contains no DynamoDB source code. ExtendDB speaks the DynamoDB wire protocol: any AWS SDK, CLI, or tool that works with DynamoDB works with ExtendDB, unchanged. - **Language:** Rust (edition 2024, MSRV 1.88+) -- **Storage backends:** PostgreSQL 14+ (default), SQLite (bundled, no server or client install required; feature flag `sqlite`, also `sqlite-memory` for dev and CI), MongoDB 7.0+ (feature flag `mongodb`). One backend per binary, selected by mutually exclusive Cargo features. +- **Storage backends:** PostgreSQL 14+ (default), SQLite (bundled, no server or client install required; feature flag `sqlite`, also `sqlite-memory` for dev and CI), MongoDB 7.0+ (feature flag `mongodb`), Cassandra 4.1+ (experimental; feature flag `cassandra`). One backend per binary, selected by mutually exclusive Cargo features. - **Architecture:** Async (tokio), trait-based storage abstraction - **Authentication:** Mandatory SigV4 with built-in IAM (users, groups, roles, policies) - **TLS:** Mandatory (self-signed cert generated by default) @@ -28,18 +28,19 @@ protocol: any AWS SDK, CLI, or tool that works with DynamoDB works with ExtendDB ``` extenddb/ ├── crates/ +│ ├── app/ # Shared CLI surface the thin bin dispatches to +│ ├── auth/ # SigV4 verification, IAM policy engine +│ ├── bin/ # CLI, config, daemon lifecycle (extenddb binary) +│ ├── cache/ # Auth and authz caches +│ ├── config/ # Config parsing shared by the CLI and the server │ ├── core/ # Pure sync types, validation, expression parsing (no async) │ ├── engine/ # DynamoDB operation handlers (PutItem, Query, etc.) +│ ├── server/ # HTTP server, management API, web console │ ├── storage/ # Storage trait definitions (TableEngine trait) -│ ├── storage-postgres/ # PostgreSQL implementation of TableEngine -│ ├── storage-sqlite/ # SQLite implementation (dev, CI, and embedded use) +│ ├── storage-cassandra/ # Cassandra implementation (experimental; feature flag) │ ├── storage-mongodb/ # MongoDB implementation (feature flag) -│ ├── cache/ # Auth and authz caches -│ ├── config/ # Config parsing shared by the CLI and the server -│ ├── app/ # Shared CLI surface the thin bin dispatches to -│ ├── auth/ # SigV4 verification, IAM policy engine -│ ├── server/ # HTTP server, management API, web console -│ └── bin/ # CLI, config, daemon lifecycle (extenddb binary) +│ ├── storage-postgres/ # PostgreSQL implementation (default/reference) +│ └── storage-sqlite/ # SQLite implementation (dev, CI, and embedded use) ├── tests/ # Python integration tests (pytest) ├── docs/ │ ├── design/ # Architecture and component design docs @@ -59,6 +60,7 @@ The `TableEngine` trait in `crates/storage/src/lib.rs` defines the storage inter - `storage-postgres` (PostgreSQL) — default backend - `storage-sqlite` (SQLite) — feature flag `sqlite`, and `sqlite-memory` for dev and CI - `storage-mongodb` (MongoDB) — feature flag `mongodb` +- `storage-cassandra` (Cassandra) — experimental; feature flag `cassandra` The trait uses RPITIT (return-position impl Trait in traits) for async methods — no `#[async_trait]` macro. @@ -73,9 +75,10 @@ extenddb (bin) │ ├─> extenddb-core (pure sync, no async) │ └─> extenddb-storage (trait definitions) ├─> extenddb-auth - ├─> extenddb-storage-postgres (feature: postgres) - ├─> extenddb-storage-sqlite (feature: sqlite / sqlite-memory) - └─> extenddb-storage-mongodb (feature: mongodb) + ├─> extenddb-storage-postgres (feature: postgres) + ├─> extenddb-storage-sqlite (feature: sqlite / sqlite-memory) + ├─> extenddb-storage-mongodb (feature: mongodb) + └─> extenddb-storage-cassandra (feature: cassandra) ``` - **extenddb-core:** Pure synchronous Rust. No async, no I/O. Types, validation, expression parsing. @@ -84,6 +87,7 @@ extenddb (bin) - **extenddb-storage-sqlite:** Concrete SQLite implementation, the dev and CI backend. It serves the same wire surface, vector search included, and needs no server. - **extenddb-storage-mongodb:** Concrete MongoDB implementation. +- **extenddb-storage-cassandra:** Concrete Cassandra implementation. Experimental. - **extenddb-engine:** Operation handlers that call storage traits. - **extenddb-server:** HTTP server, management API, web console. - **extenddb-auth:** SigV4 signature verification, IAM policy evaluation. @@ -96,7 +100,9 @@ extenddb (bin) - Rust 1.88+ (`rustup update`) - Storage backend (one of): - PostgreSQL 14+ running locally (see `docs/local-postgres-setup.md`) + - SQLite (embedded; nothing to install) - MongoDB 7.0+ with replica set (see `docs/local-mongodb-setup.md`) + - Cassandra 4.1+ (experimental; see `docs/cassandra-setup.md`) - Python 3.10+ for tests (`python3 -m venv ~/venvs/extenddb-venv && source ~/venvs/extenddb-venv/bin/activate && pip install -r requirements.txt`) ### Build @@ -107,11 +113,14 @@ cargo build --release # MongoDB backend (backends are mutually exclusive — disable default postgres) cargo build --release --no-default-features --features mongodb + +# Cassandra backend (experimental) +cargo build --release --no-default-features --features cassandra ``` Backends are selected by mutually exclusive Cargo features (`postgres`, -`mongodb`, `sqlite`); a build enabling more than one is rejected at compile -time. Each binary serves exactly one backend. +`mongodb`, `sqlite`, `cassandra`); a build enabling more than one is rejected at +compile time. Each binary serves exactly one backend. Binary: `target/release/extenddb` @@ -136,6 +145,9 @@ cargo clippy --all-targets -- -D warnings # MongoDB ./target/release/extenddb init --backend mongodb --config extenddb.toml + +# Cassandra (experimental) +./target/release/extenddb init --backend cassandra --config extenddb.toml ``` This creates: @@ -339,7 +351,7 @@ Query, Scan (key conditions, filters, projections, pagination, index selection) BatchGetItem (100 keys), BatchWriteItem (25 ops), TransactGetItems (100 items), TransactWriteItems (100 ops) ### Vector Search -SearchVectors, and vector indexes on CreateTable and UpdateTable (COSINE, EUCLIDEAN, DOT_PRODUCT; on-demand tables only; five per table). Served by the PostgreSQL and SQLite backends; MongoDB provides no implementation and refuses every vector operation. PostgreSQL additionally requires the pgvector extension on the data database, probed once at startup, and refuses without it. The refusal is two strings, which matters when grepping logs: `Vector indexes are not supported by this storage backend` for CreateTable and UpdateTable, and `SearchVectors is not supported by this storage backend` for a search. +SearchVectors, and vector indexes on CreateTable and UpdateTable (COSINE, EUCLIDEAN, DOT_PRODUCT; on-demand tables only; five per table). Served by the PostgreSQL and SQLite backends; MongoDB and Cassandra provide no implementation and refuse every vector operation. PostgreSQL additionally requires the pgvector extension on the data database, probed once at startup, and refuses without it. The refusal is two strings, which matters when grepping logs: `Vector indexes are not supported by this storage backend` for CreateTable and UpdateTable, and `SearchVectors is not supported by this storage backend` for a search. ### Streams ListStreams, DescribeStream, GetShardIterator, GetRecords @@ -387,8 +399,10 @@ Expression parsing lives in `crates/core/src/expression/`. This is pure sync Rus | Troubleshooting | `docs/troubleshooting.md` | Common errors and solutions | | Storage Component Design | `docs/design/04-component-storage.md` | Storage trait design | | MongoDB Design | `docs/design/13-storage-mongodb.md` | MongoDB backend architecture | +| Cassandra Design (experimental) | `docs/adr/0010`–`0021` | Cassandra backend design decisions (ADRs) | | Local PostgreSQL Setup | `docs/local-postgres-setup.md` | PostgreSQL installation and config | | Local MongoDB Setup | `docs/local-mongodb-setup.md` | MongoDB installation and config | +| Cassandra Setup | `docs/cassandra-setup.md` | Cassandra install, config, and deployment (experimental) | | Testing Design | `docs/design/09-testing.md` | Test strategy and infrastructure | | Extending Storage | `12-backend-plugin-architecture.md` | Guide to implementing storage backends | @@ -480,7 +494,7 @@ Activate when the user asks about installing, configuring, running, or debugging 1. **TLS is mandatory** — server refuses to start without it. Use `AWS_CA_BUNDLE` for self-signed certs. 2. **Auth is mandatory** — all requests must be SigV4-signed. Use `extenddb manage` or web console to create credentials. 3. **Account isolation** — all operations are scoped to `account_id`. Different accounts can have tables with the same name. -4. **Storage backend must be running** — `extenddb init` and `extenddb serve` require a running PostgreSQL or MongoDB instance. +4. **Storage backend must be running** — `extenddb init` and `extenddb serve` require a running external backend (PostgreSQL, MongoDB, or Cassandra). SQLite is embedded and needs no server. 5. **MongoDB requires a replica set** — even single-node MongoDB must be configured with `--replSet` for transactions and streams. 6. **Python venv** — activate the venv before running tests: `source ~/venvs/extenddb-venv/bin/activate` 7. **Test credentials** — run `devtools/provision-test-credentials` before pytest to create test users and keys. diff --git a/README.md b/README.md index a6e432e21..fe5c2412e 100755 --- a/README.md +++ b/README.md @@ -1,27 +1,31 @@ # ExtendDB -> **ExtendDB is an independent open source project managed by Amazon Web Services. It is not Amazon DynamoDB and does not contain any DynamoDB source code.** "DynamoDB" is a trademark of Amazon.com, Inc. ExtendDB is a clean-room implementation that speaks the DynamoDB wire protocol. Behavioral differences from the real service are documented in [Differences from DynamoDB](docs/differences-from-dynamodb.md). +> **ExtendDB is an independent open source project managed by Amazon Web Services. It is not Amazon DynamoDB and does +> not contain any DynamoDB source code.** "DynamoDB" is a trademark of Amazon.com, Inc. ExtendDB is a clean-room +> implementation that speaks the DynamoDB wire protocol. Behavioral differences from the real service are documented +> in [Differences from DynamoDB](docs/differences-from-dynamodb.md). -A DynamoDB-compatible API adapter, ExtendDB speaks the DynamoDB wire protocol — any AWS SDK, CLI, or tool that works with DynamoDB works with ExtendDB, unchanged. +A DynamoDB-compatible API adapter, ExtendDB speaks the DynamoDB wire protocol — any AWS SDK, CLI, or tool that works +with DynamoDB works with ExtendDB, unchanged. ## Use Cases - **Local development** — run DynamoDB workloads on your laptop with zero cloud dependency - **CI/CD pipelines** — deterministic integration tests against a DynamoDB-compatible backend - **Self-hosted deployments** — run DynamoDB workloads on your own infrastructure (on-premises, private cloud, edge) -- **Multi-cloud** — use DynamoDB semantics on any cloud that runs PostgreSQL +- **Multi-cloud** — use DynamoDB semantics on any cloud that runs an ExtendDB-supported database backend - **Air-gapped environments** — DynamoDB functionality with no internet connectivity ## Features - Full DynamoDB wire protocol: CRUD, Query, Scan, Batch, Transactions, Streams, TTL, Import/Export +- Pluggable storage: PostgreSQL (default), SQLite, MongoDB, and Cassandra - SigV4 authentication with IAM compatibility: users, groups, roles, policies, permissions boundaries - Web management console for account and credential administration - TLS with automatic self-signed certificate generation (replaceable with CA-signed certs) - CSRF protection, security headers, session management - JSON metrics endpoint with DynamoDB CloudWatch-style metric names and dimensions - Daemon mode with syslog logging, plus `--foreground` for container and supervisor environments -- PostgreSQL storage — use standard backup, replication, and HA tools ## Quick Start @@ -50,8 +54,12 @@ scripts/install-macos.sh # macOS ## Prerequisites - Rust 1.88+ (`rustup update`) -- PostgreSQL 14+ (see `docs/local-postgres-setup.md`) - Python 3.10+ (for test suites and documentation) +- A storage backend: + - PostgreSQL 14+ (see `docs/local-postgres-setup.md`) + - SQLite (embedded) + - MongoDB 7+ (see `docs/local-mongodb-setup.md`) + - Cassandra 4.1+ (see `docs/cassandra-setup.md`) ### Python Environment @@ -106,11 +114,10 @@ key_path = "/etc/extenddb/tls/key.pem" ## Running in Containers -The production images contain ExtendDB only; the database (PostgreSQL 14+ and MongoDB 7+) -runs as a separate service. See [`docker/README.md`](docker/README.md) for the PostgreSQL -image and [`docker/README-mongodb.md`](docker/README-mongodb.md) for the MongoDB -image, external database requirements, hardened local Compose stacks, smoke tests, and -explicit init/migrate/serve production lifecycle. +The production images contain ExtendDB only. The storage backend runs as a separate service. See +[`docker/README.md`](docker/README.md) for the PostgreSQL image and [`docker/README-mongodb.md`](docker/README-mongodb.md) for the MongoDB image, +external database requirements, hardened local Compose stacks, smoke tests, and explicit init/migrate/serve +production lifecycle. The SQLite and Cassandra backends have no dedicated container guide yet. By default `extenddb serve` daemonizes itself, which doesn't play well with container runtimes and process supervisors that expect PID 1 to stay attached. Pass `--foreground` (alias `--no-daemon`) to keep the process in the foreground and stream logs to stderr instead of syslog: @@ -134,11 +141,20 @@ extenddb healthcheck --config extenddb.toml # exit 0 healthy, 1 not extenddb healthcheck --endpoint https://127.0.0.1:18443 # explicit target ``` -This is a liveness check, which is what a container `HEALTHCHECK` wants: `/health` does not query the storage backend, so it reports healthy even if PostgreSQL becomes unreachable after startup. That is deliberate, since a liveness probe that failed on a database outage would restart every replica at once. A backend that is unreachable at startup does stop the server from listening, so that case is caught. There is no separate readiness endpoint yet. +This is a liveness check, which is what a container `HEALTHCHECK` wants. The `/health` path does not query the storage +backend, so it will report healthy even if the storage backend becomes unreachable after startup. That is deliberate, +since a liveness probe that failed on a database outage would restart every replica at once. A backend that is +unreachable at startup does stop the server from listening. There is no separate readiness endpoint yet. ### One instance per database -Run one `extenddb serve` per catalog. On SQLite this is enforced: the server takes an exclusive lock on `.lock` next to the database file and a second server on the same file refuses to start (`init`, `migrate`, and `destroy` take the same lock while they run). On PostgreSQL and MongoDB nothing prevents a second instance, but running more than one against the same catalog is not supported in this release: every instance runs every background worker, credential and policy caches are per instance with no cross-instance invalidation (a revoked key stays valid on the other instance for up to the cache TTL), and `/health` does not check the database. See the [deployment guide](docs/manuals/11-deployment-guide.md#multi-instance-considerations). +Run one `extenddb serve` per catalog. On SQLite this is enforced: the server takes an exclusive lock on `.lock` +next to the database file and a second server on the same file will refuse to start (`init`, `migrate`, and `destroy` take +the same lock while they run). On PostgreSQL, MongoDB and Cassandra nothing prevents a second ExtendDB instance from +running, but running more than one against the same catalog is not supported in this release: every instance runs every +background worker, credential and policy caches are per instance with no cross-instance invalidation (a revoked key +stays valid on the other instance for up to the cache TTL), and `/health` does not check the database. +See the [deployment guide](docs/manuals/11-deployment-guide.md#multi-instance-considerations). To make the generated self-signed certificate valid for the name clients use — an in-cluster service DNS name, for example — pass `--tls-san` to `init` (repeatable): @@ -174,7 +190,7 @@ Web-based administration at `https://127.0.0.1:18443/console/`. Manage accounts, ``` extenddb serve --config extenddb.toml # Start server (daemon) extenddb serve --config extenddb.toml --foreground # Start in foreground -extenddb init --catalog-db NAME # Initialize deployment +extenddb init --catalog-db NAME # Initialize deployment extenddb stop --config extenddb.toml # Graceful shutdown extenddb status --config extenddb.toml # Check if running extenddb healthcheck --config extenddb.toml # Probe /health (exit 0 healthy, 1 not) @@ -214,17 +230,22 @@ UpdateTimeToLive, DescribeTimeToLive, TagResource, UntagResource, ListTagsOfReso ``` crates/ - core/ — types, validation, expressions (pure sync Rust, no async) - engine/ — operation handlers - storage/ — storage trait definitions - storage-postgres/ — PostgreSQL backend - auth/ — SigV4 verification, IAM policy engine - server/ — HTTP server, management API, web console - bin/ — CLI, config, daemon lifecycle + auth/ — SigV4 verification, IAM policy engine + bin/ — CLI, config, daemon lifecycle + core/ — types, validation, expressions (pure sync Rust, no async) + engine/ — operation handlers + server/ — HTTP server, management API, web console + storage/ — storage trait definitions + storage-cassandra/ — Cassandra backend (experimental) + storage-mongodb/ — MongoDB backend + storage-postgres/ — PostgreSQL backend (default/reference) + storage-sqlite/ — SQLite backend (embedded) + + docs/ + adr/ — architecture decision records design/ — architecture and design documents manuals/ — user-facing guides (PDF pipeline) - adr/ — architecture decision records ``` ## Documentation diff --git a/docs/cassandra-setup.md b/docs/cassandra-setup.md new file mode 100644 index 000000000..aba504847 --- /dev/null +++ b/docs/cassandra-setup.md @@ -0,0 +1,285 @@ +# Cassandra Setup + +> See [NOTICE](NOTICE.md) for important disclaimers. + +> **Experimental.** As of October 2026, the Cassandra backend is experimental. +> Validate it against your workload before you rely on it in production. This +> guide covers both local development and production deployment. + +## Prerequisites + +- Apache Cassandra 4.1 or later +- Docker (for the local single-node setup below), or an existing Cassandra + cluster +- The `cqlsh` client, for the verification steps (bundled in the Cassandra + Docker image) + +## Local Single-Node Setup (Docker) + +These steps start a single-node Cassandra 4.1 container for local development. +They mirror `devtools/cassandra-node`, which the project uses for its own tests. + +1. Start the container: + + ```bash + docker run -d --name cassandra -p 9042:9042 \ + -e CASSANDRA_CLUSTER_NAME=extenddb-dev \ + cassandra:4.1 + ``` + +2. Wait for the node to accept CQL. The port opens before the database is + ready, so poll with `cqlsh` until it succeeds: + + ```bash + until docker exec cassandra cqlsh -e "SELECT release_version FROM system.local" >/dev/null 2>&1; do + sleep 2 + done + ``` + +3. Enable password authentication. The `cassandra:4.1` image ignores the + `CASSANDRA_AUTHENTICATOR` environment variable, so edit `cassandra.yaml` + directly and restart: + + ```bash + docker exec cassandra sed -i \ + 's/^authenticator: .*/authenticator: PasswordAuthenticator/' \ + /etc/cassandra/cassandra.yaml + docker restart cassandra + ``` + +4. Wait for authentication to come up, then confirm the default superuser login + works. Apache Cassandra's publicly documented default credentials are + `cassandra` / `cassandra`: + + ```bash + until docker exec cassandra cqlsh -u cassandra -p cassandra \ + -e "SELECT release_version FROM system.local" >/dev/null 2>&1; do + sleep 2 + done + ``` + +The node now listens on `127.0.0.1:9042` with password authentication enabled. + +> **Note:** Password authentication is optional for a purely local node. If you +> skip step 3, the node runs with the default `AllowAllAuthenticator` (no +> authentication), and you can run `extenddb init --backend cassandra` with no +> credential flags. The steps above enable authentication because production +> clusters require it, and testing against an authenticated node locally avoids +> surprises later. + +## Connection Details + +| Setting | Value | +|---------|-------| +| Contact point | `127.0.0.1:9042` | +| Superuser (default) | `cassandra` / `cassandra` | +| Keyspace prefix | `extenddb` (default) | +| Catalog keyspace | `_catalog` (e.g. `extenddb_catalog`) | +| Per-account keyspaces | `_account_` | + +ExtendDB creates its keyspaces under the configured `keyspace_prefix`, so one +cluster can host more than one deployment by giving each a different prefix. + +## Building with Cassandra Support + +The Cassandra backend is behind a feature flag. Backends are mutually exclusive +(a build enabling more than one is rejected at compile time), so disable the +default `postgres` feature: + +```bash +cargo build --release --no-default-features --features cassandra +``` + +The binary is single-backend. To run Cassandra and another backend side by +side, build one binary per backend. + +## Initializing ExtendDB with Cassandra + +Initialize against the authenticated local node from above. Pass the contact +point and the superuser credentials used to create the keyspaces and the +application role: + +```bash +./target/release/extenddb init \ + --backend cassandra \ + --cassandra-contact-points 127.0.0.1:9042 \ + --cassandra-user cassandra \ + --cassandra-pass cassandra \ + --config extenddb.toml +``` + +If your node runs without authentication (you skipped step 3 above), omit the +credential flags: + +```bash +./target/release/extenddb init --backend cassandra --config extenddb.toml +``` + +init creates: +- The catalog keyspace (`extenddb_catalog` by default) +- The application role ExtendDB uses at serve time +- Admin user credentials (printed to stdout once — save them) +- A self-signed TLS certificate at `~/.extenddb/tls/cert.pem` +- The config file `extenddb.toml` + +Per-account keyspaces are created later, when accounts are added. + +> **Save the admin credentials.** They are printed once and cannot be retrieved +> later. + +## Config Mapping + +init writes a `[storage.cassandra]` section to `extenddb.toml`: + +```toml +[storage] +backend = "cassandra" + +[storage.cassandra] +contact_points = ["127.0.0.1:9042"] +# username = "cassandra" # Application user +# password = "cassandra-password" # Application password +keyspace_prefix = "extenddb" +replication_factor = 1 # Single node (use 3 or more for production) +datacenter = "datacenter1" +max_connections = 10 +``` + +Fields: + +- `contact_points` — one or more `host:port` entries. One reachable node is + enough; the driver discovers the rest of the cluster. +- `username` / `password` — the application credentials ExtendDB uses at serve + time. Omit both for an unauthenticated node. +- `keyspace_prefix` — prefix for this deployment's keyspaces (default + `extenddb`). +- `replication_factor` — keyspace replication factor. `1` suits a single local + node; use 3 or more in production. +- `datacenter` — the local datacenter name for `NetworkTopologyStrategy` + (default `datacenter1`, which matches a stock single-node cluster). +- `max_connections` — maximum connections per host. + +Edit these values in `extenddb.toml`. Unlike the PostgreSQL and MongoDB +backends, the Cassandra backend has no single connection-string field; set +`contact_points` and the other fields directly in the config section above. + +## Starting the Server + +```bash +./target/release/extenddb serve --config extenddb.toml +``` + +## Verifying the Connection + +```bash +# Health check +curl --cacert ~/.extenddb/tls/cert.pem https://127.0.0.1:18443/health + +# List tables (should return empty on a fresh deployment) +export AWS_CA_BUNDLE=~/.extenddb/tls/cert.pem +aws dynamodb list-tables \ + --endpoint-url https://127.0.0.1:18443 \ + --region us-east-1 +``` + +You can also confirm the keyspaces exist directly in Cassandra: + +```bash +docker exec cassandra cqlsh -u cassandra -p cassandra \ + -e "DESCRIBE KEYSPACES" | tr ' ' '\n' | grep '^extenddb' +``` + +## Stopping + +```bash +# ExtendDB +./target/release/extenddb stop --config extenddb.toml + +# Docker Cassandra +docker stop cassandra +``` + +## Cluster Deployment + +For deployment concerns common to every backend — single-node vs separated +topologies, containers, the production checklist, systemd, multi-instance +limits — see the [Deployment Guide](manuals/11-deployment-guide.md). This +section covers only what is specific to Cassandra. + +> **Not recommended for production yet.** As of October 2026 the Cassandra +> backend is experimental. It has known limitations — some multi-entity +> management operations are not atomic, and a few concurrency issues are still +> open (see Known Limitations below). Treat a Cassandra deployment as +> evaluation or pre-production, and test it against your own workload before you +> depend on it. + +### Keyspace replication + +Set `replication_factor` to 3 or more for a multi-node cluster, and set +`datacenter` to your cluster's local datacenter name. ExtendDB creates its +keyspaces with `NetworkTopologyStrategy` using these values. + +```toml +[storage.cassandra] +contact_points = ["cass-1:9042", "cass-2:9042", "cass-3:9042"] +username = "extenddb" +password = "" +keyspace_prefix = "extenddb" +replication_factor = 3 +datacenter = "dc1" +max_connections = 10 +``` + +### Authentication + +Run the cluster with authentication enabled (`PasswordAuthenticator` or +stronger), and give ExtendDB a dedicated application role rather than the +`cassandra` superuser. Supply its credentials through `username` and `password`. +ExtendDB connects without authentication when both are absent, so confirm they +are set for any cluster that is not fully trusted. + +### Known Limitations + +Understand these before you evaluate a Cassandra deployment. They are why the +backend is experimental rather than production-ready. + +- **Non-atomic management operations.** Some multi-entity management operations + (for example across the IAM entities) are not all-or-nothing on Cassandra. + Cassandra has no multi-row ACID transaction, and ExtendDB uses lightweight + transactions (LWT) for referential integrity, preventing the use of logged + batches for atomicity. A failure part-way through such an operation can leave + partial state. +- **Open concurrency issues.** A small number of concurrency/race conditions + are known (observed as intermittent integration-test failures) and are being + worked. Under concurrent load, outcomes may occasionally diverge from + DynamoDB semantics. +- **Single-datacenter writes per table.** Transaction and TTL lifecycle + operations use `LOCAL_QUORUM` / `LOCAL_SERIAL`, which are linearizable only + within one datacenter. Serve writes for a given table from one Cassandra + datacenter. +- **Multi-node ExtendDB needs node-id configuration.** Running more than one + ExtendDB instance against the cluster requires a distinct node ID per + instance (used to derive stream sequence numbers). +- **TTL throughput.** The TTL sweeper is sized for roughly 1,000 expirations + per table per minute; a table expiring faster accumulates backlog. Enable TTL + before opening a table to writes, or quiesce it briefly while enable and + backfill complete. +- **Clock synchronization.** Keep Cassandra coordinators and ExtendDB hosts + time-synchronized; request batches use explicit timestamps. +- **No vector search.** The Cassandra backend does not implement vector indexes + or `SearchVectors`; every vector operation is refused. + +If you do evaluate a deployment, monitor the TTL expiration backlog, Cassandra +LWT latency, and background-worker health. The design decisions and their +rationale are documented in the Cassandra ADRs (`docs/adr/0010`–`0021`). + +--- + +## License + +Copyright 2026 ExtendDB contributors. Licensed under the Apache License, Version 2.0. +See [LICENSE](../LICENSE) for the full text. + +This software is provided "as is" without warranty of any kind. ExtendDB is not +affiliated with, endorsed by, or sponsored by Amazon Web Services. "DynamoDB" is a trademark +of Amazon.com, Inc. diff --git a/docs/getting-started.md b/docs/getting-started.md index ffbbc9b3f..ef626f24c 100755 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -38,7 +38,9 @@ See [Local Development with Docker Compose](local-dev-docker-compose.md). - **Storage backend** (one of): - PostgreSQL 14+ running locally (see `docs/local-postgres-setup.md`) + - SQLite (embedded; nothing to install) - MongoDB 7.0+ with replica set (see `docs/local-mongodb-setup.md`) + - Cassandra 4.1+ running locally (see `docs/cassandra-setup.md`) - Rust toolchain (1.88+) - AWS CLI v2 (for testing) - Python 3.10+ with virtual environment (see [Python Environment Setup](../README.md#python-environment-setup) in the README) @@ -52,12 +54,15 @@ cargo build --release # MongoDB backend # (backends are mutually exclusive; disable the default postgres feature) cargo build --release --no-default-features --features mongodb + +# Cassandra backend +cargo build --release --no-default-features --features cassandra ``` -Each build produces a single-backend binary. The `postgres`, `mongodb`, and -`sqlite` features are mutually exclusive — a build enabling more than one is -rejected at compile time — so to run two backends, build one binary per -backend. +Each build produces a single-backend binary. The `postgres`, `mongodb`, +`sqlite`, and `cassandra` features are mutually exclusive — a build enabling +more than one is rejected at compile time — so to run two backends, build one +binary per backend. The binary is at `target/release/extenddb`. @@ -71,11 +76,14 @@ Run `extenddb init` to create the catalog and data databases: # MongoDB ./target/release/extenddb init --backend mongodb + +# Cassandra +./target/release/extenddb init --backend cassandra ``` This will: -- Create the `extenddb_catalog` database (catalog metadata) -- Create the data database (user item data) +- Create the `extenddb_catalog` database/keyspace (catalog metadata) +- Create the data database/keyspace (user item data) - Generate an AES-256-GCM encryption key (for access key storage) - Create a default account and print the account ID - Create an `admin` user and print the credentials once @@ -157,6 +165,49 @@ EXTENDDB__STORAGE__SQLITE__PATH=/data/other.sqlite ./target/release/extenddb ser Use `:memory:` (in the config, the environment variable, or `init --sqlite-path`) for an ephemeral database that is discarded on shutdown. +### Cassandra backend + +ExtendDB can run against an Apache Cassandra cluster. The Cassandra backend is +experimental; see [Cassandra setup](cassandra-setup.md) for its current +limitations. Select it at init time with `--backend cassandra`: + +```bash +./target/release/extenddb init --backend cassandra +``` + +For a local single-node cluster with authentication disabled, this needs no +other flags. init uses development defaults: it contacts `localhost:9042`, +creates keyspaces under the prefix `extenddb`, and sets a replication factor of +1. It connects without authentication, which matches a stock Cassandra node +(the default `AllowAllAuthenticator`). If the node has authentication enabled +(for example `PasswordAuthenticator`), pass `--cassandra-user` and +`--cassandra-pass`. init writes a `[storage.cassandra]` section to the config: + +```toml +[storage.cassandra] +contact_points = ["localhost:9042"] +# username = "cassandra" # Application user +# password = "cassandra-password" # Application password +keyspace_prefix = "extenddb" +replication_factor = 1 # Single node (use 3+ for production) +datacenter = "datacenter1" +max_connections = 10 +``` + +To override a default, pass the matching flag at init time: + +- `--cassandra-contact-points host1:9042,host2:9042` — one or more contact + points for a remote or multi-node cluster. +- `--cassandra-user` and `--cassandra-pass` — admin credentials, if the cluster + requires authentication to create the keyspaces and the application role. +- `--keyspace-prefix myapp` — a different keyspace prefix, so one cluster can + host more than one deployment. +- `--replication-factor 3` — a higher replication factor for a multi-node + cluster. + +For replication strategy, datacenter topology, authentication, and production +tuning, see `docs/cassandra-setup.md`. + ### Generating a self-signed certificate manually `extenddb init` auto-generates a self-signed TLS certificate at `~/.extenddb/tls/`. If you need a certificate with different SANs (e.g., binding to `0.0.0.0` and connecting via a specific hostname), generate one manually with `openssl`: diff --git a/docs/manuals/01-architecture-guide.md b/docs/manuals/01-architecture-guide.md index ddcfc6fb8..64b08d672 100755 --- a/docs/manuals/01-architecture-guide.md +++ b/docs/manuals/01-architecture-guide.md @@ -4,24 +4,35 @@ ## Overview -extenddb (ExtendDB) is a standalone DynamoDB-compatible API server written in Rust. It receives DynamoDB wire protocol requests over HTTP/HTTPS, authenticates and authorizes them via SigV4 and a local IAM policy engine, executes operation logic in a backend-agnostic engine, and delegates persistence to a pluggable storage backend (currently PostgreSQL). +extenddb (ExtendDB) is a standalone DynamoDB-compatible API server written in Rust. It receives DynamoDB wire protocol +requests over HTTP/HTTPS, authenticates and authorizes them via SigV4 and a local IAM policy engine, executes operation +logic in a backend-agnostic engine, and delegates persistence to a pluggable storage backend (PostgreSQL, SQLite, +MongoDB, or Apache Cassandra). -extenddb runs as a daemon process, logging to syslog. It is designed for any environment where DynamoDB semantics are needed — local development, CI pipelines, self-hosted production, multi-cloud, or air-gapped deployments. Developers and applications point their AWS SDKs at extenddb and get identical DynamoDB behavior. +extenddb runs as a daemon process, logging to syslog. It is designed for any environment where DynamoDB semantics are +needed: local development, CI pipelines, self-hosted production, multi-cloud, or air-gapped deployments. Developers and +applications point their AWS SDKs at extenddb and get identical DynamoDB behavior. ## Cargo Workspace -The project is structured as a Cargo workspace with 7 crates. Crate boundaries enforce dependency rules at compile time. +The project is structured as a Cargo workspace with 13 crates. Crate boundaries enforce dependency rules at compile time. ``` extenddb/ ├── crates/ -│ ├── core/ Pure sync Rust: types, expressions, validation, errors -│ ├── engine/ Async operation handlers (PutItem, Query, etc.) -│ ├── storage/ Storage trait definitions and backend-agnostic utilities -│ ├── storage-postgres/ PostgreSQL backend implementation -│ ├── auth/ AuthProvider trait, SigV4 verification, IAM policy engine -│ ├── server/ HTTP server (axum), management API, web console -│ └── bin/ CLI entry point, config loading, daemon lifecycle +│ ├── app/ Application/CLI library the bin crate dispatches to +│ ├── auth/ AuthProvider trait, SigV4 verification, IAM policy engine +│ ├── bin/ CLI entry point, config loading, daemon lifecycle +│ ├── cache/ Stale-while-revalidate cache used by the auth and server layers +│ ├── config/ Configuration types and loading +│ ├── core/ Pure sync Rust: types, expressions, validation, errors +│ ├── engine/ Async operation handlers (PutItem, Query, etc.) +│ ├── server/ HTTP server (axum), management API, web console +│ ├── storage/ Storage trait definitions and backend-agnostic utilities +│ ├── storage-cassandra/ Cassandra backend implementation (experimental) +│ ├── storage-mongodb/ MongoDB backend implementation +│ ├── storage-postgres/ PostgreSQL backend implementation (default) +│ └── storage-sqlite/ SQLite backend implementation (embedded) ``` ## Crate Dependency Graph @@ -34,11 +45,14 @@ bin ──→ server ──→ engine ──→ core │ ├──→ auth ──→ core │ └──→ storage │ - ├──→ storage-postgres ──→ storage ──→ core + ├──→ storage-{postgres,sqlite,mongodb,cassandra} ──→ storage ──→ core └──→ core ``` -Key principle: `core` depends on nothing in the workspace and has no async runtime. It contains pure sync Rust — types, expression parsing and evaluation, input validation, capacity calculation, and error types. Every other crate depends on `core`. +Exactly one `storage-*` backend is linked per binary, selected by mutually exclusive Cargo features at build time. + +Key principle: `core` depends on nothing in the workspace and has no async runtime. It contains pure sync Rust: types, +expression parsing and evaluation, input validation, capacity calculation, and error types. Every other crate depends on `core`. ## Crate Roles @@ -68,7 +82,7 @@ The `dispatch` function routes `X-Amz-Target` operation names to handlers. ### storage -Trait definitions for the storage layer. Thirteen required storage traits partition backend responsibilities: +Trait definitions for the storage layer. The required storage traits partition backend responsibilities: - **TableEngine**: Table lifecycle (create, delete, describe, list, update) - **DataEngine**: Item CRUD (put, get, update, delete, query, scan, batch, transact) @@ -86,9 +100,14 @@ Trait definitions for the storage layer. Thirteen required storage traits partit Two further traits are optional and belong to vector search: **VectorSearchEngine**, which a backend hands over through `as_vector_search` or declines by returning `None`, and **VectorIndexBuild**, the storage primitives the shared build lifecycle drives. A backend that implements neither refuses every vector operation and is otherwise unaffected. - **Bootstrapper**: Initial database setup -Traits use `BoxFuture` for object safety. A binary serves exactly one backend, installed from its thin `main` via `set_backend`. The `ServerRuntimeHooks` trait allows backends to spawn backend-specific workers (PostgreSQL spawns 7). +Traits use `BoxFuture` for object safety. A binary serves exactly one backend, installed from its thin `main` via `set_backend`. The `ServerRuntimeHooks` trait allows backends to spawn backend-specific workers. + +### Storage backends -### storage-postgres +Each backend implements the storage traits for one database. A built ExtendDB binary links exactly one. PostgreSQL +is the default and reference backend; Apache Cassandra is experimental; MongoDB and SQLite are also available. + +#### storage-postgres PostgreSQL implementation of all storage traits using `sqlx`. Features: @@ -100,6 +119,20 @@ PostgreSQL implementation of all storage traits using `sqlx`. Features: - Stream records stored in a dedicated table with background cleanup - All queries parameterized (no dynamic SQL construction) +#### storage-cassandra + +Cassandra implementation of the storage traits using `cdrs-tokio`. Experimental. +Features: + +- A catalog keyspace for metadata and a keyspace per account for items, under a configurable `keyspace_prefix` +- Items stored as JSON, keyed by Cassandra partition and clustering columns +- GSI/LSI implemented as separate index tables (see ADR-0016) +- Transactions use a lightweight-transaction (LWT) two-phase commit, with a background worker to recover abandoned transactions (see ADR-0017) +- Streams and TTL run from background workers, with a durable TTL expiration queue (see ADR-0010, ADR-0018) + +> **Note:** Cassandra has no multi-row ACID primitive. Some multi-entity operations trade all-or-nothing atomicity for +> the integrity that LWT provides (see ADR-0002, ADR-0021). + ### auth Authentication and authorization: @@ -153,7 +186,7 @@ expect. (skipped entirely in `--foreground` mode, so no run directory is needed and the root filesystem can be read-only) 4. Initialize logging — syslog when daemonized, stderr in `--foreground` -5. Connect to PostgreSQL (catalog + data databases) +5. Connect to the storage backend 6. Verify catalog version matches binary expectation 7. Start axum server on the pre-bound socket @@ -180,7 +213,11 @@ A mismatch between the compiled-in version and the stored one prevents the serve ### Storage -Storage backends implement thirteen traits (see **storage** section above). The traits use `BoxFuture` for object safety. A binary serves exactly one backend: the thin `bin` crate installs it via `set_backend` before any subcommand runs, and the config file's `backend` key is validated against the installed name rather than selecting it. PostgreSQL, SQLite and MongoDB backends are implemented, selected by mutually exclusive Cargo features at build time. +Storage backends implement the storage traits (see **storage** section above). The traits use `BoxFuture` for object safety. +A built ExtendDB binary serves exactly one backend. The thin `bin` crate installs the selected backend via `set_backend` +before any subcommand runs, and the config file's `backend` key is validated against the installed name rather than +selecting it. The PostgreSQL, SQLite, MongoDB, and Cassandra backends are selected as mutually exclusive Cargo features +at build time. ### Authentication @@ -190,7 +227,7 @@ Auth providers implement the `AuthProvider` trait using `#[async_trait]` for obj ### Authentication -SigV4 signature verification follows the AWS specification. Credentials are stored encrypted (AES-256-GCM) in PostgreSQL. Access key prefixes distinguish long-term (`VDAK`) from temporary (`VDSK`) credentials. +SigV4 signature verification follows the AWS specification. Credentials are stored encrypted (AES-256-GCM) in the storage backend. Access key prefixes distinguish long-term (`AKIA`) from temporary (`ASIA`) credentials. ### Authorization @@ -262,7 +299,7 @@ Configuration precedence: CLI flags > environment variables > config file > defa ## DynamoDB Streams -extenddb implements DynamoDB Streams for change data capture. Stream records are captured atomically with data writes inside the same PostgreSQL transaction. Both the DynamoDB API and Streams API are served on the same port. +extenddb implements DynamoDB Streams for change data capture. Stream records are captured atomically with the data writes that produce them. Both the DynamoDB API and Streams API are served on the same port. Supported operations: `ListStreams`, `DescribeStream`, `GetShardIterator`, `GetRecords`. @@ -270,14 +307,14 @@ Stream records are retained for 24 hours. A background worker cleans up expired ## Deployment Models -extenddb is a single-binary server that connects to PostgreSQL. Deployment options include: +extenddb is a single-binary server that connects to a storage backend. Deployment options include: - **Single-node**: extenddb + PostgreSQL on the same host (development, small workloads) - **Separated**: extenddb on an application server, PostgreSQL on a dedicated database server or managed service (RDS, Aurora, Cloud SQL) - **Containerized**: Docker/Kubernetes with PostgreSQL as a sidecar or external service - **Air-gapped**: No internet connectivity required; all functionality is self-contained -PostgreSQL provides the durability, replication, and backup capabilities. Use standard PostgreSQL HA tools (streaming replication, Patroni, managed services) for production availability. +The storage backend provides durability, replication, and backup. Use that backend's standard HA tooling for production availability. --- diff --git a/docs/manuals/02-design-guide.md b/docs/manuals/02-design-guide.md index 00d5541a0..69657f6cf 100755 --- a/docs/manuals/02-design-guide.md +++ b/docs/manuals/02-design-guide.md @@ -4,14 +4,16 @@ ## Storage Schema -### Dual-Database Architecture +### Catalog and Data Separation -extenddb uses two PostgreSQL databases per deployment: +ExtendDB manages two kinds of stored state in every storage backend. The +examples below name the PostgreSQL databases; other backends map the same two +roles onto their own structures (see backend-specific documentation). -- **Catalog database** (e.g., `extenddb_catalog`): All metadata — table definitions, indexes, accounts, IAM entities, settings, stream metadata, and schema history. -- **Data database** (e.g., `extenddb_catalog_data`): User item data. Each DynamoDB table maps to a PostgreSQL table. GSI and LSI data are stored in separate PostgreSQL tables. +- **Catalog** (e.g., the `extenddb_catalog` PostgreSQL database): All metadata — table definitions, indexes, accounts, IAM entities, settings, stream metadata, and schema history. +- **Data** (e.g., the `extenddb_catalog_data` PostgreSQL database): User item data. Each DynamoDB table maps to a PostgreSQL table. GSI and LSI data are stored in separate PostgreSQL tables. -The data database connection string is stored in the catalog's `settings` table under the key `data_database_url`. This allows the catalog and data databases to live on different PostgreSQL instances. +On PostgreSQL, the data database connection string is stored in the catalog's `settings` table under the key `data_database_url`. This allows the catalog and data databases to live on different PostgreSQL instances. ### Catalog Tables diff --git a/docs/manuals/11-deployment-guide.md b/docs/manuals/11-deployment-guide.md index fb03c271b..59b120c13 100755 --- a/docs/manuals/11-deployment-guide.md +++ b/docs/manuals/11-deployment-guide.md @@ -4,27 +4,36 @@ This guide covers deploying extenddb in various environments beyond local development. +> **Backend-specific sections.** This guide covers deployment concerns common to +> all storage backends, and uses PostgreSQL — the default and reference backend — +> for concrete examples. Sections headed *PostgreSQL* apply only to that backend. +> For other backends, see their setup docs: +> [Cassandra setup](../cassandra-setup.md) (experimental; local and production) +> and [MongoDB setup](../local-mongodb-setup.md). SQLite does not need a separate +> backend service; see its section in the getting-started guide. + ## Architecture Overview -extenddb is a single Rust binary that connects to PostgreSQL. All durable state lives in PostgreSQL; extenddb keeps only short-lived credential, policy, and table-metadata caches in process. This means: +extenddb is a single Rust binary that connects to a storage backend. All durable state lives in the backend; extenddb keeps only short-lived credential, policy, and table-metadata caches in process. This means: -- Multiple extenddb instances can share a PostgreSQL catalog (with caveats — see Multi-Instance below) -- Standard PostgreSQL HA, backup, and replication tools provide durability -- extenddb can run anywhere PostgreSQL is reachable +- Multiple extenddb instances can share one backend catalog (with caveats — see Multi-Instance below) +- The backend's own HA, backup, and replication tools provide durability +- extenddb can run anywhere the storage backend is reachable ## Deployment Models ### Single-Node -extenddb and PostgreSQL on the same host. Simplest setup, suitable for development, CI, and small workloads. +ExtendDB and its storage backend on the same host. Simplest setup, suitable for development, CI, and small workloads. ``` -┌─────────────────────────┐ -│ Host │ -│ ┌─────┐ ┌──────────┐ │ -│ │ extenddb│──│PostgreSQL │ │ -│ └─────┘ └──────────┘ │ -└─────────────────────────┘ +┌──────────────────────────────┐ +│ Host │ +│ ┌──────────┐ ┌───────────┐ │ +│ │ ExtendDB │──│ Storage │ │ +│ │ │ │ backend │ │ +│ └──────────┘ └───────────┘ │ +└──────────────────────────────┘ ``` ```bash @@ -34,51 +43,32 @@ extenddb serve --config extenddb.toml ### Separated Database -extenddb on an application server, PostgreSQL on a dedicated database server or managed service. +ExtendDB on an application server, the storage backend on a dedicated database +server or managed service. ``` ┌──────────┐ ┌──────────────┐ -│ App Host│──────▶│ DB Host │ -│ extenddb │ │ PostgreSQL │ +│ App Host │──────▶│ DB Host │ +│ ExtendDB │ │ Backend │ └──────────┘ └──────────────┘ ``` -```bash -extenddb init \ - --pg-host db.example.com \ - --pg-pass -``` - -Configure the connection string in `extenddb.toml`: - -```toml -[storage.postgres] -connection_string = "postgresql://extenddb:@db.example.com:5432/extenddb_catalog?sslmode=require" -pool_size = 20 -``` - -Use `sslmode=require` (or `verify-full` with a CA certificate) for encrypted database connections. - -### Managed PostgreSQL +Point ExtendDB at the remote backend at init time, and use an encrypted +connection. The flags and connection-string format are backend-specific: for +PostgreSQL see [Configuring PostgreSQL](#configuring-postgresql) below; for +Cassandra see [Cassandra setup](../cassandra-setup.md). -extenddb works with any PostgreSQL 14+ service: +### Managed Database -- **Amazon RDS for PostgreSQL** / **Amazon Aurora PostgreSQL** -- **Google Cloud SQL for PostgreSQL** -- **Azure Database for PostgreSQL** -- **Self-managed PostgreSQL** with streaming replication - -```bash -# Amazon RDS / Aurora example -extenddb init \ - --pg-host mydb.cluster-abc123.us-east-1.rds.amazonaws.com \ - --pg-user extenddb_admin \ - --pg-pass -``` +ExtendDB works with managed database services as well as self-managed servers. +The managed-service topology is the same as Separated Database: ExtendDB on an +application host, the backend provided as a service. For the PostgreSQL managed +services ExtendDB supports (Amazon RDS and Aurora, Google Cloud SQL, Azure +Database), see [Configuring PostgreSQL](#configuring-postgresql). ### Containerized -extenddb runs in Docker or Kubernetes. The binary has no runtime dependencies beyond libc and network access to PostgreSQL. +ExtendDB runs in Docker or Kubernetes. The binary has no runtime dependencies beyond libc and network access to the storage backend. Example Dockerfile: @@ -124,10 +114,10 @@ graceful shutdown. Outside a container, where you may want `stop` to work, add `--write-pid-file` — it writes the PID file to the usual `run_dir` path, which then has to be writable. -What the health check covers: `extenddb healthcheck` exits 0 when the server is +`extenddb healthcheck` exits 0 when the server is listening, completing TLS, and serving HTTP. Use it as a liveness probe, which is what a Docker `HEALTHCHECK` and a Kubernetes `livenessProbe` are for. `/health` -does not query PostgreSQL, and for a liveness probe that is the behaviour you +does not query the storage backend, and for a liveness probe that is the behaviour you want: one that failed on a database outage would restart every replica at once. A backend that is unreachable at startup does stop the server from listening, so container start-up failures are still caught. @@ -141,12 +131,56 @@ For Kubernetes, run `extenddb init` as an init container or a one-time Job, then ### Air-Gapped -extenddb requires no internet connectivity. All functionality is self-contained in the binary. Build on a connected host, transfer the binary and `extenddb.toml` to the air-gapped environment, and run. +ExtendDB requires no internet connectivity. All functionality is self-contained in the binary. Build on a connected host, transfer the binary and `extenddb.toml` to the air-gapped environment, and run. Requirements in the air-gapped environment: -- PostgreSQL 14+ (reachable from the extenddb host) +- A supported storage backend, reachable from the ExtendDB host - The `extenddb` binary (statically linked or with matching libc) +## Configuring PostgreSQL + +This section applies to the PostgreSQL backend. For Cassandra, see +[Cassandra setup](../cassandra-setup.md); for MongoDB, see +[MongoDB setup](../local-mongodb-setup.md). + +### Remote and separated databases + +Point ExtendDB at a remote PostgreSQL server at init time: + +```bash +extenddb init \ + --pg-host db.example.com \ + --pg-pass +``` + +Configure the connection string in `extenddb.toml`: + +```toml +[storage.postgres] +connection_string = "postgresql://extenddb:@db.example.com:5432/extenddb_catalog?sslmode=require" +pool_size = 20 +``` + +Use `sslmode=require` (or `verify-full` with a CA certificate) for encrypted +database connections. + +### Managed PostgreSQL services + +ExtendDB works with any PostgreSQL 14+ service: + +- **Amazon RDS for PostgreSQL** / **Amazon Aurora PostgreSQL** +- **Google Cloud SQL for PostgreSQL** +- **Azure Database for PostgreSQL** +- **Self-managed PostgreSQL** with streaming replication + +```bash +# Amazon RDS / Aurora example +extenddb init \ + --pg-host mydb.cluster-abc123.us-east-1.rds.amazonaws.com \ + --pg-user extenddb_admin \ + --pg-pass +``` + ## Production Checklist ### Authentication @@ -160,15 +194,15 @@ Requirements in the air-gapped environment: - [ ] TLS is enabled by default. Verify it is not disabled in `extenddb.toml` - [ ] Replace the self-signed certificate with a CA-signed certificate -- [ ] Use `sslmode=require` or `sslmode=verify-full` in the PostgreSQL connection string +- [ ] Encrypt the connection to the storage backend. For PostgreSQL, use `sslmode=require` or `sslmode=verify-full` in the connection string ### Network - [ ] Set `bind_addr` to the appropriate interface (not `0.0.0.0` unless behind a load balancer) -- [ ] Firewall: allow only necessary ports (extenddb port, PostgreSQL port) +- [ ] Firewall: allow only necessary ports (extenddb port, storage backend port) - [ ] Consider a reverse proxy (nginx, HAProxy) for TLS termination, rate limiting, and access logging -### Database +### Database (PostgreSQL) - [ ] Use a dedicated PostgreSQL user for extenddb with minimal privileges - [ ] Configure PostgreSQL `max_connections` ≥ extenddb `pool_size` + 3 @@ -229,9 +263,12 @@ Run one extenddb instance per catalog. Multiple instances behind a load balancer - Every instance runs every background worker (TTL expiry, GSI backfill, stream retention, control-plane transitions). The workers are safe to run twice, but the work is duplicated. - `/health` is a liveness check that does not query the database, so a load balancer cannot use it to take an instance with a broken database connection out of rotation. -On SQLite this limit is enforced: `extenddb serve` holds an exclusive lock on `.lock` and a second server on the same file refuses to start. On PostgreSQL and MongoDB it is not enforced; if you start a second instance anyway, size `pool_size × instance_count + 3 × instance_count ≤ PostgreSQL max_connections`. +On SQLite this limit is enforced: `extenddb serve` holds an exclusive lock on `.lock` and a second server on the same file refuses to start. On PostgreSQL, MongoDB, and Cassandra it is not enforced. On PostgreSQL, if you start a second instance anyway, size `pool_size × instance_count + 3 × instance_count ≤ PostgreSQL max_connections`. + +## Performance Tuning (PostgreSQL) -## Performance Tuning +This section covers the PostgreSQL backend. For Cassandra tuning (consistency, +token-aware routing, LWT contention), see [Cassandra setup](../cassandra-setup.md). ### Connection Pool From 776e3fbece495704938b68ac265faae45e9a5fa9 Mon Sep 17 00:00:00 2001 From: Joel Shepherd Date: Fri, 23 Oct 2026 20:09:51 +0000 Subject: [PATCH 2/2] docs: clarify Cassandra auth requirements and extenddb init command invocation --- docs/cassandra-setup.md | 108 +++++++++++++++++++------- docs/getting-started.md | 49 +++++++----- docs/manuals/01-architecture-guide.md | 2 +- docs/manuals/11-deployment-guide.md | 8 +- 4 files changed, 117 insertions(+), 50 deletions(-) diff --git a/docs/cassandra-setup.md b/docs/cassandra-setup.md index aba504847..3e1db8461 100644 --- a/docs/cassandra-setup.md +++ b/docs/cassandra-setup.md @@ -60,12 +60,11 @@ They mirror `devtools/cassandra-node`, which the project uses for its own tests. The node now listens on `127.0.0.1:9042` with password authentication enabled. -> **Note:** Password authentication is optional for a purely local node. If you -> skip step 3, the node runs with the default `AllowAllAuthenticator` (no -> authentication), and you can run `extenddb init --backend cassandra` with no -> credential flags. The steps above enable authentication because production -> clusters require it, and testing against an authenticated node locally avoids -> surprises later. +> **Password authentication is required.** `extenddb init` always creates its +> application role with `CREATE ROLE ... WITH PASSWORD`, which Cassandra accepts +> only when the node runs `PasswordAuthenticator`. Against a stock node using the +> default `AllowAllAuthenticator`, `init` fails at the role step with +> `CassandraRoleManager doesn't support PASSWORD`. Do not skip step 3. ## Connection Details @@ -108,16 +107,20 @@ application role: --config extenddb.toml ``` -If your node runs without authentication (you skipped step 3 above), omit the -credential flags: +The node must run `PasswordAuthenticator` (step 3). `init` always creates its +application role with a password, so it fails against an unauthenticated +(`AllowAllAuthenticator`) node — there is no credential-free path. -```bash -./target/release/extenddb init --backend cassandra --config extenddb.toml -``` +> **Authorizer.** Run the node with `AllowAllAuthorizer` (the default). `init` +> creates the application role but grants it no permissions, so under +> `CassandraAuthorizer` the server later fails to start with +> `User extenddb has no SELECT permission ...`. Until `init` issues the grants, +> only `AllowAllAuthorizer` works. init creates: - The catalog keyspace (`extenddb_catalog` by default) -- The application role ExtendDB uses at serve time +- The application role ExtendDB uses at serve time (named `extenddb` by + default; override with `--extenddb-user` / `--extenddb-pass`) - Admin user credentials (printed to stdout once — save them) - A self-signed TLS certificate at `~/.extenddb/tls/cert.pem` - The config file `extenddb.toml` @@ -129,7 +132,8 @@ Per-account keyspaces are created later, when accounts are added. ## Config Mapping -init writes a `[storage.cassandra]` section to `extenddb.toml`: +init writes a `[storage.cassandra]` section to `extenddb.toml`. It writes the +application credentials **commented out**, with placeholder values: ```toml [storage] @@ -145,23 +149,50 @@ datacenter = "datacenter1" max_connections = 10 ``` +> **You must set `username` and `password` before starting the server.** On an +> authenticated node, `serve` with these two fields commented out fails with +> `Connection timeout after 10s`. Uncomment them and set them to the +> application role `init` created — by default user `extenddb` with password +> `extenddb-local-dev` (or whatever you passed to `--extenddb-user` / +> `--extenddb-pass`), **not** the placeholder values above: + +```toml +[storage.cassandra] +contact_points = ["127.0.0.1:9042"] +username = "extenddb" +password = "extenddb-local-dev" +# ... +``` + Fields: - `contact_points` — one or more `host:port` entries. One reachable node is enough; the driver discovers the rest of the cluster. - `username` / `password` — the application credentials ExtendDB uses at serve - time. Omit both for an unauthenticated node. + time. Required against any authenticated node (which `init` requires anyway). - `keyspace_prefix` — prefix for this deployment's keyspaces (default `extenddb`). -- `replication_factor` — keyspace replication factor. `1` suits a single local - node; use 3 or more in production. -- `datacenter` — the local datacenter name for `NetworkTopologyStrategy` - (default `datacenter1`, which matches a stock single-node cluster). +- `replication_factor` — keyspace replication factor. `init` always writes `1`, + regardless of the cluster. For a multi-node cluster, edit this to 3 or more + **after** `init`; keyspaces created later (per account) use the config value. + Existing keyspaces are not altered retroactively. +- `datacenter` — the local datacenter name for `NetworkTopologyStrategy`. `init` + always writes `datacenter1` and has no flag to change it, and the keyspaces it + creates reference `datacenter1`, so the cluster's local datacenter must be + named `datacenter1`. A node started with a different datacenter name (for + example `CASSANDRA_DC=dc1`) fails keyspace creation with + `Unrecognized strategy option {datacenter1} ... for keyspace extenddb_catalog`. - `max_connections` — maximum connections per host. -Edit these values in `extenddb.toml`. Unlike the PostgreSQL and MongoDB -backends, the Cassandra backend has no single connection-string field; set -`contact_points` and the other fields directly in the config section above. +Edit these values in `extenddb.toml` after `init`. Unlike the PostgreSQL and +MongoDB backends, the Cassandra backend has no single connection-string field; +set `contact_points` and the other fields directly in the config section above. + +> **Connections are not encrypted.** The Cassandra backend connects over plain +> TCP — the driver is built without TLS — so credentials and item data cross the +> network in clear text. Run ExtendDB and Cassandra on a trusted network (for +> example a private subnet or an encrypted overlay); do not send Cassandra +> traffic across an untrusted network. ## Starting the Server @@ -215,9 +246,13 @@ section covers only what is specific to Cassandra. ### Keyspace replication -Set `replication_factor` to 3 or more for a multi-node cluster, and set -`datacenter` to your cluster's local datacenter name. ExtendDB creates its -keyspaces with `NetworkTopologyStrategy` using these values. +Set `replication_factor` to 3 or more for a multi-node cluster. `init` always +writes `1`, so edit the config after `init`; keyspaces created afterward (per +account) use the value in the config. The `datacenter` must be `datacenter1`: +`init` hardcodes that name into the keyspace definitions and provides no flag to +change it, so your cluster's local datacenter has to be named `datacenter1` +until that is configurable. ExtendDB creates its keyspaces with +`NetworkTopologyStrategy` using these values. ```toml [storage.cassandra] @@ -226,17 +261,30 @@ username = "extenddb" password = "" keyspace_prefix = "extenddb" replication_factor = 3 -datacenter = "dc1" +datacenter = "datacenter1" # must be datacenter1 (hardcoded by init) max_connections = 10 ``` ### Authentication -Run the cluster with authentication enabled (`PasswordAuthenticator` or -stronger), and give ExtendDB a dedicated application role rather than the -`cassandra` superuser. Supply its credentials through `username` and `password`. -ExtendDB connects without authentication when both are absent, so confirm they -are set for any cluster that is not fully trusted. +Run the cluster with authentication enabled — `init` requires +`PasswordAuthenticator` or stronger, since it creates its application role with +a password. Give ExtendDB a dedicated application role rather than the +`cassandra` superuser, and supply its credentials through `username` and +`password`; both are required. + +Use `AllowAllAuthorizer` for now. `init` creates the application role but grants +it no permissions, so `CassandraAuthorizer` makes `serve` fail with +`User has no SELECT permission ...`. Until `init` issues the grants a +role needs, authorization must be left open at the Cassandra layer. + +### Connection encryption + +The Cassandra backend connects over plain TCP; the driver is compiled without +TLS, so credentials and item data travel in clear text. There is no config +option to enable encryption today. Deploy ExtendDB and the Cassandra cluster on +a trusted, isolated network, and do not route Cassandra traffic over an +untrusted network. ### Known Limitations diff --git a/docs/getting-started.md b/docs/getting-started.md index ef626f24c..d1a2cbccb 100755 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -169,44 +169,57 @@ Use `:memory:` (in the config, the environment variable, or `init ExtendDB can run against an Apache Cassandra cluster. The Cassandra backend is experimental; see [Cassandra setup](cassandra-setup.md) for its current -limitations. Select it at init time with `--backend cassandra`: +limitations and the full walkthrough. Select it at init time with +`--backend cassandra`: ```bash -./target/release/extenddb init --backend cassandra +./target/release/extenddb init \ + --backend cassandra \ + --cassandra-contact-points 127.0.0.1:9042 \ + --cassandra-user cassandra \ + --cassandra-pass cassandra ``` -For a local single-node cluster with authentication disabled, this needs no -other flags. init uses development defaults: it contacts `localhost:9042`, -creates keyspaces under the prefix `extenddb`, and sets a replication factor of -1. It connects without authentication, which matches a stock Cassandra node -(the default `AllowAllAuthenticator`). If the node has authentication enabled -(for example `PasswordAuthenticator`), pass `--cassandra-user` and -`--cassandra-pass`. init writes a `[storage.cassandra]` section to the config: +The node must run `PasswordAuthenticator`. `init` always creates its +application role with a password, so it fails against a stock node using the +default `AllowAllAuthenticator` — there is no credential-free path. Pass the +Cassandra admin credentials with `--cassandra-user` / `--cassandra-pass`; `init` +uses them to create the keyspaces and the application role. `init` writes a +`[storage.cassandra]` section to the config, with the application credentials +commented out: ```toml [storage.cassandra] -contact_points = ["localhost:9042"] +contact_points = ["127.0.0.1:9042"] # username = "cassandra" # Application user # password = "cassandra-password" # Application password keyspace_prefix = "extenddb" -replication_factor = 1 # Single node (use 3+ for production) -datacenter = "datacenter1" +replication_factor = 1 # init always writes 1; edit for production +datacenter = "datacenter1" # must be datacenter1 (hardcoded by init) max_connections = 10 ``` +Before starting the server, uncomment `username` and `password` and set them to +the application role `init` created — by default `extenddb` / +`extenddb-local-dev`. With them commented out, `serve` cannot authenticate and +times out. + To override a default, pass the matching flag at init time: - `--cassandra-contact-points host1:9042,host2:9042` — one or more contact points for a remote or multi-node cluster. -- `--cassandra-user` and `--cassandra-pass` — admin credentials, if the cluster - requires authentication to create the keyspaces and the application role. +- `--cassandra-user` and `--cassandra-pass` — the admin credentials `init` uses + to create the keyspaces and the application role (required). +- `--extenddb-user` and `--extenddb-pass` — override the application role's name + and password (default `extenddb` / `extenddb-local-dev`). - `--keyspace-prefix myapp` — a different keyspace prefix, so one cluster can host more than one deployment. -- `--replication-factor 3` — a higher replication factor for a multi-node - cluster. -For replication strategy, datacenter topology, authentication, and production -tuning, see `docs/cassandra-setup.md`. +For a higher replication factor, edit `replication_factor` in the config after +`init` (there is no init flag for it today). The datacenter must be +`datacenter1`. For replication strategy, datacenter topology, authentication, +authorization, and the fact that connections are unencrypted, see +`docs/cassandra-setup.md`. ### Generating a self-signed certificate manually diff --git a/docs/manuals/01-architecture-guide.md b/docs/manuals/01-architecture-guide.md index 64b08d672..678536d53 100755 --- a/docs/manuals/01-architecture-guide.md +++ b/docs/manuals/01-architecture-guide.md @@ -299,7 +299,7 @@ Configuration precedence: CLI flags > environment variables > config file > defa ## DynamoDB Streams -extenddb implements DynamoDB Streams for change data capture. Stream records are captured atomically with the data writes that produce them. Both the DynamoDB API and Streams API are served on the same port. +extenddb implements DynamoDB Streams for change data capture. On the PostgreSQL and SQLite backends, stream records are captured atomically with the data writes that produce them, inside the same transaction. On the Cassandra backend this holds for ordinary writes but not for conditional writes: a conditional put applies the item with a lightweight transaction and then writes the stream record in a separate logged batch, so a failure between the two can leave the item written with no stream record. Both the DynamoDB API and Streams API are served on the same port. Supported operations: `ListStreams`, `DescribeStream`, `GetShardIterator`, `GetRecords`. diff --git a/docs/manuals/11-deployment-guide.md b/docs/manuals/11-deployment-guide.md index 59b120c13..4524876a0 100755 --- a/docs/manuals/11-deployment-guide.md +++ b/docs/manuals/11-deployment-guide.md @@ -164,6 +164,12 @@ pool_size = 20 Use `sslmode=require` (or `verify-full` with a CA certificate) for encrypted database connections. +The Cassandra backend cannot encrypt its connection today: the driver is +compiled without TLS, so credentials and item data cross the network in clear +text and there is no config option to turn encryption on. Run ExtendDB and the +Cassandra cluster on a trusted, isolated network; do not route Cassandra traffic +over an untrusted network. + ### Managed PostgreSQL services ExtendDB works with any PostgreSQL 14+ service: @@ -194,7 +200,7 @@ extenddb init \ - [ ] TLS is enabled by default. Verify it is not disabled in `extenddb.toml` - [ ] Replace the self-signed certificate with a CA-signed certificate -- [ ] Encrypt the connection to the storage backend. For PostgreSQL, use `sslmode=require` or `sslmode=verify-full` in the connection string +- [ ] Encrypt the connection to the storage backend. For PostgreSQL, use `sslmode=require` or `sslmode=verify-full` in the connection string. The Cassandra backend cannot encrypt its connection — place ExtendDB and the Cassandra cluster on a trusted, isolated network instead ### Network