diff --git a/AGENTS.md b/AGENTS.md index 56ec6117..9b0d32e5 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -4,13 +4,13 @@ This document provides AI agents with essential context for working on the Exten ## Project Overview -**ExtendDB** is an open source DynamoDB-compatible runtime backed by PostgreSQL storage, released under the Apache 2.0 +**ExtendDB** is an open source DynamoDB-compatible runtime with pluggable storage, released under the Apache 2.0 License. ExtendDB is a clean-room implementation of the DynamoDB wire protocol, written in Rust and initially developed by AWS engineers. It is not a fork of DynamoDB and contains no DynamoDB source code. ExtendDB speaks the DynamoDB wire protocol: any AWS SDK, CLI, or tool that works with DynamoDB works with ExtendDB, unchanged. - **Language:** Rust (edition 2024, MSRV 1.88+) -- **Storage backends:** PostgreSQL 14+ (default), SQLite (bundled, no server or client install required; feature flag `sqlite`, also `sqlite-memory` for dev and CI), MongoDB 7.0+ (feature flag `mongodb`). One backend per binary, selected by mutually exclusive Cargo features. +- **Storage backends:** PostgreSQL 14+ (default), SQLite (bundled, no server or client install required; feature flag `sqlite`, also `sqlite-memory` for dev and CI), MongoDB 7.0+ (feature flag `mongodb`), Cassandra 4.1+ (experimental; feature flag `cassandra`). One backend per binary, selected by mutually exclusive Cargo features. - **Architecture:** Async (tokio), trait-based storage abstraction - **Authentication:** Mandatory SigV4 with built-in IAM (users, groups, roles, policies) - **TLS:** Mandatory (self-signed cert generated by default) @@ -28,18 +28,19 @@ protocol: any AWS SDK, CLI, or tool that works with DynamoDB works with ExtendDB ``` extenddb/ ├── crates/ +│ ├── app/ # Shared CLI surface the thin bin dispatches to +│ ├── auth/ # SigV4 verification, IAM policy engine +│ ├── bin/ # CLI, config, daemon lifecycle (extenddb binary) +│ ├── cache/ # Auth and authz caches +│ ├── config/ # Config parsing shared by the CLI and the server │ ├── core/ # Pure sync types, validation, expression parsing (no async) │ ├── engine/ # DynamoDB operation handlers (PutItem, Query, etc.) +│ ├── server/ # HTTP server, management API, web console │ ├── storage/ # Storage trait definitions (TableEngine trait) -│ ├── storage-postgres/ # PostgreSQL implementation of TableEngine -│ ├── storage-sqlite/ # SQLite implementation (dev, CI, and embedded use) +│ ├── storage-cassandra/ # Cassandra implementation (experimental; feature flag) │ ├── storage-mongodb/ # MongoDB implementation (feature flag) -│ ├── cache/ # Auth and authz caches -│ ├── config/ # Config parsing shared by the CLI and the server -│ ├── app/ # Shared CLI surface the thin bin dispatches to -│ ├── auth/ # SigV4 verification, IAM policy engine -│ ├── server/ # HTTP server, management API, web console -│ └── bin/ # CLI, config, daemon lifecycle (extenddb binary) +│ ├── storage-postgres/ # PostgreSQL implementation (default/reference) +│ └── storage-sqlite/ # SQLite implementation (dev, CI, and embedded use) ├── tests/ # Python integration tests (pytest) ├── docs/ │ ├── design/ # Architecture and component design docs @@ -59,6 +60,7 @@ The `TableEngine` trait in `crates/storage/src/lib.rs` defines the storage inter - `storage-postgres` (PostgreSQL) — default backend - `storage-sqlite` (SQLite) — feature flag `sqlite`, and `sqlite-memory` for dev and CI - `storage-mongodb` (MongoDB) — feature flag `mongodb` +- `storage-cassandra` (Cassandra) — experimental; feature flag `cassandra` The trait uses RPITIT (return-position impl Trait in traits) for async methods — no `#[async_trait]` macro. @@ -73,9 +75,10 @@ extenddb (bin) │ ├─> extenddb-core (pure sync, no async) │ └─> extenddb-storage (trait definitions) ├─> extenddb-auth - ├─> extenddb-storage-postgres (feature: postgres) - ├─> extenddb-storage-sqlite (feature: sqlite / sqlite-memory) - └─> extenddb-storage-mongodb (feature: mongodb) + ├─> extenddb-storage-postgres (feature: postgres) + ├─> extenddb-storage-sqlite (feature: sqlite / sqlite-memory) + ├─> extenddb-storage-mongodb (feature: mongodb) + └─> extenddb-storage-cassandra (feature: cassandra) ``` - **extenddb-core:** Pure synchronous Rust. No async, no I/O. Types, validation, expression parsing. @@ -84,6 +87,7 @@ extenddb (bin) - **extenddb-storage-sqlite:** Concrete SQLite implementation, the dev and CI backend. It serves the same wire surface, vector search included, and needs no server. - **extenddb-storage-mongodb:** Concrete MongoDB implementation. +- **extenddb-storage-cassandra:** Concrete Cassandra implementation. Experimental. - **extenddb-engine:** Operation handlers that call storage traits. - **extenddb-server:** HTTP server, management API, web console. - **extenddb-auth:** SigV4 signature verification, IAM policy evaluation. @@ -96,7 +100,9 @@ extenddb (bin) - Rust 1.88+ (`rustup update`) - Storage backend (one of): - PostgreSQL 14+ running locally (see `docs/local-postgres-setup.md`) + - SQLite (embedded; nothing to install) - MongoDB 7.0+ with replica set (see `docs/local-mongodb-setup.md`) + - Cassandra 4.1+ (experimental; see `docs/cassandra-setup.md`) - Python 3.10+ for tests (`python3 -m venv ~/venvs/extenddb-venv && source ~/venvs/extenddb-venv/bin/activate && pip install -r requirements.txt`) ### Build @@ -107,11 +113,14 @@ cargo build --release # MongoDB backend (backends are mutually exclusive — disable default postgres) cargo build --release --no-default-features --features mongodb + +# Cassandra backend (experimental) +cargo build --release --no-default-features --features cassandra ``` Backends are selected by mutually exclusive Cargo features (`postgres`, -`mongodb`, `sqlite`); a build enabling more than one is rejected at compile -time. Each binary serves exactly one backend. +`mongodb`, `sqlite`, `cassandra`); a build enabling more than one is rejected at +compile time. Each binary serves exactly one backend. Binary: `target/release/extenddb` @@ -136,6 +145,9 @@ cargo clippy --all-targets -- -D warnings # MongoDB ./target/release/extenddb init --backend mongodb --config extenddb.toml + +# Cassandra (experimental) +./target/release/extenddb init --backend cassandra --config extenddb.toml ``` This creates: @@ -339,7 +351,7 @@ Query, Scan (key conditions, filters, projections, pagination, index selection) BatchGetItem (100 keys), BatchWriteItem (25 ops), TransactGetItems (100 items), TransactWriteItems (100 ops) ### Vector Search -SearchVectors, and vector indexes on CreateTable and UpdateTable (COSINE, EUCLIDEAN, DOT_PRODUCT; on-demand tables only; five per table). Served by the PostgreSQL and SQLite backends; MongoDB provides no implementation and refuses every vector operation. PostgreSQL additionally requires the pgvector extension on the data database, probed once at startup, and refuses without it. The refusal is two strings, which matters when grepping logs: `Vector indexes are not supported by this storage backend` for CreateTable and UpdateTable, and `SearchVectors is not supported by this storage backend` for a search. +SearchVectors, and vector indexes on CreateTable and UpdateTable (COSINE, EUCLIDEAN, DOT_PRODUCT; on-demand tables only; five per table). Served by the PostgreSQL and SQLite backends; MongoDB and Cassandra provide no implementation and refuse every vector operation. PostgreSQL additionally requires the pgvector extension on the data database, probed once at startup, and refuses without it. The refusal is two strings, which matters when grepping logs: `Vector indexes are not supported by this storage backend` for CreateTable and UpdateTable, and `SearchVectors is not supported by this storage backend` for a search. ### Streams ListStreams, DescribeStream, GetShardIterator, GetRecords @@ -387,8 +399,10 @@ Expression parsing lives in `crates/core/src/expression/`. This is pure sync Rus | Troubleshooting | `docs/troubleshooting.md` | Common errors and solutions | | Storage Component Design | `docs/design/04-component-storage.md` | Storage trait design | | MongoDB Design | `docs/design/13-storage-mongodb.md` | MongoDB backend architecture | +| Cassandra Design (experimental) | `docs/adr/0010`–`0021` | Cassandra backend design decisions (ADRs) | | Local PostgreSQL Setup | `docs/local-postgres-setup.md` | PostgreSQL installation and config | | Local MongoDB Setup | `docs/local-mongodb-setup.md` | MongoDB installation and config | +| Cassandra Setup | `docs/cassandra-setup.md` | Cassandra install, config, and deployment (experimental) | | Testing Design | `docs/design/09-testing.md` | Test strategy and infrastructure | | Extending Storage | `12-backend-plugin-architecture.md` | Guide to implementing storage backends | @@ -480,7 +494,7 @@ Activate when the user asks about installing, configuring, running, or debugging 1. **TLS is mandatory** — server refuses to start without it. Use `AWS_CA_BUNDLE` for self-signed certs. 2. **Auth is mandatory** — all requests must be SigV4-signed. Use `extenddb manage` or web console to create credentials. 3. **Account isolation** — all operations are scoped to `account_id`. Different accounts can have tables with the same name. -4. **Storage backend must be running** — `extenddb init` and `extenddb serve` require a running PostgreSQL or MongoDB instance. +4. **Storage backend must be running** — `extenddb init` and `extenddb serve` require a running external backend (PostgreSQL, MongoDB, or Cassandra). SQLite is embedded and needs no server. 5. **MongoDB requires a replica set** — even single-node MongoDB must be configured with `--replSet` for transactions and streams. 6. **Python venv** — activate the venv before running tests: `source ~/venvs/extenddb-venv/bin/activate` 7. **Test credentials** — run `devtools/provision-test-credentials` before pytest to create test users and keys. diff --git a/README.md b/README.md index 518fdad1..f503f1ef 100755 --- a/README.md +++ b/README.md @@ -1,27 +1,31 @@ # ExtendDB -> **ExtendDB is an independent open source project managed by Amazon Web Services. It is not Amazon DynamoDB and does not contain any DynamoDB source code.** "DynamoDB" is a trademark of Amazon.com, Inc. ExtendDB is a clean-room implementation that speaks the DynamoDB wire protocol. Behavioral differences from the real service are documented in [Differences from DynamoDB](docs/differences-from-dynamodb.md). +> **ExtendDB is an independent open source project managed by Amazon Web Services. It is not Amazon DynamoDB and does +> not contain any DynamoDB source code.** "DynamoDB" is a trademark of Amazon.com, Inc. ExtendDB is a clean-room +> implementation that speaks the DynamoDB wire protocol. Behavioral differences from the real service are documented +> in [Differences from DynamoDB](docs/differences-from-dynamodb.md). -A DynamoDB-compatible API adapter, ExtendDB speaks the DynamoDB wire protocol — any AWS SDK, CLI, or tool that works with DynamoDB works with ExtendDB, unchanged. +A DynamoDB-compatible API adapter, ExtendDB speaks the DynamoDB wire protocol — any AWS SDK, CLI, or tool that works +with DynamoDB works with ExtendDB, unchanged. ## Use Cases - **Local development** — run DynamoDB workloads on your laptop with zero cloud dependency - **CI/CD pipelines** — deterministic integration tests against a DynamoDB-compatible backend - **Self-hosted deployments** — run DynamoDB workloads on your own infrastructure (on-premises, private cloud, edge) -- **Multi-cloud** — use DynamoDB semantics on any cloud that runs PostgreSQL +- **Multi-cloud** — use DynamoDB semantics on any cloud that runs an ExtendDB-supported database backend - **Air-gapped environments** — DynamoDB functionality with no internet connectivity ## Features - Full DynamoDB wire protocol: CRUD, Query, Scan, Batch, Transactions, Streams, TTL, Import/Export +- Pluggable storage: PostgreSQL (default), SQLite, MongoDB, and Cassandra - SigV4 authentication with IAM compatibility: users, groups, roles, policies, permissions boundaries - Web management console for account and credential administration - TLS with automatic self-signed certificate generation (replaceable with CA-signed certs) - CSRF protection, security headers, session management - JSON metrics endpoint with DynamoDB CloudWatch-style metric names and dimensions - Daemon mode with syslog logging, plus `--foreground` for container and supervisor environments -- PostgreSQL storage — use standard backup, replication, and HA tools ## Quick Start @@ -50,8 +54,12 @@ scripts/install-macos.sh # macOS ## Prerequisites - Rust 1.88+ (`rustup update`) -- PostgreSQL 14+ (see `docs/local-postgres-setup.md`) - Python 3.10+ (for test suites and documentation) +- A storage backend: + - PostgreSQL 14+ (see `docs/local-postgres-setup.md`) + - SQLite (embedded) + - MongoDB 7+ (see `docs/local-mongodb-setup.md`) + - Cassandra 4.1+ (see `docs/cassandra-setup.md`) ### Python Environment @@ -106,11 +114,10 @@ key_path = "/etc/extenddb/tls/key.pem" ## Running in Containers -The production images contain ExtendDB only; the database (PostgreSQL 14+ and MongoDB 7+) -runs as a separate service. See [`docker/README.md`](docker/README.md) for the PostgreSQL -image and [`docker/README-mongodb.md`](docker/README-mongodb.md) for the MongoDB -image, external database requirements, hardened local Compose stacks, smoke tests, and -explicit init/migrate/serve production lifecycle. +The production images contain ExtendDB only. The storage backend runs as a separate service. See +[`docker/README.md`](docker/README.md) for the PostgreSQL image and [`docker/README-mongodb.md`](docker/README-mongodb.md) for the MongoDB image, +external database requirements, hardened local Compose stacks, smoke tests, and explicit init/migrate/serve +production lifecycle. The SQLite and Cassandra backends have no dedicated container guide yet. By default `extenddb serve` daemonizes itself, which doesn't play well with container runtimes and process supervisors that expect PID 1 to stay attached. Pass `--foreground` (alias `--no-daemon`) to keep the process in the foreground and stream logs to stderr instead of syslog: @@ -134,11 +141,20 @@ extenddb healthcheck --config extenddb.toml # exit 0 healthy, 1 not extenddb healthcheck --endpoint https://127.0.0.1:18443 # explicit target ``` -This is a liveness check, which is what a container `HEALTHCHECK` wants: `/health` does not query the storage backend, so it reports healthy even if PostgreSQL becomes unreachable after startup. That is deliberate, since a liveness probe that failed on a database outage would restart every replica at once. A backend that is unreachable at startup does stop the server from listening, so that case is caught. There is no separate readiness endpoint yet. +This is a liveness check, which is what a container `HEALTHCHECK` wants. The `/health` path does not query the storage +backend, so it will report healthy even if the storage backend becomes unreachable after startup. That is deliberate, +since a liveness probe that failed on a database outage would restart every replica at once. A backend that is +unreachable at startup does stop the server from listening. There is no separate readiness endpoint yet. ### One instance per database -Run one `extenddb serve` per catalog. On SQLite this is enforced: the server takes an exclusive lock on `.lock` next to the database file and a second server on the same file refuses to start (`init`, `migrate`, and `destroy` take the same lock while they run). On PostgreSQL and MongoDB nothing prevents a second instance, but running more than one against the same catalog is not supported in this release: every instance runs every background worker, credential and policy caches are per instance with no cross-instance invalidation (a revoked key stays valid on the other instance for up to the cache TTL), and `/health` does not check the database. See the [deployment guide](docs/manuals/11-deployment-guide.md#multi-instance-considerations). +Run one `extenddb serve` per catalog. On SQLite this is enforced: the server takes an exclusive lock on `.lock` +next to the database file and a second server on the same file will refuse to start (`init`, `migrate`, and `destroy` take +the same lock while they run). On PostgreSQL, MongoDB and Cassandra nothing prevents a second ExtendDB instance from +running, but running more than one against the same catalog is not supported in this release: every instance runs every +background worker, credential and policy caches are per instance with no cross-instance invalidation (a revoked key +stays valid on the other instance for up to the cache TTL), and `/health` does not check the database. +See the [deployment guide](docs/manuals/11-deployment-guide.md#multi-instance-considerations). To make the generated self-signed certificate valid for the name clients use — an in-cluster service DNS name, for example — pass `--tls-san` to `init` (repeatable): @@ -174,7 +190,7 @@ Web-based administration at `https://127.0.0.1:18443/console/`. Manage accounts, ``` extenddb serve --config extenddb.toml # Start server (daemon) extenddb serve --config extenddb.toml --foreground # Start in foreground -extenddb init --catalog-db NAME # Initialize deployment +extenddb init --catalog-db NAME # Initialize deployment extenddb stop --config extenddb.toml # Graceful shutdown extenddb status --config extenddb.toml # Check if running extenddb healthcheck --config extenddb.toml # Probe /health (exit 0 healthy, 1 not) @@ -214,17 +230,22 @@ UpdateTimeToLive, DescribeTimeToLive, TagResource, UntagResource, ListTagsOfReso ``` crates/ - core/ — types, validation, expressions (pure sync Rust, no async) - engine/ — operation handlers - storage/ — storage trait definitions - storage-postgres/ — PostgreSQL backend - auth/ — SigV4 verification, IAM policy engine - server/ — HTTP server, management API, web console - bin/ — CLI, config, daemon lifecycle + auth/ — SigV4 verification, IAM policy engine + bin/ — CLI, config, daemon lifecycle + core/ — types, validation, expressions (pure sync Rust, no async) + engine/ — operation handlers + server/ — HTTP server, management API, web console + storage/ — storage trait definitions + storage-cassandra/ — Cassandra backend (experimental) + storage-mongodb/ — MongoDB backend + storage-postgres/ — PostgreSQL backend (default/reference) + storage-sqlite/ — SQLite backend (embedded) + + docs/ + adr/ — architecture decision records design/ — architecture and design documents manuals/ — user-facing guides (PDF pipeline) - adr/ — architecture decision records ``` ## Documentation diff --git a/docs/adr/0011-cassandra-foreign-key-emulation.md b/docs/adr/0011-cassandra-foreign-key-emulation.md index 4f076f5e..e721b3f2 100644 --- a/docs/adr/0011-cassandra-foreign-key-emulation.md +++ b/docs/adr/0011-cassandra-foreign-key-emulation.md @@ -1,4 +1,4 @@ -# ADR-0001: Foreign Key Constraint Emulation +# ADR-0011: Foreign Key Constraint Emulation - Status: Accepted - Date: 2026-06-04 diff --git a/docs/adr/0012-cassandra-transaction-atomicity-patterns.md b/docs/adr/0012-cassandra-transaction-atomicity-patterns.md index 9a06f084..96c2fd51 100644 --- a/docs/adr/0012-cassandra-transaction-atomicity-patterns.md +++ b/docs/adr/0012-cassandra-transaction-atomicity-patterns.md @@ -1,4 +1,4 @@ -# ADR-0002: Transaction and Atomicity Patterns +# ADR-0012: Transaction and Atomicity Patterns - Status: Accepted - Date: 2026-06-04 @@ -36,7 +36,7 @@ Check-then-batch pattern for operations requiring both duplicate detection and a ## Consequences - All multi-entity IAM operations follow this pattern: - 1. Pre-check foreign keys (via ADR-0001) + 1. Pre-check foreign keys (via ADR-0011) 2. SELECT to check for duplicate primary keys 3. Logged BATCH to perform atomic writes - Small race window: Another concurrent operation could insert the same entity between steps 2 and 3. Mitigation: IAM operations are admin-driven and infrequent. Application-layer retry with exponential backoff handles the rare collision. diff --git a/docs/adr/0013-cassandra-keyspace-awareness.md b/docs/adr/0013-cassandra-keyspace-awareness.md index 10163654..9381c75d 100644 --- a/docs/adr/0013-cassandra-keyspace-awareness.md +++ b/docs/adr/0013-cassandra-keyspace-awareness.md @@ -1,4 +1,4 @@ -# ADR-0003: Dynamic Keyspace Construction +# ADR-0013: Dynamic Keyspace Construction - Status: Accepted - Date: 2026-06-04 diff --git a/docs/adr/0014-cassandra-upsert-semantics.md b/docs/adr/0014-cassandra-upsert-semantics.md index 60236dba..c7e55c9a 100644 --- a/docs/adr/0014-cassandra-upsert-semantics.md +++ b/docs/adr/0014-cassandra-upsert-semantics.md @@ -1,4 +1,4 @@ -# ADR-0004: Natural UPSERT Semantics +# ADR-0014: Natural UPSERT Semantics - Status: Accepted - Date: 2026-06-04 diff --git a/docs/adr/0015-cassandra-account-keyspace-provisioning.md b/docs/adr/0015-cassandra-account-keyspace-provisioning.md index 703434e0..541a46df 100644 --- a/docs/adr/0015-cassandra-account-keyspace-provisioning.md +++ b/docs/adr/0015-cassandra-account-keyspace-provisioning.md @@ -1,4 +1,4 @@ -# ADR-0005: Account Keyspace Provisioning Timing +# ADR-0015: Account Keyspace Provisioning Timing - Status: Accepted - Date: 2026-06-05 diff --git a/docs/adr/0016-cassandra-index-table-primary-key-structure.md b/docs/adr/0016-cassandra-index-table-primary-key-structure.md index d0778f69..c946e4fa 100644 --- a/docs/adr/0016-cassandra-index-table-primary-key-structure.md +++ b/docs/adr/0016-cassandra-index-table-primary-key-structure.md @@ -1,4 +1,4 @@ -# ADR-0006: Index Data Table PRIMARY KEY Structure +# ADR-0016: Index Data Table PRIMARY KEY Structure - Status: Accepted - Date: 2026-06-16 diff --git a/docs/adr/0017-cassandra-transaction-implementation.md b/docs/adr/0017-cassandra-transaction-implementation.md index cf9bb378..fc505b91 100644 --- a/docs/adr/0017-cassandra-transaction-implementation.md +++ b/docs/adr/0017-cassandra-transaction-implementation.md @@ -1,4 +1,4 @@ -# ADR-0007: DynamoDB Transaction Implementation on Cassandra +# ADR-0017: DynamoDB Transaction Implementation on Cassandra - Status: Draft - Date: 2026-06-24 diff --git a/docs/adr/0018-cassandra-stream-implementation.md b/docs/adr/0018-cassandra-stream-implementation.md index cd0c4020..1c427e0c 100644 --- a/docs/adr/0018-cassandra-stream-implementation.md +++ b/docs/adr/0018-cassandra-stream-implementation.md @@ -1,4 +1,4 @@ -# ADR-0008: DynamoDB Streams Implementation on Cassandra +# ADR-0018: DynamoDB Streams Implementation on Cassandra - Status: Draft - Date: 2026-06-25 diff --git a/docs/adr/0019-cassandra-logical-backup-restore.md b/docs/adr/0019-cassandra-logical-backup-restore.md index f0fe5ddf..7e247743 100644 --- a/docs/adr/0019-cassandra-logical-backup-restore.md +++ b/docs/adr/0019-cassandra-logical-backup-restore.md @@ -1,4 +1,4 @@ -# ADR-0009: Logical backup and restore for Cassandra +# ADR-0019: Logical backup and restore for Cassandra **Status:** Accepted **Date:** 2026-08-03 diff --git a/docs/cassandra-setup.md b/docs/cassandra-setup.md new file mode 100644 index 00000000..690e873c --- /dev/null +++ b/docs/cassandra-setup.md @@ -0,0 +1,332 @@ +# Cassandra Setup + +> See [NOTICE](NOTICE.md) for important disclaimers. + +> **Experimental.** As of October 2026, the Cassandra backend is experimental. +> Validate it against your workload before you rely on it in production. This +> guide covers both local development and production deployment. + +## Prerequisites + +- Apache Cassandra 4.1 or later +- Docker (for the local single-node setup below), or an existing Cassandra + cluster +- The `cqlsh` client, for the verification steps (bundled in the Cassandra + Docker image) + +## Local Single-Node Setup (Docker) + +These steps start a single-node Cassandra 4.1 container for local development. + +1. Start the container: + + ```bash + docker run -d --name cassandra -p 9042:9042 \ + -e CASSANDRA_CLUSTER_NAME=extenddb-dev \ + cassandra:4.1 + ``` + +2. Wait for the node to accept CQL. The port opens before the database is + ready, so poll with `cqlsh` until it succeeds: + + ```bash + until docker exec cassandra cqlsh -e "SELECT release_version FROM system.local" >/dev/null 2>&1; do + sleep 2 + done + ``` + +3. Enable password authentication. The `cassandra:4.1` image ignores the + `CASSANDRA_AUTHENTICATOR` environment variable, so edit `cassandra.yaml` + directly and restart: + + ```bash + docker exec cassandra sed -i \ + 's/^authenticator: .*/authenticator: PasswordAuthenticator/' \ + /etc/cassandra/cassandra.yaml + docker restart cassandra + ``` + +4. Wait for authentication to come up, then confirm the default superuser login + works. Apache Cassandra's publicly documented default credentials are + `cassandra` / `cassandra`: + + ```bash + until docker exec cassandra cqlsh -u cassandra -p cassandra \ + -e "SELECT release_version FROM system.local" >/dev/null 2>&1; do + sleep 2 + done + ``` + +The node now listens on `127.0.0.1:9042` with password authentication enabled. + +> **Password authentication is required.** `extenddb init` always creates its +> application role with `CREATE ROLE ... WITH PASSWORD`, which Cassandra accepts +> only when the node runs `PasswordAuthenticator`. Against a stock node using the +> default `AllowAllAuthenticator`, `init` fails at the role step with +> `CassandraRoleManager doesn't support PASSWORD`. Do not skip step 3. + +## Connection Details + +| Setting | Value | +|---------|-------| +| Contact point | `127.0.0.1:9042` | +| Superuser (default) | `cassandra` / `cassandra` | +| Keyspace prefix | `extenddb` (default) | +| Catalog keyspace | `_catalog` (e.g. `extenddb_catalog`) | +| Per-account keyspaces | `_account_` | + +ExtendDB creates its keyspaces under the configured `keyspace_prefix`, so one +cluster can host more than one deployment by giving each a different prefix. + +## Building with Cassandra Support + +The Cassandra backend is behind a feature flag. Backends are mutually exclusive +(a build enabling more than one is rejected at compile time), so disable the +default `postgres` feature: + +```bash +cargo build --release --no-default-features --features cassandra +``` + +The binary is single-backend. To run Cassandra and another backend side by +side, build one binary per backend. + +## Initializing ExtendDB with Cassandra + +Initialize against the authenticated local node from above. Pass the contact +point and the superuser credentials used to create the keyspaces and the +application role: + +```bash +./target/release/extenddb init \ + --backend cassandra \ + --cassandra-contact-points 127.0.0.1:9042 \ + --cassandra-user cassandra \ + --cassandra-pass cassandra \ + --config extenddb.toml +``` + +The node must run `PasswordAuthenticator` (step 3). `init` always creates its +application role with a password, so it fails against an unauthenticated +(`AllowAllAuthenticator`) node — there is no credential-free path. + +> **Authorizer.** Run the node with `AllowAllAuthorizer` (the default). `init` +> creates the application role but grants it no permissions, so under +> `CassandraAuthorizer` the server later fails to start with +> `User extenddb has no SELECT permission ...`. Until `init` issues the grants, +> only `AllowAllAuthorizer` works. + +init creates: +- The catalog keyspace (`extenddb_catalog` by default) +- The application role ExtendDB uses at serve time (named `extenddb` by + default; override with `--extenddb-user` / `--extenddb-pass`) +- Admin user credentials (printed to stdout once — save them) +- A self-signed TLS certificate at `~/.extenddb/tls/cert.pem` +- The config file `extenddb.toml` + +Per-account keyspaces are created later, when accounts are added. + +> **Save the admin credentials.** They are printed once and cannot be retrieved +> later. + +## Config Mapping + +init writes a `[storage.cassandra]` section to `extenddb.toml`. It writes the +application credentials **commented out**, with placeholder values: + +```toml +[storage] +backend = "cassandra" + +[storage.cassandra] +contact_points = ["127.0.0.1:9042"] +username = # From extenddb init args +password = # From extenddb init args +keyspace_prefix = "extenddb" +replication_factor = 1 # Single node (use 3 or more for production) +datacenter = "datacenter1" +max_connections = 10 +``` + +> **You must set `username` and `password` before starting the server.** On an +> authenticated node, `serve` with these two fields commented out fails with +> `Connection timeout after 10s`. Uncomment them and set them to the +> application role `init` created — by default user `extenddb` with password +> `extenddb-local-dev` (or whatever you passed to `--extenddb-user` / +> `--extenddb-pass`), **not** the placeholder values above: + +```toml +[storage.cassandra] +contact_points = ["127.0.0.1:9042"] +username = "extenddb" +password = "extenddb-local-dev" +# ... +``` + +Fields: + +- `contact_points` — one or more `host:port` entries. One reachable node is + enough; the driver discovers the rest of the cluster. +- `username` / `password` — the application credentials ExtendDB uses at serve + time. Required against any authenticated node (which `init` requires anyway). +- `keyspace_prefix` — prefix for this deployment's keyspaces (default + `extenddb`). +- `replication_factor` — keyspace replication factor. `init` always writes `1`, + regardless of the cluster. For a multi-node cluster, edit this to 3 or more + **after** `init`; keyspaces created later (per account) use the config value. + Existing keyspaces are not altered retroactively. +- `datacenter` — the local datacenter name for `NetworkTopologyStrategy`. `init` + always writes `datacenter1` and has no flag to change it, and the keyspaces it + creates reference `datacenter1`, so the cluster's local datacenter must be + named `datacenter1`. A node started with a different datacenter name (for + example `CASSANDRA_DC=dc1`) fails keyspace creation with + `Unrecognized strategy option {datacenter1} ... for keyspace extenddb_catalog`. +- `max_connections` — maximum connections per host. + +Edit these values in `extenddb.toml` after `init`. Unlike the PostgreSQL and +MongoDB backends, the Cassandra backend has no single connection-string field; +set `contact_points` and the other fields directly in the config section above. + +> **Connections are not encrypted.** The Cassandra backend connects over plain +> TCP — the driver is built without TLS — so credentials and item data cross the +> network in clear text. Run ExtendDB and Cassandra on a trusted network (for +> example a private subnet or an encrypted overlay); do not send Cassandra +> traffic across an untrusted network. + +## Starting the Server + +```bash +./target/release/extenddb serve --config extenddb.toml +``` + +## Verifying the Connection + +```bash +# Health check +curl --cacert ~/.extenddb/tls/cert.pem https://127.0.0.1:18443/health + +# List tables (should return empty on a fresh deployment) +export AWS_CA_BUNDLE=~/.extenddb/tls/cert.pem +aws dynamodb list-tables \ + --endpoint-url https://127.0.0.1:18443 \ + --region us-east-1 +``` + +You can also confirm the keyspaces exist directly in Cassandra: + +```bash +docker exec cassandra cqlsh -u cassandra -p cassandra \ + -e "DESCRIBE KEYSPACES" | tr ' ' '\n' | grep '^extenddb' +``` + +## Stopping + +```bash +# ExtendDB +./target/release/extenddb stop --config extenddb.toml + +# Docker Cassandra +docker stop cassandra +``` + +## Cluster Deployment + +For deployment concerns common to every backend — single-node vs separated +topologies, containers, the production checklist, systemd, multi-instance +limits — see the [Deployment Guide](manuals/11-deployment-guide.md). This +section covers only what is specific to Cassandra. + +> **Not recommended for production yet.** As of October 2026 the Cassandra +> backend is experimental. It has known limitations — some multi-entity +> management operations are not atomic, and a few concurrency issues are still +> open (see Known Limitations below). Treat a Cassandra deployment as +> evaluation or pre-production, and test it against your own workload before you +> depend on it. + +### Keyspace replication + +Set `replication_factor` to 3 or more for a multi-node cluster. `init` always +writes `1`, so edit the config after `init`; keyspaces created afterward (per +account) use the value in the config. The `datacenter` must be `datacenter1`: +`init` hardcodes that name into the keyspace definitions and provides no flag to +change it, so your cluster's local datacenter has to be named `datacenter1` +until that is configurable. ExtendDB creates its keyspaces with +`NetworkTopologyStrategy` using these values. + +```toml +[storage.cassandra] +contact_points = ["cass-1:9042", "cass-2:9042", "cass-3:9042"] +username = "extenddb" +password = "" +keyspace_prefix = "extenddb" +replication_factor = 3 +datacenter = "datacenter1" # must be datacenter1 (hardcoded by init) +max_connections = 10 +``` + +### Authentication + +Run the cluster with authentication enabled — `init` requires +`PasswordAuthenticator` or stronger, since it creates its application role with +a password. Give ExtendDB a dedicated application role rather than the +`cassandra` superuser, and supply its credentials through `username` and +`password`; both are required. + +Use `AllowAllAuthorizer` for now. `init` creates the application role but grants +it no permissions, so `CassandraAuthorizer` makes `serve` fail with +`User has no SELECT permission ...`. Until `init` issues the grants a +role needs, authorization must be left open at the Cassandra layer. + +### Connection encryption + +The Cassandra backend connects over plain TCP; the driver is compiled without +TLS, so credentials and item data travel in clear text. There is no config +option to enable encryption today. Deploy ExtendDB and the Cassandra cluster on +a trusted, isolated network, and do not route Cassandra traffic over an +untrusted network. + +### Known Limitations + +Understand these before you evaluate a Cassandra deployment. They are why the +backend is experimental rather than production-ready. + +- **Non-atomic management operations.** Some multi-entity management operations + (for example across the IAM entities) are not all-or-nothing on Cassandra. + Cassandra has no multi-row ACID transaction, and ExtendDB uses lightweight + transactions (LWT) for referential integrity, preventing the use of logged + batches for atomicity. A failure part-way through such an operation can leave + partial state. +- **Open concurrency issues.** A small number of concurrency/race conditions + are known (observed as intermittent integration-test failures) and are being + worked. Under concurrent load, outcomes may occasionally diverge from + DynamoDB semantics. +- **Single-datacenter writes per table.** Transaction and TTL lifecycle + operations use `LOCAL_QUORUM` / `LOCAL_SERIAL`, which are linearizable only + within one datacenter. Serve writes for a given table from one Cassandra + datacenter. +- **Multi-node ExtendDB needs node-id configuration.** Running more than one + ExtendDB instance against the cluster requires a distinct node ID per + instance (used to derive stream sequence numbers). +- **TTL throughput.** The TTL sweeper is sized for roughly 1,000 expirations + per table per minute; a table expiring faster accumulates backlog. Enable TTL + before opening a table to writes, or quiesce it briefly while enable and + backfill complete. +- **Clock synchronization.** Keep Cassandra coordinators and ExtendDB hosts + time-synchronized; request batches use explicit timestamps. +- **No vector search.** The Cassandra backend does not implement vector indexes + or `SearchVectors`; every vector operation is refused. + +If you do evaluate a deployment, monitor the TTL expiration backlog, Cassandra +LWT latency, and background-worker health. The design decisions and their +rationale are documented in the Cassandra ADRs (`docs/adr/0010`–`0021`). + +--- + +## License + +Copyright 2026 ExtendDB contributors. Licensed under the Apache License, Version 2.0. +See [LICENSE](../LICENSE) for the full text. + +This software is provided "as is" without warranty of any kind. ExtendDB is not +affiliated with, endorsed by, or sponsored by Amazon Web Services. "DynamoDB" is a trademark +of Amazon.com, Inc. diff --git a/docs/getting-started.md b/docs/getting-started.md index 2f22f158..2c8f65c2 100755 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -38,7 +38,9 @@ See [Local Development with Docker Compose](local-dev-docker-compose.md). - **Storage backend** (one of): - PostgreSQL 14+ running locally (see `docs/local-postgres-setup.md`) + - SQLite (embedded; nothing to install) - MongoDB 7.0+ with replica set (see `docs/local-mongodb-setup.md`) + - Cassandra 4.1+ running locally (see `docs/cassandra-setup.md`) - Rust toolchain (1.88+) - AWS CLI v2 (for testing) - Python 3.10+ with virtual environment (see [Python Environment Setup](../README.md#python-environment-setup) in the README) @@ -53,14 +55,17 @@ cargo build --release # (backends are mutually exclusive; disable the default postgres feature) cargo build --release --no-default-features --features mongodb +# Cassandra backend +cargo build --release --no-default-features --features cassandra + # SQLite backend (developer mode, see "SQLite backend (developer mode)" below) cargo build --release --no-default-features --features sqlite ``` -Each build produces a single-backend binary. The `postgres`, `mongodb`, and -`sqlite` features are mutually exclusive — a build enabling more than one is -rejected at compile time — so to run two backends, build one binary per -backend. +Each build produces a single-backend binary. The `postgres`, `mongodb`, +`sqlite`, and `cassandra` features are mutually exclusive — a build enabling +more than one is rejected at compile time — so to run two backends, build one +binary per backend. The binary is at `target/release/extenddb`. @@ -74,11 +79,14 @@ Run `extenddb init` to create the catalog and data databases: # MongoDB ./target/release/extenddb init --backend mongodb + +# Cassandra +./target/release/extenddb init --backend cassandra ``` This will: -- Create the `extenddb_catalog` database (catalog metadata) -- Create the data database (user item data) +- Create the `extenddb_catalog` database/keyspace (catalog metadata) +- Create the data database/keyspace (user item data) - Generate an AES-256-GCM encryption key (for access key storage) - Create a default account and print the account ID - Create an `admin` user and print the credentials once @@ -168,6 +176,56 @@ EXTENDDB__STORAGE__SQLITE__PATH=/data/other.sqlite ./target/release/extenddb ser Use `:memory:` (in the config, the environment variable, or `init --sqlite-path`) for an ephemeral database that is discarded on shutdown. +### Cassandra backend + +ExtendDB can run against an Apache Cassandra cluster. The Cassandra backend is +experimental; see [Cassandra setup](cassandra-setup.md) for its current +limitations and the full walkthrough. Select it at init time with +`--backend cassandra`: + +```bash +./target/release/extenddb init \ + --backend cassandra \ + --cassandra-contact-points 127.0.0.1:9042 \ + --cassandra-user cassandra \ + --cassandra-pass cassandra +``` + +The node must run `PasswordAuthenticator`. `init` always creates its +application role with a password, so it fails against a stock node using the +default `AllowAllAuthenticator` — there is no credential-free path. Pass the +Cassandra admin credentials with `--cassandra-user` / `--cassandra-pass`; `init` +uses them to create the keyspaces and the application role. `init` writes a +`[storage.cassandra]` section to the config, with the application credentials. + +```toml +[storage.cassandra] +contact_points = ["127.0.0.1:9042"] +username = # From extenddb init args +password = # From extenddb init args +keyspace_prefix = "extenddb" +replication_factor = 1 # init always writes 1; edit for production +datacenter = "datacenter1" # must be datacenter1 (hardcoded by init) +max_connections = 10 +``` + +To override a default, pass the matching flag at init time: + +- `--cassandra-contact-points host1:9042,host2:9042` — one or more contact + points for a remote or multi-node cluster. +- `--cassandra-user` and `--cassandra-pass` — the admin credentials `init` uses + to create the keyspaces and the application role (required). +- `--extenddb-user` and `--extenddb-pass` — override the application role's name + and password (default `extenddb` / `extenddb-local-dev`). +- `--keyspace-prefix myapp` — a different keyspace prefix, so one cluster can + host more than one deployment. + +For a higher replication factor, edit `replication_factor` in the config after +`init` (there is no init flag for it today). The datacenter must be +`datacenter1`. For replication strategy, datacenter topology, authentication, +authorization, and the fact that connections are unencrypted, see +`docs/cassandra-setup.md`. + ### Generating a self-signed certificate manually `extenddb init` auto-generates a self-signed TLS certificate at `~/.extenddb/tls/`. If you need a certificate with different SANs (e.g., binding to `0.0.0.0` and connecting via a specific hostname), generate one manually with `openssl`: diff --git a/docs/manuals/01-architecture-guide.md b/docs/manuals/01-architecture-guide.md index c709f3de..bd05b655 100755 --- a/docs/manuals/01-architecture-guide.md +++ b/docs/manuals/01-architecture-guide.md @@ -4,24 +4,35 @@ ## Overview -extenddb (ExtendDB) is a standalone DynamoDB-compatible API server written in Rust. It receives DynamoDB wire protocol requests over HTTP/HTTPS, authenticates and authorizes them via SigV4 and a local IAM policy engine, executes operation logic in a backend-agnostic engine, and delegates persistence to a pluggable storage backend (currently PostgreSQL). +extenddb (ExtendDB) is a standalone DynamoDB-compatible API server written in Rust. It receives DynamoDB wire protocol +requests over HTTP/HTTPS, authenticates and authorizes them via SigV4 and a local IAM policy engine, executes operation +logic in a backend-agnostic engine, and delegates persistence to a pluggable storage backend (PostgreSQL, SQLite, +MongoDB, or Apache Cassandra). -extenddb runs as a daemon process, logging to syslog. It is designed for any environment where DynamoDB semantics are needed — local development, CI pipelines, self-hosted production, multi-cloud, or air-gapped deployments. Developers and applications point their AWS SDKs at extenddb and get identical DynamoDB behavior. +extenddb runs as a daemon process, logging to syslog. It is designed for any environment where DynamoDB semantics are +needed: local development, CI pipelines, self-hosted production, multi-cloud, or air-gapped deployments. Developers and +applications point their AWS SDKs at extenddb and get identical DynamoDB behavior. ## Cargo Workspace -The project is structured as a Cargo workspace with 7 crates. Crate boundaries enforce dependency rules at compile time. +The project is structured as a Cargo workspace with 13 crates. Crate boundaries enforce dependency rules at compile time. ``` extenddb/ ├── crates/ -│ ├── core/ Pure sync Rust: types, expressions, validation, errors -│ ├── engine/ Async operation handlers (PutItem, Query, etc.) -│ ├── storage/ Storage trait definitions and backend-agnostic utilities -│ ├── storage-postgres/ PostgreSQL backend implementation -│ ├── auth/ AuthProvider trait, SigV4 verification, IAM policy engine -│ ├── server/ HTTP server (axum), management API, web console -│ └── bin/ CLI entry point, config loading, daemon lifecycle +│ ├── app/ Application/CLI library the bin crate dispatches to +│ ├── auth/ AuthProvider trait, SigV4 verification, IAM policy engine +│ ├── bin/ CLI entry point, config loading, daemon lifecycle +│ ├── cache/ Stale-while-revalidate cache used by the auth and server layers +│ ├── config/ Configuration types and loading +│ ├── core/ Pure sync Rust: types, expressions, validation, errors +│ ├── engine/ Async operation handlers (PutItem, Query, etc.) +│ ├── server/ HTTP server (axum), management API, web console +│ ├── storage/ Storage trait definitions and backend-agnostic utilities +│ ├── storage-cassandra/ Cassandra backend implementation (experimental) +│ ├── storage-mongodb/ MongoDB backend implementation +│ ├── storage-postgres/ PostgreSQL backend implementation (default) +│ └── storage-sqlite/ SQLite backend implementation (embedded) ``` ## Crate Dependency Graph @@ -34,11 +45,14 @@ bin ──→ server ──→ engine ──→ core │ ├──→ auth ──→ core │ └──→ storage │ - ├──→ storage-postgres ──→ storage ──→ core + ├──→ storage-{postgres,sqlite,mongodb,cassandra} ──→ storage ──→ core └──→ core ``` -Key principle: `core` depends on nothing in the workspace and has no async runtime. It contains pure sync Rust — types, expression parsing and evaluation, input validation, capacity calculation, and error types. Every other crate depends on `core`. +Exactly one `storage-*` backend is linked per binary, selected by mutually exclusive Cargo features at build time. + +Key principle: `core` depends on nothing in the workspace and has no async runtime. It contains pure sync Rust: types, +expression parsing and evaluation, input validation, capacity calculation, and error types. Every other crate depends on `core`. ## Crate Roles @@ -68,7 +82,7 @@ The `dispatch` function routes `X-Amz-Target` operation names to handlers. ### storage -Trait definitions for the storage layer. Thirteen required storage traits partition backend responsibilities: +Trait definitions for the storage layer. The required storage traits partition backend responsibilities: - **TableEngine**: Table lifecycle (create, delete, describe, list, update) - **DataEngine**: Item CRUD (put, get, update, delete, query, scan, batch, transact) @@ -86,9 +100,14 @@ Trait definitions for the storage layer. Thirteen required storage traits partit Two further traits are optional and belong to vector search: **VectorSearchEngine**, which a backend hands over through `as_vector_search` or declines by returning `None`, and **VectorIndexBuild**, the storage primitives the shared build lifecycle drives. A backend that implements neither refuses every vector operation and is otherwise unaffected. - **Bootstrapper**: Initial database setup -Traits use `BoxFuture` for object safety. A binary serves exactly one backend, installed from its thin `main` via `set_backend`. The `ServerRuntimeHooks` trait allows backends to spawn backend-specific workers (PostgreSQL spawns 7). +Traits use `BoxFuture` for object safety. A binary serves exactly one backend, installed from its thin `main` via `set_backend`. The `ServerRuntimeHooks` trait allows backends to spawn backend-specific workers. + +### Storage backends -### storage-postgres +Each backend implements the storage traits for one database. A built ExtendDB binary links exactly one. PostgreSQL +is the default and reference backend; Apache Cassandra is experimental; MongoDB and SQLite are also available. + +#### storage-postgres PostgreSQL implementation of all storage traits using `sqlx`. Features: @@ -100,6 +119,20 @@ PostgreSQL implementation of all storage traits using `sqlx`. Features: - Stream records stored in a dedicated table with background cleanup - All queries parameterized (no dynamic SQL construction) +#### storage-cassandra + +Cassandra implementation of the storage traits using `cdrs-tokio`. Experimental. +Features: + +- A catalog keyspace for metadata and a keyspace per account for items, under a configurable `keyspace_prefix` +- Items stored as JSON, keyed by Cassandra partition and clustering columns +- GSI/LSI implemented as separate index tables (see ADR-0016) +- Transactions use a lightweight-transaction (LWT) two-phase commit, with a background worker to recover abandoned transactions (see ADR-0017) +- Streams and TTL run from background workers, with a durable TTL expiration queue (see ADR-0010, ADR-0018) + +> **Note:** Cassandra has no multi-row ACID primitive. Some multi-entity operations trade all-or-nothing atomicity for +> the integrity that LWT provides (see ADR-0012, ADR-0021). + ### auth Authentication and authorization: @@ -153,7 +186,7 @@ expect. (skipped entirely in `--foreground` mode, so no run directory is needed and the root filesystem can be read-only) 4. Initialize logging — syslog when daemonized, stderr in `--foreground` -5. Connect to PostgreSQL (catalog + data databases) +5. Connect to the storage backend 6. Verify catalog version matches binary expectation 7. Start axum server on the pre-bound socket @@ -180,7 +213,11 @@ A mismatch between the compiled-in version and the stored one prevents a Postgre ### Storage -Storage backends implement thirteen traits (see **storage** section above). The traits use `BoxFuture` for object safety. A binary serves exactly one backend: the thin `bin` crate installs it via `set_backend` before any subcommand runs, and the config file's `backend` key is validated against the installed name rather than selecting it. PostgreSQL, SQLite and MongoDB backends are implemented, selected by mutually exclusive Cargo features at build time. +Storage backends implement the storage traits (see **storage** section above). The traits use `BoxFuture` for object safety. +A built ExtendDB binary serves exactly one backend. The thin `bin` crate installs the selected backend via `set_backend` +before any subcommand runs, and the config file's `backend` key is validated against the installed name rather than +selecting it. The PostgreSQL, SQLite, MongoDB, and Cassandra backends are selected as mutually exclusive Cargo features +at build time. ### Authentication @@ -190,7 +227,7 @@ Auth providers implement the `AuthProvider` trait using `#[async_trait]` for obj ### Authentication -SigV4 signature verification follows the AWS specification. Credentials are stored encrypted (AES-256-GCM) in PostgreSQL. Access key prefixes distinguish long-term (`VDAK`) from temporary (`VDSK`) credentials. +SigV4 signature verification follows the AWS specification. Credentials are stored encrypted (AES-256-GCM) in the storage backend. Access key prefixes distinguish long-term (`AKIA`) from temporary (`ASIA`) credentials. ### Authorization @@ -262,7 +299,11 @@ Configuration precedence: CLI flags > environment variables > config file > defa ## DynamoDB Streams -extenddb implements DynamoDB Streams for change data capture. Stream records are captured atomically with data writes inside the same PostgreSQL transaction. Both the DynamoDB API and Streams API are served on the same port. +Extenddb implements DynamoDB Streams for change data capture. On the PostgreSQL, SQLite and MongoDB backends, stream records +are captured atomically with the data writes that produce them, inside the same transaction. On the Cassandra backend, an +item mutation and the related stream record creation are generally non-atomic (TTL-enabled tables are the exception). A +failure between the two can leave the item written with no stream record. Unlike Amazon DynamoDB, with ExtendDB both the +DynamoDB API and Streams API are served on the same port. Supported operations: `ListStreams`, `DescribeStream`, `GetShardIterator`, `GetRecords`. @@ -270,14 +311,14 @@ Stream records are retained for 24 hours. A background worker cleans up expired ## Deployment Models -extenddb is a single-binary server that connects to PostgreSQL. Deployment options include: +extenddb is a single-binary server that connects to a storage backend. Deployment options include: - **Single-node**: extenddb + PostgreSQL on the same host (development, small workloads) - **Separated**: extenddb on an application server, PostgreSQL on a dedicated database server or managed service (RDS, Aurora, Cloud SQL) - **Containerized**: Docker/Kubernetes with PostgreSQL as a sidecar or external service - **Air-gapped**: No internet connectivity required; all functionality is self-contained -PostgreSQL provides the durability, replication, and backup capabilities. Use standard PostgreSQL HA tools (streaming replication, Patroni, managed services) for production availability. +The storage backend provides durability, replication, and backup. Use that backend's standard HA tooling for production availability. --- diff --git a/docs/manuals/02-design-guide.md b/docs/manuals/02-design-guide.md index a841b827..26fac44c 100755 --- a/docs/manuals/02-design-guide.md +++ b/docs/manuals/02-design-guide.md @@ -4,14 +4,16 @@ ## Storage Schema -### Dual-Database Architecture +### Catalog and Data Separation -extenddb uses two PostgreSQL databases per deployment: +ExtendDB manages two kinds of stored state in every storage backend. The +examples below name the PostgreSQL databases; other backends map the same two +roles onto their own structures (see backend-specific documentation). -- **Catalog database** (e.g., `extenddb_catalog`): All metadata — table definitions, indexes, accounts, IAM entities, settings, stream metadata, and schema history. -- **Data database** (e.g., `extenddb_catalog_data`): User item data. Each DynamoDB table maps to a PostgreSQL table. GSI and LSI data are stored in separate PostgreSQL tables. +- **Catalog** (e.g., the `extenddb_catalog` PostgreSQL database): All metadata — table definitions, indexes, accounts, IAM entities, settings, stream metadata, and schema history. +- **Data** (e.g., the `extenddb_catalog_data` PostgreSQL database): User item data. Each DynamoDB table maps to a PostgreSQL table. GSI and LSI data are stored in separate PostgreSQL tables. -The data database connection string is stored in the catalog's `settings` table under the key `data_database_url`. This allows the catalog and data databases to live on different PostgreSQL instances. +On PostgreSQL, the data database connection string is stored in the catalog's `settings` table under the key `data_database_url`. This allows the catalog and data databases to live on different PostgreSQL instances. ### Catalog Tables diff --git a/docs/manuals/11-deployment-guide.md b/docs/manuals/11-deployment-guide.md index 7209c2cf..77624ce3 100755 --- a/docs/manuals/11-deployment-guide.md +++ b/docs/manuals/11-deployment-guide.md @@ -4,27 +4,36 @@ This guide covers deploying extenddb in various environments beyond local development. +> **Backend-specific sections.** This guide covers deployment concerns common to +> all storage backends, and uses PostgreSQL — the default and reference backend — +> for concrete examples. Sections headed *PostgreSQL* apply only to that backend. +> For other backends, see their setup docs: +> [Cassandra setup](../cassandra-setup.md) (experimental; local and production) +> and [MongoDB setup](../local-mongodb-setup.md). SQLite does not need a separate +> backend service; see its section in the getting-started guide. + ## Architecture Overview -extenddb is a single Rust binary that connects to PostgreSQL. All durable state lives in PostgreSQL; extenddb keeps only short-lived credential, policy, and table-metadata caches in process. This means: +extenddb is a single Rust binary that connects to a storage backend. All durable state lives in the backend; extenddb keeps only short-lived credential, policy, and table-metadata caches in process. This means: -- Multiple extenddb instances can share a PostgreSQL catalog (with caveats — see Multi-Instance below) -- Standard PostgreSQL HA, backup, and replication tools provide durability -- extenddb can run anywhere PostgreSQL is reachable +- Multiple extenddb instances can share one backend catalog (with caveats — see Multi-Instance below) +- The backend's own HA, backup, and replication tools provide durability +- extenddb can run anywhere the storage backend is reachable ## Deployment Models ### Single-Node -extenddb and PostgreSQL on the same host. Simplest setup, suitable for development, CI, and small workloads. +ExtendDB and its storage backend on the same host. Simplest setup, suitable for development, CI, and small workloads. ``` -┌─────────────────────────┐ -│ Host │ -│ ┌─────┐ ┌──────────┐ │ -│ │ extenddb│──│PostgreSQL │ │ -│ └─────┘ └──────────┘ │ -└─────────────────────────┘ +┌──────────────────────────────┐ +│ Host │ +│ ┌──────────┐ ┌───────────┐ │ +│ │ ExtendDB │──│ Storage │ │ +│ │ │ │ backend │ │ +│ └──────────┘ └───────────┘ │ +└──────────────────────────────┘ ``` ```bash @@ -34,51 +43,32 @@ extenddb serve --config extenddb.toml ### Separated Database -extenddb on an application server, PostgreSQL on a dedicated database server or managed service. +ExtendDB on an application server, the storage backend on a dedicated database +server or managed service. ``` ┌──────────┐ ┌──────────────┐ -│ App Host│──────▶│ DB Host │ -│ extenddb │ │ PostgreSQL │ +│ App Host │──────▶│ DB Host │ +│ ExtendDB │ │ Backend │ └──────────┘ └──────────────┘ ``` -```bash -extenddb init \ - --pg-host db.example.com \ - --pg-pass -``` - -Configure the connection string in `extenddb.toml`: - -```toml -[storage.postgres] -connection_string = "postgresql://extenddb:@db.example.com:5432/extenddb_catalog?sslmode=require" -pool_size = 20 -``` - -Use `sslmode=require` (or `verify-full` with a CA certificate) for encrypted database connections. - -### Managed PostgreSQL +Point ExtendDB at the remote backend at init time, and use an encrypted +connection. The flags and connection-string format are backend-specific: for +PostgreSQL see [Configuring PostgreSQL](#configuring-postgresql) below; for +Cassandra see [Cassandra setup](../cassandra-setup.md). -extenddb works with any PostgreSQL 14+ service: +### Managed Database -- **Amazon RDS for PostgreSQL** / **Amazon Aurora PostgreSQL** -- **Google Cloud SQL for PostgreSQL** -- **Azure Database for PostgreSQL** -- **Self-managed PostgreSQL** with streaming replication - -```bash -# Amazon RDS / Aurora example -extenddb init \ - --pg-host mydb.cluster-abc123.us-east-1.rds.amazonaws.com \ - --pg-user extenddb_admin \ - --pg-pass -``` +ExtendDB works with managed database services as well as self-managed servers. +The managed-service topology is the same as Separated Database: ExtendDB on an +application host, the backend provided as a service. For the PostgreSQL managed +services ExtendDB supports (Amazon RDS and Aurora, Google Cloud SQL, Azure +Database), see [Configuring PostgreSQL](#configuring-postgresql). ### Containerized -extenddb runs in Docker or Kubernetes. The binary has no runtime dependencies beyond libc and network access to PostgreSQL. +ExtendDB runs in Docker or Kubernetes. The binary has no runtime dependencies beyond libc and network access to the storage backend. Example Dockerfile: @@ -124,10 +114,10 @@ graceful shutdown. Outside a container, where you may want `stop` to work, add `--write-pid-file` — it writes the PID file to the usual `run_dir` path, which then has to be writable. -What the health check covers: `extenddb healthcheck` exits 0 when the server is +`extenddb healthcheck` exits 0 when the server is listening, completing TLS, and serving HTTP. Use it as a liveness probe, which is what a Docker `HEALTHCHECK` and a Kubernetes `livenessProbe` are for. `/health` -does not query PostgreSQL, and for a liveness probe that is the behaviour you +does not query the storage backend, and for a liveness probe that is the behaviour you want: one that failed on a database outage would restart every replica at once. A backend that is unreachable at startup does stop the server from listening, so container start-up failures are still caught. @@ -141,12 +131,62 @@ For Kubernetes, run `extenddb init` as an init container or a one-time Job, then ### Air-Gapped -extenddb requires no internet connectivity. All functionality is self-contained in the binary. Build on a connected host, transfer the binary and `extenddb.toml` to the air-gapped environment, and run. +ExtendDB requires no internet connectivity. All functionality is self-contained in the binary. Build on a connected host, transfer the binary and `extenddb.toml` to the air-gapped environment, and run. Requirements in the air-gapped environment: -- PostgreSQL 14+ (reachable from the extenddb host) +- A supported storage backend, reachable from the ExtendDB host - The `extenddb` binary (statically linked or with matching libc) +## Configuring PostgreSQL + +This section applies to the PostgreSQL backend. For Cassandra, see +[Cassandra setup](../cassandra-setup.md); for MongoDB, see +[MongoDB setup](../local-mongodb-setup.md). + +### Remote and separated databases + +Point ExtendDB at a remote PostgreSQL server at init time: + +```bash +extenddb init \ + --pg-host db.example.com \ + --pg-pass +``` + +Configure the connection string in `extenddb.toml`: + +```toml +[storage.postgres] +connection_string = "postgresql://extenddb:@db.example.com:5432/extenddb_catalog?sslmode=require" +pool_size = 20 +``` + +Use `sslmode=require` (or `verify-full` with a CA certificate) for encrypted +database connections. + +The Cassandra backend cannot encrypt its connection today: the driver is +compiled without TLS, so credentials and item data cross the network in clear +text and there is no config option to turn encryption on. Run ExtendDB and the +Cassandra cluster on a trusted, isolated network; do not route Cassandra traffic +over an untrusted network. + +### Managed PostgreSQL services + +ExtendDB works with any PostgreSQL 14+ service: + +- **Amazon RDS for PostgreSQL** / **Amazon Aurora PostgreSQL** +- **Google Cloud SQL for PostgreSQL** +- **Azure Database for PostgreSQL** +- **Self-managed PostgreSQL** with streaming replication + +```bash +# Amazon RDS / Aurora example +extenddb init \ + --pg-host mydb.cluster-abc123.us-east-1.rds.amazonaws.com \ + --pg-user extenddb_admin \ + --pg-pass +``` + ## Production Checklist ### Authentication @@ -160,15 +200,15 @@ Requirements in the air-gapped environment: - [ ] TLS is enabled by default. Verify it is not disabled in `extenddb.toml` - [ ] Replace the self-signed certificate with a CA-signed certificate -- [ ] Use `sslmode=require` or `sslmode=verify-full` in the PostgreSQL connection string +- [ ] Encrypt the connection to the storage backend. For PostgreSQL, use `sslmode=require` or `sslmode=verify-full` in the connection string. The Cassandra backend cannot encrypt its connection — place ExtendDB and the Cassandra cluster on a trusted, isolated network instead ### Network - [ ] Set `bind_addr` to the appropriate interface (not `0.0.0.0` unless behind a load balancer) -- [ ] Firewall: allow only necessary ports (extenddb port, PostgreSQL port) +- [ ] Firewall: allow only necessary ports (extenddb port, storage backend port) - [ ] Consider a reverse proxy (nginx, HAProxy) for TLS termination, rate limiting, and access logging -### Database +### Database (PostgreSQL) - [ ] Use a dedicated PostgreSQL user for extenddb with minimal privileges - [ ] Configure PostgreSQL `max_connections` ≥ extenddb `pool_size` + 3 @@ -229,9 +269,11 @@ Run one extenddb instance per catalog. Multiple instances behind a load balancer - Every instance runs every background worker (TTL expiry, GSI backfill, stream retention, control-plane transitions). The workers are safe to run twice, but the work is duplicated. - `/health` is a liveness check that does not query the database, so a load balancer cannot use it to take an instance with a broken database connection out of rotation. -On SQLite this limit is enforced: `extenddb serve` holds an exclusive lock on `.lock` and a second server on the same file refuses to start. On PostgreSQL and MongoDB it is not enforced; if you start a second instance anyway, size `pool_size × instance_count + 3 × instance_count ≤ PostgreSQL max_connections`. +On SQLite this limit is enforced: `extenddb serve` holds an exclusive lock on `.lock` and a second server on the same file refuses to start. On PostgreSQL, MongoDB, and Cassandra it is not enforced. On PostgreSQL, if you start a second instance anyway, size `pool_size × instance_count + 3 × instance_count ≤ PostgreSQL max_connections`. + +## Performance Tuning (PostgreSQL) -## Performance Tuning +This section covers the PostgreSQL backend. ### Connection Pool