Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 31 additions & 17 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,13 +4,13 @@ This document provides AI agents with essential context for working on the Exten

## Project Overview

**ExtendDB** is an open source DynamoDB-compatible runtime backed by PostgreSQL storage, released under the Apache 2.0
**ExtendDB** is an open source DynamoDB-compatible runtime with pluggable storage, released under the Apache 2.0
License. ExtendDB is a clean-room implementation of the DynamoDB wire protocol, written in Rust and initially developed
by AWS engineers. It is not a fork of DynamoDB and contains no DynamoDB source code. ExtendDB speaks the DynamoDB wire
protocol: any AWS SDK, CLI, or tool that works with DynamoDB works with ExtendDB, unchanged.

- **Language:** Rust (edition 2024, MSRV 1.88+)
- **Storage backends:** PostgreSQL 14+ (default), SQLite (bundled, no server or client install required; feature flag `sqlite`, also `sqlite-memory` for dev and CI), MongoDB 7.0+ (feature flag `mongodb`). One backend per binary, selected by mutually exclusive Cargo features.
- **Storage backends:** PostgreSQL 14+ (default), SQLite (bundled, no server or client install required; feature flag `sqlite`, also `sqlite-memory` for dev and CI), MongoDB 7.0+ (feature flag `mongodb`), Cassandra 4.1+ (experimental; feature flag `cassandra`). One backend per binary, selected by mutually exclusive Cargo features.
- **Architecture:** Async (tokio), trait-based storage abstraction
- **Authentication:** Mandatory SigV4 with built-in IAM (users, groups, roles, policies)
- **TLS:** Mandatory (self-signed cert generated by default)
Expand All @@ -28,18 +28,19 @@ protocol: any AWS SDK, CLI, or tool that works with DynamoDB works with ExtendDB
```
extenddb/
├── crates/
│ ├── app/ # Shared CLI surface the thin bin dispatches to
│ ├── auth/ # SigV4 verification, IAM policy engine
│ ├── bin/ # CLI, config, daemon lifecycle (extenddb binary)
│ ├── cache/ # Auth and authz caches
│ ├── config/ # Config parsing shared by the CLI and the server
│ ├── core/ # Pure sync types, validation, expression parsing (no async)
│ ├── engine/ # DynamoDB operation handlers (PutItem, Query, etc.)
│ ├── server/ # HTTP server, management API, web console
│ ├── storage/ # Storage trait definitions (TableEngine trait)
│ ├── storage-postgres/ # PostgreSQL implementation of TableEngine
│ ├── storage-sqlite/ # SQLite implementation (dev, CI, and embedded use)
│ ├── storage-cassandra/ # Cassandra implementation (experimental; feature flag)
│ ├── storage-mongodb/ # MongoDB implementation (feature flag)
│ ├── cache/ # Auth and authz caches
│ ├── config/ # Config parsing shared by the CLI and the server
│ ├── app/ # Shared CLI surface the thin bin dispatches to
│ ├── auth/ # SigV4 verification, IAM policy engine
│ ├── server/ # HTTP server, management API, web console
│ └── bin/ # CLI, config, daemon lifecycle (extenddb binary)
│ ├── storage-postgres/ # PostgreSQL implementation (default/reference)
│ └── storage-sqlite/ # SQLite implementation (dev, CI, and embedded use)
├── tests/ # Python integration tests (pytest)
├── docs/
│ ├── design/ # Architecture and component design docs
Expand All @@ -59,6 +60,7 @@ The `TableEngine` trait in `crates/storage/src/lib.rs` defines the storage inter
- `storage-postgres` (PostgreSQL) — default backend
- `storage-sqlite` (SQLite) — feature flag `sqlite`, and `sqlite-memory` for dev and CI
- `storage-mongodb` (MongoDB) — feature flag `mongodb`
- `storage-cassandra` (Cassandra) — experimental; feature flag `cassandra`

The trait uses RPITIT (return-position impl Trait in traits) for async methods — no `#[async_trait]` macro.

Expand All @@ -73,9 +75,10 @@ extenddb (bin)
│ ├─> extenddb-core (pure sync, no async)
│ └─> extenddb-storage (trait definitions)
├─> extenddb-auth
├─> extenddb-storage-postgres (feature: postgres)
├─> extenddb-storage-sqlite (feature: sqlite / sqlite-memory)
└─> extenddb-storage-mongodb (feature: mongodb)
├─> extenddb-storage-postgres (feature: postgres)
├─> extenddb-storage-sqlite (feature: sqlite / sqlite-memory)
├─> extenddb-storage-mongodb (feature: mongodb)
└─> extenddb-storage-cassandra (feature: cassandra)
```

- **extenddb-core:** Pure synchronous Rust. No async, no I/O. Types, validation, expression parsing.
Expand All @@ -84,6 +87,7 @@ extenddb (bin)
- **extenddb-storage-sqlite:** Concrete SQLite implementation, the dev and CI backend.
It serves the same wire surface, vector search included, and needs no server.
- **extenddb-storage-mongodb:** Concrete MongoDB implementation.
- **extenddb-storage-cassandra:** Concrete Cassandra implementation. Experimental.
- **extenddb-engine:** Operation handlers that call storage traits.
- **extenddb-server:** HTTP server, management API, web console.
- **extenddb-auth:** SigV4 signature verification, IAM policy evaluation.
Expand All @@ -96,7 +100,9 @@ extenddb (bin)
- Rust 1.88+ (`rustup update`)
- Storage backend (one of):
- PostgreSQL 14+ running locally (see `docs/local-postgres-setup.md`)
- SQLite (embedded; nothing to install)
- MongoDB 7.0+ with replica set (see `docs/local-mongodb-setup.md`)
- Cassandra 4.1+ (experimental; see `docs/cassandra-setup.md`)
- Python 3.10+ for tests (`python3 -m venv ~/venvs/extenddb-venv && source ~/venvs/extenddb-venv/bin/activate && pip install -r requirements.txt`)

### Build
Expand All @@ -107,11 +113,14 @@ cargo build --release

# MongoDB backend (backends are mutually exclusive — disable default postgres)
cargo build --release --no-default-features --features mongodb

# Cassandra backend (experimental)
cargo build --release --no-default-features --features cassandra
```

Backends are selected by mutually exclusive Cargo features (`postgres`,
`mongodb`, `sqlite`); a build enabling more than one is rejected at compile
time. Each binary serves exactly one backend.
`mongodb`, `sqlite`, `cassandra`); a build enabling more than one is rejected at
compile time. Each binary serves exactly one backend.

Binary: `target/release/extenddb`

Expand All @@ -136,6 +145,9 @@ cargo clippy --all-targets -- -D warnings

# MongoDB
./target/release/extenddb init --backend mongodb --config extenddb.toml

# Cassandra (experimental)
./target/release/extenddb init --backend cassandra --config extenddb.toml
```

This creates:
Expand Down Expand Up @@ -339,7 +351,7 @@ Query, Scan (key conditions, filters, projections, pagination, index selection)
BatchGetItem (100 keys), BatchWriteItem (25 ops), TransactGetItems (100 items), TransactWriteItems (100 ops)

### Vector Search
SearchVectors, and vector indexes on CreateTable and UpdateTable (COSINE, EUCLIDEAN, DOT_PRODUCT; on-demand tables only; five per table). Served by the PostgreSQL and SQLite backends; MongoDB provides no implementation and refuses every vector operation. PostgreSQL additionally requires the pgvector extension on the data database, probed once at startup, and refuses without it. The refusal is two strings, which matters when grepping logs: `Vector indexes are not supported by this storage backend` for CreateTable and UpdateTable, and `SearchVectors is not supported by this storage backend` for a search.
SearchVectors, and vector indexes on CreateTable and UpdateTable (COSINE, EUCLIDEAN, DOT_PRODUCT; on-demand tables only; five per table). Served by the PostgreSQL and SQLite backends; MongoDB and Cassandra provide no implementation and refuse every vector operation. PostgreSQL additionally requires the pgvector extension on the data database, probed once at startup, and refuses without it. The refusal is two strings, which matters when grepping logs: `Vector indexes are not supported by this storage backend` for CreateTable and UpdateTable, and `SearchVectors is not supported by this storage backend` for a search.

### Streams
ListStreams, DescribeStream, GetShardIterator, GetRecords
Expand Down Expand Up @@ -387,8 +399,10 @@ Expression parsing lives in `crates/core/src/expression/`. This is pure sync Rus
| Troubleshooting | `docs/troubleshooting.md` | Common errors and solutions |
| Storage Component Design | `docs/design/04-component-storage.md` | Storage trait design |
| MongoDB Design | `docs/design/13-storage-mongodb.md` | MongoDB backend architecture |
| Cassandra Design (experimental) | `docs/adr/0010`–`0021` | Cassandra backend design decisions (ADRs) |
| Local PostgreSQL Setup | `docs/local-postgres-setup.md` | PostgreSQL installation and config |
| Local MongoDB Setup | `docs/local-mongodb-setup.md` | MongoDB installation and config |
| Cassandra Setup | `docs/cassandra-setup.md` | Cassandra install, config, and deployment (experimental) |
| Testing Design | `docs/design/09-testing.md` | Test strategy and infrastructure |
| Extending Storage | `12-backend-plugin-architecture.md` | Guide to implementing storage backends |

Expand Down Expand Up @@ -480,7 +494,7 @@ Activate when the user asks about installing, configuring, running, or debugging
1. **TLS is mandatory** — server refuses to start without it. Use `AWS_CA_BUNDLE` for self-signed certs.
2. **Auth is mandatory** — all requests must be SigV4-signed. Use `extenddb manage` or web console to create credentials.
3. **Account isolation** — all operations are scoped to `account_id`. Different accounts can have tables with the same name.
4. **Storage backend must be running** — `extenddb init` and `extenddb serve` require a running PostgreSQL or MongoDB instance.
4. **Storage backend must be running** — `extenddb init` and `extenddb serve` require a running external backend (PostgreSQL, MongoDB, or Cassandra). SQLite is embedded and needs no server.
5. **MongoDB requires a replica set** — even single-node MongoDB must be configured with `--replSet` for transactions and streams.
6. **Python venv** — activate the venv before running tests: `source ~/venvs/extenddb-venv/bin/activate`
7. **Test credentials** — run `devtools/provision-test-credentials` before pytest to create test users and keys.
Expand Down
63 changes: 42 additions & 21 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,27 +1,31 @@
# ExtendDB

> **ExtendDB is an independent open source project managed by Amazon Web Services. It is not Amazon DynamoDB and does not contain any DynamoDB source code.** "DynamoDB" is a trademark of Amazon.com, Inc. ExtendDB is a clean-room implementation that speaks the DynamoDB wire protocol. Behavioral differences from the real service are documented in [Differences from DynamoDB](docs/differences-from-dynamodb.md).
> **ExtendDB is an independent open source project managed by Amazon Web Services. It is not Amazon DynamoDB and does
> not contain any DynamoDB source code.** "DynamoDB" is a trademark of Amazon.com, Inc. ExtendDB is a clean-room
> implementation that speaks the DynamoDB wire protocol. Behavioral differences from the real service are documented
> in [Differences from DynamoDB](docs/differences-from-dynamodb.md).

A DynamoDB-compatible API adapter, ExtendDB speaks the DynamoDB wire protocol — any AWS SDK, CLI, or tool that works with DynamoDB works with ExtendDB, unchanged.
A DynamoDB-compatible API adapter, ExtendDB speaks the DynamoDB wire protocol — any AWS SDK, CLI, or tool that works
with DynamoDB works with ExtendDB, unchanged.

## Use Cases

- **Local development** — run DynamoDB workloads on your laptop with zero cloud dependency
- **CI/CD pipelines** — deterministic integration tests against a DynamoDB-compatible backend
- **Self-hosted deployments** — run DynamoDB workloads on your own infrastructure (on-premises, private cloud, edge)
- **Multi-cloud** — use DynamoDB semantics on any cloud that runs PostgreSQL
- **Multi-cloud** — use DynamoDB semantics on any cloud that runs an ExtendDB-supported database backend
- **Air-gapped environments** — DynamoDB functionality with no internet connectivity

## Features

- Full DynamoDB wire protocol: CRUD, Query, Scan, Batch, Transactions, Streams, TTL, Import/Export
- Pluggable storage: PostgreSQL (default), SQLite, MongoDB, and Cassandra
- SigV4 authentication with IAM compatibility: users, groups, roles, policies, permissions boundaries
- Web management console for account and credential administration
- TLS with automatic self-signed certificate generation (replaceable with CA-signed certs)
- CSRF protection, security headers, session management
- JSON metrics endpoint with DynamoDB CloudWatch-style metric names and dimensions
- Daemon mode with syslog logging, plus `--foreground` for container and supervisor environments
- PostgreSQL storage — use standard backup, replication, and HA tools

## Quick Start

Expand Down Expand Up @@ -50,8 +54,12 @@ scripts/install-macos.sh # macOS
## Prerequisites

- Rust 1.88+ (`rustup update`)
- PostgreSQL 14+ (see `docs/local-postgres-setup.md`)
- Python 3.10+ (for test suites and documentation)
- A storage backend:
- PostgreSQL 14+ (see `docs/local-postgres-setup.md`)
- SQLite (embedded)
- MongoDB 7+ (see `docs/local-mongodb-setup.md`)
- Cassandra 4.1+ (see `docs/cassandra-setup.md`)

### Python Environment

Expand Down Expand Up @@ -106,11 +114,10 @@ key_path = "/etc/extenddb/tls/key.pem"

## Running in Containers

The production images contain ExtendDB only; the database (PostgreSQL 14+ and MongoDB 7+)
runs as a separate service. See [`docker/README.md`](docker/README.md) for the PostgreSQL
image and [`docker/README-mongodb.md`](docker/README-mongodb.md) for the MongoDB
image, external database requirements, hardened local Compose stacks, smoke tests, and
explicit init/migrate/serve production lifecycle.
The production images contain ExtendDB only. The storage backend runs as a separate service. See
[`docker/README.md`](docker/README.md) for the PostgreSQL image and [`docker/README-mongodb.md`](docker/README-mongodb.md) for the MongoDB image,
external database requirements, hardened local Compose stacks, smoke tests, and explicit init/migrate/serve
production lifecycle. The SQLite and Cassandra backends have no dedicated container guide yet.

By default `extenddb serve` daemonizes itself, which doesn't play well with container runtimes and process supervisors that expect PID 1 to stay attached. Pass `--foreground` (alias `--no-daemon`) to keep the process in the foreground and stream logs to stderr instead of syslog:

Expand All @@ -134,11 +141,20 @@ extenddb healthcheck --config extenddb.toml # exit 0 healthy, 1 not
extenddb healthcheck --endpoint https://127.0.0.1:18443 # explicit target
```

This is a liveness check, which is what a container `HEALTHCHECK` wants: `/health` does not query the storage backend, so it reports healthy even if PostgreSQL becomes unreachable after startup. That is deliberate, since a liveness probe that failed on a database outage would restart every replica at once. A backend that is unreachable at startup does stop the server from listening, so that case is caught. There is no separate readiness endpoint yet.
This is a liveness check, which is what a container `HEALTHCHECK` wants. The `/health` path does not query the storage
backend, so it will report healthy even if the storage backend becomes unreachable after startup. That is deliberate,
since a liveness probe that failed on a database outage would restart every replica at once. A backend that is
unreachable at startup does stop the server from listening. There is no separate readiness endpoint yet.

### One instance per database

Run one `extenddb serve` per catalog. On SQLite this is enforced: the server takes an exclusive lock on `<database>.lock` next to the database file and a second server on the same file refuses to start (`init`, `migrate`, and `destroy` take the same lock while they run). On PostgreSQL and MongoDB nothing prevents a second instance, but running more than one against the same catalog is not supported in this release: every instance runs every background worker, credential and policy caches are per instance with no cross-instance invalidation (a revoked key stays valid on the other instance for up to the cache TTL), and `/health` does not check the database. See the [deployment guide](docs/manuals/11-deployment-guide.md#multi-instance-considerations).
Run one `extenddb serve` per catalog. On SQLite this is enforced: the server takes an exclusive lock on `<database>.lock`
next to the database file and a second server on the same file will refuse to start (`init`, `migrate`, and `destroy` take
the same lock while they run). On PostgreSQL, MongoDB and Cassandra nothing prevents a second ExtendDB instance from
running, but running more than one against the same catalog is not supported in this release: every instance runs every
background worker, credential and policy caches are per instance with no cross-instance invalidation (a revoked key
stays valid on the other instance for up to the cache TTL), and `/health` does not check the database.
See the [deployment guide](docs/manuals/11-deployment-guide.md#multi-instance-considerations).

To make the generated self-signed certificate valid for the name clients use — an in-cluster service DNS name, for example — pass `--tls-san` to `init` (repeatable):

Expand Down Expand Up @@ -174,7 +190,7 @@ Web-based administration at `https://127.0.0.1:18443/console/`. Manage accounts,
```
extenddb serve --config extenddb.toml # Start server (daemon)
extenddb serve --config extenddb.toml --foreground # Start in foreground
extenddb init --catalog-db NAME # Initialize deployment
extenddb init --catalog-db NAME # Initialize deployment
extenddb stop --config extenddb.toml # Graceful shutdown
extenddb status --config extenddb.toml # Check if running
extenddb healthcheck --config extenddb.toml # Probe /health (exit 0 healthy, 1 not)
Expand Down Expand Up @@ -214,17 +230,22 @@ UpdateTimeToLive, DescribeTimeToLive, TagResource, UntagResource, ListTagsOfReso

```
crates/
core/ — types, validation, expressions (pure sync Rust, no async)
engine/ — operation handlers
storage/ — storage trait definitions
storage-postgres/ — PostgreSQL backend
auth/ — SigV4 verification, IAM policy engine
server/ — HTTP server, management API, web console
bin/ — CLI, config, daemon lifecycle
auth/ — SigV4 verification, IAM policy engine
bin/ — CLI, config, daemon lifecycle
core/ — types, validation, expressions (pure sync Rust, no async)
engine/ — operation handlers
server/ — HTTP server, management API, web console
storage/ — storage trait definitions
storage-cassandra/ — Cassandra backend (experimental)
storage-mongodb/ — MongoDB backend
storage-postgres/ — PostgreSQL backend (default/reference)
storage-sqlite/ — SQLite backend (embedded)


docs/
adr/ — architecture decision records
design/ — architecture and design documents
manuals/ — user-facing guides (PDF pipeline)
adr/ — architecture decision records
```

## Documentation
Expand Down
Loading
Loading