Repository navigation
docs: document the experimental Cassandra storage backend - #412
jcshepherd wants to merge 4 commits into
Conversation
LeeroyHannigan
left a comment
LeeroyHannigan
left a comment
There was a problem hiding this comment.
I built this branch (5a4a0a6) with --features cassandra and followed the setup guide against cassandra:4.1 (4.1.12) in Docker.
The guide doesn't get a reader to a running server. init fails on a node without authentication, and on an authenticated node serve can't connect with the config init writes. The causes are in init and the Cassandra connection code, and those fixes don't belong in this PR. The six inline comments ask the docs to describe what the backend does today. Once they're addressed, this is good to merge from my side.
Not measured: a multi-node cluster and Cassandra 5.x. The replication factor note is from the code.
|
|
||
| The node now listens on `127.0.0.1:9042` with password authentication enabled. | ||
|
|
||
| > **Note:** Password authentication is optional for a purely local node. If you |
There was a problem hiding this comment.
This path fails. On a stock cassandra:4.1 node, extenddb init --backend cassandra stops at the application role:
Error: Internal("Create user: Server 127.0.0.1:9042 error: ErrorBody { message: \"org.apache.cassandra.auth.CassandraRoleManager doesn't support PASSWORD\", ty: Invalid }")
init always runs CREATE ROLE ... WITH PASSWORD (bootstrapper.rs:288), and Cassandra accepts that only with PasswordAuthenticator on. This note, the no-flags example at line 111 and getting-started.md lines 178 to 184 should say authentication is required.
| ./target/release/extenddb init --backend cassandra --config extenddb.toml | ||
| ``` | ||
|
|
||
| init creates: |
There was a problem hiding this comment.
After steps 1 to 4 and the init command above (I mapped the node to host port 19042), serve doesn't start:
extenddb fatal: Failed to connect to cassandra: Connection error: Connection timeout after 10s. Check that Cassandra is running and accessible at ["127.0.0.1:19042"]. If authentication is enabled, verify username/password are correct.
init creates the role extenddb with password extenddb-local-dev (the --extenddb-user and --extenddb-pass defaults), then writes username and password commented out. With those two fields set, serve started and /health returned healthy, so the guide needs that step and the two flags.
With CassandraAuthorizer on, serve then exits with "User extenddb has no SELECT permission on
or any of its parents", because init grants the role nothing. The guide should list the grants the role needs, or say only AllowAllAuthorizer works for now.| password = "<application-password>" | ||
| keyspace_prefix = "extenddb" | ||
| replication_factor = 3 | ||
| datacenter = "dc1" |
There was a problem hiding this comment.
init can't create keyspaces on this cluster. The bootstrapper always uses datacenter1 (bootstrapper.rs:229) and init has no flag to change it. On a node started with CASSANDRA_DC=dc1, init failed with:
Unrecognized strategy option {datacenter1} passed to NetworkTopologyStrategy for keyspace extenddb_catalog
The guide should say the datacenter has to be named datacenter1 until that's fixed. From the code, init also writes replication_factor = 1 into the new config whatever factor it used (bootstrapper.rs:693), so accounts created later get factor 1 unless the reader edits the config after init.
| requires authentication to create the keyspaces and the application role. | ||
| - `--keyspace-prefix myapp` — a different keyspace prefix, so one cluster can | ||
| host more than one deployment. | ||
| - `--replication-factor 3` — a higher replication factor for a multi-node |
There was a problem hiding this comment.
init rejects this flag:
error: unexpected argument '--replication-factor' found
The bootstrapper reads it, but InitArgs in cmd_init.rs never declares it, which is the bug #267 fixed for --sqlite-path. I'd drop this bullet until the flag is declared.
| Use `sslmode=require` (or `verify-full` with a CA certificate) for encrypted database connections. | ||
|
|
||
| ### Managed PostgreSQL | ||
| Point ExtendDB at the remote backend at init time, and use an encrypted |
There was a problem hiding this comment.
The Cassandra backend can't encrypt this connection. engine.rs builds the session with NodeTcpConfigBuilder and TcpSessionBuilder, and cdrs-tokio is built without its TLS feature, so the password and item data cross the network in clear text. cassandra-setup.md doesn't mention encryption, so this paragraph and the checklist item at line 197 should say Cassandra traffic is unencrypted today and the cluster needs a trusted network.
| ## DynamoDB Streams | ||
|
|
||
| extenddb implements DynamoDB Streams for change data capture. Stream records are captured atomically with data writes inside the same PostgreSQL transaction. Both the DynamoDB API and Streams API are served on the same port. | ||
| extenddb implements DynamoDB Streams for change data capture. Stream records are captured atomically with the data writes that produce them. Both the DynamoDB API and Streams API are served on the same port. |
There was a problem hiding this comment.
On Cassandra this isn't true for conditional writes. put_item_if_not_exists in put_get_item.rs applies the lightweight transaction, then writes the stream record in a separate logged batch (lines 392 to 522), so a failure between the two leaves the item written with no stream record. The old sentence was scoped to PostgreSQL. I'd scope the sentence to the backends where it's true and state the Cassandra exception.
…ndDB/extenddb into docs/cassandra-storage-docs
LeeroyHannigan
left a comment
LeeroyHannigan
left a comment
There was a problem hiding this comment.
I re-checked the six comments from 22 October against a549428 and read the changed docs against origin/main. Four are addressed and correct: the PasswordAuthenticator requirement, the AllowAllAuthorizer caveat, the datacenter1 and replication_factor notes, and the unencrypted connection. Two need another pass. The credentials text is now stale: #384 changed init to write username and password live, so my earlier comment on that point predates the change. The Streams sentence states the Cassandra exception too narrowly and leaves MongoDB out. Three optional points I missed in the first round are marked as such and are not merge conditions. Once the two are fixed I will approve. Not run: init or serve against a live node.
|
|
||
| ## Config Mapping | ||
|
|
||
| init writes a `[storage.cassandra]` section to `extenddb.toml`. It writes the |
There was a problem hiding this comment.
Major: init no longer comments the credentials out. #384 (9d13202, merged into this branch in e51b789) changed backend_config_section in crates/storage-cassandra/src/bootstrapper.rs to write username and password live from the application role it created, and backend_config_section_carries_the_application_role pins that output. Update this sentence and the sample at line 144, and drop the block at line 152, which tells the reader to set two fields the file already holds.
| Cassandra admin credentials with `--cassandra-user` / `--cassandra-pass`; `init` | ||
| uses them to create the keyspaces and the application role. `init` writes a | ||
| `[storage.cassandra]` section to the config, with the application credentials | ||
| commented out: |
There was a problem hiding this comment.
Major: same point as cassandra-setup.md line 135. Since #384 the generated section carries username and password, so the sample at line 205 and the paragraph at line 213 describe a config the reader will not have.
| ## DynamoDB Streams | ||
|
|
||
| extenddb implements DynamoDB Streams for change data capture. Stream records are captured atomically with data writes inside the same PostgreSQL transaction. Both the DynamoDB API and Streams API are served on the same port. | ||
| extenddb implements DynamoDB Streams for change data capture. On the PostgreSQL and SQLite backends, stream records are captured atomically with the data writes that produce them, inside the same transaction. On the Cassandra backend this holds for ordinary writes but not for conditional writes: a conditional put applies the item with a lightweight transaction and then writes the stream record in a separate logged batch, so a failure between the two can leave the item written with no stream record. Both the DynamoDB API and Streams API are served on the same port. |
There was a problem hiding this comment.
Major: on Cassandra the separation covers ordinary writes, not only conditional ones. occ_write in crates/storage-cassandra/src/data/update_item.rs runs the data write as its own lightweight transaction and adds the stream record to a following logged batch, because Cassandra cannot mix an LWT with non-LWT statements in one batch; the base write shares that batch only when a TTL claim is held. MongoDB writes the record in the data write's session (write_stream_inline_in_session), so name it alongside PostgreSQL and SQLite in the first sentence.
| ## Local Single-Node Setup (Docker) | ||
|
|
||
| These steps start a single-node Cassandra 4.1 container for local development. | ||
| They mirror `devtools/cassandra-node`, which the project uses for its own tests. |
There was a problem hiding this comment.
Minor, optional, and my miss in the first round: devtools/cassandra-node is not in the repository. The steps match devtools/run-cassandra-tests, whose header documents the same docker run and authenticator edit, and .github/workflows/integration-cassandra.yml runs the same sequence in CI.
| ## Performance Tuning (PostgreSQL) | ||
|
|
||
| ## Performance Tuning | ||
| This section covers the PostgreSQL backend. For Cassandra tuning (consistency, |
There was a problem hiding this comment.
Minor, optional, and my miss in the first round: cassandra-setup.md at this head has no tuning section and no mention of token-aware routing or LWT contention. Point at its Known Limitations section instead, or drop the parenthetical.
| - Streams and TTL run from background workers, with a durable TTL expiration queue (see ADR-0010, ADR-0018) | ||
|
|
||
| > **Note:** Cassandra has no multi-row ACID primitive. Some multi-entity operations trade all-or-nothing atomicity for | ||
| > the integrity that LWT provides (see ADR-0002, ADR-0021). |
There was a problem hiding this comment.
Minor, optional, and my miss in the first round: ADR-0002 resolves to docs/adr/0002-sql-injection-defense.md. The transaction and atomicity document is docs/adr/0012, whose own heading reads ADR-0002, and the other four references in this section use file numbers.
What
Updates existing documentation to include the Apache Cassandra backend (marked as experimental throughout), adding one new document on Cassandra set-up.
I did make a gentle attempt to make the existing documentation less Postgres-centric where appropriate, primarily by eliding Postgres references when it didn't change the meaning of the section. There is still more work to be done here, as well as some technical issues to fix (e.g., default Postgres database names), but those should be handled in separate changes.
This is not intended to be a broad documentation rewrite.
Why
Bring user-facing documentation for the Cassandra backend up to par with the other backends.
Testing done
Checklist
cargo test --workspace)cargo fmt --check)cargo clippy -- -W clippy::pedantic)Storagetrait, auth model, on-diskformat, or public CLI surface, an RFC has been accepted or is linked
below. Otherwise, an ADR captures the decision (link below).
By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache License 2.0 and I agree to the Developer Certificate of
Origin (DCO). See CONTRIBUTING.md for details.