fix(storage-cassandra): qualify migration statements with their keyspace instead of USE by LeeroyHannigan · Pull Request #638 · ExtendDB/extenddb

LeeroyHannigan · 2026-10-29T03:33:04Z

What

The Cassandra catalog and data migration runners no longer send USE <keyspace> on the shared session. They qualify each statement of a migration file with their keyspace at run time instead:

migration_statements splits a file on semicolons and writes the keyspace in front of the table each statement names. It accepts CREATE TABLE [IF NOT EXISTS] t, CREATE INDEX [IF NOT EXISTS] [name] ON t, ALTER TABLE [IF EXISTS] t and INSERT INTO t, after any leading comment lines.
Any other statement form, a table name that is quoted or already names a keyspace, and a fragment between semicolons that holds only a comment fail the whole file before any of its statements runs. The error names the file.
The .cql files are unchanged.

The isolated_engine helper in tests/direct/backup_engine.rs built its catalog the same way, with USE and unqualified statements. It now qualifies them through migration_statements, which is #[doc(hidden)] pub for that reason.

Why

Closes #632

The cdrs_tokio driver applies USE on the connection that ran it and copies it to the session's other connections, on every node, from a background task. That task can finish after the runner sends its next statement. On a session with more than one connection, on one node or across nodes, a migration statement can then run in the keyspace selected before. The migration fails with table schema_history does not exist, or creates its tables in another keyspace.

On main the engine opens one connection per node, the driver default, so a single-node deployment does not hit this. #457 makes the pool size configurable with a default of 10. On a cluster of more than one node the race happens with one connection per node too, but migrations there also fail for a separate reason that this PR does not change: a statement can reach a node before that node has the table an earlier statement created (#637).

Testing done

New tests/direct/keyspace_migrations.rs, run by direct_integration:
- migrations_ignore_the_keyspace_another_caller_selected builds a session with 10 connections per node, sends USE for a decoy keyspace before each runner, and runs 5 catalog and 5 data migrations, each into a new keyspace. It checks that each keyspace holds every table, index and added column its runner creates, none of the other runner's tables, and that the decoy holds no table.
- migrations_run_on_a_session_with_no_keyspace_and_leave_none_selected runs both runners on a session that never selected a keyspace and checks that both schemas are complete and that no keyspace is selected afterwards.
With the test commit alone, on main's runners, both tests fail on every run. Across 5 runs of the first test, 18 of its 50 migrations failed with table schema_history does not exist.
With the fix, the full direct_integration suite passed (151 tests) against a cassandra:4.1 node with password authentication, as the CI job runs it. The two new tests passed in all 6 runs since their last change.
Unit tests cover each accepted statement form and each rejected one, and run every statement of every embedded migration file through migration_statements, checking that the keyspace appears once, after the leading comments, in front of the table. Each of 17 deliberate breakages of the qualifier fails at least one of them.
cargo test --workspace passed (2037 tests, with the database suites skipped). cargo fmt --all -- --check, clippy 1.97.0 with --all-targets -D warnings, and the 1.88.0 build of the crate are clean. Clippy pedantic reports no warning on the branch that main does not already have.

The branch merges without conflicts with #457 and #453. #453 adds comments that describe the USE these runners no longer send, so whichever of the two lands second should reword them.

Checklist

I have read CONTRIBUTING.md
All tests pass (cargo test --workspace)
Code is formatted (cargo fmt --check)
Clippy is clean (cargo clippy -- -W clippy::pedantic)
I have added or updated tests for new functionality
I have updated documentation if behavior changed
Breaking changes are noted below (if any)
If this changes the wire protocol, Storage trait, auth model, on-disk
format, or public CLI surface, an RFC has been accepted or is linked
below. Otherwise, an ADR captures the decision (link below).

ADR / RFC: n/a

By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache License 2.0 and I agree to the Developer Certificate of
Origin (DCO). See CONTRIBUTING.md for details.

…another caller's USE (#632) Add tests/direct/keyspace_migrations.rs and register it in tests/direct_integration.rs. Its tests build their own Cassandra session with 10 connections per node and run the catalog and data migration runners on it. migrations_ignore_the_keyspace_another_caller_selected sends USE for a decoy keyspace before each runner, the way the previous runner on the same session selects its own keyspace. It runs 5 catalog and 5 data migrations, each into a new keyspace, then checks that each migrated keyspace holds every table, index and added column its runner creates and none of the other runner's tables, and that the decoy keyspace holds no table. migrations_run_on_a_session_with_no_keyspace_and_leave_none_selected runs both runners on a session that has never selected a keyspace. It checks that both succeed, that both schemas are complete, and that the session still has no keyspace selected afterwards. The cdrs_tokio driver applies USE on the connection that ran it and copies it to the session's other connections from a background task, so a runner that relies on USE can send a statement to a connection that is still in the keyspace selected before. Against runners that select their keyspace with USE, the first test finds migrations that ran in the decoy keyspace, and the second finds a keyspace left selected on the session.

…ace instead of USE (#632) The catalog and data migration runners sent USE <keyspace> on the shared session and then ran the unqualified statements of each migration file. The cdrs_tokio driver applies USE on the connection that ran it and copies it to the session's other connections, on every node, from a background task that can finish after the next statement is sent. On a session with more than one connection, on one node or across nodes, a statement could run in the keyspace selected before. The migration then failed with "table schema_history does not exist" or created its tables in another keyspace. The runners no longer send USE. migration_statements splits a migration file on semicolons and writes the runner's keyspace in front of the table each statement names: CREATE TABLE [IF NOT EXISTS] t, CREATE INDEX [IF NOT EXISTS] [name] ON t, ALTER TABLE [IF EXISTS] t and INSERT INTO t, after any leading comment lines. Any other statement, a table name that is quoted or already names a keyspace, and a fragment between semicolons that holds only a comment fail the file before any of its statements runs, with an error that names the file. The .cql files are unchanged. Unit tests cover each accepted form and the rejected ones. They also run every statement of every embedded migration file through migration_statements and check that the keyspace is written once, after the leading comments, in front of the table. The isolated_engine helper in tests/direct/backup_engine.rs builds a catalog through V004 statement by statement. It no longer sends USE, and it qualifies each statement through migration_statements.

jcshepherd · 2026-10-29T22:44:23Z

+/// qualified, the file fails before any of its statements runs.
+#[doc(hidden)]
+pub fn migration_statements(keyspace: &str, filename: &str, sql: &str) -> OpResult<Vec<String>> {
+    sql.split(';')


Existing issue, but this still leaves the possibility for unintended behavior with SQL/CQL in comments. E.g.
CREATE TABLE IF NOT EXISTS b (k int PRIMARY KEY) -- was: ; INSERT INTO settings (key, value) VALUES ('catalog_version', '9.9.9') ;

Will result in two statements when only one is intended. It'd be better to strip end-of-line comments, etc. throughout before brute force parsing by splitting on ';'.

jcshepherd · 2026-10-29T22:47:53Z

+/// qualified, the file fails before any of its statements runs.
+#[doc(hidden)]
+pub fn migration_statements(keyspace: &str, filename: &str, sql: &str) -> OpResult<Vec<String>> {
+    sql.split(';')


Similarly, ';' in string literals can cause weird failures. I'm not sure how much this type of issue can really be excluded without basically writing our own SQL/CQL parser (meh) ... at some point it might be worth considering another way of specifying system schemas that is easier to parse.

jcshepherd · 2026-10-29T22:51:38Z

+    }
+    let at = table_name_offset(body).ok_or_else(|| {
+        OpError::Internal(format!(
+            "Migration file {filename} has a statement that names no table the runner can \


Strictly speaking, this may not be true. The issue isn't with the statement per se; it's that the runner doesn't no how to handle the statement. Something like this would be more truthy and point at where the fix is needed:
"uses a statement form the runner does not support: . Supported: CREATE TABLE, CREATE INDEX, ALTER TABLE, INSERT INTO."

jcshepherd

Approving, but with a couple follow-ups suggested in the comments.

Shared traits (crates/storage). `BackupEngine` is replaced by two traits every backend implements: `BackupCatalog`, the catalog index of backups and restore jobs the engine binds (one row per backup addressed by `BackupKey` (account, backup id, ARN), one restore job record per target addressed by `RestoreTargetKey` (account, name, table id), the listings, `start_backup_delete` under the row's lock, `remove_restore_target`, `list_dead_restore_jobs`, `fail_restore_job`, `check_maintenance_claim`, `catalog_id`) with its `JobClaim` (generation, heartbeat, conditional `set_status`, `read`), and `LegacyBackupReader`, the frozen in-catalog format's create, describe, list, delete, restore-source and restore, whose `create` and `restore` now take the job's `JobPermit` and whose `abandoned_creating_backups` reports a dead CREATING row. `BackupRow` derives the API's source-table shapes from `source_table_details_json` (reading only the table id and ARN, so a later build's required member does not break a listing); `BackupStatus` maps DELETING to DELETED and hides DELETED and FAILED. The in-use, not-found and LimitExceeded texts every backend answers are one implementation in `backup_catalog`, as is `JOB_HEARTBEAT_INTERVAL` (15 s), the period every backend's jobs heartbeat at. `StorageError::AlreadyExists` is the claim's refusal; `MigrationId::BackupCatalogIndex = 5` is the migration. `util::require_served_catalog_version` is the rule `serve` applies on a backend whose catalog carries a version string. Config. A `[backup]` section (`deny_unknown_fields`, every key with a default): `data_file_target_bytes`, `heartbeat_interval_secs`, `stale_job_after_secs`, `failed_row_retention_secs`, `reconcile_interval_secs`, `max_concurrent_jobs`, `spool_path`, with `stale_job_after_secs` at least three times the period the jobs heartbeat at (`max(heartbeat_interval_secs, JOB_HEARTBEAT_INTERVAL)`, 45 s at the least), `max_concurrent_jobs` at most 1024, and the floors checked at load; `init` writes the commented block and the sample config carries it. `ServerComponentsOptions.max_concurrent_jobs` lets the PostgreSQL factory raise both pools to `2 * jobs + 10` with a warning. Engine. `BackupRuntime` holds the job slots and the configured store (none on this head); a request over `max_concurrent_jobs` is `LimitExceededException` at once. Describe, List and Delete read the catalog index first and fall back to the legacy reader; List merges `list_account_backup_rows` with the reader's list under the existing paging, filtering and cursor rules, and `TableName` accepts a table ARN as CreateBackup does. CreateBackup runs the request order of the design (ACTIVE probe, ARN fits, vector refusal, permit last, deprecation warning, the legacy create); restore takes the permit last and hands it to the backend. Delete over a row runs the in-use check before the other-store check, as a restore does, and builds the description before the DELETING write; a row whose recorded source table this build cannot read is left out of ListBackups (logged) rather than failing the listing. Both spellings of a backup ARN (as written and re-spelled in this region) parse into `BackupArn` where the store-prefix rules allow; a legacy backup of a name they refuse (dot-only, over 255 bytes) skips the catalog index and describes, restores and deletes through the legacy reader as on main. Reconciler (crates/engine/src/backup/reconciler.rs). A cycle at startup and every `reconcile_interval_secs`, spawned by `serve` beside the backend workers and drained with them: checks the maintenance claim, fails each dead legacy restore (`heartbeat_at` older than `stale_job_after_secs`) and removes its CREATING target, removes the CREATING target of a FAILED record whose removal did not land once the record is that old, with no second FAILED write (the FAILED-target duty of design 4(e), on every backend), logs a dead manifest restore once per record and leaves it to a later release, deletes each abandoned in-catalog CREATING backup through the reader, and records the worker metrics (`WorkerLastSuccess`, `WorkerCycleLatency`, `WorkerErrorCount` under `BackupReconciler`). It replaces the per-backend sweeps: PostgreSQL's control-plane pass, SQLite's startup sweeps, MongoDB's startup and worker sweep, Cassandra's 15 s poller. PostgreSQL. Migration 5 (`0005_backup_catalog_index.sql`, breaking, convergent, one transaction): `backup_index`, the twelve job columns on `table_restores` with the backfill of existing registrations (DONE with `completed_at = creation_date_time`, or RUNNING with the epoch heartbeat and generation 0 where the table is CREATING), `backup_maintenance`, `settings.catalog_id`. The claim is an advisory lock on a dedicated connection (`ClaimSession`) that the backup job's heartbeats and status writes travel through (the legacy restore heartbeats through it and writes DONE and FAILED on pooled connections, each conditioned on RUNNING and the generation); the reconciler's dead rule is the heartbeat alone, the lock being the claim's exclusion, as on the other backends. The legacy restore registers RUNNING with its generation, marks DONE in the transaction that flips the table ACTIVE, and FAILED before it removes a failed target. The restore slot semaphores (per account and per server) and `sweep_abandoned_restores` are removed. `RESTORE_LOCK_SEED` and `abort_restore` are `pub` and `#[doc(hidden)]` so the tests bind them. SQLite. Migration 5 as a code body with the same objects; the twelve columns are nullable (ADD COLUMN on a populated table) and the application writes every one. The claim is an in-process slot (`JobSlots`, a drop guard), since one process serves a file; a restart ages every RUNNING record to the epoch so the first cycle removes an interrupted restore, and a CREATING `backups` row whose slot no task holds is reported dead whatever `stale_after` says (`backups` has no heartbeat column); the reader's `delete` removes such a row whole, with no DELETED tombstone. The restore heartbeats every `JOB_HEARTBEAT_INTERVAL` (15 s) under the engine write lock. Conditional writes never have an unknown outcome here. `backup_server_slot_count` stays until a later release folds it into `max_concurrent_jobs`; its slots are shared by backups and table snapshots (#395), the formula is stated once on `SqliteConfig::pool_size`, and the legacy create tries a slot without waiting, as a snapshot open does, and answers the one `LimitExceededException` text (`limit_exceeded_message`) for both. MongoDB. `backup_index` and `backup_maintenance` collections, a `restore_job` subdocument on a restore target's `tables` document, and the versioned step `002_backup_catalog_index` recorded in `schema_history`, which backfills the registrations, writes the catalog id and moves `catalog_version` to `0.0.5` (one constant, `CATALOG_VERSION`, where three copies of `0.0.2` were). The claim is the row, born owned by an upsert with `heartbeat_at: "$$NOW"`; the lock for DeleteBackup and a restore's registration is a session transaction that increments `lock_seq` on the backup's document, which replaces the `restore_in_progress`/`delete_in_progress` claim markers and the sweep that cleared them. The legacy DeleteBackup writes DELETING before the drop and DELETED after; describe and list hide DELETING. A restore request that outlives the 60 s registration bound answers InternalServerError while its task goes on; the target it leaves is stated in the differences document's restore row. The restore slot semaphores are removed. `serve` refuses a catalog whose version is not `CATALOG_VERSION`, naming `extenddb migrate --yes`. The protocol model (`backup_protocol_model.rs`) is rewritten for the new steps. Cassandra. `V006__backup_catalog_index.cql`, after which the runner writes `0.0.5` (once the backfill step and the file's `schema_history` row are in, so a run that fails in the step leaves 0.0.4 and `migrate --yes` runs the file again): the index columns (`store_kind`, `store_root`, `store_id`, `backup_id`, `owner_generation`, `heartbeat_at`, `completed_at`, `failure_reason`, `source_table_details_json`) on the reused `backups_by_arn`, the status-keyed `backup_index_by_status`, the `restore_*` job columns on `tables`, `backup_maintenance`, and a Rust backfill step keyed on the file. The file's statements are `ALTER TABLE ... ADD IF NOT EXISTS` and `CREATE TABLE IF NOT EXISTS`, which the runner qualifies with the keyspace (#638); the step and the version write name the keyspace in their own statements, and no code or test issues `USE`. An index row is a `backups_by_arn` row with `store_kind` set, and the two readers skip each other's rows. Claims are lightweight transactions `IF owner_generation = ?`; the legacy create heartbeats its CREATING row and polls its claim once per 64-item bucket, and `abandoned_creating_backups` fences a stale one to DELETING under a fresh `owner_generation` before reporting it, scanning DELETING rows first so a row it fenced is reported once. A job stops after three heartbeat errors in a row, as after a heartbeat that does not apply, and touches neither its row nor its data again. An orphan `backup_index_by_status` entry left by a crash between the row's write and the batch is skipped by every reader; the sweep that removes it is `backup sync`'s. `serve` refuses a catalog whose version is not `CATALOG_VERSION`, as on MongoDB. `poll_abandoned_restores` and its worker are removed. The crate gains a `test-hooks` feature (as MongoDB's), compiled only for the `direct_integration` binary, which CI runs with `--features test-hooks`. Migration and versions. One change, one number: migration id 5 on PostgreSQL and SQLite, `0.0.5` on MongoDB and Cassandra. The migration is breaking (stop pre-upgrade servers first); a restore in flight at the stop is a RUNNING record whose owner is dead, failed and removed on the first reconciler pass. Docs. The upgrade manual (migration 5 and its rollback, the Cassandra and MongoDB `0.0.5` section with the Cassandra resume sentence, the serve-time refusal, the removed 15 s sweep line), the admin guide (`[backup]`, the job bound in place of the per-account and half-pool restore slots, the pooler note), the deployment guide (the job bound in place of the restore slots; the connection formulas gain `+ max_concurrent_jobs`), the high-level design and getting-started connection formulas, the limits document's concurrent-restores row, the architecture guide, the MongoDB design document, the differences document's restore rows (the MongoDB registration bound among them), the sample config, `docs/schema/postgres.txt` and `sqlite.txt` regenerated, and the backup/restore design's PR 6 section and section 4 (the migration numbers as implemented, revision log items 26 and 27). Tests. extenddb-storage 196 (13 new), extenddb-config 14 (6 new), extenddb-engine 205 (22 new: catalog describe, list, delete and restore paths, permit order, the ARN rules, reconciler cycle, duties, metrics and worker). extenddb-storage-postgres: lib 69, tests/backup_restore 36 (28 kept, 1 rewritten, 2 removed, 8 new), every other binary green (fresh 5, released 12, golden 5, crash 9, guards 6, runs 5, owners 3, tag_blobs 1, vector_control_plane 42, doc_version_literals 3, table_snapshot 81 with its restore record written as a DONE legacy job, ...). extenddb-storage-sqlite 225 (3 rewritten, 11 new; 1 ignored benchmark). extenddb-storage-mongodb: lib 141 plus 27 live tests against a replica set (21 backup, 6 stream; 9 new or rewritten, 5 removed with the slots and the sweep), interpreter_selftests 18, pushdown_parity 20. extenddb-storage-cassandra: lib 78 (4 new), direct_integration 191 (11 new, 1 rewritten, 1 removed, 1 renamed; three of the new tests build with `--features test-hooks`), ttl_integration 26 passed and the one pre-existing failure below. Wire: tests/test_backup_restore_fidelity.py (37) and tests/test_table_operations.py (27) against a `serve` of this tree on PostgreSQL and on SQLite, 63 passed and 1 skipped each (the DELETING test of #395 skips where a delete finishes in the request), and on Cassandra, on the previous base bffa088, 62 passed and the vector test skipped. `cargo fmt`, `clippy -D warnings` on 1.97, `check --all-targets` on 1.88, `cargo doc` with no new warnings. Not changed here. The backup store (`[backup.store]`, the store identity file) and the store-backed CreateBackup writer, worker and `backup` commands; the manifest restore job; every `backup_index` row is "of another store" for Delete and Restore until a store is configured. The Cassandra `test_ttl_enable_rejects_asynchronous_gsi` failure predates this change (it fails the same way without it, checked on a clean bffa088: the test matches "asynchronously propagated GSIs" and the refusal says "global secondary index"). `heartbeat_interval_secs` reaches no job yet: every in-catalog job heartbeats at `JOB_HEARTBEAT_INTERVAL`, which the stale floor is computed from. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>

Shared traits (crates/storage). `BackupEngine` is replaced by two traits every backend implements: `BackupCatalog`, the catalog index of backups and restore jobs the engine binds (one row per backup addressed by `BackupKey` (account, backup id, ARN), one restore job record per target addressed by `RestoreTargetKey` (account, name, table id), the listings, `start_backup_delete` under the row's lock, `remove_restore_target`, `list_dead_restore_jobs`, `fail_restore_job`, `check_maintenance_claim`, `catalog_id`) with its `JobClaim` (generation, heartbeat, conditional `set_status`, `read`), and `LegacyBackupReader`, the frozen in-catalog format's create, describe, list, delete, restore-source and restore, whose `create` and `restore` now take the job's `JobPermit` and whose `abandoned_creating_backups` reports a dead CREATING row. `BackupRow` derives the API's source-table shapes from `source_table_details_json` (reading only the table id and ARN, so a later build's required member does not break a listing); `BackupStatus` maps DELETING to DELETED and hides DELETED and FAILED. The in-use, not-found and LimitExceeded texts every backend answers are one implementation in `backup_catalog` (`limit_exceeded_message(JobBound)`, the text naming the bound met: `[backup] max_concurrent_jobs`, SQLite's snapshot slots, PostgreSQL's connection slots), as is `JOB_HEARTBEAT_INTERVAL` (15 s), the period every backend's jobs heartbeat at. `StorageError::AlreadyExists` is the claim's refusal; `MigrationId::BackupCatalogIndex = 5` is the migration. `util::require_served_catalog_version` is the rule `serve` applies on a backend whose catalog carries a version string. Config. A `[backup]` section (`deny_unknown_fields`, every key with a default): `data_file_target_bytes`, `heartbeat_interval_secs`, `stale_job_after_secs`, `failed_row_retention_secs`, `reconcile_interval_secs`, `max_concurrent_jobs`, `spool_path`, with `stale_job_after_secs` at least three times the period the jobs heartbeat at (`max(heartbeat_interval_secs, JOB_HEARTBEAT_INTERVAL)`, 45 s at the least), `max_concurrent_jobs` at most 1024, and the floors checked at load; `init` writes the commented block and the sample config carries it. `ServerComponentsOptions.max_concurrent_jobs` lets the PostgreSQL factory raise both pools to `2 * jobs + 10` with a warning. Engine. `BackupRuntime` holds the job slots and the configured store (none on this head); a request over `max_concurrent_jobs` is `LimitExceededException` at once. Describe, List and Delete read the catalog index first and fall back to the legacy reader; List merges `list_account_backup_rows` with the reader's list under the existing paging, filtering and cursor rules, and `TableName` accepts a table ARN as CreateBackup does. CreateBackup runs the request order of the design (ACTIVE probe; the region and the name fit a backup ARN, each refusal naming its cause: `server.region` outside `[a-z0-9-]`, then a name over 255 bytes or made only of dots, checked before the ARN is formed so the limit is named whatever the length; vector refusal; permit last; deprecation warning; the legacy create); restore takes the permit last and hands it to the backend. Delete over a row runs the in-use check before the other-store check, as a restore does, and builds the description before the DELETING write; a row whose recorded source table this build cannot read is left out of ListBackups (logged) rather than failing the listing. Both spellings of a backup ARN (as written and re-spelled in this region) parse into `BackupArn` where the store-prefix rules allow; a legacy backup of a name they refuse (dot-only, over 255 bytes) skips the catalog index and describes, restores and deletes through the legacy reader as on main. Reconciler (crates/engine/src/backup/reconciler.rs). A cycle at startup and every `reconcile_interval_secs`, spawned by `serve` beside the backend workers and drained with them: checks the maintenance claim, fails each dead legacy restore (`heartbeat_at` older than `stale_job_after_secs`) and removes its CREATING target (a target the removal finds no longer CREATING is left to its owner and reported as such), removes the target of a FAILED record whose removal did not land once the record is that old, with no second FAILED write (the FAILED-target duty of design 4(e), on every backend; on Cassandra, where the removal is a claim to DELETING and then a drop, a target left DELETING with its registration is listed on the same rule and its removal resumed from the drop), logs a dead manifest restore once per record and leaves it to a later release, deletes each abandoned in-catalog CREATING backup through the reader, and records the worker metrics (`WorkerLastSuccess`, `WorkerCycleLatency`, `WorkerErrorCount` under `BackupReconciler`). It replaces the per-backend sweeps: PostgreSQL's control-plane pass, SQLite's startup sweeps, MongoDB's startup and worker sweep, Cassandra's 15 s poller. PostgreSQL. Migration 5 (`0005_backup_catalog_index.sql`, breaking, convergent, one transaction): `backup_index` (`backup_arn COLLATE "C"`, so the listings' `(created_at, backup_arn)` order is byte order whatever the database's collation), the twelve job columns on `table_restores` with the backfill of existing registrations (DONE with `completed_at = creation_date_time`, or RUNNING with the epoch heartbeat and generation 0 where the table is CREATING), `backup_maintenance`, `settings.catalog_id`. The claim is an advisory lock on a dedicated connection (`ClaimSession`) that the backup job's heartbeats and status writes travel through (the legacy restore heartbeats through it and writes DONE and FAILED on pooled connections, each conditioned on RUNNING and the generation); the reconciler's dead rule is the heartbeat alone, the lock being the claim's exclusion, as on the other backends. The legacy restore registers RUNNING with its generation, marks DONE in the transaction that flips the table ACTIVE, and FAILED before it removes a failed target. The restore slot semaphores (per account and per server) and `sweep_abandoned_restores` are removed. `RESTORE_LOCK_SEED` and `abort_restore` are `pub` and `#[doc(hidden)]` so the tests bind them. SQLite. Migration 5 as a code body with the same objects; the twelve columns are nullable (ADD COLUMN on a populated table) and the application writes every one. The claim is an in-process slot (`JobSlots`, a drop guard), since one process serves a file; a restart ages every RUNNING record to the epoch so the first cycle removes an interrupted restore, and a CREATING `backups` row whose slot no task holds is reported dead whatever `stale_after` says (`backups` has no heartbeat column); the reader's `delete` removes such a row whole, with no DELETED tombstone. The restore heartbeats every `JOB_HEARTBEAT_INTERVAL` (15 s) under the engine write lock. Conditional writes never have an unknown outcome here. `backup_server_slot_count` stays until a later release folds it into `max_concurrent_jobs`; its slots are shared by backups and table snapshots (#395), the formula is stated once on `SqliteConfig::pool_size`, and the legacy create tries a slot without waiting, as a snapshot open does, and answers the one `LimitExceededException` text (`limit_exceeded_message`), naming the snapshot slots as the bound, for both. MongoDB. `backup_index` and `backup_maintenance` collections, a `restore_job` subdocument on a restore target's `tables` document, and the versioned step `002_backup_catalog_index` recorded in `schema_history`, which backfills the registrations, writes the catalog id and moves `catalog_version` to `0.0.5` (one constant, `CATALOG_VERSION`, where three copies of `0.0.2` were). The claim is the row, born owned by an upsert with `heartbeat_at: "$$NOW"`; the lock for DeleteBackup and a restore's registration is a session transaction that increments `lock_seq` on the backup's document, which replaces the `restore_in_progress`/`delete_in_progress` claim markers and the sweep that cleared them. Both in-use checks read at the transaction read concern (majority-committed data) while DescribeTable reads at `local`, so a RestoreTableFromBackup or DeleteBackup issued within the majority-commit lag after DescribeTable first shows the target ACTIVE can still answer BackupInUseException and succeeds when retried; the differences document's concurrent-restores row states it with the measured lag. The legacy DeleteBackup writes DELETING before the drop and DELETED after; describe and list hide DELETING. A restore request that outlives the 60 s registration bound answers InternalServerError while its task goes on; the target it leaves is stated in the differences document's restore row. The restore slot semaphores are removed. `serve` refuses a catalog whose version is not `CATALOG_VERSION`, naming `extenddb migrate --yes`. The protocol model (`backup_protocol_model.rs`) is rewritten for the new steps. Cassandra. `V006__backup_catalog_index.cql`, after which the runner writes `0.0.5` (once the backfill step and the file's `schema_history` row are in, so a run that fails in the step leaves 0.0.4 and `migrate --yes` runs the file again): the index columns (`store_kind`, `store_root`, `store_id`, `backup_id`, `owner_generation`, `heartbeat_at`, `completed_at`, `failure_reason`, `source_table_details_json`) on the reused `backups_by_arn`, the status-keyed `backup_index_by_status`, the `restore_*` job columns on `tables`, `backup_maintenance`, and a Rust backfill step keyed on the file. The file's statements are `ALTER TABLE ... ADD IF NOT EXISTS` and `CREATE TABLE IF NOT EXISTS`, which the runner qualifies with the keyspace (#638); the step and the version write name the keyspace in their own statements, and no code or test issues `USE`. An index row is a `backups_by_arn` row with `store_kind` set, and the two readers skip each other's rows. Claims are lightweight transactions `IF owner_generation = ?`; the legacy create heartbeats its CREATING row and polls its claim once per 64-item bucket, and `abandoned_creating_backups` fences a stale one to DELETING under a fresh `owner_generation` before reporting it, scanning DELETING rows first so a row it fenced is reported once. A job stops after three heartbeat errors in a row, as after a heartbeat that does not apply, and touches neither its row nor its data again. An orphan `backup_index_by_status` entry left by a crash between the row's write and the batch is skipped by every reader; the sweep that removes it is `backup sync`'s. `serve` refuses a catalog whose version is not `CATALOG_VERSION`, as on MongoDB. `poll_abandoned_restores` and its worker are removed. The crate gains a `test-hooks` feature (as MongoDB's), compiled only for the `direct_integration` binary, which CI runs with `--features test-hooks`. Migration and versions. One change, one number: migration id 5 on PostgreSQL and SQLite, `0.0.5` on MongoDB and Cassandra. The migration is breaking (stop pre-upgrade servers first); a restore in flight at the stop is a RUNNING record whose owner is dead, failed and removed on the first reconciler pass. Docs. The upgrade manual (migration 5 and its rollback, the Cassandra and MongoDB `0.0.5` section with the Cassandra resume sentence, the serve-time refusal, the removed 15 s sweep line), the admin guide (`[backup]`, the job bound in place of the per-account and half-pool restore slots, the pooler note), the deployment guide (the job bound in place of the restore slots; the connection formulas gain `+ max_concurrent_jobs`), the high-level design and getting-started connection formulas, the limits document's concurrent-restores row, the architecture guide, the MongoDB design document, the differences document's restore rows (the MongoDB registration bound among them), the admin guide's `region` row, the differences document's `BackupArn` and backup creation rows, the limits document's `BackupName`/`TableName` row and the upgrade manual's `server.region` paragraph (CreateBackup refused under a region outside `[a-z0-9-]`; a name no backup ARN can carry refused with the texts), `CONTRIBUTING.md`'s migration example renumbered to 6, the sample config, `docs/schema/postgres.txt` (regenerated for the collation) and `sqlite.txt` regenerated, and the backup/restore design's PR 6 section and section 4 (the migration numbers as implemented, revision log items 26 and 27). Tests. extenddb-storage 195 (13 new; 196 beside a crate that enables its `test-hooks` feature, as `cargo test --workspace` does), extenddb-config 14 (6 new), extenddb-engine 207 (24 new: catalog describe, list, delete and restore paths, permit order, the ARN rules, reconciler cycle, duties, metrics and worker). extenddb-storage-postgres: lib 69, tests/backup_restore 37 (28 kept, 1 rewritten, 2 removed, 9 new), every other binary green (fresh 5, released 12, golden 5, crash 9, guards 6, runs 5, owners 3, tag_blobs 1, vector_control_plane 42, doc_version_literals 3, table_snapshot 81 with its restore record written as a DONE legacy job, ...). extenddb-storage-sqlite 225 (3 rewritten, 11 new; 1 ignored benchmark). extenddb-storage-mongodb: lib 141 plus 28 live tests against a replica set (22 backup, 6 stream; 10 new or rewritten, 5 removed with the slots and the sweep), interpreter_selftests 18, pushdown_parity 20. extenddb-storage-cassandra: lib 78 (4 new), direct_integration 192 (12 new, 1 rewritten, 1 removed, 1 renamed; three of the new tests build with `--features test-hooks`), ttl_integration 26 passed and the one pre-existing failure below. Wire: tests/test_backup_restore_fidelity.py (37) and tests/test_table_operations.py (27) against a `serve` of this tree: 64 passed on PostgreSQL with `EXTENDDB_TEST_BACKEND=postgres EXTENDDB_CONFIG=<config>` (the DELETING test reads the catalog), 63 and 1 skipped without them; on SQLite 63 and the DELETING test skipped without them (with them it fails on the host's sqlite3 3.7.17 module, which cannot read a partial index); on Cassandra 62 passed and 2 skipped (the vector test and the DELETING test). `cargo fmt`, `clippy -D warnings` on 1.97, `check --all-targets` on 1.88, `cargo doc` with no new warnings. Not changed here. The backup store (`[backup.store]`, the store identity file) and the store-backed CreateBackup writer, worker and `backup` commands; the manifest restore job; every `backup_index` row is "of another store" for Delete and Restore until a store is configured. The Cassandra `test_ttl_enable_rejects_asynchronous_gsi` failure predates this change (it fails the same way without it, checked on a clean bffa088: the test matches "asynchronously propagated GSIs" and the refusal says "global secondary index"). `heartbeat_interval_secs` reaches no job yet: every in-catalog job heartbeats at `JOB_HEARTBEAT_INTERVAL`, which the stale floor is computed from. Co-authored-by: Lee <leeroyhannigan@yahoo.ie> Signed-off-by: Scott Robinson <robinnsc@amazon.com>

LeeroyHannigan added 2 commits October 29, 2026 02:38

LeeroyHannigan requested review from amrith, c33howard, jcshepherd, pdf-amzn, robinnsc and yesyayen as code owners October 29, 2026 03:33

LeeroyHannigan added the v1 Needed for the 1.0 release label Oct 29, 2026

jcshepherd reviewed Oct 29, 2026

View reviewed changes

jcshepherd approved these changes Oct 29, 2026

View reviewed changes

LeeroyHannigan added this pull request to the merge queue Oct 30, 2026

This was referenced Oct 30, 2026

storage-cassandra: the migration runner splits a file at a ; inside a comment or a string literal #675

Open

storage-cassandra: the migration runner reports an UPDATE it doesn't handle as a statement that names no table #676

Open

Merged via the queue into main with commit c5fe781 Oct 30, 2026
34 checks passed

LeeroyHannigan deleted the fix/632-migrations-keyspace branch October 30, 2026 13:27

This was referenced Oct 30, 2026

feat(backup): catalog index, BackupCatalog, JobClaim, legacy reader [backup/restore 6] #671

Draft

feat(backup): [backup.store], the store identity file, backup purge [backup/restore 7] #672

Draft

Navigation Menu

Repository navigation

fix(storage-cassandra): qualify migration statements with their keyspace instead of USE - #638

fix(storage-cassandra): qualify migration statements with their keyspace instead of USE#638
LeeroyHannigan merged 2 commits into
mainfrom
fix/632-migrations-keyspace

LeeroyHannigan commented Oct 29, 2026

Uh oh!

jcshepherd Oct 29, 2026

Uh oh!

jcshepherd Oct 29, 2026

Uh oh!

jcshepherd Oct 29, 2026

Uh oh!

jcshepherd left a comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

LeeroyHannigan commented Oct 29, 2026

What

Why

Testing done

Checklist

Uh oh!

jcshepherd Oct 29, 2026

Choose a reason for hiding this comment

Uh oh!

jcshepherd Oct 29, 2026

Choose a reason for hiding this comment

Uh oh!

jcshepherd Oct 29, 2026

Choose a reason for hiding this comment

Uh oh!

jcshepherd left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants