Repository navigation
fix(backup): restore sort keys, indexes, and throughput; bound memory; recover abandoned restores - #384
fix(backup): restore sort keys, indexes, and throughput; bound memory; recover abandoned restores#384robinnsc wants to merge 1 commit into
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
robinnsc
changed the title
Nothing stopped two `extenddb serve` processes from opening the same SQLite file. The backend serializes writers with a lock inside the server process, so two servers write concurrently and can hit SQLITE_BUSY mid-transaction; and startup recovery (control-plane transitions, GSI and vector index rebuilds, and, with #384, abandoned restores) assumes no other server is running, so a second server starting up would undo the first one's work in progress. Nothing stopped `destroy` from unlinking the file under a running server either, which left that server serving an unlinked inode while a later `init` created a fresh database at the same path. `extenddb serve` on a file database now takes an exclusive flock(2) on `<database>.lock` before opening the database and holds it for the life of the process (the lock lives in the engine, so it is held while any clone is). `init` and `migrate` take the same lock for their duration through the bootstrapper's migration-lock hook, and `destroy` takes it before it removes the file; each refuses, rather than waits, when a server holds it. Read-only commands (`settings`, `manage`, `verify`, `status`) take no lock. A second holder fails with: another extenddb process is already using <db> (lock held on <db>.lock); stop it first, or point this command at a different database The lock file is named after the file SQLite actually opens, not the configured string: the location is parsed with sqlx's own SqliteConnectOptions (so `sqlite:` URL forms, percent-encoding such as `a%20b.sqlite`, and `file:` URIs resolve as the engine resolves them), and the path is canonicalized, so two spellings of one file, a `..` path, or a symlink and its target all share one lock. A disk file whose name happens to contain `mode=memory` is a file. In-memory databases take no lock. The kernel releases the lock when the process exits, including on kill -9, so there is no stale-lock cleanup. On non-Unix platforms no lock is taken and a warning is logged. flock is advisory and its behaviour on network filesystems depends on the server and mount; the docs say to keep SQLite databases on local disk. The directory holding the database must be writable so the lock file can be created. flock through libc rather than std's File::try_lock, which needs Rust 1.89; the workspace MSRV is 1.88. libc is already a dependency of the workspace and listed in every license notices file. Docs: troubleshooting entry for the error with its scope; README and the deployment and design guides now state that one instance per catalog is the supported deployment, that SQLite enforces it, and that PostgreSQL and MongoDB do not (per-instance caches without cross-instance invalidation, every worker on every instance, a liveness-only /health). The deployment guide previously said multiple PostgreSQL instances were consistent because there was no in-process cache, which was not true. Tests: database_file resolution for plain paths, sqlite: URLs, percent-encoding, file: URIs, in-memory forms, and a disk file named like a memory parameter; ServeLock refused on the same file, independent across files, shared across `..` spellings and across file and directory symlinks; the server factory refusing a second server on one file; destroy and migrate refused while a server holds the file and a server refused while migrate holds it. Signed-off-by: Scott Robinson <robinnsc@amazon.com>
13c7f24 to
dc980eb
Compare
There was a problem hiding this comment.
Blocking on two things, both inline:
- the Postgres CreateBackup barrier doesn't pin the snapshot the way the code and docs say it does, and
- SQLite CreateBackup now holds the write lock for the whole read, which is a server-wide write stall on a big table. The rest are non-blocking.
| .execute(&mut *snapshot) | ||
| .await | ||
| .map_err(db_err)?; | ||
| meta.commit().await.map_err(db_err)?; |
There was a problem hiding this comment.
Blocking. This barrier doesn't hold. LOCK TABLE is a utility statement and doesn't take a snapshot in Postgres. The REPEATABLE READ snapshot is taken by the first SELECT, which here runs after meta.commit() has already released the FOR SHARE row. So the definition is captured at one instant and the items at a later one, with UpdateTable free to commit in between. I checked this on PG 15: BEGIN ... REPEATABLE READ; LOCK TABLE t ...; then an insert from another session, then SELECT count(*) sees the new row.
There was a problem hiding this comment.
Yeah good catch, swapped that to BEGIN RR RO; LOCK TABLE … ACCESS SHARE; SELECT 1 FROM <data> LIMIT 1; and only then releases the FOR SHARE row
| // instant, including against a concurrent UpdateTable. Items are | ||
| // read in rowid batches on the same connection, so memory is | ||
| // bounded by the batch size. | ||
| let _writer = self.write_lock.lock().await; |
There was a problem hiding this comment.
Blocking. This now holds the engine write lock for the whole read and write of the table. On main the item read ran before the lock, only the inserts were under it. The pool is WAL mode, so a deferred read transaction on a second connection already gives a stable snapshot without blocking writers.
There was a problem hiding this comment.
FIxed, the lock is now held only to read the definition, insert the backup row as CREATING, and open the deferred read transaction (snapshot fixed under the lock, so definition and items agree), items stream unlocked and are written in batches under short holds
| "SSESpecificationOverride", | ||
| ]; | ||
|
|
||
| fn validate_restore_table_overrides(body: &Value) -> Result<(), DynamoDbError> { |
There was a problem hiding this comment.
Now that the definition is in hand, BillingModeOverride and ProvisionedThroughputOverride are a few lines on CreateTableInput. Refusing them is a wire regression for clients that pass them (the console does). I'd implement at least those two and refuse only the index and SSE overrides.
| | `RestoreTableFromBackup` overrides | `BillingModeOverride`, `GlobalSecondaryIndexOverride`, `LocalSecondaryIndexOverride`, `ProvisionedThroughputOverride`, `SSESpecificationOverride` apply to the restored table | Not supported; a request carrying any of them returns `ValidationException` rather than restoring a table that differs from what was asked for | | ||
| | Multi-part base table keys (preview) | Not supported | A backup of a table created with `enable_multipart_keys` is refused at restore with `ValidationException`; the item paths address such tables by their first HASH and RANGE attribute only | | ||
| | `DeleteBackup` during a restore from that backup | `BackupInUseException` | Matches: `BackupInUseException` (HTTP 400) until the restore completes or fails | | ||
| | `DeleteTable` on a table being restored | `ResourceInUseException` | Matches on a restore target. An ordinary table in `CREATING` can still be deleted during its control-plane delay, where the service refuses | |
There was a problem hiding this comment.
This says "Matches on a restore target" without a backend qualifier, but the MongoDB diff only adds RestoreSummary and the DeleteBackup refusal. DeleteTable during a Mongo restore is not refused, and Mongo has no abandoned-restore sweep. Either qualify these rows or add the behaviour.
There was a problem hiding this comment.
Qualified every row by backend, and implemented the Mongo DeleteTable, and clarified that Mongo still has no abandoned restore sweep
| ## Version History | ||
|
|
||
| ### Catalog 0.0.3 (Current) | ||
| ### Catalog 0.0.4 (Current) |
There was a problem hiding this comment.
Worth a sentence here: any Postgres target the old bug left in CREATING matches the sweep predicate and gets removed on the first control-plane pass after upgrade. Right outcome, but operators will see tables disappear from ListTables and should know why.
| } | ||
| let rows_sql = format!( | ||
| "SELECT rowid, item_data FROM {table} \ | ||
| WHERE {cond} AND rowid > ? AND rowid <= ? ORDER BY rowid" |
There was a problem hiding this comment.
The restore cursor is the implicit rowid of backup_items, read across separately committed batches. backup_items has no INTEGER PRIMARY KEY, so a VACUUM between batches can renumber rowids and the restore skips or duplicates rows. Nothing in-tree runs VACUUM and #391 keeps a second extenddb out, but not a sqlite3 CLI. Give backup_items an INTEGER PRIMARY KEY or cursor on something stable.
There was a problem hiding this comment.
Done, backup_items has id INTEGER PRIMARY KEY and the restore cursors on it. An existing table is rebuilt in place by migrate, and the fixture test now checks id is present with rows in order.
| // This session sits idle for the whole copy. A server-side | ||
| // idle_session_timeout would end it and release the lock while the | ||
| // restore is still running, so it is disabled for this session only. | ||
| sqlx::query("SET idle_session_timeout = 0") |
There was a problem hiding this comment.
If this SET fails (managed instance that rejects it), every restore becomes a 500. Make it best-effort with a warning.
There was a problem hiding this comment.
Done, now best-effort with a warning
| // mid-copy leaves its target CREATING with no scheduled transition, | ||
| // which nothing above would ever move on. Logged and skipped on | ||
| // failure, so it cannot hold up the transitions above. | ||
| match self.sweep_abandoned_restores().await { |
There was a problem hiding this comment.
This pass is notify-driven with a 60 s idle timeout, so an abandoned target is removed between 60 and ~120 s, not "after 60 s". Each pass also opens a fresh out-of-pool connection per still-locked candidate. Fine, but say so in the comment.
There was a problem hiding this comment.
Corrected the comment
| .map_err(|e| StorageError::Internal(format!("Database error: {e}")))?; | ||
| .map_err(db_err)?; | ||
| sqlx::query(&format!( | ||
| "LOCK TABLE {} IN ACCESS SHARE MODE", |
There was a problem hiding this comment.
This lock is now held across the read and the interleaved catalog writes. A DeleteTable issued during the backup reaches the control-plane worker, whose DROP TABLE blocks behind it and stalls the whole worker pass until the backup finishes. Main had the same shape for the read only. Either SET LOCAL lock_timeout in the worker's drop or document it.
There was a problem hiding this comment.
Went with using SET LOCAL lock_timeout
| .unwrap_or(i64::MAX); | ||
| sqlx::query( | ||
| "INSERT INTO backup_items (backup_arn, pk, sk, item_data) \ | ||
| VALUES (?, '', NULL, ?)", |
There was a problem hiding this comment.
SQLite stores pk = '' while Postgres stores the real pk. Harmless today but the #348 migration-from-rows will want it. Also COPY_BATCH_ITEMS is imported but this file uses its own COPY_BATCH.
There was a problem hiding this comment.
Swapped to storing the Real pk
Nothing stopped two `extenddb serve` processes from opening the same SQLite file. The backend serializes writers with a lock inside the server process, so two servers write concurrently and can hit SQLITE_BUSY mid-transaction; and startup recovery (control-plane transitions, GSI and vector index rebuilds, and, with #384, abandoned restores) assumes no other server is running, so a second server starting up would undo the first one's work in progress. Nothing stopped `destroy` from unlinking the file under a running server either, which left that server serving an unlinked inode while a later `init` created a fresh database at the same path. `extenddb serve` on a file database now takes an exclusive flock(2) on `<database>.lock` before opening the database and holds it for the life of the process (the lock lives in the engine, so it is held while any clone is). `init` and `migrate` take the same lock for their duration through the bootstrapper's migration-lock hook, and `destroy` takes it before it removes the file; each refuses, rather than waits, when a server holds it. `init` holds it through the encryption key, default account, and admin user, not only the schema, so a server cannot start in the window where the schema exists but the key does not. Read-only commands (`settings`, `manage`, `verify`, `status`) take no lock. A second holder fails with: another extenddb process is already using <db> (lock held on <db>.lock); stop it first, or point this command at a different database The lock file is named after the file SQLite actually opens, not the configured string: the location is parsed with sqlx's own SqliteConnectOptions (so `sqlite:` URL forms, percent-encoding such as `a%20b.sqlite`, and `file:` URIs resolve as the engine resolves them), and the path is canonicalized, so two spellings of one file, a `..` path, or a symlink and its target all share one lock. A dangling symlink (the target not created yet) is followed by hand, since canonicalize refuses it. A disk file whose name happens to contain `mode=memory` is a file. In-memory databases take no lock. The lock file is created 0600 and an existing one is tightened to 0600 on open, matching the database and its sidecars: flock needs only a read handle, so a world-readable lock file would let any local user hold the lock and keep the server from starting. The kernel releases the lock when the process exits, including on kill -9, so there is no stale-lock cleanup. On non-Unix platforms no lock is taken and a warning is logged. flock is advisory and its behaviour on network filesystems depends on the server and mount; the docs say to keep SQLite databases on local disk. The directory holding the database must be writable so the lock file can be created. flock through libc rather than std's File::try_lock, which needs Rust 1.89; the workspace MSRV is 1.88. libc is already a dependency of the workspace and listed in every license notices file. Docs: troubleshooting entry for the error with its scope; README and the deployment and design guides now state that one instance per catalog is the supported deployment, that SQLite enforces it, and that PostgreSQL and MongoDB do not (per-instance caches without cross-instance invalidation, every worker on every instance, a liveness-only /health). The deployment guide previously said multiple PostgreSQL instances were consistent because there was no in-process cache, which was not true. Tests: database_file resolution for plain paths, sqlite: URLs, percent-encoding, file: URIs, in-memory forms, and a disk file named like a memory parameter; ServeLock refused on the same file, independent across files, shared across `..` spellings, across file and directory symlinks, and across a dangling symlink and its future target (one and two links); the lock file created 0600 and an existing 0644 one tightened; the server factory refusing a second server on one file; destroy and migrate refused while a server holds the file and a server refused while migrate holds it. Signed-off-by: Scott Robinson <robinnsc@amazon.com>
Nothing stopped two `extenddb serve` processes from opening the same SQLite file. The backend serializes writers with a lock inside the server process, so two servers write concurrently and can hit SQLITE_BUSY mid-transaction; and startup recovery (control-plane transitions, GSI and vector index rebuilds, and, with ExtendDB#384, abandoned restores) assumes no other server is running, so a second server starting up would undo the first one's work in progress. Nothing stopped `destroy` from unlinking the file under a running server either, which left that server serving an unlinked inode while a later `init` created a fresh database at the same path. `extenddb serve` on a file database now takes an exclusive flock(2) on `<database>.lock` before opening the database and holds it for the life of the process (the lock lives in the engine, so it is held while any clone is). `init` and `migrate` take the same lock for their duration through the bootstrapper's migration-lock hook, and `destroy` takes it before it removes the file; each refuses, rather than waits, when a server holds it. `init` holds it through the encryption key, default account, and admin user, not only the schema, so a server cannot start in the window where the schema exists but the key does not. Read-only commands (`settings`, `manage`, `verify`, `status`) take no lock. A second holder fails with: another extenddb process is already using <db> (lock held on <db>.lock); stop it first, or point this command at a different database The lock file is named after the file SQLite actually opens, not the configured string: the location is parsed with sqlx's own SqliteConnectOptions (so `sqlite:` URL forms, percent-encoding such as `a%20b.sqlite`, and `file:` URIs resolve as the engine resolves them), and the path is canonicalized, so two spellings of one file, a `..` path, or a symlink and its target all share one lock. A dangling symlink (the target not created yet) is followed by hand, since canonicalize refuses it. A disk file whose name happens to contain `mode=memory` is a file. In-memory databases take no lock. The lock file is created 0600 and an existing one is tightened to 0600 on open, matching the database and its sidecars: flock needs only a read handle, so a world-readable lock file would let any local user hold the lock and keep the server from starting. The kernel releases the lock when the process exits, including on kill -9, so there is no stale-lock cleanup. On non-Unix platforms no lock is taken and a warning is logged. flock is advisory and its behaviour on network filesystems depends on the server and mount; the docs say to keep SQLite databases on local disk. The directory holding the database must be writable so the lock file can be created. flock through libc rather than std's File::try_lock, which needs Rust 1.89; the workspace MSRV is 1.88. libc is already a dependency of the workspace and listed in every license notices file. Docs: troubleshooting entry for the error with its scope; README and the deployment and design guides now state that one instance per catalog is the supported deployment, that SQLite enforces it, and that PostgreSQL and MongoDB do not (per-instance caches without cross-instance invalidation, every worker on every instance, a liveness-only /health). The deployment guide previously said multiple PostgreSQL instances were consistent because there was no in-process cache, which was not true. Tests: database_file resolution for plain paths, sqlite: URLs, percent-encoding, file: URIs, in-memory forms, and a disk file named like a memory parameter; ServeLock refused on the same file, independent across files, shared across `..` spellings, across file and directory symlinks, and across a dangling symlink and its future target (one and two links); the lock file created 0600 and an existing 0644 one tightened; the server factory refusing a second server on one file; destroy and migrate refused while a server holds the file and a server refused while migrate holds it. Signed-off-by: Scott Robinson <robinnsc@amazon.com>
dc980eb to
8f83169
Compare
8f83169 to
ba9f869
Compare
|
#395 (SnapshotEngine) is branched from this PR and depends on it; its diff shows this PR's commit until this merges. |
…t code Revision of docs/design/15-backup-restore.md after reviewing the draft against main 9f43d03 and PRs #384, #335, #336, #337, measuring the read, compression, and bulk-load paths on PostgreSQL 16 and SQLite with a 1M-item fixture, and two review rounds against the companion PRs #393, #394 and #395. The revision log at the end of the document lists each change; the ones that alter behavior: - SnapshotEngine returns the table definition together with the items from one fenced operation (TableSnapshot). The items-only signature in the draft allowed a GSI change or DeleteTable between the definition read and the data snapshot. Indexes are reported in whatever build state they had and only an index being deleted is dropped; an ACTIVE-only filter would have left a GSI mid-backfill out of the backup for good. - The snapshot is opened in the CreateBackup request so the backup represents the request time, as DynamoDB documents; backup_creation_date_time is the instant the snapshot pinned. - Per-job claims with fencing replace the single-instance lease the draft assumed; PostgreSQL and MongoDB have no such lease. The claim is an advisory lock on a dedicated catalog connection on PostgreSQL (the seed #384's restore lock already uses), the row itself on MongoDB, an in-process slot on SQLite; no row ever exists unowned. Heartbeats are a timer, staleness is measured on the database's clock, and a failed heartbeat aborts the job. The reconciler resolves a dead CREATING row at once under a fresh generation conditioned on status = CREATING and deletes its prefix only after the stale window. - DeleteBackup tombstones the row as DELETING, deletes the manifest first, and marks DELETED only when the prefix is empty; backup sync is scoped to the configured store and never adopts a CREATING, DELETING, DELETED or FAILED prefix. Failed creations and deleted backups are BackupNotFound, as on the service, and share one retention. - Restore returns CREATING from the request (table and restore_jobs row created atomically; ResourceInUseException on a name race, LimitExceededException over the job limit), spools and verifies every data file before writing any item, and records progress in a way a crash cannot split from the data: a data-database row per file on PostgreSQL (data migration 005 reserved), the same transaction per 10k-item batch on SQLite, idempotent re-application of one file on MongoDB followed by a document-count check against the manifest. Duplicate detection for unordered backups is the target's primary key, not an in-memory key set. The ACTIVE flip is conditional; DeleteTable on a restore target fails the job. - The manifest self-checksum is replaced by a sidecar over the stored bytes; the draft's canonical form depended on serde_json formatting. backup_arn is the backup's identity and source_table must agree with it. N key values are normalized before key text or key order is computed; the SQL ORDER BY recipe includes COLLATE "C" on PostgreSQL. - Index population on restore stays per backend via bulk_load; the shared restore-backfill worker the draft described exists only on MongoDB. - Format additions: item_order, per-file key_range, md5 and uncompressed_bytes, explicit nulls, three-state vector_indexes of creation specs, optional etag, dataFileS3Key as the full object key, partition-aware BackupArn as the single ARN implementation. - Filesystem store: owner-only root re-checked by every operation, owner-only rule on every descended directory including list and delete_prefix, ancestor ownership rule, objects 0600 and directories 0700 at creation, put returns object metadata; the consequence for backup pickup agents is stated. The S3 conformance suite is mandatory in CI against moto (MinIO no longer publishes images). - ListBackups paging and filtering, the restore override contract, sqlite-memory behavior, extenddb destroy leaving the store intact, a restore spool outside the store root, max_concurrent_jobs, heartbeat_interval_secs and a concrete pool-size rule are specified. - Phase 1 is rewritten as the PR series with acceptance criteria and GitHub numbers. Migration numbers are reserved up front so the PITR PR is not on the critical path. Co-authored-by: Lee Hannigan <lhnng@amazon.com>
LeeroyHannigan
left a comment
LeeroyHannigan
left a comment
There was a problem hiding this comment.
Requesting changes. The storage engineering under this PR held under reproduction: both of yesyayen's blocking items are fixed and each fix fails its test when reverted; restore atomicity against DeleteBackup held on PostgreSQL in both orders and across 100 race iterations on MongoDB; the migration, the sweep predicate, the backup_items cursor, the downgrade gate, and crash cleanup all behaved as described; cross-account access to backups is refused on every backend under every ARN variation tried; the restored data matched DynamoDB item for item on every index and projection. The problems are in the surface customers and operators touch, and four of them block the merge.
Blocking
1. The merged tree does not compile. #338 landed the Cassandra backend on main this morning with the pre-PR four-argument restore_table_from_backup. cargo check --workspace --all-targets on the merge of origin/main and ba9f869 fails in one crate, extenddb-storage-cassandra, E0050 (4 parameters, trait has 5). GitHub reports MERGEABLE because no file overlaps. Adding the parameter alone leaves three defects: the Cassandra restore hardcodes global_secondary_indexes: None and local_secondary_indexes: None, ignores both overrides, and never calls ensure_single_part_base_key or ensure_restorable, because those refusals live in each SQL storage backend rather than in the engine. Either port the definition, the overrides, and both refusals into Cassandra, or add a Cassandra row to docs/differences-from-dynamodb.md saying what its restore does not honour. In both cases move the two refusals into crates/engine/src/backup.rs so a backend cannot skip them.
2. RestoreTableFromBackup copies every item inside the request on PostgreSQL and SQLite, and a client timeout cancels it. Both backends await copy_backup_items and only then return CREATING for a table that is already ACTIVE (the comment at backup_engine.rs:1282 says so). Reproduced with 40,000 items and two GSIs, client read timeout 3 s, no retries. PostgreSQL: the target sat CREATING with zero items, then the sweep removed it 64 s after the disconnect; the restore was lost. SQLite: the target is still CREATING four minutes later, a retry into the same name returns ResourceInUseException, DeleteTable on it returns ResourceInUseException ("Table is being restored"), and DeleteBackup on the source returns BackupInUseException. The customer is locked out of the name and the backup until the process restarts. boto3 and the CLI default to a 60 s read timeout, and with one INSERT per item per index over a network a few hundred thousand items cross it, so large restores cannot succeed with stock clients. SQLite CreateBackup in this PR already detaches its copy with tokio::spawn for this reason (backup.rs:799). Do the same here on both backends: commit the CREATING target and its table_restores row, spawn the copy, return at once. Add a live test that drops the client mid-restore and asserts ACTIVE with the full count. This also makes the DeleteBackup and DeleteTable refusals observable the way the docs describe them.
3. DescribeBackup still returns the pre-PR subset. Against DynamoDB, a backup of a table with a GSI, an LSI, and TTL enabled returns SourceTableFeatureDetails with GlobalSecondaryIndexes, LocalSecondaryIndexes, and TimeToLiveDescription, and SourceTableDetails.ProvisionedThroughput ({0, 0} for on-demand; the member is required in the shape). ExtendDB omits all of them on all three backends, although backup_definitions now holds the data. A customer auditing a backup is told it has no indexes; a strongly typed SDK can fail to deserialize the response. Populate both from the stored definition.
4. The PR body and the docs state behaviour the build does not have. Checked on servers built from ba9f869: "DeleteTable on a table that is being restored returns ResourceInUseException on all three backends" (only MongoDB; see item 2); "SQLite similarly exposes CREATING while it copies" (the caller blocks until AVAILABLE; only a concurrent observer sees CREATING); "a backup is never stuck undeletable" (false after a MongoDB crash during $out, and after item 2 on SQLite); "backups of tables with vector indexes are refused" (only the restore is refused; CreateBackup returns an AVAILABLE backup). Every claim in the body and in differences-from-dynamodb.md needs a live check per backend before the next revision.
Wire behaviour, measured against DynamoDB
Each of these was run on the service and on all three ExtendDB backends.
RestoreSummary.RestoreDateTimeis the restore wall clock. The service reports the source backup's creation time (measured:RestoreDateTime1791976431.438 equal toBackupCreationDateTime, restore issued 74 s later). A client reading it to learn which point in time was restored gets the wrong instant.- The service drops
RestoreSummaryfrom DescribeTable once the table is ACTIVE (absent at ACTIVE and 30 s later). ExtendDB keeps it withRestoreInProgress: false, andtests/test_backup_restore_fidelity.py:252-254pins that. RestoreSummary.SourceTableArnis never emitted; the service sets it.- The CREATING restore response carries GSI
IndexStatus: ACTIVE; the service omits the member. - A malformed
BackupArnreturns AccessDeniedException; the service returns ValidationException. - Restoring into an existing table name returns ResourceInUseException; the service returns TableAlreadyExistsException.
- An empty
GlobalSecondaryIndexOverrideorLocalSecondaryIndexOverridelist and anSSESpecificationOverrideare refused; the service accepts them as no-ops. The blanket refusal is documented, the no-op case is not. - The PROVISIONED-with-GSIs refusal tells the caller to pass
GlobalSecondaryIndexOverride, which ExtendDB then refuses. The fidelity test pins that text at line 649. ListBackupsignoresLimitand never returnsLastEvaluatedBackupArn; every row comes back in one response. The service caps at 100 with a token.
PostgreSQL
- The restore's ownership is a session advisory lock on a dedicated connection that issues no statement for the whole copy.
SET idle_session_timeout = 0is best-effort and the comment atbackup_engine.rs:118-123says managed services may reject it. On RDS, Aurora, or behind a transaction-mode pooler the idle session can end while the copy continues; after 60 s the sweep flips the target to DELETING and the restore fails with "deleted while it was being restored". Heartbeat the lock session, or gate the sweep on a liveness signal that does not depend on an idle connection. Item 2 does not remove this; a long detached copy has the same exposure. - "Bounded memory" bounds the Rust buffer. The copy is one PostgreSQL transaction (
beginat line 325, onecommitafter the loop), and secondary-index rows are written one INSERT per item per index in the nested loop at line 464 while base rows are batched. A million-item table with four GSIs is one transaction with four million round trips, WAL held throughout, target pinned CREATING throughout. - A failure to open the lock connection (
max_connectionsreached) maps to an Internal error. Map it to LimitExceeded so clients back off. restore_table_to_point_in_time(lines 1393-1420) is unreachable, uses a fixed backup name, and discards its cleanup error. Delete it.
SQLite and MongoDB
- CreateBackup on a table with a vector index returns an AVAILABLE backup that RestoreTableFromBackup always refuses. The customer finds out at restore time. Refuse at CreateBackup with the same message.
- MongoDB: a crash during the base
$outcopy, beforerestore_backfill_pendingis set, leaves a CREATING target that nothing sweeps or resumes. DeleteTable refuses it and DeleteBackup of the source returns BackupInUse until the catalog is edited by hand. Crashes after$outself-heal. Add a sweep for CREATING targets carryingrestore_source_backup_arnwith no pending flag and no live operation, or let DeleteTable remove them. - MongoDB ignores
--data-dband--catalog-dband has no serve lock, so two processes on one replica set share one catalog, andsweep_stale_backup_claimsat startup clears the other process's in-flight markers. Pre-existing, but the sweep makes it matter. pool_size = 1is raised to 2 for a file-backed database with no log line and no mention in the config docs. A file-backed backup also pins one pooled connection for its duration, sopool_size >= 3is wanted for undisturbed concurrency; say so.BackupMetadataClaim's Drop release only runs when a runtime handle is current; a comment should say the startup sweep is the backstop.- The MongoDB backup is not a point-in-time snapshot (
$outreads the live collection). Line 93 says so in the general consistency row; the Backup and Restore section should state the per-item mixed-state consequence where an operator choosing MongoDB will read it.
Tenancy and validation
BackupTableDefinition::apply_toapplies the stored definition withoutvalidate_create_table. A definition with 30 GSIs, a GSI on an undefined attribute, 101 projected attributes, or negative throughput restores to an ACTIVE table that CreateTable would reject. Triggering it needs catalog write access; it is still wrong data stored, and the service re-validates.RESTORE_SLOTSis a process-global semaphore of four. One account can hold all four and every other account's restores wait 30 s and get LimitExceededException. Make the limit per account.converge_catalog_versionandrefuse_newer_catalogtreat a storedcatalog_versionthe binary cannot parse as "proceed" and stamp the compiled version over it (older_or_equal_or_absent_is_allowedassertsSome("garbage")is allowed). That defeats the downgrade guard in the one case it exists for. Fail closed.- Pre-existing, outside this diff:
RestoreTableFromBackup,DescribeBackup, andDeleteBackupare authorized againstarn:...:table/*becauseextract_table_namereads onlyTableName(crates/server/src/request_helpers.rs:42-46, 289-294; the server log showsresource="...:table/*"on a denial). A policy scoped to a target table or a backup ARN can never match. The service authorizes on the backup ARN and the target table. This PR makes restore a first-class feature on top of that gap; fix it here or file an issue; either way it needs recording.
Upgrade path and docs
Walked literally from docs/manuals/07-upgrade-manual.md on a 0.0.3 deployment created by the 0.1.13 binary.
extenddb migrate --config extenddb.tomlas written fails: it needs PostgreSQL admin credentials the manual never mentions, and the error isError: Internal("Internal(\"Cannot connect as admin: error returned from database: password authentication failed for user \"lhnng\"\")"). Document--pg-userandEXTENDDB_PG_PASSWORDin the command, flatten the nested error, and say what to pass.- The manual says to run
migratewithout--yesfirst to see what is pending; the tool prints the plan and thenError: --yes is required to apply migrationswith exit status 1, so a scripted upgrade underset -estops at the preview. Exit 0 for the preview, or a documented distinct code. - On SQLite the upgrade rebuilds
backup_itemsby copying every row into a new table in one write transaction (schema.rs:494-553). No manual mentions it, the dry run does not name it, and the function logs nothing. Document it, name it in the dry run when the old shape is present, and log start and finish with the row count. - The sweep that removes targets the old bug left in CREATING logs nothing. Operators will see tables vanish from ListTables with no server record of why. Log each removal by name.
- The manual's backup and rollback commands hardcode
extenddb_catalog,extenddb, andOWNER extenddb;initlets the operator choose all three. There is no SQLite subsection (stop, copy the file, migrate, roll back by restoring the copy). - 0.0.4 and migration 003 are also claimed by #368 and #335. The migration ledger keys on filename and the gate compares version numbers, so if #368 merges first and stamps 0.0.4, this binary applies its own 003 and converges with no refusal: two schemas named 0.0.4. Whoever merges second renumbers to 004 and 0.0.5 (and the two assertions in
migrations.rstests, and the manual's 0.0.4 section). The order should be agreed now. differences-from-dynamodb.mdnames three backends; the merged tree ships four.- The
Storagetrait and the on-disk format changed.CONTRIBUTING.mdanddocs/rfcs/README.mdrequire an accepted RFC for both; the PR links #348, a design PR, not a numbered accepted RFC. - The PR template asks for a
## Breaking changessection; the changes are spread through the prose, and the one customers will hit first (overrides that were silently ignored are now hard errors) deserves its own line.
Minor
Backup ARN timestamp is 13 digits where the service zero-pads to 14; BackupCreationDateTime is whole seconds where the service is sub-second; the restore response adds LastUpdateToPayPerRequestDateTime, which the service omits; ListBackups summaries omit TableId; error __type namespace differs for ValidationException and AccessDeniedException; with two invalid inputs the service checks the ARN account before the target name.
| account_id: &str, | ||
| target_table_name: &str, | ||
| backup_arn: &str, | ||
| overrides: RestoreTableOverrides, |
There was a problem hiding this comment.
main now has crates/storage-cassandra/src/backup_engine.rs (#338, merged this morning) with the old four-argument signature, so the merge fails to compile: cargo check --workspace --all-targets on origin/main plus this branch gives E0050 in extenddb-storage-cassandra and nothing else. Adding the parameter there leaves three defects: the Cassandra restore hardcodes global_secondary_indexes: None and local_secondary_indexes: None, ignores the overrides, and never calls ensure_single_part_base_key or ensure_restorable, which this PR enforces inside the PostgreSQL and SQLite backends rather than in the engine. Port the definition and both refusals into Cassandra, or document in the differences matrix what its restore does not honour, and move the two refusals into the engine so a backend cannot skip them.
There was a problem hiding this comment.
Fixed. The trait now takes a resolved RestoreRequest. The engine runs ensure_restorable, restore_input and validate_create_table for every backend, and Cassandra restores GSIs, LSIs and the overrides through its V005 migration. cargo check --workspace --all-targets is clean on the merge.
| // CREATING with no scheduled transition, so nothing else would | ||
| // move it on, and the name would stay taken. Deleting the target | ||
| // cascades its table_restores row. | ||
| let copied = self.copy_backup_items(&desc, &backup_arn).await; |
There was a problem hiding this comment.
The copy is awaited inside the request future, so the response at line 1282 says CREATING for a table that is already ACTIVE, and a client that times out drops the future mid-copy. Reproduced with 40,000 items and two GSIs, read timeout 3 s, no retries: the target sat CREATING with zero items and the sweep removed it 64 s after the disconnect. The restore was lost. boto3 and the CLI default to 60 s, so any restore whose copy takes longer cannot succeed with a stock client. Commit the CREATING target and its table_restores row, spawn the copy, and return at once, the way SQLite CreateBackup does at backup.rs:799. Add a live test that drops the client and asserts the target reaches ACTIVE with the full count.
There was a problem hiding this comment.
Fixed. Registration and copy run in a spawned task; the request waits only for the CREATING description. Once the request is authenticated and authorized, the whole handler (backup read, validation, registration) also runs in its own task, so a disconnect does not cancel it part-way. Live tests: a 1 ms-read-timeout disconnect and the copy window (DeleteTable/DeleteBackup refused while CREATING) on all four backends, plus storage-level tests that drop the request future.
| // last one; until then the target is CREATING and refuses every | ||
| // data-plane request. A failure part-way is cleaned up here, a | ||
| // crash part-way by the startup sweep. | ||
| let copied = self.copy_backup_items(&desc, &backup_arn).await; |
There was a problem hiding this comment.
The SQLite path has the same shape. With the client gone after 2 s on a 40,000-item restore, the target is still CREATING with RestoreInProgress: true four minutes later; a retry into the same name returns ResourceInUseException, DeleteTable on the target returns ResourceInUseException ("Table is being restored"), and DeleteBackup on the source returns BackupInUseException. SQLite sweeps abandoned restores only at startup, so the name and the backup are locked until the process restarts. create_backup at line 799 already detaches its copy with tokio::spawn and the comment there names this hazard; the restore path needs the same treatment.
There was a problem hiding this comment.
Fixed the same way as PostgreSQL. serve also now refuses a file database left in a pre-release 0.0.4 shape and points to extenddb migrate --yes.
| // restore is still running, so disable it for this session when the | ||
| // server permits. Some managed PostgreSQL services reject this SET; | ||
| // ownership still works there, but remains subject to their idle timeout. | ||
| if let Err(e) = sqlx::query("SET idle_session_timeout = 0") |
There was a problem hiding this comment.
This connection issues no statement after pg_try_advisory_lock for the whole copy. When the SET is refused (the comment says managed services refuse it) or the connection sits behind a transaction-mode pooler, the idle session can end while the copy continues, the lock frees, and after 60 s the sweep flips the target to DELETING; the restore then fails with "deleted while it was being restored". That covers RDS, Aurora, and PgBouncer. Heartbeat the session while the copy runs, or gate the sweep on a liveness signal that does not depend on an idle connection.
There was a problem hiding this comment.
The lock session now runs SELECT 1 every 15 s and the copy stops if that fails, which covers idle-session reaping on RDS/Aurora. A transaction-mode pooler is not supported for restores (the lock is a session advisory lock): the admin guide says so, and the comments that implied poolers were handled are removed. Replacing the lock with a lease row (as MongoDB and Cassandra use) is the durable fix if transaction pooling matters; I left that out of this PR.
| "Backup not found: {backup_arn}" | ||
| ))); | ||
| } | ||
| let mut tx = self.data_pool.begin().await.map_err(db_err)?; |
There was a problem hiding this comment.
The whole copy is one transaction: begin here, one commit after the loop. The 500-item / 4 MiB bound limits the Rust buffer only. In write_restore_batch the base rows go in one multi-row INSERT, but the loop at line 464 writes secondary-index rows one INSERT per item per index. A million-item table with four GSIs is one transaction with four million round trips, WAL held for the duration, and the target pinned CREATING throughout, which widens the sweep exposure above. Batch the index rows per index per batch, and either commit per batch with cleanup on failure or document the single-transaction cost for large restores.
There was a problem hiding this comment.
Fixed: one commit per batch, one multi-row INSERT per index per batch. The catalog side still holds one REPEATABLE READ snapshot for the copy (so the backup rows are read whole); that is documented in the rustdoc.
| # DescribeTable keeps reporting where the table came from, finished. | ||
| summary = table["RestoreSummary"] | ||
| assert summary["SourceBackupArn"] == backup_arn | ||
| assert summary["RestoreInProgress"] is False |
There was a problem hiding this comment.
DynamoDB does not return RestoreSummary on DescribeTable once the table is ACTIVE (checked at ACTIVE and 30 s later). This assertion pins the ExtendDB behaviour as correct. The test should assert the member is absent after ACTIVE, and the engine should stop emitting it.
There was a problem hiding this comment.
The test now asserts the member is absent once ACTIVE, and the engine no longer emits it.
| | What a restore carries across | Key schema, attribute definitions, items, secondary indexes, billing mode and provisioned throughput, SSE settings. Stream settings, TTL, tags, auto scaling, IAM policies, and CloudWatch settings must be reapplied by hand | The same set on PostgreSQL, SQLite, and MongoDB, plus table class and on-demand throughput. Streams, TTL, tags, and deletion protection are not restored. A backup written by a version before catalog 0.0.4 carries no table definition and restores as keys and items only, with 5/5 provisioned throughput | | ||
| | `RestoreTableFromBackup` overrides | Current request members are `BillingModeOverride`, `ProvisionedThroughputOverride`, `GlobalSecondaryIndexOverride`, `LocalSecondaryIndexOverride`, `SSESpecificationOverride`, `OnDemandThroughputOverride`, and `VectorIndexOverride`; DynamoDB applies them to the restored table | PostgreSQL, SQLite, and MongoDB support `BillingModeOverride` and `ProvisionedThroughputOverride`. Switching to `PAY_PER_REQUEST` drops table and GSI throughput. `GlobalSecondaryIndexOverride`, `LocalSecondaryIndexOverride`, `SSESpecificationOverride`, `OnDemandThroughputOverride`, and `VectorIndexOverride` are refused with `ValidationException` before the target table is created. Switching an on-demand backup with GSIs to `PROVISIONED` is likewise refused because it requires `GlobalSecondaryIndexOverride` | | ||
| | Multi-part base table keys (preview) | Not supported | A backup of a table created with `enable_multipart_keys` is refused at restore with `ValidationException`; the item paths address such tables by their first HASH and RANGE attribute only | | ||
| | `DeleteBackup` during a restore from that backup | `BackupInUseException` | `BackupInUseException` (HTTP 400) until the restore completes or fails, on all three backends. PostgreSQL and SQLite register the restore in the same transaction that creates its target, with the backup row held so a concurrent `DeleteBackup` orders strictly before (the restore then sees no backup) or after (it sees the restore). MongoDB has no cross-collection transaction and uses a claim marker on the backup's metadata instead: a restore claims the backup before reading it and releases the claim once its `CREATING` target exists; `DeleteBackup` claims it the same way before dropping data. A claim left by a crashed process is cleared at the next startup, so a backup is never stuck undeletable; until then a restore or delete of that backup gets `BackupInUseException` | |
There was a problem hiding this comment.
"A backup is never stuck undeletable" is false in two cases on this branch: a MongoDB crash during the base $out copy leaves a CREATING target that blocks DeleteBackup with BackupInUse until the catalog is hand-edited, and on SQLite a restore whose client disconnected leaves the same state until restart. The next row's "PostgreSQL, SQLite, and MongoDB match" is only observable on MongoDB: on the SQL backends the restore completes inside the request, so a sequential client's DeleteTable or DeleteBackup after the response succeeds every time.
There was a problem hiding this comment.
Rewritten for four backends. Rows that rest on code reading or storage-level tests are marked (from the code), numbers from earlier builds of the branch are marked (earlier build), and the header says so.
| | Restore interrupted by a server crash | Managed by the service | PostgreSQL removes an abandoned `CREATING` restore target on a control-plane pass; SQLite removes one when the next server starts. MongoDB has no abandoned-restore sweep: the target remains `CREATING`, the API continues to refuse `DeleteTable`, and an operator must remove its catalog and data collections by hand | | ||
| | Point-in-time recovery | 35-day window | Not supported; `UpdateContinuousBackups` and `RestoreTableToPointInTime` are refused | | ||
| | Where backups live | Managed storage, independent of the table | Inside the database the server already uses, so a lost database loses its backups too: on PostgreSQL in the catalog database (`backups`, `backup_items`, `backup_definitions`, `table_restores`), on SQLite in the one database file, on MongoDB as a `backups` metadata collection in the catalog database and one data collection per backup in the data database. Take database-level backups for off-host copies until the backup store program ships | | ||
| | Backup creation status | `CreateBackup` returns while the backup is `CREATING`; `ListBackups` and `DescribeBackup` expose that state until it becomes `AVAILABLE` | SQLite similarly exposes `CREATING` while it copies: size and item count remain zero, `DeleteBackup` is refused, and `RestoreTableFromBackup` reports the backup missing. PostgreSQL and MongoDB complete their copy before returning and expose only `AVAILABLE` | |
There was a problem hiding this comment.
"SQLite similarly exposes CREATING while it copies" reads as if the caller gets a CREATING response. The caller blocks until AVAILABLE on all three backends (1.4 s for 50,000 items; minutes for a large table), and only a concurrent observer sees the CREATING row. Say that, and note that CreateBackup blocks for the copy duration, or return CREATING and let the detached task finish, which the SQLite code already supports.
There was a problem hiding this comment.
Rewritten: CreateBackup blocks the caller for the copy on all four backends; only a concurrent observer sees CREATING. Detaching CreateBackup is left for a follow-up.
| Upgrade sequence, on PostgreSQL and SQLite alike: | ||
|
|
||
| ```bash | ||
| extenddb stop --config extenddb.toml |
There was a problem hiding this comment.
Followed literally on a 0.0.3 deployment this fails: migrate connects as a PostgreSQL admin and the manual gives no credentials, so it falls back to the OS user and prints Error: Internal("Internal(\"Cannot connect as admin: ... password authentication failed for user \"lhnng\"\")"). Show --pg-user and EXTENDDB_PG_PASSWORD here, and flatten that nested error in the tool. The preview step on line 203 also exits 1 with Error: --yes is required, so a scripted upgrade under set -e stops at the preview; exit 0 for the preview or say the non-zero exit is intended. Nothing here tells a SQLite operator about the backup_items rebuild, and the backup and rollback steps above hardcode extenddb_catalog, extenddb, and OWNER extenddb.
There was a problem hiding this comment.
--pg-user and EXTENDDB_PG_PASSWORD are shown; the error is one plain line naming the role it tried; the preview exits 3 when work is pending and 0 when up to date (documented); the commands are parameterized; and there are SQLite, Cassandra and MongoDB subsections.
|
|
||
| CREATE INDEX IF NOT EXISTS idx_table_restores_backup ON table_restores (source_backup_arn); | ||
|
|
||
| UPDATE settings SET value = '0.0.4' WHERE key = 'catalog_version'; |
There was a problem hiding this comment.
#368 and #335 also ship a 003_*.sql that writes 0.0.4. The ledger keys on filename and converge_catalog_version compares numbers, so if #368 merges first and stamps 0.0.4, this binary applies its own 003 and converges with no refusal: two different schemas both called 0.0.4, and the startup gate cannot tell them apart. Whoever merges second renumbers to 004 and 0.0.5 (plus the two assertions in the migrations.rs tests and the manual's 0.0.4 section). The order should be agreed before any of the three lands.
There was a problem hiding this comment.
Not renumbered; I need your call on the order (open question in the PR body: #396 assumes this PR keeps 003/0.0.4 and #335 moves to 004/0.0.5, and does not place #368). Meanwhile migrate applies an unrecorded catalog migration file even when the version already matches, and PostgreSQL serve refuses to start on a catalog missing one of its files, so a second 003 can no longer be skipped silently.
ba9f869 to
8dab086
Compare
8dab086 to
7b97903
Compare
|
Thanks for the reproductions; every one of them held, and 7b97903 answers them. The PR body is rewritten; the short version: Blocking items. (1) The engine now resolves a backup into one validated Wire, PostgreSQL, tenancy, upgrade items are each answered inline on your comments. Two things found along the way that are outside backup/restore: Cassandra's GSI propagation workers were stopped right after startup in Follow-ups filed: #403 (driver reconnection) and #404 (Cassandra's restore copy goes through the LWT put path, which is why it is several times slower than the others). Still open, needing your call: the 003 / catalog 0.0.4 order against #335 and #368 (question in the body), and the RFC the contributing guide requires for the Verification is in the body: fidelity file green on all four backends on the final binaries; the pytest legs fail exactly the tests that fail on main's binaries on the same host; fmt, clippy 1.97 and the 1.88 check clean. |
There was a problem hiding this comment.
Second pass on 7b97903. I re-ran the round-1 items on all four backends and against live DynamoDB.
The merged tree compiles. Cassandra restores both GSIs and the LSI, checked attribute by attribute over 2,000 items. The 40,000-item restore with a 3 s client timeout returns CREATING in 0.06 s on PostgreSQL and 0.01 s on SQLite, then reaches ACTIVE with all items. Disconnect at 1 ms, 20 concurrent disconnects, SIGTERM and SIGKILL during the copy leave a CREATING target that the sweep removes. DescribeBackup matches the service field by field. The body and the differences matrix match the running builds. The PostgreSQL heartbeat, per-batch commits, restore slots, migration matrix with a colliding 003 file, version guard, backup-ARN authorization, stored-definition tamper cases and MongoDB claim races (130 iterations) pass. The upgrade manual runs as written under set -e on PostgreSQL and SQLite, including rollback. fmt, clippy on 1.97, the 1.88 check, four notices variants and 1,173 workspace tests pass.
Six code items remain, one Blocking.
- Blocking. TagResource on a backup ARN defeats tag-conditioned backup authorization. Backup-ARN tags now feed
dynamodb:ResourceTagfor DescribeBackup, DeleteBackup and RestoreTableFromBackup, but TagResource is authorized ontable/*andvalidate_resource_arnaccepts anytable/...resource, so a backup ARN passes. Reproduced on a fresh account on PostgreSQL and SQLite: a user allowed DescribeBackup only on backups taggedclassification=dev, plus TagResource ontable/*, is denied, tags the backup, and is then allowed. An explicit Deny TagResource ontable/*/backup/*in the same policy did not fire. Fix: refuse a/backup/sub-resource invalidate_resource_arn(the service has no taggable backups) and add a wire test for the case above. test_provisioned_restore_override_requiring_gsi_override_leaves_no_targetpins, under_is_real_dynamodb(), a message the service does not emit. The service saysInvalid Request: Must override ProvisionedThroughput if BillingMode is overridden to PROVISIONED for index by_gpk. The comment above it says it was measured there; the body says the file was not run against DynamoDB this round.ListBackupsreturns the full set forBackupTypeandTimeRangeLowerBound/UpperBound; the service filters on both. Measured on all four backends. Either filter on them or reject the members.- A
GlobalSecondaryIndexOverrideequal to the backup's own indexes is refused; the service accepts it, and the console's restore flow sends that list. Accept it as a no-op. Same forOnDemandThroughputOverrideon an on-demand restore. - MongoDB
connect_guardedrefuses a non-primary read preference but accepts any write concern. The two-phase claim protocol is only safe underw: majority, which the body states as an assumption. Refuse or warn on anything weaker, as the read-preference guard does. - Cassandra
UpdateTimeToLiveon any table with a GSI returns HTTP 500 (reproduced; the log saysValidation error: TTL is not supported while a table has asynchronously propagated GSIs). Pre-existing and listed under Not changed here, but SDKs retry a 500. Map it toValidationExceptionand add the row to the differences doc.
Minor: a bad-partition BackupArn is BackupNotFoundException where the service says ValidationException; the CREATING restore response carries DeletionProtectionEnabled and NumberOfDecreasesToday, which the service omits there; serve wraps the unparseable-version refusal as "Internal error" where migrate prints a clean line; Cassandra on an unmigrated catalog returns 500 on backup operations rather than a message naming migrate; Cassandra init leaves username and password commented out and then tells the operator to run serve, which fails against the PasswordAuthenticator node this PR requires.
The RFC, the 0.0.4 numbering and #403 are not part of this review.
| /// | `DescribeBackup` | `backup` | the `BackupArn` | | ||
| /// | `DeleteBackup` | `backup` | the `BackupArn` | | ||
| /// | `RestoreTableFromBackup` | `backup`, `table` | the `BackupArn` and `table/<TargetTableName>` | | ||
| /// | `ListBackups` | none | `table/*`, also when `TableName` filters the list | |
There was a problem hiding this comment.
TagResource is not in this table, so it is authorized on table/*, and validate_resource_arn in crates/engine/src/tagging.rs:70 accepts a backup ARN because it starts with table/. With backup-ARN tags now feeding dynamodb:ResourceTag for the three operations above, a principal restricted by a tag condition writes the tag itself. Reproduced on a fresh account on PostgreSQL and SQLite: DescribeBackup denied, TagResource on the backup ARN allowed (with an explicit Deny on table/*/backup/* in the policy), DescribeBackup allowed. Refuse /backup/ sub-resources in validate_resource_arn; the service does not tag backups. Add a wire test for that sequence.
There was a problem hiding this comment.
Fixed in 5468866. TagResource, UntagResource and ListTagsOfResource refuse a backup ARN with ValidationException (after the account and region checks, so another account's backup ARN is still AccessDeniedException), and authorization never reads resource tags for a backup ARN, so tags an earlier build stored under one no longer satisfy a condition either. tests/test_abac.py::TestBackupResourceTags reproduces your case in authenticated mode and fails on the earlier build at the TagResource step, on PostgreSQL and SQLite. The tagging actions themselves are still authorized on table/, so your Deny on table//backup/* still does not reach them; the engine refuses the backup ARN regardless. That is listed under Not changed here.
| account_id: &str, | ||
| body: &Value, | ||
| ) -> Result<Value, DynamoDbError> { | ||
| let limit = match body.get("Limit") { |
There was a problem hiding this comment.
Limit and ExclusiveStartBackupArn are read here; BackupType, TimeRangeLowerBound and TimeRangeUpperBound are not, and the service filters on all three. Measured: with only USER backups created today, BackupType=SYSTEM returns 0 on the service and every backup here; TimeRangeUpperBound=2020-01-01 the same. Filter the in-memory list, or reject the members.
There was a problem hiding this comment.
Fixed. BackupType (USER when absent, every type for ALL) and TimeRangeLowerBound (inclusive) / TimeRangeUpperBound (exclusive) filter the list before paging, so Limit and LastEvaluatedBackupArn count only matching backups. Every ExtendDB backup is USER, so SYSTEM and AWS_BACKUP list nothing. Measured on SQLite and PostgreSQL; the millisecond edges are unit-tested.
| format!( | ||
| "One or more parameter values were invalid: index {index_name} has no provisioned \ | ||
| throughput to restore with BillingModeOverride PROVISIONED, and \ | ||
| GlobalSecondaryIndexOverride is not supported on ExtendDB. Restore with \ |
There was a problem hiding this comment.
The service accepts a GlobalSecondaryIndexOverride that lists the backup's own indexes, and that is what the console's restore flow sends. Accept it as a no-op when it equals the stored definition (name, key schema, projection), refuse only a changed list. OnDemandThroughputOverride on an on-demand restore is also accepted by the service and refused here.
There was a problem hiding this comment.
Fixed, with one exception I want to flag. A GSI or LSI override listing the backup's indexes unchanged (any order, NonKeyAttributes as a set) is accepted as a no-op, and so is an OnDemandThroughputOverride equal to the backup's on an on-demand restore; a changed list is a ValidationException naming the first difference. The wire test builds the override from DescribeBackup for a PROVISIONED and an on-demand backup and restores on all four backends. The exception: for an on-demand backup, DescribeBackup reports each GSI's ProvisionedThroughput as {0, 0}, and the service model requires at least 1 for each member, so boto3 and the AWS CLI refuse that list before sending. The client leaves the member out, which is accepted, and ExtendDB refuses {0, 0}. I could not measure what the service replies to {0, 0} from an SDK that does not check the minimum. If you have that measurement and the service accepts it, I will accept it too.
| "extenddb_catalog", | ||
| DatabaseOptions::builder() | ||
| .selection_criteria(SelectionCriteria::ReadPreference(ReadPreference::Primary)) | ||
| .read_concern(ReadConcern::local()) |
There was a problem hiding this comment.
The local read here is correct for the claim protocol only if every claim write is w: majority; the body states that as an assumption. connect_guarded refuses a non-primary read preference and warns on no TLS, but never looks at the write concern, so w=1 in the connection string is accepted and a claim acknowledged before a stepdown can be lost after this read saw it. Refuse or warn on a write concern weaker than majority, as the read-preference check already does.
There was a problem hiding this comment.
Fixed. connect_guarded now refuses w=0, a numeric or tagged w, and journal=false, as it refuses a non-primary read preference; an unset w (server default) and w=majority are accepted. Since an unset w can inherit a weak server default, the catalog database and the $out copies of CreateBackup and restore are also pinned to w: majority. Transactions are unaffected, because the driver clears inherited concerns inside one.
| # DynamoDB, the service refuses this request the same way. | ||
| assert error["Message"] == ( | ||
| "One or more parameter values were invalid: " | ||
| "GlobalSecondaryIndexOverride must be specified for index: by_gpk " |
There was a problem hiding this comment.
The service returns Invalid Request: Must override ProvisionedThroughput if BillingMode is overridden to PROVISIONED for index by_gpk, so this branch fails against DynamoDB. The comment above says it was measured there. Fix the text and run the file against the service before the next push.
There was a problem hiding this comment.
The DynamoDB branch now expects the text you measured, and the comment says it is your measurement. I could not run the file against the service before this push; I have no DynamoDB access in this environment, and the PR body says the file was not run against DynamoDB this round. If you can run it, I'll fix anything else it turns up.
| | What a restore carries across | Key schema, attribute definitions, items, secondary indexes, billing mode and provisioned throughput, SSE settings. Stream settings, TTL, tags, auto scaling, IAM policies, and CloudWatch settings must be reapplied by hand | The same set on all four backends, plus table class and on-demand throughput. Streams, TTL, tags, and deletion protection are not restored. A backup written before catalog 0.0.4 (PostgreSQL, SQLite, Cassandra) has no recorded table definition and restores with its keys, items, and billing mode but no secondary indexes. A provisioned one restores with 5/5 throughput on PostgreSQL and SQLite and with its recorded throughput on Cassandra | | ||
| | How `RestoreTableFromBackup` runs | Returns at once with the target `CREATING`; the service copies in the background | The same on all four backends: the response carries the target `CREATING`, and the items are copied by a background task in the server, so a client that times out or disconnects does not stop it. Once the request has been authenticated and authorized, the rest of it, from reading the backup to registering the restore, runs in a task of its own, so a client that disconnects after that does not cancel it either: the restore runs and the client finds the target with `DescribeTable`. A client that disconnects while the request is still being authenticated or authorized cancels it, and nothing is created *(from the code)*. Measured with a client whose 1 ms read timeout closed its connection before any response arrived: on all four backends the restore was registered and finished. Measured with a 60,000-item backup of a table with no secondary indexes: response in 0.032 s (PostgreSQL), 0.012 s (SQLite), 0.053 s (MongoDB), 0.151 s (Cassandra); `ACTIVE` after 1.5 s, 2.1 s, 3.6 s, and 11.2 s. On Cassandra the response waits for the target table's schema to be created: 0.75 to 1.3 s with five restores running at once *(earlier build)*, which is longer than a client read timeout of 1 s. A restore runs inside the server process; if the process stops, see the abandoned-restore row | | ||
| | `RestoreSummary` | The restore response and `DescribeTable` while `CREATING` report `SourceBackupArn`, `SourceTableArn`, `RestoreDateTime` (the backup's `BackupCreationDateTime`, not the time the restore ran), and `RestoreInProgress=true`. Once the table is `ACTIVE`, `DescribeTable` omits `RestoreSummary` | The same on all four backends. No response that describes a table (`CreateTable`, `UpdateTable`, `DeleteTable`, `DescribeTable`, `RestoreTableFromBackup`) carries `RestoreSummary` unless the table is `CREATING`; what the service's `UpdateTable` returns for a restored table was not measured | | ||
| | `RestoreTableFromBackup` overrides | Current request members are `BillingModeOverride`, `ProvisionedThroughputOverride`, `GlobalSecondaryIndexOverride`, `LocalSecondaryIndexOverride`, `SSESpecificationOverride`, `OnDemandThroughputOverride`, and `VectorIndexOverride`; DynamoDB applies them to the restored table | See the overrides table below. A refused override is refused before the target table is created | |
There was a problem hiding this comment.
Not in this table: on Cassandra, UpdateTimeToLive on any table with a GSI is HTTP 500 (log: Validation error: TTL is not supported while a table has asynchronously propagated GSIs), so TTL and a GSI cannot coexist on that backend. Pre-existing, but a 500 makes SDKs retry an unsupported call. Map it to ValidationException in the engine and add the row here.
There was a problem hiding this comment.
Fixed. The engine now maps a backend's validation refusal in the TTL handlers to ValidationException. On Cassandra, enabling TTL on a table with an asynchronously propagated GSI names the table, the index and its delay; a synchronous GSI is fine. The row is in the differences doc. Observed on a served Cassandra build at the default 10 ms, and pinned by a live storage-level test (refused at 50 ms, enabled once the GSI is set to 0).
7b97903 to
5468866
Compare
|
Thanks for the second pass. 5468866 answers each item; the PR body is updated with the runs behind them. 1. TagResource on a backup ARN (blocking). Fixed in two parts. 2. Pinned service message. The DynamoDB branch of 3. ListBackups filters. 4. Index overrides equal to the backup's. 5. MongoDB write concern. A connection string with 6. Cassandra UpdateTimeToLive. The engine now maps a backend's validation refusal in the TTL handlers to Minor items.
CI. Two Cassandra pytest runs on the previous heads failed, on different tests each time. In the first, about 1 s GSI delays came from the GSI worker lifetime change the PR used to carry. I took that change out: reviving the workers exposes defects of its own, and the bug it fixes is on |
5468866 to
3528d05
Compare
|
Follow-up: the head is now 3528d05. The only change since 5468866, which my replies above cite, is that two unit tests no longer interpolate a test password and a billing-mode variable into assertion messages. CodeQL had flagged both as cleartext logging. All 34 checks pass on 3528d05, including the Cassandra pytest leg. |
3528d05 to
7f139f2
Compare
LeeroyHannigan
left a comment
LeeroyHannigan
left a comment
There was a problem hiding this comment.
I re-ran the list on all four backends and against the live service. The restore wedge on MongoDB, the backup-flood starvation on SQLite, the mid-restore UpdateTable, the DescribeBackup masking, the read lag after a table reports ACTIVE, the migrate data-database name, the name and ARN validation, the error classes, and the backup authorization resources now behave like the service.
Two things before it merges:
-
CI is red. CodeQL flags three test asserts that interpolate the fake account id into their message. It's not a secret, but the check still blocks the merge. Get the run green (flagged inline).
-
The body still says Cassandra on an unmigrated catalog returns internal errors. The binary now answers a ValidationException naming
extenddb migrate --yes, so the sentence is out of date.
A doc comment still carries a leftover marker, flagged inline.
Not blocking this PR: the GSI and LSI override refusal is stricter than the service. I measured the service accepting overrides this code refuses, and restoring them (inline on overrides.rs). A few error messages also read differently from the service's wording while the class is right. I'll track both of those separately so they don't hold up the rest.
| } | ||
| assert!( | ||
| Instant::now() < deadline, | ||
| "{account}/{name} did not reach {status}: {:?}", |
There was a problem hiding this comment.
CodeQL is red on this line and two more, so the PR can't merge. The fake account id flows into the assert message and the scanner reads it as a logged credential. Same at backup_live_tests.rs:1541 and crates/engine/src/backup/tests.rs:3250. Drop the id from the assert text, or add a query-filter for the test paths, and get CI green.
There was a problem hiding this comment.
Fixed in 744d18a. The three assert messages no longer interpolate the account id: backup_live_tests.rs now names the table and its current status (and the request index in the admission test), and backup/tests.rs:3250 uses a fixed message instead of printing the response. No query filter was added.
| s.cleanup().await; | ||
| } | ||
|
|
||
| /// (r4-2) The definition read reports a lost connection as `Transient`, not |
There was a problem hiding this comment.
Drop the leftover marker at the start of this doc comment, the rest reads fine.
| backup | ||
| .iter() | ||
| .find(|i| name_of(i) == name) | ||
| .ok_or_else(|| invalid(format!("{kind} index {name} is not in the backup"))) |
There was a problem hiding this comment.
I measured this against the live service. It treats GlobalSecondaryIndexOverride and LocalSecondaryIndexOverride as a redefinition of the restored index set. It accepts an override that names an index the backup didn't have, and one that re-keys an existing index, and it restores both. This refuses both, here with is not in the backup and just below at line 323 with KeySchema differs. Refusing is safe, the caller gets a ValidationException rather than wrong data, so it isn't blocking. It is narrower than the service though, so I'd rather widen it in a follow-up than hold the PR on it.
There was a problem hiding this comment.
Agreed, leaving it narrower here. The body's Not changed here now says the override is accepted only for the backup's own indexes with unchanged key schema and projection, that you measured the service accepting and restoring a new index and a re-keyed one, and that you're tracking the widening and the message wording separately.
7f139f2 to
744d18a
Compare
…; recover abandoned restores
Backup and restore now reproduce the table that was backed up, behave the
same on PostgreSQL, SQLite, MongoDB, and Cassandra, and match DynamoDB on
the points measured against the service.
Engine. The engine resolves a backup into one validated CreateTableInput
before any backend sees it: it runs the multi-part-key and vector-index
refusals, applies the overrides, and runs validate_create_table. Backends
only create the target, register the restore, and copy. The trait gains
backup_restore_source; restore_table_from_backup takes a resolved
RestoreRequest; restore_table_to_point_in_time is removed. Once the request
is authenticated and authorized, the whole restore runs in a task of its
own, so a client that disconnects does not cancel it part-way.
Restore runs after the response on every backend: the target is returned
CREATING and copied by a background task. While it is CREATING, DeleteTable
on it is ResourceInUseException and DeleteBackup on its source is
BackupInUseException. A target whose server stopped mid-copy is removed and
logged: PostgreSQL after 60 s, SQLite at the next start, MongoDB and
Cassandra by a heartbeat sweep at startup and every 15 s.
Wire behaviour: RestoreDateTime is the backup's creation time and
SourceTableArn is set; RestoreSummary appears only while CREATING, on every
table description (TableDescription::prepare_for_response). DescribeBackup
reports SourceTableDetails.ProvisionedThroughput and SourceTableFeatureDetails.
ListBackups pages with Limit and LastEvaluatedBackupArn and carries TableId.
Restoring into an existing name is TableAlreadyExistsException; a malformed
BackupArn is ValidationException, checked before the target name.
SSESpecificationOverride, GlobalSecondaryIndexOverride,
LocalSecondaryIndexOverride, and OnDemandThroughputOverride are applied as
the service applies them (an empty list restores no indexes of that kind, a
subset restores those, a GSI entry's throughput is the index's); an unknown
or changed index definition, a throughput member contradicting the billing
mode, {0, 0}, and BillingModeOverride=PROVISIONED with a GSI whose
throughput no entry supplies are ValidationException with the service's
text. TableClass is not carried across; the target is STANDARD. ListBackups
filters on BackupType and the time range; its cursor must match the
account, region, and TableName filter (a listed backup's stored ARN in an
earlier region is accepted, so paging passes it), and the time bounds are
checked. A
BackupArn in another partition is a ValidationException. BackupArn and
ExclusiveStartBackupArn shapes are checked as the service checks them
("Invalid Request: BackupArn is not valid" / "Invalid Backup ARN"),
surrounding whitespace is trimmed, and an ARN of the caller's account in
another region resolves (looked up in the server's region first, then as
written, so backups created under an earlier server.region stay
reachable); the 13-digit ids of earlier builds still resolve. BackupName
and TableName are validated as the model says (3 to 255 characters,
[a-zA-Z0-9_.-]+) with the service's "validation error(s) detected" texts;
TableName may be the table's ARN; CreateBackup of a missing or CREATING
table is TableNotFoundException. DescribeBackup propagates a failure to
read the definition as an internal error and degrades only for a version
it does not know; DeleteBackup returns the full description DELETED.
Restoring into a CREATING restore target is TableInUseException and a
second restore of a backup whose restore is CREATING is
BackupInUseException, decided inside each backend's registration
transaction, claim, or lightweight transaction. The key-info cache stores
no negative entry for a name the catalog has in a non-ACTIVE state, and the
server hands a not-found answer (never a storage error) to the engine, so
the first read after ACTIVE succeeds at one catalog load per refused
request. A PROVISIONED
backup whose GSI has no throughput restores with the table's throughput. CreateTable and UpdateTable refuse GSI throughput
below 1 on PROVISIONED tables and more than 100 projected NonKeyAttributes.
CreateBackup of a table with vector indexes is refused. Backup ids are 14
zero-padded millisecond digits on every backend.
Authorization: backup operations are evaluated on the resources the service
evaluates, not on TableName. ListBackups on table/*/backup/* or
table/<TableName>/backup/* (measured on DynamoDB); DescribeBackup and
DeleteBackup on the backup ARN (Service Authorization Reference);
RestoreTableFromBackup (measured) on the backup ARN and
then Scan, Query, UpdateItem, PutItem, GetItem, DeleteItem, and
BatchWriteItem on table/<TargetTableName>, never RestoreTableFromBackup on
the target. A backup ARN is evaluated in each spelling the engine may
resolve (the server-region rebuild, then the ARN as written when it
differs); both must allow. CreateBackup with TableName as a table ARN is
evaluated on the table it names. A foreign-account ARN is refused before
policy evaluation. The seven target evaluations carry none of the
request-derived DynamoDB condition keys (LeadingKeys, Attributes, Select,
ReturnValues, ReturnConsumedCapacity, FullTableScan, EnclosingOperation);
dynamodb:ResourceTag is the target's. BREAKING: released builds evaluated
a ListBackups with a TableName on table/<TableName>, and ListBackups
without one, DescribeBackup, DeleteBackup, and RestoreTableFromBackup on
table/*, so a ListBackups grant on table/<name> no longer lists that
table's backups (grant table/<name>/backup/*), RestoreTableFromBackup on
table/* or * alone no longer restores (the seven item actions on the
target are needed), and a statement that spells out the server's region,
arn:aws:dynamodb:<server region>:<account>:table/* included, no longer
covers a backup ARN sent in another region.
Backup ARNs carry no resource tags: TagResource, UntagResource and
ListTagsOfResource refuse them, and authorization never reads tags stored
under one, so tagging a backup cannot satisfy a dynamodb:ResourceTag
condition.
PostgreSQL: the copy commits per batch with one multi-row INSERT per index;
the lock session is heartbeated; at most 4 restores per account and
min(16, pool_size / 2) per server, then LimitExceededException after 30 s,
with the whole restore, admission included, in a task the request waits
on; only SQLSTATE 53300 on the lock connection is LimitExceeded. The backup
row is locked FOR UPDATE while a restore registers. serve refuses a
catalog that does not record one of the binary's migration files, and
converges a pre-release 0.0.4 catalog shape with a 5 s lock timeout.
migrate uses the data database name init recorded and accepts --data-db.
SQLite: serve refuses a pre-release 0.0.4 shape; the backup_items rebuild is
named in the migrate preview and logged; the pool_size clamp is logged.
CreateBackup is admitted against the pool (pool_size / 2 - 1 at once, then
LimitExceededException after 30 s). A GSI added by UpdateTable with
throughput on a PROVISIONED table no longer makes DescribeTable fail, and
rows earlier builds wrote are read.
MongoDB: twenty concurrent restores no longer wedge the pool (the
registration hand-over held a checked-out connection while waiting for
another; both futures are polled together now); restores are admitted per
account (4) and per server (min(16, pool / 2)), LimitExceededException
after 30 s, and a registration longer than 60 s answers an internal error
while the restore continues. UpdateTable on a CREATING restore target is
refused. DeleteBackup and the sweep never release a delete claim once the
data drop may have started; a held delete claim answers BackupInUse; a
concurrent restore of one backup retries for 2 s, then BackupInUse; catalog
reads are pinned to the primary with read concern local and writes to
w: majority, and a weaker write concern in the connection string is
refused at startup; heartbeats refresh at once. The test-only protocol
model covers two restores, the delete and its stop points, and the sweep's
delete takeover.
Cassandra: migration V005 (catalog 0.0.4) stores backup definitions and
restore registrations (registered with a lightweight transaction that
requires the backup AVAILABLE and unclaimed); the copy uses the normal put path with GSIs written
synchronously; ordinary CreateTable does not name the V005 columns, so an
unmigrated catalog breaks only CreateBackup, RestoreTableFromBackup, and
DeleteBackup. The restore heartbeat stops on every exit of the copy. On an
unmigrated catalog the backup operations are a ValidationException naming
`extenddb migrate --yes`. UpdateTimeToLive on a table with an asynchronous
GSI is a ValidationException instead of a 500. init writes the
application credentials into the config, created with mode 0600. A
pre-existing defect is fixed: an on-demand GSI omitted
ProvisionedThroughput.
Catalog guards fail closed on an unparseable version. extenddb migrate:
the preview exits 3 when work is pending and 0 when up to date; unrecorded
catalog migration files are pending; admin connection errors are one
plain line; --cassandra-user/--cassandra-pass and
EXTENDDB_CASSANDRA_PASSWORD are accepted.
Docs: differences-from-dynamodb, dynamodb-limits, the admin, upgrade, and
architecture manuals, and the MongoDB design doc are updated for four
backends.
Tests: tests/test_backup_restore_fidelity.py covers the restore surface on
all four backends; devtools/run-cassandra-tests runs its large-backup tests
with 10,000 items, because 60,000 made cdrs-tokio drop its only
connection. tests/test_ttl.py waits for ENABLING to end before disabling.
Signed-off-by: Scott Robinson <robinnsc@amazon.com>
744d18a to
79c66b9
Compare
|
79c66b9 adds fixes from a second internal review of 744d18a:
Testing done now comes entirely from this head: the four wire files on release binaries for all four backends (PostgreSQL and SQLite 118/118, MongoDB and Cassandra 117 plus 1 vector skip), and the Rust gates (workspace, PostgreSQL live, MongoDB live 3/3, Cassandra lib and direct 149, clippy 1.97, MSRV 1.88). |
LeeroyHannigan
left a comment
LeeroyHannigan
left a comment
There was a problem hiding this comment.
I re-ran the three items from the last round and the four changes in this revision on all four backends and against the live service. CodeQL is green, the marker is gone, and the body sentence is right. During a catalog failure an item operation on an existing table now answers InternalServerError instead of ResourceNotFoundException, ListBackups pages across a server.region change, the name-before-backup order matches the service on every input I sent, and the region warning is in the banner.
The branch conflicts with main. The four marked hunks are signature and placement. The rebase also needs two changes that git does not flag.
- The
AuthorizationStoretest double inrequest_helpers.rsneedsfetch_user_created_atandfetch_role_created_atfrom #420, or the merged tree does not compile. - The backup branch of
evaluate_request_policiespassesRequestParams::default(). After #420 every other action evaluatesaws:RequestedRegion,aws:SecureTransport,aws:SourceIpandaws:UserAgentfrom the request, and the five backup actions would evaluate them as absent. What I ran is in the inline comment onrequest_helpers.rs.
The differences doc gained a sentence in this revision that is wrong about the service, also flagged inline.
The override refusal and the error texts stay tracked separately, as before. One more text goes on that list. A different backup restored into a name that is a CREATING restore target gets TableInUseException on both, and the service's text is Attempt to change a resource which is still in use: Table: <name> is being used.
| &action, | ||
| &resource_arn, | ||
| false, | ||
| extenddb_auth::policy::context::RequestParams::default(), |
There was a problem hiding this comment.
After a rebase onto main, this branch is the only one that evaluates no request context. #420 fills region, secure_transport, source_ip and user_agent into RequestParams in the batch branch above and the generic path below. This branch keeps default(), so DescribeBackup, DeleteBackup, RestoreTableFromBackup, ListBackups and CreateBackup evaluate aws:SourceIp, aws:SecureTransport, aws:RequestedRegion and aws:UserAgent as absent.
I merged main locally and ran an Allow of dynamodb:* on * with IpAddress aws:SourceIp 10.0.0.0/8. GetItem from 10.1.1.1 was allowed and DescribeBackup from 10.1.1.1 was AccessDeniedException. With the same four fields filled here, both were allowed. The backup_authz_tests pass None, None, false, so they do not reach this path; a test that puts one of the four keys on a backup action would.
| | Members of the `RestoreTableFromBackup` response | The `CREATING` description omits `DeletionProtectionEnabled`, the table's `ProvisionedThroughput.NumberOfDecreasesToday`, and each GSI's `IndexStatus` | The same on all four backends; `DescribeTable` reports the first two. Each GSI's `ProvisionedThroughput` keeps `NumberOfDecreasesToday`; what the service sends there was not measured | | ||
| | Restoring into a name that exists | `TableAlreadyExistsException`; into a name that is itself a `CREATING` restore target, `TableInUseException` "Table: <name> is already being restored from backup: <arn>" (measured) | `TableAlreadyExistsException` on all four backends for an existing table; `TableInUseException` with the service's text, naming the backup the existing target is being restored from, for a name that is a `CREATING` restore target (the backends tell the two apart inside the registration transaction or lightweight transaction, so two simultaneous requests cannot both pass). The existing table or target is unchanged | | ||
| | `UpdateTable` on a `CREATING` restore target | `ResourceNotFoundException` (measured on DynamoDB) | `ResourceInUseException` on all four backends, the engine's mapping of a table that is not `ACTIVE`. A plain `CreateTable` into the target's name is `ResourceInUseException` "Table already exists: <name>", the service's text for a taken name (measured on DynamoDB), without the backup ARN. The target is unchanged | | ||
| | `BackupArn` that is malformed or belongs to another account | Not an ARN of the right shape, too long (over 1024), a control character, a slash or `..` in the region: `ValidationException` "Invalid Request: BackupArn is not valid". The right shape with an id that is not a backup id (wrong length, not hex, a wildcard): `ValidationException` "Invalid Backup ARN" (both measured; which input gets which text was not captured). Surrounding whitespace, a trailing newline for one, is trimmed, and an ARN naming another region of the caller's account resolves (measured). One in another partition is `ValidationException` (measured; text not captured) | The same two texts on all four backends, split by ExtendDB's parser: a string that is not `arn:aws:dynamodb:<region>:<account>:table/<name>/backup/<id>` with a region in `[a-z0-9-]`, a digit-only account, a name in `[a-zA-Z0-9_.-]`, only printable ASCII, and at most 1024 bytes after trimming is "Invalid Request: BackupArn is not valid" (so is one in a partition other than `aws`, the only one ExtendDB issues); one of that shape whose id is neither `[0-9]{14}-[0-9a-f]{8}` nor a shape an earlier build issued (13 digits, `-`, 8 hex digits, which every backend issued before the shared id; 13 bare digits, which PostgreSQL v0.1.0 to v0.1.2 issued) is "Invalid Backup ARN"; no build has put a UUID in a backup ARN, so none resolves. Surrounding ASCII whitespace is trimmed (only a trailing newline was measured on the service; trimming all surrounding whitespace is ExtendDB's generalisation), and the region is checked for shape only: the backup is looked up under the ARN rebuilt in this deployment's `server.region` first and, when that is not found and the caller wrote a different ARN, under the ARN as written, so another region's resolves and a backup created under an earlier `server.region` stays reachable by the ARN `ListBackups` reports for it, though not by that ARN rewritten with the new region (see the upgrade manual on `server.region`). A bare 13-digit id (the shape PostgreSQL v0.1.0 to v0.1.2 issued) passes the id check, so such an ARN that names no backup is `BackupNotFoundException`, where the service answers "Invalid Backup ARN". A `server.region` outside `[a-z0-9-]` makes the server's own ARNs "not valid"; the server starts and logs a warning saying so. A well-formed ARN naming another account, of any number of digits, is `AccessDeniedException` on `DescribeBackup`, `DeleteBackup`, and `RestoreTableFromBackup` (not measured on the service). The same checks apply to `ListBackups`'s `ExclusiveStartBackupArn`, where another account's or region's cursor is instead a `ValidationException` (see `ListBackups` paging). Authorization evaluates an ARN the parser refuses, or one in another partition, against `table/*`, so a caller allowed on `table/*` gets the engine's `ValidationException`, as in dev mode, and a caller whose policy names only backup ARNs gets `AccessDeniedException`; the service's order for that case was not measured. Same on all four backends | |
There was a problem hiding this comment.
The new sentence says the service answers Invalid Backup ARN for a bare 13-digit id. I sent DescribeBackup with .../backup/1234567890123 to the service and got ValidationException with Invalid Request: BackupArn is not valid.
This branch has not been deployed
4 participants
What
Backup and restore now behave the same way on all four storage backends (PostgreSQL, SQLite, MongoDB, Cassandra), and closer to DynamoDB, on the points the round-2 and round-4 reviews measured against the service.
CreateTableInputbefore any backend sees it. It runs the multi-part-key and vector-index refusals, applies the overrides, and runsvalidate_create_tableon the result, for every backend. Backends now only create the target, register the restore, and copy. Cassandra (Feat/storage cassandra in tree #338) previously skipped the refusals and dropped the secondary indexes.RestoreTableFromBackupreturns the targetCREATINGand copies in a background task on all four backends, so a client that times out or disconnects no longer loses the restore or locks the name and the backup. Once the request has been authenticated and authorized, the engine runs the rest of it (reading the backup, validation, registering the restore) in a task of its own, so a client that disconnects before the response no longer cancels it part-way. A disconnect while the request is still being authenticated or authorized cancels it with nothing created (see Not changed here).CREATING,DeleteTableon it isResourceInUseExceptionandDeleteBackupon its source isBackupInUseException. A target whose server stopped mid-copy is removed without operator action, and each removal is logged with the table name:RestoreSummarymatches the service.RestoreDateTimeis the backup's creation time andSourceTableArnis set. No response that describes a table (CreateTable,UpdateTable,DeleteTable,DescribeTable, the restore response) carriesRestoreSummaryunless the table isCREATING; one rule inTableDescription::prepare_for_responseapplies it to all of them.DescribeBackupreportsSourceTableDetails.ProvisionedThroughput, which is required in the shape and{0, 0}for on-demand. It also reportsSourceTableFeatureDetails: GSIs, LSIs, TTL, stream, and SSE. A definition version this build does not know is described without feature details; any other failure to read the definition (a row that does not decode, a lost connection) is an internal error, as it is forRestoreTableFromBackup, instead of a description that hides it.DeleteBackupreturns the fullDescribeBackupdescription withBackupStatusDELETED, includingSourceTableDetails.ProvisionedThroughputandSourceTableFeatureDetails. It alone tolerates a definition that cannot be read (it answers{0, 0}and no feature details, and logs a warning), so a damaged backup stays deletable; this is ExtendDB's choice, stated in the docs.ListBackupshonoursLimit(1 to 100, default 100) andExclusiveStartBackupArn, and returnsLastEvaluatedBackupArn. Summaries carryTableId. It filters onBackupType(USERwhen absent, every type forALL; every ExtendDB backup isUSER) and onTimeRangeLowerBound(inclusive) andTimeRangeUpperBound(exclusive), before paging. The cursor is checked as the service checks it (measured on DynamoDB): one from another account isValidationException"Invalid Request: The supplied ExclusiveStartBackupArn does not match the account", one from another region or another table than theTableNamefilter is refused with the same shape of text, a time bound before the epoch is refused, and a lower bound above the upper bound is "Time range lower bound must be less than or equal to upper bound".BackupNameandTableNameonCreateBackup, andTableNameonListBackups, are validated as the service model says: 3 to 255 characters and[a-zA-Z0-9_.-]+, with the service's "1 validation error detected: Value '' at 'backupName' failed to satisfy constraint: ..." texts, joined under "2 validation errors detected:" when both fail (the order of the two clauses was not captured; the length clause comes first).TableNametakes the deployment's configured table-name limits. Before this a 200 KBBackupNamewas stored and returned in everyListBackupspage (a 500 on Cassandra) and a NUL byte was a 500 on PostgreSQL.CreateBackupacceptsTableNameas the table's ARN (measured on DynamoDB; botocore refuses the ARN form client-side, so only a hand-built request sends it);CreateBackupon a table that does not exist or is stillCREATINGisTableNotFoundException"Table not found: " (measured) on all four backends.TableAlreadyExistsException; into a name that is itself aCREATINGrestore target,TableInUseException"Table: is already being restored from backup: " (measured on DynamoDB). A secondRestoreTableFromBackupof a backup while a restore from it is stillCREATINGisBackupInUseException(measured). Both are decided inside the registration transaction (PostgreSQL, with the backup row lockedFOR UPDATE; SQLite), the claim (MongoDB), or the lightweight transaction (Cassandra), so two simultaneous requests cannot both pass; the name is checked before the backup, so the same backup restored again into its ownCREATINGtarget's name isTableInUseExceptionon all four. A plainCreateTableinto aCREATINGrestore target staysResourceInUseException"Table already exists: ".BackupArnandExclusiveStartBackupArnare checked as the service checks them (measured on DynamoDB): a string that is notarn:aws:dynamodb:<region>:<account>:table/<name>/backup/<id>with a region in[a-z0-9-], a digit-only account, a name in[a-zA-Z0-9_.-], over 1024 bytes, or carrying a control or non-ASCII byte isValidationException"Invalid Request: BackupArn is not valid"; a well-shaped ARN whose id is not a backup id is "Invalid Backup ARN" (which text goes with which input was not captured; the pairing is ExtendDB's). Surrounding ASCII whitespace is trimmed (the service was measured trimming a trailing newline). An ARN of the caller's account in another region resolves: the engine looks the backup up under the ARN rebuilt in the server's region first and, when the ARN as written differs, under that spelling, so backups created under an earlierserver.regionstay reachable by the ARNListBackupsreports (not by that ARN rewritten with the new region), andListBackupsaccepts such a backup's stored ARN asExclusiveStartBackupArn, so paging passes it. Aserver.regionoutside[a-z0-9-]logs a startup warning naming the consequence. The ARN is checked before the target name; another account's isAccessDeniedException; another partition's isValidationException. Before this, every malformed shape resolved toBackupNotFoundExceptionand a NUL byte was a 500 on PostgreSQL.SSESpecificationOverrideis applied.GlobalSecondaryIndexOverride,LocalSecondaryIndexOverride, andOnDemandThroughputOverrideare applied as the service applies them (measured on DynamoDB): an empty list restores no indexes of that kind, a subset restores those indexes and prunes the attribute definitions only they used, a GSI entry'sProvisionedThroughputis the restored index's throughput, a one-member or changedOnDemandThroughputOverrideis the restored table's on-demand throughput (including aPROVISIONEDbackup restored as on-demand), and the on-demand-to-PROVISIONEDroute works with per-index throughput in the list. Refused as on the service: an unknown or changed index definition (key schema or projection), a throughput member contradicting the billing mode,{0, 0}, and aPROVISIONEDrestore of a GSI without throughput, with the service's text "Invalid Request: Must override ProvisionedThroughput if BillingMode is overridden to PROVISIONED for index ": that refusal fires wheneverBillingModeOverride=PROVISIONEDis sent with a GSI and no list entry supplies its throughput, whatever the backup's billing mode or recorded throughput (Lee's measurement was on an on-demand backup; thePROVISIONED-backup case follows the service's own message and was not measured), and when a list entry omits throughput on aPROVISIONEDrestore without the override (text not measured). A GSI entry'sOnDemandThroughputis accepted and not recorded (ExtendDB keeps no per-index on-demand maximum).NonKeyAttributesare compared as a set.TableClassis no longer carried by a restore: the target isSTANDARD, as on the service (measured). The backup definition keeps the class.RestoreTableFromBackupresponse no longer carriesDeletionProtectionEnabledor the table'sProvisionedThroughput.NumberOfDecreasesToday, which the service omits there; a GSI'sNumberOfDecreasesTodayis kept (not measured).PROVISIONEDnow says what works: restore withPAY_PER_REQUEST, then switch withUpdateTable.PROVISIONEDbackup whose GSI has no provisioned throughput restores, and that GSI takes the table's throughput (orProvisionedThroughputOverride's). ExtendDB'sCreateTableaccepts such a table andUpdateTabletoPROVISIONEDproduces one; the service cannot produce such a backup, so the choice is ExtendDB's. The earlier restore-only stopgap that refused these backups is removed.CreateTableandUpdateTablenow apply the two checks the stopgap applied only at restore: a GSIProvisionedThroughputbelow 1 on aPROVISIONEDtable (and onUpdateTable's GSICreate) isValidationException, and so are more than 100 projectedNonKeyAttributesacross a table's GSIs and LSIs. Restore runs the same checks throughvalidate_create_table.CreateBackupof a table with vector indexes is refused up front. It no longer returns an AVAILABLE backup that can never be restored.BackupCreationDateTimeis that millisecond. The ids earlier builds issued (13 digits plus the hex suffix from v0.1.3 to v0.1.13; bare 13 digits from PostgreSQL v0.1.0 to v0.1.2) still resolve; they carry no time, so one that has been deleted or filtered out cannot serve as aListBackupscursor.DescribeTablereportsACTIVEsucceeds (measured on DynamoDB). The server's key-info cache no longer stores a negative entry for a name the catalog has in a non-ACTIVEstate: on a not-found answer it asks the catalog once more withDescribeTable, and hands a not-found result to the engine so a request against aCREATINGname costs exactly onetable_key_infoand onedescribe_tableload and caches nothing (a lookup that fails with a storage error is not handed on: the engine looks the name up again and reports the error, so anACTIVEtable is never answeredResourceNotFoundExceptionbecause of a transient failure) (measured on SQLite: 40 and 41 catalog loads for 40 refusedGetItems; 80 and 81 without the hand-off). Before this a read during the copy poisoned the cache and reads failed for up to 5 s afterACTIVE;CreateTabletargets had the same lag and get the same fix.pool_sizein total (capped at 16; 10 with the defaultpool_size). A restore that cannot get both within 30 s isLimitExceededException. The whole restore, admission included, runs in a task the request only waits on, so a client that disconnects during the wait does not cancel it; the account slots belong to the engine, not the process.extenddb migratefinds the data database through the nameinitrecorded in the catalog'ssettings, so a deployment initialised with--data-db <other-name>migrates without further flags, and accepts--data-dbto override it. Before, it derived the name from the catalog database name and failed with "database does not exist".LimitExceededExceptiononly when PostgreSQL refuses it with too_many_connections (SQLSTATE 53300); any other failure is an internal error.serverefuses a catalog whoseschema_historylacks one of the binary's catalog migration files; the message names the files andextenddb migrate --yes. The stored version alone does not show this case, because another migration of the same number can have stamped the same version.table_restores.source_table_arn) gets the column atservestartup, with a warning. That step waits at most 5 s for the table lock (lock_timeout) and otherwise refuses to start with the migrate advice.serveno longer drops the pre-releaserestore_date_timedefault, which a still-running earlier-build server relies on; onlymigrate --yesdoes.restore_table_to_point_in_timeis removed from the trait and from every backend.serverefuses to start a file database whose tables are in a pre-release 0.0.4 shape, and names the step andextenddb migrate --yes. The upgrade'sbackup_itemsrebuild is named in the preview with its row count and logged at start and finish. Thepool_sizeclamp is logged.CreateBackupis admitted against the pool: at mostpool_size / 2 - 1backups at once (4 with the default of 10; each holds its reader and its writer),LimitExceededExceptionafter a 30 s wait; a refused backup writes nothing. Before this, a flood ofCreateBackups took every pooled connection and every other request waited behind them.UpdateTablewithProvisionedThroughputon aPROVISIONEDtable made everyDescribeTableof that table fail (Internal("index pt: missing field NumberOfDecreasesToday")). The index's throughput is now written in the shapeDescribeTablereads, and rows earlier builds wrote are read as the same units withNumberOfDecreasesToday0, so such a table is describable again after the upgrade without touching the database file.DeleteBackup, and the sweep finishing an interrupted delete, no longer release their claim once the backup's data may have been dropped. An error or cancellation after that point leaves the backupAVAILABLEand in use until the sweep finishes the delete; before, it could be leftAVAILABLEwith no data, and a restore of it produced an empty table.DeleteBackupthat finds the backup claimed by another delete isBackupInUseException("Backup is being deleted"), where it wasBackupNotFoundException.local, and write concernmajoritywhatever the connection string says, and the$outcopies of CreateBackup and restore are written withw: majority, so the claim protocol's reads see every write the primary acknowledged and an acknowledged claim survives a failover. A connection string that sets a weaker write concern (w=0, a numeric or taggedw,journal=false) is refused at startup, as a non-primary read preference already was.BackupInUseException. Before, it failed at once withBackupNotFoundException.CREATING, and finishes the delete afterwards.RestoreTableFromBackuprequests no longer wedge the server for every later request (reproduced on the previous build with 20 restores of 20 distinct backups: 6 answered, 14 timed out after 60 s, and every later request timed out for the rest of the run; 20 restores of one backup did not wedge, because one backup's claim serializes them). The registration's hand-over leftcreateCollection's pooled connection checked out while it waited for another connection, and the Rust driver never times a checkout out (waitQueueTimeoutMSis not supported inmongodb3.8.0), so with ten such restores the pool was exhausted for good. Both futures are now polled together. Restores are admitted per account (4) and per server (min(16, pool/2), where the pool is the engine client'smaxPoolSize:max_connections, or the driver default of 10, so 5), withLimitExceededExceptionafter 30 s; a request whose registration takes longer than 60 s answers an internal error while the restore continues andDescribeTablereports its target once it exists. The whole restore, admission included, runs in a task the request only waits on. The one-restore-at-a-time exclusion on a backup is bounded by the heartbeat: a restore that stalls between its claim and its insert for longer than the stale window loses the claim to the sweep and can be joined by a second restore of the backup (documented; PostgreSQL and Cassandra have no such window).UpdateTableon aCREATINGrestore target is refused (ResourceInUseException, as on the other three backends; the service answersResourceNotFoundException). Before, it changed the target mid-restore: billing mode, a third GSI, a stream, and deletion protection were accepted and the finished table was not what the backup described.BackupInUsetexts match the other backends ("Backup is being used to restore table : ", or "... to restore another table: " while the claim is held and the target is not yet known).#[ignore]d;cargo test -p extenddb-storage-mongodb --lib protocol_model -- --include-ignoredruns them, and the module doc says when to. Its doc comment now says thatRReleaseis one atomic step for the release, the lost-claim recheck, and the abort, and lists what is not modelled (release failures, the sweep's pre-takeover target check).ProvisionedThroughput{0, 0}inDescribeTable, and a GSI added byUpdateTablewith throughput reports it.CreateTableno longer names the V005 restore columns, so an unmigrated catalog breaks onlyCreateBackup,RestoreTableFromBackup, andDeleteBackup.ValidationExceptionnamingextenddb migrate --yes, instead of an internal error; a running server notices the migration without a restart.UpdateTimeToLiveenabling TTL on a table with an asynchronously propagated GSI is aValidationExceptionnaming the table, the index, and its delay, instead of an internal error that SDKs retry (the engine now maps a backend's validation refusal in the TTL handlers).extenddb initwrites the application role's username and password into the generated config, soservestarts against aPasswordAuthenticatornode without editing it. The config file is created with mode 0600 instead of being written and then narrowed.backup_status = 'AVAILABLE' AND active_restores = null, so a second restore of a backup whose first restore isCREATINGisBackupInUseExceptionand a restore into aCREATINGrestore target isTableInUseException, as on the other backends. A restore whose registration attempts are all taken by a restore that was just starting answersBackupInUseException("another restore of this backup is starting; retry") instead of "Backup not found".TableName:ListBackupsontable/*/backup/*, ortable/<TableName>/backup/*whenTableNamefilters the list (measured on DynamoDB);DescribeBackupandDeleteBackupon the backup ARN (from the Service Authorization Reference);RestoreTableFromBackup(measured on DynamoDB) on the backup ARN and then, ontable/<TargetTableName>, the seven item actions the restore depends on (Scan,Query,UpdateItem,PutItem,GetItem,DeleteItem,BatchWriteItem), neverRestoreTableFromBackupon the target. A backup ARN is evaluated in each spelling the engine may resolve: the ARN rebuilt in the server's region from the trimmed parts first and, when the ARN as written differs (another region of the caller's account), that spelling too; both must allow (ExtendDB's choice: the service's evaluation of an ARN sent in another region was not measured). ACreateBackupwhoseTableNameis the table's ARN is evaluated on thetable/<name>it names. ABackupArnnaming another account is refused before policy evaluation. The seven target evaluations carry none ofdynamodb:LeadingKeys,dynamodb:Attributes,dynamodb:Select,dynamodb:ReturnValues,dynamodb:ReturnConsumedCapacity,dynamodb:FullTableScan, ordynamodb:EnclosingOperation, soForAllValues:*andNull: trueconditions on them pass and bareStringEquals/Boolconditions fail;dynamodb:ResourceTag/<key>is the target's. The keys the service sets there were not measured. Unit tests cover the decision table, including the foreign-account refusal, the two spellings, and theCreateBackupARN form. On the wire,tests/test_auth_permissions.py::TestBackupAuthorizationcoversListBackupsper table, the restore's seven item actions, and a tagged existing target (tag-conditioned Allow:TableAlreadyExistsException; without the tag:AccessDeniedException); no wire test sends a foreign-account ARN, a scopedDescribeBackuporDeleteBackup, or aCreateBackupwith a table ARN under a scoped policy.TagResource,UntagResource, andListTagsOfResourcerefuse a backup ARN (table/<name>/backup/<id>, andtable/<name>/backup,.../backup/or any longer path under one) withValidationException("Provided Arn is not a DynamoDB resource arn", the reply already given for a non-table ARN; the service's own reply was not measured), after the account and region checks. Authorization reads no tags for a backup ARN, so adynamodb:ResourceTag/<key>condition never matches a backup, including tags an earlier build stored under one. Before this, a caller allowedTagResourceontable/*could tag a backup and so satisfy a tag-conditioned Allow onDescribeBackup,DeleteBackup, orRestoreTableFromBackup(reproduced on the earlier build on PostgreSQL and SQLite; the wire testTestBackupResourceTagsintests/test_abac.pyfails on it and passes on this build on both).extenddb migrate) refuse a stored version they cannot parse instead of overwriting it, andserveprints the same refusal line asmigrateinstead of wrapping it as an internal error.extenddb migrate:--yesexits 3 when work is pending and 0 when up to date. It used to exit 1 with an error.--yes. It lists such files underdata migrationsascatalog <file>.--pg-userandEXTENDDB_PG_PASSWORD. Before, it readInternal("Internal(\"Cannot connect as admin: ...\")").initanddestroyprint bootstrapper errors the same way.--cassandra-userand--cassandra-pass, and reads the password fromEXTENDDB_CASSANDRA_PASSWORDwhen the flag is absent (so doesinit). Without them it could not migrate a Cassandra deployment that usesPasswordAuthenticator.extenddb-appgains anasync-traitdev-dependency..github/workflows/integration.ymladds--test backup_restoreto the PostgreSQL storage-level test step.devtools/run-cassandra-testsruns the fidelity file's large-backup tests with 10,000 items unlessEXTENDDB_TEST_BACKUP_ITEMSis set (see Not changed here).docs/differences-from-dynamodb.md: the Backup and Restore section, rewritten for four backends, with a seven-member overrides table; rows that rest on the code rather than a running server are marked. Two Secondary Indexes rows for the newCreateTable/UpdateTablechecks. New rows forBackupArnshapes, name validation, the cursor checks, reads on a restore target,UpdateTableon aCREATINGtarget, what a restore carries across, backup contents, and the authorization resources.docs/dynamodb-limits.md: the backup rows; projected attributes are now enforced.docs/technical-debt.md: that limit removed from the unenforced list.docs/manuals/07-upgrade-manual.md: parameterized backup and rollback commands, themigratecredentials and exit codes, and PostgreSQL, SQLite, Cassandra, and MongoDB subsections for 0.0.4 (includingserve's refusal of unrecorded catalog files, the 5 s lock wait, and MongoDB's read and write concern).docs/manuals/05-admin-guide.md:[storage.sqlite]; the PostgreSQL restore limits and the session-mode pooler requirement.docs/manuals/01-architecture-guide.md,docs/design/13-storage-mongodb.md(restore flow; catalog read preference and concern), and the version literals in the getting-started and install guides.Why
Round-2 review by Lee Hannigan, which requested changes. It found four blocking problems:
The review also listed wire differences measured against DynamoDB, PostgreSQL robustness issues, tenancy issues (a process-global restore limit, unvalidated stored definitions, and catalog guards that failed open), authorization of backup operations against
table/*, and the upgrade-path problems in the manual. A further review of this round found the MongoDB delete-claim release, and the shapes thatCreateTableaccepted and restore refused.Round-4 review by Lee Hannigan (sent directly) on the previous revision found 19 further problems, all addressed here: twenty concurrent MongoDB restores wedged the server; a SQLite
CreateBackupflood took the whole pool; MongoDBUpdateTablechanged aCREATINGrestore target;DescribeBackupmasked database failures; reads on a restored table kept failing for seconds afterACTIVE; PostgreSQLmigratecould not find a data database named atinit;BackupName,TableName,BackupArnand theListBackupscursor were not validated as the service validates them;CreateBackupof a missing table, a restore into aCREATINGtarget, and a second restore of a backup gave the wrong errors;DeleteBackupreturned a partial description;TableNameas an ARN was refused; index and on-demand overrides the service applies were refused, and two requests the service refuses were accepted;TableClasswas carried across; and the docs and test comments narrated the review. It also listed pre-existing issues, of which the SQLite GSI-throughput storage shape and the authorization resources ofListBackupsandRestoreTableFromBackupare fixed here (the others are under Not changed here).Open questions for the maintainers:
003_*.sqlthat writes catalog 0.0.4. Whichever merges second has to renumber to004_*.sqland 0.0.5 (and themigrations.rsassertions and the manual's section). The design in docs(backup): design revision for off-host backup and restore [backup/restore 1/4] #396 assumes this PR keeps 003 / 0.0.4 and fix(backup): report point-in-time recovery as unsupported #335 is rebased to 004 / 0.0.5, and reserves 005 / 0.0.6 for a later PR of the series; it does not mention Store U+0000 in strings on PostgreSQL and MongoDB #368. Is that the order you want, and where does Store U+0000 in strings on PostgreSQL and MongoDB #368 go?migratereports an unrecorded catalog migration file as pending even at the current version and applies it, and PostgreSQLserverefuses to start until it has, so after a renumbering the second PR's file is applied rather than skipped. The numbering itself still has to be settled.Testing done
Release binaries were built from the code in this PR, one per backend feature, and served on 127.0.0.1 against PostgreSQL 16, mongo:7 (single-node replica set), and cassandra:4.1 with
PasswordAuthenticator, all on one host. Timings compare backends; they do not predict a networked deployment. The fidelity table below ran on those binaries; the large-backup timings and the further checks after them ran on earlier builds of this branch, as each says.tests/test_backup_restore_fidelity.py,tests/test_ttl.py,tests/test_abac.py, andtests/test_auth_permissions.py, each run in full on release binaries built from this PR's head (Cassandra withindex_propagation_delay_msset to 0, as CI runs it; see #407):test_backup_restore_fidelity.py(38 collected)test_ttl.pytest_abac.pytest_auth_permissions.pyEarlier passes in this round found three defects, fixed before the run above:
test_first_read_after_active_succeedsraced theCREATINGtoACTIVEtransition on PostgreSQL (its copy of 60,000 items takes about 1.5 s; the test now accepts a read that succeeds onceDescribeTableconfirmsACTIVE); and SQLite, MongoDB, and Cassandra answeredBackupInUseExceptionwhere the service answersTableInUseExceptionfor a restore of a backup into its ownCREATINGtarget's name, because they checked the backup before the name (PostgreSQL checked the name first). All four now check the name first, whatever state the backup is in; storage-level tests on each pin the same-backup case and, on SQLite and MongoDB, a missing backup restored into the name. One Cassandra pass ran against a node that had accumulated about 1,640 tables over two days of test runs and was pausing over 80 s per young-generation GC (from the node's own log); four of its five failures were copies or teardowns that timed out, the fifth was the ordering defect. The run above used acassandra:4.1container created the same day.The large-backup tests use a 60,000-item on-demand backup of a table with only a partition key and no secondary indexes (secondary indexes are covered by the smaller tests; with GSIs on 60,000 items the fixture kept a backend's GSI queue or backfill worker busy for minutes and slowed other modules' GSI tests):
test_restore_response_arrives_before_its_copy_finishes: a client with a 10 s read timeout and no retries gets theCREATINGresponse within the timeout, the target is seenCREATINGwhile it is polled, and it reachesACTIVEwith every item. It shows the response does not wait for the copy; it does not drop the client.Measured on an earlier build of this PR:
ACTIVEafterCassandra's copy writes every item through the ordinary put path (a read and a lightweight-transaction write per item, 8 at a time, with the index writes synchronous); that is why it is several times slower than the others. A restore-only plain-insert path is storage-cassandra: restore copy writes every item through the LWT put path #404.
On Cassandra the response waits for the target's schema. With five restores at once it took 0.75 to 1.3 s on an earlier build, and inside the full Cassandra pytest suite it once exceeded 1 s, which is why the test's client timeout is 10 s; the check that the copy runs after the response is that the target is still
CREATINGwhen polled.test_restore_survives_a_client_that_disconnects_before_the_response: a client with a 1 ms read timeout sends the request and closes the connection before any response. The test accepts either outcome that leaves no half-made state: the restore was registered and finishes whole, or the request was cancelled before registration and a new restore into the same name from the same backup is accepted and finishes whole. It prints which. On all four backends the restore was registered and finished. The engine unit testa_restore_request_dropped_while_the_backup_is_read_still_restoresdrops the request future while the backup is being read and checks that the restore still runs.test_target_and_backup_are_held_while_the_copy_runs:DeleteBackup(ExtendDB only) andDeleteTableduring the copy. The status is read immediately before each probe; a probe on aCREATINGtarget must be refused, and each probe must land in the window. On all four backends both probes were refused while the target wasCREATING, and the restore then finished whole.Other new or changed tests: DescribeBackup feature details from an on-demand source and from a
PROVISIONEDsource with SSE (throughput values; theSSEDescriptionassertion is ExtendDB only); ListBackups withLimit=1over three backups (exactly one summary per page and three pages, ExtendDB only; the service's contract allows a cursor on the last page);TableAlreadyExistsException; malformed and foreignBackupArn; empty index overrides, the SSE override, and noSSEDescriptionwithout it; CreateBackup of a vector table; noRestoreSummaryonUpdateTableof anACTIVErestored table; (ExtendDB only) restore of aPROVISIONEDbackup whose GSI has no throughput;ListBackupsfiltered byBackupTypeand a time range; index overrides built fromDescribeBackup(as the console does) and an equalOnDemandThroughputOverride, both restoring; aBackupArnin another partition; and noDeletionProtectionEnabledin the restore response. The DynamoDB-branch message intest_provisioned_restore_override_requiring_gsi_override_leaves_no_targetis the service's reply as measured during review (Invalid Request: Must override ProvisionedThroughput if BillingMode is overridden to PROVISIONED for index by_gpk).tests/test_abac.py::TestBackupResourceTagsreproduces the TagResource case from the review in authenticated mode: a user allowedDescribeBackuponly ondynamodb:ResourceTag/classification = devcannot tag the backup and stays denied; it fails on the earlier build at the TagResource step, on PostgreSQL and SQLite, and a database tagged by the earlier build no longer grants access after the upgrade.tests/test_ttl.pygains a TTL-with-GSI test: it requires TTL to be enabled on PostgreSQL, SQLite, and MongoDB (passed on each, with the backend given by the newEXTENDDB_TEST_BACKENDthatdevtools/run-tests --backendexports), and on Cassandra accepts the refusal, which it got at the default delay (ValidationExceptionnamingby_gpkand 10 ms). The Cassandra refusal is pinned by the live storage-level testtest_ttl_on_a_table_with_an_async_gsi_is_refused_as_a_validation_error(refused at 50 ms; enabled once the GSI is synchronous).test_index_overrides_listing_the_backups_indexes_restore_themruns for a PROVISIONED and an on-demand backup and passed on all four backends.tests/test_ttl.py::test_disable_ttlnow waits forENABLINGto end before it disables TTL; it failed once on a loaded PostgreSQL run with "Time to live is being updated".Checks that only ExtendDB can satisfy, and assertions not measured on the service, are skipped against DynamoDB. The file was not run against DynamoDB in this round.
Against older builds: on the round-2 build (8473dd0e), on PostgreSQL, 15 of 19 earlier tests failed, among them the response test with
ReadTimeoutError. On SQLite, the round-3 build before this pass's engine fixes fails theUpdateTabletest (RestoreSummarywithRestoreInProgress: falsein the response) and the GSI-throughput test (ProvisionedThroughput must be specified for index).Further checks with built binaries, all four backends unless noted, on earlier builds of this branch (not repeated on the final binaries):
A SIGKILL mid-restore followed by an immediate restart. The target was removed after 60 s on PostgreSQL, 0.1 s after the restart on SQLite, 60 s on MongoDB, and 62 s on Cassandra. Afterwards
DeleteBackupand a new restore into the name succeeded.A writer updating an early item and then a late item during CreateBackup. 0 pairs out of order on PostgreSQL and SQLite; 16 of 66 on MongoDB and 123 of 800 on Cassandra. Neither MongoDB nor Cassandra takes a point-in-time backup; the docs say so.
Concurrent restores on PostgreSQL: with four restores running in one account, a fifth in the same account waited for a slot (3.1 s) while another account's restore answered in 0.05 s.
Upgrades from origin/main (catalog 0.0.3) with
extenddb migrate:--yesexit 0, preview again exit 0. A table leftCREATINGwas removed at startup and logged by name. Rolling back with the manual's parameterized dump and restore brought the 0.0.3 binary back. A catalog in the pre-release 0.0.4 shape was converged byserveand bymigrate --yes.ListBackupsandDescribeBackupworked;CreateBackup,RestoreTableFromBackup, andDeleteBackupansweredValidationException"Backups are unavailable until the ExtendDB catalog is migrated: it does not record catalog migration V005..." namingextenddb migrate --yes(re-measured on the final binary; an earlier build returned internal errors). Aftermigrate --yes, all of them worked.Backups taken before the upgrade restored afterwards on PostgreSQL, SQLite, and Cassandra.
PostgreSQL
servewith a debug build of this round's PostgreSQL changes: on a catalog whose ledger names another003_*.sqlat 0.0.4, it exits 1 naming003_backup_definitions.sqlandextenddb migrate --yes; aftermigrate --yesit starts. On a pre-release 0.0.4 shape with another session holding a transaction ontable_restores, it exits 1 after about 6 s withcould not lock the table within 5 s; once the lock is free it adds the column, keeps therestore_date_timedefault, and starts, andmigratethen reports only the default step.The
migrateerror text with no admin credentials and with a wrong password (PostgreSQL), and the SQLitepool_sizelog line.The docs mark which statements rest on the code or on storage-level tests rather than on a running server, and which numbers come from an earlier build.
Rust, on the final tree:
cargo test --workspacewithout the PostgreSQL storage crate and the Cassandra live test targets: all passed. extenddb 1, extenddb-app 28, extenddb-auth 150, extenddb-cache 27, extenddb-config 7, extenddb-core 514, extenddb-engine 173, extenddb-server 91, extenddb-storage 102, extenddb-storage-mongodb lib 122 (22 live and model tests ignored), itsinterpreter_selftests18 andpushdown_parity20, extenddb-storage-sqlite 136 (1 ignored); doc-tests 4.cargo test -p extenddb-storage-postgres --tests: lib 53,backup_restore30,catalog_converge5,doc_version_literals3,key_collation3,migrate_data_db_name2,put_create_race9,stream_shard_retention1,twi_conflict8,vector_control_plane42, all passed.cargo test -p extenddb-storage-mongodb --lib -- --ignored --skip protocol_model live_: 18 passed, three runs in a row under default test parallelism (about 76 s each); the protocol model's tests run with the workspace.cargo test -p extenddb-storage-cassandra --lib: 61 passed;--test direct_integration: 149 passed.test_restore_sweep_worker_sweeps_at_startup_and_stops_on_shutdownonce missed its 8 s deadline under the full parallel run on the two-day-old node (above) and passed alone; it passed in the full runs on the new node.cargo fmt --all -- --check,cargo +1.97.0 clippy --workspace --all-targets -- -D warnings, andcargo +1.88.0 check --workspace --locked --all-targets: clean. In one workspace runextenddb-cache'sstale_hit_serves_cached_and_spawns_refresh, a timing test in a crate this PR does not touch, failed once and then passed five times alone and in the full rerun.The engine, authorization, and the first round of backend fixes were each checked with a mutant (the fix reverted or broken, the new test run; every mutant was caught); of the later backend fixes, the name-before-backup order on SQLite and MongoDB and the cursor and key-info fixes were mutant-checked, the others were not. The MongoDB wedge was reproduced on the build before its fix (20 restores of 20 distinct backups: 6 answered, 14 timed out, every later request timed out) and the build with the fix answered all 20 in 1.1 s with every target
ACTIVEafter 6.9 s (both before the later admission change; the storage-level testlive_twenty_concurrent_restores_of_distinct_backups_do_not_wedge_the_poolruns in the live suite above); the SQLite backup flood was checked at storage level (16 backups, at most 4CREATINGat once, another account's reads unaffected).CI-equivalent legs (
devtools/run-tests):tests/rustintegration passed 512 on PostgreSQL, SQLite, and MongoDB, and the PostgreSQL no-vector leg passed 7, on an earlier build of this branch. The pytest legs (--pytest --comprehensive --parallel) fail exactly the tests that fail on origin/main's binaries on the same host (host Python 3.7 and botocore 1.33): PostgreSQL, SQLite, and MongoDB on the previous build of this branch, and Cassandra on the final one, with 2 workers as CI's 4-core runner gives: 6 failed against main's 6 failed and 1 error (main's extra error is a fixture failure intest_stream_table_reuse.py), plus 4 intests/pythonon both. The PR's Cassandra CI job has failed on two runs, on different tests each time. In the first,test_gsi_configured_delay_rangesaw about 1 s GSI delays; that came from the GSI worker change this revision takes out (#407). In the second, three FGAC tests' teardownDeleteTablereturned 500 because their account's keyspace was missing tables: a race onmainwhen two accounts are created at once (#408), which this PR's extra tests made more likely by changing how the parallel suite is split. Until #408 is fixed, that leg can fail intermittently.Not run: the CLI migration pytest suites (
tests/test_cli_*migrat*.py), because psycopg2 is not installed on the test host.tests/test_put_item_create_race.pyneeds Python 3.8 and was not run on the 3.7 host.Not changed here
RestoreTableFromBackupis still being authenticated or authorized (SigV4 verification and policy evaluation, including the target table's tag reads, run in the request before the engine handler) cancels the request: nothing is created and no error is reported to anyone. From the engine handler on, the request runs in a task of its own. The fidelity suite runs the server in dev mode, where these steps take almost no time, so its 1 ms disconnect test does not exercise this window.backup.rsmaps storageTransientandConnectionerrors to 500InternalServerError, where the generic mapping (for example increate_table.rs) gives 503ServiceUnavailable. Both are retried by the SDKs; worth aligning separately.UpdateTablechecks the 100 projectedNonKeyAttributestotal against a description read before, and outside, the backend's lock, so two concurrent GSICreates can each pass and together exceed it (whether a backend keeps the tableACTIVEwhile the first index isCREATINGwas not checked). A GSI stillDELETINGis also counted.UpdateTableGSICreaterefusesProvisionedThroughputbelow 1 on aPAY_PER_REQUESTtable, while{5, 5}on the same table is still accepted; the "should not be specified when BillingMode is PAY_PER_REQUEST" rule exists only in CreateTable validation. The ">= 1" rule also has three message texts.__typenamespaces forValidationExceptionandAccessDeniedExceptionstill differ from the service's.ListBackupsrequest reads all of the account's (or table's) backups and pages in memory.ProvisionedThroughput{0, 0}for an on-demand backup, which is whatDescribeBackupreports, is refused (leave the member out), as on the service. DuplicateNonKeyAttributesin an index override are accepted (compared as a set); whether the service refuses them was not measured. A GSI entry'sOnDemandThroughputis accepted and dropped:GsiInputhas no field for a per-index on-demand maximum, and adding one touches every backend.GlobalSecondaryIndexOverrideandLocalSecondaryIndexOverrideare accepted only for indexes the backup has, with their key schema and projection unchanged. The reviewer measured the service treating the override list as a redefinition of the restored index set: it accepts and restores an entry naming an index the backup did not have, and one that re-keys an existing index. ExtendDB refuses both withValidationException("is not in the backup", "KeySchema differs"), which is narrower than the service but never produces wrong data. The reviewer is tracking the widening, and the error texts that read differently from the service's while the class is right, separately from this PR./healthanswers 200 as soon as the listener is up; it does not probe the storage backend (extenddb healthcheckdocuments it as liveness, not readiness). The condition under which it answered 200 while every request hung (the MongoDB wedge) is removed, and the admission bound keeps half the pool free, but a readiness probe should still issue aDescribeTableorListTableswith a client timeout.UpdateTableon aCREATINGrestore target isResourceInUseExceptionon all four backends; the service answersResourceNotFoundException.ListBackups.TableNameas a table ARN is refused with the pattern clause whileCreateBackupaccepts the ARN form; the service's behaviour forListBackupswas not measured.backup_id_epoch_millisreads the time only from the 14-digit id; the 13-digit ids of earlier builds carry no time for theListBackupscursor placement.TagResource,UntagResource, andListTagsOfResourceare authorized ontable/*(below);aws:ResourceTag,aws:RequestTag, andaws:CurrentTimecondition keys are not supported; a write racing aDeleteTablecan answer 500; a Cassandra GSI backfill can fail with "Batch too large"; the Cassandra GSI workers underserve(storage-cassandra: asynchronous GSI writes are never applied under serve (default 10 ms delay) #407).CREATING; the backfill worker retries it every 5 s with a warning and has no give-up, and while it isCREATINGDeleteTablerefuses the target andDeleteBackuprefuses its source backup. Not reproduced (no item was found that makes a backfill batch fail persistently); the remedy is to fix the cause and let the worker finish, or to remove the target's catalog document by hand.DeleteBackupthat returned an error can still take effect later. One that failed after claiming the backup (for example on finding a restore from itCREATING) and whose claim could then not be released, neither in the request nor by the claim guard on drop, is finished by the stale-claim sweep once no restore from the backup isCREATING. If that restore is one that never becomesACTIVE(the backfill case above), the claim also blocks new restores of the backup. Not reproduced; it needs two consecutive release failures.extenddb_catalogandextenddb_dataand has no serve lock. The sweeps only act on work whose heartbeat is stale.BackupSizeBytes0.main, Cassandraservenever applies asynchronous GSI writes, and the default delay is 10 ms, so served GSIs miss every write; CI sets the delay to 0), and storage-cassandra: concurrent account creation puts one account's tables in another's keyspace #408 (onmain, concurrent Cassandra account creation can leave an account without its TTL and GSI-queue tables; this is what failed the FGAC tests in this PR's Cassandra CI run). A fix for the GSI worker lifetime was in an earlier revision of this PR and was taken out: reviving the workers exposes defects that belong in storage-cassandra: asynchronous GSI writes are never applied under serve (default 10 ms delay) #407.Unmatched stream id, thenNo nodes available in query plan!on every request until the end), and 90 tests failed and 314 errored. cdrs-tokio 9.0.2 reuses stream ids after 32,767 requests without checking that the old one has been answered, and closes the connection when a second response arrives for an id; it then did not reconnect to the single node. That reading is from the driver's code and the server log, not a reproduction in isolation.devtools/run-cassandra-testsnow setsEXTENDDB_TEST_BACKUP_ITEMSto 10,000 (unless it is already set), which keeps the copy running past the response; the driver behaviour itself is a production risk for a busy server: storage-cassandra: cdrs-tokio drops its only connection under sustained load and never reconnects #403.UpdateTimeToLivereturns an internal error when the backend's TTL change is still blocked by an expiration sweep after its retries (IndexesInUse); SDKs retry it.replication_factor = 1anddatacenter = "datacenter1", whatever--replication-factorwas given.servelogs that background workers did not drain within 5 s.CREATINGuntil the process stops.TagResource,UntagResource,ListTagsOfResource, andExportTableToPointInTimeare still authorized againsttable/*, not against theirResourceArnorTableArn, so a statement scoped to a table, stream, ortable/*/backup/*ARN (Allow or Deny) does not apply to them. The tagging actions refuse a backup ARN in the engine whatever the policy says. Tags an earlier build stored under a backup ARN are left in storage: nothing reads or lists them, andDeleteBackupdoes not remove them (from the code).docs/design/11-high-availability.mdstill suggests transaction-mode pooling for large deployments.table_restores(every DescribeTable) behind it on every server while it waits.REPEATABLE READcatalog snapshot, and so one catalog pool connection and the catalog's xmin, from the first item read until the last full batch is written.CREATINGwith no scheduled transition) and the fallback table ARN built in SQL are written in several SQL statements per backend rather than in one place.test_backup_arn_shape_and_account_are_checkedruns in dev mode, so its foreign-accountAccessDeniedExceptioncomes from the engine's check rather than the authorization path; the authorization path's foreign-account refusal is pinned only by its unit tests (backup_arn_of_another_account_is_access_denied_without_evaluation,another_accounts_backup_is_denied_and_its_tags_are_not_read);TestBackupAuthorizationcovers other cases in authenticated mode. No wire test sends an other-region backup ARN under a scoped policy; the two-spelling evaluation is pinned at the decision level.UpdateContinuousBackupsaccepts enabling point-in-time recovery and reports a 35-day restorable window, although nothing is recorded. This is pre-existing and now documented.Checklist
cargo test --workspace). Not run as one command over every crate: the workspace run without the PostgreSQL storage crate and the Cassandra live targets, and the PostgreSQL, MongoDB, and Cassandra live runs, are listed above and passed. The pytest legs are reported in Testing done.cargo fmt --check)cargo clippy -- -W clippy::pedantic).cargo +1.97.0 clippy --workspace --all-targets -- -D warningsis clean; pedantic was not re-run this round.Storagetrait, auth model, on-disk format, or public CLI surface, an RFC has been accepted or is linked below. Not met; see the next line.ADR / RFC: none accepted. This PR changes the
Storagetrait (BackupEngine) and the on-disk format (catalog 0.0.4, Cassandra V005, SQLitebackup_itemsandtable_restores).docs/rfcs/README.mdrequires an accepted RFC for both, and none exists for these changes. #396 is a design PR, not an RFC: it revisesdocs/design/15-backup-restore.mdfor off-host backup and restore, a series in which it lists this PR as PR 0, to merge first. It describes later changes toBackupEngine(moving create, describe, list, delete, and restore into the engine) and does not specify this PR's change toBackupEngine::restore_table_from_backup. An RFC is required before merge.Breaking changes
SSESpecificationOverride,GlobalSecondaryIndexOverride,LocalSecondaryIndexOverride, andOnDemandThroughputOverrideare applied as the service applies them (an empty index list restores no indexes of that kind; a subset restores those); an unknown or changed index definition, a throughput member contradicting the billing mode,{0, 0}, andBillingModeOverride=PROVISIONEDwith a GSI whose throughput no list entry supplies areValidationException;VectorIndexOverrideis aValidationException. A restore no longer carriesTableClass: the target isSTANDARD.ListBackupsrequests that setBackupTypeSYSTEMorAWS_BACKUP, or a time range, now return only the matching backups.BackupNameandTableNameonCreateBackupandListBackupsare validated (3 to 255 characters,[a-zA-Z0-9_.-]+;TableNametakes the configured table-name limits); aBackupArnorExclusiveStartBackupArnthat is not a well-formed backup ARN isValidationExceptionwhere it wasBackupNotFoundException; a cursor from another account, region, or table, a time bound before the epoch, and a lower bound above the upper bound areValidationException. An ARN of the caller's account in another region now resolves (it wasBackupNotFoundException); a statement in a policy must allow both spellings (below).CreateBackupof a table that does not exist or isCREATINGisTableNotFoundException"Table not found: " (it wasResourceNotFoundException).DescribeBackupanswers an internal error when the definition cannot be read (it answered a description without feature details).DeleteBackupreturns the full description withBackupStatusDELETED.CREATINGrestore target isTableInUseException(it wasTableAlreadyExistsException), and a second restore of a backup while a restore from it isCREATINGisBackupInUseExceptionon every backend (PostgreSQL, SQLite, and Cassandra accepted it).CreateBackupon SQLite is bounded topool_size / 2 - 1at once (4 by default), withLimitExceededExceptionafter 30 s. Restores on MongoDB are bounded to 4 per account andmin(16, pool/2)per server (5 with the driver's default pool), with the same refusal, and a registration that takes longer than 60 s answers an internal error while the restore continues.UpdateTableon a MongoDBCREATINGrestore target isResourceInUseException(it was accepted).CreateTableof aPROVISIONEDtable with a GSIProvisionedThroughputbelow 1 (including 0/0), andUpdateTableGSICreatewith one, are nowValidationException. So are aCreateTablewith more than 100 projectedNonKeyAttributesacross its indexes and anUpdateTableGSICreatethat would take the table over 100.TableAlreadyExistsException, where it wasResourceInUseException.RestoreSummaryonce the table is notCREATING(DescribeTableused to keep it withRestoreInProgress: false, andUpdateTablereturned it too).RestoreDateTimeis now the backup's creation time, not the time the restore ran.RestoreTableFromBackupreturns before the copy. On PostgreSQL and SQLite the table used to beACTIVEwhen the response arrived; now it isCREATINGuntil the copy ends, so callers must wait forACTIVE.pool_sizerestores in total (capped at 16), besides 4 per account; more wait up to 30 s and then getLimitExceededException.extenddb migrateon PostgreSQL migrates the data databaseinitrecorded (or--data-db), not one derived from the catalog database name.DeleteBackupholds isBackupInUseException, and so is aDeleteBackupof a backup another delete holds, where both wereBackupNotFoundException.DeleteBackupthat returns an internal error or is cancelled after it claimed the backup can still complete: the backup readsAVAILABLE,DeleteBackupandRestoreTableFromBackupreturnBackupInUseException, and the stale-claim sweep marks itDELETEDabout 45 to 80 s later (later still if a restore from it isCREATING). Before, such an error released the claim and left the backup available, with no data if the drop had already run. A client that treats an error fromDeleteBackupas "not deleted" is now wrong on MongoDB.ListBackupscursor once deleted or filtered out.BackupCreationDateTimehas millisecond precision.TagResource,UntagResource, andListTagsOfResourceon a backup ARN areValidationException; they used to succeed, storing and listing tags under it. Tags so stored no longer feeddynamodb:ResourceTag: a tag-conditioned Allow or Deny never matches a backup.w=0, a numeric or taggedw, orjournal=falseis refused at startup.TableName. Released builds (up to v0.1.13) evaluated aListBackupswith aTableNameontable/<TableName>, andListBackupswithout one,DescribeBackup,DeleteBackup, andRestoreTableFromBackupontable/*, each under its own action. NowListBackupsis evaluated ontable/*/backup/*ortable/<TableName>/backup/*;DescribeBackupandDeleteBackupon the backup ARN;RestoreTableFromBackupon the backup ARN and then the seven item actions (Scan,Query,UpdateItem,PutItem,GetItem,DeleteItem,BatchWriteItem) ontable/<TargetTableName>. Two kinds of policy stop working: aListBackupsgrant ontable/<name>no longer lists that table's backups (granttable/<name>/backup/*), and a grant ofRestoreTableFromBackupontable/*or*alone no longer allows a restore (the seven item actions on the target are needed;dynamodb:*has them). A statement whoseResourceis*orarn:aws:dynamodb:*:*:table/*still matches every resource these operations are now evaluated on (it must still grant the actions above); Allows and Denies on backup ARNs, which released builds never matched, now take effect. A backup ARN sent in another region of the caller's account is evaluated in both spellings (the server-region rebuild and the ARN as written), and both must allow, so a statement that spells out the server's region,arn:aws:dynamodb:<server region>:<account>:table/*included, no longer covers such a request.RestoreTableToPointInTimewithSourceTableNameis authorized against that table. ABackupArnnaming another account is refused at authorization.extenddb migrate --yesis required before serving. PostgreSQL and SQLite servers refuse an unmigrated catalog, and a PostgreSQL server also refuses a catalog whose ledger lacks one of its catalog migration files, even at the current version, and refuses to start a pre-release 0.0.4 catalog if its startup shape step cannot locktable_restoreswithin 5 s. SQLiteservealso refuses a file database in a pre-release 0.0.4 shape untilmigrate --yeshas run. A Cassandra server on an unmigrated catalog serves tables and items, butCreateBackup,RestoreTableFromBackup, andDeleteBackupfail until it is migrated. A stored catalog version that cannot be parsed is now refused rather than overwritten.Storagetrait changed.BackupEngine::restore_table_from_backuptakes a resolvedRestoreRequest,backup_restore_sourceis new, andrestore_table_to_point_in_timeis removed.BackupSummarygainstable_id.extenddb migratepreview without--yesexits 3 when work is pending; it used to exit 1. It exits 0 when everything is up to date.By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache License 2.0 and I agree to the Developer Certificate of
Origin (DCO). See CONTRIBUTING.md for details.