[robinnsc]

Backup/restore series (design: #396): #384 restore fidelity (merged) · #393 store · #394 format · #395 SnapshotEngine · #666 catalog migration framework · PR 6 the catalog index · PR 7 configuration, the store identity file and backup purge · this PR: the CreateBackup job · next: the restore job (PR 9a).
Closes #699 (part of #624).

Stacked on #672 (feat/backup-store-config, head 62d32f0f), itself on PR 6's a4ece880 and #666's d4187ea9: this PR adds nothing to the catalog schema (every column the job writes exists from PR 6's migration 5; docs/schema/*.txt are unchanged, checked against a fresh init on PostgreSQL and SQLite); it binds PR 7's store, identity file and health to the job, and PR 6's claim, row and reconciler to the duties. Where the PRs meet: the job takes PR 7's require_healthy_store before the permit and PR 6's claim_backup after the snapshot, and checks PR 7's maintenance claim once the row exists (the in-catalog paths keep checking before their row); the reconciler's cycle carries PR 7's store check and DELETING duty and PR 6's dead-restore duties beside this PR's dead-CREATING, FAILED-prefix, DELETED-prefix and tombstone duties, with the maintenance flag gating the last three; backup purge's zero-window pass on SQLite runs unchanged; the PostgreSQL snapshot slot's refusal now goes through PR 6's limit_exceeded_message; StatusFields (PR 6) gains one member every backend's set_status writes; BackupCatalog gains three methods beside PR 7's.

What

With a configured [backup.store] on PostgreSQL, SQLite and MongoDB, CreateBackup writes the backup to the store and returns it CREATING; a job of the server uploads it and marks the row AVAILABLE. Without a store, and on Cassandra always, the in-catalog create of PR 7 runs as before. The reconciler runs its full duty list. No request or response shape changes; the API-visible changes are the asynchronous create (clients poll DescribeBackup, as against the service), CreateBackup of a vector table succeeding on the store-backed path, CreateBackup refused with ValidationException while a configured store is unhealthy or its first check pending, LimitExceededException for a second CreateBackup over max_concurrent_jobs while one job runs, and a backup of a table holding an N value the wire did not validate (#618) failing on the store-backed path (CREATING, then BackupNotFound; the reason in backup list --failed and the log).

Core (crates/core). validation::validate_restorable_item(item, &RestorableKeySchema, &RestorableLimits): one walk of the item, every N and NS member parsed once on the accepting path with the wire's number rule, the nesting depth counted on the way, the key attributes (RestorableKeySchema::resolve from the key schema and attribute definitions, multi-part keys included) checked first with the format's own check_key_value (the declared type compared before the value is parsed), then the key sizes against the instance's hash and range key limits, every attribute name against max_attribute_name_bytes, then the size (RestorableLimits from &LimitsConfig); RestorableItemError { key, path, reason } with Display key {key}, attribute {path}: {reason}, every key value, name and path segment bounded on its own so a legal key appears whole whatever its bytes (a key value at SHOWN_KEY_VALUE_BYTES, 12,288 escaped bytes, six times the hash key limit; names, path segments and echoed numbers at SHOWN_VALUE_BYTES, 4096), and every such text written as a JSON string's content with U+0085, U+2028 and U+2029 escaped beside the control characters, so neither a control character nor a Unicode line separator in a tenant's item can add a line to the server's log (a reader using str.splitlines sees one line too); RestorableItemReason::{InvalidNumber, KeyShape, AttributeNameTooLong, TooLarge, TooDeep, LineTooLong}, the last carrying the cap. backup_format::encode_item_line_len_bounded(item, cap) measures an item's line through a counting writer that stops at the cap, so the feeder's measurement of an oversized item holds no line; limits::DEFAULT_MAX_ITEM_SIZE_BYTES is the feeder's threshold. backup_format::check_key_value and shown are pub. Metrics: MetricName::{BackupJobs, BackupJobDuration, BackupBytesUploaded, BackupSnapshotAge, BackupStoreErrors}, BackupJobOutcome, QuerySource::BackupJob, the recorders on MetricsCollector; /metrics emits them under those names as JSON like every other metric (the endpoint is not Prometheus exposition format), BackupJobDuration in microseconds from the snapshot's open to the job's last catalog or store call (for a FAILED job its FAILED write and delete_prefix included), BackupBytesUploaded for a job that ended AVAILABLE, WorkerErrorCount under BackupJob for a FAILED job (a cancelled one records neither worker metric).

Storage (crates/storage). BackupCatalog::claim_dead_backup(key, stale_after) -> Option<Box<dyn JobClaim>> (the reconciler's takeover of a dead CREATING row under a fresh generation, heartbeat_at untouched; None for a live owner, a fresh heartbeat, a row that is not CREATING or missing, or a claim another task holds), list_backup_rows_older_than(status, store, clock, age) with StaleClock::{Heartbeat, Completed} (the age measured by the backend on its clock; a row with completed_at unset is never older on the completed clock), sweep_restore_registrations (a default empty body), StatusFields.source_table_details_json. snapshot::paging::SNAPSHOT_FETCH_BYTES (64 MiB) with fetch_bytes(), record_page_bytes, record_rows_past_cut and the test-hooks readers and override; snapshot::source_table_schema(description, ttl) builds the manifest's source_table from a snapshot's description (round-trips through the manifest reader, absent members as explicit nulls); snapshot_conformance::run_page_bounds (a one-byte budget measures the row and proves the knob took effect, then an eight-row budget: every row once, pages at most the budget plus one row, the cut fired; a stream that has yielded more items than the table holds fails the check at once, naming the key yielded twice, so an inclusive seek fails in seconds rather than running until the hooks lock's wait gives out).

PostgreSQL (crates/storage-postgres). The cursor is replaced by keyset pages in the same REPEATABLE READ transaction: three layers (LIMIT n after the last key, the running byte sum and the row count, the rows under the budget in key order), the key columns under qualified aliases and bound as their column types, COLLATE "C" only where the probe found it, the rows pulled from the connection one at a time and parsed as they arrive (the page's text is never held whole), n 16 for the first page and then the budget divided by the largest row of the last pages that together hold at least a row bound's worth of rows (RowBoundMemory), never under 4 (MIN_NEXT_PAGE_ROWS) nor over 1000: the server serializes and spills every row the inner LIMIT reads, so a page costs it n rows of text and not the rows it returns, and sized from the average row a page of 984 small rows and 3 large ones asked the next page for 810 rows of a table that was then large rows only (5.9 GB of server temp files, 18 to 44 s for the 3 rows it returned), where the largest row asks for 4 (208 MiB, 1.1 s); a page of 27 MB rows reads one row past its cut rather than thirteen; sized from the previous page alone the bound oscillated on a table whose large rows are scattered among small ones (4 after a large row, 1000 after four small ones, the 1000-row page meeting large rows again every other page: 5,555 rows read for 1,000 kept and 13.4 GB of temp files on a 1,000-row table with 30 such rows), where the memory keeps it at 4 while large rows keep arriving within a row bound's worth of rows (1,901 read, 3.9 GB; a memory of four pages read the same 5,555 as one); the bound that remains, stated as it is: the second page, and then the page that follows a row bound's worth of small rows, reads up to n rows of up to the line cap, at most once per row bound's worth of small rows, and nothing in this form stops the server at the budget (a walk that does was measured and not taken, below); the floor's cost is 250 pages of four after every large row of a table whose large rows are rarer than one per row bound (2,585 page queries against 38 with one page of memory on a 20,000-row table with a 27 MB row at every 1,100th key, 25.6 against 25.1 s, the record); the stream ends when fewer rows than asked came back and none was cut; page_sql_for (taking the collation flag open_stream decides), MIN_PAGE_ROWS and MIN_NEXT_PAGE_ROWS exported for the tests; the slot refusal through limit_exceeded_message naming [storage.postgres] pool_size snapshot-slot. claim_dead_backup takes the row's advisory lock on a dedicated connection and then UPDATE ... WHERE status = 'CREATING' AND heartbeat_at < now() - make_interval(secs => $stale); list_backup_rows_older_than compares the named column with now(); set_status writes source_table_details_json = COALESCE($10, ...); try_claim sets tcp_keepalives_idle = 30, tcp_keepalives_interval = 10, tcp_keepalives_count = 3 (CLAIM_KEEPALIVES), each failure a warning.

SQLite (crates/storage-sqlite). claim_dead_backup is the slot rule: JobSlots::take is the liveness test and the exclusion in one step, stale_after unread, then a conditional UPDATE under BEGIN IMMEDIATE; list_backup_rows_older_than with julianday arithmetic; the byte cut in fetch_page as a row stream (the stream pulls no row past the cut; the driver's worker reads up to five ahead of it, below); backup::pool_size_for_jobs(pool_size, max_concurrent_jobs) and SqliteEngine::new_for_server: a file-backed pool is raised to 2 * (min(jobs, 16) + 1) with a warning naming both numbers so the snapshot slot refuses no job the permit admitted (10 at the defaults, the default pool; over 16 jobs the warning names the jobs the slot cap refuses, 17 to max_concurrent_jobs, pool_raise_message); the rule is stated once on SqliteConfig::pool_size. The pool's connections are built with sqlx's row_buffer_size at four (store.rs): the driver's worker runs each statement ahead of the task reading it, and its default of 50 buffered rows let a snapshot of large rows hold up to 50 lines the page never asked for; the rows ahead of the reader are five (the four the channel holds and the one the worker has stepped and copied while its send waits), 160 MiB at the line cap.

MongoDB (crates/storage-mongodb). claim_dead_backup is one find_one_and_update on status: CREATING and heartbeat_at < $$NOW - stale, the generation set through a $literal pipeline (literal_set, now one function for every pipeline write); list_backup_rows_older_than requires the clock column present ($type: "date") beside age_is_stale, since that helper reads an absent column as the epoch; set_status writes the member; the snapshot stream applies no cut (the 16 MiB reply bound is under the budget), stated in its module doc.

Cassandra (crates/storage-cassandra). claim_dead_backup: a LOCAL_SERIAL read, staleness against the coordinator clock, then UPDATE ... SET owner_generation = <fresh> IF owner_generation = <read> AND backup_status = 'CREATING' AND heartbeat_at = <read> (the heartbeat in the condition puts the liveness test inside the write), an Unknown outcome settled by a re-read; list_backup_rows_older_than from backup_index_by_status with the clock read before the scan; set_status writes the member; sweep_restore_registrations removes active_restores entries whose target is not a CREATING restore target of that backup, by LWT on the entry as read. Cassandra has no SnapshotEngine, so these run only on rows the reconciler meets.

Engine (crates/engine/src/backup/). create_backup branches on configured_store().is_some() && !legacy_create_only(): the store-backed path requires a healthy store, takes the permit, and spawns job::start through the runtime's task tracker, awaiting its one-shot reply; refuse_backup_of_vector_table and refuse_while_maintenance_held stay in front of the in-catalog create alone. job.rs: start opens the snapshot, mints the id from the open's instant, reads the TTL, builds the source_table, inserts the CREATING row through claim_backup (AlreadyExists is an internal error), checks the maintenance claim (held: the row FAILED maintenance, LimitExceededException), answers CREATING, and runs the job: writer.rs (a feeder task validating every item into a four-item channel and, for an item over the default item limit, validating it and measuring its line against MAX_LINE_BYTES on the blocking pool (the walk's cost grows with the item) through a counting writer that holds no line, a line over the cap refused through the same path with a reason naming the cap; one spawn_blocking encoder per data file writing 1 MiB chunks into a four-chunk channel that is the put's body; a write into the sink after the pipeline stopped is BrokenPipe, which the encoder does not retry, so a job cancelled mid-file ends within milliseconds rather than spinning through the drain; a file closes at the first item that takes it to or past data_file_target_bytes; KeyRangeTracker per item on a key-ordered table; the first refusal stops the encoding and the uploads but not the validation, each refused item logged with key, path and reason up to 1,000, the rest counted; a put that fails ends the feeder at once, so the FAILED write does not wait for the snapshot stream's next page; a put is started only when a file's first chunk exists and only when a heartbeat confirmed the claim within two periods, the claim's insert standing for the first beat (the watch starts from the instant taken before claim_backup, so a slow maintenance check cannot admit a first put on a stale row); one info line at the start with the table's count as the stats pass last counted it and one at the end with the items written, the bytes and the duration), then BackupDocuments::build with the stream's count, the three documents, a claim re-check, the manifest and the AVAILABLE write with the manifest's size, count and source_table; heartbeat.rs (a timer through the claim every max(heartbeat_interval_secs, 15) s; start takes the claim's instant; a lost, failed or unreturned beat raises the abort; confirms_the_claim is the pre-put check); failure.rs (write_status: each unknown outcome followed by a re-read, the write sent once more at most, and after the second write a re-read that shows the row in the target status counted as matched and anything else as matching no row; fail_row: FAILED first, delete_prefix only after it matched). A lost claim ends the job with no catalog write. KillPoint::{AfterDataFile, BeforeManifest, AfterManifest} pause the job under backup-test-hooks (EXTENDDB_BACKUP_PAUSE, released by the file EXTENDDB_BACKUP_PAUSE_RELEASE names). runtime.rs: JobSettings, with_metrics, with_shutdown, with_source, spawn_job, running_jobs, drain_jobs(bound), whose give-up line names the CREATING or FAILED rows the reconciler resolves or sweeps once their heartbeat is stale; ConfiguredStore gains bucket and key_prefix for the manifests. reconciler.rs: after the maintenance check, the dead restore jobs, the abandoned in-catalog backups and the store check, and while the store is healthy: the DELETING rows finished, dead CREATING rows taken over (also under a maintenance claim) and resolved to AVAILABLE when a complete, verified manifest with every data file at its recorded size is under the prefix, else FAILED (owner dead ...), nothing deleted there; then, without a maintenance claim: FAILED prefixes deleted once the heartbeat is stale (the same cycle's for a row it just failed: in that cycle on PostgreSQL, MongoDB and Cassandra, where the stale heartbeat made the owner dead, and once the dead owner's last beat is that old on SQLite, where the slot did), DELETED prefixes swept, tombstones past failed_row_retention_secs removed once their prefix is empty, the registrations swept; CycleReport and the cycle's log line carry the counts, a dead-CREATING resolution whose write matched no row counted as left to its owner.

Server and app. serve.rs creates the shutdown token before the runtime, gives it to the runtime as a child token with the metrics and the build's backend and version, and runs drain_jobs (30 s) after the listener stops and before drain_workers; backup_store.rs builds JobSettings from [backup] and the manifest names from the store configuration (S3: bucket and prefix; filesystem: the root label and ""), and its startup lines say CreateBackup writes to the store. crates/app/Cargo.toml and crates/server/Cargo.toml chain the backup-test-hooks feature. crates/config/src/backup.rs docs and the floor's error text name the in-catalog jobs' constant.

Harness and CI. tests/test_backup_restore_fidelity.py: wait_for_backup_available, used by _create_backup, _cleanup and every cell that reads a backup it just made; a session fixture reads /health once and checks that the server's backup_store component is the kind EXTENDDB_TEST_BACKUP_STORE names and healthy (absent for none), so the variable describes the server under test rather than being taken on trust; the vector test reads the variable (filesystem, s3, none), fails unset ("is unset"), prints which branch ran, and runs the refusal (capturing the ARN of a create accepted against expectation) or the store-backed create (CREATING, AVAILABLE with the GSI in SourceTableFeatureDetails, listed, the restore's ValidationException naming the store until PR 9a); the 24 restore cells skip on a server with a store, naming PR 9a. tests/test_auth_permissions.py: the backup fixture's cleanup waits for AVAILABLE before DeleteBackup, and the restore-authorization cell skips on a server with a store for the same reason; tests/test_abac.py's backup cell waits the same way before its DeleteBackup (deleting at once answered BackupInUseException and left an AVAILABLE backup on every store-backed run), and tests/test_authz_resource_arn.py's module fixture, whose cells deny DeleteBackup to other users, deletes its backup once AVAILABLE and its table when the module ends (it left a complete backup under the store on every store-backed run); the fidelity suite's skip of its 24 restore cells is gated on the target as well as the variable, so a stale EXTENDDB_TEST_BACKUP_STORE in a real-DynamoDB run skips nothing. tests/test_backup_job.py (new, its own servers, run by name): a second CreateBackup over max_concurrent_jobs = 1 is LimitExceededException naming the bound and a third is accepted once the first is AVAILABLE (a second accepted while the first is still CREATING fails the cell; one accepted after the first is AVAILABLE asks for more items); one cell per #618 write path (BatchWriteItem, TransactWriteItems, UpdateItem on an absent item in the key it creates, ImportTable) with three refused items: three log lines with the keys, the FAILED reason's count and first item naming #618, BackupNotFound, no manifest; a refused item with a UUID hash key and a ten-digit epoch range key named by its whole key, and the wire's reason for its value, in the log line and in backup list --failed (whose Rust Debug escapes the harness decodes); an item over max_item_size_bytes after a restart with a lower limit: a reason naming the size and the limit and not #618. tests/test_backup_store_config.py: the create refused on an unhealthy or checking store, the objects under the root and in the bucket once healthy, the start and completion log lines with their two counts. .github/workflows/integration.yml: the PostgreSQL job appends a filesystem store after init and sets EXTENDDB_TEST_BACKUP_STORE=filesystem, the SQLite job sets filesystem (the defaulted root), a new run-integration-sqlite-s3 job runs the backup tests against an S3 store on moto with s3 (and dumps moto's log on failure; its by-name step runs the store configuration suite's S3 cells and the job suite against the build's defaulted filesystem root, which the step says), and the PostgreSQL and SQLite jobs run the two store suites by name; the three start steps wait for the backup_store health component to be healthy rather than for the word anywhere in the reply, so a store that failed its check or is still checking is named at startup; the pytest lines carry -rP so a passed cell's printed branch reaches the job log; integration-mongodb.yml and integration-cassandra.yml set none; the Rust integration suite (tests/rust/src/backup_restore.rs) waits for AVAILABLE after every create that goes on to describe, delete or restore (create_available_backup, wait_for_backup_available), deletes the backups its tests make, and reads EXTENDDB_TEST_BACKUP_STORE as the Python suite does (backup_store: filesystem and s3 expect the restore's ValidationException naming the store, none the round trip, unset fails with the fixture's text), backup_arn_scoping.rs and restore_active_completeness.rs through it, since the SQLite Rust integration job's file-backed server has the defaulted root and its CreateBackup is the asynchronous one (five tests assumed the synchronous create and the suite left 12 complete backups under that root); the five Rust integration jobs set the variable (filesystem on run-rust-integration-sqlite, none on the PostgreSQL, MongoDB and Cassandra jobs); devtools/run-tests --pytest refuses when the variable is unset (its --help, CONTRIBUTING.md and the developer test guide name the variable, its three values and the refusal), naming the three values, ignores the two store suites in its line, and its health_check waits for the backup_store component to be healthy when the variable is filesystem or s3 (the server it restarts for [import] is the one the suites run against); the start steps' python3 reads curl's reply with its stderr quiet, so the seconds before the listener answers print no traceback; tests/test_backup_job.py's _started stops a server whose health wait or provisioning failed. crates/storage-sqlite/tests/backup_job_panic.rs (new; extenddb-engine and extenddb-backup-store as dev-dependencies): a job whose store put panics leaves a CREATING row that one reconciler cycle in the same process resolves to FAILED.

Docs. The differences document (the vector row, the restore-interruption row with the backup duties and the measured kill points, where backups live, backup creation with the record's durations and the CREATING ItemCount that can be 0, consistency, a new #618 row), the limits document (the SQLite pool raise and PostgreSQL's slot, the store row, a data-file row with the line cap and the file size, the contents row), the usage guide's backups row (Partial: CreateBackup to the store, RestoreTableFromBackup for in-catalog backups only this release), the upgrade manual (the asynchronous create, a store-backed backup listed, described and deleted but not restored until the restore job lands, the #618 paragraph, the rollback paragraph), the admin guide's [backup] rows (the memory figures, their scope at the default max_item_size_bytes, and MALLOC_ARENA_MAX=1 with its measured cost beside max_concurrent_jobs, the SQLite pool raise, the job's two log lines quoted whole, the /health examples' key order as the build's) and [backup.store] section, the deployment guide's service unit (MALLOC_ARENA_MAX=1 as a commented line with its condition and cost) and its SQLite sentence pointing at the admin guide's row, extenddb.sample.toml and the init template, the measurement record committed as docs/design/15-backup-restore-bench-pr8.md, design revision item 29 with the body sentences it corrects (section 3's byte budget as the three-layer page with up to n - 1 rows read past the cut, 4(e)'s unknown-outcome rule as write_status has it and its PostgreSQL liveness rule (a stale heartbeat and a free advisory lock; a stopped owner whose session lives keeps its row), 5.1 step 4's SIGSTOP outcome per backend and step 6's cancelled job completing AVAILABLE after its manifest put, 5.1 step 1's store-health check and step 2's unreadable maintenance claim, section 7 as a sizing rule scoped to the default item limit with the one-arena factor measured on both backends and the setting's request-path cost, section 9's metrics as emitted, section 10's table with PR 8's rows, the duty list's resume marked PR 9a's, the testing summary's kill points run by hand, the PR 8 acceptance's bounds and its paging cells).

Why

#396 section 5.1, 4(a) and 4(e), section 7's memory bound and pool rule, section 9, and the PR 8 section. A job without its reconciler leaves rows no one resolves, so both ship together (item 24). Decisions taken while implementing, each stated in the module doc of the code it concerns and in revision item 29: the job and the reconciler live in the engine's backup module and nothing crosses a channel between the request and the job but the response; the takeover is a method of its own, claim_dead_backup, since claim_backup inserts; the ages are backend predicates (list_backup_rows_older_than), so the engine holds no clock; the pre-put check is two heartbeat periods, since a beat sent at the tick returns up to a period later; a lost heartbeat ends the job with no catalog write, the claim connection being the one that failed; the pipeline ends its puts before the FAILED write, so the failure path awaits nothing; the FAILED-prefix duty of the same cycle deletes the prefix of a row the cycle just failed once the dead owner's beat is stale; SQLite's snapshot slot stays the backend's admission bound and the pool is raised for it; the PostgreSQL page is three layers because a window function runs before LIMIT; the vector round trip's create half lands here and its restore half with PR 9a; the restore cells of the fidelity suite skip on a server with a store until then; EXTENDDB_TEST_BACKUP_STORE is set by every CI job that runs a suite creating backups (the pytest jobs and the Rust integration jobs: filesystem on the SQLite jobs, s3 on the moto job, none elsewhere) and refused unset. Three decisions came from the measurements: section 7's memory paragraph is the sizing rule the worst-expansion runs measured (the terms at the line cap, about 490 MiB per job on PostgreSQL and 650 MiB on SQLite: the page's decoded rows, the items in the feeder, the channel and the encoder, one row's parse, the line buffer, the chunks and SQLite's five-row run-ahead, times the allocator's factor, times max_concurrent_jobs), since the earlier "budget plus one row plus its decoded form" did not describe the 880 MiB peak of 27.7 MB rows; the allocator's factor is documented rather than set from the binary: with glibc's default arenas the peak grows with the number of large rows a job reads (466, 588, 760 MiB over idle at 12, 24, 48 rows on a quiet host; 707, 700, 1,281 MiB on a loaded one) and under MALLOC_ARENA_MAX=1 it is flat (452 to 481 MiB in both series), so the admin guide carries that setting for hosts that back up large items and the service unit carries the line commented out, since one arena costs the request path 3.5 to 9.2% on SQLite and about 2% on PostgreSQL (the scans, 2.1%); and the item channel stays at four items after a one-item channel was measured 75% slower on a 1M-row backup for 50 MiB less at that peak, while the SQLite pool's row_buffer_size goes from sqlx's 50 to four, which bounded the driver's run-ahead at five rows at no cost to the 1M-row duration. The PostgreSQL page's row bound follows the largest row of the last pages rather than the average because the server serializes every row the inner LIMIT reads: the average sized a page after a mixed page at 810 large rows for the 3 it returned; the memory covers a row bound's worth of rows because one page of memory oscillated on a table whose large rows are scattered among small ones (and four pages read the same rows as one); a page query that stops at the budget on the server, a recursive walk of the primary key one index probe per row, read exactly the rows it returned (the interleaved table in 38.8 s with 2.5 GB of temp files against 54.6 s and 3.2 GB for the memory and 72.0 s and 9.2 GB for one page) and was not taken because the probe per row cost the 1M-row backup 95% of its duration (23.1 s against 11.9 s); the record carries both.

Testing done

Counts are of the whole tree on #672's head 62d32f0f; "new" is against that head.

• extenddb-core: lib 727 passed, 1 ignored (30 new: validate_restorable_item per shape, 1E126 and abc in a non-key attribute with the wire's reason text, the nested path a.b[2].c and the NS member position and member, validation past the first valid number, size at and over the limit, depth at and under the limit agreeing with validate_item_nesting_depth, a missing, mistyped and empty key attribute, abc and 1E126 in the range key as InvalidNumber and 1.0 as KeyShape, N("abc") under an S key as KeyShape, multi-part keys attribute by attribute, key attributes before the others, RestorableKeySchema::resolve's refusals, every key attribute of a legal key shown whole and only an oversized value cut, the attribute path and the echoed number bounded segment by segment, the key sizes and attribute name sizes at and over their limits, RestorableLimits from LimitsConfig, LineTooLong's text naming the cap, a control character in a name or a number escaped in the message, a legal key of control characters shown whole, the bounded line measurement stopping at the cap without a buffer; check_key_value comparing the type before parsing; the backup job recorders landing under the outcome and the store call; the three line separators escaped in a key, a name and a number; a 4097-byte key attribute name cut at SHOWN_VALUE_BYTES and a 20,000-byte list under a key attribute cut to a JSON string); data_file_line_bomb 1, data_file_nested_decode_memory 1, data_file_decode_peak 1 ignored.
• extenddb-storage: 201 (4 new: source_table_schema round-trips through ExtenddbManifest::from_json_bytes, absent members as explicit nulls, on-demand drops the throughputs, the TableClassSummary error); snapshot_conformance::run_page_bounds is run by the PostgreSQL and SQLite crates and fails in 3 s on a stream that repeats rows (an inclusive-seek mutant, which ran until another test's 600 s lock wait before).
• extenddb-engine: 274 (33 new: a tracing callsite first met by a thread without a scoped subscriber still reaching the test's LogLines (the suite's global default answers "sometimes" for every callsite, so a test running beside a log-asserting test cannot cache the job's callsites as "never"; the CI run of the previous push lost the lost-claim line that way); a 400 ms maintenance check at a 100 ms period refusing the first put as a lost claim with the row left CREATING; the store-backed create answering CREATING and its job writing the backup with the put order, the manifest's instant, count, order and key range, the AVAILABLE row and the metrics; the maintenance claim failing the row maintenance; an open that fails leaving nothing; a healthy store making the create store-backed and a vector table backed up; an unhealthy store refusing create, delete and restore unless bound to the in-catalog create; the data-file close rule at the knob and at its 32 MiB minimum; the manifest recording the snapshot's order; the validator failing the backup after the rest is validated, with and without BatchWriteItem, TransactWriteItems and other operations store N values that fail number validation as written #618; a put failure failing the row then deleting the prefix; a FAILED write that does not match making no store call; an Unknown write re-read, applied and lost; a lost, stalled or erroring heartbeat stopping the job with no put; the handle confirming the claim within two periods; cancellation running the failure path; a second create over the bound; the reconciler resolving a dead CREATING row with a manifest to AVAILABLE and one without to FAILED; the FAILED and DELETED prefixes and the tombstones on their clocks under and without the maintenance claim; a write after the stop answering BrokenPipe and an encoder closing at once after the stop; a put failure ending the job while the stream is held, the start and completion lines' counts, 1,001 refusals through the log cap, a line over the cap refused and named, a status write error leaving the manifest and making no store call, a second unknown outcome, a stopped heartbeat timer starting no put, the claim re-check before the manifest, the probe's seven incomplete shapes and a store error during the probe, a row failed by the cycle losing its prefix in that cycle, a takeover lost to another reconciler left to it, the tombstone guard (an old AVAILABLE row with an empty prefix kept) and the registrations sweep gate, a second unknown outcome whose write landed matched, a dead CREATING row whose status write errors left for the next cycle, a FAILED job's duration covering its FAILED write and deletion, an eighth incomplete-manifest shape (an unparseable manifest), the resolved row's source_table_details_json equal to the manifest's; the validate-the-rest cell fed one item per 20 ms; BackupSnapshotAge asserted recorded per confirmed beat; the probe-error cell asserting no row left to an owner).
• extenddb-server: 246 (1 new: the backup job metric names render from the store). extenddb-config 23, extenddb-app 44, extenddb 3, extenddb-auth 272, extenddb-backup-store lib 37 and conformance 123, extenddb-cache 27: unchanged.
• extenddb-storage-postgres (127.0.0.1:55432): lib 71; backup_restore 49 (4 new: claim_dead_backup taking over a stale row on the database's clock and refusing a live one, with the successor's AVAILABLE write carrying source_table_details_json; a terminated claim session reporting Unknown and the row re-read; rows older than an age listed on the named clock with the window crossed by waiting; the claim session's TCP keepalives); lib 72 (the unit cells the_row_bound_adapts_within_its_clamps, sizing a page of 984 small rows and 3 large ones from the large row (under 8, not about 800) and pinning the floor at four, and the_memory_covers_a_row_bound_of_rows); table_snapshot 89 passed, 2 ignored (4 new: pages_are_cut_at_the_byte_budget with run_page_bounds and the rows past the cut asserted exactly, 8 over 77 pages, all on the first page; the_row_bound_remembers_large_rows_among_small_ones: a 300-row table with a 27 MB row at every 33rd key streamed whole in order, 201 rows read past the cut, all on the second page, over 53 round trips (306 over 7 with one page of memory, 103 round trips with a floor of 2); the_page_query_is_an_index_scan_on_every_key_shape: EXPLAIN of the first page and a page after a key at n 16 and 1000 on 20,000-row tables of the six key shapes (hash only, S, N and B range keys, a multi-part range key, a legacy table recollated to the database collation), Limit over one Index Scan on the primary key with no sort on each, then each table streamed whole in 21 round trips; the_last_page_sorts_the_rows_after_the_seek: the index scan ten rows from the end of the 100k table and the bitmap scan with a Sort near the end of a 1,060-row table, the sorted page's rows in key order); the other 23 binaries as before.
• extenddb-storage-sqlite: lib 233 passed, 1 ignored (4 new: the takeover of a row whose slot no task holds, the listing on the named clock, the pool raise and its slots, the pool-raise warning's text at 8 and 20 jobs); backup_job_panic 1 (new: a job whose store put panics leaves a CREATING row one reconciler cycle in the same process resolves to FAILED).
• extenddb-storage-mongodb (replica set at 127.0.0.1:27117): lib 142 passed, 33 ignored, interpreter_selftests 18, pushdown_parity 20 (2 new unit: the $literal set, the older-than filter's one column); -- --ignored live_ 33 (2 new: the takeover on the server's clock with the fence on the old claim's write, the listing on the named clock; the two takeover cells' window is 3 s (was 1 s), the run 4 s longer), run twice green (42 and 69 s); two earlier runs under the host's 1M-item load failed 5 and 3 restore cells on their 90 s ACTIVE wait and the takeover cell on its 1 s window, and passed when rerun (.review/logs/pr8/stage3/verify/07-mongo-live-{1,2,3,4}.log).
• extenddb-storage-cassandra (127.0.0.1:9042, one binary at a time): lib 78; direct_integration 198 with --features test-hooks (3 new: the takeover with the heartbeat in the LWT's condition, the listing on the coordinator's clock, the registrations sweep keeping the live entry), 2,173 s; two earlier runs stalled in transact_write_stream_images while the container ran CMS collections every 2.5 s with its old generation at 939 MB of 1 GB, 165 extenddb_ttl_test_account_* keyspaces left by the TTL suite's runs being the heap (LESSONS 68); dropping them let the third run finish; ttl_integration 26 passed, 1 failed, 1 ignored: test_ttl_enable_rejects_asynchronous_gsi is the pre-existing failure PR 6 and PR 7 recorded; the file is not touched here.
• Wire, the way CI's jobs run it (devtools/run-tests --extenddb --pytest --comprehensive --parallel --filter "not import_export", LESSONS 84): on the PostgreSQL release build with a filesystem store appended after init and EXTENDDB_TEST_BACKUP_STORE=filesystem: 1439 passed, 52 skipped, 6 failed; with kind = "disabled" and none: 1464 passed, 27 skipped, 6 failed; on the SQLite release build with the defaulted root and filesystem: 1446 passed, 45 skipped, 6 failed; the comprehensive suite 336 passed, 4 failed on each. (These three full runs are on the binaries before the sink's BrokenPipe change and the SQLite row buffer, neither of which a wire test reaches; the backup selection and the two store suites were rerun on the final binaries.) The ten failures are the same on every run and on 62d32f0f: the host's botocore (Python 3.7) refuses ReturnValuesOnConditionCheckFailure (test_conditional_writes.py::TestConditionalPutItem, 4 cells; tests/python/test_item_edge_cases.py, the 4 rv_on_ccf cells) and OnDemandThroughput (test_config_fields.py::TestOnDemandThroughput, 2 cells) as unknown parameters before a request is sent; none is a backup test. The skipped cells include the 24 restore cells of test_backup_restore_fidelity.py on the servers with a store (PR 9a) and test_auth_permissions.py's restore-authorization cell for the same reason; test_create_backup_of_a_vector_table_is_refused ran its filesystem branch there and its none branch on the PostgreSQL run without a store. The S3 job's selection (--pytest --parallel --filter "backup") on the SQLite build against an S3 store on moto (127.0.0.1:9200) with s3: 27 passed, 28 skipped; the same selection on the SQLite build with the defaulted root: 27 passed, 28 skipped.
• tests/rust, the Rust integration suite as the jobs run it (devtools/run-tests --extenddb --rust-integration --release): the whole suite on the SQLite release build with the defaulted root and EXTENDDB_TEST_BACKUP_STORE=filesystem, 512 passed, and afterwards no complete backup under the root (find <root> -name extenddb-manifest.json finds none; extenddb-store.json alone); the backup and restore_active_completeness filters on the PostgreSQL release build without a store and none: 18 and 2 passed, the restore round trip taken. With the variable unset the restore cell fails with EXTENDDB_TEST_BACKUP_STORE is unset; set it to one of ["filesystem", "s3", "none"] ..., and with none against the filesystem server it fails on the store's ValidationException.
• tests/test_backup_store_config.py and tests/test_backup_job.py (7 cells) by name: on the PostgreSQL release build 10 passed (the SQLite class's 11 cells skipped), on the SQLite release build 18 passed, 3 skipped (the PostgreSQL class's 3) with moto reachable at 127.0.0.1:9200 for the S3 cells (the S3 job's step), and 17 passed, 4 skipped (the S3 endpoint cell too) without the endpoint (the SQLite job's step). The fidelity, ABAC and auth-permissions suites against a server of each release build with a filesystem store and EXTENDDB_TEST_BACKUP_STORE=filesystem: 144 passed, 25 skipped on each, the fixture printing backup store under test: {'kind': 'filesystem', 'status': 'healthy'} and the vector cell its filesystem branch; the ABAC cell's backup deleted.
• Kill points (kill-points.sh, a server built with --features extenddb-app/backup-test-hooks, reconcile_interval_secs = 2, stale_job_after_secs = 45, a 100k-item composite-key table with two GSIs; the SIGTERM point on a 300k-item table without GSIs), on SQLite with the defaulted root, on PostgreSQL with a filesystem store, and on the SQLite build with an S3 store on moto: after_data_file: after the SIGKILL the prefix holds data/000001.json.gz and no manifest, the row CREATING; after the restart the row is FAILED owner dead ... after 0 s on SQLite and the S3 store (the slot rule, the first cycle) and 46 s on PostgreSQL (the 45 s window), DescribeBackup BackupNotFound, the prefix deleted 45 s after the kill on SQLite and S3 (the dead owner's heartbeat stale) and in the same cycle on PostgreSQL; before_manifest (run on SQLite and PostgreSQL): the data file and the three documents (manifest-files.json, manifest-summary.json, extenddb-manifest.sha256) under the prefix after the kill and no manifest, the row CREATING, then FAILED owner dead ... after 0 s on SQLite and 47 s on PostgreSQL, the prefix deleted 46 s after the kill on SQLite and 48 s on PostgreSQL (its cycle), so a prefix without extenddb-manifest.json is never resolved AVAILABLE whatever else is under it; after_manifest: the five objects under the prefix after the kill, the row AVAILABLE after 0 s on SQLite and S3 and 47 to 48 s on PostgreSQL with the manifest's size (847,689) and count (100,000), the prefix whole, DescribeBackup AVAILABLE; sigterm: waiting up to 30 s for 1 backup job(s) to end, FAILED (shutdown); its objects were deleted within 15 ms, drained, the row FAILED shutdown, the prefix empty, the process gone 55 ms after the signal (polled at 50 ms; the first record's "5 s" was the script's own sleep), and on the S3 store list-multipart-uploads empty. The first SIGTERM runs on SQLite found the job spinning through the whole 30 s drain with its row left CREATING: the chunk sink answered the encoder's write after the stop with Interrupted, which flate2's writers retry; it answers BrokenPipe now (writer.rs, two unit tests), and the three stores pass on the rebuilt binary (.review/logs/pr8/stage3/kill/).
• Memory (bench-rss.sh, the release binary under a VmHWM sampler at 200 ms, restarted after each load so the high-water mark covers the backup alone; 16 cores, 30 GiB, PostgreSQL 16.15, bundled SQLite 3.46 via libsqlite3-sys 0.30.1, MongoDB 7.0.43; BENCH-pr8.md has every figure and caveat): 100k and 1M items of a composite-key table with two GSIs: peak over idle 11.3 and 20.6 MiB on PostgreSQL (+9.5 MiB), 45.0 and 50.5 MiB on SQLite (+7.9 MiB; the 45 is the pool's SQLite page caches), 9.6 MiB at 100k and 15.5 MiB at 500k on MongoDB (+4.8 MiB; the 1M load would have taken close to an hour through the wire, so 500k stands in its place), each under the 112 MiB bound. Durations: 1.85 s and 36.1 s on PostgreSQL (a loaded host), 1.44 s and 12.45 s on SQLite, 1.85 s and 8.81 s (500k) on MongoDB. The incompressible run (64 MiB of random B attributes, a 68 MB backup gzip did not shrink): 120.5, 74.8 and 90.2 MiB over idle, so the encoder streamed the file through its 4 MiB chunk window. The worst-expansion run (1,000 small rows then 12 of feat(core): backup format v1: streaming data files, sidecar manifest hash, typed backup ARN [backup/restore 3/4] #394's 27.7 MB worst lines, three to a page) at the first head: 880 MiB over idle on PostgreSQL, 582 to 762 MiB on SQLite across runs, 474 MiB on MongoDB with 13.5 MB lines (its 16 MiB document cap refuses the full shape). These peaks are not the design's earlier formula ("the budget plus one row of text, plus its decoded form"): the page's decoded rows, the items in the feeder, the four-item channel and the encoder, one row's parse through the deserializer's intermediate value, the line buffer, the chunks and on SQLite the driver's five-row run-ahead are terms of their own (545 MiB for this shape on SQLite, about 490 MiB per job on PostgreSQL and 650 MiB on SQLite at the 32 MiB line cap), and what glibc's allocator keeps on top of them depends on how many large rows the job reads: the series at 12, 24 and 48 such rows on one SQLite binary peaked at 466, 588 and 760 MiB over idle on a quiet host and 707, 700 and 1,281 MiB on the final build with the kill points running beside it (0.86 to 2.35 of the terms, growing with the rows), and under MALLOC_ARENA_MAX=1 at 481, 452, 480 and 452, 480, 455 MiB (0.83 to 0.88 of the terms, flat), so section 7 states the terms as a sizing rule (the terms at the line cap, times 1.2 under one arena, times max_concurrent_jobs, at the default max_item_size_bytes), the admin guide carries MALLOC_ARENA_MAX=1 for hosts that back up large items and the service unit the line commented out, and the acceptance bounds the one-arena peak at 1.2 times the terms and records the default-arena peak beside it; the record is docs/design/15-backup-restore-bench-pr8.md. On PostgreSQL the build with the row bound from the largest row and the rows parsed as they arrive took the worst shape to a VmHWM of 790 MiB in 16.5 s against 905 MiB in 18.2 s, its 1M run +4.0 MiB over the 100k run (13.3 s), and the rows read past the cut in the paging test are 8 over 77 pages, all on the first page (602 at the head); the same shape under MALLOC_ARENA_MAX=1 on the final PostgreSQL build peaked at 340 MiB VmHWM against 750 to 788 MiB under the default arenas (0.82 of the terms at that row on PostgreSQL, 413 MiB), so the one-arena factor is measured on both backends. The setting's cost on the request path (a 50k-item table, one boto3 client per thread, three alternating repeats of 20 s per operation): GetItem −3.8%, Query −3.5%, Scan −6.9%, paged Scan −9.2% on SQLite; +1.1%, −0.6%, −2.1%, −2.1% on PostgreSQL. The row bound's memory on the release binary (two 20,000-row tables with 30 worst-expansion items, scattered among the first 1,000 keys and grouped after them, both binaries back to back): the interleaved table's backup 54.6 s with 3.18 GB of server temp files against 72.0 s and 9.21 GB with one page of memory, the grouped table 38.9 against 39.8 s and 2.85 GB both; the 1M run 13.7 against 12.1 s (VmHWM 36.7 against 36.3 MiB, +4.1 MiB over the 100k run; the memory changes no page of a small-row table, and the tree's binary carries the validator's and feeder's changes of the same day). The server-side walk that was measured and not taken: 38.8 s and 2.51 GB interleaved, 38.0 s and 2.51 GB grouped, 23.07 s against 11.85 s for the 1M run, 13.4 s and 658 MiB against 17.2 s and 553 MiB for the worst shape. A one-item channel was measured and not taken: 50 MiB less at the worst-shape peak for a 1M-row backup 75% slower (12.4 s to 21.6 s, both orders). The SQLite pool's row_buffer_size is set to four (store.rs; sqlx's default of 50 let the driver's worker read every row of a 24-row worst-shape table ahead of the first page, 844 MiB over idle against 696 with four, back to back; a 1M-row backup took 11.4 s with either).

cargo fmt --all -- --check
cargo +1.97.0 clippy --locked --workspace --all-targets -- -D warnings
cargo +1.97.0 clippy --locked --workspace --all-targets --features extenddb-engine/backup-test-hooks -- -D warnings
cargo +1.97.0 clippy --locked -p extenddb-storage-cassandra --all-targets --features test-hooks -- -D warnings
CARGO_TARGET_DIR=target/t188 cargo +1.88.0 check --locked --workspace --all-targets
cargo doc --no-deps --workspace
cargo check --locked -p extenddb-storage --lib
devtools/generate-software-license-notices --check [--backend mongodb|cassandra]; devtools/generate-dev-license-notices --check
EXTENDDB_TEST_PG_REQUIRED=1 EXTENDDB_TEST_S3_REQUIRED=1 ... cargo test --locked --workspace --no-fail-fast --exclude extenddb-storage-cassandra --exclude extenddb-storage-mongodb
EXTENDDB_TEST_MONGODB_REQUIRED=1 cargo test --locked -p extenddb-storage-mongodb -- --skip a_standalone_server_is_unsupported; ... --lib -- --ignored live_ (twice)
EXTENDDB_TEST_CASSANDRA=required cargo test --locked -p extenddb-storage-cassandra --lib | --features test-hooks --test direct_integration | --test ttl_integration -- --test-threads=1

Not changed here

• The restore of a store-backed backup (PR 9a): RestoreTableFromBackup of a row of the configured store answers ValidationException "Backup is in a backup store; this release restores in-catalog backups only"; the fidelity suite's restore cells skip on a server with a store and come back with it; the vector wire test asserts that refusal where PR 9a asserts the round trip.
• The spool sweep and the restore_progress sweep (PR 9a adds the records they read); the duty that removed a CREATING restore target with no job row is withdrawn (design item 29).
• backup sync and its options, remove-target, migrate-legacy (PR 10); export and import through the store (PR 13); the legacy reader's removal (PR 14).
• The restore metrics of section 9 (restore_jobs_total, restore_job_duration_seconds) are PR 9a's.
• The in-catalog create (no store, Cassandra) is unchanged: synchronous, vector tables refused, the deprecation warning on each call.
• tests/test_backup_job.py runs on PostgreSQL and SQLite builds; its deployment fixture has no MongoDB branch and no CI job runs the store-backed create on MongoDB (the engine tests and the 100k run cover it), which the design's acceptance now says.
• The PostgreSQL page's plan near the end of a small table is the planner's (a bitmap scan and a Sort of the rows after the seek, pinned by a test and stated in the module doc); the query is not changed to force the index scan. A page query that stops at the budget on the server (a recursive walk of the primary key, one index probe per row) was measured and not taken: it reads no row past the cut on any shape and costs the 1M-row backup 95% of its duration; the record carries its figures, and the window form's bound is stated as it is.

Checklist

• I have read CONTRIBUTING.md
• All tests pass (cargo test --workspace), the pre-existing Cassandra TTL failure excepted
• Code is formatted (cargo fmt --check)
• Clippy is clean (cargo clippy --workspace --all-targets -- -D warnings)
• I have added or updated tests for new functionality
• I have updated documentation if behavior changed
• Breaking changes are noted below (if any)
• If this changes the wire protocol, Storage trait, auth model, on-disk format, or public CLI surface, an RFC has been accepted or is linked below. Otherwise, an ADR captures the decision (link below).

ADR / RFC: #396, sections 4(a), 4(e), 5.1, 7, 9 and the PR 8 section, revision item 29. Storage trait: BackupCatalog gains claim_dead_backup, list_backup_rows_older_than and sweep_restore_registrations (the last with a default body); StatusFields gains source_table_details_json; SnapshotEngine is unchanged. No wire protocol or on-disk format change (the data files and manifests are #394's).

Breaking changes

• With a configured [backup.store] on PostgreSQL, SQLite and MongoDB, CreateBackup returns the backup CREATING and it becomes AVAILABLE once the upload ends, as on the service; a client that deletes a backup right after creating it must poll DescribeBackup first (BackupInUseException otherwise). On file-backed SQLite the defaulted root makes this the default behaviour. The restore of a store-backed backup is a later release's (PR 9a): until then RestoreTableFromBackup of one answers ValidationException, so a deployment that needs to restore the backups it takes this release stays on the in-catalog format (kind = "disabled"); the usage guide's backups row and the upgrade manual say so.
• CreateBackup is refused with ValidationException naming the store's kind while a configured store is unhealthy or its first check is pending (PR 7 refused only the restore and delete).
• CreateBackup of a table with vector indexes succeeds on the store-backed path (refused as before without a store).
• A backup of a table holding an N value the wire did not validate (BatchWriteItem, TransactWriteItems and other operations store N values that fail number validation as written #618) fails on the store-backed path where the previous release backed it up; the reason reaches the operator through extenddb backup list --failed and the log.
• On SQLite a server raises [storage.sqlite] pool_size to 2 * (max_concurrent_jobs + 1) with a warning when it is smaller (no change at the defaults).
• Out-of-tree storage backends must implement claim_dead_backup and list_backup_rows_older_than, write StatusFields.source_table_details_json in JobClaim::set_status, and may override sweep_restore_registrations.
• devtools/run-tests --extenddb --pytest requires EXTENDDB_TEST_BACKUP_STORE (filesystem, s3 or none).

---

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache License 2.0 and I agree to the Developer Certificate of Origin (DCO). See CONTRIBUTING.md for details.