From a3e6c129562e9604af07bb0718b30031b8c0d19c Mon Sep 17 00:00:00 2001 From: Scott Robinson Date: Tue, 3 Nov 2026 08:04:26 +0000 Subject: [PATCH] feat(backup): RestoreTableFromBackup job and cross-backend matrix With a configured `[backup.store]`, RestoreTableFromBackup of a store-backed backup is the restore job of the design's 5.3 on PostgreSQL, SQLite, MongoDB and Cassandra: the request checks the `backup_index` row and the ARN, requires a healthy store, takes a job permit, resolves the key schema, and spawns a detached task that registers the CREATING target and its RUNNING restore record under the backend's own owner generation (`claim_restore`, atomically with the backup row's lock the in-use and delete checks take), answers CREATING with a RestoreSummary through a one-shot channel, and then runs the job: the manifest and sidecar verified (each read bounded, 4 KiB and 16 MiB), every data file downloaded and its digests checked before any is loaded (spooled under `spool_path` for an S3 store, the spooled manifest and sidecar written `.partial`, synced and renamed like the files; read twice in place for a filesystem store), each file applied by `bulk_load` with its secondary- and vector-index rows and its progress recorded in the same write, the target counted against the manifest's item total, and the ACTIVE flip. A backup restores into the backend that took it (another backend once a later release's `backup sync --adopt-store` has indexed it); a vector backup restores into PostgreSQL or SQLite and is refused before any table into MongoDB or Cassandra; a backup of a multi-part-key table restores with its item set; the restored table's size is the source table's. Progress is recorded inside `bulk_load`, in the write that applied the items: a `restore_progress` row in the file's data transaction on PostgreSQL (data database, migration 6; a plain INSERT for a file applied from its start, an upsert for one resumed past it), a `tables`-row update per batch on SQLite and Cassandra, a `restore_job.file_index` update per file on MongoDB. The memory bounds are in bytes of text: the load's chunk closes at 32 MiB of data-file line text, the SQLite and Cassandra batch at 32 MiB of serialized item text (10,000 items when smaller), PostgreSQL's at 4 MiB per INSERT. A dead owner's restore is resumed, not discarded: the reconciler of any instance with the configured store takes the record over once its heartbeat is stale (`claim_restore_dead`, on SQLite at once after a restart by the record's slot, on PostgreSQL once the owner's claim session has ended), stamping a fresh heartbeat in the same write so a second taker finds the record live, and resumes from the recorded progress, re-downloading to its own spool when the dead owner's `spool_host` differs; the resumed job's first confirmed beat is the takeover's return, and a fresh job's is the one beat it sends right after `claim_restore` returns and before the reply (on PostgreSQL `now()` is the transaction's start, so a registration stalled on the `backup_index` lock or the DDL would otherwise commit a stale heartbeat; a first beat that finds the claim lost fails the record and answers InternalServerError). Once the server's shutdown has begun the reconciler takes nothing over and writes no FAILED, since a resume spawned on the way out would stop at once and leave the record a second stale window. A resume with no progress into a non-empty target is FAILED `target_not_empty` and left for `extenddb backup remove-target --account `, which ships here (pulled forward from PR 10), under a maintenance claim as well, where the reconciler had failed such a record `OWNER_DEAD` and removed its target: on MongoDB and Cassandra, which record progress after the write, an ordinary crash before the first progress record (the whole first data file on MongoDB, the first 10,000 items or 32 MiB on Cassandra) ends in that state, and nothing else removes a target DeleteTable refuses while the backup it holds in use refuses DeleteBackup and a second restore. The command takes the maintenance claim, removes a FAILED record's target whatever the reason, a RUNNING record's dead target (dead by the backend's `list_dead_restore_jobs` on the database's clock, never by the command host's) or a legacy record's target with its record, and refuses a live RUNNING restore, a DONE restore and a table no record names; it opens the catalog without creating the default root or an identity file, and on PostgreSQL its serving registration is named `extenddb backup remove-target` and its refusals of a catalog under migration or behind the binary exit 6 and 3, as `serve`'s do. `backup purge` on SQLite runs its own reconciler cycle as the claim's holder (`ReconcilerState::for_maintenance_holder`), so a dead store-backed restore it meets is failed and its target removed, never resumed inside the command. The feeder checks the claim at every chunk boundary; MongoDB and Cassandra read the record before each write batch. A duplicate key is caught by the base table's plain INSERT on PostgreSQL and SQLite, by the unique `_id` on MongoDB (suppressed only on the one unrecorded file a resume re-applies, `BulkLoadFile.reapplying`), and by the final count on Cassandra, whose upserts collapse it; a key-ordered file that leaves its `key_range` fails the file. An `N` sort key past Decimal128 fails a MongoDB restore with the write path's text. MongoDB's `bulk_load` holds the GSI backfill test gate (a `test-hooks` build only) before the first insert of the restore's first file, as the in-catalog copy does before its `$out`, so the integration test that no index completes before the base load runs on the store path. MongoDB's snapshot open, which a store-backed CreateBackup takes, waits up to 2 s for the majority-committed snapshot to show a table a `local` read shows ACTIVE, so a CreateBackup sent as soon as DescribeTable says ACTIVE is not refused as a missing table. A `finish_restore` error is an unknown outcome re-read through the claim (DONE under the job's generation is the restore complete, RUNNING one more write, anything else the DONE write that matched no record); PostgreSQL's post-commit progress delete is logged and swept, never the restore's failure. An unknown outcome of Cassandra's progress lightweight transaction is re-read at LOCAL_SERIAL (applied, one retry, or the lost claim; a second unknown outcome fails the restore), and a Cassandra registration whose record a reconciler took over during its DDL removes nothing and answers the lost claim. PostgreSQL's `remove_restore_target` drops a vector index's `_ddb_vec_` relation with the data table. The lost-claim stop line names the heartbeat timer's reason. Fixes #742: the Cassandra write path put an index row's DELETE and its INSERT in one LOGGED BATCH, whose one timestamp let the tombstone win, so a PutItem or UpdateItem of an item whose index keys did not change deleted its GSI and LSI rows, and a resumed restore's re-applied batch reached ACTIVE with index rows missing; one shared writer now skips the DELETE when the old and new items give the same index row. Two concurrent registrations of one backup leave one target on Cassandra as on MongoDB (the registration LWT's `active_restores = null` is the fence). The download retry bound (15 min) counts from the job's first attempt. `.verified` markers hold the file's digest; a stale spool entry is removed before use; a job whose record became another's leaves its entry to the sweep, which lists the spool before the records so a restore registered between the two reads keeps its entry. A restore stopped by shutdown leaves its row RUNNING with its spool and progress and writes no status; a lost claim or a DONE write that matched no record ends the job with no table access. Metrics: `RestoreJobs`, `RestoreJobDuration`, `RestoreBytesDownloaded`, `BackupStoreErrors{get}` on every `get` the store failed. The feature chain `extenddb-app/backup-test-hooks` now pulls `extenddb-storage/test-hooks` so the backends' restore pause points compile into a hooks build. The cross-backend matrix is a committed `devtools/backup-restore-matrix` and a nightly `.github/workflows/matrix-nightly.yml`: each cell copies a source's backup prefix under a root the target owns and seeds the target's `backup_index` row as `backup sync --adopt-store` will (PR 10), so a cell proves the restore and not the adoption. Its twelve plain, four duplicate-key and eight vector cells passed on the final binaries; the nightly installs `postgresql-client` for the host `psql` the script's PostgreSQL seed and teardown call and checks it and the two drivers before the cells, and has had no run yet (`workflow_dispatch` needs the file on the default branch). The restore kill points ran on every backend they apply to, each ACTIVE outcome asserting the base table's and the GSI's counts, the two first-reach points ending in `target_not_empty` and then `remove-target`. Harness and CI: the 24 restore cells of `tests/test_backup_restore_fidelity.py` run on a server with a store, `tests/test_auth_permissions.py`'s restore cell runs, `tests/rust`'s backup suite restores the store's backups, `tests/test_backup_job.py` gains the tamper, SIGTERM, store outage, duplicate-key, key-range and multi-part-key cells on both the PostgreSQL and SQLite deployments and the `remove-target` cell on SQLite; `tests/test_mongodb_backup_restore.py` gains the vector-manifest refusal and the Decimal128 rules, `devtools/run-mongodb-tests` serves a filesystem store (or none, with `--backup-store none`, for the legacy LSI fallback cell) and starts from an empty root, and the PostgreSQL CI job starts moto for the outage cell; the nightly's cache step saves only on main. The replaced-file engine test runs at a 1 s heartbeat (its hashing pass shares the test's one runtime thread with the timer, and under load outlasted two 50 ms periods), `Progress::from_records` has a cell for a record and a progress row naming one file, the in-memory SQLite cell restores its store-backed backup, the SQLite job suite gains the purge cell above and the store-config suite a `remove-target` cell that leaves no default root, and the PostgreSQL exit-code cell covers `remove-target`. The MongoDB `--ignored live_` cells pass with `--test-threads=1`; in parallel three legacy copy cells can time out behind the backfill worker. Docs: the differences, limits, usage and upgrade documents lose the "restore is a later release's" refusal, the admin guide gains the spool's contents, the restore's measured memory with the shape and input order it was measured on, the two-GSI and the unordered-backup figures beside it (`docs/design/15-backup-restore-bench-pr9.md`; a table with indexes costs SQLite's page cache above the no-index figure, an unordered backup 1.5 to 1.8 times the key-ordered figure, and the final COUNT(*) a page cache bounded by `cache_size`, the same at 1M and 3M rows) and the `remove-target` command, the differences document labels its restore timings as the in-catalog restore's beside the store-backed ones, the test guide and CONTRIBUTING name the job suite and the matrix, and design revision item 30 records PR 9 as implemented. The admin guide's `stale_job_after_secs` row and purge paragraph and the upgrade manual's SQLite startup sentence say a dead store-backed restore is resumed by a server and failed by a purge; the upgrade manual says migration 6 is declared breaking on PostgreSQL and does not apply on SQLite; design 4(e) and the reconciler's module doc say the jobs and the reconciler are not gated by the PostgreSQL serving registration (#755); the heartbeat and SQLite restore-engine module docs say what holds a SQLite beat; and the PR 9a load paragraph gives PostgreSQL's batch as 500 rows or 4 MiB. Co-authored-by: Lee Signed-off-by: Scott Robinson --- .github/workflows/integration-mongodb.yml | 14 +- .github/workflows/integration.yml | 35 +- .github/workflows/matrix-nightly.yml | 184 + CONTRIBUTING.md | 17 + crates/app/src/cmd_backup.rs | 249 +- crates/app/src/lib.rs | 2 +- crates/config/src/backup.rs | 12 +- crates/core/src/backup_format/data_file.rs | 8 + crates/core/src/backup_format/manifest.rs | 7 + crates/core/src/metrics/collector.rs | 37 +- crates/core/src/metrics/collector_tests.rs | 58 + crates/core/src/metrics/types.rs | 45 +- crates/engine/Cargo.toml | 6 +- crates/engine/src/backup.rs | 102 +- crates/engine/src/backup/job.rs | 86 +- crates/engine/src/backup/job/failure.rs | 51 +- crates/engine/src/backup/job/heartbeat.rs | 124 +- crates/engine/src/backup/maintenance.rs | 180 +- crates/engine/src/backup/reconciler.rs | 766 +- crates/engine/src/backup/restore.rs | 1303 ++++ crates/engine/src/backup/restore/download.rs | 308 + crates/engine/src/backup/restore/load.rs | 287 + crates/engine/src/backup/restore/spool.rs | 516 ++ crates/engine/src/backup/runtime.rs | 27 +- crates/engine/src/backup/tests.rs | 6793 ++++++++++++++++- crates/engine/src/lib.rs | 6 +- crates/server/Cargo.toml | 12 +- crates/server/src/backup_store.rs | 31 + crates/server/src/metrics_endpoint.rs | 3 + crates/server/src/serve.rs | 3 +- crates/storage-cassandra/Cargo.toml | 6 +- .../storage-cassandra/src/backup_catalog.rs | 509 +- crates/storage-cassandra/src/backup_engine.rs | 300 +- crates/storage-cassandra/src/create_table.rs | 64 +- crates/storage-cassandra/src/data/index.rs | 161 +- crates/storage-cassandra/src/lib.rs | 1 + .../storage-cassandra/src/restore_engine.rs | 483 ++ crates/storage-cassandra/src/test_hooks.rs | 142 +- crates/storage-cassandra/src/workers.rs | 57 +- crates/storage-cassandra/tests/common/mod.rs | 79 + .../tests/direct/backup_engine.rs | 1863 ++++- .../storage-cassandra/tests/direct/index.rs | 189 +- crates/storage-mongodb/Cargo.toml | 7 +- crates/storage-mongodb/src/backup_catalog.rs | 421 +- crates/storage-mongodb/src/backup_engine.rs | 229 +- .../storage-mongodb/src/backup_live_tests.rs | 1117 ++- crates/storage-mongodb/src/lib.rs | 1 + crates/storage-mongodb/src/restore_engine.rs | 328 + crates/storage-mongodb/src/snapshot_engine.rs | 132 +- crates/storage-mongodb/src/table_engine.rs | 21 +- crates/storage-mongodb/src/ttl_worker.rs | 14 +- .../data_migrations/0006_restore_progress.sql | 18 + crates/storage-postgres/src/backup_catalog.rs | 409 +- crates/storage-postgres/src/backup_engine.rs | 418 +- crates/storage-postgres/src/create_table.rs | 169 +- crates/storage-postgres/src/lib.rs | 1 + crates/storage-postgres/src/migrations/mod.rs | 16 +- crates/storage-postgres/src/restore_engine.rs | 373 + crates/storage-postgres/tests/restore_job.rs | 1743 +++++ crates/storage-sqlite/src/backup.rs | 42 +- crates/storage-sqlite/src/backup_catalog.rs | 377 +- crates/storage-sqlite/src/create_table.rs | 200 +- .../storage-sqlite/src/data/vector_index.rs | 2 +- crates/storage-sqlite/src/lib.rs | 27 +- crates/storage-sqlite/src/migrations/mod.rs | 4 + crates/storage-sqlite/src/migrations/tests.rs | 9 +- crates/storage-sqlite/src/restore_engine.rs | 538 ++ .../storage-sqlite/src/restore_job_tests.rs | 1536 ++++ .../storage-sqlite/tests/backup_job_panic.rs | 4 +- .../tests/restore_job_resume.rs | 487 ++ crates/storage/src/backup_catalog.rs | 242 +- crates/storage/src/backup_definition.rs | 70 + crates/storage/src/lib.rs | 10 +- crates/storage/src/migrations/registry.rs | 9 + crates/storage/src/restore_engine.rs | 153 + devtools/backup-restore-matrix | 974 +++ devtools/run-mongodb-tests | 73 +- docs/design/15-backup-restore-bench-pr9.md | 310 + docs/design/15-backup-restore.md | 138 +- docs/design/README.md | 1 + docs/differences-from-dynamodb.md | 19 +- docs/dynamodb-limits.md | 4 +- docs/manuals/03-usage-guide.md | 2 +- docs/manuals/05-admin-guide.md | 21 +- docs/manuals/06-developer-test-guide.md | 57 +- docs/manuals/07-upgrade-manual.md | 8 +- docs/schema/postgres.txt | 6 + requirements-matrix.txt | 9 + tests/rust/src/backup_restore.rs | 82 +- tests/rust/src/restore_active_completeness.rs | 39 +- tests/test_auth_permissions.py | 16 +- tests/test_backup_job.py | 862 ++- tests/test_backup_restore_fidelity.py | 106 +- tests/test_backup_store_config.py | 46 +- tests/test_mongodb_backup_restore.py | 274 +- 95 files changed, 25881 insertions(+), 1419 deletions(-) create mode 100644 .github/workflows/matrix-nightly.yml create mode 100644 crates/engine/src/backup/restore.rs create mode 100644 crates/engine/src/backup/restore/download.rs create mode 100644 crates/engine/src/backup/restore/load.rs create mode 100644 crates/engine/src/backup/restore/spool.rs create mode 100644 crates/storage-cassandra/src/restore_engine.rs create mode 100644 crates/storage-mongodb/src/restore_engine.rs create mode 100644 crates/storage-postgres/data_migrations/0006_restore_progress.sql create mode 100644 crates/storage-postgres/src/restore_engine.rs create mode 100644 crates/storage-postgres/tests/restore_job.rs create mode 100644 crates/storage-sqlite/src/restore_engine.rs create mode 100644 crates/storage-sqlite/src/restore_job_tests.rs create mode 100644 crates/storage-sqlite/tests/restore_job_resume.rs create mode 100644 crates/storage/src/restore_engine.rs create mode 100755 devtools/backup-restore-matrix create mode 100644 docs/design/15-backup-restore-bench-pr9.md create mode 100644 requirements-matrix.txt diff --git a/.github/workflows/integration-mongodb.yml b/.github/workflows/integration-mongodb.yml index 32418c718..6a2a47dd3 100644 --- a/.github/workflows/integration-mongodb.yml +++ b/.github/workflows/integration-mongodb.yml @@ -52,9 +52,11 @@ jobs: run: cargo build --release --no-default-features --features mongodb-test-hooks - name: Run MongoDB pytest suite env: - # run-mongodb-tests initializes the server without [backup.store]: - # CreateBackup is the in-catalog create. - EXTENDDB_TEST_BACKUP_STORE: none + # run-mongodb-tests gives the server a filesystem [backup.store] + # under its output directory: CreateBackup is the store-backed job + # and the restore cells (tests/test_mongodb_backup_restore.py) + # rewrite prefixes under that root. + EXTENDDB_TEST_BACKUP_STORE: filesystem run: devtools/run-mongodb-tests -- --pytest --comprehensive --parallel # SnapshotEngine consistency check against a disposable replica set. The @@ -128,9 +130,9 @@ jobs: run: cargo build --release --no-default-features --features mongodb-test-hooks - name: Run MongoDB rust integration suite env: - # run-mongodb-tests initializes the server without [backup.store]: - # the Rust backup tests restore the in-catalog create's backups. - EXTENDDB_TEST_BACKUP_STORE: none + # run-mongodb-tests gives the server a filesystem [backup.store]: + # the Rust backup tests restore the store-backed job's backups. + EXTENDDB_TEST_BACKUP_STORE: filesystem run: devtools/run-mongodb-tests --rust-live -- --rust --rust-integration mongodb-production-build: diff --git a/.github/workflows/integration.yml b/.github/workflows/integration.yml index ec6155382..839ceeb82 100644 --- a/.github/workflows/integration.yml +++ b/.github/workflows/integration.yml @@ -170,14 +170,34 @@ jobs: EXTENDDB_TEST_BACKUP_STORE: filesystem run: devtools/run-tests --extenddb --pytest --comprehensive --parallel --filter "not import_export" + # moto for the job suite's store-outage cell, the one cell that + # exercises `spool_path` on PostgreSQL (an S3 download into the spool, + # the retry under a cut, the spool entry removed); without the endpoint + # the cell skips. Same environment as the s3 job's. + - name: Start S3 mock + run: | + python3 -m venv "$RUNNER_TEMP/moto" + "$RUNNER_TEMP/moto/bin/pip" install -r requirements-moto.txt + nohup "$RUNNER_TEMP/moto/bin/moto_server" -p 9000 > moto.log 2>&1 & + for _ in $(seq 1 30); do + if curl -sf http://127.0.0.1:9000/ > /dev/null; then exit 0; fi + sleep 1 + done + echo "moto_server did not start"; cat moto.log; exit 1 + # The backup store and job suites start servers of their own, so they # are excluded from run-tests' line and run by name. - name: Run the backup store and job suites env: EXTENDDB_TEST_PG_CONNECTION_STRING: postgresql://postgres:devpass@127.0.0.1:5432 EXTENDDB_TEST_PG_ADMIN_CONNECTION_STRING: postgresql://postgres:devpass@127.0.0.1:5432 + EXTENDDB_TEST_S3_ENDPOINT: http://127.0.0.1:9000 run: python3 -m pytest tests/test_backup_store_config.py tests/test_backup_job.py -v -rP -p no:cacheprovider + - name: Print the S3 mock's log + if: failure() + run: cat moto.log || true + # run-tests restarts the server daemonized to apply the import/export # config, and a daemonized server logs to syslog, not the job log. Without # this dump, the storage error behind a client-visible 500 leaves no @@ -296,9 +316,11 @@ jobs: # with EXTENDDB_TEST_BACKUP_STORE=s3, and the fidelity suite's health check # confirms the server's store is the S3 one; the store configuration # suite's S3 cells need the same endpoint. The job suite - # (tests/test_backup_job.py) starts servers of its own with the SQLite - # build's defaulted filesystem root, as on the SQLite job: its deployment - # fixture has no S3 branch, so it covers the job and not this store. + # (tests/test_backup_job.py) starts servers of its own: most of its cells + # on the SQLite build's defaulted filesystem root, as on the SQLite job, + # and the store-outage cell on an S3 store at the moto endpoint, which it + # cuts mid-download and restores, so this job runs one restore from an S3 + # store. run-integration-sqlite-s3: runs-on: extenddb_ubuntu-2404_4-core timeout-minutes: 40 @@ -391,7 +413,9 @@ jobs: run: devtools/run-tests --extenddb --pytest --parallel --filter "backup" # The store configuration suite's S3 cells against moto; the job suite - # against the build's defaulted filesystem root (the job comment above). + # against the build's defaulted filesystem root, except the store-outage + # cell, which runs against moto through EXTENDDB_TEST_S3_ENDPOINT (the + # job comment above). - name: Run the store configuration suite's S3 cells and the job suite env: EXTENDDB_TEST_S3_ENDPOINT: http://127.0.0.1:9000 @@ -731,7 +755,8 @@ jobs: # A file-backed SQLite server without [backup.store] has the defaulted # filesystem root beside its database file, so CreateBackup is the # asynchronous, store-backed one: the backup tests wait for AVAILABLE, - # delete what they create, and expect the restore's refusal. + # delete what they create, and restore the store-backed backups + # (the round trip). EXTENDDB_TEST_BACKUP_STORE: filesystem run: devtools/run-tests --extenddb --rust-integration --release diff --git a/.github/workflows/matrix-nightly.yml b/.github/workflows/matrix-nightly.yml new file mode 100644 index 000000000..d2a758a19 --- /dev/null +++ b/.github/workflows/matrix-nightly.yml @@ -0,0 +1,184 @@ +# Copyright 2026 ExtendDB contributors +# SPDX-License-Identifier: Apache-2.0 +name: Cross-backend restore matrix (nightly) + +# The cross-backend restore matrix (design PR 9b): a backup taken on one +# backend restores, through the store, into another. Each cell starts a source +# and a target server from the per-backend binaries, copies the source's backup +# prefix under a root the target owns, seeds the target's `backup_index` row as +# `backup sync --adopt-store` will (PR 10), restores and compares. The cells are +# heavy (two servers each, four backends, PostgreSQL + a MongoDB replica set + +# Cassandra) so they run nightly rather than on every pull request. + +on: + schedule: + - cron: "0 7 * * *" + workflow_dispatch: + +permissions: + contents: read + +# One run at a time: the cells drop and re-create fixed MongoDB database and +# Cassandra keyspace names, so two runs on one runner would destroy each +# other's catalogs. A second run queues behind the first. +concurrency: + group: ${{ github.workflow }} + cancel-in-progress: false + +jobs: + matrix: + runs-on: extenddb_ubuntu-2404_4-core + timeout-minutes: 120 + services: + postgres: + image: pgvector/pgvector:pg16 + env: + POSTGRES_PASSWORD: devpass + ports: + - 127.0.0.1:55432:5432 + options: >- + --health-cmd "pg_isready -U postgres" + --health-interval 10s + --health-timeout 5s + --health-retries 10 + cassandra: + image: cassandra:4.1 + ports: + - 127.0.0.1:9042:9042 + env: + CASSANDRA_CLUSTER_NAME: extenddb-ci + HEAP_NEWSIZE: 128M + MAX_HEAP_SIZE: 1024M + options: >- + --health-cmd "cqlsh -e 'SELECT release_version FROM system.local'" + --health-interval 15s + --health-timeout 10s + --health-retries 20 + --health-start-period 60s + steps: + - uses: actions/checkout@v6 + - uses: dtolnay/rust-toolchain@stable + - uses: Swatinem/rust-cache@v2 + with: + cache-on-failure: true + # Only runs on main save, so other runs restore main's cache; a + # scheduled run's ref is the default branch, so the nightly saves. + save-if: ${{ github.ref == 'refs/heads/main' }} + - uses: actions/setup-python@v5 + with: + python-version: "3.11" + - name: Install Python dependencies + # requirements-matrix.txt adds the MongoDB and Cassandra drivers the + # script's seed and teardown use on those backends. + run: pip install -r requirements.txt -r requirements-matrix.txt + + - name: Install the PostgreSQL client + # The matrix script seeds and tears down every PostgreSQL source and + # target through the host's psql; the runner image is not assumed to + # carry one. + run: | + sudo apt-get update + sudo apt-get install -y --no-install-recommends postgresql-client + + - name: Check the host tools the matrix script calls + run: | + psql --version + python3 -c "import pymongo, cassandra; print(pymongo.version, cassandra.__version__)" + + - name: Configure Cassandra authentication + run: | + docker exec ${{ job.services.cassandra.id }} sed -i \ + 's/^authenticator: .*/authenticator: PasswordAuthenticator/' \ + /etc/cassandra/cassandra.yaml + docker restart ${{ job.services.cassandra.id }} + for i in $(seq 1 90); do + if docker exec ${{ job.services.cassandra.id }} \ + cqlsh -u cassandra -p cassandra \ + -e 'SELECT release_version FROM system.local' >/dev/null 2>&1; then + echo "Cassandra ready with PasswordAuthenticator after ${i}s" + exit 0 + fi + sleep 2 + done + echo "Cassandra did not become ready" >&2 + exit 1 + + - name: Start a MongoDB replica set + # GitHub `services:` cannot express a replica set (--replSet + initiate + # + wait-for-PRIMARY), so start it directly, as devtools/run-mongodb-tests + # does. Published on loopback only; no authentication. + run: | + docker run -d --name matrix-mongo -p 127.0.0.1:27117:27017 mongo:7 \ + mongod --replSet rs0 --bind_ip_all + for i in $(seq 1 30); do + docker exec matrix-mongo mongosh --quiet --eval 'db.runCommand({ping:1}).ok' 2>/dev/null \ + | grep -qx 1 && break + sleep 1 + done + docker exec matrix-mongo mongosh --quiet --eval \ + 'rs.initiate({_id:"rs0",members:[{_id:0,host:"localhost:27017"}]})' >/dev/null 2>&1 || true + for i in $(seq 1 30); do + state=$(docker exec matrix-mongo mongosh --quiet --eval 'rs.status().members[0].stateStr' 2>/dev/null | tail -1) + [ "$state" = PRIMARY ] && { echo "replica set PRIMARY after ${i}s"; exit 0; } + sleep 1 + done + echo "MongoDB replica set did not elect a PRIMARY" >&2 + exit 1 + + - name: Build the four backend binaries + # One target directory, four sequential builds (cargo rebuilds only + # the feature-dependent crates between them), each binary copied to + # the path the script's --bin defaults use + # (target/rel-/release/extenddb). Every `init` the script + # runs writes the TLS material under ~/.extenddb/tls when absent, so + # nothing has to generate it here. + run: | + for b in postgres sqlite mongodb cassandra; do + cargo build --release -p extenddb --no-default-features --features "$b" + mkdir -p "target/rel-$b/release" + cp target/release/extenddb "target/rel-$b/release/extenddb" + done + + - name: Run the matrix + run: | + set -o pipefail + mkdir -p matrix-logs + rc=0 + plain="postgres sqlite mongodb" + targets="postgres sqlite mongodb cassandra" + for s in $plain; do + for t in $targets; do + echo "=== $s -> $t ===" + python3 devtools/backup-restore-matrix --source "$s" --target "$t" \ + --work "matrix-logs/$s-$t" || rc=1 + done + done + # The duplicate-key cell into every target, from a PostgreSQL source. + for t in $targets; do + echo "=== postgres -> $t --duplicate-key ===" + python3 devtools/backup-restore-matrix --source postgres --target "$t" --duplicate-key \ + --work "matrix-logs/postgres-$t-dup" || rc=1 + done + # The vector cells: into PostgreSQL and SQLite the vector index is + # restored and answers; into MongoDB and Cassandra the manifest is + # refused before any table. + for s in postgres sqlite; do + for t in postgres sqlite mongodb cassandra; do + echo "=== $s -> $t --vector ===" + python3 devtools/backup-restore-matrix --source "$s" --target "$t" --vector \ + --work "matrix-logs/$s-$t-vector" || rc=1 + done + done + exit $rc + + - name: Upload the matrix logs on failure + if: failure() + uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2 + with: + name: matrix-logs + path: matrix-logs + if-no-files-found: ignore + + - name: Stop the MongoDB replica set + if: always() + run: docker rm -f matrix-mongo >/dev/null 2>&1 || true diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 524874f76..6bd47f9fd 100755 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -50,8 +50,25 @@ cargo test --workspace # Integration tests (requires a running extenddb server; the variable says # where that server writes its backups, see the developer test guide) EXTENDDB_TEST_BACKUP_STORE=filesystem devtools/run-tests --extenddb --pytest + +# The backup store and job suites start servers of their own from the +# release build named by EXTENDDB_BINARY, so they run by name (the S3 cells +# need a moto server at EXTENDDB_TEST_S3_ENDPOINT; see the developer test +# guide for the PostgreSQL and moto variables) +EXTENDDB_BINARY=target/release/extenddb python3 -m pytest tests/test_backup_store_config.py tests/test_backup_job.py -v ``` +The MongoDB backup cells that read the store's root and the server's log +(`EXTENDDB_TEST_BACKUP_ROOT`, `EXTENDDB_TEST_SERVER_LOG`) run through +`devtools/run-mongodb-tests`, which exports both; its legacy LSI fallback +cell needs the runner's `--backup-store none`. The cross-backend restore +matrix is `devtools/backup-restore-matrix` (one cell per invocation, from the +four per-backend release binaries; `pip install -r requirements-matrix.txt` +for its MongoDB and Cassandra cells) and runs nightly from +`.github/workflows/matrix-nightly.yml`. The developer test guide +(`docs/manuals/06-developer-test-guide.md`, "Test Runner Script") has a +paragraph on each. + ### Code Style All contributions must pass: diff --git a/crates/app/src/cmd_backup.rs b/crates/app/src/cmd_backup.rs index ba93d133a..dc4f53a44 100644 --- a/crates/app/src/cmd_backup.rs +++ b/crates/app/src/cmd_backup.rs @@ -71,8 +71,11 @@ //! while the command holds the file, so before the wait the command runs //! the reconciler's duties itself with a zero stale window, as the holder //! of the maintenance claim (the cycle takes its held branch and does not -//! read the claim back through that window): a legacy restore -//! the stopped server left RUNNING is failed and its target removed, a +//! read the claim back through that window): a restore the stopped server +//! left RUNNING is failed and its target removed, a store-backed one too, +//! which the held branch fails rather than resumes (a record with no +//! recorded progress over a target that holds items is failed +//! `target_not_empty` and its target left for `remove-target`), a //! target a FAILED restore left CREATING is removed, an in-catalog CREATING //! backup it left is removed, and a DELETING row is finished; the wait and //! the CREATING-row removal then also use a zero window, so a CREATING or @@ -96,17 +99,37 @@ //! `extenddb destroy` removes the databases and leaves the store untouched: //! the store is the copy that survives destroy, and this command is the one //! that deletes it. +//! +//! `backup remove-target --account
` removes a restore target +//! DeleteTable refuses (a CREATING table) with its restore record, under the +//! maintenance claim like `purge`: a FAILED record's target whatever the +//! reason (`target_not_empty` included, the one state the reconciler leaves +//! for this command: on MongoDB and Cassandra a crash before the first +//! progress record, the whole first data file on MongoDB and the first +//! 10,000 items or 32 MiB on Cassandra, ends in it), a RUNNING record's +//! target whose owner is dead (no heartbeat within `stale_job_after_secs`; +//! on SQLite under the serve lock every owner is dead), and a legacy +//! record's target the same way. It refuses a RUNNING record whose owner is +//! live, a DONE record and a table no record names ("not a restore target"; +//! an ACTIVE table is one or the other). It needs no store: it opens the +//! catalog as `serve` does (on SQLite under the serve lock) whether or not +//! `[backup.store]` is configured, prints the claim it took and the one line +//! of what it removed (`extenddb_engine::maintenance::remove_target` states +//! each step), and logs the removal under `extenddb::audit` with the +//! account, the table and the backup's ARN. use std::sync::Arc; use std::time::{Duration, Instant}; use clap::{Args, Subcommand}; use extenddb_config as config; -use extenddb_engine::maintenance::{Interrupt, Wait, WaitOutcome, purge_row, wait_for_quiet}; +use extenddb_engine::maintenance::{ + Interrupt, RemoveTarget, Wait, WaitOutcome, purge_row, remove_target, wait_for_quiet, +}; use extenddb_engine::{BackupRuntime, ConfiguredStore, HeldClaim, StoreHealth, StoreProblem}; use extenddb_storage::StorageEngine; use extenddb_storage::backup_catalog::{ - BackupRow, BackupStatus, JOB_HEARTBEAT_INTERVAL, StoreIdentity, + BackupRow, BackupStatus, JOB_HEARTBEAT_INTERVAL, JobStatus, StoreIdentity, }; use extenddb_storage::error::StorageError; use futures::TryStreamExt; @@ -141,6 +164,17 @@ enum BackupCommand { #[arg(long)] confirm: bool, }, + /// Remove a restore target DeleteTable refuses (a CREATING table) and its restore record: + /// the target of a FAILED restore (target_not_empty included) or of a RUNNING restore whose + /// owner is dead. Takes the maintenance claim first. Refuses a RUNNING restore with a live + /// owner, a DONE restore and a table no restore record names. + RemoveTarget { + /// The account the table belongs to + #[arg(long)] + account: String, + /// The table's name + table: String, + }, } /// How often the waiting purge re-reads the rows. @@ -152,6 +186,7 @@ const WAIT_POLL: Duration = Duration::from_secs(2); /// name, which the migration prints while it waits. const LIST: &str = "extenddb backup list"; const PURGE: &str = "extenddb backup purge"; +const REMOVE_TARGET: &str = "extenddb backup remove-target"; pub async fn run(args: BackupArgs, release: &'static str) -> anyhow::Result<()> { if !std::path::Path::new(&args.config).exists() { @@ -196,6 +231,20 @@ pub async fn run(args: BackupArgs, release: &'static str) -> anyhow::Result<()> Opening::NoStore(reason) => anyhow::bail!("{reason}"), } } + BackupCommand::RemoveTarget { account, table } => { + // The catalog as `serve` opens it; the store is not needed, so + // it is described and never checked, and the default root and + // an identity file are not created. + let (catalog, _) = open_catalog( + &app_config, + &args.config, + release, + REMOVE_TARGET, + Mode::WriteCatalogOnly, + ) + .await?; + remove_restore_target(&catalog, &app_config, &account, &table).await + } } } @@ -214,47 +263,36 @@ enum Mode { /// As `serve` opens it (on SQLite under the serve lock); the store's /// check may write the identity file of an empty root. Write, + /// As `serve` opens it, for a command that needs no store + /// (`remove-target`): the store is described as `Read` leaves it, so + /// neither the default root nor an identity file is created. + WriteCatalogOnly, } -/// The storage engine, the backup runtime with the configured store, and -/// the store's identity as the command read it. -struct Opened { +/// The open catalog: the storage engine, the backup runtime (with the +/// configured store when there is one) and the backend's name. +struct Catalog { storage: Arc, runtime: Arc, backend: String, - identity: Identity, -} - -/// The store's identity as the command read it. -enum Identity { - /// The check passed, or the file was read and agrees with the rows. - Known(StoreIdentity), - /// The root has no identity file and the catalog no rows for it: a - /// fresh root nothing has written to. - Empty, - /// The check failed (a listing still runs; a purge refuses). - Problem(StoreProblem), -} - -impl Opened { - fn store(&self) -> &ConfiguredStore { - self.runtime - .configured_store() - .expect("open returns only with a configured store") - } } -/// Open the catalog as `command`. A catalog the factory refuses (on -/// PostgreSQL: a migration running, or the catalog at another migration than -/// this binary's) is an [`extenddb_server::CatalogRefusal`], with the -/// message and exit code `serve` gives for the same status. -async fn open( +/// Open the catalog as `command` in `mode` and describe the configured +/// store without checking it; the second member is why no store is +/// configured, when none is. A catalog the factory refuses (on PostgreSQL: +/// a migration running, or the catalog at another migration than this +/// binary's) is an [`extenddb_server::CatalogRefusal`], with the message and +/// exit code `serve` gives for the same status. +async fn open_catalog( app_config: &config::AppConfig, config_path: &str, release: &'static str, command: &'static str, mode: Mode, -) -> anyhow::Result { +) -> anyhow::Result<( + Catalog, + Option, +)> { let mut options = extenddb_storage::server_components::ServerComponentsOptions::default(); options.release = release; options.max_concurrent_jobs = app_config.backup.max_concurrent_jobs; @@ -293,21 +331,68 @@ async fn open( &backend, &*storage, match mode { - Mode::Read => extenddb_server::backup_store::Caller::ReadOnlyCommand, + Mode::Read | Mode::WriteCatalogOnly => { + extenddb_server::backup_store::Caller::ReadOnlyCommand + } Mode::Write => extenddb_server::backup_store::Caller::Command, }, ) .await?; - let runtime = Arc::new(setup.runtime); - let Some(store) = runtime.configured_store() else { + Ok(( + Catalog { + storage, + runtime: Arc::new(setup.runtime), + backend, + }, + setup.not_configured, + )) +} + +/// The catalog with the configured store and the store's identity as the +/// command read it. +struct Opened { + catalog: Catalog, + identity: Identity, +} + +/// The store's identity as the command read it. +enum Identity { + /// The check passed, or the file was read and agrees with the rows. + Known(StoreIdentity), + /// The root has no identity file and the catalog no rows for it: a + /// fresh root nothing has written to. + Empty, + /// The check failed (a listing still runs; a purge refuses). + Problem(StoreProblem), +} + +impl Opened { + fn store(&self) -> &ConfiguredStore { + self.catalog + .runtime + .configured_store() + .expect("open returns only with a configured store") + } +} + +async fn open( + app_config: &config::AppConfig, + config_path: &str, + release: &'static str, + command: &'static str, + mode: Mode, +) -> anyhow::Result { + let (catalog, not_configured) = + open_catalog(app_config, config_path, release, command, mode).await?; + let Some(store) = catalog.runtime.configured_store() else { // Why there is none, as `configure_described` found it: the section // absent or disabled, the default root not yet created, or the // default root failed its check with no rows for it. - let why = setup - .not_configured - .expect("a runtime without a store carries the reason"); + let why = not_configured.expect("a runtime without a store carries the reason"); return Ok(Opening::NoStore(why.to_string())); }; + let storage = &catalog.storage; + let backend = &catalog.backend; // The server's startup leaves an S3 store's check to its reconciler; // the command runs it here. A read leaves the root as it finds it: the // identity file is read, never created, under the store's check bound as @@ -315,13 +400,15 @@ async fn open( // or a mount that has stopped answering costs the command the bound and // not the store's retries. let identity = match (mode, store.health()) { - (Mode::Read, _) => match extenddb_engine::read_identity_bounded(store, &*storage).await { - Ok(Some(identity)) => Identity::Known(identity), - Ok(None) => Identity::Empty, - Err(problem) => Identity::Problem(problem), - }, + (Mode::Read | Mode::WriteCatalogOnly, _) => { + match extenddb_engine::read_identity_bounded(store, &**storage).await { + Ok(Some(identity)) => Identity::Known(identity), + Ok(None) => Identity::Empty, + Err(problem) => Identity::Problem(problem), + } + } (Mode::Write, StoreHealth::Checking) => { - let outcome = extenddb_engine::check_configured_store(store, &*storage, &backend).await; + let outcome = extenddb_engine::check_configured_store(store, &**storage, backend).await; store.record_check(outcome.clone()); match outcome { Ok(identity) => Identity::Known(identity), @@ -336,12 +423,7 @@ async fn open( }, (Mode::Write, StoreHealth::Unhealthy(problem)) => Identity::Problem(problem), }; - Ok(Opening::Store(Opened { - storage, - runtime, - backend, - identity, - })) + Ok(Opening::Store(Opened { catalog, identity })) } async fn rows_of( @@ -382,6 +464,7 @@ async fn rows_of_root( ) -> anyhow::Result<(Vec, Vec)> { let store = opened.store(); let mut ids = opened + .catalog .storage .read_store_ids(store.kind, &store.root) .await @@ -398,7 +481,7 @@ async fn rows_of_root( store_root: store.root.clone(), store_id: id.clone(), }; - rows.extend(rows_of(&*opened.storage, &identity, status).await?); + rows.extend(rows_of(&*opened.catalog.storage, &identity, status).await?); } Ok((ids, rows)) } @@ -494,7 +577,7 @@ async fn dry_run(opened: &Opened) -> anyhow::Result<()> { async fn purge(opened: &Opened, app_config: &config::AppConfig) -> anyhow::Result<()> { let backup = &app_config.backup; - let storage = &*opened.storage; + let storage = &*opened.catalog.storage; let configured = opened.store(); let identity = match &opened.identity { Identity::Known(identity) => identity, @@ -542,7 +625,7 @@ async fn purge(opened: &Opened, app_config: &config::AppConfig) -> anyhow::Resul // owner of any row is dead whatever its heartbeat says, so the window a // live owner is told apart by is zero. On the other backends purge runs // beside live servers, whose jobs heartbeat within `stale_after`. - let owner_window = if opened.backend == "sqlite" { + let owner_window = if opened.catalog.backend == "sqlite" { resolve_dead_owners(opened).await; Duration::ZERO } else { @@ -716,10 +799,10 @@ async fn signal_name() -> &'static str { /// wait decides. async fn resolve_dead_owners(opened: &Opened) { match extenddb_engine::reconciler::cycle( - &*opened.storage, - &opened.runtime, + &opened.catalog.storage, + &opened.catalog.runtime, Duration::ZERO, - &opened.backend, + &opened.catalog.backend, &mut extenddb_engine::reconciler::ReconcilerState::for_maintenance_holder(), ) .await @@ -754,3 +837,55 @@ async fn release(claim: HeldClaim) { println!("could not release the maintenance claim ({e}); it expires on its own"); } } + +/// `backup remove-target --account
` (module doc). +async fn remove_restore_target( + catalog: &Catalog, + app_config: &config::AppConfig, + account_id: &str, + table_name: &str, +) -> anyhow::Result<()> { + let stale_after = Duration::from_secs(app_config.backup.stale_job_after_secs); + let holder = format!( + "extenddb backup remove-target on {} (pid {})", + extenddb_server::backup_store::hostname(), + std::process::id() + ); + println!("=== extenddb backup remove-target ==="); + println!("Taking the maintenance claim as {holder:?}."); + // On SQLite the command holds the file alone, so every job owner is + // dead whatever its heartbeat says; on the other backends a RUNNING + // record whose heartbeat is within `stale_after` has a live owner. + let owner_window = if catalog.backend == "sqlite" { + Duration::ZERO + } else { + stale_after + }; + let removed = remove_target( + &*catalog.storage, + &RemoveTarget { + account_id: account_id.to_owned(), + table_name: table_name.to_owned(), + holder, + stale_after, + owner_window, + heartbeat_interval: JOB_HEARTBEAT_INTERVAL, + }, + ) + .await + .map_err(|e| anyhow::anyhow!("{e}"))?; + let state = match (removed.status, removed.failure_reason.as_deref()) { + (JobStatus::Running, _) => "RUNNING with a dead owner".to_owned(), + (status, Some(reason)) => format!("{status} ({reason})"), + (status, None) => status.to_string(), + }; + println!( + "removed the restore target {table_name} (account {account_id}, table id {}) and its \ + record: its {} restore from {} was {state}", + removed.table_id, + removed.kind.as_str(), + removed.backup_arn + ); + println!("The maintenance claim is released."); + Ok(()) +} diff --git a/crates/app/src/lib.rs b/crates/app/src/lib.rs index df7f2e51f..57e0ad4f3 100644 --- a/crates/app/src/lib.rs +++ b/crates/app/src/lib.rs @@ -69,7 +69,7 @@ enum Command { Init(cmd_init::InitArgs), /// Tear down a extenddb deployment (the databases; the backup store is left untouched) Destroy(cmd_destroy::DestroyArgs), - /// List or purge the backups of the configured backup store + /// List or purge the backups of the configured backup store, or remove a restore target Backup(cmd_backup::BackupArgs), /// Validate a extenddb deployment Verify(cmd_verify::VerifyArgs), diff --git a/crates/config/src/backup.rs b/crates/config/src/backup.rs index b31a80ce7..e9f31dcf5 100644 --- a/crates/config/src/backup.rs +++ b/crates/config/src/backup.rs @@ -73,10 +73,18 @@ //! either format stay readable, listable, restorable and deletable. //! //! The spool. Restores from an S3 store download to `spool_path`, created -//! `0700` with files `0600`. On file-backed SQLite it defaults to +//! `0700` with files `0600`: one entry `restore-
/` per running +//! restore, holding the manifest, its sidecar and each data file with a +//! `.verified` marker once its digests matched, so the spool costs the +//! compressed size of the backups being restored at once; the job removes +//! its entry when it ends, a server stopped by SIGTERM keeps it for the +//! resume, and the reconciler removes entries of restores no longer RUNNING +//! with this host's `spool_host`. On file-backed SQLite it defaults to //! `backup-spool/` beside the database file; on every other backend an S3 //! store needs `spool_path` and startup refuses without it (a filesystem -//! store needs no spool: files are verified in place). A `spool_path` that +//! store needs no spool: files are hashed in place and read again for the +//! load, so a filesystem store's `get` errors, which are local I/O errors, +//! are not retried as an S3 store's transport errors are). A `spool_path` that //! contains or is contained in a filesystem store root is refused (the store //! would list and delete it, and a sync would read its first component as an //! account id), and so is one that is or contains the database file's diff --git a/crates/core/src/backup_format/data_file.rs b/crates/core/src/backup_format/data_file.rs index bb5e48033..70a226541 100644 --- a/crates/core/src/backup_format/data_file.rs +++ b/crates/core/src/backup_format/data_file.rs @@ -965,6 +965,14 @@ impl DataFileDecoder { self.reader.get_ref().get_ref().hasher.is_some() } + /// The uncompressed bytes decoded so far: every line consumed, its + /// newline included. The difference across one [`Self::next_item`] is + /// that item's line length, the unit a reader bounds its memory in. + #[must_use] + pub fn bytes_decoded(&self) -> u64 { + self.bytes_decoded + } + /// The error for a failed read: [`FormatError::CompressedSizeMismatch`] /// when the source ran past its recorded size, otherwise /// [`FormatError::Decode`]. diff --git a/crates/core/src/backup_format/manifest.rs b/crates/core/src/backup_format/manifest.rs index 28658df85..5afa2afd9 100644 --- a/crates/core/src/backup_format/manifest.rs +++ b/crates/core/src/backup_format/manifest.rs @@ -588,6 +588,13 @@ impl DataFileReader { } checked } + + /// The uncompressed bytes decoded so far + /// ([`DataFileDecoder::bytes_decoded`]). + #[must_use] + pub fn bytes_decoded(&self) -> u64 { + self.decoder.bytes_decoded() + } } impl Iterator for DataFileReader { diff --git a/crates/core/src/metrics/collector.rs b/crates/core/src/metrics/collector.rs index 8975e2675..f6a3e6f68 100755 --- a/crates/core/src/metrics/collector.rs +++ b/crates/core/src/metrics/collector.rs @@ -385,13 +385,44 @@ impl MetricsCollector { self.record(MetricName::BackupSnapshotAge, age_secs, None, None, None); } - /// Record a failed store call of a backup job or of the backup - /// reconciler; `op` names the call (the job's `put` and - /// `delete_prefix`, the reconciler's `get`, `list` and `delete_prefix`). + /// Record a failed store call of a backup job, a restore job or the + /// backup reconciler; `op` names the call (a backup job's `put` and + /// `delete_prefix`, a restore's `get`, the reconciler's `get`, `list` + /// and `delete_prefix`). pub fn record_backup_store_error(&self, op: &str) { self.record(MetricName::BackupStoreErrors, 1.0, None, None, Some(op)); } + /// Record the end of a store-backed restore job: one `RestoreJobs` under + /// the outcome and its duration in microseconds under the same + /// dimension, as [`Self::record_backup_job`] records a backup's. The job + /// measures the duration from its registration to the end of its last + /// catalog or store call. + pub fn record_restore_job(&self, outcome: super::types::BackupJobOutcome, duration_us: f64) { + let outcome = outcome.to_string(); + self.record(MetricName::RestoreJobs, 1.0, None, None, Some(&outcome)); + self.record( + MetricName::RestoreJobDuration, + duration_us, + None, + None, + Some(&outcome), + ); + } + + /// Record bytes a restore job read from the store, as each download + /// arrived. + #[allow(clippy::cast_precision_loss)] // byte counts of one backup + pub fn record_restore_bytes_downloaded(&self, bytes: u64) { + self.record( + MetricName::RestoreBytesDownloaded, + bytes as f64, + None, + None, + None, + ); + } + /// Record per-segment latency breakdown for a request. pub fn record_segments( &self, diff --git a/crates/core/src/metrics/collector_tests.rs b/crates/core/src/metrics/collector_tests.rs index 385b07e83..1683aa25f 100755 --- a/crates/core/src/metrics/collector_tests.rs +++ b/crates/core/src/metrics/collector_tests.rs @@ -232,3 +232,61 @@ fn backup_job_recorders_land_under_the_outcome_and_the_store_call() { assert_eq!(outcome.to_string(), text); } } + +/// The restore job's recorders land as the backup job's do: one +/// `RestoreJobs` and one `RestoreJobDuration` per job under the outcome, +/// the bytes undimensioned, a `get` failure under `BackupStoreErrors`; and +/// the names render as the `/metrics` endpoint prints them. +#[test] +fn restore_job_recorders_land_under_the_outcome_and_render_their_names() { + use crate::metrics::types::{BackupJobOutcome, QuerySource}; + + let c = MetricsCollector::new(); + c.record_restore_job(BackupJobOutcome::Succeeded, 2_000.0); + c.record_restore_job(BackupJobOutcome::Cancelled, 500.0); + c.record_restore_bytes_downloaded(8192); + c.record_backup_store_error("get"); + + let by = |metric: MetricName| { + let mut snaps = c.query(&MetricsQuery { + metric: Some(metric), + window: Some(TimeWindow::AllTime), + ..Default::default() + }); + snaps.sort_by(|a, b| format!("{:?}", a.dimensions).cmp(&format!("{:?}", b.dimensions))); + snaps + }; + let jobs = by(MetricName::RestoreJobs); + let outcomes: Vec = jobs.iter().map(|s| format!("{:?}", s.dimensions)).collect(); + assert_eq!( + outcomes, + [ + "[Operation(\"cancelled\")]".to_owned(), + "[Operation(\"succeeded\")]".to_owned() + ] + ); + assert!(jobs.iter().all(|s| s.count == 1 && s.sum == 1.0)); + let durations = by(MetricName::RestoreJobDuration); + assert_eq!(durations.len(), 2); + assert_eq!(durations.iter().map(|s| s.sum).sum::(), 2_500.0); + let bytes = by(MetricName::RestoreBytesDownloaded); + assert_eq!(bytes.len(), 1); + assert!(bytes[0].dimensions.is_empty()); + assert_eq!(bytes[0].sum, 8192.0); + let errors = by(MetricName::BackupStoreErrors); + assert_eq!( + format!("{:?}", errors[0].dimensions), + "[Operation(\"get\")]" + ); + assert!(by(MetricName::BackupJobs).is_empty()); + assert_eq!(MetricName::RestoreJobs.to_string(), "RestoreJobs"); + assert_eq!( + MetricName::RestoreJobDuration.to_string(), + "RestoreJobDuration" + ); + assert_eq!( + MetricName::RestoreBytesDownloaded.to_string(), + "RestoreBytesDownloaded" + ); + assert_eq!(QuerySource::RestoreJob.to_string(), "RestoreJob"); +} diff --git a/crates/core/src/metrics/types.rs b/crates/core/src/metrics/types.rs index 3d1b747c6..0030ce1b2 100755 --- a/crates/core/src/metrics/types.rs +++ b/crates/core/src/metrics/types.rs @@ -75,16 +75,36 @@ pub enum MetricName { /// Seconds between a running backup job's snapshot instant and now /// (gauge), recorded at each confirmed heartbeat. BackupSnapshotAge, - /// Store operations a backup job or the backup reconciler saw fail; the - /// operation dimension names the store call (`put` by the job's - /// uploads, `delete_prefix` by its failure path and the reconciler's - /// deleting duties, `get` and `list` by the reconciler's manifest probe - /// and deleting duties). + /// Store operations a backup or restore job or the backup reconciler + /// saw fail; the operation dimension names the store call (`put` by a + /// backup job's uploads, `delete_prefix` by its failure path and the + /// reconciler's deleting duties, `get` by a restore's reads of the + /// manifest and its sidecar, in the request and in a resume, by a + /// restore job's downloads and by the reconciler's manifest probe, and + /// `list` by the reconciler's manifest probe and deleting duties). BackupStoreErrors, + /// Store-backed restore jobs ended, one per job; the operation dimension + /// carries the outcome ([`BackupJobOutcome`]: `succeeded` for a DONE + /// record, `failed` for a FAILED one or a job that stopped without a + /// status write of its own, `cancelled` for one the shutdown stopped, + /// whose record stays RUNNING for a resume). Named as `BackupJobs` is + /// (the `/metrics` name; a converter's would be + /// `restore_jobs_total{outcome}`). + RestoreJobs, + /// Duration of a store-backed restore job in microseconds, from its + /// registration to the end of its last catalog or store call, with the + /// outcome as the operation dimension. + RestoreJobDuration, + /// Bytes of each data-file download attempt of a restore job as they + /// arrived, verified or not, so an attempt the job retried counts its + /// bytes too; the manifest and sidecar reads, and a filesystem store's + /// second read of each file for the load, are not counted. + RestoreBytesDownloaded, } -/// How a store-backed backup job ended, the operation dimension of -/// [`MetricName::BackupJobs`] and [`MetricName::BackupJobDuration`]. +/// How a store-backed backup or restore job ended, the operation dimension +/// of [`MetricName::BackupJobs`], [`MetricName::BackupJobDuration`], +/// [`MetricName::RestoreJobs`] and [`MetricName::RestoreJobDuration`]. #[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] pub enum BackupJobOutcome { /// The manifest was written and the row is AVAILABLE. @@ -92,8 +112,8 @@ pub enum BackupJobOutcome { /// The row is FAILED, or the job stopped without a status write of its /// own (its claim lost). Failed, - /// The server's shutdown stopped the job; the row is FAILED with reason - /// `shutdown`. + /// The server's shutdown stopped the job; a backup row is FAILED with + /// reason `shutdown`, a restore record stays RUNNING for a resume. Cancelled, } @@ -141,6 +161,9 @@ impl std::fmt::Display for MetricName { Self::BackupBytesUploaded => f.write_str("BackupBytesUploaded"), Self::BackupSnapshotAge => f.write_str("BackupSnapshotAge"), Self::BackupStoreErrors => f.write_str("BackupStoreErrors"), + Self::RestoreJobs => f.write_str("RestoreJobs"), + Self::RestoreJobDuration => f.write_str("RestoreJobDuration"), + Self::RestoreBytesDownloaded => f.write_str("RestoreBytesDownloaded"), } } } @@ -189,6 +212,9 @@ pub enum QuerySource { /// A store-backed backup job (the worker metrics are recorded once per /// job, at its end). BackupJob, + /// A store-backed restore job (the worker metrics are recorded once per + /// job, at its end; a job the shutdown stopped records none). + RestoreJob, /// Management CLI commands. Management, } @@ -211,6 +237,7 @@ impl std::fmt::Display for QuerySource { Self::LoginAttemptCleanup => f.write_str("LoginAttemptCleanup"), Self::BackupReconciler => f.write_str("BackupReconciler"), Self::BackupJob => f.write_str("BackupJob"), + Self::RestoreJob => f.write_str("RestoreJob"), Self::Management => f.write_str("Management"), } } diff --git a/crates/engine/Cargo.toml b/crates/engine/Cargo.toml index 6800acfdd..1efd942e8 100755 --- a/crates/engine/Cargo.toml +++ b/crates/engine/Cargo.toml @@ -27,8 +27,10 @@ hmac = "0.12" sha2 = "0.10" hex = "0.4" tokio = { workspace = true } -# `TaskTracker` for the backup jobs the server drains at shutdown. -tokio-util = { workspace = true, features = ["rt"] } +# `TaskTracker` for the backup jobs the server drains at shutdown; +# `StreamReader` and `SyncIoBridge` for the restore's in-place decode of a +# store object on the blocking pool. +tokio-util = { workspace = true, features = ["rt", "io", "io-util"] } bytes = { workspace = true } futures = { workspace = true } diff --git a/crates/engine/src/backup.rs b/crates/engine/src/backup.rs index 43c7bc182..84d00dc52 100755 --- a/crates/engine/src/backup.rs +++ b/crates/engine/src/backup.rs @@ -29,14 +29,16 @@ use extenddb_storage::backup_catalog::{ }; use extenddb_storage::error::StorageError; use extenddb_storage::{ - BackupCatalog, BackupRestoreSource, LegacyBackupReader, MetadataEngine, RestoreRequest, - RestoreTableOverrides, SnapshotEngine, TableEngine, + BackupCatalog, BackupRestoreSource, DataEngine, LegacyBackupReader, MetadataEngine, + RestoreEngine, RestoreRequest, RestoreSource, RestoreTableOverrides, SnapshotEngine, + TableEngine, }; use futures::TryStreamExt; use serde_json::{Value, json}; pub use job::KillPoint; pub use maintenance::HeldClaim; +pub use restore::{OTHER_STORE, RESTORE_REGISTRATION_TIMEOUT, TARGET_NOT_EMPTY}; pub use runtime::{ BackupRuntime, ConfiguredStore, JobSettings, LOCAL_CHECK_TIMEOUT, S3_CHECK_TIMEOUT, Spool, StoreHealth, StoreHealthReport, check_configured_store, read_identity_bounded, @@ -1464,13 +1466,21 @@ async fn restore_table_from_backup_detached( after: F, ) -> Result where - S: BackupCatalog + LegacyBackupReader + TableEngine + ?Sized + Send + Sync + 'static, + S: BackupCatalog + + LegacyBackupReader + + TableEngine + + DataEngine + + RestoreEngine + + ?Sized + + Send + + Sync + + 'static, F: FnOnce(Arc, String) -> Fut + Send + 'static, Fut: std::future::Future + Send, { tokio::spawn(async move { let (target_table_name, response) = - restore_table_from_backup(&*storage, &runtime, &account_id, ®ion, &limits, &body) + restore_table_from_backup(&storage, &runtime, &account_id, ®ion, &limits, &body) .await?; after(account_id, target_table_name).await; Ok(response) @@ -1496,8 +1506,15 @@ where /// `BackupInUseException`, a DELETING, DELETED or FAILED row /// `BackupNotFoundException`, a row of another store a /// `ValidationException`, a row of the configured store while it is -/// unhealthy one naming the store's kind), else the legacy reader's restore -/// source; (5) refuse what no backend restores faithfully (multi-part key, vector +/// unhealthy one naming the store's kind). An AVAILABLE row of the healthy +/// configured store is restored from the store by the restore job +/// ([`restore::request`]: the permit, the manifest and its checks, the +/// vector refusal of a target without vector support, the overrides, the +/// detached registration whose answer the request awaits; the maintenance +/// claim is checked by the job once its record exists, the design's 5.3 +/// step 4), and the steps below are the legacy path's, taken when no row +/// exists: the legacy reader's restore source; (5) refuse what no backend +/// restores faithfully from an in-catalog backup (multi-part key, vector /// indexes); (6) check the throughput override against the backup's billing /// mode, build the target's `CreateTableInput`, and apply the index and /// on-demand throughput overrides to it, refusing what the service refuses @@ -1510,14 +1527,25 @@ where /// being restored into `TableInUseException` (the backend checks the name /// again, fenced, under the registration), a backup being restored from /// `BackupInUseException`. -async fn restore_table_from_backup( - storage: &S, +async fn restore_table_from_backup( + storage: &Arc, runtime: &BackupRuntime, account_id: &str, region: &str, limits: &LimitsConfig, body: &Value, -) -> Result<(String, Value), DynamoDbError> { +) -> Result<(String, Value), DynamoDbError> +where + S: BackupCatalog + + LegacyBackupReader + + TableEngine + + DataEngine + + RestoreEngine + + ?Sized + + Send + + Sync + + 'static, +{ // 1. The overrides, parsed and shape-checked: no storage I/O, so a // malformed request is refused before a missing backup can take // precedence over the request's own error. @@ -1532,17 +1560,31 @@ async fn restore_table_from_backup( +/// ([`BackupRuntime::require_healthy_store`]). An AVAILABLE row of the +/// healthy configured store passes, and that store is returned: the restore +/// reads the backup from it ([`restore::request`]). +async fn check_catalog_row_for_restore<'r, S: BackupCatalog + ?Sized>( storage: &S, - runtime: &BackupRuntime, + runtime: &'r BackupRuntime, arn: &BackupArn, row: &BackupRow, -) -> Result<(), DynamoDbError> { +) -> Result<&'r ConfiguredStore, DynamoDbError> { let arn_text = arn.to_arn_string(); match row.status { BackupStatus::Creating => return Err(backup_in_use(&arn_text, None)), @@ -1619,10 +1661,7 @@ async fn check_catalog_row_for_restore( if !runtime.is_configured_store(&row.store) { return Err(backup_in_another_store(&arn_text)); } - runtime.require_healthy_store()?; - Err(DynamoDbError::ValidationException(format!( - "Backup {arn_text} is in a backup store; this release restores in-catalog backups only" - ))) + runtime.require_healthy_store() } /// Step 3 of a restore, the target name under the account, read through @@ -1666,7 +1705,9 @@ async fn check_restore_target_name( } } -/// The RestoreTableFromBackup response body for the backend's description. +/// The RestoreTableFromBackup response body for the backend's description, +/// on both paths (`source` is the legacy reader's source or the manifest's +/// three members). /// /// The service returns the target CREATING with `RestoreSummary` /// { `SourceBackupArn`, `SourceTableArn`, `RestoreDateTime` = the backup's @@ -1677,7 +1718,7 @@ async fn check_restore_target_name( /// ground-truth runs of 2026-08-24). fn restore_response( mut desc: TableDescription, - source: &BackupRestoreSource, + source: &RestoreSource, on_demand: bool, ) -> Result { desc.table_status = TableStatus::Creating; @@ -1700,9 +1741,9 @@ fn restore_response( } // The readiness invariant is applied on every path that hands a - // description to a client. A restored table has no vector indexes (the - // restore refuses them), so this holds trivially, but the invariant claims - // to cover every such path. + // description to a client: a legacy restore's target has no vector + // indexes (the reader refuses them); a store-backed restore's may, and + // the backend reports them as its CreateTable does. desc.validate_vector_index_readiness()?; // Same rules as every other path that emits a table description; the // target is CREATING, so the summary set above is kept. @@ -1856,6 +1897,7 @@ mod job; pub mod maintenance; mod overrides; pub mod reconciler; +mod restore; mod runtime; pub mod store_identity; #[cfg(test)] diff --git a/crates/engine/src/backup/job.rs b/crates/engine/src/backup/job.rs index df6120eaa..a8433de0c 100644 --- a/crates/engine/src/backup/job.rs +++ b/crates/engine/src/backup/job.rs @@ -45,6 +45,8 @@ //! //! Kill points ([`kill_point`]) pause the job for the binary-level crash //! tests under the `backup-test-hooks` feature and are no-ops without it. +//! The restore job ([`super::restore`]) shares this module's environment +//! ([`JobEnv`]), its kill points, its heartbeat and its status writes. pub(crate) mod failure; pub(crate) mod heartbeat; @@ -74,10 +76,10 @@ use extenddb_storage::{ use tokio::sync::oneshot; use tokio::time::Instant; -use super::runtime::{BackupRuntime, ConfiguredStore, JobSettings}; +use super::runtime::{BackupRuntime, ConfiguredStore, JobSettings, Spool}; use super::{create_backup_storage_err_to_dynamo, storage_err_to_dynamo}; use failure::{FailurePath, Written}; -use heartbeat::HeartbeatHandle; +use heartbeat::{HeartbeatHandle, HeartbeatMetric}; use writer::{DataFiles, Pipeline, Stop}; /// The reason a row carries when the maintenance claim was held as the @@ -87,14 +89,17 @@ pub(crate) const MAINTENANCE: &str = "maintenance"; pub(crate) const SHUTDOWN: &str = "shutdown"; /// What the runtime hands a job: the store, its identity and manifest -/// names, the settings, the collector, the shutdown token and the source -/// names. +/// names, the restore spool, the settings, the collector, the shutdown +/// token and the source names. #[derive(Clone)] pub(crate) struct JobEnv { pub store: Arc, pub identity: StoreIdentity, pub bucket: String, pub key_prefix: String, + /// The restore spool of an S3 store; `None` for a filesystem store, + /// whose files a restore reads in place. + pub spool: Option, pub settings: JobSettings, pub metrics: Option>, pub shutdown: CancellationToken, @@ -120,6 +125,7 @@ impl JobEnv { identity, bucket: store.bucket.clone(), key_prefix: store.key_prefix.clone(), + spool: store.spool.clone(), settings: runtime.job_settings(), metrics: runtime.metrics().cloned(), shutdown: runtime.shutdown().clone(), @@ -349,7 +355,7 @@ async fn fail_before_start(claim: &dyn JobClaim, arn: &BackupArn, reason: &str) completed: true, ..StatusFields::default() }; - match failure::write_status(claim, JobStatus::Failed, &fields).await { + match failure::write_status(claim, JobStatus::Creating, JobStatus::Failed, &fields).await { Written::Matched => {} Written::NotMatched(why) | Written::Error(why) => tracing::error!( "backup {}: the row could not be marked FAILED ({reason}): {why}", @@ -380,8 +386,13 @@ pub(crate) async fn run(job: Job) -> BackupJobOutcome { let heartbeat = HeartbeatHandle::start( Arc::clone(&claim), env.settings.heartbeat_interval, - env.metrics.clone(), - created_at, + match env.metrics.clone() { + Some(metrics) => HeartbeatMetric::BackupSnapshotAge { + metrics, + created_at, + }, + None => HeartbeatMetric::None, + }, claimed_at, ); tracing::info!( @@ -538,7 +549,8 @@ async fn write_documents( source_table_details_json: serde_json::to_value(&manifest.source_table).ok(), completed: true, }; - match failure::write_status(&**claim, JobStatus::Available, &fields).await { + match failure::write_status(&**claim, JobStatus::Creating, JobStatus::Available, &fields).await + { Written::Matched => Ok(bytes_uploaded), Written::NotMatched(why) | Written::Error(why) => Err(Stop::ClaimLost(format!( "the AVAILABLE write did not settle: {why}" @@ -574,11 +586,17 @@ pub(crate) async fn put_document(p: &Pipeline<'_>, name: &str, bytes: &[u8]) -> } } -/// The points at which a job pauses under `backup-test-hooks` when -/// `EXTENDDB_BACKUP_PAUSE` names one, until the file -/// `EXTENDDB_BACKUP_PAUSE_RELEASE` names exists (or for good, for a test -/// that kills the server there). Each is logged as `backup job reached kill -/// point `. +/// The points at which a backup or restore job pauses under +/// `backup-test-hooks` when `EXTENDDB_BACKUP_PAUSE` names one, until the +/// file `EXTENDDB_BACKUP_PAUSE_RELEASE` names exists (or for good, for a +/// test that kills the server there). Each is logged as `job reached kill +/// point ` (the backend's own points log `restore job reached kill +/// point `; the harness matches `reached kill point `). This +/// enum is the engine's list: the points inside a backend's own writes are +/// paused by the backend through +/// `extenddb_storage::backup_catalog::restore_pause_point` under +/// `extenddb-storage/test-hooks`, and each backend's own point is named +/// beside `restore_pause_point`, once. #[derive(Debug, Clone, Copy, PartialEq, Eq)] pub enum KillPoint { /// After the put of the first data file returned, before the next file @@ -588,6 +606,22 @@ pub enum KillPoint { BeforeManifest, /// After the manifest's put returned, before the AVAILABLE write. AfterManifest, + /// Restore: after every data file downloaded and verified, before the + /// first `bulk_load`. + RestoreAfterDownload, + /// Restore: after the first file's `bulk_load` returned, before the + /// next file. + RestoreAfterFileCommit, + /// Restore: after a `finish_restore` that left the ACTIVE flip to the + /// backend's index backfill returned, before the job ends (MongoDB). + RestoreMongodbBeforeWorkerFlip, + /// Restore: after the FAILED write matched, before + /// `remove_restore_target`. + RestoreBetweenFailedAndRemoval, + /// Restore: before the DONE write, with the job's heartbeat stopped + /// first, so a reconciler takes the record over while the job pauses + /// and the job's write then matches no record. + RestoreFencedOut, } impl KillPoint { @@ -598,10 +632,30 @@ impl KillPoint { Self::AfterDataFile => "after_data_file", Self::BeforeManifest => "before_manifest", Self::AfterManifest => "after_manifest", + Self::RestoreAfterDownload => "restore_after_download", + Self::RestoreAfterFileCommit => "restore_after_file_commit", + Self::RestoreMongodbBeforeWorkerFlip => "restore_mongodb_before_worker_flip", + Self::RestoreBetweenFailedAndRemoval => "restore_between_failed_and_removal", + Self::RestoreFencedOut => "restore_fenced_out", } } } +/// Whether the environment names `point` as the pause point (so a caller +/// can do what the pause needs first, as the fenced-out point stops the +/// heartbeat); `false` without `backup-test-hooks`. +pub(crate) fn kill_point_named(point: KillPoint) -> bool { + #[cfg(feature = "backup-test-hooks")] + { + std::env::var("EXTENDDB_BACKUP_PAUSE").ok().as_deref() == Some(point.name()) + } + #[cfg(not(feature = "backup-test-hooks"))] + { + let _ = point; + false + } +} + /// Pause at `point` if the environment names it (type doc); a ready future /// without `backup-test-hooks`. #[cfg_attr(not(feature = "backup-test-hooks"), allow(clippy::unused_async))] @@ -613,7 +667,7 @@ pub(crate) async fn kill_point(point: KillPoint, shutdown: &CancellationToken) { } let release = std::env::var("EXTENDDB_BACKUP_PAUSE_RELEASE").ok(); tracing::warn!( - "backup job reached kill point {}; paused until {} exists", + "job reached kill point {}; paused until {} exists", point.name(), release .as_deref() @@ -623,12 +677,12 @@ pub(crate) async fn kill_point(point: KillPoint, shutdown: &CancellationToken) { if let Some(release) = &release && std::path::Path::new(release).exists() { - tracing::warn!("backup job released from kill point {}", point.name()); + tracing::warn!("job released from kill point {}", point.name()); return; } if shutdown.is_cancelled() { tracing::warn!( - "backup job at kill point {} released by the shutdown", + "job at kill point {} released by the shutdown", point.name() ); return; diff --git a/crates/engine/src/backup/job/failure.rs b/crates/engine/src/backup/job/failure.rs index 398ab7880..c0a2eba2d 100644 --- a/crates/engine/src/backup/job/failure.rs +++ b/crates/engine/src/backup/job/failure.rs @@ -5,15 +5,17 @@ //! path. //! //! Every status write goes through [`write_status`]: the backend writes -//! conditioned on the claim's generation and on the row being CREATING, and -//! this module gives the three outcomes one meaning. `Matched` is the -//! write. `NotMatched` means the row is no longer this claim's (gone, -//! another generation, or not CREATING) and the caller makes no further -//! store or table access of any kind. `Unknown` (the connection failed -//! after the statement was sent) is followed by a re-read of the row, not -//! treated as a failure: a row that shows the write landed is `Matched`; a -//! row still CREATING under this generation did not take the write, which -//! is sent once more, at most; anything else is `NotMatched`. Each unknown +//! conditioned on the claim's generation and on the record being in the +//! status the job holds it in (`from`: CREATING for a backup row, RUNNING +//! for a restore record), and this module gives the three outcomes one +//! meaning. `Matched` is the write. `NotMatched` means the record is no +//! longer this claim's (gone, another generation, or not in `from`) and the +//! caller makes no further store or table access of any kind. `Unknown` +//! (the connection failed after the statement was sent) is followed by a +//! re-read of the record, not treated as a failure: a record that shows the +//! write landed is `Matched`; one still in `from` under this generation did +//! not take the write, which is sent once more, at most; anything else is +//! `NotMatched`. Each unknown //! outcome has its re-read, the second write's included, so a second //! unknown outcome whose write landed is `Matched` (the row is in the //! target status under this generation) and one whose write was lost is @@ -28,11 +30,13 @@ //! job stops, the session's end frees the advisory lock, and the row is the //! reconciler's (AVAILABLE from a complete manifest, FAILED otherwise). //! -//! The failure path ([`fail_row`]) is the FAILED write followed, only when -//! it matched, by `delete_prefix` of the backup's prefix; the holder of the -//! claim owns the prefix, so a job whose FAILED write did not match never -//! deletes. A deletion that fails leaves the prefix to the reconciler, -//! which deletes a FAILED row's prefix once its heartbeat is stale. +//! The backup job's failure path ([`fail_row`]) is the FAILED write +//! followed, only when it matched, by `delete_prefix` of the backup's +//! prefix; the holder of the claim owns the prefix, so a job whose FAILED +//! write did not match never deletes. A deletion that fails leaves the +//! prefix to the reconciler, which deletes a FAILED row's prefix once its +//! heartbeat is stale. The restore job's failure path, FAILED then the +//! target's removal under the same gate, is `super::super::restore::fail_restore`. use extenddb_backup_store::BackupStore; use extenddb_core::metrics::MetricsCollector; @@ -51,23 +55,24 @@ pub(crate) enum Written { Error(String), } -/// Move the row from CREATING to `to` through `claim` (module doc). +/// Move the record from `from` to `to` through `claim` (module doc). pub(crate) async fn write_status( claim: &dyn JobClaim, + from: JobStatus, to: JobStatus, fields: &StatusFields, ) -> Written { for attempt in 0..2 { - let outcome = match claim.set_status(JobStatus::Creating, to, fields).await { + let outcome = match claim.set_status(from, to, fields).await { Ok(outcome) => outcome, Err(e) => return Written::Error(format!("the {to} write failed: {e}")), }; match outcome { WriteOutcome::Matched => return Written::Matched, WriteOutcome::NotMatched => { - return Written::NotMatched( - "the row is no longer CREATING under this claim's generation".to_owned(), - ); + return Written::NotMatched(format!( + "the record is no longer {from} under this claim's generation" + )); } WriteOutcome::Unknown => {} } @@ -89,14 +94,14 @@ pub(crate) async fn write_status( if row.status == to { return Written::Matched; } - if row.status != JobStatus::Creating || attempt == 1 { + if row.status != from || attempt == 1 { return Written::NotMatched(format!( - "the row is {} after a {to} write whose outcome was unknown", + "the record is {} after a {to} write whose outcome was unknown", row.status )); } tracing::warn!( - "the {to} write's outcome was unknown and the row is still CREATING under this \ + "the {to} write's outcome was unknown and the record is still {from} under this \ claim; writing once more" ); } @@ -129,7 +134,7 @@ pub(crate) async fn fail_row( completed: true, ..StatusFields::default() }; - match write_status(claim, JobStatus::Failed, &fields).await { + match write_status(claim, JobStatus::Creating, JobStatus::Failed, &fields).await { Written::Matched => {} Written::NotMatched(why) | Written::Error(why) => return FailurePath::NoStoreAccess(why), } diff --git a/crates/engine/src/backup/job/heartbeat.rs b/crates/engine/src/backup/job/heartbeat.rs index ed66fd175..fb99e463b 100644 --- a/crates/engine/src/backup/job/heartbeat.rs +++ b/crates/engine/src/backup/job/heartbeat.rs @@ -13,13 +13,24 @@ //! rule read the other way: the last confirmed beat is at most two periods //! old (one for the timer to fire, one for the beat to return, the bound //! the timer itself enforces) and no loss has been raised. The claim's -//! insert stands for the first confirmed beat: the check measures from the -//! instant the job took before `claim_backup`, so the maintenance check -//! that runs between the insert and the handle's start (one catalog query -//! with no bound of its own) counts against the two periods, and a job -//! whose first put comes within two periods of the insert needs no beat of -//! its own; `stale_job_after_secs` is at least three periods, so a row that -//! passes the check is never one a reconciler may take over. +//! insert stands for the first confirmed beat, and the two jobs hand its +//! instant in by their own convention: the backup job takes it before +//! `claim_backup` (the row's `heartbeat_at` is stamped inside the insert, so +//! the instant is at or before the beat and the maintenance check that runs +//! between the insert and the handle's start, one catalog query with no +//! bound of its own, counts against the two periods); the restore job takes +//! it after its first beat returned confirmed, a beat it sends itself right +//! after `claim_restore` returned (the record's `heartbeat_at` as the +//! registration stamped it can be old: on PostgreSQL `now()` is the +//! transaction's start, so a registration stalled on a lock commits a stale +//! beat; the first beat refreshes it on the catalog's clock), and after +//! `claim_restore_dead` returned for a resume (the takeover stamped +//! `heartbeat_at` inside that write). Either way the instant is at or after +//! the beat it stands for, by a write's own duration, and a slow +//! registration does not end the job at its first check. A job +//! whose first write comes within two periods of its instant needs no beat +//! of its own; `stale_job_after_secs` is at least three periods, so a row +//! that passes the check is never one a reconciler may take over. //! //! A process stopped and resumed (SIGSTOP, a paused VM) meets the same //! rule. On PostgreSQL its claim session keeps the advisory lock while it @@ -34,6 +45,16 @@ //! during the stop: a put in flight at the stop may still land (the //! reconciler's FAILED-prefix retry deletes it), and the job's next beat //! and status write match no row. +//! +//! What holds a beat is the backend's. On SQLite a beat is an `UPDATE` +//! under the engine's write lock, so it queues behind every writer that +//! holds or waits for that lock (a restore's write batches, item writes, +//! the GSI backfill, the TTL sweep, a table's drop); one that waits past a +//! period ends the job as a lost claim like any other late beat: the +//! reconciler resumes a store-backed restore from its last committed batch +//! and resolves a backup's row as it resolves any dead owner's. +//! The SQLite restore engine's module doc gives the batch times the bench +//! record measured. use std::sync::{Arc, Mutex, PoisonError}; use std::time::Duration; @@ -54,17 +75,27 @@ pub(crate) struct HeartbeatHandle { timer: tokio::task::JoinHandle<()>, } +/// What a confirmed beat records, if anything: a backup job records the +/// snapshot's age, a restore job records nothing per beat. +pub(crate) enum HeartbeatMetric { + /// `BackupSnapshotAge`, seconds since `created_at`, into `metrics`. + BackupSnapshotAge { + metrics: Arc, + created_at: f64, + }, + /// No per-beat metric. + None, +} + impl HeartbeatHandle { /// Start the timer on `claim`, beating every `period` from one period - /// after now; `claimed_at`, the instant of the claim's insert, stands - /// for the first confirmed beat. With `metrics`, each confirmed beat - /// records the snapshot's age (`BackupSnapshotAge`), seconds since - /// `created_at`. + /// after now; `claimed_at`, the instant of the claim's insert as the + /// job's convention takes it (module doc), stands for the first + /// confirmed beat. Each confirmed beat records `metric`. pub(crate) fn start( claim: Arc, period: Duration, - metrics: Option>, - created_at: f64, + metric: HeartbeatMetric, claimed_at: Instant, ) -> Self { let (tx, last_confirmed) = watch::channel(claimed_at); @@ -82,8 +113,12 @@ impl HeartbeatHandle { let why = match tokio::time::timeout(period, claim.heartbeat()).await { Ok(Ok(Heartbeat::Confirmed)) => { let _ = tx.send(Instant::now()); - if let Some(metrics) = &metrics { - metrics.record_backup_snapshot_age(seconds_since(created_at)); + if let HeartbeatMetric::BackupSnapshotAge { + metrics, + created_at, + } = &metric + { + metrics.record_backup_snapshot_age(seconds_since(*created_at)); } continue; } @@ -96,7 +131,11 @@ impl HeartbeatHandle { format!("the heartbeat did not return within {} s", period.as_secs()) } }; - *reason.lock().unwrap_or_else(PoisonError::into_inner) = Some(why); + *reason.lock().unwrap_or_else(PoisonError::into_inner) = Some(why.clone()); + tracing::warn!( + "the job's heartbeat raised the loss of its claim; the job stops at its \ + next check: {why}" + ); lost.cancel(); return; } @@ -125,19 +164,66 @@ impl HeartbeatHandle { .unwrap_or_else(|| "the claim is lost".to_owned()) } + /// Why the pre-put check failed, for the job's stop line: the timer's + /// own reason when it raised the loss ([`Self::reason`]), else the + /// second clause, that no beat confirmed the claim within two periods, + /// naming `step`, what the job was about to do ("to load X", "for the + /// DONE write"). + pub(crate) fn why_not_confirmed(&self, step: &str) -> String { + if self.lost.is_cancelled() { + self.reason() + } else { + format!("no heartbeat confirmed the claim recently enough {step}") + } + } + /// The pre-put check (module doc): no loss raised and the last confirmed /// beat at most two periods old. The second clause is what catches a /// timer task that stopped running without raising the loss. pub(crate) fn confirms_the_claim(&self) -> bool { - !self.lost.is_cancelled() && self.last_confirmed.borrow().elapsed() <= 2 * self.period + self.claim_check().confirms_the_claim() + } + + /// The pre-put check as a value a blocking thread can hold (the + /// restore's decoder checks it at every chunk boundary): the same two + /// clauses read from the same watch and token. + pub(crate) fn claim_check(&self) -> ClaimCheck { + ClaimCheck { + last_confirmed: self.last_confirmed.clone(), + lost: self.lost.clone(), + period: self.period, + } } /// End the timer without raising the loss: a timer task that stopped - /// running, for the test of the pre-put check's second clause. - #[cfg(test)] + /// running, for the test of the pre-put check's second clause and for + /// the restore's fenced-out kill point, which lets a reconciler take the + /// record over while the job pauses. pub(crate) fn stop_timer(&self) { self.timer.abort(); } + + /// A handle that ends the timer as [`Self::stop_timer`] does, for a test + /// that holds no reference to the job's handle. + #[cfg(test)] + pub(crate) fn timer_abort_handle(&self) -> tokio::task::AbortHandle { + self.timer.abort_handle() + } +} + +/// [`HeartbeatHandle::confirms_the_claim`] detached from the handle. +#[derive(Clone)] +pub(crate) struct ClaimCheck { + last_confirmed: watch::Receiver, + lost: CancellationToken, + period: Duration, +} + +impl ClaimCheck { + /// No loss raised and the last confirmed beat at most two periods old. + pub(crate) fn confirms_the_claim(&self) -> bool { + !self.lost.is_cancelled() && self.last_confirmed.borrow().elapsed() <= 2 * self.period + } } impl Drop for HeartbeatHandle { diff --git a/crates/engine/src/backup/maintenance.rs b/crates/engine/src/backup/maintenance.rs index 09e911597..a834c5e9f 100644 --- a/crates/engine/src/backup/maintenance.rs +++ b/crates/engine/src/backup/maintenance.rs @@ -2,7 +2,8 @@ // SPDX-License-Identifier: Apache-2.0 //! The holder's side of the catalog-level maintenance claim, and the steps a -//! maintenance command (`extenddb backup purge`) runs under it. +//! maintenance command (`extenddb backup purge`, `extenddb backup +//! remove-target`) runs under it. //! //! [`HeldClaim`] is the handle the command keeps from the take to the //! release, with the heartbeat running on a timer for the handle's whole @@ -23,6 +24,23 @@ //! operator's Ctrl-C ends the wait at once and the deletion loop before its //! next row, never inside one (the row's own steps are the idempotent //! sequence a second deleter finishes). +//! +//! [`remove_target`] is the whole of +//! `extenddb backup remove-target --account
`: the claim +//! taken and held for its duration, the table's +//! restore record read, the refusals, the removal through +//! `remove_restore_target`, the release. It is the one command that removes +//! a restore target DeleteTable refuses (a CREATING table): a FAILED +//! record's target, whatever the reason (the reconciler leaves a +//! `target_not_empty` one for this command; on MongoDB and Cassandra a +//! crash before the first progress record, the whole first data file on +//! MongoDB and the first 10,000 items or 32 MiB on Cassandra, ends in that +//! state), and a RUNNING record's target whose owner is dead (the backend's +//! `list_dead_restore_jobs` names it under the owner window: no heartbeat +//! within the window on the catalog's clock); a legacy record's target the +//! same way. It refuses a RUNNING record whose owner is live, a DONE record +//! (the table is ACTIVE, or on MongoDB awaiting its index backfill) and a +//! table no restore record names, the latter two as "not a restore target". use std::sync::{Arc, Mutex, PoisonError}; use std::time::Duration; @@ -30,9 +48,10 @@ use std::time::Duration; use extenddb_backup_store::BackupStore; use extenddb_core::backup_format::{BackupArn, EXTENDDB_MANIFEST_FILE}; use extenddb_storage::backup_catalog::{ - BackupRow, BackupStatus, DeleteStart, Heartbeat, JobStatus, StaleClock, StoreIdentity, - is_backup_not_found, + BackupRow, BackupStatus, DeleteStart, Heartbeat, JobStatus, RestoreJobRow, RestoreKind, + StaleClock, StoreIdentity, is_backup_not_found, }; +use extenddb_storage::error::StorageError; use extenddb_storage::{BackupCatalog, MaintenanceClaim}; use futures::TryStreamExt; use tokio::sync::watch; @@ -101,7 +120,7 @@ impl HeldClaim { /// /// The backend's, when the release could not be written; the claim then /// expires by the stale rule. - pub async fn release(mut self) -> Result<(), extenddb_storage::error::StorageError> { + pub async fn release(mut self) -> Result<(), StorageError> { self.timer.abort(); // The abort lands at the task's next await; a heartbeat already sent // finishes first, so the release never overlaps it on the claim's @@ -403,6 +422,159 @@ pub async fn purge_row( } } +/// What `extenddb backup remove-target` is asked to do. +#[derive(Debug, Clone)] +pub struct RemoveTarget { + /// The account the table belongs to. + pub account_id: String, + /// The table's name. + pub table_name: String, + /// The maintenance claim's holder label (the command and its host). + pub holder: String, + /// `stale_job_after_secs`: the claim's stale window. + pub stale_after: Duration, + /// The window `list_dead_restore_jobs` is asked with: a RUNNING record + /// the backend does not list as dead under it (a heartbeat within the + /// window on the catalog's clock) has a live owner and is refused; zero + /// where every owner is known dead (SQLite under the serve lock, where + /// a zero window lists every RUNNING record). + pub owner_window: Duration, + /// The claim's heartbeat period (the jobs', `JOB_HEARTBEAT_INTERVAL`). + pub heartbeat_interval: Duration, +} + +/// What [`remove_target`] removed: the record as it was read, for the +/// command's line. +#[derive(Debug, Clone, PartialEq)] +pub struct RemovedTarget { + /// The target's table id. + pub table_id: String, + /// The backup the restore read. + pub backup_arn: String, + /// The record's kind. + pub kind: RestoreKind, + /// The record's status when it was read: FAILED, or RUNNING with a + /// dead owner. + pub status: JobStatus, + /// The FAILED record's reason. + pub failure_reason: Option, +} + +/// `extenddb backup remove-target` (module doc): under the maintenance +/// claim, taken here and released before the return, remove the CREATING +/// target of the restore record that names `request`'s table and the record +/// with it, through `remove_restore_target`. +/// +/// # Errors +/// +/// The text for the operator: the claim held by a live holder; a table no +/// record names, a DONE record, or a target no longer a CREATING restore +/// target with the record's id ("not a restore target"); a RUNNING record +/// the backend does not list as dead under the owner window; the claim lost +/// before the removal; a catalog call failing. +pub async fn remove_target( + storage: &S, + request: &RemoveTarget, +) -> Result { + let claim = storage + .maintenance_claim(&request.holder, request.stale_after) + .await + .map_err(|e| match e { + StorageError::AlreadyExists(m) => format!( + "{m}; one maintenance command runs at a time, and a claim whose holder died \ + expires after stale_job_after_secs ({} s)", + request.stale_after.as_secs() + ), + other => format!("cannot take the maintenance claim: {other}"), + })?; + let claim = HeldClaim::start(claim, request.heartbeat_interval); + let outcome = remove_target_under(storage, request, &claim).await; + if let Err(e) = claim.release().await { + tracing::warn!("the maintenance claim could not be released; it expires on its own: {e}"); + } + outcome +} + +/// The steps of [`remove_target`] between the take and the release. +async fn remove_target_under( + storage: &S, + request: &RemoveTarget, + claim: &HeldClaim, +) -> Result { + let what = format!("{} (account {})", request.table_name, request.account_id); + let records: Vec = storage + .list_restore_jobs(None) + .await + .map_err(|e| format!("cannot read the restore job records: {e}"))?; + let record = records + .into_iter() + .find(|r| { + r.target.account_id == request.account_id && r.target.table_name == request.table_name + }) + .ok_or_else(|| format!("{what} is not a restore target: no restore record names it"))?; + match record.status { + JobStatus::Done => { + return Err(format!( + "{what} is not a restore target: its restore from {} is DONE (the table is \ + ACTIVE, or awaits its index backfill)", + record.source_backup_arn + )); + } + JobStatus::Running => { + // The owner is dead only if the backend's own listing says so: + // `list_dead_restore_jobs` compares the heartbeat with the + // catalog's clock, the one the owner stamps, so a skew of this + // host's clock can neither refuse a dead owner nor remove under + // a live one. A zero window (SQLite under the serve lock) lists + // every RUNNING record, as it does for the purge's wait. + let dead = storage + .list_dead_restore_jobs(request.owner_window) + .await + .map_err(|e| format!("cannot list the dead restore jobs: {e}"))? + .iter() + .any(|r| r.target.table_id == record.target.table_id); + if !dead { + return Err(format!( + "the restore into {what} from {} is RUNNING with a live owner (a heartbeat \ + within stale_job_after_secs); stop it or wait for it", + record.source_backup_arn + )); + } + } + JobStatus::Failed | JobStatus::Creating | JobStatus::Available => {} + } + if let Some(reason) = claim.lost() { + return Err(format!("stopped before the removal: {reason}")); + } + let removed = storage + .remove_restore_target(&record.target) + .await + .map_err(|e| format!("cannot remove the restore target {what}: {e}"))?; + if !removed { + return Err(format!( + "{what} is not a restore target: it is no longer a CREATING restore target with \ + table id {}", + record.target.table_id + )); + } + tracing::info!( + target: "extenddb::audit", + action = "backup remove-target", + account_id = %record.target.account_id, + table_name = %record.target.table_name, + table_id = %record.target.table_id, + backup_arn = %record.source_backup_arn, + "removed the restore target and its record" + ); + Ok(RemovedTarget { + table_id: record.target.table_id, + backup_arn: record.source_backup_arn, + kind: record.kind, + status: record.status, + failure_reason: record.failure_reason, + }) +} + #[cfg(test)] mod tests { use extenddb_storage::error::StorageError; diff --git a/crates/engine/src/backup/reconciler.rs b/crates/engine/src/backup/reconciler.rs index 1f47d1b92..73ea84d8e 100644 --- a/crates/engine/src/backup/reconciler.rs +++ b/crates/engine/src/backup/reconciler.rs @@ -24,40 +24,75 @@ //! //! The duties, in the order a cycle runs them: //! +//! - With a store configured: the store's check runs again +//! ([`super::check_configured_store`]: `validate` while it has not passed, +//! then the identity check and manifest probe) and sets the store's +//! health, so a store whose checks run after the listener is up, or that +//! failed at startup or later, is healthy once a check passes, and one +//! that fails is refused from the rest of this cycle on: every duty below +//! that reads the store's health reads the health this cycle's check left. +//! The check runs under the store's bound +//! ([`super::ConfiguredStore::check_timeout`]: 5 s for a filesystem root, +//! 30 s for a bucket), so a store that has stopped answering costs each +//! cycle that bound and nothing more: the duties run every `interval` plus +//! the bound, the check is recorded as a transport failure, and a +//! filesystem check leaves one blocking thread parked on the mount until +//! it answers, after which no check starts until that call returns (the +//! runtime module's in-flight rule), so a mount that stays stalled parks +//! one thread and not one per cycle. The check is also selected against +//! the shutdown token, so a cycle that is checking when the server stops +//! returns at once with no further duty and the drain finishes. //! - Over the records `list_dead_restore_jobs` returns: a RUNNING restore //! job record whose `heartbeat_at` is older than `stale_job_after` (on //! the catalog's clock) has a dead owner. One of kind `legacy` is marked //! FAILED and its target removed through `remove_restore_target`, never //! resumed: an in-catalog copy has no data files or progress to resume -//! from. One of kind `manifest` is left for PR 9a's reconciler, the -//! release with the restore job, which resumes it, and logged once per -//! record for as long as this process runs. A FAILED restore job record -//! whose table is still a CREATING restore target, the FAILED-target -//! duty: the FAILED write landed (the job's own, or an earlier cycle's) -//! and the target's removal did not; the backend lists it once the record -//! is stale, and the target is removed through `remove_restore_target` -//! with no second FAILED write. +//! from. One of kind `manifest` is resumed (`super::restore::resume`), +//! in this order: once the server's shutdown has begun (the jobs' token is +//! cancelled) the record is left as it is, with no takeover and no FAILED +//! write, for the next instance's reconciler (a resume spawned on the way +//! out would stop at its first check and leave the record with a fresh +//! heartbeat, a second stale window for the next instance to wait); +//! while the maintenance claim is held it is failed and its target +//! removed like a legacy one ([`OWNER_DEAD`]; a resume is a job, which +//! the claim excludes), except a record with no recorded progress at all +//! (no `file_index`, no progress row) whose target already holds items, +//! which is failed [`super::restore::TARGET_NOT_EMPTY`] through the same +//! unclaimed write and its target left for the operator, as below (the +//! command's wait is satisfied either way: the record is no longer +//! RUNNING); a record of a store other than the +//! configured one (no store configured included) is failed naming +//! `backup sync --move-store`, its target removed by the FAILED-target +//! duty next; while the configured store is unhealthy, or not yet checked, +//! the record is skipped (one line per cycle, naming which); with no free +//! job permit it is skipped to the next cycle; then the backend's +//! `claim_restore_dead` is the test and the takeover (a fresh generation, +//! this host's `spool_host` and a fresh `heartbeat_at`, so a second taker +//! finds the record live at once), and a record it does not hand over has +//! a live owner; the instant the takeover returned is the resumed job's +//! first confirmed beat (the record's `heartbeat_at` was stamped inside +//! the write, so the instant is at or after the beat it stands for by the +//! write's own duration, and the reads that follow count against the +//! heartbeat's window rather than widen it); under the claim, a record with no recorded progress at all +//! (no `file_index`, no progress row) whose target already holds items is +//! FAILED [`super::restore::TARGET_NOT_EMPTY`] and its target left for the +//! operator (`extenddb backup remove-target --account
`), +//! since a catalog restored +//! from a dump can name a restore that finished and was written to since; +//! otherwise the job is spawned through the runtime's tracker and runs +//! from the recorded progress. A FAILED restore job record whose table is +//! still a CREATING restore target, the FAILED-target duty: the FAILED +//! write landed (the job's own, or an earlier cycle's) and the target's +//! removal did not; the backend lists it once the record is stale, and the +//! target is removed through `remove_restore_target` with no second +//! FAILED write, except a record whose reason is `target_not_empty`, +//! whose target is the operator's and is logged with the command. //! - An in-catalog CREATING backup row whose create is dead (reported by the //! backend's `abandoned_creating_backups`) is removed through the reader's //! `delete`, which takes its items with it. -//! - With a store configured: the store's check runs again -//! ([`super::check_configured_store`]: `validate` while it has not passed, -//! then the identity check and manifest probe) and sets the store's -//! health, so a store whose checks run after the listener is up, or that -//! failed at startup or later, is healthy once a check passes, and one -//! that fails is refused from the next cycle on. The check runs under the -//! store's bound ([`super::ConfiguredStore::check_timeout`]: 5 s for a -//! filesystem root, 30 s for a bucket), so a store that has stopped -//! answering costs each cycle that bound and nothing more: the duties -//! above run every `interval` plus the bound, the check is recorded as a -//! transport failure, and a filesystem check leaves one blocking thread -//! parked on the mount until it answers, after which no check starts -//! until that call returns (the runtime module's in-flight rule), so a -//! mount that stays stalled parks one thread and not one per cycle. The -//! check is also selected against the shutdown token, so a cycle that is -//! checking when the server stops returns at once and the drain finishes. -//! While the store is unhealthy the reconciler acts on no row of the -//! store. While it is healthy, on the rows of the configured store: +//! - With a store configured: while the store is unhealthy the reconciler +//! acts on no row of the store. While it is healthy, on the rows of the +//! configured store: //! - Each DELETING row (a DeleteBackup whose deletion did not finish) is //! finished (`finish_backup_delete` in the engine's backup module): the //! manifest, then the prefix, then the row to DELETED. @@ -94,7 +129,26 @@ //! (the completed clock) is removed. Every age is the database's //! comparison; the engine holds no clock. //! - Not while the maintenance claim is held, on every backend: the -//! backend's `sweep_restore_registrations` (a no-op but on Cassandra). +//! backend's `sweep_restore_registrations` (a no-op but on Cassandra); +//! the progress-row sweep (`sweep_restore_progress` with the table ids of +//! the RUNNING restore targets: PostgreSQL's `restore_progress` rows of +//! any other table go, the other backends keep progress on the record and +//! have nothing to sweep); and, with a spool configured, the spool sweep +//! (`super::restore::spool::sweep_entries`: entries whose table id is +//! not a RUNNING restore with this host's `spool_host`). Both sweeps read +//! their candidates before the records: the progress-row sweep's +//! candidates are the backend's, the spool's are listed +//! (`super::restore::spool::list_entries`) before `list_restore_jobs`, +//! so a restore whose registration commits between the two reads is +//! never a candidate of either. A backend that cannot list the records +//! yet logs once per cycle and the two sweeps are skipped. +//! +//! The worker is not gated by the PostgreSQL serving registration: after +//! the server's registration is lost the server refuses new requests and +//! tries to register again for up to 30 s, and cancels the workers' +//! shutdown token only when that fails or a migration holds or waits for +//! the catalog; until then cycles go on, beside a migration that may +//! already have taken the catalog (GitHub issue #755 is the gate). //! //! Every duty acts only on rows of the configured store (kind, root and //! id), the legacy duty excepted. A cycle that fails logs and waits for the @@ -123,11 +177,16 @@ use extenddb_storage::backup_catalog::{ StatusFields, StoreIdentity, }; use extenddb_storage::error::StorageError; -use extenddb_storage::{BackupCatalog, CancellationToken, LegacyBackupReader, sleep_or_shutdown}; +use extenddb_storage::{ + BackupCatalog, CancellationToken, JobPermit, LegacyBackupReader, RestoreEngine, + sleep_or_shutdown, +}; use futures::TryStreamExt; -use super::BackupRuntime; +use super::job::JobEnv; use super::job::failure::{Written, write_status}; +use super::restore::{self, OTHER_STORE, TARGET_NOT_EMPTY}; +use super::{BackupRuntime, ConfiguredStore}; /// The two durations a reconciler reads from `[backup]`. #[derive(Debug, Clone, Copy, PartialEq, Eq)] @@ -160,11 +219,32 @@ pub struct CycleReport { /// this cycle wrote the FAILED or found it), left to their current owner, /// by table name. pub targets_left_to_owner: Vec, - /// Manifest restore jobs with a dead owner, left for their resumer. - pub manifest_restores_seen: Vec, - /// Those of `manifest_restores_seen` this cycle logged: the ones no - /// earlier cycle of this process had seen. - pub manifest_restores_logged: Vec, + /// Manifest restore jobs with a dead owner this cycle took over and + /// spawned a resume of, by table name. + pub manifest_restores_resumed: Vec, + /// Manifest restore jobs with a dead owner this cycle marked FAILED and + /// left their target to the FAILED-target duty or the operator, by table + /// name with the reason: another store, or `target_not_empty`. + pub manifest_restores_failed: Vec<(String, String)>, + /// Manifest restore jobs with a dead owner this cycle marked FAILED + /// `OWNER_DEAD` under the maintenance claim and removed the target of, + /// as a legacy record is, by table name. + pub manifest_restores_failed_and_removed: Vec, + /// Manifest restore jobs with a dead owner skipped to the next cycle: + /// no job permit was free, by table name. + pub restores_skipped_no_permit: Vec, + /// Manifest restore jobs with a dead owner skipped because the + /// configured store is unhealthy, by table name. + pub restores_skipped_store_unhealthy: Vec, + /// FAILED `target_not_empty` records whose target is left for the + /// operator, by table name. + pub targets_left_for_operator: Vec, + /// Progress rows of tables that are not RUNNING restore targets removed + /// (PostgreSQL's `restore_progress`). + pub progress_rows_swept: u64, + /// Spool entries of restores not RUNNING with this host's `spool_host` + /// removed, by table id. + pub spool_entries_swept: Vec, /// In-catalog CREATING backup rows removed, by ARN. pub legacy_backups_removed: Vec, /// The configured store's health after this cycle's check; `None` with @@ -196,6 +276,11 @@ impl CycleReport { pub fn acted(&self) -> bool { !(self.legacy_restores_removed.is_empty() && self.failed_targets_removed.is_empty() + && self.manifest_restores_resumed.is_empty() + && self.manifest_restores_failed.is_empty() + && self.manifest_restores_failed_and_removed.is_empty() + && self.progress_rows_swept == 0 + && self.spool_entries_swept.is_empty() && self.legacy_backups_removed.is_empty() && self.deletions_finished.is_empty() && self.creating_resolved.is_empty() @@ -206,14 +291,12 @@ impl CycleReport { } } -/// What a reconciler carries from one cycle to the next: the manifest -/// records it has already logged, so a record it cannot act on is reported -/// once rather than every cycle, the worker's shutdown token, which the -/// store's check is selected against, and, for a maintenance command's own -/// cycle, that its caller holds the maintenance claim. +/// What a reconciler carries from one cycle to the next: the worker's +/// shutdown token, which the store's check is selected against, and, for a +/// maintenance command's own cycle, that its caller holds the maintenance +/// claim. #[derive(Debug, Default)] pub struct ReconcilerState { - logged_manifest_restores: HashSet, shutdown: Option, /// `Some(true)` when the caller holds the maintenance claim: the cycle /// takes the held branch and does not read the claim. `None` reads it @@ -257,13 +340,13 @@ pub async fn run( metrics: Arc, shutdown: CancellationToken, ) where - S: BackupCatalog + LegacyBackupReader + ?Sized, + S: BackupCatalog + LegacyBackupReader + RestoreEngine + ?Sized + Send + Sync + 'static, { let mut state = ReconcilerState::with_shutdown(shutdown.clone()); loop { let started = std::time::Instant::now(); match cycle( - &*storage, + &storage, &runtime, settings.stale_job_after, &backend, @@ -277,12 +360,18 @@ pub async fn run( metrics.record_worker_success(QuerySource::BackupReconciler, cycle_us); if report.acted() { tracing::info!( - "backup reconciler: failed and removed {} dead legacy restore(s), removed \ - the target(s) of {} FAILED restore record(s) and {} abandoned in-catalog \ - backup(s), finished {} deletion(s), resolved {} dead CREATING row(s), \ - deleted {} FAILED prefix(es), swept {} DELETED prefix(es), removed {} \ - tombstone(s), swept the registrations of {} backup(s)", + "backup reconciler: failed and removed {} dead legacy restore(s), resumed \ + {}, failed {} and failed and removed {} (under the maintenance claim) \ + dead store-backed restore(s), removed the target(s) of {} FAILED restore \ + record(s) and {} abandoned in-catalog backup(s), finished {} \ + deletion(s), resolved {} dead CREATING row(s), deleted {} FAILED \ + prefix(es), swept {} DELETED prefix(es), removed {} tombstone(s), swept \ + the registrations of {} backup(s), {} progress row(s) and {} spool \ + entry(ies)", report.legacy_restores_removed.len(), + report.manifest_restores_resumed.len(), + report.manifest_restores_failed.len(), + report.manifest_restores_failed_and_removed.len(), report.failed_targets_removed.len(), report.legacy_backups_removed.len(), report.deletions_finished.len(), @@ -290,7 +379,9 @@ pub async fn run( report.failed_prefixes_deleted.len(), report.deleted_prefixes_swept.len(), report.tombstones_removed.len(), - report.registrations_swept.len() + report.registrations_swept.len(), + report.progress_rows_swept, + report.spool_entries_swept.len() ); } } @@ -315,14 +406,14 @@ pub async fn run( /// A catalog read that fails ends the cycle; a failed write on one row is /// logged and the cycle goes on to the next row. pub async fn cycle( - storage: &S, + storage: &Arc, runtime: &BackupRuntime, stale_job_after: Duration, backend: &str, state: &mut ReconcilerState, ) -> Result where - S: BackupCatalog + LegacyBackupReader + ?Sized, + S: BackupCatalog + LegacyBackupReader + RestoreEngine + ?Sized + Send + Sync + 'static, { let maintenance_held = match state.maintenance_held { Some(held) => held, @@ -339,13 +430,60 @@ where ); } + // The store's check first, so every duty of this cycle that reads the + // store's health reads what this cycle's check left. + if let Some(store) = runtime.configured_store() { + let check = super::check_configured_store(store, &**storage, backend); + let outcome = match &state.shutdown { + Some(shutdown) => tokio::select! { + outcome = check => Some(outcome), + () = shutdown.cancelled() => None, + }, + None => Some(check.await), + }; + match outcome { + Some(outcome) => { + store.record_check(outcome); + } + None => { + tracing::info!( + "backup store ({}) check was stopped by shutdown; its health is left as the \ + last check left it", + store.kind + ); + report.store_healthy = Some(store.is_healthy()); + return Ok(report); + } + } + report.store_healthy = Some(store.is_healthy()); + } + + let mut store_not_ready_logged = false; for job in storage.list_dead_restore_jobs(stale_job_after).await? { match (job.status, job.kind) { + // A `target_not_empty` target is the operator's. + (JobStatus::Failed, _) if job.failure_reason.as_deref() == Some(TARGET_NOT_EMPTY) => { + tracing::warn!( + "the restore into {} (account {}) from {} is FAILED {TARGET_NOT_EMPTY}: its \ + target ({}) holds items no record of the restore accounts for and is left \ + for the operator; inspect it at the storage level and remove it with \ + extenddb backup remove-target --account {} {}", + job.target.table_name, + job.target.account_id, + job.source_backup_arn, + job.target.table_id, + job.target.account_id, + job.target.table_name + ); + report + .targets_left_for_operator + .push(job.target.table_name.clone()); + } // The FAILED-target duty, whatever the record's kind: the job is // over and only its target is left. (JobStatus::Failed, _) => { match remove_restore_target( - storage, + &**storage, &job, "was FAILED when the reconciler reached it", ) @@ -360,7 +498,7 @@ where Removal::Failed => {} } } - (_, RestoreKind::Legacy) => match remove_dead_legacy_restore(storage, &job).await { + (_, RestoreKind::Legacy) => match remove_dead_legacy_restore(&**storage, &job).await { Some(Removal::Removed) => report .legacy_restores_removed .push(job.target.table_name.clone()), @@ -370,21 +508,41 @@ where Some(Removal::Failed) | None => {} }, (_, RestoreKind::Manifest) => { - if state.logged_manifest_restores.insert(job.target.clone()) { - tracing::info!( - "restore of {} into {} (account {}) has no live owner; a manifest \ - restore is resumed by a later release's reconciler (reported once)", - job.source_backup_arn, - job.target.table_name, - job.target.account_id - ); - report - .manifest_restores_logged - .push(job.target.table_name.clone()); + let name = job.target.table_name.clone(); + match resume_dead_manifest_restore( + storage, + runtime, + stale_job_after, + &job, + report.maintenance_held, + ) + .await + { + Resumed::Spawned => report.manifest_restores_resumed.push(name), + Resumed::Failed(reason) => report.manifest_restores_failed.push((name, reason)), + Resumed::FailedAndRemoved(removal) => match removal { + Removal::Removed => report.manifest_restores_failed_and_removed.push(name), + Removal::LeftToOwner => report.targets_left_to_owner.push(name), + Removal::Failed => {} + }, + Resumed::NoPermit => report.restores_skipped_no_permit.push(name), + Resumed::StoreUnhealthy => { + if !store_not_ready_logged { + store_not_ready_logged = true; + let state = + match runtime.configured_store().map(ConfiguredStore::health) { + Some(super::StoreHealth::Checking) => "not yet checked", + _ => "unhealthy", + }; + tracing::warn!( + "the configured backup store is {state}; dead store-backed \ + restores are not resumed this cycle" + ); + } + report.restores_skipped_store_unhealthy.push(name); + } + Resumed::LeftToOwner | Resumed::Error => {} } - report - .manifest_restores_seen - .push(job.target.table_name.clone()); } } } @@ -408,55 +566,360 @@ where } } - if let Some(store) = runtime.configured_store() { - let check = super::check_configured_store(store, storage, backend); - let outcome = match &state.shutdown { - Some(shutdown) => tokio::select! { - outcome = check => Some(outcome), - () = shutdown.cancelled() => None, - }, - None => Some(check.await), + if let Some(store) = runtime.configured_store() + && let Some(identity) = store.identity().filter(|_| store.is_healthy()) + { + let calls = StoreCalls { + store: &*store.store, + metrics: runtime.metrics().map(|m| &**m), }; - match outcome { - Some(outcome) => { - store.record_check(outcome); + report.deletions_finished = finish_deleting_rows(&**storage, store, &identity).await?; + (report.creating_resolved, report.creating_left_to_owner) = + resolve_dead_creating_rows(&**storage, calls, &identity, stale_job_after).await?; + if !report.maintenance_held { + let retention = runtime.job_settings().failed_row_retention; + report.failed_prefixes_deleted = + delete_stale_failed_prefixes(&**storage, calls, &identity, stale_job_after).await?; + report.deleted_prefixes_swept = + sweep_deleted_prefixes(&**storage, calls, &identity).await?; + report.tombstones_removed = + remove_tombstones(&**storage, calls, &identity, retention).await?; + } + } + + if !report.maintenance_held { + report.registrations_swept = storage.sweep_restore_registrations().await?; + (report.progress_rows_swept, report.spool_entries_swept) = + sweep_restore_leftovers(&**storage, runtime.configured_store()).await; + } + Ok(report) +} + +/// What the manifest arm did with one dead RUNNING record. +enum Resumed { + /// Taken over and the job spawned. + Spawned, + /// FAILED with the reason, the target left to the FAILED-target duty + /// (another store) or to the operator (`target_not_empty`). + Failed(String), + /// FAILED `OWNER_DEAD` and the target removed (or found moved, or the + /// removal failed), as a legacy record is, under the maintenance claim. + FailedAndRemoved(Removal), + /// No job permit was free; the next cycle retries. + NoPermit, + /// The configured store is unhealthy; the next cycle retries. + StoreUnhealthy, + /// The backend did not hand the record over: its owner is alive, or + /// another reconciler took it; or the server is shutting down and the + /// record is left for the next reconciler. + LeftToOwner, + /// A catalog or store call failed; logged, the next cycle retries. + Error, +} + +/// The manifest arm (module doc). +async fn resume_dead_manifest_restore( + storage: &Arc, + runtime: &BackupRuntime, + stale_job_after: Duration, + job: &RestoreJobRow, + maintenance_held: bool, +) -> Resumed +where + S: BackupCatalog + RestoreEngine + ?Sized + Send + Sync + 'static, +{ + let target = &job.target; + let what = format!( + "the restore of {} into {} (account {})", + job.source_backup_arn, target.table_name, target.account_id + ); + // Once the server's shutdown has begun the jobs' token is cancelled: a + // takeover now would spawn a resume that stops at its first check and + // leaves the record with this instance's fresh heartbeat, so the next + // instance waits a second stale window for it; a FAILED write on the + // way out is no better. The record is left as it is for the next + // reconciler to meet. + if runtime.shutdown().is_cancelled() { + tracing::info!( + "{what} has no live owner and the server is shutting down; it is left for the next \ + reconciler" + ); + return Resumed::LeftToOwner; + } + if maintenance_held { + tracing::info!( + "{what} has no live owner and a maintenance claim is held; it is failed rather than \ + resumed" + ); + // The `target_not_empty` guard first, as on the takeover arm: a + // record with no progress over a target that holds items is the + // operator's, and `OWNER_DEAD` with the target's removal would drop + // what the record does not account for. + match unaccounted_items(&**storage, target).await { + Ok(Some(count)) => { + return match storage.fail_restore_job(target, TARGET_NOT_EMPTY).await { + Ok(true) => { + tracing::error!( + "{what} recorded no progress and its target ({}) holds {count} \ + item(s); the record is FAILED {TARGET_NOT_EMPTY} and the target is \ + left for the operator: extenddb backup remove-target --account {} {}", + target.table_id, + target.account_id, + target.table_name + ); + Resumed::Failed(TARGET_NOT_EMPTY.to_owned()) + } + Ok(false) => Resumed::LeftToOwner, + Err(e) => { + tracing::error!( + "{what}: the FAILED write failed; the next cycle retries: {e}" + ); + Resumed::Error + } + }; } - None => { - tracing::info!( - "backup store ({}) check was stopped by shutdown; its health is left as the \ - last check left it", - store.kind + Ok(None) => {} + Err(e) => { + tracing::error!( + "{what}: its progress or its target could not be read; the next cycle \ + retries: {e}" ); - report.store_healthy = Some(store.is_healthy()); - return Ok(report); + return Resumed::Error; } } - report.store_healthy = Some(store.is_healthy()); - if let Some(identity) = store.identity().filter(|_| store.is_healthy()) { - let calls = StoreCalls { - store: &*store.store, - metrics: runtime.metrics().map(|m| &**m), - }; - report.deletions_finished = finish_deleting_rows(storage, store, &identity).await?; - (report.creating_resolved, report.creating_left_to_owner) = - resolve_dead_creating_rows(storage, calls, &identity, stale_job_after).await?; - if !report.maintenance_held { - let retention = runtime.job_settings().failed_row_retention; - report.failed_prefixes_deleted = - delete_stale_failed_prefixes(storage, calls, &identity, stale_job_after) - .await?; - report.deleted_prefixes_swept = - sweep_deleted_prefixes(storage, calls, &identity).await?; - report.tombstones_removed = - remove_tombstones(storage, calls, &identity, retention).await?; + return match remove_dead_legacy_restore(&**storage, job).await { + Some(removal) => Resumed::FailedAndRemoved(removal), + None => Resumed::LeftToOwner, + }; + } + let store = runtime.configured_store(); + let of_this_store = job + .store + .as_ref() + .is_some_and(|row_store| runtime.is_configured_store(row_store)); + if !of_this_store { + tracing::error!( + "{what} has no live owner and reads a store other than the configured one; the \ + record is FAILED and its target removed once the record is stale: {OTHER_STORE}" + ); + return match storage.fail_restore_job(target, OTHER_STORE).await { + Ok(true) => Resumed::Failed(OTHER_STORE.to_owned()), + Ok(false) => Resumed::LeftToOwner, + Err(e) => { + tracing::error!("{what} could not be marked FAILED; the next cycle retries: {e}"); + Resumed::Error + } + }; + } + let Some(store) = store.filter(|store| store.is_healthy()) else { + return Resumed::StoreUnhealthy; + }; + let env = match JobEnv::new(runtime, store) { + Ok(env) => env, + Err(_) => return Resumed::StoreUnhealthy, + }; + let Some(permit) = JobPermit::try_acquire(runtime.slots()) else { + tracing::info!("{what} has no live owner; no job permit is free, the next cycle retries"); + return Resumed::NoPermit; + }; + let spool_host = env.spool.as_ref().map(|s| s.spool_host.clone()); + let claim = match storage + .claim_restore_dead(target, stale_job_after, spool_host.as_deref()) + .await + { + Ok(Some(claim)) => claim, + Ok(None) => return Resumed::LeftToOwner, + Err(e) => { + tracing::error!( + "could not test whether the owner of {what} is dead; the next cycle retries: {e}" + ); + return Resumed::Error; + } + }; + // The resumed job's first confirmed beat (module doc): the takeover + // stamped `heartbeat_at` before it returned. + let claimed_at = tokio::time::Instant::now(); + // The progress under the claim: the record as it is now and, where the + // backend keeps them apart, its progress rows. + let (record, progress_rows) = match ( + storage.read_restore_job(target).await, + storage.read_restore_progress(target).await, + ) { + (Ok(Some(record)), Ok(rows)) => (record, rows), + (Ok(None), _) => { + tracing::info!("{what} was gone when its progress was read; left to whoever moved it"); + return Resumed::LeftToOwner; + } + (Err(e), _) | (_, Err(e)) => { + tracing::error!( + "{what} was taken over but its progress could not be read; the next cycle \ + retries once the record is stale again: {e}" + ); + return Resumed::Error; + } + }; + if has_no_progress(&record, &progress_rows) { + let count = match storage.count_restore_target(target).await { + Ok(count) => count, + Err(e) => { + tracing::error!( + "{what} was taken over but its target could not be counted; the next cycle \ + retries once the record is stale again: {e}" + ); + return Resumed::Error; } + }; + if count > 0 { + let fields = StatusFields { + failure_reason: Some(TARGET_NOT_EMPTY.to_owned()), + completed: true, + ..StatusFields::default() + }; + return match write_status(&*claim, JobStatus::Running, JobStatus::Failed, &fields).await + { + Written::Matched => { + tracing::error!( + "{what} recorded no progress and its target ({}) holds {count} item(s); \ + the record is FAILED {TARGET_NOT_EMPTY} and the target is left for the \ + operator: extenddb backup remove-target --account {} {}", + target.table_id, + target.account_id, + target.table_name + ); + Resumed::Failed(TARGET_NOT_EMPTY.to_owned()) + } + Written::NotMatched(why) => { + tracing::info!("{what} was no longer this cycle's ({why}); left to its owner"); + Resumed::LeftToOwner + } + Written::Error(e) => { + tracing::error!("{what}: the FAILED write failed; the next cycle retries: {e}"); + Resumed::Error + } + }; } } + // The record's progress as recorded; the job's own start line names the + // first file it opens. + tracing::info!( + "{what} has no live owner; taken over by this instance and resumed with {}", + match (record.file_index, record.items_applied) { + (Some(file), Some(items)) => + format!("{items} items of data file {file} recorded as applied"), + (Some(file), None) => format!("data file {file} recorded as applied"), + (None, _) if !progress_rows.is_empty() => + format!("{} data file(s) recorded as applied", progress_rows.len()), + (None, _) => "nothing recorded as applied".to_owned(), + } + ); + let limits = (&**runtime.limits()).into(); + runtime.spawn_job(restore::resume( + Arc::clone(storage), + env, + permit, + claim, + restore::DeadRecord { + row: record, + recorded_spool_host: job.spool_host.clone(), + progress_rows, + claimed_at, + }, + limits, + )); + Resumed::Spawned +} - if !report.maintenance_held { - report.registrations_swept = storage.sweep_restore_registrations().await?; +/// Whether a RUNNING record has no progress recorded at all: no +/// `file_index` on the record and no progress row beside it. +fn has_no_progress(record: &RestoreJobRow, progress_rows: &[(u32, u64)]) -> bool { + record.file_index.is_none() && progress_rows.is_empty() +} + +/// The `target_not_empty` test as the maintenance-held arm runs it, with +/// no claim on the record: the record and its progress rows read, and when +/// [`has_no_progress`] the target counted; `Some(count)` when the target +/// holds items no record accounts for, `None` when progress is recorded, +/// the target is empty, or the record is gone (the caller's write then +/// finds it so). +async fn unaccounted_items( + storage: &S, + target: &RestoreTargetKey, +) -> Result, StorageError> { + let Some(record) = storage.read_restore_job(target).await? else { + return Ok(None); + }; + let progress_rows = storage.read_restore_progress(target).await?; + if !has_no_progress(&record, &progress_rows) { + return Ok(None); } - Ok(report) + let count = storage.count_restore_target(target).await?; + Ok((count > 0).then_some(count)) +} + +/// The progress-row and spool sweeps (module doc). The spool directory is +/// listed first, then the RUNNING records, and only entries of that listing +/// that no record of this host names are removed, so a restore whose +/// registration commits and whose entry opens between the two reads is +/// never a candidate (its entry is not in the listing, or its record is in +/// the records). A listing the backend cannot make is logged and skips +/// both. +async fn sweep_restore_leftovers( + storage: &S, + store: Option<&ConfiguredStore>, +) -> (u64, Vec) { + let spool = store.and_then(|store| store.spool.as_ref()); + let entries = match spool.map(|spool| restore::spool::list_entries(&spool.path)) { + None => Vec::new(), + Some(Ok(entries)) => entries, + Some(Err(e)) => { + tracing::error!( + "the restore spool {} could not be listed; the spool sweep is skipped this \ + cycle: {e}", + spool + .map(|s| s.path.display().to_string()) + .unwrap_or_default() + ); + Vec::new() + } + }; + let running = match storage.list_restore_jobs(Some(JobStatus::Running)).await { + Ok(rows) => rows, + Err(e) => { + tracing::warn!( + "the restore job records could not be listed; the progress-row and spool sweeps \ + are skipped this cycle: {e}" + ); + return (0, Vec::new()); + } + }; + let running_ids: Vec = running.iter().map(|r| r.target.table_id.clone()).collect(); + let progress_rows = match storage.sweep_restore_progress(&running_ids).await { + Ok(n) => { + if n > 0 { + tracing::info!( + "removed {n} restore progress row(s) of tables that are not RUNNING restore \ + targets" + ); + } + n + } + Err(e) => { + tracing::error!( + "the restore progress rows could not be swept; the next cycle retries: {e}" + ); + 0 + } + }; + let Some(spool) = spool else { + return (progress_rows, Vec::new()); + }; + let this_host: HashSet = running + .iter() + .filter(|r| r.spool_host.as_deref() == Some(spool.spool_host.as_str())) + .map(|r| r.target.table_id.clone()) + .collect(); + let swept = restore::spool::sweep_entries(entries, &this_host); + (progress_rows, swept) } /// The duty for CREATING rows of the configured store whose owner is dead @@ -529,7 +992,7 @@ async fn resolve_dead_creating_rows( continue; } }; - match write_status(&*claim, to, &fields).await { + match write_status(&*claim, JobStatus::Creating, to, &fields).await { Written::Matched => { tracing::info!( "backup {} (account {}): its owner was dead; the row is now {to}", @@ -579,21 +1042,11 @@ impl StoreCalls<'_> { e } - /// One object's bytes, `None` when absent. - async fn read_object(self, key: &str) -> Result>, StoreError> { - let body = match self.store.get(key).await { - Ok(body) => body, - Err(StoreError::NotFound) => return Ok(None), - Err(e) => return Err(self.failed("get", e)), - }; - let bytes = body - .try_fold(Vec::new(), |mut acc, chunk| async move { - acc.extend_from_slice(&chunk); - Ok(acc) - }) + /// One object's bytes within `bound` ([`read_object`]). + async fn read_object(self, key: &str, bound: u64) -> Result { + read_object(self.store, key, bound) .await - .map_err(|e| self.failed("get", e))?; - Ok(Some(bytes)) + .map_err(|e| self.failed("get", e)) } /// The prefix's objects, key to size. @@ -639,22 +1092,38 @@ enum Probe { async fn probe_manifest(store: StoreCalls<'_>, arn: &BackupArn) -> Result { let prefix = arn.store_prefix(); let manifest = match store - .read_object(&format!("{prefix}{EXTENDDB_MANIFEST_FILE}")) + .read_object( + &format!("{prefix}{EXTENDDB_MANIFEST_FILE}"), + MANIFEST_READ_BOUND, + ) .await? { - Some(bytes) => bytes, - None => return Ok(Probe::Incomplete("no manifest".to_owned())), + ReadObject::Bytes(bytes) => bytes, + ReadObject::Absent => return Ok(Probe::Incomplete("no manifest".to_owned())), + ReadObject::Oversize => { + return Ok(Probe::Incomplete(format!( + "a manifest over {MANIFEST_READ_BOUND} bytes" + ))); + } }; let sidecar = match store - .read_object(&format!("{prefix}{MANIFEST_CHECKSUM_FILE}")) + .read_object( + &format!("{prefix}{MANIFEST_CHECKSUM_FILE}"), + SIDECAR_READ_BOUND, + ) .await? { - Some(bytes) => bytes, - None => { + ReadObject::Bytes(bytes) => bytes, + ReadObject::Absent => { return Ok(Probe::Incomplete( "a manifest without its sidecar".to_owned(), )); } + ReadObject::Oversize => { + return Ok(Probe::Incomplete(format!( + "a sidecar over {SIDECAR_READ_BOUND} bytes" + ))); + } }; if let Err(e) = verify_manifest_checksum(&manifest, &sidecar) { return Ok(Probe::Incomplete(format!( @@ -711,6 +1180,51 @@ async fn probe_manifest(store: StoreCalls<'_>, arn: &BackupArn) -> Result), + /// No object under the key. + Absent, + /// The object has more bytes than the bound; the read stopped there and + /// collected nothing. + Oversize, +} + +/// One object's bytes, collected only while they stay within `bound` +/// (the body is read in chunks and dropped once the count passes it). The +/// restore's request and resume read the manifest and sidecar through it +/// too, each with its own bound, so a prefix an attacker rewrote cannot make +/// a request hold an object of any size in memory. +pub(super) async fn read_object( + store: &dyn BackupStore, + key: &str, + bound: u64, +) -> Result { + let mut body = match store.get(key).await { + Ok(body) => body, + Err(StoreError::NotFound) => return Ok(ReadObject::Absent), + Err(e) => return Err(e), + }; + let mut bytes = Vec::new(); + while let Some(chunk) = body.try_next().await? { + if (bytes.len() as u64).saturating_add(chunk.len() as u64) > bound { + return Ok(ReadObject::Oversize); + } + bytes.extend_from_slice(&chunk); + } + Ok(ReadObject::Bytes(bytes)) +} + /// The FAILED-prefix duty (module doc): FAILED rows whose heartbeat is older /// than `stale_job_after`, their prefix deleted while it lists objects. async fn delete_stale_failed_prefixes( diff --git a/crates/engine/src/backup/restore.rs b/crates/engine/src/backup/restore.rs new file mode 100644 index 000000000..0a6e4d62c --- /dev/null +++ b/crates/engine/src/backup/restore.rs @@ -0,0 +1,1303 @@ +// Copyright 2026 ExtendDB contributors +// SPDX-License-Identifier: Apache-2.0 + +//! The store-backed RestoreTableFromBackup: the request's manifest branch +//! and the restore job (the design's 5.3). +//! +//! The request ([`request`]) runs once `check_catalog_row_for_restore` has +//! accepted the backup's row: it takes the job permit, reads the sidecar and +//! the manifest from the store (a missing one is `BackupNotFoundException` +//! and an error line naming `backup sync --prune-missing`; each read +//! collects at most its bound, [`super::reconciler::SIDECAR_READ_BOUND`] and +//! [`super::reconciler::MANIFEST_READ_BOUND`], and an object over it is a +//! `ValidationException` naming a corrupt prefix, as a bad sidecar is), +//! verifies the +//! sidecar, parses the manifest (`ExtenddbManifest::from_json_bytes`, whose +//! own gates are the `format_version` and the identity rule that the ARN, +//! `source_table.table_name` and `source_table.table_arn` name one table), +//! checks that it names the row's backup, refuses a non-empty +//! `vector_indexes` into a backend without vector support with the text the +//! legacy reader uses, builds the target's `CreateTableInput` from the +//! manifest with the overrides applied, validates it as CreateTable does, +//! resolves the key schema the validator holds every item to, and spawns +//! the detached task ([`start`]) whose answer it awaits for at most +//! [`RESTORE_REGISTRATION_TIMEOUT`]; past that it answers +//! `InternalServerError` while the task goes on (the target then appears +//! through DescribeTable). The task registers the target and its RUNNING +//! record in one write (`claim_restore`, under the backup row's lock), sends +//! the job's first beat through the claim (the registration's own +//! `heartbeat_at` stamp can be old: the design's 4(e) states each backend's +//! clock, PostgreSQL's `now()` being the transaction's start, so a +//! registration stalled on the `backup_index` lock or the DDL commits a +//! stale beat that the reconciler under a maintenance claim and +//! `remove-target`, which judge the owner by the heartbeat alone, would +//! read as dead; the beat refreshes it at the registration's end on every +//! backend, and a beat that finds the claim lost or fails ends the task as +//! a lost claim before the reply, the record failed as the maintenance +//! refusal fails it and the request answered `InternalServerError`), then +//! checks the maintenance claim: held, the record is FAILED `maintenance`, +//! the target removed, and the request refused with +//! `LimitExceededException` (a job end: `RestoreJobs{failed}` and the worker +//! error are recorded); otherwise the request is answered CREATING and the +//! job runs ([`run`]). The heartbeat's first confirmed beat is that first +//! beat, and the instant it returned is what the pre-write check +//! (`heartbeat.rs`) counts the two periods from, so the registration's own +//! duration, however long, does not end the job at its first check. A +//! resumed job's instant is the takeover's return, taken by the reconciler +//! beside `claim_restore_dead` and passed in (the takeover stamped +//! `heartbeat_at` inside its write), so the reads between the takeover and +//! the job's start count against the window rather than widen it. +//! +//! The job: step 3, every data file downloaded and verified +//! ([`download`]: into the spool entry on an S3 store, hashed in place on a +//! filesystem store, the manifest and sidecar spooled first for a resume, +//! each as a data file is: `.partial`, synced, renamed; a resume that read +//! them from the spool does not write them again); +//! step 5, each file decoded, validated and written through one +//! `bulk_load` ([`load`]), skipping the files and items a resume has +//! recorded; the count check (`count_restore_target` against the manifest's +//! total, a shortfall being a key the backup holds twice); step 6, +//! `finish_restore` (DONE and ACTIVE, or DONE with the index backfill +//! pending on MongoDB). A `finish_restore` that returns an error is a +//! conditional write whose outcome is unknown, handled as `write_status` +//! handles one: the record is re-read through the claim; DONE under this +//! generation means the write landed and the restore is reported complete +//! with a WARN naming the error; RUNNING under this generation means it did +//! not, and the write is sent once more, at most, its own error re-read the +//! same way; anything else is the DONE write that matched no record. The +//! heartbeat runs on a timer for the job's life +//! and is checked before every download attempt, every `bulk_load`, the +//! count and the flip; a lost claim ends the job with no catalog or table +//! access. Step 7 ([`fail_restore`]): FAILED with the reason, then +//! `remove_restore_target` only when that write matched (the claim holder +//! owns the table); the spool entry is removed on every end but the +//! shutdown's and the ends at which the job learnt the record is another's +//! (a lost claim, a DONE or FAILED write that matched no record): a +//! same-host successor may be reading that entry, so it is left to the +//! reconciler's sweep, which lists the spool before it reads the records and +//! removes the entries it listed that no RUNNING record of this host names. +//! The shutdown token stops the job between download attempts, +//! files and chunks with no status write: the record stays RUNNING with its +//! spool and progress for a resume ([`resume`], the reconciler's entry). +//! +//! Each job records `RestoreJobs` and `RestoreJobDuration` under its +//! outcome, `RestoreBytesDownloaded` as the files arrive, +//! `BackupStoreErrors{get}` on every failed `get` of a restore job (the +//! request's and a resume's two document reads and each download attempt +//! that the store failed, whatever the store's error), and the worker +//! metrics under `QuerySource::RestoreJob` (`WorkerLastSuccess` on DONE, +//! `WorkerErrorCount` on FAILED or a job that ended without its write; a +//! stopped job records neither); one info line at start and end, one error +//! line per failure. + +pub(crate) mod download; +pub(crate) mod load; +pub(crate) mod spool; + +use std::sync::Arc; +use std::time::Duration; + +use extenddb_core::backup_format::{ + BackupArn, EXTENDDB_MANIFEST_FILE, ExtenddbManifest, ItemOrder, MANIFEST_CHECKSUM_FILE, + verify_manifest_checksum, +}; +use extenddb_core::error::DynamoDbError; +use extenddb_core::limits::LimitsConfig; +use extenddb_core::metrics::{BackupJobOutcome, QuerySource}; +use extenddb_core::types::{BillingMode, CreateTableInput, TableDescription}; +use extenddb_core::validation::{RestorableKeySchema, RestorableLimits}; +use extenddb_storage::backup_catalog::{ + BackupKey, Heartbeat, JobStatus, RestoreJobRow, RestoreRegistrationRow, RestoreTargetKey, + StatusFields, UNOWNED_GENERATION, billing_mode_name, +}; +use extenddb_storage::backup_definition::{restore_input_from_manifest, vector_index_refusal}; +use extenddb_storage::{ + BackupCatalog, CancellationToken, DataEngine, Finished, JobClaim, JobPermit, RestoreEngine, + RestoreSource, RestoreTableOverrides, +}; +use serde_json::Value; +use tokio::sync::oneshot; +use tokio::time::Instant; + +use super::job::failure::{Written, write_status}; +use super::job::heartbeat::{HeartbeatHandle, HeartbeatMetric}; +use super::job::{JobEnv, KillPoint, MAINTENANCE, kill_point, kill_point_named}; +use super::overrides::IndexOverrides; +use super::reconciler::{MANIFEST_READ_BOUND, ReadObject, SIDECAR_READ_BOUND}; +use super::runtime::{BackupRuntime, ConfiguredStore}; +use super::{ + backup_not_found, maintenance_held, restore_response, restore_storage_err_to_dynamo, + storage_err_to_dynamo, validate_override_for_source, +}; +use download::{Dest, Downloader, Stop}; +use load::{FileLoad, Source}; + +/// How long the request waits for the detached task to register the target +/// before it answers `InternalServerError` and leaves the task to finish +/// (against a catalog that has stopped handing out connections). One value +/// on every backend. +pub const RESTORE_REGISTRATION_TIMEOUT: Duration = Duration::from_secs(60); + +/// The reason a resumed restore's record carries when no progress was +/// recorded and the target already holds items (a catalog restored from a +/// dump can name a restore that finished and was written to since): the +/// target is left for the operator (`extenddb backup remove-target`). +pub const TARGET_NOT_EMPTY: &str = "target_not_empty"; + +/// The reason the reconciler writes on a RUNNING record whose store is not +/// the configured one. +pub const OTHER_STORE: &str = "the restore reads a store other than the configured one; backup sync --move-store re-points it"; + +/// What the request resolved before it spawned the task. +pub(crate) struct RestoreRequestParts { + pub account_id: String, + pub arn: BackupArn, + pub backup: BackupKey, + pub manifest: Arc, + pub manifest_bytes: Vec, + pub sidecar_bytes: Vec, + pub input: CreateTableInput, + /// The key schema the validator holds every item to. + pub key_schema: RestorableKeySchema, + pub limits: RestorableLimits, +} + +/// The one-shot the task answers the request through. +pub(crate) type Reply = oneshot::Sender>; + +/// The request's manifest branch (module doc), from the permit to the +/// response body, for a row `check_catalog_row_for_restore` accepted whose +/// store is the healthy configured one. +#[allow(clippy::too_many_arguments)] +pub(super) async fn request( + storage: &Arc, + runtime: &BackupRuntime, + store: &ConfiguredStore, + account_id: &str, + limits: &LimitsConfig, + target_table_name: &str, + arn: &BackupArn, + overrides: &RestoreTableOverrides, + index_overrides: &IndexOverrides, +) -> Result<(String, Value), DynamoDbError> +where + S: BackupCatalog + DataEngine + RestoreEngine + ?Sized + Send + Sync + 'static, +{ + let arn_text = arn.to_arn_string(); + let env = JobEnv::new(runtime, store)?; + // The permit, last among the checks that need no store access. + let permit = runtime.take_permit()?; + + // The sidecar and the manifest; a missing one behind an AVAILABLE row is + // BackupNotFound and the operator's line; one over its bound is a + // prefix nothing this server wrote produces. + let prefix = arn.store_prefix(); + let read = async |name: &str, bound: u64| -> Result, DynamoDbError> { + match super::reconciler::read_object(&*env.store, &format!("{prefix}{name}"), bound).await { + Ok(ReadObject::Bytes(bytes)) => Ok(bytes), + Ok(ReadObject::Absent) => { + tracing::error!( + "backup {arn_text} (account {account_id}): its row is AVAILABLE but the store \ + holds no {name} under its prefix; backup sync --prune-missing marks the row" + ); + Err(backup_not_found(&arn_text)) + } + Ok(ReadObject::Oversize) => Err(DynamoDbError::ValidationException(format!( + "Backup {arn_text} has a {name} over {bound} bytes, which no backup this server \ + wrote produces; the backup's prefix is corrupt" + ))), + Err(e) => { + if let Some(metrics) = runtime.metrics() { + metrics.record_backup_store_error("get"); + } + tracing::error!("backup {arn_text}: reading {name} from the store failed: {e}"); + Err(DynamoDbError::InternalServerError( + "Internal server error".to_owned(), + )) + } + } + }; + let sidecar_bytes = read(MANIFEST_CHECKSUM_FILE, SIDECAR_READ_BOUND).await?; + let manifest_bytes = read(EXTENDDB_MANIFEST_FILE, MANIFEST_READ_BOUND).await?; + verify_manifest_checksum(&manifest_bytes, &sidecar_bytes).map_err(|e| { + DynamoDbError::ValidationException(format!( + "Backup {arn_text} has a manifest whose checksum sidecar does not match: {e}" + )) + })?; + // The parser ran the version gate and the identity rule (the ARN's + // table is `source_table.table_name`); what is left is that the + // manifest's backup is the row's. + let manifest = ExtenddbManifest::from_json_bytes(&manifest_bytes).map_err(|e| { + DynamoDbError::ValidationException(format!( + "Backup {arn_text} has a manifest this server cannot restore from: {e}" + )) + })?; + match manifest.backup_arn() { + Ok(named) if named == *arn => {} + Ok(named) => { + return Err(DynamoDbError::ValidationException(format!( + "Backup {arn_text} has a manifest that names another backup, {}", + named.to_arn_string() + ))); + } + Err(e) => { + return Err(DynamoDbError::ValidationException(format!( + "Backup {arn_text} has a manifest whose backup ARN is refused: {e}" + ))); + } + } + + // Step 2: the one feature a target can lack. + let source_table = &manifest.source_table; + if crate::vector_gate::ensure_create_table_supported( + source_table.vector_indexes.as_ref(), + storage.as_vector_search(), + ) + .is_err() + { + let count = source_table.vector_indexes.as_ref().map_or(0, Vec::len); + return Err(storage_err_to_dynamo(vector_index_refusal( + &format!("backup {arn_text}"), + count, + ))); + } + + // The target's input, with the overrides, validated as CreateTable does. + validate_override_for_source(overrides, billing_mode_name(source_table.billing_mode))?; + let mut input = restore_input_from_manifest(source_table, target_table_name, overrides); + index_overrides.apply(&mut input, overrides.billing_mode)?; + extenddb_core::validation::validate_create_table(&input, limits)?; + // The key schema the validator holds every item to: a pure function of + // the manifest, resolved here with step 2's refusals rather than after + // the client was told the target is CREATING. + let key_schema = RestorableKeySchema::resolve( + &source_table.key_schema, + &source_table.attribute_definitions, + ) + .map_err(|e| { + DynamoDbError::ValidationException(format!( + "Backup {arn_text} has a manifest whose key schema cannot be checked: {e}" + )) + })?; + let on_demand = input.billing_mode == Some(BillingMode::PayPerRequest); + let source = RestoreSource { + backup_arn: arn_text.clone(), + source_table_arn: source_table.table_arn.clone(), + backup_creation_date_time: manifest.backup_creation_date_time, + }; + + // Step 4 in the detached task; the request waits for its answer. + let parts = RestoreRequestParts { + account_id: account_id.to_owned(), + arn: arn.clone(), + backup: BackupKey::from_arn(arn), + manifest: Arc::new(manifest), + manifest_bytes, + sidecar_bytes, + input, + key_schema, + limits: limits.into(), + }; + let (reply, answer) = oneshot::channel(); + runtime.spawn_job(start(Arc::clone(storage), env, permit, parts, reply)); + let desc = match tokio::time::timeout(RESTORE_REGISTRATION_TIMEOUT, answer).await { + Ok(Ok(answer)) => answer?, + Ok(Err(_)) => { + tracing::error!( + "RestoreTableFromBackup of {arn_text} into {target_table_name} (account \ + {account_id}): the job task ended without answering" + ); + return Err(DynamoDbError::InternalServerError( + "Internal server error".to_owned(), + )); + } + Err(_) => { + tracing::error!( + "RestoreTableFromBackup of {arn_text} into {target_table_name} (account \ + {account_id}): the registration did not answer within {} s; the task goes on \ + and the target, if registered, appears through DescribeTable", + RESTORE_REGISTRATION_TIMEOUT.as_secs() + ); + return Err(DynamoDbError::InternalServerError( + "Internal server error".to_owned(), + )); + } + }; + let response = restore_response(desc, &source, on_demand)?; + Ok((target_table_name.to_owned(), response)) +} + +/// The detached task: the registration, the maintenance check, the reply, +/// then the job. +pub(crate) async fn start( + storage: Arc, + env: JobEnv, + permit: JobPermit, + parts: RestoreRequestParts, + reply: Reply, +) where + S: BackupCatalog + RestoreEngine + ?Sized + Send + Sync + 'static, +{ + let started = Instant::now(); + let RestoreRequestParts { + account_id, + arn, + backup, + manifest, + manifest_bytes, + sidecar_bytes, + input, + key_schema, + limits, + } = parts; + let arn_text = arn.to_arn_string(); + let target_table_name = input.table_name.clone(); + let spool_host = env.spool.as_ref().map(|s| s.spool_host.clone()); + let registration = RestoreRegistrationRow { + backup, + source_table_arn: manifest.source_table.table_arn.clone(), + restore_date_time: manifest.backup_creation_date_time, + store: env.identity.clone(), + spool_host: spool_host.clone(), + owner_generation: UNOWNED_GENERATION, + }; + let (desc, claim) = match storage + .claim_restore(&account_id, input, ®istration) + .await + { + Ok(registered) => registered, + Err(e) => { + drop(permit); + if let Err(Err(error)) = reply.send(Err(restore_storage_err_to_dynamo(e))) { + // The request has answered `InternalServerError` at its bound + // and told the operator the target appears if registered; + // this line is the one trace of why it will not. + tracing::error!( + "restore of {arn_text} into {target_table_name} (account {account_id}): the \ + registration failed after the request had stopped waiting for it; no target \ + was created: {error}" + ); + } + return; + } + }; + // The job's first beat, before the reply: the record's `heartbeat_at` as + // the registration stamped it is the catalog's clock at some point of + // the registration (on PostgreSQL `now()` is the transaction's start, so + // a registration stalled on the `backup_index` lock or the DDL commits a + // stale beat), and the two readers that judge the owner by the heartbeat + // alone (the reconciler under a maintenance claim, `remove-target`) + // would find a live job dead until its timer's first beat. A confirmed + // beat refreshes it at the registration's end on every backend, and its + // return is the instant the heartbeat's two-period rule counts from. A + // beat that finds the claim lost, or fails, ends the task as a lost + // claim before the reply: the record is failed as the maintenance + // refusal fails it (the write matches nothing when the claim is + // another's) and the request is answered `InternalServerError`. + let first_beat = match claim.heartbeat().await { + Ok(Heartbeat::Confirmed) => Ok(Instant::now()), + Ok(Heartbeat::Lost) => Err("the claim was lost before the job started".to_owned()), + Err(e) => Err(format!( + "the claim was lost before the job started: its first heartbeat failed: {e}" + )), + }; + let claim: Arc = Arc::from(claim); + let target = RestoreTargetKey { + account_id: account_id.clone(), + table_name: desc.table_name.clone(), + table_id: desc.table_id.clone(), + }; + // The maintenance claim, once the record exists so the command's wait + // sees it: held, the record is FAILED `maintenance`, the target removed + // and the request refused. + let (claimed_at, refused) = match first_beat { + Err(reason) => ( + started, + Some(( + reason, + DynamoDbError::InternalServerError("Internal server error".to_owned()), + )), + ), + Ok(beat) => ( + beat, + match storage + .check_maintenance_claim(env.settings.stale_job_after) + .await + { + Ok(false) => None, + Ok(true) => Some((MAINTENANCE.to_owned(), maintenance_held())), + Err(e) => Some(( + format!("the maintenance claim could not be read: {e}"), + storage_err_to_dynamo(e), + )), + }, + ), + }; + if let Some((reason, error)) = refused { + let path = fail_restore(&*storage, &*claim, &target, &reason, None, &env.shutdown).await; + tracing::error!( + "restore of {arn_text} into {target_table_name} (account {account_id}): FAILED \ + ({reason}) before it started; {path}" + ); + // A FAILED record is a job end: the outcome and the worker error. + if let Some(metrics) = env.metrics.as_deref() { + #[allow(clippy::cast_precision_loss)] + let duration_us = started.elapsed().as_micros() as f64; + metrics.record_restore_job(BackupJobOutcome::Failed, duration_us); + metrics.record_worker_error(QuerySource::RestoreJob); + } + drop(permit); + let _ = reply.send(Err(error)); + return; + } + // A request whose client went away still has its job. + let _ = reply.send(Ok(desc)); + let job = RestoreJob { + storage, + claim, + target, + arn, + manifest, + manifest_bytes, + sidecar_bytes, + key_schema, + limits, + env, + started, + claimed_at, + progress: None, + recorded_spool_host: spool_host, + documents_from_spool: false, + }; + run(job).await; + drop(permit); +} + +/// Per file, what an earlier run recorded as applied; `None` for a file +/// with no record. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) struct Progress { + recorded: Vec>, +} + +impl Progress { + /// From a dead owner's record and, on PostgreSQL, its progress rows: + /// every file before the record's `file_index` is complete, `file_index` + /// itself has `items_applied` of its items (all of them when the backend + /// records per file), and each progress row names its file's count. + pub(crate) fn from_records( + row: &RestoreJobRow, + progress_rows: &[(u32, u64)], + manifest: &ExtenddbManifest, + ) -> Self { + let counts: Vec = manifest.data_files.iter().map(|f| f.item_count).collect(); + let mut recorded = vec![None; counts.len()]; + if let Some(last) = row.file_index { + let last = last as usize; + for (i, slot) in recorded.iter_mut().enumerate().take(last.min(counts.len())) { + *slot = Some(counts[i]); + } + if let Some(slot) = recorded.get_mut(last) { + *slot = Some(row.items_applied.unwrap_or(counts[last])); + } + } + for (index, applied) in progress_rows { + if let Some(slot) = recorded.get_mut(*index as usize) { + *slot = Some((*applied).max(slot.unwrap_or(0))); + } + } + Self { recorded } + } + + /// The first file not complete, for the start line. + fn first_open(&self, counts: &[u64]) -> Option<(usize, u64)> { + self.recorded + .iter() + .enumerate() + .find_map(|(i, recorded)| match recorded { + Some(n) if *n >= counts[i] => None, + Some(n) => Some((i, *n)), + None => Some((i, 0)), + }) + } + + /// Per file, the items recorded as applied; `None` for a file with no + /// record. + #[cfg(test)] + pub(crate) fn recorded(&self) -> &[Option] { + &self.recorded + } + + /// [`Self::first_open`] against `manifest`'s counts. + #[cfg(test)] + pub(crate) fn first_open_in(&self, manifest: &ExtenddbManifest) -> Option<(usize, u64)> { + let counts: Vec = manifest.data_files.iter().map(|f| f.item_count).collect(); + self.first_open(&counts) + } +} + +/// The job once its record exists. +pub(crate) struct RestoreJob { + storage: Arc, + claim: Arc, + target: RestoreTargetKey, + arn: BackupArn, + manifest: Arc, + manifest_bytes: Vec, + sidecar_bytes: Vec, + key_schema: RestorableKeySchema, + limits: RestorableLimits, + env: JobEnv, + started: Instant, + /// The instant the heartbeat's two-period rule counts from: the return + /// of the job's first beat, sent right after the registration, for a + /// fresh job; the takeover's return for a resume (the takeover stamped + /// the record's `heartbeat_at` inside its write). + claimed_at: Instant, + /// A resumed job's recorded progress; `None` for a fresh one. + progress: Option, + /// The `spool_host` the record carried before this job's claim: the + /// registration's own for a fresh job, the dead owner's for a resume. + /// A spool entry of this table id is this job's to reuse only when it + /// is this host's. + recorded_spool_host: Option, + /// Whether the manifest and sidecar in hand were read from this host's + /// spool entry (a resume), in which case the job does not write them + /// there again. + documents_from_spool: bool, +} + +/// The job (module doc), from the download to the final write. +pub(crate) async fn run(job: RestoreJob) -> BackupJobOutcome +where + S: BackupCatalog + RestoreEngine + ?Sized + Send + Sync + 'static, +{ + let RestoreJob { + storage, + claim, + target, + arn, + manifest, + manifest_bytes, + sidecar_bytes, + key_schema, + limits, + env, + started, + claimed_at, + progress, + recorded_spool_host, + documents_from_spool, + } = job; + let arn_text = arn.to_arn_string(); + let name = format!( + "restore of {arn_text} into {} (account {})", + target.table_name, target.account_id + ); + let metrics = env.metrics.as_deref(); + let heartbeat = HeartbeatHandle::start( + Arc::clone(&claim), + env.settings.heartbeat_interval, + HeartbeatMetric::None, + claimed_at, + ); + #[cfg(test)] + test_support::register_timer(&target.table_id, &heartbeat); + let counts: Vec = manifest.data_files.iter().map(|f| f.item_count).collect(); + let total_items: u64 = counts.iter().sum(); + let resumed_from = progress.as_ref().and_then(|p| p.first_open(&counts)); + tracing::info!( + "{name}: started, {} data file(s), {total_items} items, {}{}", + manifest.data_files.len(), + match manifest.item_order { + ItemOrder::KeyOrdered => "key-ordered", + ItemOrder::Unordered => "unordered", + }, + match (&progress, resumed_from) { + (None, _) => String::new(), + (Some(_), Some((file, items))) => + format!("; resumed from data file {file} after {items} of its items"), + (Some(_), None) => "; resumed with every file recorded".to_owned(), + } + ); + let prefix = arn.store_prefix(); + let mut entry = None; + let mut bytes_downloaded = 0; + let mut applied_total = 0; + let end: Result = async { + // Step 3. The entry is the job's from the open, so a document write + // that fails (the disk full, a name taken by a directory) reaches + // the failure path with the entry in hand and it is removed as any + // other failed end's is. + if let Some(spool) = &env.spool { + let opened = spool::Entry::open( + &spool.path, + &target.table_id, + recorded_spool_host.as_deref(), + &spool.spool_host, + ) + .map_err(|e| Stop::Fail(format!("opening the spool entry: {e}")))?; + let opened = entry.insert(opened); + if !documents_from_spool { + for (doc, bytes) in [ + (EXTENDDB_MANIFEST_FILE, &manifest_bytes), + (MANIFEST_CHECKSUM_FILE, &sidecar_bytes), + ] { + opened + .write_document(doc, bytes) + .await + .map_err(|e| Stop::Fail(format!("writing {doc} to the spool: {e}")))?; + } + } + } + let dest = entry.as_ref().map_or(Dest::InPlace, Dest::Spool); + let downloader = Downloader { + store: &*env.store, + prefix: &prefix, + heartbeat: &heartbeat, + shutdown: &env.shutdown, + metrics, + }; + bytes_downloaded = download::download_all(&downloader, &manifest, &dest).await?; + kill_point(KillPoint::RestoreAfterDownload, &env.shutdown).await; + + // Step 5. + #[cfg(not(test))] + let chunk_bytes = load::LOAD_CHUNK_BYTES; + #[cfg(test)] + let chunk_bytes = test_support::chunk_bytes_for(&target.table_id); + let mut first_loaded = true; + let mut reapplying = progress.is_some(); + for (i, file) in manifest.data_files.iter().enumerate() { + let recorded = progress.as_ref().and_then(|p| p.recorded[i]); + let start_at = match recorded { + Some(n) if n >= file.item_count => continue, + Some(n) => n, + None => 0, + }; + if env.shutdown.is_cancelled() { + return Err(Stop::Cancelled); + } + if !heartbeat.confirms_the_claim() { + return Err(Stop::ClaimLost( + heartbeat.why_not_confirmed(&format!("to load {}", file.key)), + )); + } + let file_index = + u32::try_from(i).map_err(|_| Stop::Fail("too many data files".to_owned()))?; + let source = match &entry { + Some(entry) => Source::Spool(entry.file_path(file_index)), + None => Source::InPlace { + store: Arc::clone(&env.store), + key: format!("{prefix}{}", file.key), + }, + }; + let applied = load::load_file( + &*storage, + &target, + &*claim, + FileLoad { + manifest: &manifest, + entry: file, + file_index, + start_at, + reapplying, + key_schema: &key_schema, + limits, + shutdown: &env.shutdown, + heartbeat: &heartbeat, + chunk_bytes, + }, + source, + ) + .await?; + applied_total += applied; + // Only the first file a resume meets may have been applied in + // part without a record. + reapplying = false; + if first_loaded { + first_loaded = false; + kill_point(KillPoint::RestoreAfterFileCommit, &env.shutdown).await; + } + } + + // The count: every restore ends with it, so a backend that kept one + // of two rows with one key, and a key-range check that missed, are + // both caught. + if env.shutdown.is_cancelled() { + return Err(Stop::Cancelled); + } + if !heartbeat.confirms_the_claim() { + return Err(Stop::ClaimLost( + heartbeat.why_not_confirmed("to count the target"), + )); + } + let count = storage + .count_restore_target(&target) + .await + .map_err(|e| Stop::Fail(format!("counting the target's items: {e}")))?; + if count != total_items { + return Err(Stop::Fail(format!( + "restored {count} items, the manifest lists {total_items}: the backup holds a \ + key twice" + ))); + } + + // Step 6. The fenced-out kill point stops the heartbeat first, so a + // reconciler can take the record over while the job pauses, and the + // pre-write check is skipped so the conditional write itself is what + // the test observes. + let fenced = kill_point_named(KillPoint::RestoreFencedOut); + if fenced { + heartbeat.stop_timer(); + } + kill_point(KillPoint::RestoreFencedOut, &env.shutdown).await; + if env.shutdown.is_cancelled() { + return Err(Stop::Cancelled); + } + if !fenced && !heartbeat.confirms_the_claim() { + return Err(Stop::ClaimLost( + heartbeat.why_not_confirmed("for the DONE write"), + )); + } + // The restored table's size is the source table's as the manifest + // records it (the statistics pass rewrites it once ACTIVE), not the + // compressed size of the backup's files. + let table_size_bytes = u64::try_from(manifest.source_table.table_size_bytes).unwrap_or(0); + finish(&*storage, &*claim, &target, total_items, table_size_bytes).await + } + .await; + let elapsed = started.elapsed().as_secs_f64(); + let (outcome, remove_entry) = match end { + Ok(Ended::Written(Finished::Active)) => { + tracing::info!( + "{name}: ACTIVE, {total_items} items ({applied_total} applied by this run), \ + {bytes_downloaded} bytes downloaded, in {elapsed:.1} s" + ); + (BackupJobOutcome::Succeeded, true) + } + Ok(Ended::Written(Finished::Pending)) => { + tracing::info!( + "{name}: DONE, {total_items} items ({applied_total} applied by this run), \ + {bytes_downloaded} bytes downloaded, in {elapsed:.1} s; index backfill pending, \ + the backend's worker flips the table ACTIVE" + ); + kill_point(KillPoint::RestoreMongodbBeforeWorkerFlip, &env.shutdown).await; + (BackupJobOutcome::Succeeded, true) + } + // The write landed and its answer did not come back: the table is + // ACTIVE, or awaits the backend's worker, which this job did not see. + Ok(Ended::DoneOnReRead) => { + tracing::info!( + "{name}: complete (DONE), {total_items} items ({applied_total} applied by this \ + run), {bytes_downloaded} bytes downloaded, in {elapsed:.1} s; the DONE write's \ + answer was lost and the record was re-read DONE under this job's generation (the \ + table is ACTIVE, or the backend's worker flips it)" + ); + (BackupJobOutcome::Succeeded, true) + } + // The record is another's: its owner may be reading this host's + // entry, so the entry is left to the sweep. + Ok(Ended::Written(Finished::NotMatched(why))) => { + tracing::error!( + "{name}: the DONE write matched no record ({why}); the record is not this job's \ + to write and no table access was made" + ); + (BackupJobOutcome::Failed, false) + } + Err(Stop::ClaimLost(why)) => { + tracing::error!( + "{name}: stopped with no further catalog or table access: {why}; the record is \ + left to its owner or to the reconciler" + ); + (BackupJobOutcome::Failed, false) + } + Err(Stop::Cancelled) => { + tracing::info!( + "{name}: stopped by the server's shutdown; the record stays RUNNING with its \ + spool and its progress for a resume" + ); + (BackupJobOutcome::Cancelled, false) + } + Err(Stop::Fail(reason)) => { + let path = fail_restore( + &*storage, + &*claim, + &target, + &reason, + entry.as_ref(), + &env.shutdown, + ) + .await; + tracing::error!("{name}: FAILED ({reason}); {path}"); + // `fail_restore` removed the entry, or left it to the sweep. + (BackupJobOutcome::Failed, false) + } + }; + if remove_entry && let Some(entry) = &entry { + remove_spool_entry(entry).await; + } + // After the failure path too, so a FAILED job's duration covers its + // FAILED write and the target's removal. + let duration = started.elapsed(); + drop(heartbeat); + if let Some(metrics) = metrics { + #[allow(clippy::cast_precision_loss)] + let duration_us = duration.as_micros() as f64; + metrics.record_restore_job(outcome, duration_us); + match outcome { + BackupJobOutcome::Succeeded => { + metrics.record_worker_success(QuerySource::RestoreJob, duration_us); + } + BackupJobOutcome::Failed => metrics.record_worker_error(QuerySource::RestoreJob), + // A clean shutdown is not a worker error. + BackupJobOutcome::Cancelled => {} + } + } + outcome +} + +/// What step 6 settled to. +enum Ended { + /// The backend's own answer to the DONE write. + Written(Finished), + /// The DONE write's answer was lost and the record re-read DONE under + /// this generation: the restore is complete; whether the table is ACTIVE + /// or awaits the backend's worker (MongoDB) is a fact the backend holds + /// and this job did not see. + DoneOnReRead, +} + +/// Step 6's write (module doc): `finish_restore`, with an error from it +/// handled as a conditional write whose outcome is unknown. The record is +/// re-read through the claim: DONE under this generation is the write +/// landed ([`Ended::DoneOnReRead`], with a WARN naming the error), RUNNING +/// under this generation is the write not taken, sent once more at most; +/// another generation, another status, a failed re-read, or a second error +/// after a RUNNING re-read is the DONE write that matched no record, after +/// which the job makes no table access. +async fn finish( + storage: &S, + claim: &dyn JobClaim, + target: &RestoreTargetKey, + item_count: u64, + size_bytes: u64, +) -> Result { + let not_matched = |why: String| Ok(Ended::Written(Finished::NotMatched(why))); + for attempt in 0..2 { + let error = match storage + .finish_restore(target, claim, item_count, size_bytes) + .await + { + Ok(finished) => return Ok(Ended::Written(finished)), + Err(e) => e, + }; + let row = match claim.read().await { + Ok(row) => row, + Err(e) => { + return not_matched(format!( + "the DONE write failed ({error}) and the record could not be re-read: {e}" + )); + } + }; + if row.owner_generation != claim.generation() { + return not_matched(format!( + "the DONE write failed ({error}) and the record carries generation {} (this \ + claim's is {})", + row.owner_generation, + claim.generation() + )); + } + if row.status == JobStatus::Done { + tracing::warn!( + "the DONE write returned an error ({error}) and the record is DONE under this \ + job's generation: the write landed and the restore is complete" + ); + return Ok(Ended::DoneOnReRead); + } + if row.status != JobStatus::Running { + return not_matched(format!( + "the DONE write failed ({error}) and the record is {} under this claim", + row.status + )); + } + if attempt == 1 { + return not_matched(format!( + "the DONE write failed twice ({error}) and the record is still RUNNING under \ + this claim; the reconciler resumes it once the record is stale" + )); + } + tracing::warn!( + "the DONE write returned an error ({error}) and the record is still RUNNING under \ + this claim; writing once more" + ); + } + unreachable!("the loop returns on its second attempt") +} + +/// The end of the restore's failure path. +#[derive(Debug)] +pub(crate) enum RestoreFailurePath { + /// FAILED written and the target removed. + Removed, + /// FAILED written; the target was no longer a CREATING restore target + /// with this id and belongs to whoever moved it. + LeftToOwner, + /// FAILED written; the removal failed and the reconciler's FAILED-target + /// duty removes the target once the record is stale. + RemoveFailed(String), + /// FAILED not written by this claim (the record is another's, or the + /// claim connection failed): no table access was made. + NoTableAccess(String), +} + +impl std::fmt::Display for RestoreFailurePath { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + Self::Removed => f.write_str("the target was removed"), + Self::LeftToOwner => f.write_str( + "the target was no longer a CREATING restore target with this id and is left to \ + its owner", + ), + Self::RemoveFailed(e) => write!( + f, + "the target could not be removed and the reconciler removes it once the record \ + is stale: {e}" + ), + Self::NoTableAccess(why) => write!( + f, + "the record is not this job's to write ({why}); no table access was made" + ), + } + } +} + +/// Step 7 (module doc): FAILED with `reason` through `claim`, then +/// `remove_restore_target` only when that write matched; the spool entry, +/// when there is one, removed when the write matched and left to the sweep +/// when it did not (the record is another's, whose job may be reading the +/// entry). +pub(crate) async fn fail_restore( + storage: &S, + claim: &dyn JobClaim, + target: &RestoreTargetKey, + reason: &str, + entry: Option<&spool::Entry>, + shutdown: &CancellationToken, +) -> RestoreFailurePath { + let fields = StatusFields { + failure_reason: Some(reason.to_owned()), + completed: true, + ..StatusFields::default() + }; + let path = match write_status(claim, JobStatus::Running, JobStatus::Failed, &fields).await { + Written::Matched => { + kill_point(KillPoint::RestoreBetweenFailedAndRemoval, shutdown).await; + match storage.remove_restore_target(target).await { + Ok(true) => RestoreFailurePath::Removed, + Ok(false) => RestoreFailurePath::LeftToOwner, + Err(e) => RestoreFailurePath::RemoveFailed(e.to_string()), + } + } + Written::NotMatched(why) | Written::Error(why) => RestoreFailurePath::NoTableAccess(why), + }; + if let (Some(entry), false) = (entry, matches!(path, RestoreFailurePath::NoTableAccess(_))) { + remove_spool_entry(entry).await; + } + path +} + +async fn remove_spool_entry(entry: &spool::Entry) { + if let Err(e) = entry.remove().await { + tracing::error!( + "the spool entry {} could not be removed; the reconciler's sweep retries: {e}", + entry.dir().display() + ); + } +} + +/// A dead owner's record as the reconciler hands it to [`resume`]. +pub(crate) struct DeadRecord { + /// The record as read under the takeover's claim. + pub row: RestoreJobRow, + /// The `spool_host` the record carried before the takeover (which wrote + /// this host's): whose spool entry, if any, holds the files. + pub recorded_spool_host: Option, + /// The progress rows a backend keeps apart from the record. + pub progress_rows: Vec<(u32, u64)>, + /// The instant `claim_restore_dead` returned: the resumed job's first + /// confirmed beat (module doc). + pub claimed_at: Instant, +} + +/// The reconciler's entry: a dead owner's RUNNING record taken over +/// (`claim`), resumed from its progress (the record and, on PostgreSQL, its +/// progress rows). The manifest and sidecar are read from this host's spool +/// entry when the record named this host before the takeover and both are +/// there and verify, else from the store; either way they are verified and +/// must name the record's backup. A manifest that cannot be read or is +/// refused fails the restore (step 7), and this host's entry, when the +/// record named this host, goes with it. The first confirmed beat is +/// `record.claimed_at`, the takeover's return. +pub(crate) async fn resume( + storage: Arc, + env: JobEnv, + permit: JobPermit, + claim: Box, + record: DeadRecord, + limits: RestorableLimits, +) where + S: BackupCatalog + RestoreEngine + ?Sized + Send + Sync + 'static, +{ + let started = Instant::now(); + let claim: Arc = Arc::from(claim); + let DeadRecord { + row, + recorded_spool_host, + progress_rows, + claimed_at, + } = record; + let target = row.target.clone(); + let name = format!( + "restore of {} into {} (account {})", + row.source_backup_arn, target.table_name, target.account_id + ); + // This host's entry, when the record named this host before the + // takeover: the spooled documents are read from it, and a resume that + // fails before it runs removes it with the failure path, as the job + // removes its entry at every failed end. An entry that cannot be opened + // is left to the sweep and the store's documents are read. + let entry = env + .spool + .as_ref() + .filter(|spool| recorded_spool_host.as_deref() == Some(spool.spool_host.as_str())) + .and_then(|spool| { + spool::Entry::open( + &spool.path, + &target.table_id, + recorded_spool_host.as_deref(), + &spool.spool_host, + ) + .ok() + }); + let prepared = prepare_resume(&env, &row, entry.as_ref(), &progress_rows).await; + let (arn, manifest, manifest_bytes, sidecar_bytes, key_schema, progress, from_spool) = + match prepared { + Ok(prepared) => prepared, + Err(reason) => { + let path = fail_restore( + &*storage, + &*claim, + &target, + &reason, + entry.as_ref(), + &env.shutdown, + ) + .await; + tracing::error!("{name}: FAILED on resume ({reason}); {path}"); + if let Some(metrics) = env.metrics.as_deref() { + metrics.record_worker_error(QuerySource::RestoreJob); + #[allow(clippy::cast_precision_loss)] + let duration_us = started.elapsed().as_micros() as f64; + metrics.record_restore_job(BackupJobOutcome::Failed, duration_us); + } + drop(permit); + return; + } + }; + tracing::info!( + "{name}: resumed by this instance; the manifest was read from {}", + if from_spool { + "this host's spool" + } else { + "the store" + } + ); + let job = RestoreJob { + storage, + claim, + target, + arn, + manifest: Arc::new(manifest), + manifest_bytes, + sidecar_bytes, + key_schema, + limits, + env, + started, + claimed_at, + progress: Some(progress), + recorded_spool_host, + documents_from_spool: from_spool, + }; + run(job).await; + drop(permit); +} + +/// What a resume reads before it runs: the manifest and sidecar (from this +/// host's spool entry when there is one and the spooled pair verifies, else +/// the store), verified, with the key schema and the progress, and whether +/// the pair came from the spool; the `Err` is the FAILED reason. +type Prepared = ( + BackupArn, + ExtenddbManifest, + Vec, + Vec, + RestorableKeySchema, + Progress, + bool, +); + +/// The manifest of a (manifest, sidecar) pair that verifies and names +/// `arn`; the `Err` says what failed. +fn verified_manifest( + manifest_bytes: &[u8], + sidecar_bytes: &[u8], + arn: &BackupArn, +) -> Result { + verify_manifest_checksum(manifest_bytes, sidecar_bytes) + .map_err(|e| format!("the manifest's checksum sidecar does not match: {e}"))?; + let manifest = ExtenddbManifest::from_json_bytes(manifest_bytes) + .map_err(|e| format!("the manifest is refused: {e}"))?; + match manifest.backup_arn() { + Ok(named) if named == *arn => Ok(manifest), + Ok(named) => Err(format!( + "the manifest names another backup, {}", + named.to_arn_string() + )), + Err(e) => Err(format!("the manifest's backup ARN is refused: {e}")), + } +} + +async fn prepare_resume( + env: &JobEnv, + row: &RestoreJobRow, + entry: Option<&spool::Entry>, + progress_rows: &[(u32, u64)], +) -> Result { + let target = &row.target; + let arn = BackupArn::parse(&row.source_backup_arn) + .map_err(|e| format!("the record's backup ARN is refused: {e}"))?; + let prefix = arn.store_prefix(); + let spooled = entry.and_then(|entry| { + let manifest = entry.read_document(EXTENDDB_MANIFEST_FILE).ok()??; + let sidecar = entry.read_document(MANIFEST_CHECKSUM_FILE).ok()??; + Some((manifest, sidecar)) + }); + // The spooled pair, when it verifies; a pair a crash left short, or + // that names another backup, is replaced by the store's. + let spooled = spooled.and_then(|(manifest_bytes, sidecar_bytes)| { + match verified_manifest(&manifest_bytes, &sidecar_bytes, &arn) { + Ok(manifest) => Some((manifest, manifest_bytes, sidecar_bytes)), + Err(e) => { + tracing::warn!( + "the spooled manifest and sidecar of the restore into {} (account {}) do not \ + verify ({e}); the store's are read instead", + target.table_name, + target.account_id + ); + None + } + } + }); + let from_spool = spooled.is_some(); + let (manifest, manifest_bytes, sidecar_bytes) = match spooled { + Some(documents) => documents, + None => { + let read = async |name: &str, bound: u64| -> Result, String> { + match super::reconciler::read_object(&*env.store, &format!("{prefix}{name}"), bound) + .await + { + Ok(ReadObject::Bytes(bytes)) => Ok(bytes), + Ok(ReadObject::Absent) => { + Err(format!("the store holds no {name} for the backup")) + } + Ok(ReadObject::Oversize) => Err(format!( + "the store's {name} for the backup is over {bound} bytes, which no \ + backup this server wrote produces" + )), + Err(e) => { + if let Some(metrics) = env.metrics.as_deref() { + metrics.record_backup_store_error("get"); + } + Err(format!("reading {name} from the store: {e}")) + } + } + }; + let manifest_bytes = read(EXTENDDB_MANIFEST_FILE, MANIFEST_READ_BOUND).await?; + let sidecar_bytes = read(MANIFEST_CHECKSUM_FILE, SIDECAR_READ_BOUND).await?; + let manifest = verified_manifest(&manifest_bytes, &sidecar_bytes, &arn)?; + (manifest, manifest_bytes, sidecar_bytes) + } + }; + let key_schema = RestorableKeySchema::resolve( + &manifest.source_table.key_schema, + &manifest.source_table.attribute_definitions, + ) + .map_err(|e| format!("the key schema cannot be checked: {e}"))?; + let progress = Progress::from_records(row, progress_rows, &manifest); + Ok(( + arn, + manifest, + manifest_bytes, + sidecar_bytes, + key_schema, + progress, + from_spool, + )) +} + +/// What a test can do to a running job by its target table id: end its +/// heartbeat timer without raising the loss (`HeartbeatHandle::stop_timer`) +/// while the job is held at a gate of the storage double, and give its +/// loads a smaller chunk byte bound than [`load::LOAD_CHUNK_BYTES`]. Tests +/// that use it give their target a name of their own; the maps are +/// process-wide. +#[cfg(test)] +pub(crate) mod test_support { + use std::collections::HashMap; + use std::sync::{LazyLock, Mutex, PoisonError}; + + use super::HeartbeatHandle; + + static TIMERS: LazyLock>> = + LazyLock::new(|| Mutex::new(HashMap::new())); + static CHUNK_BYTES: LazyLock>> = + LazyLock::new(|| Mutex::new(HashMap::new())); + + /// Give the loads of the job into `table_id` a chunk byte bound of + /// `bytes`. + pub(crate) fn set_chunk_bytes(table_id: &str, bytes: u64) { + CHUNK_BYTES + .lock() + .unwrap_or_else(PoisonError::into_inner) + .insert(table_id.to_owned(), bytes); + } + + pub(super) fn chunk_bytes_for(table_id: &str) -> u64 { + CHUNK_BYTES + .lock() + .unwrap_or_else(PoisonError::into_inner) + .get(table_id) + .copied() + .unwrap_or(super::load::LOAD_CHUNK_BYTES) + } + + pub(super) fn register_timer(table_id: &str, heartbeat: &HeartbeatHandle) { + TIMERS + .lock() + .unwrap_or_else(PoisonError::into_inner) + .insert(table_id.to_owned(), heartbeat.timer_abort_handle()); + } + + /// End the heartbeat timer of the running job into `table_id`. + pub(crate) fn stop_timer(table_id: &str) { + let handle = TIMERS + .lock() + .unwrap_or_else(PoisonError::into_inner) + .get(table_id) + .cloned() + .expect("a job into the table is running"); + handle.abort(); + } +} diff --git a/crates/engine/src/backup/restore/download.rs b/crates/engine/src/backup/restore/download.rs new file mode 100644 index 000000000..fd61f1ce0 --- /dev/null +++ b/crates/engine/src/backup/restore/download.rs @@ -0,0 +1,308 @@ +// Copyright 2026 ExtendDB contributors +// SPDX-License-Identifier: Apache-2.0 + +//! Step 3 of a restore: every data file read from the store and compared +//! with the manifest's `sha256`, `md5` and `bytes` before any file is +//! loaded. +//! +//! On an S3 store each file is written to the job's spool entry as it +//! arrives ([`Dest::Spool`]), and committed with its `.verified` marker once +//! the digests match; a resume skips the files an earlier run marked. On a +//! filesystem store ([`Dest::InPlace`]) the file is hashed as the store +//! streams it and nothing is written: the load reads the same file through +//! a second `get`, hashing it again, so a file replaced between the two +//! passes fails there. +//! +//! A `get` that fails as a transport error, at the call or while the body +//! arrives, is retried with a backoff of `min(2^n s, 60 s)` plus jitter for +//! as long as the job has been downloading for less than +//! [`DOWNLOAD_RETRY_BOUND`], counted from the first download attempt of the +//! job (one clock for every file, so a store that flaps through a backup of +//! many files costs the restore the bound once, not once per file), the +//! record RUNNING and heartbeated meanwhile; the shutdown token and the +//! heartbeat's loss end the wait. Every other failure fails the restore at +//! once: a digest or size mismatch (the file is not the backup's), an absent +//! file (a prefix changed under an AVAILABLE row), a permission or key +//! refusal, a local I/O error (the filesystem store's `Io`, or the spool's +//! disk), or any other store error. The filesystem store reports a mount +//! that stopped answering as `Io`, so the retry covers S3 stores only. +//! Every `get` the store fails, retried or not, is one `BackupStoreErrors{get}`; +//! a mismatch and the spool's own errors are not the store's. + +use std::time::Duration; + +use extenddb_backup_store::{BackupStore, StoreError}; +use extenddb_core::backup_format::{CompressedHasher, DataFileEntry, ExtenddbManifest}; +use extenddb_core::metrics::MetricsCollector; +use extenddb_storage::CancellationToken; +use futures::TryStreamExt; +use tokio::io::AsyncWriteExt; +use tokio::time::Instant; + +use super::super::job::heartbeat::HeartbeatHandle; +use super::spool::Entry; + +/// How long the job's downloads are retried over transport errors, from +/// the job's first download attempt, before the restore fails. +pub(crate) const DOWNLOAD_RETRY_BOUND: Duration = Duration::from_secs(15 * 60); + +/// The longest wait between two attempts. +const MAX_BACKOFF: Duration = Duration::from_secs(60); + +/// Where a verified file goes. +pub(crate) enum Dest<'a> { + /// Hashed as it streams, written nowhere: the load reads the store + /// again (a filesystem store). + InPlace, + /// Written to the job's spool entry and marked `.verified`. + Spool(&'a Entry), +} + +/// Why the download stopped. +#[derive(Debug, Clone, PartialEq, Eq)] +pub(crate) enum Stop { + /// The restore fails with this reason. + Fail(String), + /// The shutdown token fired. + Cancelled, + /// The heartbeat no longer confirms the claim. + ClaimLost(String), +} + +/// What one download runs with. +pub(crate) struct Downloader<'a> { + pub store: &'a dyn BackupStore, + /// The backup's store prefix, which every key is under. + pub prefix: &'a str, + pub heartbeat: &'a HeartbeatHandle, + pub shutdown: &'a CancellationToken, + pub metrics: Option<&'a MetricsCollector>, +} + +/// Download and verify every data file of `manifest` into `dest` (module +/// doc); a file the spool holds verified with the manifest's digest is +/// skipped. Returns the bytes read from the store over every attempt. +pub(crate) async fn download_all( + d: &Downloader<'_>, + manifest: &ExtenddbManifest, + dest: &Dest<'_>, +) -> Result { + // The retry bound's clock: one for the job, started here. + let started = Instant::now(); + let mut bytes = 0; + for (index, entry) in manifest.data_files.iter().enumerate() { + let index = + u32::try_from(index).map_err(|_| Stop::Fail("too many data files".to_owned()))?; + if let Dest::Spool(spool) = dest + && spool.is_verified(index, &entry.sha256) + { + continue; + } + bytes += fetch_verified(d, entry, index, dest, started).await?; + } + Ok(bytes) +} + +/// One file: attempts until the file verifies, a non-transport failure, the +/// retry bound (measured from `started`, the job's first attempt), the +/// shutdown or a lost claim. +async fn fetch_verified( + d: &Downloader<'_>, + entry: &DataFileEntry, + index: u32, + dest: &Dest<'_>, + started: Instant, +) -> Result { + let key = format!("{}{}", d.prefix, entry.key); + let mut bytes = 0; + for attempt in 0u32.. { + if d.shutdown.is_cancelled() { + return Err(Stop::Cancelled); + } + if !d.heartbeat.confirms_the_claim() { + return Err(Stop::ClaimLost( + d.heartbeat + .why_not_confirmed(&format!("to download {}", entry.key)), + )); + } + let outcome = tokio::select! { + biased; + () = d.shutdown.cancelled() => return Err(Stop::Cancelled), + () = d.heartbeat.lost().cancelled() => return Err(Stop::ClaimLost(d.heartbeat.reason())), + outcome = attempt_once(d, entry, index, &key, dest) => outcome, + }; + bytes += outcome.bytes; + if let Some(metrics) = d.metrics + && outcome.bytes > 0 + { + metrics.record_restore_bytes_downloaded(outcome.bytes); + } + let transport = match outcome.result { + Ok(()) => return Ok(bytes), + Err(Attempt::Fail(reason)) => return Err(Stop::Fail(reason)), + Err(Attempt::StoreFailed(reason)) => { + // The store's refusal counts as its error, as a transport + // failure does; the restore then fails at once. + if let Some(metrics) = d.metrics { + metrics.record_backup_store_error("get"); + } + return Err(Stop::Fail(reason)); + } + Err(Attempt::Transport(e)) => e, + }; + if let Some(metrics) = d.metrics { + metrics.record_backup_store_error("get"); + } + let elapsed = started.elapsed(); + if elapsed >= DOWNLOAD_RETRY_BOUND { + return Err(Stop::Fail(format!( + "downloading {}: {transport} (the job has retried its downloads for {} s; the \ + bound is {} s)", + entry.key, + elapsed.as_secs(), + DOWNLOAD_RETRY_BOUND.as_secs() + ))); + } + let wait = backoff(attempt); + tracing::warn!( + "downloading {key}: {transport}; attempt {} retried in {:.1} s", + attempt + 2, + wait.as_secs_f64() + ); + tokio::select! { + biased; + () = d.shutdown.cancelled() => return Err(Stop::Cancelled), + () = d.heartbeat.lost().cancelled() => return Err(Stop::ClaimLost(d.heartbeat.reason())), + () = tokio::time::sleep(wait) => {} + } + } + unreachable!("the attempt loop returns") +} + +/// `min(2^attempt s, 60 s)` plus a jitter of up to one second. +fn backoff(attempt: u32) -> Duration { + let base = Duration::from_secs(1u64 << attempt.min(6)); + let jitter_ms = u64::from(uuid::Uuid::new_v4().as_u128() as u32) % 1000; + base.min(MAX_BACKOFF) + Duration::from_millis(jitter_ms) +} + +/// How one attempt ended. +enum Attempt { + /// The restore fails with this reason; no retry, and not the store's + /// error (a digest or size mismatch, the spool's disk). + Fail(String), + /// The store failed the `get` with an error that is not retried (an + /// absent file, a refusal, an I/O error, any other); the restore fails + /// with this reason and the error is counted as the store's. + StoreFailed(String), + /// Retried within the bound. + Transport(StoreError), +} + +struct AttemptOutcome { + result: Result<(), Attempt>, + /// Bytes this attempt read from the store. + bytes: u64, +} + +/// One `get` of the file, hashed and compared, written to the spool when +/// `dest` is one. +async fn attempt_once( + d: &Downloader<'_>, + entry: &DataFileEntry, + index: u32, + key: &str, + dest: &Dest<'_>, +) -> AttemptOutcome { + let mut bytes = 0; + let result = attempt_inner(d, entry, index, key, dest, &mut bytes).await; + AttemptOutcome { result, bytes } +} + +async fn attempt_inner( + d: &Downloader<'_>, + entry: &DataFileEntry, + index: u32, + key: &str, + dest: &Dest<'_>, + bytes: &mut u64, +) -> Result<(), Attempt> { + let mut body = match d.store.get(key).await { + Ok(body) => body, + Err(e @ StoreError::Transport { .. }) => return Err(Attempt::Transport(e)), + Err(StoreError::NotFound) => { + return Err(Attempt::StoreFailed(format!( + "data file {} is not in the store", + entry.key + ))); + } + Err(e) => { + return Err(Attempt::StoreFailed(format!( + "reading data file {} from the store: {e}", + entry.key + ))); + } + }; + let mut file = match dest { + Dest::InPlace => None, + Dest::Spool(spool) => Some(spool.begin_file(index).await.map_err(|e| { + Attempt::Fail(format!("writing data file {} to the spool: {e}", entry.key)) + })?), + }; + let mut hasher = CompressedHasher::new(); + loop { + let chunk = match body.try_next().await { + Ok(Some(chunk)) => chunk, + Ok(None) => break, + Err(e @ StoreError::Transport { .. }) => return Err(Attempt::Transport(e)), + Err(e) => { + return Err(Attempt::StoreFailed(format!( + "reading data file {} from the store: {e}", + entry.key + ))); + } + }; + *bytes += chunk.len() as u64; + hasher.update(&chunk); + if let Some(file) = &mut file { + file.write_all(&chunk).await.map_err(|e| { + Attempt::Fail(format!("writing data file {} to the spool: {e}", entry.key)) + })?; + } + } + entry + .expectations() + .verify_digest(&hasher.finish()) + .map_err(|e| { + Attempt::Fail(format!( + "data file {} does not match the manifest: {e}", + entry.key + )) + })?; + if let (Some(file), Dest::Spool(spool)) = (file, dest) { + spool + .commit_file(index, file, &entry.sha256) + .await + .map_err(|e| { + Attempt::Fail(format!("writing data file {} to the spool: {e}", entry.key)) + })?; + } + Ok(()) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn the_backoff_doubles_to_a_minute_with_jitter() { + for (attempt, base) in [(0, 1), (1, 2), (3, 8), (5, 32), (6, 64), (20, 64)] { + let wait = backoff(attempt); + let base = Duration::from_secs(base).min(MAX_BACKOFF); + assert!( + wait >= base && wait < base + Duration::from_secs(1), + "{attempt}: {wait:?}" + ); + } + } +} diff --git a/crates/engine/src/backup/restore/load.rs b/crates/engine/src/backup/restore/load.rs new file mode 100644 index 000000000..4435a685c --- /dev/null +++ b/crates/engine/src/backup/restore/load.rs @@ -0,0 +1,287 @@ +// Copyright 2026 ExtendDB contributors +// SPDX-License-Identifier: Apache-2.0 + +//! Step 5 of a restore: one verified data file decoded, validated and +//! handed to the backend's `bulk_load`. +//! +//! The decoder is synchronous (`std::io::Read`), so it runs on the blocking +//! pool: over the spooled file through `ExtenddbManifest::reader_prehashed` +//! (the download hashed those bytes), or, on a filesystem store, over a +//! second `get` of the store bridged to `Read`, through +//! `ExtenddbManifest::reader`, which hashes again. On that in-place path the +//! items are handed to `bulk_load` as they decode and the digest is checked +//! at the end of the stream, so the items of a file replaced between the two +//! passes reach the backend (and, on a backend that commits per batch, the +//! CREATING target) before the end-of-stream check fails the file and the +//! failure path removes the target; the store root's owner-only mode is what +//! keeps the file from changing. Each item past the resume's `start_at` is +//! validated against the target's limits and key schema +//! (`validate_restorable_item`), then collected into chunks of at most +//! [`LOAD_CHUNK_ITEMS`] items or [`LOAD_CHUNK_BYTES`] of line text (the +//! bytes the decoder read for the items, the unit the backup side budgets +//! in), whichever comes first, and sent over a channel of two chunks to the +//! async side, which flattens them into the item stream of one `bulk_load` +//! call per file. The memory a file's load holds is therefore the decoded +//! items of at most `LOAD_CHUNK_BYTES` plus one line of line text per chunk +//! (the line that crosses the bound is kept in the chunk it was read for) at +//! four chunk positions (two in the channel, the one being filled, the one +//! being drained), the decoder's line (at most `MAX_LINE_BYTES`) and the +//! deserializer's intermediate value for it, and the backend's own batch. +//! +//! A decode or validation failure, the shutdown token and the heartbeat +//! (its loss, or a timer that no longer confirms the claim, both read at +//! every chunk boundary) end the stream with an `Err` item, which the +//! backend answers by failing the file (the design: nothing of a failed file +//! is recorded where the backend writes a file in one transaction); the +//! reason is kept here so the job reports it rather than the backend's +//! wrapping of it. A decoder that stops without ending the stream itself (a +//! panic on the blocking thread) fails the file too: a clean end of stream +//! the backend saw is not taken for a complete file. The join of the +//! decoder's thread after `bulk_load` returned has no bound of its own: on +//! the in-place path the thread reads the store through the bridge, and a +//! filesystem mount that stopped answering holds it as it holds the store's +//! own `get`, until the mount answers. + +use std::io::Read; +use std::path::PathBuf; +use std::sync::{Arc, Mutex, PoisonError}; + +use extenddb_backup_store::BackupStore; +use extenddb_core::backup_format::{DataFileEntry, ExtenddbManifest}; +use extenddb_core::types::Item; +use extenddb_core::validation::{RestorableKeySchema, RestorableLimits, validate_restorable_item}; +use extenddb_storage::backup_catalog::RestoreTargetKey; +use extenddb_storage::error::StorageError; +use extenddb_storage::{BulkLoadFile, CancellationToken, JobClaim, RestoreEngine}; +use futures::{StreamExt, TryStreamExt}; +use tokio::sync::mpsc; + +use super::super::job::heartbeat::{ClaimCheck, HeartbeatHandle}; +pub(crate) use super::download::Stop; + +/// Items per chunk handed to the backend, at most. +pub(crate) const LOAD_CHUNK_ITEMS: usize = 1000; +/// Line bytes per chunk (the data file's text the decoder read for the +/// chunk's items), at most; a chunk closes at the first item that takes it +/// to this. +pub(crate) const LOAD_CHUNK_BYTES: u64 = 32 * 1024 * 1024; +/// Chunks the channel between the decoder and the backend holds. +const CHUNK_CHANNEL: usize = 2; + +/// Where the file's bytes are read from. +pub(crate) enum Source { + /// The spooled, verified file. + Spool(PathBuf), + /// The store again (a filesystem store, read in place). + InPlace { + store: Arc, + key: String, + }, +} + +/// One file's load. +pub(crate) struct FileLoad<'a> { + pub manifest: &'a Arc, + pub entry: &'a DataFileEntry, + pub file_index: u32, + /// Items of the file an earlier run applied, decoded and skipped. + pub start_at: u64, + pub reapplying: bool, + pub key_schema: &'a RestorableKeySchema, + pub limits: RestorableLimits, + pub shutdown: &'a CancellationToken, + pub heartbeat: &'a HeartbeatHandle, + /// The chunk's byte bound: [`LOAD_CHUNK_BYTES`], or a smaller bound a + /// test set for its job. + pub chunk_bytes: u64, +} + +/// Decode `source` from `load.start_at` on, validate every item, and write +/// the file through `storage.bulk_load` (module doc). Returns the items the +/// backend applied. +pub(crate) async fn load_file( + storage: &S, + target: &RestoreTargetKey, + claim: &dyn JobClaim, + load: FileLoad<'_>, + source: Source, +) -> Result { + let key = load.entry.key.clone(); + let (read, prehashed): (Box, bool) = match source { + Source::Spool(path) => { + let file = std::fs::File::open(&path).map_err(|e| { + Stop::Fail(format!( + "opening data file {key} in the spool ({}): {e}", + path.display() + )) + })?; + (Box::new(file), true) + } + Source::InPlace { + store, + key: store_key, + } => { + let body = store + .get(&store_key) + .await + .map_err(|e| Stop::Fail(format!("reading data file {key} from the store: {e}")))?; + let reader = tokio_util::io::StreamReader::new(body.map_err(std::io::Error::other)); + let bridge = tokio_util::io::SyncIoBridge::new_with_handle( + reader, + tokio::runtime::Handle::current(), + ); + (Box::new(bridge), false) + } + }; + let stop: Arc>> = Arc::new(Mutex::new(None)); + let (tx, rx) = mpsc::channel::, StorageError>>(CHUNK_CHANNEL); + let feeder = Feeder { + manifest: Arc::clone(load.manifest), + entry: load.entry.clone(), + start_at: load.start_at, + key_schema: load.key_schema.clone(), + limits: load.limits, + shutdown: load.shutdown.clone(), + claim: load.heartbeat.claim_check(), + chunk_bytes: load.chunk_bytes, + stop: Arc::clone(&stop), + tx, + }; + let decoder = tokio::task::spawn_blocking(move || feeder.run(read, prehashed)); + let items = futures::stream::unfold(rx, |mut rx| async move { + rx.recv().await.map(|chunk| (chunk, rx)) + }) + .flat_map(|chunk| match chunk { + Ok(items) => futures::stream::iter(items.into_iter().map(Ok)).boxed(), + Err(e) => futures::stream::once(async move { Err(e) }).boxed(), + }) + .boxed(); + let file = BulkLoadFile { + file_index: load.file_index, + start_at: load.start_at, + reapplying: load.reapplying, + }; + let result = tokio::select! { + biased; + () = load.heartbeat.lost().cancelled() => Err(None), + result = storage.bulk_load(target, claim, file, items) => result.map_err(Some), + }; + // The stream is dropped with the call, so a decoder blocked on the + // channel returns; its thread is joined before the file is reported. A + // decoder that did not return (it panicked) dropped the channel without + // an `Err` item, so the backend saw a clean end of a short stream: the + // file fails here, where a backend that writes per file rolls it back. + if decoder.await.is_err() { + return Err(Stop::Fail(format!( + "data file {key}: the decoder stopped unexpectedly" + ))); + } + let stopped = stop.lock().unwrap_or_else(PoisonError::into_inner).take(); + match (result, stopped) { + (Ok(applied), None) => Ok(applied), + (Ok(_), Some(stop)) | (Err(_), Some(stop)) => Err(stop), + (Err(None), None) => Err(Stop::ClaimLost(load.heartbeat.reason())), + (Err(Some(e)), None) => Err(Stop::Fail(format!("data file {key}: {e}"))), + } +} + +/// The blocking side: the decoder, the validator and the chunking. +struct Feeder { + manifest: Arc, + entry: DataFileEntry, + start_at: u64, + key_schema: RestorableKeySchema, + limits: RestorableLimits, + shutdown: CancellationToken, + /// The heartbeat's pre-write check, read at every chunk boundary: the + /// loss raised, or a timer that no longer confirms the claim. + claim: ClaimCheck, + chunk_bytes: u64, + stop: Arc>>, + tx: mpsc::Sender, StorageError>>, +} + +impl Feeder { + fn run(self, read: Box, prehashed: bool) { + let key = self.entry.key.clone(); + let mut reader = match if prehashed { + self.manifest.reader_prehashed(&self.entry, read) + } else { + self.manifest.reader(&self.entry, read) + } { + Ok(reader) => reader, + Err(e) => { + self.end(Stop::Fail(format!("data file {key}: {e}"))); + return; + } + }; + let mut index: u64 = 0; + let mut chunk = Vec::new(); + // The line bytes of the items in `chunk`, from the decoder's count. + let mut chunk_bytes: u64 = 0; + let mut decoded_before = reader.bytes_decoded(); + loop { + match reader.next_item() { + Ok(Some(item)) => { + let line_bytes = reader.bytes_decoded() - decoded_before; + decoded_before = reader.bytes_decoded(); + index += 1; + if index <= self.start_at { + continue; + } + if let Err(e) = validate_restorable_item(&item, &self.key_schema, &self.limits) + { + self.end(Stop::Fail(format!( + "data file {key}: item cannot be restored: {e}" + ))); + return; + } + chunk_bytes += line_bytes; + chunk.push(item); + if chunk.len() >= LOAD_CHUNK_ITEMS || chunk_bytes >= self.chunk_bytes { + if self.shutdown.is_cancelled() { + self.end(Stop::Cancelled); + return; + } + if !self.claim.confirms_the_claim() { + self.end(Stop::ClaimLost( + "the claim was lost while the file was being decoded".to_owned(), + )); + return; + } + if self + .tx + .blocking_send(Ok(std::mem::take(&mut chunk))) + .is_err() + { + // The backend ended the call; nothing more to feed. + return; + } + chunk_bytes = 0; + } + } + Ok(None) => { + if !chunk.is_empty() { + let _ = self.tx.blocking_send(Ok(chunk)); + } + return; + } + Err(e) => { + self.end(Stop::Fail(format!("data file {key}: {e}"))); + return; + } + } + } + } + + /// Record why the stream ends and end it with an `Err` item. + fn end(&self, stop: Stop) { + let text = match &stop { + Stop::Fail(reason) => reason.clone(), + Stop::Cancelled => "the restore was stopped by the server's shutdown".to_owned(), + Stop::ClaimLost(why) => why.clone(), + }; + *self.stop.lock().unwrap_or_else(PoisonError::into_inner) = Some(stop); + let _ = self.tx.blocking_send(Err(StorageError::Internal(text))); + } +} diff --git a/crates/engine/src/backup/restore/spool.rs b/crates/engine/src/backup/restore/spool.rs new file mode 100644 index 000000000..d13730864 --- /dev/null +++ b/crates/engine/src/backup/restore/spool.rs @@ -0,0 +1,516 @@ +// Copyright 2026 ExtendDB contributors +// SPDX-License-Identifier: Apache-2.0 + +//! The restore spool of an S3 store: where a restore job downloads the +//! backup's files before it loads any of them. +//! +//! One restore owns one entry, `restore-
/` under the spool +//! directory the server took at startup (`backup_store::take_spool`: owner +//! only, locked). The entry is created `0700` and every file in it `0600` +//! (the database and its sidecars are `0600`; the spool holds the same +//! data). A data file is written as `.json.gz.partial`, synced, +//! renamed to `.json.gz`, and only then marked `.json.gz.verified`, +//! after the job has compared its digests with the manifest's; the marker +//! holds the file's sha256, and a file counts as verified only while the +//! marker names the digest the manifest in hand records for it, so a +//! replaced manifest does not have an old entry's files accepted. A +//! `.partial`, an unmarked file or a marker naming another digest is one a +//! crash or another manifest left, and the next run of the job removes it +//! and downloads the file again. The manifest and its sidecar are written to +//! the entry too, the same way (`.partial`, synced, renamed), so a +//! crash during the write leaves the previous document or none and never a +//! truncated one, and a resume on the same host reads them from the spool +//! and verifies them again (a pair that fails verification is replaced by +//! the store's). An entry is this host's only while the record's +//! `spool_host` is this host's: a successor on another host finds no entry +//! of its own, and one whose own old entry survives from an earlier run of +//! the job (the record moved to another host and back) removes it before +//! use ([`Entry::open`]). The job removes its entry when it ends by any +//! path but the server's shutdown, which keeps it for a resume, and a job +//! that learnt the record is another's (its claim lost, its final write +//! matching no record), which leaves it to the sweep; the reconciler's sweep +//! ([`list_entries`], then [`sweep_entries`]) removes the entries it listed +//! before it read the records whose job is not RUNNING with this host's +//! `spool_host`, so an entry opened after the listing is never a candidate. + +use std::collections::HashSet; +use std::io; +use std::path::{Path, PathBuf}; + +use tokio::io::AsyncWriteExt; + +/// The prefix of an entry's directory name, followed by the table id. +const ENTRY_PREFIX: &str = "restore-"; +/// The suffix of a file being downloaded. +const PARTIAL: &str = ".partial"; +/// The suffix of the marker of a file whose digests the job verified. +const VERIFIED: &str = ".verified"; + +/// One restore's directory in the spool. +pub(crate) struct Entry { + dir: PathBuf, + /// Make the rename into place fail, so a test can see what a crash + /// there leaves. + #[cfg(test)] + pub(crate) fail_rename: std::sync::atomic::AtomicBool, +} + +impl Entry { + /// The entry of the restore into table `table_id`, created `0700` when + /// it does not exist. An existing entry is a resume's and is kept when + /// `record_spool_host`, the `spool_host` the record carried before this + /// job's claim, is `this_host`; otherwise the files in it were verified + /// by a run whose record has since been another host's, and the entry + /// is removed and created again, empty. + pub(crate) fn open( + spool: &Path, + table_id: &str, + record_spool_host: Option<&str>, + this_host: &str, + ) -> io::Result { + let dir = spool.join(format!("{ENTRY_PREFIX}{table_id}")); + if dir.is_dir() && record_spool_host != Some(this_host) { + tracing::info!( + "the spool entry {} is left from a run whose record moved to another host; it is \ + removed and the files downloaded again", + dir.display() + ); + std::fs::remove_dir_all(&dir)?; + } + if !dir.is_dir() { + let mut builder = std::fs::DirBuilder::new(); + #[cfg(unix)] + { + use std::os::unix::fs::DirBuilderExt; + builder.mode(0o700); + } + match builder.create(&dir) { + Ok(()) => {} + Err(e) if e.kind() == io::ErrorKind::AlreadyExists => {} + Err(e) => return Err(e), + } + } + Ok(Self { + dir, + #[cfg(test)] + fail_rename: std::sync::atomic::AtomicBool::new(false), + }) + } + + /// The entry's directory. + pub(crate) fn dir(&self) -> &Path { + &self.dir + } + + /// The path of data file `index` once downloaded and renamed. + pub(crate) fn file_path(&self, index: u32) -> PathBuf { + self.dir.join(format!("{index:06}.json.gz")) + } + + fn partial_path(&self, index: u32) -> PathBuf { + let mut name = self.file_path(index).into_os_string(); + name.push(PARTIAL); + PathBuf::from(name) + } + + fn verified_path(&self, index: u32) -> PathBuf { + let mut name = self.file_path(index).into_os_string(); + name.push(VERIFIED); + PathBuf::from(name) + } + + /// Whether data file `index` is in the entry with a `.verified` marker + /// naming `sha256`, the manifest's digest of the file: a file an earlier + /// run of the job downloaded and verified against the same manifest. + pub(crate) fn is_verified(&self, index: u32, sha256: &str) -> bool { + self.file_path(index).is_file() + && std::fs::read(self.verified_path(index)).is_ok_and(|marker| { + marker.strip_suffix(b"\n").unwrap_or(&marker) == sha256.as_bytes() + }) + } + + /// Write a document (the manifest, the sidecar) as `name` in the entry + /// the way a data file is written: `.partial`, `0600`, synced, + /// then renamed over `name` and the directory synced. An existing + /// document stands until the rename, so a crash during the write leaves + /// it whole or leaves the new one, never a truncated file. + pub(crate) async fn write_document(&self, name: &str, bytes: &[u8]) -> io::Result<()> { + let path = self.dir.join(name); + let mut partial = path.clone().into_os_string(); + partial.push(PARTIAL); + let partial = PathBuf::from(partial); + let mut file = open_private(&partial).await?; + file.write_all(bytes).await?; + file.sync_all().await?; + drop(file); + #[cfg(test)] + if self.fail_rename.load(std::sync::atomic::Ordering::Acquire) { + return Err(io::Error::other("the process stopped at the rename")); + } + tokio::fs::rename(&partial, &path).await?; + sync_dir(&self.dir).await + } + + /// The bytes of document `name`, `None` when the entry has no such file. + pub(crate) fn read_document(&self, name: &str) -> io::Result>> { + match std::fs::read(self.dir.join(name)) { + Ok(bytes) => Ok(Some(bytes)), + Err(e) if e.kind() == io::ErrorKind::NotFound => Ok(None), + Err(e) => Err(e), + } + } + + /// Start downloading data file `index`: whatever an earlier attempt + /// left (a `.partial`, an unmarked file, a stale marker) is removed, and + /// the `.partial` is created `0600` for the caller to write. + pub(crate) async fn begin_file(&self, index: u32) -> io::Result { + for path in [ + self.verified_path(index), + self.file_path(index), + self.partial_path(index), + ] { + remove_if_present(&path).await?; + } + open_private(&self.partial_path(index)).await + } + + /// Finish data file `index` after its digests matched: sync the + /// `.partial`, rename it into place, then create and sync the + /// `.verified` marker holding `sha256`, and sync the directory, in that + /// order, so the marker exists only beside a complete file. + pub(crate) async fn commit_file( + &self, + index: u32, + mut file: tokio::fs::File, + sha256: &str, + ) -> io::Result<()> { + file.flush().await?; + file.sync_all().await?; + drop(file); + self.rename_into_place(index).await?; + let mut marker = open_private(&self.verified_path(index)).await?; + marker.write_all(sha256.as_bytes()).await?; + marker.write_all(b"\n").await?; + marker.sync_all().await?; + drop(marker); + sync_dir(&self.dir).await + } + + /// The `.partial` renamed to the file's name. + async fn rename_into_place(&self, index: u32) -> io::Result<()> { + #[cfg(test)] + if self.fail_rename.load(std::sync::atomic::Ordering::Acquire) { + return Err(io::Error::other("the process stopped at the rename")); + } + tokio::fs::rename(self.partial_path(index), self.file_path(index)).await + } + + /// Remove the entry and everything in it; an entry already gone is not + /// an error. + pub(crate) async fn remove(&self) -> io::Result<()> { + match tokio::fs::remove_dir_all(&self.dir).await { + Ok(()) => Ok(()), + Err(e) if e.kind() == io::ErrorKind::NotFound => Ok(()), + Err(e) => Err(e), + } + } +} + +/// Create or truncate `path` with mode `0600`. +async fn open_private(path: &Path) -> io::Result { + let mut options = tokio::fs::OpenOptions::new(); + options.write(true).create(true).truncate(true); + #[cfg(unix)] + { + options.mode(0o600); + } + options.open(path).await +} + +async fn remove_if_present(path: &Path) -> io::Result<()> { + match tokio::fs::remove_file(path).await { + Ok(()) => Ok(()), + Err(e) if e.kind() == io::ErrorKind::NotFound => Ok(()), + Err(e) => Err(e), + } +} + +/// Sync a directory so the entries it gained or lost are durable. +async fn sync_dir(dir: &Path) -> io::Result<()> { + tokio::fs::File::open(dir).await?.sync_all().await +} + +/// The table id of an entry's directory name, `None` for any other name in +/// the spool (the `.lock` file, a name an operator put there). +fn entry_table_id(name: &str) -> Option<&str> { + let id = name.strip_prefix(ENTRY_PREFIX)?; + (!id.is_empty() + && id + .bytes() + .all(|b| b.is_ascii_alphanumeric() || b == b'-' || b == b'_')) + .then_some(id) +} + +/// The entries under `spool`, as `(table id, path)`: the reconciler's +/// sweep lists them before it reads the RUNNING records, so an entry a job +/// opens after this listing (its record committed after the listing too) +/// is never a candidate of the sweep that follows. Other names in the spool +/// (the `.lock` file, a name an operator put there, a file under an +/// entry-shaped name) are not listed. +pub(crate) fn list_entries(spool: &Path) -> io::Result> { + let mut entries = Vec::new(); + for dir_entry in std::fs::read_dir(spool)? { + let dir_entry = dir_entry?; + let name = dir_entry.file_name(); + let Some(table_id) = name.to_str().and_then(entry_table_id) else { + continue; + }; + if !dir_entry.file_type()?.is_dir() { + continue; + } + entries.push((table_id.to_owned(), dir_entry.path())); + } + Ok(entries) +} + +/// The reconciler's sweep: remove every entry of `entries` (a +/// [`list_entries`] listing made before the records were read) whose table +/// id is not in `running` (the RUNNING restore targets whose `spool_host` is +/// this host's), and return the ids removed. An entry gone by the time of +/// its removal (its own job removed it between the listing and now) is done, +/// not an error, and not counted; one that cannot be removed is logged and +/// left for the next cycle. +pub(crate) fn sweep_entries( + entries: Vec<(String, PathBuf)>, + running: &HashSet, +) -> Vec { + let mut removed = Vec::new(); + for (table_id, path) in entries { + if running.contains(&table_id) { + continue; + } + match std::fs::remove_dir_all(&path) { + Ok(()) => { + tracing::info!( + "removed the spool entry of the restore into table id {table_id}: no \ + RUNNING restore of this host's spool names it" + ); + removed.push(table_id); + } + Err(e) if e.kind() == io::ErrorKind::NotFound => {} + Err(e) => tracing::error!( + "could not remove the spool entry of table id {table_id}; the next cycle \ + retries: {e}" + ), + } + } + removed +} + +#[cfg(test)] +pub(crate) mod tests { + use super::*; + + /// A scratch directory of this test's own, removed when dropped. + pub(crate) struct Scratch(PathBuf); + + impl Scratch { + pub(crate) fn new() -> Self { + let dir = std::env::temp_dir().join(format!( + "extenddb-engine-test-{}", + uuid::Uuid::new_v4().simple() + )); + std::fs::create_dir(&dir).unwrap(); + Self(dir) + } + + pub(crate) fn path(&self) -> &Path { + &self.0 + } + } + + impl Drop for Scratch { + fn drop(&mut self) { + let _ = std::fs::remove_dir_all(&self.0); + } + } + + #[cfg(unix)] + fn mode(path: &Path) -> u32 { + use std::os::unix::fs::PermissionsExt; + std::fs::metadata(path).unwrap().permissions().mode() & 0o777 + } + + /// The entry is `0700`, its files `0600`; a committed file has its + /// marker naming the digest, a `.partial` has none, a marker naming + /// another digest does not verify the file, and `begin_file` removes + /// what an earlier attempt left. + #[tokio::test] + async fn an_entry_is_private_and_a_marker_follows_the_rename() { + let spool = Scratch::new(); + let entry = Entry::open(spool.path(), "tid-1", Some("host-a"), "host-a").unwrap(); + assert_eq!(mode(entry.dir()), 0o700); + entry + .write_document("extenddb-manifest.json", b"{}") + .await + .unwrap(); + assert_eq!(mode(&entry.dir().join("extenddb-manifest.json")), 0o600); + assert_eq!( + entry.read_document("extenddb-manifest.json").unwrap(), + Some(b"{}".to_vec()) + ); + assert_eq!(entry.read_document("missing").unwrap(), None); + + let mut file = entry.begin_file(3).await.unwrap(); + file.write_all(b"bytes").await.unwrap(); + assert_eq!(mode(&entry.dir().join("000003.json.gz.partial")), 0o600); + assert!(!entry.is_verified(3, "abc")); + entry.commit_file(3, file, "abc").await.unwrap(); + assert!(entry.is_verified(3, "abc")); + assert!( + !entry.is_verified(3, "abd"), + "the marker binds the digest, not the index" + ); + assert_eq!( + std::fs::read(entry.dir().join("000003.json.gz.verified")).unwrap(), + b"abc\n" + ); + assert_eq!(mode(&entry.file_path(3)), 0o600); + assert_eq!(mode(&entry.dir().join("000003.json.gz.verified")), 0o600); + assert!(!entry.dir().join("000003.json.gz.partial").exists()); + + // A second attempt removes the file and its marker before it writes. + let file = entry.begin_file(3).await.unwrap(); + assert!(!entry.is_verified(3, "abc")); + assert!(!entry.file_path(3).exists()); + drop(file); + // Reopening an existing entry of this host's record keeps it; one + // whose record was another host's is emptied. + let again = Entry::open(spool.path(), "tid-1", Some("host-a"), "host-a").unwrap(); + assert!(again.dir().join("000003.json.gz.partial").exists()); + let moved = Entry::open(spool.path(), "tid-1", Some("host-b"), "host-a").unwrap(); + assert!(moved.dir().is_dir()); + assert_eq!(mode(moved.dir()), 0o700); + assert!(!moved.dir().join("000003.json.gz.partial").exists()); + assert!(!moved.dir().join("extenddb-manifest.json").exists()); + let unknown = Entry::open(spool.path(), "tid-1", None, "host-a"); + assert!( + unknown.is_ok(), + "a record with no spool_host has no entry to keep" + ); + again.remove().await.unwrap(); + assert!(!again.dir().exists()); + again.remove().await.unwrap(); + } + + /// A crash at the rename leaves a `.partial` and no marker: the file is + /// never taken for verified, so the marker is written only once the + /// rename has landed. + #[tokio::test] + async fn the_marker_is_absent_until_the_file_is_renamed() { + let spool = Scratch::new(); + let entry = Entry::open(spool.path(), "tid-2", Some("host-a"), "host-a").unwrap(); + let mut file = entry.begin_file(0).await.unwrap(); + file.write_all(b"bytes").await.unwrap(); + entry + .fail_rename + .store(true, std::sync::atomic::Ordering::Release); + let err = entry.commit_file(0, file, "abc").await.unwrap_err(); + assert!(err.to_string().contains("at the rename"), "{err}"); + assert!(entry.dir().join("000000.json.gz.partial").exists()); + assert!(!entry.file_path(0).exists()); + assert!(!entry.dir().join("000000.json.gz.verified").exists()); + assert!(!entry.is_verified(0, "abc")); + } + + /// The sweep removes the entries of restores not RUNNING on this host + /// and leaves a live entry and every other name; an entry listed and + /// then removed by its own job before the sweep reached it is not an + /// error and not counted. + #[test] + fn the_sweep_removes_only_dead_entries() { + let spool = Scratch::new(); + for name in [ + "restore-live", + "restore-dead-1", + "restore-gone", + "other-dir", + ] { + std::fs::create_dir(spool.path().join(name)).unwrap(); + } + std::fs::write(spool.path().join("restore-dead-1/000000.json.gz"), b"x").unwrap(); + std::fs::write(spool.path().join(".lock"), b"").unwrap(); + // A file, not a directory, under an entry-shaped name is left. + std::fs::write(spool.path().join("restore-file"), b"").unwrap(); + let running: HashSet = ["live".to_owned()].into_iter().collect(); + let mut listed = list_entries(spool.path()).unwrap(); + listed.sort(); + assert_eq!( + listed.iter().map(|(id, _)| id.as_str()).collect::>(), + ["dead-1", "gone", "live"], + "entries only, in whatever order the directory gives" + ); + // Between the listing and the sweep, the job of `gone` removed its + // entry. + std::fs::remove_dir(spool.path().join("restore-gone")).unwrap(); + let mut removed = sweep_entries(listed, &running); + removed.sort(); + assert_eq!(removed, ["dead-1"]); + let mut left: Vec = std::fs::read_dir(spool.path()) + .unwrap() + .map(|e| e.unwrap().file_name().into_string().unwrap()) + .collect(); + left.sort(); + assert_eq!(left, [".lock", "other-dir", "restore-file", "restore-live"]); + assert_eq!(entry_table_id("restore-"), None); + assert_eq!(entry_table_id("restore-a/b"), None); + assert_eq!(entry_table_id("restore-ab_1-2"), Some("ab_1-2")); + } + + /// A document is written beside the one it replaces and renamed over + /// it: a write that stops at the rename leaves the earlier document + /// whole and a `.partial` beside it; a write that completes leaves the + /// new document and no `.partial`. + #[tokio::test] + async fn a_document_replaces_the_earlier_one_at_the_rename() { + let spool = Scratch::new(); + let entry = Entry::open(spool.path(), "tid-3", Some("host-a"), "host-a").unwrap(); + entry + .write_document("extenddb-manifest.json", b"first") + .await + .unwrap(); + entry + .fail_rename + .store(true, std::sync::atomic::Ordering::Release); + let err = entry + .write_document("extenddb-manifest.json", b"second") + .await + .unwrap_err(); + assert!(err.to_string().contains("at the rename"), "{err}"); + assert_eq!( + entry.read_document("extenddb-manifest.json").unwrap(), + Some(b"first".to_vec()), + "the earlier document stands until the rename" + ); + assert_eq!( + std::fs::read(entry.dir().join("extenddb-manifest.json.partial")).unwrap(), + b"second" + ); + assert_eq!( + mode(&entry.dir().join("extenddb-manifest.json.partial")), + 0o600 + ); + entry + .fail_rename + .store(false, std::sync::atomic::Ordering::Release); + entry + .write_document("extenddb-manifest.json", b"third") + .await + .unwrap(); + assert_eq!( + entry.read_document("extenddb-manifest.json").unwrap(), + Some(b"third".to_vec()) + ); + assert!(!entry.dir().join("extenddb-manifest.json.partial").exists()); + } +} diff --git a/crates/engine/src/backup/runtime.rs b/crates/engine/src/backup/runtime.rs index e176749b6..930d69a9d 100644 --- a/crates/engine/src/backup/runtime.rs +++ b/crates/engine/src/backup/runtime.rs @@ -39,6 +39,7 @@ use std::time::Duration; use extenddb_backup_store::{BackupStore, ByteStream, ObjectMeta, StoreError}; use extenddb_core::error::DynamoDbError; +use extenddb_core::limits::LimitsConfig; use extenddb_core::metrics::MetricsCollector; use extenddb_storage::backup_catalog::{ JOB_HEARTBEAT_INTERVAL, JobBound, StoreIdentity, StoreKind, limit_exceeded_message, @@ -138,6 +139,10 @@ pub struct BackupRuntime { source_backend: String, /// The build's version, the manifest's `source_extenddb_version`. source_extenddb_version: String, + /// The instance's limits, which a restore the reconciler resumes + /// validates its items against (a request's restore reads them from + /// the operation context). + limits: Arc, } impl std::fmt::Debug for BackupRuntime { @@ -696,9 +701,24 @@ impl BackupRuntime { jobs: TaskTracker::new(), source_backend: "test".to_owned(), source_extenddb_version: env!("CARGO_PKG_VERSION").to_owned(), + limits: Arc::new(LimitsConfig::default()), } } + /// The same runtime with the instance's limits, which a resumed restore + /// validates its items against (the default is `LimitsConfig::default`). + #[must_use] + pub fn with_limits(mut self, limits: Arc) -> Self { + self.limits = limits; + self + } + + /// The instance's limits. + #[must_use] + pub fn limits(&self) -> &Arc { + &self.limits + } + /// The same runtime with `[backup] stale_job_after_secs` set, the window /// the jobs' maintenance-claim check reads with (the default is the /// configuration's default, 300 s). @@ -802,14 +822,15 @@ impl BackupRuntime { return 0; } tracing::info!( - "waiting up to {} s for {running} backup job(s) to end", + "waiting up to {} s for {running} backup or restore job(s) to end", bound.as_secs() ); if tokio::time::timeout(bound, self.jobs.wait()).await.is_err() { let left = self.jobs.len(); tracing::warn!( - "{left} backup job(s) did not end within {} s; their rows are CREATING or FAILED \ - and the reconciler resolves or sweeps them once their heartbeat is stale", + "{left} job(s) did not end within {} s; a backup job's row is CREATING or FAILED \ + and the reconciler resolves or sweeps it once its heartbeat is stale, a restore \ + job's record is RUNNING with its progress and the reconciler resumes it", bound.as_secs() ); return left; diff --git a/crates/engine/src/backup/tests.rs b/crates/engine/src/backup/tests.rs index 3888f1b9b..8f04118e0 100644 --- a/crates/engine/src/backup/tests.rs +++ b/crates/engine/src/backup/tests.rs @@ -22,17 +22,18 @@ use extenddb_core::types::{ }; use extenddb_storage::backup_catalog::{ AbandonedLegacyBackup, BackupKey, BackupRow, BackupStatus, DeleteStart, Heartbeat, JobClaim, - JobRow, JobStatus, RestoreJobRow, RestoreTargetKey, StaleClock, StatusFields, StoreIdentity, - StoreKind, TimeBounds, WriteOutcome, new_generation, + JobRow, JobStatus, RestoreJobRow, RestoreKind, RestoreRegistrationRow, RestoreTargetKey, + StaleClock, StatusFields, StoreIdentity, StoreKind, TimeBounds, WriteOutcome, new_generation, }; use extenddb_storage::backup_definition::{BACKUP_DEFINITION_VERSION, BackupTableDefinition}; use extenddb_storage::error::StorageError; use extenddb_storage::{ - BackupCatalog, BackupRestoreSource, BoxedFuture, JobPermit, LegacyBackupReader, MetadataEngine, - RestoreRequest, SnapshotEngine, TableEngine, TableSnapshot, + BackupCatalog, BackupRestoreSource, BoxedFuture, BulkLoadFile, DataEngine, Finished, JobPermit, + LegacyBackupReader, MetadataEngine, RestoreEngine, RestoreRequest, SnapshotEngine, TableEngine, + TableSnapshot, }; -use futures::TryStreamExt; use futures::stream::BoxStream; +use futures::{StreamExt, TryStreamExt}; use serde_json::{Value, json}; use super::*; @@ -166,6 +167,308 @@ struct Fake { takeover_lost_to_another: std::sync::atomic::AtomicBool, /// What `sweep_restore_registrations` answers. swept_registrations: Mutex>, + /// The store-backed restore records, by target table id: what + /// `claim_restore` writes and the restore's claims, `read_restore_job`, + /// `list_restore_jobs` and the reconciler read. + restore_jobs: Arc>>, + /// The restore targets' data, by table id: the key schema the duplicate + /// rule reads keys with, the items `bulk_load` applied, and the + /// statuses `finish_restore` flips. + restore_tables: Mutex>, + /// Every `claim_restore` call: the account and the input. + restore_claims: Mutex>, + /// When set, the next `claim_restore` fails with it instead of + /// registering (a name race the backend lost, a backup deleted under + /// the lock). + claim_restore_error: Mutex>, + /// Every `set_status` a restore claim received, in order. + restore_writes: Arc>>, + /// The table ids `claim_restore_dead` hands over (the backend's liveness + /// test modelled as a set). + restore_dead: Mutex>, + /// Every `claim_restore_dead` that handed a claim over: the table id and + /// the `spool_host` it was given. + restore_takeovers: Mutex)>>, + /// Every `bulk_load`: the file, and the items the stream yielded before + /// it ended or failed. + bulk_loads: Mutex>, + /// When set, every `bulk_load` waits for one permit before it reads its + /// stream, so a test can hold the job between files. + bulk_load_gate: Mutex>>, + /// When set, `bulk_load` takes one permit before each item it pulls + /// from its stream, so a test can hold the job at a known item of a + /// file; `bulk_load_items_seen` counts the items pulled so far. + bulk_load_item_gate: Mutex>>, + bulk_load_items_seen: std::sync::atomic::AtomicUsize, + /// When set, `count_restore_target` answers this instead of the table's + /// item count (a backend that kept one of two rows with one key). + count_override: Mutex>, + /// When set, `count_restore_target` waits for one permit before it + /// counts, so a test can hold the job after its last `bulk_load`. + count_gate: Mutex>>, + /// `count_restore_target` calls, counted before the gate. + count_calls: std::sync::atomic::AtomicUsize, + /// When set, `claim_restore` takes this long before it registers (a + /// registration held behind the catalog's lock or pool). + claim_restore_delay: Mutex>, + /// What `finish_restore` answers when its conditions hold: ACTIVE, or + /// DONE with the backfill pending (MongoDB). + finish_pending: std::sync::atomic::AtomicBool, + /// The progress rows a backend keeps apart from the record + /// (PostgreSQL's `restore_progress`), by table id. + progress_rows: Mutex>>, + /// The `running` lists `sweep_restore_progress` received. + progress_sweeps: Mutex>>, + /// Whether `as_vector_search` answers an engine: a target with vector + /// support. + vector_search: std::sync::atomic::AtomicBool, + /// Every `list_restore_jobs` call's filter. + restore_lists: Mutex>>, + /// When set, `list_restore_jobs` fails as a backend that has no restore + /// records yet (its `Unsupported` stub). + restore_list_error: std::sync::atomic::AtomicBool, + /// `bulk_load` calls begun, counted before the gate, so a test can tell + /// a job waiting at the gate. + bulk_loads_started: std::sync::atomic::AtomicUsize, + /// When set, run inside `sweep_restore_progress`, that is, after the + /// reconciler read the RUNNING records and before its spool sweep: the + /// moment a registration can land between the sweep's two reads. + sweep_hook: Mutex>, + /// What the next `finish_restore` calls do instead of answering, in + /// order: fail after the flip landed, or fail without flipping. + finish_errors: Mutex>, + /// Every `finish_restore` call's `size_bytes`. + finish_sizes: Mutex>, + /// When set, `read_restore_job` takes this long to answer (a catalog + /// read held behind the pool). + read_restore_job_delay: Mutex>, + /// When set, `claim_restore_dead` hands the claim over and then removes + /// the record (gone between the takeover and the reads that follow). + takeover_removes_record: std::sync::atomic::AtomicBool, + /// The maintenance claims `maintenance_claim` handed out, as the test's + /// handles on them; `None` in the error slot makes the next call fail + /// as a claim a live holder has. + maintenance_holders: Mutex>>, + maintenance_claim_held_by: Arc>>, + /// When set, `claim_restore` stamps the record's `heartbeat_at` this + /// many seconds in the past: a registration that stalled on a lock, as + /// PostgreSQL's transaction-start `now()` stamps it. + registration_heartbeat_age: Mutex>, +} + +/// A closure the `Fake` runs inside `sweep_restore_progress`. +type SweepHook = Box; + +/// What a `finish_restore` of the `Fake` does instead of answering. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum FinishError { + /// The flip landed (the record DONE, the table ACTIVE) and the call + /// then failed: PostgreSQL's progress delete after the commit. + AfterFlip, + /// The call failed before anything was written. + BeforeFlip, + /// The call failed before anything was written, and another claim took + /// the record over and wrote DONE under its own generation before the + /// re-read. + TakenAndDone, +} + +/// A restore target's data in the `Fake`. +#[derive(Debug, Clone)] +struct RestoreTable { + name: String, + key_attributes: Vec, + items: Vec, + active: bool, +} + +/// One `bulk_load` the `Fake` received. +#[derive(Debug, Clone, PartialEq)] +struct BulkLoadCall { + table_id: String, + file: BulkLoadFile, + /// The items the stream yielded. + items: Vec, + /// The text of the `Err` item that ended the stream, if one did. + stream_error: Option, +} + +/// A `Fake` claim on one restore record, by table id: the backends' +/// `heartbeat` (`status = RUNNING AND owner_generation = ?`), `set_status` +/// conditioned on the generation and `from`, and `read`. +struct FakeRestoreClaim { + records: Arc>>, + table_id: String, + generation: u64, + writes: Arc>>, + heartbeat_answer: Arc>, + heartbeats: Arc, + unknown_write: Arc>>, + status_write_error: Arc, +} + +impl JobClaim for FakeRestoreClaim { + fn generation(&self) -> u64 { + self.generation + } + + fn heartbeat(&self) -> BoxedFuture<'_, Result> { + self.heartbeats + .fetch_add(1, std::sync::atomic::Ordering::AcqRel); + let answer = *self.heartbeat_answer.lock().unwrap(); + let confirmed = match self.records.lock().unwrap().get_mut(&self.table_id) { + Some(record) + if answer == HeartbeatAnswer::Confirm + && record.owner_generation == self.generation + && record.status == JobStatus::Running => + { + record.heartbeat_at = seconds_since_epoch(); + true + } + _ => false, + }; + Box::pin(async move { + match answer { + HeartbeatAnswer::Confirm if confirmed => Ok(Heartbeat::Confirmed), + HeartbeatAnswer::AlwaysConfirm => Ok(Heartbeat::Confirmed), + HeartbeatAnswer::Confirm | HeartbeatAnswer::Lost => Ok(Heartbeat::Lost), + HeartbeatAnswer::Stall => futures::future::pending().await, + HeartbeatAnswer::Error => Err(StorageError::Internal( + "the claim connection failed".to_owned(), + )), + } + }) + } + + fn set_status<'a>( + &'a self, + from: JobStatus, + to: JobStatus, + fields: &'a StatusFields, + ) -> BoxedFuture<'a, Result> { + self.writes.lock().unwrap().push(StatusWrite { + arn: self.table_id.clone(), + generation: self.generation, + from, + to, + fields: fields.clone(), + }); + if self + .status_write_error + .load(std::sync::atomic::Ordering::Acquire) + { + return Box::pin(async move { + Err(StorageError::Internal( + "claim session: connection reset".to_owned(), + )) + }); + } + let unknown = self.unknown_write.lock().unwrap().take(); + let apply = unknown != Some(UnknownWrite::Lost); + let mut outcome = WriteOutcome::NotMatched; + if let Some(record) = self.records.lock().unwrap().get_mut(&self.table_id) + && record.owner_generation == self.generation + && record.status == from + { + if unknown == Some(UnknownWrite::TakenAndWritten) { + record.owner_generation = new_generation(); + record.status = to; + } else if apply { + record.status = to; + if let Some(reason) = &fields.failure_reason { + record.failure_reason = Some(reason.clone()); + } + if fields.completed { + record.completed_at = Some(seconds_since_epoch()); + } + outcome = WriteOutcome::Matched; + } + } + if unknown.is_some() { + outcome = WriteOutcome::Unknown; + } + Box::pin(async move { Ok(outcome) }) + } + + fn read(&self) -> BoxedFuture<'_, Result> { + let found = self + .records + .lock() + .unwrap() + .get(&self.table_id) + .map(|record| JobRow { + status: record.status, + owner_generation: record.owner_generation, + heartbeat_at: record.heartbeat_at, + failure_reason: record.failure_reason.clone(), + }); + Box::pin(async move { + found.ok_or_else(|| StorageError::Internal("the record is gone".to_owned())) + }) + } +} + +impl Fake { + fn restore_claim_on(&self, table_id: &str, generation: u64) -> Box { + Box::new(FakeRestoreClaim { + records: Arc::clone(&self.restore_jobs), + table_id: table_id.to_owned(), + generation, + writes: Arc::clone(&self.restore_writes), + heartbeat_answer: Arc::clone(&self.heartbeat_answer), + heartbeats: Arc::clone(&self.heartbeats), + unknown_write: Arc::clone(&self.unknown_write), + status_write_error: Arc::clone(&self.status_write_error), + }) + } + + /// The key of `item` under `attributes`, for the duplicate rule. + fn item_key(attributes: &[String], item: &Item) -> String { + attributes + .iter() + .map(|a| serde_json::to_string(&item.get(a)).unwrap_or_default()) + .collect::>() + .join("|") + } + + /// The restore record of the target named `table`, if any. + fn restore_job(&self, table: &str) -> Option { + self.restore_jobs + .lock() + .unwrap() + .values() + .find(|r| r.target.table_name == table) + .cloned() + } + + /// The items a restore target holds, by table name. + fn restored_items(&self, table: &str) -> Vec { + self.restore_tables + .lock() + .unwrap() + .values() + .find(|t| t.name == table) + .map(|t| t.items.clone()) + .unwrap_or_default() + } + + /// The `bulk_load` calls received, as `(file_index, start_at, reapplying, + /// item count)`. + fn bulk_load_files(&self) -> Vec<(u32, u64, bool, usize)> { + self.bulk_loads + .lock() + .unwrap() + .iter() + .map(|c| { + ( + c.file.file_index, + c.file.start_at, + c.file.reapplying, + c.items.len(), + ) + }) + .collect() + } } /// A table snapshot the `Fake` serves. @@ -215,6 +518,10 @@ enum UnknownWrite { Applied, /// The write was lost with the connection. Lost, + /// The write was lost with the connection, and another claim took the + /// record over and wrote the same status under its own generation + /// before the re-read. + TakenAndWritten, } /// A `Fake` claim on one row. @@ -384,13 +691,16 @@ impl JobClaim for FakeClaim { } impl Fake { - fn with_source(source: BackupRestoreSource) -> Self { + /// A `Fake` serving `source` through the legacy reader, in the `Arc` the + /// restore request takes (its manifest branch spawns a task that owns a + /// clone). + fn with_source(source: BackupRestoreSource) -> Arc { let fake = Self::default(); fake.sources .lock() .unwrap() .insert(source.backup_arn.clone(), source); - fake + Arc::new(fake) } fn restores(&self) -> Vec { self.restores.lock().unwrap().clone() @@ -442,6 +752,27 @@ struct MemStore { /// error; every other call answers (the identity file's read among /// them, so the store stays healthy). fail_gets_under: Mutex>, + /// When set, the next `n` `get`s of a data file (a key under a `data/` + /// component) fail at the call with the error kind given; every other + /// call answers. + fail_gets: Mutex>, + /// When set, the next `get` of a data file yields this many bytes and + /// then fails as a transport error mid-body (a `GetObject` body that + /// stopped arriving). + fail_get_after_bytes: Mutex>, + /// When set, every `get` of a data file fails at the call as a transport + /// error while the clock reads before this instant (an outage of a + /// known length under a paused clock). + fail_data_gets_until: Mutex>, + /// When set, every `get` of a key ending in this text fails at the call + /// as a transport error (one object the store never serves). + fail_gets_of: Mutex>, + /// When set, the body of the `get` with this ordinal (1-based, over + /// every `get` the store saw) panics after its first chunk: a reader on + /// the blocking pool that dies mid-file. + panic_body_of_get: std::sync::atomic::AtomicUsize, + /// Every `get`'s key, in order, failed ones included. + gets: Mutex>, /// Called with the key of every `put` that stored its object, so a test /// can act at a point of the job (after the sidecar's put, before the /// claim re-check). @@ -468,6 +799,15 @@ struct MemStore { /// What `MemStore::on_put` runs after a stored `put`, with the object's key. type OnPut = Box; +/// How `MemStore::fail_gets_with` fails a data file's `get`. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum GetFailure { + /// The SDK's transport shape, which the restore retries. + Transport, + /// A denial, which it does not. + PermissionDenied, +} + /// A transport error with the SDK's shape: a top link naming only the class /// and the cause one link below it. #[derive(Debug)] @@ -530,6 +870,21 @@ impl MemStore { .unwrap() .insert(key.to_owned(), body.to_vec()); } + /// The next `times` `get`s of a data file fail with `failure`. + fn fail_gets_with(&self, failure: GetFailure, times: usize) { + *self.fail_gets.lock().unwrap() = Some((failure, times)); + } + /// The next `get` of a data file stops after `n` bytes with a transport + /// error. + fn fail_get_after_bytes(&self, n: usize) { + *self.fail_get_after_bytes.lock().unwrap() = Some(n); + } + fn gets(&self) -> Vec { + self.gets.lock().unwrap().clone() + } + fn is_data_file(key: &str) -> bool { + key.contains("/data/") + } fn keys(&self) -> Vec { self.objects.lock().unwrap().keys().cloned().collect() } @@ -624,6 +979,7 @@ impl extenddb_backup_store::BackupStore for MemStore { ) -> BoxedFuture<'a, Result> { Box::pin(async move { + self.gets.lock().unwrap().push(key.to_owned()); self.stall().await; self.check()?; if self @@ -637,6 +993,44 @@ impl extenddb_backup_store::BackupStore for MemStore { source: Box::new(DispatchFailure(ConnectError("get refused".to_owned()))), }); } + let ordinal = self.gets.lock().unwrap().len(); + if Self::is_data_file(key) + && (self + .fail_data_gets_until + .lock() + .unwrap() + .is_some_and(|until| tokio::time::Instant::now() < until) + || self + .fail_gets_of + .lock() + .unwrap() + .as_deref() + .is_some_and(|suffix| key.ends_with(suffix))) + { + return Err(extenddb_backup_store::StoreError::Transport { + source: Box::new(DispatchFailure(ConnectError( + "connection reset by peer".to_owned(), + ))), + }); + } + if Self::is_data_file(key) { + let mut fail = self.fail_gets.lock().unwrap(); + if let Some((failure, times)) = *fail { + *fail = (times > 1).then_some((failure, times - 1)); + return Err(match failure { + GetFailure::Transport => extenddb_backup_store::StoreError::Transport { + source: Box::new(DispatchFailure(ConnectError( + "connection reset by peer".to_owned(), + ))), + }, + GetFailure::PermissionDenied => { + extenddb_backup_store::StoreError::PermissionDenied(format!( + "GetObject {key}" + )) + } + }); + } + } let bytes = self .objects .lock() @@ -644,6 +1038,33 @@ impl extenddb_backup_store::BackupStore for MemStore { .get(key) .cloned() .ok_or(extenddb_backup_store::StoreError::NotFound)?; + if Self::is_data_file(key) + && let Some(n) = self.fail_get_after_bytes.lock().unwrap().take() + { + let head = bytes[..n.min(bytes.len())].to_vec(); + let stream = futures::stream::iter(vec![ + Ok(bytes::Bytes::from(head)), + Err(extenddb_backup_store::StoreError::Transport { + source: Box::new(DispatchFailure(ConnectError( + "the body stopped arriving".to_owned(), + ))), + }), + ]); + return Ok(Box::pin(stream) as extenddb_backup_store::ByteStream); + } + if ordinal + == self + .panic_body_of_get + .load(std::sync::atomic::Ordering::Acquire) + { + let head = bytes[..bytes.len().min(16)].to_vec(); + let stream = futures::stream::iter(vec![Ok(bytes::Bytes::from(head))]).chain( + futures::stream::once(async { + panic!("the body's reader died (a test fixture)") + }), + ); + return Ok(Box::pin(stream) as extenddb_backup_store::ByteStream); + } Ok(extenddb_backup_store::byte_stream_from(bytes)) }) } @@ -796,6 +1217,115 @@ fn row(arn: &str, status: BackupStatus) -> BackupRow { } } +impl Fake { + /// `claim_restore`'s registration once any delay has passed. + fn register_restore( + &self, + account_id: &str, + input: CreateTableInput, + registration: &RestoreRegistrationRow, + ) -> Result<(TableDescription, Box), StorageError> { + let arn = ®istration.backup.backup_arn; + let available = self + .rows + .lock() + .unwrap() + .get(arn) + .is_some_and(|r| r.status == BackupStatus::Available); + if !available { + return Err(extenddb_storage::backup_catalog::backup_not_found(arn)); + } + let in_use = self.in_use.lock().unwrap().get(arn).cloned().or_else(|| { + let jobs = self.restore_jobs.lock().unwrap(); + self.restore_tables + .lock() + .unwrap() + .iter() + .filter(|(_, t)| !t.active) + .find(|(id, _)| jobs.get(*id).is_some_and(|j| j.source_backup_arn == *arn)) + .map(|(_, t)| t.name.clone()) + }); + if let Some(table) = in_use { + return Err(extenddb_storage::backup_catalog::backup_in_use(&table, arn)); + } + let mut tables = self.tables.lock().unwrap(); + if let Some(taken) = tables.get(&input.table_name) { + let err = match taken + .restore_summary + .as_ref() + .filter(|_| taken.table_status == TableStatus::Creating) + .and_then(|s| s.source_backup_arn.clone()) + { + Some(backup_arn) => StorageError::TableBeingRestored { + table_name: input.table_name.clone(), + backup_arn, + }, + None => StorageError::TableAlreadyExists(input.table_name.clone()), + }; + return Err(err); + } + let table_id = format!("tid-{}", input.table_name); + let generation = new_generation(); + let mut desc = described(&RestoreRequest { + backup_arn: arn.clone(), + source_table_arn: registration.source_table_arn.clone(), + restore_date_time: registration.restore_date_time, + input: input.clone(), + }); + desc.table_id = table_id.clone(); + desc.restore_summary = Some(RestoreSummary { + source_backup_arn: Some(arn.clone()), + source_table_arn: Some(registration.source_table_arn.clone()), + restore_date_time: registration.restore_date_time, + restore_in_progress: true, + }); + tables.insert(input.table_name.clone(), desc.clone()); + drop(tables); + let target = RestoreTargetKey { + account_id: account_id.to_owned(), + table_name: input.table_name.clone(), + table_id: table_id.clone(), + }; + self.restore_jobs.lock().unwrap().insert( + table_id.clone(), + RestoreJobRow { + target, + source_backup_arn: arn.clone(), + kind: RestoreKind::Manifest, + status: JobStatus::Running, + owner_generation: generation, + heartbeat_at: seconds_since_epoch() + - self + .registration_heartbeat_age + .lock() + .unwrap() + .unwrap_or(0.0), + store: Some(registration.store.clone()), + spool_host: registration.spool_host.clone(), + file_index: None, + items_applied: None, + failure_reason: None, + completed_at: None, + }, + ); + self.restore_tables.lock().unwrap().insert( + table_id.clone(), + RestoreTable { + name: input.table_name.clone(), + key_attributes: input + .key_schema + .iter() + .map(|k| k.attribute_name.clone()) + .collect(), + items: Vec::new(), + active: false, + }, + ); + let claim = self.restore_claim_on(&table_id, generation); + Ok((desc, claim)) + } +} + impl BackupCatalog for Fake { fn catalog_id(&self) -> BoxedFuture<'_, Result> { if *self.catalog_id_error.lock().unwrap() { @@ -834,6 +1364,117 @@ impl BackupCatalog for Fake { Box::pin(async move { Ok(claim) }) } + /// The backends' registration: under the backup row's lock the row must + /// be AVAILABLE (else `Backup not found`) and not in use (else + /// `BackupInUse`), the name free (else `TableAlreadyExists`, or + /// `TableBeingRestored` for a CREATING restore target); then the target + /// CREATING with its `RestoreSummary` and the record RUNNING under a + /// fresh generation, `heartbeat_at` now, in one step. + fn claim_restore<'a>( + &'a self, + account_id: &'a str, + input: CreateTableInput, + registration: &'a RestoreRegistrationRow, + ) -> BoxedFuture<'a, Result<(TableDescription, Box), StorageError>> { + self.restore_claims + .lock() + .unwrap() + .push((account_id.to_owned(), input.clone())); + if let Some(e) = self.claim_restore_error.lock().unwrap().take() { + return Box::pin(async move { Err(e) }); + } + let delay = *self.claim_restore_delay.lock().unwrap(); + Box::pin(async move { + if let Some(delay) = delay { + tokio::time::sleep(delay).await; + } + self.register_restore(account_id, input, registration) + }) + } + + /// The takeover: a RUNNING manifest record whose table id is in the + /// `restore_dead` set gets a fresh generation, the claimant's + /// `spool_host` and a fresh `heartbeat_at` (so a second taker finds it + /// live at once); anything else is `None`. + fn claim_restore_dead<'a>( + &'a self, + key: &'a RestoreTargetKey, + _stale_after: Duration, + spool_host: Option<&'a str>, + ) -> BoxedFuture<'a, Result>, StorageError>> { + let dead = self.restore_dead.lock().unwrap().contains(&key.table_id); + let mut records = self.restore_jobs.lock().unwrap(); + let claim = match records.get_mut(&key.table_id) { + Some(record) + if dead + && record.status == JobStatus::Running + && record.kind == RestoreKind::Manifest => + { + record.owner_generation = new_generation(); + record.spool_host = spool_host.map(str::to_owned); + record.heartbeat_at = seconds_since_epoch(); + let generation = record.owner_generation; + if self + .takeover_removes_record + .load(std::sync::atomic::Ordering::Acquire) + { + records.remove(&key.table_id); + } + drop(records); + self.restore_takeovers + .lock() + .unwrap() + .push((key.table_id.clone(), spool_host.map(str::to_owned))); + Some(self.restore_claim_on(&key.table_id, generation)) + } + _ => None, + }; + Box::pin(async move { Ok(claim) }) + } + + fn read_restore_job<'a>( + &'a self, + key: &'a RestoreTargetKey, + ) -> BoxedFuture<'a, Result, StorageError>> { + let delay = *self.read_restore_job_delay.lock().unwrap(); + Box::pin(async move { + if let Some(delay) = delay { + tokio::time::sleep(delay).await; + } + Ok(self + .restore_jobs + .lock() + .unwrap() + .get(&key.table_id) + .cloned()) + }) + } + + fn list_restore_jobs<'a>( + &'a self, + status: Option, + ) -> BoxedFuture<'a, Result, StorageError>> { + self.restore_lists.lock().unwrap().push(status); + if self + .restore_list_error + .load(std::sync::atomic::Ordering::Acquire) + { + return Box::pin(async { + Err(StorageError::Unsupported("not implemented".to_owned())) + }); + } + let mut rows: Vec = self + .restore_jobs + .lock() + .unwrap() + .values() + .filter(|r| status.is_none_or(|s| r.status == s)) + .cloned() + .collect(); + rows.sort_by(|a, b| a.target.table_id.cmp(&b.target.table_id)); + Box::pin(async move { Ok(rows) }) + } + /// The takeover: a CREATING row whose ARN is in the `dead` set gets a /// fresh generation, `heartbeat_at` untouched; anything else is `None`. fn claim_dead_backup<'a>( @@ -1071,31 +1712,86 @@ impl BackupCatalog for Fake { Box::pin(futures::stream::iter(rows.into_iter().map(Ok))) } + /// The preset answer, or the name of a CREATING restore target the + /// `Fake` registered from the backup. fn backup_in_use<'a>( &'a self, key: &'a BackupKey, ) -> BoxedFuture<'a, Result, StorageError>> { - let found = self.in_use.lock().unwrap().get(&key.backup_arn).cloned(); + let found = self + .in_use + .lock() + .unwrap() + .get(&key.backup_arn) + .cloned() + .or_else(|| { + let jobs = self.restore_jobs.lock().unwrap(); + self.restore_tables + .lock() + .unwrap() + .iter() + .filter(|(_, t)| !t.active) + .find(|(id, _)| { + jobs.get(*id) + .is_some_and(|j| j.source_backup_arn == key.backup_arn) + }) + .map(|(_, t)| t.name.clone()) + }); Box::pin(async move { Ok(found) }) } + /// A target the `Fake` registered is removed with its record and its + /// progress rows when it is still CREATING; any other key answers the + /// preset `target_still_creating`. fn remove_restore_target<'a>( &'a self, key: &'a RestoreTargetKey, ) -> BoxedFuture<'a, Result> { self.removed_targets.lock().unwrap().push(key.clone()); - let answer = *self.target_still_creating.lock().unwrap(); + let mut targets = self.restore_tables.lock().unwrap(); + let answer = match targets.get(&key.table_id) { + Some(target) if !target.active && target.name == key.table_name => { + let target = targets.remove(&key.table_id).expect("present"); + self.tables.lock().unwrap().remove(&target.name); + self.restore_jobs.lock().unwrap().remove(&key.table_id); + self.progress_rows.lock().unwrap().remove(&key.table_id); + true + } + Some(_) => false, + None => *self.target_still_creating.lock().unwrap(), + }; Box::pin(async move { Ok(answer) }) } + /// The preset `dead_jobs`, plus the `Fake`'s own records that the + /// backends' rule would list: RUNNING ones in the `restore_dead` set, and + /// FAILED ones whose target is still CREATING. fn list_dead_restore_jobs( &self, _stale_after: Duration, ) -> BoxedFuture<'_, Result, StorageError>> { - let jobs = self.dead_jobs.lock().unwrap().clone(); + let mut jobs = self.dead_jobs.lock().unwrap().clone(); + let dead = self.restore_dead.lock().unwrap(); + let targets = self.restore_tables.lock().unwrap(); + let mut own: Vec = self + .restore_jobs + .lock() + .unwrap() + .values() + .filter(|r| match r.status { + JobStatus::Running => dead.contains(&r.target.table_id), + JobStatus::Failed => targets.get(&r.target.table_id).is_some_and(|t| !t.active), + _ => false, + }) + .cloned() + .collect(); + own.sort_by(|a, b| a.target.table_id.cmp(&b.target.table_id)); + jobs.extend(own); Box::pin(async move { Ok(jobs) }) } + /// A RUNNING record of the `Fake`'s own is FAILED with the reason; any + /// other key answers the preset `record_still_running`. fn fail_restore_job<'a>( &'a self, key: &'a RestoreTargetKey, @@ -1105,7 +1801,16 @@ impl BackupCatalog for Fake { .lock() .unwrap() .push((key.clone(), reason.to_owned())); - let answer = *self.record_still_running.lock().unwrap(); + let answer = match self.restore_jobs.lock().unwrap().get_mut(&key.table_id) { + Some(record) if record.status == JobStatus::Running => { + record.status = JobStatus::Failed; + record.failure_reason = Some(reason.to_owned()); + record.completed_at = Some(seconds_since_epoch()); + true + } + Some(_) => false, + None => *self.record_still_running.lock().unwrap(), + }; Box::pin(async move { Ok(answer) }) } @@ -1127,12 +1832,32 @@ impl BackupCatalog for Fake { Box::pin(async move { Ok(ids) }) } + /// A claim on the `Fake`'s maintenance record: refused naming the holder + /// while `maintenance_claim_held_by` names one, else a `FakeHolder` the + /// test keeps a handle on. fn maintenance_claim<'a>( &'a self, - _holder: &'a str, + holder: &'a str, _stale_after: Duration, ) -> BoxedFuture<'a, Result, StorageError>> { - unused() + if let Some(live) = self.maintenance_claim_held_by.lock().unwrap().clone() { + return Box::pin(async move { + Err(StorageError::AlreadyExists(format!( + "the maintenance claim is held by {live:?}" + ))) + }); + } + let fake = Arc::new(FakeHolder::default()); + self.maintenance_holders + .lock() + .unwrap() + .push(Arc::clone(&fake)); + *self.maintenance_claim_held_by.lock().unwrap() = Some(holder.to_owned()); + let claim: Box = Box::new(ReleasingForward( + Arc::clone(&self.maintenance_claim_held_by), + fake, + )); + Box::pin(async move { Ok(claim) }) } fn check_maintenance_claim( @@ -1158,9 +1883,383 @@ impl BackupCatalog for Fake { } } -impl SnapshotEngine for Fake { - /// The fixture for the table, or `TableNotFound`; the description and - /// the items are the fixture's, in its order, through the gate when it +impl RestoreEngine for Fake { + /// The backends' load: under the claim's generation and RUNNING, the + /// stream read whole; an `Err` item fails the file with nothing applied; + /// a key the target holds fails the file unless the file is + /// `reapplying`, where the duplicate is skipped; the progress recorded + /// on the record as SQLite records it (`file_index`, the running total). + fn bulk_load<'a>( + &'a self, + target: &'a RestoreTargetKey, + claim: &'a dyn JobClaim, + file: BulkLoadFile, + items: BoxStream<'a, Result>, + ) -> BoxedFuture<'a, Result> { + Box::pin(async move { + self.bulk_loads_started + .fetch_add(1, std::sync::atomic::Ordering::AcqRel); + let gate = self.bulk_load_gate.lock().unwrap().clone(); + if let Some(gate) = gate { + gate.acquire() + .await + .expect("the gate is never closed") + .forget(); + } + let mut call = BulkLoadCall { + table_id: target.table_id.clone(), + file, + items: Vec::new(), + stream_error: None, + }; + let mut items = items; + let mut failed = None; + let item_gate = self.bulk_load_item_gate.lock().unwrap().clone(); + loop { + if let Some(gate) = &item_gate { + gate.acquire() + .await + .expect("the gate is never closed") + .forget(); + } + let Some(next) = items.next().await else { + break; + }; + match next { + Ok(item) => { + self.bulk_load_items_seen + .fetch_add(1, std::sync::atomic::Ordering::AcqRel); + call.items.push(item); + } + Err(e) => { + call.stream_error = Some(e.to_string()); + failed = Some(e); + break; + } + } + } + let streamed = std::mem::take(&mut call.items); + call.items.clone_from(&streamed); + self.bulk_loads.lock().unwrap().push(call); + if let Some(e) = failed { + return Err(e); + } + let owned = self + .restore_jobs + .lock() + .unwrap() + .get(&target.table_id) + .is_some_and(|r| { + r.owner_generation == claim.generation() && r.status == JobStatus::Running + }); + if !owned { + return Err(StorageError::Internal( + "the restore record is not RUNNING under this claim".to_owned(), + )); + } + let mut targets = self.restore_tables.lock().unwrap(); + let Some(table) = targets.get_mut(&target.table_id) else { + return Err(StorageError::TableNotFound(target.table_name.clone())); + }; + let mut keys: std::collections::HashSet = table + .items + .iter() + .map(|i| Fake::item_key(&table.key_attributes, i)) + .collect(); + let mut applied = 0; + for item in streamed { + let key = Fake::item_key(&table.key_attributes, &item); + if !keys.insert(key.clone()) { + if file.reapplying { + continue; + } + return Err(StorageError::Validation(format!( + "holds a primary key the target already has (the backup holds a key \ + twice): {key}" + ))); + } + table.items.push(item); + applied += 1; + } + drop(targets); + if let Some(record) = self.restore_jobs.lock().unwrap().get_mut(&target.table_id) { + record.file_index = Some(file.file_index); + record.items_applied = Some(file.start_at + applied); + } + Ok(applied) + }) + } + + /// DONE and ACTIVE in one step under the claim's generation, RUNNING and + /// a CREATING target; `Pending` when the test says the backend's worker + /// flips the table. A queued `FinishError` makes the call fail, after + /// the flip landed or before anything was written. + fn finish_restore<'a>( + &'a self, + target: &'a RestoreTargetKey, + claim: &'a dyn JobClaim, + item_count: u64, + size_bytes: u64, + ) -> BoxedFuture<'a, Result> { + self.finish_sizes.lock().unwrap().push(size_bytes); + let error = self.finish_errors.lock().unwrap().pop_front(); + if error == Some(FinishError::TakenAndDone) + && let Some(record) = self.restore_jobs.lock().unwrap().get_mut(&target.table_id) + && record.owner_generation == claim.generation() + { + record.owner_generation = new_generation(); + record.status = JobStatus::Done; + } + if matches!( + error, + Some(FinishError::BeforeFlip | FinishError::TakenAndDone) + ) { + return Box::pin(async { + Err(StorageError::Internal( + "the claim connection failed before the DONE write".to_owned(), + )) + }); + } + let mut records = self.restore_jobs.lock().unwrap(); + let mut targets = self.restore_tables.lock().unwrap(); + let finished = match ( + records.get_mut(&target.table_id), + targets.get_mut(&target.table_id), + ) { + (Some(record), Some(table)) + if record.owner_generation == claim.generation() + && record.status == JobStatus::Running + && !table.active => + { + record.status = JobStatus::Done; + record.completed_at = Some(seconds_since_epoch()); + let pending = self + .finish_pending + .load(std::sync::atomic::Ordering::Acquire); + if !pending { + table.active = true; + if let Some(desc) = self.tables.lock().unwrap().get_mut(&table.name) { + desc.table_status = TableStatus::Active; + desc.item_count = i64::try_from(item_count).unwrap_or(i64::MAX); + for gsi in desc.global_secondary_indexes.iter_mut().flatten() { + gsi.index_status = "ACTIVE".to_owned(); + } + } + } + self.progress_rows.lock().unwrap().remove(&target.table_id); + if pending { + Finished::Pending + } else { + Finished::Active + } + } + (Some(record), _) => Finished::NotMatched(format!( + "the record is {} under generation {} (the claim's is {})", + record.status, + record.owner_generation, + claim.generation() + )), + (None, _) => Finished::NotMatched("the record is gone".to_owned()), + }; + Box::pin(async move { + if error == Some(FinishError::AfterFlip) { + return Err(StorageError::Internal( + "the data pool failed after the flip's commit".to_owned(), + )); + } + Ok(finished) + }) + } + + fn count_restore_target<'a>( + &'a self, + key: &'a RestoreTargetKey, + ) -> BoxedFuture<'a, Result> { + self.count_calls + .fetch_add(1, std::sync::atomic::Ordering::AcqRel); + let gate = self.count_gate.lock().unwrap().clone(); + Box::pin(async move { + if let Some(gate) = gate { + gate.acquire() + .await + .expect("the gate is never closed") + .forget(); + } + match *self.count_override.lock().unwrap() { + Some(count) => Ok(count), + None => self + .restore_tables + .lock() + .unwrap() + .get(&key.table_id) + .map(|t| t.items.len() as u64) + .ok_or_else(|| StorageError::TableNotFound(key.table_name.clone())), + } + }) + } + + fn read_restore_progress<'a>( + &'a self, + key: &'a RestoreTargetKey, + ) -> BoxedFuture<'a, Result, StorageError>> { + let rows = self + .progress_rows + .lock() + .unwrap() + .get(&key.table_id) + .cloned() + .unwrap_or_default(); + Box::pin(async move { Ok(rows) }) + } + + fn sweep_restore_progress<'a>( + &'a self, + running: &'a [String], + ) -> BoxedFuture<'a, Result> { + self.progress_sweeps.lock().unwrap().push(running.to_vec()); + if let Some(hook) = self.sweep_hook.lock().unwrap().take() { + hook(self); + } + let mut rows = self.progress_rows.lock().unwrap(); + let before = rows.len(); + rows.retain(|id, _| running.contains(id)); + let removed = (before - rows.len()) as u64; + Box::pin(async move { Ok(removed) }) + } +} + +/// A vector-search engine the `Fake` hands out when it stands for a target +/// with vector support; never searched. +struct CapableVectorSearch; + +impl extenddb_storage::VectorSearchEngine for CapableVectorSearch { + fn search_vectors( + &self, + _req: extenddb_storage::VectorSearch<'_>, + ) -> BoxedFuture<'_, extenddb_storage::VectorSearchResult> { + unused() + } +} + +static CAPABLE_VECTOR_SEARCH: CapableVectorSearch = CapableVectorSearch; + +/// The restore's vector gate reads `as_vector_search`; nothing else of the +/// data plane is used by these tests. +impl DataEngine for Fake { + fn put_item( + &self, + _: &TableKeyInfo, + _: Item, + _: bool, + _: Option<&extenddb_core::expression::Expr>, + _: &extenddb_core::expression::ExpressionMaps, + _: Option<&extenddb_storage::StreamCapture>, + ) -> BoxedFuture<'_, Result, StorageError>> { + unused() + } + + fn get_item( + &self, + _: &TableKeyInfo, + _: &Item, + ) -> BoxedFuture<'_, Result, StorageError>> { + unused() + } + + fn delete_item( + &self, + _: &TableKeyInfo, + _: &Item, + _: bool, + _: Option<&extenddb_core::expression::Expr>, + _: &extenddb_core::expression::ExpressionMaps, + _: Option<&extenddb_storage::StreamCapture>, + ) -> BoxedFuture<'_, Result, StorageError>> { + unused() + } + + fn update_item( + &self, + _: &TableKeyInfo, + _: &Item, + _: &[extenddb_core::expression::UpdateAction], + _: bool, + _: bool, + _: Option<&extenddb_core::expression::Expr>, + _: &extenddb_core::expression::ExpressionMaps, + _: Option<&extenddb_storage::StreamCapture>, + ) -> BoxedFuture<'_, extenddb_storage::ItemPairResult> { + unused() + } + + fn query( + &self, + _: &TableKeyInfo, + _: &extenddb_core::expression::KeyCondition, + _: &extenddb_core::expression::ExpressionMaps, + _: bool, + _: Option, + _: Option<&Item>, + _: Option<&str>, + ) -> BoxedFuture<'_, extenddb_storage::QueryResult> { + unused() + } + + fn scan( + &self, + _: &TableKeyInfo, + _: Option, + _: Option<&Item>, + _: Option, + _: Option, + _: Option<&str>, + ) -> BoxedFuture<'_, extenddb_storage::QueryResult> { + unused() + } + + fn scan_key_in_segment( + &self, + _: &TableKeyInfo, + _: &Item, + _: i64, + _: i64, + _: Option<&str>, + ) -> BoxedFuture<'_, Result> { + unused() + } + + fn as_vector_search(&self) -> Option<&dyn extenddb_storage::VectorSearchEngine> { + self.vector_search + .load(std::sync::atomic::Ordering::Acquire) + .then_some(&CAPABLE_VECTOR_SEARCH) + } + + fn transact_get_items( + &self, + _: &[extenddb_storage::TransactGetOp<'_>], + ) -> BoxedFuture<'_, Result>, StorageError>> { + unused() + } + + fn transact_write_items( + &self, + _: &[extenddb_storage::TransactWriteOp<'_>], + _: Option>, + ) -> BoxedFuture<'_, Result<(), StorageError>> { + unused() + } + + fn cleanup_expired_idempotency_tokens( + &self, + _: i64, + ) -> BoxedFuture<'_, Result> { + unused() + } +} + +impl SnapshotEngine for Fake { + /// The fixture for the table, or `TableNotFound`; the description and + /// the items are the fixture's, in its order, through the gate when it /// has one. fn open_table_snapshot( &self, @@ -1673,13 +2772,13 @@ fn restore_body(extra: Value) -> Value { body } -async fn restore(fake: &Fake, body: Value) -> Result { +async fn restore(fake: &Arc, body: Value) -> Result { restore_on(fake, &runtime(), &body).await } /// `restore` on `runtime` rather than the tests' default one. async fn restore_on( - fake: &Fake, + fake: &Arc, runtime: &BackupRuntime, body: &Value, ) -> Result { @@ -1957,7 +3056,7 @@ async fn a_backup_stored_under_another_region_resolves_as_written() { // DescribeBackup: canonical not found, as written found. let fake = with_stored(); - let desc = describe_backup(&fake, ACCOUNT, REGION, &spellings) + let desc = describe_backup(&*fake, ACCOUNT, REGION, &spellings) .await .unwrap(); assert_eq!(desc.backup_details.backup_status, "AVAILABLE"); @@ -1981,7 +3080,7 @@ async fn a_backup_stored_under_another_region_resolves_as_written() { // DeleteBackup: the full description, and the stored row is gone. let fake = with_stored(); - let deleted = delete_backup(&fake, &runtime(), ACCOUNT, REGION, &spellings) + let deleted = delete_backup(&*fake, &runtime(), ACCOUNT, REGION, &spellings) .await .unwrap(); assert_eq!(deleted.backup_details.backup_status, "DELETED"); @@ -2005,7 +3104,7 @@ async fn a_backup_stored_under_another_region_resolves_as_written() { .unwrap(); assert_eq!(fake.restores()[0].backup_arn, ours); assert_eq!(fake.source_reads(), 1); - let deleted = delete_backup(&fake, &runtime(), ACCOUNT, REGION, &spellings) + let deleted = delete_backup(&*fake, &runtime(), ACCOUNT, REGION, &spellings) .await .unwrap(); assert_eq!(deleted.backup_details.backup_status, "DELETED"); @@ -2014,7 +3113,7 @@ async fn a_backup_stored_under_another_region_resolves_as_written() { // Neither spelling: BackupNotFoundException from the backend, after both // were tried; the server's own spelling is tried once. - let fake = Fake::default(); + let fake = Arc::new(Fake::default()); for (body, reads) in [(stored.clone(), 2), (ours.clone(), 1)] { let before = fake.source_reads(); let err = restore(&fake, restore_body(json!({ "BackupArn": body }))) @@ -2026,13 +3125,13 @@ async fn a_backup_stored_under_another_region_resolves_as_written() { ); assert_eq!(fake.source_reads() - before, reads); assert!(matches!( - describe_backup(&fake, ACCOUNT, REGION, &own(&body)) + describe_backup(&*fake, ACCOUNT, REGION, &own(&body)) .await .unwrap_err(), DynamoDbError::BackupNotFoundException(_) )); assert!(matches!( - delete_backup(&fake, &runtime(), ACCOUNT, REGION, &own(&body)) + delete_backup(&*fake, &runtime(), ACCOUNT, REGION, &own(&body)) .await .unwrap_err(), DynamoDbError::BackupNotFoundException(_) @@ -2120,7 +3219,11 @@ async fn a_restore_request_dropped_while_the_backup_is_read_still_restores() { use std::task::{Context, Waker}; let gate = Arc::new(tokio::sync::Semaphore::new(0)); - let mut fake = Fake::with_source(source(Some(definition("PROVISIONED")))); + let mut fake = Fake::default(); + fake.sources + .lock() + .unwrap() + .insert(arn(ACCOUNT), source(Some(definition("PROVISIONED")))); fake.source_gate = Some(Arc::clone(&gate)); let fake = Arc::new(fake); @@ -2374,14 +3477,131 @@ fn lsi_entry() -> Value { } /// The target input of a restore of `src` with `body`'s overrides, or the -/// engine's refusal. +/// engine's refusal, through both paths: the legacy reader's, and the +/// manifest path over a store-backed backup whose `source_table` is `src`'s +/// definition (`manifest_from_source`). The two must agree (the same input, +/// or the same refusal), so one list of override cases binds both; the +/// manifest records the SSE *description*, not the specification a legacy +/// definition keeps, so that member is compared only when an override set +/// it. A source without a definition (pre-0.0.4) has no manifest form and +/// runs the legacy path alone. async fn restored_input( src: BackupRestoreSource, body: Value, ) -> Result { - let fake = Fake::with_source(src); - restore(&fake, restore_body(body)).await?; - Ok(fake.restores()[0].input.clone()) + let fake = Fake::with_source(src.clone()); + let legacy = restore(&fake, restore_body(body.clone())) + .await + .map(|_| fake.restores()[0].input.clone()); + if legacy.is_err() { + assert!( + fake.restores().is_empty(), + "the legacy path reached storage for a refused {body}" + ); + } + if src.definition.is_none() { + return legacy; + } + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + let item: Item = src + .key_schema + .iter() + .map(|k| { + ( + k.attribute_name.clone(), + extenddb_core::types::AttributeValue::S("a".to_owned()), + ) + }) + .collect(); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + manifest_from_source(&src), + &[vec![item]], + false, + ); + let runtime = job_runtime(&mem); + let manifest = restore_on(&fake, &runtime, &restore_body(body.clone())) + .await + .map(|_| fake.restore_claims.lock().unwrap()[0].1.clone()); + wait_for_jobs(&runtime).await; + // The manifest records the SSE description, from which the restore + // derives the enabled KMS specification, so the member is compared when + // the source had SSE enabled or an override set it; a legacy + // definition's disabled specification has no manifest form. + let source_sse_enabled = src + .definition + .as_ref() + .and_then(|d| d.sse_specification.as_ref()) + .and_then(|sse| sse.get("Enabled")) + .and_then(Value::as_bool) + == Some(true); + match (&legacy, &manifest) { + (Ok(a), Ok(b)) => { + let mut a = a.clone(); + let mut b = b.clone(); + if body.get("SSESpecificationOverride").is_none() && !source_sse_enabled { + a.sse_specification = None; + b.sse_specification = None; + } + assert_eq!(a, b, "the two paths built different inputs for {body}"); + } + (Err(a), Err(b)) => { + assert_eq!( + format!("{a:?}"), + format!("{b:?}"), + "the two paths refused differently for {body}" + ); + assert!( + fake.restore_claims.lock().unwrap().is_empty(), + "the manifest path registered a target it then refused for {body}" + ); + } + (a, b) => panic!("the two paths disagree for {body}: legacy {a:?}, manifest {b:?}"), + } + legacy +} + +/// A manifest's `source_table` for a legacy source with a definition: the +/// same keys, indexes, billing and throughput, so the manifest path builds +/// the input the legacy path builds. +fn manifest_from_source(src: &BackupRestoreSource) -> SourceTableSchema { + let def = src.definition.as_ref().expect("a definition"); + let sse_description = def + .sse_specification + .as_ref() + .filter(|s| s.get("Enabled").and_then(Value::as_bool) == Some(true)) + .map(|_| extenddb_core::types::SseDescription { + status: "ENABLED".to_owned(), + sse_type: Some(extenddb_core::types::SseType::KMS), + kms_master_key_arn: None, + }); + SourceTableSchema { + table_name: src.source_table_name.clone(), + table_id: src.source_table_id.clone(), + table_arn: src.source_table_arn.clone(), + table_creation_date_time: src.table_creation_date_time, + key_schema: src.key_schema.clone(), + attribute_definitions: src.attribute_definitions.clone(), + billing_mode: if def.billing_mode == "PAY_PER_REQUEST" { + BillingMode::PayPerRequest + } else { + BillingMode::Provisioned + }, + provisioned_throughput: def.provisioned_throughput.clone(), + on_demand_throughput: def.on_demand_throughput.clone(), + table_class: def.table_class.clone(), + sse_description, + global_secondary_indexes: def.global_secondary_indexes.clone(), + local_secondary_indexes: def.local_secondary_indexes.clone(), + vector_indexes: None, + stream_specification: None, + time_to_live: None, + item_count: src.item_count, + table_size_bytes: src.table_size_bytes, + } } /// The attribute names `input` defines. @@ -2757,10 +3977,12 @@ async fn index_overrides_the_service_refuses_are_refused() { ), ]; for (what, body, expect) in cases { - let fake = Fake::with_source(source(Some(definition("PROVISIONED")))); - let msg = validation_message(restore(&fake, restore_body(body)).await.unwrap_err()); + let msg = validation_message( + restored_input(source(Some(definition("PROVISIONED"))), body) + .await + .unwrap_err(), + ); assert!(msg.contains(expect), "{what}: {msg}"); - assert!(fake.restores().is_empty(), "{what}: reached storage"); } // A legacy backup restores no index, so any listed one is not in it; an @@ -2849,17 +4071,17 @@ async fn a_provisioned_restore_needs_each_gsis_throughput_overridden() { ), ]; for (what, def, body) in refused { - let fake = Fake::with_source(source(Some(def))); - let msg = validation_message(restore(&fake, restore_body(body)).await.unwrap_err()); + let msg = validation_message(restored_input(source(Some(def)), body).await.unwrap_err()); assert_eq!(msg, expected, "{what}"); - assert!(fake.restores().is_empty(), "{what}: reached storage"); } // The first GSI of two is the one named. - let fake = Fake::with_source(source(Some(two_gsis("PAY_PER_REQUEST")))); let msg = validation_message( - restore(&fake, restore_body(provisioned.clone())) - .await - .unwrap_err(), + restored_input( + source(Some(two_gsis("PAY_PER_REQUEST"))), + provisioned.clone(), + ) + .await + .unwrap_err(), ); assert_eq!(msg, expected); @@ -3059,19 +4281,32 @@ async fn table_class_is_not_restored() { assert_eq!(input.table_class, None); } +/// A source with SSE enabled restores with an enabled KMS specification on +/// both paths: the legacy definition carries the specification, the +/// manifest the description it was derived from. +#[tokio::test] +async fn an_enabled_sse_source_restores_sse_on_both_paths() { + let mut def = definition("PROVISIONED"); + def.sse_specification = Some(json!({ "Enabled": true, "SSEType": "KMS" })); + let input = restored_input(source(Some(def)), json!({})).await.unwrap(); + assert_eq!( + input.sse_specification, + Some(json!({ "Enabled": true, "SSEType": "KMS" })) + ); +} + #[tokio::test] async fn sse_override_replaces_the_recorded_specification() { let mut def = definition("PROVISIONED"); def.sse_specification = Some(json!({ "Enabled": false })); - let fake = Fake::with_source(source(Some(def))); let sse = json!({ "Enabled": true, "SSEType": "KMS", "KMSMasterKeyId": "alias/k" }); - restore( - &fake, - restore_body(json!({ "SSESpecificationOverride": sse })), + let input = restored_input( + source(Some(def)), + json!({ "SSESpecificationOverride": sse }), ) .await .unwrap(); - assert_eq!(fake.restores()[0].input.sse_specification, Some(sse)); + assert_eq!(input.sse_specification, Some(sse)); // A legacy backup takes the override too. let fake = Fake::with_source(source(None)); @@ -3096,14 +4331,10 @@ async fn sse_override_replaces_the_recorded_specification() { #[tokio::test] async fn billing_overrides_shape_the_restore_input() { // PROVISIONED -> PAY_PER_REQUEST drops table and GSI throughput. - let fake = Fake::with_source(source(Some(definition("PROVISIONED")))); - let out = restore( - &fake, - restore_body(json!({ "BillingModeOverride": "PAY_PER_REQUEST" })), - ) - .await - .unwrap(); - let input = &fake.restores()[0].input; + let body = json!({ "BillingModeOverride": "PAY_PER_REQUEST" }); + let input = restored_input(source(Some(definition("PROVISIONED"))), body.clone()) + .await + .unwrap(); assert_eq!(input.billing_mode, Some(BillingMode::PayPerRequest)); assert!(input.provisioned_throughput.is_none()); assert!( @@ -3111,6 +4342,9 @@ async fn billing_overrides_shape_the_restore_input() { .provisioned_throughput .is_none() ); + // The response zeros the throughput of an on-demand table and index. + let fake = Fake::with_source(source(Some(definition("PROVISIONED")))); + let out = restore(&fake, restore_body(body)).await.unwrap(); let td = &out["TableDescription"]; assert_eq!(td["ProvisionedThroughput"]["ReadCapacityUnits"], 0); assert_eq!( @@ -3119,16 +4353,14 @@ async fn billing_overrides_shape_the_restore_input() { ); // A throughput override on a PROVISIONED backup replaces the table's. - let fake = Fake::with_source(source(Some(definition("PROVISIONED")))); - restore( - &fake, - restore_body(json!({ + let input = restored_input( + source(Some(definition("PROVISIONED"))), + json!({ "ProvisionedThroughputOverride": { "ReadCapacityUnits": 11, "WriteCapacityUnits": 12 }, - })), + }), ) .await .unwrap(); - let input = &fake.restores()[0].input; assert_eq!(input.provisioned_throughput, Some(throughput(11, 12))); assert_eq!( input.global_secondary_indexes.as_ref().unwrap()[0].provisioned_throughput, @@ -3136,28 +4368,25 @@ async fn billing_overrides_shape_the_restore_input() { ); // A throughput override on an on-demand backup without a billing override. - let fake = Fake::with_source(source(Some(definition("PAY_PER_REQUEST")))); - let err = restore( - &fake, - restore_body(json!({ + let err = restored_input( + source(Some(definition("PAY_PER_REQUEST"))), + json!({ "ProvisionedThroughputOverride": { "ReadCapacityUnits": 1, "WriteCapacityUnits": 1 }, - })), + }), ) .await .unwrap_err(); assert!(validation_message(err).contains("can only be specified")); - assert!(fake.restores().is_empty()); } #[tokio::test] async fn provisioned_override_on_an_on_demand_gsi_has_the_services_text() { - let fake = Fake::with_source(source(Some(definition("PAY_PER_REQUEST")))); - let err = restore( - &fake, - restore_body(json!({ + let err = restored_input( + source(Some(definition("PAY_PER_REQUEST"))), + json!({ "BillingModeOverride": "PROVISIONED", "ProvisionedThroughputOverride": { "ReadCapacityUnits": 5, "WriteCapacityUnits": 5 }, - })), + }), ) .await .unwrap_err(); @@ -3166,12 +4395,11 @@ async fn provisioned_override_on_an_on_demand_gsi_has_the_services_text() { "Invalid Request: Must override ProvisionedThroughput if BillingMode is overridden to \ PROVISIONED for index gsi1" ); - assert!(fake.restores().is_empty()); } #[tokio::test] async fn override_shape_is_refused_before_any_storage_read() { - let fake = Fake::default(); + let fake = Arc::new(Fake::default()); let err = restore( &fake, restore_body(json!({ "BillingModeOverride": "PROVISIONED" })), @@ -3180,6 +4408,15 @@ async fn override_shape_is_refused_before_any_storage_read() { .unwrap_err(); assert!(validation_message(err).contains("ProvisionedThroughputOverride must")); assert_eq!(fake.source_reads(), 0); + // The manifest path refuses the shape the same way, before any store or + // row read. + let err = restored_input( + source(Some(definition("PROVISIONED"))), + json!({ "BillingModeOverride": "PROVISIONED" }), + ) + .await + .unwrap_err(); + assert!(validation_message(err).contains("ProvisionedThroughputOverride must")); } // --- refusals and validation -------------------------------------------------- @@ -3194,23 +4431,6 @@ async fn multi_part_key_backups_are_refused_in_the_engine() { assert!(fake.restores().is_empty()); } -#[tokio::test] -async fn vector_index_backups_are_refused_in_the_engine() { - let mut def = definition("PAY_PER_REQUEST"); - def.vector_index_names.push("v".to_owned()); - let fake = Fake::with_source(source(Some(def))); - let msg = validation_message(restore(&fake, restore_body(json!({}))).await.unwrap_err()); - assert!(msg.contains("vector index"), "{msg}"); - assert!(fake.restores().is_empty()); - - let mut legacy = source(None); - legacy.legacy_vector_index_count = 1; - let fake = Fake::with_source(legacy); - let msg = validation_message(restore(&fake, restore_body(json!({}))).await.unwrap_err()); - assert!(msg.contains("vector index"), "{msg}"); - assert!(fake.restores().is_empty()); -} - #[tokio::test] async fn a_stored_definition_create_table_would_refuse_is_refused() { type Corrupt = fn(&mut BackupRestoreSource); @@ -3351,7 +4571,7 @@ async fn a_table_create_table_accepted_restores() { #[tokio::test] async fn a_missing_backup_is_backup_not_found() { - let fake = Fake::default(); + let fake = Arc::new(Fake::default()); assert!(matches!( restore(&fake, restore_body(json!({}))).await.unwrap_err(), DynamoDbError::BackupNotFoundException(_) @@ -3718,7 +4938,7 @@ async fn name_lengths_are_counted_in_characters_and_table_names_take_the_limits( ); let fake = fake_with_summaries(vec![]); list_backups( - &fake, + &*fake, ACCOUNT, REGION, &raised, @@ -3737,7 +4957,7 @@ async fn name_lengths_are_counted_in_characters_and_table_names_take_the_limits( assert!(msg.contains("length less than or equal to 255"), "{msg}"); let msg = validation_message( list_backups( - &fake, + &*fake, ACCOUNT, REGION, &limits(), @@ -3866,7 +5086,7 @@ async fn delete_backup_answers_the_full_description_as_deleted() { let on_demand = def.billing_mode == "PAY_PER_REQUEST"; let fake = fake_with_backup("AVAILABLE", source(Some(def))); let described = describe(&fake).await; - let deleted = delete_backup(&fake, &runtime(), ACCOUNT, REGION, &own(&arn(ACCOUNT))) + let deleted = delete_backup(&*fake, &runtime(), ACCOUNT, REGION, &own(&arn(ACCOUNT))) .await .unwrap(); let deleted = serde_json::to_value(deleted).unwrap(); @@ -3893,7 +5113,7 @@ async fn delete_backup_answers_the_full_description_as_deleted() { assert_eq!(deleted, expected); // The backup is gone. assert!(matches!( - delete_backup(&fake, &runtime(), ACCOUNT, REGION, &own(&arn(ACCOUNT))) + delete_backup(&*fake, &runtime(), ACCOUNT, REGION, &own(&arn(ACCOUNT))) .await .unwrap_err(), DynamoDbError::BackupNotFoundException(_) @@ -3914,7 +5134,7 @@ async fn delete_backup_answers_the_full_description_as_deleted() { let what = format!("{err:?}"); let fake = fake_with_backup("AVAILABLE", source(Some(definition("PROVISIONED")))); *fake.source_error.lock().unwrap() = Some(err); - let deleted = delete_backup(&fake, &runtime(), ACCOUNT, REGION, &own(&arn(ACCOUNT))) + let deleted = delete_backup(&*fake, &runtime(), ACCOUNT, REGION, &own(&arn(ACCOUNT))) .await .unwrap_or_else(|e| panic!("{what}: {e:?}")); assert_eq!(deleted.backup_details.backup_status, "DELETED", "{what}"); @@ -3935,7 +5155,7 @@ async fn delete_backup_answers_the_full_description_as_deleted() { let fake = fake_with_backup("AVAILABLE", source(Some(definition("PROVISIONED")))); *fake.source_error.lock().unwrap() = Some(StorageError::Connection("refused".to_owned())); assert!(matches!( - delete_backup(&fake, &runtime(), ACCOUNT, REGION, &own(&arn(ACCOUNT))) + delete_backup(&*fake, &runtime(), ACCOUNT, REGION, &own(&arn(ACCOUNT))) .await .unwrap_err(), DynamoDbError::InternalServerError(_) @@ -4014,7 +5234,7 @@ fn backup_description(status: &str, billing: &str) -> BackupDescription { } } -fn fake_with_backup(status: &str, src: BackupRestoreSource) -> Fake { +fn fake_with_backup(status: &str, src: BackupRestoreSource) -> Arc { let billing = src.billing_mode.clone(); let fake = Fake::with_source(src); fake.descriptions @@ -4158,7 +5378,7 @@ async fn describe_backup_survives_a_definition_that_does_not_read() { let what = format!("{err:?}"); let fake = fake_with_backup("AVAILABLE", source(Some(definition("PROVISIONED")))); *fake.source_error.lock().unwrap() = Some(err); - let desc = describe_backup(&fake, ACCOUNT, REGION, &own(&arn_text)) + let desc = describe_backup(&*fake, ACCOUNT, REGION, &own(&arn_text)) .await .unwrap_or_else(|e| panic!("{what}: {e:?}")); assert_eq!(fake.source_reads(), 1, "{what}"); @@ -4184,7 +5404,7 @@ async fn describe_backup_survives_a_definition_that_does_not_read() { *fake.source_error.lock().unwrap() = Some(err); // The backup operations' mapping reports both as a 500 // (`storage_err_to_dynamo` in backup.rs), which SDKs retry. - match describe_backup(&fake, ACCOUNT, REGION, &own(&arn_text)).await { + match describe_backup(&*fake, ACCOUNT, REGION, &own(&arn_text)).await { Err(DynamoDbError::InternalServerError(_)) => {} other => panic!("{what} must reach the client as an error: {other:?}"), } @@ -4232,8 +5452,8 @@ fn summary(n: u64, millis: u64) -> BackupSummary { } } -fn fake_with_summaries(summaries: Vec) -> Fake { - let fake = Fake::default(); +fn fake_with_summaries(summaries: Vec) -> Arc { + let fake = Arc::new(Fake::default()); *fake.summaries.lock().unwrap() = summaries; fake } @@ -4677,8 +5897,8 @@ fn music_arn(account: &str, id: &str) -> String { format!("arn:aws:dynamodb:{REGION}:{account}:table/Music/backup/{id}") } -fn fake_with_row(status: BackupStatus) -> Fake { - let fake = Fake::default(); +fn fake_with_row(status: BackupStatus) -> Arc { + let fake = Arc::new(Fake::default()); fake.rows .lock() .unwrap() @@ -4700,7 +5920,7 @@ async fn describe_backup_reads_the_catalog_row_first() { arn(ACCOUNT), backup_description("AVAILABLE", "PAY_PER_REQUEST"), ); - let desc = describe_backup(&fake, ACCOUNT, REGION, &own(&arn(ACCOUNT))) + let desc = describe_backup(&*fake, ACCOUNT, REGION, &own(&arn(ACCOUNT))) .await .unwrap(); assert_eq!(desc.backup_details.backup_name, "music-nightly"); @@ -4732,13 +5952,13 @@ async fn describe_backup_reads_the_catalog_row_first() { #[tokio::test] async fn describe_backup_maps_the_row_status() { let fake = fake_with_row(BackupStatus::Creating); - let desc = describe_backup(&fake, ACCOUNT, REGION, &own(&arn(ACCOUNT))) + let desc = describe_backup(&*fake, ACCOUNT, REGION, &own(&arn(ACCOUNT))) .await .unwrap(); assert_eq!(desc.backup_details.backup_status, "CREATING"); let fake = fake_with_row(BackupStatus::Deleting); - let desc = describe_backup(&fake, ACCOUNT, REGION, &own(&arn(ACCOUNT))) + let desc = describe_backup(&*fake, ACCOUNT, REGION, &own(&arn(ACCOUNT))) .await .unwrap(); assert_eq!(desc.backup_details.backup_status, "DELETED"); @@ -4749,7 +5969,7 @@ async fn describe_backup_maps_the_row_status() { .lock() .unwrap() .insert(arn(ACCOUNT), backup_description("AVAILABLE", "PROVISIONED")); - let err = describe_backup(&fake, ACCOUNT, REGION, &own(&arn(ACCOUNT))) + let err = describe_backup(&*fake, ACCOUNT, REGION, &own(&arn(ACCOUNT))) .await .unwrap_err(); assert!( @@ -4768,7 +5988,7 @@ async fn describe_backup_of_another_accounts_id_is_not_found() { const OTHER: &str = "999999999999"; let fake = fake_with_row(BackupStatus::Available); let other = own_for(&music_arn(OTHER, "01791976431438-73d8d5bc"), OTHER); - let err = describe_backup(&fake, OTHER, REGION, &other) + let err = describe_backup(&*fake, OTHER, REGION, &other) .await .unwrap_err(); assert!( @@ -4930,7 +6150,7 @@ async fn delete_backup_over_a_row_refuses_before_the_lock() { for status in [BackupStatus::Deleted, BackupStatus::Failed] { let fake = fake_with_row(status); let err = delete_backup( - &fake, + &*fake, &runtime_with_store(), ACCOUNT, REGION, @@ -4946,7 +6166,7 @@ async fn delete_backup_over_a_row_refuses_before_the_lock() { } let fake = fake_with_row(BackupStatus::Creating); let err = delete_backup( - &fake, + &*fake, &runtime_with_store(), ACCOUNT, REGION, @@ -4963,7 +6183,7 @@ async fn delete_backup_over_a_row_refuses_before_the_lock() { // No store configured: every row is a row of another store. let fake = fake_with_row(BackupStatus::Available); let msg = validation_message( - delete_backup(&fake, &runtime(), ACCOUNT, REGION, &own(&arn(ACCOUNT))) + delete_backup(&*fake, &runtime(), ACCOUNT, REGION, &own(&arn(ACCOUNT))) .await .unwrap_err(), ); @@ -4986,7 +6206,7 @@ async fn delete_backup_over_a_row_refuses_before_the_lock() { .lock() .unwrap() .insert(arn(ACCOUNT), "Earlier".to_owned()); - let err = delete_backup(&fake, &runtime(), ACCOUNT, REGION, &own(&arn(ACCOUNT))) + let err = delete_backup(&*fake, &runtime(), ACCOUNT, REGION, &own(&arn(ACCOUNT))) .await .unwrap_err(); assert!( @@ -5006,7 +6226,7 @@ async fn delete_backup_over_a_row_refuses_before_the_lock() { async fn delete_backup_over_a_row_runs_the_in_use_check_under_the_lock() { let fake = fake_with_row(BackupStatus::Available); let desc = delete_backup( - &fake, + &*fake, &runtime_with_store(), ACCOUNT, REGION, @@ -5027,7 +6247,7 @@ async fn delete_backup_over_a_row_runs_the_in_use_check_under_the_lock() { let fake = fake_with_row(BackupStatus::Deleting); let desc = delete_backup( - &fake, + &*fake, &runtime_with_store(), ACCOUNT, REGION, @@ -5064,7 +6284,7 @@ async fn delete_backup_over_a_row_runs_the_in_use_check_under_the_lock() { .unwrap() .insert(arn(ACCOUNT), answer.clone()); let err = delete_backup( - &fake, + &*fake, &runtime_with_store(), ACCOUNT, REGION, @@ -5091,7 +6311,7 @@ async fn delete_backup_over_a_row_runs_the_in_use_check_under_the_lock() { .unwrap() .insert(arn(ACCOUNT), DeleteStart::Status(BackupStatus::Available)); let err = delete_backup( - &fake, + &*fake, &runtime_with_store(), ACCOUNT, REGION, @@ -5199,7 +6419,7 @@ async fn a_backup_of_a_name_the_store_prefix_refuses_resolves_through_the_legacy .unwrap() .insert(legacy_arn.clone(), description); - let desc = describe_backup(&fake, ACCOUNT, REGION, &spellings) + let desc = describe_backup(&*fake, ACCOUNT, REGION, &spellings) .await .unwrap(); assert_eq!(desc.backup_details.backup_arn, legacy_arn); @@ -5224,7 +6444,7 @@ async fn a_backup_of_a_name_the_store_prefix_refuses_resolves_through_the_legacy assert_eq!(fake.restores().len(), 1); assert_eq!(fake.restores()[0].backup_arn, legacy_arn); - let desc = delete_backup(&fake, &runtime(), ACCOUNT, REGION, &spellings) + let desc = delete_backup(&*fake, &runtime(), ACCOUNT, REGION, &spellings) .await .unwrap(); assert_eq!(desc.backup_details.backup_status, "DELETED"); @@ -5245,14 +6465,14 @@ async fn a_backup_of_a_name_the_store_prefix_refuses_resolves_through_the_legacy /// or not the row is of the configured store (the in-use check runs before /// the other-store check, as the delete's does); a row /// of another store is a ValidationException, with no store configured and -/// with a different one configured; and an AVAILABLE row of the -/// configured store is refused by this release. The runtime has no free -/// slot, so a permit taken anywhere before the row's refusal, even one -/// dropped again, would turn the refusal into LimitExceeded; the legacy -/// reader is never reached. +/// with a different one configured; and an AVAILABLE row of the healthy +/// configured store passes to the permit, the next check, with no store +/// read. The runtime has no free slot, so a permit taken anywhere before a +/// row's refusal, even one dropped again, would turn the refusal into +/// LimitExceeded; the legacy reader is never reached. #[tokio::test] async fn restore_over_a_row_refuses_before_the_permit() { - async fn check(fake: Fake, runtime: &BackupRuntime) -> DynamoDbError { + async fn check(fake: Arc, runtime: &BackupRuntime) -> DynamoDbError { assert_eq!(runtime.free_slots(), 0, "every slot is held"); let err = restore_table_from_backup( &fake, @@ -5272,10 +6492,11 @@ async fn restore_over_a_row_refuses_before_the_permit() { assert!(fake.restores().is_empty()); err } + let configured_mem = Arc::new(MemStore::default()); let configured = BackupRuntime::new(1).with_store(ConfiguredStore::new( store().store_kind, store().store_root, - Arc::new(MemStore::default()) as Arc, + Arc::clone(&configured_mem) as Arc, None, Ok(store()), )); @@ -5321,8 +6542,29 @@ async fn restore_over_a_row_refuses_before_the_permit() { matches!(err, DynamoDbError::BackupInUseException(ref m) if m.contains("Earlier")), "the in-use check runs before the other-store check: {err:?}" ); - let msg = validation_message(check(fake_with_row(BackupStatus::Available), &configured).await); - assert!(msg.contains("in-catalog backups only"), "{msg}"); + // An AVAILABLE row of the healthy configured store passes the row checks: + // the permit is the next check, before any store read, so with every + // slot held the request is LimitExceeded and the store saw no get. + let fake = fake_with_row(BackupStatus::Available); + let err = restore_table_from_backup( + &fake, + &configured, + ACCOUNT, + REGION, + &LimitsConfig::default(), + &restore_body(json!({})), + ) + .await + .unwrap_err(); + assert!( + matches!(err, DynamoDbError::LimitExceededException(_)), + "the permit is the check after the row's: {err:?}" + ); + assert!( + configured_mem.gets().is_empty(), + "no store read before the permit" + ); + assert!(fake.restore_claims.lock().unwrap().is_empty()); let msg = validation_message(check(fake_with_row(BackupStatus::Available), &none).await); assert!(msg.contains("store other than"), "{msg}"); // A configured store that is not the row's: the row is of another store, @@ -5352,7 +6594,7 @@ async fn restore_over_a_row_refuses_before_the_permit() { #[tokio::test] async fn restore_takes_the_permit_last() { async fn restore_with( - fake: &Fake, + fake: &Arc, runtime: &BackupRuntime, ) -> Result<(String, Value), DynamoDbError> { let body = restore_body(json!({})); @@ -5390,7 +6632,7 @@ async fn restore_takes_the_permit_last() { // A refusal earlier in the order takes no slot: a missing backup. let runtime = BackupRuntime::new(1); - let empty = Fake::default(); + let empty = Arc::new(Fake::default()); let err = restore_with(&empty, &runtime).await.unwrap_err(); assert!( matches!(err, DynamoDbError::BackupNotFoundException(_)), @@ -5796,7 +7038,7 @@ async fn create_backup_under_a_region_outside_the_arn_alphabet_names_the_setting /// the claim is gone both run. #[tokio::test] async fn jobs_are_refused_while_a_maintenance_claim_is_held() { - let fake = Arc::new(Fake::with_source(source(Some(definition("PROVISIONED"))))); + let fake = Fake::with_source(source(Some(definition("PROVISIONED")))); *fake.maintenance_held.lock().unwrap() = true; let runtime = BackupRuntime::new(1).with_stale_job_after(Duration::from_secs(7)); let err = create_backup( @@ -5962,6 +7204,26 @@ fn held(holder: &Arc) -> maintenance::HeldClaim { ) } +/// The claim object the `Fake`'s `maintenance_claim` hands out: `Forward` +/// whose release also frees the `Fake`'s record, so a second command may +/// take the claim once the first released it. +struct ReleasingForward(Arc>>, Arc); + +impl extenddb_storage::MaintenanceClaim for ReleasingForward { + fn generation(&self) -> u64 { + self.1.generation() + } + fn heartbeat( + &self, + ) -> BoxedFuture<'_, Result> { + self.1.heartbeat() + } + fn release(&self) -> BoxedFuture<'_, Result<(), StorageError>> { + *self.0.lock().unwrap() = None; + self.1.release() + } +} + fn seconds_since_epoch() -> f64 { std::time::SystemTime::now() .duration_since(std::time::UNIX_EPOCH) @@ -6136,9 +7398,11 @@ async fn purge_removes_a_dead_creating_row_and_leaves_a_live_one() { /// restore record whose target is still CREATING (the reconciler's duty, /// not a running job) and a FAILED backup row whose owner is dead; busy at /// the limit, with the jobs named (a CREATING row and a FAILED row with a -/// live heartbeat, a RUNNING restore) and announced once; a signal, during -/// a sleep and before the first read; the catalog unreadable; the claim -/// lost. +/// live heartbeat, a RUNNING restore) and announced once; the ages read on +/// the catalog's clock, not this host's (a skewed catalog clock decides +/// busy and quiet against what this host's clock would say); a signal, +/// during a sleep and before the first read; the catalog unreadable; the +/// claim lost. #[tokio::test(start_paused = true)] async fn the_wait_reports_each_outcome_as_itself() { use extenddb_storage::backup_catalog::{JobStatus, RestoreKind}; @@ -6259,27 +7523,29 @@ async fn the_wait_reports_each_outcome_as_itself() { // --- the reconciler ------------------------------------------------------------- fn dead_job(kind: extenddb_storage::backup_catalog::RestoreKind, name: &str) -> RestoreJobRow { - RestoreJobRow { - target: RestoreTargetKey { + RestoreJobRow::reconciler_row( + RestoreTargetKey { account_id: ACCOUNT.to_owned(), table_name: name.to_owned(), table_id: format!("tid-{name}"), }, - source_backup_arn: arn(ACCOUNT), + arn(ACCOUNT), kind, - status: extenddb_storage::backup_catalog::JobStatus::Running, - owner_generation: 0, - heartbeat_at: 0.0, - } + extenddb_storage::backup_catalog::JobStatus::Running, + 0, + 0.0, + ) } /// A legacy restore whose owner is dead is marked FAILED and its target -/// removed; a manifest restore is left for its resumer; an abandoned -/// in-catalog CREATING backup is removed through the reader's delete. +/// removed; a manifest restore whose store is not the configured one (here +/// no store is configured) is marked FAILED naming `--move-store` and its +/// target left to the FAILED-target duty; an abandoned in-catalog CREATING +/// backup is removed through the reader's delete. #[tokio::test] async fn the_reconciler_fails_dead_legacy_restores_and_removes_abandoned_backups() { use extenddb_storage::backup_catalog::RestoreKind; - let fake = Fake::default(); + let fake = Arc::new(Fake::default()); *fake.target_still_creating.lock().unwrap() = true; *fake.record_still_running.lock().unwrap() = true; fake.dead_jobs.lock().unwrap().extend([ @@ -6310,14 +7576,19 @@ async fn the_reconciler_fails_dead_legacy_restores_and_removes_abandoned_backups assert!(!report.maintenance_held); assert_eq!(report.legacy_restores_removed, ["LegacyTarget"]); assert!(report.failed_targets_removed.is_empty()); - assert_eq!(report.manifest_restores_seen, ["ManifestTarget"]); - assert_eq!(report.manifest_restores_logged, ["ManifestTarget"]); + assert!(report.manifest_restores_resumed.is_empty()); + assert_eq!( + report.manifest_restores_failed, + [("ManifestTarget".to_owned(), OTHER_STORE.to_owned())] + ); assert_eq!(report.legacy_backups_removed, [arn(ACCOUNT)]); let failed = fake.failed_jobs.lock().unwrap().clone(); - assert_eq!(failed.len(), 1); + assert_eq!(failed.len(), 2); assert_eq!(failed[0].0.table_name, "LegacyTarget"); assert_eq!(failed[0].1, reconciler::OWNER_DEAD); + assert_eq!(failed[1].0.table_name, "ManifestTarget"); + assert_eq!(failed[1].1, OTHER_STORE); assert_eq!( fake.removed_targets.lock().unwrap().as_slice(), [dead_job(RestoreKind::Legacy, "LegacyTarget").target] @@ -6330,7 +7601,7 @@ async fn the_reconciler_fails_dead_legacy_restores_and_removes_abandoned_backups #[tokio::test] async fn the_reconciler_leaves_a_target_whose_record_moved() { use extenddb_storage::backup_catalog::RestoreKind; - let fake = Fake::default(); + let fake = Arc::new(Fake::default()); *fake.record_still_running.lock().unwrap() = false; *fake.target_still_creating.lock().unwrap() = true; fake.dead_jobs @@ -6361,7 +7632,7 @@ async fn the_reconciler_leaves_a_target_whose_record_moved() { #[tokio::test] async fn the_reconciler_leaves_a_target_that_is_no_longer_creating_after_the_failed_write() { use extenddb_storage::backup_catalog::{JobStatus, RestoreKind}; - let fake = Fake::default(); + let fake = Arc::new(Fake::default()); *fake.record_still_running.lock().unwrap() = true; *fake.target_still_creating.lock().unwrap() = false; let dead = dead_job(RestoreKind::Legacy, "LegacyTarget"); @@ -6576,6 +7847,86 @@ async fn a_cycle_run_by_the_claim_holder_takes_the_held_branch_whatever_its_wind assert_eq!(report.registrations_swept, [arn_n(9)]); } +/// The restore arm of a cycle run by the holder of the maintenance claim with +/// a zero dead-owner window (`backup purge` on SQLite): a RUNNING `manifest` +/// record whose owner is dead is FAILED `OWNER_DEAD` and its target removed, +/// as 4(f) asks of a held claim, and not taken over and resumed inside the +/// command. The command's claim was heartbeated a millisecond ago, which a +/// zero window read through the clock finds stale; a cycle that read it +/// would take the arm for a free claim and spawn a resume. +#[tokio::test] +async fn a_cycle_run_by_the_claim_holder_fails_a_dead_manifest_restore_rather_than_resuming_it() { + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + let runtime = job_runtime(&mem); + let gate = closed_gate(); + *fake.bulk_load_gate.lock().unwrap() = Some(Arc::clone(&gate)); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_bulk_loads_started(&fake, 1).await; + runtime.drain_jobs(Duration::from_millis(50)).await; + gate.add_permits(10); + wait_for_jobs(&runtime).await; + *fake.bulk_load_gate.lock().unwrap() = None; + assert_eq!( + fake.restore_job("MusicRestored").unwrap().status, + JobStatus::Running, + "precondition: the stopped restore left its record RUNNING" + ); + fake.restore_dead + .lock() + .unwrap() + .insert("tid-MusicRestored".to_owned()); + *fake.maintenance_beat_age.lock().unwrap() = Some(Duration::from_millis(1)); + // The restore's own request read the claim; the cycle must not. + fake.maintenance_checks.lock().unwrap().clear(); + + let command = job_runtime(&mem); + let report = reconciler::cycle( + &fake, + &command, + Duration::ZERO, + "test", + &mut reconciler::ReconcilerState::for_maintenance_holder(), + ) + .await + .unwrap(); + wait_for_jobs(&command).await; + assert!( + report.manifest_restores_resumed.is_empty(), + "the holder's cycle resumed a restore: {:?}", + report.manifest_restores_resumed + ); + assert!(fake.restore_takeovers.lock().unwrap().is_empty()); + assert_eq!( + report.manifest_restores_failed_and_removed, + ["MusicRestored"] + ); + assert!(fake.restore_job("MusicRestored").is_none()); + assert!(!fake.tables.lock().unwrap().contains_key("MusicRestored")); + assert!( + fake.failed_jobs + .lock() + .unwrap() + .iter() + .any(|(key, reason)| key.table_name == "MusicRestored" + && reason == reconciler::OWNER_DEAD), + "{:?}", + fake.failed_jobs.lock().unwrap() + ); + assert!(fake.maintenance_checks.lock().unwrap().is_empty()); +} + /// The worker runs a cycle at once (observed through the maintenance check, /// the cycle's first read), records the worker metrics the other workers /// record, then waits an interval, and stops at the shutdown token. @@ -6680,7 +8031,7 @@ async fn a_failed_reconciler_cycle_records_a_worker_error() { #[tokio::test] async fn the_reconciler_removes_the_target_of_a_failed_record_without_writing_failed_again() { use extenddb_storage::backup_catalog::{JobStatus, RestoreKind}; - let fake = Fake::default(); + let fake = Arc::new(Fake::default()); *fake.target_still_creating.lock().unwrap() = true; let mut failed = dead_job(RestoreKind::Legacy, "FailedTarget"); failed.status = JobStatus::Failed; @@ -6708,70 +8059,14 @@ async fn the_reconciler_removes_the_target_of_a_failed_record_without_writing_fa ["FailedTarget", "FailedManifestTarget"] ); assert!(report.legacy_restores_removed.is_empty()); - assert!(report.manifest_restores_seen.is_empty()); + assert!(report.manifest_restores_resumed.is_empty()); + assert!(report.manifest_restores_failed.is_empty()); assert_eq!( fake.removed_targets.lock().unwrap().as_slice(), [failed.target, failed_manifest.target] ); } -/// A dead manifest record, which this release cannot act on, is reported -/// once per record for as long as the process runs, not every cycle. -#[tokio::test] -async fn a_dead_manifest_record_is_logged_once_per_record() { - use extenddb_storage::backup_catalog::RestoreKind; - let fake = Fake::default(); - fake.dead_jobs - .lock() - .unwrap() - .push(dead_job(RestoreKind::Manifest, "ManifestTarget")); - let mut state = reconciler::ReconcilerState::default(); - let first = reconciler::cycle( - &fake, - &runtime(), - Duration::from_secs(300), - "test", - &mut state, - ) - .await - .unwrap(); - assert_eq!(first.manifest_restores_seen, ["ManifestTarget"]); - assert_eq!(first.manifest_restores_logged, ["ManifestTarget"]); - let second = reconciler::cycle( - &fake, - &runtime(), - Duration::from_secs(300), - "test", - &mut state, - ) - .await - .unwrap(); - assert_eq!(second.manifest_restores_seen, ["ManifestTarget"]); - assert!( - second.manifest_restores_logged.is_empty(), - "the same record is not reported twice" - ); - fake.dead_jobs - .lock() - .unwrap() - .push(dead_job(RestoreKind::Manifest, "Another")); - let third = reconciler::cycle( - &fake, - &runtime(), - Duration::from_secs(300), - "test", - &mut state, - ) - .await - .unwrap(); - assert_eq!(third.manifest_restores_logged, ["Another"]); - assert!( - fake.failed_jobs.lock().unwrap().is_empty() - && fake.removed_targets.lock().unwrap().is_empty(), - "a manifest record is never acted on by this release" - ); -} - // --- the store identity file and the store's health --------------------------- const MANIFEST_KEY_SUFFIX: &str = "extenddb-manifest.json"; @@ -6798,8 +8093,8 @@ fn newer_arn(account: &str) -> String { format!("arn:aws:dynamodb:{REGION}:{account}:table/Music/backup/01791976999999-0badf00d") } -fn fake_with_rows(rows: &[(String, BackupStatus)]) -> Fake { - let fake = Fake::default(); +fn fake_with_rows(rows: &[(String, BackupStatus)]) -> Arc { + let fake = Arc::new(Fake::default()); for (arn, status) in rows { fake.rows .lock() @@ -6868,7 +8163,7 @@ async fn a_file_naming_another_catalog_is_unhealthy_until_restored() { &identity_file("store-1", "another-catalog"), ); let fake = fake_with_row(BackupStatus::Available); - let err = store_identity::establish(&mem, StoreKind::Filesystem, "/r", &fake, "test") + let err = store_identity::establish(&mem, StoreKind::Filesystem, "/r", &*fake, "test") .await .unwrap_err(); assert_eq!( @@ -6883,7 +8178,7 @@ async fn a_file_naming_another_catalog_is_unhealthy_until_restored() { // Restored: the check passes. mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); mem.insert(&manifest_key(&arn(ACCOUNT)), b"{}"); - let identity = store_identity::recheck(&mem, &store(), &fake, "test").await; + let identity = store_identity::recheck(&mem, &store(), &*fake, "test").await; assert_eq!(identity, Ok(())); } @@ -6919,7 +8214,7 @@ async fn ids_that_disagree_or_a_missing_file_with_rows_are_unhealthy() { &mem, StoreKind::Filesystem, store().store_root.as_str(), - &fake, + &*fake, "test", ) .await @@ -6939,7 +8234,7 @@ async fn ids_that_disagree_or_a_missing_file_with_rows_are_unhealthy() { &mem, StoreKind::Filesystem, store().store_root.as_str(), - &fake, + &*fake, "test", ) .await @@ -6956,7 +8251,7 @@ async fn ids_that_disagree_or_a_missing_file_with_rows_are_unhealthy() { &mem, StoreKind::Filesystem, store().store_root.as_str(), - &fake, + &*fake, "test", ) .await @@ -6971,7 +8266,7 @@ async fn ids_that_disagree_or_a_missing_file_with_rows_are_unhealthy() { &mem, StoreKind::Filesystem, store().store_root.as_str(), - &fake, + &*fake, "test", ) .await @@ -7004,7 +8299,7 @@ async fn the_manifest_probe_reads_the_newest_available_rows_manifest() { &mem, StoreKind::Filesystem, &store().store_root, - &fake, + &*fake, "test", ) .await @@ -7021,7 +8316,7 @@ async fn the_manifest_probe_reads_the_newest_available_rows_manifest() { &mem, StoreKind::Filesystem, &store().store_root, - &fake, + &*fake, "test", ) .await @@ -7035,7 +8330,7 @@ async fn the_manifest_probe_reads_the_newest_available_rows_manifest() { &empty, StoreKind::Filesystem, &store().store_root, - &fake, + &*fake, "test", ) .await @@ -7077,12 +8372,12 @@ async fn an_unreachable_store_is_unhealthy_and_recovers_on_recheck() { let mem = Arc::new(MemStore::default()); mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); mem.set_down(true); - let fake = Fake::default(); + let fake = Arc::new(Fake::default()); let err = store_identity::establish( &*mem, StoreKind::Filesystem, &store().store_root, - &fake, + &*fake, "test", ) .await @@ -7166,7 +8461,7 @@ async fn a_stalled_store_check_is_abandoned_at_its_bound() { let runtime = runtime_on(&mem, store()); let configured = runtime.configured_store().unwrap(); assert_eq!(configured.check_timeout(), LOCAL_CHECK_TIMEOUT); - let fake = Fake::default(); + let fake = Arc::new(Fake::default()); *fake.record_still_running.lock().unwrap() = true; *fake.target_still_creating.lock().unwrap() = true; fake.dead_jobs @@ -7560,7 +8855,7 @@ async fn a_catalog_read_failure_leaves_the_health_unchanged() { ); // And through a cycle: a catalog that fails `catalog_id` leaves a // healthy store healthy. - let fake = Fake::default(); + let fake = Arc::new(Fake::default()); *fake.catalog_id_error.lock().unwrap() = true; let report = reconciler::cycle( &fake, @@ -7628,7 +8923,7 @@ async fn a_store_checked_after_startup_is_checking_until_the_first_check() { ); // The first check: `validate` fails on the unreachable store. - let fake = Fake::default(); + let fake = Arc::new(Fake::default()); let report = reconciler::cycle( &fake, &runtime, @@ -7727,7 +9022,7 @@ async fn an_unhealthy_store_refuses_create_delete_and_restore_unless_bound_to_th fake.permits.lock().unwrap().clear(); // DeleteBackup of a row of the store. - let fake = Arc::new(fake_with_row(BackupStatus::Available)); + let fake = fake_with_row(BackupStatus::Available); let err = delete_backup(&*fake, &unhealthy(), ACCOUNT, REGION, &own(&arn(ACCOUNT))) .await .unwrap_err(); @@ -7826,7 +9121,7 @@ async fn delete_backup_of_a_configured_store_row_deletes_manifest_prefix_then_ma (newer_arn(ACCOUNT), BackupStatus::Available), ]); let runtime = runtime_on(&mem, store()); - let desc = delete_backup(&fake, &runtime, ACCOUNT, REGION, &own(&arn(ACCOUNT))) + let desc = delete_backup(&*fake, &runtime, ACCOUNT, REGION, &own(&arn(ACCOUNT))) .await .unwrap(); assert_eq!(desc.backup_details.backup_status, "DELETED"); @@ -7864,7 +9159,7 @@ async fn a_failed_deletion_stays_deleting_and_the_reconciler_finishes_it() { let runtime = runtime_on(&mem, store()); mem.set_down(true); - let err = delete_backup(&fake, &runtime, ACCOUNT, REGION, &own(&arn(ACCOUNT))) + let err = delete_backup(&*fake, &runtime, ACCOUNT, REGION, &own(&arn(ACCOUNT))) .await .unwrap_err(); assert!( @@ -8036,7 +9331,7 @@ async fn a_prefix_that_still_lists_an_object_is_left_deleting() { let store_handle = Refilling(MemStore::default()); let fake = fake_with_rows(&[(arn(ACCOUNT), BackupStatus::Deleting)]); let parsed = BackupArn::parse(&arn(ACCOUNT)).unwrap(); - let err = finish_backup_delete(&fake, &store_handle, &parsed) + let err = finish_backup_delete(&*fake, &store_handle, &parsed) .await .unwrap_err(); assert!(err.contains("still lists"), "{err}"); @@ -9010,8 +10305,7 @@ async fn the_heartbeat_handle_confirms_the_claim_within_two_periods() { let handle = job::heartbeat::HeartbeatHandle::start( Arc::clone(&claim), period, - None, - 0.0, + job::heartbeat::HeartbeatMetric::None, tokio::time::Instant::now(), ); // Half a period past the tenth tick, so that tick's beat has confirmed. @@ -9035,8 +10329,7 @@ async fn the_heartbeat_handle_confirms_the_claim_within_two_periods() { let handle = job::heartbeat::HeartbeatHandle::start( claim, period, - None, - 0.0, + job::heartbeat::HeartbeatMetric::None, tokio::time::Instant::now(), ); tokio::time::sleep(period + period / 2).await; @@ -9219,6 +10512,17 @@ impl LogLines { .map(|(_, m)| m.clone()) .collect() } + + /// The messages logged at `level`, in order. + fn at_level(&self, level: tracing::Level) -> Vec { + self.0 + .lock() + .unwrap() + .iter() + .filter(|(l, _)| *l == level) + .map(|(_, m)| m.clone()) + .collect() + } } struct MessageVisitor<'a>(&'a mut String); @@ -9654,8 +10958,7 @@ async fn a_stopped_heartbeat_timer_starts_no_put() { let heartbeat = job::heartbeat::HeartbeatHandle::start( claim, period, - None, - 0.0, + job::heartbeat::HeartbeatMetric::None, tokio::time::Instant::now(), ); heartbeat.stop_timer(); @@ -9868,7 +11171,7 @@ fn seed_dead_creating_with_manifest(fake: &Fake, mem: &MemStore, complete: bool) } /// One cycle on `fake` and `mem`, the store healthy. -async fn cycle_on(fake: &Fake, runtime: &BackupRuntime) -> reconciler::CycleReport { +async fn cycle_on(fake: &Arc, runtime: &BackupRuntime) -> reconciler::CycleReport { reconciler::cycle( fake, runtime, @@ -9892,7 +11195,7 @@ async fn the_reconcilers_failed_store_calls_are_counted_by_call() { use extenddb_core::metrics::{MetricName, MetricsCollector}; let mem = Arc::new(MemStore::default()); mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); - let fake = Fake::default(); + let fake = Arc::new(Fake::default()); let now = seconds_since_epoch(); let dead = seed_dead_creating_with_manifest(&fake, &mem, true); let dead_prefix = BackupArn::parse(&dead).unwrap().store_prefix(); @@ -9961,7 +11264,7 @@ async fn the_reconcilers_failed_store_calls_are_counted_by_call() { async fn the_reconciler_resolves_a_dead_creating_row_with_a_manifest_to_available() { let mem = Arc::new(MemStore::default()); mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); - let fake = Fake::default(); + let fake = Arc::new(Fake::default()); let dead = seed_dead_creating_with_manifest(&fake, &mem, true); let stats_pass = serde_json::json!({"TableName": "Music", "ItemCount": 0}); fake.rows @@ -10041,7 +11344,7 @@ async fn the_reconciler_fails_a_dead_creating_row_without_a_complete_manifest() ] { let mem = Arc::new(MemStore::default()); mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); - let fake = Fake::default(); + let fake = Arc::new(Fake::default()); let mut dead = seed_dead_creating_with_manifest(&fake, &mem, shape != "short file"); let prefix = BackupArn::parse(&dead).unwrap().store_prefix(); let golden = stored_manifest(&mem, &dead); @@ -10154,7 +11457,7 @@ async fn the_reconciler_fails_a_dead_creating_row_without_a_complete_manifest() // store answering, resolves it. let mem = Arc::new(MemStore::default()); mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); - let fake = Fake::default(); + let fake = Arc::new(Fake::default()); let dead = seed_dead_creating_with_manifest(&fake, &mem, true); *mem.fail_gets_under.lock().unwrap() = Some(BackupArn::parse(&dead).unwrap().store_prefix()); let runtime = runtime_on(&mem, store()); @@ -10189,7 +11492,7 @@ async fn the_reconciler_fails_a_dead_creating_row_without_a_complete_manifest() async fn a_dead_creating_row_whose_status_write_errors_is_left_for_the_next_cycle() { let mem = Arc::new(MemStore::default()); mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); - let fake = Fake::default(); + let fake = Arc::new(Fake::default()); let dead = seed_dead_creating_with_manifest(&fake, &mem, true); fake.status_write_error .store(true, std::sync::atomic::Ordering::Release); @@ -10220,7 +11523,7 @@ async fn a_dead_creating_row_whose_status_write_errors_is_left_for_the_next_cycl async fn a_row_failed_by_the_cycle_loses_its_prefix_in_that_cycle() { let mem = Arc::new(MemStore::default()); mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); - let fake = Fake::default(); + let fake = Arc::new(Fake::default()); let dead = seed_dead_creating_with_manifest(&fake, &mem, false); assert!(!objects_under(&mem, &dead).is_empty()); let runtime = runtime_on(&mem, store()); @@ -10250,7 +11553,7 @@ async fn a_row_failed_by_the_cycle_loses_its_prefix_in_that_cycle() { async fn a_dead_creating_row_taken_over_by_another_is_left_to_that_owner() { let mem = Arc::new(MemStore::default()); mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); - let fake = Fake::default(); + let fake = Arc::new(Fake::default()); let dead = seed_dead_creating_with_manifest(&fake, &mem, true); fake.takeover_lost_to_another .store(true, std::sync::atomic::Ordering::Release); @@ -10285,7 +11588,7 @@ async fn the_reconciler_deletes_failed_and_deleted_prefixes_and_removes_tombston { let mem = Arc::new(MemStore::default()); mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); - let fake = Fake::default(); + let fake = Arc::new(Fake::default()); let now = seconds_since_epoch(); let stale_failed = arn_n(1); let fresh_failed = arn_n(2); @@ -10499,3 +11802,5011 @@ async fn the_reconciler_deletes_failed_and_deleted_prefixes_and_removes_tombston assert!(report.failed_prefixes_deleted.is_empty()); assert!(report.creating_resolved.is_empty()); } + +// --- the store-backed restore --------------------------------------------------- + +use extenddb_core::backup_format::{ + BackupDocuments, DataFileEntry, EXTENDDB_MANIFEST_FILE, ExtenddbManifest, ItemOrder, KeyOrder, + KeyRangeTracker, MANIFEST_CHECKSUM_FILE, ManifestHeader, SnapshotDescriptor, SourceTableSchema, + data_file_key, encode_data_file, +}; +use extenddb_core::metrics::MetricName; +use extenddb_storage::snapshot::source_table_schema; + +/// The `Music` table's source schema as a manifest carries it. +fn music_source_table() -> SourceTableSchema { + source_table_schema(&music_table(), None).unwrap() +} + +/// `music_source_table` as an on-demand table (vector indexes need one). +fn on_demand_source_table() -> SourceTableSchema { + let mut source_table = music_source_table(); + source_table.billing_mode = BillingMode::PayPerRequest; + source_table.provisioned_throughput = None; + for gsi in &mut source_table.global_secondary_indexes { + gsi.provisioned_throughput = None; + } + source_table +} + +/// A backup of `source_table` under `mem` for `arn`, as the writer leaves +/// one: one data file per entry of `files`, encoded with the format's +/// encoder; key-ordered with the tracker's ranges when `key_ordered`; the +/// four documents built by `BackupDocuments::build`; and the row AVAILABLE +/// in `fake` with the manifest's `source_table`. Returns the manifest. +fn seed_manifest_backup( + fake: &Fake, + mem: &MemStore, + arn: &str, + mut source_table: SourceTableSchema, + files: &[Vec], + key_ordered: bool, +) -> ExtenddbManifest { + let parsed = BackupArn::parse(arn).unwrap(); + let prefix = parsed.store_prefix(); + let mut tracker = key_ordered.then(|| { + KeyRangeTracker::new( + &KeyOrder::resolve( + &source_table.key_schema, + &source_table.attribute_definitions, + ) + .unwrap(), + ) + }); + let mut entries = Vec::new(); + let mut total = 0; + for (i, items) in files.iter().enumerate() { + let (bytes, checksums) = encode_data_file(items.iter().cloned()).unwrap(); + let range = tracker.as_mut().map(|t| { + for item in items { + t.push(item).unwrap(); + } + t.next_file().expect("a file with items has a range") + }); + let key = data_file_key(u32::try_from(i + 1).unwrap()); + mem.insert(&format!("{prefix}{key}"), &bytes); + total += items.len(); + entries.push((DataFileEntry::from_checksums(key, &checksums, range), None)); + } + source_table.item_count = i64::try_from(total).unwrap(); + seed_manifest_documents(fake, mem, arn, source_table, entries, key_ordered) +} + +/// The documents of a backup whose data files are already under the prefix, +/// from `entries`, and the AVAILABLE row; the manifest as built. +fn seed_manifest_documents( + fake: &Fake, + mem: &MemStore, + arn: &str, + source_table: SourceTableSchema, + entries: Vec<(DataFileEntry, Option)>, + key_ordered: bool, +) -> ExtenddbManifest { + let parsed = BackupArn::parse(arn).unwrap(); + let prefix = parsed.store_prefix(); + let total: u64 = entries.iter().map(|(e, _)| e.item_count).sum(); + let docs = BackupDocuments::build( + ManifestHeader { + backup_arn: parsed, + backup_name: "nightly".to_owned(), + source_backend: "test".to_owned(), + source_extenddb_version: "0.0.0".to_owned(), + snapshot: SnapshotDescriptor { + kind: "test".to_owned(), + marker: String::new(), + }, + item_order: if key_ordered { + ItemOrder::KeyOrdered + } else { + ItemOrder::Unordered + }, + source_table, + }, + entries, + "", + &store().store_root, + ) + .unwrap(); + for (name, bytes) in docs.documents() { + mem.insert(&format!("{prefix}{name}"), bytes); + } + let manifest = docs.manifest().clone(); + let mut available = row(arn, BackupStatus::Available); + available.source_table_details_json = serde_json::to_value(&manifest.source_table).unwrap(); + available.size_bytes = manifest.backup_size_bytes; + available.item_count = i64::try_from(total).unwrap(); + fake.rows.lock().unwrap().insert(arn.to_owned(), available); + manifest +} + +/// `n` songs from `from`, in key order. +fn songs_from(from: usize, n: usize) -> Vec { + (from..from + n) + .map(|i| song(&format!("artist-{i:05}"), &format!("title-{i:05}"))) + .collect() +} + +/// A healthy runtime on `mem` with a spool at `spool` (an S3 store's shape), +/// the job settings and a collector. +fn spooled_runtime(mem: &Arc, spool: &std::path::Path) -> BackupRuntime { + runtime() + .with_store( + ConfiguredStore::new( + StoreKind::S3, + "bucket/prefix".to_owned(), + Arc::clone(mem) as Arc, + Some(Spool { + path: spool.to_path_buf(), + spool_host: "host-a:/var/spool".to_owned(), + }), + Ok(StoreIdentity { + store_kind: StoreKind::S3, + store_root: "bucket/prefix".to_owned(), + store_id: "store-1".to_owned(), + }), + ) + .with_manifest_names("bucket".to_owned(), "prefix/".to_owned()), + ) + .with_job_settings(job_settings()) + .with_metrics(Arc::new(extenddb_core::metrics::MetricsCollector::new())) +} + +/// The `s3_row` of `arn`: the row with the spooled runtime's identity. +fn s3_identity() -> StoreIdentity { + StoreIdentity { + store_kind: StoreKind::S3, + store_root: "bucket/prefix".to_owned(), + store_id: "store-1".to_owned(), + } +} + +/// The `RestoreJobs` outcomes the collector drained, with their count. +fn restore_outcomes(runtime: &BackupRuntime) -> Vec<(String, f64)> { + let mut outcomes: Vec<(String, f64)> = runtime + .metrics() + .unwrap() + .drain(Duration::ZERO) + .into_iter() + .filter(|b| b.metric == MetricName::RestoreJobs) + .map(|b| (b.operation, b.sum)) + .collect(); + outcomes.sort_by(|a, b| a.0.cmp(&b.0)); + outcomes +} + +/// Every metric bucket the collector drained, as `(name, operation, sum)`. +fn drained(runtime: &BackupRuntime) -> Vec<(MetricName, String, f64)> { + runtime + .metrics() + .unwrap() + .drain(Duration::ZERO) + .into_iter() + .map(|b| (b.metric, b.operation, b.sum)) + .collect() +} + +/// Wait until `n` `bulk_load` calls have begun (a job waiting at the gate +/// counts). +async fn wait_for_bulk_loads_started(fake: &Fake, n: usize) { + let deadline = tokio::time::Instant::now() + Duration::from_secs(30); + while fake + .bulk_loads_started + .load(std::sync::atomic::Ordering::Acquire) + < n + { + assert!(tokio::time::Instant::now() < deadline, "no bulk_load began"); + tokio::time::sleep(Duration::from_millis(5)).await; + } +} + +/// The status writes the restore claims received, as `(from, to, reason)`. +fn restore_writes(fake: &Fake) -> Vec<(JobStatus, JobStatus, Option)> { + fake.restore_writes + .lock() + .unwrap() + .iter() + .map(|w| (w.from, w.to, w.fields.failure_reason.clone())) + .collect() +} + +/// The store keys `mem` saw `get` for, without the prefix. +fn gets_under(mem: &MemStore, arn: &str) -> Vec { + let prefix = BackupArn::parse(arn).unwrap().store_prefix(); + mem.gets() + .into_iter() + .filter_map(|k| k.strip_prefix(&prefix).map(str::to_owned)) + .collect() +} + +/// The happy path. The request: the row passes, the permit is taken, the +/// sidecar and manifest are read, the registration carries the manifest's +/// table (key schema, GSI, billing) and the row's store, the maintenance +/// claim is read once after it, and the response is CREATING with the +/// manifest's provenance in its `RestoreSummary`. The job: every file is +/// read once to hash it (in order), then once more to decode it, each file +/// goes to one `bulk_load` from item 0 with `reapplying` false, the items +/// come back whole and in order, the target is counted, the record is DONE +/// and the table ACTIVE with its GSI, no status write was needed, the slot +/// is free; `RestoreJobs{succeeded}`, `RestoreBytesDownloaded` of every +/// file's bytes, `WorkerLastSuccess{RestoreJob}`. +#[tokio::test] +async fn a_store_backed_restore_answers_creating_and_its_job_loads_the_backup() { + let (lines, _guard) = LogLines::capture(); + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + let files = [songs_from(0, 3), songs_from(3, 2), songs_from(5, 4)]; + let manifest = seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &files, + true, + ); + let runtime = job_runtime(&mem); + + let response = restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + let table = &response["TableDescription"]; + assert_eq!(table["TableStatus"], "CREATING"); + assert_eq!(table["TableName"], "MusicRestored"); + assert_eq!(table["RestoreSummary"]["SourceBackupArn"], arn(ACCOUNT)); + assert_eq!(table["RestoreSummary"]["SourceTableArn"], source_arn()); + assert_eq!( + table["RestoreSummary"]["RestoreDateTime"], + json!(manifest.backup_creation_date_time) + ); + assert_eq!(table["RestoreSummary"]["RestoreInProgress"], true); + let claims = fake.restore_claims.lock().unwrap().clone(); + assert_eq!(claims.len(), 1); + assert_eq!(claims[0].0, ACCOUNT); + let input = &claims[0].1; + assert_eq!(input.table_name, "MusicRestored"); + assert_eq!(input.key_schema, music_table().key_schema); + assert_eq!(input.billing_mode, Some(BillingMode::Provisioned)); + assert_eq!(gsi_names(input), ["GenreIndex"]); + assert_eq!(input.vector_indexes, None); + let record = fake.restore_job("MusicRestored").unwrap(); + assert_eq!(record.kind, RestoreKind::Manifest); + assert_eq!(record.store, Some(store())); + assert_eq!(record.spool_host, None, "a filesystem store has no spool"); + assert_eq!( + fake.maintenance_checks.lock().unwrap().as_slice(), + [job_settings().stale_job_after], + "the maintenance claim was read once, after the registration" + ); + + wait_for_jobs(&runtime).await; + assert_eq!( + fake.bulk_load_files(), + [(0, 0, false, 3), (1, 0, false, 2), (2, 0, false, 4)] + ); + let all: Vec = files.iter().flatten().cloned().collect(); + assert_eq!(fake.restored_items("MusicRestored"), all); + let record = fake.restore_job("MusicRestored").unwrap(); + assert_eq!(record.status, JobStatus::Done); + assert!(record.completed_at.is_some()); + assert_eq!(record.file_index, Some(2)); + assert_eq!(record.items_applied, Some(4)); + let desc = fake.tables.lock().unwrap()["MusicRestored"].clone(); + assert_eq!(desc.table_status, TableStatus::Active); + assert_eq!(desc.item_count, 9); + assert!( + restore_writes(&fake).is_empty(), + "the flip is the backend's one write" + ); + assert_eq!( + fake.finish_sizes.lock().unwrap().as_slice(), + [u64::try_from(manifest.source_table.table_size_bytes).unwrap()], + "the restored table's size is the source table's, not the compressed backup's" + ); + assert_ne!( + manifest.source_table.table_size_bytes, manifest.backup_size_bytes, + "the two sizes differ, so the assertion above tells them apart" + ); + assert_eq!(runtime.free_slots(), 4); + let reads = gets_under(&mem, &arn(ACCOUNT)); + assert_eq!( + reads, + [ + MANIFEST_CHECKSUM_FILE, + EXTENDDB_MANIFEST_FILE, + "data/000001.json.gz", + "data/000002.json.gz", + "data/000003.json.gz", + "data/000001.json.gz", + "data/000002.json.gz", + "data/000003.json.gz", + ], + "every file hashed before any is decoded, each read twice in place" + ); + let bytes: u64 = manifest.data_files.iter().map(|f| f.bytes).sum(); + let metrics = drained(&runtime); + assert_eq!( + metrics + .iter() + .filter(|(m, _, _)| *m == MetricName::RestoreJobs) + .map(|(_, op, sum)| (op.clone(), *sum)) + .collect::>(), + [("succeeded".to_owned(), 1.0)] + ); + // Summed across the drain's buckets (one per wall-clock minute). + let downloaded: f64 = metrics + .iter() + .filter(|(m, _, _)| *m == MetricName::RestoreBytesDownloaded) + .map(|(_, _, sum)| *sum) + .sum(); + assert!( + (downloaded - bytes as f64).abs() < f64::EPSILON, + "{metrics:?}" + ); + assert!( + metrics + .iter() + .any(|(m, op, _)| *m == MetricName::WorkerLastSuccess && op == "RestoreJob"), + "{metrics:?}" + ); + let started = lines.containing(": started, 3 data file(s), 9 items, key-ordered"); + assert_eq!(started.len(), 1, "{started:?}"); + let active = lines.containing(": ACTIVE, 9 items (9 applied by this run), "); + assert_eq!(active.len(), 1, "{active:?}"); +} + +/// The request's order on the manifest path, each refusal before the next +/// step and the registration: a table of the target's name is +/// `TableAlreadyExists` with no row read; a missing sidecar or manifest is +/// `BackupNotFound` with the operator's line naming `--prune-missing`; a +/// sidecar that does not match, a manifest naming another backup or another +/// table are `ValidationException`s; none of them registers anything or +/// takes a slot for longer than the request. +#[tokio::test] +async fn the_manifest_request_refuses_in_order_before_the_registration() { + let (lines, _guard) = LogLines::capture(); + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + let runtime = job_runtime(&mem); + let prefix = BackupArn::parse(&arn(ACCOUNT)).unwrap().store_prefix(); + + // A taken name: before the row. + fake.tables.lock().unwrap().insert( + "MusicRestored".to_owned(), + TableDescription { + table_name: "MusicRestored".to_owned(), + table_status: TableStatus::Active, + ..Default::default() + }, + ); + let err = restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap_err(); + assert!( + matches!(err, DynamoDbError::TableAlreadyExistsException(_)), + "{err:?}" + ); + assert!(fake.row_reads.lock().unwrap().is_empty()); + assert!(mem.gets().is_empty()); + fake.tables.lock().unwrap().remove("MusicRestored"); + + // A missing sidecar, then a missing manifest. + let sidecar_key = format!("{prefix}{MANIFEST_CHECKSUM_FILE}"); + let manifest_key = format!("{prefix}{EXTENDDB_MANIFEST_FILE}"); + let sidecar = mem.objects.lock().unwrap().remove(&sidecar_key).unwrap(); + let err = restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap_err(); + assert!( + matches!(err, DynamoDbError::BackupNotFoundException(ref m) if m.contains(&arn(ACCOUNT))), + "{err:?}" + ); + let prune = lines.containing("backup sync --prune-missing"); + assert_eq!(prune.len(), 1, "{prune:?}"); + assert!(prune[0].contains(MANIFEST_CHECKSUM_FILE), "{}", prune[0]); + mem.insert(&sidecar_key, &sidecar); + let manifest = mem.objects.lock().unwrap().remove(&manifest_key).unwrap(); + let err = restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap_err(); + assert!( + matches!(err, DynamoDbError::BackupNotFoundException(_)), + "{err:?}" + ); + assert_eq!(lines.containing("backup sync --prune-missing").len(), 2); + + // A sidecar that does not match the manifest. + mem.insert(&manifest_key, &manifest); + mem.insert( + &sidecar_key, + b"0000000000000000000000000000000000000000000000000000000000000000\n", + ); + let msg = validation_message( + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap_err(), + ); + assert!(msg.contains("checksum sidecar does not match"), "{msg}"); + mem.insert(&sidecar_key, &sidecar); + + // A manifest naming another backup (another account's, the same id), + // with a sidecar that matches it. + let other = arn("999999999999"); + let mut parsed: Value = serde_json::from_slice(&manifest).unwrap(); + parsed["backup_arn"] = json!(other); + parsed["source_table"]["table_arn"] = json!(format!( + "arn:aws:dynamodb:{REGION}:999999999999:table/Music" + )); + let other_bytes = serde_json::to_vec(&parsed).unwrap(); + let other_sidecar = { + use sha2::Digest; + format!( + "{:x} {EXTENDDB_MANIFEST_FILE}\n", + sha2::Sha256::digest(&other_bytes) + ) + }; + mem.insert(&manifest_key, &other_bytes); + mem.insert(&sidecar_key, other_sidecar.as_bytes()); + let msg = validation_message( + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap_err(), + ); + assert!( + msg.contains("names another backup") && msg.contains(&other), + "{msg}" + ); + + // A manifest whose source table is not the ARN's: the reader's own + // identity rule refuses it at the parse. + let mut parsed: Value = serde_json::from_slice(&manifest).unwrap(); + parsed["source_table"]["table_name"] = json!("Other"); + let renamed = serde_json::to_vec(&parsed).unwrap(); + let renamed_sidecar = { + use sha2::Digest; + format!( + "{:x} {EXTENDDB_MANIFEST_FILE}\n", + sha2::Sha256::digest(&renamed) + ) + }; + mem.insert(&manifest_key, &renamed); + mem.insert(&sidecar_key, renamed_sidecar.as_bytes()); + let msg = validation_message( + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap_err(), + ); + assert!( + msg.contains("cannot restore from") + && msg.contains("names table \"Music\" but source_table.table_name is \"Other\""), + "{msg}" + ); + + assert!( + fake.restore_claims.lock().unwrap().is_empty(), + "nothing was registered" + ); + assert_eq!(runtime.free_slots(), 4, "every refusal returned its slot"); + assert_eq!(runtime.running_jobs(), 0); +} + +/// The maintenance claim is checked after the registration, not before: a +/// held claim leaves a record that was FAILED `maintenance` from RUNNING +/// under the job's claim, its target removed, and the request answered +/// `LimitExceededException`; the registration ran (the row is what the +/// command's wait sees). A claim that cannot be read fails the record with +/// the read's error and answers it. +#[tokio::test] +async fn a_manifest_restore_under_a_maintenance_claim_fails_its_record_after_registering() { + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + let runtime = job_runtime(&mem); + *fake.maintenance_held.lock().unwrap() = true; + let err = restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap_err(); + assert!( + matches!(err, DynamoDbError::LimitExceededException(ref m) if m.contains("maintenance claim")), + "{err:?}" + ); + wait_for_jobs(&runtime).await; + assert_eq!( + fake.restore_claims.lock().unwrap().len(), + 1, + "the registration ran first" + ); + assert_eq!( + restore_writes(&fake), + [( + JobStatus::Running, + JobStatus::Failed, + Some(job::MAINTENANCE.to_owned()) + )] + ); + assert_eq!(fake.removed_targets.lock().unwrap().len(), 1); + assert!( + fake.restore_job("MusicRestored").is_none(), + "the target and its record are gone" + ); + assert!(!fake.tables.lock().unwrap().contains_key("MusicRestored")); + assert!(fake.bulk_loads.lock().unwrap().is_empty()); + assert_eq!(runtime.free_slots(), 4); + // A FAILED record is a job end: the outcome beside the worker error. + let metrics = drained(&runtime); + assert_eq!( + metrics + .iter() + .filter(|(m, _, _)| *m == MetricName::RestoreJobs) + .map(|(_, op, sum)| (op.clone(), *sum)) + .collect::>(), + [("failed".to_owned(), 1.0)], + "{metrics:?}" + ); + assert!( + metrics + .iter() + .any(|(m, op, _)| *m == MetricName::WorkerErrorCount && op == "RestoreJob"), + "{metrics:?}" + ); + assert!( + metrics + .iter() + .any(|(m, op, _)| *m == MetricName::RestoreJobDuration && op == "failed"), + "{metrics:?}" + ); + + *fake.maintenance_held.lock().unwrap() = false; + *fake.maintenance_error.lock().unwrap() = Some("pool exhausted".to_owned()); + let err = restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap_err(); + assert!( + matches!(err, DynamoDbError::InternalServerError(_)), + "{err:?}" + ); + wait_for_jobs(&runtime).await; + let writes = restore_writes(&fake); + assert_eq!(writes.len(), 2); + assert!( + writes[1].2.as_deref().unwrap().contains("pool exhausted"), + "{writes:?}" + ); +} + +/// The registration's own refusals reach the client as the legacy +/// backend's do: a name taken under the registration is +/// `TableAlreadyExistsException`, a CREATING restore target +/// `TableInUseException`, a backup deleted under the lock +/// `BackupNotFoundException`, a backup in use `BackupInUseException`; the +/// slot is returned each time and no job runs. With no slot free the +/// request is `LimitExceededException` before any store read. +#[tokio::test] +async fn the_registrations_refusals_keep_their_classes_and_return_the_slot() { + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + let runtime = job_runtime(&mem); + type Expected = fn(&DynamoDbError) -> bool; + let cases: [(StorageError, Expected); 4] = [ + ( + StorageError::TableAlreadyExists("MusicRestored".to_owned()), + |e| matches!(e, DynamoDbError::TableAlreadyExistsException(_)), + ), + ( + StorageError::TableBeingRestored { + table_name: "MusicRestored".to_owned(), + backup_arn: arn(ACCOUNT), + }, + |e| matches!(e, DynamoDbError::TableInUseException(_)), + ), + ( + extenddb_storage::backup_catalog::backup_not_found(&arn(ACCOUNT)), + |e| matches!(e, DynamoDbError::BackupNotFoundException(_)), + ), + ( + extenddb_storage::backup_catalog::backup_in_use("Earlier", &arn(ACCOUNT)), + |e| matches!(e, DynamoDbError::BackupInUseException(_)), + ), + ]; + for (error, expected) in cases { + let shown = format!("{error:?}"); + *fake.claim_restore_error.lock().unwrap() = Some(error); + let err = restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap_err(); + assert!(expected(&err), "{shown}: {err:?}"); + wait_for_jobs(&runtime).await; + assert_eq!(runtime.free_slots(), 4, "{shown}"); + } + assert!(fake.bulk_loads.lock().unwrap().is_empty()); + assert!(restore_writes(&fake).is_empty()); + + let full = job_runtime(&mem); + let _held: Vec<_> = (0..4).map(|_| full.take_permit().unwrap()).collect(); + let before = mem.gets().len(); + let err = restore_on(&fake, &full, &restore_body(json!({}))) + .await + .unwrap_err(); + assert!( + matches!(err, DynamoDbError::LimitExceededException(_)), + "{err:?}" + ); + assert_eq!(mem.gets().len(), before, "no store read before the permit"); +} + +/// A second restore while the first's target is CREATING is +/// `BackupInUseException` naming the target, through the catalog's in-use +/// rule the registration wrote; a second restore into the same name is +/// `TableInUseException`; once the first is ACTIVE the backup is free again. +#[tokio::test] +async fn a_backup_being_restored_is_in_use_until_its_target_is_active() { + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + let runtime = job_runtime(&mem); + let gate = closed_gate(); + *fake.bulk_load_gate.lock().unwrap() = Some(Arc::clone(&gate)); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_bulk_loads_started(&fake, 1).await; + assert_eq!( + runtime.free_slots(), + 3, + "the job holds its slot past the reply, until it ends" + ); + let err = restore_on( + &fake, + &runtime, + &restore_body(json!({ "TargetTableName": "Second" })), + ) + .await + .unwrap_err(); + assert!( + matches!(err, DynamoDbError::BackupInUseException(ref m) if m.contains("MusicRestored")), + "{err:?}" + ); + assert_eq!(runtime.free_slots(), 3, "a refusal returns its own slot"); + let err = restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap_err(); + assert!( + matches!(err, DynamoDbError::TableInUseException(_)), + "{err:?}" + ); + gate.add_permits(1); + wait_for_jobs(&runtime).await; + assert_eq!( + fake.restore_job("MusicRestored").unwrap().status, + JobStatus::Done + ); + assert_eq!( + runtime.free_slots(), + 4, + "the slot is free once the job ended" + ); + *fake.bulk_load_gate.lock().unwrap() = None; + restore_on( + &fake, + &runtime, + &restore_body(json!({ "TargetTableName": "Second" })), + ) + .await + .unwrap(); + wait_for_jobs(&runtime).await; + assert_eq!(fake.restored_items("Second").len(), 2); +} + +/// The tamper cell: file 2 of 3 does not hash to the manifest's digest. +/// Every file is verified before any is loaded, so no `bulk_load` is +/// called at all, the record is FAILED naming the file, the target is +/// removed (the FAILED write matched), and the failed outcome is recorded. +#[tokio::test] +async fn a_corrupt_second_file_fails_before_any_load() { + let (lines, _guard) = LogLines::capture(); + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + let files = [songs_from(0, 3), songs_from(3, 3), songs_from(6, 3)]; + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &files, + true, + ); + let prefix = BackupArn::parse(&arn(ACCOUNT)).unwrap().store_prefix(); + let key = format!("{prefix}data/000002.json.gz"); + let mut bytes = mem.objects.lock().unwrap()[&key].clone(); + let last = bytes.len() - 1; + bytes[last] ^= 0xff; + mem.insert(&key, &bytes); + let runtime = job_runtime(&mem); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_jobs(&runtime).await; + assert!( + fake.bulk_loads.lock().unwrap().is_empty(), + "no file was loaded" + ); + let writes = restore_writes(&fake); + assert_eq!(writes.len(), 1); + assert_eq!( + (writes[0].0, writes[0].1), + (JobStatus::Running, JobStatus::Failed) + ); + let reason = writes[0].2.clone().unwrap(); + assert!( + reason.starts_with("data file data/000002.json.gz does not match the manifest: "), + "{reason}" + ); + assert_eq!(fake.removed_targets.lock().unwrap().len(), 1); + assert!(fake.restore_job("MusicRestored").is_none()); + assert!(!fake.tables.lock().unwrap().contains_key("MusicRestored")); + assert_eq!( + gets_under(&mem, &arn(ACCOUNT)).len(), + 4, + "sidecar, manifest, files 1 and 2" + ); + let metrics = drained(&runtime); + assert_eq!( + metrics + .iter() + .filter(|(m, _, _)| *m == MetricName::RestoreJobs) + .map(|(_, op, sum)| (op.clone(), *sum)) + .collect::>(), + [("failed".to_owned(), 1.0)] + ); + assert!( + metrics + .iter() + .any(|(m, op, sum)| *m == MetricName::WorkerErrorCount + && op == "RestoreJob" + && (*sum - 1.0).abs() < f64::EPSILON), + "a FAILED job is a worker error: {metrics:?}" + ); + let failed = lines.containing(": FAILED (data file data/000002.json.gz does not match"); + assert_eq!(failed.len(), 1, "{failed:?}"); + assert!( + failed[0].ends_with("the target was removed"), + "{}", + failed[0] + ); +} + +/// A transport failure at the `get` and one mid-body are each retried +/// after a backoff and the restore completes; the store's error is counted +/// once per failed attempt; a denial is not retried and fails the restore +/// naming the file; a file missing from the store fails it at once. +#[tokio::test] +async fn transport_failures_are_retried_and_other_failures_are_not() { + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + let files = [songs_from(0, 2), songs_from(2, 2)]; + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &files, + true, + ); + let runtime = job_runtime(&mem); + mem.fail_gets_with(GetFailure::Transport, 1); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + // The first data-file get failed; its retry, the next data-file get, + // stops mid-body. + let deadline = tokio::time::Instant::now() + Duration::from_secs(10); + while gets_under(&mem, &arn(ACCOUNT)).len() < 3 { + assert!(tokio::time::Instant::now() < deadline); + tokio::time::sleep(Duration::from_millis(5)).await; + } + mem.fail_get_after_bytes(5); + wait_for_jobs(&runtime).await; + assert_eq!( + fake.restore_job("MusicRestored").unwrap().status, + JobStatus::Done + ); + assert_eq!(fake.restored_items("MusicRestored").len(), 4); + let reads = gets_under(&mem, &arn(ACCOUNT)); + assert_eq!( + reads, + [ + MANIFEST_CHECKSUM_FILE, + EXTENDDB_MANIFEST_FILE, + "data/000001.json.gz", + "data/000001.json.gz", + "data/000001.json.gz", + "data/000002.json.gz", + "data/000001.json.gz", + "data/000002.json.gz", + ], + "file 1 took three attempts" + ); + // Summed across the drain's buckets: the two errors are 1 to 2 s apart + // and a run that crosses a wall-clock minute puts them in two. + let errors: Vec<_> = drained(&runtime) + .into_iter() + .filter(|(m, _, _)| *m == MetricName::BackupStoreErrors) + .collect(); + assert!(errors.iter().all(|(_, op, _)| op == "get"), "{errors:?}"); + let total: f64 = errors.iter().map(|(_, _, sum)| *sum).sum(); + assert!((total - 2.0).abs() < f64::EPSILON, "{errors:?}"); + + // A denial: no retry. + let fake = Arc::new(Fake::default()); + let mem = Arc::new(MemStore::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &files, + true, + ); + let runtime = job_runtime(&mem); + mem.fail_gets_with(GetFailure::PermissionDenied, 1); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_jobs(&runtime).await; + let writes = restore_writes(&fake); + assert_eq!(writes.len(), 1); + let reason = writes[0].2.clone().unwrap(); + assert!( + reason + .starts_with("reading data file data/000001.json.gz from the store: permission denied"), + "{reason}" + ); + assert_eq!(gets_under(&mem, &arn(ACCOUNT)).len(), 3); + let errors: Vec<_> = drained(&runtime) + .into_iter() + .filter(|(m, _, _)| *m == MetricName::BackupStoreErrors) + .collect(); + assert_eq!( + errors, + [(MetricName::BackupStoreErrors, "get".to_owned(), 1.0)], + "a denial is the store's error too" + ); + + // A file that is not there. + let fake = Arc::new(Fake::default()); + let mem = Arc::new(MemStore::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &files, + true, + ); + let prefix = BackupArn::parse(&arn(ACCOUNT)).unwrap().store_prefix(); + mem.objects + .lock() + .unwrap() + .remove(&format!("{prefix}data/000002.json.gz")); + let runtime = job_runtime(&mem); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_jobs(&runtime).await; + let writes = restore_writes(&fake); + assert_eq!( + writes[0].2.as_deref(), + Some("data file data/000002.json.gz is not in the store") + ); + assert!(fake.bulk_loads.lock().unwrap().is_empty()); + let errors: Vec<_> = drained(&runtime) + .into_iter() + .filter(|(m, _, _)| *m == MetricName::BackupStoreErrors) + .collect(); + assert_eq!( + errors, + [(MetricName::BackupStoreErrors, "get".to_owned(), 1.0)], + "an absent file is the store's error too" + ); +} + +/// A store that keeps failing is retried for the bound and then fails the +/// restore naming the file and the bound, with the record heartbeated +/// meanwhile and no `bulk_load`; the backoff never exceeds a minute. +#[tokio::test(start_paused = true)] +async fn a_download_exhausting_the_retry_bound_fails_the_restore() { + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + let runtime = job_runtime(&mem); + mem.fail_gets_with(GetFailure::Transport, usize::MAX); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + let started = tokio::time::Instant::now(); + // The paused clock runs ahead through the backoffs, so the job's end is + // waited for without `wait_for_jobs`'s wall-clock bound. + while runtime.running_jobs() > 0 { + tokio::time::sleep(Duration::from_millis(50)).await; + } + let elapsed = started.elapsed(); + assert!( + elapsed >= Duration::from_secs(15 * 60) && elapsed < Duration::from_secs(16 * 60 + 2), + "{elapsed:?}" + ); + let writes = restore_writes(&fake); + assert_eq!(writes.len(), 1); + let reason = writes[0].2.clone().unwrap(); + assert!( + reason.starts_with("downloading data/000001.json.gz: transport error"), + "{reason}" + ); + assert!(reason.contains("the bound is 900 s"), "{reason}"); + let attempts = gets_under(&mem, &arn(ACCOUNT)).len() - 2; + // 1+2+4+8+16+32 s, then 60 s each: about 20 attempts in 15 minutes. + assert!((18..=24).contains(&attempts), "{attempts} attempts"); + assert!(fake.bulk_loads.lock().unwrap().is_empty()); + assert!( + fake.heartbeats.load(std::sync::atomic::Ordering::Acquire) > 100, + "the record was heartbeated through the wait" + ); + assert_eq!(restore_outcomes(&runtime), [("failed".to_owned(), 1.0)]); +} + +/// The count after the last file: a backend that reports one item fewer +/// than the manifest (one of two rows with one key kept) fails the restore +/// with the count reason, after every file was loaded; and the duplicate +/// rule of the backend itself (an unordered backup holding one key in two +/// files) fails the second file's load with the backend's text. +#[tokio::test] +async fn a_duplicate_key_fails_the_restore_by_the_count_or_by_the_load() { + // The count runs on both orders. + for key_ordered in [false, true] { + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs_from(0, 2), songs_from(2, 2)], + key_ordered, + ); + let runtime = job_runtime(&mem); + *fake.count_override.lock().unwrap() = Some(3); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_jobs(&runtime).await; + assert_eq!( + fake.bulk_loads.lock().unwrap().len(), + 2, + "every file was loaded first ({key_ordered})" + ); + let writes = restore_writes(&fake); + assert_eq!( + writes[0].2.as_deref(), + Some("restored 3 items, the manifest lists 4: the backup holds a key twice"), + "{key_ordered}" + ); + assert!(fake.restore_job("MusicRestored").is_none()); + } + + let fake = Arc::new(Fake::default()); + let mem = Arc::new(MemStore::default()); + let mut second = songs_from(5, 2); + second.push(song("artist-00001", "title-00001")); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs_from(0, 3), second], + false, + ); + let runtime = job_runtime(&mem); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_jobs(&runtime).await; + assert_eq!(fake.bulk_loads.lock().unwrap().len(), 2); + let writes = restore_writes(&fake); + let reason = writes[0].2.clone().unwrap(); + assert!( + reason.starts_with("data file data/000002.json.gz: ") + && reason.contains("holds a primary key the target already has"), + "{reason}" + ); + assert!(fake.restore_job("MusicRestored").is_none()); + assert_eq!(fake.restored_items("MusicRestored").len(), 0); +} + +/// A key-ordered file whose first item is below its recorded range fails +/// the file through the reader's range check, naming the file; nothing of +/// the file is applied and the target is removed. +#[tokio::test] +async fn a_file_leaving_its_key_range_fails_the_restore() { + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + let files = [songs_from(0, 2), songs_from(2, 3)]; + let source_table = music_source_table(); + let parsed = BackupArn::parse(&arn(ACCOUNT)).unwrap(); + let prefix = parsed.store_prefix(); + let order = KeyOrder::resolve( + &source_table.key_schema, + &source_table.attribute_definitions, + ) + .unwrap(); + let mut tracker = KeyRangeTracker::new(&order); + let mut entries = Vec::new(); + for (i, items) in files.iter().enumerate() { + let (bytes, checksums) = encode_data_file(items.iter().cloned()).unwrap(); + for item in items { + tracker.push(item).unwrap(); + } + let mut range = tracker.next_file().unwrap(); + if i == 1 { + // The recorded range starts at the file's second item. + range.first = order.endpoint_key(&items[1]).unwrap(); + } + let key = data_file_key(u32::try_from(i + 1).unwrap()); + mem.insert(&format!("{prefix}{key}"), &bytes); + entries.push(( + DataFileEntry::from_checksums(key, &checksums, Some(range)), + None, + )); + } + seed_manifest_documents(&fake, &mem, &arn(ACCOUNT), source_table, entries, true); + let runtime = job_runtime(&mem); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_jobs(&runtime).await; + let loads = fake.bulk_loads.lock().unwrap().clone(); + assert_eq!(loads.len(), 2); + assert!(loads[1].stream_error.is_some(), "{loads:?}"); + let writes = restore_writes(&fake); + let reason = writes[0].2.clone().unwrap(); + assert!( + reason.starts_with("data file data/000002.json.gz: "), + "{reason}" + ); + assert!( + reason.contains("range") || reason.contains("key"), + "{reason}" + ); + assert!(fake.restore_job("MusicRestored").is_none()); +} + +/// An item the validator refuses (a `Year` of `1E126`, #618's shape) fails +/// the file before the backend sees it, naming the key and the attribute; +/// the backend's stream ended with the error and nothing was applied. +#[tokio::test] +async fn a_validator_refusal_fails_the_restore_naming_the_item() { + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + let mut second = songs_from(2, 1); + second.push(song_with_bad_number("artist-00003", "title-00003", "1E126")); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs_from(0, 2), second], + true, + ); + let runtime = job_runtime(&mem); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_jobs(&runtime).await; + let loads = fake.bulk_loads.lock().unwrap().clone(); + assert_eq!(loads.len(), 2); + assert_eq!(loads[1].items.len(), 0, "the chunk was never sent"); + let writes = restore_writes(&fake); + let reason = writes[0].2.clone().unwrap(); + assert!( + reason.starts_with( + "data file data/000002.json.gz: item cannot be restored: key \ + {\"Artist\":{\"S\":\"artist-00003\"},\"SongTitle\":{\"S\":\"title-00003\"}}, attribute Year: " + ), + "{reason}" + ); + assert_eq!( + loads[0].items.len(), + 2, + "file 1 was applied before file 2 failed" + ); + assert!( + fake.restore_job("MusicRestored").is_none(), + "the target was removed" + ); +} + +/// The FAILED-write gate: a record another owner took (its generation +/// changed) is not this job's to write, so a failing job makes no table +/// access: no `remove_restore_target`, the record untouched. An unknown +/// outcome is re-read: a write that landed is matched and the target is +/// removed with one write; one that was lost is sent once more; one that +/// was lost and whose record a successor then wrote FAILED under its own +/// generation is not this job's, so no removal. +#[tokio::test] +async fn a_failed_write_that_does_not_match_removes_no_target_and_an_unknown_one_is_re_read() { + let (lines, _guard) = LogLines::capture(); + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + *fake.heartbeat_answer.lock().unwrap() = HeartbeatAnswer::AlwaysConfirm; + // The job is held at its first `bulk_load`; the record changes hands + // meanwhile, so the load fails under the old generation and the FAILED + // write matches no record. + let gate = closed_gate(); + *fake.bulk_load_gate.lock().unwrap() = Some(Arc::clone(&gate)); + let runtime = job_runtime(&mem); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_bulk_loads_started(&fake, 1).await; + let other = { + let mut records = fake.restore_jobs.lock().unwrap(); + let record = records.get_mut("tid-MusicRestored").unwrap(); + record.owner_generation = new_generation(); + record.owner_generation + }; + gate.add_permits(10); + wait_for_jobs(&runtime).await; + let writes = restore_writes(&fake); + assert_eq!(writes.len(), 1); + assert_eq!(writes[0].1, JobStatus::Failed); + assert!( + fake.removed_targets.lock().unwrap().is_empty(), + "no table access" + ); + let record = fake.restore_job("MusicRestored").unwrap(); + assert_eq!(record.status, JobStatus::Running); + assert_eq!(record.owner_generation, other); + let line = lines.containing("no table access was made"); + assert_eq!(line.len(), 1, "{line:?}"); + + // A corrupt file, so the failure path runs with the claim intact. A + // write lost with its connection and followed by a takeover whose + // successor wrote FAILED itself re-reads FAILED under another + // generation: the record is not this job's, so no removal. + let prefix = BackupArn::parse(&arn(ACCOUNT)).unwrap().store_prefix(); + let key = format!("{prefix}data/000001.json.gz"); + { + let (lines, _guard) = LogLines::capture(); + let fake = Arc::new(Fake::default()); + let mem = Arc::new(MemStore::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + let mut bytes = mem.objects.lock().unwrap()[&key].clone(); + bytes[0] ^= 0xff; + mem.insert(&key, &bytes); + *fake.unknown_write.lock().unwrap() = Some(UnknownWrite::TakenAndWritten); + let runtime = job_runtime(&mem); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_jobs(&runtime).await; + assert_eq!(restore_writes(&fake).len(), 1, "no second write"); + assert!( + fake.removed_targets.lock().unwrap().is_empty(), + "no table access: the FAILED record is the successor's" + ); + assert!(fake.tables.lock().unwrap().contains_key("MusicRestored")); + let line = + lines.containing("the record is not this job's to write (the row carries generation"); + assert_eq!(line.len(), 1, "{line:?}"); + } + for (unknown, expected_writes) in [(UnknownWrite::Applied, 1), (UnknownWrite::Lost, 2)] { + let fake = Arc::new(Fake::default()); + let mem = Arc::new(MemStore::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + let mut bytes = mem.objects.lock().unwrap()[&key].clone(); + bytes[0] ^= 0xff; + mem.insert(&key, &bytes); + *fake.unknown_write.lock().unwrap() = Some(unknown); + let runtime = job_runtime(&mem); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_jobs(&runtime).await; + assert_eq!(restore_writes(&fake).len(), expected_writes, "{unknown:?}"); + assert_eq!(fake.removed_targets.lock().unwrap().len(), 1, "{unknown:?}"); + assert!(fake.restore_job("MusicRestored").is_none(), "{unknown:?}"); + } +} + +/// A lost heartbeat ends the job with no catalog write and no table access: +/// the job is held inside the first `bulk_load`, the claim is reported lost +/// at the next beat, and the job stops; the record stays RUNNING under its +/// generation, nothing is removed, the outcome is `failed`. +#[tokio::test] +async fn a_lost_heartbeat_stops_the_restore_with_no_catalog_write_and_no_table_access() { + let (lines, _guard) = LogLines::capture(); + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + let runtime = job_runtime(&mem); + let gate = closed_gate(); + *fake.bulk_load_gate.lock().unwrap() = Some(Arc::clone(&gate)); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + // The job reaches the gate; then the claim is lost at the next beat. + wait_for_bulk_loads_started(&fake, 1).await; + *fake.heartbeat_answer.lock().unwrap() = HeartbeatAnswer::Lost; + wait_for_jobs(&runtime).await; + assert!(restore_writes(&fake).is_empty(), "no catalog write"); + assert!( + fake.removed_targets.lock().unwrap().is_empty(), + "no table access" + ); + let record = fake.restore_job("MusicRestored").unwrap(); + assert_eq!(record.status, JobStatus::Running); + assert!(fake.restored_items("MusicRestored").is_empty()); + assert_eq!(restore_outcomes(&runtime), [("failed".to_owned(), 1.0)]); + // The stop line names the timer's reason, not the two-period clause. + let stopped = lines.containing( + "stopped with no further catalog or table access: the claim was lost: the row is gone \ + or carries another generation; the record is left to its owner or to the reconciler", + ); + assert_eq!(stopped.len(), 1, "{:?}", lines.containing("stopped with")); + let raised = lines.containing( + "the job's heartbeat raised the loss of its claim; the job stops at its next check: the \ + claim was lost: the row is gone or carries another generation", + ); + assert_eq!(raised.len(), 1, "{raised:?}"); + assert_eq!(lines.at_level(tracing::Level::WARN), raised); + assert_eq!(runtime.free_slots(), 4); +} + +/// A loss the timer raised while the job was inside a step (here the count) +/// is named by the next check between steps: the pre-flip check's stop line +/// carries the timer's reason, not the two-period clause. +#[tokio::test] +async fn a_loss_raised_inside_a_step_is_named_by_the_next_check() { + let (lines, _guard) = LogLines::capture(); + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + let runtime = job_runtime(&mem); + let count_gate = closed_gate(); + *fake.count_gate.lock().unwrap() = Some(Arc::clone(&count_gate)); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + let deadline = tokio::time::Instant::now() + Duration::from_secs(30); + while fake.count_calls.load(std::sync::atomic::Ordering::Acquire) == 0 { + assert!(tokio::time::Instant::now() < deadline, "no count began"); + tokio::time::sleep(Duration::from_millis(5)).await; + } + // The claim is lost at the next beat, while the job waits in the count. + *fake.heartbeat_answer.lock().unwrap() = HeartbeatAnswer::Lost; + while lines.containing("raised the loss of its claim").is_empty() { + assert!( + tokio::time::Instant::now() < deadline, + "the loss was not raised" + ); + tokio::time::sleep(Duration::from_millis(5)).await; + } + count_gate.add_permits(1); + wait_for_jobs(&runtime).await; + assert_eq!(fake.finish_sizes.lock().unwrap().len(), 0, "no DONE write"); + assert!(restore_writes(&fake).is_empty(), "no catalog write"); + assert_eq!( + fake.restore_job("MusicRestored").unwrap().status, + JobStatus::Running + ); + let stopped = lines.containing( + "stopped with no further catalog or table access: the claim was lost: the row is gone \ + or carries another generation; the record is left to its owner or to the reconciler", + ); + assert_eq!(stopped.len(), 1, "{:?}", lines.containing("stopped with")); + assert_eq!(restore_outcomes(&runtime), [("failed".to_owned(), 1.0)]); +} + +/// Shutdown between files: the job stops at its next file with no status +/// write, the record stays RUNNING with its progress, the spool entry stays +/// with every file verified, the outcome is `cancelled` and no worker +/// error is recorded; `drain_jobs` returns with every job ended. +#[tokio::test] +async fn a_shutdown_leaves_the_record_running_with_its_spool() { + let scratch = restore::spool::tests::Scratch::new(); + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + let s3_arn = arn(ACCOUNT); + let files = [songs_from(0, 2), songs_from(2, 2), songs_from(4, 2)]; + seed_manifest_backup(&fake, &mem, &s3_arn, music_source_table(), &files, true); + fake.rows.lock().unwrap().get_mut(&s3_arn).unwrap().store = s3_identity(); + let runtime = spooled_runtime(&mem, scratch.path()); + let gate = Arc::new(tokio::sync::Semaphore::new(1)); + *fake.bulk_load_gate.lock().unwrap() = Some(Arc::clone(&gate)); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + assert_eq!( + fake.restore_job("MusicRestored") + .unwrap() + .spool_host + .as_deref(), + Some("host-a:/var/spool") + ); + // File 1 loads; file 2 waits at the gate. + wait_for_bulk_loads_started(&fake, 2).await; + let left = runtime.drain_jobs(Duration::from_millis(100)).await; + assert_eq!(left, 1, "the job waits at the gate past the bound"); + gate.add_permits(10); + wait_for_jobs(&runtime).await; + assert!( + restore_writes(&fake).is_empty(), + "no status write at shutdown" + ); + let record = fake.restore_job("MusicRestored").unwrap(); + assert_eq!(record.status, JobStatus::Running); + assert!( + record.file_index.is_some(), + "the progress of the loaded file(s) stays" + ); + assert!(fake.removed_targets.lock().unwrap().is_empty()); + let entry = scratch.path().join("restore-tid-MusicRestored"); + assert!(entry.is_dir(), "the spool entry stays"); + for i in 0..3 { + assert!(entry.join(format!("{i:06}.json.gz")).is_file()); + assert!(entry.join(format!("{i:06}.json.gz.verified")).is_file()); + } + assert!(entry.join(EXTENDDB_MANIFEST_FILE).is_file()); + assert!(entry.join(MANIFEST_CHECKSUM_FILE).is_file()); + // One drain, both assertions read from it (a second drain reads an + // empty collector and asserts nothing). + let metrics = drained(&runtime); + assert_eq!( + metrics + .iter() + .filter(|(m, _, _)| *m == MetricName::RestoreJobs) + .map(|(_, op, sum)| (op.clone(), *sum)) + .collect::>(), + [("cancelled".to_owned(), 1.0)] + ); + assert!( + !metrics + .iter() + .any(|(m, op, _)| *m == MetricName::WorkerErrorCount && op == "RestoreJob"), + "a clean shutdown is not a worker error: {metrics:?}" + ); +} + +/// On an S3 store the files are spooled: each is downloaded once into the +/// entry, marked `.verified`, and the load reads the spool, not the store; +/// the entry is removed when the job ends; and a completed job's second +/// read of the store is never made. +#[tokio::test] +async fn an_s3_restore_spools_every_file_and_removes_its_entry_at_the_end() { + let scratch = restore::spool::tests::Scratch::new(); + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + let files = [songs_from(0, 2), songs_from(2, 2)]; + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &files, + true, + ); + fake.rows + .lock() + .unwrap() + .get_mut(&arn(ACCOUNT)) + .unwrap() + .store = s3_identity(); + let runtime = spooled_runtime(&mem, scratch.path()); + let gate = closed_gate(); + *fake.bulk_load_gate.lock().unwrap() = Some(Arc::clone(&gate)); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + let entry = scratch.path().join("restore-tid-MusicRestored"); + let deadline = tokio::time::Instant::now() + Duration::from_secs(10); + while !entry.join("000001.json.gz.verified").is_file() { + assert!(tokio::time::Instant::now() < deadline); + tokio::time::sleep(Duration::from_millis(5)).await; + } + #[cfg(unix)] + { + use std::os::unix::fs::PermissionsExt; + assert_eq!( + std::fs::metadata(&entry).unwrap().permissions().mode() & 0o777, + 0o700 + ); + assert_eq!( + std::fs::metadata(entry.join("000000.json.gz")) + .unwrap() + .permissions() + .mode() + & 0o777, + 0o600 + ); + } + gate.add_permits(10); + wait_for_jobs(&runtime).await; + assert_eq!( + fake.restore_job("MusicRestored").unwrap().status, + JobStatus::Done + ); + assert_eq!(fake.restored_items("MusicRestored").len(), 4); + assert_eq!( + gets_under(&mem, &arn(ACCOUNT)), + [ + MANIFEST_CHECKSUM_FILE, + EXTENDDB_MANIFEST_FILE, + "data/000001.json.gz", + "data/000002.json.gz", + ], + "each file read from the store once" + ); + assert!(!entry.exists(), "the entry is removed at the end"); +} + +/// A resumed job skips the files and items the record and the progress rows +/// account for: the first open file is loaded from its `start_at` with +/// `reapplying`, the rest from 0 without it, and the count covers the whole +/// manifest. +#[tokio::test] +async fn a_resume_loads_from_the_recorded_progress() { + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Arc::new(Fake::default()); + let files = [songs_from(0, 3), songs_from(3, 3), songs_from(6, 3)]; + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &files, + true, + ); + let runtime = job_runtime(&mem); + // A dead owner's record: file 0 complete, one item of file 1 applied, + // the target holding those four items. + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_jobs(&runtime).await; + { + let mut records = fake.restore_jobs.lock().unwrap(); + let record = records.get_mut("tid-MusicRestored").unwrap(); + record.status = JobStatus::Running; + record.completed_at = None; + record.file_index = Some(1); + record.items_applied = Some(1); + let mut targets = fake.restore_tables.lock().unwrap(); + let target = targets.get_mut("tid-MusicRestored").unwrap(); + target.items.truncate(4); + target.active = false; + } + fake.bulk_loads.lock().unwrap().clear(); + fake.restore_dead + .lock() + .unwrap() + .insert("tid-MusicRestored".to_owned()); + let report = cycle_on(&fake, &runtime).await; + assert_eq!(report.manifest_restores_resumed, ["MusicRestored"]); + wait_for_jobs(&runtime).await; + assert_eq!(fake.bulk_load_files(), [(1, 1, true, 2), (2, 0, false, 3)]); + let all: Vec = files.iter().flatten().cloned().collect(); + assert_eq!(fake.restored_items("MusicRestored"), all); + assert_eq!( + fake.restore_job("MusicRestored").unwrap().status, + JobStatus::Done + ); +} + +/// The vector refusal on the manifest path, narrowed to a target without +/// vector support: a manifest with a non-empty `vector_indexes` into a +/// target whose engine has no vector search is the legacy text's +/// `ValidationException` with nothing registered; the same manifest into a +/// target with vector support registers the table with its vector indexes; +/// `null` and `[]` register on both. +#[tokio::test] +async fn vector_index_backups_are_refused_in_the_engine() { + let spec: extenddb_core::types::VectorIndexSpecification = serde_json::from_value(json!({ + "IndexName": "vidx", + "VectorAttribute": { "AttributeName": "emb" }, + "Dimensions": 3, + "DistanceFunction": "COSINE", + "Projection": { "ProjectionType": "ALL" } + })) + .unwrap(); + let cases: [( + Option>, + bool, + bool, + ); 6] = [ + (Some(vec![spec.clone()]), false, false), + (Some(vec![spec.clone()]), true, true), + (None, false, true), + (None, true, true), + (Some(Vec::new()), false, true), + (Some(Vec::new()), true, true), + ]; + for (vector_indexes, capable, registered) in cases { + let what = format!("{vector_indexes:?} into a target with vector support {capable}"); + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + fake.vector_search + .store(capable, std::sync::atomic::Ordering::Release); + let mut source_table = on_demand_source_table(); + source_table.vector_indexes = vector_indexes.clone(); + seed_manifest_backup(&fake, &mem, &arn(ACCOUNT), source_table, &[songs(1)], true); + let runtime = job_runtime(&mem); + let result = restore_on(&fake, &runtime, &restore_body(json!({}))).await; + wait_for_jobs(&runtime).await; + if registered { + result.unwrap_or_else(|e| panic!("{what}: {e:?}")); + let claims = fake.restore_claims.lock().unwrap(); + assert_eq!(claims.len(), 1, "{what}"); + let expected = vector_indexes.filter(|v| !v.is_empty()); + assert_eq!(claims[0].1.vector_indexes, expected, "{what}"); + } else { + let msg = validation_message(result.unwrap_err()); + assert_eq!( + msg, + format!( + "backup {} has 1 vector index(es); restoring a table with vector indexes is \ + not supported by this storage backend", + arn(ACCOUNT) + ), + "{what}" + ); + assert!(fake.restore_claims.lock().unwrap().is_empty(), "{what}"); + } + assert_eq!(runtime.free_slots(), 4, "{what}"); + } +} + +/// The legacy reader's vector refusals stay as they were: a definition +/// naming a vector index and a pre-0.0.4 source counting one. +#[tokio::test] +async fn legacy_vector_backups_keep_the_reader_refusal() { + let mut def = definition("PAY_PER_REQUEST"); + def.vector_index_names.push("v".to_owned()); + let fake = Fake::with_source(source(Some(def))); + let msg = validation_message(restore(&fake, restore_body(json!({}))).await.unwrap_err()); + assert!(msg.contains("vector index"), "{msg}"); + assert!(fake.restores().is_empty()); + + let mut legacy = source(None); + legacy.legacy_vector_index_count = 1; + let fake = Fake::with_source(legacy); + let msg = validation_message(restore(&fake, restore_body(json!({}))).await.unwrap_err()); + assert!(msg.contains("vector index"), "{msg}"); + assert!(fake.restores().is_empty()); +} + +/// A multi-part-key table (two HASH attributes, backed up unordered) +/// registers on the manifest path when the target's limits allow the shape: +/// `ensure_single_part_base_key` is the legacy reader's rule alone. +#[tokio::test] +async fn a_multi_part_key_manifest_is_registered() { + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + let mut source_table = music_source_table(); + source_table.key_schema = vec![ + key("Artist", KeyType::Hash), + key("Label", KeyType::Hash), + key("SongTitle", KeyType::Range), + ]; + source_table.attribute_definitions.push(attr("Label")); + source_table + .attribute_definitions + .retain(|a| a.attribute_name != "Genre"); + source_table.global_secondary_indexes.clear(); + let mut items = songs(2); + for item in &mut items { + item.insert( + "Label".to_owned(), + extenddb_core::types::AttributeValue::S("indie".to_owned()), + ); + } + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + source_table, + &[items.clone()], + false, + ); + let runtime = job_runtime(&mem); + let limits = LimitsConfig { + allow_multipart_table_keys: true, + ..LimitsConfig::default() + }; + let (_, response) = restore_table_from_backup( + &fake, + &runtime, + ACCOUNT, + REGION, + &limits, + &restore_body(json!({})), + ) + .await + .unwrap(); + assert_eq!(response["TableDescription"]["TableStatus"], "CREATING"); + wait_for_jobs(&runtime).await; + let claims = fake.restore_claims.lock().unwrap(); + assert_eq!(claims[0].1.key_schema.len(), 3); + assert_eq!(fake.restored_items("MusicRestored"), items); +} + +/// The override contract on the manifest path is the legacy path's: +/// `restored_input` runs every case through both and asserts they agree, +/// so one list of cases binds both (the override tests above). This cell +/// pins the comparison itself with a case of each kind: an applied +/// override and a refused one. +#[tokio::test] +async fn the_override_cases_run_through_both_paths() { + let input = restored_input( + source(Some(definition("PAY_PER_REQUEST"))), + json!({ + "BillingModeOverride": "PROVISIONED", + "ProvisionedThroughputOverride": { "ReadCapacityUnits": 11, "WriteCapacityUnits": 12 }, + "GlobalSecondaryIndexOverride": [gsi_entry(Some((3, 4)))], + }), + ) + .await + .unwrap(); + assert_eq!(input.billing_mode, Some(BillingMode::Provisioned)); + assert_eq!(input.provisioned_throughput, Some(throughput(11, 12))); + let err = restored_input( + source(Some(definition("PAY_PER_REQUEST"))), + json!({ "ProvisionedThroughputOverride": { "ReadCapacityUnits": 1, "WriteCapacityUnits": 1 } }), + ) + .await + .unwrap_err(); + assert!( + matches!(err, DynamoDbError::ValidationException(_)), + "{err:?}" + ); +} + +// --- the reconciler's restore duties ------------------------------------------- + +/// A dead RUNNING manifest record of the configured store whose owner was +/// on another host: the reconciler takes it over (a fresh generation, this +/// host's `spool_host`, `heartbeat_at` refreshed so a second taker finds the +/// record live) and spawns the resume, which reads the manifest from the +/// store, removes the entry this host's earlier run of the job left (its +/// files were verified under a record that has since been another host's), +/// downloads every file again and finishes the restore from the recorded +/// progress; a live record is left alone. +#[tokio::test] +async fn the_reconciler_resumes_a_dead_manifest_restore_with_its_spool_host() { + let (lines, _guard) = LogLines::capture(); + let scratch = restore::spool::tests::Scratch::new(); + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Arc::new(Fake::default()); + let files = [songs_from(0, 2), songs_from(2, 2)]; + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &files, + true, + ); + fake.rows + .lock() + .unwrap() + .get_mut(&arn(ACCOUNT)) + .unwrap() + .store = s3_identity(); + let runtime = spooled_runtime(&mem, scratch.path()); + // A record left by a dead owner on another host, file 0 applied. + let gate = Arc::new(tokio::sync::Semaphore::new(1)); + *fake.bulk_load_gate.lock().unwrap() = Some(Arc::clone(&gate)); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_bulk_loads_started(&fake, 2).await; + runtime.drain_jobs(Duration::from_millis(100)).await; + gate.add_permits(10); + wait_for_jobs(&runtime).await; + let before = fake.restore_job("MusicRestored").unwrap(); + assert_eq!(before.status, JobStatus::Running); + assert_eq!( + before.file_index, + Some(1), + "file 1 was in its load when the shutdown fired" + ); + *fake.bulk_load_gate.lock().unwrap() = None; + fake.bulk_loads.lock().unwrap().clear(); + let entry = scratch.path().join("restore-tid-MusicRestored"); + + // A live owner on this host: nothing happens, and the sweep keeps the + // entry of a RUNNING record that names this host. + let runtime2 = spooled_runtime(&mem, scratch.path()); + let report = cycle_on(&fake, &runtime2).await; + assert!(report.manifest_restores_resumed.is_empty()); + assert!(fake.restore_takeovers.lock().unwrap().is_empty()); + assert!(report.spool_entries_swept.is_empty()); + assert!(entry.join("000000.json.gz.verified").is_file()); + assert!(entry.join("000001.json.gz.verified").is_file()); + + { + // As a backend whose file 1 write did not land would have left it: + // the record at file 0, the target holding file 0's items, the + // owner on another host (the record moved there and its job died), + // its heartbeat old. This host's entry from the stopped job still + // holds both files verified; it is not the record's and the resume + // removes it before use (the sweep would, a cycle later). + let mut records = fake.restore_jobs.lock().unwrap(); + let record = records.get_mut("tid-MusicRestored").unwrap(); + record.spool_host = Some("host-b:/var/spool".to_owned()); + record.heartbeat_at = 1234.5; + record.file_index = Some(0); + record.items_applied = Some(2); + fake.restore_tables + .lock() + .unwrap() + .get_mut("tid-MusicRestored") + .unwrap() + .items + .truncate(2); + } + mem.gets.lock().unwrap().clear(); + fake.restore_dead + .lock() + .unwrap() + .insert("tid-MusicRestored".to_owned()); + let report = cycle_on(&fake, &runtime2).await; + assert_eq!(report.manifest_restores_resumed, ["MusicRestored"]); + assert!(report.acted()); + let after = fake.restore_job("MusicRestored").unwrap(); + assert_ne!(after.owner_generation, before.owner_generation); + assert_eq!(after.spool_host.as_deref(), Some("host-a:/var/spool")); + assert!( + after.heartbeat_at > 1234.5, + "the takeover refreshed the heartbeat: {}", + after.heartbeat_at + ); + assert_eq!( + fake.restore_takeovers.lock().unwrap().as_slice(), + [( + "tid-MusicRestored".to_owned(), + Some("host-a:/var/spool".to_owned()) + )] + ); + wait_for_jobs(&runtime2).await; + assert_eq!(fake.bulk_load_files(), [(1, 0, true, 2)]); + let all: Vec = files.iter().flatten().cloned().collect(); + assert_eq!(fake.restored_items("MusicRestored"), all); + assert_eq!( + fake.restore_job("MusicRestored").unwrap().status, + JobStatus::Done + ); + assert_eq!( + gets_under(&mem, &arn(ACCOUNT)), + [ + EXTENDDB_MANIFEST_FILE, + MANIFEST_CHECKSUM_FILE, + "data/000001.json.gz", + "data/000002.json.gz", + ], + "the manifest from the store, every file downloaded again" + ); + assert_eq!( + lines + .containing( + "has no live owner; taken over by this instance and resumed with 2 items of data \ + file 0 recorded as applied" + ) + .len(), + 1, + "the reconciler's line states the record's progress" + ); + assert_eq!( + lines + .containing("started, 2 data file(s), 4 items, key-ordered; resumed from data file 1 after 0 of its items") + .len(), + 1, + "the job's line names the first open file" + ); + assert_eq!( + lines + .containing("resumed by this instance; the manifest was read from the store") + .len(), + 1 + ); + assert_eq!( + lines + .containing("is left from a run whose record moved to another host") + .len(), + 1 + ); + assert!(!entry.exists()); +} + +/// The manifest arm's refusals: no free permit skips the record to the next +/// cycle with no takeover; a held maintenance claim fails it `OWNER_DEAD` +/// and removes its target like a legacy record; a record of another store +/// is FAILED naming `--move-store` and its target left to the FAILED-target +/// duty, which removes it next cycle; an unhealthy store skips it. +#[tokio::test] +async fn the_reconciler_skips_or_fails_a_dead_manifest_restore_it_cannot_resume() { + let (lines, _guard) = LogLines::capture(); + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + let runtime = job_runtime(&mem); + let gate = closed_gate(); + *fake.bulk_load_gate.lock().unwrap() = Some(Arc::clone(&gate)); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_bulk_loads_started(&fake, 1).await; + runtime.drain_jobs(Duration::from_millis(50)).await; + gate.add_permits(10); + wait_for_jobs(&runtime).await; + *fake.bulk_load_gate.lock().unwrap() = None; + fake.restore_dead + .lock() + .unwrap() + .insert("tid-MusicRestored".to_owned()); + + // No permit. + let full = job_runtime(&mem); + let held: Vec<_> = (0..4).map(|_| full.take_permit().unwrap()).collect(); + let report = cycle_on(&fake, &full).await; + assert_eq!(report.restores_skipped_no_permit, ["MusicRestored"]); + assert!(fake.restore_takeovers.lock().unwrap().is_empty()); + assert!(!report.acted()); + drop(held); + + // The store unhealthy: the cycle's own check, which runs first, finds + // the store down. + mem.set_down(true); + let unhealthy = runtime_unhealthy(&mem, StoreProblem::Transport("down".to_owned())) + .with_job_settings(job_settings()); + let report = cycle_on(&fake, &unhealthy).await; + assert_eq!(report.restores_skipped_store_unhealthy, ["MusicRestored"]); + assert_eq!(report.store_healthy, Some(false)); + assert_eq!( + lines.containing("dead store-backed restores are not resumed this cycle"), + [ + "the configured backup store is unhealthy; dead store-backed restores are not \ + resumed this cycle" + ] + ); + assert!(fake.restore_takeovers.lock().unwrap().is_empty()); + mem.set_down(false); + + // The maintenance claim held: counted once, as failed and removed. + *fake.maintenance_held.lock().unwrap() = true; + let report = cycle_on(&fake, &full).await; + assert_eq!( + report.manifest_restores_failed_and_removed, + ["MusicRestored"] + ); + assert!(report.manifest_restores_failed.is_empty()); + assert!(report.legacy_restores_removed.is_empty()); + assert!(report.acted()); + assert!(fake.restore_job("MusicRestored").is_none()); + assert!(!fake.tables.lock().unwrap().contains_key("MusicRestored")); + *fake.maintenance_held.lock().unwrap() = false; + + // Another store. + let mem2 = Arc::new(MemStore::default()); + mem2.insert(STORE_IDENTITY_KEY, &identity_file("store-2", "catalog-1")); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem2, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + let runtime = job_runtime(&mem2); + let gate = closed_gate(); + *fake.bulk_load_gate.lock().unwrap() = Some(Arc::clone(&gate)); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_bulk_loads_started(&fake, 1).await; + runtime.drain_jobs(Duration::from_millis(50)).await; + gate.add_permits(10); + wait_for_jobs(&runtime).await; + fake.restore_dead + .lock() + .unwrap() + .insert("tid-MusicRestored".to_owned()); + let other = runtime_on( + &mem2, + StoreIdentity { + store_id: "store-2".to_owned(), + ..store() + }, + ) + .with_job_settings(job_settings()); + let report = cycle_on(&fake, &other).await; + assert_eq!( + report.manifest_restores_failed, + [("MusicRestored".to_owned(), OTHER_STORE.to_owned())] + ); + let record = fake.restore_job("MusicRestored").unwrap(); + assert_eq!(record.status, JobStatus::Failed); + assert_eq!(record.failure_reason.as_deref(), Some(OTHER_STORE)); + assert!( + fake.tables.lock().unwrap().contains_key("MusicRestored"), + "the target waits for the FAILED-target duty" + ); + assert!(fake.restore_takeovers.lock().unwrap().is_empty()); + let report = cycle_on(&fake, &other).await; + assert_eq!(report.failed_targets_removed, ["MusicRestored"]); + assert!(!fake.tables.lock().unwrap().contains_key("MusicRestored")); +} + +/// A dead record with no progress at all whose target already holds items +/// is FAILED `target_not_empty` under the takeover and its target left in +/// place; the FAILED-target duty skips it on the next cycle and logs the +/// operator's command; a record with no progress into an empty target is +/// resumed from the first file. +#[tokio::test] +async fn a_resume_without_progress_needs_an_empty_target() { + let (lines, _guard) = LogLines::capture(); + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let runtime = job_runtime(&mem); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_jobs(&runtime).await; + // The record as a dead owner that registered and recorded nothing left + // it, over a target that holds an item the record does not account for. + dead_record_without_progress(&fake, true); + fake.bulk_loads.lock().unwrap().clear(); + let runtime = job_runtime(&mem); + let report = cycle_on(&fake, &runtime).await; + assert_eq!( + report.manifest_restores_failed, + [("MusicRestored".to_owned(), TARGET_NOT_EMPTY.to_owned())] + ); + assert_eq!(fake.restore_takeovers.lock().unwrap().len(), 1); + let record = fake.restore_job("MusicRestored").unwrap(); + assert_eq!(record.status, JobStatus::Failed); + assert_eq!(record.failure_reason.as_deref(), Some(TARGET_NOT_EMPTY)); + assert!( + fake.tables.lock().unwrap().contains_key("MusicRestored"), + "the target is left" + ); + wait_for_jobs(&runtime).await; + assert!(fake.bulk_loads.lock().unwrap().is_empty()); + // The FAILED-target duty leaves it to the operator. + let report = cycle_on(&fake, &runtime).await; + assert_eq!(report.targets_left_for_operator, ["MusicRestored"]); + assert!(report.failed_targets_removed.is_empty()); + assert!(fake.tables.lock().unwrap().contains_key("MusicRestored")); + let command = lines.containing(&format!( + "extenddb backup remove-target --account {ACCOUNT} MusicRestored" + )); + assert_eq!( + command.len(), + 2, + "the takeover's line and the duty's: {command:?}" + ); + + // An empty target with no progress is resumed from the first file. + let fake = Arc::new(Fake::default()); + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + let runtime = job_runtime(&mem); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_jobs(&runtime).await; + dead_record_without_progress(&fake, false); + fake.bulk_loads.lock().unwrap().clear(); + let runtime = job_runtime(&mem); + let report = cycle_on(&fake, &runtime).await; + assert_eq!(report.manifest_restores_resumed, ["MusicRestored"]); + wait_for_jobs(&runtime).await; + assert_eq!(fake.bulk_load_files(), [(0, 0, true, 2)]); + assert_eq!( + fake.restore_job("MusicRestored").unwrap().status, + JobStatus::Done + ); +} + +/// Under the maintenance claim a dead record with no progress at all whose +/// target holds items is FAILED `target_not_empty` through the unclaimed +/// write and its target left for the operator, not `OWNER_DEAD` with the +/// target removed: the command's wait is satisfied (the record is no longer +/// RUNNING) and `remove-target`, under the same claim, removes it; one +/// whose target is empty is `OWNER_DEAD` and removed, as before. +#[tokio::test] +async fn a_dead_record_without_progress_under_the_maintenance_claim_keeps_its_target() { + let (lines, _guard) = LogLines::capture(); + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + let runtime = job_runtime(&mem); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_jobs(&runtime).await; + dead_record_without_progress(&fake, true); + *fake.maintenance_held.lock().unwrap() = true; + let report = cycle_on(&fake, &runtime).await; + assert_eq!( + report.manifest_restores_failed, + [("MusicRestored".to_owned(), TARGET_NOT_EMPTY.to_owned())] + ); + assert!(report.manifest_restores_failed_and_removed.is_empty()); + let record = fake.restore_job("MusicRestored").unwrap(); + assert_eq!(record.status, JobStatus::Failed); + assert_eq!(record.failure_reason.as_deref(), Some(TARGET_NOT_EMPTY)); + assert!( + fake.tables.lock().unwrap().contains_key("MusicRestored"), + "the target is left" + ); + assert!(fake.removed_targets.lock().unwrap().is_empty()); + assert!( + fake.restore_takeovers.lock().unwrap().is_empty(), + "no takeover under the claim" + ); + assert_eq!( + fake.failed_jobs.lock().unwrap().as_slice(), + [(record.target.clone(), TARGET_NOT_EMPTY.to_owned())], + "the unclaimed write" + ); + assert_eq!( + lines + .containing(&format!( + "extenddb backup remove-target --account {ACCOUNT} MusicRestored" + )) + .len(), + 1 + ); + // The command removes it under the same claim. + let removed = maintenance::remove_target( + &*fake, + &remove_target_request("MusicRestored", Duration::from_secs(300)), + ) + .await + .unwrap(); + assert_eq!(removed.failure_reason.as_deref(), Some(TARGET_NOT_EMPTY)); + assert!(!fake.tables.lock().unwrap().contains_key("MusicRestored")); + + // An empty target: `OWNER_DEAD` and removed. + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + let runtime = job_runtime(&mem); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_jobs(&runtime).await; + dead_record_without_progress(&fake, false); + *fake.maintenance_held.lock().unwrap() = true; + let report = cycle_on(&fake, &runtime).await; + assert_eq!( + report.manifest_restores_failed_and_removed, + ["MusicRestored"] + ); + assert!(report.manifest_restores_failed.is_empty()); + assert!(fake.restore_job("MusicRestored").is_none()); + assert!(!fake.tables.lock().unwrap().contains_key("MusicRestored")); +} + +/// Once the server's shutdown has begun (the jobs' token is cancelled) a +/// cycle takes no dead record over, writes no FAILED on the maintenance or +/// other-store arms, and spawns nothing: the record stays RUNNING under its +/// owner's generation with its heartbeat as it was, for the next instance's +/// reconciler; the cycle's other duties still run. +#[tokio::test] +async fn the_reconciler_takes_nothing_over_once_the_shutdown_began() { + let (lines, _guard) = LogLines::capture(); + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + let runtime = job_runtime(&mem); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_jobs(&runtime).await; + dead_record_without_progress(&fake, false); + let before = fake.restore_job("MusicRestored").unwrap(); + let checks_before = fake.maintenance_checks.lock().unwrap().len(); + let shutdown = extenddb_storage::CancellationToken::new(); + let draining = job_runtime(&mem).with_shutdown(shutdown.clone()); + shutdown.cancel(); + + // The takeover arm, the maintenance arm and the other-store arm, each + // with the token cancelled. + let report = cycle_on(&fake, &draining).await; + assert!(report.manifest_restores_resumed.is_empty(), "{report:?}"); + assert!(!report.acted(), "{report:?}"); + *fake.maintenance_held.lock().unwrap() = true; + let report = cycle_on(&fake, &draining).await; + assert!( + report.manifest_restores_failed_and_removed.is_empty(), + "{report:?}" + ); + assert!(report.manifest_restores_failed.is_empty(), "{report:?}"); + *fake.maintenance_held.lock().unwrap() = false; + let mem2 = Arc::new(MemStore::default()); + mem2.insert(STORE_IDENTITY_KEY, &identity_file("store-2", "catalog-1")); + let other = runtime_on( + &mem2, + StoreIdentity { + store_id: "store-2".to_owned(), + ..store() + }, + ) + .with_job_settings(job_settings()) + .with_shutdown(shutdown.clone()); + let report = cycle_on(&fake, &other).await; + assert!(report.manifest_restores_failed.is_empty(), "{report:?}"); + + assert!( + fake.restore_takeovers.lock().unwrap().is_empty(), + "no takeover" + ); + assert!( + fake.failed_jobs.lock().unwrap().is_empty(), + "no FAILED write" + ); + assert!(restore_writes(&fake).is_empty()); + assert_eq!(draining.running_jobs(), 0, "nothing spawned"); + assert_eq!(fake.restore_job("MusicRestored").unwrap(), before); + assert_eq!( + lines + .containing("has no live owner and the server is shutting down; it is left for the next reconciler") + .len(), + 3 + ); + assert_eq!( + fake.maintenance_checks.lock().unwrap().len() - checks_before, + 3, + "each cycle ran its other duties" + ); + + // The same runtime with the token intact takes it over. + let runtime = job_runtime(&mem); + let report = cycle_on(&fake, &runtime).await; + assert_eq!(report.manifest_restores_resumed, ["MusicRestored"]); + wait_for_jobs(&runtime).await; + assert_eq!( + fake.restore_job("MusicRestored").unwrap().status, + JobStatus::Done + ); +} + +/// Rewrite the finished restore into `MusicRestored` as a dead owner's +/// RUNNING record with no progress recorded, its target CREATING and +/// holding one unaccounted item when `with_item`, else empty. +fn dead_record_without_progress(fake: &Fake, with_item: bool) { + let mut records = fake.restore_jobs.lock().unwrap(); + let record = records.get_mut("tid-MusicRestored").unwrap(); + record.status = JobStatus::Running; + record.completed_at = None; + record.file_index = None; + record.items_applied = None; + let mut targets = fake.restore_tables.lock().unwrap(); + let target = targets.get_mut("tid-MusicRestored").unwrap(); + target.active = false; + target.items.clear(); + if with_item { + target.items.push(song("x", "y")); + } + if let Some(desc) = fake.tables.lock().unwrap().get_mut("MusicRestored") { + desc.table_status = TableStatus::Creating; + } + fake.restore_dead + .lock() + .unwrap() + .insert("tid-MusicRestored".to_owned()); +} + +/// The two sweeps: progress rows of tables that are not RUNNING restore +/// targets go, a RUNNING target's stay; spool entries of restores not +/// RUNNING with this host's `spool_host` go (a dead one, one taken over by +/// another host), a live one of this host and every other name stay; both +/// run only without the maintenance claim; a backend that cannot list the +/// records skips both and the cycle goes on. +#[tokio::test] +async fn the_reconciler_sweeps_progress_rows_and_spool_entries() { + let scratch = restore::spool::tests::Scratch::new(); + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Arc::new(Fake::default()); + let runtime = spooled_runtime(&mem, scratch.path()); + for (id, host) in [ + ("live", "host-a:/var/spool"), + ("moved", "host-b:/var/spool"), + ] { + fake.restore_jobs.lock().unwrap().insert( + id.to_owned(), + RestoreJobRow { + target: RestoreTargetKey { + account_id: ACCOUNT.to_owned(), + table_name: id.to_owned(), + table_id: id.to_owned(), + }, + source_backup_arn: arn(ACCOUNT), + kind: RestoreKind::Manifest, + status: JobStatus::Running, + owner_generation: 1, + heartbeat_at: seconds_since_epoch(), + store: Some(s3_identity()), + spool_host: Some(host.to_owned()), + file_index: None, + items_applied: None, + failure_reason: None, + completed_at: None, + }, + ); + } + { + let mut rows = fake.progress_rows.lock().unwrap(); + rows.insert("live".to_owned(), vec![(0, 10)]); + rows.insert("gone".to_owned(), vec![(0, 10), (1, 5)]); + } + for name in ["restore-live", "restore-moved", "restore-dead", "other"] { + std::fs::create_dir(scratch.path().join(name)).unwrap(); + } + std::fs::write(scratch.path().join(".lock"), b"").unwrap(); + + *fake.maintenance_held.lock().unwrap() = true; + let report = cycle_on(&fake, &runtime).await; + assert_eq!(report.progress_rows_swept, 0); + assert!(report.spool_entries_swept.is_empty()); + assert!(fake.progress_sweeps.lock().unwrap().is_empty()); + assert!(scratch.path().join("restore-dead").is_dir()); + *fake.maintenance_held.lock().unwrap() = false; + + let report = cycle_on(&fake, &runtime).await; + assert_eq!(report.progress_rows_swept, 1); + let mut swept = report.spool_entries_swept.clone(); + swept.sort(); + assert_eq!(swept, ["dead", "moved"]); + assert!(report.acted()); + assert_eq!( + fake.progress_sweeps.lock().unwrap().as_slice(), + [vec!["live".to_owned(), "moved".to_owned()]], + "the RUNNING targets' ids, in id order" + ); + assert_eq!( + fake.progress_rows + .lock() + .unwrap() + .keys() + .cloned() + .collect::>(), + ["live"] + ); + let mut left: Vec = std::fs::read_dir(scratch.path()) + .unwrap() + .map(|e| e.unwrap().file_name().into_string().unwrap()) + .collect(); + left.sort(); + assert_eq!(left, [".lock", "other", "restore-live"]); + assert_eq!( + fake.restore_lists.lock().unwrap().as_slice(), + [Some(JobStatus::Running)] + ); +} + +/// The sweeps on a backend whose records cannot be listed yet: the cycle +/// goes on, logs once, and sweeps nothing. +#[tokio::test] +async fn a_backend_that_cannot_list_restore_records_skips_the_sweeps() { + let (lines, _guard) = LogLines::capture(); + let fake = Arc::new(Fake::default()); + fake.progress_rows + .lock() + .unwrap() + .insert("gone".to_owned(), vec![(0, 1)]); + fake.restore_list_error + .store(true, std::sync::atomic::Ordering::Release); + let report = cycle_on(&fake, &runtime()).await; + assert_eq!(report.progress_rows_swept, 0); + assert_eq!( + lines + .containing("the restore job records could not be listed") + .len(), + 1 + ); + assert_eq!(fake.progress_rows.lock().unwrap().len(), 1, "nothing swept"); +} + +// --- the restore's liveness checks, spool rule and load bounds ----------------- + +/// The heartbeat's first confirmed beat is the instant `claim_restore` +/// returned, not the instant before it: a registration that takes four +/// periods answers CREATING to a job that runs to DONE, where a clock taken +/// before the registration would end it at its first check as a lost claim. +#[tokio::test] +async fn a_slow_registration_does_not_end_the_job_at_its_first_check() { + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + let period = job_settings().heartbeat_interval; + *fake.claim_restore_delay.lock().unwrap() = Some(4 * period); + let runtime = job_runtime(&mem); + let response = restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + assert_eq!(response["TableDescription"]["TableStatus"], "CREATING"); + wait_for_jobs(&runtime).await; + assert_eq!( + fake.restore_job("MusicRestored").unwrap().status, + JobStatus::Done + ); + assert_eq!(fake.restored_items("MusicRestored").len(), 2); + assert_eq!(restore_outcomes(&runtime), [("succeeded".to_owned(), 1.0)]); +} + +/// The job's first beat, sent right after the registration and before the +/// reply, refreshes the record's `heartbeat_at`: a registration that +/// stamped it three periods in the past (a transaction-start `now()` behind +/// a lock, as on PostgreSQL) answers CREATING with the record's heartbeat of +/// this instant, one beat received before the reply, and the job runs to +/// DONE; a first beat that finds the claim lost fails the record before the +/// job starts, with the reason, and answers `InternalServerError`. On a +/// paused clock, so no timer beat runs between the reply and the read of +/// the record. +#[tokio::test(start_paused = true)] +async fn the_jobs_first_beat_refreshes_the_registrations_heartbeat_before_the_reply() { + let (lines, _guard) = LogLines::capture(); + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + let period = job_settings().heartbeat_interval; + *fake.registration_heartbeat_age.lock().unwrap() = Some(3.0 * period.as_secs_f64() + 10.0); + let runtime = job_runtime(&mem); + let gate = closed_gate(); + *fake.bulk_load_gate.lock().unwrap() = Some(Arc::clone(&gate)); + let response = restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + assert_eq!(response["TableDescription"]["TableStatus"], "CREATING"); + let age = seconds_since_epoch() - fake.restore_job("MusicRestored").unwrap().heartbeat_at; + assert!( + age < 1.0, + "the record's heartbeat is this instant's by the reply, not the registration's: {age}" + ); + assert!( + fake.heartbeats.load(std::sync::atomic::Ordering::Acquire) >= 1, + "one beat before the reply" + ); + gate.add_permits(10); + wait_for_jobs(&runtime).await; + assert_eq!( + fake.restore_job("MusicRestored").unwrap().status, + JobStatus::Done + ); + assert_eq!(restore_outcomes(&runtime), [("succeeded".to_owned(), 1.0)]); + + // The first beat finds the claim lost. + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + *fake.heartbeat_answer.lock().unwrap() = HeartbeatAnswer::Lost; + let runtime = job_runtime(&mem); + let err = restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap_err(); + assert!( + matches!(err, DynamoDbError::InternalServerError(_)), + "{err:?}" + ); + wait_for_jobs(&runtime).await; + assert_eq!( + restore_writes(&fake), + [( + JobStatus::Running, + JobStatus::Failed, + Some("the claim was lost before the job started".to_owned()) + )] + ); + assert!(fake.bulk_loads.lock().unwrap().is_empty(), "no load"); + assert_eq!( + lines + .containing("FAILED (the claim was lost before the job started) before it started;") + .len(), + 1 + ); + assert_eq!(restore_outcomes(&runtime), [("failed".to_owned(), 1.0)]); + assert_eq!(runtime.free_slots(), 4); +} + +/// The registration's bound: a `claim_restore` that takes longer than +/// `RESTORE_REGISTRATION_TIMEOUT` has the request answer +/// `InternalServerError` at the bound with the line naming it, while the +/// task goes on and the job runs to DONE once the registration returns. +#[tokio::test(start_paused = true)] +async fn the_request_stops_waiting_at_the_registration_bound_and_the_task_goes_on() { + let (lines, _guard) = LogLines::capture(); + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + // The design's figure, 60 s on every backend; the registration takes + // one second longer. + assert_eq!(RESTORE_REGISTRATION_TIMEOUT, Duration::from_secs(60)); + *fake.claim_restore_delay.lock().unwrap() = Some(Duration::from_secs(61)); + let runtime = job_runtime(&mem); + let started = tokio::time::Instant::now(); + let err = restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap_err(); + assert!( + matches!(err, DynamoDbError::InternalServerError(_)), + "{err:?}" + ); + assert!( + started.elapsed() >= Duration::from_secs(60) && started.elapsed() < Duration::from_secs(61), + "answered at the bound: {:?}", + started.elapsed() + ); + assert_eq!(runtime.running_jobs(), 1, "the task goes on"); + assert_eq!( + lines.containing("the registration did not answer within 60 s; the task goes on"), + [format!( + "RestoreTableFromBackup of {} into MusicRestored (account {ACCOUNT}): the \ + registration did not answer within 60 s; the task goes on and the target, if \ + registered, appears through DescribeTable", + arn(ACCOUNT) + )] + ); + wait_for_jobs(&runtime).await; + assert_eq!( + fake.restore_job("MusicRestored").unwrap().status, + JobStatus::Done + ); + assert_eq!(fake.restored_items("MusicRestored").len(), 2); + assert_eq!(runtime.free_slots(), 4); +} + +/// The pre-write check's second clause on the restore: with the job held +/// inside a `bulk_load` and its heartbeat timer ended without raising the +/// loss, the next check (before the next file, before the count, before the +/// DONE write) finds the last confirmed beat more than two periods old and +/// ends the job as a lost claim naming what it was about to do, with no +/// further `bulk_load`, no count, no `finish_restore` and no status write. +#[tokio::test] +async fn a_stopped_heartbeat_timer_stops_the_restore_before_its_next_write() { + let (lines, _guard) = LogLines::capture(); + let period = job_settings().heartbeat_interval; + // (the target's name, the bulk_load the job is held in, the reason the + // next check names, the bulk_loads and counts made) + let cells: [(&str, usize, &str, usize, usize); 3] = [ + ( + "TimerStoppedBeforeFile", + 1, + "no heartbeat confirmed the claim recently enough to load data/000002.json.gz", + 1, + 0, + ), + ( + "TimerStoppedBeforeCount", + 2, + "no heartbeat confirmed the claim recently enough to count the target", + 2, + 0, + ), + ( + "TimerStoppedBeforeFlip", + 0, + "no heartbeat confirmed the claim recently enough for the DONE write", + 2, + 1, + ), + ]; + for (table, held_in, reason, loads, counts) in cells { + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs_from(0, 2), songs_from(2, 2)], + true, + ); + let runtime = job_runtime(&mem); + let gate = Arc::new(tokio::sync::Semaphore::new(held_in.saturating_sub(1))); + let count_gate = closed_gate(); + if held_in > 0 { + *fake.bulk_load_gate.lock().unwrap() = Some(Arc::clone(&gate)); + } else { + *fake.count_gate.lock().unwrap() = Some(Arc::clone(&count_gate)); + } + restore_on( + &fake, + &runtime, + &restore_body(json!({ "TargetTableName": table })), + ) + .await + .unwrap(); + let table_id = format!("tid-{table}"); + if held_in > 0 { + wait_for_bulk_loads_started(&fake, held_in).await; + } else { + // Both files loaded; the job waits in the count. + let deadline = tokio::time::Instant::now() + Duration::from_secs(30); + while fake.bulk_loads.lock().unwrap().len() < 2 { + assert!(tokio::time::Instant::now() < deadline, "{table}"); + tokio::time::sleep(Duration::from_millis(5)).await; + } + tokio::time::sleep(period / 2).await; + } + restore::test_support::stop_timer(&table_id); + tokio::time::sleep(3 * period).await; + gate.add_permits(10); + count_gate.add_permits(10); + wait_for_jobs(&runtime).await; + assert_eq!(fake.bulk_loads.lock().unwrap().len(), loads, "{table}"); + assert_eq!( + fake.count_calls.load(std::sync::atomic::Ordering::Acquire), + counts, + "{table}" + ); + assert!(restore_writes(&fake).is_empty(), "{table}: no status write"); + assert!(fake.removed_targets.lock().unwrap().is_empty(), "{table}"); + let record = fake.restore_job(table).unwrap(); + assert_eq!(record.status, JobStatus::Running, "{table}"); + let stopped = lines.containing(&format!( + "into {table} (account {ACCOUNT}): stopped with no further catalog or table access: \ + {reason}" + )); + assert_eq!(stopped.len(), 1, "{table}: {stopped:?}"); + assert_eq!( + restore_outcomes(&runtime), + [("failed".to_owned(), 1.0)], + "{table}" + ); + } +} + +/// The DONE write's `NotMatched` outcome: the generation moved after the +/// last `bulk_load` returned (the job held in the count), so +/// `finish_restore` matches no record; the job removes no target, writes no +/// status, records `RestoreJobs{failed}` and the worker error, logs the +/// "matched no record" line, and leaves its spool entry to the sweep (the +/// record's owner may be reading it). +#[tokio::test] +async fn a_done_write_that_matches_no_record_removes_no_target_and_keeps_the_entry() { + let (lines, _guard) = LogLines::capture(); + let scratch = restore::spool::tests::Scratch::new(); + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + fake.rows + .lock() + .unwrap() + .get_mut(&arn(ACCOUNT)) + .unwrap() + .store = s3_identity(); + *fake.heartbeat_answer.lock().unwrap() = HeartbeatAnswer::AlwaysConfirm; + let count_gate = closed_gate(); + *fake.count_gate.lock().unwrap() = Some(Arc::clone(&count_gate)); + let runtime = spooled_runtime(&mem, scratch.path()); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + let deadline = tokio::time::Instant::now() + Duration::from_secs(30); + while fake.bulk_loads.lock().unwrap().is_empty() { + assert!(tokio::time::Instant::now() < deadline); + tokio::time::sleep(Duration::from_millis(5)).await; + } + let other = { + let mut records = fake.restore_jobs.lock().unwrap(); + let record = records.get_mut("tid-MusicRestored").unwrap(); + record.owner_generation = new_generation(); + record.owner_generation + }; + count_gate.add_permits(10); + wait_for_jobs(&runtime).await; + assert_eq!( + fake.count_calls.load(std::sync::atomic::Ordering::Acquire), + 1 + ); + assert!( + fake.removed_targets.lock().unwrap().is_empty(), + "no table access" + ); + assert!(restore_writes(&fake).is_empty(), "no status write"); + let record = fake.restore_job("MusicRestored").unwrap(); + assert_eq!(record.status, JobStatus::Running); + assert_eq!(record.owner_generation, other); + assert!(!fake.restore_tables.lock().unwrap()["tid-MusicRestored"].active); + let line = lines.containing("the DONE write matched no record"); + assert_eq!(line.len(), 1, "{line:?}"); + assert!(line[0].contains("no table access was made"), "{}", line[0]); + let metrics = drained(&runtime); + assert_eq!( + metrics + .iter() + .filter(|(m, _, _)| *m == MetricName::RestoreJobs) + .map(|(_, op, sum)| (op.clone(), *sum)) + .collect::>(), + [("failed".to_owned(), 1.0)] + ); + assert!( + metrics + .iter() + .any(|(m, op, _)| *m == MetricName::WorkerErrorCount && op == "RestoreJob"), + "{metrics:?}" + ); + assert!( + scratch.path().join("restore-tid-MusicRestored").is_dir(), + "the entry is left to the sweep" + ); + assert_eq!(runtime.free_slots(), 4); +} + +/// A job that learnt the record is another's leaves its spool entry in +/// place for the sweep: one whose claim is lost while it is held inside a +/// `bulk_load`, and one whose FAILED write matches no record (the +/// generation moved under it). Both end with no table access. +#[tokio::test] +async fn a_job_that_learnt_the_record_is_anothers_leaves_its_spool_entry() { + for lost_claim in [true, false] { + let scratch = restore::spool::tests::Scratch::new(); + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + fake.rows + .lock() + .unwrap() + .get_mut(&arn(ACCOUNT)) + .unwrap() + .store = s3_identity(); + let gate = closed_gate(); + *fake.bulk_load_gate.lock().unwrap() = Some(Arc::clone(&gate)); + let runtime = spooled_runtime(&mem, scratch.path()); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_bulk_loads_started(&fake, 1).await; + let entry = scratch.path().join("restore-tid-MusicRestored"); + assert!(entry.join("000000.json.gz.verified").is_file()); + if lost_claim { + *fake.heartbeat_answer.lock().unwrap() = HeartbeatAnswer::Lost; + } else { + *fake.heartbeat_answer.lock().unwrap() = HeartbeatAnswer::AlwaysConfirm; + fake.restore_jobs + .lock() + .unwrap() + .get_mut("tid-MusicRestored") + .unwrap() + .owner_generation = new_generation(); + gate.add_permits(10); + } + wait_for_jobs(&runtime).await; + assert!( + fake.removed_targets.lock().unwrap().is_empty(), + "{lost_claim}: no table access" + ); + assert_eq!( + restore_writes(&fake).len(), + usize::from(!lost_claim), + "{lost_claim}: a lost claim writes nothing, a failing job one FAILED write" + ); + assert_eq!( + fake.restore_job("MusicRestored").unwrap().status, + JobStatus::Running, + "{lost_claim}" + ); + assert!( + entry.join("000000.json.gz.verified").is_file(), + "{lost_claim}: the entry is left to the sweep" + ); + assert_eq!( + restore_outcomes(&runtime), + [("failed".to_owned(), 1.0)], + "{lost_claim}" + ); + } +} + +/// `finish_restore` answering `Pending` (MongoDB: DONE with the index +/// backfill left to the backend's worker) is a success: the backfill line, +/// `RestoreJobs{succeeded}`, `WorkerLastSuccess{RestoreJob}`, the spool entry +/// removed and the record DONE, the table not flipped by the job. +#[tokio::test] +async fn a_pending_finish_is_a_success_whose_flip_is_the_backends() { + let (lines, _guard) = LogLines::capture(); + let scratch = restore::spool::tests::Scratch::new(); + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + fake.rows + .lock() + .unwrap() + .get_mut(&arn(ACCOUNT)) + .unwrap() + .store = s3_identity(); + fake.finish_pending + .store(true, std::sync::atomic::Ordering::Release); + let runtime = spooled_runtime(&mem, scratch.path()); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_jobs(&runtime).await; + let record = fake.restore_job("MusicRestored").unwrap(); + assert_eq!(record.status, JobStatus::Done); + assert!(record.completed_at.is_some()); + assert!( + !fake.restore_tables.lock().unwrap()["tid-MusicRestored"].active, + "the flip is the backend's worker's" + ); + assert!(fake.removed_targets.lock().unwrap().is_empty()); + assert!(!scratch.path().join("restore-tid-MusicRestored").exists()); + let pending = + lines.containing("index backfill pending, the backend's worker flips the table ACTIVE"); + assert_eq!(pending.len(), 1, "{pending:?}"); + assert!( + pending[0].contains(": DONE, 2 items (2 applied by this run)"), + "{}", + pending[0] + ); + let metrics = drained(&runtime); + assert_eq!( + metrics + .iter() + .filter(|(m, _, _)| *m == MetricName::RestoreJobs) + .map(|(_, op, sum)| (op.clone(), *sum)) + .collect::>(), + [("succeeded".to_owned(), 1.0)] + ); + assert!( + metrics + .iter() + .any(|(m, op, _)| *m == MetricName::WorkerLastSuccess && op == "RestoreJob"), + "{metrics:?}" + ); + assert!( + !metrics + .iter() + .any(|(m, op, _)| *m == MetricName::WorkerErrorCount && op == "RestoreJob"), + "{metrics:?}" + ); +} + +/// The retry bound is the job's, from its first download attempt: the first +/// file succeeds after ten minutes of transport failures, the second fails +/// from then on, and the restore is FAILED naming the second file and the +/// bound once fifteen minutes have passed since the job's first attempt +/// (a clock per file would run it fifteen minutes longer). +#[tokio::test(start_paused = true)] +async fn the_download_retry_bound_is_the_jobs_not_each_files() { + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs_from(0, 2), songs_from(2, 2)], + true, + ); + let runtime = job_runtime(&mem); + *mem.fail_data_gets_until.lock().unwrap() = + Some(tokio::time::Instant::now() + Duration::from_secs(10 * 60)); + *mem.fail_gets_of.lock().unwrap() = Some("data/000002.json.gz".to_owned()); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + let started = tokio::time::Instant::now(); + while runtime.running_jobs() > 0 { + tokio::time::sleep(Duration::from_millis(50)).await; + } + let elapsed = started.elapsed(); + assert!( + elapsed >= Duration::from_secs(15 * 60) && elapsed < Duration::from_secs(16 * 60 + 2), + "{elapsed:?}" + ); + let writes = restore_writes(&fake); + assert_eq!(writes.len(), 1); + let reason = writes[0].2.clone().unwrap(); + assert!( + reason.starts_with("downloading data/000002.json.gz: transport error"), + "{reason}" + ); + assert!(reason.contains("the bound is 900 s"), "{reason}"); + let reads = gets_under(&mem, &arn(ACCOUNT)); + let first: Vec<&String> = reads + .iter() + .filter(|k| k.ends_with("000001.json.gz")) + .collect(); + let second: Vec<&String> = reads + .iter() + .filter(|k| k.ends_with("000002.json.gz")) + .collect(); + // The backoff restarts per file (1+2+4+8+16+32 s, then 60 s each): about + // ten minutes of attempts on the first file, five on the second. + assert!( + (13..=18).contains(&first.len()), + "{} attempts on file 1", + first.len() + ); + assert!( + (9..=13).contains(&second.len()), + "{} attempts on file 2", + second.len() + ); + assert!(fake.bulk_loads.lock().unwrap().is_empty()); + assert_eq!(restore_outcomes(&runtime), [("failed".to_owned(), 1.0)]); +} + +/// The download's own pre-attempt check: with the first attempt failed and +/// the job in its backoff, a heartbeat timer ended without raising the loss +/// stops the job before the next attempt, naming the file; no second +/// attempt is made and nothing is written. +#[tokio::test(start_paused = true)] +async fn a_stopped_heartbeat_timer_stops_the_download_before_its_next_attempt() { + let (lines, _guard) = LogLines::capture(); + let period = job_settings().heartbeat_interval; + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + let runtime = job_runtime(&mem); + mem.fail_gets_with(GetFailure::Transport, 1); + restore_on( + &fake, + &runtime, + &restore_body(json!({ "TargetTableName": "TimerStoppedInBackoff" })), + ) + .await + .unwrap(); + // The first attempt failed; the job sleeps its first backoff (at least + // one second), during which the timer is ended. + while gets_under(&mem, &arn(ACCOUNT)).len() < 3 { + tokio::time::sleep(Duration::from_millis(5)).await; + } + restore::test_support::stop_timer("tid-TimerStoppedInBackoff"); + tokio::time::sleep(3 * period).await; + while runtime.running_jobs() > 0 { + tokio::time::sleep(Duration::from_millis(5)).await; + } + assert_eq!( + gets_under(&mem, &arn(ACCOUNT)).len(), + 3, + "no second attempt" + ); + assert!(restore_writes(&fake).is_empty()); + assert!(fake.bulk_loads.lock().unwrap().is_empty()); + let stopped = lines.containing( + "stopped with no further catalog or table access: no heartbeat confirmed the claim \ + recently enough to download data/000001.json.gz", + ); + assert_eq!(stopped.len(), 1, "{stopped:?}"); + assert_eq!( + fake.restore_job("TimerStoppedInBackoff").unwrap().status, + JobStatus::Running + ); +} + +/// A decoder that dies on the blocking pool (its reader panics mid-file) +/// drops the channel without an `Err` item, so the backend sees a clean end +/// of a short stream; the job fails the file rather than reporting it +/// applied, and the target is removed. +#[tokio::test] +async fn a_decoder_that_dies_fails_the_file_rather_than_completing_it() { + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + // The gets: the sidecar, the manifest, the file's hashing pass, then + // the file's load pass, whose body dies. + mem.panic_body_of_get + .store(4, std::sync::atomic::Ordering::Release); + let runtime = job_runtime(&mem); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_jobs(&runtime).await; + let loads = fake.bulk_loads.lock().unwrap().clone(); + assert_eq!(loads.len(), 1); + assert_eq!(loads[0].stream_error, None, "the backend saw a clean end"); + assert!(loads[0].items.len() < 2, "{} items", loads[0].items.len()); + let writes = restore_writes(&fake); + assert_eq!(writes.len(), 1); + assert_eq!( + writes[0].2.as_deref(), + Some("data file data/000001.json.gz: the decoder stopped unexpectedly") + ); + assert_eq!(fake.removed_targets.lock().unwrap().len(), 1); + assert!(fake.restore_job("MusicRestored").is_none()); + assert_eq!(restore_outcomes(&runtime), [("failed".to_owned(), 1.0)]); +} + +/// Stop a restore into `MusicRestored` on `runtime` at its first `bulk_load` +/// through the shutdown, so its spool entry holds every file verified; then +/// rewrite its record as a dead owner's with no progress recorded over an +/// empty target (the shutdown lets the held file finish, so the record +/// would otherwise name file 0). +async fn stop_a_spooled_restore_before_its_first_load( + fake: &Arc, + runtime: &BackupRuntime, + gate: &Arc, +) { + *fake.bulk_load_gate.lock().unwrap() = Some(Arc::clone(gate)); + restore_on(fake, runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_bulk_loads_started(fake, 1).await; + runtime.drain_jobs(Duration::from_millis(100)).await; + gate.add_permits(10); + wait_for_jobs(runtime).await; + *fake.bulk_load_gate.lock().unwrap() = None; + fake.bulk_loads.lock().unwrap().clear(); + { + let mut records = fake.restore_jobs.lock().unwrap(); + let record = records.get_mut("tid-MusicRestored").unwrap(); + assert_eq!(record.status, JobStatus::Running); + record.file_index = None; + record.items_applied = None; + record.heartbeat_at = 1234.5; + let mut targets = fake.restore_tables.lock().unwrap(); + let target = targets.get_mut("tid-MusicRestored").unwrap(); + target.items.clear(); + target.active = false; + } + fake.restore_dead + .lock() + .unwrap() + .insert("tid-MusicRestored".to_owned()); +} + +/// A same-host resume with every file verified in the spool reads the +/// manifest and sidecar from the spool and downloads nothing: no store +/// read at all, no `RestoreBytesDownloaded`. +#[tokio::test] +async fn a_same_host_resume_reads_the_spool_and_downloads_nothing() { + let (lines, _guard) = LogLines::capture(); + let scratch = restore::spool::tests::Scratch::new(); + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Arc::new(Fake::default()); + let files = [songs_from(0, 2), songs_from(2, 2)]; + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &files, + true, + ); + fake.rows + .lock() + .unwrap() + .get_mut(&arn(ACCOUNT)) + .unwrap() + .store = s3_identity(); + let runtime = spooled_runtime(&mem, scratch.path()); + stop_a_spooled_restore_before_its_first_load(&fake, &runtime, &closed_gate()).await; + mem.gets.lock().unwrap().clear(); + let _ = drained(&runtime); + + let runtime2 = spooled_runtime(&mem, scratch.path()); + let report = cycle_on(&fake, &runtime2).await; + assert_eq!(report.manifest_restores_resumed, ["MusicRestored"]); + wait_for_jobs(&runtime2).await; + assert!( + gets_under(&mem, &arn(ACCOUNT)).is_empty(), + "{:?}", + gets_under(&mem, &arn(ACCOUNT)) + ); + assert_eq!(fake.bulk_load_files(), [(0, 0, true, 2), (1, 0, false, 2)]); + let all: Vec = files.iter().flatten().cloned().collect(); + assert_eq!(fake.restored_items("MusicRestored"), all); + assert_eq!( + fake.restore_job("MusicRestored").unwrap().status, + JobStatus::Done + ); + assert_eq!( + lines + .containing("resumed by this instance; the manifest was read from this host's spool") + .len(), + 1 + ); + assert!( + !drained(&runtime2) + .iter() + .any(|(m, _, _)| *m == MetricName::RestoreBytesDownloaded), + "nothing downloaded" + ); + let active = lines.containing(": ACTIVE, 4 items (4 applied by this run), 0 bytes downloaded"); + assert_eq!(active.len(), 1, "{active:?}"); +} + +/// A `.verified` marker binds the digest: a marker naming another digest +/// does not make its file verified, and a same-host resume downloads that +/// file again and only it. +#[tokio::test] +async fn a_marker_naming_another_digest_has_its_file_downloaded_again() { + let scratch = restore::spool::tests::Scratch::new(); + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Arc::new(Fake::default()); + let files = [songs_from(0, 2), songs_from(2, 2)]; + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &files, + true, + ); + fake.rows + .lock() + .unwrap() + .get_mut(&arn(ACCOUNT)) + .unwrap() + .store = s3_identity(); + let runtime = spooled_runtime(&mem, scratch.path()); + stop_a_spooled_restore_before_its_first_load(&fake, &runtime, &closed_gate()).await; + let entry = scratch.path().join("restore-tid-MusicRestored"); + let marker = entry.join("000000.json.gz.verified"); + let recorded = std::fs::read_to_string(&marker).unwrap(); + assert_eq!(recorded.trim_end(), files_sha256(&mem, &arn(ACCOUNT), 1)); + std::fs::write(&marker, format!("{}\n", "0".repeat(64))).unwrap(); + mem.gets.lock().unwrap().clear(); + + let runtime2 = spooled_runtime(&mem, scratch.path()); + let report = cycle_on(&fake, &runtime2).await; + assert_eq!(report.manifest_restores_resumed, ["MusicRestored"]); + wait_for_jobs(&runtime2).await; + assert_eq!( + gets_under(&mem, &arn(ACCOUNT)), + ["data/000001.json.gz"], + "the file whose marker names another digest, and nothing else" + ); + let all: Vec = files.iter().flatten().cloned().collect(); + assert_eq!(fake.restored_items("MusicRestored"), all); + assert_eq!( + fake.restore_job("MusicRestored").unwrap().status, + JobStatus::Done + ); + assert!(!entry.exists(), "the entry is removed with the job's end"); +} + +/// The lowercase hex sha256 of data file `n` (1-based key) of `arn` in `mem`. +fn files_sha256(mem: &MemStore, arn: &str, n: u32) -> String { + use sha2::Digest; + let prefix = BackupArn::parse(arn).unwrap().store_prefix(); + let key = format!("{prefix}{}", data_file_key(n)); + let bytes = mem.objects.lock().unwrap()[&key].clone(); + format!("{:x}", sha2::Sha256::digest(&bytes)) +} + +/// The spool states a crash can leave, and what a same-host resume +/// downloads: file 0 verified (file and marker), file 1 a `.partial` only, +/// file 2 a file without its marker, file 3 a marker without its file. The +/// verified file is kept; the other three are downloaded again, every file +/// is loaded and the restore completes. +#[tokio::test] +async fn a_resume_downloads_the_spool_files_a_crash_left_unverified() { + let scratch = restore::spool::tests::Scratch::new(); + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Arc::new(Fake::default()); + let files = [ + songs_from(0, 2), + songs_from(2, 2), + songs_from(4, 2), + songs_from(6, 2), + ]; + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &files, + true, + ); + fake.rows + .lock() + .unwrap() + .get_mut(&arn(ACCOUNT)) + .unwrap() + .store = s3_identity(); + let runtime = spooled_runtime(&mem, scratch.path()); + stop_a_spooled_restore_before_its_first_load(&fake, &runtime, &closed_gate()).await; + let entry = scratch.path().join("restore-tid-MusicRestored"); + for i in 0..4 { + assert!(entry.join(format!("{i:06}.json.gz.verified")).is_file()); + } + std::fs::rename( + entry.join("000001.json.gz"), + entry.join("000001.json.gz.partial"), + ) + .unwrap(); + std::fs::remove_file(entry.join("000001.json.gz.verified")).unwrap(); + std::fs::remove_file(entry.join("000002.json.gz.verified")).unwrap(); + std::fs::remove_file(entry.join("000003.json.gz")).unwrap(); + mem.gets.lock().unwrap().clear(); + + let runtime2 = spooled_runtime(&mem, scratch.path()); + let report = cycle_on(&fake, &runtime2).await; + assert_eq!(report.manifest_restores_resumed, ["MusicRestored"]); + wait_for_jobs(&runtime2).await; + assert_eq!( + gets_under(&mem, &arn(ACCOUNT)), + [ + "data/000002.json.gz", + "data/000003.json.gz", + "data/000004.json.gz" + ], + "the verified file is kept; the .partial, the unmarked file and the marker without a \ + file are downloaded again" + ); + assert_eq!(fake.bulk_load_files().len(), 4); + let all: Vec = files.iter().flatten().cloned().collect(); + assert_eq!(fake.restored_items("MusicRestored"), all); + assert_eq!( + fake.restore_job("MusicRestored").unwrap().status, + JobStatus::Done + ); + assert!(!entry.exists()); +} + +/// Songs `from..from + n` each carrying a `Blob` of `bytes` bytes. +fn padded_songs(from: usize, n: usize, bytes: usize, pattern: impl Fn(usize) -> u8) -> Vec { + songs_from(from, n) + .into_iter() + .enumerate() + .map(|(i, mut item)| { + let blob: Vec = (0..bytes) + .map(|j| pattern(i * 7919 + j * 104_729)) + .collect(); + item.insert( + "Blob".to_owned(), + extenddb_core::types::AttributeValue::B(blob), + ); + item + }) + .collect() +} + +/// `padded_songs` whose blob is `zeros` zero bytes followed by bytes from +/// `pattern`: the zeros compress to nothing, so one item's `zeros` tunes the +/// compressed length of a file by under a byte per step while the +/// uncompressed length and the item count stay the same. +fn padded_songs_with_zeros( + from: usize, + n: usize, + bytes: usize, + zeros_in_first: usize, + pattern: impl Fn(usize) -> u8, +) -> Vec { + let mut items = padded_songs(from, n, bytes, pattern); + if let Some(first) = items.first_mut() { + let blob: Vec = (0..bytes) + .map(|j| { + if j < zeros_in_first { + 0 + } else { + (j.wrapping_mul(2_654_435_761) >> 7) as u8 + } + }) + .collect(); + first.insert( + "Blob".to_owned(), + extenddb_core::types::AttributeValue::B(blob), + ); + } + items +} + +/// The blob byte pattern of `seed`: bytes with no structure a compressor +/// finds, so two seeds give files of nearly the same compressed length. +fn blob_pattern(seed: usize) -> impl Fn(usize) -> u8 + Copy { + move |x: usize| { + let mut z = (x as u64).wrapping_add((seed as u64).wrapping_mul(0x9E37_79B9_7F4A_7C15)); + z = (z ^ (z >> 30)).wrapping_mul(0xBF58_476D_1CE4_E5B9); + z = (z ^ (z >> 27)).wrapping_mul(0x94D0_49BB_1331_11EB); + (z ^ (z >> 31)) as u8 + } +} + +/// A data file of the same item count, uncompressed length and compressed +/// length as `original` (the file of `blob_pattern(0)`), holding other items +/// of the same keys and blob length (so only the digests differ): the blobs +/// are seeded differently per try, and within a try the zeros at the head of +/// the first item's blob are swept until the compressed length matches. +fn same_length_replacement(original: &[u8], from: usize, n: usize, bytes: usize) -> Vec { + for seed in 1usize..200 { + let pattern = blob_pattern(seed); + let (base, _) = + encode_data_file(padded_songs_with_zeros(from, n, bytes, 0, pattern)).unwrap(); + if base.len() < original.len() || base.len() > original.len() + bytes / 2 { + continue; + } + for zeros in 0..=bytes { + let (candidate, _) = + encode_data_file(padded_songs_with_zeros(from, n, bytes, zeros, pattern)).unwrap(); + if candidate.len() == original.len() { + assert_ne!(candidate, original); + return candidate; + } + if candidate.len() + 8 < original.len() { + break; + } + } + } + panic!( + "no replacement of {} compressed bytes found", + original.len() + ) +} + +/// On an in-place (filesystem) store a data file replaced between the +/// hashing pass and the load pass by another valid file of the same item +/// count, uncompressed length and compressed length (so only its digests +/// differ from the manifest's): the items of the replacement reach the +/// backend as they decode (two chunks of a thousand), the end-of-stream +/// check then fails the file on its sha256, since the load's second read is +/// hashed again, the restore is FAILED naming the file and the target is +/// removed. +/// +/// The job runs at a 1 s heartbeat, not the 50 ms of `job_settings`: the +/// hashing pass over both files runs before the first load's claim check, +/// on this test's one runtime thread with the heartbeat's timer, and under +/// a loaded host it can outlast two 50 ms periods, which ends the job as a +/// lost claim before any load. +#[tokio::test] +async fn a_file_replaced_between_the_two_passes_fails_in_the_load() { + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + // Payloads that compress little, so the compressed body is long and the + // decoder's first chunks are sent before the stream ends. + let blob = 256; + let files = [ + songs_from(0, 2), + padded_songs(2, 2500, blob, blob_pattern(0)), + ]; + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &files, + false, + ); + let prefix = BackupArn::parse(&arn(ACCOUNT)).unwrap().store_prefix(); + let key = format!("{prefix}data/000002.json.gz"); + let original = mem.objects.lock().unwrap()[&key].clone(); + let replacement = same_length_replacement(&original, 2, 2500, blob); + assert_eq!(replacement.len(), original.len()); + let runtime = job_runtime(&mem).with_job_settings(JobSettings { + heartbeat_interval: Duration::from_secs(1), + ..job_settings() + }); + let gate = closed_gate(); + *fake.bulk_load_gate.lock().unwrap() = Some(Arc::clone(&gate)); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + // Both files hashed (pass 1) and file 1's second get made; file 2 not + // yet re-read. + wait_for_bulk_loads_started(&fake, 1).await; + let reads = gets_under(&mem, &arn(ACCOUNT)); + assert_eq!( + reads + .iter() + .filter(|k| k.ends_with("000002.json.gz")) + .count(), + 1, + "{reads:?}" + ); + mem.insert(&key, &replacement); + gate.add_permits(10); + wait_for_jobs(&runtime).await; + let writes = restore_writes(&fake); + assert_eq!( + writes.len(), + 1, + "the second pass hashes the file again and fails the restore: {writes:?}" + ); + let reason = writes[0].2.clone().unwrap(); + assert!( + reason.starts_with("data file data/000002.json.gz: data file sha256 checksum mismatch"), + "the second pass hashes the file again: {reason}" + ); + let loads = fake.bulk_loads.lock().unwrap().clone(); + assert_eq!(loads.len(), 2); + assert_eq!( + loads[1].items.len(), + 2000, + "two chunks of the replacement reached the backend before the end-of-stream check" + ); + assert!(loads[1].stream_error.is_some(), "{loads:?}"); + assert!( + fake.restore_job("MusicRestored").is_none(), + "target removed" + ); + assert_eq!(fake.removed_targets.lock().unwrap().len(), 1); +} + +/// A file over one chunk by line bytes (the job's bound set small for the +/// test; the items are well under the count bound) is fed in chunks that +/// close at the byte bound, and the shutdown is read at each chunk boundary. +/// The backend is held after it pulled the first item of the second chunk, +/// so the decoder has sent two chunks and holds at most two more in the +/// channel and one past its check; the shutdown fired then is seen at the +/// close of a later chunk, so the backend receives a whole number of chunks +/// (two to five of six) and then the shutdown's `Err` item, never the whole +/// file. +#[tokio::test] +async fn a_file_over_one_chunk_by_bytes_is_fed_in_chunks_and_stops_at_a_boundary() { + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + // The job's chunk bound is 32 KiB (the production bound, 32 MiB, would + // make this a 192 MiB file); items of 1,500-byte blobs whose lines are + // the same length, so a chunk is a fixed number of items. + let chunk_bytes: u64 = 32 * 1024; + restore::test_support::set_chunk_bytes("tid-ChunkedByBytes", chunk_bytes); + let blob = 1500; + let line = line_bytes(&padded_songs(0, 1, blob, |_| b'a')[0]); + let per_chunk = usize::try_from(chunk_bytes.div_ceil(line)).unwrap(); + assert!( + (10..=20).contains(&per_chunk), + "{per_chunk} items of {line} bytes per chunk" + ); + let total = 6 * per_chunk; + let items = padded_songs(0, total, blob, |_| b'a'); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[items], + true, + ); + let runtime = job_runtime(&mem); + let item_gate = Arc::new(tokio::sync::Semaphore::new(per_chunk + 1)); + *fake.bulk_load_item_gate.lock().unwrap() = Some(Arc::clone(&item_gate)); + restore_on( + &fake, + &runtime, + &restore_body(json!({ "TargetTableName": "ChunkedByBytes" })), + ) + .await + .unwrap(); + wait_for_items_seen(&fake, per_chunk + 1).await; + let left = runtime.drain_jobs(Duration::from_millis(100)).await; + assert_eq!(left, 1, "the job waits at the gate past the bound"); + item_gate.add_permits(total + 10); + wait_for_jobs(&runtime).await; + let loads = fake.bulk_loads.lock().unwrap().clone(); + assert_eq!(loads.len(), 1); + let received = loads[0].items.len(); + assert!( + received % per_chunk == 0 && received >= 2 * per_chunk && received < total, + "{received} items received of {total}, {per_chunk} per chunk" + ); + assert_eq!( + loads[0].stream_error.as_deref(), + Some("Internal error: the restore was stopped by the server's shutdown"), + "{:?}", + loads[0].stream_error + ); + assert!(restore_writes(&fake).is_empty()); + assert_eq!( + fake.restore_job("ChunkedByBytes").unwrap().status, + JobStatus::Running + ); + assert_eq!(restore_outcomes(&runtime), [("cancelled".to_owned(), 1.0)]); +} + +/// Wait until `bulk_load` has pulled `n` items from its stream. +async fn wait_for_items_seen(fake: &Fake, n: usize) { + let deadline = tokio::time::Instant::now() + Duration::from_secs(30); + while fake + .bulk_load_items_seen + .load(std::sync::atomic::Ordering::Acquire) + < n + { + assert!( + tokio::time::Instant::now() < deadline, + "the backend pulled no item" + ); + tokio::time::sleep(Duration::from_millis(5)).await; + } +} + +/// The decoder's claim check at each chunk boundary: the backend is held +/// after it pulled the first item of the second chunk (the decoder has sent +/// two chunks of a thousand and holds at most two more and one past its +/// check), the heartbeat timer is ended without raising the loss, and the +/// decoder ends the stream at the close of a later chunk with the +/// claim-lost reason: a whole number of chunks (two to five of six) reach +/// the backend, then the `Err` item, and the job stops with no catalog +/// write. +#[tokio::test] +async fn a_feeder_whose_timer_stopped_ends_the_stream_with_the_claim_lost_reason() { + let (lines, _guard) = LogLines::capture(); + let period = job_settings().heartbeat_interval; + let per_chunk = restore::load::LOAD_CHUNK_ITEMS; + let total = 6 * per_chunk; + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs_from(0, total)], + true, + ); + let runtime = job_runtime(&mem); + let item_gate = Arc::new(tokio::sync::Semaphore::new(per_chunk + 1)); + *fake.bulk_load_item_gate.lock().unwrap() = Some(Arc::clone(&item_gate)); + restore_on( + &fake, + &runtime, + &restore_body(json!({ "TargetTableName": "FeederTimerStopped" })), + ) + .await + .unwrap(); + wait_for_items_seen(&fake, per_chunk + 1).await; + restore::test_support::stop_timer("tid-FeederTimerStopped"); + tokio::time::sleep(3 * period).await; + item_gate.add_permits(total + 10); + wait_for_jobs(&runtime).await; + let loads = fake.bulk_loads.lock().unwrap().clone(); + assert_eq!(loads.len(), 1); + let received = loads[0].items.len(); + assert!( + received % per_chunk == 0 && received >= 2 * per_chunk && received < total, + "{received} items received of {total}" + ); + assert_eq!( + loads[0].stream_error.as_deref(), + Some("Internal error: the claim was lost while the file was being decoded"), + "{:?}", + loads[0].stream_error + ); + assert!(restore_writes(&fake).is_empty()); + assert!(fake.removed_targets.lock().unwrap().is_empty()); + let stopped = lines.containing( + "stopped with no further catalog or table access: the claim was lost while the file \ + was being decoded", + ); + assert_eq!(stopped.len(), 1, "{stopped:?}"); + assert_eq!( + fake.restore_job("FeederTimerStopped").unwrap().status, + JobStatus::Running + ); +} + +/// The manifest arm reads the health this cycle's check left: a store that +/// passed a check once (its identity known) and is down when the cycle +/// runs is unhealthy to the arm, which skips the record with the line +/// naming the state; a store whose check cannot read the catalog and so has +/// not yet been checked is skipped with "not yet checked"; once the store +/// answers again the next cycle resumes the record. +#[tokio::test] +async fn an_unhealthy_or_unchecked_store_with_a_known_identity_skips_the_resume() { + let (lines, _guard) = LogLines::capture(); + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + let runtime = job_runtime(&mem); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_jobs(&runtime).await; + dead_record_without_progress(&fake, false); + fake.bulk_loads.lock().unwrap().clear(); + + // Healthy with its identity known, then down when the cycle runs. + let runtime = job_runtime(&mem); + assert_eq!( + runtime.configured_store().unwrap().identity(), + Some(store()) + ); + mem.set_down(true); + let report = cycle_on(&fake, &runtime).await; + assert_eq!(report.store_healthy, Some(false)); + assert_eq!(report.restores_skipped_store_unhealthy, ["MusicRestored"]); + assert!(report.manifest_restores_resumed.is_empty()); + assert!(fake.restore_takeovers.lock().unwrap().is_empty()); + assert_eq!( + lines.containing("dead store-backed restores are not resumed this cycle"), + [ + "the configured backup store is unhealthy; dead store-backed restores are not \ + resumed this cycle" + ] + ); + mem.set_down(false); + + // Not yet checked: the store's first check cannot read the catalog. + let checking = self::runtime() + .with_store(ConfiguredStore::checking( + StoreKind::Filesystem, + store().store_root, + Arc::clone(&mem) as Arc, + None, + )) + .with_job_settings(job_settings()) + .with_metrics(Arc::new(extenddb_core::metrics::MetricsCollector::new())); + *fake.catalog_id_error.lock().unwrap() = true; + let report = cycle_on(&fake, &checking).await; + assert_eq!(report.store_healthy, Some(false)); + assert_eq!(report.restores_skipped_store_unhealthy, ["MusicRestored"]); + assert!(fake.restore_takeovers.lock().unwrap().is_empty()); + assert_eq!( + lines + .containing("the configured backup store is not yet checked; dead store-backed restores are not resumed this cycle") + .len(), + 1 + ); + *fake.catalog_id_error.lock().unwrap() = false; + + // The store answers again: resumed. + let report = cycle_on(&fake, &runtime).await; + assert_eq!(report.store_healthy, Some(true)); + assert_eq!(report.manifest_restores_resumed, ["MusicRestored"]); + wait_for_jobs(&runtime).await; + assert_eq!( + fake.restore_job("MusicRestored").unwrap().status, + JobStatus::Done + ); +} + +// --- the sweep's read order, the DONE write's unknown outcome, the resume's +// instant and documents, the request's bounds ----------------------------------- + +/// A restore that registers (RUNNING, this host's `spool_host`) and opens +/// its spool entry after the reconciler read the RUNNING records: the spool +/// was listed before the records, so the entry is not a candidate and the +/// live job keeps it (the design: the sweep never removes the spool of the +/// job that owns the record). +#[tokio::test] +async fn a_restore_registered_between_the_two_reads_keeps_its_spool_entry() { + let scratch = restore::spool::tests::Scratch::new(); + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Arc::new(Fake::default()); + let runtime = spooled_runtime(&mem, scratch.path()); + // An entry of a dead restore, so the sweep has work to do in the same + // cycle and the live entry's survival is not the sweep doing nothing. + std::fs::create_dir(scratch.path().join("restore-dead")).unwrap(); + let spool_dir = scratch.path().to_path_buf(); + *fake.sweep_hook.lock().unwrap() = Some(Box::new(move |fake: &Fake| { + fake.restore_jobs.lock().unwrap().insert( + "late".to_owned(), + RestoreJobRow { + target: RestoreTargetKey { + account_id: ACCOUNT.to_owned(), + table_name: "late".to_owned(), + table_id: "late".to_owned(), + }, + source_backup_arn: arn(ACCOUNT), + kind: RestoreKind::Manifest, + status: JobStatus::Running, + owner_generation: 1, + heartbeat_at: seconds_since_epoch(), + store: Some(s3_identity()), + spool_host: Some("host-a:/var/spool".to_owned()), + file_index: None, + items_applied: None, + failure_reason: None, + completed_at: None, + }, + ); + // The job's `Entry::open` and its first download. + let entry = restore::spool::Entry::open( + &spool_dir, + "late", + Some("host-a:/var/spool"), + "host-a:/var/spool", + ) + .unwrap(); + std::fs::write(entry.dir().join("000000.json.gz.partial"), b"bytes").unwrap(); + })); + let report = cycle_on(&fake, &runtime).await; + assert!( + fake.sweep_hook.lock().unwrap().is_none(), + "the registration landed between the two reads" + ); + assert_eq!( + fake.restore_job("late").map(|r| r.status), + Some(JobStatus::Running), + "the record is RUNNING with this host's spool_host" + ); + assert_eq!( + report.spool_entries_swept, + ["dead"], + "the dead entry went; the live restore's entry did not" + ); + assert!( + scratch.path().join("restore-late").is_dir(), + "the live job's spool entry was removed from under it" + ); + assert!( + scratch + .path() + .join("restore-late/000000.json.gz.partial") + .is_file() + ); + // The next cycle, with the record RUNNING and this host's, keeps it too. + let report = cycle_on(&fake, &runtime).await; + assert!(report.spool_entries_swept.is_empty()); + assert!(scratch.path().join("restore-late").is_dir()); +} + +/// A `finish_restore` that returns an error is a write whose outcome is +/// unknown, and the record decides: (a) the flip landed and the call then +/// failed (PostgreSQL's progress delete after the commit): the re-read finds +/// DONE under this generation, the restore is complete and `succeeded`, with +/// a WARN naming the error, no FAILED write, no table access and the entry +/// removed; (b) the call failed before anything was written: the re-read +/// finds RUNNING, the write is sent once more and lands; (c) it fails twice: +/// the record stays RUNNING for the reconciler, no FAILED write, no +/// removal, the job `failed` with the worker error and the entry left; (d) +/// it fails and a successor wrote DONE under its own generation before the +/// re-read: the write matched no record, the job `failed`, no table access. +#[tokio::test] +async fn a_failed_done_write_is_re_read_and_the_record_decides() { + async fn run_with( + errors: &[FinishError], + ) -> (Arc, Vec<(MetricName, String, f64)>, LogLines) { + let (lines, _guard) = LogLines::capture(); + let scratch = restore::spool::tests::Scratch::new(); + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + fake.rows + .lock() + .unwrap() + .get_mut(&arn(ACCOUNT)) + .unwrap() + .store = s3_identity(); + fake.finish_errors + .lock() + .unwrap() + .extend(errors.iter().copied()); + let runtime = spooled_runtime(&mem, scratch.path()); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_jobs(&runtime).await; + let entry_left = scratch.path().join("restore-tid-MusicRestored").exists(); + let mut metrics = drained(&runtime); + metrics.push(( + MetricName::RestoreJobs, + format!("entry_left={entry_left}"), + 0.0, + )); + (fake, metrics, lines) + } + fn restore_jobs(metrics: &[(MetricName, String, f64)]) -> Vec<(String, f64)> { + metrics + .iter() + .filter(|(m, op, _)| *m == MetricName::RestoreJobs && !op.starts_with("entry_left")) + .map(|(_, op, sum)| (op.clone(), *sum)) + .collect() + } + fn entry_left(metrics: &[(MetricName, String, f64)]) -> bool { + metrics.iter().any(|(_, op, _)| op == "entry_left=true") + } + + // (a) The flip landed, then the error. + let (fake, metrics, lines) = run_with(&[FinishError::AfterFlip]).await; + assert_eq!(fake.finish_sizes.lock().unwrap().len(), 1, "one DONE write"); + let record = fake.restore_job("MusicRestored").unwrap(); + assert_eq!(record.status, JobStatus::Done); + assert!(fake.restore_tables.lock().unwrap()["tid-MusicRestored"].active); + assert!( + restore_writes(&fake).is_empty(), + "no FAILED write over a DONE record: {:?}", + restore_writes(&fake) + ); + assert!(fake.removed_targets.lock().unwrap().is_empty()); + assert_eq!(restore_jobs(&metrics), [("succeeded".to_owned(), 1.0)]); + assert!( + metrics + .iter() + .any(|(m, op, _)| *m == MetricName::WorkerLastSuccess && op == "RestoreJob") + ); + assert!(!entry_left(&metrics), "the entry is removed on a success"); + let warned = lines.containing( + "the DONE write returned an error (Internal error: the data pool failed after the flip's \ + commit) and the record is DONE under this job's generation: the write landed and the \ + restore is complete", + ); + assert_eq!(warned.len(), 1, "{warned:?}"); + assert_eq!( + lines + .containing(": complete (DONE), 2 items (2 applied by this run)") + .len(), + 1, + "the line says DONE, not ACTIVE: the table's flip is the backend's" + ); + assert!(lines.containing(": ACTIVE, ").is_empty()); + + // (b) The error before anything was written: once more, and it lands. + let (fake, metrics, lines) = run_with(&[FinishError::BeforeFlip]).await; + assert_eq!( + fake.finish_sizes.lock().unwrap().len(), + 2, + "the write sent once more" + ); + assert_eq!( + fake.restore_job("MusicRestored").unwrap().status, + JobStatus::Done + ); + assert!(restore_writes(&fake).is_empty()); + assert_eq!(restore_jobs(&metrics), [("succeeded".to_owned(), 1.0)]); + assert_eq!( + lines + .containing("the record is still RUNNING under this claim; writing once more") + .len(), + 1 + ); + + // (d) The error, and a takeover whose successor wrote DONE before the + // re-read: the record is DONE under another generation, so the write + // matched no record; no second write, no table access, the entry left. + let (fake, metrics, lines) = run_with(&[FinishError::TakenAndDone]).await; + assert_eq!(fake.finish_sizes.lock().unwrap().len(), 1, "one DONE write"); + assert_eq!( + fake.restore_job("MusicRestored").unwrap().status, + JobStatus::Done + ); + assert!(restore_writes(&fake).is_empty()); + assert!(fake.removed_targets.lock().unwrap().is_empty()); + assert_eq!(restore_jobs(&metrics), [("failed".to_owned(), 1.0)]); + assert!(entry_left(&metrics), "the entry is left to the sweep"); + let ended = lines.containing( + "the DONE write matched no record (the DONE write failed (Internal error: the claim \ + connection failed before the DONE write) and the record carries generation", + ); + assert_eq!(ended.len(), 1, "{ended:?}"); + assert!( + lines.containing("complete (DONE)").is_empty(), + "a successor's DONE is not this job's" + ); + + // (c) Twice: no third write, no FAILED write, no removal; the record is + // the reconciler's. + let (fake, metrics, lines) = + run_with(&[FinishError::BeforeFlip, FinishError::BeforeFlip]).await; + assert_eq!( + fake.finish_sizes.lock().unwrap().len(), + 2, + "two writes at most" + ); + let record = fake.restore_job("MusicRestored").unwrap(); + assert_eq!(record.status, JobStatus::Running); + assert!( + restore_writes(&fake).is_empty(), + "no FAILED write: {:?}", + restore_writes(&fake) + ); + assert!(fake.removed_targets.lock().unwrap().is_empty()); + assert!( + fake.tables.lock().unwrap().contains_key("MusicRestored"), + "the target stands" + ); + assert_eq!(restore_jobs(&metrics), [("failed".to_owned(), 1.0)]); + assert!( + metrics + .iter() + .any(|(m, op, _)| *m == MetricName::WorkerErrorCount && op == "RestoreJob") + ); + assert!(entry_left(&metrics), "the entry is left to the sweep"); + let ended = lines.containing( + "the DONE write matched no record (the DONE write failed twice (Internal error: the claim \ + connection failed before the DONE write) and the record is still RUNNING under this \ + claim; the reconciler resumes it once the record is stale); the record is not this \ + job's to write and no table access was made", + ); + assert_eq!(ended.len(), 1, "{ended:?}"); +} + +/// A resumed job's first confirmed beat is the instant the takeover +/// returned, not the instant the job started: with the record read held +/// three periods after the takeover, the job's first check finds the beat +/// older than two periods and stops as a lost claim with no load and no +/// status write (the record is one a second taker may hold by then); with +/// the read held one period the job runs to DONE. +#[tokio::test(start_paused = true)] +async fn a_resumes_first_beat_is_the_takeovers_return() { + async fn resume_held(periods: u32) -> (Arc, Arc, LogLines) { + let (lines, _guard) = LogLines::capture(); + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs_from(0, 2), songs_from(2, 2)], + true, + ); + let runtime = job_runtime(&mem); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_jobs(&runtime).await; + // The record as a dead owner left it after file 0. + { + let mut records = fake.restore_jobs.lock().unwrap(); + let record = records.get_mut("tid-MusicRestored").unwrap(); + record.status = JobStatus::Running; + record.completed_at = None; + record.file_index = Some(0); + record.items_applied = Some(2); + record.heartbeat_at = 1234.5; + let mut targets = fake.restore_tables.lock().unwrap(); + let target = targets.get_mut("tid-MusicRestored").unwrap(); + target.active = false; + target.items.truncate(2); + if let Some(desc) = fake.tables.lock().unwrap().get_mut("MusicRestored") { + desc.table_status = TableStatus::Creating; + } + } + fake.restore_dead + .lock() + .unwrap() + .insert("tid-MusicRestored".to_owned()); + fake.bulk_loads.lock().unwrap().clear(); + mem.gets.lock().unwrap().clear(); + *fake.read_restore_job_delay.lock().unwrap() = + Some(job_settings().heartbeat_interval * periods); + let runtime = job_runtime(&mem); + let report = cycle_on(&fake, &runtime).await; + assert_eq!(report.manifest_restores_resumed, ["MusicRestored"]); + wait_for_jobs(&runtime).await; + (fake, mem, lines) + } + + let (fake, mem, lines) = resume_held(3).await; + assert!( + fake.bulk_loads.lock().unwrap().is_empty(), + "no load under a beat three periods old: {:?}", + fake.bulk_load_files() + ); + assert_eq!( + fake.restore_job("MusicRestored").unwrap().status, + JobStatus::Running, + "no status write" + ); + assert!(restore_writes(&fake).is_empty()); + let stopped = lines.containing( + "stopped with no further catalog or table access: no heartbeat confirmed the claim \ + recently enough to download data/000001.json.gz", + ); + assert_eq!(stopped.len(), 1, "{stopped:?}"); + assert_eq!( + gets_under(&mem, &arn(ACCOUNT)), + [EXTENDDB_MANIFEST_FILE, MANIFEST_CHECKSUM_FILE], + "the two documents of the resume's preparation and no data file" + ); + + let (fake, _mem, _lines) = resume_held(1).await; + assert_eq!(fake.bulk_load_files(), [(1, 0, true, 2)]); + assert_eq!( + fake.restore_job("MusicRestored").unwrap().status, + JobStatus::Done + ); +} + +/// A same-host resume whose spooled manifest a crash left short reads the +/// store's manifest and sidecar instead (one WARN), writes them to the entry +/// again, and resumes with the spooled files; the restore completes. +#[tokio::test] +async fn a_truncated_spooled_manifest_has_the_resume_read_the_store() { + let (lines, _guard) = LogLines::capture(); + let scratch = restore::spool::tests::Scratch::new(); + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Arc::new(Fake::default()); + let files = [songs_from(0, 2), songs_from(2, 2)]; + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &files, + true, + ); + fake.rows + .lock() + .unwrap() + .get_mut(&arn(ACCOUNT)) + .unwrap() + .store = s3_identity(); + let runtime = spooled_runtime(&mem, scratch.path()); + stop_a_spooled_restore_before_its_first_load(&fake, &runtime, &closed_gate()).await; + mem.gets.lock().unwrap().clear(); + let entry = scratch.path().join("restore-tid-MusicRestored"); + let spooled_manifest = std::fs::read(entry.join(EXTENDDB_MANIFEST_FILE)).unwrap(); + // A crash during the manifest's write, as an in-place write leaves it. + std::fs::write( + entry.join(EXTENDDB_MANIFEST_FILE), + &spooled_manifest[..spooled_manifest.len() / 2], + ) + .unwrap(); + + let runtime2 = spooled_runtime(&mem, scratch.path()); + let report = cycle_on(&fake, &runtime2).await; + assert_eq!(report.manifest_restores_resumed, ["MusicRestored"]); + wait_for_jobs(&runtime2).await; + assert_eq!( + fake.restore_job("MusicRestored").map(|r| r.status), + Some(JobStatus::Done), + "the resume read the store's documents and completed: {:?}", + restore_writes(&fake) + ); + assert_eq!( + gets_under(&mem, &arn(ACCOUNT)), + [EXTENDDB_MANIFEST_FILE, MANIFEST_CHECKSUM_FILE], + "the two documents from the store, no data file (the spooled files are verified)" + ); + assert_eq!(fake.bulk_load_files(), [(0, 0, true, 2), (1, 0, false, 2)]); + assert_eq!( + lines + .containing( + "the spooled manifest and sidecar of the restore into MusicRestored (account \ + 123456789012) do not verify (the manifest's checksum sidecar does not match" + ) + .len(), + 1 + ); + assert_eq!( + lines + .containing("resumed by this instance; the manifest was read from the store") + .len(), + 1 + ); +} + +/// A resume whose store documents fail verification is FAILED with the +/// reason, its target removed, and its duration recorded from the resume's +/// start: a sidecar that does not match the manifest; a manifest naming +/// another backup under a sidecar that matches it. +#[tokio::test] +async fn a_resume_whose_store_documents_fail_verification_is_failed() { + async fn resume_over(tamper: impl Fn(&MemStore, &str)) -> (Arc, BackupRuntime, LogLines) { + let (lines, _guard) = LogLines::capture(); + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs_from(0, 2), songs_from(2, 2)], + true, + ); + let runtime = job_runtime(&mem); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_jobs(&runtime).await; + { + let mut records = fake.restore_jobs.lock().unwrap(); + let record = records.get_mut("tid-MusicRestored").unwrap(); + record.status = JobStatus::Running; + record.completed_at = None; + record.file_index = Some(0); + record.items_applied = Some(2); + record.heartbeat_at = 1234.5; + let mut targets = fake.restore_tables.lock().unwrap(); + let target = targets.get_mut("tid-MusicRestored").unwrap(); + target.active = false; + target.items.truncate(2); + } + fake.restore_dead + .lock() + .unwrap() + .insert("tid-MusicRestored".to_owned()); + fake.bulk_loads.lock().unwrap().clear(); + let prefix = BackupArn::parse(&arn(ACCOUNT)).unwrap().store_prefix(); + tamper(&mem, &prefix); + let runtime = job_runtime(&mem); + let report = cycle_on(&fake, &runtime).await; + assert_eq!(report.manifest_restores_resumed, ["MusicRestored"]); + wait_for_jobs(&runtime).await; + drop(_guard); + (fake, runtime, lines) + } + fn failed_with(fake: &Fake, runtime: &BackupRuntime, reason_prefix: &str) { + let writes = restore_writes(fake); + assert_eq!(writes.len(), 1, "{writes:?}"); + assert_eq!( + (writes[0].0, writes[0].1), + (JobStatus::Running, JobStatus::Failed) + ); + let reason = writes[0].2.clone().unwrap(); + assert!(reason.starts_with(reason_prefix), "{reason}"); + assert!(fake.bulk_loads.lock().unwrap().is_empty(), "no load"); + assert!( + fake.restore_job("MusicRestored").is_none(), + "the target and its record are removed" + ); + assert_eq!(fake.removed_targets.lock().unwrap().len(), 1); + let metrics = drained(runtime); + let durations: Vec = metrics + .iter() + .filter(|(m, op, _)| *m == MetricName::RestoreJobDuration && op == "failed") + .map(|(_, _, sum)| *sum) + .collect(); + assert_eq!(durations.len(), 1, "{metrics:?}"); + assert!( + durations[0] > 0.0, + "the duration runs from the resume's start: {metrics:?}" + ); + assert!( + metrics + .iter() + .any(|(m, op, _)| *m == MetricName::WorkerErrorCount && op == "RestoreJob") + ); + } + + // The sidecar names another digest. + let (fake, runtime, lines) = resume_over(|mem, prefix| { + mem.insert( + &format!("{prefix}{MANIFEST_CHECKSUM_FILE}"), + format!("{} {EXTENDDB_MANIFEST_FILE}\n", "0".repeat(64)).as_bytes(), + ); + }) + .await; + failed_with( + &fake, + &runtime, + "the manifest's checksum sidecar does not match", + ); + assert_eq!( + lines + .containing("FAILED on resume (the manifest's checksum sidecar does not match") + .len(), + 1 + ); + + // The manifest names another backup of the same table and instant + // (another id; the parser's identity rule admits it), under a sidecar + // that matches it. + let (fake, runtime, lines) = resume_over(|mem, prefix| { + let other = BackupArn::parse(&arn(ACCOUNT).replace("73d8d5bc", "0badf00d")).unwrap(); + let bytes = + mem.objects.lock().unwrap()[&format!("{prefix}{EXTENDDB_MANIFEST_FILE}")].clone(); + let text = String::from_utf8(bytes).unwrap(); + let own_id = BackupArn::parse(&arn(ACCOUNT)).unwrap(); + let renamed = text.replace( + own_id.to_arn_string().as_str(), + other.to_arn_string().as_str(), + ); + assert_ne!(renamed, text); + let manifest = renamed.into_bytes(); + let sidecar = { + use sha2::Digest; + format!( + "{:x} {EXTENDDB_MANIFEST_FILE}\n", + sha2::Sha256::digest(&manifest) + ) + }; + mem.insert(&format!("{prefix}{EXTENDDB_MANIFEST_FILE}"), &manifest); + mem.insert( + &format!("{prefix}{MANIFEST_CHECKSUM_FILE}"), + sidecar.as_bytes(), + ); + }) + .await; + failed_with(&fake, &runtime, "the manifest names another backup, "); + assert_eq!( + lines + .containing("FAILED on resume (the manifest names another backup, ") + .len(), + 1 + ); +} + +/// A same-host resume that fails before it runs (the spooled pair a crash +/// left short, and the store's sidecar naming another digest) removes this +/// host's spool entry with the failure path, as the job removes its entry +/// at every failed end: the record is FAILED naming the mismatch, the +/// target removed, and the entry gone with its verified files. +#[tokio::test] +async fn a_resume_that_fails_before_it_runs_removes_this_hosts_spool_entry() { + let (lines, _guard) = LogLines::capture(); + let scratch = restore::spool::tests::Scratch::new(); + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs_from(0, 2), songs_from(2, 2)], + true, + ); + fake.rows + .lock() + .unwrap() + .get_mut(&arn(ACCOUNT)) + .unwrap() + .store = s3_identity(); + let runtime = spooled_runtime(&mem, scratch.path()); + stop_a_spooled_restore_before_its_first_load(&fake, &runtime, &closed_gate()).await; + let entry = scratch.path().join("restore-tid-MusicRestored"); + assert!(entry.join("000000.json.gz.verified").is_file()); + let spooled_manifest = std::fs::read(entry.join(EXTENDDB_MANIFEST_FILE)).unwrap(); + std::fs::write( + entry.join(EXTENDDB_MANIFEST_FILE), + &spooled_manifest[..spooled_manifest.len() / 2], + ) + .unwrap(); + let prefix = BackupArn::parse(&arn(ACCOUNT)).unwrap().store_prefix(); + mem.insert( + &format!("{prefix}{MANIFEST_CHECKSUM_FILE}"), + format!("{} {EXTENDDB_MANIFEST_FILE}\n", "0".repeat(64)).as_bytes(), + ); + + let runtime2 = spooled_runtime(&mem, scratch.path()); + let report = cycle_on(&fake, &runtime2).await; + assert_eq!(report.manifest_restores_resumed, ["MusicRestored"]); + wait_for_jobs(&runtime2).await; + let writes = restore_writes(&fake); + assert_eq!(writes.len(), 1, "{writes:?}"); + assert_eq!( + (writes[0].0, writes[0].1), + (JobStatus::Running, JobStatus::Failed) + ); + let reason = writes[0].2.clone().unwrap(); + assert!( + reason.starts_with("the manifest's checksum sidecar does not match"), + "{reason}" + ); + assert_eq!(fake.removed_targets.lock().unwrap().len(), 1); + assert!(fake.bulk_loads.lock().unwrap().is_empty()); + assert!( + !entry.exists(), + "this host's entry goes with the failed resume" + ); + let ended = + lines.containing("FAILED on resume (the manifest's checksum sidecar does not match"); + assert_eq!(ended.len(), 1, "{ended:?}"); + assert!(ended[0].ends_with("; the target was removed"), "{ended:?}"); +} + +/// A spool document write that fails (the `.partial` path is taken by a +/// directory, as a disk full fails the same arm) fails the restore naming +/// the document, removes the target, and removes the entry the job had +/// opened, as every failed end does. +#[tokio::test] +async fn a_spool_document_write_failure_fails_the_restore_and_removes_the_entry() { + let scratch = restore::spool::tests::Scratch::new(); + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + fake.rows + .lock() + .unwrap() + .get_mut(&arn(ACCOUNT)) + .unwrap() + .store = s3_identity(); + let entry = scratch.path().join("restore-tid-MusicRestored"); + std::fs::create_dir_all(entry.join(format!("{EXTENDDB_MANIFEST_FILE}.partial"))).unwrap(); + let runtime = spooled_runtime(&mem, scratch.path()); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_jobs(&runtime).await; + let writes = restore_writes(&fake); + assert_eq!(writes.len(), 1, "{writes:?}"); + let reason = writes[0].2.clone().unwrap(); + assert!( + reason.starts_with(&format!("writing {EXTENDDB_MANIFEST_FILE} to the spool: ")), + "{reason}" + ); + assert_eq!(fake.removed_targets.lock().unwrap().len(), 1); + assert!(fake.bulk_loads.lock().unwrap().is_empty()); + assert!( + !entry.exists(), + "the entry is removed after a FAILED write that matched" + ); + assert_eq!(restore_outcomes(&runtime), [("failed".to_owned(), 1.0)]); +} + +/// `Progress::from_records` on each backend's record shape: PostgreSQL's +/// (the record carries no `file_index`; the progress rows name each +/// recorded file's count), MongoDB's (`file_index` with no `items_applied`: +/// the recorded file is complete), and SQLite's and Cassandra's (both, +/// mid-file); a `file_index` past the manifest records nothing more. +#[test] +fn progress_from_records_reads_each_backends_shape() { + let mem = MemStore::default(); + let fake = Fake::default(); + let manifest = seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs_from(0, 3), songs_from(3, 2), songs_from(5, 4)], + true, + ); + let mut row = RestoreJobRow::reconciler_row( + RestoreTargetKey { + account_id: ACCOUNT.to_owned(), + table_name: "T".to_owned(), + table_id: "tid".to_owned(), + }, + arn(ACCOUNT), + RestoreKind::Manifest, + JobStatus::Running, + 1, + 0.0, + ); + + // PostgreSQL: files 0 and 2 recorded whole, file 1 not; the resume + // opens file 1 from 0. + let progress = restore::Progress::from_records(&row, &[(0, 3), (2, 4)], &manifest); + assert_eq!(progress.recorded(), [Some(3), None, Some(4)]); + assert_eq!(progress.first_open_in(&manifest), Some((1, 0))); + // A PostgreSQL row of a file a decoder panic left short (n1): the file + // opens from its row. + let progress = restore::Progress::from_records(&row, &[(0, 3), (1, 1)], &manifest); + assert_eq!(progress.recorded(), [Some(3), Some(1), None]); + assert_eq!(progress.first_open_in(&manifest), Some((1, 1))); + + // MongoDB: `file_index` 1 with no `items_applied`: files 0 and 1 are + // complete, file 2 opens from 0. + row.file_index = Some(1); + row.items_applied = None; + let progress = restore::Progress::from_records(&row, &[], &manifest); + assert_eq!(progress.recorded(), [Some(3), Some(2), None]); + assert_eq!(progress.first_open_in(&manifest), Some((2, 0))); + + // SQLite and Cassandra: file 1 after 1 of its items. + row.items_applied = Some(1); + let progress = restore::Progress::from_records(&row, &[], &manifest); + assert_eq!(progress.recorded(), [Some(3), Some(1), None]); + assert_eq!(progress.first_open_in(&manifest), Some((1, 1))); + + // Every file recorded: nothing opens. + row.file_index = Some(2); + row.items_applied = Some(4); + let progress = restore::Progress::from_records(&row, &[], &manifest); + assert_eq!(progress.first_open_in(&manifest), None); + + // A `file_index` past the manifest: every file complete, the count + // decides. + row.file_index = Some(7); + row.items_applied = Some(1); + let progress = restore::Progress::from_records(&row, &[], &manifest); + assert_eq!(progress.recorded(), [Some(3), Some(2), Some(4)]); + assert_eq!(progress.first_open_in(&manifest), None); + + // Nothing recorded. + row.file_index = None; + row.items_applied = None; + let progress = restore::Progress::from_records(&row, &[], &manifest); + assert_eq!(progress.recorded(), [None, None, None]); + assert_eq!(progress.first_open_in(&manifest), Some((0, 0))); + + // The record and a progress row for one file: the larger count is the + // file's, whichever source carries it (no backend writes both today). + row.file_index = Some(1); + row.items_applied = Some(2); + let progress = restore::Progress::from_records(&row, &[(1, 1)], &manifest); + assert_eq!( + progress.recorded(), + [Some(3), Some(2), None], + "a smaller progress row lowered the record's count" + ); + row.items_applied = Some(1); + let progress = restore::Progress::from_records(&row, &[(1, 2)], &manifest); + assert_eq!(progress.recorded(), [Some(3), Some(2), None]); +} + +/// A record gone between the takeover and the reads that follow (whoever +/// moved it owns it) is left to its owner with the info line, not reported +/// as a cycle error. +#[tokio::test] +async fn a_record_gone_after_the_takeover_is_left_to_its_owner() { + let (lines, _guard) = LogLines::capture(); + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + let runtime = job_runtime(&mem); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_jobs(&runtime).await; + dead_record_without_progress(&fake, false); + fake.takeover_removes_record + .store(true, std::sync::atomic::Ordering::Release); + let report = cycle_on(&fake, &runtime).await; + assert_eq!(fake.restore_takeovers.lock().unwrap().len(), 1); + assert!(report.manifest_restores_resumed.is_empty()); + assert!(report.manifest_restores_failed.is_empty()); + assert_eq!( + lines + .containing("was gone when its progress was read; left to whoever moved it") + .len(), + 1 + ); + assert!( + lines + .containing("was taken over but its progress could not be read") + .is_empty() + ); + assert_eq!(lines.at_level(tracing::Level::ERROR), Vec::::new()); +} + +/// A registration that fails after the request stopped waiting for it +/// (the 60 s bound answered `InternalServerError`) is logged at ERROR with +/// the ARN, the target, the account and the error, since the request's own +/// line told the operator the target appears if registered. +#[tokio::test] +async fn a_registration_that_fails_after_the_request_stopped_waiting_is_logged() { + let (lines, _guard) = LogLines::capture(); + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + let manifest = seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + let runtime = job_runtime(&mem); + let store = runtime.configured_store().unwrap(); + let env = job::JobEnv::new(&runtime, store).unwrap(); + let permit = runtime.take_permit().unwrap(); + let prefix = BackupArn::parse(&arn(ACCOUNT)).unwrap().store_prefix(); + let manifest_bytes = + mem.objects.lock().unwrap()[&format!("{prefix}{EXTENDDB_MANIFEST_FILE}")].clone(); + let sidecar_bytes = + mem.objects.lock().unwrap()[&format!("{prefix}{MANIFEST_CHECKSUM_FILE}")].clone(); + let input = extenddb_storage::backup_definition::restore_input_from_manifest( + &manifest.source_table, + "MusicRestored", + &RestoreTableOverrides::default(), + ); + let key_schema = extenddb_core::validation::RestorableKeySchema::resolve( + &manifest.source_table.key_schema, + &manifest.source_table.attribute_definitions, + ) + .unwrap(); + let parts = restore::RestoreRequestParts { + account_id: ACCOUNT.to_owned(), + arn: BackupArn::parse(&arn(ACCOUNT)).unwrap(), + backup: BackupKey::from_arn(&BackupArn::parse(&arn(ACCOUNT)).unwrap()), + manifest: Arc::new(manifest), + manifest_bytes, + sidecar_bytes, + input, + key_schema, + limits: (&LimitsConfig::default()).into(), + }; + *fake.claim_restore_error.lock().unwrap() = + Some(StorageError::TableAlreadyExists("MusicRestored".to_owned())); + let (reply, answer) = tokio::sync::oneshot::channel(); + // The request gave up at its bound. + drop(answer); + restore::start(Arc::clone(&fake), env, permit, parts, reply).await; + let logged = lines.containing( + "the registration failed after the request had stopped waiting for it; no target was \ + created", + ); + assert_eq!(logged.len(), 1, "{logged:?}"); + assert!( + logged[0].starts_with(&format!( + "restore of {} into MusicRestored (account {ACCOUNT}): ", + arn(ACCOUNT) + )), + "{}", + logged[0] + ); + assert!(logged[0].contains("MusicRestored"), "{}", logged[0]); + assert_eq!(runtime.free_slots(), 4, "the permit is returned"); +} + +/// The request's two document reads collect at most their bound: a sidecar +/// over 4 KiB or a manifest over 16 MiB behind an AVAILABLE row is refused +/// as a corrupt prefix (`ValidationException`), not read whole; a resume +/// meeting one is FAILED with the same reason. +#[tokio::test] +async fn the_document_reads_are_bounded() { + let mem = Arc::new(MemStore::default()); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + let runtime = job_runtime(&mem); + let prefix = BackupArn::parse(&arn(ACCOUNT)).unwrap().store_prefix(); + let sidecar_key = format!("{prefix}{MANIFEST_CHECKSUM_FILE}"); + let good_sidecar = mem.objects.lock().unwrap()[&sidecar_key].clone(); + let mut oversize = good_sidecar.clone(); + oversize.extend(std::iter::repeat_n(b' ', 4 * 1024)); + mem.insert(&sidecar_key, &oversize); + let err = restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap_err(); + assert_eq!( + validation_message(err), + format!( + "Backup {} has a {MANIFEST_CHECKSUM_FILE} over 4096 bytes, which no backup this \ + server wrote produces; the backup's prefix is corrupt", + arn(ACCOUNT) + ) + ); + assert!(fake.restore_claims.lock().unwrap().is_empty()); + mem.insert(&sidecar_key, &good_sidecar); + + // The manifest's bound, one byte over. + let manifest_key = format!("{prefix}{EXTENDDB_MANIFEST_FILE}"); + let good_manifest = mem.objects.lock().unwrap()[&manifest_key].clone(); + let mut oversize = good_manifest.clone(); + oversize.resize(16 * 1024 * 1024 + 1, b' '); + mem.insert(&manifest_key, &oversize); + let err = restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap_err(); + assert_eq!( + validation_message(err), + format!( + "Backup {} has a {EXTENDDB_MANIFEST_FILE} over 16777216 bytes, which no backup this \ + server wrote produces; the backup's prefix is corrupt", + arn(ACCOUNT) + ) + ); + // At the bound, the read goes through and the checksum decides. + oversize.truncate(16 * 1024 * 1024); + mem.insert(&manifest_key, &oversize); + let err = restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap_err(); + assert!( + validation_message(err).contains("whose checksum sidecar does not match"), + "a manifest at the bound is read and verified" + ); + mem.insert(&manifest_key, &good_manifest); + + // A resume over an oversize sidecar is FAILED with the reason. + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_jobs(&runtime).await; + dead_record_without_progress(&fake, false); + let mut oversize = good_sidecar.clone(); + oversize.extend(std::iter::repeat_n(b' ', 4 * 1024)); + mem.insert(&sidecar_key, &oversize); + let runtime = job_runtime(&mem); + let report = cycle_on(&fake, &runtime).await; + assert_eq!(report.manifest_restores_resumed, ["MusicRestored"]); + wait_for_jobs(&runtime).await; + let writes = restore_writes(&fake); + assert_eq!( + writes.last().and_then(|w| w.2.clone()).as_deref(), + Some( + "the store's extenddb-manifest.sha256 for the backup is over 4096 bytes, which no \ + backup this server wrote produces" + ), + "{writes:?}" + ); +} + +// --- `backup remove-target` (`maintenance::remove_target`) ------------------------ + +/// The request `remove_target` runs with, against the `Fake`. +fn remove_target_request(table: &str, owner_window: Duration) -> maintenance::RemoveTarget { + maintenance::RemoveTarget { + account_id: ACCOUNT.to_owned(), + table_name: table.to_owned(), + holder: "extenddb backup remove-target on host-a (pid 1)".to_owned(), + stale_after: Duration::from_secs(300), + owner_window, + heartbeat_interval: Duration::from_millis(20), + } +} + +/// `backup remove-target` under the maintenance claim: a FAILED record's +/// CREATING target (`target_not_empty`, the reason the reconciler leaves +/// for it) is removed with its record, and the claim was taken for the +/// removal and released after it; a RUNNING record the backend's dead-job +/// listing does not name is refused and nothing is removed, whatever its +/// `heartbeat_at` reads on this host's clock (one second old; 1234.5, an +/// age a host whose clock runs ahead of the catalog's would compute for a +/// live owner); a RUNNING record the listing names is removed, with a +/// heartbeat this host's clock reads as fresh; a DONE record (the table +/// ACTIVE) and a table no record names are refused as "not a restore +/// target"; a claim a live holder has refuses the command before any read. +#[tokio::test] +async fn remove_target_removes_a_failed_or_dead_target_under_the_claim_and_refuses_the_rest() { + let mem = Arc::new(MemStore::default()); + mem.insert(STORE_IDENTITY_KEY, &identity_file("store-1", "catalog-1")); + let fake = Arc::new(Fake::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + let runtime = job_runtime(&mem); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_jobs(&runtime).await; + + // DONE: the table is ACTIVE. + assert_eq!( + fake.restore_job("MusicRestored").unwrap().status, + JobStatus::Done + ); + let err = maintenance::remove_target( + &*fake, + &remove_target_request("MusicRestored", Duration::from_secs(300)), + ) + .await + .unwrap_err(); + assert_eq!( + err, + format!( + "MusicRestored (account {ACCOUNT}) is not a restore target: its restore from {} is \ + DONE (the table is ACTIVE, or awaits its index backfill)", + arn(ACCOUNT) + ) + ); + assert!(fake.removed_targets.lock().unwrap().is_empty()); + assert_eq!(fake.maintenance_holders.lock().unwrap().len(), 1); + assert!( + *fake.maintenance_holders.lock().unwrap()[0] + .released + .lock() + .unwrap(), + "the claim is released after a refusal too" + ); + + // No record names the table. + let err = maintenance::remove_target( + &*fake, + &remove_target_request("Nothing", Duration::from_secs(300)), + ) + .await + .unwrap_err(); + assert_eq!( + err, + format!("Nothing (account {ACCOUNT}) is not a restore target: no restore record names it") + ); + + // RUNNING with a live owner: the dead-job listing does not name the + // record (the backend's age test on the catalog's clock), whatever its + // heartbeat reads against this host's clock. + dead_record_without_progress(&fake, true); + fake.restore_dead.lock().unwrap().clear(); + for heartbeat_at in [seconds_since_epoch() - 1.0, 1234.5] { + fake.restore_jobs + .lock() + .unwrap() + .get_mut("tid-MusicRestored") + .unwrap() + .heartbeat_at = heartbeat_at; + let err = maintenance::remove_target( + &*fake, + &remove_target_request("MusicRestored", Duration::from_secs(300)), + ) + .await + .unwrap_err(); + assert_eq!( + err, + format!( + "the restore into MusicRestored (account {ACCOUNT}) from {} is RUNNING with a \ + live owner (a heartbeat within stale_job_after_secs); stop it or wait for it", + arn(ACCOUNT) + ), + "heartbeat_at {heartbeat_at}" + ); + assert!(fake.removed_targets.lock().unwrap().is_empty()); + assert!(fake.tables.lock().unwrap().contains_key("MusicRestored")); + } + fake.restore_dead + .lock() + .unwrap() + .insert("tid-MusicRestored".to_owned()); + + // The reconciler finds the dead owner, no progress and a non-empty + // target: FAILED `target_not_empty`, the target left; the command + // removes it. + let report = cycle_on(&fake, &runtime).await; + assert_eq!( + report.manifest_restores_failed, + [("MusicRestored".to_owned(), TARGET_NOT_EMPTY.to_owned())] + ); + assert!(fake.tables.lock().unwrap().contains_key("MusicRestored")); + let removed = maintenance::remove_target( + &*fake, + &remove_target_request("MusicRestored", Duration::from_secs(300)), + ) + .await + .unwrap(); + assert_eq!( + removed, + maintenance::RemovedTarget { + table_id: "tid-MusicRestored".to_owned(), + backup_arn: arn(ACCOUNT), + kind: RestoreKind::Manifest, + status: JobStatus::Failed, + failure_reason: Some(TARGET_NOT_EMPTY.to_owned()), + } + ); + assert!( + !fake.tables.lock().unwrap().contains_key("MusicRestored"), + "the target is gone" + ); + assert!( + fake.restore_job("MusicRestored").is_none(), + "the record is gone" + ); + assert_eq!( + fake.removed_targets.lock().unwrap().as_slice(), + [RestoreTargetKey { + account_id: ACCOUNT.to_owned(), + table_name: "MusicRestored".to_owned(), + table_id: "tid-MusicRestored".to_owned(), + }] + ); + { + let holders = fake.maintenance_holders.lock().unwrap(); + assert_eq!(holders.len(), 5, "a claim per command"); + assert!( + holders.iter().all(|h| *h.released.lock().unwrap()), + "every claim released" + ); + } + assert!( + fake.maintenance_claim_held_by.lock().unwrap().is_none(), + "no claim left held" + ); + + // RUNNING with a dead owner: the listing names the record (under the + // 300 s window with a heartbeat this host's clock reads as fresh; under + // the zero window SQLite's serve lock gives, with a heartbeat an hour + // ahead): removed with its record, no FAILED write. + for window in [Duration::from_secs(300), Duration::ZERO] { + let fake = Arc::new(Fake::default()); + let mem = Arc::new(MemStore::default()); + seed_manifest_backup( + &fake, + &mem, + &arn(ACCOUNT), + music_source_table(), + &[songs(2)], + true, + ); + let runtime = job_runtime(&mem); + restore_on(&fake, &runtime, &restore_body(json!({}))) + .await + .unwrap(); + wait_for_jobs(&runtime).await; + dead_record_without_progress(&fake, true); + if window.is_zero() { + fake.restore_jobs + .lock() + .unwrap() + .get_mut("tid-MusicRestored") + .unwrap() + .heartbeat_at = seconds_since_epoch() + 3600.0; + } else { + fake.restore_jobs + .lock() + .unwrap() + .get_mut("tid-MusicRestored") + .unwrap() + .heartbeat_at = seconds_since_epoch(); + } + let removed = + maintenance::remove_target(&*fake, &remove_target_request("MusicRestored", window)) + .await + .unwrap(); + assert_eq!(removed.status, JobStatus::Running); + assert_eq!(removed.failure_reason, None); + assert!(fake.restore_job("MusicRestored").is_none()); + assert!(!fake.tables.lock().unwrap().contains_key("MusicRestored")); + assert!( + fake.failed_jobs.lock().unwrap().is_empty(), + "no FAILED write" + ); + } + + // A claim a live holder has: refused before any read. + let fake = Arc::new(Fake::default()); + *fake.maintenance_claim_held_by.lock().unwrap() = Some("extenddb backup purge".to_owned()); + let err = maintenance::remove_target( + &*fake, + &remove_target_request("MusicRestored", Duration::from_secs(300)), + ) + .await + .unwrap_err(); + assert_eq!( + err, + "the maintenance claim is held by \"extenddb backup purge\"; one maintenance command \ + runs at a time, and a claim whose holder died expires after stale_job_after_secs (300 s)" + ); + assert!(fake.restore_lists.lock().unwrap().is_empty(), "no read"); +} + +/// A legacy record's target is removed the same way: the reconciler would +/// fail and remove it once stale; the command lets the operator do it now. +#[tokio::test] +async fn remove_target_removes_a_legacy_records_target() { + let fake = Arc::new(Fake::default()); + let target = RestoreTargetKey { + account_id: ACCOUNT.to_owned(), + table_name: "Legacy".to_owned(), + table_id: "tid-legacy".to_owned(), + }; + fake.restore_jobs.lock().unwrap().insert( + "tid-legacy".to_owned(), + RestoreJobRow::reconciler_row( + target.clone(), + arn(ACCOUNT), + RestoreKind::Legacy, + JobStatus::Running, + 1, + 1234.5, + ), + ); + fake.restore_dead + .lock() + .unwrap() + .insert("tid-legacy".to_owned()); + *fake.target_still_creating.lock().unwrap() = true; + let removed = maintenance::remove_target( + &*fake, + &remove_target_request("Legacy", Duration::from_secs(300)), + ) + .await + .unwrap(); + assert_eq!(removed.kind, RestoreKind::Legacy); + assert_eq!(removed.table_id, "tid-legacy"); + assert_eq!(fake.removed_targets.lock().unwrap().as_slice(), [target]); + + // A target the backend finds no longer a CREATING restore target. + *fake.target_still_creating.lock().unwrap() = false; + let err = maintenance::remove_target( + &*fake, + &remove_target_request("Legacy", Duration::from_secs(300)), + ) + .await + .unwrap_err(); + assert_eq!( + err, + format!( + "Legacy (account {ACCOUNT}) is not a restore target: it is no longer a CREATING \ + restore target with table id tid-legacy" + ) + ); +} diff --git a/crates/engine/src/lib.rs b/crates/engine/src/lib.rs index f56df9813..dfc5260c2 100755 --- a/crates/engine/src/lib.rs +++ b/crates/engine/src/lib.rs @@ -10,9 +10,9 @@ mod backup; pub use backup::{ BackupRuntime, ConfiguredStore, HeldClaim, JobSettings, KillPoint, LOCAL_CHECK_TIMEOUT, - S3_CHECK_TIMEOUT, STORE_IDENTITY_KEY, Spool, StoreHealth, StoreHealthReport, StoreIdentityFile, - StoreProblem, check_configured_store, maintenance, read_identity_bounded, reconciler, - store_identity, + OTHER_STORE, RESTORE_REGISTRATION_TIMEOUT, S3_CHECK_TIMEOUT, STORE_IDENTITY_KEY, Spool, + StoreHealth, StoreHealthReport, StoreIdentityFile, StoreProblem, TARGET_NOT_EMPTY, + check_configured_store, maintenance, read_identity_bounded, reconciler, store_identity, }; mod batch_get_item; mod batch_write_item; diff --git a/crates/server/Cargo.toml b/crates/server/Cargo.toml index 1b58f02b5..dac2d6a70 100755 --- a/crates/server/Cargo.toml +++ b/crates/server/Cargo.toml @@ -9,9 +9,15 @@ license.workspace = true [features] mongodb-test-hooks = [] -# The backup job's kill-point pauses (`EXTENDDB_BACKUP_PAUSE`), for the -# binary-level crash tests. -backup-test-hooks = ["extenddb-engine/backup-test-hooks"] +# The backup and restore jobs' kill-point pauses (`EXTENDDB_BACKUP_PAUSE`), +# for the binary-level crash tests. The engine's points are its own feature; +# the points inside the backends' writes (`restore_sqlite_mid_batch`, +# `restore_postgres_before_progress_delete`, +# `restore_mongodb_after_insert_many`, `restore_cassandra_before_progress`) +# are calls of `extenddb_storage::backup_catalog::restore_pause_point`, so +# `extenddb-storage/test-hooks` arms them in whichever backend the build +# selects without naming a backend crate here (the server depends on none). +backup-test-hooks = ["extenddb-engine/backup-test-hooks", "extenddb-storage/test-hooks"] [dependencies] extenddb-cache = { workspace = true } diff --git a/crates/server/src/backup_store.rs b/crates/server/src/backup_store.rs index 0116cf1f4..d509eadc9 100644 --- a/crates/server/src/backup_store.rs +++ b/crates/server/src/backup_store.rs @@ -936,6 +936,37 @@ mod tests { ) -> BoxedFuture<'a, Result>, StorageError>> { unused() } + fn claim_restore<'a>( + &'a self, + _account_id: &'a str, + _input: extenddb_core::types::CreateTableInput, + _registration: &'a extenddb_storage::backup_catalog::RestoreRegistrationRow, + ) -> BoxedFuture< + 'a, + Result<(extenddb_core::types::TableDescription, Box), StorageError>, + > { + unused() + } + fn claim_restore_dead<'a>( + &'a self, + _key: &'a RestoreTargetKey, + _stale_after: std::time::Duration, + _spool_host: Option<&'a str>, + ) -> BoxedFuture<'a, Result>, StorageError>> { + unused() + } + fn read_restore_job<'a>( + &'a self, + _key: &'a RestoreTargetKey, + ) -> BoxedFuture<'a, Result, StorageError>> { + unused() + } + fn list_restore_jobs<'a>( + &'a self, + _status: Option, + ) -> BoxedFuture<'a, Result, StorageError>> { + unused() + } fn list_backup_rows<'a>( &'a self, _status: Option, diff --git a/crates/server/src/metrics_endpoint.rs b/crates/server/src/metrics_endpoint.rs index 52bb5c282..f38027a4c 100755 --- a/crates/server/src/metrics_endpoint.rs +++ b/crates/server/src/metrics_endpoint.rs @@ -377,6 +377,9 @@ mod tests { MetricName::BackupBytesUploaded, MetricName::BackupSnapshotAge, MetricName::BackupStoreErrors, + MetricName::RestoreJobs, + MetricName::RestoreJobDuration, + MetricName::RestoreBytesDownloaded, ]; let rows: Vec = names .iter() diff --git a/crates/server/src/serve.rs b/crates/server/src/serve.rs index c32940838..9d753a2f1 100644 --- a/crates/server/src/serve.rs +++ b/crates/server/src/serve.rs @@ -701,7 +701,8 @@ async fn serve_inner(params: ServeParams, port: u16) -> anyhow::Result<()> { backup_runtime .with_metrics(metrics.clone()) .with_shutdown(shutdown.child_token()) - .with_source(&backend_name, build.version), + .with_source(&backend_name, build.version) + .with_limits(Arc::clone(&limits)), ); let backup_runtime_for_reconciler = Arc::clone(&backup_runtime); let state = AppState { diff --git a/crates/storage-cassandra/Cargo.toml b/crates/storage-cassandra/Cargo.toml index ba3378b32..b7e1a0a93 100644 --- a/crates/storage-cassandra/Cargo.toml +++ b/crates/storage-cassandra/Cargo.toml @@ -9,8 +9,10 @@ license.workspace = true [features] # Coordination hooks used only by this crate's integration test binary -# (`src/test_hooks.rs`). Production builds do not compile them. -test-hooks = [] +# (`src/test_hooks.rs`). Production builds do not compile them. Enabling it +# also arms `extenddb_storage::backup_catalog::restore_pause_point`, the +# `bulk_load` kill point the restore kill-point harness drives. +test-hooks = ["extenddb-storage/test-hooks"] [dependencies] extenddb-core = { workspace = true } diff --git a/crates/storage-cassandra/src/backup_catalog.rs b/crates/storage-cassandra/src/backup_catalog.rs index 76d1ada05..36ba5710b 100644 --- a/crates/storage-cassandra/src/backup_catalog.rs +++ b/crates/storage-cassandra/src/backup_catalog.rs @@ -72,11 +72,12 @@ use cdrs_tokio::types::IntoRustByName; use cdrs_tokio::types::rows::Row; use cdrs_tokio::types::value::Value; use extenddb_core::backup_format::BackupArn; +use extenddb_core::types::CreateTableInput; use extenddb_storage::backup_catalog::{ BackupCatalog, BackupKey, BackupRow, BackupStatus, DeleteStart, Heartbeat, JobClaim, JobRow, - JobStatus, MaintenanceClaim, RestoreJobRow, RestoreKind, RestoreTargetKey, StaleClock, - StatusFields, StoreIdentity, StoreKind, TimeBounds, WriteOutcome, billing_mode_name, - new_generation, + JobStatus, MaintenanceClaim, RestoreJobRow, RestoreKind, RestoreRegistered, + RestoreRegistrationRow, RestoreTargetKey, StaleClock, StatusFields, StoreIdentity, StoreKind, + TimeBounds, WriteOutcome, billing_mode_name, new_generation, }; use extenddb_storage::error::StorageError; use futures::future::BoxFuture; @@ -172,7 +173,7 @@ fn outcome_unknown(error: &DriverError) -> bool { } /// Run a conditional write and report what it did. -async fn conditional_write( +pub(crate) async fn conditional_write( session: &CassandraSession, query: &str, values: QueryValues, @@ -571,6 +572,80 @@ impl CassandraEngine { Ok(arns) } + /// Re-apply the target's `active_restores[table_id] = table_name` entry + /// on the backup's row when it is absent, under the takeover of a dead + /// manifest record (the trait doc of `claim_restore_dead`): one serial + /// read of the map, then a lightweight transaction on the entry + /// conditioned on the backup still AVAILABLE. A backup no longer + /// AVAILABLE (a delete won while nothing held it) keeps no entry: the + /// resumed load fails on the missing prefix and the target is removed. + async fn reapply_restore_registration( + &self, + key: &RestoreTargetKey, + backup_arn: &str, + ) -> Result<(), StorageError> { + use cdrs_tokio::types::AsRust; + use cdrs_tokio::types::map::Map; + let catalog = self.catalog_keyspace(); + let rows = crate::cassandra_util::query_rows_serial::( + &self.session, + &format!( + "SELECT active_restores FROM {catalog}.backups_by_arn \ + WHERE account_id = ? AND backup_arn = ?" + ), + cdrs_tokio::query_values!(key.account_id.as_str(), backup_arn), + "restore registration", + ) + .await?; + let Some(row) = rows.first() else { + return Ok(()); + }; + let map: Option = row.get_by_name("active_restores").ok().flatten(); + let entries: std::collections::HashMap = match map { + Some(map) => map + .as_rust() + .map_err(|e| StorageError::Internal(format!("Parse active_restores: {e}")))? + .unwrap_or_default(), + None => std::collections::HashMap::new(), + }; + if entries.contains_key(&key.table_id) { + return Ok(()); + } + let applied = conditional_write( + &self.session, + &format!( + "UPDATE {catalog}.backups_by_arn SET active_restores[?] = ? \ + WHERE account_id = ? AND backup_arn = ? IF backup_status = 'AVAILABLE'" + ), + QueryValues::SimpleValues(vec![ + Value::from(key.table_id.as_str()), + Value::from(key.table_name.as_str()), + Value::from(key.account_id.as_str()), + Value::from(backup_arn), + ]), + ) + .await?; + match applied { + WriteOutcome::Matched => tracing::warn!( + "restore of table {}: the backup {backup_arn} did not list the target as in \ + use (its owner died between the target's creation and the registration); the \ + entry was re-applied by the takeover", + key.table_name + ), + WriteOutcome::NotMatched => tracing::warn!( + "restore of table {}: the backup {backup_arn} is no longer AVAILABLE and does \ + not list the target; the resumed load fails on the missing prefix", + key.table_name + ), + WriteOutcome::Unknown => tracing::warn!( + "restore of table {}: re-applying the backup's in-use entry had no known \ + outcome; the registration sweep removes a stale one", + key.table_name + ), + } + Ok(()) + } + /// The `tables` row of a restore target, read at quorum. async fn restore_target_row( &self, @@ -630,6 +705,187 @@ impl RestoreTargetRow { } } +/// The `tables` columns one restore job record fills, in the order +/// [`restore_job_row_from`] reads them. +const RESTORE_JOB_COLUMNS: &str = "account_id, table_name, table_id, table_status, \ + restore_source_backup_arn, restore_kind, restore_status, restore_owner_generation, \ + restore_heartbeat_at, restore_store_kind, restore_store_root, restore_store_id, \ + restore_spool_host, restore_file_index, restore_items_applied, restore_failure_reason, \ + restore_completed_at"; + +/// One [`RestoreJobRow`] from a `tables` row that holds a restore record +/// (every member [`RESTORE_JOB_COLUMNS`] selects): the store the restore +/// reads, the claimant's spool, the recorded progress, the failure reason +/// and the completion time, beyond the six the reconciler's legacy and dead +/// arms read. +fn restore_job_row_from(row: &Row) -> Result { + let column = |name: &str| -> Result { + crate::cassandra_util::get_column(row, name, "restore job") + }; + let optional = |name: &str| -> Option { row.get_by_name(name).ok().flatten() }; + let store = optional("restore_store_kind") + .as_deref() + .and_then(StoreKind::parse) + .map(|store_kind| StoreIdentity { + store_kind, + store_root: optional("restore_store_root").unwrap_or_default(), + store_id: optional("restore_store_id").unwrap_or_default(), + }); + let generation: Option = row.get_by_name("restore_owner_generation").ok().flatten(); + let heartbeat_at: Option = row.get_by_name("restore_heartbeat_at").ok().flatten(); + let completed_at: Option = row.get_by_name("restore_completed_at").ok().flatten(); + let file_index: Option = row.get_by_name("restore_file_index").ok().flatten(); + let items_applied: Option = row.get_by_name("restore_items_applied").ok().flatten(); + Ok(RestoreJobRow { + target: RestoreTargetKey { + account_id: column("account_id")?, + table_name: column("table_name")?, + table_id: column("table_id")?, + }, + source_backup_arn: column("restore_source_backup_arn")?, + kind: RestoreKind::parse(&column("restore_kind")?)?, + status: JobStatus::parse(&column("restore_status")?)?, + owner_generation: generation_from_column(generation), + heartbeat_at: heartbeat_at.map_or(0.0, millis_to_seconds), + store, + spool_host: optional("restore_spool_host"), + file_index: file_index.and_then(|i| u32::try_from(i).ok()), + items_applied: items_applied.and_then(|i| u64::try_from(i).ok()), + failure_reason: optional("restore_failure_reason"), + completed_at: completed_at.map(millis_to_seconds), + }) +} + +/// A store-backed restore job's handle on its target's `tables` row. Unlike +/// the legacy copy's self-managed [`crate::backup_engine::JobHeartbeat`], +/// the engine drives this claim's heartbeat and status writes; every write +/// is a lightweight transaction conditioned on the restore generation and on +/// the record being RUNNING, and leaves `table_status` to +/// [`CassandraEngine::activate_restore_target_row`] (the ACTIVE flip) and to +/// `remove_restore_target` (the CREATING to DELETING step of a removal). +struct CassandraRestoreClaim { + engine: CassandraEngine, + key: RestoreTargetKey, + generation: u64, +} + +impl JobClaim for CassandraRestoreClaim { + fn generation(&self) -> u64 { + self.generation + } + + fn heartbeat(&self) -> BoxFuture<'_, Result> { + Box::pin(async move { + let query = format!( + "UPDATE {}.tables SET restore_heartbeat_at = toTimestamp(now()) \ + WHERE account_id = ? AND table_name = ? \ + IF table_id = ? AND restore_owner_generation = ? AND restore_status = 'RUNNING'", + self.engine.catalog_keyspace() + ); + let result = crate::cassandra_util::query_lwt( + &self.engine.session, + &query, + cdrs_tokio::query_values!( + self.key.account_id.as_str(), + self.key.table_name.as_str(), + self.key.table_id.as_str(), + generation_column(self.generation) + ), + ) + .await?; + Ok(if crate::cassandra_util::lwt_applied(&result)? { + Heartbeat::Confirmed + } else { + Heartbeat::Lost + }) + }) + } + + fn set_status<'a>( + &'a self, + from: JobStatus, + to: JobStatus, + fields: &'a StatusFields, + ) -> BoxFuture<'a, Result> { + Box::pin(async move { + // The record only; `table_status` is left to the ACTIVE flip and + // to the removal's CREATING to DELETING step (4(e)'s two-write + // rule). A restore record has no size/item/source-table columns, + // so those `StatusFields` are ignored. + let mut assignments = vec!["restore_status = ?".to_owned()]; + let mut values = vec![Value::from(to.as_str())]; + if let Some(reason) = fields.failure_reason.as_deref() { + assignments.push("restore_failure_reason = ?".to_owned()); + values.push(Value::from(reason)); + } + if fields.completed { + assignments.push("restore_completed_at = toTimestamp(now())".to_owned()); + } + values.extend([ + Value::from(self.key.account_id.as_str()), + Value::from(self.key.table_name.as_str()), + Value::from(self.key.table_id.as_str()), + Value::from(generation_column(self.generation)), + Value::from(from.as_str()), + ]); + let query = format!( + "UPDATE {}.tables SET {} WHERE account_id = ? AND table_name = ? \ + IF table_id = ? AND restore_owner_generation = ? AND restore_status = ?", + self.engine.catalog_keyspace(), + assignments.join(", ") + ); + conditional_write( + &self.engine.session, + &query, + QueryValues::SimpleValues(values), + ) + .await + }) + } + + fn read(&self) -> BoxFuture<'_, Result> { + Box::pin(async move { + let query = format!( + "SELECT restore_status, restore_owner_generation, restore_heartbeat_at, \ + restore_failure_reason FROM {}.tables WHERE account_id = ? AND table_name = ?", + self.engine.catalog_keyspace() + ); + let rows = crate::cassandra_util::query_rows_quorum::( + &self.engine.session, + &query, + cdrs_tokio::query_values!( + self.key.account_id.as_str(), + self.key.table_name.as_str() + ), + "restore job row", + ) + .await?; + let row = rows.first().ok_or_else(|| { + StorageError::TableNotFound(format!( + "restore target {} has no job record", + self.key.table_name + )) + })?; + let status: Option = row.get_by_name("restore_status").ok().flatten(); + let status = status.ok_or_else(|| { + StorageError::TableNotFound(format!( + "restore target {} has no job record", + self.key.table_name + )) + })?; + let generation: Option = + row.get_by_name("restore_owner_generation").ok().flatten(); + let heartbeat_at: Option = row.get_by_name("restore_heartbeat_at").ok().flatten(); + Ok(JobRow { + status: JobStatus::parse(&status)?, + owner_generation: generation_from_column(generation), + heartbeat_at: heartbeat_at.map_or(0.0, millis_to_seconds), + failure_reason: row.get_by_name("restore_failure_reason").ok().flatten(), + }) + }) + } +} + impl BackupCatalog for CassandraEngine { fn catalog_id(&self) -> BoxFuture<'_, Result> { Box::pin(async move { @@ -1399,13 +1655,11 @@ impl BackupCatalog for CassandraEngine { // window, every RUNNING record the scans saw is listed). A range // scan has no Paxos read (a serial level resolves to a local // quorum there), so it stays at quorum. - let columns = "account_id, table_name, table_id, table_status, \ - restore_source_backup_arn, restore_kind, restore_status, \ - restore_owner_generation, restore_heartbeat_at, restore_completed_at"; let mut scanned = Vec::new(); for status in ["RUNNING", "FAILED"] { let query = format!( - "SELECT {columns} FROM {}.tables WHERE restore_status = ? ALLOW FILTERING", + "SELECT {RESTORE_JOB_COLUMNS} FROM {}.tables WHERE restore_status = ? \ + ALLOW FILTERING", self.catalog_keyspace() ); let rows = crate::cassandra_util::query_rows_quorum::( @@ -1427,15 +1681,16 @@ impl BackupCatalog for CassandraEngine { row.get_by_name("restore_heartbeat_at").ok().flatten(); let completed_at: Option = row.get_by_name("restore_completed_at").ok().flatten(); - let column = |name: &str| -> Result { - crate::cassandra_util::get_column(&row, name, "dead restore jobs") - }; // A RUNNING record always carries a heartbeat (the insert // or the backfill wrote it); one without is as dead as the // epoch. let age_of = if status == "FAILED" { let target = RestoreTargetRow { - table_id: column("table_id")?, + table_id: crate::cassandra_util::get_column( + &row, + "table_id", + "dead restore jobs", + )?, table_status: row.get_by_name("table_status").ok().flatten(), source_backup_arn: row .get_by_name("restore_source_backup_arn") @@ -1453,27 +1708,227 @@ impl BackupCatalog for CassandraEngine { if age_of.is_some_and(|at| at >= cutoff) { continue; } - let kind = column("restore_kind")?; - let generation: Option = - row.get_by_name("restore_owner_generation").ok().flatten(); - dead.push(RestoreJobRow { - target: RestoreTargetKey { - account_id: column("account_id")?, - table_name: column("table_name")?, - table_id: column("table_id")?, - }, - source_backup_arn: column("restore_source_backup_arn")?, - kind: RestoreKind::parse(&kind)?, - status: JobStatus::parse(status)?, - owner_generation: generation_from_column(generation), - heartbeat_at: heartbeat_at.map_or(0.0, millis_to_seconds), - }); + dead.push(restore_job_row_from(&row)?); } } Ok(dead) }) } + fn claim_restore<'a>( + &'a self, + account_id: &'a str, + input: CreateTableInput, + registration: &'a RestoreRegistrationRow, + ) -> BoxFuture<'a, Result> { + Box::pin(async move { + Self::validate_account_id(account_id)?; + self.require_migrated_catalog().await?; + let (description, generation) = self + .register_manifest_restore_target(account_id, input, registration) + .await?; + let key = RestoreTargetKey { + account_id: account_id.to_owned(), + table_name: description.table_name.clone(), + table_id: description.table_id.clone(), + }; + Ok(( + description, + Box::new(CassandraRestoreClaim { + engine: self.clone(), + key, + generation, + }) as Box, + )) + }) + } + + fn claim_restore_dead<'a>( + &'a self, + key: &'a RestoreTargetKey, + stale_after: Duration, + spool_host: Option<&'a str>, + ) -> BoxFuture<'a, Result>, StorageError>> { + Box::pin(async move { + // The record is read at LOCAL_SERIAL, so a heartbeat an LWT just + // wrote is seen; dead is `restore_heartbeat_at` older than the + // window on the coordinator's clock. The takeover is conditioned + // on the generation, RUNNING, the manifest kind and the heartbeat + // as read: a beat landing between the read and the write (the + // owner alive after all), another taker's generation, or a record + // that left RUNNING leaves the row and answers `None`. The write + // sets the generation, the claimant's `spool_host` (`None` clears + // it) and `restore_heartbeat_at` to the coordinator's clock, so a + // second taker's read finds the record live at once (the trait + // doc says why a restore record is refreshed where a backup row + // is not). A legacy record is not a manifest restore's to take. + let catalog = self.catalog_keyspace(); + let read = format!( + "SELECT table_id, restore_status, restore_kind, restore_owner_generation, \ + restore_heartbeat_at, restore_source_backup_arn FROM {catalog}.tables \ + WHERE account_id = ? AND table_name = ?" + ); + let rows = crate::cassandra_util::query_rows_serial::( + &self.session, + &read, + cdrs_tokio::query_values!(key.account_id.as_str(), key.table_name.as_str()), + "dead restore job", + ) + .await?; + let Some(row) = rows.first() else { + return Ok(None); + }; + let table_id: Option = row.get_by_name("table_id").ok().flatten(); + if table_id.as_deref() != Some(key.table_id.as_str()) { + return Ok(None); + } + let status: Option = row.get_by_name("restore_status").ok().flatten(); + if status.as_deref() != Some(JobStatus::Running.as_str()) { + return Ok(None); + } + let owner_generation: Option = + row.get_by_name("restore_owner_generation").ok().flatten(); + let heartbeat_at: Option = row.get_by_name("restore_heartbeat_at").ok().flatten(); + let source_backup_arn: Option = + row.get_by_name("restore_source_backup_arn").ok().flatten(); + let now = coordinator_now_millis(self).await?; + let stale_ms = i64::try_from(stale_after.as_millis()).unwrap_or(i64::MAX); + if heartbeat_at.is_some_and(|at| at >= now.saturating_sub(stale_ms)) { + return Ok(None); + } + // A test may interpose a beat here, between the read and the + // write, to prove the heartbeat condition refuses the takeover. + crate::test_hooks::before_restore_takeover(&key.table_id).await; + let generation = new_generation(); + let take_over = format!( + "UPDATE {catalog}.tables SET restore_owner_generation = ?, \ + restore_spool_host = ?, restore_heartbeat_at = toTimestamp(now()) \ + WHERE account_id = ? AND table_name = ? \ + IF restore_owner_generation = ? AND restore_status = 'RUNNING' \ + AND restore_kind = 'manifest' AND restore_heartbeat_at = ?" + ); + let taken = conditional_write( + &self.session, + &take_over, + QueryValues::SimpleValues(vec![ + Value::from(generation_column(generation)), + match spool_host { + Some(host) => Value::from(host), + None => Value::from(Option::::None), + }, + Value::from(key.account_id.as_str()), + Value::from(key.table_name.as_str()), + Value::from(owner_generation), + Value::from(heartbeat_at), + ]), + ) + .await?; + let claimed = match taken { + WriteOutcome::Matched => true, + WriteOutcome::NotMatched => false, + // A takeover whose answer was lost may have committed: a + // record carrying this call's generation is this call's claim. + WriteOutcome::Unknown => { + let rows = crate::cassandra_util::query_rows_serial::( + &self.session, + &format!( + "SELECT restore_owner_generation FROM {catalog}.tables \ + WHERE account_id = ? AND table_name = ?" + ), + cdrs_tokio::query_values!(key.account_id.as_str(), key.table_name.as_str()), + "dead restore job", + ) + .await?; + let current: Option = rows + .first() + .and_then(|r| r.get_by_name("restore_owner_generation").ok().flatten()); + current == Some(generation_column(generation)) + } + }; + if !claimed { + return Ok(None); + } + // The backup's `active_restores` entry for the target, re-applied + // when it is absent: the registration writes the target's row and + // its DDL first and the entry second, so an owner that died + // between the two left a resumable record whose backup nothing + // held in use. Without the entry `start_backup_delete` would + // succeed under the resumed load. + if let Some(backup_arn) = source_backup_arn { + self.reapply_restore_registration(key, &backup_arn).await?; + } + Ok(Some(Box::new(CassandraRestoreClaim { + engine: self.clone(), + key: key.clone(), + generation, + }) as Box)) + }) + } + + fn read_restore_job<'a>( + &'a self, + key: &'a RestoreTargetKey, + ) -> BoxFuture<'a, Result, StorageError>> { + Box::pin(async move { + let query = format!( + "SELECT {RESTORE_JOB_COLUMNS} FROM {}.tables \ + WHERE account_id = ? AND table_name = ?", + self.catalog_keyspace() + ); + let rows = crate::cassandra_util::query_rows_quorum::( + &self.session, + &query, + cdrs_tokio::query_values!(key.account_id.as_str(), key.table_name.as_str()), + "restore job", + ) + .await?; + let Some(row) = rows.first() else { + return Ok(None); + }; + // No record, or a table re-created under the name with another id, + // is not this target's job. + let table_id: Option = row.get_by_name("table_id").ok().flatten(); + let status: Option = row.get_by_name("restore_status").ok().flatten(); + if table_id.as_deref() != Some(key.table_id.as_str()) || status.is_none() { + return Ok(None); + } + Ok(Some(restore_job_row_from(row)?)) + }) + } + + fn list_restore_jobs<'a>( + &'a self, + status: Option, + ) -> BoxFuture<'a, Result, StorageError>> { + Box::pin(async move { + // CQL filters one equality per scan, so a status filter is one + // scan and "every record" is a scan per restore status. + let statuses = match status { + Some(status) => vec![status], + None => vec![JobStatus::Running, JobStatus::Done, JobStatus::Failed], + }; + let mut jobs = Vec::new(); + for status in statuses { + let query = format!( + "SELECT {RESTORE_JOB_COLUMNS} FROM {}.tables WHERE restore_status = ? \ + ALLOW FILTERING", + self.catalog_keyspace() + ); + let rows = crate::cassandra_util::query_rows_quorum::( + &self.session, + &query, + cdrs_tokio::query_values!(status.as_str()), + "restore jobs", + ) + .await?; + for row in rows { + jobs.push(restore_job_row_from(&row)?); + } + } + Ok(jobs) + }) + } + fn fail_restore_job<'a>( &'a self, key: &'a RestoreTargetKey, diff --git a/crates/storage-cassandra/src/backup_engine.rs b/crates/storage-cassandra/src/backup_engine.rs index 6064211b1..78a3527d8 100755 --- a/crates/storage-cassandra/src/backup_engine.rs +++ b/crates/storage-cassandra/src/backup_engine.rs @@ -81,11 +81,13 @@ use cdrs_tokio::types::map::Map; use cdrs_tokio::types::{AsRust, IntoRustByName, rows::Row}; use extenddb_core::types::{ AttributeDefinition, BackupDescription, BackupDetails, BackupSummary, - ContinuousBackupsDescription, DeleteTableInput, GsiInput, KeySchemaElement, LsiInput, - PointInTimeRecoveryDescription, ProvisionedThroughput, SourceTableDetails, TableDescription, - TableKeyInfo, + ContinuousBackupsDescription, CreateTableInput, DeleteTableInput, GsiInput, KeySchemaElement, + LsiInput, PointInTimeRecoveryDescription, ProvisionedThroughput, SourceTableDetails, + TableDescription, TableKeyInfo, +}; +use extenddb_storage::backup_catalog::{ + AbandonedLegacyBackup, RestoreRegistrationRow, new_generation, }; -use extenddb_storage::backup_catalog::{AbandonedLegacyBackup, new_generation}; use extenddb_storage::backup_definition::{ BACKUP_DEFINITION_VERSION, BackupTableDefinition, throughput_from_catalog, }; @@ -107,8 +109,9 @@ use crate::delete_table::DeleteTableCaller; const BACKUP_ITEMS_PER_BUCKET: i64 = 64; const BACKUP_SCAN_PAGE_SIZE: i64 = 1_000; -/// Items of one bucket a restore copy writes concurrently. -const RESTORE_WRITE_CONCURRENCY: usize = 8; +/// Items of one bucket a restore copy writes concurrently. Shared by the +/// legacy copy and the store-backed `bulk_load`. +pub(crate) const RESTORE_WRITE_CONCURRENCY: usize = 8; /// Heartbeats that error in a row after which a job stops: three beats could /// not confirm the claim, so the row may already be another owner's. One @@ -568,6 +571,26 @@ impl CassandraEngine { Ok(backup) } + /// The AVAILABLE catalog-index backup (`store_kind` set) `backup_arn` + /// names; an in-catalog row is "not found" here, the mirror of + /// [`Self::load_available_backup`]. A store-backed restore reads this + /// row, not the legacy one. + async fn load_available_index_backup( + &self, + account_id: &str, + backup_arn: &str, + ) -> Result { + let backup = self + .load_stored_backup(account_id, backup_arn) + .await? + .filter(|backup| !backup.is_in_catalog()) + .ok_or_else(|| backup_not_found(backup_arn))?; + if backup.backup_status != "AVAILABLE" { + return Err(backup_not_found(backup_arn)); + } + Ok(backup) + } + /// Delete a backup's `backup_items` buckets: the `bucket_count(item_count)` /// the row accounts for, then every further bucket until an empty one. /// The create writes buckets in order and publishes `item_count` last, so @@ -801,6 +824,24 @@ impl CassandraEngine { self.ensure_backup_not_restoring(&backup).await } + /// [`Self::registration_refusal`] for a store-backed (index) backup: it + /// re-reads the catalog-index row rather than the in-catalog one. + async fn registration_refusal_index( + &self, + account_id: &str, + backup_arn: &str, + ) -> Result<(), StorageError> { + let backup = self + .load_stored_backup(account_id, backup_arn) + .await? + .filter(|backup| !backup.is_in_catalog()) + .ok_or_else(|| backup_not_found(backup_arn))?; + if backup.backup_status != "AVAILABLE" { + return Err(backup_not_found(backup_arn)); + } + self.ensure_backup_not_restoring(&backup).await + } + async fn unregister_restore( &self, account_id: &str, @@ -996,6 +1037,30 @@ impl CassandraEngine { job: &RestoreJob, item_count: i64, table_size_bytes: i64, + ) -> Result { + self.activate_restore_target_row( + &job.account_id, + &job.target_name, + &job.table_id, + job.generation, + item_count, + table_size_bytes, + ) + .await + } + + /// The ACTIVE flip's lightweight transaction, by the target's address and + /// the claim's generation. Shared by the legacy copy + /// ([`Self::activate_restore_target`]) and the store-backed + /// [`extenddb_storage::restore_engine::RestoreEngine::finish_restore`]. + pub(crate) async fn activate_restore_target_row( + &self, + account_id: &str, + target_name: &str, + table_id: &str, + generation: u64, + item_count: i64, + table_size_bytes: i64, ) -> Result { let query = format!( "UPDATE {}.tables SET table_status = 'ACTIVE', status_transition_at = NULL, \ @@ -1012,10 +1077,10 @@ impl CassandraEngine { cdrs_tokio::query_values!( item_count, table_size_bytes, - job.account_id.as_str(), - job.target_name.as_str(), - job.table_id.as_str(), - crate::backup_catalog::generation_column(job.generation) + account_id, + target_name, + table_id, + crate::backup_catalog::generation_column(generation) ), ) .await?; @@ -1375,6 +1440,9 @@ impl CassandraEngine { source_table_arn, restore_date_time_ms, owner_generation: generation, + kind: extenddb_storage::backup_catalog::RestoreKind::Legacy, + store: None, + spool_host: None, }; let table_id = columns.table_id.clone(); @@ -1453,6 +1521,218 @@ impl CassandraEngine { }; Ok((description, job, heartbeat)) } + + /// Claim the target a registration created for its own removal: CREATING + /// to DELETING and the record FAILED with `reason`, one lightweight + /// transaction conditioned on the table id, on CREATING, on the + /// registrant's generation and on RUNNING (the write + /// [`Self::abandon_restore_target`] makes). `false` when the record is no + /// longer this registration's: the registration ran no heartbeat during + /// the target's DDL, a reconciler whose stale window the DDL outlasted + /// took the record over, and the target is the successor's to load, so + /// the registrant removes nothing. The state a crash between the claim + /// and the drop leaves (DELETING with its registration and a FAILED + /// record) is the one `remove_restore_target` and the FAILED-target duty + /// resume from. + async fn claim_registered_target_for_removal( + &self, + account_id: &str, + target_name: &str, + table_id: &str, + generation: u64, + reason: &str, + ) -> Result { + let query = format!( + "UPDATE {}.tables SET table_status = 'DELETING', restore_status = 'FAILED', \ + restore_failure_reason = ?, restore_completed_at = toTimestamp(now()) \ + WHERE account_id = ? AND table_name = ? \ + IF table_id = ? AND table_status = 'CREATING' AND restore_owner_generation = ? \ + AND restore_status = 'RUNNING'", + self.catalog_keyspace() + ); + let result = crate::cassandra_util::query_lwt( + &self.session, + &query, + cdrs_tokio::query_values!( + reason, + account_id, + target_name, + table_id, + crate::backup_catalog::generation_column(generation) + ), + ) + .await?; + crate::cassandra_util::lwt_applied(&result) + } + + /// The error of a registration whose record another generation took over + /// while its target's DDL ran: the target is left to the record's owner. + fn registration_claim_lost(target_name: &str, backup_arn: &str) -> StorageError { + StorageError::Internal(format!( + "restore of table {target_name} from {backup_arn}: the registration lost its \ + record to a takeover during the target's creation (the record is not RUNNING \ + under the registration's generation); the target is left to the record's owner, \ + which finishes the restore" + )) + } + + /// Register a store-backed (manifest) restore target and return the + /// CREATING description and the claim's generation. The manifest analogue + /// of [`Self::register_restore_target`]: it reads the catalog-index backup + /// row (`store_kind` set) rather than the in-catalog one, writes the job + /// record with `restore_kind = 'manifest'` and the store columns, and + /// starts no self-managed heartbeat (the engine heartbeats through the + /// claim). The ordering against DeleteBackup is the same `active_restores` + /// lightweight transaction on the backup's partition, which + /// [`crate::backup_catalog`]'s `start_backup_delete` conflicts with. Two + /// concurrent registrations of one backup are ordered by that transaction + /// too: the in-use pre-read (`ensure_backup_not_restoring`) is not atomic + /// with it, so both may pass the pre-read and create their targets, but + /// its `active_restores = null` condition admits one; the other finds the + /// registered target in its re-read, is `BackupInUse` naming it, and + /// removes its own target, so one target is left. With no heartbeat + /// during the DDL, a registration whose DDL outlasts the stale window can + /// lose its record to a reconciler's takeover; its removal of the target + /// on a refusal is therefore conditioned on its own generation + /// ([`Self::claim_registered_target_for_removal`]) and removes nothing + /// then, answering the lost claim instead of the refusal. + pub(crate) async fn register_manifest_restore_target( + &self, + account_id: &str, + input: CreateTableInput, + registration: &RestoreRegistrationRow, + ) -> Result<(TableDescription, u64), StorageError> { + let backup_arn = registration.backup.backup_arn.clone(); + // The name is checked first (unfenced), so a restore into a CREATING + // restore target's name is refused as the name (TableBeingRestored), + // not as the backup, as on the other backends and the legacy path. + if let Some(source_backup_arn) = self + .creating_restore_target_under_name(account_id, &input.table_name) + .await? + { + return Err(StorageError::TableBeingRestored { + table_name: input.table_name.clone(), + backup_arn: source_backup_arn, + }); + } + let backup = self + .load_available_index_backup(account_id, &backup_arn) + .await?; + self.ensure_backup_not_restoring(&backup).await?; + + let target_name = input.table_name.clone(); + // The claim is the row itself: a fresh generation, written by the + // IF NOT EXISTS insert, fences every later write of this restore. + let generation = new_generation(); + #[allow(clippy::cast_possible_truncation)] + let restore_date_time_ms = (registration.restore_date_time * 1_000.0).round() as i64; + let columns = RestoreTargetColumns { + table_id: uuid::Uuid::new_v4().to_string(), + source_backup_arn: backup_arn.clone(), + source_table_arn: registration.source_table_arn.clone(), + restore_date_time_ms, + owner_generation: generation, + kind: extenddb_storage::backup_catalog::RestoreKind::Manifest, + store: Some(registration.store.clone()), + spool_host: registration.spool_host.clone(), + }; + let table_id = columns.table_id.clone(); + + // Step 1: the target row (IF NOT EXISTS reserves the name), its + // indexes and its data tables, exactly as `input` describes. + let description = match self + .create_table_for_restore_manifest(account_id, input, &columns) + .await + { + Ok(description) => description, + Err(StorageError::TableAlreadyExists(name)) => { + return Err(StorageError::TableAlreadyExists(name)); + } + Err(error) => { + if let Err(cleanup) = self + .drop_restore_target(account_id, &target_name, &table_id, &backup_arn) + .await + { + tracing::warn!( + "Restore of table {target_name}: removing the partly created \ + target failed ({cleanup}); the reconciler removes it" + ); + } + return Err(error); + } + }; + + // Step 2: register on the backup, fenced on AVAILABLE and on no other + // registration. A refusal caused only by stale registrations is + // retried once they are cleared, a bounded number of times. A test + // may hold here, between the target's creation and the fence, to + // move the backup to DELETING. + crate::test_hooks::before_restore_registration(&backup_arn).await; + let mut registered = Ok(false); + for _ in 0..REGISTRATION_ATTEMPTS { + registered = self + .register_restore(account_id, &backup_arn, &table_id, &target_name) + .await; + match registered { + Ok(true) => break, + Ok(false) => { + if let Err(refusal) = self + .registration_refusal_index(account_id, &backup_arn) + .await + { + registered = Err(refusal); + break; + } + } + Err(_) => break, + } + } + if !matches!(registered, Ok(true)) { + // The removal is fenced on this registration's generation: a + // record a reconciler took over during the DDL is the + // successor's, and so is the target. + let refusal = registration_exhausted(registered, &backup_arn); + let owned = match self + .claim_registered_target_for_removal( + account_id, + &target_name, + &table_id, + generation, + &format!("the registration was refused: {refusal}"), + ) + .await + { + Ok(owned) => owned, + Err(error) => { + tracing::warn!( + "Restore of table {target_name}: claiming the unregistered target \ + for removal failed ({error}); the reconciler removes it" + ); + return Err(refusal); + } + }; + if !owned { + tracing::warn!( + "Restore of table {target_name} from {backup_arn}: the registration's \ + record was taken over during the target's creation; the target is left \ + to its new owner" + ); + return Err(Self::registration_claim_lost(&target_name, &backup_arn)); + } + if let Err(cleanup) = self + .drop_restore_target(account_id, &target_name, &table_id, &backup_arn) + .await + { + tracing::warn!( + "Restore of table {target_name}: removing the unregistered target \ + failed ({cleanup}); the reconciler removes it" + ); + } + return Err(refusal); + } + + Ok((description, generation)) + } } impl LegacyBackupReader for CassandraEngine { diff --git a/crates/storage-cassandra/src/create_table.rs b/crates/storage-cassandra/src/create_table.rs index 904b4ff83..02e9dcf44 100644 --- a/crates/storage-cassandra/src/create_table.rs +++ b/crates/storage-cassandra/src/create_table.rs @@ -9,6 +9,7 @@ use extenddb_core::types::{ ProvisionedThroughputDescription, RestoreSummary, SseDescription, SseType, TableDescription, TableStatus, }; +use extenddb_storage::backup_catalog::{RestoreKind, StoreIdentity}; use extenddb_storage::error::StorageError; use extenddb_storage::util::{index_arn, stream_arn, table_arn}; @@ -123,6 +124,25 @@ impl CassandraEngine { Ok((description, heartbeat)) } + /// Create a store-backed (manifest) restore target: CREATING, carrying + /// the `restore_kind = 'manifest'` job record and the store columns in + /// the one `INSERT ... IF NOT EXISTS`. No self-managed [`JobHeartbeat`] + /// is started: the engine owns the manifest restore's heartbeat through + /// the [`extenddb_storage::backup_catalog::JobClaim`] this registration + /// returns. The target's DDL that follows the insert is bounded by a + /// few schema statements, inside the engine's registration timeout. + pub(crate) async fn create_table_for_restore_manifest( + &self, + account_id: &str, + input: CreateTableInput, + restore: &RestoreTargetColumns, + ) -> Result { + let (description, _heartbeat) = self + .create_table_impl_inner(account_id, input, Some(restore)) + .await?; + Ok(description) + } + async fn create_table_impl_inner( &self, account_id: &str, @@ -268,8 +288,8 @@ impl CassandraEngine { ]; let mut placeholders = vec!["?"; values.len()]; if let Some(r) = restore { - // The job record: RUNNING under the claim's generation, kind - // legacy, the heartbeat stamped on the coordinator's clock. + // The job record: RUNNING under the claim's generation, the + // restore kind, the heartbeat stamped on the coordinator's clock. columns.push_str( ", restore_source_backup_arn, restore_source_table_arn, restore_date_time, \ restore_status, restore_kind, restore_owner_generation, restore_heartbeat_at", @@ -279,11 +299,29 @@ impl CassandraEngine { Value::from(r.source_table_arn.as_str()), Value::from(r.restore_date_time_ms), Value::from("RUNNING"), - Value::from("legacy"), + Value::from(r.kind.as_str()), Value::from(crate::backup_catalog::generation_column(r.owner_generation)), ]); placeholders.extend(["?"; 6]); placeholders.push("toTimestamp(now())"); + // A store-backed restore records the store it reads and the + // claimant's spool, so the reconciler can resume it. + if let Some(store) = &r.store { + columns.push_str( + ", restore_store_kind, restore_store_root, restore_store_id, \ + restore_spool_host", + ); + values.extend([ + Value::from(store.store_kind.as_str()), + Value::from(store.store_root.as_str()), + Value::from(store.store_id.as_str()), + match r.spool_host.as_deref() { + Some(host) => Value::from(host), + None => Value::NotSet, + }, + ]); + placeholders.extend(["?"; 4]); + } } let placeholders = placeholders.join(", "); let insert_table_cql = format!( @@ -337,9 +375,11 @@ impl CassandraEngine { } } - // A restore target's heartbeat runs from here, before the DDL below. - // An error return from this function drops it, which stops it. - let heartbeat = restore.map(|r| { + // A legacy restore target's heartbeat runs from here, before the DDL + // below, and is dropped (stopped) on an error return. A store-backed + // (manifest) restore target has no self-managed heartbeat: the engine + // heartbeats it through the claim the registration returns. + let heartbeat = restore.filter(|r| r.kind == RestoreKind::Legacy).map(|r| { self.start_restore_heartbeat( account_id, &input.table_name, @@ -675,4 +715,16 @@ pub(crate) struct RestoreTargetColumns { /// The claim's fencing token (`tables.restore_owner_generation`): every /// write the copy makes to the target row is conditioned on it. pub(crate) owner_generation: u64, + /// `legacy` for an in-catalog restore (whose copy reads `backup_items`), + /// `manifest` for a store-backed one (whose copy the engine streams from + /// the store). A legacy target carries a self-managed heartbeat; a + /// manifest target is heartbeated by the engine's claim. + pub(crate) kind: RestoreKind, + /// The store a manifest restore reads its backup from; `None` on a legacy + /// restore, which reads the catalog. + pub(crate) store: Option, + /// `:` of the instance downloading the backup into + /// its spool; `None` for a filesystem store (read in place) and on a + /// legacy restore. + pub(crate) spool_host: Option, } diff --git a/crates/storage-cassandra/src/data/index.rs b/crates/storage-cassandra/src/data/index.rs index 4ebd72dac..0ae8ae05e 100644 --- a/crates/storage-cassandra/src/data/index.rs +++ b/crates/storage-cassandra/src/data/index.rs @@ -244,7 +244,9 @@ pub(crate) fn effective_delay(idx: &IndexMeta, system_default: u64) -> u64 { /// Only processes indexes where `effective_delay == 0`. Async indexes are /// handled by async queue (Phase 2). /// -/// Adds parameterized statements to the batch builder. +/// Adds parameterized statements to the batch builder through +/// [`write_index_row_change`], which skips the old row's `DELETE` when the +/// new item rewrites the same row (#742). #[allow(clippy::too_many_arguments)] pub fn sync_indexes( batch: &mut BatchQueryBuilder, @@ -263,45 +265,136 @@ pub fn sync_indexes( let idx_table = index_table_name(&idx.index_id); let idx_sks = all_sort_key_info(&idx.key_schema, attr_defs); let base_sks = all_sort_key_info(base_key_schema, attr_defs); + write_index_row_change( + batch, + account_keyspace, + &idx_table, + &idx.key_schema, + base_key_schema, + &idx_sks, + &base_sks, + &idx.projection, + old_item, + new_item, + )?; + } + Ok(()) +} - // Delete old index row if the old item had index keys - if let Some(old) = old_item - && item_has_index_keys(old, &idx.key_schema) - { - delete_index_row_multi( - batch, - account_keyspace, - &idx_table, - old, - &idx.key_schema, - base_key_schema, - &idx_sks, - &base_sks, - )?; - } +/// Add to `batch` the statements that take one index table from the row +/// `old_item` produces to the row `new_item` produces: a `DELETE` of the old +/// row when the old item has the index's keys, an `INSERT` of the new row +/// when the new item has them. The one writer of an index row change: the +/// synchronous path (`sync_indexes`) and the GSI worker (`gsi_apply_index`) +/// both call it. +/// +/// When the two items address the same row (the same index key values and +/// the same base key, compared as the row's key columns are bound), the +/// `DELETE` is skipped and the `INSERT` alone rewrites the row. Every +/// statement of a logged batch carries the batch's timestamp and a tombstone +/// wins a tie with a write at the same timestamp, so a `DELETE` and an +/// `INSERT` of one row in one batch leave the row deleted: a PutItem or +/// UpdateItem of an item whose index keys did not change (an unchanged item +/// included) would remove the item's GSI and LSI rows (GitHub issue #742). An +/// index row's projection lives in its one `item_data` column, and the +/// `INSERT` writes that column with every key column, so the `INSERT` alone +/// replaces the projection whole; a projected attribute the new item dropped +/// is gone from the row because the column is rewritten, not merged. +#[allow(clippy::too_many_arguments)] +pub(crate) fn write_index_row_change( + batch: &mut BatchQueryBuilder, + account_keyspace: &str, + idx_table: &str, + index_ks: &[KeySchemaElement], + base_ks: &[KeySchemaElement], + idx_sks: &[(&str, ScalarAttributeType)], + base_sks: &[(&str, ScalarAttributeType)], + projection: &Projection, + old_item: Option<&Item>, + new_item: Option<&Item>, +) -> Result<(), StorageError> { + let old_row = old_item.filter(|old| item_has_index_keys(old, index_ks)); + let new_row = new_item.filter(|new| item_has_index_keys(new, index_ks)); + let same_row = match (old_row, new_row) { + (Some(old), Some(new)) => same_index_row(old, new, index_ks, base_ks, idx_sks, base_sks)?, + _ => false, + }; - // Insert new index row if the new item has index keys - if let Some(new) = new_item - && item_has_index_keys(new, &idx.key_schema) - { - let projected = - project_item_for_index(new, &idx.key_schema, base_key_schema, &idx.projection); - insert_index_row_multi( - batch, - account_keyspace, - &idx_table, - new, - &projected, - &idx.key_schema, - base_key_schema, - &idx_sks, - &base_sks, - )?; - } + if let Some(old) = old_row + && !same_row + { + delete_index_row_multi( + batch, + account_keyspace, + idx_table, + old, + index_ks, + base_ks, + idx_sks, + base_sks, + )?; + } + + if let Some(new) = new_row { + let projected = project_item_for_index(new, index_ks, base_ks, projection); + insert_index_row_multi( + batch, + account_keyspace, + idx_table, + new, + &projected, + index_ks, + base_ks, + idx_sks, + base_sks, + )?; } Ok(()) } +/// Whether `old` and `new` produce the same row of one index table: the same +/// index partition key text, index sort key values, base partition key text +/// and base sort key values, the primary key as `insert_index_row_multi` and +/// `delete_index_row_multi` bind it. Both items must carry the index's keys. +/// A numeric sort key compares by value, as Cassandra's `decimal` clustering +/// column orders it. +fn same_index_row( + old: &Item, + new: &Item, + index_ks: &[KeySchemaElement], + base_ks: &[KeySchemaElement], + idx_sks: &[(&str, ScalarAttributeType)], + base_sks: &[(&str, ScalarAttributeType)], +) -> Result { + if composite_pk_to_text(old, index_ks)? != composite_pk_to_text(new, index_ks)? + || composite_pk_to_text(old, base_ks)? != composite_pk_to_text(new, base_ks)? + { + return Ok(false); + } + for &(sk_name, sk_type) in idx_sks.iter().chain(base_sks.iter()) { + let same = match (old.get(sk_name), new.get(sk_name)) { + (Some(a), Some(b)) => sort_key_eq(&parse_sk(a, sk_type)?, &parse_sk(b, sk_type)?), + (None, None) => true, + _ => false, + }; + if !same { + return Ok(false); + } + } + Ok(true) +} + +/// Equality of two sort key values as the clustering column compares them: +/// text and bytes byte for byte, numbers by value. +fn sort_key_eq(a: &SortKeyValue, b: &SortKeyValue) -> bool { + match (a, b) { + (SortKeyValue::S(a), SortKeyValue::S(b)) => a == b, + (SortKeyValue::N(a), SortKeyValue::N(b)) => a == b, + (SortKeyValue::B(a), SortKeyValue::B(b)) => a == b, + _ => false, + } +} + /// Enqueue one `gsi_pending` row per async GSI into the in-progress logged /// batch. Returns the number of rows enqueued. /// diff --git a/crates/storage-cassandra/src/lib.rs b/crates/storage-cassandra/src/lib.rs index 54b038946..f45499b8a 100644 --- a/crates/storage-cassandra/src/lib.rs +++ b/crates/storage-cassandra/src/lib.rs @@ -33,6 +33,7 @@ mod metadata_engine; pub mod migrations; pub mod operations; pub(crate) mod propagation_hold; +mod restore_engine; mod snapshot_engine; mod stream_engine; pub mod stream_util; diff --git a/crates/storage-cassandra/src/restore_engine.rs b/crates/storage-cassandra/src/restore_engine.rs new file mode 100644 index 000000000..da585eae5 --- /dev/null +++ b/crates/storage-cassandra/src/restore_engine.rs @@ -0,0 +1,483 @@ +// Copyright 2026 ExtendDB contributors +// SPDX-License-Identifier: Apache-2.0 + +//! The data-plane side of a store-backed restore on Cassandra: the +//! [`RestoreEngine`] methods the engine's restore job calls. +//! +//! `bulk_load` writes one verified data file's items into the CREATING +//! target through the same put path the legacy copy uses +//! ([`CassandraEngine::put_restored_item`], every GSI and LSI in the item's +//! own logged batch), `RESTORE_WRITE_CONCURRENCY` in flight, in batches of +//! at most [`RESTORE_BATCH_ITEMS`] items or [`RESTORE_BATCH_BYTES`] of +//! serialized item text (a batch closes at the first item that takes its +//! `item_data` text past the bound, so the batch term of a restore's memory +//! is bounded in bytes, not only in items). Before each batch's writes it +//! reads the record once at LOCAL_QUORUM and ends the file as a lost claim +//! when the record is not RUNNING under the claim's generation, so a writer +//! whose record another instance took over while it stalled lands at most +//! the one batch it was in; after each batch it records the file's progress +//! with a lightweight transaction conditioned on the restore generation and +//! on the record being RUNNING; `[applied] = false` there is the same lost +//! claim, and the load stops without touching the target again; an unknown +//! outcome of that transaction is decided by a LOCAL_SERIAL re-read of the +//! record (applied when it carries the values written, one retry when it +//! carries the earlier ones, the lost claim when it is not RUNNING under the +//! generation; `write_restore_progress`). The writes +//! are idempotent upserts, so a re-applied batch after a resume rewrites the +//! same base and index rows (`occ_write` bumps each base row's `version` and +//! writes its item again; each GSI and LSI row is rewritten by its `INSERT` +//! alone, the old row's `DELETE` skipped when the item's index keys did not +//! change, since a `DELETE` and an `INSERT` of one row in one logged batch +//! would leave the row deleted, #742), and a key the backup holds twice +//! collapses into one row: a shortfall `count_restore_target` finds against +//! the manifest's total is the duplicate (the engine's count check, 5.3 step +//! 6). An empty file records its progress too (`items_applied = start_at`), +//! so a resume does not re-read it. +//! +//! `finish_restore` is [`CassandraEngine::activate_restore_target_row`]'s +//! lightweight transaction (DONE, the table and its indexes ACTIVE, the +//! counts, conditioned on CREATING, RUNNING and the generation). The index +//! rows are created ACTIVE with the table, so Cassandra never returns +//! [`Finished::Pending`]. + +use cdrs_tokio::types::IntoRustByName; +use extenddb_core::types::{Item, TableKeyInfo}; +use extenddb_storage::backup_catalog::{ + JobClaim, JobStatus, RESTORE_CASSANDRA_BEFORE_PROGRESS, RestoreTargetKey, WriteOutcome, + restore_pause_point, +}; +use extenddb_storage::error::StorageError; +use extenddb_storage::restore_engine::{BulkLoadFile, Finished, RestoreEngine}; +pub(crate) use extenddb_storage::restore_engine::{RESTORE_BATCH_BYTES, RESTORE_BATCH_ITEMS}; +use futures::future::BoxFuture; +use futures::stream::BoxStream; +use futures::{StreamExt, TryStreamExt}; + +use crate::CassandraEngine; +use crate::backup_engine::RESTORE_WRITE_CONCURRENCY; +use crate::data::ddl::data_table_name; + +/// The error of a `bulk_load` whose progress write found the record no longer +/// this claim's: the reconciler took it over while the file was loading, and +/// whoever holds the record now owns the target. Nothing further is touched. +fn claim_lost(table_name: &str) -> StorageError { + StorageError::Internal(format!( + "restore of table {table_name}: the claim on the target was lost while a data file \ + loaded (the record is not RUNNING under this generation); the record's owner \ + finishes it" + )) +} + +impl CassandraEngine { + /// The key shape of a CREATING restore target, read from its `tables` + /// row by hand (the put path refuses a CREATING table through + /// `fetch_table_key_info`, so the restore builds the info itself, as the + /// legacy copy does). `TableNotFound` when `key` is not a CREATING + /// restore target with this table id, as the other backends answer. The + /// put path reads the target's indexes from the catalog by `table_id`, + /// so the routing fields are left empty. + async fn restore_target_key_info( + &self, + key: &RestoreTargetKey, + ) -> Result { + let query = format!( + "SELECT table_id, table_status, restore_source_backup_arn, key_schema, \ + attribute_definitions FROM {}.tables WHERE account_id = ? AND table_name = ?", + self.catalog_keyspace() + ); + let rows = crate::cassandra_util::query_rows_quorum::( + &self.session, + &query, + cdrs_tokio::query_values!(key.account_id.as_str(), key.table_name.as_str()), + "restore target key info", + ) + .await?; + let row = rows.first().filter(|row| { + let id: Option = row.get_by_name("table_id").ok().flatten(); + let status: Option = row.get_by_name("table_status").ok().flatten(); + let source: Option = + row.get_by_name("restore_source_backup_arn").ok().flatten(); + id.as_deref() == Some(&key.table_id) + && status.as_deref() == Some("CREATING") + && source.is_some() + }); + let row = row.ok_or_else(|| { + StorageError::TableNotFound(format!( + "{} is not a CREATING restore target with table id {}", + key.table_name, key.table_id + )) + })?; + let ks_text: String = + crate::cassandra_util::get_column(row, "key_schema", "restore target key info")?; + let ad_text: String = crate::cassandra_util::get_column( + row, + "attribute_definitions", + "restore target key info", + )?; + let key_schema: Vec = + serde_json::from_str(&ks_text) + .map_err(|e| StorageError::Internal(format!("key schema: {e}")))?; + let attribute_definitions = serde_json::from_str(&ad_text) + .map_err(|e| StorageError::Internal(format!("attribute definitions: {e}")))?; + Ok(TableKeyInfo { + table_name: key.table_name.clone(), + account_id: key.account_id.clone(), + table_id: key.table_id.clone(), + base_key_schema: key_schema.clone(), + key_schema, + attribute_definitions, + has_lsi: false, + global_secondary_indexes: Vec::new(), + local_secondary_indexes: Vec::new(), + stream_specification: None, + vector_indexes: Vec::new(), + }) + } + + /// The claim's liveness, read at LOCAL_QUORUM before each batch's writes: + /// the record RUNNING under `generation` with this table id. `Err` (a + /// lost claim) otherwise: the reconciler took the record over, and + /// whoever holds it owns the target, so nothing further is written. + async fn confirm_restore_claim( + &self, + key: &RestoreTargetKey, + generation: u64, + ) -> Result<(), StorageError> { + let query = format!( + "SELECT table_id, restore_status, restore_owner_generation FROM {}.tables \ + WHERE account_id = ? AND table_name = ?", + self.catalog_keyspace() + ); + let rows = crate::cassandra_util::query_rows_quorum::( + &self.session, + &query, + cdrs_tokio::query_values!(key.account_id.as_str(), key.table_name.as_str()), + "restore claim", + ) + .await?; + let live = rows.first().is_some_and(|row| { + let table_id: Option = row.get_by_name("table_id").ok().flatten(); + let status: Option = row.get_by_name("restore_status").ok().flatten(); + let owner: Option = row.get_by_name("restore_owner_generation").ok().flatten(); + table_id.as_deref() == Some(key.table_id.as_str()) + && status.as_deref() == Some(JobStatus::Running.as_str()) + && owner == Some(crate::backup_catalog::generation_column(generation)) + }); + if live { + Ok(()) + } else { + Err(claim_lost(&key.table_name)) + } + } + + /// Write one batch of items into the target, every index in each item's + /// own batch, `RESTORE_WRITE_CONCURRENCY` in flight, after confirming the + /// claim. Drains `batch`. + async fn flush_restore_batch( + &self, + key: &RestoreTargetKey, + generation: u64, + key_info: &TableKeyInfo, + batch: &mut Vec, + ) -> Result { + self.confirm_restore_claim(key, generation).await?; + let count = u64::try_from(batch.len()).unwrap_or(u64::MAX); + let mut writes = futures::stream::iter(batch.drain(..)) + .map(|item| self.put_restored_item(key_info, item)) + .buffer_unordered(RESTORE_WRITE_CONCURRENCY); + while let Some(result) = writes.next().await { + result?; + } + Ok(count) + } + + /// The progress lightweight transaction alone: `restore_file_index` and + /// `restore_items_applied`, conditioned on the table id, the restore + /// generation and the record being RUNNING. A test may make it answer + /// `Unknown`, with or without the transaction run. + async fn progress_write( + &self, + key: &RestoreTargetKey, + generation: u64, + file_index: u32, + items_applied: u64, + ) -> Result { + let fault = crate::test_hooks::progress_write_fault(&key.account_id); + if fault == Some(crate::test_hooks::ProgressWriteFault::Lost) { + return Ok(WriteOutcome::Unknown); + } + let query = format!( + "UPDATE {}.tables SET restore_file_index = ?, restore_items_applied = ? \ + WHERE account_id = ? AND table_name = ? \ + IF table_id = ? AND restore_owner_generation = ? AND restore_status = 'RUNNING'", + self.catalog_keyspace() + ); + let outcome = crate::backup_catalog::conditional_write( + &self.session, + &query, + cdrs_tokio::query::QueryValues::SimpleValues(vec![ + cdrs_tokio::types::value::Value::from( + i32::try_from(file_index).unwrap_or(i32::MAX), + ), + cdrs_tokio::types::value::Value::from( + i64::try_from(items_applied).unwrap_or(i64::MAX), + ), + cdrs_tokio::types::value::Value::from(key.account_id.as_str()), + cdrs_tokio::types::value::Value::from(key.table_name.as_str()), + cdrs_tokio::types::value::Value::from(key.table_id.as_str()), + cdrs_tokio::types::value::Value::from(crate::backup_catalog::generation_column( + generation, + )), + ]), + ) + .await?; + if fault == Some(crate::test_hooks::ProgressWriteFault::Landed) { + return Ok(WriteOutcome::Unknown); + } + Ok(outcome) + } + + /// What the record says after a progress write whose outcome is unknown, + /// read at LOCAL_SERIAL so a transaction that did commit is seen: RUNNING + /// under `generation` with this table id and carrying the values just + /// written is `Applied`; RUNNING under `generation` with other values + /// (the earlier progress: nothing but this job writes the record's + /// progress while it owns it) is `Pending`; anything else is `Lost`. + async fn progress_write_state( + &self, + key: &RestoreTargetKey, + generation: u64, + file_index: u32, + items_applied: u64, + ) -> Result { + let query = format!( + "SELECT table_id, restore_status, restore_owner_generation, restore_file_index, \ + restore_items_applied FROM {}.tables WHERE account_id = ? AND table_name = ?", + self.catalog_keyspace() + ); + let rows = crate::cassandra_util::query_rows_serial::( + &self.session, + &query, + cdrs_tokio::query_values!(key.account_id.as_str(), key.table_name.as_str()), + "restore progress", + ) + .await?; + let Some(row) = rows.first() else { + return Ok(ProgressWriteState::Lost); + }; + let table_id: Option = row.get_by_name("table_id").ok().flatten(); + let status: Option = row.get_by_name("restore_status").ok().flatten(); + let owner: Option = row.get_by_name("restore_owner_generation").ok().flatten(); + let owned = table_id.as_deref() == Some(key.table_id.as_str()) + && status.as_deref() == Some(JobStatus::Running.as_str()) + && owner == Some(crate::backup_catalog::generation_column(generation)); + if !owned { + return Ok(ProgressWriteState::Lost); + } + let recorded_file: Option = row.get_by_name("restore_file_index").ok().flatten(); + let recorded_items: Option = row.get_by_name("restore_items_applied").ok().flatten(); + let written = ( + Some(i32::try_from(file_index).unwrap_or(i32::MAX)), + Some(i64::try_from(items_applied).unwrap_or(i64::MAX)), + ); + if (recorded_file, recorded_items) == written { + Ok(ProgressWriteState::Applied) + } else { + Ok(ProgressWriteState::Pending) + } + } + + /// Record the file's progress (`restore_file_index`, the running total + /// `restore_items_applied`) with a lightweight transaction conditioned on + /// the restore generation and on the record being RUNNING. `[applied] = + /// false` is a lost claim. An unknown outcome (a write timeout or failure, + /// a lost connection: the Paxos round may still have committed) is + /// decided by re-reading the record at LOCAL_SERIAL: RUNNING under this + /// generation with the values just written is applied and the file goes + /// on; RUNNING under this generation with the earlier values is one + /// retry of the transaction, whose own unknown outcome is re-read the + /// same way, and a second unknown outcome that the record does not carry + /// fails the file (the error names it; the engine then fails the restore + /// and removes the target); anything else is the lost claim. + async fn write_restore_progress( + &self, + key: &RestoreTargetKey, + generation: u64, + file_index: u32, + items_applied: u64, + ) -> Result<(), StorageError> { + restore_pause_point(RESTORE_CASSANDRA_BEFORE_PROGRESS).await; + let mut retried = false; + loop { + match self + .progress_write(key, generation, file_index, items_applied) + .await? + { + WriteOutcome::Matched => return Ok(()), + WriteOutcome::NotMatched => return Err(claim_lost(&key.table_name)), + WriteOutcome::Unknown => {} + } + match self + .progress_write_state(key, generation, file_index, items_applied) + .await? + { + ProgressWriteState::Applied => return Ok(()), + ProgressWriteState::Lost => return Err(claim_lost(&key.table_name)), + ProgressWriteState::Pending if !retried => { + tracing::warn!( + "restore of table {}: the progress write for data file {file_index} \ + had no known outcome and the record does not carry it; written once \ + more", + key.table_name + ); + retried = true; + } + ProgressWriteState::Pending => { + return Err(StorageError::Internal(format!( + "restore of table {}: the progress write for data file {file_index} \ + had no known outcome twice and the record still carries the earlier \ + progress; the restore fails and its target is removed", + key.table_name + ))); + } + } + } + } +} + +/// What the record says about a progress write whose outcome was unknown. +#[derive(Debug, PartialEq, Eq)] +enum ProgressWriteState { + /// RUNNING under the claim's generation, the written values on it. + Applied, + /// RUNNING under the claim's generation, the earlier values on it. + Pending, + /// Not RUNNING under the claim's generation, or gone. + Lost, +} + +impl RestoreEngine for CassandraEngine { + fn bulk_load<'a>( + &'a self, + target: &'a RestoreTargetKey, + claim: &'a dyn JobClaim, + file: BulkLoadFile, + items: BoxStream<'a, Result>, + ) -> BoxFuture<'a, Result> { + Box::pin(async move { + let generation = claim.generation(); + let key_info = self.restore_target_key_info(target).await?; + let batch_size = crate::test_hooks::restore_batch_size(&target.account_id).max(1); + let mut items = items; + let mut batch: Vec = Vec::with_capacity(batch_size.min(1024)); + let mut batch_bytes = 0usize; + let mut applied: u64 = 0; + let mut wrote_progress = false; + while let Some(item) = items.try_next().await? { + // The item's `item_data` text, the unit the byte bound is + // measured in (the put path serializes it again for the row). + let text_len = serde_json::to_string(&item) + .map_err(|e| StorageError::Internal(format!("item: {e}")))? + .len(); + batch_bytes = batch_bytes.saturating_add(text_len); + batch.push(item); + if batch.len() >= batch_size || batch_bytes >= RESTORE_BATCH_BYTES { + applied += self + .flush_restore_batch(target, generation, &key_info, &mut batch) + .await?; + batch_bytes = 0; + self.write_restore_progress( + target, + generation, + file.file_index, + file.start_at.saturating_add(applied), + ) + .await?; + wrote_progress = true; + } + } + if !batch.is_empty() { + applied += self + .flush_restore_batch(target, generation, &key_info, &mut batch) + .await?; + self.write_restore_progress( + target, + generation, + file.file_index, + file.start_at.saturating_add(applied), + ) + .await?; + wrote_progress = true; + } + // An empty file still records that it is complete, so a resume + // does not re-read it. + if !wrote_progress { + self.write_restore_progress(target, generation, file.file_index, file.start_at) + .await?; + } + Ok(applied) + }) + } + + fn finish_restore<'a>( + &'a self, + target: &'a RestoreTargetKey, + claim: &'a dyn JobClaim, + item_count: u64, + size_bytes: u64, + ) -> BoxFuture<'a, Result> { + Box::pin(async move { + let activated = self + .activate_restore_target_row( + &target.account_id, + &target.table_name, + &target.table_id, + claim.generation(), + i64::try_from(item_count).unwrap_or(i64::MAX), + i64::try_from(size_bytes).unwrap_or(i64::MAX), + ) + .await?; + if activated { + Ok(Finished::Active) + } else { + Ok(Finished::NotMatched(format!( + "the restore record of {} is not RUNNING under generation {}, or the table \ + is not a CREATING restore target", + target.table_name, + claim.generation() + ))) + } + }) + } + + fn count_restore_target<'a>( + &'a self, + key: &'a RestoreTargetKey, + ) -> BoxFuture<'a, Result> { + Box::pin(async move { + // A full COUNT(*) over the target's data table: a single scan of + // a CREATING table, run once per restore (the cost is stated in + // the limits doc). The id names the data table; the account + // keyspace scopes it to the target's account, and the catalog is + // read first so a key that is not a CREATING restore target with + // this id is `TableNotFound`, as on the other backends. + self.restore_target_key_info(key).await?; + let account_keyspace = self.account_keyspace(&key.account_id); + let data_table = data_table_name(&key.table_id); + let query = format!("SELECT COUNT(*) AS c FROM {account_keyspace}.{data_table}"); + let rows = crate::cassandra_util::query_rows_quorum::( + &self.session, + &query, + cdrs_tokio::query_values!(), + "count restore target", + ) + .await?; + let count: i64 = rows + .first() + .map(|row| crate::cassandra_util::get_column(row, "c", "count restore target")) + .transpose()? + .unwrap_or(0); + Ok(u64::try_from(count).unwrap_or(0)) + }) + } +} diff --git a/crates/storage-cassandra/src/test_hooks.rs b/crates/storage-cassandra/src/test_hooks.rs index 80821442c..667ee79e5 100644 --- a/crates/storage-cassandra/src/test_hooks.rs +++ b/crates/storage-cassandra/src/test_hooks.rs @@ -19,20 +19,42 @@ //! * a pause between the in-use read and the DELETING claim of a backup //! delete (the legacy `delete` and `start_backup_delete`), so a test can //! register a restore in that gap and see the claim's own -//! `active_restores = null` condition refuse the delete. +//! `active_restores = null` condition refuse the delete; +//! * a pause in a store-backed registration between the target's creation +//! and its `active_restores` lightweight transaction, so a test can move +//! the backup to DELETING in that gap and see the transaction's own +//! `backup_status = 'AVAILABLE'` condition refuse the registration; +//! * a pause in `claim_restore_dead` between its read and its takeover +//! lightweight transaction, and the batch size of a store-backed +//! `bulk_load`; +//! * a fault on the progress lightweight transaction of a store-backed +//! `bulk_load`, so a test can see the re-read that decides an unknown +//! outcome. use std::time::Duration; use extenddb_storage::error::StorageError; +/// How a test makes a progress lightweight transaction answer +/// `WriteOutcome::Unknown`: with the transaction run first (`Landed`: the +/// answer was lost on its way back) or not run at all (`Lost`: the proposal +/// never reached a replica). Each fault is taken by one write, in order. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum ProgressWriteFault { + Landed, + Lost, +} + #[cfg(feature = "test-hooks")] mod armed { - use std::collections::HashMap; + use std::collections::{HashMap, VecDeque}; use std::sync::{Arc, LazyLock, Mutex}; use std::time::Duration; use tokio::sync::Notify; + use super::ProgressWriteFault; + /// A pause a test arms at one point of one operation: the operation /// signals `reached` when it gets there and waits for `released`. pub struct Pause { @@ -69,6 +91,10 @@ mod armed { pub(super) heartbeat_interval: HashMap, pub(super) heartbeat_errors: HashMap, pub(super) before_delete_claim: HashMap>, + pub(super) before_restore_takeover: HashMap>, + pub(super) before_restore_registration: HashMap>, + pub(super) restore_batch_size: HashMap, + pub(super) progress_write_faults: HashMap>, } pub(super) static HOOKS: LazyLock> = LazyLock::new(Mutex::default); @@ -125,6 +151,63 @@ pub fn pause_before_delete_claim(backup_arn: &str) -> std::sync::Arc { pause } +/// Arm a pause in the next `claim_restore_dead` of the restore target +/// `table_id`, between its serial read (and staleness check) and its +/// takeover lightweight transaction. A test releases it after writing a +/// fresh heartbeat, to prove the takeover's `restore_heartbeat_at = ` +/// condition refuses a beat that lands in between. Taken once, then disarmed. +#[cfg(feature = "test-hooks")] +pub fn pause_before_restore_takeover(table_id: &str) -> std::sync::Arc { + let pause = std::sync::Arc::new(Pause::new()); + armed::with_hooks(|hooks| { + hooks + .before_restore_takeover + .insert(table_id.to_owned(), std::sync::Arc::clone(&pause)); + }); + pause +} + +/// Arm a pause in the next store-backed registration of a restore of +/// `backup_arn`, between the target's creation (its row and DDL) and the +/// `active_restores` lightweight transaction on the backup. A test moves the +/// backup to DELETING in the gap to prove the transaction's own +/// `backup_status = 'AVAILABLE'` condition, not the unfenced pre-read, +/// refuses the registration. Taken once, then disarmed. +#[cfg(feature = "test-hooks")] +pub fn pause_before_restore_registration(backup_arn: &str) -> std::sync::Arc { + let pause = std::sync::Arc::new(Pause::new()); + armed::with_hooks(|hooks| { + hooks + .before_restore_registration + .insert(backup_arn.to_owned(), std::sync::Arc::clone(&pause)); + }); + pause +} + +/// Make the store-backed `bulk_load` of jobs of `account_id` flush and record +/// progress every `size` items instead of the production batch size, so a +/// test can cross a batch boundary without writing ten thousand items. +#[cfg(feature = "test-hooks")] +pub fn set_restore_batch_size(account_id: &str, size: usize) { + armed::with_hooks(|hooks| { + hooks.restore_batch_size.insert(account_id.to_owned(), size); + }); +} + +/// Make the next progress lightweight transactions of the store-backed +/// `bulk_load` of jobs of `account_id` answer `WriteOutcome::Unknown`, one +/// fault per write in the order given: [`ProgressWriteFault::Landed`] runs +/// the transaction and drops its answer, [`ProgressWriteFault::Lost`] drops +/// the transaction. Writes after the last fault run as usual. +#[cfg(feature = "test-hooks")] +pub fn fail_progress_writes(account_id: &str, faults: &[ProgressWriteFault]) { + armed::with_hooks(|hooks| { + hooks + .progress_write_faults + .insert(account_id.to_owned(), faults.iter().copied().collect()); + }); +} + /// The period the jobs of `account_id` heartbeat at. pub(crate) fn heartbeat_interval(account_id: &str) -> Duration { #[cfg(feature = "test-hooks")] @@ -168,3 +251,58 @@ pub(crate) async fn before_delete_claim(backup_arn: &str) { } let _ = backup_arn; } + +/// Hold here if a test armed a pause for the takeover of restore target +/// `table_id`. +pub(crate) async fn before_restore_takeover(table_id: &str) { + #[cfg(feature = "test-hooks")] + if let Some(pause) = armed::with_hooks(|hooks| hooks.before_restore_takeover.remove(table_id)) { + pause.hold().await; + } + let _ = table_id; +} + +/// Hold here if a test armed a pause for the store-backed registration of a +/// restore of `backup_arn`. +pub(crate) async fn before_restore_registration(backup_arn: &str) { + #[cfg(feature = "test-hooks")] + if let Some(pause) = + armed::with_hooks(|hooks| hooks.before_restore_registration.remove(backup_arn)) + { + pause.hold().await; + } + let _ = backup_arn; +} + +/// The number of items the store-backed `bulk_load` of jobs of `account_id` +/// applies before each progress write (the byte bound closes a batch +/// earlier). The production default is `RESTORE_BATCH_ITEMS`; a test may set +/// a small size with [`set_restore_batch_size`]. +pub(crate) fn restore_batch_size(account_id: &str) -> usize { + #[cfg(feature = "test-hooks")] + if let Some(size) = armed::with_hooks(|hooks| hooks.restore_batch_size.get(account_id).copied()) + { + return size; + } + let _ = account_id; + crate::restore_engine::RESTORE_BATCH_ITEMS +} + +/// The fault the next progress lightweight transaction of a job of +/// `account_id` takes, if a test armed one. +pub(crate) fn progress_write_fault(account_id: &str) -> Option { + #[cfg(feature = "test-hooks")] + { + armed::with_hooks(|hooks| { + hooks + .progress_write_faults + .get_mut(account_id) + .and_then(std::collections::VecDeque::pop_front) + }) + } + #[cfg(not(feature = "test-hooks"))] + { + let _ = account_id; + None + } +} diff --git a/crates/storage-cassandra/src/workers.rs b/crates/storage-cassandra/src/workers.rs index 2bd5996e7..a51ed5b44 100644 --- a/crates/storage-cassandra/src/workers.rs +++ b/crates/storage-cassandra/src/workers.rs @@ -372,7 +372,9 @@ async fn gsi_process_batch( Ok(total) } -/// Apply a single GSI update (delete old row, insert new row) for one pending entry. +/// Apply a single GSI update (the old row deleted, the new row inserted; the +/// one row rewritten when the index keys did not change, #742) for one +/// pending entry. async fn gsi_apply_index( engine: &CassandraEngine, account_keyspace: &str, @@ -382,9 +384,7 @@ async fn gsi_apply_index( new_item: Option<&extenddb_core::types::Item>, ) -> Result<(), extenddb_storage::error::StorageError> { use crate::data::ddl::all_sort_key_info; - use crate::data::index::{ - delete_index_row_multi, insert_index_row_multi, item_has_index_keys, project_item_for_index, - }; + use crate::data::index::write_index_row_change; use cdrs_tokio::consistency::Consistency; use cdrs_tokio::query::BatchQueryBuilder; @@ -394,43 +394,18 @@ async fn gsi_apply_index( let base_sks = all_sort_key_info(&context.base_key_schema, &context.attribute_definitions); let mut batch = BatchQueryBuilder::new().with_consistency(Consistency::LocalQuorum); - - if let Some(old) = old_item - && item_has_index_keys(old, &idx.key_schema) - { - delete_index_row_multi( - &mut batch, - account_keyspace, - &idx_table, - old, - &idx.key_schema, - &context.base_key_schema, - &idx_sks, - &base_sks, - )?; - } - - if let Some(new) = new_item - && item_has_index_keys(new, &idx.key_schema) - { - let projected = project_item_for_index( - new, - &idx.key_schema, - &context.base_key_schema, - &idx.projection, - ); - insert_index_row_multi( - &mut batch, - account_keyspace, - &idx_table, - new, - &projected, - &idx.key_schema, - &context.base_key_schema, - &idx_sks, - &base_sks, - )?; - } + write_index_row_change( + &mut batch, + account_keyspace, + &idx_table, + &idx.key_schema, + &context.base_key_schema, + &idx_sks, + &base_sks, + &idx.projection, + old_item, + new_item, + )?; // Only execute if there's something to do. let built = batch diff --git a/crates/storage-cassandra/tests/common/mod.rs b/crates/storage-cassandra/tests/common/mod.rs index c1fd1e971..95c80b94f 100644 --- a/crates/storage-cassandra/tests/common/mod.rs +++ b/crates/storage-cassandra/tests/common/mod.rs @@ -558,6 +558,85 @@ impl TestTable { owns_keyspace: true, } } + + /// A composite-key table (`id` S, `sk` N) with a GSI `gix` on `g` (S) and + /// an LSI `lix` on (`id`, `l` S), both projecting every attribute, so a + /// projected non-key attribute of an index row is observable through a + /// Query on the index. + pub async fn with_gsi_and_lsi(engine: &CassandraEngine, table_name: &str) -> Self { + use extenddb_core::types::{GsiInput, LsiInput, Projection, ProjectionType}; + + let account_id = unique_test_account(); + ensure_test_account(engine, &account_id).await.unwrap(); + + let ks = |name: &str, key_type: KeyType| KeySchemaElement { + attribute_name: name.to_string(), + key_type, + }; + let attr = |name: &str, attribute_type: ScalarAttributeType| AttributeDefinition { + attribute_name: name.to_string(), + attribute_type, + }; + let all = || Projection { + projection_type: ProjectionType::All, + non_key_attributes: None, + }; + let key_schema = vec![ks("id", KeyType::Hash), ks("sk", KeyType::Range)]; + let attribute_definitions = vec![ + attr("id", ScalarAttributeType::S), + attr("sk", ScalarAttributeType::N), + attr("g", ScalarAttributeType::S), + attr("l", ScalarAttributeType::S), + ]; + let input = extenddb_core::types::CreateTableInput { + table_name: table_name.to_string(), + key_schema: key_schema.clone(), + attribute_definitions: attribute_definitions.clone(), + local_secondary_indexes: Some(vec![LsiInput { + index_name: "lix".to_string(), + key_schema: vec![ks("id", KeyType::Hash), ks("l", KeyType::Range)], + projection: all(), + }]), + global_secondary_indexes: Some(vec![GsiInput { + index_name: "gix".to_string(), + key_schema: vec![ks("g", KeyType::Hash)], + projection: all(), + provisioned_throughput: None, + }]), + vector_indexes: None, + billing_mode: None, + table_throughput_mode: None, + provisioned_throughput: None, + on_demand_throughput: None, + stream_specification: None, + sse_specification: None, + tags: None, + table_class: None, + deletion_protection_enabled: None, + }; + + let table_desc = engine.create_table(&account_id, input).await.unwrap(); + + let key_info = TableKeyInfo { + table_name: table_name.to_string(), + account_id: account_id.to_string(), + table_id: table_desc.table_id, + key_schema: key_schema.clone(), + base_key_schema: key_schema, + attribute_definitions, + has_lsi: true, + global_secondary_indexes: Vec::new(), + local_secondary_indexes: Vec::new(), + vector_indexes: Vec::new(), + stream_specification: None, + }; + + Self { + session: engine.session_arc(), + key_info, + owns_keyspace: true, + } + } } impl Drop for TestTable { diff --git a/crates/storage-cassandra/tests/direct/backup_engine.rs b/crates/storage-cassandra/tests/direct/backup_engine.rs index a0a256d63..961d42c37 100644 --- a/crates/storage-cassandra/tests/direct/backup_engine.rs +++ b/crates/storage-cassandra/tests/direct/backup_engine.rs @@ -20,13 +20,14 @@ use extenddb_core::types::{ TimeToLiveStatus, }; use extenddb_storage::backup_catalog::{ - BackupRow, BackupStatus, DeleteStart, Heartbeat, JobStatus, RestoreTargetKey, StaleClock, - StatusFields, StoreIdentity, StoreKind, TimeBounds, UNOWNED_GENERATION, WriteOutcome, + BackupRow, BackupStatus, DeleteStart, Heartbeat, JobStatus, RestoreKind, + RestoreRegistrationRow, RestoreTargetKey, StaleClock, StatusFields, StoreIdentity, StoreKind, + TimeBounds, UNOWNED_GENERATION, WriteOutcome, }; use extenddb_storage::error::StorageError; use extenddb_storage::{ - BackupCatalog, DataEngine, JobPermit, LegacyBackupReader, RestoreRequest, - RestoreTableOverrides, TableEngine, + BackupCatalog, BulkLoadFile, DataEngine, Finished, JobPermit, LegacyBackupReader, + RestoreEngine, RestoreRequest, RestoreTableOverrides, TableEngine, }; use extenddb_storage_cassandra::CassandraEngine; use futures::{StreamExt, TryStreamExt}; @@ -5030,3 +5031,1857 @@ async fn test_the_registrations_sweep_removes_stale_entries_and_keeps_a_live_one "the kept entry is the in-use rule's" ); } + +// ── Store-backed (manifest) restore ────────────────────────────────────────── + +/// An AVAILABLE catalog-index backup row (`store_kind` set) for `table`, the +/// row a store-backed restore registers against. +async fn available_index_backup( + engine: &CassandraEngine, + account_id: &str, + table: &str, + name: &str, + store: &StoreIdentity, +) -> BackupRow { + let row = index_row(account_id, table, name, store); + let claim = engine.claim_backup(&row).await.expect("claim backup"); + assert_eq!( + claim + .set_status( + JobStatus::Creating, + JobStatus::Available, + &StatusFields { + completed: true, + ..StatusFields::default() + } + ) + .await + .expect("make the backup AVAILABLE"), + WriteOutcome::Matched + ); + row +} + +/// A hash-only `CreateTableInput` for a restore target. +fn manifest_target_input(name: &str) -> CreateTableInput { + CreateTableInput { + table_name: name.to_owned(), + key_schema: vec![ks("id", KeyType::Hash)], + attribute_definitions: vec![attr("id", ScalarAttributeType::S)], + billing_mode: Some(BillingMode::PayPerRequest), + ..Default::default() + } +} + +/// The registration a store-backed restore of `row` writes, with `spool_host`. +fn manifest_registration(row: &BackupRow, spool_host: Option<&str>) -> RestoreRegistrationRow { + RestoreRegistrationRow { + backup: row.key(), + source_table_arn: format!( + "arn:aws:dynamodb:us-east-1:{}:table/{}", + row.account_id, row.table_name + ), + restore_date_time: row.created_at, + store: row.store.clone(), + spool_host: spool_host.map(str::to_owned), + owner_generation: UNOWNED_GENERATION, + } +} + +/// A hash-keyed item with key `k{i:05}`. +fn id_item(i: usize) -> Item { + let mut it = BTreeMap::new(); + it.insert("id".to_owned(), AttributeValue::S(format!("k{i:05}"))); + it.insert("v".to_owned(), AttributeValue::S(format!("val{i}"))); + it +} + +/// The items `lo..hi` as a `bulk_load` stream. +fn item_stream( + lo: usize, + hi: usize, +) -> futures::stream::BoxStream<'static, Result> { + let items: Vec = (lo..hi).map(id_item).collect(); + futures::stream::iter(items.into_iter().map(Ok)).boxed() +} + +/// `claim_restore` registers a CREATING target with a RUNNING `manifest` +/// record carrying the store, the spool host and the claim's generation, the +/// `active_restores` entry on the backup, and a `RestoreSummary` on the +/// description; the claim heartbeats and reads the record. +#[tokio::test] +async fn test_claim_restore_registers_a_manifest_target_with_its_record() { + if crate::helpers::skip_without_cassandra() { + return; + } + let engine = setup_engine().await; + let account = TestAccount::new(&engine, "extenddb_ttl_test").await; + let account_id = account.account_id.clone(); + let store = unique_store(); + let row = available_index_backup(&engine, &account_id, "Music", "nightly", &store).await; + let registration = manifest_registration(&row, Some("host-a:/var/spool")); + + let (desc, claim) = engine + .claim_restore( + &account_id, + manifest_target_input("RManifest"), + ®istration, + ) + .await + .expect("the registration creates the target and the record"); + assert_eq!(desc.table_status, TableStatus::Creating); + let summary = desc.restore_summary.as_ref().expect("a RestoreSummary"); + assert_eq!( + summary.source_backup_arn.as_deref(), + Some(row.backup_arn.as_str()) + ); + assert!(summary.restore_in_progress); + assert_ne!(claim.generation(), UNOWNED_GENERATION); + + let key = RestoreTargetKey { + account_id: account_id.clone(), + table_name: desc.table_name.clone(), + table_id: desc.table_id.clone(), + }; + let record = engine + .read_restore_job(&key) + .await + .expect("read") + .expect("the record exists"); + assert_eq!(record.kind, RestoreKind::Manifest); + assert_eq!(record.status, JobStatus::Running); + assert_eq!(record.owner_generation, claim.generation()); + assert_eq!(record.store.as_ref(), Some(&store)); + assert_eq!(record.spool_host.as_deref(), Some("host-a:/var/spool")); + assert_eq!(record.source_backup_arn, row.backup_arn); + + // The backup is in use by the target, and the heartbeat confirms and the + // read returns RUNNING. + assert_eq!( + engine.backup_in_use(&row.key()).await.unwrap().as_deref(), + Some("RManifest") + ); + assert_eq!( + claim.heartbeat().await.expect("heartbeat"), + Heartbeat::Confirmed + ); + assert_eq!(claim.read().await.expect("read").status, JobStatus::Running); +} + +/// The two orderings against `start_backup_delete` on the backup's +/// `backup_index` row: a registration that lands first leaves the backup in +/// use; a DeleteBackup that lands first leaves the registration with no +/// backup to read. +#[tokio::test] +async fn test_claim_restore_and_start_backup_delete_order_on_the_backup_index_row() { + if crate::helpers::skip_without_cassandra() { + return; + } + let engine = setup_engine().await; + let account = TestAccount::new(&engine, "extenddb_ttl_test").await; + let account_id = account.account_id.clone(); + + // Registration wins: DeleteBackup finds the backup in use by the target. + let store = unique_store(); + let won = available_index_backup(&engine, &account_id, "Music", "a", &store).await; + let (desc, _claim) = engine + .claim_restore( + &account_id, + manifest_target_input("WonTarget"), + &manifest_registration(&won, None), + ) + .await + .expect("register"); + assert_eq!(desc.table_status, TableStatus::Creating); + assert_eq!( + engine.start_backup_delete(&won.key()).await.unwrap(), + DeleteStart::InUse("WonTarget".to_owned()) + ); + + // DeleteBackup wins: the registration finds the backup gone. + let lost = available_index_backup(&engine, &account_id, "Music", "b", &store).await; + assert_eq!( + engine.start_backup_delete(&lost.key()).await.unwrap(), + DeleteStart::Deleting + ); + let refused = engine + .claim_restore( + &account_id, + manifest_target_input("LostTarget"), + &manifest_registration(&lost, None), + ) + .await + .err() + .expect("a registration against a DELETING backup is Backup not found"); + assert!( + matches!(refused, StorageError::Validation(ref m) if m.contains("Backup not found")), + "{refused}" + ); + // The lost registration left no target behind. + assert!( + engine + .describe_table( + &account_id, + DescribeTableInput { + table_name: "LostTarget".to_owned() + } + ) + .await + .is_err() + ); +} + +/// A second restore of a backup whose target is CREATING is `BackupInUse`, +/// and a restore into a taken name is `TableAlreadyExists`. +#[tokio::test] +async fn test_a_second_restore_is_in_use_and_a_taken_name_is_refused() { + if crate::helpers::skip_without_cassandra() { + return; + } + let engine = setup_engine().await; + let account = TestAccount::new(&engine, "extenddb_ttl_test").await; + let account_id = account.account_id.clone(); + let store = unique_store(); + let row = available_index_backup(&engine, &account_id, "Music", "nightly", &store).await; + + let (_desc, _claim) = engine + .claim_restore( + &account_id, + manifest_target_input("FirstTarget"), + &manifest_registration(&row, None), + ) + .await + .expect("the first restore registers"); + + // A second restore of the same backup, into a different name, is in use. + let in_use = engine + .claim_restore( + &account_id, + manifest_target_input("SecondTarget"), + &manifest_registration(&row, None), + ) + .await + .err() + .expect("the backup is in use by the first restore"); + assert!( + matches!(in_use, StorageError::BackupInUse(ref m) if m.contains("FirstTarget")), + "{in_use}" + ); + + // A restore into an existing table's name is TableAlreadyExists. + create_active_table( + &engine, + &account_id, + CreateTableInput { + table_name: "Taken".to_owned(), + key_schema: vec![ks("id", KeyType::Hash)], + attribute_definitions: vec![attr("id", ScalarAttributeType::S)], + billing_mode: Some(BillingMode::PayPerRequest), + ..Default::default() + }, + ) + .await; + let other = available_index_backup(&engine, &account_id, "Music", "other", &store).await; + let taken = engine + .claim_restore( + &account_id, + manifest_target_input("Taken"), + &manifest_registration(&other, None), + ) + .await + .err() + .expect("the name is taken"); + assert!( + matches!(taken, StorageError::TableAlreadyExists(ref name) if name == "Taken"), + "{taken}" + ); +} + +/// `claim_restore_dead`: a live owner, and a legacy record, are not taken +/// over; a stale manifest record is, under a fresh generation, with the +/// claimant's `restore_spool_host` written and `restore_heartbeat_at` +/// refreshed to the coordinator's clock in the same lightweight transaction, +/// so a second taker right after the first (before the successor's first +/// beat) finds the record live and is refused while the first taker's beat +/// confirms; the old claim's writes are then fenced. +#[tokio::test] +async fn test_claim_restore_dead_takes_over_a_stale_manifest_record() { + if crate::helpers::skip_without_cassandra() { + return; + } + let engine = setup_engine().await; + let account = TestAccount::new(&engine, "extenddb_ttl_test").await; + let account_id = account.account_id.clone(); + let store = unique_store(); + let row = available_index_backup(&engine, &account_id, "Music", "nightly", &store).await; + let (desc, owner) = engine + .claim_restore( + &account_id, + manifest_target_input("Takeover"), + &manifest_registration(&row, Some("owner:/s")), + ) + .await + .expect("register"); + let key = RestoreTargetKey { + account_id: account_id.clone(), + table_name: desc.table_name.clone(), + table_id: desc.table_id.clone(), + }; + let window = Duration::from_secs(1); + let before = engine + .read_restore_job(&key) + .await + .unwrap() + .unwrap() + .heartbeat_at; + assert!( + engine + .claim_restore_dead(&key, Duration::from_secs(300), Some("other:/s")) + .await + .expect("claim") + .is_none(), + "a live owner within a long window is not taken over" + ); + + tokio::time::sleep(Duration::from_millis(1_300)).await; + let successor = engine + .claim_restore_dead(&key, window, Some("successor:/spool")) + .await + .expect("claim") + .expect("the window passed: taken over"); + assert_ne!(successor.generation(), owner.generation()); + let after = engine.read_restore_job(&key).await.unwrap().unwrap(); + assert_eq!(after.owner_generation, successor.generation()); + assert_eq!(after.status, JobStatus::Running); + assert_eq!( + after.spool_host.as_deref(), + Some("successor:/spool"), + "the takeover wrote the claimant's spool host" + ); + // A second taker right after the first, before the successor's first + // beat: its serial read finds the fresh heartbeat and it is refused; the + // first taker's beat confirms. + let second = engine + .claim_restore_dead(&key, window, Some("second:/s")) + .await + .expect("a second takeover attempt"); + assert!( + second.is_none(), + "a second taker right after the first is refused: {:?}", + second.as_ref().map(|c| c.generation()) + ); + assert_eq!( + successor.heartbeat().await.expect("heartbeat"), + Heartbeat::Confirmed, + "the first taker keeps the record" + ); + assert!( + after.heartbeat_at > before + 1.0, + "the takeover refreshed the heartbeat: {} was {before}", + after.heartbeat_at + ); + let now = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .expect("epoch") + .as_secs_f64(); + assert!( + (now - after.heartbeat_at).abs() < 5.0, + "the refreshed heartbeat is the coordinator's now: {} vs {now}", + after.heartbeat_at + ); + assert_eq!(owner.heartbeat().await.expect("heartbeat"), Heartbeat::Lost); + assert_eq!( + owner + .set_status( + JobStatus::Running, + JobStatus::Failed, + &StatusFields::default() + ) + .await + .expect("write"), + WriteOutcome::NotMatched, + "the old generation's write is fenced" + ); + + // The successor heartbeats again: a third taker finds it live. + assert_eq!( + successor.heartbeat().await.expect("heartbeat"), + Heartbeat::Confirmed + ); + assert!( + engine + .claim_restore_dead(&key, window, Some("third:/s")) + .await + .expect("claim") + .is_none(), + "the successor's heartbeat is fresh" + ); + + // A legacy record is not a manifest restore's to take, whatever its age. + apply( + &engine, + &format!( + "UPDATE {}.tables SET restore_kind = 'legacy' \ + WHERE account_id = ? AND table_name = ? IF table_id = ?", + engine.catalog_keyspace() + ), + cdrs_tokio::query_values!( + account_id.as_str(), + key.table_name.as_str(), + key.table_id.as_str() + ), + ) + .await; + tokio::time::sleep(Duration::from_millis(1_300)).await; + assert!( + engine + .claim_restore_dead(&key, window, Some("x:/s")) + .await + .expect("claim") + .is_none(), + "a legacy record is not taken over by the manifest takeover" + ); +} + +/// `claim_restore_dead` refuses the takeover when a heartbeat lands between +/// its serial read and its lightweight transaction: the LWT conditions on the +/// heartbeat it read, which the beat has moved. +#[cfg(feature = "test-hooks")] +#[tokio::test] +async fn test_claim_restore_dead_refuses_a_beat_between_the_read_and_the_write() { + if crate::helpers::skip_without_cassandra() { + return; + } + let engine = setup_engine().await; + let account = TestAccount::new(&engine, "extenddb_ttl_test").await; + let account_id = account.account_id.clone(); + let store = unique_store(); + let row = available_index_backup(&engine, &account_id, "Music", "nightly", &store).await; + let (desc, owner) = engine + .claim_restore( + &account_id, + manifest_target_input("BeatRace"), + &manifest_registration(&row, Some("owner:/s")), + ) + .await + .expect("register"); + let key = RestoreTargetKey { + account_id: account_id.clone(), + table_name: desc.table_name.clone(), + table_id: desc.table_id.clone(), + }; + tokio::time::sleep(Duration::from_millis(1_300)).await; + + let pause = + extenddb_storage_cassandra::test_hooks::pause_before_restore_takeover(&key.table_id); + let takeover = { + let engine = engine.clone(); + let key = key.clone(); + tokio::spawn(async move { + engine + .claim_restore_dead(&key, Duration::from_secs(1), Some("successor:/s")) + .await + }) + }; + pause.reached().await; + // A beat lands after the takeover read its stale heartbeat. + apply( + &engine, + &format!( + "UPDATE {}.tables SET restore_heartbeat_at = toTimestamp(now()) \ + WHERE account_id = ? AND table_name = ? IF table_id = ?", + engine.catalog_keyspace() + ), + cdrs_tokio::query_values!( + account_id.as_str(), + key.table_name.as_str(), + key.table_id.as_str() + ), + ) + .await; + pause.release(); + assert!( + takeover + .await + .expect("takeover task") + .expect("claim") + .is_none(), + "the beat between the read and the write refuses the takeover" + ); + let record = engine.read_restore_job(&key).await.unwrap().unwrap(); + assert_eq!( + record.owner_generation, + owner.generation(), + "the record is still the original owner's" + ); + assert_eq!( + record.spool_host.as_deref(), + Some("owner:/s"), + "the refused takeover did not rewrite the spool host" + ); +} + +/// A record whose backup does not list its target (the owner died between +/// the target's creation and the `active_restores` registration): the +/// takeover re-applies the entry, so the backup is in use for the resumed +/// load and `start_backup_delete` is refused. +#[tokio::test] +async fn test_claim_restore_dead_reapplies_a_missing_active_restores_entry() { + if crate::helpers::skip_without_cassandra() { + return; + } + let engine = setup_engine().await; + let account = TestAccount::new(&engine, "extenddb_ttl_test").await; + let account_id = account.account_id.clone(); + let store = unique_store(); + let row = available_index_backup(&engine, &account_id, "Music", "nightly", &store).await; + let (desc, owner) = engine + .claim_restore( + &account_id, + manifest_target_input("Unlisted"), + &manifest_registration(&row, Some("owner:/s")), + ) + .await + .expect("register"); + let key = RestoreTargetKey { + account_id: account_id.clone(), + table_name: desc.table_name.clone(), + table_id: desc.table_id.clone(), + }; + // The crash's shape: the target and its RUNNING record exist, the + // backup's entry does not. + apply( + &engine, + &format!( + "DELETE active_restores[?] FROM {}.backups_by_arn \ + WHERE account_id = ? AND backup_arn = ? IF EXISTS", + engine.catalog_keyspace() + ), + cdrs_tokio::query_values!( + key.table_id.as_str(), + account_id.as_str(), + row.backup_arn.as_str() + ), + ) + .await; + assert!( + registrations(&engine, &account_id, &row.backup_arn) + .await + .is_empty(), + "the entry is gone" + ); + drop(owner); + tokio::time::sleep(Duration::from_millis(1_300)).await; + let successor = engine + .claim_restore_dead(&key, Duration::from_secs(1), Some("successor:/s")) + .await + .expect("claim") + .expect("the stale record is taken over"); + let entries = registrations(&engine, &account_id, &row.backup_arn).await; + assert_eq!( + entries.get(&key.table_id).map(String::as_str), + Some("Unlisted"), + "the takeover re-applied the backup's entry for the target: {entries:?}" + ); + assert_eq!( + engine.start_backup_delete(&row.key()).await.unwrap(), + DeleteStart::InUse("Unlisted".to_owned()), + "the backup is in use for the resumed load" + ); + assert_eq!( + engine.backup_in_use(&row.key()).await.unwrap().as_deref(), + Some("Unlisted") + ); + drop(successor); +} + +/// A DeleteBackup that lands between a registration's unfenced pre-read and +/// its `active_restores` lightweight transaction: the transaction's own +/// `backup_status = 'AVAILABLE'` condition refuses the registration, which +/// answers `Backup not found` and leaves no target and no entry. +#[cfg(feature = "test-hooks")] +#[tokio::test] +async fn test_a_delete_between_the_registrations_pre_read_and_its_fence_leaves_no_target() { + if crate::helpers::skip_without_cassandra() { + return; + } + let engine = setup_engine().await; + let account = TestAccount::new(&engine, "extenddb_ttl_test").await; + let account_id = account.account_id.clone(); + let store = unique_store(); + let row = available_index_backup(&engine, &account_id, "Music", "raced", &store).await; + let pause = + extenddb_storage_cassandra::test_hooks::pause_before_restore_registration(&row.backup_arn); + let registration = { + let engine = engine.clone(); + let account_id = account_id.clone(); + let registration = manifest_registration(&row, None); + tokio::spawn(async move { + engine + .claim_restore( + &account_id, + manifest_target_input("RacedTarget"), + ®istration, + ) + .await + }) + }; + // The registration has read the backup AVAILABLE and created the target; + // the delete lands before its fence. + pause.reached().await; + assert_eq!( + engine.start_backup_delete(&row.key()).await.unwrap(), + DeleteStart::Deleting, + "nothing holds the backup before the registration's fence" + ); + pause.release(); + let refused = registration + .await + .expect("registration task") + .err() + .expect("the fence refuses the registration"); + assert!( + matches!(refused, StorageError::Validation(ref m) if m.contains("Backup not found")), + "{refused}" + ); + assert!( + matches!( + engine + .describe_table( + &account_id, + DescribeTableInput { + table_name: "RacedTarget".to_owned() + } + ) + .await, + Err(StorageError::TableNotFound(_)) + ), + "the refused registration left no target" + ); + assert!( + registrations(&engine, &account_id, &row.backup_arn) + .await + .is_empty(), + "no entry on the DELETING backup" + ); +} + +/// Two concurrent registrations of one backup leave one target: the in-use +/// pre-read is not atomic with the `active_restores` lightweight +/// transaction, so both can pass it (the first held before its fence while +/// the second runs whole), but the fence's `active_restores = null` admits +/// one; the other finds the registered target in its re-read, is +/// `BackupInUse` naming it, and removes its own target. +#[cfg(feature = "test-hooks")] +#[tokio::test] +async fn test_two_concurrent_registrations_of_one_backup_leave_one_target() { + if crate::helpers::skip_without_cassandra() { + return; + } + let engine = setup_engine().await; + let account = TestAccount::new(&engine, "extenddb_ttl_test").await; + let account_id = account.account_id.clone(); + let store = unique_store(); + let row = available_index_backup(&engine, &account_id, "Music", "twice", &store).await; + let pause = + extenddb_storage_cassandra::test_hooks::pause_before_restore_registration(&row.backup_arn); + let first = { + let engine = engine.clone(); + let account_id = account_id.clone(); + let registration = manifest_registration(&row, None); + tokio::spawn(async move { + engine + .claim_restore( + &account_id, + manifest_target_input("FirstOfTwo"), + ®istration, + ) + .await + }) + }; + // The first registration has passed the pre-read (no restore registered) + // and created its target; the second runs whole before the first's fence. + pause.reached().await; + let (second_desc, second_claim) = engine + .claim_restore( + &account_id, + manifest_target_input("SecondOfTwo"), + &manifest_registration(&row, None), + ) + .await + .expect("the second registration, whole before the first's fence"); + pause.release(); + let refused = first + .await + .expect("registration task") + .err() + .expect("the first's fence finds the second registered"); + assert!( + matches!(refused, StorageError::BackupInUse(ref m) if m.contains("SecondOfTwo")), + "{refused}" + ); + assert!( + matches!( + engine + .describe_table( + &account_id, + DescribeTableInput { + table_name: "FirstOfTwo".to_owned() + } + ) + .await, + Err(StorageError::TableNotFound(_)) + ), + "the refused registration left no target" + ); + assert_eq!( + registrations(&engine, &account_id, &row.backup_arn).await, + std::collections::HashMap::from([(second_desc.table_id.clone(), "SecondOfTwo".to_owned())]), + "one entry on the backup, the second's" + ); + assert_eq!( + engine + .backup_in_use(&row.key()) + .await + .expect("in use") + .as_deref(), + Some("SecondOfTwo") + ); + assert_eq!( + second_claim.heartbeat().await.expect("beat"), + Heartbeat::Confirmed, + "the second registration's claim is live" + ); +} + +/// A registration whose record a reconciler took over during its target's +/// DDL (no heartbeat runs there, so a DDL that outlasts the stale window +/// leaves the record dead) removes nothing: its `active_restores` transaction +/// finds the successor's entry, and its removal of the target is conditioned +/// on its own generation, which the takeover moved. The target stays, the +/// record is the successor's, and the registration answers an error naming +/// the lost claim rather than the refusal. +#[cfg(feature = "test-hooks")] +#[tokio::test] +async fn test_a_registration_taken_over_during_its_ddl_leaves_the_successors_target() { + if crate::helpers::skip_without_cassandra() { + return; + } + let engine = setup_engine().await; + let account = TestAccount::new(&engine, "extenddb_ttl_test").await; + let account_id = account.account_id.clone(); + let store = unique_store(); + let row = available_index_backup(&engine, &account_id, "Music", "slowddl", &store).await; + let pause = + extenddb_storage_cassandra::test_hooks::pause_before_restore_registration(&row.backup_arn); + let registration = { + let engine = engine.clone(); + let account_id = account_id.clone(); + let registration = manifest_registration(&row, Some("registrant:/s")); + tokio::spawn(async move { + engine + .claim_restore( + &account_id, + composite_target_input("SlowDdl"), + ®istration, + ) + .await + }) + }; + // The target's row and DDL exist; the registration is held before its + // `active_restores` transaction, as a DDL past the stale window would + // hold it. The record's heartbeat is the row insert's. + pause.reached().await; + let desc = describe(&engine, &account_id, "SlowDdl").await; + assert_eq!(desc.table_status, TableStatus::Creating); + let key = RestoreTargetKey { + account_id: account_id.clone(), + table_name: desc.table_name.clone(), + table_id: desc.table_id.clone(), + }; + let registrant_generation = engine + .read_restore_job(&key) + .await + .unwrap() + .unwrap() + .owner_generation; + tokio::time::sleep(Duration::from_millis(1_300)).await; + let successor = engine + .claim_restore_dead(&key, Duration::from_secs(1), Some("successor:/s")) + .await + .expect("claim") + .expect("the registration's record is stale and taken over"); + assert_ne!(successor.generation(), registrant_generation); + assert_eq!( + registrations(&engine, &account_id, &row.backup_arn).await, + std::collections::HashMap::from([(key.table_id.clone(), "SlowDdl".to_owned())]), + "the takeover re-applied the target's entry" + ); + + pause.release(); + let outcome = registration.await.expect("registration task"); + + // The successor's target, record and registration stand. + let target = engine + .describe_table( + &account_id, + DescribeTableInput { + table_name: "SlowDdl".to_owned(), + }, + ) + .await; + assert!( + matches!(&target, Ok(desc) if desc.table_status == TableStatus::Creating), + "the registrant's refusal path left the successor's target in place: {target:?}" + ); + let lost = outcome + .err() + .expect("the registration does not return a claim on a record another owns"); + assert!( + matches!(&lost, StorageError::Internal(m) if m.contains("lost its record to a takeover")), + "the registration names the lost claim, not the refusal: {lost}" + ); + let record = engine + .read_restore_job(&key) + .await + .expect("read") + .expect("the record stands"); + assert_eq!(record.status, JobStatus::Running); + assert_eq!(record.owner_generation, successor.generation()); + assert_eq!(record.spool_host.as_deref(), Some("successor:/s")); + assert_eq!( + registrations(&engine, &account_id, &row.backup_arn).await, + std::collections::HashMap::from([(key.table_id.clone(), "SlowDdl".to_owned())]), + "the successor's entry on the backup stands" + ); + assert_eq!( + successor.heartbeat().await.expect("beat"), + Heartbeat::Confirmed + ); + // The successor loads into the target the registrant left: its data and + // index tables are there. + assert_eq!( + engine + .bulk_load( + &key, + &*successor, + BulkLoadFile { + file_index: 0, + start_at: 0, + reapplying: false, + }, + futures::stream::iter((0..6).map(composite_item).map(Ok)).boxed(), + ) + .await + .expect("the successor's load"), + 6 + ); + assert_eq!(engine.count_restore_target(&key).await.expect("count"), 6); + assert_eq!( + index_rows(&engine, &account_id, &key.table_id, "gix").await, + 3 + ); +} + +/// A composite-key restore target with a GSI and an LSI. +fn composite_target_input(name: &str) -> CreateTableInput { + CreateTableInput { + table_name: name.to_owned(), + key_schema: vec![ks("id", KeyType::Hash), ks("sk", KeyType::Range)], + attribute_definitions: vec![ + attr("id", ScalarAttributeType::S), + attr("sk", ScalarAttributeType::N), + attr("g", ScalarAttributeType::S), + attr("l", ScalarAttributeType::S), + ], + billing_mode: Some(BillingMode::PayPerRequest), + global_secondary_indexes: Some(vec![GsiInput { + index_name: "gix".to_owned(), + key_schema: vec![ks("g", KeyType::Hash)], + projection: Projection { + projection_type: ProjectionType::KeysOnly, + non_key_attributes: None, + }, + provisioned_throughput: None, + }]), + local_secondary_indexes: Some(vec![LsiInput { + index_name: "lix".to_owned(), + key_schema: vec![ks("id", KeyType::Hash), ks("l", KeyType::Range)], + projection: Projection { + projection_type: ProjectionType::All, + non_key_attributes: None, + }, + }]), + ..Default::default() + } +} + +/// Item `i` of the composite fixture: `id` one of five, `sk` the index, +/// every second item carrying the GSI's key `g`, every third the LSI's `l`. +fn composite_item(i: usize) -> Item { + let mut it = BTreeMap::new(); + it.insert("id".to_owned(), AttributeValue::S(format!("p{}", i % 5))); + it.insert("sk".to_owned(), AttributeValue::N(i.to_string())); + if i.is_multiple_of(2) { + it.insert("g".to_owned(), AttributeValue::S(format!("g{}", i % 4))); + } + if i.is_multiple_of(3) { + it.insert("l".to_owned(), AttributeValue::S(format!("l{i}"))); + } + it +} + +/// The rows of the target's secondary index `index_name`. +async fn index_rows( + engine: &CassandraEngine, + account_id: &str, + table_id: &str, + index_name: &str, +) -> i64 { + use cdrs_tokio::types::IntoRustByName; + let rows = extenddb_storage_cassandra::cassandra_util::query_rows_quorum::( + &engine.session_arc(), + &format!( + "SELECT index_id FROM {}.indexes WHERE table_id = ? AND index_name = ?", + engine.catalog_keyspace() + ), + cdrs_tokio::query_values!(table_id, index_name), + "index id", + ) + .await + .expect("read the index"); + let index_id: String = rows + .first() + .expect("the index exists") + .get_by_name("index_id") + .ok() + .flatten() + .expect("index_id"); + let rows = extenddb_storage_cassandra::cassandra_util::query_rows_quorum::( + &engine.session_arc(), + &format!( + "SELECT COUNT(*) AS c FROM {}.{}", + engine.account_keyspace(account_id), + extenddb_storage_cassandra::data::ddl::index_table_name(&index_id) + ), + cdrs_tokio::query_values!(), + "index rows", + ) + .await + .expect("count the index rows"); + rows.first() + .expect("a count") + .get_by_name("c") + .ok() + .flatten() + .expect("c") +} + +/// `bulk_load` into a composite-key target with a GSI and an LSI: a file +/// resumed after its first three items (`start_at: 3`) applies exactly the +/// rest and records `start_at` plus the items applied as the file's total; +/// the GSI and LSI rows are written with the base rows; an empty file, and a +/// file resumed at its end, record their progress. +#[tokio::test] +async fn test_bulk_load_resumes_mid_file_with_its_index_rows_and_records_an_empty_file() { + if crate::helpers::skip_without_cassandra() { + return; + } + let engine = setup_engine().await; + let account = TestAccount::new(&engine, "extenddb_ttl_test").await; + let account_id = account.account_id.clone(); + let store = unique_store(); + let row = available_index_backup(&engine, &account_id, "Music", "nightly", &store).await; + let (desc, claim) = engine + .claim_restore( + &account_id, + composite_target_input("Composite"), + &manifest_registration(&row, None), + ) + .await + .expect("register a composite target"); + let key = RestoreTargetKey { + account_id: account_id.clone(), + table_name: desc.table_name.clone(), + table_id: desc.table_id.clone(), + }; + // File 0 resumed after three of its ten items. + let rest: Vec = (3..10).map(composite_item).collect(); + let applied = engine + .bulk_load( + &key, + &*claim, + BulkLoadFile { + file_index: 0, + start_at: 3, + reapplying: false, + }, + futures::stream::iter(rest.into_iter().map(Ok)).boxed(), + ) + .await + .expect("the rest of file 0"); + assert_eq!(applied, 7, "exactly the rest"); + let record = engine.read_restore_job(&key).await.unwrap().unwrap(); + assert_eq!(record.file_index, Some(0)); + assert_eq!( + record.items_applied, + Some(10), + "the running total is start_at plus the items applied" + ); + assert_eq!(engine.count_restore_target(&key).await.expect("count"), 7); + assert_eq!( + index_rows(&engine, &account_id, &key.table_id, "gix").await, + 3, + "the GSI rows of items 4, 6 and 8" + ); + assert_eq!( + index_rows(&engine, &account_id, &key.table_id, "lix").await, + 3, + "the LSI rows of items 3, 6 and 9" + ); + + // An empty file records its progress as complete. + let applied = engine + .bulk_load( + &key, + &*claim, + BulkLoadFile { + file_index: 1, + start_at: 0, + reapplying: false, + }, + futures::stream::iter(Vec::>::new()).boxed(), + ) + .await + .expect("an empty file"); + assert_eq!(applied, 0); + let record = engine.read_restore_job(&key).await.unwrap().unwrap(); + assert_eq!( + (record.file_index, record.items_applied), + (Some(1), Some(0)), + "an empty file's progress is recorded" + ); + // A file resumed at its end: `start_at` is the recorded total. + let applied = engine + .bulk_load( + &key, + &*claim, + BulkLoadFile { + file_index: 2, + start_at: 4, + reapplying: false, + }, + futures::stream::iter(Vec::>::new()).boxed(), + ) + .await + .expect("a file resumed at its end"); + assert_eq!(applied, 0); + let record = engine.read_restore_job(&key).await.unwrap().unwrap(); + assert_eq!( + (record.file_index, record.items_applied), + (Some(2), Some(4)) + ); + + // Another account's key, and the target once ACTIVE, name no CREATING + // restore target: `count_restore_target` is `TableNotFound`. + let foreign = RestoreTargetKey { + account_id: "111111111111".to_owned(), + ..key.clone() + }; + assert!(matches!( + engine.count_restore_target(&foreign).await, + Err(StorageError::TableNotFound(_)) + )); + assert_eq!( + engine + .finish_restore(&key, &*claim, 7, 4_096) + .await + .expect("finish"), + Finished::Active + ); + assert!( + matches!( + engine.count_restore_target(&key).await, + Err(StorageError::TableNotFound(_)) + ), + "an ACTIVE table is not a CREATING restore target" + ); +} + +/// A file resumed mid-way (`start_at: 3`) with more than one batch left: +/// the progress written after each batch inside the loop is `start_at` plus +/// the items applied so far, as the write after the tail is, so a second +/// crash between two of the resumed file's batches resumes from a running +/// total that counts the items the first run applied. Three items per batch, +/// seven items: the record reads 6 after the first batch, 9 after the +/// second, 10 at the end. +#[cfg(feature = "test-hooks")] +#[tokio::test] +async fn test_a_mid_file_resume_records_start_at_in_every_batch_progress() { + if crate::helpers::skip_without_cassandra() { + return; + } + let engine = setup_engine().await; + let account = TestAccount::new(&engine, "extenddb_ttl_test").await; + let account_id = account.account_id.clone(); + extenddb_storage_cassandra::test_hooks::set_restore_batch_size(&account_id, 3); + let store = unique_store(); + let row = available_index_backup(&engine, &account_id, "Music", "nightly", &store).await; + let (desc, claim) = engine + .claim_restore( + &account_id, + manifest_target_input("ResumedBatches"), + &manifest_registration(&row, None), + ) + .await + .expect("register"); + let key = RestoreTargetKey { + account_id: account_id.clone(), + table_name: desc.table_name.clone(), + table_id: desc.table_id.clone(), + }; + // The stream reads the record when asked for the first item of the + // second and of the third batch: by then the previous batch's progress + // is written. + let seen = std::sync::Arc::new(std::sync::Mutex::new(Vec::new())); + let seen_by_stream = std::sync::Arc::clone(&seen); + let reader = engine.clone(); + let read_key = key.clone(); + let rest: futures::stream::BoxStream<'static, Result> = + Box::pin(futures::stream::unfold(3usize, move |i| { + let reader = reader.clone(); + let key = read_key.clone(); + let seen = std::sync::Arc::clone(&seen_by_stream); + async move { + if i >= 10 { + return None; + } + if i == 6 || i == 9 { + let record = reader.read_restore_job(&key).await.unwrap().unwrap(); + seen.lock() + .expect("seen") + .push((record.file_index, record.items_applied)); + } + Some((Ok(id_item(i)), i + 1)) + } + })); + let applied = engine + .bulk_load( + &key, + &*claim, + BulkLoadFile { + file_index: 0, + start_at: 3, + reapplying: false, + }, + rest, + ) + .await + .expect("the rest of file 0"); + assert_eq!(applied, 7, "exactly the rest"); + assert_eq!( + *seen.lock().expect("seen"), + vec![(Some(0), Some(6)), (Some(0), Some(9))], + "each batch's progress is start_at plus the items applied so far" + ); + let record = engine.read_restore_job(&key).await.unwrap().unwrap(); + assert_eq!( + (record.file_index, record.items_applied), + (Some(0), Some(10)), + "the tail's progress is the file's total" + ); + assert_eq!(engine.count_restore_target(&key).await.expect("count"), 7); +} + +/// A progress lightweight transaction whose outcome is unknown (a write +/// timeout, a lost connection) does not fail the file: the record is re-read +/// at LOCAL_SERIAL. One whose transaction committed (the answer alone was +/// lost) is found applied and the file goes on; one whose transaction never +/// ran is found carrying the earlier progress and is written once more; a +/// second unknown outcome the record does not carry fails the file with an +/// error naming it, and the batches before it stand with their progress. +#[cfg(feature = "test-hooks")] +#[tokio::test] +async fn test_an_unknown_progress_write_outcome_is_decided_by_a_re_read() { + use extenddb_storage_cassandra::test_hooks::ProgressWriteFault; + if crate::helpers::skip_without_cassandra() { + return; + } + let engine = setup_engine().await; + let account = TestAccount::new(&engine, "extenddb_ttl_test").await; + let account_id = account.account_id.clone(); + extenddb_storage_cassandra::test_hooks::set_restore_batch_size(&account_id, 3); + let store = unique_store(); + let row = available_index_backup(&engine, &account_id, "Music", "nightly", &store).await; + let (desc, claim) = engine + .claim_restore( + &account_id, + manifest_target_input("UnknownProgress"), + &manifest_registration(&row, None), + ) + .await + .expect("register"); + let key = RestoreTargetKey { + account_id: account_id.clone(), + table_name: desc.table_name.clone(), + table_id: desc.table_id.clone(), + }; + + // File 0, three batches: the first batch's write lands but its answer is + // lost (the re-read finds the values written), the second's never runs + // (the re-read finds the first batch's values; written once more), the + // tail's runs as usual. The file completes with every item and its + // progress. + extenddb_storage_cassandra::test_hooks::fail_progress_writes( + &account_id, + &[ProgressWriteFault::Landed, ProgressWriteFault::Lost], + ); + let applied = engine + .bulk_load( + &key, + &*claim, + BulkLoadFile { + file_index: 0, + start_at: 0, + reapplying: false, + }, + item_stream(0, 7), + ) + .await + .expect("an unknown outcome the record decides does not fail the file"); + assert_eq!(applied, 7); + let record = engine.read_restore_job(&key).await.unwrap().unwrap(); + assert_eq!( + (record.file_index, record.items_applied), + (Some(0), Some(7)), + "the file's progress is recorded" + ); + assert_eq!(engine.count_restore_target(&key).await.expect("count"), 7); + + // File 1: the first batch's write is lost twice over; the second unknown + // outcome fails the file. The batch's items landed (the writes are + // idempotent upserts a resume rewrites) and the record carries file 0's + // progress, so a resume re-applies file 1 from its start. + extenddb_storage_cassandra::test_hooks::fail_progress_writes( + &account_id, + &[ProgressWriteFault::Lost, ProgressWriteFault::Lost], + ); + let failed = engine + .bulk_load( + &key, + &*claim, + BulkLoadFile { + file_index: 1, + start_at: 0, + reapplying: false, + }, + item_stream(7, 14), + ) + .await + .expect_err("two unknown outcomes the record does not carry fail the file"); + assert!( + matches!(&failed, StorageError::Internal(m) if m.contains("had no known outcome twice")), + "{failed}" + ); + let record = engine.read_restore_job(&key).await.unwrap().unwrap(); + assert_eq!( + (record.file_index, record.items_applied), + (Some(0), Some(7)), + "file 1 recorded no progress" + ); + assert_eq!( + engine.count_restore_target(&key).await.expect("count"), + 10, + "the first batch of file 1 landed before its progress write" + ); + assert_eq!( + claim.heartbeat().await.expect("beat"), + Heartbeat::Confirmed, + "the claim is this job's throughout" + ); +} + +/// A batch closes at the first item whose text takes it past +/// `RESTORE_BATCH_BYTES`, long before the count cap: a file of 100 items of +/// about 350 KB each (35 MB of text against the 32 MiB bound) is written in +/// two batches, the first's progress recorded before the stream hands over +/// its last item. +#[tokio::test] +async fn test_a_batch_closes_at_the_byte_bound_before_the_count_cap() { + if crate::helpers::skip_without_cassandra() { + return; + } + const N: usize = 100; + const TEXT: usize = 350_000; + const { + assert!(N < extenddb_storage::restore_engine::RESTORE_BATCH_ITEMS); + // 100 items cross the byte bound once. + assert!( + N * TEXT > extenddb_storage::restore_engine::RESTORE_BATCH_BYTES + && (N - 2) * TEXT < 2 * extenddb_storage::restore_engine::RESTORE_BATCH_BYTES + ); + } + let engine = setup_engine().await; + let account = TestAccount::new(&engine, "extenddb_ttl_test").await; + let account_id = account.account_id.clone(); + let store = unique_store(); + let row = available_index_backup(&engine, &account_id, "Music", "nightly", &store).await; + let (desc, claim) = engine + .claim_restore( + &account_id, + manifest_target_input("BigItems"), + &manifest_registration(&row, None), + ) + .await + .expect("register"); + let key = RestoreTargetKey { + account_id: account_id.clone(), + table_name: desc.table_name.clone(), + table_id: desc.table_id.clone(), + }; + // The stream reads the record when asked for the last item: by then the + // byte bound has closed the first batch and recorded its progress. + let seen = std::sync::Arc::new(std::sync::Mutex::new(None)); + let seen_by_stream = std::sync::Arc::clone(&seen); + let reader = engine.clone(); + let read_key = key.clone(); + let big: futures::stream::BoxStream<'static, Result> = + Box::pin(futures::stream::unfold(0usize, move |i| { + let reader = reader.clone(); + let key = read_key.clone(); + let seen = std::sync::Arc::clone(&seen_by_stream); + async move { + if i >= N { + return None; + } + if i == N - 1 { + let record = reader.read_restore_job(&key).await.unwrap().unwrap(); + *seen.lock().expect("seen") = Some((record.file_index, record.items_applied)); + } + let mut item = id_item(i); + item.insert("v".to_owned(), AttributeValue::S("x".repeat(TEXT))); + Some((Ok(item), i + 1)) + } + })); + let applied = engine + .bulk_load( + &key, + &*claim, + BulkLoadFile { + file_index: 0, + start_at: 0, + reapplying: false, + }, + big, + ) + .await + .expect("bulk_load"); + assert_eq!(applied as usize, N); + let (file_index, items_applied) = seen + .lock() + .expect("seen") + .expect("the stream read the record at its last item"); + assert_eq!(file_index, Some(0), "the first batch recorded its progress"); + let first_batch = items_applied.expect("the first batch's total") as usize; + assert!( + 0 < first_batch && first_batch < N, + "the first batch closed at the byte bound with {first_batch} of {N} items" + ); + assert!( + first_batch * TEXT + < extenddb_storage::restore_engine::RESTORE_BATCH_BYTES + TEXT + 1_000 * N, + "the batch closed at the first item past the bound: {first_batch} items" + ); + let record = engine.read_restore_job(&key).await.unwrap().unwrap(); + assert_eq!( + (record.file_index, record.items_applied), + (Some(0), Some(N as u64)) + ); + assert_eq!( + engine.count_restore_target(&key).await.expect("count") as usize, + N + ); +} + +/// A claim lost between two batches of one file (the record's generation +/// moved by a takeover while the load ran): the read before the next batch +/// finds the record no longer this claim's, the file ends as a lost claim, +/// and the second batch's items never land. +#[cfg(feature = "test-hooks")] +#[tokio::test] +async fn test_bulk_load_stops_at_a_lost_claim_between_two_batches() { + if crate::helpers::skip_without_cassandra() { + return; + } + let engine = setup_engine().await; + let account = TestAccount::new(&engine, "extenddb_ttl_test").await; + let account_id = account.account_id.clone(); + extenddb_storage_cassandra::test_hooks::set_restore_batch_size(&account_id, 3); + let store = unique_store(); + let row = available_index_backup(&engine, &account_id, "Music", "nightly", &store).await; + let (desc, claim) = engine + .claim_restore( + &account_id, + manifest_target_input("LostMidFile"), + &manifest_registration(&row, None), + ) + .await + .expect("register"); + let key = RestoreTargetKey { + account_id: account_id.clone(), + table_name: desc.table_name.clone(), + table_id: desc.table_id.clone(), + }; + // The stream hands the first batch's three items, then moves the + // record's generation when asked for the first item of the second. + let mover = engine.clone(); + let moved_key = key.clone(); + let generation = i64::try_from(claim.generation()).unwrap(); + let stream: futures::stream::BoxStream<'static, Result> = + Box::pin(futures::stream::unfold(0usize, move |i| { + let mover = mover.clone(); + let key = moved_key.clone(); + async move { + if i >= 7 { + return None; + } + if i == 3 { + apply( + &mover, + &format!( + "UPDATE {}.tables SET restore_owner_generation = ? \ + WHERE account_id = ? AND table_name = ? \ + IF table_id = ? AND restore_owner_generation = ?", + mover.catalog_keyspace() + ), + cdrs_tokio::query_values!( + generation + 1, + key.account_id.as_str(), + key.table_name.as_str(), + key.table_id.as_str(), + generation + ), + ) + .await; + } + Some((Ok(id_item(i)), i + 1)) + } + })); + let out = engine + .bulk_load( + &key, + &*claim, + BulkLoadFile { + file_index: 0, + start_at: 0, + reapplying: false, + }, + stream, + ) + .await; + assert!( + matches!(&out, Err(StorageError::Internal(m)) if m.contains("claim on the target was lost")), + "the read before the second batch finds another owner: {out:?}" + ); + assert_eq!( + engine.count_restore_target(&key).await.expect("count"), + 3, + "the first batch's items alone landed" + ); + let record = engine.read_restore_job(&key).await.unwrap().unwrap(); + assert_eq!( + (record.file_index, record.items_applied), + (Some(0), Some(3)), + "the first batch's progress stands" + ); +} + +/// `bulk_load` crosses a batch boundary, recording the file's running total +/// after each batch, and a re-applied batch changes nothing observable: the +/// writes are idempotent upserts of the base rows and of each GSI and LSI row +/// (the old index row's `DELETE` is skipped when the item's index keys did +/// not change, #742), so the base count and both index row counts equal +/// those of the first load, after the last batch alone is re-applied +/// (`start_at: 8`, a resume's shape) and after the whole file is. +#[cfg(feature = "test-hooks")] +#[tokio::test] +async fn test_bulk_load_crosses_a_batch_boundary_and_a_reapply_changes_nothing() { + if crate::helpers::skip_without_cassandra() { + return; + } + let engine = setup_engine().await; + let account = TestAccount::new(&engine, "extenddb_ttl_test").await; + let account_id = account.account_id.clone(); + // Four items per batch, so twelve items cross two boundaries. + extenddb_storage_cassandra::test_hooks::set_restore_batch_size(&account_id, 4); + let store = unique_store(); + let row = available_index_backup(&engine, &account_id, "Music", "nightly", &store).await; + let (desc, claim) = engine + .claim_restore( + &account_id, + composite_target_input("BulkTarget"), + &manifest_registration(&row, None), + ) + .await + .expect("register"); + let key = RestoreTargetKey { + account_id: account_id.clone(), + table_name: desc.table_name.clone(), + table_id: desc.table_id.clone(), + }; + let composite_items = |range: std::ops::Range| { + futures::stream::iter(range.map(composite_item).map(Ok)).boxed() + }; + // Items 0, 2, 4, 6, 8 and 10 carry the GSI's key; 0, 3, 6 and 9 the LSI's. + let (gsi_rows, lsi_rows) = (6, 4); + + let applied = engine + .bulk_load( + &key, + &*claim, + BulkLoadFile { + file_index: 0, + start_at: 0, + reapplying: false, + }, + composite_items(0..12), + ) + .await + .expect("bulk_load"); + assert_eq!(applied, 12); + let record = engine.read_restore_job(&key).await.unwrap().unwrap(); + assert_eq!(record.file_index, Some(0)); + assert_eq!(record.items_applied, Some(12)); + assert_eq!(engine.count_restore_target(&key).await.expect("count"), 12); + assert_eq!( + index_rows(&engine, &account_id, &key.table_id, "gix").await, + gsi_rows + ); + assert_eq!( + index_rows(&engine, &account_id, &key.table_id, "lix").await, + lsi_rows + ); + + // The last batch re-applied (a resume whose progress write was lost): + // the same base and index rows. + let reapplied = engine + .bulk_load( + &key, + &*claim, + BulkLoadFile { + file_index: 0, + start_at: 8, + reapplying: true, + }, + composite_items(8..12), + ) + .await + .expect("bulk_load re-apply of the last batch"); + assert_eq!(reapplied, 4); + assert_eq!(engine.count_restore_target(&key).await.expect("count"), 12); + assert_eq!( + index_rows(&engine, &account_id, &key.table_id, "gix").await, + gsi_rows, + "a re-applied batch keeps every GSI row" + ); + assert_eq!( + index_rows(&engine, &account_id, &key.table_id, "lix").await, + lsi_rows, + "a re-applied batch keeps every LSI row" + ); + + // The whole file re-applied: the count stays 12 and the index rows stand. + let reapplied = engine + .bulk_load( + &key, + &*claim, + BulkLoadFile { + file_index: 0, + start_at: 0, + reapplying: true, + }, + composite_items(0..12), + ) + .await + .expect("bulk_load re-apply"); + assert_eq!(reapplied, 12); + assert_eq!( + engine.count_restore_target(&key).await.expect("count"), + 12, + "a re-applied file changes nothing" + ); + assert_eq!( + ( + index_rows(&engine, &account_id, &key.table_id, "gix").await, + index_rows(&engine, &account_id, &key.table_id, "lix").await + ), + (gsi_rows, lsi_rows), + "a re-applied file keeps every index row" + ); +} + +/// An unordered backup carrying one key in two data files applies both items +/// (the second overwrites the first), so `count_restore_target` is one short +/// of the manifest's total: the shortfall is the duplicate, which the +/// engine's count check turns into a FAILED restore. +#[tokio::test] +async fn test_an_unordered_duplicate_key_is_caught_by_the_count_alone() { + if crate::helpers::skip_without_cassandra() { + return; + } + let engine = setup_engine().await; + let account = TestAccount::new(&engine, "extenddb_ttl_test").await; + let account_id = account.account_id.clone(); + let store = unique_store(); + let row = available_index_backup(&engine, &account_id, "Music", "nightly", &store).await; + let (desc, claim) = engine + .claim_restore( + &account_id, + manifest_target_input("DupTarget"), + &manifest_registration(&row, None), + ) + .await + .expect("register"); + let key = RestoreTargetKey { + account_id: account_id.clone(), + table_name: desc.table_name.clone(), + table_id: desc.table_id.clone(), + }; + // File 0: items 0..6. File 1: items 6..12 and item 0 again (the backup + // holds k00000 twice). The manifest's total is 13 items. + engine + .bulk_load( + &key, + &*claim, + BulkLoadFile { + file_index: 0, + start_at: 0, + reapplying: false, + }, + item_stream(0, 6), + ) + .await + .expect("file 0"); + let file_one: Vec = (6..12) + .map(id_item) + .chain(std::iter::once(id_item(0))) + .collect(); + engine + .bulk_load( + &key, + &*claim, + BulkLoadFile { + file_index: 1, + start_at: 0, + reapplying: false, + }, + futures::stream::iter(file_one.into_iter().map(Ok)).boxed(), + ) + .await + .expect("file 1"); + let count = engine.count_restore_target(&key).await.expect("count"); + assert_eq!(count, 12, "the duplicated key collapsed into one row"); + assert!(count < 13, "the count is one short of the manifest's 13"); +} + +/// `bulk_load` stops with a lost-claim error when the record's generation has +/// moved (the reconciler took it over): the progress lightweight transaction +/// matches no RUNNING record under the claim's generation. +#[tokio::test] +async fn test_bulk_load_is_refused_when_the_claim_was_lost() { + if crate::helpers::skip_without_cassandra() { + return; + } + let engine = setup_engine().await; + let account = TestAccount::new(&engine, "extenddb_ttl_test").await; + let account_id = account.account_id.clone(); + let store = unique_store(); + let row = available_index_backup(&engine, &account_id, "Music", "nightly", &store).await; + let (desc, claim) = engine + .claim_restore( + &account_id, + manifest_target_input("LostTarget"), + &manifest_registration(&row, None), + ) + .await + .expect("register"); + let key = RestoreTargetKey { + account_id: account_id.clone(), + table_name: desc.table_name.clone(), + table_id: desc.table_id.clone(), + }; + // Another owner bumps the generation. + apply( + &engine, + &format!( + "UPDATE {}.tables SET restore_owner_generation = ? \ + WHERE account_id = ? AND table_name = ? \ + IF table_id = ? AND restore_owner_generation = ?", + engine.catalog_keyspace() + ), + cdrs_tokio::query_values!( + i64::try_from(claim.generation()).unwrap() + 1, + account_id.as_str(), + key.table_name.as_str(), + key.table_id.as_str(), + i64::try_from(claim.generation()).unwrap() + ), + ) + .await; + let error = engine + .bulk_load( + &key, + &*claim, + BulkLoadFile { + file_index: 0, + start_at: 0, + reapplying: false, + }, + item_stream(0, 3), + ) + .await + .expect_err("the progress write finds another owner"); + assert!( + error.to_string().contains("claim on the target was lost"), + "{error}" + ); +} + +/// `finish_restore` flips the target ACTIVE and marks the record DONE, and a +/// stale generation's `finish_restore` is `NotMatched`. +#[tokio::test] +async fn test_finish_restore_flips_the_target_and_refuses_a_stale_generation() { + if crate::helpers::skip_without_cassandra() { + return; + } + let engine = setup_engine().await; + let account = TestAccount::new(&engine, "extenddb_ttl_test").await; + let account_id = account.account_id.clone(); + let store = unique_store(); + let row = available_index_backup(&engine, &account_id, "Music", "nightly", &store).await; + let (desc, claim) = engine + .claim_restore( + &account_id, + manifest_target_input("FinishTarget"), + &manifest_registration(&row, None), + ) + .await + .expect("register"); + let key = RestoreTargetKey { + account_id: account_id.clone(), + table_name: desc.table_name.clone(), + table_id: desc.table_id.clone(), + }; + engine + .bulk_load( + &key, + &*claim, + BulkLoadFile { + file_index: 0, + start_at: 0, + reapplying: false, + }, + item_stream(0, 5), + ) + .await + .expect("bulk_load"); + let count = engine.count_restore_target(&key).await.expect("count"); + assert_eq!(count, 5); + + assert_eq!( + engine + .finish_restore(&key, &*claim, count, 4_096) + .await + .expect("finish"), + Finished::Active + ); + let described = describe(&engine, &account_id, "FinishTarget").await; + assert_eq!(described.table_status, TableStatus::Active); + assert_eq!(described.item_count, 5); + + // A second flip, the record no longer RUNNING, matches nothing. + assert!(matches!( + engine + .finish_restore(&key, &*claim, count, 4_096) + .await + .expect("finish again"), + Finished::NotMatched(_) + )); +} + +/// After a claim writes the record FAILED (leaving `table_status` alone), +/// `remove_restore_target` performs the two-write removal (CREATING to +/// DELETING, then the drop) and the target is gone. +#[tokio::test] +async fn test_remove_restore_target_after_a_failed_write() { + if crate::helpers::skip_without_cassandra() { + return; + } + let engine = setup_engine().await; + let account = TestAccount::new(&engine, "extenddb_ttl_test").await; + let account_id = account.account_id.clone(); + let store = unique_store(); + let row = available_index_backup(&engine, &account_id, "Music", "nightly", &store).await; + let (desc, claim) = engine + .claim_restore( + &account_id, + manifest_target_input("FailTarget"), + &manifest_registration(&row, None), + ) + .await + .expect("register"); + let key = RestoreTargetKey { + account_id: account_id.clone(), + table_name: desc.table_name.clone(), + table_id: desc.table_id.clone(), + }; + assert_eq!( + claim + .set_status( + JobStatus::Running, + JobStatus::Failed, + &StatusFields { + failure_reason: Some("a data file failed".to_owned()), + completed: true, + ..StatusFields::default() + } + ) + .await + .expect("fail the record"), + WriteOutcome::Matched + ); + // The FAILED write left the table CREATING; the record carries the reason. + assert_eq!( + describe(&engine, &account_id, "FailTarget") + .await + .table_status, + TableStatus::Creating, + "the FAILED write leaves table_status alone for the removal's two-write step" + ); + let record = engine.read_restore_job(&key).await.unwrap().unwrap(); + assert_eq!(record.status, JobStatus::Failed); + assert_eq!(record.failure_reason.as_deref(), Some("a data file failed")); + + assert!( + engine.remove_restore_target(&key).await.expect("remove"), + "the two-write removal of a FAILED target succeeds" + ); + assert!( + matches!( + engine + .describe_table( + &account_id, + DescribeTableInput { + table_name: "FailTarget".to_owned() + } + ) + .await, + Err(StorageError::TableNotFound(_)) + ), + "the target is gone" + ); + assert!( + engine.read_restore_job(&key).await.expect("read").is_none(), + "the record went with the target" + ); +} diff --git a/crates/storage-cassandra/tests/direct/index.rs b/crates/storage-cassandra/tests/direct/index.rs index ddb4ac02c..9830072c5 100644 --- a/crates/storage-cassandra/tests/direct/index.rs +++ b/crates/storage-cassandra/tests/direct/index.rs @@ -624,19 +624,30 @@ async fn gsi_query_count( gsi_pk_attr: &str, gsi_pk_value: &str, ) -> usize { + index_items(engine, table, gsi_name, gsi_pk_attr, gsi_pk_value) + .await + .len() +} + +/// The items a Query on `index_name` with `pk_attr = pk_value` returns. +async fn index_items( + engine: &CassandraEngine, + table: &crate::helpers::TestTable, + index_name: &str, + pk_attr: &str, + pk_value: &str, +) -> Vec { use extenddb_core::expression::{Expr, KeyCondition, PathElement}; let key_condition = KeyCondition { - pk_path: vec![PathElement::Attribute(gsi_pk_attr.to_string())], + pk_path: vec![PathElement::Attribute(pk_attr.to_string())], pk_value: Expr::Placeholder(":v".to_string()), sk_condition: None, extra_pk_conditions: vec![], extra_sk_conditions: vec![], }; let mut maps = ExpressionMaps::default(); - maps.values.insert( - ":v".to_string(), - AttributeValue::S(gsi_pk_value.to_string()), - ); + maps.values + .insert(":v".to_string(), AttributeValue::S(pk_value.to_string())); engine .query( &table.key_info, @@ -645,11 +656,173 @@ async fn gsi_query_count( true, None, None, - Some(gsi_name), + Some(index_name), ) .await - .map(|(items, _)| items.len()) - .unwrap_or(0) + .map(|(items, _)| items) + .unwrap_or_default() +} + +/// The `S` value of `attr` in the one item `items` holds. +#[track_caller] +fn the_one(items: &[Item], attr: &str) -> String { + assert_eq!(items.len(), 1, "one index row: {items:?}"); + match items[0].get(attr) { + Some(AttributeValue::S(s)) => s.clone(), + other => panic!("{attr} is not a string: {other:?}"), + } +} + +/// Item `("a", 1)` of the composite fixture with the GSI key `g`, the LSI +/// key `l` and a projected non-key attribute `v`. +fn composite_item(g: &str, l: &str, v: &str) -> Item { + let mut item = Item::new(); + item.insert("id".to_string(), AttributeValue::S("a".to_string())); + item.insert("sk".to_string(), AttributeValue::N("1".to_string())); + item.insert("g".to_string(), AttributeValue::S(g.to_string())); + item.insert("l".to_string(), AttributeValue::S(l.to_string())); + item.insert("v".to_string(), AttributeValue::S(v.to_string())); + item +} + +/// A PutItem of an item whose index keys did not change rewrites its GSI and +/// LSI rows instead of deleting them (#742: a `DELETE` and an `INSERT` of one +/// row in one logged batch share the batch's timestamp and the tombstone +/// wins): the same item put twice keeps both rows; a put that changes a +/// projected non-key attribute updates the row; a put that changes the GSI +/// key moves the row (the old gone, the new present); the same for the LSI +/// key. +#[tokio::test] +async fn test_a_put_of_an_item_with_unchanged_index_keys_keeps_its_gsi_and_lsi_rows() { + if crate::helpers::skip_without_cassandra() { + return; + } + let config = test_config(); + let engine = CassandraEngine::new(&config, "us-east-1").await.unwrap(); + let table = crate::helpers::TestTable::with_gsi_and_lsi(&engine, "RePutKeepsIndexRows").await; + set_gsi_delay(&engine, &table.key_info.table_id, "gix", 0).await; + let maps = ExpressionMaps::default(); + let put = |item: Item| engine.put_item(&table.key_info, item, false, None, &maps, None); + + put(composite_item("g1", "l1", "v1")).await.expect("put"); + assert_eq!( + the_one(&index_items(&engine, &table, "gix", "g", "g1").await, "v"), + "v1" + ); + assert_eq!( + the_one(&index_items(&engine, &table, "lix", "id", "a").await, "v"), + "v1" + ); + + // The same item again: both rows stand. + put(composite_item("g1", "l1", "v1")).await.expect("re-put"); + assert_eq!( + the_one(&index_items(&engine, &table, "gix", "g", "g1").await, "v"), + "v1", + "the GSI row of an unchanged item survives its re-put" + ); + assert_eq!( + the_one(&index_items(&engine, &table, "lix", "id", "a").await, "v"), + "v1", + "the LSI row of an unchanged item survives its re-put" + ); + + // A projected non-key attribute changes: the rows carry the new value. + put(composite_item("g1", "l1", "v2")).await.expect("put v2"); + assert_eq!( + the_one(&index_items(&engine, &table, "gix", "g", "g1").await, "v"), + "v2", + "the GSI row carries the new projected value" + ); + assert_eq!( + the_one(&index_items(&engine, &table, "lix", "id", "a").await, "v"), + "v2", + "the LSI row carries the new projected value" + ); + + // The GSI key changes: the row moves. + put(composite_item("g2", "l1", "v2")).await.expect("put g2"); + assert_eq!( + gsi_query_count(&engine, &table, "gix", "g", "g1").await, + 0, + "the old GSI row is gone" + ); + assert_eq!( + the_one(&index_items(&engine, &table, "gix", "g", "g2").await, "v"), + "v2", + "the new GSI row is present" + ); + assert_eq!( + the_one(&index_items(&engine, &table, "lix", "id", "a").await, "l"), + "l1", + "the LSI row is untouched by a GSI key change" + ); + + // The LSI key changes: the one row under `id = a` carries the new key. + put(composite_item("g2", "l2", "v2")).await.expect("put l2"); + assert_eq!( + the_one(&index_items(&engine, &table, "lix", "id", "a").await, "l"), + "l2", + "the LSI row moved to the new key and the old row is gone" + ); + assert_eq!( + the_one(&index_items(&engine, &table, "gix", "g", "g2").await, "l"), + "l2", + "the GSI row carries the new LSI key as a projected attribute" + ); +} + +/// The GSI worker's path (`gsi_apply_index`, a GSI with a propagation delay) +/// applies the same rule as the synchronous one: a pending entry whose old +/// and new items address the same index row rewrites the row instead of +/// deleting it (#742), so an item put twice unchanged, then with a new +/// projected value, has one GSI row carrying the latest value once the queue +/// drains. +#[tokio::test] +async fn test_the_gsi_worker_keeps_the_row_of_an_item_with_an_unchanged_key() { + if crate::helpers::skip_without_cassandra() { + return; + } + let config = test_config(); + let engine = Arc::new(CassandraEngine::new(&config, "us-east-1").await.unwrap()); + let table = + crate::helpers::TestTable::with_gsi(&engine, "AsyncGsiRePutTable", "GsiIdx", "gpk").await; + set_gsi_delay(&engine, &table.key_info.table_id, "GsiIdx", 50).await; + let account_keyspace = engine.account_keyspace(&table.key_info.account_id); + let maps = ExpressionMaps::default(); + + // The same item twice, then with a new projected value; the GSI key `gpk` + // never changes, so every pending entry's old and new rows are one row. + for v in ["v1", "v1", "v2"] { + let mut item = Item::new(); + item.insert("id".to_string(), AttributeValue::S("X".to_string())); + item.insert("gpk".to_string(), AttributeValue::S("same".to_string())); + item.insert("v".to_string(), AttributeValue::S(v.to_string())); + engine + .put_item(&table.key_info, item, false, None, &maps, None) + .await + .expect("put_item"); + } + assert_eq!(gsi_pending_count(&engine, &account_keyspace).await, 3); + + let _worker_guard = extenddb_storage_cassandra::workers::spawn_gsi_workers(engine.clone()); + let deadline = tokio::time::Instant::now() + std::time::Duration::from_secs(15); + while gsi_pending_count(&engine, &account_keyspace).await > 0 { + assert!( + tokio::time::Instant::now() < deadline, + "gsi_pending did not drain within 15 s" + ); + tokio::time::sleep(std::time::Duration::from_millis(100)).await; + } + + assert_eq!( + the_one( + &index_items(&engine, &table, "GsiIdx", "gpk", "same").await, + "v" + ), + "v2", + "the worker rewrote the one GSI row with the latest projection instead of deleting it" + ); } #[tokio::test] diff --git a/crates/storage-mongodb/Cargo.toml b/crates/storage-mongodb/Cargo.toml index f455971dc..87a0e7899 100644 --- a/crates/storage-mongodb/Cargo.toml +++ b/crates/storage-mongodb/Cargo.toml @@ -10,9 +10,10 @@ license.workspace = true [features] # Enables deterministic coordination hooks used only by the MongoDB integration -# test binary (the mongodb-test-hooks server build). Production builds do not -# compile these hooks. -test-hooks = [] +# test binary (the mongodb-test-hooks server build), and the restore pause +# point of `extenddb-storage` the kill-point harness drives. Production builds +# do not compile these hooks. +test-hooks = ["extenddb-storage/test-hooks"] # Compiles the snapshot page-size setter, the open pause and the shared # snapshot conformance check the crate's snapshot tests use. Separate from # `test-hooks` so the crate's own tests build the production commit and diff --git a/crates/storage-mongodb/src/backup_catalog.rs b/crates/storage-mongodb/src/backup_catalog.rs index 20acf3627..a8eb32e8c 100644 --- a/crates/storage-mongodb/src/backup_catalog.rs +++ b/crates/storage-mongodb/src/backup_catalog.rs @@ -40,10 +40,12 @@ use std::time::Duration; use extenddb_core::backup_format::BackupArn; +use extenddb_core::types::{CreateTableInput, RestoreSummary, TableDescription}; use extenddb_storage::backup_catalog::{ BackupCatalog, BackupKey, BackupRow, BackupStatus, DeleteStart, Heartbeat, JobClaim, JobRow, - JobStatus, MaintenanceClaim, RestoreJobRow, RestoreKind, RestoreTargetKey, StaleClock, - StatusFields, StoreIdentity, StoreKind, TimeBounds, WriteOutcome, new_generation, + JobStatus, MaintenanceClaim, RestoreJobRow, RestoreKind, RestoreRegistered, + RestoreRegistrationRow, RestoreTargetKey, StaleClock, StatusFields, StoreIdentity, StoreKind, + TimeBounds, WriteOutcome, new_generation, }; use extenddb_storage::error::StorageError; use futures::TryStreamExt; @@ -52,7 +54,9 @@ use futures::stream::BoxStream; use mongodb::bson::{self, Bson, Document, doc}; use crate::MongoEngine; -use crate::backup_engine::{age_is_stale, restoring_target_filter}; +use crate::backup_engine::{ + RestoreRecord, RestoreRegistration, age_is_stale, restoring_target_filter, +}; use crate::data_engine::{ CommitOutcome, TRANSIENT_RETRY_ATTEMPTS, TxErr, backoff_sleep, is_duplicate_key, }; @@ -70,10 +74,13 @@ pub(crate) const LOCK_SEQ: &str = "lock_seq"; /// document, with the field names the job record has on every backend. pub(crate) const RESTORE_JOB: &str = "restore_job"; pub(crate) const RESTORE_JOB_STATUS: &str = "restore_job.status"; +pub(crate) const RESTORE_JOB_KIND: &str = "restore_job.kind"; pub(crate) const RESTORE_JOB_GENERATION: &str = "restore_job.owner_generation"; pub(crate) const RESTORE_JOB_HEARTBEAT: &str = "restore_job.heartbeat_at"; pub(crate) const RESTORE_JOB_FAILURE_REASON: &str = "restore_job.failure_reason"; pub(crate) const RESTORE_JOB_COMPLETED_AT: &str = "restore_job.completed_at"; +pub(crate) const RESTORE_JOB_SPOOL_HOST: &str = "restore_job.spool_host"; +pub(crate) const RESTORE_JOB_FILE_INDEX: &str = "restore_job.file_index"; fn internal(e: impl std::fmt::Display) -> StorageError { StorageError::Internal(e.to_string()) @@ -398,6 +405,161 @@ impl JobClaim for MongoBackupClaim { } } +/// The filter of the `tables` document `key` names: the `_id` and the table +/// id, so a table re-created under the name is never the one addressed. +fn target_filter(key: &RestoreTargetKey) -> Document { + doc! { + "_id": { "account_id": &key.account_id, "table_name": &key.table_name }, + "table_id": &key.table_id, + } +} + +/// One restore target's `tables` document as a [`RestoreJobRow`]: the key, +/// the provenance and every member of the `restore_job` subdocument; the +/// store is read when all three of its fields are there. +pub(crate) fn restore_job_from_doc(doc: &Document) -> Result { + let id = doc + .get_document("_id") + .map_err(|_| StorageError::Internal("table is missing _id".to_owned()))?; + let job = doc.get_document(RESTORE_JOB).map_err(|_| { + StorageError::Internal("restore target is missing its job record".to_owned()) + })?; + let source_backup_arn = required_str(doc, "restore_source_backup_arn")?; + let store = match ( + job.get_str("store_kind"), + job.get_str("store_root"), + job.get_str("store_id"), + ) { + (Ok(kind), Ok(store_root), Ok(store_id)) => Some(StoreIdentity { + store_kind: StoreKind::parse(kind).ok_or_else(|| { + StorageError::Internal(format!( + "restore of {source_backup_arn} carries an unknown store kind {kind:?}" + )) + })?, + store_root: store_root.to_owned(), + store_id: store_id.to_owned(), + }), + _ => None, + }; + Ok(RestoreJobRow { + target: RestoreTargetKey { + account_id: required_str(id, "account_id")?, + table_name: required_str(id, "table_name")?, + table_id: required_str(doc, "table_id")?, + }, + source_backup_arn, + kind: RestoreKind::parse(&required_str(job, "kind")?)?, + status: JobStatus::parse(&required_str(job, "status")?)?, + owner_generation: generation_from_field(job, "owner_generation"), + heartbeat_at: optional_datetime(job, "heartbeat_at").unwrap_or(0.0), + store, + spool_host: job.get_str("spool_host").ok().map(str::to_owned), + file_index: match job.get("file_index") { + Some(Bson::Int32(i)) => u32::try_from(*i).ok(), + Some(Bson::Int64(i)) => u32::try_from(*i).ok(), + _ => None, + }, + items_applied: None, + failure_reason: job.get_str("failure_reason").ok().map(str::to_owned), + completed_at: optional_datetime(job, "completed_at"), + }) +} + +/// A store-backed restore job's handle on the job record of its target's +/// `tables` document: every write carries the target's key, its table id +/// and the generation, and the heartbeat and the status writes also require +/// the record RUNNING (or the `from` status). +struct MongoRestoreClaim { + tables: mongodb::Collection, + key: RestoreTargetKey, + generation: u64, +} + +impl MongoRestoreClaim { + /// The filter of the record while it is in `status` under this claim. + fn owned_filter(&self, status: JobStatus) -> Document { + let mut filter = target_filter(&self.key); + filter.insert(RESTORE_JOB_STATUS, status.as_str()); + filter.insert(RESTORE_JOB_GENERATION, generation_field(self.generation)); + filter + } +} + +impl JobClaim for MongoRestoreClaim { + fn generation(&self) -> u64 { + self.generation + } + + fn heartbeat(&self) -> BoxFuture<'_, Result> { + Box::pin(async move { + let result = self + .tables + .update_one( + self.owned_filter(JobStatus::Running), + vec![doc! { "$set": { RESTORE_JOB_HEARTBEAT: "$$NOW" } }], + ) + .await + .map_err(|e| StorageError::Internal(format!("restore heartbeat: {e}")))?; + Ok(if result.matched_count == 1 { + Heartbeat::Confirmed + } else { + Heartbeat::Lost + }) + }) + } + + fn set_status<'a>( + &'a self, + from: JobStatus, + to: JobStatus, + fields: &'a StatusFields, + ) -> BoxFuture<'a, Result> { + Box::pin(async move { + let mut set = doc! { RESTORE_JOB_STATUS: to.as_str() }; + if let Some(reason) = &fields.failure_reason { + set.insert(RESTORE_JOB_FAILURE_REASON, reason.as_str()); + } + if fields.completed { + set.insert(RESTORE_JOB_COMPLETED_AT, "$$NOW"); + } + let set = literal_set(set); + match self + .tables + .update_one(self.owned_filter(from), vec![doc! { "$set": set }]) + .await + { + Ok(result) if result.matched_count == 1 => Ok(WriteOutcome::Matched), + Ok(_) => Ok(WriteOutcome::NotMatched), + Err(e) if outcome_unknown(&e) => Ok(WriteOutcome::Unknown), + Err(e) => Err(internal(e)), + } + }) + } + + fn read(&self) -> BoxFuture<'_, Result> { + Box::pin(async move { + let doc = self + .tables + .find_one(target_filter(&self.key)) + .await + .map_err(internal)? + .ok_or_else(|| { + StorageError::TableNotFound(format!( + "restore target {} (table id {}) has no catalog document", + self.key.table_name, self.key.table_id + )) + })?; + let row = restore_job_from_doc(&doc)?; + Ok(JobRow { + status: row.status, + owner_generation: row.owner_generation, + heartbeat_at: row.heartbeat_at, + failure_reason: row.failure_reason, + }) + }) + } +} + impl MongoEngine { pub(crate) fn backup_index(&self) -> mongodb::Collection { self.catalog_db.collection::(BACKUP_INDEX) @@ -499,6 +661,143 @@ impl MongoEngine { key.backup_arn ))) } + + /// A store-backed restore's registration (the design's 5.3 step 4) as + /// one session transaction: `$inc` the `backup_index` row's `lock_seq` + /// while it is AVAILABLE (the lock `start_backup_delete` takes, so the + /// two order wholly one before the other), run the in-use rule, insert + /// the CREATING target with its provenance and its RUNNING `kind = + /// manifest` record under a fresh generation, the store and the + /// claimant's spool, with the server's clock on its heartbeat. The + /// transaction commits right after the insert, before the target's + /// collections and indexes are created; one that lost a write conflict + /// (a DeleteBackup or another registration of the backup) is re-run and + /// sees the other's result. + /// + /// Refusals: a missing or non-AVAILABLE row is `Backup not found`; a + /// CREATING target of the backup is `BackupInUse` naming it; a taken + /// name is what `create_table_for_restore` reports (`TableAlreadyExists`, + /// or `TableBeingRestored` when the name is a CREATING restore target). + async fn claim_restore_tx( + &self, + account_id: &str, + input: CreateTableInput, + registration: &RestoreRegistrationRow, + ) -> Result { + let target_table_name = input.table_name.clone(); + let backup_arn = registration.backup.backup_arn.as_str(); + let index = self.backup_index(); + let mut session = self.client.start_session().await.map_err(internal)?; + let tx_options = self.write_transaction_options(); + let generation = new_generation(); + for attempt in 0..TRANSIENT_RETRY_ATTEMPTS { + session + .start_transaction() + .with_options(tx_options.clone()) + .await + .map_err(internal)?; + let body: Result, TxErr> = async { + let mut lock = key_filter(®istration.backup); + lock.insert("status", BackupStatus::Available.as_str()); + let locked = index + .find_one_and_update(lock, doc! { "$inc": { LOCK_SEQ: 1_i64 } }) + .session(&mut session) + .await + .map_err(TxErr::from)?; + if locked.is_none() { + return Ok(Err(not_found(backup_arn))); + } + if let Some(restoring) = self + .creating_restore_target(account_id, backup_arn, Some(&mut session)) + .await + .map_err(TxErr::from)? + { + return Ok(Err(backup_in_use(&restoring, backup_arn))); + } + let mut target = RestoreRegistration { + backup_arn, + source_table_arn: ®istration.source_table_arn, + restore_date_time: epoch_secs_datetime(registration.restore_date_time), + owner_generation: generation, + // This process's clock, replaced by the server's inside + // the transaction (`stamp_server_clock`); test builds set + // it behind to prove the replacement. + first_heartbeat: self.local_now(), + record: RestoreRecord::Manifest { + store: ®istration.store, + spool_host: registration.spool_host.as_deref(), + }, + session: Some(&mut session), + }; + match self + .create_table_for_restore(account_id, input.clone(), &mut target) + .await + { + Ok(desc) => Ok(Ok(desc)), + Err(StorageError::Transient(_)) => Err(TxErr::Transient), + Err(e) => { + // A creation that failed after its insert committed + // (a collection or index could not be created) left a + // registered target under this generation; a name + // refusal left nothing, and the removal matches + // nothing. + if let Err(cleanup) = self + .remove_target_of_generation(account_id, &target_table_name, generation) + .await + { + tracing::error!( + "restore of {backup_arn} into {target_table_name} failed during \ + target creation ({e}), and cleanup failed: {cleanup}" + ); + } + Ok(Err(e)) + } + } + } + .await; + match body { + // The insert committed its transaction before the target's + // collections were created; nothing is left to commit. + Ok(Ok(mut desc)) => { + desc.restore_summary = Some(RestoreSummary { + source_backup_arn: Some(backup_arn.to_owned()), + source_table_arn: Some(registration.source_table_arn.clone()), + restore_date_time: registration.restore_date_time, + restore_in_progress: true, + }); + let key = RestoreTargetKey { + account_id: account_id.to_owned(), + table_name: target_table_name, + table_id: desc.table_id.clone(), + }; + return Ok(( + desc, + Box::new(MongoRestoreClaim { + tables: self.tables(), + key, + generation, + }) as Box, + )); + } + Ok(Err(refusal)) => { + let _ = session.abort_transaction().await; + return Err(refusal); + } + Err(TxErr::Transient) => { + let _ = session.abort_transaction().await; + backoff_sleep(attempt).await; + } + Err(TxErr::Fatal(e)) => { + let _ = session.abort_transaction().await; + return Err(e); + } + } + } + Err(StorageError::Internal(format!( + "the registration of the restore of {backup_arn} into {target_table_name} did not \ + commit after {TRANSIENT_RETRY_ATTEMPTS} conflicting attempts" + ))) + } } impl BackupCatalog for MongoEngine { @@ -839,30 +1138,98 @@ impl BackupCatalog for MongoEngine { .try_collect() .await .map_err(internal)?; - docs.iter() - .map(|doc| { - let id = doc - .get_document("_id") - .map_err(|_| StorageError::Internal("table is missing _id".to_owned()))?; - let job = doc.get_document(RESTORE_JOB).map_err(|_| { - StorageError::Internal( - "restore target is missing its job record".to_owned(), - ) - })?; - Ok(RestoreJobRow { - target: RestoreTargetKey { - account_id: required_str(id, "account_id")?, - table_name: required_str(id, "table_name")?, - table_id: required_str(doc, "table_id")?, - }, - source_backup_arn: required_str(doc, "restore_source_backup_arn")?, - kind: RestoreKind::parse(&required_str(job, "kind")?)?, - status: JobStatus::parse(&required_str(job, "status")?)?, - owner_generation: generation_from_field(job, "owner_generation"), - heartbeat_at: optional_datetime(job, "heartbeat_at").unwrap_or(0.0), - }) - }) - .collect() + docs.iter().map(restore_job_from_doc).collect() + }) + } + + fn claim_restore<'a>( + &'a self, + account_id: &'a str, + input: CreateTableInput, + registration: &'a RestoreRegistrationRow, + ) -> BoxFuture<'a, Result> { + Box::pin(self.claim_restore_tx(account_id, input, registration)) + } + + fn claim_restore_dead<'a>( + &'a self, + key: &'a RestoreTargetKey, + stale_after: Duration, + spool_host: Option<&'a str>, + ) -> BoxFuture<'a, Result>, StorageError>> { + Box::pin(async move { + // The heartbeat is the liveness test, as for a backup row: a + // RUNNING manifest record whose heartbeat is older than the window + // on the server's clock has no owner, and the one write below + // tests that and takes the record under a fresh generation with + // the claimant's spool and `heartbeat_at` at the server's clock, + // so a second taker (another instance's reconciler in the period + // before the successor's first beat) finds it live; the trait doc + // says why a restore record is refreshed where a backup row is + // not. A legacy record is never taken over (the reconciler fails + // it), and a record that is not RUNNING is its owner's or + // finished. + let generation = new_generation(); + let mut filter = target_filter(key); + filter.insert(RESTORE_JOB_STATUS, JobStatus::Running.as_str()); + filter.insert(RESTORE_JOB_KIND, RestoreKind::Manifest.as_str()); + filter.insert("$expr", age_is_stale(&[RESTORE_JOB_HEARTBEAT], stale_after)); + let mut update = vec![doc! { "$set": literal_set(doc! { + RESTORE_JOB_GENERATION: generation_field(generation), + RESTORE_JOB_HEARTBEAT: "$$NOW", + }) }]; + match spool_host { + Some(host) => update.push(doc! { "$set": literal_set(doc! { + RESTORE_JOB_SPOOL_HOST: host, + }) }), + None => update.push(doc! { "$unset": RESTORE_JOB_SPOOL_HOST }), + } + let taken = self + .tables() + .find_one_and_update(filter, update) + .await + .map_err(internal)?; + Ok(taken.map(|_| { + Box::new(MongoRestoreClaim { + tables: self.tables(), + key: key.clone(), + generation, + }) as Box + })) + }) + } + + fn read_restore_job<'a>( + &'a self, + key: &'a RestoreTargetKey, + ) -> BoxFuture<'a, Result, StorageError>> { + Box::pin(async move { + let mut filter = target_filter(key); + filter.insert(RESTORE_JOB, doc! { "$exists": true }); + let doc = self.tables().find_one(filter).await.map_err(internal)?; + doc.as_ref().map(restore_job_from_doc).transpose() + }) + } + + fn list_restore_jobs<'a>( + &'a self, + status: Option, + ) -> BoxFuture<'a, Result, StorageError>> { + Box::pin(async move { + let mut filter = doc! { RESTORE_JOB: { "$exists": true } }; + if let Some(status) = status { + filter.insert(RESTORE_JOB_STATUS, status.as_str()); + } + let docs: Vec = self + .tables() + .find(filter) + .sort(doc! { "_id": 1 }) + .await + .map_err(internal)? + .try_collect() + .await + .map_err(internal)?; + docs.iter().map(restore_job_from_doc).collect() }) } diff --git a/crates/storage-mongodb/src/backup_engine.rs b/crates/storage-mongodb/src/backup_engine.rs index d4561e1f7..4049df81f 100644 --- a/crates/storage-mongodb/src/backup_engine.rs +++ b/crates/storage-mongodb/src/backup_engine.rs @@ -97,7 +97,7 @@ use extenddb_core::types::{ }; use extenddb_storage::backup_catalog::{ AbandonedLegacyBackup, BackupCatalog, JOB_HEARTBEAT_INTERVAL, JobStatus, RestoreKind, - RestoreTargetKey, new_generation, + RestoreTargetKey, StoreIdentity, new_generation, }; use extenddb_storage::backup_definition::{BACKUP_DEFINITION_VERSION, BackupTableDefinition}; use extenddb_storage::error::StorageError; @@ -362,6 +362,19 @@ pub(crate) fn age_is_stale(fields: &[&str], stale_after: Duration) -> Document { } } +/// How the job of a registration reads its backup, and what the record +/// carries for it beyond the members every record has. +pub(crate) enum RestoreRecord<'a> { + /// The in-catalog backup: `kind = legacy`, nothing more. + Legacy, + /// A backup in the store: `kind = manifest`, the store the backup row + /// names and the claimant's spool. + Manifest { + store: &'a StoreIdentity, + spool_host: Option<&'a str>, + }, +} + /// What a restore's registration writes into the target's first catalog /// document: the provenance `DescribeTable` reports and the job record. pub(crate) struct RestoreRegistration<'a> { @@ -372,8 +385,11 @@ pub(crate) struct RestoreRegistration<'a> { pub(crate) owner_generation: u64, /// The record's first heartbeat. An insert cannot use `$$NOW`, so this is /// the inserting process's clock; the copy task's first refresh replaces - /// it with the server's as soon as the task starts. + /// it with the server's as soon as the task starts. A manifest record + /// has it rewritten with the server's clock inside the registration's + /// transaction ([`Self::stamp_server_clock`]). pub(crate) first_heartbeat: bson::DateTime, + pub(crate) record: RestoreRecord<'a>, /// The session whose transaction the target's insert runs in and which /// the insert commits; `None` inserts and commits nothing beyond the /// document itself. @@ -385,15 +401,56 @@ impl RestoreRegistration<'_> { table_doc.insert("restore_source_backup_arn", self.backup_arn); table_doc.insert("restore_source_table_arn", self.source_table_arn); table_doc.insert("restore_date_time", self.restore_date_time); - table_doc.insert( - RESTORE_JOB, - doc! { - "status": JobStatus::Running.as_str(), - "kind": RestoreKind::Legacy.as_str(), - "owner_generation": generation_field(self.owner_generation), - "heartbeat_at": self.first_heartbeat, - }, - ); + let kind = match self.record { + RestoreRecord::Legacy => RestoreKind::Legacy, + RestoreRecord::Manifest { .. } => RestoreKind::Manifest, + }; + let mut job = doc! { + "status": JobStatus::Running.as_str(), + "kind": kind.as_str(), + "owner_generation": generation_field(self.owner_generation), + "heartbeat_at": self.first_heartbeat, + }; + if let RestoreRecord::Manifest { store, spool_host } = &self.record { + job.insert("store_kind", store.store_kind.as_str()); + job.insert("store_root", &store.store_root); + job.insert("store_id", &store.store_id); + if let Some(spool_host) = spool_host { + job.insert("spool_host", *spool_host); + } + } + table_doc.insert(RESTORE_JOB, job); + } + + /// Rewrite the record's first heartbeat with the server's clock, inside + /// the registration's transaction: a manifest record is heartbeated by + /// the engine's timer, whose first beat comes one period after the + /// registration, so the record must not start on the inserting + /// process's clock. The driver's error is returned as it is, so a write + /// conflict keeps its transient label for the caller's retry. Nothing + /// for a legacy record, whose copy task beats at once. + pub(crate) async fn stamp_server_clock( + &mut self, + tables_coll: &mongodb::Collection, + table_id: &str, + ) -> Result<(), mongodb::error::Error> { + if !matches!(self.record, RestoreRecord::Manifest { .. }) { + return Ok(()); + } + let Some(session) = self.session.as_deref_mut() else { + return Ok(()); + }; + tables_coll + .update_one( + doc! { + "table_id": table_id, + RESTORE_JOB_GENERATION: generation_field(self.owner_generation), + }, + vec![doc! { "$set": { RESTORE_JOB_HEARTBEAT: "$$NOW" } }], + ) + .session(session) + .await + .map(|_| ()) } } @@ -456,7 +513,7 @@ impl MongoEngine { /// This process's clock, for the one heartbeat that cannot use the /// server's (the target insert). Test builds can set it behind. - fn local_now(&self) -> bson::DateTime { + pub(crate) fn local_now(&self) -> bson::DateTime { let now = bson::DateTime::now(); #[cfg(test)] let now = bson::DateTime::from_millis( @@ -589,6 +646,7 @@ impl MongoEngine { restore_date_time: epoch_secs_datetime(*restore_date_time), owner_generation, first_heartbeat: self.local_now(), + record: RestoreRecord::Legacy, session: Some(&mut session), }; match self @@ -659,7 +717,7 @@ impl MongoEngine { /// Remove the CREATING restore target `target_table_name` registered /// under `owner_generation`, with its collections; `Ok(false)` when there /// is none. - async fn remove_target_of_generation( + pub(crate) async fn remove_target_of_generation( &self, account_id: &str, target_table_name: &str, @@ -775,6 +833,52 @@ impl MongoEngine { Ok(result.matched_count == 1) } + /// The end of a restore's load (the design's 5.3 step 6 on this + /// backend): the job record DONE and the target handed to the GSI + /// backfill worker (`restore_backfill_pending`), with `item_count` and, + /// when given, `table_size_bytes`, in one write conditioned on the + /// generation, on the record RUNNING and on the table CREATING. From + /// then on the worker owns the target: it promotes each index and flips + /// the table ACTIVE, conditioned on the flag and on CREATING, and the + /// reconciler leaves a DONE record alone. `Ok(false)` when the record is + /// not RUNNING under `owner_generation` or the table is not CREATING. + pub(crate) async fn hand_over_to_backfill( + &self, + target: &RestoreTargetKey, + owner_generation: u64, + item_count: u64, + size_bytes: Option, + ) -> Result { + let mut set = doc! { + "item_count": { "$literal": i64::try_from(item_count).unwrap_or(i64::MAX) }, + "restore_backfill_pending": { "$literal": true }, + RESTORE_JOB_STATUS: { "$literal": JobStatus::Done.as_str() }, + RESTORE_JOB_COMPLETED_AT: "$$NOW", + }; + if let Some(size_bytes) = size_bytes { + set.insert( + "table_size_bytes", + doc! { "$literal": i64::try_from(size_bytes).unwrap_or(i64::MAX) }, + ); + } + let handed_over = self + .catalog_db + .collection::("tables") + .update_one( + doc! { + "_id": { "account_id": &target.account_id, "table_name": &target.table_name }, + "table_id": &target.table_id, + "table_status": "CREATING", + RESTORE_JOB_STATUS: JobStatus::Running.as_str(), + RESTORE_JOB_GENERATION: generation_field(owner_generation), + }, + vec![doc! { "$set": set }], + ) + .await + .map_err(internal)?; + Ok(handed_over.matched_count == 1) + } + /// The detached copy of a restore: copy the backup's items into the /// target with `$out` while the job record is heartbeated, then hand the /// target to the backfill worker, marking the record DONE in the same @@ -833,26 +937,18 @@ impl MongoEngine { // the flag and on CREATING, and the reconciler leaves a DONE // record alone. let handed_over = self - .catalog_db - .collection::("tables") - .update_one( - doc! { - "_id": { "account_id": account_id, "table_name": target_table_name }, - "table_id": table_id, - "table_status": "CREATING", - RESTORE_JOB_STATUS: JobStatus::Running.as_str(), - RESTORE_JOB_GENERATION: generation_field(owner_generation), + .hand_over_to_backfill( + &RestoreTargetKey { + account_id: account_id.to_owned(), + table_name: target_table_name.to_owned(), + table_id: table_id.to_owned(), }, - vec![doc! { "$set": { - "item_count": { "$literal": i64::try_from(item_count).unwrap_or(i64::MAX) }, - "restore_backfill_pending": { "$literal": true }, - RESTORE_JOB_STATUS: { "$literal": JobStatus::Done.as_str() }, - RESTORE_JOB_COMPLETED_AT: "$$NOW", - } }], + owner_generation, + item_count, + None, ) - .await - .map_err(internal)?; - Ok::<_, StorageError>((item_count, handed_over.matched_count == 1)) + .await?; + Ok::<_, StorageError>((item_count, handed_over)) }; let heartbeat = self.restore_heartbeat_loop(account_id, target_table_name, table_id, owner_generation); @@ -1921,8 +2017,9 @@ mod tests { #[cfg(test)] mod definition_tests { use super::{ - LockMiss, RestoreRegistration, backup_definition_from_doc, description_from_doc, - gone_statuses, lock_miss, restore_source_from_doc, restoring_target_filter, + LockMiss, RestoreRecord, RestoreRegistration, backup_definition_from_doc, + description_from_doc, gone_statuses, lock_miss, restore_source_from_doc, + restoring_target_filter, }; use extenddb_core::types::{ OnDemandThroughput, ProvisionedThroughput, StreamViewType, TimeToLiveStatus, @@ -2166,6 +2263,7 @@ mod definition_tests { restore_date_time: bson::DateTime::from_millis(1_791_976_431_438), owner_generation: 7, first_heartbeat: bson::DateTime::from_millis(1_791_976_500_000), + record: RestoreRecord::Legacy, session: None, }; let mut table_doc = doc! { "table_status": "CREATING" }; @@ -2193,6 +2291,71 @@ mod definition_tests { assert!(!table_doc.contains_key("restore_operation_id")); assert!(!table_doc.contains_key("restore_heartbeat_at")); } + + /// The manifest arm writes `kind = manifest`, the store the backup row + /// names and the claimant's spool; a filesystem store leaves + /// `spool_host` out. + #[test] + fn a_manifest_registration_writes_the_kind_store_and_spool() { + use extenddb_storage::backup_catalog::{StoreIdentity, StoreKind}; + let store = StoreIdentity { + store_kind: StoreKind::S3, + store_root: "s3://bucket/prefix".to_owned(), + store_id: "store-7".to_owned(), + }; + let registration = RestoreRegistration { + backup_arn: ARN, + source_table_arn: "arn:aws:dynamodb:us-east-1:123456789012:table/src", + restore_date_time: bson::DateTime::from_millis(1_791_976_431_438), + owner_generation: 9, + first_heartbeat: bson::DateTime::from_millis(1_791_976_500_000), + record: RestoreRecord::Manifest { + store: &store, + spool_host: Some("host-b:/var/spool"), + }, + session: None, + }; + let mut table_doc = doc! { "table_status": "CREATING" }; + registration.insert_into(&mut table_doc); + let job = table_doc.get_document("restore_job").expect("job record"); + assert_eq!(job.get_str("status"), Ok("RUNNING")); + assert_eq!(job.get_str("kind"), Ok("manifest")); + assert_eq!(job.get_str("store_kind"), Ok(StoreKind::S3.as_str())); + assert_eq!(job.get_str("store_root"), Ok("s3://bucket/prefix")); + assert_eq!(job.get_str("store_id"), Ok("store-7")); + assert_eq!(job.get_str("spool_host"), Ok("host-b:/var/spool")); + + // A filesystem store spools nothing. + let fs = StoreIdentity { + store_kind: StoreKind::Filesystem, + store_root: "/var/lib/extenddb/backups".to_owned(), + store_id: "store-1".to_owned(), + }; + let registration = RestoreRegistration { + backup_arn: ARN, + source_table_arn: "x", + restore_date_time: bson::DateTime::from_millis(0), + owner_generation: 9, + first_heartbeat: bson::DateTime::from_millis(0), + record: RestoreRecord::Manifest { + store: &fs, + spool_host: None, + }, + session: None, + }; + let mut table_doc = doc! { "table_status": "CREATING" }; + registration.insert_into(&mut table_doc); + let job = table_doc.get_document("restore_job").expect("job record"); + assert_eq!(job.get_str("kind"), Ok("manifest")); + assert_eq!( + job.get_str("store_kind"), + Ok(StoreKind::Filesystem.as_str()) + ); + assert!( + job.get_str("spool_host").is_err(), + "no spool for a filesystem store" + ); + } } /// The protocol model; see the module documentation. diff --git a/crates/storage-mongodb/src/backup_live_tests.rs b/crates/storage-mongodb/src/backup_live_tests.rs index e139eb513..bca94e766 100644 --- a/crates/storage-mongodb/src/backup_live_tests.rs +++ b/crates/storage-mongodb/src/backup_live_tests.rs @@ -30,23 +30,28 @@ use extenddb_core::types::{ ProjectionType, ScalarAttributeType, TableStatus, }; use extenddb_storage::backup_catalog::{ - BackupRow, BackupStatus, DeleteStart, Heartbeat, JobStatus, RestoreKind, RestoreTargetKey, - StaleClock, StatusFields, StoreIdentity, StoreKind, TimeBounds, UNOWNED_GENERATION, - WriteOutcome, + BackupRow, BackupStatus, DeleteStart, Heartbeat, JobStatus, RestoreKind, + RestoreRegistrationRow, RestoreTargetKey, StaleClock, StatusFields, StoreIdentity, StoreKind, + TimeBounds, UNOWNED_GENERATION, WriteOutcome, }; use extenddb_storage::error::StorageError; +use extenddb_storage::restore_engine::{BulkLoadFile, Finished, RestoreEngine}; use extenddb_storage::{ BackupCatalog, JobPermit, LegacyBackupReader, RestoreRequest, RestoreTableOverrides, TableEngine, }; -use futures::{FutureExt, TryStreamExt}; +use futures::stream::BoxStream; +use futures::{FutureExt, StreamExt, TryStreamExt}; use mongodb::bson::{self, Document, doc}; use super::{AfterBackupDrop, BackupLiveness, HeartbeatFault}; use crate::MongoEngine; -use crate::backup_catalog::{BACKUP_INDEX, RESTORE_JOB, RESTORE_JOB_HEARTBEAT, RESTORE_JOB_STATUS}; +use crate::backup_catalog::{ + BACKUP_INDEX, RESTORE_JOB, RESTORE_JOB_GENERATION, RESTORE_JOB_HEARTBEAT, RESTORE_JOB_STATUS, +}; use crate::bootstrapper::{CATALOG_VERSION, apply_backup_catalog_index, require_served_catalog}; -use crate::data::{data_collection_name, item_to_document}; +use crate::data::{data_collection_name, document_to_item, item_to_document}; +use crate::restore_engine::INSERT_MANY_DOCUMENTS; const ACCOUNT: &str = "123456789012"; const OTHER_ACCOUNT: &str = "210987654321"; @@ -412,6 +417,25 @@ async fn registered_targets(engine: &MongoEngine, targets: &[String]) -> Vec bson::DateTime { + let mut cursor = engine + .catalog_db + .collection::("tables") + .aggregate(vec![ + doc! { "$limit": 1 }, + doc! { "$project": { "_id": 0, "now": "$$NOW" } }, + ]) + .await + .expect("aggregate"); + let doc = cursor + .next() + .await + .expect("one document") + .expect("the server's clock"); + *doc.get_datetime("now").expect("now") +} + async fn job_record(engine: &MongoEngine, name: &str) -> Document { table_doc(engine, name) .await @@ -3294,3 +3318,1084 @@ async fn live_update_table_on_a_creating_restore_target_is_refused() { assert!(desc.deletion_protection_enabled); fx.finish().await; } + +// ───────────────────────────────────────────────────────────────────────── +// Store-backed restore: claim_restore, the claim, claim_restore_dead, +// read/list_restore_jobs, bulk_load, finish_restore, count_restore_target. +// The registration takes the `backup_index` row's lock (as the legacy path +// takes the `backups` row's), writes the record `kind = manifest` with the +// store and the claimant's spool, and the record's heartbeat is the server's +// clock from the registration on; bulk_load writes base documents only, the +// GSI backfill worker fills the indexes and flips the table ACTIVE. +// ───────────────────────────────────────────────────────────────────────── + +/// A composite-key input with a GSI (ALL) and an LSI (KEYS_ONLY), the shape +/// `source_table_for` creates, for a restore target. +fn manifest_input(name: &str) -> CreateTableInput { + CreateTableInput { + table_name: name.to_owned(), + key_schema: vec![key("pk", KeyType::Hash), key("sk", KeyType::Range)], + attribute_definitions: vec![ + attr("pk", ScalarAttributeType::S), + attr("sk", ScalarAttributeType::N), + attr("gpk", ScalarAttributeType::S), + attr("lsk", ScalarAttributeType::N), + ], + billing_mode: Some(BillingMode::PayPerRequest), + global_secondary_indexes: Some(vec![GsiInput { + index_name: "g1".to_owned(), + key_schema: vec![key("gpk", KeyType::Hash)], + projection: Projection { + projection_type: ProjectionType::All, + non_key_attributes: None, + }, + provisioned_throughput: None, + }]), + local_secondary_indexes: Some(vec![LsiInput { + index_name: "l1".to_owned(), + key_schema: vec![key("pk", KeyType::Hash), key("lsk", KeyType::Range)], + projection: Projection { + projection_type: ProjectionType::KeysOnly, + non_key_attributes: None, + }, + }]), + ..Default::default() + } +} + +/// A `backup_index` row of a backup of `table`, left AVAILABLE: what a +/// store-backed restore registers against. Returns the row (its `key()` is +/// the registration's backup key). +async fn available_backup(engine: &MongoEngine, table: &str, name: &str) -> BackupRow { + let row = row(ACCOUNT, table, name); + let claim = engine.claim_backup(&row).await.expect("claim backup row"); + assert_eq!( + claim + .set_status( + JobStatus::Creating, + JobStatus::Available, + &StatusFields::default() + ) + .await + .expect("to AVAILABLE"), + WriteOutcome::Matched + ); + drop(claim); + row +} + +/// The registration row for `row`, with the claimant's `spool_host` +/// (`None` for a filesystem store). `owner_generation` is unowned: the +/// backend mints its own. +fn registration(row: &BackupRow, spool: Option<&str>) -> RestoreRegistrationRow { + RestoreRegistrationRow { + backup: row.key(), + source_table_arn: format!( + "arn:aws:dynamodb:{REGION}:{ACCOUNT}:table/{}", + row.table_name + ), + restore_date_time: row.created_at, + store: store(), + spool_host: spool.map(str::to_owned), + owner_generation: UNOWNED_GENERATION, + } +} + +/// Item `i` of a restore: every attribute the source shape carries, both +/// index keys, and a 38-digit non-key number that survives as a string. +fn manifest_item(i: usize) -> Item { + [ + ("pk".to_owned(), AttributeValue::S(format!("p{}", i % 50))), + ("sk".to_owned(), AttributeValue::N(i.to_string())), + ("gpk".to_owned(), AttributeValue::S(format!("g{}", i % 7))), + ("lsk".to_owned(), AttributeValue::N((i * 3).to_string())), + ( + "big".to_owned(), + AttributeValue::N("12345678901234567890123456789012345678".to_owned()), + ), + ] + .into_iter() + .collect() +} + +/// The items `lo..hi` as the stream `bulk_load` reads. +fn item_stream(lo: usize, hi: usize) -> BoxStream<'static, Result> { + let items: Vec<_> = (lo..hi).map(manifest_item).collect(); + futures::stream::iter(items.into_iter().map(Ok)).boxed() +} + +/// The registration writes the target CREATING with its `RestoreSummary` +/// and a RUNNING `kind = manifest` record carrying the store, the +/// claimant's `spool_host`, a fresh generation and the server's clock on +/// its heartbeat (the registering process's clock is an hour behind, and +/// the stored heartbeat is within seconds of the server's `$$NOW`); the +/// claim heartbeats and reads the record; `read_restore_job` and +/// `list_restore_jobs` fill the store and spool; `backup_in_use` names the +/// target. +#[tokio::test(flavor = "multi_thread")] +#[ignore = "needs a MongoDB replica set in EXTENDDB_TEST_MONGODB_URI"] +async fn live_claim_restore_registers_the_target_and_its_record() { + let fx = Fixture::new().await; + fx.worker.abort(); + let backup = available_backup(&fx.engine, "src", "b").await; + let reg = registration(&backup, Some("host-a:/var/spool/x")); + let behind = fx + .second_instance(BackupLiveness { + local_clock_behind: Duration::from_secs(3_600), + ..test_liveness() + }) + .await; + let (desc, claim) = behind + .claim_restore(ACCOUNT, manifest_input("dst"), ®) + .await + .expect("register"); + assert_eq!(desc.table_status, TableStatus::Creating); + let summary = desc.restore_summary.as_ref().expect("a RestoreSummary"); + assert_eq!( + summary.source_backup_arn.as_deref(), + Some(backup.backup_arn.as_str()) + ); + assert!(summary.restore_in_progress); + let generation = claim.generation(); + assert_ne!(generation, UNOWNED_GENERATION); + + let doc = table_doc(&fx.engine, "dst").await.expect("target"); + let job = doc.get_document(RESTORE_JOB).expect("job record"); + assert_eq!(job.get_str("status"), Ok("RUNNING")); + assert_eq!(job.get_str("kind"), Ok("manifest")); + assert_eq!( + u64::try_from(job.get_i64("owner_generation").unwrap()).unwrap(), + generation + ); + assert_eq!( + job.get_str("store_kind"), + Ok(StoreKind::Filesystem.as_str()) + ); + assert_eq!(job.get_str("store_root"), Ok(store().store_root.as_str())); + assert_eq!(job.get_str("store_id"), Ok(store().store_id.as_str())); + assert_eq!(job.get_str("spool_host"), Ok("host-a:/var/spool/x")); + // The heartbeat is the server's clock stamped inside the registration, + // not the inserting process's (an hour behind): within seconds of the + // server's `$$NOW`, read back through an aggregation. + let heartbeat = *job.get_datetime("heartbeat_at").expect("heartbeat"); + let now = server_now(&fx.engine).await; + let age_ms = now.timestamp_millis() - heartbeat.timestamp_millis(); + assert!( + (-2_000..=5_000).contains(&age_ms), + "the record's heartbeat is the server's clock, not the slow process's: {heartbeat:?} \ + is {age_ms} ms behind {now:?}" + ); + + let key = target_key(&doc); + let read = fx + .engine + .read_restore_job(&key) + .await + .expect("read") + .expect("a record"); + assert_eq!(read.kind, RestoreKind::Manifest); + assert_eq!(read.status, JobStatus::Running); + assert_eq!(read.owner_generation, generation); + assert_eq!( + read.store.as_ref().map(|s| s.store_kind), + Some(StoreKind::Filesystem) + ); + assert_eq!(read.spool_host.as_deref(), Some("host-a:/var/spool/x")); + assert_eq!(read.file_index, None); + + let listed = fx + .engine + .list_restore_jobs(Some(JobStatus::Running)) + .await + .expect("list"); + assert_eq!(listed.len(), 1); + assert_eq!(listed[0].target, key); + assert!(listed[0].store.is_some()); + assert_eq!(listed[0].spool_host.as_deref(), Some("host-a:/var/spool/x")); + + assert_eq!( + fx.engine + .backup_in_use(&backup.key()) + .await + .unwrap() + .as_deref(), + Some("dst") + ); + assert_eq!(claim.heartbeat().await.unwrap(), Heartbeat::Confirmed); + drop(claim); + fx.finish().await; +} + +/// A DeleteBackup and a store-backed registration of one backup order wholly +/// one before the other, on the `backup_index` row's lock: a delete that +/// wins leaves the registration `Backup not found` and no target; a +/// registration that wins makes the delete `BackupInUse`. A loop of +/// concurrent races holds the invariant that no target is ever registered +/// against a row that is not AVAILABLE. +#[tokio::test(flavor = "multi_thread")] +#[ignore = "needs a MongoDB replica set in EXTENDDB_TEST_MONGODB_URI"] +async fn live_the_two_orderings_against_delete_backup_on_the_backup_index_row() { + let fx = Fixture::new().await; + fx.worker.abort(); + + // Order A: the delete wins. + let a = available_backup(&fx.engine, "srcA", "ba").await; + assert_eq!( + fx.engine.start_backup_delete(&a.key()).await.unwrap(), + DeleteStart::Deleting + ); + let refused = fx + .engine + .claim_restore(ACCOUNT, manifest_input("dstA"), ®istration(&a, None)) + .await + .err(); + assert!( + matches!(&refused, Some(StorageError::Validation(m)) if m.contains("Backup not found")), + "the registration finds the backup gone: {refused:?}" + ); + assert!(table_doc(&fx.engine, "dstA").await.is_none()); + + // Order B: the registration wins. + let b = available_backup(&fx.engine, "srcB", "bb").await; + let (_, claim) = fx + .engine + .claim_restore(ACCOUNT, manifest_input("dstB"), ®istration(&b, None)) + .await + .expect("register"); + assert_eq!( + fx.engine.start_backup_delete(&b.key()).await.unwrap(), + DeleteStart::InUse("dstB".to_owned()) + ); + assert_eq!( + fx.engine + .read_backup_row(&arn_of(&b)) + .await + .unwrap() + .unwrap() + .status, + BackupStatus::Available, + "a refused delete leaves the row AVAILABLE" + ); + drop(claim); + + // Concurrent races: the lock keeps the two consistent every time, and + // both outcomes occur across the loop (no frozen answer). + let mut delete_won = 0; + let mut restore_won = 0; + for i in 0..16 { + let backup = available_backup(&fx.engine, &format!("srcC{i}"), "bc").await; + let name = format!("dstC{i}"); + let e1 = fx.engine.clone(); + let e2 = fx.engine.clone(); + let key = backup.key(); + let reg = registration(&backup, None); + let del = tokio::spawn(async move { e1.start_backup_delete(&key).await }); + let res = + tokio::spawn( + async move { e2.claim_restore(ACCOUNT, manifest_input(&name), ®).await }, + ); + let (del, res) = (del.await.unwrap().unwrap(), res.await.unwrap()); + let res_ok = res.is_ok(); + let res_err = res.err(); + let target = table_doc(&fx.engine, &format!("dstC{i}")).await; + let status = fx + .engine + .read_backup_row(&arn_of(&backup)) + .await + .unwrap() + .expect("the backup_index row") + .status; + if target.is_some() { + assert_eq!( + status, + BackupStatus::Available, + "a target was registered, so the delete must have lost" + ); + assert!( + matches!(del, DeleteStart::InUse(_)), + "iteration {i}: {del:?}" + ); + assert!(res_ok, "iteration {i}"); + restore_won += 1; + } else { + assert!( + matches!(&res_err, Some(StorageError::Validation(m)) if m.contains("Backup not found")), + "iteration {i}: no target but the registration did not refuse: {res_err:?}" + ); + assert_eq!(status, BackupStatus::Deleting, "iteration {i}"); + delete_won += 1; + } + } + assert!( + delete_won > 0 && restore_won > 0, + "both orders occur: {delete_won} / {restore_won}" + ); + fx.finish().await; +} + +/// Two concurrent registrations of one backup leave one CREATING target and +/// answer `BackupInUse` to the other (the in-use rule under the lock). +#[tokio::test(flavor = "multi_thread")] +#[ignore = "needs a MongoDB replica set in EXTENDDB_TEST_MONGODB_URI"] +async fn live_two_concurrent_registrations_of_one_backup_leave_one_target() { + let fx = Fixture::new().await; + fx.worker.abort(); + let backup = available_backup(&fx.engine, "src", "b").await; + let e1 = fx.engine.clone(); + let e2 = fx.engine.clone(); + let r1 = registration(&backup, None); + let r2 = registration(&backup, None); + let t1 = + tokio::spawn(async move { e1.claim_restore(ACCOUNT, manifest_input("r1"), &r1).await }); + let t2 = + tokio::spawn(async move { e2.claim_restore(ACCOUNT, manifest_input("r2"), &r2).await }); + let o1 = t1.await.unwrap(); + let o2 = t2.await.unwrap(); + let oks = [&o1, &o2].iter().filter(|o| o.is_ok()).count(); + let in_use = [&o1, &o2] + .iter() + .filter(|o| matches!(o, Err(StorageError::BackupInUse(_)))) + .count(); + let errs: Vec<_> = [&o1, &o2] + .iter() + .filter_map(|o| o.as_ref().err().map(ToString::to_string)) + .collect(); + assert_eq!( + (oks, in_use), + (1, 1), + "one target, one BackupInUse: oks={oks}, errs={errs:?}" + ); + let targets = registered_targets(&fx.engine, &["r1".to_owned(), "r2".to_owned()]).await; + assert_eq!( + targets.len(), + 1, + "exactly one target registered: {targets:?}" + ); + for (_, claim) in [o1, o2].into_iter().flatten() { + drop(claim); + } + fx.finish().await; +} + +/// A name a table or a CREATING restore target already holds is refused as +/// `TableAlreadyExists` or `TableBeingRestored`, with nothing registered. +#[tokio::test(flavor = "multi_thread")] +#[ignore = "needs a MongoDB replica set in EXTENDDB_TEST_MONGODB_URI"] +async fn live_claim_restore_refuses_a_taken_name() { + let fx = Fixture::new().await; + fx.worker.abort(); + // A plain ACTIVE table. + fx.engine + .create_table_impl(ACCOUNT, manifest_input("taken"), false) + .await + .expect("create a table"); + let backup = available_backup(&fx.engine, "src", "b").await; + let exists = fx + .engine + .claim_restore( + ACCOUNT, + manifest_input("taken"), + ®istration(&backup, None), + ) + .await + .err(); + assert!( + matches!(&exists, Some(StorageError::TableAlreadyExists(n)) if n == "taken"), + "{exists:?}" + ); + + // A CREATING restore target. + let (_, claim) = fx + .engine + .claim_restore( + ACCOUNT, + manifest_input("being"), + ®istration(&backup, None), + ) + .await + .expect("first registration"); + let b2 = available_backup(&fx.engine, "src2", "b2").await; + let restoring = fx + .engine + .claim_restore(ACCOUNT, manifest_input("being"), ®istration(&b2, None)) + .await + .err(); + assert!( + matches!( + &restoring, + Some(StorageError::TableBeingRestored { table_name, backup_arn }) + if table_name == "being" && backup_arn == &backup.backup_arn + ), + "{restoring:?}" + ); + drop(claim); + fx.finish().await; +} + +/// `claim_restore_dead` takes over a RUNNING manifest record whose heartbeat +/// is stale: a fresh generation, the claimant's `spool_host` and +/// `heartbeat_at` refreshed to the server's clock in one write, so a second +/// taker issued right after the first (before the successor's first beat) +/// finds the record live and is refused, and the first taker's beat +/// confirms; the old generation's writes are fenced out. A live record, and +/// a legacy record whatever its age, are left alone. +#[tokio::test(flavor = "multi_thread")] +#[ignore = "needs a MongoDB replica set in EXTENDDB_TEST_MONGODB_URI"] +async fn live_claim_restore_dead_takes_over_a_stale_manifest_record() { + let fx = Fixture::new().await; + fx.worker.abort(); + let backup = available_backup(&fx.engine, "src", "b").await; + let (doc, claim) = fx + .engine + .claim_restore( + ACCOUNT, + manifest_input("t"), + ®istration(&backup, Some("old:/s")), + ) + .await + .expect("register"); + let desc_doc = table_doc(&fx.engine, "t").await.expect("target"); + let key = target_key(&desc_doc); + let g1 = claim.generation(); + let _ = doc; + + // Live: fresh heartbeat, so not dead. + assert!( + fx.engine + .claim_restore_dead(&key, STALE_AFTER, Some("new:/s2")) + .await + .unwrap() + .is_none(), + "a live record is not taken over" + ); + + // Age it past the window and take it over. + age_running_record(&fx.engine, "t", Duration::from_secs(600)).await; + let aged = *job_record(&fx.engine, "t") + .await + .get_datetime("heartbeat_at") + .expect("heartbeat"); + let taken = fx + .engine + .claim_restore_dead(&key, STALE_AFTER, Some("new:/s2")) + .await + .expect("takeover") + .expect("a stale record is taken over"); + let g2 = taken.generation(); + assert_ne!(g2, g1, "a fresh generation"); + let after = job_record(&fx.engine, "t").await; + assert_eq!( + u64::try_from(after.get_i64("owner_generation").unwrap()).unwrap(), + g2 + ); + assert_eq!( + after.get_str("spool_host"), + Ok("new:/s2"), + "the claimant's spool is written" + ); + // A second taker right after the first, before the successor's first + // beat: the record is live to it, and the first taker's beat confirms. + let second = fx + .engine + .claim_restore_dead(&key, STALE_AFTER, Some("third:/s3")) + .await + .expect("a second takeover attempt"); + assert!( + second.is_none(), + "a second taker right after the first is refused: {:?}", + second.as_ref().map(|c| c.generation()) + ); + assert_eq!( + taken.heartbeat().await.unwrap(), + Heartbeat::Confirmed, + "the first taker keeps the record" + ); + let refreshed = *after.get_datetime("heartbeat_at").expect("heartbeat"); + assert!( + refreshed > aged, + "the takeover refreshed heartbeat_at: {refreshed:?} was {aged:?}" + ); + let now = server_now(&fx.engine).await; + let age_ms = now.timestamp_millis() - refreshed.timestamp_millis(); + assert!( + (-2_000..=5_000).contains(&age_ms), + "the refreshed heartbeat is the server's now: {refreshed:?} is {age_ms} ms behind {now:?}" + ); + assert!( + fx.engine + .list_dead_restore_jobs(STALE_AFTER) + .await + .unwrap() + .is_empty(), + "the taken-over record is live to the listing at once" + ); + + // The old generation is fenced out; the new one owns the record. + assert_eq!( + claim + .set_status( + JobStatus::Running, + JobStatus::Failed, + &StatusFields::default() + ) + .await + .unwrap(), + WriteOutcome::NotMatched + ); + assert!( + matches!( + fx.engine.finish_restore(&key, &*claim, 0, 0).await.unwrap(), + Finished::NotMatched(_) + ), + "the dead generation cannot finish the restore" + ); + drop(claim); + drop(taken); + fx.finish().await; +} + +/// A legacy record, and a record that is not RUNNING, are never taken over. +#[tokio::test(flavor = "multi_thread")] +#[ignore = "needs a MongoDB replica set in EXTENDDB_TEST_MONGODB_URI"] +async fn live_claim_restore_dead_leaves_a_legacy_or_finished_record() { + let fx = Fixture::new().await; + fx.worker.abort(); + let backup = available_backup(&fx.engine, "src", "b").await; + let (doc, claim) = fx + .engine + .claim_restore(ACCOUNT, manifest_input("leg"), ®istration(&backup, None)) + .await + .expect("register"); + let key = target_key(&table_doc(&fx.engine, "leg").await.expect("target")); + let _ = doc; + // Make the record legacy and stale. + fx.engine + .catalog_db + .collection::("tables") + .update_one( + doc! { "_id": { "account_id": ACCOUNT, "table_name": "leg" } }, + doc! { "$set": { "restore_job.kind": "legacy" } }, + ) + .await + .expect("to legacy"); + age_running_record(&fx.engine, "leg", Duration::from_secs(600)).await; + // age_running_record rewrites the record RUNNING; set kind legacy again. + fx.engine + .catalog_db + .collection::("tables") + .update_one( + doc! { "_id": { "account_id": ACCOUNT, "table_name": "leg" } }, + doc! { "$set": { "restore_job.kind": "legacy" } }, + ) + .await + .expect("to legacy"); + assert!( + fx.engine + .claim_restore_dead(&key, STALE_AFTER, Some("new:/s")) + .await + .unwrap() + .is_none(), + "a legacy record is not taken over" + ); + drop(claim); + fx.finish().await; +} + +/// `bulk_load` writes a file's base documents in `insert_many` calls and +/// records `file_index` under the claim; a resume re-applies the one +/// unrecorded file with `reapplying`, every duplicate suppressed, and the +/// count equals the file's full set; a duplicate on a file not being +/// re-applied fails the file. +#[tokio::test(flavor = "multi_thread")] +#[ignore = "needs a MongoDB replica set in EXTENDDB_TEST_MONGODB_URI"] +async fn live_bulk_load_records_each_file_and_a_resume_reapplies() { + let fx = Fixture::new().await; + fx.worker.abort(); + let backup = available_backup(&fx.engine, "src", "b").await; + let (doc, claim) = fx + .engine + .claim_restore(ACCOUNT, manifest_input("dst"), ®istration(&backup, None)) + .await + .expect("register"); + let key = target_key(&table_doc(&fx.engine, "dst").await.expect("target")); + let _ = doc; + + // File 0: 2500 items across three insert_many calls. + let applied = fx + .engine + .bulk_load( + &key, + &*claim, + BulkLoadFile { + file_index: 0, + start_at: 0, + reapplying: false, + }, + item_stream(0, 2500), + ) + .await + .expect("file 0"); + assert_eq!(applied, 2500); + assert_eq!(fx.engine.count_restore_target(&key).await.unwrap(), 2500); + assert_eq!( + fx.engine + .read_restore_job(&key) + .await + .unwrap() + .unwrap() + .file_index, + Some(0) + ); + + // Simulate a resume of file 1 whose record never landed: insert the + // first 500 of its items directly, then re-apply the whole file with + // `reapplying` set. + let coll = fx + .engine + .data_db + .collection::(&data_collection_name(&key.table_id)); + let input = manifest_input("dst"); + let half: Vec = (2500..3000) + .map(|i| { + item_to_document( + &manifest_item(i), + &input.key_schema, + &input.attribute_definitions, + ) + .unwrap() + }) + .collect(); + coll.insert_many(half) + .await + .expect("seed a half-applied file"); + let resumed = fx + .engine + .bulk_load( + &key, + &*claim, + BulkLoadFile { + file_index: 1, + start_at: 0, + reapplying: true, + }, + item_stream(2500, 3500), + ) + .await + .expect("resume file 1"); + assert_eq!(resumed, 500, "only the unrecorded half is counted"); + assert_eq!(fx.engine.count_restore_target(&key).await.unwrap(), 3500); + assert_eq!( + fx.engine + .read_restore_job(&key) + .await + .unwrap() + .unwrap() + .file_index, + Some(1) + ); + + // A duplicate key on a file not being re-applied fails the file. + let dup = fx + .engine + .bulk_load( + &key, + &*claim, + BulkLoadFile { + file_index: 1, + start_at: 0, + reapplying: false, + }, + item_stream(2500, 3500), + ) + .await; + assert!( + matches!(&dup, Err(StorageError::Validation(m)) if m.contains("holds a key twice")), + "{dup:?}" + ); + assert_eq!(fx.engine.count_restore_target(&key).await.unwrap(), 3500); + drop(claim); + fx.finish().await; +} + +/// A claim lost between two `insert_many` calls of one file (the record's +/// generation moved by a takeover while the load ran): the read before the +/// next call finds the record no longer this claim's, the file ends as a +/// lost claim, and the second call's documents never land. +#[tokio::test(flavor = "multi_thread")] +#[ignore = "needs a MongoDB replica set in EXTENDDB_TEST_MONGODB_URI"] +async fn live_bulk_load_stops_at_a_lost_claim_between_two_insert_many_calls() { + let fx = Fixture::new().await; + fx.worker.abort(); + let backup = available_backup(&fx.engine, "src", "b").await; + let (_doc, claim) = fx + .engine + .claim_restore( + ACCOUNT, + manifest_input("lost"), + ®istration(&backup, None), + ) + .await + .expect("register"); + let key = target_key(&table_doc(&fx.engine, "lost").await.expect("target")); + // The stream hands the first call's documents, then moves the record's + // generation when asked for the first item of the second call. + let tables = fx.engine.catalog_db.collection::("tables"); + let moved_key = key.clone(); + let stream: BoxStream<'static, Result> = Box::pin(futures::stream::unfold( + 0usize, + move |i| { + let tables = tables.clone(); + let key = moved_key.clone(); + async move { + if i >= 2 * INSERT_MANY_DOCUMENTS + 500 { + return None; + } + if i == INSERT_MANY_DOCUMENTS { + tables + .update_one( + doc! { "_id": { "account_id": &key.account_id, "table_name": &key.table_name } }, + doc! { "$inc": { RESTORE_JOB_GENERATION: 1_i64 } }, + ) + .await + .expect("another owner takes the record"); + } + Some((Ok(manifest_item(i)), i + 1)) + } + }, + )); + let out = fx + .engine + .bulk_load( + &key, + &*claim, + BulkLoadFile { + file_index: 0, + start_at: 0, + reapplying: false, + }, + stream, + ) + .await; + assert!( + matches!(&out, Err(StorageError::Internal(m)) if m.contains("claim on the target was lost")), + "the second call's read finds another owner: {out:?}" + ); + assert_eq!( + count(&fx.engine, &data_collection_name(&key.table_id)).await, + INSERT_MANY_DOCUMENTS as u64, + "the first call's documents alone landed" + ); + assert_eq!( + fx.engine + .read_restore_job(&key) + .await + .unwrap() + .unwrap() + .file_index, + None, + "nothing was recorded" + ); + drop(claim); + fx.finish().await; +} + +/// `bulk_load` refuses an `N` sort key beyond Decimal128's precision with +/// the write path's `ValidationException`, and a 38-digit non-key number +/// survives a restore as a string. +#[tokio::test(flavor = "multi_thread")] +#[ignore = "needs a MongoDB replica set in EXTENDDB_TEST_MONGODB_URI"] +async fn live_bulk_load_refuses_a_long_sort_key_and_keeps_a_long_number() { + let fx = Fixture::new().await; + fx.worker.abort(); + let backup = available_backup(&fx.engine, "src", "b").await; + let (_, claim) = fx + .engine + .claim_restore(ACCOUNT, manifest_input("dec"), ®istration(&backup, None)) + .await + .expect("register"); + let key = target_key(&table_doc(&fx.engine, "dec").await.expect("target")); + + let long_sk: Item = [ + ("pk".to_owned(), AttributeValue::S("p".to_owned())), + // 35 significant digits: beyond Decimal128. + ( + "sk".to_owned(), + AttributeValue::N("12345678901234567890123456789012345".to_owned()), + ), + ("gpk".to_owned(), AttributeValue::S("g".to_owned())), + ("lsk".to_owned(), AttributeValue::N("1".to_owned())), + ] + .into_iter() + .collect(); + let refused = fx + .engine + .bulk_load( + &key, + &*claim, + BulkLoadFile { + file_index: 0, + start_at: 0, + reapplying: false, + }, + futures::stream::iter(vec![Ok(long_sk)]).boxed(), + ) + .await; + assert!( + matches!(&refused, Err(StorageError::Validation(m)) if m.contains("Decimal128")), + "{refused:?}" + ); + + // A good item with a 38-digit non-key number restores intact. + fx.engine + .bulk_load( + &key, + &*claim, + BulkLoadFile { + file_index: 0, + start_at: 0, + reapplying: false, + }, + item_stream(0, 1), + ) + .await + .expect("a good item"); + let stored = fx + .engine + .data_db + .collection::(&data_collection_name(&key.table_id)) + .find_one(doc! {}) + .await + .unwrap() + .expect("the item"); + let item = document_to_item(&stored).expect("item"); + assert_eq!( + item.get("big"), + Some(&AttributeValue::N( + "12345678901234567890123456789012345678".to_owned() + )), + "the 38-digit non-key number survives" + ); + drop(claim); + fx.finish().await; +} + +/// `finish_restore` hands the target to the backfill worker (`Pending`, +/// DONE + `restore_backfill_pending`), the backup stays in use until the +/// worker flips the table ACTIVE, and `list_dead_restore_jobs` leaves the +/// DONE record alone. A stale generation's `finish_restore` is `NotMatched`. +#[tokio::test(flavor = "multi_thread")] +#[ignore = "needs a MongoDB replica set in EXTENDDB_TEST_MONGODB_URI"] +async fn live_finish_restore_hands_over_and_the_worker_flips_active() { + const FEW: usize = 300; + let fx = Fixture::new().await; + fx.worker.abort(); + let backup = available_backup(&fx.engine, "src", "b").await; + let (_, claim) = fx + .engine + .claim_restore(ACCOUNT, manifest_input("fin"), ®istration(&backup, None)) + .await + .expect("register"); + let key = target_key(&table_doc(&fx.engine, "fin").await.expect("target")); + fx.engine + .bulk_load( + &key, + &*claim, + BulkLoadFile { + file_index: 0, + start_at: 0, + reapplying: false, + }, + item_stream(0, FEW), + ) + .await + .expect("load"); + + // A stale generation cannot finish it. + age_running_record(&fx.engine, "fin", Duration::from_secs(600)).await; + let taken = fx + .engine + .claim_restore_dead(&key, STALE_AFTER, None) + .await + .unwrap() + .expect("takeover"); + assert_eq!( + fx.engine + .finish_restore(&key, &*claim, FEW as u64, 1) + .await + .unwrap(), + Finished::NotMatched(format!( + "the job record of fin is not RUNNING under generation {}, or the table is not \ + CREATING with table id {}", + claim.generation(), + key.table_id + )) + ); + + assert_eq!( + fx.engine + .finish_restore(&key, &*taken, FEW as u64, 1) + .await + .unwrap(), + Finished::Pending + ); + let doc = table_doc(&fx.engine, "fin").await.expect("target"); + assert_eq!(doc.get_str("table_status"), Ok("CREATING")); + assert!(doc.get_bool("restore_backfill_pending").unwrap_or(false)); + assert_eq!( + doc.get_document(RESTORE_JOB).unwrap().get_str("status"), + Ok("DONE") + ); + assert_eq!(doc.get_i64("item_count"), Ok(FEW as i64)); + + // Between the DONE write and the worker's flip the backup is in use and + // the DONE record is not dead. + assert_eq!( + fx.engine + .backup_in_use(&backup.key()) + .await + .unwrap() + .as_deref(), + Some("fin") + ); + assert!( + fx.engine + .list_dead_restore_jobs(STALE_AFTER) + .await + .unwrap() + .iter() + .all(|j| j.target.table_name != "fin"), + "the DONE+pending record is not dead" + ); + + // The worker flips it ACTIVE with every index populated. + let worker = spawn_worker(&fx.engine); + let active = wait_for_status(&fx.engine, "fin", "ACTIVE").await; + let table_id = active.get_str("table_id").expect("table_id"); + assert_eq!(active.get_i64("item_count"), Ok(FEW as i64)); + assert_eq!( + count(&fx.engine, &data_collection_name(table_id)).await, + FEW as u64 + ); + for (index_name, index_id, status) in index_ids(&fx.engine, table_id).await { + assert_eq!(status, "ACTIVE", "{index_name}"); + assert_eq!( + count(&fx.engine, &data_collection_name(&index_id)).await, + FEW as u64, + "index {index_name}" + ); + } + let job = active.get_document(RESTORE_JOB).expect("job record"); + assert_eq!(job.get_str("status"), Ok("DONE")); + assert_eq!(job.get_str("kind"), Ok("manifest")); + assert!(!active.contains_key("restore_backfill_pending")); + assert_eq!(fx.engine.backup_in_use(&backup.key()).await.unwrap(), None); + worker.abort(); + drop(claim); + drop(taken); + fx.finish().await; +} + +/// The `restore_mongodb_after_insert_many` kill point and the file-index +/// write's fence: a claim taken over while `bulk_load` is paused between its +/// `insert_many` and the record write finds the write matched nothing (the +/// generation moved), so the lost claim's progress never lands. Runs only +/// under `test-hooks` (the pause point is compiled then) and must run alone +/// (`--test-threads=1`): the pause is driven by a process-wide environment +/// variable. +#[cfg(feature = "test-hooks")] +#[tokio::test(flavor = "multi_thread")] +#[ignore = "needs a MongoDB replica set and --features test-hooks, run with --test-threads=1"] +async fn a_takeover_during_bulk_load_fences_the_file_index_write() { + let fx = Fixture::new().await; + fx.worker.abort(); + let backup = available_backup(&fx.engine, "src", "b").await; + let (_, claim) = fx + .engine + .claim_restore( + ACCOUNT, + manifest_input("fence"), + ®istration(&backup, None), + ) + .await + .expect("register"); + let key = target_key(&table_doc(&fx.engine, "fence").await.expect("target")); + let release = std::env::temp_dir().join(format!( + "extenddb-fence-release-{}", + uuid::Uuid::new_v4().simple() + )); + let _ = std::fs::remove_file(&release); + // SAFETY: a single-threaded test run; no other thread reads the + // environment concurrently, and the variables are cleared at the end. + unsafe { + std::env::set_var("EXTENDDB_BACKUP_PAUSE", "restore_mongodb_after_insert_many"); + std::env::set_var( + "EXTENDDB_BACKUP_PAUSE_RELEASE", + release.display().to_string(), + ); + } + + let engine = fx.engine.clone(); + let loaded_key = key.clone(); + let task = tokio::spawn(async move { + engine + .bulk_load( + &loaded_key, + &*claim, + BulkLoadFile { + file_index: 0, + start_at: 0, + reapplying: false, + }, + item_stream(0, 10), + ) + .await + }); + + // Wait for the documents to land: `bulk_load` is now paused before its + // record write. + let coll = data_collection_name(&key.table_id); + let deadline = Instant::now() + Duration::from_secs(30); + while count(&fx.engine, &coll).await < 10 { + assert!( + Instant::now() < deadline, + "bulk_load did not reach the pause" + ); + tokio::time::sleep(Duration::from_millis(50)).await; + } + + // Take the claim over while the load is paused. + age_running_record(&fx.engine, "fence", Duration::from_secs(600)).await; + let taken = fx + .engine + .claim_restore_dead(&key, STALE_AFTER, None) + .await + .expect("takeover") + .expect("a stale record is taken over"); + + // Release the paused load: its record write is fenced out. + std::fs::write(&release, b"go").expect("release file"); + let out = task.await.unwrap(); + assert!( + matches!(&out, Err(StorageError::Internal(m)) if m.contains("no longer carries this claim")), + "the fenced-out load must not record its file: {out:?}" + ); + // The successor's record never advanced its file_index. + assert_eq!( + fx.engine + .read_restore_job(&key) + .await + .unwrap() + .unwrap() + .file_index, + None, + "the lost claim's file_index write did not land" + ); + + unsafe { + std::env::remove_var("EXTENDDB_BACKUP_PAUSE"); + std::env::remove_var("EXTENDDB_BACKUP_PAUSE_RELEASE"); + } + let _ = std::fs::remove_file(&release); + drop(taken); + fx.finish().await; +} diff --git a/crates/storage-mongodb/src/lib.rs b/crates/storage-mongodb/src/lib.rs index d9a544f46..a8921ca87 100644 --- a/crates/storage-mongodb/src/lib.rs +++ b/crates/storage-mongodb/src/lib.rs @@ -21,6 +21,7 @@ mod management_store; mod metadata_engine; mod operations; pub mod pushdown; +mod restore_engine; mod snapshot_engine; mod stream_engine; mod table_engine; diff --git a/crates/storage-mongodb/src/restore_engine.rs b/crates/storage-mongodb/src/restore_engine.rs new file mode 100644 index 000000000..65e14c232 --- /dev/null +++ b/crates/storage-mongodb/src/restore_engine.rs @@ -0,0 +1,328 @@ +// Copyright 2026 ExtendDB contributors +// SPDX-License-Identifier: Apache-2.0 + +//! The data-plane side of a store-backed restore on `MongoDB` +//! ([`RestoreEngine`]): the items of one verified data file written into the +//! CREATING target's data collection, the per-file record, the hand-over to +//! the index backfill worker and the count. +//! +//! The target's documents are built by [`item_to_document`], the write +//! path's function, so a restored table is laid out as a natively created +//! one and refuses what the write path refuses (an `N` sort key beyond +//! Decimal128's 34 significant digits is the write path's `Validation` +//! error, and fails the file). Base documents only: the target's indexes +//! are created CREATING with it and the GSI backfill worker fills them from +//! the base collection and flips the table ACTIVE once every index has +//! left CREATING, so `finish_restore` is the hand-over +//! ([`crate::MongoEngine::hand_over_to_backfill`]) and answers +//! [`Finished::Pending`]. +//! +//! One file is `insert_many(ordered = false)` calls of +//! [`INSERT_MANY_DOCUMENTS`] documents at `w: majority`, each preceded by +//! one read of the record (`find_one` on the target's `_id`, `table_id`, +//! `restore_job.status` and `restore_job.owner_generation`) that ends the +//! file as a lost claim when the record is not RUNNING under the claim's +//! generation, so a writer whose record another instance took over while it +//! stalled lands at most the one call it was in; then the record's +//! `restore_job.file_index` written conditioned on the generation and on +//! RUNNING: the progress record is per file, so a job that died between the +//! inserts and the record leaves a file applied in part, and the resume +//! re-applies it with [`BulkLoadFile::reapplying`] set, under which a +//! duplicate key (E11000, the data collection's unique `_id`) is suppressed +//! and not counted; on any other file a duplicate key is the backup's and +//! fails the file, as any other write error does. Nothing here keeps progress +//! outside the record, so `read_restore_progress` and +//! `sweep_restore_progress` keep their defaults. + +use extenddb_core::types::{AttributeDefinition, Item, KeySchemaElement}; +use extenddb_storage::backup_catalog::{ + JobClaim, JobStatus, RESTORE_MONGODB_AFTER_INSERT_MANY, RestoreTargetKey, restore_pause_point, +}; +use extenddb_storage::error::StorageError; +use extenddb_storage::restore_engine::{BulkLoadFile, Finished, RestoreEngine}; +use futures::future::BoxFuture; +use futures::stream::BoxStream; +use mongodb::bson::{self, Document, doc}; + +use crate::MongoEngine; +use crate::backup_catalog::{ + RESTORE_JOB_FILE_INDEX, RESTORE_JOB_GENERATION, RESTORE_JOB_STATUS, generation_field, +}; +use crate::data::{data_collection_name, item_to_document}; + +/// Documents per `insert_many` call. Not a message bound: 1,000 documents at +/// the item size limit are about 390 MiB, and the driver splits one call by +/// the server's message and write-batch bounds on its own. It is the unit a +/// half-applied file is re-applied in, and the unit between two reads of the +/// record's claim. +pub(crate) const INSERT_MANY_DOCUMENTS: usize = 1000; + +/// The server's duplicate-key error code. +const DUPLICATE_KEY: i32 = 11000; + +fn internal(e: impl std::fmt::Display) -> StorageError { + StorageError::Internal(e.to_string()) +} + +/// The key shape of a CREATING restore target, read from its catalog +/// document. +struct TargetShape { + key_schema: Vec, + attribute_definitions: Vec, +} + +impl MongoEngine { + /// The target's catalog document, when `key` names a CREATING restore + /// target with this table id. + async fn restore_target_doc(&self, key: &RestoreTargetKey) -> Result { + self.catalog_db + .collection::("tables") + .find_one(doc! { + "_id": { "account_id": &key.account_id, "table_name": &key.table_name }, + "table_id": &key.table_id, + "table_status": "CREATING", + "restore_source_backup_arn": { "$exists": true }, + }) + .await + .map_err(internal)? + .ok_or_else(|| { + StorageError::TableNotFound(format!( + "{} is not a CREATING restore target with table id {}", + key.table_name, key.table_id + )) + }) + } + + /// The claim's liveness, read before each `insert_many`: the record + /// RUNNING under `generation` with this table id. `Err` (a lost claim) + /// otherwise: the reconciler took the record over, and whoever holds it + /// owns the target, so nothing further is written. + async fn confirm_restore_claim( + &self, + target: &RestoreTargetKey, + generation: u64, + ) -> Result<(), StorageError> { + let live = self + .catalog_db + .collection::("tables") + .find_one(doc! { + "_id": { "account_id": &target.account_id, "table_name": &target.table_name }, + "table_id": &target.table_id, + RESTORE_JOB_STATUS: JobStatus::Running.as_str(), + RESTORE_JOB_GENERATION: generation_field(generation), + }) + .projection(doc! { "_id": 1 }) + .await + .map_err(internal)?; + if live.is_none() { + return Err(StorageError::Internal(format!( + "restore of table {}: the claim on the target was lost while a data file \ + loaded (the record is not RUNNING under generation {generation}); the \ + record's owner finishes it", + target.table_name + ))); + } + Ok(()) + } + + /// Write `batch` into the target's data collection and clear it, after + /// confirming the claim; the documents the server took. Under + /// `reapplying` a duplicate key is a document an earlier run of this + /// file wrote and is left out of the count; otherwise it is the backup's + /// and fails the file. + async fn insert_restore_batch( + &self, + collection: &mongodb::Collection, + target: &RestoreTargetKey, + generation: u64, + reapplying: bool, + batch: &mut Vec, + ) -> Result { + self.confirm_restore_claim(target, generation).await?; + let sent = batch.len(); + let result = collection + .insert_many(batch.drain(..)) + .ordered(false) + .write_concern(crate::majority_write_concern(self.client.write_concern())) + .await; + match result { + Ok(_) => Ok(u64::try_from(sent).unwrap_or(u64::MAX)), + Err(e) => match &*e.kind { + mongodb::error::ErrorKind::InsertMany(failure) + if failure.write_concern_error.is_none() + && failure.write_errors.as_ref().is_some_and(|errors| { + errors.iter().all(|w| w.code == DUPLICATE_KEY) + }) => + { + let duplicates = failure.write_errors.as_ref().map_or(0, Vec::len); + if reapplying { + Ok(u64::try_from(sent - duplicates).unwrap_or(u64::MAX)) + } else { + // The engine prefixes the file's key, so the file + // is not named here. + Err(StorageError::Validation(format!( + "holds a primary key the target already has (the backup holds a \ + key twice): {e}" + ))) + } + } + _ => Err(internal(e)), + }, + } + } +} + +impl RestoreEngine for MongoEngine { + fn bulk_load<'a>( + &'a self, + target: &'a RestoreTargetKey, + claim: &'a dyn JobClaim, + file: BulkLoadFile, + mut items: BoxStream<'a, Result>, + ) -> BoxFuture<'a, Result> { + Box::pin(async move { + use futures::StreamExt; + + // The shape from the target's catalog document; the record is + // confirmed RUNNING under the claim before each `insert_many` + // (`confirm_restore_claim`), the first included, and the file's + // record below is the fence itself. + let generation = claim.generation(); + let table_doc = self.restore_target_doc(target).await?; + let shape = TargetShape { + key_schema: bson::from_bson( + table_doc + .get("key_schema") + .cloned() + .ok_or_else(|| internal("restore target is missing key_schema"))?, + ) + .map_err(|e| StorageError::Internal(format!("key schema: {e}")))?, + attribute_definitions: bson::from_bson( + table_doc + .get("attribute_definitions") + .cloned() + .ok_or_else(|| { + internal("restore target is missing attribute_definitions") + })?, + ) + .map_err(|e| StorageError::Internal(format!("attribute definitions: {e}")))?, + }; + let source = format!("data file {}", file.file_index); + let collection = self + .data_db + .collection::(&data_collection_name(&target.table_id)); + // The test-hook gate the in-catalog copy also honours: it holds + // the restore after the target's CREATING index metadata exists + // and before the first insert of its first file, so a test can + // run a backfill worker tick through that window and see that + // the worker completes no index of a restore whose base load has + // not started. Inert without the `test-hooks` feature and unless + // a test armed it. + if file.file_index == 0 && file.start_at == 0 { + self.wait_for_gsi_backfill_test_gate(&target.table_name) + .await?; + } + + let mut batch: Vec = Vec::with_capacity(INSERT_MANY_DOCUMENTS); + let mut applied: u64 = 0; + while let Some(item) = items.next().await { + let item = item?; + batch.push(item_to_document( + &item, + &shape.key_schema, + &shape.attribute_definitions, + )?); + if batch.len() == INSERT_MANY_DOCUMENTS { + applied += self + .insert_restore_batch( + &collection, + target, + generation, + file.reapplying, + &mut batch, + ) + .await?; + } + } + if !batch.is_empty() { + applied += self + .insert_restore_batch( + &collection, + target, + generation, + file.reapplying, + &mut batch, + ) + .await?; + } + restore_pause_point(RESTORE_MONGODB_AFTER_INSERT_MANY).await; + // The file's record, conditioned on the claim: a record another + // generation took over is not written, and the job stops. + let recorded = self + .catalog_db + .collection::("tables") + .update_one( + doc! { + "_id": { "account_id": &target.account_id, "table_name": &target.table_name }, + "table_id": &target.table_id, + RESTORE_JOB_STATUS: JobStatus::Running.as_str(), + RESTORE_JOB_GENERATION: generation_field(claim.generation()), + }, + doc! { "$set": { RESTORE_JOB_FILE_INDEX: i64::from(file.file_index) } }, + ) + .await + .map_err(internal)?; + if recorded.matched_count != 1 { + return Err(StorageError::Internal(format!( + "the restore job record of {} no longer carries this claim (generation {}); \ + {source} was written but not recorded", + target.table_name, + claim.generation() + ))); + } + Ok(applied) + }) + } + + fn finish_restore<'a>( + &'a self, + target: &'a RestoreTargetKey, + claim: &'a dyn JobClaim, + item_count: u64, + size_bytes: u64, + ) -> BoxFuture<'a, Result> { + Box::pin(async move { + let handed_over = self + .hand_over_to_backfill(target, claim.generation(), item_count, Some(size_bytes)) + .await?; + if handed_over { + Ok(Finished::Pending) + } else { + Ok(Finished::NotMatched(format!( + "the job record of {} is not RUNNING under generation {}, or the table is \ + not CREATING with table id {}", + target.table_name, + claim.generation(), + target.table_id + ))) + } + }) + } + + fn count_restore_target<'a>( + &'a self, + key: &'a RestoreTargetKey, + ) -> BoxFuture<'a, Result> { + Box::pin(async move { + // The id names the data collection; the key's account and name + // are checked against the catalog so a bare id reads no other + // account's table. + self.restore_target_doc(key).await?; + self.data_db + .collection::(&data_collection_name(&key.table_id)) + .count_documents(doc! {}) + .await + .map_err(internal) + }) + } +} diff --git a/crates/storage-mongodb/src/snapshot_engine.rs b/crates/storage-mongodb/src/snapshot_engine.rs index aa2cd0878..717688be0 100644 --- a/crates/storage-mongodb/src/snapshot_engine.rs +++ b/crates/storage-mongodb/src/snapshot_engine.rs @@ -74,6 +74,20 @@ //! `paging::SNAPSHOT_FETCH_BYTES` (64 MiB): the reply limit is already under //! that budget, and the byte cut the SQL backends make exists because their //! row-bounded fetches have no such limit. +//! +//! A snapshot read serves the majority-committed snapshot, which can trail +//! what a `readConcern: local` read of the primary already returns (the +//! catalog handle's level, so `DescribeTable`'s): a table flipped ACTIVE a +//! few milliseconds earlier can be missing, or still CREATING, at the +//! snapshot's time, and a CreateBackup sent as soon as DescribeTable said +//! ACTIVE would be refused as a table that does not exist. So an open whose +//! snapshot finds the table missing or not ACTIVE reads the `tables` +//! document at `local`; when that read shows it ACTIVE, the open starts over +//! with a fresh session after a short sleep (5 ms, doubling to 100 ms), +//! for at most [`MAJORITY_CATCH_UP`], and answers the snapshot's refusal +//! once a local read no longer shows the table ACTIVE or the bound has +//! passed. The bound is far above the lag a w:majority write leaves (one +//! journal commit, 100 ms at most by default). use extenddb_core::types::Item; use extenddb_storage::error::StorageError; @@ -105,6 +119,10 @@ const STANDALONE_SERVER: &str = "MongoDB refused the first read at readConcern s standalone server cannot serve snapshot reads, and table snapshots need a replica set \ member"; +/// How long an open waits for the majority-committed snapshot to show a +/// table that a `local` read shows ACTIVE (module doc). +const MAJORITY_CATCH_UP: std::time::Duration = std::time::Duration::from_secs(2); + /// Why an open cannot proceed when the first read succeeded but the session /// reports no snapshot time. const NO_SNAPSHOT_TIME: &str = "MongoDB served the first read without a snapshot time: a read \ @@ -119,13 +137,51 @@ impl SnapshotEngine for MongoEngine { let account_id = account_id.to_owned(); let table_name = table_name.to_owned(); Box::pin(async move { - self.open_table_snapshot_impl(&account_id, &table_name) - .await + let deadline = tokio::time::Instant::now() + MAJORITY_CATCH_UP; + let mut delay = std::time::Duration::from_millis(5); + loop { + match self + .open_table_snapshot_impl(&account_id, &table_name) + .await + { + Err(StorageError::TableNotFound(_) | StorageError::TableNotActive(_)) + if tokio::time::Instant::now() < deadline + && self + .table_is_active_locally(&account_id, &table_name) + .await? => + { + tokio::time::sleep(delay).await; + delay = (delay * 2).min(std::time::Duration::from_millis(100)); + } + outcome => return outcome, + } + } }) } } impl MongoEngine { + /// Whether a `local` read of the `tables` document shows the table + /// ACTIVE: what `DescribeTable` reads, and the snapshot's time may trail. + async fn table_is_active_locally( + &self, + account_id: &str, + table_name: &str, + ) -> Result { + Ok(self + .catalog_db + .collection::("tables") + .find_one(doc! { + "_id": { "account_id": account_id, "table_name": table_name }, + "table_status": "ACTIVE", + }) + .read_concern(ReadConcern::local()) + .projection(doc! { "_id": 1 }) + .await + .map_err(|e| map_mongo_error(&e))? + .is_some()) + } + async fn open_table_snapshot_impl( &self, account_id: &str, @@ -646,6 +702,78 @@ mod tests { .await; } + /// A table a `local` read shows ACTIVE opens although the majority- + /// committed snapshot does not show it yet. The flip to ACTIVE is + /// written at `w: 1` without the journal, so the majority snapshot + /// trails it by up to one journal commit; the test observes that + /// precondition with a snapshot read of its own before each open, and + /// asserts that it saw it at least once. + #[tokio::test] + async fn an_open_waits_for_the_snapshot_to_show_a_table_a_local_read_shows_active() { + let Some(uri) = uri() else { + eprintln!( + "SKIP an_open_waits_for_the_snapshot_to_show_a_table_a_local_read_shows_active: \ + no MongoDB" + ); + return; + }; + let _hooks = hooks().await; + let engine = fixture(&uri).await; + engine + .create_table(ACCOUNT, snapshot_conformance::table_input("lagging")) + .await + .expect("create"); + snapshot_conformance::wait_active(&*engine, ACCOUNT, "lagging", || async { + let _ = + extenddb_storage::WorkerStore::process_control_plane_transitions(&*engine).await; + }) + .await; + let tables = engine.catalog_db.collection::("tables"); + let id = doc! { "_id": { "account_id": ACCOUNT, "table_name": "lagging" } }; + let unjournaled = mongodb::options::WriteConcern::builder() + .w(mongodb::options::Acknowledgment::Nodes(1)) + .journal(false) + .build(); + let mut trailing = 0; + for _ in 0..20 { + set_updating(&engine, "lagging").await; + tables + .update_one(id.clone(), doc! { "$set": { "table_status": "ACTIVE" } }) + .write_concern(unjournaled.clone()) + .await + .expect("flip to ACTIVE"); + let mut session = engine + .client + .start_session() + .snapshot(true) + .await + .expect("session"); + let at_snapshot = tables + .find_one(id.clone()) + .read_concern(ReadConcern::snapshot()) + .session(&mut session) + .await + .expect("snapshot read") + .and_then(|d| d.get_str("table_status").ok().map(str::to_owned)); + if at_snapshot.as_deref() != Some("ACTIVE") { + trailing += 1; + } + let snapshot = engine + .open_table_snapshot(ACCOUNT, "lagging") + .await + .expect("a table a local read shows ACTIVE opens"); + assert_eq!( + snapshot.description.table_status, + extenddb_core::types::TableStatus::Active + ); + } + eprintln!("the snapshot trailed the local read in {trailing} of 20 flips"); + assert!( + trailing > 0, + "precondition: no flip left the majority snapshot behind the local read" + ); + } + /// A document the decoder cannot read ends the stream at its error, as on /// the other backends: nothing after it, so a consumer cannot mistake the /// rest for the whole table. diff --git a/crates/storage-mongodb/src/table_engine.rs b/crates/storage-mongodb/src/table_engine.rs index e744e4b70..9d5abd34b 100644 --- a/crates/storage-mongodb/src/table_engine.rs +++ b/crates/storage-mongodb/src/table_engine.rs @@ -267,11 +267,28 @@ impl MongoEngine { let tables_coll = self.catalog_db.collection::("tables"); let for_restore = restore.is_some(); - let mut session = restore.and_then(|registration| registration.session.as_deref_mut()); - let inserted = match session.as_deref_mut() { + let mut restore = restore; + let inserted = match restore + .as_deref_mut() + .and_then(|registration| registration.session.as_deref_mut()) + { Some(session) => tables_coll.insert_one(table_doc).session(session).await, None => tables_coll.insert_one(table_doc).await, }; + // A manifest record's first heartbeat is rewritten with the server's + // clock in the same transaction (nothing for a legacy one). + let inserted = match inserted { + Ok(_) => match restore.as_deref_mut() { + Some(registration) => { + registration + .stamp_server_clock(&tables_coll, &table_id) + .await + } + None => Ok(()), + }, + Err(e) => Err(e), + }; + let session = restore.and_then(|registration| registration.session.as_deref_mut()); if let Err(e) = inserted { if let Some(session) = session { let _ = session.abort_transaction().await; diff --git a/crates/storage-mongodb/src/ttl_worker.rs b/crates/storage-mongodb/src/ttl_worker.rs index c8e54ef61..38754786b 100644 --- a/crates/storage-mongodb/src/ttl_worker.rs +++ b/crates/storage-mongodb/src/ttl_worker.rs @@ -76,9 +76,11 @@ pub(crate) async fn stream_record_cleanup_worker(storage: Arc) { /// says the table has no indexes; an invalidated in-flight observation /// cannot publish that stale no-index result. /// -/// A restore target whose copy stopped before its hand-over is not this +/// A restore target whose load stopped before its hand-over is not this /// worker's: its job record keeps a stale heartbeat, and the backup -/// reconciler fails the record and removes the target. +/// reconciler acts on it, by its kind: a legacy (in-catalog) record is +/// failed and its target removed; a store-backed (`manifest`) record is +/// taken over and its load resumed, and the hand-over comes from the resume. pub(crate) async fn gsi_backfill_worker(storage: Arc) { loop { tokio::time::sleep(GSI_BACKFILL_INTERVAL).await; @@ -274,9 +276,11 @@ async fn restore_backfill_is_ready( } /// Complete restored tables after every restored GSI/LSI has been populated. -/// The restore request records `restore_backfill_pending` only after the base -/// `$out` copy succeeds, so this remains safe to retry after a process restart -/// or a client disconnect during index backfill. +/// `restore_backfill_pending` is set only once every base document is in +/// place: by the legacy copy after its `$out` succeeds, and by a store-backed +/// restore's `finish_restore` after its count check (`hand_over_to_backfill`), +/// so this remains safe to retry after a process restart or a client +/// disconnect during index backfill. async fn finish_ready_restores(storage: &MongoEngine) { let tables_coll = storage.catalog_db.collection::("tables"); let mut cursor = match tables_coll diff --git a/crates/storage-postgres/data_migrations/0006_restore_progress.sql b/crates/storage-postgres/data_migrations/0006_restore_progress.sql new file mode 100644 index 000000000..6588fc90c --- /dev/null +++ b/crates/storage-postgres/data_migrations/0006_restore_progress.sql @@ -0,0 +1,18 @@ +-- Copyright 2026 ExtendDB contributors +-- SPDX-License-Identifier: Apache-2.0 +-- +-- Migration 6, restore_progress, PostgreSQL data database. +-- +-- The progress record of a store-backed restore: one row per data file the +-- restore applied into its CREATING target, written in the data transaction +-- that wrote the file's rows, so a file is recorded exactly when its rows +-- are durable. The catalog's `table_restores` row names the target; this +-- table lives beside the target's rows because the two databases cannot +-- share a transaction. The restore's end deletes the target's rows, and the +-- reconciler removes rows of targets no RUNNING restore owns. +CREATE TABLE IF NOT EXISTS restore_progress ( + table_id TEXT NOT NULL, + file_index BIGINT NOT NULL, + items_applied BIGINT NOT NULL, + PRIMARY KEY (table_id, file_index) +); diff --git a/crates/storage-postgres/src/backup_catalog.rs b/crates/storage-postgres/src/backup_catalog.rs index f766e1cef..349e164d4 100644 --- a/crates/storage-postgres/src/backup_catalog.rs +++ b/crates/storage-postgres/src/backup_catalog.rs @@ -2,7 +2,8 @@ // SPDX-License-Identifier: Apache-2.0 //! The catalog index on `PostgreSQL`: `backup_index`, the restore job fields -//! of `table_restores`, and `backup_maintenance`. +//! of `table_restores`, and `backup_maintenance`. The data-plane side of a +//! store-backed restore is `restore_engine.rs`. //! //! The claim of a backup job is `pg_try_advisory_lock(hashtextextended( //! 'backup:' || backup_id, RESTORE_LOCK_SEED))` on a dedicated, non-pooled @@ -42,10 +43,12 @@ use std::time::Duration; use extenddb_core::backup_format::BackupArn; +use extenddb_core::types::{CreateTableInput, RestoreSummary}; use extenddb_storage::backup_catalog::{ BackupCatalog, BackupKey, BackupRow, BackupStatus, DeleteStart, Heartbeat, JobClaim, JobRow, - JobStatus, MaintenanceClaim, RestoreJobRow, RestoreKind, RestoreTargetKey, StaleClock, - StatusFields, StoreIdentity, StoreKind, TimeBounds, WriteOutcome, + JobStatus, MaintenanceClaim, RestoreJobRow, RestoreKind, RestoreRegistered, + RestoreRegistrationRow, RestoreTargetKey, StaleClock, StatusFields, StoreIdentity, StoreKind, + TimeBounds, WriteOutcome, }; use extenddb_storage::error::StorageError; use futures::future::BoxFuture; @@ -55,8 +58,10 @@ use tokio::sync::Mutex; use crate::PostgresEngine; use crate::backup_engine::{ - ClaimSession, backup_lock_key, generation_column, try_claim, try_claim_with, + ClaimSession, RESTORE_HEARTBEAT, backup_lock_key, generation_column, restore_lock_key, + try_claim, try_claim_with, }; +use crate::create_table::RestoreSource; /// The name of a CREATING restore target registered from `backup_arn`, if /// any: the one in-use rule DeleteBackup, a restore's registration and @@ -300,6 +305,174 @@ impl JobClaim for PostgresBackupClaim { } } +/// A store-backed restore job's handle on its `table_restores` record: the +/// advisory lock on the target name (`restore_lock_key`) on the claim +/// session, the generation, and the pool for reads. The heartbeat and the +/// status writes go through the session, so a session the server dropped +/// surfaces as a lost heartbeat or an unknown write outcome. +struct PostgresRestoreClaim { + session: Mutex, + generation: u64, + table_id: String, + pool: sqlx::PgPool, +} + +impl JobClaim for PostgresRestoreClaim { + fn generation(&self) -> u64 { + self.generation + } + + fn heartbeat(&self) -> BoxFuture<'_, Result> { + Box::pin(async move { + let mut session = self.session.lock().await; + let confirmed = session + .confirm(RESTORE_HEARTBEAT, &[&self.table_id]) + .await + .map_err(|e| StorageError::Internal(format!("claim session: {e}")))?; + Ok(if confirmed { + Heartbeat::Confirmed + } else { + Heartbeat::Lost + }) + }) + } + + fn set_status<'a>( + &'a self, + from: JobStatus, + to: JobStatus, + fields: &'a StatusFields, + ) -> BoxFuture<'a, Result> { + Box::pin(async move { + let mut session = self.session.lock().await; + let written = sqlx::query( + "UPDATE table_restores SET status = $3, \ + failure_reason = COALESCE($4, failure_reason), \ + completed_at = CASE WHEN $5 THEN now() ELSE completed_at END \ + WHERE table_id = $1 AND owner_generation = $2 AND status = $6", + ) + .bind(&self.table_id) + .bind(generation_column(self.generation)) + .bind(to.as_str()) + .bind(fields.failure_reason.as_deref()) + .bind(fields.completed) + .bind(from.as_str()) + .execute(session.connection()) + .await; + match written { + Ok(result) if result.rows_affected() == 1 => Ok(WriteOutcome::Matched), + Ok(_) => Ok(WriteOutcome::NotMatched), + Err(e) if outcome_unknown(&e) => Ok(WriteOutcome::Unknown), + Err(e) => Err(db_err(e)), + } + }) + } + + fn read(&self) -> BoxFuture<'_, Result> { + Box::pin(async move { + let row: Option<(String, i64, f64, Option)> = sqlx::query_as( + "SELECT status, owner_generation, EXTRACT(EPOCH FROM heartbeat_at)::FLOAT8, \ + failure_reason FROM table_restores WHERE table_id = $1", + ) + .bind(&self.table_id) + .fetch_optional(&self.pool) + .await + .map_err(db_err)?; + let (status, generation, heartbeat_at, failure_reason) = row.ok_or_else(|| { + StorageError::TableNotFound(format!( + "restore target {} has no job record", + self.table_id + )) + })?; + Ok(JobRow { + status: JobStatus::parse(&status)?, + owner_generation: u64::try_from(generation).unwrap_or_default(), + heartbeat_at, + failure_reason, + }) + }) + } +} + +/// The columns of one restore job record, joined with its table, in the +/// order [`RestoreJobTuple`] reads them. +const RESTORE_ROW_COLUMNS: &str = "r.table_id, t.account_id, t.table_name, r.source_backup_arn, \ + r.kind, r.status, r.owner_generation, EXTRACT(EPOCH FROM r.heartbeat_at)::FLOAT8, \ + r.store_kind, r.store_root, r.store_id, r.spool_host, r.file_index, r.items_applied, \ + r.failure_reason, EXTRACT(EPOCH FROM r.completed_at)::FLOAT8"; + +/// One restore job record as the driver reads it. +#[allow(clippy::type_complexity)] +type RestoreJobTuple = ( + String, + String, + String, + String, + String, + String, + i64, + f64, + Option, + Option, + Option, + Option, + Option, + Option, + Option, + Option, +); + +fn restore_job_from_tuple(t: RestoreJobTuple) -> Result { + let ( + table_id, + account_id, + table_name, + source_backup_arn, + kind, + status, + owner_generation, + heartbeat_at, + store_kind, + store_root, + store_id, + spool_host, + file_index, + items_applied, + failure_reason, + completed_at, + ) = t; + let store = match (store_kind, store_root, store_id) { + (Some(kind), Some(store_root), Some(store_id)) => Some(StoreIdentity { + store_kind: StoreKind::parse(&kind).ok_or_else(|| { + StorageError::Internal(format!( + "restore of {source_backup_arn} carries an unknown store kind {kind:?}" + )) + })?, + store_root, + store_id, + }), + _ => None, + }; + Ok(RestoreJobRow { + target: RestoreTargetKey { + account_id, + table_name, + table_id, + }, + source_backup_arn, + kind: RestoreKind::parse(&kind)?, + status: JobStatus::parse(&status)?, + owner_generation: u64::try_from(owner_generation).unwrap_or_default(), + heartbeat_at, + store, + spool_host, + file_index: file_index.and_then(|i| u32::try_from(i).ok()), + items_applied: items_applied.and_then(|i| u64::try_from(i).ok()), + failure_reason, + completed_at, + }) +} + impl BackupCatalog for PostgresEngine { fn catalog_id(&self) -> BoxFuture<'_, Result> { Box::pin(async move { @@ -444,10 +617,9 @@ impl BackupCatalog for PostgresEngine { // row must meet: it locks the same row before its own in-use // check, so the two order wholly one before the other (a // registration that commits first leaves this `InUse`; one that - // waits behind this finds the row DELETING). On this head the - // legacy registration locks the legacy `backups` row and no - // restore of a `backup_index` row exists; PR 9a's registration - // is the first to take this lock. + // waits behind this finds the row DELETING). `claim_restore` + // (`create_table.rs`, the manifest arm) takes this lock; the + // legacy registration locks the legacy `backups` row instead. let status: Option = sqlx::query_scalar( "SELECT status FROM backup_index WHERE account_id = $1 AND backup_id = $2 \ FOR UPDATE", @@ -638,17 +810,18 @@ impl BackupCatalog for PostgresEngine { if !is_target { return Ok(false); } - // Progress rows belong to a later release's restore path; a - // catalog without that table has none to delete. + // Progress rows live in the data database beside the target's + // rows (migration 6); a data database from before that + // migration has none to delete. let has_progress: bool = sqlx::query_scalar("SELECT to_regclass('restore_progress') IS NOT NULL") - .fetch_one(&self.pool) + .fetch_one(&self.data_pool) .await .map_err(db_err)?; if has_progress { sqlx::query("DELETE FROM restore_progress WHERE table_id = $1") .bind(&key.table_id) - .execute(&self.pool) + .execute(&self.data_pool) .await .map_err(db_err)?; } @@ -668,51 +841,21 @@ impl BackupCatalog for PostgresEngine { // job's own failure path stamp `completed_at`, and a record // without one is aged by its heartbeat). The delay is what // leaves a job still on its own failure path to finish it. - #[allow(clippy::type_complexity)] - let rows: Vec<(String, String, String, String, String, String, i64, f64)> = - sqlx::query_as( - "SELECT r.table_id, t.account_id, t.table_name, r.source_backup_arn, \ - r.kind, r.status, r.owner_generation, \ - EXTRACT(EPOCH FROM r.heartbeat_at)::FLOAT8 \ - FROM table_restores r JOIN tables t ON t.table_id = r.table_id \ - WHERE (r.status = 'RUNNING' \ - AND r.heartbeat_at < now() - make_interval(secs => $1)) \ - OR (r.status = 'FAILED' AND t.table_status = 'CREATING' \ - AND COALESCE(r.completed_at, r.heartbeat_at) \ - < now() - make_interval(secs => $1)) \ - ORDER BY t.account_id, t.table_name", - ) - .bind(stale_after.as_secs_f64()) - .fetch_all(&self.pool) - .await - .map_err(db_err)?; - rows.into_iter() - .map( - |( - table_id, - account_id, - table_name, - source_backup_arn, - kind, - status, - owner_generation, - heartbeat_at, - )| { - Ok(RestoreJobRow { - target: RestoreTargetKey { - account_id, - table_name, - table_id, - }, - source_backup_arn, - kind: RestoreKind::parse(&kind)?, - status: JobStatus::parse(&status)?, - owner_generation: u64::try_from(owner_generation).unwrap_or_default(), - heartbeat_at, - }) - }, - ) - .collect() + let rows: Vec = sqlx::query_as(&format!( + "SELECT {RESTORE_ROW_COLUMNS} \ + FROM table_restores r JOIN tables t ON t.table_id = r.table_id \ + WHERE (r.status = 'RUNNING' \ + AND r.heartbeat_at < now() - make_interval(secs => $1)) \ + OR (r.status = 'FAILED' AND t.table_status = 'CREATING' \ + AND COALESCE(r.completed_at, r.heartbeat_at) \ + < now() - make_interval(secs => $1)) \ + ORDER BY t.account_id, t.table_name" + )) + .bind(stale_after.as_secs_f64()) + .fetch_all(&self.pool) + .await + .map_err(db_err)?; + rows.into_iter().map(restore_job_from_tuple).collect() }) } @@ -740,6 +883,158 @@ impl BackupCatalog for PostgresEngine { }) } + fn claim_restore<'a>( + &'a self, + account_id: &'a str, + input: CreateTableInput, + registration: &'a RestoreRegistrationRow, + ) -> BoxFuture<'a, Result> { + Box::pin(async move { + let table_name = input.table_name.clone(); + // Ownership first, before the target exists, on the lock the + // legacy restore takes for a name (`restore_lock_key`), so a + // legacy and a store-backed restore into one name exclude each + // other too. A lock already held is another restore into this + // name in flight: TableBeingRestored naming its source once its + // CREATING row is visible, else the name is taken as CreateTable + // would report it. The record's `heartbeat_at` is `now()` of the + // registration transaction, its start (`create_table.rs`), so a + // registration that waited on the backup's row lock or whose data + // DDL took long returns a record already that old; the engine's + // first beat refreshes it before the reply. + let Some(session) = + try_claim(&self.pool, &restore_lock_key(account_id, &table_name), None).await? + else { + return Err(self.name_taken_error(account_id, &table_name, true).await); + }; + let generation = session.generation(); + let registration = RestoreRegistrationRow { + owner_generation: generation, + ..registration.clone() + }; + let mut desc = match self + .create_table_for_restore(account_id, input, RestoreSource::Manifest(®istration)) + .await + { + Ok(desc) => desc, + Err(e) => { + session.close().await; + return Err(e); + } + }; + desc.restore_summary = Some(RestoreSummary { + source_backup_arn: Some(registration.backup.backup_arn.clone()), + source_table_arn: Some(registration.source_table_arn.clone()), + restore_date_time: registration.restore_date_time, + restore_in_progress: true, + }); + let table_id = desc.table_id.clone(); + Ok(( + desc, + Box::new(PostgresRestoreClaim { + session: Mutex::new(session), + generation, + table_id, + pool: self.pool.clone(), + }) as Box, + )) + }) + } + + fn claim_restore_dead<'a>( + &'a self, + key: &'a RestoreTargetKey, + stale_after: Duration, + spool_host: Option<&'a str>, + ) -> BoxFuture<'a, Result>, StorageError>> { + Box::pin(async move { + // The same advisory lock the owner took on the target name: a + // session that still holds it is alive whatever its heartbeat + // says, and a second taker is refused here. + let Some(mut session) = try_claim( + &self.pool, + &restore_lock_key(&key.account_id, &key.table_name), + None, + ) + .await? + else { + return Ok(None); + }; + let generation = session.generation(); + // The generation, the claimant's spool and a fresh heartbeat in + // one write: a second taker, on this or another instance, finds + // the record live at once (the trait doc says why a restore + // record is refreshed where a backup row is not). + let taken = sqlx::query( + "UPDATE table_restores r SET owner_generation = $4, spool_host = $5, \ + heartbeat_at = now() \ + FROM tables t WHERE t.table_id = r.table_id AND r.table_id = $1 \ + AND t.account_id = $2 AND t.table_name = $3 \ + AND r.status = 'RUNNING' AND r.kind = 'manifest' \ + AND r.heartbeat_at < now() - make_interval(secs => $6)", + ) + .bind(&key.table_id) + .bind(&key.account_id) + .bind(&key.table_name) + .bind(generation_column(generation)) + .bind(spool_host) + .bind(stale_after.as_secs_f64()) + .execute(session.connection()) + .await + .map_err(db_err)? + .rows_affected(); + if taken != 1 { + session.close().await; + return Ok(None); + } + Ok(Some(Box::new(PostgresRestoreClaim { + session: Mutex::new(session), + generation, + table_id: key.table_id.clone(), + pool: self.pool.clone(), + }) as Box)) + }) + } + + fn read_restore_job<'a>( + &'a self, + key: &'a RestoreTargetKey, + ) -> BoxFuture<'a, Result, StorageError>> { + Box::pin(async move { + let row: Option = sqlx::query_as(&format!( + "SELECT {RESTORE_ROW_COLUMNS} \ + FROM table_restores r JOIN tables t ON t.table_id = r.table_id \ + WHERE r.table_id = $1 AND t.account_id = $2 AND t.table_name = $3" + )) + .bind(&key.table_id) + .bind(&key.account_id) + .bind(&key.table_name) + .fetch_optional(&self.pool) + .await + .map_err(db_err)?; + row.map(restore_job_from_tuple).transpose() + }) + } + + fn list_restore_jobs<'a>( + &'a self, + status: Option, + ) -> BoxFuture<'a, Result, StorageError>> { + Box::pin(async move { + let rows: Vec = sqlx::query_as(&format!( + "SELECT {RESTORE_ROW_COLUMNS} \ + FROM table_restores r JOIN tables t ON t.table_id = r.table_id \ + WHERE ($1::TEXT IS NULL OR r.status = $1) \ + ORDER BY t.account_id, t.table_name" + )) + .bind(status.map(JobStatus::as_str)) + .fetch_all(&self.pool) + .await + .map_err(db_err)?; + rows.into_iter().map(restore_job_from_tuple).collect() + }) + } + fn read_store_ids<'a>( &'a self, store_kind: StoreKind, diff --git a/crates/storage-postgres/src/backup_engine.rs b/crates/storage-postgres/src/backup_engine.rs index ac18fcd6c..18bc0bd78 100755 --- a/crates/storage-postgres/src/backup_engine.rs +++ b/crates/storage-postgres/src/backup_engine.rs @@ -31,10 +31,10 @@ use extenddb_storage::{BackupRestoreSource, JobPermit, LegacyBackupReader, Resto use futures::future::BoxFuture; use crate::PostgresEngine; -use crate::create_table::RestoreRegistration; +use crate::create_table::{RestoreRegistration, RestoreSource}; use crate::data::{ all_sort_key_info, data_table_name, index_table_name, insert_index_rows_multi, - item_has_index_keys, project_item_for_index, + item_has_index_keys, project_item_for_index, table_storage_relations, }; /// Convert a `PostgreSQL` `TIMESTAMPTZ` to epoch seconds as `f64`, keeping the @@ -65,10 +65,198 @@ fn pg_timestamp_to_epoch(ts: time::OffsetDateTime) -> f64 { pub const RESTORE_LOCK_SEED: i64 = 0x0045_4452; // 'E', 'D', 'R' /// A secondary index of a restore target, as the copy needs it. -struct RestoreIndex { - table: String, - key_schema: Vec, - projection: Projection, +pub(crate) struct RestoreIndex { + pub(crate) table: String, + pub(crate) key_schema: Vec, + pub(crate) projection: Projection, +} + +/// The secondary indexes of `table_id` as the catalog records them, each +/// with the data table its rows go into. +pub(crate) async fn restore_indexes( + pool: &sqlx::PgPool, + table_id: &str, +) -> Result, StorageError> { + let index_rows: Vec<(String, serde_json::Value, serde_json::Value)> = + sqlx::query_as("SELECT index_id, key_schema, projection FROM indexes WHERE table_id = $1") + .bind(table_id) + .fetch_all(pool) + .await + .map_err(db_err)?; + let mut indexes = Vec::with_capacity(index_rows.len()); + for (index_id, ks, proj) in index_rows { + indexes.push(RestoreIndex { + table: index_table_name(&index_id), + key_schema: serde_json::from_value(ks) + .map_err(|e| StorageError::Internal(format!("index key schema: {e}")))?, + projection: serde_json::from_value(proj) + .map_err(|e| StorageError::Internal(format!("index projection: {e}")))?, + }); + } + Ok(indexes) +} + +/// Why [`insert_restore_rows`] stopped: the base table refused a key it +/// already holds (the backup carries a key twice), or anything else. The +/// legacy copy reports both as the database error ([`From`]); the +/// store-backed load names the data file for the first. +pub(crate) enum RowInsertError { + DuplicateKey(sqlx::Error), + Storage(StorageError), +} + +impl From for RowInsertError { + fn from(e: StorageError) -> Self { + Self::Storage(e) + } +} + +impl From for StorageError { + fn from(e: RowInsertError) -> Self { + match e { + RowInsertError::DuplicateKey(e) => db_err(e), + RowInsertError::Storage(e) => e, + } + } +} + +/// SQLSTATE `unique_violation`. +const SQLSTATE_UNIQUE_VIOLATION: &str = "23505"; + +/// A restore target as the row writes need it, built once per restore: the +/// base data table, the key shape, and the secondary indexes with their +/// data tables. `source` names the backup or data file in errors. +pub(crate) struct RestoreTarget<'a> { + pub(crate) ddb_table: String, + pub(crate) key_schema: &'a [KeySchemaElement], + pub(crate) attribute_definitions: &'a [extenddb_core::types::AttributeDefinition], + pub(crate) sort_keys: Vec<(&'a str, ScalarAttributeType)>, + pub(crate) indexes: Vec, + pub(crate) source: String, +} + +impl<'a> RestoreTarget<'a> { + /// The target `table_id` with `key_schema` and `attribute_definitions`, + /// its indexes read from the catalog. + pub(crate) async fn load( + pool: &sqlx::PgPool, + table_id: &str, + key_schema: &'a [KeySchemaElement], + attribute_definitions: &'a [extenddb_core::types::AttributeDefinition], + source: String, + ) -> Result { + Ok(Self { + ddb_table: data_table_name(table_id), + key_schema, + attribute_definitions, + sort_keys: all_sort_key_info(key_schema, attribute_definitions), + indexes: restore_indexes(pool, table_id).await?, + source, + }) + } +} + +/// Write one batch of restored items inside `tx`: one multi-row INSERT into +/// the base table, then one multi-row INSERT into each secondary index for +/// the batch's items that carry that index's keys. Plain INSERTs: two rows +/// for one key fail the statement rather than collapsing into one item. +pub(crate) async fn insert_restore_rows( + tx: &mut sqlx::Transaction<'_, sqlx::Postgres>, + target: &RestoreTarget<'_>, + batch: &[(Item, String)], +) -> Result<(), RowInsertError> { + if batch.is_empty() { + return Ok(()); + } + let RestoreTarget { + ddb_table, + key_schema, + attribute_definitions, + sort_keys, + indexes, + source, + } = target; + let mut cols = vec!["pk".to_owned()]; + cols.extend( + sort_keys + .iter() + .enumerate() + .map(|(i, &(_, t))| sk_column_n(i, t)), + ); + cols.push("item_data".to_owned()); + let width = cols.len(); + // The last column is the item, bound as text and cast in SQL. + let tuples: Vec = (0..batch.len()) + .map(|r| { + let ps: Vec = (1..=width) + .map(|c| { + if c == width { + format!("${}::jsonb", r * width + c) + } else { + format!("${}", r * width + c) + } + }) + .collect(); + format!("({})", ps.join(", ")) + }) + .collect(); + let sql = format!( + "INSERT INTO {ddb_table} ({}) VALUES {}", + cols.join(", "), + tuples.join(", ") + ); + let mut query = sqlx::query(&sql); + for (item, item_json) in batch { + query = query.bind(composite_pk_to_text(item, key_schema)?); + for &(name, sk_type) in sort_keys { + let value = item.get(name).ok_or_else(|| { + StorageError::Internal(format!("{source} has an item without sort key {name}")) + })?; + query = match parse_sk(value, sk_type)? { + SortKeyValue::S(v) => query.bind(v), + SortKeyValue::N(v) => query.bind(v), + SortKeyValue::B(v) => query.bind(v), + }; + } + query = query.bind(item_json.as_str()); + } + query.execute(&mut **tx).await.map_err(|e| { + let duplicate = e + .as_database_error() + .and_then(|d| d.code()) + .is_some_and(|code| code == SQLSTATE_UNIQUE_VIOLATION); + if duplicate { + RowInsertError::DuplicateKey(e) + } else { + RowInsertError::Storage(db_err(e)) + } + })?; + + for idx in indexes { + let idx_sks = all_sort_key_info(&idx.key_schema, attribute_definitions); + let projected: Vec<(&Item, Item)> = batch + .iter() + .filter(|(item, _)| item_has_index_keys(item, &idx.key_schema)) + .map(|(item, _)| { + ( + item, + project_item_for_index(item, &idx.key_schema, key_schema, &idx.projection), + ) + }) + .collect(); + let rows: Vec<(&Item, &Item)> = projected.iter().map(|(i, p)| (*i, p)).collect(); + insert_index_rows_multi( + tx, + &idx.table, + &rows, + &idx.key_schema, + key_schema, + &idx_sks, + sort_keys, + ) + .await?; + } + Ok(()) } /// Session-scoped ownership of one job, held for the job's life: a @@ -168,9 +356,9 @@ pub(crate) fn generation_column(generation: u64) -> i64 { i64::try_from(generation).unwrap_or(i64::MAX) } -/// The heartbeat of a legacy restore: refresh its job record while it is -/// RUNNING under this generation. -const RESTORE_HEARTBEAT: &str = "UPDATE table_restores SET heartbeat_at = now() \ +/// The heartbeat of a restore: refresh its job record while it is RUNNING +/// under this generation. +pub(crate) const RESTORE_HEARTBEAT: &str = "UPDATE table_restores SET heartbeat_at = now() \ WHERE table_id = $1 AND status = 'RUNNING' AND owner_generation = $2"; /// The advisory-lock key for a restore into `table_name` in `account_id`. @@ -181,7 +369,7 @@ const RESTORE_HEARTBEAT: &str = "UPDATE table_restores SET heartbeat_at = now() /// `hashtextextended`, so a collision is negligible, and one would only make /// a second restore into the other name wait, never remove a table it should /// not. -fn restore_lock_key(account_id: &str, table_name: &str) -> String { +pub(crate) fn restore_lock_key(account_id: &str, table_name: &str) -> String { format!("{account_id}/{table_name}") } @@ -537,24 +725,14 @@ impl PostgresEngine { use futures::TryStreamExt; let ddb_table = data_table_name(&desc.table_id); - let sort_keys = all_sort_key_info(&desc.key_schema, &desc.attribute_definitions); - let index_rows: Vec<(String, serde_json::Value, serde_json::Value)> = sqlx::query_as( - "SELECT index_id, key_schema, projection FROM indexes WHERE table_id = $1", + let target = RestoreTarget::load( + &self.pool, + &desc.table_id, + &desc.key_schema, + &desc.attribute_definitions, + format!("backup {backup_arn}"), ) - .bind(&desc.table_id) - .fetch_all(&self.pool) - .await - .map_err(db_err)?; - let mut indexes = Vec::with_capacity(index_rows.len()); - for (index_id, ks, proj) in index_rows { - indexes.push(RestoreIndex { - table: index_table_name(&index_id), - key_schema: serde_json::from_value(ks) - .map_err(|e| StorageError::Internal(format!("index key schema: {e}")))?, - projection: serde_json::from_value(proj) - .map_err(|e| StorageError::Internal(format!("index projection: {e}")))?, - }); - } + .await?; // The backup's rows are read from one catalog snapshot in which the // backup is still AVAILABLE. DeleteBackup refuses while this restore @@ -596,10 +774,7 @@ impl PostgresEngine { batch_bytes += item_json.len(); batch.push((item, item_json)); if batch.len() == COPY_BATCH_ITEMS || batch_bytes >= COPY_BATCH_BYTES { - self.write_restore_batch( - &ddb_table, desc, &sort_keys, &indexes, &batch, backup_arn, - ) - .await?; + self.write_restore_batch(&target, &batch).await?; item_count += i64::try_from(batch.len()).unwrap_or(i64::MAX); batch.clear(); batch_bytes = 0; @@ -608,8 +783,7 @@ impl PostgresEngine { drop(rows); snapshot.commit().await.map_err(db_err)?; if !batch.is_empty() { - self.write_restore_batch(&ddb_table, desc, &sort_keys, &indexes, &batch, backup_arn) - .await?; + self.write_restore_batch(&target, &batch).await?; item_count += i64::try_from(batch.len()).unwrap_or(i64::MAX); } @@ -624,18 +798,47 @@ impl PostgresEngine { // and its job record DONE, in one catalog transaction. The table was // created with the transition deferred, so this is the first point it // can become ACTIVE, by ordering rather than by timing. - // - // Only from CREATING, and only while the job record is RUNNING under - // this generation. DeleteTable refuses a restore target, but the - // reconciler can claim one whose owner it found dead (moving it to - // DELETING); that claim wins, and flipping the row back to ACTIVE - // would leave a table whose data tables are being dropped. + if !self + .activate_restore_target(&desc.table_id, generation, item_count, table_size) + .await? + { + // The target was claimed for removal while the copy ran, or the + // job record passed to another claim; the copy's rows go with it. + return Err(StorageError::TableNotFound(format!( + "restore of {backup_arn} into {} did not complete: the table was deleted \ + while it was being restored", + desc.table_name + ))); + } + Ok(()) + } + + /// The end of a restore in one catalog transaction: the job record + /// RUNNING to DONE under `generation`, the table CREATING to ACTIVE with + /// its counts, and every secondary and vector index of the table ACTIVE + /// (the indexes are created ACTIVE with a restore target, so this is the + /// rule stated once rather than a change). `Ok(false)`, with nothing + /// written, when the record is not RUNNING under `generation` or the + /// table is not CREATING. + /// + /// Only from CREATING, and only while the job record is RUNNING under + /// this generation. DeleteTable refuses a restore target, but the + /// reconciler can claim one whose owner it found dead (moving it to + /// DELETING); that claim wins, and flipping the row back to ACTIVE + /// would leave a table whose data tables are being dropped. + pub(crate) async fn activate_restore_target( + &self, + table_id: &str, + generation: u64, + item_count: i64, + table_size: i64, + ) -> Result { let mut tx = self.pool.begin().await.map_err(db_err)?; let done = sqlx::query( "UPDATE table_restores SET status = 'DONE', completed_at = now() \ WHERE table_id = $1 AND status = 'RUNNING' AND owner_generation = $2", ) - .bind(&desc.table_id) + .bind(table_id) .bind(generation_column(generation)) .execute(&mut *tx) .await @@ -649,7 +852,7 @@ impl PostgresEngine { ) .bind(item_count) .bind(table_size) - .bind(&desc.table_id) + .bind(table_id) .execute(&mut *tx) .await .map_err(db_err)? @@ -659,16 +862,26 @@ impl PostgresEngine { }; if activated == 0 { tx.rollback().await.map_err(db_err)?; - // The target was claimed for removal while the copy ran, or the - // job record passed to another claim; the copy's rows go with it. - return Err(StorageError::TableNotFound(format!( - "restore of {backup_arn} into {} did not complete: the table was deleted \ - while it was being restored", - desc.table_name - ))); + return Ok(false); } + sqlx::query( + "UPDATE indexes SET index_status = 'ACTIVE' \ + WHERE table_id = $1 AND index_status <> 'ACTIVE'", + ) + .bind(table_id) + .execute(&mut *tx) + .await + .map_err(db_err)?; + sqlx::query( + "UPDATE vector_indexes SET index_status = 'ACTIVE', backfilling = NULL \ + WHERE table_id = $1 AND index_status <> 'ACTIVE'", + ) + .bind(table_id) + .execute(&mut *tx) + .await + .map_err(db_err)?; tx.commit().await.map_err(db_err)?; - Ok(()) + Ok(true) } /// RUNNING to FAILED on the job record of `table_id`, conditioned on @@ -696,96 +909,15 @@ impl PostgresEngine { Ok(affected == 1) } - /// Write and commit one batch of restored items: one multi-row INSERT - /// into the base table, then one multi-row INSERT into each secondary - /// index for the batch's items that carry that index's keys. + /// Write and commit one batch of restored items ([`insert_restore_rows`] + /// in one data transaction). async fn write_restore_batch( &self, - ddb_table: &str, - desc: &TableDescription, - sort_keys: &[(&str, ScalarAttributeType)], - indexes: &[RestoreIndex], + target: &RestoreTarget<'_>, batch: &[(Item, String)], - backup_arn: &str, ) -> Result<(), StorageError> { let mut tx = self.data_pool.begin().await.map_err(db_err)?; - let mut cols = vec!["pk".to_owned()]; - cols.extend( - sort_keys - .iter() - .enumerate() - .map(|(i, &(_, t))| sk_column_n(i, t)), - ); - cols.push("item_data".to_owned()); - let width = cols.len(); - // The last column is the item, bound as text and cast in SQL. - let tuples: Vec = (0..batch.len()) - .map(|r| { - let ps: Vec = (1..=width) - .map(|c| { - if c == width { - format!("${}::jsonb", r * width + c) - } else { - format!("${}", r * width + c) - } - }) - .collect(); - format!("({})", ps.join(", ")) - }) - .collect(); - let sql = format!( - "INSERT INTO {ddb_table} ({}) VALUES {}", - cols.join(", "), - tuples.join(", ") - ); - let mut query = sqlx::query(&sql); - for (item, item_json) in batch { - query = query.bind(composite_pk_to_text(item, &desc.key_schema)?); - for &(name, sk_type) in sort_keys { - let value = item.get(name).ok_or_else(|| { - StorageError::Internal(format!( - "backup {backup_arn} has an item without sort key {name}" - )) - })?; - query = match parse_sk(value, sk_type)? { - SortKeyValue::S(v) => query.bind(v), - SortKeyValue::N(v) => query.bind(v), - SortKeyValue::B(v) => query.bind(v), - }; - } - query = query.bind(item_json.as_str()); - } - query.execute(&mut *tx).await.map_err(db_err)?; - - for idx in indexes { - let idx_sks = all_sort_key_info(&idx.key_schema, &desc.attribute_definitions); - let projected: Vec<(&Item, Item)> = batch - .iter() - .filter(|(item, _)| item_has_index_keys(item, &idx.key_schema)) - .map(|(item, _)| { - ( - item, - project_item_for_index( - item, - &idx.key_schema, - &desc.key_schema, - &idx.projection, - ), - ) - }) - .collect(); - let rows: Vec<(&Item, &Item)> = projected.iter().map(|(i, p)| (*i, p)).collect(); - insert_index_rows_multi( - &mut tx, - &idx.table, - &rows, - &idx.key_schema, - &desc.key_schema, - &idx_sks, - sort_keys, - ) - .await?; - } + insert_restore_rows(&mut tx, target, batch).await?; tx.commit().await.map_err(db_err)?; Ok(()) } @@ -804,6 +936,12 @@ impl PostgresEngine { /// DELETING removal, so if a drop below fails, the row stays DELETING with /// its index rows and the control plane finishes the job later. /// + /// The relations dropped are the ones `DeleteTable` drops + /// (`table_storage_relations`): the data table, every secondary index's + /// table and every vector index's table, their ids read in the claiming + /// transaction while the catalog rows still exist (the `tables` row's + /// deletion cascades them away). + /// /// Public for the test that runs it against an ACTIVE table; not an /// interface. #[doc(hidden)] @@ -825,13 +963,17 @@ impl PostgresEngine { .fetch_all(&mut *tx) .await .map_err(db_err)?; + let vector_index_ids: Vec = + sqlx::query_scalar("SELECT index_id FROM vector_indexes WHERE table_id = $1") + .bind(table_id) + .fetch_all(&mut *tx) + .await + .map_err(db_err)?; tx.commit().await.map_err(db_err)?; if !claimed { return Ok(false); } - let mut drops = vec![data_table_name(table_id)]; - drops.extend(index_ids.iter().map(|id| index_table_name(id))); - for t in drops { + for t in table_storage_relations(table_id, &index_ids, &vector_index_ids) { sqlx::query(&format!("DROP TABLE IF EXISTS {t}")) .execute(&self.data_pool) .await @@ -926,7 +1068,7 @@ impl PostgresEngine { owner_generation: owner.generation(), }; let mut desc = match self - .create_table_for_restore(&account_id, input, ®istration) + .create_table_for_restore(&account_id, input, RestoreSource::Legacy(®istration)) .await { Ok(desc) => desc, diff --git a/crates/storage-postgres/src/create_table.rs b/crates/storage-postgres/src/create_table.rs index 7e4098a82..61e8928c2 100755 --- a/crates/storage-postgres/src/create_table.rs +++ b/crates/storage-postgres/src/create_table.rs @@ -7,13 +7,14 @@ use extenddb_core::types::{ BillingMode, BillingModeSummary, CreateTableInput, GsiDescription, LsiDescription, ProvisionedThroughputDescription, SseDescription, SseType, TableDescription, TableStatus, }; +use extenddb_storage::backup_catalog::RestoreRegistrationRow; use extenddb_storage::error::StorageError; use extenddb_storage::util::{index_arn, stream_arn, table_arn}; use crate::PostgresEngine; -/// What a restore target records about where it came from, written in the -/// same catalog transaction that creates it. +/// What a legacy restore target records about where it came from, written +/// in the same catalog transaction that creates it. pub(crate) struct RestoreRegistration<'a> { pub(crate) backup_arn: &'a str, /// Reported as `RestoreSummary.SourceTableArn`. @@ -26,6 +27,18 @@ pub(crate) struct RestoreRegistration<'a> { pub(crate) owner_generation: u64, } +/// Which backup a restore target is registered from, and so which row its +/// registration locks: a legacy restore locks the `backups` row of the +/// in-catalog backup, a store-backed one the `backup_index` row (the lock +/// DeleteBackup's `start_backup_delete` takes). The two paths keep their own +/// locks: a legacy and a store-backed restore are of two different backups +/// and never meet, and a DeleteBackup of a `backup_index` row meets only +/// store-backed registrations. +pub(crate) enum RestoreSource<'a> { + Legacy(&'a RestoreRegistration<'a>), + Manifest(&'a RestoreRegistrationRow), +} + impl PostgresEngine { /// Core implementation of `create_table` (Fix #4: wrapped in a transaction). /// Create a table. When `defer_active` is set (the restore path), the row @@ -54,7 +67,7 @@ impl PostgresEngine { &self, account_id: &str, input: CreateTableInput, - registration: &RestoreRegistration<'_>, + registration: RestoreSource<'_>, ) -> Result { self.create_table_impl_inner(account_id, input, true, Some(registration)) .await @@ -107,7 +120,7 @@ impl PostgresEngine { account_id: &str, input: CreateTableInput, defer_active: bool, - restore: Option<&RestoreRegistration<'_>>, + restore: Option>, ) -> Result { Self::validate_account_id(account_id)?; let table_id = uuid::Uuid::new_v4().to_string(); @@ -149,32 +162,67 @@ impl PostgresEngine { .await .map_err(|e| StorageError::Internal(e.to_string()))?; - if let Some(RestoreRegistration { backup_arn, .. }) = restore { - // FOR UPDATE, not FOR SHARE: two restores of one backup must not - // both pass the CREATING-target check below, so they serialize on - // the backup row (DeleteBackup locks it the same way). - let available: Option = sqlx::query_scalar( - "SELECT 1 FROM backups WHERE backup_arn = $1 AND account_id = $2 \ - AND backup_status = 'AVAILABLE' FOR UPDATE", - ) - .bind(backup_arn) - .bind(account_id) - .fetch_optional(&mut *tx) - .await - .map_err(|e| StorageError::Internal(e.to_string()))?; - if available.is_none() { - return Err(StorageError::Validation(format!( - "Backup not found: {backup_arn}" - ))); + match &restore { + Some(RestoreSource::Legacy(RestoreRegistration { backup_arn, .. })) => { + // FOR UPDATE, not FOR SHARE: two restores of one backup must not + // both pass the CREATING-target check below, so they serialize on + // the backup row (DeleteBackup locks it the same way). + let available: Option = sqlx::query_scalar( + "SELECT 1 FROM backups WHERE backup_arn = $1 AND account_id = $2 \ + AND backup_status = 'AVAILABLE' FOR UPDATE", + ) + .bind(backup_arn) + .bind(account_id) + .fetch_optional(&mut *tx) + .await + .map_err(|e| StorageError::Internal(e.to_string()))?; + if available.is_none() { + return Err(StorageError::Validation(format!( + "Backup not found: {backup_arn}" + ))); + } + // One restore of a backup at a time, as the service: while a + // target restored from it is CREATING the backup is in use. Same + // predicate as DeleteBackup's. + if let Some(table) = + crate::backup_catalog::creating_restore_target(&mut *tx, backup_arn).await? + { + return Err(crate::backup_catalog::backup_in_use(&table, backup_arn)); + } } - // One restore of a backup at a time, as the service: while a - // target restored from it is CREATING the backup is in use. Same - // predicate as DeleteBackup's. - if let Some(table) = - crate::backup_catalog::creating_restore_target(&mut *tx, backup_arn).await? - { - return Err(crate::backup_catalog::backup_in_use(&table, backup_arn)); + Some(RestoreSource::Manifest(registration)) => { + // The `backup_index` row FOR UPDATE: the lock + // `start_backup_delete` takes before its own in-use check, so + // a DeleteBackup and this registration order wholly one + // before the other, and two registrations of one backup + // serialize on the row before the CREATING-target check. + let status: Option = sqlx::query_scalar( + "SELECT status FROM backup_index WHERE account_id = $1 AND backup_id = $2 \ + FOR UPDATE", + ) + .bind(account_id) + .bind(®istration.backup.backup_id) + .fetch_optional(&mut *tx) + .await + .map_err(|e| StorageError::Internal(e.to_string()))?; + if status.as_deref() != Some("AVAILABLE") { + return Err(extenddb_storage::backup_catalog::backup_not_found( + ®istration.backup.backup_arn, + )); + } + if let Some(table) = crate::backup_catalog::creating_restore_target( + &mut *tx, + ®istration.backup.backup_arn, + ) + .await? + { + return Err(crate::backup_catalog::backup_in_use( + &table, + ®istration.backup.backup_arn, + )); + } } + None => {} } // Insert table metadata, returning creation timestamp and actual status. @@ -236,24 +284,53 @@ impl PostgresEngine { Err(e) => return Err(StorageError::Internal(e.to_string())), }; - if let Some(r) = restore { - // The job record: RUNNING under the claim's generation, with a - // heartbeat on the catalog's clock, so no CREATING target ever - // exists without an owned record. - sqlx::query( - "INSERT INTO table_restores \ - (table_id, source_backup_arn, source_table_arn, restore_date_time, \ - status, kind, owner_generation, heartbeat_at) \ - VALUES ($1, $2, $3, to_timestamp($4), 'RUNNING', 'legacy', $5, now())", - ) - .bind(&table_id) - .bind(r.backup_arn) - .bind(r.source_table_arn) - .bind(r.restore_date_time) - .bind(crate::backup_engine::generation_column(r.owner_generation)) - .execute(&mut *tx) - .await - .map_err(|e| StorageError::Internal(e.to_string()))?; + // The job record: RUNNING under the claim's generation, with a + // heartbeat on the catalog's clock, so no CREATING target ever + // exists without an owned record. `now()` is this transaction's + // start, not the insert's instant: a registration that waited on the + // `backup_index` row's lock above commits a heartbeat that old, and + // the data DDL after the commit adds to the age the engine's first + // beat (sent before the registration's reply) then refreshes. + match &restore { + Some(RestoreSource::Legacy(r)) => { + sqlx::query( + "INSERT INTO table_restores \ + (table_id, source_backup_arn, source_table_arn, restore_date_time, \ + status, kind, owner_generation, heartbeat_at) \ + VALUES ($1, $2, $3, to_timestamp($4), 'RUNNING', 'legacy', $5, now())", + ) + .bind(&table_id) + .bind(r.backup_arn) + .bind(r.source_table_arn) + .bind(r.restore_date_time) + .bind(crate::backup_engine::generation_column(r.owner_generation)) + .execute(&mut *tx) + .await + .map_err(|e| StorageError::Internal(e.to_string()))?; + } + Some(RestoreSource::Manifest(r)) => { + sqlx::query( + "INSERT INTO table_restores \ + (table_id, source_backup_arn, source_table_arn, restore_date_time, \ + status, kind, store_kind, store_root, store_id, spool_host, \ + owner_generation, heartbeat_at) \ + VALUES ($1, $2, $3, to_timestamp($4), 'RUNNING', 'manifest', $5, $6, $7, \ + $8, $9, now())", + ) + .bind(&table_id) + .bind(&r.backup.backup_arn) + .bind(&r.source_table_arn) + .bind(r.restore_date_time) + .bind(r.store.store_kind.as_str()) + .bind(&r.store.store_root) + .bind(&r.store.store_id) + .bind(r.spool_host.as_deref()) + .bind(crate::backup_engine::generation_column(r.owner_generation)) + .execute(&mut *tx) + .await + .map_err(|e| StorageError::Internal(e.to_string()))?; + } + None => {} } // Insert GSI metadata diff --git a/crates/storage-postgres/src/lib.rs b/crates/storage-postgres/src/lib.rs index 310a416bb..e54b9f68f 100755 --- a/crates/storage-postgres/src/lib.rs +++ b/crates/storage-postgres/src/lib.rs @@ -25,6 +25,7 @@ mod metadata_engine; pub mod migrations; mod operations; mod pg_util; +mod restore_engine; mod snapshot_engine; #[doc(hidden)] pub use snapshot_engine::{ diff --git a/crates/storage-postgres/src/migrations/mod.rs b/crates/storage-postgres/src/migrations/mod.rs index a8ef03a74..95c687f4d 100644 --- a/crates/storage-postgres/src/migrations/mod.rs +++ b/crates/storage-postgres/src/migrations/mod.rs @@ -127,6 +127,19 @@ pub fn migration(id: MigrationId) -> Migration { "0005_backup_catalog_index.sql" )], }, + MigrationId::RestoreProgress => Migration::Applies { + // A new table is additive by CONTRIBUTING's step 4, but declaring + // `Kind::Additive` needs the old-binary job, which does not exist + // yet (the paragraph after step 4; `refuse_additive` in + // `crates/bin` pins it). + kind: Kind::Breaking, + convergent: true, + bodies: vec![file_body!( + StoreKind::Data, + "data_migrations", + "0006_restore_progress.sql" + )], + }, } } @@ -148,7 +161,8 @@ pub(crate) const fn runs_in(id: MigrationId) -> RunsIn { MigrationId::Baseline | MigrationId::ResourceTagConditionKeys | MigrationId::SortKeyDefinitions - | MigrationId::BackupCatalogIndex => RunsIn::Transaction, + | MigrationId::BackupCatalogIndex + | MigrationId::RestoreProgress => RunsIn::Transaction, MigrationId::IndexTableBaseKey => RunsIn::Autocommit, } } diff --git a/crates/storage-postgres/src/restore_engine.rs b/crates/storage-postgres/src/restore_engine.rs new file mode 100644 index 000000000..dcdcf3ab7 --- /dev/null +++ b/crates/storage-postgres/src/restore_engine.rs @@ -0,0 +1,373 @@ +// Copyright 2026 ExtendDB contributors +// SPDX-License-Identifier: Apache-2.0 + +//! The data-plane side of a store-backed restore on `PostgreSQL` +//! ([`RestoreEngine`]): the items of one verified data file written into the +//! CREATING target, the progress record, the ACTIVE flip and the counts. +//! +//! One data transaction per file: the base rows in multi-row INSERTs of +//! `COPY_BATCH_ITEMS` rows or `COPY_BATCH_BYTES` of item text, each +//! secondary index's rows beside them (the `insert_restore_rows` the legacy +//! copy shares), each vector index's rows through the write path's +//! `VectorInsertPlan`, and the file's `restore_progress` row last, so a file +//! commits whole with its record or not at all. The data database has no +//! `tables` row and no job record, so nothing in `bulk_load` reads or +//! conditions on the claim's generation: the fence is the base table's +//! primary key, which refuses a file applied twice, and the claim's +//! liveness, which the engine confirms before each file. Plain INSERTs, so +//! a key the backup holds twice, within a file or across files, fails the +//! file (SQLSTATE 23505) instead of collapsing two items. The progress row +//! is a plain INSERT for a file applied from its start and an upsert +//! (`ON CONFLICT (table_id, file_index) DO UPDATE SET items_applied`) for a +//! file resumed past it, so a row left short of the file's count by a +//! decoder that died after the commit is completed by the resume. +//! +//! `finish_restore` is `activate_restore_target` (the catalog transaction +//! the legacy copy ends with: the record DONE, the table and every index +//! ACTIVE, conditioned on the generation and on CREATING), then the +//! progress rows deleted on the data pool. The delete runs after the commit +//! and cannot fail the restore: its error is logged at WARN and +//! `Finished::Active` returned, since the restore is complete once the flip +//! committed and the rows are the reconciler's `sweep_restore_progress` to +//! remove, as they are after a crash between the two. + +use std::collections::HashSet; + +use extenddb_core::types::{AttributeDefinition, Item, KeySchemaElement}; +use extenddb_storage::backup_catalog::{ + JobClaim, RESTORE_POSTGRES_BEFORE_PROGRESS_DELETE, RestoreTargetKey, restore_pause_point, +}; +use extenddb_storage::backup_definition::{COPY_BATCH_BYTES, COPY_BATCH_ITEMS}; +use extenddb_storage::error::StorageError; +use extenddb_storage::restore_engine::{BulkLoadFile, Finished, RestoreEngine}; +use futures::future::BoxFuture; +use futures::stream::BoxStream; + +use crate::PostgresEngine; +use crate::backup_engine::{RestoreTarget, RowInsertError, insert_restore_rows}; +use crate::data::data_table_name; +use crate::data::vector_index::{VectorInsertPlan, fetch_vector_indexes_for_table}; + +fn db_err(e: sqlx::Error) -> StorageError { + StorageError::Internal(format!("Database error: {e}")) +} + +/// The key shape of a CREATING restore target, read from its catalog row. +struct TargetShape { + key_schema: Vec, + attribute_definitions: Vec, +} + +impl PostgresEngine { + /// The target's key schema and attribute definitions, when `key` names + /// a CREATING restore target with this table id. + async fn restore_target_shape( + &self, + key: &RestoreTargetKey, + ) -> Result { + let row: Option<(serde_json::Value, serde_json::Value)> = sqlx::query_as( + "SELECT t.key_schema, t.attribute_definitions FROM tables t \ + JOIN table_restores r ON r.table_id = t.table_id \ + WHERE t.table_id = $1 AND t.account_id = $2 AND t.table_name = $3 \ + AND t.table_status = 'CREATING'", + ) + .bind(&key.table_id) + .bind(&key.account_id) + .bind(&key.table_name) + .fetch_optional(&self.pool) + .await + .map_err(db_err)?; + let (key_schema, attribute_definitions) = row.ok_or_else(|| { + StorageError::TableNotFound(format!( + "{} is not a CREATING restore target with table id {}", + key.table_name, key.table_id + )) + })?; + Ok(TargetShape { + key_schema: serde_json::from_value(key_schema) + .map_err(|e| StorageError::Internal(format!("key schema: {e}")))?, + attribute_definitions: serde_json::from_value(attribute_definitions) + .map_err(|e| StorageError::Internal(format!("attribute definitions: {e}")))?, + }) + } +} + +/// The error for a key the target already holds: the backup carries it +/// twice, within the file or in a file applied before it. The engine +/// prefixes the file's key, so the file is not named here. +fn duplicate_key_error(e: &sqlx::Error) -> StorageError { + StorageError::Validation(format!( + "holds a primary key the target already has (the backup holds a key twice): {e}" + )) +} + +/// Write `batch` into the base table, every secondary index and every vector +/// index inside the file's transaction, and clear it. +async fn flush_batch( + tx: &mut sqlx::Transaction<'_, sqlx::Postgres>, + target: &RestoreTarget<'_>, + vector_plans: &[VectorInsertPlan<'_>], + batch: &mut Vec<(Item, String)>, +) -> Result { + insert_restore_rows(tx, target, batch) + .await + .map_err(|e| match e { + RowInsertError::DuplicateKey(e) => duplicate_key_error(&e), + RowInsertError::Storage(e) => e, + })?; + for plan in vector_plans { + for (item, _) in batch.iter() { + plan.insert(tx, item).await?; + } + } + let written = u64::try_from(batch.len()).unwrap_or(u64::MAX); + batch.clear(); + Ok(written) +} + +impl RestoreEngine for PostgresEngine { + fn bulk_load<'a>( + &'a self, + target: &'a RestoreTargetKey, + _claim: &'a dyn JobClaim, + file: BulkLoadFile, + mut items: BoxStream<'a, Result>, + ) -> BoxFuture<'a, Result> { + Box::pin(async move { + use futures::StreamExt; + + let shape = self.restore_target_shape(target).await?; + let source = format!("data file {}", file.file_index); + let rows = RestoreTarget::load( + &self.pool, + &target.table_id, + &shape.key_schema, + &shape.attribute_definitions, + source.clone(), + ) + .await?; + let vector_metas = fetch_vector_indexes_for_table(&self.pool, &target.table_id).await?; + let vector_plans: Vec> = vector_metas + .iter() + .map(|(meta, _)| { + VectorInsertPlan::new(meta, &shape.key_schema, &shape.attribute_definitions) + }) + .collect(); + + let mut tx = self.data_pool.begin().await.map_err(db_err)?; + let mut batch: Vec<(Item, String)> = Vec::with_capacity(COPY_BATCH_ITEMS); + let mut batch_bytes = 0usize; + let mut applied: u64 = 0; + while let Some(item) = items.next().await { + let item = item?; + let item_json = serde_json::to_string(&item) + .map_err(|e| StorageError::Internal(format!("item: {e}")))?; + batch_bytes += item_json.len(); + batch.push((item, item_json)); + if batch.len() == COPY_BATCH_ITEMS || batch_bytes >= COPY_BATCH_BYTES { + applied += flush_batch(&mut tx, &rows, &vector_plans, &mut batch).await?; + batch_bytes = 0; + } + } + if !batch.is_empty() { + applied += flush_batch(&mut tx, &rows, &vector_plans, &mut batch).await?; + } + // The file's record, in the file's transaction. A file applied from + // its start writes a new row: a duplicate `(table_id, file_index)` + // is a file applied twice, which the base table's primary key has + // refused above for any file with items. A file resumed past its + // start (`start_at > 0`) updates the row the earlier run left + // (one with `items_applied` short of the file's count: the + // decoder died after the file's commit and the job's FAILED write + // never landed), so the resume completes the file instead of + // failing on the row's key. + let progress_row = if file.start_at > 0 { + "INSERT INTO restore_progress (table_id, file_index, items_applied) \ + VALUES ($1, $2, $3) \ + ON CONFLICT (table_id, file_index) DO UPDATE \ + SET items_applied = EXCLUDED.items_applied" + } else { + "INSERT INTO restore_progress (table_id, file_index, items_applied) \ + VALUES ($1, $2, $3)" + }; + sqlx::query(progress_row) + .bind(&target.table_id) + .bind(i64::from(file.file_index)) + .bind(i64::try_from(file.start_at.saturating_add(applied)).unwrap_or(i64::MAX)) + .execute(&mut *tx) + .await + .map_err(|e| { + let duplicate = e + .as_database_error() + .and_then(|d| d.code()) + .is_some_and(|code| code == "23505"); + if duplicate { + StorageError::Validation(format!( + "{source} was applied before and its progress recorded: {e}" + )) + } else { + db_err(e) + } + })?; + tx.commit().await.map_err(db_err)?; + Ok(applied) + }) + } + + fn finish_restore<'a>( + &'a self, + target: &'a RestoreTargetKey, + claim: &'a dyn JobClaim, + item_count: u64, + size_bytes: u64, + ) -> BoxFuture<'a, Result> { + Box::pin(async move { + // The key names the table; the flip itself is conditioned on the + // table id, CREATING, RUNNING and the generation. + let is_target: bool = sqlx::query_scalar( + "SELECT EXISTS(SELECT 1 FROM tables WHERE table_id = $1 AND account_id = $2 \ + AND table_name = $3)", + ) + .bind(&target.table_id) + .bind(&target.account_id) + .bind(&target.table_name) + .fetch_one(&self.pool) + .await + .map_err(db_err)?; + if !is_target { + return Ok(Finished::NotMatched(format!( + "{} is not a table with id {} in account {}", + target.table_name, target.table_id, target.account_id + ))); + } + let activated = self + .activate_restore_target( + &target.table_id, + claim.generation(), + i64::try_from(item_count).unwrap_or(i64::MAX), + i64::try_from(size_bytes).unwrap_or(i64::MAX), + ) + .await?; + if !activated { + return Ok(Finished::NotMatched(format!( + "the job record of {} is not RUNNING under generation {}, or the table is \ + not CREATING", + target.table_name, + claim.generation() + ))); + } + restore_pause_point(RESTORE_POSTGRES_BEFORE_PROGRESS_DELETE).await; + // The flip has committed: the table is ACTIVE and the record DONE, + // so the restore is complete whatever happens to the progress + // rows. A failure here is logged, not returned: an `Err` would make + // the engine fail a finished restore, and the rows are harmless + // (the data database reads them for nothing once the record is + // DONE) and the reconciler's `sweep_restore_progress` removes them. + if let Err(e) = sqlx::query("DELETE FROM restore_progress WHERE table_id = $1") + .bind(&target.table_id) + .execute(&self.data_pool) + .await + { + tracing::warn!( + "restore of table {}: the table is ACTIVE and its record DONE, but the \ + restore_progress rows of table id {} could not be deleted: {e}; the progress \ + rows are the reconciler's to sweep", + target.table_name, + target.table_id + ); + } + Ok(Finished::Active) + }) + } + + fn count_restore_target<'a>( + &'a self, + key: &'a RestoreTargetKey, + ) -> BoxFuture<'a, Result> { + Box::pin(async move { + // The id names the data table; the key's account and name are + // checked against the catalog so a bare id reads no other + // account's table. + self.restore_target_shape(key).await?; + let count: i64 = sqlx::query_scalar(&format!( + "SELECT count(*) FROM {}", + data_table_name(&key.table_id) + )) + .fetch_one(&self.data_pool) + .await + .map_err(db_err)?; + Ok(u64::try_from(count).unwrap_or_default()) + }) + } + + fn read_restore_progress<'a>( + &'a self, + key: &'a RestoreTargetKey, + ) -> BoxFuture<'a, Result, StorageError>> { + Box::pin(async move { + let rows: Vec<(i64, i64)> = sqlx::query_as( + "SELECT file_index, items_applied FROM restore_progress WHERE table_id = $1 \ + ORDER BY file_index", + ) + .bind(&key.table_id) + .fetch_all(&self.data_pool) + .await + .map_err(db_err)?; + Ok(rows + .into_iter() + .map(|(index, applied)| { + ( + u32::try_from(index).unwrap_or(u32::MAX), + u64::try_from(applied).unwrap_or_default(), + ) + }) + .collect()) + }) + } + + fn sweep_restore_progress<'a>( + &'a self, + running: &'a [String], + ) -> BoxFuture<'a, Result> { + Box::pin(async move { + // A positive list, candidates before the live set (the trait + // doc): the ids the progress rows name now, then the catalog's + // RUNNING records; a target registered after `running` was + // listed is RUNNING here, and one whose first row lands after + // the first read is not among the candidates. + let named: Vec = + sqlx::query_scalar("SELECT DISTINCT table_id FROM restore_progress") + .fetch_all(&self.data_pool) + .await + .map_err(db_err)?; + if named.is_empty() { + return Ok(0); + } + let live: Vec = + sqlx::query_scalar("SELECT table_id FROM table_restores WHERE status = 'RUNNING'") + .fetch_all(&self.pool) + .await + .map_err(db_err)?; + let protected: HashSet<&str> = running + .iter() + .chain(live.iter()) + .map(String::as_str) + .collect(); + let stale: Vec = named + .into_iter() + .filter(|id| !protected.contains(id.as_str())) + .collect(); + if stale.is_empty() { + return Ok(0); + } + let removed = + sqlx::query("DELETE FROM restore_progress WHERE table_id = ANY($1::TEXT[])") + .bind(&stale) + .execute(&self.data_pool) + .await + .map_err(db_err)? + .rows_affected(); + Ok(removed) + }) + } +} diff --git a/crates/storage-postgres/tests/restore_job.rs b/crates/storage-postgres/tests/restore_job.rs new file mode 100644 index 000000000..e16e5faa0 --- /dev/null +++ b/crates/storage-postgres/tests/restore_job.rs @@ -0,0 +1,1743 @@ +// Copyright 2026 ExtendDB contributors +// SPDX-License-Identifier: Apache-2.0 +//! Storage-level tests of the store-backed restore on PostgreSQL: the +//! registration under the `backup_index` row's lock (`claim_restore`), the +//! takeover of a dead record (`claim_restore_dead`), the per-file load with +//! its progress row (`bulk_load`), the ACTIVE flip (`finish_restore`), the +//! counts and the progress sweep. The engine's restore job drives these +//! through `BackupCatalog` and `RestoreEngine`; what each must guarantee on +//! this backend is what a test here pins. +//! +//! Each test builds a throwaway catalog and data database and drops them +//! when it ends, passing or failing, as `backup_restore.rs` does; the tests +//! run one at a time. Requires `EXTENDDB_TEST_PG_CONNECTION_STRING`; +//! without it every test reports a skip and passes, unless +//! `EXTENDDB_TEST_PG_REQUIRED` says the job must reach a server. + +use std::time::Duration; + +use extenddb_core::types::{ + CreateTableInput, DistanceFunction, Item, Projection, ProjectionType, TableDescription, + TableStatus, VectorAttribute, VectorIndexSpecification, +}; +use extenddb_storage::backup_catalog::{ + BackupRow, BackupStatus, DeleteStart, Heartbeat, JobClaim, JobStatus, RestoreJobRow, + RestoreKind, RestoreRegistrationRow, RestoreTargetKey, StatusFields, StoreIdentity, StoreKind, + WriteOutcome, +}; +use extenddb_storage::error::StorageError; +use extenddb_storage::migrations::Runner; +use extenddb_storage::restore_engine::{BulkLoadFile, Finished, RestoreEngine}; +use extenddb_storage::{BackupCatalog, TableEngine}; +use extenddb_storage_postgres::RESTORE_LOCK_SEED; +use extenddb_storage_postgres::migrations::{PostgresMigrations, PostgresTarget}; +use extenddb_storage_postgres::{PostgresConfig, PostgresEngine}; +use sqlx::PgPool; +use sqlx::postgres::PgPoolOptions; + +const ACCOUNT: &str = "123456789012"; +const REGION: &str = "us-east-1"; +/// A stale window no test's heartbeat crosses on its own. +const STALE: Duration = Duration::from_secs(300); + +// --- the scratch databases -------------------------------------------------------- + +/// A catalog database and a separate data database, the layout `extenddb +/// init` creates, with pgvector installed in the data database when the +/// server offers it. +struct Scratch { + engine: PostgresEngine, + /// The data database. + db: PgPool, + catalog: PgPool, + admin: PgPool, + db_names: [String; 2], + vector: bool, + _guard: DbGuard, + _serial: tokio::sync::MutexGuard<'static, ()>, +} + +/// Drops the scratch databases if the test panics, from the moment they are +/// created, including during setup. +struct DbGuard { + names: Vec, +} + +/// One test at a time: each opens its own pools and claim sessions. +static SERIAL: tokio::sync::Mutex<()> = tokio::sync::Mutex::const_new(()); + +impl Scratch { + async fn cleanup(self) { + self.db.close().await; + self.catalog.close().await; + drop_databases(&self.admin, &self.db_names).await; + self.admin.close().await; + } +} + +async fn drop_databases(admin: &PgPool, names: &[String]) { + for name in names { + let _ = sqlx::query(&format!("DROP DATABASE IF EXISTS \"{name}\" WITH (FORCE)")) + .execute(admin) + .await; + } +} + +impl Drop for DbGuard { + fn drop(&mut self) { + if !std::thread::panicking() { + return; + } + let (Some(base), names) = (base_conn(), self.names.clone()) else { + return; + }; + let _ = std::thread::spawn(move || { + let Ok(rt) = tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + else { + return; + }; + rt.block_on(async { + if let Ok(admin) = PgPoolOptions::new() + .max_connections(1) + .connect(&format!("{base}/postgres")) + .await + { + drop_databases(&admin, &names).await; + admin.close().await; + } + }); + }) + .join(); + } +} + +/// The server's base URL, or `None` to skip. With `EXTENDDB_TEST_PG_REQUIRED` +/// set to any value but `0`, a job that must reach a server fails here +/// instead. +fn base_conn() -> Option { + let conn = std::env::var("EXTENDDB_TEST_PG_CONNECTION_STRING").unwrap_or_default(); + if conn.trim().is_empty() { + let required = + std::env::var("EXTENDDB_TEST_PG_REQUIRED").is_ok_and(|v| !v.is_empty() && v != "0"); + assert!( + !required, + "EXTENDDB_TEST_PG_CONNECTION_STRING is not set, and EXTENDDB_TEST_PG_REQUIRED says \ + this job must reach a PostgreSQL server" + ); + return None; + } + Some(conn.trim_end_matches('/').to_owned()) +} + +/// `true` when the test can run; prints the skip line otherwise. +fn pg_available(test: &str) -> bool { + if base_conn().is_none() { + eprintln!("SKIP {test}: no PostgreSQL"); + return false; + } + true +} + +async fn connect(url: &str) -> PgPool { + PgPoolOptions::new() + .max_connections(3) + .connect(url) + .await + .expect("connect to a scratch database") +} + +async fn scratch() -> Scratch { + let serial = SERIAL.lock().await; + let base = base_conn().expect("caller checks base_conn() first"); + let stem = format!("eddb_rstr_{}", uuid::Uuid::new_v4().simple())[..24].to_owned(); + let (catalog_name, data_name) = (format!("{stem}_c"), format!("{stem}_d")); + let admin = PgPoolOptions::new() + .max_connections(1) + .connect(&format!("{base}/postgres")) + .await + .expect("connect to the postgres maintenance database"); + let guard = DbGuard { + names: vec![catalog_name.clone(), data_name.clone()], + }; + for name in [&catalog_name, &data_name] { + sqlx::query(&format!("CREATE DATABASE \"{name}\"")) + .execute(&admin) + .await + .expect("create a scratch database"); + } + let catalog_url = format!("{base}/{catalog_name}"); + let data_url = format!("{base}/{data_name}"); + // A fresh install, migrated by the runner as `init` does: migration 6 + // creates `restore_progress` in the data database. + let target = PostgresTarget::new(catalog_url.parse().expect("a postgres URL"), &data_name) + .with_data_connection_row(&data_url); + let runner = PostgresMigrations::open_unlocked(&target) + .await + .expect("open the migration runner"); + Runner::new(&runner, "test") + .migrate() + .await + .expect("migrate the scratch databases"); + runner.close().await; + let catalog = connect(&catalog_url).await; + let db = connect(&data_url).await; + let vector: bool = sqlx::query_scalar( + "SELECT EXISTS(SELECT 1 FROM pg_available_extensions WHERE name = 'vector')", + ) + .fetch_one(&db) + .await + .expect("read the available extensions"); + if vector { + sqlx::query("CREATE EXTENSION IF NOT EXISTS vector") + .execute(&db) + .await + .expect("create the pgvector extension"); + } else { + eprintln!("NOTE: no pgvector on this server; the vector index cells are skipped"); + } + for key in ["control_plane_delay_seconds", "index_propagation_delay_ms"] { + sqlx::query( + "INSERT INTO settings (key, value) VALUES ($1, '0') \ + ON CONFLICT (key) DO UPDATE SET value = EXCLUDED.value", + ) + .bind(key) + .execute(&catalog) + .await + .expect("set a delay to zero"); + } + sqlx::query( + "INSERT INTO settings (key, value) VALUES ('data_database_connection_string', $1) \ + ON CONFLICT (key) DO UPDATE SET value = EXCLUDED.value", + ) + .bind(&data_url) + .execute(&catalog) + .await + .expect("point the catalog at the data database"); + sqlx::query("INSERT INTO accounts (account_id, account_name) VALUES ($1, $2)") + .bind(ACCOUNT) + .bind(format!("acct-{stem}")) + .execute(&catalog) + .await + .expect("seed the account row"); + let engine = PostgresEngine::new( + &PostgresConfig { + connection_string: catalog_url.clone(), + pool_size: 6, + max_item_size_bytes: 400_000, + }, + REGION, + ) + .await + .expect("open a PostgresEngine on the scratch databases"); + Scratch { + engine, + db, + catalog, + admin, + db_names: [catalog_name, data_name], + vector, + _guard: guard, + _serial: serial, + } +} + +// --- rows, inputs and items ----------------------------------------------------- + +fn store() -> StoreIdentity { + StoreIdentity { + store_kind: StoreKind::S3, + store_root: "s3://bucket/extenddb".to_owned(), + store_id: "store-1".to_owned(), + } +} + +/// An AVAILABLE `backup_index` row of a backup of `table`, written through +/// the claim as a backup job writes it. +async fn available_backup(s: &Scratch, table: &str) -> BackupRow { + let backup_id = extenddb_storage::util::new_backup_id(); + let backup_arn = extenddb_storage::util::backup_arn(REGION, ACCOUNT, table, &backup_id); + let manifest: serde_json::Value = serde_json::from_str(include_str!( + "../../core/testdata/extenddb-manifest/v1.json" + )) + .expect("fixture"); + let row = BackupRow { + backup_id: backup_id.clone(), + account_id: ACCOUNT.to_owned(), + table_name: table.to_owned(), + backup_arn: backup_arn.clone(), + backup_name: "nightly".to_owned(), + status: BackupStatus::Creating, + store: store(), + size_bytes: 0, + item_count: 0, + created_at: extenddb_storage::util::backup_creation_date_time(&backup_id).unwrap(), + completed_at: None, + failure_reason: None, + source_table_details_json: manifest["source_table"].clone(), + owner_generation: 0, + heartbeat_at: 0.0, + }; + let claim = s.engine.claim_backup(&row).await.expect("claim the row"); + let written = claim + .set_status( + JobStatus::Creating, + JobStatus::Available, + &StatusFields { + item_count: Some(3), + size_bytes: Some(1234), + completed: true, + ..StatusFields::default() + }, + ) + .await + .expect("the AVAILABLE write"); + assert_eq!(written, WriteOutcome::Matched); + drop(claim); + BackupRow { + status: BackupStatus::Available, + ..row + } +} + +fn registration(row: &BackupRow, spool_host: Option<&str>) -> RestoreRegistrationRow { + RestoreRegistrationRow { + backup: row.key(), + source_table_arn: format!( + "arn:aws:dynamodb:{REGION}:{ACCOUNT}:table/{}", + row.table_name + ), + restore_date_time: row.created_at, + store: row.store.clone(), + spool_host: spool_host.map(str::to_owned), + owner_generation: 0, + } +} + +/// The target: a composite-key table with a GSI on `g`, an LSI on `l`, and +/// a vector index on `emb` (three dimensions) when the server has pgvector. +fn target_input(name: &str, vector: bool) -> CreateTableInput { + let mut input: CreateTableInput = serde_json::from_value(serde_json::json!({ + "TableName": name, + "KeySchema": [ + {"AttributeName": "pk", "KeyType": "HASH"}, + {"AttributeName": "sk", "KeyType": "RANGE"} + ], + "AttributeDefinitions": [ + {"AttributeName": "pk", "AttributeType": "S"}, + {"AttributeName": "sk", "AttributeType": "N"}, + {"AttributeName": "g", "AttributeType": "S"}, + {"AttributeName": "l", "AttributeType": "S"} + ], + "BillingMode": "PAY_PER_REQUEST", + "GlobalSecondaryIndexes": [{ + "IndexName": "gix", + "KeySchema": [{"AttributeName": "g", "KeyType": "HASH"}], + "Projection": {"ProjectionType": "KEYS_ONLY"} + }], + "LocalSecondaryIndexes": [{ + "IndexName": "lix", + "KeySchema": [ + {"AttributeName": "pk", "KeyType": "HASH"}, + {"AttributeName": "l", "KeyType": "RANGE"} + ], + "Projection": {"ProjectionType": "ALL"} + }] + })) + .expect("input"); + if vector { + input.vector_indexes = Some(vec![VectorIndexSpecification { + index_name: "vix".to_owned(), + dimensions: 3, + distance_function: DistanceFunction::Cosine, + vector_attribute: VectorAttribute { + attribute_name: "emb".to_owned(), + }, + search_schema: None, + projection: Some(Projection { + projection_type: ProjectionType::All, + non_key_attributes: None, + }), + }]); + } + input +} + +/// Item `i` of the fixture: every item has `pk` and `sk`, every second one +/// `g` (the GSI's key), every third one `l` (the LSI's), every fifth one +/// `emb` (the vector). +fn item(i: usize) -> Item { + let mut item = + serde_json::json!({"pk": {"S": format!("p{}", i % 7)}, "sk": {"N": i.to_string()}}); + if i.is_multiple_of(2) { + item["g"] = serde_json::json!({"S": format!("g{}", i % 4)}); + } + if i.is_multiple_of(3) { + item["l"] = serde_json::json!({"S": format!("l{i}")}); + } + if i.is_multiple_of(5) { + item["emb"] = serde_json::json!({"L": [{"N": "0.1"}, {"N": "0.2"}, {"N": "0.3"}]}); + } + serde_json::from_value(item).expect("item") +} + +fn items(range: std::ops::Range) -> Vec { + range.map(item).collect() +} + +fn stream(items: Vec) -> futures::stream::BoxStream<'static, Result> { + Box::pin(futures::stream::iter(items.into_iter().map(Ok))) +} + +fn key_of(desc: &TableDescription) -> RestoreTargetKey { + RestoreTargetKey { + account_id: ACCOUNT.to_owned(), + table_name: desc.table_name.clone(), + table_id: desc.table_id.clone(), + } +} + +/// Register a restore of `row` into `target` and return the description, its +/// key and the claim. +async fn register( + s: &Scratch, + row: &BackupRow, + target: &str, + spool_host: Option<&str>, +) -> (TableDescription, RestoreTargetKey, Box) { + let (desc, claim) = s + .engine + .claim_restore( + ACCOUNT, + target_input(target, s.vector), + ®istration(row, spool_host), + ) + .await + .expect("register the restore"); + let key = key_of(&desc); + (desc, key, claim) +} + +// --- reads -------------------------------------------------------------------- + +async fn table_status(s: &Scratch, name: &str) -> Option { + sqlx::query_scalar("SELECT table_status FROM tables WHERE account_id = $1 AND table_name = $2") + .bind(ACCOUNT) + .bind(name) + .fetch_optional(&s.catalog) + .await + .expect("table status") +} + +async fn record(s: &Scratch, key: &RestoreTargetKey) -> RestoreJobRow { + s.engine + .read_restore_job(key) + .await + .expect("read the record") + .expect("the record exists") +} + +/// `(status, kind, owner_generation, heartbeat_at, spool_host, completed_at)` +/// of the record, straight from the table. +async fn record_columns( + s: &Scratch, + table_id: &str, +) -> (String, String, i64, f64, Option, Option) { + sqlx::query_as( + "SELECT status, kind, owner_generation, EXTRACT(EPOCH FROM heartbeat_at)::FLOAT8, \ + spool_host, EXTRACT(EPOCH FROM completed_at)::FLOAT8 FROM table_restores \ + WHERE table_id = $1", + ) + .bind(table_id) + .fetch_one(&s.catalog) + .await + .expect("read table_restores") +} + +async fn rows_in(db: &PgPool, physical: &str) -> i64 { + sqlx::query_scalar(&format!("SELECT count(*) FROM {physical}")) + .fetch_one(db) + .await + .expect("count rows") +} + +async fn base_rows(s: &Scratch, table_id: &str) -> i64 { + rows_in(&s.db, &format!("\"_ddb_{table_id}\"")).await +} + +/// `(index_name, index_id, index_status)` of the table's secondary indexes. +async fn indexes(s: &Scratch, table_id: &str) -> Vec<(String, String, String)> { + sqlx::query_as( + "SELECT index_name, index_id, index_status FROM indexes WHERE table_id = $1 \ + ORDER BY index_name", + ) + .bind(table_id) + .fetch_all(&s.catalog) + .await + .expect("read indexes") +} + +async fn index_rows(s: &Scratch, table_id: &str, name: &str) -> i64 { + let id = indexes(s, table_id) + .await + .into_iter() + .find(|(n, _, _)| n == name) + .map(|(_, id, _)| id) + .expect("the index exists"); + rows_in(&s.db, &format!("\"_ddb_{id}\"")).await +} + +/// `(index_id, index_status)` of the table's vector indexes. +async fn vector_indexes(s: &Scratch, table_id: &str) -> Vec<(String, String)> { + sqlx::query_as("SELECT index_id, index_status FROM vector_indexes WHERE table_id = $1") + .bind(table_id) + .fetch_all(&s.catalog) + .await + .expect("read vector_indexes") +} + +async fn vector_rows(s: &Scratch, table_id: &str) -> i64 { + let (id, _) = vector_indexes(s, table_id) + .await + .pop() + .expect("the vector index exists"); + rows_in(&s.db, &format!("\"_ddb_vec_{id}\"")).await +} + +/// Whether the data database has a relation named `relation` (the physical +/// name without quotes), by `to_regclass`. +async fn relation_exists(s: &Scratch, relation: &str) -> bool { + let regclass: Option = sqlx::query_scalar("SELECT to_regclass($1)::TEXT") + .bind(format!("\"{relation}\"")) + .fetch_one(&s.db) + .await + .expect("probe the relation"); + regclass.is_some() +} + +async fn progress_rows(s: &Scratch, table_id: &str) -> Vec<(i64, i64)> { + sqlx::query_as( + "SELECT file_index, items_applied FROM restore_progress WHERE table_id = $1 \ + ORDER BY file_index", + ) + .bind(table_id) + .fetch_all(&s.db) + .await + .expect("read restore_progress") +} + +/// The backend pid holding the restore lock on `table`, if any. +async fn lock_holder(s: &Scratch, table: &str) -> Option { + sqlx::query_scalar( + "SELECT pid FROM pg_locks WHERE locktype = 'advisory' AND granted \ + AND database = (SELECT oid FROM pg_database WHERE datname = current_database()) \ + AND ((classid::bigint << 32) | objid::bigint) = hashtextextended($1, $2)", + ) + .bind(format!("{ACCOUNT}/{table}")) + .bind(RESTORE_LOCK_SEED) + .fetch_optional(&s.catalog) + .await + .expect("find the advisory lock holder") +} + +/// Date the record's heartbeat `age` into the past by hand. This proves the +/// takeover's predicate on a stale row, not the window boundary on the +/// database's clock, which `abandoned_restore_is_removed_only_when_its_heartbeat_is_stale` +/// in `backup_restore.rs` crosses. +async fn age_heartbeat(s: &Scratch, table_id: &str, age: Duration) -> f64 { + sqlx::query_scalar( + "UPDATE table_restores SET heartbeat_at = now() - make_interval(secs => $2) \ + WHERE table_id = $1 RETURNING EXTRACT(EPOCH FROM heartbeat_at)::FLOAT8", + ) + .bind(table_id) + .bind(age.as_secs_f64()) + .fetch_one(&s.catalog) + .await + .expect("age the heartbeat") +} + +// --- the registration ----------------------------------------------------------- + +/// `claim_restore` creates the CREATING target with no scheduled transition +/// and its RUNNING `manifest` record carrying the store, the spool host and +/// the claim's generation, in one transaction; the description carries the +/// `RestoreSummary`; the record reads back through `read_restore_job` and +/// `list_restore_jobs`; the claim holds the name's advisory lock and its +/// heartbeat confirms. +#[tokio::test] +async fn claim_restore_registers_the_target_with_its_record() { + if !pg_available("claim_restore_registers_the_target_with_its_record") { + return; + } + let s = scratch().await; + let row = available_backup(&s, "Music").await; + assert_eq!(lock_holder(&s, "Restored").await, None); + let (desc, key, claim) = register(&s, &row, "Restored", Some("host-a:/spool")).await; + assert_eq!(desc.table_status, TableStatus::Creating); + let summary = desc.restore_summary.as_ref().expect("a RestoreSummary"); + assert_eq!( + summary.source_backup_arn.as_deref(), + Some(row.backup_arn.as_str()) + ); + assert_eq!( + summary.source_table_arn.as_deref(), + Some(format!("arn:aws:dynamodb:{REGION}:{ACCOUNT}:table/Music").as_str()) + ); + assert_eq!(summary.restore_date_time, row.created_at); + assert!(summary.restore_in_progress); + assert!( + lock_holder(&s, "Restored").await.is_some(), + "the claim holds the name lock" + ); + + let (status, kind, generation, heartbeat_at, spool_host, completed_at) = + record_columns(&s, &key.table_id).await; + assert_eq!((status.as_str(), kind.as_str()), ("RUNNING", "manifest")); + assert_eq!(generation, i64::try_from(claim.generation()).unwrap()); + assert!(heartbeat_at > 0.0); + assert_eq!(spool_host.as_deref(), Some("host-a:/spool")); + assert_eq!(completed_at, None); + let transition: Option = sqlx::query_scalar( + "SELECT EXTRACT(EPOCH FROM status_transition_at)::FLOAT8 FROM tables WHERE table_id = $1", + ) + .bind(&key.table_id) + .fetch_one(&s.catalog) + .await + .expect("read the transition"); + assert_eq!( + transition, None, + "no scheduled transition: the job flips the table" + ); + + let job = record(&s, &key).await; + assert_eq!(job.target, key); + assert_eq!(job.source_backup_arn, row.backup_arn); + assert_eq!(job.kind, RestoreKind::Manifest); + assert_eq!(job.status, JobStatus::Running); + assert_eq!(job.owner_generation, claim.generation()); + assert_eq!(job.store, Some(store())); + assert_eq!(job.spool_host.as_deref(), Some("host-a:/spool")); + assert_eq!((job.file_index, job.items_applied), (None, None)); + assert_eq!( + (job.failure_reason.as_deref(), job.completed_at), + (None, None) + ); + let running = s + .engine + .list_restore_jobs(Some(JobStatus::Running)) + .await + .expect("list"); + assert_eq!(running, vec![job.clone()]); + assert!( + s.engine + .list_restore_jobs(Some(JobStatus::Done)) + .await + .expect("list") + .is_empty() + ); + assert_eq!( + s.engine.backup_in_use(&row.key()).await.expect("in use"), + Some("Restored".to_owned()) + ); + assert_eq!(claim.heartbeat().await.expect("beat"), Heartbeat::Confirmed); + let heartbeat_now = record_columns(&s, &key.table_id).await.3; + assert!( + heartbeat_now >= heartbeat_at, + "the heartbeat refreshed the row" + ); + + // The indexes a restore target is created with are ACTIVE from the start + // (the flip has nothing to move), and the GSI and LSI data tables exist. + let index_rows = indexes(&s, &key.table_id).await; + assert_eq!(index_rows.len(), 2); + assert!(index_rows.iter().all(|(_, _, status)| status == "ACTIVE")); + assert_eq!(base_rows(&s, &key.table_id).await, 0); + drop(claim); + s.cleanup().await; +} + +/// The ordering DeleteBackup wins: with `start_backup_delete` holding the +/// `backup_index` row FOR UPDATE, a registration waits behind it and, once +/// the row is DELETING, answers `Backup not found` and creates no target. +#[tokio::test] +async fn delete_backup_winning_before_the_registration_leaves_no_target() { + if !pg_available("delete_backup_winning_before_the_registration_leaves_no_target") { + return; + } + let s = scratch().await; + let row = available_backup(&s, "Music").await; + // `start_backup_delete` locks the row FOR UPDATE and then reads + // `table_restores`; a gate on that table holds it with the row lock + // taken. + let mut gate = s.catalog.begin().await.expect("begin the gate"); + sqlx::query("LOCK TABLE table_restores IN ACCESS EXCLUSIVE MODE") + .execute(&mut *gate) + .await + .expect("hold DeleteBackup after its row lock"); + let key = row.key(); + let mut delete = Box::pin(s.engine.start_backup_delete(&key)); + assert!( + tokio::time::timeout(Duration::from_secs(2), &mut delete) + .await + .is_err(), + "DeleteBackup did not reach the gate" + ); + let delete_waits: bool = sqlx::query_scalar( + "SELECT EXISTS(SELECT 1 FROM pg_locks \ + WHERE relation = 'table_restores'::regclass AND NOT granted)", + ) + .fetch_one(&s.catalog) + .await + .expect("observe the blocked DeleteBackup"); + assert!( + delete_waits, + "DeleteBackup was not paused after its row lock" + ); + + let input = target_input("Late", s.vector); + let reg = registration(&row, None); + let mut restore = Box::pin(s.engine.claim_restore(ACCOUNT, input, ®)); + assert!( + tokio::time::timeout(Duration::from_millis(700), &mut restore) + .await + .is_err(), + "the registration did not wait for DeleteBackup's row lock" + ); + assert_eq!( + table_status(&s, "Late").await, + None, + "the registration created its target before the row lock" + ); + + gate.commit().await.expect("let DeleteBackup finish"); + assert_eq!( + delete.await.expect("DeleteBackup wins"), + DeleteStart::Deleting + ); + let err = restore + .await + .err() + .expect("the registration finds the backup gone"); + assert!( + extenddb_storage::backup_catalog::is_backup_not_found(&err), + "{err:?}" + ); + assert_eq!( + table_status(&s, "Late").await, + None, + "no target was created" + ); + assert_eq!( + lock_holder(&s, "Late").await, + None, + "the refused registration released the name lock" + ); + assert!( + s.engine + .list_restore_jobs(None) + .await + .expect("list") + .is_empty() + ); + s.cleanup().await; +} + +/// The other ordering: a registration that commits first leaves +/// `start_backup_delete` with `InUse` naming the target, and a second +/// registration of the same backup is `BackupInUse` while the target is +/// CREATING. +#[tokio::test] +async fn a_registration_winning_before_delete_backup_keeps_the_backup_in_use() { + if !pg_available("a_registration_winning_before_delete_backup_keeps_the_backup_in_use") { + return; + } + let s = scratch().await; + let row = available_backup(&s, "Music").await; + let (_desc, key, claim) = register(&s, &row, "First", None).await; + assert_eq!( + s.engine + .start_backup_delete(&row.key()) + .await + .expect("DeleteBackup answers"), + DeleteStart::InUse("First".to_owned()) + ); + let second = s + .engine + .claim_restore( + ACCOUNT, + target_input("Second", s.vector), + ®istration(&row, None), + ) + .await + .err() + .expect("a second restore of the backup is refused"); + assert!( + matches!(&second, StorageError::BackupInUse(m) if m.contains("First")), + "{second:?}" + ); + assert_eq!(table_status(&s, "Second").await, None); + assert_eq!(lock_holder(&s, "Second").await, None); + // The row stayed AVAILABLE through both refusals. + let status: String = sqlx::query_scalar("SELECT status FROM backup_index WHERE backup_id = $1") + .bind(&row.backup_id) + .fetch_one(&s.catalog) + .await + .expect("status"); + assert_eq!(status, "AVAILABLE"); + drop(claim); + let _ = key; + s.cleanup().await; +} + +/// A taken name: an ACTIVE table of the name is `TableAlreadyExists`; a +/// CREATING restore target of the name is `TableBeingRestored` naming its +/// source backup, whether the first registration's lock is still held or +/// its owner is gone. +#[tokio::test] +async fn a_taken_name_is_refused_as_create_table_or_as_a_restore_in_progress() { + if !pg_available("a_taken_name_is_refused_as_create_table_or_as_a_restore_in_progress") { + return; + } + let s = scratch().await; + let row = available_backup(&s, "Music").await; + s.engine + .create_table(ACCOUNT, target_input("Taken", false)) + .await + .expect("create a table of the name"); + let err = s + .engine + .claim_restore( + ACCOUNT, + target_input("Taken", s.vector), + ®istration(&row, None), + ) + .await + .err() + .expect("refused"); + assert!( + matches!(&err, StorageError::TableAlreadyExists(n) if n == "Taken"), + "{err:?}" + ); + + let (_desc, _key, claim) = register(&s, &row, "Restoring", None).await; + let other = available_backup(&s, "Other").await; + // With the first claim's lock held, the name lock refuses first. + let err = s + .engine + .claim_restore( + ACCOUNT, + target_input("Restoring", s.vector), + ®istration(&other, None), + ) + .await + .err() + .expect("refused"); + assert!( + matches!(&err, StorageError::TableBeingRestored { table_name, backup_arn } + if table_name == "Restoring" && *backup_arn == row.backup_arn), + "{err:?}" + ); + // With the lock gone (a dead owner), the unique index refuses the same way. + drop(claim); + let deadline = tokio::time::Instant::now() + Duration::from_secs(10); + while lock_holder(&s, "Restoring").await.is_some() { + assert!( + tokio::time::Instant::now() < deadline, + "the lock session did not end" + ); + tokio::time::sleep(Duration::from_millis(20)).await; + } + let err = s + .engine + .claim_restore( + ACCOUNT, + target_input("Restoring", s.vector), + ®istration(&other, None), + ) + .await + .err() + .expect("refused"); + assert!( + matches!(&err, StorageError::TableBeingRestored { table_name, .. } if table_name == "Restoring"), + "{err:?}" + ); + s.cleanup().await; +} + +// --- the takeover ------------------------------------------------------------- + +/// `claim_restore_dead` is refused while the owner's lock session lives and +/// while the heartbeat is young, and taken once both are gone: a fresh +/// generation, the claimant's `spool_host` and a refreshed `heartbeat_at` +/// (the database's clock) in one write, so a second taker finds the record +/// live at once; `list_dead_restore_jobs` lists the record with its members +/// only once stale; the old generation's writes are `NotMatched`. +#[tokio::test] +async fn a_dead_record_is_taken_over_with_the_claimants_spool_and_a_fresh_heartbeat() { + if !pg_available("a_dead_record_is_taken_over_with_the_claimants_spool_and_a_fresh_heartbeat") { + return; + } + let s = scratch().await; + let row = available_backup(&s, "Music").await; + let (_desc, key, claim) = register(&s, &row, "Target", Some("host-a:/spool")).await; + let old_generation = claim.generation(); + // A live owner: the lock is held. + assert!( + s.engine + .claim_restore_dead(&key, STALE, Some("host-b:/spool")) + .await + .expect("takeover") + .is_none(), + "a held lock is a live owner" + ); + // The lock gone (the session ended) but the heartbeat young. + let holder = lock_holder(&s, "Target") + .await + .expect("the owner's session"); + sqlx::query("SELECT pg_terminate_backend($1)") + .bind(holder) + .execute(&s.catalog) + .await + .expect("end the owner's session"); + let deadline = tokio::time::Instant::now() + Duration::from_secs(10); + while lock_holder(&s, "Target").await.is_some() { + assert!( + tokio::time::Instant::now() < deadline, + "the lock did not go" + ); + tokio::time::sleep(Duration::from_millis(20)).await; + } + // The owner's next heartbeat is lost: its session is gone. + assert!( + !matches!(claim.heartbeat().await, Ok(Heartbeat::Confirmed)), + "a terminated claim session cannot confirm" + ); + assert!( + s.engine + .claim_restore_dead(&key, STALE, Some("host-b:/spool")) + .await + .expect("takeover") + .is_none(), + "a young heartbeat is a live owner" + ); + assert!( + s.engine + .list_dead_restore_jobs(STALE) + .await + .expect("list") + .is_empty(), + "a young record is not listed" + ); + assert_eq!( + record(&s, &key).await.spool_host.as_deref(), + Some("host-a:/spool") + ); + + // Stale: listed with its members, and taken over. + let aged = age_heartbeat(&s, &key.table_id, Duration::from_secs(3600)).await; + let dead = s.engine.list_dead_restore_jobs(STALE).await.expect("list"); + assert_eq!(dead.len(), 1); + assert_eq!(dead[0].target, key); + assert_eq!(dead[0].kind, RestoreKind::Manifest); + assert_eq!(dead[0].status, JobStatus::Running); + assert_eq!(dead[0].store, Some(store())); + assert_eq!(dead[0].spool_host.as_deref(), Some("host-a:/spool")); + assert_eq!(dead[0].owner_generation, old_generation); + let successor = s + .engine + .claim_restore_dead(&key, STALE, Some("host-b:/spool")) + .await + .expect("takeover") + .expect("a stale record with no lock is dead"); + assert_ne!(successor.generation(), old_generation); + let (status, _, generation, heartbeat_at, spool_host, _) = + record_columns(&s, &key.table_id).await; + assert_eq!(status, "RUNNING"); + assert_eq!(generation, i64::try_from(successor.generation()).unwrap()); + assert_eq!( + spool_host.as_deref(), + Some("host-b:/spool"), + "the claimant's spool" + ); + assert!( + heartbeat_at > aged + 3_000.0, + "the takeover refreshed heartbeat_at: {heartbeat_at} vs the aged {aged}" + ); + let db_now: f64 = sqlx::query_scalar("SELECT EXTRACT(EPOCH FROM now())::FLOAT8") + .fetch_one(&s.catalog) + .await + .expect("the database's clock"); + assert!( + (db_now - heartbeat_at).abs() < 5.0, + "the refreshed heartbeat is the database's now: {heartbeat_at} vs {db_now}" + ); + assert!( + lock_holder(&s, "Target").await.is_some(), + "the successor holds the lock" + ); + // A second taker is refused while the successor lives: the lock, and + // the fresh heartbeat (the window a second instance's reconciler would + // otherwise find the record stale in). + assert!( + s.engine + .claim_restore_dead(&key, STALE, Some("host-c:/spool")) + .await + .expect("takeover") + .is_none() + ); + assert!( + s.engine + .list_dead_restore_jobs(STALE) + .await + .expect("list") + .is_empty(), + "the taken-over record is live to the listing at once" + ); + // The old generation's writes match nothing: the flip through the old + // claim, and the old claim's status write (its session is gone, so the + // write's outcome is unknown and the re-read shows the successor). + assert!(matches!( + s.engine + .finish_restore(&key, &*claim, 0, 0) + .await + .expect("finish"), + Finished::NotMatched(_) + )); + assert_eq!(record(&s, &key).await.status, JobStatus::Running); + let old_write = claim + .set_status( + JobStatus::Running, + JobStatus::Failed, + &StatusFields { + failure_reason: Some("stale".to_owned()), + completed: true, + ..StatusFields::default() + }, + ) + .await; + assert!( + matches!(old_write, Ok(WriteOutcome::Unknown) | Err(_)), + "{old_write:?}" + ); + let reread = claim.read().await.expect("the pool re-read"); + assert_eq!(reread.owner_generation, successor.generation()); + assert_eq!(reread.status, JobStatus::Running); + // The successor's heartbeat confirms and its status write matches; after + // its FAILED write a second write from it is NotMatched (not RUNNING). + assert_eq!( + successor.heartbeat().await.expect("beat"), + Heartbeat::Confirmed + ); + assert_eq!( + successor + .set_status( + JobStatus::Running, + JobStatus::Failed, + &StatusFields { + failure_reason: Some("count".to_owned()), + completed: true, + ..StatusFields::default() + }, + ) + .await + .expect("write"), + WriteOutcome::Matched + ); + assert_eq!( + successor + .set_status( + JobStatus::Running, + JobStatus::Done, + &StatusFields::default() + ) + .await + .expect("write"), + WriteOutcome::NotMatched + ); + let job = record(&s, &key).await; + assert_eq!(job.status, JobStatus::Failed); + assert_eq!(job.failure_reason.as_deref(), Some("count")); + assert!(job.completed_at.is_some()); + assert!( + s.engine + .list_dead_restore_jobs(STALE) + .await + .expect("list") + .is_empty(), + "a fresh FAILED record is left to its job's removal" + ); + drop(successor); + s.cleanup().await; +} + +// --- the load ------------------------------------------------------------------- + +/// `bulk_load` writes the base rows, the GSI, LSI and vector rows and the +/// file's progress row in one data transaction, across the 500-row batch +/// boundary; `start_at` is added to the recorded total; a duplicate key +/// within a file or against an earlier file fails the file with nothing of +/// it committed, no progress row included; a file whose progress is already +/// recorded fails whole, its index rows included. +#[tokio::test] +async fn bulk_load_writes_each_file_with_its_indexes_and_progress_in_one_transaction() { + if !pg_available("bulk_load_writes_each_file_with_its_indexes_and_progress_in_one_transaction") + { + return; + } + let s = scratch().await; + let row = available_backup(&s, "Music").await; + let (_desc, key, claim) = register(&s, &row, "Loaded", None).await; + + // File 0: 1200 items, past the 500-row batch. + let applied = s + .engine + .bulk_load( + &key, + &*claim, + BulkLoadFile { + file_index: 0, + start_at: 0, + reapplying: false, + }, + stream(items(0..1200)), + ) + .await + .expect("load file 0"); + assert_eq!(applied, 1200); + assert_eq!(base_rows(&s, &key.table_id).await, 1200); + assert_eq!( + index_rows(&s, &key.table_id, "gix").await, + 600, + "every second item has g" + ); + assert_eq!( + index_rows(&s, &key.table_id, "lix").await, + 400, + "every third item has l" + ); + if s.vector { + assert_eq!( + vector_rows(&s, &key.table_id).await, + 240, + "every fifth item has emb" + ); + } + assert_eq!(progress_rows(&s, &key.table_id).await, vec![(0, 1200)]); + assert_eq!( + s.engine.count_restore_target(&key).await.expect("count"), + 1200 + ); + + // File 1 resumed at item 5 of its 8: the record carries the running total. + let applied = s + .engine + .bulk_load( + &key, + &*claim, + BulkLoadFile { + file_index: 1, + start_at: 5, + reapplying: false, + }, + stream(items(1205..1208)), + ) + .await + .expect("load file 1"); + assert_eq!(applied, 3); + assert_eq!( + progress_rows(&s, &key.table_id).await, + vec![(0, 1200), (1, 8)] + ); + assert_eq!(base_rows(&s, &key.table_id).await, 1203); + + // File 2 holds a key twice: nothing of it lands, no progress row. + let mut twice = items(2000..2010); + twice.push(item(2003)); + let err = s + .engine + .bulk_load( + &key, + &*claim, + BulkLoadFile { + file_index: 2, + start_at: 0, + reapplying: false, + }, + stream(twice), + ) + .await + .expect_err("a duplicate key fails the file"); + assert!( + matches!(&err, StorageError::Validation(m) + if m.starts_with("holds a primary key the target already has") + && m.contains("the backup holds a key twice")), + "the backend names no file (the engine prefixes the file's key): {err:?}" + ); + assert_eq!(base_rows(&s, &key.table_id).await, 1203); + assert_eq!( + progress_rows(&s, &key.table_id).await, + vec![(0, 1200), (1, 8)] + ); + + // File 3 repeats a key file 0 wrote: the same. + let mut across = items(3000..3004); + across.push(item(7)); + let err = s + .engine + .bulk_load( + &key, + &*claim, + BulkLoadFile { + file_index: 3, + start_at: 0, + reapplying: false, + }, + stream(across), + ) + .await + .expect_err("a key of an earlier file fails the file"); + assert!( + matches!(&err, StorageError::Validation(m) + if m.starts_with("holds a primary key the target already has")), + "{err:?}" + ); + assert_eq!(base_rows(&s, &key.table_id).await, 1203); + assert_eq!( + index_rows(&s, &key.table_id, "gix").await, + 600 + 1, + "1206 has g" + ); + assert_eq!( + progress_rows(&s, &key.table_id).await, + vec![(0, 1200), (1, 8)] + ); + + // A decode or validation failure the engine found mid-stream: the items + // before it are not committed either. + let failing: futures::stream::BoxStream<'static, Result> = + Box::pin(futures::stream::iter( + items(4000..4003) + .into_iter() + .map(Ok) + .chain(std::iter::once(Err(StorageError::Validation( + "key {\"pk\": \"p1\"}, attribute n: 1E126 is not a number".to_owned(), + )))), + )); + let err = s + .engine + .bulk_load( + &key, + &*claim, + BulkLoadFile { + file_index: 4, + start_at: 0, + reapplying: false, + }, + failing, + ) + .await + .expect_err("an Err item fails the file"); + assert!( + matches!(&err, StorageError::Validation(m) if m.contains("1E126")), + "{err:?}" + ); + assert_eq!(base_rows(&s, &key.table_id).await, 1203); + assert_eq!( + progress_rows(&s, &key.table_id).await, + vec![(0, 1200), (1, 8)] + ); + + // File 0 again with keys the target lacks: its rows and index rows land + // in the file's transaction, and the progress row's primary key rolls + // the whole file back, index rows included. + let err = s + .engine + .bulk_load( + &key, + &*claim, + BulkLoadFile { + file_index: 0, + start_at: 0, + reapplying: false, + }, + stream(items(5000..5004)), + ) + .await + .expect_err("a recorded file is not applied twice"); + assert!( + matches!(&err, StorageError::Validation(m) if m.contains("data file 0")), + "{err:?}" + ); + assert_eq!(base_rows(&s, &key.table_id).await, 1203); + assert_eq!(index_rows(&s, &key.table_id, "gix").await, 601); + assert_eq!( + index_rows(&s, &key.table_id, "lix").await, + 400 + 1, + "1206 has l" + ); + if s.vector { + assert_eq!( + vector_rows(&s, &key.table_id).await, + 240 + 1, + "1205 has emb" + ); + } + assert_eq!( + progress_rows(&s, &key.table_id).await, + vec![(0, 1200), (1, 8)] + ); + + // File 1's row says 8 of its items were applied and the file holds 11 + // (the decoder died after the file's commit and the job's FAILED write + // never landed): the resume starts the file at 8, and the rest completes + // it, the row updated in place instead of refused on its key. + let applied = s + .engine + .bulk_load( + &key, + &*claim, + BulkLoadFile { + file_index: 1, + start_at: 8, + reapplying: false, + }, + stream(items(1208..1211)), + ) + .await + .expect("a file resumed from a short progress row completes"); + assert_eq!(applied, 3); + assert_eq!( + progress_rows(&s, &key.table_id).await, + vec![(0, 1200), (1, 11)], + "the short row is updated to the file's total" + ); + assert_eq!(base_rows(&s, &key.table_id).await, 1206); + assert_eq!(index_rows(&s, &key.table_id, "gix").await, 603); + assert_eq!(index_rows(&s, &key.table_id, "lix").await, 402); + if s.vector { + assert_eq!(vector_rows(&s, &key.table_id).await, 242); + } + + // Another account's key, or a wrong table id, reaches no table. + let foreign = RestoreTargetKey { + account_id: "210987654321".to_owned(), + ..key.clone() + }; + assert!(matches!( + s.engine.count_restore_target(&foreign).await, + Err(StorageError::TableNotFound(_)) + )); + assert!(matches!( + s.engine + .bulk_load( + &foreign, + &*claim, + BulkLoadFile { + file_index: 9, + start_at: 0, + reapplying: false, + }, + stream(items(9000..9001)), + ) + .await, + Err(StorageError::TableNotFound(_)) + )); + drop(claim); + s.cleanup().await; +} + +/// `finish_restore` marks the record DONE and the table and every index +/// ACTIVE in one transaction, conditioned on the claim's generation and on +/// CREATING, then removes the progress rows; a second call, or one under a +/// stale generation, is `NotMatched` and writes nothing. The progress sweep +/// keeps a RUNNING target's rows and removes any other's; +/// `remove_restore_target` deletes a target's rows. +#[tokio::test] +async fn finish_restore_flips_the_table_and_every_index_and_clears_the_progress() { + if !pg_available("finish_restore_flips_the_table_and_every_index_and_clears_the_progress") { + return; + } + let s = scratch().await; + let row = available_backup(&s, "Music").await; + let (_desc, key, claim) = register(&s, &row, "Done", None).await; + let other_row = available_backup(&s, "Other").await; + let (_desc2, other, other_claim) = register(&s, &other_row, "Still", None).await; + for (k, c) in [(&key, &claim), (&other, &other_claim)] { + s.engine + .bulk_load( + k, + &**c, + BulkLoadFile { + file_index: 0, + start_at: 0, + reapplying: false, + }, + stream(items(0..30)), + ) + .await + .expect("load"); + } + // An index a backend left CREATING (none does on this backend; the flip + // states the rule once) is moved with the table. + sqlx::query( + "UPDATE indexes SET index_status = 'CREATING' WHERE table_id = $1 AND index_name = 'gix'", + ) + .bind(&key.table_id) + .execute(&s.catalog) + .await + .expect("set an index CREATING"); + if s.vector { + sqlx::query("UPDATE vector_indexes SET index_status = 'CREATING' WHERE table_id = $1") + .bind(&key.table_id) + .execute(&s.catalog) + .await + .expect("set the vector index CREATING"); + } + + // The sweep: `Still`'s rows stay while it is RUNNING; a row of a table + // that is gone goes. + sqlx::query( + "INSERT INTO restore_progress (table_id, file_index, items_applied) VALUES ('gone', 0, 5)", + ) + .execute(&s.db) + .await + .expect("an orphan row"); + // A caller whose RUNNING list predates both registrations (the list is + // read before the sweep; a registration and its first file can land in + // between): the sweep protects the RUNNING records it reads itself after + // the candidates, so neither live row goes, only the orphan. + assert_eq!( + s.engine.sweep_restore_progress(&[]).await.expect("sweep"), + 1, + "the orphan row alone, with an empty caller list" + ); + assert_eq!(progress_rows(&s, &key.table_id).await, vec![(0, 30)]); + assert_eq!(progress_rows(&s, &other.table_id).await, vec![(0, 30)]); + sqlx::query( + "INSERT INTO restore_progress (table_id, file_index, items_applied) VALUES ('gone', 0, 5)", + ) + .execute(&s.db) + .await + .expect("the orphan row again"); + let running: Vec = s + .engine + .list_restore_jobs(Some(JobStatus::Running)) + .await + .expect("list") + .into_iter() + .map(|j| j.target.table_id) + .collect(); + assert_eq!(running.len(), 2); + assert_eq!( + s.engine + .sweep_restore_progress(&running) + .await + .expect("sweep"), + 1 + ); + assert_eq!(progress_rows(&s, &key.table_id).await, vec![(0, 30)]); + assert_eq!(progress_rows(&s, &other.table_id).await, vec![(0, 30)]); + assert_eq!( + s.engine.read_restore_progress(&key).await.expect("read"), + vec![(0, 30)] + ); + + // A stale generation flips nothing. + let stale_row = available_backup(&s, "Stale").await; + let (_d, stale_key, stale_claim) = register(&s, &stale_row, "StaleOwner", None).await; + let _ = stale_key; + assert!(matches!( + s.engine + .finish_restore(&key, &*stale_claim, 30, 4096) + .await + .expect("finish"), + Finished::NotMatched(_) + )); + assert_eq!(table_status(&s, "Done").await.as_deref(), Some("CREATING")); + assert_eq!(record(&s, &key).await.status, JobStatus::Running); + assert_eq!(progress_rows(&s, &key.table_id).await, vec![(0, 30)]); + drop(stale_claim); + + assert_eq!( + s.engine + .finish_restore(&key, &*claim, 30, 4096) + .await + .expect("finish"), + Finished::Active + ); + assert_eq!(table_status(&s, "Done").await.as_deref(), Some("ACTIVE")); + let (status, _, _, _, _, completed_at) = record_columns(&s, &key.table_id).await; + assert_eq!(status, "DONE"); + assert!(completed_at.is_some()); + let counts: (i64, i64) = + sqlx::query_as("SELECT item_count, table_size_bytes FROM tables WHERE table_id = $1") + .bind(&key.table_id) + .fetch_one(&s.catalog) + .await + .expect("counts"); + assert_eq!(counts, (30, 4096)); + assert!( + indexes(&s, &key.table_id) + .await + .iter() + .all(|(_, _, status)| status == "ACTIVE"), + "every secondary index is ACTIVE" + ); + if s.vector { + assert!( + vector_indexes(&s, &key.table_id) + .await + .iter() + .all(|(_, status)| status == "ACTIVE"), + "the vector index is ACTIVE" + ); + } + assert!( + progress_rows(&s, &key.table_id).await.is_empty(), + "the progress rows are gone" + ); + assert_eq!(progress_rows(&s, &other.table_id).await, vec![(0, 30)]); + let job = record(&s, &key).await; + assert_eq!(job.status, JobStatus::Done); + assert!(job.completed_at.is_some()); + // The job record's heartbeat is no longer refreshed: not RUNNING. + assert_eq!(claim.heartbeat().await.expect("beat"), Heartbeat::Lost); + // A second flip matches nothing. + assert!(matches!( + s.engine + .finish_restore(&key, &*claim, 30, 4096) + .await + .expect("finish"), + Finished::NotMatched(_) + )); + // The restored table reads through the data plane as any ACTIVE table. + let desc = s + .engine + .describe_table( + ACCOUNT, + extenddb_core::types::DescribeTableInput { + table_name: "Done".to_owned(), + }, + ) + .await + .expect("describe"); + assert_eq!(desc.table_status, TableStatus::Active); + assert_eq!(desc.item_count, 30); + + // `remove_restore_target` on the other target deletes its progress rows + // with the target. + assert!( + s.engine + .remove_restore_target(&other) + .await + .expect("remove") + ); + assert!(progress_rows(&s, &other.table_id).await.is_empty()); + assert_eq!(table_status(&s, "Still").await, None); + drop(other_claim); + drop(claim); + s.cleanup().await; +} + +/// `remove_restore_target` of a target with a GSI, an LSI and (with pgvector) +/// a vector index drops every relation the target's rows named: the data +/// table, each secondary index's table and the vector index's `_ddb_vec_` +/// table, so a failed vector restore leaves no relation holding its items +/// with no catalog row naming it. +#[tokio::test] +async fn remove_restore_target_drops_the_vector_relation_with_the_data_table() { + if !pg_available("remove_restore_target_drops_the_vector_relation_with_the_data_table") { + return; + } + let s = scratch().await; + let row = available_backup(&s, "Music").await; + let (_desc, key, claim) = register(&s, &row, "Vectors", None).await; + s.engine + .bulk_load( + &key, + &*claim, + BulkLoadFile { + file_index: 0, + start_at: 0, + reapplying: false, + }, + stream(items(0..30)), + ) + .await + .expect("load"); + let mut relations: Vec = vec![format!("_ddb_{}", key.table_id)]; + relations.extend( + indexes(&s, &key.table_id) + .await + .into_iter() + .map(|(_, id, _)| format!("_ddb_{id}")), + ); + if s.vector { + assert_eq!( + vector_rows(&s, &key.table_id).await, + 6, + "every fifth item has emb" + ); + relations.extend( + vector_indexes(&s, &key.table_id) + .await + .into_iter() + .map(|(id, _)| format!("_ddb_vec_{id}")), + ); + assert_eq!( + relations.len(), + 4, + "the data table, two index tables, one vector table" + ); + } + for relation in &relations { + assert!( + relation_exists(&s, relation).await, + "{relation} exists before the removal" + ); + } + + assert!(s.engine.remove_restore_target(&key).await.expect("remove")); + assert_eq!(table_status(&s, "Vectors").await, None); + if s.vector { + assert!(vector_indexes(&s, &key.table_id).await.is_empty()); + } + for relation in &relations { + assert!( + !relation_exists(&s, relation).await, + "{relation} is dropped with the target" + ); + } + drop(claim); + s.cleanup().await; +} + +/// A `finish_restore` whose progress delete fails after the flip committed +/// (the data database cannot run the `DELETE`: the `restore_progress` table +/// is renamed under it) is `Active`: the table is ACTIVE and the record DONE, +/// the failure is logged and not returned, the rows are left, and the +/// reconciler's `sweep_restore_progress` removes them once the record is no +/// longer RUNNING. +#[tokio::test] +async fn a_progress_delete_failing_after_the_flip_leaves_the_restore_finished() { + if !pg_available("a_progress_delete_failing_after_the_flip_leaves_the_restore_finished") { + return; + } + let s = scratch().await; + let row = available_backup(&s, "Music").await; + let (_desc, key, claim) = register(&s, &row, "Flipped", None).await; + s.engine + .bulk_load( + &key, + &*claim, + BulkLoadFile { + file_index: 0, + start_at: 0, + reapplying: false, + }, + stream(items(0..30)), + ) + .await + .expect("load"); + assert_eq!(progress_rows(&s, &key.table_id).await, vec![(0, 30)]); + + sqlx::query("ALTER TABLE restore_progress RENAME TO restore_progress_away") + .execute(&s.db) + .await + .expect("take the progress table away"); + let finished = s.engine.finish_restore(&key, &*claim, 30, 4096).await; + sqlx::query("ALTER TABLE restore_progress_away RENAME TO restore_progress") + .execute(&s.db) + .await + .expect("bring the progress table back"); + assert!( + matches!(finished, Ok(Finished::Active)), + "a progress delete failing after the flip is not the restore's failure: {finished:?}" + ); + assert_eq!(table_status(&s, "Flipped").await.as_deref(), Some("ACTIVE")); + assert_eq!(record(&s, &key).await.status, JobStatus::Done); + assert_eq!( + progress_rows(&s, &key.table_id).await, + vec![(0, 30)], + "the rows the delete could not remove are left" + ); + assert_eq!( + s.engine.sweep_restore_progress(&[]).await.expect("sweep"), + 1, + "the reconciler's sweep removes the DONE target's rows" + ); + assert!(progress_rows(&s, &key.table_id).await.is_empty()); + drop(claim); + s.cleanup().await; +} + +/// Terminating the claim session: the next heartbeat is lost, the status +/// write's outcome is unknown, and the record stays RUNNING for the +/// reconciler once stale. +#[tokio::test] +async fn a_terminated_claim_session_loses_its_next_heartbeat() { + if !pg_available("a_terminated_claim_session_loses_its_next_heartbeat") { + return; + } + let s = scratch().await; + let row = available_backup(&s, "Music").await; + let (_desc, key, claim) = register(&s, &row, "Cut", None).await; + assert_eq!(claim.heartbeat().await.expect("beat"), Heartbeat::Confirmed); + let holder = lock_holder(&s, "Cut").await.expect("the claim session"); + sqlx::query("SELECT pg_terminate_backend($1)") + .bind(holder) + .execute(&s.catalog) + .await + .expect("terminate"); + let beat = claim.heartbeat().await; + assert!(!matches!(beat, Ok(Heartbeat::Confirmed)), "{beat:?}"); + let write = claim + .set_status( + JobStatus::Running, + JobStatus::Done, + &StatusFields::default(), + ) + .await; + assert!( + matches!(write, Ok(WriteOutcome::Unknown) | Err(_)), + "{write:?}" + ); + assert_eq!(record(&s, &key).await.status, JobStatus::Running); + assert_eq!(table_status(&s, "Cut").await.as_deref(), Some("CREATING")); + s.cleanup().await; +} + +/// A legacy restore record is not a takeover candidate: `claim_restore_dead` +/// leaves a stale `kind = legacy` record to the reconciler's legacy duty. +#[tokio::test] +async fn a_legacy_record_is_not_taken_over() { + if !pg_available("a_legacy_record_is_not_taken_over") { + return; + } + let s = scratch().await; + let row = available_backup(&s, "Music").await; + let (_desc, key, claim) = register(&s, &row, "Legacyish", None).await; + drop(claim); + let deadline = tokio::time::Instant::now() + Duration::from_secs(10); + while lock_holder(&s, "Legacyish").await.is_some() { + assert!(tokio::time::Instant::now() < deadline); + tokio::time::sleep(Duration::from_millis(20)).await; + } + sqlx::query("UPDATE table_restores SET kind = 'legacy' WHERE table_id = $1") + .bind(&key.table_id) + .execute(&s.catalog) + .await + .expect("make the record legacy"); + age_heartbeat(&s, &key.table_id, Duration::from_secs(3600)).await; + assert!( + s.engine + .claim_restore_dead(&key, STALE, None) + .await + .expect("takeover") + .is_none() + ); + assert_eq!(record(&s, &key).await.kind, RestoreKind::Legacy); + assert_eq!(lock_holder(&s, "Legacyish").await, None); + s.cleanup().await; +} diff --git a/crates/storage-sqlite/src/backup.rs b/crates/storage-sqlite/src/backup.rs index e7538553a..94b0e5005 100644 --- a/crates/storage-sqlite/src/backup.rs +++ b/crates/storage-sqlite/src/backup.rs @@ -73,9 +73,7 @@ use extenddb_storage::util::{composite_pk_to_text, parse_sk, sk_column_n}; use extenddb_storage::{BackupRestoreSource, JobPermit, LegacyBackupReader, RestoreRequest}; use futures::future::BoxFuture; -use crate::backup_catalog::{ - JobSlots, NOW, SqliteJobClaim, generation_column, legacy_backup_slot_key, -}; +use crate::backup_catalog::{JobSlots, SqliteJobClaim, generation_column, legacy_backup_slot_key}; use crate::create_table::RestoreRegistration; use crate::data::{ BoundValue, all_sort_key_info, data_table_name, index_table_name, insert_index_row_multi, @@ -440,19 +438,16 @@ impl SqliteEngine { // generation, and the table ACTIVE only from CREATING, which // the check above confirmed under the same lock. A record // another claim took over leaves the table to that owner. - let done = sqlx::query(&format!( - "UPDATE table_restores SET status = 'DONE', items_applied = ?, \ - completed_at = {NOW} \ - WHERE table_id = ? AND status = 'RUNNING' AND owner_generation = ?" - )) - .bind(count) - .bind(&desc.table_id) - .bind(generation_column(generation)) - .execute(&mut *tx) - .await - .map_err(db_err)? - .rows_affected(); - if done != 1 { + let activated = crate::restore_engine::activate_restore_target( + &mut tx, + &desc.table_id, + generation, + count, + table_size, + Some(count), + ) + .await?; + if !activated.record { tx.rollback().await.map_err(db_err)?; return Err(StorageError::TableNotFound(format!( "restore of {backup_arn} into {} did not complete: its job record \ @@ -460,17 +455,6 @@ impl SqliteEngine { desc.table_name ))); } - sqlx::query( - "UPDATE tables SET item_count = ?, table_size_bytes = ?, \ - table_status = 'ACTIVE', status_transition_at = NULL \ - WHERE table_id = ? AND table_status = 'CREATING'", - ) - .bind(count) - .bind(table_size) - .bind(&desc.table_id) - .execute(&mut *tx) - .await - .map_err(db_err)?; tx.commit().await.map_err(db_err)?; return Ok(count); }; @@ -3619,7 +3603,7 @@ mod restore_tests { #[tokio::test] async fn a_restart_ages_an_interrupted_restore_so_the_reconciler_removes_it() { use extenddb_storage::BackupCatalog; - use extenddb_storage::backup_catalog::{RestoreTargetKey, UNOWNED_GENERATION}; + use extenddb_storage::backup_catalog::{RestoreKind, RestoreTargetKey, UNOWNED_GENERATION}; let engine = engine().await; indexed_table(&engine, "src").await; let backup = engine @@ -3663,7 +3647,7 @@ mod restore_tests { .expire_restore_job_heartbeats() .await .expect("expire"); - assert_eq!(aged, vec!["dst".to_owned()]); + assert_eq!(aged, vec![("dst".to_owned(), RestoreKind::Legacy)]); let (heartbeat, record_generation): (String, i64) = sqlx::query_as( "SELECT heartbeat_at, owner_generation FROM table_restores WHERE table_id = ?", ) diff --git a/crates/storage-sqlite/src/backup_catalog.rs b/crates/storage-sqlite/src/backup_catalog.rs index 36c76bd9e..2a8cf3688 100644 --- a/crates/storage-sqlite/src/backup_catalog.rs +++ b/crates/storage-sqlite/src/backup_catalog.rs @@ -14,7 +14,12 @@ //! ([`BackupCatalog::claim_dead_backup`]) reads no stale window here, and //! every RUNNING restore job record of the stopped process is aged at startup //! so the reconciler's first cycle finds it -//! ([`SqliteEngine::expire_restore_job_heartbeats`]). +//! ([`SqliteEngine::expire_restore_job_heartbeats`]). A store-backed +//! restore's claim holds the slot of its target name +//! ([`restore_slot_key`]) the same way: the registration takes it before the +//! target exists, so two registrations into one name exclude each other in +//! this process, and a RUNNING `manifest` record whose slot no task holds is +//! taken over at once by [`BackupCatalog::claim_restore_dead`]. //! //! Every write a job makes to its row goes through its claim and carries the //! claim's `owner_generation`, so a row a successor took over matches no @@ -37,11 +42,13 @@ use std::sync::{Arc, Mutex}; use std::time::Duration; use extenddb_core::backup_format::BackupArn; +use extenddb_core::types::{CreateTableInput, RestoreSummary}; use extenddb_storage::JobPermit; use extenddb_storage::backup_catalog::{ BackupCatalog, BackupKey, BackupRow, BackupStatus, DeleteStart, Heartbeat, JobClaim, JobRow, - JobStatus, MaintenanceClaim, RestoreJobRow, RestoreKind, RestoreTargetKey, StaleClock, - StatusFields, StoreIdentity, StoreKind, TimeBounds, WriteOutcome, new_generation, + JobStatus, MaintenanceClaim, RestoreJobRow, RestoreKind, RestoreRegistered, + RestoreRegistrationRow, RestoreTargetKey, StaleClock, StatusFields, StoreIdentity, StoreKind, + TimeBounds, WriteOutcome, new_generation, }; use extenddb_storage::error::StorageError; use futures::future::BoxFuture; @@ -108,6 +115,14 @@ pub(crate) fn legacy_backup_slot_key(backup_arn: &str) -> String { format!("legacy-backup:{backup_arn}") } +/// The slot key of a store-backed restore's claim on its target name: taken +/// before the target exists, so two registrations into one name in this +/// process exclude each other before either reaches the catalog, and held +/// by the claim for the job's life. +pub(crate) fn restore_slot_key(account_id: &str, table_name: &str) -> String { + format!("restore:{account_id}/{table_name}") +} + /// An `owner_generation` as the catalog stores it (`INTEGER`); generations /// are minted below `i64::MAX`. pub(crate) fn generation_column(generation: u64) -> i64 { @@ -155,6 +170,27 @@ impl SqliteJobClaim { } } + /// A claim on a store-backed restore's job record, born with the + /// registration that writes `generation` on the row, holding the + /// target name's slot ([`restore_slot_key`]) for the job's life. The + /// engine's job holds the permit; none is held here. + pub(crate) fn for_manifest_restore( + engine: &SqliteEngine, + table_id: &str, + generation: u64, + slot: JobSlot, + ) -> Self { + Self { + engine: engine.clone(), + _slot: Some(slot), + _permit: None, + generation, + record: JobRecord::Restore { + table_id: table_id.to_owned(), + }, + } + } + /// Heartbeat the record every `period` until a heartbeat is not /// confirmed; returns why. Never returns while the claim is confirmed. pub(crate) async fn heartbeat_loop(&self, period: Duration) -> StorageError { @@ -263,15 +299,15 @@ impl JobClaim for SqliteJobClaim { .await } JobRecord::Restore { table_id } => { + // The record's progress (`file_index`, `items_applied`) + // is `bulk_load`'s write; a status write leaves it. sqlx::query(&format!( "UPDATE table_restores SET status = ?, \ - items_applied = COALESCE(?, items_applied), \ failure_reason = COALESCE(?, failure_reason), \ completed_at = CASE WHEN ? THEN {NOW} ELSE completed_at END \ WHERE table_id = ? AND owner_generation = ? AND status = ?" )) .bind(to.as_str()) - .bind(fields.item_count) .bind(fields.failure_reason.as_deref()) .bind(fields.completed) .bind(table_id) @@ -465,32 +501,117 @@ where ) } +/// The columns of one restore job record joined with its table, in the +/// order [`RestoreJobTuple`] reads them; every read of a record selects +/// them. +const RESTORE_ROW_COLUMNS: &str = "r.table_id, t.account_id, t.table_name, r.source_backup_arn, \ + r.kind, r.status, r.owner_generation, r.heartbeat_at, r.store_kind, r.store_root, \ + r.store_id, r.spool_host, r.file_index, r.items_applied, r.failure_reason, r.completed_at"; + +/// One restore job record as the driver reads it. +#[allow(clippy::type_complexity)] +type RestoreJobTuple = ( + String, + String, + String, + String, + String, + String, + i64, + String, + Option, + Option, + Option, + Option, + Option, + Option, + Option, + Option, +); + +fn restore_job_from_tuple(t: RestoreJobTuple) -> Result { + let ( + table_id, + account_id, + table_name, + source_backup_arn, + kind, + status, + owner_generation, + heartbeat_at, + store_kind, + store_root, + store_id, + spool_host, + file_index, + items_applied, + failure_reason, + completed_at, + ) = t; + let store = match (store_kind, store_root, store_id) { + (Some(kind), Some(store_root), Some(store_id)) => Some(StoreIdentity { + store_kind: StoreKind::parse(&kind).ok_or_else(|| { + StorageError::Internal(format!( + "restore of {source_backup_arn} carries an unknown store kind {kind:?}" + )) + })?, + store_root, + store_id, + }), + _ => None, + }; + Ok(RestoreJobRow { + target: RestoreTargetKey { + account_id, + table_name, + table_id, + }, + source_backup_arn, + kind: RestoreKind::parse(&kind)?, + status: JobStatus::parse(&status)?, + owner_generation: u64::try_from(owner_generation).unwrap_or_default(), + heartbeat_at: timestamp_epoch_seconds(&heartbeat_at), + store, + spool_host, + file_index: file_index.and_then(|i| u32::try_from(i).ok()), + items_applied: items_applied.and_then(|i| u64::try_from(i).ok()), + failure_reason, + completed_at: completed_at.as_deref().map(timestamp_epoch_seconds), + }) +} + impl SqliteEngine { /// Every RUNNING restore job record at startup was owned by the process /// that stopped: one process serves a file, so a restart is proof of /// death. Set each record's heartbeat to the epoch, so the reconciler's - /// first cycle fails it and removes its target without waiting the stale - /// window. Returns the names of the targets whose record was aged (one - /// already at the epoch is not reported again). Runs before the server - /// takes requests, so no restore of this process can be in flight. - pub(crate) async fn expire_restore_job_heartbeats(&self) -> Result, StorageError> { + /// first cycle acts on it (a store-backed record is taken over and + /// resumed, a legacy one failed and its target removed) without waiting + /// the stale window. Returns the name and kind of each target whose + /// record was aged (one already at the epoch is not reported again). + /// Runs before the server takes requests, so no restore of this process + /// can be in flight. + pub(crate) async fn expire_restore_job_heartbeats( + &self, + ) -> Result, StorageError> { let _writer = self.write_lock.lock().await; let mut tx = self .pool .begin_with("BEGIN IMMEDIATE") .await .map_err(db_err)?; - let names: Vec = sqlx::query_scalar( + let rows: Vec<(String, String)> = sqlx::query_as( "UPDATE table_restores SET heartbeat_at = '1970-01-01T00:00:00.000Z' \ WHERE status = 'RUNNING' AND heartbeat_at > '1970-01-01T00:00:00.000Z' \ RETURNING COALESCE((SELECT table_name FROM tables t \ - WHERE t.table_id = table_restores.table_id), table_id)", + WHERE t.table_id = table_restores.table_id), table_id), kind", ) .fetch_all(&mut *tx) .await .map_err(db_err)?; tx.commit().await.map_err(db_err)?; - Ok(names) + rows.into_iter() + .map(|(name, kind)| Ok((name, RestoreKind::parse(&kind)?))) + .collect() } /// Remove the maintenance claim, whoever holds it: the server's startup @@ -873,23 +994,11 @@ impl BackupCatalog for SqliteEngine { if !is_target { return Ok(false); } - // Progress rows belong to a later release's restore path; a - // catalog without that table has none to delete. - let has_progress: bool = sqlx::query_scalar( - "SELECT EXISTS(SELECT 1 FROM sqlite_master \ - WHERE type = 'table' AND name = 'restore_progress')", - ) - .fetch_one(&self.pool) - .await - .map_err(db_err)?; - if has_progress { - let _writer = self.write_lock.lock().await; - sqlx::query("DELETE FROM restore_progress WHERE table_id = ?") - .bind(&key.table_id) - .execute(&self.pool) - .await - .map_err(db_err)?; - } + // A store-backed restore's progress lives on the `table_restores` + // row (`file_index`, `items_applied`); SQLite has + // no `restore_progress` table (migration 6 is `NotApplicable` + // here), so the record goes with the target below and there are + // no rows to delete. self.abort_restore(&key.table_id).await }) } @@ -907,58 +1016,170 @@ impl BackupCatalog for SqliteEngine { // without one is aged by its heartbeat). The delay is what leaves // a task still on its own failure path to finish it. Both ages // are measured on the database's clock (`julianday('now')`). - #[allow(clippy::type_complexity)] - let rows: Vec<( - String, - String, - String, - String, - String, - String, - i64, - String, - )> = sqlx::query_as( - "SELECT r.table_id, t.account_id, t.table_name, r.source_backup_arn, \ - r.kind, r.status, r.owner_generation, r.heartbeat_at \ - FROM table_restores r JOIN tables t ON t.table_id = r.table_id \ - WHERE (r.status = 'RUNNING' \ - AND julianday(r.heartbeat_at) < julianday('now') - ?1 / 86400.0) \ - OR (r.status = 'FAILED' AND t.table_status = 'CREATING' \ - AND julianday(COALESCE(r.completed_at, r.heartbeat_at)) \ - < julianday('now') - ?1 / 86400.0) \ - ORDER BY t.account_id, t.table_name", - ) + let rows: Vec = sqlx::query_as(&format!( + "SELECT {RESTORE_ROW_COLUMNS} \ + FROM table_restores r JOIN tables t ON t.table_id = r.table_id \ + WHERE (r.status = 'RUNNING' \ + AND julianday(r.heartbeat_at) < julianday('now') - ?1 / 86400.0) \ + OR (r.status = 'FAILED' AND t.table_status = 'CREATING' \ + AND julianday(COALESCE(r.completed_at, r.heartbeat_at)) \ + < julianday('now') - ?1 / 86400.0) \ + ORDER BY t.account_id, t.table_name" + )) .bind(stale_after.as_secs_f64()) .fetch_all(&self.pool) .await .map_err(db_err)?; - rows.into_iter() - .map( - |( - table_id, - account_id, - table_name, - source_backup_arn, - kind, - status, - owner_generation, - heartbeat_at, - )| { - Ok(RestoreJobRow { - target: RestoreTargetKey { - account_id, - table_name, - table_id, - }, - source_backup_arn, - kind: RestoreKind::parse(&kind)?, - status: JobStatus::parse(&status)?, - owner_generation: u64::try_from(owner_generation).unwrap_or_default(), - heartbeat_at: timestamp_epoch_seconds(&heartbeat_at), - }) - }, - ) - .collect() + rows.into_iter().map(restore_job_from_tuple).collect() + }) + } + + fn claim_restore<'a>( + &'a self, + account_id: &'a str, + input: CreateTableInput, + registration: &'a RestoreRegistrationRow, + ) -> BoxFuture<'a, Result> { + Box::pin(async move { + let table_name = input.table_name.clone(); + // The name's slot first, before the target exists (module doc): + // a slot already held is another store-backed restore into this + // name in flight in this process. It is refused as the name is + // taken: TableBeingRestored naming its source once its CREATING + // row is visible, else as CreateTable would report it. + let Some(slot) = + JobSlots::take(&self.job_slots, restore_slot_key(account_id, &table_name)) + else { + return Err(self.name_taken_error(account_id, &table_name).await); + }; + let generation = new_generation(); + let registration = RestoreRegistrationRow { + owner_generation: generation, + ..registration.clone() + }; + // The registration transaction (`BEGIN IMMEDIATE` under the + // engine write lock): the `backup_index` row's status, the + // in-use check, the CREATING target and its RUNNING record. + let mut desc = match self + .create_table_for_manifest_restore(account_id, input, ®istration) + .await + { + Ok(desc) => desc, + Err(e) => { + drop(slot); + return Err(e); + } + }; + desc.restore_summary = Some(RestoreSummary { + source_backup_arn: Some(registration.backup.backup_arn.clone()), + source_table_arn: Some(registration.source_table_arn.clone()), + restore_date_time: registration.restore_date_time, + restore_in_progress: true, + }); + let claim = + SqliteJobClaim::for_manifest_restore(self, &desc.table_id, generation, slot); + Ok((desc, Box::new(claim) as Box)) + }) + } + + fn claim_restore_dead<'a>( + &'a self, + key: &'a RestoreTargetKey, + _stale_after: Duration, + spool_host: Option<&'a str>, + ) -> BoxFuture<'a, Result>, StorageError>> { + Box::pin(async move { + // The slot is the liveness test, as for `claim_dead_backup`: a + // job of this process that holds the name's slot is alive + // whatever its heartbeat says, and a record without a holder is + // dead at once (a restart frees every slot, and ages every + // RUNNING record so the reconciler lists it). Taking the slot is + // the test and the exclusion of a second taker in one step. + let Some(slot) = JobSlots::take( + &self.job_slots, + restore_slot_key(&key.account_id, &key.table_name), + ) else { + return Ok(None); + }; + let generation = new_generation(); + let _writer = self.write_lock.lock().await; + let mut tx = self + .pool + .begin_with("BEGIN IMMEDIATE") + .await + .map_err(db_err)?; + // The generation, the claimant's spool and a fresh heartbeat in + // one write (the trait doc says why a restore record is refreshed + // where a backup row is not; here the slot already excludes a + // second taker of this process, and the beat keeps the record + // out of the reconciler's dead listing). A legacy record is + // never taken over: the reconciler fails and removes it. + let taken = sqlx::query(&format!( + "UPDATE table_restores SET owner_generation = ?, spool_host = ?, \ + heartbeat_at = {NOW} \ + WHERE table_id = ? AND status = 'RUNNING' AND kind = 'manifest' \ + AND EXISTS(SELECT 1 FROM tables t WHERE t.table_id = table_restores.table_id \ + AND t.account_id = ? AND t.table_name = ?)" + )) + .bind(generation_column(generation)) + .bind(spool_host) + .bind(&key.table_id) + .bind(&key.account_id) + .bind(&key.table_name) + .execute(&mut *tx) + .await + .map_err(db_err)? + .rows_affected(); + tx.commit().await.map_err(db_err)?; + if taken != 1 { + drop(slot); + return Ok(None); + } + Ok(Some(Box::new(SqliteJobClaim::for_manifest_restore( + self, + &key.table_id, + generation, + slot, + )) as Box)) + }) + } + + fn read_restore_job<'a>( + &'a self, + key: &'a RestoreTargetKey, + ) -> BoxFuture<'a, Result, StorageError>> { + Box::pin(async move { + let row: Option = sqlx::query_as(&format!( + "SELECT {RESTORE_ROW_COLUMNS} \ + FROM table_restores r JOIN tables t ON t.table_id = r.table_id \ + WHERE r.table_id = ? AND t.account_id = ? AND t.table_name = ?" + )) + .bind(&key.table_id) + .bind(&key.account_id) + .bind(&key.table_name) + .fetch_optional(&self.pool) + .await + .map_err(db_err)?; + row.map(restore_job_from_tuple).transpose() + }) + } + + fn list_restore_jobs<'a>( + &'a self, + status: Option, + ) -> BoxFuture<'a, Result, StorageError>> { + Box::pin(async move { + let rows: Vec = sqlx::query_as(&format!( + "SELECT {RESTORE_ROW_COLUMNS} \ + FROM table_restores r JOIN tables t ON t.table_id = r.table_id \ + WHERE (?1 IS NULL OR r.status = ?1) \ + ORDER BY t.account_id, t.table_name" + )) + .bind(status.map(JobStatus::as_str)) + .fetch_all(&self.pool) + .await + .map_err(db_err)?; + rows.into_iter().map(restore_job_from_tuple).collect() }) } diff --git a/crates/storage-sqlite/src/create_table.rs b/crates/storage-sqlite/src/create_table.rs index 509e73086..1b183037a 100644 --- a/crates/storage-sqlite/src/create_table.rs +++ b/crates/storage-sqlite/src/create_table.rs @@ -16,6 +16,7 @@ use extenddb_core::types::{ BillingMode, BillingModeSummary, CreateTableInput, GsiDescription, LsiDescription, ProvisionedThroughputDescription, SseDescription, SseType, TableDescription, TableStatus, }; +use extenddb_storage::backup_catalog::RestoreRegistrationRow; use extenddb_storage::error::StorageError; use extenddb_storage::util::{index_arn, stream_arn, table_arn}; @@ -35,7 +36,57 @@ pub(crate) struct RestoreRegistration<'a> { pub owner_generation: u64, } +/// Which backup a restore target is created for, and so which row the +/// registration re-checks in its write transaction: an in-catalog `backups` +/// row (the legacy reader's restore, `kind = legacy`) or a `backup_index` row +/// of a backup in the store (`kind = manifest`, with the store, the spool +/// host and the claim's generation on the record). +pub(crate) enum RestoreSource<'a> { + Legacy(&'a RestoreRegistration<'a>), + Manifest(&'a RestoreRegistrationRow), +} + +impl RestoreSource<'_> { + fn backup_arn(&self) -> &str { + match self { + Self::Legacy(r) => r.backup_arn, + Self::Manifest(r) => &r.backup.backup_arn, + } + } +} + impl SqliteEngine { + /// The error for a restore into a name another restore holds the slot + /// of: `TableBeingRestored` naming that restore's backup once its + /// CREATING row is committed, else `TableAlreadyExists`, as CreateTable + /// reports a taken name. Reads committed state through the pool; a read + /// failure is "not a restore target": an error is being produced + /// already. + pub(crate) async fn name_taken_error( + &self, + account_id: &str, + table_name: &str, + ) -> StorageError { + let restoring: Option = sqlx::query_scalar( + "SELECT r.source_backup_arn FROM tables t \ + JOIN table_restores r ON r.table_id = t.table_id \ + WHERE t.account_id = ? AND t.table_name = ? AND t.table_status = 'CREATING'", + ) + .bind(account_id) + .bind(table_name) + .fetch_optional(&self.pool) + .await + .ok() + .flatten(); + match restoring { + Some(backup_arn) => StorageError::TableBeingRestored { + table_name: table_name.to_owned(), + backup_arn, + }, + None => StorageError::TableAlreadyExists(table_name.to_owned()), + } + } + /// Create a table. When `defer_active` is set (the restore path), the row /// is written `CREATING` with **no** scheduled transition, so the /// background control-plane worker cannot flip it to `ACTIVE` while the @@ -61,8 +112,33 @@ impl SqliteEngine { input: CreateTableInput, registration: &RestoreRegistration<'_>, ) -> Result { - self.create_table_impl_inner(account_id, input, true, Some(registration)) - .await + self.create_table_impl_inner( + account_id, + input, + true, + Some(RestoreSource::Legacy(registration)), + ) + .await + } + + /// Create a store-backed restore's target and register the restore in + /// the write transaction that also re-checks, on the backup's + /// `backup_index` row, that it is AVAILABLE and has no CREATING target. + /// The job record is born RUNNING, `kind = manifest`, with the + /// registration's store, spool host and generation. + pub(crate) async fn create_table_for_manifest_restore( + &self, + account_id: &str, + input: CreateTableInput, + registration: &RestoreRegistrationRow, + ) -> Result { + self.create_table_impl_inner( + account_id, + input, + true, + Some(RestoreSource::Manifest(registration)), + ) + .await } async fn create_table_impl_inner( @@ -70,7 +146,7 @@ impl SqliteEngine { account_id: &str, input: CreateTableInput, defer_active: bool, - restore: Option<&RestoreRegistration<'_>>, + restore: Option>, ) -> Result { Self::validate_account_id(account_id)?; // D1: every writer holds the engine write lock. This method runs two @@ -153,7 +229,8 @@ impl SqliteEngine { .await .map_err(|e| StorageError::Internal(e.to_string()))?; - if let Some(RestoreRegistration { backup_arn, .. }) = restore { + if let Some(source) = &restore { + let backup_arn = source.backup_arn(); // A restore into a name that is itself a CREATING restore target // is refused as such before the backup is checked, so a second // restore of the same backup into the same name is @@ -175,19 +252,45 @@ impl SqliteEngine { backup_arn: source_backup_arn, }); } - let available: bool = sqlx::query_scalar( - "SELECT EXISTS(SELECT 1 FROM backups WHERE backup_arn = ? \ - AND account_id = ? AND backup_status = 'AVAILABLE')", - ) - .bind(backup_arn) - .bind(account_id) - .fetch_one(&mut *tx) - .await - .map_err(|e| StorageError::Internal(e.to_string()))?; - if !available { - return Err(StorageError::Validation(format!( - "Backup not found: {backup_arn}" - ))); + match source { + RestoreSource::Legacy(_) => { + let available: bool = sqlx::query_scalar( + "SELECT EXISTS(SELECT 1 FROM backups WHERE backup_arn = ? \ + AND account_id = ? AND backup_status = 'AVAILABLE')", + ) + .bind(backup_arn) + .bind(account_id) + .fetch_one(&mut *tx) + .await + .map_err(|e| StorageError::Internal(e.to_string()))?; + if !available { + return Err(StorageError::Validation(format!( + "Backup not found: {backup_arn}" + ))); + } + } + RestoreSource::Manifest(registration) => { + // The `backup_index` row's status, in the write + // transaction `start_backup_delete` reads it in: the + // engine write lock and `BEGIN IMMEDIATE` are the lock + // on the row, so a DeleteBackup and this registration + // order wholly one before the other, and two + // registrations of one backup serialize before the + // CREATING-target check below. + let status: Option = sqlx::query_scalar( + "SELECT status FROM backup_index WHERE account_id = ? AND backup_id = ?", + ) + .bind(account_id) + .bind(®istration.backup.backup_id) + .fetch_optional(&mut *tx) + .await + .map_err(|e| StorageError::Internal(e.to_string()))?; + if status.as_deref() != Some("AVAILABLE") { + return Err(extenddb_storage::backup_catalog::backup_not_found( + backup_arn, + )); + } + } } // One restore of a backup at a time, as the service: while a // target restored from it is CREATING the backup is in use. Same @@ -258,24 +361,51 @@ impl SqliteEngine { return Err(StorageError::TableAlreadyExists(input.table_name.clone())); } - if let Some(registration) = restore { - sqlx::query(&format!( - "INSERT INTO table_restores \ - (table_id, source_backup_arn, source_table_arn, restore_date_time, \ - status, kind, owner_generation, heartbeat_at) \ - VALUES (?, ?, ?, ?, 'RUNNING', 'legacy', ?, {})", - crate::backup_catalog::NOW - )) - .bind(&table_id) - .bind(registration.backup_arn) - .bind(registration.source_table_arn) - .bind(format_epoch_seconds(registration.restore_date_time)) - .bind(crate::backup_catalog::generation_column( - registration.owner_generation, - )) - .execute(&mut *tx) - .await - .map_err(|e| StorageError::Internal(e.to_string()))?; + match &restore { + Some(RestoreSource::Legacy(registration)) => { + sqlx::query(&format!( + "INSERT INTO table_restores \ + (table_id, source_backup_arn, source_table_arn, restore_date_time, \ + status, kind, owner_generation, heartbeat_at) \ + VALUES (?, ?, ?, ?, 'RUNNING', 'legacy', ?, {})", + crate::backup_catalog::NOW + )) + .bind(&table_id) + .bind(registration.backup_arn) + .bind(registration.source_table_arn) + .bind(format_epoch_seconds(registration.restore_date_time)) + .bind(crate::backup_catalog::generation_column( + registration.owner_generation, + )) + .execute(&mut *tx) + .await + .map_err(|e| StorageError::Internal(e.to_string()))?; + } + Some(RestoreSource::Manifest(registration)) => { + sqlx::query(&format!( + "INSERT INTO table_restores \ + (table_id, source_backup_arn, source_table_arn, restore_date_time, \ + status, kind, store_kind, store_root, store_id, spool_host, \ + owner_generation, heartbeat_at) \ + VALUES (?, ?, ?, ?, 'RUNNING', 'manifest', ?, ?, ?, ?, ?, {})", + crate::backup_catalog::NOW + )) + .bind(&table_id) + .bind(®istration.backup.backup_arn) + .bind(®istration.source_table_arn) + .bind(format_epoch_seconds(registration.restore_date_time)) + .bind(registration.store.store_kind.as_str()) + .bind(®istration.store.store_root) + .bind(®istration.store.store_id) + .bind(registration.spool_host.as_deref()) + .bind(crate::backup_catalog::generation_column( + registration.owner_generation, + )) + .execute(&mut *tx) + .await + .map_err(|e| StorageError::Internal(e.to_string()))?; + } + None => {} } // GSI / LSI metadata. diff --git a/crates/storage-sqlite/src/data/vector_index.rs b/crates/storage-sqlite/src/data/vector_index.rs index 02bc09836..80acca8cd 100644 --- a/crates/storage-sqlite/src/data/vector_index.rs +++ b/crates/storage-sqlite/src/data/vector_index.rs @@ -126,7 +126,7 @@ fn base_key_binds( } /// Column names for the base key, in key-schema order. -fn base_key_columns( +pub(crate) fn base_key_columns( base_key_schema: &[KeySchemaElement], attr_defs: &[AttributeDefinition], ) -> Vec { diff --git a/crates/storage-sqlite/src/lib.rs b/crates/storage-sqlite/src/lib.rs index a6d7aa8a9..2e9f146e8 100644 --- a/crates/storage-sqlite/src/lib.rs +++ b/crates/storage-sqlite/src/lib.rs @@ -37,6 +37,9 @@ mod metadata; pub mod migrations; mod number_key; mod operations; +mod restore_engine; +#[cfg(test)] +mod restore_job_tests; mod schema; mod serve_lock; mod snapshot_engine; @@ -319,18 +322,28 @@ fn sqlite_server_components_factory( // A restore the previous run left mid-copy: one process serves this // file, so its owner is gone. Age every RUNNING job record so the - // engine's reconciler fails it and removes its target on its first - // cycle rather than after the stale window. Before the server takes - // requests, so no restore of this process can be in flight. A backup - // the previous run left CREATING is reported to the reconciler by + // engine's reconciler acts on it on its first cycle rather than + // after the stale window: a store-backed (`manifest`) record is + // taken over and resumed, an in-catalog (`legacy`) one is failed and + // its target removed. Before the server takes requests, so no + // restore of this process can be in flight. A backup the previous + // run left CREATING is reported to the reconciler by // `abandoned_creating_backups` without a step here: no task of this // process holds its slot. match engine.expire_restore_job_heartbeats().await { - Ok(names) => { - for name in &names { + Ok(aged) => { + for (name, kind) in &aged { + let duty = match kind { + extenddb_storage::backup_catalog::RestoreKind::Manifest => { + "the reconciler resumes it (a store-backed restore)" + } + extenddb_storage::backup_catalog::RestoreKind::Legacy => { + "the reconciler removes it (an in-catalog restore)" + } + }; tracing::warn!( "table '{name}': its restore did not finish before the last shutdown; \ - the reconciler removes it" + {duty}" ); } } diff --git a/crates/storage-sqlite/src/migrations/mod.rs b/crates/storage-sqlite/src/migrations/mod.rs index eb2f1e0ff..6b0af1f7b 100644 --- a/crates/storage-sqlite/src/migrations/mod.rs +++ b/crates/storage-sqlite/src/migrations/mod.rs @@ -108,6 +108,10 @@ pub fn migration(id: MigrationId) -> Migration { m0005_backup_catalog_index::SOURCE, )], }, + MigrationId::RestoreProgress => Migration::NotApplicable( + "the restore progress of a store-backed restore is on table_restores (file_index, \ + items_applied) on SQLite; the table is PostgreSQL's data database's", + ), } } diff --git a/crates/storage-sqlite/src/migrations/tests.rs b/crates/storage-sqlite/src/migrations/tests.rs index 5fa0d8739..cf1f1fe96 100644 --- a/crates/storage-sqlite/src/migrations/tests.rs +++ b/crates/storage-sqlite/src/migrations/tests.rs @@ -270,7 +270,14 @@ async fn a_fresh_database_is_current_after_one_run_and_seeds_nothing() { "SELECT COUNT(*) FROM extenddb_migrations WHERE outcome = 'applied'" ) .await, - i64::from(LATEST) + i64::from(LATEST) - 1, + "every id but 6 (the PostgreSQL progress table) applies on SQLite" + ); + assert_eq!( + scalar(&db, "SELECT outcome FROM extenddb_migrations WHERE id = 6") + .await + .as_deref(), + Some("not_applicable") ); assert_eq!(version_row(&db).await, Some(LATEST.to_string())); assert_eq!( diff --git a/crates/storage-sqlite/src/restore_engine.rs b/crates/storage-sqlite/src/restore_engine.rs new file mode 100644 index 000000000..42714d0d5 --- /dev/null +++ b/crates/storage-sqlite/src/restore_engine.rs @@ -0,0 +1,538 @@ +// Copyright 2026 ExtendDB contributors +// SPDX-License-Identifier: Apache-2.0 + +//! The data-plane side of a store-backed restore on SQLite +//! ([`RestoreEngine`]): the items of one verified data file written into the +//! CREATING target, the progress record, the ACTIVE flip and the count. +//! +//! One write transaction (`BEGIN IMMEDIATE` under the engine write lock) per +//! batch of at most [`RESTORE_BATCH_ITEMS`] items or [`RESTORE_BATCH_BYTES`] +//! of serialized item text (a batch closes at the first item that takes its +//! `item_data` text past the bound, so the batch term of a restore's memory +//! is bounded in bytes, not only in items): the base rows (plain INSERTs, so +//! a key the backup holds twice, within a file or across files, fails the +//! file instead of collapsing two items), each secondary index's rows and +//! each vector index's rows beside them, and last the record's progress, +//! `UPDATE table_restores SET file_index, items_applied` conditioned on the +//! claim's generation and on RUNNING: a batch commits whole with its +//! progress or not at all, and a batch whose record another owner took +//! rolls back and ends the file as a lost claim. Other writers on the server +//! wait for a batch rather than the whole file. A resume reads the record's +//! `(file_index, items_applied)` and the engine skips that many items of the +//! file before the stream starts. An empty file records its progress too +//! (`items_applied = start_at`), so a resume does not re-read it. +//! +//! The restore's own buffer is this one batch of `(Item, String)`, bounded +//! by [`RESTORE_BATCH_BYTES`] whatever the table's indexes are. A table with +//! secondary or vector indexes nonetheless restores in more resident memory +//! than one without (about 90 MiB over idle at 100,000 200-byte items with +//! two full-projection GSIs against 43 MiB with none), and the excess is +//! SQLite's own: its per-connection page cache (`PRAGMA cache_size = +//! -32000`, 32 MiB) holds the index tables' b-tree pages, which a load of +//! scattered index keys dirties across the whole tree, and glibc's +//! per-arena free lists hold freed blocks (the figure falls by about a third +//! under `MALLOC_ARENA_MAX=1`). The excess is bounded by `cache_size` and +//! grows only sub-linearly with the table (about 90 MiB at 100,000 items, +//! 175 MiB at 300,000), not with the restore's buffers; a smaller batch does +//! not lower it (measured: the peak is unchanged at a third of the item +//! cap), because the cache persists across batches. The published restore +//! figures are for a table with no secondary index and state that shape. +//! +//! A job's heartbeat on SQLite (`SqliteJobClaim::heartbeat`) is an `UPDATE` +//! under the same engine write lock, so a beat waits for the batch that +//! holds the lock and for every writer queued ahead of it: other jobs' +//! batches, item writes, the GSI backfill, the TTL sweep, a table's drop. +//! The bench record (`docs/design/15-backup-restore-bench-pr9.md`) puts one +//! batch at about 1.1 s for items of 27.7 MB (24 of them, one per batch, +//! restored in 26.8 s with their download and decode) and about 0.27 s for +//! 10,000 items of 200 bytes (1,000,000 in 26.8 s). A beat that has not +//! returned within one heartbeat period (15 s at the least) ends the job as +//! a lost claim: it stops with its record RUNNING and makes no further +//! table access, its slot is freed, and the reconciler's next cycle takes +//! the record over and resumes it from the last committed batch. At the +//! default four jobs a beat waits a few seconds at most; many jobs of large +//! items, or the drop of a large table, can hold it past a period, which +//! costs a restart of the file from its last batch and a lost-claim log +//! line, not data. +//! +//! `finish_restore` is [`activate_restore_target`], the one transaction the +//! legacy copy ends with too: the record DONE conditioned on the generation +//! and on RUNNING, the table ACTIVE conditioned on CREATING, and every +//! secondary and vector index of the table ACTIVE. + +use extenddb_core::types::{AttributeDefinition, Item, KeySchemaElement, Projection}; +use extenddb_storage::backup_catalog::{ + JobClaim, RESTORE_SQLITE_MID_BATCH, RestoreTargetKey, restore_pause_point, +}; +use extenddb_storage::error::StorageError; +use extenddb_storage::restore_engine::{BulkLoadFile, Finished, RestoreEngine}; +pub(crate) use extenddb_storage::restore_engine::{RESTORE_BATCH_BYTES, RESTORE_BATCH_ITEMS}; +use extenddb_storage::util::{composite_pk_to_text, parse_sk, sk_column_n}; +use extenddb_storage::vector_lifecycle::VectorIndexMeta; +use futures::future::BoxFuture; +use futures::stream::BoxStream; + +use crate::SqliteEngine; +use crate::backup_catalog::{NOW, generation_column}; +use crate::data::vector_index::{ + VectorRowConflict, base_key_columns, fetch_vector_indexes_for_table, insert_vector_row, +}; +use crate::data::{ + BoundValue, all_sort_key_info, data_table_name, index_table_name, insert_index_row_multi, + item_has_index_keys, project_item_for_index, sk_bound, +}; +use crate::sqlite_util::is_unique_violation; + +fn db_err(e: sqlx::Error) -> StorageError { + StorageError::Internal(format!("Database error: {e}")) +} + +/// The error for a key the target already holds: the backup carries it +/// twice, within the file or in a file applied before it. The engine +/// prefixes the file's key, so the file is not named here. +fn duplicate_key_error(e: &sqlx::Error) -> StorageError { + StorageError::Validation(format!( + "holds a primary key the target already has (the backup holds a key twice): {e}" + )) +} + +/// The key shape of a CREATING restore target, read from its catalog row. +struct TargetShape { + key_schema: Vec, + attribute_definitions: Vec, +} + +/// A secondary index of the target, as the load writes it. +struct TargetIndex { + table: String, + key_schema: Vec, + projection: Projection, +} + +/// What `bulk_load` writes per item and per batch, read once per file. +struct LoadPlan { + insert_sql: String, + shape: TargetShape, + indexes: Vec, + vector_key_columns: Vec, + file: BulkLoadFile, + generation: u64, + /// The file as an error names it. + source: String, +} + +/// What [`activate_restore_target`] matched. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub(crate) struct Activated { + /// The record was RUNNING under the generation and is DONE. + pub record: bool, + /// The table was CREATING and is ACTIVE. + pub table: bool, +} + +/// The end of a restore, inside the caller's write transaction: the record +/// DONE (`completed_at` now; `items_applied` left as the progress stands +/// unless the caller gives a total) only while it is RUNNING under +/// `generation`; the table ACTIVE with its counts only from CREATING; every +/// secondary and vector index of the table ACTIVE (`backfilling` cleared on +/// the vector rows, which a restore target creates CREATING with the table). +/// Reports what each conditional write matched; the caller rolls back when +/// it needed a match it did not get. +pub(crate) async fn activate_restore_target( + tx: &mut sqlx::Transaction<'_, sqlx::Sqlite>, + table_id: &str, + generation: u64, + item_count: i64, + table_size_bytes: i64, + items_applied: Option, +) -> Result { + let record = sqlx::query(&format!( + "UPDATE table_restores SET status = 'DONE', \ + items_applied = COALESCE(?, items_applied), completed_at = {NOW} \ + WHERE table_id = ? AND status = 'RUNNING' AND owner_generation = ?" + )) + .bind(items_applied) + .bind(table_id) + .bind(generation_column(generation)) + .execute(&mut **tx) + .await + .map_err(db_err)? + .rows_affected() + == 1; + if !record { + return Ok(Activated { + record, + table: false, + }); + } + let table = sqlx::query( + "UPDATE tables SET item_count = ?, table_size_bytes = ?, \ + table_status = 'ACTIVE', status_transition_at = NULL \ + WHERE table_id = ? AND table_status = 'CREATING'", + ) + .bind(item_count) + .bind(table_size_bytes) + .bind(table_id) + .execute(&mut **tx) + .await + .map_err(db_err)? + .rows_affected() + == 1; + sqlx::query("UPDATE indexes SET index_status = 'ACTIVE' WHERE table_id = ? AND index_status <> 'ACTIVE'") + .bind(table_id) + .execute(&mut **tx) + .await + .map_err(db_err)?; + sqlx::query( + "UPDATE vector_indexes SET index_status = 'ACTIVE', backfilling = NULL \ + WHERE table_id = ? AND index_status <> 'ACTIVE'", + ) + .bind(table_id) + .execute(&mut **tx) + .await + .map_err(db_err)?; + Ok(Activated { record, table }) +} + +impl SqliteEngine { + /// The target's key schema and attribute definitions, when `key` names + /// a CREATING restore target with this table id. + async fn restore_target_shape( + &self, + key: &RestoreTargetKey, + ) -> Result { + let row: Option<(String, String)> = sqlx::query_as( + "SELECT t.key_schema, t.attribute_definitions FROM tables t \ + JOIN table_restores r ON r.table_id = t.table_id \ + WHERE t.table_id = ? AND t.account_id = ? AND t.table_name = ? \ + AND t.table_status = 'CREATING'", + ) + .bind(&key.table_id) + .bind(&key.account_id) + .bind(&key.table_name) + .fetch_optional(&self.pool) + .await + .map_err(db_err)?; + let (key_schema, attribute_definitions) = row.ok_or_else(|| { + StorageError::TableNotFound(format!( + "{} is not a CREATING restore target with table id {}", + key.table_name, key.table_id + )) + })?; + Ok(TargetShape { + key_schema: serde_json::from_str(&key_schema) + .map_err(|e| StorageError::Internal(format!("key schema: {e}")))?, + attribute_definitions: serde_json::from_str(&attribute_definitions) + .map_err(|e| StorageError::Internal(format!("attribute definitions: {e}")))?, + }) + } + + /// Everything a file's writes need that does not change between batches. + async fn load_plan( + &self, + key: &RestoreTargetKey, + file: BulkLoadFile, + generation: u64, + ) -> Result { + let shape = self.restore_target_shape(key).await?; + let sort_keys = all_sort_key_info(&shape.key_schema, &shape.attribute_definitions); + let mut cols = vec!["pk".to_owned()]; + cols.extend( + sort_keys + .iter() + .enumerate() + .map(|(i, &(_, t))| sk_column_n(i, t)), + ); + cols.push("item_data".to_owned()); + let data_table = data_table_name(&key.table_id); + let insert_sql = format!( + "INSERT INTO {data_table} ({}) VALUES ({})", + cols.join(", "), + vec!["?"; cols.len()].join(", ") + ); + let index_rows: Vec<(String, String, String)> = sqlx::query_as( + "SELECT index_id, key_schema, projection FROM indexes WHERE table_id = ?", + ) + .bind(&key.table_id) + .fetch_all(&self.pool) + .await + .map_err(db_err)?; + let mut indexes = Vec::with_capacity(index_rows.len()); + for (index_id, ks, proj) in index_rows { + indexes.push(TargetIndex { + table: index_table_name(&index_id), + key_schema: serde_json::from_str(&ks) + .map_err(|e| StorageError::Internal(format!("index key schema: {e}")))?, + projection: serde_json::from_str(&proj) + .map_err(|e| StorageError::Internal(format!("index projection: {e}")))?, + }); + } + let vector_key_columns = base_key_columns(&shape.key_schema, &shape.attribute_definitions); + Ok(LoadPlan { + insert_sql, + shape, + indexes, + vector_key_columns, + file, + generation, + source: format!("data file {}", file.file_index), + }) + } + + /// One batch: the base rows, the index rows, the vector rows and the + /// record's progress in one write transaction under the write lock. + /// `applied` is the file's running total before this batch, `start_at` + /// included; each item comes with its `item_data` text, serialized when + /// the batch was collected. `Ok(false)` when the record was not RUNNING + /// under the plan's generation: nothing of the batch is committed. + async fn write_restore_batch( + &self, + key: &RestoreTargetKey, + plan: &LoadPlan, + applied: u64, + batch: &mut Vec<(Item, String)>, + ) -> Result { + let batch_len = batch.len(); + let source = plan.source.as_str(); + let _writer = self.write_lock.lock().await; + let mut tx = self + .pool + .begin_with("BEGIN IMMEDIATE") + .await + .map_err(db_err)?; + let shape = &plan.shape; + let sort_keys = all_sort_key_info(&shape.key_schema, &shape.attribute_definitions); + // The vector indexes, read inside the transaction as the write path + // does (the id names the data table). + let vector_metas: Vec = + fetch_vector_indexes_for_table(&mut tx, &key.table_id).await?; + for (item, item_json) in batch.drain(..) { + let mut values = vec![BoundValue::Text(composite_pk_to_text( + &item, + &shape.key_schema, + )?)]; + for &(name, sk_type) in &sort_keys { + let value = item.get(name).ok_or_else(|| { + StorageError::Validation(format!( + "{source} holds an item without its sort key {name}" + )) + })?; + values.push(sk_bound(&parse_sk(value, sk_type)?)); + } + values.push(BoundValue::Text(item_json)); + let mut q = sqlx::query(&plan.insert_sql); + for v in values { + q = crate::data::bind_bound!(q, v); + } + q.execute(&mut *tx).await.map_err(|e| { + if is_unique_violation(&e) { + duplicate_key_error(&e) + } else { + db_err(e) + } + })?; + for idx in &plan.indexes { + if !item_has_index_keys(&item, &idx.key_schema) { + continue; + } + let idx_sks = all_sort_key_info(&idx.key_schema, &shape.attribute_definitions); + let projected = project_item_for_index( + &item, + &idx.key_schema, + &shape.key_schema, + &idx.projection, + ); + insert_index_row_multi( + &mut tx, + &idx.table, + &item, + &projected, + &idx.key_schema, + &shape.key_schema, + &idx_sks, + &sort_keys, + ) + .await?; + } + for meta in &vector_metas { + insert_vector_row( + &mut tx, + &key.table_id, + meta, + &item, + &shape.key_schema, + &shape.attribute_definitions, + &plan.vector_key_columns, + VectorRowConflict::Fail, + ) + .await?; + } + } + // The progress, last and in the same transaction, conditioned on the + // generation and on RUNNING: a record another owner took rolls the + // batch back with it. + let total = applied.saturating_add(u64::try_from(batch_len).unwrap_or(u64::MAX)); + let matched = sqlx::query( + "UPDATE table_restores SET file_index = ?, items_applied = ? \ + WHERE table_id = ? AND owner_generation = ? AND status = 'RUNNING'", + ) + .bind(i64::from(plan.file.file_index)) + .bind(i64::try_from(total).unwrap_or(i64::MAX)) + .bind(&key.table_id) + .bind(generation_column(plan.generation)) + .execute(&mut *tx) + .await + .map_err(db_err)? + .rows_affected(); + if matched != 1 { + tx.rollback().await.map_err(db_err)?; + return Ok(false); + } + tx.commit().await.map_err(db_err)?; + Ok(true) + } +} + +impl RestoreEngine for SqliteEngine { + fn bulk_load<'a>( + &'a self, + target: &'a RestoreTargetKey, + claim: &'a dyn JobClaim, + file: BulkLoadFile, + mut items: BoxStream<'a, Result>, + ) -> BoxFuture<'a, Result> { + Box::pin(async move { + use futures::StreamExt; + + let generation = claim.generation(); + let plan = self.load_plan(target, file, generation).await?; + let mut batch: Vec<(Item, String)> = Vec::with_capacity(RESTORE_BATCH_ITEMS.min(1024)); + let mut batch_bytes = 0usize; + let mut applied: u64 = 0; + let mut batches_written = 0u32; + let mut ended = false; + while !ended { + // Collect a batch, closed at the count cap or at the first + // item whose text takes it past the byte bound; an `Err` + // item ends the file with that error, and the batch in hand + // is never written. + while batch.len() < RESTORE_BATCH_ITEMS && batch_bytes < RESTORE_BATCH_BYTES { + match items.next().await { + Some(item) => { + let item = item?; + let item_json = serde_json::to_string(&item) + .map_err(|e| StorageError::Internal(format!("item: {e}")))?; + batch_bytes = batch_bytes.saturating_add(item_json.len()); + batch.push((item, item_json)); + } + None => { + ended = true; + break; + } + } + } + if batch.is_empty() && batches_written > 0 { + break; + } + if batches_written > 0 { + restore_pause_point(RESTORE_SQLITE_MID_BATCH).await; + } + let batch_len = batch.len(); + let written = self + .write_restore_batch( + target, + &plan, + file.start_at.saturating_add(applied), + &mut batch, + ) + .await?; + if !written { + return Err(StorageError::Internal(format!( + "claim lost: the job record of {} is not RUNNING under generation \ + {generation}; the batch was rolled back", + target.table_name + ))); + } + applied += u64::try_from(batch_len).unwrap_or(u64::MAX); + batches_written += 1; + batch_bytes = 0; + } + Ok(applied) + }) + } + + fn finish_restore<'a>( + &'a self, + target: &'a RestoreTargetKey, + claim: &'a dyn JobClaim, + item_count: u64, + size_bytes: u64, + ) -> BoxFuture<'a, Result> { + Box::pin(async move { + // The key names the table; the flip itself is conditioned on the + // table id, CREATING, RUNNING and the generation. + let is_target: bool = sqlx::query_scalar( + "SELECT EXISTS(SELECT 1 FROM tables WHERE table_id = ? AND account_id = ? \ + AND table_name = ?)", + ) + .bind(&target.table_id) + .bind(&target.account_id) + .bind(&target.table_name) + .fetch_one(&self.pool) + .await + .map_err(db_err)?; + if !is_target { + return Ok(Finished::NotMatched(format!( + "{} is not a table with id {} in account {}", + target.table_name, target.table_id, target.account_id + ))); + } + let _writer = self.write_lock.lock().await; + let mut tx = self + .pool + .begin_with("BEGIN IMMEDIATE") + .await + .map_err(db_err)?; + let activated = activate_restore_target( + &mut tx, + &target.table_id, + claim.generation(), + i64::try_from(item_count).unwrap_or(i64::MAX), + i64::try_from(size_bytes).unwrap_or(i64::MAX), + None, + ) + .await?; + if !(activated.record && activated.table) { + tx.rollback().await.map_err(db_err)?; + return Ok(Finished::NotMatched(format!( + "the job record of {} is not RUNNING under generation {}, or the table is \ + not CREATING", + target.table_name, + claim.generation() + ))); + } + tx.commit().await.map_err(db_err)?; + Ok(Finished::Active) + }) + } + + fn count_restore_target<'a>( + &'a self, + key: &'a RestoreTargetKey, + ) -> BoxFuture<'a, Result> { + Box::pin(async move { + // The id names the data table; the key's account and name are + // checked against the catalog so a bare id reads no other + // account's table. + self.restore_target_shape(key).await?; + let count: i64 = sqlx::query_scalar(&format!( + "SELECT COUNT(*) FROM {}", + data_table_name(&key.table_id) + )) + .fetch_one(&self.pool) + .await + .map_err(db_err)?; + Ok(u64::try_from(count).unwrap_or_default()) + }) + } +} diff --git a/crates/storage-sqlite/src/restore_job_tests.rs b/crates/storage-sqlite/src/restore_job_tests.rs new file mode 100644 index 000000000..6d28da0a0 --- /dev/null +++ b/crates/storage-sqlite/src/restore_job_tests.rs @@ -0,0 +1,1536 @@ +// Copyright 2026 ExtendDB contributors +// SPDX-License-Identifier: Apache-2.0 + +//! Storage-level tests of the store-backed restore on SQLite: the +//! registration under the write lock (`claim_restore`), the takeover of a +//! record whose slot no task holds (`claim_restore_dead`), the per-batch load +//! with its progress write (`bulk_load`), the ACTIVE flip (`finish_restore`) +//! and the count. The engine's restore job drives these through +//! `BackupCatalog` and `RestoreEngine`; what each must guarantee on this +//! backend is what a test here pins. Every test builds its own in-memory +//! engine. + +use std::sync::Arc; +use std::time::Duration; + +use extenddb_core::types::{ + AttributeValue, CreateTableInput, DistanceFunction, Item, Projection, ProjectionType, + TableDescription, TableStatus, VectorAttribute, VectorIndexSpecification, +}; +use extenddb_storage::backup_catalog::{ + BackupRow, BackupStatus, DeleteStart, Heartbeat, JobClaim, JobStatus, RestoreJobRow, + RestoreKind, RestoreRegistrationRow, RestoreTargetKey, StatusFields, StoreIdentity, StoreKind, + WriteOutcome, is_backup_not_found, +}; +use extenddb_storage::error::StorageError; +use extenddb_storage::restore_engine::{BulkLoadFile, Finished, RestoreEngine}; +use extenddb_storage::{BackupCatalog, JobPermit, TableEngine}; +use futures::StreamExt; +use futures::stream::BoxStream; + +use crate::SqliteEngine; +use crate::backup_catalog::{SqliteJobClaim, restore_slot_key}; +use crate::restore_engine::{RESTORE_BATCH_BYTES, RESTORE_BATCH_ITEMS}; + +const ACCOUNT: &str = "123456789012"; +const REGION: &str = "us-east-1"; +/// A stale window no test's heartbeat crosses on its own. +const STALE: Duration = Duration::from_secs(300); + +// --- the engine, rows, inputs and items ------------------------------------------- + +async fn engine() -> SqliteEngine { + let engine = SqliteEngine::new(":memory:", 2, REGION, 409_600) + .await + .expect("engine"); + crate::migrations::apply_for_tests(&engine.pool).await; + for (key, value) in [ + ("control_plane_delay_seconds", "0"), + ("index_propagation_delay_ms", "0"), + ] { + sqlx::query( + "INSERT INTO settings (key, value) VALUES (?, ?) \ + ON CONFLICT(key) DO UPDATE SET value = excluded.value", + ) + .bind(key) + .bind(value) + .execute(&engine.pool) + .await + .expect("setting"); + } + sqlx::query("INSERT INTO accounts (account_id, account_name) VALUES (?, 'default')") + .bind(ACCOUNT) + .execute(&engine.pool) + .await + .expect("account"); + engine +} + +fn store() -> StoreIdentity { + StoreIdentity { + store_kind: StoreKind::Filesystem, + store_root: "/var/lib/extenddb/backups".to_owned(), + store_id: "store-1".to_owned(), + } +} + +/// An AVAILABLE `backup_index` row of a backup of `table`, written through +/// the claim as a backup job writes it. +async fn available_backup(engine: &SqliteEngine, table: &str) -> BackupRow { + let backup_id = extenddb_storage::util::new_backup_id(); + let backup_arn = extenddb_storage::util::backup_arn(REGION, ACCOUNT, table, &backup_id); + let manifest: serde_json::Value = serde_json::from_str(include_str!( + "../../core/testdata/extenddb-manifest/v1.json" + )) + .expect("fixture"); + let row = BackupRow { + backup_id: backup_id.clone(), + account_id: ACCOUNT.to_owned(), + table_name: table.to_owned(), + backup_arn, + backup_name: "nightly".to_owned(), + status: BackupStatus::Creating, + store: store(), + size_bytes: 0, + item_count: 0, + created_at: extenddb_storage::util::backup_creation_date_time(&backup_id).unwrap(), + completed_at: None, + failure_reason: None, + source_table_details_json: manifest["source_table"].clone(), + owner_generation: 0, + heartbeat_at: 0.0, + }; + let claim = engine.claim_backup(&row).await.expect("claim the row"); + let written = claim + .set_status( + JobStatus::Creating, + JobStatus::Available, + &StatusFields { + item_count: Some(3), + size_bytes: Some(1234), + completed: true, + ..StatusFields::default() + }, + ) + .await + .expect("the AVAILABLE write"); + assert_eq!(written, WriteOutcome::Matched); + drop(claim); + BackupRow { + status: BackupStatus::Available, + ..row + } +} + +fn registration(row: &BackupRow, spool_host: Option<&str>) -> RestoreRegistrationRow { + RestoreRegistrationRow { + backup: row.key(), + source_table_arn: format!( + "arn:aws:dynamodb:{REGION}:{ACCOUNT}:table/{}", + row.table_name + ), + restore_date_time: row.created_at, + store: row.store.clone(), + spool_host: spool_host.map(str::to_owned), + owner_generation: 0, + } +} + +/// The target: a composite-key table with a GSI on `g`, an LSI on `l` and, +/// when asked, a vector index on `emb` (three dimensions). +fn target_input(name: &str, vector: bool) -> CreateTableInput { + let mut input: CreateTableInput = serde_json::from_value(serde_json::json!({ + "TableName": name, + "KeySchema": [ + {"AttributeName": "pk", "KeyType": "HASH"}, + {"AttributeName": "sk", "KeyType": "RANGE"} + ], + "AttributeDefinitions": [ + {"AttributeName": "pk", "AttributeType": "S"}, + {"AttributeName": "sk", "AttributeType": "N"}, + {"AttributeName": "g", "AttributeType": "S"}, + {"AttributeName": "l", "AttributeType": "S"} + ], + "BillingMode": "PAY_PER_REQUEST", + "GlobalSecondaryIndexes": [{ + "IndexName": "gix", + "KeySchema": [{"AttributeName": "g", "KeyType": "HASH"}], + "Projection": {"ProjectionType": "KEYS_ONLY"} + }], + "LocalSecondaryIndexes": [{ + "IndexName": "lix", + "KeySchema": [ + {"AttributeName": "pk", "KeyType": "HASH"}, + {"AttributeName": "l", "KeyType": "RANGE"} + ], + "Projection": {"ProjectionType": "ALL"} + }] + })) + .expect("input"); + if vector { + input.vector_indexes = Some(vec![VectorIndexSpecification { + index_name: "vix".to_owned(), + dimensions: 3, + distance_function: DistanceFunction::Cosine, + vector_attribute: VectorAttribute { + attribute_name: "emb".to_owned(), + }, + search_schema: None, + projection: Some(Projection { + projection_type: ProjectionType::All, + non_key_attributes: None, + }), + }]); + } + input +} + +/// Item `i` of the fixture: every item has `pk` and `sk`, every second one +/// `g` (the GSI's key), every third one `l` (the LSI's), every fifth one +/// `emb` (the vector). +fn item(i: usize) -> Item { + let mut item = + serde_json::json!({"pk": {"S": format!("p{}", i % 7)}, "sk": {"N": i.to_string()}}); + if i.is_multiple_of(2) { + item["g"] = serde_json::json!({"S": format!("g{}", i % 4)}); + } + if i.is_multiple_of(3) { + item["l"] = serde_json::json!({"S": format!("l{i}")}); + } + if i.is_multiple_of(5) { + item["emb"] = serde_json::json!({"L": [{"N": "0.1"}, {"N": "0.2"}, {"N": "0.3"}]}); + } + serde_json::from_value(item).expect("item") +} + +fn items(range: std::ops::Range) -> Vec { + range.map(item).collect() +} + +fn stream(items: Vec) -> BoxStream<'static, Result> { + Box::pin(futures::stream::iter(items.into_iter().map(Ok))) +} + +fn file(index: u32, start_at: u64) -> BulkLoadFile { + BulkLoadFile { + file_index: index, + start_at, + reapplying: false, + } +} + +fn key_of(desc: &TableDescription) -> RestoreTargetKey { + RestoreTargetKey { + account_id: ACCOUNT.to_owned(), + table_name: desc.table_name.clone(), + table_id: desc.table_id.clone(), + } +} + +/// Register a restore of `row` into `target` and return the description, its +/// key and the claim. +async fn register( + engine: &SqliteEngine, + row: &BackupRow, + target: &str, + vector: bool, + spool_host: Option<&str>, +) -> (TableDescription, RestoreTargetKey, Box) { + let (desc, claim) = engine + .claim_restore( + ACCOUNT, + target_input(target, vector), + ®istration(row, spool_host), + ) + .await + .expect("register the restore"); + let key = key_of(&desc); + (desc, key, claim) +} + +/// A claim on the record under `generation`, as a job whose record passed to +/// another owner still holds one. +fn stale_claim(engine: &SqliteEngine, table_id: &str, generation: u64) -> SqliteJobClaim { + SqliteJobClaim::for_restore(engine, table_id, generation, JobPermit::detached()) +} + +// --- reads -------------------------------------------------------------------- + +async fn table_status(engine: &SqliteEngine, name: &str) -> Option { + sqlx::query_scalar("SELECT table_status FROM tables WHERE account_id = ? AND table_name = ?") + .bind(ACCOUNT) + .bind(name) + .fetch_optional(&engine.pool) + .await + .expect("table status") +} + +async fn record(engine: &SqliteEngine, key: &RestoreTargetKey) -> RestoreJobRow { + engine + .read_restore_job(key) + .await + .expect("read the record") + .expect("the record exists") +} + +/// `(status, kind, owner_generation, heartbeat_at, spool_host, file_index, +/// items_applied, completed_at)` of the record, straight from the table. +type RecordColumns = ( + String, + String, + i64, + String, + Option, + Option, + Option, + Option, +); + +async fn record_columns(engine: &SqliteEngine, table_id: &str) -> RecordColumns { + sqlx::query_as( + "SELECT status, kind, owner_generation, heartbeat_at, spool_host, file_index, \ + items_applied, completed_at FROM table_restores WHERE table_id = ?", + ) + .bind(table_id) + .fetch_one(&engine.pool) + .await + .expect("read table_restores") +} + +async fn rows_in(engine: &SqliteEngine, physical: &str) -> i64 { + sqlx::query_scalar(&format!("SELECT COUNT(*) FROM {physical}")) + .fetch_one(&engine.pool) + .await + .expect("count rows") +} + +async fn base_rows(engine: &SqliteEngine, table_id: &str) -> i64 { + rows_in(engine, &crate::data::data_table_name(table_id)).await +} + +/// `(index_name, index_id, index_status)` of the table's secondary indexes. +async fn indexes(engine: &SqliteEngine, table_id: &str) -> Vec<(String, String, String)> { + sqlx::query_as( + "SELECT index_name, index_id, index_status FROM indexes WHERE table_id = ? \ + ORDER BY index_name", + ) + .bind(table_id) + .fetch_all(&engine.pool) + .await + .expect("read indexes") +} + +async fn index_rows(engine: &SqliteEngine, table_id: &str, name: &str) -> i64 { + let id = indexes(engine, table_id) + .await + .into_iter() + .find(|(n, _, _)| n == name) + .map(|(_, id, _)| id) + .expect("the index exists"); + rows_in(engine, &crate::data::index_table_name(&id)).await +} + +/// `(index_id, index_status, backfilling)` of the table's vector indexes. +async fn vector_indexes( + engine: &SqliteEngine, + table_id: &str, +) -> Vec<(String, String, Option)> { + sqlx::query_as( + "SELECT index_id, index_status, backfilling FROM vector_indexes WHERE table_id = ?", + ) + .bind(table_id) + .fetch_all(&engine.pool) + .await + .expect("read vector_indexes") +} + +async fn vector_rows(engine: &SqliteEngine, table_id: &str) -> i64 { + let (id, _, _) = vector_indexes(engine, table_id) + .await + .pop() + .expect("the vector index exists"); + rows_in(engine, &crate::data::vector_table_name(table_id, &id)).await +} + +async fn progress(engine: &SqliteEngine, table_id: &str) -> (Option, Option) { + let (_, _, _, _, _, file_index, items_applied, _) = record_columns(engine, table_id).await; + (file_index, items_applied) +} + +async fn tables_named(engine: &SqliteEngine, name: &str) -> i64 { + sqlx::query_scalar("SELECT COUNT(*) FROM tables WHERE account_id = ? AND table_name = ?") + .bind(ACCOUNT) + .bind(name) + .fetch_one(&engine.pool) + .await + .expect("count tables") +} + +// --- the registration ----------------------------------------------------------- + +/// `claim_restore` takes the name's slot and creates the CREATING target with +/// no scheduled transition and its RUNNING `manifest` record carrying the +/// store, the spool host and the claim's generation in one transaction; the +/// description carries the `RestoreSummary`; the record reads back through +/// `read_restore_job` and `list_restore_jobs`; the heartbeat confirms and +/// refreshes the row; the secondary indexes are ACTIVE from creation and the +/// vector index CREATING with its table; dropping the claim frees the slot. +#[tokio::test] +async fn claim_restore_registers_the_target_with_its_record() { + let engine = engine().await; + let row = available_backup(&engine, "Music").await; + let slot = restore_slot_key(ACCOUNT, "Restored"); + assert!(!engine.job_slots.holds(&slot)); + let (desc, key, claim) = register(&engine, &row, "Restored", true, Some("host-a:/spool")).await; + assert_eq!(desc.table_status, TableStatus::Creating); + let summary = desc.restore_summary.as_ref().expect("a RestoreSummary"); + assert_eq!( + summary.source_backup_arn.as_deref(), + Some(row.backup_arn.as_str()) + ); + assert_eq!( + summary.source_table_arn.as_deref(), + Some(format!("arn:aws:dynamodb:{REGION}:{ACCOUNT}:table/Music").as_str()) + ); + assert_eq!(summary.restore_date_time, row.created_at); + assert!(summary.restore_in_progress); + assert!( + engine.job_slots.holds(&slot), + "the claim holds the name's slot" + ); + + let (status, kind, generation, heartbeat_at, spool_host, file_index, items_applied, done) = + record_columns(&engine, &key.table_id).await; + assert_eq!((status.as_str(), kind.as_str()), ("RUNNING", "manifest")); + assert_eq!(generation, i64::try_from(claim.generation()).unwrap()); + assert!(heartbeat_at.starts_with("20"), "{heartbeat_at}"); + assert_eq!(spool_host.as_deref(), Some("host-a:/spool")); + assert_eq!((file_index, items_applied, done), (None, None, None)); + let transition: Option = + sqlx::query_scalar("SELECT status_transition_at FROM tables WHERE table_id = ?") + .bind(&key.table_id) + .fetch_one(&engine.pool) + .await + .expect("read the transition"); + assert_eq!( + transition, None, + "no scheduled transition: the job flips the table" + ); + + let job = record(&engine, &key).await; + assert_eq!(job.target, key); + assert_eq!(job.source_backup_arn, row.backup_arn); + assert_eq!(job.kind, RestoreKind::Manifest); + assert_eq!(job.status, JobStatus::Running); + assert_eq!(job.owner_generation, claim.generation()); + assert_eq!(job.store, Some(store())); + assert_eq!(job.spool_host.as_deref(), Some("host-a:/spool")); + assert_eq!((job.file_index, job.items_applied), (None, None)); + assert_eq!( + (job.failure_reason.as_deref(), job.completed_at), + (None, None) + ); + let running = engine + .list_restore_jobs(Some(JobStatus::Running)) + .await + .expect("list"); + assert_eq!(running, vec![job.clone()]); + assert_eq!( + engine.list_restore_jobs(None).await.expect("list all"), + vec![job.clone()] + ); + assert!( + engine + .list_restore_jobs(Some(JobStatus::Done)) + .await + .expect("list done") + .is_empty() + ); + assert_eq!( + engine.backup_in_use(&row.key()).await.expect("in use"), + Some("Restored".to_owned()), + "the backup is in use while its target is CREATING" + ); + assert!( + engine + .list_dead_restore_jobs(STALE) + .await + .expect("dead") + .is_empty(), + "a fresh record is not dead" + ); + + tokio::time::sleep(Duration::from_millis(20)).await; + assert_eq!( + claim.heartbeat().await.expect("heartbeat"), + Heartbeat::Confirmed + ); + let after = record(&engine, &key).await; + assert!( + after.heartbeat_at > job.heartbeat_at, + "{} > {}", + after.heartbeat_at, + job.heartbeat_at + ); + assert_eq!( + after.owner_generation, + claim.generation(), + "the heartbeat left the rest of the record" + ); + let read = claim.read().await.expect("read through the claim"); + assert_eq!(read.status, JobStatus::Running); + assert_eq!(read.owner_generation, claim.generation()); + + for (name, _, status) in indexes(&engine, &key.table_id).await { + assert_eq!(status, "ACTIVE", "secondary index {name} is created ACTIVE"); + } + let vectors = vector_indexes(&engine, &key.table_id).await; + assert_eq!(vectors.len(), 1); + assert_eq!( + (vectors[0].1.as_str(), vectors[0].2), + ("CREATING", None), + "the vector index is CREATING with its table and not a backfill" + ); + + drop(claim); + assert!( + !engine.job_slots.holds(&slot), + "dropping the claim frees the slot" + ); + assert_eq!( + table_status(&engine, "Restored").await.as_deref(), + Some("CREATING") + ); +} + +/// DeleteBackup winning: `start_backup_delete` moves the row to DELETING and a +/// registration after it finds the backup gone, creates no target, keeps no +/// slot and writes no record. The two also serialize when they contend on +/// the write lock: started together behind it, the outcomes are one of the +/// two consistent pairs and never a target beside a DELETING row. +#[tokio::test] +async fn delete_backup_winning_before_the_registration_leaves_no_target() { + let engine = engine().await; + let row = available_backup(&engine, "Music").await; + assert_eq!( + engine + .start_backup_delete(&row.key()) + .await + .expect("delete"), + DeleteStart::Deleting + ); + let refused = engine + .claim_restore( + ACCOUNT, + target_input("Restored", false), + ®istration(&row, None), + ) + .await + .err() + .expect("the registration finds the backup gone"); + assert!( + is_backup_not_found(&refused), + "the registration finds the backup gone: {refused:?}" + ); + assert_eq!(tables_named(&engine, "Restored").await, 0, "no target"); + assert!( + !engine + .job_slots + .holds(&restore_slot_key(ACCOUNT, "Restored")) + ); + assert!( + engine + .list_restore_jobs(None) + .await + .expect("list") + .is_empty(), + "no record" + ); + + // Contended: both behind the write lock, released together. + let engine = Arc::new(engine); + for attempt in 0..6 { + let row = available_backup(&engine, &format!("Music{attempt}")).await; + let name = format!("Restored{attempt}"); + let held = engine.write_lock.lock().await; + let registering = tokio::spawn({ + let engine = Arc::clone(&engine); + let row = row.clone(); + let name = name.clone(); + async move { + engine + .claim_restore( + ACCOUNT, + target_input(&name, false), + ®istration(&row, None), + ) + .await + } + }); + let deleting = tokio::spawn({ + let engine = Arc::clone(&engine); + let key = row.key(); + async move { engine.start_backup_delete(&key).await } + }); + tokio::time::sleep(Duration::from_millis(20)).await; + assert!(!registering.is_finished() && !deleting.is_finished()); + drop(held); + let registered = tokio::time::timeout(Duration::from_secs(10), registering) + .await + .expect("the registration ends") + .expect("no panic"); + let deleted = tokio::time::timeout(Duration::from_secs(10), deleting) + .await + .expect("the delete ends") + .expect("no panic") + .expect("start_backup_delete"); + match (registered, deleted) { + (Ok((desc, claim)), DeleteStart::InUse(table)) => { + assert_eq!(table, name, "attempt {attempt}: the registration won"); + assert_eq!( + table_status(&engine, &name).await.as_deref(), + Some("CREATING") + ); + drop(claim); + assert!( + engine + .remove_restore_target(&key_of(&desc)) + .await + .expect("remove") + ); + } + (Err(e), DeleteStart::Deleting) => { + assert!(is_backup_not_found(&e), "attempt {attempt}: {e:?}"); + assert_eq!(tables_named(&engine, &name).await, 0, "attempt {attempt}"); + } + (registered, deleted) => panic!( + "attempt {attempt}: the two outcomes are not one ordering: {:?} / {deleted:?}", + registered.map(|(d, _)| d.table_status) + ), + } + } +} + +/// The registration winning: the backup is in use (`start_backup_delete` +/// answers the target's name), a second registration of the backup into +/// another name is `BackupInUse` naming the target and leaves no slot, and +/// the row stays AVAILABLE. +#[tokio::test] +async fn a_registration_winning_before_delete_backup_keeps_the_backup_in_use() { + let engine = engine().await; + let row = available_backup(&engine, "Music").await; + let (_desc, _key, _claim) = register(&engine, &row, "First", false, None).await; + assert_eq!( + engine + .start_backup_delete(&row.key()) + .await + .expect("delete"), + DeleteStart::InUse("First".to_owned()) + ); + let second = engine + .claim_restore( + ACCOUNT, + target_input("Second", false), + ®istration(&row, None), + ) + .await + .err() + .expect("a second registration of the backup is refused"); + match second { + StorageError::BackupInUse(m) => assert!(m.contains("First"), "{m}"), + other => panic!("{other:?}"), + } + assert_eq!(tables_named(&engine, "Second").await, 0); + assert!(!engine.job_slots.holds(&restore_slot_key(ACCOUNT, "Second"))); + let arn = extenddb_core::backup_format::BackupArn::parse(&row.backup_arn).unwrap(); + assert_eq!( + engine + .read_backup_row(&arn) + .await + .expect("read") + .expect("the row") + .status, + BackupStatus::Available + ); +} + +/// A taken name: an existing table is `TableAlreadyExists`; a CREATING +/// restore target is `TableBeingRestored` naming its backup, both while its +/// claim holds the slot and after the claim is gone (the committed row). +#[tokio::test] +async fn a_taken_name_is_refused_as_create_table_or_as_a_restore_in_progress() { + let engine = engine().await; + let row = available_backup(&engine, "Music").await; + engine + .create_table_impl(ACCOUNT, target_input("Plain", false), false) + .await + .expect("a table"); + let taken = engine + .claim_restore( + ACCOUNT, + target_input("Plain", false), + ®istration(&row, None), + ) + .await + .err() + .expect("the name is taken"); + assert!( + matches!(&taken, StorageError::TableAlreadyExists(n) if n == "Plain"), + "{taken:?}" + ); + assert!(!engine.job_slots.holds(&restore_slot_key(ACCOUNT, "Plain"))); + + let other = available_backup(&engine, "Other").await; + let (_desc, _key, claim) = register(&engine, &row, "Target", false, None).await; + async fn refuse(engine: &SqliteEngine, other: &BackupRow) -> StorageError { + engine + .claim_restore( + ACCOUNT, + target_input("Target", false), + ®istration(other, None), + ) + .await + .err() + .expect("the name is a restore target") + } + // While the first claim holds the slot: refused on the slot, the + // committed CREATING row naming the backup. + match refuse(&engine, &other).await { + StorageError::TableBeingRestored { + table_name, + backup_arn, + } => { + assert_eq!(table_name, "Target"); + assert_eq!(backup_arn, row.backup_arn, "the slot held"); + } + other => panic!("slot held: {other:?}"), + } + // The claim gone, the target still CREATING: refused in the + // registration transaction on the committed row. + drop(claim); + assert!(!engine.job_slots.holds(&restore_slot_key(ACCOUNT, "Target"))); + match refuse(&engine, &other).await { + StorageError::TableBeingRestored { + table_name, + backup_arn, + } => { + assert_eq!(table_name, "Target"); + assert_eq!( + backup_arn, row.backup_arn, + "the slot free, the row committed" + ); + } + other => panic!("slot free: {other:?}"), + } + assert!( + !engine.job_slots.holds(&restore_slot_key(ACCOUNT, "Target")), + "a refused registration returns the slot" + ); +} + +// --- the takeover --------------------------------------------------------------- + +/// `claim_restore_dead`: refused while the owner's claim holds the slot, +/// whatever the heartbeat says; a record whose slot no task holds is taken at +/// once, fresh heartbeat or not, with a fresh generation, the claimant's +/// `spool_host` and `heartbeat_at` refreshed to the database's clock (an +/// aged record is live to the dead listing at once); a third taker is +/// refused; the old generation's heartbeat is lost and its writes match +/// nothing; the successor's FAILED write matches and leaves the progress; a +/// DONE record, a legacy record and a missing one are `None` and hold no +/// slot. +#[tokio::test] +async fn a_dead_record_is_taken_over_with_the_claimants_spool_and_a_fresh_heartbeat() { + let engine = engine().await; + let row = available_backup(&engine, "Music").await; + let (_desc, key, claim) = register(&engine, &row, "Restored", false, Some("host-a:/a")).await; + let old_generation = claim.generation(); + engine + .bulk_load(&key, claim.as_ref(), file(0, 0), stream(items(0..10))) + .await + .expect("a first file"); + assert_eq!(progress(&engine, &key.table_id).await, (Some(0), Some(10))); + assert!( + engine + .claim_restore_dead(&key, STALE, Some("host-b:/b")) + .await + .expect("claim") + .is_none(), + "a live claim's slot refuses the takeover" + ); + // The dead owner's last beat, an hour ago: the record is in the dead + // listing until the takeover refreshes it. + sqlx::query( + "UPDATE table_restores SET heartbeat_at = strftime('%Y-%m-%dT%H:%M:%fZ', 'now', \ + '-1 hour') WHERE table_id = ?", + ) + .bind(&key.table_id) + .execute(&engine.pool) + .await + .expect("age the record"); + let before = record_columns(&engine, &key.table_id).await; + assert_eq!(before.4.as_deref(), Some("host-a:/a")); + + drop(claim); + assert_eq!( + engine + .list_dead_restore_jobs(STALE) + .await + .expect("dead") + .len(), + 1, + "the aged record is listed before the takeover" + ); + let successor = engine + .claim_restore_dead(&key, STALE, Some("host-b:/b")) + .await + .expect("claim") + .expect("a record whose slot no task holds is dead at once"); + assert_ne!(successor.generation(), old_generation); + let after = record_columns(&engine, &key.table_id).await; + assert_eq!(after.0, "RUNNING"); + assert_eq!( + after.2, + i64::try_from(successor.generation()).unwrap(), + "the successor's generation" + ); + assert!( + after.3 > before.3, + "the takeover refreshed heartbeat_at: {} was {}", + after.3, + before.3 + ); + let age_secs: f64 = sqlx::query_scalar( + "SELECT (julianday('now') - julianday(heartbeat_at)) * 86400.0 FROM table_restores \ + WHERE table_id = ?", + ) + .bind(&key.table_id) + .fetch_one(&engine.pool) + .await + .expect("the record's age"); + assert!( + age_secs.abs() < 5.0, + "the refreshed heartbeat is the database's now: {age_secs} s old" + ); + assert!( + engine + .list_dead_restore_jobs(STALE) + .await + .expect("dead") + .is_empty(), + "the taken-over record is live to the listing at once" + ); + assert_eq!( + after.4.as_deref(), + Some("host-b:/b"), + "the claimant's spool" + ); + assert_eq!( + (after.5, after.6), + (Some(0), Some(10)), + "the progress stays" + ); + assert!( + engine + .job_slots + .holds(&restore_slot_key(ACCOUNT, "Restored")) + ); + assert!( + engine + .claim_restore_dead(&key, STALE, None) + .await + .expect("claim") + .is_none(), + "a third taker is refused" + ); + + let old = stale_claim(&engine, &key.table_id, old_generation); + assert_eq!(old.heartbeat().await.expect("beat"), Heartbeat::Lost); + assert_eq!( + old.set_status( + JobStatus::Running, + JobStatus::Failed, + &StatusFields { + failure_reason: Some("old owner".to_owned()), + completed: true, + ..StatusFields::default() + } + ) + .await + .expect("write"), + WriteOutcome::NotMatched + ); + assert_eq!( + engine + .finish_restore(&key, &old, 10, 100) + .await + .expect("finish"), + Finished::NotMatched(format!( + "the job record of Restored is not RUNNING under generation {old_generation}, or \ + the table is not CREATING" + )) + ); + assert_eq!( + table_status(&engine, "Restored").await.as_deref(), + Some("CREATING") + ); + assert_eq!( + successor.heartbeat().await.expect("beat"), + Heartbeat::Confirmed + ); + assert_eq!( + successor + .set_status( + JobStatus::Running, + JobStatus::Failed, + &StatusFields { + item_count: Some(99), + failure_reason: Some("target_not_empty".to_owned()), + completed: true, + ..StatusFields::default() + } + ) + .await + .expect("write"), + WriteOutcome::Matched + ); + let failed = record_columns(&engine, &key.table_id).await; + assert_eq!(failed.0, "FAILED"); + assert!(failed.7.is_some(), "the FAILED write stamps completed_at"); + assert_eq!( + (failed.5, failed.6), + (Some(0), Some(10)), + "a status write leaves the progress; item_count is not items_applied" + ); + let read = successor.read().await.expect("read"); + assert_eq!(read.status, JobStatus::Failed); + assert_eq!(read.failure_reason.as_deref(), Some("target_not_empty")); + assert_eq!( + successor + .set_status( + JobStatus::Running, + JobStatus::Failed, + &StatusFields::default() + ) + .await + .expect("write"), + WriteOutcome::NotMatched, + "a FAILED record is not RUNNING" + ); + assert!( + engine + .list_dead_restore_jobs(STALE) + .await + .expect("dead") + .is_empty(), + "a fresh FAILED record is not listed yet" + ); + drop(successor); + assert!( + !engine + .job_slots + .holds(&restore_slot_key(ACCOUNT, "Restored")), + "dropping the successor frees the slot" + ); + assert!( + engine + .claim_restore_dead(&key, STALE, None) + .await + .expect("claim") + .is_none(), + "a FAILED record is not taken over" + ); + assert!( + !engine + .job_slots + .holds(&restore_slot_key(ACCOUNT, "Restored")), + "a refused takeover holds no slot" + ); + let listed = engine + .list_restore_jobs(Some(JobStatus::Failed)) + .await + .expect("list"); + assert_eq!(listed.len(), 1); + assert_eq!( + listed[0].failure_reason.as_deref(), + Some("target_not_empty") + ); + assert_eq!(listed[0].spool_host.as_deref(), Some("host-b:/b")); + assert!(listed[0].completed_at.is_some()); + + // A legacy record is never taken over: the reconciler fails and removes it. + let legacy = engine + .create_table_impl(ACCOUNT, target_input("Legacy", false), true) + .await + .expect("a CREATING table"); + sqlx::query(&format!( + "INSERT INTO table_restores (table_id, source_backup_arn, source_table_arn, \ + restore_date_time, status, kind, owner_generation, heartbeat_at) \ + VALUES (?, 'arn:legacy', 'arn:aws:dynamodb:us-east-1:123456789012:table/Music', {NOW}, \ + 'RUNNING', 'legacy', 7, '1970-01-01T00:00:00.000Z')", + NOW = crate::backup_catalog::NOW + )) + .bind(&legacy.table_id) + .execute(&engine.pool) + .await + .expect("a legacy record"); + let legacy_key = key_of(&legacy); + assert!( + engine + .claim_restore_dead(&legacy_key, STALE, None) + .await + .expect("claim") + .is_none(), + "a legacy record is not taken over" + ); + assert!(!engine.job_slots.holds(&restore_slot_key(ACCOUNT, "Legacy"))); + let dead = engine.list_dead_restore_jobs(STALE).await.expect("dead"); + assert_eq!(dead.len(), 1, "{dead:?}"); + assert_eq!(dead[0].kind, RestoreKind::Legacy); + assert_eq!(dead[0].target, legacy_key); + assert_eq!( + (dead[0].store.clone(), dead[0].spool_host.clone()), + (None, None) + ); + let missing = RestoreTargetKey { + account_id: ACCOUNT.to_owned(), + table_name: "Nothing".to_owned(), + table_id: "no-such-id".to_owned(), + }; + assert!( + engine + .claim_restore_dead(&missing, STALE, None) + .await + .expect("claim") + .is_none() + ); + assert!( + !engine + .job_slots + .holds(&restore_slot_key(ACCOUNT, "Nothing")) + ); + assert!( + engine + .read_restore_job(&missing) + .await + .expect("read") + .is_none() + ); +} + +// --- the load ------------------------------------------------------------------- + +/// `bulk_load` writes the base rows, every secondary index's rows and the +/// vector rows across the batch boundary, recording `(file_index, +/// items_applied)` on the record; a file resumed at `start_at` records the +/// running total; a key the backup holds twice, within a file or against an +/// earlier file, fails the file with nothing of its batch committed and the +/// progress untouched; an `Err` item fails the file the same way; another +/// account's key is refused before any write. +#[tokio::test] +async fn bulk_load_writes_each_batch_with_its_indexes_and_progress() { + let engine = engine().await; + let row = available_backup(&engine, "Music").await; + let (_desc, key, claim) = register(&engine, &row, "Restored", true, None).await; + let n = RESTORE_BATCH_ITEMS + 500; + let applied = engine + .bulk_load(&key, claim.as_ref(), file(0, 0), stream(items(0..n))) + .await + .expect("load file 0"); + assert_eq!(applied as usize, n); + assert_eq!(base_rows(&engine, &key.table_id).await as usize, n); + assert_eq!( + index_rows(&engine, &key.table_id, "gix").await as usize, + n / 2 + ); + assert_eq!( + index_rows(&engine, &key.table_id, "lix").await as usize, + n.div_ceil(3) + ); + assert_eq!(vector_rows(&engine, &key.table_id).await as usize, n / 5); + assert_eq!( + progress(&engine, &key.table_id).await, + (Some(0), Some(n as i64)) + ); + assert_eq!( + engine.count_restore_target(&key).await.expect("count") as usize, + n + ); + let listed = record(&engine, &key).await; + assert_eq!( + (listed.file_index, listed.items_applied), + (Some(0), Some(n as u64)) + ); + + // A file resumed after five of its items: the record carries the total. + let applied = engine + .bulk_load(&key, claim.as_ref(), file(1, 5), stream(items(n..n + 3))) + .await + .expect("load the rest of file 1"); + assert_eq!(applied, 3); + assert_eq!(progress(&engine, &key.table_id).await, (Some(1), Some(8))); + assert_eq!(base_rows(&engine, &key.table_id).await as usize, n + 3); + + // A key twice within a file. + let mut twice = items(n + 3..n + 6); + twice.push(item(n + 4)); + let e = engine + .bulk_load(&key, claim.as_ref(), file(2, 0), stream(twice)) + .await + .expect_err("a key twice fails the file"); + match &e { + StorageError::Validation(m) => { + assert!( + m.starts_with("holds a primary key the target already has"), + "the backend names no file (the engine prefixes the file's key): {m}" + ); + assert!(m.contains("the backup holds a key twice"), "{m}"); + } + other => panic!("{other:?}"), + } + assert_eq!( + base_rows(&engine, &key.table_id).await as usize, + n + 3, + "nothing of the failed batch committed" + ); + assert_eq!( + index_rows(&engine, &key.table_id, "gix").await as usize, + (n + 3).div_ceil(2), + "no index row of the failed batch" + ); + assert_eq!(progress(&engine, &key.table_id).await, (Some(1), Some(8))); + + // A key of file 0 in file 3. + let mut across = items(n + 6..n + 8); + across.push(item(1)); + let e = engine + .bulk_load(&key, claim.as_ref(), file(3, 0), stream(across)) + .await + .expect_err("a key of an earlier file fails the file"); + assert!( + matches!(&e, StorageError::Validation(m) if m.starts_with("holds a primary key the target already has")), + "{e:?}" + ); + assert_eq!(base_rows(&engine, &key.table_id).await as usize, n + 3); + assert_eq!(progress(&engine, &key.table_id).await, (Some(1), Some(8))); + + // An `Err` item mid-stream: the error, nothing committed. + let failing: BoxStream<'static, Result> = Box::pin( + futures::stream::iter(items(n + 10..n + 12).into_iter().map(Ok)).chain( + futures::stream::once(async { + Err(StorageError::Validation( + "item cannot be restored".to_owned(), + )) + }), + ), + ); + let e = engine + .bulk_load(&key, claim.as_ref(), file(4, 0), failing) + .await + .expect_err("an Err item fails the file"); + assert!( + matches!(&e, StorageError::Validation(m) if m == "item cannot be restored"), + "{e:?}" + ); + assert_eq!(base_rows(&engine, &key.table_id).await as usize, n + 3); + assert_eq!(progress(&engine, &key.table_id).await, (Some(1), Some(8))); + + // Another account's key names no restore target of this account. + let foreign = RestoreTargetKey { + account_id: "111111111111".to_owned(), + ..key.clone() + }; + assert!(matches!( + engine.count_restore_target(&foreign).await, + Err(StorageError::TableNotFound(_)) + )); + assert!(matches!( + engine + .bulk_load(&foreign, claim.as_ref(), file(5, 0), stream(items(0..1))) + .await, + Err(StorageError::TableNotFound(_)) + )); + assert_eq!(base_rows(&engine, &key.table_id).await as usize, n + 3); +} + +/// A resume from a mid-file `items_applied`: the engine hands the rest of the +/// file with `start_at` set, and exactly those items land, the record's total +/// becoming the file's count. +#[tokio::test] +async fn a_resume_from_a_mid_file_progress_applies_exactly_the_rest() { + let engine = engine().await; + let row = available_backup(&engine, "Music").await; + let (_desc, key, claim) = register(&engine, &row, "Restored", false, None).await; + let applied = engine + .bulk_load(&key, claim.as_ref(), file(0, 0), stream(items(0..60))) + .await + .expect("the first run's batch"); + assert_eq!(applied, 60); + assert_eq!(progress(&engine, &key.table_id).await, (Some(0), Some(60))); + // The successor: `start_at` 60, the stream from item 60. + let applied = engine + .bulk_load(&key, claim.as_ref(), file(0, 60), stream(items(60..100))) + .await + .expect("the resumed rest of the file"); + assert_eq!(applied, 40, "exactly the rest"); + assert_eq!(progress(&engine, &key.table_id).await, (Some(0), Some(100))); + assert_eq!(engine.count_restore_target(&key).await.expect("count"), 100); + assert_eq!(index_rows(&engine, &key.table_id, "gix").await, 50); + assert_eq!(index_rows(&engine, &key.table_id, "lix").await, 34); + + // An empty file (a resume whose `start_at` is the file's whole count, + // or a data file with no items) records its progress as complete. + let applied = engine + .bulk_load(&key, claim.as_ref(), file(1, 0), stream(Vec::new())) + .await + .expect("an empty file"); + assert_eq!(applied, 0); + assert_eq!( + progress(&engine, &key.table_id).await, + (Some(1), Some(0)), + "an empty file's progress is recorded" + ); + let applied = engine + .bulk_load(&key, claim.as_ref(), file(2, 7), stream(Vec::new())) + .await + .expect("a file resumed at its end"); + assert_eq!(applied, 0); + assert_eq!( + progress(&engine, &key.table_id).await, + (Some(2), Some(7)), + "`start_at` is the recorded total of an empty rest" + ); +} + +/// A batch closes at the first item whose text takes it past +/// `RESTORE_BATCH_BYTES`, long before the count cap: a file of 100 items of +/// about 350 KB each (35 MB of text against the 32 MiB bound) is written in +/// two transactions, the first committed with its progress before the +/// stream hands over its last item. +#[tokio::test] +async fn a_batch_closes_at_the_byte_bound_before_the_count_cap() { + let engine = engine().await; + let row = available_backup(&engine, "Music").await; + let (_desc, key, claim) = register(&engine, &row, "Restored", false, None).await; + const N: usize = 100; + const TEXT: usize = 350_000; + const { + assert!(N < RESTORE_BATCH_ITEMS); + // 100 items cross the byte bound once. + assert!(N * TEXT > RESTORE_BATCH_BYTES && (N - 2) * TEXT < 2 * RESTORE_BATCH_BYTES); + } + let big = |i: usize| -> Item { + let mut it = item(i); + it.insert("v".to_owned(), AttributeValue::S("x".repeat(TEXT))); + it + }; + // The stream reads the record when asked for the last item: by then the + // byte bound has closed and committed the first batch. + let pool = engine.pool.clone(); + let table_id = key.table_id.clone(); + let seen = Arc::new(std::sync::Mutex::new(None)); + let seen_by_stream = Arc::clone(&seen); + let bounded: BoxStream<'static, Result> = + Box::pin(futures::stream::unfold(0usize, move |i| { + let pool = pool.clone(); + let table_id = table_id.clone(); + let seen = Arc::clone(&seen_by_stream); + async move { + if i >= N { + return None; + } + if i == N - 1 { + let progress: (Option, Option) = sqlx::query_as( + "SELECT file_index, items_applied FROM table_restores WHERE table_id = ?", + ) + .bind(&table_id) + .fetch_one(&pool) + .await + .expect("progress"); + *seen.lock().expect("seen") = Some(progress); + } + Some((Ok(big(i)), i + 1)) + } + })); + let applied = engine + .bulk_load(&key, claim.as_ref(), file(0, 0), bounded) + .await + .expect("load"); + assert_eq!(applied as usize, N); + let (file_index, items_applied) = seen + .lock() + .expect("seen") + .expect("the stream read the record at its last item"); + assert_eq!( + file_index, + Some(0), + "the first batch committed with its progress" + ); + let first_batch = items_applied.expect("the first batch's total") as usize; + assert!( + 0 < first_batch && first_batch < N, + "the first batch closed at the byte bound with {first_batch} of {N} items" + ); + assert!( + first_batch * TEXT < RESTORE_BATCH_BYTES + TEXT + 1_000 * N, + "the batch closed at the first item past the bound: {first_batch} items" + ); + assert_eq!( + progress(&engine, &key.table_id).await, + (Some(0), Some(N as i64)) + ); + assert_eq!(base_rows(&engine, &key.table_id).await as usize, N); +} + +/// A claim lost between two batches of one file: the second batch's progress +/// write matches no record, the batch rolls back whole (base, index and +/// vector rows), and `bulk_load` fails as a lost claim; the first batch and +/// its progress stand. +#[tokio::test] +async fn a_lost_claim_mid_file_rolls_its_batch_back() { + let engine = engine().await; + let row = available_backup(&engine, "Music").await; + let (_desc, key, claim) = register(&engine, &row, "Restored", true, None).await; + let n = RESTORE_BATCH_ITEMS + 200; + // The stream hands the first batch, then takes the record away before + // the first item of the second. + let pool = engine.pool.clone(); + let table_id = key.table_id.clone(); + let taken: BoxStream<'static, Result> = + Box::pin(futures::stream::unfold(0usize, move |i| { + let pool = pool.clone(); + let table_id = table_id.clone(); + async move { + if i >= n { + return None; + } + if i == RESTORE_BATCH_ITEMS { + sqlx::query( + "UPDATE table_restores SET owner_generation = owner_generation + 1 \ + WHERE table_id = ?", + ) + .bind(&table_id) + .execute(&pool) + .await + .expect("another owner takes the record"); + } + Some((Ok(item(i)), i + 1)) + } + })); + let e = engine + .bulk_load(&key, claim.as_ref(), file(0, 0), taken) + .await + .expect_err("the lost claim ends the file"); + assert!( + matches!(&e, StorageError::Internal(m) if m.starts_with("claim lost")), + "{e:?}" + ); + assert_eq!( + base_rows(&engine, &key.table_id).await as usize, + RESTORE_BATCH_ITEMS, + "the second batch's base rows rolled back" + ); + assert_eq!( + index_rows(&engine, &key.table_id, "gix").await as usize, + RESTORE_BATCH_ITEMS / 2, + "the second batch's index rows rolled back" + ); + assert_eq!( + vector_rows(&engine, &key.table_id).await as usize, + RESTORE_BATCH_ITEMS / 5, + "the second batch's vector rows rolled back" + ); + assert_eq!( + progress(&engine, &key.table_id).await, + (Some(0), Some(RESTORE_BATCH_ITEMS as i64)), + "the first batch's progress stands" + ); + assert_eq!(claim.heartbeat().await.expect("beat"), Heartbeat::Lost); +} + +// --- the flip ------------------------------------------------------------------- + +/// `finish_restore`: a stale generation is `NotMatched` and changes nothing; +/// the claim's flips the record DONE with `completed_at` and the progress +/// left, the table ACTIVE with its counts and no transition, and every +/// secondary and vector index ACTIVE in one transaction; the heartbeat is +/// lost after DONE; a second flip is `NotMatched`; the backup is no longer +/// in use; `describe_table` reports ACTIVE with the count. +/// `remove_restore_target` of a CREATING target removes it with its vector +/// data table. +#[tokio::test] +async fn finish_restore_flips_the_table_and_every_index() { + let engine = engine().await; + let row = available_backup(&engine, "Music").await; + let (_desc, key, claim) = register(&engine, &row, "Restored", true, None).await; + engine + .bulk_load(&key, claim.as_ref(), file(0, 0), stream(items(0..30))) + .await + .expect("load"); + // A secondary index left CREATING by hand, beside the vector index the + // creation left CREATING; the vector row's `backfilling` set as a backfill + // in progress leaves it, so the flip's `backfilling = NULL` is observed + // (an ACTIVE row may not carry it: `chk_vector_active_has_no_backfilling`). + sqlx::query( + "UPDATE indexes SET index_status = 'CREATING' WHERE table_id = ? AND index_name = 'gix'", + ) + .bind(&key.table_id) + .execute(&engine.pool) + .await + .expect("a CREATING index"); + sqlx::query("UPDATE vector_indexes SET backfilling = 1 WHERE table_id = ?") + .bind(&key.table_id) + .execute(&engine.pool) + .await + .expect("a backfilling vector index"); + assert_eq!( + vector_indexes(&engine, &key.table_id) + .await + .iter() + .map(|(_, status, backfilling)| (status.as_str(), *backfilling)) + .collect::>(), + vec![("CREATING", Some(true))], + "the fixture's one vector index is CREATING and backfilling before the flip" + ); + + let stale = stale_claim(&engine, &key.table_id, claim.generation() + 1); + assert!(matches!( + engine + .finish_restore(&key, &stale, 30, 3000) + .await + .expect("finish"), + Finished::NotMatched(_) + )); + assert_eq!( + table_status(&engine, "Restored").await.as_deref(), + Some("CREATING") + ); + assert_eq!(record_columns(&engine, &key.table_id).await.0, "RUNNING"); + + assert_eq!( + engine + .finish_restore(&key, claim.as_ref(), 30, 3000) + .await + .expect("finish"), + Finished::Active + ); + let (status, _, _, _, _, file_index, items_applied, completed_at) = + record_columns(&engine, &key.table_id).await; + assert_eq!(status, "DONE"); + assert!(completed_at.is_some(), "DONE stamps completed_at"); + assert_eq!( + (file_index, items_applied), + (Some(0), Some(30)), + "the DONE write leaves the progress" + ); + let table: (String, i64, i64, Option) = sqlx::query_as( + "SELECT table_status, item_count, table_size_bytes, status_transition_at \ + FROM tables WHERE table_id = ?", + ) + .bind(&key.table_id) + .fetch_one(&engine.pool) + .await + .expect("the table row"); + assert_eq!(table, ("ACTIVE".to_owned(), 30, 3000, None)); + for (name, _, status) in indexes(&engine, &key.table_id).await { + assert_eq!(status, "ACTIVE", "every secondary index is ACTIVE ({name})"); + } + for (id, status, backfilling) in vector_indexes(&engine, &key.table_id).await { + assert_eq!( + (status.as_str(), backfilling), + ("ACTIVE", None), + "every vector index is ACTIVE with `backfilling` cleared ({id})" + ); + } + assert_eq!(claim.heartbeat().await.expect("beat"), Heartbeat::Lost); + assert!(matches!( + engine + .finish_restore(&key, claim.as_ref(), 30, 3000) + .await + .expect("finish"), + Finished::NotMatched(_) + )); + assert_eq!( + engine.backup_in_use(&row.key()).await.expect("in use"), + None + ); + let described = engine + .describe_table( + ACCOUNT, + extenddb_core::types::DescribeTableInput { + table_name: "Restored".to_owned(), + }, + ) + .await + .expect("describe"); + assert_eq!(described.table_status, TableStatus::Active); + assert_eq!(described.item_count, 30); + let done = engine + .list_restore_jobs(Some(JobStatus::Done)) + .await + .expect("list"); + assert_eq!(done.len(), 1); + assert!(done[0].completed_at.is_some()); + assert!( + matches!( + engine.count_restore_target(&key).await, + Err(StorageError::TableNotFound(_)) + ), + "an ACTIVE table is no longer a CREATING restore target" + ); + + // Removal of another target: the table, its record, its index and + // vector data tables. + let other = available_backup(&engine, "Other").await; + let (desc, other_key, other_claim) = register(&engine, &other, "Gone", true, None).await; + engine + .bulk_load( + &other_key, + other_claim.as_ref(), + file(0, 0), + stream(items(0..10)), + ) + .await + .expect("load"); + let (vector_id, _, _) = vector_indexes(&engine, &other_key.table_id) + .await + .pop() + .unwrap(); + let data_tables = || async { + sqlx::query_scalar::<_, i64>( + "SELECT COUNT(*) FROM sqlite_master WHERE type = 'table' AND name IN (?, ?)", + ) + .bind(crate::data::data_table_name(&desc.table_id).trim_matches('"')) + .bind(crate::data::vector_table_name(&desc.table_id, &vector_id).trim_matches('"')) + .fetch_one(&engine.pool) + .await + .expect("sqlite_master") + }; + assert_eq!(data_tables().await, 2); + drop(other_claim); + assert!( + engine + .remove_restore_target(&other_key) + .await + .expect("remove") + ); + assert_eq!(tables_named(&engine, "Gone").await, 0); + assert_eq!( + data_tables().await, + 0, + "the data and vector tables are dropped" + ); + assert!( + engine + .read_restore_job(&other_key) + .await + .expect("read") + .is_none() + ); + assert!( + !engine + .remove_restore_target(&other_key) + .await + .expect("remove again") + ); +} diff --git a/crates/storage-sqlite/tests/backup_job_panic.rs b/crates/storage-sqlite/tests/backup_job_panic.rs index d366de136..14bd52ed8 100644 --- a/crates/storage-sqlite/tests/backup_job_panic.rs +++ b/crates/storage-sqlite/tests/backup_job_panic.rs @@ -285,7 +285,7 @@ async fn a_job_task_that_panics_leaves_a_row_the_next_cycle_resolves_without_a_r // One reconciler cycle in the same process: the slot no task holds makes // the row dead at once, and without a manifest it is FAILED. let mut state = ReconcilerState::with_shutdown(shutdown.child_token()); - let report = reconciler::cycle(&*storage, &runtime, stale_job_after, "sqlite", &mut state) + let report = reconciler::cycle(&storage, &runtime, stale_job_after, "sqlite", &mut state) .await .expect("a reconciler cycle"); assert_eq!(report.store_healthy, Some(true), "{report:?}"); @@ -326,7 +326,7 @@ async fn a_job_task_that_panics_leaves_a_row_the_next_cycle_resolves_without_a_r Ok(answer) => panic!("DescribeBackup of the FAILED row answered {}", answer.body), } // A second cycle has nothing left to resolve. - let report = reconciler::cycle(&*storage, &runtime, stale_job_after, "sqlite", &mut state) + let report = reconciler::cycle(&storage, &runtime, stale_job_after, "sqlite", &mut state) .await .expect("a second cycle"); assert!(report.creating_resolved.is_empty(), "{report:?}"); diff --git a/crates/storage-sqlite/tests/restore_job_resume.rs b/crates/storage-sqlite/tests/restore_job_resume.rs new file mode 100644 index 000000000..fcf6dc83f --- /dev/null +++ b/crates/storage-sqlite/tests/restore_job_resume.rs @@ -0,0 +1,487 @@ +// Copyright 2026 ExtendDB contributors +// SPDX-License-Identifier: Apache-2.0 + +//! A store-backed restore on SQLite whose owner died mid-file is resumed by +//! one reconciler cycle in the same process, from the record's +//! `(file_index, items_applied)`, and ends ACTIVE with the backup's item +//! count (design 4(e) and 5.3 step 5 on this backend: a RUNNING `manifest` +//! record whose in-process slot no task holds is dead at once, and the +//! progress on the record is where the resume starts). +//! +//! The engine is built the way the server builds it (the backend's server +//! components factory, its workers spawned through the runtime hooks), the +//! store is a real filesystem store, and the backup is taken through the +//! engine's `dispatch` with data files small enough to make several. The +//! dead owner is played at the storage level: `claim_restore`, `bulk_load` +//! of the first file and of half of the second, the claim dropped (its slot +//! freed) and the record's heartbeat aged as the server's restart ages it. +//! The cycle takes the record over, the resume loads the rest, and the table +//! reaches ACTIVE with every item once; a second cycle finds nothing. + +use std::os::unix::fs::DirBuilderExt; +use std::path::PathBuf; +use std::sync::Arc; +use std::time::Duration; + +use extenddb_backup_store::FilesystemStore; +use extenddb_core::backup_format::{ + BackupArn, EXTENDDB_MANIFEST_FILE, ExtenddbManifest, data_file_key, +}; +use extenddb_core::limits::LimitsConfig; +use extenddb_core::types::Item; +use extenddb_engine::reconciler::{self, ReconcilerState}; +use extenddb_engine::{ + BackupRuntime, ConfiguredStore, JobSettings, OperationContext, check_configured_store, dispatch, +}; +use extenddb_storage::backup_catalog::{ + JobStatus, RestoreKind, RestoreRegistrationRow, RestoreTargetKey, StoreKind, UNOWNED_GENERATION, +}; +use extenddb_storage::restore_engine::BulkLoadFile; +use extenddb_storage::server_components::ServerComponentsOptions; +use extenddb_storage::{ + CancellationToken, RestoreTableOverrides, WorkerContext, + backup_definition::restore_input_from_manifest, +}; +use extenddb_storage_sqlite::SqliteConfig; +use serde_json::json; + +const REGION: &str = "us-east-1"; +const ITEMS: usize = 300; + +/// A scratch directory under the cargo target directory, removed on drop. +struct Scratch(PathBuf); + +impl Drop for Scratch { + fn drop(&mut self) { + let _ = std::fs::remove_dir_all(&self.0); + } +} + +async fn call( + ctx: &OperationContext, + operation: &str, + body: serde_json::Value, +) -> serde_json::Value { + dispatch(operation, body, ctx, "127.0.0.1:0") + .await + .unwrap_or_else(|e| panic!("{operation}: {e:?}")) + .body +} + +async fn wait_table_status(ctx: &OperationContext, table: &str, status: &str) -> serde_json::Value { + let deadline = std::time::Instant::now() + Duration::from_secs(30); + loop { + let desc = call(ctx, "DescribeTable", json!({"TableName": table})).await; + if desc["Table"]["TableStatus"] == status { + return desc; + } + assert!( + std::time::Instant::now() < deadline, + "{table} did not reach {status}: {desc}" + ); + tokio::time::sleep(Duration::from_millis(50)).await; + } +} + +async fn scan_count(ctx: &OperationContext, table: &str, index: Option<&str>) -> usize { + let mut count = 0; + let mut start: Option = None; + loop { + let mut body = json!({"TableName": table, "Select": "COUNT"}); + if let Some(index) = index { + body["IndexName"] = json!(index); + } + if let Some(key) = &start { + body["ExclusiveStartKey"] = key.clone(); + } + let page = call(ctx, "Scan", body).await; + count += usize::try_from(page["Count"].as_u64().expect("Count")).unwrap(); + match page.get("LastEvaluatedKey") { + Some(key) if !key.is_null() => start = Some(key.clone()), + _ => return count, + } + } +} + +/// The items of data file `index` of the backup under `root`, decoded as the +/// engine's load decodes them. +fn file_items( + root: &std::path::Path, + arn: &BackupArn, + manifest: &ExtenddbManifest, + index: usize, +) -> Vec { + let entry = &manifest.data_files[index]; + // Data file keys count from one; `BulkLoadFile::file_index` from zero. + assert_eq!(entry.key, data_file_key(u32::try_from(index + 1).unwrap())); + let compressed = std::fs::File::open(root.join(arn.store_prefix()).join(&entry.key)) + .expect("the data file in the store"); + let mut reader = manifest + .reader_prehashed(entry, compressed) + .expect("a reader"); + let mut items = Vec::new(); + while let Some(item) = reader.next_item().expect("an item") { + items.push(item); + } + assert_eq!(items.len() as u64, entry.item_count); + items +} + +fn stream( + items: Vec, +) -> futures::stream::BoxStream<'static, Result> { + Box::pin(futures::stream::iter(items.into_iter().map(Ok))) +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 4)] +async fn a_dead_restore_is_resumed_by_one_cycle_from_its_recorded_progress() { + let scratch = Scratch( + PathBuf::from(env!("CARGO_TARGET_TMPDIR")) + .join(format!("restore-job-resume-{}", uuid::Uuid::new_v4())), + ); + std::fs::create_dir_all(&scratch.0).expect("scratch directory"); + let db_path = scratch.0.join("catalog.sqlite"); + let root = scratch.0.join("backups"); + std::fs::DirBuilder::new() + .mode(0o700) + .create(&root) + .expect("root"); + + // The engine as the server builds it. + let config = SqliteConfig { + path: db_path.to_str().expect("UTF-8 path").to_owned(), + pool_size: 4, + }; + let components = (extenddb_storage_sqlite::backend().server_components)(&config, REGION, { + let mut options = ServerComponentsOptions::default(); + options.bootstrap_if_uninitialized = true; + options.release = "test"; + options.max_concurrent_jobs = 4; + options + }) + .await + .unwrap_or_else(|e| panic!("server components: {e:?}")); + let storage = components.engine; + let shutdown = CancellationToken::new(); + let (_layer, reload_handle) = + tracing_subscriber::reload::Layer::new(tracing_subscriber::EnvFilter::new("info")); + let worker_ctx = WorkerContext { + metrics: Arc::new(extenddb_core::metrics::MetricsCollector::new()), + catalog_store: components.catalog_store.clone(), + reload_handle, + config_log_level: "info".to_owned(), + shutdown: shutdown.clone(), + }; + let hooks = components + .runtime_hooks + .expect("the SQLite backend has runtime hooks"); + let workers = hooks.spawn_workers(&worker_ctx).await; + let catalog = sqlx::SqlitePool::connect(&format!("sqlite://{}", db_path.display())) + .await + .expect("a second connection to the catalog"); + for (key, value) in [ + ("control_plane_delay_seconds", "0.05"), + ("index_propagation_delay_ms", "0"), + ] { + sqlx::query("INSERT OR REPLACE INTO settings (key, value) VALUES (?, ?)") + .bind(key) + .bind(value) + .execute(&catalog) + .await + .expect("setting"); + } + let account_id: String = + sqlx::query_scalar("SELECT value FROM settings WHERE key = 'default_account_id'") + .fetch_one(&catalog) + .await + .expect("the bootstrapped account"); + + // A healthy filesystem store, its identity file written by the check + // the server runs at startup; data files of about 4 KB so the backup + // has several. + let store = Arc::new( + FilesystemStore::open(&root) + .await + .expect("filesystem store"), + ); + let configured = ConfiguredStore::checking( + StoreKind::Filesystem, + root.to_str().expect("UTF-8 path").to_owned(), + store, + None, + ); + let outcome = check_configured_store(&configured, &*storage, "sqlite").await; + assert!(outcome.is_ok(), "the store's first check: {outcome:?}"); + configured.record_check(outcome); + let stale_job_after = Duration::from_secs(300); + let runtime = Arc::new( + BackupRuntime::new(4) + .with_stale_job_after(stale_job_after) + .with_store(configured) + .with_source("sqlite", "test") + .with_job_settings(JobSettings { + data_file_target_bytes: 4096, + ..JobSettings::default() + }) + .with_limits(Arc::new(LimitsConfig::default())) + .with_shutdown(shutdown.child_token()), + ); + let ctx = OperationContext { + storage: storage.clone(), + backup: runtime.clone(), + limits: Arc::new(LimitsConfig::default()), + region: Arc::from(REGION), + account_id: Arc::from(account_id.as_str()), + import_paths: Arc::from(Vec::new()), + export_paths: Arc::from(Vec::new()), + authorized_key_info: Default::default(), + auth_cache: extenddb_auth::AuthCacheRegistry::empty(), + table_key_info_lookup: None, + }; + + call( + &ctx, + "CreateTable", + json!({ + "TableName": "Music", + "KeySchema": [ + {"AttributeName": "pk", "KeyType": "HASH"}, + {"AttributeName": "sk", "KeyType": "RANGE"} + ], + "AttributeDefinitions": [ + {"AttributeName": "pk", "AttributeType": "S"}, + {"AttributeName": "sk", "AttributeType": "N"}, + {"AttributeName": "g", "AttributeType": "S"} + ], + "BillingMode": "PAY_PER_REQUEST", + "GlobalSecondaryIndexes": [{ + "IndexName": "gix", + "KeySchema": [{"AttributeName": "g", "KeyType": "HASH"}], + "Projection": {"ProjectionType": "KEYS_ONLY"} + }] + }), + ) + .await; + wait_table_status(&ctx, "Music", "ACTIVE").await; + for i in 0..ITEMS { + let mut item = json!({ + "pk": {"S": format!("p{}", i % 7)}, + "sk": {"N": i.to_string()}, + "note": {"S": format!("item {i} of the fixture, padded to make the files small {}", "x".repeat(40))} + }); + if i % 2 == 0 { + item["g"] = json!({"S": format!("g{}", i % 4)}); + } + call(&ctx, "PutItem", json!({"TableName": "Music", "Item": item})).await; + } + + // The backup, to the store, AVAILABLE. + let details = call( + &ctx, + "CreateBackup", + json!({"TableName": "Music", "BackupName": "nightly"}), + ) + .await; + let arn_text = details["BackupDetails"]["BackupArn"] + .as_str() + .expect("an ARN") + .to_owned(); + let deadline = std::time::Instant::now() + Duration::from_secs(60); + loop { + let described = call(&ctx, "DescribeBackup", json!({"BackupArn": arn_text})).await; + if described["BackupDescription"]["BackupDetails"]["BackupStatus"] == "AVAILABLE" { + break; + } + assert!( + std::time::Instant::now() < deadline, + "the backup did not become AVAILABLE: {described}" + ); + tokio::time::sleep(Duration::from_millis(50)).await; + } + while runtime.running_jobs() > 0 { + tokio::time::sleep(Duration::from_millis(20)).await; + } + let arn = BackupArn::parse(&arn_text).expect("a backup ARN"); + let row = storage + .read_backup_row(&arn) + .await + .expect("read the row") + .expect("the row exists"); + let manifest_bytes = std::fs::read(root.join(arn.store_prefix()).join(EXTENDDB_MANIFEST_FILE)) + .expect("manifest"); + let manifest = ExtenddbManifest::from_json_bytes(&manifest_bytes).expect("a manifest"); + assert!( + manifest.data_files.len() >= 3, + "the backup has several data files: {}", + manifest.data_files.len() + ); + let total: u64 = manifest.data_files.iter().map(|f| f.item_count).sum(); + assert_eq!(total as usize, ITEMS); + + // The dead owner: the registration, file 0 whole, half of file 1, then + // gone (the claim dropped frees the slot; the heartbeat aged as the + // server's restart ages every RUNNING record). + let input = restore_input_from_manifest( + &manifest.source_table, + "Restored", + &RestoreTableOverrides::default(), + ); + let registration = RestoreRegistrationRow { + backup: row.key(), + source_table_arn: manifest.source_table.table_arn.clone(), + restore_date_time: row.created_at, + store: row.store.clone(), + spool_host: None, + owner_generation: UNOWNED_GENERATION, + }; + let (desc, claim) = storage + .claim_restore(&account_id, input, ®istration) + .await + .expect("register the restore"); + let key = RestoreTargetKey { + account_id: account_id.clone(), + table_name: desc.table_name.clone(), + table_id: desc.table_id.clone(), + }; + let file0 = file_items(&root, &arn, &manifest, 0); + let file1 = file_items(&root, &arn, &manifest, 1); + let half = file1.len() / 2; + assert!(half > 0); + storage + .bulk_load( + &key, + claim.as_ref(), + BulkLoadFile { + file_index: 0, + start_at: 0, + reapplying: false, + }, + stream(file0.clone()), + ) + .await + .expect("file 0"); + storage + .bulk_load( + &key, + claim.as_ref(), + BulkLoadFile { + file_index: 1, + start_at: 0, + reapplying: false, + }, + stream(file1[..half].to_vec()), + ) + .await + .expect("half of file 1"); + let dead_generation = claim.generation(); + drop(claim); + let before = storage + .read_restore_job(&key) + .await + .expect("read") + .expect("the record"); + assert_eq!(before.status, JobStatus::Running); + assert_eq!(before.kind, RestoreKind::Manifest); + assert_eq!( + (before.file_index, before.items_applied), + (Some(1), Some(half as u64)) + ); + assert_eq!( + storage.count_restore_target(&key).await.expect("count") as usize, + file0.len() + half + ); + sqlx::query( + "UPDATE table_restores SET heartbeat_at = '1970-01-01T00:00:00.000Z' WHERE table_id = ?", + ) + .bind(&key.table_id) + .execute(&catalog) + .await + .expect("age the heartbeat as a restart does"); + let desc_before = call(&ctx, "DescribeTable", json!({"TableName": "Restored"})).await; + assert_eq!(desc_before["Table"]["TableStatus"], "CREATING"); + assert!( + desc_before["Table"]["RestoreSummary"]["RestoreInProgress"] + .as_bool() + .unwrap_or(false), + "{desc_before}" + ); + + // One cycle: the record's slot no task holds makes it dead at once; the + // resume is spawned through the runtime. + let mut state = ReconcilerState::with_shutdown(shutdown.child_token()); + let report = reconciler::cycle(&storage, &runtime, stale_job_after, "sqlite", &mut state) + .await + .expect("a reconciler cycle"); + assert_eq!(report.store_healthy, Some(true), "{report:?}"); + assert_eq!( + report.manifest_restores_resumed, + vec!["Restored".to_owned()], + "{report:?}" + ); + assert!(report.manifest_restores_failed.is_empty(), "{report:?}"); + assert!(report.legacy_restores_removed.is_empty(), "{report:?}"); + let taken = storage + .read_restore_job(&key) + .await + .expect("read") + .expect("the record"); + assert_ne!( + taken.owner_generation, dead_generation, + "the takeover minted a generation" + ); + assert_eq!(taken.spool_host, None, "a filesystem store has no spool"); + let deadline = std::time::Instant::now() + Duration::from_secs(60); + while runtime.running_jobs() > 0 { + assert!( + std::time::Instant::now() < deadline, + "the resumed restore did not end" + ); + tokio::time::sleep(Duration::from_millis(20)).await; + } + + // The table ACTIVE with every item once; the record DONE at the last + // file's count; the GSI answering. + let desc_after = wait_table_status(&ctx, "Restored", "ACTIVE").await; + assert_eq!(desc_after["Table"]["ItemCount"], json!(ITEMS)); + assert!( + desc_after["Table"].get("RestoreSummary").is_none(), + "{desc_after}" + ); + assert_eq!(scan_count(&ctx, "Restored", None).await, ITEMS); + assert_eq!(scan_count(&ctx, "Restored", Some("gix")).await, ITEMS / 2); + assert_eq!(scan_count(&ctx, "Music", None).await, ITEMS); + let done = storage + .read_restore_job(&key) + .await + .expect("read") + .expect("the record"); + assert_eq!(done.status, JobStatus::Done); + assert!(done.completed_at.is_some()); + let last = manifest.data_files.len() - 1; + assert_eq!( + (done.file_index, done.items_applied), + ( + Some(u32::try_from(last).unwrap()), + Some(manifest.data_files[last].item_count) + ), + "the progress names the last file complete" + ); + assert_eq!(done.owner_generation, taken.owner_generation); + assert_eq!( + storage.backup_in_use(&row.key()).await.expect("in use"), + None + ); + + // A second cycle has nothing left to resume. + let report = reconciler::cycle(&storage, &runtime, stale_job_after, "sqlite", &mut state) + .await + .expect("a second cycle"); + assert!(report.manifest_restores_resumed.is_empty(), "{report:?}"); + assert!(report.manifest_restores_failed.is_empty(), "{report:?}"); + + catalog.close().await; + shutdown.cancel(); + for worker in workers { + let _ = tokio::time::timeout(Duration::from_secs(5), worker).await; + } +} diff --git a/crates/storage/src/backup_catalog.rs b/crates/storage/src/backup_catalog.rs index 23ed2cfdd..efb3111d6 100644 --- a/crates/storage/src/backup_catalog.rs +++ b/crates/storage/src/backup_catalog.rs @@ -28,9 +28,9 @@ use std::time::Duration; use extenddb_core::backup_format::{BackupArn, SourceTableSchema}; use extenddb_core::types::{ - BackupDescription, BackupDetails, BackupSummary, BillingMode, GlobalSecondaryIndexInfo, - LocalSecondaryIndexInfo, ProvisionedThroughput, SourceTableDetails, SourceTableFeatureDetails, - TableDescription, + BackupDescription, BackupDetails, BackupSummary, BillingMode, CreateTableInput, + GlobalSecondaryIndexInfo, LocalSecondaryIndexInfo, ProvisionedThroughput, SourceTableDetails, + SourceTableFeatureDetails, TableDescription, }; use futures::future::BoxFuture; use futures::stream::BoxStream; @@ -70,12 +70,13 @@ pub fn pool_size_for_jobs(max_concurrent_jobs: usize) -> u32 { jobs.saturating_mul(2).saturating_add(10) } -/// How often a running job refreshes the heartbeat of the record it owns, -/// on every backend, until a later release wires `[backup] -/// heartbeat_interval_secs` to the jobs. `BackupConfig::validate` floors -/// `stale_job_after_secs` on this period, so a configured interval shorter -/// than it cannot make the reconciler declare a live job dead between two -/// heartbeats. +/// The floor of a job's heartbeat period on every backend: the store-backed +/// jobs run at `max([backup] heartbeat_interval_secs, JOB_HEARTBEAT_INTERVAL)` +/// (`BackupConfig::effective_heartbeat_interval_secs`, which the server +/// hands the jobs), the in-catalog paths and the maintenance claim at the +/// constant. `BackupConfig::validate` floors `stale_job_after_secs` on this +/// period, so a configured interval shorter than it cannot make the +/// reconciler declare a live job dead between two heartbeats. pub const JOB_HEARTBEAT_INTERVAL: Duration = Duration::from_secs(15); /// The bound a job refusal names, so the text says which one was met @@ -614,21 +615,152 @@ impl RestoreKind { } } -/// A restore job record the reconciler acts on: RUNNING with a dead owner, -/// or FAILED with its target still CREATING ([`BackupCatalog::list_dead_restore_jobs`]). +/// A restore job record: the row [`BackupCatalog::claim_restore`] writes +/// beside a CREATING target, as [`BackupCatalog::read_restore_job`] and +/// [`BackupCatalog::list_restore_jobs`] return it, and the one the +/// reconciler acts on (RUNNING with a dead owner, or FAILED with its target +/// still CREATING, [`BackupCatalog::list_dead_restore_jobs`]). #[derive(Debug, Clone, PartialEq)] pub struct RestoreJobRow { pub target: RestoreTargetKey, pub source_backup_arn: String, pub kind: RestoreKind, - /// RUNNING or FAILED; the reconciler reads it to decide whether the - /// FAILED write is still to be made. + /// RUNNING, DONE or FAILED; the reconciler reads it to decide whether + /// the FAILED write is still to be made. pub status: JobStatus, pub owner_generation: u64, /// Epoch seconds, on the database's clock. pub heartbeat_at: f64, + /// The store the job reads its backup from; `None` on a legacy record, + /// which reads the catalog. + pub store: Option, + /// `:` of the instance that downloads the backup + /// into its spool; `None` for a filesystem store, which is read in + /// place, and on a legacy record. A takeover rewrites it to the + /// claimant's. + pub spool_host: Option, + /// The last data file whose progress the job recorded, as the backend + /// records it (SQLite and Cassandra write it per batch, MongoDB per + /// file; PostgreSQL keeps its progress in the data database and leaves + /// this `None`). + pub file_index: Option, + /// Items of `file_index` applied so far, where the backend records + /// mid-file progress (SQLite and Cassandra). + pub items_applied: Option, + pub failure_reason: Option, + /// Epoch seconds, on the database's clock; set by the DONE and FAILED + /// writes. + pub completed_at: Option, +} + +impl RestoreJobRow { + /// The row as the four backends' `list_dead_restore_jobs` built it + /// before the restore job existed: the six members the reconciler's + /// duties on legacy and dead records read, the rest `None`. + #[must_use] + pub fn reconciler_row( + target: RestoreTargetKey, + source_backup_arn: String, + kind: RestoreKind, + status: JobStatus, + owner_generation: u64, + heartbeat_at: f64, + ) -> Self { + Self { + target, + source_backup_arn, + kind, + status, + owner_generation, + heartbeat_at, + store: None, + spool_host: None, + file_index: None, + items_applied: None, + failure_reason: None, + completed_at: None, + } + } +} + +/// What [`BackupCatalog::claim_restore`] returns: the CREATING target's +/// description with its `RestoreSummary`, and the claim on its record. +pub type RestoreRegistered = (TableDescription, Box); + +/// What a restore's registration writes beside the target table it creates +/// (the design's 5.3 step 4): the backup the job reads, the provenance the +/// target's `RestoreSummary` reports, the store, the claimant's spool and +/// the claim's generation. The record is born RUNNING and `kind = manifest`. +#[derive(Debug, Clone, PartialEq)] +pub struct RestoreRegistrationRow { + pub backup: BackupKey, + /// Reported as `RestoreSummary.SourceTableArn`. + pub source_table_arn: String, + /// Reported as `RestoreSummary.RestoreDateTime`, epoch seconds: the + /// backup's creation time, as the service reports it. + pub restore_date_time: f64, + pub store: StoreIdentity, + /// `:` when the job downloads into a spool (an S3 + /// store); `None` for a filesystem store. + pub spool_host: Option, + /// The claim's fencing token, written on the record with the target. + pub owner_generation: u64, +} + +/// The names of the pause points inside the backends' restore writes, one +/// spelling each: the backend passes its constant to +/// [`restore_pause_point`], and the kill-point harness names the same +/// constant in `EXTENDDB_BACKUP_PAUSE`. The engine's own kill points are its +/// `KillPoint` enum; these four live here because the backends cannot see +/// the engine. +/// +/// SQLite's `bulk_load`: between two batches of one file, after the first +/// batch's transaction committed with its progress. +pub const RESTORE_SQLITE_MID_BATCH: &str = "restore_sqlite_mid_batch"; +/// MongoDB's `bulk_load`: after the file's last `insert_many` returned, +/// before the record's `file_index` write. +pub const RESTORE_MONGODB_AFTER_INSERT_MANY: &str = "restore_mongodb_after_insert_many"; +/// Cassandra's `bulk_load`: after a batch's writes, before its progress +/// lightweight transaction. +pub const RESTORE_CASSANDRA_BEFORE_PROGRESS: &str = "restore_cassandra_before_progress"; +/// PostgreSQL's `finish_restore`: after the ACTIVE flip committed, before +/// the `restore_progress` rows are deleted. +pub const RESTORE_POSTGRES_BEFORE_PROGRESS_DELETE: &str = "restore_postgres_before_progress_delete"; + +/// A pause point inside a backend's `bulk_load` or its progress write, for +/// the binary-level kill-point harness: with `test-hooks` it pauses while +/// `EXTENDDB_BACKUP_PAUSE` names `name` (one of the constants above), until +/// the file `EXTENDDB_BACKUP_PAUSE_RELEASE` names exists, logging `restore +/// job reached kill point ` as the engine's own kill points do (one +/// harness drives both); without the feature it is a no-op. +#[cfg(feature = "test-hooks")] +pub async fn restore_pause_point(name: &str) { + if std::env::var("EXTENDDB_BACKUP_PAUSE").ok().as_deref() != Some(name) { + return; + } + let release = std::env::var("EXTENDDB_BACKUP_PAUSE_RELEASE").ok(); + tracing::warn!( + "restore job reached kill point {name}; paused until {} exists", + release + .as_deref() + .unwrap_or("no release file (the pause does not end)") + ); + loop { + if let Some(release) = &release + && std::path::Path::new(release).exists() + { + tracing::warn!("restore job released from kill point {name}"); + return; + } + tokio::time::sleep(Duration::from_millis(100)).await; + } } +/// [`restore_pause_point`] without `test-hooks`: nothing. +#[cfg(not(feature = "test-hooks"))] +#[allow(clippy::unused_async)] +pub async fn restore_pause_point(_name: &str) {} + /// A legacy CREATING backup row whose create died. #[derive(Debug, Clone, PartialEq, Eq)] pub struct AbandonedLegacyBackup { @@ -867,13 +999,95 @@ pub trait BackupCatalog: Send + Sync { /// RUNNING to FAILED with `reason` on the target's job record, conditional /// on RUNNING; `false` when the record was not RUNNING or is gone. The - /// reconciler's write for a dead legacy restore, which is never resumed. + /// reconciler's write for a dead legacy restore, which is never resumed, + /// and for a manifest restore it cannot resume (a store other than the + /// configured one, a maintenance claim held). fn fail_restore_job<'a>( &'a self, key: &'a RestoreTargetKey, reason: &'a str, ) -> BoxFuture<'a, Result>; + /// A store-backed restore's registration (the design's 5.3 step 4): + /// create the target table CREATING with no scheduled transition and + /// its RUNNING `kind = manifest` job record, `registration`'s store, + /// `spool_host` and generation on it, in one transaction, under the + /// lock on the backup's `backup_index` row that + /// [`Self::start_backup_delete`] takes, re-checking there that the row + /// is AVAILABLE and that no CREATING target of the backup exists. A + /// DeleteBackup and a registration of one backup therefore order wholly + /// one before the other. Returns the CREATING description, its + /// `RestoreSummary` filled from `registration`, and the claim the job + /// heartbeats and writes its status through; the record is born owned, + /// as [`Self::claim_backup`]'s row is. The record's `heartbeat_at` is + /// the database's clock as the backend stamps it at the insert: on + /// PostgreSQL `now()` is the registration transaction's start, so a + /// registration that waited on the `backup_index` row's lock, or whose + /// data DDL after the commit took long, hands back a record whose + /// heartbeat is that much older than its return; the engine's first beat, + /// sent before the registration's reply, refreshes it. + /// + /// # Errors + /// + /// `Validation("Backup not found: ")` ([`is_backup_not_found`]) + /// when the row is missing or not AVAILABLE; `BackupInUse` when a target + /// of the backup is CREATING; `TableAlreadyExists` when the name is + /// taken, or `TableBeingRestored` when it is a CREATING restore target; + /// `LimitExceeded` when the backend admits no further job. + fn claim_restore<'a>( + &'a self, + account_id: &'a str, + input: CreateTableInput, + registration: &'a RestoreRegistrationRow, + ) -> BoxFuture<'a, Result>; + + /// Take over a RUNNING manifest record whose owner is dead (the design's + /// 4(e)): a fresh generation, the claimant's `spool_host` and + /// `heartbeat_at` set to the database's clock, in one conditional write, + /// so a second taker finds the record live at once. The refresh is where + /// this takeover differs from [`Self::claim_dead_backup`]: the backup + /// takeover leaves `heartbeat_at` because the FAILED-prefix duty's stale + /// window counts from it, while nothing waits on a restore record's old + /// heartbeat after a takeover (the FAILED-target duty ages on + /// `completed_at`, which every FAILED write of a resume stamps, and the + /// maintenance claim waits on RUNNING, not on the beat); without the + /// refresh a second reconciler's cycle landing before the successor's + /// first beat (one period after the takeover) would take the same record + /// again. Dead is `heartbeat_at` older than `stale_after` on the + /// database's clock; on PostgreSQL the owner's advisory lock on the + /// target name is tried first, so a stopped owner whose session lives + /// keeps its record, as [`Self::claim_dead_backup`] says; on SQLite the + /// name's in-process slot is the test instead and `stale_after` is not + /// read: a record whose slot no task of this process holds is dead at + /// once (a restart frees every slot and ages every RUNNING record), and + /// one whose slot a task holds is alive whatever its heartbeat says. + /// `None` when the record is not RUNNING, is gone, or has a live owner; + /// the old generation's writes find [`WriteOutcome::NotMatched`] from + /// then on. On Cassandra the takeover also re-applies the target's + /// `active_restores` entry on the backup's row when it is absent (a + /// crash between the target's DDL and the registration), so the backup + /// stays in use for the resumed load. + fn claim_restore_dead<'a>( + &'a self, + key: &'a RestoreTargetKey, + stale_after: Duration, + spool_host: Option<&'a str>, + ) -> BoxFuture<'a, Result>, StorageError>>; + + /// The job record of the target `key` names, of whatever status; `None` + /// when the table is not a restore target with this table id. + fn read_restore_job<'a>( + &'a self, + key: &'a RestoreTargetKey, + ) -> BoxFuture<'a, Result, StorageError>>; + + /// Every job record with `status`, or of every status for `None`: the + /// reconciler's listing of RUNNING targets for its sweeps. + fn list_restore_jobs<'a>( + &'a self, + status: Option, + ) -> BoxFuture<'a, Result, StorageError>>; + /// The distinct `store_id`s the catalog's rows record for a root, of /// every status: the store identity check reads them before it knows an /// id. Empty when no row names the root. diff --git a/crates/storage/src/backup_definition.rs b/crates/storage/src/backup_definition.rs index b8f014f09..db7f5247a 100644 --- a/crates/storage/src/backup_definition.rs +++ b/crates/storage/src/backup_definition.rs @@ -24,6 +24,7 @@ //! alter the meaning of backups already on disk. A reader that finds a version //! it does not know refuses the restore rather than guessing. +use extenddb_core::backup_format::SourceTableSchema; use extenddb_core::types::{ AttributeDefinition, BillingMode, CreateTableInput, GsiInput, KeySchemaElement, KeyType, LsiInput, OnDemandThroughput, ProvisionedThroughput, StreamSpecification, @@ -437,6 +438,75 @@ impl BackupRestoreSource { } } +/// What the engine answers a `RestoreTableFromBackup` with, beside the +/// target's description: the three members of the `RestoreSummary`, from a +/// legacy backup's [`BackupRestoreSource`] or a store-backed backup's +/// manifest. +#[derive(Debug, Clone, PartialEq)] +pub struct RestoreSource { + pub backup_arn: String, + pub source_table_arn: String, + /// Epoch seconds; reported as `RestoreSummary.RestoreDateTime`, as the + /// service does. + pub backup_creation_date_time: f64, +} + +impl From<&BackupRestoreSource> for RestoreSource { + fn from(source: &BackupRestoreSource) -> Self { + Self { + backup_arn: source.backup_arn.clone(), + source_table_arn: source.source_table_arn.clone(), + backup_creation_date_time: source.backup_creation_date_time, + } + } +} + +/// The restore target's `CreateTableInput` for a store-backed backup: the +/// manifest's `source_table` with the request's billing, throughput and SSE +/// overrides applied, by the rule [`BackupTableDefinition::apply_to`] applies +/// to a legacy definition (one rule for both paths: the GSI throughput +/// defaulting, the on-demand normalization, `TableClass` not restored). The +/// engine applies the index and on-demand overrides to the result and +/// validates it with `validate_create_table`. `vector_indexes` is the +/// manifest's list when it is non-empty (a target without vector support has +/// refused it before this is called), else `None`; the recorded SSE +/// description maps back to the specification the backends derive it from +/// (an `ENABLED` description was an enabled `KMS` specification). Streams, +/// TTL and tags are not restored, as the service does not. +#[must_use] +pub fn restore_input_from_manifest( + source: &SourceTableSchema, + target_table_name: &str, + overrides: &RestoreTableOverrides, +) -> CreateTableInput { + let mut input = CreateTableInput { + table_name: target_table_name.to_owned(), + key_schema: source.key_schema.clone(), + attribute_definitions: source.attribute_definitions.clone(), + ..Default::default() + }; + let definition = BackupTableDefinition { + version: BACKUP_DEFINITION_VERSION, + billing_mode: crate::backup_catalog::billing_mode_name(source.billing_mode).to_owned(), + provisioned_throughput: source.provisioned_throughput.clone(), + global_secondary_indexes: source.global_secondary_indexes.clone(), + local_secondary_indexes: source.local_secondary_indexes.clone(), + table_class: source.table_class.clone(), + sse_specification: source + .sse_description + .as_ref() + .filter(|sse| sse.status == "ENABLED") + .map(|_| serde_json::json!({"Enabled": true, "SSEType": "KMS"})), + on_demand_throughput: source.on_demand_throughput.clone(), + vector_index_names: Vec::new(), + time_to_live: None, + stream_specification: None, + }; + definition.apply_to(&mut input, overrides); + input.vector_indexes = source.vector_indexes.clone().filter(|v| !v.is_empty()); + input +} + /// What a backend is asked to do for one `RestoreTableFromBackup`, after the /// engine has resolved and validated it. #[derive(Debug, Clone, PartialEq)] diff --git a/crates/storage/src/lib.rs b/crates/storage/src/lib.rs index f7fbd6cf6..9d99aa136 100755 --- a/crates/storage/src/lib.rs +++ b/crates/storage/src/lib.rs @@ -22,6 +22,7 @@ pub mod management_store_factory; pub mod migrations; pub mod operations; pub mod policy_migration; +pub mod restore_engine; pub mod server_components; pub mod settings_store; pub mod snapshot; @@ -39,7 +40,10 @@ pub use backend::{Backend, BackendAlreadySet, backend_name, set_backend, try_bac pub use backup_catalog::{ BackupCatalog, JobClaim, JobPermit, LegacyBackupReader, MaintenanceClaim, }; -pub use backup_definition::{BackupRestoreSource, RestoreRequest, RestoreTableOverrides}; +pub use backup_definition::{ + BackupRestoreSource, RestoreRequest, RestoreSource, RestoreTableOverrides, +}; +pub use restore_engine::{BulkLoadFile, Finished, RestoreEngine}; pub use snapshot::{SnapshotEngine, SnapshotInfo, TableSnapshot}; pub use transact::{IdempotencyKey, TransactGetOp, TransactWriteOp}; @@ -721,6 +725,7 @@ pub trait StorageEngine: + StreamEngine + BackupCatalog + LegacyBackupReader + + RestoreEngine + SnapshotEngine + WorkerStore + Send @@ -728,7 +733,7 @@ pub trait StorageEngine: { } -// Blanket implementation: any type implementing all 8 traits is a StorageEngine +// Blanket implementation: any type implementing all 9 traits is a StorageEngine impl StorageEngine for T where T: TableEngine + DataEngine @@ -736,6 +741,7 @@ impl StorageEngine for T where + StreamEngine + BackupCatalog + LegacyBackupReader + + RestoreEngine + SnapshotEngine + WorkerStore + Send diff --git a/crates/storage/src/migrations/registry.rs b/crates/storage/src/migrations/registry.rs index 21261634c..160257824 100644 --- a/crates/storage/src/migrations/registry.rs +++ b/crates/storage/src/migrations/registry.rs @@ -66,6 +66,15 @@ migration_ids! { BackupCatalogIndex = 5, "backup_catalog_index", "create the backup catalog index, the restore job fields, the maintenance claim \ record and the catalog id"; + /// Creates `restore_progress` in the PostgreSQL data database, where a + /// store-backed restore records each data file it applied; the other + /// backends keep that progress on the job record and have nothing to do. + /// Declared `Kind::Breaking` on PostgreSQL although a new table is + /// additive by CONTRIBUTING's step 4: declaring `Kind::Additive` needs + /// the old-binary job, which does not exist yet. + RestoreProgress = 6, "restore_progress", + "create the restore progress table of store-backed restores in the PostgreSQL data \ + database"; } /// Highest id that has shipped in a release. Bodies at or below it are frozen: diff --git a/crates/storage/src/restore_engine.rs b/crates/storage/src/restore_engine.rs new file mode 100644 index 000000000..6ce51c080 --- /dev/null +++ b/crates/storage/src/restore_engine.rs @@ -0,0 +1,153 @@ +// Copyright 2026 ExtendDB contributors +// SPDX-License-Identifier: Apache-2.0 + +//! The data-plane side of a store-backed restore: how a backend takes the +//! items of one verified data file into a CREATING target, records how far +//! it got, flips the target ACTIVE, and counts what it holds. +//! +//! The engine (`crates/engine/src/backup/restore.rs`) owns the job: it +//! downloads and verifies every file, decodes and validates the items, and +//! hands them to [`RestoreEngine::bulk_load`] one file at a time. The +//! backend owns the writes: the base rows, the index rows and the vector +//! rows in its own batches and transactions, the progress record its +//! resume reads, and the ACTIVE flip conditioned on the claim. Progress is +//! recorded inside `bulk_load`, never by a separate call, because the +//! backends record it at different points (PostgreSQL per file in the data +//! database, SQLite and Cassandra per batch, MongoDB per file) and only the +//! write that applied the items knows when they are durable. + +use extenddb_core::types::Item; +use futures::future::BoxFuture; +use futures::stream::BoxStream; + +use crate::backup_catalog::{JobClaim, RestoreTargetKey}; +use crate::error::StorageError; + +/// Items per write batch of the SQLite and Cassandra `bulk_load` (SQLite's +/// write transaction, Cassandra's run of upserts before a progress write): +/// the count cap of the batch term of a restore's memory (the design's +/// section 7), and the unit other writers wait for on SQLite. +pub const RESTORE_BATCH_ITEMS: usize = 10_000; + +/// Bytes of serialized item text per write batch of the SQLite and Cassandra +/// `bulk_load`: a batch closes at the first item that takes its `item_data` +/// text past this, so the batch term of a restore's memory is bounded in +/// bytes as well as in items (10,000 items of the 400 KB limit would be +/// 3.9 GiB). Measured on the JSON text each backend writes as the row's +/// `item_data`, not on the service's `item_size_bytes`. +pub const RESTORE_BATCH_BYTES: usize = 32 * 1024 * 1024; + +/// Which data file a [`RestoreEngine::bulk_load`] call carries, and where in +/// it the items start. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct BulkLoadFile { + /// The file's index in the manifest's `data_files`. + pub file_index: u32, + /// Items of this file already applied by an earlier run, which the + /// engine has skipped: the stream holds the rest. The backend records + /// `start_at` plus the items it applied as the file's running total. + pub start_at: u64, + /// The file is the one a resumed job may have applied in part without a + /// record of it (a backend that records progress per file and died + /// after some of the file's writes). A backend whose writes are plain + /// inserts may suppress duplicate-key failures for this file only; + /// elsewhere a duplicate key is the backup's and fails the file. + pub reapplying: bool, +} + +/// What [`RestoreEngine::finish_restore`] did. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum Finished { + /// The record is DONE and the table and every index are ACTIVE. + Active, + /// The record is DONE and the table waits for its index backfill; the + /// backend's worker flips it ACTIVE once every index has left CREATING + /// (MongoDB). + Pending, + /// The record was not RUNNING under the claim's generation, or the + /// table was not a CREATING restore target: nothing was written. Carries + /// the backend's account of what it found. + NotMatched(String), +} + +/// The data-plane operations of a store-backed restore, implemented by each +/// backend beside [`crate::backup_catalog::BackupCatalog`]. +pub trait RestoreEngine: Send + Sync { + /// Write one verified data file's items, from `file.start_at` on, into + /// the CREATING target: the base rows and the rows of every secondary + /// and vector index the target has, in the backend's own batches, and + /// the file's progress as the backend records it (the design's 5.3 step + /// 5), under `claim`'s generation where the backend's record carries it. + /// Returns the items it applied from the stream. + /// + /// A duplicate primary key, within the file or against rows an earlier + /// file wrote, fails the file (the backup holds a key twice), except as + /// [`BulkLoadFile::reapplying`] says. An `Err` item of the stream (a + /// decode or validation failure the engine found) fails the file with + /// that error, and nothing of a failed file is recorded as applied where + /// the backend writes the file in one transaction. + fn bulk_load<'a>( + &'a self, + target: &'a RestoreTargetKey, + claim: &'a dyn JobClaim, + file: BulkLoadFile, + items: BoxStream<'a, Result>, + ) -> BoxFuture<'a, Result>; + + /// The end of a restore (the design's 5.3 step 6): the record RUNNING to + /// DONE and the table and every index CREATING to ACTIVE with + /// `item_count` and `size_bytes`, in one write where the backend can, + /// conditional on the claim's generation and on the record RUNNING and + /// the table CREATING ([`Finished::Active`]); or DONE with the ACTIVE + /// flip left to the backend's index backfill ([`Finished::Pending`]). + /// [`Finished::NotMatched`] when the record or the table was not the + /// claim's. A backend that keeps progress outside the record removes it + /// after the flip. + fn finish_restore<'a>( + &'a self, + target: &'a RestoreTargetKey, + claim: &'a dyn JobClaim, + item_count: u64, + size_bytes: u64, + ) -> BoxFuture<'a, Result>; + + /// The items the target's base table holds: the count check every + /// restore ends with (equal to the manifest's total, else the backup + /// holds a key twice), and the emptiness check a resume with no recorded + /// progress makes. + fn count_restore_target<'a>( + &'a self, + key: &'a RestoreTargetKey, + ) -> BoxFuture<'a, Result>; + + /// The progress rows a backend keeps outside its job record, as + /// `(file_index, items_applied)` pairs: PostgreSQL's `restore_progress` + /// rows in the data database. Empty by default: the other backends + /// carry the progress on the record ([`crate::backup_catalog::RestoreJobRow`]). + fn read_restore_progress<'a>( + &'a self, + key: &'a RestoreTargetKey, + ) -> BoxFuture<'a, Result, StorageError>> { + let _ = key; + Box::pin(async { Ok(Vec::new()) }) + } + + /// Remove progress rows of targets not among `running` (the table ids + /// of RUNNING restore targets, read from the catalog): the reconciler's + /// sweep of rows a crash left behind where the progress lives apart + /// from the record. The candidates are a positive list: the backend + /// reads the distinct table ids the progress rows name first, then the + /// catalog's RUNNING records, and deletes only the rows of ids in the + /// first set that are in neither the second nor `running`; a target + /// whose first row lands after the first read is never a candidate, and + /// one registered before it is RUNNING at the second read, so a row of + /// a live job is never deleted. Returns the rows removed; `0` by + /// default. + fn sweep_restore_progress<'a>( + &'a self, + running: &'a [String], + ) -> BoxFuture<'a, Result> { + let _ = running; + Box::pin(async { Ok(0) }) + } +} diff --git a/devtools/backup-restore-matrix b/devtools/backup-restore-matrix new file mode 100755 index 000000000..589c52479 --- /dev/null +++ b/devtools/backup-restore-matrix @@ -0,0 +1,974 @@ +#!/usr/bin/env python3 +"""One cell of the cross-backend restore matrix, at the binary level. + +A cell proves that a backup taken on the source backend restores, through the +store, into the target backend: the restored table has the source's items, +key schema, secondary indexes, throughput, SSE and (where the target supports +them) vector indexes, and its class is STANDARD. The target's catalog has no +`backup_index` row for the source's backup until `backup sync --adopt-store` +runs (PR 10); the cell stands in for that command by copying the prefix under +a root the target owns and seeding the row at the storage level, so it proves +the restore and not the adoption. `matrix-nightly.yml` runs the twelve plain +cells and the duplicate-key and vector cells through this committed script. + + backup-restore-matrix --source --target + [--items N] [--vector] [--duplicate-key] [--work DIR] + [--bin-sqlite PATH] [--bin-postgres PATH] [--bin-mongodb PATH] [--bin-cassandra PATH] + +Starts a source server from the backend's binary with a filesystem store at +its own root, creates a composite-key table with a GSI and an LSI (and, with +--vector, a vector index), writes --items items carrying every attribute +type, takes a backup and waits for AVAILABLE. Starts the target server with +its own empty root (it writes its own extenddb-store.json), copies the +source's backup prefix under the target root, seeds the target's +backup_index row at the storage level with the target's store id (what +`backup sync --adopt-store` will do in PR 10; the cell proves the restore, +not the adoption), restores, waits for ACTIVE, compares the item set and the +description (key schema, indexes, throughput, SSE; the table class STANDARD), +queries each GSI, and tears both servers down by PID and drops their +databases. With --duplicate-key the copied prefix is replaced by a hand-built +unordered backup whose two data files share one key, and the cell passes +when the restore fails (the target gone, no table) instead. + +The two servers connect to PostgreSQL at 127.0.0.1:55432 (postgres/devpass), +MongoDB at 127.0.0.1:27117 and Cassandra at 127.0.0.1:9042, the coordinates +`matrix-nightly.yml` publishes its services on. Each server is an `init` with +its port patched, `[backup]` with short intervals and `serve --foreground`, +the test credentials provisioned through devtools/provision-test-credentials. +The MongoDB and Cassandra deployments use the backends' fixed names (the +`extenddb_catalog` and `extenddb_data` databases; the `extenddb_catalog` and +`extenddb_account_` keyspaces) and drop them in `init` and +`teardown`, so two cells with a MongoDB or Cassandra role, or a cell beside +another process on those names, destroy each other's catalogs: run such cells +one at a time, as the nightly does. The MongoDB and Cassandra roles need +`pymongo` and `cassandra-driver` (requirements-matrix.txt). + +Logs go under --work (default `$TMPDIR/backup-restore-matrix/`). +Exit 0 when the cell passes, 1 on an assertion failure or a setup error. +""" +from __future__ import annotations + +import argparse +import base64 +import gzip +import hashlib +import json +import os +import re +import shutil +import signal +import subprocess +import sys +import tempfile +import time +import urllib.parse + +import boto3 +import requests +import urllib3 +from botocore.auth import SigV4Auth +from botocore.awsrequest import AWSRequest +from botocore.config import Config +from botocore.exceptions import ClientError + +urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning) + +HERE = os.path.dirname(os.path.abspath(__file__)) +# The script lives at /devtools/backup-restore-matrix; the repo root +# is its parent. A caller overrides the binaries with --bin-. +REPO = os.path.abspath(os.path.join(HERE, "..")) +ACCOUNT = "123456789012" +REGION = "us-east-1" +ADMIN_PASSWORD = "TestPass1!" +PG = {"host": "127.0.0.1", "port": "55432", "user": "postgres", "password": "devpass"} +MONGODB_URI = "mongodb://127.0.0.1:27117/?directConnection=true" + +DEFAULT_BINS = { + "sqlite": os.path.join(REPO, "target", "rel-sqlite", "release", "extenddb"), + "postgres": os.path.join(REPO, "target", "rel-postgres", "release", "extenddb"), # or target/release + "mongodb": os.path.join(REPO, "target", "rel-mongodb", "release", "extenddb"), + "cassandra": os.path.join(REPO, "target", "rel-cassandra", "release", "extenddb"), +} + + +def say(msg: str) -> None: + print(f"[{time.strftime('%H:%M:%S')}] {msg}", flush=True) + + +class CellFailed(Exception): + pass + + +def check(cond: bool, msg: str) -> None: + if not cond: + raise CellFailed(msg) + + +def sqlite_module(): + """pysqlite3 when installed (an old sqlite3 library bundled with the + interpreter cannot read the catalog), else the standard module.""" + try: + import pysqlite3 as sqlite3 # type: ignore + except ImportError: + import sqlite3 # type: ignore + return sqlite3 + + +def free_port() -> int: + import socket + + with socket.socket() as s: + s.bind(("127.0.0.1", 0)) + return s.getsockname()[1] + + +# --- deployments ----------------------------------------------------------------- + + +class Deployment: + """One server: its binary, work directory, config, store root, PID and + the credentials provisioned on it.""" + + def __init__(self, backend: str, binary: str, work: str, role: str): + self.backend = backend + self.bin = binary + self.work = os.path.join(work, role) + self.role = role + os.makedirs(self.work, exist_ok=True) + self.config = os.path.join(self.work, "extenddb.toml") + self.root = os.path.join(self.work, "root") + self.port = free_port() + self.endpoint = f"https://127.0.0.1:{self.port}" + self.tag = f"mx{int(time.time()) % 1_000_000}{role[0]}" + self.proc: subprocess.Popen | None = None + self.creds: dict[str, str] = {} + + # -- init and config -- + + def init(self) -> None: + env = dict(os.environ, EXTENDDB_ADMIN_PASSWORD=ADMIN_PASSWORD) + os.makedirs(self.root, mode=0o700, exist_ok=True) + if self.backend == "mongodb": + # The MongoDB backend uses fixed database names + # (extenddb_catalog / extenddb_data), so a cell drops them before + # it inits and after it tears down, and a MongoDB source and a + # MongoDB target never run at once (the source stops before the + # target starts). + self.mongo_drop_databases() + with open(self.config, "w") as f: + f.write( + f'[storage]\nbackend = "mongodb"\n\n[storage.mongodb]\n' + f'connection_string = "{MONGODB_URI}"\n' + ) + cmd = [self.bin, "init", "--backend", "mongodb", "--config", self.config, + "--overwrite"] + with open(os.path.join(self.work, "init.log"), "w") as log: + r = subprocess.run(cmd, cwd=self.work, env=env, stdout=log, stderr=subprocess.STDOUT) + if r.returncode != 0: + print(open(os.path.join(self.work, "init.log")).read()) + raise SystemExit(f"init of the {self.role} failed") + # `init` regenerates the config without the port, run_dir and + # backup sections the cell needs; write the whole config, as + # devtools/run-mongodb-tests does. + home = os.path.expanduser("~") + with open(self.config, "w") as f: + f.write( + f'[server]\nbind_addr = "127.0.0.1"\nport = {self.port}\n' + f'region = "{REGION}"\nrun_dir = "{self.work}/run"\n\n' + f'[server.tls]\ncert_path = "{home}/.extenddb/tls/cert.pem"\n' + f'key_path = "{home}/.extenddb/tls/key.pem"\n\n' + f'[storage]\nbackend = "mongodb"\n\n[storage.mongodb]\n' + f'connection_string = "{MONGODB_URI}"\n\n' + f'[auth]\nprovider = "builtin"\n\n' + f'[backup]\nreconcile_interval_secs = 2\nstale_job_after_secs = 45\n\n' + f'[backup.store]\nkind = "filesystem"\n' + f'[backup.store.filesystem]\npath = "{self.root}"\n' + ) + return + if self.backend == "sqlite": + cmd = [self.bin, "init", "--backend", "sqlite", "--sqlite-path", + os.path.join(self.work, "db.sqlite"), "--config", self.config] + elif self.backend == "postgres": + cmd = [self.bin, "init", "--pg-host", PG["host"], "--pg-port", PG["port"], + "--pg-user", PG["user"], "--pg-pass", PG["password"], + "--data-db", f"{self.tag}_data", "--catalog-db", f"{self.tag}_catalog", + "--extenddb-user", f"{self.tag}_app", "--config", self.config] + elif self.backend == "cassandra": + cmd = [self.bin, "init", "--backend", "cassandra", + "--cassandra-contact-points", "127.0.0.1:9042", + "--cassandra-user", "cassandra", "--cassandra-pass", "cassandra", + "--config", self.config] + else: + raise NotImplementedError(f"deployment of a {self.backend} server") + with open(os.path.join(self.work, "init.log"), "w") as log: + r = subprocess.run(cmd, cwd=self.work, env=env, stdout=log, stderr=subprocess.STDOUT) + if r.returncode != 0: + print(open(os.path.join(self.work, "init.log")).read()) + raise SystemExit(f"init of the {self.role} failed") + text = open(self.config).read() + text, n = re.subn(r"(?m)^[ \t]*#?[ \t]*port[ \t]*=[ \t]*\d+.*$", f"port = {self.port}", text, count=1) + assert n == 1, "no port line in the config" + text = re.sub(r'(?m)^run_dir = .*$', f'run_dir = "{self.work}/run"', text) + os.makedirs(self.root, mode=0o700, exist_ok=True) + text += ( + "\n[backup]\nreconcile_interval_secs = 2\nstale_job_after_secs = 45\n" + f'\n[backup.store]\nkind = "filesystem"\n[backup.store.filesystem]\npath = "{self.root}"\n' + ) + open(self.config, "w").write(text) + + def mongo_drop_databases(self) -> None: + """Drop the MongoDB backend's fixed catalog and data databases and wait + until each drop has completed. On a replica set `dropDatabase` is + two-phase: the database stays drop-pending until the drop is + majority-committed, a collection created meanwhile fails with + `DatabaseDropPending` (code 215) or vanishes with the database when + the drop lands, and a server started over a pending drop then answers + `ResourceNotFoundException` to writes into a table it reported + ACTIVE. The probe below creates a collection in each database until + the server accepts it, then drops the collection (a collection drop + leaves no pending database).""" + from pymongo import MongoClient + from pymongo.errors import OperationFailure + + client = MongoClient(MONGODB_URI) + try: + for name in ("extenddb_catalog", "extenddb_data"): + client.drop_database(name) + deadline = time.monotonic() + 120 + for name in ("extenddb_catalog", "extenddb_data"): + while True: + try: + client[name].create_collection("drop_probe") + except OperationFailure as e: + if e.code == 215 and time.monotonic() < deadline: + time.sleep(0.25) + continue + raise + client[name].drop_collection("drop_probe") + break + finally: + client.close() + + # -- the process -- + + def start(self, log_name: str = "serve.log", extra_env: dict | None = None) -> None: + env = dict(os.environ, **(extra_env or {})) + self.log = open(os.path.join(self.work, log_name), "a") + self.proc = subprocess.Popen( + [self.bin, "serve", "--foreground", "--config", self.config], + cwd=self.work, env=env, stdout=self.log, stderr=subprocess.STDOUT, + ) + for _ in range(300): + try: + if requests.get(f"{self.endpoint}/health", verify=False, timeout=2).status_code == 200: + break + except requests.RequestException: + pass + if self.proc.poll() is not None: + break + time.sleep(0.2) + else: + self.stop() + raise SystemExit(f"the {self.role} server did not start; see {self.log.name}") + if self.proc.poll() is not None: + raise SystemExit(f"the {self.role} server exited at start; see {self.log.name}") + say(f"{self.role} ({self.backend}) pid {self.proc.pid} on {self.port}") + + def stop(self) -> None: + """SIGTERM by PID, then SIGKILL if it outlives the drain.""" + if self.proc is None or self.proc.poll() is not None: + return + self.proc.send_signal(signal.SIGTERM) + try: + self.proc.wait(timeout=60) + except subprocess.TimeoutExpired: + self.proc.kill() + self.proc.wait(timeout=10) + say(f"{self.role} pid {self.proc.pid} stopped (exit {self.proc.returncode})") + + def teardown(self) -> None: + self.stop() + if self.backend == "postgres": + psql(f"DROP DATABASE IF EXISTS {self.tag}_data WITH (FORCE)") + psql(f"DROP DATABASE IF EXISTS {self.tag}_catalog WITH (FORCE)") + psql(f"DROP ROLE IF EXISTS {self.tag}_app") + elif self.backend == "mongodb": + self.mongo_drop_databases() + elif self.backend == "cassandra": + # The Cassandra target server used the default keyspace prefix + # `extenddb`; drop its catalog and this account's data keyspace. + # The names are fixed, so another cell or process on the same + # node loses its catalog here (module docstring). + try: + from cassandra.auth import PlainTextAuthProvider + from cassandra.cluster import Cluster + + cluster = Cluster( + ["127.0.0.1"], port=9042, + auth_provider=PlainTextAuthProvider("cassandra", "cassandra"), + ) + session = cluster.connect() + for ks in ("extenddb_catalog", f"extenddb_account_{ACCOUNT}"): + session.execute(f"DROP KEYSPACE IF EXISTS {ks}") + cluster.shutdown() + except Exception as e: # noqa: BLE001 + print(f"cassandra teardown: {e}") + + # -- credentials and clients -- + + def provision(self) -> None: + env = dict(os.environ, EXTENDDB_TEST_ENDPOINT=self.endpoint, EXTENDDB_ADMIN_USER="admin", + EXTENDDB_ADMIN_PASSWORD=ADMIN_PASSWORD) + r = subprocess.run([sys.executable, os.path.join(REPO, "devtools", "provision-test-credentials")], + env=env, capture_output=True, text=True) + if r.returncode != 0: + raise SystemExit(f"provisioning on the {self.role} failed: {r.stderr}") + for line in r.stdout.splitlines(): + m = re.match(r"export (\w+)=(.*)", line.strip()) + if m: + self.creds[m.group(1)] = m.group(2) + for key, value in [("control_plane_delay_seconds", "0.05"), ("index_propagation_delay_ms", "0")]: + subprocess.run([self.bin, "settings", "--config", self.config, "set", key, value], + cwd=self.work, capture_output=True) + + def client(self): + return boto3.client( + "dynamodb", endpoint_url=self.endpoint, region_name=REGION, verify=False, + aws_access_key_id=self.creds["AWS_ACCESS_KEY_ID"], + aws_secret_access_key=self.creds["AWS_SECRET_ACCESS_KEY"], + config=Config(retries={"max_attempts": 10, "mode": "standard"}, read_timeout=120), + ) + + def signed_post(self, operation: str, body: dict) -> requests.Response: + """A hand-signed request for an operation botocore's model lacks + (CreateTable with VectorIndexes, DescribeTable's VectorIndexes, + SearchVectors).""" + body_bytes = json.dumps(body).encode() + headers = {"X-Amz-Target": f"DynamoDB_20120810.{operation}", + "Content-Type": "application/x-amz-json-1.0"} + frozen = boto3.Session( + aws_access_key_id=self.creds["AWS_ACCESS_KEY_ID"], + aws_secret_access_key=self.creds["AWS_SECRET_ACCESS_KEY"], + ).get_credentials().get_frozen_credentials() + req = AWSRequest(method="POST", url=self.endpoint, data=body_bytes, headers=headers) + SigV4Auth(frozen, "dynamodb", REGION).add_auth(req) + return requests.post(self.endpoint, data=body_bytes, headers=dict(req.headers), verify=False) + + def store_id(self) -> str: + return json.load(open(os.path.join(self.root, "extenddb-store.json")))["store_id"] + + +def psql(statement: str, database: str = "postgres", tuples: bool = False) -> str: + env = dict(os.environ, PGPASSWORD=PG["password"]) + cmd = ["psql", "-h", PG["host"], "-p", PG["port"], "-U", PG["user"], "-d", database, "-v", "ON_ERROR_STOP=1", "-q"] + if tuples: + cmd.append("-tA") + r = subprocess.run(cmd + ["-c", statement], env=env, capture_output=True, text=True) + if r.returncode != 0: + raise SystemExit(f"psql failed: {r.stderr}\n{statement}") + return r.stdout.strip() + + +# --- the table, the items, the backup -------------------------------------------- + +TABLE = "MatrixSource" +TARGET = "MatrixTarget" + + +def table_input(name: str, vector: bool) -> dict: + body = { + "TableName": name, + "KeySchema": [{"AttributeName": "pk", "KeyType": "HASH"}, {"AttributeName": "sk", "KeyType": "RANGE"}], + "AttributeDefinitions": [ + {"AttributeName": "pk", "AttributeType": "S"}, + {"AttributeName": "sk", "AttributeType": "N"}, + {"AttributeName": "g", "AttributeType": "S"}, + {"AttributeName": "l", "AttributeType": "S"}, + ], + "BillingMode": "PROVISIONED", + "ProvisionedThroughput": {"ReadCapacityUnits": 7, "WriteCapacityUnits": 9}, + "GlobalSecondaryIndexes": [{ + "IndexName": "by-g", + "KeySchema": [{"AttributeName": "g", "KeyType": "HASH"}, {"AttributeName": "sk", "KeyType": "RANGE"}], + "Projection": {"ProjectionType": "ALL"}, + "ProvisionedThroughput": {"ReadCapacityUnits": 3, "WriteCapacityUnits": 4}, + }], + "LocalSecondaryIndexes": [{ + "IndexName": "by-l", + "KeySchema": [{"AttributeName": "pk", "KeyType": "HASH"}, {"AttributeName": "l", "KeyType": "RANGE"}], + "Projection": {"ProjectionType": "KEYS_ONLY"}, + }], + "SSESpecification": {"Enabled": True, "SSEType": "KMS"}, + } + if vector: + # Vector indexes are accepted on on-demand tables only. + body["BillingMode"] = "PAY_PER_REQUEST" + del body["ProvisionedThroughput"] + del body["GlobalSecondaryIndexes"][0]["ProvisionedThroughput"] + body["VectorIndexes"] = [{ + "IndexName": "by-emb", "Dimensions": 3, "DistanceFunction": "COSINE", + "VectorAttribute": {"AttributeName": "emb"}, "Projection": {"ProjectionType": "ALL"}, + }] + return body + + +def item(i: int, vector: bool) -> dict: + """Item `i` with every attribute type; every second item carries the GSI + key, every third the LSI key, every fifth the vector.""" + it = { + "pk": {"S": f"p{i % 11:03d}"}, + "sk": {"N": str(i)}, + "s": {"S": f"text {i} \u00e9\u2028"}, + "n": {"N": f"{i}.25"}, + "big": {"N": "12345678901234567890123456789012345678"}, + "b": {"B": bytes([i % 256, 0, 255, 7])}, + "ss": {"SS": [f"a{i}", "b", "c"]}, + "ns": {"NS": ["1", "2.5", str(-i)]}, + "bs": {"BS": [b"\x00\x01", bytes([i % 256])]}, + "bool": {"BOOL": i % 2 == 0}, + "null": {"NULL": True}, + "list": {"L": [{"S": "x"}, {"N": "1"}, {"M": {"k": {"BOOL": False}}}]}, + "map": {"M": {"nested": {"M": {"deep": {"L": [{"NULL": True}]}}}, "n": {"N": "3"}}}, + } + if i % 2 == 0: + it["g"] = {"S": f"g{i % 4}"} + if i % 3 == 0: + it["l"] = {"S": f"l{i:05d}"} + if vector and i % 5 == 0: + it["emb"] = {"L": [{"N": "0.1"}, {"N": str(0.01 * (i % 50))}, {"N": "0.3"}]} + return it + + +def create_and_fill(d: Deployment, items: int, vector: bool) -> None: + body = table_input(TABLE, vector) + r = d.signed_post("CreateTable", body) + check(r.status_code == 200, f"CreateTable on the source: {r.status_code} {r.text}") + c = d.client() + wait_active(d, TABLE) + reqs = [] + for i in range(items): + reqs.append({"PutRequest": {"Item": item(i, vector)}}) + if len(reqs) == 25 or i == items - 1: + pending = {TABLE: reqs} + while pending: + pending = c.batch_write_item(RequestItems=pending).get("UnprocessedItems") or {} + reqs = [] + say(f"source: {items} items written") + + +def wait_active(d: Deployment, name: str, timeout: float = 600) -> dict: + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + desc = d.signed_post("DescribeTable", {"TableName": name}) + if desc.status_code == 200: + table = desc.json()["Table"] + indexes = (table.get("GlobalSecondaryIndexes") or []) + (table.get("VectorIndexes") or []) + if table["TableStatus"] == "ACTIVE" and all(x.get("IndexStatus", "ACTIVE") == "ACTIVE" for x in indexes): + return table + time.sleep(0.2) + raise CellFailed(f"{name} did not become ACTIVE on the {d.role} within {timeout} s") + + +def take_backup(d: Deployment) -> tuple[str, dict]: + c = d.client() + arn = c.create_backup(TableName=TABLE, BackupName="matrix")["BackupDetails"]["BackupArn"] + deadline = time.monotonic() + 600 + while True: + desc = c.describe_backup(BackupArn=arn)["BackupDescription"] + status = desc["BackupDetails"]["BackupStatus"] + if status == "AVAILABLE": + say(f"source: backup {arn} AVAILABLE ({desc['BackupDetails']['BackupSizeBytes']} bytes)") + return arn, desc + check(status == "CREATING", f"the backup is {status}") + check(time.monotonic() < deadline, "the backup did not become AVAILABLE") + time.sleep(0.2) + + +def copy_prefix(source_prefix: str, target_prefix: str, target_root: str) -> None: + """Copy the backup's prefix under the target root, every directory + 0700 and every file 0600 (the store refuses a root others can read).""" + rel = os.path.relpath(os.path.dirname(target_prefix), target_root) + path = target_root + for part in rel.split(os.sep): + path = os.path.join(path, part) + if not os.path.isdir(path): + os.mkdir(path, 0o700) + os.chmod(path, 0o700) + shutil.copytree(source_prefix, target_prefix) + for dirpath, dirnames, filenames in os.walk(target_prefix): + os.chmod(dirpath, 0o700) + for name in filenames: + os.chmod(os.path.join(dirpath, name), 0o600) + + +def prefix_of(root: str, arn: str) -> str: + backup_id = arn.rsplit("/", 1)[1] + table = arn.split(":table/")[1].split("/")[0] + return os.path.join(root, ACCOUNT, table, backup_id) + + +# --- a hand-built unordered backup with one key in two files ----------------------- + + +def write_duplicate_key_prefix(prefix: str, manifest: dict, vector: bool) -> dict: + """Replace the data files under `prefix` with two unordered files that + share the key of item 0, and rewrite the manifest and its sidecar; every + file written here is 0600 and the data directory 0700, as `copy_prefix` + leaves the rest of the prefix.""" + data_dir = os.path.join(prefix, "data") + shutil.rmtree(data_dir) + os.makedirs(data_dir, mode=0o700) + os.chmod(data_dir, 0o700) + files = [] + total = 0 + for index, rows in enumerate([[item(i, vector) for i in range(0, 6)], [item(i, vector) for i in range(6, 12)] + [item(0, vector)]], start=1): + lines = b"".join(json.dumps({"Item": r}, separators=(",", ":"), default=b64).encode() + b"\n" for r in rows) + import io + + buffer = io.BytesIO() + with gzip.GzipFile(fileobj=buffer, mode="wb", mtime=0) as gz: + gz.write(lines) + compressed = buffer.getvalue() + key = f"data/{index:06d}.json.gz" + with open(os.path.join(prefix, key), "wb") as f: + f.write(compressed) + os.chmod(os.path.join(prefix, key), 0o600) + files.append({ + "key": key, "item_count": len(rows), + "sha256": hashlib.sha256(compressed).hexdigest(), + "md5": base64.b64encode(hashlib.md5(compressed).digest()).decode(), + "bytes": len(compressed), "uncompressed_bytes": len(lines), "key_range": None, + }) + total += len(compressed) + manifest = dict(manifest) + manifest["item_order"] = "unordered" + manifest["data_files"] = files + manifest["backup_size_bytes"] = total + manifest["source_table"] = dict(manifest["source_table"], item_count=13) + text = json.dumps(manifest, separators=(",", ":")).encode() + with open(os.path.join(prefix, "extenddb-manifest.json"), "wb") as f: + f.write(text) + os.chmod(os.path.join(prefix, "extenddb-manifest.json"), 0o600) + with open(os.path.join(prefix, "extenddb-manifest.sha256"), "w") as f: + f.write(f"{hashlib.sha256(text).hexdigest()} extenddb-manifest.json\n") + os.chmod(os.path.join(prefix, "extenddb-manifest.sha256"), 0o600) + write_ddb_manifests(prefix, files, total) + return manifest + + +def write_ddb_manifests(prefix: str, files: list[dict], size_bytes: int) -> None: + """Rewrite the two DynamoDB export manifests for the data files now under + `prefix`, so the prefix is a whole backup in the format's sense (one + `md5Checksum` and one `itemCount` per file, agreeing with the + extenddb manifest by construction): `manifest-files.json` is one line per + file, `manifest-summary.json` keeps the source's document with the new + item count and size.""" + summary_path = os.path.join(prefix, "manifest-summary.json") + with open(summary_path) as f: + summary = json.load(f) + s3_prefix = summary["s3Prefix"] + with open(os.path.join(prefix, "manifest-files.json"), "w") as f: + for entry in files: + line = { + "itemCount": entry["item_count"], + "md5Checksum": entry["md5"], + "dataFileS3Key": f"{s3_prefix}/{entry['key']}" if s3_prefix else entry["key"], + } + f.write(json.dumps(line, separators=(",", ":")) + "\n") + os.chmod(os.path.join(prefix, "manifest-files.json"), 0o600) + summary["itemCount"] = sum(entry["item_count"] for entry in files) + summary["billedSizeBytes"] = size_bytes + with open(summary_path, "w") as f: + f.write(json.dumps(summary, separators=(",", ":"))) + os.chmod(summary_path, 0o600) + + +def b64(value): + if isinstance(value, bytes): + return base64.b64encode(value).decode() + raise TypeError(type(value)) + + +# --- the seed ------------------------------------------------------------------ + + +def seed_row(target: Deployment, arn: str, manifest: dict, size_bytes: int, item_count: int) -> None: + """Insert the target's `backup_index` row for the copied prefix, as + `backup sync` will (PR 10): AVAILABLE, the target's store identity, the + manifest's source table, `created_at` from the backup id.""" + backup_id = arn.rsplit("/", 1)[1] + table = arn.split(":table/")[1].split("/")[0] + created_ms = int(backup_id.split("-")[0]) + # `created_at` as the job writes it (the id's millisecond, six fraction + # digits on SQLite); `completed_at` and `heartbeat_at` are the seed's own + # instant, distinct from `created_at` as the job's AVAILABLE write and + # its last beat are. + created_iso = time.strftime("%Y-%m-%dT%H:%M:%S", time.gmtime(created_ms / 1000)) + f".{created_ms % 1000:03d}000Z" + now_ms = int(time.time() * 1000) + now_iso = time.strftime("%Y-%m-%dT%H:%M:%S", time.gmtime(now_ms / 1000)) + f".{now_ms % 1000:03d}Z" + store_root = os.path.realpath(target.root) + source_table = json.dumps(manifest["source_table"], separators=(",", ":")) + if target.backend == "postgres": + catalog = f"{target.tag}_catalog" + psql( + "INSERT INTO backup_index (account_id, backup_id, table_name, backup_arn, backup_name, status, " + "store_kind, store_root, store_id, size_bytes, item_count, created_at, completed_at, " + "source_table_details_json, owner_generation, heartbeat_at) VALUES " + f"('{ACCOUNT}', '{backup_id}', '{table}', '{arn}', 'matrix', 'AVAILABLE', 'filesystem', " + f"'{store_root}', '{target.store_id()}', {size_bytes}, {item_count}, " + f"to_timestamp({created_ms} / 1000.0), now(), '{source_table.replace(chr(39), chr(39) * 2)}'::jsonb, 0, now())", + database=catalog, + ) + elif target.backend == "sqlite": + conn = sqlite_module().connect(os.path.join(target.work, "db.sqlite"), timeout=10) + try: + conn.execute( + "INSERT INTO backup_index (account_id, backup_id, table_name, backup_arn, backup_name, status, " + "store_kind, store_root, store_id, size_bytes, item_count, created_at, completed_at, " + "source_table_details_json, owner_generation, heartbeat_at) VALUES (?, ?, ?, ?, 'matrix', 'AVAILABLE', " + "'filesystem', ?, ?, ?, ?, ?, ?, ?, 0, ?)", + (ACCOUNT, backup_id, table, arn, store_root, target.store_id(), size_bytes, item_count, + created_iso, now_iso, source_table, now_iso), + ) + conn.commit() + finally: + conn.close() + elif target.backend == "mongodb": + import datetime + + from bson import Int64 + from pymongo import MongoClient + + created = datetime.datetime.fromtimestamp(created_ms / 1000, tz=datetime.timezone.utc) + now = datetime.datetime.fromtimestamp(now_ms / 1000, tz=datetime.timezone.utc) + client = MongoClient(MONGODB_URI) + try: + client["extenddb_catalog"]["backup_index"].insert_one({ + "account_id": ACCOUNT, + "backup_id": backup_id, + "table_name": table, + "backup_arn": arn, + "backup_name": "matrix", + "status": "AVAILABLE", + "store_kind": "filesystem", + "store_root": store_root, + "store_id": target.store_id(), + "size_bytes": Int64(size_bytes), + "item_count": Int64(item_count), + "created_at": created, + "completed_at": now, + "source_table_details_json": manifest["source_table"], + "owner_generation": Int64(0), + "heartbeat_at": now, + "lock_seq": Int64(0), + }) + finally: + client.close() + elif target.backend == "cassandra": + # The catalog-index row (`store_kind` set) in `backups_by_arn`, plus + # its `backup_index_by_status` entry; DescribeTable and the restore + # re-read the authoritative row, so the two list rows are optional. + import datetime + + from cassandra.auth import PlainTextAuthProvider + from cassandra.cluster import Cluster + + st = manifest["source_table"] + created_dt = datetime.datetime.utcfromtimestamp(created_ms / 1000) + tct = st.get("table_creation_date_time") + table_created_dt = ( + datetime.datetime.utcfromtimestamp(tct) if tct else created_dt + ) + now_dt = datetime.datetime.utcnow() + cluster = Cluster( + ["127.0.0.1"], port=9042, + auth_provider=PlainTextAuthProvider("cassandra", "cassandra"), + ) + session = cluster.connect() + try: + insert = session.prepare( + "INSERT INTO extenddb_catalog.backups_by_arn (account_id, backup_arn, backup_id, " + "backup_name, table_id, table_name, table_arn, backup_status, backup_type, " + "backup_size_bytes, item_count, key_schema, attribute_definitions, billing_mode, " + "table_created_at, created_at, store_kind, store_root, store_id, " + "source_table_details_json, owner_generation, heartbeat_at) " + "VALUES (?, ?, ?, 'matrix', ?, ?, ?, 'AVAILABLE', 'USER', ?, ?, ?, ?, ?, ?, ?, " + "'filesystem', ?, ?, ?, 0, ?)" + ) + session.execute( + insert, + ( + ACCOUNT, arn, backup_id, st["table_id"], table, st["table_arn"], + size_bytes, item_count, + json.dumps(st["key_schema"], separators=(",", ":")), + json.dumps(st["attribute_definitions"], separators=(",", ":")), + str(st.get("billing_mode", "PAY_PER_REQUEST")), + table_created_dt, created_dt, store_root, target.store_id(), + source_table, now_dt, + ), + ) + status_entry = session.prepare( + "INSERT INTO extenddb_catalog.backup_index_by_status (store_root, backup_status, " + "backup_arn, account_id, store_kind, store_id) " + "VALUES (?, 'AVAILABLE', ?, ?, 'filesystem', ?)" + ) + session.execute(status_entry, (store_root, arn, ACCOUNT, target.store_id())) + finally: + cluster.shutdown() + else: + raise NotImplementedError(f"seed_row for {target.backend}") + say(f"target: backup_index row seeded for {arn} (store {target.store_id()})") + + +# --- comparison --------------------------------------------------------------- + + +def normalize(it: dict) -> str: + def norm(v): + if isinstance(v, dict): + if len(v) == 1: + (k, inner), = v.items() + if k in ("SS", "NS"): + return {k: sorted(inner)} + if k == "BS": + return {k: sorted(base64.b64encode(x).decode() for x in inner)} + if k == "B": + return {k: base64.b64encode(inner).decode()} + if k == "N": + return {k: str(inner)} + return {k: norm(x) for k, x in v.items()} + if isinstance(v, list): + return [norm(x) for x in v] + return v + return json.dumps(norm(it), sort_keys=True) + + +def scan_all(d: Deployment, name: str, index: str | None = None) -> list[str]: + c = d.client() + out, kwargs = [], {"TableName": name} + if index: + kwargs["IndexName"] = index + while True: + page = c.scan(**kwargs) + out.extend(normalize(i) for i in page["Items"]) + if "LastEvaluatedKey" not in page: + return sorted(out) + kwargs["ExclusiveStartKey"] = page["LastEvaluatedKey"] + + +def compare_descriptions(source: dict, target: dict) -> None: + def keys(x): + return [(k["AttributeName"], k["KeyType"]) for k in x["KeySchema"]] + check(keys(source) == keys(target), "key schema differs") + check(sorted((a["AttributeName"], a["AttributeType"]) for a in source["AttributeDefinitions"]) + == sorted((a["AttributeName"], a["AttributeType"]) for a in target["AttributeDefinitions"]), "attribute definitions differ") + for kind in ("GlobalSecondaryIndexes", "LocalSecondaryIndexes"): + s = {g["IndexName"]: (keys(g), g["Projection"]["ProjectionType"]) for g in source.get(kind) or []} + t = {g["IndexName"]: (keys(g), g["Projection"]["ProjectionType"]) for g in target.get(kind) or []} + check(s == t, f"{kind} differ: {s} vs {t}") + def units(x): + pt = x.get("ProvisionedThroughput") or {} + return (pt.get("ReadCapacityUnits", 0), pt.get("WriteCapacityUnits", 0)) + for g in target.get("GlobalSecondaryIndexes") or []: + sg = next(x for x in source["GlobalSecondaryIndexes"] if x["IndexName"] == g["IndexName"]) + check(units(g) == units(sg), f"GSI {g['IndexName']} throughput differs: {units(g)} vs {units(sg)}") + check(units(source) == units(target), f"table throughput differs: {units(source)} vs {units(target)}") + check((source.get("BillingModeSummary") or {}).get("BillingMode", "PROVISIONED") + == (target.get("BillingModeSummary") or {}).get("BillingMode", "PROVISIONED"), "billing mode differs") + check((source.get("SSEDescription") or {}).get("Status") == (target.get("SSEDescription") or {}).get("Status"), "SSE differs") + check((target.get("TableClassSummary") or {}).get("TableClass", "STANDARD") == "STANDARD", "the target's table class is not STANDARD") + sv = {v["IndexName"]: (v["Dimensions"], v["DistanceFunction"]) for v in source.get("VectorIndexes") or []} + tv = {v["IndexName"]: (v["Dimensions"], v["DistanceFunction"]) for v in target.get("VectorIndexes") or []} + check(sv == tv, f"vector indexes differ: {sv} vs {tv}") + + +# --- the cell ------------------------------------------------------------------ + + +def run(args) -> int: + cell_tag = f"{args.source}-{args.target}" + ("-vector" if args.vector else "") + ("-dup" if args.duplicate_key else "") + work = args.work or os.path.join(tempfile.gettempdir(), "backup-restore-matrix", cell_tag) + work = os.path.abspath(work) + shutil.rmtree(work, ignore_errors=True) + os.makedirs(work) + bins = {b: os.path.abspath(getattr(args, f"bin_{b}") or DEFAULT_BINS[b]) for b in DEFAULT_BINS} + for role, backend in (("source", args.source), ("target", args.target)): + check(os.path.isfile(bins[backend]), f"no {backend} binary at {bins[backend]} for the {role}") + source = Deployment(args.source, bins[args.source], work, "source") + target = Deployment(args.target, bins[args.target], work, "target") + rc = 0 + try: + source.init() + source.start() + source.provision() + create_and_fill(source, args.items, args.vector) + source_desc = wait_active(source, TABLE) + arn, backup_desc = take_backup(source) + source_items = scan_all(source, TABLE) + source_prefix = prefix_of(source.root, arn) + manifest = json.load(open(os.path.join(source_prefix, "extenddb-manifest.json"))) + source.stop() + + target.init() + target.start() + target.provision() + check(os.path.isfile(os.path.join(target.root, "extenddb-store.json")), "the target wrote no identity file") + target_prefix = prefix_of(target.root, arn) + copy_prefix(source_prefix, target_prefix, target.root) + if args.duplicate_key: + manifest = write_duplicate_key_prefix(target_prefix, manifest, args.vector) + say("target: the prefix now holds an unordered backup with one key in two files") + seed_row(target, arn, manifest, manifest["backup_size_bytes"], manifest["source_table"]["item_count"]) + + c = target.client() + described = c.describe_backup(BackupArn=arn)["BackupDescription"] + check(described["BackupDetails"]["BackupStatus"] == "AVAILABLE", "the seeded row does not describe AVAILABLE") + started = time.monotonic() + + if args.vector and args.target == "mongodb": + # A manifest with vector indexes into a backend without vector + # support is refused before any table is created (section 1, + # 5.3 step 2). + try: + c.restore_table_from_backup(TargetTableName=TARGET, BackupArn=arn) + except ClientError as e: + err = e.response["Error"] + check(err.get("Code") == "ValidationException", + f"the refusal is a ValidationException: {err}") + check("vector" in err.get("Message", "").lower(), + f"the refusal names vector indexes: {err}") + probe = target.signed_post("DescribeTable", {"TableName": TARGET}) + check(probe.status_code == 400 and "ResourceNotFound" in probe.text, + f"a table was created despite the refusal: {probe.status_code} {probe.text}") + say(f"target: the vector manifest was refused before any table: {err['Message'][:160]}") + print("CELL PASSED (vector manifest refused)") + return 0 + raise CellFailed("a vector manifest into MongoDB was not refused") + + # A vector manifest is refused into a backend without vector support + # (Cassandra), synchronously and before any table is created. + if args.vector and args.target == "cassandra": + try: + c.restore_table_from_backup(TargetTableName=TARGET, BackupArn=arn) + except ClientError as e: + err = e.response["Error"] + check( + err.get("Code") == "ValidationException" and "vector index" in err.get("Message", ""), + f"the vector manifest was not refused with a vector ValidationException: {err}", + ) + probe = target.signed_post("DescribeTable", {"TableName": TARGET}) + check( + probe.status_code == 400 and "ResourceNotFound" in probe.text, + f"a table was created for a refused vector restore: {probe.status_code} {probe.text}", + ) + say("target: the vector manifest was refused before any table was created") + print("CELL PASSED (vector manifest refused)") + return 0 + raise CellFailed("the vector manifest into Cassandra was not refused") + + try: + resp = c.restore_table_from_backup(TargetTableName=TARGET, BackupArn=arn) + except ClientError as e: + raise CellFailed(f"RestoreTableFromBackup refused: {e.response['Error']}") from e + desc = resp["TableDescription"] + check(desc["TableStatus"] == "CREATING", f"the restore answered {desc['TableStatus']}") + summary = desc.get("RestoreSummary") or {} + check(summary.get("SourceBackupArn") == arn and summary.get("RestoreInProgress") is True, + f"the response carries no RestoreSummary: {summary}") + say(f"target: restore of {arn} into {TARGET} registered CREATING") + + if args.duplicate_key: + deadline = time.monotonic() + 600 + while time.monotonic() < deadline: + r = target.signed_post("DescribeTable", {"TableName": TARGET}) + if r.status_code == 400 and "ResourceNotFound" in r.text: + break + # Cassandra removes a failed restore target in two writes + # (CREATING to DELETING, then the drop), so DELETING is a + # transient state on the way out, not a reached ACTIVE. + check( + r.status_code != 200 + or r.json()["Table"]["TableStatus"] in ("CREATING", "DELETING"), + f"a backup with a duplicated key reached {r.json()['Table']['TableStatus']}", + ) + time.sleep(0.5) + else: + raise CellFailed("the target of the duplicated-key backup was not removed") + # The job logs its FAILED line just after it removes the target, + # so poll the log for a short window rather than reading it the + # instant the table disappears. + log_deadline = time.monotonic() + 15 + failed = [] + while time.monotonic() < log_deadline: + log = open(os.path.join(target.work, "serve.log")).read() + failed = [l for l in log.splitlines() if "FAILED" in l and TARGET in l] + if failed: + break + time.sleep(0.25) + check(bool(failed), "the server log does not record the FAILED restore") + check(any("holds a key twice" in l or "the manifest lists" in l for l in failed), + f"the failure names neither the duplicate key nor the count: {failed[-1][:300]}") + say(f"target: {failed[-1].split(': FAILED', 1)[-1][:200]}") + say(f"target: the duplicated-key restore failed and the target is gone after {time.monotonic() - started:.1f} s") + print("CELL PASSED (duplicate key refused)") + return 0 + + target_desc = wait_active(target, TARGET) + say(f"target: {TARGET} ACTIVE after {time.monotonic() - started:.1f} s") + check(target_desc.get("RestoreSummary") is None, "an ACTIVE table still reports a RestoreSummary") + target_items = scan_all(target, TARGET) + check(len(target_items) == args.items, f"the target holds {len(target_items)} items, the source wrote {args.items}") + check(target_items == source_items, "the item sets differ") + compare_descriptions(source_desc, target_desc) + for gsi in target_desc.get("GlobalSecondaryIndexes") or []: + name = gsi["IndexName"] + rows = c.query(TableName=TARGET, IndexName=name, + KeyConditionExpression="g = :g", ExpressionAttributeValues={":g": {"S": "g0"}})["Items"] + expected = sum(1 for i in range(args.items) if i % 2 == 0 and i % 4 == 0) + check(len(rows) == expected, f"Query on {name} returned {len(rows)} items, expected {expected}") + lsi_rows = c.query(TableName=TARGET, IndexName="by-l", + KeyConditionExpression="pk = :p", ExpressionAttributeValues={":p": {"S": "p000"}})["Items"] + expected_l = sum(1 for i in range(args.items) if i % 11 == 0 and i % 3 == 0) + check(len(lsi_rows) == expected_l, f"Query on by-l returned {len(lsi_rows)} items, expected {expected_l}") + if args.vector: + r = target.signed_post("SearchVectors", {"TableName": TARGET, "IndexName": "by-emb", + "SearchVector": [{"N": "0.1"}, {"N": "0.0"}, {"N": "0.3"}], + "TopK": 3}) + check(r.status_code == 200, f"SearchVectors on the target: {r.status_code} {r.text}") + results = r.json().get("SearchResults") or [] + check(len(results) == 3, f"SearchVectors returned {len(results)} results, expected 3") + # The index strips the vector from the projected item; the items it + # answers are the ones that carried one (every fifth). + keys = [int(x["Item"]["sk"]["N"]) for x in results] + check(all(k % 5 == 0 for k in keys), f"SearchVectors answered items that carried no vector: {keys}") + check(described["SourceTableDetails"]["ItemCount"] == args.items, "DescribeBackup's item count differs from the items written") + say("target: item set, description, GSI and LSI queries match the source") + print("CELL PASSED") + return 0 + except CellFailed as e: + print(f"CELL FAILED: {e}") + return 1 + finally: + for d in (target, source): + try: + d.teardown() + except Exception as e: # noqa: BLE001 + print(f"teardown of the {d.role}: {e}") + print(f"logs under {work}") + + +def main() -> None: + p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) + p.add_argument("--source", required=True, choices=["sqlite", "postgres", "mongodb"]) + p.add_argument("--target", required=True, choices=["sqlite", "postgres", "mongodb", "cassandra"]) + p.add_argument("--items", type=int, default=1200) + p.add_argument("--vector", action="store_true") + p.add_argument("--duplicate-key", action="store_true") + p.add_argument("--work") + for b in DEFAULT_BINS: + p.add_argument(f"--bin-{b}") + sys.exit(run(p.parse_args())) + + +if __name__ == "__main__": + main() diff --git a/devtools/run-mongodb-tests b/devtools/run-mongodb-tests index 4a3b03885..cde51351c 100755 --- a/devtools/run-mongodb-tests +++ b/devtools/run-mongodb-tests @@ -16,6 +16,13 @@ # --mongo-port PORT Host port to publish mongo on (default: 27021) # --output DIR Directory for logs and generated config # (default: /tmp/run-mongodb-tests-) +# --backup-store KIND filesystem (default): the server gets a filesystem +# [backup.store] under the output directory and the +# suites read EXTENDDB_TEST_BACKUP_STORE=filesystem; +# none: no [backup.store], the in-catalog backup path, +# EXTENDDB_TEST_BACKUP_STORE=none. The legacy LSI +# fallback cell of tests/test_mongodb_backup_restore.py +# runs only under none. # --keep Do not tear down the mongo container / server # on exit (useful for post-run inspection) # --mongo-only Start the replica set, print its URI, and exit without @@ -37,6 +44,7 @@ # devtools/run-mongodb-tests -- --pytest --filter test_put_item # devtools/run-mongodb-tests --keep -- --pytest --filter test_rfc0003 # devtools/run-mongodb-tests --rust-live -- --rust --rust-integration +# devtools/run-mongodb-tests --backup-store none -- --pytest --filter test_mongodb_backup_restore # # Prerequisites: # - Docker running @@ -60,6 +68,7 @@ OUTPUT_DIR="" KEEP=false MONGO_ONLY=false RUST_LIVE=false +BACKUP_STORE=filesystem RUN_TESTS_ARGS=() while [[ $# -gt 0 ]]; do @@ -67,11 +76,17 @@ while [[ $# -gt 0 ]]; do --port) BIND_PORT="$2"; shift 2 ;; --mongo-port) MONGO_PORT="$2"; shift 2 ;; --output) OUTPUT_DIR="$2"; shift 2 ;; + --backup-store) + case "$2" in + filesystem|none) BACKUP_STORE="$2" ;; + *) echo "error: --backup-store takes filesystem or none, not: $2" >&2; exit 1 ;; + esac + shift 2 ;; --keep) KEEP=true; shift ;; --mongo-only) MONGO_ONLY=true; KEEP=true; shift ;; --rust-live) RUST_LIVE=true; shift ;; --) shift; RUN_TESTS_ARGS=("$@"); break ;; - -h|--help) sed -n '5,45p' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;; + -h|--help) sed -n '5,53p' "$0" | sed 's/^# \{0,1\}//'; exit 0 ;; *) echo "error: unknown option: $1" >&2; exit 1 ;; esac done @@ -134,7 +149,7 @@ cleanup() { if $KEEP; then echo "" echo "=== --keep set; leaving mongo + extenddb running ===" - echo " server log (startup): $SERVER_LOG" + echo " server log: $SERVER_LOG" echo " mongo container: $CONTAINER_NAME (port $MONGO_PORT)" echo " tear down manually: $BINARY stop --config $CONFIG; docker rm -f $CONTAINER_NAME" return @@ -221,7 +236,33 @@ EXTENDDB_ADMIN_PASSWORD="$ADMIN_PASSWORD" "$BINARY" init \ # `init` regenerates the config with default port 18443 and no # `[import]` / `[export]` sections. Rewrite with the port + defaults the # integration suite needs. This overwrite falls away once `extenddb init` -# grows flags for these knobs. +# grows flags for these knobs. With `--backup-store filesystem` (the +# default) the `[backup.store]` is a filesystem root under the output +# directory: CreateBackup is the store-backed job and RestoreTableFromBackup +# the store-backed restore, which the MongoDB restore cells +# (tests/test_mongodb_backup_restore.py) reach through the root +# (EXTENDDB_TEST_BACKUP_ROOT) and the server's log (EXTENDDB_TEST_SERVER_LOG); +# the suites read EXTENDDB_TEST_BACKUP_STORE=filesystem to expect that path. +# With `--backup-store none` the section is absent, CreateBackup writes the +# in-catalog format and the suites read EXTENDDB_TEST_BACKUP_STORE=none. +BACKUP_ROOT="$OUTPUT_DIR/backups" +STORE_SECTION="" +if [[ "$BACKUP_STORE" == "filesystem" ]]; then + # A root left by an earlier run of a reused --output directory carries + # that run's `extenddb-store.json`, which names that run's catalog; the + # container is recreated every run, so the new catalog's id differs and + # the server would find the store "owned by another catalog". Start from + # an empty root. + rm -rf "$BACKUP_ROOT" + mkdir -m 0700 -p "$BACKUP_ROOT" + STORE_SECTION=" +[backup.store] +kind = \"filesystem\" + +[backup.store.filesystem] +path = \"$BACKUP_ROOT\" +" +fi cat > "$CONFIG" <"$SERVER_LOG" 2>&1 +# The server runs in the foreground in the background of this shell, with its +# PID file written so `extenddb stop` (the cleanup) can signal it, and its +# whole log in SERVER_LOG: the restore cells read the job's log lines there, +# where a daemonized server's would go to syslog. +"$BINARY" serve --foreground --write-pid-file --config "$CONFIG" >"$SERVER_LOG" 2>&1 & for i in $(seq 1 30); do if curl -sk "https://127.0.0.1:$BIND_PORT/health" >/dev/null 2>&1; then @@ -310,6 +355,18 @@ export EXTENDDB_TEST_MONGODB_CONTAINER="$CONTAINER_NAME" # physical index lifecycle after exercising the DynamoDB API. export EXTENDDB_TEST_MONGODB_CONTAINER="$CONTAINER_NAME" +# Where the server writes its backups, which run-tests requires for --pytest +# and the backup suites read to pick their branch (filesystem: the +# store-backed job and restore; none: the in-catalog format). The MongoDB +# restore cells rewrite a backup's prefix under the store's root and read the +# restore job's log lines from the server's log; without a store there is no +# root and those cells skip, while the legacy LSI fallback cell runs. +export EXTENDDB_TEST_BACKUP_STORE="$BACKUP_STORE" +if [[ "$BACKUP_STORE" == "filesystem" ]]; then + export EXTENDDB_TEST_BACKUP_ROOT="$BACKUP_ROOT" +fi +export EXTENDDB_TEST_SERVER_LOG="$SERVER_LOG" + # Some tests read EXTENDDB_CONFIG to invoke `extenddb settings` against # the running instance. Without this, they fall back to `./extenddb.toml` # in the repo root, which is a stale dev config that points at a diff --git a/docs/design/15-backup-restore-bench-pr9.md b/docs/design/15-backup-restore-bench-pr9.md new file mode 100644 index 000000000..24e190dd1 --- /dev/null +++ b/docs/design/15-backup-restore-bench-pr9.md @@ -0,0 +1,310 @@ +# PR 9 restore memory and duration record (2026-11-02; the index, 60,000-item, input-order and 3M-row runs added the same day) + +This is the record the design cites for the restore (section 7, revision item +30 of `15-backup-restore.md`) and the admin guide's `max_concurrent_jobs` row +quotes. The scripts (`bench-restore-rss.sh`, `kill-points.py`) and the raw +samples (`/proc//status` every 200 ms per run, the servers' logs) are +kept beside the PR outside the repository, on the measuring host; the figures +below are what they produced. The backup side's record is +`15-backup-restore-bench-pr8.md`. + +## Method + +The release binary (`cargo build --release -p extenddb --no-default-features +--features sqlite` and `--features postgres`, built from the PR's final tree) +running `serve --foreground` with a filesystem backup store on local disk. +Each run: a fresh deployment (`init`, the port patched, `[backup] +max_concurrent_jobs = 2`), the table loaded through the wire +(`kill-points.py create-table`: BatchWriteItem from 16 threads, composite +key, no GSI; or `put-worst` for the worst shape), `CreateBackup` and the wait +for `AVAILABLE`, then the server stopped and started again so the process's +high-water mark covers the restore alone; `RestoreTableFromBackup` into a new +table; `VmHWM` read right before the request (the idle figure) and once +`DescribeTable` reports `ACTIVE` (the peak), sampled every 200 ms in between; +"over idle" is the difference. The duration is from the request to `ACTIVE` +as the client saw it; the job's own line gives the same figure within 0.2 s. + +The worst shape is #394's worst-expansion item (`data_file.rs`'s line-cap +table): a list of 2,262 number sets whose members are `1.0E-130`, 409 KB as +the service sizes an item, 27.7 MB as a data-file line, about 34 MB as a +decoded `Item`; each `worst` table holds 1,000 small items and then `N` +such items. `big1200` is 1,200 items of a 350,000-byte string under one +hash key, written into one data file (`data_file_target_bytes` raised past +the table), the shape on which the SQLite batch was measured on the PR's first +build, before it had a byte bound. The 100k and 1M tables are 200-byte `S` pads under a +composite key and have no secondary index, and their data files are in key +order (the server's own backup of a table loaded through the wire is +`key_ordered`); every figure in the first table below is for that shape, a +table with secondary indexes restores in more memory on SQLite (the second +table), and an unordered backup of the same items restores in more memory and +time on SQLite (the input-order table). The 60,000-item runs are the same items +and key, no index: the shape the differences document's in-catalog restore +timings were measured on. + +Host: 16 cores (Xeon Platinum 8124M), 30 GiB, Linux 5.10 (amzn2int), glibc +2.26; PostgreSQL 16.15 in a container on loopback (55432); the SQLite the +binary bundles. What was not held constant: the host ran other work during +the runs (the four hooks builds during the first six runs, with the load +average between 8 and 13 on 16 cores; a MongoDB kill-point run and the +matrix cells beside the later ones, load 2 to 7), so the durations are upper +bounds and comparable only within one back-to-back pair; the peaks are the +process's own and do not depend on the host's load. Everything is on +loopback and on a filesystem store: nothing here says what a network database +or an S3 store's download does to the duration, and the spool's disk is not +in these figures. + +## The figures + +| backend | shape | data files | idle `VmHWM` | peak `VmHWM` | peak − idle | `ACTIVE` after | downloaded | +|---|---|---|---|---|---|---|---| +| SQLite | worst 12 | 4 | 22.1 MiB | 870.5 MiB | **848 MiB** | 17.7 s | 3,793,555 B | +| SQLite | worst 24 | 8 | 20.4 MiB | 869.2 MiB | **848 MiB** | 26.8 s | 7,567,821 B | +| SQLite | worst 48 | 16 | 22.1 MiB | 1,035.1 MiB | **1,013 MiB** | 57.0 s | 15,116,357 B | +| SQLite | worst 24, `MALLOC_ARENA_MAX=1` | 8 | 20.2 MiB | 412.9 MiB | **392 MiB** | 20.5 s | 7,567,821 B | +| PostgreSQL | worst 24 | 8 | 21.4 MiB | 799.8 MiB | **778 MiB** | 22.5 s | 7,567,821 B | +| PostgreSQL | worst 24, `MALLOC_ARENA_MAX=1` | 8 | 21.4 MiB | 448.8 MiB | **427 MiB** | 22.7 s | 7,567,821 B | +| SQLite | big1200 (1,200 × 350 KB, one file) | 1 | 20.8 MiB | 291.5 MiB | **270 MiB** | 3.1 s | 450,922 B | +| PostgreSQL | big1200 | 1 | 21.4 MiB | 177.8 MiB | **156 MiB** | 3.3 s | 450,922 B | +| SQLite | 100k × 200 B | 1 | 21.9 MiB | 60.2 MiB | **38 MiB** | 2.9 s | 366,631 B | +| SQLite | 1M × 200 B, key-ordered lines | 4 | 21.2 MiB | 110.4 MiB | **89 MiB** | 26.8 s | 3,017,864 B | +| PostgreSQL | 100k × 200 B | 1 | 21.4 MiB | 32.0 MiB | **10.7 MiB** | 1.1 s | 366,631 B | +| PostgreSQL | 1M × 200 B, key-ordered lines | 4 | 21.5 MiB | 37.8 MiB | **16.2 MiB** | 9.5 s | 3,017,864 B | + +The SQLite 1M run's high-water mark over time: 60 MiB at 2.7 s (the 100k +run's peak), rising to 87 MiB by 20.8 s as the load went on, then 110 MiB in +the last second, at the count and the flip. The growth during the load is the +write connection's page cache filling (`PRAGMA cache_size = -32000`, +`crates/storage-sqlite/src/sqlite_util.rs`: 31 MiB per connection, bounded by +the pool, as the backup record says of its 45 MiB); the final 23 MiB is the +`COUNT(*)` over the base table: the counting connection's page cache filling +with the table's b-tree pages, bounded by the same `cache_size` (32 MiB) and +measured at the same size at 3M rows (22 to 28 MiB, the input-order table +below), so it does not grow with the table past the cache. On PostgreSQL the 1M run's high-water mark reached 90% of its final +value within 3 s and the count added nothing visible. 1M − 100k: +51 MiB on +SQLite (+27 of them during the load), +5.5 MiB on PostgreSQL. Neither +materializes the table (1M items of 200 bytes are over 200 MB of items). + +## The terms the code bounds + +Per running restore, times `max_concurrent_jobs`: + +- the parsed manifest, for the whole job (one entry per data file); +- the decoder's line, at most `MAX_LINE_BYTES` (32 MiB, `data_file.rs`), and + the deserializer's intermediate value for it (the backup record's "one + row's parse", `r + 2d`: 109 MiB at the line cap); +- the decoded items of up to `LOAD_CHUNK_BYTES` (32 MiB, + `crates/engine/src/backup/restore/load.rs`) of data-file line text per + chunk, the chunk closing at the first item that takes it to the bound (so at + most the bound plus one line, 64 MiB of text at the cap) or at + `LOAD_CHUNK_ITEMS` (1,000) items, at four chunk positions: two in the + channel to `bulk_load`, the one being filled and the one being drained. The + bound counts the bytes the decoder read for the chunk's items, the unit the + backup side budgets in, not the service's `item_size_bytes` (which is 68 + times smaller for the worst shape and bounded nothing). `bulk_load` is one + data file at a time, so the chunks in flight never hold more than one file's + items: at most the source's `data_file_target_bytes` plus one line of text + (64 + 32 MiB at the defaults), a knob the restoring host does not set; +- the backend's write batch: on SQLite and Cassandra at most + `RESTORE_BATCH_BYTES` (32 MiB, `crates/storage/src/restore_engine.rs`) of + serialized item text plus one item, or `RESTORE_BATCH_ITEMS` (10,000) + items when they are smaller, held beside the decoded items of that text + (SQLite collects `(Item, item_json)`; Cassandra serializes once to measure + and again to write); on PostgreSQL at most `COPY_BATCH_BYTES` (4 MiB, + `crates/storage/src/backup_definition.rs`) of item text plus one item, or + `COPY_BATCH_ITEMS` (500) rows, as `Vec<(Item, String)>` (the item and its + JSON text, two copies), one transaction per file; on MongoDB 1,000 + documents per `insert_many`, bounded in bytes only by the data file + (`bulk_load` is per file), since the driver splits a call by the server's + message bound on its own; +- on SQLite, the `COUNT(*)` after the last file: the counting connection's + page cache (`PRAGMA cache_size = -32000`, 32 MiB), which the count fills + with the base table's b-tree pages and which stops at that setting whatever + the table holds (22 to 28 MiB measured at 1M and at 3M rows, the input-order + table below). + +None of the terms grows with the table: the chunk and batch terms grow with +the item, which `max_item_size_bytes` bounds, the line cap bounds the line, +and `cache_size` bounds the count's pages. These are the restore's own terms; a table with +secondary or vector indexes adds, on SQLite, the backend's own per-connection +page cache for the index b-trees (below), which is not a buffer of the +restore and which the batch bound does not cover. The decoded `Item` of the worst shape is about 1.2 times +its line, so at the line cap the chunk positions hold up to 4 × 64 MiB × 1.2 ≈ +307 MiB of decoded items when a file holds that much, the SQLite or Cassandra +batch up to 32 MiB of text plus about 77 MiB of decoded items, and the +PostgreSQL batch 4 MiB plus one 32 MiB item twice. The measured worst-shape +peaks are above those terms under glibc's default allocator for the reason +the backup record gives: the items' strings are allocated on one thread (the +decoder's blocking thread) and freed on another (the backend's write), and +the per-thread arenas keep that memory, so the high-water mark grows with the +number of large items a restore reads (848 MiB at 12 and 24 items, 1,013 MiB +at 48 on SQLite, with no plateau measured) and varies between runs. Under +`MALLOC_ARENA_MAX=1` the peak is the terms' and did not grow from 12 to 24 +items on the backup side; here it is 392 MiB (SQLite) and 427 MiB +(PostgreSQL) at 24 items. + +Before the bounds (the figures measured the same way on the PR's first +build, kept beside this record): SQLite worst 12, +24 and 48 at 616, 891 and 925 MiB, worst 24 under one arena 370 MiB; +PostgreSQL worst 24 at 861 MiB, under one arena 428 MiB; the 1,200 × 350 KB +table on SQLite 435 MiB over idle (`VmRSS`, sampled every 100 ms in the +process that had taken the backup) against 127 MiB on PostgreSQL, the whole +file's items in one batch. With the batch bounded at 32 MiB of text the same +table holds 270 MiB on SQLite (`VmHWM` from a fresh start, so not the same +figure's method; the one-file table of 420 MB of items now loads in batches +of about 95 items) and 156 MiB on PostgreSQL. The worst-shape SQLite figures +with the default arenas moved with the allocator between the two series (616 +to 848 at 12 items, 925 to 1,013 at 48), as the backup record's series did; +the one-arena figures, which measure the terms, are 370 to 392 MiB on SQLite +and 428 to 427 MiB on PostgreSQL. + +## Tables with secondary indexes + +The same 200-byte items under the same composite key with two GSIs +(`by-g0`, `by-g1`, hash key `g` with 97 values, range key `sk`, `ALL` +projection), measured the same way (a fresh start, `VmHWM`), after every GSI +reported `ACTIVE` and with each GSI's `Scan` count checked against the item +count (equal in every run). "head" is the build of the tree before the +spool-sweep order, the `finish_restore` re-read and the `remove-target` +command were added, "final" the build with them; nothing in the restore's +load changed between the two, and the runs differ as the allocator's runs do. + +| backend | shape | build | idle `VmHWM` | peak `VmHWM` | peak − idle | `ACTIVE` after | load average at start | +|---|---|---|---|---|---|---|---| +| SQLite | 100k × 200 B, two GSIs | head | 21.4 MiB | 164.8 MiB | **143 MiB** | 10.2 s | 2.9 | +| SQLite | 100k × 200 B, two GSIs | final | 21.3 MiB | 246.4 MiB | **225 MiB** | 12.3 s | 12.1 (release builds running) | +| SQLite | 100k × 200 B, two GSIs | final, repeated | 21.1 MiB | 164.1 MiB | **143 MiB** | 13.9 s | 1.8 | +| SQLite | 300k × 200 B, two GSIs | head | 22.0 MiB | 187.1 MiB | **165 MiB** | 51.7 s | 8.1 | +| SQLite | 300k × 200 B, two GSIs | final | 21.7 MiB | 193.5 MiB | **172 MiB** | 51.9 s | 6.1 | +| PostgreSQL | 100k × 200 B, two GSIs | head | 21.4 MiB | 50.8 MiB | **29.3 MiB** | 9.9 s | 8.4 | +| PostgreSQL | 100k × 200 B, two GSIs | final | 23.3 MiB | 46.2 MiB | **22.9 MiB** | 6.2 s | 4.6 | +| PostgreSQL | 300k × 200 B, two GSIs | head | 21.3 MiB | 52.1 MiB | **30.8 MiB** | 19.7 s | 4.8 | +| PostgreSQL | 300k × 200 B, two GSIs | final | 24.0 MiB | 57.6 MiB | **33.5 MiB** | 21.4 s | 9.7 | + +With two GSIs the SQLite restore holds three to six times the no-index +figure at 100k (143 to 225 MiB in three runs against 38 and 43 MiB) and +grows from 100k to 300k (143 to 165 and 172 MiB); PostgreSQL holds 23 to +34 MiB against 10.7 MiB and grows little. In the SQLite runs the load itself +climbs to about 105 to 125 MiB of `VmRSS` before the last batch and the final +`COUNT(*)` and flip add the rest (50 MiB in the 143 MiB runs, 120 MiB in the +225 MiB run). The term is SQLite's, not the restore's: the restore's own +buffer is the one batch of `(Item, item_json)` bounded by `RESTORE_BATCH_BYTES` +whatever the indexes are, and the extra resident memory is the write +connection's page cache (`PRAGMA cache_size = -32000`, 32 MiB per connection, +`crates/storage-sqlite/src/sqlite_util.rs`) holding the index tables' b-tree +pages, which a load of scattered index keys dirties across the whole tree, +plus glibc's per-arena free lists (the figure falls by about a third under +`MALLOC_ARENA_MAX=1`: 72 MiB against 119 MiB at 100k in another run on the +same shape). It is bounded by `cache_size` per connection and grows +sub-linearly with the table, not with the restore's buffers; a smaller batch +for an indexed table was built and measured (91.7 MiB against 92.7 MiB over +idle at 100k in that pair of runs: the cache persists across batches, so +smaller transactions do not lower the peak) and not kept. The published +no-index figures stand for the no-index shape; size a SQLite host that +restores indexed tables for the no-index figure plus about 32 MiB of page +cache per running restore, and read the run-to-run spread above as the +allocator's. + +## Input order and the count term + +The 1M figures above are for the server's own backup, whose data files are +in key order. The same items (200-byte `S` pads, composite key, no index) +as an unordered backup, the prefix built by hand with the hash keys cycling +(`pk = i % 1000`, the shape of every MongoDB-sourced backup and every +multi-part-key table), and once more in key order (`pk = i // 1000`, 1,000 +items per hash key), at 1M and at 3M rows, on two builds of this PR: +"committed", the pushed head's, and "final", the build after the last +corrections (the PostgreSQL vector-relation drop on a removed target, the +Cassandra progress re-read, the job's first beat before the reply; nothing in +the SQLite load changed between the two). Sampled every 50 ms. +"count + flip" is the resident set's growth in the restore's last stretch +(the peak less the `VmRSS` one second before `ACTIVE`), which is the final +`COUNT(*)` and the flip; the host's load average at the start is beside each +run (the hooks builds and other suites ran beside the SQLite runs). + +| backend | shape | build | peak − idle | count + flip | `ACTIVE` after | load | +|---|---|---|---|---|---|---| +| SQLite | 1M, key-ordered lines | final | **98.1 MiB** | 21.9 MiB | 56.3 s | 16.2 | +| SQLite | 1M, unordered lines | final | **163.5 MiB** | 13.6 MiB | 90.3 s | 18.0 | +| SQLite | 1M, unordered lines | committed | **144.4 MiB** | 13.4 MiB | 66.2 s | 3.1 | +| SQLite | 3M, key-ordered lines | final | **109.3 MiB** | 27.6 MiB | 102.7 s | 19.1 | +| SQLite | 3M, key-ordered lines | committed | **116.4 MiB** | 22.9 MiB | 84.9 s | 4.5 | +| SQLite | 3M, unordered lines | final | **173.5 MiB** | 12.8 MiB | 264.9 s | 10.9 | +| SQLite | 3M, unordered lines | committed | **170.8 MiB** | 14.4 MiB | 274.7 s | 6.0 | +| PostgreSQL | 1M, unordered lines | final | **16.4 MiB** | 0.1 MiB | 17.5 s | 5.6 | + +Two things the table shows. The input order is a term of SQLite's, not of +the restore's: an unordered backup of 1M items holds 1.5 to 1.8 times the +key-ordered figure (144 to 164 MiB against 87 to 98) and takes two to three +times as long, because scattered inserts dirty pages across the whole base +b-tree where ordered inserts append to its right edge, and the write +connection's page cache (`cache_size`, 32 MiB) plus the allocator's arenas +hold the difference; the restore's own buffers (the chunk and the batch) are +the same in both runs, and PostgreSQL does not depend on the order (16.2 MiB +ordered, 16.4 unordered). The ordered 1M run on this binary reproduced the +first table's figure (98 against 89 MiB over idle; 86.9 MiB in another run +of the same shape on the committed build). And the count's term does not grow +with the table: 22 to 28 MiB at 1M and at 3M rows in key order, 13 to 14 MiB +unordered at both sizes, every figure under the counting connection's +`cache_size` (32 MiB), which is what bounds it; from 1M to 3M rows the whole +peak moved by +11 MiB key-ordered and +7 to +10 MiB unordered while the +table tripled. The durations are upper bounds: the SQLite runs on the final +build ran under a load average of 11 to 19 (the ordered 1M run took 56 s +against 26.8 s on the quieter host of the first table). + +## The 60,000-item shape, store-backed + +The differences document's "How `RestoreTableFromBackup` runs" row quotes +response and `ACTIVE` times measured on an earlier build's in-catalog restore +of a 60,000-item table with no secondary index. The same shape (60,000 items +of 200-byte `S` pads under a composite key, no index) restored from a +filesystem store on the final binaries, the host's load average 5.4 to 7.6 +with other runs beside it: + +| backend | response | `ACTIVE` after | peak − idle | data files / bytes | +|---|---|---|---|---| +| SQLite | 0.016 s | 1.85 s | 38.3 MiB | 1 / 360,532 | +| PostgreSQL | 0.051 s | 0.67 s | 9.3 MiB | 1 / 360,532 | +| MongoDB | 0.055 s | 3.74 s (the job's DONE at 1.8 s; the backfill worker's flip after) | 30.1 MiB | 1 / 360,532 | + +Cassandra was not measured on this shape (its own CreateBackup is in-catalog, +so a store-backed Cassandra restore needs a backup taken elsewhere and +adopted, which the kill-point and matrix runs do at 12,000 items). The +in-catalog figures the differences document quotes (response 0.032, 0.012, +0.053, 0.151 s; `ACTIVE` after 1.5, 2.1, 3.6, 11.2 s on PostgreSQL, SQLite, +MongoDB, Cassandra) were measured on another build and host and are not +comparable run for run; the store-backed response includes the sidecar and +manifest reads, and its `ACTIVE` time the download and the per-file +verification. + +## The sizing sentence + +Size a host for about 450 MiB per running restore of a table of large items +under `MALLOC_ARENA_MAX=1` on SQLite or PostgreSQL (392 and 427 MiB measured +at 24 items of 27.7 MB), times `max_concurrent_jobs`; under the default +arenas the peak grows with the number of large items a restore reads and no +bound was measured (1,013 MiB over idle at 48 such items on SQLite is the +largest seen; 778 MiB at 24 on PostgreSQL). A restore of 200-byte items holds +under 100 MiB on SQLite and under 20 MiB on PostgreSQL at 1M items when the +table has no secondary index and the backup's lines are in key order (the +server's own backups of hash-only and single-range-key tables); an unordered +backup of the same items (every MongoDB-sourced backup and every +multi-part-key table) held 144 to 164 MiB over idle on SQLite at 1M and 171 to +174 MiB at 3M (the input-order section), PostgreSQL 16 MiB either way; with two GSIs SQLite held 143 to 225 MiB at +100k and 165 to 172 MiB at 300k (the section above), PostgreSQL 23 to 34 MiB. A +restore is not "much less" than a backup of the same table: the backup +record's worst-shape figures are 466 to 1,281 MiB under the default arenas +and 452 to 481 MiB under one arena. + +## Durations (the RTO baseline the testing summary names) + +On this host, loopback, filesystem store: 1M items of 200 bytes in key order +restore in 9.5 s on PostgreSQL and 26.8 s on SQLite (100k: 1.1 s and 2.9 s; +60k: 0.67 s and 1.85 s, and 3.74 s on MongoDB); the same 1M items as an +unordered backup in 17.5 s on PostgreSQL and 66 to 90 s on SQLite, and 3M +items on SQLite in 85 to 103 s key-ordered and 265 to 275 s unordered (the +input-order section, on a host whose load average ran between 4 and 19); with two GSIs 100k in 6.2 to 9.9 s on +PostgreSQL and 10.2 to 13.9 s on SQLite and 300k in 19.7 to 21.4 s and 51.7 +to 51.9 s, on a host whose load average ran between 1.8 and 12; 24 items of +27.7 MB in 22.5 s on PostgreSQL and 26.8 s on SQLite; 1,200 items of 350 KB in +3.3 s and 3.1 s. The backup side's 1M figures are in the PR 8 record (36 s on +PostgreSQL under a loaded host, 12.5 s on SQLite). diff --git a/docs/design/15-backup-restore.md b/docs/design/15-backup-restore.md index 53be72f9e..cdac26332 100644 --- a/docs/design/15-backup-restore.md +++ b/docs/design/15-backup-restore.md @@ -61,9 +61,9 @@ A data file line (one `{"Item": ...}` object and its newline) is at most 32 MiB `backup_creation_date_time` is the instant `open_table_snapshot` returned, read just after the snapshot pinned, and it is the instant encoded in the backup id: CreateBackup opens the snapshot first (after any wait behind an in-flight UpdateTable), then mints the id from that millisecond (5.1 step 2), so the ARN, the manifest and the catalog's `created_at` carry one time. -A restore rejects a backup whose data files contain the same primary key twice, across files as well as within one. For key-ordered backups the manifest's `key_range` check rejects overlap before any file is downloaded and the per-file key-range check proves the rest, on every target. For unordered backups a plain `INSERT` on PostgreSQL and SQLite fails on a duplicate and so does MongoDB's `_id` unique index (E11000), but a Cassandra write is an upsert and detects nothing, so the general rule is a count: after the last file the restore counts the target's items and requires the count to equal the sum of the manifest's `item_count`; a shortfall is a duplicate and fails the restore. +A restore rejects a backup whose data files contain the same primary key twice, across files as well as within one. For key-ordered backups the manifest's `key_range` check rejects overlap before any file is downloaded and the per-file key-range check proves the rest, on every target. For unordered backups a plain `INSERT` on PostgreSQL and SQLite fails on a duplicate and so does MongoDB's `_id` unique index (E11000, suppressed only on the one unrecorded file a resume re-applies, 5.3 step 5), but a Cassandra write is an upsert and detects nothing, so the general rule is a count: after the last file of every backup, key-ordered or not, the restore counts the target's items (`count_restore_target`, section 3) and requires the count to equal the sum of the manifest's `item_count`; a shortfall is a duplicate and fails the restore. -- Numbers are DynamoDB JSON strings, so the 38-digit precision survives the file. A restore into MongoDB of an `N` sort-key value beyond Decimal128's 34 significant digits fails with ValidationException and the restore reports FAILED (see 5.3); it never rounds. Other `N` values are stored as strings there and keep their 38 digits. +- Numbers are DynamoDB JSON strings, so the 38-digit precision survives the file. A restore into MongoDB of an `N` sort-key value beyond Decimal128's 34 significant digits fails in `bulk_load` with the write path's ValidationException text, which becomes the restore's FAILED reason (5.3 step 7: no wire exception, the client sees the target disappear); it never rounds. Other `N` values are stored as strings there and keep their 38 digits, so a 38-digit non-key number restores intact. ### 2. Backup store abstraction @@ -82,7 +82,7 @@ Three store facts that rules of sections 4 and 5 rest on, and the one method a l - Cassandra: `open_table_snapshot` returns `Unsupported`. Cassandra has no `SnapshotEngine`, so the engine routes CreateBackup there to the legacy create (`LegacyBackupReader::create`, the #384 in-catalog path) whether or not `[backup.store]` is configured, the path every backend takes while no store is configured (section 7); the off-host path is never taken there. That stays until PR 14, by which time either a token-range `SnapshotEngine` with a documented non-point-in-time `SnapshotInfo.kind` (`cassandra_token_scan`) has been accepted as a design change to this section, or Cassandra backups are documented as unavailable. RFC-0003 (backend acceptance criteria) has no backup criterion today; the criterion this trait implies ("a backend that cannot fix an instant across a table returns `Unsupported` rather than a best-effort stream") is to be proposed there in a separate docs change, not in PR 5. The interfaces later PRs add to `extenddb-storage`, each with its contract; the signatures and the row keys are in the section of the PR that adds them (Phasing): -- `bulk_load(target_table_id, claim, file_index, start_at, items) -> Result`, a new `StorageEngine` method each backend owns (PR 9a), writes the items of one verified data file from item `start_at` on, and their index rows, into a CREATING restore target, records progress as 5.3 step 5 specifies for the backend under the job's `claim` (so a progress write conditioned on the generation has it) and returns how many items it applied. There is one spelling of that signature; 5.3 step 5 calls it. +- `RestoreEngine` (PR 9a), a trait in `StorageEngine`'s bound (`StorageEngine` has no methods of its own) that each backend implements: `bulk_load(target, claim, BulkLoadFile { file_index, start_at, reapplying }, items) -> Result` writes the items of one verified data file from item `start_at` on, and their index rows, into a CREATING restore target, records progress in the write that applied the items as 5.3 step 5 specifies for the backend (the job's `claim` carries the generation the record's writes are conditioned on) and returns how many items it applied; `finish_restore(target, claim, item_count, size_bytes) -> Finished { Active, Pending, NotMatched }` is the ACTIVE flip of 5.3 step 6; `count_restore_target(target)` is the count of section 1; `read_restore_progress` and `sweep_restore_progress(running)` read and sweep PostgreSQL's `restore_progress` rows (defaults elsewhere). There is one spelling of each signature, in PR 9a's section; 5.3 step 5 calls `bulk_load`. - The engine is backend-agnostic (main's `crates/engine` reaches every backend through the `extenddb_storage` traits), and nothing in sections 4 and 5 that touches a catalog row (the claim, the heartbeat, the conditional status writes, the store-scoped listings, the restore job record) can be written in it. `BackupCatalog`, a trait in `extenddb-storage` that PR 6 adds and each backend implements, is how the engine reaches the catalog index of section 4; PRs 8, 9a and 10 add methods to the same trait as their sections need them. Section 4's "Authority" line names this trait beside the engine module that binds it. - `LegacyBackupReader` is a trait in `extenddb-storage` with `describe`, `list`, `restore` and `delete`, plus `create`, which every backend keeps until PR 14: the engine calls it for CreateBackup when no `[backup.store]` is configured (section 7), logging a deprecation warning that names `[backup.store]` on each call, and on Cassandra always, since it has no `SnapshotEngine` (the decision before PR 14 settles whether that stays), and a legacy create takes a `max_concurrent_jobs` permit like any job, returning `LimitExceededException` when none is free (5.1 step 1), since PR 6 removes the slot semaphores #384's creates wait on; each backend implements it with a frozen copy of its #384 code for reading and decoding the legacy shape (not calls through trait methods later PRs remove), and the engine holds `Option>`. PR 14 removes the trait. @@ -92,13 +92,13 @@ The invariants and state machines every later PR honours, by lettered part; the (a) The backup row. The engine keeps a catalog row per backup, in `backup_index` (PR 6's schema), as the fast index for Describe and List, and as the state machine record; `status` is CREATING, AVAILABLE, DELETING, DELETED or FAILED. Status mapping to the API: CREATING and AVAILABLE as themselves; DELETING is reported as DELETED by DescribeBackup and DeleteBackup (a client retrying DeleteBackup sees the terminal status) and is not listed; DELETED and FAILED rows are not described (BackupNotFoundException, which is what the live service returns for a deleted backup) and not listed. FAILED and DELETED rows are tombstones the reconciler owns. Transitions and who performs them: CreateBackup inserts the row CREATING under the job's claim (5.1 step 2); the job writes AVAILABLE after the manifest, or FAILED, each conditional on its generation and on `status = CREATING` (5.1 steps 4, 5); the reconciler resolves a CREATING row whose owner is dead to AVAILABLE if a complete, verified manifest exists in the configured store, else to FAILED, under the same condition (5.1 step 6); DeleteBackup moves an AVAILABLE row to DELETING under the row's lock and to DELETED once the prefix lists empty (5.2); the reconciler finishes a DELETING row, deletes the prefix of a FAILED row once its `heartbeat_at` is older than `stale_job_after_secs` and of a DELETED row that still lists objects, and removes a FAILED or DELETED row once its prefix is empty and its `completed_at` is older than `failed_row_retention_secs`. -(b) The restore job record. The engine keeps one restore job record per restore target, called `restore_jobs` in this document. It is #384's restore registration extended with `status` (RUNNING, FAILED, DONE), `kind` (legacy or manifest), the store it reads, the claim (`owner_generation`, `heartbeat_at`), `spool_host`, per-batch progress, `failure_reason` and `completed_at` (PR 6's schema). Transitions: the request writes the row RUNNING with the target table in one transaction (5.3 step 4); the job marks it DONE with the ACTIVE flip, or FAILED, each conditional on its generation and on RUNNING (5.3 steps 6, 7); the reconciler claims a RUNNING row whose owner is dead and resumes or fails it (5.3 step 7); on MongoDB the job's DONE write precedes the worker's ACTIVE flip and the reconciler leaves such a row alone (5.3 step 6). The record is the table's child (`table_restores` cascades with the table on PostgreSQL and SQLite; it is fields of the table on MongoDB and Cassandra), so it lives exactly as long as the table: a DONE record stays while the table exists, `RestoreSummary` is reported only while the table is CREATING (as main does and `restore_summary_is_reported_only_while_creating` pins), and a FAILED record goes with its table when step 7 deletes it, the failure reported by the job's log line and `restore_jobs_total{outcome="failed"}`. +(b) The restore job record. The engine keeps one restore job record per restore target, called `restore_jobs` in this document. It is #384's restore registration extended with `status` (RUNNING, FAILED, DONE), `kind` (legacy or manifest), the store it reads, the claim (`owner_generation`, `heartbeat_at`), `spool_host`, its progress (`file_index` and `items_applied`: per batch on SQLite and Cassandra, `file_index` per file on MongoDB; PostgreSQL records progress in `restore_progress` rows of the data database instead, 5.3 step 5), `failure_reason` and `completed_at` (PR 6's schema; `RestoreJobRow` carries every member). Transitions: the request writes the row RUNNING with the target table in one transaction (5.3 step 4); the job marks it DONE with the ACTIVE flip, or FAILED, each conditional on its generation and on RUNNING (5.3 steps 6, 7); the reconciler claims a RUNNING row whose owner is dead and resumes or fails it (5.3 step 7); on MongoDB the job's DONE write precedes the worker's ACTIVE flip and the reconciler leaves such a row alone (5.3 step 6). The record is the table's child (`table_restores` cascades with the table on PostgreSQL and SQLite; it is fields of the table on MongoDB and Cassandra), so it lives exactly as long as the table: a DONE record stays while the table exists, `RestoreSummary` is reported only while the table is CREATING (as main does and `restore_summary_is_reported_only_while_creating` pins), and a FAILED record goes with its table when step 7 deletes it, the failure reported by the job's log line and `RestoreJobs{failed}` (section 9). (c) Store identity and health. Each catalog has a random `catalog_id` (`BackupCatalog::catalog_id`); the store's root carries `extenddb-store.json`, holding a random store id and the `catalog_id` of the catalog that owns the root, and every backup row records the store id in `store_id`; server startup, every sync and every reconciler cycle read the file and check it against the catalog (PR 7's section: how the file is created). The store is unhealthy when the file names another catalog, when its store id differs from the `store_id` of the catalog's rows for the configured `store_kind` and `store_root`, when the catalog has rows for the root and the root has no file, or when those rows carry more than one `store_id` (`BackupCatalog::read_store_ids`, section 3). An unhealthy store is not a startup refusal (section 7): the reason is logged, `/health` reports the store component unhealthy, the operations that touch the store are refused, the reconciler acts on no row of the store, and the check runs again each reconciler cycle; a root whose file names another catalog stays unhealthy until an operator adopts it with `backup sync --adopt-store` (PR 10), and the previous owner then finds the store unhealthy and stops acting on it. Instances that share a catalog must share one store root (an S3 bucket and prefix or a shared mount): host-local roots at one path would give each instance different bytes under one `store_root`, so startup and each reconciler cycle also probe the root independently of the host, `head`ing the manifest of the most recent AVAILABLE row and marking the store unhealthy if it is absent. Every duty that reads or writes a backup row acts only on rows of the configured store (`store_kind`, `store_root` and `store_id`), as sync does, the legacy duty excepted: a CREATING or DELETING row of another store is reported and left alone (`backup sync --move-store`, PR 10, resolves it), a FAILED or DELETED row of another store is never removed and its prefix never deleted, and the emptiness check before a row is removed looks in the configured store, which is also what finds an object left by a put that overlapped the prefix's deletion (section 2); so pointing the configuration at a copy of a store never deletes the copy, and no row is resolved against a store that does not hold its bytes. (d) Identities. The ARN encodes partition, region, account, table, and backup id, and the key prefix is derived from account, table and id deterministically by `BackupArn::store_prefix`, under the configured store's prefix, and no row stores it, so restore, sync, purge and the reconciler derive it the same way and `describe_backup(arn)` works from the store alone after a sync. The engine, the writer and sync all parse and format ARNs through one implementation, `BackupArn` in `extenddb_core::backup_format` (#394); the engine keeps #384's region handling: it resolves a backup under the ARN as written and under the ARN re-spelled in its own region, and refuses a partition other than its own. `backup_creation_date_time` and `created_at` are the id's instant, so the ARN, the manifest, the API's reported time and the ListBackups cursor agree by construction. One row is addressed by `BackupKey` (`account_id`, `backup_id`, `backup_arn`): PostgreSQL, SQLite and MongoDB key `backup_index` by (`account_id`, `backup_id`), Cassandra keys `backups_by_arn` by (`account_id`, `backup_arn`); no method takes a bare backup id, and the engine treats a row whose `account_id`, `table_name` or `backup_arn` (after that re-spelling) differs from the request's as absent. One target is addressed by `RestoreTargetKey` (`account_id`, `table_name`, `table_id`), the table id guarding against a table re-created under the name. -(e) The claim. There is no single-instance lease on PostgreSQL or MongoDB, so each job holds a claim that both excludes a second owner and lets a successor detect a dead one. Every backend sets `owner_generation` (a fresh random value) at claim time, and every catalog write by the job is `WHERE owner_generation = $gen`, so a job that lost its claim cannot publish over a successor's work. No row ever exists unowned: on every backend the claim precedes the row's insert or is the insert itself (5.1 step 2); what the claim is per backend is PR 6's section. Who is dead: on SQLite every CREATING row at startup, and at runtime a row whose in-process slot no task holds; on PostgreSQL, MongoDB and Cassandra a row whose `heartbeat_at` is older than `stale_job_after_secs` (default 300), the rule MongoDB and Cassandra share; PostgreSQL adds the lock. On PostgreSQL the advisory lock is the claim's proof against a second owner and not a liveness test on its own: a free lock shows that the owner's catalog session ended, not that its process stopped (a session a managed server dropped leaves a process whose data writes carry no generation), and the owner itself aborts at its next heartbeat inside the stale window, so the reconciler takes over only a row whose heartbeat is stale **and** whose lock is free: a held lock with a stale heartbeat is a process that is stopped, not dead (SIGSTOP, a paused VM), and its row stays with it until its session ends or it resumes and finishes; a host that vanished releases the lock when the server's keepalives give up (about a minute with `CLAIM_KEEPALIVES`), so the takeover waits for the later of the two. Staleness is always measured on the database's clock, never on an application host's: the engine holds no clock, and every age test is a predicate of the backend's own (`BackupCatalog::claim_dead_backup(key, stale_after)`, `list_backup_rows_older_than(status, store, clock, age)` on the heartbeat or the completion column, PR 8's section). The reconciler takes over a dead row through `claim_dead_backup`, the same claim as a worker's with a fresh `owner_generation` that fences the dead job's generation out (PostgreSQL's lock and SQLite's slot refuse a second taker outright; on MongoDB and Cassandra the generation fence orders two takers, since the successor is a live owner to a later taker only once it heartbeats, and the reconciler writes its one status before any beat, so two instances' reconcilers can both take over one dead row within that gap and exactly one resolution lands, the other's write finding no row; Cassandra's conditional write also carries the heartbeat it read, so a beat landing between the read and the write refuses the takeover), leaves `heartbeat_at` untouched (a refreshed heartbeat would restart the stale window the prefix deletion waits for), writes AVAILABLE or FAILED at once, both conditioned on `status = CREATING`, and never continues the upload. A job whose conditional write matches no row, or whose conditional write's outcome stays unknown after the re-read, performs no further store or table operation, deletes of a prefix or of a restore target included (5.1 step 5, 5.3 step 7); each unknown outcome is followed by a re-read; the write is sent once more at most; after the second write a re-read that shows the row in the target status counts as matched and anything else as matching no row. The second write is for a backend whose claim connection survives the unknown outcome (MongoDB and Cassandra, whose claim writes go through the shared client); on PostgreSQL an unknown outcome is an error after which the claim session's one connection is dead, so the second write fails with the session, its re-read settles it as matching no row, and the row is the reconciler's once the session's end has freed the advisory lock. A FAILED `restore_jobs` row whose target table is still CREATING (the FAILED write landed and the target's removal did not, whether the job's own or the reconciler's) is the FAILED-target duty: the reconciler lists it once the row's `completed_at` (or, where none was stamped, its `heartbeat_at`) is older than `stale_job_after_secs`, so a writer still on its own failure path finishes it, and removes the target through `remove_restore_target` with no second FAILED write; PR 6's reconciler runs it for every kind of record, since on this rule the kind does not matter (PR 8 adds the `target_not_empty` exception). On Cassandra the removal is itself two writes, a lightweight transaction claiming the target CREATING to DELETING and then the drop, so the duty also lists a FAILED row whose target is DELETING with its registration still present, on the same staleness rule, and `remove_restore_target` resumes the removal from the drop; PostgreSQL and SQLite remove in one transaction and MongoDB deletes the document first, so neither leaves that state. +(e) The claim. There is no single-instance lease on PostgreSQL or MongoDB, so each job holds a claim that both excludes a second owner and lets a successor detect a dead one. Every backend sets `owner_generation` (a fresh random value) at claim time, and every catalog write by the job is `WHERE owner_generation = $gen`, so a job that lost its claim cannot publish over a successor's work. No row ever exists unowned: on every backend the claim precedes the row's insert or is the insert itself (5.1 step 2); what the claim is per backend is PR 6's section. Who is dead: on SQLite every CREATING backup row and every RUNNING store-backed restore record at startup (the restart ages every RUNNING restore's heartbeat to the epoch), and at runtime a row or record whose in-process slot no task holds (on SQLite the store-backed restore's claim holds the `restore:/
` slot from before the target exists to the task's end; the legacy restore holds no slot, and a legacy and a store-backed restore into one name exclude each other through the name's `tables` row under the write transaction (`BEGIN IMMEDIATE`: the CREATING-target check, then the unique name)); on PostgreSQL, MongoDB and Cassandra a row whose `heartbeat_at` is older than `stale_job_after_secs` (default 300), the rule MongoDB and Cassandra share; PostgreSQL adds the lock. On PostgreSQL the advisory lock is the claim's proof against a second owner and not a liveness test on its own: a free lock shows that the owner's catalog session ended, not that its process stopped (a session a managed server dropped leaves a process whose data writes carry no generation), and the owner itself aborts at its next heartbeat inside the stale window, so the reconciler takes over only a row whose heartbeat is stale **and** whose lock is free: a held lock with a stale heartbeat is a process that is stopped, not dead (SIGSTOP, a paused VM), and its row stays with it until its session ends or it resumes and finishes; a host that vanished releases the lock when the server's keepalives give up (about a minute with `CLAIM_KEEPALIVES`), so the takeover waits for the later of the two. Staleness is always measured on the database's clock, never on an application host's: the engine holds no clock, and every age test is a predicate of the backend's own (`BackupCatalog::claim_dead_backup(key, stale_after)`, `list_backup_rows_older_than(status, store, clock, age)` on the heartbeat or the completion column, PR 8's section). The reconciler takes over a dead row through `claim_dead_backup`, the same claim as a worker's with a fresh `owner_generation` that fences the dead job's generation out (PostgreSQL's lock and SQLite's slot refuse a second taker outright; on MongoDB and Cassandra the generation fence orders two takers, since the successor is a live owner to a later taker only once it heartbeats, and the reconciler writes its one status before any beat, so two instances' reconcilers can both take over one dead row within that gap and exactly one resolution lands, the other's write finding no row; Cassandra's conditional write also carries the heartbeat it read, so a beat landing between the read and the write refuses the takeover), leaves `heartbeat_at` untouched (a refreshed heartbeat would restart the stale window the prefix deletion waits for), writes AVAILABLE or FAILED at once, both conditioned on `status = CREATING`, and never continues the upload. A dead RUNNING store-backed restore record is taken over through `claim_restore_dead(key, stale_after, spool_host)` under the same rules (PostgreSQL's name lock and SQLite's slot refuse a taker outright; MongoDB and Cassandra condition on the stale heartbeat, Cassandra's lightweight transaction also on the heartbeat it read and on `kind = manifest`), which writes the claimant's `spool_host` with the fresh generation and never takes over a legacy record; the successor resumes the restore from the recorded progress (5.3 step 7). A restore record's takeover differs from a backup row's in one write: `claim_restore_dead` sets `heartbeat_at` to the database's clock in the same conditional write as the generation and `spool_host` (PostgreSQL `now()`, SQLite `strftime('%Y-%m-%dT%H:%M:%fZ', 'now')`, MongoDB `$$NOW`, Cassandra `toTimestamp(now())`), so a second instance's reconciler, whose cycle lands before the successor's first beat (one `heartbeat_interval_secs` after the takeover), finds the record live and is refused. The backup takeover leaves `heartbeat_at` because the FAILED-prefix duty's stale window counts from the owner's last beat; nothing waits on a restore record's old heartbeat after a takeover (the FAILED-target duty ages on `COALESCE(completed_at, heartbeat_at)` and every FAILED write of a resume stamps `completed_at`; the maintenance claim waits on RUNNING, not on the beat). Per backend, the second taker is refused by: PostgreSQL the successor's advisory lock on the target name, then the fresh heartbeat; SQLite the successor's in-process slot (one process serves the file), and the record the startup step had aged is no longer listed as dead once the takeover refreshed its heartbeat; MongoDB the `$expr` staleness term of the takeover's own filter against the refreshed `heartbeat_at`; Cassandra the takeover's serial read, which sees the refreshed `restore_heartbeat_at` and stops before its lightweight transaction. So the two-taker argument of the backup takeover is, for a restore, "a second taker finds the record live"; the belt under it is that every progress write is fenced by the generation (a first taker's next write would match nothing and its job end with no table access) and that a resume's first file is `reapplying`, which covers a partial file on MongoDB and which Cassandra's upserts collapse, so a later change to either mechanism (a progress write without the generation, `reapplying` on a later file) is seen to matter here. The takeover on Cassandra also re-applies the target's `active_restores[table_id] = name` entry on the backup's row when it is absent (one serial read, then `UPDATE ... IF backup_status = 'AVAILABLE'`): the registration writes the target's row and its DDL before the entry, so an owner that died between the two left a resumable record whose backup nothing held in use, and `DeleteBackup` would have succeeded under the resumed load. A backup no longer AVAILABLE keeps no entry; the resumed load fails on the missing prefix and the target is removed. The engine's first confirmed beat of a fresh restore is the return of the one heartbeat the task sends right after `claim_restore` returns and before the reply (5.3 step 4), and of a resumed restore the instant `claim_restore_dead` returned, taken by the reconciler beside the takeover and handed to the job; a takeover's `heartbeat_at` was stamped inside that write, so the instant is at or after the beat it stands for by the write's own duration, and the catalog reads between a takeover and the resumed job's start count against the two-period window rather than widen it. The registration's own `heartbeat_at` stamp is the catalog's clock at some point of the registration, not its end: on PostgreSQL `now()` is the transaction's start, so a registration stalled on the `backup_index` lock or the DDL commits a stale `heartbeat_at`; the job's first beat, sent before the reply, refreshes it. Two readers judge a RUNNING record's owner by the heartbeat alone and need that beat: the reconciler under a maintenance claim (a dead record is failed and its target removed, or failed `target_not_empty`) and `extenddb backup remove-target`; the lock fences only `claim_restore_dead`. A first beat that finds the claim lost, or fails, ends the task before the reply: the record is failed with reason "the claim was lost before the job started" (a write that matches nothing when the claim is another's) and the request is answered `InternalServerError`. What bounds a registration's duration per backend: PostgreSQL holds the `backup_index` row's `FOR UPDATE` lock and the target name's advisory lock through the registration, so `claim_restore_dead` is refused however long it takes; SQLite holds the in-process `restore:/
` slot from before the target exists; MongoDB runs the registration in a transaction bounded by `transactionLifetimeLimitSeconds` (default 60 s); Cassandra's registration writes are lightweight transactions bounded by the coordinator's `write_request_timeout_in_ms` and `cas_contention_timeout_in_ms` (Cassandra defaults 2 s and 1 s), and the manifest registration runs no heartbeat during the target's DDL; a registration whose DDL outlasts the stale window can lose its record to a takeover, and its refusal path then conditions the target's removal on its own generation and removes nothing (the registration answers an error naming the lost claim, and the record's owner finishes the restore). A registration that took two periods does not end the job at its first check, since the two-period rule counts from the first beat's return. A job whose conditional write matches no row, or whose conditional write's outcome stays unknown after the re-read, performs no further store or table operation, deletes of a prefix or of a restore target included (5.1 step 5, 5.3 step 7); each unknown outcome is followed by a re-read; the write is sent once more at most; after the second write a re-read that shows the row in the target status counts as matched and anything else as matching no row. The second write is for a backend whose claim connection survives the unknown outcome (MongoDB and Cassandra, whose claim writes go through the shared client); on PostgreSQL an unknown outcome is an error after which the claim session's one connection is dead, so the second write fails with the session, its re-read settles it as matching no row, and the row is the reconciler's once the session's end has freed the advisory lock. PostgreSQL's lock rule applies to the resume's takeover alone: the two writes the reconciler makes to a dead RUNNING manifest record without resuming it, FAILED `OWNER_DEAD` (the target removed) or `target_not_empty` (the target left, 5.3 step 7) under a maintenance claim and FAILED `OTHER_STORE`, test the heartbeat alone on every backend, because the operator's command waits on the first and the second is a configuration this design forbids (4(c): instances of one catalog share one root), so neither waits for a stopped owner's session to end: both go through `fail_restore_job`, which conditions on RUNNING alone after `list_dead_restore_jobs` selected the record by heartbeat age, and on PostgreSQL they do not try the owner's advisory lock, so a stopped owner whose session lives keeps its record on the resume path and loses it on these two. A FAILED `restore_jobs` row whose target table is still CREATING (the FAILED write landed and the target's removal did not, whether the job's own or the reconciler's) is the FAILED-target duty: the reconciler lists it once the row's `completed_at` (or, where none was stamped, its `heartbeat_at`) is older than `stale_job_after_secs`, so a writer still on its own failure path finishes it, and removes the target through `remove_restore_target` with no second FAILED write; PR 6's reconciler runs it for every kind of record, since on this rule the kind does not matter (PR 8 adds the `target_not_empty` exception). On Cassandra the removal is itself two writes, a lightweight transaction claiming the target CREATING to DELETING and then the drop, so the duty also lists a FAILED row whose target is DELETING with its registration still present, on the same staleness rule, and `remove_restore_target` resumes the removal from the drop; PostgreSQL and SQLite remove in one transaction and MongoDB deletes the document first, so neither leaves that state. The jobs and the reconciler are not gated by the PostgreSQL serving registration (#666's `crates/server/src/registration.rs`): after the server's registration is lost it refuses new requests and tries to register again for up to 30 s, and only when that fails, or the backend refuses because a migration holds or waits for the catalog, does it cancel the workers' shutdown token, which the jobs' token is a child of; until then the running jobs and the reconciler's cycles go on writing the catalog and the store, beside a migration that may already have taken the catalog. Gating them on the registration (the reconciler skipping its cycle and `JobPermit` refusing while the server is unregistered) is GitHub issue #755. (f) The maintenance claim. `backup purge`, `backup remove-target` and `backup sync` with `--move-store`, `--forget-store`, `--forget-backup`, `--prune-missing` or `--adopt-store` change rows or bytes that jobs on any instance may be using, and a job permit (5.1 step 1) is per instance, so these commands take a catalog-level maintenance claim first (PR 10's section: the claim per backend and the SQLite serve lock). Every backup and restore job checks the claim after its own claim and row exist (5.1 step 2, 5.3 step 4; this release's in-catalog create and restore check before their row, since the backends commit it and detach, revision item 28) and refuses while it is held; while a claim is held the reconciler runs only the duties that resolve the states the command waits on (DELETING rows, rows whose owner is dead, the FAILED-target duty) and skips every other duty; the command waits until no CREATING or FAILED backup row of the configured store has a live owner (a heartbeat within `stale_job_after_secs` on the catalog's clock, read through `list_backup_rows_older_than`; a FAILED row's heartbeat may still be refreshed by a job that has not seen its claim lost) and no RUNNING `restore_jobs` row remains, a DELETING row not being waited on (nothing heartbeats one; the command finishes it as it finishes every row), and refuses and releases the claim if they have not gone within `2 * stale_job_after_secs`, which it prints; a second maintenance command while one holds the claim fails at once. Because the job checks after its row exists and the command checks rows after its claim exists, one of the two always sees the other. @@ -108,7 +108,7 @@ The invariants and state machines every later PR honours, by lettered part; the (i) Authority: the catalog migration PR 6 adds (migration id 5 on PostgreSQL and SQLite; Cassandra V006 and MongoDB's step `002_backup_catalog_index`, both writing catalog 0.0.5) -the `BackupCatalog` trait of section 3 that each backend implements, and the engine's `backup` module (`crates/engine/src/backup/`), which binds it; this section is the specification they implement. The MongoDB version step and the mixed-version rule are PR 6's section. +the `BackupCatalog` trait of section 3 that each backend implements, and the engine's `backup` module (`crates/engine/src/backup/`), which binds it; this section is the specification they implement. The MongoDB version step and the mixed-version rule are PR 6's section. PR 9 adds migration id 6, `restore_progress` (`MigrationId::RestoreProgress`), a `StoreKind::Data` body on PostgreSQL only (`data_migrations/0006_restore_progress.sql`), `NotApplicable` on SQLite and nothing on MongoDB and Cassandra, whose bootstrappers have no `MigrationId` table (PR 9a's section). ### 5. Operations @@ -126,19 +126,19 @@ Per operation, the contract: the request checks in their order, the error each r #### 5.2 DescribeBackup, ListBackups, DeleteBackup Served from the catalog index with the legacy fallback. DescribeBackup reads the row by the ARN's account and backup id (`BackupCatalog::read_backup_row`) and returns BackupNotFoundException unless the row's `account_id`, `table_name` and `backup_arn` (after the region re-spelling of section 4) equal the request's, as 5.3 step 1 does. ListBackups implements the full request as #384 implements it (the members, the (`created_at`, `backup_arn`) order and the cursor rules with #384's two exceptions), except that `TableName` may be a table name or a table ARN, as the service accepts it (measured on DynamoDB) and as main's CreateBackup already does; the differences doc's `ListBackups` paging row changes with PR 6 (section 10). -DeleteBackup on a CREATING row returns `BackupInUseException`, as on the service; while the backup is in use likewise; on a DELETING row it resumes the deletion inline; on a DELETED or FAILED row it returns `BackupNotFoundException`, as Restore does; on a row of another store it returns the ValidationException of section 4, and while the configured store is unhealthy, on a row of that store the ValidationException of section 7. A backup is in use while a restore target registered from it is CREATING (`BackupCatalog::backup_in_use`), main's rule on every backend, which 5.3 step 1 applies to a second restore of the backup; the rule reads the target's status, not the job's, so on MongoDB it also covers the window between the job's DONE write and the worker's ACTIVE flip (5.3 step 6). DeleteBackup runs the in-use check and its DELETING write in one transaction under the backup row's lock (`BackupCatalog::start_backup_delete`), and a restore's registration (5.3 step 4) re-checks the in-use rule and `status = AVAILABLE` under the same lock, so the two order wholly one before the other, as on main: a DeleteBackup that wins leaves the registration a row that is not AVAILABLE (`BackupNotFoundException`, no target left), and a registration that wins makes the DeleteBackup return `BackupInUseException`; so no restore loads from a backup a DeleteBackup has started to delete, on any backend. Otherwise: set the row DELETING (a tombstone sync honours), delete `extenddb-manifest.json` first so the prefix is no longer a complete backup to any reader (`delete_prefix` on the manifest's exact key, section 2), then `delete_prefix` for the rest, then set the row DELETED (`set_backup_deleted`) once a `list` of the prefix returns nothing; if any step fails the row stays DELETING and the API returns InternalServerError, and the reconciler finishes the deletion later (section 4). The request returns the description with BackupStatus DELETED after the row reaches DELETED. +DeleteBackup on a CREATING row returns `BackupInUseException`, as on the service; while the backup is in use likewise; on a DELETING row it resumes the deletion inline; on a DELETED or FAILED row it returns `BackupNotFoundException`, as Restore does; on a row of another store it returns the ValidationException of section 4, and while the configured store is unhealthy, on a row of that store the ValidationException of section 7. A backup is in use while a restore target registered from it is CREATING (`BackupCatalog::backup_in_use`), main's rule on every backend, which 5.3 step 1 applies to a second restore of the backup; the rule reads the target's status, not the job's, so on MongoDB it also covers the window between the job's DONE write and the worker's ACTIVE flip (5.3 step 6). DeleteBackup runs the in-use check and its DELETING write in one transaction under the backup row's lock (`BackupCatalog::start_backup_delete`), and a store-backed restore's registration (5.3 step 4) re-checks the in-use rule and `status = AVAILABLE` under the same lock (the legacy registration keeps the `backups` row's), so the two order wholly one before the other, as on main: a DeleteBackup that wins leaves the registration a row that is not AVAILABLE (`BackupNotFoundException`, no target left), and a registration that wins makes the DeleteBackup return `BackupInUseException`; so no restore loads from a backup a DeleteBackup has started to delete, on any backend. Otherwise: set the row DELETING (a tombstone sync honours), delete `extenddb-manifest.json` first so the prefix is no longer a complete backup to any reader (`delete_prefix` on the manifest's exact key, section 2), then `delete_prefix` for the rest, then set the row DELETED (`set_backup_deleted`) once a `list` of the prefix returns nothing; if any step fails the row stays DELETING and the API returns InternalServerError, and the reconciler finishes the deletion later (section 4). The request returns the description with BackupStatus DELETED after the row reaches DELETED. #### 5.3 RestoreTableFromBackup -The request runs steps 1, 2 and 4 and returns CREATING in milliseconds plus the manifest parse, as DynamoDB does; the job runs steps 3, 5 and 6. On MongoDB the request's wait for step 4 is bounded (`RESTORE_REGISTRATION_TIMEOUT`, 60 s, against a connection pool that has stopped handing out connections): past it the request answers `InternalServerError` while the detached task goes on, so the target can appear after the error, holding the backup in use and the name until the copy ends, and DescribeTable is where the client finds it (the differences doc's row on how `RestoreTableFromBackup` runs states it for operators). The request's own awaits are the checks of step 1 in their order, the permit, the manifest read and the validation of step 2; the step 4 transaction and the spawn of the job run together in a detached task whose result the request awaits, so a client disconnect cannot commit the table and the RUNNING row without an owner. The table exists from the response onwards, so no second restore or CreateTable can take the name while files download, but it accepts no reads or writes until every file has verified and loaded. +The request runs steps 1, 2 and 4 and returns CREATING in milliseconds plus the manifest parse, as DynamoDB does; the job runs steps 3, 5 and 6. On every backend the request's wait for step 4 is bounded (`RESTORE_REGISTRATION_TIMEOUT`, 60 s, the engine's constant; a connection pool that has stopped handing out connections is the case that meets it): past it the request answers `InternalServerError` while the detached task goes on, so the target can appear after the error, holding the backup in use and the name until the load ends, and DescribeTable is where the client finds it (the differences doc's row on how `RestoreTableFromBackup` runs states it for operators). There are two detaches. Once the request is authenticated and authorized, its whole body (the checks of step 1 in their order, the permit, the manifest read and the validation of step 2, then step 4) runs in a task of its own, so a client disconnect after authorization cancels nothing (#384's rule); inside it, the step 4 transaction and the spawn of the job run together in a second detached task whose result the request awaits under the 60 s bound, so the bound can return while the registration goes on and a client disconnect cannot commit the table and the RUNNING row without an owner. The table exists from the response onwards, so no second restore or CreateTable can take the name while files download, but it accepts no reads or writes until every file has verified and loaded. -1. Check the request in the order main uses and the differences doc records, every refusal that needs no store access before the permit: the overrides' shape (below); the ARN with `BackupArn` (the account must equal the caller's before any store access); the target name (`TableAlreadyExistsException` when a table of that name exists, `TableInUseException` when it is a CREATING restore target); the backup row, read by the ARN's account and backup id and accepted only when its `account_id`, `table_name` and `backup_arn` (after the region re-spelling of section 4) equal the request's, or, if there is no such row, resolved through the `LegacyBackupReader` (section 3): an ARN with neither is `BackupNotFoundException`, so an ARN of account B that carries the id of account A's backup is BackupNotFound and never A's row, a CREATING row returns `BackupInUseException`, a backup in use (the rule of 5.2) returns `BackupInUseException` naming that table, a DELETING, DELETED or FAILED row returns `BackupNotFoundException`, a row of another store returns the ValidationException of section 4 and a row of an unhealthy configured store the one of section 7, none of them reading the store; then, last, the job permit (`LimitExceededException` when none is free). Only then read `extenddb-manifest.sha256` and `extenddb-manifest.json`, verify the sidecar against the manifest bytes, then `format_version`, the ARN and its agreement with `source_table`, and the data-file list rules (`ExtenddbManifest::from_json_bytes`). A manifest missing behind an AVAILABLE row returns `BackupNotFoundException` and logs the row for `backup sync --prune-missing`. +1. Check the request in the order main uses and the differences doc records, every refusal that needs no store access before the permit: the overrides' shape (below); the ARN with `BackupArn` (the account must equal the caller's before any store access); the target name (`TableAlreadyExistsException` when a table of that name exists, `TableInUseException` when it is a CREATING restore target); the backup row, read by the ARN's account and backup id and accepted only when its `account_id`, `table_name` and `backup_arn` (after the region re-spelling of section 4) equal the request's, or, if there is no such row, resolved through the `LegacyBackupReader` (section 3): an ARN with neither is `BackupNotFoundException`, so an ARN of account B that carries the id of account A's backup is BackupNotFound and never A's row, a CREATING row returns `BackupInUseException`, a backup in use (the rule of 5.2) returns `BackupInUseException` naming that table, a DELETING, DELETED or FAILED row returns `BackupNotFoundException`, a row of another store returns the ValidationException of section 4 and a row of an unhealthy configured store the one of section 7, none of them reading the store; then, last, the job permit (`LimitExceededException` when none is free). Only then read `extenddb-manifest.sha256` and `extenddb-manifest.json`, verify the sidecar against the manifest bytes, then `format_version`, the ARN and its agreement with `source_table`, and the data-file list rules (`ExtenddbManifest::from_json_bytes`). The request's sidecar and manifest reads collect at most 4 KiB and 16 MiB; an object over its bound behind an AVAILABLE row is a `ValidationException` naming a corrupt prefix, as a bad sidecar is, and a resume meeting one is FAILED with the same reason. A manifest missing behind an AVAILABLE row returns `BackupNotFoundException` and logs the row for `backup sync --prune-missing`. 2. Validate that the target backend supports every feature in `source_table` that a restore reproduces (a non-empty `vector_indexes` into a backend without vector support is the one case, section 1); refuse before creating anything otherwise. Unlike #384 (`ensure_single_part_base_key`), this path restores multi-part-key tables (as `unordered` backups, section 1): `bulk_load` keys each row and derives its index rows with the functions the backend's item write path uses (section 3), so a restored multi-part-key table is laid out as a natively created one. -3. (Job.) Write the verified manifest and sidecar to the spool, then download every data file to the spool (PR 7's section), hashing each as it arrives and comparing with the manifest's `sha256`, `md5` and `bytes`; on a filesystem store the files are verified in place and no copy is made. A transport error during a download is retried with backoff for up to 15 minutes (a constant, not a knob in v1), the row kept RUNNING and heartbeated, before the restore fails (step 7); a checksum or size mismatch is not retried and fails the restore, naming the file, before any item has been written. Spooling everything before loading is what makes "no item is written before every checksum verifies" true for the second file as well as the first. The spool's `.verified` markers, `spool_host` and the resume rules are PR 9a's section. -4. Create the target table and the `restore_jobs` row (RUNNING, with the job's claim, `kind = manifest` and the store the backup row names; section 4) in one catalog transaction (PostgreSQL and SQLite), one session transaction (MongoDB) or, on Cassandra, one `INSERT ... IF NOT EXISTS` of the `tables` row, whose columns are the job row, so no crash leaves a CREATING table without a job row. If another CreateTable or restore took the name between steps 1 and 4, the request returns `TableAlreadyExistsException` (or `TableInUseException` when the winner is itself a CREATING restore target) and the job row insert rolls back with it; CreateTable keeps `ResourceInUseException`. The registration (`BackupCatalog::claim_restore`) re-checks inside its transaction, under the backup row's lock that DeleteBackup's `start_backup_delete` takes (5.2), that the backup row is still AVAILABLE and that no restore target registered from it is CREATING: a row that is no longer AVAILABLE returns `BackupNotFoundException`, a backup in use returns `BackupInUseException`, and no target is left. After the transaction commits the job checks the maintenance claim (section 4; this release's in-catalog restore checks it before the registration, revision item 28); if it is held, the job runs step 7 with reason `maintenance` and the request returns `LimitExceededException`. Otherwise return `TableDescription` with TableStatus CREATING. Overrides are applied here (below). A restore the reconciler resumes is a job and takes a `max_concurrent_jobs` permit like a new one; when none is free the row is skipped until the next cycle. -5. (Job, after step 3.) Per data file: read the verified file through #394's `ExtenddbManifest::reader_prehashed`, apply the per-file key-range check of section 1 when the backup is key-ordered, validate every item with PR 8's item validator with the target's limits and fail the restore on the first invalid item, then write the file through `bulk_load(target_table_id, claim, file_index, start_at, items)` (section 3), with `start_at` the recorded `items_applied` of a file a resume stopped in and 0 otherwise. Each backend writes base rows and index rows together and records progress so that a crash cannot leave an item applied twice: in one transaction with the data on PostgreSQL and SQLite, and by idempotent re-application of the one unrecorded file or batch on MongoDB and Cassandra (PR 9a's section: the batch sizes and progress records). The restored table has no streams and no TTL, so there is nothing to bypass. -6. (Job.) After all files, and for an unordered backup after the count of section 1 matched, the job flips the target to ACTIVE conditional on the table still being CREATING as a restore target and on the job's `owner_generation`; on PostgreSQL and SQLite the table and every index of the target flip in one catalog transaction, which also marks the `restore_jobs` row DONE; on Cassandra the index rows are created ACTIVE with the table and the lightweight transaction on the `tables` row flips the table and marks the job row DONE. A Query on each GSI, and SearchVectors where the target supports it, therefore sees every restored item as soon as the target reports ACTIVE. MongoDB's sequence differs, because the existing backfill worker, not the job, promotes the indexes: the job marks the `restore_jobs` row DONE (conditioned on its generation and on RUNNING) and sets `restore_backfill_pending` in one session transaction, and the job task ends there; the worker promotes each index and then flips the table to ACTIVE, conditioned on `restore_backfill_pending` and on CREATING, not on the job's generation; and the reconciler never treats a row with `restore_backfill_pending` as a dead RUNNING job (the row is DONE). DeleteTable on a restore target is refused with `ResourceInUseException`, as #384 does and as the service does, which is why the job and the reconciler remove a target through `remove_restore_target` (step 7); there is no user-facing way to cancel a running restore. UpdateTable on a CREATING table is refused as today. -7. Failure (verification or load): the job marks the `restore_jobs` row FAILED with the reason, conditional on its generation. Only if that write matched exactly one row does the job remove the partially created table, through `BackupCatalog::remove_restore_target` (section 3), keyed by the target's account, name and table id and conditional on the table still being a CREATING restore target, which also deletes its progress rows, never through the DeleteTable API; a write that matches zero rows ends the job with only its spool removed, because the claim holder owns the table. DescribeTable then returns ResourceNotFound. Crash mid-restore: the reconciler claims RUNNING `restore_jobs` rows whose owner is dead (section 4(e)) and resumes from the recorded progress, reading the store the job row records if it is the configured store, failing the restore and naming `backup sync --move-store` if it is not, and resuming nothing while the configured store is unhealthy; on PostgreSQL, as on MongoDB and Cassandra, once the row's `heartbeat_at` is older than `stale_job_after_secs`, the delay the prefix deletion already waits (4(e): the advisory lock is the claim's exclusion, not the liveness rule; the owner aborts at its next heartbeat, inside that window). A legacy RUNNING restore whose owner is dead is marked FAILED and its target removed, never resumed (section 3). Shutdown: a restore job stops at its next batch without writing a status, so step 7 does not run; its row stays RUNNING with its spool and its progress, for a successor once the owner is dead by its backend's rule. +3. (Job.) Write the verified manifest and sidecar to the spool, then download every data file to the spool (PR 7's section), hashing each as it arrives and comparing with the manifest's `sha256`, `md5` and `bytes`; on a filesystem store the files are verified in place, with no spool: a hashing `get` pass over every file before any load, then a second `get` of each file as it is loaded, hashed again through `ExtenddbManifest::reader`, so a file replaced between the passes fails in the load; on a filesystem store the load's second `get` hands items to `bulk_load` as they decode and checks the digest at the end of the stream, so the items of a file replaced between the two passes may be written to the CREATING target (committed, on a backend that commits per batch) before the end-of-stream check fails the file and the failure path removes the target; the store root's owner-only mode (section 2) is the guard. A `StoreError::Transport` during a download (at the `get` or mid-body) is retried for up to `DOWNLOAD_RETRY_BOUND`, 15 minutes (a constant, not a knob in v1), after `min(2^n s, 60 s)` plus up to a second of jitter, the row kept RUNNING and heartbeated, before the restore fails (step 7); the bound counts from the job's first download attempt, one clock for every file (`download_all` takes it once and every file's retries count against it), so a store that stays down costs a restore the bound once, not the bound per file; the failure names the file whose attempt crossed it; a filesystem store's `get` errors are `Io` and fail at once, so the retry covers S3 stores; `NotFound`, a denial, and a checksum or size mismatch are not retried and fail the restore, naming the file, before any item has been written. Spooling everything before loading is what makes "no item is written before every checksum verifies" true for the second file as well as the first. On an S3 store the manifest and its sidecar are written to the spool entry before the first download, each as a data file is (`.partial`, synced, renamed), so a crash during the write leaves the earlier document or none and never a truncated one; a resume that read the pair from the spool does not write it again, and a spooled pair that fails verification (the sidecar, the parse, the record's ARN) is replaced by the store's, with one WARN line. The spool's `.verified` markers, `spool_host` and the resume rules are PR 9a's section. +4. Create the target table and the `restore_jobs` row (RUNNING, with the job's claim, `kind = manifest` and the store the backup row names; section 4) in one catalog transaction (PostgreSQL and SQLite), one session transaction (MongoDB) or, on Cassandra, one `INSERT ... IF NOT EXISTS` of the `tables` row, whose columns are the job row, so no crash leaves a CREATING table without a job row. If another CreateTable or restore took the name between steps 1 and 4, the request returns `TableAlreadyExistsException` (or `TableInUseException` when the winner is itself a CREATING restore target) and the job row insert rolls back with it; CreateTable keeps `ResourceInUseException`. The registration (`BackupCatalog::claim_restore`) re-checks inside its transaction, under the `backup_index` row's lock that DeleteBackup's `start_backup_delete` takes (5.2: PostgreSQL `SELECT ... FOR UPDATE`, SQLite the `BEGIN IMMEDIATE` write transaction, MongoDB `$inc lock_seq` on the document, Cassandra the `active_restores` lightweight transaction on the backup's partition), that the backup row is still AVAILABLE and that no restore target registered from it is CREATING: a row that is no longer AVAILABLE returns `BackupNotFoundException`, a backup in use returns `BackupInUseException`, and no target is left. The task then sends one heartbeat through the claim and takes its return as the instant the heartbeat's two-period rule counts from (`heartbeat.rs`), checks the maintenance claim, and replies: a first beat that finds the claim lost, or fails, runs step 7 with reason "the claim was lost before the job started" and the request returns `InternalServerError` (4(e)); a held maintenance claim (section 4; this release's in-catalog restore checks it before the registration, revision item 28) runs step 7 with reason `maintenance` and the request returns `LimitExceededException`. Otherwise return the `TableDescription` the registration produced, with TableStatus CREATING and its `RestoreSummary`. Overrides are applied here (below). A restore the reconciler resumes is a job and takes a `max_concurrent_jobs` permit like a new one; when none is free the row is skipped until the next cycle. +5. (Job, after step 3.) Per data file: read the verified file through #394's `ExtenddbManifest::reader_prehashed` from the spool and `ExtenddbManifest::reader` in place (a filesystem store's file, hashed again as it is read), apply the per-file key-range check of section 1 when the backup is key-ordered, validate every item with PR 8's item validator with the target's limits and fail the restore on the first invalid item, then write the file through one `bulk_load(target, claim, BulkLoadFile { file_index, start_at, reapplying }, items)` (section 3) over a stream fed in chunks of at most `LOAD_CHUNK_ITEMS` (1000) items or `LOAD_CHUNK_BYTES` (32 MiB) of data-file line text (the unit the backup side budgets in; a chunk closes at the first item that takes it to the bound, so it holds at most the bound plus one line) through a channel of two chunks, with `start_at` the recorded `items_applied` of a file a resume stopped in and 0 otherwise, and `reapplying` true for the first file a resume meets and false after. Each backend writes base rows and index rows together and records progress in the write that applied the items, so that a crash cannot leave an item applied twice: in one transaction with the data on PostgreSQL (a `restore_progress` row per file in the data database, fenced by the base table's primary key and the progress row's own key rather than by the generation, which fences only the record's writes in the catalog) and SQLite (the record's `(file_index, items_applied)` per batch), and by idempotent re-application of the one unrecorded file or batch on MongoDB (the record's `file_index` per file; E11000 suppressed only on the `reapplying` file) and Cassandra (the record's progress per batch under a lightweight transaction; `put_restored_item` is an upsert). The batches: PostgreSQL one data transaction per file, the base rows in multi-row INSERTs closed at 500 rows or 4 MiB of item text (`COPY_BATCH_ITEMS`, `COPY_BATCH_BYTES`), the progress row last in the transaction; SQLite one write transaction per batch, closed at 10,000 items or at the first item that takes the batch's serialized `item_data` text past 32 MiB (`RESTORE_BATCH_ITEMS`, `RESTORE_BATCH_BYTES` in `extenddb_storage::restore_engine`), the record's progress last in the transaction; MongoDB `insert_many(ordered = false)` calls of 1,000 documents (the driver splits a call by the server's message and write-batch bounds; 1,000 is the unit a half-applied file is re-applied in), the record's `file_index` written after the file; Cassandra upserts in batches closed at 10,000 items or 32 MiB of serialized item text, the same two constants, each batch followed by the progress lightweight transaction; an empty file records its progress on every backend: `items_applied = start_at` on PostgreSQL, SQLite and Cassandra, the `file_index` alone on MongoDB, whose record carries no `items_applied` (PR 9a's section). Liveness inside a file: the feeder checks at every chunk boundary that the heartbeat confirmed the claim within two periods and ends the stream as a lost claim otherwise; on MongoDB and Cassandra `bulk_load` reads the record before each write batch (MongoDB one `find_one` on the target's `_id`, `table_id`, `restore_job.status` and `restore_job.owner_generation` before each `insert_many`; Cassandra one LOCAL_QUORUM read of `table_id`, `restore_status` and `restore_owner_generation` before each batch of upserts) and ends the file as a lost claim when the record is not RUNNING under its generation, so a writer whose record another instance took over while it stalled lands at most the one batch it was in; on Cassandra an unknown outcome of the progress lightweight transaction (a write timeout or failure, a lost connection) is decided by a `LOCAL_SERIAL` re-read of the record: RUNNING under the generation and carrying the values written is applied; RUNNING under the generation with the earlier values is one retry, whose own unknown outcome is re-read the same way, and a second unknown outcome the record does not carry fails the restore; anything else is the lost claim. A stalled owner whose record a successor took over, flipped ACTIVE and whose clients then wrote, can land at most the one write unit it was in before its next liveness read ends it: on MongoDB an `insert_many` that re-inserts a key a client has since deleted, as a base document the backfill will not revisit; on Cassandra upserts of the backup's items over client writes, and (with #742 fixed) an unchanged item's upsert rewrites the same base and index rows rather than deleting the index rows. The window is a stall longer than `stale_job_after_secs` plus the successor's whole restore. PostgreSQL's window stays as stated: nothing in its `bulk_load` conditions on the generation; the base table's primary key and the claim's liveness, confirmed before each file, are its fence, the advisory lock holds the record through a stall while the owner's session lives, and the window that remains is a session the database server dropped during a stall longer than the stale window (an `idle_session_timeout`, not a default): the stalled file's transaction can commit after a takeover, and the successor's own load of that file then fails on the duplicate key and the restore is FAILED and its target removed, a lost restore and not wrong data. The restored table has no streams and no TTL, so there is nothing to bypass. +6. (Job.) After all files, and after the count of section 1 matched, the job flips the target to ACTIVE through `finish_restore` (section 3), conditional on the table still being CREATING as a restore target and on the job's `owner_generation` (`NotMatched` otherwise: an error line and no table access); on PostgreSQL and SQLite the table and every index of the target flip in one catalog transaction, which also marks the `restore_jobs` row DONE; on Cassandra the index rows are created ACTIVE with the table and the lightweight transaction on the `tables` row flips the table and marks the job row DONE (`Active`, never `Pending`, there). A `finish_restore` that returns an error is a conditional write whose outcome is unknown and is handled as the status writes are: the record is re-read through the claim; DONE under the job's generation is the write landed and the restore complete (one WARN naming the error); RUNNING under it is the write not taken, sent once more at most and its own error re-read the same way; anything else, a second error included, is the DONE write that matched no record, after which the job makes no table access and the reconciler resumes the record once stale. `finish_restore` writes the manifest's `source_table.table_size_bytes` as the restored table's size until the statistics pass rewrites it; `DescribeTable` of a restored table reports the source table's size, not the compressed size of the backup's files. A Query on each GSI, and SearchVectors where the target supports it, therefore sees every restored item as soon as the target reports ACTIVE (on Cassandra this holds for a resumed restore's re-applied batch because the write path rewrites an unchanged item's index rows instead of deleting them, #742, the PR 9a load paragraph). MongoDB's sequence differs, because the existing backfill worker, not the job, promotes the indexes: the job marks the `restore_jobs` row DONE (conditioned on its generation and on RUNNING) and sets `restore_backfill_pending` in one session transaction (`finish_restore` returns `Pending`), and the job task ends there; the worker promotes each index and then flips the table to ACTIVE, conditioned on `restore_backfill_pending` and on CREATING, not on the job's generation; and the reconciler never treats a row with `restore_backfill_pending` as a dead RUNNING job (the row is DONE). DeleteTable on a restore target is refused with `ResourceInUseException`, as #384 does and as the service does, which is why the job and the reconciler remove a target through `remove_restore_target` (step 7); there is no user-facing way to cancel a running restore. UpdateTable on a CREATING table is refused as today. +7. Failure (verification or load): the job marks the `restore_jobs` row FAILED with the reason, conditional on its generation. Only if that write matched exactly one row does the job remove the partially created table, through `BackupCatalog::remove_restore_target` (section 3), keyed by the target's account, name and table id and conditional on the table still being a CREATING restore target, which also deletes its progress rows, never through the DeleteTable API; a write that matches zero rows ends the job with no further table access and its spool entry left to the sweep, because the claim holder owns the table and may be reading that entry (the entry is keyed by table id, so a same-host successor shares it). DescribeTable then returns ResourceNotFound. Crash mid-restore: once the server's shutdown has begun (the jobs' token is cancelled) the reconciler takes no dead record over and writes no FAILED on the maintenance or other-store arm: the record is left as it is for the next instance's reconciler, since a resume spawned on the way out would stop at its first check and leave the record with a fresh heartbeat, a second stale window for the next instance to wait. Otherwise the reconciler claims RUNNING `restore_jobs` rows whose owner is dead (`claim_restore_dead`, section 4(e)) and resumes from the recorded progress (a dead record met under a maintenance claim is failed through the unclaimed RUNNING-only write before its store is looked at: `target_not_empty` with its target left for `remove-target` (which runs under the same claim) when it recorded no progress and its target holds items, else `OWNER_DEAD` with its target removed; the command's wait is satisfied either way, the record being no longer RUNNING; a record of another store is failed `OTHER_STORE` below, both arms on the heartbeat alone on every backend, 4(e); a resume takes a job permit and is skipped to the next cycle without one; a resume with no recorded progress into a non-empty target is FAILED `target_not_empty` and its target left for `extenddb backup remove-target --account
` (section 4's duty list), which ships with PR 9: on MongoDB a crash before the first progress record (the whole first data file) and on Cassandra before the first (the first 10,000 items or 32 MiB) leaves written items and no progress, so the ordinary crash there ends in this state and the release that ships the restore ships its remedy), reading the store the job row records if it is the configured store, failing the restore and naming `backup sync --move-store` if it is not (`OTHER_STORE`, the target left to the FAILED-target duty), and resuming nothing while the configured store is unhealthy; on SQLite at once after a restart, by the slot rule; on PostgreSQL, as on MongoDB and Cassandra, once the row's `heartbeat_at` is older than `stale_job_after_secs`, the delay the prefix deletion already waits (4(e): the advisory lock is the claim's exclusion, not the liveness rule; the owner aborts at its next heartbeat, inside that window). A legacy RUNNING restore whose owner is dead is marked FAILED and its target removed, never resumed (section 3). The reconciler's sweep of `restore_progress` rows (PostgreSQL) is a positive list: the backend reads the distinct table ids the progress rows name, then the catalog's RUNNING records, and deletes only the rows of ids in the first set that are in neither the second nor the reconciler's own RUNNING list; a target whose first row lands after the first read is never a candidate, and one registered before it is RUNNING at the second read, so a live job's row is never deleted. Shutdown: a restore job stops at its next chunk without writing a status, so step 7 does not run; its row stays RUNNING with its spool and its progress, for a successor once the owner is dead by its backend's rule; it records `RestoreJobs{cancelled}` and no worker metric (section 9), where a lost claim or a DONE write that matched no record records `RestoreJobs{failed}` and `WorkerErrorCount{RestoreJob}` and leaves the spool entry to the sweep: the reconciler's spool sweep lists the spool directory first and the RUNNING records second, and removes only the entries it listed that no RUNNING record of this host's `spool_host` names, so a restore whose registration commits and whose entry opens between the two reads is never a candidate; an entry its own job removed between the listing and the removal is done, not an error. Restore overrides. #384 applies every override the operation models as the service applies it (`crates/engine/src/backup/overrides.rs`) and refuses only `VectorIndexOverride`, which the botocore model this project targets does not carry, with a typed ValidationException. The v1 contract keeps exactly that: the new path applies the overrides at step 4 to the manifest's `source_table` before the target is created, through the same `overrides` module, and `VectorIndexOverride` keeps the refusal. The differences and limits docs' override rows stay as they are. @@ -171,7 +171,7 @@ spool_path = "" # restore download area for an S3 store; d On SQLite (file-backed) a missing `[backup]` section is equivalent to `kind = "filesystem"` with `path` = `backups/` beside the database file; a `validate` failure of that defaulted root is a logged warning that leaves the store unconfigured, not a startup error, so an upgrade never stops a server that never configured backups, while an explicitly configured root that fails `validate` is still a startup error. The warning holds only while the catalog has no backup rows for the defaulted root (`read_store_ids`); once it has, the root has been used and is handled as an explicitly configured one (revision item 28). On PostgreSQL and MongoDB a missing `[backup.store]` leaves the off-host path unconfigured and startup logs that; `kind = "disabled"` leaves it unconfigured on purpose on any backend. Without a configured store, CreateBackup on every backend uses the legacy in-catalog create (`LegacyBackupReader::create`, section 3) and logs a deprecation warning naming `[backup.store]` on each call, until PR 14 removes that path; after it CreateBackup without a configured store returns a ValidationException naming `[backup.store]`. Legacy backups stay readable, listable, restorable and deletable either way. `sqlite-memory` (any `:memory:` database path) has no database file and so no default root: CreateBackup there takes the off-host path only with `[backup.store]` set and the legacy create without it. Cassandra has no `SnapshotEngine`, so it uses the legacy create even with a store configured (section 3); a configured store serves restores of off-host backups into Cassandra and nothing else there. -Memory per running job, which operators size hosts from. A backup's terms, for a row of `r` bytes of item text whose decoded `Item` is `d` bytes (`d` is about 1.2 × `r` for a row of many short strings, #394's worst shape, where every `N` is a 133-character string): the read path holds the page's decoded rows, `d` each, while they wait for the pipeline, at most the rows of the budget PR 8 adds to the PostgreSQL and SQLite streams (`SNAPSHOT_FETCH_BYTES`, 64 MiB, a constant beside `DEFAULT_FETCH_BATCH`, not a `[backup]` key in v1) plus one row, and one row's parse, `r + 2d`, since the deserializer builds an intermediate value before the item; PostgreSQL pulls the page's rows from the connection one at a time and parses each as it arrives, so it holds no page text beyond the row being parsed, and SQLite pulls a row at a time with the driver's worker at most five rows ahead of the pull, `r` each (`row_buffer_size` on the pool, four, plus the row the worker holds while its `send` waits; 160 MiB at the line cap; the default of 50 made the run-ahead, not the page, the largest term on a table of large rows). The pipeline holds the feeder's item, the item channel's items (`ITEM_CHANNEL`, four) and the encoder's item, `d` each; the encoder's line buffer (one line, at most 32 MiB, the line cap) and gzip window; the feeder's measurement of an item over the default `max_item_size_bytes` holds no line (it counts the line's bytes through a writer that stops at the cap), so the terms are the same on an instance that raises the limit, at the largest stored row; the chunk sink's 1 MiB buffer and the chunk channel (four 1 MiB chunks); the upload window (`upload_part_size_bytes * max_concurrent_uploads` on an S3 store, one write buffer on a filesystem store); and the manifest's data-file entries until the manifest is written. No term grows with `data_file_target_bytes` or with the table, except on `sqlite-memory`, whose open copies the whole table (section 3); every term but the budget and the chunks grows with the row, which `max_item_size_bytes` bounds (the line cap bounds the line the job writes and names the item it refuses; the row is read and decoded before the feeder measures it). The sizing rule, per running job, is the terms at the line cap times the allocator's factor, times `max_concurrent_jobs`; the rule is stated at the default `max_item_size_bytes`, where no stored item's line exceeds the cap (27.7 MB is the largest line a 400 KB item expands to, `MAX_LINE_BYTES`'s doc); on an instance that raises it the page's decoded rows, one row's parse and the pipeline's items are at the largest stored row, which only `max_item_size_bytes` bounds (about 68 times the item's size as a line, for #394's worst shape). The terms at the line cap (`r` = 32 MiB, `d` = 1.2 `r`): the page's rows, 3 `d` = 115 MiB; one row's parse, `r` + 2 `d` = 109 MiB; the pipeline's six items, 6 `d` = 230 MiB; the line buffer, 32 MiB; the chunks, 5 MiB; on SQLite the run-ahead, 5 `r` = 160 MiB: about 490 MiB on PostgreSQL and 650 MiB on SQLite, an upper bound, since the run-ahead and a full pipeline are not both at their largest at one instant. The allocator's factor is glibc's: the row's 200,000 short strings are allocated on the tokio worker that parses the row and freed on the blocking thread that encodes it, and with glibc's default per-thread arenas that memory stays in the arenas between pages, so the process's high-water mark grows with the number of large rows the job reads and varies by about 200 MiB between identical runs; under `MALLOC_ARENA_MAX=1` (one arena, set in the server's environment on a host that backs up tables of large items; the admin guide's `max_concurrent_jobs` row carries it and the deployment guide's service unit carries the line commented out with that condition, since one arena serializes the worker threads' allocations on one lock and costs the request path 3.5 to 9.2% on SQLite and about 2% on PostgreSQL (the scans, 2.1%), measured on GetItem, Query and Scan with a Python client, the record) the peak is the terms' and does not grow with the table. Measured on PR 8's 16-core host at 27.7 MB rows (`r` = 26.4 MiB, three rows to a page; the terms' sum 545 MiB on SQLite), 1,000 small rows then 12, 24 and 48 such rows, each series six runs back to back on one SQLite binary (`docs/design/15-backup-restore-bench-pr8.md`; revision item 29): with the default arenas the peak over idle was 466, 588 and 760 MiB on a quiet host and 707, 700 and 1,281 MiB on the final build with the kill-point runs beside it (0.86 to 2.35 of the terms, growing with the rows and varying by hundreds of MiB between series; runs of the PR's first build on a loaded host reached 1,096 MiB at 48 rows; no plateau was measured), and under `MALLOC_ARENA_MAX=1` 481, 452 and 480 MiB and then 452, 480 and 455 MiB (0.83 to 0.88 of the terms, flat from 12 to 48 rows in both series; the SQLite ratios are of the peak over idle), and on PostgreSQL at 12 rows 340 MiB `VmHWM` under one arena against 750 to 788 MiB under the default arenas (0.82 of the terms at that row on PostgreSQL, 413 MiB, which has no run-ahead; that ratio is of the `VmHWM`, the process's whole high-water mark, and about 0.47 over its idle of 144 MiB); so the bound an operator can hold the process to is the terms at the line cap times 1.2 per job under `MALLOC_ARENA_MAX=1`, and under the default arenas a figure that grows with the large rows a job reads, 2.35 of the terms at 48 rows being the largest measured. A one-item channel saved about 50 MiB of the peak and cost a 1M-row backup 75% of its duration, so the channel stays at four. With 200-byte rows every term that grows with the row is under 1 MiB, the chunk channel and the sink hold their 5 MiB whatever the row, and the high-water mark of a 1M-row backup is within 10 MiB of a 100k-row backup's on PostgreSQL and SQLite (the bound the acceptance asserts is 112 MiB: the budget, the line cap and the channels) and the 500k-row backup's within 5 MiB on MongoDB, where the 500k run stands in for the 1M run (its load alone took 27 minutes through the wire). A restore holds one line's decode plus one insert batch plus the parsed manifest for the whole job; neither grows with the table. Each running backup also holds one connection from the PostgreSQL data pool for its duration, and on the separate layout one catalog-pool connection while its snapshot opens (section 3); the backend admits at most half the pool (after the raise below) as open snapshots and refuses more with `LimitExceeded`; a running restore holds one data connection and has no slot of its own after PR 6; and each PostgreSQL job holds one dedicated claim connection outside the pool. `pool_size` sizes both pools, and the rule is `pool_size >= max(MIN_POOL_SIZE, 2 * max_concurrent_jobs + 10)`: twice the jobs, so the half-pool bound admits every job `max_concurrent_jobs` allows, plus ten for requests; 18 at the defaults, under the default `pool_size` of 20. It applies whether or not `[backup.store]` is configured, because legacy restores are jobs too. Startup raises a smaller `pool_size` to that value with a warning naming both numbers, as it already raises a `pool_size` under `MIN_POOL_SIZE` (10) today. On SQLite the snapshot slot (`backup_server_slot_count`, `pool_size / 2 - 1` capped at 16, shared by the in-catalog create and the snapshot open) stays the backend's admission bound, as PostgreSQL's half-pool slot does, and a server raises `[storage.sqlite] pool_size` to `2 * (max_concurrent_jobs + 1)` (at most 34) with a warning naming both numbers so the slot refuses no job the permit admitted; 10 at the defaults, the default pool; above 16 jobs the slot refuses the seventeenth. +Memory per running job, which operators size hosts from. A backup's terms, for a row of `r` bytes of item text whose decoded `Item` is `d` bytes (`d` is about 1.2 × `r` for a row of many short strings, #394's worst shape, where every `N` is a 133-character string): the read path holds the page's decoded rows, `d` each, while they wait for the pipeline, at most the rows of the budget PR 8 adds to the PostgreSQL and SQLite streams (`SNAPSHOT_FETCH_BYTES`, 64 MiB, a constant beside `DEFAULT_FETCH_BATCH`, not a `[backup]` key in v1) plus one row, and one row's parse, `r + 2d`, since the deserializer builds an intermediate value before the item; PostgreSQL pulls the page's rows from the connection one at a time and parses each as it arrives, so it holds no page text beyond the row being parsed, and SQLite pulls a row at a time with the driver's worker at most five rows ahead of the pull, `r` each (`row_buffer_size` on the pool, four, plus the row the worker holds while its `send` waits; 160 MiB at the line cap; the default of 50 made the run-ahead, not the page, the largest term on a table of large rows). The pipeline holds the feeder's item, the item channel's items (`ITEM_CHANNEL`, four) and the encoder's item, `d` each; the encoder's line buffer (one line, at most 32 MiB, the line cap) and gzip window; the feeder's measurement of an item over the default `max_item_size_bytes` holds no line (it counts the line's bytes through a writer that stops at the cap), so the terms are the same on an instance that raises the limit, at the largest stored row; the chunk sink's 1 MiB buffer and the chunk channel (four 1 MiB chunks); the upload window (`upload_part_size_bytes * max_concurrent_uploads` on an S3 store, one write buffer on a filesystem store); and the manifest's data-file entries until the manifest is written. No term grows with `data_file_target_bytes` or with the table, except on `sqlite-memory`, whose open copies the whole table (section 3); every term but the budget and the chunks grows with the row, which `max_item_size_bytes` bounds (the line cap bounds the line the job writes and names the item it refuses; the row is read and decoded before the feeder measures it). The sizing rule, per running job, is the terms at the line cap times the allocator's factor, times `max_concurrent_jobs`; the rule is stated at the default `max_item_size_bytes`, where no stored item's line exceeds the cap (27.7 MB is the largest line a 400 KB item expands to, `MAX_LINE_BYTES`'s doc); on an instance that raises it the page's decoded rows, one row's parse and the pipeline's items are at the largest stored row, which only `max_item_size_bytes` bounds (about 68 times the item's size as a line, for #394's worst shape). The terms at the line cap (`r` = 32 MiB, `d` = 1.2 `r`): the page's rows, 3 `d` = 115 MiB; one row's parse, `r` + 2 `d` = 109 MiB; the pipeline's six items, 6 `d` = 230 MiB; the line buffer, 32 MiB; the chunks, 5 MiB; on SQLite the run-ahead, 5 `r` = 160 MiB: about 490 MiB on PostgreSQL and 650 MiB on SQLite, an upper bound, since the run-ahead and a full pipeline are not both at their largest at one instant. The allocator's factor is glibc's: the row's 200,000 short strings are allocated on the tokio worker that parses the row and freed on the blocking thread that encodes it, and with glibc's default per-thread arenas that memory stays in the arenas between pages, so the process's high-water mark grows with the number of large rows the job reads and varies by about 200 MiB between identical runs; under `MALLOC_ARENA_MAX=1` (one arena, set in the server's environment on a host that backs up tables of large items; the admin guide's `max_concurrent_jobs` row carries it and the deployment guide's service unit carries the line commented out with that condition, since one arena serializes the worker threads' allocations on one lock and costs the request path 3.5 to 9.2% on SQLite and about 2% on PostgreSQL (the scans, 2.1%), measured on GetItem, Query and Scan with a Python client, the record) the peak is the terms' and does not grow with the table. Measured on PR 8's 16-core host at 27.7 MB rows (`r` = 26.4 MiB, three rows to a page; the terms' sum 545 MiB on SQLite), 1,000 small rows then 12, 24 and 48 such rows, each series six runs back to back on one SQLite binary (`docs/design/15-backup-restore-bench-pr8.md`; revision item 29): with the default arenas the peak over idle was 466, 588 and 760 MiB on a quiet host and 707, 700 and 1,281 MiB on the final build with the kill-point runs beside it (0.86 to 2.35 of the terms, growing with the rows and varying by hundreds of MiB between series; runs of the PR's first build on a loaded host reached 1,096 MiB at 48 rows; no plateau was measured), and under `MALLOC_ARENA_MAX=1` 481, 452 and 480 MiB and then 452, 480 and 455 MiB (0.83 to 0.88 of the terms, flat from 12 to 48 rows in both series; the SQLite ratios are of the peak over idle), and on PostgreSQL at 12 rows 340 MiB `VmHWM` under one arena against 750 to 788 MiB under the default arenas (0.82 of the terms at that row on PostgreSQL, 413 MiB, which has no run-ahead; that ratio is of the `VmHWM`, the process's whole high-water mark, and about 0.47 over its idle of 144 MiB); so the bound an operator can hold the process to is the terms at the line cap times 1.2 per job under `MALLOC_ARENA_MAX=1`, and under the default arenas a figure that grows with the large rows a job reads, 2.35 of the terms at 48 rows being the largest measured. A one-item channel saved about 50 MiB of the peak and cost a 1M-row backup 75% of its duration, so the channel stays at four. With 200-byte rows every term that grows with the row is under 1 MiB, the chunk channel and the sink hold their 5 MiB whatever the row, and the high-water mark of a 1M-row backup is within 10 MiB of a 100k-row backup's on PostgreSQL and SQLite (the bound the acceptance asserts is 112 MiB: the budget, the line cap and the channels) and the 500k-row backup's within 5 MiB on MongoDB, where the 500k run stands in for the 1M run (its load alone took 27 minutes through the wire). A running restore holds the parsed manifest, the decoder's line (at most `MAX_LINE_BYTES`) and the deserializer's intermediate value for it, the decoded items of `LOAD_CHUNK_BYTES` (32 MiB) of data-file text plus one line per chunk (the chunk closes with the item that reaches the bound) at four chunk positions (two in the channel to `bulk_load`, the one being filled, the one being drained), and the backend's write batch, times `max_concurrent_jobs`; the chunk bound counts the bytes the decoder read for the chunk's items, the unit the backup side budgets in, not the service's item sizing, and a chunk also closes at `LOAD_CHUNK_ITEMS` (1,000) items. The backend's write batch: at most 32 MiB of item text on SQLite and Cassandra (10,000 items when they are smaller), held beside the decoded items of that text, 4 MiB per INSERT statement on PostgreSQL (500 rows when they are smaller; one transaction per file; the item and its text, two copies), 1,000 documents per `insert_many` on MongoDB (the driver splits the call; the documents are held until it returns, bounded in bytes only by the data file, since `bulk_load` is per file). A `bulk_load` is one data file at a time, so the chunks in flight never hold more than one file's items, at most the source's `data_file_target_bytes` plus one line of text. None of these grows with the table: they grow with the item, which `max_item_size_bytes` bounds, and SQLite's `COUNT(*)` after the last file costs the counting connection's page cache, bounded by `cache_size` (32 MiB; 22 to 28 MiB measured at 1M and at 3M rows, the record). The input order of the backup's lines is a term of SQLite's beside these: a key-ordered backup (the server's own, of a hash-only or single-range-key table) appends to the base b-tree's right edge, an unordered one (every MongoDB-sourced backup and every multi-part-key table) dirties pages across the whole tree, and the write connection's page cache plus the allocator's arenas hold the difference (the record's input-order table: 1M items of 200 bytes 87 to 98 MiB key-ordered against 144 to 164 MiB unordered on SQLite, in 27 to 56 s against 66 to 90 s; PostgreSQL 16 MiB either way). A table with secondary or vector indexes adds no term of the restore's: on SQLite it adds the backend's own per-connection page cache for the index b-trees (`cache_size`, 32 MiB), bounded by that setting and growing sub-linearly with the table (the record's index table), on PostgreSQL a few MiB. Measured (`docs/design/15-backup-restore-bench-pr9.md`; revision item 30), `VmHWM` over idle from a fresh start, 24 worst-shape items of 27.7 MB: 848 MiB on SQLite and 778 MiB on PostgreSQL under the default arenas (848 and 1,013 MiB at 12 and 48 items on SQLite: the peak grows with the large items a restore reads, as the backup's does, and no bound was measured), 392 MiB on SQLite and 427 MiB on PostgreSQL under `MALLOC_ARENA_MAX=1`; 1,200 items of 350 KB in one file 270 MiB on SQLite and 156 MiB on PostgreSQL; 1M items of 200 bytes under a composite key with no secondary index, the backup's lines in key order, 89 MiB on SQLite and 16 MiB on PostgreSQL (100k: 38 and 10.7 MiB), restored in 26.8 s and 9.5 s; the same items under the same key with two full-projection GSIs 143 to 225 MiB on SQLite at 100,000 items in three runs and 165 to 172 MiB at 300,000, and 23 to 34 MiB on PostgreSQL at both: the excess on SQLite is the backend's own per-connection page cache (`cache_size`, 32 MiB) holding the index tables' b-tree pages plus the allocator's arenas (about a third less under `MALLOC_ARENA_MAX=1`), not the restore's buffers, which the 32 MiB batch bound holds whatever the indexes; it is bounded by `cache_size` and grows sub-linearly with the table. A restore of a table of large items is of the same order as its backup, not less; the sizing rule is the record's: about 450 MiB per running restore of large items under one arena, times `max_concurrent_jobs`. Each running backup also holds one connection from the PostgreSQL data pool for its duration, and on the separate layout one catalog-pool connection while its snapshot opens (section 3); the backend admits at most half the pool (after the raise below) as open snapshots and refuses more with `LimitExceeded`; a running restore holds one data connection and has no slot of its own after PR 6; and each PostgreSQL job holds one dedicated claim connection outside the pool. `pool_size` sizes both pools, and the rule is `pool_size >= max(MIN_POOL_SIZE, 2 * max_concurrent_jobs + 10)`: twice the jobs, so the half-pool bound admits every job `max_concurrent_jobs` allows, plus ten for requests; 18 at the defaults, under the default `pool_size` of 20. It applies whether or not `[backup.store]` is configured, because legacy restores are jobs too. Startup raises a smaller `pool_size` to that value with a warning naming both numbers, as it already raises a `pool_size` under `MIN_POOL_SIZE` (10) today. On SQLite the snapshot slot (`backup_server_slot_count`, `pool_size / 2 - 1` capped at 16, shared by the in-catalog create and the snapshot open) stays the backend's admission bound, as PostgreSQL's half-pool slot does, and a server raises `[storage.sqlite] pool_size` to `2 * (max_concurrent_jobs + 1)` (at most 34) with a warning naming both numbers so the slot refuses no job the permit admitted; 10 at the defaults, the default pool; above 16 jobs the slot refuses the seventeenth. A configuration error that needs no request to the store (a part size, prefix or key-length problem, an insecure endpoint, a root that fails the owner-only rule or whose ancestors others can write to, a filesystem root that is or contains the database directory, a spool rule) is a startup error naming the path or bucket, except for the defaulted SQLite root above while the catalog has no rows for it, where it and a failure of the root's identity check are a warning that leaves the store unconfigured. A filesystem root's `validate` also writes a probe file, a request to the store: an I/O failure there or in resolving the root is a store condition that degrades the store (`failed validation`), as the same condition one reconciler cycle later does, not a startup error. A filesystem store's `validate` and identity check run before the listener, bounded by a timeout, so a mount that stops answering after the root is opened leaves the server starting with the store `checking` (a root that does not answer at startup holds the server before the listener, as the database directory would); an S3 store's are requests to the endpoint and run after the listener is up, in the reconciler's first cycle, with the store `checking` until the first result reports, so a stalled endpoint never holds the data plane (revision item 28). Every later check runs under the same bound (5 s for a filesystem root; 30 s for a bucket, under the SDK's per-attempt timeout so no retry outlives the check) and against the shutdown token: a store that has stopped answering costs a reconciler cycle its bound and nothing more, the cycle's other duties run, the check is recorded as a transport failure, and neither the drain at shutdown nor the process's exit waits on it (a filesystem root's abandoned call stays parked on a thread until the mount answers, no later check starts while it is, and the binary ends its runtime with a 5 s bound). A transport failure (no region, no credentials, an unreachable endpoint, a check that did not finish within its bound), a missing or denied bucket found after the listener, and a failure of section 4's identity check or manifest probe, do not stop startup, because a store problem must not take the data plane out of rotation (section 9): the failure is logged with its cause, the store component is reported unhealthy with the problem's category as its reason, the operations that touch the store refuse with a ValidationException that names the store's kind and the category and not its root (CreateBackup on PostgreSQL, SQLite and MongoDB, and RestoreTableFromBackup and DeleteBackup of an off-host backup; Describe and List, legacy backups and the in-catalog CreateBackup are unaffected), the reconciler acts on no row of the store (its legacy duty, which touches no store, still runs) and resumes no restore, and the check runs again each reconciler cycle until it passes. @@ -187,7 +187,7 @@ Authority: the `[backup]` struct in `crates/config` (PR 6 adds it with the job k ### 9. Observability -Metrics: `backup_jobs_total{outcome=succeeded|failed|cancelled}`, `backup_job_duration_seconds`, `backup_bytes_uploaded_total`, `restore_jobs_total{outcome}`, `restore_job_duration_seconds`, `backup_snapshot_age_seconds` (gauge while a job runs), `backup_store_errors_total{op}`. Logs: one info line at job start and completion with ARN, item count, bytes, duration; one error line per failure with the reason. The `/health` readiness work from the operations report should include store reachability as a component, not as a hard failure (a backup store outage should not take the data plane out of rotation). `/health` is a static 200 today; the component model is new work in the configuration PR, and a transport failure of the store, or a failure of its identity check or manifest probe (section 4), marks the component unhealthy rather than stopping the server (section 7). Metric names are variants of the closed `MetricName` enum in `crates/core/src/metrics`, and `/metrics` emits them under those names as JSON (`render_metrics`, `docs/design/06-component-server.md`: the endpoint is not Prometheus exposition format; the deployment guide tells operators to convert); the snake-case spellings above are the names a converter would give them, not a rendering the server does. PR 8 adds `BackupJobs` (one count per job end, `operation` = the outcome, `succeeded`, `failed` or `cancelled`), `BackupJobDuration` (the job's duration in microseconds, like every duration the collector records, with the same `operation`), `BackupBytesUploaded` (bytes put to the store), `BackupSnapshotAge` (seconds since the snapshot instant, set at each confirmed heartbeat) and `BackupStoreErrors` (`operation` = the store call that failed: the job's `put` and `delete_prefix`, and the reconciler's `get` and `list` of the manifest probe and the deleting duties and its `delete_prefix`, an absent object not counted), with `QuerySource::BackupJob`; `BackupBytesUploaded` is recorded for a job that ended AVAILABLE, and `WorkerErrorCount` under `BackupJob` for a FAILED job and not for one the shutdown cancelled (which records `BackupJobs` with `cancelled` and neither worker metric); the restore variants are PR 9a's. The existing `WorkerLastSuccess`, `WorkerCycleLatency` and `WorkerErrorCount` are emitted by the reconciler like the other workers and by the backup job at its end under `BackupJob`. +Metrics: `backup_jobs_total{outcome=succeeded|failed|cancelled}`, `backup_job_duration_seconds`, `backup_bytes_uploaded_total`, `restore_jobs_total{outcome}`, `restore_job_duration_seconds`, `restore_bytes_downloaded_total`, `backup_snapshot_age_seconds` (gauge while a job runs), `backup_store_errors_total{op}`. Logs: one info line at job start and completion with ARN, item count, bytes, duration; one error line per failure with the reason. The `/health` readiness work from the operations report should include store reachability as a component, not as a hard failure (a backup store outage should not take the data plane out of rotation). `/health` is a static 200 today; the component model is new work in the configuration PR, and a transport failure of the store, or a failure of its identity check or manifest probe (section 4), marks the component unhealthy rather than stopping the server (section 7). Metric names are variants of the closed `MetricName` enum in `crates/core/src/metrics`, and `/metrics` emits them under those names as JSON (`render_metrics`, `docs/design/06-component-server.md`: the endpoint is not Prometheus exposition format; the deployment guide tells operators to convert); the snake-case spellings above are the names a converter would give them, not a rendering the server does. PR 8 adds `BackupJobs` (one count per job end, `operation` = the outcome, `succeeded`, `failed` or `cancelled`), `BackupJobDuration` (the job's duration in microseconds, like every duration the collector records, with the same `operation`), `BackupBytesUploaded` (bytes put to the store), `BackupSnapshotAge` (seconds since the snapshot instant, set at each confirmed heartbeat) and `BackupStoreErrors` (`operation` = the store call that failed: the job's `put` and `delete_prefix`, from PR 9a a restore's `get`, and the reconciler's `get` and `list` of the manifest probe and the deleting duties and its `delete_prefix`, an absent object not counted), with `QuerySource::BackupJob`; `BackupBytesUploaded` is recorded for a job that ended AVAILABLE, and `WorkerErrorCount` under `BackupJob` for a FAILED job and not for one the shutdown cancelled (which records `BackupJobs` with `cancelled` and neither worker metric); the restore variants are PR 9a's: `RestoreJobs` (`operation` = the outcome), `RestoreJobDuration` and `RestoreBytesDownloaded`, with `QuerySource::RestoreJob` for the worker metrics, and `BackupStoreErrors{get}` on every failed `get` of a restore job: the request's and a resume's two document reads and each download attempt the store failed, retried (a transport error) or not (an absent file, a refusal, an I/O error); a digest mismatch and the spool's own errors are not the store's. `RestoreBytesDownloaded` counts the bytes of each data-file download attempt as they arrived, verified or not; the manifest and sidecar reads and a filesystem store's second read of each file are not counted. `RestoreJobs{failed}` counts the job's FAILED writes and the ends at which it learnt the record is another's; the reconciler's own FAILED writes (`target_not_empty`, `OTHER_STORE`, `OWNER_DEAD` under a maintenance claim) are counted in its cycle summary line and `CycleReport`, not in `RestoreJobs`. `WorkerErrorCount{RestoreJob}` accompanies each of those job ends (the maintenance refusal before the job ran, a FAILED write, a lost claim, a DONE write that matched no record); a job the shutdown stopped records `RestoreJobs{cancelled}` and no worker metric; `WorkerLastSuccess{RestoreJob}` is recorded on a restore that reached DONE. The existing `WorkerLastSuccess`, `WorkerCycleLatency` and `WorkerErrorCount` are emitted by the reconciler like the other workers and by the backup job at its end under `BackupJob`. ### 10. Documentation @@ -202,7 +202,9 @@ Which document and which row each PR changes; what each row says is in the ownin | differences doc: vector-backup row | PR 8 (CreateBackup) and PR 9a (the restore's refusal) | | differences doc: `ListBackups` paging row (table ARN in `TableName`) | PR 6 | | differences doc: multi-part-key row | PR 9a | -| differences doc: restore-interruption row | PR 8 | +| differences doc: restore-interruption row | PR 8 (the reconciler) and PR 9a (the store-backed resume per backend, SIGTERM, the kill-point outcomes) | +| differences doc: how `RestoreTableFromBackup` runs (the 60 s registration bound on every backend; the measured durations in that row are the base's in-catalog figures), the DeleteBackup and concurrent-restore rows (the `backup_index` lock per backend) | PR 9a | +| differences doc: new row for the MongoDB Decimal128 sort-key rule | PR 9a | | differences doc: Import and Export rows | PR 13 | | differences doc: new row for CreateBackup of a table holding an `N` value #618 let in | PR 8 | | differences doc: where-backups-live and backup-creation rows (the store-backed, asynchronous create) | PR 8 | @@ -212,7 +214,14 @@ Which document and which row each PR changes; what each row says is in the ownin | admin guide: the `[backup]` rows (`data_file_target_bytes`, `heartbeat_interval_secs`, `stale_job_after_secs`, `failed_row_retention_secs`, `reconcile_interval_secs`, `max_concurrent_jobs` with the memory figures) and the `[backup.store]` section (the store-backed create, the job's two log lines, the unhealthy-store refusal) | PR 8 | | deployment guide: the service unit's `MALLOC_ARENA_MAX=1` line | PR 8 | | `docs/design/README.md` row 15b (the measurement record) | PR 8 | +| `docs/design/15-backup-restore-bench-pr9.md` (the restore memory and duration record) and `docs/design/README.md` row 15c | PR 9a | | usage guide: "backups not supported" line | PR 8 | +| usage guide: the backups row's RestoreTableFromBackup ✓ (from the store into every backend) | PR 9a | +| upgrade manual: the restore of a store-backed backup lands, `kind = "disabled"` no longer needed for it, `spool_path` for an S3 store, migration 6 and `extenddb migrate`, the rollback paragraph's progress-row sentence | PR 9a | +| admin guide: the `spool_path` row (what the spool holds, its disk cost, the sweep), the `max_concurrent_jobs` row's restore memory terms, `backup list` lists no restore | PR 9a | +| limits doc: the concurrent-restores row (the registration bound, the spool term, Cassandra's `COUNT(*)`) and the backup-contents row (restores into any backend, vector where supported, multi-part keys) | PR 9a | +| `docs/schema/postgres.txt`: the `restore_progress` table | PR 9a | +| `.github/workflows/matrix-nightly.yml` and `devtools/backup-restore-matrix` (the nightly's script) | PR 9b | | upgrade manual: stop pre-PR-6 servers before `extenddb migrate`; a restore in flight at the stop; the rollback paragraph | PR 6 (and PR 8 for the store-backed backups a restored catalog no longer lists) | | upgrade manual: CreateBackup without `[backup.store]`, `kind = "disabled"`, the SQLite default root, shared store roots and `--adopt-store` | PR 7 (and PR 10 for `--adopt-store`) | | upgrade manual: the SQLite serve lock; `migrate-legacy` and its refusals | PR 10 | @@ -279,7 +288,7 @@ store_kind, store_root, store_id the store the restore reads, recorded at 5.3 owner_generation the same claim as a backup job (5.1 step 3); replaces #384's restore_operation_id (MongoDB) and restore_owner (Cassandra, where a non-key column cannot be renamed: V006 adds restore_owner_generation beside restore_owner, which PR 14 drops) heartbeat_at replaces #384's restore_heartbeat / restore_heartbeat_at spool_host the host whose spool holds the downloaded files (5.3 step 3) -file_index, items_applied per-batch progress on SQLite and Cassandra (5.3 step 5); unused on PostgreSQL (see `restore_progress`) and MongoDB (per-file record) +file_index, items_applied the restore's progress: per batch on SQLite and Cassandra, `file_index` per file on MongoDB (`items_applied` unused there); unused on PostgreSQL, whose progress is `restore_progress` rows in the data database (migration 6, 5.3 step 5) failure_reason, completed_at ``` PR 6's migration backfills the new fields on the registrations that exist at migration time, on all four backends: `kind = legacy`, `status = DONE` where the table is not CREATING, and where it is (a restore the upgrade's stop interrupted), `status = RUNNING` with `heartbeat_at` set to the epoch (1970-01-01T00:00:00Z), not NULL, on every backend, so every backend's staleness test selects them (the column is NOT NULL after the migration on PostgreSQL, so its test has no `IS NULL` arm), and a sentinel `owner_generation` (0) that no job holds; the first reconciler pass fails those rows as legacy restores (section 3) and removes their targets (`remove_restore_target`), which is what #384's startup sweeps did for such a target. @@ -372,53 +381,76 @@ Documentation rows it corrects: limits doc backup rows (the knobs); upgrade manu ### PR 8: CreateBackup job, worker and reconciler Goal: the CreateBackup job of 5.1 on the filesystem store, with the reconciler and its full duty list, the byte budget on the snapshot streams and the item validator. Its line names the interfaces and carries the acceptance: -- PR 8: CreateBackup job and worker per 5.1 on the filesystem store: snapshot in the request with the per-backend claim, generation-fenced catalog writes, pipelined read/compress/upload, timer heartbeat through the claim connection with abort on any unconfirmed heartbeat, the failure path's delete gated on a matched FAILED write, every job status write conditioned on `status = CREATING` and an ambiguous AVAILABLE update re-read, a table name over 255 bytes refused, cancellation runs the failure path, id minted from the snapshot instant, `LimitExceeded` over `max_concurrent_jobs` and while the maintenance claim is held, the byte budget `SNAPSHOT_FETCH_BYTES` on the PostgreSQL and SQLite snapshot streams, in the page query on PostgreSQL and while the rows are pulled on SQLite, PostgreSQL's cursor replaced by the three-layer keyset page under the same snapshot (section 3, 7), the item validator `validate_restorable_item` in `extenddb-core` (one walk, each `N` parsed once, an error naming the item's key and the attribute path), which PR 9a's restore binds, run on every item, with the backup failed after the rest of the snapshot is validated, every refused item logged up to the cap and the count and the first item in the FAILED reason (5.1 step 4, #618), #394's `KeyRangeTracker` and `BackupDocuments::build` in the writer, the server's shutdown awaiting each job's failure path, the PostgreSQL claim connection's keepalive settings, the data-file close rule and the knob's minimum, the SQLite runtime dead-owner test (a slot no task holds), `refuse_backup_of_vector_table` taken out of the off-host path and kept in front of `LegacyBackupReader::create`, `create_backup_refuses_a_table_with_vector_indexes` kept for that path, the create half of an off-host round trip of a vector table added (CreateBackup succeeds, DescribeBackup reaches AVAILABLE with the table's features, ListBackups lists it, and the restore answers the ValidationException naming the store until PR 9a adds the restore half) and `test_create_backup_of_a_vector_table_is_refused` expecting the outcome of the configured path, read from `EXTENDDB_TEST_BACKUP_STORE`, which every CI pytest job that runs the suite sets (`filesystem` on the PostgreSQL job, which appends a filesystem store after `init`, and on the SQLite job, the defaulted root; `s3` on the SQLite job against moto; `none` on the MongoDB and Cassandra jobs) and `devtools/run-tests --pytest` refuses to run without (5.1 step 1); the restore cells of `tests/test_backup_restore_fidelity.py` skip on a server with a store, naming PR 9a, until its restore job lands; FAILED tombstones, metrics. Acceptance: backup of a 100k-item composite-key table with GSIs on each backend with a `SnapshotEngine` (PostgreSQL, SQLite, MongoDB; Cassandra keeps the legacy create and its existing cells); SIGKILL mid-upload leaves no prefix with a manifest and the row resolves after restart (binary-level); SIGKILL between the manifest put and the AVAILABLE update resolves to AVAILABLE after restart and the prefix survives; SIGTERM mid-upload leaves a FAILED row, an empty prefix and no multipart upload; failure deletes the prefix; a second CreateBackup on the same instance while one runs returns `LimitExceededException`; a 1M-item run records duration and peak RSS, and the 1M run's peak must be within 112 MiB of the 100k run's (the budget, the line cap and the channels; on every backend but `sqlite-memory`, whose open copies the table; on MongoDB a 500k run stands in for the 1M run, its load alone taking 27 minutes through the wire); the memory test with #394's worst-expansion items, with the worst lines placed after a page of small rows, has its peak over idle under 1.2 times the sum of section 7's terms at the measured row (the decoded rows in the page and the pipeline, one row's parse, the line buffer, the chunks and SQLite's run-ahead) with `MALLOC_ARENA_MAX=1`, at 12, 24 and 48 worst rows on one SQLite binary and at 12 rows on PostgreSQL, and its peak with glibc's default arenas recorded beside it at the same counts, and no page holds more than the budget plus one row, counted in the stream by a check in `extenddb_storage::snapshot_conformance` that the PostgreSQL and SQLite tests both run; the rows a PostgreSQL page reads past its cut are at most `n - 1`, measured by `EXPLAIN (ANALYZE, BUFFERS)` on a 1,000-small-then-60-worst table (4 read, 1 past the cut at the floor) and pinned by `the_row_bound_remembers_large_rows_among_small_ones` (201 over 53 round trips on a 300-row table with nine 27 MB rows); `EXPLAIN` of the page query on the six key shapes a table can have (hash only, `S`, `N` and `B` range keys, a multi-part range key, a legacy table on the database collation) shows `Limit` over one `Index Scan` on the primary key and no other scan of the relation where the planner's estimate is above `n`, and the bitmap scan with a `Sort` near the end of a small table (PostgreSQL 16), both pinned by a test and stated in the module doc; one cell for each write path #394's `key_order.rs` doc names (BatchWriteItem, TransactWriteItems, UpdateItem on an absent item, ImportTable): a table holding `1E126` or `abc` written through that path (in a non-key attribute and, on a key-ordered table, in the range key; for UpdateItem in the key it creates) gives a FAILED row and no manifest on PostgreSQL and SQLite (the two backends `tests/test_backup_job.py`'s deployment fixture starts; MongoDB's store-backed create is covered by the engine tests and the 100k run, and gets its cell when the fixture gains a MongoDB deployment) (an `abc` range key is refused by the number column on PostgreSQL and SQLite, so the range-key cell carries `1E126`; UpdateItem validates the values its expression sets, so its cell carries the bad value in the key alone); with three such items the job validates past the first, logs all three with key, attribute path and reason, and the FAILED reason gives the count and the first and names #618; an item refused for another rule (over `max_item_size_bytes`) gives a reason that does not name #618; the vector wire test fails when `EXTENDDB_TEST_BACKUP_STORE` is unset; without `[backup.store]` CreateBackup of a vector table is refused as on main; a job task that panics on SQLite leaves a row the reconciler resolves without a restart. The reconciler ships in the same PR, with the full duty list of section 4, the S3 leg in CI, and memory tests with incompressible items (64 MiB of random binary attributes, which gzip cannot shrink) and with the worst-expansion shapes of section 7; RSS is recorded against the terms of section 7 in each, proving the encoder streams into the upload (the incompressible run's 68 MB file passes through a 4 MiB chunk window) and that the byte budget and the channel bound hold at the largest line (under `MALLOC_ARENA_MAX=1` the worst-expansion run's peak does not grow with the number of worst rows, 452, 480 and 455 MiB over idle at 12, 24 and 48 rows on the final SQLite binary; with glibc's default arenas it does, 707, 700 and 1,281 MiB in the same series, the allocator's retention and not the pipeline's terms, which the one-arena series shows do not grow). Depends on PR 4, 5, 6, 7. +- PR 8: CreateBackup job and worker per 5.1 on the filesystem store: snapshot in the request with the per-backend claim, generation-fenced catalog writes, pipelined read/compress/upload, timer heartbeat through the claim connection with abort on any unconfirmed heartbeat, the failure path's delete gated on a matched FAILED write, every job status write conditioned on `status = CREATING` and an ambiguous AVAILABLE update re-read, a table name over 255 bytes refused, cancellation runs the failure path, id minted from the snapshot instant, `LimitExceeded` over `max_concurrent_jobs` and while the maintenance claim is held, the byte budget `SNAPSHOT_FETCH_BYTES` on the PostgreSQL and SQLite snapshot streams, in the page query on PostgreSQL and while the rows are pulled on SQLite, PostgreSQL's cursor replaced by the three-layer keyset page under the same snapshot (section 3, 7), the item validator `validate_restorable_item` in `extenddb-core` (one walk, each `N` parsed once, an error naming the item's key and the attribute path), which PR 9a's restore binds, run on every item, with the backup failed after the rest of the snapshot is validated, every refused item logged up to the cap and the count and the first item in the FAILED reason (5.1 step 4, #618), #394's `KeyRangeTracker` and `BackupDocuments::build` in the writer, the server's shutdown awaiting each job's failure path, the PostgreSQL claim connection's keepalive settings, the data-file close rule and the knob's minimum, the SQLite runtime dead-owner test (a slot no task holds), `refuse_backup_of_vector_table` taken out of the off-host path and kept in front of `LegacyBackupReader::create`, `create_backup_refuses_a_table_with_vector_indexes` kept for that path, the create half of an off-host round trip of a vector table added (CreateBackup succeeds, DescribeBackup reaches AVAILABLE with the table's features, ListBackups lists it, and the restore answers the ValidationException naming the store until PR 9a adds the restore half) and `test_vector_table_backup_refused_in_catalog_and_restored_from_store` (named `test_create_backup_of_a_vector_table_is_refused` until PR 9 added the restore half) expecting the outcome of the configured path, read from `EXTENDDB_TEST_BACKUP_STORE`, which every CI pytest job that runs the suite sets (`filesystem` on the PostgreSQL job, which appends a filesystem store after `init`, and on the SQLite job, the defaulted root; `s3` on the SQLite job against moto; `filesystem` on the MongoDB jobs, whose `devtools/run-mongodb-tests` serves a filesystem root under its output directory from PR 9 on, where it was `none` before; `none` on the Cassandra job) and `devtools/run-tests --pytest` refuses to run without (5.1 step 1); the restore cells of `tests/test_backup_restore_fidelity.py` skip on a server with a store, naming PR 9a, until its restore job lands; FAILED tombstones, metrics. Acceptance: backup of a 100k-item composite-key table with GSIs on each backend with a `SnapshotEngine` (PostgreSQL, SQLite, MongoDB; Cassandra keeps the legacy create and its existing cells); SIGKILL mid-upload leaves no prefix with a manifest and the row resolves after restart (binary-level); SIGKILL between the manifest put and the AVAILABLE update resolves to AVAILABLE after restart and the prefix survives; SIGTERM mid-upload leaves a FAILED row, an empty prefix and no multipart upload; failure deletes the prefix; a second CreateBackup on the same instance while one runs returns `LimitExceededException`; a 1M-item run records duration and peak RSS, and the 1M run's peak must be within 112 MiB of the 100k run's (the budget, the line cap and the channels; on every backend but `sqlite-memory`, whose open copies the table; on MongoDB a 500k run stands in for the 1M run, its load alone taking 27 minutes through the wire); the memory test with #394's worst-expansion items, with the worst lines placed after a page of small rows, has its peak over idle under 1.2 times the sum of section 7's terms at the measured row (the decoded rows in the page and the pipeline, one row's parse, the line buffer, the chunks and SQLite's run-ahead) with `MALLOC_ARENA_MAX=1`, at 12, 24 and 48 worst rows on one SQLite binary and at 12 rows on PostgreSQL, and its peak with glibc's default arenas recorded beside it at the same counts, and no page holds more than the budget plus one row, counted in the stream by a check in `extenddb_storage::snapshot_conformance` that the PostgreSQL and SQLite tests both run; the rows a PostgreSQL page reads past its cut are at most `n - 1`, measured by `EXPLAIN (ANALYZE, BUFFERS)` on a 1,000-small-then-60-worst table (4 read, 1 past the cut at the floor) and pinned by `the_row_bound_remembers_large_rows_among_small_ones` (201 over 53 round trips on a 300-row table with nine 27 MB rows); `EXPLAIN` of the page query on the six key shapes a table can have (hash only, `S`, `N` and `B` range keys, a multi-part range key, a legacy table on the database collation) shows `Limit` over one `Index Scan` on the primary key and no other scan of the relation where the planner's estimate is above `n`, and the bitmap scan with a `Sort` near the end of a small table (PostgreSQL 16), both pinned by a test and stated in the module doc; one cell for each write path #394's `key_order.rs` doc names (BatchWriteItem, TransactWriteItems, UpdateItem on an absent item, ImportTable): a table holding `1E126` or `abc` written through that path (in a non-key attribute and, on a key-ordered table, in the range key; for UpdateItem in the key it creates) gives a FAILED row and no manifest on PostgreSQL and SQLite (the two backends `tests/test_backup_job.py`'s deployment fixture starts; MongoDB's store-backed create is covered by the engine tests and the 100k run, and gets its cell when the fixture gains a MongoDB deployment) (an `abc` range key is refused by the number column on PostgreSQL and SQLite, so the range-key cell carries `1E126`; UpdateItem validates the values its expression sets, so its cell carries the bad value in the key alone); with three such items the job validates past the first, logs all three with key, attribute path and reason, and the FAILED reason gives the count and the first and names #618; an item refused for another rule (over `max_item_size_bytes`) gives a reason that does not name #618; the vector wire test fails when `EXTENDDB_TEST_BACKUP_STORE` is unset; without `[backup.store]` CreateBackup of a vector table is refused as on main; a job task that panics on SQLite leaves a row the reconciler resolves without a restart. The reconciler ships in the same PR, with the full duty list of section 4, the S3 leg in CI, and memory tests with incompressible items (64 MiB of random binary attributes, which gzip cannot shrink) and with the worst-expansion shapes of section 7; RSS is recorded against the terms of section 7 in each, proving the encoder streams into the upload (the incompressible run's 68 MB file passes through a 4 MiB chunk window) and that the byte budget and the channel bound hold at the largest line (under `MALLOC_ARENA_MAX=1` the worst-expansion run's peak does not grow with the number of worst rows, 452, 480 and 455 MiB over idle at 12, 24 and 48 rows on the final SQLite binary; with glibc's default arenas it does, 707, 700 and 1,281 MiB in the same series, the allocator's retention and not the pipeline's terms, which the one-arena series shows do not grow). Depends on PR 4, 5, 6, 7. Invariants honoured: 4(a) (every transition of the backup row a job or the reconciler performs), 4(c) (the reconciler acts only on rows of the configured store and on none while it is unhealthy), 4(e) (the claim, the fencing of every status write, the FAILED-write gate before any delete), 4(f) (the duties that run under a maintenance claim), 5.1 as written, section 2's three store facts, section 7's memory bound and pool rule, and the Drawbacks' #618 regression. Interfaces it adds: `extenddb_core::validation::validate_restorable_item(item, &RestorableKeySchema, &RestorableLimits)` (5.1 step 4; the key argument is a schema every table has, multi-part keys included, each key attribute checked with the format's own `check_key_value`; `RestorableLimits` is built from `&LimitsConfig`: the item size, the hash and range key sizes and the attribute name size), `RestorableItemReason::{AttributeNameTooLong, LineTooLong}` (the latter carrying `{ cap }`), `SHOWN_VALUE_BYTES` (4096, the bound on each text an error carries other than a key value) and `SHOWN_KEY_VALUE_BYTES` (12,288, the bound on a key value), `extenddb_core::limits::DEFAULT_MAX_ITEM_SIZE_BYTES` (the default item limit, the feeder's threshold), `extenddb_core::backup_format::encode_item_line_len_bounded(item, cap)` (the line's length up to `cap`, measured without a buffer), `extenddb_storage::snapshot::paging::SNAPSHOT_FETCH_BYTES` with `fetch_bytes()`, `record_page_bytes` and `record_rows_past_cut` (counters under `test-hooks`), `extenddb_storage::snapshot::source_table_schema` (5.1 step 2), the job (`crates/engine/src/backup/job.rs` with `job/{writer,heartbeat,failure}.rs`) and the reconciler's new duties (`crates/engine/src/backup/reconciler.rs`) in the engine's backup module, the server handing the engine the shutdown token, the metrics collector, the job settings (`JobSettings`) and the manifest's bucket and key prefix through `BackupRuntime` and `ConfiguredStore`, the metrics of section 9, and on `BackupCatalog`: `claim_dead_backup(key, stale_after)` (the takeover of 4(e); `claim_backup` inserts and takes over nothing), `list_backup_rows_older_than(status, store, clock, age)` (`StaleClock::{Heartbeat, Completed}`, the backend's clock; a row with `completed_at` unset is never older on the completed clock), `sweep_restore_registrations` (Cassandra's `active_restores` sweep, a default empty body elsewhere) and `StatusFields.source_table_details_json` (the AVAILABLE write carries the manifest's `source_table`). The request's detached task opens the snapshot, claims the row, checks the maintenance claim and returns the response through a one-shot channel, then continues as the job with the same owned values, so nothing but the response crosses a channel between the request and the job (the pipeline's item and chunk channels and its blocking encoder are the job's own); the runtime's task tracker holds every job, and the server's shutdown cancels and awaits the jobs' failure paths (`BackupRuntime::drain_jobs`, a 30 s bound) before it drains its workers. It is needed because four write paths store an `N` value such as `1E126` or `abc` unvalidated on main (#618): BatchWriteItem, TransactWriteItems, UpdateItem on an absent item (which creates the item from its `Key`) and ImportTable, the list #394's `key_order.rs` doc gives. So that one run finds every offending item, the first refusal stops encoding and uploading but not validation: the job reads and checks the rest of the snapshot, logs each refused item (its key, attribute path and reason, with the backup ARN) up to 1000 items and counts the rest, and then fails the backup (step 5) with a FAILED reason that gives the number of refused items and the first one, naming #618 only when that item's refusal is an `N` value; so no backup that no restore can load is reported AVAILABLE. The byte budget (section 7 states the bound): PR 8 replaces #395's cursor on PostgreSQL with pages bounded by bytes in the query, each page a keyset `SELECT` in the same `REPEATABLE READ` transaction, ordered by the data table's primary key and starting after the last key of the page before, in three layers (the innermost takes `n` rows after the key under `LIMIT`, since a window function runs before `LIMIT`; the middle adds the running byte sum and the rows' count; the outer keeps the rows whose predecessors' bytes are under `SNAPSHOT_FETCH_BYTES` (64 MiB, a constant beside `DEFAULT_FETCH_BATCH`, not a configuration key in v1) and orders them), so a page holds at least one row and at most the budget plus one, and the stream ends when the inner query returned fewer rows than asked and the cut dropped none of them; `n` starts at 16 and is then the budget divided by the largest row of the last pages holding at least a row bound's worth of rows (`RowBoundMemory`), never under 4 and never over `DEFAULT_FETCH_BATCH`; the rows read past the cut (at most `n - 1`) are read by the server and dropped by the outer layer, and each costs the server one serialization of its item text and two spills of it (the window functions buffer the `n` rows before the outer layer drops them), so a page costs the server `n` rows of text, not the rows it returns: sized from the average, a page of 984 small rows and 3 large ones asked the next page for 810 rows of a table that was then large rows only, 5.9 GB of server temp files and 18 to 44 s for the three rows it returned; sized from the largest row it asks for 4 (208 MiB, 1.1 s); the over-read that remains is the second page, and then the page that follows a row bound's worth of small rows, up to `n` rows of up to the line cap, at most once per row bound's worth of small rows; the key columns come back under qualified aliases so the next cursor is the row's own typed key, and the outer `ORDER BY` states a guarantee the executor gives anyway (a mutant without it passes every behavioural test, and the query text pins it); the snapshot, the lock and which tables stream `key_ordered` are unchanged. On SQLite a page stops at `DEFAULT_FETCH_BATCH` rows or after the row that reaches `SNAPSHOT_FETCH_BYTES` of item text, whichever comes first, and holds at least one row; the rows are pulled one at a time, so the stream pulls none past the cut, while the driver's worker thread reads at most five rows ahead of the pull (sqlx's `row_buffer_size`, set on the pool to four from its default of 50, which read every row of a 24-row worst-shape table ahead of the first page, plus the row the worker holds while its `send` waits; 160 MiB at the line cap), and the stream ends only when a page is short of the row bound and no cut happened. The cut rule exists in SQL on PostgreSQL and in Rust on SQLite, so the check that no page holds more than the budget plus one row lives in `extenddb_storage::snapshot_conformance`, which both run; a MongoDB batch is already bounded by the server's 16 MiB reply. -Tests that change (5.1 step 1): `create_backup_refuses_a_table_with_vector_indexes` (`crates/engine/src/backup/tests.rs`) stays for that path, PR 8 adds the create half of an off-host round trip of a vector table beside it (PR 9a adds the restore half), and `test_create_backup_of_a_vector_table_is_refused` (`tests/test_backup_restore_fidelity.py`) expects the refusal or the round trip according to the server's configured path, which it reads from `EXTENDDB_TEST_BACKUP_STORE=`, a variable the test harness exports and every CI job that runs a suite creating backups sets it (the Rust integration suite reads it the same way, `tests/rust/src/backup_restore.rs`): it derives its branch from the variable, prints which branch ran, and fails if the variable is unset, rather than accepting either outcome; the restore's `ensure_restorable` refusal of every vector backup narrows to "the target has no vector support" only in PR 9a, which rewrites `vector_index_backups_are_refused_in_the_engine` to that case; the frozen legacy reader keeps #384's behaviour for a legacy vector backup (section 3), and section 10 corrects the differences doc's vector row. +Tests that change (5.1 step 1): `create_backup_refuses_a_table_with_vector_indexes` (`crates/engine/src/backup/tests.rs`) stays for that path, PR 8 adds the create half of an off-host round trip of a vector table beside it (PR 9a adds the restore half), and `test_vector_table_backup_refused_in_catalog_and_restored_from_store` (`tests/test_backup_restore_fidelity.py`; `test_create_backup_of_a_vector_table_is_refused` until PR 9) expects the refusal or the round trip according to the server's configured path, which it reads from `EXTENDDB_TEST_BACKUP_STORE=`, a variable the test harness exports and every CI job that runs a suite creating backups sets it (the Rust integration suite reads it the same way, `tests/rust/src/backup_restore.rs`): it derives its branch from the variable, prints which branch ran, and fails if the variable is unset, rather than accepting either outcome; the restore's `ensure_restorable` refusal of every vector backup narrows to "the target has no vector support" only in PR 9a, which rewrites `vector_index_backups_are_refused_in_the_engine` to that case; the frozen legacy reader keeps #384's behaviour for a legacy vector backup (section 3), and section 10 corrects the differences doc's vector row. -State transitions it owns: CREATING to AVAILABLE and to FAILED by the job (5.1 steps 4, 5) and by the reconciler (5.1 step 6); every deleting duty of 4(a). The reconciler, in one place: The reconciler runs at startup and then every `reconcile_interval_secs` (default 60, section 7); it acts on no row of the store while the configured store is unhealthy (the legacy duty touches no store and still runs), and it checks the maintenance claim first, and while one is held it runs only the duties that resolve the states a maintenance command waits on: it finishes DELETING rows of the configured store, resolves CREATING backup rows whose owner is dead and fails RUNNING `restore_jobs` rows whose owner is dead rather than resuming them (a resume is a job, which the claim excludes), removing their targets, and runs the FAILED-target duty (such a target keeps its backup in use, so `--forget-backup` refuses until the reconciler has removed it); it skips every other duty, so it never deletes the prefix of a FAILED or DELETED row a `--move-store` is rewriting. Its duties: resolve a CREATING backup row whose owner is dead to AVAILABLE or FAILED; delete the prefix of a FAILED row once its `heartbeat_at` is older than `stale_job_after_secs`, retrying while the prefix lists non-empty; finish a DELETING row (delete the manifest, then the prefix, then mark DELETED once the prefix lists empty); delete the prefix of a DELETED row that still lists objects; remove a FAILED or DELETED backup row once its prefix is empty and its `completed_at` is older than `failed_row_retention_secs` (default 7 days); claim and resume a RUNNING `restore_jobs` row whose owner is dead (the resume is PR 9a's, with the restore job; this release's reconciler logs such a record once; once its `heartbeat_at` is older than `stale_job_after_secs` on every backend, 5.3 step 7, and a row with no `restore_progress` row is resumed only if the target's data table is empty, otherwise marked FAILED with reason `target_not_empty`, its target left in place and reported for the operator, since a catalog restored from a dump can name a restore that finished and was written to since; the operator inspects that target at the storage level (a CREATING table accepts no reads) and removes it with `extenddb backup remove-target
` (PR 10), the one command that removes a FAILED restore's CREATING target and its record, through `remove_restore_target`, under the maintenance claim like the other writing commands, since DeleteTable refuses a restore target (5.3 step 6); a legacy restore (`kind = legacy`) is marked FAILED and its target removed the same way instead, section 3; on MongoDB a row with `restore_backfill_pending` is DONE, not RUNNING, and is never resumed while the backfill worker finishes it, 5.3 step 6), skipping it to the next cycle when no job permit is free; finish a FAILED `restore_jobs` row whose target table is still CREATING, the FAILED-target duty (4(e); landed in PR 6, whose `list_dead_restore_jobs` returns such a row once it is stale; `BackupCatalog::remove_restore_target`, section 3, not DeleteTable, which refuses a restore target; never for reason `target_not_empty`, whose target is the operator's, above); remove `restore_progress` rows whose table is not a CREATING restore target and spool entries under this instance's `spool_path` whose job is not RUNNING with this instance's `spool_host` (5.3 step 3), both from PR 9a, which adds the records they read (the duty that removed a CREATING restore target with no job row is withdrawn: on every backend the job record is the registration itself, so no such state exists after migration 5); on Cassandra, remove an `active_restores` entry of a backup row whose target is not a CREATING restore target of that backup; remove a legacy CREATING backup row whose create is dead, with its payload (section 3). The reconciler's writes to a backup row carry what the job's would (`size_bytes`, `item_count`, `source_table_details_json` from the manifest for AVAILABLE; `completed_at` for both outcomes, the retention clock). On PostgreSQL its claim is an advisory lock on a dedicated connection that it closes when the row is resolved. `extenddb backup list --failed` prints FAILED rows with their `failure_reason`. +State transitions it owns: CREATING to AVAILABLE and to FAILED by the job (5.1 steps 4, 5) and by the reconciler (5.1 step 6); every deleting duty of 4(a). The reconciler, in one place: The reconciler runs at startup and then every `reconcile_interval_secs` (default 60, section 7); it acts on no row of the store while the configured store is unhealthy (the legacy duty touches no store and still runs), and it checks the maintenance claim first, and while one is held it runs only the duties that resolve the states a maintenance command waits on: it finishes DELETING rows of the configured store, resolves CREATING backup rows whose owner is dead and fails RUNNING `restore_jobs` rows whose owner is dead rather than resuming them (a resume is a job, which the claim excludes), removing their targets, and runs the FAILED-target duty (such a target keeps its backup in use, so `--forget-backup` refuses until the reconciler has removed it); it skips every other duty, so it never deletes the prefix of a FAILED or DELETED row a `--move-store` is rewriting. Its duties: resolve a CREATING backup row whose owner is dead to AVAILABLE or FAILED; delete the prefix of a FAILED row once its `heartbeat_at` is older than `stale_job_after_secs`, retrying while the prefix lists non-empty; finish a DELETING row (delete the manifest, then the prefix, then mark DELETED once the prefix lists empty); delete the prefix of a DELETED row that still lists objects; remove a FAILED or DELETED backup row once its prefix is empty and its `completed_at` is older than `failed_row_retention_secs` (default 7 days); claim and resume a RUNNING `restore_jobs` row whose owner is dead (the resume is PR 9a's, with the restore job, through `claim_restore_dead`, 4(e); once its `heartbeat_at` is older than `stale_job_after_secs` on every backend, 5.3 step 7, and a row with no `restore_progress` row is resumed only if the target's data table is empty, otherwise marked FAILED with reason `target_not_empty`, its target left in place and reported for the operator, since a catalog restored from a dump can name a restore that finished and was written to since; the operator inspects that target at the storage level (a CREATING table accepts no reads) and removes it with `extenddb backup remove-target --account
` (PR 10's command, shipped with PR 9: on MongoDB and Cassandra a crash before the first progress record, the whole first data file on MongoDB and the first 10,000 items or 32 MiB on Cassandra, leaves written items and no progress, so the ordinary crash there ends in this state and the release that ships the restore ships its remedy, item 30; until it runs, DeleteTable refuses the target, 5.3 step 6, and DeleteBackup and a second restore of the backup are `BackupInUseException`, 5.2). `extenddb backup remove-target --account
` removes a restore target DeleteTable refuses (a CREATING table) and its restore record, under the maintenance claim like the other writing commands: a FAILED record's target whatever the reason (`target_not_empty` included), a RUNNING record's target whose owner is dead (the backend's `list_dead_restore_jobs` names it under `stale_job_after_secs` on the database's clock; on SQLite under the serve lock the window is zero and every RUNNING record is listed), and a legacy record's target the same way; it refuses a RUNNING record whose owner is live ("stop it or wait for it"), a DONE record and a table no record names (an ACTIVE table is one or the other), the latter two as "not a restore target". It needs no store, prints the claim it took and one line naming what it removed and the record's state, exits 1 with the refusal otherwise, and logs the removal under `extenddb::audit` with the account, the table and the backup's ARN. The removal is `remove_restore_target`, section 3; a legacy restore (`kind = legacy`) is marked FAILED and its target removed the same way instead, section 3; on MongoDB a row with `restore_backfill_pending` is DONE, not RUNNING, and is never resumed while the backfill worker finishes it, 5.3 step 6), skipping it to the next cycle when no job permit is free; finish a FAILED `restore_jobs` row whose target table is still CREATING, the FAILED-target duty (4(e); landed in PR 6, whose `list_dead_restore_jobs` returns such a row once it is stale; `BackupCatalog::remove_restore_target`, section 3, not DeleteTable, which refuses a restore target; never for reason `target_not_empty`, whose target is the operator's, above); remove `restore_progress` rows whose table is not a CREATING restore target and spool entries under this instance's `spool_path` whose job is not RUNNING with this instance's `spool_host` (5.3 step 3), both from PR 9a, which adds the records they read (the duty that removed a CREATING restore target with no job row is withdrawn: on every backend the job record is the registration itself, so no such state exists after migration 5); on Cassandra, remove an `active_restores` entry of a backup row whose target is not a CREATING restore target of that backup; remove a legacy CREATING backup row whose create is dead, with its payload (section 3). The reconciler's writes to a backup row carry what the job's would (`size_bytes`, `item_count`, `source_table_details_json` from the manifest for AVAILABLE; `completed_at` for both outcomes, the retention clock). On PostgreSQL its claim is an advisory lock on a dedicated connection that it closes when the row is resolved. `extenddb backup list --failed` prints FAILED rows with their `failure_reason`. Kill points (testing summary): between a data file upload and the manifest; between the three documents and the manifest (`before_manifest`); between the manifest put and the AVAILABLE update; SIGTERM during a backup. Main tests that must pass unchanged: #384's backup tests other than the vector tests named above. Documentation rows it corrects: the differences doc's backup consistency row (its MongoDB half, "not point-in-time" today, which the snapshot read concern of section 3 makes one; its Cassandra half stays true), its vector-backup row ("Backing up and restoring a table that has vector indexes": supported on the off-host path from PR 8 and 9a, refused only into a target without vector support, CreateBackup on the legacy create, without `[backup.store]` or on Cassandra, still refused as on main, the legacy reader's behaviour kept for legacy backups; 5.1 step 1), its restore-interruption row (one reconciler with the duties above in place of the per-backend sweeps), a new row for CreateBackup of a table holding an `N` value #618 let in (the backup fails on the off-host path, where main backs it up and restores it; 5.1 step 4), the limits doc's line cap and file size rows, the usage guide's "backups not supported" line; upgrade manual: on the off-host path a backup of a table holding an `N` value the wire refuses (#618) fails where main backs it up, which meets file-backed SQLite without a `[backup]` section and any deployment once a store is configured (5.1 step 4), the reason reaches only the operator (`backup list --failed`, and the log, which names each offending item's key and attribute path with the backup ARN a tenant's support case can cite), and the way out is to rewrite each named item. ### PR 9a: restore job Goal: the restore job of 5.3 on the same backend over a filesystem store. Its line names the interfaces and carries the acceptance: -- PR 9a: restore job per 5.3, same backend, filesystem store (PR 8 leaves it the engine's `check_catalog_row_for_restore`, which answers `ValidationException` "Backup is in a backup store; this release restores in-catalog backups only" for a row of the configured store, the 24 restore cells of `tests/test_backup_restore_fidelity.py` and `tests/test_auth_permissions.py`'s restore-authorization cell skipped on a server with a store, and the vector wire test's restore assertion, all four to replace with the restore): request creates table and `restore_jobs` row atomically in a detached task and returns CREATING (`TableAlreadyExistsException` or `TableInUseException` on a name race, as #384 returns them; `LimitExceededException` over the job limit); Cassandra claim and heartbeat by lightweight transaction; job verifies every file before loading any, with `.verified` spool markers; `bulk_load(target_table_id, claim, file_index, start_at, items)` per backend with the progress record as 5.3 step 5 specifies for each (PostgreSQL data migration 005 for `restore_progress`, reserved here) and the `BackupCatalog` restore methods of section 3; the MongoDB terminal sequence of 5.3 step 6 (DONE and `restore_backfill_pending` in one transaction, the worker's flip conditioned on the flag); `spool_host` as host and spool path; the item validator PR 8 adds, bound with the target's `RestorableLimits` (item, hash key, range key and attribute name sizes; `N` normalization included); items read through `ExtenddbManifest::reader_prehashed`; the restore's vector refusal narrowed to a target without vector support, with `vector_index_backups_are_refused_in_the_engine` rewritten to that case; duplicate rejection across and within files (#394's per-file key-range check on key-ordered backups, the count after the last file on unordered ones, the MongoDB post-resume count check); the manifest and sidecar spooled for a resume; a PostgreSQL resume or target deletion only after the stale window; conditional ACTIVE flip before the progress rows are deleted, the job's target removal (`remove_restore_target`, not DeleteTable) gated on a matched FAILED write, DeleteTable on a restore target refused as #384 refuses it, the in-use rule of 5.2 checked in step 1 before the permit and again in step 4's registration together with `status = AVAILABLE` under the backup row's lock (MongoDB's `$inc`, Cassandra's `active_restores` lightweight transaction), multi-part-key tables restored; `bulk_load` writing index rows in the batch's transaction, with the base rows, on PostgreSQL and SQLite and the ACTIVE flip setting every index ACTIVE there; download transport errors retried for 15 minutes; a restore stopped by shutdown left RUNNING; the store and `kind` recorded on the job row and a takeover writing its `spool_host`; a PostgreSQL resume with no progress row only into an empty target. Acceptance: round trip preserves items, keys, GSIs, LSIs, throughput, SSE, vector where supported, and restores the table `STANDARD` whatever its class (#384's `table_class_is_not_restored` passes unchanged); tampering file 2's sha256 leaves a table that was never writable and is gone after the job fails, with no item ever written; SIGKILL after file 1's commit then restart writes exactly the backup's item count on every backend (MongoDB via duplicate-key suppression); SIGKILL at each kill point of the testing summary leaves a table that either finishes with the right count or is gone (on MongoDB, killed between the DONE write and the worker's flip, the restarted worker finishes it and the reconciler leaves it alone); an unordered backup carrying one key in two files fails into every target, Cassandra included, and a key-ordered backup whose file leaves its `key_range` fails; a second restore of a backup while a target restored from it is CREATING, and a DeleteBackup of that backup, return `BackupInUseException` on every backend, on MongoDB also between the job's DONE write and the worker's ACTIVE flip, and #384's `test_restore_into_a_creating_target_and_a_second_restore_of_its_backup_are_refused` passes unchanged against the new path; the two orderings over an off-host backup (a `backup_index` row), gated on the `backup_index` row lock, run here for the first time, deferred from PR 6 (which has no restore of such a row): a DeleteBackup that wins leaves the registration BackupNotFound and no target, a registration that wins leaves DeleteBackup BackupInUse; on MongoDB two concurrent registrations of one backup leave one target and return `BackupInUseException` to the other; a restore of an ARN of account B that carries the id of account A's backup returns BackupNotFound; on SQLite and PostgreSQL a Query on each GSI and SearchVectors return the restored items as soon as the target reports ACTIVE; a SIGTERM during a restore leaves its row RUNNING and the restore finishes after the restart; a store outage shorter than the retry bound mid-download leaves the restore RUNNING, and it completes once the store answers. Depends on PR 8. +- PR 9a: restore job per 5.3, same backend, filesystem store (shipped in PR 9; the engine's `check_catalog_row_for_restore` no longer refuses a `backup_index` row with "Backup is in a backup store; this release restores in-catalog backups only", the 24 restore cells of `tests/test_backup_restore_fidelity.py` and `tests/test_auth_permissions.py`'s restore-authorization cell run on a server with a store, and the vector wire test's restore asserts the round trip into a vector-capable target and the refusal into one without): request creates table and `restore_jobs` row atomically in a detached task and returns CREATING (`TableAlreadyExistsException` or `TableInUseException` on a name race, as #384 returns them; `LimitExceededException` over the job limit); Cassandra claim and heartbeat by lightweight transaction; job verifies every file before loading any, with `.verified` spool markers; `bulk_load(target, claim, BulkLoadFile { file_index, start_at, reapplying }, items)` per backend with the progress record as 5.3 step 5 specifies for each (PostgreSQL migration 6 for `restore_progress`) and the `BackupCatalog` restore methods of section 3; the MongoDB terminal sequence of 5.3 step 6 (DONE and `restore_backfill_pending` in one transaction, the worker's flip conditioned on the flag); `spool_host` as host and spool path; the item validator PR 8 adds, bound with the target's `RestorableLimits` (item, hash key, range key and attribute name sizes; `N` normalization included); items read through `ExtenddbManifest::reader_prehashed` from the spool and `ExtenddbManifest::reader` in place; the restore's vector refusal narrowed to a target without vector support, with `vector_index_backups_are_refused_in_the_engine` rewritten to that case; duplicate rejection across and within files (#394's per-file key-range check on key-ordered backups, the count after the last file on unordered ones, the MongoDB post-resume count check); the manifest and sidecar spooled for a resume; a PostgreSQL resume or target deletion only after the stale window; conditional ACTIVE flip before the progress rows are deleted, the job's target removal (`remove_restore_target`, not DeleteTable) gated on a matched FAILED write (`remove_restore_target` on PostgreSQL drops the relations `DeleteTable` drops (`table_storage_relations`): the data table, every secondary index's table and every vector index's `_ddb_vec_` table, their ids read in the claiming transaction), DeleteTable on a restore target refused as #384 refuses it, the in-use rule of 5.2 checked in step 1 before the permit and again in step 4's registration together with `status = AVAILABLE` under the backup row's lock (MongoDB's `$inc`, Cassandra's `active_restores` lightweight transaction), multi-part-key tables restored; `bulk_load` writing index rows in the batch's transaction, with the base rows, on PostgreSQL and SQLite and the ACTIVE flip setting every index ACTIVE there; download transport errors retried for 15 minutes; a restore stopped by shutdown left RUNNING; the store and `kind` recorded on the job row and a takeover writing its `spool_host`; a PostgreSQL resume with no progress row only into an empty target. Acceptance: round trip preserves items, keys, GSIs, LSIs, throughput, SSE, vector where supported, and restores the table `STANDARD` whatever its class (#384's `table_class_is_not_restored` passes unchanged); tampering file 2's sha256 leaves a table that was never writable and is gone after the job fails, with no item ever written; SIGKILL after file 1's commit then restart writes exactly the backup's item count on every backend (MongoDB's duplicate-key suppression is proved by `restore_mongodb_after_insert_many` at its second reach, since a kill after a file's commit resumes MongoDB at the next file with nothing of it written; the Cassandra points run with a SQLite source, since a Cassandra server's own backup is in-catalog, item 30); SIGKILL at each kill point of the testing summary leaves a table that finishes with the right count (the base table's and every GSI's), is gone, or, at the first reach of the two points that pause before a first progress record (`restore_mongodb_after_insert_many`, `restore_cassandra_before_progress`), is FAILED `target_not_empty` for `extenddb backup remove-target`, which removes it (on MongoDB, killed between the DONE write and the worker's flip, the restarted worker finishes it and the reconciler leaves it alone); an unordered backup carrying one key in two files fails into every target, Cassandra included, and a key-ordered backup whose file leaves its `key_range` fails; a second restore of a backup while a target restored from it is CREATING, and a DeleteBackup of that backup, return `BackupInUseException` on every backend, on MongoDB also between the job's DONE write and the worker's ACTIVE flip, and #384's `test_restore_into_a_creating_target_and_a_second_restore_of_its_backup_are_refused` passes unchanged against the new path; the two orderings over an off-host backup (a `backup_index` row), gated on the `backup_index` row lock, run here for the first time, deferred from PR 6 (which has no restore of such a row): a DeleteBackup that wins leaves the registration BackupNotFound and no target, a registration that wins leaves DeleteBackup BackupInUse; on MongoDB and on Cassandra two concurrent registrations of one backup leave one target and return `BackupInUseException` to the other (on Cassandra the registration's lightweight transaction, `IF backup_status = 'AVAILABLE' AND active_restores = null`, is linearized on the backup's partition, so the second finds `active_restores` non-null; the in-use pre-read is advisory); a restore of an ARN of account B that carries the id of account A's backup returns BackupNotFound; on SQLite and PostgreSQL a Query on each GSI and SearchVectors return the restored items as soon as the target reports ACTIVE; a SIGTERM during a restore leaves its row RUNNING and the restore finishes after the restart; a store outage shorter than the retry bound mid-download leaves the restore RUNNING, and it completes once the store answers; a 1M-item restore on PostgreSQL and SQLite records its duration and peak memory (the record above: 9.5 s and 26.8 s, 16 and 89 MiB over idle), the MongoDB and Cassandra figures being PR 11's. Depends on PR 8. Invariants honoured: 4(b) (the restore record and its transitions), 4(d) (every target by `RestoreTargetKey`), 4(e) (the claim, the FAILED-write gate before the target is removed, the PostgreSQL stale window), 4(f) (the job checks the maintenance claim after its row exists), 5.2 (the registration under the backup row's lock), 5.3 as written, section 1's duplicate rule and vector refusal, and the Goals' fidelity list. -Interfaces it adds: `bulk_load` (section 3) and the restore methods of `BackupCatalog`: +Interfaces it adds: `RestoreEngine` (section 3) and the restore methods of `BackupCatalog`: ```rust - // Restore job record (section 4). PR 9a. One target is addressed by `RestoreTargetKey` (`account_id`, - // `table_name`, `table_id`): MongoDB's `tables` `_id` and Cassandra's `tables` primary key are - // (`account_id`, `table_name`), and the table id guards against a table re-created under the name. - fn claim_restore(&self, row: &RestoreJobRow) -> BoxFuture, StorageError>>; // same contract as claim_backup - // at creation it is 5.3 step 4's registration: the target, the job row, and the re-check under the backup row's - // lock that the row is AVAILABLE and not in use (BackupNotFound or BackupInUse otherwise, no target left); - // a takeover writes the claimant's `spool_host` in the same conditional write that changes the generation - fn read_restore_job(&self, key: &RestoreTargetKey) -> BoxFuture, StorageError>>; - fn list_restore_jobs(&self, status: Option) -> BoxStream>; - fn record_progress(&self, claim: &dyn JobClaim, file_index: u32, items_applied: u64) -> BoxFuture>; - // the per-file (or per-batch) progress write of 5.3 step 5, conditioned on the claim's generation - fn count_restore_target(&self, key: &RestoreTargetKey) -> BoxFuture>; - // a storage-level count of a CREATING table: Scan is refused on one, so the count is a backend method + // RestoreEngine, a trait in StorageEngine's bound (PR 9). StorageEngine has no + // methods of its own; bulk_load, finish_restore and the counters live here. + fn bulk_load<'a>(&'a self, target: &'a RestoreTargetKey, claim: &'a dyn JobClaim, + file: BulkLoadFile, items: BoxStream<'a, Result>) + -> BoxFuture<'a, Result>; // BulkLoadFile { file_index, start_at, reapplying } + // applies one data file with its secondary- and vector-index rows, recording progress in the + // same write; `reapplying` lets MongoDB suppress E11000 on the one unrecorded file a resume meets + fn finish_restore<'a>(&'a self, target: &'a RestoreTargetKey, claim: &'a dyn JobClaim, + item_count: u64, size_bytes: u64) -> BoxFuture<'a, Result>; + // Finished { Active, Pending (MongoDB: the worker flips the table), NotMatched(reason) } + fn count_restore_target<'a>(&'a self, key: &'a RestoreTargetKey) -> BoxFuture<'a, Result>; + fn read_restore_progress<'a>(&'a self, key: &'a RestoreTargetKey) // default Ok(vec![]) + -> BoxFuture<'a, Result, StorageError>>; + fn sweep_restore_progress<'a>(&'a self, running: &'a [String]) // default Ok(0) + -> BoxFuture<'a, Result>; + + // Restore job record on BackupCatalog (section 4). PR 9. A target is addressed by RestoreTargetKey + // (account_id, table_name, table_id); the table id guards a name re-created under it. + fn claim_restore<'a>(&'a self, account_id: &'a str, input: CreateTableInput, + registration: &'a RestoreRegistrationRow) + -> BoxFuture<'a, Result<(TableDescription, Box), StorageError>>; + // 5.3 step 4's registration: create the CREATING target and the RUNNING record under the backend's + // own generation (registration.owner_generation is UNOWNED_GENERATION), the re-check under the + // backup_index row's lock that it is AVAILABLE and not in use (BackupNotFound / BackupInUse else); + // returns the description the request answers + fn claim_restore_dead<'a>(&'a self, key: &'a RestoreTargetKey, stale_after: Duration, + spool_host: Option<&'a str>) + -> BoxFuture<'a, Result>, StorageError>>; + // the restore analogue of claim_dead_backup: take over a RUNNING manifest record whose heartbeat is + // stale, writing the claimant's spool_host, a fresh generation and heartbeat_at = now() in one write (4(e)) + fn read_restore_job<'a>(&'a self, key: &'a RestoreTargetKey) + -> BoxFuture<'a, Result, StorageError>>; + fn list_restore_jobs<'a>(&'a self, status: Option) + -> BoxFuture<'a, Result, StorageError>>; + // remove_restore_target and fail_restore_job are PR 6's, used by the job's and the reconciler's failure + // paths; RestoreJobRow carries store, spool_host, file_index, items_applied, failure_reason, completed_at. ``` -`count_restore_target` exists because a CREATING table accepts no reads (5.3), so the count of section 1 cannot be a Scan. The job holds the same claim as a backup job (5.1 step 3), keyed `/
` on PostgreSQL (the key and seed #384's `try_restore_lock` already uses, so a legacy restore and a new restore into one name exclude each other through the lock as well as the name), the row itself on MongoDB and Cassandra, and heartbeats on the same timer. -The spool and the resume (5.3 step 3): A file that verifies gets a `.verified` marker in the spool, written after the file's `fsync` and renamed into place, so a spooled file without its marker (a crash mid-download) is re-downloaded on resume. The spool is per instance: the `restore_jobs` row records `spool_host` as `:`, a successor whose `spool_host` differs has an empty spool and downloads everything again (its claim writes its own `spool_host` in the conditional write that takes the row, so the sweep of section 4 never removes the spool of the job that owns the row), and two instances must not share a `spool_path` (startup holds a `.lock` file in the spool with an exclusive `flock`, the serve-lock mechanism SQLite already uses, and refuses a spool whose lock another live process holds). A resume with the same `spool_host` reads the manifest and sidecar from the spool; the sweep of section 4 removes only entries of this `spool_host`. - The restore spool holds plaintext items while a restore runs; it is created `0700` with `0600` files, lives outside the store root, and is removed when the job ends. -The load (5.3 step 5): PostgreSQL: multi-row INSERT in 500-row batches with the GSI, LSI and vector-index rows in the same transaction, not through the propagation queue (section 3), one data transaction per file. The catalog and the data live in different databases, so the per-file progress record is a row in the data database (`restore_progress(table_id, file_index)`, data migration 005, reserved for PR 9a next to the catalog numbers in PR 1 and PR 6), written in the file's own transaction; the catalog job row holds only the terminal status. A file is therefore applied exactly once. SQLite: 10k-item batches, each its own transaction holding the batch's index rows, with the progress record `(file_index, items_applied)` committed in each batch's transaction; MongoDB: `insert_many(ordered=false)` per file, the progress record written after the insert, a resumed restore re-applying the one file whose record is missing with E11000 duplicate-key errors ignored on that file only (any other write error fails the restore), then `countDocuments` checked against the cumulative `item_count` (section 1); Cassandra: #384's restore write path (`put_restored_item`, eight in flight) in 10k-item batches, the progress record written after each batch with a lightweight transaction conditioned on the job's `owner_generation`, so a crash re-applies at most one batch, which is safe because the target accepts no other writes and a put of an identical item changes nothing. The table's `restore_progress` rows live in the data database on PostgreSQL (step 5), so they cannot be in that transaction; they are deleted after it commits, and a crash between the two leaves orphan rows for an ACTIVE table, which are harmless because the reconciler reads progress only for a CREATING restore target and removes progress rows whose table is not one (section 4). The other order, progress rows first, would make a crash between the two resume the restore from file 0 and fail a finished restore on the first duplicate key. +There is no `record_progress`: each backend records progress inside `bulk_load`, in the write that applied the items (item 30). `count_restore_target` exists because a CREATING table accepts no reads (5.3), so the count of section 1 cannot be a Scan. The job holds the same claim as a backup job (5.1 step 3), keyed `/
` on PostgreSQL (the key and seed #384's `try_restore_lock` already uses, so a legacy restore and a new restore into one name exclude each other through the lock as well as the name), the in-process slot `restore:/
` on SQLite (4(e)), the row itself on MongoDB and Cassandra, and heartbeats on the same timer. +The spool and the resume (5.3 step 3): A file is downloaded to `restore-
/.json.gz.partial` (`` the file's index zero-padded to six digits, `000003.json.gz`), synced, renamed to `.json.gz` and then given its `.verified` marker, created and synced after the rename, so a spooled file without its marker (a crash mid-download) is re-downloaded on resume and one with it is read through `reader_prehashed`. `.verified` holds the file's sha256; a file is verified for a run only while its marker names the digest the manifest in hand records, so a replaced manifest has no old file accepted. A spool entry is a job's to reuse only while the record named this host before the job's claim: a resume whose record moved to another host and back removes its own old entry before use, and a job that learnt the record is another's (its claim lost, a DONE or FAILED write that matched no record) leaves its entry to the sweep rather than removing it from under the record's owner. The spool is per instance: the `restore_jobs` row records `spool_host` as `:`, a successor whose `spool_host` differs has an empty spool and downloads everything again (its claim writes its own `spool_host` in the conditional write that takes the row, and the sweep of section 4 lists the spool before the records, 5.3 step 7, so it never removes the spool of the job that owns the row), and two instances must not share a `spool_path` (startup holds a `.lock` file in the spool with an exclusive `flock`, the serve-lock mechanism SQLite already uses, and refuses a spool whose lock another live process holds). A resume with the same `spool_host` reads the manifest and sidecar from the spool; the sweep of section 4 removes only entries of this `spool_host`. The restore spool holds plaintext items while a restore runs; it is created `0700` with `0600` files, lives outside the store root, and its entry is removed at every end of the job but the shutdown's, which keeps it for the resume, and the ends at which the job learnt the record is another's (a lost claim, a DONE or FAILED write that matched no record), which leave it to the sweep because a same-host successor may be reading it (5.3 step 7). +The load (5.3 step 5): PostgreSQL: multi-row INSERT in batches of 500 rows or 4 MiB of item text, whichever is reached first, with the GSI, LSI and vector-index rows in the same transaction, not through the propagation queue (section 3), one data transaction per file. The catalog and the data live in different databases, so the per-file progress record is a row in the data database (`restore_progress(table_id, file_index, items_applied)`, migration 6 (a StoreKind::Data body on PostgreSQL only; NotApplicable on SQLite, nothing on MongoDB and Cassandra)), written in the file's own transaction; the catalog job row holds only the terminal status. A file is therefore applied exactly once. On PostgreSQL a file's `restore_progress` row is a plain INSERT when the file is applied from its start and an upsert on `(table_id, file_index)` when it is resumed past its start, so a row a decoder panic left short of the file's item count (its progress committed, the job's FAILED write lost to a second fault) is completed by the resume rather than failing the restore on the row's primary key. SQLite: batches of at most 10,000 items or `RESTORE_BATCH_BYTES` (32 MiB) of serialized item text (the batch closes at the first item that takes it past the bound), each its own `BEGIN IMMEDIATE` transaction under the engine write lock holding the batch's base, secondary-index and vector-index rows, with the progress record `(file_index, items_applied)` committed in each batch's transaction conditioned on the generation and on RUNNING (a record another owner took rolls the batch back); MongoDB: `insert_many(ordered=false)` in 1000-document calls at `w: majority` per file, base documents only (the backfill worker fills the index collections), the record's `file_index` written after the insert conditioned on the generation and on RUNNING, a resumed restore re-applying the one file whose record is missing (`reapplying`) with E11000 duplicate-key errors ignored on that file only (any other write error fails the restore), then `count_documents` checked against the cumulative `item_count` (section 1); the re-application needs an earlier file's record, so a crash before the first file's record (anywhere in the first file) is not re-applied: the resume finds written items and no progress and answers `target_not_empty` (5.3 step 7, `extenddb backup remove-target`); Cassandra: #384's restore write path (`put_restored_item`, eight in flight) in batches of at most 10,000 items or `RESTORE_BATCH_BYTES` (32 MiB) of serialized item text, the progress record written after each batch with a lightweight transaction conditioned on the job's `owner_generation` and on RUNNING, so a crash after the first batch's record re-applies at most one batch, which is safe because the target accepts no other writes while CREATING and a put of an identical item rewrites the same base and index rows: a PutItem or UpdateItem of an item whose secondary-index keys do not change rewrites its GSI and LSI rows rather than deleting them (the write path's per-index delete-and-insert in one LOGGED BATCH would share the batch's timestamp, and a tombstone wins a tie, so the two statements left the row deleted, GitHub issue #742; the write path now skips the DELETE when the old and new items produce the same index row and lets the INSERT rewrite it, so a re-applied restore batch, a resume re-applying at most the one batch it was in, rewrites the same base and index rows and a Query on each GSI sees every restored item as soon as the target reports ACTIVE); a crash before the first batch's record (in the first 10,000 items or 32 MiB) is not re-applied: the resume finds written items and no progress and answers `target_not_empty` (5.3 step 7). The table's `restore_progress` rows live in the data database on PostgreSQL (step 5), so they cannot be in that transaction; they are deleted after it commits, and a crash between the two leaves orphan rows for an ACTIVE table, which are harmless because the reconciler reads progress only for a CREATING restore target and removes progress rows whose table is not one (section 4). On PostgreSQL the `restore_progress` rows are deleted on the data pool after the ACTIVE flip has committed; that delete cannot fail the restore (its error is logged and `RestoreTableFromBackup` still finishes ACTIVE), because the table is ACTIVE and the record DONE once the flip commits, and the reconciler's `sweep_restore_progress` removes any rows the delete left, exactly as it does after a crash between the flip and the delete. The other order, progress rows first, would make a crash between the two resume the restore from file 0 and fail a finished restore on the first duplicate key. State transitions it owns: RUNNING at registration (5.3 step 4); RUNNING to DONE with the ACTIVE flip, on MongoDB DONE then the worker's flip (5.3 step 6); RUNNING to FAILED with the target removed through `remove_restore_target` only after the FAILED write matched (5.3 step 7); the resume of a dead RUNNING row, on PostgreSQL only after the stale window (5.3 step 7). Kill points (testing summary): every restore kill point, the fenced-out restore job and the `pg_terminate_backend` case; SIGTERM during a restore. Main tests that must pass unchanged: #384's override tests (`crates/engine/src/backup/tests.rs`), `table_class_is_not_restored` and `test_restore_into_a_creating_target_and_a_second_restore_of_its_backup_are_refused`; the two orderings over a `backup_index` row are PR 9a's own new tests (deferred from PR 6). Documentation rows it corrects: the differences doc's vector-backup row (the restore's refusal narrowed to a target without vector support) and its multi-part-key row (restored, 5.3 step 2); the differences and limits docs' override rows stay (5.3). ### PR 9b: cross-backend matrix -- PR 9b: cross-backend matrix (twelve cells, nightly: PostgreSQL, SQLite and MongoDB sources into those three and Cassandra), #384's override tests run against the new path per 5.3, MongoDB target refuses a vector manifest, an `N` sort-key value beyond 34 significant digits into MongoDB fails with ValidationException, and a 38-digit non-key number into MongoDB restores intact; the Cassandra-target cells include an unordered backup with a duplicated key. Depends on PR 9a. +- PR 9b: cross-backend matrix (shipped in PR 9): the twelve plain cells (PostgreSQL, SQLite and MongoDB sources into PostgreSQL, SQLite, MongoDB and Cassandra targets), the `--duplicate-key` cells into every target (an unordered backup carrying one key in two files fails into every target, into Cassandra by the count alone) and the `--vector` cells (PostgreSQL and SQLite sources restored into PostgreSQL and SQLite with `SearchVectors` answering, refused before any table into MongoDB and Cassandra), run by `devtools/backup-restore-matrix`: a cell takes a backup on the source server, copies its prefix under a root the target catalog owns and inserts the target's `backup_index` row as `backup sync --adopt-store` will (PR 10; the seed stands in for it until then), so the cell proves the restore and not the adoption, then restores and compares the item sets, the descriptions and the index queries; `.github/workflows/matrix-nightly.yml` (a `schedule` cron and `workflow_dispatch`) runs every cell nightly, building the four backend binaries and starting PostgreSQL, a MongoDB replica set and Cassandra as the integration workflows do. #384's override tests run against the new path per 5.3; the MongoDB target refuses a vector manifest before any table, an `N` sort-key value beyond 34 significant digits into MongoDB fails the restore with the write path's ValidationException text as its reason and the target gone, and a 38-digit non-key number into MongoDB restores intact (`tests/test_mongodb_backup_restore.py`). Depends on PR 9a. ### PR 10: `backup sync`, its options, `remove-target`, `migrate-legacy` Goal: the catalog-loss recovery and the maintenance commands of section 4. Its line names the interfaces and carries the acceptance: -- PR 10: `extenddb backup sync` per section 4 with store-root scoping, the store id, the maintenance claim with its printed wait bound, tombstone and identity checks, the row check before the manifest read and the insert-if-absent, `--prune-missing`, `--forget-store`, `--forget-backup`, `--move-store` (with `list_backup_rows_by_root` and `rewrite_store`), `--adopt-store`, `extenddb backup remove-target
` (section 4), the SQLite serve lock on every command that writes and the dead-owner resolutions such a command runs first, and `extenddb backup migrate-legacy |--all` with its re-read, validation and count checks and its refusals (section 3). Acceptance: drop the catalog rows, run sync, Describe and List and Restore work; a prefix without a manifest, with a bad sidecar, with a mismatched ARN, under `changelog/`, or with a CREATING, DELETING or FAILED row is not adopted; rows for another store root are reported and untouched, and the reconciler deletes none of their prefixes; after a store tree is copied to a new root, `--move-store` re-points its rows and Describe and Restore work against the new root; `--move-store` resolves the old root's CREATING and DELETING rows, stops on a row whose data files were not all copied, and refuses a DELETING, DELETED or FAILED row whose copied prefix holds a verified manifest; `--forget-backup` refuses a CREATING row with a live owner and a backup from which a restore target is CREATING (a RUNNING restore's target, a FAILED restore's target not yet removed, and on MongoDB a DONE restore's target awaiting the backfill), and says so and does nothing for a legacy ARN; a second maintenance command while one holds the claim fails at once; on SQLite every writing command refuses to run beside a server; `migrate-legacy` on a backup the PR 0 binary created produces a prefix Describe, List and Restore serve under the original ARN, and the legacy rows are gone; with the volume unmounted (an empty root, no id or another id) the store is unhealthy, sync without the option that resolves it refuses, and no row changes; `--adopt-store` of a root another catalog owns makes this catalog the owner, and the other catalog finds the store unhealthy at its next check; with host-local roots on two instances of one catalog no reconciler or sync on either resolves or deletes a row because its own root lacks the bytes; `migrate-legacy` refuses a backup with a 13-digit id, one holding a value #618 let in and a pre-0.0.4 SQLite backup, keeps their legacy rows and names the reason; without `--prune-missing` a row whose manifest is gone is reported and stays AVAILABLE; after `--forget-store` and the store's return, sync adopts its prefixes; a CreateBackup started on another instance during `--move-store` either finishes before the rewrite or is refused; after `--move-store A` stops on a row with one data file missing under the configured root and `--forget-backup` lets the move finish, sync reports that prefix as incomplete and does not adopt it, and adopts it once the file is copied; a maintenance command started while a DELETING row and a RUNNING restore with a dead owner exist proceeds once the reconciler, which runs those two duties under the claim, has finished the one and failed the other. Depends on PR 6, 8. +- PR 10: `extenddb backup sync` per section 4 with store-root scoping, the store id, the maintenance claim with its printed wait bound, tombstone and identity checks, the row check before the manifest read and the insert-if-absent, `--prune-missing`, `--forget-store`, `--forget-backup`, `--move-store` (with `list_backup_rows_by_root` and `rewrite_store`), `--adopt-store`, the SQLite serve lock on every command that writes (`extenddb backup remove-target --account
`, section 4, shipped with PR 9 rather than here, item 30: a crash before the first progress record on MongoDB or Cassandra leaves a `target_not_empty` target that no other command removes) and the dead-owner resolutions such a command runs first, and `extenddb backup migrate-legacy |--all` with its re-read, validation and count checks and its refusals (section 3). Acceptance: drop the catalog rows, run sync, Describe and List and Restore work; a prefix without a manifest, with a bad sidecar, with a mismatched ARN, under `changelog/`, or with a CREATING, DELETING or FAILED row is not adopted; rows for another store root are reported and untouched, and the reconciler deletes none of their prefixes; after a store tree is copied to a new root, `--move-store` re-points its rows and Describe and Restore work against the new root; `--move-store` resolves the old root's CREATING and DELETING rows, stops on a row whose data files were not all copied, and refuses a DELETING, DELETED or FAILED row whose copied prefix holds a verified manifest; `--forget-backup` refuses a CREATING row with a live owner and a backup from which a restore target is CREATING (a RUNNING restore's target, a FAILED restore's target not yet removed, and on MongoDB a DONE restore's target awaiting the backfill), and says so and does nothing for a legacy ARN; a second maintenance command while one holds the claim fails at once; on SQLite every writing command refuses to run beside a server; `migrate-legacy` on a backup the PR 0 binary created produces a prefix Describe, List and Restore serve under the original ARN, and the legacy rows are gone; with the volume unmounted (an empty root, no id or another id) the store is unhealthy, sync without the option that resolves it refuses, and no row changes; `--adopt-store` of a root another catalog owns makes this catalog the owner, and the other catalog finds the store unhealthy at its next check; with host-local roots on two instances of one catalog no reconciler or sync on either resolves or deletes a row because its own root lacks the bytes; `migrate-legacy` refuses a backup with a 13-digit id, one holding a value #618 let in and a pre-0.0.4 SQLite backup, keeps their legacy rows and names the reason; without `--prune-missing` a row whose manifest is gone is reported and stays AVAILABLE; after `--forget-store` and the store's return, sync adopts its prefixes; a CreateBackup started on another instance during `--move-store` either finishes before the rewrite or is refused; after `--move-store A` stops on a row with one data file missing under the configured root and `--forget-backup` lets the move finish, sync reports that prefix as incomplete and does not adopt it, and adopts it once the file is copied; a maintenance command started while a DELETING row and a RUNNING restore with a dead owner exist proceeds once the reconciler, which runs those two duties under the claim, has finished the one and failed the other. Depends on PR 6, 8. Invariants honoured: 4(c) (store identity, scoping, adoption only through `--adopt-store`), 4(f) (every writing command takes the maintenance claim), 4(g) (what sync adopts and never adopts; the opt-ins), 4(d) (the ARN checked against the path and `source_table`), and 4(a)'s rule that no deletion duty acts on the bytes of a row the operator forgot. Interfaces it adds: @@ -431,7 +463,7 @@ Interfaces it adds: // listing reads each row's `backups_by_arn` row for the rest of the row fn rewrite_store(&self, keys: &[BackupKey], store: &StoreIdentity) -> BoxFuture>; // `--move-store`, PR 10 ``` -`backup sync` and its options, each with its contract: Sync lists the root (every object, since the trait has no delimiter listing; only the distinct `/
//` prefixes are kept, the reserved `changelog` and `exports/` components skipped), checks each prefix against the catalog as the listing reaches it, reads the manifest and sidecar of the prefixes that have no row, and inserts an AVAILABLE row for each verified manifest whose data files the listing contains, every `data_files` key present with its `bytes`, with an insert-if-absent (`BackupCatalog::insert_backup_row`): a CreateBackup whose row landed between the check and the insert keeps its row and sync skips the prefix. Before adopting, sync parses the manifest's `backup_arn` with the one `BackupArn` type in `extenddb-core` and requires the parsed account, table and backup id to equal the listed path components and the `source_table` identity; a mismatch is logged and skipped. Foreign region or partition in an adopted ARN is accepted and logged; requests check only the account. `--move-store ` re-points the rows of that root to the configured store. It first resolves that root's CREATING and DELETING rows itself, against the configured store: a CREATING row becomes AVAILABLE if a verified manifest is there and FAILED otherwise, and a DELETING row whose prefix there holds no manifest has its deletion finished in the configured store. For each AVAILABLE row it compares the configured store's listing of the row's prefix with the manifest's `data_files` (every key present, with its `bytes`) as well as checking the verified manifest, and stops on a row whose copy is incomplete, naming it; `--forget-backup ` deletes one row's catalog entry so the move can go ahead for the rest; it runs under the maintenance claim like the other commands, refuses a CREATING row with a live owner (the job's next conditional write would match nothing and its manifest would be adopted by the next sync) and a backup in use, one from which a restore target is CREATING, and for a legacy ARN, which has no new-format row, says so and does nothing. One rule covers the three tombstone states: the move refuses to carry a DELETING, DELETED or FAILED row whose prefix in the configured store holds a verified manifest, and names it (that prefix is a backup the operator copied before or during its deletion, not a leftover; `--forget-backup` on the row is the way out). Then it rewrites `store_kind`, `store_root` and `store_id`; the reconciler then cleans the leftovers of the moved tombstones in the configured store. `--move-store` and `--forget-store` match `store_root` exactly as stored, the `?endpoint=` suffix of an S3 root included. A root whose file names another catalog stays unhealthy until an operator adopts it with `backup sync --adopt-store`, which takes the maintenance claim, rewrites the file's owner to this catalog and then syncs, recording the file's store id on the rows it adopts; the previous owner finds the store unhealthy at its next check and stops acting on it. On `sqlite-memory` the catalog, and with it the `catalog_id`, is new at every start, so the server runs a plain `backup sync` itself at startup when a store is configured, which adopts a root whose file names a `sqlite-memory` catalog as `--adopt-store` does, rewriting the file's owner to the new `catalog_id` before it syncs, so the root has one owner at a time there too; a root owned by any other catalog is unhealthy there as anywhere, and the backup CLI commands do not exist there (no file to open), so that startup sync is the only catalog-loss recovery there. +`backup sync` and its options, each with its contract: Sync lists the root (every object, since the trait has no delimiter listing; only the distinct `/
//` prefixes are kept, the reserved `changelog` and `exports/` components skipped), checks each prefix against the catalog as the listing reaches it, reads the manifest and sidecar of the prefixes that have no row, and inserts an AVAILABLE row for each verified manifest whose data files the listing contains, every `data_files` key present with its `bytes`, with an insert-if-absent (`BackupCatalog::insert_backup_row`): a CreateBackup whose row landed between the check and the insert keeps its row and sync skips the prefix. Before adopting, sync parses the manifest's `backup_arn` with the one `BackupArn` type in `extenddb-core` and requires the parsed account, table and backup id to equal the listed path components and the `source_table` identity; a mismatch is logged and skipped. Foreign region or partition in an adopted ARN is accepted and logged; requests check only the account. `--move-store ` re-points the rows of that root to the configured store. It first resolves that root's CREATING and DELETING rows itself, against the configured store: a CREATING row becomes AVAILABLE if a verified manifest is there and FAILED otherwise, and a DELETING row whose prefix there holds no manifest has its deletion finished in the configured store. For each AVAILABLE row it compares the configured store's listing of the row's prefix with the manifest's `data_files` (every key present, with its `bytes`) as well as checking the verified manifest, and stops on a row whose copy is incomplete, naming it; `--forget-backup ` deletes one row's catalog entry so the move can go ahead for the rest; it runs under the maintenance claim like the other commands, refuses a CREATING row with a live owner (the job's next conditional write would match nothing and its manifest would be adopted by the next sync) and a backup in use, one from which a restore target is CREATING, and for a legacy ARN, which has no new-format row, says so and does nothing. One rule covers the three tombstone states: the move refuses to carry a DELETING, DELETED or FAILED row whose prefix in the configured store holds a verified manifest, and names it (that prefix is a backup the operator copied before or during its deletion, not a leftover; `--forget-backup` on the row is the way out). Then it rewrites `store_kind`, `store_root` and `store_id`; the reconciler then cleans the leftovers of the moved tombstones in the configured store. `--move-store` and `--forget-store` match `store_root` exactly as stored, the `?endpoint=` suffix of an S3 root included. A root whose file names another catalog stays unhealthy until an operator adopts it with `backup sync --adopt-store`, which takes the maintenance claim, rewrites the file's owner to this catalog and then syncs, recording the file's store id on the rows it adopts (the `backup_index` row PR 9b's matrix seeds by hand is the row this adoption writes); the previous owner finds the store unhealthy at its next check and stops acting on it. On `sqlite-memory` the catalog, and with it the `catalog_id`, is new at every start, so the server runs a plain `backup sync` itself at startup when a store is configured, which adopts a root whose file names a `sqlite-memory` catalog as `--adopt-store` does, rewriting the file's owner to the new `catalog_id` before it syncs, so the root has one owner at a time there too; a root owned by any other catalog is unhealthy there as anywhere, and the backup CLI commands do not exist there (no file to open), so that startup sync is the only catalog-loss recovery there. `migrate-legacy`: PR 10 adds `extenddb backup migrate-legacy |--all`, which streams a legacy backup through the frozen reader's item stream, checking each item with the backup job's item validator (5.1 step 4), into `DataFileEncoder` and the configured store under the legacy backup id, so the ARN survives, with a manifest whose `snapshot.kind` says it is a copy of an in-catalog backup (`legacy_in_catalog`, no marker). Before it inserts the AVAILABLE row or deletes anything it re-reads the manifest it wrote through `ExtenddbManifest::from_json_bytes` and checks that the manifest's item count equals the legacy backup's; only then does it insert the row and delete the legacy rows through the reader's `delete`. It refuses a backup, keeping its legacy rows, deleting the prefix it wrote and naming the reason, when the id carries no instant (the 13-digit and bare shapes, which the manifest parser refuses, section 4), when the validator refuses an item (a value #618 let in, which main's legacy restore copies as stored), and when it cannot state the definition (a pre-0.0.4 SQLite backup recorded no vector state, so `vector_indexes` would be a guess); the way out for such a backup is to restore it through the legacy reader and take a new backup of the restored table, which has a new ARN (section 10). The maintenance claim per backend and the SQLite serve lock (4(f) states the rules): On PostgreSQL the claim is an advisory lock keyed `hashtextextended('backup_maintenance', RESTORE_LOCK_SEED)` (`MAINTENANCE_LOCK_KEY`), held exclusively on the command's dedicated connection, with the `backup_maintenance` row carrying the holder, the generation and the heartbeat; a job tests the lock with `pg_try_advisory_xact_lock_shared` in one autocommit statement, so the shared lock ends with the statement, and then reads the row's heartbeat age, so a holder whose session ended is still seen held until its heartbeat is older than `stale_job_after_secs`; a command's take tries the exclusive lock a few times over about 10 ms on its connection, so a job's one-statement shared lock does not refuse it. On MongoDB it is a document in a `backup_maintenance` collection, claimed and heartbeated like a job row (5.1 step 3) and held while its heartbeat is younger than `stale_job_after_secs`; on Cassandra a `backup_maintenance` row claimed by lightweight transaction with a coordinator-clock heartbeat, read by jobs at `LOCAL_SERIAL`. On SQLite every backup CLI command that writes the catalog takes the database's serve lock, as `destroy` and `migrate` do, so the server must be stopped first and no other claim is needed; with the serve lock held every job owner is dead, and so is the holder of any maintenance claim row, which such a command takes over whatever its heartbeat says; before the wait it runs the reconciler's dead-owner resolutions itself, as the claim's holder (the cycle takes the held branch without reading the claim back through the zero window) (a CREATING backup row to AVAILABLE or FAILED, a RUNNING restore to FAILED with its target removed, a DELETING row finished) and then waits and removes CREATING rows with a zero stale window, so a job the stop interrupted never makes it wait or refuse whatever its heartbeat says (a RUNNING `manifest` restore, which PR 9a introduces, is the exception revision item 28 states), and a server starting under the serve lock removes a maintenance claim a stopped command left; `backup list` is a read and runs beside a server. @@ -440,12 +472,12 @@ Documentation rows it corrects: upgrade manual: on SQLite the backup CLI command ### PR 11: CI and documentation -- PR 11: CI and documentation: drop the `not import_export` filters, the nightly schedule for the matrix, and the documentation per section 10 with RPO and RTO numbers from the PR 8 and 9a runs on a stated host class. Depends on PR 3, 9a and 9b. +- PR 11: CI and documentation: drop the `not import_export` filters; measure the 1M-item restore on MongoDB and Cassandra and the 1M-item vector cell on PostgreSQL and SQLite (the testing summary's two open cells; the nightly schedule for the matrix shipped with PR 9); the documentation per section 10 with RPO and RTO numbers from the PR 8, PR 9 and these runs on a stated host class. Depends on PR 3, 9a and 9b. Documentation it writes (section 10's table): New `docs/manuals/14-backup-and-restore.md`: strategy, RPO and RTO statements per store kind (RPO equals backup interval, no PITR; RTO measured in the acceptance tests and published), configuration, IAM policy example, restore drill procedure, catalog loss recovery with `backup sync`, migrating legacy backups, what a restore preserves and what it does not (the table from the readiness discussion), moving a backup to or from Amazon DynamoDB with ImportTable and export. The IAM and S3-compatible-server material points at `crates/backup-store/README.md` rather than copying it; the admin guide's database names; ADR-0019 and the Cassandra RFC's "Backup and restore" section marked as superseded; the admin and upgrade manuals' backup sections rewritten. Acceptance: the statements the series defers to the deployment guide, each stated in the manual: a filesystem store keeps the same-host failure domain, said in the first sentence of the backup section, with S3 as the production recommendation; encryption at rest is the store's job; a pickup agent runs as the server's user or as root, and a root under a storage administrator's directory, an NFS mount owned by `nobody` or a group-writable parent without the sticky bit is refused; a bind mount or a filesystem without Unix permission semantics is outside what the store can observe; an `AbortIncompleteMultipartUpload` lifecycle rule, and for a versioned bucket the `NoncurrentVersionExpiration` and `ExpiredObjectDeleteMarker` rules; the least-privilege IAM policy example; a PostgreSQL snapshot retains every dead tuple written during the backup and the SQLite WAL grows by the write volume during it, with the sizing figures the manual measures; the claim connection's keepalive bound (about a minute after the loss of the owner's host) and `pg_terminate_backend` on the claim's session as the way to end it sooner; a transaction-mode pooler is unsupported for the claim; the SDK's `trace` logs carry the session token and are not enabled in production; write access to an account's prefix is backup-administrator authority for that account, with per-account IAM prefix policies; MongoDB's `minSnapshotHistoryWindowInSeconds` as the operator setting a slow backup needs; a root that reports `0700` while enforcing nothing. -Parallel today: PR 1, 2, 3, 4 (PR 0 has merged); PR 5 follows PR 4 and PR 6 follows both. Critical path: PR 4, 5, 6, 8, 9a. PR 1 is not on it: migration numbers are reserved up front so a PITR PR and a catalog PR do not serialize on a shared number. +Parallel today: PR 1, 2, 3, 4 (PR 0 has merged); PR 5 follows PR 4 and PR 6 follows both. Critical path: PR 4, 5, 6, 8, 9a. PR 1 is not on it: migration numbers are reserved up front so a PITR PR and a catalog PR do not serialize on a shared number (PR 6 took id 5 and PR 9a's `restore_progress` id 6, 4(i)). Phase 2, unify export and import (v1 if time allows, otherwise v1.1): ### PR 13: export and import through the store @@ -470,28 +502,28 @@ Phase 3, later (v1.x): - Unit: manifest serde and golden files, key mapping and ARN parsing, store conformance. - Integration, every backend, every PR: round trip with full configuration preservation, composite keys, GSIs and LSIs, vector indexes where supported, legacy fallback, PITR surface. - Consistency: concurrent-writer snapshot test on every backend. -- Crash: SIGKILL during upload and during restore, verified by the reconciler tests. The backup kill points are run by hand against the built binary with the script posted with PR 8 (`kill-points.sh`, outside the repository: a server built with `extenddb-app/backup-test-hooks`, paused by `EXTENDDB_BACKUP_PAUSE` at the named point, killed by PID and restarted, with the row and the prefix watched through `extenddb backup list` and the store), not by a pytest cell; the restore kill points extend the SIGKILL harness the migration tests already use. Backup kill points: between a data file upload and the manifest; between the three documents and the manifest; between the manifest put and the AVAILABLE update (the one path to the reconciler's AVAILABLE outcome). Restore kill points: after a file's transaction commits and before the next file starts; MongoDB between `insert_many` and the progress write (the path that exercises E11000 suppression and the count check); Cassandra between a batch's last write and its progress write; SQLite mid-batch; PostgreSQL between the ACTIVE flip and the progress-row deletion; MongoDB between the DONE write (with `restore_backfill_pending`) and the worker's ACTIVE flip, where the reconciler must leave the row alone and the restarted worker must finish; between the FAILED write and the target removal of step 7; a fenced-out restore job that resumes after a successor claimed the row (its FAILED write matches zero rows and the successor's table survives); PostgreSQL, a restore job whose catalog session ended (`pg_terminate_backend` on its claim connection) while its data writes continue: no successor loads into or deletes the target until `heartbeat_at` is stale, and the job aborts at its next heartbeat. DeleteBackup kill points: after the DELETING write; after the manifest delete; after `delete_prefix` and before the DELETED write. Shutdown: SIGTERM during a backup (FAILED, empty prefix, no multipart upload left) and during a restore (row left RUNNING, finished after the restart). +- Crash: SIGKILL during upload and during restore, verified by the reconciler tests. The backup kill points are run by hand against the built binary with the script posted with PR 8 (`kill-points.sh`, outside the repository: a server built with `extenddb-app/backup-test-hooks`, paused by `EXTENDDB_BACKUP_PAUSE` at the named point, killed by PID and restarted, with the row and the prefix watched through `extenddb backup list` and the store), not by a pytest cell; the restore kill points are run the same way with the script posted with PR 9 (`kill-points.py`, outside the repository: a server built with `extenddb-app/backup-test-hooks`, whose feature chain pulls `extenddb-storage/test-hooks` beside `extenddb-engine/backup-test-hooks` so each backend's own pause point compiles in without the server naming a backend crate, a backup of enough items for two data files, a restore paused at the named point by `EXTENDDB_BACKUP_PAUSE`, `kill -9` by PID, a restart, and the outcome asserted: the exact count after the resume, of the base table and of the GSI `by-g` every item of the fixture carries, or the target gone for a failing restore, or, for the two first-reach cells below, FAILED `target_not_empty` and the target removed by `extenddb backup remove-target --account
`, after which DescribeTable is ResourceNotFound and DeleteBackup allowed), on every backend the point applies to (item 30 lists the cells as run); a Cassandra kill point takes its backup on a SQLite source and seeds the target's row as the matrix does, Cassandra's own CreateBackup being in-catalog (section 3); the two points that pause before a file's or batch's first progress record are run at their first reach (the `target_not_empty` outcome) and at their second (`--second-reach`, the re-application path). Backup kill points: between a data file upload and the manifest; between the three documents and the manifest; between the manifest put and the AVAILABLE update (the one path to the reconciler's AVAILABLE outcome). Restore kill points: after every file has verified and before the first load (`restore_after_download`; on an S3 store the spool holds every `.verified` marker, the manifest and the sidecar, and no item was written); after a file's transaction commits and before the next file starts (`restore_after_file_commit`); MongoDB between `insert_many` and the progress write (`restore_mongodb_after_insert_many`, the path that exercises E11000 suppression and the count check); Cassandra between a batch's last write and its progress write (`restore_cassandra_before_progress`); SQLite mid-batch (`restore_sqlite_mid_batch`); PostgreSQL between the ACTIVE flip and the progress-row deletion (`restore_postgres_before_progress_delete`); MongoDB between the DONE write (with `restore_backfill_pending`) and the worker's ACTIVE flip (`restore_mongodb_before_worker_flip`), where the reconciler must leave the row alone and the restarted worker must finish; between the FAILED write and the target removal of step 7 (`restore_between_failed_and_removal`); a fenced-out restore job that resumes after a successor claimed the row (`restore_fenced_out`: its DONE write matches zero rows, it makes no table access and the successor's table survives with the exact count); PostgreSQL, a restore job whose catalog session ended (`pg_terminate_backend` on its claim connection) while its data writes continue: no successor loads into or deletes the target until `heartbeat_at` is stale, and the job aborts at its next heartbeat. DeleteBackup kill points: after the DELETING write; after the manifest delete; after `delete_prefix` and before the DELETED write. Shutdown: SIGTERM during a backup (FAILED, empty prefix, no multipart upload left) and during a restore (row left RUNNING, finished after the restart). - Distribution matrix for the filesystem default path: unprivileged local launch, the npm launcher, the container image with and without a volume, `sqlite-memory`. - Cross-backend matrix: nightly. - Live DynamoDB: Phase 2, measured. -- Performance: backup and restore of a 1M-item table per backend, and of a 1M-item table carrying a vector index on the backends that support one, duration and peak memory recorded as the published RTO baseline; memory must stay flat with table size (streaming, never materializing the table). Index contents are not stored in the backup; every index is rebuilt from the items on restore, so the vector cell measures the dominant restore cost. +- Performance: backup and restore of a 1M-item table per backend, and of a 1M-item table carrying a vector index on the backends that support one, duration and peak memory recorded as the published RTO baseline; memory must stay flat with table size (streaming, never materializing the table) in the restore's own terms: on SQLite a table with secondary indexes costs SQLite's page cache for the index b-trees above the no-index figure, an unordered backup costs its page cache across the base b-tree above the key-ordered figure, and the final `COUNT(*)` costs the counting connection's page cache, each bounded by `cache_size` (section 7, the record). The backup side's 1M figures are PR 8's record; the restore's are PR 9a's (`docs/design/15-backup-restore-bench-pr9.md`): 1M items of 200 bytes in key order in 9.5 s on PostgreSQL and 26.8 s on SQLite, the peak over idle 16 and 89 MiB, the growth from 100k to 1M 5.5 MiB on PostgreSQL and 51 MiB on SQLite (the write connection's page cache and the final `COUNT(*)`, each bounded by `cache_size`, not the items; from 1M to 3M rows the SQLite peak moved 11 MiB), and the same 1M items as an unordered backup 16 MiB on PostgreSQL and 144 to 164 MiB on SQLite in 66 to 90 s (SQLite's base b-tree under scattered inserts, not the restore's buffers). The MongoDB and Cassandra 1M restore and the vector cell are still to be measured (PR 11). Index contents are not stored in the backup; every index is rebuilt from the items on restore, so the vector cell measures the dominant restore cost. ## Review checklist for implementation PRs - No data file is uploaded outside a snapshot; the snapshot is opened before the first read and closed after the last. - `extenddb-manifest.json` is written last and only after every other object succeeded. - Every failure path deletes the prefix or leaves it unreachable from List, and the row never says AVAILABLE for an incomplete prefix. -- Restore writes no item before every data file's checksum has verified; a verification failure deletes the CREATING target table, which never accepted a read or a write; the request refuses before creating the table when the target cannot represent the manifest or the key ranges overlap. -- A file's data and its progress record commit in one transaction on PostgreSQL and SQLite (per 10k-item batch on SQLite, with the items-applied count), so no item is applied twice; on MongoDB the progress record follows the insert, re-application of the one unrecorded file is idempotent by duplicate-key suppression (code 11000 only; any other write error fails the restore), and the post-resume document count is checked against the manifest; on Cassandra the progress record follows each batch under a lightweight transaction and at most one batch is re-applied. A key-ordered restore runs the per-file key-range check on every file, and an unordered one ends with a count of the target equal to the manifest's item count. On PostgreSQL the ACTIVE flip precedes the deletion of progress rows; on MongoDB the job's DONE write and `restore_backfill_pending` are one transaction, and the worker's ACTIVE flip is conditioned on that flag and CREATING, not on the job's generation. -- Every status write by a job is conditional on its owner generation and on the row still being in the job's state (`CREATING` for a backup, `RUNNING` for a restore); a job whose conditional write matches no row, or whose conditional write's outcome stays unknown after the re-read, performs no further store or table operation, deletes (of a prefix or of a restore target) included; the reconciler takes the same claim as a worker and leaves `heartbeat_at` untouched. +- Restore writes no item before every data file's checksum has verified once; on a filesystem store the load's second read is hashed again and a file replaced between the passes fails the file, its items removed with the target (5.3 step 3); a verification failure deletes the CREATING target table, which never accepted a read or a write; the request refuses before creating the table when the target cannot represent the manifest or the key ranges overlap. +- A file's data and its progress record commit in one transaction on PostgreSQL (the `restore_progress` row, per file) and SQLite (per batch of at most 10,000 items or 32 MiB of item text, with the items-applied count), so no item is applied twice; on MongoDB the `file_index` record follows the insert, re-application of the one unrecorded file (`BulkLoadFile::reapplying`) is idempotent by duplicate-key suppression (code 11000 only, on that file only; any other write error fails the restore); on Cassandra the progress record follows each batch under a lightweight transaction and at most one batch is re-applied. A key-ordered restore runs the per-file key-range check on every file, and every restore ends with a count of the target equal to the manifest's item count. On PostgreSQL the ACTIVE flip precedes the deletion of progress rows; on MongoDB the job's DONE write and `restore_backfill_pending` are one transaction, and the worker's ACTIVE flip is conditioned on that flag and CREATING, not on the job's generation. +- Every status write by a job is conditional on its owner generation and on the row still being in the job's state (`CREATING` for a backup, `RUNNING` for a restore); a job whose conditional write matches no row, or whose conditional write's outcome stays unknown after the re-read, performs no further store or table operation, deletes (of a prefix or of a restore target) included; the reconciler takes the same claim as a worker and leaves `heartbeat_at` untouched on a backup row and refreshes it on a restore record (4(e)). - The ARN is parsed by `BackupArn` once, in the engine, before any catalog or store access; no other code splits ARN strings. - Account scoping is checked on the ARN before any store access, every backup row is read by account and backup id and accepted only when its account, table and ARN equal the request's, and every restore-target method takes the target's account, name and table id. - The backup job validates every item with `validate_restorable_item` before encoding it, keeps validating after the first refusal, and the restore binds the same function. -- A restore's registration re-checks, under the backup row's lock, that the row is AVAILABLE and not in use, and DeleteBackup's in-use check and DELETING write run in one transaction under the same lock. -- A failed restore's target is removed through `remove_restore_target`, never DeleteTable; on PostgreSQL and SQLite `bulk_load` writes index rows in the batch's transaction, with the base rows, and the ACTIVE flip sets every index of the target ACTIVE. +- A restore's registration re-checks, under the `backup_index` row's lock (5.3 step 4 names it per backend), that the row is AVAILABLE and not in use, and DeleteBackup's in-use check and DELETING write run in one transaction under the same lock. +- A failed restore's target is removed through `remove_restore_target`, never DeleteTable, and only after the FAILED write matched one record (`fail_restore`); on PostgreSQL and SQLite `bulk_load` writes the secondary- and vector-index rows in the batch's transaction, with the base rows, and the ACTIVE flip sets every index of the target ACTIVE. - No API operation or job touches the store while it is unhealthy, the reconciler acts on no row of the store then (its legacy duty, which touches no store, still runs), and only sync with a resolving option and the health check itself read it. - No credential, bucket name with credentials, or presigned URL appears in a log line, and no API error names a store root. -- New workers stop on the shutdown signal within the drain window: a backup runs its failure path, which the server's shutdown awaits, and a restore stops with its row left RUNNING. +- New workers stop on the shutdown signal within the drain window: a backup runs its failure path, which the server's shutdown awaits, and a restore stops with its row left RUNNING and its spool kept. - Tests create composite-key tables, not only hash-only tables. ## Drawbacks @@ -556,3 +588,5 @@ Changed after a review of the draft against main `9f43d03` and PRs #384, #335, # 27. (2026-10-30, PR 6.) The FAILED-target duty lands in PR 6: `list_dead_restore_jobs` also returns a FAILED `restore_jobs` row whose target is still CREATING once the row is stale, and the reconciler removes the target without a second FAILED write, which closes the state the job's or the reconciler's failure path leaves when the FAILED write lands and the removal does not (4(e), PR 6, PR 8). PostgreSQL's dead rule is the heartbeat alone, as on MongoDB and Cassandra: the advisory lock is the claim's exclusion, not the liveness test, and the `IS NULL` arm item 21 named does not exist, the column being NOT NULL (4(e), 5.3 step 7, PR 6). The stale floor is computed from the period the jobs run at, `max(heartbeat_interval_secs, JOB_HEARTBEAT_INTERVAL)`, the one constant every backend's heartbeat loop reads until the knob reaches the jobs, and `max_concurrent_jobs` has an upper bound (5.1 step 4, section 7, PR 6). SQLite's `backup_server_slot_count` is kept for snapshot opens until PR 8, the legacy create trying it without waiting and answering the one `LimitExceededException` text (PR 6). The engine's `BackupArnSpellings` carries an optional parsed ARN per spelling, so a legacy backup of a name the store-prefix rules refuse (dot-only, over 255 bytes) still describes, restores and deletes through the legacy reader (4(d), PR 6). DeleteBackup over a row runs the in-use check before the other-store check, as a restore does, builds the description before the DELETING write, and a row whose source table the build cannot read is left out of ListBackups rather than failing the listing (5.2, PR 6). The acceptance cell running the two orderings over a `backup_index` row is deferred to PR 9a, with the row lock observed by a two-transaction test of `start_backup_delete` in PR 6 (PR 6). On Cassandra an index row's list rows exist for as long as the row does, whatever its status, readers re-reading the authoritative row; the legacy create keeps its publish-time rule (PR 6). The reconciler records the worker metrics of section 9 (PR 6). On Cassandra the migration runner writes `catalog_version` after the last file's statements, its backfill step and its `schema_history` row, so a run that fails in the step leaves the previous version and `migrate --yes` runs the file again; a Cassandra job stops after three heartbeat errors in a row, as after a heartbeat that does not apply, and the fence the reconciler puts on a dead create's row takes a fresh `owner_generation` (4(e), PR 6). MongoDB's RestoreTableFromBackup waits a bounded time for the registration and answers `InternalServerError` past it while the registration goes on; what the request leaves is stated in 5.3 and the differences document (5.3, PR 6). CreateBackup checks `server.region` before forming a backup ARN and refuses, naming the setting, under a region outside `[a-z0-9-]`, where main created a backup no operation could then address; the store-prefix rules on the name are applied before the ARN is formed so a name over 255 bytes is refused naming the limit whatever its length (5.1 step 1, PR 6). The `LimitExceededException` text names the bound that was met beside its number (`max_concurrent_jobs`, SQLite's snapshot slots, or PostgreSQL's connection slots for the claim connection), one implementation (`limit_exceeded_message`) every refusal binds (5.1 step 1, section 7, PR 6). Section 7 states, where it bounds snapshots by half the pool, that the pool is the raised one (section 7, PR 6). Item 21's `IS NULL` arm is withdrawn by this item alone; item 21 keeps its wording, as item 22 requires. On Cassandra, where the removal is a claim and then a drop, the FAILED-target duty also lists a FAILED record whose target is DELETING with its registration, and `remove_restore_target` resumes the removal from the drop (4(e), PR 6). On MongoDB the in-use check of a restore's registration and of DeleteBackup reads at the transaction read concern, majority-committed data, while DescribeTable reads at `local`, so a request issued within the majority-commit lag after DescribeTable first shows the target ACTIVE can still answer `BackupInUseException` and succeeds when retried; the differences document's concurrent-restores row states it with the measured lag (5.3, PR 6). The rule that a job performs no further operation after a write it cannot confirm is scoped to the conditional write itself: a legacy restore whose claim connection (the heartbeat's) errors runs 5.3 step 7, the conditional FAILED write on a pooled connection and the removal only when that matched, so on PostgreSQL the target is removed at the next heartbeat rather than by the reconciler once the record is stale (4(e), 5.3 step 7, PR 6). On MongoDB `remove_restore_target` deletes the target's catalog document first and returns the target removed once that is gone: a collection drop that fails after it is logged naming the collections and leaves them for `backup sync` (PR 10), there being no record for a later reconciler cycle to retry from (4(e), PR 6). PR 6's interface block lists `list_dead_restore_jobs`, `fail_restore_job` and `LegacyBackupReader::abandoned_creating_backups` with their contracts, as the trait ships them (PR 6). On MongoDB `serve` checks the catalog version before it builds the data engine or starts the startup key repair of documents stored without their key (#473), so a server that refuses the catalog has written nothing, as on PostgreSQL (PR 6). 28. (2026-10-30, PR 7 as implemented.) A store problem never gates the in-catalog create: with a configured store that is unhealthy, CreateBackup writes the in-catalog format as it does with no store, and only the store-backed create of PR 8 is refused by `require_healthy_store`; a defaulted SQLite root whose identity check fails (a catalog re-created by `destroy` and `init` under the old root's `extenddb-store.json`) is the same warning as a failed validation, leaving the store unconfigured, and `destroy` and the admin guide say the file must go with the catalog (section 7, PR 7). An S3 store's `validate` and identity check run after the listener is up, in the reconciler's first cycle, with the store `checking` until the first result reports; a filesystem store's run before the listener; the configuration errors that need no request still stop startup; a `validate` refusal of an S3 bucket after the listener makes the store unhealthy (`failed validation`) rather than stopping startup (section 7, 9, PR 7). `/health`'s `reason` is the problem's category alone; the ids and ARNs go to the operator's log line (section 9, PR 7). The identity file is read up to 64 KiB. The S3 root label is built from the endpoint the client uses, in one spelling. A check that cannot read the catalog leaves the store's health unchanged. An existing spool must be owner-only. The maintenance claim is heartbeated on a timer from the take to the release, and PostgreSQL's `check_maintenance_claim` tests the advisory lock before the heartbeat's age (4(f), PR 7). The jobs' side of the claim is implemented in the engine for the in-catalog create and restore, before the permit and, since those paths commit their row inside the backend, before the row exists; the ordering 5.1 step 2 states applies to the store-backed job PR 8 adds (5.1, 5.3, PR 7). `backup list` opens the SQLite catalog without the serve lock, reads the identity file without creating it and runs on an unhealthy store; on SQLite `backup purge` runs the reconciler's duties with a zero window before its wait; a CREATING row with a dead owner is purged as any row, through `remove_dead_creating_backup_row`, a `BackupCatalog` method of PR 7; a FAILED row with a young heartbeat is waited on; the dry run exits 0 (PR 7, PR 10). Cassandra's claim reads run at `LOCAL_SERIAL` and an insert whose answer was lost is settled by a re-read of the generation (PR 7). The PR 10 comment on `backup_index_by_status` is corrected: V006's entry carries `store_kind` and `store_id`. The body sentences that paraphrased the earlier rules are edited with this item (section 7's configuration-error paragraph, 4(f) and 5.1 step 2 and 5.3 step 4's claim check, PR 6's `store_root` row, PR 7's interfaces and acceptance), and the filesystem store's startup checks are bounded by a timeout that leaves the store `checking` when a mount stops answering after the root is opened (a root that does not answer at startup holds the server before the listener, as the database directory would). Also: the PostgreSQL check takes its shared lock at transaction scope so a cancelled request leaves no lock on the pool; a dropped `HeldClaim` stops its timer; the purge's dry run is a read like `list`; a CREATING row is removed before its prefix; an empty `filesystem.path` is left for `open` to refuse; the spool meets the root's whole rule, parents included; `backup list` reads the catalog's status without the serve lock. The defaulted SQLite root's warning holds only while the catalog has no backup rows for the root (`read_store_ids(Filesystem, root)`: the upgrade, fresh and re-init cases); with rows for it the root has been used and is handled as an explicitly configured one, so its backups never vanish behind a warning: a failure of its identity check or manifest probe is an unhealthy store (reported, its rows' restores and deletes refused, rechecked each cycle), and a refusal of its `open` or `validate` a startup error naming the root (section 7, PR 7's acceptance). The server's `configure` carries the reason for an absent store out (`NotConfigured`), and `backup list` and `backup purge` print it, so a defaulted root that failed its check is no longer reported as "[backup.store] is absent or disabled". The store's check runs under a per-kind bound in every reconciler cycle (`check_configured_store`: 5 s for a filesystem root, the startup bound; 30 s for a bucket, under the SDK's 60 s per-attempt timeout so a timed-out check is dropped before the SDK retries its request), recording a transport failure naming the bound, and the cycle selects the check against the shutdown token; an unbounded check let a stalled S3 endpoint hold each cycle for the SDK's whole retry budget and a stalled mount hold it for good, with shutdown hitting the drain timeout (section 7). A filesystem root's `validate` reports no transport class; an I/O failure of its probe write or of resolving the root at startup degrades the store (`failed validation`) as the same condition one cycle later does, while the owner-only and existence refusals stay configuration errors (section 7). A transport failure's operator line carries the store error's whole cause chain (the SDK's top link is a bare class). Every store error of a check that is neither a transport failure nor a denial is `failed validation`, the category `validate` gives it, so a bucket deleted under a running server reads as a bucket missing at startup. A filesystem root that is or contains the database directory is refused in `resolve_store`, beside the spool rule (section 7). A filesystem store's synchronous root checks run on the blocking pool so the startup bound fires whichever call stalls. The checking store's refusal says its first check is pending; `/health`'s reason list carries `catalog unreadable`; `record_check` reports whether the category changed. Purge's wait set is CREATING and FAILED rows with a live heartbeat and RUNNING restores, DELETING rows being finished rather than waited on, and its refusal rule is that set at the wait's end; 4(f), the PR 7 "Destroy and purge" paragraph and the PR 10 claim paragraph (`MAINTENANCE_LOCK_KEY`, `pg_try_advisory_xact_lock_shared` then the heartbeat age, `LOCAL_SERIAL`) and the PR 6 and PR 7 signatures (`check_maintenance_claim(stale_after)`, `maintenance_claim(holder, stale_after) -> Box`) are edited to the code with this item. A row a DeleteBackup moves to DELETING or DELETED, or removes, between purge's read and its turn is finished and counted as deleted, not left (the deletion is one idempotent sequence on both sides). On SQLite purge waits and removes CREATING rows with a zero stale window (under the serve lock every owner is dead, so a young heartbeat proves nothing), and the server's startup under the serve lock removes a maintenance claim a stopped command left, so a killed purge costs no stale window; on every backend purge releases the claim on `SIGINT` or `SIGTERM`, at once during the wait and before the next row during the deletion. The wait's three failures (jobs still running at the limit, the claim lost, the catalog unreadable) and the signal are reported each as itself. PostgreSQL's factory starts the GSI queue workers only for a server, not for a command reading beside one. Purge's restore part of the wait set is the RUNNING `restore_jobs` records alone: the dead-job listing it reads with a zero window also returns the FAILED records whose target is still CREATING (the FAILED-target duty's, 4(e)), which are the reconciler's to finish and no job runs under, so they are not waited on; the wait finishes at once on a stopped deployment that holds one. On SQLite the zero-window pass resolves the kinds this release writes; a RUNNING `restore_jobs` record of kind `manifest`, which PR 9a introduces, is left for the reconciler that resumes it and purge waits on it and refuses naming the table, the one state the PR 10 paragraph's "never makes it wait or refuse" does not cover. The reconciler finishes the configured store's DELETING rows while the maintenance claim is held, as 4(f) lists. The binary ends its tokio runtime with a bounded shutdown (5 s) after `serve` and after every command: a filesystem store's check abandoned at its bound leaves its call parked on a blocking thread until the mount answers, and a plain drop of the runtime joined that thread, so a server whose root had stopped answering drained its workers at SIGTERM and then stayed alive until SIGKILL. Each call of a store check runs as a task of its own that an abandoned check does not take with it, and the store records that a call is in flight; no check starts while one is (the store is recorded `unreachable` again, with the reason), so a mount that stays stalled parks one thread rather than one per cycle. A command's identity read (`backup list`, the dry run) runs under the store's check bound as the server's check does, where the unbounded read waited out the SDK's three attempts of 60 s on an endpoint that accepts and never answers. An empty `filesystem.path` is compared with nothing in `resolve_store` (compared as written it was the working directory, refused by name as the database directory when the server starts there) and left for `open` to refuse. A root that refuses `link(2)` with `EPERM` is reported `failed validation` (the README and the deployment guide said `unreachable`). `tests/test_backup_store_config.py` is excluded from `devtools/run-tests`'s pytest run, as its docstring says; a catalog whose rows cannot be read keeps a defaulted root configured, and that arm is tested. On SQLite the purge's own reconciler cycle is told that its caller holds the maintenance claim (`ReconcilerState::for_maintenance_holder`) and takes the held branch without reading the claim; read through the zero dead-owner window, the command's own claim was seen held only when its heartbeat and the check fell in one millisecond, so the branch was decided by scheduler timing (PR 7, PR 10). Purge's test of a live owner is the backend's `list_backup_rows_older_than` on the heartbeat clock, a `BackupCatalog` method of PR 7 that PR 8's reconciler also lists by, where it compared `heartbeat_at` with this host's clock, which a host running ahead of the database read as a shorter wait, not a longer one (4(e), 4(f), PR 7). On SQLite a command under the serve lock takes over a maintenance claim row whatever its heartbeat says, and PostgreSQL's take tries the exclusive lock again over about 10 ms, so a job's one-statement shared lock does not refuse it (PR 10's claim paragraph). On MongoDB a command that opens the catalog beside a server (`backup list`, the dry run) starts no key repair of documents stored without their key (#473); the server does. On PostgreSQL every `backup` command registers as a server does and its registration's session carries the command's name as `application_name`, so a migration that waits for it names it, and a catalog the factory refuses ends the command with the message and exit code `serve` gives (3 behind, 6 while a migration runs). The jobs' refusal while the claim is held names a backup maintenance command, not `purge`, since every maintenance command holds the one claim. The console banner shows a filesystem root in the canonical spelling the rows and the log line use, with the configured spelling only where it differs (PR 7). 29. (2026-10-31, PR 8 as implemented.) The job and the reconciler live in the engine's backup module (`crates/engine/src/backup/{job,reconciler}.rs`), not in `crates/server/src/workers.rs`, and nothing crosses an in-process channel between the request and the job but the response (the pipeline's item and chunk channels are the job's own): the request's detached task opens the snapshot, claims the row, checks the maintenance claim and answers through a one-shot channel, then continues as the job; the server hands the engine the shutdown token, the metrics collector, the job settings and the manifest's bucket and key prefix through `BackupRuntime` and `ConfiguredStore`, and `drain_jobs` (30 s) runs before the workers drain (PR 8); a stopped pipeline's sink answers the encoder's write with `BrokenPipe`, which nothing retries, where an `Interrupted` answer (std's "try again") kept a job cancelled in the middle of a 300k-row table spinning in the compressor's trailer write through the whole drain with its row left CREATING; the job ends within milliseconds of the signal on every store (5.1 step 5, PR 8). The reconciler's takeover is `BackupCatalog::claim_dead_backup(key, stale_after)`, not `claim_backup`, which inserts; `list_backup_rows_older_than(status, store, clock, age)` with `StaleClock::{Heartbeat, Completed}` carries the database's clock as a backend predicate for the FAILED-prefix delay and the tombstone retention, so the engine never holds a clock; `sweep_restore_registrations` is the Cassandra `active_restores` sweep with a default empty body; `StatusFields` gains `source_table_details_json`, which the job's and the reconciler's AVAILABLE writes fill with the manifest's `source_table` (4(e), PR 8). The PostgreSQL page is three layers, since a window function runs before `LIMIT`, and the stream ends when the inner query returned fewer rows than asked and none was cut (a page cut short of the row bound is not the end); the outer `ORDER BY` is a stated guarantee the executor gives anyway; SQLite's stream pulls no row past its cut (the driver's worker reads at most five rows ahead of it: the four `row_buffer_size` on the pool buffers and the one it holds while its `send` waits), so the rows-past-cut figure is PostgreSQL's alone (section 3, PR 8). `heartbeat_interval_secs` reaches the store-backed job as `max(heartbeat_interval_secs, JOB_HEARTBEAT_INTERVAL)`; the in-catalog paths keep the constant until PR 14, which corrects item 27's "until the knob reaches the jobs" (5.1 step 4, section 7). The job's pre-put check is two periods, not one (a beat sent at the tick returns up to a period later), which stays inside the three-period floor; the check measures from the instant taken before the claim's insert, not from the heartbeat handle's start, so an unbounded maintenance check (a catalog query held behind the pool) cannot admit a first put on a row already past `stale_job_after`; when the pipeline returns no put is in flight, so the failure path's `delete_prefix` follows the FAILED write with nothing to await; a lost or stalled heartbeat ends the job with no catalog write, the claim connection being the one that failed; each unknown outcome of a status write is followed by a re-read; the write is sent once more at most; after the second write a re-read that shows the row in the target status counts as matched and anything else as matching no row; within one cycle the duties run dead-owner first and FAILED-prefix after it, so a row the cycle just failed loses its prefix in that cycle once the dead owner's beat is stale (5.1 steps 4 to 6, 4(e)). SQLite's snapshot slot is kept as the backend's admission bound and the pool raised to `2 * (max_concurrent_jobs + 1)`, at most 34, which is what item 27's "folds it into `max_concurrent_jobs`" became (section 7, PR 6). On SQLite `claim_dead_backup` reads no window: a row whose slot no task holds is dead at once, and the engine calls it on every CREATING row each cycle (one set probe per live row); the dead owner's handle cannot observe `NotMatched` after a takeover on SQLite or PostgreSQL, whose slot and lock are the owner's life, and the engine reaches a taken-over row through `Unknown` and the re-read (4(e)). On MongoDB and Cassandra a successor is a live owner to a later taker only once it heartbeats, and the reconciler writes its one status before any beat, so two instances' reconcilers can both take over one dead row in that gap; the generation fence makes the loser's write `NotMatched` and one resolution lands; Cassandra's takeover condition carries the heartbeat it read (`IF owner_generation = AND backup_status = 'CREATING' AND heartbeat_at = `), which puts the liveness test inside the write; Cassandra's `list_backup_rows_older_than` reads the coordinator clock before its scan where `list_dead_restore_jobs` reads it after, both conservative; MongoDB's `Completed` listing requires the column present, since `age_is_stale` alone reads an absent column as the epoch (4(e), PR 8). MongoDB's stream applies no cut of its own (the server's 16 MiB reply bound is under the budget) and feeds no page-bytes counter, so `run_page_bounds` runs on PostgreSQL and SQLite only (section 3, PR 8); Cassandra's registrations sweep is one range read of `backups_by_arn` per cycle, enough while a catalog holds thousands of backups and not more, and the `backup_size_bytes` of `backups_by_account` and `backups_by_table` is written at the claim and not refreshed by `set_status`, which the index listings do not read (they re-read the row), as before PR 8 (PR 6). The spool sweep and the `restore_progress` sweep land with PR 9a, which adds the records they read, and the duty that removed a CREATING restore target with no job row is withdrawn: the job record is the registration on every backend (section 4, PR 8). The vector "round trip" PR 8 adds is its create half, the restore half being PR 9a's; the vector wire test asserts the store's ValidationException on the restore until then, and the fidelity suite's 24 restore cells skip on a server with a store (PR 8, PR 9a). `source_table.time_to_live` is read beside the open, outside the snapshot, and may trail it by the open's duration; DescribeBackup's `ItemCount` is the stats pass's figure while CREATING and the stream's count once AVAILABLE (5.1 step 2). Section 9 lists the metrics as `/metrics` emits them, the `MetricName` variants PR 8 adds (`BackupJobs`, `BackupJobDuration` in microseconds, `BackupBytesUploaded`, `BackupSnapshotAge`, `BackupStoreErrors`, the outcome and the store call in the operation dimension) as JSON under those names, and says the snake-case spellings are a converter's names and not a rendering the server does (section 9). `EXTENDDB_TEST_BACKUP_STORE` is set by every CI job that runs a suite creating backups, the pytest jobs and the Rust integration jobs (`filesystem` on PostgreSQL, which appends a store after `init`, and on SQLite, the defaulted root; `s3` on the SQLite job against moto; `none` on MongoDB and Cassandra and on the PostgreSQL Rust integration jobs; `run-integration-dev-mode` runs `tests/test_dev_mode_authorization.py` alone, which creates no backup, without it) and `devtools/run-tests --pytest` refuses without it (section 10, PR 8). PostgreSQL's snapshot-slot refusal now goes through `limit_exceeded_message` naming `[storage.postgres] pool_size snapshot-slot` (item 26's one-implementation rule). The acceptance's "on each backend" for the 100k backup reads "on each backend with a `SnapshotEngine`", Cassandra keeping the legacy create, and for the #618 cells "on PostgreSQL and SQLite", the two deployments `tests/test_backup_job.py`'s fixture starts (MongoDB's store-backed create is covered by the engine tests and the 100k run; no CI job runs the suite on a MongoDB build); an `abc` range key is refused by the number column on PostgreSQL and SQLite, so the range-key cell carries `1E126`, and UpdateItem validates the values its expression sets, so its cell carries the bad value in the key alone (PR 8). The 1M-item RSS runs and the kill points were measured on a 16-core, 30 GiB host (PostgreSQL 16.15 and MongoDB 7.0 in containers on loopback, the bundled SQLite 3.46) and the figures recorded with the PR in `docs/design/15-backup-restore-bench-pr8.md`; the memory bound holds (the 1M run's peak within 10 MiB of the 100k run's on PostgreSQL and SQLite, and the 500k run's within 5 MiB on MongoDB, where the 500k load alone took 27 minutes through the wire on that host; the bound is 112 MiB). Section 7's memory paragraph is rewritten to the terms the worst-expansion run measured: the earlier "the budget plus one row of text, plus its decoded form" left out the page's decoded rows waiting for the pipeline, the items in the feeder, the item channel and the encoder, one row's parse through the deserializer's intermediate value, and the line buffer, and it said nothing of the allocator; with 27.7 MB rows (three to a page) the terms sum to about 400 MB and the high-water mark over idle was 880 MiB on PostgreSQL and 580 to 760 MiB on SQLite across runs (507 MiB under `MALLOC_ARENA_MAX=1`), with MongoDB measured at 474 MiB on 13.5 MB rows, its 16 MiB document cap refusing the full shape; SQLite's driver run-ahead was bounded (sqlx's `row_buffer_size`, 50 rows by default, set to four on the pool: the 24-row run peaked at 844 MiB over idle with 50 and 696 MiB with four, back to back, and a 1M-row backup took 11.4 s with either), and whether the peak then grows with the number of worst rows is settled by the 12, 24 and 48-row series of the 2026-11-01 sentences below (it does with glibc's default arenas and does not under `MALLOC_ARENA_MAX=1`), and a one-item channel saved about 50 MiB of it for a 1M-row backup 75% slower (12.4 s to 21.6 s, both orders), so `ITEM_CHANNEL` stays at four. The PR 8 acceptance's "stays within section 7's formula" and "RSS must match the formula" read "under 1.2 times the sum of section 7's terms at the measured row with `MALLOC_ARENA_MAX=1`, at 12, 24 and 48 worst rows on one binary, the default-arena peak recorded beside it" (section 7, PR 8). The body sentences each of these corrects are edited with this item (4(e), 5.1 steps 2, 4, 5 and 6, section 7's knob comments, memory paragraph and pool paragraph, section 9, section 10's table, PR 6's slot sentence, PR 8's acceptance, interfaces, page paragraph and duty list, PR 9a's lead-in); earlier items keep their wording, as item 22 requires. (2026-11-01, PR 8 as measured and corrected.) The PostgreSQL page's row bound `n` is sized from the previous page's largest row, not its average, and the floor after the first page is 4 (`MIN_NEXT_PAGE_ROWS`; `MIN_PAGE_ROWS`, 16, is the first page's alone), because the server serializes and spills every one of the `n` rows the inner `LIMIT` reads, so a page costs the server `n` rows of text and not the rows it returns: after a page of 984 small rows and 3 large ones the average sized the next page at 810 rows of a table that was then large rows only, 5.9 GB of server temp files and 18 to 44 s for the three rows it returned, where the largest row sizes it at 4 (208 MiB of temp, 1.1 s), and a page of 27 MB rows now reads one row past its cut rather than thirteen (its steady state 1.2 s a page, 4.4 s before); the over-read that remains is the page that follows a row bound's worth of small rows, up to `n` rows of up to the line cap, which the sentence of the same day below bounds (section 3, 7, PR 8). The `EXPLAIN` acceptance holds for a page with the planner's estimate above `n` and not everywhere: on a table of 1,060 rows a page near the end whose `n` is above the rows left gets a bitmap scan of the primary key index and a `Sort` of the rows after the seek under the `Limit` (PostgreSQL 16), while on the 100k-row table the index scan holds ten rows from the end; both are pinned by a test and stated in the module doc (PR 8). PostgreSQL's `fetch_page` pulls the page's rows from the connection one at a time and parses each as it arrives, so the page's text is no longer a term of section 7 (the record's 83 MB at the worst shape); the item channel and the decoded page are unchanged; with that change the PostgreSQL worst-shape run peaked at 790 MiB (`VmHWM`) in 16.5 s against the head's 905 MiB in 18.2 s, and its 1M run's peak was 4.0 MiB over the 100k run's (section 7, PR 8). `run_page_bounds` fails at once, naming the key yielded twice, when a stream has yielded more items than the table holds, so an inclusive seek fails in 3 s where it ran until another test's lock wait gave out (PR 8). SQLite's run-ahead is five rows, not four: sqlx's worker steps and copies the next row before its `send` on the four-row channel blocks, 160 MiB at the line cap (section 7, PR 8). The allocator: the peak over idle of the worst-expansion run grows with the number of large rows under glibc's default arenas and is flat under `MALLOC_ARENA_MAX=1`, which is documented for operators (the admin guide's `[backup]` rows, the deployment guide's service unit, section 7) rather than set from the binary; section 7's memory paragraph is the sizing rule (the terms at the line cap, about 490 MiB per job on PostgreSQL and 650 MiB on SQLite, times the allocator's factor, times `max_concurrent_jobs`), measured at 27.7 MB rows, 1,000 small rows first, 12, 24 and 48 such rows back to back, in two series of six runs each on one SQLite binary: 466, 588 and 760 MiB over idle with the default arenas and 481, 452 and 480 MiB under `MALLOC_ARENA_MAX=1` on a quiet host, then 707, 700 and 1,281 MiB and 452, 480 and 455 MiB on the final build with the kill-point runs beside it (the terms' sum at that row 545 MiB; runs of the PR's first build on a loaded host reached 1,096 MiB at 48 rows), which replaces this item's earlier "the peak did not move between 12 and 24 worst rows" (section 7, PR 8). The #618 cells run on PostgreSQL and SQLite; the RSS acceptance states its 112 MiB bound and MongoDB's 500k run; the testing summary says the backup kill points are run by hand with the posted script, a `before_manifest` point added (the three documents and the data file under the prefix, no manifest, FAILED `owner dead` and the prefix deleted as at `after_data_file`) and the SIGTERM exit timed by polling (55 ms on SQLite, not the script's 5 s sleep); 5.1 step 1 names the store-health check in its order, before the permit; the record is committed as `docs/design/15-backup-restore-bench-pr8.md` and cited by that path, with its SQLite cache explanation corrected to `cache_size = -32000` (31 MiB per connection, `sqlite_util.rs`) and its two 12-row rows labelled by build; the differences document's backup-creation figures are the record's and its restore-interruption row says the prefix of a SQLite row goes once the dead owner's heartbeat is stale, not in the first cycle; the usage guide's backups row says RestoreTableFromBackup restores in-catalog backups only this release and the upgrade manual says a store-backed backup is listed, described and deleted but its restore answers the ValidationException until the restore job lands (section 5.1, 7, 10, testing summary, PR 8). The harness: `tests/test_abac.py`'s backup cell waits for AVAILABLE before DeleteBackup; the job-bound cell's two diagnostics say what each case is; the fidelity suite reads `/health` once and checks that the server's `backup_store` component is the kind `EXTENDDB_TEST_BACKUP_STORE` names and healthy (absent for `none`), the `none` branch of the vector cell captures the ARN of a create accepted against expectation, `_failed_rows` decodes Rust's `Debug` escapes, a cell pins the whole key of a refused item with a UUID hash key and a ten-digit `N` range key in the log line and in `backup list --failed`; the CI start steps of the PostgreSQL, SQLite and S3 jobs wait for the `backup_store` component to be healthy, the pytest lines carry `-rP` so a passed cell's printed branch reaches the log, and the S3 job dumps moto's log on failure and names what its by-name step covers (PR 8). Also: `validate_restorable_item` takes `&RestorableLimits` (the item size, hash key size, range key size and attribute name size of `LimitsConfig`), not `max_item_size_bytes` alone, so the restore's "every item against the target's limits" (5.3 step 5) is the same call; every untrusted text its error carries (a key value, an attribute name, a path segment, the number the wire's reason echoes) is bounded on its own at 4096 bytes of the text the message writes (a key value at 12,288, six times the hash key limit, so a key of control characters is whole too; 4096 is a round figure that keeps a path segment to four times the longest key attribute name), with a 1024-byte prefix and the length for a longer one, so a key of any legal size appears whole in the log line and the FAILED reason; every such text, the attribute path and the echoed number included, is written as a JSON string's content, with U+0085, U+2028 and U+2029 escaped as `\u0085`, `\u2028`, `\u2029` beside the control characters, so neither a control character nor a Unicode line separator in a tenant's item can add a line to the server's log, for a byte-oriented reader and for one that breaks lines the Unicode way (Python's `str.splitlines`) alike; an item whose data file line would exceed `MAX_LINE_BYTES` under a raised `max_item_size_bytes` is refused by the job's feeder through the same count-and-log path (`RestorableItemReason::LineTooLong`, whose reason names the cap, the line's own length being unknown once the count stops), the line measured on the blocking pool for items over the default item limit only, through a counting writer that holds no line, and such an item's validation (the number rule over every member, CPU work proportional to the item's size) run on the blocking pool with it, an item within the default validated on the feeder's own task, so the encoder's own refusal is never the first to see it; a put that fails while the snapshot stream is between pages ends the pipeline at once (the uploader cancels a token the feeder selects on), so the FAILED write does not wait for the stream; the completion log line carries the stream's item count; a job the shutdown cancelled records `BackupJobs{cancelled}` and neither `WorkerLastSuccess` nor `WorkerErrorCount`; `BackupJobDuration` is microseconds from the snapshot's open to the end of the job's last catalog or store call (the AVAILABLE write, or the FAILED write and the prefix's deletion); the reconciler's `CycleReport` counts a dead-CREATING resolution whose write matched no row as left to its owner (`creating_left_to_owner`), and its cycle line counts the dead legacy restores and the FAILED-record targets apart; `reconciler.rs`'s module doc states the FAILED-prefix duty's timing as the code has it, in this cycle on PostgreSQL, MongoDB and Cassandra and once the dead owner's last beat is stale on SQLite (5.1 steps 4 and 5, section 9, PR 8). (2026-11-01, PR 8 as measured and corrected, the row bound's memory.) The sentence above that the over-read happens "once" was false on a table whose large rows are scattered among small ones: with the bound sized from the previous page alone, a large row set the next bound to 4, four small rows set the one after back to 1000, and that page met large rows again, so the 1000-row page recurred every other page for the whole table (5,555 rows read for 1,000 kept, 13.4 GB of server temp files, on a 1,000-row table with 30 such rows; on the release binary the same bytes took 1.9 times as long and 3.2 times the temp files by key layout alone). The bound is now sized from the largest row of the last pages that together hold at least a row bound's worth of rows (`RowBoundMemory`), so it stays at the floor while large rows keep arriving within that span and returns to the row bound a row bound's worth of small rows after the last large one: 1,901 rows read for 1,000 kept and 3.9 GB on the probe (a memory of four pages read the same 5,555 as one page, since four pages of four rows are sixteen rows), and the interleaved table's backup 54.6 s and 3.2 GB of temp against 72.0 s and 9.2 GB, the grouped table unchanged. The bound that remains, stated as it is: the page that follows a row bound's worth of small rows reads up to `n` rows of up to the line cap, at most once per row bound's worth of small rows, and nothing in the window form stops the server at the budget. A page query that does stop there, a recursive walk of the primary key one index probe per row, was measured and not taken: it read exactly the rows it returned on every shape (the interleaved table in 38.8 s with 2.5 GB of temp) and cost one probe per row, 23.1 s against 11.9 s for the 1M-row backup; the record carries both. `the_row_bound_remembers_large_rows_among_small_ones` pins the memory's figures on a 300-row table with nine such rows (201 rows read past the cut over 53 round trips), `the_page_query_is_an_index_scan_on_every_key_shape` pins the index scan on the six key shapes (hash only, `S`, `N` and `B` range keys, a multi-part range key, a legacy table on the database collation, each at `n` 16 and 1000, first page and a page after a key, and the stream over 20,000 rows), and `page_sql_for` takes the collation flag `open_stream` decides, so the legacy table's query is tested as the engine writes it (section 3, 7, PR 8). Section 7's rule is scoped: the terms are at the default `max_item_size_bytes`, where no stored item's line exceeds the cap, and on an instance that raises it they are at the largest line a stored item expands to; the feeder's measurement of an item over the default limit holds no line (a counting writer that stops at the cap), where the earlier text said its buffer was bounded by the cap and the code held the whole line (section 7, PR 8). The one-arena factor is measured on both backends: the PostgreSQL worst shape at 12 rows peaked at 340 MiB `VmHWM` under `MALLOC_ARENA_MAX=1` against 750 to 788 MiB under the default arenas, 0.82 of the terms at that row on PostgreSQL (413 MiB, no run-ahead), and the setting's cost on the request path is measured: 3.5 to 9.2% on SQLite (GetItem, Query, Scan, paged Scan) and about 2% on PostgreSQL (the scans, 2.1%), so the deployment guide's service unit carries the line commented out with the condition and the cost, and section 7 states both (section 7, PR 8). 4(e)'s unknown-outcome rule reads as `write_status` has it: each unknown outcome is followed by a re-read, the write is sent once more at most, and after the second write a re-read that shows the row in the target status counts as matched, anything else as matching no row; the review checklist's bullet says the same (4(e), PR 8). The PR 8 duty list marks the resume of a dead RUNNING restore as PR 9a's, this release's reconciler logging such a record once; section 10's table gains PR 8's rows (the admin guide's `[backup]` rows and `[backup.store]` section, the deployment guide's service unit, the upgrade manual's asynchronous-create and rollback paragraphs, the design README's row 15b); the PR 9a lead-in names `RestorableLimits`; the ratios are given to two places (0.86 to 2.35; 2.35 at 48 rows); the SIGTERM exit is 55 ms in every statement; the record attributes the PostgreSQL worst run's idle to the restarted process's startup work on a table of 27.7 MB rows, not to the 1M load (sections 4, 7, 10, PR 8, PR 9a). (2026-11-01, PR 8 as corrected.) Section 3's parenthesis on the PostgreSQL cut, the PR 8 bullet, its two paging acceptance cells and the Interfaces paragraph's `n` sentence described the recursive walk that was measured and not taken (one row per step, no row read past the cut, two index probes) and the rule before the memory (`n` from the previous page's largest row, the over-read "once"); each is corrected to the shipped form: the three-layer window page with up to `n - 1` rows read past the cut, `n` sized from the memory, the first paging cell naming the record's `EXPLAIN (ANALYZE, BUFFERS)` figure (4 read, 1 past the cut at the floor) and `the_row_bound_remembers_large_rows_among_small_ones` (201 over 53 round trips), the `EXPLAIN` cell the plans the tests pin (`Limit` over one `Index Scan`; the bitmap scan with a `Sort` near the end of a small table) (section 3, PR 8). The over-read's statement gains the second page, which follows the first page's 16 rows and asks for the row bound: the over-read that remains is the second page, and then the page that follows a row bound's worth of small rows, each up to `n` rows of up to the line cap, said the same way in the PR 8 page paragraph, the module doc, the test's doc and the record (the earlier sentence of this item said "the page that follows a row bound's worth of small rows" alone); the floor's cost is stated beside it: after a page holding a row at the line cap the bound stays at the floor for a row bound's worth of rows, 250 pages of four, whatever those rows hold, so a table whose large rows are rarer than one per row bound pays 250 round trips per large row where one page of memory pages it at the row bound (2,585 page queries against 38 on a 20,000-row table with a 27 MB row at every 1,100th key, 25.6 against 25.1 s, 1.06 against 1.09 GB of temp; the record), and the record's "changes no page of a small-row table" reads "of a table with no large row"; the one-page-memory figure the test's doc, the record and the PR text cite is 306 rows past the cut over 7 pages (section 7, PR 8). 4(e) said the PostgreSQL reconciler "reads the heartbeat alone and never probes the lock" and named "the one liveness rule the three share": PostgreSQL's `claim_dead_backup` tries the advisory lock before it tests the heartbeat, so it takes over only a row whose heartbeat is stale and whose lock is free; a stopped owner (SIGSTOP, a paused VM) whose session lives keeps its row until the session ends or it finishes, and a vanished host's lock is released when the server's keepalives give up (about a minute with `CLAIM_KEEPALIVES`), so the takeover waits for the later of the two; 4(e) and 5.1 step 4 now say so (the SIGSTOP sentence scoped to MongoDB and Cassandra, with the PostgreSQL outcome: no takeover while the session lives; on resume the job completes under its own generation or, when the stop outlasted two heartbeat periods, stops as a lost claim with no catalog write and the reconciler resolves the row), the trait doc of `claim_dead_backup`, the admin guide's `stale_job_after_secs` row and the differences document's dead-owner clause gain "on PostgreSQL, once the owner's claim session has ended as well", and item 27's "the heartbeat alone" stands for PR 6's `remove_dead_creating_backup_row`, which `backup purge` still uses (4(e), 5.1 step 4, PR 8). The one-arena cost on PostgreSQL reads "about 2% (the scans, 2.1%)" in section 7, this item, the admin guide, the deployment guide and the PR text, the record's scans being −2.1%; section 7 labels the base of each ratio (the peak over idle on SQLite, the `VmHWM` on PostgreSQL, about 0.47 over idle); the admin guide's `max_concurrent_jobs` row restates section 7's sizing rule (1.2 times the terms per job under `MALLOC_ARENA_MAX=1`, no bound measured under the default arenas, the PostgreSQL figure as `VmHWM`, 2.35) and its `stale_job_after_secs` row the dead-row outcomes (AVAILABLE with a complete manifest, FAILED without, a restore job's target removed); the upgrade manual's rollback paragraph names `backup sync --adopt-store`, as 4(c) requires (section 7, 10, PR 8). `EXTENDDB_TEST_BACKUP_STORE` is set by every CI job that runs a suite creating backups, the Rust integration jobs included: the SQLite Rust integration job's file-backed server has the defaulted root, so its CreateBackup is the asynchronous one, and `tests/rust`'s backup tests assumed the synchronous create and left their backups; the suite now waits for AVAILABLE, deletes what it makes and reads the variable as the Python suite does, and `run-tests`' usage, `CONTRIBUTING.md` and the developer test guide name the variable and the refusal (section 10, PR 8). The PR 8 kill-point list names `before_manifest`, and section 10's limits-doc row the rewritten "Backup store" row (PR 8). 5.1 step 6 says a job cancelled after its manifest put completes AVAILABLE, the manifest being the commit record, where it said a cancelled job can never complete (5.1 step 6, PR 8). On PostgreSQL a job stopped and resumed after more than two heartbeat periods stops as a lost claim with no catalog write, its session's end frees the advisory lock and the reconciler resolves the row, where 5.1 step 4 said it resumes and completes; the unknown-outcome rule's second write is stated as the rule of a backend whose claim connection survives the outcome (MongoDB, Cassandra), PostgreSQL's failing with the session; the reconciler's failed `get`, `list` and `delete_prefix` calls are `BackupStoreErrors` as the job's are; and `extenddb stop` waits for the sum of the server's shutdown bounds, the 30 s job drain included (4(e), 5.1 step 4, section 9, PR 8). + +30. (2026-11-01, PR 9 as implemented — the design's PR 9a and PR 9b as one PR, the maintainer's decision.) The restore job runs from the store on PostgreSQL, SQLite, MongoDB and Cassandra, same-backend and cross-backend, from a filesystem or an S3 store; the matrix and the nightly are PR 9b's, shipped here. `RestoreTableFromBackup` of a `backup_index` row no longer answers the "in a backup store; this release restores in-catalog backups only" refusal `check_catalog_row_for_restore` carried; a healthy configured store's AVAILABLE row is restored, and the refusal is gone from the engine and the differences, usage and upgrade docs (5.3, section 10, PR 9a). The interface: `RestoreEngine`, a new trait in `StorageEngine`'s bound with `bulk_load(target, claim, BulkLoadFile { file_index, start_at, reapplying }, items) -> applied`, `finish_restore(target, claim, item_count, size_bytes) -> Finished { Active, Pending, NotMatched }`, `count_restore_target(target) -> u64`, `read_restore_progress` (default `[]`) and `sweep_restore_progress(running) -> u64` (default 0); `StorageEngine` has no methods of its own, so `bulk_load` is not "a new `StorageEngine` method" (section 3's PR 9a interface list). `BackupCatalog` gains `claim_restore(account_id, input: CreateTableInput, &RestoreRegistrationRow) -> (TableDescription, Box)` (the registration creates the CREATING target and returns the description the request answers; the record is born owned under the backend's generation, `RestoreRegistrationRow.owner_generation` being `UNOWNED_GENERATION` and the backend minting its own), `claim_restore_dead(key, stale_after, spool_host) -> Option>` (the restore analogue of `claim_dead_backup`, writing the claimant's `spool_host`, a fresh generation and a fresh `heartbeat_at` on the database's clock in one conditional write, where the backup takeover leaves the heartbeat: nothing waits on a restore record's old heartbeat after a takeover, and a second taker must find the record live at once, 4(e)), `read_restore_job`, `list_restore_jobs(Option)` and `remove_restore_target`/`fail_restore_job` as the reconciler uses them; `RestoreJobRow` carries `store`, `spool_host`, `file_index`, `items_applied`, `failure_reason`, `completed_at`, so the FAILED-target duty reads a row's reason and the sweeps read its store and `spool_host` (section 3, 4, 5.3, PR 9a, PR 9b). The design's `claim_restore(&RestoreJobRow) -> Box` and `record_progress(&dyn JobClaim, file_index, items_applied)` are gone: there is no `record_progress`; every backend records progress inside `bulk_load`, in the write that applied the items (PostgreSQL a `restore_progress` row in the file's data transaction, SQLite and Cassandra a `tables`-row update per batch, MongoDB a `restore_job.file_index` update per file), and the design's "conditioned on the generation" holds on SQLite, MongoDB and Cassandra but on PostgreSQL the progress row lives in the data database, which holds no job record, so the fence there is the base table's primary key (a file applied twice fails on its first key, a recorded file on the progress row's own key) and the claim's liveness, which the engine confirms before each file and whose loss, the heartbeat's, aborts a file in flight; the generation fences only the record's own writes in the catalog (DONE, FAILED, the takeover) (5.3 step 5, PR 9a). `bulk_load` carries `reapplying` so MongoDB's duplicate suppression is explicit: on a resume the first file the engine meets is `reapplying = true` and every later file false; MongoDB suppresses an all-E11000 `insert_many` only when `reapplying`, else "the backup holds a key twice"; PostgreSQL and SQLite never suppress (their base rows are plain `INSERT`s, a duplicate fails the file whatever the flag, and SQLite has no unrecorded partial write, its progress being in the batch's transaction); Cassandra's `put_restored_item` is an upsert, so a duplicate collapses and `count_restore_target` one short of the manifest's total is the only duplicate check there (5.3 step 6, PR 9a). The restore ends by counting the target against the manifest's item total on every backend, so an unordered backup holding one key in two files and a key-range check that missed are both caught; a key-ordered file runs #394's per-file `KeyRange` check, which ends the file's stream at an item outside its range. The migration for PostgreSQL's `restore_progress` is id 6 (`MigrationId::RestoreProgress`), a `StoreKind::Data` body on PostgreSQL (`data_migrations/0006_restore_progress.sql`), `NotApplicable` on SQLite, and nothing on MongoDB and Cassandra, whose bootstrappers have no `MigrationId` table; this corrects "data migration 005 for `restore_progress`" in the PR 9a section (its acceptance line and its load paragraph) and the PR 6 line on reserving numbers; item 14's "data migration 005 reserved" stands as the history it is (4(i), PR 6, PR 9a). The manifest registration locks the `backup_index` row (the lock `start_backup_delete` takes: PostgreSQL `SELECT ... FOR UPDATE`, SQLite the `BEGIN IMMEDIATE` write transaction, MongoDB `$inc lock_seq` on the document, Cassandra the `active_restores` lightweight transaction on the backup's partition), while the legacy registration keeps the `backups` row; a legacy and a store-backed restore into one name exclude each other (PostgreSQL the advisory `restore_lock_key`, which both paths take; on SQLite the store-backed path's `restore:/
` slot and, for a legacy restore, the `tables` row under the write transaction), so a `DeleteBackup` of a `backup_index` row meets only store-backed registrations and the two orderings resolve one before the other (DeleteBackup wins → the registration is `BackupNotFound` and no target; the registration wins → DeleteBackup is `BackupInUse` and the row stays AVAILABLE) (5.2, 5.3, PR 9a). The restore's ACTIVE flip sets the table and every secondary and vector index of the target in one catalog transaction on PostgreSQL and SQLite (an index of a restore target is created ACTIVE, and `activate_restore_target` sets ACTIVE any `indexes` or `vector_indexes` row that is not), on MongoDB the job writes DONE and `restore_backfill_pending` and the backfill worker flips the table ACTIVE once every index has left CREATING (`finish_restore` returns `Pending` there), and on Cassandra the one lightweight transaction flips the table alone, its index rows created ACTIVE (`Active`, never `Pending`), so a Query on each GSI and `SearchVectors` answer as soon as the target reports ACTIVE (5.3 step 6, PR 9a). The request bounds its wait for the detached registration at `RESTORE_REGISTRATION_TIMEOUT` (60 s) on every backend; past it it answers `InternalServerError` while the task goes on and the target appears through `DescribeTable`, so the differences document's MongoDB-only sentence becomes every backend's (5.3 step 4, PR 9a). The maintenance check runs after the registration on the manifest path (5.3 step 4 as written) and before the permit on the legacy path (item 28), both orders in `restore_table_from_backup` and `restore::start`; the restore's first confirmed beat stands at the instant `claim_restore` returned (the record's `heartbeat_at` was stamped inside the registration, before the return, so the instant is at or after the beat it stands for by the registration's remaining duration, which each backend bounds, 4(e)), where the backup job takes the instant before its claim's insert (5.1 step 4), so a registration that took longer than two heartbeat periods does not end the job at its first check; a held claim on the manifest path leaves a FAILED `maintenance` record whose target the job removes before it answers `LimitExceededException`. Only `StoreError::Transport` is retried on a data-file download, for up to `DOWNLOAD_RETRY_BOUND` (15 min, counted from the job's first download attempt, every file's retries against it) with `min(2^n s, 60 s)` plus up to a second of jitter, so the retry covers S3 stores (a filesystem store's `get` errors are `Io` and fail at once); `NotFound`, a denial, and a digest or size mismatch fail at once naming the file. On a filesystem store "verified in place" is two `get` passes, a hashing pass before any load and a decoding pass through `ExtenddbManifest::reader` that hashes again (so a file replaced between the passes fails in the load) with no spool; on an S3 store the files are spooled as `restore-
/.json.gz` with `.partial` → `sync_all` → rename → `.verified` (the marker holding the file's sha256, compared with the manifest's before the file is trusted; an entry left by a run of this host for a record another host then owned is removed before use) and read through `reader_prehashed`, the spool at `spool_path` (the defaulted `backup-spool/` on file-backed SQLite). A restore the shutdown stops records `RestoreJobs{cancelled}` and no worker metric, keeps its spool and its progress and writes no status (the `SHUTDOWN` reason is a backup's alone); a lost claim or a DONE write that matched no record records `RestoreJobs{failed}` and `WorkerErrorCount{RestoreJob}` with no catalog or table access; `fail_restore` writes FAILED then runs `remove_restore_target` only when that write matched. A job that learnt the record is another's (the lost claim, the DONE write's `NotMatched`, `fail_restore`'s no-table-access arm) leaves its spool entry to the sweep, since a same-host successor may be reading it (the entry is keyed by table id); every other end but the shutdown's removes it (4(e), 5.3 step 7, PR 9a). The load hands the backend one `bulk_load` per file over a stream fed in chunks of at most `LOAD_CHUNK_ITEMS` (1000) items or `LOAD_CHUNK_BYTES` (32 MiB of data-file line text) through a channel of two chunks; a restore holds the parsed manifest, one decoded line and its parse, the decoded items of four chunk positions and the backend's batch (32 MiB of item text on SQLite and Cassandra, 4 MiB per INSERT on PostgreSQL, 1,000 documents on MongoDB), which is section 7's restore sentence, with the figures measured on this PR's final tree (`docs/design/15-backup-restore-bench-pr9.md`: 848 MiB over idle on SQLite and 778 MiB on PostgreSQL at 24 worst-shape items under the default arenas, 392 and 427 MiB under one arena; 89 and 16 MiB at 1M key-ordered items of 200 bytes, in 26.8 s and 9.5 s, 144 to 164 and 16 MiB for the same items unordered), which replace the earlier sentence, "A restore holds one line's decode plus one insert batch plus the parsed manifest for the whole job; neither grows with the table": the terms stand, the figures are measured, and a restore of large items is of the backup's order. The metrics are `MetricName::{RestoreJobs, RestoreJobDuration, RestoreBytesDownloaded}` with the outcome in the operation dimension, `QuerySource::RestoreJob` for the worker metrics, and `BackupStoreErrors{get}` on every failed `get` of a restore job (the request's and a resume's two document reads and each download attempt the store failed, retried or not; a digest mismatch and the spool's own errors are not the store's); section 9's `restore_jobs_total{outcome}` and `restore_job_duration_seconds` are a converter's spellings, as item 29 said for the backup names (section 9, PR 9a). A backend's refusal of an item in `bulk_load` is not a wire exception: on MongoDB an `N` sort-key value past Decimal128's 34 significant digits is the write path's `ValidationException`, which becomes the restore's FAILED reason, and the client sees the target disappear; a 38-digit non-key number is stored as text and restored intact on every backend. The reconciler resumes a dead store-backed restore from its recorded progress (per file on PostgreSQL and MongoDB, per batch on SQLite and Cassandra) under `claim_restore_dead` once the heartbeat is stale (on SQLite at once after a restart, by the record's slot; on PostgreSQL once the owner's claim session has ended as well), re-downloading to its own spool when the dead owner's `spool_host` is not this host's; it fails a dead manifest record under a maintenance claim before it looks at the record's store (`OWNER_DEAD`, the target removed; `target_not_empty`, the target left for `remove-target`, when it recorded no progress and the target holds items), fails a record of a store other than the configured one or of no store with `OTHER_STORE` (naming `backup sync --move-store`, the target left to the FAILED-target duty), skips a resume without a permit, and FAILs a resume with no recorded progress into a non-empty target as `target_not_empty`, left for `extenddb backup remove-target --account
` (PR 10's command, shipped with PR 9, below); its two sweeps (`restore_progress` rows of tables that are not RUNNING restore targets; spool entries of restores not RUNNING with this host's `spool_host`) run only without the maintenance claim and read `list_restore_jobs(Some(RUNNING))`, a backend that cannot list them logging once and skipping both (section 4, 5.3 step 7, PR 9a). A store-backed backup of a multi-part-key table is written `unordered` and restores on every backend, the target laid out as a natively created table (`bulk_load` keys rows with the backend's own key functions), while a legacy backup of one keeps the reader's refusal; the #384 override tests run through both the legacy and the manifest paths for every case with a definition and assert the same input or the same refusal, the manifest path deriving the SSE specification from the manifest's `sse_description` where the legacy path keeps the specification as written. The cross-backend matrix (PR 9b): a cell copies the source's backup prefix into a root the target catalog owns and inserts the target's `backup_index` row as `backup sync --adopt-store` will (PR 10), so the cell proves the restore and not the adoption; it runs from the committed `devtools/backup-restore-matrix`, and `.github/workflows/matrix-nightly.yml` (a `schedule` cron and `workflow_dispatch`) runs the twelve plain cells and the duplicate-key and vector cells nightly, building the four backend binaries and starting PostgreSQL, a MongoDB replica set and Cassandra as the integration workflows do; the seed stands in for `backup sync --adopt-store` until PR 10 replaces it. The restore kill points are run by hand with the `kill-points.py` posted with PR 9 (outside the repository, as PR 8's was), one mode per backend where it applies: `restore_after_download`, `restore_after_file_commit`, `restore_sqlite_mid_batch`, `restore_cassandra_before_progress`, `restore_mongodb_after_insert_many`, `restore_postgres_before_progress_delete`, `restore_mongodb_before_worker_flip`, `restore_between_failed_and_removal`, `restore_fenced_out`, `pg_terminate_backend` and `sigterm`, each a SIGKILL or a signal at the named point with a restart and the outcome asserted (the exact count after a resume, or the target gone for a failing restore). Two points pause before the first progress record of a file or batch (`restore_mongodb_after_insert_many`, `restore_cassandra_before_progress`), so a kill at their first reach leaves written items and no progress and the resume answers `target_not_empty` (the rule of 5.3 step 7, asserted); the re-application path (MongoDB's E11000 suppression of the one unrecorded file, Cassandra's re-applied batch) is run by killing at the point's second reach, after an earlier file's or batch's progress is recorded (the script's `--second-reach`): on MongoDB a 24,000-item backup in three files resumed from file 1 with file 0 recorded and reached ACTIVE with the exact count (file 1's 8,227 items re-applied as duplicates, 7,544 applied by the resume); on Cassandra a one-file backup of 12,000 items killed before the second batch's progress write resumed from item 10,000 and reached ACTIVE with 12,000 (2,000 applied by the resume), and `restore_after_file_commit` resumed from file 1 to the exact count. A Cassandra server's CreateBackup is the in-catalog format (section 3), so a Cassandra kill point takes its backup on a SQLite source, copies the prefix under the Cassandra target's root and seeds the target's `backup_index` row as the matrix does; the single-node Cassandra of the harness needs `replication_factor = 1` in the server's configuration, as `init` writes it, since under the default of 3 every lightweight transaction and quorum read on a keyspace the server creates answers "No nodes available in query plan", which is what had blocked these points. The feature chain for the backends' pause points: `extenddb-app/backup-test-hooks` → `extenddb-server/backup-test-hooks` now pulls `extenddb-storage/test-hooks` beside `extenddb-engine/backup-test-hooks`, so a hooks build compiles `extenddb_storage::backup_catalog::restore_pause_point` and each backend's own point (`restore_sqlite_mid_batch`, `restore_cassandra_before_progress`, `restore_mongodb_after_insert_many`, `restore_postgres_before_progress_delete`, each a `pub const` beside `restore_pause_point` in `extenddb_storage::backup_catalog`, the one spelling the backends and the kill-point script use) in without naming a backend crate (the server depends on none directly); the engine's five restore points are its `KillPoint` variants (`RestoreAfterDownload`, `RestoreAfterFileCommit`, `RestoreMongodbBeforeWorkerFlip`, `RestoreBetweenFailedAndRemoval`, `RestoreFencedOut`), and that enum is the engine's list, not the whole list. Corrections made before PR 9 merged: (1) the restore takeover writes `heartbeat_at = now()` in the one conditional write with the generation and `spool_host`, on every backend, so a second taker finds the record live at once; the backup takeover keeps leaving it (4(e) states why the rules differ); (2) `LOAD_CHUNK_BYTES` is summed from the decoded line's length (the decoder's `bytes_decoded` across one item), not `item_size_bytes`; section 7's terms are restated in that unit; (3) the download retry bound is per job: `download_all` takes the job's download start once and every file's retries count against it; the module doc, 5.3 step 3 and this item say so; (4) a job that learnt the record is another's (a lost claim, a DONE or FAILED write that matched no record) leaves its spool entry to the reconciler's sweep; (5) the `.verified` marker holds the file's sha256 and `is_verified` compares it with the manifest's; `Entry::open` removes an entry whose record's `spool_host` was another host's before use; the resume reads the manifest from the spool only when the record named this host before the takeover (it read the record after the takeover, which had just written this host's name); (6) the decoder checks the heartbeat's pre-write rule (`confirms_the_claim`) at every chunk boundary beside the shutdown token; MongoDB and Cassandra re-read the record per batch; (7) a restore's `claimed_at` is the instant `claim_restore` returned, the record's `heartbeat_at` having been stamped inside the registration. Cassandra's takeover re-applies the `active_restores` entry when absent; the SQLite and Cassandra batch is bounded by `RESTORE_BATCH_BYTES` (32 MiB of item text) beside `RESTORE_BATCH_ITEMS` (10,000), both in `extenddb_storage::restore_engine`; MongoDB and Cassandra read the record before each write batch; `sweep_restore_progress` deletes from a positive list; the backends' pause-point names are `pub const`s beside `restore_pause_point` in `extenddb_storage::backup_catalog`, one spelling each (`RESTORE_SQLITE_MID_BATCH`, `RESTORE_MONGODB_AFTER_INSERT_MANY`, `RESTORE_CASSANDRA_BEFORE_PROGRESS`, `RESTORE_POSTGRES_BEFORE_PROGRESS_DELETE`), and the engine's `KillPoint` is its own list; Cassandra's `count_restore_target` reads the catalog first and answers `TableNotFound` for a key that is not a CREATING restore target with that id, as the other three do; the MongoDB registration stamps the record's heartbeat from the server's clock inside its transaction (the registering process's clock is replaced; the test registers from a process an hour behind); a duplicate primary key fails the file with `data file : holds a primary key the target already has (the backup holds a key twice): `, the engine naming the file by its key once and the backends naming no file; SQLite's startup line for an aged RUNNING record names the reconciler's duty per kind. The reconciler's store check runs first in the cycle, so the manifest arm reads this cycle's health, and the skip line names `not yet checked` for a store whose first check has not recorded; a maintenance-held dead manifest restore is counted once, in `manifest_restores_failed_and_removed`; `RestorableKeySchema::resolve` runs in the request with step 2's refusals; the request's second table-name check and second `check_supported` are gone: the parser's identity rule and version gate run first; the engine's kill points log `job reached kill point `; each backend's own point is named beside `restore_pause_point`; `finish_restore`'s `Pending`, the DONE write's `NotMatched`, the pre-write claim checks of `run`, the download's pre-attempt check and `WorkerErrorCount{RestoreJob}` on a FAILED job have engine tests; the #384 override tests run through both paths for every case with a definition. The wire test `test_create_backup_of_a_vector_table_is_refused` is `test_vector_table_backup_refused_in_catalog_and_restored_from_store`, since it asserts the round trip on a server with a store; the multi-part-key wire cell lives in `tests/test_backup_job.py` on a deployment that enables `allow_multipart_table_keys`, compares the restored item set with the source's `Scan` and makes no `GetItem` by the whole key (the preview's item paths address such a table by its first HASH attribute on every backend: a `GetItem` by the whole key finds nothing on PostgreSQL and the first HASH key's one item on SQLite, restored or not). Corrections made before PR 9 merged, second set: `extenddb backup remove-target --account
` ships with PR 9 rather than PR 10 (the PR 10 section's clause says so): the `target_not_empty` rule was written for a catalog restored from a dump that names a restore which finished and was written to since, but its common trigger on MongoDB and Cassandra, which record progress after the write, is an ordinary crash before the first progress record (the whole first data file on MongoDB, the first 10,000 items or 32 MiB on Cassandra: the kill leaves written items and no progress), and without the command this release had no way to remove such a target (DeleteTable refuses a restore target, the FAILED-target duty skips the reason, DeleteBackup and a second restore of the backup are `BackupInUseException`); the first-reach runs of `restore_mongodb_after_insert_many` and `restore_cassandra_before_progress` are that state, so the PR 9a acceptance line, the PR 9a load paragraph, the testing summary and the differences doc's restore-interruption row name it as the third outcome beside "finishes with the right count" and "gone", where they had named two; the command is the one section 4's duty list describes (the maintenance claim, a FAILED record's target whatever the reason, a RUNNING record's dead target, a legacy record's target, the three refusals, no store needed, the audit line), its tests are the engine's maintenance tests and one wire cell on SQLite seeded into the state at the catalog, and the two first-reach kill points run it and assert the target gone and DeleteBackup allowed. The spool sweep lists the spool before the records (the design's rule, which the shipped order broke); the spooled manifest and sidecar are written as the data files are and a spooled pair that fails verification yields to the store's; the resume's first confirmed beat is the takeover's return, taken beside `claim_restore_dead`; a `finish_restore` error is an unknown outcome re-read through the claim; `BackupStoreErrors{get}` counts every `get` the store failed; the request's and the resume's document reads are bounded (4 KiB, 16 MiB); the restored table's size is the source table's; `RestorePostgresBeforeProgressDelete` is gone from the engine's `KillPoint` (one spelling each holds: the four backend names are the storage crate's constants and nothing else); a registration that fails after the request stopped waiting is logged. The Cassandra write path skips an index row's DELETE when a PutItem/UpdateItem leaves the index keys unchanged, so a re-put rewrites the GSI/LSI row instead of deleting it (#742); the restore's re-applied batch relies on it, and the `restore_cassandra_before_progress` second-reach run now reaches ACTIVE with the GSI count exact where before #742 was corrected it was one batch short. PostgreSQL's `finish_restore` deletes the progress rows after the flip's commit and does not fail the restore if that delete errors (logged and swept). A file's PostgreSQL progress row is a plain INSERT from `start_at == 0` and an upsert from `start_at > 0`. Two concurrent registrations of one backup leave one target on Cassandra (the registration LWT's `active_restores = null` is the fence). What bounds a registration's duration per backend (the advisory/`FOR UPDATE` lock, the slot, `transactionLifetimeLimitSeconds`, the LWT timeouts), so the `claimed_at`-minus-`heartbeat_at` gap can never reach the stale window, replaces the earlier "conservative" in 4(e) and in this item, which had the direction wrong: an instant later than the stamp makes the pre-write check admit a beat older than two periods by the gap, so the gap needs a bound, not a name. The SQLite restore's published memory figures are the no-index shape; a table with indexes costs SQLite's page cache (`cache_size`) plus arena fragmentation, bounded, not the restore's buffers (the record's index table: 143 to 225 MiB at 100k and 165 to 172 MiB at 300k with two GSIs on SQLite, 23 to 34 MiB on PostgreSQL), and the 60,000-item shape of the differences doc's in-catalog timings was measured store-backed on PostgreSQL, SQLite and MongoDB (0.67, 1.85 and 3.74 s to ACTIVE), the row now labelling which path each figure is. The MongoDB CI jobs' `EXTENDDB_TEST_BACKUP_STORE` is `filesystem` from this PR (`devtools/run-mongodb-tests` serves a filesystem root), where it was `none`; the Cassandra job's stays `none`; the runner's `--backup-store none` is the one way to run the legacy LSI fallback cell, the in-catalog restore path's one wire cell on MongoDB, which no CI job runs. The PR 11 line owns the two unmeasured cells (the MongoDB and Cassandra 1M restores, the vector cell) and no longer the nightly schedule, shipped here. The kill-point cells as run on the pushed head's hooks binaries (4c353191), every ACTIVE outcome asserting the base table's and the GSI's `Scan` counts: `restore_after_download`, `restore_after_file_commit`, `restore_between_failed_and_removal` and `sigterm` on all four backends; `restore_sqlite_mid_batch`; `restore_postgres_before_progress_delete`, `restore_fenced_out` and `pg_terminate_backend` on PostgreSQL; `restore_mongodb_after_insert_many` at its first reach (`target_not_empty`, then `remove-target`) and its second (the re-application, 24,000 items in three files); `restore_mongodb_before_worker_flip`; `restore_cassandra_before_progress` at its first reach (`target_not_empty`, then `remove-target`) and its second (12,000 items, the GSI exact); the Cassandra points with a SQLite source. Rerun on the build after the last corrections (the first beat before the reply, the Cassandra re-read and fence, the PostgreSQL vector drop): `pg_terminate_backend` as reworked (the lost claim at the next check 20 s after the termination, the record RUNNING with its `(0, 8228)` progress row, the takeover 45.6 s after the termination and 46.7 s after the last beat, ACTIVE 24,000 on base and GSI), `restore_after_file_commit` and `restore_mongodb_before_worker_flip` on MongoDB (resumed from data file 1 after 0 of its items, 3,771 applied by the resume, DONE then the worker's flip, 12,000 exact; DONE before the kill and the restarted worker's flip after it), `restore_cassandra_before_progress` at its second reach (resumed after 10,000, 2,000 applied, 12,000 on base and GSI) and `restore_between_failed_and_removal` on PostgreSQL with a vector backup (the target gone after the stale window, its data table and `_ddb_vec_` relation gone with it, where the pushed head's build left the relation). The matrix's four `--duplicate-key` and eight `--vector` cells ran on the final binaries. Corrections made before PR 9 merged, third set. The job's first heartbeat is sent right after `claim_restore` returns and before the reply, and the pre-write check counts its two periods from that beat's return; a `Lost` or failed first beat fails the record ("the claim was lost before the job started") and answers `InternalServerError`. `extenddb backup remove-target` decides a RUNNING owner dead through `list_dead_restore_jobs(owner_window)` on the database's clock, and the purge's wait decides a backup row's owner live through `list_backup_rows_older_than` on the heartbeat clock; neither reads a host clock. The reconciler takes nothing over and writes no FAILED on its manifest arm once the server's shutdown has begun. Under a maintenance claim a dead record with no progress over a target that holds items is failed `target_not_empty` through `fail_restore_job`, not `OWNER_DEAD`. The reconciler's `target_not_empty` lines print `extenddb backup remove-target --account
`. The restore job's stop line names the heartbeat timer's reason when the timer raised the loss, and the timer logs the reason at WARN. A DONE write whose answer was lost and whose record re-reads DONE under the job's generation ends with "complete (DONE)", not "ACTIVE". A spool document write that fails and a resume that fails before it runs remove this host's entry. The engine's tests of the 60 s bound, the slot held to the job's end and the re-reads' generation checks are cells of their own. Cassandra's progress lightweight transaction re-reads an unknown outcome (applied, one retry, or the lost claim; a second unknown outcome fails the restore), where the earlier text said the restore fails and the reconciler resumes it, which the FAILED write's removal made false. The Cassandra registration's refusal path conditions the target's removal on its own generation; a record a reconciler took over during the DDL keeps its target. PostgreSQL's `abort_restore` drops a vector index's relation with the data table. The PostgreSQL record's `heartbeat_at` is the registration transaction's start. Tests: `test_an_unknown_progress_write_outcome_is_decided_by_a_re_read`, `test_a_registration_taken_over_during_its_ddl_leaves_the_successors_target`, `test_a_mid_file_resume_records_start_at_in_every_batch_progress`, `remove_restore_target_drops_the_vector_relation_with_the_data_table`. The engine's `KillPoint` has five restore variants, named above. SQLite's takeover clock is `strftime('%Y-%m-%dT%H:%M:%fZ', 'now')`, the `NOW` constant of its catalog (4(e) names it). `extenddb backup remove-target` refuses a RUNNING record with a live owner by its own text ("stop it or wait for it"); the DONE and no-record refusals are the two "not a restore target" ones. The published 1M figures are the key-ordered shape's and the record says so beside them, with the same items measured as an unordered backup (SQLite 144 to 164 MiB over idle in 66 to 90 s against 87 to 98 MiB in 27 to 56 s key-ordered; PostgreSQL 16 MiB either way), and SQLite's final `COUNT(*)` is the counting connection's page cache, bounded by `cache_size` (32 MiB) and the same size at 1M and at 3M rows (22 to 28 MiB), where the record had called it the one term that grows with the table from a single measurement. The `pg_terminate_backend` cell of `kill-points.py` pauses the job at `restore_after_file_commit`, terminates the claim session while it is paused (psql must answer a row), waits past one heartbeat period, releases, and asserts the lost-claim stop line, the record RUNNING with its progress row, the takeover no earlier than `stale_job_after_secs` after the owner's last beat and the exact counts; the shipped cell had terminated the session after a fixed sleep and asserted the final count alone, which a 2.6 s restore satisfied without the case ever running. The script's `--vector` switch gives the source table a vector index, and the tamper cell on PostgreSQL then asserts the removed target's data table and `_ddb_vec_` relation gone. `.github/workflows/matrix-nightly.yml` installs `postgresql-client` and checks `psql --version` and the two drivers' import before the cells; it has had no run, since `workflow_dispatch` needs the file on the default branch, so its first run after the merge is the proof of the runner image. The upgrade manual counts migrations 1 to 5 as breaking and migration 6 as declared breaking on PostgreSQL for want of the additive-declaration job (on SQLite it does not apply). The body sentences each of these corrects are edited with this item (section 1's duplicate rule and Decimal128 bullets, section 3's PR 9a interface list, 4(b)'s progress members and metric name, 4(e) and 4(i), 5.2, 5.3's preamble and steps 1 and 3 to 7, section 7's restore memory sentence, section 9, section 10's table, the PR 6, PR 8, PR 9a and PR 9b sections, the PR 10 clause naming what the seed stands in for and the command shipped here, the PR 11 line, the testing summary and the review checklist); earlier items keep their wording, as item 22 requires. With the rebase onto #666's head (2026-11-03): the jobs and the reconciler are not gated by the PostgreSQL serving registration, and 4(e) says what they do in the registration's retry window, GitHub issue #755 being the gate; `backup purge`'s own cycle under the SQLite serve lock fails a dead store-backed restore and removes its target rather than resuming it (4(f)), since the command tells the cycle it holds the claim instead of the cycle reading it through a zero window; the PR 9a load paragraph gives PostgreSQL's batch as 500 rows or 4 MiB, as 5.3 step 5 does; and MongoDB's snapshot open waits, at most 2 s, for the majority-committed snapshot to show a table a `local` read shows ACTIVE, so a store-backed CreateBackup sent as soon as DescribeTable says ACTIVE is not refused as a table that does not exist. diff --git a/docs/design/README.md b/docs/design/README.md index cc403a0ac..e419a4c83 100755 --- a/docs/design/README.md +++ b/docs/design/README.md @@ -21,6 +21,7 @@ This directory contains the authoritative design documents for ExtendDB (extendd | 13 | [MongoDB Storage](13-storage-mongodb.md) | MongoDB storage backend: engine and catalog traits over the official driver, replica-set transactions | | 15 | [Backup and Restore](15-backup-restore.md) | Overview of the off-host backup series: the invariants, the per-operation contracts and a short section per later PR; the format, store and snapshot documentation lives with their crates (`crates/core/src/backup_format`, `crates/backup-store`, `crates/storage/src/snapshot.rs`) | | 15b | [Backup memory and duration record, PR 8](15-backup-restore-bench-pr8.md) | The measurements section 7 and revision item 29 of the design cite: peak RSS and duration of the 100k, 1M, worst-expansion and incompressible backups per backend, the allocator series, the PostgreSQL page's cost | +| 15c | [Restore memory and duration record, PR 9](15-backup-restore-bench-pr9.md) | The measurements section 7 and revision item 30 of the design cite for the restore: peak RSS and duration of the 100k, 1M, worst-expansion and large-item restores on SQLite and PostgreSQL, with and without one allocator arena, the terms the code bounds and the sizing sentence | ## How to Use These Docs diff --git a/docs/differences-from-dynamodb.md b/docs/differences-from-dynamodb.md index f69e669bc..cbb12a379 100755 --- a/docs/differences-from-dynamodb.md +++ b/docs/differences-from-dynamodb.md @@ -87,11 +87,11 @@ adaptation when switching between ExtendDB and the real service. | Vector index update propagation | Eventually consistent, the same model as a GSI | Matches DynamoDB. Maintenance is queued on the same propagation queue as async GSIs, so a search immediately after a write may not see it. Governed by the same `index_propagation_delay_ms` setting; unlike a GSI there is no per-index override. A value of 0 applies maintenance inline in the write's own transaction, which is stricter than the service and exists so a test can assert steady state without waiting. Inline additionally requires the table's propagation queue to be empty: while rows queued during an index build are still draining, a new write queues behind them instead of overtaking them, so a brief search-after-write window exists even at 0 until the queue drains. That window is ordering, not data loss; the write is applied, in order, by the queue worker. That zero behaves differently while an index is still building, and the two backends differ: **PostgreSQL** applies inline only to an index that is already `ACTIVE`, and defers a write to a building index whatever the delay says, because the write must not reach the index ahead of the backfill's older snapshot of the same item; **SQLite** does not check the status on this path, so with a zero delay a write during a backfill is applied inline and bypasses the hold that keeps the two ordered. The SQLite behaviour is tracked as a defect (F-20), not intended, and it is reachable only with the zero delay, which is a test setting. | | `SearchVectors` score at extreme magnitudes (PostgreSQL backend only) | Returns the true distance as a number for any vector of finite components | The score is bounded to a finite value instead, and the bound is not a measured service answer. pgvector accumulates distances in single precision, so magnitudes far below `f32::MAX` overflow inside the extension: Euclidean above about 9.2e18, dot product above about 1.8e19, and cosine at both ends, above about 1.8e19 and below about 3.7e-23, which is where a component's single-precision square rounds to zero. A non-finite score cannot be serialised as JSON at all, so the result is bounded in SQL: `1e308` for an overflowed distance, `-1e308` for an overflowed negated inner product, which the score contract negates so a client sees `1e308` in `Score`, and 1.0 for a cosine that comes back NaN. Ranking is unaffected at the overflow end, because each bound sits at the end its metric overflows towards, so the farthest row stays farthest and the most similar stays most similar; two rows that both overflow tie, and the tie breaks on the base key. At the underflow end cosine loses resolution rather than being bounded, and which side is tiny decides how much. For a tiny **query** vector, its norm underflows while the inner product usually does not, so the quotient is an infinity that pgvector clamps and the reported distance collapses to one of 0, 1 or 2 following the sign of the inner product, with the 1.0 substitute firing only when the vectors are exactly orthogonal and the quotient is therefore 0/0 (measured). For a tiny **stored** vector it is worse: the stored norm is computed in single precision and reaches zero, so the zero-vector guard fires and that row reports 1.0 at every angle, parallel included. A corpus of tiny embeddings therefore loses ranking altogether rather than losing resolution. For a tiny query vector, by contrast, ranking still separates nearer-than-orthogonal from farther and loses resolution only within each half. The SQLite backend owns its own arithmetic, computes in double precision, and reports the true value, which is why this row is scoped to PostgreSQL. | | Vector index deletion window | `UpdateTable` Delete leaves the index in `DELETING` long enough to observe, then removes it | No observable `DELETING` window on either backend: the catalog row is removed inside the `UpdateTable` transaction, so a `DescribeTable` immediately afterwards already omits the index. The index's data table is dropped after that commit, in a separate transaction, and the two backends handle a failure there differently. **PostgreSQL** treats it as best effort, because the data table lives in a different database entirely: a failure is logged and skipped rather than failing the request. **SQLite** propagates it, so a failed drop returns an error from an `UpdateTable` whose catalog change has already committed, which is the more surprising outcome of the two: the index is gone from the catalog and the caller saw a failure. So an operator debugging a leftover `_ddb_vec_*` table should look for that warning rather than assume the delete was incomplete. | -| Backing up and restoring a table that has vector indexes | Restores the table with its vector indexes intact: the configuration survives, items keep their vector attributes, and `SearchVectors` works as soon as the table is `ACTIVE` (measured) | Partial. With a configured `[backup.store]`, `CreateBackup` of such a table succeeds on PostgreSQL and SQLite, the two backends with vector indexes: the backup is written to the store and its manifest records the index specifications, and `DescribeBackup` and `ListBackups` report it as any other backup (wire test on both backends). Without a store (the section absent on PostgreSQL, or `kind = "disabled"`), where `CreateBackup` writes the in-catalog format, it is refused with `ValidationException` "table has vector index(es); restoring a table with vector indexes is not supported by this storage backend" and no backup is created, because that format cannot carry the index data across. `RestoreTableFromBackup` of any backup in the store answers `ValidationException` "Backup is in a backup store; this release restores in-catalog backups only" until a later release's restore job, which restores a vector table into a backend with vector support and refuses it into one without. A backup of a vector table taken before catalog 0.0.4 is refused at restore on PostgreSQL with the same message naming the backup; on SQLite such a backup restores without its vector indexes. A backup of a table with no vector indexes restores normally. | +| Backing up and restoring a table that has vector indexes | Restores the table with its vector indexes intact: the configuration survives, items keep their vector attributes, and `SearchVectors` works as soon as the table is `ACTIVE` (measured) | Partial. With a configured `[backup.store]`, `CreateBackup` of such a table succeeds on PostgreSQL and SQLite, the two backends with vector indexes: the backup is written to the store and its manifest records the index specifications, and `DescribeBackup` and `ListBackups` report it as any other backup (wire test on both backends). Without a store (the section absent on PostgreSQL, or `kind = "disabled"`), where `CreateBackup` writes the in-catalog format, it is refused with `ValidationException` "table has vector index(es); restoring a table with vector indexes is not supported by this storage backend" and no backup is created, because that format cannot carry the index data across. `RestoreTableFromBackup` of a store-backed vector backup restores the table with its vector indexes into PostgreSQL and SQLite, the two backends with vector support (`SearchVectors` answers as soon as the table is `ACTIVE`; matrix cells), and is refused into MongoDB or Cassandra with `ValidationException` "backup has vector index(es); restoring a table with vector indexes is not supported by this storage backend" before any target table is created. A backup of a vector table taken before catalog 0.0.4 is refused at restore on PostgreSQL with the same message naming the backup; on SQLite such a backup restores without its vector indexes. A backup of a table with no vector indexes restores normally. | | `SearchVectors` endpoint | Served only on `search-dynamodb..amazonaws.com`. The standard `dynamodb..amazonaws.com` endpoint answers the same request with HTTP 400 `UnknownOperationException` ("This operation is not supported by this endpoint"); every control-plane and item operation stays on the standard endpoint. Signing is unchanged either way (service name `dynamodb`, target prefix `DynamoDB_20120810`) | Served on the same endpoint as every other operation, so a client is pointed at one ExtendDB endpoint for all of them. Two consequences worth knowing: an SDK that resolves a separate search hostname from its endpoint ruleset needs its endpoint overridden to reach ExtendDB, and ExtendDB does **not** reproduce the service's refusal, so a test asserting `UnknownOperationException` for vector search on the base endpoint passes against Amazon DynamoDB and fails here. | | `SearchVectors` result order for equal distances | Measured unstable: three identical searches returned tied rows in three different orders, and a top-k that truncates a tie group keeps an arbitrary subset of it | Deterministic on both backends, by different means. **PostgreSQL** sorts explicitly on the base table's full primary key after the score, so the order is a property of the query. **SQLite** issues no `ORDER BY` and resolves ties by scan order through a stable top-k, so its order is a property of the plan rather than something the query guarantees. Either way a client re-issuing an identical search sees the same order, which is stricter than the service rather than divergent in outcome. Do not rely on the two backends agreeing on which subset of a truncated tie group they keep. | | Vector indexes on the MongoDB backend | Vector indexes and `SearchVectors` are available | Not supported at all. That backend provides no vector search implementation, so the engine's capability gate refuses `CreateTable` and `UpdateTable` carrying `VectorIndexes` and every `SearchVectors` request, with the same capability message a PostgreSQL deployment without pgvector returns. Every other vector row in this document describes the PostgreSQL and SQLite backends. | -| Multi-part base table keys | Not supported | Preview extension (opt-in via `enable_multipart_keys` setting). Standard single/composite keys work identically. | +| Multi-part base table keys | Not supported | Preview extension (opt-in via the `limits.allow_multipart_table_keys` setting). Standard single/composite keys work identically. | | GSI throughput on a `PROVISIONED` table | Every GSI needs `ProvisionedThroughput` of at least 1 read and 1 write unit | A GSI `ProvisionedThroughput` below 1 (including 0/0) is `ValidationException` on `CreateTable` of a `PROVISIONED` table and on `UpdateTable`'s GSI `Create` ("ReadCapacityUnits and WriteCapacityUnits must both be greater than or equal to 1 for index: "; the service's text was not captured). A GSI with no `ProvisionedThroughput` is still accepted on a `PROVISIONED` table, and `UpdateTable` to `PROVISIONED` leaves an on-demand table's GSIs without one; see the `BillingModeOverride` row under Backup and Restore for how a backup of such a table restores | | Projected attributes across all indexes | At most 100 | Enforced on `CreateTable` (and so on `RestoreTableFromBackup`, which validates its target the same way) and on `UpdateTable`'s GSI `Create`: every `NonKeyAttributes` entry of every GSI and LSI is counted, so an attribute projected into two indexes counts twice. Over 100 is `ValidationException` ("Number of projected attributes in all indexes exceeds limit of 100, projected: "; the service's text was not captured) | | `Query` on a GSI with several RANGE attributes (MongoDB and Cassandra backends) | Orders results over every RANGE attribute in key-schema order and serves a key condition on each of them (equality on every one before the last, any operator on the last) | SQLite and PostgreSQL match the service. The MongoDB and Cassandra backends filter the first RANGE attribute of such an index only: a key condition on a later RANGE attribute is refused with `ValidationException` naming the attribute and the backend. MongoDB orders results by the first RANGE attribute then the base table key. Cassandra orders a page over every RANGE attribute then the base table key, and refuses a `Query` that carries an `ExclusiveStartKey` on such an index with `ValidationException` (the cursor compares the first RANGE attribute only). Every refusal the service gives for the condition's shape (one condition per key, condition count, missing HASH attribute, RANGE gap, non-key attribute, value type) is the same on every backend. | @@ -102,8 +102,8 @@ Behaviour stated as measured, and every number below, was observed with release | Area | DynamoDB | ExtendDB | |------|----------|------| -| What a restore carries across | Key schema, attribute definitions, items, secondary indexes, billing mode and provisioned throughput, on-demand throughput (table and GSI level), SSE settings. Table class is not carried: a restore is `STANDARD` whatever the source's class (measured). Stream settings, TTL, tags, auto scaling, IAM policies, and CloudWatch settings must be reapplied by hand | The same set on all four backends, with one gap: the table's on-demand throughput is carried, a GSI's is not (ExtendDB records no per-index on-demand maximum, and `OnDemandThroughput` is not a member of its GSI definition). Table class is not restored: a `STANDARD_INFREQUENT_ACCESS` source restores as `STANDARD`, as on the service. Streams, TTL, tags, and deletion protection are not restored. A backup written before catalog 0.0.4 (PostgreSQL, SQLite, Cassandra) has no recorded table definition and restores with its keys, items, and billing mode but no secondary indexes. A provisioned one restores with 5/5 throughput on PostgreSQL and SQLite and with its recorded throughput on Cassandra | -| How `RestoreTableFromBackup` runs | Returns at once with the target `CREATING`; the service copies in the background | The same on all four backends: the response carries the target `CREATING`, and the items are copied by a background task in the server, so a client that times out or disconnects does not stop it. Once the request has been authenticated and authorized, the rest of it, from reading the backup to registering the restore, runs in a task of its own, so a client that disconnects after that does not cancel it either: the restore runs and the client finds the target with `DescribeTable`. A client that disconnects while the request is still being authenticated or authorized cancels it, and nothing is created *(from the code)*. Measured with a client whose 1 ms read timeout closed its connection before any response arrived: on all four backends the restore was registered and finished. Measured with a 60,000-item backup of a table with no secondary indexes: response in 0.032 s (PostgreSQL), 0.012 s (SQLite), 0.053 s (MongoDB), 0.151 s (Cassandra); `ACTIVE` after 1.5 s, 2.1 s, 3.6 s, and 11.2 s. On Cassandra the response waits for the target table's schema to be created: 0.75 to 1.3 s with five restores running at once *(earlier build)*, which is longer than a client read timeout of 1 s. On MongoDB the request waits at most 60 s for its background task to register the `CREATING` target (a bound against a connection pool that has stopped handing out connections); past that it answers `InternalServerError` while the task goes on, so the target can appear after the error, holding the backup in use (`BackupInUseException` to `DeleteBackup`) and the name until the copy ends; `DescribeTable` is where a client finds it *(from the code)*. A restore runs inside the server process; if the process stops, see the abandoned-restore row | +| What a restore carries across | Key schema, attribute definitions, items, secondary indexes, billing mode and provisioned throughput, on-demand throughput (table and GSI level), SSE settings. Table class is not carried: a restore is `STANDARD` whatever the source's class (measured). Stream settings, TTL, tags, auto scaling, IAM policies, and CloudWatch settings must be reapplied by hand | The same set on all four backends, with one gap: the table's on-demand throughput is carried, a GSI's is not (ExtendDB records no per-index on-demand maximum, and `OnDemandThroughput` is not a member of its GSI definition). Table class is not restored: a `STANDARD_INFREQUENT_ACCESS` source restores as `STANDARD`, as on the service. Streams, TTL, tags, and deletion protection are not restored. `DescribeTable` on a restored table reports `ItemCount` as the backup's count and `TableSizeBytes` as the manifest's `source_table.table_size_bytes` (the source's size when the backup was taken) until the statistics pass next rewrites both from the restored rows *(from the code)*. A backup written before catalog 0.0.4 (PostgreSQL, SQLite, Cassandra) has no recorded table definition and restores with its keys, items, and billing mode but no secondary indexes. A provisioned one restores with 5/5 throughput on PostgreSQL and SQLite and with its recorded throughput on Cassandra | +| How `RestoreTableFromBackup` runs | Returns at once with the target `CREATING`; the service copies in the background | The same on all four backends: the response carries the target `CREATING`, and the items are copied by a background task in the server, so a client that times out or disconnects does not stop it. Once the request has been authenticated and authorized, the rest of it, from reading the backup to registering the restore, runs in a task of its own, so a client that disconnects after that does not cancel it either: the restore runs and the client finds the target with `DescribeTable`. A client that disconnects while the request is still being authenticated or authorized cancels it, and nothing is created *(from the code)*. Measured with a client whose 1 ms read timeout closed its connection before any response arrived: on all four backends the restore was registered and finished. Measured *(in-catalog restore, an earlier build)* with a 60,000-item backup of a table with no secondary indexes: response in 0.032 s (PostgreSQL), 0.012 s (SQLite), 0.053 s (MongoDB), 0.151 s (Cassandra); `ACTIVE` after 1.5 s, 2.1 s, 3.6 s, and 11.2 s. Measured *(store-backed restore, this release)* with the same shape (60,000 items of 200 bytes, composite key, no secondary index) from a filesystem store on loopback: response in 0.051 s (PostgreSQL), 0.016 s (SQLite), 0.055 s (MongoDB); `ACTIVE` after 0.67 s, 1.85 s and 3.74 s (on MongoDB the job's DONE at 1.8 s and the index backfill worker's flip after); 1M items of 200 bytes in key order (the server's own backups) `ACTIVE` after 9.5 s on PostgreSQL and 26.8 s on SQLite, the same items as an unordered backup (a MongoDB source, a multi-part-key table) after 17.5 s and 66 to 90 s (`docs/design/15-backup-restore-bench-pr9.md`, which has the host and its load); the store-backed response includes the sidecar and manifest reads and its `ACTIVE` time the download and the per-file verification, and Cassandra as a store-backed target was measured at 12,000 items only (the kill points: a resumed restore of 12,000 items of 4,000 bytes in two data files reached `ACTIVE` 8.3 s after its takeover on a single-node container). On Cassandra the response waits for the target table's schema to be created: 0.75 to 1.3 s with five restores running at once *(earlier build)*, which is longer than a client read timeout of 1 s. On every backend the request waits at most 60 s (`RESTORE_REGISTRATION_TIMEOUT`) for its background task to register the `CREATING` target of a store-backed restore (a bound against a connection pool that has stopped answering; the manifest and sidecar reads that precede the registration are not under it); past that it answers `InternalServerError` while the task goes on, so the target can appear after the error, holding the backup in use (`BackupInUseException` to `DeleteBackup`) and the name until the restore ends; `DescribeTable` is where a client finds it *(from the code)*. A restore runs inside the server process; if the process stops, see the abandoned-restore row | | `RestoreSummary` | The restore response and `DescribeTable` while `CREATING` report `SourceBackupArn`, `SourceTableArn`, `RestoreDateTime` (the backup's `BackupCreationDateTime`, not the time the restore ran), and `RestoreInProgress=true`. Once the table is `ACTIVE`, `DescribeTable` omits `RestoreSummary` | The same on all four backends. No response that describes a table (`CreateTable`, `UpdateTable`, `DeleteTable`, `DescribeTable`, `RestoreTableFromBackup`) carries `RestoreSummary` unless the table is `CREATING`; what the service's `UpdateTable` returns for a restored table was not measured | | Reads on a restore target | A `GetItem` or `Scan` on the target while `CREATING` is `ResourceNotFoundException`; the first read after `DescribeTable` reports `ACTIVE` succeeds (measured) | The same on all four backends. The server's key-info cache stores a negative entry (5 s TTL by default) for a name the catalog does not have, and does not store one for a name the catalog has in a state other than `ACTIVE`: on a not-found answer it asks the catalog once more with `DescribeTable`, and authorization records the name as missing in the key-info snapshot it hands the engine, so the engine does not look it up again. A `GetItem`, `PutItem`, `UpdateItem`, `DeleteItem`, `Query`, or `Scan` on the target while it is `CREATING` therefore costs exactly one `table_key_info` and one `describe_table` catalog load, uncached, per request (on SQLite five SQL statements: one for the key info, four for `DescribeTable`; on PostgreSQL `describe_table` is two queries), plus one `describe_table` on the first such request for the throttle registry; the first read after `ACTIVE` loads once more and is then cached. Measured on SQLite with a `GetItem` loop on a `CREATING` restore target of 100,000 items: 40 requests refused, 40 key-info and 41 `DescribeTable` loads in the server's statement log; with the hand-off to the engine removed, 80 and 81. In an earlier run of the fidelity test the first `GetItem` after `ACTIVE` answered in 3 ms after 72 refused reads. | | `RestoreTableFromBackup` overrides | Current request members are `BillingModeOverride`, `ProvisionedThroughputOverride`, `GlobalSecondaryIndexOverride`, `LocalSecondaryIndexOverride`, `SSESpecificationOverride`, `OnDemandThroughputOverride`, and `VectorIndexOverride`; DynamoDB applies them to the restored table | See the overrides table below. A refused override is refused before the target table is created | @@ -114,17 +114,18 @@ Behaviour stated as measured, and every number below, was observed with release | Tag conditions on backups | The backup resource has no `aws:ResourceTag` condition key: a backup has no tags to match | `DescribeBackup`, `DeleteBackup`, and `RestoreTableFromBackup` are authorized on the backup ARN as a resource with no tags. An `aws:ResourceTag/` condition finds the key absent for a backup (and `dynamodb:ResourceTag/` is absent for every resource), so, for example, an Allow conditioned on `StringEquals` never grants on a backup and a Deny conditioned on `StringEquals` never denies one. This holds whatever an earlier build stored under the backup ARN (it accepted `TagResource` on one; see Tagging), and a backup does not take its table's tags. For `RestoreTableFromBackup`, the seven item actions evaluated on `table/` (see the authorization rows below) read that ARN's resource tags. A target that does not exist has no tags, so an `aws:ResourceTag/` condition on those actions finds none and does not grant the restore; a target that exists is evaluated with its tags, and a request that is otherwise valid then passes authorization and fails `TableAlreadyExistsException` (or `TableInUseException` for a `CREATING` restore target) when the condition is met, and fails `AccessDeniedException` naming the first item action when it is not (measured on SQLite). Whether the service reads a target's tags in these evaluations was not measured. Scope backup permissions by ARN (`table//backup/*`). Measured on PostgreSQL and SQLite for the Allow case, including tags written under a backup ARN by the earlier build and then served by this one; the Deny case and role sessions rest on unit tests of the authorization path, which is the same for every backend | | Authorization of `ListBackups` | `dynamodb:ListBackups` is evaluated on `arn:aws:dynamodb:::table/*/backup/*`, or on `table//backup/*` when `TableName` filters the list (measured on DynamoDB) | The same on all four backends (the check runs in the server before the backend is reached). A statement on `*` or `table/*` covers every list; one on `table//backup/*` (or `table//*`) covers only the list filtered to that table; one on the table's own ARN, `table/`, covers none. The unfiltered list is one resource, so a Deny on `table//backup/*` denies the list filtered to that table and not the unfiltered list that would include its backups, which follows from the resources the service evaluates. A `TableName` that is not a string is evaluated as no filter; the engine then refuses the request. These resources have been evaluated since v0.1.14; the admin guide's compatibility notes describe what earlier builds evaluated | | Authorization of `RestoreTableFromBackup` | `dynamodb:RestoreTableFromBackup` on the backup ARN, and `dynamodb:Scan`, `dynamodb:Query`, `dynamodb:GetItem`, `dynamodb:PutItem`, `dynamodb:UpdateItem`, `dynamodb:DeleteItem`, and `dynamodb:BatchWriteItem` on `table/`, each of which must be allowed; `dynamodb:RestoreTableFromBackup` is not evaluated on the target (measured on DynamoDB) | The same eight evaluations on all four backends, in that order: the backup first, then the target under each item action in the order listed, stopping at the first denial, which the `AccessDeniedException` names (`User: ... is not authorized to perform: dynamodb:Scan on resource: arn:aws:dynamodb:::table/`); the service's order was not measured. A missing `TargetTableName` is evaluated as `table/` (an empty name, which no statement but `*`-style wildcards covers); the engine then refuses the request. The seven item-action evaluations read the target ARN's resource tags: a target that does not exist has none, one that exists is evaluated with its tags and the restore then fails `TableAlreadyExistsException` (see Tag conditions on backups). Every DynamoDB condition key (`dynamodb:LeadingKeys`, `dynamodb:Attributes`, `dynamodb:Select`, `dynamodb:ReturnValues`, `dynamodb:ReturnConsumedCapacity`, `dynamodb:FullTableScan`, `dynamodb:EnclosingOperation`) is absent in these seven evaluations, so `ForAllValues:*` and `Null: true` conditions pass and bare `StringEquals` and `Bool` conditions fail: an Allow of the item actions conditioned on `ForAllValues:StringEquals dynamodb:LeadingKeys` grants the restore of every item in the backup, an Allow of `dynamodb:Scan` conditioned on `Bool dynamodb:FullTableScan: false` blocks every restore, and a Deny of `dynamodb:Scan` conditioned on `Bool dynamodb:FullTableScan: true` blocks none; the keys the service sets in these evaluations were not measured. The backup ARN is evaluated as the backup it names in this deployment: the table and backup id from the request, in the server's region and the caller's account, whatever partition, region or account the caller wrote, with surrounding whitespace trimmed as the engine trims it. The engine resolves the same table and id (under the server's region first, then as written for a backup created under an earlier `server.region`), so it never acts on a backup other than the one evaluated. `DescribeBackup` and `DeleteBackup` are evaluated the same way. These resources have been evaluated since v0.1.14 (see the admin guide's compatibility notes). A Deny of `dynamodb:RestoreTableFromBackup` on a table does not block restores into it; deny one of the seven item actions on it instead | -| Multi-part base table keys (preview) | Not supported | A backup of a table created with `enable_multipart_keys` is refused at restore with `ValidationException`; the item paths address such tables by their first HASH and RANGE attribute only *(from the code and storage-level tests; not checked on a running server)* | +| Multi-part base table keys (preview) | Not supported | A store-backed backup of a table created under `limits.allow_multipart_table_keys` is written `unordered`, and its restore preserves the item set: the restored table holds what the source's `Scan` held, with the source's key schema and attribute definitions, the target laid out as a natively created table (`bulk_load` keys rows with the backend's own key functions) *(measured on PostgreSQL and SQLite with the preview enabled, `tests/test_backup_job.py`; the engine test `a_multi_part_key_manifest_is_registered`)*. The preview's item paths address such a table by its first HASH attribute on every backend, restored or not: on PostgreSQL a `GetItem` by the whole key finds nothing on the source or on the restored table (a `Query` naming both key attributes does find the item), on SQLite the source itself keeps one item per first HASH key value and a `GetItem` returns that item whatever second key it was given, and a `Query` on the first HASH key alone is refused on both ("Query condition missed key schema element") *(measured on PostgreSQL and SQLite with the preview enabled)*. An in-catalog backup of such a table (no store, or Cassandra) is refused at restore with `ValidationException` *(from the code and storage-level tests)* | | `DeleteBackup` and `DeleteTable` during a restore | `DeleteTable` on the `CREATING` target is `ResourceInUseException`. `DeleteBackup` during a restore was not measured | While the target is `CREATING`, `DeleteTable` on it is `ResourceInUseException` on all four backends, as on the service. `DeleteBackup` on its source backup is `BackupInUseException` on all four: that refusal is ExtendDB's contract, not measured against the service. Both succeed once the target is `ACTIVE`. How the restore and `DeleteBackup` are ordered differs *(from the code)*. PostgreSQL and SQLite register the restore in the transaction that creates the target, holding the backup row, so a concurrent `DeleteBackup` is ordered strictly before (the restore then finds no backup) or after (it finds the restore). MongoDB runs the restore's registration and `DeleteBackup`'s claim each as one session transaction that first increments a lock field on the backup's catalog document while it is `AVAILABLE`, so the two conflict and one orders wholly before the other: a `DeleteBackup` that wins leaves the registration no `AVAILABLE` backup (`BackupNotFoundException`), a registration that wins makes the `DeleteBackup` `BackupInUseException`. Cassandra uses two lightweight transactions on different partitions: the target's row is inserted first (`IF NOT EXISTS`, which reserves the name), then the restore is registered on the backup's row (`IF backup_status = 'AVAILABLE'`), which `DeleteBackup` claims on the same partition; a concurrent `DescribeTable` can therefore see a `CREATING` target for a restore that then fails with `BackupNotFoundException`, after which the target is removed. On MongoDB `DeleteBackup` writes the backup `DELETING` before it drops the data and `DELETED` after; a `DeleteBackup` whose request fails or is cancelled after the drop started leaves the backup `DELETING`, which `DescribeBackup`, `ListBackups` and a restore treat as gone (`BackupNotFoundException`), and a retry of `DeleteBackup` finishes the drop and answers `DELETED` *(from the code and storage-level tests)*. MongoDB reads the catalog from the primary with read concern `local` regardless of the connection string, which the lock and the job records rely on; the connection string must use `readPreference=primary` (any other read preference is refused at startup). The write concern follows the connection string and must be `majority` (the server default): a connection string with `w=1` is refused at startup, because a `DELETING` write or a job record acknowledged under `w=1` just before a primary failover can be rolled back | -| Concurrent restores of one backup | A second `RestoreTableFromBackup` of a backup while a restore from it is still `CREATING` is `BackupInUseException` (measured) | The same on all four backends: the second restore is refused with `BackupInUseException` and creates nothing; the check runs inside the transaction that registers the restore (PostgreSQL, with the backup row locked `FOR UPDATE`; SQLite), inside the claim on the backup document followed by a check for a `CREATING` target of the backup (MongoDB), or inside the lightweight transaction on the backup row (Cassandra), so two simultaneous requests cannot both pass. Once the first target is `ACTIVE` the backup restores again; on MongoDB the check reads majority-committed data (at the default transaction read concern) while `DescribeTable` reads at `local`, so a `RestoreTableFromBackup` or `DeleteBackup` issued within the majority-commit lag after `DescribeTable` first shows the target `ACTIVE` can still be `BackupInUseException` and succeeds when retried (the lag is the journal flush plus any concurrent batch write still holding an earlier oplog slot; measured at 7 to 12 ms, once in about 2,000 observations, with a 3,000-item insert running beside the write). A restore of a backup whose `DeleteBackup` is in progress answers `BackupNotFoundException` on every backend (the service's answer was not measured), by two mechanisms: on MongoDB and Cassandra the delete marks the backup `DELETING` before it drops the data, and the restore finds no `AVAILABLE` backup; on PostgreSQL and SQLite the restore waits for the delete's transaction and then finds no backup. Earlier builds accepted concurrent restores of one backup on PostgreSQL, SQLite, and Cassandra, and on MongoDB after a 2 s retry | -| Restore interrupted by the server stopping | Managed by the service | The copy runs in the server process, so a stopped process leaves its target `CREATING` with a RUNNING restore job record whose heartbeat stops. The engine's reconciler, once at startup and then every `backup.reconcile_interval_secs` (default 60 s), marks the record FAILED and removes the target once the heartbeat is older than `backup.stale_job_after_secs` (default 300 s) on the database's clock, on every backend alike, logs the removal with the table name, and then lets `DeleteBackup` and a new restore into the name succeed *(from the code)*. On SQLite a restart ages every record of the stopped process at once (one process serves the file), so the first cycle removes it without the stale window. Until then `DeleteTable` and `DeleteBackup` are refused as during a running restore. Not covered *(from the code)*: on MongoDB, a target whose base copy had finished has its record DONE and is left alone by the reconciler; the index backfill worker finishes it after a restart, and if that backfill fails on every attempt the worker retries it every 5 s with a warning and never gives up, so the target stays `CREATING` (and `DeleteTable` and `DeleteBackup` stay refused) until the cause is fixed or the target's catalog document is removed by hand (no failing case was found to reproduce this). A target left `CREATING` by a build before this release is backfilled by the catalog migration as a RUNNING record with a heartbeat at the epoch, so the first cycle removes it. Heartbeats and the stale window are measured on the database server's clock, so the application hosts' clocks play no part. The same reconciler resolves a store-backed `CreateBackup` the server stopped (one reconciler with one duty list, in place of per-backend sweeps): a `CREATING` backup row whose owner is dead (on PostgreSQL and MongoDB a heartbeat older than `stale_job_after_secs`, on PostgreSQL once the owner's claim session has ended as well; on SQLite, where one process serves the file, every `CREATING` row after a restart and, at runtime, a row whose in-process slot no task holds, so a job task that died is resolved by the next cycle without a restart) is taken over under a fresh owner generation and becomes `AVAILABLE` when a complete, verified manifest is under its prefix in the configured store (the process died between the manifest write and the status write) and `FAILED` otherwise, never continuing the upload; the objects of a `FAILED` backup are deleted once its heartbeat is older than `stale_job_after_secs`; a `DELETING` backup's deletion is finished; a `DELETED` backup's prefix that still lists objects is swept; and `FAILED` and `DELETED` rows are removed once their prefix is empty and their `completed_at` is older than `failed_row_retention_secs` (7 days). Measured *(binary-level, this release)*: SIGKILL between the data file and the manifest left the row `CREATING` and the data file under the prefix, and after the restart the row was `FAILED` in the first cycle on SQLite and 46 s after the kill on PostgreSQL (`stale_job_after_secs = 45`), with the prefix deleted in the same cycle on PostgreSQL (the stale heartbeat is what made the owner dead) and once the dead owner's heartbeat was stale on SQLite, 45 s after the kill (the slot made the owner dead at once; the prefix waits for the heartbeat clock on every backend); the same at a kill between the three documents and the manifest, which leaves the documents under the prefix and no manifest; SIGKILL between the manifest write and the status write left the five objects, and the row was `AVAILABLE` with the manifest's size and count after the restart (SQLite at once, PostgreSQL after 48 s); SIGTERM during an upload left the row `FAILED` with reason `shutdown` and an empty prefix. These duties run only while the configured store is healthy, and under a maintenance claim only the dead-owner and `DELETING` duties run. A `FAILED` backup is `BackupNotFound` to clients; `extenddb backup list --failed` prints it with its reason | +| Concurrent restores of one backup | A second `RestoreTableFromBackup` of a backup while a restore from it is still `CREATING` is `BackupInUseException` (measured) | The same on all four backends: the second restore is refused with `BackupInUseException` and creates nothing; the check runs inside the transaction that registers the restore (PostgreSQL, with the backup row locked `FOR UPDATE`; SQLite), inside the claim on the backup document followed by a check for a `CREATING` target of the backup (MongoDB), or inside the lightweight transaction on the backup row (Cassandra), so two simultaneous requests cannot both pass. For a store-backed restore the registration takes the same lock on the `backup_index` row instead of the legacy `backups` row (PostgreSQL `SELECT ... FOR UPDATE`, SQLite the `BEGIN IMMEDIATE` write transaction, MongoDB a `$inc lock_seq` on the `backup_index` document, Cassandra the `active_restores` lightweight transaction on the backup's partition), the same ordering guarantee, so a legacy restore and a store-backed restore of two different backups never contend and a `DeleteBackup` of a store-backed backup meets only store-backed registrations. Two concurrent registrations of one backup leave one restore target on Cassandra as on MongoDB: the registration's lightweight transaction (`IF backup_status = 'AVAILABLE' AND active_restores = null`) is linearized on the backup's partition, so the second finds `active_restores` non-null and is `BackupInUseException`; the in-use pre-read is advisory *(storage-level tests on both)*. Once the first target is `ACTIVE` the backup restores again; on MongoDB the check reads majority-committed data (at the default transaction read concern) while `DescribeTable` reads at `local`, so a `RestoreTableFromBackup` or `DeleteBackup` issued within the majority-commit lag after `DescribeTable` first shows the target `ACTIVE` can still be `BackupInUseException` and succeeds when retried (the lag is the journal flush plus any concurrent batch write still holding an earlier oplog slot; measured at 7 to 12 ms, once in about 2,000 observations, with a 3,000-item insert running beside the write). A restore of a backup whose `DeleteBackup` is in progress answers `BackupNotFoundException` on every backend (the service's answer was not measured), by two mechanisms: on MongoDB and Cassandra the delete marks the backup `DELETING` before it drops the data, and the restore finds no `AVAILABLE` backup; on PostgreSQL and SQLite the restore waits for the delete's transaction and then finds no backup. Earlier builds accepted concurrent restores of one backup on PostgreSQL, SQLite, and Cassandra, and on MongoDB after a 2 s retry | +| Restore interrupted by the server stopping | Managed by the service | The copy runs in the server process, so a stopped process leaves its target `CREATING` with a RUNNING restore job record whose heartbeat stops. The engine's reconciler, once at startup and then every `backup.reconcile_interval_secs` (default 60 s), marks the record FAILED and removes the target once the heartbeat is older than `backup.stale_job_after_secs` (default 300 s) on the database's clock, on every backend alike, logs the removal with the table name, and then lets `DeleteBackup` and a new restore into the name succeed *(from the code)*. On SQLite a restart ages every record of the stopped process at once (one process serves the file), so the first cycle removes it without the stale window. Until then `DeleteTable` and `DeleteBackup` are refused as during a running restore. A store-backed restore (one run from a `[backup.store]`) is instead resumed, not discarded: its owner's death leaves a RUNNING restore job record with a stale heartbeat and the target `CREATING`, and the reconciler of any instance with the configured store takes the record over under a fresh owner generation once the heartbeat is stale (on SQLite at once after a restart, by the record's slot; on PostgreSQL once the owner's claim session has ended as well), stamping a fresh heartbeat in the same write so no second instance takes it too, and resumes the load from its recorded progress (per data file on PostgreSQL and MongoDB, per batch on SQLite and Cassandra), re-downloading to its own spool when the dead owner's `spool_host` is not this host's; the target reaches `ACTIVE` with the backup's full item count, its secondary indexes complete: a resumed Cassandra restore re-applies its last batch as idempotent upserts that rewrite the same base and index rows (#742 corrected the index-row delete-plus-insert that a re-applied unchanged item would otherwise suffer; before #742 was corrected a resumed Cassandra restore reached `ACTIVE` with up to one batch of GSI and LSI rows missing). A resume that finds no recorded progress and a non-empty target is `FAILED` with reason `target_not_empty` and the target is left in place: the rule protects a catalog restored from a dump that names a restore which finished and was written to since, but on MongoDB and Cassandra, which record progress after the write, an ordinary crash before the first progress record (anywhere in the first data file on MongoDB; in the first 10,000 items or 32 MiB on Cassandra) leaves written items and no progress and ends in this state too. The target stays `CREATING` with a `FAILED` record: `DeleteTable` on it is `ResourceInUseException`, `DeleteBackup` of its backup and a second restore of that backup are `BackupInUseException`, and the reconciler does not remove it; `extenddb backup remove-target --account
` (the admin guide's `backup` section) removes the target and its record, after which `DeleteBackup` and a new restore into the name succeed. A legacy (in-catalog) restore is still failed and its target removed. Measured *(binary-level, this release)*: SIGTERM during a store-backed restore left the record RUNNING and the restarted server resumed it to `ACTIVE` with every item (on PostgreSQL after the 45 s stale window, on SQLite at once); a SIGKILL at each restore kill point left the target to finish with the exact count (the base table's and the GSI's) or, for a failing restore, gone, except at the first reach of the two points that pause before a first progress record: on MongoDB (killed after the first file's `insert_many`, before its record) and on Cassandra (killed after the first batch's 10,000 upserts, before their record) the resume answered `target_not_empty` and left the target `CREATING`, and `extenddb backup remove-target` then removed it, after which `DescribeTable` was `ResourceNotFoundException` and `DeleteBackup` succeeded. Not covered *(from the code)*: on MongoDB, a target whose base copy had finished has its record DONE and is left alone by the reconciler; the index backfill worker finishes it after a restart, and if that backfill fails on every attempt the worker retries it every 5 s with a warning and never gives up, so the target stays `CREATING` (and `DeleteTable` and `DeleteBackup` stay refused) until the cause is fixed or the target's catalog document is removed by hand (no failing case was found to reproduce this). A target left `CREATING` by a build before this release is backfilled by the catalog migration as a RUNNING record with a heartbeat at the epoch, so the first cycle removes it. Heartbeats and the stale window are measured on the database server's clock, so the application hosts' clocks play no part. The same reconciler resolves a store-backed `CreateBackup` the server stopped (one reconciler with one duty list, in place of per-backend sweeps): a `CREATING` backup row whose owner is dead (on PostgreSQL and MongoDB a heartbeat older than `stale_job_after_secs`, on PostgreSQL once the owner's claim session has ended as well; on SQLite, where one process serves the file, every `CREATING` row after a restart and, at runtime, a row whose in-process slot no task holds, so a job task that died is resolved by the next cycle without a restart) is taken over under a fresh owner generation and becomes `AVAILABLE` when a complete, verified manifest is under its prefix in the configured store (the process died between the manifest write and the status write) and `FAILED` otherwise, never continuing the upload; the objects of a `FAILED` backup are deleted once its heartbeat is older than `stale_job_after_secs`; a `DELETING` backup's deletion is finished; a `DELETED` backup's prefix that still lists objects is swept; and `FAILED` and `DELETED` rows are removed once their prefix is empty and their `completed_at` is older than `failed_row_retention_secs` (7 days). Measured *(binary-level, this release)*: SIGKILL between the data file and the manifest left the row `CREATING` and the data file under the prefix, and after the restart the row was `FAILED` in the first cycle on SQLite and 46 s after the kill on PostgreSQL (`stale_job_after_secs = 45`), with the prefix deleted in the same cycle on PostgreSQL (the stale heartbeat is what made the owner dead) and once the dead owner's heartbeat was stale on SQLite, 45 s after the kill (the slot made the owner dead at once; the prefix waits for the heartbeat clock on every backend); the same at a kill between the three documents and the manifest, which leaves the documents under the prefix and no manifest; SIGKILL between the manifest write and the status write left the five objects, and the row was `AVAILABLE` with the manifest's size and count after the restart (SQLite at once, PostgreSQL after 48 s); SIGTERM during an upload left the row `FAILED` with reason `shutdown` and an empty prefix. These duties run only while the configured store is healthy, and under a maintenance claim only the dead-owner and `DELETING` duties run. A `FAILED` backup is `BackupNotFound` to clients; `extenddb backup list --failed` prints it with its reason | | Point-in-time recovery | 35-day window | Not supported. `RestoreTableToPointInTime` is refused with `ValidationException` ("Point-in-time recovery restore is not yet supported"). `UpdateContinuousBackups` is not refused: enabling point-in-time recovery succeeds, and `DescribeContinuousBackups` then reports it `ENABLED` with an `EarliestRestorableDateTime` 35 days back, although nothing is recorded and no restore to any of those times is possible. Same on all four backends | | Where backups live | Managed storage, independent of the table | With a configured `[backup.store]` (a filesystem root, which is the default beside a SQLite database file, or an S3 bucket and prefix) on PostgreSQL, SQLite and MongoDB: under the store's root, at `/
//`, as gzip data files of DynamoDB JSON lines (`data/000001.json.gz`, ...) and the manifest documents (`extenddb-manifest.json` last, the commit record), with a `backup_index` row in the catalog as the index; a lost database loses the rows, not the bytes (a later release's `backup sync` re-adopts a root's backups into a new catalog). Without a store, and on Cassandra always, inside the database the server already uses, so a lost database loses its backups too: on PostgreSQL in the catalog database (`backups`, `backup_items`, `backup_definitions`, `table_restores`), on SQLite in the one database file, on MongoDB as a `backups` metadata collection in the catalog database and one data collection per backup in the data database, on Cassandra as backup metadata in the catalog keyspace and item payloads in the account's keyspace; take database-level backups for off-host copies there | | Backup creation | `CreateBackup` returns while the backup is `CREATING`; `ListBackups` and `DescribeBackup` expose that state until it becomes `AVAILABLE`. `BackupName` and `TableName` are checked as the model says: length 3 to 255 and `[a-zA-Z0-9_.-]+`, with "1 validation error detected: Value '' at 'backupName' failed to satisfy constraint: " texts, joined under "2 validation errors detected:" when both fail (measured). `TableName` may be the table's ARN (measured). A table that does not exist or is still `CREATING` is `TableNotFoundException` "Table not found: " (measured) | With a configured `[backup.store]` on PostgreSQL, SQLite and MongoDB, `CreateBackup` takes the table's snapshot and writes the catalog row in the request and returns the backup `CREATING`; a job then uploads it and the backup becomes `AVAILABLE`, which `DescribeBackup` and `ListBackups` show as on the service (`BackupSizeBytes` is 0 while `CREATING` and the compressed size once `AVAILABLE`; `ItemCount` is the table's count as last counted while `CREATING`, which is 0 for a table the stats pass has not reached yet, such as one loaded seconds before CreateBackup, and the backup's exact count once `AVAILABLE`). The backup holds every write acknowledged before the request was received, which is stricter than the service's one-minute window. A backup the job cannot finish (a store failure, an item no restore can load, a shutdown) is `FAILED`, which is `BackupNotFound` to clients, and its objects are deleted (the restore-interruption row). Measured *(this release, 16-core host, filesystem store, the record in `docs/design/15-backup-restore-bench-pr8.md`)*: 100,000 items of a composite-key table with two GSIs `AVAILABLE` in 1.85 s on PostgreSQL and 1.44 s on SQLite, 1,000,000 in 36.1 s and 12.45 s. Without a store, and on Cassandra always, `CreateBackup` copies the items before it returns and returns the backup `AVAILABLE`. On PostgreSQL, MongoDB, and Cassandra a concurrent `ListBackups` never saw an in-catalog backup `CREATING` *(earlier build)*; on SQLite the backup row is visible as `CREATING` to other requests while the copy runs. Measured *(earlier build, in-catalog)*: 1.3 s for 60,000 items on PostgreSQL, 1.9 s for 60,000 on SQLite, 0.2 s for 20,000 on MongoDB, 5.1 s for 20,000 on Cassandra. MongoDB's in-catalog backups report `BackupSizeBytes` 0. The name checks are the service's, with its texts (the length clause before the pattern clause when both fail; the service's order was not captured), counting characters, with `TableName` taking the deployment's table-name limits; before them a 200 KB name was stored and returned in every `ListBackups` page (a 500 on Cassandra) and a NUL byte was a 500 on PostgreSQL. `TableName` as an ARN (`arn:aws:dynamodb:::table/`) is accepted on all four backends: another region's resolves, another account's is `AccessDeniedException` (neither measured on the service); an SDK built from the model refuses the ARN form client-side, so only a hand-built request sends it. `TableNotFoundException` "Table not found: " for a missing or `CREATING` table on all four backends, built from the request's name (the backends' own payloads differ). A table the deployment could create but whose name no backup ARN can carry is refused with `ValidationException` where the service backs up any table it can create: a name over 255 bytes (reachable only with `[limits] max_table_name_length` raised) or made only of dots ("TableName cannot be backed up: a backup ARN cannot be formed for it: invalid backup ARN: table name of N bytes is longer than 255" / "... invalid backup ARN: table name \"...\" is only dots"), and every table under a `server.region` outside `[a-z0-9-]` (the `BackupArn` row); the checks run after the table's ACTIVE probe and before the job permit, so the refusal takes no slot | | `DeleteBackup` response | The full `DescribeBackup` description with `BackupStatus` `DELETED`, including `SourceTableDetails.ProvisionedThroughput` (a required member) and `SourceTableFeatureDetails` (measured) | The same on all four backends: the engine reads the backup's recorded definition before the delete and fills the description as `DescribeBackup` does, with `BackupStatus` `DELETED`. A backup whose definition this build cannot read or that does not parse is still deleted and described without feature details (logged at warn), which means a damaged `PROVISIONED` backup is described with on-demand `ProvisionedThroughput` `{0, 0}`: ExtendDB's choice, deletability over the throughput member of a row that is being removed, where `DescribeBackup` on the same row is an `InternalServerError`; a failure to reach the database fails the request and deletes nothing. | | `DescribeBackup` when the definition cannot be read | Not applicable | A definition row that does not parse, a renamed column, or a revoked privilege (`Internal` from the backend) is an `InternalServerError`, as it is for `RestoreTableFromBackup` on the same read; only a definition version this build does not know is described without `SourceTableFeatureDetails`. | -| Consistency of a backup | Point-in-time snapshot of the table | **PostgreSQL** and **SQLite** take the items from one snapshot, and so does **MongoDB** with a configured `[backup.store]`. How *(from the code)*: with a store, the request opens the table snapshot (`SnapshotEngine`, the module docs of `crates/storage/src/snapshot.rs` and the backends' `snapshot_engine.rs` are the record): on PostgreSQL one `REPEATABLE READ` transaction holds the definition and the items on the shared layout, and on the separate layout the catalog row is held shared until the data transaction is pinned, the items then paged by the data table's primary key under that transaction, each page cut at 64 MiB of item text; on SQLite one deferred read transaction opened without the engine's write lock, paged by key with the same cut; on MongoDB a session with `readConcern: snapshot`, so items copied late reflect the same instant as items copied early. Without a store, PostgreSQL holds the table's catalog row shared from the first definition read until the data snapshot (`REPEATABLE READ`, pinned by a read of the table before the row is released) is taken, so the definition recorded is the one in force for the items read; the catalog and the data are separate databases, so this is a barrier, not one cross-database snapshot. SQLite reads all items in one deferred read transaction opened while the engine's write lock was held for the definition read, so writers continue during the copy; that long reader pins WAL checkpointing, so the WAL grows for the duration of the backup by the size of the backup rows written plus any concurrent writes. **MongoDB without a store** (`$out` from the live collection) and **Cassandra** (paging the live table) are not point-in-time: items copied early reflect the table as it was when the copy started, items copied late reflect writes made during the copy. A backup can therefore hold a write without an earlier write it followed, a state the table was never in. Measured *(earlier build)* with a writer that updated an early item and then a late item, pair after pair, during `CreateBackup`: 16 of 66 pairs on MongoDB and 123 of 800 on Cassandra had the late write without the early one; 0 on PostgreSQL (of 228) and SQLite (of 230). Back up a Cassandra table, or a MongoDB table on a server without a store, while it is not being written if the backup must be consistent | +| Consistency of a backup | Point-in-time snapshot of the table | **PostgreSQL** and **SQLite** take the items from one snapshot, and so does **MongoDB** with a configured `[backup.store]`. How *(from the code)*: with a store, the request opens the table snapshot (`SnapshotEngine`, the module docs of `crates/storage/src/snapshot.rs` and the backends' `snapshot_engine.rs` are the record): on PostgreSQL one `REPEATABLE READ` transaction holds the definition and the items on the shared layout, and on the separate layout the catalog row is held shared until the data transaction is pinned, the items then paged by the data table's primary key under that transaction, each page cut at 64 MiB of item text; on SQLite one deferred read transaction opened without the engine's write lock, paged by key with the same cut; on MongoDB a session with `readConcern: snapshot`, so items copied late reflect the same instant as items copied early. That snapshot is the majority-committed one, which can trail `DescribeTable`'s `local` read by the majority-commit lag, so a CreateBackup sent as soon as `DescribeTable` shows a new table `ACTIVE` could find it missing; the open then waits, at most 2 s, until the snapshot shows the table, and answers `TableNotFoundException` only for a table a `local` read does not show `ACTIVE` either. Without a store, PostgreSQL holds the table's catalog row shared from the first definition read until the data snapshot (`REPEATABLE READ`, pinned by a read of the table before the row is released) is taken, so the definition recorded is the one in force for the items read; the catalog and the data are separate databases, so this is a barrier, not one cross-database snapshot. SQLite reads all items in one deferred read transaction opened while the engine's write lock was held for the definition read, so writers continue during the copy; that long reader pins WAL checkpointing, so the WAL grows for the duration of the backup by the size of the backup rows written plus any concurrent writes. **MongoDB without a store** (`$out` from the live collection) and **Cassandra** (paging the live table) are not point-in-time: items copied early reflect the table as it was when the copy started, items copied late reflect writes made during the copy. A backup can therefore hold a write without an earlier write it followed, a state the table was never in. Measured *(earlier build)* with a writer that updated an early item and then a late item, pair after pair, during `CreateBackup`: 16 of 66 pairs on MongoDB and 123 of 800 on Cassandra had the late write without the early one; 0 on PostgreSQL (of 228) and SQLite (of 230). Back up a Cassandra table, or a MongoDB table on a server without a store, while it is not being written if the backup must be consistent | | `CreateBackup` of a table holding an `N` value the wire did not validate | Every stored value is a valid number, and a backup holds the table as it is | With a configured `[backup.store]` the backup fails. Four write paths of earlier builds stored an `N` value such as `1E126` or `abc` without validating it (#618: `BatchWriteItem`, `TransactWriteItems`, `UpdateItem` on an absent item in the key it creates, `ImportTable`), and the store-backed job validates every item before encoding it with the rule the restore applies (`validate_restorable_item`: every `N` and `NS` member a valid number, the nesting depth, the key shape (each key attribute present, of its declared type, not empty, an `N` key normalized), the item size against the instance's `max_item_size_bytes`, the hash and range key sizes and the attribute name sizes against the instance's limits, and for an item over the default item limit the data file line against the format's 32 MiB cap), so no backup that no restore can load is reported `AVAILABLE`. After the first refusal the job reads and validates the rest of the snapshot, logs each refused item with its key, attribute path and reason beside the backup ARN (up to 1,000, then a count), and fails the backup with a reason giving the count and the first item, naming #618 only when that item's refusal is an `N` value ("3 item(s) cannot be restored; first: key {...}, attribute amount: Number overflow... (#618)"); an item over `max_item_size_bytes` (stored under a higher limit) fails it the same way with a reason naming the size and the limit and not #618. The client sees `CREATING` and then `BackupNotFound`; the reason reaches the operator (`extenddb backup list --failed`, and the log, which a tenant's support case can cite; every tenant-supplied text in the line is written as a JSON string's content, control characters and the Unicode line separators U+0085, U+2028 and U+2029 escaped, so a newline or a line separator in an attribute name or a number value cannot forge a second log record); the way out is to rewrite or delete each named item through the wire (a bad range key cannot be rewritten, since a PutItem with that key is refused; `DeleteItem` removes it). The in-catalog create (no store, Cassandra) backs such a table up and restores it as before (wire tests on PostgreSQL and SQLite for each path) | +| Restoring into MongoDB an `N` key value past Decimal128 | Not applicable | A store-backed restore into the MongoDB backend of a backup holding an `N` sort-key value of more than 34 significant digits (the sort-key column is BSON Decimal128) fails: the restore job applies the file through the write path, whose `ValidationException` ("Numeric sort key value '' exceeds supported precision (Decimal128, 34 significant digits)") becomes the restore's `FAILED` reason, so the client sees the target `CREATING` and then gone and the server log carries the text. PostgreSQL, SQLite and Cassandra restore the same value (their sort-key columns are numeric or decimal). A non-key `N` of up to 38 digits is stored as the item's text and restores intact on every backend, MongoDB included *(wire tests against the MongoDB runner)* | | `ListBackups` paging | At most 100 summaries per page, with `LastEvaluatedBackupArn`; `Limit` 1 to 100. `TimeRangeLowerBound`, `TimeRangeUpperBound`, and `BackupType` filter the list. `TableName` is checked as the model says (3 to 255, `[a-zA-Z0-9_.-]+`, the "N validation error(s) detected" texts). An `ExclusiveStartBackupArn` of another account is `ValidationException` "Invalid Request: The supplied ExclusiveStartBackupArn does not match the account"; of another region, or of a table other than `TableName`'s, `ValidationException` ("does not match the region", "does not match the table name predicate"); a time bound before the epoch is `ValidationException`; a lower bound above the upper bound is `ValidationException` "Time range lower bound must be less than or equal to upper bound" (all measured) | The same paging on all four backends: `Limit` outside 1 to 100 is `ValidationException`, `ExclusiveStartBackupArn` resumes after the given backup, and summaries are ordered by creation time then ARN. Summaries carry `TableId`. `TimeRangeLowerBound` (inclusive) and `TimeRangeUpperBound` (exclusive) keep backups created in that range, and `BackupType` keeps backups of that type (`USER` when absent, every type for `ALL`); every ExtendDB backup is `USER`, so `SYSTEM` and `AWS_BACKUP` list nothing. The filters apply before paging, so `Limit` and the cursor count only backups that pass. Measured on SQLite and PostgreSQL with whole-second bounds between two backups taken 2 s apart; the millisecond edges are from the engine's unit tests. The request checks above are the service's, with its texts where captured: the account text in full; the region and table-name texts complete the captured fragments on the account text's pattern ("Invalid Request: The supplied ExclusiveStartBackupArn does not match the region" / "... the table name predicate"), checked in that order after the ARN's shape; a bound before the epoch is refused with ExtendDB's own text naming the member; equal bounds list nothing. An `ExclusiveStartBackupArn` that is not in the listing (excluded by the filters, or deleted since the previous page) is placed by the creation time in its id; one whose id carries no time (the 13-digit ids of earlier builds, with or without the hex suffix: only the 14-digit form carries one) is then a `ValidationException`. A cursor in another region is accepted only when it is the stored ARN of a backup of the account (or of `TableName`), which is what `LastEvaluatedBackupArn` hands out for a backup created under an earlier `server.region`, so paging passes such a backup; any other cursor in another region is refused ("does not match the region"). If that backup is deleted between two pages, its ARN is no longer a stored one and the next page is refused the same way (a same-region cursor would be placed by its time instead); start the listing again. Under a `server.region` outside `[a-z0-9-]` the cursor is still the stored ARN: a listing of backups created under a valid region pages, and a cursor that is the ARN of a backup an earlier release created under the invalid region is refused with "Invalid Request: BackupArn is not valid". `TableName` is a table name or a table ARN, as `CreateBackup` accepts it (the service accepts both, measured on DynamoDB): the ARN form lists what the name lists, another account's ARN is `AccessDeniedException`, and the cursor's table check runs against the resolved name. A numeric `TableName` is refused with the text the service uses for a map ("Start of structure or map found where not expected"), not measured for a number. Every `ListBackups` request reads all of the account's backups (or all of the table's, with `TableName`) from storage and pages them in memory, so walking N backups 100 at a time reads about N²/100 rows in total *(from the code)* | | Backup ARN and creation time | Backup id is 14-digit zero-padded epoch milliseconds, `-`, 8 hex digits; `BackupCreationDateTime` has sub-second precision | The same shape on all four backends; `BackupCreationDateTime` is the millisecond in the ARN | | Running two servers against one MongoDB catalog | Not applicable | MongoDB uses the fixed database names `extenddb_catalog` and `extenddb_data` whatever `--catalog-db` and `--data-db` say, and takes no serve lock: two servers on one replica set share one catalog and both start. Each server's reconciler acts only on a restore whose job record's heartbeat is older than `backup.stale_job_after_secs` (default 300 s) on the database's clock, so one server does not remove another live server's restore *(from the code and storage-level tests)*. Running more than one server per catalog is otherwise unsupported (see the deployment guide) | @@ -171,7 +172,7 @@ ExtendDB exposes runtime settings that have no DynamoDB equivalent: | `control_plane_delay_seconds` | 5 | Simulated delay for table state transitions (CREATING → ACTIVE, DELETING → removed) | | `index_propagation_delay_ms` | 10 | System-wide default propagation delay for asynchronous secondary-index maintenance (milliseconds), covering GSIs and vector indexes alike. Per-GSI overrides stored in catalog; vector indexes have no per-index override. 0 = synchronous. Accepts the pre-rename name `gsi_propagation_delay_ms` as a deprecated alias, and a catalog created before the rename keeps honouring a value stored under it. | | `throttling_enabled` | `true` | Enable provisioned capacity throttling (token bucket per table/partition) | -| `enable_multipart_keys` | `false` | Enable multi-part base table key extension | +| `limits.allow_multipart_table_keys` | `false` | Enable multi-part base table key extension | | `log_level` | `info` | Runtime log level (trace, debug, info, warn, error) | | `sqlx_log_level` | `warn` | Separate log level for sqlx query traces | | `allow_credential_import` | `true` | Allow importing credentials via the management API | diff --git a/docs/dynamodb-limits.md b/docs/dynamodb-limits.md index c2ac3d19b..5abb95599 100755 --- a/docs/dynamodb-limits.md +++ b/docs/dynamodb-limits.md @@ -128,13 +128,13 @@ Source: [AWS DynamoDB Service Quotas](https://docs.aws.amazon.com/amazondynamodb | Limit | DynamoDB Value | Status | Notes | |-------|---------------|--------|-------| -| Concurrent restores | 50 | Partial | Every backend: one server process runs at most `[backup] max_concurrent_jobs` (default 4, at most 1024) backup and restore jobs at once, `CreateBackup` and `RestoreTableFromBackup` together and not per account; a request over the bound is refused at once with `LimitExceededException` ("Too many backup and restore jobs are running on this server (at most N at once, the [backup] max_concurrent_jobs bound); retry later") and creates nothing, with no wait (storage-level and engine tests; measured on PostgreSQL with `max_concurrent_jobs = 1`: a running restore made a CreateBackup and a second restore refuse in under 60 ms). On PostgreSQL `pool_size` and `catalog_pool_size` are raised to at least `2 * max_concurrent_jobs + 10` so every admitted job has a connection. SQLite also bounds the snapshot slots `CreateBackup` and table snapshots (#395) share, a count derived from `[storage.sqlite] pool_size` (the formula is stated once, on that setting's documentation in `SqliteConfig`; 4 for the default pool) so half the pool stays free for ordinary requests; a server raises a smaller `pool_size` to `2 * (max_concurrent_jobs + 1)`, at most 34 (the slots are capped at 16), with a warning naming both numbers, so the slot bound refuses no job the `max_concurrent_jobs` permit admitted (no change at the defaults: a pool of 10 has 4 slots for 4 jobs); above 16 jobs a 17th concurrent job is refused by the slot. A `CreateBackup` that finds no slot free is refused at once with the same `LimitExceededException`, its parenthesis naming that bound and its count in place of `max_concurrent_jobs` ("at most N at once, the [storage.sqlite] pool_size snapshot-slot bound"); PostgreSQL bounds open table snapshots at half its (raised) pool the same way, naming `[storage.postgres] pool_size snapshot-slot`; a PostgreSQL job whose dedicated claim connection the database server refuses is refused with the same exception naming the server's connection slots ("the database server has no connection slot left for the job"), with no count; the same error while `extenddb backup purge` holds the catalog's maintenance claim names a backup maintenance command and the claim, not a bound. A MongoDB restore request that is not registered within 60 seconds answers an internal error while the restore continues | +| Concurrent restores | 50 | Partial | Every backend: one server process runs at most `[backup] max_concurrent_jobs` (default 4, at most 1024) backup and restore jobs at once, `CreateBackup` and `RestoreTableFromBackup` together and not per account; a request over the bound is refused at once with `LimitExceededException` ("Too many backup and restore jobs are running on this server (at most N at once, the [backup] max_concurrent_jobs bound); retry later") and creates nothing, with no wait (storage-level and engine tests; measured on PostgreSQL with `max_concurrent_jobs = 1`: a running restore made a CreateBackup and a second restore refuse in under 60 ms). On PostgreSQL `pool_size` and `catalog_pool_size` are raised to at least `2 * max_concurrent_jobs + 10` so every admitted job has a connection. SQLite also bounds the snapshot slots `CreateBackup` and table snapshots (#395) share, a count derived from `[storage.sqlite] pool_size` (the formula is stated once, on that setting's documentation in `SqliteConfig`; 4 for the default pool) so half the pool stays free for ordinary requests; a server raises a smaller `pool_size` to `2 * (max_concurrent_jobs + 1)`, at most 34 (the slots are capped at 16), with a warning naming both numbers, so the slot bound refuses no job the `max_concurrent_jobs` permit admitted (no change at the defaults: a pool of 10 has 4 slots for 4 jobs); above 16 jobs a 17th concurrent job is refused by the slot. A `CreateBackup` that finds no slot free is refused at once with the same `LimitExceededException`, its parenthesis naming that bound and its count in place of `max_concurrent_jobs` ("at most N at once, the [storage.sqlite] pool_size snapshot-slot bound"); PostgreSQL bounds open table snapshots at half its (raised) pool the same way, naming `[storage.postgres] pool_size snapshot-slot`; a PostgreSQL job whose dedicated claim connection the database server refuses is refused with the same exception naming the server's connection slots ("the database server has no connection slot left for the job"), with no count; the same error while `extenddb backup purge` holds the catalog's maintenance claim names a backup maintenance command and the claim, not a bound. A restore request on any backend that is not registered within 60 seconds (`RESTORE_REGISTRATION_TIMEOUT`) answers an internal error while the restore continues in the background. A restore from an S3 store holds the compressed backup size at `spool_path` while it runs; a restore's write batch on SQLite and Cassandra is bounded at 32 MiB of item text, measured on the serialized item (not on the service's `item_size_bytes`), and on Cassandra the item is serialized once more to measure it, a CPU cost per item the bound buys; a restore ends by counting the target against the manifest's item total, which on Cassandra is a full `COUNT(*)` of the restored table (one scan of a CREATING table, run once per restore and once more on a resume without recorded progress; no run has measured its cost past 12,000 rows, the largest Cassandra restore measured) | | `BackupName` and `TableName` on `CreateBackup`, `TableName` on `ListBackups` | Length 3 to 255, `[a-zA-Z0-9_.-]+` | Enforced | The service model's constraints, with its texts ("1 validation error detected: Value '' at 'backupName' failed to satisfy constraint: ...", "2 validation errors detected: ...; ..." when length and pattern both fail). Length is counted in characters. `BackupName` takes the model's 3 to 255; a `TableName` member takes the deployment's `[limits]` `min_table_name_length` and `max_table_name_length` (defaults 3 and 255), as `CreateTable` does, so a table the deployment can create can be listed, and can be backed up unless no backup ARN can carry its name: a name over 255 bytes (with `max_table_name_length` raised) or made only of dots can be created but `CreateBackup` refuses it with `ValidationException` "TableName cannot be backed up: a backup ARN cannot be formed for it: invalid backup ARN: table name of N bytes is longer than 255" or "... invalid backup ARN: table name \"...\" is only dots" (the service backs up any table it can create; see the differences document's backup creation row). Both members may also be the table's ARN. The value is echoed in the message as sent | | `BackupArn` | 37 to 1024 characters | Maximum enforced; minimum implied | Surrounding ASCII whitespace is trimmed first, then over 1024 bytes, not of the backup ARN shape, a control or non-ASCII character, or a region outside `[a-z0-9-]`, an account that is not digits, or a table name outside `[a-zA-Z0-9_.-]` is `ValidationException` "Invalid Request: BackupArn is not valid"; the right shape with an id that is not a backup id is "Invalid Backup ARN"; a digit-only account that is not the caller's is `AccessDeniedException`. No minimum is checked: the shortest string of the right shape is longer than 37 characters. Another region of the caller's account resolves (the server's region is tried first, then the ARN as written). `ListBackups`'s `ExclusiveStartBackupArn` must also match the account, the region, and `TableName` when given | | `ListBackups` page size | 100 | Enforced | `Limit` 1 to 100, default 100; more rows come back through `LastEvaluatedBackupArn` and `ExclusiveStartBackupArn`. Each request reads every backup of the account (or of the table, with `TableName`) and pages in memory, so walking N backups reads about N²/100 rows. `TimeRangeLowerBound`, `TimeRangeUpperBound`, and `BackupType` filter the list before paging; a bound before the epoch, or a lower bound above the upper bound ("Time range lower bound must be less than or equal to upper bound"), is `ValidationException` | | Backup store | A backup lives in the service's own storage | N/A | `[backup.store]` selects where CreateBackup writes its backups on PostgreSQL, SQLite and MongoDB (Cassandra, and any backend without a store, writes the in-catalog format) and where restores and deletes of store-backed backups read: a filesystem root (`backups/` beside a SQLite database file when the section is absent), an S3 bucket and prefix, or `disabled`. The knobs are `[backup]`: `data_file_target_bytes` 64 MiB (minimum 32 MiB), `heartbeat_interval_secs` 15, `stale_job_after_secs` 300 (at least three heartbeats), `failed_row_retention_secs` 7 days, `reconcile_interval_secs` 60, `max_concurrent_jobs` 4 per instance, `spool_path` (required with an S3 store except beside a SQLite database file). A filesystem store holds every backup's compressed size under its root, beside the database on the SQLite default, and `extenddb destroy` leaves it; a restore from an S3 store holds the compressed backup size at `spool_path` while it runs. Instances that share a catalog must share one store root; an unhealthy store refuses CreateBackup where it writes to the store and the restore and delete of store-backed backups with `ValidationException` naming the store's kind, and the server keeps serving (`/health` reports `degraded`). See the admin guide's `[backup.store]` | | Backup data file | Not specified | N/A | A data file holds one item per line as DynamoDB JSON, gzip-compressed, in key order on a table whose key schema is orderable (`item_order` in the manifest); a line is at most 32 MiB (`MAX_LINE_BYTES`, which holds the largest line a 400 KB item can expand to, 27.7 MB); a file closes at the first item that takes its uncompressed bytes to or past `[backup] data_file_target_bytes` (default 64 MiB, minimum 32 MiB, the line cap), so it exceeds the knob by less than one line. The snapshot streams on PostgreSQL and SQLite page the table by key and cut each page at 64 MiB of item text (`SNAPSHOT_FETCH_BYTES`, a constant), so a page holds at most that plus one row; MongoDB's batches are bounded by the server's 16 MiB reply | -| Backup contents | Key schema, attribute definitions, items, indexes, billing mode, provisioned throughput, on-demand throughput (table and GSI level), SSE. Table class is not carried (a restore is `STANDARD`) | Partial | ExtendDB restores the same set, except a GSI's on-demand throughput (it records none per index); table class is not restored, as on the service. Streams, TTL, tags, and deletion protection are not restored, as on the service (a store-backed backup's manifest records the stream specification and TTL for `DescribeBackup`). With a configured store on PostgreSQL, SQLite and MongoDB the backup is written under the store's root (the differences document's "Where backups live" row); without one, and on Cassandra, it is stored inside the database the server uses, so take database-level backups for off-host copies there | +| Backup contents | Key schema, attribute definitions, items, indexes, billing mode, provisioned throughput, on-demand throughput (table and GSI level), SSE. Table class is not carried (a restore is `STANDARD`) | Partial | ExtendDB restores the same set, except a GSI's on-demand throughput (it records none per index); table class is not restored, as on the service. Streams, TTL, tags, and deletion protection are not restored, as on the service (a store-backed backup's manifest records the stream specification and TTL for `DescribeBackup`). With a configured store on PostgreSQL, SQLite and MongoDB the backup is written under the store's root and restores into the backend that took it, and into another backend once `backup sync --adopt-store` (a later release) has indexed the backup in the target's catalog; this release restores the backups its own catalog indexes (the differences document's "Where backups live" row). Vector indexes are restored where the target supports them, and a backup of a multi-part-key table (the `limits.allow_multipart_table_keys` preview) restores with its item set intact (the engine's `a_multi_part_key_manifest_is_registered` and `tests/test_backup_job.py`'s multi-part cell, which restores such a backup on a server with the preview enabled and compares the restored item set with the source's); without one, and on Cassandra, it is stored inside the database the server uses, so take database-level backups for off-host copies there | | Point-in-time recovery | 35-day window | N/A | `RestoreTableToPointInTime` is refused. `UpdateContinuousBackups` accepts enabling point-in-time recovery and reports a 35-day restorable window, but nothing is recorded (see differences-from-dynamodb.md) | | `RestoreTableFromBackup` overrides | `BillingModeOverride`, `ProvisionedThroughputOverride`, `GlobalSecondaryIndexOverride`, `LocalSecondaryIndexOverride`, `SSESpecificationOverride`, `OnDemandThroughputOverride`, `VectorIndexOverride` | Partial | All but `VectorIndexOverride` are applied as the service applies them: an index override list replaces the backup's indexes of that kind (empty restores none, a subset restores those, a GSI entry's throughput is the restored index's), `OnDemandThroughputOverride` is the restored table's on-demand throughput on an on-demand restore. Refused with `ValidationException` before the target is created: an index the backup does not have or a changed index definition, a throughput member contradicting the billing mode, `{0, 0}`, a `PROVISIONED` restore of a GSI without throughput ("Invalid Request: Must override ProvisionedThroughput if BillingMode is overridden to PROVISIONED for index "), a GSI entry's `OnDemandThroughput` on a `PROVISIONED` restore (on an on-demand restore it is accepted and not recorded), and `VectorIndexOverride`. See differences-from-dynamodb.md | diff --git a/docs/manuals/03-usage-guide.md b/docs/manuals/03-usage-guide.md index 2f78383b2..886cf508e 100755 --- a/docs/manuals/03-usage-guide.md +++ b/docs/manuals/03-usage-guide.md @@ -571,7 +571,7 @@ extenddb reproduces DynamoDB error responses exactly. Common errors: | PartiQL | ✗ (not planned) | | DAX | ✗ (not applicable) | | Global Tables | ✗ (future) | -| On-demand backups (CreateBackup, RestoreTableFromBackup) | Partial: CreateBackup ✓ (to the configured filesystem or S3 backup store on PostgreSQL, SQLite and MongoDB; in-catalog without one and on Cassandra); RestoreTableFromBackup ✓ for in-catalog backups only, store-backed backups from a later release (their restore answers a `ValidationException` naming the store this release; the upgrade manual) | +| On-demand backups (CreateBackup, RestoreTableFromBackup) | ✓: CreateBackup to the configured filesystem or S3 backup store on PostgreSQL, SQLite and MongoDB (in-catalog without one and on Cassandra); RestoreTableFromBackup from the store into every backend (a filesystem or S3 store; a backup taken on one backend restores into another once `backup sync --adopt-store` (a later release) has indexed the backup in the target's catalog; this release restores the backups its own catalog indexes), and of in-catalog backups | | PITR | ✗ (future) | | Import/Export (local filesystem) | ✓ (FileSource instead of S3) | diff --git a/docs/manuals/05-admin-guide.md b/docs/manuals/05-admin-guide.md index 53083a8df..be1abeea1 100755 --- a/docs/manuals/05-admin-guide.md +++ b/docs/manuals/05-admin-guide.md @@ -148,11 +148,11 @@ The knobs backup and restore jobs and the reconciler read; every key has a defau |-----|---------|-------------| | `data_file_target_bytes` | `67108864` | Uncompressed bytes per backup data file (minimum 32 MiB, the cap on one line): a data file closes at the first item that takes it to or past this, so it exceeds it by less than one line | | `heartbeat_interval_secs` | `15` | The job heartbeat timer. The store-backed backup job heartbeats every `max(heartbeat_interval_secs, 15)` s and starts no store write unless a beat confirmed its claim within two periods, the claim's insert counting as the first beat; the in-catalog backup and restore paths heartbeat every 15 s whatever this says | -| `stale_job_after_secs` | `300` | Heartbeat age after which a job's owner is dead (on PostgreSQL, once the owner's claim session has ended as well) and the reconciler resolves the job: a dead backup job with a complete manifest under its prefix is AVAILABLE, one without is FAILED and its prefix deleted; a dead restore job is FAILED and its target removed. Must be at least three heartbeat periods, where the period is the larger of `heartbeat_interval_secs` and the 15 s the in-catalog jobs heartbeat at (so 45 at the least, whatever `heartbeat_interval_secs` says); on SQLite a long write transaction delays a heartbeat, so keep a margin above that floor | +| `stale_job_after_secs` | `300` | Heartbeat age after which a job's owner is dead (on PostgreSQL, once the owner's claim session has ended as well) and the reconciler resolves the job: a dead backup job with a complete manifest under its prefix is AVAILABLE, one without is FAILED and its prefix deleted; a dead in-catalog restore job is FAILED and its target removed; a dead store-backed restore is resumed from its recorded progress (the differences document's restore-interruption row). Must be at least three heartbeat periods, where the period is the larger of `heartbeat_interval_secs` and the 15 s the in-catalog jobs heartbeat at (so 45 at the least, whatever `heartbeat_interval_secs` says); on SQLite a long write transaction delays a heartbeat, so keep a margin above that floor | | `failed_row_retention_secs` | `604800` | How long FAILED and DELETED backup rows of the configured store are kept once their objects are gone (7 days); `extenddb backup list --failed` prints the FAILED ones with their reason | | `reconcile_interval_secs` | `60` | The reconciler's cycle; it also runs once at startup. Each cycle checks the store, resolves backup and restore jobs whose owner is dead, finishes deletions, deletes the objects of FAILED backups once their heartbeat is older than `stale_job_after_secs`, and removes tombstones past `failed_row_retention_secs` (the differences document's restore-interruption row lists the duties) | -| `max_concurrent_jobs` | `4` | Backup and restore jobs this instance runs at once (at most 1024); a request over it is `LimitExceededException`. On PostgreSQL the pool sizes are raised to `2 * max_concurrent_jobs + 10` when smaller; on SQLite `[storage.sqlite] pool_size` is raised to `2 * (max_concurrent_jobs + 1)`, at most 34, when its snapshot slots (`pool_size / 2 - 1`, between 1 and 16) would refuse a job this knob admits. Memory: a running backup holds about 490 MiB (PostgreSQL) or 650 MiB (SQLite) at most in the pipeline's own terms for a table of items at the 32 MiB line cap, a few tens of MiB for 200-byte items (the figures are for the default `max_item_size_bytes`, 400 KB, where no stored item's line exceeds the cap; on an instance that raises it the terms are at the largest line a stored item expands to, about 68 times the item's size for the worst shape); with glibc's default allocator the process's high-water mark grows with the number of large items a job reads (measured 760 MiB over idle at 48 items of 27.7 MB on a quiet host and 1,281 MiB on a loaded one, 1.39 and 2.35 times the terms at that item size) because the items' strings are allocated on one thread and freed on another and the per-thread arenas keep that memory. Set `MALLOC_ARENA_MAX=1` in the server's environment (the deployment guide's service unit carries the line, commented out) on a host that backs up large items: the peak is then the terms' and does not grow with the table (measured 452 to 481 MiB over idle at 12, 24 and 48 items of 27.7 MB on SQLite, in two series on two hosts' loads, and 340 MiB `VmHWM` (about 200 MiB over idle) at 12 items on PostgreSQL); one arena serializes the worker threads' allocations on one lock and costs the request path a few percent (measured 3.5 to 9.2% on GetItem, Query and Scan on SQLite and about 2% on PostgreSQL (the scans, 2.1%), where the request path waits on the database on PostgreSQL). Size the host for 1.2 times the terms (about 590 MiB on PostgreSQL, 780 MiB on SQLite) times this knob under `MALLOC_ARENA_MAX=1`; without it the peak grows with the large rows a job reads and no bound was measured (1,281 MiB over idle at 48 items of 27.7 MB is the largest seen) | -| `spool_path` | (unset) | Restore download area for an S3 store, created `0700` with files `0600`, holding a `.lock` another instance on the host refuses. Required with an S3 store, except on file-backed SQLite where it defaults to `backup-spool/` beside the database file. Refused inside or containing a filesystem store root, and when it is or contains the database file's directory | +| `max_concurrent_jobs` | `4` | Backup and restore jobs this instance runs at once (at most 1024); a request over it is `LimitExceededException`. On PostgreSQL the pool sizes are raised to `2 * max_concurrent_jobs + 10` when smaller; on SQLite `[storage.sqlite] pool_size` is raised to `2 * (max_concurrent_jobs + 1)`, at most 34, when its snapshot slots (`pool_size / 2 - 1`, between 1 and 16) would refuse a job this knob admits. Memory: a running backup holds about 490 MiB (PostgreSQL) or 650 MiB (SQLite) at most in the pipeline's own terms for a table of items at the 32 MiB line cap, a few tens of MiB for 200-byte items (the figures are for the default `max_item_size_bytes`, 400 KB, where no stored item's line exceeds the cap; on an instance that raises it the terms are at the largest line a stored item expands to, about 68 times the item's size for the worst shape); with glibc's default allocator the process's high-water mark grows with the number of large items a job reads (measured 760 MiB over idle at 48 items of 27.7 MB on a quiet host and 1,281 MiB on a loaded one, 1.39 and 2.35 times the terms at that item size) because the items' strings are allocated on one thread and freed on another and the per-thread arenas keep that memory. Set `MALLOC_ARENA_MAX=1` in the server's environment (the deployment guide's service unit carries the line, commented out) on a host that backs up large items: the peak is then the terms' and does not grow with the table (measured 452 to 481 MiB over idle at 12, 24 and 48 items of 27.7 MB on SQLite, in two series on two hosts' loads, and 340 MiB `VmHWM` (about 200 MiB over idle) at 12 items on PostgreSQL); one arena serializes the worker threads' allocations on one lock and costs the request path a few percent (measured 3.5 to 9.2% on GetItem, Query and Scan on SQLite and about 2% on PostgreSQL (the scans, 2.1%), where the request path waits on the database on PostgreSQL). Size the host for 1.2 times the terms (about 590 MiB on PostgreSQL, 780 MiB on SQLite) times this knob under `MALLOC_ARENA_MAX=1`; without it the peak grows with the large rows a job reads and no bound was measured (1,281 MiB over idle at 48 items of 27.7 MB is the largest seen). A running restore holds the parsed manifest, one decoded data-file line and its parse, the decoded items of 32 MiB of data-file text plus one line per chunk (the chunk closes with the item that reaches the bound) at four chunk positions between the decoder and the load (two in the channel, the one being filled, the one being drained; a chunk also closes at 1,000 items, and the chunks never hold more than one data file's items), and the backend's write batch: at most 32 MiB of item text on SQLite and Cassandra (10,000 items when they are smaller), held beside the decoded items of that text; 500 items (or 4 MiB) per INSERT statement inside one transaction per data file on PostgreSQL; 1,000 documents per `insert_many` on MongoDB (the driver splits the call); times this knob. For a table of large items that is the same order as the backup, not less: measured `VmHWM` over idle from a fresh start (`docs/design/15-backup-restore-bench-pr9.md`), 24 items of 27.7 MB restore with a peak of 848 MiB on SQLite and 778 MiB on PostgreSQL under the default allocator (848 and 1,013 MiB at 12 and 48 such items on SQLite: the peak grows with the large items a restore reads, as the backup's does, and no bound was measured), and 392 MiB on SQLite and 427 MiB on PostgreSQL under `MALLOC_ARENA_MAX=1`, the same setting as above; 1,200 items of 350 KB in one data file 270 MiB on SQLite and 156 MiB on PostgreSQL; 1M items of 200 bytes under a composite key with no secondary index, the backup's lines in key order as the server's own backups are, 89 MiB on SQLite (the write connection's page cache and the final `COUNT(*)`, each bounded by `cache_size`; the count's share was 22 to 28 MiB at 1M and at 3M rows) and 16 MiB on PostgreSQL, and the same 1M items as an unordered backup (every MongoDB-sourced backup and every multi-part-key table) 144 to 164 MiB on SQLite in 66 to 90 s against 27 to 56 s key-ordered (scattered inserts dirty the whole base b-tree; the restore's buffers are the same) and 16 MiB on PostgreSQL; the same items under the same key with two full-projection GSIs 143 to 225 MiB on SQLite at 100,000 items in three runs and 165 to 172 MiB at 300,000, and 23 to 34 MiB on PostgreSQL at both: the excess on SQLite is the backend's own per-connection page cache (`cache_size`, 32 MiB) holding the index tables' b-tree pages plus the allocator's arenas (about a third less under `MALLOC_ARENA_MAX=1`), not the restore's buffers, which the 32 MiB batch bound holds whatever the indexes; it is bounded by `cache_size` and grows sub-linearly with the table. Size the host for about 450 MiB per running restore of large items under `MALLOC_ARENA_MAX=1`, times this knob, plus about 32 MiB of page cache per running restore of an indexed table on SQLite, and read the 1M figure for 200-byte items as the key-ordered shape's: an unordered backup of them restores in about 1.5 to 1.8 times that on SQLite | +| `spool_path` | (unset) | Restore download area for an S3 store, created `0700` with files `0600`, holding a `.lock` another instance on the host refuses. A store-backed restore from an S3 store downloads each data file here under `restore-
/` (`.json.gz`, `` zero-padded to six digits, with a `.verified` marker holding the file's sha256, written only after the file's digests matched and trusted only while it names the digest the manifest records, plus the manifest and sidecar), reads it from there, and removes the entry when the restore ends, except when it ended because the record had become another instance's (its claim lost, a status write that matched no record), where the sweep removes it; an entry left by an earlier run of this host for a record another host then owned is removed before a new run of the same restore uses it; the reconciler sweeps the `restore-` entry of a restore no longer RUNNING with this host's `spool_host`. Its disk cost is the compressed backup of the restore in flight. A filesystem store needs no spool (files are hashed in place and read again). Required with an S3 store, except on file-backed SQLite where it defaults to `backup-spool/` beside the database file. Refused inside or containing a filesystem store root, and when it is or contains the database file's directory | #### [backup.store] @@ -164,13 +164,24 @@ The store CreateBackup writes to on PostgreSQL, SQLite and MongoDB, and the one | `filesystem.path` | `backups/` beside a SQLite database file | The root directory of a filesystem store. It must exist, be owned by the extenddb user with mode `0700`, and sit under directories owned by root or that user which others cannot write to; `validate` refuses anything else at startup. A relative path, like a relative `spool_path`, is resolved against the working directory the server was started from (the daemon's own working directory is `/`) | | `s3.bucket`, `s3.prefix`, `s3.region`, `s3.endpoint`, `s3.force_path_style`, `s3.allow_insecure_endpoint`, `s3.upload_part_size_bytes`, `s3.max_concurrent_uploads` | | The S3 store; see the crate's `config` module doc. Credentials come from the AWS SDK's chain and are never in the file | -When the section is absent, the backend decides: on file-backed SQLite it is a filesystem store at `backups/` beside the database file, created `0700` if missing, whose validation or identity-check failure is a logged warning that leaves the store unconfigured rather than a startup error or an unhealthy store while the catalog has no backup rows for that root (so an upgrade never stops a server that never configured backups, and CreateBackup keeps working on one); once the catalog has rows for the default root, the root has been used and is handled as a configured one: a failure of its identity check or manifest probe is an unhealthy store (reported on `/health`, its rows' restores and deletes refused, `backup purge` refusing on it rather than reporting no store), and a refusal of its `open` or validation a startup error naming the root; on PostgreSQL, MongoDB, Cassandra and an in-memory SQLite database there is no default and the store is unconfigured, logged at startup. `kind = "disabled"` leaves it unconfigured on purpose on any backend. Without a configured store, CreateBackup writes the in-catalog backup format and logs a deprecation warning naming `[backup.store]` on each call. With one configured, CreateBackup on PostgreSQL, SQLite and MongoDB writes the backup to the store: the request takes the table's snapshot, writes the catalog row and answers `CREATING`, and a job of the server uploads gzip data files and the manifest under `/
//` below the root and marks the row `AVAILABLE` (the job's log lines carry the ARN: `backup : started, items in the table as last counted by the stats pass, key-ordered` (or `unordered`, for a table whose key shape or column collation has no format order) at the start, the stats pass's count, which is 0 on a table the pass has not reached, and `backup : AVAILABLE, items, bytes written in s` at the end, with the backup's own count); Cassandra, which has no table snapshot, keeps the in-catalog create with a store configured. Backups of either format stay listable, describable and deletable, and in-catalog backups restorable; the restore of a store-backed backup is a later release's and answers `ValidationException` until then. +When the section is absent, the backend decides: on file-backed SQLite it is a filesystem store at `backups/` beside the database file, created `0700` if missing, whose validation or identity-check failure is a logged warning that leaves the store unconfigured rather than a startup error or an unhealthy store while the catalog has no backup rows for that root (so an upgrade never stops a server that never configured backups, and CreateBackup keeps working on one); once the catalog has rows for the default root, the root has been used and is handled as a configured one: a failure of its identity check or manifest probe is an unhealthy store (reported on `/health`, its rows' restores and deletes refused, `backup purge` refusing on it rather than reporting no store), and a refusal of its `open` or validation a startup error naming the root; on PostgreSQL, MongoDB, Cassandra and an in-memory SQLite database there is no default and the store is unconfigured, logged at startup. `kind = "disabled"` leaves it unconfigured on purpose on any backend. Without a configured store, CreateBackup writes the in-catalog backup format and logs a deprecation warning naming `[backup.store]` on each call. With one configured, CreateBackup on PostgreSQL, SQLite and MongoDB writes the backup to the store: the request takes the table's snapshot, writes the catalog row and answers `CREATING`, and a job of the server uploads gzip data files and the manifest under `/
//` below the root and marks the row `AVAILABLE` (the job's log lines carry the ARN: `backup : started, items in the table as last counted by the stats pass, key-ordered` (or `unordered`, for a table whose key shape or column collation has no format order) at the start, the stats pass's count, which is 0 on a table the pass has not reached, and `backup : AVAILABLE, items, bytes written in s` at the end, with the backup's own count); Cassandra, which has no table snapshot, keeps the in-catalog create with a store configured. Backups of either format stay listable, describable, deletable and restorable: a store-backed backup is restored by the restore job (the differences document's "How `RestoreTableFromBackup` runs" row; the `spool_path` row above and the `remove-target` command below are its operator surface), an in-catalog one by the previous path. Startup opens the configured store and, for a filesystem store, validates it and checks its identity before the listener is up (bounded by 5 s: a mount that stops answering after the root is opened leaves the server starting with the store `checking`; a root that does not answer at startup holds the server before the listener, as the database directory would, with the port bound and silent); an S3 store's validation and identity check are requests to the endpoint, and run after the listener is up, in the reconciler's first cycle, so an endpoint that stalls never holds the data plane down (the component is `checking` until the first result, and the operations that touch the store refuse meanwhile as they do while it is unhealthy). A configuration error, every one decided without a request to the store, stops startup with the key, path or bucket in the message: a root that does not exist or fails the owner-only rule, a filesystem root another user can reach, a filesystem root that is or contains the database directory, a bucket name the SDK routes to S3 Express, a bad part size or prefix, an insecure endpoint, an S3 store without `spool_path` on a backend with no default, a `spool_path` that breaks its placement rule, that another live instance holds or that exists with a mode other than `0700`. A transport failure (no region, no credentials, an endpoint that refuses connections or does not answer within the check's bound), a missing or denied bucket, an I/O failure of a filesystem root's probe write or resolution (a full or read-only disk, a stale network handle), or a failed identity check leaves the server running with the store unhealthy: the problem is logged once at error, `/health` reports the `backup_store` component unhealthy (status `degraded`), CreateBackup (where it writes to the store) and the restore and delete of a store-backed backup are refused with a `ValidationException` that names the store's kind and the problem's category and not its root, the reconciler acts on no row of the store, and the check runs again each `reconcile_interval_secs` until it passes, at which point the server logs `is healthy again` and resumes without a restart. Each check is bounded (5 s for a filesystem root, 30 s for a bucket): a store that has stopped answering costs a reconciler cycle that bound and nothing more, the cycle's other duties run as before, and the abandoned check is logged; on a filesystem root the abandoned check's call stays parked on one thread until the mount answers, no later cycle starts a check while it is (logged each cycle, the store staying `unreachable`), and `stop` or SIGTERM still ends the server, the runtime giving that thread 5 s before the process exits without it (after the job wait of up to 30 s when a backup job is running; see Stopping). CreateBackup on Cassandra is not refused by an unhealthy store: its in-catalog create touches no store. The store identity. The root carries `extenddb-store.json`: a random store id and the id of the catalog that owns the root. Startup creates it when the root has no file and the catalog has no rows for the root, and otherwise reads it and compares: a file naming another catalog, a store id that differs from the one the catalog's rows for the root carry, a root with no file while rows exist, or rows carrying more than one id each mark the store unhealthy. Instances that share a catalog must share one store root (one S3 bucket and prefix, or one shared mount): host-local roots configured at the same path give each instance different bytes under one root label, so each check also `head`s the manifest of the newest AVAILABLE backup and marks the store unhealthy when it is absent, which is what a second instance with a local copy of the directory sees. The file is left by `backup purge` and by every deletion; remove it only when retiring the root, or after `extenddb destroy`: a catalog re-created with `init` has a new id, and the old root's file names the old one, so the re-initialised server reports the store owned by another catalog until the file is removed or a new root configured (on the default SQLite root, that is the logged warning above and CreateBackup keeps the in-catalog path). While the catalog has no rows for a root, two host-local directories at one path cannot be told apart (each instance creates its own file and id); point every instance at one shared root before the first backup. The identity file is read up to 64 KiB; a larger object under its key is reported unreadable. -`extenddb destroy` removes the databases and leaves the store untouched: it is the copy that survives destroy. `extenddb backup list [--failed]` prints the catalog's rows for the configured store's root, every store id they carry, with each row's `store_root` and `store_id`; it is a read that runs beside a server (on SQLite it takes no serve lock), reads the identity file without creating one, and runs on an unhealthy store, naming the problem in its header. `extenddb backup purge --confirm` deletes every backup of the configured store, each backup's objects (its own prefix, never a whole account prefix, so a prefix the catalog has no row for survives) and its catalog row outright, leaving no tombstone, and leaves `extenddb-store.json` and the `exports/` component; it needs the store healthy. It takes the catalog-level maintenance claim first and heartbeats it for the command's whole run, waits up to `2 * stale_job_after_secs` until no store-backed CreateBackup is running on any instance (a CREATING or FAILED row with a heartbeat within `stale_job_after_secs` on the catalog database's clock) and no restore is RUNNING, refuses and releases the claim if they have not finished, deletes a CREATING row the wait saw no live owner for as it deletes any other row, finishes any DeleteBackup left in progress as part of the purge (a backup a DeleteBackup finishes while the purge runs is counted as deleted, not left), and logs each deletion under the `extenddb::audit` target with the account and the ARN. While the claim is held, CreateBackup and RestoreTableFromBackup on every instance answer `LimitExceededException` saying that a backup maintenance command holds the catalog's maintenance claim (the message does not say which command; the second command's refusal and the claim's row do), and a second maintenance command fails at once naming the holder. Without `--confirm` it prints the plan, changes nothing and exits 0. `--config` is accepted before or after the subcommand. On SQLite the command opens the database as `serve` does and so takes the serve lock: stop the server first (while one runs on the file the command refuses with exit 6, as a second `serve` does); under that lock it first resolves what the stopped server left (a RUNNING restore is failed and its target removed, the CREATING target of a FAILED restore removed, an in-catalog CREATING backup removed, a DELETING row finished) and deletes a CREATING or FAILED row whatever its heartbeat says (the server that wrote it is stopped, and its clock is this host's), so such a job never makes it wait or refuse; the one exception is a RUNNING restore of kind `manifest`, which no release before PR 9a writes: the command leaves it for the reconciler that resumes it, waits on it as on any RUNNING restore and refuses naming the table; and a server starting on the file removes a claim a killed purge left, so no job is refused for the stale window, as a purge run again after the killed one takes its claim over. On the other backends it runs beside live servers, and a claim whose holder died expires after `stale_job_after_secs`. On PostgreSQL both commands open the catalog as a server does and hold a serving registration for their run, whose session is named after the command (`extenddb backup list`, `extenddb backup purge`): `extenddb migrate --yes` waits for them as for a server and names them while it waits, and while a migration runs they refuse at once (exit 6), as they do on a catalog at another migration than the binary's (exit 3, with the `extenddb migrate --yes` advice `serve` prints). On every backend Ctrl-C or `SIGTERM` makes the command release the claim and exit: at once while it waits, and before its next backup while it deletes, so an interrupted purge leaves no half-deleted backup and no claim to wait out; run it again for the rest. +`extenddb destroy` removes the databases and leaves the store untouched: it is the copy that survives destroy. `extenddb backup list [--failed]` prints the catalog's rows for the configured store's root, every store id they carry, with each row's `store_root` and `store_id`; it is a read that runs beside a server (on SQLite it takes no serve lock), reads the identity file without creating one, and runs on an unhealthy store, naming the problem in its header. It lists backups only, never restore jobs: a restore's record is the job's log line, not a `backup list` row. `extenddb backup purge --confirm` deletes every backup of the configured store, each backup's objects (its own prefix, never a whole account prefix, so a prefix the catalog has no row for survives) and its catalog row outright, leaving no tombstone, and leaves `extenddb-store.json` and the `exports/` component; it needs the store healthy. It takes the catalog-level maintenance claim first and heartbeats it for the command's whole run, waits up to `2 * stale_job_after_secs` until no store-backed CreateBackup is running on any instance (a CREATING or FAILED row with a heartbeat within `stale_job_after_secs` on the catalog database's clock) and no restore is RUNNING, refuses and releases the claim if they have not finished, deletes a CREATING row the wait saw no live owner for as it deletes any other row, finishes any DeleteBackup left in progress as part of the purge (a backup a DeleteBackup finishes while the purge runs is counted as deleted, not left), and logs each deletion under the `extenddb::audit` target with the account and the ARN. While the claim is held, CreateBackup and RestoreTableFromBackup on every instance answer `LimitExceededException` saying that a backup maintenance command holds the catalog's maintenance claim (the message does not say which command; the second command's refusal and the claim's row do), and a second maintenance command fails at once naming the holder. Without `--confirm` it prints the plan, changes nothing and exits 0. `--config` is accepted before or after the subcommand. On SQLite the command opens the database as `serve` does and so takes the serve lock: stop the server first (while one runs on the file the command refuses with exit 6, as a second `serve` does); under that lock it first resolves what the stopped server left (a RUNNING restore is failed and its target removed, the CREATING target of a FAILED restore removed, an in-catalog CREATING backup removed, a DELETING row finished) and deletes a CREATING or FAILED row whatever its heartbeat says (the server that wrote it is stopped, and its clock is this host's), so such a job never makes it wait or refuse. That includes a RUNNING restore from a backup store, which a server would resume: under the serve lock the command fails it and removes its target, as for an in-catalog restore (a resume is a job, which the maintenance claim excludes), except one that recorded no progress while its target holds items, which is failed `target_not_empty` and its target left for `extenddb backup remove-target`. A server starting on the file removes a claim a killed purge left, so no job is refused for the stale window, as a purge run again after the killed one takes its claim over. On the other backends it runs beside live servers, and a claim whose holder died expires after `stale_job_after_secs`. On PostgreSQL the backup commands (`list`, `purge` and `remove-target` below) open the catalog as a server does and hold a serving registration for their run, whose session is named after the command (`extenddb backup list`, `extenddb backup purge`, `extenddb backup remove-target`): `extenddb migrate --yes` waits for them as for a server and names them while it waits, and while a migration runs they refuse at once (exit 6), as they do on a catalog at another migration than the binary's (exit 3, with the `extenddb migrate --yes` advice `serve` prints). On every backend Ctrl-C or `SIGTERM` makes the command release the claim and exit: at once while it waits, and before its next backup while it deletes, so an interrupted purge leaves no half-deleted backup and no claim to wait out; run it again for the rest. + +`extenddb backup remove-target --account
` removes a restore target that `DeleteTable` refuses (a `CREATING` table) together with its restore record: the target of a `FAILED` restore whatever the reason (`target_not_empty` included: the state a resume leaves when it finds items in the target and no recorded progress, which on MongoDB and Cassandra an ordinary crash before the first progress record produces, anywhere in the first data file on MongoDB and in the first 10,000 items or 32 MiB on Cassandra; until the command runs that target keeps its backup in use, so `DeleteBackup` and a second restore of the backup are `BackupInUseException` and the name is held), the target of a `RUNNING` restore whose owner is dead (no heartbeat within `stale_job_after_secs` on the database's clock, the listing the reconciler uses; on SQLite under the serve lock every owner is dead), and a legacy (in-catalog) restore's target the same way. It refuses a `RUNNING` restore with a live owner ("stop it or wait for it"), a `DONE` restore and a table no restore record names, the latter two as "not a restore target", and exits 1. It needs no store (it runs whether or not `[backup.store]` is configured, and creates neither the default root nor its identity file), takes the catalog-level maintenance claim first as `purge` does and releases it before it returns, logs the removal under `extenddb::audit` with the account, the table and the backup's ARN, and prints what it did: + +``` +=== extenddb backup remove-target === +Taking the maintenance claim as "extenddb backup remove-target on (pid )". +removed the restore target
(account , table id
) and its record: its manifest restore from was FAILED (target_not_empty) +The maintenance claim is released. +``` + +The third line's tail names the record's state: `was FAILED ()`, `was RUNNING with a dead owner`, or `its legacy restore from ` for an in-catalog restore. A refusal is one `Error:` line on stderr (`
(account ) is not a restore target: no restore record names it`; `... its restore from is DONE (the table is ACTIVE, or awaits its index backfill)`; `the restore into
(account ) from is RUNNING with a live owner (a heartbeat within stale_job_after_secs); stop it or wait for it`; `the maintenance claim is held by ""; one maintenance command runs at a time, and a claim whose holder died expires after stale_job_after_secs ( s)`). On SQLite it opens the database as `serve` does and takes the serve lock: stop the server first, as for `purge`; under that lock every owner is dead, so a `RUNNING` record's target is removed too. After it, `DescribeTable` of the name is `ResourceNotFoundException`, `DeleteBackup` of the backup succeeds and a new restore into the name runs. #### [auth] diff --git a/docs/manuals/06-developer-test-guide.md b/docs/manuals/06-developer-test-guide.md index 3eefa6ea2..c8999615e 100755 --- a/docs/manuals/06-developer-test-guide.md +++ b/docs/manuals/06-developer-test-guide.md @@ -157,14 +157,63 @@ devtools/run-tests --extenddb --pytest --filter test_put_item server without the section has by default (`backups/` beside its database file); `s3` for an S3 store; `none` for a server without a store (`kind = "disabled"`, or PostgreSQL, MongoDB and Cassandra without the - section). With a configured store CreateBackup is asynchronous and this - release refuses to restore its backups, so the backup tests derive their - expected outcome from the variable and fail rather than guess when it is - unset; `run-tests --pytest` refuses before any test runs with + section). With a configured store CreateBackup is asynchronous and + RestoreTableFromBackup of its backups is the store-backed restore job + (the response is `CREATING` and the table is loaded in the background), + so the backup tests derive their expected outcome from the variable and + fail rather than guess when it is unset; `run-tests --pytest` refuses before any test runs with `error: --extenddb --pytest requires EXTENDDB_TEST_BACKUP_STORE to be set.` The suites check the value against the server's `/health` `backup_store` component, so a wrong value fails too. +**The backup store and job suites** (`tests/test_backup_store_config.py` +and `tests/test_backup_job.py`) start servers of their own, so `run-tests` +excludes them and they run by name: +`python3 -m pytest tests/test_backup_store_config.py tests/test_backup_job.py -v`. +`EXTENDDB_BINARY` names the release build under test and decides which +deployment the cells use: a SQLite build (the default root beside the +database file) or a PostgreSQL build, which also needs +`EXTENDDB_TEST_PG_CONNECTION_STRING` and +`EXTENDDB_TEST_PG_ADMIN_CONNECTION_STRING`. The S3 cells (the store +configuration suite's S3 rows and the job suite's store-outage cell) need a +moto server at `EXTENDDB_TEST_S3_ENDPOINT` (`http://127.0.0.1:`); +without it they skip. They set the credentials moto accepts +(`testing`/`testing`, `us-east-1`) themselves, in the server's environment +and on their own client, so nothing else needs to be exported. The job suite's restore cells size their +backup from `EXTENDDB_TEST_RESTORE_ITEMS` (default 10,000; at least two data +files are needed), and the multi-part-key cell enables +`[limits] allow_multipart_table_keys` on its own deployment. CI runs both +suites on the PostgreSQL job and the two SQLite jobs; the PostgreSQL job +and the SQLite S3 job start moto, so the outage cell runs there and skips on +the plain SQLite job (`.github/workflows/integration.yml`). + +**The MongoDB backup cells** of `tests/test_mongodb_backup_restore.py` that +read the store's root and the server's log skip unless +`EXTENDDB_TEST_BACKUP_ROOT` (the server's filesystem store root) and +`EXTENDDB_TEST_SERVER_LOG` (the file the server logs to) are set; +`devtools/run-mongodb-tests` serves a filesystem store and exports both +(and `EXTENDDB_TEST_BACKUP_STORE=filesystem`), so run them through it: +`devtools/run-mongodb-tests -- --pytest --filter test_mongodb_backup_restore`. +The suite's legacy LSI fallback cell edits an in-catalog backup and runs +only on a server without a store: +`devtools/run-mongodb-tests --backup-store none -- --pytest --filter test_mongodb_backup_restore` +(the runner then writes no `[backup.store]` and exports +`EXTENDDB_TEST_BACKUP_STORE=none`; the store-backed cells skip under it). +The CI jobs run the suite with `filesystem`. + +**The cross-backend restore matrix** is `devtools/backup-restore-matrix`: +one cell per invocation (`--source --target +`, plus `--duplicate-key` or `--vector`), +each starting a source and a target server from the per-backend release +binaries (`--bin-`, defaulting to +`target/rel-/release/extenddb`) against PostgreSQL at +`127.0.0.1:55432`, MongoDB at `127.0.0.1:27117` and Cassandra at +`127.0.0.1:9042`; its MongoDB and Cassandra cells need `pymongo` and +`cassandra-driver` (`requirements-matrix.txt`). The nightly workflow +`.github/workflows/matrix-nightly.yml` builds the four binaries, starts the +three services and runs the twelve plain cells, the four duplicate-key +cells and the eight vector cells; `workflow_dispatch` runs it on demand. + The `run-tests` script automatically: - Performs health check on the target endpoint - Provisions test credentials via `devtools/provision-test-credentials` diff --git a/docs/manuals/07-upgrade-manual.md b/docs/manuals/07-upgrade-manual.md index 671b884de..93eaac5a1 100755 --- a/docs/manuals/07-upgrade-manual.md +++ b/docs/manuals/07-upgrade-manual.md @@ -4,9 +4,9 @@ ## Current Status -This release versions the PostgreSQL and SQLite catalogs by a migration ledger and carries catalog migrations 1 to 5, which `extenddb migrate --list` prints. **Every PostgreSQL and SQLite deployment created by a 0.1.x release runs `extenddb migrate --yes` once** before a server of this release starts on it: the server refuses a catalog with pending migrations. Migration 1 brings the catalog of every 0.1.x release to the v1.0 schema. Cassandra and MongoDB keep their version strings and both move to catalog 0.0.5 (Cassandra `0.0.4 -> 0.0.5` through `V006__backup_catalog_index.cql`, MongoDB `0.0.2 -> 0.0.5` through its step `002_backup_catalog_index`), the same change as migration 5: run `extenddb migrate --yes` once on each with every pre-upgrade server stopped, as [the section below](#cassandra-and-mongodb-catalog-005) says. +This release versions the PostgreSQL and SQLite catalogs by a migration ledger and carries catalog migrations 1 to 6, which `extenddb migrate --list` prints. **Every PostgreSQL and SQLite deployment created by a 0.1.x release runs `extenddb migrate --yes` once** before a server of this release starts on it: the server refuses a catalog with pending migrations. Migration 1 brings the catalog of every 0.1.x release to the v1.0 schema. Cassandra and MongoDB keep their version strings and both move to catalog 0.0.5 (Cassandra `0.0.4 -> 0.0.5` through `V006__backup_catalog_index.cql`, MongoDB `0.0.2 -> 0.0.5` through its step `002_backup_catalog_index`), the same change as migration 5: run `extenddb migrate --yes` once on each with every pre-upgrade server stopped, as [the section below](#cassandra-and-mongodb-catalog-005) says. -Configure `[backup.store]` before the release that stops creating in-catalog backups. From this release CreateBackup writes to the configured store on PostgreSQL, SQLite and MongoDB, and is asynchronous there as on the service: the response is `CREATING` and the backup becomes `AVAILABLE` once the upload ends, so a client that deletes a backup it just created has to poll DescribeBackup first (the previous release returned `AVAILABLE`). A store-backed backup is listed, described and deleted like any other, but its restore is a later release's: until the restore job lands, RestoreTableFromBackup of a backup in the configured store answers `ValidationException` ("is in a backup store; this release restores in-catalog backups only"), while in-catalog backups, those of the previous release included, restore as before; a deployment that needs to restore the backups it takes this release stays on the in-catalog format (`kind = "disabled"`, below) until then. Without a store CreateBackup writes the in-catalog format and logs a deprecation warning naming `[backup.store]` on each call; a later release's CreateBackup refuses without one. Cassandra writes the in-catalog format with or without a store. `kind = "disabled"` under `[backup.store]` is the opt-out that keeps a deployment on the in-catalog format on purpose and silences the warning. On file-backed SQLite an absent section is a filesystem store at `backups/` beside the database file, created `0700` at startup; that root must be owned by the extenddb user with mode `0700`, under directories owned by root or that user which others cannot write to. A default root that breaks the rule (a database file in a group-writable directory, say), or whose `extenddb-store.json` names another catalog (a catalog re-created by `destroy` and `init` under the old root), is logged at startup as a warning (`backup store: the default root ... failed its check`) and leaves the store unconfigured, not unhealthy, so the upgrade never stops a server that never configured backups and CreateBackup keeps working on it; the same root configured explicitly is a startup error or an unhealthy store. The warning applies only while the catalog has no backup rows for the default root (an upgrade, a fresh deployment, a catalog re-created by `destroy` and `init`): once a server has written rows for it, the default root is handled as a configured one, so its backups never vanish behind a warning (a failed identity check is an unhealthy store, reported on `/health`; a root that can no longer be opened is a startup error naming it). Instances that share a catalog must share one store root, an S3 bucket and prefix or a shared mount: a second instance pointed at a host-local directory under the same configured path starts with the store unhealthy (`/health` reports `backup_store` unhealthy; the admin guide's `[backup.store]` section says what the identity file and the manifest probe check). `extenddb destroy` leaves the store; `extenddb backup purge --confirm` is what deletes its backups, and a catalog re-created with `init` needs the old root's `extenddb-store.json` removed or a new root, since the file names the old catalog. +Configure `[backup.store]` before the release that stops creating in-catalog backups. From this release CreateBackup writes to the configured store on PostgreSQL, SQLite and MongoDB, and is asynchronous there as on the service: the response is `CREATING` and the backup becomes `AVAILABLE` once the upload ends, so a client that deletes a backup it just created has to poll DescribeBackup first (the previous release returned `AVAILABLE`). A store-backed backup is listed, described, deleted and restored like any other. `RestoreTableFromBackup` of a backup in the configured store runs the restore job from the store on every backend (PostgreSQL, SQLite, MongoDB and Cassandra); in-catalog backups, those of the previous release included, restore as before. An S3 store needs `spool_path` set (the restore's download area) on PostgreSQL, MongoDB and Cassandra, and that must be a directory of its own, not the database directory; a filesystem store needs no spool. Migration 6 adds the `restore_progress` table to the PostgreSQL data database; run `extenddb migrate --yes` before starting the new binary. The progress rows of a restore in flight live in the data database, so a data database restored from a dump taken before migration 6 has none of them and no `restore_progress` table: the new binary refuses to serve it until `extenddb migrate --yes` has run again, and its reconciler then finds no progress for the restore and resumes it from its first data file into the (empty) target, or `FAILED`s it as `target_not_empty` if the target already holds items, in which case `extenddb backup remove-target --account
` (the admin guide's `backup` section) removes the target and its record and the backup restores again. The same `target_not_empty` state is what a server stopped inside a restore's first data file on MongoDB, or inside its first batch of 10,000 items or 32 MiB on Cassandra, leaves on restart, since both record progress after the write; the command is the remedy there too. `backup remove-target` writes the catalog, so on SQLite the server is stopped for it, as for `purge`. The previous binary resumes no restore (a store-backed restore in flight at a rollback is lost with the catalog rows, as the rollback paragraph below says). Quiesce restores before a downgrade. `kind = "disabled"` is no longer needed to keep a deployment able to restore the backups it takes. Without a store CreateBackup writes the in-catalog format and logs a deprecation warning naming `[backup.store]` on each call; a later release's CreateBackup refuses without one. Cassandra writes the in-catalog format with or without a store. `kind = "disabled"` under `[backup.store]` is the opt-out that keeps a deployment on the in-catalog format on purpose and silences the warning. On file-backed SQLite an absent section is a filesystem store at `backups/` beside the database file, created `0700` at startup; that root must be owned by the extenddb user with mode `0700`, under directories owned by root or that user which others cannot write to. A default root that breaks the rule (a database file in a group-writable directory, say), or whose `extenddb-store.json` names another catalog (a catalog re-created by `destroy` and `init` under the old root), is logged at startup as a warning (`backup store: the default root ... failed its check`) and leaves the store unconfigured, not unhealthy, so the upgrade never stops a server that never configured backups and CreateBackup keeps working on it; the same root configured explicitly is a startup error or an unhealthy store. The warning applies only while the catalog has no backup rows for the default root (an upgrade, a fresh deployment, a catalog re-created by `destroy` and `init`): once a server has written rows for it, the default root is handled as a configured one, so its backups never vanish behind a warning (a failed identity check is an unhealthy store, reported on `/health`; a root that can no longer be opened is a startup error naming it). Instances that share a catalog must share one store root, an S3 bucket and prefix or a shared mount: a second instance pointed at a host-local directory under the same configured path starts with the store unhealthy (`/health` reports `backup_store` unhealthy; the admin guide's `[backup.store]` section says what the identity file and the manifest probe check). `extenddb destroy` leaves the store; `extenddb backup purge --confirm` is what deletes its backups, and a catalog re-created with `init` needs the old root's `extenddb-store.json` removed or a new root, since the file names the old catalog. A backup that a 0.1.x release took can restore without its global and local secondary indexes: a SQLite backup taken by 0.1.14 records only the table's key schema and attribute definitions, and its restore creates the table with its items and no index. Take a new backup of each table after upgrading to v1.0, and keep the 0.1.x backups for their items only. @@ -41,7 +41,7 @@ Exit status 6 means the migration lock, or a running server, was still held when `verify` also checks the connection to the data database. When that check fails, `verify` exits 1 on a Behind catalog instead of 3, because `migrate --yes` cannot apply the pending migrations while the data database cannot be reached. It still exits 4 or 5 for the statuses that need a person. -A migration is additive when the previous release serves the catalog correctly with it applied, and breaking otherwise. Migrations 1 to 5 are breaking: do not run a 0.1.x server against a migrated catalog. A 0.1.x binary that writes to a migrated catalog (its `migrate --yes`, or a SQLite dev server) leaves it WrittenByOlderRelease. `migrate --yes --reconverge` repairs it by running the convergent migrations again. Migration 2, which rewrites stored policies, is not convergent: the repair skips it and prints a line that says so (see the v0.1.14 notes below). +A migration is additive when the previous release serves the catalog correctly with it applied, and breaking otherwise. Migrations 1 to 5 are breaking. Migration 6 creates a table in the PostgreSQL data database that the previous release never reads; on PostgreSQL it is declared breaking only because the job that proves an additive declaration against the old binary does not yet exist (CONTRIBUTING, step 4 of adding a migration), and on SQLite it does not apply (`migrate --list` prints it as not applicable, and `migrate` counts it as additive). Do not run a 0.1.x server against a migrated catalog. A 0.1.x binary that writes to a migrated catalog (its `migrate --yes`, or a SQLite dev server) leaves it WrittenByOlderRelease. `migrate --yes --reconverge` repairs it by running the convergent migrations again. Migration 2, which rewrites stored policies, is not convergent: the repair skips it and prints a line that says so (see the v0.1.14 notes below). A backup of a table that holds an `N` value the wire did not validate fails on the store-backed path where the previous release backed it up. Four write paths stored such a value (`1E126`, `abc`) unvalidated (#618: BatchWriteItem, TransactWriteItems, UpdateItem on an absent item in the key it creates, ImportTable), and the store-backed job validates every item with the rule the restore applies, so no backup that no restore can load is reported `AVAILABLE`. This meets file-backed SQLite without a `[backup.store]` section (the defaulted root) and any deployment once a store is configured. The client sees `CREATING` and then `BackupNotFound`; the reason reaches the operator only: `extenddb backup list --failed` prints the FAILED row with the count of refused items and the first one, and the server log names each refused item's key and attribute path beside the backup ARN (up to 1,000 per backup), which a tenant's support case can cite (every tenant-supplied text in the line is written as a JSON string's content, control characters and the Unicode line separators U+0085, U+2028 and U+2029 escaped, so a newline or a line separator in an attribute name or a number value cannot forge a second log record). The way out is to rewrite or delete each named item through the wire, which validates the value (a bad range key cannot be rewritten, since a PutItem with that key is refused; `DeleteItem` removes it); the in-catalog create of a server without a store still backs such a table up. @@ -218,7 +218,7 @@ On SQLite, `extenddb serve` reads the catalog's migration ledger before it opens The `extenddb migrate` command that `serve`, `migrate` and `verify` advise names the config file by its absolute path, quoted for a POSIX shell, so it can be pasted as printed. A relative `path` in `[storage.sqlite]`, such as the `extenddb.sqlite` that `init` writes, resolves against the directory a command runs in. So when the config file names the database by a relative `path`, or names none, the advised command starts with `EXTENDDB__STORAGE__SQLITE__PATH=`, and it reaches the same database from any directory. -On startup the SQLite server ages every RUNNING restore record to the epoch, which is only safe because one server at a time can hold the file, and its reconciler's first cycle then fails an in-catalog restore that a server stopped mid-copy and removes its `CREATING` target (the cycle runs beside the listener, so a request can still see the target `CREATING` until it ends). A backup keeps a WAL read snapshot open while it copies; checkpoints cannot reclaim pages needed by that snapshot, so allow disk headroom for WAL growth proportional to the backup duration and concurrent write volume. Set `pool_size` to 3 or more (see the Admin Guide); the server runs at most `pool_size / 2 - 1` backups at once (4 with the default of 10) and answers `LimitExceededException` at once to a `CreateBackup` that finds no slot free. Running backups hold one connection each for their read snapshot plus one shared writer (`n` backups hold `n + 1` connections), so from `pool_size` 4 up a flood of backups leaves at least half the pool to other requests; with a `pool_size` of 2 or 3 the one admitted backup holds two of them. A GSI added by `UpdateTable` with `ProvisionedThroughput` on a `PROVISIONED` table made every `DescribeTable` of that table fail on earlier builds (`Internal("index pt: missing field NumberOfDecreasesToday")`); this build writes the index's throughput in the shape `DescribeTable` reads and reads the shape earlier builds wrote (as the same units with `NumberOfDecreasesToday` 0), so such a table is describable again after the upgrade without any change to the database file. +On startup the SQLite server ages every RUNNING restore record to the epoch, which is only safe because one server at a time can hold the file, and its reconciler's first cycle then fails an in-catalog restore that a server stopped mid-copy and removes its `CREATING` target (the cycle runs beside the listener, so a request can still see the target `CREATING` until it ends); a restore from a backup store is resumed instead, from its recorded progress, and its target stays `CREATING` until it finishes. A backup keeps a WAL read snapshot open while it copies; checkpoints cannot reclaim pages needed by that snapshot, so allow disk headroom for WAL growth proportional to the backup duration and concurrent write volume. Set `pool_size` to 3 or more (see the Admin Guide); the server runs at most `pool_size / 2 - 1` backups at once (4 with the default of 10) and answers `LimitExceededException` at once to a `CreateBackup` that finds no slot free. Running backups hold one connection each for their read snapshot plus one shared writer (`n` backups hold `n + 1` connections), so from `pool_size` 4 up a flood of backups leaves at least half the pool to other requests; with a `pool_size` of 2 or 3 the one admitted backup holds two of them. A GSI added by `UpdateTable` with `ProvisionedThroughput` on a `PROVISIONED` table made every `DescribeTable` of that table fail on earlier builds (`Internal("index pt: missing field NumberOfDecreasesToday")`); this build writes the index's throughput in the shape `DescribeTable` reads and reads the shape earlier builds wrote (as the same units with `NumberOfDecreasesToday` 0), so such a table is describable again after the upgrade without any change to the database file. #### Cassandra diff --git a/docs/schema/postgres.txt b/docs/schema/postgres.txt index 24671a937..d888c7bce 100644 --- a/docs/schema/postgres.txt +++ b/docs/schema/postgres.txt @@ -354,6 +354,7 @@ sequence stream_seq: bigint START 1 INCREMENT 1 table extenddb_migrations table gsi_pending table idempotency_tokens +table restore_progress table schema_history table stream_records table stream_shards @@ -377,6 +378,9 @@ column idempotency_tokens.account_id: text NOT NULL column idempotency_tokens.token: text NOT NULL column idempotency_tokens.fingerprint: text NOT NULL column idempotency_tokens.created_at: timestamp with time zone NOT NULL DEFAULT now() +column restore_progress.table_id: text NOT NULL +column restore_progress.file_index: bigint NOT NULL +column restore_progress.items_applied: bigint NOT NULL column schema_history.filename: text NOT NULL column schema_history.applied_at: timestamp with time zone NOT NULL DEFAULT now() column stream_records.shard_id: text NOT NULL @@ -402,6 +406,7 @@ index idx_idempotency_tokens_created: CREATE INDEX idx_idempotency_tokens_create index idx_stream_records_created: CREATE INDEX idx_stream_records_created ON public.stream_records USING btree (created_at) index idx_stream_shards_table: CREATE INDEX idx_stream_shards_table ON public.stream_shards USING btree (table_id) index idx_vector_index_holds_table: CREATE INDEX idx_vector_index_holds_table ON public.vector_index_holds USING btree (table_id) +index restore_progress_pkey: CREATE UNIQUE INDEX restore_progress_pkey ON public.restore_progress USING btree (table_id, file_index) index schema_history_pkey: CREATE UNIQUE INDEX schema_history_pkey ON public.schema_history USING btree (filename) index stream_records_pkey: CREATE UNIQUE INDEX stream_records_pkey ON public.stream_records USING btree (shard_id, sequence_number) index stream_shards_pkey: CREATE UNIQUE INDEX stream_shards_pkey ON public.stream_shards USING btree (shard_id) @@ -409,6 +414,7 @@ index vector_index_holds_pkey: CREATE UNIQUE INDEX vector_index_holds_pkey ON pu constraint extenddb_migrations.extenddb_migrations_pkey: PRIMARY KEY (id) constraint gsi_pending.gsi_pending_pkey: PRIMARY KEY (id) constraint idempotency_tokens.idempotency_tokens_pkey: PRIMARY KEY (account_id, token) +constraint restore_progress.restore_progress_pkey: PRIMARY KEY (table_id, file_index) constraint schema_history.schema_history_pkey: PRIMARY KEY (filename) constraint stream_records.stream_records_pkey: PRIMARY KEY (shard_id, sequence_number) constraint stream_records.stream_records_shard_id_fkey: FOREIGN KEY (shard_id) REFERENCES stream_shards(shard_id) ON DELETE CASCADE diff --git a/requirements-matrix.txt b/requirements-matrix.txt new file mode 100644 index 000000000..9489acf13 --- /dev/null +++ b/requirements-matrix.txt @@ -0,0 +1,9 @@ +# Copyright 2026 ExtendDB contributors +# SPDX-License-Identifier: Apache-2.0 +# The drivers devtools/backup-restore-matrix uses to seed and tear down its +# MongoDB and Cassandra deployments (.github/workflows/matrix-nightly.yml +# installs this file beside requirements.txt). Separate from requirements.txt +# because nothing else in the test suites talks to those services directly +# and cassandra-driver builds a C extension on install. +pymongo==4.7.3 +cassandra-driver==3.29.3 diff --git a/tests/rust/src/backup_restore.rs b/tests/rust/src/backup_restore.rs index c086a545a..dac1cd16f 100755 --- a/tests/rust/src/backup_restore.rs +++ b/tests/rust/src/backup_restore.rs @@ -23,9 +23,9 @@ use std::time::{Duration, Instant}; /// Where the server under test writes its backups, from /// `EXTENDDB_TEST_BACKUP_STORE`: `filesystem` or `s3` (a configured -/// `[backup.store]`, under which a backup is in the store and this release -/// refuses to restore it, PR 9a adding the restore) or `none` (the -/// in-catalog create, which restores). Against the service the variable is +/// `[backup.store]`, under which a backup is in the store and the restore +/// job restores it) or `none` (the in-catalog create, which the legacy +/// reader restores). Against the service the variable is /// not read (the restore runs). Against ExtendDB a test whose outcome depends /// on it fails when it is unset, with the text the Python fixture prints, /// rather than accept either outcome. @@ -306,49 +306,39 @@ async fn restore_table_from_backup() { .backup_arn(&arn) .send() .await; - match backup_store() { - // A backup in the configured store is listed, described and deleted - // but not restored this release (the restore job is PR 9a's). - BackupStore::Filesystem | BackupStore::S3 => { - let err = restore.expect_err("a store-backed backup restored"); - assert_eq!(err_code(&err), Some("ValidationException"), "{err:?}"); - let msg = err_msg(&err); - assert!( - msg.contains("is in a backup store"), - "the refusal does not name the store: {msg}" - ); - println!("restore of a store-backed backup refused: {msg}"); - assert!( - c.describe_table() - .table_name(&restored) - .send() - .await - .is_err(), - "the refused restore created {restored}" - ); - } - BackupStore::None => { - let resp = restore.unwrap(); - assert_eq!( - resp.table_description() - .unwrap() - .table_status() - .unwrap() - .as_str(), - "CREATING" - ); - - wait_for_active(c, &restored).await; - let scan = c.scan().table_name(&restored).send().await.unwrap(); - assert_eq!(scan.count(), 5); - println!( - "restore of an in-catalog backup completed with {} items", - scan.count() - ); - - c.delete_table().table_name(&restored).send().await.ok(); - } - } + // The round trip on either path: a backup in the configured store is + // restored by the restore job, an in-catalog one by the legacy reader. + let resp = restore.unwrap(); + let desc = resp.table_description().unwrap(); + assert_eq!(desc.table_status().unwrap().as_str(), "CREATING"); + let summary = desc + .restore_summary() + .expect("a CREATING restore target carries its RestoreSummary"); + assert_eq!(summary.source_backup_arn(), Some(arn.as_str())); + assert!(summary.restore_in_progress()); + + wait_for_active(c, &restored).await; + let scan = c.scan().table_name(&restored).send().await.unwrap(); + assert_eq!(scan.count(), 5); + let after = c + .describe_table() + .table_name(&restored) + .send() + .await + .unwrap(); + assert!( + after.table().unwrap().restore_summary().is_none(), + "RestoreSummary is reported only while CREATING" + ); + println!( + "restore of a {} backup completed with {} items", + match backup_store() { + BackupStore::Filesystem | BackupStore::S3 => "store-backed", + BackupStore::None => "in-catalog", + }, + scan.count() + ); + c.delete_table().table_name(&restored).send().await.ok(); delete_backup_quietly(c, &arn).await; c.delete_table().table_name(&table).send().await.ok(); diff --git a/tests/rust/src/restore_active_completeness.rs b/tests/rust/src/restore_active_completeness.rs index 93ca7bcd5..bab83e8f1 100644 --- a/tests/rust/src/restore_active_completeness.rs +++ b/tests/rust/src/restore_active_completeness.rs @@ -25,9 +25,8 @@ //! window. //! //! On a server with a configured `[backup.store]` (`EXTENDDB_TEST_BACKUP_STORE` -//! at `filesystem` or `s3`) the restore of a backup this server created is -//! refused this release (the restore job is PR 9a's), so the race test pins -//! that refusal in place of the race. +//! at `filesystem` or `s3`) the restore is the restore job's, from the store, +//! and the race runs against it as against the in-catalog copy. use crate::backup_restore::{ backup_store, delete_backup_quietly, wait_for_backup_available, BackupStore, @@ -169,33 +168,13 @@ async fn restored_table_has_all_items_when_first_active() { wait_for_backup_available(c, &arn).await; let dst = format!("RestoreRaceDst_{}", ts()); - // A backup in the configured store is not restored this release (the - // restore job is PR 9a's): the race has nothing to observe, and the test - // pins the refusal instead. - if matches!(backup_store(), BackupStore::Filesystem | BackupStore::S3) { - let err = c - .restore_table_from_backup() - .target_table_name(&dst) - .backup_arn(&arn) - .send() - .await - .expect_err("a store-backed backup restored"); - assert_eq!(err_code(&err), Some("ValidationException"), "{err:?}"); - let msg = err_msg(&err); - assert!( - msg.contains("is in a backup store"), - "the refusal does not name the store: {msg}" - ); - println!("restore of a store-backed backup refused: {msg}"); - assert!( - c.describe_table().table_name(&dst).send().await.is_err(), - "the refused restore created {dst}" - ); - delete_backup_quietly(c, &arn).await; - c.delete_table().table_name(&src).send().await.ok(); - return; - } - println!("restore of an in-catalog backup: racing the observer"); + println!( + "restore of {} backup: racing the observer", + match backup_store() { + BackupStore::Filesystem | BackupStore::S3 => "a store-backed", + BackupStore::None => "an in-catalog", + } + ); // Race an observer against the in-flight restore: the moment it sees // ACTIVE, it counts. Returns None if ACTIVE never arrives within the // bound, which means the restore itself failed. diff --git a/tests/test_auth_permissions.py b/tests/test_auth_permissions.py index 3c2e33465..6cf89d714 100755 --- a/tests/test_auth_permissions.py +++ b/tests/test_auth_permissions.py @@ -1356,6 +1356,14 @@ def source_backup(self): backup_arn = admin_client.create_backup( TableName=table, BackupName="authz", )["BackupDetails"]["BackupArn"] + # A store-backed backup is CREATING until its job ends; a restore + # of a CREATING backup is BackupInUse, so the cells wait here. + deadline = time.monotonic() + 120 + while admin_client.describe_backup( + BackupArn=backup_arn + )["BackupDescription"]["BackupDetails"]["BackupStatus"] == "CREATING": + assert time.monotonic() < deadline, f"backup {backup_arn} still CREATING" + time.sleep(0.2) yield admin_client, table, backup_arn finally: for t in self.created: @@ -1446,14 +1454,6 @@ def test_restore_needs_the_item_actions_on_the_target(self, source_backup): """RestoreTableFromBackup on the backup plus the seven item actions on the target, each required; RestoreTableFromBackup on the target is neither required nor evaluated.""" - if os.environ.get("EXTENDDB_TEST_BACKUP_STORE", "").strip() in ("filesystem", "s3"): - # The denials hold on any server; the restores that must run are - # of a backup in the configured store, which RestoreTableFromBackup - # refuses until the restore job lands (PR 9a). - pytest.skip( - "RestoreTableFromBackup of a backup in the configured store is refused " - "until the restore job lands (PR 9a)" - ) admin_client, table, backup_arn = source_backup user = f"u-{uuid.uuid4().hex[:8]}" ak, sk = self._user_with_key(user) diff --git a/tests/test_backup_job.py b/tests/test_backup_job.py index 6bca868a9..5d8b6f3b8 100644 --- a/tests/test_backup_job.py +++ b/tests/test_backup_job.py @@ -24,6 +24,28 @@ named by its whole key, and the wire's reason for its value, in the log line and in `backup list --failed`. +The store-backed RestoreTableFromBackup job, on both deployments: + +- a data file tampered with under the root fails the restore before any item + is loaded, the target gone and the log naming the file; +- SIGTERM mid-restore leaves the record RUNNING and the restarted server + resumes it (PostgreSQL after the stale window, SQLite at once); +- an S3 store (moto) cut mid-download leaves the record RUNNING and the + restore completes once the store answers; +- an unordered backup holding one key in two files, and a key-ordered file + leaving its `key_range` (both hand-built under the root), fail the restore + with the target gone; +- a table keyed on two HASH attributes (the `allow_multipart_table_keys` + preview, enabled on the cell's own deployment) backs up and restores with + the source's item set, key schema and attribute definitions; +- (SQLite) a RUNNING restore record with no recorded progress whose target + holds items, the state a crash before the first progress record leaves on + MongoDB and Cassandra, seeded at the catalog, is FAILED `target_not_empty` + by the restarted server's reconciler, the target left CREATING and + `DeleteTable`, `DeleteBackup` and a second restore refused, and `extenddb + backup remove-target` removes the target and its record, after which + `DescribeTable` is `ResourceNotFoundException` and `DeleteBackup` succeeds. + The binary at `EXTENDDB_BINARY` decides which deployment runs: a SQLite build or a PostgreSQL build (with `EXTENDDB_TEST_PG_CONNECTION_STRING`). Excluded from the backend-agnostic suite; run by name. @@ -52,13 +74,18 @@ EXTENDDB_BINARY, _append, _client, + _Forwarder, + _free_port, + _moto_or_skip, _pg_build, _provision, + _s3_env, _Server, _sqlite_build, _SqliteDeployment, _wait_backup_available, _wait_health, + moto_bucket, # noqa: F401 - the fixture of the store-outage cell ) ACCOUNT = "123456789012" @@ -77,15 +104,19 @@ class _Deployment: - """A deployment with a filesystem store, its config and its root.""" + """A deployment with a filesystem store, its config, its root and its + catalog (`catalog`: the SQLite database file, or the PostgreSQL catalog + database's connection string), where a cell edits a `backup_index` row + it rewrote the prefix of.""" - def __init__(self, backend: str, config: str, port: int, root: str, cert: str, cwd: str): + def __init__(self, backend: str, config: str, port: int, root: str, cert: str, cwd: str, catalog: str = ""): self.backend = backend self.config = config self.port = port self.root = root self.cert = cert self.cwd = cwd + self.catalog = catalog def run(self, *args): import subprocess @@ -122,6 +153,7 @@ def deployment(request, tmp_path) -> _Deployment: root = os.path.join(base, "backups") store = "" backend = "sqlite" + catalog = d.db_path elif _pg_build(): cli_env = request.getfixturevalue("cli_env") result = lh._run_extenddb( @@ -136,6 +168,7 @@ def deployment(request, tmp_path) -> _Deployment: os.mkdir(root, 0o700) store = f'[backup.store]\nkind = "filesystem"\n[backup.store.filesystem]\npath = "{root}"\n' backend = "postgres" + catalog = cli_env["conn_string"] else: pytest.skip("EXTENDDB_BINARY is neither a SQLite nor a PostgreSQL build") os.mkdir(os.path.join(base, ACCOUNT)) @@ -145,7 +178,7 @@ def deployment(request, tmp_path) -> _Deployment: + store + f'[import]\npaths = ["{base}"]\n', ) - return _Deployment(backend, config, port, root, cert, base) + return _Deployment(backend, config, port, root, cert, base, catalog) def _started(d: _Deployment, log_path: str) -> tuple[_Server, object, dict]: @@ -573,3 +606,826 @@ def test_an_item_over_max_item_size_bytes_fails_the_backup_without_naming_618(de assert listed == ["under-default"], listed finally: server.stop() + + +# --- the restore job --------------------------------------------------------------- + +# Items of this count and padding make a backup of two data files (the knob's +# floor is 32 MiB of uncompressed lines) whose restore runs for long enough, +# on a PostgreSQL build of this host, that a SIGTERM sent right after the +# restore's response finds the job inside its download or its first file's +# load. The cells check the precondition they rely on (the target still +# CREATING when the signal is sent) rather than assume it. +RESTORE_ITEMS = int(os.environ.get("EXTENDDB_TEST_RESTORE_ITEMS", "10000")) +RESTORE_PAD = 4000 + + +# The `[backup]` keys the restore cells add to the deployment's: data files +# close at the knob's floor, so RESTORE_ITEMS padded items make two of them. +RESTORE_BACKUP_SETTINGS = "data_file_target_bytes = 33554432\n" + + +def _add_backup_settings(d: _Deployment, extra: str) -> None: + """Add `extra` to the deployment's `[backup]` table before the server + starts (TOML refuses a second `[backup]` header).""" + with open(d.config) as f: + text = f.read() + head = "[backup]\nreconcile_interval_secs = 1\n" + assert text.count(head) == 1, text + with open(d.config, "w") as f: + f.write(text.replace(head, head + extra)) + + +def _load_padded(d: _Deployment, creds: dict, name: str, n: int, pad: int) -> None: + """`_load` with items of `pad` bytes of padding and a `g` the GSI keys + on, from 8 threads with one client each.""" + local = threading.local() + + def write(start: int) -> None: + if not hasattr(local, "client"): + local.client = _client(d.port, creds) + requests_ = [ + { + "PutRequest": { + "Item": { + "pk": {"S": f"p{i % 500:04d}"}, + "sk": {"N": str(i)}, + "g": {"S": f"g{i % 50}"}, + "pad": {"S": "x" * pad}, + } + } + } + for i in range(start, min(start + 25, n)) + ] + pending = {name: requests_} + while pending: + pending = local.client.batch_write_item(RequestItems=pending).get("UnprocessedItems") or {} + if pending: + time.sleep(0.05) + + with ThreadPoolExecutor(max_workers=8) as pool: + list(pool.map(write, range(0, n, 25))) + + +def _count(ddb, name: str) -> int: + total, start = 0, None + while True: + kwargs = {"TableName": name, "Select": "COUNT", "ConsistentRead": True} + if start: + kwargs["ExclusiveStartKey"] = start + resp = ddb.scan(**kwargs) + total += resp["Count"] + start = resp.get("LastEvaluatedKey") + if not start: + return total + + +def _wait_table_gone(ddb, name: str, timeout: float = 120.0) -> None: + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + try: + status = ddb.describe_table(TableName=name)["Table"]["TableStatus"] + except ClientError as e: + assert e.response["Error"]["Code"] == "ResourceNotFoundException", e.response["Error"] + return + assert status == "CREATING", f"{name} is {status}, it was never writable" + time.sleep(0.2) + raise AssertionError(f"{name} still exists after {timeout} s") + + +def _backed_up_table(d: _Deployment, ddb, creds: dict, prefix: str) -> tuple[str, str]: + """A composite-key table with a GSI holding RESTORE_ITEMS padded items, + backed up to AVAILABLE in two data files; the name and the ARN.""" + name = f"{prefix}-{uuid.uuid4().hex[:8]}" + _create_composite_table(ddb, name, gsi=True) + _load_padded(d, creds, name, RESTORE_ITEMS, RESTORE_PAD) + arn = ddb.create_backup(TableName=name, BackupName="nightly")["BackupDetails"]["BackupArn"] + _wait_backup_available(ddb, arn, timeout=180) + files = sorted(os.listdir(os.path.join(_prefix(d, name, arn), "data"))) + assert len(files) >= 2, f"the backup has one data file ({files}); raise EXTENDDB_TEST_RESTORE_ITEMS" + return name, arn + + +def test_a_tampered_data_file_fails_the_restore_before_any_item(deployment, tmp_path): + """A byte of the backup's second data file flipped under the root: the + restore request is accepted (CREATING with a RestoreSummary), the target + is removed once the job fails, and the server log names the file. That + the job verified every file before it loaded any is pinned by the FAILED + reason's text: "does not match the manifest" is the hashing pass's, which + runs over every file before the first `bulk_load`; a mismatch found while + a file is loaded reads "data file : data file sha256 checksum + mismatch" or, when the flipped byte sits in the gzip trailer so the + decoder's own CRC fires first, "data file : data file decode error: + corrupt gzip stream does not have a matching checksum" instead. A + CREATING target answers no read whatever the job did, so the GetItem + probe below only confirms the target never became readable. The backup + itself stays AVAILABLE.""" + d = deployment + _add_backup_settings(d, RESTORE_BACKUP_SETTINGS) + server, ddb, creds = _started(d, str(tmp_path / "serve.log")) + try: + name, arn = _backed_up_table(d, ddb, creds, "tamper") + path = os.path.join(_prefix(d, name, arn), "data", "000002.json.gz") + with open(path, "r+b") as f: + f.seek(-1, os.SEEK_END) + last = f.read(1) + f.seek(-1, os.SEEK_END) + f.write(bytes([last[0] ^ 0xFF])) + target = f"{name}-r" + resp = ddb.restore_table_from_backup(TargetTableName=target, BackupArn=arn)["TableDescription"] + assert resp["TableStatus"] == "CREATING", resp + assert resp["RestoreSummary"]["SourceBackupArn"] == arn, resp + readable = [] + deadline = time.monotonic() + 120 + while time.monotonic() < deadline: + try: + got = ddb.get_item(TableName=target, Key={"pk": {"S": "p0000"}, "sk": {"N": "0"}}, ConsistentRead=True) + if "Item" in got: + readable.append(got["Item"]) + except ClientError: + pass + try: + ddb.describe_table(TableName=target) + except ClientError as e: + assert e.response["Error"]["Code"] == "ResourceNotFoundException", e.response["Error"] + break + time.sleep(0.2) + else: + raise AssertionError(f"{target} was not removed within 120 s") + assert readable == [], f"an item of the failed restore was readable: {readable[:1]}" + log = server.log_text() + failed = [line for line in log.splitlines() if "FAILED (data file data/000002.json.gz does not match the manifest" in line] + assert len(failed) == 1, log[-4000:] + assert "the target was removed" in failed[0], failed[0] + assert ddb.describe_backup(BackupArn=arn)["BackupDescription"]["BackupDetails"]["BackupStatus"] == "AVAILABLE" + finally: + server.stop() + + +def test_sigterm_leaves_the_restore_running_and_it_finishes_after_restart(deployment, tmp_path): + """SIGTERM while the restore runs: the server drains, the job stops at + its next file or chunk with no status write, the record stays RUNNING + (the target CREATING with its RestoreSummary across the restart), and the + restarted server's reconciler resumes it: on PostgreSQL once the dead + owner's heartbeat is older than `stale_job_after_secs` (45 s, the floor, + on the database's clock); on SQLite at the first cycle, since the restart + frees the record's slot and ages every RUNNING record. The target reaches + ACTIVE with every item and the GSI answers.""" + d = deployment + _add_backup_settings(d, RESTORE_BACKUP_SETTINGS + "stale_job_after_secs = 45\n") + server, ddb, creds = _started(d, str(tmp_path / "serve.log")) + try: + name, arn = _backed_up_table(d, ddb, creds, "sigterm") + target = f"{name}-r" + resp = ddb.restore_table_from_backup(TargetTableName=target, BackupArn=arn)["TableDescription"] + assert resp["TableStatus"] == "CREATING", resp + status = ddb.describe_table(TableName=target)["Table"]["TableStatus"] + assert status == "CREATING", ( + f"the restore finished ({status}) before the signal was sent; raise EXTENDDB_TEST_RESTORE_ITEMS" + ) + server.proc.terminate() + code = server.wait_exit(timeout=60) + assert code == 0, server.log_text()[-4000:] + log = server.log_text() + assert "stopped by the server's shutdown; the record stays RUNNING" in log, log[-4000:] + assert ": FAILED (" not in log, log[-4000:] + finally: + server.stop() + stopped_at = time.monotonic() + server, ddb, creds = _started(d, str(tmp_path / "serve2.log")) + try: + table = ddb.describe_table(TableName=target)["Table"] + if d.backend == "postgres": + # PostgreSQL resumes only once the dead owner's heartbeat is + # stale (45 s), so right after the restart the target is still + # CREATING, carries its RestoreSummary, and a retry into the name + # is refused as a restore in progress. On SQLite the restart frees + # the record's slot and the first reconciler cycle resumes at + # once, so the target may already be ACTIVE here; the + # resume-timing assertion below pins the at-once behaviour. + assert table["TableStatus"] == "CREATING", table + assert table["RestoreSummary"]["SourceBackupArn"] == arn, table + with pytest.raises(ClientError) as refused: + ddb.restore_table_from_backup(TargetTableName=target, BackupArn=arn) + assert refused.value.response["Error"]["Code"] == "TableInUseException", refused.value.response + _wait_active(ddb, target, timeout=240) + resumed_after = time.monotonic() - stopped_at + if d.backend == "postgres": + assert resumed_after >= 45 - 5, f"the resume waited for the stale window ({resumed_after:.1f} s)" + else: + assert resumed_after < 45 - 5, f"the restart's slot rule resumes at once ({resumed_after:.1f} s)" + assert _count(ddb, target) == RESTORE_ITEMS + by_g = ddb.query( + TableName=target, + IndexName="by-g", + KeyConditionExpression="g = :g", + ExpressionAttributeValues={":g": {"S": "g7"}}, + Select="COUNT", + )["Count"] + assert by_g == RESTORE_ITEMS // 50, by_g + log = server.log_text() + assert "taken over by this instance and resumed" in log, log[-4000:] + assert ": ACTIVE, " in log, log[-4000:] + finally: + server.stop() + + +def test_a_store_outage_mid_download_leaves_the_restore_running_and_it_completes(request, tmp_path, moto_bucket): + """An S3 store (moto) reached through a TCP forwarder, with a spool: the + forwarder is cut right after the restore's response, so the job's data + file downloads fail as transport errors; the record stays RUNNING (the + target CREATING) while the job retries with backoff; once the forwarder + is back the download completes and the target reaches ACTIVE with every + item, the spool entry removed. On PostgreSQL the spool is `spool_path`; + on SQLite the defaulted `backup-spool/` beside the database file.""" + moto_host, moto_port = _moto_or_skip() + bucket = moto_bucket + base = os.path.realpath(str(tmp_path)) + port = _free_port() + store = ( + '[backup.store]\nkind = "s3"\n[backup.store.s3]\n' + f'bucket = "{bucket}"\nendpoint = "http://127.0.0.1:{port}"\n' + 'force_path_style = true\nregion = "us-east-1"\n' + ) + if _sqlite_build(): + sd = _SqliteDeployment(str(tmp_path)) + spool = os.path.join(base, "backup-spool") + _append(sd.config, "[backup]\nreconcile_interval_secs = 1\nmax_concurrent_jobs = 1\n" + RESTORE_BACKUP_SETTINGS + store) + d = _Deployment("sqlite", sd.config, sd.port, "", sd.cert, base) + elif _pg_build(): + cli_env = request.getfixturevalue("cli_env") + result = lh._run_extenddb( + "init", *lh._init_args(cli_env), config=cli_env["config_path"], + env_override={"EXTENDDB_ADMIN_PASSWORD": ADMIN_PASSWORD}, + ) + assert result.returncode == 0, result.stdout + result.stderr + lh._patch_config_port(cli_env["config_path"], cli_env["port"]) + spool = os.path.join(base, "spool") + _append( + cli_env["config_path"], + "[backup]\nreconcile_interval_secs = 1\nmax_concurrent_jobs = 1\n" + + RESTORE_BACKUP_SETTINGS + + f'spool_path = "{spool}"\n' + + store, + ) + d = _Deployment("postgres", cli_env["config_path"], cli_env["port"], "", os.path.join(cli_env["tls_dir"], "cert.pem"), base) + else: + pytest.skip("EXTENDDB_BINARY is neither a SQLite nor a PostgreSQL build") + forwarder = _Forwarder(port, (moto_host, moto_port)) + server = _Server(d.config, str(tmp_path / "serve.log"), env=_s3_env()) + try: + _wait_health( + d.port, + lambda h: h.get("components", {}).get("backup_store", {}).get("status") == "healthy", + timeout=60, + server=server, + ) + creds = _provision(d.port, d.cert, ADMIN_PASSWORD) + ddb = _client(d.port, creds) + name = f"outage-{uuid.uuid4().hex[:8]}" + _create_composite_table(ddb, name, gsi=True) + _load_padded(d, creds, name, RESTORE_ITEMS, RESTORE_PAD) + arn = ddb.create_backup(TableName=name, BackupName="nightly")["BackupDetails"]["BackupArn"] + _wait_backup_available(ddb, arn, timeout=180) + target = f"{name}-r" + resp = ddb.restore_table_from_backup(TargetTableName=target, BackupArn=arn)["TableDescription"] + assert resp["TableStatus"] == "CREATING", resp + # The cut: every data file get fails until the forwarder is back. + forwarder.stop() + cut_at = time.monotonic() + while time.monotonic() - cut_at < 8: + table = ddb.describe_table(TableName=target)["Table"] + assert table["TableStatus"] == "CREATING", table + assert table["RestoreSummary"]["RestoreInProgress"] is True, table + time.sleep(0.5) + log = server.log_text() + retried = [line for line in log.splitlines() if "retried in" in line and "data/" in line] + assert retried, "no download attempt was retried during the cut:\n" + log[-4000:] + forwarder = _Forwarder(port, (moto_host, moto_port)) + _wait_active(ddb, target, timeout=240) + assert _count(ddb, target) == RESTORE_ITEMS + entries = [e for e in os.listdir(spool) if e.startswith("restore-")] + assert entries == [], f"the spool entry stays: {entries}" + log = server.log_text() + assert ": ACTIVE, " in log and ": FAILED (" not in log, log[-4000:] + finally: + server.stop() + forwarder.stop() + + +# --- `backup remove-target`: the target_not_empty state ------------------------------ + + +def _seed_target_not_empty(d: _Deployment, target: str) -> str: + """Put a finished restore (table ACTIVE, record DONE) back into the state + a crash before the first progress record leaves on MongoDB and Cassandra: + the target CREATING with its items and a RUNNING manifest record with no + progress, which the restarted server's reconciler ages, takes over, and + fails `target_not_empty` since the target is not empty. SQLite records + progress in the batch's own transaction, so no kill produces the state on + this deployment; the catalog is edited with the server stopped. Returns + the target's table id.""" + try: + import pysqlite3 as sqlite3 + except ImportError: + import sqlite3 + conn = sqlite3.connect(d.catalog, timeout=10) + try: + row = conn.execute( + "SELECT table_id FROM tables WHERE account_id = ? AND table_name = ?", (ACCOUNT, target) + ).fetchone() + assert row, f"no catalog row for {target}" + table_id = row[0] + n = conn.execute( + "UPDATE tables SET table_status = 'CREATING' WHERE table_id = ?", (table_id,) + ).rowcount + assert n == 1, n + n = conn.execute( + "UPDATE table_restores SET status = 'RUNNING', file_index = NULL, items_applied = NULL, " + "failure_reason = NULL, completed_at = NULL, heartbeat_at = '1970-01-01T00:00:00.000Z' " + "WHERE table_id = ? AND kind = 'manifest' AND status = 'DONE'", + (table_id,), + ).rowcount + assert n == 1, n + conn.commit() + finally: + conn.close() + return table_id + + +def test_remove_target_removes_a_target_not_empty_restore(deployment, tmp_path): + """The operator's remedy for a resume that found items and no progress. + A backup restored to ACTIVE is seeded back into the `target_not_empty` + state (`_seed_target_not_empty`: the shape a SIGKILL inside the first + data file on MongoDB, or inside the first batch on Cassandra, leaves; on + SQLite only a catalog edit produces it). The restarted server's + reconciler fails the record `target_not_empty` and leaves the target + CREATING, which `DeleteTable` refuses (`ResourceInUseException`) and + which keeps the backup in use (`BackupInUseException` to `DeleteBackup` + and to a second restore). `extenddb backup remove-target`, run with the + server stopped (on SQLite it takes the serve lock), removes the target + and its record and prints what it did; the restarted server then answers + `ResourceNotFoundException` for the target, `DeleteBackup` succeeds, and + a new restore into the name runs to ACTIVE with every item.""" + d = deployment + if d.backend != "sqlite": + pytest.skip("the seeded target_not_empty state is built on the SQLite catalog (the PostgreSQL command runs beside a live server and is the engine tests')") + server, ddb, creds = _started(d, str(tmp_path / "serve.log")) + try: + name, arn = _small_backed_up_table(ddb, "rmt") + target = f"{name}-r" + resp = ddb.restore_table_from_backup(TargetTableName=target, BackupArn=arn)["TableDescription"] + assert resp["TableStatus"] == "CREATING", resp + _wait_active(ddb, target) + assert _count(ddb, target) == HAND_BUILT_ITEMS + finally: + server.stop() + table_id = _seed_target_not_empty(d, target) + + server, ddb, creds = _started(d, str(tmp_path / "serve2.log")) + try: + deadline = time.monotonic() + 60 + while "target_not_empty" not in server.log_text(): + assert time.monotonic() < deadline, server.log_text()[-4000:] + time.sleep(0.2) + log = server.log_text() + failed = [line for line in log.splitlines() if "target_not_empty" in line] + assert any(target in line for line in failed), failed + assert ddb.describe_table(TableName=target)["Table"]["TableStatus"] == "CREATING" + with pytest.raises(ClientError) as refused: + ddb.delete_table(TableName=target) + assert refused.value.response["Error"]["Code"] == "ResourceInUseException", refused.value.response + with pytest.raises(ClientError) as refused: + ddb.delete_backup(BackupArn=arn) + assert refused.value.response["Error"]["Code"] == "BackupInUseException", refused.value.response + with pytest.raises(ClientError) as refused: + ddb.restore_table_from_backup(TargetTableName=f"{name}-again", BackupArn=arn) + assert refused.value.response["Error"]["Code"] == "BackupInUseException", refused.value.response + # The reconciler's next cycles leave it: the FAILED-target duty skips + # this reason, and the target is still CREATING after two more cycles. + time.sleep(2.5) + assert ddb.describe_table(TableName=target)["Table"]["TableStatus"] == "CREATING" + finally: + server.stop() + + res = d.run("backup", "remove-target", "--account", ACCOUNT, target) + assert res.returncode == 0, res.stdout + res.stderr + assert "=== extenddb backup remove-target ===" in res.stdout, res.stdout + res.stderr + assert 'Taking the maintenance claim as "extenddb backup remove-target on ' in res.stdout, res.stdout + assert ( + f"removed the restore target {target} (account {ACCOUNT}, table id {table_id}) and its record: " + f"its manifest restore from {arn} was FAILED (target_not_empty)" + ) in res.stdout, res.stdout + res.stderr + assert "The maintenance claim is released." in res.stdout, res.stdout + # A second run finds no record for the name and refuses, exit 1. + res = d.run("backup", "remove-target", "--account", ACCOUNT, target) + assert res.returncode == 1, res.stdout + res.stderr + assert f"{target} (account {ACCOUNT}) is not a restore target: no restore record names it" in res.stderr, ( + res.stdout + res.stderr + ) + + server, ddb, creds = _started(d, str(tmp_path / "serve3.log")) + try: + with pytest.raises(ClientError) as gone: + ddb.describe_table(TableName=target) + assert gone.value.response["Error"]["Code"] == "ResourceNotFoundException", gone.value.response + # The name is free and the backup no longer in use: a new restore runs. + resp = ddb.restore_table_from_backup(TargetTableName=target, BackupArn=arn)["TableDescription"] + assert resp["TableStatus"] == "CREATING", resp + _wait_active(ddb, target) + assert _count(ddb, target) == HAND_BUILT_ITEMS + ddb.delete_table(TableName=target) + _wait_table_gone_any_status(ddb, target) + ddb.delete_backup(BackupArn=arn) + _wait_backup_gone(ddb, arn) + log = server.log_text() + assert "target_not_empty" not in log, log[-4000:] + finally: + server.stop() + + +def _seed_running_with_progress(d: _Deployment, target: str) -> None: + """Put a finished restore (table ACTIVE, record DONE) back into the state + a server killed after the restore's first data file committed leaves + (the `restore_after_file_commit` kill point): the target CREATING with + its items and a RUNNING manifest record whose progress names the file + and its item count. Edited at the catalog with the server stopped, as + `_seed_target_not_empty` does.""" + try: + import pysqlite3 as sqlite3 + except ImportError: + import sqlite3 + conn = sqlite3.connect(d.catalog, timeout=10) + try: + row = conn.execute( + "SELECT table_id FROM tables WHERE account_id = ? AND table_name = ?", (ACCOUNT, target) + ).fetchone() + assert row, f"no catalog row for {target}" + table_id = row[0] + n = conn.execute( + "UPDATE tables SET table_status = 'CREATING' WHERE table_id = ?", (table_id,) + ).rowcount + assert n == 1, n + n = conn.execute( + "UPDATE table_restores SET status = 'RUNNING', file_index = 0, items_applied = ?, " + "failure_reason = NULL, completed_at = NULL " + "WHERE table_id = ? AND kind = 'manifest' AND status = 'DONE'", + (HAND_BUILT_ITEMS, table_id), + ).rowcount + assert n == 1, n + conn.commit() + finally: + conn.close() + + +def test_purge_fails_a_killed_servers_store_backed_restore_rather_than_resuming_it(deployment, tmp_path): + """`backup purge --confirm` under the SQLite serve lock, after a server + that was running a store-backed restore stopped (the record RUNNING with + recorded progress, `_seed_running_with_progress`): the command's own + reconciler cycle holds the maintenance claim, so it fails the record and + removes its target, as it does a legacy restore, and resumes nothing (no + `taken over by this instance` line, no restore job in the command). The + restarted server finds no target and no backup.""" + d = deployment + if d.backend != "sqlite": + pytest.skip("the purge's own reconciler cycle runs under the SQLite serve lock only") + server, ddb, creds = _started(d, str(tmp_path / "serve.log")) + try: + name, arn = _small_backed_up_table(ddb, "prg") + target = f"{name}-r" + resp = ddb.restore_table_from_backup(TargetTableName=target, BackupArn=arn)["TableDescription"] + assert resp["TableStatus"] == "CREATING", resp + _wait_active(ddb, target) + assert _count(ddb, target) == HAND_BUILT_ITEMS + finally: + server.stop() + _seed_running_with_progress(d, target) + + res = d.run("backup", "purge", "--confirm") + out = res.stdout + res.stderr + assert res.returncode == 0, out + assert "taken over by this instance" not in out, out + assert "resumed by this instance" not in out, out + assert "a maintenance claim is held; it is failed rather than resumed" in out, out + assert "Waiting for running jobs" not in out, out + + server, ddb, creds = _started(d, str(tmp_path / "serve2.log")) + try: + with pytest.raises(ClientError) as gone: + ddb.describe_table(TableName=target) + assert gone.value.response["Error"]["Code"] == "ResourceNotFoundException", gone.value.response + with pytest.raises(ClientError) as gone: + ddb.describe_backup(BackupArn=arn) + assert gone.value.response["Error"]["Code"] == "BackupNotFoundException", gone.value.response + log = server.log_text() + assert "taken over by this instance" not in log, log[-4000:] + finally: + server.stop() + + +def _wait_table_gone_any_status(ddb, name: str, timeout: float = 60.0) -> None: + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + try: + ddb.describe_table(TableName=name) + except ClientError as e: + assert e.response["Error"]["Code"] == "ResourceNotFoundException", e.response["Error"] + return + time.sleep(0.2) + raise AssertionError(f"{name} still exists after {timeout} s") + + +# --- hand-built prefixes: the format's rules met at the server's boundary ---------- + +HAND_BUILT_ITEMS = 12 + + +def _hand_item(i: int) -> dict: + """Item `i` of a hand-built data file, in the shape `_create_composite_table` + keys on (`pk` S, `sk` N) with the GSI's `g`.""" + return {"pk": {"S": f"p{i % 4:04d}"}, "sk": {"N": str(i)}, "g": {"S": f"g{i % 3}"}, "v": {"S": f"value {i}"}} + + +def _gzip_lines(rows: list[dict]) -> tuple[bytes, int]: + """The data file of `rows`: one `{"Item": ..}` line each, gzipped; the + compressed bytes and the uncompressed length.""" + import gzip + import io + + lines = b"".join(json.dumps({"Item": r}, separators=(",", ":")).encode() + b"\n" for r in rows) + buffer = io.BytesIO() + with gzip.GzipFile(fileobj=buffer, mode="wb", mtime=0) as gz: + gz.write(lines) + return buffer.getvalue(), len(lines) + + +def _write_manifest(prefix: str, manifest: dict) -> None: + """Write the manifest and its sha256 sidecar under `prefix`, the files + private as the store writes them, and the two DynamoDB export manifests + for the manifest's data files, so the prefix stays a whole backup in the + format's sense (one `md5Checksum` and one `itemCount` per file, agreeing + with the extenddb manifest by construction).""" + import hashlib + + text = json.dumps(manifest, separators=(",", ":")).encode() + path = os.path.join(prefix, "extenddb-manifest.json") + with open(path, "wb") as f: + f.write(text) + os.chmod(path, 0o600) + sidecar = os.path.join(prefix, "extenddb-manifest.sha256") + with open(sidecar, "w") as f: + f.write(f"{hashlib.sha256(text).hexdigest()} extenddb-manifest.json\n") + os.chmod(sidecar, 0o600) + summary_path = os.path.join(prefix, "manifest-summary.json") + with open(summary_path) as f: + summary = json.load(f) + s3_prefix = summary["s3Prefix"] + files_path = os.path.join(prefix, "manifest-files.json") + with open(files_path, "w") as f: + for entry in manifest["data_files"]: + line = { + "itemCount": entry["item_count"], + "md5Checksum": entry["md5"], + "dataFileS3Key": f"{s3_prefix}/{entry['key']}" if s3_prefix else entry["key"], + } + f.write(json.dumps(line, separators=(",", ":")) + "\n") + os.chmod(files_path, 0o600) + summary["itemCount"] = sum(entry["item_count"] for entry in manifest["data_files"]) + summary["billedSizeBytes"] = manifest["backup_size_bytes"] + with open(summary_path, "w") as f: + f.write(json.dumps(summary, separators=(",", ":"))) + os.chmod(summary_path, 0o600) + + +def _write_data_files(prefix: str, files: list[list[dict]]) -> list[dict]: + """Replace the data files under `prefix` with one file per list of rows + (unordered: no `key_range`); the manifest entries.""" + import base64 + import hashlib + import shutil + + data_dir = os.path.join(prefix, "data") + shutil.rmtree(data_dir) + os.mkdir(data_dir, 0o700) + entries = [] + for index, rows in enumerate(files, start=1): + compressed, uncompressed = _gzip_lines(rows) + key = f"data/{index:06d}.json.gz" + path = os.path.join(prefix, key) + with open(path, "wb") as f: + f.write(compressed) + os.chmod(path, 0o600) + entries.append({ + "key": key, "item_count": len(rows), + "sha256": hashlib.sha256(compressed).hexdigest(), + "md5": base64.b64encode(hashlib.md5(compressed).digest()).decode(), + "bytes": len(compressed), "uncompressed_bytes": uncompressed, "key_range": None, + }) + return entries + + +def _update_row(d: _Deployment, arn: str, size_bytes: int, item_count: int) -> None: + """Set the `backup_index` row's `size_bytes` and `item_count` to the + rewritten prefix's, as `backup sync` would record them.""" + backup_id = arn.rsplit("/", 1)[1] + if d.backend == "sqlite": + try: + import pysqlite3 as sqlite3 + except ImportError: + import sqlite3 + conn = sqlite3.connect(d.catalog, timeout=10) + try: + n = conn.execute( + "UPDATE backup_index SET size_bytes = ?, item_count = ? WHERE account_id = ? AND backup_id = ?", + (size_bytes, item_count, ACCOUNT, backup_id), + ).rowcount + conn.commit() + finally: + conn.close() + else: + import psycopg2 + + conn = psycopg2.connect(d.catalog) + try: + with conn.cursor() as cur: + cur.execute( + "UPDATE backup_index SET size_bytes = %s, item_count = %s WHERE account_id = %s AND backup_id = %s", + (size_bytes, item_count, ACCOUNT, backup_id), + ) + n = cur.rowcount + conn.commit() + finally: + conn.close() + assert n == 1, f"{n} backup_index rows match {arn}" + + +def _small_backed_up_table(ddb, prefix: str) -> tuple[str, str]: + """A composite-key table with a GSI holding HAND_BUILT_ITEMS items, + backed up to AVAILABLE; the name and the ARN.""" + name = f"{prefix}-{uuid.uuid4().hex[:8]}" + _create_composite_table(ddb, name, gsi=True) + pending = {name: [{"PutRequest": {"Item": _hand_item(i)}} for i in range(HAND_BUILT_ITEMS)]} + while pending: + pending = ddb.batch_write_item(RequestItems=pending).get("UnprocessedItems") or {} + arn = ddb.create_backup(TableName=name, BackupName="nightly")["BackupDetails"]["BackupArn"] + _wait_backup_available(ddb, arn, timeout=60) + return name, arn + + +def _restore_fails(server: _Server, ddb, target: str, arn: str, *needles: str) -> str: + """The restore of `arn` into `target` is accepted CREATING and the target + is gone once the job fails, no item of it ever readable; the server log's + FAILED line, which must carry every needle. The job logs the line right + after it removes the target, so the log is polled for a moment.""" + resp = ddb.restore_table_from_backup(TargetTableName=target, BackupArn=arn)["TableDescription"] + assert resp["TableStatus"] == "CREATING", resp + assert resp["RestoreSummary"]["SourceBackupArn"] == arn, resp + _wait_table_gone(ddb, target) + deadline = time.monotonic() + 15 + while True: + failed = [line for line in server.log_text().splitlines() if ": FAILED (" in line and f" into {target} " in line] + if failed or time.monotonic() > deadline: + break + time.sleep(0.25) + assert len(failed) == 1, server.log_text()[-4000:] + for needle in needles: + assert needle in failed[0], failed[0] + assert "the target was removed" in failed[0], failed[0] + assert ddb.describe_backup(BackupArn=arn)["BackupDescription"]["BackupDetails"]["BackupStatus"] == "AVAILABLE" + return failed[0] + + +DUPLICATE_KEY_TEXT = "holds a primary key the target already has (the backup holds a key twice)" + + +def test_an_unordered_backup_holding_one_key_in_two_files_fails_the_restore(deployment, tmp_path): + """The backup's prefix rewritten under the root as an unordered backup of + two data files that share item 0's key (the manifest and its sidecar + rewritten, the row's size and count updated): the restore is accepted, + the job fails the second file on the duplicate, the target is gone and + the log names the reason. On PostgreSQL and SQLite, the two deployments + here, the detector is the base row's plain INSERT and the reason is its + text (`DUPLICATE_KEY_TEXT`); the engine's final count against the + manifest's total ("the manifest lists") is the detector on a backend + whose write is an upsert (Cassandra, through the matrix). A backend with + neither would restore 12 of 13 items.""" + d = deployment + server, ddb, creds = _started(d, str(tmp_path / "serve.log")) + try: + name, arn = _small_backed_up_table(ddb, "dup") + prefix = _prefix(d, name, arn) + with open(os.path.join(prefix, "extenddb-manifest.json")) as f: + manifest = json.load(f) + rows = [_hand_item(i) for i in range(HAND_BUILT_ITEMS)] + entries = _write_data_files(prefix, [rows[:6], rows[6:] + [rows[0]]]) + manifest["item_order"] = "unordered" + manifest["data_files"] = entries + manifest["backup_size_bytes"] = sum(e["bytes"] for e in entries) + manifest["source_table"]["item_count"] = HAND_BUILT_ITEMS + 1 + _write_manifest(prefix, manifest) + _update_row(d, arn, manifest["backup_size_bytes"], HAND_BUILT_ITEMS + 1) + line = _restore_fails(server, ddb, f"{name}-r", arn, "data/000002.json.gz", DUPLICATE_KEY_TEXT) + assert "the manifest lists" not in line, line + finally: + server.stop() + + +def test_a_key_ordered_file_leaving_its_key_range_fails_the_restore(deployment, tmp_path): + """The backup's manifest rewritten so its first data file's `key_range` + ends at its first item: the reader's key-range check ends the file's + stream at the second item, the restore is FAILED naming the file and the + rule, and the target is gone.""" + d = deployment + server, ddb, creds = _started(d, str(tmp_path / "serve.log")) + try: + name, arn = _small_backed_up_table(ddb, "range") + prefix = _prefix(d, name, arn) + with open(os.path.join(prefix, "extenddb-manifest.json")) as f: + manifest = json.load(f) + assert manifest["item_order"] == "key_ordered", manifest["item_order"] + first = manifest["data_files"][0] + assert first["item_count"] >= 2, first + first["key_range"]["last"] = first["key_range"]["first"] + _write_manifest(prefix, manifest) + _restore_fails(server, ddb, f"{name}-r", arn, "data/000001.json.gz", "key_range") + finally: + server.stop() + + +# --- the multi-part-key preview ------------------------------------------------------ + +MULTIPART_ITEMS = 60 + + +def _scan_items(ddb, name: str) -> list[dict]: + items: list[dict] = [] + kwargs: dict = {"TableName": name, "ConsistentRead": True} + while True: + resp = ddb.scan(**kwargs) + items += resp["Items"] + if "LastEvaluatedKey" not in resp: + return items + kwargs["ExclusiveStartKey"] = resp["LastEvaluatedKey"] + + +def _canonical(item: dict) -> str: + return json.dumps(item, sort_keys=True, separators=(",", ":")) + + +def test_a_multi_part_key_table_restores_with_the_sources_item_set(deployment, tmp_path): + """A table keyed on two HASH attributes (`[limits] + allow_multipart_table_keys = true` on this cell's deployment, so a + server that refuses the second HASH key fails the cell rather than + skipping it) backs up to the store as an `unordered` backup and + restores: the restored table's `Scan` holds what the source's held, + with the source's key schema and attribute definitions. The comparison + is with the source's `Scan`, not with the items written: the preview's + item paths address such a table by its first HASH attribute on every + backend (on SQLite the source keeps one item per `h1` value; on + PostgreSQL a GetItem by the whole key finds nothing on the source + either), and that is the preview's behaviour, not the restore's; for the + same reason the cell makes no GetItem by the whole key.""" + d = deployment + d.set_limit("allow_multipart_table_keys", "true") + server, ddb, creds = _started(d, str(tmp_path / "serve.log")) + try: + name = f"mpk-{uuid.uuid4().hex[:8]}" + key_schema = [ + {"AttributeName": "h1", "KeyType": "HASH"}, + {"AttributeName": "h2", "KeyType": "HASH"}, + ] + attr_defs = [ + {"AttributeName": "h1", "AttributeType": "S"}, + {"AttributeName": "h2", "AttributeType": "N"}, + ] + ddb.create_table(TableName=name, KeySchema=key_schema, AttributeDefinitions=attr_defs, BillingMode="PAY_PER_REQUEST") + _wait_active(ddb, name) + for i in range(MULTIPART_ITEMS): + ddb.put_item( + TableName=name, + Item={"h1": {"S": f"tenant-{i % 5}"}, "h2": {"N": str(i)}, "v": {"S": f"value-{i}"}, "n": {"N": str(i * 1.5)}}, + ) + expected = sorted(_canonical(i) for i in _scan_items(ddb, name)) + assert expected, "the source holds no item" + print(f"branch: the source holds {len(expected)} of {MULTIPART_ITEMS} items written ({d.backend})") + arn = ddb.create_backup(TableName=name, BackupName="nightly")["BackupDetails"]["BackupArn"] + _wait_backup_available(ddb, arn, timeout=60) + with open(os.path.join(_prefix(d, name, arn), "extenddb-manifest.json")) as f: + manifest = json.load(f) + assert manifest["item_order"] == "unordered", manifest["item_order"] + assert manifest["source_table"]["key_schema"] == key_schema, manifest["source_table"]["key_schema"] + target = f"{name}-r" + resp = ddb.restore_table_from_backup(TargetTableName=target, BackupArn=arn)["TableDescription"] + assert resp["TableStatus"] == "CREATING", resp + assert resp["RestoreSummary"]["SourceBackupArn"] == arn, resp + _wait_active(ddb, target, timeout=120) + table = ddb.describe_table(TableName=target)["Table"] + assert table["KeySchema"] == key_schema, table["KeySchema"] + assert sorted(table["AttributeDefinitions"], key=lambda a: a["AttributeName"]) == attr_defs, table["AttributeDefinitions"] + assert "RestoreSummary" not in table, table + got = sorted(_canonical(i) for i in _scan_items(ddb, target)) + assert got == expected, f"{len(got)} restored items against {len(expected)} in the source" + log = server.log_text() + assert f" into {target} " in log and ": ACTIVE, " in log and ": FAILED (" not in log, log[-4000:] + finally: + server.stop() diff --git a/tests/test_backup_restore_fidelity.py b/tests/test_backup_restore_fidelity.py index 8d152b607..12da741c7 100644 --- a/tests/test_backup_restore_fidelity.py +++ b/tests/test_backup_restore_fidelity.py @@ -6,8 +6,9 @@ Every backend; hash-only keys and composite keys with S, N, and B sort keys, plus secondary indexes and provisioned throughput. A restore must reach ACTIVE with the source's key schema, attribute definitions, and exactly the source's -items, attribute for attribute. (Multi-part base keys are a preview that -restore refuses; that refusal is tested at the storage level.) Comparing whole items rather than counts is what catches a +items, attribute for attribute. (Multi-part base keys are a preview: a +store-backed backup of such a table restores through the store, an in-catalog +one is refused, and the refusal is tested at the storage level.) Comparing whole items rather than counts is what catches a backend that restores the right number of rows under the wrong keys. Readiness assessment P0-2: on PostgreSQL every table with a sort key backed up @@ -44,29 +45,15 @@ # Where the server under test writes its backups: `filesystem` or `s3` (a # configured `[backup.store]`, under which CreateBackup is asynchronous and -# backs up vector tables) or `none` (the in-catalog create). The CI jobs set -# it beside EXTENDDB_TEST_ENDPOINT; the one test whose expected outcome -# depends on it fails when it is unset rather than accept either outcome. +# backs up vector tables, and RestoreTableFromBackup runs the restore job +# from the store) or `none` (the in-catalog create). The CI jobs set it +# beside EXTENDDB_TEST_ENDPOINT; the one test whose expected outcome depends +# on it fails when it is unset rather than accept either outcome. Every cell +# that restores a backup it created waits for AVAILABLE first +# (`_create_backup`), on either path. BACKUP_STORE = os.environ.get("EXTENDDB_TEST_BACKUP_STORE", "").strip() or None BACKUP_STORE_VALUES = ("filesystem", "s3", "none") -# RestoreTableFromBackup of a backup written to a configured store answers -# ValidationException ("is in a backup store; this release restores -# in-catalog backups only") until the restore job of PR 9a lands. The cells -# that restore a backup this server created skip on such a server and come -# back with PR 9a; the in-catalog create's backups (`none`, and the service) -# restore as before. The marker is gated on the target as well as the -# variable, so a stale `EXTENDDB_TEST_BACKUP_STORE=filesystem` in the -# environment of a real-DynamoDB run (which does not require the variable) -# cannot skip the 24 cells against the service. -restores_a_backup_of_this_server = pytest.mark.skipif( - BACKUP_STORE in ("filesystem", "s3") and not _is_real_dynamodb(), - reason=( - f"EXTENDDB_TEST_BACKUP_STORE={BACKUP_STORE}: RestoreTableFromBackup of a backup " - "in the configured store is refused until the restore job lands (PR 9a)" - ), -) - @pytest.fixture(scope="session") def backup_store_component(endpoint_url): @@ -428,13 +415,11 @@ def _round_trip(client, sort_kind: str | None, hash_kind: str = "S") -> None: _cleanup(client, backup_arn, restored, source) -@restores_a_backup_of_this_server @pytest.mark.parametrize("hash_kind", ["S", "N", "B"]) def test_restore_hash_only_table(dynamodb_client, hash_kind): _round_trip(dynamodb_client, None, hash_kind) -@restores_a_backup_of_this_server @pytest.mark.parametrize("sort_kind", ["S", "N", "B"]) def test_restore_composite_key_table(dynamodb_client, sort_kind): _round_trip(dynamodb_client, sort_kind) @@ -451,7 +436,6 @@ def _index_by_name(indexes: list[dict]) -> dict: return {i["IndexName"]: i for i in indexes} -@restores_a_backup_of_this_server def test_restore_preserves_indexes_and_provisioned_throughput(dynamodb_client): """GSIs and LSIs, their key schemas, projections, and throughput, the table's provisioned throughput, and every index's contents come back.""" @@ -639,7 +623,6 @@ def ranked(table: str) -> list[str]: _cleanup(client, backup_arn, restored, source) -@restores_a_backup_of_this_server def test_restore_normalizes_stale_throughput_after_on_demand_switch(dynamodb_client): """A backup taken after an on-demand switch does not restore stale capacity.""" client = dynamodb_client @@ -714,7 +697,6 @@ def test_restore_normalizes_stale_throughput_after_on_demand_switch(dynamodb_cli _cleanup(client, backup_arn, restored, source) -@restores_a_backup_of_this_server def test_restore_override_pay_per_request_to_provisioned(dynamodb_client): client = dynamodb_client source = f"restore-override-ppr-{uuid.uuid4().hex[:10]}" @@ -753,7 +735,6 @@ def test_restore_override_pay_per_request_to_provisioned(dynamodb_client): _cleanup(client, backup_arn, restored, source) -@restores_a_backup_of_this_server def test_restore_override_provisioned_to_pay_per_request(dynamodb_client): client = dynamodb_client source = f"restore-override-prov-{uuid.uuid4().hex[:10]}" @@ -788,7 +769,6 @@ def test_restore_override_provisioned_to_pay_per_request(dynamodb_client): _cleanup(client, backup_arn, restored, source) -@restores_a_backup_of_this_server def test_provisioned_restore_override_requiring_gsi_override_leaves_no_target( dynamodb_client, ): @@ -1031,7 +1011,6 @@ def large_backup(dynamodb_client): _cleanup(client, backup_arn, source) -@restores_a_backup_of_this_server def test_restore_response_arrives_before_its_copy_finishes(dynamodb_client, large_backup): """The CREATING response reaches a client with a 10 s read timeout and no retries before the copy finishes, and the table then reaches ACTIVE with @@ -1094,7 +1073,6 @@ def _table_appears(client, name: str, timeout: float) -> bool: time.sleep(_poll_interval() * 5) -@restores_a_backup_of_this_server def test_restore_survives_a_client_that_disconnects_before_the_response( dynamodb_client, large_backup ): @@ -1150,7 +1128,6 @@ def test_restore_survives_a_client_that_disconnects_before_the_response( _cleanup(dynamodb_client, None, target) -@restores_a_backup_of_this_server def test_first_read_after_active_succeeds(dynamodb_client, large_backup): """A read on the restore target while it is CREATING is ResourceNotFoundException, and the first read after DescribeTable reports @@ -1207,7 +1184,6 @@ def test_first_read_after_active_succeeds(dynamodb_client, large_backup): _cleanup(client, None, target) -@restores_a_backup_of_this_server def test_create_backup_of_a_creating_table_is_table_not_found(dynamodb_client, large_backup): """CreateBackup on a table that is still CREATING is TableNotFoundException "Table not found: " (measured on DynamoDB), @@ -1237,7 +1213,6 @@ def test_create_backup_of_a_creating_table_is_table_not_found(dynamodb_client, l _cleanup(client, None, target) -@restores_a_backup_of_this_server def test_restore_into_a_creating_target_and_a_second_restore_of_its_backup_are_refused( dynamodb_client, large_backup ): @@ -1280,7 +1255,6 @@ def test_restore_into_a_creating_target_and_a_second_restore_of_its_backup_are_r _cleanup(client, None, target, other) -@restores_a_backup_of_this_server def test_target_and_backup_are_held_while_the_copy_runs(dynamodb_client, large_backup): """DeleteTable of a CREATING restore target is ResourceInUseException, and (ExtendDB) DeleteBackup of its source is BackupInUseException. @@ -1731,7 +1705,6 @@ def test_backup_arn_shape_and_account_are_checked(dynamodb_client): _cleanup(client, backup_arn, source) -@restores_a_backup_of_this_server def test_empty_index_overrides_restore_no_indexes_and_sse_override_is_applied(dynamodb_client): client = dynamodb_client source = f"restore-noop-{uuid.uuid4().hex[:10]}" @@ -1782,7 +1755,6 @@ def test_empty_index_overrides_restore_no_indexes_and_sse_override_is_applied(dy _cleanup(client, backup_arn, plain, restored, source) -@restores_a_backup_of_this_server @pytest.mark.parametrize("billing", ["PROVISIONED", "PAY_PER_REQUEST"]) def test_index_overrides_listing_the_backups_indexes_restore_them(dynamodb_client, billing): """GlobalSecondaryIndexOverride and LocalSecondaryIndexOverride built from @@ -1953,7 +1925,6 @@ def test_index_overrides_listing_the_backups_indexes_restore_them(dynamodb_clien _cleanup(client, backup_arn, changed, restored, source) -@restores_a_backup_of_this_server def test_on_demand_throughput_override_is_applied(dynamodb_client): """OnDemandThroughputOverride is the restored table's on-demand throughput, equal to the backup's or changed (measured on DynamoDB). @@ -2009,7 +1980,7 @@ def test_on_demand_throughput_override_is_applied(dynamodb_client): @extenddb_only -def test_create_backup_of_a_vector_table_is_refused(dynamodb_client, backup_store_component): +def test_vector_table_backup_refused_in_catalog_and_restored_from_store(dynamodb_client, backup_store_component): """CreateBackup of a table with vector indexes: the outcome of the server's configured path, read from EXTENDDB_TEST_BACKUP_STORE and checked against the server's `/health` (`backup_store_component`). @@ -2019,17 +1990,18 @@ def test_create_backup_of_a_vector_table_is_refused(dynamodb_client, backup_stor created. `filesystem` or `s3`: the store-backed create backs the table up (the manifest records its vector indexes): CreateBackup answers CREATING, DescribeBackup reaches AVAILABLE with the table's features, - ListBackups lists it, and RestoreTableFromBackup of it answers the - ValidationException naming the store, which PR 9a's restore job - replaces with the restore (its refusal then narrows to a target without - vector support). The test fails rather than accept either outcome when - the variable is unset (the fixture fails first), and prints which branch - ran. + ListBackups lists it, and RestoreTableFromBackup of it restores the + table with its vector index: the response is CREATING with a + RestoreSummary, the target reaches ACTIVE, DescribeTable shows the + vector index ACTIVE, and SearchVectors on it answers the item. The test + fails rather than accept either outcome when the variable is unset (the + fixture fails first), and prints which branch ran. """ from test_vector_index_query_scan import _error_message, _signed_post client = dynamodb_client name = f"restore-vec-{uuid.uuid4().hex[:10]}" + restored = f"{name}-r" resp = _signed_post( "CreateTable", { @@ -2106,21 +2078,39 @@ def test_create_backup_of_a_vector_table_is_refused(dynamodb_client, backup_stor assert [g["IndexName"] for g in gsis] == ["by-g"], desc["SourceTableFeatureDetails"] listed = client.list_backups(TableName=name)["BackupSummaries"] assert [b["BackupArn"] for b in listed] == [backup_arn], listed - # Until PR 9a's restore job, the restore of a backup in the store is - # refused; PR 9a replaces this assertion with the round trip. - with pytest.raises(ClientError) as exc_info: - client.restore_table_from_backup(TargetTableName=f"{name}-r", BackupArn=backup_arn) - error = exc_info.value.response["Error"] - assert error["Code"] == "ValidationException", error - assert error["Message"] == ( - f"Backup {backup_arn} is in a backup store; this release restores in-catalog " - "backups only" - ), error + # The round trip: the restore runs from the store and the target + # comes back with its vector index. + resp = client.restore_table_from_backup(TargetTableName=restored, BackupArn=backup_arn) + table = resp["TableDescription"] + assert table["TableStatus"] == "CREATING", table + assert table["RestoreSummary"]["SourceBackupArn"] == backup_arn, table + assert table["RestoreSummary"]["RestoreInProgress"] is True, table + wait_for_active(client, restored, timeout=RESTORE_TIMEOUT_S) + desc = _signed_post("DescribeTable", {"TableName": restored}).json()["Table"] + assert "RestoreSummary" not in desc, desc + indexes = desc.get("VectorIndexes") or [] + assert [v["IndexName"] for v in indexes] == ["vidx"], desc + assert indexes[0]["IndexStatus"] == "ACTIVE", indexes + assert indexes[0]["Dimensions"] == 4, indexes + assert [g["IndexName"] for g in desc["GlobalSecondaryIndexes"]] == ["by-g"], desc + search = _signed_post( + "SearchVectors", + { + "TableName": restored, + "IndexName": "vidx", + "SearchVector": [{"N": "0.1"}] * 4, + "TopK": 1, + }, + ) + assert search.status_code == 200, search.text + results = search.json()["SearchResults"] + assert [r["Item"]["pk"] for r in results] == [{"S": "a"}], results + got = client.get_item(TableName=restored, Key={"pk": {"S": "a"}}, ConsistentRead=True) + assert got["Item"]["g"] == {"S": "g1"}, got finally: - _cleanup(client, backup_arn, name) + _cleanup(client, backup_arn, name, restored) -@restores_a_backup_of_this_server def test_update_table_on_a_restored_table_reports_no_restore_summary(dynamodb_client): """Once a restored table is ACTIVE, no description-returning response carries RestoreSummary: UpdateTable's no more than DescribeTable's. @@ -2153,7 +2143,6 @@ def test_update_table_on_a_restored_table_reports_no_restore_summary(dynamodb_cl _cleanup(client, backup_arn, restored, source) -@restores_a_backup_of_this_server @extenddb_only def test_a_provisioned_gsi_without_throughput_restores_with_the_table_throughput( dynamodb_client, @@ -2522,7 +2511,6 @@ def test_create_backup_accepts_the_table_arn(dynamodb_client): _cleanup(client, backup_arn, source) -@restores_a_backup_of_this_server def test_table_class_is_not_restored(dynamodb_client): """A STANDARD_INFREQUENT_ACCESS source restores as STANDARD (measured on DynamoDB); DescribeBackup still reports nothing of the diff --git a/tests/test_backup_store_config.py b/tests/test_backup_store_config.py index 2af5a565f..86e6866d1 100644 --- a/tests/test_backup_store_config.py +++ b/tests/test_backup_store_config.py @@ -506,8 +506,8 @@ def test_an_in_memory_database_with_a_filesystem_store_and_without_one(self, sql default of that build) on the SQLite build: with `[backup.store.filesystem] path` set the store is configured and healthy, its identity file is written, and a backup created there is written to the store (CREATING, - then AVAILABLE with its manifest under the root; its restore is - refused naming the store until the restore job lands); without + then AVAILABLE with its manifest under the root) and restores from it + with its item; without `[backup.store]` there is no default root (no database file to put it beside), no component, and CreateBackup writes a legacy backup with the warning naming `[backup.store]`.""" @@ -537,10 +537,15 @@ def test_an_in_memory_database_with_a_filesystem_store_and_without_one(self, sql _wait_backup_available(ddb, arn) prefix = os.path.join(root, "123456789012", "Music", arn.rsplit("/", 1)[1]) assert os.path.isfile(os.path.join(prefix, "extenddb-manifest.json")), os.listdir(root) - with pytest.raises(ClientError) as refused: - ddb.restore_table_from_backup(TargetTableName="Restored", BackupArn=arn) - assert refused.value.response["Error"]["Code"] == "ValidationException", refused.value.response - assert "is in a backup store" in refused.value.response["Error"]["Message"], refused.value.response + restored = ddb.restore_table_from_backup(TargetTableName="Restored", BackupArn=arn) + assert restored["TableDescription"]["TableStatus"] == "CREATING", restored + deadline = time.monotonic() + 60 + while ddb.describe_table(TableName="Restored")["Table"]["TableStatus"] != "ACTIVE": + assert time.monotonic() < deadline, "the restore did not finish" + time.sleep(0.1) + item = ddb.get_item(TableName="Restored", Key={"pk": {"S": "a"}})["Item"] + assert item == {"pk": {"S": "a"}, "v": {"N": "1"}}, item + assert "restore of " + arn in server.log_text(), "the restore ran as a store-backed job" assert "configure [backup.store]" not in server.log_text(), server.log_text() finally: server.stop() @@ -1317,6 +1322,22 @@ def claim_rows(): finally: server.stop() + def test_remove_target_creates_neither_the_default_root_nor_its_identity_file(self, sqlite_deployment): + # `backup remove-target` needs no store: on a fresh deployment it + # refuses a table no restore record names and leaves the default root + # as it found it, missing, then present and empty. + d = sqlite_deployment + root = os.path.join(d.dir, "backups") + res = d.run("backup", "remove-target", "--account", "000000000000", "nosuch") + assert res.returncode == 1, res.stdout + res.stderr + assert "nosuch (account 000000000000) is not a restore target" in res.stderr, res.stderr + assert not os.path.exists(root), "remove-target created the default root" + os.mkdir(root, 0o700) + res = d.run("backup", "remove-target", "--account", "000000000000", "nosuch") + assert res.returncode == 1, res.stdout + res.stderr + assert "is not a restore target" in res.stderr, res.stderr + assert os.listdir(root) == [], "remove-target wrote into the default root" + def test_backup_list_and_purge_on_an_empty_store_and_the_serve_lock(self, sqlite_deployment, tmp_path): d = sqlite_deployment # Create the default root with a server once, then stop it. @@ -1496,7 +1517,12 @@ def command(*args): with conn.cursor() as cur: cur.execute("SET application_name = 'holds the serving key'") cur.execute("SELECT pg_advisory_lock(%s, 2)", (0x00454442,)) - for args in (("backup", "list"), ("backup", "purge"), ("backup", "purge", "--confirm")): + for args in ( + ("backup", "list"), + ("backup", "purge"), + ("backup", "purge", "--confirm"), + ("backup", "remove-target", "--account", "000000000000", "nosuch"), + ): res = command(*args) out = res.stdout + res.stderr assert res.returncode == 6, f"{args}: {out}" @@ -1509,7 +1535,11 @@ def command(*args): res = command("migrate", "--forget", "5", *lh._pg_args()) assert res.returncode == 0, res.stdout + res.stderr - for args in (("backup", "list"), ("backup", "purge", "--confirm")): + for args in ( + ("backup", "list"), + ("backup", "purge", "--confirm"), + ("backup", "remove-target", "--account", "000000000000", "nosuch"), + ): res = command(*args) out = res.stdout + res.stderr assert res.returncode == 3, f"{args}: {out}" diff --git a/tests/test_mongodb_backup_restore.py b/tests/test_mongodb_backup_restore.py index c68c34cdb..3f05c7bc0 100644 --- a/tests/test_mongodb_backup_restore.py +++ b/tests/test_mongodb_backup_restore.py @@ -5,7 +5,10 @@ The table operations are exercised through the DynamoDB API. These tests are gated by the MongoDB runner because the backup metadata being checked is part -of the MongoDB implementation; other backends retain the shared API tests. +of the MongoDB implementation; other backends retain the shared API tests. The +runner's server has a filesystem backup store, so CreateBackup is the +store-backed job and RestoreTableFromBackup the store-backed restore; the +cells at the end meet the MongoDB target's own rules through that path. """ from __future__ import annotations @@ -29,7 +32,13 @@ def mongodb_container() -> str: return container +BACKUP_STORE = os.environ.get("EXTENDDB_TEST_BACKUP_STORE", "").strip() or None + + def _create_backup(client, table_name: str, backup_name: str) -> str: + """CreateBackup, waited for AVAILABLE: with the runner's filesystem store + the create is the asynchronous job, and a restore or a delete of a + CREATING backup is BackupInUseException.""" deadline = time.monotonic() + 30 while True: try: @@ -37,7 +46,8 @@ def _create_backup(client, table_name: str, backup_name: str) -> str: TableName=table_name, BackupName=backup_name, ) - return response["BackupDetails"]["BackupArn"] + arn = response["BackupDetails"]["BackupArn"] + break except ClientError as exc: if ( exc.response["Error"]["Code"] @@ -46,6 +56,8 @@ def _create_backup(client, table_name: str, backup_name: str) -> str: ): raise time.sleep(0.2) + _wait_backup_available(client, arn) + return arn def _remove_legacy_capacity_metadata(container: str, backup_arn: str) -> None: @@ -239,7 +251,16 @@ def test_restore_preserves_provisioned_throughput_and_secondary_indexes( def test_restore_legacy_backup_falls_back_and_populates_lsi( dynamodb_client, unique_table_name, mongodb_container ): - """A pre-throughput backup still restores and its LSI remains queryable.""" + """A pre-throughput backup still restores and its LSI remains queryable. + The backup edited is an in-catalog one (the legacy `backups` document), + so the cell runs only on a server without a backup store: + `devtools/run-mongodb-tests --backup-store none -- --pytest --filter + test_mongodb_backup_restore` (the CI jobs run the suite with a filesystem + store, where this cell skips). With a store, CreateBackup writes the + store and the fallback is pinned by the crate's + `restore_uses_legacy_fallback_when_capacity_metadata_is_missing`.""" + if BACKUP_STORE != "none": + pytest.skip("the legacy throughput fallback is a rule of in-catalog backups (EXTENDDB_TEST_BACKUP_STORE is not none)") source = unique_table_name restored = f"{source}-restored" dynamodb_client.create_table( @@ -310,3 +331,250 @@ def test_restore_legacy_backup_falls_back_and_populates_lsi( _cleanup(dynamodb_client, restored, source) if backup_arn: dynamodb_client.delete_backup(BackupArn=backup_arn) + + +# --- the store-backed restore's MongoDB rules --------------------------------------- +# +# These cells rewrite a backup's prefix under the runner's filesystem store +# (EXTENDDB_TEST_BACKUP_ROOT) and read the restore job's log lines from the +# server's log (EXTENDDB_TEST_SERVER_LOG); devtools/run-mongodb-tests exports +# both. A rule the backend's write path enforces (an `N` sort key beyond +# Decimal128) is met inside the job, so the restore is FAILED with that text +# as its reason and the client sees the target disappear; a rule the request +# checks (vector indexes into a backend without vector support) is a +# synchronous ValidationException before any table exists. + +ACCOUNT = "123456789012" + + +@pytest.fixture() +def backup_root(mongodb_container) -> str: + root = os.environ.get("EXTENDDB_TEST_BACKUP_ROOT", "").strip() + log = os.environ.get("EXTENDDB_TEST_SERVER_LOG", "").strip() + if not root or not log: + pytest.skip("requires devtools/run-mongodb-tests (EXTENDDB_TEST_BACKUP_ROOT, EXTENDDB_TEST_SERVER_LOG)") + return root + + +def _wait_backup_available(client, arn: str, timeout: float = 120.0) -> None: + deadline = time.monotonic() + timeout + while True: + status = client.describe_backup(BackupArn=arn)["BackupDescription"]["BackupDetails"]["BackupStatus"] + if status == "AVAILABLE": + return + assert status == "CREATING", status + assert time.monotonic() < deadline, f"backup {arn} still CREATING after {timeout} s" + time.sleep(0.2) + + +def _prefix(root: str, table: str, arn: str) -> str: + return os.path.join(root, ACCOUNT, table, arn.rsplit("/", 1)[1]) + + +def _read_manifest(prefix: str) -> dict: + with open(os.path.join(prefix, "extenddb-manifest.json")) as f: + return json.load(f) + + +def _write_manifest(prefix: str, manifest: dict) -> None: + """The manifest and its sha256 sidecar rewritten under `prefix`, private + as the store writes them, and the two DynamoDB export manifests for the + manifest's data files (one `md5Checksum` and one `itemCount` per file, + agreeing with the extenddb manifest by construction), so the prefix + stays a whole backup in the format's sense, as `tests/test_backup_job.py` + and `devtools/backup-restore-matrix` rewrite theirs.""" + import hashlib + + text = json.dumps(manifest, separators=(",", ":")).encode() + path = os.path.join(prefix, "extenddb-manifest.json") + with open(path, "wb") as f: + f.write(text) + os.chmod(path, 0o600) + sidecar = os.path.join(prefix, "extenddb-manifest.sha256") + with open(sidecar, "w") as f: + f.write(f"{hashlib.sha256(text).hexdigest()} extenddb-manifest.json\n") + os.chmod(sidecar, 0o600) + summary_path = os.path.join(prefix, "manifest-summary.json") + with open(summary_path) as f: + summary = json.load(f) + s3_prefix = summary["s3Prefix"] + files_path = os.path.join(prefix, "manifest-files.json") + with open(files_path, "w") as f: + for entry in manifest["data_files"]: + line = { + "itemCount": entry["item_count"], + "md5Checksum": entry["md5"], + "dataFileS3Key": f"{s3_prefix}/{entry['key']}" if s3_prefix else entry["key"], + } + f.write(json.dumps(line, separators=(",", ":")) + "\n") + os.chmod(files_path, 0o600) + summary["itemCount"] = sum(entry["item_count"] for entry in manifest["data_files"]) + summary["billedSizeBytes"] = manifest["backup_size_bytes"] + with open(summary_path, "w") as f: + f.write(json.dumps(summary, separators=(",", ":"))) + os.chmod(summary_path, 0o600) + + +def _write_unordered_file(prefix: str, rows: list[dict]) -> dict: + """The one data file of an unordered backup holding `rows`; its manifest + entry.""" + import base64 + import gzip + import hashlib + import io + + lines = b"".join(json.dumps({"Item": r}, separators=(",", ":")).encode() + b"\n" for r in rows) + buffer = io.BytesIO() + with gzip.GzipFile(fileobj=buffer, mode="wb", mtime=0) as gz: + gz.write(lines) + compressed = buffer.getvalue() + path = os.path.join(prefix, "data", "000001.json.gz") + for name in os.listdir(os.path.join(prefix, "data")): + os.remove(os.path.join(prefix, "data", name)) + with open(path, "wb") as f: + f.write(compressed) + os.chmod(path, 0o600) + return { + "key": "data/000001.json.gz", "item_count": len(rows), + "sha256": hashlib.sha256(compressed).hexdigest(), + "md5": base64.b64encode(hashlib.md5(compressed).digest()).decode(), + "bytes": len(compressed), "uncompressed_bytes": len(lines), "key_range": None, + } + + +def _composite_table(client, name: str) -> None: + client.create_table( + TableName=name, + AttributeDefinitions=[ + {"AttributeName": "pk", "AttributeType": "S"}, + {"AttributeName": "sk", "AttributeType": "N"}, + ], + KeySchema=[ + {"AttributeName": "pk", "KeyType": "HASH"}, + {"AttributeName": "sk", "KeyType": "RANGE"}, + ], + BillingMode="PAY_PER_REQUEST", + ) + wait_for_active(client, name) + + +def _delete_backup_once_available(client, arn: str | None) -> None: + if arn: + _wait_backup_available(client, arn) + client.delete_backup(BackupArn=arn) + + +def test_a_vector_manifest_is_refused_before_any_table(dynamodb_client, unique_table_name, backup_root): + """A backup whose manifest records a vector index (the prefix of a real + backup rewritten under the root) is refused by RestoreTableFromBackup + with the vector ValidationException, and no table of the target's name + exists afterwards: the request checks the manifest against the backend + before it registers anything.""" + client = dynamodb_client + source = unique_table_name + target = f"{source}-restored" + backup_arn = None + try: + _composite_table(client, source) + client.put_item(TableName=source, Item={"pk": {"S": "a"}, "sk": {"N": "1"}, "emb": {"L": [{"N": "0.1"}]}}) + backup_arn = _create_backup(client, source, f"{source}-backup") + _wait_backup_available(client, backup_arn) + prefix = _prefix(backup_root, source, backup_arn) + manifest = _read_manifest(prefix) + manifest["source_table"]["vector_indexes"] = [{ + "IndexName": "by-emb", "Dimensions": 3, "DistanceFunction": "COSINE", + "VectorAttribute": {"AttributeName": "emb"}, + }] + _write_manifest(prefix, manifest) + with pytest.raises(ClientError) as refused: + client.restore_table_from_backup(TargetTableName=target, BackupArn=backup_arn) + err = refused.value.response["Error"] + assert err["Code"] == "ValidationException", err + assert err["Message"] == ( + f"backup {backup_arn} has 1 vector index(es); restoring a table with " + "vector indexes is not supported by this storage backend" + ), err + with pytest.raises(client.exceptions.ResourceNotFoundException): + client.describe_table(TableName=target) + finally: + _cleanup(client, target, source) + _delete_backup_once_available(client, backup_arn) + + +def test_a_sort_key_beyond_decimal128_fails_the_restore(dynamodb_client, unique_table_name, backup_root): + """A data file holding an `N` sort key of 35 significant digits (more + than Decimal128's 34, which the sort-key column is): the restore is + accepted CREATING, the job fails the file with the write path's + ValidationException text, the target is gone, and the server log carries + the Decimal128 text as the FAILED reason.""" + client = dynamodb_client + source = unique_table_name + target = f"{source}-restored" + backup_arn = None + try: + _composite_table(client, source) + for i in range(3): + client.put_item(TableName=source, Item={"pk": {"S": "a"}, "sk": {"N": str(i)}}) + backup_arn = _create_backup(client, source, f"{source}-backup") + _wait_backup_available(client, backup_arn) + prefix = _prefix(backup_root, source, backup_arn) + manifest = _read_manifest(prefix) + long_key = "12345678901234567890123456789012345" + rows = [{"pk": {"S": "a"}, "sk": {"N": str(i)}} for i in range(3)] + rows.append({"pk": {"S": "a"}, "sk": {"N": long_key}}) + entry = _write_unordered_file(prefix, rows) + manifest["item_order"] = "unordered" + manifest["data_files"] = [entry] + manifest["backup_size_bytes"] = entry["bytes"] + manifest["source_table"]["item_count"] = len(rows) + _write_manifest(prefix, manifest) + resp = client.restore_table_from_backup(TargetTableName=target, BackupArn=backup_arn)["TableDescription"] + assert resp["TableStatus"] == "CREATING", resp + deadline = time.monotonic() + 120 + while True: + try: + status = client.describe_table(TableName=target)["Table"]["TableStatus"] + except client.exceptions.ResourceNotFoundException: + break + assert status in ("CREATING", "DELETING"), f"{target} reached {status}" + assert time.monotonic() < deadline, f"{target} still exists after 120 s" + time.sleep(0.2) + deadline = time.monotonic() + 15 + while True: + with open(os.environ["EXTENDDB_TEST_SERVER_LOG"], errors="replace") as f: + failed = [line for line in f if ": FAILED (" in line and f" into {target} " in line] + if failed or time.monotonic() > deadline: + break + time.sleep(0.25) + assert len(failed) == 1, failed + assert "Decimal128" in failed[0] and long_key in failed[0], failed[0] + assert "the target was removed" in failed[0], failed[0] + finally: + _cleanup(client, target, source) + _delete_backup_once_available(client, backup_arn) + + +def test_a_non_key_number_of_38_digits_restores_intact(dynamodb_client, unique_table_name, backup_root): + """A non-key `N` of 38 significant digits (the service's precision, past + Decimal128's) is stored as the item's text and comes back from a restore + equal to what the source serves.""" + client = dynamodb_client + source = unique_table_name + target = f"{source}-restored" + backup_arn = None + big = "12345678901234567890123456789012345678" + try: + _composite_table(client, source) + client.put_item(TableName=source, Item={"pk": {"S": "a"}, "sk": {"N": "1"}, "big": {"N": big}}) + key = {"pk": {"S": "a"}, "sk": {"N": "1"}} + served = client.get_item(TableName=source, Key=key, ConsistentRead=True)["Item"] + assert served["big"] == {"N": big}, served + backup_arn = _create_backup(client, source, f"{source}-backup") + _wait_backup_available(client, backup_arn) + client.restore_table_from_backup(TargetTableName=target, BackupArn=backup_arn) + wait_for_active(client, target) + restored = client.get_item(TableName=target, Key=key, ConsistentRead=True)["Item"] + assert restored == served, (restored, served) + finally: + _cleanup(client, target, source) + _delete_backup_once_available(client, backup_arn)