From 56c59a334c835d85db73a62f26dbf111056486bb Mon Sep 17 00:00:00 2001 From: Lee Hannigan <29015211+LeeroyHannigan@users.noreply.github.com> Date: Sun, 1 Nov 2026 18:31:57 +0000 Subject: [PATCH] docs: back up and restore SQLite and PostgreSQL with their own tools The admin guide and the security model showed two pg_dump commands with a data database name that init does not create, and no page said how to copy a SQLite deployment or whether two pg_dump runs agree. Add a Database Backup and Restore section to the admin guide. Each procedure that needs the server stopped runs as one block under set -e and ends before it touches a file or a database unless extenddb status reports the server stopped, because extenddb stop stops nothing for a server started with serve --foreground. For SQLite the section gives VACUUM INTO while the server runs, a copy of the database file and whichever of its -wal and -shm files exist with the server stopped, and a restore that checks the copy with PRAGMA integrity_check before it moves the old files aside. For PostgreSQL it states that two pg_dump runs disagree when a table is created or deleted between them, and gives a dump pair taken with the servers stopped, pg_basebackup with its prerequisites, and the restore of each. The dump pair restore renames the databases it replaces and removes them only after the restored server passes a check. It prints the catalog's data_database_name and data_database_connection_string rows and sets both to their full new values when they name another database or address, because the server opens the database the connection string names. Every restore ends with a read of every table. The section also says what a copy holds: the catalog stores the encryption key next to the secrets it encrypts. The upgrade manual, security model and deployment guide now point at the section. The upgrade manual sends its plain-text dumps to the Rollback Procedure, which restores them with psql -f. --- docs/manuals/05-admin-guide.md | 250 ++++++++++++++++++++++++++-- docs/manuals/07-upgrade-manual.md | 2 + docs/manuals/10-security-model.md | 7 +- docs/manuals/11-deployment-guide.md | 2 +- 4 files changed, 243 insertions(+), 18 deletions(-) diff --git a/docs/manuals/05-admin-guide.md b/docs/manuals/05-admin-guide.md index a0cf56e7e..e079e0550 100755 --- a/docs/manuals/05-admin-guide.md +++ b/docs/manuals/05-admin-guide.md @@ -876,6 +876,244 @@ curl --cacert ~/.extenddb/tls/cert.pem https://127.0.0.1:18443/health # {"status":"healthy"} ``` +## Database Backup and Restore + +`CreateBackup` stores each table backup in the deployment's own database, in the catalog's `backups` and `backup_items` tables, so losing that database loses those backups as well. To keep a copy of the whole deployment, use the database's own tools as this section describes. The copy holds what ExtendDB keeps in its databases: the accounts, IAM users, roles and policies, access keys, tables, items, stream records and table backups. + +The copies taken while the server ran were measured with a client writing to two tables, one of them with a GSI, during the copy. The copies taken with the server stopped were measured after the client's last acknowledged write. A server started on each restored copy returned all writes acknowledged before the copy began. Its GSI queries returned the same items as the base table, and the access keys created before the copy authenticated. + +### What a copy holds + +The catalog holds the AES-256-GCM key that encrypts the stored secret access keys, in the `encryption_key` row of its `settings` table, next to the encrypted secrets in `access_keys` and `iam_sessions`. Anyone who can read a copy of the catalog can decrypt the secret access keys in it. The catalog also holds the bcrypt hashes of the admin and console passwords. On SQLite the catalog is the database file. On PostgreSQL it is the catalog database, and its `data_database_connection_string` row also holds the application role's password. The PostgreSQL data database holds the items, index entries and stream records. It holds no keys. + +Keep copies on encrypted storage, readable by their owner only. Under umask 022, `VACUUM INTO` and the `pg_dump` redirections below create files with mode 0644, which every local user can read. Under `umask 077` they create files with mode 0600. + +### Stop the server first + +`extenddb stop` finds the server through the PID file the daemon writes. It stops nothing and exits 1 for a server started with `serve --foreground`, which writes no PID file, and when its `--config` path does not name the server's config file. Stop the server through whatever runs it: `sudo systemctl stop extenddb` for the Deployment Guide's systemd unit, the container runtime for a container, otherwise `extenddb stop --config /path/to/extenddb.toml`. Start it again the same way when the procedure is done. + +Each procedure below that needs the server stopped is one block that runs in a subshell under `set -e`, so a failed command ends the block and leaves your shell open. Before it touches a file or a database, the block runs `extenddb status --config /path/to/extenddb.toml | grep -q 'is not running'`, which ends the block unless `extenddb status` reports that nothing listens on the configured port. Give the config file as an absolute path: a relative path names another file when the block runs from another directory. `extenddb status` connects to the port on 127.0.0.1, so for a server whose `bind_addr` is another address, also confirm the stop through whatever runs the server. + +### SQLite + +The database is the file named by `[storage.sqlite] path`, plus the `-wal` and `-shm` files SQLite keeps beside it. `extenddb serve` sets all three to mode 0600 when it opens them, so run the commands below as the user that owns them. The commands keep each copy in a directory of its own under `/path/to/backups`, in a file named `extenddb.sqlite` like the database. + +#### Copy while the server runs + +```bash +( + set -e + umask 077 + d=/path/to/backups/$(date +%Y%m%d%H%M%S) + mkdir -p "$d" + sqlite3 -cmd ".timeout 5000" /path/to/extenddb.sqlite "VACUUM INTO '$d/extenddb.sqlite'" +) +``` + +`VACUUM INTO` reads the database in one read transaction and writes a new database file with no `-wal` or `-shm` file. The copy holds the writes committed before its read transaction began, a few milliseconds after the command starts: one copy held 3 writes acknowledged after the command started, the last of them 21 ms after. The server keeps serving reads and writes while it runs. On a 284 MB database the copy took 2.0 s, with a client making about 100 writes a second; 206 of those writes were acknowledged during the copy. `.timeout 5000` sets the shell's busy timeout (its help reads `Try opening locked tables for MS milliseconds`), so the command waits for a lock instead of failing at once. + +The commands here were run with sqlite3 3.46.0. SQLite's release history dates `VACUUM INTO` to version 3.27.0. The sqlite3 3.7.17 shell rejects it (`Error: near "INTO": syntax error`) and cannot read the database's schema (`malformed database schema (idx_login_attempts_source_ip_time)`). + +Do not use the sqlite3 `.backup` command while the server takes writes. The shell copies 100 pages at a time through SQLite's online backup API, which the SQLite documentation says starts again from the first page when another connection writes to the database between two steps. With the client making about 100 writes a second to the 284 MB database, `.backup` had not finished after 300 s. With no client writing it took 5.0 s. + +#### Copy with the server stopped + +A non-empty `-wal` file can hold committed writes that are not yet in the database file, and one database file copied without it did not open (`database disk image is malformed`). `extenddb stop` does not always leave the `-wal` file: after seven of eight measured stops the `-wal` and `-shm` files stayed, with up to 4.2 MB in the `-wal` file, and after one they were gone. The block copies each of the three files that exists. Stop the server (see [Stop the server first](#stop-the-server-first)), then run: + +```bash +( + set -e + umask 077 + extenddb status --config /path/to/extenddb.toml | grep -q 'is not running' + d=/path/to/backups/$(date +%Y%m%d%H%M%S) + mkdir -p "$d" + cd /path/to + for f in extenddb.sqlite extenddb.sqlite-wal extenddb.sqlite-shm; do + if [ -e "$f" ]; then cp -p "$f" "$d/"; fi + done +) +``` + +#### Restore a SQLite copy + +SQLite applies a `-wal` file to the database file of the same name, so a copy put beside the old `-wal` file gets pages of the old database written over it. With the old `-wal` and `-shm` files left in place, one server answered `ListTables`, returned `InternalServerError` to `Scan` on both tables, and logged `database disk image is malformed`. Another could not open the database and exited 34 s after it started. + +The block makes a work directory, `restore-