Fence incremental mobile backups with server revisions

README.md

@@ -47,11 +47,11 @@ The earlier public-source paired setup was checked on October 6, 2026 with isola
 
 ## Backup snapshot and repair
 
-`POST /getBackupSnapshot` reads an account's encrypted recovery records in one database snapshot and returns a SHA-256 `revision` covering encrypted content plus a retained account mutation generation, the app image, account-scoped vault images, labels, and any unavailable vault IDs. It does not replace the backup. Invalid account identifiers return `400`; unavailable storage returns a fixed `503` response.
+`POST /getBackupSnapshot` reads an account's encrypted recovery records in one database snapshot and returns a SHA-256 `revision` covering encrypted content plus a retained account mutation generation, the app image, account-scoped vault images, labels, and unavailable vault or label IDs. Missing labels and historically shared label IDs are withheld from generic recovery; shared IDs require an operator migration with ownership evidence. It does not replace the backup. Invalid account identifiers return `400`; unavailable storage returns a fixed `503` response.
 
 `POST /repairAppBackup` requires the revision just read. A stale revision returns `409` with `BACKUP_CHANGED`; the existing backup stays intact. The generation advances transactionally on backup mutations and survives Delete Backup, so returning from content A to B and back to A never reuses an earlier token. Explicit current-state replacement must use the existing app verification/resubmission flow. Legacy full replacements cannot bypass this requirement after a revisioned repair. Collaborative accounts retain independently encrypted vault copies; no other account's ciphertext is substituted for a missing copy.
 
-The source includes the guarded backup implementation used by the mobile recovery protocol. Publishing it does not deploy a backend or migrate a hosted database. Before any hosted adoption, maintainers must review transaction/replica-set support and the account-scoped unique index with staging fixtures.
+The source includes the guarded backup implementation used by the mobile recovery protocol. Publishing it does not deploy a backend or migrate a hosted database. Before any hosted adoption, maintainers must review transaction/replica-set support and the account-scoped unique index with staging fixtures. The shared-label check also needs the `AppImages` multikey index `{ labels: 1 }`: build and verify that index on existing hosted databases before enabling the new snapshot route, then check its query plan and historical shared-reference counts in staging. A source change alone does not guarantee the index exists when automatic index creation is disabled.
 
 ## Hosted integrations
 
@@ -68,6 +68,8 @@ Deployment and provider permissions are maintainer responsibilities. Do not use
 
 Run `yarn test --runInBand` (which compiles first) and `python3 scripts/check-source-secrets.py` with Gitleaks 8.30.1. The database tests use `READINESS_MONGO_URL` pointing at a disposable MongoDB. Run `yarn test:backup` with a disposable loopback replica set named `keeperQA` (default `127.0.0.1:27028`, or set `BACKUP_TEST_MONGO_URL` to its loopback address). These tests create and drop only uniquely named test databases; they never read `.env` or hosted credentials. CI runs both suites. See [CONTRIBUTING.md](CONTRIBUTING.md) and [SECURITY.md](SECURITY.md).
 
+The optional paired HTTP test uses a compatible public Keeper app checkout for its actual AES encryption/decryption exports. With a disposable loopback MongoDB replica set named `keeperQA`, run `KEEPER_APP_SOURCE=/path/to/bitcoin-keeper BACKUP_TEST_MONGO_URL=mongodb://127.0.0.1:27028 yarn test:automatic-http`. It mounts the actual local-mode allowlist and the exact `/getBackupSnapshot`, `/revisionedBackupMutation` and `/getAppImage` production handler source on loopback Express with the actual Relay service graph, Mongoose models and transactions. Local mode now permits the revisioned mutation route while hosted provider routes remain blocked. The legacy `/getAppImage` endpoint is a restore read only; `prepareRestore(appId)` seeds an actual disposable free-tier App record so the production receipt check uses its normal no-provider branch. No legacy write handler or automatic fallback is mounted. The seven-operation matrix checks canonical acknowledgements, stale CAS conflicts, an old-route 404, malformed acknowledgement (including an unchanged old revision), snapshot metadata faults, and a lost reply after real commits; encrypted readback checks the actual outcome. It reads no `.env`, uses a uniquely named disposable database, and drops only that database afterward. The supporting module also lets the matching app's real Realm/Redux/transport fixture share this HTTP server. Response fault injection does not mock CAS or cryptographic success. Hosted authentication/rate limits, provider bootstrap, native mobile acceptance and production deployment topology are separate checks; this test does not establish them.
+
 ## License
 
 Relay is licensed under the [MIT License](LICENSE). Dependencies retain their respective licenses.

package.json

@@ -10,6 +10,7 @@
     "test": "npm run compile && jest ./dist --detectOpenHandles",
     "gcp-build": "tsc -p .",
     "test:backup": "node --test tests/*.test.cjs",
+    "test:automatic-http": "node --test tests/integration/automaticBackupHttp.test.cjs",
     "check:secrets": "python3 scripts/check-source-secrets.py"
   },
   "jest": {

src/db.ts

@@ -153,6 +153,9 @@ class Database {
     nodes: [{ type: String, required: false }],
     // Once repaired, unversioned full replacements must never overwrite this image.
     backupRevisionRequired: { type: Boolean, default: false },
+    // A revisioned incremental writer has claimed this account. Older clients
+    // must not apply delayed unversioned mutations after that writer commits.
+    incrementalRevisionRequired: { type: Boolean, default: false },
     // Retained by Delete Backup, preventing stale A → B → A revisions.
     backupGeneration: { type: Number, default: 0, min: 0 },
   });
@@ -327,6 +330,8 @@ class Database {
       { unique: true },
     );
     this.articleChunkSchema.index({ ragTimestamp: -1 });
+    // Snapshot ownership checks must not scan every account for each label.
+    this.appImageSchema.index({ labels: 1 });
     this.appVaultImageSchema.index({ appId: 1, vaultId: 1 }, { unique: true });
     this.faucetQuotaSchema.index({ appId: 1, utcDate: 1 }, { unique: true });
     this.faucetQuotaSchema.index(

src/localDevelopment.ts

@@ -27,7 +27,7 @@ const localDatabaseRoutes = new Set([
   "/getvaultmetadata", "/getsigneridinfo", "/migratexfps", "/modifylabels",
   "/getmessages", "/getsubscriptiondetails", "/updatecontactskey",
   "/createremotekey", "/getremotekey", "/backupallsignersandvaults",
-  "/getbackupsnapshot", "/repairappbackup",
+  "/getbackupsnapshot", "/repairappbackup", "/revisionedbackupmutation",
   "/deletebackup", "/updatecollaborativechannel", "/fetchcollaborativechannel",
   "/gethardwarereferrallinks", "/getactivecampaign", "/getadvisors",
 ]);

src/routes/routes.ts

@@ -77,6 +77,15 @@ const memoryStorage = multer.memoryStorage();
 const upload = multer({ memoryStorage });
 const screenshotUpload = multer({ storage: multer.memoryStorage() });
 
+function sendBackupMutationError(res: Response, error: unknown) {
+  const conflict = error instanceof BackupConflict;
+  const upgrade = error instanceof BackupUpgradeRequired;
+  return res.status(conflict || upgrade ? 409 : error instanceof InvalidBackupRequest ? 400 : 503).json({
+    updated: false,
+    error: conflict ? "BACKUP_CHANGED" : upgrade ? "BACKUP_UPGRADE_REQUIRED" : "BACKUP_NOT_UPDATED",
+  });
+}
+
 export default class Routes {
   public app: Express;
   constructor(app: Express) {
@@ -348,13 +357,11 @@ export default class Routes {
         const updated = await bhr.migrateXfp(
           req.body.appId,
           req.body.signerChanges,
+          req.body.expectedRevision,
         );
         res.status(200).json({ updated });
       } catch (err) {
-        res.status(400).json({
-          updated: false,
-          err: err.message,
-        });
+        sendBackupMutationError(res, err);
       }
     });
 
@@ -519,10 +526,11 @@ export default class Routes {
           hasField("signersObject") ? req.body.signersObject : req.body.signersObjects,
           req.body.nodes,
           req.body.replaceNodes === true,
+          req.body.expectedRevision,
         );
         res.status(200).json(result);
       } catch (err) {
-        res.status(400).json(err);
+        sendBackupMutationError(res, err);
       }
     });
 
@@ -535,10 +543,11 @@ export default class Routes {
           req.body.appId,
           req.body.walletIds,
           req.body.signers,
+          req.body.expectedRevision,
         );
         res.status(200).json(result);
       } catch (err) {
-        res.status(400).json(err);
+        sendBackupMutationError(res, err);
       }
     });
 
@@ -547,10 +556,10 @@ export default class Routes {
         return res.status(400).json({ err: "Input param missing - app's id" });
       }
       try {
-        const result = await bhr.deleteVaults(req.body.appId, req.body.vaults);
+        const result = await bhr.deleteVaults(req.body.appId, req.body.vaults, req.body.expectedRevision);
         res.status(200).json(result);
       } catch (err) {
-        res.status(400).json(err);
+        sendBackupMutationError(res, err);
       }
     });
 
@@ -566,6 +575,7 @@ export default class Routes {
             req.body.appId ?? req.body.appID,
             req.body.isArchived,
             req.body.signersData,
+            req.body.expectedRevision,
           );
           if (updated) {
             return res.status(200).json({
@@ -588,13 +598,11 @@ export default class Routes {
           req.body.subscription,
           req.body.archiveVaultId,
           req.body.isArchived,
+          req.body.expectedRevision,
         );
         res.status(200).json(result);
-      } catch {
-        res.status(400).json({
-          updated: false,
-          error: "Vault backup update failed",
-        });
+      } catch (error) {
+        sendBackupMutationError(res, error);
       }
     });
 
@@ -765,13 +773,11 @@ export default class Routes {
           req.body.appId,
           req.body?.addLabels,
           req.body?.deleteLabels,
+          req.body.expectedRevision,
         );
         res.status(200).json(result);
       } catch (err) {
-        res
-          .status(400)
-          .json({ updated: false, err: "failed to update the labels" });
-        console.log(err);
+        sendBackupMutationError(res, err);
       }
     });
 
@@ -1066,9 +1072,75 @@ export default class Routes {
 
     router.post("/getBackupSnapshot", async (req, res) => {
       if (typeof req.body.appId !== "string" || !req.body.appId) return res.status(400).json({ error: "Invalid backup account" });
-      try { return res.status(200).json(await getBackupSnapshot(req.body.appId)); }
+      try { return res.status(200).json({
+        ...await getBackupSnapshot(req.body.appId), revisionedMutations:true,
+      }); }
       catch { return res.status(503).json({ error: "Backup snapshot unavailable" }); }
     });
+    // A distinct route prevents a revisioned client from writing through an old
+    // Relay instance that would silently ignore expectedRevision. Deploy this
+    // route to every instance before enabling it in mobile clients.
+    router.post("/revisionedBackupMutation", async (req, res) => {
+      const body = req.body || {};
+      if (typeof body.appId !== "string" || !body.appId ||
+          typeof body.expectedRevision !== "string" || !/^[a-f0-9]{64}$/.test(body.expectedRevision))
+        return res.status(400).json({updated:false,error:"BACKUP_NOT_UPDATED"});
+      if (body.appID !== undefined && body.appID !== body.appId)
+        return res.status(400).json({updated:false,error:"BACKUP_NOT_UPDATED"});
+      try {
+        let result: any;
+        switch (body.operation) {
+          case "updateAppImage":
+            if (("walletsObject" in body && "walletObject" in body) ||
+                ("signersObject" in body && "signersObjects" in body))
+              throw new InvalidBackupRequest("Conflicting backup record fields");
+            result = await bhr.updateAppImage(body.appId, body.publicId,
+              "walletsObject" in body ? body.walletsObject : body.walletObject,
+              body.subscription, body.version,
+              "signersObject" in body ? body.signersObject : body.signersObjects,
+              body.nodes, body.replaceNodes === true, body.expectedRevision);
+            break;
+          case "migrateXfps":
+            result = await bhr.migrateXfp(body.appId, body.signerChanges, body.expectedRevision);
+            break;
+          case "deleteAppImageEntity":
+            result = await bhr.deleteAppImageEntity(body.appId, body.walletIds,
+              body.signers, body.expectedRevision);
+            break;
+          case "deleteVaults":
+            result = await bhr.deleteVaults(body.appId, body.vaults, body.expectedRevision);
+            break;
+          case "updateVaultImage":
+            if (typeof body.vaultId !== "string" || !body.vaultId)
+              throw new InvalidBackupRequest("Invalid vault ID");
+            result = body.isUpdate
+              ? await bhr.updateVault(body.vaultId, body.vault, body.appId,
+                  body.isArchived, body.signersData, body.expectedRevision)
+              : await bhr.addVaultImage(body.appId, body.vaultShellId, body.vaultId,
+                  body.scheme, body.vault, body.signersData, body.subscription,
+                  body.archiveVaultId, body.isArchived, body.expectedRevision);
+            break;
+          case "modifyLabels":
+            result = await bhr.modifyLabels(body.appId, body.addLabels,
+              body.deleteLabels, body.expectedRevision);
+            break;
+          case "deleteBackup":
+            result = await bhr.deleteBackup(body.appId, body.expectedRevision);
+            break;
+          case "repairAppBackup":
+            result = await repairAppBackup(body, true);
+            break;
+          default:
+            throw new InvalidBackupRequest("Unknown backup mutation");
+        }
+        if (result === false || result?.updated === false ||
+            typeof result?.newRevision !== "string" || !/^[a-f0-9]{64}$/.test(result.newRevision))
+          return res.status(503).json({updated:false,error:"BACKUP_NOT_UPDATED"});
+        return res.status(200).json({updated:true,preconditionApplied:true,newRevision:result.newRevision});
+      } catch (error) {
+        return sendBackupMutationError(res, error);
+      }
+    });
     router.post("/repairAppBackup", async (req, res) => {
       try { return res.status(200).json(await repairAppBackup(req.body)); }
       catch (error) {
@@ -1098,11 +1170,10 @@ export default class Routes {
         return res.status(400).json({ err: "Input param missing - app's id" });
       }
       try {
-        const result = await bhr.deleteBackup(req.body.appId);
+        const result = await bhr.deleteBackup(req.body.appId, req.body.expectedRevision);
         res.status(200).json(result);
       } catch (err) {
-        console.log("🚀 ~deleteBackup err:", err);
-        res.status(400).json(err);
+        sendBackupMutationError(res, err);
       }
     });