Repository navigation
[Bug] MongoDB restore can remain CREATING after client disconnect during base copy #334
Description
Describe the bug
If the client disconnects while RestoreTableFromBackup is copying the
base data with MongoDB $out, the restore can remain permanently stuck in
CREATING.
The restore request creates the target table and its secondary-index metadata
before copying the base data. The restore_backfill_pending marker is written
only after the copy completes.
To Reproduce
- Start ExtendDB with MongoDB and a replica set.
- Create a table containing a Global Secondary Index or Local Secondary Index.
- Create a backup containing data and index entries.
- Start
RestoreTableFromBackupfor that backup. - Disconnect or cancel the client while the
$outbase-data copy is running. - Describe the restored table and inspect its index status.
Expected behavior
An interrupted restore should either be safely retryable or be recovered by
the background worker. It should not remain permanently stuck in CREATING.
Actual behavior
The restore may exit before setting restore_backfill_pending. The table
remains in CREATING, and its restore-mode index jobs are not processed because
the worker waits for that marker. The restore therefore has no completion path.
Environment
- ExtendDB version: 0.1.10
- Operating system: macOS 26.6.2, Apple Silicon
- MongoDB version: 7.0
- Rust version: 1.97.1
- Client SDK/driver and version: aws-sdk-dynamodb 1.113.0 for Rust
- Deployment method: source
- MongoDB deployment: single-node replica set
- Number of ExtendDB instances: 1
Logs / stack trace
No error is necessarily reported after the client disconnects.
Additional context
The current restore flow intentionally creates secondary-index metadata in
CREATING before the base $out copy. The worker now correctly avoids
processing those restore indexes until restore_backfill_pending is set.
This follow-up issue covers recovery when the restore request is interrupted
before that marker is written. Possible solutions include making the restore
state resumable, adding recovery for abandoned restore jobs, or cleaning up
the incomplete target so the restore can be retried safely.
Checklist
- I have searched existing issues for duplicates
- I have included the ExtendDB version and environment details
- I can reproduce this with the latest release
Activity
LeeroyHannigan commentedon Oct 24, 2026
Resolved by #384 (merged 2026-10-23). On
main744ea29 the MongoDB restore no longer runs inside the request future:RestoreTableFromBackupinserts the CREATING target and runs the$outcopy in a task detached from the request (crates/storage-mongodb/src/backup_engine.rs, module doc and the restore path), so a client disconnect during the copy does not cancel it. The target carriesrestore_heartbeat_at, refreshed every 15 s while the copy runs;sweep_stale_backup_claims(run at startup and periodically by the backfill worker) removes a CREATING target whose heartbeat is older than the stale threshold together with its collections (step 2, "Abandoned restore targets"), so a restore killed mid-copy is cleaned up rather than left CREATING. Once the copy completes andrestore_backfill_pendingis set, the backfill worker owns the target and finishes it after a restart, as the issue asked.The scenario as written (disconnect during
$outon a table with a GSI) is covered by the detached task; the crash variant by the heartbeat sweep.