You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
storage-cassandra: concurrent account creation puts one account's tables in another's keyspace #408
Two accounts created at about the same time (two CreateAccount calls, or any two first requests that call ensure_account_keyspace) can leave one account's keyspace missing tables for good. In the PR #384 Cassandra CI run 38886321929, account 066309124626 ended up without gsi_pending, the five ttl_* tables and backup_items. Its schema_history still recorded V001 and V002 as applied, so nothing ever retried. Every DeleteTable in that account then returned HTTP 500 (clear_ttl_entries selects from the missing ttl_expiration_buckets), after the data table had already been dropped, leaving a half-deleted table. This is on main.
Why
crates/storage-cassandra/src/migrations.rsrun_data_migrations runs session.query("USE {keyspace}") and then the V001/V002 data migrations, whose DDL is unqualified (CREATE TABLE IF NOT EXISTS gsi_pending ...). record_migration is keyspace-qualified.
cdrs-tokio keeps one current keyspace per Session. A SetKeyspace result is stored in the shared KeyspaceHolder, and the pool re-sends USE <ks> to every connection. The whole process shares one Session.
ensure_account_keyspace (catalog_store.rs, called from CreateAccount) takes no lock. Two concurrent calls interleave their USE, so the rest of account A's unqualified DDL runs in account B's keyspace, while A still records the migration as applied in its own schema_history.
Evidence from the CI log above: account 066309124626's last DDL is index_propagation_holds at 10:29:56.174. At 10:29:56.212 account 807809417490 (created concurrently by test_stream_table_reuse.py) starts migrating, and its first table after schema_history is gsi_pending (.308), before its own stream_shards (.353). That is A's next statement. A single sequential V001 run cannot produce that order. On main's green run the two test modules ran one after the other on the same xdist worker, so they did not overlap.
Fix
Keyspace-qualify the data-migration DDL and drop the USE (or, minimally, a process-wide mutex around run_*_migrations; the USE state is per Session).
Repair accounts already broken: their ledger says V001/V002 are applied while tables are missing. The statements are all CREATE ... IF NOT EXISTS, so re-running them regardless of schema_history is safe.
A test that creates two accounts concurrently and checks every expected table exists in each keyspace; it should fail on main.
Activity
jcshepherd commentedon Oct 22, 2026
https://github.com/ExtendDB/extenddb/pull/453/changes