Skip to content

storage-cassandra: concurrent account creation puts one account's tables in another's keyspace #408

Open

Description

@robinnsc

Impact

Two accounts created at about the same time (two CreateAccount calls, or any two first requests that call ensure_account_keyspace) can leave one account's keyspace missing tables for good. In the PR #384 Cassandra CI run 38886321929, account 066309124626 ended up without gsi_pending, the five ttl_* tables and backup_items. Its schema_history still recorded V001 and V002 as applied, so nothing ever retried. Every DeleteTable in that account then returned HTTP 500 (clear_ttl_entries selects from the missing ttl_expiration_buckets), after the data table had already been dropped, leaving a half-deleted table. This is on main.

Why

  • crates/storage-cassandra/src/migrations.rs run_data_migrations runs session.query("USE {keyspace}") and then the V001/V002 data migrations, whose DDL is unqualified (CREATE TABLE IF NOT EXISTS gsi_pending ...). record_migration is keyspace-qualified.
  • cdrs-tokio keeps one current keyspace per Session. A SetKeyspace result is stored in the shared KeyspaceHolder, and the pool re-sends USE <ks> to every connection. The whole process shares one Session.
  • ensure_account_keyspace (catalog_store.rs, called from CreateAccount) takes no lock. Two concurrent calls interleave their USE, so the rest of account A's unqualified DDL runs in account B's keyspace, while A still records the migration as applied in its own schema_history.

Evidence from the CI log above: account 066309124626's last DDL is index_propagation_holds at 10:29:56.174. At 10:29:56.212 account 807809417490 (created concurrently by test_stream_table_reuse.py) starts migrating, and its first table after schema_history is gsi_pending (.308), before its own stream_shards (.353). That is A's next statement. A single sequential V001 run cannot produce that order. On main's green run the two test modules ran one after the other on the same xdist worker, so they did not overlap.

Fix

  • Keyspace-qualify the data-migration DDL and drop the USE (or, minimally, a process-wide mutex around run_*_migrations; the USE state is per Session).
  • Repair accounts already broken: their ledger says V001/V002 are applied while tables are missing. The statements are all CREATE ... IF NOT EXISTS, so re-running them regardless of schema_history is safe.
  • A test that creates two accounts concurrently and checks every expected table exists in each keyspace; it should fail on main.

Activity

  1. self-assigned this
    on Oct 22, 2026
  2. jcshepherd commented on Oct 22, 2026

    @jcshepherd
    Collaborator
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions