Skip to content

storage-cassandra: migrations fail on a cluster of more than one node, because a statement can reach a node before the schema change it depends on #637

Open

Description

@LeeroyHannigan

On a Cassandra cluster of two nodes, the catalog and data migrations fail in most runs. A DDL statement that depends on the one before it can go to the other node before that node has applied the earlier change:

  • Catalog V001's CREATE INDEX IF NOT EXISTS ON tables (table_id) runs right after CREATE TABLE tables and fails with Table 'tables' doesn't exist. Data V001's indexes on stream_shards and transaction_ledger fail the same way.
  • A write into a table created a moment before can fail with INCOMPATIBLE_SCHEMA from /<other node>.
  • On main, the USE right after CREATE KEYSPACE can fail with Keyspace ... does not exist.

Ten migration runs per row, two cassandra:4.1 nodes, statements round robin across both:

Code 1 connection per node 10 connections per node
main at cf321f0 9 failed 10 failed
#632 fix 10 failed 9 failed
#632 fix merged with #457 9 failed 8 failed

The first six errors of every failed run were all of these kinds. The #632 fix doesn't change this: it stops statements from running in the wrong keyspace, and these statements name the right keyspace but reach a node that doesn't have the table yet.

#457 adds wait_for_schema_agreement, which misses these failures:

  1. It runs once after each migration file, and the failing statements depend on statements earlier in the same file.
  2. schema_versions_agree sends SELECT schema_version FROM system.local and SELECT schema_version FROM system.peers as two queries. Round robin sends them to different nodes, so on two nodes it compares node A's version with node B's gossip copy of node A's version, and never reads node B's own version.

I measured the second point on two nodes with one connection each: 40 tables, each CREATE TABLE followed by an ALTER TABLE on it.

Wait between the two statements ALTER failures
None 12 of 40
#457's check, polled every 200 ms 8 of 40
system.local read on each node 0 of 40

A fix would wait for schema agreement after every DDL statement, reading schema_version from system.local on each node through a request pinned to that node. CI runs a single node, so no test covers this today.

Activity

  1. LeeroyHannigan commented on Oct 29, 2026

    @LeeroyHannigan
    CollaboratorAuthor

    Measurements from reviewing #457, which adds a schema agreement wait after each migration (migrations.rs line 229 at b4cf251).

    On a two node cluster with node 2 stopped, the wait times out on the first migration: account creation failed after 51s and a fresh server start after 37s, both with Timed out waiting for Cassandra schema agreement after a migration. main did the same two operations in 5.5s and 7.7s with the same node down. The cause is that a down node keeps its old schema_version row in system.peers, so the check cannot agree until the node is back.

    The wait also did not prevent the schema_history does not exist failure in #457's CI run 40398447509. The test added with it (migration_schema_agreement.rs) passed 20 of 20 runs with the wait removed, so it does not exercise the wait.

    Whatever fixes this issue needs to skip peers that are down, or bound the wait by the live peers only.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

v1Needed for the 1.0 release

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions