You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
storage-cassandra: migrations fail on a cluster of more than one node, because a statement can reach a node before the schema change it depends on #637
On a Cassandra cluster of two nodes, the catalog and data migrations fail in most runs. A DDL statement that depends on the one before it can go to the other node before that node has applied the earlier change:
Catalog V001's CREATE INDEX IF NOT EXISTS ON tables (table_id) runs right after CREATE TABLE tables and fails with Table 'tables' doesn't exist. Data V001's indexes on stream_shards and transaction_ledger fail the same way.
A write into a table created a moment before can fail with INCOMPATIBLE_SCHEMA from /<other node>.
On main, the USE right after CREATE KEYSPACE can fail with Keyspace ... does not exist.
Ten migration runs per row, two cassandra:4.1 nodes, statements round robin across both:
The first six errors of every failed run were all of these kinds. The #632 fix doesn't change this: it stops statements from running in the wrong keyspace, and these statements name the right keyspace but reach a node that doesn't have the table yet.
#457 adds wait_for_schema_agreement, which misses these failures:
It runs once after each migration file, and the failing statements depend on statements earlier in the same file.
schema_versions_agree sends SELECT schema_version FROM system.local and SELECT schema_version FROM system.peers as two queries. Round robin sends them to different nodes, so on two nodes it compares node A's version with node B's gossip copy of node A's version, and never reads node B's own version.
I measured the second point on two nodes with one connection each: 40 tables, each CREATE TABLE followed by an ALTER TABLE on it.
A fix would wait for schema agreement after every DDL statement, reading schema_version from system.local on each node through a request pinned to that node. CI runs a single node, so no test covers this today.
Measurements from reviewing #457, which adds a schema agreement wait after each migration (migrations.rs line 229 at b4cf251).
On a two node cluster with node 2 stopped, the wait times out on the first migration: account creation failed after 51s and a fresh server start after 37s, both with Timed out waiting for Cassandra schema agreement after a migration. main did the same two operations in 5.5s and 7.7s with the same node down. The cause is that a down node keeps its old schema_version row in system.peers, so the check cannot agree until the node is back.
The wait also did not prevent the schema_history does not exist failure in #457's CI run 40398447509. The test added with it (migration_schema_agreement.rs) passed 20 of 20 runs with the wait removed, so it does not exercise the wait.
Whatever fixes this issue needs to skip peers that are down, or bound the wait by the live peers only.
Activity
LeeroyHannigan commentedon Oct 29, 2026
Measurements from reviewing #457, which adds a schema agreement wait after each migration (
migrations.rsline 229 at b4cf251).On a two node cluster with node 2 stopped, the wait times out on the first migration: account creation failed after 51s and a fresh server start after 37s, both with
Timed out waiting for Cassandra schema agreement after a migration. main did the same two operations in 5.5s and 7.7s with the same node down. The cause is that a down node keeps its oldschema_versionrow insystem.peers, so the check cannot agree until the node is back.The wait also did not prevent the
schema_history does not existfailure in #457's CI run 40398447509. The test added with it (migration_schema_agreement.rs) passed 20 of 20 runs with the wait removed, so it does not exercise the wait.Whatever fixes this issue needs to skip peers that are down, or bound the wait by the live peers only.