storage-cassandra: cdrs-tokio drops its only connection under sustained load and never reconnects #403

What happened

Running the full Cassandra pytest suite (devtools/run-cassandra-tests -- --pytest --comprehensive --parallel, 2 workers) against a single-node cassandra:4.1 with a 60,000-item backup-restore fixture in the mix, the server lost its connection to Cassandra about 25 minutes in and never recovered. Every request after that failed with No nodes available in query plan! (surfaced to clients as InternalServerError, including Internal error during authentication), until the process was restarted. 90 tests failed and 314 errored in that run.

Server log at the moment it happened:

ERROR extenddb_engine::create_table: storage internal error internal_error=Internal error: Failed to drop data table: General error: Connection closed while waiting for response!
ERROR cdrs_tokio::transport: Transport error! error=General error: Unmatched stream id: 7914
WARN  cdrs_tokio::cluster::connection_pool: Connection down. Starting reconnection. broadcast_rpc_address=Some(240.10.0.6:7000)
ERROR extenddb_storage_cassandra::cassandra_util: account_exists query failed: General error: No nodes available in query plan!
... (every request from here on)

Reading of the driver (cdrs-tokio 9.0.2, src/transport.rs)

ResponseHandlerMap::next_stream_id hands out ids 1..=i16::MAX and wraps to 1 without checking that the id's previous request has been answered. A slow request (here, a DROP TABLE under load) can still be outstanding when its id is reused; when the late response arrives the handler is already gone, send_response returns Unmatched stream id, the ? in start_reading_normal_frames ends the read loop, and the connection is closed. The reconnection then targets broadcast_rpc_address (240.10.0.6:7000 in this container), not the contact point the server was configured with, and never succeeds, so the pool stays empty.

This is from reading the code and the log, not a reproduction in isolation.

Why it matters

A busy production server with one Cassandra node (or one connection per node) can hit this after ~32k requests with any one slow statement in flight, and then serves nothing until restarted.

Options

• Open more than one connection per node (cdrs-tokio supports a per-node pool size), which divides the id space.
• Make next_stream_id skip ids that still have a handler.
• Reconnect to the configured contact point rather than broadcast_rpc_address, or make reconnection retry.
• Upstream fix or driver change.

Workaround in #384

devtools/run-cassandra-tests now sets EXTENDDB_TEST_BACKUP_ITEMS=10000 unless it is already set, which keeps the test suite under the threshold. Nothing changes for production.