storage-cassandra: asynchronous GSI writes are never applied under serve (default 10 ms delay) #407

Impact

On a Cassandra deployment run with extenddb serve, every write to a table with a GSI goes to the asynchronous propagation queue (gsi_pending) and is never applied. GSI queries never see new, changed, or deleted items. This is on main today.

Why

• crates/server/src/serve.rs (~line 621) binds the backend's runtime hooks by value: if let Some(hooks) = runtime_hooks { ... hooks.spawn_workers(&worker_ctx).await }. The Box is dropped at the end of that block, before the server starts.
• CassandraRuntimeHooks stores the GSI workers' guard on itself (gsi_worker_guard: OnceLock<GsiWorkerGuard>). Dropping the hooks drops the guard, which sets the shutdown flag, and all four GSI workers exit within about a second of startup.
• Cassandra never seeds index_propagation_delay_ms: neither init nor the Cassandra migrations insert it. The engine starts at AtomicU64::new(10) and poll_gsi_delay falls back to unwrap_or(10). With no per-index override, every GSI therefore uses 10 ms, which takes the asynchronous path.
• CI does not see it, because devtools/run-tests sets index_propagation_delay_ms to 0 (synchronous) before the suites run.

Workaround: extenddb settings set index_propagation_delay_ms 0. This affects later writes only; already-queued rows stay unapplied until a server with live workers runs.

What a fix has to cover

Keeping the guard alive (an attempt was in #384 and was taken out of it) turns on a worker path that has never run in a served deployment. It exposed these defects in crates/storage-cassandra/src/workers.rs and gsi_queue.rs:

• gsi_next_ready_wait is a TODO returning None, so a worker woken before its row is due sleeps the full 1 s MAX_IDLE. With a 50 ms delay configured, CI measured about 1,010 ms.
• notify_workers() uses Notify::notify_waiters(), which drops a wake sent while a worker is not parked. One Notify per partition with notify_one (which stores a permit) avoids that.
• The idle loop lists all keyspaces and SELECTs every account's gsi_pending every second, on each of 4 workers: about 4 × (1 + accounts) queries per second at idle.
• A ServerError on one keyspace (for example a keyspace still being created: "Unknown CF") returns from the whole batch instead of skipping that keyspace, as is_table_not_found already does.
• enqueue_async_indexes reads last_ready_at and then writes it without an LWT, so concurrent writers to one partition can get out-of-order ready_at values, and images applied out of order leave stale GSI entries.
• The first start after the fix drains whatever backlog has been queued so far; its size and duration are unbounded.
• The docs that describe asynchronous propagation as working on Cassandra: docs/differences-from-dynamodb.md (GSI update propagation row) and docs/troubleshooting.md.

A test that drives the hooks the way serve does (spawn the workers, drop the hooks, then write with a non-zero delay) catches the guard lifetime; the crate's own tests hold the guard in a local, so they never saw it.