[LeeroyHannigan]

Summary

#476 reports that, on the PostgreSQL backend, another table's GSI build left a table's 21 queued index rows unapplied for 38 s, and names two candidate causes: queue partitions shared across tables, and connection pool occupancy. Reproducing on main (744ea291) with pg_stat_activity and per-table gsi_pending counts sampled every 2 s settles it.

The issue's own shape does not stall on main: UpdateTable builds the GSI inside the request and enqueues nothing, and the build holds one of the pool's twenty connections, so the 21 rows apply 1.96 s after the last write with the build running (2.14 s with nothing else running). The issue's measurement was taken on the #401 head, whose online build does enqueue. The shape that stalls on main is a backlog: with 50,000 BatchWriteItem puts to table A queued, 21 puts to table B apply 9 to 11 s after the last write. Each worker claims ORDER BY id LIMIT 1 over its whole partition and partitions are shared across tables, so every row A enqueued earlier is applied before B's first row; B's latency is A's backlog divided by the drain rate (about 1,500 rows/s, so 50,000 rows is about 34 s, the magnitude the issue measured). The samples show zero ungranted locks, twenty idle connections and no batch error, and everything applies when the other table's work ends, which is the issue's picture.

Change

Each batch now lists the tables with a ready row in the worker's partition and claims each table's oldest ready row in turn (gsi_rotation.rs, TableRotation). Within a table the claims stay in id order, so per-key FIFO holds (a key's rows are in one table and one partition). The worker carries the table it served last from batch to batch, so more than BATCH_SIZE (100) tables with ready rows in one partition are all served once per pass instead of the tail waiting for the head to empty. The partition hash is unchanged, so no drain or operator step follows from the claim order.

The listing returns only tables whose lowest-id row is ready. The enqueue floors ready_at at the partition's current maximum, so within a partition ready_at is non-decreasing with id and a table's first row carries its earliest ready_at; the one exception (two writers reading the floor before either commits) delays a table's listing by at most one jitter with one delay in force, or by the difference between the two rows' delays if an operator has set a per-index propagation_delay_ms above the default (nothing writes that column today). Without this filter a partition holding N tables inside their delay cost N claim transactions per wakeup.

Data migration 005_gsi_pending_table_claim.sql adds idx_gsi_pending_table_claim on gsi_pending (worker_partition, table_id, id). The per-table claim walks it to the first ready row (0.02 to 0.04 ms whatever the backlog), and the listing is a recursive probe over it, one index entry per table with rows. Without the index the per-table claim sorted the partition's ready rows (drain fell from about 1,600 to about 240 rows/s in the first measurement). The hold check is bound to the claimed table (NOT EXISTS ... WHERE h.table_id = $2), evaluated once per claim in the same snapshot.

Operator step

Stop the servers, extenddb migrate --config extenddb.toml --pg-user <admin> --yes, start the servers. The catalog version does not change (005 is a data migration, recorded in the data database's schema_history, as #307's 004 was); the preview lists the file under data migrations. migrate builds the index with the servers stopped at about 1.7 s per million queued rows (plain CREATE INDEX, which takes SHARE on gsi_pending for the build; 3.45 s at 2,000,000 rows). A server started without the index serves and propagates every row and logs one warning at startup naming the two slower plans: each claim for a table behind another table's backlog reads past that backlog (3.6 ms at 40,000 queued rows), and the per-batch listing scans the queue table (33 ms at 40,000 rows, once per 100 claims). The upgrade manual carries the entry.

Measurements

Single pgvector/pgvector:pg16 container on one host, index_propagation_delay_ms = 2000, release binaries, from the last write on table B to its rows applied:

shape	main	this branch
nothing else running	2.13 to 2.14 s	1.83 to 2.14 s
A's 50,000-row backlog queued	9.13 to 10.97 s	2.46 to 3.11 s
A and C each with a 25,000-row backlog	not measured on main	within the delay plus one wakeup
the issue's UpdateTable build shape	1.96 s (no stall)	2.32 s; build time 8.9 s to 9.8 s

Per-batch cost with tables inside their delay (one listing statement, no claims): 10 tables 0.49 ms, 100 tables 1.26 ms, 500 tables 4.5 ms; main's one claim is 0.3 to 0.4 ms. Before the ready-only listing the same shapes cost 4.8, 29.7 and 164 ms (one claim transaction per idle table).

Drain rate, isolated 36,000 to 40,000 row backlogs, ten runs alternating binaries in the same hour: 770 to 2,000 rows/s with the change, 938 to 1,902 without; the same within run-to-run spread (the apply is one transaction per row and WAL-bound; WALWrite and WALSync waits in every sample). The stable quantity is the claim plan, 0.02 to 0.04 ms on both. Every index count equalled its table in every run, in two-, three- and fifty-table shapes.

Tests

• tests/gsi_queue_fairness.rs (2): a 20,000-row backlog for table A (whose table_id sorts first) and eight writes to table B; B's rows apply while more than three quarters of A's backlog is still queued (observed 19,579 to 19,612 of 20,000; 0 under the old claim); and a held table's rows stay queued with zero index rows until the vector_index_holds row is removed, then apply.
• tests/gsi_queue_many_tables.rs (1): 104 tables with rows in one partition, 100 rows for each of the hundred sorting first and 5 for each of the four sorting last; the tail tables' rows apply while more than three quarters of the head rows are still queued (observed 9,494 of 10,000).
• tests/data_migrations.rs (1): a data database one migration behind reports 005 pending, applies it, and pg_indexes has idx_gsi_pending_table_claim; claim_index_present is false before and true after.
• gsi_rotation unit tests (9): ring order, cursor placement and wrap, a cursor naming a table no longer listed, six tables in batches of four over three batches.
• The registration guard now covers DATA_MIGRATIONS as well as the catalog list.
• Negative controls on the committed tree: main's claim written over the fixed file, the cursor ignored, served() a no-op, the hold predicate removed, 005 emptied, a duplicated include_str!: each fails the test written for it.
• The three new test binaries are in the PostgreSQL storage-level CI list (.github/workflows/integration.yml).
• Crate suite with PostgreSQL: 169 passed (62 unit, 107 across 12 integration files). cargo fmt, clippy 1.97.0 with -D warnings, 1.88.0 build: clean. No dependency change. Files added or grown stay under 600 (gsi_rotation.rs 314, gsi_queue.rs 501 after a pure move of the claim loop); lib.rs, migrations.rs and the upgrade manual (pre-existing over 600) grew by a few lines.

Merge notes

• fix(postgres): build a GSI added by UpdateTable correctly while writes continue #401 (fix/pg-online-gsi-backfill): two files, seven hunks. .github/workflows/integration.yml (keep both --test lists). crates/storage-postgres/src/gsi_queue.rs: the module doc, the use block, the worker loop's state (cursor here, parked there), the body of process_batch (which this branch moved to gsi_rotation.rs and fix(postgres): build a GSI added by UpdateTable correctly while writes continue #401 rewrites in place), and apply_claimed_row against decode_row. They compose: the merged loop lives in gsi_rotation.rs::process_batch(worker_id, q, cursor, parked); the claim takes table_id = $2 and NOT (id = ANY($3)) together, with the parked filter applied while walking the index; fix(postgres): build a GSI added by UpdateTable correctly while writes continue #401's break on an empty claim becomes rotation.drained(); its two continues (an already-parked item, a lock timeout) keep the rotation on the same table, whose next claim excludes the parked id. fix(postgres): build a GSI added by UpdateTable correctly while writes continue #401 bounds the loop as for _ in 0..BATCH_SIZE where this branch uses while processed < BATCH_SIZE; the merger picks one. fix(postgres): build a GSI added by UpdateTable correctly while writes continue #401's next_ready_wait takes the parked ids and carries over unchanged.
• Fix PostgreSQL global secondary index creation under concurrent writers #454 (fix/pg-gsi-create-race): one hunk, the same --test list; keep both.

Not in this PR

• The ready_at floor is still partition-wide across tables. With one global delay its effect is bounded by one delay; it becomes a stall only if rows carry different delays, and nothing sets the per-index propagation_delay_ms column today. Scoping the floor to (table_id, worker_partition) needs an index leading on table_id for the MAX.
• DescribeTable on main reports a GSI added by UpdateTable as ACTIVE while its backfill is still running inside the request (the service reports CREATING); fix(postgres): build a GSI added by UpdateTable correctly while writes continue #401 is the change that makes the build online.
• The idle path's next_ready_wait keeps the correlated hold anti-join (13 ms on a 40,000-row queue, once per empty batch).
• The drain rate is bounded by one transaction per row (about 400 rows/s per worker here); a bulk write still queues at write speed and drains at that rate.
• Replacing "a table whose claim finds nothing leaves the rotation" with main's break passes every test; it changes only the drain rate while a held table or a concurrently drained table is listed. Recorded in the test module comment as an accepted gap.

Closes #476

Rebase on #401

The branch is rebased onto main at abf8082f. #401's Parked, held_item, decode_row, is_lock_timeout and ROW_LOCK_TIMEOUT are pub(crate) in gsi_queue.rs, so gsi_rotation::process_batch parks a row whose lock wait on its base row times out and leaves parked ids out of each table's claim. The loop keeps this branch's while processed < BATCH_SIZE, so a passed-over row does not count toward the batch.

The rebase also changes two tests:

• gsi_queue_support::enqueue_rows writes the base items as well as the queue rows. fix(postgres): build a GSI added by UpdateTable correctly while writes continue #401 applies a secondary index row from the base item as the table holds it at apply time, so seeded rows with no base item left the index empty (gsi_queue_fairness counted 0 index rows for a 20,000 row backlog, gsi_queue_many_tables 0 for a table's 100 rows).
• gsi_online_build::queue_rows_applied_out_of_order_converge_to_the_item holds the older queue row with FOR KEY SHARE from an outside transaction and makes both rows due, in place of making only the newer row due. This branch lists a table only when its lowest id row is due, so the newer row was never claimed. The changed test passes on main's code as well.

gsi_queue_fairness (2), gsi_queue_many_tables (1), data_migrations (1), gsi_online_build (12), gsi_stale_writer (1) and the crate's unit tests (71) pass together.