Skip to content

fix(transaction): detect DDL conflicts with staged inserts at statement time - #1506

Merged
adsharma merged 2 commits into
LadybugDB:mainfrom
aikins01:fix/ddl-insert-conflicts-at-statement-time
Nov 4, 2026
Merged

adsharma merged 2 commits into
LadybugDB:mainfrom
aikins01:fix/ddl-insert-conflicts-at-statement-time

Conversation

@aikins01

@aikins01 aikins01 commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Description

Follow-up to #1435. Its DDL-vs-insert check ran in LocalStorage::commit, which runs after writeCommitToWAL has already logged the transaction and its commit record. A commit it rejected was rolled back in memory but replayed on the next open: with the repro from #1435, reopening without a checkpoint brings back 2|20|0, the zero-filled column the check was meant to stop. It also had three other problems:

  • An ALTER could still reshape the table between the check and the append.
  • An insert followed by the transaction's own DROP COLUMN was rejected at COMMIT as a conflict with another transaction, since dropColumn bumped the schema epoch without refreshing the transaction's own snapshot. That COMMIT no longer reports a conflict, but it still fails a runtime check (and writes the wrong values in a release build) until Bug: DROP in a transaction and ADD after a rolled-back ADD write node column values into the wrong column #1221 is fixed, so there's no test for it here.
  • After ALTER TABLE A RENAME TO B, inserts into B could fail with the conflict error, because getHeadEntryOfOID returned whichever head came first in the catalog map, including the old name's tombstone.

This replaces the commit-time check with conflicts at statement time, so a rejected insert never reaches the WAL. The transaction manager keeps, per table, the transactions that have inserts staged in a local table. A transaction's first insert into a table fails if the newest catalog version (committed or not) no longer exists or has different column IDs or types than its snapshot. Column IDs rather than names are compared, so a concurrent rename or comment doesn't conflict, but DROP b followed by ADD b does. ALTER TABLE ... ADD/DROP and DROP TABLE fail while another transaction has staged inserts into the table. The DDL changes the catalog before it checks the registry, and both checks run under one mutex, so whichever comes second sees the other.

That changes which side loses. With #1435, an ALTER always went through and the inserting transaction failed at COMMIT. Now the second statement fails: an ALTER or DROP TABLE is refused while another transaction holds uncommitted inserts into that table. DROP TABLE beside staged inserts had no check before #1435. Without one, the inserting commit goes through and the WAL then fails to replay.

The insert-side check reads the newest catalog version under the catalog set's lock. Catalog rollback now takes that lock too, so a rolled-back ALTER can't free the entry while it's being read.

ALTER TABLE ... DROP FROM A TO B against staged rel inserts isn't covered; neither version of the check handled it.

Types of changes

  • Bug fix
  • New feature
  • Breaking change
  • Documentation Update

Checklist

  • I have changed storage version if on disk format has changed.
    Not needed: the on-disk format is unchanged.
  • I have requested a review from a maintainer.
  • I have updated the documentation (if needed).

…nt time

b6649b7 checked for a concurrent ALTER TABLE ADD/DROP in
LocalStorage::commit, which runs after the transaction's records and
commit record are already in the WAL. A commit it rejected was rolled
back in memory but replayed on the next open, with the zero-filled
column it was meant to prevent. An ALTER could also reshape the table
between the check and the append, an insert followed by the
transaction's own DROP COLUMN failed at COMMIT, and an insert into a
renamed table failed once the old name's tombstone came first in the
catalog map.

The conflict now fails at statement time instead. The transaction
manager keeps, per table, the transactions with inserts staged in a
local table. A transaction's first insert into a table fails if the
newest catalog version, committed or not, no longer exists or has
different column IDs or types than its snapshot. ALTER TABLE ADD/DROP
COLUMN and DROP TABLE fail while another transaction has staged
inserts. The DDL changes the catalog before it checks and both checks
run under one mutex, so one side always sees the other.

The newest catalog version is read under the catalog set's lock, which
catalog rollback now takes too, so a rolled-back ALTER cannot free the
entry while it is read.
…bugDB#1221

Committing rows staged before the transaction's own DROP COLUMN still
fails a runtime check in NodeGroupCollection::append (LadybugDB#1221), so the
case cannot pass until that is fixed.
@adsharma

adsharma commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

4 followups (non-blocking):

  • ADD/DROP FROM/TO vs staged rel inserts still uncovered (as noted for DROP FROM); confirm intentional or file followup.
  • removeLocalWriter only in ~LocalStorage — works via clearTransactionNoLock destroy today, but fragile; prefer explicit deregister on commit/rollback.
  • Lock order (localWritersMtx -> catalogSet.shared, never held reverse) is load-bearing but undocumented; alter/drop must keep catalog change outside hasLocalWriter check.
  • Add partitioned RENAME parent -> insert case (PartitionedAlterRestrictions on main hits the same tombstone); parent ADD/DROP path only checks parent ID for when propagation lands.

@adsharma
adsharma merged commit 1a9f83c into LadybugDB:main Nov 4, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants