Repository navigation
Fix rebase conflicts, log and ATTACH durability, recovery and row diff OOM - #1
Merged
Merged
Conversation
…ry, OOM in the row diff Rebase - A statement's own conflict policy (INSERT/UPDATE OR ..., and the ON CONFLICT clause of an UPSERT) is not in the row images; replaying it with the policy of the schema could delete or ignore a row that another writer committed. Such a transaction is refused (BUSY) instead of rebased. - A column of type ANY in a STRICT table has no affinity: 1 and 1.0 are different values there (noaff). - bind_col: no left shift of a negative value when decoding an integer. Log - synchronous=NORMAL/OFF: the commit returns only when its record is in the file, not while another writer is still flushing an earlier prefix (stage_wait_written). Measured: about 25% fewer commits/s with 16 agents on 8 threads, p50 35 -> 48 us. - PRAGMA synchronous: the level of the engine follows SQLite (names, booleans, hexadecimal and out-of-range values, mask of the pager), and a PRAGMA that SQLite rejects inside a transaction does not change it. The connection is known to the lane from the start (rd_db). Recovery - A record that cannot be read leaves the end of the log unknown: the segments after it are no longer unlinked, so that a later open can retry. Row diff - Allocation failures are reported (unsupported: "no memory", the commit is refused) in ov_put, the overflow chain, the freelist copy and the changes. Tests - mw_strictany, mw_rebasepolicy, mw_rebaseneg, mw_syncpragma, mw_stagednormal (pauses the real pwritev of the log), mw_recoverread (EIO at every n-th read), mw_rowdiffoom (fails every allocation of the commit in turn). - test/sanitize.sh: pipefail, every test runs and the first failing status is the exit status; the tests with an injected function are built there too. test/sanitize_exit.sh checks that, and runs from make test / test-ci. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…log fails The append of one commit failed with a later commit already behind it, so the log is failed for good. A commit parked in sync_staged whose record the running fsync was making durable saw `failed` first and returned IOERR_FSYNC; the leader then finished, published the epoch and the commit was visible (and in the log after a reopen) although it was told that it had failed. mw_rollbackrace showed it as "t4 has 80 rows, 79 acknowledged" (2 in 150 runs on two CPUs, also on main; every run on the CI runner of the PR). A waiter that finds the log failed now waits while an fsync is in flight and decides after it: durable -> acknowledged, otherwise refused. No fsync begins once the log is failed, so what the running one does not cover stays invisible. mw_holesync reproduces it in every run (x covered by the held fsync, f fails, s behind the hole): without the change w is refused and visible. The fsync is held by building multiwriter_log.c with -Dfsync=inject_fsync, as the other tests do with pwritev and malloc. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Log: a commit covered by the fsync in flight is not refused when the log fails
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cosa cambia
Rebase
INSERT/UPDATE OR ..., clausolaON CONFLICTdi un UPSERT) non è nelle row image: riprodurlo con la policy dello schema poteva cancellare o ignorare una riga già confermata da un altro writer. La transazione ora è rifiutata (BUSY) invece di essere rebasata.ANYdi una tabellaSTRICTnon ha affinità (noaff):1e1.0sono valori diversi.bind_col: niente shift a sinistra di un valore negativo nella decodifica degli interi.Log
synchronous=NORMAL/OFF: il commit ritorna solo quando il suo record è nel file, non mentre un altro writer sta ancora scrivendo un prefisso precedente (stage_wait_written).PRAGMA synchronous: il livello dell'engine segue SQLite (nomi, booleani, esadecimali, valori fuori range, maschera del pager) e un PRAGMA rifiutato da SQLite dentro una transazione non lo cambia.SQLITE_FCNTL_PDB, anche perATTACH: la verifica dell'autocommit funziona prima della prima lettura/scrittura e anche quando l'applicazione sostituisce il trace hook. Dopo un cambio a OFF rifiutato da SQLite, un commit FULL continua a sincronizzare il log.Recovery
Row diff
unsupported: "no memory", il commit è rifiutato) inov_put, catena overflow, freelist e lista dei cambi.Test
mw_attachsync: 32 casi con main standard o multiwriter, modalità thread o multiprocesso, rebase abilitato/disabilitato, trace hook presente/rimosso e DETACH seguito da un nuovo ATTACH. Verifica la connessione della lane, i PRAGMA rifiutati prima e dopo la scrittura e il sync effettivo del commit FULL.mw_strictany,mw_rebasepolicy,mw_rebaseneg,mw_syncpragma,mw_stagednormal(mette in pausa il veropwritevdel log),mw_recoverread(EIO a ogni n-esima lettura),mw_rowdiffoom(fallisce a turno ogni allocazione del commit).test/sanitize.sh:pipefail, tutti i test girano e il primo stato non-zero è l'exit status; i test con funzione iniettata sono compilati anche lì.test/sanitize_exit.shlo verifica ed è eseguito damake test/test-ci.Da sapere prima del merge
Costo in NORMAL/OFF: circa -25% di commit/s misurato con
mw_bench --workload independent --agents 16 --threads 8(base della PR ~206k tx/s, questo branch ~157k; p50 da 35 a 48 µs). È il prezzo del non confermare un commit il cui record è ancora solo nel ring in memoria.Scelta conservativa:
INSERT OR ABORT/FAIL/ROLLBACKsu tabelle senzaON CONFLICTnello schema non è più rebasabile, anche se lo sarebbe senza rischio. Restringerlo guardando lo schema è possibile in un seguito.Verifica
mw_attachsync.make test,make test-mp,make test-ioemake test-loadable: exit 0.mw_diskfullviene saltato perché l'ambiente non può creare l'immagine disco; i casi di disco pieno simulati inmw_ioerrpassano.mw_attachsync,mw_syncpragma,mw_strictany,mw_rebasepolicy,mw_rebaseneg,mw_recoverread,mw_rowdiffoom,mw_stagednormalemw_rebasecases.mw_rebasenegsoltanto sotto UBSan).mw_rebasepolicypassano anche conMW_TEST_MP=1; senza il controlloCONFLICTperdono la riga confermata dal secondo writer.