Conversation
… terminating `MergeTreeTransaction::afterCommit` and `rollback` are `noexcept`. They write part version metadata (`creation_csn`, `removal_csn`, `removal_tid`) and the CSN of a mutation to disk. A storage error in one of these writes escaped and terminated the server, although the transaction was already committed (or rolled back) in the transaction log and a restart repairs the files from it. The six writes now go through `retryMetadataStore`: the error is retried with backoff for up to 60 seconds per object; `LOGICAL_ERROR` and `NOT_IMPLEMENTED` are rethrown at once; an exhausted budget, or a server shutdown, rethrows as before, so a lost write is never hidden. `MergeTreeMutationEntry::writeCSN` used to append one line to the mutation file; a write that fails half-way, or is repeated, could leave a partial or duplicated `csn` line, which the loader rejects. The whole record is now written through a temporary file and replaces the old one, so a retry is idempotent. `loadMutations` tolerates a temporary file that a repair earlier in the same pass has already consumed. `KILL MUTATION` between the log write and `afterCommit` erases the mutation entry and cannot roll the committing transaction back; `setMutationCSN` then threw `LOGICAL_ERROR` under `noexcept`. It now logs a warning: the parts are already mutated and there is nothing left to write. Two `ONCE` failpoints and the stateless test `05053_transaction_metadata_store_retry` cover the commit of parts, the commit of a mutation and a rollback. Closes: #2344 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
…a-store-retry-squashed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
MergeTreeTransaction::afterCommitandrollbackarenoexceptand write part version metadata and mutation CSNs to disk. A storage error in one of these writes escaped and terminated the server, although the transaction was already committed (or rolled back) in the transaction log and a restart repairs the files from it. This is what killed the server in the CAS test runs of the linked issue.The six writes now go through one helper,
retryMetadataStore: the error is retried with backoff for up to 60 seconds per object;LOGICAL_ERRORandNOT_IMPLEMENTEDare rethrown at once; an exhausted budget, or a server shutdown, rethrows as before, so a lost write is never hidden.MergeTreeMutationEntry::writeCSNrewrites the whole mutation record through a temporary file, so a retry is idempotent.setMutationCSNlogs a warning instead of throwingLOGICAL_ERRORwhenKILL MUTATIONerased the entry during the commit window. TwoONCEfailpoints and the stateless test05053_transaction_metadata_store_retrycover the commit of parts, the commit of a mutation and a rollback.The change is generic MergeTree code and is meant to be cherry-picked to upstream unchanged.
Closes: #2344
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Fixed a server termination when a disk write of transaction metadata (part CSN or mutation CSN) failed inside the commit or rollback of a MergeTree transaction; such writes are now retried for a bounded time. Also fixed a termination when
KILL MUTATIONraced with the commit of a transactional mutation.Documentation entry for user-facing changes
CI/CD Options
Exclude tests:
Regression jobs to run:
🤖 Generated with Claude Code
https://claude.ai/code/session_01GhVd7eMAWdFubNk4g1B2Tx