Skip to content

perf(processor): zero factorized table blocks as tuples are claimed - #1300

Merged
adsharma merged 1 commit into
LadybugDB:mainfrom
zahariash:perf/ftable-zero-on-claim
Oct 19, 2026
Merged

adsharma merged 1 commit into
LadybugDB:mainfrom
zahariash:perf/ftable-zero-on-claim

Conversation

@zahariash

Copy link
Copy Markdown
Contributor

This work was produced with the help of language models.

Fixes #1299. Part of #1290.

Every FactorizedTable zero-filled a full 256 KB page for its first flat tuple block, even for one row. RETURN 1 created two (the dummy scan's one-row table and the result collector's), so each query cleared 512 KB: about 13% of a warm RETURN 1.

Fix. Flat tuple blocks are taken without zeroing and DataBlock::claim zeroes 4 KB at a time as tuples are handed out; the block counts the bytes at its end that are neither zeroed nor written. Claimed tuples read as zero exactly as before, and clear() zeroes only the part that was used. Hash-slot and sort blocks still zero up front.

Performance. RETURN 1 in a fresh in-memory database, query-summary execute time, median of 8 processes; warm is a C API loop of 20000 queries.

main this PR #1297 #1297 + this PR
1st query 0.94 ms 0.74 ms 0.54 ms 0.32 ms
warm 157 us 147 us 59 us 53 us

Heavy LDBC SF1 queries at one thread: identical rows, instructions within ±0.4%, except a 1M-group aggregate at +0.9% instructions; its cycles stay within run-to-run noise (1747M vs 1756M per execution, median of 8 runs that vary by up to 12%). It zeroes 12 MB less per execution, in more and smaller memset calls. Mallocs per execution are unchanged: 601 for RETURN 1 at one thread, and within ±2 on the heavy queries.

Tests. FactorizedTableTest.ClaimedTuplesAreZeroOnRecycledPages dirties a page, frees it, and checks that tuples handed out from the recycled page are zero across several 4 KB steps, after a resize past a 4 KB step and after clear() (fails if a claim stops zeroing). ClaimsAfterMergeKeepCopiedTuples: after a merge copies tuples past the other block's zeroed part, they stay intact and the next tuple handed out is zero (fails without the merge bookkeeping). Full ctest on this branch: 3102/3102.

Every FactorizedTable zero-filled a 256 KB page for its first flat tuple
block, so a one-row RETURN cleared half a megabyte across the dummy scan
and the result collector. Flat tuple blocks now count the bytes at their
end that are neither zeroed nor written, and claim() zeroes 4 KB at a
time as tuples are handed out. Claimed tuples read as zero exactly as
before; clear() zeroes only the part that was used. Blocks used for hash
slots and sorting still zero up front.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@adsharma adsharma left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved. I statically verified the two uninitialized-memory paths rather than leaving them as open questions:

  • clear() ordering is correct: resetToZero() (zeroes size - uninit, i.e. everything ever touched) runs before freeSize is reset to full, so post-clear claims into the used prefix hit already-zeroed bytes and claims past the old high-water mark trip zero-on-claim. No bug.
  • resize() shrink matches main: the shrink branch bumps freeSize without touching numUninitializedBytes, but the old code never zeroed on claim either, so shrink-then-regrow exposes exactly what main exposes — no regression. The only shrink caller (HashTableQueue::mergeInto) truncates a pre-sized-but-unwritten tail, so no stale tuple data is reachable in practice.

One non-blocking nit: setting numUninitializedBytes = freeSize on shrink would harden the invariant for future callers (it only ever over-zeroes), ideally with a shrink-regrow test alongside ClaimedTuplesAreZeroOnRecycledPages.

@adsharma
adsharma merged commit 16df8bf into LadybugDB:main Oct 19, 2026
4 checks passed
@zahariash

zahariash commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor Author

This was merged before the nit was pushed, so 46a65aa is not in main. The same change (truncate() on shrink plus TuplesAreZeroAfterShrinkAndRegrow) is now #1323; full ctest there: 3118/3118.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Factorized tables zero-fill a 256 KB block up front, 512 KB cleared per constant query

2 participants