Skip to content

[codex] Prototype frontier scans and compare DuckDB against V1 - #9884

Draft
joseph-isaacs wants to merge 75 commits into
developfrom
ji/push-frontier-all-bench
Draft

[codex] Prototype frontier scans and compare DuckDB against V1#9884
joseph-isaacs wants to merge 75 commits into
developfrom
ji/push-frontier-all-bench

Conversation

@joseph-isaacs

@joseph-isaacs joseph-isaacs commented Sep 15, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Benchmark run links: The completed real TPC-H + ClickBench benchmark job ran on e199904d (the same source tree as this PR head). A repeat on the exact current head 09edaaa0 also completed successfully. Its recomputed summed medians show TPC-H W8 12.46% / prefetch 13.64% less time than V1, and ClickBench W8 15.99% / prefetch 16.24% less time. Both runs therefore show aggregate wins on both suites. The repeat's ClickBench prefetch increment is +0.30%, versus −0.32% in the first run; there is still no consistent additional ClickBench prefetch benefit. These are manually dispatched Actions runs; GitHub currently omits them from this PR’s check summary. In each run, open frontier / bench. The alternative sql job is intentionally skipped for the pr-frontier preset. A completed benchmark run does not mean the PR’s normal checks have passed.

This draft introduces the grouped-I/O frontier scan prototype and its integration with DuckDB. It retains the real V1 path for direct comparison and keeps frontier pairing and projection prefetch opt-in.

The frontier executor groups reads according to predicate/projection demand, shares file-level I/O admission, and bounds speculative work. DuckDB's external execution path owns its output work and conversion caches. This branch also contains the supporting scan abstractions, diagnostics, tests, and experiment history; it is a broad prototype draft rather than a production-default change.

The latest local comparison on real TPC-H SF10 files, at 14 threads, reduced the sum of all 22 query medians from 2947.30 ms with V1 to 2497.73 ms with W8 plus projection prefetch (15.25%). Projection prefetch contributed 2.24% beyond W8. Q6 regressed with prefetch and Q9 was flat, so prefetch remains disabled by default. These are local Apple Silicon results, not CI results.

The earlier local ClickBench timing campaign was excluded after unrelated IDE/Cargo activity contaminated it. The completed dedicated CI comparison below now provides a valid real-file ClickBench result. Earlier diagnostics show that the W8 per-file pairing threshold does not activate for the small shards tested; these gains describe the full frontier implementation, not pairing alone.

CI comparison

The pr-frontier preset of PR SQL Benchmarks builds the comparison drivers and runs TPC-H SF10 and the full 100-shard ClickBench dataset sequentially on a dedicated bare-metal runner. Each suite compares V1, W8, and projection prefetch with one binary, 14 threads, six balanced rounds, and an immediate same-arm warmup per measurement. It checks independent Parquet results, identical V1/frontier plans, result cardinality, file/binary/SQL hashes, and no spilling. Per-query tables, paired-round intervals, exact commands, raw results, and operator profiles are uploaded as workflow artifacts.

SQL comparison run 34962772259 completed successfully on 2026-09-15. It tested commit e199904d7a57b88312ecbc880eab0ec07edbe79b; the subsequent DCO-only commit has the identical source tree (a4d9bf0dd016ccb096d5dda64b8c4bbbeae811fe). Hardware: dedicated AWS c6id.metal, Intel Xeon Platinum 8375C, DuckDB 14 threads, CPU/memory affinity to NUMA node 0.

These are full-query elapsed times with warm real-file scans. Totals are sums of per-query medians, not scan-only time or one timed suite execution.

Suite Queries V1 total ms Frontier W8 total ms W8 time reduction W8 + prefetch total ms Prefetch time reduction
TPC-H SF10 22 4757.83 4159.91 12.57% 4064.19 14.58%
ClickBench, 99,997,497 rows / 100 shards 43 19188.03 16092.11 16.13% 16143.17 15.87%

Both fixed frontier configurations beat V1 in aggregate on both suites. Prefetch reduces TPC-H time by a further 2.30% versus W8, but is 0.32% slower on ClickBench; its paired interval there crosses parity, so there is no demonstrated ClickBench prefetch benefit. Prefetch remains opt-in.

The six paired round totals favor frontier in every round for both configurations and both suites. Exploratory 95% bootstrap intervals for frontier/V1 total-time ratios are TPC-H W8 [0.8673, 0.8778], TPC-H prefetch [0.8467, 0.8565], ClickBench W8 [0.8362, 0.8441], and ClickBench prefetch [0.8370, 0.8472]. These quantify within-run repeatability, not variation across machines or independent CI runs.

TPC-H has lower medians on 20/22 queries in each frontier arm; Q2 and Q16 regress by up to 2.27%. ClickBench has lower medians on 43/43 queries in each arm, although small differences such as W8 Q39 (+0.30%) are not established individual wins. TPC-H Q9 remains a modest 5.89% reduction with prefetch. ClickBench Q28 remains dominant and improves only 1.82% with W8. No per-query multiple-comparison correction is applied.

Validation completed 1,170 measured executions, 2,535 warm/measured/validation checks, and 195 separate no-spill profiles. All 65 queries passed independent Parquet-reference validation for all three Vortex arms. Post-download auditing rechecked 2,535 result hashes and 2,031 full-value comparisons (the other 504 canonical executions have ambiguous LIMIT ties and use row-count checks), all plan-equality gates, measurement order, recomputed medians, source identity, and profile spill counters.

CI input sizes: TPC-H Vortex 2,379,939,292 bytes across eight files (59,986,052 lineitem rows); ClickBench Vortex 11,616,661,328 bytes across 100 files. Each arm used exactly the same files; manifests include their SHA-256 hashes, SQL, binaries, and linked DuckDB. CI-generated TPC-H compression differs from the earlier local archive, so absolute local and CI timings are not directly compared.

Download all raw results, manifests, profiles, and complete 65-query tables (artifact duckdb-frontier-results, archive digest sha256:05c6d68f33ba843fda4e63d8cfcf26fded6dd5276f814b3b2e1d815053557b62). Exact CI binaries are also available.

The comparison reports improvements and regressions and does not require a speedup to make the job green. The runner's optional global workqueue CPU-mask write was rejected by sysfs; process NUMA affinity was recorded successfully. The runner's post-job metrics visualizer also failed after results were uploaded. Neither is hidden as a successful check.

Known limitations

  • The draft currently conflicts with develop. DCO formatting was repaired without rewriting history; older commit 4974fd147d7492b57d37702b90ec1b3c03fa0bb5 still needs author attention because its signer is EC2 Default User rather than its author. These do not prevent the explicit branch-head benchmark run.

  • TPC-H Q22 has an unresolved default-plan correctness bug shared by V1/frontier. The comparison disables common_subplan only for Q22, equally in every arm and the Parquet reference. This benchmark is not a claim that the default Q22 path is fixed.

  • ClickBench's canonical SQL is timed unchanged; correctness uses separate tie-breakers for ambiguous LIMIT queries, including the Q24 and Q30 ties found during this work. Floating aggregates use a tight numeric tolerance; other values and schemas are exact.

  • The W8 policy is experimental: SQL LIMIT propagation and memory/resource acceptance remain open. There is no claim here about cold-cache or remote-storage performance.

  • Historical handover links under /private/tmp identify local evidence archives. New CI artifacts provide downloadable evidence for the exact PR head.

Validation

  • Existing retained runtime changes: 396 targeted tests passed (2 skipped); 167 frontier tests also passed with prefetch enabled.
  • Current checkpoint: 464 targeted Rust tests passed (2 skipped), all six CI runner unit tests passed, and both real-file smoke campaigns passed correctness/plan/cardinality/hash/spill gates.
  • Formatting and targeted strict Clippy pass. Workspace all-feature Clippy is blocked by the existing compress-bench/src/main.rs:408 cognitive-complexity error (30 versus 25); it is not hidden by this draft.

The canonical implementation/experiment notes are in vortex-morsel-push/IO_FRONTIER_HANDOVER.md.

joseph-isaacs and others added 30 commits August 27, 2026 12:59
@joseph-isaacs
feat(morsel): add the P1 morsel-driven scan executor spine
57ff482 
Adds `vortex-morsel`, the P1 spine of the morsel-based plan execution design:
stateful per-morsel `ExecNode` state machines driven inline off one atomic
morsel cursor, with a scheduler-visible IO plane that nodes name reads
against but never read through.

- `ExecNode` is `next_plan` / `execute` / `retire`. Planning names IO as
  `IoUse`s keyed to whole stored units and is budget-bounded and resumable;
  execution may only wait on tickets its own planning stream emitted.
- Five operators: FLAT, CHUNKED, STRUCT, CONJUNCT (cascade and parallel as
  one node behind a policy flag) and FILTER.
- Nodes live in a per-thread arena instantiated once from a shared immutable
  plan, so nothing is allocated per morsel and nothing on the hot path is
  shared between threads.
- A per-thread decoded-chunk cache means a chunk straddling several morsels
  is decoded once rather than once per morsel that touches it.
- Unsupported layouts and expression shapes are build errors, never silent
  fallbacks, so an unsupported query cannot be timed as if it had run.

Correctness is differential against the V1 `LayoutReader`: 16 tests covering
misaligned chunking (including the design document's [0,3,10) vs [0,6,10)
case), thread counts, morsel sizes, conjunct policy and decode-cache budget,
each asserting the same rows in the same order as V1.

Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
@joseph-isaacs
feat(morsel): evaluate the P1 spine against the V1 LayoutReader
cca7f7a 
Adds the comparison harness, the evaluation binary and the P1 findings.

The harness makes V1 both a row in the matrix and the oracle: every
configuration's output is validated against V1's (equal row count, equal
ordered content) before anything is timed, so a configuration that drops rows
is reported as a failure rather than as a fast run. Timing then alternates the
executors over five iterations and reports the median.

Because the named suites need multi-gigabyte downloads this environment cannot
fetch, the workloads are shape-matched synthetic fixtures reproducing what the
plan says those suites lower to: struct-of-chunked-flat columns whose
per-column chunk boundaries deliberately disagree, under conjunctive filters of
varying selectivity. This is stated in the findings rather than implied by the
numbers, along with the rest of what this run does not establish.

Measured over 15 queries, all 105 configuration-query pairs matching the
oracle: geomean 0.553 vs V1 at equal thread count, 0.251 at four threads with
coalesced morsels. The decode cache accounts for most of the string-heavy win;
morsel coalescing accounts for most of the wide-table win.

Gate E1 as specified was NOT evaluated: it requires rows B and C (the
self-paced graph/reactor and pipeline executors), and neither exists at any
commit reachable in this repository. The findings say so explicitly and do not
compare a synthetic-fixture ratio against the recorded real-suite ratios.

Also carries the scan execution model design documents onto this branch,
unchanged from the design branch, so the findings' references resolve.

Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
@joseph-isaacs
fix(morsel): remove the decoded-chunk cache the V1 LayoutReader does …
dabd419 
…not have

The prototype carried a per-thread decoded-array cache and retained resolved
IO cells across morsels. V1 has neither, so timing against it with them was
measuring a cache, not an executor. Both are removed: there is no decode
cache, and a morsel's IO cells are released at retire, so a segment straddling
two morsels is read and decoded once per morsel exactly as V1 does per
evaluation. Within-morsel cell sharing remains — two uses naming the same
stored unit resolve to one read — which is the registration mechanism itself,
not retained state.

The eval counters now show requests = uses = decodes in every configuration,
which also removes the previous 4-thread byte-duplication asymmetry: the
cold-scan IO invariant holds at every thread count.

Re-measured, all 90 configuration-query pairs matching the V1 oracle:
geomean 0.650 vs V1 at equal thread count (was 0.553 with the cache),
0.238 at four threads with coalesced morsels. The string-heavy select-all
moves from 0.49 to 0.97 — the honest number for a decode-dominated query —
and the delta is recorded in the findings as the measured value of
cross-morsel decode reuse (~2x), to be rebuilt in P2 as a shared, budgeted,
scheduler-visible facility rather than executor-private state.

Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
@joseph-isaacs
feat(morsel): recover cross-morsel decode reuse with leased shared cells
f22008b 
Rebuilds the ~2x decode-reuse win on misaligned string layouts as the P1
slice of P2's keyed cells, with retention derived from demand rather than
from a budget. Before the scan starts the driver counts, from the morsel cut
and the plan's flat nodes alone, how many (node, morsel) pairs will touch
each stored unit. The first morsel to decode a unit publishes the array into
its cell; every retiring morsel releases its lease whether it used the cell
or not; the last release drops the array. Nothing is held speculatively,
nothing has an eviction policy, nothing survives the scan, and the ledger is
asserted to drain to zero. A morsel whose planning finds the cell populated
skips issuing the read — its own unreleased lease pins the value until it
retires — so requests fall together with decodes.

The cell map is sharded 16 ways: a first cut with one mutex made the
wide-numeric workload slower at 4 threads than at 1 (lease traffic scales
with nodes x morsels), and sharding restored the scaling. Recorded in the
findings as an early admission-plane lesson.

Sharing can be disabled (`with_share_decodes(false)`), leaving no state
across morsels at all; that configuration is kept as the state-for-state
fairness row against V1 and as the chaos check: per query, a sharing run's
decodes + reuses must exactly equal a non-sharing run's decodes, asserted in
tests along with each straddled chunk decoding exactly once per scan.

Re-measured, all 105 configuration-query pairs matching the V1 oracle:
geomean 0.539 vs V1 at equal thread count (0.644 with sharing disabled),
0.249 at four threads with coalesced morsels. String-heavy select-all:
310 decodes without sharing, 121 with — one per chunk — moving 0.87 to 0.46.

Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
@joseph-isaacs
feat(morsel): benchmark real TPC-H queries end to end on both executors
27cf7d4 
Replaces the shape-matched synthetic TPC-H rows with the real thing.

Data comes from tpchgen at a real scale factor — dbgen's schema,
distributions and correlations, decimal(15,2) money, date[days] columns —
imported through the session's own Arrow path, the same one vortex-bench
uses to build its TPC-H files. No download required.

The fixture is written through a real btrblocks compressing pipeline
(repartition 8192 rows, coalesce to 1 MiB, compress, buffer, chunk, flat),
so decode cost is what a real file imposes rather than the raw-buffer cost
the synthetic fixtures had. Zone maps and the dictionary layout are left out
because P1 supports neither: enabling them would compare a pruning executor
against a non-pruning one instead of comparing executors, which is stated as
a capability gap rather than folded into a ratio.

Queries are the scan portion of the TPC-H queries in vortex-bench/sql/tpch/
— Q6, Q1, Q14, Q15, Q12, Q19 — transcribed predicate for predicate, plus a
bare six-column scan and a highly selective shape. The exactness contract is
stricter than the synthetic eval's: every configuration must reproduce V1's
dtype, row count and ordered content before any timing, and a mismatch
aborts the run rather than dropping a row from the table.

Also stops registering single-lease units in the shared cells. A unit
touched by exactly one (node, morsel) pair can never be reused, and
registering it cost ~20% on a pure six-column scan for bookkeeping that
could never pay off.

Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
@joseph-isaacs
docs(morsel): record the real TPC-H SF=1 results
e9fd51f 
The prototype is ~1.3x faster than the V1 LayoutReader at one thread and
~1.5x at four (1.85x with coalesced morsels), on real TPC-H lineitem with
real decimals, dates and btrblocks encodings. All 48 configuration-query
pairs reproduced V1's dtype, row count and ordered content exactly.

Two results that revise the earlier synthetic findings, both recorded rather
than smoothed over:

The single-thread margin drops from 0.54x to 0.75x. The P1 findings
predicted exactly this: writing uncompressed leaves understated decode cost,
which both executors share, and so inflated the prototype's apparent margin.
Compressed decode now dominates and the margin narrows to what the executor
controls. Q6 (0.92x) and Q12 (0.85x) are the clearest cases.

Cross-morsel decode reuse turns out to be neutral on a real file (0.75x with
sharing, 0.76x without) because the real write pipeline repartitions every
column onto the same row blocks, so the misalignment the synthetic fixtures
forced does not occur. Q19 is the exception, at 0.54x against 0.69x, because
its string columns compress to sizes that land on different block
boundaries. Keyed cells earn their place on width-divergent schemas and on
filter-and-project column overlap, not as a general win.

Also removes a stale mut and refreshes the raw evaluation output.

Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
@joseph-isaacs
docs(morsel): explain the four-thread win, and add a handoff
6867f49 
Adds a sweep mode to tpch-eval that measures rather than asserts why the
four-thread rows win, and a handoff document for re-running the whole
evaluation on other hardware.

The sweep varies three things: the morsel driver's thread count from one to
four times the core count, V1's per-worker split concurrency (so the baseline
is tuned to its own best rather than left at a possibly unfavourable
default), and morsel size.

Against V1 tuned to its best concurrency the morsel executor at four threads
is 0.61x geomean, decomposing as ~0.73x single-thread base advantage times
~1.19x better scaling — roughly three quarters base, one quarter scaling.

The win is not oversubscription. This host has one thread per core, and one
driving thread per physical core is optimal on seven of eight queries (x8
costs ~10%, x16 ~20%). V1 reaches its best at four workers times concurrency
sixteen, so 64 in-flight split tasks against the morsel driver's four
morsels: the driver needs 16x fewer concurrent units and still wins.

Corrects an overstatement in the previous commit. Morsel coalescing does not
generally help: excluding Q19 it is 1.02x, no effect at all, and the whole
geomean advantage of the 64k row was Q19 alone (0.42x), whose string columns
land on different block boundaries and give it 366 natural splits where every
other query has 92. Past 64k rows coalescing hurts, with Q12 going from 9.3ms
to 22.7ms at 1M-row morsels.

Also records that single-query differences under ~20% are within this host's
noise, which is one of the reasons the handoff asks for a re-run elsewhere.

Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
@joseph-isaacs
feat(morsel): add affinity scheduler and background IO
287d005 
Keep one thread-local morsel and reusable arena per worker while sharing required and speculative IO queues with exact-ticket wakeups.

Move file reads into planning and the background service, improve local coalescing, add hot/cold and morsel-size benchmark controls, and record the validated TPC-H results.

Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
@joseph-isaacs
feat(morsel): integrate push-read SQL scan benchmarks
ec00f09 
Add the morsel executor, push-read scan integration, per-morsel statistics pruning, compression and random-access runners, persistent worker optimizations, and loadable DuckDB extension benchmark support. Layout-v27 and AVX-512 experiments are intentionally excluded.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
@joseph-isaacs
feat(morsel-push): import shared push primitives
ab32824 
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
@joseph-isaacs
feat(morsel-push): import optimized push reader tree
960a997 
Source-tree: ae8b9800409a60d1ceebb2b8181a144581a0cc45:vortex-morsel

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
@joseph-isaacs
feat(morsel): select independent v1 pull or push readers
069a77f 
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
@joseph-isaacs
docs(morsel-push): document scoped upstream refresh
624142e 
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
@joseph-isaacs
refactor(morsel): isolate SQL backend selection
f67e595 
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
@joseph-isaacs
fix(morsel): preserve selection through stats pruning
314836c 
Own plan caches per fixed-layout executor to remove stale global pointer identity and retain reusable plans safely.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
@joseph-isaacs
chore(morsel-push): verify scoped upstream refreshes
ee252f7 
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
@joseph-isaacs
perf(morsel): avoid dense pruning mask intersections
f2b439a 
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
@joseph-isaacs
Drive DuckDB morsels on scan threads
aedee32 
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
@joseph-isaacs
feat(morsel): optimize SSD random access execution
d7cf588 
Add mask-aware morsel planning, persistent workers and arenas, batched priority-aware buffered I/O, safe reopen plan reuse, dictionary and nullable layout support, and opt-in scan observability. Document the retained design, rejected experiments, cache boundaries, reproduction workflow, and final SSD comparison against V1.

Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
@joseph-isaacs
Merge remote-tracking branch 'origin/ji/morsel-push-pull-ci' into ji/…
ce2c3aa 
…morsel-perfect-io-scheduling

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

# Conflicts:
#	Cargo.lock
#	Cargo.toml
#	vortex-bench/Cargo.toml
#	vortex-bench/src/random_access/take.rs
#	vortex-io/src/read_at.rs
#	vortex-layout/src/segments/source.rs
#	vortex-morsel/Cargo.toml
#	vortex-morsel/README.md
#	vortex-morsel/src/bin/tpch-eval.rs
#	vortex-morsel/src/build.rs
#	vortex-morsel/src/driver.rs
#	vortex-morsel/src/harness.rs
#	vortex-morsel/src/io.rs
#	vortex-morsel/src/lib.rs
#	vortex-morsel/src/node.rs
#	vortex-morsel/src/nodes/chunked.rs
#	vortex-morsel/src/nodes/conjunct.rs
#	vortex-morsel/src/nodes/flat.rs
#	vortex-morsel/src/nodes/mod.rs
#	vortex-morsel/src/nodes/struct_.rs
#	vortex-morsel/src/stats.rs
#	vortex-morsel/src/tests.rs
@joseph-isaacs
Merge remote-tracking branch 'origin/develop' into ji/morsel-perfect-…
38b460c 
…io-scheduling

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

# Conflicts:
#	scripts/random-access-split.py
@joseph-isaacs
feat(morsel): record and replay exact in-memory I/O demand
ce227ce 
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
@joseph-isaacs
bench(morsel): run real ClickBench scans on all engines
f3980d9 
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
@joseph-isaacs
test(morsel): compare dictionary scans through ScanBuilder
4974fd1 
Signed-off-by: EC2 Default User <ec2-user@ip-172-31-75-25.ec2.internal>
@joseph-isaacs @claude
refactor(morsel): scan through a dedicated MorselScanBuilder
bcc5469 
Drop the ScanExecutor hook from vortex-layout's ScanBuilder and give the
morsel executors their own builder in vortex-morsel-scan. The builder
mirrors the ScanBuilder surface but never constructs a LayoutReader:
layout planning and predicate execution belong to the selected morsel
implementation. DataFusion and DuckDB pick the builder by backend and
only build a LayoutReader for the V1 path.

Both executors expose full_file_splits and build one output future per
morsel, completed through a sink as each morsel retires. The pull driver
keeps a filtered-scan lookahead window ahead of the active workers; the
push driver issues lookahead batches through request_background_batch
and can issue every source in the window eagerly.

duckdb-bench links the workspace extension in-process when no loadable
extension is configured, and falls back to the library vortex-duckdb
built when DUCKDB_LIB_DIR is unset.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NAzndXAjqTKVjgcSbiYyDo
@joseph-isaacs @claude
refactor(morsel): hand I/O out of plan execution
75ed9d5 
Plan execution no longer owns a segment source. MorselScan::new takes a
plan and a session; the reads its scheduler wants started leave the scan
as IoDemand on a stream taken with take_io (Start batches that preserve
the planning-wave boundary, and Promote when execution blocks) and are
answered through IoCompletions. Workers park on their exact cells and
never poll a storage future; wait accounting moves into the service.

SegmentSourceDriver serves that demand from any SegmentSource, either as
a task on a runtime handle or on a dedicated thread for runtime-less
callers. It registers each batch together so coalescing sources see
neighbours, polls required and promoted reads immediately, and polls
speculative reads through a bounded window so a demand-driven source's
promotion order stays meaningful. The non-blocking inline probe survives
as an optional hook the driver installs from the source's request_nowait.

Both executors connect the driver outside the scan. The push crate drops
its urgent and ready I/O work queues and the wakers that requeued them;
the DuckDB current-thread path shares one service and driver across a
file's morsels. Benchmarks, harnesses, and tests connect the driver the
same way.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NAzndXAjqTKVjgcSbiYyDo
@joseph-isaacs @claude
fix(morsel): address review findings and add a primer
2b9df23 
Driver: remember a promotion that arrives before its own start batch so a
worker blocked on a read another worker just started is still served
first; count the speculative window separately from required reads; fail
the reads a segment source returns no future for instead of leaving them
outstanding.

Service: when the demand stream is closed, fail the affected cells and
wake their waiters so the scan errors instead of parking forever, and
refuse to run a scan whose demand was never taken. Skip the separate
promotion when a read just went out as required. Document io_waits and
io_wait_time as completion-based.

Executors: apply an unfiltered LIMIT to the morsel cut before any I/O,
and stop assigning morsels once every consumer has dropped its future.
The push scheduler again attributes planning-wave starts to their morsel
so the per-morsel request counters are meaningful.

duckdb-bench prefers the DuckDB library vortex-duckdb was built against.

Add docs/developer-guide/internals/scan-execution-models/morsel-executor-primer.md:
each trait and type simplified, one scan end to end, and what the
experiment taught.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NAzndXAjqTKVjgcSbiYyDo
@joseph-isaacs @claude
fix(morsel): cancel scans by output lifetime and cap unfiltered limits
606de89 
Every output future now carries a guard; when the last one is consumed or
dropped the scan is cancelled through ScanCancellation (pull) or
StreamCancellation (push), which stops the scheduler and wakes parked
workers, so a scan stalled on slow reads no longer outlives its consumers.
The completion-sink cancellation check is gone with it.

An unfiltered limit is now exact at the executor: morsels past it are
never read and the last one is capped, and a zero limit builds no tasks.
Filtered scans still return every matching row for the builder to trim.

Two tests cover the limit cut and cancellation of a stalled scan; the
primer's limit rule matches the implementation.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NAzndXAjqTKVjgcSbiYyDo
@joseph-isaacs @claude
fix(morsel): honour cancellation at startup and cap external outputs
c16effe 
An already-cancelled scan no longer submits its startup lookahead or
refills it, so a query cancelled before its coordinator starts issues no
I/O. A cancelled pull scan leaves the leases of unfinished morsels
outstanding by design, so the lease-ledger assertion only runs when every
morsel retired; the cancellation test now keeps decode sharing on. The
DuckDB current-thread path applies the same exact row caps as the shared
path when an unfiltered limit ends inside a morsel.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NAzndXAjqTKVjgcSbiYyDo
@joseph-isaacs @claude
refactor(morsel): extract the layout planner, node blueprint, and ans…
ddd6bae 
…werer traits

Three extension points of the pull executor become traits, each with a
reason the primer records:

LayoutPlanner turns one kind of stored layout into nodes. It owns both
answers about its layout, where chunks start and which nodes execute it,
so the morsel cut and the plan agree by construction. The built-in
planners (zoned wrappers, flat, dictionary, struct, chunked) move to
layouts.rs, and LayoutPlanners is an ordered registry a caller can extend
ahead of them. This replaces two duplicated layout matches in build.rs.

NodeBlueprint is the immutable half of a node, instantiated once per
worker arena. Planners push blueprints and only stored_use is inspected
from outside, so a planner can introduce node types the scheduler has
never seen. The closed NodeSpec enum is gone; each nodes/*.rs file now
holds its spec next to its exec.

IoAnswerer is whatever serves the demand stream. MorselScan::connect and
connect_on_thread attach any answerer; SegmentSourceDriver is one of
them. IoPlane::ready no longer hands a read out itself on a probe miss,
since the scheduler already does so when it parks the worker.

Two tests register a planner ahead of the built-ins and serve a scan from
an answerer with no SegmentSource. The primer gains a section on which
extension points are traits and why the rest stay concrete.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NAzndXAjqTKVjgcSbiYyDo
@joseph-isaacs
feat(morsel): add grouped I/O frontier API
714d20e 
Represent logical I/O groups with an incremental plan cursor that can move across groups and row ranges without materializing a range-by-group matrix.

Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
@joseph-isaacs
feat(morsel): schedule bounded frontier I/O
7e64ed6 
Consume grouped plan frontiers with bounded lookahead, exact raw-cell leases, speculative-read cancellation, and owned driver teardown.

Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
@joseph-isaacs
test(morsel): cover grouped frontier scheduling
fa9dd59 
Exercise frontier group shapes, scheduling depths, pruning, retention, cancellation, shared cells, and scan teardown.

Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
@joseph-isaacs
bench(morsel): compare push frontier I/O
cb2dddb 
Add push and V1 discard-output timings, frontier configuration matrices, hot and cold disk controls, injected latency, raw retention metrics, and reusable SF10 packs.

Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
@joseph-isaacs
docs(morsel): record frontier I/O experiment
08d146a 
Document the grouped frontier design, implementation caveats, benchmark history, validation state, and handover instructions.

Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
@joseph-isaacs
bench(morsel): default comparisons to frontier path
2022bd5 
Keep V1 on the current LayoutReader scan defaults while selecting grouped frontier I/O for ordinary push rows. Label both paths explicitly and test that the frontier benchmark defaults change only the scheduler selector.

Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
@joseph-isaacs
feat: add fair push-frontier scan routing
c69bd62 
Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
@joseph-isaacs
feat: add exact push-frontier benchmark verification
0b1eef5 
Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
@joseph-isaacs
docs: record push-frontier benchmark protocol
9fc6989 
Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
@joseph-isaacs
fix(datafusion): leave unsupported projections above scans
ade2bee 
Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
@joseph-isaacs
docs: record FineWeb and TPC-DS correctness
6d08d4f 
Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
@joseph-isaacs
bench: separate correctness matrix policy
eab26ff 
Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
@joseph-isaacs
bench: gate matrix measurements on correctness
10e6135 
Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
@joseph-isaacs
bench: record TPC-H frontier matrix
ae8c638 
Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
@joseph-isaacs
bench: record ClickBench frontier matrix
568fcb5 
Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
@joseph-isaacs
bench: record FineWeb frontier matrix
4bae722 
Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
@joseph-isaacs
bench: record TPC-DS frontier matrix
f102ab2 
Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
@joseph-isaacs
perf: increase push frontier lookahead
121a045 
Use one additional down frontier per worker in the production push-frontier policy while keeping right speculation disabled and 32-range refills.

Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
@joseph-isaacs
revert: reject push frontier lookahead trial
761ff9f 
Restore the production push-frontier policy to zero additional down lookahead after the clean interleaved TPC-H Q6 A/B produced conflicting internal and supervisor evidence. Preserve the trial attestation in the benchmark checklist.

Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
@joseph-isaacs
perf: increase frontier morsel target
d25daa1 
Use 512K-row morsels for the production push-frontier route while preserving the plain push executor fallback and explicit benchmark controls.

Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
@joseph-isaacs
perf: reject larger frontier morsels
fbb232a 
Restore the 128K-row production frontier target after the all-core Q6 trial showed no robust speedup and a consistent RSS penalty. Record the clean build, exact-result, ordering, and normalization evidence.

Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
@joseph-isaacs
feat: publish push scan metrics
f2822e7 
Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
@joseph-isaacs
docs: record Q6 I/O metrics isolation
7b0b268 
Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
@joseph-isaacs
perf: cohort frontier predicate reads
d36cd71 
Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
@joseph-isaacs
revert: reject predicate frontier cohorts
58e2012 
Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
@joseph-isaacs
feat: add scan attribution diagnostics
104297a 
Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
@joseph-isaacs
checkpoint(io): park bounded frontier I/O work
8295653 
Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
@joseph-isaacs
checkpoint(io): document DuckDB frontier W8 performance
ad0749e 
Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
@joseph-isaacs
perf(duckdb): retain frontier prefetch and add paired SQL CI
e199904 
Retain opt-in projection admission and conversion-cache improvements, add
persistent real-file benchmark drivers, and compare V1/W8/prefetch on
independently validated TPC-H SF10 and full ClickBench data in CI.

Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
@joseph-isaacs
DCO Remediation Commit for Joe Isaacs <joe.isaacs@live.co.uk>
09edaaa 
Normalize recognition of existing sign-offs with quoted author names.
No source files or historical commits are changed.

I, Joe Isaacs <joe.isaacs@live.co.uk>, hereby add my Signed-off-by to this commit: 57ff482
I, Joe Isaacs <joe.isaacs@live.co.uk>, hereby add my Signed-off-by to this commit: cca7f7a
I, Joe Isaacs <joe.isaacs@live.co.uk>, hereby add my Signed-off-by to this commit: dabd419
I, Joe Isaacs <joe.isaacs@live.co.uk>, hereby add my Signed-off-by to this commit: f22008b
I, Joe Isaacs <joe.isaacs@live.co.uk>, hereby add my Signed-off-by to this commit: 27cf7d4
I, Joe Isaacs <joe.isaacs@live.co.uk>, hereby add my Signed-off-by to this commit: e9fd51f
I, Joe Isaacs <joe.isaacs@live.co.uk>, hereby add my Signed-off-by to this commit: 6867f49
I, Joe Isaacs <joe.isaacs@live.co.uk>, hereby add my Signed-off-by to this commit: 287d005
I, Joe Isaacs <joe.isaacs@live.co.uk>, hereby add my Signed-off-by to this commit: d7cf588
I, Joe Isaacs <joe.isaacs@live.co.uk>, hereby add my Signed-off-by to this commit: bd1f3cb
I, Joe Isaacs <joe.isaacs@live.co.uk>, hereby add my Signed-off-by to this commit: 381487e
I, Joe Isaacs <joe.isaacs@live.co.uk>, hereby add my Signed-off-by to this commit: 4934e3b
I, Joe Isaacs <joe.isaacs@live.co.uk>, hereby add my Signed-off-by to this commit: 2c3f7f8
I, Joe Isaacs <joe.isaacs@live.co.uk>, hereby add my Signed-off-by to this commit: 442e385
I, Joe Isaacs <joe.isaacs@live.co.uk>, hereby add my Signed-off-by to this commit: 714d20e
I, Joe Isaacs <joe.isaacs@live.co.uk>, hereby add my Signed-off-by to this commit: 7e64ed6
I, Joe Isaacs <joe.isaacs@live.co.uk>, hereby add my Signed-off-by to this commit: fa9dd59
I, Joe Isaacs <joe.isaacs@live.co.uk>, hereby add my Signed-off-by to this commit: cb2dddb
I, Joe Isaacs <joe.isaacs@live.co.uk>, hereby add my Signed-off-by to this commit: 08d146a
I, Joe Isaacs <joe.isaacs@live.co.uk>, hereby add my Signed-off-by to this commit: 2022bd5
I, Joe Isaacs <joe.isaacs@live.co.uk>, hereby add my Signed-off-by to this commit: c69bd62
I, Joe Isaacs <joe.isaacs@live.co.uk>, hereby add my Signed-off-by to this commit: 0b1eef5
I, Joe Isaacs <joe.isaacs@live.co.uk>, hereby add my Signed-off-by to this commit: 9fc6989
I, Joe Isaacs <joe.isaacs@live.co.uk>, hereby add my Signed-off-by to this commit: ade2bee
I, Joe Isaacs <joe.isaacs@live.co.uk>, hereby add my Signed-off-by to this commit: 6d08d4f
I, Joe Isaacs <joe.isaacs@live.co.uk>, hereby add my Signed-off-by to this commit: eab26ff
I, Joe Isaacs <joe.isaacs@live.co.uk>, hereby add my Signed-off-by to this commit: 10e6135
I, Joe Isaacs <joe.isaacs@live.co.uk>, hereby add my Signed-off-by to this commit: ae8c638
I, Joe Isaacs <joe.isaacs@live.co.uk>, hereby add my Signed-off-by to this commit: 568fcb5
I, Joe Isaacs <joe.isaacs@live.co.uk>, hereby add my Signed-off-by to this commit: 4bae722
I, Joe Isaacs <joe.isaacs@live.co.uk>, hereby add my Signed-off-by to this commit: f102ab2
I, Joe Isaacs <joe.isaacs@live.co.uk>, hereby add my Signed-off-by to this commit: 121a045
I, Joe Isaacs <joe.isaacs@live.co.uk>, hereby add my Signed-off-by to this commit: 761ff9f
I, Joe Isaacs <joe.isaacs@live.co.uk>, hereby add my Signed-off-by to this commit: d25daa1
I, Joe Isaacs <joe.isaacs@live.co.uk>, hereby add my Signed-off-by to this commit: fbb232a
I, Joe Isaacs <joe.isaacs@live.co.uk>, hereby add my Signed-off-by to this commit: f2822e7
I, Joe Isaacs <joe.isaacs@live.co.uk>, hereby add my Signed-off-by to this commit: 7b0b268
I, Joe Isaacs <joe.isaacs@live.co.uk>, hereby add my Signed-off-by to this commit: d36cd71
I, Joe Isaacs <joe.isaacs@live.co.uk>, hereby add my Signed-off-by to this commit: 58e2012
I, Joe Isaacs <joe.isaacs@live.co.uk>, hereby add my Signed-off-by to this commit: 104297a
I, Joe Isaacs <joe.isaacs@live.co.uk>, hereby add my Signed-off-by to this commit: 8295653
I, Joe Isaacs <joe.isaacs@live.co.uk>, hereby add my Signed-off-by to this commit: ad0749e
I, Joe Isaacs <joe.isaacs@live.co.uk>, hereby add my Signed-off-by to this commit: e199904

Signed-off-by: "Joe Isaacs" <joe.isaacs@live.co.uk>
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@joseph-isaacs