You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Improvement] Aggregation and partial-update merges build a one-row batch per key #1336
Reading a primary-key table with 2M keys written in 3 overlapping rounds takes about 1.97 s with merge-engine=aggregation (sum) and 1.42 s with partial-update, against 0.44 s for deduplicate (release build, in-memory FileIO, medians of 8 reads). Writes show the same gap since the write buffer uses the same merge. In the profile, malloc, GenericByteArray::slice and Arc<dyn Array> drops are near the top: merge_ordered and merge_unreduced return a one-row RecordBatch per key, built from slice(row, 1) for key columns and a one-row array from each aggregator's result().
Batching those one-row batches before interleave made no difference, so the cost is in building them. One option is letting aggregators append their result to a column builder, so a merged key adds a row to per-column builders instead of creating arrays. That touches every aggregator, so I wanted to check whether this direction makes sense before opening a PR.
Opened #1338 for the per-row slice in the pick aggregators, about 13% of the aggregation read here. The rest is building a one-row batch per key, which still needs the builder change, so I'll wait for your view on that direction.
Activity
JeonDaehong commentedon Nov 7, 2026
Opened #1338 for the per-row slice in the pick aggregators, about 13% of the aggregation read here. The rest is building a one-row batch per key, which still needs the builder change, so I'll wait for your view on that direction.