Skip to content

[Improvement] Aggregation and partial-update merges build a one-row batch per key #1336

Open

Description

@JeonDaehong

Reading a primary-key table with 2M keys written in 3 overlapping rounds takes about 1.97 s with merge-engine=aggregation (sum) and 1.42 s with partial-update, against 0.44 s for deduplicate (release build, in-memory FileIO, medians of 8 reads). Writes show the same gap since the write buffer uses the same merge. In the profile, malloc, GenericByteArray::slice and Arc<dyn Array> drops are near the top: merge_ordered and merge_unreduced return a one-row RecordBatch per key, built from slice(row, 1) for key columns and a one-row array from each aggregator's result().

Batching those one-row batches before interleave made no difference, so the cost is in building them. One option is letting aggregators append their result to a column builder, so a merged key adds a row to per-column builders instead of creating arrays. That touches every aggregator, so I wanted to check whether this direction makes sense before opening a PR.

Activity

  1. JeonDaehong commented on Nov 7, 2026

    @JeonDaehong
    ContributorAuthor

    Opened #1338 for the per-row slice in the pick aggregators, about 13% of the aggregation read here. The rest is building a one-row batch per key, which still needs the builder change, so I'll wait for your view on that direction.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions