Skip to content

[core] Keep the winning cell by reference in pick-value aggregators - #1338

Open
JeonDaehong wants to merge 1 commit into
apache:mainfrom
JeonDaehong:perf-pick-agg-ref
Open

JeonDaehong wants to merge 1 commit into
apache:mainfrom
JeonDaehong:perf-pick-agg-ref

Conversation

@JeonDaehong

Copy link
Copy Markdown
Contributor

Purpose

Linked issue: part of #1336

last_value, first_value, last_non_null_value, first_non_null_value and primary-key hold the current winner as a one-row slice and take a new slice for every input row. Columns without an aggregate function default to last_non_null_value, so almost every aggregation table pays this per row.

Brief change log

  • FieldAggregator::agg_array(&ArrayRef, row) defaults to agg. The pick aggregators override it to keep the input array and row index, and result() slices once.
  • merge_ordered and the partial-update aggregator path pass the column as an ArrayRef. Columns without an aggregator keep the existing path.

PK table, 2M keys written in 3 overlapping rounds, sum on one column, full read (release, in-memory FileIO): 2,105–2,135 → 1,827–1,856 ms, measured in one binary with the old and new call paths switched by an env var, 3 alternating rounds. Allocations per merged key go from 10.2 to 8.2. Deduplicate and partial-update are unchanged.

Tests

  • agg_array_matches_agg_for_every_policy: each pick policy gives the same result through agg_array and agg, with NULLs among the inputs.
  • agg_array_winner_feeds_reversed_and_delete_paths: agg_reversed and replace_with_delete after agg_array.

Ran the ci.yml steps locally (fmt, clippy, msrv, unit, rest-server, integration, datafusion, python, go).

API and Format

None. agg_array is crate-internal.

Documentation

None.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant