perf(csr): one-pass n-ary merge, row gather for permuting reindex by FabianHofmann · Pull Request #1014 · PyPSA/linopy

FabianHofmann · 2026-10-06T14:00:43Z

Closes #1010. Part of #972 (phase 2, PR 7).

Note

The following content was generated by AI.

Changes proposed in this Pull Request

One-pass n-ary merge. CSRLinearExpression.added(self, *others) now takes any number of operands on the same grid and builds the sum in a single COO pass. _try_csr_merge calls it once instead of folding pairwise. Explicit zero coefficients and absent cells keep their current behaviour. Auxiliary coordinates are united, earlier operands take precedence, and conflicting ones still raise.
Row gather for permuting reindex. When reindexed only permutes the existing cells (no dropped and no new labels), it gathers the CSR rows and constants directly instead of going through COO.
Index dtype in aggregated. The aggregated row indices are cast to the store's index dtype before the COO to CSR conversion.

The added(self, *others) change is byte-identical to the one in open PR #992, so the two PRs merge without conflict.

Chained + is unchanged, because + is binary.

Benchmark (scratch script dev-scripts/bench_1010.py, before → after)

case	before	after
merge of 3 sparse operands	118 ms	84 ms
transposed merge	58 ms	45 ms
permuting reindex	16 ms	4 ms
chained `+`	unchanged	unchanged

Checklist

AI-generated content is marked (see AGENTS.md).
Code changes are sufficiently documented; i.e. new functions contain docstrings and further explanations may be given in doc.
Unit tests for new features were added (if applicable).
A note for the release notes doc/release_notes.rst of the upcoming release is included.
I consent to the release of this PR's code under the MIT license.

) Make CSRLinearExpression.added n-ary in one COO pass (as in #992), gather rows directly when reindexed only permutes cells, and index aggregated rows in the store's index dtype.

github-actions · 2026-10-06T14:04:52Z

Build cost — v1 vs legacy

v1 build peak & time relative to legacy, on this commit — not a comparison against master (that is CodSpeed).

peak — v1 / legacy	time — v1 / legacy

Full table (time + peak, mean)

benchmarks/drivers/test_build.py::test_build[basic-n=10]
                  time (s)         peak (KiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.08834 (1.01)   │   15.03 (1.00) 
 (v1)        0.08758 (1.0)   │    15.00 (1.0) 

benchmarks/drivers/test_build.py::test_build[basic-n=250]
                  time (s)         peak (MiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.09443 (1.02)   │   12.04 (1.00) 
 (v1)        0.09261 (1.0)   │    12.04 (1.0) 

benchmarks/drivers/test_build.py::test_build[cumsum-severity=0]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)     0.0383 (1.0)   │   15.20 (1.0) 
 (v1)       0.03869 (1.01)   │   15.20 (1.0) 

benchmarks/drivers/test_build.py::test_build[cumsum-severity=100]
                  time (s)        peak (MiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.05464 (1.08)   │   44.93 (1.0) 
 (v1)        0.05062 (1.0)   │   44.93 (1.0) 

benchmarks/drivers/test_build.py::test_build[cumsum-severity=50]
                  time (s)        peak (MiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.04195 (1.02)   │   11.51 (1.0) 
 (v1)         0.0413 (1.0)   │   11.51 (1.0) 

benchmarks/drivers/test_build.py::test_build[expression_arithmetic-n=10]
                 time (s)         peak (KiB) 
 name                mean   │           mean 
─────────────────────────────────────────────
 (legacy)   0.09881 (1.0)   │   24.34 (1.06) 
 (v1)        0.102 (1.03)   │    23.04 (1.0) 

benchmarks/drivers/test_build.py::test_build[expression_arithmetic-n=250]
                 time (s)         peak (MiB) 
 name                mean   │           mean 
─────────────────────────────────────────────
 (legacy)   0.1099 (1.02)   │   16.12 (1.00) 
 (v1)        0.1076 (1.0)   │    16.12 (1.0) 

benchmarks/drivers/test_build.py::test_build[knapsack-n=10000]
                 time (s)          peak (KiB) 
 name                mean   │            mean 
──────────────────────────────────────────────
 (legacy)   0.02388 (1.0)   │   752.18 (1.10) 
 (v1)       0.0241 (1.01)   │    685.15 (1.0) 

benchmarks/drivers/test_build.py::test_build[knapsack-n=100]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)    0.02339 (1.0)   │   3.12 (1.33) 
 (v1)       0.02381 (1.02)   │    2.34 (1.0) 

benchmarks/drivers/test_build.py::test_build[kvl_cycles-severity=0]
                  time (s)          peak (MiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.06413 (1.22)   │   126.16 (1.44) 
 (v1)        0.05271 (1.0)   │     87.71 (1.0) 

benchmarks/drivers/test_build.py::test_build[kvl_cycles-severity=100]
                  time (s)          peak (MiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.06299 (1.21)   │   126.16 (1.44) 
 (v1)        0.05194 (1.0)   │     87.71 (1.0) 

benchmarks/drivers/test_build.py::test_build[kvl_cycles-severity=50]
                  time (s)          peak (MiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.06332 (1.21)   │   126.16 (1.44) 
 (v1)        0.05244 (1.0)   │     87.71 (1.0) 

benchmarks/drivers/test_build.py::test_build[masked-n=100]
                 time (s)          peak (KiB) 
 name                mean   │            mean 
──────────────────────────────────────────────
 (legacy)   0.05484 (1.0)   │    715.12 (1.0) 
 (v1)       0.0575 (1.05)   │   787.73 (1.10) 

benchmarks/drivers/test_build.py::test_build[masked-n=10]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.05341 (1.02)   │   4.54 (1.27) 
 (v1)         0.0522 (1.0)   │    3.57 (1.0) 

benchmarks/drivers/test_build.py::test_build[merge_balance-severity=0]
                 time (s)          peak (KiB) 
 name                mean   │            mean 
──────────────────────────────────────────────
 (legacy)    0.3752 (1.0)   │   704.12 (1.09) 
 (v1)       0.3873 (1.03)   │    643.85 (1.0) 

benchmarks/drivers/test_build.py::test_build[merge_balance-severity=100]
                time (s)        peak (MiB) 
 name               mean   │          mean 
───────────────────────────────────────────
 (legacy)   0.392 (1.00)   │   18.34 (1.0) 
 (v1)       0.3914 (1.0)   │   18.34 (1.0) 

benchmarks/drivers/test_build.py::test_build[merge_balance-severity=50]
                 time (s)       peak (MiB) 
 name                mean   │         mean 
───────────────────────────────────────────
 (legacy)   0.3897 (1.04)   │   9.54 (1.0) 
 (v1)        0.3754 (1.0)   │   9.54 (1.0) 

benchmarks/drivers/test_build.py::test_build[milp-n=10]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.07532 (1.01)   │   3.77 (1.12) 
 (v1)        0.07489 (1.0)   │    3.37 (1.0) 

benchmarks/drivers/test_build.py::test_build[milp-n=50]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.07553 (1.02)   │   216.59 (1.10) 
 (v1)        0.07404 (1.0)   │    196.23 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance-severity=0]
                  time (s)         peak (KiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.03821 (1.03)   │   938.49 (1.0) 
 (v1)        0.03707 (1.0)   │   938.49 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance-severity=100]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)   0.03936 (1.08)   │   9.66 (1.0) 
 (v1)         0.0363 (1.0)   │   9.66 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance-severity=50]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)   0.03885 (1.04)   │   5.32 (1.0) 
 (v1)         0.0374 (1.0)   │   5.32 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance_sparse-severity=0]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)    0.01907 (1.0)   │   1.28 (1.0) 
 (v1)       0.01998 (1.05)   │   1.28 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance_sparse-severity=100]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)    0.01906 (1.0)   │   1.28 (1.0) 
 (v1)       0.02004 (1.05)   │   1.28 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance_sparse-severity=50]
                 time (s)       peak (MiB) 
 name                mean   │         mean 
───────────────────────────────────────────
 (legacy)   0.01918 (1.0)   │   1.28 (1.0) 
 (v1)       0.0196 (1.02)   │   1.28 (1.0) 

benchmarks/drivers/test_build.py::test_build[piecewise-n=1000]
                 time (s)          peak (KiB) 
 name                mean   │            mean 
──────────────────────────────────────────────
 (legacy)    0.1903 (1.0)   │   946.85 (1.06) 
 (v1)       0.1933 (1.02)   │    891.54 (1.0) 

benchmarks/drivers/test_build.py::test_build[piecewise-n=10]
                time (s)         peak (KiB) 
 name               mean   │           mean 
────────────────────────────────────────────
 (legacy)   0.1841 (1.0)   │   12.01 (1.00) 
 (v1)       0.185 (1.00)   │    11.99 (1.0) 

benchmarks/drivers/test_build.py::test_build[qp-n=1000]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)    0.04731 (1.0)   │   147.70 (1.06) 
 (v1)       0.04801 (1.01)   │    139.87 (1.0) 

benchmarks/drivers/test_build.py::test_build[qp-n=10]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)    0.04712 (1.0)   │   2.60 (1.09) 
 (v1)       0.04827 (1.02)   │    2.38 (1.0) 

benchmarks/drivers/test_build.py::test_build[rolling-severity=0]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.03984 (1.02)   │   696.75 (1.03) 
 (v1)        0.03889 (1.0)   │    673.70 (1.0) 

benchmarks/drivers/test_build.py::test_build[rolling-severity=100]
                  time (s)         peak (MiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)    0.08325 (1.0)   │   137.97 (1.0) 
 (v1)       0.08609 (1.03)   │   137.97 (1.0) 

benchmarks/drivers/test_build.py::test_build[rolling-severity=50]
                  time (s)        peak (MiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.06165 (1.08)   │   69.22 (1.0) 
 (v1)        0.05717 (1.0)   │   69.22 (1.0) 

benchmarks/drivers/test_build.py::test_build[sos-n=1000]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.04535 (1.03)   │   402.33 (1.00) 
 (v1)        0.04403 (1.0)   │    402.30 (1.0) 

benchmarks/drivers/test_build.py::test_build[sos-n=10]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.04495 (1.03)   │   3.19 (1.19) 
 (v1)        0.04353 (1.0)   │    2.69 (1.0) 

benchmarks/drivers/test_build.py::test_build[sparse_network-n=10]
                  time (s)         peak (KiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)    0.04928 (1.0)   │   29.00 (1.54) 
 (v1)       0.05029 (1.02)   │    18.84 (1.0) 

benchmarks/drivers/test_build.py::test_build[sparse_network-n=250]
                  time (s)         peak (MiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.05891 (1.06)   │   37.95 (1.43) 
 (v1)        0.05537 (1.0)   │    26.51 (1.0) 

benchmarks/drivers/test_build.py::test_build[storage-n=10]
                 time (s)          peak (KiB) 
 name                mean   │            mean 
──────────────────────────────────────────────
 (legacy)   0.09298 (1.0)   │    410.93 (1.0) 
 (v1)       0.1039 (1.12)   │   427.84 (1.04) 

benchmarks/drivers/test_build.py::test_build[storage-n=250]
                 time (s)         peak (MiB) 
 name                mean   │           mean 
─────────────────────────────────────────────
 (legacy)   0.09901 (1.0)   │     9.94 (1.0) 
 (v1)       0.1062 (1.07)   │   10.22 (1.03)

📊 Interactive plots + CSV: download the semantics-report-v1-vs-legacy artifact from this run.

_{Report-only · not a gate · refreshed on every push · obsolete once legacy is dropped.}

codspeed · 2026-10-06T14:06:42Z

Merging this PR will not alter performance

✅ 181 untouched benchmarks
⏩ 181 skipped benchmarks¹

_{Comparing perf/csr-kernel-1010 (a50c11b) with master (f665a26)}

181 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

FabianHofmann added 2 commits October 6, 2026 15:42

perf(csr): one-pass n-ary added, row gather for permuting reindex (#1010

9b43746

) Make CSRLinearExpression.added n-ary in one COO pass (as in #992), gather rows directly when reindexed only permutes cells, and index aggregated rows in the store's index dtype.

test: reuse densified helper for mypy; add release note (#1010)

a50c11b

This was referenced Oct 6, 2026

perf(csr): sparse shift/roll/diff and merge(dim=...); densify notice on by default in sparse models #1016

Open

docs: document the sparse expression path #1017

Open

Navigation Menu

Repository navigation

perf(csr): one-pass n-ary merge, row gather for permuting reindex - #1014

perf(csr): one-pass n-ary merge, row gather for permuting reindex#1014
FabianHofmann wants to merge 2 commits into
masterfrom
perf/csr-kernel-1010

FabianHofmann commented Oct 6, 2026

Uh oh!

github-actions Bot commented Oct 6, 2026

Uh oh!

codspeed Bot commented Oct 6, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

FabianHofmann commented Oct 6, 2026

Changes proposed in this Pull Request

Checklist

Uh oh!

github-actions Bot commented Oct 6, 2026

Build cost — v1 vs legacy

Uh oh!

codspeed Bot commented Oct 6, 2026

Merging this PR will not alter performance

Footnotes

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant