perf(constraints): direct rectangle-to-CSR freeze, sort only when nterm > 1 by FabianHofmann · Pull Request #1015 · PyPSA/linopy

FabianHofmann · 2026-10-06T14:51:00Z

Closes #1009. Part of #972 (phase 2, PR 6). Stacked on #1012: until it merges, this PR also shows its commits; only the last commit belongs here.

Note

The following content was generated by AI.

Changes proposed in this Pull Request

Direct rectangle-to-CSR in Constraint._matrix_export_data. One validity mask over the N-d term array and a single np.flatnonzero pick the entries. Row counts come from np.bincount on the same index buffer, divided in place. No row-masked copies of vars and coeffs, and no reshape copies: a strided (non-contiguous) term array is gathered through .flat when at most a quarter of its cells are entries, and copied once otherwise (_take_flat).
Canonical flag for nterm == 1. A one-term row cannot hold duplicates or unsorted columns, and zeros are already dropped, so CSRConstraint.from_dense sets has_canonical_format instead of calling sum_duplicates/eliminate_zeros. Constraints with more terms still canonicalise.
_is_uniform instead of np.unique for the sign check, in from_dense and on the to_matrix_with_rhs path.
Tests: the dense-freeze matrix gains cancelling terms, an all-masked one-term constraint, empty trailing rows and a strided term array with mostly zero coefficients. The frozen-equality helper ignores terms whose coefficients sum to zero, since freezing drops them (existing policy); the canonical-format check still covers the explicit zeros.

No behaviour change; frozen output byte-identical to before (checked on 14 edge cases: all-masked, zero-size, 0-d, cancelling, zero and NaN coefficients, trailing empty rows, mixed and masked signs, nterm == 0).

Benchmark (benchmark/benchmark_sparse_export.py frozen, 200 buses x 720 snapshots, 2.09M vars, 1.73M cons; best of 3, interleaved runs)

phase	before	after
`cons: balance`	466 ms, peak 1739 MB	182 ms, peak 253 MB
`cons: soc`	44 ms, peak 24 MB	40 ms, peak 21 MB
`expr+cons: p <= pmax`	117 ms, peak 153 MB	89 ms, peak 143 MB
`matrices`	85 ms	82 ms
total	2679 ms	2337 ms

Peak traced memory of to_highspy on the kvl_cycles pattern (mutable constraint with a strided term array, 16800 rows x 300 terms):

severity	before	after
0	148.6 MB	148.3 MB
50	123.2 MB	98.5 MB
100	123.2 MB	12.2 MB

Checklist

AI-generated content is marked (see AGENTS.md).
Code changes are sufficiently documented; i.e. new functions contain docstrings and further explanations may be given in doc.
Unit tests for new features were added (if applicable).
A note for the release notes doc/release_notes.rst of the upcoming release is included.
I consent to the release of this PR's code under the MIT license.

…xport path (#1008)

…w-ups

github-actions · 2026-10-06T15:04:27Z

Build cost — v1 vs legacy

v1 build peak & time relative to legacy, on this commit — not a comparison against master (that is CodSpeed).

peak — v1 / legacy	time — v1 / legacy

Full table (time + peak, mean)

benchmarks/drivers/test_build.py::test_build[basic-n=10]
                  time (s)         peak (KiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.09146 (1.09)   │   15.03 (1.00) 
 (v1)         0.0837 (1.0)   │    15.00 (1.0) 

benchmarks/drivers/test_build.py::test_build[basic-n=250]
                  time (s)         peak (MiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.09823 (1.08)   │   12.04 (1.00) 
 (v1)        0.09057 (1.0)   │    12.04 (1.0) 

benchmarks/drivers/test_build.py::test_build[cumsum-severity=0]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.03952 (1.07)   │   15.20 (1.0) 
 (v1)        0.03703 (1.0)   │   15.20 (1.0) 

benchmarks/drivers/test_build.py::test_build[cumsum-severity=100]
                  time (s)        peak (MiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.05725 (1.04)   │   44.93 (1.0) 
 (v1)        0.05501 (1.0)   │   44.93 (1.0) 

benchmarks/drivers/test_build.py::test_build[cumsum-severity=50]
                  time (s)        peak (MiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.04355 (1.07)   │   11.51 (1.0) 
 (v1)        0.04054 (1.0)   │   11.51 (1.0) 

benchmarks/drivers/test_build.py::test_build[expression_arithmetic-n=10]
                 time (s)         peak (KiB) 
 name                mean   │           mean 
─────────────────────────────────────────────
 (legacy)   0.1025 (1.04)   │   24.34 (1.06) 
 (v1)       0.09829 (1.0)   │    23.04 (1.0) 

benchmarks/drivers/test_build.py::test_build[expression_arithmetic-n=250]
                 time (s)         peak (MiB) 
 name                mean   │           mean 
─────────────────────────────────────────────
 (legacy)   0.1126 (1.07)   │   16.12 (1.00) 
 (v1)        0.1057 (1.0)   │    16.12 (1.0) 

benchmarks/drivers/test_build.py::test_build[knapsack-n=10000]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.02461 (1.06)   │   752.18 (1.10) 
 (v1)        0.02323 (1.0)   │    685.15 (1.0) 

benchmarks/drivers/test_build.py::test_build[knapsack-n=100]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.02416 (1.08)   │   3.12 (1.33) 
 (v1)        0.02247 (1.0)   │    2.34 (1.0) 

benchmarks/drivers/test_build.py::test_build[kvl_cycles-severity=0]
                  time (s)          peak (MiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.06549 (1.18)   │   126.16 (1.44) 
 (v1)        0.05535 (1.0)   │     87.71 (1.0) 

benchmarks/drivers/test_build.py::test_build[kvl_cycles-severity=100]
                  time (s)          peak (MiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.06431 (1.18)   │   126.16 (1.44) 
 (v1)        0.05444 (1.0)   │     87.71 (1.0) 

benchmarks/drivers/test_build.py::test_build[kvl_cycles-severity=50]
                  time (s)          peak (MiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.06492 (1.18)   │   126.16 (1.44) 
 (v1)        0.05509 (1.0)   │     87.71 (1.0) 

benchmarks/drivers/test_build.py::test_build[masked-n=100]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.05714 (1.02)   │    715.12 (1.0) 
 (v1)        0.05609 (1.0)   │   787.73 (1.10) 

benchmarks/drivers/test_build.py::test_build[masked-n=10]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.05602 (1.14)   │   4.54 (1.27) 
 (v1)        0.04927 (1.0)   │    3.57 (1.0) 

benchmarks/drivers/test_build.py::test_build[merge_balance-severity=0]
                 time (s)          peak (KiB) 
 name                mean   │            mean 
──────────────────────────────────────────────
 (legacy)   0.3886 (1.04)   │   704.12 (1.09) 
 (v1)        0.3745 (1.0)   │    643.85 (1.0) 

benchmarks/drivers/test_build.py::test_build[merge_balance-severity=100]
                 time (s)        peak (MiB) 
 name                mean   │          mean 
────────────────────────────────────────────
 (legacy)   0.4055 (1.03)   │   18.34 (1.0) 
 (v1)        0.3938 (1.0)   │   18.34 (1.0) 

benchmarks/drivers/test_build.py::test_build[merge_balance-severity=50]
                 time (s)       peak (MiB) 
 name                mean   │         mean 
───────────────────────────────────────────
 (legacy)   0.3986 (1.02)   │   9.54 (1.0) 
 (v1)        0.3916 (1.0)   │   9.54 (1.0) 

benchmarks/drivers/test_build.py::test_build[milp-n=10]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.07906 (1.10)   │   3.77 (1.12) 
 (v1)        0.07156 (1.0)   │    3.37 (1.0) 

benchmarks/drivers/test_build.py::test_build[milp-n=50]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.07853 (1.10)   │   216.59 (1.10) 
 (v1)        0.07161 (1.0)   │    196.23 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance-severity=0]
                  time (s)         peak (KiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.03948 (1.08)   │   938.49 (1.0) 
 (v1)        0.03661 (1.0)   │   938.49 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance-severity=100]
                 time (s)       peak (MiB) 
 name                mean   │         mean 
───────────────────────────────────────────
 (legacy)   0.0406 (1.07)   │   9.66 (1.0) 
 (v1)       0.03812 (1.0)   │   9.66 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance-severity=50]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)   0.04088 (1.08)   │   5.32 (1.0) 
 (v1)        0.03788 (1.0)   │   5.32 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance_sparse-severity=0]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)   0.02013 (1.01)   │   1.28 (1.0) 
 (v1)        0.01997 (1.0)   │   1.28 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance_sparse-severity=100]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)   0.02059 (1.04)   │   1.28 (1.0) 
 (v1)        0.01981 (1.0)   │   1.28 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance_sparse-severity=50]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)   0.02011 (1.00)   │   1.28 (1.0) 
 (v1)        0.02009 (1.0)   │   1.28 (1.0) 

benchmarks/drivers/test_build.py::test_build[piecewise-n=1000]
                time (s)          peak (KiB) 
 name               mean   │            mean 
─────────────────────────────────────────────
 (legacy)    0.19 (1.04)   │   946.85 (1.06) 
 (v1)       0.1822 (1.0)   │    891.54 (1.0) 

benchmarks/drivers/test_build.py::test_build[piecewise-n=10]
                 time (s)         peak (KiB) 
 name                mean   │           mean 
─────────────────────────────────────────────
 (legacy)   0.1928 (1.07)   │   12.01 (1.00) 
 (v1)        0.1801 (1.0)   │    11.99 (1.0) 

benchmarks/drivers/test_build.py::test_build[qp-n=1000]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.04807 (1.04)   │   147.70 (1.06) 
 (v1)        0.04604 (1.0)   │    139.87 (1.0) 

benchmarks/drivers/test_build.py::test_build[qp-n=10]
                 time (s)        peak (KiB) 
 name                mean   │          mean 
────────────────────────────────────────────
 (legacy)   0.0481 (1.06)   │   2.60 (1.09) 
 (v1)       0.04554 (1.0)   │    2.38 (1.0) 

benchmarks/drivers/test_build.py::test_build[rolling-severity=0]
                 time (s)          peak (KiB) 
 name                mean   │            mean 
──────────────────────────────────────────────
 (legacy)   0.0411 (1.08)   │   696.75 (1.03) 
 (v1)       0.03798 (1.0)   │    673.70 (1.0) 

benchmarks/drivers/test_build.py::test_build[rolling-severity=100]
                  time (s)         peak (MiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.09005 (1.03)   │   137.97 (1.0) 
 (v1)        0.08703 (1.0)   │   137.97 (1.0) 

benchmarks/drivers/test_build.py::test_build[rolling-severity=50]
                  time (s)        peak (MiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.06311 (1.09)   │   69.22 (1.0) 
 (v1)        0.05768 (1.0)   │   69.22 (1.0) 

benchmarks/drivers/test_build.py::test_build[sos-n=1000]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.04663 (1.07)   │   402.33 (1.00) 
 (v1)        0.04359 (1.0)   │    402.30 (1.0) 

benchmarks/drivers/test_build.py::test_build[sos-n=10]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.04597 (1.10)   │   3.19 (1.19) 
 (v1)        0.04176 (1.0)   │    2.69 (1.0) 

benchmarks/drivers/test_build.py::test_build[sparse_network-n=10]
                  time (s)         peak (KiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.05004 (1.05)   │   29.00 (1.54) 
 (v1)        0.04749 (1.0)   │    18.84 (1.0) 

benchmarks/drivers/test_build.py::test_build[sparse_network-n=250]
                  time (s)         peak (MiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.06014 (1.07)   │   37.95 (1.43) 
 (v1)        0.05605 (1.0)   │    26.51 (1.0) 

benchmarks/drivers/test_build.py::test_build[storage-n=10]
                 time (s)          peak (KiB) 
 name                mean   │            mean 
──────────────────────────────────────────────
 (legacy)   0.09486 (1.0)   │    410.93 (1.0) 
 (v1)       0.1007 (1.06)   │   427.84 (1.04) 

benchmarks/drivers/test_build.py::test_build[storage-n=250]
                time (s)         peak (MiB) 
 name               mean   │           mean 
────────────────────────────────────────────
 (legacy)   0.1015 (1.0)   │     9.94 (1.0) 
 (v1)        0.11 (1.08)   │   10.22 (1.03)

📊 Interactive plots + CSV: download the semantics-report-v1-vs-legacy artifact from this run.

_{Report-only · not a gate · refreshed on every push · obsolete once legacy is dropped.}

codspeed · 2026-10-06T15:06:05Z

Merging this PR will improve performance by ×3.6

⚡ 12 improved benchmarks
✅ 169 untouched benchmarks
⏩ 181 skipped benchmarks¹

Performance Changes

	Benchmark	`BASE`	`HEAD`	Efficiency
⚡	`test_to_solver[highs-kvl_cycles-severity=100]`	117 MB	11 MB	×11
⚡	`test_to_solver[gurobi-kvl_cycles-severity=100]`	117.3 MB	15.7 MB	×7.5
⚡	`test_to_solver[highs-sparse_network-n=250]`	21.9 MB	3.4 MB	×6.5
⚡	`test_to_solver[highs-nodal_balance-severity=100]`	11.6 MB	2.1 MB	×5.6
⚡	`test_to_solver[gurobi-sparse_network-n=250]`	22.2 MB	4.7 MB	×4.7
⚡	`test_to_solver[highs-merge_balance-severity=100]`	18.6 MB	4.9 MB	×3.8
⚡	`test_to_solver[highs-nodal_balance-severity=50]`	6.2 MB	2.1 MB	×3
⚡	`test_to_solver[highs-merge_balance-severity=50]`	9.7 MB	3.3 MB	×2.9
⚡	`test_to_solver[gurobi-nodal_balance-severity=100]`	11.8 MB	4.3 MB	×2.7
⚡	`test_to_solver[gurobi-merge_balance-severity=100]`	18.9 MB	10 MB	+87.94%
⚡	`test_to_solver[gurobi-nodal_balance-severity=50]`	6.4 MB	4.3 MB	+46.68%
⚡	`test_to_solver[gurobi-merge_balance-severity=50]`	9.9 MB	6.9 MB	+43.32%

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.

_{Comparing perf/fast-freeze-1009 (f03954b) with master (f665a26)}

181 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

… once per caller Model.matrices returns a fresh accessor again. Dualization binds it once and expression solutions map labels directly, without building the matrices.

…model

…rm > 1 (#1009) Single mask + flatnonzero export without reshape copies of strided term arrays, canonical flag for one-term constraints, uniform-sign check without np.unique. Tests cover cancelling, all-masked, empty-trailing-row and strided sparse freezes.

FabianHofmann added 3 commits October 6, 2026 12:38

perf(matrices): cache Model.matrices for frozen models and trim the e…

f21f603

…xport path (#1008)

fix(xpress): stop editing cached matrix bounds in place; review follo…

f9058a6

…w-ups

docs: note the shared Model.matrices contract in the release notes

2c9dee4

FabianHofmann changed the base branch from perf/matrices-cache-1008 to master October 6, 2026 15:00

FabianHofmann closed this Oct 6, 2026

FabianHofmann reopened this Oct 6, 2026

FabianHofmann force-pushed the perf/fast-freeze-1009 branch 2 times, most recently from 54df852 to 249ecce Compare October 6, 2026 16:20

fix(matrices): key the cache on the objective expression data

fbf7c74

FabianHofmann force-pushed the perf/fast-freeze-1009 branch from 249ecce to 4991718 Compare October 6, 2026 19:12

FabianHofmann added 2 commits October 6, 2026 21:55

refactor(matrices): drop the Model.matrices cache, build the matrices…

087292b

… once per caller Model.matrices returns a fresh accessor again. Dualization binds it once and expression solutions map labels directly, without building the matrices.

fix(expressions): raise when a solution maps labels missing from the …

d757470

…model

FabianHofmann force-pushed the perf/fast-freeze-1009 branch from 4991718 to c4da3a6 Compare October 6, 2026 20:17

docs: restate the matrices build saving without the cache

89560f3

FabianHofmann force-pushed the perf/fast-freeze-1009 branch from c4da3a6 to 8f25086 Compare October 6, 2026 20:19

FabianHofmann added 2 commits October 6, 2026 22:52

test(matrices): cover the solution read-back of the accessor

eeb63e7

FabianHofmann force-pushed the perf/fast-freeze-1009 branch from 8f25086 to f03954b Compare October 6, 2026 20:52

Navigation Menu

Repository navigation

perf(constraints): direct rectangle-to-CSR freeze, sort only when nterm > 1 - #1015

perf(constraints): direct rectangle-to-CSR freeze, sort only when nterm > 1#1015
FabianHofmann wants to merge 9 commits into
masterfrom
perf/fast-freeze-1009

FabianHofmann commented Oct 6, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 6, 2026 •

edited

Loading

Uh oh!

codspeed Bot commented Oct 6, 2026 •

edited

Loading

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

FabianHofmann commented Oct 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes proposed in this Pull Request

Checklist

Uh oh!

github-actions Bot commented Oct 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Build cost — v1 vs legacy

Uh oh!

codspeed Bot commented Oct 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will improve performance by ×3.6

Performance Changes

Footnotes

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

FabianHofmann commented Oct 6, 2026 •

edited

Loading

github-actions Bot commented Oct 6, 2026 •

edited

Loading

codspeed Bot commented Oct 6, 2026 •

edited

Loading