perf(matrices): trim the Model.matrices build and drop redundant builds by FabianHofmann · Pull Request #1012 · PyPSA/linopy

FabianHofmann · 2026-10-06T10:38:53Z

Closes #1008. Part of #972 (phase 2, PR 5).

Note

The following content was generated by AI.

Changes proposed in this Pull Request

Model.matrices still builds a fresh MatrixAccessor on every access. This PR makes the build cheaper and removes callers that built it more than once.

No label-sized scaling lookups. Variable scaling is collected in the existing per-variable loop. Row scaling is read from each constraint's active rows (CSRConstraint._scaling directly).
Prune explicit zeros per block. eliminate_zeros now runs only on blocks of mutable constraints, because freezing already drops explicit zeros. Indicator blocks are always frozen.
Xpress no longer edits M.lb/M.ub in place (np.place → np.where).
Fewer matrix builds.
- Dualization binds M = m.matrices once instead of building it four times.
- expr.solution maps variable labels to solution values directly and no longer builds the whole accessor. An expression that references a removed variable still raises a KeyError.
Tracked benchmark benchmark/benchmark_sparse_export.py (sparse, dense, frozen modes), so the next phase-2 PRs can show before and after numbers. The matrices, to_highspy and to_file(lp) phases each build the matrices, by design.

An earlier version of this PR also cached Model.matrices for frozen models. The cache was dropped after @coroa's review.

Benchmark (200 buses x 720 snapshots, 2.09M vars, 1.73M cons; best of 3, interleaved runs against master)

phase	sparse before → after	frozen dense before → after	dense before → after
`matrices`	113 → 96 ms	125 → 91 ms	718 → 734 ms
`to_highspy`	475 → 437 ms	473 → 444 ms	1008 → 1020 ms
`to_file(lp)`	665 → 705 ms	721 → 722 ms	1810 → 1896 ms
total	1.96 → 1.93 s	3.22 → 3.01 s	4.83 → 4.89 s

Peak traced memory per phase is unchanged. The dense mode and to_file(lp) differences are within run-to-run noise.

Checklist

AI-generated content is marked (see AGENTS.md).
Code changes are sufficiently documented; i.e. new functions contain docstrings and further explanations may be given in doc.
Unit tests for new features were added (if applicable).
A note for the release notes doc/release_notes.rst of the upcoming release is included.
I consent to the release of this PR's code under the MIT license.

…xport path (#1008)

github-actions · 2026-10-06T10:42:39Z

Build cost — v1 vs legacy

v1 build peak & time relative to legacy, on this commit — not a comparison against master (that is CodSpeed).

peak — v1 / legacy	time — v1 / legacy

Full table (time + peak, mean)

benchmarks/drivers/test_build.py::test_build[basic-n=10]
                  time (s)         peak (KiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.07871 (1.12)   │   15.03 (1.00) 
 (v1)        0.07055 (1.0)   │    15.00 (1.0) 

benchmarks/drivers/test_build.py::test_build[basic-n=250]
                  time (s)         peak (MiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.08685 (1.09)   │   12.04 (1.00) 
 (v1)        0.07948 (1.0)   │    12.04 (1.0) 

benchmarks/drivers/test_build.py::test_build[cumsum-severity=0]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.03297 (1.07)   │   15.20 (1.0) 
 (v1)        0.03088 (1.0)   │   15.20 (1.0) 

benchmarks/drivers/test_build.py::test_build[cumsum-severity=100]
                  time (s)        peak (MiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.05431 (1.08)   │   44.93 (1.0) 
 (v1)         0.0505 (1.0)   │   44.93 (1.0) 

benchmarks/drivers/test_build.py::test_build[cumsum-severity=50]
                  time (s)        peak (MiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.03774 (1.05)   │   11.51 (1.0) 
 (v1)        0.03597 (1.0)   │   11.51 (1.0) 

benchmarks/drivers/test_build.py::test_build[expression_arithmetic-n=10]
                  time (s)         peak (KiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.08569 (1.06)   │   24.34 (1.06) 
 (v1)        0.08096 (1.0)   │    23.04 (1.0) 

benchmarks/drivers/test_build.py::test_build[expression_arithmetic-n=250]
                  time (s)         peak (MiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.09967 (1.06)   │   16.12 (1.00) 
 (v1)        0.09375 (1.0)   │    16.12 (1.0) 

benchmarks/drivers/test_build.py::test_build[knapsack-n=10000]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.02098 (1.07)   │   752.18 (1.10) 
 (v1)        0.01956 (1.0)   │    685.15 (1.0) 

benchmarks/drivers/test_build.py::test_build[knapsack-n=100]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.02038 (1.06)   │   3.12 (1.33) 
 (v1)        0.01927 (1.0)   │    2.34 (1.0) 

benchmarks/drivers/test_build.py::test_build[kvl_cycles-severity=0]
                  time (s)          peak (MiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.07043 (1.24)   │   126.16 (1.44) 
 (v1)        0.05657 (1.0)   │     87.71 (1.0) 

benchmarks/drivers/test_build.py::test_build[kvl_cycles-severity=100]
                  time (s)          peak (MiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.07088 (1.28)   │   126.16 (1.44) 
 (v1)        0.05532 (1.0)   │     87.71 (1.0) 

benchmarks/drivers/test_build.py::test_build[kvl_cycles-severity=50]
                  time (s)          peak (MiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.07046 (1.27)   │   126.16 (1.44) 
 (v1)        0.05563 (1.0)   │     87.71 (1.0) 

benchmarks/drivers/test_build.py::test_build[masked-n=100]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.04704 (1.01)   │    715.12 (1.0) 
 (v1)        0.04644 (1.0)   │   787.73 (1.10) 

benchmarks/drivers/test_build.py::test_build[masked-n=10]
                 time (s)        peak (KiB) 
 name                mean   │          mean 
────────────────────────────────────────────
 (legacy)   0.0455 (1.09)   │   4.54 (1.27) 
 (v1)       0.04189 (1.0)   │    3.57 (1.0) 

benchmarks/drivers/test_build.py::test_build[merge_balance-severity=0]
                 time (s)          peak (KiB) 
 name                mean   │            mean 
──────────────────────────────────────────────
 (legacy)   0.3311 (1.07)   │   704.12 (1.09) 
 (v1)        0.3097 (1.0)   │    643.85 (1.0) 

benchmarks/drivers/test_build.py::test_build[merge_balance-severity=100]
                 time (s)        peak (MiB) 
 name                mean   │          mean 
────────────────────────────────────────────
 (legacy)   0.3427 (1.05)   │   18.34 (1.0) 
 (v1)        0.3256 (1.0)   │   18.34 (1.0) 

benchmarks/drivers/test_build.py::test_build[merge_balance-severity=50]
                 time (s)       peak (MiB) 
 name                mean   │         mean 
───────────────────────────────────────────
 (legacy)   0.3387 (1.06)   │   9.54 (1.0) 
 (v1)         0.321 (1.0)   │   9.54 (1.0) 

benchmarks/drivers/test_build.py::test_build[milp-n=10]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.06485 (1.08)   │   3.77 (1.12) 
 (v1)        0.06025 (1.0)   │    3.37 (1.0) 

benchmarks/drivers/test_build.py::test_build[milp-n=50]
                 time (s)          peak (KiB) 
 name                mean   │            mean 
──────────────────────────────────────────────
 (legacy)    0.065 (1.08)   │   216.59 (1.10) 
 (v1)       0.06011 (1.0)   │    196.23 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance-severity=0]
                  time (s)         peak (KiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.03346 (1.09)   │   938.49 (1.0) 
 (v1)        0.03064 (1.0)   │   938.49 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance-severity=100]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)   0.03537 (1.09)   │   9.66 (1.0) 
 (v1)        0.03248 (1.0)   │   9.66 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance-severity=50]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)   0.03456 (1.09)   │   5.32 (1.0) 
 (v1)        0.03181 (1.0)   │   5.32 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance_sparse-severity=0]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)   0.01753 (1.02)   │   1.28 (1.0) 
 (v1)        0.01713 (1.0)   │   1.28 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance_sparse-severity=100]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)   0.01744 (1.03)   │   1.28 (1.0) 
 (v1)        0.01697 (1.0)   │   1.28 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance_sparse-severity=50]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)   0.01738 (1.02)   │   1.28 (1.0) 
 (v1)        0.01707 (1.0)   │   1.28 (1.0) 

benchmarks/drivers/test_build.py::test_build[piecewise-n=1000]
                time (s)          peak (KiB) 
 name               mean   │            mean 
─────────────────────────────────────────────
 (legacy)   0.163 (1.06)   │   946.85 (1.06) 
 (v1)       0.1542 (1.0)   │    891.54 (1.0) 

benchmarks/drivers/test_build.py::test_build[piecewise-n=10]
                 time (s)         peak (KiB) 
 name                mean   │           mean 
─────────────────────────────────────────────
 (legacy)   0.1579 (1.03)   │   12.01 (1.00) 
 (v1)        0.1537 (1.0)   │    11.99 (1.0) 

benchmarks/drivers/test_build.py::test_build[qp-n=1000]
                 time (s)          peak (KiB) 
 name                mean   │            mean 
──────────────────────────────────────────────
 (legacy)   0.0413 (1.07)   │   147.70 (1.06) 
 (v1)       0.03857 (1.0)   │    139.87 (1.0) 

benchmarks/drivers/test_build.py::test_build[qp-n=10]
                 time (s)        peak (KiB) 
 name                mean   │          mean 
────────────────────────────────────────────
 (legacy)   0.0412 (1.06)   │   2.60 (1.09) 
 (v1)       0.03892 (1.0)   │    2.38 (1.0) 

benchmarks/drivers/test_build.py::test_build[rolling-severity=0]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.03505 (1.09)   │   696.75 (1.03) 
 (v1)        0.03229 (1.0)   │    673.70 (1.0) 

benchmarks/drivers/test_build.py::test_build[rolling-severity=100]
                 time (s)         peak (MiB) 
 name                mean   │           mean 
─────────────────────────────────────────────
 (legacy)    0.1042 (1.0)   │   137.97 (1.0) 
 (v1)       0.1074 (1.03)   │   137.97 (1.0) 

benchmarks/drivers/test_build.py::test_build[rolling-severity=50]
                  time (s)        peak (MiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.06244 (1.05)   │   69.22 (1.0) 
 (v1)        0.05921 (1.0)   │   69.22 (1.0) 

benchmarks/drivers/test_build.py::test_build[sos-n=1000]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.03916 (1.09)   │   402.33 (1.00) 
 (v1)        0.03579 (1.0)   │    402.30 (1.0) 

benchmarks/drivers/test_build.py::test_build[sos-n=10]
                 time (s)        peak (KiB) 
 name                mean   │          mean 
────────────────────────────────────────────
 (legacy)    0.039 (1.11)   │   3.19 (1.19) 
 (v1)       0.03524 (1.0)   │    2.69 (1.0) 

benchmarks/drivers/test_build.py::test_build[sparse_network-n=10]
                  time (s)         peak (KiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.04335 (1.09)   │   29.00 (1.54) 
 (v1)        0.03982 (1.0)   │    18.84 (1.0) 

benchmarks/drivers/test_build.py::test_build[sparse_network-n=250]
                  time (s)         peak (MiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.05644 (1.12)   │   37.95 (1.43) 
 (v1)         0.0506 (1.0)   │    26.51 (1.0) 

benchmarks/drivers/test_build.py::test_build[storage-n=10]
                 time (s)          peak (KiB) 
 name                mean   │            mean 
──────────────────────────────────────────────
 (legacy)   0.08102 (1.0)   │    410.93 (1.0) 
 (v1)       0.0827 (1.02)   │   427.84 (1.04) 

benchmarks/drivers/test_build.py::test_build[storage-n=250]
                 time (s)         peak (MiB) 
 name                mean   │           mean 
─────────────────────────────────────────────
 (legacy)   0.08947 (1.0)   │     9.94 (1.0) 
 (v1)       0.0926 (1.03)   │   10.22 (1.03)

📊 Interactive plots + CSV: download the semantics-report-v1-vs-legacy artifact from this run.

_{Report-only · not a gate · refreshed on every push · obsolete once legacy is dropped.}

codspeed · 2026-10-06T10:43:51Z

Merging this PR will not alter performance

✅ 181 untouched benchmarks
⏩ 181 skipped benchmarks¹

_{Comparing perf/matrices-cache-1008 (eeb63e7) with master (f665a26)}

181 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

…w-ups

… once per caller Model.matrices returns a fresh accessor again. Dualization binds it once and expression solutions map labels directly, without building the matrices.

…model

perf(matrices): cache Model.matrices for frozen models and trim the e…

f21f603

…xport path (#1008)

FabianHofmann added the performance This improves performance while not (meaningfully) altering behaviour for users label Oct 6, 2026

FabianHofmann added 2 commits October 6, 2026 12:51

fix(xpress): stop editing cached matrix bounds in place; review follo…

f9058a6

…w-ups

docs: note the shared Model.matrices contract in the release notes

2c9dee4

FabianHofmann mentioned this pull request Oct 6, 2026

perf(constraints): direct rectangle-to-CSR freeze, sort only when nterm > 1 #1015

Merged

5 tasks done

FabianHofmann added 4 commits October 6, 2026 21:02

fix(matrices): key the cache on the objective expression data

fbf7c74

refactor(matrices): drop the Model.matrices cache, build the matrices…

087292b

… once per caller Model.matrices returns a fresh accessor again. Dualization binds it once and expression solutions map labels directly, without building the matrices.

fix(expressions): raise when a solution maps labels missing from the …

d757470

…model

docs: restate the matrices build saving without the cache

89560f3

FabianHofmann changed the title ~~perf(matrices): cache Model.matrices for frozen models and trim the export path~~ perf(matrices): trim the Model.matrices build and drop redundant builds Oct 6, 2026

test(matrices): cover the solution read-back of the accessor

eeb63e7

FabianHofmann merged commit 120dff9 into master Oct 7, 2026
24 checks passed

FabianHofmann deleted the perf/matrices-cache-1008 branch October 7, 2026 08:01

FabianHofmann mentioned this pull request Oct 7, 2026

perf(sparse): trim the stages around the CSR solver hand-off #992

Merged

4 of 5 tasks

Navigation Menu

Repository navigation

perf(matrices): trim the Model.matrices build and drop redundant builds - #1012

perf(matrices): trim the Model.matrices build and drop redundant builds#1012
FabianHofmann merged 8 commits into
masterfrom
perf/matrices-cache-1008

FabianHofmann commented Oct 6, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 6, 2026 •

edited

Loading

Uh oh!

codspeed Bot commented Oct 6, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

FabianHofmann commented Oct 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes proposed in this Pull Request

Checklist

Uh oh!

github-actions Bot commented Oct 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Build cost — v1 vs legacy

Uh oh!

codspeed Bot commented Oct 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will not alter performance

Footnotes

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

FabianHofmann commented Oct 6, 2026 •

edited

Loading

github-actions Bot commented Oct 6, 2026 •

edited

Loading

codspeed Bot commented Oct 6, 2026 •

edited

Loading