feat(io): add compression option to Model.to_netcdf by coroa · Pull Request #1007 · PyPSA/linopy

coroa · 2026-10-06T09:17:19Z

The linopy model does currently not have an easy way to add compression options to all variables since the name of quite some variables depend on factors like whether they are frozen or MultiIndex'ed.

Thus, this PR adds a simple compression argument to Model.to_netcdf which is applied to the encoding and enabled by default.

Note

The following content was generated by AI.

Add `compression` option to `Model.to_netcdf`, on by default

Model.to_netcdf forwards **kwargs to xarray.Dataset.to_netcdf, but compressing the file required a per-variable encoding keyed by linopy's internal names (variables-x-labels, constraints-c-…), which callers can't know in advance.

This adds a keyword-only compression argument, applied to every array in the dataset:

m.to_netcdf("model.nc")                                          # default: {"zlib": True, "complevel": 3}
m.to_netcdf("model.nc", compression={"zlib": True, "complevel": 9})
m.to_netcdf("model.nc", compression=False)                       # uncompressed, as before

On by default. compression=True applies {"zlib": True, "complevel": 3}. This also affects the internal writes in remote/ssh.py and remote/oetc.py.
Set as a fallback in each array's .encoding, on the merged dataset that gets written, not on the model's own arrays. As a result:
- keys a user already set on an array's .encoding (e.g. dtype, complevel) win over compression;
- an array's entry in encoding= passed to xarray replaces its encoding entirely, as in plain xarray;
- engines without compression support (scipy/netCDF3, in-memory bytes) silently ignore it, so the default never breaks a write. An explicit dict is likewise ignored there.
String arrays are written as variable-length strings, and both netcdf4 and h5netcdf skip compression on those without raising an error.

Works for regular and frozen (CSRConstraint) constraints, whose CSR components are written as plain 1-D arrays. On a 2000-row toy model with zlib/complevel=4, the file shrinks about 5×.

Tests

test_model_to_netcdf_compression:
- the compressed file is less than half the size of the uncompressed one;
- an explicit encoding entry wins over compression;
- the model round-trips with assert_model_equal.
test_model_to_netcdf_compression_non_numeric_coords: datetime and timedelta coordinates are compressed on variables and frozen constraints, and string coordinates don't break writing or reading back.
test_model_to_netcdf_default_compression: the default applies complevel=3, and compression=False turns it off.
test_model_to_netcdf_compression_keeps_array_encoding: a user-set .encoding on a model array wins, and the model itself isn't modified.
test_model_to_netcdf_compression_scipy: scipy writes with both the default and an explicit dict.

All pass with netcdf4, and with h5netcdf installed.

🤖 Generated with Claude Code

Applies the given encoding (e.g. {"zlib": True, "complevel": 4}) to every array in the dataset, so callers no longer need linopy's internal variable names to compress the file. Explicit `encoding` entries take precedence. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

github-actions · 2026-10-06T09:20:07Z

Build cost — v1 vs legacy

v1 build peak & time relative to legacy, on this commit — not a comparison against master (that is CodSpeed).

peak — v1 / legacy	time — v1 / legacy

Full table (time + peak, mean)

benchmarks/drivers/test_build.py::test_build[basic-n=10]
                  time (s)         peak (KiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.05182 (1.03)   │   15.03 (1.00) 
 (v1)        0.05021 (1.0)   │    15.00 (1.0) 

benchmarks/drivers/test_build.py::test_build[basic-n=250]
                  time (s)         peak (MiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.06341 (1.16)   │   12.04 (1.00) 
 (v1)        0.05463 (1.0)   │    12.04 (1.0) 

benchmarks/drivers/test_build.py::test_build[cumsum-severity=0]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.02345 (1.14)   │   15.20 (1.0) 
 (v1)        0.02066 (1.0)   │   15.20 (1.0) 

benchmarks/drivers/test_build.py::test_build[cumsum-severity=100]
                 time (s)        peak (MiB) 
 name                mean   │          mean 
────────────────────────────────────────────
 (legacy)   0.0376 (1.07)   │   44.93 (1.0) 
 (v1)       0.03528 (1.0)   │   44.93 (1.0) 

benchmarks/drivers/test_build.py::test_build[cumsum-severity=50]
                  time (s)        peak (MiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.02786 (1.15)   │   11.51 (1.0) 
 (v1)         0.0242 (1.0)   │   11.51 (1.0) 

benchmarks/drivers/test_build.py::test_build[expression_arithmetic-n=10]
                  time (s)         peak (KiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)    0.05787 (1.0)   │   24.34 (1.06) 
 (v1)       0.05964 (1.03)   │    23.04 (1.0) 

benchmarks/drivers/test_build.py::test_build[expression_arithmetic-n=250]
                 time (s)         peak (MiB) 
 name                mean   │           mean 
─────────────────────────────────────────────
 (legacy)     0.07 (1.10)   │   16.12 (1.00) 
 (v1)       0.06355 (1.0)   │    16.12 (1.0) 

benchmarks/drivers/test_build.py::test_build[knapsack-n=10000]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.01523 (1.16)   │   752.18 (1.10) 
 (v1)        0.01316 (1.0)   │    685.15 (1.0) 

benchmarks/drivers/test_build.py::test_build[knapsack-n=100]
                 time (s)        peak (KiB) 
 name                mean   │          mean 
────────────────────────────────────────────
 (legacy)   0.0137 (1.07)   │   3.12 (1.33) 
 (v1)       0.01281 (1.0)   │    2.34 (1.0) 

benchmarks/drivers/test_build.py::test_build[kvl_cycles-severity=0]
                  time (s)          peak (MiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.05657 (1.39)   │   126.16 (1.44) 
 (v1)        0.04079 (1.0)   │     87.71 (1.0) 

benchmarks/drivers/test_build.py::test_build[kvl_cycles-severity=100]
                  time (s)          peak (MiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.05692 (1.37)   │   126.16 (1.44) 
 (v1)        0.04169 (1.0)   │     87.71 (1.0) 

benchmarks/drivers/test_build.py::test_build[kvl_cycles-severity=50]
                  time (s)          peak (MiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.05751 (1.42)   │   126.16 (1.44) 
 (v1)        0.04059 (1.0)   │     87.71 (1.0) 

benchmarks/drivers/test_build.py::test_build[masked-n=100]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.03403 (1.09)   │    715.12 (1.0) 
 (v1)        0.03122 (1.0)   │   787.73 (1.10) 

benchmarks/drivers/test_build.py::test_build[masked-n=10]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.03118 (1.12)   │   4.54 (1.27) 
 (v1)        0.02795 (1.0)   │    3.57 (1.0) 

benchmarks/drivers/test_build.py::test_build[merge_balance-severity=0]
                 time (s)          peak (KiB) 
 name                mean   │            mean 
──────────────────────────────────────────────
 (legacy)   0.2192 (1.00)   │   704.12 (1.09) 
 (v1)        0.2181 (1.0)   │    643.85 (1.0) 

benchmarks/drivers/test_build.py::test_build[merge_balance-severity=100]
                 time (s)        peak (MiB) 
 name                mean   │          mean 
────────────────────────────────────────────
 (legacy)   0.2468 (1.06)   │   18.34 (1.0) 
 (v1)        0.2336 (1.0)   │   18.34 (1.0) 

benchmarks/drivers/test_build.py::test_build[merge_balance-severity=50]
                 time (s)       peak (MiB) 
 name                mean   │         mean 
───────────────────────────────────────────
 (legacy)    0.2289 (1.0)   │   9.54 (1.0) 
 (v1)       0.2362 (1.03)   │   9.54 (1.0) 

benchmarks/drivers/test_build.py::test_build[milp-n=10]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.04593 (1.10)   │   3.77 (1.12) 
 (v1)        0.04173 (1.0)   │    3.37 (1.0) 

benchmarks/drivers/test_build.py::test_build[milp-n=50]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.04497 (1.12)   │   216.59 (1.10) 
 (v1)        0.04027 (1.0)   │    196.23 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance-severity=0]
                  time (s)         peak (KiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.02326 (1.09)   │   938.49 (1.0) 
 (v1)        0.02141 (1.0)   │   938.49 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance-severity=100]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)   0.02447 (1.07)   │   9.66 (1.0) 
 (v1)         0.0229 (1.0)   │   9.66 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance-severity=50]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)   0.02431 (1.00)   │   5.32 (1.0) 
 (v1)        0.02423 (1.0)   │   5.32 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance_sparse-severity=0]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)   0.01185 (1.00)   │   1.28 (1.0) 
 (v1)        0.01185 (1.0)   │   1.28 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance_sparse-severity=100]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)   0.01178 (1.01)   │   1.28 (1.0) 
 (v1)        0.01171 (1.0)   │   1.28 (1.0) 

benchmarks/drivers/test_build.py::test_build[nodal_balance_sparse-severity=50]
                  time (s)       peak (MiB) 
 name                 mean   │         mean 
────────────────────────────────────────────
 (legacy)    0.01179 (1.0)   │   1.28 (1.0) 
 (v1)       0.01237 (1.05)   │   1.28 (1.0) 

benchmarks/drivers/test_build.py::test_build[piecewise-n=1000]
                 time (s)          peak (KiB) 
 name                mean   │            mean 
──────────────────────────────────────────────
 (legacy)   0.1098 (1.06)   │   946.85 (1.06) 
 (v1)        0.1038 (1.0)   │    891.54 (1.0) 

benchmarks/drivers/test_build.py::test_build[piecewise-n=10]
                 time (s)         peak (KiB) 
 name                mean   │           mean 
─────────────────────────────────────────────
 (legacy)   0.1155 (1.05)   │   12.01 (1.00) 
 (v1)        0.1105 (1.0)   │    11.99 (1.0) 

benchmarks/drivers/test_build.py::test_build[qp-n=1000]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.02852 (1.05)   │   147.70 (1.06) 
 (v1)        0.02709 (1.0)   │    139.87 (1.0) 

benchmarks/drivers/test_build.py::test_build[qp-n=10]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.02962 (1.13)   │   2.60 (1.09) 
 (v1)        0.02614 (1.0)   │    2.38 (1.0) 

benchmarks/drivers/test_build.py::test_build[rolling-severity=0]
                 time (s)          peak (KiB) 
 name                mean   │            mean 
──────────────────────────────────────────────
 (legacy)   0.0238 (1.04)   │   696.75 (1.03) 
 (v1)       0.02287 (1.0)   │    673.70 (1.0) 

benchmarks/drivers/test_build.py::test_build[rolling-severity=100]
                  time (s)         peak (MiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.07905 (1.05)   │   137.97 (1.0) 
 (v1)        0.07493 (1.0)   │   137.97 (1.0) 

benchmarks/drivers/test_build.py::test_build[rolling-severity=50]
                  time (s)        peak (MiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)    0.04424 (1.0)   │   69.22 (1.0) 
 (v1)       0.04585 (1.04)   │   69.22 (1.0) 

benchmarks/drivers/test_build.py::test_build[sos-n=1000]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)   0.02864 (1.19)   │   402.33 (1.00) 
 (v1)        0.02405 (1.0)   │    402.30 (1.0) 

benchmarks/drivers/test_build.py::test_build[sos-n=10]
                  time (s)        peak (KiB) 
 name                 mean   │          mean 
─────────────────────────────────────────────
 (legacy)   0.02617 (1.11)   │   3.19 (1.19) 
 (v1)        0.02351 (1.0)   │    2.69 (1.0) 

benchmarks/drivers/test_build.py::test_build[sparse_network-n=10]
                  time (s)         peak (KiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.02908 (1.08)   │   29.00 (1.54) 
 (v1)        0.02686 (1.0)   │    18.84 (1.0) 

benchmarks/drivers/test_build.py::test_build[sparse_network-n=250]
                  time (s)         peak (MiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)   0.04278 (1.18)   │   37.95 (1.43) 
 (v1)        0.03611 (1.0)   │    26.51 (1.0) 

benchmarks/drivers/test_build.py::test_build[storage-n=10]
                  time (s)          peak (KiB) 
 name                 mean   │            mean 
───────────────────────────────────────────────
 (legacy)    0.05498 (1.0)   │    410.93 (1.0) 
 (v1)       0.05598 (1.02)   │   427.84 (1.04) 

benchmarks/drivers/test_build.py::test_build[storage-n=250]
                  time (s)         peak (MiB) 
 name                 mean   │           mean 
──────────────────────────────────────────────
 (legacy)    0.06258 (1.0)   │     9.94 (1.0) 
 (v1)       0.06361 (1.02)   │   10.22 (1.03)

📊 Interactive plots + CSV: download the semantics-report-v1-vs-legacy artifact from this run.

_{Report-only · not a gate · refreshed on every push · obsolete once legacy is dropped.}

codspeed · 2026-10-06T09:21:54Z

Merging this PR will not alter performance

✅ 181 untouched benchmarks
⏩ 181 skipped benchmarks¹

_{Comparing feature/netcdf-compression (d37e94c) with master (f665a26)}

181 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

Navigation Menu

Repository navigation

feat(io): add compression option to Model.to_netcdf - #1007

feat(io): add compression option to Model.to_netcdf#1007
coroa wants to merge 1 commit into
masterfrom
feature/netcdf-compression

coroa commented Oct 6, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 6, 2026

Uh oh!

codspeed Bot commented Oct 6, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

coroa commented Oct 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Add compression option to Model.to_netcdf, on by default

Tests

Uh oh!

github-actions Bot commented Oct 6, 2026

Build cost — v1 vs legacy

Uh oh!

codspeed Bot commented Oct 6, 2026

Merging this PR will not alter performance

Footnotes

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

coroa commented Oct 6, 2026 •

edited

Loading

Add `compression` option to `Model.to_netcdf`, on by default