Skip to content

feat: add netCDF output option for case caches - #437

Merged
darothen merged 6 commits into
brightbandtech:developfrom
AswaniSahoo:feat/netcdf-case-cache
Nov 16, 2026
Merged

darothen merged 6 commits into
brightbandtech:developfrom
AswaniSahoo:feat/netcdf-case-cache

Conversation

@AswaniSahoo

@AswaniSahoo AswaniSahoo commented Oct 31, 2026 •

Copy link
Copy Markdown

EWB Pull Request

Description

Closes #286.

Adds cache_format to ExtremeWeatherBench and --cache-format to the CLI. The default stays "zarr"; "netcdf" writes each case cache as one zlib-compressed .nc file ({case_id}_{name}.nc) in utils.maybe_cache_and_compute and reads it back with xr.open_dataset / xr.open_dataarray, the same way the zarr branch does.

Two things to check while reviewing:

  • netCDF can't store some attr values that zarr accepts: dicts like LSR's report_type_mapping (set at inputs.py:965), None, bools or timestamps. Those are written as JSON strings on the dataset, every variable and every coordinate, and only in the netCDF files. This is the string coercion you agreed to on Add option for NetCDF output in case caches #286.
  • cache_format is passed through the call chain next to cache_dir instead of in **kwargs, because **kwargs also feeds pipeline and metric kwargs.

No new dependency, netCDF4 already comes with xarray[io]. The first three commits are split by layer (utils, evaluate, CLI and CHANGELOG), and the last three follow the review, one commit per point.

Type of change

  • Refactor
  • Bug fix
  • New feature
  • Chore

If this PR is a breaking change, please explain:

Nothing changes for existing users, with the default cache_format="zarr" caches are written and read as before.

How Has This Been Tested?

  • New tests in tests/test_utils.py, tests/test_evaluate.py and tests/test_evaluate_cli.py: .nc round trip, reusing an existing .nc cache, unnamed sparse DataArray, unsupported attr values as JSON at every level, zlib compression (also for data opened from netCDF), serial and parallel run_evaluation, the CLI flag, unknown format rejected.
  • uv run pytest: 1500 passed.
  • uv run pre-commit run --all-files: passed.

@AswaniSahoo

Copy link
Copy Markdown
Author

@darothen this PR needs a review. Please take a look when you get time.

@darothen darothen left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @AswaniSahoo thanks for working on this! A couple of small areas of feedback to address before we merge this in:

  1. As-is, the NetCDF path doesn't apply any compression to the output files. They can get pretty big pretty quickly and bottleneck the evaluation execution. I don't think this needs to be a generally configurable thing; we use sensible defaults for Zarr and that approach should work just fine for NetCDF.

  2. The way that attributes on forecast datasets are coerced into the cached dataset has to be handled a bit more carefully. Zarr is happy to run with any type of attribute all the way down to the Variable level. NetCDF will fail if there is any attributes other than the global ones. Would recommend adding some code to handle those edge cases and trying to coerce non-global attributes to the cached NetCDF so it can safely write.

Comment thread src/extremeweatherbench/utils.py Resolved
@AswaniSahoo

Copy link
Copy Markdown
Author

@darothen I've pushed changes for all three points from your review. Could you take another look when you get a chance?

@AswaniSahoo
AswaniSahoo requested a review from darothen November 14, 2026 06:43

@darothen darothen left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great @AswaniSahoo; LGTM and will merge once CI runs.

@darothen
darothen merged commit 7cf81bc into brightbandtech:develop Nov 16, 2026
6 checks passed
@darothen

Copy link
Copy Markdown
Collaborator

@AswaniSahoo this is merged into our develop branch, so we'll catch it on the next version bump of the core library.

@AswaniSahoo
AswaniSahoo deleted the feat/netcdf-case-cache branch November 18, 2026 15:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants