Repository navigation
feat: add netCDF output option for case caches - #437
Conversation
|
@darothen this PR needs a review. Please take a look when you get time. |
darothen
left a comment
There was a problem hiding this comment.
Hi @AswaniSahoo thanks for working on this! A couple of small areas of feedback to address before we merge this in:
-
As-is, the NetCDF path doesn't apply any compression to the output files. They can get pretty big pretty quickly and bottleneck the evaluation execution. I don't think this needs to be a generally configurable thing; we use sensible defaults for Zarr and that approach should work just fine for NetCDF.
-
The way that attributes on forecast datasets are coerced into the cached dataset has to be handled a bit more carefully. Zarr is happy to run with any type of attribute all the way down to the Variable level. NetCDF will fail if there is any attributes other than the global ones. Would recommend adding some code to handle those edge cases and trying to coerce non-global attributes to the cached NetCDF so it can safely write.
|
@darothen I've pushed changes for all three points from your review. Could you take another look when you get a chance? |
darothen
left a comment
There was a problem hiding this comment.
Looks great @AswaniSahoo; LGTM and will merge once CI runs.
|
@AswaniSahoo this is merged into our |
EWB Pull Request
Description
Closes #286.
Adds
cache_formattoExtremeWeatherBenchand--cache-formatto the CLI. The default stays"zarr";"netcdf"writes each case cache as one zlib-compressed.ncfile ({case_id}_{name}.nc) inutils.maybe_cache_and_computeand reads it back withxr.open_dataset/xr.open_dataarray, the same way the zarr branch does.Two things to check while reviewing:
report_type_mapping(set atinputs.py:965), None, bools or timestamps. Those are written as JSON strings on the dataset, every variable and every coordinate, and only in the netCDF files. This is the string coercion you agreed to on Add option for NetCDF output in case caches #286.cache_formatis passed through the call chain next tocache_dirinstead of in**kwargs, because**kwargsalso feeds pipeline and metric kwargs.No new dependency, netCDF4 already comes with
xarray[io]. The first three commits are split by layer (utils, evaluate, CLI and CHANGELOG), and the last three follow the review, one commit per point.Type of change
If this PR is a breaking change, please explain:
Nothing changes for existing users, with the default
cache_format="zarr"caches are written and read as before.How Has This Been Tested?
tests/test_utils.py,tests/test_evaluate.pyandtests/test_evaluate_cli.py:.ncround trip, reusing an existing.nccache, unnamed sparse DataArray, unsupported attr values as JSON at every level, zlib compression (also for data opened from netCDF), serial and parallelrun_evaluation, the CLI flag, unknown format rejected.uv run pytest: 1500 passed.uv run pre-commit run --all-files: passed.