Repository navigation
feat: publish IFS HRES icechunk store for EWB paper cases (#438) - #441
Merged
Merged
Conversation
Copies the Arraylake ewb-hres forecast archive into an open Icechunk store coerced to the GraphCast MLWP layout (long variable names, latitude -90->90, level 50->1000, one level per chunk, GraphCast CF attrs, 2-D valid_time). Writes are batched and resumable via commit messages, and --validate checks schema parity against the GraphCast reference and bit-identity against the source. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…438) Reads the public IFS HRES Icechunk store and writes the ifs-hres/hres group in brightband/mlwp-forecast-archive with the same five variables, encoding and attrs as graphcast/hres (thickness = (z300 - z500) / g0). Writes go to a branch, one resumable commit per init batch; --validate checks schema parity with graphcast/hres and bit-identity vs the source. Also pins icechunk>=2.3.1 and arraylake>=1.8.0 in ewb_hres_to_icechunk.py, since the target repos are Icechunk spec v2. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replaces the attribution placeholder with ECMWF's required copyright, source, licence, disclaimer and modification statements. The Arraylake group carries them from the source store (failing if absent) and adds its own modifications note; both validators check the attrs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Opens gs://brightband-public-mlwp-forecast-archive/ifs-hres.hres.icechunk with credentials from the environment and wraps it in an XarrayForecast using HRES_metadata_variable_mapping, mirroring get_cira_icechunk. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
4 x nproc threads were OOM-killed at ~29 GB RSS on 32 vCPU; 64 threads gave the same throughput at ~9-17 GB. Default (nproc) is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
5 of 6 tasks
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Branch cache-key collisions and missing source-identity checks during resumable copies can silently reuse another branch’s data or mix incompatible archive versions.
3 open issues
What changed in this PR
Adds publishing and access tooling for the 2023–24 IFS HRES forecasts used by the EWB paper, supporting reproducible evaluation without access to the private Arraylake archive. The published Icechunk store requires Google authentication.
Changes:
- Add resumable publishing scripts for the GCS archive and Arraylake subset, with validation and ECMWF attribution.
- Add Icechunk dataset and forecast helpers with network-free unit tests.
- Document coverage, authentication, usage and licensing.
| File | Description |
|---|---|
tests/test_inputs.py |
Tests helper authentication settings, forecast construction and variable mapping. |
src/extremeweatherbench/inputs.py |
Adds authenticated IFS HRES Icechunk openers and forecast wrapping. |
docs/data.md |
Documents the archive, access requirements, examples and attribution. |
data_prep/ifs_hres_to_arraylake_mlwp.py |
Publishes and validates a five-variable Arraylake subset, including derived thickness. |
data_prep/ewb_hres_to_icechunk.py |
Converts the private archive into a standardized Icechunk store using resumable batches. |
Review effort: Balanced
…it (#438) tests/test_docs_api.py parses fenced Python in docs; the "> "-prefixed block failed with "invalid syntax" on CI. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Both scripts resume by skipping init_time ranges named in commit messages, but never checked the source recorded alongside them. Resuming with a different source snapshot (or bucket/prefix for the Arraylake subset) could silently mix archive versions. Check the recorded source first and exit with a clear error on mismatch. Behaviour for same-source runs is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
evaluate caches pipelines by (case, type, name, source, variables), so two get_ifs_hres_icechunk() forecasts from different branches with the default name shared a cache entry. Append @<branch> to source and test it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
4 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Summary
The EWB paper's 2023–24 cases are scored against a private Arraylake HRES archive (
brightband/ecmwf→forecast-archive/ewb-hres), so outside users get 403 errors. This PR adds the tooling that publishes a copy, plus docs and a helper for using it.data_prep/ewb_hres_to_icechunk.pycopies the archive into a standalone Icechunk repo laid out like the neighbouring GraphCast store: WB2/ERA5 names, ascending latitude and levels,timedelta64lead times, a 2-Dvalid_time, one 2-D field per chunk. Writes go in resumable batches, and--validatechecks the result. The docstring now records the--workersmemory limits: 4 x nproc was OOM-killed at ~29 GB; 64 threads gave the same ~24.5 s/init at ~9-17 GB.data_prep/ifs_hres_to_arraylake_mlwp.pywrites the same data into the Arraylake MLWP forecast archive.gs://brightband-public-mlwp-forecast-archive/ifs-hres.hres.icechunk, branchmain, tip snapshotNG8BP8BHAXKQGA3GA230. It has 1462 init times (2023-01-01 to 2024-12-31, 00/12 UTC) and 41 lead times (0-240 h every 6 h) on 13 levels at 0.25°. Variables:2m_temperature,10m_u/v_component_of_wind,mean_sea_level_pressure,geopotential,temperature,u/v_component_of_wind,specific_humidity.ewb.inputs.get_ifs_hres_icechunk()(andopen_ifs_hres_icechunk()) open the store with credentials from the environment and return anXarrayForecastwithHRES_metadata_variable_mapping. This follows theget_cira_icechunk()pattern. Unit tests mock icechunk and xarray, so they make no network calls.docs/data.mdsection covers the store's coverage, variables, auth, example usage (helper and plainXarrayForecast) and the licence/attribution.graphcast.hres.icechunk,panguweather.hres.icechunkandaifs-single.hres.icechunk, reads need an authenticated Google account (gcloud auth application-default login). Anonymous reads are refused on purpose, and the docs say so. This differs from the issue's original "anonymous reads" task.processed_by, and the docs repeat the required credit.Validation
--validate --samples 10gave 236 PASS / 0 FAIL. The store is 9.37 TB and every chunk is present.brightband/mlwp-forecast-archiveifs-hres/hresonmainatG27QRC7QQG8EFV0VH7NG, 146 PASS / 0 FAIL.uv run pytest tests/test_inputs.py -k "IFSHRES or CIRAIcechunk": 21 passed.zensical buildreports no issues.Test plan
uv run pytest tests/test_inputs.py -k IFSHRES(3 passed)ewb.inputs.get_ifs_hres_icechunk(variables=["surface_air_temperature"])opens and shows the expected dims (verified: returnsXarrayForecast"IFS-HRES"; dims init_time=1462, lead_time=41, level=13, latitude=721, longitude=1440; 9 variables; 2m_temperature at the first init/lead is 216–314 K)just build-docs, then check the new "IFS HRES for the EWB paper cases" section in DataPart of #438
🤖 Generated with Claude Code