Skip to content

feat: publish IFS HRES icechunk store for EWB paper cases (#438) - #441

Merged
darothen merged 9 commits into
mainfrom
feat/438-public-hres-icechunk
Nov 17, 2026
Merged

darothen merged 9 commits into
mainfrom
feat/438-public-hres-icechunk

Conversation

@darothen

@darothen darothen commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

The EWB paper's 2023–24 cases are scored against a private Arraylake HRES archive (brightband/ecmwf → forecast-archive/ewb-hres), so outside users get 403 errors. This PR adds the tooling that publishes a copy, plus docs and a helper for using it.

  • Scripts
    • data_prep/ewb_hres_to_icechunk.py copies the archive into a standalone Icechunk repo laid out like the neighbouring GraphCast store: WB2/ERA5 names, ascending latitude and levels, timedelta64 lead times, a 2-D valid_time, one 2-D field per chunk. Writes go in resumable batches, and --validate checks the result. The docstring now records the --workers memory limits: 4 x nproc was OOM-killed at ~29 GB; 64 threads gave the same ~24.5 s/init at ~9-17 GB.
    • data_prep/ifs_hres_to_arraylake_mlwp.py writes the same data into the Arraylake MLWP forecast archive.
  • Store: gs://brightband-public-mlwp-forecast-archive/ifs-hres.hres.icechunk, branch main, tip snapshot NG8BP8BHAXKQGA3GA230. It has 1462 init times (2023-01-01 to 2024-12-31, 00/12 UTC) and 41 lead times (0-240 h every 6 h) on 13 levels at 0.25°. Variables: 2m_temperature, 10m_u/v_component_of_wind, mean_sea_level_pressure, geopotential, temperature, u/v_component_of_wind, specific_humidity.
  • Helper: ewb.inputs.get_ifs_hres_icechunk() (and open_ifs_hres_icechunk()) open the store with credentials from the environment and return an XarrayForecast with HRES_metadata_variable_mapping. This follows the get_cira_icechunk() pattern. Unit tests mock icechunk and xarray, so they make no network calls.
  • Docs: a new docs/data.md section covers the store's coverage, variables, auth, example usage (helper and plain XarrayForecast) and the licence/attribution.
  • Auth by design: like graphcast.hres.icechunk, panguweather.hres.icechunk and aifs-single.hres.icechunk, reads need an authenticated Google account (gcloud auth application-default login). Anonymous reads are refused on purpose, and the docs say so. This differs from the issue's original "anonymous reads" task.
  • Attribution: the data are ECMWF IFS HRES under CC BY 4.0. The root attrs carry the attribution, licence, licence URL, modifications, disclaimer and processed_by, and the docs repeat the required credit.

Validation

  • GCS store: all 183 batches committed. --validate --samples 10 gave 236 PASS / 0 FAIL. The store is 9.37 TB and every chunk is present.
  • Arraylake subset: brightband/mlwp-forecast-archive ifs-hres/hres on main at G27QRC7QQG8EFV0VH7NG, 146 PASS / 0 FAIL.
  • uv run pytest tests/test_inputs.py -k "IFSHRES or CIRAIcechunk": 21 passed.
  • pre-commit (ruff, ruff-format, mypy) passes on the changed files, and zensical build reports no issues.

Test plan

  • uv run pytest tests/test_inputs.py -k IFSHRES (3 passed)
  • With ADC set up, ewb.inputs.get_ifs_hres_icechunk(variables=["surface_air_temperature"]) opens and shows the expected dims (verified: returns XarrayForecast "IFS-HRES"; dims init_time=1462, lead_time=41, level=13, latitude=721, longitude=1440; 9 variables; 2m_temperature at the first init/lead is 216–314 K)
  • Docs render: just build-docs, then check the new "IFS HRES for the EWB paper cases" section in Data
  • Follow-up (not in this PR): point the paper code at the public store
  • Follow-up (not in this PR): spot-check one case per event type against the private copy

Part of #438

🤖 Generated with Claude Code

darothen and others added 6 commits November 12, 2026 22:03
Copies the Arraylake ewb-hres forecast archive into an open Icechunk
store coerced to the GraphCast MLWP layout (long variable names,
latitude -90->90, level 50->1000, one level per chunk, GraphCast CF
attrs, 2-D valid_time). Writes are batched and resumable via commit
messages, and --validate checks schema parity against the GraphCast
reference and bit-identity against the source.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…438)

Reads the public IFS HRES Icechunk store and writes the ifs-hres/hres
group in brightband/mlwp-forecast-archive with the same five variables,
encoding and attrs as graphcast/hres (thickness = (z300 - z500) / g0).
Writes go to a branch, one resumable commit per init batch; --validate
checks schema parity with graphcast/hres and bit-identity vs the source.

Also pins icechunk>=2.3.1 and arraylake>=1.8.0 in ewb_hres_to_icechunk.py,
since the target repos are Icechunk spec v2.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replaces the attribution placeholder with ECMWF's required copyright,
source, licence, disclaimer and modification statements. The Arraylake
group carries them from the source store (failing if absent) and adds
its own modifications note; both validators check the attrs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Opens gs://brightband-public-mlwp-forecast-archive/ifs-hres.hres.icechunk
with credentials from the environment and wraps it in an XarrayForecast
using HRES_metadata_variable_mapping, mirroring get_cira_icechunk.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
4 x nproc threads were OOM-killed at ~29 GB RSS on 32 vCPU; 64 threads
gave the same throughput at ~9-17 GB. Default (nproc) is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings November 17, 2026 04:25

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Branch cache-key collisions and missing source-identity checks during resumable copies can silently reuse another branch’s data or mix incompatible archive versions.

3 open issues
What changed in this PR

Adds publishing and access tooling for the 2023–24 IFS HRES forecasts used by the EWB paper, supporting reproducible evaluation without access to the private Arraylake archive. The published Icechunk store requires Google authentication.

Changes:

  • Add resumable publishing scripts for the GCS archive and Arraylake subset, with validation and ECMWF attribution.
  • Add Icechunk dataset and forecast helpers with network-free unit tests.
  • Document coverage, authentication, usage and licensing.
File Description
tests/​test_inputs.py Tests helper authentication settings, forecast construction and variable mapping.
src/​extremeweatherbench/​inputs.py Adds authenticated IFS HRES Icechunk openers and forecast wrapping.
docs/​data.md Documents the archive, access requirements, examples and attribution.
data_prep/​ifs_hres_to_arraylake_mlwp.py Publishes and validates a five-variable Arraylake subset, including derived thickness.
data_prep/​ewb_hres_to_icechunk.py Converts the private archive into a standardized Icechunk store using resumable batches.

Review effort: Balanced

Comment thread data_prep/ewb_hres_to_icechunk.py Resolved
Comment thread data_prep/ifs_hres_to_arraylake_mlwp.py Resolved
Comment thread src/extremeweatherbench/inputs.py Resolved
darothen and others added 3 commits November 16, 2026 21:34
…it (#438)

tests/test_docs_api.py parses fenced Python in docs; the "> "-prefixed block
failed with "invalid syntax" on CI.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Both scripts resume by skipping init_time ranges named in commit messages,
but never checked the source recorded alongside them. Resuming with a
different source snapshot (or bucket/prefix for the Arraylake subset) could
silently mix archive versions. Check the recorded source first and exit with
a clear error on mismatch. Behaviour for same-source runs is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
evaluate caches pipelines by (case, type, name, source, variables), so two
get_ifs_hres_icechunk() forecasts from different branches with the default
name shared a cache entry. Append @<branch> to source and test it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@darothen
darothen merged commit 0bb0f39 into main Nov 17, 2026
6 checks passed
@darothen
darothen deleted the feat/438-public-hres-icechunk branch November 17, 2026 16:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants