Skip to content

perf(events): search the neighbourhood of a peak in tropical_cyclone.py, not the whole globe - #442

Open
aaTman wants to merge 3 commits into
brightbandtech:developfrom
aaTman:perf/tc-locality-search
Open

aaTman wants to merge 3 commits into
brightbandtech:developfrom
aaTman:perf/tc-locality-search

Conversation

@aaTman

@aaTman aaTman commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Two array-side searches in the tropical-cyclone detector scan the whole (global) field to produce results that are local by construction:

  1. _create_spatial_mask builds a (n_track_points, nlat, nlon) great-circle distance field — chunked 32 points at a time (thanks to the earlier vectorisation) — only to mark the union of discs of radius max_distance_degrees. At 0.25° resolution a 5° disc is ~0.2 % of the globe, so ~99 % of that temporary always compares False; it was the largest temporary in detection (~266 MB float64 per 32-row chunk, plus its haversine intermediates).
  2. find_contours_from_point_specified_field subtracts the point value from the entire field and contours the entire field, once per peak per field (SLP and thickness). find_valid_candidates then rejects any contour whose furthest point is more than max_gc_distance_slp_contour_degrees / max_gc_distance_dz_contour_degrees from the peak — so every contour that can be accepted lies inside a ≤ 6.5° box around the peak.

This PR restricts both searches to a conservative window around the point. The window comes from Clairaut's relation: a geodesic of length ≤ r between two points no farther from the equator than φ_max has |Δlon| ≤ r / cos(φ_max); cos is smallest at the highest |latitude| inside the latitude window, so taking it there cannot clip a point that is in range (and correctly opens to the whole globe at a point near a pole). Latitude uses the exact bound |Δlat| ≤ r. A contour clipped by a window edge extends farther than any contour that can pass the tracker's distance test, so the accept/reject decision cannot change; contours are still returned in full-field index space and find_contours_from_point_specified_field keeps its old 3-argument behaviour when search_radius_degrees is omitted, so no caller changes.

The whole-field path is kept as an unconditional fallback: a window that would cross the antimeridian (contour path; the mask wraps instead), a grid with fewer than two coordinates per axis, a non-uniform or non-finite coordinate array, or a degenerate spacing all keep the previous whole-field computation, so results are identical for those grids too.

Type of change

  • Refactor (performance; behaviour-preserving)
  • Bug fix
  • New feature
  • Chore

No new dependencies.

Benchmarks

Measured with a profiling script kept outside the repo (gist) on a synthetic-but-realistic global 0.25° field (721 × 1440, 40 localised closed lows, 12 IBTrACS-like rows per valid time, surface_pressure_threshold = 101 000 Pa); no network or data download.

Per-step pricing (8-core machine, 24 steps):

v1.1.1 this branch
_create_spatial_mask (12 rows) 496 ms 3 ms
_find_peaks_for_time_slice 725 ms/step 24 ms/step

Real 2400-step peak-search loop (the array work of a 20-init × 120-lead forecast evaluation, 16-core machine, one run each):

v1.1.1 this branch
wall time 30.7 min 0.9 min
peak RSS 2.19 GB 0.64 GB

After the change the per-step profile is flat (windowed contouring, the peak_local_max max-filter, the per-peak DataArray construction) — there is no second hot spot worth a PR here.

To reproduce the timings, run the gist from a checkout on v1.1.1 and on this branch:

uv run python tc_detection_profile.py --steps 24

To run the tests:

uv run pytest tests/test_tropical_cyclone_locality.py tests/test_tropical_cyclone.py -q

How Has This Been Tested?

tests/test_tropical_cyclone_locality.py (72 cases, ~25 s, seeded, no network or data files) pins the behaviour against reference implementations of the replaced whole-field code:

  • _create_spatial_mask exactly equals the whole-field mask over random point sets × {0.25°, 1°} grids × {1, 5, 12}° radii × {1, 7, 40} points; explicitly for discs straddling the dateline and sitting on the poles; on 0–360/descending, −180–180/ascending and descending-longitude grids; and on non-uniform and NaN-coordinate grids (fallback path).
  • contour equivalence, stated precisely: the contours that lie well inside the window are the same objects either way (asserted pointwise, atol=1e-9); windowed and global contour sets produce identical find_valid_candidates accept/reject decisions for every minimum of the fixture at several levels.
  • end-to-end: _find_peaks_for_time_slice returns the same peak set with the window as when the window is patched out; the fixture is asserted non-vacuous so the comparison cannot pass trivially.

Existing suite: tests/test_tropical_cyclone.py (43 cases) and the rest of tests/ pass unchanged; pre-commit (ruff, ruff-format, mypy) passes.

Notes for the reviewer

  • The window width is derived from parameters the tracker already has (max_gc_distance_*_degrees + a 1° marching-squares margin); nothing new is tunable.
  • The decision-preservation argument relies on find_valid_candidates measuring the contour-to-peak distance from the contour's furthest point in index space; every contour that can reach a window edge is farther from the peak than any contour that can pass that test.
  • One input where behaviour differs: track coordinates outside ±90 (missing-value placeholders). Both implementations mark cells based on meaningless haversine values there; which cells they mark differs slightly. IBTrACS coordinates are in range.
  • Not touched, but visible once the mask is cheap: _find_peaks_for_time_slice rebuilds two xr.DataArrays per time slice per peak (the next-largest line in the profile), and calc.haversine_distance re-evaluates np.radians/np.cos on both operands every call; neither can be hoisted as long as the coordinate arrays are rebuilt per slice.

🤖 Generated with Claude Code

aaTman and others added 3 commits November 17, 2026 19:54
_create_spatial_mask built a (n_track_points, nlat, nlon) great-circle distance
field per timestep and find_contours_from_point_specified_field contoured the
whole (global) field per peak per field. Both results are local by
construction: a track point only marks a disc of max_distance_degrees, and a
contour is only ever accepted when its furthest point lies within
max_gc_distance_*_degrees of the peak.

Restrict both searches to a conservative window (Clairaut's relation bounds the
longitude sweep of a geodesic of length r to r/cos(phi_max)). Degenerate grids
and windows that would cross the antimeridian fall back to the whole-field path,
so results are identical.

On a synthetic global 0.25 degree field (benchmarks/tc_detection_profile.py):
spatial mask 504 ms -> 3 ms, peak search 760 ms -> 24 ms per timestep, i.e. a
20-init x 120-lead forecast eval drops from ~30 min to ~1 min of peak search.

tests/test_tropical_cyclone_locality.py pins the equivalence against reference
implementations of the previous whole-field code (random point sets, wrapped and
ascending grids, dateline and pole cases, contour-level sweeps, and an
end-to-end _find_peaks_for_time_slice peak-set comparison).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The windows size themselves from the first cell spacing, so a non-uniform
grid (or a NaN in a coordinate) could make the latitude window too small
or the longitude half-width NaN: the mask silently dropped cells that the
whole-field code marked, and the contourer raised on the NaN spacing.
Guard both sites with a uniform-spacing check and keep the previous
whole-field path for such grids; add regression tests.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@aaTman
aaTman changed the base branch from main to develop November 17, 2026 22:38
@aaTman

aaTman commented Nov 17, 2026

Copy link
Copy Markdown
Collaborator Author

This PR update is from an optimization that Claude + Qwen3.8 Flash Next landed on while setting up an EWB near-real-time dashboard. I'm running the dashboard and the EWB code on a local machine with 512GB RAM total (256GB dedicated to any EWB runs) and 12 cores/24 threads, so not incredibly large. As far as I can tell, this is a solid improvement. Not going to merge it just yet, want to double check if any edge cases get caught.

@aaTman aaTman changed the title perf(events): search the neighbourhood of a peak, not the whole globe perf(events): search the neighbourhood of a peak in tropical_cyclone.py, not the whole globe Nov 18, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant