perf(planner): the snow bake runs its chunks on every core and keeps its tiles on disk

[timohueser]

Summary

tools/planner_snow.py bakes each zoom-9 chunk in a worker process, one per core. Before, it used 4 threads.

• No copies. The workers fork from the bake, so they read the planes of the sources without a copy.
• Tiles on disk. The tiles of each chunk go to a spill file beside the output as the chunk ends.
• Same bytes. The archive takes the tiles in tile order, so its bytes do not depend on which chunk ends first. The layer revision does not change.
• Less memory. The 4 GiB dict of gzip tiles that the step held until the end is gone. A chunk body for the parent zooms comes back from its tile in the spill file.
• S3 limit. The command line keeps 4 workers when it reads HR-WSI from CDSE S3, which allows 4 connections per user.

Why

On the DACH bake of 2026-10-17 (runs/2026-10-17-071235), planner/snow ran alone for 8.7 h:

• One chunk every 81 s, for 374 chunks.
• CPU time divided by wall time was 1.49 on 12 cores.
• The threads held the interpreter lock most of the time. Before perf(planner): the snow bake skips HR-WSI files that miss a chunk #2730, about 6.5 h of that was transform_bounds on every file for every chunk.

Estimate, not measured on DACH. About 60 CPU-s per chunk remain after #2730.

• With perf(planner): the snow bake skips HR-WSI files that miss a chunk #2730 alone: about 374 × 40 s plus 16 min of MODIS ≈ 4.4 h.
• With this change on 12 cores: about 374 × 60 / 12 s plus 16 min ≈ 50 min.

This is finding 4 of the bake system review.

Memory note

Each worker holds the planes of one chunk: about 0.5 GB for HR-WSI at 20 m. That is an estimate from the chunk size. On 12 cores, a worker peak of 0.5 GB means 6 GB in all.

The receipt shows ru_maxrss: the peak of the largest process, not the sum of the processes. The engine review (finding 1) proposes per-step processes. A cgroup memory.peak would count the sum.

Tests

• Byte check. Old (develop) and new code baked the same synthetic HR-WSI + MODIS input across the corner of four zoom-9 chunks. The archives have the same SHA-256:
  • 21 tiles: 9269efeb…
  • 67 tiles: bbe6e18a…
• Peak memory on the 67-tile case. The parent went from 197 MiB to 80 MiB; each worker peaks at 109 MiB.
• tools/tests suite, as CI runs it (--group dev --group planner-snow --group planner-sun): 331 tests OK. There are no fork warnings.
• test_the_bake_reads_one_zoom_9_chunk_and_stores_to_the_max_zoom now counts reads through a file, because a chunk reads in a worker process.

Not run: a real snow bake, because the machine runs other work.

Public docs: no change.

Review round

An adversarial review confirmed that old and new give the same bytes on four more synthetic inputs. It also found the issues below; all are fixed in cacf667.

• The parent kept every chunk's tiles in RAM. The list of futures held them until the bake returned. The list is gone, and as_completed lets go of each future. On the inputs above, the parent now peaks at 69 to 77 MiB, against 183 to 201 MiB on develop.
• Workers outlived the bake. They survived a SIGKILL of the bake, such as an out-of-memory kill, and kept their memory. Each worker now sets PR_SET_PDEATHSIG. I killed a running bake with SIGKILL and both of its workers ended.
• The worker count ignored limits. The default now follows the cores that the process can use (sched_getaffinity) and is at most one worker per GB of free memory. A worker holds about 0.6 GB for nine seasons.
• Fork is not safe on macOS. Outside Linux the chunks now bake in threads.

The receipt still records the peak of the largest process, not the sum of the workers. That is finding 1 of the review: the session "Run every obc data step in its own process" works on it.

Requirements: none

🤖 Generated with Claude Code

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

• Configuration used: defaults
• Review profile: CHILL
• Plan: Advanced
• Run ID: 5415b1e3-d6fb-4f79-bc70-d640eb2371af

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}