feat(obc-data): each step runs in a process of its own

[timohueser]

Summary

Each step of obc data runs in a process of its own. The engine stays small, and each receipt holds the peak memory and the CPU time of its own step. The scheduler then admits steps by these peaks, the longest chain first.

A process per step

• A Rust step is a child of the same binary. The engine starts <binary> obc_data_step <function> (on Linux /proc/self/exe, so a rebuild during a run does not change the code), with the request as JSON on standard input, as for a command step. Each binary registers its step functions at the start of main with obc_data::engine::child(FUNCTIONS): obc data with host/obc-data-steps/src/functions.rs, obc-data-plumbing with obc-data/pass. A function name is <crate>/<function>.
• Tests use the same path. A test binary that runs Rust steps has one test, obc_data_step, that registers its test functions. libtest reads obc_data_step <function> as test names, so the child runs that test alone. A name with / matches no test, so order refuses a function name without /.
• Deleted: in_process, the CHILDREN and STOP thread-locals, check_stop (with its calls in obc-pack), run_child (obc-osm waits for Osmium itself, and wait4 of the step counts it), process_peak_rss_bytes of the review, and the spec caveats "exact only while no other step runs" and "an upper bound".
• Stop. Each step leads a process group of its own. When a run stops (signal, stop file, lost lock), the engine sends SIGTERM to the group of each step that runs, so the programs of the step end too.
• Fork, not vfork. A pre_exec sets the group, so Command forks. Linux starts the peak of a vfork child at the peak of the engine, and that of a fork child at its current resident set.
• Dead runs (fix(obc-data): a run that died frees the lock that it kept #2733). The programs of a step are no longer in the process group of the run. A dead run now keeps its lock while a process of its session runs: the detached run leads its session, and every step stays in it. Linux reads /proc. Other systems cannot list a session, so they take no lock over: unlock frees it after 10 minutes.
• TMPDIR (fix(obc-data): a step program writes its temporary files on the disk of the store #2732) now applies to Rust steps too.
• Exact floats. obc-data enables serde_json float_roundtrip: the child parses the floats of its options as the key reads them (the default parser was 1 ULP off for about 12 % of 17-digit coordinates). The feature unifies into every crate of a build that links obc-data, so those parse JSON floats exactly, as Python does. No baked bytes change:
  • Options. Before this PR a Rust step got its options in-process, exactly. The child now gets the same values: a test sends 10,000 17-digit coordinates through a child.
  • JSON input files. I emulated serde_json's default algorithm (the digits as an integer, then one multiply or divide by a power of ten) on every float literal in the JSON files that the 38 receipts and the snapshots of the timosPC store name. The Geofabrik index (292,100 floats) and the landmark content (4,697) parse the same. Only the planner */grid index.json files (16,778 of 45,811 floats) parse otherwise, and only Python steps read their contents (planner/index, planner/sun, overlays); the catalog pointer reads their digests.
  • Floats that a Rust step writes again. The planner catalog pointer copies bounds from release.json into catalog.json. These bounds are microdegrees divided by 1e6, so short decimals, which both parsers read exactly.
  • Byte-pinned tests in CI. The terrain shard digests of obc-dem (the_same_inputs_bake_the_same_bytes_forever and the lifted plane) and the semantic geometry digests of obc-draw pass. They pin device bytes, but no JSON floats reach them, so they are weak evidence here. The planner outputs have no digest pin: planner_build checks their structure.
• A step ends with its engine. On Linux a step gets PR_SET_PDEATHSIG SIGKILL, so a killed run leaves no step behind (the programs that a step started can stay; the session check below keeps the lock).
• ps shows a Rust step as obc-data obc_data_step <function> (arg0).

Admission (the owner chose rule A, scaled by input size)

• Of the steps that can start, the one with the longest chain of planned steps from it to the end of the run starts first: its estimated wall_ms plus the longest chain of the steps that read it. A step without an estimate counts as the longest. I rank by the chain, not by the step's own time, because the chain keeps the order a dependency order: the scan of ready steps relies on that.
• Only true peaks count. A receipt has own_process (absent = false). An estimate reads a peak only from a receipt with it; the cost in a release never gives one, as a release does not tell whose peak it is. So the in-process high-water marks of old receipts and of a release baked on a bigger machine refuse no update: their steps count as without a peak.
• A step reserves its estimated peak. A step without one reserves from the measured steps of its layer (in the store or of this run): their smallest peak, or, when that is more, its input bytes times the peak per input byte of the measured step that read the most. This keeps the owner's per-byte rule, but takes the ratio from the largest input, where the fixed memory of a process matters least: 100 MB of a 10 kB sea leaf no longer makes a 5 MB leaf reserve 50 GB. A layer without a measured step runs its first step alone. A reservation is at most the memory, so the sums cannot overflow.
• A step that does not fit in the free memory holds back the steps after it, so that the long step is not starved by short ones.

Notes

• ru_maxrss is the largest process of a step, not the sum. A step with worker processes, such as the snow bake after perf(planner): the snow bake runs its chunks on every core and keeps its tiles on disk #2734, reports its largest worker. A cgroup memory.peak per step counts them all. That is not in this PR: it needs a delegated cgroup per step (a systemd user scope), and memory.peak also counts page cache, so a step that reads a 6 GB extract would look larger than it is. It is a follow-up.
• Known limit: a step with worker processes (snow, climate) records and reserves its largest process, not the sum. The proposed follow-up is a cgroup memory.peak per step.
• The land index of obc-draw (LAND_INDEX) is now built once per draw-cells process instead of once per engine. Measured on timosPC: the shapefile has 848,110 records; a warm read of the 1.3 GB file takes 0.15 s (the first read 1.4 s), and a scan with the same read pattern 0.5 s in Python. Against minutes per leaf this is not material, so there is no cross-process cache.
• Each child has its own rayon pool, so up to cores × jobs threads can exist. The OS shares the cores; a fixed split would starve a step that runs alone, so I left it.
• A step that reads no bytes and has no estimate reserves the smallest measured peak of its layer.

Checks

• OBC_UPDATE_DATA_SPEC=1 obc test -p obc-data: 202 passed. New: a receipt has the peak of its own step; a Rust step gets the floats of its options exactly (10,000 17-digit coordinates); a peak comes only from a receipt with own_process; a stop ends a step with the programs that it started; the memory budget with a scaled reservation; the first step of a layer without a peak runs alone; the longest step starts first and holds back the rest; FUNCTIONS has unique names and lists every static.
• After a rebuild of every crate of host/ and planner/ from this branch (the worktrees share one target directory), obc test -p passed for obc-data-steps (37), obc-osm (3), obc-pack (95), obc-draw (102), obc-network (30), obc-dem (48), obc-search-bake (16), planner-router-build (65) and obc-bake (187). These hold the golden tests of the Rust steps: exact float parsing changed none of their outputs. planner_build needs builder/web/node_modules; the worktree had none, so I linked the one of the main checkout for the run.
• obc test fixtures -p obc-data-steps: cells_fixture, terrain_fixture and planner_build passed, also after the merge of develop with fix(deps): zip 9 for RUSTSEC-2026-0367, -0368 and -0369 #2745.
• cargo clippy -p <crate> --all-targets -- -D warnings for obc-data, obc-data-steps, obc-osm and obc-pack: clean. cargo fmt --all -- --check, obc suites check, obc prose --check and docs/build_docs.py --check-links: clean.
• Not run: the web suites that the changed example JSONs select (they only lose process_peak_rss_bytes), and no bake (shared PC).

Public docs changed: specs/obc-data.md (step contract, Receipt, Runs, locks, Review) and host/obc-data/README.md, in their own docs: commit. The generated JSON schema stays with the code.

Requirements: none (internal bake tooling; I could not read the verification console, as this machine has no agent token).

🤖 Generated with Claude Code

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

• Configuration used: defaults
• Review profile: CHILL
• Plan: Advanced
• Run ID: fe5f8ef3-2c26-4376-9dfa-40c11fff0a85

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}