obc data: the planet region reads osm-planet and the daily diffs after it

[timohueser]

Part of #2655 (P7-D).

Stacked on #2701. This branch starts from #2701 (claude/obc-data-v2-p6-wiki-readers, c056967fe), so its diff holds #2701 until #2701 merges. After that, I merge develop into this branch. I do not rebase. The commits of this PR are 87735e305, eeaec4250 and ee3c1883b.

Problem

No region can select the OSM planet. osm-planet and osm-replication have fetchers, but no product reads them.

Behavior

• A region of kind planet is the OSM planet. A union that holds it is the planet too. data/regions/planet.toml is the preset. areas --json and the web type know the kind. Its overview box is the world.
• region_sources::resolve for a planet: B = the version of osm-planet, and E = the version of osm-replication with from=B. The outline is the world box.
• One source step, device/source/planet, runs obc_osm::step::planet. It does osmium apply-changes -F pbf --change-file-format osc.gz --generator obc-data of the diffs, in the order of their sequences, and writes source.osm.pbf and source.poly, the same files as the area step. With no diff, source.osm.pbf is a link to the planet file. device/osm reads it as the one source.
• Links: link moves from target to engine, next to view and pass. The two OSM source steps and engine::pass use it. It copies only across file systems and never opens an existing file for writing, so no step can write through a link to a store object.
• Version pair: the release records B and E in the snapshots of the step. The catalog writes CellSource { extract_id: "planet", snapshot: E }.
• Store: the step has no client file, so a target never gets the updated planet. Another machine makes it again from the copies of osm-planet and osm-replication on the target.
• Content (feat(obc-data): landmark and peak content read the wiki shards (P6-3b) #2701's wiki readers): the planet content reads source.osm.pbf and source.poly of device/source/planet, and the wiki shards of every world cell that the manifest names. obc-pack content_view now also finds its .osm.pbf and .poly in step files. A Geofabrik content reads the same inputs as before.
• Catalog: the planet is no Geofabrik area. The catalog picks every Geofabrik area and the planet.
• Planner: it refuses a planet region as Invalid. This is the recommended answer to Q1. If the owner chooses the other answer, that is a later PR.
• An unsuited product refuses before a fetch of data:
  • freshness::discover lists with only small metadata fetched, and it now reports the products that do not suit the settings.
  • The update and --dry-run then fail with layers cannot be planned before the bulk listing. A planet update, and the "Planet" preset on the operator page, no longer download the planet and its diffs before the planner refuses.
  • The early refusal judges with the metadata in the store. So its fix says to change the region or the layers, or, when a reason names Geofabrik data, to run obc data fetch geofabrik-index and update again.
• First listing and review: the first listing of a planet fetches osm-planet and the diffs to learn B and E, as a Geofabrik listing fetches its extract. So a review lists them as downloads only for a later B or E, or on another machine. The size of the diffs is unknown.
• terrain-reference prepare: with a planet scope, it keeps every tile of the model.
• Q2: osm-planet refresh is now 28 days (it was 90).
• Dead rule removed: sources::status lost its base parameter and the "before osm-planet" rule. freshness.rs looked up the base with the params of the replication, so the rule never fired. The diffs now name their base in from=B.
• Spec: the region table and sentence, the stale row, the OSM planet section, the update rule for an unsuited product, the device section, a device/source/planet row, the content row, and the regenerated schemas.

Planet-scale risks (to measure, not solved here)

• osmium apply-changes holds every change in memory: ≈0.55 GB per daily diff, so ≈20 GB at 35 days. The 28-day refresh bounds it.
• Peak disk of a planet update: the planet (≈95 GB), the updated planet, the halves of the split and the leaves, so about 4 × 95 GB. The review's disk check does not count partial/. The collection keeps one updated planet in the store for each kept release.
• copy_key includes the version, so each weekly E uploads every diff since B again under sources/osm-replication/E/…. That is up to ≈35 × 105 MB ≈ 3.7 GB a week, and the copies of the serving and the rollback release overlap.
• device/source/planet and device/osm take hours, and a step cannot resume.
• The release manifest lists every file: ≈700k planet cell files (≈100 MB). The device/catalog options list each cell of each pick, about 2M ids, which grows the key and the manifest.
• GLO-30 for the planet: ≈26k tiles, which every machine downloads, because GLO-30 is not redistributed.
• The planet content compiles landmarks and peaks from the whole planet in one step, and reads every wiki shard of its manifest.
• Each leaf parses the whole manifest of each terrain-reference model that it meets. The bake fetches tiles one HTTPS GET at a time.
• Listing a planet makes steps for every world leaf and cell, including polar ones. The listing time and memory are not measured.

Lines

git diff --shortstat c056967fe..ee3c1883b: 23 files changed, 374 insertions(+), 132 deletions(-).

• First commit: +300 −84.
• Review round 1: +60 −39.
• Review round 2: +36 −31.
• About 100 lines are tests.
• About 15 spec lines are generated schema.

Tests

• region_sources::tests::a_planet_lists_one_source_step_that_reads_the_planet_and_the_diffs_after_it:
  • Without osm-planet in the store, the listing first asks for osm-planet.
  • Then a planet lists exactly one step, device/source/planet. It reads osm-planet B and osm-replication E with from=B, has no client file, and its outline is the world.
  • With a newer day of diffs in the store, target A, which serves the older day, moves to the newer day. Target B, which serves it, keeps the same step. B does not change for either.
• separate_input_layers_retain_distinct_versions_in_release_provenance now also asserts that a Geofabrik listing requests no osm-replication.
• obc-osm the_planet_step_applies_the_daily_diffs_to_the_planet: a tiny planet and two .osc.gz diffs, stored without a file extension. The output holds only the newest node, the deleted node is gone, and source.poly is the option.
• The planner test refuses a planet region as invalid, before a fetch.
• a_machine_with_less_memory_or_disk_than_the_estimates_refuses_the_update now has an unsuited product next to one that needs a bulk fetch that the registry cannot give. The update fails with layers cannot be planned: test (test): …, not with the error of that fetch. So it refuses before the bulk fetch.

Checks

Run on the head, with the shared lock and debug off:

• obc test -p obc-data: 181 passed, 1 skipped. The spec schema test passes.
• obc test -p obc-data-steps: 34 passed. builder/web/node_modules came from another worktree with the same lock file.
• cargo test -p obc-osm: 4 passed, with Osmium 1.16.0.
• cargo clippy -p obc-data -p obc-data-steps -p obc-osm --all-targets -- -D warnings: clean.
• cargo fmt --all --check: clean.
• obc suites check: test plan OK.

Run on the first commit only:

• obc test -p obc-pack: 95 passed.
• clippy on obc-pack: clean.
• tools/prose.py --check: pass.
• tools/check_data_sources.py: pass.
• tools/check_obc_data_licence.py: pass.
• tools.tests.test_fixtures: pass.
• svelte-check: no error from this change. Its 16 errors are the wasm packages that this machine does not build.

Left to CI: the rest of obc test affected, and the web suites.

Public docs: unchanged (contract change in specs/obc-data.md only).

Requirements: none. No planet target can publish until the planner builds the planet. I could not run obc req here.

🤖 Generated with Claude Code

https://claude.ai/code/session_01MdLzBWpdcQRZLATaT7poQZ

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}