feat(obc-data): landmark and peak content read the wiki shards (P6-3b)

[timohueser]

Part of #2648. Part of #2654.

Draft until the owner publishes the first wiki version to prod. Without a version of wiki, the landmark and peak content blocks on every target, so this merges only after prod lists one. The branch is based on develop after #2699 (merged as 1d4b111), so the diff holds only P6-3b.

Problem

The landmark and peak content of a region still asks Wikimedia itself: it captures facts by exact identity through tools/wikimedia_acquire.py and keeps them in the store. The prepared source wiki (#2695, #2699) now holds those facts for every OSM subject, in geographic shards on the target.

Behavior

• Readers. device/landmark-content[/<area>] and device/peak-content[/<area>] read wiki at the version that the target reads (prepared::resolve):
  • The step list reads the manifest of the version and names common.tar.zst plus the shards of the 2^20 µdeg cells that the region polygon touches, kept only when the manifest lists them.
  • Polygon cells, not the bounding box: Coverage::cells(20) already gives the cells that the .poly touches (boundary or centre inside), so this costs 3 lines.
  • The extract and .poly inputs stay. planner/search reads the content steps, as before.
• obc-pack. {landmark,peak}_content unpack the shards (tar under zstd; adds tar 0.4.46, MIT OR Apache-2.0; zstd was in the lockfile already) into the view, then run the views and the compile as before.
• Absent facts (feat(tools): port the Wikimedia dump processing into tools/wiki #2664 decision). shared::view, landmark_view and peak_view take a fact that the version lacks as no content and count it, instead of failing. The step writes {"absent": {"article": n, "category": n, "commons": n, "entity": n, "link": n}} to its metrics, so the receipt shows it, and prints one line with the counts to the run log. A fact with the status missing stays a confirmed outcome and is not counted. This also covers OSM identities newer than the version, and it never blocks the bake.
• In the producer, an absent fact is a fault. The export reads the database of the version, so the round-2 compile (feat(obc-data): wiki photos from Commons for the subjects of the compilers (P6-3a-2) #2699) turns a non-empty count into a refused compile: it gets the per-subject retry, the photo_compile_failures count and the majority stop. The summits take the link tags that subjects kept, so the export has a link fact of each; subjects keeps only linked subjects, so round 2 compiles every summit, and one whose link fact is absent fails its compile instead of being dropped.
• No version. The content layers block with "no target lists a version of wiki. Fix: Run obc data source prepare wiki." The device catalog (it needs every article layer) and planner/search block with them; the maps and the terrain still bake. A version whose manifest lists no shard of the area blocks them too: a snapshot input without files would read every file of the version.
• Staleness. A new version changes the snapshot input of the content steps. device/landmarks shows wiki 2026-10-01 (…) → 2026-11-01 (…); planner/search, device/catalog, planner/places and planner/index follow. No map, terrain or routing layer is stale.
• Rows. wikidata, wikipedia and commons stay in data/sources.toml as the named inputs of wiki, without a fetcher: a fetch of one fails with "source wikidata is read through wiki". A target keeps no copy of them.

Deleted

• host/obc-data/src/fetch/wikimedia.rs (778 lines); the Wikimedia arm and legacy in fetch/capture.rs; the Acquisition plumbing and the acquisition field of the run journal; the Wikimedia special case in cli/listing.rs.
• Http::fresh_download, the request and byte counters, and the rate limit: only the Wikimedia fetcher used them.
• gc's "resumable operation" for partial/wikimedia-* work.
• host/obc-data-steps/src/device/shared_content.rs (253 lines) and captures_at.
• obc-pack: CAPTURES, the capture filter of content_view, shared_files, and shared::bundles with its test.
• tools/wikimedia_acquire.py (536), tools/wikimedia_adopt.py (195) and their tests (475).
• landmark_capture.py stays until P8 (obc-bake, peak_capture.py); discover::candidates and peaks::discover stay for obc-bake.

Decisions to check

• The content layers publish nothing now (accepted in review). They published shared-content/ bundles so that another machine could restore the facts. With wiki on the target, the bundles had no reader, and bundles() would only write an empty folder. The content steps are Client::None; planner/search keeps its exemption in the "client files" rule, because a machine without the content compiles it again from the wiki shards on the target.
• Risk: the size of common.tar.zst. Every regional bake downloads all of it: the classes and every subject without a box or with a box over more than 16 cells. Its size is known only after the first version. If it is large, splitting it is a later change.
• With publish_text = false, devices get no landmarks. Landmarks require article text, so such a version gives only peaks. That is the owner's Q2.

Before merge

• On the first prod wiki version, measure the memory, disk and time of one region content step. Each step unpacks all of common.tar.zst, parses every fact, and copies every fact into the view (the entities three times: classes, locales, entities), twice per area. If that is large, make view copy only the facts that the chosen subjects reach.

Spec sections updated

• specs/obc-data.md: the opening reference to the content source; the redistribute row; the capture table (the Wikimedia row and the content bullet removed; the three rows have no fetcher); the device/landmark-content/device/peak-content row (inputs, outputs, metrics, no client); the paragraph after the device table (shards, absent facts, no version, no shard of the area); the run journal schema (acquisition removed).
• specs/wikimedia-content.md: the introduction; the producer/reader seam in place of the Store seam; "Acquisition adapter bytes" becomes "Compact facts" without the adapter's request, input and output formats, --adopt and exit status; the rounds sentence; Store retention and shared-content bundles replaced by one paragraph on versions on the target; the transfer metrics of the fetcher removed.

Public docs

Changed, in its own docs: commit: docs/content/software/data.md, "Shared landmark and peak content". host/obc-data/README.md uses geofabrik-extracts in the --refresh example.

Size

git diff --shortstat origin/develop...HEAD: 30 files changed, 575 insertions(+), 2,864 deletions(−). The review rounds add +67/−30 and +31/−23 (the last two rounds cancel out).

• Rust: about +470/−1,530. Python: −1,206 (the two tools and their tests). The rest is specs, docs and Cargo.lock.
• New and changed Rust: the device.rs reader and its tests; the obc-pack shard reader, the absent counts and their tests; test fixtures with a wiki version (device, planner, cells, planner_build).

Checks

All obc-data and obc-data-steps builds and tests ran under flock /home/user/cargo-target/obc-data.lock with debug=0, at c843b9b; at the head, obc test -p obc-data-steps (33 passed), its clippy and fmt ran again.

• obc test -p obc-data: 181 passed, 1 skipped. The spec's run schema was regenerated with OBC_UPDATE_DATA_SPEC=1 in the first round.
• obc test -p obc-data-steps: 33 passed. obc test fixtures -p obc-data-steps: passed (cells_fixture, terrain_fixture, planner_build). planner_build needs builder/web/node_modules; I linked one from another checkout for the run and removed it after. New and changed tests:
  • content_reads_the_extract_the_poly_and_the_wiki_shards_of_its_cells: the region reads common.tar.zst and its own cell, not the far cell of the manifest; no step reads wikidata, wikipedia or commons; without a version, and with a version that lists no shard of the region, the content layers block, and the catalog with them.
  • a_new_wiki_version_stales_the_landmarks_and_the_search: device + planner steps; the stale layers are exactly device/landmarks (the wiki cause), planner/search, device/catalog, planner/places and planner/index.
  • The scripted resume of the producer: the castle without facts counts as one failed compile.
  • planner_build reads a real, empty common shard.
  • The existing a_leaf_plans_from_the_manifest_of_a_model… still checks that no step reads a dtm-* source.
• obc test -p obc-pack: 95 passed; obc test fixtures -p obc-pack: passed (first round). New: a_fact_that_the_version_lacks_is_no_content_and_is_counted (landmark: entity, article, category, Commons; peak: link; the summit stays an OSM summit). The content step test unpacks a real shard and checks the metrics.
• tools.tests.test_wiki, tools.tests.test_data_registry: 18 OK.
• cargo clippy -p obc-data -p obc-data-steps -p obc-pack -p obc-bake --all-targets --features obc-data-steps/external-fixtures -- -D warnings: clean.
• cargo fmt --all -- --check, obc prose --check, obc suites check, tools/check_data_sources.py, tools/check_obc_data_licence.py, firmware/tools/check_dependencies.py, tools/licenses/gen-third-party.sh --check (THIRD-PARTY.md is current), python3 docs/build_docs.py --check-links: OK.
• Not changed: the test Manifest is still built in three places (device.rs unit tests, cells_fixture.rs, planner_build.rs). The unit tests of the lib and the integration tests are separate crates, so one helper would need a new public function or a shared test module.
• Left to CI: cargo deny (not installed here) and the rest of obc test affected. Cargo.lock changed, so CI selects the whole workspace.
• Not run: obc req. This machine has no agent token.

Requirements: none

🤖 Generated with Claude Code

https://claude.ai/code/session_01MdLzBWpdcQRZLATaT7poQZ

---

Generated by Claude Code

---

Generated by Claude Code

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[timohueser]

Merged develop and #2703 (codex/wiki-region-languages) into this branch. The owner bakes DACH from this branch with the first wiki version, which #2703 prepared. Until #2703 merges, this diff also shows #2703.

Conflicts, in 5e5f62b9d:

• fetch/http.rs: the bearer origin, headers and post_json of feat(data): prepare regional wiki data with automatic downloads #2703 stay. The rate limit, the counters and limited go, as in this PR. The download loop counts received again, because the size check and the progress line of feat(data): prepare regional wiki data with automatic downloads #2703 read it.
• device.rs: the prepared test helper of this PR stays, with the new languages field of Manifest. The test manifests of planner_build and cells_fixture get the field too.

Checks after the merge: cargo nextest run -p obc-data -p obc-data-steps -p obc-pack: 315 passed, 1 skipped. cargo check -p obc-data-steps --tests --features external-fixtures, clippy with -D warnings, cargo fmt --all --check, the repository tools Python suite, obc suites check and python3 docs/build_docs.py --check-links: passed.

🤖 Generated with Claude Code