Add manual Wikimedia snapshot preparation for landmark and peak bakes

[timohueser]

New landmark and peak preparation still depends on Wikimedia API availability. This PR adds a manual bulk-snapshot refresh. An operator prepares reusable content on a machine with enough disk space, reviews its immutable publication, and publishes it to R2. Regional bakes select that content version and resolve their facts without Wikimedia API requests.

Changes

• Add obc data content configure|prepare|status|plan|publish|use and tab 6 · Content in the TUI. Preparation and publication use retained operations. Closing the terminal does not stop an admitted worker; stopping preparation retains successful work.
• Stream Wikidata JSON, Wikipedia Enterprise HTML, and identity-proof SQL archives into a temporary SQLite catalogue. Inputs can be local files or configured HTTPS URLs. Downloads resume against matching upstream validators. Processing keeps a 4 GiB disk reserve and does not extract complete archives.
• Discover exact subjects from configured OSM inputs or explicit identities. Reuse native classification, language, article, photo, and credit rules. Only manual preparation permits Commons metadata and selected 500 px thumbnail requests. Convert selected photos to the existing 216 × 240 dithered RGB222 format.
• Publish a SQLite lookup index, compact JSON fact bundles of at most 16 MiB, final device photos, and a version manifest. Objects use SHA-256 identities. Publication verifies objects and writes the manifest last. It changes no Live pointer and removes no remote objects.
• Restore a selected manifest and index on a bake machine. Download required fact bundles and photos as needed, then reuse the local Store. Incomplete identity proofs and uncovered OSM identities remain explicit preparation failures; the selected snapshot path has no Wikimedia API fallback.
• Preserve rendered article revisions, source dates, credits, identity proofs, and the photo-transform identity. Reject incompatible prepared photos before fetching data. Keep selected versions and interrupted work reachable through Store cleanup.
• Use platform certificate trust for Rust HTTP fetches.

Validation

Completed during implementation:

• Repository tools Python suite: 382 tests passed, including compressed imports, SQL redirects and language links, revision proofs, interruption, compact export, and offline restoration.
• obc test -p obc-data --lib: 213 tests passed.
• obc test -p obc-data-steps: 39 tests passed across five binaries; doc tests passed with no cases.
• obc test -p obc-pack --lib: 94 tests passed.
• cargo clippy -p obc-data -p obc-data-steps -p obc-pack --all-targets -- -D warnings: passed.
• CLI fixture preparation, publication to a local bucket, restoration into a fresh Store, native content selection and compilation, and actual TUI preparation/review: passed. The publication test does not write to R2.
• Small prefixes from real Wikipedia Enterprise and current multiline SQL archives validate the input parsers. These checks are not complete source scans.

The rebase adds unrelated planner and iOS changes and does not change the tested implementation files. After the rebase, cargo fmt --all -- --check, obc suites check, python3 tools/check_data_sources.py, python3 tools/check_obc_data_licence.py, obc prose --check, python3 docs/build_docs.py --check-links, and diff checks pass.

Storage and remaining validation

The three-record CLI fixture publishes three objects: an 8,192-byte SQLite index, a 1,484-byte fact bundle, and a 2,976-byte manifest. It contains no photos. Each prepared device photo occupies 51,840 bytes. These fixture sizes do not estimate worldwide storage.

Fact bundles and device photo objects are currently uncompressed. Compression remains a follow-up before worldwide publication. Raw archives, temporary catalogues, API request logs, and source thumbnails are not publication outputs.

A real Teningen bake, real bulk-source photo coverage and visual quality, worldwide retained/temporary storage and timing, overlapping-region measurements, and actual R2 publication remain unvalidated. No VPS changes, Live apply, offload, or full-world download ran. The full workspace, firmware, iOS, and web suites were not run locally. CI and independent review remain the merge gates.

Public docs changed in a separate docs: commit. The README, public data guide, content contract, and generated CLI schema describe the new workflow.

Requirements: SYS-007, SYS-008, SYS-014, SYS-034

These IDs follow the shared acquisition and Store integration PRs #2630 and #2632. The verification console cannot be read in this workspace because its agent token is absent. Requirement-plan review remains open; no coverage linkage or plan update is claimed.

Related: #2410, #2612, #2627. This PR does not close the outstanding real-data acceptance work.

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

• Configuration used: defaults
• Review profile: CHILL
• Plan: Advanced
• Run ID: 9be984e5-8349-4dfb-be5f-8b6901876608

---

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

---

_{Comment @coderabbitai help to get the list of available commands.}

[timohueser]

Closed as superseded by the prepared source wiki: #2695, #2699, #2703 (merged), and the readers in #2701.

This PR added obc data content with its own SQLite index, fact bundles and publication, so that bakes need no Wikimedia API. wiki now does that job inside obc data source prepare. It streams the same dumps (Wikidata JSON, Enterprise HTML, SQL tables) into a resumable catalog, asks Commons only for the selected subjects, and publishes geographic shards as a version on the target. The first real version (DACH, en,de) is being prepared to prod now. #2701 deletes the Wikimedia API path that this PR worked around.

The branch is 273 commits behind develop and its deny check failed. Ideas worth keeping, if a later change needs them: device photos prepared ahead of time (216 × 240 RGB222), and an upstream checksum check of the downloaded archives (also a follow-up on #2703).

🤖 Generated with Claude Code