feat(tools): port the Wikimedia dump processing into tools/wiki

[timohueser]

Part of #2654.

Problem

The wiki prepared source needs the dump processing of PR #2647 without its TUI tab, its content command tree and its store coupling. Nothing calls this package yet. The mechanism (P6-1) and the producer (P6-3a) wire it in.

Behavior

python -m tools.wiki STEP runs one resumable step in a work directory. Every input is a local path: the producer downloads the dumps (fetch/http.rs), one at a time. inputs.json is {"wikidata": PATH, "wikidata_pages": PATH, "wikidata_redirects": PATH, "wikipedia": {LANG: PATH}, "langlinks": {LANG: PATH}}; the import requires the first three. A path may be a store object without a suffix: its first bytes name a bz2, gzip or tar file, and any other input is refused.

Step	Reads	Keeps
subjects	an OSM PBF (Osmium)	each object with a wikidata or wikipedia tag, with the box from osmium export
import	Wikidata page and redirect SQL, Wikidata JSON dump	pass 1: subject entities, P279 and P131 edges of every item, item sitelinks that tags name; pass 2: the class, country and locale closure
articles	langlinks SQL, Enterprise HTML	only the pages that a selected subject or a tag names; link facts for the tags
commons	Commons API through Capture, with its limits unchanged	Commons facts and 500 px thumbnails, only for the requests of the selected subjects
shards	the work catalog	cells/<i>-<j>.tar.zst per 2^20 µdeg cell, common.tar.zst, index.json

• Shard members are content/<kind>/<sha256>.json and content/assets/<sha256>, as obc_pack::landmarks::shared reads them now. A subject goes to each cell that its box touches, up to 16 cells (chosen by the orchestrator: it bounds the duplicate copies, and a box up to about 4° × 4° stays in cells). A subject without a box, or with a larger box, goes to common. Classes go to common. Cell names are zero-padded like obc_formats::grid::id_width (0300-0262).
• content/assets keeps the 500 px thumbnail; the compiler converts it with photo::prepare, as now.
• The producer calls commons once per round with all requests so far: categories, then the metadata of every candidate, then the files that the compilers choose. A stop, also "Wikimedia is busy", resumes when the step is called again. When the two fresh checks around a thumbnail agree, they become the Commons fact, so a description edit keeps the photo. A check that answers an older revision than the stored fact is no witness. A deleted file is stored as missing.
• articles and shards refuse to run before the Wikidata import is complete.
• Compressed input streams through lbzip2 or pigz when present, else through Python. A substring prefilter runs before json.loads. No table holds every Wikidata item. A shard streams from a temporary table. zstd runs at level 10 with threads: most bytes are JPEG thumbnails, which do not compress, so level 19 costs much time for a small gain on the JSON facts.
• An import that stops starts again with the same input. A changed input never resumes. Unresolved work is a failure row, never a missing fact: a cyclic redirect chain, a tag whose article fails to parse, a thumbnail whose metadata changes during the download. index.json counts subjects, subjects in common, subjects without a link and failures.
• The compact fact helpers and the Commons response parsing move from wikimedia_acquire.py into tools/wiki/facts.py. The old module imports them from there, so P6-3b can delete it.

Not ported from #2647: the downloader (the producer downloads), photo_transform_sha256, blocking on OSM identities outside coverage, the TUI tab, the content commands, admit/use/publish, settings/content-source.json, OBC_CONTENT_CONTROL.

Changes from #2647 that a reviewer should check:

• An Enterprise archive may hold more than one NDJSON member.
• SQL columns are read by the names that CREATE TABLE declares (page_is_redirect changed position between MediaWiki versions). The tuple parser no longer copies the line for each token.
• Article languages import in reverse order (es, fr, de, en), so that a page without an item can want its canonical page in a preferred language. The link of such a page names the requested titles of the page and its canonical page; the aliases map them to the canonical titles.
• The Commons step takes per-subject requests (categories, commons, files). The photo candidates and the photo choice stay compiler policy in Rust.

Known gap: a wikipedia tag that names a redirect of an item page (mostly peaks) resolves through the article to a QID that pass 1 did not keep. That link has an identity, but no entity or article fact. P6-3b readers treat absent facts as no content.

PR #2647 stays open until the producer (P6-3a) lands. The wiki row of P6-3a declares the Wikimedia hosts in data/sources.toml. The shard layout spec lands in P6-3a in specs/wikimedia-content.md.

Public docs

Unchanged.

Size

git diff --shortstat origin/develop...HEAD: 9 files changed, 1534 insertions(+), 189 deletions(-). All Python. tools/wiki/ has 1204 lines, of which about 165 moved from wikimedia_acquire.py. tools/tests/test_wiki.py has 314 lines.

Checks

• obc test affected --base origin/develop: python.repository-tools (379 tests, OK, with Osmium, zstd, lbzip2 and pigz present; the first commit also passed without lbzip2 and pigz), ci.data-sources-guard, ci.fixture-policy, ci.prose: all pass.
• obc suites check: OK.
• pyflakes on the changed Python: clean.
• obc ready --base origin/develop --dry-run: selects only the two gates above.
• No Cargo package changed, so no Rust suite runs. CI is the gate for the same selection.

Requirements: none

🤖 Generated with Claude Code

https://claude.ai/code/session_01MdLzBWpdcQRZLATaT7poQZ

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

• Configuration used: defaults
• Review profile: CHILL
• Plan: Advanced
• Run ID: 8884ab8d-3761-4fd2-8a48-24bcb7370b52

---

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

---

_{Comment @coderabbitai help to get the list of available commands.}