perf(obc-pack): the content steps find sources and facts by key

[timohueser]

Summary

The in-process steps device/landmark-content and device/peak-content scanned a whole collection once per item. On a large region these scans were quadratic. This change builds each lookup once.

• read_pinned (landmarks/mod.rs): a new Sources type holds the pinned sources in a map by path, under the snapshot root. The root is canonicalized once. Sources::new also rejects a duplicate path, so load_snapshot does not do that check separately. The (root, sources) parameter pair is now one &Sources.
• shared::view: a private read reads and parses each fact file once. It returns the facts and a pin (key, digest, length) for each file. landmark_view and peak_view give these to view_of, so they do not read the files again. An asset fact is found by its digest in a map. A set holds the paths that can repeat (canonical aliases and proofs). The old code did a scan of sources for this.
• peaks::compile: resolutions are grouped by node ID, and places are put in a map by identity. The first place of an identity wins, as find did before.

The public signatures of shared::{facts, view, landmark_view, peak_view} do not change.

Output

The output bytes do not change. A synthetic bake of 2,000 places ran with the old code and with the new code, from the same inputs:

• The view manifest.json for landmarks and for peaks is byte-identical.
• content.json and peaks.json are different in policy_sha256 only. That field hashes the compiler source (include_bytes!("mod.rs") and the other files), so it changes with any edit. All other fields are equal.

The existing landmark and peak tests also pin the output.

Measurement

I did not have a local data store, so I could not do a DACH bake. I used a synthetic region instead: N places, each with one entity, one article, one Commons file, one asset, and one link fact. The landmark view and compile ran on it, then the peak view and compile. Release build on the dev laptop (15 GB RAM). Times are in seconds.

Places	Step	Before view	Before compile	After view	After compile
2,000	landmarks	6.02	1.59	0.73	0.94
2,000	peaks	5.14	1.70	0.68	0.91
6,000	landmarks	59.44	7.10	2.17	2.90
6,000	peaks	57.68	26.44	2.12	2.78
20,000	landmarks	—	—	7.59	9.93
20,000	peaks	—	—	7.16	9.36

The time is now linear in the number of places. Before, three times the places took about ten times as long. I did not run the 20,000 baseline. If the trend continues, it would take approximately 10 minutes for each view.

A remaining scan: shared::subject falls back to a scan of all facts when an ID is not an entity key. In a peak view this occurs for a summit whose link resolves to a non-QID identity (wiki-<lang>-<page>). That case is rare, so this change does not address it.

Checks

• obc test -p obc-pack: 95 passed.
• cargo clippy -p obc-pack --all-targets -- -D warnings: clean.
• cargo fmt -p obc-pack --check: clean.
• Not run: obc test affected, and any check outside obc-pack. CI is the gate.

Public docs did not change.

Requirements: none

🤖 Generated with Claude Code

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

• Configuration used: defaults
• Review profile: CHILL
• Plan: Advanced
• Run ID: f4939ea3-2b42-47ef-8770-17b83cc3a148

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}