Skip to content

feat(fs): support an optional indexed RomFS image - #2600

Open
Arkex-ai wants to merge 12 commits into
NotRexed:mainfrom
Arkex-ai:feat/optional-packed-romfs
Open

Arkex-ai wants to merge 12 commits into
NotRexed:mainfrom
Arkex-ai:feat/optional-packed-romfs

Conversation

@Arkex-ai

@Arkex-ai Arkex-ai commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Summary

A folder-backed RomFS needs a separate host file for every asset. This adds an optional read-only indexed image while preserving folder installs, mods and writable mounts. Related to #583, tracked in #1007; no loading-speed or memory improvement is claimed.

Technical changes

  • Add a streaming, copy-only packer with atomic publication and source-change checks, plus a bounded immutable index and positional native reads.
  • Preserve mod/prompt/overlay precedence and guest-range/directory locking. A null directory base now explicitly lists mod-only entries without probing a fictitious host folder.
  • Select the virtual native-font callback only for packed mode, before renderer initialization. Folder installs retain the original direct-file Reader and its size behavior; the 64 MiB virtual-input limit applies only to packed assets. Both main and translated-menu font callers support the image.
  • Keep renderer API 20, preserving main's version19 binding semantics. Main still uses19; whichever competing API20 change lands second must rebase/bump. Builder/launcher auto-selection is not included; acceptance of the opt-in CLI path remains a maintainer call on Feature requests (collected) #1007.

Validation

Current head 99ae3374aa554fcfca793f4bdcdb71579274f58e, rebased on main f3dd4486. All eleven code/test commits retain equivalent patches; the only conflict was the tested-configurations table, where upstream rows and the earlier two runtime rows are preserved. All 12 focused checks pass on Windows and Linux at this head. New hosted checks are pending.

Earlier native/game and full-suite evidence below remains pinned to 329b2a93/72f107bf; no fresh native build or gameplay is claimed after this rebase. Main now changes guest generation, so the earlier generated guest is not claimed current.

Earlier-head CI (72f107bf) Result / run
Linux AppImage success
Game-free source regressions success
Windows builder success
PR title success
  • 12 focused checks per OS pass, covering actual filesystem/loader, packer, mounts/startup, mods, native fonts, UTF-8 and links. Two causal controls per OS fail on their intended runtime assertion: old folder-font selection and host probes with a null mod base.
  • Earlier 329b2a93 matching Windows native build passes (184.4 s) and prompts are regenerated (123.8 s). Verified guest generation retains 23c50087 provenance: existing generator modules are unchanged; only the standalone packer is added.
  • Folder and image-only runs load the copied Kokiri checkpoint, log FZLTHB merged into the zh-Hans font atlas and exit0; the source checkpoint is preserved. Image mode has no loose RomFS root. Captures show normal gameplay, not native-menu/per-glyph visual coverage. Individual runs.
  • Full source suites at 329b2a93: Windows465/3/35, with all three failures reproduced on exact main; Linux478/1/24, failing the unchanged control fixture's newest-save ordering assertion. Three isolated repetitions each on exact main and candidate pass; the full-suite failure remains recorded. No all-green local-suite claim.

Earlier source bdefcade established complete private asset/guest-read parity (186,583 files and 598 directories); 836692b2 established packed translated-font acquisition. These are earlier-head results. Current Windows coverage is early Kokiri and font acquisition, not campaign, all scripts, Linux gameplay, physical ultrawide, loading speed or memory savings. Game inputs, generated images and private runtime material are excluded.

@Joey-Magnus Joey-Magnus left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I merged this into current main (it merges cleanly) and ran the RomFS, game font, vcore_fs_open and fs tests on Linux. All 12 pass. The reader in host/romfs_pack.h holds up. Names are capped at 0x300 bytes and validated. Entries must be sorted and have a parent directory, and file data has to be contiguous and inside the file. read_at still checks the guest range before romfs_pack_read. Before it goes in:

  • MKWVK_API_VERSION 20 is also claimed by #2604, #2317, #2261 and #2566. Whichever lands second has to move to 21.
  • The description claims no loading or memory gain. Nothing passes --romfs-pack yet either (the Builder and launcher don't know about it), so this is a second filesystem path behind a command-line flag. Whether that's wanted is a maintainer call on #1007.
  • In folder mode, the menu fonts now go through host->romfs_read too, with a new 64 MB cap in Reader::load (vk_game_font.h). That changes things for every player, not only image users.
  • OpenDirectory passes "<" as base_dir to hos_mods_dir_extras, so it relies on "<\name" never existing. That works, but a NULL base_dir handled in mods.c would be clearer.
  • The new docs/worklist.md row (0.4) sits above the 0.6 row, but the table is sorted by score.

@Arkex-ai

Copy link
Copy Markdown
Contributor Author

Thanks @Joey-Magnus. Updated at 72f107bf on main 51da0464:

  • Folder fonts now retain the legacy Reader path; the virtual callback and its 64 MiB input cap are packed-mode only. The actual loader regression catches the old unconditional selection.
  • hos_mods_dir_extras accepts a null base explicitly. Its real-loader test checks zero filesystem probes for that case and retains filtering for a real loose base; both causal controls are rejected on Windows/Linux.
  • The worklist row is sorted. API20 still sits on main's19, with the later-merge bump/rebase gate retained. No Builder/launcher default is added; the acceptance question remains with the maintainers on Feature requests (collected) #1007.

Twelve focused checks pass per OS. Matching native build/current prompts and both folder/image-only Kokiri runs pass, with FZLTHB acquisition logged and the source checkpoint preserved. The new captures do not establish native-menu glyph coverage. Full-suite failures and the isolated base controls are recorded in the description; new exact-head CI is starting.

Arkex-ai and others added 12 commits November 12, 2026 13:53
Keep the extracted folder available while adding copy-only image creation,
bounded positional reads, directory/mod integration, explicit startup selection
and native UI font asset reads through a versioned renderer interface.

Focused authored checks pass on Windows and Linux. Full regression suites,
native game build and game/corpus validation remain pending for this local
candidate; no loading-speed improvement is claimed.

Assisted-by: OpenAI Codex
Co-Authored-By: OpenAI Codex <noreply@openai.com>
Preserve source mutation checks while accepting directory metadata size changes. Exercise real mod indexing against an authored packed filesystem.

Assisted-by: OpenAI Codex

Co-Authored-By: OpenAI Codex <noreply@openai.com>
Reproduce the hosted Python 3.12 Windows short TEMP alias: canonical pack input previously bypassed lstat mutation fixtures. Preserve and assert mutation visits.

Assisted-by: OpenAI Codex
Keep main's version 19 binding semantics and add the RomFS reader callback as version 20.

Assisted-by: OpenAI Codex
Preserve main's directory synchronization on the packed-range rejection path. Verify the lock can be acquired again and reject an unlock-removal mutation on both platforms.

Assisted-by: OpenAI Codex
Preserve the new script-font fallback with image-backed RomFS. Exercise non-Latin fonts, both filename suffixes, missing assets, I/O refusal and recovery through callback and loose-folder sources.

Assisted-by: OpenAI Codex
Exercise the actual script_fonts caller with the real reader, authored dictionary and non-Latin TrueType font, without a loose root or system-font fallback. The new caller control fails before wiring the callback.

Assisted-by: OpenAI Codex

Co-Authored-By: OpenAI Codex <noreply@openai.com>
Select the virtual font reader only for an open packed image before renderer initialization, preserving the legacy folder Reader path and its size behavior. Use a nullable base path for mod-only enumeration and verify that it makes no host filesystem probes. Address the review worklist ordering and API collision documentation.

Assisted-by: OpenAI Codex
Record exact native/source evidence and limits, including the retained control-fixture failure and lack of native-menu visual coverage in the new captures.

Assisted-by: OpenAI Codex
@Arkex-ai
Arkex-ai force-pushed the feat/optional-packed-romfs branch from 72f107b to 99ae337 Compare November 12, 2026 16:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants