[klimovich008]

The game decodes every streamed file with its own lifted copy of zstd (ParallelDecompThread, about 3.5 ms of CPU a frame in the standard test). This adds an opt-in switch, HOS_NATIVE_ZSTD, that hands that decode to the host's libzstd 1.5.7, already vendored and linked for host/prompts.c. In a stub, libzstd takes about a third of the lifted decoder's time per MiB. In the game the difference is too small for 4 pairs of the standard test to resolve, so it stays default off.

This is a performance change, not a bug fix.

What changed

• host/hos_zstd.c (new): hos_zstd_stream, a replacement for the game's per-chunk wrapper f_80115c80 around zstd's buffer-less API (frame header from the first chunk, begin_usingDDict, then the nextSrcSizeToDecompress / decompressContinue loop with the game's staging buffer for a piece split between chunks). Every field of the game's stream object (state, dst, room, bytes still to come, staged fill) and every result (bytes to come, 0, -1, -2 for dstSize_tooSmall, -3 for input past the frame's end) is kept as the game's code leaves it. libzstd keeps its own ZSTD_DCtx per stream; the game's context is never written. The game's dictionary is used by reference: a ZSTD_DDict made from the game's content and checked against a copy of the game's entropy tables, so a dictionary remade at the same address is never served stale. Output pages are marked with HOS_WRITTEN after the writes. States 3 and 4 (failed, done) and anything unexpected (a misaligned workspace, an allocation failure) go back to the lifted code. Fixed tables of 64 streams and 64 dictionaries, under a spin lock; the peak in use is logged.
  • HOS_NATIVE_ZSTD=1: decodes on the host and gives the virtual core up while it decodes (hos_vcore_away / hos_vcore_back), as a sleep does.
  • HOS_NATIVE_ZSTD=2: decodes on the host and keeps the core.
  • unset or 0: off, the lifted code runs as before.
• nx2/oot_patches.py: patch_zstd_stream puts if (hos_zstd_stream(c)) return; first in f_80115c80, for the supported build only, counted by report() like the other hooks (warns unless found exactly once).
• nx2/build.py: ZSTD_HOST, the host files that get the zstd include path and headers in their dependency key (prompts.c, hos_zstd.c).
• tests/test_native_zstd.py (new, game-free; zstandard from RUN_ALL_MODULES makes the frames): compiles hos_zstd.c as it is, with the vendored zstd decoder, over a small fake guest window. It feeds frames with and without checksums, with a dictionary (entropy tables and raw content), empty and one byte long, RLE and raw blocks, whole and in chunks from 7 bytes to 128 KB + 3 and at random, so headers and blocks are split through the staging buffer. Each call's result is checked against the stream object's remaining-byte field and state byte, and the final state against what the game's wrapper leaves. Failures must answer as the game's do: wrong checksum or dictionary, bad magic, unknown content size -1; too little room -2; bytes after the frame -3. Also covered: a dictionary remade in place with the same id; the fallbacks (switch off, misaligned workspace, all 64 entries used); page marks after the last write; the core given up once a call with =1 and kept with =2. A broken copy (the staging fill never stored) runs inside the test and must fail.
• docs/tested-configurations.md: the three rows below.

How it was tested

• The game's own decoder, locally (no game data here or in the test): the game's f_80115c80 and its callees, run in Unicorn from a local image, against hos_zstd.c on the same inputs. 110 cases (the test's kinds plus larger and multi-frame sets), about 88,000 calls. Results, every stream field, the staging buffer, the dictionary region and the decoded output were identical at every call. Differences were only in the up-to-32 bytes just past the decoded output, which both decoders' fast copies over-write: the game's writes 3-11 bytes past it, libzstd up to 27. For a corrupt block, the partial output's last bytes differ; both report -1. A broken variant fails.
• python -I -B tests/run_all.py on the merged tree (Windows, llvm-mingw): 457 passed, 36 skipped, 1 failed. The failure is nx2/test_nup_paths.py, which cannot create its file-link fixture on this PC (no symlink right); this PR does not touch that test or what it covers.
• Stub timing (quiet machine, under the GPU lock; A11's 1,528 KiB synthetic set, levels 3-19, with and without dictionary and checksum): libzstd through hos_zstd.c at the build's flags (-O2 -march=x86-64-v3, ZSTD_DISABLE_ASM) takes 0.99 ms per MiB, against 3.09 ms for the lifted decoder at -O2 (PGO hot list: the game's Zstd decoder at the hot flags whatever the profile says #909) and 3.70 ms at -O1. That is 0.27-0.36 of the lifted -O2 time per frame.
• The standard test (Hyrule Field, camera turning at full speed, HOS_FPS_MAX=240, 20 s window): exe cacdc864b4239f07 (1bde186, before the merge with main), hookshot_vk.dll bfdac38fbf7d7c23. RX 9070 XT, Vulkan driver 0x8001a0, Ryzen 7 5700X3D, 64 GB, Windows 11, 5120x1440 (32:9). 2 warm-ups, then 4 alternating pairs per arm, same binaries:

arm	fps	CPU ms a frame	p99 ms	worst ms	frames > 100 / > 250 ms	game clock	save load s (median)	core escapes (with ParallelDecompThread holding)
off	44.05	115.05	29.1	220.6	2 / 0	0.98	29.0	4 (0)
=1 (core given up)	44.15	114.01	31.0	333.1	4 / 2	0.98	28.7	4 (0)
=2 (core kept)	44.70	112.71	29.5	56.7	0 / 0	1.00	28.5	3 (0)

The CPU-time differences (-1.0 and -2.3 ms a frame) are inside the run-to-run range (111.5-117.9 ms). The stub predicts about 2.4 ms a frame (3.54 ms x 0.68), which 4 pairs cannot resolve. Save load: -0.3 and -0.5 s, also inside the range. All of =1's frames over 100 ms came in its first run, which also had 5 virtual-core waits over 200 ms, against 1-2 in every other run. The control's first run (220.6 ms) and a warm-up (267.2 ms) spiked too, so I don't attribute them. The escapes were measured at main's 2 s escape default (#2127). None had ParallelDecompThread holding a core, so giving the core up buys nothing measurable here. =2 is the mode to try.

These runs were on a main from before #2602, which fixed a regression on main (this machine: about 41 → 53 fps and 125 → 78 ms of CPU a frame on the standard test), so every arm ran on the slower main. I can rerun the A/B on a main with #2602 if wanted.

Not covered

• Linux (the code has no OS branch, but it was not built or run there), NVIDIA, Intel, the Steam Deck, other CPUs.
• More than 4 pairs, other scenes, first visits with cold caches, sessions longer than a minute.
• The second zstd path in the game, f_8010b380 (whole-buffer decode): not hooked. A follow-up if this one proves out.
• libzstd's assembly Huffman decoder (ZSTD_DISABLE_ASM stays, as for the rest of the build): a follow-up.
• Corrupt frames: the partial output's last bytes before the error differ from the game's decoder (both return -1).

Related

• PGO hot list: the game's Zstd decoder at the hot flags whatever the profile says #909 (the lifted decoder at -O2, merged).
• fix(vcores): no core held while a movie picture is decoded (#1929) #2322 / fix(vcores): no core held while OpenFile builds the button prompts (#1929) #2301: the same pattern of a host routine in place of a lifted one, behind a switch.
• Virtual cores: a thread that runs without a core (escape) shares the held core's per-core state, crashing after the intro #1929 and CIB's rule on virtual-core escapes: the core give-up here is measured at the 2 s default only, and is a by-product, not the point.
• Prior art: zortos293/Lichii#869 puts this game's zstd decode on the host's libzstd in an emulator fork, one level lower (the block decoder, with the game's history buffer). This PR replaces the game's own per-chunk wrapper instead, so libzstd keeps its own state. I did not read its code; it's credited as related work.

🤖 Generated with Claude Code

[CIB]

Linux test

I built and ran this on Linux (the part the PR says is not covered).

• Arch Linux, kernel 6.19.8, Ryzen 7 5700X3D, RTX 4090, NVIDIA 590.48.01 (open kernel module), Vulkan 1.4.
• Game-free test: python3 -I -B tests/test_native_zstd.py -> both cases PASS (zstandard 0.25.0).
• Build: passes on current main fce53453 with the host file and the build.py change. My shared lift predates the PR, so I applied patch_zstd_stream to the generated zstd stream wrapper manually; a fresh lift will do it. (Using the PR's own older base in that lift hit an unrelated a64_stubs/nn_audio duplicate, so the runs below are current main + the PR's host side.)
• In-game: fresh profile, anonymous window, 45 s from the same Start, 2 runs per arm:

HOS_NATIVE_ZSTD	crashes	fps	peak host streams
off	0/2	60.0	-
1	0/2	60.0	3-4
2	0/2	59.9-60.0	2-4

The hook is active (zstd: the game's stream decode on the host's libzstd 1.5.7 ...) and no arm crashed or regressed.

Caveat: the intro is vsync-capped at 60, so this is not a CPU-time measurement. I did not run the Hyrule Field standard test or measure CPU ms/frame; the 2.4 ms/frame prediction remains untested on Linux. If a Linux perf row is wanted, I can run an uncapped Hyrule Field A/B with process CPU time, but it needs the save/camera automation.

No game files or saves used.

---

🤖 Generated with opencode using DeepSeek V4.1 Flash.

[Joey-Magnus]

I merged this into current main and ran tests/test_native_zstd.py (zstandard 0.25.0) and test_patch_hooks_defined.py: both pass, and so does the broken-copy control. I read host/hos_zstd.c. Off, hos_zstd_stream returns after a cached env_on (line 295), so the default path costs one call per chunk. The stream table only frees an entry on failure, on done, or when a new frame starts. An object the game abandons mid-stream keeps its entry, which is documented, and after 64 of those new streams fall back to the lifted decoder. The DDict is checked against a copy of the game's entropy tables, so one remade in place isn't served stale. CIB's Linux run covers the "not covered: Linux" line.

I see no correctness problem. What's left is whether to carry 336 lines of host code and a hook in the lifted f_80115c80 for a gain 4 pairs couldn't resolve. Those runs were also on main before #2602. With #2602 CPU time a frame is about 78 ms instead of 115, so the predicted ~2.4 ms is a bigger share and might show. I'd rerun =2 against off on current main before deciding.

[jordanella]

Test request T-M4T5: pass

Hardware: NVIDIA GeForce RTX 3080 (10 GB), driver 621.14, AMD Ryzen 9 3900X (12 cores, 24 threads), Windows 11. Built f0f7dfd (the PR head; v2.1.0-rc.3-260-gf0f7dfda), with the Builder's llvm-mingw; the build log names the hook: oot_patches: the zstd stream decode's host hook in code_000.c.

What I checked: the same build, switch off against HOS_NATIVE_ZSTD=2 and =1, alternating (off, =2, =1, off, =2, =1). Each run starts headless from a save through the control bridge with HOS_FPS_MAX=240, teleports to Hyrule Field (Lost Woods bridge), settles 10 s, then holds the right stick fully right (the camera turning at full speed) and samples the status fps once a second for 20 s.

Result: no difference beyond run-to-run spread, and nothing broken.

arm	run 1 median fps (min-max)	run 2 median fps (min-max)
off	27.2 (13.6-37.0)	25.2 (14.7-32.1)
=2 (core kept)	25.2 (11.7-31.2)	26.6 (14.7-31.1)
=1 (core given up)	not counted: the teleport was refused (an event was playing), so it sampled Kokiri Forest	26.6 (12.7-36.9)

• No device lost, crash or exception line in any run.

• The host decoder was in use in every =1 and =2 run, and in none of the off runs:

  zstd: the game's stream decode on the host's libzstd 1.5.7 (HOS_NATIVE_ZSTD=2)
  zstd: 1 stream(s) decoding on the host at once
  zstd: 2 stream(s) decoding on the host at once
  

  =1 reached 3 streams at once; nothing came near the 64-entry tables.

• This scene is CPU-bound here (about 26 fps against the 44 in your RX 9070 XT / 5700X3D table), so a ~2 ms/frame saving would show as about 1.3 fps, inside the 2 fps spread between the two off runs. Two runs per arm can't resolve it; it confirms no regression on NVIDIA and a Zen 2 CPU, not a gain.

[hckrman101]

Test request T-M4T5: pass

Hardware: AMD Radeon RX 580 8 GB (RADV Polaris10), Mesa 26.2.6, CachyOS Linux, 8 threads, 15 GB RAM. Built f0f7dfd (the PR head). One build; HOS_NATIVE_ZSTD unset, =1 and =2 alternated (3 runs each, the order rotated each round, after one warm-up run).

What I checked: start to play from the test save, then Debug teleports Kokiri Forest → Hyrule Field → Market, with a 34 s window in each scene, the camera turning (HOS_FPS_MAX=240 RX_VK_FRAMEGEN=0 RX_VK_PRECOMPILE=0, headless). CPU time of the process and of the four ParallelDecompThread threads, sampled from /proc once a second. Loads are timed from the log: game start to first present and to the first frame of play; each teleport from its request to the first frame of 300+ draws after the loading screen. Frame times from RX_VK_FRAME_TRACE.

• libzstd: the build uses the vendored copy. The log says zstd: the game's stream decode on the host's libzstd 1.5.7. The binary has no runtime libzstd dependency (ldd shows none; the only dynamic zstd symbols are libzstd's two weak trace hooks). The system libzstd here is also 1.5.7, but it isn't used, so a missing or different system library can't affect it.
• tests/test_native_zstd.py passes on Linux (gcc, the PR's own fixture, including the fallback cases and the broken-copy control): "the host's zstd stream decode answers, writes and marks as the game's wrapper does"; "the broken copy fails".

Results, per run → median:

	off	=1	=2
decode-thread CPU, start → play	12.4 / 12.0 / 7.2 → 12.0 s	4.4 / 4.0 / 3.7 → 4.0 s	5.6 / 5.3 / 4.6 → 5.3 s
decode-thread CPU, Field load	1.03 / 1.56 / 1.24 → 1.24 s	0.41 / 0.55 / 0.65 → 0.55 s	0.60 / 0.66 / 0.63 → 0.63 s
decode-thread CPU, Market load	5.31 / 2.89 / 4.28 → 4.28 s	2.23 / 1.40 / 2.03 → 2.03 s	1.11 / 1.39 / 1.51 → 1.39 s
start → first present	24.5 / 14.5 / 19.7 → 19.7 s	27.3 / 13.6 / 13.0 → 13.6 s	25.2 / 16.4 / 13.9 → 16.4 s
start → in play (save loaded)	78.1 / 63.7 / 63.8 → 63.8 s	74.6 / 55.4 / 49.0 → 55.4 s	72.3 / 64.9 / 49.4 → 64.9 s
Field teleport → play	12.5 / 15.0 / 12.2 → 12.5 s	12.0 / 10.3 / 12.8 → 12.0 s	12.8 / 10.7 / 10.0 → 10.7 s
Market teleport → play	29.9 / 25.2 / 30.1 → 29.9 s	32.3 / 21.7 / 22.6 → 22.6 s	25.6 / 28.1 / 19.3 → 25.6 s
Kokiri fps median; 1% low	21.9; 8.1	21.6; 9.3	21.6; 9.6
Field fps median; 1% low; frames > 100 ms	32.2; 3.8; 45	31.1; 4.0; 45	30.5; 3.8; 46
Market fps median; 1% low; frames > 100 ms	39.2; 3.6; 72	36.9; 3.5; 65	38.9; 3.8; 73

• The decode threads use 2-3x less CPU with it on, in every run. They do nothing in the play windows (0.000 cores in all arms), so the saving is all in loads.
• The load times move the same way but stay inside the run-to-run range. The first round was slower for every arm (a colder page cache), and the harness presses A once a second to reach play, so start → play is coarse. The Field and Market hitches (frames > 100 ms) are pipeline compiles with background precompile off, the same in every arm.
• fps in play is unchanged within noise. =1's Market median came out a little lower in all three runs (37.8 / 34.3 / 36.9 vs 40.2 / 39.2 / 38.9 off), and =2 matched off. The decode threads are idle in those windows, so I don't think it's the decode; with 3 runs I can't tell.

Correctness:

• Every on run logs the switch line and then 1, 2, 3 stream(s) decoding on the host at once. No stream fell back to the lifted code (a stream decodes lifted: 0 in all runs).
• Screenshots once a second, =1 vs off, no camera turn, matched by time after each scene drew: mean absolute difference 3-8 of 255 and 0.4-3% of pixels more than 32 off. That's the same as two runs of one build (wind and animation), with the same mean brightness per scene. No missing or corrupted textures or models in Kokiri, the Field or the Market.
• No crash, hang or error in 12 runs (=1 / =2 / off plus the warm-up and shot runs).
• Not covered: audio. These runs are headless with audio off (HOS_AUDIO=0), so audio streams weren't checked.