Skip to content

Vulkan: pool imports take the game pools across a chunk's end whole, and commit their pages in one call - #1267

Merged
hckrman101 merged 1 commit into
NotRexed:mainfrom
jordanella:import-span
Nov 9, 2026
Merged

hckrman101 merged 1 commit into
NotRexed:mainfrom
jordanella:import-span

Conversation

@jordanella

Copy link
Copy Markdown
Contributor

Two changes to how guest memory is imported into Vulkan, both aimed at the stalls a pool import puts on the replay thread.

1. A game pool across a chunk's end is imported in one piece. Imports are made in 16 MB chunks, at the first binding in each. A game pool that runs across a chunk's end therefore had its two chunks imported separately, at the first binding in each. The first binding that crosses the end then imported both chunks again as one pool, synchronously, on the replay thread. In Kokiri Forest this happens with a 16 MB pool at 0x1021745000, bound at 0x1021ffff00+0x160 across 0x1022000000: 130–147 ms, the one big hitch of a panning run. Of the 4547 pools the game makes in two minutes, 88 cross a chunk's end.

span_game_pools now grows an import's chunk-aligned range over any game pool that runs across its ends, so long as the range stays within 64 MB (RX_VK_IMPORT_SPAN_MB; 0 turns it off). Larger pools are chunked as before. If a spanned import fails, the import is made again without the span, so a pool that imported before still imports.

2. import_pool commits the range with one VirtualAlloc call instead of querying it one region at a time. In the game's 512 GB reservation the queries took ~46 ms of an import's ~95 ms, and the single call takes ~3 ms. This is safe because:

  • the guest window is a single reservation;
  • committing pages that are already committed leaves them as they are;
  • guest pages are read-write (only the RX_NVN_WATCH_* diagnostics protect some);
  • pages under an import stay committed, as hos_mem.c and hos_svc.c zero them instead of decommitting.

Where the call fails, the old query loop runs as before.

Measured

Ryzen 9 3900X, RTX 3080, Windows 11, on main c94d80e. Kokiri Forest after loading a save, headless, input timed in game frames (HOS_INPUT_FRAME_CLOCK=1), presents 3120–5520. Three rounds, each building main and then this branch, each run standing still and panning (RX_VK_SLOW_LOG=35). Something outside the game slowed both builds in round 2 (main panning 32.1 fps, this branch standing 34.1), so its fps numbers aren't comparable, but its hitches are included.

main this PR
panning: frames over 100 ms (3 runs) 0 / 4 / 1 0 / 0 / 0
panning: worst frame 89 / 244 / 156 ms 86 / 88 / 77 ms
panning: 0.1% low 13.1 / 4.6 / 8.3 fps 12.0 / 11.6 / 13.1 fps
imports per session 90–97 45–47
time in imports per session 4.2–5.2 s 1.7–3.1 s
an import, on average (commit part) 89–103 ms (44–50 ms) 45–73 ms (~3 ms)
fps, rounds 1 and 3 (standing / panning) 39.8, 38.9 / 37.0, 36.5 39.5, 39.4 / 36.7, 36.9

Costs and limits

  • More host memory imported: ~1520 MB live against ~1410 MB (+7%), because a span takes in whole neighbouring chunks.
  • Larger first imports: a session makes many more 32–160 MB imports and far fewer 16 MB ones. In four sessions each, the slowest single import was 152 ms against 182 ms on main, and the one-call commit shortens them all.
  • One hitch remains: a chunk whose first binding happens during play. In Kokiri that's a 4 KB pool at 0x1034ef6000, made during the load, alone in its chunk. That import is still synchronous, but about half as long as before. Importing chunks when pools are made isn't a fix: the game's pools touch 419 chunks, against ~45 ever imported.
  • Tested only on Windows with NVIDIA. On Linux the one-call commit may fall back to the query loop, and nothing else changes there. Nothing new runs in the background (background imports are what slowed AMD before).
  • Log line: the import line now names the range imported (... imported in X ms as lo..hi ...). Nothing in the repository parses it.

Tests

  • tests/test_import_span.py checks span_game_pools on its own: the Kokiri layout from both sides, growth chained across several pools up to the cap, a long pool behind a shorter one (the host's pools can overlap), pools only touching an end, the cap, and 0 for off.
  • nx2/test_import_windows.py now has the new functions in its fixture. It also exercises the span through pool_lookup: a pool across a chunk's end imported as both chunks, and a spanned import over the device limit made again without the span.

Mutation-checked: removing the span, the retry, the cap, the overlap handling or the chunk alignment each fails a test.

The branch is rebased onto main 96fece9 (v2.0.11 merged). Built there, a panning run had no frame over 100 ms, with 50 imports and 1.7 s spent in them.

python -I -B tests/run_all.py: the same 14 failures as main c94d80e on this machine (missing tools, and an older g++ that rejects volatile compound assignment). Everything else passes, including the new test.

🤖 Generated with Claude Code

…and commit their pages in one call

A game pool across a 16 MB chunk's end had each chunk imported on its own, at the first binding in it, and a later
binding across the end imported the two again as one, on the replay thread: OoT's 16 MB pool at 0x1021745000, bound at
0x1021ffff00+0x160 across 0x1022000000, 130-147 ms, the hitch of a panning run in Kokiri. span_game_pools grows an
import's chunk-aligned range over the game pools across its ends, while it stays within 64 MB (RX_VK_IMPORT_SPAN_MB;
0: off), so those chunks come in one import. A spanned import that fails is made again without the span.

import_pool commits the range with one VirtualAlloc instead of querying it a region at a time: the queries were ~46 ms
of an import's ~95 ms in the game's 512 GB reservation, the call ~3 ms. The guest window is one reservation, committing
committed pages leaves them as they are, and guest pages are read-write (only the RX_NVN_WATCH_* diagnostics protect
some); where the call fails, the query loop as before.

Ryzen 9 3900X, RTX 3080, Kokiri Forest panning, three rounds against main: no frame over 100 ms (main: 0, 4, 1),
worst 77-88 ms (main 89-244); imports per session 45-47 (main 90-97), 1.7-3.1 s of them (main 4.2-5.2 s). fps the
same. ~7% more host memory imported (~1520 against ~1410 MB live).

tests/test_import_span.py: span_game_pools on its own; nx2/test_import_windows.py: its functions in the fixture, and
pool_lookup's span and its retry.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@lucasmazzoni45-del

Copy link
Copy Markdown
Contributor

AMD / Windows data for this PR, since the numbers above are NVIDIA: RX 7800 XT 16 GB, Ryzen 7 5800X3D, Windows 11, 2560x1440, frame generation x2, v2.0.11 with this PR merged on top (it merged cleanly; I also had #1150 and a few AMD changes of my own in the same build, so this is not a clean single-PR A/B). Normal play with a save, not a scripted route, so the two sessions are not the same scenes. Both started within the same hour on the same machine, and the pool imports are counted by the log's own (N imports, M ms in all):

up to present ~9700 v2.0.11 without the PR with the PR
pool imports 106 71
time in imports (log's running total) 1654 ms 952 ms
imports of 30 ms or more 29 2
longest single import 100 ms 46 ms
mean import 19.5 ms 14.7 ms

A second session without it (21:45) is close to the first: 121 imports, 2081 ms, 30 imports of 30 ms or more, longest 90 ms. So on this card too the PR cuts the number of imports and nearly all of the 30-100 ms ones.

Frame times are less clear: frames over 40 ms went from 100 in 13615 presents to 50 in 12316, but intervals over 22 ms went from 4.9% to 11.2%. The session with the PR was the first one after a DLL change, so the pipeline cache was cold (compiles in the background), which probably explains the second number; I would not read it as a regression, nor as a result. No device lost and no new warning in the log.

I have not measured the 7% extra host memory the description mentions. RX_VK_IMPORT_SPAN_MB=0 turns it off if anyone wants to compare on the same save.

@hckrman101
hckrman101 merged commit 373adca into NotRexed:main Nov 9, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants