Skip to content

fix(vk): read the registrations kept from before the renderer started from the host's copies (#1117) - #2289

Merged
hckrman101 merged 1 commit into
NotRexed:mainfrom
mooki000:fix/vk-early-registration-copies
Nov 11, 2026
Merged

hckrman101 merged 1 commit into
NotRexed:mainfrom
mooki000:fix/vk-early-registration-copies

Conversation

@mooki000

@mooki000 mooki000 commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

The textures and samplers the game registers before the renderer has started can reach the Vulkan back end as all zeros: the game's sampler 258 then filters nearest instead of trilinear and a few texture slots have no image. Six of the logs attached to #1117 show exactly that, and they are the logs of the two reporters with the blocky light halos.

Refs #1117.

The cause

  • The game registers its first textures and samplers before the window's thread calls rx_start. host/rx_backend.c keeps those calls (keep_early) and afterwards hands them to the renderer as pointers to its own copies on the heap (flush_early, forward): the log's rx: 123 texture and sampler calls made before the renderer started ..., given to it.
  • descriptor_ok (gpu/vk/vk_bank.cpp) decides which memory a registration is copied from. A pointer outside the guest window was first given to host->gpu_ptr() and only then treated as host memory.
  • With the native NVN layer (nvn_native_on() returns 1), hos_gpu_ptr answers through nvn_gpu_guest (host/nvn_native.c): every value from 16 MB (HOS_SYSTEM_BASE) to 512 GB (HOS_SPACE_END) is a guest address, and a value from 1 TB (VBASE) up is one where a virtual pool has a page mapped. A kept copy at such a host address was therefore read from guest memory at that address, not from the copy.
  • A registration's pointer is never a GPU address. nvnTexturePoolRegisterTexture, nvnTexturePoolRegisterImage and nvnSamplerPoolRegisterSampler pass host pointers inside the guest window (P(i)), which descriptor_ok's first check takes; forward passes the kept copies. So asking gpu_ptr could only ever misread a host pointer.

What changed

  • gpu/vk/vk_bank.cpp: descriptor_ok no longer asks host->gpu_ptr(). A pointer inside the guest window is taken as before; any other pointer is the host's own memory when VirtualQuery says it is committed and readable, and is refused otherwise (as before for memory that is not there).
  • tests/test_registration_host_copies.py with tests/registration_host_copies_test.cpp: runs descriptor_ok, cut from the source, against nvn_gpu_guest, cut from the source, with the guest window's real bounds read from the headers. Seven cases: a pointer in the guest window up to its last bytes; a kept copy at 0x9873eec (the address from the log below); a kept copy at a mapped virtual pool's address above 1 TB; a copy above the window; no memory at the address; memory that is not readable; a region that ends inside the registration. It also checks that the four registration entries still go through descriptor_ok.

vk_bank.cpp is one of the files the shader cache's stamp is made from (nx2/build.py), so the program cache is prepared again at the first start after this change, as after any change to that file.

Evidence: the reporters' logs

Counted by a script with asserts over the 13 logs attached to #1117 and #1727 (7 and 6):

logs version card sampler 258 texture slots registered all zero
wartechnology: 4 of 5 v2.0.7 RTX 5080 nearest, every field zero 0x102, 0x104, 0x107, 0x124
wartechnology: 1 of 5 (2160 presents) v2.0.7 RTX 5080 trilinear none
dmajchrowski: 2 of 2 (with the blocks, and with RX_VK_FORCE_LINEAR=1) v2.1.0 RTX 5080 nearest, every field zero 0x102, 0x107, 0x124 (and 0x104 in the second)
arukode: 5 of 5 (#1727) v2.0.12 RX 9060 XT trilinear none
liveat40watt: 1 of 1 (#1727) v2.0.14 RTX 3080 Ti trilinear none
  • All 13 logs kept 123 calls. The six with the zero sampler are the six with the zero texture slots; the seven others have neither.
  • The lines, from dmajchrowski's log of the run with the blocks: vk: sampler 258 is nearest: min filter 0, mag filter 0, wrap 0/0/0, lod 0..0 bias 0, aniso 0, compare 0 and vk: texture slot 0x102 (handle 0x10200102): no image for it; NVN target 0 format 0 0x0x0 levels 0 gpu 0, .... In the logs without them the first sampler anisotropy line is the game asks for 1 (min filter 5, ...).
  • Where the nearest samplers are logged with their values (v2.0.12 on), the all-zero set is the only one the logs with the fault have and the others lack.
  • That the zeros come from guest memory is shown by one log: in dmajchrowski's run with the blocks the host thread logs mem: 0x9873eec committed on first touch (nothing handed it out) between rx: renderer: Vulkan (hookshot_vk.dll) and rx: 123 ... given to it, where rx_start only hands the kept calls over; 0x9873eec is inside the 16 MB to 512 GB range. The other five logs with the zeros have no such line there. Why one run of the same PC was right is not shown; the address of the heap block is my guess.

Added 2026-11-11 13:54 UTC, two more pieces of evidence (no game run):

  • The address in that log fits the kept calls' layout. A kept call (early_call in host/rx_backend.c) is 256 bytes on both Windows targets, with the texture at offset 8, the view at 136 and the sampler's parameters at 156 (static asserts over the struct cut from the source). 0x9873eec minus 156 is 0x9873e50, a 16-byte-aligned address, as realloc returns; minus 8 or minus 136 it is not aligned. So the first byte that was read from guest memory instead of from the copy is the first byte of a kept sampler registration's parameters, which is what register_sampler copies.
  • A Linux run with a window keeps the same 123 calls and reads them correctly. Main bdfe09c, RTX 4090, NVIDIA 615.78.08, a 1920x1080 window: the log has rx: 123 texture and sampler calls made before the renderer started (33 texel writes), given to it, the first sampler anisotropy line is the game asks for 1 (min filter 5, ...), and no line about sampler 258 or a texture slot without an image. That is what the reading expects where the heap lies above 512 GB: the hand-over itself is not at fault, the address check is.

Added 2026-11-11 16:20 UTC: the Windows hand-over emulated in the game. A local test switch (not part of this pull request) hands the first 123 calls' registrations to the renderer as host copies at 0x9870000, the address range of the Windows log. On Linux with an RTX 4090, main 49f75dc then logs the reporters' lines (sampler 258 is nearest: min filter 0, ...; texture slots 0x102, 0x107, 0x124 all zero) and draws the glow around Navi and the lanterns in blocks; with this change and the same hand-over the lines are gone and the glow is smooth. The table and the pictures are in this comment.

How it was tested

  • python -I -B tests/test_registration_host_copies.py --cxx g++ and --cxx clang++ (g++ 16.2.1, clang 23.1.3, Arch Linux): 7 cases pass on the branch. With --source pointed at main 49f75dc the same test fails at the case copy-in-guest-address-range.
  • python -I -B tests/run_all.py on the branch (5f0de4d, Linux): 363 passed, 2 failed, 10 skipped. The two failures are this machine's, the same on main: tests/test_updater_asset_digest.py (only the .NET 10 SDK is installed, the fixture wants net8.0) and tests/test_wayland_libdecor_hint.py (clang 23's -Wunused-but-set-global under -Werror in the test's own harness).

Not tested

  • No normal game run shows the fault or the fix (the emulated hand-over above shows both; a Windows run is still missing). My machine is Linux (RTX 4090, NVIDIA 615.78.08, Arch): a run without a window starts the renderer before the game and keeps no calls, and the heap normally lies above 512 GB there, so main does not misread anything here (my log of Kokiri Forest on main bdfe09c: no kept calls, sampler 258 trilinear, smooth glows around the lanterns and Navi).
  • No Windows run, with or without the change; no AMD or Intel card.
  • The reports in Blocky light halos and checkerboard smoke go away when the few NEAREST samplers are made linear #1117 without a log (the RX 7800 XT's grid in light shafts, the RTX 5070, the RTX 3090 Ti) and the checkerboard haze: this change explains the six logs above; whether those reports are the same fault needs their logs (the two lines above).

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 noreply@anthropic.com

@Joey-Magnus

Copy link
Copy Markdown
Contributor

A Linux data point: this fault does not happen there, and with the PR the logs come out the same.

Setup:

  • PC: Ubuntu 26.04.1, KDE Plasma 6.6 on Wayland. RTX 4070 Ti 12 GB with driver 595.99.02, Ryzen 7 9800X3D.
  • Build: main 49f75dc8 plus about 30 open PRs. The two renderers differ only in this PR: one built with it, one with its merge reverted (vk_bank.cpp only).
  • Test: the same Kokiri Forest save and the same scripted 90 s, with DLAA at 2560x1440.
Without this PR With this PR
rx: N ... calls made before the renderer started 123 123
vk: sampler 258 is nearest (the all-zero set) none none
Early texture slots with an all-zero registration none: slot 0x102 is logged in both runs as a real 4x4 format 0x25 texture ("no view of it") the same
Errors, device lost none none

Why it doesn't happen here: this fits your cause. On Linux the kept copies are ordinary glibc heap allocations, far above the range nvn_gpu_guest takes for guest addresses, so descriptor_ok reads the copies even without the fix. Windows heap addresses can fall inside that range.

Results:

  • No regression from the PR on Linux.
  • This setup can't reproduce the halos to confirm the fix. A Windows run would have to.

🤖 Generated with Claude Code

@mooki000

Copy link
Copy Markdown
Contributor Author

Thanks for building it both ways: that is the check for a regression on a second Linux setup, which I could not give.

Your reason is the one I see here too. On my RTX 4090 (NVIDIA 615.78.08) a run with a window on main keeps the same 123 calls and reads them correctly, because the kept copies lie above the range that is taken for guest addresses.

What stays open is the Windows side: a run of this branch on one of the PCs from #1117 that show the halos (both are RTX 5080s). If someone there can build it, the two log lines to look for are vk: sampler 258 is nearest: min filter 0, ... and vk: texture slot 0x102 ...: no image for it. With the fix they should be gone, and the first sampler anisotropy line should read the game asks for 1 (min filter 5, ...).

@mooki000

Copy link
Copy Markdown
Contributor Author

The Windows hand-over emulated in the game (Linux, RTX 4090): blocks on main, none with this change.

I cannot get a low heap address on Linux, so I built a local test switch (not part of this pull request): it hands the registrations among the first 123 texture and sampler calls to the renderer as host copies in a block mapped at 0x9870000, the address range of the Windows log. Nothing else differs from a normal run without a window. Three runs in one hold, Kokiri Forest from the same save (Link on his porch, the lanterns over the door, Navi), output 2048x1152 (the game's own), default settings:

run build log picture
1. main + the emulated hand-over main 49f75dc + the switch sampler 258 is nearest: min filter 0, mag filter 0, wrap 0/0/0, lod 0..0 bias 0, aniso 0, compare 0; texture slots 0x102, 0x107, 0x124 registered all zero: the same lines as the Windows log with the blocks square steps in the glow around Navi; the lanterns' glow in rectangles
2. this branch's change + the same hand-over main 49f75dc + this change + the switch sampler 258 not nearest (first sampler line: the game asks for 1 (min filter 5, ...)), no texture slot registered all zero smooth
3. main as it is main 49f75dc no calls kept, sampler 258 not nearest, no zero slot smooth

Navi: 1 main with the emulated hand-over, 2 this branch with it, 3 main as it is

The lanterns over the door, the same three runs

In run 1 the game samples through sampler 258 in at least 30 programs (a local log line, capped at 64 lines), three of them among the seven this thread named as the bloom and light-shaft passes: 27ee9883e2a20323 (textures of 128x72, 64x36 and 4x4), 1e2fcee33a0c7209 (128x72) and b7914ae65792d0c8 (2048x1152). Small buffers read without filtering are the blocks. So the zero sampler alone gives the picture of this issue, and reading the kept copies from where they are removes it.

All three runs: RTX 4090, NVIDIA 615.78.08, Arch Linux, no window, exit at the wall, no FATAL, crash or device-lost line, no driver line in the kernel log (one NVRM: memMap_IMPL line each, the known refusal of the dma-buf import). Load average 9 to 40 during the hold.

Still not shown: a Windows run, with or without the change. The emulation shows what the zeros do to the picture and that the change stops them under a low address; it does not show which addresses a Windows heap really gives.

@mooki000

Copy link
Copy Markdown
Contributor Author

Thanks, also for checking the flush's locking and the Linux side. About the workflows: GitHub now starts them for my pushes without an approval (it did for #2380). Main's game-free tests fail until #2380 is in, so I'll rebase this onto main right after that merge and push; the runs should then show here on their own. The Windows run by a #1117 reporter is still open, as you say; I asked for it in the issue and will link the result here.

…om the host's copy (NotRexed#1117)

The game registers its first textures and samplers before the window's
thread starts the renderer. host/rx_backend.c keeps those calls and hands
them over as pointers to its own copies. descriptor_ok (gpu/vk/vk_bank.cpp)
decides which memory a registration is copied from, and after the guest
window it asked the host whether the pointer was a GPU address. With the
native NVN layer every address from 16 MB to 512 GB is one, so a copy on a
heap in that range (Windows gives such addresses) was read from guest
memory at that address instead: untouched pages, all zero. The game's
sampler 258 then filtered nearest with every field zero, and texture slot
0x102 had no image: the blocky light halos of issue 1117.

A registration's pointer is a host pointer, never a GPU address (the NVN
layer passes host pointers in the guest window, rx_backend.c its copies),
so descriptor_ok no longer asks. A pointer outside the guest window is
taken as the host's own memory when VirtualQuery says it is readable, and
refused otherwise.

tests/test_registration_host_copies.py runs descriptor_ok, cut from the
source, against nvn_gpu_guest, cut from the source, with the guest
window's real bounds; it fails without the change.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mooki000

Copy link
Copy Markdown
Contributor Author

Rebased onto main b29967bf so the workflows run here: they had waited for an approval on the old head, and main's game-free tests are green again since #2380. The patch is unchanged (git range-diff: the commit is equal), and its test passes on the new base.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants