Skip to content

fix(vk): a pool whose mirror found no VRAM is asked for again, not refused for good - #2618

Open
aquinas22 wants to merge 1 commit into
NotRexed:mainfrom
aquinas22:fix/mirror-no-vram-retry
Open

aquinas22 wants to merge 1 commit into
NotRexed:mainfrom
aquinas22:fix/mirror-no-vram-retry

Conversation

@aquinas22

Copy link
Copy Markdown
Contributor

On cards with little VRAM, the scene's hottest memory pool can lose its VRAM mirror for the whole session: when mirror_make's allocation fails once (VRAM full of textures and retired pools' mirrors at that moment), the pool is MIRROR_REFUSED for good and the GPU reads it over PCIe from then on. This asks for it again 64 submissions later instead.

What I saw. RTX A1000 Laptop (4 GB), Windows, v2.1.0, Kokiri Forest. One 144 MB pool takes ~2.3 GB of the ~2.4 GB the GPU reads over PCIe in 600 submits (the bound in guest memory, by pool log line), state 3 (refused). The process is over what the driver allows (~4.2 of 3.3 GB), so VMA's WITHIN_BUDGET allocation fails at the pool's first binding; allocating without the budget flag fails too here (NVIDIA does not oversubscribe), so a "make it anyway" fallback is not an option. Whether the pool got its mirror depended on load timing, which made frame rates swing between launches of the same build.

The change. On an allocation failure mirror_make leaves the pool MIRROR_NONE and notes the submission (Pool::no_vram_epoch); gpu_buffer_bind neither makes nor asks for its mirror again until 64 submissions on, so a full VRAM costs one failed allocation per pool per 64 submissions, not one a binding. By then colder or retired pools' mirrors have usually been given back. Both outcomes are logged (8 lines each at most). RX_VK_MIRROR_NO_RETRY=1 keeps the old behaviour; documented in docs/development.md.

Measured (headless, HOS_ENV_FILE per run, same save, ~4240 draws a frame, 90 s after 45 s of play; with #1377's mirrors-by-heat and RX_VK_WB_MAX_KB=64, built on v2.1.0):

fps PCIe reads / 600 submits
refused for good (current) 18-22 (30 on a launch that happened to mirror it first) ~2.4 GB
asked for again (this PR) 30.2, 30.3 ~0 MB

The pool was mirrored ~70 submissions after its first failure in every run. A third launch measured after only 20 s of play read 25.1 fps while mirrors were still arriving.

Checks. tests/run_all.py for every test that reads vk_memory.cpp or docs/development.md passes locally on Windows (llvm-mingw), except test_env_on (HOS_FPS_LOG in host/oot_high_fps.c, failing on main too) and test_vram_budget_row (fails on main locally, unrelated). test_parallel_mirror_make still finds !par on the first-binding line.

Independent of #1377, which makes the retry more likely to find room (it gives back colder mirrors for hot pools).

…fused for good

On a 4 GB card the textures in use and the retired pools' mirrors often have VRAM full when a scene's hottest pool is
first bound. mirror_make's allocation failed, the pool was MIRROR_REFUSED, and the GPU read it over PCIe for the rest of
the session: Kokiri Forest's 144 MB pool, ~2.3 GB bound in 600 submits. Now it is asked for again 64 submissions on.

RTX A1000 Laptop (4 GB), Kokiri Forest, headless, 90 s after 45 s of play, v2.1.0 with NotRexed#1377 and RX_VK_WB_MAX_KB=64:
refused 18-22 fps (the pool mirrored only when a launch happened to bind it before VRAM filled); asked for again,
mirrored ~70 submissions after its first refusal, ~0 MB read over PCIe, 30.2 and 30.3 fps (25.1 in a third launch measured after only 20 s, the mirrors still arriving).
RX_VK_MIRROR_NO_RETRY=1: as before.
@sheepbun

Copy link
Copy Markdown

Measured on a 10 GB card too: the refusal is not only a 4 GB problem.

Setup. RTX 3080 10 GB, driver 615.78, Linux (GNOME on Wayland), i7-11700, 2560x1440 at 100% render scale, cap 170, frame generation off. Built from upstream main of 2026-11-11 with the renderer's profile and ThinLTO, with and without this PR's commit (58834871) cherry-picked. The renderer was swapped between runs and everything else held fixed. RX_VK_WB_MAX_KB was unset. Runs were interleaved, each started below 70 C, with other programs at 2.5-4.7% of the CPU.

scene_changes (the bench route that teleports through several maps; 20 s measured):

fps Pools refused for good Readback waits (of 3,000)
with #2618 68.7 none 1,551 waited, 0.04-0.05 ms each
without 43.9 one (bound in guest memory ... state 3, 84 log lines) 339-349 waited, 14.7-16.2 ms each, all for their submission's last part
with #2618 61.4 none 1,561 waited, 0.04-0.06 ms each
without 45.5 one (96 log lines) 341-358 waited, 6.6-7.9 ms each, all for their submission's last part

With the PR, 8 pools a run (16-240 MB) hit "no VRAM for its mirror" during the scene loads. All of them got their mirror 64-87 submissions later; one 240 MB pool took 468. VRAM was at ~7.8 of 9.4 GB.

Without it, the refused pool's binds in guest memory raise the in-place part, so every readback waits for its whole submission. That wait is what costs the frame rate here, more than the PCIe reads themselves.

The same refusal turned up in a Hyrule Field run on this card. A 224 MB pool, merged from three imports during the teleport while VRAM was at 8.05 of 9.34 GB, stayed refused, and the run measured 52.2 fps. The runs that found room measured 62.9-71.5.

The 1% lows (the scene loads themselves) are unchanged. I saw no visual difference, but I did not check screenshots.

sheepbun added a commit to sheepbun/Bunny-Hood that referenced this pull request Nov 12, 2026
…d for good in Hyrule Field

RX_VK_WB_SKIP_LEARN keeps +1.4% in Kokiri Forest with stale first touches: not taken. The slow no-cap run in Hyrule Field
was a 224 MB pool whose mirror allocation failed once and was refused for good (5.1 ms readback waits); upstream NotRexed#2618
is taken for it, and NotRexed#2731 is to be measured against this branch's learning cap.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sheepbun added a commit to sheepbun/Bunny-Hood that referenced this pull request Nov 12, 2026
…of upstream, this repository and every fork

The fork's first rule, above every other priority: before any task, search upstream's main and every other branch, its
pull requests (open, merged, closed), every branch and pull request of this repository, and every branch and open pull
request of every fork of upstream and of their forks; build on what exists instead of writing it again. AGENTS.md,
CONTRIBUTING.md and docs/workflow.md (a new section 0) say so; the claim and the pull request say what was found.

scripts/prior_work.py does the search: `sync` lists every branch head and open pull request of upstream, this
repository and all forks over GraphQL (743 repositories, 4,973 branches today) and fetches only the commits a local cache
lacks (198 s the first time, 100 s after); `search` greps every branch's commits by group (upstream's default branch, its
other branches, this repository's, the other forks', each without the groups before it), with -G for code, plus
GitHub's search of the pull requests and the forks' own open ones. On 2026-11-12 it finds upstream NotRexed#2618 and its fork
branch for "mirror refused", work this fork had just written again. Fork-only (FORK_ONLY).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit 9a53a9f)
sheepbun added a commit to sheepbun/Bunny-Hood that referenced this pull request Nov 12, 2026
…o mirror refused for good

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sheepbun added a commit to sheepbun/Bunny-Hood that referenced this pull request Nov 12, 2026
…ull request

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@Joey-Magnus Joey-Magnus left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I merged this into current main. The only conflict is in docs/development.md, where #2651 added the RX_VK_CAPTURE_KEEP_OLD row in the same spot; both rows can stay. The code merges cleanly. Every test that reads vk_memory.cpp or development.md passes on Linux, including test_import_buffer_binding, test_texel_buffer_ownership, test_parallel_mirror_make and test_mirror_inherit.
The logic is sound. A failed create_device_buffer leaves the pool MIRROR_NONE with no_vram_epoch set, and gpu_buffer_bind neither makes nor queues its mirror for 64 submissions. A success logs how long it took and clears the epoch. Two small notes, neither blocking:

  • While a pool waits it isn't REFUSED, so its read bindings miss the lock-free fast path (vk_memory.cpp, the MIRROR_REFUSED || MIRROR_WANTED test) and take pools_mu for those 64 submissions. That's the same as a pool over the mirror budget on main, so it's fine.
  • no_vram_epoch == 0 means "none", so a failure in the very first submission retries at the next binding. That's harmless.
    Your A1000 numbers and sheepbun's RTX 3080 runs (43.9/45.5 against 68.7/61.4 fps in scene_changes) both come from the game. Please rebase; once it's rebased, I'd merge it.

sheepbun added a commit to sheepbun/Bunny-Hood that referenced this pull request Nov 12, 2026
…d for good in Hyrule Field

RX_VK_WB_SKIP_LEARN keeps +1.4% in Kokiri Forest with stale first touches: not taken. The slow no-cap run in Hyrule Field
was a 224 MB pool whose mirror allocation failed once and was refused for good (5.1 ms readback waits); upstream NotRexed#2618
is taken for it, and NotRexed#2731 is to be measured against this branch's learning cap.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sheepbun added a commit to sheepbun/Bunny-Hood that referenced this pull request Nov 12, 2026
…o mirror refused for good

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sheepbun added a commit to sheepbun/Bunny-Hood that referenced this pull request Nov 12, 2026
…ull request

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants