[wrocketdev]

Draft, not tried in game. It is opt-in (RX_VK_MEMORY_PRIORITY=1), so nothing changes unless a player turns it on.
I am looking for players on 6-8 GB cards (#993, #995) to try it before it goes further.

Problem

On 8 GB cards the process sits at the driver's VRAM limit in Hyrule Field, Castle Town and Kakariko (#993, #995):
the logs show VRAM: ... used of ... allowed over the allowance. The driver then pages VRAM to system memory, and
nothing tells it which allocations matter: a terrain array of an area left behind and the textures the scene draws
are the same to it.

Change

Everything below is behind RX_VK_MEMORY_PRIORITY=1. Unset, the device is created with the same extensions as
before, and the renderer behaves as it does now.

• vk_main.cpp: asks for VK_EXT_memory_priority and, with it, VK_EXT_pageable_device_local_memory, each only
  with its feature. One log line says which: vk: memory priorities: on, pageable device-local memory ...,
  on, without pageable device-local memory ..., not supported, or off (RX_VK_MEMORY_PRIORITY=1: on).
• vk_vma.cpp: VMA_ALLOCATOR_CREATE_EXT_MEMORY_PRIORITY_BIT. Every VmaAllocationCreateInfo and both image
  pools' VmaPoolCreateInfo (the textures' and, since vk: render targets' images in VMA blocks of their own; dead targets swept every 10 presents (follow-up to #1416) #1436, the render targets') now set priority = 0.5. With that flag on, VMA passes the create-info's value as
  given, and a create-info left at {} is 0, the lowest: without this, every dedicated allocation and both image
  pools would be the first paged out. (The field is ignored while the flag is off.) set_allocation_priority()
  changes a dedicated allocation's priority, never a shared block's.
• vk_texture.cpp: texture_priorities(), every 60 presents when the pageable extension is on. A texture unused for
  RX_VK_PRIORITY_IDLE presents (600, about 10 s) drops to 0.1, and goes back to 0.5 once used. Only memory a
  texture owns changes: dedicated allocations (the big ones, past the 128 MB image blocks) and sparse chunks.
  Textures inside pool blocks keep the block's 0.5. Every 600 presents it logs
  vk: memory priority: N textures (X MB) at the idle priority.
• memory_priority.h: the rule and the switches, tested without a GPU by tests/test_memory_priority.py. The test
  also fails if a VMA create-info is added later without a priority.
• Docs: the README switch section and gpu/vk/VULKAN.md.

Testing

• Game-free: tests/test_memory_priority.py (C++ rule test plus source checks). The three changed renderer files
  pass a syntax-only compile against the pinned Vulkan-Headers v1.4.345 and VMA v3.3.0.
• In game: not tried. I have no PC to run it on right now; that is why it is a draft and opt-in.

How to try it: add RX_VK_MEMORY_PRIORITY=1 to hookshot_env.txt, play the same route with and without it (for
example Kokiri Forest → Hyrule Field → Kakariko → Castle Town market and back), and compare the fps, the worst freezes,
and these lines of hookshot_log.txt: vk: memory priorities:, vk: memory priority: and
VRAM: ... used of ... allowed.

Windows already pages video memory on its own, so the gain there may be small. It is aimed at the stalls when VRAM is
full, not at the garbled textures of #993.

Not in this PR

• Priorities by role (render targets higher). Since vk: render targets' images in VMA blocks of their own; dead targets swept every 10 presents (follow-up to #1416) #1436 the render targets have a pool of their own, so this is now
  one line (pi.priority in target_images_pool()); left at 0.5 here to keep this PR to the mechanism.
• A higher priority for the GPU mirrors.

Related: #993, #995.

Written with AI assistance (Claude Opus 5.5); the commit carries the Assisted-by: and Co-Authored-By: trailers.

🤖 Generated with Claude Code

[wrocketdev]

Rebased onto current main (d224dc0) and force-pushed: no conflicts now, still a single commit.

Two things changed with the rebase:

• The test declares RUN_ALL_ARGS in its own file, following the new run_all.py format.
• The render targets' pool from vk: render targets' images in VMA blocks of their own; dead targets swept every 10 presents (follow-up to #1416) #1436 (target_images_pool()) now sets priority = 0.5 too. Left at {}, it would be 0 once the extension is on, so render targets would be the first paged out. The test's create-info check caught it.

Still opt-in (RX_VK_MEMORY_PRIORITY=1) and not tried in game yet. Thanks for taking a look!

[Joey-Magnus]

Two things for the rebase. The branch conflicts with main in gpu/vk/vk_texture.cpp.

• Main's gpu/vk/vk_vma.cpp now has a third image pool, textures_pool(), with a VmaAllocationCreateInfo and a VmaPoolCreateInfo at about lines 122 and 127. Both need priority = memory_priority::kNormal. Without it, once the extension is on, every texture in that pool is created at priority 0, the lowest. That is the opposite of what this PR is for. It would help if test_memory_priority.py counted the create-infos in vk_vma.cpp, so that a new one can't slip through without a priority.
• memory_priority::wanted reads RX_VK_MEMORY_PRIORITY with atoi, and RX_VK_LOG_PRIORITY turns on for any value, including 0. Since #2042, switches use env_on and env_value_off from gpu/vk/compat/hookshot_env.h, so that =on and =true work and =0 means off.

CI has never run on this branch. test_memory_priority.py passes locally on e2ec9b8.

[eliasbruno124-dev]

Tested this on a 6 GB card, as you asked for. Short version: the extensions work here, but on this card the idle demotion never reaches a texture, and I could not show that the switch helps. What the runs did show is a slow state that comes and goes with the texture cache staying far over its budget, with and without the switch. Details below.

Setup

• Build: v2.1.0 (633eb435) with this PR's commit e2ec9b84 applied on top (so not on the PR's own base). Most hunks applied as they are. In vk_texture.cpp / vk_texture.h three hunks had moved context and I placed them by hand: the #include "memory_priority.h", the texture_priorities() call (it now sits right before 2.1.0's compact_step(completed)), and the two new Texture fields. The code lines are the same as in the PR. The tests/ and readme.md hunks were not applied (the builder's tools folder has no tests), so tests/test_memory_priority.py was not run. Built with make_game.py into a folder of its own: 473 files, 0 failed.
• Hardware: RTX 3060 Laptop 6 GB, driver 621.01, i5-11400H, 32 GB RAM, Windows 11 (26300), romfs on NVMe.
• Settings: Handheld profile, DLSS Quality, quality_preset=2, frame generation off, frame pacing off.
• Switches, the same in every run: RX_VK_MIRROR_MB=2048 RX_VK_VRAM_MB=1800 RX_VK_FIFO=1 RX_VK_COMPACT=0 HOS_VCORE_ESCAPE_US=2000000 HOS_VCORE_ESCAPE_EQ_US=2000000. The mirror budget is far above the Low preset's, so this card is overcommitted on purpose: every run sits at 5.4-6.7 GB against about 5.15 GB allowed.

Method

Scripted runs, no hands on the pad: a fresh copy of the same save every run (autosave at the Castle Town gate), input through HOS_INPUT_LIVE, menu steps keyed to the draw counts in the log, load screens detected on screen.

• Route 1, town only: walk to the market, turn the camera both ways, about 3 minutes.
• Route 2, area changes: four times through the gate (Town, Hyrule Field, Town, Field, Town), a look around after each, then 60 s of camera turns in town.
• A = stock v2.1.0. B = patched build, RX_VK_MEMORY_PRIORITY=1. C = patched build, switch not set (control).

Results

fps is the mean of the log's presents blocks in game scenes; "sync" is memory sync from a frame's replay time.

Route 1 (town only), whole route:

run	fps	sync	slowest frame	VRAM max
A1	22.0	12.8 ms	1491 ms	5.9 GB
A2	18.8	24.6 ms	1623 ms	5.8 GB
B1	22.8	12.2 ms	227 ms	5.9 GB
B2	25.0	9.4 ms	1084 ms	5.7 GB

Route 2 (area changes), the last 60 s in town. The last column is from the final vk: VRAM: line of each run.

run	fps	sync	GPU time	VRAM used / allowed	textures held (budget)
A1	31.3	5.8 ms	15.8 ms	5680 / 5172 MB	1619 MB (1800), 1097 textures
A2	31.9	6.2 ms	16.3 ms	5834 / 5171 MB	1651 MB (1800), 1075 textures
C1	31.1	5.3 ms	12.4 ms	5774 / 5145 MB	1640 MB (1800), 1066 textures
C2	17.5	25.1 ms	47.6 ms	5652 / 5175 MB	2812 MB (1800), 2313 textures
B1	17.2	22.8 ms	46.9 ms	6154 / 5171 MB	2938 MB (1653), 2410 textures
B2	23.1	10.8 ms	24.6 ms	6042 / 5171 MB	2832 MB (1639), 2308 textures
B3	20.3	18.4 ms	36.0 ms	6180 / 5175 MB	2952 MB (1633), 2637 textures

So with the switch on all three runs ended slow, and with it off three of four ended fast. But C2 is the patched build with the switch off and it ended just as slow as B1, so I cannot say the switch causes it. In Route 1 it was the other way round (B a little ahead, A2 slow). With this few runs I would call the switch's effect on fps unproven in either direction.

What the logs and the code say

1. The extensions are there and on: vk: memory priorities: on, pageable device-local memory (unused textures drop to the lowest).
2. vk: image over budget, retrying without WITHIN_BUDGET is in every run without the switch (10 times, the line's cap, in A1, A2, C1; 3 times in C2) and in no run with it. That part of the switch does something measurable.
3. No texture is ever demoted. Every B run logs vk: memory priority: 0 textures (0 MB) at the idle priority, 0 changed now, start to end. I think this is structural on this card, not a matter of waiting longer: set_allocation_priority returns 0 for anything that is not a dedicated allocation, and the textures are made with dedicated = false into the 128 MB pool blocks (vk: image blocks: 17 blocks 2176 MB, 2080 images 2075 MB, of about 2350 textures). So the textures the PR means to demote are the ones it cannot reach.
4. Everything is made at kNormal, the mirrors too (create_device_buffer in vk_memory.cpp). So when the process is over the allowance the driver has no order to go by: a mirror that is synced every frame ranks the same as a texture nobody samples.
5. The slow state has one clear mark in the table above: in the fast runs the texture cache is down to about 1.6 GB by the end (around 1070-1100 textures, 1.4 GB free in the blocks). In the slow runs it still holds 2.8-2.95 GB (2300-2600 textures) against a budget of 1.6-1.8 GB, for the whole last minute, and memory sync is 18-25 ms instead of 5-6 ms with GPU time two to three times as high for the same scene. That is the same in A, B and C, so it is not this PR's doing. It looks like Slowdown after moving between areas: GPU memory spills into system RAM #2412 to me. I have not found yet why the cache is not cut back to its budget in those runs.

What I would try

• Give the priorities an order: mirrors (and the render targets' pool) above the default, the textures' pool blocks below it. Textures can be made again from guest memory; a mirror in system memory costs every frame.
• Let the demotion reach pooled textures: by block (a block whose textures are all idle), or by putting textures that go idle into blocks of their own.

I am going to look at why the cache stays over its budget, try the first point here, and run the same two routes again. I will post the numbers and the diff.

[wrocketdev]

Thanks, both fixed in c7cd90c (rebased onto 3189a42). textures_pool()'s two create-infos now set priority = kNormal. The test's create-info rule did catch them: it failed on vk_vma.cpp:130 and :135 before the fix. It now matches every declaration form in gpu/vk ({}, {...}, = {...}, copies) and fails if it finds fewer than 10, so it can't go quiet if the code moves. RX_VK_MEMORY_PRIORITY and RX_VK_LOG_PRIORITY are read with env_on, and memory_priority::wanted is gone.

[wrocketdev]

Thank you, this is exactly the run it needed. You're right about point 3: on main every texture goes into pool blocks (dedicated = false), so the demotion only reaches sparse images. As it stands, the PR does nothing for textures.

The fix I have in mind works per block. With the textures in their own pool (the default; not with RX_VK_COMPACT=0), a block whose textures have all been idle for 600 presents would drop to 0.1, and go back to 0.5 as soon as one of them is used. I'd rather not write it blind, though. Would you be up for running your two routes on it once your role-priority numbers are in? If those show priorities don't move the slow state, it's probably not worth more code. I also agree the cache staying over budget looks like the real problem here (#2412).