Repository navigation
perf(vk): make part of the image pool's blocks at the start - #2438
Conversation
Each 128 MB block of the images' pool is a vkAllocateMemory on the replay thread, ~170 ms on AMD's Windows driver. Walking across Hyrule Field on an RX 7800 XT made ~70 of them, and the frame rate dropped to the low 40s each time new ground came in. The pool now holds 15% of the card's device-local memory as blocks from the start (2 to 24 blocks; 19 on a 16 GB card). RX_VK_IMAGE_PREALLOC=<blocks> sets the count, 0 keeps the old behaviour; if the blocks do not fit, the pool is made empty as before. Scripted 1080p walk, same route: mean fps 52.9 -> 59.8, min 41.6 -> 56.7 (measured on v2.0.5-v2.0.12 trees; not yet re-measured on v2.1.0 with real play). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Joey-Magnus
left a comment
There was a problem hiding this comment.
Thanks for chasing the vkAllocateMemory stalls on AMD's Windows driver; making blocks up front is a sensible way to take them off the replay thread. The fallback when the pool can't be made with its blocks is handled correctly, and the checks that read vk_vma.cpp (test_target_image_pool, test_vram_compact, test_texture_staging_publication) pass at this head. The block count, though, is sized from the heap's total size rather than what the process may actually use, and it lands badly on two configurations the repo already has special handling for. Details:
blocking gpu/vk/vk_vma.cpp:103-109: the blocks are made without regard to the memory budget. VMA's CreateMinBlocks goes straight to vkAllocateMemory (it only honours a heap size limit, never VK_EXT_memory_budget), whereas every image create_image makes asks for VMA_ALLOCATION_CREATE_WITHIN_BUDGET_BIT. When other programs already hold VRAM (the tested-configurations table has a 12 GB card with only 10.2-10.5 GB allowed), 2.4 GB made up front can push the process over what the driver allows. On Windows that doesn't fail the allocation: WDDM pages it out, which is the "GPU time 13 → 55 ms for good" failure the texture budget's comments in vk_texture.cpp describe. Please cap n by the heap's free budget (vmaGetHeapBudgets: budget - usage, with some margin), not by memoryHeaps[h].size.
blocking gpu/vk/vk_vma.cpp:103-106: the heap is the largest device-local heap, not the heap of type (dev.memory.memoryTypes[type].heapIndex), and integrated GPUs aren't excluded. On UMA drivers the device-local heap is a share of system RAM, so a 32 GB laptop could pin up to 24 × 128 MB = 3 GB of RAM at the first image. That is more than the 2560 MB textures are capped at on an integrated GPU (vk_texture.cpp, the 780M laptop that ran out of memory), and the 2-block minimum alone takes a quarter of a Steam Deck's default 1 GB carve-out. I'd skip pre-making on VK_PHYSICAL_DEVICE_TYPE_INTEGRATED_GPU unless RX_VK_IMAGE_PREALLOC asks for it, and use the type's heap.
should fix: interaction with RX_VK_COMPACT=1. With compaction on, textures go to texture_pool (create_image, textures && textures_pool()), so images_pool keeps only DLSS's, frame interpolation's and scratch images. The pre-made blocks then sit mostly empty for the whole session (VMA never frees blocks below minBlockCount). Under compaction the texture budget only subtracts the textures' pool's free space (vram_compact::reclaimable(block_free, pool_free)), so those empty blocks count as pressure and the held budget evicts roughly that much of the textures. The two features work against each other: compaction exists to give blocks back. Pre-make nothing (or far fewer) when compaction is on, or pre-make in the pool the textures actually use.
question: the numbers (52.9 → 59.8 mean, 41.6 → 56.7 min) come from v2.0.5-v2.0.12 trees together with other changes. Since #1436 (in v2.0.12), render targets have their own pool (target_images_pool), and this change doesn't pre-make blocks there, so some of the ~70 allocations you counted may no longer be in images_pool at all. Could you re-run the Hyrule Field walk on current main with RX_VK_IMAGE_PREALLOC=0 vs. the default and post the vk: image blocks log lines? That would show how many blocks each pool really grows to. The PR also says the test suite wasn't run, and only the title check has run in CI here.
minor gpu/vk/vk_vma.cpp:109: the first image now pays for all the blocks at once on the replay thread (19 × ~170 ms ≈ 3 s on your card, per your own timing). That's probably fine while booting, but worth saying in the log line, and worth checking it can't land mid-gameplay (for example on the first image after a device-lost recovery, if the pools are re-made).
Joey-Magnus
left a comment
There was a problem hiding this comment.
The earlier review is still open, and I'd add:
vk_vma.cpp:103-109makes up to 24 x 128 MB blocks sized from the largest device-local heap. VMA's minimum blocks skipWITHIN_BUDGET, and that heap isn't necessarilytype's.- Integrated GPUs aren't excluded, so the blocks come out of system RAM: up to 3 GB on a laptop, and 256 MB of a Steam Deck's 1 GB carve-out.
- With
RX_VK_COMPACT=1, textures go to the other pool, so these blocks sit empty and count as pressure.
|
On an 8 GB card (RTX 3070, Windows) the blocks made at the start cost a lot in Castle Town Market: 31-36 fps on main against 54-56 with |
Port of upstream NotRexed#2438. Each 128 MB block is a vkAllocateMemory on the replay thread (~170 ms on AMD's Windows driver); a walk into new ground made ~70 of them and dropped the frame rate. The pool now starts with 15% of device-local memory as blocks (2 to 24). RX_VK_IMAGE_PREALLOC sets the count; 0 keeps the old behaviour, and a card without room falls back to making them as needed.
Each 128 MB block of the images' pool is a
vkAllocateMemoryon the replay thread, ~170 ms on AMD's Windows driver. A walk across Hyrule Field on an RX 7800 XT made ~70 of them and the frame rate dropped to the low 40s each time new ground came in. The pool now starts with 15% of the device-local memory as blocks (2 to 24; 19 on a 16 GB card).RX_VK_IMAGE_PREALLOC=<blocks>sets the count,0keeps the old behaviour; if the blocks do not fit the pool is made empty as before.Scripted 1080p walk, same route, measured on older trees together with other changes (v2.0.5-v2.0.12): mean fps 52.9 -> 59.8, min 41.6 -> 56.7, replay submit 14.5 -> 9.9 ms. I have not re-measured this change alone on v2.1.0. Log line on start:
images from a pool of 128 MB blocks (memory type 0), 19 made at the start. Costs VRAM up front (about 2.4 GB on this card), so a card with little free memory may want a lower value.Tested: the Vulkan back end builds (llvm-mingw,
RX_VK_ONLY) on top of v2.1.0, and a full build of that tree boots to the title at 60 fps on an RX 7800 XT / Ryzen 7 5800X3D (Windows 11, driver 0x8001a0) with the new log lines present. The repository test suite was not run, and I have not yet played a long session on this build, so the effect in real play is not shown. Related data: #995.🤖 Generated with Claude Code