[Erfboom]

Problem

When /dev/udmabuf exists on the host system but fails to import guest memory (e.g. kernel/driver rejection with EINVAL, EBADFD, or EPERM, or when the driver reports no valid memory type), the Vulkan backend continuously attempts imports on every buffer pool allocation. This causes continuous ioctl failure logging and wasted kernel transitions instead of cleanly falling back to host-pointer imports (VK_EXT_external_memory_host).

In addition, pools that failed device mirror creation under transient VRAM pressure remained permanently marked as MIRROR_REFUSED. If memory was later freed or other mirrors were reclaimed, these pools would never be retried, leaving hot buffers in slower system memory permanently.

Solution

• Add disable_dmabuf_imports(const char* why): if udmabuf creation or driver import is rejected before any successful import has occurred, close the udmabuf handle and permanently fallback to host-pointer imports for subsequent allocations.
• In mirror_make, if device buffer creation fails due to VRAM pressure, trigger mirrors_give_back() and retry once.
• If mirror allocation still fails due to transient pressure, set mirror_retry_epoch = mirror_epoch + 120 rather than permanently refusing mirrors for the pool. In gpu_buffer_bind, allow pools to exit MIRROR_REFUSED once the retry epoch is reached.

Impact

• Eliminates repeated failing ioctl(..., kUdmabufCreate) syscalls and log spam on Linux configurations where /dev/udmabuf is present but unsupported or restricted.
• Restores mirroring for pools that experienced temporary allocation failures during high VRAM spikes once pressure subsides.
• Verified with tests/run_all.py (359 passed, 0 failed).

[Joey-Magnus]

I ran the suite merged into main and it passes, but:

• mirror_make now calls mirrors_give_back(n) (vk_memory.cpp:3060), which is reached from gpu_buffer_bind mid-recording (:2752). mirror_free then rewrites the page table and bumps pools_gen while the recording is still open. That used to happen only at the submission boundary.
• With VRAM genuinely full, the retry at mirror_epoch + 120 (:3063) repeats free-and-retry every 120 submissions, freeing idle mirrors for an allocation that keeps failing. Low-VRAM cards churn.
• Imports are switched off for the session on the first failed import before any success (:204), so one transient VK_ERROR_OUT_OF_DEVICE_MEMORY disables dma-buf imports for good. The log was already capped at 8 lines.

I'd split out or drop the retry, and attach a run from the game.

[ghost]

Thanks. The udmabuf half makes sense to me: EBADFD/EINVAL from udmabuf on a memfd it won't take are permanent, and so is "no memory type", so giving up on the first one is fine there. Two things I'd still change, plus I agree with @Joey-Magnus's review:

• His third point stands: vkImportMemoryFdKHR/bind failing with VK_ERROR_OUT_OF_DEVICE_MEMORY is not proof the driver can't import, so I'd only disable on the ioctl errnos and the missing memory type, and keep the import-failure path as it was.
• His first point (mirrors_give_back reached from gpu_buffer_bind mid-recording, page table rewritten and pools_gen bumped outside the submission boundary) is a real behaviour change and not mentioned in the description. More below on why the immediate retry rarely helps anyway, which makes it easy to just drop.
• The 120-submission retry of MIRROR_REFUSED pools on its own is reasonable, but I'd split it from the udmabuf fix into its own PR with a run from the game on a low-VRAM card, as Joey asked.

Nit: the description says the failures meant "continuous ioctl failure logging"; the log was already capped at 8 lines, so this is about the wasted ioctls/imports, not log spam.

CI has no checks reported on this branch yet.

[ghost]

We only get here after the budget check above passed, so inside mirrors_give_back the mirror_bytes + want_bytes > mirror_budget() clause is false: it frees only retired mirrors and ones idle past kLongIdle, which memory_begin_submission already does every 64 submissions. So the second create_device_buffer is usually the same allocation against the same VRAM, and the cost is the mid-recording mirror_free Joey pointed out. I'd drop the give-back-and-retry here and keep just the mirror_retry_epoch part (in its own PR).

[ghost]

Minor: dma_buf_memory can run on the background import thread (bg_main calls import_pool without pools_mu) as well as on the recording thread, so closing dmabuf_dev here can race the other thread's ioctl on the same fd (or a reused fd number). The background imports are opt-in (RX_VK_BG_IMPORT, RX_VK_ASYNC_IMPORT, prefetch), so not a default-path problem; setting dmabuf_memfd = -1 and leaving the fd open would avoid it.

[felipefl142]

I tested this PR in a real play session. It fixes the stuck refused mirrors I reported in #2412 (#2412 (comment)).

Setup: RTX 3070 Laptop 8 GB, NVIDIA 620.70.06, CachyOS (Linux). I merged this PR into main at 57ed3aa (v2.1.0-rc.3-177-gf302c495). DLSS (built with the SDK), output 1920x1080, RX_VK_VRAM_MB=3072, RX_VK_COMPACT=1, default mirror budget (2048 MB). In the game: the whole Water Temple, then Lake Hylia, Hyrule Field, and burning Kakariko. Before the merge, tests/ scripts that read vk_memory.cpp passed: 26 run, one skipped because it needs Windows.

Timeline

Time	Where	Game fps	Draws a frame	Guest memory read over PCIe
0-54 min	Water Temple	38-46 (52-55 in light rooms)	800-1100	none
56-60 min	Leaving for Lake Hylia	30-44	600-1700	up to 474 MB per 600 submits
62-74 min	Hyrule Field, Kakariko	25-30	1500-2200	almost none

At Lake Hylia, VRAM ran out the same way as in my earlier report. The log showed image over budget (x10), 40 textures evicted, and pools left in system memory (0x7fb8000000 (176 MB) stays in system memory: the mirrors have 1920 MB of 2048). This time the pools went back to state 1 (MIRROR_WANTED), got mirrors again, and the PCIe reads dropped to almost nothing within about 3 minutes. Over the whole session, pools appeared in state 3 in only 4 summaries. My bad session on 7759a6f had 23,649.

Same scene size, four sessions (same PC and settings)

Draws a frame	7759a6f, good session	7759a6f, bad session (mirrors stuck refused)	940eb9e, RX_VK_VRAM_MB=2560	57ed3aa + #2482
600-1000	42.5 fps	36.6	33.7	42.0
1400-1800	34.4	28.8	27.3	34.5
1800+	32.3	25.1	22.0	27.9
memory sync a frame	1.1-2.0 ms	3.0-4.6 ms	1.4-2.3 ms	0.7-0.9 ms

(fps: mean of the vi: lines, grouped by the draw count in the nearest presents line.)

Lowering the texture budget to 2560 did not leave room for the mirrors. VRAM still filled up to 6.4-6.7 GB, textures were missing or blurry in some areas, and fps was worse than with 3072. With this PR and the budget back at 3072, the session at up to 1800 draws a frame matched my best earlier one, memory sync stayed under 1 ms the whole time, and nothing was missing in the temple.

The lower fps in Hyrule Field comes from the size of the scene, not from memory. At 1500-2200 draws a frame, both the replay (submit 13-17 ms) and GPU time (13-16 ms) grow with the draw count, while PCIe reads and memory sync stay low. That is #995's territory.

On NVIDIA the udmabuf part of the PR doesn't come into play: the log has no disabling udmabuf pool imports line, since #2340 already keeps dma-buf imports off there.

[YakuzaFanA]

Tested this on Windows with an 8 GB card and it fixes the slowdown for me.

Setup: RTX 5060 Ti 8 GB, driver 610.88, Core Ultra 7 265F, Windows 11, 1080p. Built on main at 497060f with this PR merged on top. My build also has a few other local changes (#1377 and some switches), but those were identical in both runs below. The only difference between them is this PR.

Same route both times: Graveyard, Kakariko, Hyrule Field, then into the Market from the field, then back out. Without the PR, the Market entry pushed VRAM over the limit and two pools (160 MB and 240 MB) ended up in state 3 for the rest of the session. The GPU read them over PCIe the whole time and the game never recovered, even after leaving. With the PR they got their mirrors back.

	without #2482	with #2482
mean fps	27.1	49.0
frames over 33 ms	47.5%	4.0%
"bound in guest memory", median MB per 600 submits	3,818	9.1
GPU time a frame	25.3 ms	10.4 ms
Hyrule Field after the Market	16-22 fps	46-56 fps

The Market itself still drops to around 22 fps, but that one looks CPU-bound (#995), not this.

(I used Claude to help read the logs; the playing and testing were mine.)

[Erfboom]

Thanks @Joey-Magnus and @hckrman101 for the feedback, and @YakuzaFanA for the benchmark run on Windows!

Pushed an update addressing all points:

1. Transient OOM no longer disables udmabuf: In dma_buf_memory, VK_ERROR_OUT_OF_DEVICE_MEMORY is explicitly guarded so a temporary allocation failure during a VRAM spike will not disable udmabuf imports for the remainder of the session.
2. Removed mid-recording mirror eviction: Dropped mirrors_give_back(n) entirely from inside mirror_make. Evictions and page table rewrites continue to only happen safely at submission boundaries.
3. Budget-gated mirror retry: In gpu_buffer_bind, mirror retry is gated on budget headroom (mirror_bytes + (p->hi - p->lo) <= mirror_budget()). If VRAM is genuinely full and no room exists, the retry epoch is deferred by 120 submissions without evicting active mirrors or churning on low-VRAM GPUs. Pools > 1 GB remain permanently refused.

Tested locally on Linux (6 GB hardware):

• Unit test suite passes cleanly.
• udmabuf remains active across heavy transitions (80 live imports, 2.9 GB imported, clean lifecycle).
• Scene transitions and Hyrule Field run smoothly with stable frame pacing (21–25 FPS under heavy scene load, 60 FPS in menus) with zero churn or mid-recording invalidations.

[Joey-Magnus]

The update addresses my first round: no mirrors_give_back mid-recording, the retry is budget-gated and frees nothing, and an out-of-device-memory import no longer turns dma-buf imports off (gpu/vk/vk_memory.cpp:213).

I merged it into current main and ran tests/run_all.py. One failure is this PR's: tests/test_direct_buffer_writes.py cuts gpu_buffer_bind out, and the retry block (:3042) uses mirror_bytes and mirror_budget(), which tests/direct_buffer_writes_test.cpp doesn't declare, so it no longer compiles. Please add the two stubs. The other CI reds are main's from early Nov 12.

Still open:

• hckrman101's point on :163: disable_dmabuf_imports closes dmabuf_dev while the background import thread (RX_VK_BG_IMPORT, RX_VK_ASYNC_IMPORT) can be in ioctl on it. Setting dmabuf_memfd = -1 and leaving the fd open avoids that.
• The description still describes the give-back retry and log spam.
• felipefl142's and YakuzaFanA's runs were on 2fa7431; a short re-run on 993f7f0 would confirm the budget-gated retry still recovers.

[felipefl142]

I played the updated head (993f7f0) for about 2.5 h, merged into main 51da046 (v2.1.0-rc.3-259-gc25b656d). Same machine as in my earlier comment: RTX 3070 Laptop 8 GB, NVIDIA 620.70.06, CachyOS, DLSS, RX_VK_VRAM_MB=3072, RX_VK_COMPACT=1, default mirror budget (2048 MB on this card).

Route: Desert Colossus → Spirit Temple → Death Mountain Crater → Market → Ganon's Castle → Ganondorf → the collapsing castle → Ganon → credits.

The three changes:

• udmabuf: never disabled. No "disabling udmabuf" line and no udmabuf or import errors in the log; 75–103 live imports during the session.
• No mid-recording give-back: no visual problems or corruption through the end of the game.
• Budget-gated retry: no churn. Refused pools (state 3) showed up in 36 of 574 "bound in guest memory" summaries; on the first version of this PR it was a handful, before the PR it was thousands. The GPU read 96 MB over PCIe per 600-submit summary on average.

Separate from this PR, for reference: what is left is the budget itself, not the retry. In the collapsing castle the 2048 MB mirror budget was full, and a 208–272 MB pool stayed in state 1 (WANTED). The GPU read it over PCIe at up to 1.9 GB per 600 submits (~30–33 fps, readbacks waiting 2–6 ms each instead of ~0.1 ms). I then tried RX_VK_MIRROR_MB=2560 in Hyrule Field: VRAM filled within ~10 minutes (images placed in system memory, a 308 ms staging arena, a 3.4 s hitch). So on an 8 GB card the default 2048 MB looks like the right trade-off.

CI: merged into current main, the only check that fails is tests/test_direct_buffer_writes.py, and it comes from this PR. The check cuts gpu_buffer_bind out of vk_memory.cpp and compiles it with stubs, which don't declare mirror_bytes and mirror_budget() that the new budget gate uses:

direct_write_actual.inc:189:13: error: use of undeclared identifier 'mirror_bytes'
direct_write_actual.inc:189:47: error: use of undeclared identifier 'mirror_budget'

The other red checks in this PR's CI run pass once current main is merged in.