[hckrman101]

Hyrule Field runs about 15 fps slower on main than on v2.0.11-588 (ba86aee), and uses almost twice the CPU. One of the game's job threads sits blocked for 5.5 ms of every frame in a CPU read of GPU results, and the other job threads poll while they wait for it. This PR removes that wait without undoing what #2125 and #2419 fixed.

What regressed

RTX 3090, Ryzen 9 7900X, Linux, 2560x1440, uncapped, standing in Hyrule Field from the same save:

	v2.0.11-588 (ba86aee)	main (fce53453)
fps	79	64-71
CPU a frame, all threads	39-40 ms	64-73 ms
readbacks that waited, of 3000	~1300, 0.13 ms each	~172, 5.5 ms each
time blocked in readbacks, per 3000	~170 ms	~940 ms

GPU time and the replay's own time are the same on both.

Cause

An all-thread sample of both builds (HOS_SAMPLE_ALL) shows the extra CPU is one item: sched_yield in the host's short-sleep loop, 0.28 cores on the old build and 1.92 on main. It is the game's six job threads (ModuleSystemWorker1-5 and sead::MainThread), which poll with 20 us sleeps while one of them is blocked. On main a job thread is inside a readback at 38.7 % of all instants, with 3.9 threads polling meanwhile.

The blocked read is of the results that the first part of the scene's submission writes (two buffers, alternating by frame). That submission then stops at an event wait and is finished later, on the presenting thread. Three things held the read up:

1. It waited for the whole submission. Since fix(vk): a fence wait covers every part of a split submission, not only the first #2419 a submission's value maps to its last part's recording, which is right for a fence. A readback used the same map, so it waited for all of the scene's GPU work (3-4 ms) for pages the first part had already written back.
2. It took sync_order_mu whenever any queue was stopped. fix(vk): with per-queue fences, a CPU read-back waits for the GPU work that writes it again #2125 added a look at the stopped queues before the read. The presenting thread holds that mutex for the whole replay of its submission and for its present, so the read waited for both (24 % of all instants, 3.9 ms a frame) for a look that let nothing on.
3. It took dev.present_mu to flush held submits. Held submits exist only with batched submits (RADV). A present holds that mutex about 4 ms a frame. A readback whose recording was already done waited there, and so did the host's copy bookkeeping (rx_nvn_replayed, called under its own lock), which queued the other job threads behind it. The old build has this one too.

The fix

All of it is on by default; the two switches only turn parts off, for A/B.

1. A readback waits for the part that writes its pages back. Each part of a split submission copies back, at its own end, what the GPU wrote in it. The page table now keeps that part with the page's value, and the read waits for the newest part of its pages. Fence waits are unchanged. RX_VK_READBACK_PART=0 restores the last part. The read still waits for its submission's last part, as on main, in two cases:
   • a page of its range was written in place (no mirror, as on the iGPUs of issue 997);
   • a part after its own reads guest memory in place: a pool bound with no mirror, a mirror's dirty pages that no capture holds, a mirror's first copy, a large texture unswizzled on the GPU, a texel buffer's view. The log's readbacks line counts these.
2. The look at the stopped queues happens only when it can let something on. That is after a CPU signal (nvnEventSignal) nobody has looked at since, or while a queue is stopped at an event that neither the stream nor nvnEventSignal ever set, whose value is read from guest memory. A look counts as taken only once the burst it ran in has finished its recording, so its write-backs are in the page table. In the field 0-9 of 3000 readbacks look; on main every readback that found a queue stopped did. RX_VK_READBACK_SETTLE_ALWAYS=1 restores the look whenever anything is stopped.
3. A flush with nothing held does not take dev.present_mu. An atomic says whether any submit is held. With batched submits a recording is now put where waiters find it only after its submit is in the held list, under the mutex, which also closes a window main has. A wait returns at once when the recordings' timeline has already reached it. No switch.

The trade-off against main

Two reads of guest memory at the GPU's time are deliberately not counted for the part wait. Both behave as they did in v2.0.14 and before, that is before #2419:

• Shaders following pointers into a pool that was never bound, and so has no mirror. No call of the renderer's is made for those reads, so they cannot be noted. On a discrete GPU this is the remaining way to a frame of issue 997's kind that current main's whole-submission wait would have prevented: the game, let go after the first part, writes memory that a later part still reads through such a pointer. GPUs with no mirrors at all (997's iGPUs) are not affected: every page there is written in place and keeps the whole wait.
• The refresh of readback pages into their mirrors (memory_rb_refresh, in every part's uploads). These copies are how a CPU write to a page it reads back reaches the mirror at all, and a later part that writes such a page again is waited for through its own part. Counting them would put an in-place read in every part and cancel the change. What they carry along is any other data that shares a 4 KB page with a readback range. mooki000 counted it on an RTX 4090 (see their comment in this PR). In Kokiri Forest 0 of 20,371 readbacks shared a page this way. In Hyrule Field 610 of 45,916 did, all with read-only bindings, and in every case the reading part's recording was already complete when the readback's wait returned.

Review

Three commits:

• 798d625a: the three parts above.
• 9daa0718: first review. The part wait gives way to in-place reads in later parts; the look also covers events whose value is in guest memory.
• 4ba31740: mooki000's two findings on this PR, both real, and the second review.
  • A look was counted at its own end, before its burst's recording was finished; a readback in between took no look and read the page table without that recording's pages (fix(vk): with per-queue fences, a CPU read-back waits for the GPU work that writes it again #2125's hole again). Fixed: counted at the burst's end.
  • A held submit's recording could be found before the submit was in the held list, so a waiter skipped the flush and waited until something else sent the batch. Fixed as in part 3.
  • The second review added the unswizzle and texel-buffer reads, and bindings made outside a recording's uploads, to the in-place reads.

Measured

Hyrule Field, 30 s each, alternating builds in one session. Rounds 1-3 are 798d625a's renderer (built before its rebase onto 6f7ab2a7) on main's game binary (fce53453); the last two rows are the later commits, built as committed.

round	fix	v2.0.11-588	main
1	78.6 fps, 46.7 ms CPU	81.3 fps, 38.3 ms	63.1 fps, 71.0 ms
2	78.7 fps, 45.1 ms	73.9 fps, 40.0 ms	65.9 fps, 69.5 ms
3	69.9 fps, 50.5 ms (disturbed)	74.8 fps, 42.5 ms	67.2 fps, 68.8 ms
9daa0718	76.8 fps, 47.6 ms	66.4 fps, 42.4 ms (hitched)	60.9 fps, 73.4 ms
4ba31740	77.2 fps, 47.0 ms	not run	61.1 fps, 75.5 ms

Round 3's fix run had 367,000 pages swapped in on the PC during its 30 s (the others 20,000-135,000), so I count it as disturbed. The PC was short of memory for the whole session, which is also why the old build swings between 66 and 81.

Time blocked in readbacks, per 3000: about 220 ms with the fix (1500-1540 waits of 0.14-0.16 ms), about 230 ms on the old build, about 1100 ms on main. The counts themselves are not comparable between builds: main counts a readback as waited whenever the replay was behind.

Kokiri Forest, 798d625a, two rounds: fix 65.3 / 58.2 fps at 68-69 ms of CPU a frame, main 40.5 (a hitched run) / 45.9 at 113-117 ms, the old build 50.8 / 56.0 at 84-88 ms. The number of draws there varies from run to run (955 to 1255) on every build, so these are rougher than the field's.

The CPU is still 6-8 ms a frame above the old build in the field. The read now makes one real wait a frame, for the first part on the GPU (1-2 ms), and about three job threads poll during it; on the old build those threads sat blocked on mutexes instead. That is the yield loop for sleeps under a millisecond. HOS_SHORT_SLEEP_TIMER=1 on main brings the CPU to 37.5 ms a frame at the same fps, so it is a separate follow-up in the host, not part of this PR.

SadTransGirl measured the cost of #2419 independently with an RX_VK_WAIT_FIRST_PART switch (branch) on an RTX 4070 SUPER, DLAA 1440p: about 4-7 fps (Town to Hyrule Field 55.9-56.2 before #2419, 47.4-47.5 on main, 52.7-54.2 with the switch).

Tested

• python -I -B tests/run_all.py on Linux at 4ba31740: 447 passed, 3 failed, 25 skipped. The three are not this change: test_bake_portability and test_builder_log_read_only (no zstandard or tkinter on this PC) and test_env_on (HOS_FPS_LOG in host/oot_high_fps.c, on main).
• New: tests/test_readback_part.py runs the page table, the part bookkeeping, the readback's choice of wait, the look at the stopped queues and the burst's end, cut from the source, and counts every copy out of a guest pool. tests/test_flush_nothing_held.py checks the order in which a held submit is listed, flagged and made findable. Both fail on the commit before the fix they cover.
• Updated for the new code: tests/test_readback_settle.py, tests/test_recorded_wait_flush.py, and five fixtures that compile the changed functions.
• Flashes, RX_VK_FRAMELOG with scripts/framelog.py, 30 s each, 798d625a against the same build with both switches off:
  • Hyrule Field: 0 events in 2374 frames with the fix, 0 in 1869 without.
  • Kokiri Forest: 0 events without; 4 with the fix, all one 20x20 block on four consecutive frames. I looked at them: a leaf falling across the sky.
  • Mean brightness is the same with and without (field 37.6 / 37.6, Kokiri 89.7 / 90.0), so nothing is hidden by a blown-out picture.
• The same pass on 4ba31740, fix only: Hyrule Field 0 events in 2558 frames (brightness 37.6); Kokiri Forest 0 events in 1692 frames. That Kokiri run was the 964-draw variant of the scene, darker at 79.1, and I have no switches-off run of that variant to compare its brightness with.
• The game ran on NVIDIA 620.70.06, Vulkan 1.4, RTX 3090 (24 GB), VRAM mirrors on, batched submits off. In these runs no readback fell back for in-place reads and no pages were read live from guest memory.

Not yet verified

• 4ba31740 in the game beyond one field pair and one framelog pass (above): no Kokiri timing, no run of the old build beside it.
• The batched-submit path (RADV) on hardware. Part 3 changed in 4ba31740, so a check of 798d625a does not cover it; it is covered by a source test only.
• A GPU with no VRAM mirrors (issue 997's iGPUs), and a discrete GPU short of VRAM, where the in-place fallback would actually trigger.
• Windows.
• If a part's pages are noted and that part is then never submitted (a lost device), a read of them waits 250 ms before it falls back.

[mooki000]

Testing this on Linux with an RTX 4090 (NVIDIA 615.78.08, i9-13900K, Arch); I'll post the numbers for this exact commit as each run is read. Two things from your "Not yet verified" list are what I can give: Kokiri Forest timing, and the flash check with RX_VK_FRAMELOG and scripts/framelog.py against the same build with both switches off. I am building 798d625a itself, so that is the third.

What I have already, measured tonight before this pull request existed, and it agrees with your "one item": in a forest scene the whole loss of main is the readback wait, and nothing else in the range costs anything there.

Kokiri Forest, a save with camera turns, 2048x1152, the game's own upscaler, uncapped (HOS_FPS_MAX=240), headless, a 30 s window after the load, builds alternated A B A B inside one exclusive hold each, the same counting patch in every build (it counts every buffer readback and how long it waited):

build	fps	mean frame	1 % low	frames over 20 ms	waited in readbacks, a frame	load before → after
49f75dc	78.6	12.73 ms	45.0	21	0.02 ms	8.0 → 7.8
49f75dc	78.5	12.74 ms	38.9	18	0.02 ms	7.2 → 7.6
main 9df055f	62.9	15.89 ms	34.0	425	4.96 ms	9.3 → 21.2
main 9df055f	62.6	15.97 ms	37.1	445	5.02 ms	9.8 → 8.0
main 9df055f	62.5	15.99 ms	36.2	432	4.89 ms	8.1 → 7.6
main, both taken back	79.2	12.63 ms	45.9	23	0.02 ms	7.8 → 7.2
main, both taken back	80.0	12.50 ms	47.1	19	0.02 ms	7.6 → 5.9
main, RX_VK_READBACK_NO_CATCHUP=1	69.0	14.49 ms	43.2	33	3.11 ms	7.6 → 7.2
main, RX_VK_READBACK_NO_CATCHUP=1	68.0	14.70 ms	42.3	35	3.23 ms	13.8 → 8.6

• 49f75dc is v2.1.0-rc.3-53, the last build I had from before fix(vk): with per-queue fences, a CPU read-back waits for the GPU work that writes it again #2125 and fix(vk): a fence wait covers every part of a split submission, not only the first #2419. "Both taken back" is main with those two pull requests' code hunks reverted, a local diagnostic only.
• On main about one readback a frame waits, 4.8 to 5.0 ms; about 3.2 ms of that is in fix(vk): with per-queue fences, a CPU read-back waits for the GPU work that writes it again #2125's catch-up before the write-back lookup, which is what RX_VK_READBACK_NO_CATCHUP=1 removes. With both taken back the build is at the old build's rate (79.2 and 80.0 against 78.6 and 78.5 fps).
• Left out: a fourth run of plain main at 54.4 fps (1 % low 11.5) that was disturbed. The main-against-old runs had a load of 8 to 21 from other work on this PC; the revert runs were quiet.
• GPU time was the same in all nine (5.94 to 6.22 ms a frame).

Coming for 798d625a against its base 6f7ab2a7 and against 49f75dc: the forest and an open-field scene uncapped, the forest at a 60 cap with DLAA, your two switches one at a time, the flash check, and your new tests with gcc 16.2 and clang 23.1. Not something I can test: Windows, AMD, an integrated GPU, RADV's batched submits.

[mooki000]

One thing from reading the code while the timing runs build, in part 2 (the look only after a CPU signal). I think there is a window in which a readback can go through without waiting for work that schedule() has just let on. It comes from where ev_cpu_settled is stored. Lines are gpu/vk/vk_replay.cpp at 798d625a.

The order today

1. A thread settles: sync_settle() (2140) takes sync_order_mu, burst_begin(0), schedule() (2144).
2. schedule() lets the met queues on, which replays their work into the open recording, and at its end stores ev_cpu_settled = signals (2094).
3. Only then burst_end() (2145) takes replay_mu (1920) and finish_recording() runs memory_end_submission (1931), which is where the pages that work writes back get their new value in the page table.

The window is between 2 and 3. A readback on another thread calls sync_settle_met() (2463): the two counters are equal (2159), so it returns without taking sync_order_mu. It then asks the page table (memory_writeback_value, 2476), gets the value from before the work that was just let on, finds it complete, and returns without a wait. That is the read #2125 was for: the range's writer is not recorded yet and its write-back not known. On main the readback took sync_order_mu whenever anything was stopped, so it could not get past a settle that was still open; with RX_VK_READBACK_SETTLE_ALWAYS=1 it still does. The window is short, but burst_end() waits for replay_mu inside it. The same holds for the other places that call schedule() between burst_begin and burst_end (2171, 4559, 4635), and drain() (2105) goes on recording forced queues after the store.

Shown without a game. A fixture that cuts schedule, sync_settle, sync_settle_met, the page table and the readback's choice of wait out of this branch and holds the settling thread at burst_end:

default:                  before burst publication: signals=1 settled=1 read_returned=1 waited_recording=0   FAIL
RX_VK_READBACK_SETTLE_ALWAYS:  before burst publication: signals=1 settled=1 read_returned=0                  ok (it then waits for recording 2)

The fixture (it imports the helpers of your tests/test_readback_part.py): python -I <fixture> --source <checkout of this branch> --output <a folder> [--always]. I ran it with g++ 16.2; I have not seen a wrong frame from this in the game, and the flash check is still to come.

A way to close it, if you agree it is one: store ev_cpu_settled after the burst's recording is finished and its pages are noted (after burst_end() in each caller, with schedule() returning the count it read), or let the readback's quick test also see that a settle is open.

[mooki000]

A second one, in part 3 (the flush with nothing held), and only on the batched path, which your description lists as checked by a source test only. Same file and commit.

What the flag is meant to guarantee (your comment at 397): a thread that finds a recording in the maps and waits for it never skips the flush its wait needs. The flag is raised before the maps are written (399) and again after the append (435). In between, a flush of older submits clears it (249).

The order that breaks it

1. Producer: pend_held = true (399), its recording goes into the maps (406 to 424).
2. Another thread flushes the older batch: queue_flush_locked() ends with pend_held = false (249).
3. The producer waits for, or already holds, dev.present_mu (428) and has not appended yet (434) nor raised the flag again (435).
4. A readback finds the new recording in the maps and waits for it: queue_flush_for(FR_WAIT) (2256) sees the flag down and returns without the mutex (263); the wait that follows has no time limit (2488, 2489) and is for work that is still to be held.

After that the readback waits until something else sends the batch: the producer's own batch_gpu_idle() check (443), the cap, a poll flush or a present. So it is a wait of up to a frame on the path this pull request frees from exactly that, not a wrong read. With the mutex always taken, as on main, the waiter in step 4 queued behind the producer when the producer already held the mutex, and flushed after it; when both were waiting for the mutex main could lose the same way, so main was not strict here either, but the quick test makes the skip certain for the whole of step 3.

Shown without a game. The real submit_recording, queue_flush_locked, queue_flush_for and wait_rec_seq bodies cut from this branch, Vulkan replaced by a model that says what was sent, the producer paused before its append with the mutex held:

default:                      new part mapped=2 after old flush pend_held=0 ... wait_ok=0 GPU=1 target=2 held_cbs=1   FAIL: waiter skipped its flush
quick test removed (control): new part mapped=2 after old flush pend_held=0 ... wait_ok=1 GPU=2 target=2 held_cbs=0   PASS

The fixture: python -I <fixture> --source <checkout of this branch> --output <a folder> [--locked-control]. It fails by default and passes with the control with g++ 16.2 and clang++ 23.1 here. (The first fixture now builds with clang++ too, same result: updated.) I have no RADV card, so I cannot say how often this order happens in play.

A way to close it: count the producers that have raised the flag and not appended yet (up at 399, down after 435 under the mutex), and let the quick test return only when the flag is down and that count is zero; or have the flush leave the flag up while that count is not zero.

[hckrman101]

@mooki000 thanks for testing. Heads up: I pushed 9daa071 on top of 798d625. It fixes two review findings: the readback's part wait now gives way to the whole-submission wait while any part reads guest memory in place (direct-bound pools, live mirror syncs), and the look at stopped queues also covers events in guest memory. On a GPU with enough VRAM for mirrors it shouldn't change timing (no in-place reads in our Field/Kokiri logs). Your 798d625 numbers still count for perf; the flash check would be most useful on 9daa071.

[mooki000]

@hckrman101 thanks, the flash check and the timing will be on 9daa0718 then (I build that one; the forest table above stays as the picture of main).

One note so the two comments above do not get lost under the new head: both still apply to 9daa0718 as far as I can see.

• The settle counter: schedule() still stores ev_cpu_settled at its end (now line 2107) and sync_settle() still calls burst_end() after it, so the quick test in sync_settle_met() can still pass before the pages of the work just let on are noted. The new ev_wait_in_memory test sits beside it and does not change that order. My fixture cuts the old quick test out of the source and no longer builds against the new one; I'll adjust it and update the link.
• The held-submit flag: lines 249, 263, 400 and 436 are as before, and the batch fixture gives the same result on 9daa0718 (fails by default, passes with the quick test removed; g++ 16.2).

[hckrman101]

@mooki000 thanks, both are real as far as I can see from the code, and both are good catches.

• Settle window (part 2): agreed. ev_cpu_settled is published before burst_end() notes the pages, so a readback can pass the quick test and read the old value: exactly the read fix(vk): with per-queue fences, a CPU read-back waits for the GPU work that writes it again #2125 is for. The fix will publish the count only after the burst's recording is finished and its pages are noted, for every schedule() caller and drain().
• Held-submit flag (part 3): agreed. A flush of the older batch can drop the flag between the producer's mapping and its append, and the waiter then skips the flush it needs. Not a wrong read, but up to a frame of waiting on exactly the path this PR is meant to free. The fix will keep the quick path closed while any producer has raised the flag and not appended yet.

Both get tests of their own that fail on 9daa0718 and pass with the fix. I'll push them on top and ping here. Until then the PR stays in draft. Your flash check and timing on 9daa0718 are still useful, since neither change should move the frame rate on NVIDIA.

[mooki000]

Thanks. In case it helps for the two tests: the settle fixture adjusted to 9daa0718 is here. On that head, with g++ 16.2 and clang++ 23.1:

default:   before burst publication: signals=1 settled=1 read_returned=1 waited_recording=0   FAIL
--always:  before burst publication: signals=1 settled=1 read_returned=0                      ok (then waits for recording 2)

It models an event the CPU signalled (ev_wait_in_memory false) and asserts that no later in-place part hides the window, so 9daa0718's new guards are not what it trips over. The batch fixture linked above runs on 9daa0718 unchanged.

Timing and the flash check on 9daa0718 are building; numbers follow here.

[mooki000]

A third one, on 9daa0718's new guard (the part wait gives way while a later part reads guest memory in place). The guard goes by note_in_place_read(), and one way of reading guest memory in place does not call it.

• note_in_place_read() (gpu/vk/vk_memory.cpp 1652) is called at five places: 2443, 2750, 2782, 2810, 3118. Its comment names them: a pool bound with no mirror, a mirror's dirty pages no capture holds, a mirror's first copy.
• gpu_buffer() (2578) hands out the imported guest buffer itself (p->buffer) and does not call it. Its two users give that buffer to the GPU to read at the GPU's time: the GPU unswizzle's source (gpu/vk/vk_texture.cpp 1142, bound as a storage buffer at 1177) and the texel buffer views (3580, 3601).
• So a submission whose first part writes a page back and whose later part reads guest memory through one of those two leaves memory_in_place_part() where it was; in_place_after is false (vk_replay.cpp 2509) and the readback waits for the first part only (2511). That is the early return issue 997 was about, for these two readers.

Shown without a game. The real page table, readback choice, recording bookkeeping, marker and gpu_buffer cut from 9daa0718; two parts, the first writes the page back, the second takes it through gpu_buffer:

default:                         FAIL  a later raw guest-buffer read keeps the whole-submission wait   (waits for part 1)
note added at that one call (control):  ok                                                              (waits for the last part)

The fixture: python -I <fixture> --source <checkout> --output <a folder> [--noted-control]; g++ 16.2 and clang++ 23.1, same result. I am not claiming a wrong picture from it: whether the game overwrites a texture's source or a texel buffer's memory that soon I have not shown.

By reading only, not run: memory_rb_refresh copies pp->buffer into the mirror at the GPU's time (2567) without the note as well.

A way to close it: call note_in_place_read() in gpu_buffer() itself, so every present and future user of the raw buffer is covered, and at the refresh copy.

[mooki000]

The refresh copy from my last comment is now run as well, no longer only read: the same fixture with --refresh cuts the real memory_rb_refresh body, which records one copy from the imported guest buffer into the mirror, and the marker stays where it was:

--refresh:                  FAIL  a later guest-to-mirror refresh keeps the whole-submission wait   (waits for part 1)
--refresh --noted-control:  ok                                                                       (waits for the last part)

g++ 16.2 and clang++ 23.1 on 9daa0718, same result; the gpu_buffer case is unchanged.

[hckrman101]

@mooki000 thanks again. Both of these were on 9daa0718; 4ba31740 (now the head) handles them:

• gpu_buffer(): it now calls note_in_place_read() itself, as you suggested, so the GPU unswizzle's source and the texel buffer views are covered, and so is any future user of the raw buffer.
• memory_rb_refresh: deliberately not noted, with the reason in the comment at note_in_place_read and in the PR description's "trade-off against main" section. Those copies run in every part's uploads and are how a CPU write to a page it reads back reaches the mirror at all. A later part that writes such a page again is waited for through its own part. Noting the refresh would put an in-place read in every part and cancel the change. So what it leaves open is other data sharing a 4 KB page with a readback range, the same as before fix(vk): a fence wait covers every part of a split submission, not only the first #2419. If you think that residual is worse than it sounds (e.g. a readback range that regularly shares its page with data a later part reads), I'd like to hear it.

4ba31740 also has your two races fixed (the count is published at burst_end, and a held submit is mapped only after it is appended under present_mu), plus a second review. First field pair on it: fix 77.2 fps / 47.0 ms CPU, main 61.1 / 75.5. The flash check is running.

[hckrman101]

Status: ready for review at 4ba31740.

Measured on 4ba31740 (Hyrule Field, RTX 3090 / 7900X, 2560x1440, uncapped, same session): fix 77.2 fps / 47.0 ms CPU a frame, main 61.1 fps / 75.5 ms. Earlier builds of the change: 76.8–78.7 fps in the field; Kokiri 65.3 / 58.2 fps against main's 40.5 / 45.9 (798d625a).

Review:

• An independent review of 798d625a found the in-place reads in later parts and the events in guest memory; both fixed in 9daa0718.
• @mooki000 found two races on 9daa0718: the look being counted before its burst's pages were noted, and a held submit being found before it was held. Both fixed in 4ba31740, each with a test that fails on 9daa0718.
• The direct reader through gpu_buffer() is noted in 4ba31740. The readback-page refresh is left out on purpose; see "The trade-off against main" in the description.
• A final review pass of 4ba31740 found it correct: no new lock-order problem (the one new edge is present_mu → rec_map_mu, and rec_map_mu is a leaf).

Still running: the framelog flash check on 4ba31740 in the field and Kokiri, and the batched-submit path (RADV, RX 580). Results will be posted here.

Not testable here: Windows, a no-mirror iGPU.

If something turns up after merging: RX_VK_READBACK_PART=0 and RX_VK_READBACK_SETTLE_ALWAYS=1 restore main's readback waits.

[hckrman101]

Flash check on 4ba31740 (RX_VK_FRAMELOG + scripts/framelog.py): Hyrule Field 0 events in 2558 frames, Kokiri Forest 0 events in 1692 frames. Readbacks in the field pair: 1534-1551 of 3000 waited, at 0.10-0.12 ms each (main: 172-176 at 6.3-6.8 ms). 0-1 of 3000 took the look at the stopped queues, and 0 fell back to the whole-submission wait for in-place reads. The RADV batched-submit check is still to come.

[mooki000]

Checked 4ba31740 against the three fixtures, each adjusted to the moved code with its assertion unchanged (settle, held submit, in-place readers). g++ 16.2 and clang++ 23.1, same result with both:

fixture	9daa0718	4ba31740
the settle count and the burst's pages	fails	passes: the readback waits for recording 2
a held submit found before it is held	fails	passes: nothing is in the maps before the append, then the waiter flushes
gpu_buffer() as an in-place reader	fails	passes: waits for the last part
the refresh copy (--refresh)	fails	fails, as you decided: it is not noted

So the two races and the gpu_buffer() reader are closed as far as these can show. What the fixtures cannot show is a path that reaches the same state another way; I am reading the callers of schedule() and drain() and the two mutexes' order for that and will say what I find.

On the refresh copy: your argument makes sense to me from the code, and I would rather answer your question with counts than with a guess. I am taking them from this PC's runs (how often a readback range's page also holds data a later part reads through a refresh, forest and open field) and will post them with the timing and the flash check on 4ba31740.

[mooki000]

First timing here, on an RTX 4090 (NVIDIA 615.78.08, i9-13900K, Arch Linux). This hold ran on 9daa0718, just before 4ba31740 came; the same arms on 4ba31740 are building and follow.

Kokiri Forest, uncapped (a save with camera turns, 2048x1152, the game's own upscaler, headless, 30 s after the load, base and fix in turn inside one exclusive hold, the same counting patch in both):

build	fps	mean frame	1 % low	frames over 20 ms	CPU a frame, all the game's threads	waited in readbacks, a frame	other work on the PC (cores, highest 5 s)
6f7ab2a7 (base)	63.2	15.81 ms	33.1	399 of 1895	77.0 ms	4.85 ms	1.8 (2.0)
9daa0718	80.5	12.42 ms	46.3	20 of 2415	54.9 ms	0.60 ms	1.6 (1.7)
6f7ab2a7 (base)	64.6	15.48 ms	37.0	375 of 1936	74.9 ms	4.71 ms	1.7 (2.1)
9daa0718	80.1	12.48 ms	46.5	17 of 2402	54.9 ms	0.59 ms	1.6 (1.8)

The fix: 3.20 ms a frame less (15.65 to 12.45), 16.4 fps more, 21.1 ms less CPU a frame, and the long frames gone (399 and 375 over 20 ms against 20 and 17). GPU time is the same in all four (5.91 to 5.99 ms a frame). For scale, in other holds tonight the build from before #2125 and #2419 (49f75dc) ran this scene at 78.4 to 78.6 fps; the comparison with it in one hold comes on 4ba31740.

What main costs under a 60 cap, since most people play capped: the same scene with DLAA and the 60 cap, 49f75dc against main 9df055f2 (one exclusive hold, run before this pull request's builds were ready):

build	fps	mean frame	1 % low	frames over 16.7 / 20 / 25 ms	waited in readbacks, a frame	load before → after
49f75dc	58.4	17.12 ms	33.0	1024 / 85 / 11 of 1746	0.02 ms	7.9 → 19.4
main 9df055f2	54.5	18.34 ms	32.1	1005 / 456 / 43 of 1635	5.88 ms	19.4 → 9.9
49f75dc	58.9	16.99 ms	41.1	1000 / 85 / 3 of 1764	0.02 ms	9.9 → 6.5
main 9df055f2	54.2	18.44 ms	31.6	1040 / 477 / 44 of 1626	5.97 ms	6.5 → 7.1

So even capped, main is 4.2 fps lower here, with about 5 times the frames over 20 ms. The same arm with the fix follows.

Reading 4ba31740 for the two races moved elsewhere: nothing found. All six callers of schedule() leave the count to burst_end() when a burst is open, and burst_end() publishes it after finish_recording(); drain()'s early return comes before any look and its forced resumes stay inside the burst. rec_map_mu is taken at four places, only ever inside present_mu and never the other way round.

Still to come from here: forest and open field on 4ba31740 against its base and against 49f75dc, the 60-cap arm, your two switches, the flash check, and counts for the refresh question.

[mooki000]

Counts for the refresh question (whether leaving memory_rb_refresh out of the in-place readers can hand later work bytes newer than its submission expected).

How it was counted: 4ba31740 plus an observation-only counter, no wait changed, RX_VK_PARALLEL=0 so that a part belongs to one recording. It logs every readback after its wait, the refresh copies actually made (guest to mirror), the mirror bindings on pages that have had a readback, and each part's recording with whether it had completed. A readback is "sharing" when a later part of the same submission value refreshes its page and binds bytes of that page outside the readback's own range. RTX 4090, headless, one 30 s window a scene.

scene	frames	readbacks	sharing	of those, the binding part's recording not complete when the readback's wait returned
Kokiri Forest	930	20,371	0	0
Hyrule Field	2,707	45,916	610 (1.33 %; 0.23 a frame, on 305 frames)	0

All 610 are one shape: a 192-byte readback with a read-only 16-byte binding that starts 464 bytes before it and a read-only 64-byte binding that starts 192 bytes after it, both outside every readback range of that submission. In each case the recording of the part that binds them had completed when the readback's wait returned, so the game could not have changed those bytes under it.

So in these two scenes the choice costs nothing: the sharing is rare, read-only, and never still in flight. This is an answer in numbers for one card and two scenes, not a proof:

• a bound range is an upper bound on what a shader reads, and a load through a pointer without a binding is not seen at all;
• bindings made before a page first had a readback are not logged;
• serial replay only, 30 s a scene.

The counter is a small local patch on top of 4ba31740 (off unless its switch is set). If you want it for another scene or for RADV, say so and I put it in a gist.

[hckrman101]

Batched-submit path on hardware: RX 580 (RADV, GCN), with batching forced on (RX_VK_BATCH=1; it is off by default on GCN), Kokiri Forest, 150 s of play per run, headless.

run	build / env	fps (min)	readbacks waited per 3000
A1 (cold)	4ba31740 defaults	21.3 (18.5)	401 × 12.5 ms
A2 (warm)	4ba31740 defaults	22.0 (19.7)	366 × 13.7 ms
B1 (warm)	4ba31740, RX_VK_READBACK_PART=0 RX_VK_READBACK_SETTLE_ALWAYS=1	23.3 (23.1)	164 × 30.5 ms
C1 (cold)	main 6f7ab2a7	23.3 (21.6)	164 × 31.5 ms

• No hang, no device loss, no Vulkan errors, and nothing misrendered in screenshots of every run. The held-submit order in 4ba31740 works on the batched path.
• On this GPU-bound card the part wait does not help. Readbacks wait about 2.3× as often and each wait is about 2.3× shorter, so the total wait is unchanged, and fps is about 1.3 (5-6 %) lower than with the switches off or on main. That is one or two runs each, so it is a small gap, but it showed in both the cold and the warm pair. I'm looking into why.
• Not covered: RDNA / Steam Deck, where batching is on by default.

[luiz-august0]

The same readback wait on Windows, with an 8 GB NVIDIA card and a 6-core CPU

My logs show the wait this PR removes, on a build that contains fce53453. These are the "main" numbers for another platform; I have not run this PR's build yet.

• Build: v2.1.0-rc.3-191-gef81aee7 (nightly)
• PC: RTX 4060 8 GB (NVIDIA 610.88, PCIe 3.0 x8 on an A320 board), Ryzen 5 5600X (6 cores, 12 threads), 32 GB, Windows 11 Pro 26200
• Settings: fullscreen, output 1920x1080 on a 3840x2160 60 Hz display, DLSS Quality (1280x720 in), Reflex on + boost, V-Sync Auto (mailbox), frame generation off,
  RX_VK_WB_MAX_KB=1024 in hookshot_env.txt. VRAM budget Auto in session A, 4.8 GB in session B.

	session A (hookshot_log.1.txt)	session B (hookshot_log.txt)
frames counted	100,920	4,560
fps in gameplay, median (p10-p90)	41.7 (29.1-53.0)	38.5 (31.9-42.1)
GPU time a frame, median	9.9 ms	12.9 ms
readbacks that waited, of 3000 (median)	110	183
ms each (weighted mean)	9.1	12.5
time blocked in readbacks, per 3000	~1000 ms	~2290 ms
waits a frame	0.96	0.94
time blocked a frame (a job thread, sometimes sead::MainThread)	8.6 ms	11.5 ms

• The Presentation Thread's GPU-bound wait is 0.0 ms in every report, so this blocked read is where the GPU's time enters the frame here.
• Each wait is about one frame's GPU time. That fits cause 1, the wait for the whole submission.
• Compared with your main numbers (172-176 of 3000 at 6.3-6.8 ms on the 3090), each wait here is about twice as long, which fits the slower GPU.
• Batched submits are RADV only, so I take it cause 3 does not apply here.
• Session B also had the VRAM budget at 4.8 GB. That caps the buffer mirrors at 1 GB, and one 144 MB pool stayed in system memory, read over PCIe. So B's GPU time is higher, and so are
  its waits. That is a separate topic.

How I counted: from the log's readbacks of GPU results lines, against the N presents counter. Gameplay is the report periods with more than 300 draws a frame.

[hckrman101]

@mooki000 thank you, that's a lot of useful data.

• Kokiri: +16 fps on 9daa0718 (63-65 → 80), 21 ms less CPU a frame, and the frames over 20 ms down from ~390 to ~20. That puts it just above your pre-fix(vk): with per-queue fences, a CPU read-back waits for the GPU work that writes it again #2125/fix(vk): a fence wait covers every part of a split submission, not only the first #2419 build (78.4-78.6), which matches what we see in the field.
• The 60-cap numbers are the most important part for players: main costs ~4 fps and about 5× the frames over 20 ms even when capped. I'm looking forward to the fix arm of that.
• The refresh counts settle the trade-off well for these scenes: 0 sharing in Kokiri, 610 of 45,916 in the field, all read-only, all with the reading part's recording already complete. I'll link your comment from the description's trade-off section. No need for a gist for now; if the RX 580 slowdown leads us to touch that path, I'll ask.
• And thanks for re-reading 4ba31740 for the races.

@luiz-august0 thanks, this is the Windows data we were missing. About one wait a frame at 9-12 ms, roughly one frame's GPU time, is the same cause-1 picture on a 4060 / 5600X. Right: batched submits are RADV only, so cause 3 doesn't apply to you. If you can run this PR's build (the Windows CI artifact of 4ba31740) with the same settings, the readbacks line and gameplay fps against that nightly would be very valuable. Windows is the one platform nobody has run it on yet. The 4.8 GB budget / 144 MB pool in system memory is worth its own issue.

[mooki000]

The capped arm with the fix, as asked: Kokiri Forest at a player's settings (DLAA, the 60 cap, 2048x1152), headless, RTX 4090 (NVIDIA 615.78.08), base and fix in turn in one exclusive hold, all four runs counting. This hold ran on the pull request's head 4ba31740 just before the merge; the same on merged main is running now.

build	fps	mean frame	1 % low	frames over 16.7 / 20 / 25 ms	CPU a frame, all threads	waited in readbacks, a frame
6f7ab2a7 (before the fix)	54.2	18.44 ms	28.6	1013 / 468 / 39 of 1625	83.3 ms	5.95 ms
4ba31740 (the fix)	59.3	16.87 ms	38.9	952 / 64 / 10 of 1777	57.3 ms	0.70 ms
6f7ab2a7 (before the fix)	53.9	18.55 ms	23.6	1009 / 484 / 47 of 1616	83.1 ms	5.95 ms
4ba31740 (the fix)	59.2	16.89 ms	39.6	966 / 71 / 7 of 1775	58.6 ms	0.70 ms

With the fix the scene holds 59.3 and 59.2 of 60 (before: 54.2 and 53.9), frames over 20 ms go from 468 and 484 to 64 and 71, over 25 ms from 39 and 47 to 10 and 7, and the game uses about 25 ms less CPU a frame. For scale, the build from before #2125 and #2419 in another hold tonight (so not a paired comparison): 58.4 and 58.9 fps, 85 and 85 frames over 20 ms.

Uncapped, the fix against the build from before the two changes, in one hold each (own upscaler), and the field against the base:

scene	build	fps (two runs)	mean frame	CPU a frame	waited in readbacks, a frame
Kokiri Forest	4ba31740 (the fix)	79.2 / 81.2	12.63 / 12.31 ms	55.5 / 54.4 ms	0.57 / 0.56 ms
Kokiri Forest	49f75dc (before #2125 and #2419)	79.3 / 79.5	12.61 / 12.58 ms	54.6 / 53.6 ms	0.02 / 0.02 ms
Hyrule Field	4ba31740 (the fix)	92.6 / 92.1	10.80 / 10.86 ms	49.8 / 50.0 ms	0.33 / 0.33 ms
Hyrule Field	49f75dc (before #2125 and #2419)	91.5 / 91.9	10.93 / 10.88 ms	49.4 / 49.4 ms	0.02 / 0.02 ms
Hyrule Field	6f7ab2a7 (before the fix)	69.2 / 68.6	14.45 / 14.59 ms	71.1 / 71.8 ms	4.83 / 4.71 ms
Hyrule Field	4ba31740 (the fix)	92.8 / 92.7	10.78 / 10.79 ms	50.2 / 50.2 ms	0.41 / 0.31 ms

Against the old build the ranges overlap in both scenes, so nothing is left over on this PC: the fix gives back all of it. What remains of the readback wait is 0.6 to 0.7 ms a frame in the forest and about 0.3 ms in the field (0.02 ms on the old build).

Still coming from here, on merged main: this capped hold again, and the flash check with a window and the pixel frame log (a run with the readback wait switched off first, to show the check sees a flash at all).

[felipefl142]

Results from an 8 GB laptop card on Linux. This PR fixes the main thing that held my game back: one job thread blocked in readbacks about a third of the time. At the same scene size, fps went up 30-50%.

Setup: RTX 3070 Laptop 8 GB, NVIDIA 620.70.06, CachyOS (Linux). DLSS (built with the SDK), output 1920x1080, RX_VK_VRAM_MB=3072, RX_VK_COMPACT=1, built-in frame interpolation (DLSS-G needs an RTX 40). Both builds also have #2482 merged.

• Before: main ef81aee + fix(vk): fall back to host pointers on udmabuf rejection and retry mirror allocation #2482 (v2.1.0-rc.3-195-geb0c4a2b), a 102-minute session.
• After: main 09aac43 (this PR as 76bbe4aa) + fix(vk): fall back to host pointers on udmabuf rejection and retry mirror allocation #2482 (v2.1.0-rc.3-200-gd82f41c7), a 52-minute session: the Shadow Temple, then Lake Hylia and the Desert Colossus.

The commits in between are #2598, #2347 and #2556. On this card DLSS-G is not available, so none of those three should change performance. The two sessions are normal play, not an A/B run in one spot, and they took different routes. So I compare them by draw count rather than by place.

Readbacks

	Before	After
Readbacks that waited, of 3000 (median)	145	503
Each wait (median)	9.9 ms	0.1 ms
Time blocked, per 3000 readbacks (median)	~1100 ms	~60 ms
Replay submit a frame (median)	11.7 ms	8.9 ms
GPU time a frame (median)	11.6 ms	11.8 ms

More readbacks wait now, but each wait is about 100 times shorter. Before this PR the line came out about every 5.4 s. So a game thread was blocked for about 1.1 s of every 5.4, the same picture as in your Hyrule Field trace. GPU time did not change, so the gain is all on the CPU side.

Fps at the same scene size

Draws a frame	Before	After
400-700	34.8 fps	51.6
700-1000	30.7	46.3
1000-1300	28.3	37.7
1300-1700	28.1	33.0
1700-2200	24.4	30.8
2200+	21.7	26.5

(Median of the presents lines' fps, grouped by their draw count.)

On an 8 GB card the gain is larger than the 3090 numbers in the description. Before this PR my waits were 10-20 ms each, not 5.5 ms. The gain shrinks as scenes get bigger, where the replay itself (#995) takes over.

I saw no visual problems: no flashes, no missing shadows or geometry. There was one 930 ms hitch, when VRAM ran out entering a large area (VRAM full x4, a 64 MB staging arena took 159 ms to make). It is memory pressure (#2412), not this PR.

[DFN-master]

Windows test: Hyrule Field, stock and with our backend switches

Setup

• Windows 11, 64-bit; Intel Core i7-11700 (8C/16T); 32 GB RAM; NVIDIA GeForce RTX 3080 10 GB; 1920x1080 @ 60 Hz.
• Build: 55b976c0 (one commit past the PR's base 6f7ab2a7, only the DLSS frame-generation commit, no vk_replay/vk_memory change) + the three commits of this PR (798d625a, 9daa0718, 4ba31740), applied cleanly. Vanilla lift, no PGO. hookshot.exe 469,956,608 B, hookshot_vk.dll 11,190,272 B.
• Same exe for every arm; the "main waits" arm is the same build with RX_VK_READBACK_PART=0 and RX_VK_READBACK_SETTLE_ALWAYS=1. Headless, fixed save, scripted menu input, teleport to HyruleField_Adult @ HyruleField_FrontOfLostWoodBridge, 330 s per run, ~1430 draws; fps counted after the teleport and 600 frames, readback lines from steady play.
• All arms share the same 6.4 GB program cache and pipeline cache. No shader/pipeline failures.

Result (Hyrule Field, 1080p, same save and route)

arm	fps avg (min-max)	readbacks that waited, of 3000	ms each	last-part fallback
stock waits (PART=0, SETTLE_ALWAYS=1)	48.12 (42.7-49.2)	172	5.53-5.68	n/a
stock fix (default)	59.29 (56.3-60.5)	1536-1552	0.02-0.03	0
fix + RX_VK_VRAM_MB=4096, RX_VK_COMPACT_PCT=90	59.83 (57.6-60.4)	1539-1550	0.02-0.03	0
fix + RX_VK_READBACK_NO_CATCHUP=1	59.96 (58.1-60.3)	1531-1550	0.02	0
fix + our full config without SYNC_LIVE (4 GB, pct 90, NO_CATCHUP, WB_MAX 1024)	59.77 (57.2-60.5)	1340	0.01	0

This is the same direction as your RTX 3090 result: +11.2 fps stock (+23%), 5.5 ms -> 0.02 ms, and the long frames go away (min 42.7 -> 56.3). Windows was on the "not yet verified" list; this covers it for one NVIDIA machine at 1080p.

Kakariko Village (same build and final config, ~2510 draws): 47.4 fps avg (40.4-50.0, last 48.2), 1227-1245 readbacks of 3000 at 0.15-0.16 ms, 0 fallback. The town is CPU-bound; the readback wait is no longer what limits it.

The one interaction we found: RX_VK_SYNC_LIVE=1

Our config sets RX_VK_SYNC_LIVE=1 (dirty pages not in the submission's capture are read from guest memory). With only that switch added to the stock build:

arm	fps avg (min-max)	readbacks that waited	ms each	last-part fallback
SYNC_LIVE=1 fix	51.39 (49.4-52.7)	482-1068	0.55-5.43	309-1022
our full config fix (SYNC_LIVE + 4 GB + pct 90 + NO_CATCHUP + WB_MAX)	53.25 (51.1-54.9)	838	0.83	820
our full config, main waits	53.86 (52.6-55.2)	753	0.91	n/a

The log's own counter shows why: with SYNC_LIVE=1 the mirror syncs read 80,954 pages "from guest memory as it was at the replay" per 600 submits (0 with it off), so a later part reads guest memory in place and the part wait gives way to the whole-submission wait for ~90% of the readbacks. That is exactly the guard you added in 9daa0718, not a wrong read; the switch and the part wait just cannot both apply. We will drop SYNC_LIVE in our config and keep the part wait.

RX_VK_VRAM_MB=4096 + RX_VK_COMPACT_PCT=90 alone and RX_VK_READBACK_NO_CATCHUP=1 alone do not trigger the fallback (both at 59.8-60.0 fps, 0 fallback), so the 4 GB texture budget is not the problem.

Checks

• tests/test_readback_part.py, tests/test_flush_nothing_held.py, tests/test_readback_settle.py, tests/test_recorded_wait_flush.py: pass on Windows at this head.
• 0 looked at the stopped queues first in every fix line matches your field count; SETTLE_ALWAYS=1 restores the look (172 of 3000 in the stock-waits arm), as documented.
• No framelog/false-flash pass on Windows yet; that is the one check we have not run here. No screenshots were uploaded.

Limits

• One NVIDIA machine, 1080p, one save and route; no AMD/Intel, no iGPU without mirrors.
• The stock-waits arm is the same build with the two switches, not a separate main build; the third part (flush with nothing held) stays on in both arms and has no switch.
• The byte sizes above are this build's; the PR's own base and lift may differ.

[DFN-master]

One more data point for the absolute level, same save and route, 1080p: my current playable build (4a53c897 + zstd + BC7, readbacks 0.02 ms) runs it at 59.67 fps avg (56.4-60.5, last 60.3), 1325/3000 readbacks at 0.02 ms, 7142 MB VRAM, 3490 MB textures. The PR at 55b976c0 with the 4 GB config runs it at 59.77 fps avg (57.2-60.5, last 60.3), 1340/3000 at 0.01 ms, 7280 MB VRAM, 3495 MB textures. So the PR brings newest main to the old build's level in this scene, against 48.12 fps on main without it.

[mooki000]

An integrated GPU without VRAM mirrors, which the description listed as untested: Intel UHD Graphics 770 (RPL-S), Mesa 26.2.5 (ANV), Linux 7.2.15-1-omarchy, i9-13900K. Merged main 09aac439 plus a local counting patch, with what the game chose for this device at its first start (the log: the Minimum (4GB) preset, which sets Resolution to 50 %; VRAM mirror of buffer pools off; the compute fallback for mesh shaders), 2048x1152 output without a window, Kokiri Forest from a save, standing still. Started with RX_VK_PRECOMPILE=0 RX_VK_NO_WARM=1 RX_VK_NO_PREDICT=1: without them the driver's shader compiles take more memory than a run may have here.

The part wait does not act on this card, so there is nothing for it to cost.

• In the code: a readback's part is noted only where a mirror's pages are copied back (wb_pages_note(..., part) in the write-back of gpu/vk/vk_memory.cpp); results the GPU writes in place are noted with part 0, and without mirrors that is every one of them. invalidate then has no part to wait for and waits for the whole submission, as before this pull request.
• In the runs: the log's line counts 0 readbacks that "waited for their submission's last part" in every run (5 runs, 15 of its lines), with the defaults and with the switches alike.
• The pair, in turn (defaults, switches, defaults, switches): fps 8.5 and 8.6 against 8.4 and 9.4; readbacks that waited, of 3000: 604 and 593 against 236 and 224, each 30.7 and 31.2 ms against 83.7 and 87.2 ms; of them looked at the stopped queues first 51 and 57 against 236 and 224.

run	arm	frames	fps	the log's 120-presents lines: fps mean (lowest .. highest)	draws a frame	GPU time a frame	readbacks waited of 3000, each	looked at the stopped queues first	waited for their last part	readbacks a frame, of them 0.1 ms or longer (mean)	waited in readbacks a frame	frame time	rest of the PC, cores (highest 5 s)
I1/1	defaults	2463	10.5	11.4 (8.5 .. 19.8)	4322	84.6 ms	610, 26.8 ms	18	0	45.8, 2.20 (33.9 ms)	74.7 ms	95.4 ms	11.1 (30.8)
I2/1	defaults	328	8.5	8.6 (8.6 .. 8.6)	4335	97.2 ms	604, 30.7 ms	51	0	45.6, 2.06 (41.4 ms)	85.4 ms	117.0 ms	30.0 (30.5)
I2/2	RX_VK_READBACK_PART=0 RX_VK_READBACK_SETTLE_ALWAYS=1	335	8.4	8.4 (8.4 .. 8.4)	4325	97.7 ms	236, 83.7 ms	236	0	45.4, 1.04 (84.6 ms)	88.5 ms	119.0 ms	29.9 (30.3)
I2/3	defaults	344	8.6	8.6 (8.5 .. 8.6)	4342	97.3 ms	593, 31.2 ms	57	0	45.6, 1.92 (43.6 ms)	83.6 ms	116.0 ms	28.0 (30.3)
I2/4	RX_VK_READBACK_PART=0 RX_VK_READBACK_SETTLE_ALWAYS=1	374	9.4	9.4 (9.3 .. 9.4)	4328	94.8 ms	224, 87.2 ms	224	0	45.4, 1.02 (86.0 ms)	87.7 ms	106.4 ms	13.3 (17.9)

The card is GPU-bound in this scene (85 ms of GPU time in a 95 ms frame in the first, long run), and the game spends 75 ms of each frame waiting in readbacks: here that is the wait for the GPU's frame itself, the whole-submission wait your follow-up would fall back to.
How to read the table: the Intel device has no exclusive hold on this PC, so other work ran beside these runs (the last column; a run counts for me only under 3 cores on average with no 5 s over 6), and the frame rates are rough. I1 is the copy's first start on this device (one run of 440 s, the window 15 to 250 s after the load); I2 is the pair (105 s runs, the window 30 to 70 s after the load).

What does differ between the two arms is how the same wait is cut up, and by the code that can only be the settle switch, since no readback here has a part: with the defaults 604 and 593 of 3000 readbacks wait 30.7 and 31.2 ms each (51 and 57 of them after a look at the stopped queues); with both switches 236 and 224 wait 83.7 and 87.2 ms each, every one after that look. About 2.6 times as many waits, each about 2.8 times shorter, for about the same time a frame (85.4 and 83.6 ms against 88.5 and 87.7 ms): the shape you describe on the RX 580, on a card where the part wait cannot be its cause. Whether it costs frames here I cannot say: other work used 13 to 30 cores beside all four runs of the pair, so none of them counts as a timing.

Not a correct picture on this device, with or without this change: on today's main the UHD 770 draws this area in false colours (dispatches are left out while their compute pipelines compile; reported separately). "The same scene" above means the same draws a frame. No device lost, no i915 line in the kernel log, in any run.

[dankman42]

Windows run for @hckrman101's request: main against the last nightly before this PR, on an RTX 3080.

This complements DFN-master's RTX 3080 run above: there, one build with the switches that restore the old waits, at a 60 Hz cap; here, two separate builds (main and the nightly itself), uncapped, at the same teleport spot. The nightly matches DFN-master's "stock waits" arm (172 of 3000, ~5.6 ms each), and without the cap main's gain shows in full.

Setup. Windows 11, RTX 3080 10 GB (driver 621.14), i9-12900K, 64 GB. Both built from source with llvm-mingw, no DLSS: main 87991bc8 and 55b976c0 (this PR's parent, the last commit whose Windows and Linux nightly runs both succeeded before it). Main also carries about 50 other merges after #2602, so this is main against that nightly, not #2602 alone. Window 1280x720 (mailbox, V-Sync Auto), default options, HOS_FPS_MAX=1000 (uncapped). Scene: the same save, teleported to Hyrule Field (adult, by the Lost Woods bridge, the teleport's own view), standing still; 45 s to settle, then 120 s measured. A warm-up run in each build first, so no pipeline was compiled during the measurement (1481-1495 before and after). Two pairs, interleaved main, nightly, main, nightly.

	nightly 55b976c0	main 87991bc8
readbacks that waited, of 3000	171-182 (median 172)	1542-1573 (median 1550)
wait each	4.92-6.34 ms (medians 5.80, 5.56)	0.03-0.10 ms (medians 0.07, 0.04)
fps, 120 s	59.7 / 60.8	78.4 / 79.3
median frame time	16.39 / 16.32 ms	12.18 / 12.23 ms
p99 frame time	21.46 / 18.98 ms	17.18 / 18.06 ms
frames over 20 ms	264 / 22	30 / 37
draws a frame	1355-1445	1335-1453

So on this card about +31 % fps (median frame 16.35 -> 12.2 ms), in line with the 3090's 63-66 -> 77-79. On main, more readbacks wait (about half, against 6 %), but each one for 0.04-0.10 ms instead of ~5.7 ms. "Looked at the stopped queues first" and "waited for their submission's last part" are 0 in every line of both main runs.

Log lines, the first five of each measured window (55 and 55 lines on main, 42 and 43 on the nightly):

main, pair 1
vk: readbacks of GPU results: 1550 of 3000 waited for their write-back, 0.09 ms each; 0 looked at the stopped queues first, 0 waited for their submission's last part (guest memory read in place after theirs)
vk: readbacks of GPU results: 1548 of 3000 waited for their write-back, 0.09 ms each; 0 looked at the stopped queues first, 0 waited for their submission's last part (guest memory read in place after theirs)
vk: readbacks of GPU results: 1548 of 3000 waited for their write-back, 0.09 ms each; 0 looked at the stopped queues first, 0 waited for their submission's last part (guest memory read in place after theirs)
vk: readbacks of GPU results: 1549 of 3000 waited for their write-back, 0.10 ms each; 0 looked at the stopped queues first, 0 waited for their submission's last part (guest memory read in place after theirs)
vk: readbacks of GPU results: 1551 of 3000 waited for their write-back, 0.10 ms each; 0 looked at the stopped queues first, 0 waited for their submission's last part (guest memory read in place after theirs)
nightly 55b976c0, pair 1
vk: readbacks of GPU results: 172 of 3000 waited for their write-back, 5.87 ms each
vk: readbacks of GPU results: 174 of 3000 waited for their write-back, 5.74 ms each
vk: readbacks of GPU results: 172 of 3000 waited for their write-back, 5.86 ms each
vk: readbacks of GPU results: 172 of 3000 waited for their write-back, 5.88 ms each
vk: readbacks of GPU results: 172 of 3000 waited for their write-back, 5.89 ms each
main, pair 2
vk: readbacks of GPU results: 1549 of 3000 waited for their write-back, 0.10 ms each; 0 looked at the stopped queues first, 0 waited for their submission's last part (guest memory read in place after theirs)
vk: readbacks of GPU results: 1552 of 3000 waited for their write-back, 0.10 ms each; 0 looked at the stopped queues first, 0 waited for their submission's last part (guest memory read in place after theirs)
vk: readbacks of GPU results: 1552 of 3000 waited for their write-back, 0.10 ms each; 0 looked at the stopped queues first, 0 waited for their submission's last part (guest memory read in place after theirs)
vk: readbacks of GPU results: 1549 of 3000 waited for their write-back, 0.09 ms each; 0 looked at the stopped queues first, 0 waited for their submission's last part (guest memory read in place after theirs)
vk: readbacks of GPU results: 1549 of 3000 waited for their write-back, 0.10 ms each; 0 looked at the stopped queues first, 0 waited for their submission's last part (guest memory read in place after theirs)
nightly 55b976c0, pair 2
vk: readbacks of GPU results: 172 of 3000 waited for their write-back, 5.74 ms each
vk: readbacks of GPU results: 173 of 3000 waited for their write-back, 5.70 ms each
vk: readbacks of GPU results: 173 of 3000 waited for their write-back, 5.72 ms each
vk: readbacks of GPU results: 172 of 3000 waited for their write-back, 5.64 ms each
vk: readbacks of GPU results: 172 of 3000 waited for their write-back, 5.77 ms each

Not tested: DLSS, frame generation, Reflex, a frame rate cap, other areas, moving through the field.

[Brett-D]

Windows numbers for this one, after the merge: it recovered the loss from #2125/#2419 on an RTX 5090. (#2700 has RTX 3080 data on the same question.)

Setup. i7-13700K (24 threads), 98 GB, Windows 11, RTX 5090 (driver 621.14), 165 Hz. Every build was made from source with the Hookshot Builder's toolchain and v2.1.0's profile and hot list. Kokiri Forest, the same autosave loaded by a scripted input each run, 2048x1152. Frame generation off and HOS_FPS_MAX=240 ("frame rate cap 240" in the log). Frame times come from HOS_PRESENT_TRACE, 40-70 s after the first present, about 2,330 draws. Builds alternated within each round, with a warm-up run each; 3 runs per build (6 for main).

build	fps	p50 / p95 / p99 ms	readback wait, ms a frame	CPU, cores
main 9776c36 (with this PR)	61.9 (65.5 in a second set)	16.0 / 18.9 / 20.9	2.4-2.9	5.8-6.0
55b976c (main just before this PR)	47.6	20.9 / 23.9 / 27.1	8.2-9.4 (about 8 ms each)	6.2-6.3
49f75dc (before #2125/#2419)	61.9	15.9 / 18.9 / 22.2	0.09-0.10	5.1-5.3
v2.1.0	61.2	16.1 / 19.4 / 22.8	0.09-0.14	5.2-5.3

The readback wait is the sum of "waited x ms each" from the readbacks of GPU results lines, divided by the game's frames in play (frames while the scene draws 1,000 or more draws). Edited: the first version divided by every frame of the run, title screen included, which halved these numbers. CPU is the process's CPU time over the measured window.

With the player's own settings (frame generation Auto) the game's limiter holds every build at 58-60 fps, so this is the uncapped view. No run had a stall, a "timed out" line or a device loss, and screenshots of main and 49f75dc in the same view match.

What's left after this PR

• Readbacks still wait 2.4-2.9 ms a frame, against about 0.1 ms before fix(vk): with per-queue fences, a CPU read-back waits for the GPU work that writes it again #2125. That is about the one real wait a frame for the first part that the description expects.
• Main uses about 0.65 more cores at the same frame rate. A sampled profile (scripts/cpu_report.py) puts the extra in os/sleep-and-timers (44% of a core against 21% on 49f75dc: the job workers yielding in their 20 us sleeps while one waits in a readback) and render/replay-and-submit (28% against 12%).
• HOS_SHORT_SLEEP_TIMER=1 on main, 2 runs each: 0.9 fewer cores, but about 2 fps lower with worse p95/p99 on this 24-thread CPU. So it's not a default here; CPUs with fewer cores may differ.

[DFN-master]

On top of this PR I built the stack I actually play (zstd program cache + BC7 big textures + the 4 GB texture budget) and ran the same save and route:

scene	fps avg (min-max)	readbacks/3000	ms each	last-part fallback	VRAM
Hyrule Field	60.0 (57.2-60.5)	1343-1360	0.01	0	7402/9085 MB
Lake Hylia (adult)	45.94 (41.6-51.1, last 46.4)	1356-1363	0.03	0	6704/8944 MB
Kakariko Village	50-51 (last 51.1)	1227-1261	0.42-0.46	1221-1261	7502/9017 MB

One cross-PR note from the Kakariko run: the BC7 path's GPU unswizzle reads the source through gpu_buffer(), which now notes an in-place read (4ba31740), so the part wait gives way to the last part for essentially every readback there (1221-1261 of 3000, 0.42-0.46 ms). Field and Lake show 0 fallback. It is the guard working as designed; it just means BC7 users lose the part wait in texture-heavy scenes. Town fps is CPU-bound and unchanged.

No device loss on this stack; the earlier crash in my runs was stale pipeline play-cache files from older builds (mkw_vk_pipelines_play_*.bin), not this PR; the zstd program cache was fine.

[mooki000]

Thanks @dankman42 for the uncapped two-build RTX 3080 comparison, @Brett-D for including v2.1.0 and separating the remaining readback/CPU cost on the RTX 5090, and @DFN-master for the RTX 3080 switch comparisons and follow-up with your play stack. The Linux three-column comparison is on #2595, and I'm measuring Brett-D's remaining cost on Linux now. I'm also checking the BC7 interaction you reported, without drawing a conclusion yet.