Repository navigation
perf(vk): the next frame's CPU work overlaps the GPU's end of the current frame (presenter thread, fixed and on by default on Linux) - #2622
Draft
hckrman101 wants to merge 11 commits into
Conversation
…nd no game thread waits behind a present for nothing Hyrule Field lost about 15 fps between v2.0.11-588 (ba86aee) and main (RTX 3090, 2560x1440, uncapped: 79 -> 64-71 fps, CPU 39 -> 64-73 ms a frame). One of the game's job threads was blocked 5.5 ms a frame in a CPU read of GPU results (0.1 ms before), and the game's other job threads polled with 20 us sleeps meanwhile (1.9 cores in sched_yield against 0.3). The read is of the culling results the first part of the scene's submission writes; that submission then stops at an event wait and is finished later, on the presenting thread. Three things held the read up: - since NotRexed#2419 a submission's value maps to its last part's recording, so the read waited for the GPU's whole submission. Each part writes back the pages the GPU wrote in it, at its own end: the page table now keeps the part with the value, and the read waits for the newest part of its pages. A page written in place (no mirror) keeps the wait for its whole submission, whose later parts read and write guest memory itself (issue 997). RX_VK_READBACK_PART=0: the last part, as before. - since NotRexed#2125 the read took sync_order_mu whenever a queue was stopped, behind the presenting thread's whole replay and its present. A queue can be stopped at a met wait with nobody to let it on only after a CPU signal; the read looks at the stopped queues when the CPU has signalled since the last look (2-9 of 3000 readbacks, all of them before). RX_VK_READBACK_SETTLE_ALWAYS=1: whenever anything is stopped, as before. - a wait whose recording was already done, and the host's copy bookkeeping (rx_nvn_replayed, under its own lock), took dev.present_mu to flush held submits, which exist only with batched submits. A present holds that mutex 4 ms a frame. The flushes look at an atomic first; a wait returns at once when the recordings' timeline is already there. Hyrule Field, same session, fix / base / main: 78.6 / 81.3 / 63.1 and 78.7 / 73.9 / 65.9 fps; CPU 46.7 / 38.3 / 71.0 and 45.1 / 40.0 / 69.5 ms a frame. Readbacks that waited: ~1350 of 3000 at 0.14 ms (main 172 at 5.5 ms, base ~1300 at 0.13 ms). Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…s look at the stopped queues covers events in guest memory Review of the readback change: - The part wait asked only whether the read's own pages were written in place. A later part of the same submission may read other guest memory in place: a pool bound with no mirror (refused, wanted, or bound outside a recording's uploads), a mirror's dirty pages that no capture holds, a mirror's first copy. Let go after the first part, the game could write what those parts still read (issue 997's kind of frame, on a card short of VRAM). vk_memory.cpp now notes the newest part with such a read, and a readback whose part is older waits for its submission's last part, as before this branch. The log's readbacks line counts them. - The look at the stopped queues was skipped unless nvnEventSignal had been called. An event neither the stream nor nvnEventSignal ever set has its value read from guest memory, where a plain store meets a wait with no signal. While a queue is stopped at such an event the readback looks every time, as before this branch. - memory_writeback_value's page loop skips chunks never made, as max_in does. - The readbacks line counts a readback as waited whenever the replay was behind, as main does, so the counts compare. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…d submit is found only once it is held, and more in-place reads noted mooki000's two findings on the pull request, and the second review's: - A look at the stopped queues said it was done at the look's end, before the burst it ran in had finished its recording (burst_end: the write-backs noted, the recording submitted). A readback in between found the counts equal, took no look, and read the page table without that recording's pages: the hole NotRexed#2125 closed. The count is now said at the burst's end, after the recording is finished (found by mooki000). - A held submit's recording was in value_rec / part_rec before the submit was in the list of held ones. A thread that found it there and waited for it had nothing to flush, and waited for a signal still held until something else sent the batch. The recording is now put in the maps under dev.present_mu, after the submit is in the list and pend_held is up; without batched submits as before. The early raise of pend_held is gone with its window (found by mooki000; main has the same window with the mutex). - In-place reads noted for the readback's part wait: the pool handed out whole (gpu_buffer: a large texture unswizzled on the GPU, a texel buffer's view), and a binding made outside a recording's uploads now counts for the next part. The counters are read and written in order (no relaxed loads). - Not noted, with the reason at note_in_place_read: memory_rb_refresh's copies of the readback pages into their mirrors (in every part's uploads, they are how a CPU write to such a page reaches the mirror; noted, no read would wait for less than its submission), and the shaders' own pointers into a pool with no mirror. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… swapchain, and the waits for the presenter never give up Fixes from a review of the presenter thread (NotRexed#2325, off by default since NotRexed#2544): - One thread on the swapchain at a time. present_image waits for the presenter to be idle before vkAcquireNextImageKHR: acquire, present and vkWaitForPresentKHR all need the swapchain externally synchronized. The held submits go out first, so the GPU has them meanwhile. The same wait comes before the frame slot's fence is reset, which the presenter may still be waiting on (frame generation's present-ahead path). - The waits for the presenter have no limit. A drain that gave up after 5 s let its callers remake the swapchain, or present out of order, while the presenter still used it. The presenter's own waits are what is bounded: the copy's fence for 5 s, the present wait in 20 ms steps. - After 5 s on the copy's fence the presenter presents all the same, since swap_rendered orders the present on the GPU. Before, it skipped the present and left the image acquired for good. - Only the game's own presents are handed over: not frame generation's paced presents, not the start-up screen. Never while NVIDIA Reflex is on or has numbered a present of this swapchain, since its present ids and the presenter's would not go up together. - A lost surface marks the presenter's queue as not presenting to it, until the new surface is checked. The RX_VK_BENCH replay's device-idle waits take every queue's lock. The "present failed" line names the presenter's queue. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
With the swapchain shared in turns and the waits that never give up (the change before this one), the presenter thread is on by default on Linux. RX_VK_PRESENT_THREAD=0 turns it off. On Windows it stays off unless RX_VK_PRESENT_THREAD=1: nothing was run there. Its queue there would be a compute family's, the same arrangement as RX_VK_PRESENT_QUEUE, which is off on Windows for issue 1161. The switch is read once (presenter_wanted), by both the device's queue choice and the presenter. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
retire_swapchain waits for the presenter thread before it destroys the swapchain and its semaphores, so the rule holds on every path to it, not only on the ones whose callers drain first (a lost surface found by a present made on the game's or the display thread). The presenter never takes dev.present_mu, so the wait is safe with that lock held; the test checks both. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…er thread From the review of the presenter thread's fixes: - A copy whose vkQueueSubmit2 failed signals neither its fence nor swap_rendered. Its present is no longer handed to the presenter, which would wait 5 s for the fence; it stays on the game's thread, as before. - present_thread_on's "said once" flag is atomic: the game's thread and frame generation's display thread both ask. - Comments: why AMD Anti-Lag 2's present stage stays at the hand-off, and the two known limits (a present the driver never returns from, nothing waited for at exit). Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Collaborator
Author
|
First measurements of
Next on this PR:
Testers: Linux with Reflex and frame generation off for now. Compare against main in the same spot, or toggle |
With the present off the game's thread, the wait for the frame before's GPU work moved into the capture: a page the CPU wrote that work still on the GPU writes back (OoT's exposure meter page) waited for its submission's value, which is done with the submission's last part. In Hyrule Field 88-96 of 600 takes waited 0.8 ms (0 of 600 before), on the game's main thread and its presentation thread, both on the frame's path. A page written back from a mirror is in guest memory once the part that copies it back is done, and the page table keeps that part (wb_parts, as the readbacks use it). So a page whose part is done is not late at all, and the others wait for the newest of their parts. A late page written in place keeps the wait for the submission's value, for the whole capture. RX_VK_CAPTURE_PART=0 restores the value always. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Collaborator
Author
|
Testers: please hold off on this build for now. As pushed ( Hyrule Field, 5 runs with it on against 3 off, same session:
Cause: once the game runs a frame ahead, a capture finds pages still to be written back (the exposure meter's) and waits for that submission's value. That value is reached only with the submission's last part. When the submission is stopped at an event, the presentation thread has to finish it, while the main thread is the one waiting, so the wait runs tens of ms, up to the 250 ms limit. Before this PR the present had already waited for the whole frame, so this never happened (0 of 600 takes). The same wait already happens on main with frame generation on, where the present is already off the game's thread, so it is probably part of the stutter frame-generation users see today. Fix, committed locally, not yet run in the game: a capture's late pages wait for the part that writes them back (#2602's page-to-part table), not the submission's last part. Only pages written in place keep the old wait. |
hckrman101
added a commit
that referenced
this pull request
Nov 12, 2026
…led list (#2627) Team updates with several items are hard to scan as one run-on message. This makes the skill's default a one-line summary, then a short labelled list. ## The format ``` 🤖 Perf: main's readback regression is fixed and merged; the overlap draft is on hold while its hitch fix is timed. - Done: #2602 merged: Hyrule Field 63-66 -> 77-79 fps on a 3090. - Working on: #2622: late pages wait only for the part that writes them back. - Wanted: a Windows tester to compare main against the last nightly. ``` - **The first line** says in a few words what changed. It follows the script's 🤖 prefix. - **Each line after it** starts with `- ` and a label, using only the labels that apply, in this order: `Done:`, `Found:`, `Claimed:`, `PR:`, `Working on:`, `Blocked:`, `Wanted:`. - **One sentence a line,** with the numbers that matter and full URLs, at most about five list lines. - **A single short update can stay one line.** ## What changed Only `.claude/skills/team-updates/SKILL.md`, section 3 (*How to post*): the example post and the format rules. The script and its limits (one post an hour, no repeats) are unchanged. ## Testing - `tests/test_team_chat.py` passes. No test reads the skill text. - The format was used for a live post in `#agent-updates`. I have not checked how the list renders in Discord #dev-agents. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
… queue's lock let go NVIDIA's driver on Linux waits inside vkQueuePresentKHR until the frame's copy into the window is done on the GPU. A present made on the replay's queue held dev.present_mu through that wait, and every submit of the replay waited behind it. With frame interpolation the display thread presents four times a game frame: the game's presentation thread was in submit_recording's wait for the lock at 17 % of all instants, 3.8 ms a frame (Hyrule Field, DLSS + interpolation 4x + Reflex, RTX 3090). present_out now waits for the copy's fence first, with the lock let go, and takes it again for the present. This covers frame generation's presents and the game's own when the presenter thread does not take them (NVIDIA Reflex on). The present goes out when the driver would have let it: it waited for the same copy. On by default only with NVIDIA's driver on Linux, where the wait was seen; RX_VK_PRESENT_COPY_WAIT=0 / 1 over it. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…whole submission The 600-take line counts the takes that waited for the submission's value (a late page written in place, or a part not known), apart from those that waited for a part. RX_VK_LOG_CAPTURE_LATE=1 logs each late wait: its time, its pages, the value and the part. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Contributor
|
A second machine for the hitch question on One hold with nothing else on the GPU and no builds running, the same build four times in turn. Memory available 66.9 GB before and 75.7 GB after; pages swapped in 5632 and out 0 during the hold. Runs of 105 s; the numbers are from the window that starts 15 s after the save has loaded (55 s, judged in two halves: a half counts when everything else on the PC used under 3 cores on average over it and no 5 s sample of that was over 6). Frame times are the swapchain's present intervals (
Thread on against off, from the parts that count: frames over 33 ms 0 and 0 against 0 and 0; over 20 ms 1 and 1 against 2 and 2; longest frame 32.7 and 20.5 ms against 20.7 and 22.6 ms; median 10.48 and 10.37 ms against 10.37 and 10.45 ms; fps 93.4 and 94.3 against 94.0 and 93.5; main-thread memory-sync So the hitches do not show on this PC: no frame over 33 ms in any of the four windows (110 s and about 10,300 frames a side). The main thread's memory-sync Quitting, as far as a script reaches: each run ends with the window being closed by SIGTERM at its time limit. 4 of 4 runs (2 with the thread on) ended 2.9 to 3.2 s after the signal with No line in any of the logs that names a swapchain error, a lost surface, an out-of-date or suboptimal swapchain, or a wait that gave up. The build is the exact commit plus one local counting patch (it logs each buffer readback). The Not tested: resizing, alt-tab and the fullscreen toggle (the test window takes no input and its fullscreen requests are refused), V-Sync On, Reflex or frame generation on, Kokiri Forest and the 60 cap on this head (waiting for the new head), other GPUs, X11. |
sheepbun
added a commit
to sheepbun/Bunny-Hood
that referenced
this pull request
Nov 12, 2026
…d here (NotRexed#2622) Five runs on the synced build with NotRexed#2602: Kokiri Forest 51.8 fps (51.2-52.0), 1% low 40.8, frame 19.3 ms (session 5: 36.9 fps). The shader preparation works again with a driver cache of the benchmarks' own. Upstream draft NotRexed#2622's presenter thread changed nothing here: 51.1 and 50.9 fps on, 51.4 and 50.2 off. Area: docs Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tead of the capture waiting for the GPU The one late page left (OoT's exposure meter page) is written back by the last part of the frame before, and take_capture waited for that part on the game's main thread: the next frame's replay did not start before the GPU had finished the frame before, 1.4 ms a frame at best and the GPU's whole tail in the frame's path. The GPU cannot run the next frame's commands before that part anyway: one queue, in order. So the capture is taken as it is, the replay records at once, and what it submits is held in the list batched submits use until the part is done and the page has been copied again into the capture's buffer. Then the held submits go out, in order. - One hold at a time, armed only with nothing held and the part's recording already on the queue, never with batched submits or a parallel replay, never for a texture's storage page. - Every flush looks first. One that must send (a wait for held work, a present, another submit, a slot, the cap) waits for the part, 250 ms at most, as the capture did; a progress poll and each submit_recording send as soon as the part is done, without a wait. - A submit that is not held sends the held ones before it; whether one is held is decided under dev.present_mu, and a recording is put where waiters find it only after that. - If the CPU writes a held page again before its copy, it is said once and no hold is armed again in that run: the captures wait, as before. RX_VK_CAPTURE_HOLD=0 restores the wait at the capture. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
sheepbun
added a commit
to sheepbun/Bunny-Hood
that referenced
this pull request
Nov 12, 2026
…t steps The frame is the Presentation Thread's replay (15.6 ms) and the main thread's ~3 ms tail after it; the game's job threads are off the path, and the parallel replay only moves them onto it. Upstream NotRexed#2622's presenter thread changes nothing on this machine. Area: docs Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
hckrman101
pushed a commit
that referenced
this pull request
Nov 12, 2026
**What this is for:** restore ten game-free checks whose extracted fixtures or expectations no longer match intentional changes on main; [main's CI log](https://github.com/NotRexed/Hookshot/actions/runs/43151254256) shows these failures. **What is measured:** those ten checks on Linux with GCC 16.2.1 and Clang 23.1.3, plus the strict full suite and deliberate regression mutations, as detailed below. **Not measured:** a game run, Windows execution, audio output or GPU behavior; no production file changes in this diff. ## Cause and changes Each row is a separate commit, naming the change that made the fixture stale; the quoted diagnostics come from [main's failed jobs](https://github.com/NotRexed/Hookshot/actions/runs/43149508038). | Introduced by | Checks / CI diagnostic | Repair | |---|---|---| | #1544, after #2042's switch helper | `audio_aux`, `audio_biquad`: `implicit declaration of function ‘env_on’` | Include the real helper before the extracted DSP. | | #2116 | `display_startup_failure`, `wayland_libdecor_hint`: `use of undeclared identifier 'window_title'` | Extract the real title helper along with its callers. | | #1906 | `texel_buffer_ownership`: `use of undeclared identifier 'dead_import_bytes'` | Model the counter and check that it stays charged until completion. | | #2458 | `readback_part`: `extraneous closing brace ('}')` | Select the complete `wait_recorded` definition, not its new forward declaration. | | #2574 | `dlss_checks`: `FAIL under a third wide at line 40`; `framegen_preset` / `rumble_menu`: undeclared `temporal_reset_failure` | Check both sides of the intended two-pixel tolerance; supply a counted reset hook and exercise the real scaling rows. | | #2435 | `mods_size_table`: `undefined reference to hos_xinput_pads` | Supply the fixture's no-controller answer; retain real merge and routing checks. | The diff retains the existing assertions: no check is deleted or skipped, and it does not revert the production behavior named above. ## How it was tested Recorded on 2026-11-11, Linux kernel `7.2.15-1-omarchy`; base `de866017`, head `eeb09c09`, GCC 16.2.1 and Clang 23.1.3 (compiler version output and test records). - Untouched main `a016b7a6`: `455 passed, 15 failed, 20 skipped`; each of these ten failures also reproduced alone with both compilers, matching the linked CI diagnostics. - Branch: all ten repaired checks pass independently with each compiler (`python -I -B tests/run_all.py --strict --jobs 1 <test>` records). - Negative controls: 14 deliberate source regressions were each rejected with both compilers, without a compile failure—28 behavioral rejections (mutation records). These remove Aux sizing, biquad filtering, startup failure handling, the libdecor hint, view ownership release, part-specific waits, preset preservation, temporal reset or mod-table routing; the DLSS controls independently make each axis's tolerance too narrow and too broad. - One full `python -I -B tests/run_all.py --strict` run: **`464 passed, 6 failed, 20 skipped in 141 s`**, 32 workers, load 7.24 → 28.38 (complete suite output). ### Remaining failures and the path to green The complete branch run names these six failures; the first five are outside this fixture-only diff: | Remaining check | Separate fix | |---|---| | `test_env_on.py` | `fix/capture-switch-values`: the three presence-only capture switch reads from #2437. | | `test_linux_docker_packages.py` | `test/docker-ppa-options`: distinguish `--no-update` from its PPA operand. | | `test_mouse_native.py` | `fix/mouse-camera-handover`: discard retained angles while another camera owns the view. | | `test_menu_i18n.py` | `fix/menu-hint-translations`: update three complete hint keys and their translations. | | `test_hdr_encode.py` | Already repaired in #2614; deliberately not duplicated here. | | `test_release_scripts.py` | An intermittent, unchanged release-script concurrency failure: `SHA256SUMS still differs from the files after 5 uploads`. It passed untouched main and current main CI; isolated reruns passed once and failed once. Neither run invokes a C/C++ compiler. | ## Not tested / risks No Windows, game, display, controller, listening or GPU run was performed; the evidence above is game-free only. The fixture extraction still follows source structure, so future helper/signature changes must update its inputs rather than remove its behavioral guards. ## Overlaps The inspected merge hunks overlap #1834's frame-generation fixture helpers and #2622's older readback extraction; neither contains these repairs. #2614 owns the separate HDR initializer check. Other same-file changes merge cleanly in the recorded overlap comparison. Assisted-by: GPT-6 Astra Co-Authored-By: GPT-6 Astra <noreply@openai.com> --------- Co-authored-by: GPT-6 Astra <noreply@openai.com>
sheepbun
added a commit
to sheepbun/Bunny-Hood
that referenced
this pull request
Nov 12, 2026
…d here (NotRexed#2622) Five runs on the synced build with NotRexed#2602: Kokiri Forest 51.8 fps (51.2-52.0), 1% low 40.8, frame 19.3 ms (session 5: 36.9 fps). The shader preparation works again with a driver cache of the benchmarks' own. Upstream draft NotRexed#2622's presenter thread changed nothing here: 51.1 and 50.9 fps on, 51.4 and 50.2 off. Area: docs Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sheepbun
added a commit
to sheepbun/Bunny-Hood
that referenced
this pull request
Nov 12, 2026
…t steps The frame is the Presentation Thread's replay (15.6 ms) and the main thread's ~3 ms tail after it; the game's job threads are off the path, and the parallel replay only moves them onto it. Upstream NotRexed#2622's presenter thread changes nothing on this machine. Area: docs Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Tested in the game on Linux + NVIDIA: the hold works as described and the capture's wait is gone, but the frame rate did not change here, because this machine is GPU-limited in this scene, so the wait moves to the hold's flush. Hardware: RTX 3060 12 GB, NVIDIA 615.71.09, Manjaro (kernel 6.12), Core Ultra 7 265KF, Wayland. Built on main
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Parked. Not for testing or merging. The gain turned out small and step 3 is not safe: see "Why it is parked". The branch stays at
e59c8c80; the work continues as a different design (the game running a frame ahead with readback results delivered in order), not as fixes to this PR.The game's frame is strictly serial today: the next frame's CPU work does not start until the GPU has finished the current one. This PR lets the two overlap.
Cause
In Hyrule Field with #2602 (RTX 3090, Ryzen 9 7900X, Linux, 2560x1440, uncapped) a frame is 12.6 ms:
The 4 ms is
vkQueuePresentKHR. NVIDIA's driver on Wayland waits there on the CPU until the frame's copy into the window is done on the GPU, which is after all of the frame's GPU work. The game's main thread waits for that present to return before it builds the next frame, so the GPU's tail of one frame and the CPU's start of the next never overlap. It is not V-Sync: the swapchain is mailbox, and the wait does not follow the refresh.What is in it
RX_VK_PRESENT_THREAD=1.RX_VK_CAPTURE_HOLD=0turns it off. It has no platform check, so it is on everywhere, and it has only run on Linux with NVIDIA.Measured
Hyrule Field, RTX 3090, Linux/Wayland, 2560x1440, uncapped, Reflex and frame generation off, 15 s runs against #2602's build:
e59c8c80(paired run)e59c8c80(a second run)e59c8c80withRX_VK_PRESENT_THREAD=0Why it is parked
RX_VK_PRESENT_THREAD=0on the same build the median is 11.93 ms against 11.64 ms for perf(vk): a readback waits for the part that writes its range back, and no game thread waits behind a present for nothing #2602, so there is no gain without a present that blocks on the game's thread (NVIDIA's driver on Wayland here). A tester on an RTX 4090 whose present does not block saw 93-94 fps on and off.54d1ab82)e59c8c80)RX_VK_CAPTURE_HOLD=0: presenter + part wait)54d1ab82), unpushed.Never tested
Off switches
RX_VK_PRESENT_THREAD=0: the present back on the game's thread. The log says which:vk: presenter thread: ....RX_VK_CAPTURE_HOLD=0: the capture waits for its late page as before.Tests
tests/test_present_thread.py: the held submits are sent and the presenter idle before the acquire and before the fence's reset; the drain has no limit; no present is skipped; only the game's own present is handed over, never while Reflex numbers the presents; onevkDeviceWaitIdle, under every queue's lock; the lost-surface reset; the default per platform.tests/presenter_queue_test.cpp: the queue choice on NVIDIA's and RADV's layouts.tests/test_fps_stats.pyandtests/test_present_semaphores.pyupdated for the changed functions.