Repository navigation
perf(vk): cheaper replay: uniform-only descriptor rebuilds and flat program picks - #2644
hckrman101 wants to merge 3 commits into
Conversation
dw() is a call and a thread-local read, and the descriptor build asked it about ten times a descriptor: 12 of build_descriptors' 178 samples in a Hyrule Field profile of the replay thread (~0.07 ms a frame). The build takes the reference once, after the two lines the mesh fallback's check reads. Nothing else changes. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
… once A pick that the pair of last picks does not answer compared its program's answers with the bound textures through the program: each answer went to its stage and the stage's descriptor list for where its handle is read, cold cache lines every time (17 of answer_handle's 20 samples in a Hyrule Field profile are those two loads), and a pick with other textures of the same types, a quarter of all picks there, read every handle a second time in still_valid. answer_stage_mask walked the answers once more for each. A pick now keeps where each answer's handle is read (AnswerSource: the NVN stage and the c2 offsets, with Eden's TexturePair shifts) and its mask. pick_handles reads the handles from the state through driver_bank_word, which is what stage_cbuf_word calls for c2; an answer read anywhere else (a handle program, another bank) is still asked of the program. still_valid_handles judges the handles already read. The same words, comparison and verdict as before: the answer domains' check holds pick_handles and still_valid_handles to answer_handles and still_valid on the real descriptor types. Expected: 0.1-0.2 ms a frame of the replay's "programs" time; not measured in the game yet. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…ges alone OoT binds a uniform buffer for nearly every draw, which moves bind_gen, so nine draws in ten built their whole descriptor set again (124 of 1262 kept their neighbour's in Hyrule Field) although a third of the picks there are "the same program, the same textures, a uniform buffer bound since". Such a draw now writes its uniform ranges (uniform_range, as the whole build does) over the last build, which still holds the storage buffers, textures and samplers, and bind_descriptors pushes what differs as before. It is taken only when nothing the rest was made from can have moved since that build (desc_rest_stands): - the same recording and program, and the last build is this program's and was bound; - the program reads its storage buffers through c0 and its texture handles through c2 of its own graphics stages, writes none of them, and has no images, texel buffers, tables or handle programs (uniform_only_of), so the rest is made from those stages' ssbo_gen and tex_gen alone; - those generations, the slots' versions, the samplers' and the destroy generations, the texture epoch, the pools' and mirrors' stamp, and a new generation that moves when a texture becomes GPU-written, the last three as they were before the build began. RX_VK_DESC_PARTIAL=0 turns it off; it is off too with RX_VK_DESC_REUSE=0 or never, in a dump, and with the switches that log, count, widen or copy each binding. The frame log counts the draws that take it. Expected: 0.15-0.25 ms a frame of the replay's "descriptors" time; not measured in the game yet. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
|
Timing result: no measurable gain, so this stays a draft. One alternating pair, standing in Hyrule Field, RTX 3090 + Ryzen 9 7900X, Linux (NVIDIA, Wayland), uncapped, pipeline cache loading off on both sides:
|
…he capture's late-page wait Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Setup: RX 5600 XT 6 GB (RDNA1, no mesh shaders), Adrenalin 26.10.1, Windows 11, Ryzen 5 5600X, 24 GB, 1920x1080, frame generation off. Every run starts from the same copied save and follows a scripted route through the control bridge (Kokiri Forest and Hyrule Field: settle 15 s, then 60 s standing); fps are the medians of the log's Merged on main The mechanism works on this card:
|
|
Tested in the game: the partial rebuild is taken, but no change in frame rate here. Hardware: RTX 3060 12 GB, NVIDIA 615.71.09, Manjaro (kernel 6.12), Core Ultra 7 265KF (8 P-cores + 12 E-cores), Wayland. Built on main
(replay columns: medians of the last 20
|
|
Some data from another machine, in case it helps (RTX 5070 Laptop, Core Ultra 9 275HX, v2.0.14, two 30 s HOS_SAMPLE_ALL profiles in gameplay at 20-40 fps):
I also tried the same idea independently, with a wider eligibility (texture and image descriptors kept across uniform buffer binds, handles re-read and compared when they are not plain c2 words, so handle programs and c3+ handles qualify too). A differential test cut from the real code (38k draws over random streams with UBO rebinds and updates, slot re-registrations, texture destroys, handle programs, captures and recordings) matched a whole build at every draw, and dropping any of its 24 conditions made it fail. But on a frame-shaped synthetic stream it was neutral: keeping saved about 22-34 ns a draw, while recording what is kept cost every whole build 17-38 ns, because the draws that qualify reuse their neighbour's handles and their lookups are already warm. So the gain may be at the low end of your estimate, and an in-game A/B ( Two things from that attempt that could be useful here: a check switch that runs a whole build after each partial one and logs any difference (so a player can verify it in game), and the differential test. Happy to share either. |
…he capture's late-page wait Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After #2602 the frame is serial and the replay of the game's command stream is its longest stretch (~7 ms a frame in Hyrule Field on an RTX 3090 / 7900X), so CPU time taken off the replay thread is frame time. This takes three pieces of per-draw work off it; none changes what is drawn.
Not tested in the game yet (code reading, existing profiles, a build and the checks only): see the last section for what to measure.
Where the time goes (existing profile, Hyrule Field, replay thread)
HOS_SAMPLE_ALLprofile of the #2602 build, 951 samples underreplay(~6.4 ms a frame, one sample ~6.7 us):drawbuild_descriptorsprogram_forbind_descriptors(50 in the driver's push)dispatchStageand descriptor (answer_handle), the stages' descriptor lists inbuild_descriptors.124 draws kept their neighbour's descriptorsof 1262: OoT binds a uniform buffer for nearly every draw, which movesbind_gen, so nine draws in ten build their whole set again. Yet the picks' own log says a third of the graphics picks areby the last picks' textures, a uniform buffer bound since(2.36 of 7.2 million), and another quarterwith other textures.The changes
1. A draw after a uniform buffer's bind builds its uniform ranges alone (
RX_VK_DESC_PARTIAL=0: off)rebuild_uniform_rangeswrites the uniform ranges (throughuniform_range, as the full build does) over that build;bind_descriptorsthen pushes what differs, as it already does. The storage buffers, textures, samplers and the list of sampled textures are the last build's.desc_rest_stands):bind_descriptors(desc_built_prog; a dispatch's build over it is seen throughdw().prog);uniform_only_of, asked once a program). Those words are the stages'ssbo_genandtex_genand nothing else of the state, so no uniform buffer's contents can change them;desc_slots_kept, as for a draw that keeps its whole set), the texture epoch,gpu_bind_stamp()(pools and mirrors: what a buffer binding resolves to) and a new generation that moves when a texture becomes GPU-written (what the build kept for the pass hazards). The last three are read before the build begins, so a build that moves one itself is followed by a whole build.RX_VK_DESC_REUSE=0/never, and with the switches that log, count or widen each binding (RX_VK_DRAW_STATS,RX_VK_WATCH_PROG,RX_VK_SSBO_STRETCH, ...).descriptors(1.35-1.4 ms today). Up to a third of the graphics draws qualify by the picks' log; for those the build skipstexture_view_of(28 samples), the storage buffers'gpu_buffer_bind, the handle reads and most of the function's own 67 samples. Honest low end: an earlier per-stage texture memo measured nothing above the noise, andRX_VK_PROF_FINE's large "textures" time is mostly its own clock reads.a frame: N draws kept their neighbour's descriptors (...), M more built their uniform ranges alone.2. Program picks read their answers' handles from a flat list, once
AnswerSource: the stage and the c2 offsets, Eden's TexturePair shifts) and itsanswer_stage_mask.pick_handlesreads the handles straight from the state; anything that is not a c2 word of the answer's own stage is still asked of the program. A pick "with other textures" checked each handle twice (answer_handles, thenstill_valid);still_valid_handlesjudges the handles already read.driver_bank_word(s, 2, stage, offset)is whatstage_cbuf_wordcalls for c2), the same comparison, the same verdict.tests/test_program_answer_domains.pyholdspick_handlesandstill_valid_handlestoanswer_handlesandstill_validon the real descriptor types for every kind of answer. No switch: identical output.programs(0.7-1.1 ms).answer_handleis 21 samples, 17 of them on two loads of the stage's descriptor;still_valid17; the mask walked the answers a third time.3.
build_descriptorsasks for its writer oncedw()is a call and a thread-local read; the build asked it about ten times a descriptor (12 samples underbuild_descriptors). Expected: ~0.07 ms. Identical output.In all: an estimated 0.3-0.5 ms of the ~6.8 ms replay (5-8 %). Estimates from sample counts, not measurements.
Switches
RX_VK_DESC_PARTIAL=0: every draw builds its whole set, as before (indocs/development.md). Changes 2 and 3 have no switch: the same output by construction.Looked at and left for later
mesh_program_forevery time (two locks, three hash lookups). Only ~25-40 of the field's 1262 draws are mesh draws, so a picks cache for them is worth ~0.01 ms there; written and dropped from this PR.Shader::Info), and a flat table for the picks.memory_end_submission(0.38 ms:cpu_words_merge, the sort inshadows_forget_written): left alone, next to the capture work of perf(vk): the next frame's CPU work overlaps the GPU's end of the current frame (presenter thread, fixed and on by default on Linux) #2622.Tests
tests/test_desc_uniform_only.py: which programs may, what refuses a draw, what is written, and the conditions indraw(), on the realProgram/ Eden descriptor types.tests/test_program_answer_domains.py(the flat sources against the program's own reads),tests/test_pick_tex_gen.py(the mask kept with the pick).hookshot-build --vk-only: builds, ABI check ok.python -I -B tests/run_all.py(Linux): 444-446 pass, 25 skipped; 21 fail, the same 21 on unchanged main (a016b7a6) with the same headers, none in files this PR touches (audio, control, docker packages, menu i18n,vk_memory.cppfixtures, ...).tests/test_release_scripts.pyfailed once in a full run (its two-uploads-at-once case) and passes alone.Not yet tested in the game
Nothing here was run in the game. What testers should measure, Hyrule Field and Kokiri Forest, default options, prepared shaders:
replay(N): submit, draw, descriptors, programs, commandsline ofoot-status --timingruns, orvk: a frame's replay time (ms)in the log):descriptorsshould fall by 0.2-0.3 ms andprogramsby 0.1-0.2 ms;submitby 0.3-0.5 ms. Alternate runs: two runs of one build differ by 10 %.M more built their uniform ranges alonein the log. If M is near 0, change 1 does nothing here and the eligibility (uniform_only_of) is too narrow for this game's materials.RX_VK_DESC_PARTIAL=0, andRX_VK_DESC_REUSE=neveras the reference for descriptors.RX_VK_FRAMELOG/ the flash detector in Kokiri and the field for wrong or missing textures and one-frame flashes; scene changes, the pause menu, a texture streaming burst (riding across the field), DLSS/FSR and frame generation on (their passes push descriptors of their own between draws); an AMD card (RADV) and a card whose sets go over the push limit (allocated sets instead of pushes).RX_VK_PARALLEL, off by default): the new state is per recording thread like the state beside it, but it was not run.