fix(vk): a shader preparation the GPU driver stops answering in goes on or ends, and says why (#2107) by nullifyshot · Pull Request #2195 · NotRexed/Hookshot

nullifyshot · 2026-11-11T07:02:15Z

Since v2.1.0, some builds stop for good while preparing shaders (#2107). #2186 made the slow-but-working case visible: the pipeline steps now log their progress every 10 s. A run where the GPU driver itself stops answering still waits forever, though. This makes that run go on, or end with a reason, so the game's first start doesn't hang in the same place.

This replaces #2123. It's the same work, rebased onto #2186, with hckrman101's review there addressed, and with the status lines #2186 now covers taken out.

What happens

One player's log (an RX 580) shows a likely cause:

vk: the driver faulted compiling a graphics pipeline (exception 0xc0000005): that one is left out
vk: graphics pipeline for 6ee656c480d6beea recovered with optimization disabled

guarded_graphics catches the driver's crash by jumping out of the compile, and that leaves the compile unfinished inside the driver. fix(vk): keep programs the driver faulted on out of the driver (#996) #1823's process dump of issue 996 shows what follows on AMD: later builds of that program wait inside the driver forever.
The step it stops in is predict_run, which runs at build time since fix(vk): build every predicted pipeline before the game (the prediction found nothing on fresh builds; no 90 s budget; one progress bar) #2032. Unlike precompile_run, it had no stall guard.
It also started all its workers' first pipelines at once. precompile_run already avoids that, because of a known NVIDIA fault (621.14, RTX 4070 Ti Super).
Most reports in after compiling it crashes when it is goin to start compiling shaders #2107 were slow rather than stuck, which fix(prepare): the pipeline steps log their progress every 10 s, with the time left ("stuck on preparing shaders") #2186 now shows. This PR changes nothing for those (no fault, still moving) beyond the lines below.

Changes

predict_run (gpu/vk/vk_draw.cpp)

Builds its first pipeline alone, as precompile_run does.
If no pipeline finishes for 30 s (RX_VK_PREDICT_STALL_S, 0 waits for all), it goes on and leaves the stuck workers in the driver. Their state lives on the heap, as in precompile_run. This happens:
- anywhere in the run, only after a driver fault was caught in it (driver_faults());
- otherwise only once every program has been handed out, as precompile_run's guard does. So only the last stragglers are given up, never the bulk. A slow driver that still works keeps its pipelines, at the game's own start too. A real hang before then is the watchdog's to end.
An abandoned worker builds no more of its program's list.
The count moves per pipeline, not per program, so a slow program doesn't look stuck. fix(prepare): the pipeline steps log their progress every 10 s, with the time left ("stuck on preparing shaders") #2186's progress line reads the same count.
predict_run and precompile_run record when they leave workers in the driver (prepare_left_workers()).

A watchdog for the whole --prepare-shaders run (gpu/vk/vk_main.cpp), covering every part on every GPU:

What it's waiting on. Every 30 s that nothing has moved, it logs the part, its count, and what it's waiting on: "the GPU driver or the disk", "the GPU driver, which faulted (above)", or "the GPU driver, which still has pipelines it did not finish". It logs this even while fix(prepare): the pipeline steps log their progress every 10 s, with the time left ("stuck on preparing shaders") #2186's 10-second lines keep the log busy, because those repeat the same count during a hang. A driver fault it caught gets a line of its own as soon as it happens.
The shader programs' count. That step logs no count of its own, so the watchdog writes one after 30 s of a quiet log. It writes nothing of its own while the pipeline steps are moving, since fix(prepare): the pipeline steps log their progress every 10 s, with the time left ("stuck on preparing shaders") #2186's lines cover them.
Stops when nothing moves for 5 minutes (RX_VK_PREPARE_STALL_S, 0 never stops). It then:
- logs STOPPED: ... the GPU driver stopped responding
- leaves hookshot_prepare_skipped, so the game's own starts don't hang in the same place (what wasn't prepared is built when first drawn)
- ends with exit status 4 through TerminateProcess, not exit(), which would unload a driver that can still hold workers
The 5-minute limit is long on purpose: reading the caches and deciding the predictions report no count for a while on a slow disk or CPU.
The pipeline cache's save, the last step:
- If workers were left in the driver, or a fault was caught: the save is stopped the same way, since vkGetPipelineCacheData can wait on them. A run that gets through it ends at once with TerminateProcess (after @@done shaders).
- Otherwise: the save is only reported, and its lines stop after 300 s, so make_game's PREPARE_SILENCE (15 minutes with no line) still ends a save that never finishes.
The part, its count and the time are read under one lock.

Elsewhere

make_game.py maps exit status 4 to "Left to the game's first start: the GPU driver stopped responding". Before, players waited out PREPARE_SILENCE. fix(builder): an update ends a leftover windowless game process, and names a running one #2155 then cleans up the leftover process, if any, before an update.
driver_faults(): the fault guards count what they catch. These are the same four guard blocks fix(vk): keep programs the driver faulted on out of the driver (#996) #1823 changes, so whichever lands second needs a mechanical rebase. After fix(vk): keep programs the driver faulted on out of the driver (#996) #1823, the count would follow its kDriverFault path.

Tested

tests/test_prepare_stall.py (new) compiles predict_run and the watchdog from the source with a stand-in driver. All 14 cases pass:
- predict_run:
  - every pipeline built, with the first one alone
  - the last pipelines never finish: the run goes on
  - a slow driver that still finishes pipelines, with each program longer than the limit: no stop
  - a fault, then nothing finishes early in the run: the run goes on
  - the same with no fault: waited for
- The watchdog:
  - stuck, after a fault, and while the shader programs move
  - stuck and moving while other lines come every 200 ms, as fix(prepare): the pipeline steps log their progress every 10 s, with the time left ("stuck on preparing shaders") #2186's do: it still says what it's waiting on and stops, and says nothing of its own while things move
  - a long clean save: reported, then silent, never stopped
  - a save with workers left in the driver: stopped
  - a run ending through prepare_end
  - in each case, the lines it logs, the exit status and the skip file
- make_game.py: its message for exit status 4
vk_main.cpp and vk_draw.cpp compile with the Windows toolchain's flags (llvm-mingw, -fms-extensions).
Updated fixtures for the new calls: test_prepare_cache_save, test_precompile_compute_guard, test_linux_driver_fault_guard.
python -I -B tests/run_all.py on Windows: 341 passed, 3 failed. test_recorder_sync, test_fpx_sticky and test_precompile_compute_guard fail the same way on an untouched copy of main on this PC (the last needs MSVC's libraries). The Linux-only checks, test_linux_driver_fault_guard among them, are left to CI.
Not tested: a real hang on one of the affected cards. I couldn't reproduce it here (RTX 5080).

🤖 Generated with Claude Code

hckrman101

Thanks for carrying this over onto #2186. I checked it against my review of #2123, and all of it is addressed:

Windows test build: -D_CRT_SECURE_NO_WARNINGS is on the compile line, and both windows-2022 parts are green.
The save hanging behind left-over workers: prepare_left_workers() is set by both predict_run and precompile_run when they go on. In that case, or after a fault, the save is stopped at the limit. A clean save only reports, and its lines stop after 300 s, so PREPARE_SILENCE still applies. Runs that get through end via prepare_end (TerminateProcess / _Exit) after @@done shaders instead of exit(). pipeline_cache_save writes to .tmp and renames, so a stop in the middle can't leave a torn cache file.
The 30 s guard catching slow drivers: it now fires anywhere in the run only after a counted fault, and otherwise only on the tail (next >= keys.size()), as precompile_run does. The slow-program case is in the test.
Nits: the abandoned check is per pipeline now, and PrepareSeen is read under one lock.

Moving the "waiting on ..." line to every 30 s of no movement, instead of 30 s of a quiet log, is the right call now that #2186's 10 s lines repeat the same count during a hang. The watchdog only starts in the prepare_only() branch, so it can't end a real game start. The detached predict workers only touch Pre and their own stack (program_cache_borrow holds no lock while it runs fn), so leaving them behind is safe.

#1823: the overlap is the same four guard blocks plus the two test stubs, and it's mechanical. Your note there (the AMD retry recovers before fault_quarantine().end(...) runs, so the program isn't quarantined) is a good catch for #1823 to look at. Either order works.

CI: the red Linux, SPIR-V and AppImage jobs are the runners failing to resolve the Ubuntu apt mirrors (Temporary failure resolving 'azure.archive.ubuntu.com'), so clang, spirv-val and glslang never got installed. The only failures are the checks that need those tools. test_prepare_stall, test_linux_driver_fault_guard and test_prepare_cache_save pass on Linux even in that run. A re-run should go green.

Two nits inline, neither blocking. Approving.

hckrman101 · 2026-11-11T16:30:31Z

+    //   game's own start too, where no watchdog is; a real hang before then is the --prepare-shaders watchdog's (vk_main.cpp).
+    // Before, it waited for ever. RX_VK_PREDICT_STALL_S=<s>, 0: wait for all
+    static const int stall_s = getenv("RX_VK_PREDICT_STALL_S") ? atoi(getenv("RX_VK_PREDICT_STALL_S")) : 30;
+    const int faults_before = driver_faults();


Nit: faults_before is read after the 64 workers have started, so a fault in their first few milliseconds is counted as "before", and that run falls back to the tail-only rule. Reading it just above the threads.emplace_back loop would close that window.

Good catch, fixed. faults_before is now read just above the threads.emplace_back loop, so a fault in the workers' first milliseconds counts as this run's.

hckrman101 · 2026-11-11T16:30:31Z

+            const int64_t still = (now - at.moved_ms) / 1000;
+            const int faults = driver_faults();
+            const bool left = prepare_left_workers();
+            const bool risky = faults > 0 || left;   // (the driver may be stuck: the save too can wait for ever)


Nit: faults > 0 also counts a fault the AMD retry recovered from ("recovered with optimization disabled", the exact line in the #2107 RX 580 log). On that card every run is "risky": a save that's just slow for 5 minutes gets stopped with exit 4 and leaves the skip file. That's probably rare enough not to matter, and it does keep the TerminateProcess ending, which is what you want. It might still be worth a word in the comment, or a longer cap for the save than for the other parts.

Agreed, I did both. A risky save now gets twice the parts' limit (600 s by default) before it's stopped. The "still writing" and "STOPPED" lines say 600 s. The header comment now says why: a fault that AMD's retry recovered from ("recovered with optimization disabled") still counts, so on that card every run is risky.

test_prepare_stall's save-left case now checks that the save isn't stopped at the parts' 4 s. It's stopped at 8 s instead.

…on or ends, and says why (NotRexed#2107) Builds on Radeon and GeForce RTX 40 cards stopped for good while preparing shaders, the Builder's details showing "vertex format NVN 0xe -> vk 17". The driver had faulted inside a pipeline compile; the fault guard caught it and left the driver holding a lock of its own, so every other worker waited on it for ever. - predict_run (new at build time since NotRexed#2032) builds its first pipeline alone, as precompile_run does for NVIDIA's fault at many first pipelines at once, and goes on without the pipelines the driver never finishes once nothing has finished for 30 s (RX_VK_PREDICT_STALL_S). Its count moves per pipeline. - The --prepare-shaders run has a watchdog for every part: after 30 s with no other line it logs what it is working on, or what it is waiting on (the GPU driver, or the disk) and since when, and any driver fault it caught. When nothing has moved for 5 minutes (RX_VK_PREPARE_STALL_S) it logs why, leaves hookshot_prepare_skipped so the game's starts don't stop in the same place, and ends with exit status 4 (TerminateProcess, not exit(), which would unload the stuck driver). The pipeline cache's save is reported, never stopped. - make_game.py says "the GPU driver stopped responding" for exit status 4. - tests/test_prepare_stall.py compiles predict_run and the watchdog from the source with a stand-in driver: a stuck pipeline, a slow one, a fault, the save. Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…t one vendor or series Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…; the save stopped when workers were left - tests/test_prepare_stall.py: -D_CRT_SECURE_NO_WARNINGS, as the other compiled checks (CI's MSVC-target clang calls getenv and fopen deprecated, an error under -Werror). - predict_run goes on anywhere in the run only once the driver faulted in it; otherwise only once every program was handed out, as precompile_run's guard. A slow driver that still works (up to 30 s on one pipeline, NotRexed#2107), here or at the game's own start, keeps its pipelines; a real hang before then is the watchdog's. - predict_run and precompile_run say when they left workers in the driver (prepare_left_workers). Then, or after a driver fault, the --prepare-shaders watchdog stops the pipeline cache's save as it does the parts, and a run that gets through ends with TerminateProcess (prepare_end), not exit(). Otherwise the save is reported, and its lines end after 300 s so make_game.py's 15 minutes with no line still apply. - An abandoned predict_run worker builds no more of its program's list. - The watchdog reads the part and its count under one lock. Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…n while stuck, no count of its own for the pipeline steps NotRexed#2186 has the pipeline steps log their progress every 10 s, the same count again while they are stuck. So the watchdog: - says what the run waits on every 30 s that nothing has moved, whatever else the log says (it waited for a quiet log before, which NotRexed#2186's lines never leave during the pipeline steps); - no longer logs "still working on" for the pipeline steps (NotRexed#2186's lines say it), only for the shader programs, which log no count of their own, after 30 s of a quiet log. tests/test_prepare_stall.py: a stuck pipeline step with other lines every 200 ms still says what it waits on and stops; a moving one with them says nothing of its own. predict_run's fixture has NotRexed#2186's prepare_progress_log. Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… save stopped at twice the parts' limit - predict_run reads the driver's faults before it starts its workers, so a fault in their first milliseconds counts as this run's - prepare_watchdog stops the pipeline cache's save after twice RX_VK_PREPARE_STALL_S (600 s), not the parts' 300 s. A fault AMD's retry recovered from ("recovered with optimization disabled") still counts, so on such a card every run is risky, and a save that is only slow should not be stopped as early - test_prepare_stall: save-left is stopped at 8 s, not at the parts' 4 s; a stand-in for env_on (NotRexed#2042) in the predict fixture Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

nullifyshot · 2026-11-11T17:13:28Z

Thanks for the review. I rebased onto main and fixed both nits (replies inline).

The rebase over #1823 and #2042:

fix(vk): keep programs the driver faulted on out of the driver (#996) #1823: the four guard blocks keep both its *pl = VK_NULL_HANDLE; and this PR's note_driver_fault();.
fix: switches that behave as players expect, and dead ones removed #2042: predict_run uses env_on("RX_VK_NO_PREDICT"). test_prepare_stall's predict fixture has a stand-in for it.

test_prepare_stall passes, vk_main.cpp and vk_draw.cpp compile cleanly, and the rest of the suite shows no new failures. The earlier CI failures were the runners' DNS (the apt mirrors), as you said, not this change.

predict_run takes main's stall guard (NotRexed#2195: the state on the heap, the first graphics pipeline alone, the run goes on without a stuck driver) with this branch's unfinished-run marker, type counts, per-key duplicate check, usable_cpus pool and sample. A prediction that gave up on stuck pipelines removes its marker and keeps no sample. The program cache's disk_init, now in vk_progcache.cpp, takes main's env_on; the driver-cache check's two switches use env_on too. Tests follow: the stall test's stand-ins for what predict_run now calls, test_program_stamp_knobs reads translator_stamp (disk_stamp before), and the recorder test keeps main's vk_main.cpp reason with vk_progcache.cpp's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… usable CPUs, and keep translated programs across cache-code changes (#2323) ## What changes A fresh prepare (first start after a build or an update, or the Builder's `--prepare-shaders`) is mostly the driver compiling pipelines. This PR makes it build fewer pipelines that no draw can use, run its thread pools at the right size, and stop throwing away every player's translated programs when only the cache code changes. It also fixes a gap where some predicted pipelines were never built, and adds clearer log lines. 1. **Skip predicted pipelines no draw can ask for.** The seed builds every skinned program three times: with float bone inputs, with unsigned integer bone inputs at attribute 6, and at 6 and 7. All three were given the vertex layout that their signature's notes used most. A draw with that layout resolves to exactly one of those three program keys (`graphics_key` puts the input types in the key), so the other two pipelines could never be looked up. Each of them was still a full driver compile with its own modules. - The predictor now reads the types of each module's input and output locations from its SPIR-V. - It drops a candidate layout or target set only when the types definitely differ: an integer where the module has a float, a float where it has an integer, or the other signedness. Scaled types count as integers of their signedness, because `make_profile` turns `support_scaled_attributes` off on every GPU. - A program whose module has notes of its own keeps every class those notes had that fits. A program that falls back on its signature (the notes of other modules with the same input or output locations) now gets the most common class *that fits it*, so each bone-input variant gets a layout of its own types instead of none. - As a self-check, the log counts the notes (real draws) whose own types fail the rule. It should be 0, and it is 0 on all 15,330 notes of the current shipped list. 2. **Build each predicted pipeline once.** `predict_run` skips a prediction whose `pipeline_key` (with the program's mesh or mesh-fallback form) was already built for that program key. Some vertex classes differ only in dynamic state (exact topology, stride), and mesh programs have no vertex layout, so those predictions were the same pipeline. 3. **Coverage fix.** A borrowed program (`program_cache_load(..., keep=true)`) now also includes the records already read into memory (`disk_taken`). Before, the predicted pipelines of programs that the state list had loaded were silently never built: 64,797 predicted, 64,325 made, and nothing in the log said why. Now "made" plus skipped duplicates equals "predicted". 4. **The deciding pass reads the program cache on up to 16 threads**, each with its own file handle, in runs of 64 records (as `program_cache_load_some` does). It was one record at a time on one shared handle under a lock. 5. **Thread pools follow the CPUs the process may use.** The pools counted every CPU of the PC (`hardware_concurrency`), not the ones its affinity mask allows: started on 12 of 24 CPUs, the prediction and the precompile ran 24 threads each. `usable_cpus()` counts the process's affinity mask (`GetProcessAffinityMask` on Windows, `sched_getaffinity` through host/posix on Linux). The prediction, the precompile and the program cache readers use it. The translation, the state list and the stage libraries keep half the PC's CPUs, but never more than `usable_cpus()` (`half_cpus()`); halving the usable count instead made the translation about 30% slower in a test. 6. **The translated programs survive changes to the cache code.** `hookshot_programs.bin` (6.9 GB with the whole game) is only valid for the build's translator stamp, and `nx2/build.py` hashes all of `vk_shader.cpp` for it. The cache's reads, the prediction's reads and the pool sizes lived in that file too, so any change to how the cache is read or scheduled (items 3-5 above, for example) made every player translate the whole game again. - What a record writes byte for byte from `vk_shader.h` (`Program::Answer`, the questions, `Stage::c1`, the head's words), which the translator stamp does not hash, is pinned by `static_assert`s beside `kProgramCacheFormat`: a layout change stops the build until the format is raised. - The cache code moves to `gpu/vk/vk_progcache.cpp` / `.h`, outside the hashed files: `disk_init`, the record and index format, `program_cache_save`, `_read`, `_load`, `_load_some`, `_borrow`, `_keys`, `_each`, `usable_cpus`, `half_cpus`. - The cache, its index, the pipeline notes and `hookshot_prepared.bin` are now stamped with `program_cache_stamp()`: the translator's stamp (`translator_stamp()` in `vk_shader.cpp`, the same inputs `disk_stamp` had) mixed with a new `kProgramCacheFormat`. A change to what a record holds, or to how the file or its index is laid out, must raise `kProgramCacheFormat`. The old cache is then not read, and the next start prepares again, as after a translator change. - `nx2/build.py`'s stamp block and its mirrors in `nx2/bake.py` / `nx2/bake_mimo.py` are unchanged (same file list). The seed reader (`seed_open`) and the seed's translation drivers stay in `vk_shader.cpp`: they parse stage code and decide what is translated. - Keying each program by its own translation (so that only the programs a translator change affects are redone) is not done here. The program keys are already content keys of the stage code and inputs. What is missing is knowing, without translating again, whether a translator change alters a given program's SPIR-V. 7. **An unfinished prediction is finished at the next start.** The predicted pipelines are built after the programs are translated and `hookshot_prepared.bin` says so. A start closed (or ended) during the prediction left the rest unbuilt for good: the next start found the programs prepared and skipped the prediction, so play compiled those pipelines when they were first drawn. `predict_run` now keeps `hookshot_pipelines.bin.predicting` from its start to its end, and the Builder's run, the start-up screen and the console start run the prediction again when it is there. What the driver kept from the first try comes back from its cache quickly. After two starts that ended inside it (a driver that hangs or ends the process on one of these pipelines), the marker is let go and the rest are left to play, as `precompile_run` does with its notes. Closing the window during the prediction does not count toward those two: the marker keeps its count, and the next start goes on. 8. **Check that the driver kept the predicted pipelines.** They are built into the driver's own disk cache only. On a driver without one, or with one too small (NVIDIA on Windows takes its size from the control panel), that work leaves nothing, and nothing said so. `predict_run` keeps a sample of 32 of them (`hookshot_pipelines.bin.predicted`), and the next game start asks the driver for each on an idle thread behind the game, without compiling and as play asks: `vk: the driver's shader cache kept N of M predicted pipelines asked`. It needs `pipelineCreationCacheControl` (as play's own asks do) and does not run in a `--prepare-shaders` run, which exits right after. The driver cache folders' size is logged at the start and after the prediction. `RX_VK_NO_PREDICT_CHECK=1` turns the check off. 9. **Log which program fails and why.** The prepare's summary said "1 failed" and nothing else. A seed stage that does not parse now logs the program, its NVN stage and the reason. On the current seed it is one compute program, which no local play folder has ever dispatched: `vk: program 79116b290d46542f not translated: its NVN stage 5 (code 79116b290d46542f) the branch at instruction 16404 (+0x40140) goes outside the program`. 10. **Progress bar guess.** `prepare_guess` counted one predicted pipeline per seed program (38,809). The prediction now makes 56,588 with the shipped list, so the guess is one and a half per program. Before, the bar ran ahead through the translation and then slowed down once the real count replaced the guess. The log line of the prediction reports the new counts, for example: `56168 made, 420 the same pipeline as another ... left out as no draw could ask for them: 8209 by their vertex input types, 0 by their colour output types (7555 program variants with none left; 0 notes whose own types do not fit; 0 took a less common layout or targets that fit)`. New tests: `tests/test_predict_types.py` (the SPIR-V type reader and fit rule, compiled on hand-made SPIR-V, plus source checks for items 2-3), `tests/test_prepare_cpus.py` (`usable_cpus` / `half_cpus` against stand-in masks, and which pools use them), `tests/test_program_cache_format.py` (runs `nx2/build.py`'s real stamp block: an edit to `vk_progcache.cpp`/`.h` keeps the stamp, an edit to `vk_shader.cpp` changes it; and the stamps' sources), `tests/test_predict_resume.py` and `tests/test_predicted_check.py` (source checks for items 7 and 8). The tests that read the cache code from `vk_shader.cpp` now read `vk_progcache.cpp`/`.h` beside it. **Note for anyone working on the stamp or the bake (#1707):** the translator stamp's file list did not change. `HOOKSHOT_SHADER_STAMP` still covers `vk_shader.cpp`, `vk_bank.cpp`, `vk_state.h`, `vk_api.h` and the recompiler folders. What changed is the runtime stamp that the program cache, the notes and `hookshot_prepared.bin` carry: `program_cache_stamp() = mix(translator_stamp(), kProgramCacheFormat)`, where `translator_stamp()` is the old `disk_stamp()`. The record layout (`put_head`/`get_head`, `put_info`/`get_info`, `program_cache_save`/`_read`) now lives in `vk_progcache.cpp` and is versioned by `kProgramCacheFormat` (1). Because this PR changes `vk_shader.cpp` and the cache stamp, existing installs prepare once more after updating. Later changes to `vk_progcache.cpp` or the scheduling in `vk_draw.cpp` no longer cause that. ## Relation to #2248 #2248 (klimovich008) and item 1 fix the same bug, and they edit the same lines of `predict_pipelines` and `Face` in `vk_predict.cpp`, so only one of them should land. The bug: the prediction's fallback gave a program the vertex layout most common among shaders with the same input *locations*, whatever their types, so some predicted pipelines had vertex attribute formats of another numeric type than the shader's inputs (VUID-08733, which #2248 reports on an RX 9070 XT). - #2248 (head 5d34f63) keys the fallback by a signature of locations *and* numeric types, and gives an input interface it cannot classify (grouped decorations, component-packed inputs, other widths) only its exact module's notes. On their AMD subset that took the predictions from 276 to 4, so most of the prediction goes. - This PR drops the same wrong-type guesses and keeps the rest. On the shipped list it leaves out the 8,209 predicted pipelines whose vertex inputs are of another numeric type than the shader reads (scaled formats are kept) and builds the other 56,588: about 87% of what main predicted. It applies the same check to the colour outputs (0 left out today), and none of the 15,330 real notes fails it. - Items 2-10 do not depend on #2248. If #2248 lands first, this branch keeps its version of those lines on top. #2248's fixture in `tests/test_predict_sees_loaded.py` compiles `vk_predict.cpp` with stand-ins and would need stand-ins for what this branch adds there (`Locations`, `PredictCounts`, `attribute_type_of`, `note_color_types`). ## Measured Setup: RTX 3090, Ryzen 9 7900X, Linux, NVIDIA driver. Each timed run was a cold `hookshot --prepare-shaders` in a reflinked copy of a game folder built from main: no program cache, notes, pipeline cache or computes list, an empty `__GL_SHADER_DISK_CACHE_PATH`, headless. The shipped list was 25,251 pipeline states (15,330 notes). Every timed run held the exclusive timing lock, which gives it 12 logical CPUs (6 cores) and moves builds to the other CPUs. The PC was shared with other work all night and the next day (other test games and prepares running beside some runs, load averages of 20-50), so the seconds are noisy; base and new runs were interleaved. The translation step runs below normal priority, so it suffers most from that load. The main numbers are from the night's runs on main fab8cb3 and were not taken again for the final head. Main's later commits merged here (#2042 switches, #2195 the prediction's stall guard, #1823 the driver-fault quarantine, #2363 the driver cache folder) change neither the work the prepare does nor its thread counts, but the final head's run is a single run on a loaded PC. Whole prepare (wall), seconds: | | runs | |---|---| | main (fab8cb3) | 733.6, 712.0 (earlier, quieter), 728.7 | | items 1-4 only | 666.3, 665.3 (earlier, quieter), 708.0, 708.1 | | items 1-5, 9 and 10 | 699.6, 659.5, 679.4 (each with another test game running) | | final branch before the last main merge (62711af, items 1-10 and the review fixes) | 661.1 | By step (the same runs; main / items 1-4 / items 1-5): | step | main | items 1-4 | items 1-5 | 62711af | |---|---|---|---|---| | translation | 187-205 s | 190-213 s | 223-248 s (contended) | 198.2 s | | prediction | 379-384 s on 24 threads (64,797 pipelines, 64,325 built) | 307-315 s on 24 threads (56,588: 56,168 built + 420 duplicates) | 271-280 s on 12 threads | 279.0 s on 12 threads (56,168 built + 420 duplicates) | | precompile (noted) | 145-147 s | 166-180 s | 159-172 s | 178.2 s | - The prediction is about 105 s shorter than on main (about 28%). About 70 s of that is building fewer pipelines (items 1-2), and about 35 s is running 12 threads instead of 24 on the 12 CPUs the process may use (item 5). - The precompile got 15-30 s longer. The dropped float bone-input variants shared fragment modules with noted programs, and building them used to put those modules in the driver's cache before the precompile. NVIDIA's driver cache appears to work per shader stage, so a pipeline's cost is mostly its not-yet-compiled modules. - Whole prepare: about 675 s on average for this branch (659.5-699.6 s, and 661.1 s for the final code) against about 725 s for main, roughly 7-9%, with a large spread. Predicted pipelines, shipped list (the prediction's log line; the same in every run): | | main | this branch | |---|---|---| | predicted | 64,797 | 56,588 | | built | 64,325 (472 silently never built: item 3) | 56,168 | | left out: same `pipeline_key` as another (item 2) | - | 420 | | left out: a vertex input of another numeric type (item 1) | - | 8,209 | | left out: a colour output of another numeric type (item 1) | - | 0 | | program variants left with no prediction | - | 7,555 | | program variants whose signature fallback took a less common class that fits | - | 0 | | notes (real draws) that fail the type rule | - | 0 of 15,330 | The 8,209 left out are the kind of pipeline VUID-08733 is about: their vertex attribute formats are of another numeric type than the shader reads (scaled formats not counted), so no draw can ask for them (`graphics_key` puts the input types in the program key). This branch builds none of them; main built all of them. The 7,555 program variants left with no prediction are the bone-input variants whose signature's notes all have the other input types: no note anywhere has a layout of their own types (the fallback asks the fit before it picks, and found nothing to take for any of them: the "less common class" count is 0). Main gave each a pipeline that no draw can look up. What that warmed in the driver's cache was the variant's own vertex module, which no noted draw has used; a draw that ever does needs a layout of these types, which no note has to predict from. Thread count for the prediction alone (programs kept, cold driver cache, same 12 CPUs): 6 threads 302.0 s, 12 threads 271.0 s, 24 threads 313-315 s. One thread per logical CPU is best here, so the prediction and precompile use `usable_cpus()`. Grouping the predicted pipelines by fragment module on one worker (to reuse the driver's per-stage cache) took 283.3 s against 271.0 s: no gain, not kept. Program cache stamp (item 6), each a run of `--prepare-shaders` on a folder prepared by this branch: - A build with a change only in `vk_progcache.cpp`: `46676 program keys kept, 6790077 KB`, nothing translated, no prediction; the whole run took 28 s. - A build with a change only in `vk_shader.cpp`: `0 program keys kept`, the translation starts again. - This branch on a folder prepared by the build before it (the old stamp): `0 program keys kept`, a full preparation with the same counts as a cold one (38,809 programs, 46,666 records, 56,588 predicted pipelines). Play after the prepare: a headless start from a Kokiri Forest save, A pressed until play, then the Debug teleports to areas the shipped list covers less (Hyrule Field, Dodongo's Cavern, Jabu-Jabu's Belly, Bottom of the Well), about 40 s in each. This branch (final code, with main merged) ran with its own prepare's driver cache; main (fab8cb3) ran from its prepared folder (the 15,330 shipped notes) with an empty driver cache, so for main only the counts compare, not the milliseconds: | area | pipelines built in play, this branch / main | programs translated in play | waits for shaders or pipelines | |---|---|---|---| | Kokiri Forest | 0 / 0 | 0 / 0 | 0 ms / 0 ms | | Hyrule Field | 0 / 0 | 0 / 0 | 0 ms / 0 ms | | Dodongo's Cavern | 0 / 0 | 0 / 0 | 0 ms / 0 ms | | Jabu-Jabu's Belly | 21 / 21 (374 ms from the driver's cache / 2,563 ms compiled) | 0 / 0 | 0 ms / 0 ms | | Bottom of the Well | 0 / 0 | 0 / 0 | 0 ms / 0 ms | Both had draws left out in one 2-second window, at Jabu-Jabu's load. The 21 pipelines there are neither noted nor predicted by main or this branch. The start's check said `the driver's shader cache kept 32 of 32 predicted pipelines asked (... in 216 ms)`. ## Possible future improvement (not in this PR) About 77% of the translation step is the driver compiling compute pipelines: 9,925 of them, for 4,127 compute programs and their texture-answer variants (each program is tried with every texture answer that changes its SPIR-V, up to 64 variants). Across 39 local play folders, only 606 of the 4,127 compute programs were ever dispatched. Building every variant only for programs on a shipped "dispatched" list, and the base variant alone for the rest, would remove about 4,800 compute compiles: roughly 70 s of a fresh prepare and 0.45 GB of program cache. The risk is a hitch in play if an unlisted program is used with a non-base variant. This PR keeps compiling everything. Also for later, from review: the driver-cache check runs at idle priority but borrows programs through the program cache's lock, which play's threads take too, so a busy PC could hold play up behind it; `driver_cache_log` walks the whole driver cache folder twice per prepare (cheap on NVIDIA's few large files, maybe not on Mesa's); and on Windows PCs with more than 64 logical CPUs `usable_cpus()` reads one processor group only. Also for later: stage libraries (`RX_VK_GPL`) would cover every vertex layout and target set for two compiles per program. The program cache holds 17,247 distinct vertex and 23,155 distinct fragment SPIR-V modules for about 43,000 graphics programs; how many of those share a descriptor layout (which a library also depends on) was not counted here. Translating the seed on all usable logical CPUs instead of half (the start-up screen holds the game anyway) needs a measurement on a whole PC, which the shared timing slot (6 cores) could not give. ## Not tested - AMD, Intel and Windows drivers. The thread counts and the program cache move are plain code, but the timings above are NVIDIA on Linux only. On Windows, `usable_cpus()` reads `GetProcessAffinityMask`, which covers one processor group: on PCs with more than 64 logical CPUs it falls back to all of them. - Item 7 (finishing an unfinished prediction at the next start) has its source tests, but has not yet run in a game: that check is running now, and its result will be added here. Item 8 (the driver-cache check) ran at a start after the prepare: `kept 32 of 32 predicted pipelines asked`. - The VUID-08733 count with the Khronos validation layer: not taken on the NVIDIA PC. A before/after count, with prepare timings, is running on an AMD RX 580 (Mesa RADV) and will be added here. The prediction's own type check (table above) is the same comparison: 8,209 predicted pipelines on main had a vertex input of another numeric type than their shader; this branch builds none of them. - A full playthrough. The play check above covers five areas for about 40 s each, not every area or cutscene. - Hard disks and 16 GB PCs: the multithreaded deciding pass and the program cache readers were only measured with the program cache in memory. - The driver cache folder sizes on Windows (`%LOCALAPPDATA%\NVIDIA\GLCache`, `AMD\VkCache`, `Intel\ShaderCache`) are a best guess of where those drivers keep it. - Not done: the driver's shader cache size on Windows (`driver_cache_room` stays Linux-only: NVIDIA on Windows takes it from the control panel, and nothing here could check an environment variable there) and `__GL_SHADER_DISK_CACHE_SKIP_CLEANUP` (it stops the driver from ever trimming its cache, which would grow by about 2 GB with every update's preparation). --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

nullifyshot mentioned this pull request Nov 11, 2026

fix(vk): a shader preparation the GPU driver stops answering in goes on or ends, and says why (#2107) #2123

Closed

nullifyshot force-pushed the fix/prepare-hang-guard branch from 3dac755 to 0b04734 Compare November 11, 2026 15:51

hckrman101 mentioned this pull request Nov 11, 2026

fix(vk): keep programs the driver faulted on out of the driver (#996) #1823

Merged

hckrman101 approved these changes Nov 11, 2026

View reviewed changes

nullifyshot and others added 5 commits November 11, 2026 12:06

docs(vk): the stuck preparation is reported on cards of all kinds, no…

0679439

…t one vendor or series Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

nullifyshot force-pushed the fix/prepare-hang-guard branch from 0b04734 to 665d7fd Compare November 11, 2026 17:13

hckrman101 merged commit 7759a6f into NotRexed:main Nov 11, 2026
9 of 11 checks passed

github-actions Bot mentioned this pull request Nov 11, 2026

chore(main): release 2.2.0-rc.1 #2071

Open

unixman69 mentioned this pull request Nov 11, 2026

feat(vk): the game's anti-aliasing networks on AMD RDNA3/4 matrix units (16x16x16 cooperative matrices) #1975

Open

Zarigis mentioned this pull request Nov 12, 2026

#2419 (fence wait covers every part of a split submission) costs about 9 fps on NVIDIA at 4K: 60 → 51 in Hyrule Field #2628

Open

Navigation Menu

Repository navigation

fix(vk): a shader preparation the GPU driver stops answering in goes on or ends, and says why (#2107) - #2195

fix(vk): a shader preparation the GPU driver stops answering in goes on or ends, and says why (#2107)#2195
hckrman101 merged 5 commits into
NotRexed:mainfrom
nullifyshot:fix/prepare-hang-guard

nullifyshot commented Nov 11, 2026

Uh oh!

hckrman101 left a comment

Uh oh!

hckrman101 Nov 11, 2026

Uh oh!

nullifyshot Nov 11, 2026

Uh oh!

hckrman101 Nov 11, 2026

Uh oh!

nullifyshot Nov 11, 2026

Uh oh!

nullifyshot commented Nov 11, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

nullifyshot commented Nov 11, 2026

What happens

Changes

Tested

Uh oh!

hckrman101 left a comment

Choose a reason for hiding this comment

Uh oh!

hckrman101 Nov 11, 2026

Choose a reason for hiding this comment

Uh oh!

nullifyshot Nov 11, 2026

Choose a reason for hiding this comment

Uh oh!

hckrman101 Nov 11, 2026

Choose a reason for hiding this comment

Uh oh!

nullifyshot Nov 11, 2026

Choose a reason for hiding this comment

Uh oh!

nullifyshot commented Nov 11, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants