fix(vk): a shader preparation the GPU driver stops answering in goes on or ends, and says why (#2107) by nullifyshot · Pull Request #2123 · NotRexed/Hookshot

nullifyshot · 2026-11-11T04:02:49Z

Since v2.1.0, builds can stop for good while preparing shaders (#2107). The reports come from players on a range of graphics cards, AMD and NVIDIA alike, so this isn't one vendor's or one series' bug. The Builder's details freeze on vertex format NVN 0xe -> vk 17 and nothing says why. This makes the preparation go on, or end with a reason, and keeps the log saying what it's doing.

What happens

One player's log (an RX 580) shows a likely cause:

vk: the driver faulted compiling a graphics pipeline (exception 0xc0000005): that one is left out
vk: graphics pipeline for 6ee656c480d6beea recovered with optimization disabled

guarded_graphics catches the driver's crash by jumping out of the compile, and that leaves the compile unfinished inside the driver. fix(vk): keep programs the driver faulted on out of the driver (#996) #1823's process dump of issue 996 shows what follows on AMD: later builds of that program wait inside the driver for ever.
The step it stops in is predict_run, which runs at build time since fix(vk): build every predicted pipeline before the game (the prediction found nothing on fresh builds; no 90 s budget; one progress bar) #2032 (before that, new builds predicted 0 pipelines). Unlike precompile_run, it had no stall guard.
It also started all its workers' first pipelines at once. precompile_run already avoids that, because of a known NVIDIA fault (621.14, RTX 4070 Ti Super).
Other reports in after compiling it crashes when it is goin to start compiling shaders #2107 read as very slow rather than stuck. This PR changes nothing for those (no fault, still moving) except that the log now says what's going on.

Changes

predict_run (gpu/vk/vk_draw.cpp)

Builds its first pipeline alone, as precompile_run does.
If no pipeline finishes for 30 s (RX_VK_PREDICT_STALL_S, 0 waits for all), it goes on and leaves the stuck workers in the driver, with their state on the heap as in precompile_run. This happens:
- anywhere in the run, only after a driver fault was caught in it (driver_faults());
- otherwise only once every program was handed out, as precompile_run's guard does, so only the last stragglers are given up, never the bulk. A slow driver that still works keeps its pipelines, at the game's own start too. A real hang before then is the watchdog's to end.
An abandoned worker builds no more of its program's list.
The count now moves per pipeline, not per program, so a slow program doesn't look stuck.
predict_run and precompile_run record when they leave workers in the driver (prepare_left_workers()).

A watchdog for the whole --prepare-shaders run (gpu/vk/vk_main.cpp), covering every part on every GPU:

Live status in the log. After 30 s with no other line, it writes what it's working on with its count, or, if the count hasn't moved, what it's waiting on and for how long. That's "the GPU driver or the disk", "the GPU driver, which faulted (above)", or "the GPU driver, which still has pipelines it did not finish". A driver fault it caught gets a line of its own as soon as it happens.
Stops when nothing moves for 5 minutes (RX_VK_PREPARE_STALL_S, 0 never stops). It then:
- logs STOPPED: ... the GPU driver stopped responding
- leaves hookshot_prepare_skipped, so the game's own starts don't hang in the same place (what wasn't prepared is built when first drawn)
- ends with exit status 4 through TerminateProcess, not exit(), which would unload the stuck driver and could wait on it
Going on to the next part wouldn't help once the driver is stuck, because every part needs it. The 5-minute limit is long on purpose: reading the caches and deciding the predictions report no count for a while on a slow disk or CPU.
The pipeline cache's save, the last step:
- If workers were left in the driver, or a fault was caught: the save is stopped the same way, since vkGetPipelineCacheData can wait on them. A run that gets through ends at once with TerminateProcess (after @@done shaders), not exit().
- Otherwise: the save is only reported, and its lines stop after 300 s so that make_game's PREPARE_SILENCE (15 minutes with no line) still ends a save that never finishes.
The part, its count and the time are read under one lock.

Elsewhere

make_game.py maps exit status 4 to "Left to the game's first start: the GPU driver stopped responding". Before, players waited for PREPARE_SILENCE, 15 minutes, and the game's first start could then stop in the same place.
driver_faults(): the fault guards count what they catch. These are the same four guard blocks fix(vk): keep programs the driver faulted on out of the driver (#996) #1823 changes, so whichever lands second needs a mechanical rebase. After fix(vk): keep programs the driver faulted on out of the driver (#996) #1823, the count would follow its kDriverFault path.

Tested

tests/test_prepare_stall.py (new) compiles predict_run and the watchdog from the source with a stand-in driver. All 12 cases pass:
- predict_run:
  - every pipeline built, with the first one alone
  - the last pipelines never finish: the run goes on
  - a slow driver that still finishes pipelines, each program past the limit: no stop
  - a fault, then nothing finishes early in the run: the run goes on
  - the same with no fault: waited for
- The watchdog:
  - stuck, after a fault, and while the count moves
  - a long clean save: reported, then silent, never stopped
  - a save with workers left in the driver: stopped
  - a run ending through prepare_end
  - in each case, the lines it logs, the exit status and the skip file
- make_game.py: its message for exit status 4
vk_main.cpp and vk_draw.cpp compile with the Windows toolchain's flags (llvm-mingw, -fms-extensions), and the new functions link between them.
Updated fixtures for the new calls: test_prepare_cache_save, test_precompile_compute_guard, test_linux_driver_fault_guard.
python -I -B tests/run_all.py on Windows: 341 passed, 3 failed. test_recorder_sync, test_fpx_sticky and test_precompile_compute_guard fail the same way on an untouched copy of main on this PC (the last needs MSVC's libraries). The Linux-only checks, test_linux_driver_fault_guard among them, are left to CI.
Not tested: a real build on one of the affected cards. I couldn't reproduce the hang here (RTX 5080).

🤖 Generated with Claude Code

…on or ends, and says why (NotRexed#2107) Builds on Radeon and GeForce RTX 40 cards stopped for good while preparing shaders, the Builder's details showing "vertex format NVN 0xe -> vk 17". The driver had faulted inside a pipeline compile; the fault guard caught it and left the driver holding a lock of its own, so every other worker waited on it for ever. - predict_run (new at build time since NotRexed#2032) builds its first pipeline alone, as precompile_run does for NVIDIA's fault at many first pipelines at once, and goes on without the pipelines the driver never finishes once nothing has finished for 30 s (RX_VK_PREDICT_STALL_S). Its count moves per pipeline. - The --prepare-shaders run has a watchdog for every part: after 30 s with no other line it logs what it is working on, or what it is waiting on (the GPU driver, or the disk) and since when, and any driver fault it caught. When nothing has moved for 5 minutes (RX_VK_PREPARE_STALL_S) it logs why, leaves hookshot_prepare_skipped so the game's starts don't stop in the same place, and ends with exit status 4 (TerminateProcess, not exit(), which would unload the stuck driver). The pipeline cache's save is reported, never stopped. - make_game.py says "the GPU driver stopped responding" for exit status 4. - tests/test_prepare_stall.py compiles predict_run and the watchdog from the source with a stand-in driver: a stuck pipeline, a slow one, a fault, the save. Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…t one vendor or series Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

hckrman101

Thanks for picking this up. The status lines alone are a real improvement for #2107: players can't tell "slow" from "stuck" right now. Moving the count per pipeline and building the first pipeline alone in predict_run both look right to me. Keeping the stuck workers' state on the heap and detaching them follows precompile_run's pattern and is safe as written: they only touch Pre, never the stack. Exit status 4 and its make_game message line up, and RUN_ALL_ARGS is declared.

Some things I'd like sorted before this goes in:

CI is red on Windows because of the new test. test_prepare_stall.py fails in part 2/2. clang++ targeting MSVC turns getenv (and fopen in the watchdog piece too) into -Wdeprecated-declarations errors under -Werror. Add -D_CRT_SECURE_NO_WARNINGS on Windows, as the other compiled tests do. This is the PR's own test, not one of main's known failures.
The save step can now hang forever (inline). Part 3 is never stopped, because "every pipeline is made". That's no longer true once predict_run (or precompile_run) has moved on and left a worker inside vkCreateGraphicsPipelines. If that worker holds the driver's lock or the cache's lock, vkGetPipelineCacheData in pipeline_cache_save(true) waits on it. Worse, the watchdog then logs "still writing ... this is the disk" every 30 s. That output keeps make_game's PREPARE_SILENCE from ever firing, so the 15-minute backstop this PR replaces is gone exactly in the #2107 case. The same goes for the plain return 0 → exit() after @@done, with a detached thread still in the driver (the unload hazard the description itself cites for exit()). Some ideas: let part 3 stop too when a part moved on (or a fault was counted), or after a longer cap. Stop the status lines after N minutes so the silence timeout still applies. And end the run with _Exit/TerminateProcess after the save when workers were left behind.
30 s for predict_run may catch slow drivers that still work (inline). precompile_run's guard only fires once everything has been handed out (next >= total), so it only ever drops the last stragglers. This one fires whenever nothing finishes for 30 s, wherever the run is, and gives up on everything left (tens of thousands of pipelines, which are then built in play). It also runs at the game's own start (the start screen and console paths), where there's no watchdog. In #2107, oniryuuko describes a run that was "working, just very very slow", with the progress sometimes stopping "for up to 30 seconds as it hangs on a particular pipeline". That's right at this limit, and right after the first pipeline, which the others now wait for. I'd only fire early once driver_faults() has counted something (the case the log shows). Otherwise use the tail-only rule, or a much longer limit, and leave the real hang to the watchdog in --prepare-shaders.
Overlap with #1823 (approved, open). Both change the __except / sigsetjmp blocks of guarded_graphics/guarded_compute, test_linux_driver_fault_guard.py and test_precompile_compute_guard.py, so whichever lands second needs a rebase. Two semantic points as well. #1823 makes the graphics guard return -77 like compute and adds *pl = VK_NULL_HANDLE, and its gate serializes a program's builds on AMD so that "a build behind a faulted one" doesn't hang (issue 996). That sounds a lot like the fault-then-stuck theory here, and the #2107 reporter is on an RX 580. Please say how the two fit together, e.g. whether the fault counter should also count #1823's kDriverFault path and whether its quarantine would already have kept this run from hanging.

Smaller:

Nit: a worker that's been abandoned keeps building the rest of its current program's list. abandoned is only checked at the top of the loop, so those pipelines can run alongside precompile_run and the save. A check inside the per-pipeline loop would stop it.
Nit: prepare_moved updates part/done/total as three separate atomics, so the watchdog can read a mixed snapshot. That's harmless for a log line, but worth knowing if it ever decides anything.
The description's root cause comes from one log, and nobody has reproduced it on an affected card yet ("Not tested"). Other reports in #2107 read like slowness rather than a hang. That's fine for a diagnostic change, but the 30 s default (point 3) makes it change behaviour too.

The watchdog/diagnostic part is close. Points 1–3 are what I'd want fixed first.

hckrman101 · 2026-11-11T04:13:14Z

        Sleep(100);
+        const auto now = std::chrono::steady_clock::now();
+        if (done.load() != last_done) last_done = done.load(), last_change = now;
+        else if (stall_s > 0 && now - last_change > std::chrono::seconds(stall_s)) {


Unlike precompile_run's guard (next.load() >= total &&), this fires anywhere in the run and gives up on all that's left, here and at the game's own start (no watchdog there). With the first pipeline built alone, a single 30 s compile (which #2107 reports describe on slow-but-working runs) is enough. I'd fire early only once driver_faults() > 0, and otherwise keep the tail-only rule or a much longer limit.

Done in c69b5a6. It goes on from anywhere in the run only if driver_faults() rose during this run. Otherwise it uses the tail-only rule, as precompile_run does (next >= keys.size()). test_prepare_stall.py checks that a hang early in the run with no fault is waited for.

hckrman101 · 2026-11-11T04:13:14Z

+        prepare_moved(2, 0, 0);
        bar.reach(2);
        precompile_run(progress);
+        prepare_moved(3, 0, 0);   // (the pipeline cache written out: the watchdog reports it, never stops it)


"Every pipeline is made" isn't true when predict/precompile moved on and left a worker inside vkCreate*Pipelines. vkGetPipelineCacheData on the same cache can then wait on it forever. Because part 3 keeps logging "still writing", make_game's 15 min PREPARE_SILENCE never fires either. I think the save needs a cap, or a stop, when workers were left behind.

Agreed. predict_run and precompile_run now call prepare_left_workers_note() when they go on. When that's set, or a fault was counted, the watchdog stops the save the same way it stops the other parts.

hckrman101 · 2026-11-11T04:13:14Z

+                std::_Exit(4);
+            }
+            if (now - last_log_ms.load(std::memory_order_relaxed) < quiet_s * 1000) continue;
+            if (part >= 3)


These lines reset make_game's silence timer, so if the save (or the exit() after @@done) hangs, the build waits forever. Maybe stop emitting after a few minutes in part 3, or end the run here too when driver_faults() or a moved-on part says the driver may be stuck.

Both are handled in c69b5a6. In a clean run, the save's status lines stop after 300 s, so PREPARE_SILENCE applies again. In a risky run (workers left, or a fault), the save is stopped at 300 s, and a run that gets through ends with TerminateProcess (prepare_end), not exit().

hckrman101 · 2026-11-11T04:13:14Z

+    src = Path(tmp) / (name + ".cpp")
+    exe = Path(tmp) / (name + (".exe" if os.name == "nt" else ""))
+    src.write_text(code, encoding="utf-8")
+    cmd = [cxx, "-std=c++20", "-O1", "-Wall", "-Wextra", "-Werror", "-Wno-unused-function", str(src), "-o", str(exe)]


This is what fails on windows-2022 (part 2/2): getenv is deprecated under MSVC's CRT, and with -Werror that's an error (fopen in the watchdog piece would hit the same). Add -D_CRT_SECURE_NO_WARNINGS when os.name == "nt".

Fixed: -D_CRT_SECURE_NO_WARNINGS is added to the compile command, as in the other compiled checks.

…; the save stopped when workers were left - tests/test_prepare_stall.py: -D_CRT_SECURE_NO_WARNINGS, as the other compiled checks (CI's MSVC-target clang calls getenv and fopen deprecated, an error under -Werror). - predict_run goes on anywhere in the run only once the driver faulted in it; otherwise only once every program was handed out, as precompile_run's guard. A slow driver that still works (up to 30 s on one pipeline, NotRexed#2107), here or at the game's own start, keeps its pipelines; a real hang before then is the watchdog's. - predict_run and precompile_run say when they left workers in the driver (prepare_left_workers). Then, or after a driver fault, the --prepare-shaders watchdog stops the pipeline cache's save as it does the parts, and a run that gets through ends with TerminateProcess (prepare_end), not exit(). Otherwise the save is reported, and its lines end after 300 s so make_game.py's 15 minutes with no line still apply. - An abandoned predict_run worker builds no more of its program's list. - The watchdog reads the part and its count under one lock. Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

nullifyshot · 2026-11-11T04:32:26Z

Thanks for the careful review, hckrman101. All of points 1–3 and both nits are in c69b5a6.

CI: the test compiles with -D_CRT_SECURE_NO_WARNINGS, as the other compiled checks do.
The save: predict_run and precompile_run now record when they go on and leave workers in the driver (prepare_left_workers()).
- If workers were left, or a driver fault was counted: the watchdog stops the save as it does the other parts (exit status 4, with the skip file). A run that gets through the save ends with TerminateProcess (prepare_end), not with return → exit(). It prints @@done shaders first, as the host would.
- A clean run: the save is only reported. Its status lines stop after 300 s, so PREPARE_SILENCE still ends a save that never finishes.
The 30 s in predict_run: it now goes on from anywhere in the run only if driver_faults() rose during that run. Otherwise it uses precompile_run's tail-only rule (next >= keys.size()), so a slow driver that still works keeps its pipelines, at the game's own start too. A real hang before the tail is left to the watchdog. The test has a case for each: a fault and then a hang early in the run goes on, the same hang with no fault is waited for, and a slow program (1.5 s, past a 1 s limit) isn't given up.

Nits:

An abandoned worker checks abandoned before each pipeline of its list.
The watchdog now reads part, count and time under one lock (PrepareSeen).

fix(vk): keep programs the driver faulted on out of the driver (#996) #1823: I read its dump, and it describes the same mechanism. After a caught fault, AMD's driver leaves that program's compile unfinished, and later builds of the program wait inside the driver for ever. The RX 580 log in after compiling it crashes when it is goin to start compiling shaders #2107 fits that, so I've reworded the code comment to point at it instead of a general "lock". As I see it, the two are complementary:
- fix(vk): keep programs the driver faulted on out of the driver (#996) #1823 prevents the known cause on AMD: one build of a program at a time, and a faulted program is never asked of the driver again.
- This PR is the backstop and the diagnostics for what that doesn't cover: other drivers (where fix(vk): keep programs the driver faulted on out of the driver (#996) #1823's gate is off), other causes, and saying what the run is doing either way.
- Would fix(vk): keep programs the driver faulted on out of the driver (#996) #1823 alone have prevented this run's hang? I'm not sure it would. In the after compiling it crashes when it is goin to start compiling shaders #2107 log the faulted program then logs recovered with optimization disabled. As I read fix(vk): keep programs the driver faulted on out of the driver (#996) #1823's build_graphics, that AMD retry runs before fault_quarantine().end(p.key, r == kDriverFault), so r is VK_SUCCESS by then and the program isn't quarantined. Later builds of it would go back to the driver. It may be worth quarantining on the first fault even when the retry recovers, but that's fix(vk): keep programs the driver faulted on out of the driver (#996) #1823's call.
- The fault counter: yes, it should follow fix(vk): keep programs the driver faulted on out of the driver (#996) #1823's kDriverFault path. My note_driver_fault() calls sit in the same four guard blocks, next to fix(vk): keep programs the driver faulted on out of the driver (#996) #1823's *pl = VK_NULL_HANDLE and return (VkResult)-77, and so do the two test stubs. The conflict is mechanical. Since fix(vk): keep programs the driver faulted on out of the driver (#996) #1823 is approved, I'd let it land first and rebase this onto it.

On the evidence: you're right that the root cause comes from one log, and that other reports read as slowness. With point 3 changed, the only behaviour change without a counted fault is the watchdog's 5-minute stop in --prepare-shaders. Slow-but-working runs now get status lines and otherwise run as before.

Locally: test_prepare_stall passes all 12 cases. Full suite on Windows: 341 passed, 3 failed. The 3 (test_recorder_sync, test_fpx_sticky, test_precompile_compute_guard) fail the same way on an untouched copy of main on this PC.

nullifyshot · 2026-11-11T07:02:29Z

Closing in favour of #2195.

#2186 landed on main in the meantime. It has the preparation's pipeline steps log their progress every 10 s, which covers the "is it slow or stuck?" half of this PR and conflicted with it. #2195 is this work rebased onto #2186, with two changes on top:

Dropped: this PR's "still working on …" lines for the pipeline steps. fix(prepare): the pipeline steps log their progress every 10 s, with the time left ("stuck on preparing shaders") #2186's lines report that better, with time left.
Changed: the "waiting on the GPU driver …" line now comes every 30 s that nothing has moved. It used to wait for 30 s of a quiet log, and fix(prepare): the pipeline steps log their progress every 10 s, with the time left ("stuck on preparing shaders") #2186's lines, which repeat the same count during a hang, would never have let it appear.

Everything else carries over unchanged, including the changes from hckrman101's review here:

predict_run's first pipeline built alone
the stall guard: anywhere in the run only after a driver fault, otherwise the tail-only rule
the 5-minute stop, with its reason, skip file and exit status 4
the cache-save handling, and ending through TerminateProcess when workers were left
the driver-fault lines

A new PR keeps this one's review and history readable as they were, instead of force-pushing over them.

@done

…on or ends, and says why (#2107) (#2195) Since v2.1.0, some builds stop for good while preparing shaders (#2107). #2186 made the slow-but-working case visible: the pipeline steps now log their progress every 10 s. A run where the GPU driver itself stops answering still waits forever, though. This makes that run go on, or end with a reason, so the game's first start doesn't hang in the same place. This replaces #2123. It's the same work, rebased onto #2186, with hckrman101's review there addressed, and with the status lines #2186 now covers taken out. ### What happens One player's log (an RX 580) shows a likely cause: ``` vk: the driver faulted compiling a graphics pipeline (exception 0xc0000005): that one is left out vk: graphics pipeline for 6ee656c480d6beea recovered with optimization disabled ``` - `guarded_graphics` catches the driver's crash by jumping out of the compile, and that leaves the compile unfinished inside the driver. #1823's process dump of issue 996 shows what follows on AMD: later builds of that program wait inside the driver forever. - The step it stops in is `predict_run`, which runs at build time since #2032. Unlike `precompile_run`, it had no stall guard. - It also started all its workers' first pipelines at once. `precompile_run` already avoids that, because of a known NVIDIA fault (621.14, RTX 4070 Ti Super). - Most reports in #2107 were slow rather than stuck, which #2186 now shows. This PR changes nothing for those (no fault, still moving) beyond the lines below. ### Changes **`predict_run`** (`gpu/vk/vk_draw.cpp`) - Builds its first pipeline alone, as `precompile_run` does. - If no pipeline finishes for 30 s (`RX_VK_PREDICT_STALL_S`, 0 waits for all), it goes on and leaves the stuck workers in the driver. Their state lives on the heap, as in `precompile_run`. This happens: - **anywhere in the run**, only after a driver fault was caught in it (`driver_faults()`); - **otherwise only once every program has been handed out**, as `precompile_run`'s guard does. So only the last stragglers are given up, never the bulk. A slow driver that still works keeps its pipelines, at the game's own start too. A real hang before then is the watchdog's to end. - An abandoned worker builds no more of its program's list. - The count moves per pipeline, not per program, so a slow program doesn't look stuck. #2186's progress line reads the same count. - `predict_run` and `precompile_run` record when they leave workers in the driver (`prepare_left_workers()`). **A watchdog for the whole `--prepare-shaders` run** (`gpu/vk/vk_main.cpp`), covering every part on every GPU: - **What it's waiting on.** Every 30 s that nothing has moved, it logs the part, its count, and what it's waiting on: "the GPU driver or the disk", "the GPU driver, which faulted (above)", or "the GPU driver, which still has pipelines it did not finish". It logs this even while #2186's 10-second lines keep the log busy, because those repeat the same count during a hang. A driver fault it caught gets a line of its own as soon as it happens. - **The shader programs' count.** That step logs no count of its own, so the watchdog writes one after 30 s of a quiet log. It writes nothing of its own while the pipeline steps are moving, since #2186's lines cover them. - **Stops when nothing moves for 5 minutes** (`RX_VK_PREPARE_STALL_S`, 0 never stops). It then: - logs `STOPPED: ... the GPU driver stopped responding` - leaves `hookshot_prepare_skipped`, so the game's own starts don't hang in the same place (what wasn't prepared is built when first drawn) - ends with exit status 4 through `TerminateProcess`, not `exit()`, which would unload a driver that can still hold workers The 5-minute limit is long on purpose: reading the caches and deciding the predictions report no count for a while on a slow disk or CPU. - **The pipeline cache's save, the last step:** - **If workers were left in the driver, or a fault was caught:** the save is stopped the same way, since `vkGetPipelineCacheData` can wait on them. A run that gets through it ends at once with `TerminateProcess` (after `@@done shaders`). - **Otherwise:** the save is only reported, and its lines stop after 300 s, so make_game's `PREPARE_SILENCE` (15 minutes with no line) still ends a save that never finishes. - The part, its count and the time are read under one lock. **Elsewhere** - `make_game.py` maps exit status 4 to "Left to the game's first start: the GPU driver stopped responding". Before, players waited out `PREPARE_SILENCE`. #2155 then cleans up the leftover process, if any, before an update. - `driver_faults()`: the fault guards count what they catch. These are the same four guard blocks #1823 changes, so whichever lands second needs a mechanical rebase. After #1823, the count would follow its `kDriverFault` path. ### Tested - `tests/test_prepare_stall.py` (new) compiles `predict_run` and the watchdog from the source with a stand-in driver. All 14 cases pass: - **`predict_run`:** - every pipeline built, with the first one alone - the last pipelines never finish: the run goes on - a slow driver that still finishes pipelines, with each program longer than the limit: no stop - a fault, then nothing finishes early in the run: the run goes on - the same with no fault: waited for - **The watchdog:** - stuck, after a fault, and while the shader programs move - stuck and moving while other lines come every 200 ms, as #2186's do: it still says what it's waiting on and stops, and says nothing of its own while things move - a long clean save: reported, then silent, never stopped - a save with workers left in the driver: stopped - a run ending through `prepare_end` - in each case, the lines it logs, the exit status and the skip file - **`make_game.py`:** its message for exit status 4 - `vk_main.cpp` and `vk_draw.cpp` compile with the Windows toolchain's flags (llvm-mingw, `-fms-extensions`). - Updated fixtures for the new calls: `test_prepare_cache_save`, `test_precompile_compute_guard`, `test_linux_driver_fault_guard`. - `python -I -B tests/run_all.py` on Windows: 341 passed, 3 failed. `test_recorder_sync`, `test_fpx_sticky` and `test_precompile_compute_guard` fail the same way on an untouched copy of main on this PC (the last needs MSVC's libraries). The Linux-only checks, `test_linux_driver_fault_guard` among them, are left to CI. - **Not tested:** a real hang on one of the affected cards. I couldn't reproduce it here (RTX 5080). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: breakthetargets <81792782+breakthetargets@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

nullifyshot and others added 2 commits November 10, 2026 23:02

docs(vk): the stuck preparation is reported on cards of all kinds, no…

ac32233

…t one vendor or series Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

hckrman101 reviewed Nov 11, 2026

View reviewed changes

nullifyshot mentioned this pull request Nov 11, 2026

fix(vk): a shader preparation the GPU driver stops answering in goes on or ends, and says why (#2107) #2195

Merged

nullifyshot closed this Nov 11, 2026

hckrman101 mentioned this pull request Nov 11, 2026

fix(vk): keep programs the driver faulted on out of the driver (#996) #1823

Merged

Navigation Menu

Repository navigation

fix(vk): a shader preparation the GPU driver stops answering in goes on or ends, and says why (#2107) - #2123

fix(vk): a shader preparation the GPU driver stops answering in goes on or ends, and says why (#2107)#2123
nullifyshot wants to merge 3 commits into
NotRexed:mainfrom
nullifyshot:fix/predict-stall-guard

nullifyshot commented Nov 11, 2026 •

edited

Loading

Uh oh!

hckrman101 left a comment

Uh oh!

hckrman101 Nov 11, 2026

Uh oh!

nullifyshot Nov 11, 2026

Uh oh!

hckrman101 Nov 11, 2026

Uh oh!

nullifyshot Nov 11, 2026

Uh oh!

hckrman101 Nov 11, 2026

Uh oh!

nullifyshot Nov 11, 2026

Uh oh!

hckrman101 Nov 11, 2026

Uh oh!

nullifyshot Nov 11, 2026

Uh oh!

nullifyshot commented Nov 11, 2026

Uh oh!

nullifyshot commented Nov 11, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

nullifyshot commented Nov 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What happens

Changes

Tested

Uh oh!

hckrman101 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nullifyshot commented Nov 11, 2026

Uh oh!

nullifyshot commented Nov 11, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

nullifyshot commented Nov 11, 2026 •

edited

Loading