Skip to content

fix(vk): build every predicted pipeline before the game (the prediction found nothing on fresh builds; no 90 s budget; one progress bar) - #2032

Merged
hckrman101 merged 2 commits into
NotRexed:mainfrom
hckrman101:predict-all-in-prepare
Nov 11, 2026
Merged

hckrman101 merged 2 commits into
NotRexed:mainfrom
hckrman101:predict-all-in-prepare

Conversation

@hckrman101

Copy link
Copy Markdown
Collaborator

The pipelines the prediction makes are meant to be built before the game. On a fresh build, though, none were predicted at all. Where they were, a 90 s budget left most of them to be compiled in play, each one a stall on its first draw. This PR builds every predicted pipeline in the preparation, on a progress bar that counts them from the start.

What changes

  1. fix(vk): the prediction sees the programs the preparation read in.
    • program_cache_load and program_cache_load_some take a key's records out of disk_records when they read a program in.
    • The preparation reads in every program the state list names (state_list_prepare) just before predict_run.
    • program_cache_each listed disk_records alone, so no note found its program. Every new build logged predicted 0 pipelines from 1261 notes for 0 program variants.
    • The records taken out are now kept in disk_taken, and program_cache_each lists both.
    • A played folder whose start did not run state_list_prepare (for example, a start after a driver update) predicted fine, which is why this went unnoticed.
  2. fix(vk): build every predicted pipeline in the preparation, on one progress bar.
    • No budget by default. RX_VK_PREDICT_S no longer defaults to 90 s; setting it still sets a budget.
    • Every CPU, normal priority. The workers now use every CPU at normal priority, as precompile_run already does; nothing else runs before the game. They used half the CPUs below normal priority. (fix(vk): bound precompile thread concurrency via half_cpus (#1593) #1610's job limit would apply here the same way.)
    • Closing the window stops them, through prepare_stopped(). What the driver has built is kept.
    • One bar. The start-up screen and the builder's --prepare-shaders bar now cover the programs, the predicted pipelines and the noted pipelines together (PrepareBar). The total holds all three from the start.
      • Until a part counts itself, it stands at a guess from the files on disk (prepare_guess: the seed's program count, the notes and the lists).
      • A pipeline weighs as two programs, the measured ratio.
      • The bar never goes back and never reads 100% while pipelines are still being built.
      • Before, the screen showed a second bar after a full one, and the builder gave the predicted pipelines a fixed 20% share.
    • Log. It now says how long deciding the predictions took, separately from building them.

Numbers

Setup: RTX 3090, Ryzen 9 7900X (24 threads), Linux, OoT 1.1.1, an empty NVIDIA shader cache, and the shipped 1,261-pipeline list.

Case Predicted Built in the preparation Time
main, fresh build 0 (the bug) 0 prepare 116 s with the program cache present
main, played folder after a driver update 43,406-45,882 8,634-10,858 (90 s budget, 12 threads) the rest in play, up to 205 draws a frame left out
this PR with RX_VK_PREDICT_S=90 43,414 13,941 93 s
this PR 43,414 43,414 7-11 s deciding + 306-320 s building
  • A whole cold preparation takes 457-581 s.
    • 457 s from nothing: translation 131 s, prediction 313 s, noted pipelines 10 s.
    • 581 s with the programs already translated but no compute list, so every compute pipeline was built.
  • A second preparation took 96 s, with the driver cache kept from the first.
  • Peak memory was 4.3 GB for the process.

Tests

  • tests/test_predict_sees_loaded.py (new): checks that both take-out sites keep what they take, and that program_cache_each lists those records too.
  • tests/test_startup_progress.py covers PrepareBar:
    • The finished programs leave room on the bar for the pipelines.
    • The bar holds while the prediction decides.
    • A part's count replaces its guess without the bar going back.
    • The bar stays under 1 until the last part is done.
    • A start with nothing to prepare fills the bar with its pipelines alone.
    • The screen shows the whole bar's share.
    • It also checks the source: predict_run has no default budget, and does not use half the CPUs or lower its priority.
  • tests/test_prepare_cache_save.py and tests/test_program_cache_cleanup.py: stubs updated.

Not tested: the Windows start-up screen and Windows builder bar (the logic is shared and tested here), and AMD or Intel GPUs.

hckrman101 and others added 2 commits November 10, 2026 19:20
A first start's prediction found none of the programs its notes name: program_cache_load and
program_cache_load_some take a key's records out of disk_records as they read the program in,
the state list's programs are read in by the preparation just before the prediction, and
program_cache_each listed disk_records alone. Every new build logged "predicted 0 pipelines
from 1261 notes for 0 program variants", so the predicted pipelines were all built in play.
The records taken out are kept (disk_taken) and program_cache_each lists both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ogress bar

The predicted pipelines stopped at a 90 s budget (RX_VK_PREDICT_S) and the rest were built
in play, where each one stalls its first draw. They are now all built before the game: no
budget by default (RX_VK_PREDICT_S=<s> still sets one), on every CPU at normal priority,
and stopped by closing the window like the rest of the preparation.

The start-up screen and the builder's --prepare-shaders bar now cover the programs, the
predicted pipelines and the noted ones as one bar whose total has all three from the start
(PrepareBar, guessed from the files on disk until each part counts itself), so it no longer
sits full while thousands of pipelines are still built. The log says how long deciding the
predictions took apart from building them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@hckrman101
hckrman101 merged commit ff85d7d into NotRexed:main Nov 11, 2026
11 checks passed
@hckrman101
hckrman101 deleted the predict-all-in-prepare branch November 11, 2026 00:47
NotRexed pushed a commit that referenced this pull request Nov 11, 2026
🤖 I have created a release *beep* *boop*
---


##
[2.1.0-rc.1](v2.0.14...v2.1.0-rc.1)
(2026-11-11)


### Features

* Add a control to disable camera centering
([2dcf210](2dcf210))
* Add an option to disable automatic camera centering
([657da4f](657da4f))
* Audio tab in the Insert menu with Volume and Mute in background
([52ca067](52ca067))
* Audio tab in the Insert menu with Volume and Mute in background
([df2d84c](df2d84c))
* **builder:** let the player choose how much memory a build may take
([90055d0](90055d0))
* **builder:** the Linux builder looks like the Windows one
([8c1b5e9](8c1b5e9))
* **builder:** the Linux builder looks like the Windows one
([8f2e1a1](8f2e1a1))
* **builder:** use the saved build memory choice for updates
([86ff22c](86ff22c))
* **builder:** warn before building on a hard drive
([b526a84](b526a84))
* **builder:** warn before building on a hard drive
([c769305](c769305))
* **build:** Probe for system Vulkan packages, add proper requirements
([3b84291](3b84291))
* **build:** Probe for system Vulkan packages, add proper
requirements.txt
([155bfa2](155bfa2))
* **linux:** opt-in NVIDIA DLSS in Linux Builder and AppImage
([484cfd1](484cfd1))
* **linux:** opt-in NVIDIA DLSS in Linux Builder and AppImage
([66b86fb](66b86fb))
* **menu:** a DualShock 4 or DualSense touchpad click opens the menu as
F1
([8790b8f](8790b8f))
* **menu:** a Mesh shaders row (Automatic, Native, Compute)
([#1971](#1971))
([c8022e7](c8022e7))
* **menu:** a Text size row for dialogue, Navi and signs
([4d0645f](4d0645f))
* **menu:** a Text size row for dialogue, Navi and signs
([4bae30c](4bae30c))
* **menu:** add 5120x2880 (5K) to Output resolution
([ce175ac](ce175ac))
* **menu:** centre the menu in a 16:9 column on ultrawide
([70e2209](70e2209))
* **menu:** DLAA as DLSS's Native render resolution, and Render
resolution as a slider in real sizes
([#2000](#2000))
([e187859](e187859))
* **menu:** Esc can open the Insert menu (Menu shortcut "Insert, F1 or
Esc") ([#1998](#1998))
([d926317](d926317))
* **menu:** let players choose the keyboard shortcut
([#1609](#1609))
([191cfdc](191cfdc))
* **menu:** put the debug tools on the Debug tab
([e256616](e256616))
* **menu:** redesign the Debug tab, Inventory and Teleport pages
([c805d9b](c805d9b))
* **menu:** save Output resolution as a size, and take any size from the
config
([a78ee4d](a78ee4d))
* **menu:** the debug tools' Teleport row opens the map list
([a4c3a8d](a4c3a8d))
* RecompLauncher support (recomplauncher.json, make_game
--game/--prepare)
([ab5a1f6](ab5a1f6))
* **startup:** lower the start-up screen, smaller text, fade to black at
the end
([8649805](8649805))
* **startup:** lower the start-up screen, smaller text, fade to black at
the end
([fb60134](fb60134))
* **vk:** add AMD Anti-Lag 2 (VK_AMD_anti_lag)
([8ea4e7f](8ea4e7f))
* **vk:** add opt-in VRAM compaction and hold the driver's count below
its limit
([3932421](3932421))
* **vk:** AMD Anti-Lag 2 through VK_AMD_anti_lag
([b9e7559](b9e7559))
* **vk:** Anisotropic filtering option, 16x by default, and the
upscalers' texture LOD bias
([0af7629](0af7629))
* **vk:** Anisotropic filtering row, 16x by default
([d9b0ce2](d9b0ce2))
* **vk:** draw the menu, FPS counter and start-up screen with Dear ImGui
([509579b](509579b))
* **vk:** opt-in VRAM compaction (RX_VK_COMPACT=1): the textures' free
space goes back to the driver
([d218438](d218438))
* **vk:** trace effective presentation geometry on changes
([#1457](#1457))
([a350759](a350759))
* **vk:** upscalers' texture LOD bias below the game's 3/4 render
([3b9ffcd](3b9ffcd))
* **vk:** warm the stage libraries from the program cache into the
pipeline cache
([24ebac7](24ebac7))


### Bug Fixes

* a game folder whose deepest file passes 260 characters builds and
plays ([#1468](#1468))
([d231d17](d231d17))
* a game folder whose deepest file passes 260 characters builds and
plays ([#1468](#1468))
([45c7418](45c7418))
* Address camera centering review feedback
([103e2cd](103e2cd))
* **appimage:** copy sysroot library aliases by value for updates
([#1536](#1536))
([ed66422](ed66422))
* **audio:** keep 512-frame power-of-two default on SDL2 fallback
([#1550](#1550))
([fa85175](fa85175))
* **build:** a game built from extracted folders refuses links in them
([233e072](233e072))
* **builder:** --cli says make_game's own error, on stdout too
([a3859f1](a3859f1))
* **builder:** bound writes from compressed NUP members
([#1596](#1596))
([56e31e7](56e31e7))
* **builder:** constrain the Show details hit area
([d45e262](d45e262))
* **builder:** constrain the Show details hit area
([080bf91](080bf91)),
closes [#1842](#1842)
* **builder:** give the prepare shaders tick its own row
([1ace933](1ace933))
* **builder:** honor DLSS opt-in when updating a game
([#1643](#1643))
([c3aa9c0](c3aa9c0))
* **builder:** report bytes during NUP extraction
([#1671](#1671))
([c4a9085](c4a9085))
* **builder:** select the saved build memory in BeginUpdate
([d08aaed](d08aaed))
* **builder:** the Linux builder's details box is read-only
([b2ac2bf](b2ac2bf))
* **builder:** the Linux builder's details box is read-only
([d7cc1d4](d7cc1d4))
* **build:** escape the build folder in glob patterns (a work folder
with [ ] found no lifted code)
([#1722](#1722))
([caaee0b](caaee0b))
* **build:** invalidate DLSS objects when SDK headers change
([#1648](#1648))
([c716137](c716137))
* **build:** package scripts/ in the Linux AppImage, so NVIDIA DLSS can
be fetched
([b0aa6f6](b0aa6f6))
* **build:** package scripts/ in the Linux AppImage, so NVIDIA DLSS can
be fetched
([af9e98c](af9e98c))
* **build:** stop sampling memory for exited compiler roots
([#1562](#1562))
([4a7ae1b](4a7ae1b))
* **build:** the logo and fonts are made only where the Vulkan back end
is compiled (an AppImage build stopped at LibertinusSans-Regular.ttf)
([#1567](#1567))
([6add2f1](6add2f1))
* Discord status starts on Windows (it was only started in the Linux
build's main)
([#1539](#1539))
([7d1562d](7d1562d))
* **display:** address pointer sampling review feedback
([acf9dae](acf9dae))
* **display:** do not toggle fullscreen on F11 when modifier keys are
held ([#1309](#1309))
([#1608](#1608))
([635ba1a](635ba1a))
* **display:** integrate coherent game-menu pointer samples
([09685de](09685de))
* **display:** integrate main and declare pointer fixture requirements
([8a6f1e5](8a6f1e5))
* **display:** open the window on COSMIC's Wayland with SDL2
([#992](#992))
([e937168](e937168))
* **display:** publish coherent pointer and drawable samples
([fd93173](fd93173))
* **display:** publish coherent pointer and drawable samples
([ee53072](ee53072))
* **env:** hookshot_env.txt sets no test switch that names a file or
presses buttons
([74cee1e](74cee1e))
* **framegen:** 2x made by the renderer's own blend again; AMD FSR 3.1's
interpolation opt-in (RX_VK_FSR_FG=1)
([eb5b9a0](eb5b9a0))
* **framegen:** 2x uses the renderer's own blend again; FSR 3.1
interpolation opt-in
([#1635](#1635))
([3b76e95](3b76e95))
* hardening part 3, updates, shared files, guest buffers and texture
sizes
([d5feb7b](d5feb7b))
* **hid:** deduplicate gamepad polling when SDL controllers are active
([#1480](#1480),
[#1583](#1583))
([2b0f762](2b0f762))
* **hid:** deduplicate gamepad polling when SDL controllers are active
([#1480](#1480),
[#1583](#1583))
([731898a](731898a))
* **hos:** host copies, file reads and writes, and the NV12 reader stay
in guest memory
([6f8f243](6f8f243))
* **host:** room in the drivers' shader caches for the prepared
pipelines (Linux)
([#2033](#2033))
([c9ec3bb](c9ec3bb))
* **host:** the cache cores only where the L3s are shared alike and the
largest has 8 cores (review of
[#1638](#1638))
([428633a](428633a))
* **input:** a stick axis steers once it has been seen at rest (camera
spinning on its own,
[#1673](#1673),
[#1661](#1661))
([#1677](#1677))
([d8f0513](d8f0513))
* **linux:** safely rebase legacy sysroot links
([#1553](#1553))
([ca3a252](ca3a252))
* **loader:** add the addend to GLOB_DAT and JUMP_SLOT relocations
([1372edb](1372edb))
* **loader:** add the addend to GLOB_DAT and JUMP_SLOT relocations
([95a9a07](95a9a07))
* **make_game:** --prepare says errors and --check on stdout, and
refuses a missing --game
([2d27d08](2d27d08))
* **menu:** A and B confirm and go back on a controller, as in the game
([02b0903](02b0903))
* **menu:** a save keeps the options file's lines the menu does not
write ([#1493](#1493))
([a2e6505](a2e6505))
* **menu:** a size from the config shows as its size alone, no label
([bc9406c](bc9406c))
* **menu:** clarify frame-generation preset help
([#1554](#1554))
([77685a5](77685a5))
* **menu:** distinguish game and generated frame rates
([#1659](#1659))
([b2bb214](b2bb214))
* **menu:** F1 always opens the menu (Menu shortcut: Insert or F1, or F1
only) ([#1632](#1632))
([d448ccc](d448ccc))
* **menu:** Frame generation is the player's choice; Quality presets
leave it alone
([2ef499d](2ef499d))
* **menu:** hide rows for features this PC lacks, and keep mouse clicks
on the row meant
([51de681](51de681))
* **menu:** hide rows for features this PC lacks, and keep mouse clicks
on the row meant
([849b63e](849b63e))
* **menu:** keep the FPS counter off the banner on a window just wider
than 16:9 ([#1545](#1545))
([50d71bd](50d71bd))
* **menu:** keep the help box and rows above the prompt strip
([bd1616f](bd1616f))
* **menu:** Menu size is Medium on every input, and the version shrinks
to fit
([73feec0](73feec0))
* **menu:** Quality presets leave Frame generation alone, so it stays
off by default
([a72690c](a72690c))
* **menu:** run the upscalers' one-time probes during the start-up
screen, not at the menu's first open
([#1886](#1886))
([#1916](#1916))
([74075de](74075de))
* **menu:** the FPS counter looks as it did (green outlined text in the
corner, no plate)
([#1957](#1957))
([92907f7](92907f7))
* **menu:** the mouse wheel scrolls, it never changes a value
([#1557](#1557))
([be541fa](be541fa))
* **nup:** name the member and the drive when extraction runs out of
space ([#1595](#1595))
([218d8de](218d8de))
* **packaging:** include the prompt encoder's runtime dependency
([#1598](#1598))
([53a5fd5](53a5fd5))
* preserve UTF-8 and extraction guards in long-path support
([#1592](#1592))
([38a5794](38a5794))
* **render:** publish coherent shared temporal jitter samples
([a38a07d](a38a07d))
* **render:** publish coherent temporal jitter samples
([0291727](0291727))
* **render:** reset temporal history when reprojection is invalid
([0cf9936](0cf9936))
* **render:** reset temporal history when reprojection is invalid
([19b81b7](19b81b7))
* **rt:** mark an exclusive store's page after the store, not before
([7cb4028](7cb4028))
* **rt:** mark an exclusive store's page after the store, not before
([db6324f](db6324f))
* **scripts:** the profile readers name and count Linux samples
([#1746](#1746))
([6269088](6269088))
* **security:** a font's cmap offset, a microphone's name, and the
game's library loads stay in bounds
([9323390](9323390))
* **security:** verify Windows updater asset digests
([6d5f277](6d5f277))
* **security:** verify Windows updater asset digests
([50b6866](50b6866)),
closes [#1679](#1679)
* **setup:** explain how to restore missing Windows tar
([1c1995e](1c1995e))
* **setup:** explain recovery when Windows tar is missing
([fbaf722](fbaf722))
* **shaders:** a loop that leaves by predicated EXITs ends at its
closing self-branch (the last program that did not lower)
([a8e7ee5](a8e7ee5))
* **shaders:** a program ends at its closing self-branch after an
unconditional jump back
([7e1db37](7e1db37))
* **shaders:** an immediate's low byte is not a scratch register
([d0f3511](d0f3511))
* **shaders:** ISBERD.O, a task or mesh program reading back its output
region
([67c048e](67c048e))
* **shaders:** ISBERD.O, a task program reading back its payload (12
failing pipelines)
([0904155](0904155))
* **shaders:** LDG [URn+imm] with both ordering bits 77-78 lowers
([692a1c0](692a1c0))
* **shaders:** LDG/STG [URn+imm] with ordering bit 78 lower (2 compute
programs)
([5ec9668](5ec9668))
* **shaders:** lower TLD.MS (multisampled texel fetch) to Maxwell TLDS
2D.LZ.MS
([6860451](6860451))
* **shaders:** lower TLD.MS, a multisampled texture's texel, to Maxwell
TLDS 2D.LZ.MS
([20dd78b](20dd78b))
* **shaders:** R2UR's source is one register, not a pair
([597edcb](597edcb))
* **shaders:** R2UR's source is one register, not a pair (a vertex
program refused)
([9edb3a5](9edb3a5))
* **shaders:** STG [URn+imm] with ordering bit 78 lowers too
([923a334](923a334))
* **shaders:** two large compute programs lower (an immediate is not
R254) and get their local memory
([10cfd70](10cfd70))
* the display stays on while playing with a controller
([#1532](#1532))
([#1538](#1538))
([c38df20](c38df20))
* **update:** Linux downloads are GETs again; a part's links stay in the
install; the requested tag is installed
([df9a52e](df9a52e))
* **vi:** a cutscene's cues stay with its movie above a 60 cap
([a7cc747](a7cc747))
* **vi:** a cutscene's cues stay with its movie above a 60 cap
([#1742](#1742))
([6c596af](6c596af))
* **vi:** keep the game at real speed under a Frame rate cap above 60
([#1612](#1612))
([1a429f6](1a429f6))
* **vk:** a masked clear that backed off is tried again now and then
([#1787](#1787))
([fa04b6f](fa04b6f))
* **vk:** a masked clear whose pipeline the driver did not make is tried
again, not cached as null
([#1787](#1787))
([9d94f4c](9d94f4c))
* **vk:** a masked clear whose pipeline the driver did not make is tried
again, not cached as null
([#1787](#1787))
([f002c90](f002c90))
* **vk:** a shadow is made only where the textures' own lines are not
passed (review of
[#1637](#1637))
([ecb6e9a](ecb6e9a))
* **vk:** a switch for an engine that cannot run leaves a saved FSR 3.1
([#1773](#1773) review)
([a9ea73a](a9ea73a))
* **vk:** a texture's storage size is computed in 64 bits, and one past
the GPU's limits is not made
([81a6da5](81a6da5))
* **vk:** add VMA image allocation retry without WITHIN_BUDGET
([39d978b](39d978b))
* **vk:** add VMA image allocation retry without WITHIN_BUDGET
([560cc5a](560cc5a))
* **vk:** admit only successful samplers across cache layers
([52ac35c](52ac35c))
* **vk:** admit only successful samplers across cache layers
([3379314](3379314))
* **vk:** ask for task and mesh shader counts only where the driver
offers them
([ba86aee](ba86aee))
* **vk:** ask for task and mesh shader counts only where the driver
offers them
([a3d88ea](a3d88ea))
* **vk:** ask the pipeline cache for a compute pipeline before queueing
its build
([0db1e74](0db1e74))
* **vk:** avoid aligned uniform-word cache collisions
([1b25b22](1b25b22))
* **vk:** build every predicted pipeline before the game (the prediction
found nothing on fresh builds; no 90 s budget; one progress bar)
([#2032](#2032))
([ff85d7d](ff85d7d))
* **vk:** build the precompile's first graphics pipeline alone
([41b3438](41b3438))
* **vk:** capture jitter cache frame under its lock
([1f32a16](1f32a16))
* **vk:** check cached descriptor counts and the formats of noted
pipelines
([59f9284](59f9284))
* **vk:** coloured specks on fallback foliage (AMD): the generated
vertex stage's outputs start at (0,0,0,1)
([16631b6](16631b6))
* **vk:** commit bank cache generations after successful stream upload
([60ea984](60ea984))
* **vk:** compute programs get their local memory size (control section
+0x68)
([a083825](a083825))
* **vk:** dark lighting and glowing bloom on AMD: 64-lane graphics warps
and Lichii [#273](#273
fragment shuffle fallback
([030e605](030e605))
* **vk:** decode Deck gameplay upscale as RGB9E5
([903bc4a](903bc4a))
* **vk:** decode gameplay RGB9E5 upscale fallback
([6c936d4](6c936d4))
* **vk:** dense grass cut off and far trees flickering without mesh
shaders: fallback records from an allocator, 65535 slots
([7047756](7047756))
* **vk:** disable failed frame log sinks and unwind startup
([#1531](#1531))
([5fde3cf](5fde3cf))
* **vk:** establish capture copy visibility for host reads
([7ef03d4](7ef03d4))
* **vk:** expose the retirement helper across renderer units
([d5b7437](d5b7437))
* **vk:** flush custom UI texture staging before GPU use
([#1559](#1559))
([188baa6](188baa6))
* **vk:** invalidate capture readbacks before CPU reads
([0c6c047](0c6c047))
* **vk:** keep adjacent uniform words in separate cache slots
([7b81752](7b81752))
* **vk:** keep Windows prepare times in FILETIME ticks
([#1517](#1517))
([edeaa9d](edeaa9d))
* **vk:** let a draw run its own queued shader translation
([#1714](#1714))
([8bac8aa](8bac8aa))
* **vk:** log a driver fault caught by the Windows pipeline guards
([ac7d4a7](ac7d4a7))
* **vk:** make capture readbacks visible before CPU reads
([6e06af5](6e06af5))
* **vk:** make frame dump copies visible to host reads
([#1621](#1621))
([c1a4fde](c1a4fde))
* **vk:** mark a clear's targets in the parallel replay's dry pass (the
one-frame black scenes)
([adb7a5d](adb7a5d))
* **vk:** missing floor grass without mesh shaders: indirect task draws
ran no mesh workgroups
([ccf7f41](ccf7f41))
* **vk:** own native FSR constants for each recording
([2982420](2982420))
* **vk:** own native FSR constants for each recording
([895d673](895d673))
* **vk:** parallel replay: a clear's targets are marked in the dry pass
(the one-frame black scenes)
([168e4d1](168e4d1))
* **vk:** present ahead's vk_replay.cpp half, lost in
[#1720](#1720 merge
([be8b512](be8b512))
* **vk:** present ahead's vk_replay.cpp half, lost in
[#1720](#1720 merge (main
doesn't link)
([1392ea3](1392ea3))
* **vk:** preserve brightness in flash sampling across capture sizes
([#1537](#1537))
([6f5ecab](6f5ecab))
* **vk:** preserve contiguous words in the mixed uniform cache
([70ffaf2](70ffaf2))
* **vk:** preserve CPU write watches during pool imports
([#1434](#1434))
([74dcb75](74dcb75))
* **vk:** prevent aligned uniform words from thrashing the cache
([78e1112](78e1112))
* **vk:** publish complete PNG captures and report failures
([#1604](#1604))
([ba40a1a](ba40a1a))
* **vk:** publish complete screenshot resource sets
([#1472](#1472))
([ff69134](ff69134))
* **vk:** publish recording stream writes before GPU submission
([d71739d](d71739d))
* **vk:** publish recording stream writes before GPU submission
([0e843bd](0e843bd))
* **vk:** publish texture staging slices before GPU reads
([#1672](#1672))
([a6e2e14](a6e2e14))
* **vk:** Read texture handles from stage constant banks
([31933e4](31933e4))
* **vk:** Read texture handles from stage constant banks
([4654870](4654870))
* **vk:** recover recording stream and descriptor allocation failures
([fad4d16](fad4d16))
* **vk:** recover recording stream and descriptor allocation failures
([6e93f80](6e93f80))
* **vk:** reject failed descriptor-pool creation results
([6a51fc2](6a51fc2))
* **vk:** reject failed samplers before descriptor publication
([2914a73](2914a73))
* **vk:** reject missing RGB9E5 fallback views
([6f5f769](6f5f769))
* **vk:** reject unbound imported buffers and roll back ownership
([325583b](325583b))
* **vk:** reject unbound imported buffers and roll back ownership
([af8f012](af8f012))
* **vk:** replace menu upload buffers only after successful growth
([#1453](#1453))
([60b0c9b](60b0c9b))
* **vk:** report device loss throughout native presentation
([5997b2d](5997b2d))
* **vk:** report device loss throughout native presentation
([8e1abe3](8e1abe3))
* **vk:** reset swap_spare with the swapchain; leave a skipped present
out of the counts
([08f3fe4](08f3fe4))
* **vk:** retire native FSR images after GPU completion
([8fc2587](8fc2587))
* **vk:** retire native FSR images after GPU completion
([4aaf336](4aaf336))
* **vk:** retry transient texture image view failures
([8686552](8686552))
* **vk:** retry transient texture image view failures
([394b797](394b797))
* **vk:** RX_VK_FSR4 and RX_VK_DLSS go over a saved FSR 3.1
([#1008](#1008) follow-up)
([4cde5c6](4cde5c6))
* **vk:** RX_VK_FSR4 and RX_VK_DLSS go over a saved FSR 3.1
([#1008](#1008) review)
([6229c02](6229c02))
* **vk:** RX_VK_VRAM_MB and the integrated-GPU cap win over a preset's
texture budget
([#1376](#1376))
([e27740a](e27740a))
* **vk:** share validated source crops across presentation paths
([187e70b](187e70b))
* **vk:** share validated source crops across presentation paths
([d4c0429](d4c0429))
* **vk:** shared_pipeline_cache outside the unnamed namespace
(vk_fsr3.cpp could not link to it)
([9d44621](9d44621))
* **vk:** slow resume and black HUD hearts on 4 GB cards: the texture
squeeze waits while VMA's blocks have room
([189287f](189287f))
* **vk:** snapshot pending shots before reusing their readback slot
([#1519](#1519))
([aaf6ac1](aaf6ac1))
* **vk:** submit a 2x set's copies together so its presents keep their
times
([64cfe26](64cfe26))
* **vk:** submit a 2x set's copies together so its presents keep their
times
([79e42ac](79e42ac))
* **vk:** the upscale stand-in and the texture clear call
texture_written as before
([73c6dd3](73c6dd3))
* **vk:** tie texel buffer views to their imported backing
([692e87d](692e87d))
* **vk:** tie texel buffer views to their imported backing
([a23587f](a23587f))
* **vk:** Timestamp frame captures with the host clock
([3563479](3563479))
* **vk:** Use host timestamps in frame captures
([b2ed731](b2ed731))
* **vk:** validate capture dimensions before allocating readbacks
([#1551](#1551))
([17e8cd8](17e8cd8))
* **vk:** validate HDR crops through the shared source rectangle
([f20afae](f20afae))
* **vulkan:** avoid the AMD cloth pipeline optimizer hang
([bebe51e](bebe51e))
* **vulkan:** finish the RX 6700 XT merge resolution
([5ffdaeb](5ffdaeb))
* **vulkan:** refresh direct-write tracking on current main
([a4c8f85](a4c8f85))
* **vulkan:** stop distant trees flickering in the mesh fallback
([7fc0a61](7fc0a61))
* **vulkan:** track writes from direct buffer bindings
([be1aabc](be1aabc))
* write-back stale-shadow edgecase
([c5aa906](c5aa906))


### Performance Improvements

* **host:** keep the game's threads on the 3D V-Cache cores of two-CCD
X3D CPUs
([1360a44](1360a44))
* **host:** on X3D CPUs with two CCDs the game's threads stay on the
cores that share the 3D V-Cache
([b3c0982](b3c0982))
* **host:** write the log in batches, a few writes a second instead of
one per line
([c7e41fd](c7e41fd))
* **host:** write the log in batches, a few writes a second instead of
one per line
([7f12643](7f12643))
* **vk:** a captured page's words read from the capture, not a 4 KB copy
kept a thread
([0b951ec](0b951ec))
* **vk:** copy captured pages with non-temporal stores
([d8ce191](d8ce191))
* **vk:** copy captured pages with non-temporal stores
([146fe38](146fe38))
* **vk:** Descriptor reuse on by default (options_version 4)
([f758d9f](f758d9f))
* **vk:** Descriptor reuse on by default (options_version 4)
([f214c0c](f214c0c))
* **vk:** DLSS keeps its feature through loading screens while it is the
Upscaler (a 44-78 ms stutter at each area entry)
([#1520](#1520))
([549a43b](549a43b))
* **vk:** flush only each FSR uniform slice
([c7c88fa](c7c88fa))
* **vk:** FSR 3.1's passes go through the pipeline cache kept across
runs
([e805e02](e805e02))
* **vk:** FSR 3.1's passes go through the pipeline cache kept across
runs (a 1.7 s longer first frame of play)
([b27d9b6](b27d9b6))
* **vk:** FSR 3.1's pipelines through the play caches too
(shared_compute_pipeline)
([46efc77](46efc77))
* **vk:** in play an ask without compiling goes through the play caches
too ([#1765](#1765
dispatch, the stage libraries)
([6593de7](6593de7))
* **vk:** keep 32 submitted pages a thread, not one
([338837d](338837d))
* **vk:** keep four pages for the replay's reads as submitted, not one
([3bae557](3bae557))
* **vk:** keep four pages for the replay's reads as submitted, not one
(Kokiri village view 50 -&gt; 57 fps)
([72133cb](72133cb))
* **vk:** keep the write-back shadows while VRAM is full of free space
inside VMA's blocks
([6a4f3ea](6a4f3ea))
* **vk:** less copying of captured memory on the game and replay threads
([e63d30d](e63d30d))
* **vk:** RX_VK_SUBMITTED_PAGES=1 keeps one submitted page, as before
[#1762](#1762); tests follow
[#1762](#1762) and
[#1619](#1619)
([cfefd0e](cfefd0e))
* **vk:** skip write-backs of large GPU-written bindings
([2ea6386](2ea6386))
* **vk:** skip write-backs of large GPU-written bindings
([30911ca](30911ca))
* **vk:** the capture's copies split by one function; a caller runs its
own parts no helper has taken
([c747a2c](c747a2c))
* **vk:** the capture's streaming-store copy is opt-in
(RX_VK_CAPTURE_NT=1): more stutters, no lower capture time here
([#1841](#1841))
([f4e6fdd](f4e6fdd))
* **vk:** the texture shadow's pages kept in place and copied by the
capture helpers
([a4ccf07](a4ccf07))
* **vk:** the write-back shadows stay while VRAM is full of free space
inside VMA's blocks
([23738b6](23738b6))
* **vk:** what play makes goes into a pipeline cache of its own, one
file per save (no 0.5-1.2 GB rewrite in play)
([74c15be](74c15be))
* **vk:** what play makes goes into a pipeline cache of its own, one per
save, not the whole cache again
([1b0ea22](1b0ea22))
* **vk:** write the pipeline cache only when its data changed (no 1.1 GB
rewrite at every start)
([bd5086a](bd5086a))
* **vk:** write the pipeline cache only when its data changed since it
was read or last written
([5a3fd42](5a3fd42))
* **vk:** write the pipeline cache only when its data changed since it
was read or last written
([9f4fb8e](9f4fb8e))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
nullifyshot added a commit to nullifyshot/Hookshot that referenced this pull request Nov 11, 2026
…on or ends, and says why (NotRexed#2107)

Builds on Radeon and GeForce RTX 40 cards stopped for good while preparing shaders, the Builder's details
showing "vertex format NVN 0xe -> vk 17". The driver had faulted inside a pipeline compile; the fault guard
caught it and left the driver holding a lock of its own, so every other worker waited on it for ever.

- predict_run (new at build time since NotRexed#2032) builds its first pipeline alone, as precompile_run does
  for NVIDIA's fault at many first pipelines at once, and goes on without the pipelines the driver never
  finishes once nothing has finished for 30 s (RX_VK_PREDICT_STALL_S). Its count moves per pipeline.
- The --prepare-shaders run has a watchdog for every part: after 30 s with no other line it logs what
  it is working on, or what it is waiting on (the GPU driver, or the disk) and since when, and any driver
  fault it caught. When nothing has moved for 5 minutes (RX_VK_PREPARE_STALL_S) it logs why, leaves
  hookshot_prepare_skipped so the game's starts don't stop in the same place, and ends with exit status 4
  (TerminateProcess, not exit(), which would unload the stuck driver). The pipeline cache's save is
  reported, never stopped.
- make_game.py says "the GPU driver stopped responding" for exit status 4.
- tests/test_prepare_stall.py compiles predict_run and the watchdog from the source with a stand-in
  driver: a stuck pipeline, a slow one, a fault, the save.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nullifyshot added a commit to nullifyshot/Hookshot that referenced this pull request Nov 11, 2026
…on or ends, and says why (NotRexed#2107)

Builds on Radeon and GeForce RTX 40 cards stopped for good while preparing shaders, the Builder's details
showing "vertex format NVN 0xe -> vk 17". The driver had faulted inside a pipeline compile; the fault guard
caught it and left the driver holding a lock of its own, so every other worker waited on it for ever.

- predict_run (new at build time since NotRexed#2032) builds its first pipeline alone, as precompile_run does
  for NVIDIA's fault at many first pipelines at once, and goes on without the pipelines the driver never
  finishes once nothing has finished for 30 s (RX_VK_PREDICT_STALL_S). Its count moves per pipeline.
- The --prepare-shaders run has a watchdog for every part: after 30 s with no other line it logs what
  it is working on, or what it is waiting on (the GPU driver, or the disk) and since when, and any driver
  fault it caught. When nothing has moved for 5 minutes (RX_VK_PREPARE_STALL_S) it logs why, leaves
  hookshot_prepare_skipped so the game's starts don't stop in the same place, and ends with exit status 4
  (TerminateProcess, not exit(), which would unload the stuck driver). The pipeline cache's save is
  reported, never stopped.
- make_game.py says "the GPU driver stopped responding" for exit status 4.
- tests/test_prepare_stall.py compiles predict_run and the watchdog from the source with a stand-in
  driver: a stuck pipeline, a slow one, a fault, the save.

Assisted-by: Claude Opus 5.5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
hckrman101 pushed a commit that referenced this pull request Nov 11, 2026
…on or ends, and says why (#2107) (#2195)

Since v2.1.0, some builds stop for good while preparing shaders (#2107).
#2186 made the slow-but-working case visible: the pipeline steps now log
their progress every 10 s. A run where the GPU driver itself stops
answering still waits forever, though. This makes that run go on, or end
with a reason, so the game's first start doesn't hang in the same place.

This replaces #2123. It's the same work, rebased onto #2186, with
hckrman101's review there addressed, and with the status lines #2186 now
covers taken out.

### What happens

One player's log (an RX 580) shows a likely cause:

```
vk: the driver faulted compiling a graphics pipeline (exception 0xc0000005): that one is left out
vk: graphics pipeline for 6ee656c480d6beea recovered with optimization disabled
```

- `guarded_graphics` catches the driver's crash by jumping out of the
compile, and that leaves the compile unfinished inside the driver.
#1823's process dump of issue 996 shows what follows on AMD: later
builds of that program wait inside the driver forever.
- The step it stops in is `predict_run`, which runs at build time since
#2032. Unlike `precompile_run`, it had no stall guard.
- It also started all its workers' first pipelines at once.
`precompile_run` already avoids that, because of a known NVIDIA fault
(621.14, RTX 4070 Ti Super).
- Most reports in #2107 were slow rather than stuck, which #2186 now
shows. This PR changes nothing for those (no fault, still moving) beyond
the lines below.

### Changes

**`predict_run`** (`gpu/vk/vk_draw.cpp`)
- Builds its first pipeline alone, as `precompile_run` does.
- If no pipeline finishes for 30 s (`RX_VK_PREDICT_STALL_S`, 0 waits for
all), it goes on and leaves the stuck workers in the driver. Their state
lives on the heap, as in `precompile_run`. This happens:
- **anywhere in the run**, only after a driver fault was caught in it
(`driver_faults()`);
- **otherwise only once every program has been handed out**, as
`precompile_run`'s guard does. So only the last stragglers are given up,
never the bulk. A slow driver that still works keeps its pipelines, at
the game's own start too. A real hang before then is the watchdog's to
end.
- An abandoned worker builds no more of its program's list.
- The count moves per pipeline, not per program, so a slow program
doesn't look stuck. #2186's progress line reads the same count.
- `predict_run` and `precompile_run` record when they leave workers in
the driver (`prepare_left_workers()`).

**A watchdog for the whole `--prepare-shaders` run**
(`gpu/vk/vk_main.cpp`), covering every part on every GPU:
- **What it's waiting on.** Every 30 s that nothing has moved, it logs
the part, its count, and what it's waiting on: "the GPU driver or the
disk", "the GPU driver, which faulted (above)", or "the GPU driver,
which still has pipelines it did not finish". It logs this even while
#2186's 10-second lines keep the log busy, because those repeat the same
count during a hang. A driver fault it caught gets a line of its own as
soon as it happens.
- **The shader programs' count.** That step logs no count of its own, so
the watchdog writes one after 30 s of a quiet log. It writes nothing of
its own while the pipeline steps are moving, since #2186's lines cover
them.
- **Stops when nothing moves for 5 minutes** (`RX_VK_PREPARE_STALL_S`, 0
never stops). It then:
  - logs `STOPPED: ... the GPU driver stopped responding`
- leaves `hookshot_prepare_skipped`, so the game's own starts don't hang
in the same place (what wasn't prepared is built when first drawn)
- ends with exit status 4 through `TerminateProcess`, not `exit()`,
which would unload a driver that can still hold workers

The 5-minute limit is long on purpose: reading the caches and deciding
the predictions report no count for a while on a slow disk or CPU.
- **The pipeline cache's save, the last step:**
- **If workers were left in the driver, or a fault was caught:** the
save is stopped the same way, since `vkGetPipelineCacheData` can wait on
them. A run that gets through it ends at once with `TerminateProcess`
(after `@@done shaders`).
- **Otherwise:** the save is only reported, and its lines stop after 300
s, so make_game's `PREPARE_SILENCE` (15 minutes with no line) still ends
a save that never finishes.
- The part, its count and the time are read under one lock.

**Elsewhere**
- `make_game.py` maps exit status 4 to "Left to the game's first start:
the GPU driver stopped responding". Before, players waited out
`PREPARE_SILENCE`. #2155 then cleans up the leftover process, if any,
before an update.
- `driver_faults()`: the fault guards count what they catch. These are
the same four guard blocks #1823 changes, so whichever lands second
needs a mechanical rebase. After #1823, the count would follow its
`kDriverFault` path.

### Tested

- `tests/test_prepare_stall.py` (new) compiles `predict_run` and the
watchdog from the source with a stand-in driver. All 14 cases pass:
  - **`predict_run`:**
    - every pipeline built, with the first one alone
    - the last pipelines never finish: the run goes on
- a slow driver that still finishes pipelines, with each program longer
than the limit: no stop
    - a fault, then nothing finishes early in the run: the run goes on
    - the same with no fault: waited for
  - **The watchdog:**
    - stuck, after a fault, and while the shader programs move
- stuck and moving while other lines come every 200 ms, as #2186's do:
it still says what it's waiting on and stops, and says nothing of its
own while things move
    - a long clean save: reported, then silent, never stopped
    - a save with workers left in the driver: stopped
    - a run ending through `prepare_end`
    - in each case, the lines it logs, the exit status and the skip file
  - **`make_game.py`:** its message for exit status 4
- `vk_main.cpp` and `vk_draw.cpp` compile with the Windows toolchain's
flags (llvm-mingw, `-fms-extensions`).
- Updated fixtures for the new calls: `test_prepare_cache_save`,
`test_precompile_compute_guard`, `test_linux_driver_fault_guard`.
- `python -I -B tests/run_all.py` on Windows: 341 passed, 3 failed.
`test_recorder_sync`, `test_fpx_sticky` and
`test_precompile_compute_guard` fail the same way on an untouched copy
of main on this PC (the last needs MSVC's libraries). The Linux-only
checks, `test_linux_driver_fault_guard` among them, are left to CI.
- **Not tested:** a real hang on one of the affected cards. I couldn't
reproduce it here (RTX 5080).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: breakthetargets <81792782+breakthetargets@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant