Repository navigation
feat(vk): Compile shaders (All / Most / Later) and When a shader isn't ready (Wait / Wait briefly / Skip) - #2465
Draft
hckrman101 wants to merge 6 commits into
Draft
hckrman101 wants to merge 6 commits into
hckrman101 wants to merge 6 commits into
Conversation
…t ready (Wait / Wait briefly / Skip) Before the game, All prepares every shader as before. Most (the default) translates the programs and builds the noted pipelines; compute programs play is known to dispatch get every texture variant, the others their base one. The predicted pipelines and the other compute variants are then built behind the game on idle-priority threads, gated on the frame's own work and on memory, resumable across starts, until the folder has what All gives it. Later does all of it behind the game. A "Shaders: NN%" counter shows while it runs; an area's first frames are held (up to 3 s) until their pipelines are ready. When a shader isn't ready: Wait, Wait briefly then skip (15% of the capped frame, the default), or Skip. A draw that writes memory or draws into a target made this frame always waits. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…and the dispatched-compute list make_game.py --compile-shaders all|most|later writes compile_shaders= into hookshot_options.ini (the key the game's menu writes; none given: left as it is, and the game's default is Most) and skips the build's shader step on Later. The WPF, Qt and Tk builders get a Compile shaders box in place of a tick, saved with their settings and passed on builds and updates. assets/computes_dispatched.txt: the 622 compute programs seen dispatched in 53 local play and test folders, by hash only. make_game.py writes it beside the game as hookshot_computes_shipped.bin; release packaging carries it with the pipeline state list. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Joey-Magnus
reviewed
Nov 11, 2026
Joey-Magnus
left a comment
Contributor
There was a problem hiding this comment.
The description still has unfilled template placeholders (MEASURED_TABLE, VERIFIED_SUITE, VERIFIED_RUNS, NOT_TESTED_EXTRA). The new defaults (Most, Wait briefly) change behaviour for everyone across 42 files, so those numbers matter for review.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e shaders) - A dispatch waits main's 50 ms for a compile, never more; a compute variant not translated yet (Most and Later leave the unlisted ones to the background) follows When a shader isn't ready and Wait briefly's frame budget instead of an open-ended wait. All waits as main did. - The background threads run below normal, not idle (they take play's program-cache, layout and notes locks for one record or call at a time; an idle holder could be starved for seconds), at most 6; the layout lock is only taken while the background runs; the game's thread only posts its area, the prefetch thread does the rest. - The gate is checked before each compute variant's compile and each predicted or noted pipeline, not once per program; a slot is taken under the same lock a checkpoint pauses with. - The start's pipeline cache is read for a save only by the background's own saves, with its workers paused (also on the save's own thread). Checkpoints every 5 minutes and only after 256 new pipelines; the position is recorded only after the save, and a stop records nothing new. - A new GPU driver brings the predicted pipelines back on Most; a missing record on Most means the parts Most leaves are done again (All now records that everything is done). - Resume positions carry their list's hash: a list that grew starts its part again. - A prediction that gave up on the driver is not recorded done. - The hold of a load's own frames is capped like an area's hold; held frames are paced at the cap's frame time; the unused area_loaded and the unshipped areas list are gone. - At a close, a worker still in the driver after 20 s ends the process without tearing the device down under it. - When a shader isn't ready defaults to Skip (main's) on All when it was never chosen. - The builders pass --compile-shaders only when it was chosen there, so a rebuild keeps the game menu's choice. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Nov 12, 2026
hckrman101
pushed a commit
that referenced
this pull request
Nov 12, 2026
) **What this is for:** contributors rebuilding from source should not spend another minute drawing the same button prompts. **What is measured:** on an Intel Core i9-13900K running Arch Linux, the finish stage at main `c544828f` took 67.707 and 66.898 seconds; two unchanged runs took 0.603 and 0.651 seconds, and the final head took 0.616 seconds, clearing the run-time files as well. Other work was running throughout. **Not measured:** Windows, a packaged Builder update, a complete compile/lift, or gameplay performance. Refs #2636. ## Problem `nx2/make_game.py` removes the game folder's `prompts` directory and runs `prompt_assets.py` on every build when fontTools is available. Most source edits do not change anything those assets depend on. This avoids drawing unchanged assets; it does not make the drawing itself faster. #2636's A1 plans to measure the finish stage, but its A1–A10 items do not already propose this prompt cache. It complements #2674 and #2675, which address unnecessary retranslation rather than prompt generation. ## What changed - **`nx2/make_game.py`:** source-checkout builds request incremental prompt generation. Packaged builders still regenerate every time, using the finish stage's existing launcher distinction. `RX_RELEASE` cannot distinguish them: make_game sets it for source builds too. Missing fontTools still removes the replacement prompts, keeping the existing fallback to the game's icons. - **`nx2/prompt_assets.py`:** fingerprint the inputs before generating, draw into a fresh sibling directory, and record the successful outputs only if a second fingerprint still matches. Reuse requires both matching inputs and every recorded output's size and SHA-256. **A failed generation still removes the prompts, as before**, so the game shows its own icons. An interrupted builder can leave the previous complete set without a stamp; the next build regenerates it. The drawing, encoding and asset formats are unchanged. - **`tests/test_prompt_skip.py`:** game-free fixtures run the actual finish-stage prompt call and production cache/publication functions, with small authored files and a stand-in for expensive drawing. - **`docs/development.md`:** explain reuse, its limits, and `HOOKSHOT_FORCE_PROMPTS=1`. The force switch follows `env_on`: unset, empty, `0`, `false`, `off` and `no` are off, ignoring case and surrounding spaces. ### What prevents a wrong skip The input record covers: - The four prompt source modules, including controller, keyboard and binding tables. - The game's icon font and optional bold label font, compression dictionary, common texture/layout archives, and matching resource-size table names and contents. - The repository's controller and keyboard fonts, and added, changed or removed PNG overrides. - Python/platform; fontTools, Pillow, ASTC encoder, texture decoder, NumPy, zstandard and archspec availability and versions; Pillow's native library versions and the selected ASTC backend. - Generation parameters and relevant library switches, including `SOURCE_DATE_EPOCH`, `FREETYPE_PROPERTIES` and the encoder's custom CPU definitions. Languages and saved bindings are not build inputs here: the generator uses its source tables, and the runtime applies the selected controls. The output record includes files **and directories**: the keyboard directory itself is a runtime marker. Legitimately absent optional outputs stay absent; a missing, truncated or same-size-corrupted recorded file forces regeneration. The runtime's own compressed copies and control-specific keys are excluded before hashing, not mistaken for generator output. **The files the game derives at run time are still cleared by every build**, including an unchanged hit: their own keys do not cover changes to `host/prompts.c`. Only the known runtime filenames are removed; if one cannot be cleared, the build regenerates instead of keeping on doubt. An unreadable input cannot authorize reuse. The log gives the reason, even if a forced regeneration subsequently fails. A source edited during generation leaves no stamp, addressing the stale-input certification problem discussed in [#2674's review](#2674 (comment)). ## How it was tested 2026-11-12; Arch Linux, kernel `7.2.15-1-omarchy`, x86-64, Intel Core i9-13900K, 32 logical CPUs, Python 3.12.16. Base `c544828ff`; final head `2ec89de8`. No game or GPU program was launched for these checks. ### Finish-stage measurements These execute the real finish-stage statements through completion, including binary/image copies, the unchanged shader-list check, pipeline-list work and prompt generation, in a disposable copy of an existing built game folder. Compilation, translation and shader preparation are not included. They are **not whole rebuild times**. All optional prompt packages were present: fontTools 4.68.0, Pillow 13.0.0, astc-encoder-py 0.1.12, texture2ddecoder 1.0.6, archspec 0.2.7, NumPy 2.5.4 and zstandard 0.25.0. | Run | Source | Finish time | 1-minute load before → after | Top other work | |---|---|---:|---:|---| | Before 1 | `c544828f` | 67.707 s | 7.13 → 7.40 | Existing game 316 → 315% CPU; file indexing 66 → 79%; relay check also active | | Before 2 | `c544828f` | 66.898 s | 7.06 → 8.24 | Existing game 315 → 318%; terminal process about 41% | | First generation with cache | `63d5c965` | 66.400 s | 8.24 → 8.19 | Existing game 318 → 320%; terminal process about 42% | | Unchanged 1 | `63d5c965` | 0.603 s | 8.19 → 8.19 | Existing game 320%; terminal process about 42% | | Unchanged 2 | `63d5c965` | 0.651 s | 8.19 → 8.19 | Existing game 320%; terminal process about 42% | | Changed table source | `681208dd` | 63.147 s | 6.99 → 6.95 | Existing game 320 → 321%; terminal process about 42% | | Regenerate, final head | `2ec89de8` | 65.007 s | 5.77 → 5.86 | Existing game 317 → 316%; terminal process about 43% | | Unchanged, final head, with run-time files present | `2ec89de8` | 0.616 s | 5.79 → 5.79 | Existing game 316%; terminal process about 43% | Other work was running; none was a quiet measurement. CPU percentages are lifetime averages from snapshots, not interval measurements. The first generation is included to show that this is a skip, not a faster drawing algorithm. The unchanged runs print: ```text prompts: unchanged, kept ``` Adding a harmless comment to the button-table source forces a fresh generation: ```text prompts: regenerated (table changed) ``` The comment was removed afterwards. A timestamp-only touch does not regenerate: the key checks content. An additional run on the final head verifies an unchanged hit with all six runtime-derived files present: all six are removed, while the 16 generated originals and the stamp retain their sizes and SHA-256 values. The focused fixture also compares the retained originals byte for byte. The final timing rows above include that cleanup. **Asset parity:** all 16 regenerated files, 274,704,680 bytes, compare byte for byte with a full generation by main `c544828f`. Only the new stamp is excluded. Both sides used the same `SOURCE_DATE_EPOCH` because fontTools otherwise writes the current time into the OpenType header. This comparison setting is not introduced into normal builds. A development cache hit retains the earlier font timestamp; packaged builds retain their old full-generation behavior. ### Game-free checks - **23 focused tests pass** on the final change. They cover unchanged reuse, every input kind, package versions/presence, switch/parameter changes, missing/truncated/same-size-corrupted output, empty directories, clearing runtime derivatives while retaining byte-identical originals, regeneration when runtime cleanup fails, interruption, mid-generation edits, unreadable input, force-switch parsing, incomplete output, publication failure, and the packaged/no-fontTools paths. A non-zero generator result removes prompts for both source and packaged builds; interruption before that result leaves the old set unstamped. - The same unchanged-build test against main's finish stage fails behaviorally: `2 != 1: unchanged finish must not draw the prompts twice`. - Removing each of **18 guards/key components** in memory makes its corresponding behavior test fail on the final change. Three later corrections also fail against the earlier version of this patch: returned failure retained old prompts in both build paths, runtime files survived a hit, and an uncleared runtime file did not force regeneration. The unreadable-input diagnostic has a separate before-fails/after-passes proof. - The whole suite one commit earlier, at `f20ea58b` (before a failed generation was made to remove the prompts again and the run-time files to be cleared), `python -I -B tests/run_all.py --jobs 8`: ```text 472 passed, 0 failed, 20 skipped in 133 s (linux, 8 at once) ``` Load was 7.37 → 19.48. The first full run, at 32 checks at once, was **471 passed, 1 failed, 20 skipped**: a clang++ child compiling `texture_pass.cpp` for `nx2/test_depth_sample_types.py` died with SIGKILL. That fixture and its input trees are unchanged from main. No kernel OOM line was found, so the cause is not established. The complete run with eight workers passes, including that check. The failed run is retained, not counted as a pass. On the final head `2ec89de8` the focused checks, the baseline failure and all guard mutations were rerun, and the whole suite once more with eight workers: **471 passed, 1 failed, 20 skipped in 136 s** (load 11.14 → 22.62). The one failure is `tests/test_release_scripts.py`, `test_nightly_two_runs_at_once_end_at_the_newer_commit`, the intermittent race reported in #2626; this change touches none of its files. - No core dumps were recorded. Nothing generated from the game is in this change. ## Not tested - Windows filesystem behavior or a Windows build. - The packaged Windows Builder or Linux AppImage build/update end to end. Their always-regenerate selection is covered by a game-free test only. - A full compile/lift or a new gameplay run, other CPUs, or gameplay/frame-rate benefits. No screenshots: the regenerated asset bytes match, and no visual change is intended. ## Risks The input list must grow when the generator gains new file dependencies or outputs outside the prompt directory. A regression flags new local module imports, but it does not replace reviewing new file reads. Generation temporarily needs room for both old and new assets. Publication uses completed directories and rollback on a rename error; interruption cannot create a stamped half-set, though a process killed between the two directory renames may require regeneration on the next build. ## Overlaps The open-PR scan found 47 PRs touching one of these files. The actual conflict hunks were read: - **#2151:** adjacent item-button generation conflicts with the finish-stage prompt call/error branch. Preserve both steps and account for added prompt dependencies. - **#2261:** substantive overlap. It adds `--wheel-icons`, another input folder and output outside `prompts`. Keep that work outside this skip or extend the input/output proof and staging; accepting both hunks mechanically is not enough. - #1872, #1958, #2304, #2317, #2430, #2465, #2474, #2478, #2608, #2673, #2675 and #2686 share code files but merged cleanly in the scan. That is not proof that any newly added generator dependency is covered. Assisted-by: GPT-6 Astra Co-Authored-By: GPT-6 Astra <noreply@openai.com> Co-authored-by: GPT-6 Astra <noreply@openai.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
A first start today prepares every shader before the title: about 11 minutes on an RTX 3090 with a 12-core CPU, longer on smaller PCs. This adds two settings so a player can start sooner, and lets the rest of the work happen while they play.
Compile shaders (builders and the Insert menu's Graphics tab, applies after a restart):
On Most and Later the background work runs until the folder has what All gives it. It saves where it got to, carries on at the next start, then stops until an update changes the shaders. While it runs, a small "Shaders: NN%" counter shows in the corner, and the log says when it is done.
When a shader isn't ready (Insert menu, changes at once):
Under every option, a draw that writes the game's memory, or draws into a picture made in the same frame, always waits. Leaving one of those out would leave the game's state wrong or the picture empty. The menu help says so. This is the same rule
vk_draw.cppuses today (RX_VK_PIPE_SKIP=0still means wait for everything).How the background work stays out of the way
mkw_vk_pipelines.bin), not play's tables.hookshot_prepare_play.txt).hookshot_prepared.bin) is only written once everything is done. A start that ends halfway still knows it has work left.The dispatched-compute list
assets/computes_dispatched.txt: 622 of the game's 4,127 compute programs, by hash only (no game code or data). They are the union ofhookshot_computes.binfrom 53 local play and test folders.make_game.pywrites it beside the game ashookshot_computes_shipped.bin. The release packaging copies it with the pipeline state list.Builders
The WPF, Qt and Tk builders get a Compile shaders box in place of a tick.
make_game.py --compile-shaders all|most|laterwritescompile_shaders=intohookshot_options.ini, which is the same key the game's menu writes. Any launcher can pass the flag or write the key; without either, the game uses Most. On Later the build skips its own shader step.Experimental, and the trade-off
Most and Later trade a shorter first start for work during play. Until the background finishes, some objects may appear a moment late, and an area's first frames may be held briefly. All stays available and behaves as before.
Measured
(RTX 3090, 12-core/24-thread CPU, Linux, headless, the game's own frames)
MEASURED_TABLE
Verified
tests/test_prepare_play.py(the rules compiled and run: choices and defaults, thread counts, the wait decision for every option including memory-writing draws, the frame budget, the hold, the resume record, the percentage; and the plumbing in the back end, menu,make_game.pyand the three builders).tests/test_linux_builder_jobs.py, and the updated fixtures of the tests that compile parts ofvk_draw.cpp/vk_main.cpp/the menu. Full suite: VERIFIED_SUITENot tested
Overlaps with open PRs
draw()(may_skipbecomeswrites_memory(p)). Whichever lands second has a small, mechanical rebase; the rule stays the same.Stage::module. The background's compute warm-up now goes through one function,warm_compute_one, shared withprepare_run, so perf(vk): keep program stages' SPIR-V packed, make shader modules only per pipeline (~1.5 GB less RAM on NVIDIA) #2309 only has one place to change.--compile-shadersflag or thecompile_shaders=key, so the launcher's Settings can offer it the same way.