[jordanella]

Editing any renderer header recompiles the whole back end: on Windows a comment in vk_present_rect.h, which two files include, compiled 35 files again (417 s of compiler time; a 44 s build against 26 s for a .cpp edit, and minutes on a 4–8 thread PC). This is the dev-loop proposal's A6 (#2636), validated by hckrman101's Linux numbers there (16 objects for one header).

Change

• Depfiles: the back end's own files (gpu/vk/*.cpp) are compiled with -MD, and clang writes a depfile beside each object (vk_x.d).
• Narrower keys: the next build keys each object by the headers its depfile lists under the same folders as before (gpu/vk, gpu/render, gpu/compat, suyu's texture decoders), by name and content (renderer_deps). A header the file doesn't read no longer changes its key; one it reads, or one that's gone, does.
• The blob stays as the fallback: for a file with no depfile yet, or one whose depfile names nothing under those folders.
• No forced recompiles: flag_text leaves -MD out, so no key changes and nothing compiles again for this change itself. Depfiles appear as files are compiled.
• Re-keying: an object compiled now is cached and stamped under the key its new depfile gives (rekey_depfiles), so the next build doesn't compile it once more. Only an object of this build, so made from this build's source and headers, is re-keyed.
• Cached by its own depfile only: an object whose planned key came from the last compile's depfile is neither cached nor stamped under that key (it hashes the new source with the old include list, so it doesn't vouch for a header read for the first time). compile_one stamps it depfile-pending, and rekey_depfiles stamps and caches it by its own depfile's key, also when that's the planned key. A build that stops before re-keying leaves it pending, and the next build compiles it again. (hckrman101's review: an include added, removed, the header edited, the include back, used to get the first object back from the cache.)
• Unchanged: Eden's and sirit's files keep their own headers' blob, and the host keeps its blob (it mixes in non-header inputs such as the FFmpeg version key, so it's left for a separate change).
• Same limit as make/ninja: a newly added header that shadows another earlier in the include path isn't seen until something the file reads changes.

Measured

On Windows (Ryzen 9 3900X, NVMe):

Build	Files compiled	Compiler time	Build
one comment in vk_present_rect.h, before	35	417 s	44 s
the same, with depfiles	4	20 s	27 s
the header reverted	0	0 s	18 s

The four are the files that read it: vk_framegen.cpp, vk_menu.cpp, vk_replay.cpp, and host/hos_main.c, through the host's own blob. Getting depfiles took one compile of each back-end file, done here by commenting them all. The re-keying then kept the reverted files from compiling a third time.

In the game

Windows 11, RTX 3080 (NVIDIA 621.14), from this branch's build.py (with the review fix) on main ee29dd5: a build with depfiles, then one with a comment added to vk_present_rect.h (5 files compiled: the four above, and vk_hangdump.cpp, which reads no repo header, so its depfile names nothing under those folders and it keeps the blob; no object left pending; with the comment taken out again, 0 files). That game, launched headless through the control bridge from a save: in play in Kokiri Forest, 27–34 fps over five samples, nothing in the log about a lost device or a crash.

Testing

• tests/test_renderer_depfiles.py (new, 11 cases):

  • the depfile read as clang 23 writes it on Windows (/ and \ in paths with spaces, CRLF continuations), plus $$ and \#;
  • -MD is in no key;
  • no depfile keeps the blob;
  • an unread header or one outside the folders changes nothing, while a read header or a gone one changes the key;
  • the re-keying, and that it leaves alone an object not of this build;
  • the wiring;
  • hckrman101's four steps, with a stand-in compiler that writes the headers it read into the object and a real depfile: on the previous commit step 4 got back step 2's object, built with the old b.h.

  It fails against main's build.py.

• test_compiler_failure.py and test_mem_gate_leases.py take DEPFILE_DEPS and DEPFILE_PENDING from build.py with compile_one.

• test_dlss_sdk_cache.py and test_shader_stamp_cache.py run the job loop with its new helpers. Both still take an older build.py (--build-script), and test_shader_stamp_cache passes against main's.

• Every check that reads build.py (39), on Windows, passes apart from test_driver_cache_folder, test_fpx_sticky and test_linux_update, which fail the same way on main here.

• Not tested: Linux. gcc and clang write the same make-style depfile.

🤖 Generated with Claude Code

[jordanella]

Thanks: that's a real hole, and your steps reproduce it exactly. Fixed in f0f345b:

• No caching under the planned key. compile_one no longer caches, or stamps, an object under a key that came from the last compile's depfile. It stamps it depfile-pending instead.
• rekey_depfiles does the keying. It stamps and caches the object by the key its own new depfile gives, also when that's the planned key. The cache now only ever holds blob keys (every header) or a compile's own depfile key.
• A stamp-only variant of the same hole. The old code also stamped the object with the lossy key. A build that stopped before re-keying (some other file failing) could leave that stamp for a later build whose depfile gives that key. A pending stamp never matches, so that build compiles the file again.
• test_renderer_depfiles.py walks your four steps with a stand-in compiler that writes its included headers into the "object" and a real depfile. On the previous commit it fails: step 4 got back step 2's object, built with the old b.h.

On running the game: with this branch's build.py (the fix included) on Windows 11 and an RTX 3080, I built once to get the depfiles, then built again after adding a comment to vk_present_rect.h. That compiled the four files plus vk_hangdump.cpp (no repo headers, so it keeps the blob), and left no object pending. The game from that build, launched headless from a save, plays in Kokiri Forest at 27-34 fps with nothing in the log about a lost device or a crash. With the comment taken out, the next build compiled nothing. That's in the description now.

test_compiler_failure.py and test_mem_gate_leases.py also cut compile_one out, so they now take the two new constants from build.py as well.

[Joey-Magnus]

hckrman101's stale-object hole is closed in f0f345b. compile_one stamps an object it compiled under a depfile key depfile-pending and doesn't cache it (nx2/build.py:952). rekey_depfiles then caches and stamps it under the key from its own new depfile, so the cache only holds blob keys or a compile's own depfile key. I traced the cache-hit path (compile_one restoring from OBJ_CACHE). A hit needs the planned key to equal a key computed from that object's own depfile, so the source and the set of hashed headers match, and leaving the old .d on disk is sound. A build that fails before rekey_depfiles leaves pending stamps, which never match, so those files compile again. I merged this into current main and ran the depfile, compile, mem_gate, dlss_sdk, shader_stamp and build checks: 31 passed. test_precompile_compute_guard and test_portable_local_builder also fail on main on this machine, so they're not from this PR. The in-game run after the header edit is what was asked for. The red appimage job timed out in the PPA apt step, which is infrastructure, so a re-run should clear it.

[mooki000]

Linux result on f0f345b6, against its main base 46780503: the identical header edit recompiles 5 files instead of 35, and both resulting games load Kokiri Forest and exit cleanly.

Setup: Arch Linux, kernel 7.2.15, i9-13900K, RTX 4090, NVIDIA 615.78.08, Clang 23.1.3 release builds with the same PGO profile and compatible translated inputs. Separate checkouts and game folders. After the initial builds, I added the same harmless comment to gpu/vk/vk_present_rect.h in each checkout and rebuilt through nx2/build.py.

Build	Files actually compiled after the edit
Base 46780503	35
PR f0f345b6	5

The five were vk_framegen.cpp, vk_menu.cpp, vk_replay.cpp, host/hos_main.c, and vk_hangdump.cpp. This matches your explanation: the host still uses its header group, and vk_hangdump.cpp has no repository-header dependency and keeps the fallback. Counts come from the build's own @@file ... new records, asserted by script. Other work was running, so I make no wall-time speed claim.

The stale-cache review fix also passes with real compilers. A separate check drives the actual cache/key/compile/rekey functions, adds an include, removes it, edits that header, and adds it back. It links and runs each resulting object. GCC 16.2 and Clang 23.1.3 both return the new value 50 at step 4 on f0f345b6; both return the stale value 30 on 632b55ac. This does not use the stand-in compiler from the PR's test.

Game check after the header edit: windowless, fresh copied save, 2048×1152 game output / 1920×1080 offscreen presentation, 60 cap, DLSS/frame generation off. Base: 37.9 seconds, 1,505 game-present rows, exit 0. PR: 34.3 seconds, 1,454 rows, exit 0. Both show Link, the forest and HUD; neither logs a crash, lost device or kernel driver fault. Load averages were 36.37→34.31 and 13.85→13.41, respectively: these are not frame-rate comparisons. The experimental comments were removed afterward; the retained tested binaries are explicitly the post-comment builds.

All six changed checks pass in both compiler environments. Against base source, the new depfile check reports one flag-key failure and ten missing-helper errors; the compile-failure/memory-gate adaptations reject missing constants. The existing compile-retry, DLSS-cache and shader-stamp checks pass on base too, as compatibility coverage. Full suite on the head: 473 passed, 0 failed, 20 skipped.

The hosted AppImage job separately timed out in System packages, exit 124, before compiling this change: https://github.com/NotRexed/Hookshot/actions/runs/43211349705/job/131159467435.

Not tested: Windows runtime, all possible include-path shadowing changes, other game scenes, or quiet repeated build timings. No new defect found in the exercised head.