[hckrman101]

Rebuilding the game spends most of a warm build in the lift (translating the game to C), and most of a cold build in the eight hot pieces. This PR makes both shorter. The game's code comes out the same.

What changes

1. A player's update skips the lift when the tools did not change (make_game.keep_translation / kept_before, build.py KEPT).
   • Before, the clean-up after a player's build deleted all of sa/, including the 1.5 GB of lifted C. So every update lifted again (1-3 minutes), even when nothing the lift reads had changed.
   • Now the clean-up deletes only the lifted code_*.c files and the build outputs. It keeps:
     • the rest of the lift (~124 MB: headers, runtime tables, image);
     • a list that build.py writes after the compile: each lifted file's object key, its size, and a hash of its flags and headers (gen/lifted_objects.txt). Its first line names the lift key it was written for and a format number; a new lift deletes the list of the one before.
   • The lift key in .lift_key gets a third line, kept.
   • The next build with the same lift key, and a list naming that key, skips the lift. make_game passes the list as build.py --kept-lift. build.py then takes the lifted files' objects from the object cache by the keys in that list. These are the same objects as before, so nothing is compiled again.
   • If an object is missing, or the flags or headers changed, build.py stops before compiling anything (exit status 75, KEPT_STALE; not 3, which is abort()'s on Windows). make_game then lifts again and compiles as usual.
   • If the clean-up cannot remove a lifted file (Windows: one still held open), the build still succeeds: it lets sa/ go as far as it can, and the next update lifts again.
   • The two builder apps' after-build clean-up (Storage.Free(afterBuild) / core.free(after_build=True)) also deleted sa/. It now leaves a kept lift in place. "Free up space" still deletes it.
2. The big chunks are split in one process per file (chunks.split_big, using the same worker pool as the edits). Before, it read all 1.5 GB of chunks on one core.
3. The runtime tables and standalone files are written beside the edits. These are a64_rtgen, hos_image.c, the import stubs and nvn2_exports.c. They read the image, not the lifted C, so a forked process now makes them while the C is being edited. Without fork (Windows), or if that process fails, they are made after the edits, as before.
4. The shader list is made again only when its inputs changed (make_game.shader_list_current).
   • Before, every build ran shader_census.py (about 4 s) and rewrote the 514 MB hookshot_shaders.bin.
   • Now a note in the work folder records what the list was made from: the script, the Python, the names, sizes and times of the files the census reads, and the list file itself. If none of these changed, the census is skipped.
5. The Windows setup parts are zipped at the same time (scripts/make_setup_parts.py), one thread each, as the Linux make_parts.py has done since ci: cache the AppImage's back-end objects for pull requests, and a lighter deps cache #2079. The zips come out byte for byte the same.
6. (Dropped: compiling the hot pieces without loop unrolling. It was in this PR and is reverted by a later commit; see "Not in this PR".)
7. The hot pieces of the last lift are kept (split_hot.py).
   • Before, split_hot dealt the hot functions out into eight pieces from scratch at every lift. So a few bytes more in one hot function (a patch) moved functions between most of the pieces, and every player compiled all eight again. That is the longest third of a cold compile.
   • Now the assignment is kept in <work>/image/hot_pieces.txt, next to the chunk boundaries:
     • functions from the last lift stay in their piece;
     • a new function goes to the piece of the function it shares the most calls with;
     • a fresh split is made when the kept split's longest piece would cost more than 15% over a fresh split's.
   • Each piece now declares only the functions and host calls it uses. A new declaration therefore no longer changes every piece.
   • The first split after this PR puts the same functions in each piece as before.
   • hot_pieces.txt is part of the lift key, as chunks.txt is: the pieces' C follows it.
8. On Windows, the lift's per-file passes run in several processes (chunks.file_jobs, _each). Before, they used forked workers only, so without fork (Windows) they ran in one process. Their workers take a file's path and read the file themselves, so without fork they now start as new processes (spawn). The emit still runs in one process there. HOOKSHOT_LIFT_NO_FORK=1 takes this path on Linux, for testing.
9. Earlier in this PR (first pass):
   • the lift's edits run in one process per file;
   • the lift prints a timing line for each step;
   • the lift key covers only the modules the lift imports;
   • a compile that failed before the link keeps the objects it already made.

Measured

Setup: Linux, Ryzen 9 7900X (24 threads), 30 GB RAM, other jobs running on the PC.

• Timed runs used oot-status --lock --timing, which gives the run its own core complex (6 cores / 12 threads).
• Old and new were alternated.
• The bundled clang 23 and Python were used.

Player update, same tools (make_game.py --nup, no HOOKSHOT_KEEP_BUILD_FILES, objects cached, shader prepare skipped). Times are from start to @@done:

	update 1	update 2
before (lifts each time)	100 s (lift 83 s)	193 s (lift 175 s, PC busier)
after (lift kept)	9 s	7 s

Caveat: these four runs went through make_game.py under oot-status --timing. A queue bug that night moved make_game.py / build.py processes onto the busy build CPUs during timed runs, so the absolute seconds of both rows are inflated by load. The gap does not come from load, though: the "before" runs spent 83 / 175 s in a lift that the "after" runs skip entirely. The lift-step, hot-piece and game-speed numbers below ran lift.py, clang or the game directly, and the bug did not affect them.

A second pair after the queue bug was fixed (--timing --cpu-only, one round, the PC heavily loaded and the builds at idle priority, so both absolute times are high again): make_game took 640 s before (lift 614 s: discovery alone 162 s, against 26 s on a quiet PC) and 42 s after (lift skipped; the rest is the finish stage's file copies).

• After a full build at the same commit, a kept update produced the same hookshot, hookshot_vk.so and image.bin (sha1).
• Fallback test: I deleted one kept object (code_010) and ran make_game again. It printed "the translation kept from the last build does not fit this one (1 of 80 files: code_010.c): translating the game again", lifted, compiled that one file, and produced the same hookshot sha1 as the full build.
• The kept files take 124 MB in the work folder, on top of the objects.

Lift steps (lift.py + split_hot.py run alone, two rounds each). All 95 output files and chunks.txt were sha1-identical to the old lift:

step	old (round 1 / 2)	new (round 1 / 2)
big chunks split	6.0 / 5.6 s	2.3 / 2.1 s
patches	2.7 / 2.6 s	3.5 / 3.1 s
output size + page-mark groups	5.7 / 5.7 s	6.0 / 6.2 s
runtime tables + standalone files	4.3 / 4.0 s	0.0 s (beside the edits)
these four steps	18.7 / 17.9 s	11.8 / 11.4 s

Shader list: the finish stage took 1 s when the list was kept, against 4-7 s when it was made.

Windows setup parts: I could not run a real package here. On a 104 MB stand-in package (three parts), three rounds took 1.81 / 1.50 / 1.50 s before and 1.05 / 1.15 / 1.04 s after. The zips and hookshot-setup.txt were identical.

Windows path for the per-file passes: measured on Linux with HOOKSHOT_LIFT_NO_FORK=1, against the old one-process lift (HOOKSHOT_LIFT_JOBS=1), two rounds. The three per-file passes (big chunk split, patches, output size + page-mark groups) took 9.9 / 16.1 s, against 40.6 / 35.6 s. The non-hot lifted files were byte-identical. The hot pieces differ only because of item 7. The whole lift's time varied more than that, because the one-process emit took 56-84 s from run to run.

Hot pieces kept: this was measured without a game run, on the game's real code_hot.c (849 functions). In 12 trials, each adding 1-60 lines to one random hot function, the next split changed 5.42 of the 8 pieces on average before (1-6 per trial) and exactly 1 after. A fresh split put the same functions in each piece as the old code.

Tried and dropped:

• A faster discovery walk in the lifter (reading the image through a 32-bit view). The output was identical, but discovery ended at 21 vs 23 s in one round and 26 vs 25 s in the other: no clear gain.
• The coverage report in a side process during the emit. The output was identical, but over four rounds the lift took 92.5 / 61.7 / 82.0 / 68.4 s against 71.6 / 72.4 / 88.4 / 69.8 s without it, a mean change of +0.6 s. The emit slowed by about what the report saved.
• Writing the lifter's small constant-count loops (NEON permutes and lane loops) out as straight-line C in the emitter. The instruction test gave the same results as before (Unicorn compare, 41,552 cases, 744 of 749 forms pass, and the same 5 forms fail without the change). But one hot piece then compiled for over 16 minutes against about 3, so the idea was dropped.

Full build of the branch (before merging main, still with the dropped item 6): lift 94 s, 9 of 264 files compiled (the eight hot pieces, 165-363 s each, and one more), then a kept update from it as above.

Merge with main: main removed oot_native_resolution.py (#2042) and added patches to oot_patches.apply (the text hooks of #2366, Camera speed of #1945). They are in the per-file patch_file / report split now; the ripple fix of #2409 goes through the frame-time table as before. A full build of the merge booted from the test save into Kokiri Forest and played 20 s of scripted running and sword swings (headless) without an error. It lifted with no patch warnings: every patch found once, the camera speed at its 4 functions, the three text hooks, both step divides and both frame counters (the ripple one included). 3 files compiled.

Tests: tests/run_all.py at the head (the merge and the review fixes): 402 passed, 0 failed, 12 skipped. New or changed for the fixes: a kept list of another lift or format is not used, a clean-up that cannot remove a file is no failed build, the kept hot pieces change the lift key, and no module of the lift imports by a name made at run time. Before the merge: 364 passed, 1 failed (test_wayland_libdecor_hint, a harness -Werror warning that also fails without this PR). New tests: test_kept_translation.py, test_shader_list_skip.py and test_sticky_hot_pieces.py. Updated tests that cut functions out of build.py and make_game.py: test_lift_cleanup_error.py and test_compile_retry_jobs.py.

Not tested

• Windows, for all of it (CI runs the Windows test suite; two of its tests failed on this branch at first, line endings and a missing lz4, and were fixed):
  • the kept lift on Windows, and the C# builder change (Storage.cs): no .NET SDK here, only the Python builder's twin was run;
  • the Windows setup zips with a real package;
  • the lift's per-file passes in spawned processes on Windows (item 8: only the Linux spawn path was run), and the side process for the tables there (it falls back to one process).
• While checking this I noticed that the play profile is out of date for this lift: clang ignored the profile for 58 of the 134 functions in code_hot_1.c (-Wprofile-instr-out-of-date). This happens without this PR too.
• A kept lift across a real release-to-release update. It was tested with two builds of the same tools, and with a deleted object for the fallback.
• Only one round of the re-timed update (above); the first pair is in the table.
• A full build of the last head, with the review fixes (unrolling back, the kept list's header, the hot pieces in the lift key): the merge just before them was built and run, the fixes only by the game-free tests. With unrolling back, the hot pieces compile with main's flags again.

Not in this PR (measured or written, for later)

• Compiling the hot pieces without loop unrolling (-fno-unroll-loops). It cut the hot pieces' compile a lot (code_hot_4.c 261 → 216 s alone; all eight 4547 → 2807 s of CPU in a loaded build), and Kokiri Forest ran as fast (62.8/64.8 fps before, 65.1/65.2 after). But a review found that it also stops clang from fully unrolling the lifter's ~25,000 fixed-count lane loops (for (_i < 4) and the like): on six lane-heavy functions of code_hot_5.c the loops left in the code went from 65 to 263 and the stack accesses from 744 to 1165. The scenes measured barely run those functions, so they could not show it. Reverted. A narrower set (-mllvm -unroll-runtime=false -mllvm -unroll-allow-partial=false, perhaps -mllvm -unroll-max-upperbound=0) gave code byte-identical to today's on that sample; its compile-time gain with the play profile is not measured yet.

• Smaller chunks of ordinary code: three 24 MB chunks compiled whole took 145 s of CPU and 1.4-1.6 GB each; the same code cut in thirds (at function starts, as split_big cuts) took 117 s (-19%) and 0.52-0.70 GB each. More compilers would fit on a 16 GB PC, and less CPU on any PC. It changes every ordinary chunk's object (one full compile for everyone) and needs a game-speed check, since fewer callees share a file.

• Unpacking the romfs while the game is translated and compiled (about as long as the unpack, ~1-2 minutes, off a first build): written and tested game-free, not measured end to end.

[Arkex-ai]

There is a cleanup integration point with #2367 (portable Builder): both touch Storage.cs, and your kept-translation marker changes what after-build cleanup may remove. When combining them, Storage.Free(afterBuild) should keep the marked lift in the selected portable work home; explicit Free up space should retain your current semantics. My #2250 separately scopes stale-process cleanup to proven work/output paths, so retaining a translation must not broaden which processes cleanup may terminate. These look complementary rather than duplicate work; could we preserve those two contracts when rebasing whichever lands second?