Skip to content

perf(build): skip unchanged links, write compile_commands.json, key the hot pieces by the profile's contents - #2893

Open
jordanella wants to merge 1 commit into
NotRexed:mainfrom
jordanella:perf/link-skip
Open

jordanella wants to merge 1 commit into
NotRexed:mainfrom
jordanella:perf/link-skip

Conversation

@jordanella

Copy link
Copy Markdown
Contributor

A build with nothing to compile still relinked the 440 MB executable and the renderer, a build from a second checkout of the same tree recompiled all eight hot pieces (~10-20 minutes) because their keys named the play profile by its path, and clangd had no compile database. Three small changes to nx2/build.py from the dev-loop proposal (#2636, A4 and A8); one PR because they touch the same lines, and I can split them if you prefer.

Change

nx2/build.py:

  • A4, no link when nothing changed: run_link skips a link whose inputs are the last successful one's. Its key is kept beside the output (<output>.linkkey) with the output's size:

    • the command, the compiler and the work folder's system-headers stamp;
    • each object's key (its .o.sha1; its size and time without one);
    • the contents of the other files it names, inside options too (the hot symbol order, the libraries, the resources).

    A failed link, or an output of another size or gone, links again. The executable's, the renderer's and the game DLL's links go through it.

  • A8, compile_commands.json: every compile as compile_one runs it, written to the build folder for clangd and clang-tidy (compile_commands.json at the checkout's top is git-ignored, for a copy there).

  • The hot pieces' keys name the profile by its contents: flag_text leaves -fprofile-instr-use=<path> out of the key, since -DHOOKSHOT_PROFILE_SHA1 beside it already says what the key needs of the profile. This changes their keys once.

Measured

Windows 11, Ryzen 9 3900X, a dev work folder:

  • A build with nothing to do: 16 s against 6 s, both links skipped.
  • One renderer edit: only the renderer's link runs.
  • A second checkout of the same tree building in the same work folder: the eight hot pieces from the cache, where main compiled them again.

In the game

Every build I ran in the game today used this build.py. The latest: Windows 11, RTX 3080 (NVIDIA 621.14), a dev-profile build, from a save, headless through the control bridge: in play in Kokiri Forest, 28-31 fps, nothing in the log about a lost device or a crash. A skipped link leaves the same binary in place.

Testing

  • tests/test_link_skip.py (new): a stand-in linker through the real run_link: the same inputs aren't linked again; a changed object, library or symbol order is; so is an output of another size or gone; a failed link forgets its key; the files inside options are inputs; the compile commands are the compiles; the build's links go through it.
  • tests/test_profile_path_key.py (new): two paths, one profile give one key; another profile, another key; every use of the profile's path names its contents too.
  • Every check that reads the changed files (37), through run_all.py, rebased on today's main, Windows: pass, apart from four that fail the same way on main here (test_driver_cache_folder, test_fpx_sticky, test_linux_update, nx2/test_dll.py). Rebasing found that main's build.py doesn't import re (perf(build): a renderer header edit compiles the files that read it, not the whole back end #2706 adds it); this PR imports it too.
  • Not tested: Linux.

See #2636 (A4, A8).

🤖 Generated with Claude Code

…he hot pieces by the profile's contents

Three changes to nx2/build.py:

- A4: a link whose inputs are the last successful one's is not run again
  (run_link). Its key, beside the output as <output>.linkkey: the command,
  the compiler and the work folder's system-headers stamp, each object's
  key (.o.sha1), and the contents of the other files it names, inside
  options too (the hot symbol order). A build with nothing to do went from
  16 s to 6 s on Windows. Players' builds let sa/build go, so it helps a
  work folder that keeps its files (the checkout's work/,
  HOOKSHOT_KEEP_BUILD_FILES=1).
- A8: <build>/compile_commands.json, every compile as compile_one runs it,
  for clangd and clang-tidy (git-ignored at the checkout's top).
- The hot pieces' keys name the play profile by its contents
  (-DHOOKSHOT_PROFILE_SHA1), not its path: a build from a checkout other
  than the work folder's compiled all eight again (~10-20 minutes). This
  changes their keys once.

tests/test_link_skip.py (a stand-in linker: skipped, relinked on a changed
object, library, symbol order or output, a failed link forgets the key;
the compile commands) and tests/test_profile_path_key.py.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant