Skip to content

perf(build): lifted chunks of ~4 MB, by guest address, besides chunks.txt's boundaries - #2890

Open
jordanella wants to merge 1 commit into
NotRexed:mainfrom
jordanella:perf/smaller-chunks
Open

jordanella wants to merge 1 commit into
NotRexed:mainfrom
jordanella:perf/smaller-chunks

Conversation

@jordanella

Copy link
Copy Markdown
Contributor

A small update that changes a few of the game's functions still recompiles every lifted chunk that holds one, and a chunk is ~24 MB of C: a typical one takes ~227 s of compiler time, and the few heavy ones far more. Splitting the lifted C into ~4 MB pieces by guest address makes that ~41 s, with no cost in play or in a full build. Part of the dev-loop proposal's work on small updates (#2636).

Change

nx2/chunks.py:

  • Sub-boundaries: besides image/chunks.txt's stable boundaries, sub_starts adds one every SUB_STRIDE (0x40000) of guest address, at the first function at or after it. They are guest addresses, so they stay put when a function grows: a change moves no boundary but its own chunk's.
  • Not saved: chunks.txt keeps only the boundaries it had (the sub-boundaries are worked out again each lift from the same functions), so dropping this change later is clean.
  • A guard: more than 1000 chunks stops the lift with a message, since split_big, split_hot and build.py match three-digit chunk names.

The first build after this compiles every lifted chunk once (all boundaries are new): for a player, that update takes about as long as a full build, once. Updates after it recompile only the ~4 MB pieces that changed.

Measured

Windows 11, Ryzen 9 3900X:

  • One chunk's compile: a typical 24 MB chunk ~227 s of compiler time against ~41 s for a 4 MB piece. The heavy-function cluster (old chunks 021-023) still takes 300-900 s per piece.
  • A full build with my local build changes together (this, the lift key's environment, link skips and others), each in a fresh player-like work folder: 36:17 on main against 32:30, at the same total compiler time (25,700 against 25,500 s) over 698 files instead of 486. The smaller pieces spread better over the cores.
  • A two-PR update (fix(vi): the time of day passes again above a 60 fps cap #2093 and feat: add separate button mappings for the ocarina notes #1930, the same setup): 6:06 against 4:36; compiling 2:16 against 1:00.

In the game

No change in play: the 4 MB-chunk game against the 24 MB one, Kokiri Forest from a save (the game logic bounds its frame on the CPU there), RTX 3080 (NVIDIA 621.14), Windows 11, two interleaved pairs of 30 samples: median 29.6 / 30.0 fps against 31.0 / 30.0. No errors in either.

Testing

  • tests/test_sub_chunks.py (new): the first function past each stride; a long function spans strides without a boundary inside it; a function found since moves one boundary at most; chunks end at them as well as at the saved boundaries; write uses them and saves only chunks.txt's; the stride makes chunks of a few MB. (The 1000-chunk guard isn't tested: the game's code makes ~330 pieces.)
  • Every check that reads the changed files (9), through run_all.py, rebased on today's main (with perf(lift): emit the lifted C in several processes on Windows too #2748's spawned emit), Windows: 8 pass; test_spawn_emit fails the same way on main here.
  • Not tested: Linux.

See #2636.

🤖 Generated with Claude Code

….txt's boundaries

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant