[jordanella]

Today a one-line renderer or host change costs a relink of the 426 MB exe, a cold start and a trip back to the scene, and a stray RX_* switch or a failed build makes the next build translate the whole game again. This adds a proposal, docs/dev-loop-proposal.md, for making those edits cheap and predictable and the code easier to navigate, without changing what players build or run.

Decisions needed (top of the doc, each before its work starts):

1. What agents may read: never work/, single excerpts on request, or local models only. scripts/cpu_categories.py already treats "a map of the game's code" as local; the same question covers a hand-written table of game-function addresses in the repo.
2. Where the entry point lives: nx2/dev.py (proposed) or scripts/dev.py.
3. Where scenario files live: scenarios/ or tests/scenarios/.
4. Dev warnings and clang-tidy in CI: advisory or failing, given a baseline.
5. Whether a dev-only split of the host out of the exe is acceptable at all, if the measurements call for it.
6. Whether to make players' start-up update check asynchronous: a player-facing change, decided on its own.

What it plans, as small PRs behind a week of measurement (Phase 0):

• Build graph. Skip links whose inputs haven't changed. Use depfiles instead of header blobs. Make the lift key hash only the environment the lift reads. Don't relift after a compile error in a tracked source. Add compile_commands.json, and one nx2/dev.py whose status names why anything rebuilds.
• Middle loop. A relaunch command on the control bridge (feat(control): drive a running game over a local socket, with a CLI #926) with a staged DLL swap in dev builds, so a renderer edit reaches a scripted scene in one command. Built-in shaders loadable from disk. A live renderer swap is designed but deferred: the renderer has no teardown today.
• Testing. run_all.py --changed, one shared source-cutting helper in place of about 165 copies of block(), a sanitizer lane, scenario files on hookshot_ctl.py, and run summaries that can be compared.
• Diagnostics. One list of switches with a check, a header on captures, memory counters, and warnings and clang-tidy on our own code.
• Docs and code maps. A docs/README.md index, subsystem maps, every generated file naming its generator, a dev.py where repo map that tells people and agents which tests cut the code they're changing, and the data boundary for agents written into AGENTS.md.

How it scores. Under workflow.md's formula, contributor tooling is S3 with no player reach, so it shouldn't take a worklist row from anyone. The case for it is throughput. The three pieces that do reach players (the full rebuild after a failed link, the blocking update check, the single-process emit on Windows) get their own worklist rows, scored normally.

Where the facts come from. Read from the source at 7297d93d, partly by AI agents, and rechecked against main where the plan leans on it. #2324's kept lift and #2518's COMPILE_ONLY_ENV were already doing parts of it, and the doc builds on them rather than re-proposing them. Nothing was measured; timings are quoted from the PRs that measured them.

Testing. Docs only: one new file, no code changed. Every relative link and anchor in it resolves; tests/test_hookshot_ctl.py (the one check that reads docs/) passes. I didn't run the full run_all.py locally, since nothing it checks changed; CI runs it. No screenshots: nothing changes on screen.

🤖 Generated with Claude Code

[jordanella]

Thanks for this. It's exactly the data Phase 0 was missing. Pushed 3c32ad6:

• Decisions, recorded at the top as you settled them:
  1. Strict, including the address table.
  2. nx2/dev.py.
  3. tests/scenarios/, marked as needing the game.
  4. Advisory to start; a later PR of its own makes new findings in changed renderer files fail CI.
  5. Not now, pending Windows link times.
  6. Its own PR with a worklist row.
• New A10: the opt-in object store shared across work folders. It has both catches (seeding hot_funcs.txt/hot_order.txt/chunks.txt, and the PGO profile at a content-addressed path) and your numbers, credited to your setup.
• New D6: the lock so timing runs never share the machine with a build.
• A2, B1, D4/D5: changed as your inline comments asked (replies there). A4/A6/C2: your Linux data points are in; C2 is "not now".
• The length: agreed. §2 stays until F1's subsystem maps exist, then moves into them, and the proposal keeps only the plan. The status line says so.

Yes please to upstreaming the shared-store wrapper and the in-session A/B pieces as their own PRs.

We'll take Phase 0 on Windows: a no-op rebuild, a one-file renderer edit and a host header edit, with the link and compile times from today's output posted here, then A1 to cover the phases that output doesn't show. The first code PR follows from those numbers: A4 if Windows' links are slow, otherwise the rest of A7.

[jordanella]

Phase 0 on Windows, and the two fixes it turned up.

The setup: Ryzen 9 3900X (24 threads), 64 GB, NVMe work folder, main at 0ec31ff, llvm-mingw clang 23.1.5, a work folder kept between builds (--work = the checkout's work/), --no-prepare-shaders. The game folder was on a hard drive, which only slows the read stage.

Build	Total	Notes
Clean (new worktree)	2,265 s	lift 331 s (of it the emit: 170.5 s in 1 process), compile 1,282 s at 24 jobs, link 3 s (440 MB exe) + <1 s (DLL); the other ~650 s is reading the .nup onto the hard drive and finishing the game folder
No-op, the next build	319 s	translated the game again: #2674
One gpu/vk/vk_draw.cpp edit	33 s	compile 18 s, links 3 s
One host/hos.h edit	26 s	compile 11 s (the host blob, as §2.2 says)
A typo in host/hos_update.c	fails in 16 s
The build after the typo	301 s	translated the game again: #2675

What follows from these numbers:

• Windows links in seconds, as Linux does, so C2 stays dropped, and A4 is worth only the 3 s it saves. I've put the fixes first instead.
• fix(build): a new work folder's second build no longer translates the game again #2674: the lift key hashes image/hot_pieces.txt, which the first lift in a work folder writes, but the key was taken before that lift. Every new work folder's second build translated again, and so did a player's first update after a new install. With it: the second build takes 14 s instead of 319 s.
• fix(build): a compile error in a source file no longer translates the game again #2675 (A7): a compile error in a repository source no longer drops the translation. With it: the build after the typo takes 21 s instead of 301 s.
• A3: the emit is 145–170 s of a 280–330 s lift on Windows (perf(build): updates skip the lift, a faster lift, and hot pieces that compile sooner and stay put #2324 measured 56–84 s on Linux). Since perf(build): updates skip the lift, a faster lift, and hot pieces that compile sooner and stay put #2324, that's paid only when the lift really changes.
• A1: a typical edit's ~10 s outside compile and link isn't broken down by today's output. That's next.

All numbers are from one PC, read off make_game.py's own lines (build_times.json, the linked ... (Ns) and compiled in lines); the logs stay local.

[Jiro131]

Claiming A3 (the emit in several processes on Windows), with it already working: the PR is #2748.

• Spawned workers get the lifter's small tables once and the bodies with each run of functions; they translate the words themselves rather than mapping image.bin.
• On an i9-13900K (32 threads, 64 GB, NVMe), main vs the branch, three runs each: emit and write 64.9–66.5 s → 20.9–21.2 s with 16 processes, the whole standalone lift 117–120 s → 73–74 s. The gen folder is byte-identical.
• Linux and macOS keep their forked workers.

If you'd rather fold it into your step 4.1, say so and I'll adapt or close it.

[mooki000]

Thanks @jordanella—Phase 0 has already turned into useful fixes. #2674 and your A5 #2702 are merged, and @Jiro131's A3 #2748 has landed too; #2686, A7 #2675, A1 #2693 and A6 #2706 are still open. #2732 also supplies a concrete base/pack approach to the middle loop, still under review rather than something to design from scratch.

My prompt skip #2698 is merged: the finish stage here went from roughly 67 s to 0.6 s on the i9-13900K/Arch source build, with other work running—not a whole-build or gameplay measurement. Its cleanup/wording follow-up #2710 remains under review, with an open finding being addressed.

Only two concrete proposal details I'd still flag:

• D6: the timing lock needs one shared scope across worktrees, not a separate lock in each work folder. Make timing an explicit request; screenshots and correctness assertions alone need not block every build.
• F5: put the source-only index outside work/, so the tool intended for agents is consistent with the strict data boundary recorded in decision 1.

The landed/open links above would also let the proposal retire its “not started” status and keep the remaining plan short.

[jordanella]

Thanks @mooki000, both taken:

• D6: the lock is the one nx2/kit.py already had, .bench-lock beside the checkout, so every worktree in that folder shares it. What I added (local for now): builds mark themselves beside it while they run, and nx2/dev.py bench -- <command> takes the lock and waits for those marks, so a timing run also waits for a build already going. And timing is now explicit: a scenario run (dev.py run) holds the lock only when the scenario says "timing": true; screenshots and assertions alone don't stop builds.
• F5: dev.py where keeps no index at all: it scans the repo's own source (never work/) when asked, in about 3 s, so there is nothing to put anywhere.
• feat(dev): test a change against a base, as a pack of what it changes #2732: noted as the middle-loop approach to build on; B1's relaunch and DLL swap would sit on its packs rather than redo them.

I'll refresh the proposal's status (landed, open, local) when I next push it.

[Joey-Magnus]

I read the doc against current main and the review thread. hckrman101's points are all reflected: the decisions are recorded at the top, A2 now leaves the path variables out, B1 has the ABI check, A10 and D6 are in, and D4/D5 have the in-session A/B. The red CI is main's own breakage from early Nov 12 and has nothing to do with a docs-only change.

What I'd still change before merging:

• The status line (line 3) still says "not started". #2674 (A7's lift-key half), #2702 (A5) and #2748 (A3) are merged, and #2675, #2693 and #2706 are open. A short "done / open" list would keep the doc honest.
• D6 puts the lock "in work/". That's one lock per worktree, but the point is that a timing run must not share the machine with a build in another checkout, so it needs a machine-wide location (as mooki000 also said).
• F5 puts the index in work/index/, while decision 1 and F6 say agents never read work/. The tool meant for agents should keep its source-only index outside work/.

With those fixed I'd be happy to see it merged as the plan of record.

[jordanella]

Thanks @Joey-Magnus, all three are in 295a2e8:

• Status: the top now lists what's merged (fix(build): a new work folder's second build no longer translates the game again #2674, perf(build): a source whose size, time and id are unchanged is not read again to name its object #2702, perf(lift): emit the lifted C in several processes on Windows too #2748), what's open (fix(build): a compile error in a source file no longer translates the game again #2675, fix(build): the lift key is taken again only if the lift's other inputs did not change #2686, feat(build): build_times.json and the build's last lines say where the time went #2693, perf(build): a renderer header edit compiles the files that read it, not the whole back end #2706) and what's built and measured but not opened yet.
• D6: the lock is machine-wide: nx2/kit.py's .bench-lock beside the checkout, which every worktree there shares. A timing run asks for it explicitly and waits for the builds already running; screenshots and assertions alone take nothing.
• F5: no stored index: dev.py where scans the repository's own source when asked, so nothing goes in work/.