Repository navigation
Conversation
ed4a11b to
51560ca
Compare
f7721ae to
048cbff
Compare
|
Hey yeetbot, I'm testing this. Would love to get Hookshot running on Mac |
|
Ok this is the third part and my macbook broke so currently it needs testers. So thank you. |
62cd846 to
d7d63a3
Compare
d7d63a3 to
b336d8a
Compare
|
I found overlap with your compiler discovery/cache work while auditing #1931, so I removed that part from my local candidate rather than opening a competing implementation. Two complementary fixes remain: the pinned Python zstandard dependency can run the DS dictionary fixtures/reader on Windows (the existing test passes without a zstd CLI or Linux libzstd soname), and Clang 18 rejects the current unconditional-static constexpr probe with I am preparing a focused bake portability PR for these two failures, retaining the Windows check. Your discovery/cache changes can stay here; please avoid the DS Windows skip if the portable reader/fixture lands first. I will link the PR once its final Linux translator/full-suite validation completes. |
|
Ok! |
85e7cfe to
70f226e
Compare
|
Quick coordination: I see the Python reader fallback is now in this PR too. My tested #1931 candidate is ready at Arkex-ai@4e9ac9e : decoder plus dictionary fixture compression, the actual constexpr probe, and strict truncated/unknown-size frame validation. Full Windows: 321 passed / 3 failed / 21 skipped versus main 319 / 4 / 21; all DS/Mimo/skeleton checks execute and pass. Linux Clang 18 integration and the 512 MiB memory regression pass too. Could we land that focused prerequisite first, then keep discovery/cache/macOS changes here and rebase onto it? I have held opening the PR while we align the reader overlap. The bounded scan is followed by complete-frame validation so truncated frames are rejected. One regression to preserve during consolidation: I executed the actual Windows Python fallback at |
|
ok I understand I will try to do that |
|
Ok, so I'll continue implementing the macOS and the implementation will stay |
# Conflicts: # nx2/build.py
Resolve conflicts with current main while keeping the native macOS port, Builder packaging, and offline-bake portability work. Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # .github/workflows/source-regressions.yml # builder/linux/hookshot_builder.py # nx2/bake_ds/romenum.py # nx2/bake_mimo.py # tests/test_bake_portability.py # tests/test_finish_output.py
toolchain.find() names xcrun's clang by its full path, and clang run that way has no SDK, so zbic, make_game's compiler probes and the compiled checks failed with '<string.h>' not found (the .nup could not be read). native_target() now returns the macOS target and sysroot, and build.py and build_emit.py use it rather than adding them by hand. The macOS check tests the compiler first and names it when it is missing.
…ne wake cannot be lost InitializeCriticalSection, SleepConditionVariableSRW and SleepConditionVariableCS were only in the Linux branch, so the macOS game did not link. They only use pthreads and cv_wait, which both branches have: defined after them now. macOS's condition variables share 64 wait buckets, so WakeConditionVariable broadcasts its bucket: a signal could wake another variable's waiter, which sleeps again, and leave this one's asleep.
…e ran before a local import) Five 'import shutil' statements inside main() made shutil local to the whole function, so the macOS step that copies libwgpu_native.dylib beside the game, which runs before any of them, stopped the build with UnboundLocalError after the link. The module imports shutil at the top; the local imports go.
…st a triangle on Metal) wgpu's Metal back end counts a binding's elements as size / stride, so a binding of (n - 1) * stride + end bytes, valid in WebGPU, read its last vertex as zeros: the title's composite quad (4 vertices of 16 bytes, 8 used: 56 bytes) lost a triangle. Both the cached and the copied vertex paths bind whole strides now. From NotRexed#1088.
…(the menu text was missing) NVN's draw macros put them in the vertex stage's c0 at 0x640, 0x644 and 0x648, as gpu/vk does (GetReplaceConstBuffer), and OoT's text finds each string's glyph records with the base instance there. The WebGPU renderer left them 0, so the title menu and subtitles drew nothing. From NotRexed#1088.
The compute driver bank puts the push constants first, right after the counts, as gpu/vk/vk_bank.cpp does: +0xa9 is their size in words (OoT's NV12 -> RGBA shader: 15), the area rounded to 16 bytes. Without it the tables sat 64 bytes early, the movie's conversion shader read its output image's handle as 0 and was never dispatched. NOP_PUSH_CONST now fills them. RX_NO_CS_PUSH=1 keeps the old layout for debugging. From NotRexed#1088.
…s 16 The WGSL back end and the renderer shared samplers from the 17th texture only in the web build. Metal allows 16 a stage too: with 17 or 18, OoT's pixel programs' layouts were invalid and every batch holding one of their draws was dropped. Now from the 16th sampler declared, on macOS and the web (RX_SHARE_SAMPLERS). From NotRexed#1088.
…ower_to_maxwell.h The WebGPU side kept an older copy of the Ampere-to-Maxwell lowering, without the forms of opcodes 0x8a5, 0xc0b and the texture dimension query that OoT's programs use, so those programs did not translate. It is now the one the Vulkan back end (Eden) uses. The WGSL emitter emulates image atomics (read, operate, write). From NotRexed#1088.
…WebGPU (the title and opening movies) The movie and gameplay upscalers are HMMA networks, which do not translate here (their emulation takes Metal minutes to compile). As the Vulkan back end's upscale stand-in does, the network's last pass is replaced by its input sampled linearly into its output; gameplay's TAAU network by its RGB9E5 scene colour decoded, upscaled and squared. RX_NO_UPSCALE_STUB=1 leaves them out as before. From NotRexed#1088.
… texture (the gameplay's picture was black)
Past 16 samplers a stage, a texture shares the sampler of the last earlier texture of its kind, as the WGSL back end
declares them; the kind there is only depth comparison or not. A depth buffer read as a color is unfilterable, so
when it shared a filtering sampler wgpu refused the pipeline ("Non-filterable float textures can't be sampled with a
filtering sampler"), and OoT's gameplay picture with it: black on an M1 Max. The owner's sampler is now made
non-filtering in that layout (its texture is then sampled nearest), with the never-filtering sampler each bound
texture now carries.
…play Wire the NVN2 storage-buffer table into the WebGPU translator (as Vulkan does), emit WGSL LoadGlobal/WriteGlobal helpers for leftover accesses, and sync small dispatch writebacks so CPU-side exposure reads see GPU results. Co-authored-by: Cursor <cursoragent@cursor.com>
Release scripts need bash 4+ inherit_errexit and the Linux play diagnostics are Builder-specific; skip both on macOS. Split macOS source regressions into two shards like Linux/Windows so the job finishes faster. Co-authored-by: Cursor <cursoragent@cursor.com>
… uploads Metal Occluded (196609) was reconfiguring the swapchain and blacking the window; skip that frame instead. Save mounts use / on POSIX, SDL focus recovers from Metal churn, depth textures clear instead of illegal WriteTexture, and large SSBO writebacks sync a 64 KiB prefix so exposure can reach the CPU. Co-authored-by: Cursor <cursoragent@cursor.com>
Add docs/macos-port-status.md for NotRexed#1872 lineage and hardware status, and link it from the main readme and development guide. Co-authored-by: Cursor <cursoragent@cursor.com>
A build stops at the link, after the whole compile, when the folder chosen for the build files has a comma in its path (for example `D:\Games, Zelda` picked in the builder, or `--work`): the compiler splits the linker options that name the link map and the link order at that comma. A failed link also makes the next build translate and compile everything again (`make_game.py`), and it fails the same way. Fixes #2257. ## What changed - `nx2/build.py`: `linker()` passes linker options each after `-Xlinker`, as one argument (with llvm-mingw as `-Xlink=<option>`, as `toolchain.link_opt` does). It is used for the options that name a file in the build files folder, which went through `-Wl,` (directly on Linux, through `toolchain.link_opt` on Windows): - the link map of the game (`-Map=` on Linux, `/map:` on Windows), of the Vulkan back end, and of `--dll`; - the link order, `hot_order.txt` in that folder (`--symbol-ordering-file=` on Linux, `/order:@` on Windows). The other linker options are constants and stay as they are. - `tests/test_link_path_comma.py` (new, game-free): - no `-Wl,` option in `build.py` carries anything but a constant, and `toolchain.link_opt` is given only constants; - what `linker()` and `HOT_ORDER()` give for Linux, the MSVC target and llvm-mingw, with a build files folder named `Games, Zelda`; - where clang and LLVM's lld are installed: a small program linked into that folder with these options, through ld.lld, lld-link and the MinGW driver, writes its map there. Without lld that part skips itself (CI's Ubuntu has clang but perhaps not lld); the first two parts need nothing. ## How it was tested Linux (Arch, clang and lld 23.1.3), no GPU involved: - `nx2/build.py --standalone` with `OOT_WORK` set to a folder named `x,y` (a copy of a full build's files from main, so only the files changed since were compiled). On main (fab8cb3) the link stops: ``` link FAILED (0s), 0 undefined symbols ld.lld: error: cannot open <build files folder>/x: No such file or directory ld.lld: error: cannot open y/sa/build/hookshot.map: No such file or directory ld.lld: error: cannot open y/hot_order.txt: No such file or directory ``` With this change both links pass and write `hookshot.map` and `hookshot_vk.so.map` in `x,y`. All 3682 functions of the link order are in the new map at the same addresses as in main's own build (built in a folder with no comma), and in the list's order: the order still takes effect through `-Xlinker`. - The Windows options, linked from Linux with the same clang for both Windows targets: a small program linked into a folder named `a,b` with the old options fails (`lld-link: error: could not open 'b/hookshot.map'`; llvm-mingw's ld.lld gives `ld.lld: error: could not open 'b/hookshot.map'`; `/order:@` the same with `b/hot_order.txt`). With `-Xlinker` it links and writes the map. - The compile options that also carry paths in the build files folder (`-I`, `-include`, `-fprofile-instr-use=`) are not split at commas: a small compile with a profile in `a,b` passes. - The new test passes; run against main's `build.py` (`--build-script`) it fails 8 times, and names the 7 options that carried a path (build.py lines 136, 137, 1073, 1092, 1142, 1147, 1161 on main). - `python -I -B tests/run_all.py`: 362 passed, 2 failed, 10 skipped. The same 2 fail on main here, for reasons of this machine (`tests/test_updater_asset_digest.py`: dotnet cannot restore its packages offline; `tests/test_wayland_libdecor_hint.py`: clang 23's new `-Wunused-but-set-global` on its harness). Every other check has the same result as on main; the new one passes. - After a rebase onto current `main` (e0578ed) the new test was run again and passes (6 tests, with the three real links); the full set above was run on fab8cb3, before the rebase. ## Not tested - A Windows build: there is no Windows here. The Windows options were checked only by linking for the two Windows targets from Linux with clang 23's lld-link and ld.lld (MinGW). 2026-11-11: @Arkex-ai ran the new test's six cases on Windows with llvm-mingw's clang 23 and LLVM's lld, with the three real links, at `0d240b30` (see the comments). Still not run: a full Builder build on Windows. - A full `make_game.py` build into a folder with a comma: the lift and the full compile were not run there, only `build.py`'s compile of the changed files and its two links. - The game was not started from these binaries. - macOS (#1872 is where its link lines come in; its `-Wl,-map,<path>` would have the same fault). Assisted-by: Claude Opus 5.5 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
) **What this is for:** contributors rebuilding from source should not spend another minute drawing the same button prompts. **What is measured:** on an Intel Core i9-13900K running Arch Linux, the finish stage at main `c544828f` took 67.707 and 66.898 seconds; two unchanged runs took 0.603 and 0.651 seconds, and the final head took 0.616 seconds, clearing the run-time files as well. Other work was running throughout. **Not measured:** Windows, a packaged Builder update, a complete compile/lift, or gameplay performance. Refs #2636. ## Problem `nx2/make_game.py` removes the game folder's `prompts` directory and runs `prompt_assets.py` on every build when fontTools is available. Most source edits do not change anything those assets depend on. This avoids drawing unchanged assets; it does not make the drawing itself faster. #2636's A1 plans to measure the finish stage, but its A1–A10 items do not already propose this prompt cache. It complements #2674 and #2675, which address unnecessary retranslation rather than prompt generation. ## What changed - **`nx2/make_game.py`:** source-checkout builds request incremental prompt generation. Packaged builders still regenerate every time, using the finish stage's existing launcher distinction. `RX_RELEASE` cannot distinguish them: make_game sets it for source builds too. Missing fontTools still removes the replacement prompts, keeping the existing fallback to the game's icons. - **`nx2/prompt_assets.py`:** fingerprint the inputs before generating, draw into a fresh sibling directory, and record the successful outputs only if a second fingerprint still matches. Reuse requires both matching inputs and every recorded output's size and SHA-256. **A failed generation still removes the prompts, as before**, so the game shows its own icons. An interrupted builder can leave the previous complete set without a stamp; the next build regenerates it. The drawing, encoding and asset formats are unchanged. - **`tests/test_prompt_skip.py`:** game-free fixtures run the actual finish-stage prompt call and production cache/publication functions, with small authored files and a stand-in for expensive drawing. - **`docs/development.md`:** explain reuse, its limits, and `HOOKSHOT_FORCE_PROMPTS=1`. The force switch follows `env_on`: unset, empty, `0`, `false`, `off` and `no` are off, ignoring case and surrounding spaces. ### What prevents a wrong skip The input record covers: - The four prompt source modules, including controller, keyboard and binding tables. - The game's icon font and optional bold label font, compression dictionary, common texture/layout archives, and matching resource-size table names and contents. - The repository's controller and keyboard fonts, and added, changed or removed PNG overrides. - Python/platform; fontTools, Pillow, ASTC encoder, texture decoder, NumPy, zstandard and archspec availability and versions; Pillow's native library versions and the selected ASTC backend. - Generation parameters and relevant library switches, including `SOURCE_DATE_EPOCH`, `FREETYPE_PROPERTIES` and the encoder's custom CPU definitions. Languages and saved bindings are not build inputs here: the generator uses its source tables, and the runtime applies the selected controls. The output record includes files **and directories**: the keyboard directory itself is a runtime marker. Legitimately absent optional outputs stay absent; a missing, truncated or same-size-corrupted recorded file forces regeneration. The runtime's own compressed copies and control-specific keys are excluded before hashing, not mistaken for generator output. **The files the game derives at run time are still cleared by every build**, including an unchanged hit: their own keys do not cover changes to `host/prompts.c`. Only the known runtime filenames are removed; if one cannot be cleared, the build regenerates instead of keeping on doubt. An unreadable input cannot authorize reuse. The log gives the reason, even if a forced regeneration subsequently fails. A source edited during generation leaves no stamp, addressing the stale-input certification problem discussed in [#2674's review](#2674 (comment)). ## How it was tested 2026-11-12; Arch Linux, kernel `7.2.15-1-omarchy`, x86-64, Intel Core i9-13900K, 32 logical CPUs, Python 3.12.16. Base `c544828ff`; final head `2ec89de8`. No game or GPU program was launched for these checks. ### Finish-stage measurements These execute the real finish-stage statements through completion, including binary/image copies, the unchanged shader-list check, pipeline-list work and prompt generation, in a disposable copy of an existing built game folder. Compilation, translation and shader preparation are not included. They are **not whole rebuild times**. All optional prompt packages were present: fontTools 4.68.0, Pillow 13.0.0, astc-encoder-py 0.1.12, texture2ddecoder 1.0.6, archspec 0.2.7, NumPy 2.5.4 and zstandard 0.25.0. | Run | Source | Finish time | 1-minute load before → after | Top other work | |---|---|---:|---:|---| | Before 1 | `c544828f` | 67.707 s | 7.13 → 7.40 | Existing game 316 → 315% CPU; file indexing 66 → 79%; relay check also active | | Before 2 | `c544828f` | 66.898 s | 7.06 → 8.24 | Existing game 315 → 318%; terminal process about 41% | | First generation with cache | `63d5c965` | 66.400 s | 8.24 → 8.19 | Existing game 318 → 320%; terminal process about 42% | | Unchanged 1 | `63d5c965` | 0.603 s | 8.19 → 8.19 | Existing game 320%; terminal process about 42% | | Unchanged 2 | `63d5c965` | 0.651 s | 8.19 → 8.19 | Existing game 320%; terminal process about 42% | | Changed table source | `681208dd` | 63.147 s | 6.99 → 6.95 | Existing game 320 → 321%; terminal process about 42% | | Regenerate, final head | `2ec89de8` | 65.007 s | 5.77 → 5.86 | Existing game 317 → 316%; terminal process about 43% | | Unchanged, final head, with run-time files present | `2ec89de8` | 0.616 s | 5.79 → 5.79 | Existing game 316%; terminal process about 43% | Other work was running; none was a quiet measurement. CPU percentages are lifetime averages from snapshots, not interval measurements. The first generation is included to show that this is a skip, not a faster drawing algorithm. The unchanged runs print: ```text prompts: unchanged, kept ``` Adding a harmless comment to the button-table source forces a fresh generation: ```text prompts: regenerated (table changed) ``` The comment was removed afterwards. A timestamp-only touch does not regenerate: the key checks content. An additional run on the final head verifies an unchanged hit with all six runtime-derived files present: all six are removed, while the 16 generated originals and the stamp retain their sizes and SHA-256 values. The focused fixture also compares the retained originals byte for byte. The final timing rows above include that cleanup. **Asset parity:** all 16 regenerated files, 274,704,680 bytes, compare byte for byte with a full generation by main `c544828f`. Only the new stamp is excluded. Both sides used the same `SOURCE_DATE_EPOCH` because fontTools otherwise writes the current time into the OpenType header. This comparison setting is not introduced into normal builds. A development cache hit retains the earlier font timestamp; packaged builds retain their old full-generation behavior. ### Game-free checks - **23 focused tests pass** on the final change. They cover unchanged reuse, every input kind, package versions/presence, switch/parameter changes, missing/truncated/same-size-corrupted output, empty directories, clearing runtime derivatives while retaining byte-identical originals, regeneration when runtime cleanup fails, interruption, mid-generation edits, unreadable input, force-switch parsing, incomplete output, publication failure, and the packaged/no-fontTools paths. A non-zero generator result removes prompts for both source and packaged builds; interruption before that result leaves the old set unstamped. - The same unchanged-build test against main's finish stage fails behaviorally: `2 != 1: unchanged finish must not draw the prompts twice`. - Removing each of **18 guards/key components** in memory makes its corresponding behavior test fail on the final change. Three later corrections also fail against the earlier version of this patch: returned failure retained old prompts in both build paths, runtime files survived a hit, and an uncleared runtime file did not force regeneration. The unreadable-input diagnostic has a separate before-fails/after-passes proof. - The whole suite one commit earlier, at `f20ea58b` (before a failed generation was made to remove the prompts again and the run-time files to be cleared), `python -I -B tests/run_all.py --jobs 8`: ```text 472 passed, 0 failed, 20 skipped in 133 s (linux, 8 at once) ``` Load was 7.37 → 19.48. The first full run, at 32 checks at once, was **471 passed, 1 failed, 20 skipped**: a clang++ child compiling `texture_pass.cpp` for `nx2/test_depth_sample_types.py` died with SIGKILL. That fixture and its input trees are unchanged from main. No kernel OOM line was found, so the cause is not established. The complete run with eight workers passes, including that check. The failed run is retained, not counted as a pass. On the final head `2ec89de8` the focused checks, the baseline failure and all guard mutations were rerun, and the whole suite once more with eight workers: **471 passed, 1 failed, 20 skipped in 136 s** (load 11.14 → 22.62). The one failure is `tests/test_release_scripts.py`, `test_nightly_two_runs_at_once_end_at_the_newer_commit`, the intermittent race reported in #2626; this change touches none of its files. - No core dumps were recorded. Nothing generated from the game is in this change. ## Not tested - Windows filesystem behavior or a Windows build. - The packaged Windows Builder or Linux AppImage build/update end to end. Their always-regenerate selection is covered by a game-free test only. - A full compile/lift or a new gameplay run, other CPUs, or gameplay/frame-rate benefits. No screenshots: the regenerated asset bytes match, and no visual change is intended. ## Risks The input list must grow when the generator gains new file dependencies or outputs outside the prompt directory. A regression flags new local module imports, but it does not replace reviewing new file reads. Generation temporarily needs room for both old and new assets. Publication uses completed directories and rollback on a rename error; interruption cannot create a stamped half-set, though a process killed between the two directory renames may require regeneration on the next build. ## Overlaps The open-PR scan found 47 PRs touching one of these files. The actual conflict hunks were read: - **#2151:** adjacent item-button generation conflicts with the finish-stage prompt call/error branch. Preserve both steps and account for added prompt dependencies. - **#2261:** substantive overlap. It adds `--wheel-icons`, another input folder and output outside `prompts`. Keep that work outside this skip or extend the input/output proof and staging; accepting both hunks mechanically is not enough. - #1872, #1958, #2304, #2317, #2430, #2465, #2474, #2478, #2608, #2673, #2675 and #2686 share code files but merged cleanly in the scan. That is not proof that any newly added generator dependency is covered. Assisted-by: GPT-6 Astra Co-Authored-By: GPT-6 Astra <noreply@openai.com> Co-authored-by: GPT-6 Astra <noreply@openai.com>
Resolve conflicts keeping Apple Silicon / Metal paths while taking main's dav1d CI packages, Builder update cancel checks, env_on THP, linker helpers, control-bridge docs, and game-free test fixtures. Co-authored-by: Cursor <cursoragent@cursor.com>
The host_os() rewrite dropped WINDOWS while find_tools still gated dav1d on it, and the THP guard assert still expected getenv. Also coalesce adjacent dispatch writebacks into one sync.
Keep Windows cleanup tests patchable via LINUX/MACOS, pass MACOS into HOT_ORDER fixtures, wire sys into Builder handler tests, use linker() for the Apple map path, and skip Windows-only/dav1d checks on macOS.
Windows CI has no bare c++ on PATH, so the presenter queue fixture now takes --cxx from run_all. Release script inherit_errexit stays on bash 4+, and the stand-in sleep yields so concurrent SHA256SUMS finish loops can converge.
Control/AppImage/driver-cache need Linux APIs; cleanup tests force MACOS off so Windows process selection still runs; Builder cancel fixtures get sys for the shared darwin branch.
macOS (Apple Silicon) port — progress
Status of the native Apple Silicon / WebGPU-Metal port that lands through
PR #1872.
How to build and run: development.md § macOS and the macOS section of
readme.md. This page is the changelog of what landed, who did it, and which PRs carry the work.
Current status (2026-11-11)
toolchain.native_target, pthread CS/CV, wgpu/SDL copy)macos-builder/macos-posixgreen/mounts)WriteTextureinto depth)bcf9e1a(MERGEABLE / CLEAN)CI does not run the game. Please try a Mac Builder artifact and report chip, macOS version, steps, and log.
Thanks
This port is the work of many people. In particular, thanks to:
mainRelated pull requests and issues
Lineage (macOS port)
main+ WebGPU/Metal renderer fixes (title/menu/movies; 3D still white)Adjacent / unblocking work folded into #1872
driver_cache_roomon Linux; broke the display-startup fixture until #2051driver_cache_roomin the display-startup test so game-free CI is green againProgress timeline (high level)
main.lower_to_maxwell.h, HMMA upscale stubs. Title/menu/movies good; 3D white. Closed for conflicts / no Mac CI.main(conflicts resolved), Not checking my rom #4+[Bug] lift.py fails with "ValueError: .../exefs/rtld: not an NSO" during translation stage #5 renderer/build commits folded in, bake portability kept, then:LoadGlobal/WriteGlobal+ small/prefix dispatch writebacks (exposure)inherit_errexitWhat still needs a Mac
Report against PR #1872: Mac model/chip, macOS version, Builder or source build, steps, and log (Metal frame capture for render bugs if you can).