Repository navigation
Builder: the damaged-texture check uses what the extraction saw (no ~100,000 files reopened) - #1885
Merged
Merged
Conversation
…100,000 files opened again Since v2.0.12 the first build after an unpack opened every romfs .zs file (about 100,000) to find the ones a 23.3 GB dump has damaged (#1002). Under a virus scanner each open is scanned, and the build sat for a long time at 0% CPU after "main: ZBIC -> plain NSO" or "Reading your game files" (#1876, #1870; v2.0.11 built fine). nup.py now keeps the first 4 bytes of each romfs .zs member as it writes it (HeadSink) and, when the whole romfs was walked, leaves the list of the ones that do not start as zstd beside it (romfs_zstd_heads.txt; a re-run opens only the files the run before left). make_game keeps that list in the game folder under the unpack's stamp and the stand-in step takes it; a folder without it, or with one from another unpack, is checked as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
NotRexed
added a commit
that referenced
this pull request
Nov 10, 2026
…ram Files message (#1889) * fix(builder): honor DLSS opt-in when updating a game (#1643) Use the saved license-accepted DLSS choice during updates, including after the builder restarts, while preserving existing DLSS installations. Exercise the real update decision without WPF, NVIDIA files or a game. Assisted-by: OpenAI Codex (cherry picked from commit c3aa9c0) * fix(build): invalidate DLSS objects when SDK headers change (#1648) Include the selected NGX header tree in the DLSS compilation job dependencies. Preserve content-based reuse across SDK relocation and keep unrelated and non-NGX jobs unchanged. Assisted-by: OpenAI Codex (cherry picked from commit c716137) * Builder: a game folder it cannot write to stops at once with the reason A game folder under Program Files stopped the build minutes in, inside nup.py, with a bare "PermissionError: [WinError 5]" on its .game folder (a player on Discord, v2.0.14). make_game now writes a small check file into the folder before anything is read; refused, it names the folder, says Program Files needs administrator rights when the folder is under it, and asks for a folder of the player's own. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 71635f3) * Builder: the damaged-texture check after an unpack shows its progress After a fresh unpack, the first build checks every romfs .zs file for damage (#1002). It opens about 102,000 files and printed nothing until done: the Builder sat on "the game was read from this .nup before: reading it again is skipped" for minutes (about 5 on a Windows PC with Defender scanning each open), and looked stuck. - nup.damaged_game_files lists the files first, then reports each one it opened (progress(done, total)). - make_game says the check is starting, sends @@progress for the Read row's bar at most 4 times a second, and a plain line at each tenth ("checked 40,874 of 102,185 files (40%, 6 s)") for the log and terminals. - tests/test_damaged_check_progress.py: the reports, the found file, the lines' throttle and text. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 6ad9f1c) * tests: test_nup_stand_in's stand-in of nup.stand_in_damaged takes the progress (#1764) Since #1764 the Python source checks fail on main: make_game.stand_in_damaged_files now passes a progress callback, and this check's stand-in took the romfs alone (TypeError: <lambda>() takes 1 positional argument but 2 were given). The stand-in takes it now, and the check also says each check was given one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 167685b) * Builder: the damaged-texture check uses what the extraction saw, no ~100,000 files opened again (cherry picked from commit b039629, #1885, ported to the v2.0.14 line: its extract has no ProgressSink, so HeadSink wraps the file directly) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(update): Linux downloads are GETs again (the URL was passed as the POST body) linux_update._request built Request(url, url, ...): Request's second argument is the body, so every update download was a POST with a str body and stopped before connecting: "POST data should be bytes, an iterable of bytes, or a file object. It cannot be of type str" (Discord, Arch Linux, 2.0.13 -> 2.0.14). From main's df9a52e, this part only (the rest builds on Linux updater changes this line does not have). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Issai <279266118+Arkex-ai@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: breakthetargets <81792782+breakthetargets@users.noreply.github.com>
3 tasks done
3 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
For #1876 and #1870 (closed into it). This should remove the long stall they hit, but I haven't reproduced their machine, so the issue stays open until a reporter confirms.
Problem: since v2.0.12, the first build after an unpack opens every romfs
.zsfile, about 100,000 of them, to find the few a 23.3 GB dump has damaged (#1002). Under a virus scanner every open is scanned, so the worker sits at 0% CPU for a long time:main: ZBIC -> plain NSO;Players read that as a freeze. v2.0.11, which didn't have the check, built fine. Main's #6ad9f1cd now shows the check's progress, but it still opens every file.
Fix:
nx2/nup.py's extraction keeps each romfs.zsmember's first 4 bytes as it writes them (HeadSink). When the whole romfs was walked, it leaves the list of the files that don't start as zstd beside it (romfs_zstd_heads.txt). A re-run opens only the files left by the run before.nx2/make_game.pykeeps that list in the game folder under the unpack's stamp (.hookshot_romfs_heads). The stand-in step takes it instead of reopening every file, then removes it.Tests: new
nx2/test_nup_zstd_heads.pychecks that:damaged_game_filespatched to fail) and still puts the stand-in in;The nup and stand-in tests (
test_nup_stand_in,test_damaged_check_progress,test_nup_paths,test_nup_codec,test_nup_read_name,test_missing_nup) pass.test_nup_out_of_spacefails the same way without this change; #1793 fixes it.🤖 Generated with Claude Code