Repository navigation
fix(builder): find damaged textures still in the .nup's zstd frame - #2391
Conversation
|
Thanks. Rebased onto main. I'll wait for #2096: once it lands I'll rebase onto it and make the combined stamp |
c474bd0 to
83bd366
Compare
|
Thanks for coordinating with #2096. |
Builders before NotRexed#456 left three damaged .bntx.zs files wrapped in the .nup's own zstd frame: they start as zstd, so the check passed them and marked the folder checked. A .zs frame whose content is as large as the file is now decompressed at its start; zeros, or no decompression, count as damaged. .hookshot_romfs_checked now carries a check version, so folders checked before are checked once more. Fixes NotRexed#2361.
83bd366 to
971a983
Compare
Joey-Magnus
left a comment
Joey-Magnus
left a comment
There was a problem hiding this comment.
The blocker is resolved: #2096 is merged and this is rebased onto it. resolved-v2 (nx2/make_game.py:463) re-checks folders stamped v1, and a test covers that case. still_in_nup_frame only decompresses when the frame's content size equals the file size, so real .zs files are skipped without the cost. CI is green.
… a time runs the full check once on its first build with the next release. With no list from the extraction left (it is deleted after the first check), that check opens every romfs .zs file, 102,185 of them, one at a time. On Windows each first open of a file took about 4-5 ms here: the check took 486 s and 492 s in two builds (i9-13900K, NVMe, Windows Defender on). We attribute the wait to the virus scanner scanning each file on its first open; this was not confirmed with the scanner off. Fixes NotRexed#2754 Those waits overlap when several files are open at once, so the files are opened by a thread pool of min(32, CPU threads + 4), Python's own default for I/O work. Results and progress stay in the listed order. The opens are queued 4096 at a time, not all 102,000 at once. The whole check after a fresh unpack, with the stamp removed as NotRexed#2391's change does, two new game folders each, interleaved: 486 s and 492 s before, 27 s and 28 s after, the same files found. On files already opened once (three runs each, alternating): 5.0-5.9 s before, 4.9-5.6 s after. Opening fresh copies of the game's .zs files, 2,044 files per run, ms per file: threads 1 2 4 8 16 32 64 ms 3.5 2.4 1.16 0.56 0.34 0.25 0.21 Not measured: Linux, HDDs, network drives, machines with few cores, and the check with the virus scanner off. tests/test_damaged_check_progress.py: - the first two opens wait for each other at a barrier, which a check opening one file at a time never gets past (fails before the change); - 4,500 files in three folders, more than a slice and than the threads: the damaged ones and the progress in the listed order, and an unreadable file still stops the check. docs: the worklist row for NotRexed#2754 and the tested-configurations row of the game run on this build. Assisted-by: Claude Code Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Builders before #456 left three damaged .bntx.zs files wrapped in the .nup's own zstd frame. They start as zstd, so the damaged-file check passed them and the Water Temple, Great Fairy Fountain and Ganon's Stage kept their invisible walls and floors.
.hookshot_romfs_checkedcarries a check version, so folders already checked are checked once more.Closes #2361