Repository navigation
build(release): a pipeline list converted from Fritchii's shader reci… - #2133
Conversation
…pe, 1461 pipelines Fritchii (a Lichii fork) ships a recipe of the pipelines Ocarina of Time draws: its testers' sessions and the states it predicts, each shader named by a fingerprint, with no code. 1,461 of them are pipelines no list here holds that convert to whole version 2 records. They are contrib/pmwork-personal-fritchii-recipe. The fingerprint (SHA-256 of the stage's control section with four bytes masked, then its code area) and code_key hash the same bytes, so hookshot_shaders.bin names the recipe's shaders by code key (41,165 of 41,216). Target and depth formats, samples, topology class, blend, channel mask, polygon mode, depth clamp and alpha to coverage map by meaning. The recipe has no vertex formats (recorded with dynamic vertex input) and no texture answers, so a pipeline is here only when the lists here fix both for its program, or it needs neither (mesh draws; programs that ask about no texture). Of the recipe's pipelines the lists here hold too, converted the same way, 13,323 of 19,738 match one of their records field for field; the others are other states of the same draws (blend, masks, clamp) or blend enables past the last target, which Lichii clears. The README says this one list is converted, not recorded. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adds the three pipeline lists posted in #2034 that aren't in `main` or in an open pull request yet. I added each one with `nx2/pipeline_states.py add`, so each list keeps only the pipelines that no other contributed list already has. Each one has a note beside it. ## What's added | List | From | GPU, OS | Build and settings | Route | Pipelines sent | Kept in the list | New to the shipped list | |---|---|---|---|---|---|---|---| | `contrib/yeetbot90-amd-rx-9070-xt-fsr3-4k` | @yeetbot90 (second list, "MORE PIPELINES") | AMD RX 9070 XT, driver 32.0.32015.7029, Windows 11 | v2.1.0, 3840x2160, FSR 3.1 Auto, frame generation 3x, HDR on | Kokiri Forest through Jabu-Jabu's Belly | 2,047 | 1,013 | 992 | | `contrib/jabbaxipil-rtx-4070-linux-dlaa` | @jabbaxipil | NVIDIA RTX 4070, driver 615.71.09, Ubuntu 26.04.1 (Wayland) | nightly v2.1.0-rc.3-46, 2560x1440, DLAA (model K), frame generation 3x | child, by day: Kokiri Forest out to Gerudo Valley | 960 | 778 | 710 | | `contrib/disgustingusting-web-intel-uhd-intro` | @disgustingusting-web | Intel UHD (i3-1115G4, integrated), driver 32.0.101.7088, Windows 11 | v2.1.0, 1920x1080, no frame generation | the intro only | 9 | 6 | 0 (the tour's list already has all 6) | All three are version 2 lists (`HKSTATE2`). The merged list grows from 25,793 to 27,441 pipelines (1,648 new; the two big lists share 54). I kept the Intel list even though it adds nothing today: the tour's list is replaced whole each time, and this one holds pipelines no other contributed list has. ## What's left out - @pmwork-personal's first list, @yeetbot90's first list and @Imasvpp's list: already in `main` (#2075 and #2082). - Lists from @Brett-D (#2143), @pmwork-personal (#2133) and @jarynclouatre (#2064): each is in its contributor's own open pull request. - @disgustingusting-web's `hookshot_log.txt`: it's a log, not a list. ## How it was checked - I downloaded each zip into its own empty folder outside the repository and listed it without extracting. Each one held a single `hookshot_pipeline_states.bin`: no paths, no symlinks, nothing encrypted, and sane sizes and compression ratios. Only that one file was extracted, with a size cap. - Each file starts with `HKSTATE2`, uses 552-byte records, has a whole number of records, and has no duplicates. - I also checked every field of every record: mesh, sample, depth and flag fields in range, no attributes or answers past their counts, padding zero, and each variant matching its answers' hash. The only exceptions were records written with no answers, which `tour.bin.xz` has in the same form. - `python -I nx2/pipeline_states.py check`: passes. The merge gives the same bytes twice. - Tests: `tests/test_pipeline_state_sources.py`, `test_ship_state_list.py`, `test_pipeline_states.py`, `test_pipeline_notes_open.py`, `test_pipeline_state_counts.py` and `test_pipeline_state_portable.py` all pass. ## Not tested - I didn't run the game: no first-start shader preparation with the new shipped list, no game build, and no check that these pipelines remove stutter on these GPUs. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Joey-Magnus
left a comment
There was a problem hiding this comment.
Nice piece of cross-checking: matching 13,323 of 19,738 field for field against main's lists is good evidence that the mapping is right. The merge is clean and CI is green. Before this goes in, I think it needs a maintainer decision and a couple of changes, because it is the first list in assets/pipeline_states/ that Hookshot did not record.
question (maintainer) assets/pipeline_states/README.md:5-7: the folder's rule until now was "only lists written by Hookshot and this folder's tool go in here". This PR adds an exception for one named list. That deserves an explicit yes from the maintainer, since it opens the door to converted lists in general. Related: the recipe is Fritchii's data, built from its testers' sessions. Credit is given, but the PR doesn't say under what terms it may be redistributed inside a GPL-3.0 repo. Was permission given (the #dev-work discussion?), or is the recipe published under a licence that allows this? A link or a line in the .md note would settle it. The list holds only hashes and draw state, so this isn't a game-data problem. It's a question of whose data it is.
should fix: the converter isn't in the PR (the body offers to add it). For a list that is converted rather than recorded, the script is the only way to review or regenerate it. That matters when main's lists change (inclusion depends on "main's lists fix one vertex input and one variant"), and when code_key or the record format changes. Please add it under scripts/ (standard library only, per CONTRIBUTING), and have the note name the command that produced this file.
should fix: predicted states. The recipe holds "the pipelines its testers' sessions drew plus the states it predicts". Your own check found that the point-topology predictions disagreed with main in 728 of 734 checkable cases. That is a strong signal that the prediction isn't reliable, and the other predicted states are kept. A wrong record costs a pipeline compiled for nothing during the Builder's preparation, on every player's PC. Could you split the 1,461 into drawn and predicted? Then either ship only the drawn ones, or show what share of the predicted ones match main where they overlap.
minor: the note says that blend enables past the last target are cleared by Lichii, and that Hookshot keys on them only on GPUs without dynamic blend state. On those GPUs, such records will not match what the game asks for. They are harmless but wasted there. Worth a sentence in the note so nobody later reads the mismatch as a bug.
I didn't review the .bin.xz record by record. CI's test_pipeline_state* checks validate its format, and they pass.
Joey-Magnus
left a comment
There was a problem hiding this comment.
This is the first list in assets/pipeline_states/ that Hookshot didn't record, so it needs a maintainer yes or no on converted lists, and a word on whether Fritchii's recipe may be redistributed under GPL-3.0. There's no game data in it.
If it goes in:
- The converter should be under
scripts/, so the list can be reviewed and regenerated whencode_keychanges. - Predicted states should be split from drawn ones. 728 of the 734 checkable point-topology predictions disagreed with main, though 13,323 of the 19,738 overlapping pipelines match field for field.
The title is cut off with "…".
…s) (NotRexed#2445) Adds the three pipeline lists posted in NotRexed#2034 that aren't in `main` or in an open pull request yet. I added each one with `nx2/pipeline_states.py add`, so each list keeps only the pipelines that no other contributed list already has. Each one has a note beside it. ## What's added | List | From | GPU, OS | Build and settings | Route | Pipelines sent | Kept in the list | New to the shipped list | |---|---|---|---|---|---|---|---| | `contrib/yeetbot90-amd-rx-9070-xt-fsr3-4k` | @yeetbot90 (second list, "MORE PIPELINES") | AMD RX 9070 XT, driver 32.0.32015.7029, Windows 11 | v2.1.0, 3840x2160, FSR 3.1 Auto, frame generation 3x, HDR on | Kokiri Forest through Jabu-Jabu's Belly | 2,047 | 1,013 | 992 | | `contrib/jabbaxipil-rtx-4070-linux-dlaa` | @jabbaxipil | NVIDIA RTX 4070, driver 615.71.09, Ubuntu 26.04.1 (Wayland) | nightly v2.1.0-rc.3-46, 2560x1440, DLAA (model K), frame generation 3x | child, by day: Kokiri Forest out to Gerudo Valley | 960 | 778 | 710 | | `contrib/disgustingusting-web-intel-uhd-intro` | @disgustingusting-web | Intel UHD (i3-1115G4, integrated), driver 32.0.101.7088, Windows 11 | v2.1.0, 1920x1080, no frame generation | the intro only | 9 | 6 | 0 (the tour's list already has all 6) | All three are version 2 lists (`HKSTATE2`). The merged list grows from 25,793 to 27,441 pipelines (1,648 new; the two big lists share 54). I kept the Intel list even though it adds nothing today: the tour's list is replaced whole each time, and this one holds pipelines no other contributed list has. ## What's left out - @pmwork-personal's first list, @yeetbot90's first list and @Imasvpp's list: already in `main` (NotRexed#2075 and NotRexed#2082). - Lists from @Brett-D (NotRexed#2143), @pmwork-personal (NotRexed#2133) and @jarynclouatre (NotRexed#2064): each is in its contributor's own open pull request. - @disgustingusting-web's `hookshot_log.txt`: it's a log, not a list. ## How it was checked - I downloaded each zip into its own empty folder outside the repository and listed it without extracting. Each one held a single `hookshot_pipeline_states.bin`: no paths, no symlinks, nothing encrypted, and sane sizes and compression ratios. Only that one file was extracted, with a size cap. - Each file starts with `HKSTATE2`, uses 552-byte records, has a whole number of records, and has no duplicates. - I also checked every field of every record: mesh, sample, depth and flag fields in range, no attributes or answers past their counts, padding zero, and each variant matching its answers' hash. The only exceptions were records written with no answers, which `tour.bin.xz` has in the same form. - `python -I nx2/pipeline_states.py check`: passes. The merge gives the same bytes twice. - Tests: `tests/test_pipeline_state_sources.py`, `test_ship_state_list.py`, `test_pipeline_states.py`, `test_pipeline_notes_open.py`, `test_pipeline_state_counts.py` and `test_pipeline_state_portable.py` all pass. ## Not tested - I didn't run the game: no first-start shader preparation with the new shipped list, no game build, and no check that these pipelines remove stutter on these GPUs. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
NotRexed#2133) Fritchii ships a recipe of the pipelines Ocarina of Time draws, and 1,461 of them are pipelines no list here holds that convert to whole HKSTATE2 records. This adds them as `contrib/pmwork-personal-fritchii-recipe`, as discussed in #dev-work. **What the recipe is.** Fritchii (a Lichii fork) ships `OoT v131072 (cache v54).nx2recipe`: the pipelines its testers' sessions drew plus the states it predicts. Each shader is named by a fingerprint, and the recipe holds no code (its format is in `nx2_recipe.h` / `nx2_shader_extract.h` on Fritchii's `oot-gather-fix` branch). **How it maps.** - **Shaders.** The fingerprint is SHA-256 of the stage's control section (first 0x48 bytes, +0x30..0x33 zeroed), then its code area. `code_key` hashes the same control and code bytes with FNV, so a player's `hookshot_shaders.bin` names the recipe's shaders by code key: 41,165 of 41,216. - **Render state.** It maps by meaning: - target and depth formats (Maxwell render-target formats to NVN's, one to one for every format OoT uses) - samples and topology class - blend factors and equations, into `nvn_blend_st` - channel mask, polygon mode, depth clamp and alpha to coverage **What it can't hold.** The recipe was recorded with dynamic vertex input, so it has no vertex formats or divisors. It has no texture answers either. So a pipeline is in this list only when: - main's lists fix both for its program (one vertex input for the vertex stage and one variant), or - it needs neither: mesh draws take no vertex input, and a program none of whose stages asks about a texture or image has variant 0. All 40 such programs in main's lists have variant 0 there. Point-topology states that the recipe predicts for programs main's lists draw as triangles are left out: 728 of the 734 checkable ones disagreed with main. **Checked against main.** I converted every recipe pipeline main's lists already hold the same way. 13,323 of 19,738 match one of their records field for field, including blend, masks and clamp. The rest are: - other states of the same draws, e.g. additive vs alpha blend; or - blend enables past the last target, which Lichii clears and Hookshot keys only on a GPU without dynamic blend state. There was no systematic difference in any field. **Files.** - `assets/pipeline_states/contrib/pmwork-personal-fritchii-recipe.bin.xz` and `.md`, written by `nx2/pipeline_states.py add`. The note says how the list was made. - `assets/pipeline_states/README.md`: one paragraph saying the lists named `pmwork-personal-fritchii-recipe*` are converted, not recorded. They hold the same records and nothing more. The converter is a small script (recipe + `hookshot_shaders.bin` + main's merged list → HKSTATE2). It isn't in this PR; happy to add it under `scripts/` if you want the list reproducible. **Tested.** - `nx2/pipeline_states.py check`: merged list 26,712 pipelines (main 25,251 + 1,461). - Tests: `test_pipeline_states`, `test_pipeline_state_sources`, `test_pipeline_state_counts`, `test_pipeline_state_portable`, `test_ship_state_list`. - `python -I -B tests/run_all.py` on main with both PRs (Windows): 342 passed, 2 failed, 30 skipped (Linux-only). The two (`test_precompile_compute_guard.py`, `test_recorder_sync.py`) fail the same on main without these changes on this PC. - In the game's preparation: a build of main with this list and the follow-up's (NotRexed#2134) ran the Builder's shader preparation (RTX 5090, driver 621.14): `vk: state list: 38697 pipelines (38697 shipped, version 2), 14866 programs (1200 translated now, 0 with stages not in the seed), 953 variants by their textures (953 made), 4798 by nothing bound (4798 made), 38584 noted (2438 native mesh, 0 mesh fallback, 11985 by nothing bound), 113 whose program variant was not made, in 10.2 s` Every record of this list was noted. The 113 not made are records main's lists already hold that were noted without answers (1,285 such records there, none here). Credit: the recipe is Fritchii's (Frizti), built from its testers' sessions. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit b8fd5fd)
… 11,985 pipelines from Fritchii's recipe (#2134) Lists converted from records that hold no texture answers, like Fritchii's shader recipe, cannot name a program variant. 11,985 of the recipe's pipelines that no list here holds have vertex input the lists here fix but no known variant; a build preparing them with the variant nothing bound makes builds most of them as the draws will ask. state_list_prepare: a record with variant 0 and no answers or questions stands for the variant the program's translation makes with nothing bound, each texture as its instruction declares it. When the record's own variant is not found (a program without textures has variant 0 already and is found as before), it is made with seed_plain_of and the pipeline noted with it. Draws whose textures differ make their own variant as before; the prepared pipeline is just not used. Not seed_variant_of with empty answers: that variant would hold no answers and stand for any textures. The log's state list line counts them ("by nothing bound"). seed_plain_of shares seed_variant_of's body (seed_variant_made), which hands the translation questions and answers only when given; without them it answers what nothing bound answers and notes the questions, as the seed's own translation does. contrib/pmwork-personal-fritchii-recipe-plain: those 11,985 records (11,111 graphics, 874 mesh, 4,785 programs). A build without this rule leaves them out ("whose program variant was not made"). test_state_list_plain.py checks the rule in the source. **Since review** (hckrman101): - The plain variant's own variant is the hash of its answers, not 0, so `program_find_portable(key, 0)` never found it, and each prepare translated the list's ~4.8k plain programs again and appended them to `hookshot_programs.bin`. Now `program_find_plain(key)` finds the loaded variant by its answers (an answer for every slot, each its declared type; one `declared_answer` helper shared with `record_answers`), and `state_list_prepare` queues a plain record only when that finds nothing. The prepare's log before this fix: a first prepare in a fresh folder made "4798 by nothing bound (4798 made)", and every later prepare made them again. With it, even a cold first prepare shows "0 by nothing bound (0 made)": a plain variant is the program's own first translation, so the 11,985 pipelines are noted "by nothing bound" with no extra translation (cost measured in the comments). - Disjoint lists: by pipeline (the record without its variant, answers and questions), none of the 11,985 is in #2133's list, any list on main as of 34f0578, or #2143's lists. - This lands after #2133 (the contrib note and README text come from there). - Tests: `test_state_list_plain.py` checks the lookup and both its uses; `test_program_answer_domains.py` copies `declared_answer` with `record_answers`; main's new `test_seed_variant_cleanup.py` (#2323) copies `seed_variant_made` with `seed_variant_of`. `tests/run_all.py` on Windows after merging main at 34f0578: 379 passed, 31 skipped, 2 failed, both failing on untouched main here too (`test_recorder_sync`, `test_vram_budget_row`). A Windows build compiles. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Fritchii ships a recipe of the pipelines Ocarina of Time draws, and 1,461 of them are pipelines no list here holds that convert to whole HKSTATE2 records. This adds them as
contrib/pmwork-personal-fritchii-recipe, as discussed in #dev-work.What the recipe is. Fritchii (a Lichii fork) ships
OoT v131072 (cache v54).nx2recipe: the pipelines its testers' sessions drew plus the states it predicts. Each shader is named by a fingerprint, and the recipe holds no code (its format is innx2_recipe.h/nx2_shader_extract.hon Fritchii'soot-gather-fixbranch).How it maps.
code_keyhashes the same control and code bytes with FNV, so a player'shookshot_shaders.binnames the recipe's shaders by code key: 41,165 of 41,216.nvn_blend_stWhat it can't hold. The recipe was recorded with dynamic vertex input, so it has no vertex formats or divisors. It has no texture answers either. So a pipeline is in this list only when:
Point-topology states that the recipe predicts for programs main's lists draw as triangles are left out: 728 of the 734 checkable ones disagreed with main.
Checked against main. I converted every recipe pipeline main's lists already hold the same way. 13,323 of 19,738 match one of their records field for field, including blend, masks and clamp. The rest are:
There was no systematic difference in any field.
Files.
assets/pipeline_states/contrib/pmwork-personal-fritchii-recipe.bin.xzand.md, written bynx2/pipeline_states.py add. The note says how the list was made.assets/pipeline_states/README.md: one paragraph saying the lists namedpmwork-personal-fritchii-recipe*are converted, not recorded. They hold the same records and nothing more.The converter is a small script (recipe +
hookshot_shaders.bin+ main's merged list → HKSTATE2). It isn't in this PR; happy to add it underscripts/if you want the list reproducible.Tested.
nx2/pipeline_states.py check: merged list 26,712 pipelines (main 25,251 + 1,461).Tests:
test_pipeline_states,test_pipeline_state_sources,test_pipeline_state_counts,test_pipeline_state_portable,test_ship_state_list.python -I -B tests/run_all.pyon main with both PRs (Windows): 342 passed, 2 failed, 30 skipped (Linux-only). The two (test_precompile_compute_guard.py,test_recorder_sync.py) fail the same on main without these changes on this PC.In the game's preparation: a build of main with this list and the follow-up's (feat(vk): a state list record with no variant prepares the plain one; 11,985 pipelines from Fritchii's recipe #2134) ran the Builder's shader preparation (RTX 5090, driver 621.14):
vk: state list: 38697 pipelines (38697 shipped, version 2), 14866 programs (1200 translated now, 0 with stages not in the seed), 953 variants by their textures (953 made), 4798 by nothing bound (4798 made), 38584 noted (2438 native mesh, 0 mesh fallback, 11985 by nothing bound), 113 whose program variant was not made, in 10.2 sEvery record of this list was noted. The 113 not made are records main's lists already hold that were noted without answers (1,285 such records there, none here).
Credit: the recipe is Fritchii's (Frizti), built from its testers' sessions.
🤖 Generated with Claude Code