Repository navigation
fix(vk): mesh fallback grass is left out while its shaders build, not drawn as holes - #2382
Conversation
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1a07909 to
dc17986
Compare
…2382) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dc17986 to
39fc960
Compare
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
hckrman101
left a comment
There was a problem hiding this comment.
Thanks, the diagnosis (depth from the same culled list drawn while its colour is still building) is clear, and holding the depth draw is a sensible way to turn holes into missing grass. The test is good: it checks the roles, the 120-frame release and the order in draw(). A few things before this goes in:
-
The new skip doesn't exclude memory-writing draws. Since #1573, a draw is only left out while its pipeline builds when none of its stages store to global memory or write storage buffers/images (the
may_skiploop above). The newSKIP_HERE()for held depth-only draws skips the whole draw, including the fallback's task/mesh stages, without that check. If a held depth draw's program writes memory, those writes are lost while it is held. Please pull the #1573 check into a small helper (e.g.writes_memory(const Program&)) and returnFallbackRole::Nonefor programs that write memory, with a case in the test. -
The hold also triggers when the colour draw is left out for good, not just while it builds.
fallback_record()also returns false for a count-buffer draw (issue 640), descriptors of another program, a records budget over 128 MB, an indirect draw with no converted arguments, and a compute build that failed (compute_state == 2,fallback_computeempty). In those cases the colour draw is retried every frame, so its depth draw is now hidden for good where before it was drawn. That may look better than holes, but it differs from what the description says ("they appear once their colour program is ready"). I'd only mark the list as held when the colour program is still compiling (p.compute_state != 2afterfallback_recordfails), or else say in the description that permanent failures hide the depth too. -
Gap (not blocking): when the colour draw's graphics pipeline is the one still building,
draw()returns atif (may_skip) return void(st_pending_skips++);before it gets to the role code, so the list isn't marked and the depth still draws as holes. On ANV the compute build is probably the slow one, so this may not matter in practice. Worth a sentence in the description if so. -
Nit: the key is the indirect-argument address. If the game reuses the same args address for another object's list, an unrelated depth-only fallback draw can be held for up to 120 frames (about 7 s at your 17-20 fps). Probably rare. A comment saying the address is assumed to belong to one list would help.
Overlaps: #1573 (merged) owns the pending-pipeline skip rule this should follow (point 1). #1946 changes how a waiting draw gets its pipeline, not fallback_pipelines_ready, so the two shouldn't conflict textually, but if it lands the graphics-side gap in point 3 shrinks. #2362 (RADV mesh shaders off by default on Linux AMD) and #2215 (more Navi 21 boards on the task + mesh fallback) send more GPUs down this path, so the hold will affect AMD too, and only Iris Xe has been tested. A run on one of those GPUs would help.
CI: only the title check has been queued so far. The build and test workflows haven't run on this head yet.
| } | ||
| } | ||
| const FallbackRole fallback_role = p.fallback && d.mesh ? fallback_role_of(p, s, at, d) : FallbackRole::None; | ||
| if (fallback_role == FallbackRole::DepthOnly && fallback_list_held(d.indirect)) SKIP_HERE(); |
There was a problem hiding this comment.
This skips the whole draw, including the fallback's compute stages, without the memory-write check that #1573 requires for every other draw left out while building. Could fallback_role_of return None when any stage stores global memory or writes storage buffers/images?
There was a problem hiding this comment.
Done. writes_memory() now owns the #1573 check and is reused by the pending graphics-pipeline skip; fallback_role_of() returns None for those programs. The fixture covers global stores and written storage buffers, image buffers, and images.
| } | ||
| const FallbackRole fallback_role = p.fallback && d.mesh ? fallback_role_of(p, s, at, d) : FallbackRole::None; | ||
| if (fallback_role == FallbackRole::DepthOnly && fallback_list_held(d.indirect)) SKIP_HERE(); | ||
| if (fallback_role == FallbackRole::Colour) fallback_list_hold(d.indirect, true); // released below once it records |
There was a problem hiding this comment.
If fallback_record then fails for a permanent reason (count buffer, failed build, records budget, no converted args), the list stays held for good and its depth draw never comes back. Marking it held only while p.compute_state isn't 2 would limit the hold to the build wait.
There was a problem hiding this comment.
Done. A failed colour record marks the list only when compute_state == 1. That also excludes permanent failures which never start a build (0) and completed or failed builds (2); any earlier hold still expires after 120 frames. Rebased onto current main.
5d652ca to
d8ba77a
Compare
… drawn as holes On a GPU without mesh shaders, the mesh fallback's colour draws are left out until their pipelines build (minutes each on an Iris Xe), while the depth-only draws of the same culled list are already built. Depth with no colour let the sky show through the grass, and bloom blew it up into bright specks. A depth-only fallback draw is now held while its list's colour draw was left out in the last 120 frames, so the objects are missing until they build. RX_VK_FALLBACK_DEPTH_HOLD=0 draws them as before. tests/test_fallback_depth_hold.py checks the roles, the hold and its release, and draw()'s order around the fallback's dispatch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Keep memory-writing fallback draws untouched, hold depth only while the colour compute pipeline is compiling, and cover the shared side-effect guard and source anchors. Assisted-by: Kimi Code CLI Co-Authored-By: Kimi Code CLI <noreply@moonshot.ai>
d8ba77a to
ef91bda
Compare
Joey-Magnus
left a comment
There was a problem hiding this comment.
All four earlier points are handled (writes_memory() with tests for all four write kinds, the hold only while compute_state == 1, the graphics-build gap documented, the address-reuse comment). The red Windows part 2 ran on a base before #2397, so updating the branch should clear it.
The one open question for the maintainers: the hold is on by default for every GPU on the mesh fallback, but it's only tested on an Iris Xe. #2362 and #2215 send more AMD cards down this path. RX_VK_FALLBACK_DEPTH_HOLD=0 turns it off.
hckrman101
left a comment
There was a problem hiding this comment.
Thanks for the quick turnaround. All four points from my last review are fixed:
writes_memory()now holds the #1573 check, the pending-pipeline skip uses it, andfallback_role_of()returnsNonefor memory-writing programs. The test covers all four write kinds.- The list is only marked held when
compute_state == 1. I checkedfallback_record(): the count-buffer and other-descriptors cases return before a build starts, and a finished or failed build is 2. So a permanent failure can only extend a hold that a real build started, and that hold still runs out after 120 frames. - The graphics-pipeline gap is now in the description.
- The comment on the address-reuse assumption is there.
CI is green on this head, including Windows part 2, which Joey-Magnus saw red before. The branch merges cleanly.
One nit inline. Joey-Magnus's question is still open: the hold is on by default for every GPU on the mesh fallback, but it has only been tested on an Iris Xe. Whether to ship it on by default is up to the maintainer. The review changes only narrow the hold, so the Iris Xe result should still hold, though it hasn't been rerun in the game.
| fallback_list_hold(d.indirect, true); | ||
| SKIP_HERE(); | ||
| } | ||
| if (fallback_role == FallbackRole::Colour) fallback_list_hold(d.indirect, false); |
There was a problem hiding this comment.
Nit: every recorded colour fallback draw now takes the global fallback_held_mu to erase a key that is usually absent, and recording threads can contend on it. An if (fallback_held.empty()) check is cheap, but it needs the lock too, or a relaxed atomic count beside the map, so this is only worth doing if it shows up in a profile.
On Intel GPUs under Mesa, the game's culling programs never finish compiling, and Intel builds compute pipelines one at a time (issue 496), so every compute build queued behind them waits for good. In Kokiri Forest on an Iris Xe that leaves the forest grey, with no foliage or light, and the rocks and doors the mesh fallback draws stay white. Part of #999 and #1001 ## Why it happens A global memory access the storage buffer pass could not bind is translated into a search of the storage buffer tables: four tables of 16 entries, about 70 entries in all, each written out in `DefineGlobalMemoryFunctions`. These helpers are marked `DontInline` for NVIDIA, but Mesa inlines every function anyway. The foliage culling program 53caac5f has 46 global accesses, so it ends up with over 100k instructions. At the 32 lanes compute is pinned to, Intel's compiler (`brw`, register allocation at SIMD32) did not finish it in 5+ minutes. ## What changed - `spirv_emit_context.cpp`: when `global_table_loop` is set, each table is searched in a loop the driver is asked not to unroll (`DontUnroll`). The loop runs the same test as the written-out form, entry by entry: the same constant-buffer words, the same masked address and bounds, and the first match taken. Its counter is the same in every invocation that runs it, so this is dynamic indexing, not non-uniform, and modules with the loop declare `StorageBufferArrayDynamicIndexing`. The CAS loops' callers open their own block and keep the written-out form. - `vk_coopmat.cpp`: `table_loop_wanted()` turns it on for Intel's Mesa driver (`VK_DRIVER_ID_INTEL_OPEN_SOURCE_MESA`), only where the device has `shaderStorageBufferArrayDynamicIndexing`. The cause is Mesa's, so Intel's Windows driver, whose compiler is not Mesa's, keeps the written-out form until it has been tried. `RX_VK_GM_LOOP` overrides the default on any GPU, read as every switch is (`env_value_off`). - `vk_shader.cpp`: `translator_stamp()` takes the switch (flag bit 8). Programs cached in one form are not used in the other. The bit is 0 everywhere the loop is off, so on other GPUs the SPIR-V, the stamp and the cache are unchanged. - `docs/development.md`: a row for `RX_VK_GM_LOOP`. The effect on the four table-heavy programs: 53caac5f's SPIR-V halves (998 KB to 448 KB), and each now builds in 12-15 s instead of not finishing. ## Tests - `tests/test_global_table_loop.py` (new): - compiles `table_loop_wanted` and runs it for Intel's Mesa and Windows drivers, NVIDIA and AMD, with and without dynamic indexing and `RX_VK_GM_LOOP` (`1`, `true`, `0`); - checks that `translator_stamp` takes the switch; - checks that the loop reads the words the written-out form reads (cbuf_offset / 8 + 2k for every offset and entry) and runs the same bounds test. - `tests/global_load_16_ir.cpp`: the global load fixture emits one module whose storage buffer table is searched in the loop, so the SPIR-V fixtures job runs `spirv-val` on the loop's form (its counter, the return inside its body). - The checks that read the same sources pass: `python -I -B tests/run_all.py global_table_loop bake_mimo amd_fragment_warp bake_skeleton bake_ds cas_loop_ceiling global_atomic_add_f32 mesh_point_topology prepared_stamps recorder_sync program_cache_format program_stamp_knobs` gave 12 passed on Linux. `test_mesh_unreachable_outputs` was skipped (no `spirv-val` here). ## Tested in the game Iris Xe (Raptor Lake-P, i9-13900H, 32 GB), Mesa ANV 26.2.4, Bazzite. Kokiri Forest, walking out of Link's house: - Before: 53caac5f still compiling after 5+ minutes, and every compute build queued behind it waiting. The forest stays grey and the mesh-drawn rocks, doors and trees stay white. - After: every compute program the forest dispatches builds and runs, and the forest gets its foliage and light. With everything built: about 17 fps in the forest, 52 ms of GPU time per frame (`RX_VK_PASS_TIMING=2`), 20 ms of it the game's own dispatches. That work was never run before, so earlier frame rates in this scene are not comparable. Once the culling builds, the mesh fallback's colour draws still wait for their own (slow) builds. Until then the grass shows as sky-coloured holes, which bloom blows up into specks. #2382 holds those draws' depth while their colour programs build. **Not tested:** - Other Intel GPUs (Arc, UHD). - Windows: untested for now. Intel's Windows driver keeps the written-out form; `RX_VK_GM_LOOP=1` turns the loop on there. - GPUs where `RX_VK_GM_LOOP=1` is set by hand. - Intel users' program caches are rebuilt once, because the stamp changes. **Overlap:** #2306 also edits `DefineGlobalMemoryFunctions` (the float atomics). The code merges cleanly with it; only the `docs/development.md` row conflicts. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…piler never finishes (#2525) ## Why On an RX 580 (Polaris 10, Mesa 26.2.6 RADV), a cold `--prepare-shaders` got through the translation and the prediction, then ran out of memory in the precompile. It grew past 24 GB and was killed, on main too. The game's own first start does the same precompile, so a player with an RX 400/500-class card can't get through the first start: it pushes the desktop into swap or the OOM killer. The cause is two mesh programs' fallback compute pipelines, `93e3e999f2fb9fd8` (stage 1) and `ea4dd64df9c8f2b7` (stage 0). RADV never finishes compiling them on GCN 1-4 (GFX6-8), and its thread keeps the memory it grew to. A time limit wouldn't help for that reason. What makes them different from the other mesh programs: - **Size.** Both mesh stages are 4.3 MB of SPIR-V: about 28,000 blocks and 80,000 phis. The native programs already have about 11,000 blocks each. The next biggest mesh programs have up to 29,400 phis. - **Live values.** About 400 values are live at once (≈300 32-bit, ≈100 booleans), far more than a wave's 256 VGPRs and ~100 SGPRs. ACO has to spill tens of thousands of them. - **Not the 126 vertices.** Their max vertices/primitives of 126 isn't what sets them apart: 246 of the first 400 mesh programs the prepare logs are 126/126, and they build normally. - **Not the fallback's own code.** The CFG comes from the native program's structurization, which the native mesh path shares. I compiled the dumped modules with RADV itself, through Mesa's amdgpu drm-shim (no AMD GPU needed), under a 6 GB cap: | module | Polaris 10 / 12, Stoney, Bonaire, Pitcairn (GFX6-8) | Vega 10, Raven (GFX9) | Navi 10 / 21 | |---|---|---|---| | `ea4dd64df9c8f2b7` mesh (79,947 phis) | not done after 45-90 s, killed at the 6 GB cap (the lab: 25 GB at 150 s) | 14 s, 2.9 GB, 24,879 SGPR + 10,644 VGPR spills | 13-14 s, 2.9 GB | | `ecbbff63280b68e2` mesh (29,372 phis) | 4.5 s, 480 MB | | | | `cf987f56b971bdb3` mesh (28,506 phis) | 5.5 s, 664 MB | | | `RX_VK_NOOPT` (DISABLE_OPTIMIZATION) doesn't change it. The spiller runs either way. GFX6-8 have no scratch instructions. ACO spills there through buffer instructions, and these programs' scratch is past their offset range, so ACO takes its overflow path for every spill and reload (`aco_spill.cpp`). With this many spills it doesn't finish. That's my reading of the source, not profiled. I couldn't find a translation-side change that compiles these two normally. Their register pressure and block count come from the native program and the shared goto-elimination structurizer. Changing either would retranslate every shader for every GPU. So this PR leaves just these two out on the cards whose compiler can't build them. ## What changes - `gpu/vk/fallback_size_policy.h` (new): `spirv_phis()` counts a module's OpPhi instructions. `too_big()` decides whether a mesh fallback compute stage should be skipped: the device is RADV on GCN 1-4, and the stage has more than 50,000 phis. 50,000 sits between the 29,400 that build in seconds and the 80,000 that never finish. - `vk_progcache.cpp`: each stage's phi count is recorded where its SPIR-V passes through the program cache. That is on save, which every translated program goes through, and on read. It's done there and not in `vk_shader.cpp` because `vk_shader.cpp` is part of the translator stamp, and touching it would retranslate every player's programs. The record format is unchanged. - `vk_main.cpp` / `vk_internal.h`: `dev.radv_gcn` is set when the driver is RADV and `storageInputOutput16` is false. RADV sets that feature from packed 16-bit math, which means GFX9 and later. Checked through the drm-shim: false on Polaris and Pitcairn, true on Vega 10 and Navi 10. - `vk_draw.cpp` `fallback_compute_pipelines`: before anything is built, a program with a stage over the limit goes into the same "failed" set as a pipeline the driver refuses. It logs one line, and its draws are left out (as #2382 leaves out draws whose fallback isn't ready). The precompile and the in-play builds both go through this function. - `RX_VK_FALLBACK_PHI_LIMIT=<n>` sets another limit, and `0` turns it off. It's documented in `docs/development.md`. - Tests: `tests/test_fallback_size_policy.py` checks the counter on well-formed, cut-short and non-SPIR-V input, checks the limit, and checks that the source still calls the policy before any fallback pipeline is built. `program_cache_cleanup_test.cpp` now includes the header. **What's lost:** on RADV with an RX 400/500, R9 2xx/3xx or Fury card, the draws of these two mesh programs. Neither was drawn in the lab's Kokiri Forest or Hyrule Field runs, so I don't know yet which objects they are. Nothing changes on Vega and later, on AMD's Windows driver, on NVIDIA, or on Intel. The SPIR-V of every program is unchanged. ## Verified - Dumped the mesh fallback SPIR-V on Linux (RADV Polaris 10 through the drm-shim, `RX_VK_DUMP_MESH_SPV`). Both modules pass `spirv-val --target-env vulkan1.3 --uniform-buffer-standard-layout --scalar-block-layout`. This PR doesn't change any SPIR-V. - Ran the precompile on the drm-shim's Polaris 10, translating only these two and the two biggest mesh programs that do build (a local-only key filter), with a 6 GB cap: - main: OOM-killed at the 6 GB cap after 12 s. - this PR: completes in 16 s with the two log lines (79,947 phis, the most 50,000) and the other two fallback programs built. - Ran the phi count over all ~800 mesh fallback modules: only these two are over 50,000. - Ran `python -I -B tests/run_all.py fallback_size_policy program_cache_cleanup mesh_fallback mesh_fallback_failed_once program_cache_reader driver_faults run_all_declarations` (8 passed), plus the program cache, prediction and recorder tests (all pass). `hookshot_vk.so` builds on the current main, ABI check ok. ## Not tested - A real RX 580: no full cold prepare + precompile with this build yet. - Windows, and AMD's own driver (the check is RADV only). - In play: which objects the two programs draw, and so what is missing on these cards. - A Mesa release with a fixed spiller would build them. The limit would then leave them out for no reason until `RX_VK_FALLBACK_PHI_LIMIT=0` or a version check is added. --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Without mesh shaders, grass and foliage go through the mesh fallback. Its colour programs take long to build (minutes each on an Iris Xe), and until they are ready their draws are left out. The depth pass's programs for the same objects are small and already built, so the depth pass still draws: what is left is depth with no colour, the sky showing through the grass and bloom spreading it into bright specks. A depth-only mesh-fallback draw is now left out while the same culled list's (same indirect arguments) colour fallback compute pipeline is compiling, and for up to 120 frames after its last wait, so the objects are missing while they build instead of being holes. Draws whose colour draw ran, programs that write memory, and permanent fallback-record failures are not held. RX_VK_FALLBACK_DEPTH_HOLD=0 draws them as before. The roles and the hold moved to gpu/vk/fallback_depth_hold.h so tests/fallback_depth_hold_test.cpp can check them without a device; the pipeline-wait readiness block now shares writes_memory(). Ported from upstream 3189a42 (NotRexed#2382). Intel/no-mesh-shader hardware not available here: the rules are checked, the in-game holes are unverified.
On a GPU without mesh shaders, the grass in an area you enter for the first time is drawn as holes the sky shows through, blown up into bright specks by the bloom, for as long as the grass's shaders take to build. On an Iris Xe that is minutes per shader and up to an hour for a whole area.
Part of #1001
Why it happens
Without mesh shaders, grass and foliage go through the mesh fallback. Its colour programs take long to build (on Intel they spill at 32 lanes), and until they are ready their draws are left out. The depth pass's programs for the same objects are small and already built, so the depth pass still draws the grass. What is left is depth with no colour: the sky behind shows through, and bloom spreads it. Both passes draw from the same culled list (the draw's indirect arguments), e.g. 43ec86f3's colour and 8a8304ed's depth in Kokiri Forest, and d4abbcb4's with 3bde059f's and the shadow map's 6b7f79e2.
What changed
gpu/vk/vk_draw.cpp: a depth-only mesh-fallback draw is left out while the same culled list's colour fallback compute pipeline is compiling, and for up to 120 frames after its last wait. The objects are missing while they build instead of being holes, and they appear once their colour program is ready. Draws whose colour draw ran, programs that write memory, and permanent fallback-record failures are not held.RX_VK_FALLBACK_DEPTH_HOLD=0draws them as before.Tests
tests/test_fallback_depth_hold.py: checks the draw roles, all four memory-write cases, the compute-build-only hold, its 120-frame release, anddraw()'s order around the fallback's dispatch. It fails on currentmain.tests/test_draw_side_effects.pynow compiles the sharedwrites_memory()helper too;tests/test_mesh_fallback.pyfollows the new fallback-record block.python -I -B tests/run_all.py fallback_depth_hold draw_side_effects mesh_fallbackon the rebased branch: 5 passed on Linux.python -I -B tests/run_all.py: 354 passed, 31 skipped. Of the 13 failures, 12 were unrelated local environment/cache-test failures (missing clang/clang++/zstandard andtest_make_game_cache.py); the other wastest_mesh_fallback.py's old source anchor, fixed and passing in the focused run above.Tested in the game
Iris Xe (Raptor Lake-P, i9-13900H, 32 GB), Mesa ANV 26.2.4, Bazzite (KDE Wayland). Kokiri Forest, walking out of Link's house:
That build also had the Intel table-loop change (
RX_VK_GM_LOOP, a separate PR to come). Without it the forest's culling programs do not finish building on Intel at all, so the colour draws there never arrive.Review changes were not rerun in the game. Not tested: other GPUs without mesh shaders (GTX 9xx/10xx, RX 400/500/5000), where the fallback's colour programs build faster and the hold is shorter; Windows.