Skip to content

fix(vk): build task and mesh programs with the optimizer on RADV - #1965

Merged
hckrman101 merged 2 commits into
NotRexed:mainfrom
addisonbair:fix/radv-mesh-optimizer
Nov 11, 2026
Merged

hckrman101 merged 2 commits into
NotRexed:mainfrom
addisonbair:fix/radv-mesh-optimizer

Conversation

@addisonbair

@addisonbair addisonbair commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

On Linux with Mesa's RADV driver, the GPU resets while the save loads at the Great Deku Tree on the Steam Deck (and RDNA2 desktop cards) unless players find RX_VK_MESH_OPT=1 in an issue thread. The cause is the PR 430 rule that builds task and mesh programs without the driver's optimizer on every AMD GPU. That rule was made and tested on AMD's Windows driver; on RADV, the unoptimized foliage task program runs long enough to trip the GPU timeout.

What changed

build_graphics (gpu/vk/vk_draw.cpp) now applies the no-optimizer rule to AMD's own drivers only, not to VK_DRIVER_ID_MESA_RADV. On RADV, task and mesh programs are built with the optimizer, like every other pipeline. AMD on Windows is unchanged, and RX_VK_MESH_OPT=1 still forces the optimizer on any driver. The check reads dev.driver_id, which create_device already stores. tests/test_radv_mesh_optimizer.py checks that RADV stays out of the rule and the override stays.

What it doesn't fix

This fixes the reset while the save loads. Other RADV hangs remain, and the optimizer doesn't cure them: rere225 (RX 6900 XT) still hangs entering the Great Deku Tree with RX_VK_MESH_OPT=1, and KewinBR (Deck LCD) still hangs in Lon Lon Ranch's open view with it (RX_VK_FORCE_MESH_FALLBACK=1 played). Both runs had the optimizer on already, so neither is a regression from this change.

Testing

Steam Deck OLED: RADV VANGOGH, Mesa 26.2.0-devel, SteamOS 3.9.5. v2.0.14 with only this change, Kokiri Forest, loading the same save each run, RX_VK_MESH_OPT not set:

Renderer Result
v2.0.14 as released ring gfx_0.0.0 timeout and MODE2 reset while the save loads. The log says "task and mesh programs built without the driver's optimizer"
v2.0.14 + this change Plays with no reset. That line is not logged. 15.4 fps (the same as RX_VK_MESH_OPT=1 on the release)

RX_VK_MESH_OPT=1 on the release has also run about 30 minutes across five launches with no reset on this Deck. @manianac reports the same on an RX 6800 XT, RADV 26.2.6 (#1000).

tests/run_all.py (Ubuntu 22.04 with g++-13, as in CI): 321 passed, 13 failed, 11 skipped, both on main and with this change: the same 13 fail without it (tools missing in my container), none new.

Related: #1000, #817, #430.

🤖 Generated with Claude Code

@hckrman101 hckrman101 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. The change does what the description says, and it's narrow. It's the same as setting RX_VK_MESH_OPT=1 by default on RADV, which is what Deck and RADV desktop users already do by hand.

Detection. vendorID == 0x1002 && driverID != VK_DRIVER_ID_MESA_RADV is the right axis: the hang depends on the driver, not the chip. It covers every RADV GPU with mesh shaders: RDNA2 desktop (6800 XT, 6900 XT), the Deck (VanGogh, LCD and OLED), RDNA3/4 and the APUs. RDNA1 has no mesh shaders, so it runs the issue 640 fallback and never builds task/mesh graphics pipelines. Unchanged: AMD's Windows driver, and AMDVLK on Linux (AMD_OPEN_SOURCE, the workaround NotRexed posted in #1000). Both keep the PR 430 rule.

Correctness and the cloth hang. The cloth guard (p.key == 0x5fa56485a6b07619, #461/#1540) is a separate block: _WIN32 plus VK_DRIVER_ID_AMD_PROPRIETARY. This PR doesn't touch it, and it never applied on RADV. Windows AMD still builds task/mesh programs without the optimizer, so the cloth hang and the RX 6600 hang that PR 430 was made for both stay avoided there. Nothing comes back on that side.

Compile time. Task/mesh pipelines on RADV now go through ACO's full optimizer, so a cold cache spends a bit longer on them. The driver cache is keyed on the create flags, so the first start after this change rebuilds those pipelines once. Main has no optimizer time budget any more (the old build_raced), so a slow build just takes longer. If you have it, the shader-prep time on the Deck before and after would be worth adding.

Evidence, and what it doesn't cover. Your Deck OLED A/B plus about 30 min with RX_VK_MESH_OPT=1, and manianac's RX 6800 XT RADV report in #1000, both support it. But two newer reports in #1000 show the optimizer is not the whole RADV story:

  • rere225, RX 6900 XT (RADV, 1002:73af): still hangs every time entering the Great Deku Tree with RX_VK_MESH_OPT=1, a real hang, still stuck at a 30 s lockup timeout.
  • KewinBR, Deck LCD: still hangs in Lon Lon Ranch's open view with RX_VK_MESH_OPT=1. RX_VK_FORCE_MESH_FALLBACK=1 played 15 min.

Both also had RX_VK_MESH_NOCULL set, so the setup differs a little. Neither is a regression from this PR, since they already had the optimizer on. Still, I'd soften "and built with it they play" in the code comment and the description: this fixes the reset while the save loads, and some RADV hangs remain. On main, RX_VK_TASK_MESH_FALLBACK is read outside the #ifdef _WIN32, so rere225's request to try it on RADV Navi 21 works with a main build already.

Fit with main. The PR is 48 commits behind, but it merges cleanly. It is independent of the Mesh shaders row (#1971): under Native or Automatic, task/mesh pipelines are built with the optimizer on RADV, and under Compute none get built. It is also independent of the mesh-query gating (dev.mesh_queries, RADV has meshShaderQueries off on RDNA2).

Relation to #1992 (Navi 21 on the task + mesh compute fallback on Windows): no shared code. #1992 only edits the #ifdef _WIN32 default in vk_main.cpp, and this one only edits the amd check in vk_draw.cpp, so they can land in either order.

Before merge:

  • CI hasn't run. The three workflows are waiting for a maintainer's approval (action_required).
  • There's no source check. Many rules like this one have a test in tests/ that cuts the block out by text. A small tests/test_radv_mesh_optimizer.py, asserting that VK_DRIVER_ID_MESA_RADV is excluded and the RX_VK_MESH_OPT override stays, would keep this from being undone by accident. Optional, but in keeping with the repo.

The code itself looks right to me. Once CI is green, I have no concerns about the change.

Comment thread gpu/vk/vk_draw.cpp Outdated
VkPhysicalDeviceProperties2 p2{VK_STRUCTURE_TYPE_PHYSICAL_DEVICE_PROPERTIES_2};
p2.pNext = &dp;
vkGetPhysicalDeviceProperties2(dev.physical, &p2);
return p2.properties.vendorID == 0x1002 && dp.driverID != VK_DRIVER_ID_MESA_RADV;

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: create_device already stores this as dev.driver_id (vk_main.cpp, the subgroup block), and vk_shader.cpp reads it the same way (dev.driver_id == VK_DRIVER_ID_AMD_PROPRIETARY). So this could be dev.props.vendorID == 0x1002 && dev.driver_id != VK_DRIVER_ID_MESA_RADV with no second properties query.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 55c48f7: the rule reads dev.props.vendorID and dev.driver_id now, no second properties query. Thanks.

@omgitsgela

Copy link
Copy Markdown

Additional Steam Deck LCD measurements: 3601726 and pinned main

Hardware in the test logs: Steam Deck LCD, RADV VANGOGH, driver 0x6801063.

Legacy 3601726 baseline: enabling RX_VK_MESH_OPT=1 on the original renderer let the opening sequence reach player control, where earlier trials had failed. It did not resolve the later color/particle defects or camera hitches. Our subsequently tested legacy branch combines the RADV optimizer policy with upstream-derived DMA-buf imports, allocator-aware texture squeeze, and smaller/growing recording streams. It is not an optimizer-only performance result.

In a matched legacy host/renderer/save/settings DMA A/B, settled room samples rose from roughly 20 to 25–26 fps and queue-submit CPU timing fell from roughly 21 to 2 ms. That gain belongs to the DMA change, not this PR. The combined legacy build was subjectively playable during Kokiri traversal. A separate 30-minute mixed play/menu/view observation had median 20.0 fps and range 10.2–27.7 in five-second VI reports, with no captured kernel timeout. This is not locked-25/30-fps or continuous-route acceptance.

Pinned main ebbba2ea868458cde654a008e481f474841f0dff: official nightly toolchain and matching packaged renderer, FSR 3.1 Performance, frame generation off, isolated copies of the same Kokiri save:

  • Default optimizer policy: the log says task/mesh optimization is disabled on AMD; loading Kokiri caused gfx_0.0.0 timeout, a GPU reset, and VK_ERROR_DEVICE_LOST.
  • Same binaries/options with only RX_VK_MESH_OPT=1 added: Kokiri loaded and rendered terrain/characters; no kernel timeout during the captured gameplay window. The sampled docked Saria view was about 16 fps.
  • A subsequent Handheld-profile sample of the comparable view was about 18–19 fps. Red particles remain; full intro/audio-sync and long-route stability are not cleared.

The reset and cache-stamped option change prevented a cache-matched performance A/B, so I am treating this as additional hardware corroboration for the RADV policy, not proof of a universal speedup or a complete correctness fix.

Public legacy source milestone (not current-main tested source patches): omgitsgela@8d1347f . No game files, saves, cache files, or raw game-data logs attached.

@NotRexed

Copy link
Copy Markdown
Owner

Note from the duplicate #2311 (closed): on the Steam Deck (Van Gogh), the foliage culling mesh program 89ca750f9852b7ca built without the optimizer wedged the GPU about 30 s after a save loaded in Kokiri Forest, every run (ring gfx timeout, then a reset); built with it, the forest plays.

@addisonbair

Copy link
Copy Markdown
Contributor Author

Thanks. All three are in c81cf97:

  • The rule reads dev.driver_id. The second properties query is gone.
  • The comment and description now say this fixes the reset while the save loads, and that other RADV hangs remain. rere225's and KewinBR's reports are listed as not covered.
  • New tests/test_radv_mesh_optimizer.py checks that RADV stays excluded and RX_VK_MESH_OPT stays. I checked that it fails if the RADV exclusion is removed.

The renderer builds clean. I don't have before/after shader-prep times on the Deck yet.

🤖 Generated with Claude Code

@Joey-Magnus Joey-Magnus left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is still needed: #2311 was closed as a duplicate of this PR, not merged, and main still builds task/mesh programs without the optimizer on every AMD driver (gpu/vk/vk_draw.cpp:2187). Your original commit merges cleanly and run_all passes on it (376 passed, Linux).

The new commit c81cf97 conflicts with main, though. Since #2042, main reads the override as env_on("RX_VK_MESH_OPT") (vk_draw.cpp:2192), and the branch still has getenv(...) != nullptr. Either way of resolving it fails a test. If you keep getenv, tests/test_env_on.py flags RX_VK_MESH_OPT (gpu/vk/vk_draw.cpp:2191). If you keep env_on, your new tests/test_radv_mesh_optimizer.py fails with "RX_VK_MESH_OPT=1 no longer builds them with the optimizer", because it looks for the getenv text. I tried both. The fix is to rebase, keep your amd line with dev.driver_id (create_device always sets it), keep main's env_on, and change the test's needle to env_on("RX_VK_MESH_OPT"). Using dev.driver_id and the softer comment both look right to me. Since the RC is today, this is worth doing quickly.

addisonbair and others added 2 commits November 11, 2026 22:43
The PR 430 rule builds task and mesh programs without the driver's
optimizer on every AMD GPU. It was made and tested on AMD's Windows
driver. On Mesa's RADV the unoptimized foliage task program runs long
enough to reset the GPU at the Great Deku Tree (Steam Deck OLED,
RX 6800 XT; NotRexed#1000), and the optimized one plays. Keep the rule for
AMD's own drivers and leave RADV to build them as everything else.
RX_VK_MESH_OPT=1 still forces the optimizer on any driver.

Assisted-by: Claude Opus 5.5 (Anthropic)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ich hang it fixes, and has a source check

The rule reads the driver ID create_device already stored instead of a second properties query. The comment now says it
fixes the GPU reset while the save loads at the Great Deku Tree, and that other RADV hangs remain (issue 1000).
tests/test_radv_mesh_optimizer.py checks that RADV is left out of the rule and that RX_VK_MESH_OPT=1 still overrides it.

Assisted-by: Claude Opus 5.5 (Anthropic)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@addisonbair
addisonbair force-pushed the fix/radv-mesh-optimizer branch from c81cf97 to 55c48f7 Compare November 11, 2026 22:47

@hckrman101 hckrman101 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the rebase. Everything from my last review and from Joey-Magnus's is addressed at 55c48f7:

  • Joey-Magnus's conflict is gone. The branch is rebased on main, keeps main's env_on("RX_VK_MESH_OPT"), and the test's needle is now "RX_VK_MESH_OPT", so it matches env_on and tests/test_env_on.py has nothing to flag. Against main, the only change in vk_draw.cpp is the amd line and its comment. It's 8 commits behind but merges cleanly, and none of those commits touch this block.
  • dev.driver_id: done. create_device sets dev.props (vkGetPhysicalDeviceProperties) and dev.driver_id (the driver-properties query) before any pipeline gets built, so the static is safe to initialize on the first build.
  • Wording: the code comment and the description now say this fixes the reset while the save loads, and that other RADV hangs remain (rere225, KewinBR). That's the scope it has.
  • Source check: tests/test_radv_mesh_optimizer.py checks the window from the amd line to told++ for the AMD check, the RADV exclusion, the override, the if (amd && !mesh_opt) gate and the flag. It needs no arguments, so it's fine with no RUN_ALL_ declarations.

The other evidence agrees with this: NotRexed's note from the closed duplicate #2311 (Van Gogh, the foliage program built without the optimizer wedges the GPU about 30 s after the save loads, plays with it), and omgitsgela's Deck LCD run on pinned main (the device is lost on the default policy, Kokiri loads with only RX_VK_MESH_OPT=1 added).

With #2362 (Automatic picks Compute on Linux AMD, issue 2245): the two don't share code. #2362 edits vk_main.cpp/vk_menu.cpp and this one edits vk_draw.cpp, so they merge in either order. Both are worth having. Once #2362 is in, Automatic on RADV builds no task/mesh pipelines at all, so this PR then matters for anyone who picks Native by hand (and for every RADV user until #2362 lands). It fixes the load reset there, and #2362's default covers the hangs this one leaves open. I'd merge this one first: it's narrower, it has been tested on the Deck by several people, and it doesn't depend on the #2362 decision.

The only thing left before merge is CI. The three workflows at this head are still action_required and need a maintainer to approve them. Approving on the code.

@hckrman101
hckrman101 merged commit eed4d91 into NotRexed:main Nov 11, 2026
1 check passed
@mooki000 mooki000 mentioned this pull request Nov 12, 2026
3 tasks done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants