Skip to content

Avoid repeated CUDA device-property queries during convolution plan rebuilds - #33776

Draft
Tianlei Wu (tianleiwu) with Copilot wants to merge 4 commits into
mainfrom
copilot/performance-fix-cuda-conv
Draft

Tianlei Wu (tianleiwu) with Copilot wants to merge 4 commits into
mainfrom
copilot/performance-fix-cuda-conv

Conversation

Copilot AI commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Description

  • Convolution: Initialize Conv and ConvTranspose frontend graphs with the SM version from cached kernel properties:
    const auto& device_prop = GetDeviceProp();
    s_.cudnn_fe_graph->set_sm_version(device_prop.major * 10 + device_prop.minor);
  • Other queries: Pass cached properties to multinomial sampling; replace CUTLASS’s full-property query with the required shared-memory attribute query.
  • Coverage: Add dynamic-shape convolution cases with bias fusion enabled/disabled and deterministic sampling cases.

Motivation and Context

cuDNN frontend ≥1.24 queries full device properties during support checks on rebuilt graphs. Dynamic-shape convolution models incur ~1 ms per node per shape change, causing a reported 22–42% end-to-end slowdown; reusing the EP’s cached properties avoids this overhead.

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

Copilot AI and others added 3 commits October 30, 2026 20:25
Co-authored-by: tianleiwu <30328909+tianleiwu@users.noreply.github.com>
…pancy queries

Co-authored-by: tianleiwu <30328909+tianleiwu@users.noreply.github.com>
Co-authored-by: tianleiwu <30328909+tianleiwu@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix performance issue with CUDA Conv on input-shape change Avoid repeated CUDA device-property queries during convolution plan rebuilds Oct 30, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Performance] CUDA Conv: cuDNN frontend >= 1.24 calls cudaGetDeviceProperties on every plan rebuild (~1 ms per Conv node per input-shape change)

2 participants