[Silvio Tomatis (silviot)]

Description

Call Graph::set_sm_version() on each cuDNN frontend graph that Conv and ConvTranspose build in CreateCudnnFeExecutionPlan. The value comes from the EP's cached cudaDeviceProp (GetDeviceProp()).

Starting with cudnn_frontend 1.24 (pinned since ORT 1.28; main pins 1.27.0), Graph::check_support() calls Context::populate_sm_version_from_device(). That function calls cudaGetDeviceProperties() unless an SM version was already set. Both kernels build a new graph per node whenever the input or weight shape changes, so a model with dynamic shapes pays one cudaGetDeviceProperties() call per Conv node every time the shape changes. When target_sm_version > 0, populate_sm_version_from_device() returns early. The value set here is the same one cudnn_frontend would compute for the device, so plan selection does not change.

• Two files, five lines each. The commented-out bias-graph code in both files is unchanged.
• contrib FusedConv uses the legacy cuDNN API (no frontend graph) and is unaffected.
• The cuDNN SDPA graphs in contrib_ops/cuda/bert/cudnn_fmha are left as they are. They are cached per full graph configuration (per thread), so they rebuild only for new configurations; left out of scope.
• This is neutral with cudnn_frontend >= 1.32.0, which reads the SM version with cudaDeviceGetAttribute() instead ([Auto-FE] shim: call ABI-changed CUDA runtime entry points with the loaded runtime's prototype (#1563) NVIDIA/cudnn-frontend#1566).

Motivation and Context

Fixes #33769

On an RTX 5060 (driver 580), cudaGetDeviceProperties() costs about 0.9-1.0 ms per call, almost all of it sys time. Tested with a FastConformer speech encoder (77 Conv nodes) where every audio segment has a new length:

• every encoder run whose input shape changed costs 77 extra calls, about 75 ms;
• end-to-end transcription is 22-42% slower on 1.28.2 than on 1.27.1;
• with input lengths alternating over 20 runs: 1,543 calls (1.53 s), and 136.6 ms per run against 64.8 ms on 1.27.1.

When a process-level interposer returns a cached cudaDeviceProp, the 1.27.1 timings come back and the outputs are byte-identical. This change removes those queries from Conv/ConvTranspose graph construction; I expect the same effect, but it has not been measured in an ORT build. Details and permalinks are in the linked issue.

This complements #33123 (an LRU cache of Conv plans), which helps when shapes recur but not when every shape is new. If #33123 lands first, the same two lines apply to plan->graph.

Testing

• Not built locally. The host where I measured has no CUDA toolkit (it has only the pip CUDA/cuDNN runtime wheels), 8 cores and 16 GB RAM. Building the CUDA EP (about 580 .cu files) plus core was not practical there. I rely on CI for build and test coverage.
• Unit tests. The change does not alter results, and the existing CUDA Conv / ConvTranspose tests run this path. They are in onnxruntime/test/providers/cpu/nn/conv_op_test.cc, conv_fp16_test.cc and conv_transpose_op_test.cc. I did not add a test that counts cudaGetDeviceProperties calls, because ORT's test infrastructure has no clean way to observe them. I am happy to add one if you want.
• Standalone check of the mechanism, on the same GPU.
  • Setup: a small program with cudnn_frontend v1.24.0, ORT's cudnn_frontend_win_dynamic_loading.patch applied, and NV_CUDNN_FRONTEND_USE_DYNAMIC_LOADING defined.
  • It builds 100 fp32 conv_fprop graphs with new shapes, in the same order as CreateCudnnFeExecutionPlan: validate, build_operation_graph, create_execution_plans, check_support, build_plans.
  • Counted with an LD_PRELOAD interposer: 101 calls without set_sm_version and 1 call with it (the program's one-time query, standing in for the EP's cache). Time per graph went from 2.15-2.28 ms to 1.29-1.67 ms over 3 runs each.
• The patch applies cleanly to v1.28.2.
• Not yet measured: an ORT build with this patch on the real model. The interposer result above is the end-to-end evidence.

🤖 Generated with Claude Code

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

🟢 Approval recommended

No active review issues remain, although full ORT build and end-to-end performance validation are still pending.

No open issues 🎉

What changed in this PR

This PR addresses #33769 by passing the CUDA execution provider’s cached SM version to cuDNN frontend Conv and ConvTranspose graphs, avoiding repeated device-property queries during plan rebuilds. It complements plan caching by also helping workloads where every input shape is new.

Changes:

• Set each new graph’s SM version from cached cudaDeviceProp values.
• Document why this avoids costly queries in cuDNN frontend versions 1.24–1.31.

No actionable findings were identified. Full ORT build and end-to-end performance validation remain pending.

File	Description
onnxruntime/​core/​providers/​cuda/​nn/​conv.cc	Sets the cached SM version on new Conv frontend graphs.
onnxruntime/​core/​providers/​cuda/​nn/​conv_transpose.cc	Applies the same optimization to ConvTranspose frontend graphs.

_{Review effort: Balanced}

[Silvio Tomatis (silviot)]

@microsoft-github-policy-service agree