Repository navigation
[CUDA] Cache cuDNN frontend Conv execution plans - #33123
Tianlei Wu (tianleiwu) wants to merge 10 commits into
Conversation
Signed-off-by: Tianlei Wu <tlwu@microsoft.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🟡 Changes recommended
NHWC cache behavior lacks regression coverage, and the new test file is missing the required license preamble.
2 open findings
What changed in this PR
Adds bounded cuDNN 9 convolution plan caching to avoid rebuilding plans for recurring shapes.
Changes:
- Adds an eight-entry per-kernel LRU plan cache.
- Refreshes tensor bindings and workspace per execution.
- Adds CUDA cache regression tests and inspection helpers.
| File | Description |
|---|---|
cuda_test_bridge.h |
Defines cache snapshot test API. |
cuda_test_bridge.cc |
Exposes internal Conv cache state. |
conv_plan_cache_test.cc |
Tests reuse, eviction, bindings, and recovery. |
conv.h |
Adds cached plan and state definitions. |
conv.cc |
Builds, selects, and executes cached plans. |
🧠 Review effort: Balanced
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Retry unsupported heuristic plans with the unfused fallback and validate bias dimensions before publishing cached plans. Clear synthetic residuals on the compute stream so nonblocking-stream execution and capture remain correctly ordered. Extend fresh FP32, FP16 and BF16 coverage for cache recovery, empty batches, invalid unfused bias and stream capture/replay. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add recurring-shape NHWC plan-cache tests for FP32 and FP16 with prepacked and dynamic weights. Preserve prior device buffers and verify cache reuse, layouts, workspace, and refreshed bindings against exact CPU references. Avoid fused bias for FP16 NHWC activations with dynamic NCHW weights: cuDNN can read those weights as NHWC despite their descriptor strides. Keep the existing unfused bias path for that combination without changing weight inputs or disabling prepacked-weight fusion. Validated fresh focused production Conv build: all 37 NHWC/NCHW cache tests pass without skips on A100 with cuDNN 9.19 and 9.20; focused formatting checks pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…is unsupported Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
### Description Call `Graph::set_sm_version()` on each cuDNN frontend graph that `Conv` and `ConvTranspose` build in `CreateCudnnFeExecutionPlan`. The value comes from the EP's cached `cudaDeviceProp` (`GetDeviceProp()`). Starting with cudnn_frontend 1.24 (pinned since ORT 1.28; `main` pins 1.27.0), `Graph::check_support()` calls `Context::populate_sm_version_from_device()`. That function calls `cudaGetDeviceProperties()` unless an SM version was already set. Both kernels build a new graph per node whenever the input or weight shape changes, so a model with dynamic shapes pays one `cudaGetDeviceProperties()` call per Conv node every time the shape changes. When `target_sm_version > 0`, `populate_sm_version_from_device()` returns early. The value set here is the same one cudnn_frontend would compute for the device, so plan selection does not change. - Two files, five lines each. The commented-out bias-graph code in both files is unchanged. - contrib `FusedConv` uses the legacy cuDNN API (no frontend graph) and is unaffected. - The cuDNN SDPA graphs in `contrib_ops/cuda/bert/cudnn_fmha` are left as they are. They are cached per full graph configuration (per thread), so they rebuild only for new configurations; left out of scope. - This is neutral with cudnn_frontend >= 1.32.0, which reads the SM version with `cudaDeviceGetAttribute()` instead (NVIDIA/cudnn-frontend#1566). ### Motivation and Context Fixes #33769 On an RTX 5060 (driver 580), `cudaGetDeviceProperties()` costs about 0.9-1.0 ms per call, almost all of it sys time. Tested with a FastConformer speech encoder (77 Conv nodes) where every audio segment has a new length: - every encoder run whose input shape changed costs 77 extra calls, about 75 ms; - end-to-end transcription is 22-42% slower on 1.28.2 than on 1.27.1; - with input lengths alternating over 20 runs: 1,543 calls (1.53 s), and 136.6 ms per run against 64.8 ms on 1.27.1. When a process-level interposer returns a cached `cudaDeviceProp`, the 1.27.1 timings come back and the outputs are byte-identical. This change removes those queries from Conv/ConvTranspose graph construction; I expect the same effect, but it has not been measured in an ORT build. Details and permalinks are in the linked issue. This complements #33123 (an LRU cache of Conv plans), which helps when shapes recur but not when every shape is new. If #33123 lands first, the same two lines apply to `plan->graph`. ### Testing - **Not built locally.** The host where I measured has no CUDA toolkit (it has only the pip CUDA/cuDNN runtime wheels), 8 cores and 16 GB RAM. Building the CUDA EP (about 580 `.cu` files) plus core was not practical there. I rely on CI for build and test coverage. - **Unit tests.** The change does not alter results, and the existing CUDA Conv / ConvTranspose tests run this path. They are in `onnxruntime/test/providers/cpu/nn/conv_op_test.cc`, `conv_fp16_test.cc` and `conv_transpose_op_test.cc`. I did not add a test that counts `cudaGetDeviceProperties` calls, because ORT's test infrastructure has no clean way to observe them. I am happy to add one if you want. - **Standalone check of the mechanism, on the same GPU.** - Setup: a small program with cudnn_frontend v1.24.0, ORT's `cudnn_frontend_win_dynamic_loading.patch` applied, and `NV_CUDNN_FRONTEND_USE_DYNAMIC_LOADING` defined. - It builds 100 fp32 `conv_fprop` graphs with new shapes, in the same order as `CreateCudnnFeExecutionPlan`: validate, build_operation_graph, create_execution_plans, check_support, build_plans. - Counted with an `LD_PRELOAD` interposer: 101 calls without `set_sm_version` and 1 call with it (the program's one-time query, standing in for the EP's cache). Time per graph went from 2.15-2.28 ms to 1.29-1.67 ms over 3 runs each. - The patch applies cleanly to `v1.28.2`. - Not yet measured: an ORT build with this patch on the real model. The interposer result above is the end-to-end evidence. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Merge origin/main at d93f544 without rewriting branch history. Retain cached SM-version setup and stream-ordered synthetic-residual zeroing. Reuse per-plan binding slots and lookup-key storage on recurring cache hits. Prevent constant bias prepacking from changing the layout of dynamic weights in the internal NHWC domain. Add binding-storage, constant-bias, and capture-after-real-eviction regressions. Targeted native A100 validation with CUDA 13.0/cuDNN 9.23 passed all 45 cache tests and 52 standard Conv/NHWC tests; two DML-only tests were skipped on Linux. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
TensorShapeVector uses Abseil InlinedVector, whose clear() frees heap storage. Range erase retains capacity across recurring-shape lookups and avoids added cache-hit allocation churn. The unchanged f8998fc head passes the three key-storage tests under the normal local allocator but fails all three with glibc malloc tcache disabled, matching the Windows/Linux CUDA CI pointer mismatches. Keep the existing regression assertions unchanged. Fresh A100/SM80 CUDA 13.0/cuDNN 9.23 validation: all 45 cache tests pass under both allocator settings; 52 standard Conv/NHWC tests pass with two Linux-inapplicable DML skips. Production Conv was rebuilt and both CUDA modules relinked. Focused clang-format and diff checks pass. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
Maintenance update at The earlier native failure was demonstrated on both Linux FP16 and Windows FP32/FP16/BF16 RecurringShapesReuseBindingStorage cases: TensorShapeVector's Abseil InlinedVector clear() releases heap storage. Local malloc address reuse hid the churn.
The genuine current-head React Native workflow 42606543397 completed successfully. Fresh iOS artifact The original 240-second Detox deadline, uninstall/reinstall and cleanup policies, all device/native assertions and CUDA launch behavior remain unchanged. The fresh complete all-20 audit at 2026-11-09T13:46:34Z verifies this exact current49 head open, non-draft, mergeable, with zero unresolved review threads and all current-head checks green. Current simulator/device execution and retained native/cache/lifetime proof remain valid. This PR's CI preparation is complete; normal human review remains required. |
iOS device runs have stalled during cold simulator setup before app installation. Reset only the simulator selected by the Detox configuration, then wait for boot and app services without restarting the global CoreSimulator service. Keep app reinstallation, device assertions, and the existing Detox setup deadline unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Complete native workers before promise dispatch can release their final reference. Preserve exactly-once completion and assert completion at success/failure settlement in all existing device checks. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Description
Cache up to eight cuDNN 9 frontend Conv execution plans per kernel in an LRU keyed
by input, weight, output, bias and residual shapes plus convolution configuration.
Recurring shapes reuse plans and workspace requirements while rebinding current
activation buffers. Plans and fallback descriptors are published only after
successful initialization.
publishing a cached plan.
covering fused FP16 configurations rejected by cuDNN.
and capture/replay on caller-owned nonblocking streams.
failures, empty batches, nonblocking streams and graph capture.
Validation (fresh isolated A100 build, CUDA 13.0/cuDNN 9.20):
cmake --build build/readiness/Release --target onnxruntime_provider_test --parallel 2— passed using a local focused source/registration projection that compiles
the exact production Conv implementation and its test bridge, omitting
unrelated CUDA kernels/tests.
build/readiness/Release,./onnxruntime_provider_test --gtest_filter='CUDA_EP_Unittest.All'— passed: the wrapper executed 33 Conv cache tests, zero skipped.
GTEST_FILTER='ConvPlanCacheTest/0.CapturingCachedPlanOnNonblockingStream' ./onnxruntime_provider_test --gtest_filter='CUDA_EP_Unittest.All'— failed as expected. Restoring the fix and rebuilding passed all 33 tests.
lintrunner --take CLANGFORMAT onnxruntime/core/providers/cuda/nn/conv.cc onnxruntime/core/providers/cuda/nn/conv.h onnxruntime/test/providers/cuda/test_cases/conv_plan_cache_test.cc onnxruntime/test/providers/cuda/test_cases/cuda_test_bridge.cc onnxruntime/test/providers/cuda/test_cases/cuda_test_bridge.h— passed.
The full unrelated CUDA operator suite and Windows DML were not run locally;
fresh-head CI remains required.
Motivation and Context
Alternating recurring input shapes previously rebuilt cuDNN frontend plans.
Retaining a bounded cache avoids that setup cost without retaining activation
buffers.
Merged pinned main (
58bacbc148) to remove the 87-commit lag and incorporateAndroid binary-size baseline fixes: the old run exceeded its stale threshold by
3,374 bytes; main already updated the threshold and passed Android CI. The old
DML failure was a Slice GPU readback
DXGI_ERROR_DEVICE_HUNG(0x887A0006);the same test passed on newer main. No DML/Slice code or test exclusion was added.