Skip to content

Fix release planning for cross-stream aliased buffers - #33881

Open
Tianlei Wu (tianleiwu) with Copilot wants to merge 6 commits into
mainfrom
copilot/cuda-ep-fix-expand-shape
Open

Tianlei Wu (tianleiwu) with Copilot wants to merge 6 commits into
mainfrom
copilot/cuda-ep-fix-expand-shape

Conversation

Copilot AI commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Description

Fix premature release of host allocations whose buffers are reused and then aliased across execution streams. Deallocation planning now follows the full kReuse chain, so consumers of Squeeze/Reshape-style aliases retain the allocation that owns the buffer.

Summary of Changes

  • Resolve explicit and implicit input consumers to the owning allocation before constructing release actions, with a bounded diagnostic for cyclic reuse chains.
  • Preserve the existing allocation/reuse plans and release policy: single-stream temporaries release at their last consumer, externally allocated temporary references remain releasable, and graph inputs, graph outputs, and initializers are not deallocated by the execution plan.
  • Retain the GPU-independent cross-stream host-buffer regression and add five single-stream alias ownership cases.
  • Exercise the same host-buffer reuse and alias chain through InferenceSession with enable_cpu_mem_arena both disabled and enabled, verifying allocator type, root release accounting, and runtime outputs.
  • Respect the CPU provider's arena capability when checking allocator type: requesting an arena does not enable one in ASan, 32-bit, or alternative-allocator builds. Both runtime variants still execute all lifetime and output checks.

Testing

  • Configured and built a fresh, private CPU-only Release onnxruntime_test_all with streams enabled and at most eight build jobs.
  • Before the production fix: six selected tests ran; the cross-stream regression failed with two root consumers instead of four, while all five ownership cases passed.
  • After recompiling the production fix and runtime test source and relinking the private binary: 38 selected tests passed, including the unchanged cross-stream regression, all five ownership cases, and both arena variants:
    ./onnxruntime_test_all --gtest_filter='PlannerTest.*:AllocationPlannerTest.*:FreeBufferIndexTest.*:AllocationKinds/AliasedBufferReleasePlannerTest.*:CpuArena/AliasedCpuBufferRuntimeTest.*'
  • The two runtime variants execute 16 inference runs each (32 total), use real CPU allocators, and disable memory patterns so they cannot mask the host-buffer lifetime. The device stream uses synchronous test kernels, not CUDA hardware.
  • Targeted lintrunner checks passed for both modified C++ files.
  • The arena-capability portability correction was validated in a fresh, private Clang 18 Debug ASan build: 39 focused C++ tests passed, including the allocator-policy test, unchanged cross-stream regression, five ownership cases, and both 16-run runtime variants. No sanitizer findings or skips; no sanitizer suppression or test exclusion was added.
  • Linux and Windows CUDA test jobs and both CUDA plugin test jobs passed on production-fix head 660105743cec40554650e6995d639c31480bb196. The test-only portability push starts new exact-head CI, which is not yet claimed green.
  • The full CUDA model reproduction was not run locally. Current-head Linux and Windows CUDA test jobs must finish before declaring validation complete; code pushes start fresh CI, and previous-head jobs were not rerun.

Motivation and Context

Per-device stream planning can leave an alias pointing to a value whose buffer was itself reused on another stream. Following only one reuse link omits later alias consumers from the root release action, allowing a live CPU shape buffer to be recycled before a CUDA Expand reads it.

Fixes #33807.

Checklist

  • Regression tests added/updated
  • Existing ownership and single-stream release behavior preserved
  • No public API changes

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

Co-authored-by: tianleiwu <30328909+tianleiwu@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix Expand shape issue for CUDA EP Add regression coverage for cross-stream host-buffer alias lifetimes Nov 4, 2026
Follow the full reuse chain when assigning consumers to buffer release actions. Per-device stream planning can retain cross-stream alias links; following only one link drops late consumers and releases live host shape tensors prematurely.

Preserve single-stream release and existing allocation ownership behavior, with five focused alias ownership regressions.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@tianleiwu Tianlei Wu (tianleiwu) changed the title Add regression coverage for cross-stream host-buffer alias lifetimes Fix release planning for cross-stream aliased buffers Nov 9, 2026
@tianleiwu
Tianlei Wu (tianleiwu) marked this pull request as ready for review November 9, 2026 00:53
Copilot AI balanced review requested due to automatic review settings November 9, 2026 00:53
@tianleiwu

Copy link
Copy Markdown
Contributor

/azp run Linux Android Emulator QNN CI Pipeline

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
No pipelines were found matching this branch/path.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Cross-stream host-buffer lifetime changes lack end-to-end CUDA validation, leaving the reported failure and arena-disabled case unverified by the planner-level, CPU-only checks.

No open issues 🎉

What changed in this PR

Fixes premature release of host buffers reused and aliased across execution streams, addressing the allocation-lifetime issue reported in #33807.

Changes:

  • Follows the full buffer-reuse chain when assigning consumers to the owning allocation, with bounded cycle detection.
  • Preserves existing allocation ownership and release policies.
  • Adds a GPU-independent cross-stream regression and five single-stream ownership cases.

No actionable findings were identified.

File Description
onnxruntime/​test/​framework/​allocation_planner_test.cc Adds cross-stream alias-lifetime and single-stream ownership coverage.
onnxruntime/​core/​framework/​allocation_planner.cc Resolves reuse chains to the owning allocation before planning releases.

Review effort: Balanced

@tianleiwu

Tianlei Wu (tianleiwu) commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Re: #33881 (review) — the current published head is d26a89dd78184501cdd5fc777b4eac35e990d7e5. The full #33807 CUDA model remains unverified locally, as stated in the PR body; neither the native unit validation nor the device checks below is presented as that full-model reproduction.

The GPU-independent accounting repair in da395cddaa3888e4492fadf6ea78a004016d33b2 remains intact: two root consumers before, four after. The retained exact-native evidence comprises 39 ASan planner/ownership cases and 32 inference executions with CPU arena enabled/disabled, no skips or sanitizer findings, and allocator expectations respecting ASan/32-bit capability. The earlier exact-head Linux CUDA success satisfied the draft gate; the PR remains non-draft and normal reviewer approval is external.

The simulator-preparation repair in d26a89dd78 changes only the React Native workflow and related documentation. It resets only the configured simulator, waits for boot/app-service readiness, and retains the 240-second Detox deadline, uninstall/reinstall behavior, and every device assertion. All nine protocol regressions executed against this branch's actual YAML passed.

The genuine current-head React Native workflow 42603603023 completed successfully. Fresh iOS artifact 13669391885 and Android artifact 13668422663 each contain exactly three real passing MNIST, Basic Types and EPContext cases, with no failures/errors/skips and the unchanged 14/14 checks passed assertion in the device logs/traces. This is actual new-head device execution, not a green setup step or an older-head artifact.

The fresh complete all-20 audit at 2026-11-09T13:20:03Z verified this exact currentd26 head open, non-draft, mergeable, with zero unresolved review threads and all current-head checks green. Current Android/iOS device proof and the retained native accounting proof remain valid. This PR's CI preparation is complete; normal human approval remains external. The full #33807 CUDA model remains unverified locally.

@tianleiwu
Tianlei Wu (tianleiwu) marked this pull request as draft November 9, 2026 01:14
Exercise the cross-stream host-buffer reuse and alias chain through InferenceSession with the CPU arena enabled and disabled. Verify the allocator type, root release accounting, and sixteen runtime outputs for each setting without requiring a separate GPU build.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use the CPU provider arena-capability policy when checking the allocator selected for each session option. ASan and 32-bit builds deliberately use a device allocator even when an arena is requested. Keep both runtime variants and all alias-lifetime and output checks.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@tianleiwu
Tianlei Wu (tianleiwu) marked this pull request as ready for review November 9, 2026 06:31
iOS device runs have stalled during cold simulator setup before app installation. Reset only the simulator selected by the Detox configuration, then wait for boot and app services without restarting the global CoreSimulator service. Keep app reinstallation, device assertions, and the existing Detox setup deadline unchanged.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[CUDA EP] Expand receives a wrong shape: CPU buffer of a live int64 shape tensor is reused (1.29.0, 1.31.1)

3 participants