[Copilot]

Description

• Initialization: Pack constant FC1/FC3 weights once into per-expert [FC1 gate | FC3 value] buffers; retain shapes so original initializers can be released.
• Inference: Reuse packed weights. Preserve existing behavior for dynamic or mixed constant/dynamic weights and disabled prepacking.

Motivation and Context

CUDA FP16 SiLU-gated MoE shows reported slowdowns of 3× for decode and 1.25× for prefill between 1.23.2 and 1.29.0. At Granite’s dimensions, per-inference weight repacking adds 64 device-to-device copies totaling 64 MiB—even when weights are constant.

• Fixes CUDA MoE is up to 3x slower in 1.29 than in 1.23 #33864

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

🟡 Changes recommended

The new FP16/BF16 tests fail on the TMA warp-specialized dispatch path because the backend’s SwiGLU gate/value chunk order conflicts with the test layout, requiring correction and explicit coverage.

1 open issue

• Fix TMA SwiGLU chunk order and cover its dispatch path · New

What changed in this PR

This PR addresses the CUDA SiLU-gated MoE slowdown reported in #33864 by packing constant FC1/FC3 weights during initialization rather than on every inference.

Changes:

• Cache per-expert [FC1 gate | FC3 value] weights and retain shapes after original initializers are released.
• Reuse packed weights while preserving dynamic, mixed-weight, and disabled-prepacking paths.
• Add CUDA tests for supported data types, repeated inference, prefill, and prepacking controls.

File	Description
onnxruntime/​test/​contrib_ops/​moe_test.cc	Adds correctness and prepack-count tests for CUDA SiLU-gated MoE.
onnxruntime/​contrib_ops/​cuda/​moe/​moe.h	Declares the prepacking hook and cached weight state.
onnxruntime/​contrib_ops/​cuda/​moe/​moe.cc	Packs constant weights at initialization and reuses them during inference.

_{Review effort: Balanced}

[Tianlei Wu (tianleiwu)]

Final maintenance verification at a40151cd3a: the complete, freshly paginated all-20 audit at2026-11-09T15:00:56Z verifies this PR's complete current-head CI green (100 SUCCESS, one NEUTRAL, no pending or failed contexts), OPEN/non-draft/MERGEABLE state and zero unresolved feedback. Normal reviewer approval remains external; no merge or self-approval was performed.

The TMA main refresh, reviewer fixes, forced-SM90 gated-value/bias coverage and native Button testID navigation repair remain intact. CI maintenance preserves production TMA kernels, CPUIDInfo/CPUINFO/provider behavior and the original39a AsyncWorker header. The preceding zero-case98 iOS cancellation showed cold boot/setup and cleanup failures before app installation, not an executed native worker or TMA-kernel failure; no speculative kernel/worker change was made.

The retained98 CPUINFO repair changes only two Windows regression-test files. It observes real allocation cleanup after CPUIDInfo destruction and before later CRT/loader address reuse, repeats16 unload cycles and safely preserves process-lifetime result storage for delayed module callbacks. Mandatory initial-allocation validation and genuine deallocation checks remain enabled. Five native-loader protocol fixtures passed, but actual currenta401 Windows CPU Debug42618374821 supplies the decisive proof: real job129163863702 executed the mandatory cleanup detector successfully in5.16seconds, the refcount case in7.27seconds and35/35 CTest cases passed. Currenta401 Windows CUDA42618374597 also completed SUCCESS, including its separate actual test job.

The new two-file simulator preparation passed all nine regressions against actual TMA YAML/Node configuration. It only resets the uniquely configured simulator and waits for bootstatus/app-service readiness; no global CoreSimulator restart, relaxed240-second Detox deadline or weakened reinstall/cleanup/assertion policy is used. Actual currenta401 React Native42618374715 completed SUCCESS. Fresh Android artifact13676477466 and iOS artifact13677473990 each verify exactly three real passing MNIST, Basic Types and EPContext cases, no failures/errors/skips and the unchanged14/14 checks passed assertion in device logs/traces. Historical98/f8 outcomes and protocol mocks are not substituted for current device or Windows runtime execution.

All remaining non-mobile integrations, including the final WebGPU static-plugin, test and Dawn-package gates, have now succeeded on currenta401. No TMA behavior, tolerances, hardware gates, teardown tests or runtime assertions were bypassed. Local Hopper hardware remains unavailable; no local Hopper execution or performance result is claimed.