[Rui Ren (rui-ren)]

Latest qualified update: exact R8 FC1 tiling

Adds commit a51424a8f3f78b3c57a9ce1dc99096380e5b87e0 above the previously accepted 0357a7dfe47d3c383bf36be62015162b6a612d5e, in place on native stack #33967. No stack members or PR bases change.

Aligned eight-row IQ2 FC1 now computes 16 rather than 32 independent output columns per CTA, reducing registers from 71 to 64. The original four K partitions, reconstruction, BF16 casts, MMA and FP32 reduction order remain unchanged. FC2, one-token decoding, and offset bindings retain their existing paths.

Fresh H100 real30 comparison against 0357, with unchanged GenAI db6ffd29fc96e319f645ff461efa05c643235f29 from native stack microsoft/onnxruntime-genai#2924:

Input tokens	Baseline ordinary	Candidate ordinary	Baseline k7	Candidate k7
178-254	120.07	120.16	361.69	364.34
427-932	119.79	119.91	299.14	301.36
1028-1663	119.04	119.16	323.94	327.01

Rates are pooled decoded tokens / decode time, excluding TTFT, over two reversed same-physical-GPU passes and 240 complete measured generations. Both arms use the same previously qualified hybrid ordinary graph and the same original R8 k7 graph. Original prompts, warmups, mode order, EOS/512 cap, complete token IDs, and entire speculative counter dictionaries are preserved. Other sessions occupied the remaining GPUs; this comparison ran serially on idle GPU1 under the unchanged foreign-client/VRAM guards.

Both passes improve every k7 bucket. Long-bucket improvement is 0.95%; ordinary performance is preserved. Actual longest-prompt CUDA profiles show about 4% lower FC1 device work, unchanged FC2 dispatch, and identical work counts/tokens/counters. This is an incremental gain, not resolution of the remaining speculative throughput gap.

Qualification:

• 23 native CUDA tests executed, zero skips; two compute-sanitizer memchecks, zero errors.
• 1,782 complete finite tensor/state hashes per arm, bitwise identical.
• 15 GenAI tests including 388 live committed-state boundaries, capture/reset and full sequence comparisons.
• All 240 measured generations match complete same-mode token sequences and all speculative statistics.
• All nine original packed-package files rehashed, including the unchanged 26.33 GB weights.
• Changed C++/CUDA lines pass the repository clang-format configuration; git whitespace checks pass.
• Exact runtime/test binaries, raw evidence, commands, profiles and changed source are preserved in a checksummed, stream-verified durable archive.

CUDA provider SHA-256: 567960ed30c6135f146fe8ff8b8216e3975ec382cbc36eaa757f8cb23c5e0a49.

Durable evidence: /opt/bolt-runtime-fixes-20261108-v1/evidence/k7-fc1-tiling-v1/FINAL.json, SHA-256 eec6bc6c45472c9ac056777c4520c14d113ca366b0dcdf062a446a4d7e31a0f5.

Limitations: incremental production-flag rebuild, not a clean full build, hosted CI, new independent HF comparison, or rebase onto the advanced lowest stack head. No merge is requested. No experimental Q8/scheduling changes are included.

---

Earlier layer qualification (historical)

Description

Extend #34002 on the actual native merge stack #33967 with two guarded loader/tile optimizations:

• Remove redundant bounds/alignment predicates only for complete aligned NVFP4 R8 output tiles.
• Use bounded aligned record windows for eligible IQ2 FC2 weights; retain the original FC1 path.
• Preserve odd-block, offset and partial-tile fallbacks, native operands, FP32 reductions and BF16 boundaries.
• Extend the existing precision regression with the minimum eligible FC2 K=512 width.

Motivation and Context

The actual loaded CUDA profiles attribute remaining latency to packed projection/weight reconstruction. A broader FC1 window was rejected after production profiling; no repacking or precision change is included.

Same native model package and corrected GenAI c0fd01914265d5157d6da1376c38b4df521a76c5 in both arms. H100 unprofiled real30 acceptance, six GPUs, 240 complete measured generations, same physical GPU per pair and two reversed passes:

Input tokens	#34002 ordinary	Candidate ordinary	#34002 DFlash k7	Candidate DFlash k7
1028-1663	92.03	93.96	318.89	324.39
178-254	92.44	94.36	355.31	361.53
427-932	92.14	94.09	293.72	298.91

Rates are pooled decoded tokens / decode time, excluding TTFT; exact same-mode token IDs and all speculative counters are unchanged. This is a small additional gain, not resolution of the remaining public-reference gap.

Validation on the exact candidate head:

• 22 native CUDA tests executed, zero skips.
• Two production-width compute-sanitizer memcheck tests executed, zero skips/errors.
• 1,782 complete finite tensor/state hashes per arm across ordinary and k7 prefixes: bitwise identical.
• Full LCB145 512-token ordinary/k7 sequences and counters identical, including disk-loaded deployment.
• All nine unchanged native package files rehashed, including the 26.33 GB packed weights.
• Actual paired CUPTI profiles verified loaded-provider provenance and unchanged sequences/counters.
• Source formatting/lint and targeted precision regressions passed.

Candidate: f0a529d9fd4f7a134701e908130b2dfaac66810a; baseline: 1b5063c4af651e12bf955bc978c9312e2734a9f9.
CUDA provider SHA-256: 7da68e2302f0bab39d980cdc7d880c0d9e1ddea170aab60343a4b5a997d9eefd.
Packed weights SHA-256: 06d3d508e8d11235d0b770484b6b59764c2f297c3383b6d0de75648d26c11867.

Local checksummed evidence: /opt/bolt-runtime-fixes-20261108-v1/evidence/merge-stack-kernel-gap-v1/production-v2/FINAL.json (SHA-256 0d0617eb1df8f93960416b857c71f181a1fb9eb37b63310cd7ab3033668a83ec). Complete commands, outputs, token IDs, state hashes, provider maps and profiles are preserved in its gates.

Limitations: incremental production-flag provider/test rebuild, not a clean full build, hosted CI, fresh independent HF comparison, or rebase onto the advanced lowest stack head. No merge is requested.

[Rui Ren (rui-ren)]

Qualified opt-in ordinary decode: about 27% faster

Updated the actual stack33967 tip
in-place, without changing its seven-member order, lower heads, PR bases, or draft status.
Companion runtime: microsoft/onnxruntime-genai#2916.

The ordinary decoder was computing eight target rows while committing only the first.
The opt-in package keeps R8 prefill and R1 ordinary decode in one session, with shared
backing buffers and separate captured row geometries. Attention remains physically R8
to preserve Flash/cache arithmetic; other target work runs only the logical rows.

R1 also exposed a real Mamba inconsistency: its readout reduced two partials while R8
reduced four. The native correction retains four-part readout for multi-slot replay.
The new nonzero-state regression fails the old kernel and passes the corrected one.
Original R8 and single-slot R1 dispatch are unchanged.

Unprofiled frozen real30, six H100s, two reversed same-GPU passes, 240 complete measured
generations; original warmups, mode order, EOS/512 cap.

Actual input tokens	Before ordinary tok/s	Hybrid ordinary tok/s	Gain	Original R8/k7 tok/s
178-254	94.40	119.95	+27.1%	361.46 → 361.57
427-932	94.01	119.59	+27.2%	298.68 → 298.94
1028-1663	93.89	118.90	+26.6%	324.25 → 324.31

Ordinary TTFT medians: short 357.35→359.00ms, medium 826.97→831.29ms,
long 2153.15→2161.92ms. The earlier static-R1 version's prefill regression is not
the package being qualified here.

Separate same-GPU synthetic-2048 CUDA profiles, 64 captured target passes:
IQ2 expert work 3.259→1.496ms/pass, Mamba 0.824→0.423ms/pass.
These are device-work attribution, not an additional throughput claim.

Correctness: all240 exact same-mode sequences and complete counters match;
388 complete live committed-state boundaries (66.43GB compared), capture off/on,
two resets, prefix through130; original R8 ordinary/k7 retains1782 complete finite
tensor hashes. Three native kernel tests, seven Mamba operator tests, fifteen
GenAI tests execute without skips. The new replay test also passes CUDA memcheck
with zero errors (100 repetitions).

Scope: hybrid is ordinary-only. Both speculative arms use the original R8 package.
The benchmark explicitly keeps two per-mode models resident in each arm; it is not
a unified hybrid-speculative package or a k7 speedup. No precision/cache/mask workaround,
no weight repack, no independent HF rerun, no clean-build/full-CI claim.

Reproduce the exact tested graph/config with:

python examples/python/nemotron-bolt-hybrid.py --input <packed-r8> --output <fresh-same-filesystem-hybrid>

Requires onnx-ir/numpy/ml-dtypes and both updated runtimes. External-data validation
stays enabled; payloads are hardlinked inside the output package, not copied.

Pins:

• ORT: 0357a7dfe47d3c383bf36be62015162b6a612d5e
• Compiled/common GenAI: d73a6e627d6d20a5a1688b9fbd46237069a7be89
• Published GenAI: db6ffd29fc96e319f645ff461efa05c643235f29 (converter/docs-only child)
• CUDA provider: fac33c75d0336236736aa96456268057b628d93d0b9901d49598e275bf335275
• GenAI shared library: f977660a408bc0a01a31151c7280282ba443fa256785680ae3e841654eec9be7
• Tested hybrid graph: aaf5d6ae06b6740ab5b3af523359581930d2b6c42c38a18529a22737c2cfe530
• Config: 77f2d920d2b04e17884c3d2ee691d9053225b957c047454ebd0a4ea3649dac44
• Unchanged packed payload: 06d3d508e8d11235d0b770484b6b59764c2f297c3383b6d0de75648d26c11867

Full commands, per-case exact tokens/counters, raw kernel traces, provider maps,
UUID/admission receipts, and rejection evidence remain on the validation host under
/opt/bolt-runtime-fixes-20261108-v1/evidence/ordinary-r1-v1. Sealed manifest SHA256: 412b3898f09565557b01e82906358f017f9df3227db2bf330850245a52d32bbc.
Root-disk binaries were reloaded and verified with full ordinary/k7 generation;
the large model payload still resides on the existing tmpfs, not reboot-persistent.
Original package/deployment untouched; no merge.