[Rui Ren (rui-ren)]

Measured CUDA kernel latency - 2026-11-07

BF16, IQ2_XXS, ReLU-square experts, latent hidden dimension 1024, top-k 9. Internal softmax routing versus preselected FP32 indices/weights using identical routes and the same corrected expert arithmetic. Selected logits are +16 and others -16; selected weights are 1/9. Shared/distinct labels describe expert reuse across rows (the one-row variants are equivalent workload controls). Sixteen rows create 144 routes, crossing the 128-route prefill-tiles dispatch threshold.

The ratio measures routing-path overhead removed, not this entire PR versus its parent. All 24 complete BF16 outputs are bitwise equal in both runs. Benefits diminish for distinct-expert, bandwidth-heavy batches; all cases, including those smaller gains, are shown.

Intermediate	Experts	Rows	Routes across rows	Internal routing us	Preselected us	Ratio
1536	240	1	distinct	54.71	41.35	1.32x
1536	240	1	shared	53.31	40.51	1.32x
1536	240	8	distinct	205.41	190.60	1.08x
1536	240	8	shared	56.53	43.36	1.30x
1536	240	16	distinct	396.04	381.20	1.04x
1536	240	16	shared	94.66	80.74	1.17x
2048	240	1	distinct	63.38	49.42	1.28x
2048	240	1	shared	62.54	48.42	1.29x
2048	240	8	distinct	252.80	238.01	1.06x
2048	240	8	shared	64.43	51.36	1.25x
2048	240	16	distinct	495.45	480.54	1.03x
2048	240	16	shared	112.66	99.46	1.13x
2304	432	1	distinct	79.56	57.62	1.38x
2304	432	1	shared	77.32	56.37	1.37x
2304	432	8	distinct	322.00	297.79	1.08x
2304	432	8	shared	82.72	61.52	1.34x
2304	432	16	distinct	670.85	653.92	1.03x
2304	432	16	shared	182.34	161.18	1.13x
2560	432	1	distinct	80.40	59.66	1.35x
2560	432	1	shared	78.78	58.72	1.34x
2560	432	8	distinct	325.76	308.10	1.06x
2560	432	8	shared	82.50	61.28	1.35x
2560	432	16	distinct	689.52	666.84	1.03x
2560	432	16	shared	196.47	175.37	1.12x

H100 80GB HBM3, CUDA 12.8.93, driver 580.159.04, GCC 11.4.0. CUDA-event median per dispatch; 64 dispatches per captured CUDA graph, 50 warmups, three graph warmups, and 15 paired timed batches alternating AB/BA. Four rotating allocation banks contain deterministic synthetic weights (seed 617); this is not a cold-cache guarantee or a checkpoint-weight benchmark. Raw p95 values describe batch-average latency, not individual-invocation tails. No profiler runs in the timing window. The assigned GPU had empty admission, one stable compute client during execution, and clean release afterward; clocks were not locked. Two independent 39-case runs passed, including the published replay runner.

Scope: same-build production CUDA dispatch alternatives, not historical parent-commit binaries. The measured runtime is an isolated CUDA provider relink with preserved core/support, not a clean full-source ORT build or end-to-end model qualification. Each measured operator's CUDA source hash matches its owning PR head. A clean source build is a separate replication variant and must be measured again. Kernel-only timings do not establish model TTFT/tok/s, MMLU/LCB, speculative integration, or full-model readiness.

Complete primary/repeat results, numerical metrics, source hashes and benchmark code. All 39 cases passed in both supervised runs; full-model work remains paused. No source push, stack change, merge, auto-merge or ready-for-review promotion is included.

---

Current source and CI repair - 2026-11-07

Head: a9bdbce374a3b2757e1fd11ef83ce6f83502b0d9. Native stack #33967 remains #33953 -> #33962 -> #33963 -> #33969, ultimately targeting main. All three followups remain draft; no merge or auto-merge was requested.

Inherits #33962's NVFP4 guard, projection fixture and test-dependency fixes. Repairs the MoE cancellation fixture by keeping unit terms outside the large-magnitude MMA partials. Kernel arithmetic and exact output tolerance remain unchanged. Operator entries in ContribOperators.md and OperatorKernels.md were refreshed from source-pinned CI artifacts. The Mamba artifact was generated against its former intermediate base, so its stale unrelated GQA/WebGPU changes are excluded and newer inherited documentation is preserved.

Local validation: 10/10 current packed CUDA regression tests passed, zero skips, including the repaired cancellation cases and BF16/FP16 NVFP4 scale-before-cast. This is a bounded production-object test overlay with a preserved core, not a complete source-built runtime or full-model qualification. The pinned sentencepiece_model_pb2.ModelProto import and serialization roundtrip pass with sentencepiece 0.2.2 and protobuf 6.33.6.

CI status: fresh GitHub Actions started automatically on this exact head; results are pending, not claimed green. There are 32 current-head workflow runs. The earlier missing-check diagnosis is corrected: the native final-base merge was blocked by our sentencepiece 0.2.1 pin conflicting with main's concurrent 0.2.2 fix. Matching main resolved the conflict; all three PRs are now mergeable. No workflow-filter changes were needed or made. Live affected checks: ONNX Runtime CUDA Builds, ONNX Runtime WebGPU Builds, Linux CI, ONNX Runtime Windows GPU Doc Gen CI. The stack, PR bases and draft states were not altered to force CI. No failed old-SHA workflows were rerun. Earlier WebGPU failures were GRU/LSTM tests outside these CUDA changes; the static WebGPU job was a downstream test-result gate, not a compiler failure. No unrelated kernels or thresholds were modified.

Model builds, MMLU-Pro, LCB and performance testing remain paused at the user's request. No new full-model readiness or performance claim is made.

Operator design and preserved earlier evidence

Description

MoE layer of native stack #33967: #33953 -> #33962 -> #33963, with ultimate base main. Its three commits were replayed onto projection head c1b4daf5f209600bf3e118935822b5780c62b070, producing b9b776b540b15a2c73e41b2559c1f1d662654206. git range-diff reports three unchanged patches, and all four own source/test files remain byte-identical. The original MoE head 520f375edcce90f52e92f3f44c76a44c589b3ae3 is preserved in a backup ref. The dependency branch is the review base, not the landing destination: native stack landing is intended for main after review and validation, not a merge of our work into the coworker's feature branch. Coworker #33953 remains unchanged at ad04a0448c8a995b4557c12bab12bc2288497fbc.

• Extend the coworker MoEPacked operator for ReLU-square experts and preselected routes with FP32 routing weights through k=32.
• Support fused sigmoid routing with correction bias while deriving mixture weights from the unbiased scores.
• Preserve malformed-route isolation and the existing default SwiGLU behavior.
• Preserve BF16 IQ2 FC1/FC2 contiguous-K partitions, MMA order, BF16 store boundaries and FP32 final routing/reduction. Reuse existing routing, workspaces and finalization, including four eight-column subtiles for 32-route prefill tiles.

Motivation and Context

Nemotron Bolt uses ReLU-square rather than SwiGLU and needs its existing routing and precision contract preserved. The source-qualified cancellation regression distinguishes the old source-compatible result (1024) from the former interleaved result (256), including rows 1/8/17/129.

Pre-restack validation: production-flag compilation, scoped clang-format and whitespace checks pass. All 10 combined CUDA tests and all 10 compute-sanitizer memcheck tests execute, with zero skips and zero memcheck errors. On identical diagnostic operands with the actual first MoE weights (240 experts, k=9), both the latent projection and all 8192 MoE output values match the preserved runtime bitwise. That reduced intervention is separate from live acceptance.

Four actual current Mobius-generated packed MoE cases at 1/8/9/17 rows produce exact outputs with three-repeat identity and verified CUDA placement through the C++ API. These use pinned sparse four-expert/topk2 codec fixtures, not the production 240-expert/k9 checkpoint. Row1 passed in the initial attempt; its remaining rows were rejected before worker launch by the storage guard, and the original overall FAIL report remains unchanged. A fresh remaining-row attempt passed rows8/9/17. The two fixture models have identical IR and initializer values after normalizing initializer insertion order; both original serialization hashes are preserved. These are component/source-packing/Scan/row-tiling checks, not full-model or Python-wheel integration.

With both follow-ups applied to the pre-restack overlay, the live frozen eight-row production package matches the previous runtime bitwise for all 891 complete ordinary outputs and all 891 k=7 outputs, including logits, hidden outputs and every KV/conv/SSM state. All outputs are finite; generated sequences and speculative counters match exactly. Ordinary generation produces [1784,3330,17000,1639,1317,15047,1261,6165]. No threshold or reference was loosened. This establishes preserved-control parity for this package, not independent HF parity or MoE-only full-model acceptance.

Reproduction uses qualify_coworker_runtime.py --attempt 8 --build-attempt 6 --component all --checks memcheck --lazy-modules and qualify_coworker_model.py --attempt 7 --build-attempt 6 --package-attempt 3 --focused-attempt 8, with explicitly selected idle H100 GPU UUIDs. Source snapshots, exact commands, hashes, provider maps, token sequences and counters are retained in COWORKER_RUNTIME_VALIDATION_20261107_V8_ALL.json, COWORKER_FULL_MODEL_20261107_V7.json and COWORKER_POST_CORRECTION_VALIDATION_20261107_V2.json.

Restacked-source validation: head b9b776b540b15a2c73e41b2559c1f1d662654206 passes production-flag compilation of the CUDA implementation, host registration, routing tests and combined contrib schemas. This does not qualify the linked runtime: the new parent supplies a changed shared-header context, and current-head runtime/full-model requalification is NOT_RUN. The earlier GPU evidence is retained as pre-restack evidence, not relabeled.

Combined executing CUDA provider SHA256: 3b83346bbc1034fffe876d8efadc5bee4810e9b9df336301bfc1e4c160593636.

The matching Mobius exporter implementation is published at 740cbc6f, on rui-ren/bolt-coworker-packed-abi-20261107. Previous exporter branches remain unchanged.

Draft / remaining gates: pre-restack qualification used a CUDA-operator overlay on the previously qualified core, not a clean whole-latest-PR runtime build. Post-restack linked-runtime/full-model requalification is NOT_RUN. The full package is a lossless ABI rewrite with unchanged original initializers, not a fresh full-checkpoint Mobius export. Four paired-TPS attempts, including serialized execution with the original zero-core-dump policy, stopped at the unchanged 128 GiB persistent-storage guard during old-baseline execution; no complete paired TPS or performance promotion is claimed. Original implementations, packages, runtimes, the pre-restack MoE head and numerical/TPS evidence remain preserved. No GenAI source changes, merge, auto-merge or ready-for-review promotion are included.

[Rui Ren (rui-ren)]

Source-pinned H100 kernel evidence - 2026-11-07

BF16, IQ2_XXS, ReLU-square experts, latent hidden dimension 1024, top-k 9. Internal softmax routing versus preselected FP32 indices/weights using identical routes and the same corrected expert arithmetic. Selected logits are +16 and others -16; selected weights are 1/9. Shared/distinct labels describe expert reuse across rows (the one-row variants are equivalent workload controls). Sixteen rows create 144 routes, crossing the 128-route prefill-tiles dispatch threshold.

The ratio measures routing-path overhead removed, not this entire PR versus its parent. All 24 complete BF16 outputs are bitwise equal in both runs. Benefits diminish for distinct-expert, bandwidth-heavy batches; all cases, including those smaller gains, are shown.

Intermediate	Experts	Rows	Routes across rows	Internal routing us	Preselected us	Ratio
1536	240	1	distinct	54.71	41.35	1.32x
1536	240	1	shared	53.31	40.51	1.32x
1536	240	8	distinct	205.41	190.60	1.08x
1536	240	8	shared	56.53	43.36	1.30x
1536	240	16	distinct	396.04	381.20	1.04x
1536	240	16	shared	94.66	80.74	1.17x
2048	240	1	distinct	63.38	49.42	1.28x
2048	240	1	shared	62.54	48.42	1.29x
2048	240	8	distinct	252.80	238.01	1.06x
2048	240	8	shared	64.43	51.36	1.25x
2048	240	16	distinct	495.45	480.54	1.03x
2048	240	16	shared	112.66	99.46	1.13x
2304	432	1	distinct	79.56	57.62	1.38x
2304	432	1	shared	77.32	56.37	1.37x
2304	432	8	distinct	322.00	297.79	1.08x
2304	432	8	shared	82.72	61.52	1.34x
2304	432	16	distinct	670.85	653.92	1.03x
2304	432	16	shared	182.34	161.18	1.13x
2560	432	1	distinct	80.40	59.66	1.35x
2560	432	1	shared	78.78	58.72	1.34x
2560	432	8	distinct	325.76	308.10	1.06x
2560	432	8	shared	82.50	61.28	1.35x
2560	432	16	distinct	689.52	666.84	1.03x
2560	432	16	shared	196.47	175.37	1.12x

H100 80GB HBM3, CUDA 12.8.93, driver 580.159.04, GCC 11.4.0. CUDA-event median per dispatch; 64 dispatches per captured CUDA graph, 50 warmups, three graph warmups, and 15 paired timed batches alternating AB/BA. Four rotating allocation banks contain deterministic synthetic weights (seed 617); this is not a cold-cache guarantee or a checkpoint-weight benchmark. Raw p95 values describe batch-average latency, not individual-invocation tails. No profiler runs in the timing window. The assigned GPU had empty admission, one stable compute client during execution, and clean release afterward; clocks were not locked. Two independent 39-case runs passed, including the published replay runner.

Scope: same-build production CUDA dispatch alternatives, not historical parent-commit binaries. The measured runtime is an isolated CUDA provider relink with preserved core/support, not a clean full-source ORT build or end-to-end model qualification. Each measured operator's CUDA source hash matches its owning PR head. A clean source build is a separate replication variant and must be measured again. Kernel-only timings do not establish model TTFT/tok/s, MMLU/LCB, speculative integration, or full-model readiness.

Source/runtime hashes and supervision receipts

{
  "operator_sources": {
    "projection": {
      "head": "ca5bf6a4c1308b4374238b37884d07edf4c3bae5",
      "cuda_sha256": "9df42138bbe8cbd40114307053398db59f64a5ab25908361a8a655230ce90e71"
    },
    "moe": {
      "head": "a9bdbce374a3b2757e1fd11ef83ce6f83502b0d9",
      "cuda_sha256": "7d9a1454e416753b61342cd78c3ba11f90308320d58f96cc5361c6bfa6ffa486"
    },
    "mamba": {
      "head": "374c7d51d93bfdd29cb0f2efbc76b88e1e812aa7",
      "cuda_sha256": "5e2568fe8aa5794456bf84f122b2e50020f050572043d578fb736ea03796923a"
    }
  },
  "loaded_library_sha256": {
    "libonnxruntime.so.1.32.0": "8e08a5f9b0ae8e67f31bbeaea1942a14f97d2a568d62f1e6b13ad72fcc187d4c",
    "libonnxruntime_providers_cuda.so": "c1b303583039d412416a5aabc04c1748a2d5433ce691c4aa10b9d58ab3d76e0a",
    "libonnxruntime_providers_shared.so": "98cc35d20a8701b425d3b51ebbbd9186744c9abcec5d2270879f6c165a735228"
  },
  "cuda": "12.8.93",
  "driver": "580.159.04",
  "host_compiler": "GCC 11.4.0",
  "device": "NVIDIA H100 80GB HBM3",
  "seed": 617,
  "allocation_banks": 4,
  "primary_supervisor": {
    "status": "PASS",
    "worker_exit": 0,
    "peak_process_bytes": 3147825152
  },
  "repeat_supervisor": {
    "status": "PASS",
    "worker_exit": 0,
    "peak_process_bytes": 3147825152
  },
  "scope": "Isolated production CUDA dispatch microbenchmark; recorded production objects/flags linked with preserved ORT support, NOT a clean full runtime or end-to-end model benchmark.",
  "cuda_and_host_dependency_sha256": {
    "libcudnn_graph.so.9": "2e8a0e4e66c0c8f9dc273c486698614c0561832626fadf76a8a37b94d0e2f895",
    "libcudnn.so.9": "0f43113ec6b171f93236560541f3b95104fed2516b19000a0e7047a5cc12f8ee",
    "libcuda.so.580.159.04": "821992937649fe6b01b0633fa4f47665ff5c02f273ea4696763ad19154cb7c55",
    "libcudart.so.12.8.90": "c3a75b33af334a3486d197dbd1584a2985183ba4688d237a2be5f2f679329920",
    "libcublas.so.12.8.4.1": "031ce6c2cbfbb9468f040527cab5c599069ce5609e73e28f87503881063eac21",
    "libcublasLt.so.12.8.4.1": "10b5e6631cf8115c661eb895ed1533826308b58f7956466f53d236a40c9b622c",
    "libstdc++.so.6": "1fd75fe70354a416d75aef22bcae68c47bd25d20e2d0568c30b1a9838cf62f11",
    "libc.so.6": "3a15d66867d83762c7f2f1e37359cb8f6c5743edb369c65285cb0b1c4f7498bf"
  },
  "production_cuda_codegen_options": [
    "-O3",
    "-std=c++20",
    "--generate-code=arch=compute_90a,code=[sm_90a]"
  ]
}

Primary run: all cases, numerical metrics and all paired latency samples

{"pr":33963,"op":"MoEPacked","rows":1,"hidden":1024,"inter":1536,"experts":240,"topk":9,"shared":false,"banks":4,"correctness":{"nrmse":0,"cosine":1,"max_abs":0,"bitwise":true},"baseline":{"median_us":54.714001715,"batch_p95_us":54.76350151,"trials_us":[54.736498743,54.788000882,54.721999913,54.753001779,54.721999913,54.714001715,54.724998772,54.667498916,54.724998772,54.624501616,54.689500481,53.364999592,53.33949998,53.307000548,53.416501731]},"current":{"median_us":41.353501379,"batch_p95_us":41.417748854,"trials_us":[41.384998709,41.398499161,41.312500834,41.419498622,41.38199985,41.353501379,41.349500418,41.414000094,41.416998953,41.373498738,41.101001203,40.514998138,40.489498526,40.542498231,40.538001806]}}
{"pr":33963,"op":"MoEPacked","rows":1,"hidden":1024,"inter":1536,"experts":240,"topk":9,"shared":true,"banks":4,"correctness":{"nrmse":0,"cosine":1,"max_abs":0,"bitwise":true},"baseline":{"median_us":53.314499557,"batch_p95_us":53.367898613,"trials_us":[53.35899815,53.314499557,53.346499801,53.316999227,53.328998387,53.343001753,53.383998573,53.360998631,53.112998605,52.875000983,52.908498794,52.913501859,52.906498313,52.845999599,52.888501436]},"current":{"median_us":40.507499129,"batch_p95_us":40.636901185,"trials_us":[40.674000978,40.566999465,40.543001145,40.577501059,40.621001273,40.591999888,40.507499129,40.543999523,40.242001414,40.177501738,40.247000754,40.233001113,40.203999728,40.266498923,40.228001773]}}
{"pr":33963,"op":"MoEPacked","rows":8,"hidden":1024,"inter":1536,"experts":240,"topk":9,"shared":false,"banks":4,"correctness":{"nrmse":0,"cosine":1,"max_abs":0,"bitwise":true},"baseline":{"median_us":205.412507057,"batch_p95_us":205.464103818,"trials_us":[205.483004451,205.4104954,205.35799861,205.32900095,205.412507057,205.434501171,205.344006419,205.434501171,205.351501703,205.411493778,205.385997891,205.429494381,205.438002944,205.456003547,205.4399997]},"current":{"median_us":190.604999661,"batch_p95_us":190.725697577,"trials_us":[190.756499767,190.525993705,190.551996231,190.670996904,190.60100615,190.673500299,190.518006682,190.706506371,190.623998642,190.522000194,190.604999661,190.556004643,190.585494041,190.712496638,190.623000264]}}
{"pr":33963,"op":"MoEPacked","rows":8,"hidden":1024,"inter":1536,"experts":240,"topk":9,"shared":true,"banks":4,"correctness":{"nrmse":0,"cosine":1,"max_abs":0,"bitwise":true},"baseline":{"median_us":56.53449893,"batch_p95_us":56.628049538,"trials_us":[56.579001248,56.487999856,56.432999671,56.602999568,56.53449893,56.552000344,56.347500533,56.542001665,56.559998542,56.665498763,56.439001113,56.504998356,56.5200001,56.611999869,56.494001299]},"current":{"median_us":43.356001377,"batch_p95_us":43.435349315,"trials_us":[43.356999755,43.264001608,43.365500867,43.364498764,43.483998626,43.414499611,43.264999986,43.309498578,43.364498764,43.386500329,43.356001377,43.269500136,43.330501765,43.342500925,43.293498456]}}
{"pr":33963,"op":"MoEPacked","rows":16,"hidden":1024,"inter":1536,"experts":240,"topk":9,"shared":false,"banks":4,"correctness":{"nrmse":0,"cosine":1,"max_abs":0,"bitwise":true},"baseline":{"median_us":396.042495966,"batch_p95_us":397.37970531,"trials_us":[400.231510401,396.037012339,396.157503128,396.01251483,396.067500114,396.058499813,396.003991365,396.061509848,396.039485931,396.07000351,396.021991968,396.026998758,396.091490984,396.037012339,396.042495966]},"current":{"median_us":381.204009056,"batch_p95_us":382.775253057,"trials_us":[384.99251008,381.825000048,381.15799427,381.161510944,381.246507168,381.195485592,381.237506866,381.206989288,381.22099638,381.143510342,381.179988384,381.152004004,381.223499775,381.161004305,381.204009056]}}
{"pr":33963,"op":"MoEPacked","rows":16,"hidden":1024,"inter":1536,"experts":240,"topk":9,"shared":true,"banks":4,"correctness":{"nrmse":0,"cosine":1,"max_abs":0,"bitwise":true},"baseline":{"median_us":94.662003219,"batch_p95_us":94.726397842,"trials_us":[94.728499651,94.664499164,94.622500241,94.725497067,94.653502107,94.628997147,94.662003219,94.680503011,94.618998468,94.691000879,94.64699775,94.668000937,94.647496939,94.718001783,94.65649724]},"current":{"median_us":80.738000572,"batch_p95_us":80.777700245,"trials_us":[80.771498382,80.721996725,80.763496459,80.775000155,80.756500363,80.714501441,80.76749742,80.723002553,80.72900027,80.737002194,80.784000456,80.744996667,80.733001232,80.738000572,80.73399961]}}
{"pr":33963,"op":"MoEPacked","rows":1,"hidden":1024,"inter":2048,"experts":240,"topk":9,"shared":false,"banks":4,"correctness":{"nrmse":0,"cosine":1,"max_abs":0,"bitwise":true},"baseline":{"median_us":63.381001353,"batch_p95_us":63.671547174,"trials_us":[64.181499183,63.377998769,63.381001353,63.353002071,63.417002559,63.409999013,63.3585006,63.343003392,63.380502164,63.412502408,63.406996429,63.358999789,63.452996314,63.323996961,63.395000994]},"current":{"median_us":49.424499273,"batch_p95_us":49.601699784,"trials_us":[49.779500812,49.38749969,49.476999789,49.335498363,49.394998699,49.428999424,49.456000328,49.424499273,49.412999302,49.422498792,49.525499344,49.412000924,49.486998469,49.430999905,49.054998904]}}
{"pr":33963,"op":"MoEPacked","rows":1,"hidden":1024,"inter":2048,"experts":240,"topk":9,"shared":true,"banks":4,"correctness":{"nrmse":0,"cosine":1,"max_abs":0,"bitwise":true},"baseline":{"median_us":62.535502017,"batch_p95_us":62.619448453,"trials_us":[62.637999654,62.597498298,62.52399832,62.554001808,62.535502017,62.611497939,62.541499734,62.578000128,62.54799664,62.063001096,62.109000981,62.127500772,62.075998634,62.100999057,62.146998942]},"current":{"median_us":48.418499529,"batch_p95_us":48.523400724,"trials_us":[48.509001732,48.414498568,48.418499529,48.556998372,48.501998186,48.454001546,48.478499055,48.464000225,48.448499292,48.354499042,48.050999641,48.113498837,48.076998442,48.110999167,48.152498901]}}
{"pr":33963,"op":"MoEPacked","rows":8,"hidden":1024,"inter":2048,"experts":240,"topk":9,"shared":false,"banks":4,"correctness":{"nrmse":0,"cosine":1,"max_abs":0,"bitwise":true},"baseline":{"median_us":252.797007561,"batch_p95_us":252.847740054,"trials_us":[252.828985453,252.813011408,252.783507109,252.828985453,252.772003412,252.718508244,252.809494734,252.797007561,252.824991941,252.837985754,252.68599391,252.697497606,252.718001604,252.870500088,252.716511488]},"current":{"median_us":238.014504313,"batch_p95_us":238.075254858,"trials_us":[237.899005413,237.952500582,238.073006272,238.052994013,238.037496805,238.014504313,237.982004881,237.994998693,238.017499447,237.975001335,238.027498126,237.995997071,238.080501556,237.930998206,238.038003445]}}
{"pr":33963,"op":"MoEPacked","rows":8,"hidden":1024,"inter":2048,"experts":240,"topk":9,"shared":true,"banks":4,"correctness":{"nrmse":0,"cosine":1,"max_abs":0,"bitwise":true},"baseline":{"median_us":64.428001642,"batch_p95_us":64.717750251,"trials_us":[64.562499523,64.478002489,64.341001213,64.437501132,64.418502152,64.493499696,64.363501966,65.08000195,64.525000751,64.412496984,64.413003623,64.428001642,64.441002905,64.404003322,64.359501004]},"current":{"median_us":51.362000406,"batch_p95_us":51.63714923,"trials_us":[51.38900131,51.339000463,51.347501576,51.383998245,51.337499171,51.43449828,52.110001445,51.366999745,51.378499717,51.42249912,51.344998181,51.346499473,51.362000406,51.274001598,51.33099854]}}
{"pr":33963,"op":"MoEPacked","rows":16,"hidden":1024,"inter":2048,"experts":240,"topk":9,"shared":false,"banks":4,"correctness":{"nrmse":0,"cosine":1,"max_abs":0,"bitwise":true},"baseline":{"median_us":495.451003313,"batch_p95_us":496.883350611,"trials_us":[500.025987625,494.74349618,495.332509279,495.482504368,495.536506176,495.435506105,495.503008366,495.397508144,495.451003313,495.470494032,495.438486338,495.435506105,495.53000927,495.42298913,495.455503464]},"current":{"median_us":480.535507202,"batch_p95_us":482.087543607,"trials_us":[484.900504351,480.881989002,480.628997087,480.51649332,480.535507202,480.498999357,480.538487434,480.585992336,480.51199317,480.54549098,480.571508408,480.489999056,480.525493622,480.524003506,480.525493622]}}
{"pr":33963,"op":"MoEPacked","rows":16,"hidden":1024,"inter":2048,"experts":240,"topk":9,"shared":true,"banks":4,"correctness":{"nrmse":0,"cosine":1,"max_abs":0,"bitwise":true},"baseline":{"median_us":112.655997276,"batch_p95_us":113.029748946,"trials_us":[112.847499549,113.45499754,112.742498517,112.72650212,112.771503627,112.675003707,112.650997937,112.644001842,112.631998956,112.620003521,112.61600256,112.634003162,112.629503012,112.655997276,112.659499049]},"current":{"median_us":99.460996687,"batch_p95_us":99.662648141,"trials_us":[99.515497684,99.511496723,100.005999207,99.490500987,99.496997893,99.480502307,99.411502481,99.466502666,99.435500801,99.377498031,99.460996687,99.435500801,99.381498992,99.399000406,99.432498217]}}
{"pr":33963,"op":"MoEPacked","rows":1,"hidden":1024,"inter":2304,"experts":432,"topk":9,"shared":false,"banks":4,"correctness":{"nrmse":0,"cosine":1,"max_abs":0,"bitwise":true},"baseline":{"median_us":79.559497535,"batch_p95_us":79.674852639,"trials_us":[79.695500433,79.655997455,79.561501741,79.563997686,79.569503665,79.505503178,79.563498497,79.666003585,79.559497535,78.007496893,77.998496592,78.0095011,78.020498157,78.037500381,78.019499779]},"current":{"median_us":57.619001716,"batch_p95_us":57.67140016,"trials_us":[57.661000639,57.66050145,57.588998228,57.647500187,57.642500848,57.665999979,57.684000582,57.640999556,57.619001716,57.019498199,56.814998388,56.881498545,56.832499802,56.914500892,56.837998331]}}
{"pr":33963,"op":"MoEPacked","rows":1,"hidden":1024,"inter":2304,"experts":432,"topk":9,"shared":true,"banks":4,"correctness":{"nrmse":0,"cosine":1,"max_abs":0,"bitwise":true},"baseline":{"median_us":77.320002019,"batch_p95_us":77.554950118,"trials_us":[77.95149833,77.385000885,77.233500779,77.353000641,77.320002019,77.345997095,77.321998775,77.283002436,77.212497592,77.218003571,77.293999493,77.279999852,77.256001532,77.325999737,77.358499169]},"current":{"median_us":56.373998523,"batch_p95_us":56.837899238,"trials_us":[56.875001639,56.821998209,56.347500533,56.389499456,56.3150011,56.386999786,56.386001408,56.373998523,56.354001164,56.407500058,56.393500417,56.3150011,56.361500174,56.366000324,56.340001523]}}
{"pr":33963,"op":"MoEPacked","rows":8,"hidden":1024,"inter":2304,"experts":432,"topk":9,"shared":false,"banks":4,"correctness":{"nrmse":0,"cosine":1,"max_abs":0,"bitwise":true},"baseline":{"median_us":321.999996901,"batch_p95_us":327.073353529,"trials_us":[326.173990965,317.896008492,317.875504494,326.22051239,326.108008623,321.512490511,317.802011967,317.808508873,321.999996901,326.33548975,326.393008232,317.892998457,317.869514227,326.844006777,327.608495951]},"current":{"median_us":297.794997692,"batch_p95_us":302.658894658,"trials_us":[297.478497028,295.025497675,294.955998659,298.752009869,302.298486233,302.130490541,295.046508312,295.058488846,302.451491356,302.570998669,298.586010933,294.901490211,294.997990131,297.794997692,302.8639853]}}
{"pr":33963,"op":"MoEPacked","rows":8,"hidden":1024,"inter":2304,"experts":432,"topk":9,"shared":true,"banks":4,"correctness":{"nrmse":0,"cosine":1,"max_abs":0,"bitwise":true},"baseline":{"median_us":82.718998194,"batch_p95_us":82.849451154,"trials_us":[82.839496434,82.684502006,82.740999758,82.718998194,82.868002355,82.84150064,82.824997604,82.777000964,82.722000778,82.433998585,82.049496472,82.175500691,82.105502486,82.14700222,82.076497376]},"current":{"median_us":61.516501009,"batch_p95_us":61.638549343,"trials_us":[61.632998288,61.651501805,61.512500048,61.551999301,61.529498547,61.567500234,61.565998942,61.505500227,61.516501009,61.542000622,61.088498682,61.183501035,61.114501208,61.064001173,61.163499951]}}
{"pr":33963,"op":"MoEPacked","rows":16,"hidden":1024,"inter":2304,"experts":432,"topk":9,"shared":false,"banks":4,"correctness":{"nrmse":0,"cosine":1,"max_abs":0,"bitwise":true},"baseline":{"median_us":670.848488808,"batch_p95_us":678.508472443,"trials_us":[669.746994972,663.124024868,663.083970547,673.277974129,663.573503494,678.454518318,669.561982155,677.990496159,678.008019924,670.848488808,663.513481617,678.005993366,666.549503803,678.511977196,678.506970406]},"current":{"median_us":653.924524784,"batch_p95_us":654.88781929,"trials_us":[654.899001122,654.09052372,650.030016899,654.347479343,640.762507915,653.924524784,640.812516212,643.269002438,654.405474663,654.468476772,642.328977585,654.348015785,640.416026115,645.613491535,654.883027077]}}
{"pr":33963,"op":"MoEPacked","rows":16,"hidden":1024,"inter":2304,"experts":432,"topk":9,"shared":true,"banks":4,"correctness":{"nrmse":0,"cosine":1,"max_abs":0,"bitwise":true},"baseline":{"median_us":182.336002588,"batch_p95_us":182.900595665,"trials_us":[182.899996638,182.901993394,182.874500751,182.370498776,182.328999043,182.346493006,182.270005345,182.292506099,182.252004743,182.38799274,182.291001081,182.331994176,182.334005833,182.425007224,182.336002588]},"current":{"median_us":161.176502705,"batch_p95_us":161.659252644,"trials_us":[161.639496684,161.648005247,161.685496569,161.176502705,161.235004663,161.15899384,161.117002368,161.141499877,161.257997155,161.157503724,161.219000816,161.197006702,161.15449369,161.174997687,161.149993539]}}
{"pr":33963,"op":"MoEPacked","rows":1,"hidden":1024,"inter":2560,"experts":432,"topk":9,"shared":false,"banks":4,"correctness":{"nrmse":0,"cosine":1,"max_abs":0,"bitwise":true},"baseline":{"median_us":80.395497382,"batch_p95_us":81.357548386,"trials_us":[81.363499165,81.354998052,81.265501678,80.43000102,80.414496362,80.364502966,80.395497382,80.385498703,80.358996987,80.416999757,80.332502723,80.403998494,80.362997949,80.384999514,79.136997461]},"current":{"median_us":59.660501778,"batch_p95_us":60.136950761,"trials_us":[60.239501297,60.093000531,60.001499951,59.762500226,59.741500765,59.627499431,59.660501778,59.641998261,59.698499739,59.632498771,59.701498598,59.645000845,59.622500092,59.63049829,58.879498392]}}
{"pr":33963,"op":"MoEPacked","rows":1,"hidden":1024,"inter":2560,"experts":432,"topk":9,"shared":true,"banks":4,"correctness":{"nrmse":0,"cosine":1,"max_abs":0,"bitwise":true},"baseline":{"median_us":78.776501119,"batch_p95_us":79.069699347,"trials_us":[79.107500613,79.047001898,79.037003219,79.048499465,79.053498805,79.045496881,78.754998744,78.763999045,78.69400084,78.776501119,78.7229985,78.769497573,78.745499253,78.810997307,78.764498234]},"current":{"median_us":58.71900171,"batch_p95_us":58.853299543,"trials_us":[58.797501028,58.857500553,58.829501271,58.807499707,58.825999498,58.85149911,58.598499745,58.683000505,58.696500957,58.71900171,58.752000332,58.701500297,58.674000204,58.667998761,58.614000678]}}
{"pr":33963,"op":"MoEPacked","rows":8,"hidden":1024,"inter":2560,"experts":432,"topk":9,"shared":false,"banks":4,"correctness":{"nrmse":0,"cosine":1,"max_abs":0,"bitwise":true},"baseline":{"median_us":325.75750351,"batch_p95_us":334.487098455,"trials_us":[334.385514259,325.675994158,325.55949688,325.446486473,325.75750351,334.684491158,333.083987236,325.455009937,325.398504734,334.324508905,334.353506565,325.435996056,325.455486774,328.507512808,334.402501583]},"current":{"median_us":308.095008135,"batch_p95_us":310.756349564,"trials_us":[310.487002134,310.462504625,302.612513304,302.59001255,310.829490423,310.725003481,302.516013384,302.61901021,304.046005011,310.451000929,310.427486897,308.095008135,302.61400342,302.554011345,310.369491577]}}
{"pr":33963,"op":"MoEPacked","rows":8,"hidden":1024,"inter":2560,"experts":432,"topk":9,"shared":true,"banks":4,"correctness":{"nrmse":0,"cosine":1,"max_abs":0,"bitwise":true},"baseline":{"median_us":82.501500845,"batch_p95_us":83.308201283,"trials_us":[85.040003061,82.527503371,82.432001829,82.516998053,82.526497543,82.566000521,82.464501262,82.54750073,82.425497472,82.534499466,82.501500845,82.318000495,81.803999841,81.881500781,81.877499819]},"current":{"median_us":61.279501766,"batch_p95_us":61.836950108,"trials_us":[62.74099648,61.449501663,61.279501766,61.290498823,61.364501715,61.256501824,61.399500817,61.266500503,61.289999634,61.286501586,61.270501465,61.227999628,60.86749956,60.994498432,60.962498188]}}
{"pr":33963,"op":"MoEPacked","rows":16,"hidden":1024,"inter":2560,"experts":432,"topk":9,"shared":false,"banks":4,"correctness":{"nrmse":0,"cosine":1,"max_abs":0,"bitwise":true},"baseline":{"median_us":689.517498016,"batch_p95_us":702.397239208,"trials_us":[685.881018639,684.118509293,684.156477451,696.353971958,686.49148941,702.409505844,689.517498016,701.856017113,698.754489422,684.524476528,684.15248394,697.081506252,686.456501484,702.391982079,690.101504326]},"current":{"median_us":666.837990284,"batch_p95_us":678.702801466,"trials_us":[661.427974701,661.364018917,672.85400629,678.66897583,664.840996265,678.710520267,663.734972477,670.783996582,666.837990284,666.828513145,672.095477581,678.699493408,664.291977882,678.677499294,663.816511631]}}
{"pr":33963,"op":"MoEPacked","rows":16,"hidden":1024,"inter":2560,"experts":432,"topk":9,"shared":true,"banks":4,"correctness":{"nrmse":0,"cosine":1,"max_abs":0,"bitwise":true},"baseline":{"median_us":196.47449255,"batch_p95_us":196.507802606,"trials_us":[196.482494473,196.504503489,196.47449255,196.449995041,196.468502283,196.442499757,196.48000598,196.515500546,196.483999491,196.46500051,196.499496698,196.419999003,196.464002132,196.487501264,196.448504925]},"current":{"median_us":175.374493003,"batch_p95_us":175.427144766,"trials_us":[175.391495228,175.362497568,175.379499793,175.288498402,175.402000546,175.355002284,175.394505262,175.36200583,175.345003605,175.347507,175.367504358,175.374493003,175.455495715,175.399497151,175.414994359]}}

Independent replay: all cases, numerical metrics and all paired latency samples

{"pr":33963,"op":"MoEPacked","rows":1,"hidden":1024,"inter":1536,"experts":240,"topk":9,"shared":false,"banks":4,"correctness":{"nrmse":0,"cosine":1,"max_abs":0,"bitwise":true},"baseline":{"median_us":54.374501109,"batch_p95_us":55.055799335,"trials_us":[55.09499833,54.977498949,54.993998259,55.038999766,54.944999516,54.543498904,54.336000234,54.333500564,54.345000535,54.235998541,54.288998246,54.374501109,54.324999452,54.386999458,54.338000715]},"current":{"median_us":41.326500475,"batch_p95_us":41.705400497,"trials_us":[41.721500456,41.698500514,41.634000838,41.652999818,41.688501835,41.684001684,41.324499995,41.291501373,41.285000741,41.306998581,41.326500475,41.335999966,41.313998401,41.258499026,41.299000382]}}
{"pr":33963,"op":"MoEPacked","rows":1,"hidden":1024,"inter":1536,"experts":240,"topk":9,"shared":true,"banks":4,"correctness":{"nrmse":0,"cosine":1,"max_abs":0,"bitwise":true},"baseline":{"median_us":53.417999297,"batch_p95_us":54.332950711,"trials_us":[54.47050184,54.274000227,53.702499717,53.42650041,53.366500884,53.397499025,53.449999541,53.401999176,53.330998868,53.387001157,53.465500474,53.403001279,53.390499204,53.420498967,53.417999297]},"current":{"median_us":40.784500539,"batch_p95_us":41.390099376,"trials_us":[41.402000934,41.384998709,40.784500539,40.783498436,40.769498795,40.848501027,40.79850018,40.81299901,40.784500539,40.754999965,40.775999427,40.849998593,40.722999722,40.801499039,40.734000504]}}
{"pr":33963,"op":"MoEPacked","rows":8,"hidden":1024,"inter":1536,"experts":240,"topk":9,"shared":false,"banks":4,"correctness":{"nrmse":0,"cosine":1,"max_abs":0,"bitwise":true},"baseline":{"median_us":205.401003361,"batch_p95_us":205.503247678,"trials_us":[205.347001553,205.39650321,205.414503813,205.38200438,205.455005169,205.455496907,205.452501774,205.360993743,205.515503883,205.392494798,205.362007022,205.42499423,205.399006605,205.401003361,205.497995019]},"current":{"median_us":190.635994077,"batch_p95_us":190.772049129,"trials_us":[190.628498793,190.62499702,190.770998597,190.660506487,190.651506186,190.688997507,190.635994077,190.610006452,190.599501133,190.544992685,190.563499928,190.64450264,190.717995167,190.63000381,190.77450037]}}
{"pr":33963,"op":"MoEPacked","rows":8,"hidden":1024,"inter":1536,"experts":240,"topk":9,"shared":true,"banks":4,"correctness":{"nrmse":0,"cosine":1,"max_abs":0,"bitwise":true},"baseline":{"median_us":56.512501091,"batch_p95_us":56.614299864,"trials_us":[56.513499469,56.566499174,56.464500725,56.600499898,56.542500854,56.646499783,56.472498924,56.464999914,56.372001767,56.584998965,56.511998177,56.462500244,56.455500424,56.512501091,56.582000107]},"current":{"median_us":43.324500322,"batch_p95_us":43.449398875,"trials_us":[43.297499418,43.29650104,43.384999037,43.340001255,43.442498893,43.324500322,43.288998306,43.366000056,43.223001063,43.423000723,43.267000467,43.303500861,43.223999441,43.465498835,43.440498412]}}
{"pr":33963,"op":"MoEPacked","rows":16,"hidden":1024,"inter":1536,"experts":240,"topk":9,"shared":false,"banks":4,"correctness":{"nrmse":0,"cosine":1,"max_abs":0,"bitwise":true},"baseline":{"median_us":397.146493196,"batch_p95_us":405.727794766,"trials_us":[404.707491398,397.935003042,397.338986397,396.049499512,396.019011736,396.062999964,399.404495955,408.108502626,404.329001904,397.808492184,397.146493196,395.956486464,396.046996117,396.032512188,396.075993776]},"current":{"median_us":381.209999323,"batch_p95_us":392.336460948,"trials_us":[382.901489735,382.882505655,381.217509508,381.190508604,381.156504154,381.154000759,392.358511686,392.327010632,382.87499547,382.831007242,381.117999554,381.182491779,381.166994572,381.209999323,381.141006947]}}
{"pr":33963,"op":"MoEPacked","rows":16,"hidden":1024,"inter":1536,"experts":240,"topk":9,"shared":true,"banks":4,"correctness":{"nrmse":0,"cosine":1,"max_abs":0,"bitwise":true},"baseline":{"median_us":94.654999673,"batch_p95_us":94.702199847,"trials_us":[94.676502049,94.691000879,94.610996544,94.678997993,94.652496278,94.646498561,94.635501504,94.663001597,94.596996903,94.70500052,94.595000148,94.700999558,94.654999673,94.663500786,94.617500901]},"current":{"median_us":80.737501383,"batch_p95_us":80.764549226,"trials_us":[80.759502947,80.732502043,80.7434991,80.677002668,80.741502345,80.744497478,80.738499761,80.715000629,80.718502402,80.733001232,80.727502704,80.761998892,80.770500004,80.737501383,80.696001649]}}
{"pr":33963,"op":"MoEPacked","rows":1,"hidden":1024,"inter":2048,"experts":240,"topk":9,"shared":false,"banks":4,"correctness":{"nrmse":0,"cosine":1,"max_abs":0,"bitwise":true},"baseline":{"median_us":63.193500042,"batch_p95_us":64.350598305,"trials_us":[64.411498606,64.324498177,64.31800127,63.199996948,63.166998327,63.188999891,63.124999404,63.171498477,63.161998987,63.193500042,63.199996948,63.248500228,63.19899857,63.10749799,63.176497817]},"current":{"median_us":49.394000322,"batch_p95_us":50.128300488,"trials_us":[50.091501325,50.153501332,50.117500126,50.113998353,49.318999052,49.350000918,49.438498914,49.355000257,49.387998879,49.373999238,49.385499209,49.390498549,49.394000322,49.395501614,49.398999661]}}
{"pr":33963,"op":"MoEPacked","rows":1,"hidden":1024,"inter":2048,"experts":240,"topk":9,"shared":true,"banks":4,"correctness":{"nrmse":0,"cosine":1,"max_abs":0,"bitwise":true},"baseline":{"median_us":62.375999987,"batch_p95_us":62.456549704,"trials_us":[62.472999096,62.449499965,62.426999211,62.383998185,62.411498278,62.327001244,62.408000231,62.39400059,62.375999987,62.375500798,62.365498394,62.306001782,62.302000821,62.235500664,62.210001051]},"current":{"median_us":48.215001822,"batch_p95_us":48.266201466,"trials_us":[48.263501376,48.239499331,48.215001822,48.272501677,48.165500164,48.191998154,48.191498965,48.258498311,48.188000917,48.255000263,48.255499452,48.228498548,48.053499311,48.025999218,48.05649817]}}
{"pr":33963,"op":"MoEPacked","rows":8,"hidden":1024,"inter":2048,"experts":240,"topk":9,"shared":false,"banks":4,"correctness":{"nrmse":0,"cosine":1,"max_abs":0,"bitwise":true},"baseline":{"median_us":252.770006657,"batch_p95_us":252.907192707,"trials_us":[252.838492393,252.892494202,252.822011709,252.716988325,252.799987793,252.761989832,252.732008696,252.718508244,252.739995718,252.78198719,252.766489983,252.783000469,252.94148922,252.770006657,252.739995718]},"current":{"median_us":237.983003259,"batch_p95_us":238.035444915,"trials_us":[237.993493676,238.036498427,237.983003259,237.961500883,237.968504429,237.954005599,238.03499341,237.948000431,237.989500165,238.018006086,238.018497825,237.94350028,237.994506955,237.976506352,237.936496735]}}
{"pr":33963,"op":"MoEPacked","rows":8,"hidden":1024,"inter":2048,"experts":240,"topk":9,"shared":true,"banks":4,"correctness":{"nrmse":0,"cosine":1,"max_abs":0,"bitwise":true},"baseline":{"median_us":64.425498247,"batch_p95_us":64.550200105,"trials_us":[64.458496869,64.436003566,64.433500171,64.482502639,64.358502626,64.373999834,64.32300061,64.608998597,64.338497818,64.425498247,64.27449733,64.42400068,64.4050017,64.525000751,64.43399936]},"current":{"median_us":51.317501813,"batch_p95_us":51.385251805,"trials_us":[51.425501704,51.368001848,51.306001842,51.316000521,51.341999322,51.317501813,51.323998719,51.286000758,51.316000521,51.312498748,51.362998784,51.320999861,51.29950121,51.288999617,51.358498633]}}
{"pr":33963,"op":"MoEPacked","rows":16,"hidden":1024,"inter":2048,"experts":240,"topk":9,"shared":false,"banks":4,"correctness":{"nrmse":0,"cosine":1,"max_abs":0,"bitwise":true},"baseline":{"median_us":494.893491268,"batch_p95_us":495.989856124,"trials_us":[497.119992971,494.642496109,494.695514441,494.616985321,494.690001011,494.893491268,494.65149641,494.540005922,494.601488113,494.965493679,495.505511761,495.370000601,495.483011007,495.424509048,495.444506407]},"current":{"median_us":479.931503534,"batch_p95_us":482.06820786,"trials_us":[482.059508562,482.088506222,479.74100709,479.725509882,479.708999395,479.683488607,479.931503534,479.696512222,479.685992002,479.737490416,480.479508638,480.462014675,480.478495359,480.473011732,480.488985777]}}
{"pr":33963,"op":"MoEPacked","rows":16,"hidden":1024,"inter":2048,"experts":240,"topk":9,"shared":true,"banks":4,"correctness":{"nrmse":0,"cosine":1,"max_abs":0,"bitwise":true},"baseline":{"median_us":112.714000046,"batch_p95_us":112.775948644,"trials_us":[112.752497196,112.726002932,112.661503255,112.72650212,112.677998841,112.7069965,112.658500671,112.775497139,112.684503198,112.777002156,112.714000046,112.729497254,112.664498389,112.750500441,112.706497312]},"current":{"median_us":99.471502006,"batch_p95_us":99.506551027,"trials_us":[99.523000419,99.487997591,99.473997951,99.447503686,99.478498101,99.499501288,99.498003721,99.446997046,99.436998367,99.455498159,99.445998669,99.434502423,99.496498704,99.471502006,99.460497499]}}
{"pr":33963,"op":"MoEPacked","rows":1,"hidden":1024,"inter":2304,"experts":432,"topk":9,"shared":false,"banks":4,"correctness":{"nrmse":0,"cosine":1,"max_abs":0,"bitwise":true},"baseline":{"median_us":79.158499837,"batch_p95_us":79.22179848,"trials_us":[79.226002097,79.190999269,79.12299782,79.164996743,79.17650044,79.125002027,79.092502594,79.219996929,79.18599993,79.205997288,79.158499837,77.962502837,78.031502664,78.063003719,78.011997044]},"current":{"median_us":57.399500161,"batch_p95_us":57.478649169,"trials_us":[57.422999293,57.399500161,57.409498841,57.438999414,57.433500886,57.387001812,57.517498732,57.440500706,57.394500822,57.461999357,57.098001242,56.853499264,56.901499629,56.855998933,56.894499809]}}
{"pr":33963,"op":"MoEPacked","rows":1,"hidden":1024,"inter":2304,"experts":432,"topk":9,"shared":true,"banks":4,"correctness":{"nrmse":0,"cosine":1,"max_abs":0,"bitwise":true},"baseline":{"median_us":77.371001244,"batch_p95_us":78.227750957,"trials_us":[78.792996705,77.976502478,77.985502779,77.409997582,77.39199698,77.31500268,77.333502471,77.371001244,77.369503677,77.393002808,77.335000038,77.394001186,77.345497906,77.262498438,77.340498567]},"current":{"median_us":56.43799901,"batch_p95_us":57.058600336,"trials_us":[57.448498905,56.885998696,56.89150095,56.62349984,56.43799901,56.379500777,56.494001299,56.395001709,56.413501501,56.405998766,56.553501636,56.430500001,56.44999817,56.379500777,56.436501443]}}
{"pr":33963,"op":"MoEPacked","rows":8,"hidden":1024,"inter":2304,"experts":432,"topk":9,"shared":false,"banks":4,"correctness":{"nrmse":0,"cosine":1,"max_abs":0,"bitwise":true},"baseline":{"median_us":323.358505964,"batch_p95_us":326.776793599,"trials_us":[326.096504927,317.817509174,317.763000727,324.434012175,326.319485903,326.375991106,319.183498621,317.822009325,317.813485861,326.200485229,326.140999794,317.855507135,317.806005478,323.358505964,327.711999416]},"current":{"median_us":296.90399766,"batch_p95_us":302.519991994,"trials_us":[302.239000797,295.769989491,294.985502958,294.987499714,302.468985319,302.453488111,294.969499111,294.938504696,300.559997559,302.145987749,302.325993776,296.90399766,295.033991337,294.96049881,302.639007568]}}
{"pr":33963,"op":"MoEPacked","rows":8,"hidden":1024,"inter":2304,"experts":432,"topk":9,"shared":true,"banks":4,"correctness":{"nrmse":0,"cosine":1,"max_abs":0,"bitwise":true},"baseline":{"median_us":82.290999591,"batch_p95_us":83.29609856,"trials_us":[85.380002856,82.309499383,82.249000669,82.323499024,82.276500762,82.367002964,82.290999591,82.402996719,82.342997193,82.401499152,82.227498293,82.116000354,82.234501839,82.097500563,81.987001002]},"current":{"median_us":61.228498816,"batch_p95_us":61.323399842,"trials_us":[61.406001449,61.14749983,61.287999153,61.207499355,61.234500259,61.25099957,61.248000711,61.201501638,61.232000589,61.173498631,61.238501221,61.228498816,61.089001596,61.04850024,61.066500843]}}
{"pr":33963,"op":"MoEPacked","rows":16,"hidden":1024,"inter":2304,"experts":432,"topk":9,"shared":false,"banks":4,"correctness":{"nrmse":0,"cosine":1,"max_abs":0,"bitwise":true},"baseline":{"median_us":664.550483227,"batch_p95_us":678.549540043,"trials_us":[664.550483227,667.835474014,663.427472115,677.997529507,663.558483124,678.565979004,674.616992474,661.584973335,661.281526089,677.079975605,663.423001766,678.542494774,672.756493092,661.283016205,661.215007305]},"current":{"median_us":645.83748579,"batch_p95_us":654.943853617,"trials_us":[654.888510704,654.859483242,645.385026932,654.35397625,640.813529491,649.040997028,640.800476074,640.780985355,645.83748579,655.07298708,640.780508518,650.77149868,640.774488449,640.755474567,647.783517838]}}
{"pr":33963,"op":"MoEPacked","rows":16,"hidden":1024,"inter":2304,"experts":432,"topk":9,"shared":true,"banks":4,"correctness":{"nrmse":0,"cosine":1,"max_abs":0,"bitwise":true},"baseline":{"median_us":182.533994317,"batch_p95_us":183.37879926,"trials_us":[183.452993631,183.304995298,183.347001672,183.086499572,182.545006275,182.57650733,182.533994317,182.582005858,182.506993413,182.324498892,182.295501232,182.330995798,182.237997651,182.157501578,182.080507278]},"current":{"median_us":161.442503333,"batch_p95_us":161.968551576,"trials_us":[161.954000592,161.923497915,161.888495088,162.002503872,161.461502314,161.46799922,161.442503333,161.468505859,161.155506968,161.181002855,161.128997803,161.116495728,161.149501801,161.109998822,161.008998752]}}
{"pr":33963,"op":"MoEPacked","rows":1,"hidden":1024,"inter":2560,"experts":432,"topk":9,"shared":false,"banks":4,"correctness":{"nrmse":0,"cosine":1,"max_abs":0,"bitwise":true},"baseline":{"median_us":79.985499382,"batch_p95_us":81.305997819,"trials_us":[81.404000521,81.263996661,81.234000623,81.246502697,81.243999302,80.028496683,79.917497933,79.985499382,79.913496971,79.972997308,79.928003252,79.981997609,79.925999045,79.960502684,79.985998571]},"current":{"median_us":59.60150063,"batch_p95_us":60.184699297,"trials_us":[60.169499367,60.183499008,60.139998794,60.143500566,60.18749997,59.985499829,59.60150063,59.545498341,59.581000358,59.592500329,59.546500444,59.483498335,59.59700048,59.578001499,59.658501297]}}
{"pr":33963,"op":"MoEPacked","rows":1,"hidden":1024,"inter":2560,"experts":432,"topk":9,"shared":true,"banks":4,"correctness":{"nrmse":0,"cosine":1,"max_abs":0,"bitwise":true},"baseline":{"median_us":78.774496913,"batch_p95_us":78.826347739,"trials_us":[78.7955001,78.850500286,78.815996647,78.772999346,78.71799916,78.779503703,78.795999289,78.753501177,78.753001988,78.786000609,78.713998199,78.815996647,78.772500157,78.774496913,78.734003007]},"current":{"median_us":58.692000806,"batch_p95_us":58.742599934,"trials_us":[58.724500239,58.665499091,58.715000749,58.71450156,58.684501797,58.670498431,58.768499643,58.64899978,58.701999485,58.664500713,58.731500059,58.710001409,58.56249854,58.655500412,58.692000806]}}
{"pr":33963,"op":"MoEPacked","rows":8,"hidden":1024,"inter":2560,"experts":432,"topk":9,"shared":false,"banks":4,"correctness":{"nrmse":0,"cosine":1,"max_abs":0,"bitwise":true},"baseline":{"median_us":329.624503851,"batch_p95_us":335.026440024,"trials_us":[325.471997261,326.644986868,333.847492933,334.738999605,333.611011505,325.648486614,325.390487909,334.805488586,334.616005421,325.277507305,325.379490852,328.066498041,335.54199338,334.415495396,329.624503851]},"current":{"median_us":303.640991449,"batch_p95_us":311.12306118,"trials_us":[302.539497614,302.554011345,310.829013586,310.822993517,303.021013737,302.601486444,303.640991449,310.819506645,310.903012753,308.815509081,302.576988935,302.593499422,311.636507511,310.48899889,302.538990974]}}
{"pr":33963,"op":"MoEPacked","rows":8,"hidden":1024,"inter":2560,"experts":432,"topk":9,"shared":true,"banks":4,"correctness":{"nrmse":0,"cosine":1,"max_abs":0,"bitwise":true},"baseline":{"median_us":81.903003156,"batch_p95_us":82.18324855,"trials_us":[82.110002637,82.20949769,82.171998918,81.882499158,81.861503422,81.863999367,81.927500665,81.908501685,81.834502518,81.932999194,81.989496946,81.903003156,81.885501742,81.876002252,81.854499876]},"current":{"median_us":60.931999236,"batch_p95_us":61.007050052,"trials_us":[61.012998223,60.931000859,61.004500836,60.997001827,60.998499393,60.946501791,60.886498541,60.845501721,60.913998634,60.931999236,60.997001827,60.839999467,60.945998877,60.883000493,60.885000974]}}
{"pr":33963,"op":"MoEPacked","rows":16,"hidden":1024,"inter":2560,"experts":432,"topk":9,"shared":false,"banks":4,"correctness":{"nrmse":0,"cosine":1,"max_abs":0,"bitwise":true},"baseline":{"median_us":694.537520409,"batch_p95_us":702.974128723,"trials_us":[686.540484428,694.537520409,686.847507954,703.296482563,687.627017498,702.835977077,696.637511253,684.252977371,684.426009655,695.10102272,686.733007431,702.735483646,688.379526138,702.831506729,697.892487049]},"current":{"median_us":672.545015812,"batch_p95_us":679.047197104,"trials_us":[675.967991352,679.018497467,668.245971203,679.038524628,664.108991623,673.573493958,664.104998112,664.086520672,674.883008003,679.042994976,666.665494442,679.057002068,664.089977741,672.545015812,667.716026306]}}
{"pr":33963,"op":"MoEPacked","rows":16,"hidden":1024,"inter":2560,"experts":432,"topk":9,"shared":true,"banks":4,"correctness":{"nrmse":0,"cosine":1,"max_abs":0,"bitwise":true},"baseline":{"median_us":196.701005101,"batch_p95_us":196.788798273,"trials_us":[196.785494685,196.677997708,196.716502309,196.711495519,196.757003665,196.667999029,196.64350152,196.733504534,196.654498577,196.708500385,196.796506643,196.644499898,196.701005101,196.648001671,196.627005935]},"current":{"median_us":175.594493747,"batch_p95_us":175.662703812,"trials_us":[175.603494048,175.570994616,175.647005439,175.537005067,175.657004118,175.573498011,175.613492727,175.590500236,175.594493747,175.532504916,175.676003098,175.519496202,175.612494349,175.557494164,175.621494651]}}

Benchmark source, synthetic setup graph and validated replay runner: #33962 (comment)

[Rui Ren (rui-ren)]

Standalone Bolt routing is now isolated in draft follow-up #33994 above #33969, extending native stack #33967 without rewriting this PR or the coworker's #32935. The existing integrated route remains compatible and uses the same shared CUDA implementation. The complete fresh production source build passed native tests, logits/cache/state and route comparisons, plus 240 exact same-mode generation/reset/counter checks. The repeated 12k/24k/32k/124k before/after timings and limitations are in #33994; those timings use the frozen extraction build, not a new benchmark of the production refactor. Preserved quantized control is not independent HF, and inherited ordinary-versus-k7 differences remain.

[Rui Ren (rui-ren)]

The native kernel optimizations are now pushed in #34002 and registered as the
new top of merge stack #33967.
This changes the intended merge stack, not alternative #33940. Existing layer
heads and their history are preserved.

Actual #33994-tip baseline versus the new native candidate, real 1028-1663-token
inputs: 54.25 -> 92.04 ordinary tok/s; 201.48 -> 318.91 k=7 tok/s.
Both arms used the same native package and corrected GenAI. All 240 complete
measured generations/counters, 1,782 full live tensor/state hashes per arm and
22 native CUDA regressions passed; no skips.

The earlier ~92 tok/s private baseline already contained these optimizations:
it was not the unmodified remote stack. The separate #33940 ~109/~370 result is
still not a result of this native candidate. That remaining gap is outstanding.
Full protocol, source pins and limits are in #34002; it remains draft.