[Klaud Cold] Add H200 Qwen3.8-27B-FP8 vLLM TP1 native MTP 1k1k / 新增 H200 Qwen3.8-27B-FP8 vLLM TP1 原生 MTP 1k1k 配方 - #3293
Open
functionstackx wants to merge 2 commits into
Open
functionstackx wants to merge 2 commits into
functionstackx wants to merge 2 commits into
Conversation
functionstackx
requested review from
Ankur-singh and
kedarpotdar-nv
as code owners
September 19, 2026 19:55
… 1k1k recipe on H200 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
functionstackx
force-pushed
the
feat/qwen38-27b-fp8-h200-vllm-mtp
branch
from
September 19, 2026 19:55
43f9940 to
c51e82a
Compare
Contributor
|
Thanks for the contribution!
中文感谢你的贡献!
|
Contributor
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=35465800643 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=35465800643 |
Collaborator
Author
|
/reuse-sweep-run 35465800643 |
Sync with origin/main after the green sweep run 35465800643; the reuse gate authorizes that run on this head. 在绿色 sweep 运行 35465800643 之后与 origin/main 同步;reuse gate 在此 head 上授权该运行。 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add
qwen3.827b-fp8-h200-vllm-mtp: Qwen3.8-27B-FP8 (Qwen/Qwen3.8-27B-FP8, fp8 e4m3 weights with dynamic activation scaling) on a single H200 (141 GB), served by vLLMnightly-cd10ed6f9f6b37a8ace9cf380007e66fe12ec0c3with TP1 and the checkpoint's native MTP head (mtp_num_hidden_layers=1, themtp.*tensors ship in the repo). No external drafter and no--enforce-eager. Single-turn 1k1k only, concurrency 1-128. Sibling of the bf16 DSpark recipe in theqwen3.8-27bPRs; sameqwen3.827bmodel-prefix, distinguished by precision.Serve shape:
--speculative-config '{"method":"mtp","num_speculative_tokens":3}'(the setting the vLLM recipe page gives for this model; it reports fp8 MTP acceptance 0.771), fp8 KV cache,--max-num-seqssized to the sweep point (floor 16, 256 for accuracy evals: vLLM's default 1024 exceeded the GDN cache blocks next to the weights on H100 in run 35357364404),--no-enable-prefix-cachingso every request prefills its full prompt,--language-model-only(text-only serving ofQwen3_5ForConditionalGeneration),--reasoning-parser qwen3,--tool-call-parser qwen3_xml, and the matrix context as--max-model-len. Prompts go through the chat template so MTP acceptance reflects real text rather than random tokens; acceptance is real. The MODELS.md / MODELS_zh.md rows for Qwen3.8-27B land with the bf16 H100 PR (#3260).Validation:
bash -n, YAML parse, exact-key matrix generation, and changelog append (no deletions) pass. GPU validation pending;full-sweep-fail-fastis set on this PR.One PR per SKU.
中文
新增
qwen3.827b-fp8-h200-vllm-mtp:Qwen3.8-27B-FP8(Qwen/Qwen3.8-27B-FP8,fp8 e4m3 权重、动态激活缩放)在单张 H200(141 GB)上由 vLLMnightly-cd10ed6f9f6b37a8ace9cf380007e66fe12ec0c3以 TP1 提供服务,并使用 checkpoint 自带的原生 MTP 头(mtp_num_hidden_layers=1,仓库内含mtp.*张量)。无外部草稿模型,也不使用--enforce-eager。仅单轮 1k1k,并发 1-128。与qwen3.8-27b各 PR 中的 bf16 DSpark 配方为姊妹配方;沿用qwen3.827bmodel-prefix,以精度区分。服务参数:
--speculative-config '{"method":"mtp","num_speculative_tokens":3}'(vLLM recipe 页面为该模型给出的设置;其报告 fp8 MTP 接受率 0.771),fp8 KV cache,--max-num-seqs按并发点设定(下限 16,精度评测为 256:运行 35357364404 中 vLLM 默认的 1024 超过 H100 上权重旁的 GDN cache block 数),--no-enable-prefix-caching使每个请求完整 prefill 其提示词,--language-model-only(仅文本服务Qwen3_5ForConditionalGeneration),--reasoning-parser qwen3,--tool-call-parser qwen3_xml,--max-model-len使用矩阵上下文。提示词经 chat template 处理,使 MTP 接受率反映真实文本而非随机 token;接受率为真实值。 Qwen3.8-27B 的 MODELS.md / MODELS_zh.md 行随 bf16 H100 PR(#3260)提交。验证:
bash -n、YAML 解析、精确 key 矩阵生成、changelog 追加(无删除)均通过。GPU 验证待完成;本 PR 已添加full-sweep-fail-fast。每个 SKU 一个 PR。
🤖 Generated with Claude Code