[Klaud Cold] Add MI300X Qwen3.8-27B bf16 vLLM TP1 DSpark 1k1k / 新增 MI300X Qwen3.8-27B bf16 vLLM TP1 DSpark 1k1k 配方 - #3264
functionstackx wants to merge 3 commits into
Conversation
… 1k1k recipe on MI300X Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
067b49a to
faf0b94
Compare
|
Thanks for the contribution!
中文感谢你的贡献！
|
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=35363221115 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=35363221115 |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| description: | ||
| - "Add the MI300X Qwen3.8-27B bf16 recipe: vLLM nightly-eed1f3d0c6043bd494424a22443ee198dd56f657 on one GPU (TP1), 1k1k only, with the RadixArk DSpark drafter Doopeworld/Qwen3.8-27B-DSpark-vLLM (a config-only copy vLLM loads as Qwen3DSparkModel) drafting seven tokens with probabilistic draft sampling, Triton attention with a bf16 KV cache (gfx942 has no calibrated fp8 attention scales), --max-num-seqs sized to the sweep point (floor 16; 256 for accuracy evals) because vLLM's default 1024 exceeds the GDN cache blocks that fit next to the weights, prefix caching disabled, the matrix context as --max-model-len, --language-model-only, and chat-templated prompts; concurrency 1-128" | ||
| - "新增 MI300X Qwen3.8-27B bf16 配方：vLLM nightly-eed1f3d0c6043bd494424a22443ee198dd56f657 单卡（TP1），仅 1k1k，使用 RadixArk 的 DSpark 草稿模型 Doopeworld/Qwen3.8-27B-DSpark-vLLM（仅修改配置以便 vLLM 按 Qwen3DSparkModel 加载）预测 7 个 token 并采用 probabilistic 草稿采样，Triton attention 与 bf16 KV cache（gfx942 无校准的 fp8 attention scale），--max-num-seqs 按并发点设定（下限 16；精度评测为 256），因 vLLM 默认的 1024 超过权重旁可容纳的 GDN cache block 数，关闭 prefix caching，--max-model-len 使用矩阵上下文，--language-model-only，提示词经 chat template 处理；并发 1-128" | ||
| pr-link: TBD |
There was a problem hiding this comment.
🔴 This PR's changelog entry uses pr-link: TBD, which fails CI's changelog gate and blocks every recipe in this PR from merging. validate_added_pr_link in infx/workflows/validate_perf_changelog.py:134-144 only accepts the real .../pull/<N> URL or a placeholder in PR_LINK_PLACEHOLDERS ("XXX" or "https://github.com/SemiAnalysisAI/InferenceX/pull/XXX", line 21-24); "TBD" matches neither, so compare_entries raises ChangelogValidationError for this appended entry. Fix: use the XXX placeholder (or the real PR URL once known) so the appended entry satisfies validate_added_pr_link.
Extended reasoning...
CI runs infx/workflows/validate_perf_changelog.py on this PR with pr_number set (this PR's number, not None). compare_entries() at line 178 computes additions = head_entries[len(base_entries):], which includes the new qwen3.827b-bf16-mi300x-vllm-mtp entry ending at perf-changelog.yaml:8091. For each addition it calls validate_added_pr_link(str(entry.get('pr-link') or ''), pr_number) at line 188. Inside validate_added_pr_link (line 134), pr_number is not None so it takes the branch at line 140: expected = '.../pull/<pr_number>'; link not in PR_LINK_PLACEHOLDERS ('TBD' is not 'XXX' or the XXX URL) and link != expected, so it raises ChangelogValidationError("new PR entry must use ... or an XXX placeholder; found 'TBD'"). This propagates and fails the validation job, blocking merge until pr-link is corrected to XXX or the real link.
Verification: normal (with a correction to the candidate's cited stage). The root defect is real and reachable: the appended entry at perf-changelog.yaml:8091 uses pr-link: TBD, and TBD is not an accepted pr-link value. validate_added_pr_link (infx/workflows/validate_perf_changelog.py:134-145) raises ChangelogValidationError whenever pr_number is not None and the link is neither in… | normal…
| if [[ "${EVAL_ONLY:-false}" == true ]]; then | ||
| run_eval --framework lm-eval --port "$PORT" |
There was a problem hiding this comment.
🔴 An eval-only CI run of this recipe always fails: EVAL_ONLY=true never stages lm-eval results, so the job errors even though the eval itself succeeded. Line 94 calls run_eval --framework lm-eval but never calls append_lm_eval_summary, unlike the RUN_EVAL branch at lines 111-113 which calls both. append_lm_eval_summary is what copies results*.json/meta_env.json out of EVAL_RESULT_DIR into the workspace root; without it, benchmark-tmpl.yml's launcher check ls results*.json (and the eval-artifact upload with if-no-files-found: error when eval-only) fails. Fix: call run_eval followed by append_lm_eval_summary on the EVAL_ONLY path too, mirroring every sibling _mtp.sh script's pattern.
Extended reasoning...
e2e-tests.yml always pairs eval-only:true with run-eval:true (see lines 399-400, 457-458, 519-520, 573-574), so this recipe's accuracy-eval CI job sets EVAL_ONLY=true. The script hits line 93's if-branch and calls run_eval (line 94) only. run_eval's internal auto-staging only fires for agentic scenarios (scenario_is_agentic) or kimi/minimax/bfcl frameworks on failure (benchmark_lib.sh ~3055-3062); fixed-seq-len + lm-eval matches neither, so append_lm_eval_summary is never invoked. run_lm_eval sets EVAL_RESULT_DIR to a tmp dir (benchmark_lib.sh:2114) holding results.json, but nothing copies it to $(pwd). benchmark-tmpl.yml's Launch job script step then runs if ! ls results*.json; then echo 'Eval-only run failed...'; exit 1; fi, which fails since no results*.json exists in the checkout root. Every eval-only invocation of qwen3.827b-bf16-mi300x-vllm-mtp therefore fails after a successful eval, losing accuracy coverage for this recipe.
Verification: normal. The eval-only accuracy path for this new recipe always fails because results are never staged into the workspace. Chain, all reachable: (1) Recipe qwen3.827b_bf16_mi300x_mtp.sh:93-94 — when EVAL_ONLY=true it takes the if-branch calling only run_eval --framework lm-eval --port "$PORT"; append_lm_eval_summary (present only in the else/RUN_EVAL block at line 113) is never reached. (2)…
Runs 35360742600 (attempts 1 and 2) landed c4/c16/c64 on smci300x-ccs-aus-e06-40 and e06-01, where pyxis cannot create a user namespace, and each attempt failed before the container started. This is the launch_mi300x-amd.sh from #3246 verbatim (retry on a fresh allocation, exclude and drain the broken node); the diff disappears once #3246 merges. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
/reuse-sweep-run 35363221115 |
Sync with origin/main after the green sweep run 35363221115; the reuse gate authorizes that run on this head. 在绿色 sweep 运行 35363221115 之后与 origin/main 同步；reuse gate 在此 head 上授权该运行。 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
functionstackx
added
the
qwen3.8-27b
1 participant
Add
qwen3.827b-bf16-mi300x-vllm-mtp: Qwen3.8-27B in bf16 (Qwen/Qwen3.8-27B) on a single MI300X (192 GB), served by vLLMnightly-eed1f3d0c6043bd494424a22443ee198dd56f657with TP1 and the RadixArk DSpark drafterDoopeworld/Qwen3.8-27B-DSpark-vLLM(byte-identical weights toRadixArk/Qwen3.8-27B-DSpark; onlyarchitecturesis changed so vLLM routes it toQwen3DSparkForCausalLM). Single-turn 1k1k only, concurrency 1-128.Serve shape:
--speculative-config '{"method":"dspark","model":"Doopeworld/Qwen3.8-27B-DSpark-vLLM","num_speculative_tokens":7,"draft_sample_method":"probabilistic"}'(seven tokens is the drafter's trained block size; its card measured k=4 and k=6 slower, and probabilistic sampling ~23% faster than greedy), Triton attention with a bf16 KV cache (gfx942 has no calibrated fp8 attention scales),--max-num-seqssized to the sweep point (floor 16, 256 for accuracy evals: vLLM's default 1024 exceeded the 472 GDN cache blocks next to the weights on H100 in run 35357364404),--no-enable-prefix-cachingso every request prefills its full prompt,--language-model-only(text-only serving ofQwen3_5ForConditionalGeneration),--reasoning-parser qwen3,--tool-call-parser qwen3_xml, and the matrix context as--max-model-len. Adaptive verification stays off because vLLM's GDN attention backend rejects it for this hybrid architecture. Prompts go through the chat template so draft acceptance reflects real text rather than random tokens; acceptance is real (no golden AL exists yet for this drafter). On gfx942 the recipe follows the MiniMax-M3 and DSv4.1 Flash ROCm arms:--attention-backend TRITON_ATTN, bf16 KV cache,VLLM_USE_BREAKABLE_CUDAGRAPH=0, imagevllm/vllm-openai-rocmnightly (tag verified on Docker Hub). The MODELS.md / MODELS_zh.md rows for Qwen3.8-27B land in the H100 PR (#3260).Validation:
bash -n, YAML parse, exact-key matrix generation, and changelog append (no deletions) pass. GPU validation pending;full-sweep-fail-fastis set on this PR.One PR per SKU.
中文
新增
qwen3.827b-bf16-mi300x-vllm-mtp：bf16 的 Qwen3.8-27B（Qwen/Qwen3.8-27B）在单张 MI300X（192 GB）上由 vLLMnightly-eed1f3d0c6043bd494424a22443ee198dd56f657以 TP1 提供服务，并使用 RadixArk 的 DSpark 草稿模型Doopeworld/Qwen3.8-27B-DSpark-vLLM（权重与RadixArk/Qwen3.8-27B-DSpark逐字节一致，仅修改architectures以便 vLLM 路由到Qwen3DSparkForCausalLM）。仅单轮 1k1k，并发 1-128。服务参数：
--speculative-config '{"method":"dspark","model":"Doopeworld/Qwen3.8-27B-DSpark-vLLM","num_speculative_tokens":7,"draft_sample_method":"probabilistic"}'（7 为草稿模型训练时的 block size；其模型卡实测 k=4、k=6 更慢，probabilistic 采样比 greedy 快约 23%），Triton attention 与 bf16 KV cache（gfx942 无校准的 fp8 attention scale），--max-num-seqs按并发点设定（下限 16，精度评测为 256：运行 35357364404 中 vLLM 默认的 1024 超过 H100 上权重旁仅剩的 472 个 GDN cache block），--no-enable-prefix-caching使每个请求完整 prefill 其提示词，--language-model-only（仅文本服务Qwen3_5ForConditionalGeneration），--reasoning-parser qwen3，--tool-call-parser qwen3_xml，--max-model-len使用矩阵上下文。由于 vLLM 的 GDN attention 后端不支持该混合架构的自适应验证，该选项保持关闭。提示词经 chat template 处理，使草稿接受率反映真实文本而非随机 token；接受率为真实值（该草稿模型尚无黄金 AL）。 在 gfx942 上遵循 MiniMax-M3 与 DSv4.1 Flash 的 ROCm 配方：--attention-backend TRITON_ATTN、bf16 KV cache、VLLM_USE_BREAKABLE_CUDAGRAPH=0，镜像为vllm/vllm-openai-rocmnightly（tag 已在 Docker Hub 验证）。 Qwen3.8-27B 的 MODELS.md / MODELS_zh.md 行随 H100 PR（#3260）提交。验证：
bash -n、YAML 解析、精确 key 矩阵生成、changelog 追加（无删除）均通过。GPU 验证待完成；本 PR 已添加full-sweep-fail-fast。每个 SKU 一个 PR。
🤖 Generated with Claude Code