[PowerX] measure Grace CPU-side power on NVL72 power lanes / 在 NVL72 功耗 lane 上实测 Grace CPU 侧功耗 - #3296
Draft
edwingao28 wants to merge 9 commits into
Draft
edwingao28 wants to merge 9 commits into
edwingao28 wants to merge 9 commits into
Conversation
Read srt-slurm's power/cpu/{samples.csv,cpu_manifest.json} (v1 long and v2
wide) next to the GPU DCGM package, integrate each (host, socket) Grace-side
and module series over the bound formal window with the shared trapezoid and
gap rule, and emit cpu_power_valid plus avg_cpu_socket_power_w,
avg_total_cpu_power_w, total_cpu_energy_j and the module keys. The verdict is
independent of power_valid and REQUIRE_POWER; a package without cpu/ produces
byte-identical output (GB200 fixture replay).
中文：在多节点功耗校验器中接入 srt-slurm 的 power/cpu/ 子产物包（v1 长格式与 v2
宽格式），按 (主机, socket) 在同一正式测量窗口内用共享的梯形法和采样间隔规则积分
Grace 侧及模块功耗，输出 cpu_power_valid 与 avg_cpu_socket_power_w、
avg_total_cpu_power_w、total_cpu_energy_j 及模块字段；结论独立于 power_valid 和
REQUIRE_POWER，无 cpu/ 的产物包输出逐字节不变（GB200 fixture 回放验证）。
validate_and_integrate() forced cpu_power_valid=0 with cpu_producer_unverified
whenever the GPU leg's producer pin failed, so a run with valid Grace samples
lost its CPU energy for a GPU-side reason, against the ticket contract
("independent of power_valid") and the spec's independent-leg rule. Drop that
gate and the unused blocking_reasons hook: the CPU leg borrows only the bound
formal window and the host topology as inputs, and no GPU verdict reaches it.
Fold the pin-mismatch and pin-missing cases into the existing invalid-GPU-leg
test. Docs (EN/ZH) drop the contradictory sentence and list the codes the leg
actually emits, including the shared non_finite_power_metric and
aggregate_result_* ones. GB200 fixture replay stays byte-identical.
中文：validate_and_integrate() 原本在 GPU 侧 producer 固定版本校验失败时，用
cpu_producer_unverified 强制 cpu_power_valid=0，导致 Grace 采样有效的运行因 GPU 侧
原因丢失 CPU 能耗，与工单契约（“独立于 power_valid”）和 spec 的独立测量环节规则相悖。
移除该门控及未使用的 blocking_reasons 参数：CPU 侧只借用已绑定的正式窗口和主机拓扑
作为输入，GPU 侧的任何结论都不会传导过来。把固定版本不匹配与缺失两种情形并入现有的
GPU 侧无效测试。中英文文档删除自相矛盾的表述，并列出实际输出的原因码，包括共享的
non_finite_power_metric 与 aggregate_result_*。GB200 fixture 回放输出逐字节不变。
…anes Add telemetry.cpu_power_exporter (port 9405, source acpi) to the 1P1D TP4/TP4 STP recipes of qwen3.5-fp8-gb200-dynamo-sglang and qwen3.5-fp8-gb300-dynamo-sglang, the smallest active non-Kimi-K3 NVL72 power lanes; nothing else in the recipes changes. srt-slurm v2.2.1 runs the exporter on the bare worker host and writes power/cpu/samples.csv plus cpu_manifest.json beside the GPU DCGM package for the multinode validator's CPU leg. Append the perf-changelog entry. DSV4 8k1k lanes are not used because that scenario was deprecated after 2026-09-08. 中文：在 qwen3.5-fp8-gb200-dynamo-sglang 与 qwen3.5-fp8-gb300-dynamo-sglang 的 1P1D TP4/TP4 STP recipe 中加入 telemetry.cpu_power_exporter（port 9405， source acpi），这是当前活跃、非 Kimi-K3 的最小 NVL72 功耗 lane；recipe 其余内容不变。srt-slurm v2.2.1 在 worker 裸主机上运行 exporter，并在 GPU DCGM 产物包旁写入 power/cpu/samples.csv 与 cpu_manifest.json，供多节点 校验器的 CPU 侧测量环节使用。追加 perf-changelog 条目。未采用 DSV4 8k1k lane，因为该场景已于 2026-09-08 之后弃用。
…e tail Bring in main (96bccfe), including the #3252 and #3254 image bumps of the two Qwen3.5 FP8 NVL72 recipes this branch edits. perf-changelog.yaml is resolved as main's bytes followed by this branch's entry so the file stays append-only. 中文：合并 main（96bccfe12），包含 #3252 与 #3254 对本分支所改两份 Qwen3.5 FP8 NVL72 recipe 的镜像更新。perf-changelog.yaml 按 main 的原始字节加本分支 条目的方式解决冲突，保持仅追加。
Replace the pr-link placeholder in the appended perf-changelog.yaml entry with the opened PR URL; no other bytes change. 中文：将新增 perf-changelog.yaml 条目中的 pr-link 占位符替换为已开启的 PR 链接，其余内容不变。
…e tail Synchronize with main so the perf-changelog validator sees only additions from this branch; the appended cpu power leg entry stays the last entry. 中文：同步 origin/main，使 perf-changelog 校验只看到本分支的新增内容；新增的 CPU 功耗测量条目保持在文件末尾。
Contributor
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=35632024733 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=35632024733 |
1 task
…e tail Synchronize with main before re-pinning srt-slurm so the perf-changelog validator sees only additions from this branch; the appended cpu power leg entry stays the last entry. 中文：在重新固定 srt-slurm 之前同步 origin/main，使 perf-changelog 校验只看到本分支的新增内容；新增的 CPU 功耗测量条目保持在文件末尾。
…branch The first sweep showed the pinned cpu-power-exporter exiting with "no ACPI power_meter hwmon sensors found" on every GB200 node: on 6.8.0-nvidia-64k the acpi_power_meter hwmon node has no `name` attribute, only `device/name`. Pin utils/srt-slurm to SemiAnalysisAI/srt-slurm#23 head 473ecaa8 (984180e5 plus the gap-budget fix and the discovery fix) so the CPU leg can publish; the wire contract stays v1 long, so no consumer change. 中文：首次 sweep 显示固定版本的 cpu-power-exporter 在所有 GB200 节点上以 "no ACPI power_meter hwmon sensors found" 退出：6.8.0-nvidia-64k 内核下 acpi_power_meter 的 hwmon 节点没有 `name` 属性，只有 `device/name`。将 utils/srt-slurm 固定到 SemiAnalysisAI/srt-slurm#23 的 head 473ecaa8（984180e5 + 采样间隙预算修复 + 传感器发现修复），使 CPU 侧采集可以发布；CSV 契约仍为 v1 长表，消费端无需改动。
…e tail Synchronize with main before re-pinning srt-slurm so the perf-changelog validator sees only additions from this branch; the appended cpu power leg entry stays the last entry. 中文：在重新固定 srt-slurm 之前同步 origin/main，使 perf-changelog 校验只看到本分支的新增内容；新增的 CPU 功耗测量条目保持在文件末尾。
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
NVL72 rows have measured GPU-board power but no Grace-side measurement, so smart provisioning never renders for GB200/GB300. The multinode power validator now integrates srt-slurm's
power/cpu/package over the formal GPU window and addscpu_power_validplus additive CPU-side keys under schema 2; GPU fields andpower_validare unchanged (contract). Thetelemetry.cpu_power_exporterleg (port 9405,source: acpi) is enabled onqwen3.5-fp8-gb200-dynamo-sglangandqwen3.5-fp8-gb300-dynamo-sglang, the smallest active non-K3 NVL72 lanes.Testing: local only:
pytest301 + 424 passed,ruffclean, pinnedsrtctlvalidates both recipes, matrix rows identical,validate_perf_changelogvsorigin/mainpasses. No GPU run yet.Delivery:
full-sweep-fail-faston both keys (6 rows). Accept when thepower_audit_*bundle hasLOGS/power/cpu/samples.csvwithGrace Power Socket 0/1rows per host andcpu_manifest.jsonmodeacpi, the aggregate hascpu_power_valid: 1withavg_cpu_socket_power_w,avg_total_cpu_power_w,total_cpu_energy_j, and GPUpower_validis unchanged; keep that bundle as the validator golden fixture.Producer pin: the first sweep (run 35467617921) hit
no ACPI power_meter hwmon sensors foundon every GB200 node: on 6.8.0-nvidia-64k the hwmon node has noname, onlydevice/name.utils/srt-slurmpins SemiAnalysisAI/srt-slurm#23 head473ecaa8(984180e5 + gap budget + discovery fix) until it merges; the CSV contract stays v1 long.中文
NVL72 数据行只有实测 GPU 板卡功耗，没有 Grace 侧实测，GB200/GB300 的智能预配因此从不渲染。多节点功耗校验器现在会在正式 GPU 测量窗口内对 srt-slurm 的
power/cpu/产物包积分，并在 schema 2 下新增cpu_power_valid与 CPU 侧字段；GPU 字段与power_valid不变（契约）。在当前活跃、非 K3 的最小 NVL72 laneqwen3.5-fp8-gb200-dynamo-sglang与qwen3.5-fp8-gb300-dynamo-sglang上启用telemetry.cpu_power_exporter（port 9405，source: acpi）。测试： 仅本地：
pytest301 + 424 通过，ruff无问题，固定版本srtctl校验两份 recipe 通过，矩阵生成前后一致，validate_perf_changelog对origin/main通过。尚未跑 GPU。交付： 对两个 config key 跑
full-sweep-fail-fast（6 行）。验收条件：power_audit_*产物包含LOGS/power/cpu/samples.csv（每台主机均有Grace Power Socket 0/1行）与解析模式为acpi的cpu_manifest.json，聚合结果含cpu_power_valid: 1及avg_cpu_socket_power_w、avg_total_cpu_power_w、total_cpu_energy_j，GPUpower_valid不变；将该产物包保留为校验器的黄金固定样本。生产端固定版本： 首次 sweep（run 35467617921）在所有 GB200 节点上遇到
no ACPI power_meter hwmon sensors found：6.8.0-nvidia-64k 内核下 hwmon 节点没有name，只有device/name。utils/srt-slurm现固定到 SemiAnalysisAI/srt-slurm#23 的 head473ecaa8（984180e5 + 采样间隙预算修复 + 传感器发现修复），待其合并；CSV 契约仍为 v1 长表。AI 模型： 本 PR 描述由
claude-fable-5-1（Claude Fable 5.1，Claude Code）撰写；此前的提交与委派的 review 未记录运行环境提供的模型标识，确切模型无法确认。角色：起草代码、文档、changelog 与 PR 文本；作者审阅、运行本地检查，并负责 push、label 与 sweep。AI model disclosure
claude-fable-5-1(Claude Fable 5.1, Claude Code) for this body; earlier commits and delegated reviews recorded no runtime identifier, so their exact model could not be verified.Related Issue
None.
Type of Change
Checklist
perf-changelog.yamland have not edited historical entriesOWNER/MEMBER/COLLABORATOR) has commented/use <run_id>(or the legacy/reuse-sweep-run) on this PR. Do this only once there is a final full sweep that is all green with evals passing, since after this comment the sweep label will no longer automatically kick off new sweeps. Remove and re-add the label to force one.