[Power] Require GB multinode telemetry / 强制启用 GB 多节点功耗采集 - #2687
[Power] Require GB multinode telemetry / 强制启用 GB 多节点功耗采集#2687edwingao28 wants to merge 7 commits into
Conversation
…telemetry # Conflicts: # perf-changelog.yaml
|
Dispatched trusted sweep run for approved external revision 已为获批的外部提交 |
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 675dd92. Configure here.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 675dd92ce3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…telemetry # Conflicts: # perf-changelog.yaml
|
Dispatched trusted sweep run for approved external revision 已为获批的外部提交 |

What changed
dcgm-powerwithrequired: trueon all 59 eligible recipes: 8 previously hardware-validated shapes plus the 51 remaining eligible recipes (24 GB200 + 27 GB300), now that both pools are back in service.configs/nvidia-master.yaml) so every candidate and eligible recipe remains covered when master config mappings change.Why
The producer and dashboard paths already support GB telemetry, but most eligible recipes were not opted in. With the GB200 and GB300 pools available again, telemetry is required across the board so a run whose power collection never started cannot silently publish without energy metrics (
required: truepromotes exporter startup failures to blocking validation failures inutils/aggregate_power_multinode.py).Validation
origin/main.false.git diff --checkpassed.lmsysorg/sglangnightly tags from May 2026 that Docker Hub no longer serves — a pre-existing image-pin issue onmain, independent of this PR.中文说明
变更内容
dcgm-power并设为required: true:包括 8 个已有真机验证的拓扑和其余 51 个(24 个 GB200 + 27 个 GB300)——两个 pool 均已恢复服务。configs/nvidia-master.yaml),确保 master config 映射变化后,所有候选及 eligible recipe 仍被测试覆盖。原因
producer 和 dashboard 链路已支持 GB 功耗数据,但大部分符合条件的 recipe 尚未开启。GB200/GB300 pool 恢复可用后,全量设为 required,可防止功耗采集未启动的 run 静默发布无能耗指标的结果(
required: true会在utils/aggregate_power_multinode.py中把 exporter 启动失败提升为阻断性校验失败)。验证
origin/main验证通过。git diff --check通过。lmsysorg/sglangnightly 镜像已被 Docker Hub 回收——这是main上已存在的镜像 pin 问题,与本 PR 无关。Note
Medium Risk
Publishing gates tighten for most official GB multinode configs—missing or failed power collection can block results—but changes are recipe YAML and test/CI coverage, not launcher logic.
Overview
Rolls out mandatory
dcgm-powertelemetry (required: true) across 59 eligible GB200/GB300 disaggregated multinode recipes (DeepSeek-V4 and Qwen3.5), using a shared contract: 1 Hz sampling,storage_subdir: power, DCGM exporter on port 9401 (GB200) or 19401 (GB300). Two dedicated-infra8p1dQwen recipes stay without power opt-in.Replaces a small hardcoded recipe list in
test_gb200_power_official_contract.pywith dynamic eligibility (61 candidates from recipe shape +configs/nvidia-master.yamlmappings). CI Test Process Result path filters now use broad recipe globs and watchnvidia-master.yamlso contract tests run when mappings change.Adds the corresponding perf-changelog entry; launcher scripts are unchanged.
Reviewed by Cursor Bugbot for commit 6055b67. Bugbot is set up for automated code reviews on this repo. Configure here.