Skip to content

[Power] Require GB multinode telemetry / 强制启用 GB 多节点功耗采集 - #2687

Open
edwingao28 wants to merge 7 commits into
SemiAnalysisAI:mainfrom
edwingao28:feat/gb-recipes-bulk-telemetry
Open

[Power] Require GB multinode telemetry / 强制启用 GB 多节点功耗采集#2687
edwingao28 wants to merge 7 commits into
SemiAnalysisAI:mainfrom
edwingao28:feat/gb-recipes-bulk-telemetry

Conversation

@edwingao28

@edwingao28 edwingao28 commented Aug 20, 2026

Copy link
Copy Markdown
Collaborator

What changed

  • Evaluate 61 GB200/GB300 multinode recipes against the existing power-lane eligibility contract.
  • Enable dcgm-power with required: true on all 59 eligible recipes: 8 previously hardware-validated shapes plus the 51 remaining eligible recipes (24 GB200 + 27 GB300), now that both pools are back in service.
  • Leave 2 dedicated-infrastructure recipes disabled because their current topology is not eligible for this launcher path.
  • Add dynamic contract coverage and CI path filters (including configs/nvidia-master.yaml) so every candidate and eligible recipe remains covered when master config mappings change.
  • Append the required performance changelog entry without changing runner behavior.

Why

The producer and dashboard paths already support GB telemetry, but most eligible recipes were not opted in. With the GB200 and GB300 pools available again, telemetry is required across the board so a run whose power collection never started cannot silently publish without energy metrics (required: true promotes exporter startup failures to blocking validation failures in utils/aggregate_power_multinode.py).

Validation

  • CI-equivalent power and result-processing suite: 205 passed.
  • Official performance changelog validator passed against origin/main.
  • Contract coverage includes 61/61 candidates and 59/59 eligible recipes; the required-contract assertion was tripwire-tested by flipping one recipe back to false.
  • git diff --check passed.
  • Full-sweep validation on the GB pools is dispatched from this PR; note the dsv4 GB lanes currently reference lmsysorg/sglang nightly tags from May 2026 that Docker Hub no longer serves — a pre-existing image-pin issue on main, independent of this PR.

中文说明

变更内容

  • 按现有功耗 lane 资格契约检查 61 个 GB200/GB300 多节点 recipe。
  • 为全部 59 个符合条件的 recipe 启用 dcgm-power 并设为 required: true:包括 8 个已有真机验证的拓扑和其余 51 个(24 个 GB200 + 27 个 GB300)——两个 pool 均已恢复服务。
  • 2 个使用独立基础设施节点且不符合当前 launcher 资格条件的 recipe 保持关闭。
  • 增加动态 contract 覆盖与 CI path 过滤(含 configs/nvidia-master.yaml),确保 master config 映射变化后,所有候选及 eligible recipe 仍被测试覆盖。
  • 追加必需的 performance changelog,不修改 runner 行为。

原因

producer 和 dashboard 链路已支持 GB 功耗数据,但大部分符合条件的 recipe 尚未开启。GB200/GB300 pool 恢复可用后,全量设为 required,可防止功耗采集未启动的 run 静默发布无能耗指标的结果(required: true 会在 utils/aggregate_power_multinode.py 中把 exporter 启动失败提升为阻断性校验失败)。

验证

  • CI 同款功耗与结果处理测试:205 个通过。
  • 官方 performance changelog validator 基于 origin/main 验证通过。
  • contract 覆盖 61/61 候选及 59/59 eligible recipe;required 断言经过"翻回 false 触发失败"的 tripwire 验证。
  • git diff --check 通过。
  • 已从本 PR 派发 GB pool 的 full-sweep 验证;注意 dsv4 GB lane 引用的 2026 年 5 月 lmsysorg/sglang nightly 镜像已被 Docker Hub 回收——这是 main 上已存在的镜像 pin 问题,与本 PR 无关。

Note

Medium Risk
Publishing gates tighten for most official GB multinode configs—missing or failed power collection can block results—but changes are recipe YAML and test/CI coverage, not launcher logic.

Overview
Rolls out mandatory dcgm-power telemetry (required: true) across 59 eligible GB200/GB300 disaggregated multinode recipes (DeepSeek-V4 and Qwen3.5), using a shared contract: 1 Hz sampling, storage_subdir: power, DCGM exporter on port 9401 (GB200) or 19401 (GB300). Two dedicated-infra 8p1d Qwen recipes stay without power opt-in.

Replaces a small hardcoded recipe list in test_gb200_power_official_contract.py with dynamic eligibility (61 candidates from recipe shape + configs/nvidia-master.yaml mappings). CI Test Process Result path filters now use broad recipe globs and watch nvidia-master.yaml so contract tests run when mappings change.

Adds the corresponding perf-changelog entry; launcher scripts are unchanged.

Reviewed by Cursor Bugbot for commit 6055b67. Bugbot is set up for automated code reviews on this repo. Configure here.

@edwingao28
edwingao28 marked this pull request as ready for review August 24, 2026 19:21
@edwingao28
edwingao28 requested a review from a team August 24, 2026 19:21

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@github-actions

Copy link
Copy Markdown
Contributor

Dispatched trusted sweep run for approved external revision 675dd92ce39878e634c2df4bbfcc32b925911f4b.
New commits are not trusted automatically; remove and re-add the primary sweep label to approve a new SHA.

已为获批的外部提交 675dd92ce39878e634c2df4bbfcc32b925911f4b 调度trusted sweep run
后续新提交不会自动获得信任;如需批准新的 SHA,请移除并重新添加主扫描标签。

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 675dd92. Configure here.

Comment thread utils/test_gb200_power_official_contract.py

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 675dd92ce3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .github/workflows/test-process-result.yml
@edwingao28 edwingao28 changed the title [Power] Bulk-enable GB multinode telemetry / 批量启用 GB 多节点功耗采集 [Power] Require GB multinode telemetry / 强制启用 GB 多节点功耗采集 Aug 24, 2026
…telemetry

# Conflicts:
#	perf-changelog.yaml
@github-actions

Copy link
Copy Markdown
Contributor

Dispatched trusted sweep run for approved external revision 6055b67eafa5b30f923b7e9770ce0df079d351c1.
New commits are not trusted automatically; remove and re-add the primary sweep label to approve a new SHA.

已为获批的外部提交 6055b67eafa5b30f923b7e9770ce0df079d351c1 调度trusted sweep run
后续新提交不会自动获得信任;如需批准新的 SHA,请移除并重新添加主扫描标签。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

1 participant