[AgentX] DeepSeek-V4 B300 SGLang update - #2704
Conversation
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
2 similar comments
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Beyond the inline changelog-arithmetic nit, this run also checked several other candidates and ruled them out: the DEP8 mem-fraction-static values are internally consistent with the comment (0.9 setpoint at conc>=256, not the ratio-validation figure); the HiCache ratio default table is unaffected by the low-conc DEP8 mem-fraction difference; the config comment describing dp-attn true/false paths matches the script logic; and the perf-changelog pr-link (2704) matches this PR.
Extended reasoning...
Beyond the inline finding, finder agents raised four additional candidates this run, and each was examined against the actual script/config and refuted: (1) a claim that the config comment for dsv4-fp4-b300-sglang-mtp describes the wrong script's behavior — checked against the sh file, matches; (2) a claim that DEP8 mem-fraction-static at conc>=256 uses an unvalidated 0.9 contradicting the OOM measurement comment — the measurement text refers to the ratio-tuning experiment at conc 256, and the tiered mem-fraction schedule (0.93/0.9/0.89/0.875) is a separate, consistent scale-down; (3) a claim that the HiCache ratio default is validated at one mem-fraction but low-conc DEP8 runs at another — the ratio and mem-fraction knobs are independent and the described interaction is about host-capacity sizing, not runtime correctness; (4) the perf-changelog pr-link was checked and correctly points to 2704, not 2701.
| - "Scale chunked-prefill-size with dp_size (8192 * TP) instead of pinning 65536. server_args.py divides this global budget by dp_size, so a fixed 65536 gives the tuned 8192/rank at DEP8 but 16384/rank at DEP4 -- which exceeds MegaMoE's per-rank token cap and raises at startup, and which measured slower when tried directly at DEP8. Also export SGLANG_OPT_DEEPGEMM_MEGA_MOE_NUM_MAX_TOKENS_PER_RANK=8320, since the 8192 default is an exact fit with no headroom." | ||
| - "On the DP path, replace --cuda-graph-max-bs with --cuda-graph-max-bs-decode 544 and add --enable-prefill-delayer. The generic flag aliases the same knob, and its CONC-capped value of 64 does not cover the padded MTP decode batch across DP ranks." | ||
| - "Scale DP-path mem-fraction-static down as concurrency rises: DEP8 uses 0.93 at conc 64/128, 0.9 at 256, 0.89 at 384 and 0.875 at 512/576; DEP4 keeps upstream's 0.95. MegaMoE's transient workspace is allocated outside the static budget and needs a single ~7 GB contiguous block, so the headroom left by mem-fraction-static has to grow with batch size. Measured at conc 256: 0.835 (~42 GB free) completes, while 0.93 (~16 GB free) and 0.95 (~11 GB free) both hit a CUDA OOM on a single DP rank -- which then hangs the whole engine, because the surviving ranks block forever in the MLP-sync collective and the client sees in-flight requests never return with zero errors. DEP4 cannot go lower: it shards the model over half the node, so per-rank weights roughly double and the weights-only floor rises above 0.9 (the engine reports a minimum viable 0.9013 and refuses to start). Also lower swa-full-tokens-ratio from 0.1 to 0.075." | ||
| - "Restructure the search space from 47 points to 12. Cut the TP-8 no-offload row to conc 1/4/8 and drop both TP-4 rows and the DEP4 no-offload row; keep DEP4+hicache at conc 32/48/64; and replace the DEP8 no-offload row with a DEP8+hicache row at conc 64/128/256/384/512/576. This tracks the vLLM agentic lane on the same runner, which sweeps DEP4 with a DRAM tier at 48/64 and DEP8 with a DRAM tier at 128/256/384/512/576. Measured at conc 256 with a CPU tier, SGLang 4,168 vs vLLM 4,199 tok/s." |
There was a problem hiding this comment.
🟡 The perf-changelog entry says the search space was restructured 'from 47 points to 12', but summing the conc-list lengths of the five removed rows in configs/nvidia-master.yaml (7 + 12 + 6 + 10 + 7) gives 42, not 47. The new total of 12 is correct; only the old-count figure in the rationale text is wrong.
Extended reasoning...
The newly added changelog entry in perf-changelog.yaml (around line 6304) states:
"Restructure the search space from 47 points to 12."
This is a factual/arithmetic error in the descriptive rationale text. The diff to configs/nvidia-master.yaml removes five search-space rows from the dsv4-fp4-b300-sglang-agentic-hicache-mtp / agentic-coding scenario, each carrying a conc-list:
{ tp: 4, kv-offloading: none, ... }→[1, 4, 8, 16, 20, 24, 32]= 7 points{ tp: 8, kv-offloading: none, ... }→[1, 4, 8, 16, 32, 40, 48, 52, 56, 60, 64, 72]= 12 points{ tp: 4, ep: 4, dp-attn: true, kv-offloading: none, ... }→[8, 16, 24, 32, 40, 64]= 6 points{ tp: 4, ep: 4, dp-attn: true, kv-offloading: dram, ... }→[32, 40, 48, 56, 64, 72, 80, 88, 96, 128]= 10 points{ tp: 8, ep: 8, dp-attn: true, kv-offloading: none, ... }→[52, 72, 100, 128, 144, 196, 512]= 7 points
Step-by-step proof: 7 + 12 + 6 + 10 + 7 = 42, not 47. All five rows use the plain conc-list form (no conc-start/conc-end range expansion that could add hidden extra points), so 42 is the complete count of the removed old search space. The new search space consists of three rows — [1,4,8] (3) + [32,48,64] (3) + [64,128,256,384,512,576] (6) — which sums to 12, matching the changelog's claimed new total exactly. So only the 'old' half of the '47 points to 12' claim is wrong; it should read 42.
The code path here is purely documentation: perf-changelog.yaml is a human-readable rationale log describing why a config changed, and this line has no effect on which benchmarks run or how they're configured — the actual behavior is driven entirely by the search-space list in configs/nvidia-master.yaml, which is correct. Nothing in the review or CI tooling cross-checks these prose point-counts against the YAML, which is why the off-by-five slipped through.
Impact is limited to a slightly misleading changelog entry that future readers (or an agent skimming perf history for context) might use to reason about how much the search space was pruned. Fix is trivial: change "from 47 points to 12" to "from 42 points to 12" in the changelog description string.
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32457429222 |
460f75b to
a1338fb
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32501225217 |
a1338fb to
b730489
Compare
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32501423418 |
|
see unofficial run visualizer at https://inferencex.semianalysis.com/inference?unofficialRun=32501828617 |
…ill-decode-interval 10 at conc 512 基于 #2701 的 recipe（prefill-decode-interval 改为 10），临时将 agentic sweep 缩减到 conc 512 单点，recipe 脚本在启动 sglang 前把 sglang-loadback-multipin.patch（sgl-project/sglang#35880，即 #34975 cherry-pick 到 dev-nightly-0820 基线并保留 #34519 write-back 门控） 打到镜像内 editable 安装的 sglang 源码上，验证 unified_cache commit_load_back 单 pin 断言的修复。仅为验证用途，不用于合入。
b730489 to
c9443fd
Compare
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit c9443fd. Configure here.
| + node.load_back_pending_ids.discard(anchor_node_id) | ||
| + # The loaded copies become tracked duplicates only once the | ||
| + # last in-flight load-back on this node acks. | ||
| self._update_duplicate_tracking(node) |
There was a problem hiding this comment.
Premature duplicate tracking on ack
High Severity
In finish_load_back, after load_back_pending_ids.discard, _update_duplicate_tracking still runs even when other anchors remain in the set. The new comment says tracking should wait until the last in-flight load-back acks, which matches the old single-pin clear-then-update behavior. Calling it early under overlapping pins can mark host copies settled while another H→D transfer is still reading them—the multipin case this patch is meant to fix at conc 512.
Reviewed by Cursor Bugbot for commit c9443fd. Configure here.
| else | ||
| echo "sglang load-back multipin patch not applicable (already applied?), skipping" | ||
| fi | ||
| fi |
There was a problem hiding this comment.
Patch apply failure is ignored
Medium Severity
The TEMP load-back multipin apply path never aborts the job. A missing /sgl-workspace/sglang is silent, git apply --check failure is treated as already applied, and a failed git apply only prints ERROR because of || echo. This PR’s only purpose is to validate that patch at conc 512, so continuing unpatched will reproduce the old commit_load_back assertion and look like the fix failed.
Reviewed by Cursor Bugbot for commit c9443fd. Configure here.
重新触发 sweep CI。
1 participant
Same recipe as #2701 but with prefill-decode-interval 10, plus exactly one difference: the recipe script applies sglang-loadback-multipin.patch (sgl-project/sglang#35880) to the image's editable sglang source before server start, and the sweep is temporarily reduced to the conc-512 point only, to validate the unified-cache load-back fix. Just test, do not merge.
recipe 与 #2701 相同但 prefill-decode-interval 改为 10，另有唯一差异：recipe 脚本会在启动 sglang 前把 sglang-loadback-multipin.patch（sgl-project/sglang#35880）打到镜像内 editable 安装的 sglang 源码上，且 sweep 临时缩减到 conc 512 单点，用于验证 unified-cache load-back 修复。仅测试，请勿合入。
Note
Medium Risk
Benchmark-only, but it git-applies a cache-correctness patch into the container’s SGLang tree and retunes GPU/host memory knobs that can OOM or hang the engine if wrong.
Overview
Validation-only AgentX recipe for DeepSeek-V4-Pro on B300 SGLang. Not intended to merge: the sweep is cut to a single DEP8+HiCache conc-512 point to A/B the unified-cache load-back fix against #2701.
The recipe now applies
sglang-loadback-multipin.patch(sgl-project/sglang#35880) to the image’s editable source before launch socommit_load_backcan pin a node from multiple anchors. Image islmsysorg/sglang:dev-nightly-0820. TEMP knobs:--prefill-decode-interval 5and DEP8 conc-512mem-fraction-static0.86.On the DP-attention path the engine switches to MegaMoE + FP4 indexer, scales chunked-prefill as
8192 * TP, uses decode CUDA graphs of 544, and lowers mem-fraction as concurrency rises. HiCache TP≥8 ratio goes 2 → 3 anddram-utilization0.80 → 0.95. TP-only decode graphs are sized at4*CONC(capped at 64) so subagent fan-out does not fall back to eager decode.Reviewed by Cursor Bugbot for commit c423e7e. Bugbot is set up for automated code reviews on this repo. Configure here.