Skip to content

feat(closed_loop): shard evaluation across all DDP ranks - #16

Draft
HansRobo wants to merge 1 commit into
ky/closed-loop-ddp-fail-fastfrom
ky/closed-loop-pooled-ddp
Draft

feat(closed_loop): shard evaluation across all DDP ranks#16
HansRobo wants to merge 1 commit into
ky/closed-loop-ddp-fail-fastfrom
ky/closed-loop-pooled-ddp

Conversation

@HansRobo

@HansRobo HansRobo commented Aug 12, 2026

Copy link
Copy Markdown
Owner

Original PR: tier4#337

Problem

closed_loop_validate runs inside if global_rank == 0:, so one GPU does the evaluation while
the rest idle at the next collective. On a 4-GPU job that is 48 minutes of 3 idle GPUs per
checkpoint (102 minutes on the final save, which also renders media).

The DDP machinery for this already exists (shard_items, run_distributed, merge_ddp_shards)
but has no caller. It also barriers once per (site, object-mode), so a site with one route runs at
single-GPU speed regardless of GPU count — measured ceiling 1.96x on 4 GPUs.

Fix

run_evaluations_ddp() pools all (combo, route) pairs into one list, splits it once with
cost-weighted LPT, runs with no intermediate barriers, then barriers once and merges per combo.
closed_loop_validate moves out of the rank-0 guard.

Result

4xH100, real manifest, 38 segments:

sequential 4 GPU
media OFF 2862 s 1023 s 2.80x
media ON 6121 s 2736 s 2.24x

Summed per-rank rollout time matches the sequential baselines to 0.04% / 0.1% — same workload,
negligible contention.

Output is identical. All 38 mp4s match, including 14 routes written concurrently
into one directory.

For reviewers

  • elapsed_sec is now all-reduced per combo, preserving its per-site meaning. This touches an
    existing per-site W&B metric deliberately.
  • run_distributed() now has zero callers (it had none before either). Deleting it is your call.
  • 2.80x vs an ideal 4.00x is the cost proxy, not the mechanism: frame count orders routes well
    within a site, poorly across sites (0.026-0.079 s/frame). Feeding measured per-route seconds
    back would reach 4.00x; that needs rank-0 to plan and broadcast, so it is left as follow-up.

Tests

25 tests, where there were 0. The multi-rank ones spawn real gloo groups on CPU and drive the
production merge path. Failure propagation is mutation-verified.

@HansRobo
HansRobo marked this pull request as draft August 17, 2026 08:27
@HansRobo
HansRobo changed the base branch from ky/closed-loop-ddp-fail-fast to tier4-main August 19, 2026 06:59
@HansRobo
HansRobo changed the base branch from tier4-main to worktree-pr20-ddp-barrier-fix August 19, 2026 07:02
closed_loop_validate ran inside `if global_rank == 0:`, so one GPU evaluated while the rest idled
at the next collective -- 48 minutes per checkpoint on a 4-GPU job, 102 on the final save.

The existing machinery barriers once per (site, object-mode), which caps at 1.96x on 4 GPUs
because a one-route site cannot be split. run_evaluations_ddp() pools all (combo, route) pairs
instead, splits once with cost-weighted LPT, runs with no intermediate barriers, then barriers
once and merges per combo.

4xH100: media off 2862 -> 1023 s (2.80x), media on 6121 -> 2736 s (2.24x). Summed per-rank time
matches the sequential baselines to 0.04% / 0.1%. Output identical apart from order-dependent
reductions. 25 tests, where there were 0.
@HansRobo
HansRobo force-pushed the ky/closed-loop-pooled-ddp branch from 00a1077 to 3de3de7 Compare August 19, 2026 07:53
@HansRobo
HansRobo changed the base branch from worktree-pr20-ddp-barrier-fix to ky/closed-loop-ddp-fail-fast August 19, 2026 08:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant