perf(open-loop): render the scenario PNGs on a worker pool - #417
Merged
Merged
Conversation
HansRobo
force-pushed
the
perf/open-loop-render-pool
branch
2 times, most recently
from
September 11, 2026 08:58
591af59 to
05cdbd0
Compare
The 4-in-1 model evaluation runs 110.8 minutes and 13.5 of them are the scenario-based open-loop validation, which spends 93% of its wall drawing 1,296 matplotlib figures one at a time on rank 0. They now go to render_pool, the process pool the closed-loop renderer already submits to. Measured on 4 GPUs against c58ab06, both arms launched the same minute on separate nodes: 880 s -> 101 s. All 1,296 PNGs are byte-identical and summary.json is unchanged. Sixteen workers is where the footprint stops paying for itself rather than where the curve peaks: it recovers 98% of what the pool can recover for 21.6 GB on rank 0, and by 32 the main thread, not the renderers, sets the pace -- its own per-item work is 28.0 s there against 26.9 s of waiting. OPEN_LOOP_RENDER_WORKERS takes the rest, the way REPLAY_SAVE_WORKERS does in replay.py.
HansRobo
force-pushed
the
perf/open-loop-render-pool
branch
from
September 11, 2026 09:34
05cdbd0 to
a8451e0
Compare
HansRobo
requested review from
MasatoSaeki,
TatsukiNishimura,
kosuke55 and
xtk8532704
September 11, 2026 09:45
HansRobo
marked this pull request as ready for review
September 11, 2026 09:45
TatsukiNishimura
approved these changes
Sep 11, 2026
kosuke55
approved these changes
Sep 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The 4-in-1 model evaluation drops from 110.8 to 98.9 minutes (−10.7%).
Problem
13.5 of those minutes are the scenario-based open-loop validation, and 93.2% of it is
matplotlib: 1,296 figures drawn one at a time on rank 0.
Change
The figures go to
render_pool, the process pool the closed-loop renderer already uses.Result
Same minute, separate nodes, 4 GPUs each. All 1,296 PNGs byte-identical,
summary.jsonunchanged.
c58ab06fFor review
Default thirty-two. 48 buys nothing: by 32 the main thread, not the renderers, sets the
pace. It costs 36 GB on rank 0, 4% of what a four-GPU job is allocated;
OPEN_LOOP_RENDER_WORKERSlowers it where that is not true.Not sharded across the four DDP ranks. They are idle here, but sharding caps at 4x where
a pool does not, and would need the per-category totals, sample indices and
details/writes to survive a merge. The GPUs stay idle either way — the allocation is held for the
whole evaluation.