slurm: batch the queries and parallelize the writes in the GPU power/clock helpers - #1393
Draft
100milliongold wants to merge 1 commit into
Draft
slurm: batch the queries and parallelize the writes in the GPU power/clock helpers#1393100milliongold wants to merge 1 commit into
100milliongold wants to merge 1 commit into
Conversation
…clock helpers set_gpu_power_levels.sh and set_gpu_clocks.sh called nvidia-smi once per GPU to read the target value and once more to apply it, all serially. Both "nvidia-smi -pl" and "nvidia-smi -ac" take roughly a second per GPU, so on an 8-GPU node the two helpers together add about 8 s to the prolog of every job that 50-exclusive-gpu runs for. srun reports this as: srun: Prolog hung on node <node> Read the values for all GPUs in a single --query-gpu call, then apply them in parallel and collect each child's exit status so a failure on any GPU still fails the script. Behaviour is otherwise unchanged: the same values are written to the same GPUs. The "default" branch of set_gpu_clocks.sh already operated on all GPUs at once and is untouched. Observed on DGX OS 7.5.0 (8x B300), Slurm 26.05.1: prolog took 6-8 s per job while 50-exclusive-gpu was running. Signed-off-by: Jea-Eok-Kim <je.kim@xiilab.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
set_gpu_power_levels.shandset_gpu_clocks.shcallnvidia-smionce per GPU to read the target value and once more to apply it, all serially. Bothnvidia-smi -plandnvidia-smi -actake roughly a second per GPU, so on an 8-GPU node the two helpers together add about 8 s to the prolog of every job for which50-exclusive-gpuruns.srunsurfaces it as:Observed on DGX OS 7.5.0 (8× B300), Slurm 26.05.1.
Fix
--query-gpucall (the per-GPU-iloop was only needed because the value was read one at a time).The same values are written to the same GPUs; only the number of
nvidia-smiinvocations and their concurrency change. Thedefaultbranch ofset_gpu_clocks.shalready operated on all GPUs at once and is untouched.Verification
Not yet timed with the patched scripts on hardware — the system where this was found has
50-exclusive-gpuremoved fromprolog.d(an 8-GPU node shared between jobs should not have every job reset limits and clocks on all GPUs). The serial cost is reproducible there: prolog took 6–8 s per job while the script was in place. Marked as draft for that reason; happy to run a timed before/after if that would help.Related
The reason every job ran
50-exclusive-gpuin the first place is a separate defect in the exclusive-job detection, addressed in #1391.