slurm: make run-parts.sh exclusive detection work with custom prefix and recent Slurm - #1391
Open
100milliongold wants to merge 1 commit into
Open
slurm: make run-parts.sh exclusive detection work with custom prefix and recent Slurm#1391100milliongold wants to merge 1 commit into
100milliongold wants to merge 1 commit into
Conversation
…and recent Slurm
The exclusive-job check in run-parts.sh had two independent failures:
1. It called scontrol/squeue through PATH. slurmd's environment does not
include a custom slurm_install_prefix, so both commands produced empty
output, numcpus_sys and numcpus_job were both "", the comparison was
true, and every job ran the *-exclusive-* prolog/epilog scripts. On a
shared node this reset power limits and clocks on all GPUs and dropped
page caches for every job.
2. It parsed "scontrol show job" with grep -Eio "TRES=cpu=[0-9]+". On
recent Slurm the output has both ReqTRES= and AllocTRES= lines, so the
pattern matched twice and numcpus_job became a multi-line value that
never compared equal. With scontrol on PATH, exclusive jobs were
therefore never detected.
Use {{ slurm_install_prefix }}/bin/squeue by absolute path (the file is
already deployed via the template module) and read allocated CPUs and node
count with -o %C / -o %D instead of parsing scontrol. Guard against an
empty result so a lookup failure means "not exclusive" rather than
"exclusive".
Observed on DGX OS 7.5.0, Slurm 26.05.1, slurm_install_prefix=/raid/slurm/usr/local:
- before the binaries were symlinked into /usr/local/bin, every srun
--gres=gpu:1 job logged "Running .../50-exclusive-gpu" and prolog took
6-8 s (srun: Prolog hung on node);
- after symlinking, a bash -x run of the script showed numcpus_job='1<nl>56'.
Signed-off-by: Jea-Eok-Kim <je.kim@xiilab.com>
100milliongold
marked this pull request as ready for review
September 3, 2026 13:54
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The exclusive-job detection in
roles/slurm/templates/etc/slurm/shared/bin/run-parts.shfails in two independent ways:PATH. It calls
scontrol/squeuethrough PATH. slurmd's environment does not include a customslurm_install_prefix, so both commands return nothing,numcpus_sysandnumcpus_jobare both empty,"" == ""is true, and every job runs the*-exclusive-*scripts. On a shared node that resets power limits and application clocks on all GPUs and drops page caches for each job (prolog took 6–8 s;srun: Prolog hung on node).Parsing.
grep -Eio "TRES=cpu=[0-9]+"matches bothReqTRES=andAllocTRES=lines on recent Slurm, sonumcpus_jobbecomes a multi-line value ('1\n56'in abash -xtrace) that never compares equal. Withscontrolon PATH, exclusive jobs are therefore never detected.Reproduced on DGX OS 7.5.0, Slurm 26.05.1,
slurm_install_prefix: /raid/slurm/usr/local.Fix
{{ slurm_install_prefix }}/bin/squeueby absolute path (the script is deployed with thetemplatemodule, so the variable is available).squeue -o %C/-o %Dinstead of parsingscontrol show job.Verification
On the system above, before the fix every
srun --gres=gpu:1job loggedRunning .../50-exclusive-gpu; after symlinking the binaries onto PATH (which exercises failure 2) abash -xrun showednumcpus_job='1\n56'. The patched logic yieldsnumcpus_job=56,numcpus_sys=256,exclusive=0for that job.