Skip to content

fix: harden GPU runtime convergence - #1396

Open
dholt wants to merge 1 commit into
masterfrom
dholt/release-26.09-blockers
Open

fix: harden GPU runtime convergence#1396
dholt wants to merge 1 commit into
masterfrom
dholt/release-26.09-blockers

Conversation

@dholt

@dholt dholt commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Problem

Several GPU-cluster paths diverged as newer DGX OS, Slurm, and Ubuntu releases
changed their service, module, package, and security defaults. In combination,
these gaps prevent a complete Slurm deployment with monitoring and Pyxis from
converging and validating reliably.

Changes

  • Select the correct SSH service when applying the Slurm login GPU guard.
  • Transition DGX systems safely from legacy nv_peer_mem to the in-tree
    nvidia_peermem module.
  • Make exclusive-job detection custom-prefix aware and fail closed when Slurm
    queries fail or return malformed data.
  • Install a command-scoped AppArmor profile for Enroot instead of changing the
    host-wide unprivileged-user-namespace policy.
  • Select and verify the DGX OS 7 DCGM 4 package on Ubuntu 24.04 and newer.
  • Put custom-prefix Slurm client binaries on the exporter container's PATH as
    read-only mounts.
  • Avoid repeating a direct-login driver test after Slurm has intentionally
    hidden GPUs from SSH; allocated srun validation remains authoritative.

This incorporates and hardens the useful changes proposed in #1389 through
#1395. The GPU power/clock batching from #1393 remains separate because it is
an optimization rather than part of this correctness fix.

Validation

  • python3 -m unittest discover scripts/validation/tests — 25 tests passed,
    including failed, empty, malformed, exclusive, and last-user Slurm query
    cases.
  • Full Ansible role lint passed with ansible-lint 26.1.1.
  • Syntax checks passed for the Slurm, Kubernetes, monitoring, GPU, and MAAS
    playbooks.
  • Public-content sanitization and the complete non-live release QA pipeline
    passed.
  • Exact commit bc571ff9 passed a fresh Ubuntu 24.04 GPU-backed Slurm
    deployment and its immediate convergence rerun. The final playbook completed
    with failed=0; the validator proved direct-login GPU isolation, allocated
    srun GPU access, a populated custom-prefix Slurm exporter scrape with zero
    collector errors, a GPU container through Pyxis/Enroot, and correct
    exclusive/nonexclusive prolog selection.
  • The live run also loaded the command-scoped AppArmor profile while preserving
    the host-wide unprivileged-user-namespace restriction.

All 30 public CI checks pass at the validated commit.

Resolve the release-blocking SSH guard, peer-memory, DCGM, exporter, and Pyxis contracts while preserving standalone component behavior and fail-closed validation.
@dholt
dholt force-pushed the dholt/release-26.09-blockers branch from adbefea to bc571ff Compare September 4, 2026 22:23
@dholt
dholt marked this pull request as ready for review September 4, 2026 23:08
@dholt
dholt requested a review from michael-balint September 4, 2026 23:08
@dholt dholt changed the title Fix GPU runtime convergence across Slurm deployment paths fix: harden GPU runtime convergence Sep 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant