nvidia_dcgm: do not replace the DCGM that DGX OS already ships

[100-JM]

Problem

roles/nvidia_dcgm/tasks/install-dgx.yml installs the unversioned datacenter-gpu-manager package on every DGX. In the CUDA apt repo that name resolves to DCGM 3.3.9. DGX OS 7.x already ships datacenter-gpu-manager-4-cuda13 (4.5.2). The two series conflict, so apt removes the 4.x packages and installs 3.3.9. The task reports ok, so nothing in the play fails.

The regression is only visible afterwards on the node:

$ dcgmi diag -r 1
Detected unsupported Cuda version        # 3.3.9 ships cuda10/11/12 plugins only; driver reports CUDA 13.0
$ dcgmi discovery -l | grep found
8 GPUs found.
0 NvSwitches found.                      # DGX B300 has two

The Ubuntu branch (install-ubuntu.yml) already uses {{ dcgm_pkg_name }}; only the DGX branch hardcodes the name. roles/nvidia-dgx/vars/ubuntu-24.04.yml lists datacenter-gpu-manager-4-cuda13 for DGX OS 7, so the two roles currently disagree.

Reproduced on DGX B300, DGX OS 7.5.0, driver 580.126.20, install_dcgm: true (default) via playbooks/slurm-cluster.yml.

Fix

On DGX, gather package facts and skip the install when any datacenter-gpu-manager* package is already present (DGX OS provides it). When nothing is present, install {{ dcgm_pkg_name }} like the Ubuntu branch does. Non-DGX branches are unchanged.

Verification

Restored the DGX OS version on the node (apt-get install datacenter-gpu-manager-4-cuda13=1:4.5.2-1 datacenter-gpu-manager-4-core=1:4.5.2-1):

$ dcgmi discovery -l | grep found
8 GPUs found (Active).
2 NvSwitches found.
$ dcgmi diag -r 1
DCGM Version 4.5.2, Driver 580.126.20, software: Pass on GPU0-7

Fabric Manager logs show the same two switches with 74 active links each. With the patched task, a re-run of slurm-cluster.yml leaves the 4.5.2 packages in place.

[dholt]

Please make the DGX task validate the specific acceptable DCGM package/version rather than treating any datacenter-gpu-manager* package as sufficient. Handle at least the existing legacy 3.x state, partial package sets, and the fully absent state so each converges to the correct DGX OS package; also ensure the absent-state package name is explicitly DGX-compatible instead of relying on the current unversioned default.

---

Automated triage review (agent-generated on the maintainer's behalf; a human maintainer decides merges).