prometheus-slurm-exporter: mount Slurm client tools onto the container PATH

[100milliongold]

Problem

roles/prometheus-slurm-exporter/templates/docker.slurm-exporter.service.j2 mounts the Slurm client tools into the container at the host path:

-v {{ slurm_install_prefix }}/bin/sinfo:{{ slurm_install_prefix }}/bin/sinfo

The exporter invokes them by name, so the destination must be on the container's PATH. With the default slurm_install_prefix the destination is /usr/local/bin, which already is; with a custom prefix it is not, and every scrape run dies:

exec: "sinfo": executable file not found in $PATH

Restart=always means systemctl status keeps reporting active and the play completes normally. The only visible symptoms are an empty Grafana dashboard and, in Prometheus, Get "http://<node>:8080/metrics": EOF. On our node the unit had restarted 3289 times before anyone noticed.

Reproduced on DGX B300 / DGX OS 7.5.0 with slurm_install_prefix: /raid/slurm/usr/local.

Fix

Pin the three destination paths to /usr/local/bin. Behaviour with the default prefix is unchanged, since that is where they already landed.

The library mount is deliberately left as-is: shared objects are resolved by path, not by PATH.

Verification

After changing the destinations on the affected node:

$ systemctl status docker.slurm-exporter.service | grep Active
     Active: active (running) since ...; 20s ago      # previously restarting every ~15s

$ curl -s localhost:8080/metrics | grep -c "^slurm"
34

$ curl -s localhost:9090/api/v1/targets | grep -o '"health":"[a-z]*"' | sort | uniq -c
      4 "health":"up"

The Grafana SLURM dashboard populates as expected.