[gspark-etri]

Background

MPSManager.start decides whether an MPS daemon is running with _mps_daemon_running, which fell back to a host-wide pgrep -x nvidia-cuda-mps-server|control (#166). A daemon of another job, serving another pipe directory, therefore looked like ours. The health probe (get_server_list on our pipe dir) then failed, and _force_cleanup_stale_daemon ran pkill -u $UID -x on every MPS server and control process of the user. For example: job A runs with /tmp/mps-a, job B starts with the default dir, and job A's daemon is killed.

Changes

• opt_prime/opt_prime/mps_manager.py _mps_daemon_running (was :504-530): liveness is our pipe directory's control socket only; the two pgrep probes are removed.
• opt_prime/opt_prime/mps_manager.py _force_cleanup_stale_daemon (was :801-808): the pkill loop is removed. The cleanup still sends quit to our pipe dir (via CUDA_MPS_PIPE_DIRECTORY) and removes our control socket so a new daemon can bind.
• New opt_prime/tests/test_mps_manager.py: with a fake pipe dir and a subprocess.run stub on which every command succeeds (as if another job's daemon were up), _mps_daemon_running is False until our control socket exists, and the stale cleanup issues no pkill/killall and removes our socket.

Net -30 lines in mps_manager.py.

Verification

Environment: Python 3.10, torch 2.10.0+cu128, CPU only (no MPS daemon started). Base: optimus-rm @253be20.

cd opt_prime && TORCH_DEVICE_BACKEND_AUTOLOAD=0 CUDA_VISIBLE_DEVICES=1 PYTHONPATH=$PWD python3 -m pytest -q tests/test_mps_manager.py

• Before the fix: 2 failed. _mps_daemon_running returned True for an empty pipe dir, and the cleanup issued pkill -u <uid> -x nvidia-cuda-mps-server and ... nvidia-cuda-mps-control.
• After the fix: 2 passed.

Full suite (cd opt_prime && TORCH_DEVICE_BACKEND_AUTOLOAD=0 CUDA_VISIBLE_DEVICES=1 PYTHONPATH=$PWD python3 -m pytest -q tests):

• Before (253be20): 400 passed, 23 skipped, 135 subtests passed
• After: 402 passed, 23 skipped, 135 subtests passed

Compatibility and limits

• A daemon of ours that was started without our pipe directory, or that lost its control socket, is no longer detected. start() then spawns a new daemon on our pipe dir, which is the intended scope.
• If our own daemon hangs and ignores quit, its process is no longer force-killed; the socket is removed and a new daemon is started. Killing it would need its PID tied to our pipe dir, which the manager does not track today.
• Not exercised against a real MPS daemon here (the GPUs were in use by another job).

Related Issue

Closes #166

🤖 Generated with Claude Code