Repository navigation
feat(runner): berth train --exclusive locks the card in the driver - #318
Merged
Merged
Conversation
berth train --exclusive puts the granted cards in exclusive-process compute mode after the grant and before the command starts, so a program that opens CUDA without Berth cannot open them, and puts them back to the default mode when the run ends. It refuses to start when another compute process is on the card, when no sudo rule lets nvidia-smi -c run without a password, and on a card with no NVIDIA UUID. Linux only.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
berth train --exclusivelocks the card in the driver too, on Linux with an NVIDIA card.(
sudo -n nvidia-smi -i <UUIDs> -c EXCLUSIVE_PROCESS). Only the training's process can then open a CUDA contextthere. When the run ends, it puts them back to
DEFAULT.lets
nvidia-smi -crun without a password, or when the card has no NVIDIA UUID.--exclusivebelongs toberth trainalone.berth run --exclusiveis refused.Why
The training lock (#314, #317) covers everything that asks Berth for the card. A program that opens CUDA without
Berth was only reported as a stowaway, and could still push a training out of memory. Only the driver can stop it.
How
acquirereturns the grant message, so the run reads the granted cards' UUIDs.exclusiveininternal/runner/exclusive.gochecks the compute apps, sets the mode through the runner'sCLIhook, and returns the restore func the run defers.
Proof
.claude/skills/berth-verify/proof.sh: full proof OK.TestExclusiveSetsAndRestoresTheMode,TestExclusiveRefusesWhenAProcessIsOnTheCard,TestExclusiveRefusesWithoutSudoandTestExclusiveNeedsAnNVIDIACardfakenvidia-smiand assert the argv.TestTrainExclusiveRefusesWithoutAnNVIDIACardrunsberth train --exclusiveagainst a real berthd whose card hasno UUID: refused after the grant, command never started. It fails when the runner skips the exclusive step.
berth train --exclusiverun after the sudo rule is in place.