Skip to content

Let a tong request host GPUs via resources.gpus - #92

Merged
CrypticSwarm merged 3 commits into
masterfrom
gpu-support
Oct 9, 2026
Merged

CrypticSwarm merged 3 commits into
masterfrom
gpu-support

Conversation

@CrypticSwarm

Copy link
Copy Markdown
Owner

A tong could not ask for GPU access, so a model server like Ollama could not run as a tong with the GPU support make run_ollama gives it today (--gpus=all). This adds resources.gpus, passed to docker run --gpus, so that tong can be built outside this repo. The existing Ollama make targets are unchanged.

What changes

  • resources.gpus: a new field, parsed by one helper (parse_gpus in swarmforge/tongs/model.py) that both validation and argv assembly call, so the two cannot disagree. Only these forms are accepted:

    • all
    • a positive count (an int or ASCII digits, at most 2³¹−1)
    • device=<id>, or the docker-quoted list "device=<id>,<id>", where each id starts with a letter or digit and continues with letters, digits, and ._:-

    Docker's other --gpus keys are refused. driver=, capabilities= and options= can grant more than GPU access (driver=cdi injects any device registered on the host). count= is redundant with a plain count. Leaving --gpus free-form would also have let control characters reach docker. An unquoted device=0,1, the likeliest mistake given docker's CSV parsing, is refused with the exact YAML quoting to use instead.

  • Approval gate: GPU access is a host-device grant, so the privilege summary reports it and a workspace tong's approval prompt shows a gpus: <value> (host GPU access) line. The config hash already covers the whole definition, so adding or changing gpus asks for approval again and recreates a running shared tong. Tests cover both.

  • Docs: a new Resources section in docs/tongs/definitions.md covers the accepted forms, the quoting, why the other keys are refused, and the host requirement. GPU containers need the NVIDIA Container Toolkit on Linux; Docker Desktop's WSL 2 backend provides GPU support itself. Without it docker run fails and the launch stops. docs/tongs/approval.md lists GPU access among what the prompt shows.

Testing

  • make test: 948 tests pass (923 on master). Each commit also passes on its own.
  • ruff check is clean.
  • New tests cover every accepted form and every refused class (control characters, trailing newlines, bools, zero, over the cap, non-ASCII digits, the refused --gpus keys), plus the --gpus flag placement before the image in the tong's docker run argv and through the orchestrator, the approval prompt line, and a non-mapping resources value.
  • Not exercised here: a real docker run --gpus against a GPU host. That needs a host with NVIDIA tooling. The quickest manual check is a user-layer shared tong with resources: {gpus: all} running nvidia-smi.

A tong could not reach the host's GPUs, so a GPU-backed service such as
a model server could not run as one. `resources.gpus` now maps to
`docker run --gpus`.

The value is held to a strict grammar rather than passed through free
form: `all`, a count from 1 to 2^31-1 (an int or ASCII digits),
`device=<id>`, or the docker-quoted list `"device=<id>,<id>,..."`.
Docker's `driver=`, `capabilities=`, and `options=` keys are refused
because they grant more than GPU access -- `driver=cdi` injects any
registered device -- and `count=` because a plain count says the same.
A free-form string could also carry control bytes that rewrite the
approval prompt yet still launch; no such value passes validation now.
An unquoted `device=0,1`, the likeliest mistake since docker reads the
value as CSV, gets an error that shows the quoting it needs.

Validation and argv assembly both go through `parse_gpus`, so the two
cannot disagree about what is accepted, and a bad value stops the
launch before docker runs.

The config hash already covers the whole definition, so adding or
changing a GPU request recreates a running `shared` tong.
GPU access is a grant of host devices, so a workspace-sourced tong that
asks for it must say so before the user approves it. The privilege
summary now carries `resources.gpus` and the prompt renders it on its
own line; a tong that asks for none shows nothing.

The gate runs before validation, so the summary reports the value as
declared rather than parsed: whatever could reach docker is what the
user sees. Approval is keyed by the definition hash, so adding a GPU
request to an approved tong re-prompts.
Describe `resources.gpus` next to `memory`: the forms it accepts, the
literal double quotes docker's CSV parsing needs around a device list,
why docker's other `--gpus` keys are refused, the GPU-capable docker
host it needs, and that a workspace tong's request shows in the
approval prompt.
@CrypticSwarm
CrypticSwarm merged commit 5ecf5d7 into master Oct 9, 2026
4 checks passed
@CrypticSwarm
CrypticSwarm deleted the gpu-support branch October 9, 2026 01:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant