Repository navigation
Let a tong mount a persistent named volume - #94
Merged
Merged
Conversation
A tong had nowhere to keep state: anything it wrote inside its container was lost whenever the container was recreated, which for a shared tong happens on every definition change. A model store such as ollama's is tens of GB to download again. Tongs may never bind a raw host path, so the new `volume:<name>:/target[:mode]` magic word asks for a docker named volume that the launcher names, and docker creates on first use and keeps when the container goes. The docker name is `swarmforge-volume-[<hint>-]<digest>_<name>`. The digest is the volume's identity: the first 32 hex digits of a SHA-256 over the tong's scope and its raw name. Joining a scope token and a sanitized name with `-` would let distinct tongs collide, both by aliasing (`github.` and `github` sanitize alike) and by brute-forcing a short digest. The hint is the sanitized name, kept only so `docker volume ls` is readable. Volume names may not contain `_` and the digest has a fixed width, so the last `_` always splits the volume name off. A volume mount must be given a scope; choosing one per layer comes next. - `parse_mount` keeps returning `(word, target, mode)` so its callers are untouched; `mount_volume_name` reads the name through the same grammar, so validation and argv assembly cannot disagree on it. - A volume target follows the workspace target's rules: it must be absolute, must not resolve to `/`, and may not overlap the tong's wiring or its other mounts. - A tong mounts each volume once. A second target for the same data, possibly with a different access mode, is much more likely a mistake than something anyone needs. - Volume names are capped at 64 characters and the readable hint at 32, keeping the docker name well inside a 255-byte directory name.
A volume named only after its tong would hand one org's data to another org that ships a tong with the same filename, and one cloned repo's data to any other clone. `volume_scope` now picks the scope that goes into the volume's digest from the layer the tong came from: - org: the org's tongs directory, on both lifecycles - workspace: the checkout's top-level path, with symlinks resolved, so every session in the checkout shares the volume whichever subdirectory it starts from. The path is the key, as it is for approvals, so a later checkout at the same path inherits it. - user and repo: this machine, so a repo tong and the same-named user tong it replaces in the merge share one volume An org or workspace tong that mounts a volume refuses to start when there is no directory to scope it by, instead of falling back to the machine-wide scope. A workspace-sourced `shared` tong that mounts a volume is refused. Its container name is global but its volume belongs to one checkout, so a session in another checkout that ships the same definition would reuse the container with the first checkout's volume still mounted. Refusing it follows the existing rule against a `shared` tong mounting the workspace. Teardown never touches a volume, and a shared tong recreated after a definition change mounts the same volume again.
A workspace tong that mounts a volume leaves data on the host's docker after the tong is removed. The prompt already lists the raw `volume:...` mount, but nothing in that line says the data persists. The prompt now adds a `volumes:` line that names each volume and says it is kept. The prompt does not show the resolved docker volume name. The launcher derives that name from the tong's name and the checkout, so the definition cannot choose it, and it would not help the reviewer decide whether to approve. Leaving it out also keeps `privilege_summary` a function of the definition alone. The docs explain how volumes are named, for anyone who wants to find or remove one. The gate runs before validation, so a malformed volume entry is left off the `volumes:` line rather than raising. It still appears among the raw mounts, and validation refuses it before anything starts.
The definition reference covers the `volume:<name>:/target[:mode]` mount. It describes how the docker volume is named and scoped per layer, explains that the launcher never removes a volume and how to remove one with `docker volume rm`, and says why the mount is unrelated to the not-yet-wired `interface.kind: volume`. The lifecycle notes now tell `shared` tong authors where to keep state that must survive a recreation, and the approval page mentions the new `volumes:` line.
With GPUs and volumes both available, an ollama-style model server can run as a tong: a `shared` `port` tong with `resources.gpus` and its model store in a `volume:` mount, so recreating the container does not download the models again. The definition reference now shows one complete, minimal example. The example tong is named `models` rather than `ollama`. `make run_ollama` already answers to `ollama` on the same network, so a tong by that name would collide with it.
Conflicts:
swarmforge/anvil/orchestrate.py
- run_with_tongs docstring: this branch's note that named volumes
are never removed vs. master's SIGHUP/SIGTERM and teardown_guard()
wording -> kept master's exit/teardown text and added the volume
note after it
A workspace tong's filename reached stderr raw through the merge, refusal and readiness warnings; only the skipped-file warning was escaped. Escaping inside `tongs.warn` covers them all, and the git-dir and `tongs validate` messages now route through the same escape.
`_has_socket_mount` was `_mounts_word` with the word fixed. Move the general check down to `tongs.mounts` and have the socket, workspace and volume checks all use it.
`parse_mount` dropped the volume name, so every overlap check re-parsed the new mount and each placed one to get it back. Return the name from the parse and carry it in the placed entries instead.
Volumes resolved symlinks in the workspace path but approvals did not, so one checkout reached two ways needed two approvals yet shared one volume. Both now use `workspace_key`. A missing org or workspace dir is refused once, in `tong_mount_specs`, instead of at three layers.
`tong_volume_name` repeated `dir_hint`'s truncate-and-trim rule. Split that rule out as `names.name_hint` and use it for both.
A linked worktree's git dir is bind-mounted at its host path beside the workspace, outside the definition's own mounts, so the overlap check never saw it. A volume at /home could enclose it, and docker would leave its mountpoints inside the persistent volume.
The prompt listed bare volume names, so a reviewer could not see where the data lands or which existing docker volume it reuses. It now names both and says that concurrent sessions mount the same volume.
State that every concurrent session mounts a tong's volume, that the user and repo layers share one across checkouts, and that docker seeds a volume from the image only once, so targets should hold data only.
The never-removed, workspace-shared-refusal and scope rules were each restated across several docstrings and doc pages. Keep one copy of each and cut the rest back.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #93, which is stacked on #92. Merge those first, in that order, and GitHub will retarget this to
master. The newvolumes:line in the approval prompt relies on #93's escaping, and the docs example uses #92'sresources.gpus.Why
A tong had nowhere to keep state. A
sharedtong is recreated whenever its definition changes or its container is gone, so anything written inside it is lost. For a model server like Ollama that means re-downloading a multi-GB model store. Tongs never bind raw host paths, somake run_ollama's bind ofollama/is not an option. This adds a docker named volume that the launcher names and scopes, so the Ollama tong being built outside this repo can keep its models. The existing Ollama make targets are unchanged.What changes
volume:<name>:/target[:mode]mount word:[A-Za-z0-9.-], starting with a letter or digit. The target is required and follows the same rules as aworkspacetarget (absolute, no whitespace, not/, no overlap with the tong's reserved paths or its other mounts).parse_mountkeeps its(word, target, mode)shape. The name comes frommount_volume_name, and both share one splitter, so validation, argv assembly and the approval prompt agree.swarmforge-volume-[<hint>-]<digest>_<name>.[scope class, scope path, raw tong filename]. The hint is the sanitized filename cut to 32 characters, fordocker volume lsonly.sharedtong may not mount a volume. Its container is global and reused by sessions in other checkouts, which would then mount this checkout's volume. This follows the existing refusal of asharedtong that mounts the workspace.sharedtong is recreated. No new docker seam was needed.volumes: <names> (persistent: outlives the tong and reuses any data left by an earlier definition or checkout at this path).docs/tongs/definitions.mdcovers the grammar, naming, the scope per layer, and that the workspace scope is keyed by path: a later checkout at the same path inherits the volume, even if it is a different repository, just as it inherits approvals. It also covers cleanup withdocker volume rm, and how this differs from the still-unwiredinterface.kind: volume. It also has a short GPU model-server example. That tong is namedmodelsbecause a tong namedollamawould clash withmake run_ollamaon the same network.docs/tongs/README.mdanddocs/tongs/approval.mdare updated to match.Testing
make test: 996 tests pass (959 on the base). Each commit passes on its own.ruff checkis clean.docker runagainst a daemon. A manual check is to run themodelsexample from the docs twice, with an edit to the definition in between, and confirm pulled models survive.Follow-ups not in this PR
OLLAMA_HOST). Both are needed before the Ollama handling here can be retired.