diff --git a/docs/tongs/README.md b/docs/tongs/README.md index efaef4c..fe03a58 100644 --- a/docs/tongs/README.md +++ b/docs/tongs/README.md @@ -30,7 +30,7 @@ The workspace layer is gated by [first-run approval](approval.md). ## Lifecycle - **`session`** — started with the anvil, torn down when it exits. Per-session isolation; the default for credential tongs. -- **`shared`** — long-lived, survives across anvil sessions (ollama-style). Started on first use, connected to each session's network via a network alias, and left running on teardown (no refcounting). A running `shared` container whose config-hash docker label still matches the current definition is reused untouched; a missing, stopped, or stale one is recreated automatically. A rotated secret behind an unchanged reference does **not** churn it — force a restart with `docker rm -f `. A `shared` tong may not mount the `workspace` (it would leak one session's workspace into the next); use a `session` tong for per-workspace mounts. +- **`shared`** — long-lived, survives across anvil sessions (ollama-style). Started on first use, connected to each session's network via a network alias, and left running on teardown (no refcounting). A running `shared` container whose config-hash docker label still matches the current definition is reused untouched; a missing, stopped, or stale one is recreated automatically. A rotated secret behind an unchanged reference does **not** churn it — force a restart with `docker rm -f `. Recreating the container loses what the tong wrote inside it; state worth keeping, such as a model store, belongs in a [`volume:` mount](definitions.md#mounts). A `shared` tong may not mount the `workspace` (it would leak one session's workspace into the next); use a `session` tong for per-workspace mounts. ## Reference diff --git a/docs/tongs/approval.md b/docs/tongs/approval.md index be7de8f..8982a89 100644 --- a/docs/tongs/approval.md +++ b/docs/tongs/approval.md @@ -3,7 +3,7 @@ The user, org, and repo layers are installed deliberately and are trusted; they skip the gate. A **workspace**-sourced tong (from a repo you cloned) could otherwise request your secrets, host mounts, the docker socket, or the host's GPUs simply by being present, so the launcher gates it: -- Before starting, it prints exactly what the tong requests — image, secret references, mounts, networks, docker-socket access, and GPU access — and asks you to approve. Control and formatting characters in those values are shown as escapes such as `\x1b` or `\n`, so a value cannot move the cursor, restyle, or hide any part of the summary. -- Approval is keyed by workspace path + tong name + a hash of the merged definition, stored in `~/.swarmforge/approvals.json`. Any change to the definition re-prompts. +- Before starting, it prints exactly what the tong requests — image, secret references, mounts, networks, docker-socket access, and GPU access — and asks you to approve. Control and formatting characters in those values are shown as escapes such as `\x1b` or `\n`, so a value cannot move the cursor, restyle, or hide any part of the summary. Every `tongs:` warning escapes them the same way. Each volume is listed with its target and docker volume name, with a note that its data persists and is shared by concurrent sessions. +- Approval is keyed by workspace path (symlinks resolved, as for its [volumes](definitions.md#volume-naming-and-scope)) + tong name + a hash of the merged definition, stored in `~/.swarmforge/approvals.json`. Any change to the definition re-prompts. - The gate defaults to **No**, and a non-interactive stdin reads as No. A scripted `--no-prompt` run **fails closed** rather than auto-approving. - Approving `image: foo:latest` approves a moving target; **pinned digests are the recommended convention** for workspace tongs. diff --git a/docs/tongs/definitions.md b/docs/tongs/definitions.md index 5d4687c..757af7b 100644 --- a/docs/tongs/definitions.md +++ b/docs/tongs/definitions.md @@ -16,6 +16,7 @@ interface: # required; how (or whether) the anvil reaches the # aliases: [gh, git.example] # optional extra DNS names the tong also answers to mounts: # opt-in magic words only, never raw host paths - workspace:ro # or workspace:/code:ro to bind it somewhere else +# - volume:cache:/var/cache # a persistent named volume, kept between containers resources: memory: 512m # string or number # gpus: all # optional host GPU access (see Resources) @@ -65,10 +66,21 @@ readiness: ## Mounts -Mounts are opt-in **magic words**, never raw host paths. Only two are recognized, spelled `[:/target][:mode]` with the access mode (`ro`/`rw`) last: +Mounts are opt-in **magic words**, never raw host paths. Only three are recognized, each spelled as shown below with the optional access mode (`ro`/`rw`) last: - `workspace[:/target][:mode]` — bind-mounts the session workspace, at `/workspace` unless an absolute `target` says otherwise (e.g. `workspace:ro`, `workspace:/code`, `workspace:/code:ro`). A custom target lets an image that expects its sources elsewhere be used unmodified (it does not set the working directory — the process still starts in the image's own `WORKDIR`). A target is refused unless it is an absolute path free of whitespace, and refused if it resolves to `/`, overlaps another of the tong's mounts, or overlaps a path the tong's own wiring occupies — the secret-delivery tmpfs at `/run/swarmforge` and the `/bin/sh` its wrapper execs (for a tong with secret references), or the docker socket (for a tong that mounts it). The workspace bind is paired with the same git-dir mounts the anvil gets from `swarmforge/gitguard.py`: read-only guards over the config and hooks the host's git obeys, and — when the workspace is a linked worktree or another checkout whose git dir lives outside it — that git dir at its own absolute path, which is where the checkout's `.git` pointer file says to look (without it, git inside the tong fails with "not a git repository"). The one mount the anvil gets and a tong does not is the restated worktree registration — a tong keeps the host's copy, so `git worktree list` inside one names the checkout by its host path (the [git guard](../git-guard.md) has the rest). When every `workspace` mount is `ro`, the ride-along git-dir mounts are forced read-only too. - `docker-socket[:mode]` — bind-mounts the host docker socket onto the same path inside the container (so it takes no target). This is full host docker control and is always called out explicitly in the workspace approval prompt; it is the grant a broker tong needs. +- `volume::/target[:mode]` — mounts a docker **named volume** at `target`, for state the tong keeps between containers (a model store, a cache). The name comes first and is at most 64 letters, digits, `.` and `-`, starting with a letter or digit; the target is required and held to the same rules as a `workspace` target, may not overlap the git dir that a `workspace` mount brings along, and a tong mounts each volume only once. Docker creates the volume on first use and seeds a new, empty one with whatever the image has at `target`. Only that first seeding happens: a later image's files at `target` never replace what the volume holds, so point `target` at a data directory, not at files the image ships. The launcher never removes a volume; remove one by hand with `docker volume rm ` (`docker volume ls --filter name=swarmforge-volume-` lists them). A `volume:` mount is private to its tong; it is unrelated to `interface.kind: volume`, which would share a volume with the anvil and is not wired up yet. + +### Volume naming and scope + +The docker volume is named `swarmforge-volume-[-]_`. The hint is the tong's filename made docker-safe and cut to 32 characters, only for reading. The digest, a 128-bit SHA-256 prefix over the tong's exact filename and its scope, is the identity, so two tongs share a volume only with the same filename and scope. The scope depends on the tong's layer: + +- **org** — the org's tongs directory. This applies to both lifecycles. +- **workspace** — the checkout's top-level path, with symlinks resolved. A clone or worktree at another path gets its own volume; a later checkout at the same path, even of a different repository, inherits it, as it does the approvals. Deleting a checkout leaves its volumes behind. A workspace tong that mounts a volume may not be `shared`: its container would be reused by other checkouts, still mounting this one's volume. +- **user** and **repo** — this machine, across every checkout, so anything such a tong copies from one workspace is visible in the others. A repo tong and the same-named user tong it replaces use one volume. + +A volume belongs to the tong's name, not to one definition or one session: an edited definition reuses its data, and concurrent sessions of a `session` tong mount it read-write at once. Keep state there that tolerates that, or that the program locks itself. ## Resources @@ -87,3 +99,22 @@ Both are optional and passed to `docker run` as written. `gpus` grants the tong Docker's other `--gpus` keys are deliberately refused. `driver=`, `capabilities=`, and `options=` can grant more than GPU access — `driver=cdi`, for one, injects any device registered on the host — and `count=` is redundant with a plain count. GPU access needs a docker host set up for GPU containers — the NVIDIA Container Toolkit on Linux; Docker Desktop's WSL 2 backend provides GPU support itself. Without it `docker run` fails and the launch stops. A workspace tong's GPU request is called out in its [approval prompt](approval.md). + +## Example: a GPU model server + +A `shared` `port` tong that serves models from the host's GPUs and keeps its multi-GB model store in a volume, so a recreated container does not download it again: + +```yaml +# ~/.swarmforge/tongs/models.yaml +lifecycle: shared +image: ollama/ollama@sha256:... +interface: + kind: port + port: 11434 +mounts: + - volume:store:/root/.ollama +resources: + gpus: all +``` + +The anvil gets `SWARMFORGE_TONG_MODELS_HOST=models` and `SWARMFORGE_TONG_MODELS_PORT=11434`, and the store lives in the volume `swarmforge-volume-models-_store`. It is named `models` because `make run_ollama` already answers to `ollama` on that network. diff --git a/swarmforge/anvil/approval.py b/swarmforge/anvil/approval.py index 164bec2..ba88ff5 100644 --- a/swarmforge/anvil/approval.py +++ b/swarmforge/anvil/approval.py @@ -41,6 +41,16 @@ def render_privilege_summary(name, summary): mounts = summary.get("mounts") or [] if mounts: lines.append(" mounts: %s" % ", ".join(str(m) for m in mounts)) + volumes = summary.get("volumes") or [] + for volume in volumes: + where = "%s at %s" % (volume["name"], volume["target"]) + if volume["docker"]: + where += " (docker volume %s)" % volume["docker"] + lines.append(" volume: %s" % where) + if volumes: + lines.append(" persistent: outlives the tong, is mounted by each of its " + "concurrent sessions, and reuses any data left by an earlier " + "definition or checkout at this path") networks = summary.get("networks") or [] if networks: lines.append(" networks: %s" % ", ".join(str(n) for n in networks)) @@ -108,7 +118,9 @@ def gate_workspace_tongs(merged, workspace, approvals_path, prompt=True, out=Non for name, defn in pending: if tongs.is_approved(approvals, workspace, name, defn): continue - out.write(render_privilege_summary(name, tongs.privilege_summary(defn)) + "\n") + summary = tongs.privilege_summary( + defn, name, tongs.volume_scope(tongs.WORKSPACE, workspace=workspace)) + out.write(render_privilege_summary(name, summary) + "\n") if not prompt: raise ApprovalDenied( "workspace tong %r is unapproved and --no-prompt fails closed" % name diff --git a/swarmforge/anvil/orchestrate.py b/swarmforge/anvil/orchestrate.py index 1d880ff..9d6fe09 100644 --- a/swarmforge/anvil/orchestrate.py +++ b/swarmforge/anvil/orchestrate.py @@ -18,7 +18,6 @@ import json import os import shutil -import sys import tempfile import time @@ -32,18 +31,6 @@ from .secretchan import SecretChannel, make_secret_resolver -def _mounts_workspace(defn): - """True if a tong's `mounts:` request the session workspace. - - The magic word may carry a target and/or a mode (e.g. `workspace:/code:ro`), - so compare only the word before the first colon. - """ - for mount in defn.get("mounts") or []: - if isinstance(mount, str) and mount.split(":", 1)[0] == tongs.WORKSPACE_MOUNT: - return True - return False - - def _workspace_git_dir_specs(defn, workspace, warn=None): """Git-dir mounts a workspace-mounting tong needs beyond the workspace bind. @@ -91,7 +78,9 @@ def unsupported_tong_reasons(merged): long-lived container reused across sessions, so binding one session's workspace into it would expose that workspace to every later session that reuses the container (a `session` tong is the right home for a - per-workspace mount). + per-workspace mount); + * a workspace-sourced `shared` tong that mounts a `volume` -- its volume + is scoped to one checkout but its container is not. A refused tong is reported rather than started half-wired. Returns a list of human-readable reason strings (empty == every discovered tong is startable). @@ -105,12 +94,21 @@ def unsupported_tong_reasons(merged): "tong '%s' has a 'volume' interface, which this launcher does not " "wire up" % name ) - if defn.get("lifecycle") == "shared" and _mounts_workspace(defn): + if defn.get("lifecycle") == "shared" and tongs.mounts_word(defn, tongs.WORKSPACE_MOUNT): reasons.append( "tong '%s' is a 'shared' tong that mounts the workspace; a shared " "container is reused across sessions, so it would leak one " "session's workspace into the next" % name ) + if (defn.get("lifecycle") == "shared" + and tongs.is_workspace_sourced(merged[name]["source"]) + and tongs.mounts_word(defn, tongs.VOLUME_MOUNT)): + reasons.append( + "tong '%s' is a workspace 'shared' tong that mounts a volume; its " + "volume belongs to this checkout but its container is reused by " + "sessions in other checkouts, so it would leak this checkout's " + "data into theirs (make it a 'session' tong)" % name + ) return reasons @@ -135,7 +133,7 @@ def ensure_mcp_harness_supported(merged, harness): def _start_one_tong(docker, name, defn, *, container, network, alias, - resolver, workspace, label_hash, make_channel): + resolver, workspace, label_hash, make_channel, volume_scope): """Start one tong container detached, delivering any secret env over a FIFO. Resolves the definition's secret references through `resolver` and splits the @@ -165,8 +163,7 @@ def _start_one_tong(docker, name, defn, *, container, network, alias, try: git_dir_specs = _workspace_git_dir_specs( defn, workspace, - warn=lambda message: print("tong '%s': %s" % (name, message), - file=sys.stderr)) + warn=lambda message: tongs.warn("tong '%s': %s" % (name, message))) except ValueError as exc: raise OrchestrationError("tong '%s': %s" % (name, exc)) @@ -176,7 +173,7 @@ def _start_one_tong(docker, name, defn, *, container, network, alias, name, defn, container_name=container, network=network, alias=alias, env=plain_env, label_hash=label_hash, workspace=workspace, - extra_mount_specs=git_dir_specs, + extra_mount_specs=git_dir_specs, volume_scope=volume_scope, ) except ValueError as exc: raise OrchestrationError("tong '%s': %s" % (name, exc)) @@ -199,7 +196,7 @@ def _start_one_tong(docker, name, defn, *, container, network, alias, container_name=container, network=network, alias=alias, env=plain_env, label_hash=label_hash, workspace=workspace, secret_channel=True, entrypoint=entrypoint, command=command, - extra_mount_specs=git_dir_specs, + extra_mount_specs=git_dir_specs, volume_scope=volume_scope, ) except ValueError as exc: raise OrchestrationError("tong '%s': %s" % (name, exc)) @@ -213,7 +210,8 @@ def _start_one_tong(docker, name, defn, *, container, network, alias, def _ensure_shared_tong(docker, name, defn, *, container, network, alias, - resolver, workspace, label_hash, make_channel): + resolver, workspace, label_hash, make_channel, + volume_scope): """Start a `shared` tong, or reuse the running one, recreating it if stale. A `shared` tong is one long-lived container keyed by `shared_container_name`. @@ -232,7 +230,7 @@ def _ensure_shared_tong(docker, name, defn, *, container, network, alias, docker, name, defn, container=container, network=network, alias=alias, resolver=resolver, workspace=workspace, label_hash=label_hash, - make_channel=make_channel, + make_channel=make_channel, volume_scope=volume_scope, ) @@ -334,7 +332,8 @@ def run_with_tongs(merged, anvil_cmd, opts, *, docker, providers=None, ) # No org layer means no token, and every shared tong keeps its global unscoped name. - org_token = tongs.org_scope_token(dict(opts.layer_dirs).get(tongs.ORG)) + org_tongs_dir = dict(opts.layer_dirs).get(tongs.ORG) + org_token = tongs.org_scope_token(org_tongs_dir) has_org_shared = bool(org_token) and any( merged[name]["definition"].get("lifecycle") != "session" and merged[name]["source"] == tongs.ORG @@ -346,6 +345,12 @@ def run_with_tongs(merged, anvil_cmd, opts, *, docker, providers=None, "join their isolated network" ) + volume_scopes = { + name: tongs.volume_scope(merged[name]["source"], org_tongs_dir=org_tongs_dir, + workspace=opts.workspace) + for name in merged + } + plan = tongs.plan_network(merged, base_network, session_id) injection = tongs.plan_injection(merged, opts.harness) @@ -373,6 +378,7 @@ def run_with_tongs(merged, anvil_cmd, opts, *, docker, providers=None, container=container, network=plan["network"], alias=alias, resolver=resolver, workspace=opts.workspace, label_hash=tongs.config_hash(defn), make_channel=make_channel, + volume_scope=volume_scopes[name], ) probe_network = plan["network"] else: @@ -392,6 +398,7 @@ def run_with_tongs(merged, anvil_cmd, opts, *, docker, providers=None, container=container, network=tong_network, alias=alias, resolver=resolver, workspace=opts.workspace, label_hash=tongs.config_hash(defn), make_channel=make_channel, + volume_scope=volume_scopes[name], ) ready_checks.append((name, defn, alias, container, probe_network)) diff --git a/swarmforge/names.py b/swarmforge/names.py index 78a3ff7..1966eda 100644 --- a/swarmforge/names.py +++ b/swarmforge/names.py @@ -53,12 +53,17 @@ def path_digest(path): return digest.hexdigest()[:DIGEST_LENGTH] -def dir_hint(path, limit=None): - """`path`'s basename as a docker name, cut to `limit` and not left on a separator.""" - hint = sanitize_token(os.path.basename(canonical_path(path))) +def name_hint(name, limit=None): + """`name` as a docker name, cut to `limit` and not left on a separator.""" + hint = sanitize_token(name) return hint[:limit].rstrip("-_.") if limit is not None else hint +def dir_hint(path, limit=None): + """`path`'s basename as a `name_hint`.""" + return name_hint(os.path.basename(canonical_path(path)), limit) + + def path_token(path, hint): """Docker-name token for one host directory: `hint`, then `path`'s digest. diff --git a/swarmforge/tongs/__init__.py b/swarmforge/tongs/__init__.py index 6a67683..6b5468e 100644 --- a/swarmforge/tongs/__init__.py +++ b/swarmforge/tongs/__init__.py @@ -15,7 +15,7 @@ discovery reading the layer directories and merging by name validate schema validation secrets secret references, the provider table, FIFO delivery - mounts the `mounts:` magic words and their docker bind specs + mounts the `mounts:` magic words and their docker `-v` specs mcp what each `interface:` contributes to the anvil network per-session network planning approvals config hash, privilege summary, approval keying @@ -107,20 +107,30 @@ printable, readiness_settings, warn, + workspace_key, ) from .mounts import ( DEFAULT_DOCKER_SOCKET, DEFAULT_WORKSPACE_MOUNT_TARGET, + LOCAL_VOLUME_SCOPE, MOUNT_MODES, MOUNT_WORDS, + ORG_VOLUME_SCOPE, + VOLUME_DIGEST_LENGTH, + VOLUME_MOUNT, + VOLUME_NAME_MAX_LENGTH, WORKSPACE_MOUNT, + WORKSPACE_VOLUME_SCOPE, mount_destination, mount_target_error, + mounts_word, normalize_mount_target, overlapping_mount_error, parse_mount, reserved_mount_targets, tong_mount_specs, + tong_volume_name, + volume_scope, workspace_mount_placements, ) from .network import SESSION_NET_PREFIX, plan_network, session_network_name @@ -210,18 +220,28 @@ "printable", "readiness_settings", "warn", + "workspace_key", "DEFAULT_DOCKER_SOCKET", "DEFAULT_WORKSPACE_MOUNT_TARGET", + "LOCAL_VOLUME_SCOPE", "MOUNT_MODES", "MOUNT_WORDS", + "ORG_VOLUME_SCOPE", + "VOLUME_DIGEST_LENGTH", + "VOLUME_MOUNT", + "VOLUME_NAME_MAX_LENGTH", "WORKSPACE_MOUNT", + "WORKSPACE_VOLUME_SCOPE", "mount_destination", "mount_target_error", + "mounts_word", "normalize_mount_target", "overlapping_mount_error", "parse_mount", "reserved_mount_targets", "tong_mount_specs", + "tong_volume_name", + "volume_scope", "workspace_mount_placements", "SESSION_NET_PREFIX", "plan_network", diff --git a/swarmforge/tongs/approvals.py b/swarmforge/tongs/approvals.py index 37d9a4d..3ba552c 100644 --- a/swarmforge/tongs/approvals.py +++ b/swarmforge/tongs/approvals.py @@ -11,8 +11,8 @@ import json import os -from .model import WORKSPACE -from .mounts import _has_socket_mount +from .model import SOCKET_MOUNT, WORKSPACE, workspace_key +from .mounts import mounts_word, parse_mount, tong_volume_name from .secrets import find_secret_refs @@ -28,20 +28,22 @@ def config_hash(defn): return hashlib.sha256(canonical.encode("utf-8")).hexdigest() -def privilege_summary(defn): +def privilege_summary(defn, name=None, volume_scope=None): """Structured summary of what a definition asks for, for the approval gate. Gathers the privileges a reviewer must see before approving a - workspace-sourced tong: image, secret references, mounts, networks, - docker-socket access, and host GPU access. Rendering and prompting are the - caller's job; this just assembles the facts. + workspace-sourced tong: image, secret references, mounts, volumes, networks, + docker-socket access, and host GPU access. Each volume is `{"name", "target", + "docker"}`; `docker` is None unless `name` and `volume_scope` are given. + Rendering and prompting are the caller's job; this just assembles the facts. """ return { "image": defn.get("image"), "secrets": [{"provider": p, "ref": r} for p, r in find_secret_refs(defn)], "mounts": list(defn.get("mounts") or []), + "volumes": _declared_volumes(defn, name, volume_scope), "networks": list(defn.get("networks") or []), - "socket": _has_socket_mount(defn), + "socket": mounts_word(defn, SOCKET_MOUNT), "gpus": _requested_gpus(defn), } @@ -56,6 +58,28 @@ def _requested_gpus(defn): return resources.get("gpus") if isinstance(resources, dict) else None +def _declared_volumes(defn, name, volume_scope): + """The volumes a definition's well-formed `volume:` mounts declare. + + The gate runs before validation, so a malformed entry is skipped here rather + than raised: it is still listed among the raw mounts, and validation refuses + it before anything starts. + """ + volumes = [] + for mount in defn.get("mounts") or []: + if not isinstance(mount, str): + continue + try: + _, volume, target, _ = parse_mount(mount) + except ValueError: + continue + if volume: + docker = (tong_volume_name(name, volume, volume_scope) + if name and volume_scope else None) + volumes.append({"name": volume, "target": target, "docker": docker}) + return volumes + + def is_workspace_sourced(source_layer): """True if a tong's winning layer is the (untrusted) workspace and so gates.""" return source_layer == WORKSPACE @@ -87,7 +111,7 @@ def is_approved(approvals, workspace_path, name, defn): Fails closed (returns False) on a missing or malformed store entry rather than raising -- a hand-edited approvals.json must never crash the gate. """ - entry = approvals.get(workspace_path) + entry = approvals.get(workspace_key(workspace_path)) if not isinstance(entry, dict): return False return entry.get(name) == config_hash(defn) @@ -98,5 +122,5 @@ def record_approval(approvals, workspace_path, name, defn): Mutates and returns the store (same object) so callers can persist it. """ - approvals.setdefault(workspace_path, {})[name] = config_hash(defn) + approvals.setdefault(workspace_key(workspace_path), {})[name] = config_hash(defn) return approvals diff --git a/swarmforge/tongs/argv.py b/swarmforge/tongs/argv.py index 4020bf6..019b75c 100644 --- a/swarmforge/tongs/argv.py +++ b/swarmforge/tongs/argv.py @@ -12,8 +12,14 @@ from swarmforge.names import canonical_path, dir_hint, path_token, sanitize_token from .mcp import _is_network_facing, _ordered_aliases -from .model import LABEL_CONFIG_HASH, LABEL_TONG_NAME, WORKSPACE_HOST_ENV, parse_gpus -from .mounts import DEFAULT_DOCKER_SOCKET, _has_socket_mount, tong_mount_specs +from .model import ( + LABEL_CONFIG_HASH, + LABEL_TONG_NAME, + SOCKET_MOUNT, + WORKSPACE_HOST_ENV, + parse_gpus, +) +from .mounts import DEFAULT_DOCKER_SOCKET, mounts_word, tong_mount_specs from .secrets import SECRET_FIFO_TMPFS, declared_run_override @@ -125,6 +131,7 @@ def tong_run_argv( entrypoint=None, command=None, extra_mount_specs=None, + volume_scope=None, ): """Full `docker run -d` argv that starts one tong container. @@ -158,6 +165,8 @@ def tong_run_argv( outside the definition (today the git-dir mounts that ride along with a `workspace` mount -- see swarmforge.anvil); they are appended after the definition's own mounts, mirroring how the Makefile orders the anvil's. + + `volume_scope` names the tong's `volume:` mounts (see `volume_scope`). """ if entrypoint is None and command is None: entrypoint, command = declared_run_override(defn) @@ -176,11 +185,13 @@ def tong_run_argv( if secret_channel: argv += ["--tmpfs", SECRET_FIFO_TMPFS] effective_env = dict(env or {}) - if workspace and _has_socket_mount(defn) and defn.get("lifecycle") == "session": + if workspace and mounts_word(defn, SOCKET_MOUNT) and defn.get("lifecycle") == "session": effective_env.setdefault(WORKSPACE_HOST_ENV, workspace) for key in sorted(effective_env): argv += ["-e", "%s=%s" % (key, effective_env[key])] - for spec in tong_mount_specs(defn, workspace, socket_path=socket_path): + for spec in tong_mount_specs(defn, workspace, socket_path=socket_path, + tong_name=name, volume_scope=volume_scope, + extra_mount_specs=extra_mount_specs or ()): argv += ["-v", spec] for spec in extra_mount_specs or []: argv += ["-v", spec] diff --git a/swarmforge/tongs/cli.py b/swarmforge/tongs/cli.py index 22c7945..e231b32 100644 --- a/swarmforge/tongs/cli.py +++ b/swarmforge/tongs/cli.py @@ -8,7 +8,7 @@ import sys from .discovery import discover, merge_tongs -from .model import LAYERS +from .model import LAYERS, printable from .validate import validate_tong @@ -31,7 +31,7 @@ def main(argv): problems = 0 for name in sorted(merged): for error in validate_tong(name, merged[name]["definition"]): - print(error) + print(printable(error)) problems += 1 if not problems: print("ok: %d tong(s) valid" % len(merged)) diff --git a/swarmforge/tongs/discovery.py b/swarmforge/tongs/discovery.py index f9dd1be..6d27d21 100644 --- a/swarmforge/tongs/discovery.py +++ b/swarmforge/tongs/discovery.py @@ -11,7 +11,7 @@ from swarmforge.yamlite import parse_map -from .model import TRUSTED_LAYERS, WORKSPACE, printable, warn +from .model import TRUSTED_LAYERS, WORKSPACE, warn def load_yaml(text): @@ -48,7 +48,7 @@ def load_tong_dir(path): try: out[name] = load_tong_file(full) except (ValueError, OSError) as exc: - warn(printable("skipping %s: %s" % (full, exc))) + warn("skipping %s: %s" % (full, exc)) return out diff --git a/swarmforge/tongs/model.py b/swarmforge/tongs/model.py index 7cd4f5c..269eff1 100644 --- a/swarmforge/tongs/model.py +++ b/swarmforge/tongs/model.py @@ -5,10 +5,11 @@ dispatches on (layers, lifecycles, interface kinds, readiness modes), the docker labels it stamps, and the pure resolution of a definition's `readiness:` block and `resources.gpus` request. -`warn` lives here too, so the one `tongs: ` stderr prefix has a single home, with -`printable` beside it for echoing definition-sourced text to the terminal. +`warn` lives here too, so the one `tongs: ` stderr prefix has a single home, and +every message passes through `printable` beside it. """ +import os import re import sys @@ -35,8 +36,14 @@ WORKSPACE_HOST_ENV = "SWARMFORGE_WORKSPACE_HOST_PATH" +def workspace_key(workspace): + """The key a checkout's approvals and volumes are stored under: its path, symlinks resolved.""" + return os.path.realpath(workspace) + + def warn(message): - print("tongs: %s" % message, file=sys.stderr) + """Print `message` to stderr through `printable`, since it may quote definition-sourced text.""" + print("tongs: %s" % printable(message), file=sys.stderr) def printable(text): diff --git a/swarmforge/tongs/mounts.py b/swarmforge/tongs/mounts.py index aa6d127..6873a78 100644 --- a/swarmforge/tongs/mounts.py +++ b/swarmforge/tongs/mounts.py @@ -1,29 +1,58 @@ -"""The `mounts:` magic words and the docker bind specs they turn into. +"""The `mounts:` magic words and the docker `-v` specs they turn into. A definition never names a raw host path: it asks for a mount by word, and this module decides where that lands inside the container and what `-v` value docker -is given. Validation and argv assembly both come through here, so a mount is -judged by the same rules whichever asks. +is given: a host path the launcher knows, or a named volume it names. Validation +and argv assembly both come through here, so a mount is judged by the same rules +whichever asks. """ +import hashlib +import json import posixpath +import re -from .model import SOCKET_MOUNT +from swarmforge.names import canonical_path, name_hint + +from .model import ORG, REPO, SOCKET_MOUNT, USER, WORKSPACE, workspace_key from .secrets import SECRET_FIFO_DIR, SECRET_INJECT_SHELL, partition_secret_env WORKSPACE_MOUNT = "workspace" -MOUNT_WORDS = (WORKSPACE_MOUNT, SOCKET_MOUNT) +VOLUME_MOUNT = "volume" +MOUNT_WORDS = (WORKSPACE_MOUNT, SOCKET_MOUNT, VOLUME_MOUNT) +TARGETED_MOUNT_WORDS = (WORKSPACE_MOUNT, VOLUME_MOUNT) DEFAULT_WORKSPACE_MOUNT_TARGET = "/workspace" DEFAULT_DOCKER_SOCKET = "/var/run/docker.sock" # An allowlist, so an unvetted mount option cannot reach the daemon. MOUNT_MODES = ("ro", "rw") +VOLUME_NAME_PREFIX = "swarmforge-volume" + +# 128 bits, so no definition can search its way onto another scope's volume. +VOLUME_DIGEST_LENGTH = 32 + +ORG_VOLUME_SCOPE = "org" +WORKSPACE_VOLUME_SCOPE = "workspace" +LOCAL_VOLUME_SCOPE = "local" + +# No `_`, so the last `_` of a docker volume name splits the volume back off. +VOLUME_NAME_RE = re.compile(r"[A-Za-z0-9][A-Za-z0-9.-]*") + +# With the hint cap, keeps a docker volume name far below a 255-byte directory name. +VOLUME_NAME_MAX_LENGTH = 64 +VOLUME_HINT_LIMIT = 32 -def _has_socket_mount(defn): + +def mounts_word(defn, word): + """True if a tong's `mounts:` request the magic `word`. + + The word may carry a name, a target and/or a mode (e.g. `workspace:/code:ro`), + so compare only the word before the first colon. + """ for mount in defn.get("mounts") or []: - if isinstance(mount, str) and mount.split(":", 1)[0] == SOCKET_MOUNT: + if isinstance(mount, str) and mount.split(":", 1)[0] == word: return True return False @@ -49,19 +78,22 @@ def _targets_overlap(one, other): def parse_mount(mount, words=MOUNT_WORDS): - """Split a `mounts:` entry into `(word, target, mode)`. + """Split a `mounts:` entry into `(word, volume, target, mode)`. The grammar is `[:/target][:mode]`: the magic word, an optional absolute path naming where the container sees the mount, and an optional access mode, which is always last -- so `workspace`, `workspace:ro`, `workspace:/work` and `workspace:/work:ro` are all valid. `target`/`mode` are None when not declared; - the caller supplies the default target for its magic word. + the caller supplies the default target for its magic word. The `volume` word + is spelled `volume::/target[:mode]`, its target required; `volume` is + that name, and None for every other word. The word is checked against `words` before the rest of the entry, since a mount nobody recognizes has no meaningful target or mode; narrow that set to accept fewer words, never widen it to accept a raw host path. Raises `ValueError` for an unaccepted word, a field that is neither an absolute path nor a mode, a mode - that is not last, more than one target, or a target resolving to the root. + that is not last, more than one target, a target resolving to the root, or a + volume mount whose name is malformed or whose target is missing. """ fields = mount.split(":") word, fields = fields[0], fields[1:] @@ -76,6 +108,20 @@ def parse_mount(mount, words=MOUNT_WORDS): "unknown mount %r (expected %s)" % (mount, " or ".join(repr(known) for known in words)) ) + volume = None + if word == VOLUME_MOUNT: + if not fields or not VOLUME_NAME_RE.fullmatch(fields[0]): + raise ValueError( + "mount %r: a volume mount is spelled volume::/target[:mode], " + "with a name of letters, digits, '.' and '-' that starts with a " + "letter or digit" % (mount,) + ) + if len(fields[0]) > VOLUME_NAME_MAX_LENGTH: + raise ValueError( + "mount %r: a volume name is at most %d characters" + % (mount, VOLUME_NAME_MAX_LENGTH) + ) + volume, fields = fields[0], fields[1:] target = None mode = None for index, field in enumerate(fields): @@ -105,7 +151,9 @@ def parse_mount(mount, words=MOUNT_WORDS): "mount %r: %r is neither an absolute target path nor an access " "mode (%s)" % (mount, field, "/".join(MOUNT_MODES)) ) - return word, target, mode + if word == VOLUME_MOUNT and target is None: + raise ValueError("mount %r: a volume mount needs an absolute target path" % (mount,)) + return word, volume, target, mode def reserved_mount_targets(defn, socket_path=DEFAULT_DOCKER_SOCKET): @@ -123,7 +171,7 @@ def reserved_mount_targets(defn, socket_path=DEFAULT_DOCKER_SOCKET): if isinstance(env, dict) and partition_secret_env(env)[1]: paths[SECRET_FIFO_DIR] = "the tmpfs where the launcher delivers this tong's secrets" paths[SECRET_INJECT_SHELL] = "the shell the secret wrapper execs" - if _has_socket_mount(defn): + if mounts_word(defn, SOCKET_MOUNT): paths[socket_path] = "where the docker socket is mounted" return paths @@ -133,21 +181,23 @@ def mount_destination(word, target, socket_path=DEFAULT_DOCKER_SOCKET): The declared target when there is one, otherwise the word's default: /workspace for the workspace, its own host path for the socket (that is where a docker - client looks for it). Raises `ValueError` for any other word, so a magic word - added without a default cannot inherit the socket's. + client looks for it); a volume has none. Raises `ValueError` for a targetless + volume and any other word, so a new word cannot inherit the socket's default. """ if word == WORKSPACE_MOUNT: return normalize_mount_target(target or DEFAULT_WORKSPACE_MOUNT_TARGET) if word == SOCKET_MOUNT: return normalize_mount_target(socket_path) + if word == VOLUME_MOUNT and target is not None: + return normalize_mount_target(target) raise ValueError("mount %r has no destination" % (word,)) def mount_target_error(mount, word, target, destination, reserved): """Why a mount's target is unusable, or None if it is fine. - Two rules beyond the grammar: only `workspace` takes a target, and no mount may - land on one of the `reserved` destinations (`{path: why}`, from + Two rules beyond the grammar: only `workspace` and `volume` take a target, and + no mount may land on one of the `reserved` destinations (`{path: why}`, from `reserved_mount_targets`) -- docker layers overlapping mounts, which either buries the tong's wiring or has docker create a mountpoint inside the user's workspace on the host. `destination` is where the mount actually lands @@ -155,10 +205,10 @@ def mount_target_error(mount, word, target, destination, reserved): Validation and argv assembly both ask this, so both give the same verdict and the same message for one `reserved` map. """ - if word != WORKSPACE_MOUNT: + if word not in TARGETED_MOUNT_WORDS: if target is not None: - return ("mount %r: only the '%s' mount takes a target path" - % (mount, WORKSPACE_MOUNT)) + return ("mount %r: only the %s mounts take a target path" + % (mount, " and ".join(repr(w) for w in TARGETED_MOUNT_WORDS))) # The socket lands on its own reserved path by construction. return None for path, why in sorted(reserved.items()): @@ -167,42 +217,86 @@ def mount_target_error(mount, word, target, destination, reserved): return None -def overlapping_mount_error(mount, destination, placed): +def overlapping_mount_error(mount, destination, volume, placed): """Why a mount collides with one already placed, or None if it is clear. - `placed` is the `(mount, destination)` pairs accepted so far. Docker refuses two - binds onto one destination outright, and creates the inner mountpoint of nested - ones inside the outer bind -- inside the user's workspace, for a workspace bind. + `volume` is the name `parse_mount` read from `mount`, and `placed` the + `(mount, destination, volume)` triples accepted so far. Docker refuses two binds onto one + destination outright, and creates the inner mountpoint of nested ones inside + the outer bind -- inside the user's workspace, for a workspace bind. A volume + mounts only once. """ - for other, other_destination in placed: + for other, other_destination, other_volume in placed: if _targets_overlap(destination, other_destination): return ("mount %r: %s overlaps mount %r at %s" % (mount, destination, other, other_destination)) + if volume is not None and other_volume == volume: + return "mount %r: volume %r is already mounted by %r" % (mount, volume, other) return None -def tong_mount_specs(defn, workspace, socket_path=DEFAULT_DOCKER_SOCKET): +def tong_volume_name(tong_name, volume, scope): + """The docker named volume behind one tong's `volume:` mount. + + `swarmforge-volume-[-]_`. The digest, over `scope` + (from `volume_scope`) and the raw tong name, is the identity; the hint is the + sanitized name, only for reading in `docker volume ls`. + """ + scope_class, scope_path = scope + key = json.dumps([scope_class, scope_path, tong_name]) + digest = hashlib.sha256(key.encode("utf-8")).hexdigest()[:VOLUME_DIGEST_LENGTH] + hint = name_hint(tong_name, VOLUME_HINT_LIMIT) + parts = [VOLUME_NAME_PREFIX] + ([hint] if hint else []) + [digest] + return "%s_%s" % ("-".join(parts), volume) + + +def volume_scope(layer, org_tongs_dir=None, workspace=None): + """The `(class, path)` scope that names the volumes of a tong from `layer`. + + Org: its tongs directory, canonical as `org_scope_token` reads it. + Workspace: the checkout's `workspace_key`, like its approvals. User and + repo: `(LOCAL_VOLUME_SCOPE, None)`. None when the org or workspace directory + is unknown, which `tong_mount_specs` refuses rather than falling back to the + local scope. Raises `ValueError` for an unknown layer. + """ + if layer == ORG: + return (ORG_VOLUME_SCOPE, canonical_path(org_tongs_dir)) if org_tongs_dir else None + if layer == WORKSPACE: + return (WORKSPACE_VOLUME_SCOPE, workspace_key(workspace)) if workspace else None + if layer in (USER, REPO): + return (LOCAL_VOLUME_SCOPE, None) + raise ValueError("unknown layer %r has no volume scope" % (layer,)) + + +def tong_mount_specs(defn, workspace, socket_path=DEFAULT_DOCKER_SOCKET, + tong_name=None, volume_scope=None, extra_mount_specs=()): """Concrete docker `-v` specs for a tong's `mounts:` magic words. Returns the list of `-v` *values* (the orchestrator pairs each with a `-v` flag). `workspace[:/target][:mode]` mounts the session workspace, at /workspace unless the definition names another target; `docker-socket[:mode]` - bind-mounts the host docker socket onto the same path it has on the host. + bind-mounts the host docker socket onto the same path it has on the host; + `volume::/target[:mode]` mounts the named volume + `tong_volume_name(tong_name, name, volume_scope)`, which docker creates on + first use and the launcher never removes. A volume may not overlap one of + `extra_mount_specs` (see `tong_run_argv`), or docker would leave their + mountpoints inside it. Raises `ValueError` for a non-string entry, a malformed mount, an unusable or - colliding destination, or a `workspace` mount when no workspace path is known - -- a definition never names a raw host path, so anything else is a mistake that + colliding destination, a `workspace` mount when no workspace path is known, or + a `volume` mount without a `tong_name` and `volume_scope` to name it by -- a + definition never names a raw host path, so anything else is a mistake that should stop the launch. """ specs = [] reserved = reserved_mount_targets(defn, socket_path) - placed = [] # (mount, destination) already emitted + placed = [] # (mount, destination, volume) already emitted for mount in defn.get("mounts") or []: if not isinstance(mount, str): raise ValueError("mount entries must be strings, got %r" % (mount,)) - word, target, mode = parse_mount(mount) + word, volume, target, mode = parse_mount(mount) destination = mount_destination(word, target, socket_path) reason = (mount_target_error(mount, word, target, destination, reserved) - or overlapping_mount_error(mount, destination, placed)) + or overlapping_mount_error(mount, destination, volume, placed)) if reason: raise ValueError(reason) if word == WORKSPACE_MOUNT: @@ -211,6 +305,18 @@ def tong_mount_specs(defn, workspace, socket_path=DEFAULT_DOCKER_SOCKET): source = workspace elif word == SOCKET_MOUNT: source = socket_path + elif word == VOLUME_MOUNT: + if not tong_name or volume_scope is None: + raise ValueError( + "mount %r needs the tong's name and scope to name its volume" % (mount,)) + source = tong_volume_name(tong_name, volume, volume_scope) + for extra in extra_mount_specs: + # Ride-along sources never contain a colon; gitguard refuses those. + extra_destination = extra.split(":")[1] + if _targets_overlap(destination, extra_destination): + raise ValueError( + "mount %r: %s overlaps %s, which is mounted alongside the " + "workspace" % (mount, destination, extra_destination)) else: # Unreachable via `parse_mount`; a new word must not inherit the socket bind. raise ValueError("mount %r has no docker spec" % (mount,)) @@ -218,7 +324,7 @@ def tong_mount_specs(defn, workspace, socket_path=DEFAULT_DOCKER_SOCKET): if mode: spec += ":" + mode specs.append(spec) - placed.append((mount, destination)) + placed.append((mount, destination, volume)) return specs @@ -237,7 +343,7 @@ def workspace_mount_placements(defn, socket_path=DEFAULT_DOCKER_SOCKET): for mount in defn.get("mounts") or []: if not isinstance(mount, str): raise ValueError("mount entries must be strings, got %r" % (mount,)) - word, target, mode = parse_mount(mount) + word, _, target, mode = parse_mount(mount) if word == WORKSPACE_MOUNT: placements.append((mount_destination(word, target, socket_path), mode)) return placements diff --git a/swarmforge/tongs/validate.py b/swarmforge/tongs/validate.py index 02dfe99..4107d75 100644 --- a/swarmforge/tongs/validate.py +++ b/swarmforge/tongs/validate.py @@ -150,24 +150,24 @@ def err(msg): mounts = defn.get("mounts") if isinstance(mounts, list): reserved = reserved_mount_targets(defn) - placed = [] # (mount, destination) already accounted for + placed = [] # (mount, destination, volume) already accounted for for mount in mounts: if not isinstance(mount, str): err("mount entries must be strings, got %r" % (mount,)) continue try: - word, target, _ = parse_mount(mount) + word, volume, target, _ = parse_mount(mount) destination = mount_destination(word, target) except ValueError as exc: err(str(exc)) continue reason = mount_target_error(mount, word, target, destination, reserved) if reason is None: - reason = overlapping_mount_error(mount, destination, placed) + reason = overlapping_mount_error(mount, destination, volume, placed) if reason: err(reason) continue - placed.append((mount, destination)) + placed.append((mount, destination, volume)) networks = defn.get("networks") if isinstance(networks, list): diff --git a/tests/test_anvil_approval.py b/tests/test_anvil_approval.py index ba72be9..d3551df 100644 --- a/tests/test_anvil_approval.py +++ b/tests/test_anvil_approval.py @@ -58,10 +58,25 @@ def test_non_mapping_resources_renders_no_gpus_line(self): text = launcher.render_privilege_summary("github", tongs.privilege_summary(defn)) self.assertNotIn("gpus", text) + def test_calls_out_a_volume_as_persistent(self): + defn = dict(WORKSPACE_TONG, mounts=["volume:models:/models"]) + scope = (tongs.WORKSPACE_VOLUME_SCOPE, "/home/me/proj") + text = launcher.render_privilege_summary( + "cache", tongs.privilege_summary(defn, "cache", scope)) + self.assertIn("volume:models:/models", text) + lines = text.splitlines() + self.assertIn(" volume: models at /models (docker volume %s)" + % tongs.tong_volume_name("cache", "models", scope), lines) + self.assertIn( + " persistent: outlives the tong, is mounted by each of its concurrent " + "sessions, and reuses any data left by an earlier definition or checkout at " + "this path", lines) + def test_omits_unrequested_sections(self): defn = {"image": "x", "interface": {"kind": "none"}, "resources": {"memory": "1g"}} text = launcher.render_privilege_summary("x", tongs.privilege_summary(defn)) self.assertNotIn("secrets:", text) + self.assertNotIn("volume", text) self.assertNotIn("docker socket", text) self.assertNotIn("gpus", text) @@ -189,6 +204,12 @@ def test_accepted_workspace_tong_records_and_persists(self): stored = tongs.load_approvals(self.approvals) self.assertTrue(tongs.is_approved(stored, self.ws, "gh", WORKSPACE_TONG)) + def test_prompt_names_the_volume_this_checkout_would_mount(self): + merged = _merged("cache", dict(WORKSPACE_TONG, mounts=["volume:models:/models"])) + out = self._gate(merged, answer="y\n") + scope = tongs.volume_scope(tongs.WORKSPACE, workspace=self.ws) + self.assertIn(tongs.tong_volume_name("cache", "models", scope), out) + def test_declined_workspace_tong_raises_and_does_not_persist(self): merged = _merged("gh", WORKSPACE_TONG) with self.assertRaises(launcher.ApprovalDenied): diff --git a/tests/test_anvil_orchestrate.py b/tests/test_anvil_orchestrate.py index 4ace90a..22eebc1 100644 --- a/tests/test_anvil_orchestrate.py +++ b/tests/test_anvil_orchestrate.py @@ -106,6 +106,30 @@ def test_shared_workspace_refused_with_a_custom_mount_target(self): }) )) + def test_workspace_shared_tong_with_a_volume_refused(self): + # Its container is shared by every checkout while its volume belongs to one. + defn = { + "lifecycle": "shared", "image": "x", "mounts": ["volume:v:/data"], + "interface": {"kind": "none"}, "readiness": {"mode": "none"}, + } + reasons = launcher.unsupported_tong_reasons(_merged("t", defn)) + self.assertTrue( + any("workspace 'shared' tong that mounts a volume" in r for r in reasons), reasons + ) + + def test_volume_allowed_where_container_and_volume_scope_agree(self): + shared = { + "lifecycle": "shared", "image": "x", "mounts": ["volume:v:/data"], + "interface": {"kind": "none"}, "readiness": {"mode": "none"}, + } + for source in (tongs.USER, tongs.ORG, tongs.REPO): + self.assertEqual( + launcher.unsupported_tong_reasons(_merged("t", shared, source=source)), + [], source, + ) + session = dict(shared, lifecycle="session") + self.assertEqual(launcher.unsupported_tong_reasons(_merged("t", session)), []) + def test_workspace_refusal_is_shared_scoped(self): # A `session` tong is torn down with the anvil, so it cannot leak. self.assertEqual( @@ -921,6 +945,102 @@ def test_org_shared_tong_without_anvil_name_raises_before_any_docker_call(self): self.assertEqual(docker.calls, []) +SESSION_VOLUME = { + "lifecycle": "session", + "image": "model-cache", + "mounts": ["volume:models:/models"], + "interface": {"kind": "none"}, + "readiness": {"mode": "none"}, +} + + +def _mounted(argv): + return [argv[i + 1] for i, part in enumerate(argv) if part == "-v"] + + +class VolumeScopeTests(unittest.TestCase): + """A tong's named volumes are scoped by the layer it came from, and kept.""" + + _ACME = "/orgs/acme/.swarmforge/tongs" + + def _run(self, docker, merged, workspace=None, org_dir=None, anvil=None): + layer_dirs = [(tongs.ORG, org_dir)] if org_dir else [] + opts = launcher.LauncherOptions( + layer_dirs=layer_dirs, workspace=workspace, approvals=None, + providers=None, harness="opencode", anvil_image="anvil:img", no_prompt=False, + ) + return launcher.run_with_tongs( + merged, anvil or ANVIL_ARGV, opts, + docker=docker, sleep=lambda _s: None, monotonic=_Clock(), + ) + + def test_user_and_repo_tong_volumes_share_the_local_scope(self): + local = [tongs.tong_volume_name("cache", "models", (tongs.LOCAL_VOLUME_SCOPE, None)) + + ":/models"] + for source in (tongs.USER, tongs.REPO): + docker = FakeDocker() + self._run(docker, _merged("cache", SESSION_VOLUME, source=source), + workspace="/home/me/proj") + self.assertEqual(_mounted(docker.run_argvs[0]), local, source) + + def test_org_tong_volume_scoped_by_its_org_on_either_lifecycle(self): + org = [tongs.tong_volume_name("cache", "models", (tongs.ORG_VOLUME_SCOPE, self._ACME)) + + ":/models"] + shared = dict(SESSION_VOLUME, lifecycle="shared") + for defn in (SESSION_VOLUME, shared): + docker = FakeDocker() + self._run(docker, _merged("cache", defn, source=tongs.ORG), org_dir=self._ACME) + self.assertEqual(_mounted(docker.run_argvs[0]), org, defn["lifecycle"]) + + def test_workspace_tong_volume_scoped_by_its_checkout(self): + mounted = [] + for workspace in ("/home/me/proj", "/home/me/fork"): + docker = FakeDocker() + self._run(docker, _merged("cache", SESSION_VOLUME), workspace=workspace) + mounted.append(_mounted(docker.run_argvs[0])) + scope = (tongs.WORKSPACE_VOLUME_SCOPE, os.path.realpath("/home/me/proj")) + self.assertEqual(mounted[0], [tongs.tong_volume_name("cache", "models", scope) + + ":/models"]) + self.assertNotEqual(mounted[0], mounted[1]) + + def test_volume_without_a_directory_to_scope_it_starts_no_tong(self): + # Never the local scope: that would share the volume across orgs or checkouts. + for source in (tongs.WORKSPACE, tongs.ORG): + docker = FakeDocker() + with self.assertRaisesRegex(launcher.OrchestrationError, + "tong 'cache'.*name and scope", msg=source): + self._run(docker, _merged("cache", SESSION_VOLUME, source=source)) + self.assertEqual(docker.run_argvs, [], source) + + def test_workspace_tong_without_a_volume_needs_no_workspace_path(self): + docker = FakeDocker() + self.assertEqual(self._run(docker, _merged("pg", SESSION_PORT)), 0) + + def test_recreated_shared_tong_mounts_the_same_volume(self): + # A changed definition recreates the container, never the volume. + defn = dict(SESSION_VOLUME, lifecycle="shared") + states = {"swarmforge-shared-cache": {"running": True, "label": "stale"}} + docker = FakeDocker(states=states) + self._run(docker, _merged("cache", defn, source=tongs.REPO)) + self.assertIn(("rm_force", "swarmforge-shared-cache"), docker.calls) + self.assertEqual( + _mounted(docker.run_argvs[0]), + [tongs.tong_volume_name("cache", "models", (tongs.LOCAL_VOLUME_SCOPE, None)) + + ":/models"], + ) + + def test_teardown_only_force_removes_the_container_and_network(self): + # No volume step: a named volume outlives `docker rm` of its container. + docker = FakeDocker() + self._run(docker, _merged("cache", SESSION_VOLUME, source=tongs.REPO)) + teardown = docker.calls[[c[0] for c in docker.calls].index("run_foreground_multi") + 1:] + self.assertEqual(teardown, [ + ("rm_force", "claude-myproject-tong-cache"), + ("rm_force", "claude-myproject"), + ("network_rm", tongs.session_network_name("claude-myproject")), + ]) + + class WorkspaceGitDirSpecTests(unittest.TestCase): """_workspace_git_dir_specs: the git-dir mounts riding along with `workspace`.""" diff --git a/tests/test_names.py b/tests/test_names.py index 79c820a..50c65d2 100644 --- a/tests/test_names.py +++ b/tests/test_names.py @@ -82,6 +82,16 @@ def test_directories_sharing_a_basename_digest_differently(self): names.path_digest("/home/me/repo2/master")) +class NameHint(unittest.TestCase): + """The readable half of a token, from any name.""" + + def test_a_name_is_reduced_to_what_docker_accepts(self): + self.assertEqual(names.name_hint("my tong!"), "my-tong") + + def test_a_cut_never_leaves_the_hint_on_a_separator(self): + self.assertEqual(names.name_hint("abc-defg", limit=4), "abc") + + class DirHint(unittest.TestCase): """The readable half of a token.""" diff --git a/tests/test_tongs_approvals.py b/tests/test_tongs_approvals.py index c109c89..3d310c8 100644 --- a/tests/test_tongs_approvals.py +++ b/tests/test_tongs_approvals.py @@ -73,6 +73,31 @@ def test_summary_reports_an_unvalidated_gpus_value_as_declared(self): defn["resources"] = {"gpus": 0} self.assertEqual(tongs.privilege_summary(defn)["gpus"], 0) + def test_summary_names_each_declared_volume(self): + defn = def_of(GITHUB_TONG) + defn["mounts"] = ["workspace:ro", "volume:models:/a", "volume:cache:/c:ro"] + self.assertEqual(tongs.privilege_summary(defn)["volumes"], [ + {"name": "models", "target": "/a", "docker": None}, + {"name": "cache", "target": "/c", "docker": None}, + ]) + self.assertEqual(tongs.privilege_summary(def_of(GITHUB_TONG))["volumes"], []) + + def test_summary_names_the_docker_volume_given_its_scope(self): + defn = dict(def_of(GITHUB_TONG), mounts=["volume:models:/a"]) + scope = (tongs.WORKSPACE_VOLUME_SCOPE, "/home/me/proj") + self.assertEqual( + tongs.privilege_summary(defn, "cache", scope)["volumes"][0]["docker"], + tongs.tong_volume_name("cache", "models", scope), + ) + + def test_summary_tolerates_malformed_mounts(self): + # The gate runs before validation, which refuses these on its own. + defn = def_of(GITHUB_TONG) + defn["mounts"] = ["volume:bad_name:/a", "volume:v", 7, "/etc:/etc", "volume:ok:/b"] + summary = tongs.privilege_summary(defn) + self.assertEqual([v["name"] for v in summary["volumes"]], ["ok"]) + self.assertEqual(summary["mounts"], defn["mounts"]) + class ApprovalKeyingTests(unittest.TestCase): def setUp(self): @@ -104,6 +129,17 @@ def test_keyed_by_workspace_path(self): tongs.record_approval(approvals, self.ws, "github", self.defn) self.assertFalse(tongs.is_approved(approvals, "/other/ws", "github", self.defn)) + def test_a_symlinked_checkout_shares_its_approvals_and_volumes(self): + with tempfile.TemporaryDirectory() as tmp: + checkout = os.path.join(tmp, "checkout") + os.mkdir(checkout) + link = os.path.join(tmp, "link") + os.symlink(checkout, link) + approvals = tongs.record_approval({}, link, "github", self.defn) + self.assertTrue(tongs.is_approved(approvals, checkout, "github", self.defn)) + self.assertEqual(list(approvals), + [tongs.volume_scope(tongs.WORKSPACE, workspace=link)[1]]) + def test_load_save_roundtrip(self): with tempfile.TemporaryDirectory() as tmp: path = os.path.join(tmp, "nested", "approvals.json") diff --git a/tests/test_tongs_argv.py b/tests/test_tongs_argv.py index 140d503..d9aa42e 100644 --- a/tests/test_tongs_argv.py +++ b/tests/test_tongs_argv.py @@ -193,6 +193,21 @@ def argv_for(mounts): [part for part in default if part != "/ws:/workspace"], ) + def test_run_argv_names_a_volume_after_the_tong_and_its_scope(self): + defn = def_of(NONE_TONG) + defn["mounts"] = ["volume:models:/root/.ollama"] + for scope in ((tongs.LOCAL_VOLUME_SCOPE, None), + (tongs.ORG_VOLUME_SCOPE, "/orgs/acme/.swarmforge/tongs")): + argv = tongs.tong_run_argv( + "ollama", defn, container_name="c", network="n", alias="ollama", + volume_scope=scope, + ) + self.assertEqual( + [argv[i + 1] for i, part in enumerate(argv) if part == "-v"], + [tongs.tong_volume_name("ollama", "models", scope) + ":/root/.ollama"], + scope, + ) + def test_run_argv_extra_mount_specs_follow_definition_mounts(self): defn = def_of(NONE_TONG) defn["mounts"] = ["workspace"] @@ -206,6 +221,16 @@ def test_run_argv_extra_mount_specs_follow_definition_mounts(self): ["/ws:/workspace", "/ws/.git/config:/workspace/.git/config:ro"], ) + def test_run_argv_refuses_a_volume_over_an_extra_mount(self): + defn = def_of(NONE_TONG) + defn["mounts"] = ["workspace", "volume:v:/home"] + with self.assertRaisesRegex(ValueError, "mounted alongside the workspace"): + tongs.tong_run_argv( + "w", defn, container_name="c", network="n", alias="w", workspace="/ws", + extra_mount_specs=["/home/u/repo/.git:/home/u/repo/.git"], + volume_scope=(tongs.LOCAL_VOLUME_SCOPE, None), + ) + def test_run_argv_without_extra_mount_specs_is_unchanged(self): defn = def_of(NONE_TONG) defn["mounts"] = ["workspace"] diff --git a/tests/test_tongs_model.py b/tests/test_tongs_model.py index fe73ca8..2dbdfc1 100644 --- a/tests/test_tongs_model.py +++ b/tests/test_tongs_model.py @@ -1,6 +1,8 @@ #!/usr/bin/env python3 """Unit tests for swarmforge.tongs.model. Run: python3 tests/test_tongs_model.py""" +import contextlib +import io import os import sys import unittest @@ -36,6 +38,12 @@ def test_output_is_printable_for_every_code_point(self): every = "".join(chr(cp) for cp in range(0x110000)) self.assertTrue(tongs.printable(every).isprintable()) + def test_warn_escapes_its_message(self): + err = io.StringIO() + with contextlib.redirect_stderr(err): + tongs.warn("tong 'x\x1b[2K\napproved' is refused") + self.assertEqual(err.getvalue(), "tongs: tong 'x\\x1b[2K\\napproved' is refused\n") + class ReadinessTests(unittest.TestCase): def test_parse_duration_units(self): diff --git a/tests/test_tongs_mounts.py b/tests/test_tongs_mounts.py index fc5f631..1ca93fc 100644 --- a/tests/test_tongs_mounts.py +++ b/tests/test_tongs_mounts.py @@ -3,6 +3,7 @@ import os import sys +import tempfile import unittest REPO_ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) @@ -14,20 +15,63 @@ from swarmforge import tongs +LOCAL = (tongs.LOCAL_VOLUME_SCOPE, None) + + class MountGrammarTests(unittest.TestCase): """The mount grammar and target policy, exercised directly -- `validate_tong` and `tong_mount_specs` both delegate here.""" def test_parse_mount_splits_the_optional_fields(self): - self.assertEqual(tongs.parse_mount("workspace"), ("workspace", None, None)) - self.assertEqual(tongs.parse_mount("workspace:ro"), ("workspace", None, "ro")) - self.assertEqual(tongs.parse_mount("workspace:/code"), ("workspace", "/code", None)) + self.assertEqual(tongs.parse_mount("workspace"), ("workspace", None, None, None)) + self.assertEqual(tongs.parse_mount("workspace:ro"), ("workspace", None, None, "ro")) + self.assertEqual( + tongs.parse_mount("workspace:/code"), ("workspace", None, "/code", None) + ) + self.assertEqual( + tongs.parse_mount("workspace:/code:rw"), ("workspace", None, "/code", "rw") + ) self.assertEqual( - tongs.parse_mount("workspace:/code:rw"), ("workspace", "/code", "rw") + tongs.parse_mount("docker-socket:ro"), ("docker-socket", None, None, "ro") ) + + def test_parse_mount_reads_a_volume_name(self): self.assertEqual( - tongs.parse_mount("docker-socket:ro"), ("docker-socket", None, "ro") + tongs.parse_mount("volume:models:/data"), ("volume", "models", "/data", None) ) + self.assertEqual( + tongs.parse_mount("volume:v1.2-x:/data:ro"), ("volume", "v1.2-x", "/data", "ro") + ) + + def test_parse_mount_rejects_a_malformed_volume_name(self): + # `_` is what splits the volume back off its docker name, so it cannot be in one. + for mount in ("volume", "volume::/data", "volume:/data", "volume:my_models:/data", + "volume:-models:/data", "volume:.models:/data", "volume:mo dels:/data", + "volume:models\n:/data"): + with self.assertRaisesRegex(ValueError, "volume::/target", msg=mount): + tongs.parse_mount(mount) + + def test_parse_mount_caps_a_volume_name(self): + longest = "v" * tongs.VOLUME_NAME_MAX_LENGTH + self.assertEqual(tongs.parse_mount("volume:%s:/data" % longest)[1], longest) + with self.assertRaisesRegex(ValueError, "at most 64 characters"): + tongs.parse_mount("volume:%s:/data" % (longest + "v")) + + def test_parse_mount_requires_a_volume_target(self): + for mount in ("volume:models", "volume:models:ro"): + with self.assertRaisesRegex(ValueError, "needs an absolute target path", msg=mount): + tongs.parse_mount(mount) + + def test_parse_mount_judges_a_volume_target_like_a_workspace_one(self): + for mount, message in ( + ("volume:models:data", "neither an absolute target path"), + ("volume:models:/", "not a usable target path"), + ("volume:models:/a:/b", "more than one target path"), + ("volume:models:ro:/data", "must be the last field"), + ("volume:models:/da ta", "whitespace"), + ): + with self.assertRaisesRegex(ValueError, message, msg=mount): + tongs.parse_mount(mount) def test_parse_mount_rejects_a_raw_host_path(self): with self.assertRaisesRegex(ValueError, "unknown mount"): @@ -37,11 +81,19 @@ def test_parse_mount_word_set_can_be_narrowed(self): # A known word refused by a narrowed set is a policy refusal, not a misspelling. narrowed = (tongs.WORKSPACE_MOUNT,) self.assertEqual( - tongs.parse_mount("workspace:/code", words=narrowed), ("workspace", "/code", None) + tongs.parse_mount("workspace:/code", words=narrowed), + ("workspace", None, "/code", None), ) with self.assertRaisesRegex(ValueError, "not allowed here"): tongs.parse_mount("docker-socket", words=narrowed) + def test_mounts_word_compares_only_the_word(self): + defn = {"mounts": ["volume:workspace:/data", "docker-socket:ro", 7]} + self.assertTrue(tongs.mounts_word(defn, tongs.VOLUME_MOUNT)) + self.assertTrue(tongs.mounts_word(defn, "docker-socket")) + self.assertFalse(tongs.mounts_word(defn, tongs.WORKSPACE_MOUNT)) + self.assertFalse(tongs.mounts_word({}, tongs.WORKSPACE_MOUNT)) + def test_mount_destination_refuses_a_word_with_no_default(self): # A word without its own default must not inherit the socket's destination. with self.assertRaisesRegex(ValueError, "no destination"): @@ -77,6 +129,92 @@ def test_mount_destination_defaults_per_word(self): self.assertEqual( tongs.mount_destination("docker-socket", None, "/run/d.sock"), "/run/d.sock" ) + self.assertEqual(tongs.mount_destination("volume", "//data/"), "/data") + + def test_mount_destination_refuses_a_volume_without_a_target(self): + with self.assertRaisesRegex(ValueError, "no destination"): + tongs.mount_destination("volume", None) + + def test_mount_target_error_holds_a_volume_to_the_reserved_paths(self): + reserved = {"/run/x": "where the launcher delivers this tong's secrets"} + self.assertIsNone(tongs.mount_target_error( + "volume:v:/data", "volume", "/data", "/data", reserved)) + self.assertIn("overlaps /run/x", tongs.mount_target_error( + "volume:v:/run", "volume", "/run", "/run", reserved)) + + def test_tong_volume_name_is_a_hint_then_a_fixed_width_digest(self): + name = tongs.tong_volume_name("my tong_x", "models", LOCAL) + self.assertRegex(name, r"\Aswarmforge-volume-my-tong_x-[0-9a-f]{32}_models\Z") + self.assertEqual(tongs.VOLUME_DIGEST_LENGTH, 32) + self.assertEqual(name.rsplit("_", 1)[1], "models") + self.assertEqual(name, tongs.tong_volume_name("my tong_x", "models", LOCAL)) + # A name that sanitizes to nothing keeps the digest alone. + self.assertRegex( + tongs.tong_volume_name("__", "v", LOCAL), r"\Aswarmforge-volume-[0-9a-f]{32}_v\Z" + ) + + def test_tong_volume_name_is_pinned(self): + # Literal on purpose: any change to the formula orphans every existing volume. + cases = ( + ("ollama", LOCAL, + "swarmforge-volume-ollama-fbdbeb6d19743ef7c7724f97755c49d1_models"), + ("ollama", (tongs.ORG_VOLUME_SCOPE, "/orgs/acme/.swarmforge/tongs"), + "swarmforge-volume-ollama-a0afff5cc924fc93f5f25e86a52f5888_models"), + ("ollama", (tongs.WORKSPACE_VOLUME_SCOPE, "/home/me/proj"), + "swarmforge-volume-ollama-9d5322a3cc671ff97654fc52727d1dd3_models"), + ("caf\u00e9", LOCAL, + "swarmforge-volume-caf-a3195456dbda485c7f802622dae166e0_models"), + # The hint is cut to 32 and loses the separator it would end on. + ("a" * 31 + "-" + "b" * 10, LOCAL, + "swarmforge-volume-" + "a" * 31 + "-c9719a32926cd73d826a8ed165eb2523_models"), + ) + for tong_name, scope, expected in cases: + self.assertEqual(tongs.tong_volume_name(tong_name, "models", scope), expected) + + def test_tong_volume_name_digests_the_raw_name_not_the_hint(self): + # Both sanitize to `github`; the hint is for reading, never for identity. + dotted = tongs.tong_volume_name("github.", "v", LOCAL) + plain = tongs.tong_volume_name("github", "v", LOCAL) + self.assertTrue(dotted.startswith("swarmforge-volume-github-")) + self.assertTrue(plain.startswith("swarmforge-volume-github-")) + self.assertNotEqual(dotted, plain) + + def test_tong_volume_name_differs_per_scope(self): + names = { + tongs.tong_volume_name("cache", "v", scope) + for scope in (LOCAL, (tongs.ORG_VOLUME_SCOPE, "/orgs/acme/.swarmforge/tongs"), + (tongs.ORG_VOLUME_SCOPE, "/orgs/globex/.swarmforge/tongs"), + (tongs.WORKSPACE_VOLUME_SCOPE, "/orgs/acme/.swarmforge/tongs"), + (tongs.WORKSPACE_VOLUME_SCOPE, "/home/me/proj")) + } + self.assertEqual(len(names), 5) + + def test_volume_scope_follows_the_layer(self): + self.assertEqual(tongs.volume_scope(tongs.USER), LOCAL) + self.assertEqual(tongs.volume_scope(tongs.REPO, "/o/tongs", "/ws"), LOCAL) + self.assertEqual( + tongs.volume_scope(tongs.ORG, org_tongs_dir="/orgs//acme/./tongs/"), + (tongs.ORG_VOLUME_SCOPE, "/orgs/acme/tongs"), + ) + self.assertIsNone(tongs.volume_scope(tongs.ORG, workspace="/ws")) + self.assertIsNone(tongs.volume_scope(tongs.WORKSPACE, org_tongs_dir="/o/tongs")) + with self.assertRaisesRegex(ValueError, "unknown layer"): + tongs.volume_scope("elsewhere") + + def test_volume_scope_resolves_a_symlinked_checkout(self): + with tempfile.TemporaryDirectory() as tmp: + checkout = os.path.join(tmp, "checkout") + os.mkdir(checkout) + link = os.path.join(tmp, "link") + os.symlink(checkout, link) + self.assertEqual( + tongs.volume_scope(tongs.WORKSPACE, workspace=link), + tongs.volume_scope(tongs.WORKSPACE, workspace=checkout), + ) + self.assertEqual( + tongs.volume_scope(tongs.WORKSPACE, workspace=link)[1], + os.path.realpath(checkout), + ) def test_mount_target_error_names_the_mount_and_the_reason(self): reserved = {"/run/x": "where the launcher delivers this tong's secrets"} @@ -90,7 +228,7 @@ def error(mount, word, target): self.assertIsNone(error("workspace:/code", "workspace", "/code")) self.assertIsNone(error("docker-socket", "docker-socket", None)) self.assertIn( - "only the 'workspace' mount takes a target path", + "only the 'workspace' and 'volume' mounts take a target path", error("docker-socket:/s", "docker-socket", "/s"), ) # Overlap in both directions: the target above the reserved path and under it. @@ -106,14 +244,24 @@ def test_mount_target_error_judges_the_default_destination_too(self): ) def test_overlapping_mount_error_catches_duplicates_and_nesting(self): - placed = [("workspace:/code", "/code")] - self.assertIsNone(tongs.overlapping_mount_error("workspace:/src", "/src", placed)) + placed = [("workspace:/code", "/code", None)] + self.assertIsNone( + tongs.overlapping_mount_error("workspace:/src", "/src", None, placed)) for destination in ("/code", "/code/sub", "/"): self.assertIn( "overlaps mount 'workspace:/code'", - tongs.overlapping_mount_error("m", destination, placed), + tongs.overlapping_mount_error("workspace", destination, None, placed), ) + def test_overlapping_mount_error_refuses_one_volume_mounted_twice(self): + placed = [("volume:models:/a", "/a", "models")] + self.assertIsNone( + tongs.overlapping_mount_error("volume:cache:/b", "/b", "cache", placed)) + self.assertIn( + "volume 'models' is already mounted by 'volume:models:/a'", + tongs.overlapping_mount_error("volume:models:/b:ro", "/b", "models", placed), + ) + class MountSpecTests(unittest.TestCase): """The mount specs a tong contributes to its own `docker run` argv, and the @@ -138,7 +286,7 @@ def test_mount_specs_workspace_custom_target(self): def test_mount_specs_socket_target_raises(self): with self.assertRaises(ValueError) as caught: tongs.tong_mount_specs({"mounts": ["docker-socket:/run/d.sock"]}, "/ws") - self.assertIn("only the 'workspace' mount takes a target path", str(caught.exception)) + self.assertIn("only the 'workspace' and 'volume' mounts take a target path", str(caught.exception)) def test_mount_specs_normalize_the_target(self): # docker cleans a bind destination, so it is handed the spelling the overlap checks judged. @@ -163,6 +311,48 @@ def test_mount_specs_socket_honors_custom_path(self): specs = tongs.tong_mount_specs({"mounts": ["docker-socket:ro"]}, "/ws", socket_path="/run/d.sock") self.assertEqual(specs, ["/run/d.sock:/run/d.sock:ro"]) + def test_mount_specs_volume_names_a_docker_volume(self): + defn = {"mounts": ["volume:models:/root/.ollama", "volume:cache:/cache:ro"]} + scope = (tongs.ORG_VOLUME_SCOPE, "/orgs/acme/.swarmforge/tongs") + self.assertEqual( + tongs.tong_mount_specs(defn, None, tong_name="ollama", volume_scope=scope), + [tongs.tong_volume_name("ollama", "models", scope) + ":/root/.ollama", + tongs.tong_volume_name("ollama", "cache", scope) + ":/cache:ro"], + ) + + def test_mount_specs_volume_target_normalized(self): + self.assertEqual( + tongs.tong_mount_specs( + {"mounts": ["volume:v://data/"]}, None, tong_name="t", volume_scope=LOCAL), + [tongs.tong_volume_name("t", "v", LOCAL) + ":/data"], + ) + + def test_mount_specs_volume_without_a_tong_name_or_scope_raises(self): + # Either would let tongs that must stay apart land on one volume. + for kwargs in ({"volume_scope": LOCAL}, {"tong_name": "t"}): + with self.assertRaisesRegex(ValueError, "name and scope", msg=kwargs): + tongs.tong_mount_specs({"mounts": ["volume:v:/data"]}, None, **kwargs) + + def test_mount_specs_volume_overlapping_another_mount_raises(self): + for mounts in (["workspace", "volume:v:/workspace/cache"], + ["volume:a:/data", "volume:b:/data"]): + with self.assertRaisesRegex(ValueError, "overlaps", msg=mounts): + tongs.tong_mount_specs( + {"mounts": mounts}, "/ws", tong_name="t", volume_scope=LOCAL) + + def test_mount_specs_volume_overlapping_a_ride_along_mount_raises(self): + git_dir = "/home/u/repo/.git:/home/u/repo/.git" + defn = {"mounts": ["workspace", "volume:v:/home"]} + with self.assertRaisesRegex(ValueError, "/home overlaps /home/u/repo/.git"): + tongs.tong_mount_specs(defn, "/ws", tong_name="t", volume_scope=LOCAL, + extra_mount_specs=[git_dir]) + self.assertEqual( + len(tongs.tong_mount_specs( + {"mounts": ["workspace", "volume:v:/data"]}, "/ws", tong_name="t", + volume_scope=LOCAL, extra_mount_specs=[git_dir])), + 2, + ) + def test_mount_specs_no_mounts_is_empty(self): self.assertEqual(tongs.tong_mount_specs({}, "/ws"), []) @@ -171,6 +361,9 @@ def test_workspace_mount_placements_empty_without_workspace_mount(self): self.assertEqual( tongs.workspace_mount_placements({"mounts": ["docker-socket"]}), [] ) + self.assertEqual( + tongs.workspace_mount_placements({"mounts": ["volume:v:/data"]}), [] + ) def test_workspace_mount_placements_default_target_and_mode(self): self.assertEqual( diff --git a/tests/test_tongs_validate.py b/tests/test_tongs_validate.py index b3ed75c..1b3dc67 100644 --- a/tests/test_tongs_validate.py +++ b/tests/test_tongs_validate.py @@ -179,7 +179,40 @@ def test_two_target_paths_rejected(self): def test_socket_mount_target_rejected(self): # A docker client looks for the socket at its host path, so it cannot move. errors = tongs.validate_tong("t", self._base(mounts=["docker-socket:/run/d.sock"])) - self.assertTrue(any("only the 'workspace' mount takes a target path" in e for e in errors)) + self.assertTrue(any( + "only the 'workspace' and 'volume' mounts take a target path" in e for e in errors + )) + + def test_volume_mount_accepted(self): + for mount in ("volume:models:/root/.ollama", "volume:models:/data:ro"): + self.assertEqual(tongs.validate_tong("t", self._base(mounts=[mount])), [], mount) + + def test_malformed_volume_mount_rejected(self): + for mount, message in ( + ("volume:my_models:/data", "volume::/target"), + ("volume:/data", "volume::/target"), + ("volume:models", "needs an absolute target path"), + ("volume:models:/", "not a usable target path"), + ): + errors = tongs.validate_tong("t", self._base(mounts=[mount])) + self.assertTrue(any(message in e for e in errors), (mount, errors)) + + def test_volume_target_overlapping_the_tongs_wiring_or_mounts_rejected(self): + secret = self._base(mounts=["volume:v:/run"]) + secret["env"] = {"TOKEN": "${secret:op:op://Work/t}"} + for defn in ( + secret, + self._base(mounts=["docker-socket", "volume:v:/var/run"]), + self._base(mounts=["workspace", "volume:v:/workspace/.cache"]), + self._base(mounts=["volume:a:/data", "volume:b:/data/sub"]), + ): + errors = tongs.validate_tong("t", defn) + self.assertTrue(any("overlaps" in e for e in errors), defn["mounts"]) + + def test_one_volume_mounted_twice_rejected(self): + errors = tongs.validate_tong( + "t", self._base(mounts=["volume:models:/a", "volume:models:/b:ro"])) + self.assertTrue(any("already mounted" in e for e in errors), errors) def test_extra_aliases_accepted_on_network_facing_kinds(self): # A client matching a certificate CN dials the dotted name, not the canonical alias.