Tear down session tongs on SIGHUP and SIGTERM

docs/development/testing.md

@@ -4,7 +4,7 @@ The full variable list for each target is in [Environment variables](../referenc
 
 ## Unit Tests
 
-The launcher, the tongs layer, and the container-side translators are covered by stdlib `unittest` tests in `tests/test_*.py`. A test module is named for the source module it covers — `tests/test_tongs_<module>.py` for `swarmforge/tongs/<module>.py`, `tests/test_anvil_<module>.py` for `swarmforge/anvil/<module>.py` — so the file that covers a change is the one named after it. Two modules have no namesake file because they have nothing to assert on their own: `swarmforge/anvil/readiness.py` is exercised through `run_with_tongs`, and `swarmforge/anvil/errors.py` holds one exception class. Fixtures that more than one test module needs live in `tests/tongs_fixtures.py`, `tests/anvil_fixtures.py` and `tests/harness_fixtures.py`, which the discovery glob skips.
+The launcher, the tongs layer, and the container-side translators are covered by stdlib `unittest` tests in `tests/test_*.py`. A test module is named for the source module it covers — `tests/test_tongs_<module>.py` for `swarmforge/tongs/<module>.py`, `tests/test_anvil_<module>.py` for `swarmforge/anvil/<module>.py` — so the file that covers a change is the one named after it. Two modules have no namesake file because they have nothing to assert on their own: `swarmforge/anvil/readiness.py` is exercised through `run_with_tongs`, and `swarmforge/anvil/errors.py` holds the launcher's shared exception classes. Fixtures that more than one test module needs live in `tests/tongs_fixtures.py`, `tests/anvil_fixtures.py` and `tests/harness_fixtures.py`, which the discovery glob skips.
 
 Four files assert on the shape of the repo rather than on any one module. `tests/test_image_layout.py` holds the Dockerfile and the entrypoint to the same import root, `tests/test_package_layering.py` keeps the package's imports acyclic and keeps loading a module from a file path out of everything but the `bin/` shims, `tests/test_harness_conformance.py` holds every registered harness to the contract described under [Harness lifecycle](architecture.md), and `tests/test_docs_links.py` keeps every relative link in this tree and in the root `README.md` resolving, and every page reachable from [the index](../README.md). All four fail the way a build should — before anything reaches a container.
 

docs/tongs/README.md

@@ -13,7 +13,7 @@ The host-side launcher (`swarmforge/anvil/`, run through `bin/run-anvil`) discov
 1. (Only if the tong needs secrets) configure the secret-provider table (see [Secret providers](secrets.md)).
 2. Drop a tong definition into a layer directory, e.g. `~/.swarmforge/tongs/<name>.yaml` (personal) or `<workspace>/.swarmforge/tongs/<name>.yaml` (project).
 3. Run the anvil as usual (`oc`, or `make run_claude PROJECT_DIR=$(pwd)`). A **workspace**-sourced tong prints a privilege summary and asks for approval on first run (see [First-run approval](approval.md)). The launcher resolves secrets (which may prompt your provider CLI to unlock), starts the tong, waits for readiness, injects reachability into the anvil, then runs the anvil in the foreground.
-4. On exit (including Ctrl-C), the per-session network and its `session` tongs are torn down.
+4. On exit — Ctrl-C, a closed terminal or tmux pane (SIGHUP), or a plain `kill` (SIGTERM) — the `session` tongs and the per-session network are removed. If the anvil command has no `--network`, its container can outlive a killed docker client and hold the network, which then stays behind. Only `kill -9`, `Ctrl-\` (SIGQUIT), the OOM killer, a launcher crash, or a host reboot skips teardown.
 
 ## Where definitions live
 

swarmforge/anvil/__init__.py

@@ -23,7 +23,7 @@
     docker      the one seam every docker invocation goes through
     readiness   waiting for a started tong to report ready
     orchestrate starting the tongs, running the anvil, tearing down after it
-    errors      the one failure raised from more than one of the above
+    errors      the failures and interruptions raised from more than one of the above
 
 Every public name is re-exported below, so a caller imports `swarmforge.anvil`
 and never has to know which module a function sits in; a module's own private
@@ -41,11 +41,12 @@
 `session` tong is per-session: when any exists the launcher creates a per-session
 network, starts the `session` tongs on it under their canonical aliases, connects
 each network-facing `shared` tong to it, and joins the anvil to it (plus the base
-`NETWORK=` network). On exit -- including SIGINT -- the `session` tongs and the
-per-session network are torn down (and the connected `shared` tongs disconnected)
-while the long-lived `shared` tongs keep running. A `port` tong's reachability is
-injected into the anvil as environment; an `mcp` tong's as generated MCP config
-(see "MCP config"); a `none` tong has no anvil-facing surface.
+`NETWORK=` network). On exit -- including SIGINT, SIGHUP, and SIGTERM -- the
+`session` tongs and the per-session network are torn down (and the connected
+`shared` tongs disconnected) while the long-lived `shared` tongs keep running. A
+`port` tong's reachability is injected into the anvil as environment; an `mcp`
+tong's as generated MCP config (see "MCP config"); a `none` tong has no
+anvil-facing surface.
 
 A tong's secret references are resolved on the host (see "Secret delivery") and
 handed to the tong as environment, so the launcher starts `shared` and `session`
@@ -130,7 +131,7 @@
     parse_args,
 )
 from .docker import DockerCLI, DockerError
-from .errors import OrchestrationError
+from .errors import OrchestrationError, TerminationSignal
 from .orchestrate import (
     MCP_CONFIG_CONTAINER_PATH,
     MCP_FILE_ENV,
@@ -162,6 +163,7 @@
     "DockerCLI",
     "DockerError",
     "OrchestrationError",
+    "TerminationSignal",
     "MCP_CONFIG_CONTAINER_PATH",
     "MCP_FILE_ENV",
     "ensure_mcp_harness_supported",

swarmforge/anvil/cli.py

@@ -6,16 +6,21 @@
 then either execs the anvil argv verbatim (no tongs discovered) or hands the
 launch to the orchestrator. Every failure the launcher reports is mapped to a
 process exit code here.
+
+While the orchestrator runs, SIGHUP and SIGTERM unwind it through its teardown
+the way Ctrl-C does.
 """
 
 import collections
+import contextlib
 import os
+import signal
 
 from swarmforge import tongs
 
 from .approval import ApprovalDenied, gate_workspace_tongs
 from .docker import DockerCLI, DockerError
-from .errors import OrchestrationError
+from .errors import OrchestrationError, TerminationSignal
 from .orchestrate import (
     ensure_mcp_harness_supported,
     exec_anvil,
@@ -157,6 +162,73 @@ def discover_tongs(layer_dirs):
     return tongs.merge_tongs(tongs.discover(layer_dirs))
 
 
+_TERMINATION_SIGNALS = (signal.SIGHUP, signal.SIGTERM)
+
+
+class _TerminationTrap:
+    """SIGHUP/SIGTERM handler that raises `TerminationSignal`, or only records it.
+
+    Only the first signal is kept in `received`; later ones are dropped. It
+    raises unless `deferred` is entering or leaving, when it is only recorded
+    for `main` to turn into the exit status. SIGINT is left alone.
+    """
+
+    def __init__(self):
+        self.received = None
+        self._deferring = False
+
+    def handle(self, signum, frame):
+        if self.received is not None:
+            return
+        self.received = signum
+        if not self._deferring:
+            raise TerminationSignal(signum)
+
+    @contextlib.contextmanager
+    def deferred(self):
+        """Teardown guard: ignore SIGHUP/SIGTERM until the block has finished.
+
+        Ignored rather than blocked, because teardown's `docker` children
+        inherit it: a Go program clears an inherited signal mask but keeps an
+        inherited SIGHUP ignore, so a later hangup cannot cut a `docker rm`
+        short. Go does not keep SIGTERM ignored. A signal ignored here is lost.
+        """
+        self._deferring = True
+        trapped = [s for s in _TERMINATION_SIGNALS if signal.getsignal(s) == self.handle]
+        try:
+            for signum in trapped:
+                signal.signal(signum, signal.SIG_IGN)
+            yield
+        finally:
+            for signum in trapped:
+                signal.signal(signum, self.handle)
+            self._deferring = False
+
+
+@contextlib.contextmanager
+def _termination_signals_raise():
+    """Install a `_TerminationTrap` on SIGHUP/SIGTERM for the duration; yields it.
+
+    Python's default action for both skips every `finally`, where the
+    orchestrator tears down. A signal ignored on entry (`nohup`) stays ignored,
+    and the previous handlers are put back on the way out.
+    """
+    trap = _TerminationTrap()
+    previous = {}
+    try:
+        for signum in _TERMINATION_SIGNALS:
+            handler = signal.getsignal(signum)
+            if handler is signal.SIG_IGN:
+                continue
+            previous[signum] = handler
+            signal.signal(signum, trap.handle)
+        yield trap
+    finally:
+        for signum, handler in previous.items():
+            # None means a handler installed outside Python; the default is the best match.
+            signal.signal(signum, signal.SIG_DFL if handler is None else handler)
+
+
 def main(argv):
     try:
         opts, anvil_cmd = parse_args(argv)
@@ -221,13 +293,25 @@ def main(argv):
         tongs.warn(str(exc))
         return 1
 
+    # Outermost, so a signal landing in an except clause is caught too.
     try:
-        return run_with_tongs(
-            merged, anvil_cmd, opts, docker=DockerCLI(), providers=providers
-        )
-    except (OrchestrationError, DockerError, SecretResolutionError) as exc:
-        tongs.warn(str(exc))
-        return 1
-    except KeyboardInterrupt:
-        # 128 + SIGINT; the shared tongs stay running by design.
-        return 130
+        with _termination_signals_raise() as trap:
+            try:
+                rc = run_with_tongs(
+                    merged, anvil_cmd, opts, docker=DockerCLI(), providers=providers,
+                    teardown_guard=trap.deferred,
+                )
+            except (OrchestrationError, DockerError, SecretResolutionError) as exc:
+                # Nothing written after a hangup: the terminal may be gone.
+                if trap.received is not None:
+                    return 128 + trap.received
+                tongs.warn(str(exc))
+                return 1
+            except KeyboardInterrupt:
+                # 128 + SIGINT, even if a HUP/TERM followed.
+                return 130
+            if trap.received is not None:
+                return 128 + trap.received
+            return rc
+    except TerminationSignal as exc:
+        return 128 + exc.signum

swarmforge/anvil/docker.py

@@ -11,6 +11,8 @@
 
 from swarmforge import tongs
 
+from .errors import TerminationSignal
+
 
 class DockerError(Exception):
     """A docker command the launch depends on failed; the launch must stop."""
@@ -22,6 +24,9 @@ class DockerError(Exception):
 )
 _INSPECT_HEALTH_FORMAT = "{{if .State.Health}}{{.State.Health.Status}}{{end}}"
 
+# Teardown waits on this; under a TTY the forwarded signal just ends the client.
+_FOREGROUND_STOP_GRACE_S = 3
+
 
 class DockerCLI:
     def __init__(self, run=None):
@@ -209,22 +214,34 @@ def run_foreground_multi(self, argv, extra_networks, container):
         )
 
     def _wait_foreground(self, argv):
-        """Run a foreground command, reaping it on Ctrl-C before re-raising.
+        """Run a foreground command, reaping it on any interrupt before re-raising.
 
-        Popen + wait (rather than exec) so the launcher regains control after the
-        process exits. On Ctrl-C the SIGINT reaches both this process and the child
-        through the controlling terminal's process group; the child handles it and
-        exits, we reap it, and the KeyboardInterrupt propagates to the caller.
+        A SIGHUP/SIGTERM may have reached this process alone, so it is forwarded
+        to the child, which gets `_FOREGROUND_STOP_GRACE_S` to exit before it is
+        killed. The child is always reaped before the exception propagates, so
+        teardown never races a live client.
         """
         try:
             proc = subprocess.Popen(argv)
         except OSError as exc:
             raise DockerError("cannot run anvil %r: %s" % (argv[:2], exc))
         try:
-            return proc.wait()
-        except KeyboardInterrupt:
-            proc.wait()
+            try:
+                return proc.wait()
+            except KeyboardInterrupt:
+                proc.wait()
+                raise
+        except TerminationSignal as exc:
+            proc.send_signal(exc.signum)
+            try:
+                proc.wait(timeout=_FOREGROUND_STOP_GRACE_S)
+            except subprocess.TimeoutExpired:
+                pass
             raise
+        finally:
+            if proc.poll() is None:
+                proc.kill()
+            proc.wait()
 
 
 def _decode(output):

swarmforge/anvil/errors.py

@@ -1,11 +1,23 @@
-"""The one launch failure raised from more than one launcher module.
+"""The launch failures and interruptions more than one launcher module needs.
 
 Every other error the launcher raises is defined by the module that raises it.
-This one has two sources -- the secret channel, when a tong never takes delivery
-of its secrets, and the orchestrator, when a tong cannot be started or made
-ready -- so it sits on its own and neither has to import the other.
+These two cross modules: `OrchestrationError` is raised by both the secret
+channel and the orchestrator, and `TerminationSignal` is raised by the entry
+point but caught by the docker seam. Keeping them here means none of those
+modules imports another.
 """
 
 
 class OrchestrationError(Exception):
     """A tong could not be started/made ready; the launch stops."""
+
+
+class TerminationSignal(BaseException):
+    """SIGHUP or SIGTERM `signum` arrived mid-launch; unwinds through teardown.
+
+    A `BaseException`, like `KeyboardInterrupt`, so no `except Exception` swallows it.
+    """
+
+    def __init__(self, signum):
+        super().__init__(signum)
+        self.signum = signum