Skip to content

[Bug]: One tool's out-of-memory kill stops the whole Claude agent and its subagents #37876

Open

Description

@josephv123

What happened

A Claude agent ran a Python test that loaded a model on the CPU, and the machine ran out of memory. The kernel killed only the Python process:

kernel: oom-kill:constraint=CONSTRAINT_NONE,...,global_oom,task_memcg=/user.slice/user-1000.slice/user@1000.service/app.slice/app-t3code.slice/app-t3code-agents.slice/app-t3code-agents-7a9722fa0109.slice/t3code-claude-8fdb8ce3.scope,task=python,pid=443587
kernel: Out of memory: Killed process 443587 (python) total-vm:28182680kB, anon-rss:4431432kB, ... oom_score_adj:800
systemd[1815]: t3code-claude-8fdb8ce3.scope: The kernel OOM killer killed some processes in this unit.
systemd[1815]: t3code-claude-8fdb8ce3.scope: Failed with result 'oom-kill'.

Because the scope has OOMPolicy=stop, systemd then stopped the whole scope. That took down the claude process and the two background subagents it was running. The turn failed with "Killed: out of memory", and the user had to type "continue". The agent later explained: "the shared-prefix test ran the 0.8B model in full precision on lenovolinux's CPU, and that took the session and both helpers down with it."

Without the stop, the Bash tool would have returned exit 137 for that one command, and the agent could have run the test somewhere else.

Background

The trade-off

Switching thread scopes to continue (apps/server/src/process/agentScope.ts) is not a one-line change:

  • classifyScope treats oom_kill > 0 as "stopping". After a tool kill the scope survives, so every later failed turn would poll for about 2 seconds in oomKilled().
  • stop() leaves scopes with a non-zero counter for systemd to stop. Under continue, nothing stops them, so tool processes would outlive the agent.
  • The "Killed: out of memory" label would partly go away:
    • When claude itself is the victim: memory.events can still show the kill, but only if the cgroup still exists when it is read, which races once claude was the last process.
    • When systemd-oomd does the kill: the label is lost entirely, because oomd does not move the kernel counter.

Two alternatives don't fit:

  • memory.oom.group kills the whole group, which is the opposite of what's wanted.
  • Per-tool scopes aren't possible, because Claude Code starts its own Bash children inside its cgroup.

The fix I'd propose:

  1. Use continue for thread scopes.
  2. Stop treating a non-zero counter as "systemd is stopping this" in classifyScope and stop().
  3. Label a failed turn from a counter increase together with the agent's own SIGKILL exit.
  4. Accept that the label is lost for oomd kills and in the race above.

That's roughly 50 to 100 lines with tests. I can send a PR if you want that trade.

Version or commit

Nightly 0.1.10-nightly.20261116.4800; agentScope.ts is unchanged on main at 1d20ad0.

Environment

Ubuntu 26.04, systemd 259, kernel 7.0, 16 GB RAM, Claude Code 2.1.326, Claude Opus 5.5.

Activity

  1. fsioni commented on Nov 17, 2026

    @fsioni
    Contributor

    +1, hitting this on a second machine, with one difference worth noting: here the killer is systemd-oomd, not the kernel OOM killer, so OOMPolicy=continue alone would not help.

    Setup: Ubuntu, 16 cores / 26 GB RAM, nightly 0.1.8 build, Claude agents compiling and testing a Rust workspace in several worktrees at once.

    In 45 minutes, oomd killed nine whole Claude agent scopes. Every one of those turns failed with "Killed: out of memory":

    09:01:10 t3code-claude-20c149e4.scope: systemd-oomd killed 10 process(es) in this unit.
    09:15:11 t3code-claude-155c0f2e.scope: systemd-oomd killed 10 process(es) in this unit.
    09:19:56 t3code-claude-4b1b5aa0.scope: systemd-oomd killed 5 process(es) in this unit.
    09:27:21 t3code-claude-da909738.scope: systemd-oomd killed 25 process(es) in this unit.
    09:32:30 t3code-claude-7d33f19a.scope: systemd-oomd killed 10 process(es) in this unit.
    09:40:22 t3code-claude-e01b7061.scope: systemd-oomd killed 21 process(es) in this unit.
    09:44:00 t3code-claude-5ab4becf.scope: systemd-oomd killed 9 process(es) in this unit.
    09:45:27 t3code-claude-e6466cbe.scope: systemd-oomd killed 16 process(es) in this unit.
    09:45:43 t3code-claude-aad5aedb.scope: systemd-oomd killed 5 process(es) in this unit.
    systemd-oomd: Killed .../app-t3code-agents.slice/app-t3code-agents-<hash>.slice/t3code-claude-aad5aedb.scope
      due to memory pressure for /user.slice/user-<uid>.slice/user@<uid>.service being 84.19% > 50.00% for > 20s with reclaim activity
    

    user@.service has Ubuntu's default ManagedOOMMemoryPressure=kill at a 50% limit. The agents slice had MemoryHigh=21.9G and MemoryMax=24.1G.

    What this adds to the analysis above:

    • oomd always kills the whole cgroup, whatever its OOMPolicy. Switching thread scopes to continue fixes the kernel case but not this one. Each scope still holds claude together with every tool it spawned.
    • The MemoryHigh throttle on the agents slice may make this worse. Throttling forces reclaim, the pressure that oomd measures rises, and oomd can then kill a scope before MemoryMax is ever reached.
    • The label was shown correctly in this case: systemd recorded Result=oom-kill for the oomd kills. Point 4 above may be too pessimistic, at least on this systemd version.

    A fix for this case probably needs the agent's tools in a child cgroup that oomd can pick instead of the agent. Another option is ManagedOOMPreference=avoid on the agent process, though that only works for cgroups the user owns.

    Investigated with Claude Code (Opus 5.5).

  2. caniko commented on Nov 17, 2026

    @caniko

    Same on a NixOS workstation (kernel OOM killer, not oomd). On 2026-11-14 single tool kills (rust-analyzer, nix, a linker) stopped nine Claude sessions mid-turn.

    We've run per-thread scopes with OOMPolicy=continue since then in a fork: caniko@3d53030 (12 lines added, 24 removed in agentScope.ts and its test). Besides the policy, it drops the early return in stop() that left scopes with a non-zero OOM counter for systemd, so a scope is reaped when its agent exits even after a tool was killed. That covers your second point. It doesn't touch classifyScope, so the ~2 s oomKilled() poll after a tool kill and the "Killed: out of memory" label are still open. With continue, the killed tool fails its own command and the agent keeps its turn.

    (Written with Claude Opus 5.5 in Claude Code.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions