Skip to content

Lost-run re-attach window (15s) too tight: daemon restart orphans the run, model serves unleased, bridge livelocks in acquire/refuse #300

Open

Description

@azrael-229

Observed (Windows 11, RTX 5070 Ti, v0.4.1, 2026-11-05):

  1. 20:13:21 granted lease=qwen-3.8-27b-9 … attached pid=61524 (bridge wrapper launches llama-server).
  2. 20:22:53 daemon restart (service reinstall). Wrapper survives, writes lost-run record (berthd went away mid-run), keeps server up (fail-open).
  3. 20:22:54 new daemon up. 20:23:10 squatter: pid=61524 … mib=14579 unprotected_run=qwen-3.8-27b — 15s re-attach window (default, broker.go:599) already expired; no warning logged when the window lapses un-re-attached.
  4. 20:30:57 bridge acquires qwen-3.8-27b-1; 20:42:37 refused … needs 14000 MiB; at most 690 MiB can be freed; squatters hold 14814 MiB. Repeats: -2 refused 20:54:30, -3 waiting. Each cycle respawns a berth run wrapper.

Two issues:

  1. Re-attach race. The wrapper must notice the disconnect, write the record, and re-dial a socket being re-bound concurrently, inside 15s of the new daemon's start. That is not reliable; the failure is silent (no log line when the window expires with the run still un-re-attached). Suggest: longer default (or config), and a WARN when the window lapses with runs still due.
  2. No self-admission. A waiter for engine E can never board while E's own orphaned server (already identified by the broker via unprotected_run) holds the card — the wait is unsatisfiable by construction, so the bridge livelocks in an acquire→refuse loop while serving continues from the orphan. Suggest: when a waiter's engine matches a live lost run of the same model, offer adoption (attach the orphan's pid to the granted lease) instead of an infinite wait; and/or let the requester's -wait refusal carry a distinct reason ("model resident but unleased") so integrators can recover.

Severity: medium. Serving is unharmed (llama-swap routes to the live server); the card is genuinely full so no scheduling loss vs. a legitimately resident model; but the state never self-heals until the orphan is killed, the board misreports, and wrappers churn indefinitely.

Activity

  1. T-Crypt commented on Nov 6, 2026

    @T-Crypt
    Owner

    Thanks for the detailed timeline. The timestamps made this one quick to follow.

    One correction to the premise first. berthd accepts a re-attach at any time, not only inside the 15 s window. The window does two things. It holds new grants while the new daemon waits for engines that ran under the old one, and it decides when an orphan starts to count as a squatter. Your wrapper keeps dialing every 2 s until a daemon answers. So a longer window would not have changed what you saw.

    The wrapper stops for good after one case: the new daemon answers and refuses the re-attach. It then prints berthd would not re-attach qwen-3.8-27b (REASON); it stays unprotected to its own stderr and never tries again. berthd logged nothing about it. That fits your log, where pid 61524 stays a squatter from 20:23:10 on. The other way to get there is a wrapper that never reaches the new daemon at all, for example because the reinstall moved the socket. That one keeps dialing in silence.

    #301 fixes the two silent parts:

    • berthd logs re-attach refused at WARN with the engine, the PID and the reason.
    • A request that cannot fit because an unleased run of the same engine holds the card says so in its refusal: squatters hold 14814 MiB, 14579 MiB of that is an unleased qwen-3.8-27b (pid 61524). Your bridge can match that text and stop the orphan instead of asking again.

    I left adoption out. The orphan already has an owner, its old berth run, which is still alive and holds the server's process handle and job object. A new wrapper that took over the PID would make two owners of one process, and a drain from either would race the other. The right owner to re-attach is the old wrapper, so I'd rather find out why it was refused.

    Could you look for these two things from that evening?

    1. The stderr of the berth run that started pid 61524, from about 20:22:53 on. Look for the would not re-attach line and its reason. If you find only UNPROTECTED: lost berthd with nothing after it, please also check that the old and the new daemon use the same socket path.
    2. berth events from 20:22:50 to 20:24, and the command line your bridge passes to berth run (its flags matter, --gpu and --needs-mib above all).

    Until a fix lands, the way out of the loop is to end the orphan, Stop-Process -Id 61524 in this case. The bridge's next berth run then gets the card and starts a server under a lease. berth stop cannot reach it, because it only acts on leases.

  2. azrael-229 commented on Nov 6, 2026

    @azrael-229
    Author

    Corrected comment below

  3. T-Crypt commented on Nov 8, 2026

    @T-Crypt
    Owner

    Your last comment came through as a file path on your machine (C:\Users\...\issue300-reply.md) and not the text inside it. Could you paste the reply itself here?

    The parts that decide the next step are:

    • the wrapper's stderr line berthd would not re-attach qwen-3.8-27b (REASON), with the reason,
    • berth events from 20:22:50 to 20:24,
    • the bridge's flags and the socket path it dials.

    v0.5.0-rc.1 is out today. It has #301, so berthd now logs a refused re-attach with its reason in its own log. If the orphan happens again on the rc, berth logs (or the service's event log on Windows) shows why without the wrapper's stderr.

  4. azrael-229 commented on Nov 8, 2026

    @azrael-229
    Author

    Sorry about that, I am really unsure how this happened. The 1. part is somewhat obsolete considering the RC tagged code, but I leave it for completeness. The original comment:

    The wrapper's stderr itself didn't survive, but the daemon log did, and it answers both questions.

    1. Wrapper stderr (pid 61524, from 20:22:53). Not recoverable — precisely why: llama-swap pipes the child's stdout+stderr into an in-memory log ring (cmd.Stderr = cmd.Stdout = processLogger, internal/process/process_command.go:537-538, logmon.Monitor), served at GET /logs only. There is no file sink, and my config leaves logToStdout at its default (proxy), so the upstream stream never hit the console either. The llama-swap process holding that ring was restarted 05:47 on Nov 6 (/logs now begins with that boot), so the Nov 5 lines are gone.

    The fallback check: old and new daemon used the same socket. Old daemon (up 15:14:14, v0.4.1) and new daemon (20:22:54.790) both log socket=C:\Users\jakub\AppData\Local\berth\berth.sock. So "never reached the new daemon" is out — the refusal branch is what happened.

    The daemon log also shows why the refusal must be the "cannot fit" one, plus one thing that may sharpen #301. The run started under a valid lease on the old daemon:

    time=2026-11-05T20:13:21.481+01:00 level=INFO msg=acquire lease=qwen-3.8-27b-9 engine=qwen-3.8-27b priority=foreground needs_mib=14000
    time=2026-11-05T20:13:21.955+01:00 level=INFO msg=attached lease=qwen-3.8-27b-9 pid=61524
    

    The old daemon then died with no exit log (that attach is its last line; the new daemon bound the socket at 20:22:54.681). The new daemon saw the run at 20:23:10.804 holding 14,579 MiB unleased — more than the declared --needs-mib 14000. So the re-attach on the fresh daemon could never fit, whatever the re-attach path does, because the run itself holds more than it declared: free space was ~1.3 GiB against a 14,000 MiB need.

    2. Events 20:22:50 → 20:24, and the bridge command line. The daemon's event ring has long since rolled (in-memory, ~1000), but berthd.log persists the same stream. The full window — this is everything between 20:22:50 and 20:24 (nothing else until 20:30:57):

    time=2026-11-05T20:22:54.681+01:00 level=INFO msg="collisions board restored" file=C:\Users\jakub\.local\state\berth\counters.json since=2026-10-21T02:37:47+02:00 collisions_avoided=78
    time=2026-11-05T20:22:54.723+01:00 level=INFO msg="gpu found" index=0 name="NVIDIA GeForce RTX 5070 Ti" pci=00000000:01:00.0 total_mib=16303 scheduled=true
    time=2026-11-05T20:22:54.723+01:00 level=INFO msg=governing cards=gpu0
    time=2026-11-05T20:22:54.782+01:00 level=WARN msg="squatter: GPU memory held without a lease" pid=38856 name="C:\\Program Files\\WindowsApps\\Microsoft.WindowsTerminal_1.25.2733.0_x64__8wekyb3d8bbwe\\WindowsTerminal.exe" mib=156 gtt_mib=0 unprotected_run="" container=""
    time=2026-11-05T20:22:54.782+01:00 level=WARN msg="squatter: GPU memory held without a lease" pid=21872 name="C:\\Program Files\\WindowsApps\\Microsoft.YourPhone_1.26092.119.0_x64__8wekyb3d8bbwe\\PhoneExperienceHost.exe" mib=139 gtt_mib=0 unprotected_run="" container=""
    time=2026-11-05T20:22:54.790+01:00 level=INFO msg="berthd up" version=0.4.1 socket=C:\Users\jakub\AppData\Local\berth\berth.sock gpu=all
    time=2026-11-05T20:22:54.791+01:00 level=INFO msg="claims armed" hold=10 track=6 builtin=true
    time=2026-11-05T20:22:54.791+01:00 level=INFO msg="reading engine" name=llama-swap type=llama-swap url=http://127.0.0.1:1233
    time=2026-11-05T20:22:54.791+01:00 level=INFO msg="reading engine" name=deduper type=custom url=http://localhost:8086
    time=2026-11-05T20:22:54.791+01:00 level=INFO msg="reading engine" name=pet-tagger type=custom url=http://127.0.0.1:59998
    time=2026-11-05T20:22:54.791+01:00 level=INFO msg="reading engine" name=immichml type=custom url=http://127.0.0.1:3003
    time=2026-11-05T20:22:54.791+01:00 level=INFO msg="reading engine" name=nvidiabr type=custom url=http://localhost:59999
    time=2026-11-05T20:22:54.791+01:00 level=INFO msg="discovering engines" every=1m0s hosts=[] extra_ports=[]
    time=2026-11-05T20:22:54.791+01:00 level=INFO msg="comfy squat-free not armed: no [[engine]] with type \"comfyui\""
    time=2026-11-05T20:23:10.804+01:00 level=WARN msg="squatter: GPU memory held without a lease" pid=61524 name=F:\AI\llama.cpp-adaptive-kv-streaming-v2\build\bin\Release\llama-server.exe mib=14579 gtt_mib=0 unprotected_run=qwen-3.8-27b container=""
    

    Command line the bridge passes to berth run (from the live wrapper, identical template in my llama-swap config):

    F:\AI\berth\berth.exe run --as qwen-3.8-27b --group llama-swap --priority foreground --needs-mib 14000 -wait 700s --drain-post http://127.0.0.1:1233/api/models/unload/qwen-3.8-27b --drain-slots http://127.0.0.1:1233/upstream/qwen-3.8-27b -- <llama-server …>
    

    Flags you asked about: no --gpu (default applies), --needs-mib 14000 — and note the actual hold (14,579 MiB) exceeded the declaration by ~580 MiB (KV + MTP draft), which is what made the re-attach un-fittable above.

    For completeness, the loop that followed: the bridge re-ran berth run six more times (20:30:57, 20:42:50, 20:55:42, 21:09:48, 04:43:51, 04:57:37 → leases qwen-3.8-27b-1 … -6), each refused after 11m40s, e.g.

    time=2026-11-05T20:42:37.397+01:00 level=WARN msg=refused lease=qwen-3.8-27b-1 engine=qwen-3.8-27b reason="gave up after 11m40s: next in line: needs 14000 MiB; at most 690 MiB can be freed; squatters hold 14814 MiB"
    

    (the squatter figure includes the orphan itself). I ended the orphan at 05:15:54 (the taskkill from the issue); lease -7 (acquired 05:09:37) was granted 0.4 s later and attached a fresh server (pid 51504).

  5. T-Crypt commented on Nov 12, 2026

    @T-Crypt
    Owner

    Thanks for pasting it, and for digging the window out of berthd.log. It answers more than it looks like, though it points somewhere other than the fit.

    A re-attach has no fit check. berthd grants it at once, whatever the card holds, because the engine is already on the card. In v0.4.1 it refuses for only three reasons:

    • the PID is not the process that started as the engine (its start time differs),
    • the card is an APU and the run sets --needs-mib,
    • the PID already holds a lease.

    Your log rules out the first one. At 20:23:10 berthd still listed the run's record (unprotected_run=qwen-3.8-27b), and it keeps a record only while the PID's start time matches the one recorded. A 5070 Ti is not an APU, and nothing else held a lease for pid 61524. So I don't think the wrapper was refused. I think it never asked.

    The six later berth run starts point the same way. llama-swap starts a new process for a model only after it sees the old one end. On Windows, an engine outlives its berth run on purpose. Its job object has no kill-on-close, so a dead wrapper never takes a serving model down with it. My reading, which is inferred and not reproduced, is that the berth run for pid 61524 ended at or soon after 20:22:53. Its llama-server kept serving, with nothing left to re-attach it. I don't know yet what ended it.

    #306 (merged into main, in the next release) makes this visible without the wrapper's stderr. The unprotected-run record now names the berth run that wrote it. The squatter line logs berth_run=exited or berth_run=alive, and a request refused behind its own orphan adds one of these:

    • its berth run (pid N) has exited, so nothing will re-attach it
    • its berth run (pid N) is alive and has not re-attached

    If you can, try this once on your current build. With a model loaded through the bridge, restart berthd the way the reinstall did, then check in Task Manager whether that model's berth.exe run process is still there. If it is gone, something ended it, either the reinstall itself or llama-swap, and that is what to find next. If it is still there and the model stays a squatter, that is a bug on my side and I want the details.

    One more thing from your log. The run declared --needs-mib 14000 and held 14,579 MiB. That did not cause the orphan, but Berth reserves the declared size and grants other engines around it, so a run that grows past it can push a neighbour out of memory. --needs-mib 14700 matches what that model really holds.

  6. azrael-229 commented on Nov 12, 2026

    @azrael-229
    Author

    One more thing from your log. The run declared --needs-mib 14000 and held 14,579 MiB. That did not cause the orphan, but Berth reserves the declared size and grants other engines around it, so a run that grows past it can push a neighbour out of memory. --needs-mib 14700 matches what that model really holds.

    This is a head scratcher for me. As in both how to set it up for myself and how Berth should handle it. In Windows and non-headless environment when using the VRAM to the very limit like I do (I usually have only 300MB or less free) I am either hitting the situation above or I am hitting unending refusal from Berth when I know the model will load (the desktop will be evicted, some memory will be reclaimed by defragmentation). This is really a matter of how NVidia driver manages memory, not Berth issue and this is more general for using LLMs or ML loads in general. Sometimes even 1-2 retries will make something fit. Computing shouldn't work like this but here we are.

    If you can, try this once on your current build. With a model loaded through the bridge, restart berthd the way the reinstall did, then check in Task Manager whether that model's berth.exe run process is still there. If it is gone, something ended it, either the reinstall itself or llama-swap, and that is what to find next. If it is still there and the model stays a squatter, that is a bug on my side and I want the details.

    I will test it as soon as I can get some free GPU time. Probably within a day or two.

  7. T-Crypt commented on Nov 12, 2026

    @T-Crypt
    Owner

    Thanks, no rush. v0.5.1-rc.1 has #306, so if you run the restart test on it, berthd's log says berth_run=exited or berth_run=alive on the squatter line, and Task Manager is only a cross-check. It is a pre-release, so berth update will not offer it: download it from the release page.

    On sizing at the edge of the card: you are right that the driver decides, and that a typed --needs-mib is a guess you have to keep up to date. Berth already measures what each run holds. The next step I want is for a run with no --needs-mib to be sized from what that model held last time, so you never type the number. I'll open an issue for it with your case in it.

  8. T-Crypt commented on Nov 14, 2026

    @T-Crypt
    Owner

    v0.6.0-rc.1 is out and has two changes for the restart test:

    If you run the restart test on rc.1, berth why for the model from that moment would show whether the re-attach still misses the window. I'm leaving this open for that. No rush.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions