You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Lost-run re-attach window (15s) too tight: daemon restart orphans the run, model serves unleased, bridge livelocks in acquire/refuse #300
20:22:53 daemon restart (service reinstall). Wrapper survives, writes lost-run record (berthd went away mid-run), keeps server up (fail-open).
20:22:54 new daemon up. 20:23:10 squatter: pid=61524 … mib=14579 unprotected_run=qwen-3.8-27b — 15s re-attach window (default, broker.go:599) already expired; no warning logged when the window lapses un-re-attached.
20:30:57 bridge acquires qwen-3.8-27b-1; 20:42:37 refused … needs 14000 MiB; at most 690 MiB can be freed; squatters hold 14814 MiB. Repeats: -2 refused 20:54:30, -3 waiting. Each cycle respawns a berth run wrapper.
Two issues:
Re-attach race. The wrapper must notice the disconnect, write the record, and re-dial a socket being re-bound concurrently, inside 15s of the new daemon's start. That is not reliable; the failure is silent (no log line when the window expires with the run still un-re-attached). Suggest: longer default (or config), and a WARN when the window lapses with runs still due.
No self-admission. A waiter for engine E can never board while E's own orphaned server (already identified by the broker via unprotected_run) holds the card — the wait is unsatisfiable by construction, so the bridge livelocks in an acquire→refuse loop while serving continues from the orphan. Suggest: when a waiter's engine matches a live lost run of the same model, offer adoption (attach the orphan's pid to the granted lease) instead of an infinite wait; and/or let the requester's -wait refusal carry a distinct reason ("model resident but unleased") so integrators can recover.
Severity: medium. Serving is unharmed (llama-swap routes to the live server); the card is genuinely full so no scheduling loss vs. a legitimately resident model; but the state never self-heals until the orphan is killed, the board misreports, and wrappers churn indefinitely.
Thanks for the detailed timeline. The timestamps made this one quick to follow.
One correction to the premise first. berthd accepts a re-attach at any time, not only inside the 15 s window. The window does two things. It holds new grants while the new daemon waits for engines that ran under the old one, and it decides when an orphan starts to count as a squatter. Your wrapper keeps dialing every 2 s until a daemon answers. So a longer window would not have changed what you saw.
The wrapper stops for good after one case: the new daemon answers and refuses the re-attach. It then prints berthd would not re-attach qwen-3.8-27b (REASON); it stays unprotected to its own stderr and never tries again. berthd logged nothing about it. That fits your log, where pid 61524 stays a squatter from 20:23:10 on. The other way to get there is a wrapper that never reaches the new daemon at all, for example because the reinstall moved the socket. That one keeps dialing in silence.
berthd logs re-attach refused at WARN with the engine, the PID and the reason.
A request that cannot fit because an unleased run of the same engine holds the card says so in its refusal: squatters hold 14814 MiB, 14579 MiB of that is an unleased qwen-3.8-27b (pid 61524). Your bridge can match that text and stop the orphan instead of asking again.
I left adoption out. The orphan already has an owner, its old berth run, which is still alive and holds the server's process handle and job object. A new wrapper that took over the PID would make two owners of one process, and a drain from either would race the other. The right owner to re-attach is the old wrapper, so I'd rather find out why it was refused.
Could you look for these two things from that evening?
The stderr of the berth run that started pid 61524, from about 20:22:53 on. Look for the would not re-attach line and its reason. If you find only UNPROTECTED: lost berthd with nothing after it, please also check that the old and the new daemon use the same socket path.
berth events from 20:22:50 to 20:24, and the command line your bridge passes to berth run (its flags matter, --gpu and --needs-mib above all).
Until a fix lands, the way out of the loop is to end the orphan, Stop-Process -Id 61524 in this case. The bridge's next berth run then gets the card and starts a server under a lease. berth stop cannot reach it, because it only acts on leases.
Your last comment came through as a file path on your machine (C:\Users\...\issue300-reply.md) and not the text inside it. Could you paste the reply itself here?
The parts that decide the next step are:
the wrapper's stderr line berthd would not re-attach qwen-3.8-27b (REASON), with the reason,
berth events from 20:22:50 to 20:24,
the bridge's flags and the socket path it dials.
v0.5.0-rc.1 is out today. It has #301, so berthd now logs a refused re-attach with its reason in its own log. If the orphan happens again on the rc, berth logs (or the service's event log on Windows) shows why without the wrapper's stderr.
Sorry about that, I am really unsure how this happened. The 1. part is somewhat obsolete considering the RC tagged code, but I leave it for completeness. The original comment:
The wrapper's stderr itself didn't survive, but the daemon log did, and it answers both questions.
1. Wrapper stderr (pid 61524, from 20:22:53). Not recoverable — precisely why: llama-swap pipes the child's stdout+stderr into an in-memory log ring (cmd.Stderr = cmd.Stdout = processLogger, internal/process/process_command.go:537-538, logmon.Monitor), served at GET /logs only. There is no file sink, and my config leaves logToStdout at its default (proxy), so the upstream stream never hit the console either. The llama-swap process holding that ring was restarted 05:47 on Nov 6 (/logs now begins with that boot), so the Nov 5 lines are gone.
The fallback check: old and new daemon used the same socket. Old daemon (up 15:14:14, v0.4.1) and new daemon (20:22:54.790) both log socket=C:\Users\jakub\AppData\Local\berth\berth.sock. So "never reached the new daemon" is out — the refusal branch is what happened.
The daemon log also shows why the refusal must be the "cannot fit" one, plus one thing that may sharpen #301. The run started under a valid lease on the old daemon:
The old daemon then died with no exit log (that attach is its last line; the new daemon bound the socket at 20:22:54.681). The new daemon saw the run at 20:23:10.804 holding 14,579 MiB unleased — more than the declared --needs-mib 14000. So the re-attach on the fresh daemon could never fit, whatever the re-attach path does, because the run itself holds more than it declared: free space was ~1.3 GiB against a 14,000 MiB need.
2. Events 20:22:50 → 20:24, and the bridge command line. The daemon's event ring has long since rolled (in-memory, ~1000), but berthd.log persists the same stream. The full window — this is everything between 20:22:50 and 20:24 (nothing else until 20:30:57):
Flags you asked about: no --gpu (default applies), --needs-mib 14000 — and note the actual hold (14,579 MiB) exceeded the declaration by ~580 MiB (KV + MTP draft), which is what made the re-attach un-fittable above.
For completeness, the loop that followed: the bridge re-ran berth run six more times (20:30:57, 20:42:50, 20:55:42, 21:09:48, 04:43:51, 04:57:37 → leases qwen-3.8-27b-1 … -6), each refused after 11m40s, e.g.
time=2026-11-05T20:42:37.397+01:00 level=WARN msg=refused lease=qwen-3.8-27b-1 engine=qwen-3.8-27b reason="gave up after 11m40s: next in line: needs 14000 MiB; at most 690 MiB can be freed; squatters hold 14814 MiB"
(the squatter figure includes the orphan itself). I ended the orphan at 05:15:54 (the taskkill from the issue); lease -7 (acquired 05:09:37) was granted 0.4 s later and attached a fresh server (pid 51504).
Thanks for pasting it, and for digging the window out of berthd.log. It answers more than it looks like, though it points somewhere other than the fit.
A re-attach has no fit check. berthd grants it at once, whatever the card holds, because the engine is already on the card. In v0.4.1 it refuses for only three reasons:
the PID is not the process that started as the engine (its start time differs),
the card is an APU and the run sets --needs-mib,
the PID already holds a lease.
Your log rules out the first one. At 20:23:10 berthd still listed the run's record (unprotected_run=qwen-3.8-27b), and it keeps a record only while the PID's start time matches the one recorded. A 5070 Ti is not an APU, and nothing else held a lease for pid 61524. So I don't think the wrapper was refused. I think it never asked.
The six later berth run starts point the same way. llama-swap starts a new process for a model only after it sees the old one end. On Windows, an engine outlives its berth run on purpose. Its job object has no kill-on-close, so a dead wrapper never takes a serving model down with it. My reading, which is inferred and not reproduced, is that the berth run for pid 61524 ended at or soon after 20:22:53. Its llama-server kept serving, with nothing left to re-attach it. I don't know yet what ended it.
#306 (merged into main, in the next release) makes this visible without the wrapper's stderr. The unprotected-run record now names the berth run that wrote it. The squatter line logs berth_run=exited or berth_run=alive, and a request refused behind its own orphan adds one of these:
its berth run (pid N) has exited, so nothing will re-attach it
its berth run (pid N) is alive and has not re-attached
If you can, try this once on your current build. With a model loaded through the bridge, restart berthd the way the reinstall did, then check in Task Manager whether that model's berth.exe run process is still there. If it is gone, something ended it, either the reinstall itself or llama-swap, and that is what to find next. If it is still there and the model stays a squatter, that is a bug on my side and I want the details.
One more thing from your log. The run declared --needs-mib 14000 and held 14,579 MiB. That did not cause the orphan, but Berth reserves the declared size and grants other engines around it, so a run that grows past it can push a neighbour out of memory. --needs-mib 14700 matches what that model really holds.
One more thing from your log. The run declared --needs-mib 14000 and held 14,579 MiB. That did not cause the orphan, but Berth reserves the declared size and grants other engines around it, so a run that grows past it can push a neighbour out of memory. --needs-mib 14700 matches what that model really holds.
This is a head scratcher for me. As in both how to set it up for myself and how Berth should handle it. In Windows and non-headless environment when using the VRAM to the very limit like I do (I usually have only 300MB or less free) I am either hitting the situation above or I am hitting unending refusal from Berth when I know the model will load (the desktop will be evicted, some memory will be reclaimed by defragmentation). This is really a matter of how NVidia driver manages memory, not Berth issue and this is more general for using LLMs or ML loads in general. Sometimes even 1-2 retries will make something fit. Computing shouldn't work like this but here we are.
If you can, try this once on your current build. With a model loaded through the bridge, restart berthd the way the reinstall did, then check in Task Manager whether that model's berth.exe run process is still there. If it is gone, something ended it, either the reinstall itself or llama-swap, and that is what to find next. If it is still there and the model stays a squatter, that is a bug on my side and I want the details.
I will test it as soon as I can get some free GPU time. Probably within a day or two.
Thanks, no rush. v0.5.1-rc.1 has #306, so if you run the restart test on it, berthd's log says berth_run=exited or berth_run=alive on the squatter line, and Task Manager is only a cross-check. It is a pre-release, so berth update will not offer it: download it from the release page.
On sizing at the edge of the card: you are right that the driver decides, and that a typed --needs-mib is a guess you have to keep up to date. Berth already measures what each run holds. The next step I want is for a run with no --needs-mib to be sized from what that model held last time, so you never type the number. I'll open an issue for it with your case in it.
berthd keeps its event log in events.jsonl across restarts, and lease IDs no longer start again at 1 (feat(events): keep the event stream across restarts, and berth why ID #330). After a restart, berth why ENGINE prints that engine's last lease from the request to the re-attach, with each wait and drain reason, and names the lease it re-attached as.
If you run the restart test on rc.1, berth why for the model from that moment would show whether the re-attach still misses the window. I'm leaving this open for that. No rush.
Activity
T-Crypt commentedon Nov 6, 2026
Thanks for the detailed timeline. The timestamps made this one quick to follow.
One correction to the premise first. berthd accepts a re-attach at any time, not only inside the 15 s window. The window does two things. It holds new grants while the new daemon waits for engines that ran under the old one, and it decides when an orphan starts to count as a squatter. Your wrapper keeps dialing every 2 s until a daemon answers. So a longer window would not have changed what you saw.
The wrapper stops for good after one case: the new daemon answers and refuses the re-attach. It then prints
berthd would not re-attach qwen-3.8-27b (REASON); it stays unprotectedto its own stderr and never tries again. berthd logged nothing about it. That fits your log, where pid 61524 stays a squatter from 20:23:10 on. The other way to get there is a wrapper that never reaches the new daemon at all, for example because the reinstall moved the socket. That one keeps dialing in silence.#301 fixes the two silent parts:
re-attach refusedat WARN with the engine, the PID and the reason.squatters hold 14814 MiB, 14579 MiB of that is an unleased qwen-3.8-27b (pid 61524). Your bridge can match that text and stop the orphan instead of asking again.I left adoption out. The orphan already has an owner, its old
berth run, which is still alive and holds the server's process handle and job object. A new wrapper that took over the PID would make two owners of one process, and a drain from either would race the other. The right owner to re-attach is the old wrapper, so I'd rather find out why it was refused.Could you look for these two things from that evening?
berth runthat started pid 61524, from about 20:22:53 on. Look for thewould not re-attachline and its reason. If you find onlyUNPROTECTED: lost berthdwith nothing after it, please also check that the old and the new daemon use the same socket path.berth eventsfrom 20:22:50 to 20:24, and the command line your bridge passes toberth run(its flags matter,--gpuand--needs-mibabove all).Until a fix lands, the way out of the loop is to end the orphan,
Stop-Process -Id 61524in this case. The bridge's nextberth runthen gets the card and starts a server under a lease.berth stopcannot reach it, because it only acts on leases.azrael-229 commentedon Nov 6, 2026
Corrected comment below
T-Crypt commentedon Nov 8, 2026
Your last comment came through as a file path on your machine (
C:\Users\...\issue300-reply.md) and not the text inside it. Could you paste the reply itself here?The parts that decide the next step are:
berthd would not re-attach qwen-3.8-27b (REASON), with the reason,berth eventsfrom 20:22:50 to 20:24,v0.5.0-rc.1 is out today. It has #301, so berthd now logs a refused re-attach with its reason in its own log. If the orphan happens again on the rc,
berth logs(or the service's event log on Windows) shows why without the wrapper's stderr.azrael-229 commentedon Nov 8, 2026
Sorry about that, I am really unsure how this happened. The 1. part is somewhat obsolete considering the RC tagged code, but I leave it for completeness. The original comment:
The wrapper's stderr itself didn't survive, but the daemon log did, and it answers both questions.
1. Wrapper stderr (pid 61524, from 20:22:53). Not recoverable — precisely why: llama-swap pipes the child's stdout+stderr into an in-memory log ring (
cmd.Stderr = cmd.Stdout = processLogger, internal/process/process_command.go:537-538,logmon.Monitor), served atGET /logsonly. There is no file sink, and my config leaveslogToStdoutat its default (proxy), so the upstream stream never hit the console either. The llama-swap process holding that ring was restarted 05:47 on Nov 6 (/logsnow begins with that boot), so the Nov 5 lines are gone.The fallback check: old and new daemon used the same socket. Old daemon (up 15:14:14, v0.4.1) and new daemon (20:22:54.790) both log
socket=C:\Users\jakub\AppData\Local\berth\berth.sock. So "never reached the new daemon" is out — the refusal branch is what happened.The daemon log also shows why the refusal must be the "cannot fit" one, plus one thing that may sharpen #301. The run started under a valid lease on the old daemon:
The old daemon then died with no exit log (that attach is its last line; the new daemon bound the socket at 20:22:54.681). The new daemon saw the run at 20:23:10.804 holding 14,579 MiB unleased — more than the declared
--needs-mib 14000. So the re-attach on the fresh daemon could never fit, whatever the re-attach path does, because the run itself holds more than it declared: free space was ~1.3 GiB against a 14,000 MiB need.2. Events 20:22:50 → 20:24, and the bridge command line. The daemon's event ring has long since rolled (in-memory, ~1000), but
berthd.logpersists the same stream. The full window — this is everything between 20:22:50 and 20:24 (nothing else until 20:30:57):Command line the bridge passes to
berth run(from the live wrapper, identical template in my llama-swap config):Flags you asked about: no
--gpu(default applies),--needs-mib 14000— and note the actual hold (14,579 MiB) exceeded the declaration by ~580 MiB (KV + MTP draft), which is what made the re-attach un-fittable above.For completeness, the loop that followed: the bridge re-ran
berth runsix more times (20:30:57, 20:42:50, 20:55:42, 21:09:48, 04:43:51, 04:57:37 → leasesqwen-3.8-27b-1…-6), each refused after 11m40s, e.g.(the squatter figure includes the orphan itself). I ended the orphan at 05:15:54 (the taskkill from the issue); lease
-7(acquired 05:09:37) was granted 0.4 s later and attached a fresh server (pid 51504).T-Crypt commentedon Nov 12, 2026
Thanks for pasting it, and for digging the window out of
berthd.log. It answers more than it looks like, though it points somewhere other than the fit.A re-attach has no fit check. berthd grants it at once, whatever the card holds, because the engine is already on the card. In v0.4.1 it refuses for only three reasons:
--needs-mib,Your log rules out the first one. At 20:23:10 berthd still listed the run's record (
unprotected_run=qwen-3.8-27b), and it keeps a record only while the PID's start time matches the one recorded. A 5070 Ti is not an APU, and nothing else held a lease for pid 61524. So I don't think the wrapper was refused. I think it never asked.The six later
berth runstarts point the same way. llama-swap starts a new process for a model only after it sees the old one end. On Windows, an engine outlives itsberth runon purpose. Its job object has no kill-on-close, so a dead wrapper never takes a serving model down with it. My reading, which is inferred and not reproduced, is that theberth runfor pid 61524 ended at or soon after 20:22:53. Its llama-server kept serving, with nothing left to re-attach it. I don't know yet what ended it.#306 (merged into
main, in the next release) makes this visible without the wrapper's stderr. The unprotected-run record now names theberth runthat wrote it. The squatter line logsberth_run=exitedorberth_run=alive, and a request refused behind its own orphan adds one of these:its berth run (pid N) has exited, so nothing will re-attach itits berth run (pid N) is alive and has not re-attachedIf you can, try this once on your current build. With a model loaded through the bridge, restart berthd the way the reinstall did, then check in Task Manager whether that model's
berth.exe runprocess is still there. If it is gone, something ended it, either the reinstall itself or llama-swap, and that is what to find next. If it is still there and the model stays a squatter, that is a bug on my side and I want the details.One more thing from your log. The run declared
--needs-mib 14000and held 14,579 MiB. That did not cause the orphan, but Berth reserves the declared size and grants other engines around it, so a run that grows past it can push a neighbour out of memory.--needs-mib 14700matches what that model really holds.azrael-229 commentedon Nov 12, 2026
This is a head scratcher for me. As in both how to set it up for myself and how Berth should handle it. In Windows and non-headless environment when using the VRAM to the very limit like I do (I usually have only 300MB or less free) I am either hitting the situation above or I am hitting unending refusal from Berth when I know the model will load (the desktop will be evicted, some memory will be reclaimed by defragmentation). This is really a matter of how NVidia driver manages memory, not Berth issue and this is more general for using LLMs or ML loads in general. Sometimes even 1-2 retries will make something fit. Computing shouldn't work like this but here we are.
I will test it as soon as I can get some free GPU time. Probably within a day or two.
T-Crypt commentedon Nov 12, 2026
Thanks, no rush. v0.5.1-rc.1 has #306, so if you run the restart test on it, berthd's log says
berth_run=exitedorberth_run=aliveon the squatter line, and Task Manager is only a cross-check. It is a pre-release, soberth updatewill not offer it: download it from the release page.On sizing at the edge of the card: you are right that the driver decides, and that a typed
--needs-mibis a guess you have to keep up to date. Berth already measures what each run holds. The next step I want is for a run with no--needs-mibto be sized from what that model held last time, so you never type the number. I'll open an issue for it with your case in it.T-Crypt commentedon Nov 14, 2026
v0.6.0-rc.1 is out and has two changes for the restart test:
min_turn.events.jsonlacross restarts, and lease IDs no longer start again at 1 (feat(events): keep the event stream across restarts, and berth why ID #330). After a restart,berth why ENGINEprints that engine's last lease from the request to the re-attach, with each wait and drain reason, and names the lease it re-attached as.If you run the restart test on rc.1,
berth whyfor the model from that moment would show whether the re-attach still misses the window. I'm leaving this open for that. No rush.