diff --git a/CHANGELOG.md b/CHANGELOG.md
index 847f0887..c4fb98a2 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -6,6 +6,135 @@ has a matching GitHub release whose body comes from its section below.
## [Unreleased]
+## [v0.6.0] - 2026-11-14
+
+v0.6.0 gives each card a schedule and makes Berth explain its choices. `berth schedule` books an engine's hours,
+a priority change or downtime up to 14 days ahead, agents book their own runs through the gateway, and the
+harbor's SCHEDULE tab shows the week. `berth train` runs a training that nothing automated can interrupt.
+`berth fit`, `berth contract`, `berth queue` and `berth why` say where a request would run, what an engine costs,
+who is next and why a lease waited or was drained. The event log now survives restarts. It was tagged first as
+`v0.6.0-rc.1`.
+
+### Added
+
+- `berth why ID` tells one lease's story in plain words: when it asked for the card, each reason it waited, the
+ grant, each drain and who it was for, the release, and the berthd restarts it lived through. `berth why ENGINE`
+ reads that engine's newest lease. A lease berthd still holds ends with where it stands now, and one a restart
+ cut off names the lease its engine re-attached as.
+- berthd keeps its event stream in `events.jsonl` in the state directory (`--events`, `""` for memory only). A
+ restart reads the last 1000 events back, goes on numbering after them, and goes on with the lease numbers, so
+ `berth events --since N` and `berth why` read across restarts and deploys. The file moves to `events.jsonl.1`
+ at 8 MiB. Each start is a `node.started` event that names the version.
+
+- A schedule for each card, up to 14 days ahead. `berth schedule` shows what holds each card, the waiters in grant
+ order and every booking. `berth schedule book` books an engine's run (it goes first, runs at the booking's
+ priority, and keeps the card against everything but a hold), a priority change for set hours, or downtime (no
+ new work, holders drain at the start). `move` and `cancel` change a booking, and `--days` and `--earliest`
+ repeat one or find free hours. Hours nobody booked run by priority, then arrival, as before. berthd keeps the
+ bookings in `schedule.json`.
+- The gateway's `GET`, `POST`, `PATCH` and `DELETE /v1/berth/schedule` let agents and devices read the scheduler
+ and book, move or cancel their own runs. Downtime and priority changes stay the operator's.
+- The harbor's SCHEDULE tab: what runs and waits now, a 7-day board of 24-hour tracks with every booking to the
+ minute, each engine's window and the live holder, and a list of bookings. A paired device books by clicking an
+ empty hour and moves or cancels a booking by clicking it. Each booking form says what the booking will do before
+ you save it.
+
+- `berth train --as NAME [--gpu CARDS] -- CMD` runs a training or fine-tune that nothing automated can interrupt.
+ It takes the whole of each card it names, waits for them with no deadline, and keeps them until the command
+ exits. While it holds a card or waits for it, berthd refuses every other request on that card at once: gateway
+ requests, `berth run`, llama-swap models, scheduled jobs and `berth hold`. Clients get the refusal right away
+ instead of waiting out their timeout. A second `berth train` waits in line and starts when the first one ends.
+ Other cards keep serving. `berth train` refuses to start without berthd, and `berth status` shows the lease as
+ `hold,pinned,training`.
+- `berth train --exclusive` also locks the card in the driver on Linux with an NVIDIA card. It puts the granted cards
+ in exclusive-process compute mode for the run's life, so a program that opens CUDA without Berth cannot open
+ them, and puts them back to the default mode when the run ends. It needs a sudo rule for `nvidia-smi -c`, and
+ refuses to start without one, or when another compute process is already on the card.
+- A `berth train` run keeps its lock while berthd is away. With berthd down, `berth run` refuses to start an engine
+ while a training runs on the node, instead of starting it without a lease. A berthd that starts while a training
+ runs refuses every other request until the training re-attaches, with no time limit, where other engines get
+ 15 seconds.
+
+- On Windows, `berth notify` is a tray icon in the taskbar. Its notifications are Windows' own toasts, and a click
+ opens the harbor. The tooltip shows the latest event, and keeps it after explorer restarts. The menu opens the
+ harbor, turns the toasts off and on, lists the last 10 events and quits. `berth service install --notify`
+ starts it at logon for the current user ([#186](https://github.com/T-Crypt/berth/pull/186)).
+- `POST /v1/berth/yield` lets a ComfyUI node that calls an LLM in the middle of a job give the card to that LLM
+ and take it back. Before, the LLM's drain of ComfyUI waited for the job to end, the job waited for the LLM, and
+ neither moved until the node's call timed out. The node frees ComfyUI's models in-process and yields. The drain
+ then parks ComfyUI as it would between jobs, the LLM runs, and `{"end":true}` answers once ComfyUI holds the card
+ again. A yield runs out after its seconds (300 by default), so a node that dies mid-yield cannot leave it on
+ ([#299](https://github.com/T-Crypt/berth/issues/299)).
+- `learn_size = true` on an engine entry, or `berth run --learn-size`, sizes the lease from the most that engine
+ has been measured holding plus 512 MiB, in place of the whole card. Engines that fit together then share the card
+ without anyone typing `needs_mib` or keeping it up to date. An engine not yet measured takes the whole card, and
+ the grant line in `berth events` names the size it used
+ ([#311](https://github.com/T-Crypt/berth/issues/311)).
+- `berth status` has a SCHEDULE table and the harbor a SCHEDULES panel. Each lists every engine that has a
+ `window`, with its hours, whether they are open, and how long until they close or open. An engine shows before its
+ window opens, and a `berth run --window` lease shows its own hours. The status JSON carries the same rows as
+ `schedules`, and each lease carries its `window`.
+- `berth contract ENGINE` prints one engine's terms in one place: its priority, the VRAM a lease asks for, the last
+ run's measured peak, the RAM it needs and holds parked, its minimum turn and window, and how long a switch to it
+ takes from each tier. Each number says whether berth.toml declares it, Berth's default applies, Berth measured
+ it, or nothing is known. ENGINE can be an `[[engine]]` name, a discovered engine, or a llama-swap model's
+ `berth run --as` name. Without berthd it prints the declared terms. The status JSON's `footprints` now carry each
+ engine's `hot_peak_vram_bytes`, `ram_peak_bytes` and `parked_ram_bytes`. A llama-swap model whose last-run
+ figure sits on its group's entry is now listed under its own name too. A llama-swap model has no berth.toml
+ entry, so its priority and minimum turn come from its last run's lease, which carried the flags on its
+ `berth run` line. Its last peak is the most a run under its name held. The footprint entry carries them as
+ `priority` and `min_turn_s`.
+- `berth fit NAME` says where a request for an engine or a model would run right now and what the start costs. It
+ ranks each candidate by path: already running on the card, a resume from RAM, a free card, a drain of the holder,
+ or blocked by a VM, a cordon or a pinned lease. Each candidate names its reason, the tier its weights sit in, and
+ the switch time the gateway measured from that tier, or unknown. For a model, every engine that serves it is
+ ranked the way the gateway picks one. A model that no `[[engine]]` names or serves, such as a llama-swap model, is
+ found the way the gateway finds it, on a live engine whose `/v1/models` lists it. `--explain` lists every
+ candidate. It is read only, and the gateway now picks through the same ranking.
+- `berth queue` lists the waiters in the order berthd will grant them, with their lease IDs. `berth queue bump ID`
+ moves one to the front of its priority class, `berth queue prio ID batch|interactive|foreground` moves it to
+ another class, and `berth queue drop ID` refuses it now. Holds and trainings cannot be changed this way. Each
+ change is a `lease.queue_override` event.
+- The harbor's SWITCH tab has a QUEUE panel. It lists the waiters in grant order, and a paired device gets FIRST, a
+ priority picker and DROP on each one. These buttons go from the harbor straight to berthd, never through the
+ gateway, because agents hold the gateway's API key and a queue change is the operator's to make.
+- A gateway request whose start times out (`not_ready`) while its lease is still in line for the card says where
+ it stands. The 503 carries `X-Berth-Queue-Position` and, when berthd has measured the starts ahead,
+ `X-Berth-Start-Estimate: N; lower-bound`. `Retry-After` is the larger of the estimate and 30 s. The estimate
+ leaves out the holder's time left, so it is a minimum, not a promise.
+- `berth wire llama-swap` names each llama-swap group where llama-swap swaps two or more wired models itself.
+ There llama-swap unloads the old model before Berth sees the request, so `min_turn`, the engine ranking and the
+ juggle guard never apply between them. The JSON carries the groups as `swapping`. The llama-swap section of
+ contrib/INTEGRATION.md shows the `swap: false` group and the `healthCheckTimeout` that let Berth decide the turns.
+
+### Fixed
+
+- A drain that arrives while a parked engine is still loading back no longer stops it. `berth run` now waits for
+ the resume to finish, then parks or drains as usual. Before, the park call reached the engine in the middle of
+ its load, the engine answered 409, and the run fell back to stopping it. That cold-started the LLM on its next
+ request and could leave it marked unresponsive ([#275](https://github.com/T-Crypt/berth/issues/275)).
+- The swap bench (`contrib/bench/swap`) no longer reports `OUTPUT IDENTICAL yes` for thinking models that ran out
+ of tokens before answering. It now compares the reasoning a model streams as well as its answer, prints
+ `n/a (empty)` when every run came back empty, asks for 256 tokens by default (`-max-tokens`), and can ask the
+ model to skip its thinking (`-no-think`).
+- A subcommand's help and error hints name its group. `berth engine show --help` printed `berth show`, which is no
+ command, and a bad flag said to run `berth show -h`.
+- A berthd restart no longer gives every running engine a fresh minimum turn. `berth run` now tells the new berthd
+ how long its engine had held the card, and the re-attached lease goes on with the turn it was in. Before, each
+ deploy reset the TURN column to the full `min_turn` and kept waiters out for that long again.
+- `berth wire llama-swap` now finds a `berth run` line with a flag written before `--as`, such as
+ `berth run --min-turn 5m --as qwen ...`. Before, it took the model for unwired, and `--write` added a second
+ `berth run` in front of the first.
+- `berth wire llama-swap --write --update` keeps the words before the berth binary on a `berth run` line, such as
+ a wrapper script or `env FOO=1`. Before, it rewrote the line from the berth binary on and dropped them.
+
+- A model that fills the card no longer loads beside a parked engine it cannot fit next to. Berth granted any
+ whole-card engine the card beside a parked residual under `residual_max_gib`, so a 23.4 GiB model on a 24 GiB
+ card started beside a kept-up ComfyUI's 768 MiB context and failed to load. Berth now remembers the most each
+ engine has held, by its own name inside a llama-swap group, and evicts the parked engine first when that, plus
+ 512 MiB the load needs past its measured peak, does not fit beside it. A smaller model still loads beside it,
+ and an engine never measured is granted as before.
+
### Changed
- An engine left without a lease after berthd restarts now tells you whether its `berth run` can still re-attach
diff --git a/README.md b/README.md
index 398c10b4..c23ae6da 100644
--- a/README.md
+++ b/README.md
@@ -29,7 +29,7 @@ berth status # who holds the card, who
## How it works
-
+
Every engine asks berthd for the card: through `berth run`, the gateway, ComfyUI's API or a game
claim. When another engine holds it, Berth:
@@ -42,7 +42,7 @@ claim. When another engine holds it, Berth:
The numbered markers on the timeline are those four steps. A process that holds VRAM without a lease
shows up as a stowaway: Berth reports its PID and memory and leaves it running.
-> **Status: alpha, v0.5.0.** Runs daily on an RTX 4090 workstation beside llama-swap, NInfer,
+> **Status: alpha, v0.6.0.** Runs daily on an RTX 4090 workstation beside llama-swap, NInfer,
> Strata, ComfyUI and games, and testers run it on Windows. The [docs](docs/README.md) cover what works and has tests; the
> [roadmap](docs/roadmap.md) lists the phases and what is still missing.
@@ -110,8 +110,21 @@ line goes away.
render farm job, a cracking session, by name, command line or launcher.
- **`berth hold`** frees the card for a call or a game you start yourself. Ctrl-C gives it back. On a node with
several cards, `--gpu` names the ones it takes.
-- **`berth run --pinned`** for an app or a training run that must never be stopped mid-work, on one card or
- several. See [Training runs](docs/leasing.md#training-runs).
+- **`berth train`** runs a training or fine-tune that nothing automated can interrupt, on one card or several.
+ It takes the whole of each card, and berthd refuses every other request on those cards until it exits, so
+ agents and scheduled jobs get an answer at once instead of waiting. A second training waits in line. See
+ [Training runs](docs/leasing.md#training-runs).
+- **`berth run --pinned`** for an app that must never be stopped mid-work but may share the card, such as Blender.
+
+**Plans the card's hours**
+- **A schedule per card.** `berth schedule book` gives an engine the card for set hours, changes its priority for
+ them, or books downtime, up to 14 days ahead. A booked run goes first and keeps the card against everything but
+ a hold. Hours nobody booked run by priority, then arrival. See [Book the card](docs/leasing.md#book-the-card).
+- **Agents book their own runs** through `/v1/berth/schedule` on the gateway, and the harbor's SCHEDULE tab shows
+ the week to the minute. A paired phone books by clicking an empty hour.
+- **`berth queue`** lists the waiters in grant order. `bump`, `prio` and `drop` change one by hand.
+- **`learn_size = true`** sizes a lease from what the engine held last time, so engines that fit together share
+ the card without anyone typing `needs_mib`.
**One API for your engines** *(preview)*
- **`berth gateway`**: one OpenAI-compatible port. Name a model and Berth starts its engine,
@@ -158,6 +171,10 @@ line goes away.
leaderboard; 30 days of history on disk.
- **The harbor web UI** (`berth harbor`): a departures board of leases, a collisions-avoided
counter, charts and the event log. Read-only, loopback by default.
+- **Explains itself.** `berth why ENGINE` tells that engine's last lease in plain words: each wait and its
+ reason, the grant, each drain and who it was for, and the restarts it lived through. `berth fit MODEL` says where
+ a request would run now and what the start costs, and `berth contract ENGINE` prints an engine's VRAM, RAM, turn
+ and switch times, each marked declared, measured or unknown. The event log is kept on disk across restarts.
- **Alerts** to the desktop or a webhook, `/metrics` for Prometheus, and `berth export` for
CSV or JSON.
@@ -253,8 +270,8 @@ which `berth service status` prints.
## Preview
-The first two screenshots show the harbor web UI with its built-in demo data, on the CARD tab and on
-the Engines tab in advanced mode. The third shows the CARD tab of a live node on an iPhone.
+The screenshots show the harbor web UI with its built-in demo data: the CARD tab, the Engines tab in advanced
+mode, and the CARD tab at phone width.
[The harbor and telemetry](docs/harbor.md) explains every panel and shows how to add the harbor to an
iPhone's Home Screen.
@@ -262,7 +279,7 @@ iPhone's Home Screen.
-
+
## Install
@@ -328,6 +345,25 @@ request to the reply, lease included (2026-10-23):
Strata's numbers and the method are in [Benchmarks](docs/benchmarks.md#swap-numbers).
+## A month on the reference workstation
+
+berthd counts every handoff it makes, and the harbor's collisions board shows the counts. These are the counts
+from one RTX 4090 workstation between 2026-10-12 and 2026-11-13, running llama-swap, NInfer, Strata, ComfyUI,
+coding agents and games:
+
+| What Berth did | Count |
+|---|---|
+| Requests that waited for the card instead of loading on top of the engine holding it | 1,136 |
+| Engines drained at a reply or job boundary to make room | 901 |
+| VRAM handed from one engine to the next | 23.9 TiB |
+| Processes holding VRAM without a lease, reported and left running | 573 |
+| Swaps between two engines of equal priority that the ranking stopped | 57 |
+| Releases that stalled and were caught before the card stayed full | 4 |
+| Requests refused after their wait ran out | 24 |
+
+In the last 7 days, 23 engines and jobs held the card for 143 lease-hours. Leases that share the card count their
+own hours each.
+
## Roadmap
The phases, the shipped releases and what comes next are in the [roadmap](docs/roadmap.md).
diff --git a/assets/how-it-works.svg b/assets/how-it-works.svg
index 59797e9a..0eeaa64f 100644
--- a/assets/how-it-works.svg
+++ b/assets/how-it-works.svg
@@ -1,6 +1,6 @@
How berth works
-Clients ask berth for the GPU. Agents such as Hermes and ComfyUI workflows go through the berth gateway, one endpoint that starts engines on demand; engines under berth run, games and a VM passage (berth vfio run) ask directly. berthd queues them by priority, drains the holder at its boundary (saving KV caches or parking it in RAM), waits for the driver to show the memory free, then grants the lease. gpu0 serves strata while gpu1 is passed through to a VM. A timeline of one 24 GiB card shows llama-swap, strata, a game, a VM passage and ComfyUI taking turns with no overlap.
+Clients ask berth for the GPU. Agents such as Hermes and ComfyUI workflows go through the berth gateway, one endpoint (and one per card) that starts engines on demand; engines under berth run, games, training runs under berth train and a VM passage (berth vfio run) ask directly. A training run locks its cards, and every other request there is refused at once. berthd queues them by priority, drains the holder at its boundary (saving KV caches or parking it in RAM), waits for the driver to show the memory free, then grants the lease. gpu0 serves strata while gpu1 is passed through to a VM. A timeline of one 24 GiB card shows llama-swap, strata, a game, a VM passage and ComfyUI taking turns with no overlap.
@@ -26,7 +26,7 @@
WHO ASKS
berth gateway
-:9799 · one endpoint
+:9799 · /cards/gpuN
Agents · Hermes · chat apps
POST /v1/chat/completions
@@ -44,8 +44,8 @@
-A game launches
-claim → hold the card
+A game or a training run
+claim or berth train
diff --git a/assets/preview-harbor.png b/assets/preview-harbor.png
index 9f01a8c6..7e855ae1 100644
Binary files a/assets/preview-harbor.png and b/assets/preview-harbor.png differ
diff --git a/assets/preview-mobile.png b/assets/preview-mobile.png
index 3045166f..a2d45d79 100644
Binary files a/assets/preview-mobile.png and b/assets/preview-mobile.png differ
diff --git a/assets/preview-requests.png b/assets/preview-requests.png
index 99dcd446..6a865997 100644
Binary files a/assets/preview-requests.png and b/assets/preview-requests.png differ
diff --git a/assets/social-preview.png b/assets/social-preview.png
new file mode 100644
index 00000000..6d373f8a
Binary files /dev/null and b/assets/social-preview.png differ
diff --git a/contrib/INTEGRATION.md b/contrib/INTEGRATION.md
index 5df7c39e..344c3bb6 100644
--- a/contrib/INTEGRATION.md
+++ b/contrib/INTEGRATION.md
@@ -69,7 +69,30 @@ models:
```
llama-swap stops a model with SIGTERM; `berth run` forwards it to the engine's process group.
-While Berth waits for the card, llama-swap's `healthCheckTimeout` keeps the request open.
+
+By default llama-swap runs one model at a time and unloads the old model itself when a request names
+another. Berth sees that as the old model giving the card back, so `min_turn`, the engine ranking and
+the juggle guard never decide between two llama-swap models. Two clients on two models then swap the
+card on every request. Put the wired models in a group with `swap: false` so a request for another
+model waits for a Berth lease instead, and the old model unloads through its drain once Berth says so:
+
+```yaml
+healthCheckTimeout: 600 # longer than min_turn plus the slowest model's load
+routing:
+ router:
+ use: group
+ settings:
+ groups:
+ "chat":
+ swap: false
+ exclusive: false
+ members: ["qwen3-30b", "gemma-27b"]
+```
+
+While a model waits for the card, llama-swap's `healthCheckTimeout` keeps the request open. If the
+timeout is shorter than the wait, llama-swap gives up on the start and the request fails. `berth wire
+llama-swap` names every group where llama-swap still swaps wired models itself. Models whose learned
+size fits beside each other (`learn_size`) now share the card instead of swapping.
Reading: `type = "llama-swap"` follows every running model's upstream metrics, which exist only
when the model's `cmd` starts `llama-server` with `--metrics`; without it the engine stays up and
the harbor shows n/a for decode, prefill and token totals.
diff --git a/contrib/bench/README.md b/contrib/bench/README.md
index 637a6ce2..403e9de8 100644
--- a/contrib/bench/README.md
+++ b/contrib/bench/README.md
@@ -45,6 +45,8 @@ in its usual place. The full flag set:
-json print the raw samples as JSON
-key string the gateway's API key file
-leak int run N A-B cycles and print VRAM after drain and the gateway RSS at cycle 1 and N
+ -max-tokens int the max_tokens of each request (default 256)
+ -no-think send chat_template_kwargs enable_thinking=false, which llama.cpp-style servers honour
```
With `-comfy WORKFLOW.json` the B slot becomes a ComfyUI workflow posted
@@ -97,7 +99,11 @@ model-b 15.10 s 0.2 GiB 40.1 GiB yes
- `OUTPUT IDENTICAL` says whether every run of that engine produced the
same greedy answer. A parked resume that is not bit-identical to the
fresh run is not the same model state, and this column is how you see
- it.
+ it. The answer and the reasoning a thinking model streams first
+ (`reasoning_content` or `reasoning`) both count. When every run of an
+ engine came back empty, the column reads `n/a (empty)`: equal empty
+ strings prove nothing. Give a thinking model room with `-max-tokens`, or
+ ask it to skip the thinking with `-no-think`.
The parked tier needs no separate mode. When park is on for an engine,
its warm switches go through the park and resume path, and the warm row
diff --git a/contrib/bench/swap/main.go b/contrib/bench/swap/main.go
index 7e88885c..b7b9ab0a 100644
--- a/contrib/bench/swap/main.go
+++ b/contrib/bench/swap/main.go
@@ -45,7 +45,7 @@ import (
)
// prompt is the fixed greedy prompt of every request: short enough that
-// 32 tokens finish it, specific enough that a broken resume is obvious.
+// a short answer finishes it, specific enough that a broken resume is obvious.
const prompt = "Reply with exactly one line: swap bench ok"
// requestTimeout bounds one request. A cold start may wait ten minutes
@@ -65,6 +65,8 @@ type flags struct {
gpuBackend string
keyFile string
jsonOut bool
+ maxTokens int
+ noThink bool
}
func parseFlags() (flags, error) {
@@ -80,6 +82,8 @@ func parseFlags() (flags, error) {
gpuBackend := fs.String("gpu-backend", "auto", "GPU probe: auto, nvidia or amd (AMD index: DRM cardN)")
key := fs.String("key", "", "the gateway's API key file (default: the berth api.key)")
jsonOut := fs.Bool("json", false, "print the raw samples as JSON")
+ maxTokens := fs.Int("max-tokens", 256, "the max_tokens of each request; a thinking model needs room to finish thinking")
+ noThink := fs.Bool("no-think", false, "ask the model to answer without thinking (chat_template_kwargs enable_thinking=false, where the server honours it)")
fs.Usage = func() {
usage(fs)
}
@@ -89,7 +93,7 @@ func parseFlags() (flags, error) {
}
return flags{}, err
}
- f := flags{*gateway, *a, *b, *cycles, *comfy, *leak, *gwPID, *gpuIndex, *gpuBackend, *key, *jsonOut}
+ f := flags{*gateway, *a, *b, *cycles, *comfy, *leak, *gwPID, *gpuIndex, *gpuBackend, *key, *jsonOut, *maxTokens, *noThink}
if err := gpu.ValidateSelection(f.gpuBackend, f.gpuIndex); err != nil {
return f, err
}
@@ -301,13 +305,13 @@ func (r *run) request(ctx context.Context, model string) error {
s.MemBefore = host.MemAvailable("")
t0 := time.Now()
- res, err := r.post(ctx, "/v1/chat/completions", chatBody(model), t0)
+ res, err := r.post(ctx, "/v1/chat/completions", chatBody(model, r.f.maxTokens, r.f.noThink), t0)
if err != nil {
return err
}
s.TTFT = res.TTFT
s.Total = res.Total
- s.Text = res.Text
+ s.Text, s.Reasoning = res.Text, res.Reasoning
if res.Fail != "" {
return fmt.Errorf("the gateway answered with an error: %s", res.Fail)
}
@@ -322,15 +326,26 @@ func (r *run) request(ctx context.Context, model string) error {
return nil
}
-func chatBody(model string) []byte {
- b, _ := json.Marshal(struct {
+// chatBody is one greedy request. noThink adds the chat-template switch
+// that llama.cpp-style servers read to skip the thinking; without it the
+// request stays plain OpenAI.
+func chatBody(model string, maxTokens int, noThink bool) []byte {
+ type kwargs struct {
+ EnableThinking bool `json:"enable_thinking"`
+ }
+ body := struct {
Model string `json:"model"`
Stream bool `json:"stream"`
Temperature float64 `json:"temperature"`
MaxTokens int `json:"max_tokens"`
Seed int `json:"seed"`
Messages []message `json:"messages"`
- }{model, true, 0, 32, 1, []message{{Role: "user", Content: prompt}}})
+ Kwargs *kwargs `json:"chat_template_kwargs,omitempty"`
+ }{model, true, 0, maxTokens, 1, []message{{Role: "user", Content: prompt}}, nil}
+ if noThink {
+ body.Kwargs = &kwargs{}
+ }
+ b, _ := json.Marshal(body)
return b
}
diff --git a/contrib/bench/swap/measure.go b/contrib/bench/swap/measure.go
index 9c8ae129..630794ed 100644
--- a/contrib/bench/swap/measure.go
+++ b/contrib/bench/swap/measure.go
@@ -35,6 +35,7 @@ type Sample struct {
TTFT time.Duration `json:"ttft"`
Total time.Duration `json:"total"`
Text string `json:"text"`
+ Reasoning string `json:"reasoning,omitempty"`
VramBefore uint64 `json:"vram_before"`
VramAfter uint64 `json:"vram_after"`
MemBefore uint64 `json:"mem_before"`
@@ -60,7 +61,9 @@ type streamResult struct {
TTFT time.Duration
Total time.Duration
Text string
- Fail string
+ // Reasoning is what a thinking model streamed before its answer.
+ Reasoning string
+ Fail string
}
// parseSSE reads an SSE stream into the first-token time and the text it
@@ -114,12 +117,16 @@ func parseSSE(r io.Reader, at func() time.Time, start time.Time) (streamResult,
var ev struct {
Choices []struct {
Delta struct {
- Content string `json:"content"`
+ Content string `json:"content"`
+ ReasoningContent string `json:"reasoning_content"`
+ Reasoning string `json:"reasoning"`
} `json:"delta"`
} `json:"choices"`
}
if json.Unmarshal([]byte(payload), &ev) == nil && len(ev.Choices) > 0 {
- res.Text += ev.Choices[0].Delta.Content
+ d := ev.Choices[0].Delta
+ res.Text += d.Content
+ res.Reasoning += d.ReasoningContent + d.Reasoning
}
}
@@ -181,11 +188,16 @@ func outputsIdentical(ss []Sample) string {
if len(ss) == 0 {
return "n/a"
}
- first := ss[0].Text
- for _, s := range ss[1:] {
- if s.Text != first {
+ first := ss[0]
+ empty := true
+ for _, s := range ss {
+ if s.Text != first.Text || s.Reasoning != first.Reasoning {
return "no"
}
+ empty = empty && s.Text == "" && s.Reasoning == ""
+ }
+ if empty {
+ return "n/a (empty)" // equal empty strings say nothing about a resume
}
return "yes"
}
diff --git a/contrib/bench/swap/swap_test.go b/contrib/bench/swap/swap_test.go
index 74149733..9d5d4816 100644
--- a/contrib/bench/swap/swap_test.go
+++ b/contrib/bench/swap/swap_test.go
@@ -89,6 +89,22 @@ data: [DONE]
clock: clockOf(t0.Add(3*time.Second), t0.Add(6*time.Second)),
want: streamResult{TTFT: 3 * time.Second, Total: 6 * time.Second, Text: "swap"},
},
+ {
+ // A thinking model streams its reasoning before its answer:
+ // llama.cpp, NInfer and Strata as reasoning_content, others as
+ // reasoning. It is a token, and it is kept apart from the answer.
+ name: "reasoning then answer",
+ in: `data: {"choices":[{"delta":{"reasoning_content":"one "}}]}
+
+data: {"choices":[{"delta":{"reasoning":"line"}}]}
+
+data: {"choices":[{"delta":{"content":"ok"}}]}
+
+data: [DONE]
+`,
+ clock: clockOf(t0.Add(2*time.Second), t0.Add(4*time.Second)),
+ want: streamResult{TTFT: 2 * time.Second, Total: 4 * time.Second, Text: "ok", Reasoning: "one line"},
+ },
{
name: "trailing event without blank line or DONE",
in: "data: {\"choices\":[{\"delta\":{\"content\":\"hi\"}}]}",
@@ -235,6 +251,9 @@ func TestOutputsIdentical(t *testing.T) {
{"one", []Sample{{Text: "swap bench ok"}}, "yes"},
{"all the same", []Sample{{Text: "swap bench ok"}, {Text: "swap bench ok"}, {Text: "swap bench ok"}}, "yes"},
{"a resume that drifts", []Sample{{Text: "swap bench ok"}, {Text: "swap bench ok "}}, "no"},
+ {"thinking that drifts", []Sample{{Reasoning: "ok, one line", Text: ""}, {Reasoning: "ok one line", Text: ""}}, "no"},
+ {"the same thinking, no answer yet", []Sample{{Reasoning: "ok, one line"}, {Reasoning: "ok, one line"}}, "yes"},
+ {"empty answers prove nothing", []Sample{{}, {}}, "n/a (empty)"},
}
for _, tt := range tests {
if got := outputsIdentical(tt.samples); got != tt.want {
@@ -265,13 +284,20 @@ func TestChatBody(t *testing.T) {
Seed int `json:"seed"`
Messages []message `json:"messages"`
}
- if err := json.Unmarshal(chatBody("model-x"), &body); err != nil {
+ raw := chatBody("model-x", 256, false)
+ if err := json.Unmarshal(raw, &body); err != nil {
t.Fatalf("chatBody is not JSON: %v", err)
}
- if body.Model != "model-x" || !body.Stream || body.Temperature != 0 || body.MaxTokens != 32 || body.Seed != 1 {
+ if body.Model != "model-x" || !body.Stream || body.Temperature != 0 || body.MaxTokens != 256 || body.Seed != 1 {
t.Errorf("chatBody fields = %+v", body)
}
if len(body.Messages) != 1 || body.Messages[0].Role != "user" || body.Messages[0].Content != prompt {
t.Errorf("chatBody message = %+v", body.Messages)
}
+ if strings.Contains(string(raw), "chat_template_kwargs") {
+ t.Errorf("a plain request carries chat_template_kwargs: %s", raw)
+ }
+ if got := string(chatBody("model-x", 64, true)); !strings.Contains(got, `"chat_template_kwargs":{"enable_thinking":false}`) || !strings.Contains(got, `"max_tokens":64`) {
+ t.Errorf("-no-think body = %s", got)
+ }
}
diff --git a/docs/design.md b/docs/design.md
index 0b17715e..af6359e4 100644
--- a/docs/design.md
+++ b/docs/design.md
@@ -116,7 +116,8 @@ When a waiter does not fit on a card, the scheduler tries the cheapest way to ma
stops at the first that works:
1. **Fits now**: grant.
-2. **Beside a parked residual**: grant beside it.
+2. **Beside a parked residual**: grant beside it, when the most the waiter's engine has held fits beside it.
+ An engine that has filled the card evicts the parked engine instead of loading into a card it cannot fit.
3. **Lend**: a holder that can lend gives the waiter's size on that card, and keeps serving.
4. **Park or checkpoint**: the holder sleeps in RAM, if the RAM budget admits it.
5. **Drain**: the holder stops.
@@ -124,6 +125,12 @@ stops at the first that works:
Priorities, minimum turns, holds and the juggle guard decide whether the holder has to make room at
all; this list only decides how. Each step that fails within the grace falls through to the next.
+A **yield** moves a holder's boundary, not its books. A ComfyUI job that calls an LLM can say that it freed its
+models in-process and is at a safe point for a while ([#299](https://github.com/T-Crypt/berth/issues/299)). A drain
+then parks it without waiting for the job to end. Rule 1 still holds, because the park counts only what the card
+read shows free, and rule 4 still holds, because a park that frees nothing falls back to a stop. The holder's way
+back is a resume, which the end of the yield asks for.
+
## Many machines
Each machine runs one berthd for its cards and one gateway. A node's gateway lists its engines,
diff --git a/docs/gateway.md b/docs/gateway.md
index ce2aea79..6651c1eb 100644
--- a/docs/gateway.md
+++ b/docs/gateway.md
@@ -18,6 +18,13 @@
every retry: for 15 s, doubling to 5 min, the gateway answers at once with the same cause,
and `Retry-After` says when the next start is allowed. A start that answers clears it. A
model listed under `serves` by two engines fails `berth config check`.
+- **Queue position.** A `not_ready` start whose lease is still in line for the card says where it
+ stands. `X-Berth-Queue-Position` is its place in line, 1 for next. `X-Berth-Start-Estimate:
+ 45; lower-bound` is the seconds berthd expects the start to take once the card frees. It counts
+ only the measured starts ahead in the queue and not the holder's time left, so treat it as a
+ minimum. It is missing when nothing was measured. `Retry-After` is then the larger of that
+ estimate and 30 s. A streaming request gets the same error as an SSE event, without these
+ headers.
- **API key.** Bearer token on every route; `berth key` prints it (created on first use, mode
600). Loopback only; browser origins and non-loopback Host names are refused.
@@ -268,8 +275,15 @@ curl -s http://127.0.0.1:9799/v1/berth/switch -H "Authorization: Bearer $(berth
curl -s http://127.0.0.1:9799/v1/berth/stop -H "Authorization: Bearer $(berth key)" -d '{"engine":"strata"}'
curl -s http://127.0.0.1:9799/v1/berth/load -H "Authorization: Bearer $(berth key)" -d '{"engine":"llama-swap","model":"qwen3.6-35b-a3b"}'
curl -s http://127.0.0.1:9799/v1/berth/batch -H "Authorization: Bearer $(berth key)" -d '{"engine":"comfyui","grace":30}'
+curl -s http://127.0.0.1:9799/v1/berth/yield -H "Authorization: Bearer $(berth key)" -d '{"engine":"comfyui","seconds":120}'
```
+- **Yield** is for a ComfyUI node that calls an LLM in the middle of a job (see [leasing](leasing.md), "A yield").
+ `{"engine":"comfyui","seconds":120}` answers `200` once berthd has told ComfyUI's run, and the yield lasts the
+ seconds given (absent: 300, at most 3600). `{"engine":"comfyui","end":true}` ends it and answers `200` only once
+ ComfyUI holds the card again. An engine with no lease is a `409`, and so is one whose lease covers several cards,
+ since that lease cannot park yet. An engine not in `berth.toml` is a `404`.
+
- **Batch** is `berth batch` on the network (see [leasing](leasing.md), "A batch"): `202` once berthd has the batch,
and an engine with a `start` command is switched to as **Switch** does, so one call comes before the jobs.
`"grace"` and `"max"` are seconds (absent: 30 and 14400). `{"engine":"comfyui","end":true}` ends it, and a `409`
@@ -295,6 +309,34 @@ on the node (`card`, such as `gpu1`), its base URL on this gateway (`base`, `/ca
`berth cordon` closed it, and the holders on that card alone. Each engine names the cards its entry runs on
(`cards`), so a board for several cards can put each engine under its card.
+## Book the card from an agent
+
+An agent, a timer or another device books its own run on the card's schedule (read
+[Book the card](leasing.md#book-the-card)) with the node's key:
+
+```sh
+K="Authorization: Bearer $(berth key)"
+curl -s http://127.0.0.1:9799/v1/berth/schedule -H "$K"
+curl -s http://127.0.0.1:9799/v1/berth/schedule -H "$K" \
+ -d '{"who":"pi-pocket","engine":"pocket-train","start":"23:00","for":"5h","priority":"batch","days":7}'
+curl -s -X PATCH http://127.0.0.1:9799/v1/berth/schedule/b4 -H "$K" -d '{"start":"01:00"}'
+curl -s -X DELETE http://127.0.0.1:9799/v1/berth/schedule/b4 -H "$K"
+```
+
+- `GET` is the scheduler in one read: `now`, `running` (the leases that hold a card), `queued` (the waiters in
+ grant order) and `bookings` (every booking that has not ended, by start). A lease that a booking governs names it
+ in `booking`.
+- `POST` books a run and answers `201` with `bookings`, the bookings it made. `start` takes what
+ `berth schedule --at` takes, and `for` is a length such as `5h` or `90m`. `days`, `earliest`, `card`, `priority`
+ and `note` are optional. A clash or an invalid booking is a `409` with the reason, such as
+ `run Sat 23:30-Sun 00:30 clashes with b1 (run for pi-pocket, Sat 23:00-Sun 04:00)`.
+- `PATCH /v1/berth/schedule/ID` moves a run. It takes `start`, `for`, `priority` and `note`, keeps the rest and
+ keeps the ID. `DELETE` cancels a run. Both answer `200` with the booking.
+
+The gateway books, moves and cancels runs only. A priority change or downtime acts on everyone else's work, so it
+is the operator's: `berth schedule`, or a paired device in the harbor. `PATCH` or `DELETE` on one is a `403`. Any
+holder of the key can change any run; per-device ownership comes with device keys.
+
## One URL per card
On a node with several cards, each card has its own base URL on the gateway: `/cards/CARD/v1`, where `CARD` is
diff --git a/docs/harbor.md b/docs/harbor.md
index 7e25dd07..6aa59430 100644
--- a/docs/harbor.md
+++ b/docs/harbor.md
@@ -2,10 +2,9 @@
## Preview
-The harbor UI (basic mode, served from the built-in demo fixture - every number on the screen
-comes from `demo.json`). The tide shows the card's memory by holder over six hours; the
-hatched stretch is a stowaway the fixture replays: a model that ran without a lease after a
-daemon restart, reported and left alone.
+The harbor UI in basic mode, served from the built-in demo fixture. Every number on the screen comes from
+`demo.json`. The tide shows the card's memory by holder over the last hour. The hatched stretch is a stowaway the
+fixture replays: a model that ran without a lease after a daemon restart, which Berth reported and left alone.
@@ -15,10 +14,10 @@ decode throughput, the token ledger, the per-model leaderboard, storage and resi
-On a phone, the harbor fits the screen: the tabs move to a bar at the bottom and the panels
-stack. This is the CARD tab on an iPhone, read from a live node on the reference workstation.
+On a phone, the harbor fits the screen. The tabs move to a bar at the bottom and the panels stack. This is the
+CARD tab at phone width, from the same demo fixture.
-
+
### The harbor on your iPhone's Home Screen
@@ -66,7 +65,11 @@ for a phone.
first-seen time; optional desktop notification and webhook. Report-only by design.
- **Contested grants:** a grant made while stowaways hold memory is flagged, not hidden.
- **Event log:** every request, wait, drain, release, free, stuck lease, stowaway and
- probe change, with gap-free sequence numbers; `berth events -f` follows it live.
+ probe change, with gap-free sequence numbers; `berth events -f` follows it live. berthd keeps the
+ stream in `~/.local/state/berth/events.jsonl` (`--events`), so the numbers and the last 1000 events carry
+ on across a restart. `berth why ID` tells one lease's part of it in plain words: why it waited, the
+ grant, each drain and who it was for, and the restarts it lived through. `berth why ENGINE` reads that
+ engine's newest lease.
- **The collisions board:** collisions avoided, VRAM handed off, drains, stuck releases
caught, stowaways caught, contested grants, refusals. Each count links back to a
recorded event. berthd keeps the board in `~/.local/state/berth/counters.json`
@@ -122,8 +125,11 @@ for a phone.
adds every switch, drain, park, resume and VM handoff. A burst of one event shows once. On a machine with several cards the title leads with
the card (`gpu1: GPU passed to a VM`) and each card's events are gated apart.
Set it in `berth.toml` (`[notify] enabled = false` turns it off) or with `--level`.
- `berth service install --notify` runs it as a user unit. It is Linux only for now and
- uses `notify-send`, so any desktop or notification daemon shows it.
+ On Linux it uses `notify-send`, so any desktop or notification daemon shows it, and
+ `berth service install --notify` runs it as a user unit. On Windows it is a tray icon in
+ the taskbar: its notifications are Windows' own toasts (a click opens the harbor), its
+ tooltip is the latest event, and its menu opens the harbor, turns the toasts off and on,
+ lists the last 10 events and quits. `berth service install --notify` starts it at logon.
## The harbor's views
Everything the node collects shows in one of four tabs, and every panel answers a
diff --git a/docs/interfaces.md b/docs/interfaces.md
index f5f5dbe6..feab98ad 100644
--- a/docs/interfaces.md
+++ b/docs/interfaces.md
@@ -2,7 +2,7 @@
- **CLI:** one command tree with one output contract. `berth status`, `berth events -f`,
`berth run`, `berth hold`, `berth stop`, `berth cordon|uncordon`, `berth notify`, `berth claims`, `berth pull`, `berth verify`, `berth models`, `berth gateway`,
- `berth wire llama-swap`, `berth engine check`, `berth config check`, `berth config show`,
+ `berth wire llama-swap`, `berth engine check`, `berth contract ENGINE`, `berth fit NAME`, `berth queue`, `berth why ID|ENGINE`, `berth config check`, `berth config show`,
`berth export`, `berth key`, `berth skill install|show`, `berth update`, `berth version`,
and for the node itself `berth daemon`, `berth harbor`, `berth service install|uninstall|status`,
`berth reclaim`, `berth resume`, `berth prefetch`, `berth storage`, `berth civitai search|versions`
diff --git a/docs/leasing.md b/docs/leasing.md
index 59b2a099..84e50d91 100644
--- a/docs/leasing.md
+++ b/docs/leasing.md
@@ -31,7 +31,11 @@
Windows, Ctrl-C, Ctrl-Break, closing the console window and logging off do the same, and the engine's drain
has to finish in the few seconds Windows allows after a close. `berth gateway` starts each run in its own
process group on Windows, so Ctrl-C in the gateway's console does not drain engines it meant to leave running.
-- **Whole-card or sized leases.** `--needs-mib N` lets engines that fit share a card.
+- **Whole-card or sized leases.** `--needs-mib N` lets engines that fit share a card. `--learn-size`, or
+ `learn_size = true` on the engine entry, sizes the lease for you. It takes the most the engine has been measured
+ holding plus 512 MiB for its load, so you never type or update a number. The first run takes the whole card,
+ because nothing is measured yet. The grant in `berth events` says which size it used. A run that grows past its
+ size still counts at what the driver measures, so a waiter never gets memory the engine holds.
- **Several cards.** One berthd governs every card by default. A foreground request on one card never drains a
holder on another. `berth run --gpu gpu1` puts a lease on one card, and `--gpu gpu0,gpu1` holds one lease on
both (see [Training runs](#training-runs)). A running game holds every card, because Berth cannot tell which
@@ -88,7 +92,7 @@
minimum turn, 3m12s left"), and `berth status` shows the time a lease has left in the
TURN column. When the turn ends the normal rules apply: a drain still starts at the slot
or job boundary, and a minimum turn never cuts a request off. After a daemon restart, a
- re-attach starts a fresh turn.
+ re-attach goes on with the turn it was in.
- **A batch.** A minimum turn is a fixed time; a batch lasts as long as the work. Before an
agent or a script queues many jobs on one engine, `berth batch comfyui` (or
@@ -104,6 +108,23 @@
A batch is not the `batch` priority. That one makes an engine wait behind everything
else; this one keeps an engine on the card while its queue has work.
+ - **A yield.** A drain of ComfyUI waits for the job boundary, so a job that calls an LLM
+ in the middle can deadlock: the job waits on the LLM, the LLM waits for the card, and the
+ drain waits on the job ([#299](https://github.com/T-Crypt/berth/issues/299)). A node in
+ that job can yield the card instead:
+
+ 1. It frees ComfyUI's models in-process, which `/free` cannot do while a job runs.
+ 2. It calls `POST /v1/berth/yield` with `{"engine":"comfyui","seconds":120}`. For that
+ long, a drain treats the job as at its boundary, so the LLM's drain parks ComfyUI as
+ it would between jobs. The card read after the park still decides what is free.
+ 3. It makes its LLM call. The LLM is granted in the room the park left.
+ 4. It calls `POST /v1/berth/yield` with `{"engine":"comfyui","end":true}`, which resumes
+ ComfyUI and answers once it holds the card again. The next node loads its models.
+
+ A yield only ends the wait for the job boundary. A node that yields without freeing
+ its models gets the usual fallback: the park does not free the card within the grace,
+ and the drain stops ComfyUI. A yield that is never ended runs out after its seconds.
+
## Games, apps and holds
- **Claims cover work Berth does not start.** berthd scans the process table every 2 s.
A *hold* claim keeps the card while its process runs: engines are drained and wait, with a
@@ -142,6 +163,67 @@
- A hold cannot free memory held by something outside Berth: that memory is still reported,
as contested.
+## Reorder the queue
+
+Berth grants waiters by priority first, then in the order they arrived. Many devices can send work to one
+card and nobody has to sort it out. When you want a different order, `berth queue` changes it:
+
+```sh
+berth queue # the waiters in grant order, with their lease IDs
+berth queue bump llama-swap-41 # first in its priority class
+berth queue prio comfyui-12 foreground
+berth queue drop tagger-7 # refuse it now, as a timeout would
+```
+
+A bump puts the waiter ahead of everyone else in its priority class. To go ahead of a higher class, change its
+priority with `prio`. Holds and trainings cannot be changed here: a hold is a pin, not a place in the queue, and
+a training's place is fixed by the card's lock. Each change is a `lease.queue_override` event in `berth events`.
+
+## Book the card
+
+The queue decides who goes next. A booking decides whose hours they are, up to 14 days ahead. A booking has one of
+three kinds:
+
+- `run` is an engine's turn. While it lasts, the engine's waiter goes first in line and runs at the booking's
+ priority (foreground when the booking names none). It wins a tie against a holder that has no booking. Once it
+ holds the card, nothing but a hold drains it until the booking ends. A minimum turn (`min_turn`) still protects
+ the holder it takes over from.
+- `priority` moves an engine to another class for set hours, such as llama-swap at batch while a training runs. It
+ claims no time on the card, so it may overlap a run.
+- `down` is downtime. No new work starts, and holders drain at their usual boundary when it begins. A hold and a
+ training stay.
+
+An hour nobody booked runs by priority, then arrival. A booked engine that does not ask for the card leaves it to
+everyone else, so a booking never keeps the card idle.
+
+```sh
+berth schedule # what holds each card, the waiters in grant order, every booking
+berth schedule book --who pi-pocket --engine pocket-train --at 23:00 --for 5h --days 7 --priority batch
+berth schedule book --who training --engine sft --at "sun 18:00" --for 3h
+berth schedule book --who operator --kind priority --engine llama-swap --priority batch --at "sun 18:00" --for 3h
+berth schedule book --who operator --kind down --at "tue 04:00" --for 1h --note "driver update"
+berth schedule move b4 --at 01:00 # new hours, same ID; --for, --priority and --note change those
+berth schedule cancel b4
+```
+
+`--at` takes `now`, `22:00` (the next time the clock reads it), `tue 22:00`, `2026-11-14 22:00` or RFC 3339, on the
+node's clock. `--days 7` books the same hours on 7 days in a row, all of them or none. `--earliest` moves the
+booking to the first free hours at or after `--at`. `--engine` takes an engine or a group, such as `llama-swap`
+for every model it serves.
+
+Two runs or downtimes on one card may not overlap, and neither may two priority changes for the same engine. A
+clash is refused with the booking it hits. A class set with `berth queue prio` wins over a booking, and a booking
+never moves a hold or a training.
+
+berthd keeps the bookings in `schedule.json` in its state directory (`berth daemon --schedule`). Each change is a
+`schedule.booked` event, and each class a booking moves, or moves back at its end, is a `lease.booking_priority`
+event. A lease that a booking governs shows the booking's ID in `berth status --format json` (`booking`).
+
+The harbor's SCHEDULE tab shows the same scheduler. NOW lists what holds each card, the waiters and the booking in
+force. 7 DAYS draws one 24-hour track per day with every booking to the minute, each engine's window and the live
+holder. A paired device books by clicking an empty hour, and moves or cancels a booking by clicking it. Agents
+book over the gateway: read [Book the card from an agent](gateway.md#book-the-card-from-an-agent).
+
## Cordon a card
On a node with several cards, `berth cordon gpu1` stops new grants on that card while the others keep serving, for
@@ -166,8 +248,9 @@ maintenance or before a VM takes it. Name the card by its name or its PCI addres
[Run an engine whenever the card is free](gateway.md#run-an-engine-whenever-the-card-is-free).
- **A window puts either on the clock.** `window = "23:00-07:00"` on the entry, or `berth run --window 23:00-07:00`,
holds the lease to those hours on the node's wall clock. It may cross midnight. A lease that asks outside them
- waits for them with no deadline (unless `--wait` sets one) and holds up nobody behind it. `berth status` says when
- the window opens, and a background engine is not started at all. When the window closes its holder drains at its usual boundary, so a slot or a job boundary comes first and a
+ waits for them with no deadline (unless `--wait` sets one) and holds up nobody behind it. A background engine is
+ not started at all outside its hours. `berth status` (SCHEDULE) and the harbor (SCHEDULES) list every engine that
+ has a window, whether or not it runs, with its hours and when it next opens or closes. When the window closes its holder drains at its usual boundary, so a slot or a job boundary comes first and a
request in flight finishes.
```toml
@@ -224,25 +307,62 @@ berth run --as llama-70b --priority foreground --gpu gpu0,gpu1 \
## Training runs
-A training run that loses its cards loses everything since its last checkpoint. Run it under one pinned lease:
+A training run that loses its cards loses everything since its last checkpoint. Start it with `berth train`:
```sh
-berth run --as train-mymodel --pinned --priority foreground --gpu gpu0,gpu1,gpu2 \
- -- torchrun --nproc_per_node 3 train.py
+berth train --as train-mymodel --gpu gpu0,gpu1,gpu2 -- torchrun --nproc_per_node 3 train.py
```
-- `--pinned` keeps every card of the lease. A foreground run, a game's hold, `berth hold` and another lease on
- several cards all wait until the run exits. A gateway request never drains it, and the gateway's stop answers
- `409`. `berth stop train-mymodel` still stops it, because a person asked.
-- `--priority` decides only how the run gets its cards. `foreground` drains interactive and batch engines so the
- run can start. A hold or a pinned app already on a card makes it wait.
+- The run takes the whole of every card `--gpu` names. It drains the engines on them, waits with no deadline, and
+ keeps the cards until the command exits or you press Ctrl-C. Without `--gpu` it takes the node's first card.
+- While a training holds a card or waits for it, berthd locks that card. It refuses every other request there at
+ once: a gateway request, `berth run`, a llama-swap model, a scheduled job and `berth hold`. A client gets the
+ refusal right away and can try another node, instead of waiting out its timeout. Cards the training does not
+ name keep serving.
+- A second `berth train` for the same card waits in line with no deadline and starts when the first one ends.
+- A game or a hold that already has the card is not drained. The training waits for it.
+- `berth train` refuses to start when berthd is not running, because the lock lives in berthd.
- berthd grants the cards together and starts the command with `CUDA_VISIBLE_DEVICES` set to their UUIDs, in the
order you named them. Do not set `CUDA_VISIBLE_DEVICES` yourself, since Berth leaves it alone when it is set. Set
`--nproc_per_node` to the number of cards.
-- Leave `--needs-mib` at 0, the whole card. Training memory grows after the first steps.
- The workers torchrun or `accelerate launch` starts belong to the lease, so their memory is the run's on every card.
-- A berthd restart or upgrade does not stop the run. It keeps running, and berthd re-attaches it with its pin on
- every card.
+- `berth status` shows the lease as `hold,pinned,training`. `berth stop train-mymodel` still stops it, because a
+ person asked.
+- A berthd restart or upgrade does not stop the run, and the lock holds while berthd is away. With berthd down,
+ `berth run` refuses to start an engine while the training runs, instead of starting it without a lease. A new
+ berthd refuses every other request until the training re-attaches, with no time limit, and the training comes
+ back as a training.
+
+`berth run --pinned --priority foreground` still works for a run that must not be drained but may share its card.
+
+### Lock the card in the driver too
+
+The training lock covers everything that asks Berth for the card. A program that opens CUDA without Berth only
+shows up as a stowaway. `berth train --exclusive` closes that gap on Linux with an NVIDIA card. After the grant, it
+puts the cards in the driver's exclusive-process compute mode, so the training's process is the only one that can
+open a CUDA context there. When the run ends, it puts them back to the default mode.
+
+```sh
+berth train --exclusive --as train-mymodel -- python train.py
+```
+
+- Setting the mode needs root. Give your user a sudo rule for `nvidia-smi -c` alone, with `sudo visudo -f
+ /etc/sudoers.d/berth-exclusive`:
+
+ ```
+ YOUR_USER ALL=(root) NOPASSWD: /usr/bin/nvidia-smi -i * -c EXCLUSIVE_PROCESS, /usr/bin/nvidia-smi -i * -c DEFAULT
+ ```
+
+ Without the rule, `berth train --exclusive` refuses to start instead of running without the guard.
+- The run refuses to start when another compute process is already on the card, because that process would own
+ the card's one context and the training could not open it.
+- Exclusive mode allows one process per card. A training that opens CUDA from several processes on one card
+ (DataLoader workers that use CUDA, two ranks on one card) fails to start in it. One rank per card, as torchrun
+ gives each card, works.
+- Graphics are not compute: the desktop and games keep drawing on the card.
+- If the `berth train` process is killed with SIGKILL, the mode stays set. `sudo nvidia-smi -c DEFAULT` puts it
+ back.
+A pinned run does not lock the card: other requests wait for it until their own deadline.
A pin does not stop memory taken outside Berth. A game started on a leased card waits behind the run in Berth but
still allocates memory, and that can run the training out of memory. Keep the card that drives your display out of
diff --git a/docs/prerelease.md b/docs/prerelease.md
index 4695f311..9b80954b 100644
--- a/docs/prerelease.md
+++ b/docs/prerelease.md
@@ -1,7 +1,7 @@
# Testing a pre-release
Work for the next release lands on the `next` branch before it reaches `main`. When it is ready for testers, it is
-tagged as a pre-release (`v0.5.0-rc.1`, then `rc.2` and so on) on the
+tagged as a pre-release (`v0.6.0-rc.1`, then `rc.2` and so on) on the
[releases page](https://github.com/T-Crypt/berth/releases), marked **Pre-release**.
Your normal install is not affected. The README's download link, `go install ...@latest` and `berth update` all stay
@@ -15,7 +15,7 @@ how it works. Back up `berth.toml` before you switch.
Pick the tag from the releases page, then on Linux:
```sh
-TAG=v0.5.0-rc.1
+TAG=v0.6.0-rc.1
cd /tmp
curl -fLO https://github.com/T-Crypt/berth/releases/download/$TAG/berth-linux-amd64
curl -fLO https://github.com/T-Crypt/berth/releases/download/$TAG/SHA256SUMS
@@ -31,7 +31,7 @@ it where your current `berth.exe` is, and restart Berth the way you started it.
Or build it with Go 1.27 or newer:
```sh
-go install github.com/T-Crypt/berth/cmd/berth@v0.5.0-rc.1 # a tagged pre-release
+go install github.com/T-Crypt/berth/cmd/berth@v0.6.0-rc.1 # a tagged pre-release
go install github.com/T-Crypt/berth/cmd/berth@next # the newest commit on next
```
diff --git a/docs/roadmap.md b/docs/roadmap.md
index d0802ea0..485dd26a 100644
--- a/docs/roadmap.md
+++ b/docs/roadmap.md
@@ -130,16 +130,38 @@ v0.5.0 was tagged first as `v0.5.0-rc.1`.
- **Restarts.** A run that started before berthd, or lost it, re-attaches when a daemon answers, and berthd
logs a refused re-attach with its reason.
-## Next: v0.5.x, models across machines
-
+## Shipped: v0.6.0, a schedule per card (2026-11-14)
+
+v0.6.0 was tagged first as `v0.6.0-rc.1`.
+
+- **A schedule.** `berth schedule book` books an engine's run, a priority change or downtime on a card, up to 14
+ days ahead. Agents book their own runs through `/v1/berth/schedule`, and the harbor's SCHEDULE tab shows the
+ week. Hours nobody booked run by priority, then arrival.
+- **Training runs.** `berth train` takes whole cards for a fine-tune, refuses every other request on them at once,
+ and with `--exclusive` locks the card in the driver.
+- **Answers about the card.** `berth fit` ranks where a request would run, `berth contract` lists an engine's
+ terms, `berth queue` shows and changes the grant order, and `berth why` tells one lease's story. The gateway
+ picks through the same ranking as `berth fit`.
+- **Learned sizes.** `learn_size = true` sizes a lease from the engine's measured peak.
+- **A yield for ComfyUI jobs that call an LLM** ([#299](https://github.com/T-Crypt/berth/issues/299)), and
+ `berth notify` as a Windows tray icon ([#186](https://github.com/T-Crypt/berth/pull/186)).
+- **Restarts.** The event log and lease numbers carry on across a berthd restart, a re-attached engine keeps the
+ turn it was in, and a drain no longer stops an engine that is still resuming
+ ([#275](https://github.com/T-Crypt/berth/issues/275)).
+
+## Next: v0.6.x
+
+- **A warmth ladder.** An engine steps down from live to parked to stopped by idle time and the next booking, and
+ `berth why` names each step.
+- **A booked run starts its engine** at the start of its hours, for engines the gateway can launch.
- **The soonest node.** When several nodes serve a model, send the request to the one where it is warm or the
- card is free, by the same reach order a node uses for its own cards.
-- **One board.** The harbor's SWITCH tab shows each peer's cards and engines beside this node's.
+ card is free.
+- **Device names.** A key per device, so the queue and the schedule say "mac" or "pi-pocket", and a booking
+ belongs to the device that made it.
Still to decide, with data from multi-card users:
- Whether a cordon survives a berthd restart. Today it lives in berthd's memory, and a restart clears it.
-- Whether measured switch costs rank the cards for a plain `/v1` request. Today the gateway ranks them by reach.
## Also queued
@@ -149,8 +171,6 @@ Each item has its building blocks merged or a spec written:
- Models started on demand for a standalone llama-server.
- Parked resume numbers for vLLM and SGLang in [Swap numbers](benchmarks.md#swap-numbers), which take
their park adapters out of experimental.
-- `berth notify` on Windows: a tray icon with native toasts ([#186](https://github.com/T-Crypt/berth/pull/186)),
- after a test on a real Windows desktop.
- VFIO on a box with two NVIDIA cards: keep the other card's berthd from opening the card being
switched, after a measurement on a real two-card box.
- Hermes Agent: an overnight recipe in the setup page, built on the gateway workflow test, where Hermes
diff --git a/internal/broker/booking.go b/internal/broker/booking.go
new file mode 100644
index 00000000..e3d36405
--- /dev/null
+++ b/internal/broker/booking.go
@@ -0,0 +1,135 @@
+// Copyright 2026 Trevin Tindall (T-Crypt)
+// SPDX-License-Identifier: Apache-2.0
+
+package broker
+
+// The card's schedule. berthd's booking file says who has the card when,
+// and the broker reads it on every tick. A run booking is an engine's turn:
+// its waiter goes first, it runs at the booking's priority, and while it
+// holds the card nothing but a hold drains it. A priority booking moves an
+// engine to another class for its hours. Downtime takes no new work and
+// drains its holders at the start. An hour nobody booked runs as it always
+// did, by priority then arrival, so an unused booking leaves the card idle
+// for nobody.
+
+import (
+ "fmt"
+ "slices"
+ "time"
+
+ "github.com/T-Crypt/berth/internal/proto"
+)
+
+// SetBookings replaces the card's schedule with bs, and schedules the queue
+// under it at once.
+func (b *Broker) SetBookings(bs []proto.Booking) {
+ b.mu.Lock()
+ defer b.mu.Unlock()
+ b.bookings = slices.Clone(bs)
+ now := b.now()
+ b.closeForDowntime(now)
+ b.schedule(now)
+}
+
+// booked is the booking that sets l's priority at now: a run, then a
+// priority change, for its engine or its group. A hold, a training and a
+// class the operator set by hand are not moved.
+func (b *Broker) booked(l *lease, now time.Time) (proto.Booking, bool) {
+ if l.basePrio == Hold || l.training || l.operatorPrio {
+ return proto.Booking{}, false
+ }
+ for _, k := range []proto.BookingKind{proto.BookRun, proto.BookPriority} {
+ for _, name := range []string{l.engine, l.group} {
+ if name == "" {
+ continue
+ }
+ if x, ok := proto.BookingAt(b.bookings, k, name, now); ok {
+ return x, true
+ }
+ }
+ }
+ return proto.Booking{}, false
+}
+
+// inRun is whether a run booking holds for l at now.
+func (b *Broker) inRun(l *lease, now time.Time) bool {
+ x, ok := b.booked(l, now)
+ return ok && x.Kind == proto.BookRun
+}
+
+// bookedPrio is the class a booking gives: its own, and foreground for a
+// run that names none.
+func bookedPrio(x proto.Booking) Priority {
+ if x.Priority == "" {
+ return Foreground
+ }
+ p, err := ParsePriority(x.Priority)
+ if err != nil {
+ return Foreground
+ }
+ return p
+}
+
+// applyBookings sets every lease's class from the schedule at now: the
+// booking's while one holds, its own when none does. Each move is an event.
+// Called with b.mu held.
+func (b *Broker) applyBookings(now time.Time) {
+ moved := false
+ for _, l := range slices.Concat(b.queue, mapValues(b.leases)) {
+ p, id := l.basePrio, ""
+ if x, ok := b.booked(l, now); ok {
+ p, id = bookedPrio(x), x.ID
+ }
+ if p == l.prio && id == l.booking {
+ continue
+ }
+ detail := fmt.Sprintf("priority %s -> %s: booking %s", l.prio, p, id)
+ if id == "" {
+ detail = fmt.Sprintf("priority %s -> %s: booking %s ended", l.prio, p, l.booking)
+ }
+ l.prio, l.booking, moved = p, id, true
+ b.emit(proto.Event{Kind: proto.EvBookingPrio, Lease: l.id, Engine: l.engine, Detail: detail})
+ b.log.Info("booking priority", "lease", l.id, "engine", l.engine, "change", detail)
+ }
+ if moved {
+ b.sortQueue()
+ }
+}
+
+// closeForDowntime drains the active holders, and evicts the parked ones,
+// when downtime holds at now. A hold and a training keep the card. A lease
+// already on its way out is not asked twice. Called with b.mu held.
+func (b *Broker) closeForDowntime(now time.Time) {
+ d, ok := proto.BookingAt(b.bookings, proto.BookDown, "", now)
+ if !ok {
+ return
+ }
+ for _, l := range b.leases {
+ if l.prio == Hold || l.training {
+ continue
+ }
+ switch l.state {
+ case active:
+ b.drain(l, downWhy(d))
+ case parked:
+ b.evict(l, downWhy(d))
+ }
+ }
+}
+
+// downWhy is what a waiter and a drained holder read during downtime.
+func downWhy(d proto.Booking) string {
+ why := fmt.Sprintf("the card is down until %s (booking %s by %s", d.End.Format("Mon 15:04"), d.ID, d.Who)
+ if d.Note != "" {
+ why += ": " + d.Note
+ }
+ return why + ")"
+}
+
+func mapValues[K comparable, V any](m map[K]V) []V {
+ out := make([]V, 0, len(m))
+ for _, v := range m {
+ out = append(out, v)
+ }
+ return out
+}
diff --git a/internal/broker/booking_test.go b/internal/broker/booking_test.go
new file mode 100644
index 00000000..edb6a894
--- /dev/null
+++ b/internal/broker/booking_test.go
@@ -0,0 +1,185 @@
+// Copyright 2026 Trevin Tindall (T-Crypt)
+// SPDX-License-Identifier: Apache-2.0
+
+package broker
+
+import (
+ "context"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/T-Crypt/berth/internal/procs"
+ "github.com/T-Crypt/berth/internal/proto"
+)
+
+// book is a booking on b's card from turnBase+from for length.
+func book(_ *Broker, id string, k proto.BookingKind, engine, prio string, from, length time.Duration) proto.Booking {
+ return proto.Booking{ID: id, Card: "gpu0", Kind: k, Who: "test", Engine: engine, Priority: prio,
+ Start: turnBase.Add(from), End: turnBase.Add(from + length)}
+}
+
+func leaseRow(t *testing.T, b *Broker, engine string) proto.Lease {
+ t.Helper()
+ st := b.Status()
+ for _, l := range append(st.Leases, st.Waiting...) {
+ if l.Engine == engine {
+ return l
+ }
+ }
+ t.Fatalf("no lease for %s in %+v", engine, st)
+ return proto.Lease{}
+}
+
+// A run booking is the engine's turn: its batch waiter runs at foreground,
+// so an interactive holder drains for it, and status names the booking.
+func TestRunBookingLiftsItsWaiterAndDrainsALowerHolder(t *testing.T) {
+ b, _ := startTurnBroker(t, procs.Map{})
+ b.SetBookings([]proto.Booking{book(b, "b1", proto.BookRun, "pi-pocket", "", -time.Minute, 5*time.Hour)})
+ holder := must(t, acquire(b, Request{Engine: "comfyui", Priority: Interactive, Needs: 20 * GiB}))
+
+ w := acquireW(b, Request{Engine: "pi-pocket", Priority: Batch, Needs: 20 * GiB})
+ pending(t, w.ch)
+ if got, want := drained(t, holder), "pi-pocket (foreground) needs the card"; got != want {
+ t.Errorf("drain = %q, want %q", got, want)
+ }
+ if row := leaseRow(t, b, "pi-pocket"); row.Priority != "foreground" || row.Booking != "b1" {
+ t.Errorf("waiter = %s booking %q, want foreground booking b1", row.Priority, row.Booking)
+ }
+ holder.Release()
+ b.Tick(context.Background())
+ granted(t, b, "pi-pocket", w.ch)
+}
+
+// A tie is the holder's, except against the waiter's booked turn: a
+// foreground holder with no booking drains for a run lifted to foreground.
+func TestRunBookingWinsATie(t *testing.T) {
+ b, _ := startTurnBroker(t, procs.Map{})
+ b.cfg.JuggleWindow = time.Hour
+ b.SetBookings([]proto.Booking{book(b, "b1", proto.BookRun, "sft", "", 0, time.Hour)})
+ holder := must(t, acquire(b, Request{Engine: "chat", Priority: Foreground, Needs: 20 * GiB}))
+ w := acquireW(b, Request{Engine: "sft", Priority: Batch, Needs: 20 * GiB})
+ pending(t, w.ch)
+ if got, want := drained(t, holder), "sft (foreground) needs the card"; got != want {
+ t.Errorf("drain = %q, want %q", got, want)
+ }
+}
+
+// Inside its run booking a holder keeps the card against a foreground
+// waiter; once the booking ends it is back to its own class and drains.
+func TestRunBookingKeepsItsHolderUntilItEnds(t *testing.T) {
+ b, clk := startTurnBroker(t, procs.Map{})
+ b.SetBookings([]proto.Booking{book(b, "b1", proto.BookRun, "train", "", 0, 3*time.Hour)})
+ holder := must(t, acquire(b, Request{Engine: "train", Priority: Batch, Needs: 20 * GiB}))
+
+ w := acquireW(b, Request{Engine: "game", Priority: Foreground, Needs: 20 * GiB})
+ pending(t, w.ch)
+ select {
+ case why := <-holder.Drains():
+ t.Fatalf("a booked holder was drained inside its booking: %s", why)
+ default:
+ }
+ if got, want := firstReason(t, w), "held by train [foreground, active]"; !strings.HasSuffix(got, want) {
+ t.Errorf("reason = %q, want it to end %q", got, want)
+ }
+
+ clk.set(turnBase.Add(3 * time.Hour))
+ b.Tick(context.Background())
+ if got, want := drained(t, holder), "game (foreground) needs the card"; got != want {
+ t.Errorf("drain = %q, want %q", got, want)
+ }
+ if row := leaseRow(t, b, "train"); row.Priority != "batch" || row.Booking != "" {
+ t.Errorf("holder after the booking = %s booking %q, want batch and none", row.Priority, row.Booking)
+ }
+}
+
+// A run booking puts its waiter ahead of an earlier foreground one.
+func TestRunBookingGoesFirstInLine(t *testing.T) {
+ b, _ := startTurnBroker(t, procs.Map{})
+ b.SetBookings([]proto.Booking{book(b, "b1", proto.BookRun, "pi-pocket", "batch", 0, time.Hour)})
+ holder := must(t, acquire(b, Request{Engine: "hold", Priority: Hold, Needs: 20 * GiB}))
+ early := acquireW(b, Request{Engine: "chat", Priority: Foreground, Needs: 20 * GiB})
+ pending(t, early.ch)
+ booked := acquireW(b, Request{Engine: "pi-pocket", Priority: Batch, Needs: 20 * GiB})
+ pending(t, booked.ch)
+ if st := b.Status(); len(st.Waiting) != 2 || st.Waiting[0].Engine != "pi-pocket" {
+ t.Fatalf("queue = %+v, want pi-pocket first", st.Waiting)
+ }
+ holder.Release()
+ b.Tick(context.Background())
+ granted(t, b, "pi-pocket", booked.ch)
+}
+
+// A priority booking moves an engine's class for its hours and back after,
+// with an event each way; the operator's own berth queue prio wins over it.
+func TestPriorityBookingMovesTheClassAndBack(t *testing.T) {
+ b, clk := startTurnBroker(t, procs.Map{})
+ b.SetBookings([]proto.Booking{book(b, "b2", proto.BookPriority, "dirk", "batch", time.Hour, time.Hour)})
+ must(t, acquire(b, Request{Engine: "hold", Priority: Hold, Needs: 20 * GiB}))
+ w := acquireW(b, Request{Engine: "dirk", Priority: Foreground, Needs: 20 * GiB})
+ pending(t, w.ch)
+ if row := leaseRow(t, b, "dirk"); row.Priority != "foreground" {
+ t.Fatalf("before the booking = %s, want foreground", row.Priority)
+ }
+ clk.set(turnBase.Add(90 * time.Minute))
+ b.Tick(context.Background())
+ if row := leaseRow(t, b, "dirk"); row.Priority != "batch" || row.Booking != "b2" {
+ t.Errorf("inside the booking = %s booking %q, want batch b2", row.Priority, row.Booking)
+ }
+ clk.set(turnBase.Add(2 * time.Hour))
+ b.Tick(context.Background())
+ if row := leaseRow(t, b, "dirk"); row.Priority != "foreground" || row.Booking != "" {
+ t.Errorf("after the booking = %s booking %q, want foreground and none", row.Priority, row.Booking)
+ }
+ evs, _, _ := b.Events(0, false)
+ var moves []string
+ for _, e := range evs {
+ if e.Kind == proto.EvBookingPrio {
+ moves = append(moves, e.Detail)
+ }
+ }
+ want := []string{"priority foreground -> batch: booking b2", "priority batch -> foreground: booking b2 ended"}
+ if len(moves) != 2 || moves[0] != want[0] || moves[1] != want[1] {
+ t.Errorf("events = %q, want %q", moves, want)
+ }
+
+ clk.set(turnBase.Add(90 * time.Minute))
+ id := leaseRow(t, b, "dirk").ID
+ if _, err := b.Queue(id, QueuePrio, "interactive"); err != nil {
+ t.Fatal(err)
+ }
+ b.Tick(context.Background())
+ if row := leaseRow(t, b, "dirk"); row.Priority != "interactive" {
+ t.Errorf("after berth queue prio inside the booking = %s, want interactive", row.Priority)
+ }
+}
+
+// Downtime drains its holder at the start, refuses new work with the
+// booking's reason, and still lets a hold in.
+func TestDowntimeDrainsAndTakesNoNewWork(t *testing.T) {
+ b, clk := startTurnBroker(t, procs.Map{})
+ d := book(b, "b3", proto.BookDown, "", "", time.Hour, 30*time.Minute)
+ d.Who, d.Note = "operator", "driver update"
+ b.SetBookings([]proto.Booking{d})
+ holder := must(t, acquire(b, Request{Engine: "chat", Priority: Interactive, Needs: 8 * GiB}))
+
+ clk.set(turnBase.Add(time.Hour))
+ b.Tick(context.Background())
+ want := "the card is down until " + d.End.Format("Mon 15:04") + " (booking b3 by operator: driver update)"
+ if got := drained(t, holder); got != want {
+ t.Errorf("drain = %q, want %q", got, want)
+ }
+ holder.Release()
+ b.Tick(context.Background())
+
+ w := acquireW(b, Request{Engine: "chat", Priority: Foreground, Needs: 8 * GiB})
+ pending(t, w.ch)
+ if got := firstReason(t, w); got != "next in line: "+want {
+ t.Errorf("waiter reason = %q, want %q", got, want)
+ }
+ must(t, acquire(b, Request{Engine: "game", Priority: Hold, Needs: 8 * GiB}))
+
+ clk.set(turnBase.Add(90 * time.Minute))
+ b.Tick(context.Background())
+ granted(t, b, "chat", w.ch)
+}
diff --git a/internal/broker/broker.go b/internal/broker/broker.go
index 1ce1f6fa..4efe10b1 100644
--- a/internal/broker/broker.go
+++ b/internal/broker/broker.go
@@ -209,11 +209,19 @@ type Request struct {
Group string // the engine family a lease counts for (llama-swap's models); "" = Engine
Priority Priority
Needs uint64 // bytes; 0 = the whole card
- Wait time.Duration
+ // LearnSize: with Needs 0, the lease takes the most this engine has
+ // been measured holding plus the load headroom, not the whole card. An
+ // engine never measured still takes the whole card.
+ LearnSize bool
+ Wait time.Duration
// Pinned: never a drain victim. A hold (berth hold, a game claim) or an
// app that must not be stopped (berth run --pinned) keeps the card until
// it lets go; waiters are told who pins it.
Pinned bool
+ // Training: berth train. The lease is a pinned hold on the whole card
+ // with no wait deadline, and it locks the card: while it holds or waits
+ // there, every request that is not a training is refused at once.
+ Training bool
// Switched: a person asked for this engine (the wake page's Launch,
// /v1/berth/switch). It is the last switch, so it never loses a
// juggle, and inside the juggle window a request no person made does
@@ -240,9 +248,12 @@ type Request struct {
// MinTurn is a minimum turn: while the lease is younger than it after
// the grant, no waiter drains it, whatever the waiter's priority,
// except a hold (a game, a call, the operator's own berth hold). When
- // the turn ends the normal rules apply. A re-attach starts a fresh
- // turn. 0 = off.
+ // the turn ends the normal rules apply. A re-attach goes on with the
+ // turn it was in (Held). 0 = off.
MinTurn time.Duration
+ // Held is how long a re-attach's engine had held the card since its
+ // last grant or resume, under the daemon that went away.
+ Held time.Duration
// ParkTTL is how long the lease may stay parked before the broker
// stops its engine, in place of the node's ParkTTL. 0 = the node's.
ParkTTL time.Duration
@@ -275,15 +286,25 @@ type lease struct {
lostWhy string
reattach bool // re-attached after a daemon restart: granted is the re-attach, not a fresh switch
pinned bool // never drained (Request.Pinned)
+ training bool // locks the card (Request.Training)
+ // lockCh tells a waiter that a training locked its card: it gives up.
+ lockCh chan string
switched bool // a person's switch (Request.Switched)
minTurn time.Duration
parkTTL time.Duration // 0 = the node's
window proto.Window // zero = no window
prio Priority
- needs uint64
- state state
- pid int
- pidStart uint64
+ // basePrio is the class the lease asked for, or the operator set; prio
+ // differs from it while a booking holds (booking.go).
+ basePrio Priority
+ // booking is the ID of the booking that sets prio now, "" when none.
+ booking string
+ // operatorPrio: berth queue prio set the class, and no booking moves it.
+ operatorPrio bool
+ needs uint64
+ state state
+ pid int
+ pidStart uint64
// goneTicks counts the consecutive ticks that saw the attached pid
// gone (no process with its start time); the reaper needs two, so a
// pid lookup hiccup on one tick cannot free a live lease.
@@ -322,6 +343,9 @@ type lease struct {
// two leases granted in one quantum would otherwise tie, and which one
// is older, and so drained first, would be arbitrary.
arrivedN, grantedN, parkedN uint64
+ // bumpedN is set by berth queue bump: within its priority class the
+ // latest bump goes first, ahead of every waiter never bumped.
+ bumpedN uint64
// parkedAt is when the lease became parked; parkedCall is when the
// runner said its park call was done (the parked transition waits for
// a later card read). resumed marks a waiting lease that came from
@@ -331,6 +355,7 @@ type lease struct {
parkedCall time.Time
resumed bool
beside string
+ sized string // how a LearnSize request got its size, for the grant line
// The lend sub-state (docs/design.md): at most one open lend per holder,
// and never a park at the same time. lentBytes is what the engine said it
@@ -360,6 +385,7 @@ type lease struct {
resumeCh chan struct{}
drainCh chan string
waitCh chan string
+ yieldCh chan time.Duration // Yield; only the newest is kept
// lendAsk and returnAsk are the asks the broker made and this lease's
// runner has not picked up yet; lendSig and returnSig wake its connection.
// One ask of each kind per lease, so a runner that stops reading loses
@@ -415,8 +441,11 @@ func newLease(id string, r Request, now time.Time) *lease {
engine: r.Engine,
group: r.Group,
prio: r.Priority,
+ basePrio: r.Priority,
needs: r.Needs,
pinned: r.Pinned,
+ training: r.Training,
+ lockCh: make(chan string, 1),
switched: r.Switched,
minTurn: r.MinTurn,
parkTTL: r.ParkTTL,
@@ -430,6 +459,7 @@ func newLease(id string, r Request, now time.Time) *lease {
resumeCh: make(chan struct{}, 1),
drainCh: make(chan string, 1),
waitCh: make(chan string, 8),
+ yieldCh: make(chan time.Duration, 1),
lendSig: make(chan struct{}, 1),
returnSig: make(chan struct{}, 1),
}
@@ -495,6 +525,7 @@ type Broker struct {
verdicts map[string]verdict // loser engine key -> the juggle it lost
batches map[string]*batch // berth batch: engine or group -> its batch
cordoned bool // berth cordon: grant only holds
+ bookings []proto.Booking // this card's schedule (SetBookings)
parkOff bool // berth park off: drains stop their engine, nothing parks
// tracked is the claim watcher's last scan: a process (and every child)
// a track claim names. Its card memory is the claim's, never a stowaway.
@@ -951,6 +982,7 @@ func (b *Broker) Tick(ctx context.Context) {
}
b.claimed = claimed
b.closeWindows(now)
+ b.closeForDowntime(now)
b.evictParked(now)
b.schedule(now)
}
@@ -1053,7 +1085,12 @@ func (b *Broker) runFor(pid int) *runs.Record {
// found none at its start, while this daemon is young enough that its berth
// run is still dialling, and it has not re-attached yet.
func (b *Broker) dueToReattach(r runs.Record, now time.Time) bool {
- if (r.Reason != runs.ReasonLost && r.Reason != runs.ReasonStart) || now.Sub(b.started) >= b.cfg.ReattachWindow {
+ if r.Reason != runs.ReasonLost && r.Reason != runs.ReasonStart {
+ return false
+ }
+ // A training waits as long as its process lives: the card is its own
+ // until it comes back, however long the restart took.
+ if !r.Training && now.Sub(b.started) >= b.cfg.ReattachWindow {
return false
}
for _, l := range b.leases {
@@ -1101,6 +1138,7 @@ func (b *Broker) allowed(name string) bool {
// holders to drain for the first waiter that does not fit. Nobody behind a
// blocked waiter jumps it, so a big request is not starved by small ones.
func (b *Broker) schedule(now time.Time) {
+ b.applyBookings(now)
if b.cardState != "" {
// The card does not serve the node, so nothing new is granted, and
// every waiter reads the same reason. The VFIO hold itself keeps
@@ -1223,17 +1261,28 @@ func (b *Broker) grant(w *lease, now time.Time) {
if beside != "" {
detail += "; " + beside
}
+ if w.sized != "" {
+ detail += "; " + w.sized
+ }
b.emit(proto.Event{Kind: proto.EvGranted, Lease: w.id, Engine: w.engine, Bytes: w.contested, Detail: detail})
b.log.Info("granted", "lease", w.id, "engine", w.engine, "priority", w.prio, "needs_mib", w.needs/gpu.MiB, "waited", now.Sub(w.arrived).Round(time.Millisecond))
}
-// sortQueue orders the waiters: higher priority first, then by arrival on
-// the node. b.mu must be held.
+// sortQueue orders the waiters: higher priority first, then the operator's
+// latest bump (berth queue bump), then by arrival on the node. b.mu must be
+// held.
func (b *Broker) sortQueue() {
+ now := b.now()
sort.SliceStable(b.queue, func(i, j int) bool {
+ if ri, rj := b.inRun(b.queue[i], now), b.inRun(b.queue[j], now); ri != rj {
+ return ri
+ }
if b.queue[i].prio != b.queue[j].prio {
return b.queue[i].prio > b.queue[j].prio
}
+ if b.queue[i].bumpedN != b.queue[j].bumpedN {
+ return b.queue[i].bumpedN > b.queue[j].bumpedN
+ }
return b.queue[i].arrivedN < b.queue[j].arrivedN
})
}
@@ -1434,6 +1483,9 @@ func (b *Broker) evaluate(w *lease, now time.Time) (bool, string, []*lease) {
if b.cordoned && w.prio != Hold {
return false, "the card is cordoned; berth uncordon lets new work in", nil
}
+ if d, ok := proto.BookingAt(b.bookings, proto.BookDown, "", now); ok && w.prio != Hold && !w.training {
+ return false, downWhy(d), nil
+ }
if !w.window.Contains(now) {
// Its own hours decide this waiter alone: no holder is a victim,
// because nothing it drains would put it inside its window.
@@ -1502,7 +1554,7 @@ func (b *Broker) evaluate(w *lease, now time.Time) (bool, string, []*lease) {
// for it to become parked (or fall back to a drain); evicting
// it mid-park would throw the park away.
pending = append(pending, h)
- case h.pinned || h.prio > w.prio:
+ case h.pinned || h.prio > w.prio, w.prio != Hold && b.inRun(h, now) && !b.inRun(w, now):
blockers = append(blockers, h)
case w.prio != Hold && h.inTurn(now):
// A minimum turn: the holder keeps the card until its turn
@@ -1517,7 +1569,8 @@ func (b *Broker) evaluate(w *lease, now time.Time) (bool, string, []*lease) {
keptWhy = batchWhy(k, bt)
continue
case h.prio == w.prio:
- if keep, why := b.keeps(h, w, now); keep {
+ // A tie is the holder's, unless it is the waiter's booked turn.
+ if keep, why := b.keeps(h, w, now); keep && (!b.inRun(w, now) || b.inRun(h, now)) {
kept, keptWhy = append(kept, h), why
continue
}
@@ -1549,7 +1602,13 @@ func (b *Broker) evaluate(w *lease, now time.Time) (bool, string, []*lease) {
for _, p := range warm {
residual += p.used()
}
- if len(warm) > 0 && residual <= b.cfg.ParkResidualMax && !w.wholeMachine {
+ // A waiter that has held more than the card keeps beside the
+ // residual would fail to load there, so the residual must go.
+ need := b.hotPeak(w.engine)
+ besideOK := func(left uint64) bool {
+ return left <= b.cfg.ParkResidualMax && !w.wholeMachine && (need == 0 || need+left+besideHeadroom <= b.ceiling())
+ }
+ if len(warm) > 0 && besideOK(residual) {
w.beside = besideLine(warm, residual)
return true, "", nil
}
@@ -1559,7 +1618,7 @@ func (b *Broker) evaluate(w *lease, now time.Time) (bool, string, []*lease) {
var victims []*lease
var evicted uint64
for _, p := range sortParked(warm) {
- if residual-evicted <= b.cfg.ParkResidualMax && !w.wholeMachine {
+ if besideOK(residual - evicted) {
break
}
victims = append(victims, p)
@@ -1758,6 +1817,7 @@ func (b *Broker) Reattach(r Request, pid int, start uint64) (*Lease, error) {
// reattach is Reattach under a lease ID the caller picked.
func (b *Broker) reattach(id string, r Request, pid int, start uint64) (_ *Lease, err error) {
+ r = r.asTraining()
defer func() {
if err != nil {
b.log.Warn("re-attach refused", "engine", r.Engine, "pid", pid, "reason", err)
@@ -1778,13 +1838,14 @@ func (b *Broker) reattach(id string, r Request, pid int, start uint64) (_ *Lease
}
}
now := b.now()
+ r.Needs, _ = b.learnSize(r)
l := newLease(id, r, now)
l.arrivedN = b.order()
l.containers = containerSet(r.Containers)
l.state, l.reattach = active, true
l.pid, l.pidStart = pid, start
l.scope = scope
- l.granted = now
+ l.granted = now.Add(-max(r.Held, 0))
l.grantedN = b.order()
l.releaseBaseline = b.snap.Baseline()
l.releaseGTTBaseline = b.snap.GTTBaseline()
@@ -1838,6 +1899,10 @@ func IsNotFound(err error) bool {
// cancelled, or the daemon exits.
func (b *Broker) waitFor(r Request) time.Duration {
switch {
+ case r.Training:
+ // A training waits for the card however long the run before it
+ // takes: the next one queues, it never dies.
+ return -1
case r.Wait != 0:
return r.Wait
case r.Window.Set():
@@ -1852,16 +1917,22 @@ func (b *Broker) waitFor(r Request) time.Duration {
// Acquire queues for a berth and blocks until it is granted, ctx ends, or
// the wait runs out. onWait hears each new reason for waiting.
func (b *Broker) Acquire(ctx context.Context, r Request, onWait func(string)) (*Lease, error) {
+ r = r.asTraining()
b.mu.Lock()
if err := b.refuse(r); err != nil {
b.mu.Unlock()
return nil, err
}
+ var sized string
+ r.Needs, sized = b.learnSize(r)
l := newLease(fmt.Sprintf("%s-%d", r.Engine, b.cfg.IDs.Add(1)), r, b.now())
- l.arrivedN = b.order()
+ l.arrivedN, l.sized = b.order(), sized
b.queue = append(b.queue, l)
b.log.Info("acquire", "lease", l.id, "engine", l.engine, "priority", l.prio, "needs_mib", l.needs/gpu.MiB)
b.emit(proto.Event{Kind: proto.EvRequested, Lease: l.id, Engine: l.engine, Bytes: l.needs, Detail: l.prio.String()})
+ if l.training {
+ b.lockOut(l)
+ }
b.schedule(b.now())
b.mu.Unlock()
@@ -1882,6 +1953,8 @@ func (b *Broker) Acquire(ctx context.Context, r Request, onWait func(string)) (*
}
case <-ctx.Done():
return nil, b.abandon(l, ctx.Err().Error())
+ case why := <-l.lockCh:
+ return nil, b.abandon(l, why)
case <-timeout:
// A grant may have landed in the same instant, possibly after a
// holder was already drained for it; keep it rather than waste the drain.
@@ -1895,11 +1968,81 @@ func (b *Broker) Acquire(ctx context.Context, r Request, onWait func(string)) (*
}
}
+// asTraining is r as a training lease takes the card: a pinned hold on the
+// whole card, held to no hours and no turn. Any other request is r.
+func (r Request) asTraining() Request {
+ if !r.Training {
+ return r
+ }
+ r.Priority, r.Pinned = Hold, true
+ r.Needs, r.LearnSize = 0, false
+ r.Window, r.MinTurn = proto.Window{}, 0
+ return r
+}
+
+// training is the training lease that locks this card, holding it or
+// waiting for it, nil when none does. b.mu must be held.
+func (b *Broker) training() *lease {
+ for _, l := range b.leases {
+ if l.training {
+ return l
+ }
+ }
+ for _, l := range b.queue {
+ if l.training {
+ return l
+ }
+ }
+ return nil
+}
+
+// lockedBy names the training that locks this card: a training lease, or a
+// training run of the last daemon that has not re-attached yet. "" when
+// none does. b.mu must be held.
+func (b *Broker) lockedBy(now time.Time) string {
+ if t := b.training(); t != nil {
+ return t.engine
+ }
+ for _, r := range b.runs {
+ if r.Training && b.dueToReattach(r, now) {
+ return r.Engine
+ }
+ }
+ return ""
+}
+
+// lockedFor is the refusal a request that is not a training gets while
+// the training engine locks the card.
+func (b *Broker) lockedFor(engine string) string {
+ card := b.cfg.Card
+ if card == "" {
+ card = "the card"
+ }
+ return fmt.Sprintf("%s is locked for training by %s; try again after it finishes", card, engine)
+}
+
+// lockOut refuses every waiter on the card that is not a training, as a
+// training arrives. b.mu must be held.
+func (b *Broker) lockOut(t *lease) {
+ why := b.lockedFor(t.engine)
+ for _, q := range b.queue {
+ if !q.training {
+ select {
+ case q.lockCh <- why:
+ default:
+ }
+ }
+ }
+}
+
// refuse is why r can never be granted on this card, nil when it can be.
// b.mu must be held.
func (b *Broker) refuse(r Request) error {
why := ""
+ lock := b.lockedBy(b.now())
switch limit := sat(b.snap.Total, b.cfg.Reserve); {
+ case lock != "" && !r.Training:
+ why = b.lockedFor(lock)
case b.snap.APU && r.Needs > 0:
why = "APU requires an exclusive lease; omit needs-mib (VRAM and GTT are separate pools)"
case r.Needs > 0 && b.snap.Total > 0 && r.Needs > limit:
diff --git a/internal/broker/footprint.go b/internal/broker/footprint.go
index 965bb5cb..441e0c21 100644
--- a/internal/broker/footprint.go
+++ b/internal/broker/footprint.go
@@ -29,6 +29,50 @@ type engineFootprint struct {
ParkedRAM uint64 // the RAM a parked run last held (its scope's anon + shmem); 0 = never measured
ParkBase uint64 // what that run held hot, just before it parked
RAMPeak uint64 // the most RAM any run held while active (its scope's anon + shmem); 0 = never measured
+ // HotPeak is the most VRAM a run of this engine name held, kept on the
+ // name's entry: a group's Peak is its largest model's, and one model of
+ // a llama-swap may fit beside a parked engine while another does not.
+ HotPeak uint64
+ // Priority and MinTurn are what the name's last run asked for, kept on
+ // the name's entry: a llama-swap model's terms sit on its berth run
+ // line, which no berth.toml entry declares. Priority "" = no run yet.
+ Priority string `json:",omitempty"`
+ MinTurn time.Duration `json:",omitempty"`
+}
+
+// besideHeadroom is the free card memory a whole-card engine needs past its
+// measured peak to load beside a parked engine. Measured on an RTX 4090: a
+// 23710 MiB model served alone, and died loading with 398 MiB left beside a
+// 456 MiB residual. The 2 s card read misses the load's last allocations.
+const besideHeadroom = 512 << 20
+
+// hotPeak is the most VRAM a run of engine has held, 0 until one was measured.
+// b.mu must be held.
+func (b *Broker) hotPeak(engine string) uint64 {
+ if f := b.footprints[engine]; f != nil {
+ return f.HotPeak
+ }
+ return 0
+}
+
+// learnSize is the size a request takes, and for a LearnSize request the
+// grant's line saying where the size came from: the most VRAM a run of the
+// engine has held plus besideHeadroom. A request with a size, for the whole
+// machine or on an APU keeps what it asked. An engine never measured, or one
+// whose peak leaves the card nothing to share, takes the whole card. b.mu
+// must be held.
+func (b *Broker) learnSize(r Request) (uint64, string) {
+ if !r.LearnSize || r.Needs > 0 || r.WholeMachine || b.snap.APU {
+ return r.Needs, ""
+ }
+ peak := b.hotPeak(r.Engine)
+ if peak == 0 {
+ return 0, "not measured yet, so the whole card"
+ }
+ if peak+besideHeadroom >= b.ceiling() {
+ return 0, fmt.Sprintf("its measured peak %s fills the card", sizeHuman(peak))
+ }
+ return peak + besideHeadroom, fmt.Sprintf("sized from its measured peak %s", sizeHuman(peak))
}
// footprints is the table keyed by engine key: the lease's group, else its
@@ -75,6 +119,13 @@ func (b *Broker) foldFootprint(l *lease, now time.Time) {
f.Last = l.peak
f.Runs++
f.At = now
+ n := b.footprints[l.engine]
+ if n == nil {
+ n = &engineFootprint{}
+ b.footprints[l.engine] = n
+ }
+ n.HotPeak = max(n.HotPeak, l.peak)
+ n.Priority, n.MinTurn = l.prio.String(), l.minTurn
b.footprintDirty = true
}
@@ -87,15 +138,19 @@ func sizeHuman(n uint64) string {
return fmt.Sprintf("%d MiB", n/gpu.MiB)
}
-// footprintStatus is the table as the status reply carries it: the last
-// run's peak per engine, sorted by name. b.mu must be held.
+// footprintStatus is the table as the status reply carries it: per engine
+// key, the last run's peak and the peaks learn_size and RAM figures read,
+// sorted by name. A llama-swap model's entry may carry only its own peaks;
+// the last-run figure sits on its group's. b.mu must be held.
func (b *Broker) footprintStatus() []proto.EngineFootprint {
out := []proto.EngineFootprint{}
for k, f := range b.footprints {
- if f == nil || f.Last == 0 {
+ if f == nil || f.Last == 0 && f.HotPeak == 0 && f.RAMPeak == 0 && f.ParkedRAM == 0 {
continue
}
- out = append(out, proto.EngineFootprint{Engine: k, Bytes: f.Last, Runs: f.Runs, At: f.At})
+ out = append(out, proto.EngineFootprint{Engine: k, Bytes: f.Last, Runs: f.Runs, At: f.At,
+ HotPeakBytes: f.HotPeak, RAMPeakBytes: f.RAMPeak, ParkedRAMBytes: f.ParkedRAM,
+ Priority: f.Priority, MinTurnS: f.MinTurn.Seconds()})
}
sort.Slice(out, func(i, j int) bool { return out[i].Engine < out[j].Engine })
return out
diff --git a/internal/broker/footprint_test.go b/internal/broker/footprint_test.go
index 971e3a58..dfc0f82f 100644
--- a/internal/broker/footprint_test.go
+++ b/internal/broker/footprint_test.go
@@ -117,6 +117,35 @@ func TestFootprintIgnoresACorruptFile(t *testing.T) {
}
}
+// A LearnSize lease takes the engine's measured peak plus the load headroom,
+// not the whole card, so a sized engine is granted beside it and the holder
+// is never drained. An engine never measured still takes the whole card.
+func TestLearnSizeSharesTheCard(t *testing.T) {
+ b, c, _ := setup(t, procs.Map{101: 100})
+ runFootprint(t, b, c, "llama", 12*GiB)
+
+ llama := granted(t, b, "llama", acquire(b, Request{Engine: "llama", Priority: Foreground, LearnSize: true}))
+ llama.Attach(100)
+ c.set(gpu.Proc{PID: 101, Name: "llama-server", Used: 12 * GiB})
+ b.Tick(context.Background())
+ if d := grantDetail(t, b, "llama"); !strings.Contains(d, "sized from its measured peak 12.0 GiB") {
+ t.Fatalf("grant detail = %q, want the learned size named", d)
+ }
+
+ granted(t, b, "whisper", acquire(b, Request{Engine: "whisper", Priority: Foreground, Needs: 8 * GiB}))
+ select {
+ case why := <-llama.Drains():
+ t.Fatalf("llama was drained for a waiter that fits beside its learned size: %s", why)
+ default:
+ }
+
+ fresh := acquire(b, Request{Engine: "fresh", Priority: Batch, LearnSize: true})
+ pending(t, fresh)
+ if w := lastWaiting(b); w == "" {
+ t.Fatal("an unmeasured LearnSize engine was not left waiting for the whole card")
+ }
+}
+
// A run that holds far more than the learned footprint warns once, once
// the engine has three runs to learn from.
func TestFootprintWarnsOnceOnADrift(t *testing.T) {
@@ -179,3 +208,30 @@ func TestFootprintFoldsAtTheFreeNotTheRelease(t *testing.T) {
t.Fatalf("Footprint(llama) = %d, %v after the free; want %d, true", last, ok, 9*GiB)
}
}
+
+// A llama-swap model's own peak and terms sit on its name's entry, its last
+// run on the group's. The status reply carries both, so a reader of the
+// model's name (berth contract, learn_size) finds the peak the broker sizes
+// it from and the priority and turn its berth run line asks for.
+func TestFootprintStatusCarriesAModelsOwnPeak(t *testing.T) {
+ b, c, _ := setup(t, procs.Map{101: 100})
+ l := must(t, acquire(b, Request{Engine: "dirk", Group: "llama-swap", Priority: Foreground, MinTurn: 5 * time.Minute}))
+ l.Attach(100)
+ c.set(gpu.Proc{PID: 101, Name: "llama-server", Used: 12 * GiB})
+ b.Tick(context.Background())
+ l.Release()
+ c.set()
+ b.Tick(context.Background())
+
+ got := map[string]proto.EngineFootprint{}
+ for _, f := range b.Status().Footprints {
+ f.At = time.Time{}
+ got[f.Engine] = f
+ }
+ if f := got["dirk"]; f != (proto.EngineFootprint{Engine: "dirk", HotPeakBytes: 12 * GiB, Priority: "foreground", MinTurnS: 300}) {
+ t.Fatalf("dirk's entry = %+v, want its own 12 GiB peak, foreground, a 300 s turn", f)
+ }
+ if f := got["llama-swap"]; f.Bytes != 12*GiB || f.Runs != 1 {
+ t.Fatalf("llama-swap's entry = %+v, want the last run's 12 GiB", f)
+ }
+}
diff --git a/internal/broker/gang.go b/internal/broker/gang.go
index c84e037c..97be88c7 100644
--- a/internal/broker/gang.go
+++ b/internal/broker/gang.go
@@ -86,6 +86,10 @@ func (g *gang) commit(now time.Time) bool {
// the order of cards; nil gives every share r.Needs. The cards must share
// the node's IDs and Seq.
func AcquireAll(ctx context.Context, cards []*Broker, needs []uint64, r Request, onWait func(string)) ([]*Lease, error) {
+ if r.Training {
+ needs = nil
+ }
+ r = r.asTraining()
per := shareRequests(cards, needs, r)
if len(cards) == 1 {
l, err := cards[0].Acquire(ctx, per[cards[0]], onWait)
@@ -108,6 +112,7 @@ func AcquireAll(ctx context.Context, cards []*Broker, needs []uint64, r Request,
arrived := sorted[0].cfg.Seq.Add(1)
g := &gang{kick: make(chan struct{}, 1)}
waits := make(chan string, 8)
+ locked := make(chan string, len(sorted))
done := make(chan struct{})
defer close(done)
for _, b := range sorted {
@@ -117,6 +122,9 @@ func AcquireAll(ctx context.Context, cards []*Broker, needs []uint64, r Request,
b.queue = append(b.queue, l)
b.log.Info("acquire", "lease", l.id, "engine", l.engine, "priority", l.prio, "needs_mib", l.needs>>20, "cards", len(sorted))
b.emit(proto.Event{Kind: proto.EvRequested, Lease: l.id, Engine: l.engine, Bytes: l.needs, Detail: l.prio.String()})
+ if l.training {
+ b.lockOut(l)
+ }
g.shares = append(g.shares, share{b, l})
b.mu.Unlock()
if shareQueued != nil {
@@ -131,6 +139,12 @@ func AcquireAll(ctx context.Context, cards []*Broker, needs []uint64, r Request,
case <-done:
return
}
+ case why := <-l.lockCh:
+ select {
+ case locked <- why:
+ case <-done:
+ }
+ return
case <-done:
return
}
@@ -179,6 +193,8 @@ func AcquireAll(ctx context.Context, cards []*Broker, needs []uint64, r Request,
}
case <-ctx.Done():
return nil, giveUp(ctx.Err().Error())
+ case why := <-locked:
+ return nil, giveUp(why)
case <-timeout:
return nil, giveUp(fmt.Sprintf("gave up after %s", wait))
}
diff --git a/internal/broker/minturn_test.go b/internal/broker/minturn_test.go
index fe9aa81e..b66c1a2d 100644
--- a/internal/broker/minturn_test.go
+++ b/internal/broker/minturn_test.go
@@ -221,3 +221,21 @@ func holderDrains(b *Broker, engine string) <-chan string {
}
return nil
}
+
+// A re-attach after a daemon restart keeps the turn the engine was in: a
+// deploy that restarts berthd must not hand every holder a fresh minimum turn.
+func TestReattachKeepsTheTurnItWasIn(t *testing.T) {
+ b, _, _ := setup(t, procs.Fake{Starts: map[int]uint64{100: 7}})
+ clk := newMinClock(turnBase)
+ b.nowFn = clk.now
+ if _, err := b.Reattach(Request{Engine: "comfyui", Priority: Interactive, MinTurn: 5 * time.Minute, Held: 4 * time.Minute}, 100, 7); err != nil {
+ t.Fatal(err)
+ }
+ clk.set(turnBase.Add(30 * time.Second))
+
+ w := acquireW(b, Request{Engine: "strata", Priority: Interactive})
+ pending(t, w.ch)
+ if got, want := firstReason(t, w), "next in line: comfyui has its minimum turn, 30s left"; got != want {
+ t.Errorf("reason = %q, want %q", got, want)
+ }
+}
diff --git a/internal/broker/override.go b/internal/broker/override.go
new file mode 100644
index 00000000..b6fcac93
--- /dev/null
+++ b/internal/broker/override.go
@@ -0,0 +1,77 @@
+// Copyright 2026 Trevin Tindall (T-Crypt)
+// SPDX-License-Identifier: Apache-2.0
+
+package broker
+
+// The operator's hand on the queue. Berth orders waiters by priority, then
+// arrival, and nobody has to manage it; berth queue is for the times the
+// operator wants it otherwise. Every change is an event, so the log shows
+// that a person moved the line, not the scheduler.
+
+import (
+ "fmt"
+
+ "github.com/T-Crypt/berth/internal/proto"
+)
+
+// QueueAction is one operator change to a waiter.
+type QueueAction string
+
+const (
+ // QueueBump moves the waiter to the front of its priority class.
+ QueueBump QueueAction = "bump"
+ // QueuePrio moves the waiter to another priority class.
+ QueuePrio QueueAction = "prio"
+ // QueueDrop refuses the waiter, the way a timeout would.
+ QueueDrop QueueAction = "drop"
+)
+
+// dropReason is what a dropped waiter is told.
+const dropReason = "dropped from the queue by the operator (berth queue drop)"
+
+// Queue applies one operator change to the waiter with lease ID id. prio is
+// the new class for QueuePrio: batch, interactive or foreground. A hold is a
+// pin, not a queue position, and a training's place is fixed by the lock, so
+// neither can be set here. The returned line says what changed.
+func (b *Broker) Queue(id string, action QueueAction, prio string) (string, error) {
+ b.mu.Lock()
+ defer b.mu.Unlock()
+ var w *lease
+ for _, q := range b.queue {
+ if q.id == id {
+ w = q
+ }
+ }
+ if w == nil {
+ return "", notFound(fmt.Sprintf("no waiter %s on this card", id))
+ }
+ var detail string
+ switch action {
+ case QueueBump:
+ w.bumpedN = b.order()
+ b.sortQueue()
+ detail = "moved to the front of its class by the operator"
+ case QueuePrio:
+ if w.training {
+ return "", fmt.Errorf("%s is a training: its place is fixed by the card's lock", id)
+ }
+ p, err := ParsePriority(prio)
+ if err != nil || p == Hold {
+ return "", fmt.Errorf("priority %q: use batch, interactive or foreground", prio)
+ }
+ detail = fmt.Sprintf("priority %s -> %s by the operator", w.prio, p)
+ w.prio, w.basePrio, w.operatorPrio, w.booking = p, p, true, ""
+ b.sortQueue()
+ case QueueDrop:
+ select {
+ case w.lockCh <- dropReason:
+ default:
+ }
+ detail = "dropped by the operator"
+ default:
+ return "", fmt.Errorf("unknown queue action %q (bump, prio, drop)", action)
+ }
+ b.emit(proto.Event{Kind: proto.EvQueueOverride, Lease: w.id, Engine: w.engine, Detail: detail})
+ b.log.Info("queue override", "lease", w.id, "engine", w.engine, "change", detail)
+ return detail, nil
+}
diff --git a/internal/broker/override_test.go b/internal/broker/override_test.go
new file mode 100644
index 00000000..b28ce91d
--- /dev/null
+++ b/internal/broker/override_test.go
@@ -0,0 +1,109 @@
+// Copyright 2026 Trevin Tindall (T-Crypt)
+// SPDX-License-Identifier: Apache-2.0
+
+package broker
+
+import (
+ "context"
+ "errors"
+ "slices"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/T-Crypt/berth/internal/procs"
+ "github.com/T-Crypt/berth/internal/proto"
+)
+
+// queueOrder is the queue as the status reports it, engine names in order.
+func queueOrder(b *Broker) []string {
+ var out []string
+ for _, l := range b.Status().Waiting {
+ out = append(out, l.Engine)
+ }
+ return out
+}
+
+func waiterID(t *testing.T, b *Broker, engine string) string {
+ t.Helper()
+ for _, l := range b.Status().Waiting {
+ if l.Engine == engine {
+ return l.ID
+ }
+ }
+ t.Fatalf("no waiter %s in %v", engine, queueOrder(b))
+ return ""
+}
+
+// The operator reorders the queue: a bump puts a waiter first in its class,
+// a prio moves it to another class, a drop refuses it. Each is an event, and
+// the next grant follows the new order.
+func TestQueueOverride(t *testing.T) {
+ b, _, _ := setup(t, procs.Map{})
+ holder := must(t, acquire(b, Request{Engine: "holder", Priority: Foreground, Pinned: true}))
+ a := acquire(b, Request{Engine: "a", Priority: Interactive})
+ time.Sleep(10 * time.Millisecond)
+ bb := acquire(b, Request{Engine: "bb", Priority: Interactive})
+ time.Sleep(10 * time.Millisecond)
+ c := acquire(b, Request{Engine: "c", Priority: Batch})
+ time.Sleep(20 * time.Millisecond)
+ if got := queueOrder(b); !slices.Equal(got, []string{"a", "bb", "c"}) {
+ t.Fatalf("queue = %v, want priority then arrival", got)
+ }
+
+ if _, err := b.Queue(waiterID(t, b, "bb"), QueueBump, ""); err != nil {
+ t.Fatal(err)
+ }
+ if got := queueOrder(b); !slices.Equal(got, []string{"bb", "a", "c"}) {
+ t.Fatalf("after bump bb: %v", got)
+ }
+ detail, err := b.Queue(waiterID(t, b, "c"), QueuePrio, "foreground")
+ if err != nil || detail != "priority batch -> foreground by the operator" {
+ t.Fatalf("prio c: %q, %v", detail, err)
+ }
+ if got := queueOrder(b); !slices.Equal(got, []string{"c", "bb", "a"}) {
+ t.Fatalf("after prio c: %v", got)
+ }
+
+ if _, err := b.Queue(waiterID(t, b, "c"), QueuePrio, "hold"); err == nil {
+ t.Fatal("prio hold was accepted")
+ }
+ if _, err := b.Queue("nope-9", QueueBump, ""); !IsNotFound(err) {
+ t.Fatalf("unknown id: %v, want not found", err)
+ }
+
+ if _, err := b.Queue(waiterID(t, b, "a"), QueueDrop, ""); err != nil {
+ t.Fatal(err)
+ }
+ select {
+ case r := <-a:
+ if !errors.Is(r.err, ErrRefused) || !strings.Contains(r.err.Error(), "dropped from the queue by the operator") {
+ t.Fatalf("dropped a: %v", r.err)
+ }
+ case <-time.After(time.Second):
+ t.Fatal("dropped waiter still waiting")
+ }
+
+ var overrides []string
+ events, _, _ := b.EventLog().Events(0, false)
+ for _, e := range events {
+ if e.Kind == proto.EvQueueOverride {
+ overrides = append(overrides, e.Engine+": "+e.Detail)
+ }
+ }
+ want := []string{
+ "bb: moved to the front of its class by the operator",
+ "c: priority batch -> foreground by the operator",
+ "a: dropped by the operator",
+ }
+ if !slices.Equal(overrides, want) {
+ t.Fatalf("override events = %q, want %q", overrides, want)
+ }
+
+ holder.Release()
+ b.Tick(context.Background())
+ l := must(t, c)
+ pending(t, bb)
+ l.Release()
+ must(t, bb)
+}
diff --git a/internal/broker/park_test.go b/internal/broker/park_test.go
index 22c1a85a..91d736e8 100644
--- a/internal/broker/park_test.go
+++ b/internal/broker/park_test.go
@@ -987,3 +987,80 @@ func TestWholeMachineWaiterRefusesThePark(t *testing.T) {
t.Fatalf("park without a whole-machine waiter refused: %s", why)
}
}
+
+// runOnce leases engine in group, measures it at used on the card, and frees
+// it, so the broker has learned what one run of it holds.
+func runOnce(t *testing.T, b *Broker, c *card, engine, group string, root, eng int, used uint64) {
+ t.Helper()
+ l := must(t, acquire(b, Request{Engine: engine, Group: group}))
+ l.Attach(root)
+ c.set(gpu.Proc{PID: eng, Name: engine, Used: used})
+ b.Tick(context.Background())
+ l.Release()
+ c.set()
+ b.Tick(context.Background())
+}
+
+// A whole-card waiter is granted beside a parked residual only when what it
+// has held fits beside it. Dirk-27B holds 23.4 GiB of a 24 GiB card, so a
+// grant beside a kept-up ComfyUI's 768 MiB context fails to load; it evicts
+// the parked engine instead. A small model of the same llama-swap still goes
+// beside it, and an engine never measured keeps today's grant.
+func TestWholeCardWaiterBesideParkedOnlyWhenItFits(t *testing.T) {
+ tree := procs.Fake{
+ Parents: map[int]int{101: 100, 100: 1, 301: 300, 300: 1, 401: 400, 400: 1},
+ Starts: map[int]uint64{100: 1, 101: 1, 300: 1, 301: 1, 400: 1, 401: 1},
+ }
+ b, c, _ := parkSetup(t, tree, 32*GiB, 0)
+ runOnce(t, b, c, "Dirk-27B", "llama-swap", 300, 301, 23*GiB+400<<20)
+ runOnce(t, b, c, "small-8b", "llama-swap", 400, 401, 6*GiB)
+ comfy := warmLease(t, b, c, "comfyui", 8*GiB, 100, 101)
+ c.set(gpu.Proc{PID: 101, Name: "comfyui-engine", Used: 768 << 20})
+ b.Tick(context.Background())
+
+ small := acquire(b, Request{Engine: "small-8b", Group: "llama-swap"})
+ must(t, small).Release()
+ b.Tick(context.Background())
+ select {
+ case why := <-comfy.Drains():
+ t.Fatalf("a model that fits beside the residual evicted it: %q", why)
+ default:
+ }
+
+ unknown := acquire(b, Request{Engine: "never-run"})
+ must(t, unknown).Release()
+ b.Tick(context.Background())
+
+ dirk := acquire(b, Request{Engine: "Dirk-27B", Group: "llama-swap"})
+ pending(t, dirk)
+ if why := drained(t, comfy); !strings.Contains(why, "Dirk-27B") {
+ t.Fatalf("eviction drain reason %q, want Dirk-27B's need", why)
+ }
+ comfy.Release()
+ c.set()
+ b.Tick(context.Background())
+ must(t, dirk)
+}
+
+// Measured on aphotic (RTX 4090, 24564 MiB): Dirk-27B held 23710 MiB alone and
+// served; beside a parked ComfyUI's 456 MiB it reached 24147 MiB on the card and
+// died loading, though its peak plus the residual is 398 MiB under the total. A
+// model that fills the card needs headroom past its steady peak while it loads.
+func TestFillingModelKeepsLoadHeadroomBesideParked(t *testing.T) {
+ tree := procs.Fake{
+ Parents: map[int]int{101: 100, 100: 1, 301: 300, 300: 1},
+ Starts: map[int]uint64{100: 1, 101: 1, 300: 1, 301: 1},
+ }
+ b, c, _ := parkSetup(t, tree, 32*GiB, 0)
+ c.total = 24564 << 20
+ runOnce(t, b, c, "Dirk-27B", "llama-swap", 300, 301, 23710<<20)
+ comfy := warmLease(t, b, c, "comfyui", 8*GiB, 100, 101)
+ c.set(gpu.Proc{PID: 101, Name: "comfyui-engine", Used: 456 << 20})
+ b.Tick(context.Background())
+
+ dirk := acquire(b, Request{Engine: "Dirk-27B", Group: "llama-swap"})
+ pending(t, dirk)
+ if why := drained(t, comfy); !strings.Contains(why, "Dirk-27B") {
+ t.Fatalf("eviction drain reason %q", why)
+ }
+}
diff --git a/internal/broker/status.go b/internal/broker/status.go
index bf932105..a5acc929 100644
--- a/internal/broker/status.go
+++ b/internal/broker/status.go
@@ -86,6 +86,9 @@ func (b *Broker) Status() proto.Status {
GPUPIDs: pids, Used: l.used(),
GTTUsed: l.gttUsed(), GTTContested: l.contestedGTT,
Contested: l.contested, Age: now.Sub(since).Seconds(), Pinned: l.pinned,
+ Window: l.window.String(),
+ Booking: l.booking,
+ Training: l.training,
}
if l.state == parked {
pl.ParkedSince = parkedAge(l, now)
diff --git a/internal/broker/training_test.go b/internal/broker/training_test.go
new file mode 100644
index 00000000..a7288dc2
--- /dev/null
+++ b/internal/broker/training_test.go
@@ -0,0 +1,199 @@
+// Copyright 2026 Trevin Tindall (T-Crypt)
+// SPDX-License-Identifier: Apache-2.0
+
+package broker
+
+import (
+ "context"
+ "errors"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/T-Crypt/berth/internal/procs"
+ "github.com/T-Crypt/berth/internal/runs"
+)
+
+// refusedNow is the refusal a request gets at once, well inside its own wait.
+func refusedNow(t *testing.T, what string, ch chan result) string {
+ t.Helper()
+ select {
+ case r := <-ch:
+ if !errors.Is(r.err, ErrRefused) {
+ t.Fatalf("%s: got lease=%v err=%v, want a refusal", what, r.l, r.err)
+ }
+ return r.err.Error()
+ case <-time.After(time.Second):
+ t.Fatalf("%s was not refused at once", what)
+ }
+ return ""
+}
+
+// While a training holds the card, every other request is refused at once,
+// a hold included, so nothing waits on the card or drains the run. Another
+// training queues and gets the card when the first one ends.
+func TestTrainingRefusesEverythingButATraining(t *testing.T) {
+ b, _, _ := setup(t, procs.Map{})
+ train := must(t, acquire(b, Request{Engine: "train-a", Training: true}))
+
+ why := refusedNow(t, "a foreground request", acquire(b, Request{Engine: "chat", Priority: Foreground, Wait: time.Hour}))
+ if !strings.Contains(why, "locked for training by train-a") {
+ t.Errorf("refusal %q does not name the training", why)
+ }
+ refusedNow(t, "a hold", acquire(b, Request{Engine: "hold:rogue", Priority: Hold, Pinned: true, Wait: time.Hour}))
+ refusedNow(t, "a pinned run", acquire(b, Request{Engine: "smoke", Priority: Foreground, Pinned: true, Wait: time.Hour}))
+
+ next := acquire(b, Request{Engine: "train-b", Training: true})
+ pending(t, next)
+ notDrained(t, train)
+
+ train.Release()
+ granted(t, b, "train-b", next)
+}
+
+// A training that arrives refuses the waiters already queued for the card,
+// so a request does not sit out its wait behind a run that can take hours.
+// The holder it replaces is drained as a hold drains it.
+func TestTrainingLocksOutTheWaitersAlreadyQueued(t *testing.T) {
+ b, _, _ := setup(t, procs.Map{})
+ holder := must(t, acquire(b, Request{Engine: "dirk", Priority: Foreground}))
+ chat := acquire(b, Request{Engine: "chat", Priority: Interactive, Wait: time.Hour})
+ pending(t, chat)
+
+ train := acquire(b, Request{Engine: "train-a", Training: true})
+ refusedNow(t, "the queued chat", chat)
+ drained(t, holder)
+ holder.Release()
+ granted(t, b, "train-a", train)
+}
+
+// A training never drains a pinned holder (a game, the operator's hold): it
+// waits for it, and the card is locked for everything else meanwhile.
+func TestTrainingWaitsForAPinnedHolderAndLocksTheCard(t *testing.T) {
+ b, _, _ := setup(t, procs.Map{})
+ game := must(t, acquire(b, Request{Engine: "game", Priority: Hold, Pinned: true}))
+ train := acquire(b, Request{Engine: "train-a", Training: true})
+ pending(t, train)
+ notDrained(t, game)
+
+ refusedNow(t, "a request while the training waits", acquire(b, Request{Engine: "chat", Priority: Foreground, Wait: time.Hour}))
+ game.Release()
+ granted(t, b, "train-a", train)
+}
+
+// A training has no wait deadline, whatever the request or the node says.
+func TestTrainingWaitsWithNoDeadline(t *testing.T) {
+ b, _, _ := setup(t, procs.Map{})
+ for _, r := range []Request{{Training: true}, {Training: true, Wait: time.Second}} {
+ if got := b.waitFor(r); got >= 0 {
+ t.Errorf("waitFor(%+v) = %v, want no deadline", r, got)
+ }
+ }
+}
+
+// The status names a training lease as one, at hold priority and pinned.
+func TestStatusCarriesTheTraining(t *testing.T) {
+ b, _, _ := setup(t, procs.Map{})
+ must(t, acquire(b, Request{Engine: "train-a", Training: true}))
+ ls := b.Status().Leases
+ if len(ls) != 1 || !ls[0].Training || !ls[0].Pinned || ls[0].Priority != "hold" {
+ t.Fatalf("leases = %+v, want train-a as a pinned hold training", ls)
+ }
+}
+
+// On a node with several cards the lock covers the cards the training
+// holds: a request there is refused at once, and another card keeps serving.
+func TestTrainingOnSeveralCardsLocksOnlyItsCards(t *testing.T) {
+ bs := node(t, "gpu0", "gpu1", "gpu2")
+ r := <-acquireAll(bs[:2], Request{Engine: "train-a", Training: true})
+ if r.err != nil || len(r.ls) != 2 {
+ t.Fatalf("grant = %+v", r)
+ }
+
+ refusedNow(t, "chat on gpu1", acquire(bs[1], Request{Engine: "chat", Priority: Foreground, Wait: time.Hour}))
+ gang := acquireAll(bs[1:], Request{Engine: "maya", Priority: Foreground, Wait: time.Hour})
+ select {
+ case got := <-gang:
+ if !errors.Is(got.err, ErrRefused) {
+ t.Fatalf("maya on gpu1+gpu2: %+v, want a refusal", got)
+ }
+ case <-time.After(time.Second):
+ t.Fatal("maya on gpu1+gpu2 was not refused at once")
+ }
+ must(t, acquire(bs[2], Request{Engine: "voice", Priority: Foreground}))
+ for _, l := range r.ls {
+ notDrained(t, l)
+ }
+}
+
+// A training on several cards that arrives refuses a gang already queued on
+// one of them.
+func TestTrainingLocksOutAQueuedGang(t *testing.T) {
+ bs := node(t, "gpu0", "gpu1")
+ holder := must(t, acquire(bs[0], Request{Engine: "game", Priority: Hold, Pinned: true}))
+ gang := acquireAll(bs, Request{Engine: "maya", Priority: Foreground, Wait: time.Hour})
+ time.Sleep(50 * time.Millisecond)
+
+ train := acquireAll(bs, Request{Engine: "train-a", Training: true})
+ select {
+ case got := <-gang:
+ if !errors.Is(got.err, ErrRefused) {
+ t.Fatalf("maya: %+v, want a refusal", got)
+ }
+ case <-time.After(time.Second):
+ t.Fatal("the queued gang was not refused when the training arrived")
+ }
+ holder.Release()
+ tickAll(bs)
+ select {
+ case got := <-train:
+ if got.err != nil || len(got.ls) != 2 {
+ t.Fatalf("training grant = %+v", got)
+ }
+ case <-time.After(2 * time.Second):
+ t.Fatal("the training was not granted once the game ended")
+ }
+}
+
+// trainingLost is a node whose training (pid 200) lost the last daemon: its
+// record is on disk, and this daemon's re-attach window is long past.
+func trainingLost(t *testing.T) *Broker {
+ t.Helper()
+ b, _, _ := setup(t, procs.Fake{Starts: map[int]uint64{200: 9}})
+ b.cfg.RunsDir = t.TempDir()
+ b.cfg.ReattachWindow = time.Nanosecond
+ if _, err := runs.Write(b.cfg.RunsDir, runs.Record{
+ Engine: "train-a", PID: 200, Start: 9, Started: time.Now(), Reason: runs.ReasonLost, Training: true,
+ }); err != nil {
+ t.Fatal(err)
+ }
+ time.Sleep(time.Millisecond)
+ b.Tick(context.Background())
+ return b
+}
+
+// A training that has not re-attached yet still locks the card after a
+// daemon restart, with no time limit: other requests are refused at once.
+func TestTrainingLocksTheCardUntilItReattaches(t *testing.T) {
+ b := trainingLost(t)
+ why := refusedNow(t, "a request before the training re-attaches", acquire(b, Request{Engine: "chat", Priority: Foreground, Wait: time.Hour}))
+ if !strings.Contains(why, "locked for training by train-a") {
+ t.Errorf("refusal %q does not name the training", why)
+ }
+
+ l, err := b.Reattach(Request{Engine: "train-a", Training: true}, 200, 9)
+ if err != nil {
+ t.Fatalf("the training's re-attach: %v", err)
+ }
+ refusedNow(t, "a request after the training re-attached", acquire(b, Request{Engine: "chat", Priority: Foreground, Wait: time.Hour}))
+ l.Release()
+}
+
+// The lock ends with the training: once its process is gone, the record
+// goes and the card serves again.
+func TestTrainingRecordLocksOnlyWhileItsProcessLives(t *testing.T) {
+ b := trainingLost(t)
+ b.tree = procs.Fake{Starts: map[int]uint64{}}
+ b.Tick(context.Background())
+ must(t, acquire(b, Request{Engine: "chat", Priority: Foreground}))
+}
diff --git a/internal/broker/window_test.go b/internal/broker/window_test.go
index 2374637f..74b04a4f 100644
--- a/internal/broker/window_test.go
+++ b/internal/broker/window_test.go
@@ -197,3 +197,19 @@ func TestWindowWaiterOnSeveralCardsOutlastsTheDefaultWait(t *testing.T) {
t.Fatal("nightly is not on both cards")
}
}
+
+// The status names the hours a lease is held to, waiting or granted, and
+// leaves a lease with no window without one.
+func TestStatusCarriesTheLeaseWindow(t *testing.T) {
+ b, clk := startTurnBroker(t, procs.Map{})
+ clk.set(at2330)
+ must(t, acquire(b, Request{Engine: "nightly", Priority: Batch, Needs: 8 * GiB, Window: night}))
+ must(t, acquire(b, Request{Engine: "chat", Priority: Interactive, Needs: 4 * GiB}))
+ got := map[string]string{}
+ for _, l := range b.Status().Leases {
+ got[l.Engine] = l.Window
+ }
+ if got["nightly"] != "23:00-07:00" || got["chat"] != "" {
+ t.Errorf("lease windows = %v, want nightly 23:00-07:00 and chat none", got)
+ }
+}
diff --git a/internal/broker/yield.go b/internal/broker/yield.go
new file mode 100644
index 00000000..378f1a53
--- /dev/null
+++ b/internal/broker/yield.go
@@ -0,0 +1,57 @@
+// Copyright 2026 Trevin Tindall (T-Crypt)
+// SPDX-License-Identifier: Apache-2.0
+
+package broker
+
+import (
+ "fmt"
+ "sort"
+ "time"
+)
+
+// Yield tells the run of every leased engine named engine (by name or group)
+// that its job has let go of the card from inside, for d, or that it has
+// taken it back when d is not positive (#299). A ComfyUI job that waits on an
+// LLM never reaches its job boundary, so a drain for that LLM waits on the
+// job while the job waits on the drain. While the yield lasts, a drain finds
+// the job at its boundary and parks or stops the engine as it would there.
+// The broker's books do not move on the yield: the card read after the park
+// still decides what is free (rule 1). It answers the leases it told.
+func (b *Broker) Yield(engine string, d time.Duration) ([]string, error) {
+ b.mu.Lock()
+ defer b.mu.Unlock()
+ var ids []string
+ for _, l := range b.leases {
+ // A parked or parking lease hears the end too: its run must stop
+ // counting the job as at its boundary before the job runs again.
+ if l.pid == 0 || (l.state != active && l.state != draining && l.state != parking && l.state != parked) ||
+ (l.engine != engine && l.group != engine) {
+ continue
+ }
+ if l.gang != nil {
+ // A share cannot park yet, so a drain that skipped the job
+ // boundary would stop the engine mid-job.
+ return nil, fmt.Errorf("%w: %s runs on several cards, and a lease on several cards cannot park, so it cannot yield", ErrRefused, engine)
+ }
+ select {
+ case <-l.yieldCh:
+ default:
+ }
+ l.yieldCh <- d
+ ids = append(ids, l.id)
+ }
+ if len(ids) == 0 {
+ return nil, notFound(fmt.Sprintf("no running engine named %s under a lease", engine))
+ }
+ sort.Strings(ids)
+ if d > 0 {
+ b.log.Info("yield", "engine", engine, "for", d, "leases", ids)
+ } else {
+ b.log.Info("yield ended", "engine", engine, "leases", ids)
+ }
+ return ids, nil
+}
+
+// Yields carries each yield for this lease's run, newest first: how long it
+// lasts, or 0 when it ended.
+func (l *Lease) Yields() <-chan time.Duration { return l.l.yieldCh }
diff --git a/internal/broker/yield_test.go b/internal/broker/yield_test.go
new file mode 100644
index 00000000..45aea512
--- /dev/null
+++ b/internal/broker/yield_test.go
@@ -0,0 +1,56 @@
+// Copyright 2026 Trevin Tindall (T-Crypt)
+// SPDX-License-Identifier: Apache-2.0
+
+package broker
+
+import (
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/T-Crypt/berth/internal/procs"
+)
+
+// A share of a lease on several cards cannot park, so a drain that skipped
+// the job boundary would stop the engine mid-job. The yield is refused with
+// the reason, and its run never hears it.
+func TestYieldRefusesALeaseOnSeveralCards(t *testing.T) {
+ bs := node(t, "gpu0", "gpu1")
+ got := <-acquireAll(bs, Request{Engine: "comfyui", Priority: Foreground})
+ if got.err != nil {
+ t.Fatal(got.err)
+ }
+ for _, l := range got.ls {
+ l.Attach(100)
+ }
+ for _, b := range bs {
+ _, err := b.Yield("comfyui", time.Minute)
+ if err == nil || IsNotFound(err) || !strings.Contains(err.Error(), "several cards") {
+ t.Fatalf("yield on %s = %v, want a refusal that names the cards", b.cfg.Card, err)
+ }
+ }
+ for _, l := range got.ls {
+ select {
+ case d := <-l.Yields():
+ t.Fatalf("the run heard a yield of %s it cannot act on", d)
+ default:
+ }
+ }
+}
+
+// A lease on one card hears its yield, and the newest yield replaces one
+// the run has not read yet.
+func TestYieldReachesItsRunNewestFirst(t *testing.T) {
+ b, _, _ := setup(t, procs.Fake{Starts: map[int]uint64{100: 7}})
+ l := must(t, acquire(b, Request{Engine: "comfyui", Priority: Foreground}))
+ l.Attach(100)
+ if _, err := b.Yield("comfyui", time.Minute); err != nil {
+ t.Fatal(err)
+ }
+ if _, err := b.Yield("comfyui", 0); err != nil {
+ t.Fatal(err)
+ }
+ if d := <-l.Yields(); d != 0 {
+ t.Fatalf("the run heard %s, want the newer end", d)
+ }
+}
diff --git a/internal/cli/cli.go b/internal/cli/cli.go
index 5901bf80..8de01294 100644
--- a/internal/cli/cli.go
+++ b/internal/cli/cli.go
@@ -63,7 +63,7 @@ func Run(env Env) int {
return c.fail(Systemf("unknown command %q", args[0]).
WithHint("run \"berth help\" for the command list"))
}
- rest := args[1:]
+ rest, path := args[1:], cmd.Name
if len(cmd.Sub) > 0 {
if len(rest) == 0 {
c.groupHelp(cmd, c.Err)
@@ -80,8 +80,9 @@ func Run(env Env) int {
WithHint("run \"berth help %s\" for the list", cmd.Name))
}
cmd, rest = sub, rest[1:]
+ path += " " + cmd.Name
}
- c.Cmd = cmd
+ c.Cmd, c.Path = cmd, path
c.Args = rest
if e := c.setFormat(c.rawFormat); e != nil {
return c.fail(e)
@@ -138,7 +139,7 @@ func (c *Context) helpCommand(args []string) int {
return c.fail(Systemf("unknown command %q", args[0]).
WithHint("run \"berth help\" for the command list"))
}
- rest := args[1:]
+ rest, path := args[1:], cmd.Name
if len(cmd.Sub) > 0 {
if len(rest) == 0 {
c.groupHelp(cmd, c.Out)
@@ -150,11 +151,12 @@ func (c *Context) helpCommand(args []string) int {
WithHint("run \"berth help %s\" for the list", args[0]))
}
cmd, rest = sub, rest[1:]
+ path += " " + cmd.Name
}
if len(rest) > 0 {
- return c.fail(Systemf("berth %s takes no further argument %q", cmd.Name, rest[0]))
+ return c.fail(Systemf("berth %s takes no further argument %q", path, rest[0]))
}
- c.Cmd = cmd
+ c.Cmd, c.Path = cmd, path
c.Args = []string{"-h"}
return cmd.Run(c)
}
diff --git a/internal/cli/commands.go b/internal/cli/commands.go
index bf667452..a1bbfb4a 100644
--- a/internal/cli/commands.go
+++ b/internal/cli/commands.go
@@ -52,6 +52,22 @@ materialised, and the environment that decides which file that is.
},
},
},
+ {
+ Name: "contract",
+ Summary: "one engine's terms: what berth.toml declares and what Berth measured",
+ Args: "ENGINE [--config PATH] [--costs PATH] [--socket P] [--format F]",
+ Help: `Print one engine's terms in one place: its priority, the VRAM a lease asks for,
+the last run's measured peak, the RAM it needs and holds parked, its minimum
+turn and window, and how long a switch to it takes from each tier. Every
+number says where it came from: declared in berth.toml, Berth's default,
+measured on the card or by the gateway, or unknown.
+
+ENGINE is an [[engine]] name, an engine berthd discovered, or a name Berth
+measured leases under (a llama-swap model's berth run --as). Read only: it
+changes nothing, and without berthd it prints the declared terms.`,
+ Formats: []Format{FormatHuman, FormatJSON, FormatYAML},
+ Run: runContract,
+ },
{
Name: "civitai",
Summary: "search and browse CivitAI, for finding things to pull",
@@ -202,10 +218,43 @@ exits; with it, new events stream until you interrupt it.`,
notifications. The error level, the default, shows refused requests, memory
that never came back, stowaways and a parked engine loaded behind Berth's
back. info adds every switch, drain, park and resume. [notify] enabled = false
-in berth.toml turns it off; [notify] level sets the level. Linux only for now:
-it uses notify-send, so any desktop or notification daemon shows it.`,
+in berth.toml turns it off; [notify] level sets the level. On Linux it uses
+notify-send, so any desktop or notification daemon shows it. On Windows it is a
+tray icon whose notifications are Windows' own toasts; a click opens the harbor.`,
Run: runNotify,
},
+ {
+ Name: "why",
+ Summary: "one lease's story: why it waited, was granted, drained or refused",
+ Args: "ID|ENGINE [--socket P] [--format F]",
+ Help: `Print every event berthd kept about one lease, oldest first, in plain words:
+when it asked for the card, each reason it waited, the grant, each drain and
+who it was for, the release, and the berthd restarts it lived through. ENGINE
+picks that engine's newest lease. A lease that berthd still holds ends with
+where it stands now; one a restart cut off names the lease its engine
+re-attached as.
+
+berthd keeps its events in events.jsonl in the state directory (--events), so
+the story reads across restarts and deploys, back to the last 1000 events.`,
+ Formats: []Format{FormatHuman, FormatJSON, FormatYAML},
+ Run: runWhy,
+ },
+ {
+ Name: "fit",
+ Summary: "where a request for an engine or model would run now, and what the start costs",
+ Args: "NAME [--explain] [--config PATH] [--costs PATH] [--socket P] [--format F]",
+ Help: `Rank what it takes to bring NAME up right now, cheapest first: a resume from
+RAM, a free card, a drain of what holds the card, or a wait while a VM, a
+cordon or a pinned lease (a game, a hold, a training) keeps it. NAME is an
+engine, or a model that engines list in serves; for a model every engine that
+serves it is ranked in berth.toml's order, the way the gateway picks one.
+
+Each candidate names its reason, the tier its weights sit in, and the start
+time the gateway measured from that tier, or unknown when it has none.
+--explain lists every candidate. Read only: Fit advises, berthd decides.`,
+ Formats: []Format{FormatHuman, FormatJSON, FormatYAML},
+ Run: runFit,
+ },
{
Name: "export",
Summary: "dump the harbor's stored history: requests and samples",
@@ -365,6 +414,62 @@ Gated and private models read a token from $HF_TOKEN or $CIVITAI_API_KEY.`,
Formats: []Format{FormatHuman, FormatJSON, FormatYAML},
Run: runPull,
},
+ {
+ Name: "queue",
+ Summary: "the waiters in grant order; bump, re-rank or drop one",
+ Args: "[bump ID | prio ID batch|interactive|foreground | drop ID] [--socket P] [--format F]",
+ Help: `Without arguments, list every waiter in the order berthd will grant them:
+priority first, then arrival. Nobody has to manage the queue; these are for
+when you want it otherwise.
+
+ bump ID move the waiter to the front of its priority class
+ prio ID P move it to another class: batch, interactive or foreground
+ drop ID refuse it now, the way a timeout would
+
+A hold is a pin, not a queue place, and a training's place is fixed by the
+card's lock, so neither can be set here. Every change is journaled as a
+lease.queue_override event (berth events).`,
+ Formats: []Format{FormatHuman, FormatJSON, FormatYAML},
+ Run: runQueue,
+ },
+ {
+ Name: "schedule",
+ Summary: "the card's scheduler: what runs, what waits, what is booked; book, move or cancel",
+ Args: "[book --who W --at T --for D [--engine E] [--kind run|priority|down] [--priority P] [--days N] [--earliest] [--note N] | move ID [--at T] [--for D] | cancel ID] [--socket P] [--format F]",
+ Help: `Without arguments, show the scheduler: the leases that hold a card, the
+waiters in the order berthd grants them, and every booking that has not
+ended. The harbor draws the same bookings on its 7-day board.
+
+A booking gives a card's hours to someone:
+
+ run the engine's turn. Its waiter goes first, it runs at the
+ booking's priority (foreground when none is given), and while
+ it holds the card nothing but a hold drains it.
+ priority the engine runs at another class for these hours, like batch
+ during a training. It claims no time on the card.
+ down downtime. No new work starts and holders drain at the start; a
+ hold and a training stay.
+
+An hour nobody booked runs as always, by priority then arrival, and a booked
+engine that does not show up leaves the card to everyone else. Two runs or
+downtimes on one card may not overlap. --earliest moves a booking to the
+first free hours at or after --at, and --days books the same hours on
+consecutive days. --engine also takes a group, like llama-swap. A class set
+with berth queue prio wins over a booking.
+
+ berth schedule book --who pi-pocket --engine pi-train --at 23:00 --for 5h --days 7 --priority batch
+ berth schedule book --who operator --kind down --at "sun 04:00" --for 1h --note "driver update"
+ berth schedule book --who operator --kind priority --engine llama-swap --priority batch --at 13:00 --for 3h
+ berth schedule move b4 --at 01:00
+ berth schedule cancel b4
+
+Times are the node's clock. Agents use the same scheduler over the gateway:
+GET, POST and DELETE /v1/berth/schedule (docs/gateway.md). Each change is a
+schedule.booked event and each class a booking moves is a
+lease.booking_priority event (berth events).`,
+ Formats: []Format{FormatHuman, FormatJSON, FormatYAML},
+ Run: runSchedule,
+ },
{
Name: "reclaim",
Summary: "what a stuck lease left on the card; --yes stops it",
@@ -431,6 +536,28 @@ The engine's own stdout goes to stdout, so --quiet has nothing to silence here:
berth's notices are diagnostics on stderr and always print.`,
Run: runRun,
},
+ {
+ Name: "train",
+ Summary: "run a training job that nothing automated can interrupt",
+ Args: "--as NAME [--gpu CARDS] -- CMD [ARGS...]",
+ Help: `Run CMD as a training lease: a pinned hold on the whole card, on every card
+--gpu names. It waits for the card with no deadline, takes it from the engines
+that hold it, and keeps it until CMD exits or you press Ctrl-C.
+
+While a training holds a card or waits for it, berthd refuses every other
+request on that card at once: engines, gateway requests, berth run and berth
+hold. Clients get the refusal right away instead of waiting. Another berth train
+waits in line and starts when the first one ends. Other cards keep serving.
+
+A game or a hold that already has the card is not drained: the training waits
+for it. berth train refuses to start when berthd is not running, because the
+lock lives in berthd.
+
+--exclusive (Linux, NVIDIA) also puts the cards in the driver's exclusive-process
+compute mode for the run's life, so a program that opens CUDA without Berth
+cannot open them. It needs a sudo rule for nvidia-smi -c (see docs/leasing.md).`,
+ Run: runTrain,
+ },
{
Name: "skill",
Summary: "the agent skills: teach a coding agent to use Berth and ComfyUI",
diff --git a/internal/cli/config.go b/internal/cli/config.go
index 86e6e86c..4457356f 100644
--- a/internal/cli/config.go
+++ b/internal/cli/config.go
@@ -146,6 +146,9 @@ func engineExtra(e config.Engine) string {
if len(e.NeedsMiB.Bytes()) > 0 {
parts = append(parts, "needs_mib: "+e.NeedsMiB.String())
}
+ if e.LearnSize {
+ parts = append(parts, "learn_size")
+ }
if len(e.Start) > 0 {
parts = append(parts, "start: "+e.Start[0])
}
diff --git a/internal/cli/context.go b/internal/cli/context.go
index a9e37f40..fdfd917e 100644
--- a/internal/cli/context.go
+++ b/internal/cli/context.go
@@ -21,10 +21,13 @@ type View interface {
// Context is one command's run: the streams, the globals parsed once, and the
// format decided for the whole run.
type Context struct {
- Out io.Writer // stdout, quiet-filtered once the flags are parsed
- Err io.Writer // stderr: diagnostics, never suppressed by --quiet
- Args []string // the command's own args
- Cmd *Command
+ Out io.Writer // stdout, quiet-filtered once the flags are parsed
+ Err io.Writer // stderr: diagnostics, never suppressed by --quiet
+ Args []string // the command's own args
+ Cmd *Command
+ // Path is the command as typed after "berth": "status", or "engine
+ // show" for a subcommand, whose Cmd.Name is only "show".
+ Path string
Version string
Format Format
Verbose bool
@@ -42,7 +45,7 @@ type Context struct {
// FlagSet is the command's flag set: the globals first, then whatever the
// command registers on it. -h and --help are the flag package's own.
func (c *Context) FlagSet() *flag.FlagSet {
- fs := flag.NewFlagSet(c.Cmd.Name, flag.ContinueOnError)
+ fs := flag.NewFlagSet(c.Path, flag.ContinueOnError)
// The help and the errors are the framework's to print, from an existing
// Error: the flag package's own output would say everything twice.
fs.SetOutput(io.Discard)
@@ -89,8 +92,8 @@ func (c *Context) Parse(fs *flag.FlagSet) (int, bool) {
return ExitOK, true
}
if err != nil {
- c.fail(Systemf("berth %s: %v", c.Cmd.Name, err).
- WithHint("run \"berth %s -h\" for flags", c.Cmd.Name))
+ c.fail(Systemf("berth %s: %v", c.Path, err).
+ WithHint("run \"berth %s -h\" for flags", c.Path))
return ExitSystem, true
}
c.setFlags = map[string]bool{}
@@ -140,7 +143,7 @@ func (c *Context) noArgs() (int, bool) {
// failUnexpected reports one unexpected positional argument.
func (c *Context) failUnexpected(arg string) (int, bool) {
c.fail(Systemf("unexpected argument %q", arg).
- WithHint("run \"berth %s -h\" for flags", c.Cmd.Name))
+ WithHint("run \"berth %s -h\" for flags", c.Path))
return ExitSystem, true
}
@@ -239,7 +242,7 @@ func (c *Context) help(fs *flag.FlagSet) {
if c.Cmd.Args != "" {
args = " " + c.Cmd.Args
}
- fmt.Fprintf(c.Out, "berth %s%s\n\n%s\n\nFlags:\n", c.Cmd.Name, args, c.Cmd.Help)
+ fmt.Fprintf(c.Out, "berth %s%s\n\n%s\n\nFlags:\n", c.Path, args, c.Cmd.Help)
fs.SetOutput(c.Out)
fs.PrintDefaults()
}
@@ -251,7 +254,7 @@ func (c *Context) setFormat(v string) *Error {
return nil
}
if len(c.Cmd.Formats) == 0 {
- return Systemf("--format does not apply to berth %s", c.Cmd.Name)
+ return Systemf("--format does not apply to berth %s", c.Path)
}
for _, f := range c.Cmd.Formats {
if Format(v) == f {
diff --git a/internal/cli/contract.go b/internal/cli/contract.go
new file mode 100644
index 00000000..9f3630b5
--- /dev/null
+++ b/internal/cli/contract.go
@@ -0,0 +1,185 @@
+// Copyright 2026 Trevin Tindall (T-Crypt)
+// SPDX-License-Identifier: Apache-2.0
+
+package cli
+
+import (
+ "fmt"
+ "sort"
+ "strings"
+ "text/tabwriter"
+ "time"
+
+ "github.com/T-Crypt/berth/internal/config"
+ "github.com/T-Crypt/berth/internal/contract"
+ "github.com/T-Crypt/berth/internal/proto"
+ "github.com/T-Crypt/berth/internal/server"
+ "github.com/T-Crypt/berth/internal/swapcost"
+)
+
+// contractView is one engine's contract.
+type contractView struct {
+ c contract.Contract
+ offline string // why berthd gave no status, "" when it did
+}
+
+func (v contractView) Data() any { return v.c }
+
+func (v contractView) Human(c *Context, ok bool) error {
+ k := v.c
+ where := "not in berth.toml"
+ if k.InConfig {
+ where = "in berth.toml"
+ }
+ if k.Type != "" {
+ where = k.Type + ", " + where
+ }
+ c.Printf("%s (%s)\n", k.Engine, where)
+ if v.offline != "" {
+ c.Bullet("berthd unreachable (%s): measured numbers are unknown", v.offline)
+ }
+ tw := tabwriter.NewWriter(c.Out, 0, 4, 2, ' ', 0)
+ row := func(name, value string, src contract.Source, why string) {
+ fmt.Fprintf(tw, " %s\t%s\t%s\t%s\n", name, value, src, why)
+ }
+ row("priority", k.Priority.Value, k.Priority.Source, k.Priority.Why)
+ row("vram", contractSize(k.VRAM, "whole card"), k.VRAM.Source, k.VRAM.Why)
+ row("last peak", contractSize(k.LastPeak, "-"), k.LastPeak.Source, k.LastPeak.Why)
+ row("ram", contractSize(k.RAM, "-"), k.RAM.Source, k.RAM.Why)
+ row("parked ram", contractSize(k.ParkedRAM, "-"), k.ParkedRAM.Source, k.ParkedRAM.Why)
+ minTurn := "off"
+ if k.MinTurn.Seconds > 0 {
+ minTurn = (time.Duration(k.MinTurn.Seconds) * time.Second).String()
+ }
+ minWhy := ""
+ if k.MinTurn.Source == contract.Measured {
+ minWhy = k.MinTurn.Why
+ }
+ row("min turn", minTurn, k.MinTurn.Source, minWhy)
+ row("window", k.Window.Value, k.Window.Source, "")
+ for _, s := range k.Switch {
+ took, why := "-", s.Took.Why
+ switch s.Took.Source {
+ case contract.Measured:
+ took, why = fmt.Sprintf("%.1f s", s.Took.Seconds), fmt.Sprintf("median of %d", s.Samples)
+ case contract.Default:
+ took = "0 s"
+ }
+ row("switch "+string(s.Tier), took, s.Took.Source, why)
+ }
+ return tw.Flush()
+}
+
+// contractSize is a contract size for people: MiB, or none when it carries no bytes.
+func contractSize(b contract.Bytes, none string) string {
+ if b.Bytes == 0 {
+ return none
+ }
+ return server.MiB(b.Bytes)
+}
+
+func runContract(c *Context) int {
+ fs := c.FlagSet()
+ sock := fs.String("socket", defaultSocket(), "berthd socket ($BERTH_SOCKET)")
+ cfgPath := fs.String("config", config.DefaultPath(), "berth.toml ($BERTH_CONFIG): the engine's declared terms")
+ costs := fs.String("costs", defaultStateFile("swapcost.json"), "the gateway's measured switch times")
+ if code, done := c.ParseInterspersed(fs); done {
+ return code
+ }
+ if len(c.Args) != 1 {
+ return c.fail(Systemf("berth contract takes one engine name").
+ WithHint("for example: berth contract comfyui"))
+ }
+ name := c.Args[0]
+
+ cfg, err := config.Load(*cfgPath)
+ if err != nil {
+ return c.fail(Systemf("%s: %v", *cfgPath, err).WithHint("run berth config check"))
+ }
+ var entry *config.Engine
+ for i := range cfg.Engine {
+ if cfg.Engine[i].Name == name {
+ entry = &cfg.Engine[i]
+ }
+ }
+
+ st, offline := contractStatus(*sock)
+ if entry == nil && !knownTo(st, name) {
+ known := knownNames(cfg, st)
+ e := Failf("no engine named %q in berth.toml or berthd's status", name)
+ if len(known) > 0 {
+ e.WithHint("known: %s", strings.Join(known, ", "))
+ }
+ return c.fail(e)
+ }
+
+ book, err := swapcost.NewBook(*costs)
+ if err != nil {
+ book = nil
+ }
+ return c.Write(contractView{c: contract.Build(name, entry, st, book), offline: offline}, nil)
+}
+
+// contractStatus is berthd's status, or nil and why when it did not answer.
+func contractStatus(sock string) (*proto.Status, string) {
+ conn, raw, err := server.Dial(sock)
+ if err != nil {
+ return nil, err.Error()
+ }
+ defer raw.Close()
+ if err := conn.Send(proto.Msg{Op: proto.OpStatus}); err != nil {
+ return nil, err.Error()
+ }
+ m, err := conn.Recv()
+ if err != nil {
+ return nil, err.Error()
+ }
+ if m.Status == nil {
+ return nil, "empty reply"
+ }
+ return m.Status, ""
+}
+
+// knownTo says whether berthd's status names the engine: discovered, or
+// measured under its own name (a llama-swap model).
+func knownTo(st *proto.Status, name string) bool {
+ for _, n := range statusNames(st) {
+ if n == name {
+ return true
+ }
+ }
+ return false
+}
+
+func statusNames(st *proto.Status) []string {
+ if st == nil {
+ return nil
+ }
+ var out []string
+ for _, e := range st.Engines {
+ out = append(out, e.Name)
+ }
+ for _, f := range st.Footprints {
+ out = append(out, f.Engine)
+ }
+ return out
+}
+
+func knownNames(cfg config.Config, st *proto.Status) []string {
+ seen := map[string]bool{}
+ var out []string
+ for _, e := range cfg.Engine {
+ if !seen[e.Name] {
+ seen[e.Name] = true
+ out = append(out, e.Name)
+ }
+ }
+ for _, n := range statusNames(st) {
+ if !seen[n] {
+ seen[n] = true
+ out = append(out, n)
+ }
+ }
+ sort.Strings(out)
+ return out
+}
diff --git a/internal/cli/contract_test.go b/internal/cli/contract_test.go
new file mode 100644
index 00000000..29aef295
--- /dev/null
+++ b/internal/cli/contract_test.go
@@ -0,0 +1,61 @@
+// Copyright 2026 Trevin Tindall (T-Crypt)
+// SPDX-License-Identifier: Apache-2.0
+
+package cli
+
+import (
+ "encoding/json"
+ "os"
+ "path/filepath"
+ "strings"
+ "testing"
+)
+
+// Without berthd, berth contract still prints what berth.toml declares and
+// marks every measured number unknown; a name nobody knows is refused with
+// the names that exist.
+func TestContractWithoutBerthd(t *testing.T) {
+ home := isolate(t)
+ cfg := filepath.Join(home, "berth.toml")
+ if err := os.WriteFile(cfg, []byte("[[engine]]\nname = \"qwen\"\ntype = \"llamacpp\"\nurl = \"http://127.0.0.1:8080\"\npriority = \"foreground\"\nneeds_mib = 14000\nmin_turn = \"5m\"\n"), 0o600); err != nil {
+ t.Fatal(err)
+ }
+
+ code, out, errOut := runCLI(t, "contract", "qwen", "--config", cfg, "--format", "json")
+ if code != 0 {
+ t.Fatalf("exit %d: %s", code, errOut)
+ }
+ var got struct {
+ InConfig bool `json:"in_config"`
+ VRAM struct {
+ Bytes uint64 `json:"bytes"`
+ Source string `json:"source"`
+ } `json:"vram"`
+ LastPeak struct {
+ Source string `json:"source"`
+ Why string `json:"why"`
+ } `json:"last_peak"`
+ MinTurn struct {
+ Seconds float64 `json:"seconds"`
+ Source string `json:"source"`
+ } `json:"min_turn"`
+ }
+ if err := json.Unmarshal([]byte(out), &got); err != nil {
+ t.Fatalf("%v: %s", err, out)
+ }
+ if !got.InConfig || got.VRAM.Bytes != 14000<<20 || got.VRAM.Source != "declared" ||
+ got.LastPeak.Source != "unknown" || got.LastPeak.Why != "berthd did not answer" ||
+ got.MinTurn.Seconds != 300 || got.MinTurn.Source != "declared" {
+ t.Fatalf("contract = %+v", got)
+ }
+
+ code, out, _ = runCLI(t, "contract", "qwen", "--config", cfg)
+ if code != 0 || !strings.Contains(out, "berthd unreachable") || !strings.Contains(out, "14000 MiB") {
+ t.Fatalf("human (exit %d):\n%s", code, out)
+ }
+
+ code, _, errOut = runCLI(t, "contract", "nope", "--config", cfg)
+ if code == 0 || !strings.Contains(errOut, `no engine named "nope"`) || !strings.Contains(errOut, "known: qwen") {
+ t.Fatalf("unknown name (exit %d): %s", code, errOut)
+ }
+}
diff --git a/internal/cli/daemon.go b/internal/cli/daemon.go
index 16188dce..2a5a6307 100644
--- a/internal/cli/daemon.go
+++ b/internal/cli/daemon.go
@@ -21,6 +21,7 @@ import (
"github.com/T-Crypt/berth/internal/claims"
"github.com/T-Crypt/berth/internal/config"
"github.com/T-Crypt/berth/internal/engines"
+ "github.com/T-Crypt/berth/internal/eventlog"
"github.com/T-Crypt/berth/internal/gpu"
"github.com/T-Crypt/berth/internal/history"
"github.com/T-Crypt/berth/internal/host"
@@ -29,6 +30,7 @@ import (
"github.com/T-Crypt/berth/internal/procs"
"github.com/T-Crypt/berth/internal/proto"
"github.com/T-Crypt/berth/internal/runs"
+ "github.com/T-Crypt/berth/internal/schedule"
"github.com/T-Crypt/berth/internal/server"
"github.com/T-Crypt/berth/internal/suite"
)
@@ -50,8 +52,10 @@ func runDaemon(c *Context) int {
cfgPath := fs.String("config", config.DefaultPath(), "berth.toml: engines to read, /metrics listener ($BERTH_CONFIG); missing = none")
countersFile := fs.String("counters", defaultStateFile("counters.json"), "keeps the collisions board across restarts; \"\" = start at zero every time")
devicesFile := fs.String("devices", defaultStateFile("harbor-devices.json"), "devices paired to act from the harbor (berth pair); \"\" = pairing off")
+ scheduleFile := fs.String("schedule", defaultStateFile("schedule.json"), "the cards' bookings (berth schedule): runs, priority changes, downtime; \"\" = no schedule")
usageFile := fs.String("usage", defaultStateFile("usage.json"), "card-hours per engine, the default juggle ranking; \"\" = memory only")
footprintFile := fs.String("footprint", defaultStateFile("footprint.json"), "VRAM measured per engine across restarts, the estimate behind EST. VRAM; \"\" = memory only")
+ eventsFile := fs.String("events", defaultStateFile("events.jsonl"), "the event stream across restarts, what berth events and berth why read; \"\" = memory only")
if code, done := c.Parse(fs); done {
return code
}
@@ -125,6 +129,16 @@ func runDaemon(c *Context) int {
allowList = splitList(*allow)
}
events, ids, seq, ledger := node.Shared()
+ if *eventsFile != "" {
+ if kept, err := eventlog.Open(*eventsFile); err != nil {
+ log.Error("event file; events are kept in memory only", "path", *eventsFile, "err", err)
+ } else {
+ events = kept
+ defer kept.Close()
+ ids.Store(kept.LastLease())
+ }
+ }
+ events.Emit(proto.Event{At: time.Now(), Kind: proto.EvNodeStarted, Detail: "berth " + c.Version})
procTable := procs.Shared(procs.List, *poll/2)
var cards []*node.Card
var snaps []gpu.Snapshot
@@ -279,10 +293,22 @@ func runDaemon(c *Context) int {
go samplerLoop(ctx, rec, hr, n, watcher, ev, &lastHost, &lastAlerts)
x := server.Extras{Engines: watcher.Snapshot, Host: lastHost.Load, History: rec.Get,
Insights: func() *proto.Insights { return watcher.Insights(time.Now()) },
- Requests: watcher.Requests, Alerts: lastAlerts.Load, Finished: watcher.Finished}
+ Requests: watcher.Requests, Alerts: lastAlerts.Load, Finished: watcher.Finished,
+ Windows: map[string]proto.Window{}}
+ for _, e := range cfg.Engine {
+ x.Windows[e.Name] = e.Window
+ }
if *devicesFile != "" {
x.Pair = &pair.Store{Path: *devicesFile}
}
+ if *scheduleFile != "" {
+ s, err := schedule.Open(*scheduleFile)
+ if err != nil {
+ return c.fail(Systemf("schedule: %v", err).WithHint("fix or move %s; berthd keeps the cards' bookings there", *scheduleFile))
+ }
+ x.Schedule = s
+ n.SetBookings(s.List(time.Now()))
+ }
if err := server.Serve(ctx, ln, n, x, log); err != nil {
return c.fail(Systemf("serve: %v", err))
}
diff --git a/internal/cli/fit.go b/internal/cli/fit.go
new file mode 100644
index 00000000..66bfcdc7
--- /dev/null
+++ b/internal/cli/fit.go
@@ -0,0 +1,166 @@
+// Copyright 2026 Trevin Tindall (T-Crypt)
+// SPDX-License-Identifier: Apache-2.0
+
+package cli
+
+import (
+ "context"
+ "fmt"
+ "net/http"
+ "slices"
+ "strings"
+ "text/tabwriter"
+ "time"
+
+ "github.com/T-Crypt/berth/internal/config"
+ "github.com/T-Crypt/berth/internal/contract"
+ "github.com/T-Crypt/berth/internal/fit"
+ "github.com/T-Crypt/berth/internal/gateway"
+ "github.com/T-Crypt/berth/internal/proto"
+ "github.com/T-Crypt/berth/internal/swapcost"
+)
+
+// fitDoc is berth fit's answer: what Fit would pick for the name, and every
+// candidate it ranked, cheapest first.
+type fitDoc struct {
+ Name string `json:"name" yaml:"name"`
+ Pick fit.Candidate `json:"pick" yaml:"pick"`
+ Candidates []fit.Candidate `json:"candidates" yaml:"candidates"`
+}
+
+type fitView struct {
+ doc fitDoc
+ explain bool
+ offline string
+}
+
+func (v fitView) Data() any { return v.doc }
+
+func (v fitView) Human(c *Context, ok bool) error {
+ p := v.doc.Pick
+ if p.Engine == v.doc.Name {
+ c.Printf("%s: %s, %s\n", p.Engine, p.Path, p.Why)
+ } else {
+ c.Printf("%s: %s, %s, %s\n", v.doc.Name, p.Engine, p.Path, p.Why)
+ }
+ c.Bullet("start %s", startText(p))
+ if v.offline != "" {
+ c.Bullet("berthd unreachable (%s): every card reads as out of reach", v.offline)
+ }
+ if !v.explain && len(v.doc.Candidates) < 2 {
+ return nil
+ }
+ tw := tabwriter.NewWriter(c.Out, 0, 4, 2, ' ', 0)
+ fmt.Fprint(tw, "\nENGINE\tPATH\tTIER\tSTART\tWHY\n")
+ for _, k := range v.doc.Candidates {
+ fmt.Fprintf(tw, "%s\t%s\t%s\t%s\t%s\n", k.Engine, k.Path, k.Tier, startText(k), k.Why)
+ }
+ return tw.Flush()
+}
+
+// startText is a candidate's start for people: the time and where it came from.
+func startText(k fit.Candidate) string {
+ s := k.Start
+ if s.Source == contract.Unknown {
+ return "unknown (" + s.Why + ")"
+ }
+ t := fmt.Sprintf("%.1f s %s", s.Seconds, s.Source)
+ if s.Why != "" {
+ t += " (" + s.Why + ")"
+ }
+ return t
+}
+
+func runFit(c *Context) int {
+ fs := c.FlagSet()
+ sock := fs.String("socket", defaultSocket(), "berthd socket ($BERTH_SOCKET)")
+ cfgPath := fs.String("config", config.DefaultPath(), "berth.toml ($BERTH_CONFIG): the engines and what they serve")
+ costs := fs.String("costs", defaultStateFile("swapcost.json"), "the gateway's measured switch times")
+ explain := fs.Bool("explain", false, "list every candidate, cheapest first, with its path and reason")
+ if code, done := c.ParseInterspersed(fs); done {
+ return code
+ }
+ if len(c.Args) != 1 {
+ return c.fail(Systemf("berth fit takes one engine or model name").
+ WithHint("for example: berth fit comfyui, or berth fit qwen3-30b"))
+ }
+ name := c.Args[0]
+
+ cfg, err := config.Load(*cfgPath)
+ if err != nil {
+ return c.fail(Systemf("%s: %v", *cfgPath, err).WithHint("run berth config check"))
+ }
+ st, offline := contractStatus(*sock)
+ cands := fitCandidates(cfg, name)
+ if len(cands) == 0 && st != nil && slices.ContainsFunc(st.Engines, func(e proto.Engine) bool { return e.Name == name }) {
+ cands = []*config.Engine{{Name: name}}
+ }
+ if len(cands) == 0 {
+ cands = listedBy(context.Background(), st, cfg, name)
+ }
+ if len(cands) == 0 {
+ e := Failf("no engine named %q, and none serves it", name)
+ var known []string
+ for _, en := range cfg.Engine {
+ known = append(known, en.Name)
+ }
+ if st != nil {
+ for _, en := range st.Engines {
+ if !slices.Contains(known, en.Name) {
+ known = append(known, en.Name)
+ }
+ }
+ }
+ if len(known) > 0 {
+ e.WithHint("engines: %s", strings.Join(known, ", "))
+ }
+ return c.fail(e)
+ }
+
+ book, err := swapcost.NewBook(*costs)
+ if err != nil {
+ book = nil
+ }
+ ranked := fit.Rank(st, cands, book)
+ return c.Write(fitView{doc: fitDoc{Name: name, Pick: ranked[0], Candidates: ranked}, explain: *explain, offline: offline}, nil)
+}
+
+// listedBy is a model no berth.toml engine names or serves, the way the
+// gateway resolves it: an engine that is up, not parked, and lists it on its
+// /v1/models. llama-swap lists every model it can load and starts the model
+// under the model's own lease, on the lister's card.
+func listedBy(ctx context.Context, st *proto.Status, cfg config.Config, name string) []*config.Engine {
+ if st == nil {
+ return nil
+ }
+ client := &http.Client{Timeout: 3 * time.Second}
+ for _, e := range st.Engines {
+ if !e.Up || fit.Parked(st, e.Name) || !slices.Contains(gateway.ListModels(ctx, client, e.URL), name) {
+ continue
+ }
+ m := &config.Engine{Name: name}
+ if i := slices.IndexFunc(cfg.Engine, func(c config.Engine) bool { return c.Name == e.Name }); i >= 0 {
+ m.GPU = cfg.Engine[i].GPU
+ }
+ return []*config.Engine{m}
+ }
+ return nil
+}
+
+// fitCandidates is the engines a name means: the [[engine]] of that name, else
+// every engine whose serves lists it, in berth.toml order, which is what the
+// gateway ranks for a request naming that model.
+func fitCandidates(cfg config.Config, name string) []*config.Engine {
+ for i := range cfg.Engine {
+ if cfg.Engine[i].Name == name {
+ return []*config.Engine{&cfg.Engine[i]}
+ }
+ }
+ var out []*config.Engine
+ for i := range cfg.Engine {
+ if slices.Contains(cfg.Engine[i].Serves, name) {
+ out = append(out, &cfg.Engine[i])
+ }
+ }
+ return out
+}
diff --git a/internal/cli/fit_test.go b/internal/cli/fit_test.go
new file mode 100644
index 00000000..2d3e5bb7
--- /dev/null
+++ b/internal/cli/fit_test.go
@@ -0,0 +1,103 @@
+// Copyright 2026 Trevin Tindall (T-Crypt)
+// SPDX-License-Identifier: Apache-2.0
+
+package cli
+
+import (
+ "encoding/json"
+ "fmt"
+ "net/http"
+ "net/http/httptest"
+ "os"
+ "path/filepath"
+ "strings"
+ "testing"
+
+ "github.com/T-Crypt/berth/internal/proto"
+ "github.com/T-Crypt/berth/internal/server"
+)
+
+// berth fit for a model ranks every engine that serves it in berth.toml's
+// order, against a real berthd; without berthd every card is out of reach;
+// a name nothing serves is refused with the engines that exist.
+func TestFitAgainstADaemon(t *testing.T) {
+ home := isolate(t)
+ sock := startDaemon(t)
+ cfg := filepath.Join(home, "berth.toml")
+ toml := "[[engine]]\nname = \"a\"\ntype = \"llamacpp\"\nurl = \"http://127.0.0.1:8080\"\nserves = [\"m\"]\ngpu = \"gpu1\"\n" +
+ "[[engine]]\nname = \"b\"\ntype = \"llamacpp\"\nurl = \"http://127.0.0.1:8081\"\nserves = [\"m\"]\ngpu = \"gpu0\"\n"
+ if err := os.WriteFile(cfg, []byte(toml), 0o600); err != nil {
+ t.Fatal(err)
+ }
+
+ code, out, errOut := runCLI(t, "fit", "m", "--config", cfg, "--socket", sock, "--format", "json")
+ if code != 0 {
+ t.Fatalf("exit %d: %s", code, errOut)
+ }
+ var doc struct {
+ Pick struct {
+ Engine, Path, Why string
+ } `json:"pick"`
+ Candidates []struct {
+ Engine, Path string
+ Start struct{ Source string }
+ } `json:"candidates"`
+ }
+ if err := json.Unmarshal([]byte(out), &doc); err != nil {
+ t.Fatalf("%v: %s", err, out)
+ }
+ // a comes first in berth.toml, but its card is not one berthd reports:
+ // b on the free card wins, and a waits.
+ if doc.Pick.Engine != "b" || doc.Pick.Path != "free" || doc.Pick.Why != "the card is free" ||
+ len(doc.Candidates) != 2 || doc.Candidates[1].Engine != "a" || doc.Candidates[1].Path != "blocked" ||
+ doc.Candidates[0].Start.Source != "unknown" {
+ t.Fatalf("fit = %s", out)
+ }
+
+ code, out, _ = runCLI(t, "fit", "m", "--config", cfg, "--socket", sock)
+ if code != 0 || !strings.HasPrefix(out, "m: b, free, the card is free\n") || !strings.Contains(out, "ENGINE") {
+ t.Fatalf("human (exit %d):\n%s", code, out)
+ }
+
+ code, out, _ = runCLI(t, "fit", "a", "--config", cfg, "--socket", filepath.Join(home, "none.sock"))
+ if code != 0 || !strings.HasPrefix(out, "a: blocked, berthd did not report the card\n") || !strings.Contains(out, "berthd unreachable") {
+ t.Fatalf("offline (exit %d):\n%s", code, out)
+ }
+
+ code, _, errOut = runCLI(t, "fit", "nope", "--config", cfg, "--socket", sock)
+ if code == 0 || !strings.Contains(errOut, `no engine named "nope", and none serves it`) || !strings.Contains(errOut, "engines: a, b") {
+ t.Fatalf("unknown (exit %d): %s", code, errOut)
+ }
+}
+
+// A model no berth.toml engine names or serves is still found the way the
+// gateway finds it: a live llama-swap lists it on /v1/models, so berth fit
+// ranks the model itself on llama-swap's card.
+func TestFitFindsAModelALiveEngineLists(t *testing.T) {
+ home := isolate(t)
+ ls := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ if r.URL.Path != "/v1/models" {
+ http.NotFound(w, r)
+ return
+ }
+ fmt.Fprint(w, `{"object":"list","data":[{"id":"Defiance-27B"},{"id":"Dirk-27B"}]}`)
+ }))
+ t.Cleanup(ls.Close)
+ sock := startDaemonWith(t, server.Extras{Engines: func() []proto.Engine {
+ return []proto.Engine{{Name: "llama-swap", Type: "llama-swap", URL: ls.URL, Up: true, State: "up"}}
+ }})
+ cfg := filepath.Join(home, "berth.toml")
+ if err := os.WriteFile(cfg, []byte("[[engine]]\nname = \"llama-swap\"\ntype = \"llama-swap\"\nurl = \""+ls.URL+"\"\n"), 0o600); err != nil {
+ t.Fatal(err)
+ }
+
+ code, out, errOut := runCLI(t, "fit", "Dirk-27B", "--config", cfg, "--socket", sock)
+ if code != 0 || !strings.HasPrefix(out, "Dirk-27B: free, the card is free\n") {
+ t.Fatalf("exit %d:\n%s%s", code, out, errOut)
+ }
+
+ code, _, errOut = runCLI(t, "fit", "nope", "--config", cfg, "--socket", sock)
+ if code == 0 || !strings.Contains(errOut, `no engine named "nope", and none serves it`) {
+ t.Fatalf("unknown (exit %d): %s", code, errOut)
+ }
+}
diff --git a/internal/cli/help_path_test.go b/internal/cli/help_path_test.go
new file mode 100644
index 00000000..eb7b1636
--- /dev/null
+++ b/internal/cli/help_path_test.go
@@ -0,0 +1,24 @@
+// Copyright 2026 Trevin Tindall (T-Crypt)
+// SPDX-License-Identifier: Apache-2.0
+
+package cli
+
+import (
+ "strings"
+ "testing"
+)
+
+// A subcommand's help and hints name its group: "berth engine show", not
+// "berth show", which is no command.
+func TestSubcommandHelpNamesItsGroup(t *testing.T) {
+ for _, args := range [][]string{{"engine", "show", "--help"}, {"help", "engine", "show"}} {
+ code, out, _ := runCLI(t, args...)
+ if code != 0 || !strings.HasPrefix(out, "berth engine show ") {
+ t.Errorf("berth %s (exit %d) starts %q, want \"berth engine show \"", strings.Join(args, " "), code, firstLine(out))
+ }
+ }
+ _, _, errOut := runCLI(t, "engine", "show", "--nope")
+ if !strings.Contains(errOut, `berth engine show -h`) {
+ t.Errorf("bad flag hint = %q, want it to name berth engine show -h", errOut)
+ }
+}
diff --git a/internal/cli/notify.go b/internal/cli/notify.go
index c53aa1a0..b0167deb 100644
--- a/internal/cli/notify.go
+++ b/internal/cli/notify.go
@@ -5,6 +5,7 @@ package cli
import (
"context"
+ "net"
"os"
"os/signal"
"syscall"
@@ -44,15 +45,19 @@ func runNotify(c *Context) int {
if err != nil {
return c.fail(Failf("%v", err))
}
- sender, err := notify.Desktop()
- if err != nil {
- return c.fail(Systemf("%v", err))
- }
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()
log := c.Log()
log.Info("notifications on", "level", raw, "socket", *sock)
- notify.Follow(ctx, eventSource(*sock), func() int { return nodeCards(*sock) }, sender, level, time.Now(), 2*time.Second, log)
+ harbor := localURL(resolveListen("", cfg.Harbor.Listen, DefaultHarborListen))
+ start := time.Now()
+ err = notify.Serve(ctx, harbor, func(s notify.Sender) {
+ notify.Follow(ctx, eventSource(*sock), func() int { return nodeCards(*sock) }, s, level, start, 2*time.Second, log)
+ stop()
+ })
+ if err != nil {
+ return c.fail(Systemf("%v", err))
+ }
return ExitOK
}
@@ -90,3 +95,12 @@ func eventSource(sock string) notify.Source {
return out, nil
}
}
+
+// localURL is the harbor's address as this machine opens it: a wildcard
+// listen is reached on loopback.
+func localURL(addr string) string {
+ if host, port, err := net.SplitHostPort(addr); err == nil && (host == "" || host == "0.0.0.0" || host == "::") {
+ addr = net.JoinHostPort("127.0.0.1", port)
+ }
+ return "http://" + addr + "/"
+}
diff --git a/internal/cli/notify_logon_other.go b/internal/cli/notify_logon_other.go
new file mode 100644
index 00000000..39c4a966
--- /dev/null
+++ b/internal/cli/notify_logon_other.go
@@ -0,0 +1,10 @@
+// Copyright 2026 Trevin Tindall (T-Crypt)
+// SPDX-License-Identifier: Apache-2.0
+
+//go:build !windows
+
+package cli
+
+// notifyAtLogon has nothing to do here: on Linux berth notify is a user
+// unit, written with the others.
+func notifyAtLogon(string, bool, bool) (string, error) { return "", nil }
diff --git a/internal/cli/notify_logon_windows.go b/internal/cli/notify_logon_windows.go
new file mode 100644
index 00000000..3dff828d
--- /dev/null
+++ b/internal/cli/notify_logon_windows.go
@@ -0,0 +1,49 @@
+// Copyright 2026 Trevin Tindall (T-Crypt)
+// SPDX-License-Identifier: Apache-2.0
+
+package cli
+
+import (
+ "errors"
+
+ "golang.org/x/sys/windows/registry"
+)
+
+// The tray runs in the user's session, so it starts at logon from the
+// user's Run key, never as a service: a service runs in session 0, which
+// has no desktop.
+const (
+ logonRunKey = `Software\Microsoft\Windows\CurrentVersion\Run`
+ notifyValue = "BerthNotify"
+)
+
+// notifyAtLogon starts berth notify at the user's logon, or stops starting
+// it when on is false. It returns the change in words for the report.
+func notifyAtLogon(exe string, on, dryRun bool) (string, error) {
+ if !on {
+ line := `reg delete HKCU\` + logonRunKey + ` /v ` + notifyValue + ` /f`
+ if dryRun {
+ return line, nil
+ }
+ k, err := registry.OpenKey(registry.CURRENT_USER, logonRunKey, registry.SET_VALUE)
+ if err != nil {
+ return line, nil // no Run key: nothing starts it
+ }
+ defer k.Close()
+ if err := k.DeleteValue(notifyValue); err != nil && !errors.Is(err, registry.ErrNotExist) {
+ return line, err
+ }
+ return line, nil
+ }
+ cmd := `"` + exe + `" notify`
+ line := `reg add HKCU\` + logonRunKey + ` /v ` + notifyValue + ` /d "` + cmd + `"`
+ if dryRun {
+ return line, nil
+ }
+ k, _, err := registry.CreateKey(registry.CURRENT_USER, logonRunKey, registry.SET_VALUE)
+ if err != nil {
+ return line, err
+ }
+ defer k.Close()
+ return line, k.SetStringValue(notifyValue, cmd)
+}
diff --git a/internal/cli/notify_test.go b/internal/cli/notify_test.go
new file mode 100644
index 00000000..75feca6f
--- /dev/null
+++ b/internal/cli/notify_test.go
@@ -0,0 +1,19 @@
+// Copyright 2026 Trevin Tindall (T-Crypt)
+// SPDX-License-Identifier: Apache-2.0
+
+package cli
+
+import "testing"
+
+func TestLocalURL(t *testing.T) {
+ for in, want := range map[string]string{
+ "127.0.0.1:9798": "http://127.0.0.1:9798/",
+ "0.0.0.0:50980": "http://127.0.0.1:50980/",
+ "[::]:9798": "http://127.0.0.1:9798/",
+ "10.0.0.200:9798": "http://10.0.0.200:9798/",
+ } {
+ if got := localURL(in); got != want {
+ t.Errorf("localURL(%q) = %q, want %q", in, got, want)
+ }
+ }
+}
diff --git a/internal/cli/queue.go b/internal/cli/queue.go
new file mode 100644
index 00000000..a7a20a13
--- /dev/null
+++ b/internal/cli/queue.go
@@ -0,0 +1,102 @@
+// Copyright 2026 Trevin Tindall (T-Crypt)
+// SPDX-License-Identifier: Apache-2.0
+
+package cli
+
+import (
+ "fmt"
+ "text/tabwriter"
+
+ "github.com/T-Crypt/berth/internal/proto"
+ "github.com/T-Crypt/berth/internal/server"
+)
+
+// queueView is the waiters in the order berthd will grant them.
+type queueView struct {
+ waiting []proto.Lease
+ cards int
+}
+
+func (v queueView) Data() any { return v.waiting }
+
+func (v queueView) Human(c *Context, ok bool) error {
+ if len(v.waiting) == 0 {
+ c.Printf("%s nobody is waiting\n", c.Style.OK(markOK))
+ return nil
+ }
+ tw := tabwriter.NewWriter(c.Out, 0, 4, 2, ' ', 0)
+ fmt.Fprint(tw, "POS\tID\tENGINE\tPRIORITY\t"+claimCardCol(v.cards, "CARD")+"WAITING\n")
+ for i, l := range v.waiting {
+ prio := l.Priority
+ if l.Training {
+ prio += ",training"
+ }
+ fmt.Fprintf(tw, "%d\t%s\t%s\t%s\t%s%s\n", i+1, l.ID, l.Holder(), prio, claimCardCol(v.cards, l.Card), age(l.Age))
+ }
+ return tw.Flush()
+}
+
+// queueChange is the reply to bump, prio and drop.
+type queueChange struct {
+ Lease string `json:"lease" yaml:"lease"`
+ Change string `json:"change" yaml:"change"`
+}
+
+type queueChangeView struct{ c queueChange }
+
+func (v queueChangeView) Data() any { return v.c }
+
+func (v queueChangeView) Human(c *Context, ok bool) error {
+ c.Printf("%s %s: %s\n", c.Style.OK(markOK), v.c.Lease, v.c.Change)
+ return nil
+}
+
+func runQueue(c *Context) int {
+ fs := c.FlagSet()
+ sock := fs.String("socket", defaultSocket(), "berthd socket ($BERTH_SOCKET)")
+ if code, done := c.ParseInterspersed(fs); done {
+ return code
+ }
+ usage := "usage: berth queue | berth queue bump ID | berth queue prio ID batch|interactive|foreground | berth queue drop ID"
+ msg := proto.Msg{Op: proto.OpQueue}
+ switch args := c.Args; {
+ case len(args) == 0:
+ msg = proto.Msg{Op: proto.OpStatus}
+ case len(args) == 2 && (args[0] == "bump" || args[0] == "drop"):
+ msg.Action, msg.Lease = args[0], args[1]
+ case len(args) == 3 && args[0] == "prio":
+ msg.Action, msg.Lease, msg.Priority = args[0], args[1], args[2]
+ default:
+ return c.fail(Failf("berth queue: unexpected arguments %q", args).WithHint("%s", usage))
+ }
+
+ conn, raw, err := server.Dial(*sock)
+ if err != nil {
+ return c.fail(Systemf("berthd unreachable on %s: %v", *sock, err))
+ }
+ defer raw.Close()
+ if err := conn.Send(msg); err != nil {
+ return c.fail(Systemf("berthd on %s: %v", *sock, err))
+ }
+ m, err := conn.Recv()
+ if err != nil {
+ return c.fail(Systemf("no reply from berthd: %v", err))
+ }
+ switch {
+ case m.Op == proto.OpRefused:
+ return c.fail(Failf("%s", m.Reason).WithHint("berth queue lists the waiters and their IDs"))
+ case msg.Op == proto.OpStatus:
+ if m.Status == nil {
+ return c.fail(Systemf("no status from berthd: empty reply"))
+ }
+ w := m.Status.Waiting
+ if w == nil {
+ w = []proto.Lease{}
+ }
+ return c.Write(queueView{waiting: w, cards: scheduledCards(m.Status)}, nil)
+ case m.Op != proto.OpQueue:
+ return c.fail(Systemf("berthd answered %q: it may be older than berth queue", m.Op).
+ WithHint("restart berthd on this build"))
+ }
+ return c.Write(queueChangeView{queueChange{Lease: m.Lease, Change: m.Reason}}, nil)
+}
diff --git a/internal/cli/queue_test.go b/internal/cli/queue_test.go
new file mode 100644
index 00000000..40fd9c73
--- /dev/null
+++ b/internal/cli/queue_test.go
@@ -0,0 +1,94 @@
+// Copyright 2026 Trevin Tindall (T-Crypt)
+// SPDX-License-Identifier: Apache-2.0
+
+package cli
+
+import (
+ "encoding/json"
+ "strings"
+ "testing"
+
+ "github.com/T-Crypt/berth/internal/proto"
+ "github.com/T-Crypt/berth/internal/server"
+)
+
+// ask sends one acquire over the real socket and leaves the connection open,
+// so the lease holds or waits for the rest of the test.
+func ask(t *testing.T, sock string, m proto.Msg) *proto.Conn {
+ t.Helper()
+ conn, raw, err := server.Dial(sock)
+ if err != nil {
+ t.Fatal(err)
+ }
+ t.Cleanup(func() { raw.Close() })
+ m.Op = proto.OpAcquire
+ if err := conn.Send(m); err != nil {
+ t.Fatal(err)
+ }
+ return conn
+}
+
+type queued struct {
+ ID string `json:"id"`
+ Engine string `json:"engine"`
+}
+
+func queueJSON(t *testing.T, sock string) []queued {
+ t.Helper()
+ code, out, errOut := runCLI(t, "queue", "--socket", sock, "--format", "json")
+ if code != 0 {
+ t.Fatalf("berth queue: exit %d: %s", code, errOut)
+ }
+ var w []queued
+ if err := json.Unmarshal([]byte(out), &w); err != nil {
+ t.Fatalf("%v: %s", err, out)
+ }
+ return w
+}
+
+// berth queue lists the waiters in grant order over the real wire, and bump
+// and drop change that order on a real berthd.
+func TestQueueAgainstADaemon(t *testing.T) {
+ isolate(t)
+ sock := startDaemon(t)
+ holder := ask(t, sock, proto.Msg{Engine: "holder", Priority: "foreground", Pinned: true})
+ if m, err := holder.Recv(); err != nil || m.Op != proto.OpGranted {
+ t.Fatalf("holder: %+v, %v", m, err)
+ }
+ ask(t, sock, proto.Msg{Engine: "first", Priority: "interactive", Wait: 30})
+ waitUntil(t, func() bool { return len(queueJSON(t, sock)) == 1 })
+ ask(t, sock, proto.Msg{Engine: "second", Priority: "interactive", Wait: 30})
+ waitUntil(t, func() bool { return len(queueJSON(t, sock)) == 2 })
+
+ w := queueJSON(t, sock)
+ if w[0].Engine != "first" || w[1].Engine != "second" {
+ t.Fatalf("queue = %+v, want first then second", w)
+ }
+
+ code, out, errOut := runCLI(t, "queue", "bump", w[1].ID, "--socket", sock)
+ if code != 0 || !strings.Contains(out, "moved to the front of its class by the operator") {
+ t.Fatalf("bump (exit %d): %s %s", code, out, errOut)
+ }
+ if w = queueJSON(t, sock); w[0].Engine != "second" {
+ t.Fatalf("after bump: %+v", w)
+ }
+
+ code, out, _ = runCLI(t, "queue", "--socket", sock)
+ if code != 0 || !strings.Contains(out, "POS") || strings.Index(out, "second") > strings.Index(out, "first") {
+ t.Fatalf("human list (exit %d):\n%s", code, out)
+ }
+
+ if code, _, errOut = runCLI(t, "queue", "drop", w[1].ID, "--socket", sock); code != 0 {
+ t.Fatalf("drop: exit %d: %s", code, errOut)
+ }
+ waitUntil(t, func() bool { return len(queueJSON(t, sock)) == 1 })
+
+ code, _, errOut = runCLI(t, "queue", "prio", w[0].ID, "hold", "--socket", sock)
+ if code == 0 || !strings.Contains(errOut, "use batch, interactive or foreground") {
+ t.Fatalf("prio hold (exit %d): %s", code, errOut)
+ }
+ code, _, errOut = runCLI(t, "queue", "bump", "nope-9", "--socket", sock)
+ if code == 0 || !strings.Contains(errOut, "no waiter nope-9") {
+ t.Fatalf("unknown id (exit %d): %s", code, errOut)
+ }
+}
diff --git a/internal/cli/run.go b/internal/cli/run.go
index 29fda412..a1d29b84 100644
--- a/internal/cli/run.go
+++ b/internal/cli/run.go
@@ -20,7 +20,18 @@ import (
"github.com/T-Crypt/berth/internal/sysd"
)
-func runRun(c *Context) int {
+func runRun(c *Context) int { return runEngine(c, false) }
+
+// runTrain is berth train: berth run as a training lease, which berthd
+// grants as a pinned hold on the whole card with no wait deadline and which
+// locks its cards against every other request. It refuses to start without
+// berthd, since the lock lives there.
+func runTrain(c *Context) int { return runEngine(c, true) }
+
+// trainSets are the run flags berth train sets itself.
+var trainSets = []string{"priority", "pinned", "needs-mib", "learn-size", "wait", "min-turn", "window", "park", "front", "fail-closed"}
+
+func runEngine(c *Context, training bool) int {
fs := c.FlagSet()
sock := fs.String("socket", defaultSocket(), "berthd socket ($BERTH_SOCKET)")
engine := fs.String("as", "", "engine name shown in status and drain messages (required)")
@@ -30,6 +41,7 @@ func runRun(c *Context) int {
switched := fs.Bool("switched", false, "a person's switch (the gateway sets it for the wake page's Launch and /v1/berth/switch): wins a juggle")
var needs config.MiBs
fs.Var(&needs, "needs-mib", "memory this engine needs in MiB; 0 = the whole card; with --gpu gpu0,gpu1, one size for every card or one per card in the same order (18000,6000)")
+ learnSize := fs.Bool("learn-size", false, "with no --needs-mib: take the most this engine has been measured holding plus a load headroom, not the whole card, so engines that fit together share it; the whole card until a run is measured")
wait := fs.Duration("wait", 0, "how long to wait for a berth; 0 = berthd's default; negative = no deadline")
drainPost := fs.String("drain-post", "", "URL to POST when asked to drain, before signalling (e.g. llama-swap's unload)")
drainSlots := fs.String("drain-slots", "", "engine base URL whose /slots must show idle before a drain unloads or stops it (Strata, or a llama-swap model's /upstream/)")
@@ -50,6 +62,7 @@ func runRun(c *Context) int {
sliceName := fs.String("slice", "", "NAME.slice to start the engine's scope under (systemd user manager, Linux): the slice unit's CPU, memory and swap policy applies to the engine and everything it forks; over the entry's slice")
grace := fs.Duration("grace", 30*time.Second, "time each drain step gets before the next")
failClosed := fs.Bool("fail-closed", false, "do not start CMD without berthd or stop it without a safe boundary")
+ exclusiveMode := fs.Bool("exclusive", false, "berth train only: put the cards in exclusive-process compute mode for the run's life, so no program outside Berth can open them (Linux, NVIDIA, needs a sudo rule for nvidia-smi -c)")
cfgPath := fs.String("config", config.DefaultPath(), "berth.toml: the engine entry matching --as (or --group) supplies the settings a flag does not set; a set flag wins and says so; missing = no defaults ($BERTH_CONFIG)")
if code, done := c.Parse(fs); done {
return code
@@ -59,6 +72,16 @@ func runRun(c *Context) int {
if *engine == "" {
return c.fail(required)
}
+ if *exclusiveMode && !training {
+ return c.fail(Systemf("--exclusive is for berth train: exclusive mode allows one process on the card, which only a training should claim"))
+ }
+ if training {
+ for _, name := range trainSets {
+ if c.Changed(name) {
+ return c.fail(Systemf("berth train sets --%s itself: a training is a pinned hold on the whole card that waits with no deadline and needs berthd", name))
+ }
+ }
+ }
win, err := proto.ParseWindow(*window)
if err != nil {
return c.fail(Systemf("%v", err))
@@ -89,6 +112,9 @@ func runRun(c *Context) int {
drainComfyUI: drainComfy, keepComfyUI: keepComfy, drainMax: drainMax, minTurn: minTurn,
window: &win, container: container, compose: composeFile,
})
+ if c.Changed("learn-size") {
+ e.LearnSize = *learnSize
+ }
if c.Changed("api-key-env") {
e.APIKeyEnv = *keyEnv
}
@@ -139,6 +165,10 @@ func runRun(c *Context) int {
return c.fail(Systemf("%v", err))
}
e.Name = *engine
+ if training {
+ e.Priority, *pinned, *failClosed, *wait = "hold", true, true, -1
+ e.NeedsMiB, e.LearnSize, e.Window, e.MinTurn, e.Park = nil, false, proto.Window{}, config.Duration{}, ""
+ }
if _, err := broker.ParsePriority(e.Priority); err != nil {
return c.fail(Systemf("%v", err))
}
@@ -151,10 +181,13 @@ func runRun(c *Context) int {
if err := config.CheckNeeds(e.NeedsMiB, e.GPU); err != nil {
return c.fail(Systemf("--needs-mib: %v", err))
}
+ if err := config.CheckLearnSize(e); err != nil {
+ return c.fail(Systemf("--learn-size: %v", err))
+ }
// Nothing here goes to stdout: the engine owns it. berth's own notices are
// diagnostics on stderr, and --quiet never hides the unprotected warning.
return runner.Run(runner.OptionsFor(&e, runner.Over{
- Socket: *sock, Front: *frontAddr, Group: *group, Pinned: *pinned, Switched: *switched, Wait: *wait, Grace: *grace,
+ Socket: *sock, Front: *frontAddr, Group: *group, Pinned: *pinned, Training: training, Exclusive: *exclusiveMode, Switched: *switched, Wait: *wait, Grace: *grace,
FailClosed: *failClosed, Argv: argv, Stderr: c.Err, Log: c.Log(),
}))
}
diff --git a/internal/cli/schedule.go b/internal/cli/schedule.go
new file mode 100644
index 00000000..e063e263
--- /dev/null
+++ b/internal/cli/schedule.go
@@ -0,0 +1,188 @@
+// Copyright 2026 Trevin Tindall (T-Crypt)
+// SPDX-License-Identifier: Apache-2.0
+
+package cli
+
+import (
+ "fmt"
+ "slices"
+ "text/tabwriter"
+ "time"
+
+ "github.com/T-Crypt/berth/internal/proto"
+ "github.com/T-Crypt/berth/internal/schedule"
+ "github.com/T-Crypt/berth/internal/server"
+)
+
+type scheduleView struct {
+ b proto.ScheduleBoard
+ cards int
+}
+
+func (v scheduleView) Data() any { return v.b }
+
+func (v scheduleView) Human(c *Context, ok bool) error {
+ tw := tabwriter.NewWriter(c.Out, 0, 4, 2, ' ', 0)
+ card := func(s string) string { return claimCardCol(v.cards, s) }
+ fmt.Fprint(tw, "RUNNING\t"+card("CARD")+"ENGINE\tPRIORITY\tSTATE\tFOR\tBOOKING\n")
+ if len(v.b.Running) == 0 {
+ fmt.Fprint(tw, "-\t"+card("")+"nothing holds a card\t\t\t\t\n")
+ }
+ for _, l := range v.b.Running {
+ fmt.Fprintf(tw, "%s\t%s%s\t%s\t%s\t%s\t%s\n", l.ID, card(l.Card), l.Holder(), l.Priority, l.State, age(l.Age), dash(l.Booking))
+ }
+ fmt.Fprint(tw, "\nQUEUED\t"+card("CARD")+"ENGINE\tPRIORITY\tWAITING\tBOOKING\t\n")
+ if len(v.b.Queued) == 0 {
+ fmt.Fprint(tw, "-\t"+card("")+"nobody is waiting\t\t\t\t\n")
+ }
+ for i, l := range v.b.Queued {
+ fmt.Fprintf(tw, "%d. %s\t%s%s\t%s\t%s\t%s\t\n", i+1, l.ID, card(l.Card), l.Holder(), l.Priority, age(l.Age), dash(l.Booking))
+ }
+ fmt.Fprint(tw, "\nBOOKED\t"+card("CARD")+"KIND\tWHO\tENGINE\tPRIORITY\tWHEN\tNOTE\n")
+ if len(v.b.Bookings) == 0 {
+ fmt.Fprint(tw, "-\t"+card("")+"nothing booked; berth schedule book --help\t\t\t\t\t\n")
+ }
+ for _, x := range v.b.Bookings {
+ when := x.Start.Format("Mon Jan 2 15:04") + "-" + x.End.Format("15:04")
+ if x.End.YearDay() != x.Start.YearDay() {
+ when = x.Start.Format("Mon Jan 2 15:04") + "-" + x.End.Format("Mon 15:04")
+ }
+ if x.Holds(v.b.Now) {
+ when += " (now)"
+ }
+ fmt.Fprintf(tw, "%s\t%s%s\t%s\t%s\t%s\t%s\t%s\n", x.ID, card(x.Card), x.Kind, x.Who, dash(x.Engine), dash(x.Priority), when, x.Note)
+ }
+ return tw.Flush()
+}
+
+type bookingsView struct {
+ op string
+ bs []proto.Booking
+}
+
+func (v bookingsView) Data() any { return v.bs }
+
+func (v bookingsView) Human(c *Context, ok bool) error {
+ verb := map[string]string{proto.OpBook: "booked", proto.OpUnbook: "cancelled", proto.OpRebook: "moved"}[v.op]
+ for _, x := range v.bs {
+ c.Printf("%s %s %s: %s for %s, %s-%s\n", c.Style.OK(markOK), verb, x.ID, x.Kind, x.Who,
+ x.Start.Format("Mon Jan 2 15:04"), x.End.Format("Mon 15:04"))
+ }
+ return nil
+}
+
+func runSchedule(c *Context) int {
+ fs := c.FlagSet()
+ sock := fs.String("socket", defaultSocket(), "berthd socket ($BERTH_SOCKET)")
+ kind := fs.String("kind", "run", "book: run (the engine's turn), priority (change its class) or down (downtime)")
+ who := fs.String("who", "", "book: who the booking is for: a device, an agent, a person")
+ engine := fs.String("engine", "", "book: the engine or group the booking is for (not for down)")
+ prio := fs.String("priority", "", "book: batch, interactive or foreground (run default foreground; priority needs one)")
+ atFlag := fs.String("at", "", "book, move: the start: now, 22:00, tue 22:00, 2026-11-14 22:00")
+ forFlag := fs.Duration("for", 0, "book, move: how long, like 5h or 90m")
+ days := fs.Int("days", 1, "book: the same hours on this many days in a row")
+ earliest := fs.Bool("earliest", false, "book: take the first free hours at or after --at")
+ card := fs.String("card", "", "book: the card (default the first)")
+ note := fs.String("note", "", "book, move: a note the board shows")
+ if code, done := c.ParseInterspersed(fs); done {
+ return code
+ }
+ usage := "usage: berth schedule | berth schedule book --who W --at T --for D [--engine E] [--kind run|priority|down] | berth schedule move ID --at T [--for D] | berth schedule cancel ID"
+ now := time.Now()
+ var msg proto.Msg
+ switch args := c.Args; {
+ case len(args) == 0:
+ msg = proto.Msg{Op: proto.OpStatus}
+ case len(args) == 1 && args[0] == "book":
+ start, err := schedule.ParseAt(*atFlag, now)
+ if err != nil || *forFlag <= 0 {
+ return c.fail(Failf("berth schedule book needs --at and --for").WithHint("%s", usage))
+ }
+ b := proto.Booking{Card: *card, Kind: proto.BookingKind(*kind), Who: *who, Engine: *engine, Priority: *prio,
+ Start: start, End: start.Add(*forFlag), Note: *note}
+ if err := b.Check(now); err != nil {
+ return c.fail(Failf("%v", err).WithHint("%s", usage))
+ }
+ msg = proto.Msg{Op: proto.OpBook, Booking: &b, Days: *days, Earliest: *earliest}
+ case len(args) == 2 && args[0] == "move":
+ msg = proto.Msg{Op: proto.OpRebook}
+ case len(args) == 2 && args[0] == "cancel":
+ msg = proto.Msg{Op: proto.OpUnbook, Booking: &proto.Booking{ID: args[1]}}
+ default:
+ return c.fail(Failf("berth schedule: unexpected arguments %q", args).WithHint("%s", usage))
+ }
+
+ st, code := scheduleStatus(c, *sock)
+ if st == nil {
+ return code
+ }
+ if msg.Op == proto.OpStatus {
+ return c.Write(scheduleView{b: proto.BoardOf(st, now), cards: scheduledCards(st)}, nil)
+ }
+ if msg.Op == proto.OpRebook {
+ i := slices.IndexFunc(st.Bookings, func(x proto.Booking) bool { return x.ID == c.Args[1] })
+ if i < 0 {
+ return c.fail(Failf("no booking %s on the schedule", c.Args[1]).WithHint("berth schedule lists them"))
+ }
+ b := st.Bookings[i]
+ length := b.End.Sub(b.Start)
+ if *atFlag != "" {
+ start, err := schedule.ParseAt(*atFlag, now)
+ if err != nil {
+ return c.fail(Failf("%v", err))
+ }
+ b.Start = start
+ }
+ if *forFlag > 0 {
+ length = *forFlag
+ }
+ b.End = b.Start.Add(length)
+ if *prio != "" {
+ b.Priority = *prio
+ }
+ if *note != "" {
+ b.Note = *note
+ }
+ msg.Booking = &b
+ }
+ m, code := askBerthd(c, *sock, msg)
+ if m == nil {
+ return code
+ }
+ if m.Op == proto.OpRefused {
+ return c.fail(Failf("%s", m.Reason).WithHint("berth schedule shows the board"))
+ }
+ if m.Op != msg.Op {
+ return c.fail(Systemf("berthd answered %q: it may be older than berth schedule", m.Op).WithHint("restart berthd on this build"))
+ }
+ return c.Write(bookingsView{op: msg.Op, bs: m.Bookings}, nil)
+}
+
+func scheduleStatus(c *Context, sock string) (*proto.Status, int) {
+ m, code := askBerthd(c, sock, proto.Msg{Op: proto.OpStatus})
+ if m == nil {
+ return nil, code
+ }
+ if m.Status == nil {
+ return nil, c.fail(Systemf("no status from berthd: empty reply"))
+ }
+ return m.Status, ExitOK
+}
+
+// askBerthd sends one message and reads one reply; nil means it failed and
+// the int is the exit code.
+func askBerthd(c *Context, sock string, msg proto.Msg) (*proto.Msg, int) {
+ conn, raw, err := server.Dial(sock)
+ if err != nil {
+ return nil, c.fail(Systemf("berthd unreachable on %s: %v", sock, err))
+ }
+ defer raw.Close()
+ if err := conn.Send(msg); err != nil {
+ return nil, c.fail(Systemf("berthd on %s: %v", sock, err))
+ }
+ m, err := conn.Recv()
+ if err != nil {
+ return nil, c.fail(Systemf("no reply from berthd: %v", err))
+ }
+ return &m, ExitOK
+}
diff --git a/internal/cli/schedule_test.go b/internal/cli/schedule_test.go
new file mode 100644
index 00000000..6b269944
--- /dev/null
+++ b/internal/cli/schedule_test.go
@@ -0,0 +1,79 @@
+// Copyright 2026 Trevin Tindall (T-Crypt)
+// SPDX-License-Identifier: Apache-2.0
+
+package cli
+
+import (
+ "encoding/json"
+ "path/filepath"
+ "strings"
+ "testing"
+
+ "github.com/T-Crypt/berth/internal/proto"
+ "github.com/T-Crypt/berth/internal/schedule"
+ "github.com/T-Crypt/berth/internal/server"
+)
+
+// Booking, listing, moving and cancelling on a real berthd: the board shows
+// the booking, a clash is refused with what it hits, and a booked waiter is
+// queued first at the booking's class.
+func TestScheduleAgainstADaemon(t *testing.T) {
+ isolate(t)
+ s, err := schedule.Open(filepath.Join(t.TempDir(), "schedule.json"))
+ if err != nil {
+ t.Fatal(err)
+ }
+ sock := startDaemonWith(t, server.Extras{Schedule: s})
+
+ code, out, errOut := runCLI(t, "schedule", "book", "--who", "pi-pocket", "--engine", "pocket", "--priority", "batch",
+ "--at", "now", "--for", "5h", "--socket", sock)
+ if code != 0 || !strings.Contains(out, "booked b1: run for pi-pocket") {
+ t.Fatalf("book (exit %d): %s %s", code, out, errOut)
+ }
+ code, _, errOut = runCLI(t, "schedule", "book", "--who", "op", "--kind", "down", "--at", "now", "--for", "1h", "--socket", sock)
+ if code != 1 || !strings.Contains(errOut, "clashes with b1 (run for pi-pocket") {
+ t.Fatalf("clash (exit %d): %s", code, errOut)
+ }
+
+ holder := ask(t, sock, proto.Msg{Engine: "holder", Priority: "foreground", Pinned: true})
+ if m, err := holder.Recv(); err != nil || m.Op != proto.OpGranted {
+ t.Fatalf("holder: %+v, %v", m, err)
+ }
+ ask(t, sock, proto.Msg{Engine: "chat", Priority: "foreground", Wait: 30})
+ waitUntil(t, func() bool { return len(queueJSON(t, sock)) == 1 })
+ ask(t, sock, proto.Msg{Engine: "pocket", Priority: "interactive", Wait: 30})
+ waitUntil(t, func() bool { return len(queueJSON(t, sock)) == 2 })
+
+ code, out, errOut = runCLI(t, "schedule", "--socket", sock, "--format", "json")
+ if code != 0 {
+ t.Fatalf("schedule (exit %d): %s", code, errOut)
+ }
+ var board proto.ScheduleBoard
+ if err := json.Unmarshal([]byte(out), &board); err != nil {
+ t.Fatalf("%v: %s", err, out)
+ }
+ if len(board.Running) != 1 || board.Running[0].Engine != "holder" {
+ t.Errorf("running = %+v, want holder", board.Running)
+ }
+ if len(board.Queued) != 2 || board.Queued[0].Engine != "pocket" || board.Queued[0].Priority != "batch" || board.Queued[0].Booking != "b1" {
+ t.Errorf("queued = %+v, want pocket first at batch under b1", board.Queued)
+ }
+ if len(board.Bookings) != 1 || board.Bookings[0].ID != "b1" {
+ t.Errorf("bookings = %+v, want b1", board.Bookings)
+ }
+
+ code, out, errOut = runCLI(t, "schedule", "move", "b1", "--at", "23:59", "--socket", sock)
+ if code != 0 || !strings.Contains(out, "moved b1") {
+ t.Fatalf("move (exit %d): %s %s", code, out, errOut)
+ }
+ waitUntil(t, func() bool { return queueJSON(t, sock)[0].Engine == "chat" })
+
+ code, out, errOut = runCLI(t, "schedule", "cancel", "b1", "--socket", sock)
+ if code != 0 || !strings.Contains(out, "cancelled b1") {
+ t.Fatalf("cancel (exit %d): %s %s", code, out, errOut)
+ }
+ code, out, _ = runCLI(t, "schedule", "--socket", sock)
+ if code != 0 || !strings.Contains(out, "nothing booked") || !strings.Contains(out, "RUNNING") {
+ t.Fatalf("board after cancel (exit %d): %s", code, out)
+ }
+}
diff --git a/internal/cli/service.go b/internal/cli/service.go
index 5851b545..d4465b50 100644
--- a/internal/cli/service.go
+++ b/internal/cli/service.go
@@ -180,7 +180,7 @@ func runServiceInstall(c *Context) int {
fs := c.FlagSet()
gateway := fs.Bool("gateway", false, "also install the gateway")
harbor := fs.Bool("harbor", false, "also install the harbor")
- notifyToo := fs.Bool("notify", false, "also install berth notify, desktop notifications (Linux)")
+ notifyToo := fs.Bool("notify", false, "also start berth notify, desktop notifications: a user unit on Linux, the tray at logon on Windows")
dryRun := fs.Bool("dry-run", false, "print what would be written and run, change nothing")
backend := fs.String("gpu-backend", "auto", "daemon GPU probe: auto, nvidia or amd")
card := fs.String("gpu", "all", "daemon cards: all, or a comma list of GPU indices (AMD: DRM cardN numbers) and NVIDIA PCI addresses (0000:02:00.0)")
@@ -213,13 +213,21 @@ func runServiceInstall(c *Context) int {
return c.fail(Systemf("find the berth binary: %v", err))
}
units := serviceUnitsForGPU(exe, *gateway, *harbor, *backend, *card)
- if *notifyToo {
- if runtime.GOOS != "linux" {
- return c.fail(Failf("--notify installs a Linux user unit; on %s, run berth notify in your own session", runtime.GOOS))
- }
+ switch {
+ case !*notifyToo:
+ case runtime.GOOS == "linux":
units = append(units, notifyUnit(exe))
+ case runtime.GOOS != "windows":
+ return c.fail(Failf("--notify is for Linux and Windows; on %s, run berth notify in your own session", runtime.GOOS))
}
doc, e := serviceInstall(context.Background(), liveServiceOps(), units, *dryRun)
+ if e == nil && doc != nil && *notifyToo && runtime.GOOS == "windows" {
+ line, err := notifyAtLogon(exe, true, *dryRun)
+ doc.Commands = append(doc.Commands, line)
+ if err != nil {
+ e = Systemf("start berth notify at logon: %v", err)
+ }
+ }
return writeService(c, "install", doc, e)
}
@@ -242,6 +250,13 @@ func runServiceUninstall(c *Context) int {
return code
}
doc, e := serviceUninstall(context.Background(), liveServiceOps(), allUnits(), *dryRun)
+ if e == nil && doc != nil && runtime.GOOS == "windows" {
+ line, err := notifyAtLogon("", false, *dryRun)
+ doc.Commands = append(doc.Commands, line)
+ if err != nil {
+ e = Systemf("stop starting berth notify at logon: %v", err)
+ }
+ }
return writeService(c, "uninstall", doc, e)
}
diff --git a/internal/cli/status.go b/internal/cli/status.go
index 68cebaf2..30c5b953 100644
--- a/internal/cli/status.go
+++ b/internal/cli/status.go
@@ -111,6 +111,9 @@ func (v statusView) Human(c *Context, ok bool) error {
if l.Pinned {
prio += ",pinned"
}
+ if l.Training {
+ prio += ",training"
+ }
row := fmt.Sprintf("%s\t%s\t%s\t%s\t%s\t%s\t%v", cardCol(l.ID, l.Card), l.Engine, prio, state, needs, memoryHeld(l.Used, l.GTTUsed), l.GPUPIDs)
if c.Verbose {
row += fmt.Sprintf("\t%d", l.PID)
@@ -134,6 +137,16 @@ func (v statusView) Human(c *Context, ok bool) error {
}
lease("LEASE", st.Leases, false)
lease("WAITING", st.Waiting, true)
+ if len(st.Schedules) > 0 {
+ fmt.Fprint(tw, "\nSCHEDULE\tHOURS\tNOW\n")
+ for _, s := range st.Schedules {
+ now := c.Style.Warn("closed") + ", opens in " + age(s.ChangeS)
+ if s.Open {
+ now = c.Style.OK("open") + ", closes in " + age(s.ChangeS)
+ }
+ fmt.Fprintf(tw, "%s\t%s\t%s\n", s.Engine, s.Window, esc(now))
+ }
+ }
if len(st.Squatters) > 0 {
fmt.Fprint(tw, "\n"+cardCol("SQUATTER", "CARD")+"\tPID\tHOLDS\tSEEN\tUNPROTECTED RUN\n")
for _, s := range st.Squatters {
@@ -170,7 +183,9 @@ func (v statusView) Human(c *Context, ok bool) error {
// of what a run needs; it is never a reservation.
est := map[string]uint64{}
for _, f := range st.Footprints {
- est[f.Engine] = f.Bytes
+ if f.Bytes > 0 {
+ est[f.Engine] = f.Bytes
+ }
}
fmt.Fprint(tw, "\nENGINE\tTYPE\tUP\tMODELS\tDECODE\tPREFILL\tOUT TOKENS\tQUEUE\tEST. VRAM\n")
for _, e := range st.Engines {
diff --git a/internal/cli/status_test.go b/internal/cli/status_test.go
index 1dcf5fde..616e9e51 100644
--- a/internal/cli/status_test.go
+++ b/internal/cli/status_test.go
@@ -40,6 +40,13 @@ func (emptyCard) Read(context.Context) (gpu.Snapshot, error) {
// through the real wire, which is the only way to verify a document that has to
// match what the daemon sends.
func startDaemon(t *testing.T) string {
+ t.Helper()
+ return startDaemonWith(t, server.Extras{})
+}
+
+// startDaemonWith is startDaemon serving x's hooks (the engines berthd
+// found, say); its alerts are the daemon's own.
+func startDaemonWith(t *testing.T, x server.Extras) string {
t.Helper()
sock := filepath.Join(t.TempDir(), "berth.sock")
log := slog.New(slog.NewTextHandler(io.Discard, nil))
@@ -60,7 +67,8 @@ func startDaemon(t *testing.T) string {
ev := alerts.New(alerts.FromConfig(config.AlertsConfig{}), 3, nil)
lastAlerts := &atomic.Pointer[proto.Alerts]{}
lastAlerts.Store(ev.Snapshot())
- go server.Serve(ctx, ln, node.One(b), server.Extras{Alerts: lastAlerts.Load}, log)
+ x.Alerts = lastAlerts.Load
+ go server.Serve(ctx, ln, node.One(b), x, log)
return sock
}
@@ -442,3 +450,26 @@ func TestClaimsTablesSayHoldsAndNameCards(t *testing.T) {
t.Errorf("one card claims table:\n%s", s)
}
}
+
+// berth status lists every engine held to hours and where the node's clock
+// sits in them, so a window shows before its engine runs.
+func TestStatusShowsTheSchedules(t *testing.T) {
+ var out, errOut bytes.Buffer
+ c := &Context{Out: &out, Err: &errOut, Format: FormatHuman, Style: newStyle(&out), errStyle: newStyle(&errOut)}
+ st := &proto.Status{
+ API: proto.APIVersion, Version: "test",
+ Schedules: []proto.Schedule{
+ {Engine: "nightly", Window: "23:00-07:00", Open: false, ChangeS: 3 * 3600},
+ {Engine: "render", Window: "18:00-22:00", Open: true, ChangeS: 2*3600 + 15*60},
+ },
+ }
+ if err := (statusView{st: st}).Human(c, true); err != nil {
+ t.Fatal(err)
+ }
+ s := out.String()
+ for _, want := range []string{"SCHEDULE", "HOURS", "nightly", "23:00-07:00", "closed, opens in 3h", "render", "open, closes in 2h15m"} {
+ if !strings.Contains(s, want) {
+ t.Errorf("stdout lacks %q:\n%s", want, s)
+ }
+ }
+}
diff --git a/internal/cli/train_test.go b/internal/cli/train_test.go
new file mode 100644
index 00000000..2698280f
--- /dev/null
+++ b/internal/cli/train_test.go
@@ -0,0 +1,63 @@
+// Copyright 2026 Trevin Tindall (T-Crypt)
+// SPDX-License-Identifier: Apache-2.0
+
+package cli
+
+import (
+ "os"
+ "path/filepath"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/T-Crypt/berth/internal/procs"
+ "github.com/T-Crypt/berth/internal/runner"
+ "github.com/T-Crypt/berth/internal/runs"
+)
+
+// berth train sets the lease's shape itself, so a flag that would change it
+// is refused before berthd is dialed.
+func TestTrainRefusesTheFlagsItSets(t *testing.T) {
+ isolate(t)
+ for _, flag := range [][]string{{"--priority", "batch"}, {"--needs-mib", "8000"}, {"--wait", "5m"}, {"--window", "23:00-07:00"}} {
+ args := append([]string{"train", "--as", "x"}, flag...)
+ code, _, errOut := runCLI(t, append(args, "--", "true")...)
+ if code != ExitSystem || !strings.Contains(errOut, "berth train sets "+flag[0]+" itself") {
+ t.Errorf("%v: exit %d, stderr %q", flag, code, errOut)
+ }
+ }
+}
+
+// The lock lives in berthd, so a training never starts unprotected.
+func TestTrainRefusesToStartWithoutADaemon(t *testing.T) {
+ isolate(t)
+ missing := filepath.Join(t.TempDir(), "nope.sock")
+ code, out, _ := runCLI(t, "train", "--as", "x", "--socket", missing, "--", "true")
+ if code != runner.ExitRefused {
+ t.Fatalf("exit = %d, want %d", code, runner.ExitRefused)
+ }
+ if out != "" {
+ t.Errorf("stdout = %q, want empty", out)
+ }
+}
+
+// With berthd down, a run refuses to start while a training that lost the
+// daemon still runs on the node, instead of starting unprotected beside it.
+func TestRunRefusesWithoutADaemonWhileATrainingRuns(t *testing.T) {
+ isolate(t)
+ sock := filepath.Join(t.TempDir(), "berth.sock")
+ self := os.Getpid()
+ if _, err := runs.Write(runs.Dir(sock), runs.Record{
+ Engine: "train-a", PID: self, Start: procs.System{}.StartTime(self), Started: time.Now(),
+ Reason: runs.ReasonLost, Training: true,
+ }); err != nil {
+ t.Fatal(err)
+ }
+ code, out, errOut := runCLI(t, "run", "--as", "chat", "--socket", sock, "--", "echo", "started")
+ if code != runner.ExitRefused || strings.Contains(out, "started") {
+ t.Fatalf("exit %d, stdout %q, stderr %q; want a refusal and no start", code, out, errOut)
+ }
+ if !strings.Contains(errOut, "train-a is training") {
+ t.Errorf("stderr %q does not name the training", errOut)
+ }
+}
diff --git a/internal/cli/train_unix_test.go b/internal/cli/train_unix_test.go
new file mode 100644
index 00000000..7b8175e0
--- /dev/null
+++ b/internal/cli/train_unix_test.go
@@ -0,0 +1,75 @@
+// Copyright 2026 Trevin Tindall (T-Crypt)
+// SPDX-License-Identifier: Apache-2.0
+
+//go:build unix
+
+package cli
+
+import (
+ "strings"
+ "testing"
+ "time"
+)
+
+// Through the real wire: while berth train runs, a berth run for the same
+// card is refused at once, though it asked to wait an hour, and its command
+// never starts.
+func TestTrainLocksTheCardAgainstBerthRun(t *testing.T) {
+ isolate(t)
+ sock := startDaemon(t)
+ done := make(chan int, 1)
+ go func() {
+ code, _, _ := runCLI(t, "train", "--as", "train-a", "--socket", sock, "--", "sleep", "3")
+ done <- code
+ }()
+ deadline := time.Now().Add(2 * time.Second)
+ for {
+ _, out, _ := runCLI(t, "status", "--socket", sock)
+ if strings.Contains(out, "train-a") {
+ break
+ }
+ if time.Now().After(deadline) {
+ t.Fatalf("train-a never held the card:\n%s", out)
+ }
+ time.Sleep(50 * time.Millisecond)
+ }
+
+ start := time.Now()
+ code, out, errOut := runCLI(t, "run", "--as", "chat", "--priority", "foreground", "--wait", "1h", "--socket", sock, "--", "echo", "started")
+ if code == ExitOK || strings.Contains(out, "started") {
+ t.Fatalf("berth run beside a training: exit %d, stdout %q, stderr %q", code, out, errOut)
+ }
+ if !strings.Contains(errOut, "locked for training by train-a") {
+ t.Errorf("stderr %q does not name the training", errOut)
+ }
+ if took := time.Since(start); took > 2*time.Second {
+ t.Errorf("the refusal took %v, want it at once", took)
+ }
+ if code := <-done; code != ExitOK {
+ t.Errorf("berth train exit = %d, want 0", code)
+ }
+}
+
+// --exclusive is part of the run: on a card the driver names no UUID for,
+// berth train --exclusive is refused after the grant and its command never
+// starts, so it never runs without the guard it asked for.
+func TestTrainExclusiveRefusesWithoutAnNVIDIACard(t *testing.T) {
+ isolate(t)
+ sock := startDaemon(t)
+ code, out, errOut := runCLI(t, "train", "--exclusive", "--as", "train-a", "--socket", sock, "--", "echo", "started")
+ if code == ExitOK || strings.Contains(out, "started") {
+ t.Fatalf("exit %d, stdout %q, stderr %q; want a refusal and no start", code, out, errOut)
+ }
+ if !strings.Contains(errOut, "--exclusive") {
+ t.Errorf("stderr %q does not name --exclusive", errOut)
+ }
+}
+
+// --exclusive belongs to berth train alone.
+func TestRunRefusesExclusive(t *testing.T) {
+ isolate(t)
+ code, _, errOut := runCLI(t, "run", "--exclusive", "--as", "x", "--", "true")
+ if code != ExitSystem || !strings.Contains(errOut, "--exclusive is for berth train") {
+ t.Fatalf("exit %d, stderr %q", code, errOut)
+ }
+}
diff --git a/internal/cli/unpark.go b/internal/cli/unpark.go
index 952716e7..f95be274 100644
--- a/internal/cli/unpark.go
+++ b/internal/cli/unpark.go
@@ -89,9 +89,9 @@ func parkCall(sock string, req proto.Msg, want string) (proto.Msg, *Error) {
return m, nil
}
-func nonNil(s []string) []string {
+func nonNil[T any](s []T) []T {
if s == nil {
- return []string{}
+ return []T{}
}
return s
}
diff --git a/internal/cli/why.go b/internal/cli/why.go
new file mode 100644
index 00000000..1ae53795
--- /dev/null
+++ b/internal/cli/why.go
@@ -0,0 +1,206 @@
+// Copyright 2026 Trevin Tindall (T-Crypt)
+// SPDX-License-Identifier: Apache-2.0
+
+package cli
+
+import (
+ "slices"
+ "time"
+
+ "github.com/T-Crypt/berth/internal/proto"
+ "github.com/T-Crypt/berth/internal/server"
+)
+
+// whyDoc is one lease's story: every event berthd kept about it, oldest
+// first, with the restarts it lived through, and where it stands now.
+type whyDoc struct {
+ Lease string `json:"lease" yaml:"lease"`
+ Engine string `json:"engine" yaml:"engine"`
+ Card string `json:"card,omitempty" yaml:"card,omitempty"`
+ Events []proto.Event `json:"events" yaml:"events"`
+ // Now is the lease as berthd holds it now, nil once it has ended.
+ Now *proto.Lease `json:"now,omitempty" yaml:"now,omitempty"`
+ // Next is the lease its engine re-attached as after a berthd restart
+ // cut this one off, "" when none did.
+ Next string `json:"next,omitempty" yaml:"next,omitempty"`
+}
+
+// whyVerb says what each kind of event did to the lease, in the words the
+// story prints. A kind not listed prints as its name.
+var whyVerb = map[string]string{
+ proto.EvRequested: "asked for the card",
+ proto.EvWaiting: "waited",
+ proto.EvGranted: "granted",
+ proto.EvReattached: "re-attached after a berthd restart",
+ proto.EvDraining: "asked to give the card back",
+ proto.EvReleased: "released the card",
+ proto.EvFreed: "its memory is free",
+ proto.EvRefused: "refused",
+ proto.EvOrphaned: "its process was gone, so berthd ended the lease",
+ proto.EvStuck: "its memory stayed on the card after the release",
+ proto.EvParked: "parked in RAM",
+ proto.EvResumed: "resumed from RAM",
+ proto.EvUnparked: "loaded back onto the card without a resume",
+ proto.EvParkRefused: "could not park",
+ proto.EvParkEvicted: "stopped while parked",
+ proto.EvJuggle: "a juggle for the card",
+ proto.EvQueueOverride: "the operator changed its place in line",
+ proto.EvBookingPrio: "a booking changed its priority",
+ proto.EvLending: "asked to lend part of the card",
+ proto.EvLent: "lent part of the card",
+ proto.EvReturned: "took its lent memory back",
+ proto.EvLendFailed: "the lend failed",
+ proto.EvNodeStarted: "berthd restarted",
+}
+
+// ends is the kinds after which a lease has no more to say.
+var ends = []string{proto.EvFreed, proto.EvRefused, proto.EvOrphaned}
+
+// whyOf finds the lease name means, a lease ID or else the engine's newest
+// lease, and tells its story from evs. live is the leases berthd holds now.
+func whyOf(evs []proto.Event, live []proto.Lease, name string) (whyDoc, bool) {
+ id := ""
+ for i := len(evs) - 1; i >= 0 && id == ""; i-- {
+ if evs[i].Lease == name {
+ id = name
+ }
+ }
+ for i := len(evs) - 1; i >= 0 && id == ""; i-- {
+ if evs[i].Engine == name && evs[i].Lease != "" {
+ id = evs[i].Lease
+ }
+ }
+ if id == "" {
+ return whyDoc{}, false
+ }
+ doc := whyDoc{Lease: id}
+ for _, l := range live {
+ if l.ID == id {
+ doc.Now = &l
+ }
+ }
+ first, last := -1, -1
+ for i, e := range evs {
+ if e.Lease == id {
+ if first < 0 {
+ first = i
+ }
+ last = i
+ doc.Engine, doc.Card = e.Engine, e.Card
+ }
+ }
+ ended := slices.Contains(ends, evs[last].Kind)
+ for i := first; i < len(evs); i++ {
+ e := evs[i]
+ switch {
+ case e.Lease == id:
+ doc.Events = append(doc.Events, e)
+ case e.Kind == proto.EvNodeStarted && (i < last || (!ended && doc.Now == nil)):
+ doc.Events = append(doc.Events, e)
+ if i > last {
+ doc.Next = reattachedAs(evs[i+1:], doc.Engine)
+ return doc, true
+ }
+ }
+ }
+ return doc, true
+}
+
+// reattachedAs is the lease engine re-attached as in evs, which start just
+// after a berthd restart, "" when it did not before the next restart.
+func reattachedAs(evs []proto.Event, engine string) string {
+ for _, e := range evs {
+ if e.Kind == proto.EvNodeStarted {
+ return ""
+ }
+ if e.Kind == proto.EvReattached && e.Engine == engine {
+ return e.Lease
+ }
+ }
+ return ""
+}
+
+type whyView struct{ doc whyDoc }
+
+func (v whyView) Data() any { return v.doc }
+
+func (v whyView) Human(c *Context, _ bool) error {
+ d := v.doc
+ head := d.Lease
+ if d.Engine != "" && d.Engine != d.Lease {
+ head += " " + d.Engine
+ }
+ if d.Card != "" {
+ head += " " + d.Card
+ }
+ c.Printf("%s\n", head)
+ for _, e := range d.Events {
+ verb, ok := whyVerb[e.Kind]
+ if !ok {
+ verb = e.Kind
+ }
+ line := verb
+ if e.Bytes != 0 {
+ line += ", " + server.MiB(e.Bytes)
+ }
+ if e.Detail != "" {
+ line += ": " + e.Detail
+ }
+ c.Printf(" %s %s\n", c.Style.Dim(e.At.Local().Format("Jan 02 15:04:05")), line)
+ }
+ switch {
+ case d.Now != nil:
+ c.Printf("now %s, %s", d.Now.State, d.Now.Priority)
+ if d.Now.Position > 0 {
+ c.Printf(", %d in line", d.Now.Position)
+ }
+ if d.Now.TurnS > 0 {
+ c.Printf(", %s of its minimum turn left", (time.Duration(d.Now.TurnS) * time.Second).Round(time.Second))
+ }
+ c.Printf("\n")
+ case d.Next != "":
+ c.Printf("its engine re-attached as %s: berth why %s\n", d.Next, d.Next)
+ }
+ return nil
+}
+
+func runWhy(c *Context) int {
+ fs := c.FlagSet()
+ sock := fs.String("socket", defaultSocket(), "berthd socket ($BERTH_SOCKET)")
+ if code, done := c.ParseInterspersed(fs); done {
+ return code
+ }
+ if len(c.Args) != 1 {
+ return c.fail(Systemf("berth why takes one lease ID or engine name").
+ WithHint("for example: berth why strata-24, or berth why comfyui"))
+ }
+ conn, raw, err := server.Dial(*sock)
+ if err != nil {
+ return c.fail(Systemf("berthd unreachable on %s: %v", *sock, err))
+ }
+ defer raw.Close()
+ if err := conn.Send(proto.Msg{Op: proto.OpEvents}); err != nil {
+ return c.fail(Systemf("berthd on %s: %v", *sock, err))
+ }
+ var evs []proto.Event
+ for {
+ m, err := conn.Recv()
+ if err != nil {
+ break
+ }
+ if m.Event != nil {
+ evs = append(evs, *m.Event)
+ }
+ }
+ st, _ := contractStatus(*sock)
+ var live []proto.Lease
+ if st != nil {
+ live = append(st.Leases, st.Waiting...)
+ }
+ doc, found := whyOf(evs, live, c.Args[0])
+ if !found {
+ return c.fail(Failf("no lease or engine named %q in berthd's last %d events", c.Args[0], len(evs)).
+ WithHint("berth status lists the leases now; berth events lists what berthd kept"))
+ }
+ return c.Write(whyView{doc: doc}, nil)
+}
diff --git a/internal/cli/why_test.go b/internal/cli/why_test.go
new file mode 100644
index 00000000..63a68a2e
--- /dev/null
+++ b/internal/cli/why_test.go
@@ -0,0 +1,108 @@
+// Copyright 2026 Trevin Tindall (T-Crypt)
+// SPDX-License-Identifier: Apache-2.0
+
+package cli
+
+import (
+ "encoding/json"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/T-Crypt/berth/internal/proto"
+ "github.com/T-Crypt/berth/internal/server"
+)
+
+func at(min int) time.Time { return time.Date(2026, 11, 12, 23, min, 0, 0, time.UTC) }
+
+// The 11-12 night: strata waited for OrcaSAQ to drain, held the card, and
+// was drained for Dirk-27B. Then berthd restarted and Dirk re-attached.
+var night = []proto.Event{
+ {Seq: 1, At: at(22), Kind: proto.EvRequested, Lease: "strata-24", Engine: "strata", Detail: "foreground"},
+ {Seq: 2, At: at(22), Kind: proto.EvDraining, Lease: "OrcaSAQ-23", Engine: "OrcaSAQ", Detail: "strata (foreground) needs the card"},
+ {Seq: 3, At: at(22), Kind: proto.EvWaiting, Lease: "strata-24", Engine: "strata", Detail: "next in line: draining OrcaSAQ [foreground, active]"},
+ {Seq: 4, At: at(23), Kind: proto.EvGranted, Lease: "strata-24", Engine: "strata", Detail: "waited 1.631s"},
+ {Seq: 5, At: at(25), Kind: proto.EvRequested, Lease: "Dirk-27B-25", Engine: "Dirk-27B", Detail: "foreground"},
+ {Seq: 6, At: at(25), Kind: proto.EvDraining, Lease: "strata-24", Engine: "strata", Detail: "Dirk-27B (foreground) needs the card"},
+ {Seq: 7, At: at(25), Kind: proto.EvReleased, Lease: "strata-24", Engine: "strata"},
+ {Seq: 8, At: at(26), Kind: proto.EvFreed, Lease: "strata-24", Engine: "strata", Bytes: 16 << 30},
+ {Seq: 9, At: at(26), Kind: proto.EvGranted, Lease: "Dirk-27B-25", Engine: "Dirk-27B", Detail: "waited 40s"},
+ {Seq: 10, At: at(30), Kind: proto.EvNodeStarted, Detail: "berth 0.6.0"},
+ {Seq: 11, At: at(30), Kind: proto.EvReattached, Lease: "Dirk-27B-26", Engine: "Dirk-27B", Detail: "foreground"},
+}
+
+func kindsOf(evs []proto.Event) string {
+ var k []string
+ for _, e := range evs {
+ k = append(k, e.Kind)
+ }
+ return strings.Join(k, " ")
+}
+
+func TestWhyTellsOneLeasesStory(t *testing.T) {
+ doc, ok := whyOf(night, nil, "strata")
+ if !ok {
+ t.Fatal("strata not found")
+ }
+ if doc.Lease != "strata-24" || doc.Engine != "strata" {
+ t.Fatalf("lease = %q engine = %q", doc.Lease, doc.Engine)
+ }
+ if got, want := kindsOf(doc.Events), "lease.requested lease.waiting lease.granted lease.draining lease.released lease.freed"; got != want {
+ t.Fatalf("events = %s, want %s", got, want)
+ }
+ if doc.Next != "" {
+ t.Fatalf("an ended lease names a re-attach: %q", doc.Next)
+ }
+}
+
+// A lease a restart cut off ends at the restart and names the lease its
+// engine re-attached as.
+func TestWhyFollowsARestart(t *testing.T) {
+ doc, ok := whyOf(night, nil, "Dirk-27B-25")
+ if !ok {
+ t.Fatal("Dirk-27B-25 not found")
+ }
+ if got, want := kindsOf(doc.Events), "lease.requested lease.granted node.started"; got != want {
+ t.Fatalf("events = %s, want %s", got, want)
+ }
+ if doc.Next != "Dirk-27B-26" {
+ t.Fatalf("next = %q, want Dirk-27B-26", doc.Next)
+ }
+
+ doc, _ = whyOf(night, []proto.Lease{{ID: "Dirk-27B-26", State: "active", Priority: "foreground"}}, "Dirk-27B")
+ if doc.Lease != "Dirk-27B-26" || doc.Now == nil || doc.Now.State != "active" {
+ t.Fatalf("the engine's newest lease = %+v", doc)
+ }
+}
+
+func TestWhyAgainstADaemon(t *testing.T) {
+ isolate(t)
+ sock := startDaemonWith(t, server.Extras{})
+ holder := ask(t, sock, proto.Msg{Engine: "comfyui", Priority: "interactive"})
+ if m, err := holder.Recv(); err != nil || m.Op != proto.OpGranted {
+ t.Fatalf("holder: %+v, %v", m, err)
+ }
+ ask(t, sock, proto.Msg{Engine: "strata", Priority: "foreground", Wait: 30})
+ waitUntil(t, func() bool { return len(queueJSON(t, sock)) == 1 })
+
+ code, out, errOut := runCLI(t, "why", "comfyui", "--socket", sock)
+ if code != 0 {
+ t.Fatalf("why (exit %d): %s", code, errOut)
+ }
+ for _, want := range []string{"asked for the card", "granted", "asked to give the card back: strata (foreground) needs the card", "now "} {
+ if !strings.Contains(out, want) {
+ t.Fatalf("why comfyui = %q, want it to contain %q", out, want)
+ }
+ }
+
+ code, out, _ = runCLI(t, "why", "strata", "--socket", sock, "--format", "json")
+ var doc whyDoc
+ if code != 0 || json.Unmarshal([]byte(out), &doc) != nil || doc.Now == nil || doc.Now.State != "waiting" {
+ t.Fatalf("why strata json (exit %d): %s", code, out)
+ }
+
+ code, _, errOut = runCLI(t, "why", "nobody", "--socket", sock)
+ if code != 1 || !strings.Contains(errOut, `no lease or engine named "nobody"`) {
+ t.Fatalf("why nobody (exit %d): %s", code, errOut)
+ }
+}
diff --git a/internal/cli/wire.go b/internal/cli/wire.go
index eb1b8d84..e99bd1b5 100644
--- a/internal/cli/wire.go
+++ b/internal/cli/wire.go
@@ -5,6 +5,7 @@ package cli
import (
"errors"
+ "fmt"
"os"
"strings"
@@ -25,6 +26,9 @@ type wireDoc struct {
Skipped int `json:"skipped" yaml:"skipped"`
Models []wire.Model `json:"models" yaml:"models"`
Backup string `json:"backup,omitempty" yaml:"backup,omitempty"`
+ // Swapping is the groups whose wired models llama-swap swaps itself,
+ // out of Berth's turns. A setting to change, not drift: --check passes.
+ Swapping []wire.SwapGroup `json:"swapping,omitempty" yaml:"swapping,omitempty"`
}
// wireView is one run over one llama-swap config.
@@ -75,6 +79,13 @@ func (v wireView) Human(c *Context, ok bool) error {
}
}
}
+ for _, s := range d.Swapping {
+ group := "the default group (models no group names)"
+ if s.Group != "" {
+ group = fmt.Sprintf("group %q", s.Group)
+ }
+ c.Bullet("llama-swap swaps %d wired models in %s itself, so min_turn and the ranking never decide between them: set swap: false on it (contrib/INTEGRATION.md#llama-swap)", len(s.Members), group)
+ }
if d.Skipped > 0 && !c.Verbose {
c.Bullet("%d skipped (--verbose lists them)", d.Skipped)
}
@@ -160,7 +171,7 @@ func runWireLlamaSwap(c *Context) int {
doc := wireDoc{
Status: "ok", Server: "llama-swap", Path: *pathArg, Mode: mode, Changed: res.Changed,
Wired: res.Count("wired"), Add: res.Count("add"), Update: res.Count("update"),
- Skipped: res.Count("skip"), Models: res.Models,
+ Skipped: res.Count("skip"), Models: res.Models, Swapping: res.Swapping,
}
if *write && res.Changed {
backup, err := wire.Write(res)
diff --git a/internal/config/config.go b/internal/config/config.go
index 6a13d5d5..5ba7cf07 100644
--- a/internal/config/config.go
+++ b/internal/config/config.go
@@ -59,6 +59,10 @@ type Engine struct {
Serves []string `toml:"serves,omitempty" json:"serves,omitempty" yaml:"serves,omitempty"` // model names this engine answers for
Priority string `toml:"priority,omitempty" json:"priority,omitempty" yaml:"priority,omitempty"` // batch, interactive (default) or foreground
NeedsMiB MiBs `toml:"needs_mib,omitempty" json:"needs_mib,omitempty" yaml:"needs_mib,omitempty"` // per card, in gpu order; none = the whole card
+ // LearnSize: with no needs_mib, the lease takes the most this engine has
+ // been measured holding plus a load headroom, so engines that fit
+ // together share the card. Until a run is measured, the whole card.
+ LearnSize bool `toml:"learn_size,omitempty" json:"learn_size,omitempty" yaml:"learn_size,omitempty"`
// Background: the gateway starts this engine at batch priority whenever
// the card has been free (no lease, no waiter, the card serving this
// machine) for IdleAfter. Any other request drains it as any batch
@@ -707,6 +711,9 @@ func (c *Config) Validate() error {
if err := CheckNeeds(e.NeedsMiB, e.GPU); err != nil {
errs = append(errs, fmt.Errorf("%s: %w", where, err))
}
+ if err := CheckLearnSize(e); err != nil {
+ errs = append(errs, fmt.Errorf("%s: %w", where, err))
+ }
if e.MinTurn.Duration < 0 {
errs = append(errs, fmt.Errorf("%s: min_turn %s must be >= 0", where, e.MinTurn))
}
diff --git a/internal/config/mibs.go b/internal/config/mibs.go
index bf7219d4..35542e15 100644
--- a/internal/config/mibs.go
+++ b/internal/config/mibs.go
@@ -144,6 +144,23 @@ func (m *MiBs) UnmarshalYAML(node *yaml.Node) error {
return nil
}
+// CheckLearnSize is the rule for learn_size and berth run --learn-size: it
+// learns one size on one card, so it takes no typed size, no list of cards
+// and no whole_machine.
+func CheckLearnSize(e Engine) error {
+ switch {
+ case !e.LearnSize:
+ return nil
+ case len(e.NeedsMiB.Bytes()) > 0:
+ return fmt.Errorf("learn_size and needs_mib %s both size the lease: keep one", e.NeedsMiB)
+ case strings.Contains(e.GPU, ","):
+ return fmt.Errorf("learn_size sizes a lease on one card, and gpu %q names several", e.GPU)
+ case e.WholeMachine:
+ return errors.New("learn_size and whole_machine disagree: whole_machine takes every card")
+ }
+ return nil
+}
+
// CheckNeeds is the rule for needs_mib and berth run --needs-mib: a list of
// sizes has one size per card that gpu names.
func CheckNeeds(m MiBs, gpu string) error {
diff --git a/internal/config/mibs_test.go b/internal/config/mibs_test.go
index 2dfeb483..0ea120e4 100644
--- a/internal/config/mibs_test.go
+++ b/internal/config/mibs_test.go
@@ -48,6 +48,27 @@ func TestEngineNeedsPerCard(t *testing.T) {
}
}
+// learn_size learns one size on one card: it loads alone or with gpu naming
+// one card, and refuses a typed size, several cards or whole_machine.
+func TestEngineLearnSize(t *testing.T) {
+ const entry = "[[engine]]\nname = \"llama\"\ntype = \"custom\"\nurl = \"http://127.0.0.1:8080\"\nhealth = \"/health\"\nlearn_size = true\n"
+ for _, body := range []string{"", "gpu = \"gpu1\"\n"} {
+ c, err := Load(write(t, entry+body))
+ if err != nil || !c.Engine[0].LearnSize {
+ t.Fatalf("%q: learn_size = %v, err %v; want true, nil", body, c.Engine[0].LearnSize, err)
+ }
+ }
+ for _, bad := range []struct{ body, wantErr string }{
+ {"needs_mib = 8192\n", "learn_size and needs_mib 8192 both size the lease"},
+ {"gpu = \"gpu0,gpu1\"\n", `learn_size sizes a lease on one card, and gpu "gpu0,gpu1" names several`},
+ {"whole_machine = true\n", "learn_size and whole_machine disagree"},
+ } {
+ if _, err := Load(write(t, entry+bad.body)); err == nil || !strings.Contains(err.Error(), bad.wantErr) {
+ t.Errorf("%q: error %v, want it to say %q", bad.body, err, bad.wantErr)
+ }
+ }
+}
+
// One size writes back as the plain number it always was, and a list as a
// list, in every encoding the config has.
func TestNeedsKeepTheirSpelling(t *testing.T) {
diff --git a/internal/contract/contract.go b/internal/contract/contract.go
new file mode 100644
index 00000000..b6d9caaa
--- /dev/null
+++ b/internal/contract/contract.go
@@ -0,0 +1,223 @@
+// Copyright 2026 Trevin Tindall (T-Crypt)
+// SPDX-License-Identifier: Apache-2.0
+
+// Package contract is one engine's terms as a single view: what berth.toml
+// declares for it, joined with what Berth measured, every number tagged with
+// where it came from. It is read only and keeps no file of its own, so it
+// cannot drift from the config it reads.
+package contract
+
+import (
+ "fmt"
+ "time"
+
+ "github.com/T-Crypt/berth/internal/config"
+ "github.com/T-Crypt/berth/internal/proto"
+ "github.com/T-Crypt/berth/internal/swapcost"
+)
+
+// Source is where a number came from.
+type Source string
+
+const (
+ Declared Source = "declared" // berth.toml says so
+ Default Source = "default" // berth.toml is silent; Berth's default applies
+ Measured Source = "measured" // Berth saw it on the card, in a switch or on the last run's lease
+ Unknown Source = "unknown" // nothing declares it and nothing measured it
+)
+
+// besideHeadroom is the broker's margin on a learned size (broker.besideHeadroom).
+const besideHeadroom = 512 << 20
+
+// Bytes is a size and its source. Bytes is 0 when the source is unknown, or
+// when the size is the whole card.
+type Bytes struct {
+ Bytes uint64 `json:"bytes,omitempty" yaml:"bytes,omitempty"`
+ Source Source `json:"source" yaml:"source"`
+ Why string `json:"why,omitempty" yaml:"why,omitempty"`
+}
+
+// Duration is a time and its source.
+type Duration struct {
+ Seconds float64 `json:"seconds" yaml:"seconds"`
+ Source Source `json:"source" yaml:"source"`
+ Why string `json:"why,omitempty" yaml:"why,omitempty"`
+}
+
+// Text is a setting and its source.
+type Text struct {
+ Value string `json:"value" yaml:"value"`
+ Source Source `json:"source" yaml:"source"`
+ Why string `json:"why,omitempty" yaml:"why,omitempty"`
+}
+
+// Switch is how long a switch to the engine takes from one tier: the median of
+// the gateway's measured switches, or unknown with no samples.
+type Switch struct {
+ Tier swapcost.Tier `json:"tier" yaml:"tier"`
+ Took Duration `json:"took" yaml:"took"`
+ Samples int `json:"samples,omitempty" yaml:"samples,omitempty"`
+}
+
+// Contract is one engine's terms.
+type Contract struct {
+ Engine string `json:"engine" yaml:"engine"`
+ // Type is the engine type from berth.toml or discovery, "" for a name
+ // Berth knows only from its leases (a llama-swap model).
+ Type string `json:"type,omitempty" yaml:"type,omitempty"`
+ InConfig bool `json:"in_config" yaml:"in_config"`
+ Priority Text `json:"priority" yaml:"priority"`
+ // VRAM is what a lease asks for: needs_mib, the learned size, or the
+ // whole card.
+ VRAM Bytes `json:"vram" yaml:"vram"`
+ // LastPeak is the most VRAM the last run held, what a next run most
+ // likely needs. A reading, never a reservation.
+ LastPeak Bytes `json:"last_peak" yaml:"last_peak"`
+ // RAM is the host RAM a run needs: needs_ram_mib, else the most a run
+ // held while active.
+ RAM Bytes `json:"ram" yaml:"ram"`
+ // ParkedRAM is the host RAM the engine last held parked.
+ ParkedRAM Bytes `json:"parked_ram" yaml:"parked_ram"`
+ MinTurn Duration `json:"min_turn" yaml:"min_turn"`
+ Window Text `json:"window" yaml:"window"`
+ Switch []Switch `json:"switch" yaml:"switch"`
+}
+
+// Tiers is the switch tiers in the order a contract lists them, fastest first.
+var Tiers = []swapcost.Tier{swapcost.Hot, swapcost.Warm, swapcost.Cool, swapcost.Cold}
+
+// Build joins an engine's berth.toml entry (nil when it has none) with what
+// st measured for it and the switch book (nil when there is none). st is nil
+// when berthd did not answer; every measured number is then unknown.
+func Build(name string, e *config.Engine, st *proto.Status, book *swapcost.Book) Contract {
+ c := Contract{Engine: name, InConfig: e != nil}
+ var fp *proto.EngineFootprint
+ var card uint64
+ if st != nil {
+ for i := range st.Footprints {
+ if st.Footprints[i].Engine == name {
+ fp = &st.Footprints[i]
+ }
+ }
+ for _, en := range st.Engines {
+ if en.Name == name {
+ c.Type = en.Type
+ }
+ }
+ for _, g := range st.GPUs {
+ if g.Scheduled {
+ card = g.Total - min(g.Total, g.Baseline)
+ break
+ }
+ }
+ }
+ if e != nil && e.Type != "" {
+ c.Type = e.Type
+ }
+
+ // A llama-swap model has no berth.toml entry: its terms are the flags on
+ // its berth run line, which the last run's lease carried.
+ ran := fp != nil && fp.Priority != ""
+ c.Priority = Text{Value: "interactive", Source: Default}
+ switch {
+ case e != nil && e.Priority != "":
+ c.Priority = Text{Value: e.Priority, Source: Declared}
+ case ran:
+ c.Priority = Text{Value: fp.Priority, Source: Measured, Why: "its last run's lease"}
+ }
+
+ c.VRAM = vram(e, fp, card, st != nil)
+
+ c.LastPeak = measured(fp, func(f *proto.EngineFootprint) uint64 { return f.Bytes }, st != nil)
+ switch {
+ case fp != nil && fp.Bytes > 0:
+ c.LastPeak.Why = fmt.Sprintf("%d runs measured", fp.Runs)
+ case fp != nil && fp.HotPeakBytes > 0:
+ // A model of a group: its last-run figure sits on the group's entry.
+ c.LastPeak = Bytes{Bytes: fp.HotPeakBytes, Source: Measured, Why: "the most a run under this name held"}
+ }
+
+ c.RAM = measured(fp, func(f *proto.EngineFootprint) uint64 { return f.RAMPeakBytes }, st != nil)
+ if c.RAM.Source == Measured {
+ c.RAM.Why = "the most a run held while active"
+ }
+ if e != nil && e.NeedsRAMMiB > 0 {
+ c.RAM = Bytes{Bytes: e.NeedsRAMMiB << 20, Source: Declared, Why: "needs_ram_mib"}
+ }
+ c.ParkedRAM = measured(fp, func(f *proto.EngineFootprint) uint64 { return f.ParkedRAMBytes }, st != nil)
+
+ c.MinTurn = Duration{Source: Default, Why: "off"}
+ switch {
+ case e != nil && e.MinTurn.Duration > 0:
+ c.MinTurn = Duration{Seconds: e.MinTurn.Seconds(), Source: Declared}
+ case ran:
+ c.MinTurn = Duration{Seconds: fp.MinTurnS, Source: Measured, Why: "its last run's lease"}
+ }
+ c.Window = Text{Value: "always", Source: Default}
+ if e != nil && e.Window.Set() {
+ c.Window = Text{Value: e.Window.String(), Source: Declared}
+ }
+
+ for _, t := range Tiers {
+ s := Switch{Tier: t, Took: Duration{Source: Unknown, Why: "no switch measured"}}
+ if book != nil {
+ if est := book.Estimate(name, t); est.Known && est.Samples == 0 {
+ s.Took = Duration{Source: Default, Why: "already on the card, no switch"}
+ } else if est.Known {
+ s.Took = Duration{Seconds: est.Took.Round(100 * time.Millisecond).Seconds(), Source: Measured}
+ s.Samples = est.Samples
+ }
+ }
+ c.Switch = append(c.Switch, s)
+ }
+ return c
+}
+
+// vram is the size a lease of the engine asks for, the way the broker sizes
+// it: needs_mib as declared, else the learned size under learn_size, else
+// the whole card.
+func vram(e *config.Engine, fp *proto.EngineFootprint, card uint64, read bool) Bytes {
+ if e != nil {
+ if b := e.NeedsMiB.Bytes(); len(b) > 0 {
+ why := "needs_mib"
+ if len(b) > 1 {
+ why = fmt.Sprintf("needs_mib, the largest of %d cards", len(b))
+ }
+ return Bytes{Bytes: largest(b), Source: Declared, Why: why}
+ }
+ if e.LearnSize {
+ switch {
+ case !read:
+ return Bytes{Source: Unknown, Why: "learn_size, but berthd did not answer"}
+ case fp == nil || fp.HotPeakBytes == 0:
+ return Bytes{Source: Default, Why: "learn_size, not measured yet, so the whole card"}
+ case card > 0 && fp.HotPeakBytes+besideHeadroom >= card:
+ return Bytes{Source: Measured, Why: "learn_size, its measured peak fills the card, so the whole card"}
+ default:
+ return Bytes{Bytes: fp.HotPeakBytes + besideHeadroom, Source: Measured,
+ Why: fmt.Sprintf("learn_size: measured peak %d MiB + 512 MiB", fp.HotPeakBytes>>20)}
+ }
+ }
+ }
+ return Bytes{Source: Default, Why: "the whole card"}
+}
+
+// measured is one footprint figure, unknown when berthd did not answer or
+// never measured it.
+func measured(fp *proto.EngineFootprint, get func(*proto.EngineFootprint) uint64, read bool) Bytes {
+ if !read {
+ return Bytes{Source: Unknown, Why: "berthd did not answer"}
+ }
+ if fp == nil || get(fp) == 0 {
+ return Bytes{Source: Unknown, Why: "not measured yet"}
+ }
+ return Bytes{Bytes: get(fp), Source: Measured}
+}
+
+func largest(b []uint64) uint64 {
+ var m uint64
+ for _, v := range b {
+ m = max(m, v)
+ }
+ return m
+}
diff --git a/internal/contract/contract_test.go b/internal/contract/contract_test.go
new file mode 100644
index 00000000..ac5328db
--- /dev/null
+++ b/internal/contract/contract_test.go
@@ -0,0 +1,141 @@
+// Copyright 2026 Trevin Tindall (T-Crypt)
+// SPDX-License-Identifier: Apache-2.0
+
+package contract
+
+import (
+ "path/filepath"
+ "testing"
+ "time"
+
+ "github.com/T-Crypt/berth/internal/config"
+ "github.com/T-Crypt/berth/internal/proto"
+ "github.com/T-Crypt/berth/internal/swapcost"
+)
+
+const mib = 1 << 20
+
+func card(fps ...proto.EngineFootprint) *proto.Status {
+ return &proto.Status{
+ GPUs: []proto.GPU{{Scheduled: true, Total: 24564 * mib, Baseline: 10 * mib}},
+ Footprints: fps,
+ }
+}
+
+// The VRAM a lease asks for comes out the way the broker sizes it, and says
+// where it came from.
+func TestBuildVRAM(t *testing.T) {
+ cases := []struct {
+ name string
+ e *config.Engine
+ st *proto.Status
+ want Bytes
+ }{
+ {"needs_mib", &config.Engine{NeedsMiB: config.MiBs{14000}}, card(),
+ Bytes{Bytes: 14000 * mib, Source: Declared, Why: "needs_mib"}},
+ {"needs_mib per card", &config.Engine{NeedsMiB: config.MiBs{9000, 12000}}, card(),
+ Bytes{Bytes: 12000 * mib, Source: Declared, Why: "needs_mib, the largest of 2 cards"}},
+ {"learned", &config.Engine{LearnSize: true}, card(proto.EngineFootprint{Engine: "e", HotPeakBytes: 10000 * mib}),
+ Bytes{Bytes: 10512 * mib, Source: Measured, Why: "learn_size: measured peak 10000 MiB + 512 MiB"}},
+ {"learned peak fills the card", &config.Engine{LearnSize: true}, card(proto.EngineFootprint{Engine: "e", HotPeakBytes: 24100 * mib}),
+ Bytes{Source: Measured, Why: "learn_size, its measured peak fills the card, so the whole card"}},
+ {"learn_size never measured", &config.Engine{LearnSize: true}, card(),
+ Bytes{Source: Default, Why: "learn_size, not measured yet, so the whole card"}},
+ {"learn_size without berthd", &config.Engine{LearnSize: true}, nil,
+ Bytes{Source: Unknown, Why: "learn_size, but berthd did not answer"}},
+ {"nothing declared", &config.Engine{}, card(proto.EngineFootprint{Engine: "e", HotPeakBytes: 10000 * mib}),
+ Bytes{Source: Default, Why: "the whole card"}},
+ {"not in berth.toml", nil, card(),
+ Bytes{Source: Default, Why: "the whole card"}},
+ }
+ for _, tc := range cases {
+ t.Run(tc.name, func(t *testing.T) {
+ if got := Build("e", tc.e, tc.st, nil).VRAM; got != tc.want {
+ t.Fatalf("VRAM = %+v, want %+v", got, tc.want)
+ }
+ })
+ }
+}
+
+// Measured figures come from the footprint, declared ones win where
+// berth.toml names them, and without berthd every measured figure is unknown.
+func TestBuildMeasuredAndDeclared(t *testing.T) {
+ fp := proto.EngineFootprint{Engine: "e", Bytes: 13000 * mib, Runs: 4, RAMPeakBytes: 9000 * mib, ParkedRAMBytes: 20000 * mib}
+ e := &config.Engine{Type: "llamacpp", Priority: "foreground", MinTurn: config.Duration{Duration: 5 * time.Minute}}
+
+ c := Build("e", e, card(fp), nil)
+ if c.Type != "llamacpp" || !c.InConfig || c.Priority != (Text{Value: "foreground", Source: Declared}) {
+ t.Fatalf("type %q in_config %v priority %+v", c.Type, c.InConfig, c.Priority)
+ }
+ if c.LastPeak != (Bytes{Bytes: 13000 * mib, Source: Measured, Why: "4 runs measured"}) {
+ t.Fatalf("LastPeak = %+v", c.LastPeak)
+ }
+ if c.RAM != (Bytes{Bytes: 9000 * mib, Source: Measured, Why: "the most a run held while active"}) {
+ t.Fatalf("RAM = %+v", c.RAM)
+ }
+ if c.ParkedRAM != (Bytes{Bytes: 20000 * mib, Source: Measured}) {
+ t.Fatalf("ParkedRAM = %+v", c.ParkedRAM)
+ }
+ if c.MinTurn != (Duration{Seconds: 300, Source: Declared}) || c.Window != (Text{Value: "always", Source: Default}) {
+ t.Fatalf("MinTurn %+v Window %+v", c.MinTurn, c.Window)
+ }
+
+ e.NeedsRAMMiB = 16000
+ if got := Build("e", e, card(fp), nil).RAM; got != (Bytes{Bytes: 16000 * mib, Source: Declared, Why: "needs_ram_mib"}) {
+ t.Fatalf("declared RAM = %+v", got)
+ }
+
+ off := Build("e", nil, nil, nil)
+ if off.LastPeak != (Bytes{Source: Unknown, Why: "berthd did not answer"}) || off.Priority != (Text{Value: "interactive", Source: Default}) {
+ t.Fatalf("without berthd: LastPeak %+v Priority %+v", off.LastPeak, off.Priority)
+ }
+}
+
+// A llama-swap model has no berth.toml entry and no last-run figure of its
+// own (that sits on its group's entry): its peak and its terms come from
+// what its own runs measured and asked for.
+func TestBuildModelOfAGroup(t *testing.T) {
+ fp := proto.EngineFootprint{Engine: "Dirk-27B", HotPeakBytes: 23738 * mib, Priority: "foreground", MinTurnS: 300}
+ c := Build("Dirk-27B", nil, card(fp), nil)
+ if c.Priority != (Text{Value: "foreground", Source: Measured, Why: "its last run's lease"}) {
+ t.Fatalf("Priority = %+v", c.Priority)
+ }
+ if c.MinTurn != (Duration{Seconds: 300, Source: Measured, Why: "its last run's lease"}) {
+ t.Fatalf("MinTurn = %+v", c.MinTurn)
+ }
+ if c.LastPeak != (Bytes{Bytes: 23738 * mib, Source: Measured, Why: "the most a run under this name held"}) {
+ t.Fatalf("LastPeak = %+v", c.LastPeak)
+ }
+
+ e := &config.Engine{Priority: "batch"}
+ if got := Build("Dirk-27B", e, card(fp), nil).Priority; got != (Text{Value: "batch", Source: Declared}) {
+ t.Fatalf("declared priority = %+v, want berth.toml to win", got)
+ }
+}
+
+// A switch time is the book's median for the tier, hot is free by rule, and
+// a tier with no samples is unknown.
+func TestBuildSwitch(t *testing.T) {
+ book, err := swapcost.NewBook(filepath.Join(t.TempDir(), "swapcost.json"))
+ if err != nil {
+ t.Fatal(err)
+ }
+ for _, s := range []time.Duration{6 * time.Second, 8 * time.Second, 7 * time.Second} {
+ book.Record(swapcost.Sample{Engine: "e", Tier: swapcost.Cold, Took: s, At: time.Now()})
+ }
+ want := []Switch{
+ {Tier: swapcost.Hot, Took: Duration{Source: Default, Why: "already on the card, no switch"}},
+ {Tier: swapcost.Warm, Took: Duration{Source: Unknown, Why: "no switch measured"}},
+ {Tier: swapcost.Cool, Took: Duration{Source: Unknown, Why: "no switch measured"}},
+ {Tier: swapcost.Cold, Took: Duration{Seconds: 7, Source: Measured}, Samples: 3},
+ }
+ got := Build("e", nil, nil, book).Switch
+ if len(got) != len(want) {
+ t.Fatalf("switch = %+v", got)
+ }
+ for i := range want {
+ if got[i] != want[i] {
+ t.Errorf("switch[%d] = %+v, want %+v", i, got[i], want[i])
+ }
+ }
+}
diff --git a/internal/eventlog/eventlog.go b/internal/eventlog/eventlog.go
index b07c98e9..27707b63 100644
--- a/internal/eventlog/eventlog.go
+++ b/internal/eventlog/eventlog.go
@@ -4,10 +4,20 @@
// Package eventlog is berthd's event stream: one numbered sequence for the
// whole machine, kept in a ring and fanned out to watchers. Every card's
// broker on a node writes to the same log, so `berth events` is one ordered
-// stream however many cards the node governs.
+// stream however many cards the node governs. Opened on a file, the stream
+// outlives berthd: a restart reads the last Ring events back and numbers on
+// from them.
package eventlog
import (
+ "bufio"
+ "encoding/json"
+ "errors"
+ "io/fs"
+ "os"
+ "path/filepath"
+ "strconv"
+ "strings"
"sync"
"github.com/T-Crypt/berth/internal/proto"
@@ -20,29 +30,173 @@ const Ring = 1000
// closed.
const watchBuffer = 256
+// maxFile is how large the event file grows before it is moved to
+// path.1, which keeps one file of history behind the live one.
+const maxFile = 8 << 20
+
// Log is one node's event stream.
type Log struct {
mu sync.Mutex
seq uint64
events []proto.Event // ring, oldest first
subs map[chan proto.Event]struct{}
+ // file is where each event is appended, nil for a log kept in memory
+ // only or one whose file failed a write.
+ file *os.File
+ path string
+ size int64
+ maxFile int64
}
-// New returns an empty log.
+// New returns an empty log kept in memory only.
func New() *Log { return &Log{subs: map[chan proto.Event]struct{}{}} }
-// Emit numbers e, keeps it, and hands it to every watcher. A watcher whose
-// buffer is full is dropped, so one slow client never stalls a broker.
-func (l *Log) Emit(e proto.Event) {
+// Open returns the log kept in path: the newest Ring events in path.1 and
+// path come back, numbering goes on after the newest, and each new event is
+// appended. A line a crash cut short is skipped.
+func Open(path string) (*Log, error) {
+ l := New()
+ for _, p := range []string{path + ".1", path} {
+ if err := l.load(p); err != nil {
+ return nil, err
+ }
+ }
+ if err := os.MkdirAll(filepath.Dir(path), 0o700); err != nil {
+ return nil, err
+ }
+ f, err := os.OpenFile(path, os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0o600)
+ if err != nil {
+ return nil, err
+ }
+ fi, err := f.Stat()
+ if err != nil {
+ f.Close()
+ return nil, err
+ }
+ l.file, l.path, l.size, l.maxFile = f, path, fi.Size(), maxFile
+ if torn(path, fi.Size()) {
+ n, _ := f.Write([]byte{'\n'})
+ l.size += int64(n)
+ }
+ return l, nil
+}
+
+// torn is whether the file's last line has no newline: a crash cut it, and
+// the next event must start a line of its own.
+func torn(path string, size int64) bool {
+ if size == 0 {
+ return false
+ }
+ f, err := os.Open(path)
+ if err != nil {
+ return false
+ }
+ defer f.Close()
+ b := make([]byte, 1)
+ _, err = f.ReadAt(b, size-1)
+ return err == nil && b[0] != '\n'
+}
+
+func (l *Log) load(path string) error {
+ f, err := os.Open(path)
+ if errors.Is(err, fs.ErrNotExist) {
+ return nil
+ }
+ if err != nil {
+ return err
+ }
+ defer f.Close()
+ sc := bufio.NewScanner(f)
+ sc.Buffer(make([]byte, 64<<10), 1<<20)
+ for sc.Scan() {
+ var e proto.Event
+ if json.Unmarshal(sc.Bytes(), &e) != nil || e.Seq <= l.seq {
+ continue
+ }
+ l.seq = e.Seq
+ l.keep(e)
+ }
+ return sc.Err()
+}
+
+// Close stops writing to the file. The log keeps working in memory.
+func (l *Log) Close() error {
l.mu.Lock()
defer l.mu.Unlock()
- l.seq++
- e.Seq = l.seq
+ if l.file == nil {
+ return nil
+ }
+ err := l.file.Close()
+ l.file = nil
+ return err
+}
+
+// LastLease is the highest lease number the kept events name, so a
+// restarted berthd does not hand out lease IDs its history already uses.
+func (l *Log) LastLease() uint64 {
+ l.mu.Lock()
+ defer l.mu.Unlock()
+ var n uint64
+ for _, e := range l.events {
+ i := strings.LastIndexByte(e.Lease, '-')
+ if i < 0 {
+ continue
+ }
+ if v, err := strconv.ParseUint(e.Lease[i+1:], 10, 64); err == nil && v > n {
+ n = v
+ }
+ }
+ return n
+}
+
+func (l *Log) keep(e proto.Event) {
if len(l.events) == Ring {
copy(l.events, l.events[1:])
l.events = l.events[:Ring-1]
}
l.events = append(l.events, e)
+}
+
+// write appends e to the file, moving the file aside first when it is
+// full. A failed write ends the file: the stream goes on in memory.
+func (l *Log) write(e proto.Event) {
+ if l.file == nil {
+ return
+ }
+ line, err := json.Marshal(e)
+ if err != nil {
+ return
+ }
+ line = append(line, '\n')
+ if l.size+int64(len(line)) > l.maxFile && l.size > 0 {
+ l.file.Close()
+ l.file = nil
+ if os.Rename(l.path, l.path+".1") != nil {
+ return
+ }
+ f, err := os.OpenFile(l.path, os.O_CREATE|os.O_WRONLY|os.O_TRUNC|os.O_APPEND, 0o600)
+ if err != nil {
+ return
+ }
+ l.file, l.size = f, 0
+ }
+ n, err := l.file.Write(line)
+ l.size += int64(n)
+ if err != nil {
+ l.file.Close()
+ l.file = nil
+ }
+}
+
+// Emit numbers e, keeps it, and hands it to every watcher. A watcher whose
+// buffer is full is dropped, so one slow client never stalls a broker.
+func (l *Log) Emit(e proto.Event) {
+ l.mu.Lock()
+ defer l.mu.Unlock()
+ l.seq++
+ e.Seq = l.seq
+ l.keep(e)
+ l.write(e)
for ch := range l.subs {
select {
case ch <- e:
diff --git a/internal/eventlog/eventlog_test.go b/internal/eventlog/eventlog_test.go
new file mode 100644
index 00000000..c4fe16a9
--- /dev/null
+++ b/internal/eventlog/eventlog_test.go
@@ -0,0 +1,114 @@
+// Copyright 2026 Trevin Tindall (T-Crypt)
+// SPDX-License-Identifier: Apache-2.0
+
+package eventlog
+
+import (
+ "os"
+ "path/filepath"
+ "testing"
+ "time"
+
+ "github.com/T-Crypt/berth/internal/proto"
+)
+
+func open(t *testing.T, path string) *Log {
+ t.Helper()
+ l, err := Open(path)
+ if err != nil {
+ t.Fatal(err)
+ }
+ t.Cleanup(func() { _ = l.Close() })
+ return l
+}
+
+func kinds(evs []proto.Event) []string {
+ var out []string
+ for _, e := range evs {
+ out = append(out, e.Kind)
+ }
+ return out
+}
+
+// A restarted berthd shows the events from before it, and numbers its own
+// after them, so `berth events --since N` and `berth why` read across a
+// deploy.
+func TestOpenKeepsTheStreamAcrossARestart(t *testing.T) {
+ path := filepath.Join(t.TempDir(), "events.jsonl")
+ l := open(t, path)
+ l.Emit(proto.Event{Kind: proto.EvRequested, Lease: "strata-7", At: time.Unix(10, 0)})
+ l.Emit(proto.Event{Kind: proto.EvGranted, Lease: "strata-7", At: time.Unix(11, 0)})
+ if err := l.Close(); err != nil {
+ t.Fatal(err)
+ }
+
+ l2 := open(t, path)
+ if got := l2.Last(); got != 2 {
+ t.Fatalf("Last after the restart = %d, want 2", got)
+ }
+ l2.Emit(proto.Event{Kind: proto.EvDraining, Lease: "strata-7"})
+ evs, _, _ := l2.Events(0, false)
+ if got := kinds(evs); len(got) != 3 || got[0] != proto.EvRequested || got[2] != proto.EvDraining {
+ t.Fatalf("events after the restart = %v", got)
+ }
+ if evs[2].Seq != 3 {
+ t.Fatalf("the first event after the restart is seq %d, want 3", evs[2].Seq)
+ }
+ if got := l2.LastLease(); got != 7 {
+ t.Fatalf("LastLease = %d, want 7", got)
+ }
+}
+
+// A line cut short by a crash is skipped, not the whole file.
+func TestOpenSkipsATornLine(t *testing.T) {
+ path := filepath.Join(t.TempDir(), "events.jsonl")
+ data := `{"seq":1,"kind":"lease.requested","lease":"a-1"}` + "\n" + `{"seq":2,"kind":"lease.gr`
+ if err := os.WriteFile(path, []byte(data), 0o600); err != nil {
+ t.Fatal(err)
+ }
+ l := open(t, path)
+ l.Emit(proto.Event{Kind: proto.EvGranted, Lease: "a-1"})
+ evs, _, _ := l.Events(0, false)
+ if got := kinds(evs); len(got) != 2 || got[1] != proto.EvGranted || evs[1].Seq != 2 {
+ t.Fatalf("events = %v (%+v)", got, evs)
+ }
+ l.Close()
+ l3 := open(t, path)
+ if got := l3.Last(); got != 2 {
+ t.Fatalf("Last after a torn line and a restart = %d, want 2", got)
+ }
+}
+
+// The file is moved aside once it passes its cap, so it never grows without
+// bound, and the events in the moved file still come back.
+func TestOpenRotatesTheFile(t *testing.T) {
+ path := filepath.Join(t.TempDir(), "events.jsonl")
+ l := open(t, path)
+ l.maxFile = 200
+ for range 10 {
+ l.Emit(proto.Event{Kind: proto.EvRequested, Lease: "comfyui-1", Detail: "a detail line"})
+ }
+ l.Close()
+ if fi, err := os.Stat(path); err != nil || fi.Size() > 400 {
+ t.Fatalf("the live file was not rotated: %v %v", fi, err)
+ }
+ if _, err := os.Stat(path + ".1"); err != nil {
+ t.Fatalf("no rotated file: %v", err)
+ }
+ l2 := open(t, path)
+ if got := l2.Last(); got != 10 {
+ t.Fatalf("Last after rotation = %d, want 10", got)
+ }
+}
+
+// A first start has no state directory yet; Open makes it, as the other
+// state files do.
+func TestOpenMakesTheStateDirectory(t *testing.T) {
+ path := filepath.Join(t.TempDir(), "state", "berth", "events.jsonl")
+ l := open(t, path)
+ l.Emit(proto.Event{Kind: proto.EvNodeStarted})
+ l.Close()
+ if fi, err := os.Stat(path); err != nil || fi.Size() == 0 {
+ t.Fatalf("no events written: %v %v", fi, err)
+ }
+}
diff --git a/internal/fit/fit.go b/internal/fit/fit.go
new file mode 100644
index 00000000..dc31a12b
--- /dev/null
+++ b/internal/fit/fit.go
@@ -0,0 +1,175 @@
+// Copyright 2026 Trevin Tindall (T-Crypt)
+// SPDX-License-Identifier: Apache-2.0
+
+// Package fit ranks where a request can run: which engine, by what path
+// (resume from RAM, a free card, a drain of the holder, or a wait for a card
+// something keeps), and what the start costs, each cost with its source. It
+// reads a status and the engines' contracts and does no I/O, so a recorded
+// status replays the same answer. Fit advises; the broker decides.
+package fit
+
+import (
+ "fmt"
+ "slices"
+ "strings"
+
+ "github.com/T-Crypt/berth/internal/config"
+ "github.com/T-Crypt/berth/internal/contract"
+ "github.com/T-Crypt/berth/internal/node"
+ "github.com/T-Crypt/berth/internal/proto"
+ "github.com/T-Crypt/berth/internal/swapcost"
+)
+
+// Path is how an engine gets its card, cheapest first.
+type Path int
+
+const (
+ Running Path = iota // its own lease holds the card: nothing to switch
+ Resume // parked in RAM: a resume puts it back
+ Free // nothing holds, uses or waits for the card
+ Drain // the holder is drained first
+ Blocked // a VM, a cordon or a pinned lease keeps the card
+)
+
+var pathNames = [...]string{Running: "running", Resume: "resume", Free: "free", Drain: "drain", Blocked: "blocked"}
+
+func (p Path) String() string { return pathNames[p] }
+
+// MarshalText writes the path's name.
+func (p Path) MarshalText() ([]byte, error) { return []byte(p.String()), nil }
+
+// Candidate is one engine and what it takes to bring it up.
+type Candidate struct {
+ Engine string `json:"engine" yaml:"engine"`
+ Path Path `json:"path" yaml:"path"`
+ Why string `json:"why" yaml:"why"`
+ // Tier is where the engine's weights sit before the start.
+ Tier swapcost.Tier `json:"tier" yaml:"tier"`
+ // Start is the measured switch time from Tier, unknown without samples
+ // or while the card is out of reach.
+ Start contract.Duration `json:"start" yaml:"start"`
+
+ Entry *config.Engine `json:"-" yaml:"-"`
+}
+
+// Rank orders engines by path, cheapest first; engines on one path keep
+// their order (berth.toml's). book is the gateway's switch book, nil for
+// none: every start is then unknown.
+func Rank(st *proto.Status, engines []*config.Engine, book *swapcost.Book) []Candidate {
+ out := make([]Candidate, 0, len(engines))
+ for _, e := range engines {
+ out = append(out, candidate(st, e, book))
+ }
+ slices.SortStableFunc(out, func(a, b Candidate) int { return int(a.Path) - int(b.Path) })
+ return out
+}
+
+func candidate(st *proto.Status, e *config.Engine, book *swapcost.Book) Candidate {
+ c := Candidate{Engine: e.Name, Entry: e, Tier: swapcost.Cold}
+ v := node.View(st, e.GPU)
+ switch {
+ case running(v, e.Name):
+ c.Path, c.Tier, c.Why = Running, swapcost.Hot, "already on the card"
+ case Parked(st, e.Name):
+ c.Path, c.Tier, c.Why = Resume, swapcost.Warm, "parked in RAM"
+ case CardFree(v):
+ c.Path, c.Why = Free, "the card is free"
+ case v != nil && OutOfReach(v) == "":
+ c.Path, c.Why = Drain, "drains "+holders(v)
+ default:
+ c.Path, c.Why = Blocked, OutOfReach(v)
+ if c.Why == "" {
+ c.Why = "berthd did not report the card"
+ }
+ }
+ if c.Path == Blocked {
+ c.Start = contract.Duration{Source: contract.Unknown, Why: "the card is out of reach"}
+ return c
+ }
+ for _, s := range contract.Build(e.Name, e, st, book).Switch {
+ if s.Tier == c.Tier {
+ c.Start = s.Took
+ }
+ }
+ return c
+}
+
+// holders names what holds or waits for the card, for a drain's reason.
+func holders(st *proto.Status) string {
+ var names []string
+ for _, l := range st.Leases {
+ names = append(names, l.Holder())
+ }
+ switch {
+ case len(names) > 0:
+ return strings.Join(names, " and ")
+ case len(st.Squatters) > 0:
+ return "memory no lease accounts for"
+ default:
+ return "the waiters ahead"
+ }
+}
+
+// running is whether a lease of the engine, or of a model in its group (a
+// llama-swap's), is active on the card: a request for it needs no switch.
+func running(st *proto.Status, name string) bool {
+ return st != nil && slices.ContainsFunc(st.Leases, func(l proto.Lease) bool {
+ return l.State == "active" && (l.Engine == name || l.Group == name)
+ })
+}
+
+// Parked is whether the engine's own lease sits parked in RAM. A parked model
+// behind a router (a llama-swap model under the llama-swap group) has its own
+// front, which wakes it when the router forwards a request, so the router is
+// never parked on its behalf.
+func Parked(st *proto.Status, name string) bool {
+ if st == nil {
+ return false
+ }
+ for _, l := range st.Leases {
+ if l.Engine == name && l.State == "parked" {
+ return true
+ }
+ }
+ return false
+}
+
+// CardFree is whether nothing holds, uses or waits for the card and the card
+// serves this machine. A parked lease holds it: a batch job must not evict an
+// engine someone parked in RAM to come back to. A stowaway or a tracked app
+// (Blender) holds memory no lease accounts for, and a start beside it is the
+// collision Berth exists to prevent.
+func CardFree(st *proto.Status) bool {
+ if st == nil || st.Passthrough != nil || st.ProbeErr != "" ||
+ len(st.Leases) > 0 || len(st.Waiting) > 0 || len(st.Squatters) > 0 || Cordoned(st) {
+ return false
+ }
+ // Status lists every claim rule; only one matching right now is in use.
+ return !slices.ContainsFunc(st.Claims, func(c proto.Claim) bool { return c.Active })
+}
+
+// OutOfReach says why nothing can start on the card now, "" when something
+// can: the card is in a VM, cordoned, or a pinned lease (a game, a hold, a
+// training) has it.
+func OutOfReach(st *proto.Status) string {
+ if st == nil {
+ return ""
+ }
+ if st.Passthrough != nil {
+ return "the card is passed through to a VM"
+ }
+ if Cordoned(st) {
+ return "the card is cordoned"
+ }
+ for _, l := range st.Leases {
+ if l.Pinned {
+ return fmt.Sprintf("%s holds the card", l.Holder())
+ }
+ }
+ return ""
+}
+
+// Cordoned is whether a scheduled card in the status is cordoned.
+func Cordoned(st *proto.Status) bool {
+ return slices.ContainsFunc(st.GPUs, func(g proto.GPU) bool { return g.Scheduled && g.Cordoned })
+}
diff --git a/internal/fit/fit_test.go b/internal/fit/fit_test.go
new file mode 100644
index 00000000..e6f3122f
--- /dev/null
+++ b/internal/fit/fit_test.go
@@ -0,0 +1,165 @@
+// Copyright 2026 Trevin Tindall (T-Crypt)
+// SPDX-License-Identifier: Apache-2.0
+
+package fit
+
+import (
+ "encoding/json"
+ "os"
+ "path/filepath"
+ "testing"
+ "time"
+
+ "github.com/T-Crypt/berth/internal/config"
+ "github.com/T-Crypt/berth/internal/contract"
+ "github.com/T-Crypt/berth/internal/proto"
+ "github.com/T-Crypt/berth/internal/swapcost"
+)
+
+// recorded is a status berthd answered on aphotic on 2026-11-13, with a
+// training lease holding the card pinned. Footprints of jobs Fit never ranks
+// were cut and the training's name generalised; nothing else was changed.
+func recorded(t *testing.T) *proto.Status {
+ t.Helper()
+ b, err := os.ReadFile(filepath.Join("testdata", "aphotic-training-pinned.json"))
+ if err != nil {
+ t.Fatal(err)
+ }
+ var st proto.Status
+ if err := json.Unmarshal(b, &st); err != nil {
+ t.Fatal(err)
+ }
+ return &st
+}
+
+func engines(names ...string) []*config.Engine {
+ out := make([]*config.Engine, len(names))
+ for i, n := range names {
+ out[i] = &config.Engine{Name: n}
+ }
+ return out
+}
+
+type got struct {
+ Engine string
+ Path Path
+ Why string
+ Tier swapcost.Tier
+ Start contract.Duration
+}
+
+func summary(cs []Candidate) []got {
+ out := make([]got, len(cs))
+ for i, c := range cs {
+ out[i] = got{c.Engine, c.Path, c.Why, c.Tier, c.Start}
+ }
+ return out
+}
+
+func check(t *testing.T, cs []Candidate, want []got) {
+ t.Helper()
+ g := summary(cs)
+ if len(g) != len(want) {
+ t.Fatalf("candidates = %+v, want %+v", g, want)
+ }
+ for i := range want {
+ if g[i] != want[i] {
+ t.Errorf("candidate %d = %+v\nwant %+v", i, g[i], want[i])
+ }
+ }
+}
+
+var unknownCold = contract.Duration{Source: contract.Unknown, Why: "no switch measured"}
+
+// The recorded training holds the card pinned: every engine waits for it,
+// no start is estimated, and the reason names the holder.
+func TestRankRecordedTraining(t *testing.T) {
+ blocked := contract.Duration{Source: contract.Unknown, Why: "the card is out of reach"}
+ check(t, Rank(recorded(t), engines("strata", "comfyui"), nil), []got{
+ {"strata", Blocked, "training holds the card", swapcost.Cold, blocked},
+ {"comfyui", Blocked, "training holds the card", swapcost.Cold, blocked},
+ })
+}
+
+// Paths rank running, resume, free, drain, blocked; engines on one path keep
+// berth.toml's order. Starts come from the switch book by tier, unknown
+// where it has no samples.
+func TestRankOrdersByPath(t *testing.T) {
+ st := recorded(t)
+ st.Leases = nil
+ book, err := swapcost.NewBook(filepath.Join(t.TempDir(), "swapcost.json"))
+ if err != nil {
+ t.Fatal(err)
+ }
+ book.Record(swapcost.Sample{Engine: "comfyui", Tier: swapcost.Cold, Took: 8 * time.Second, At: time.Now()})
+ book.Record(swapcost.Sample{Engine: "strata", Tier: swapcost.Warm, Took: 4 * time.Second, At: time.Now()})
+
+ check(t, Rank(st, engines("comfyui", "acestep"), book), []got{
+ {"comfyui", Free, "the card is free", swapcost.Cold, contract.Duration{Seconds: 8, Source: contract.Measured}},
+ {"acestep", Free, "the card is free", swapcost.Cold, unknownCold},
+ })
+
+ st.Leases = []proto.Lease{
+ {Engine: "strata", State: "parked"},
+ {Engine: "dirk", Group: "llama-swap", State: "active"},
+ }
+ check(t, Rank(st, engines("comfyui", "strata"), book), []got{
+ {"strata", Resume, "parked in RAM", swapcost.Warm, contract.Duration{Seconds: 4, Source: contract.Measured}},
+ {"comfyui", Drain, "drains strata and llama-swap (dirk)", swapcost.Cold, contract.Duration{Seconds: 8, Source: contract.Measured}},
+ })
+
+ // The model that holds the card, and the router whose model holds it,
+ // are already there: no drain of themselves, no switch.
+ hot := contract.Duration{Source: contract.Default, Why: "already on the card, no switch"}
+ check(t, Rank(st, engines("comfyui", "dirk", "llama-swap"), book), []got{
+ {"dirk", Running, "already on the card", swapcost.Hot, hot},
+ {"llama-swap", Running, "already on the card", swapcost.Hot, hot},
+ {"comfyui", Drain, "drains strata and llama-swap (dirk)", swapcost.Cold, contract.Duration{Seconds: 8, Source: contract.Measured}},
+ })
+}
+
+// The card is free only when nothing holds, uses or waits for it and it
+// serves this machine.
+func TestCardFree(t *testing.T) {
+ for _, tc := range []struct {
+ name string
+ st *proto.Status
+ want bool
+ }{
+ {"no reading", nil, false},
+ {"empty card", &proto.Status{}, true},
+ {"a lease", &proto.Status{Leases: []proto.Lease{{Engine: "chat"}}}, false},
+ {"a waiter", &proto.Status{Waiting: []proto.Lease{{Engine: "chat"}}}, false},
+ {"a stowaway", &proto.Status{Squatters: []proto.Squatter{{PID: 7}}}, false},
+ {"claim rules, none matching", &proto.Status{Claims: []proto.Claim{{Name: "steam-game"}, {Name: "blender"}}}, true},
+ {"a tracked app running", &proto.Status{Claims: []proto.Claim{{Name: "steam-game"}, {Name: "blender", Active: true}}}, false},
+ {"in a VM", &proto.Status{Passthrough: &proto.Passthrough{State: proto.CardPassthrough}}, false},
+ {"probe failing", &proto.Status{ProbeErr: "nvidia-smi: exit 9"}, false},
+ {"cordoned", &proto.Status{GPUs: []proto.GPU{{Scheduled: true, Cordoned: true}}}, false},
+ } {
+ if got := CardFree(tc.st); got != tc.want {
+ t.Errorf("%s: CardFree = %v, want %v", tc.name, got, tc.want)
+ }
+ }
+}
+
+// The card is out of reach while it is in a VM, cordoned, or a pinned lease
+// has it; an ordinary holder is drained the usual way.
+func TestOutOfReach(t *testing.T) {
+ for _, tc := range []struct {
+ name string
+ st *proto.Status
+ want string
+ }{
+ {"no reading", nil, ""},
+ {"free card", &proto.Status{}, ""},
+ {"an ordinary holder", &proto.Status{Leases: []proto.Lease{{Engine: "llama-swap"}}}, ""},
+ {"a game", &proto.Status{Leases: []proto.Lease{{Engine: "claim:steam-game", Pinned: true}}}, "claim:steam-game holds the card"},
+ {"in a VM", &proto.Status{Passthrough: &proto.Passthrough{State: proto.CardPassthrough}}, "the card is passed through to a VM"},
+ {"cordoned", &proto.Status{GPUs: []proto.GPU{{Scheduled: true, Cordoned: true}}}, "the card is cordoned"},
+ } {
+ if got := OutOfReach(tc.st); got != tc.want {
+ t.Errorf("%s: OutOfReach = %q, want %q", tc.name, got, tc.want)
+ }
+ }
+}
diff --git a/internal/fit/testdata/aphotic-training-pinned.json b/internal/fit/testdata/aphotic-training-pinned.json
new file mode 100644
index 00000000..69287bfe
--- /dev/null
+++ b/internal/fit/testdata/aphotic-training-pinned.json
@@ -0,0 +1,1250 @@
+{
+ "api": 1,
+ "version": "0.5.1-rc.2-next-deploy.1bfd344",
+ "gpus": [
+ {
+ "card": "gpu0",
+ "addr": "0000:01:00.0",
+ "index": 0,
+ "name": "NVIDIA GeForce RTX 4090",
+ "scheduled": true,
+ "total": 25757220864,
+ "used": 13128171520,
+ "free": 12209618944,
+ "baseline": 10485760,
+ "age_seconds": 1.6876834120000002,
+ "util_pct": 100,
+ "temp_c": 52,
+ "power_w": 235.7,
+ "power_limit_w": 350,
+ "fan_pct": 0,
+ "clock_sm_mhz": 2595,
+ "clock_mem_mhz": 10251,
+ "pcie_gen": 4,
+ "pcie_width": 16
+ }
+ ],
+ "leases": [
+ {
+ "card": "gpu0",
+ "id": "training-203",
+ "engine": "training",
+ "priority": "hold",
+ "state": "active",
+ "needs": 0,
+ "pid": 2053292,
+ "pid_start": 41728978,
+ "gpu_pids": [
+ 2053292
+ ],
+ "used": 13117685760,
+ "age_seconds": 23739.29616739,
+ "pinned": true,
+ "training": true
+ }
+ ],
+ "waiting": [],
+ "squatters": [],
+ "unprotected": [],
+ "engines": [
+ {
+ "name": "llama-swap",
+ "type": "llama-swap",
+ "url": "http://127.0.0.1:9090",
+ "up": true,
+ "state": "up",
+ "models": [],
+ "metrics": {
+ "prompt_tokens": 0,
+ "output_tokens": 0,
+ "prompt_seconds": 0,
+ "output_seconds": 0,
+ "processing": 0,
+ "waiting": 0
+ },
+ "decode_tps": 0,
+ "prefill_tps": 0,
+ "metrics_reported": false,
+ "at": "2026-11-13T09:21:32.63308401-07:00",
+ "age_seconds": 4.319828335,
+ "kind": "llm"
+ },
+ {
+ "name": "strata",
+ "type": "strata",
+ "url": "http://127.0.0.1:8080",
+ "up": false,
+ "error": "Get \"http://127.0.0.1:8080/health\": dial tcp 127.0.0.1:8080: connect: connection refused",
+ "state": "down",
+ "models": [],
+ "metrics": {
+ "prompt_tokens": 0,
+ "output_tokens": 0,
+ "prompt_seconds": 0,
+ "output_seconds": 0,
+ "processing": 0,
+ "waiting": 0
+ },
+ "decode_tps": 0,
+ "prefill_tps": 0,
+ "metrics_reported": false,
+ "at": "2026-11-13T09:21:32.633853518-07:00",
+ "age_seconds": 4.319059136,
+ "kind": "llm"
+ },
+ {
+ "name": "strata-sc117",
+ "type": "strata",
+ "url": "http://127.0.0.1:8081",
+ "up": false,
+ "error": "Get \"http://127.0.0.1:8081/health\": dial tcp 127.0.0.1:8081: connect: connection refused",
+ "state": "down",
+ "models": [],
+ "metrics": {
+ "prompt_tokens": 0,
+ "output_tokens": 0,
+ "prompt_seconds": 0,
+ "output_seconds": 0,
+ "processing": 0,
+ "waiting": 0
+ },
+ "decode_tps": 0,
+ "prefill_tps": 0,
+ "metrics_reported": false,
+ "at": "2026-11-13T09:21:32.633964241-07:00",
+ "age_seconds": 4.318948621,
+ "kind": "llm"
+ },
+ {
+ "name": "comfyui",
+ "type": "comfyui",
+ "url": "http://127.0.0.1:8188",
+ "up": false,
+ "error": "Get \"http://127.0.0.1:8188/queue\": dial tcp 127.0.0.1:8188: connect: connection refused",
+ "state": "down",
+ "models": [],
+ "metrics": {
+ "prompt_tokens": 0,
+ "output_tokens": 0,
+ "prompt_seconds": 0,
+ "output_seconds": 0,
+ "processing": 0,
+ "waiting": 0
+ },
+ "decode_tps": 0,
+ "prefill_tps": 0,
+ "metrics_reported": false,
+ "at": "2026-11-13T09:21:32.634014252-07:00",
+ "age_seconds": 4.31889875,
+ "kind": "image"
+ },
+ {
+ "name": "acestep",
+ "type": "custom",
+ "url": "http://127.0.0.1:8001",
+ "up": false,
+ "error": "Get \"http://127.0.0.1:8001/health\": dial tcp 127.0.0.1:8001: connect: connection refused",
+ "state": "down",
+ "models": [],
+ "metrics": {
+ "prompt_tokens": 0,
+ "output_tokens": 0,
+ "prompt_seconds": 0,
+ "output_seconds": 0,
+ "processing": 0,
+ "waiting": 0
+ },
+ "decode_tps": 0,
+ "prefill_tps": 0,
+ "metrics_reported": false,
+ "at": "2026-11-13T09:21:32.634061104-07:00",
+ "age_seconds": 4.3188521,
+ "kind": "other"
+ },
+ {
+ "name": "Ollama",
+ "type": "ollama",
+ "url": "http://127.0.0.1:11434",
+ "up": true,
+ "state": "up",
+ "models": [],
+ "metrics": {
+ "prompt_tokens": 0,
+ "output_tokens": 0,
+ "prompt_seconds": 0,
+ "output_seconds": 0,
+ "processing": 0,
+ "waiting": 0
+ },
+ "decode_tps": 0,
+ "prefill_tps": 0,
+ "metrics_reported": false,
+ "at": "2026-11-13T09:21:32.634100694-07:00",
+ "age_seconds": 4.318812787,
+ "kind": "llm"
+ }
+ ],
+ "host": {
+ "supported": true,
+ "cpu_pct": 4.690618762475052,
+ "cores": 32,
+ "ram_total": 67099635712,
+ "ram_used": 19988193280,
+ "swap_total": 4294963200,
+ "swap_used": 4294938624,
+ "load": [
+ 3.44,
+ 2.78,
+ 2.45
+ ]
+ },
+ "insights": {
+ "requests_5m": 0,
+ "rpm": 0,
+ "p50_ms": 0,
+ "p95_ms": 0,
+ "prefill_ms": 0,
+ "tpot_ms": 0,
+ "decode_tps": 0,
+ "prefill_tps": 0,
+ "cache_pct": 0,
+ "reingest_pct": 0,
+ "mtp_pct": 0,
+ "ledger": [
+ {
+ "label": "1h",
+ "requests": 0,
+ "generated": 0,
+ "fresh": 0,
+ "cached": 0
+ },
+ {
+ "label": "24h",
+ "requests": 10031,
+ "generated": 114870,
+ "fresh": 10200544,
+ "cached": 7556
+ },
+ {
+ "label": "since Nov 13 00:58",
+ "requests": 10031,
+ "generated": 114870,
+ "fresh": 10200544,
+ "cached": 7556
+ }
+ ],
+ "leaderboard": [
+ {
+ "engine": "llama-swap",
+ "model": "Dirk-27B",
+ "requests": 290,
+ "prompt_tokens": 9981225,
+ "output_tokens": 72072,
+ "decode_tps_median": 64.2149693436799,
+ "cache_pct": 0,
+ "mtp_pct": 52.065310492505354
+ },
+ {
+ "engine": "llama-swap",
+ "model": "Qwen3.6-35B-A3B-IQ4-Agent",
+ "requests": 9741,
+ "prompt_tokens": 226875,
+ "output_tokens": 42798,
+ "decode_tps_median": 220.83870094064787,
+ "cache_pct": 3.3304683195592286,
+ "mtp_pct": 50.13787034320915
+ }
+ ],
+ "recent": [
+ {
+ "id": "llama-swap/10973@1794565611",
+ "engine": "llama-swap",
+ "model": "Qwen3.6-35B-A3B-IQ4-Agent",
+ "t": "2026-11-13T03:26:51-07:00",
+ "status": 500,
+ "prompt_tokens": 0,
+ "cache_tokens": 0,
+ "output_tokens": 0,
+ "prefill_tps": 0,
+ "decode_tps": 0,
+ "duration_ms": 250,
+ "prefill_ms": 0
+ },
+ {
+ "id": "llama-swap/10974@1794565611",
+ "engine": "llama-swap",
+ "model": "Qwen3.6-35B-A3B-IQ4-Agent",
+ "t": "2026-11-13T03:26:51-07:00",
+ "status": 500,
+ "prompt_tokens": 0,
+ "cache_tokens": 0,
+ "output_tokens": 0,
+ "prefill_tps": 0,
+ "decode_tps": 0,
+ "duration_ms": 251,
+ "prefill_ms": 0
+ },
+ {
+ "id": "llama-swap/10975@1794565611",
+ "engine": "llama-swap",
+ "model": "Qwen3.6-35B-A3B-IQ4-Agent",
+ "t": "2026-11-13T03:26:51-07:00",
+ "status": 500,
+ "prompt_tokens": 0,
+ "cache_tokens": 0,
+ "output_tokens": 0,
+ "prefill_tps": 0,
+ "decode_tps": 0,
+ "duration_ms": 251,
+ "prefill_ms": 0
+ },
+ {
+ "id": "llama-swap/10976@1794565611",
+ "engine": "llama-swap",
+ "model": "Qwen3.6-35B-A3B-IQ4-Agent",
+ "t": "2026-11-13T03:26:51-07:00",
+ "status": 500,
+ "prompt_tokens": 0,
+ "cache_tokens": 0,
+ "output_tokens": 0,
+ "prefill_tps": 0,
+ "decode_tps": 0,
+ "duration_ms": 251,
+ "prefill_ms": 0
+ },
+ {
+ "id": "llama-swap/10969@1794565610",
+ "engine": "llama-swap",
+ "model": "Qwen3.6-35B-A3B-IQ4-Agent",
+ "t": "2026-11-13T03:26:50-07:00",
+ "status": 500,
+ "prompt_tokens": 0,
+ "cache_tokens": 0,
+ "output_tokens": 0,
+ "prefill_tps": 0,
+ "decode_tps": 0,
+ "duration_ms": 251,
+ "prefill_ms": 0
+ },
+ {
+ "id": "llama-swap/10970@1794565610",
+ "engine": "llama-swap",
+ "model": "Qwen3.6-35B-A3B-IQ4-Agent",
+ "t": "2026-11-13T03:26:50-07:00",
+ "status": 500,
+ "prompt_tokens": 0,
+ "cache_tokens": 0,
+ "output_tokens": 0,
+ "prefill_tps": 0,
+ "decode_tps": 0,
+ "duration_ms": 251,
+ "prefill_ms": 0
+ },
+ {
+ "id": "llama-swap/10971@1794565610",
+ "engine": "llama-swap",
+ "model": "Qwen3.6-35B-A3B-IQ4-Agent",
+ "t": "2026-11-13T03:26:50-07:00",
+ "status": 500,
+ "prompt_tokens": 0,
+ "cache_tokens": 0,
+ "output_tokens": 0,
+ "prefill_tps": 0,
+ "decode_tps": 0,
+ "duration_ms": 250,
+ "prefill_ms": 0
+ },
+ {
+ "id": "llama-swap/10972@1794565610",
+ "engine": "llama-swap",
+ "model": "Qwen3.6-35B-A3B-IQ4-Agent",
+ "t": "2026-11-13T03:26:50-07:00",
+ "status": 500,
+ "prompt_tokens": 0,
+ "cache_tokens": 0,
+ "output_tokens": 0,
+ "prefill_tps": 0,
+ "decode_tps": 0,
+ "duration_ms": 251,
+ "prefill_ms": 0
+ },
+ {
+ "id": "llama-swap/10965@1794565609",
+ "engine": "llama-swap",
+ "model": "Qwen3.6-35B-A3B-IQ4-Agent",
+ "t": "2026-11-13T03:26:49-07:00",
+ "status": 500,
+ "prompt_tokens": 0,
+ "cache_tokens": 0,
+ "output_tokens": 0,
+ "prefill_tps": 0,
+ "decode_tps": 0,
+ "duration_ms": 251,
+ "prefill_ms": 0
+ },
+ {
+ "id": "llama-swap/10966@1794565609",
+ "engine": "llama-swap",
+ "model": "Qwen3.6-35B-A3B-IQ4-Agent",
+ "t": "2026-11-13T03:26:49-07:00",
+ "status": 500,
+ "prompt_tokens": 0,
+ "cache_tokens": 0,
+ "output_tokens": 0,
+ "prefill_tps": 0,
+ "decode_tps": 0,
+ "duration_ms": 251,
+ "prefill_ms": 0
+ },
+ {
+ "id": "llama-swap/10967@1794565609",
+ "engine": "llama-swap",
+ "model": "Qwen3.6-35B-A3B-IQ4-Agent",
+ "t": "2026-11-13T03:26:49-07:00",
+ "status": 500,
+ "prompt_tokens": 0,
+ "cache_tokens": 0,
+ "output_tokens": 0,
+ "prefill_tps": 0,
+ "decode_tps": 0,
+ "duration_ms": 251,
+ "prefill_ms": 0
+ },
+ {
+ "id": "llama-swap/10968@1794565609",
+ "engine": "llama-swap",
+ "model": "Qwen3.6-35B-A3B-IQ4-Agent",
+ "t": "2026-11-13T03:26:49-07:00",
+ "status": 500,
+ "prompt_tokens": 0,
+ "cache_tokens": 0,
+ "output_tokens": 0,
+ "prefill_tps": 0,
+ "decode_tps": 0,
+ "duration_ms": 251,
+ "prefill_ms": 0
+ },
+ {
+ "id": "llama-swap/10961@1794565608",
+ "engine": "llama-swap",
+ "model": "Qwen3.6-35B-A3B-IQ4-Agent",
+ "t": "2026-11-13T03:26:48-07:00",
+ "status": 500,
+ "prompt_tokens": 0,
+ "cache_tokens": 0,
+ "output_tokens": 0,
+ "prefill_tps": 0,
+ "decode_tps": 0,
+ "duration_ms": 251,
+ "prefill_ms": 0
+ },
+ {
+ "id": "llama-swap/10962@1794565608",
+ "engine": "llama-swap",
+ "model": "Qwen3.6-35B-A3B-IQ4-Agent",
+ "t": "2026-11-13T03:26:48-07:00",
+ "status": 500,
+ "prompt_tokens": 0,
+ "cache_tokens": 0,
+ "output_tokens": 0,
+ "prefill_tps": 0,
+ "decode_tps": 0,
+ "duration_ms": 251,
+ "prefill_ms": 0
+ },
+ {
+ "id": "llama-swap/10963@1794565608",
+ "engine": "llama-swap",
+ "model": "Qwen3.6-35B-A3B-IQ4-Agent",
+ "t": "2026-11-13T03:26:48-07:00",
+ "status": 500,
+ "prompt_tokens": 0,
+ "cache_tokens": 0,
+ "output_tokens": 0,
+ "prefill_tps": 0,
+ "decode_tps": 0,
+ "duration_ms": 250,
+ "prefill_ms": 0
+ },
+ {
+ "id": "llama-swap/10964@1794565608",
+ "engine": "llama-swap",
+ "model": "Qwen3.6-35B-A3B-IQ4-Agent",
+ "t": "2026-11-13T03:26:48-07:00",
+ "status": 500,
+ "prompt_tokens": 0,
+ "cache_tokens": 0,
+ "output_tokens": 0,
+ "prefill_tps": 0,
+ "decode_tps": 0,
+ "duration_ms": 251,
+ "prefill_ms": 0
+ },
+ {
+ "id": "llama-swap/10959@1794565607",
+ "engine": "llama-swap",
+ "model": "Qwen3.6-35B-A3B-IQ4-Agent",
+ "t": "2026-11-13T03:26:47-07:00",
+ "status": 500,
+ "prompt_tokens": 0,
+ "cache_tokens": 0,
+ "output_tokens": 0,
+ "prefill_tps": 0,
+ "decode_tps": 0,
+ "duration_ms": 250,
+ "prefill_ms": 0
+ },
+ {
+ "id": "llama-swap/10960@1794565607",
+ "engine": "llama-swap",
+ "model": "Qwen3.6-35B-A3B-IQ4-Agent",
+ "t": "2026-11-13T03:26:47-07:00",
+ "status": 500,
+ "prompt_tokens": 0,
+ "cache_tokens": 0,
+ "output_tokens": 0,
+ "prefill_tps": 0,
+ "decode_tps": 0,
+ "duration_ms": 251,
+ "prefill_ms": 0
+ },
+ {
+ "id": "llama-swap/10957@1794565607",
+ "engine": "llama-swap",
+ "model": "Qwen3.6-35B-A3B-IQ4-Agent",
+ "t": "2026-11-13T03:26:47-07:00",
+ "status": 500,
+ "prompt_tokens": 0,
+ "cache_tokens": 0,
+ "output_tokens": 0,
+ "prefill_tps": 0,
+ "decode_tps": 0,
+ "duration_ms": 251,
+ "prefill_ms": 0
+ },
+ {
+ "id": "llama-swap/10958@1794565607",
+ "engine": "llama-swap",
+ "model": "Qwen3.6-35B-A3B-IQ4-Agent",
+ "t": "2026-11-13T03:26:47-07:00",
+ "status": 500,
+ "prompt_tokens": 0,
+ "cache_tokens": 0,
+ "output_tokens": 0,
+ "prefill_tps": 0,
+ "decode_tps": 0,
+ "duration_ms": 251,
+ "prefill_ms": 0
+ },
+ {
+ "id": "llama-swap/10953@1794565606",
+ "engine": "llama-swap",
+ "model": "Qwen3.6-35B-A3B-IQ4-Agent",
+ "t": "2026-11-13T03:26:46-07:00",
+ "status": 500,
+ "prompt_tokens": 0,
+ "cache_tokens": 0,
+ "output_tokens": 0,
+ "prefill_tps": 0,
+ "decode_tps": 0,
+ "duration_ms": 251,
+ "prefill_ms": 0
+ },
+ {
+ "id": "llama-swap/10954@1794565606",
+ "engine": "llama-swap",
+ "model": "Qwen3.6-35B-A3B-IQ4-Agent",
+ "t": "2026-11-13T03:26:46-07:00",
+ "status": 500,
+ "prompt_tokens": 0,
+ "cache_tokens": 0,
+ "output_tokens": 0,
+ "prefill_tps": 0,
+ "decode_tps": 0,
+ "duration_ms": 250,
+ "prefill_ms": 0
+ },
+ {
+ "id": "llama-swap/10955@1794565606",
+ "engine": "llama-swap",
+ "model": "Qwen3.6-35B-A3B-IQ4-Agent",
+ "t": "2026-11-13T03:26:46-07:00",
+ "status": 500,
+ "prompt_tokens": 0,
+ "cache_tokens": 0,
+ "output_tokens": 0,
+ "prefill_tps": 0,
+ "decode_tps": 0,
+ "duration_ms": 251,
+ "prefill_ms": 0
+ },
+ {
+ "id": "llama-swap/10956@1794565606",
+ "engine": "llama-swap",
+ "model": "Qwen3.6-35B-A3B-IQ4-Agent",
+ "t": "2026-11-13T03:26:46-07:00",
+ "status": 500,
+ "prompt_tokens": 0,
+ "cache_tokens": 0,
+ "output_tokens": 0,
+ "prefill_tps": 0,
+ "decode_tps": 0,
+ "duration_ms": 251,
+ "prefill_ms": 0
+ },
+ {
+ "id": "llama-swap/10949@1794565605",
+ "engine": "llama-swap",
+ "model": "Qwen3.6-35B-A3B-IQ4-Agent",
+ "t": "2026-11-13T03:26:45-07:00",
+ "status": 500,
+ "prompt_tokens": 0,
+ "cache_tokens": 0,
+ "output_tokens": 0,
+ "prefill_tps": 0,
+ "decode_tps": 0,
+ "duration_ms": 251,
+ "prefill_ms": 0
+ },
+ {
+ "id": "llama-swap/10950@1794565605",
+ "engine": "llama-swap",
+ "model": "Qwen3.6-35B-A3B-IQ4-Agent",
+ "t": "2026-11-13T03:26:45-07:00",
+ "status": 500,
+ "prompt_tokens": 0,
+ "cache_tokens": 0,
+ "output_tokens": 0,
+ "prefill_tps": 0,
+ "decode_tps": 0,
+ "duration_ms": 251,
+ "prefill_ms": 0
+ },
+ {
+ "id": "llama-swap/10951@1794565605",
+ "engine": "llama-swap",
+ "model": "Qwen3.6-35B-A3B-IQ4-Agent",
+ "t": "2026-11-13T03:26:45-07:00",
+ "status": 500,
+ "prompt_tokens": 0,
+ "cache_tokens": 0,
+ "output_tokens": 0,
+ "prefill_tps": 0,
+ "decode_tps": 0,
+ "duration_ms": 251,
+ "prefill_ms": 0
+ },
+ {
+ "id": "llama-swap/10952@1794565605",
+ "engine": "llama-swap",
+ "model": "Qwen3.6-35B-A3B-IQ4-Agent",
+ "t": "2026-11-13T03:26:45-07:00",
+ "status": 500,
+ "prompt_tokens": 0,
+ "cache_tokens": 0,
+ "output_tokens": 0,
+ "prefill_tps": 0,
+ "decode_tps": 0,
+ "duration_ms": 251,
+ "prefill_ms": 0
+ },
+ {
+ "id": "llama-swap/10945@1794565604",
+ "engine": "llama-swap",
+ "model": "Qwen3.6-35B-A3B-IQ4-Agent",
+ "t": "2026-11-13T03:26:44-07:00",
+ "status": 500,
+ "prompt_tokens": 0,
+ "cache_tokens": 0,
+ "output_tokens": 0,
+ "prefill_tps": 0,
+ "decode_tps": 0,
+ "duration_ms": 251,
+ "prefill_ms": 0
+ },
+ {
+ "id": "llama-swap/10946@1794565604",
+ "engine": "llama-swap",
+ "model": "Qwen3.6-35B-A3B-IQ4-Agent",
+ "t": "2026-11-13T03:26:44-07:00",
+ "status": 500,
+ "prompt_tokens": 0,
+ "cache_tokens": 0,
+ "output_tokens": 0,
+ "prefill_tps": 0,
+ "decode_tps": 0,
+ "duration_ms": 251,
+ "prefill_ms": 0
+ },
+ {
+ "id": "llama-swap/10947@1794565604",
+ "engine": "llama-swap",
+ "model": "Qwen3.6-35B-A3B-IQ4-Agent",
+ "t": "2026-11-13T03:26:44-07:00",
+ "status": 500,
+ "prompt_tokens": 0,
+ "cache_tokens": 0,
+ "output_tokens": 0,
+ "prefill_tps": 0,
+ "decode_tps": 0,
+ "duration_ms": 250,
+ "prefill_ms": 0
+ },
+ {
+ "id": "llama-swap/10948@1794565604",
+ "engine": "llama-swap",
+ "model": "Qwen3.6-35B-A3B-IQ4-Agent",
+ "t": "2026-11-13T03:26:44-07:00",
+ "status": 500,
+ "prompt_tokens": 0,
+ "cache_tokens": 0,
+ "output_tokens": 0,
+ "prefill_tps": 0,
+ "decode_tps": 0,
+ "duration_ms": 250,
+ "prefill_ms": 0
+ },
+ {
+ "id": "llama-swap/10941@1794565603",
+ "engine": "llama-swap",
+ "model": "Qwen3.6-35B-A3B-IQ4-Agent",
+ "t": "2026-11-13T03:26:43-07:00",
+ "status": 500,
+ "prompt_tokens": 0,
+ "cache_tokens": 0,
+ "output_tokens": 0,
+ "prefill_tps": 0,
+ "decode_tps": 0,
+ "duration_ms": 251,
+ "prefill_ms": 0
+ },
+ {
+ "id": "llama-swap/10942@1794565603",
+ "engine": "llama-swap",
+ "model": "Qwen3.6-35B-A3B-IQ4-Agent",
+ "t": "2026-11-13T03:26:43-07:00",
+ "status": 500,
+ "prompt_tokens": 0,
+ "cache_tokens": 0,
+ "output_tokens": 0,
+ "prefill_tps": 0,
+ "decode_tps": 0,
+ "duration_ms": 250,
+ "prefill_ms": 0
+ },
+ {
+ "id": "llama-swap/10943@1794565603",
+ "engine": "llama-swap",
+ "model": "Qwen3.6-35B-A3B-IQ4-Agent",
+ "t": "2026-11-13T03:26:43-07:00",
+ "status": 500,
+ "prompt_tokens": 0,
+ "cache_tokens": 0,
+ "output_tokens": 0,
+ "prefill_tps": 0,
+ "decode_tps": 0,
+ "duration_ms": 251,
+ "prefill_ms": 0
+ },
+ {
+ "id": "llama-swap/10944@1794565603",
+ "engine": "llama-swap",
+ "model": "Qwen3.6-35B-A3B-IQ4-Agent",
+ "t": "2026-11-13T03:26:43-07:00",
+ "status": 500,
+ "prompt_tokens": 0,
+ "cache_tokens": 0,
+ "output_tokens": 0,
+ "prefill_tps": 0,
+ "decode_tps": 0,
+ "duration_ms": 250,
+ "prefill_ms": 0
+ },
+ {
+ "id": "llama-swap/10939@1794565602",
+ "engine": "llama-swap",
+ "model": "Qwen3.6-35B-A3B-IQ4-Agent",
+ "t": "2026-11-13T03:26:42-07:00",
+ "status": 500,
+ "prompt_tokens": 0,
+ "cache_tokens": 0,
+ "output_tokens": 0,
+ "prefill_tps": 0,
+ "decode_tps": 0,
+ "duration_ms": 250,
+ "prefill_ms": 0
+ },
+ {
+ "id": "llama-swap/10940@1794565602",
+ "engine": "llama-swap",
+ "model": "Qwen3.6-35B-A3B-IQ4-Agent",
+ "t": "2026-11-13T03:26:42-07:00",
+ "status": 500,
+ "prompt_tokens": 0,
+ "cache_tokens": 0,
+ "output_tokens": 0,
+ "prefill_tps": 0,
+ "decode_tps": 0,
+ "duration_ms": 250,
+ "prefill_ms": 0
+ },
+ {
+ "id": "llama-swap/10937@1794565602",
+ "engine": "llama-swap",
+ "model": "Qwen3.6-35B-A3B-IQ4-Agent",
+ "t": "2026-11-13T03:26:42-07:00",
+ "status": 500,
+ "prompt_tokens": 0,
+ "cache_tokens": 0,
+ "output_tokens": 0,
+ "prefill_tps": 0,
+ "decode_tps": 0,
+ "duration_ms": 250,
+ "prefill_ms": 0
+ },
+ {
+ "id": "llama-swap/10938@1794565602",
+ "engine": "llama-swap",
+ "model": "Qwen3.6-35B-A3B-IQ4-Agent",
+ "t": "2026-11-13T03:26:42-07:00",
+ "status": 500,
+ "prompt_tokens": 0,
+ "cache_tokens": 0,
+ "output_tokens": 0,
+ "prefill_tps": 0,
+ "decode_tps": 0,
+ "duration_ms": 250,
+ "prefill_ms": 0
+ },
+ {
+ "id": "llama-swap/10933@1794565601",
+ "engine": "llama-swap",
+ "model": "Qwen3.6-35B-A3B-IQ4-Agent",
+ "t": "2026-11-13T03:26:41-07:00",
+ "status": 500,
+ "prompt_tokens": 0,
+ "cache_tokens": 0,
+ "output_tokens": 0,
+ "prefill_tps": 0,
+ "decode_tps": 0,
+ "duration_ms": 251,
+ "prefill_ms": 0
+ },
+ {
+ "id": "llama-swap/10934@1794565601",
+ "engine": "llama-swap",
+ "model": "Qwen3.6-35B-A3B-IQ4-Agent",
+ "t": "2026-11-13T03:26:41-07:00",
+ "status": 500,
+ "prompt_tokens": 0,
+ "cache_tokens": 0,
+ "output_tokens": 0,
+ "prefill_tps": 0,
+ "decode_tps": 0,
+ "duration_ms": 251,
+ "prefill_ms": 0
+ },
+ {
+ "id": "llama-swap/10935@1794565601",
+ "engine": "llama-swap",
+ "model": "Qwen3.6-35B-A3B-IQ4-Agent",
+ "t": "2026-11-13T03:26:41-07:00",
+ "status": 500,
+ "prompt_tokens": 0,
+ "cache_tokens": 0,
+ "output_tokens": 0,
+ "prefill_tps": 0,
+ "decode_tps": 0,
+ "duration_ms": 250,
+ "prefill_ms": 0
+ },
+ {
+ "id": "llama-swap/10936@1794565601",
+ "engine": "llama-swap",
+ "model": "Qwen3.6-35B-A3B-IQ4-Agent",
+ "t": "2026-11-13T03:26:41-07:00",
+ "status": 500,
+ "prompt_tokens": 0,
+ "cache_tokens": 0,
+ "output_tokens": 0,
+ "prefill_tps": 0,
+ "decode_tps": 0,
+ "duration_ms": 250,
+ "prefill_ms": 0
+ },
+ {
+ "id": "llama-swap/10929@1794565600",
+ "engine": "llama-swap",
+ "model": "Qwen3.6-35B-A3B-IQ4-Agent",
+ "t": "2026-11-13T03:26:40-07:00",
+ "status": 500,
+ "prompt_tokens": 0,
+ "cache_tokens": 0,
+ "output_tokens": 0,
+ "prefill_tps": 0,
+ "decode_tps": 0,
+ "duration_ms": 250,
+ "prefill_ms": 0
+ },
+ {
+ "id": "llama-swap/10930@1794565600",
+ "engine": "llama-swap",
+ "model": "Qwen3.6-35B-A3B-IQ4-Agent",
+ "t": "2026-11-13T03:26:40-07:00",
+ "status": 500,
+ "prompt_tokens": 0,
+ "cache_tokens": 0,
+ "output_tokens": 0,
+ "prefill_tps": 0,
+ "decode_tps": 0,
+ "duration_ms": 251,
+ "prefill_ms": 0
+ },
+ {
+ "id": "llama-swap/10931@1794565600",
+ "engine": "llama-swap",
+ "model": "Qwen3.6-35B-A3B-IQ4-Agent",
+ "t": "2026-11-13T03:26:40-07:00",
+ "status": 500,
+ "prompt_tokens": 0,
+ "cache_tokens": 0,
+ "output_tokens": 0,
+ "prefill_tps": 0,
+ "decode_tps": 0,
+ "duration_ms": 251,
+ "prefill_ms": 0
+ },
+ {
+ "id": "llama-swap/10932@1794565600",
+ "engine": "llama-swap",
+ "model": "Qwen3.6-35B-A3B-IQ4-Agent",
+ "t": "2026-11-13T03:26:40-07:00",
+ "status": 500,
+ "prompt_tokens": 0,
+ "cache_tokens": 0,
+ "output_tokens": 0,
+ "prefill_tps": 0,
+ "decode_tps": 0,
+ "duration_ms": 250,
+ "prefill_ms": 0
+ },
+ {
+ "id": "llama-swap/10925@1794565599",
+ "engine": "llama-swap",
+ "model": "Qwen3.6-35B-A3B-IQ4-Agent",
+ "t": "2026-11-13T03:26:39-07:00",
+ "status": 500,
+ "prompt_tokens": 0,
+ "cache_tokens": 0,
+ "output_tokens": 0,
+ "prefill_tps": 0,
+ "decode_tps": 0,
+ "duration_ms": 251,
+ "prefill_ms": 0
+ },
+ {
+ "id": "llama-swap/10926@1794565599",
+ "engine": "llama-swap",
+ "model": "Qwen3.6-35B-A3B-IQ4-Agent",
+ "t": "2026-11-13T03:26:39-07:00",
+ "status": 500,
+ "prompt_tokens": 0,
+ "cache_tokens": 0,
+ "output_tokens": 0,
+ "prefill_tps": 0,
+ "decode_tps": 0,
+ "duration_ms": 251,
+ "prefill_ms": 0
+ }
+ ],
+ "since": "2026-11-13T00:58:47.64393093-07:00"
+ },
+ "counters": {
+ "collisions_avoided": 1399,
+ "vram_handed_off": 32646610550784,
+ "drains": 903,
+ "stuck_caught": 4,
+ "squatters_caught": 573,
+ "contested_grants": 198,
+ "refused": 9788,
+ "juggles_stopped": 57,
+ "send_timeouts": 0,
+ "dropped_consumers": 0,
+ "since": "2026-10-12T06:45:03.131986561-06:00"
+ },
+ "last_event": 10780,
+ "alerts": {
+ "table": {
+ "probe": {
+ "warn": 1,
+ "crit": 1
+ },
+ "stuck": {
+ "warn": 1,
+ "crit": 1
+ }
+ }
+ },
+ "priority": {
+ "source": "order",
+ "order": [
+ "llama-swap",
+ "strata",
+ "strata-sc117",
+ "comfyui",
+ "acestep"
+ ],
+ "juggle_window": 15,
+ "engines": [
+ {
+ "engine": "llama-swap",
+ "rank": 1,
+ "hours_7d": 41.22
+ },
+ {
+ "engine": "strata",
+ "rank": 2,
+ "hours_7d": 8.69
+ },
+ {
+ "engine": "strata-sc117",
+ "rank": 3,
+ "hours_7d": 4.03
+ },
+ {
+ "engine": "comfyui",
+ "rank": 4,
+ "hours_7d": 73.27
+ },
+ {
+ "engine": "acestep",
+ "rank": 5,
+ "hours_7d": 0.12
+ },
+ {
+ "engine": "training",
+ "rank": 6,
+ "hours_7d": 6.59
+ },
+ {
+ "engine": "hf-eval",
+ "rank": 7,
+ "hours_7d": 6.08
+ },
+ {
+ "engine": "heretic",
+ "rank": 8,
+ "hours_7d": 4.85
+ },
+ {
+ "engine": "mem-heal",
+ "rank": 9,
+ "hours_7d": 1.89
+ },
+ {
+ "engine": "mem-smf",
+ "rank": 10,
+ "hours_7d": 1.76
+ },
+ {
+ "engine": "tool-sft",
+ "rank": 11,
+ "hours_7d": 1.37
+ },
+ {
+ "engine": "lora-base",
+ "rank": 12,
+ "hours_7d": 0.72
+ },
+ {
+ "engine": "tool-grpo",
+ "rank": 13,
+ "hours_7d": 0.51
+ },
+ {
+ "engine": "bench-27b",
+ "rank": 14,
+ "hours_7d": 0.2
+ },
+ {
+ "engine": "heretic-export",
+ "rank": 15,
+ "hours_7d": 0.2
+ },
+ {
+ "engine": "smoke-27b",
+ "rank": 16,
+ "hours_7d": 0.09
+ },
+ {
+ "engine": "studio-finish",
+ "rank": 17,
+ "hours_7d": 0.09
+ },
+ {
+ "engine": "bench-4b",
+ "rank": 18,
+ "hours_7d": 0.08
+ },
+ {
+ "engine": "idol",
+ "rank": 19,
+ "hours_7d": 0.07
+ },
+ {
+ "engine": "studio-flashvsr",
+ "rank": 20,
+ "hours_7d": 0.04
+ },
+ {
+ "engine": "mem-smoke",
+ "rank": 21,
+ "hours_7d": 0.02
+ },
+ {
+ "engine": "claim",
+ "rank": 22,
+ "hours_7d": 0.01
+ },
+ {
+ "engine": "studio-nr",
+ "rank": 23,
+ "hours_7d": 0.01
+ },
+ {
+ "engine": "studio-diag",
+ "rank": 24,
+ "hours_7d": 0
+ }
+ ]
+ },
+ "claims": [
+ {
+ "name": "steam-game",
+ "action": "hold",
+ "active": false
+ },
+ {
+ "name": "lutris-game",
+ "action": "hold",
+ "builtin": true,
+ "active": false
+ },
+ {
+ "name": "gamescope",
+ "action": "hold",
+ "builtin": true,
+ "active": false
+ },
+ {
+ "name": "steam-library",
+ "action": "hold",
+ "builtin": true,
+ "active": false
+ },
+ {
+ "name": "epic-game",
+ "action": "hold",
+ "builtin": true,
+ "active": false
+ },
+ {
+ "name": "gog-game",
+ "action": "hold",
+ "builtin": true,
+ "active": false
+ },
+ {
+ "name": "ea-game",
+ "action": "hold",
+ "builtin": true,
+ "active": false
+ },
+ {
+ "name": "ubisoft-game",
+ "action": "hold",
+ "builtin": true,
+ "active": false
+ },
+ {
+ "name": "xbox-game",
+ "action": "hold",
+ "builtin": true,
+ "active": false
+ },
+ {
+ "name": "battlenet-game",
+ "action": "hold",
+ "builtin": true,
+ "active": false
+ },
+ {
+ "name": "blender",
+ "action": "track",
+ "builtin": true,
+ "active": false
+ },
+ {
+ "name": "davinci-resolve",
+ "action": "track",
+ "builtin": true,
+ "active": false
+ },
+ {
+ "name": "obs",
+ "action": "track",
+ "builtin": true,
+ "active": false
+ },
+ {
+ "name": "premiere",
+ "action": "track",
+ "builtin": true,
+ "active": false
+ },
+ {
+ "name": "after-effects",
+ "action": "track",
+ "builtin": true,
+ "active": false
+ },
+ {
+ "name": "kdenlive",
+ "action": "track",
+ "builtin": true,
+ "active": false
+ }
+ ],
+ "footprints": [
+ {
+ "engine": "acestep",
+ "estimated_vram_bytes": 22559064064,
+ "runs": 13,
+ "at": "2026-11-07T12:05:22.120250212-07:00"
+ },
+ {
+ "engine": "comfyui",
+ "estimated_vram_bytes": 478150656,
+ "runs": 146,
+ "at": "2026-11-12T09:52:22.309629621-07:00"
+ },
+ {
+ "engine": "llama-swap",
+ "estimated_vram_bytes": 24882708480,
+ "runs": 688,
+ "at": "2026-11-13T02:45:57.656694691-07:00"
+ },
+ {
+ "engine": "strata",
+ "estimated_vram_bytes": 24823988224,
+ "runs": 96,
+ "at": "2026-11-12T23:46:19.489608747-07:00"
+ },
+ {
+ "engine": "strata-sc117",
+ "estimated_vram_bytes": 24918360064,
+ "runs": 70,
+ "at": "2026-11-09T16:24:01.584090665-07:00"
+ }
+ ]
+}
diff --git a/internal/gateway/background.go b/internal/gateway/background.go
index d6da564c..141f9117 100644
--- a/internal/gateway/background.go
+++ b/internal/gateway/background.go
@@ -12,32 +12,17 @@ package gateway
import (
"context"
- "slices"
"time"
"github.com/T-Crypt/berth/internal/config"
+ "github.com/T-Crypt/berth/internal/fit"
"github.com/T-Crypt/berth/internal/node"
- "github.com/T-Crypt/berth/internal/proto"
)
// DefaultIdleAfter is how long the card must be free before a background
// engine starts, when its entry names no idle_after.
const DefaultIdleAfter = 60 * time.Second
-// cardFree is whether nothing holds, uses or waits for the card and the
-// card serves this machine. A parked lease holds it: a batch job must not
-// evict an engine someone parked in RAM to come back to. A stowaway or a
-// tracked app (Blender) holds memory no lease accounts for, and a start
-// beside it is the collision Berth exists to prevent.
-func cardFree(st *proto.Status) bool {
- if st == nil || st.Passthrough != nil || st.ProbeErr != "" ||
- len(st.Leases) > 0 || len(st.Waiting) > 0 || len(st.Squatters) > 0 || cordoned(st) {
- return false
- }
- // Status lists every claim rule; only one matching right now is in use.
- return !slices.ContainsFunc(st.Claims, func(c proto.Claim) bool { return c.Active })
-}
-
// Background starts the background engines while ctx lives, checking the
// card every `every` (0 = 5 s): queued ComfyUI jobs as soon as the card is
// free, then one background engine at a time, the first in berth.toml whose
@@ -69,7 +54,7 @@ func (g *Gateway) Background(ctx context.Context, every time.Duration) {
st := g.nodeStatus()
// Work someone asked for while the card was out of reach goes
// first, as soon as the card is free, before any background engine.
- if q := g.queueingComfy(); q != nil && cardFree(node.View(st, q.GPU)) && g.submitting.CompareAndSwap(false, true) {
+ if q := g.queueingComfy(); q != nil && fit.CardFree(node.View(st, q.GPU)) && g.submitting.CompareAndSwap(false, true) {
go func() {
defer g.submitting.Store(false)
g.submitDeferred(ctx)
@@ -85,7 +70,7 @@ func (g *Gateway) Background(ctx context.Context, every time.Duration) {
if !e.Window.Contains(now) {
continue
}
- if !cardFree(node.View(st, e.GPU)) {
+ if !fit.CardFree(node.View(st, e.GPU)) {
delete(freeSince, e.GPU)
continue
}
diff --git a/internal/gateway/background_test.go b/internal/gateway/background_test.go
index 06ceaefb..ac44a578 100644
--- a/internal/gateway/background_test.go
+++ b/internal/gateway/background_test.go
@@ -134,27 +134,3 @@ func windowFrom(offset, span int) proto.Window {
}
const minutesDay = 24 * 60
-
-// The card is free only when nothing holds, uses or waits for it and it
-// serves this machine.
-func TestCardFree(t *testing.T) {
- for _, tc := range []struct {
- name string
- st *proto.Status
- want bool
- }{
- {"no reading", nil, false},
- {"empty card", &proto.Status{}, true},
- {"a lease", &proto.Status{Leases: []proto.Lease{{Engine: "chat"}}}, false},
- {"a waiter", &proto.Status{Waiting: []proto.Lease{{Engine: "chat"}}}, false},
- {"a stowaway", &proto.Status{Squatters: []proto.Squatter{{PID: 7}}}, false},
- {"claim rules, none matching", &proto.Status{Claims: []proto.Claim{{Name: "steam-game"}, {Name: "blender"}}}, true},
- {"a tracked app running", &proto.Status{Claims: []proto.Claim{{Name: "steam-game"}, {Name: "blender", Active: true}}}, false},
- {"in a VM", &proto.Status{Passthrough: &proto.Passthrough{State: proto.CardPassthrough}}, false},
- {"probe failing", &proto.Status{ProbeErr: "nvidia-smi: exit 9"}, false},
- } {
- if got := cardFree(tc.st); got != tc.want {
- t.Errorf("%s: cardFree = %v, want %v", tc.name, got, tc.want)
- }
- }
-}
diff --git a/internal/gateway/board.go b/internal/gateway/board.go
index 95da52a6..9aa3cd04 100644
--- a/internal/gateway/board.go
+++ b/internal/gateway/board.go
@@ -135,7 +135,7 @@ func boardOf(host string, status *proto.Status, rows []guiRow, listed map[string
c.Base = "/cards/" + gp.Card + "/v1"
}
for _, l := range node.View(status, gp.Card).Leases {
- c.Holders = append(c.Holders, guiHold{Name: holderName(l), State: l.State, Pinned: l.Pinned})
+ c.Holders = append(c.Holders, guiHold{Name: l.Holder(), State: l.State, Pinned: l.Pinned})
if c.State == "free" {
c.State = "held"
}
diff --git a/internal/gateway/cordon_test.go b/internal/gateway/cordon_test.go
index 047c6c0c..98d4881d 100644
--- a/internal/gateway/cordon_test.go
+++ b/internal/gateway/cordon_test.go
@@ -14,6 +14,7 @@ import (
"time"
"github.com/T-Crypt/berth/internal/config"
+ "github.com/T-Crypt/berth/internal/fit"
"github.com/T-Crypt/berth/internal/node"
"github.com/T-Crypt/berth/internal/proto"
)
@@ -30,11 +31,11 @@ func TestCordonedCardIsNotFree(t *testing.T) {
st = n.Status()
return node.View(&st, "gpu0") != nil && node.View(&st, "gpu1") != nil
})
- if why := outOfReach(nodeView(t, &st, "gpu1")); why != "the card is cordoned" {
- t.Errorf("outOfReach(gpu1) = %q, want the cordon", why)
+ if why := fit.OutOfReach(nodeView(t, &st, "gpu1")); why != "the card is cordoned" {
+ t.Errorf("OutOfReach(gpu1) = %q, want the cordon", why)
}
- if why := outOfReach(nodeView(t, &st, "gpu0")); why != "" {
- t.Errorf("outOfReach(gpu0) = %q, want nothing", why)
+ if why := fit.OutOfReach(nodeView(t, &st, "gpu0")); why != "" {
+ t.Errorf("OutOfReach(gpu0) = %q, want nothing", why)
}
addr := freeAddr(t)
diff --git a/internal/gateway/cost.go b/internal/gateway/cost.go
index 4082c2fd..efbbaea7 100644
--- a/internal/gateway/cost.go
+++ b/internal/gateway/cost.go
@@ -17,6 +17,7 @@ import (
"time"
"github.com/T-Crypt/berth/internal/config"
+ "github.com/T-Crypt/berth/internal/fit"
"github.com/T-Crypt/berth/internal/pagecache"
"github.com/T-Crypt/berth/internal/swapcost"
)
@@ -51,7 +52,7 @@ type position struct {
// checked, leaves the share 0: a part of the answer would be a wrong
// fraction, and cold is the honest reading of not knowing.
func (g *Gateway) readPosition(e *config.Engine) *position {
- pos := &position{parked: parkedIn(g.nodeStatus(), e.Name)}
+ pos := &position{parked: fit.Parked(g.nodeStatus(), e.Name)}
files, _ := prefetchFiles(e.Prefetch)
if len(files) == 0 {
return pos
diff --git a/internal/gateway/deferred.go b/internal/gateway/deferred.go
index fb31baec..8b0ba526 100644
--- a/internal/gateway/deferred.go
+++ b/internal/gateway/deferred.go
@@ -19,14 +19,12 @@ import (
"io"
"net/http"
"os"
- "slices"
"strconv"
"strings"
"time"
"github.com/T-Crypt/berth/internal/config"
"github.com/T-Crypt/berth/internal/fsx"
- "github.com/T-Crypt/berth/internal/proto"
)
const deferredPrefix = "berth-"
@@ -51,26 +49,6 @@ type deferredBook struct {
// /history; older ones drop off.
const keepSubmitted = 200
-// outOfReach says why a ComfyUI job cannot run now, "" when it can: the
-// card is in a VM, or a pinned lease (a game, a hold, a pinned app) has it.
-func outOfReach(st *proto.Status) string {
- if st == nil {
- return ""
- }
- if st.Passthrough != nil {
- return "the card is passed through to a VM"
- }
- if cordoned(st) {
- return "the card is cordoned"
- }
- for _, l := range st.Leases {
- if l.Pinned {
- return fmt.Sprintf("%s holds the card", holderName(l))
- }
- }
- return ""
-}
-
// queueingComfy is the ComfyUI engine whose jobs queue while the card is
// out of reach, nil when none does.
func (g *Gateway) queueingComfy() *config.Engine {
@@ -266,8 +244,3 @@ func (g *Gateway) deferredHistory(w http.ResponseWriter, r *http.Request, id str
}
_ = json.NewEncoder(w).Encode(out)
}
-
-// cordoned is whether berth cordon closed a card the status covers.
-func cordoned(st *proto.Status) bool {
- return slices.ContainsFunc(st.GPUs, func(g proto.GPU) bool { return g.Scheduled && g.Cordoned })
-}
diff --git a/internal/gateway/deferred_test.go b/internal/gateway/deferred_test.go
index 477d9991..088d7fb7 100644
--- a/internal/gateway/deferred_test.go
+++ b/internal/gateway/deferred_test.go
@@ -20,7 +20,6 @@ import (
"github.com/T-Crypt/berth/internal/broker"
"github.com/T-Crypt/berth/internal/config"
- "github.com/T-Crypt/berth/internal/proto"
)
// A render asked for while a game holds the card is queued, answered at
@@ -82,23 +81,3 @@ func TestComfyJobQueuedWhileTheCardIsHeldRunsWhenItIsFree(t *testing.T) {
return strings.Contains(h, `"`+got.PromptID+`"`) && strings.Contains(h, "history-marker")
})
}
-
-// A ComfyUI job is out of reach while the card is in a VM or a pinned
-// lease has it; an ordinary holder is drained the usual way.
-func TestOutOfReach(t *testing.T) {
- for _, tc := range []struct {
- name string
- st *proto.Status
- want bool
- }{
- {"no reading", nil, false},
- {"free card", &proto.Status{}, false},
- {"an ordinary holder", &proto.Status{Leases: []proto.Lease{{Engine: "llama-swap"}}}, false},
- {"a game", &proto.Status{Leases: []proto.Lease{{Engine: "claim:steam-game", Pinned: true}}}, true},
- {"in a VM", &proto.Status{Passthrough: &proto.Passthrough{State: proto.CardPassthrough}}, true},
- } {
- if got := outOfReach(tc.st) != ""; got != tc.want {
- t.Errorf("%s: out of reach = %v, want %v", tc.name, got, tc.want)
- }
- }
-}
diff --git a/internal/gateway/gateway.go b/internal/gateway/gateway.go
index 403eee65..d187d247 100644
--- a/internal/gateway/gateway.go
+++ b/internal/gateway/gateway.go
@@ -37,6 +37,7 @@ import (
"github.com/T-Crypt/berth/internal/apikey"
"github.com/T-Crypt/berth/internal/config"
+ "github.com/T-Crypt/berth/internal/fit"
"github.com/T-Crypt/berth/internal/host"
"github.com/T-Crypt/berth/internal/node"
"github.com/T-Crypt/berth/internal/proto"
@@ -231,7 +232,12 @@ func (g *Gateway) Handler() http.Handler {
mux.HandleFunc("POST /v1/berth/switch", g.apiSwitch)
mux.HandleFunc("POST /v1/berth/stop", g.apiStop)
mux.HandleFunc("POST /v1/berth/batch", g.apiBatch)
+ mux.HandleFunc("POST /v1/berth/yield", g.apiYield)
mux.HandleFunc("POST /v1/berth/load", g.apiLoad)
+ mux.HandleFunc("GET /v1/berth/schedule", g.scheduleRead)
+ mux.HandleFunc("POST /v1/berth/schedule", g.scheduleBook)
+ mux.HandleFunc("PATCH /v1/berth/schedule/{id}", g.scheduleMove)
+ mux.HandleFunc("DELETE /v1/berth/schedule/{id}", g.scheduleCancel)
for _, p := range []string{"/v1/chat/completions", "/v1/completions", "/v1/embeddings"} {
mux.HandleFunc("POST "+p, g.forward)
mux.HandleFunc("POST /cards/{card}"+p, g.onCard(g.forward))
@@ -369,11 +375,27 @@ func (g *Gateway) listed(ctx context.Context, base string) []string {
return c.models
}
g.mu.Unlock()
+ out := ListModels(ctx, g.client(), base)
+ if out == nil {
+ return nil
+ }
+ g.mu.Lock()
+ if g.lists == nil {
+ g.lists = map[string]cached{}
+ }
+ g.lists[base] = cached{time.Now(), out}
+ g.mu.Unlock()
+ return out
+}
+
+// ListModels is the model names an engine's /v1/models lists, nil when it
+// does not answer with a list. A GET there loads nothing.
+func ListModels(ctx context.Context, c *http.Client, base string) []string {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, strings.TrimRight(base, "/")+"/v1/models", nil)
if err != nil {
return nil
}
- resp, err := g.client().Do(req)
+ resp, err := c.Do(req)
if err != nil {
return nil
}
@@ -390,12 +412,6 @@ func (g *Gateway) listed(ctx context.Context, base string) []string {
for _, d := range r.Data {
out = append(out, d.ID)
}
- g.mu.Lock()
- if g.lists == nil {
- g.lists = map[string]cached{}
- }
- g.lists[base] = cached{time.Now(), out}
- g.mu.Unlock()
return out
}
@@ -447,7 +463,7 @@ func (g *Gateway) resolve(ctx context.Context, model, card string) (target, bool
for _, e := range g.live() {
// A parked engine answers its list with the weights in RAM; the
// resume, not the list, says when it may be proxied.
- if e.Up && g.engineOn(st, e.Name, card) && !parkedIn(st, e.Name) && slices.Contains(g.listed(ctx, e.URL), model) {
+ if e.Up && g.engineOn(st, e.Name, card) && !fit.Parked(st, e.Name) && slices.Contains(g.listed(ctx, e.URL), model) {
return target{url: e.URL, live: true}, true
}
}
@@ -460,29 +476,12 @@ func (g *Gateway) resolve(ctx context.Context, model, card string) (target, bool
if len(serving) == 0 {
return target{}, false
}
- // One engine per card serves the model: the cheapest card to reach
+ // One engine per card serves the model: the cheapest path to a card
// wins, and berth.toml's order breaks a tie.
- e := slices.MinFunc(serving, func(a, b *config.Engine) int { return reach(st, a) - reach(st, b) })
+ e := fit.Rank(st, serving, nil)[0].Entry
return target{url: e.URL, engine: e}, true
}
-// reach ranks what it takes to bring e up on its card, cheapest first: a
-// resume from RAM, a free card, a drain of the holder, then a wait for a
-// card a VM or a pinned lease (a game, a hold) keeps.
-func reach(st *proto.Status, e *config.Engine) int {
- v := node.View(st, e.GPU)
- switch {
- case parkedIn(st, e.Name):
- return 0
- case cardFree(v):
- return 1
- case v != nil && outOfReach(v) == "":
- return 2
- default:
- return 3
- }
-}
-
// notServed is the 404 for a model no engine serves, on card when one is
// named.
func notServed(w http.ResponseWriter, model, card string) {
@@ -585,6 +584,11 @@ type startError struct {
code string // no_start, outranked, drained_while_starting, no_berth, start_failed, engine_oom, not_ready, no_headroom
msg string
retry time.Duration // when set, the client is told to try again after this (Retry-After)
+ // queue is the engine's lease's place in line when the start timed out
+ // waiting for the card, 0 when it was not waiting; estimate is the
+ // broker's lower-bound start estimate for it, 0 for unknown.
+ queue int
+ estimate time.Duration
}
func (e *startError) Error() string { return e.msg }
@@ -796,6 +800,12 @@ func (g *Gateway) startFailed(w http.ResponseWriter, engine, what string, err er
if se != nil && se.retry > 0 {
w.Header().Set("Retry-After", strconv.Itoa(int((se.retry+time.Second-1)/time.Second)))
}
+ if se != nil && se.queue > 0 {
+ w.Header().Set("X-Berth-Queue-Position", strconv.Itoa(se.queue))
+ if se.estimate > 0 {
+ w.Header().Set("X-Berth-Start-Estimate", fmt.Sprintf("%d; lower-bound", int((se.estimate+time.Second-1)/time.Second)))
+ }
+ }
apiErrorCode(w, http.StatusServiceUnavailable, code, err.Error())
}
@@ -805,7 +815,7 @@ func (g *Gateway) startFailed(w http.ResponseWriter, engine, what string, err er
func (g *Gateway) comfyTarget(ctx context.Context) (target, bool) {
st := g.nodeStatus()
for _, e := range g.live() {
- if e.Up && e.Type == "comfyui" && !parkedIn(st, e.Name) && g.comfyAnswers(ctx, e.URL) {
+ if e.Up && e.Type == "comfyui" && !fit.Parked(st, e.Name) && g.comfyAnswers(ctx, e.URL) {
return target{url: e.URL, live: true}, true
}
}
@@ -867,7 +877,7 @@ func (g *Gateway) comfyPrompt(w http.ResponseWriter, r *http.Request, switched b
return
}
if e := g.queueingComfy(); e != nil {
- if why := outOfReach(node.View(g.nodeStatus(), e.GPU)); why != "" {
+ if why := fit.OutOfReach(node.View(g.nodeStatus(), e.GPU)); why != "" {
g.deferPrompt(w, r, e, why)
return
}
@@ -1016,11 +1026,32 @@ func (g *Gateway) ensure(ctx context.Context, e *config.Engine, switched bool) e
case <-time.After(250 * time.Millisecond):
}
if time.Now().After(deadline) {
- return &startError{code: "not_ready", msg: fmt.Sprintf("%s did not answer within %s", e.Name, ready)}
+ return g.queued(e, &startError{code: "not_ready", msg: fmt.Sprintf("%s did not answer within %s", e.Name, ready)})
}
}
}
+// queued adds where the engine's lease waits to a start that timed out: its
+// place in line and the broker's start estimate. The estimate counts only the
+// measured starts ahead of it, never the holder's time left, so it is a lower
+// bound, and the client is told to retry no sooner than it or 30 s.
+func (g *Gateway) queued(e *config.Engine, se *startError) *startError {
+ v := node.View(g.nodeStatus(), e.GPU)
+ if v == nil {
+ return se
+ }
+ for _, l := range v.Waiting {
+ if l.Engine == e.Name && l.Position > 0 && (se.queue == 0 || l.Position < se.queue) {
+ se.queue, se.estimate = l.Position, time.Duration(l.EstimateS*float64(time.Second))
+ }
+ }
+ if se.queue > 0 {
+ se.retry = max(30*time.Second, se.estimate)
+ se.msg += fmt.Sprintf("; its lease is number %d in line for the card", se.queue)
+ }
+ return se
+}
+
func (g *Gateway) ready() time.Duration {
if g.Ready > 0 {
return g.Ready
diff --git a/internal/gateway/gateway_test.go b/internal/gateway/gateway_test.go
index 00f4a06c..e9e2dd03 100644
--- a/internal/gateway/gateway_test.go
+++ b/internal/gateway/gateway_test.go
@@ -31,6 +31,7 @@ import (
"github.com/T-Crypt/berth/internal/gpu"
"github.com/T-Crypt/berth/internal/node"
"github.com/T-Crypt/berth/internal/procs"
+ "github.com/T-Crypt/berth/internal/proto"
"github.com/T-Crypt/berth/internal/runner"
"github.com/T-Crypt/berth/internal/server"
)
@@ -276,6 +277,12 @@ func daemon(t *testing.T) (string, *broker.Broker) {
// daemonWith is daemon with cfg's fields on top of the test defaults.
func daemonWith(t *testing.T, cfg broker.Config) (string, *broker.Broker) {
+ t.Helper()
+ return daemonServing(t, cfg, server.Extras{})
+}
+
+// daemonServing is daemonWith serving x as well (a schedule, say).
+func daemonServing(t *testing.T, cfg broker.Config, x server.Extras) (string, *broker.Broker) {
t.Helper()
sock := filepath.Join(t.TempDir(), "berth.sock")
log := slog.New(slog.NewTextHandler(io.Discard, nil))
@@ -288,7 +295,7 @@ func daemonWith(t *testing.T, cfg broker.Config) (string, *broker.Broker) {
ctx, cancel := context.WithCancel(context.Background())
t.Cleanup(cancel)
go b.Run(ctx)
- go func() { _ = server.Serve(ctx, ln, node.One(b), server.Extras{}, log) }()
+ go func() { _ = server.Serve(ctx, ln, node.One(b), x, log) }()
// Cleanups run last first, so this runs before the cancel above. An
// engine a test's gateway started may still be up; stopping the daemon
// under it makes its berth run write an unprotected-run record beside
@@ -1008,3 +1015,58 @@ func TestComfyPromptResolves(t *testing.T) {
t.Fatalf("queue with comfyui down = %d %s, want the 503", resp.StatusCode, body)
}
}
+
+// A start that times out while its lease still waits for the card tells the
+// client where it stands: its place in line, and a Retry-After of at least
+// 30 s. Here a pinned hold keeps the card, so the gateway's run waits.
+func TestNotReadyCarriesTheQueue(t *testing.T) {
+ sock, b := daemon(t)
+ hold, err := b.Acquire(context.Background(), broker.Request{Engine: "game", Priority: broker.Hold, Pinned: true, Wait: time.Second}, nil)
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer hold.Release()
+ g := &Gateway{Socket: sock, Key: testKey, Ready: 500 * time.Millisecond, Log: slogDiscard(),
+ Engines: []config.Engine{{Name: "e", URL: "http://" + freeAddr(t), Start: []string{os.Args[0]}, Serves: []string{"m"}}}}
+ gw := httptest.NewServer(g.Handler())
+ defer gw.Close()
+
+ resp := call(t, "POST", gw.URL+"/v1/chat/completions", `{"model":"m"}`)
+ body, _ := io.ReadAll(resp.Body)
+ resp.Body.Close()
+ if resp.StatusCode != http.StatusServiceUnavailable || resp.Header.Get("X-Berth-Queue-Position") != "1" ||
+ resp.Header.Get("Retry-After") != "30" || !strings.Contains(string(body), "number 1 in line") {
+ t.Fatalf("status %d, queue %q, Retry-After %q: %s", resp.StatusCode,
+ resp.Header.Get("X-Berth-Queue-Position"), resp.Header.Get("Retry-After"), body)
+ }
+}
+
+// The broker's start estimate for the engine's lease reaches the client
+// rounded up and marked as a lower bound, and Retry-After waits for the
+// larger of it and 30 s. Another engine's place in line is not this one's.
+func TestQueuedStartHeaders(t *testing.T) {
+ for _, tc := range []struct {
+ estimate float64
+ retry, queue, start string
+ }{
+ {44.2, "45", "2", "45; lower-bound"},
+ {5, "30", "2", "5; lower-bound"},
+ {0, "30", "2", ""},
+ } {
+ sock := statusDaemon(t, proto.Status{
+ GPUs: []proto.GPU{{Card: "gpu0", Scheduled: true}},
+ Waiting: []proto.Lease{
+ {Card: "gpu0", Engine: "other", Position: 1, EstimateS: 900},
+ {Card: "gpu0", Engine: "e", Position: 2, EstimateS: tc.estimate},
+ },
+ })
+ g := &Gateway{Socket: sock, Log: slogDiscard()}
+ w := httptest.NewRecorder()
+ g.startFailed(w, "e", "m", g.queued(&config.Engine{Name: "e"}, &startError{code: "not_ready", msg: "e did not answer"}))
+ h := w.Result().Header
+ got := [...]string{h.Get("Retry-After"), h.Get("X-Berth-Queue-Position"), h.Get("X-Berth-Start-Estimate")}
+ if got != [...]string{tc.retry, tc.queue, tc.start} {
+ t.Errorf("estimate %v s: Retry-After, queue, start = %q", tc.estimate, got)
+ }
+ }
+}
diff --git a/internal/gateway/gui.go b/internal/gateway/gui.go
index cbfe59f0..92462079 100644
--- a/internal/gateway/gui.go
+++ b/internal/gateway/gui.go
@@ -300,7 +300,7 @@ func cardView(engine string, node *proto.Status, wk *wake) guiStatus {
ours = true
continue
}
- st.Card = append(st.Card, guiHold{Name: holderName(l), State: l.State, Pinned: l.Pinned})
+ st.Card = append(st.Card, guiHold{Name: l.Holder(), State: l.State, Pinned: l.Pinned})
}
}
switch {
@@ -352,14 +352,6 @@ func waitLine(card []guiHold) string {
return "Waiting for " + h.Name + " release"
}
-// holderName names a lease for people: llama-swap's models by their family.
-func holderName(l proto.Lease) string {
- if l.Group != "" && l.Group != l.Engine {
- return l.Group + " (" + l.Engine + ")"
- }
- return l.Engine
-}
-
func names(card []guiHold) string {
out := make([]string, len(card))
for i, h := range card {
diff --git a/internal/gateway/resume.go b/internal/gateway/resume.go
index 6c18c8e6..3b40aa8f 100644
--- a/internal/gateway/resume.go
+++ b/internal/gateway/resume.go
@@ -90,24 +90,6 @@ func (g *Gateway) waitForWake(ctx context.Context, engine string, since uint64)
}
}
-// parkedIn reports whether the node's lease table shows the engine
-// parked: a live answer from a parked engine is not one the gateway may
-// proxy. Only the engine's own lease counts. A parked model behind a router
-// (a llama-swap model under the llama-swap group) has its own front, which
-// wakes it when the router forwards a request, so the router is never
-// parked on its behalf.
-func parkedIn(st *proto.Status, name string) bool {
- if st == nil {
- return false
- }
- for _, l := range st.Leases {
- if l.Engine == name && l.State == "parked" {
- return true
- }
- }
- return false
-}
-
// switchQueued tells berthd a person switched to e while its start waits in
// the queue. Nothing queued is no error: the start may be granted already.
func (g *Gateway) switchQueued(e *config.Engine) {
diff --git a/internal/gateway/resume_test.go b/internal/gateway/resume_test.go
index 8ce37e0e..0698106f 100644
--- a/internal/gateway/resume_test.go
+++ b/internal/gateway/resume_test.go
@@ -21,6 +21,7 @@ import (
"strings"
"sync"
"sync/atomic"
+ "syscall"
"testing"
"time"
@@ -155,12 +156,12 @@ func TestGatewayResumesAParkedEngine(t *testing.T) {
t.Fatal("the vllm adapter is missing")
}
engineLog := &syncBuf{}
- go runner.Run(runner.Options{Socket: sock, Engine: "vllm", Priority: "interactive", Needs: []uint64{8 << 30},
+ var pid int
+ runUntilEnd(t, &pid, runner.Options{Socket: sock, Engine: "vllm", Priority: "interactive", Needs: []uint64{8 << 30},
Park: &adapter, ParkBase: srv.URL, ParkProbe: "/v1/models",
Grace: 200 * time.Millisecond, Argv: []string{"sleep", "30"}, Stderr: engineLog})
// Measure, then park it behind a foreground holder that stays.
- var pid int
deadline := time.Now().Add(5 * time.Second)
for pid == 0 {
if time.Now().After(deadline) {
@@ -267,6 +268,25 @@ func TestGatewayResumesAParkedEngine(t *testing.T) {
}
}
+// runUntilEnd starts a berth run on its own goroutine. At cleanup it kills
+// the engine's process group (*pid, once the test has read it) and waits for
+// Run to return: a Run left going sees its daemon close at cleanup and
+// writes its unprotected-run record into a temp dir that is being removed.
+func runUntilEnd(t *testing.T, pid *int, o runner.Options) {
+ t.Helper()
+ exited := make(chan int, 1)
+ go func() { exited <- runner.Run(o) }()
+ t.Cleanup(func() {
+ if *pid > 0 {
+ _ = syscall.Kill(-*pid, syscall.SIGKILL)
+ }
+ select {
+ case <-exited:
+ case <-time.After(5 * time.Second):
+ }
+ })
+}
+
func slogDiscard() *slog.Logger { return slog.New(slog.NewTextHandler(io.Discard, nil)) }
// statusDaemon is a berthd that answers every status with st.
diff --git a/internal/gateway/schedule.go b/internal/gateway/schedule.go
new file mode 100644
index 00000000..5d02a9a8
--- /dev/null
+++ b/internal/gateway/schedule.go
@@ -0,0 +1,171 @@
+// Copyright 2026 Trevin Tindall (T-Crypt)
+// SPDX-License-Identifier: Apache-2.0
+
+package gateway
+
+// The scheduler over HTTP, for agents and devices that hold the node's key.
+// GET /v1/berth/schedule is the board: what holds a card, the waiters in
+// grant order, and the bookings. POST books a run, PATCH /{id} moves one and
+// DELETE /{id} cancels one. A priority change and downtime act on everyone
+// else's work, so they stay the operator's: berth schedule or a paired
+// harbor device.
+
+import (
+ "encoding/json"
+ "fmt"
+ "io"
+ "net/http"
+ "slices"
+ "time"
+
+ "github.com/T-Crypt/berth/internal/proto"
+ "github.com/T-Crypt/berth/internal/schedule"
+ "github.com/T-Crypt/berth/internal/server"
+)
+
+// bookBody is a booking as a client writes it. Start takes what berth
+// schedule --at takes, RFC 3339 included; For is a Go duration, "5h".
+type bookBody struct {
+ Who string `json:"who"`
+ Engine string `json:"engine"`
+ Priority string `json:"priority,omitempty"`
+ Start string `json:"start"`
+ For string `json:"for"`
+ Days int `json:"days,omitempty"`
+ Earliest bool `json:"earliest,omitempty"`
+ Card string `json:"card,omitempty"`
+ Note string `json:"note,omitempty"`
+}
+
+func (g *Gateway) scheduleRead(w http.ResponseWriter, r *http.Request) {
+ st := g.nodeStatus()
+ if st == nil {
+ apiError(w, http.StatusServiceUnavailable, "berthd is not answering; no schedule to show")
+ return
+ }
+ writeJSON(w, http.StatusOK, proto.BoardOf(st, time.Now()))
+}
+
+func (g *Gateway) scheduleBook(w http.ResponseWriter, r *http.Request) {
+ var in bookBody
+ if !readBody(w, r, &in) {
+ return
+ }
+ now := time.Now()
+ start, err := schedule.ParseAt(in.Start, now)
+ length, lerr := time.ParseDuration(in.For)
+ if err != nil || lerr != nil || length <= 0 {
+ apiError(w, http.StatusBadRequest, `the body must be JSON like {"who":"pi-pocket","engine":"NAME","start":"23:00","for":"5h"}`)
+ return
+ }
+ b := proto.Booking{Card: in.Card, Kind: proto.BookRun, Who: in.Who, Engine: in.Engine, Priority: in.Priority,
+ Start: start, End: start.Add(length), Note: in.Note}
+ g.scheduleAsk(w, proto.Msg{Op: proto.OpBook, Booking: &b, Days: in.Days, Earliest: in.Earliest}, http.StatusCreated)
+}
+
+// runBooking is the run booking with the path's ID, or false after the
+// reply says why not.
+func (g *Gateway) runBooking(w http.ResponseWriter, r *http.Request) (proto.Booking, bool) {
+ st := g.nodeStatus()
+ if st == nil {
+ apiError(w, http.StatusServiceUnavailable, "berthd is not answering; the schedule is unchanged")
+ return proto.Booking{}, false
+ }
+ id := r.PathValue("id")
+ i := slices.IndexFunc(st.Bookings, func(b proto.Booking) bool { return b.ID == id })
+ switch {
+ case i < 0:
+ apiError(w, http.StatusNotFound, fmt.Sprintf("no booking %s on the schedule", id))
+ return proto.Booking{}, false
+ case st.Bookings[i].Kind != proto.BookRun:
+ apiError(w, http.StatusForbidden, fmt.Sprintf("%s is a %s booking: only the operator changes it (berth schedule, or the harbor)", id, st.Bookings[i].Kind))
+ return proto.Booking{}, false
+ }
+ return st.Bookings[i], true
+}
+
+func (g *Gateway) scheduleMove(w http.ResponseWriter, r *http.Request) {
+ b, ok := g.runBooking(w, r)
+ if !ok {
+ return
+ }
+ var in bookBody
+ if !readBody(w, r, &in) {
+ return
+ }
+ length := b.End.Sub(b.Start)
+ if in.Start != "" {
+ start, err := schedule.ParseAt(in.Start, time.Now())
+ if err != nil {
+ apiError(w, http.StatusBadRequest, err.Error())
+ return
+ }
+ b.Start = start
+ }
+ if in.For != "" {
+ d, err := time.ParseDuration(in.For)
+ if err != nil || d <= 0 {
+ apiError(w, http.StatusBadRequest, fmt.Sprintf("for %q is not a length like 5h", in.For))
+ return
+ }
+ length = d
+ }
+ b.End = b.Start.Add(length)
+ if in.Priority != "" {
+ b.Priority = in.Priority
+ }
+ if in.Note != "" {
+ b.Note = in.Note
+ }
+ g.scheduleAsk(w, proto.Msg{Op: proto.OpRebook, Booking: &b}, http.StatusOK)
+}
+
+func (g *Gateway) scheduleCancel(w http.ResponseWriter, r *http.Request) {
+ b, ok := g.runBooking(w, r)
+ if !ok {
+ return
+ }
+ g.scheduleAsk(w, proto.Msg{Op: proto.OpUnbook, Booking: &proto.Booking{ID: b.ID}}, http.StatusOK)
+}
+
+// scheduleAsk sends one booking change to berthd and answers with the
+// bookings it changed, or 409 with berthd's reason.
+func (g *Gateway) scheduleAsk(w http.ResponseWriter, msg proto.Msg, ok int) {
+ conn, raw, err := server.Dial(g.Socket)
+ if err != nil {
+ apiError(w, http.StatusServiceUnavailable, "berthd is not answering; the schedule is unchanged")
+ return
+ }
+ defer raw.Close()
+ if err := conn.Send(msg); err != nil {
+ apiError(w, http.StatusServiceUnavailable, "berthd: "+err.Error())
+ return
+ }
+ m, err := conn.Recv()
+ switch {
+ case err != nil:
+ apiError(w, http.StatusServiceUnavailable, "no schedule reply from berthd: "+err.Error())
+ case m.Op == proto.OpRefused:
+ apiError(w, http.StatusConflict, m.Reason)
+ case m.Op != msg.Op:
+ apiError(w, http.StatusServiceUnavailable, fmt.Sprintf("berthd answered %q: restart it on this build", m.Op))
+ default:
+ writeJSON(w, ok, map[string][]proto.Booking{"bookings": m.Bookings})
+ }
+}
+
+func readBody(w http.ResponseWriter, r *http.Request, v any) bool {
+ body, err := io.ReadAll(io.LimitReader(r.Body, 1<<16))
+ if err != nil || json.Unmarshal(body, v) != nil {
+ apiError(w, http.StatusBadRequest, "the body must be JSON")
+ return false
+ }
+ return true
+}
+
+func writeJSON(w http.ResponseWriter, code int, v any) {
+ w.Header().Set("Content-Type", "application/json")
+ w.Header().Set("Cache-Control", "no-store")
+ w.WriteHeader(code)
+ _ = json.NewEncoder(w).Encode(v)
+}
diff --git a/internal/gateway/schedule_test.go b/internal/gateway/schedule_test.go
new file mode 100644
index 00000000..83168d9d
--- /dev/null
+++ b/internal/gateway/schedule_test.go
@@ -0,0 +1,112 @@
+// Copyright 2026 Trevin Tindall (T-Crypt)
+// SPDX-License-Identifier: Apache-2.0
+
+//go:build unix
+
+package gateway
+
+import (
+ "encoding/json"
+ "io"
+ "log/slog"
+ "net/http"
+ "net/http/httptest"
+ "path/filepath"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/T-Crypt/berth/internal/broker"
+ "github.com/T-Crypt/berth/internal/proto"
+ "github.com/T-Crypt/berth/internal/schedule"
+ "github.com/T-Crypt/berth/internal/server"
+)
+
+func scheduleGateway(t *testing.T) (*httptest.Server, *schedule.Store) {
+ t.Helper()
+ s, err := schedule.Open(filepath.Join(t.TempDir(), "schedule.json"))
+ if err != nil {
+ t.Fatal(err)
+ }
+ sock, _ := daemonServing(t, broker.Config{}, server.Extras{Schedule: s})
+ g := &Gateway{Socket: sock, Key: testKey, Log: slog.New(slog.NewTextHandler(io.Discard, nil))}
+ srv := httptest.NewServer(g.Handler())
+ t.Cleanup(srv.Close)
+ return srv, s
+}
+
+func decode[T any](t *testing.T, resp *http.Response) T {
+ t.Helper()
+ defer resp.Body.Close()
+ var v T
+ if err := json.NewDecoder(resp.Body).Decode(&v); err != nil {
+ t.Fatal(err)
+ }
+ return v
+}
+
+// An agent books its run, reads it on the board, moves it and cancels it;
+// a clash is a 409 naming the booking it hits.
+func TestScheduleRoutes(t *testing.T) {
+ srv, _ := scheduleGateway(t)
+ resp := call(t, "POST", srv.URL+"/v1/berth/schedule", `{"who":"pi-pocket","engine":"pocket","start":"23:00","for":"5h","priority":"batch"}`)
+ if resp.StatusCode != http.StatusCreated {
+ t.Fatalf("book = %d %s", resp.StatusCode, readAll(resp))
+ }
+ made := decode[map[string][]proto.Booking](t, resp)["bookings"]
+ if len(made) != 1 || made[0].ID != "b1" || made[0].Card != "gpu0" || made[0].End.Sub(made[0].Start) != 5*time.Hour {
+ t.Fatalf("booked = %+v", made)
+ }
+
+ resp = call(t, "POST", srv.URL+"/v1/berth/schedule", `{"who":"train","engine":"sft","start":"23:30","for":"1h"}`)
+ if body := readAll(resp); resp.StatusCode != http.StatusConflict || !strings.Contains(body, "clashes with b1") {
+ t.Fatalf("clash = %d %s", resp.StatusCode, body)
+ }
+
+ board := decode[proto.ScheduleBoard](t, call(t, "GET", srv.URL+"/v1/berth/schedule", ""))
+ if len(board.Bookings) != 1 || board.Bookings[0].Who != "pi-pocket" || board.Running == nil || board.Queued == nil {
+ t.Fatalf("board = %+v", board)
+ }
+
+ resp = call(t, "PATCH", srv.URL+"/v1/berth/schedule/b1", `{"for":"2h"}`)
+ moved := decode[map[string][]proto.Booking](t, resp)["bookings"]
+ if resp.StatusCode != http.StatusOK || len(moved) != 1 || moved[0].End.Sub(moved[0].Start) != 2*time.Hour {
+ t.Fatalf("move = %d %+v", resp.StatusCode, moved)
+ }
+
+ resp = call(t, "DELETE", srv.URL+"/v1/berth/schedule/b1", "")
+ if resp.StatusCode != http.StatusOK {
+ t.Fatalf("cancel = %d %s", resp.StatusCode, readAll(resp))
+ }
+ resp.Body.Close()
+ if board := decode[proto.ScheduleBoard](t, call(t, "GET", srv.URL+"/v1/berth/schedule", "")); len(board.Bookings) != 0 {
+ t.Fatalf("after cancel = %+v", board.Bookings)
+ }
+}
+
+// Downtime and priority changes are the operator's: the gateway books only
+// runs, and will not move or cancel anything else.
+func TestScheduleRoutesLeaveTheOperatorsBookings(t *testing.T) {
+ srv, s := scheduleGateway(t)
+ now := time.Now()
+ if _, err := s.Book(proto.Booking{Card: "gpu0", Kind: proto.BookDown, Who: "operator", Start: now.Add(time.Hour), End: now.Add(2 * time.Hour)}, 1, false, now); err != nil {
+ t.Fatal(err)
+ }
+ resp := call(t, "POST", srv.URL+"/v1/berth/schedule", `{"who":"agent","engine":"x","start":"now","for":"30m","kind":"down"}`)
+ made := decode[map[string][]proto.Booking](t, resp)["bookings"]
+ if resp.StatusCode != http.StatusCreated || made[0].Kind != proto.BookRun {
+ t.Fatalf("a kind in the body booked %+v (%d), want a run", made, resp.StatusCode)
+ }
+ for _, method := range []string{"PATCH", "DELETE"} {
+ resp := call(t, method, srv.URL+"/v1/berth/schedule/b1", `{"for":"3h"}`)
+ if body := readAll(resp); resp.StatusCode != http.StatusForbidden || !strings.Contains(body, "only the operator") {
+ t.Errorf("%s downtime = %d %s, want 403", method, resp.StatusCode, body)
+ }
+ }
+}
+
+func readAll(resp *http.Response) string {
+ defer resp.Body.Close()
+ b, _ := io.ReadAll(resp.Body)
+ return string(b)
+}
diff --git a/internal/gateway/yield.go b/internal/gateway/yield.go
new file mode 100644
index 00000000..df1e1475
--- /dev/null
+++ b/internal/gateway/yield.go
@@ -0,0 +1,96 @@
+// Copyright 2026 Trevin Tindall (T-Crypt)
+// SPDX-License-Identifier: Apache-2.0
+
+package gateway
+
+import (
+ "encoding/json"
+ "fmt"
+ "io"
+ "net/http"
+ "time"
+
+ "github.com/T-Crypt/berth/internal/proto"
+ "github.com/T-Crypt/berth/internal/server"
+)
+
+// A yield lasts this long when the caller names no length, and never longer
+// than the cap: a node that dies mid-yield must not leave its engine's next
+// drains skipping the job boundary for long.
+const (
+ defaultYield = 5 * time.Minute
+ maxYield = time.Hour
+)
+
+// apiYield is POST /v1/berth/yield, for a ComfyUI node that calls an LLM the
+// card's lease would otherwise keep waiting (#299). The node first frees its
+// models in-process. {"engine":"comfyui","seconds":120} says the running job
+// is at a safe point, so a drain for the LLM parks ComfyUI as it would between
+// jobs. {"engine":"comfyui","end":true} takes the card back: the answer comes
+// once ComfyUI holds it again, so the job's next node can load its models.
+func (g *Gateway) apiYield(w http.ResponseWriter, r *http.Request) {
+ var req struct {
+ Engine string `json:"engine"`
+ Seconds float64 `json:"seconds"`
+ End bool `json:"end"`
+ }
+ body, err := io.ReadAll(io.LimitReader(r.Body, 1<<16))
+ if err != nil || json.Unmarshal(body, &req) != nil || req.Engine == "" {
+ apiError(w, http.StatusBadRequest, `the body must be JSON like {"engine":"NAME","seconds":120} or {"engine":"NAME","end":true}`)
+ return
+ }
+ if req.Seconds < 0 || req.Seconds > maxYield.Seconds() {
+ apiError(w, http.StatusBadRequest, fmt.Sprintf("seconds must be between 0 and %d", int(maxYield.Seconds())))
+ return
+ }
+ e := g.engineByName(req.Engine)
+ if e == nil {
+ apiError(w, http.StatusNotFound, fmt.Sprintf("no engine %q in berth.toml", req.Engine))
+ return
+ }
+ d := req.Seconds
+ switch {
+ case req.End:
+ d = 0
+ case d == 0:
+ d = defaultYield.Seconds()
+ }
+ // The cursor comes before the end, so the wake it may queue is after it.
+ var since uint64
+ if st := g.nodeStatus(); st != nil {
+ since = st.LastEvent
+ }
+ conn, raw, err := server.Dial(g.Socket)
+ if err != nil {
+ apiError(w, http.StatusServiceUnavailable, "berthd is not answering; no yield was set")
+ return
+ }
+ defer raw.Close()
+ if err := conn.Send(proto.Msg{Op: proto.OpYield, Engine: req.Engine, Yield: d}); err != nil {
+ apiError(w, http.StatusServiceUnavailable, "berthd: "+err.Error())
+ return
+ }
+ m, err := conn.Recv()
+ switch {
+ case err != nil:
+ apiError(w, http.StatusServiceUnavailable, "no yield reply from berthd: "+err.Error())
+ return
+ case m.Op == proto.OpRefused:
+ apiError(w, http.StatusConflict, m.Reason)
+ return
+ }
+ // A parked ComfyUI answers its queue the whole time, so only the wake
+ // says it holds the card again.
+ if m.Resume {
+ if err := g.waitForWake(r.Context(), e.Name, since); err != nil {
+ g.startFailed(w, e.Name, e.Name, err)
+ return
+ }
+ }
+ w.Header().Set("Content-Type", "application/json")
+ _ = json.NewEncoder(w).Encode(struct {
+ Engine string `json:"engine"`
+ Yield bool `json:"yield"`
+ Seconds float64 `json:"seconds"`
+ }{req.Engine, !req.End, d})
+}
diff --git a/internal/gateway/yield_test.go b/internal/gateway/yield_test.go
new file mode 100644
index 00000000..5036d8f5
--- /dev/null
+++ b/internal/gateway/yield_test.go
@@ -0,0 +1,183 @@
+// Copyright 2026 Trevin Tindall (T-Crypt)
+// SPDX-License-Identifier: Apache-2.0
+
+//go:build unix
+
+package gateway
+
+import (
+ "context"
+ "encoding/json"
+ "fmt"
+ "net/http"
+ "net/http/httptest"
+ "strings"
+ "sync/atomic"
+ "testing"
+ "time"
+
+ "github.com/T-Crypt/berth/internal/broker"
+ "github.com/T-Crypt/berth/internal/config"
+ "github.com/T-Crypt/berth/internal/gpu"
+ "github.com/T-Crypt/berth/internal/park"
+ "github.com/T-Crypt/berth/internal/runner"
+)
+
+// The node's two calls: the yield answers with its length, and its end
+// answers once the engine holds the card again.
+func TestYieldRoute(t *testing.T) {
+ srv, b := switchGateway(t)
+ resp := call(t, "POST", srv.URL+"/v1/berth/batch", `{"engine":"strata"}`)
+ resp.Body.Close()
+ until(t, "strata to hold the card", func() bool {
+ st := b.Status()
+ return len(st.Leases) == 1 && st.Leases[0].Engine == "strata" && st.Leases[0].State == "active"
+ })
+
+ var got struct {
+ Yield bool `json:"yield"`
+ Seconds float64 `json:"seconds"`
+ }
+ resp = call(t, "POST", srv.URL+"/v1/berth/yield", `{"engine":"strata"}`)
+ err := json.NewDecoder(resp.Body).Decode(&got)
+ resp.Body.Close()
+ if resp.StatusCode != http.StatusOK || err != nil || !got.Yield || got.Seconds != defaultYield.Seconds() {
+ t.Fatalf("yield = %d %+v %v, want 200 and the default length", resp.StatusCode, got, err)
+ }
+
+ resp = call(t, "POST", srv.URL+"/v1/berth/yield", `{"engine":"strata","end":true}`)
+ err = json.NewDecoder(resp.Body).Decode(&got)
+ resp.Body.Close()
+ if resp.StatusCode != http.StatusOK || err != nil || got.Yield {
+ t.Fatalf("end = %d %+v %v", resp.StatusCode, got, err)
+ }
+ if st := b.Status(); len(st.Leases) != 1 || st.Leases[0].State != "active" {
+ t.Fatalf("after the end, leases = %+v", st.Leases)
+ }
+}
+
+func TestYieldRefusals(t *testing.T) {
+ srv, _ := switchGateway(t)
+ for _, c := range []struct {
+ body string
+ want int
+ }{
+ {`{"engine":"nope"}`, http.StatusNotFound},
+ {`not json`, http.StatusBadRequest},
+ {`{"engine":"strata","seconds":-1}`, http.StatusBadRequest},
+ {`{"engine":"strata","seconds":7200}`, http.StatusBadRequest},
+ {`{"engine":"strata","seconds":60}`, http.StatusConflict}, // strata holds no lease
+ } {
+ resp := call(t, "POST", srv.URL+"/v1/berth/yield", c.body)
+ resp.Body.Close()
+ if resp.StatusCode != c.want {
+ t.Errorf("%s = %d, want %d", c.body, resp.StatusCode, c.want)
+ }
+ }
+}
+
+// busyComfyStub is a ComfyUI whose one job waits on an LLM: its queue never
+// empties, /free lands, and the run probes /system_stats after the park.
+func busyComfyStub(t *testing.T, freed *atomic.Bool) *httptest.Server {
+ t.Helper()
+ srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ switch r.URL.Path {
+ case "/queue":
+ fmt.Fprint(w, `{"queue_running":[[1,"job"]],"queue_pending":[]}`)
+ case "/free":
+ freed.Store(true)
+ fmt.Fprint(w, `{}`)
+ default:
+ fmt.Fprint(w, `{"devices":[{"vram_free":1,"vram_total":2}]}`)
+ }
+ }))
+ t.Cleanup(srv.Close)
+ return srv
+}
+
+// #299 through the gateway: the end of a yield answers only once ComfyUI
+// holds the card again. A parked ComfyUI answers its queue the whole time, so
+// a probe says nothing; the answer must wait for the wake, or the job's next
+// node loads its models onto memory the LLM still holds.
+func TestYieldEndWaitsForTheCardBack(t *testing.T) {
+ card := &wakeCard{total: 24 << 30}
+ sock, b := parkDaemon(t, card, 16<<30)
+ var freed atomic.Bool
+ comfy := busyComfyStub(t, &freed)
+ adapter, ok := park.For("comfyui", "", "")
+ if !ok {
+ t.Fatal("the comfyui adapter is missing")
+ }
+ engineLog := &syncBuf{}
+ var pid int
+ runUntilEnd(t, &pid, runner.Options{Socket: sock, Engine: "comfyui", Priority: "interactive", Needs: []uint64{8 << 30},
+ DrainComfyUI: comfy.URL, Park: &adapter, ParkBase: comfy.URL, ParkProbe: "/system_stats",
+ Retry: 10 * time.Millisecond, Grace: 200 * time.Millisecond, Argv: []string{"sleep", "30"}, Stderr: engineLog})
+ until(t, "the run to attach", func() bool {
+ if st := b.Status(); len(st.Leases) > 0 && st.Leases[0].PID != 0 {
+ pid = st.Leases[0].PID
+ }
+ return pid != 0
+ })
+ card.set(gpu.Proc{PID: pid, Name: "sleep", Used: 8 << 30})
+ until(t, "the broker to measure ComfyUI", func() bool {
+ st := b.Status()
+ return len(st.Leases) > 0 && st.Leases[0].Used > 0
+ })
+
+ g := &Gateway{Socket: sock, Ready: 10 * time.Second, Key: testKey, Log: slogDiscard(),
+ Engines: []config.Engine{{Name: "comfyui", Type: "comfyui", URL: comfy.URL,
+ DrainComfyUI: comfy.URL, KeepComfyUI: true}}}
+ gw := httptest.NewServer(g.Handler())
+ defer gw.Close()
+
+ llm := make(chan *broker.Lease, 1)
+ go func() {
+ l, _ := b.Acquire(context.Background(), broker.Request{Engine: "gemma", Priority: broker.Foreground}, nil)
+ llm <- l
+ }()
+ until(t, "the drain to wait on the job", func() bool {
+ return strings.Contains(engineLog.String(), "waiting for ComfyUI to finish")
+ })
+ resp := call(t, "POST", gw.URL+"/v1/berth/yield", `{"engine":"comfyui","seconds":60}`)
+ resp.Body.Close()
+ if resp.StatusCode != http.StatusOK {
+ t.Fatalf("yield = %d", resp.StatusCode)
+ }
+ until(t, "the park unload", freed.Load)
+ card.set(gpu.Proc{PID: pid, Name: "sleep", Used: 512 << 20})
+ hold := <-llm
+ if hold == nil {
+ t.Fatalf("gemma was never granted beside the parked job; engine log:\n%s", engineLog.String())
+ }
+
+ ended := make(chan time.Time, 1)
+ go func() {
+ resp := call(t, "POST", gw.URL+"/v1/berth/yield", `{"engine":"comfyui","end":true}`)
+ resp.Body.Close()
+ if resp.StatusCode != http.StatusOK {
+ t.Errorf("end = %d", resp.StatusCode)
+ }
+ ended <- time.Now()
+ }()
+ time.Sleep(400 * time.Millisecond)
+ select {
+ case <-ended:
+ t.Fatal("the end answered while gemma still held the card")
+ default:
+ }
+ released := time.Now()
+ hold.Release()
+ card.set(gpu.Proc{PID: pid, Name: "sleep", Used: 512 << 20})
+ select {
+ case at := <-ended:
+ if at.Before(released) {
+ t.Fatal("the end answered before gemma let go")
+ }
+ case <-time.After(10 * time.Second):
+ t.Fatalf("the end never answered; engine log:\n%s", engineLog.String())
+ }
+ if st := b.Status(); len(st.Leases) != 1 || st.Leases[0].Engine != "comfyui" || st.Leases[0].State != "active" {
+ t.Fatalf("after the end, leases = %+v", st.Leases)
+ }
+}
diff --git a/internal/harbor/harbor.go b/internal/harbor/harbor.go
index ac1afc9e..982c3e18 100644
--- a/internal/harbor/harbor.go
+++ b/internal/harbor/harbor.go
@@ -132,6 +132,8 @@ func (h *Harbor) Handler() http.Handler {
mux.HandleFunc("POST /v1/board/switch", h.boardAction("switch"))
mux.HandleFunc("POST /v1/board/stop", h.boardAction("stop"))
mux.HandleFunc("POST /v1/board/load", h.boardAction("load"))
+ mux.HandleFunc("POST /v1/board/queue", h.boardQueue)
+ mux.HandleFunc("POST /v1/board/schedule", h.boardSchedule)
mux.HandleFunc("POST /v1/pair/code", h.pairCode)
return h.guard(mux)
}
@@ -190,7 +192,7 @@ func (h *Harbor) guard(next http.Handler) http.Handler {
}
// writePaths are the only paths the harbor takes a POST on.
-var writePaths = map[string]bool{"/pair": true, "/v1/board/switch": true, "/v1/board/stop": true, "/v1/board/load": true, "/v1/pair/code": true}
+var writePaths = map[string]bool{"/pair": true, "/v1/board/switch": true, "/v1/board/stop": true, "/v1/board/load": true, "/v1/board/queue": true, "/v1/board/schedule": true, "/v1/pair/code": true}
// mode tells the page which guard this harbor runs under. It is a harbor
// fact, stable for the process lifetime, so the page fetches it once: unlike
diff --git a/internal/harbor/queue_test.go b/internal/harbor/queue_test.go
new file mode 100644
index 00000000..d98b489d
--- /dev/null
+++ b/internal/harbor/queue_test.go
@@ -0,0 +1,86 @@
+// Copyright 2026 Trevin Tindall (T-Crypt)
+// SPDX-License-Identifier: Apache-2.0
+
+package harbor
+
+import (
+ "context"
+ "encoding/json"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/T-Crypt/berth/internal/broker"
+)
+
+// A paired device reorders and drops waiters through the harbor, straight to
+// berthd; an unpaired one is refused and the queue is unchanged; nothing goes
+// through the gateway, whose key agents hold.
+func TestBoardQueue(t *testing.T) {
+ base, gw, token, b := brokerHarbor(t, testKey)
+ if _, err := b.Acquire(context.Background(), broker.Request{Engine: "game", Priority: broker.Hold, Pinned: true}, nil); err != nil {
+ t.Fatal(err)
+ }
+ refused := make(chan error, 2)
+ for _, e := range []string{"first", "second"} {
+ go func() {
+ _, err := b.Acquire(context.Background(), broker.Request{Engine: e, Priority: broker.Interactive, Wait: time.Minute}, nil)
+ refused <- err
+ }()
+ time.Sleep(20 * time.Millisecond)
+ }
+ order := func() (ids, names []string) {
+ for _, l := range b.Status().Waiting {
+ ids, names = append(ids, l.ID), append(names, l.Engine)
+ }
+ return
+ }
+ ids, names := order()
+ if strings.Join(names, ",") != "first,second" {
+ t.Fatalf("queue = %v", names)
+ }
+ paired := map[string]string{"Cookie": cookieName + "=" + token, "X-Berth": "1", "Sec-Fetch-Site": "same-origin"}
+ unpaired := map[string]string{"X-Berth": "1", "Sec-Fetch-Site": "same-origin"}
+
+ code, body := call(t, "POST", base+"/v1/board/queue", `{"lease":"`+ids[1]+`","action":"bump"}`, unpaired)
+ if code != 401 {
+ t.Fatalf("unpaired bump = %d %s, want 401", code, body)
+ }
+ if _, names = order(); strings.Join(names, ",") != "first,second" {
+ t.Fatalf("an unpaired bump moved the queue: %v", names)
+ }
+
+ code, body = call(t, "POST", base+"/v1/board/queue", `{"lease":"`+ids[1]+`","action":"bump"}`, paired)
+ var got struct{ Lease, Change string }
+ if code != 200 || json.Unmarshal([]byte(body), &got) != nil || got.Lease != ids[1] ||
+ got.Change != "moved to the front of its class by the operator" {
+ t.Fatalf("bump = %d %s", code, body)
+ }
+ if _, names = order(); strings.Join(names, ",") != "second,first" {
+ t.Fatalf("after bump: %v", names)
+ }
+
+ for _, bad := range []string{`{"lease":"x","action":"jump"}`, `{"lease":"x","action":"prio"}`, `{"action":"drop"}`} {
+ if code, body = call(t, "POST", base+"/v1/board/queue", bad, paired); code != 400 {
+ t.Fatalf("%s = %d %s, want 400", bad, code, body)
+ }
+ }
+ if code, body = call(t, "POST", base+"/v1/board/queue", `{"lease":"`+ids[0]+`","action":"prio","priority":"hold"}`, paired); code != 409 {
+ t.Fatalf("prio hold = %d %s, want 409", code, body)
+ }
+
+ if code, body = call(t, "POST", base+"/v1/board/queue", `{"lease":"`+ids[0]+`","action":"drop"}`, paired); code != 200 {
+ t.Fatalf("drop = %d %s", code, body)
+ }
+ select {
+ case err := <-refused:
+ if err == nil || !strings.Contains(err.Error(), "dropped from the queue by the operator") {
+ t.Fatalf("dropped waiter: %v", err)
+ }
+ case <-time.After(time.Second):
+ t.Fatal("dropped waiter still waiting")
+ }
+ if gw.last() != "" {
+ t.Fatalf("a queue action reached the gateway: %q", gw.last())
+ }
+}
diff --git a/internal/harbor/schedule_test.go b/internal/harbor/schedule_test.go
new file mode 100644
index 00000000..67ea1cd0
--- /dev/null
+++ b/internal/harbor/schedule_test.go
@@ -0,0 +1,60 @@
+// Copyright 2026 Trevin Tindall (T-Crypt)
+// SPDX-License-Identifier: Apache-2.0
+
+package harbor
+
+import (
+ "encoding/json"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/T-Crypt/berth/internal/proto"
+)
+
+// A paired device books downtime, moves it and cancels it through the
+// harbor, straight to berthd; the board reads it from /v1/status; an
+// unpaired device changes nothing; nothing reaches the gateway.
+func TestBoardSchedule(t *testing.T) {
+ base, gw, token, _ := brokerHarbor(t, testKey)
+ paired := map[string]string{"Cookie": cookieName + "=" + token, "X-Berth": "1", "Sec-Fetch-Site": "same-origin"}
+ unpaired := map[string]string{"X-Berth": "1", "Sec-Fetch-Site": "same-origin"}
+ start := time.Now().Add(time.Hour).Truncate(time.Minute)
+ down := func(id string, from time.Time, d time.Duration) string {
+ b, _ := json.Marshal(proto.Booking{ID: id, Kind: proto.BookDown, Who: "operator", Start: from, End: from.Add(d), Note: "driver update"})
+ return string(b)
+ }
+
+ if code, body := call(t, "POST", base+"/v1/board/schedule", `{"action":"book","booking":`+down("", start, time.Hour)+`}`, unpaired); code != 401 {
+ t.Fatalf("unpaired book = %d %s, want 401", code, body)
+ }
+ code, body := call(t, "POST", base+"/v1/board/schedule", `{"action":"book","days":2,"booking":`+down("", start, time.Hour)+`}`, paired)
+ var got struct{ Bookings []proto.Booking }
+ if code != 200 || json.Unmarshal([]byte(body), &got) != nil || len(got.Bookings) != 2 || got.Bookings[0].Card != "gpu0" {
+ t.Fatalf("book = %d %s", code, body)
+ }
+
+ code, body = call(t, "GET", base+"/v1/status", "", nil)
+ var st proto.Status
+ if code != 200 || json.Unmarshal([]byte(body), &st) != nil || len(st.Bookings) != 2 || st.Bookings[0].Note != "driver update" {
+ t.Fatalf("status bookings = %d %s", code, body)
+ }
+
+ if code, body = call(t, "POST", base+"/v1/board/schedule", `{"action":"move","booking":`+down("b1", start.Add(30*time.Minute), time.Hour)+`}`, paired); code != 200 {
+ t.Fatalf("move = %d %s", code, body)
+ }
+ if code, body = call(t, "POST", base+"/v1/board/schedule", `{"action":"move","booking":`+down("b1", start.AddDate(0, 0, 1), time.Hour)+`}`, paired); code != 409 || !strings.Contains(body, "clashes with b2") {
+ t.Fatalf("move onto b2 = %d %s, want 409", code, body)
+ }
+ if code, body = call(t, "POST", base+"/v1/board/schedule", `{"action":"cancel","booking":{"id":"b2"}}`, paired); code != 200 {
+ t.Fatalf("cancel = %d %s", code, body)
+ }
+ for _, bad := range []string{`{"action":"nap","booking":{}}`, `{"action":"cancel","booking":{}}`, `{"action":"book","booking":{"id":"b1"}}`, `{"action":"book"}`} {
+ if code, body = call(t, "POST", base+"/v1/board/schedule", bad, paired); code != 400 {
+ t.Fatalf("%s = %d %s, want 400", bad, code, body)
+ }
+ }
+ if gw.last() != "" {
+ t.Fatalf("a schedule action reached the gateway: %q", gw.last())
+ }
+}
diff --git a/internal/harbor/switchboard.go b/internal/harbor/switchboard.go
index 6ff5f5ee..c7c10a4b 100644
--- a/internal/harbor/switchboard.go
+++ b/internal/harbor/switchboard.go
@@ -75,6 +75,70 @@ func (h *Harbor) boardAction(action string) http.HandlerFunc {
}
}
+// boardQueue is POST /v1/board/queue: a paired device bumps, re-ranks or
+// drops one waiter. It goes to berthd's socket, not the gateway: the
+// gateway's API key is held by agents, and a queue change is the
+// operator's, so only a paired device makes one.
+func (h *Harbor) boardQueue(w http.ResponseWriter, r *http.Request) {
+ if !h.writeOK(w, r) {
+ return
+ }
+ var req struct {
+ Lease string `json:"lease"`
+ Action string `json:"action"`
+ Priority string `json:"priority,omitempty"`
+ }
+ body, err := io.ReadAll(io.LimitReader(r.Body, 1<<16))
+ if err != nil || json.Unmarshal(body, &req) != nil || req.Lease == "" ||
+ (req.Action != "bump" && req.Action != "prio" && req.Action != "drop") || (req.Action == "prio") != (req.Priority != "") {
+ apiError(w, http.StatusBadRequest, "bad_body", `the body must be JSON like {"lease":"ID","action":"bump|drop"} or {"lease":"ID","action":"prio","priority":"batch|interactive|foreground"}`)
+ return
+ }
+ m, err := h.ask(proto.Msg{Op: proto.OpQueue, Lease: req.Lease, Action: req.Action, Priority: req.Priority})
+ switch {
+ case err != nil:
+ apiError(w, http.StatusServiceUnavailable, "berthd_down", "berthd is not answering; the queue is unchanged")
+ case m.Op == proto.OpRefused:
+ apiError(w, http.StatusConflict, "refused", m.Reason)
+ default:
+ w.Header().Set("Content-Type", "application/json")
+ _ = json.NewEncoder(w).Encode(map[string]string{"lease": m.Lease, "change": m.Reason})
+ }
+}
+
+// boardSchedule is POST /v1/board/schedule: a paired device books, moves or
+// cancels any booking, downtime and priority changes included. Like the
+// queue, it goes to berthd's socket: agents holding the gateway's key book
+// only their own runs there.
+func (h *Harbor) boardSchedule(w http.ResponseWriter, r *http.Request) {
+ if !h.writeOK(w, r) {
+ return
+ }
+ var req struct {
+ Action string `json:"action"`
+ Booking *proto.Booking `json:"booking"`
+ Days int `json:"days,omitempty"`
+ Earliest bool `json:"earliest,omitempty"`
+ }
+ body, err := io.ReadAll(io.LimitReader(r.Body, 1<<16))
+ op := map[string]string{"book": proto.OpBook, "move": proto.OpRebook, "cancel": proto.OpUnbook}
+ if err != nil || json.Unmarshal(body, &req) != nil || op[req.Action] == "" || req.Booking == nil ||
+ (req.Action != "book") != (req.Booking.ID != "") {
+ apiError(w, http.StatusBadRequest, "bad_body", `the body must be JSON like {"action":"book","booking":{...}} or {"action":"move|cancel","booking":{"id":"b1",...}}`)
+ return
+ }
+ m, err := h.ask(proto.Msg{Op: op[req.Action], Booking: req.Booking, Days: req.Days, Earliest: req.Earliest})
+ switch {
+ case err != nil:
+ apiError(w, http.StatusServiceUnavailable, "berthd_down", "berthd is not answering; the schedule is unchanged")
+ case m.Op == proto.OpRefused:
+ apiError(w, http.StatusConflict, "refused", m.Reason)
+ default:
+ w.Header().Set("Content-Type", "application/json")
+ _ = json.NewEncoder(w).Encode(map[string][]proto.Booking{"bookings": m.Bookings})
+ }
+}
+
// toGateway sends one request to the gateway with the node's key and hands
// its status and body back to the page.
func (h *Harbor) toGateway(w http.ResponseWriter, r *http.Request, method, path string, body []byte) {
diff --git a/internal/harbor/switchboard_test.go b/internal/harbor/switchboard_test.go
index b20ecad8..95aa6de1 100644
--- a/internal/harbor/switchboard_test.go
+++ b/internal/harbor/switchboard_test.go
@@ -20,6 +20,7 @@ import (
"github.com/T-Crypt/berth/internal/node"
"github.com/T-Crypt/berth/internal/pair"
"github.com/T-Crypt/berth/internal/procs"
+ "github.com/T-Crypt/berth/internal/schedule"
"github.com/T-Crypt/berth/internal/server"
)
@@ -63,11 +64,19 @@ func (f *fakeGateway) last() string {
// switchHarbor is a harbor in front of a real berthd that owns a pairing
// file and a fake gateway, with one paired device's token.
func switchHarbor(t *testing.T, key string) (base string, gw *fakeGateway, token string) {
+ t.Helper()
+ base, gw, token, _ = brokerHarbor(t, key)
+ return base, gw, token
+}
+
+// brokerHarbor is switchHarbor with the broker behind it, for a test that
+// puts leases on the card.
+func brokerHarbor(t *testing.T, key string) (base string, gw *fakeGateway, token string, b *broker.Broker) {
t.Helper()
dir := t.TempDir()
sock := filepath.Join(dir, "berth.sock")
log := slog.New(slog.NewTextHandler(io.Discard, nil))
- b := broker.New(broker.Config{Poll: 20 * time.Millisecond, Log: log}, idleCard{}, procs.System{})
+ b = broker.New(broker.Config{Poll: 20 * time.Millisecond, Log: log}, idleCard{}, procs.System{})
ln, err := server.Listen(sock)
if err != nil {
t.Fatal(err)
@@ -75,7 +84,11 @@ func switchHarbor(t *testing.T, key string) (base string, gw *fakeGateway, token
ctx, cancel := context.WithCancel(context.Background())
t.Cleanup(cancel)
s := &pair.Store{Path: filepath.Join(dir, "harbor-devices.json")}
- go server.Serve(ctx, ln, node.One(b), server.Extras{Pair: s}, log)
+ sched, err := schedule.Open(filepath.Join(dir, "schedule.json"))
+ if err != nil {
+ t.Fatal(err)
+ }
+ go server.Serve(ctx, ln, node.One(b), server.Extras{Pair: s, Schedule: sched}, log)
code, err := s.Code()
if err != nil {
t.Fatal(err)
@@ -88,7 +101,7 @@ func switchHarbor(t *testing.T, key string) (base string, gw *fakeGateway, token
t.Cleanup(gsrv.Close)
srv := httptest.NewServer((&Harbor{Socket: sock, Log: log, Gateway: gsrv.URL, GatewayKey: key}).Handler())
t.Cleanup(srv.Close)
- return srv.URL, gw, token
+ return srv.URL, gw, token, b
}
func call(t *testing.T, method, url, body string, hdr map[string]string) (int, string) {
diff --git a/internal/harbor/web/demo.json b/internal/harbor/web/demo.json
index 0e6951ea..af73c4db 100644
--- a/internal/harbor/web/demo.json
+++ b/internal/harbor/web/demo.json
@@ -127,6 +127,10 @@
"age_seconds": 40
}
],
+ "schedules": [
+ {"engine": "nightly-train", "window": "23:00-07:00", "open": false, "change_s": 10800},
+ {"engine": "render", "window": "18:00-22:00", "open": true, "change_s": 8100}
+ ],
"waiting": [
{
"id": "LIT-0145",
diff --git a/internal/harbor/web/harbor.css b/internal/harbor/web/harbor.css
index 1a91cf0f..1f61314a 100644
--- a/internal/harbor/web/harbor.css
+++ b/internal/harbor/web/harbor.css
@@ -1036,6 +1036,10 @@ body.ready #panel-engines > :nth-child(n+2) { animation-delay: 60ms; }
.sq.sw-up { border-left-color: var(--green); }
.sq.sw-waiting, .sq.sw-starting { border-left-color: var(--amber); }
.sq.sw-failed { border-left-color: var(--magenta); }
+/* A schedule row: green while its hours are open, dim while shut. */
+.sq.sched { border-left-color: var(--dim-2); }
+.sq.sched .name { color: var(--ink-strong); }
+.sq.sched-open { border-left-color: var(--green); }
.sw-state { display: inline-flex; align-items: center; gap: var(--s1); white-space: nowrap; }
.sw-chip { border: 1px solid var(--hair-2); border-radius: var(--r-pill); padding: 0 var(--s2); font-size: var(--fs-micro); color: var(--dim); }
.sw-cordon, .sw-pinned { color: var(--amber); border-color: var(--amber-dim); }
@@ -1044,6 +1048,16 @@ body.ready #panel-engines > :nth-child(n+2) { animation-delay: 60ms; }
.sw-btn { min-height: var(--touch); min-width: 88px; }
.sw-btn:disabled { opacity: 0.5; cursor: progress; }
.sw-stop { color: var(--amber); border-color: var(--amber-dim); }
+/* the queue's priority picker: a select drawn like .btn, not the browser's default */
+.sw-prio {
+ min-height: var(--touch); padding: 6px 14px;
+ background: var(--panel-2); color: var(--ink);
+ border: 1px solid var(--edge); border-radius: var(--r2);
+ font-family: var(--mono); font-size: var(--fs-small); letter-spacing: 1px;
+ cursor: pointer;
+}
+.sw-prio:hover, .sw-prio:focus-visible { color: var(--ink-strong); border-color: var(--dim-2); }
+.sw-prio:disabled { opacity: 0.5; cursor: progress; }
.sw-error { color: var(--magenta); }
.sw-detail { color: var(--amber); }
.sw-paired { font-family: var(--mono); font-size: var(--fs-small); color: var(--dim); display: flex; flex-wrap: wrap; align-items: center; justify-content: space-between; gap: var(--s2); margin-bottom: var(--s3); }
@@ -1980,3 +1994,79 @@ body.reduced-motion .si-live, body.reduced-motion .si-connecting, body.reduced-m
.pair input { padding: 0.6rem; font-size: 1rem; color: var(--ink-strong); background: var(--panel-3); border: 1px solid var(--edge); border-radius: 6px; }
.pair button { padding: 0.75rem; font-size: 1rem; color: var(--ink-strong); background: var(--cyan-deep); border: 1px solid var(--cyan-dim); border-radius: 6px; }
.pair a { color: var(--cyan); }
+
+/* ---------------- SCHEDULE: the card's scheduler ---------------- */
+.sb-now { display: flex; flex-direction: column; gap: var(--s2); }
+.sb-line { font-family: var(--mono); font-size: var(--fs-small); color: var(--ink); border-left: 3px solid var(--hair-2); padding: var(--s1) var(--s2); }
+.sb-line-run { border-left-color: var(--cyan); }
+.sb-line-priority { border-left-color: var(--amber); }
+.sb-line-down { border-left-color: var(--magenta); color: var(--ink-strong); }
+.sb-chip-run { color: var(--cyan); border-color: var(--cyan-dim); }
+.sb-head-actions { display: inline-flex; align-items: center; gap: var(--s3); flex-wrap: wrap; justify-content: flex-end; }
+.sb-legend { display: inline-flex; align-items: center; gap: var(--s1); font-size: var(--fs-micro); color: var(--dim); }
+.sb-key { display: inline-block; width: 12px; height: 8px; border-radius: 2px; margin-left: var(--s2); }
+.sb-k-run { background: var(--cyan-deep); border: 1px solid var(--cyan); }
+.sb-k-priority { background: transparent; border: 1px dashed var(--amber); }
+.sb-k-down { background: repeating-linear-gradient(135deg, var(--magenta-deep) 0 3px, transparent 3px 6px); border: 1px solid var(--magenta); }
+.sb-k-window { background: var(--green); height: 3px; opacity: 0.6; }
+.sb-k-live { background: var(--green); }
+.sb-card-pick { display: flex; gap: var(--s2); margin-bottom: var(--s2); flex-wrap: wrap; }
+.sb-card-pick .btn[aria-pressed="true"] { color: var(--cyan); border-color: var(--cyan); background: var(--cyan-wash); }
+.sb-board { display: flex; flex-direction: column; gap: 3px; overflow-x: auto; padding-bottom: var(--s1); }
+.sb-row { display: grid; grid-template-columns: 64px minmax(560px, 1fr); align-items: stretch; gap: var(--s2); }
+.sb-day { font-family: var(--mono); font-size: var(--fs-small); color: var(--dim); align-self: center; white-space: nowrap; }
+.sb-today { color: var(--amber); font-weight: 700; }
+.sb-track {
+ position: relative; height: 46px; border-radius: var(--r1); cursor: copy;
+ background-color: var(--panel-3);
+ /* an hour line every 1/24, stronger every 6 h */
+ background-image:
+ repeating-linear-gradient(to right, var(--hair) 0 1px, transparent 1px calc(100% / 4)),
+ repeating-linear-gradient(to right, var(--chart-grid) 0 1px, transparent 1px calc(100% / 24));
+}
+.sb-hours .sb-track { height: 16px; background: none; cursor: default; }
+.sb-tick { position: absolute; top: 0; transform: translateX(-1px); font-family: var(--mono); font-size: var(--fs-micro); color: var(--dim-2); }
+.sb-past { position: absolute; inset: 0 auto 0 0; background: rgba(6, 11, 17, 0.55); pointer-events: none; border-radius: var(--r1) 0 0 var(--r1); }
+.sb-nowline { position: absolute; top: -2px; bottom: -2px; width: 2px; background: var(--amber); box-shadow: 0 0 6px var(--amber); pointer-events: none; z-index: 3; }
+.sb-window { position: absolute; bottom: 1px; height: 3px; background: var(--green); opacity: 0.45; pointer-events: none; border-radius: 2px; }
+.sb-live { position: absolute; top: 1px; height: 4px; background: var(--green); border-radius: 2px; pointer-events: none; z-index: 2; }
+.sb-bar {
+ position: absolute; top: 7px; height: 22px; min-width: 3px; padding: 0 var(--s1);
+ overflow: hidden; border-radius: 3px; cursor: pointer; text-align: left;
+ font-family: var(--mono); font-size: var(--fs-micro); line-height: 20px; color: var(--ink-strong);
+ border: 1px solid var(--cyan); background: var(--cyan-deep);
+}
+.sb-bar:hover, .sb-bar:focus-visible { filter: brightness(1.25); outline: 1px solid var(--ink-strong); z-index: 4; }
+.sb-bar-text { white-space: nowrap; }
+.sb-priority { background: rgba(232, 181, 111, 0.12); border: 1px dashed var(--amber); color: var(--amber); }
+.sb-down { background: repeating-linear-gradient(135deg, var(--magenta-deep) 0 6px, var(--magenta-dim) 6px 12px); border-color: var(--magenta); }
+.sb-lane2 { top: 31px; height: 13px; line-height: 11px; }
+.sb-hint { margin-top: var(--s2); font-size: var(--fs-micro); color: var(--dim-2); }
+.sq.sb-item-run { border-left-color: var(--cyan); }
+.sq.sb-item-priority { border-left-color: var(--amber); }
+.sq.sb-item-down { border-left-color: var(--magenta); }
+
+dialog#sched-dlg { border: 1px solid var(--hair-2); padding: 0; background: var(--panel); color: var(--ink); border-radius: 8px; box-shadow: var(--shadow-pop); max-width: 460px; width: calc(100vw - 32px); }
+dialog#sched-dlg::backdrop { background: var(--scrim); backdrop-filter: blur(3px); }
+.sched-form { padding: var(--s3); display: flex; flex-direction: column; gap: var(--s2); font-family: var(--mono); font-size: var(--fs-small); }
+.sched-form label { display: flex; flex-direction: column; gap: 2px; color: var(--dim); font-size: var(--fs-micro); letter-spacing: 0.5px; }
+.sched-form label[hidden], .sched-form .sf-row[hidden] { display: none; }
+.sched-form input, .sched-form select {
+ min-height: 36px; padding: 0 var(--s2); border: 1px solid var(--edge); border-radius: var(--r1);
+ background: var(--panel-3); color: var(--ink-strong); font: inherit; font-size: var(--fs-body);
+}
+.sched-form input:focus-visible, .sched-form select:focus-visible { outline: 2px solid var(--cyan); outline-offset: 1px; }
+.sf-row { display: flex; gap: var(--s3); flex-wrap: wrap; }
+.sf-row label { flex: 1 1 140px; }
+.sf-row input[type="number"] { width: 100%; }
+.sched-form .sf-check { flex-direction: row; align-items: center; gap: var(--s2); align-self: flex-end; min-height: 36px; }
+.sched-form .sf-check input { min-height: 0; }
+.sf-explain { margin: 0; color: var(--ink); line-height: 1.5; border-left: 3px solid var(--cyan-dim); padding-left: var(--s2); }
+.sf-error { margin: 0; color: var(--magenta); }
+.sf-error:empty, .sf-explain:empty { display: none; }
+@media (max-width: 640px) {
+ .sb-row { grid-template-columns: 44px minmax(520px, 1fr); }
+ .sb-legend { display: none; }
+}
+.sb-edit { cursor: pointer; background: none; color: var(--cyan); border-color: var(--cyan-dim); font-family: var(--mono); min-height: 24px; }
+.sb-edit:hover, .sb-edit:focus-visible { color: var(--ink-strong); border-color: var(--cyan); }
diff --git a/internal/harbor/web/harbor.js b/internal/harbor/web/harbor.js
index b6aa942e..e0d87d7a 100644
--- a/internal/harbor/web/harbor.js
+++ b/internal/harbor/web/harbor.js
@@ -13,6 +13,7 @@ import { EngineView } from "./engines.js";
import { LibraryView } from "./library.js";
import { ResidencyView } from "./residency.js";
import { initSwitch } from "./switch.js";
+import { initSchedule } from "./schedule.js";
import { sparkline, lineChart, stackedArea, timelineChart } from "./charts.js";
import { PAL } from "./palette.js";
import { GIB } from "./util.js";
@@ -124,7 +125,7 @@ function initialLbRange() {
return "24h";
}
-let board, ticket, engineView, residency, priorityView, libraryView, switchView;
+let board, ticket, engineView, residency, priorityView, libraryView, switchView, scheduleView;
let connPill, statePill, vmPill, nodeVersion, lastUpdate, demoStrip, alertStripEl;
let odometerReels, counterSince, counterMeta;
let ctiles;
@@ -133,7 +134,7 @@ let shortcutsDlg, railMenuDlg;
let lastLogSig = null; // filter+query+pause signature: repaint only on change, append otherwise
let clearLogFilter = null; // set by buildLogSearch: the page's Esc reaches it (R-07)
let lastTick = 0; // the relative-time ticker's own 1 s cadence
-let berthsRoot, claimsRoot, claimsMeta, stowRoot, notickRoot, machineRoot, machineCharts, machineMeta, socketLine;
+let berthsRoot, schedulesRoot, schedulesMeta, claimsRoot, claimsMeta, stowRoot, notickRoot, machineRoot, machineCharts, machineMeta, socketLine;
let kpiStripEl, tideEl, tideLegend, tideMeta;
let boardMeta, berthsMeta, stowMeta, noticketMeta, enginesMeta;
let throughputMeta, throughputDecode, throughputRpm;
@@ -168,6 +169,8 @@ function main() {
shortcutsDlg = $("shortcuts");
railMenuDlg = $("rail-menu");
berthsRoot = $("berths");
+ schedulesRoot = $("schedules");
+ schedulesMeta = $("schedules-meta");
claimsRoot = $("claims");
claimsMeta = $("claims-meta");
stowRoot = $("stowaways");
@@ -203,6 +206,7 @@ function main() {
residency = new ResidencyView($("residency-canvas"), $("residency-legend"), $("residency-sessions"), $("residency-meta"));
residency.colorOf = (name) => engineColor(name);
switchView = initSwitch({ demo: state.demo });
+ scheduleView = initSchedule({ demo: state.demo });
buildKpiStrip();
wireMode();
applyMode();
@@ -469,8 +473,8 @@ function applyMode() {
// ---------------- tabs ----------------
function wireTabs() {
- const tabs = { harbor: $("tab-harbor"), switch: $("tab-switch"), engines: $("tab-engines"), about: $("tab-about") };
- const panels = { harbor: $("panel-harbor"), switch: $("panel-switch"), engines: $("panel-engines"), about: $("panel-about") };
+ const tabs = { harbor: $("tab-harbor"), switch: $("tab-switch"), schedule: $("tab-schedule"), engines: $("tab-engines"), about: $("tab-about") };
+ const panels = { harbor: $("panel-harbor"), switch: $("panel-switch"), schedule: $("panel-schedule"), engines: $("panel-engines"), about: $("panel-about") };
const show = (key) => {
if (!tabs[key]) key = "harbor";
for (const k of Object.keys(tabs)) {
@@ -480,6 +484,7 @@ function wireTabs() {
}
if (key === "engines" && state.status) engineView.render(state.status);
if (key === "switch") switchView.show(); else switchView.hide();
+ if (key === "schedule") scheduleView.show(); else scheduleView.hide();
};
for (const key of Object.keys(tabs)) {
tabs[key].addEventListener("click", () => {
@@ -508,14 +513,15 @@ let pendingSection = null;
const HARBOR_SECTIONS = [
["sec-counter", "Counter"], ["sec-tide", "Tide"], ["sec-timeline", "Timeline"],
["sec-board", "Board"], ["sec-log", "Log"], ["sec-berths", "Berths"],
- ["sec-machine", "Machine"], ["sec-claims", "Claims"], ["sec-stow", "Stowaways"], ["sec-noticket", "No ticket"],
+ ["sec-machine", "Machine"], ["sec-schedules", "Schedules"], ["sec-claims", "Claims"], ["sec-stow", "Stowaways"], ["sec-noticket", "No ticket"],
];
const ENGINES_SECTIONS = [
["sec-engines", "Engines"], ["sec-library", "Library"], ["priority-panel", "Ranking"], ["sec-pool", "KV pool"],
["sec-throughput", "Throughput"], ["sec-ledger", "Ledger"], ["sec-lb", "Leaderboard"],
["sec-recent", "Recent"], ["sec-residency", "Residency"], ["sec-storage", "Storage"],
];
-const SWITCH_SECTIONS = [["sec-switch-cards", "Cards"], ["sec-switch-engines", "Engines"]];
+const SWITCH_SECTIONS = [["sec-switch-cards", "Cards"], ["sec-switch-queue", "Queue"], ["sec-switch-engines", "Engines"]];
+const SCHEDULE_SECTIONS = [["sec-sched-now", "Now"], ["sec-sched-board", "7 days"], ["sec-sched-list", "Bookings"]];
const ABOUT_SECTIONS = [
["about-top", "Berth"], ["about-cards", "Cards"], ["about-modes", "Modes"], ["about-nav", "Getting around"],
["about-keys", "Keyboard"], ["about-states", "Status words"], ["about-kpis", "KPIs"], ["about-counters", "Counters"],
@@ -529,6 +535,7 @@ function wireChrome() {
} catch (e) { state.collapsed = []; }
buildSubnav($("subnav-harbor"), HARBOR_SECTIONS);
buildSubnav($("subnav-switch"), SWITCH_SECTIONS);
+ buildSubnav($("subnav-schedule"), SCHEDULE_SECTIONS);
buildSubnav($("subnav-engines"), ENGINES_SECTIONS);
buildSubnav($("subnav-about"), ABOUT_SECTIONS);
wireCollapse();
@@ -627,7 +634,7 @@ function wireShortcuts() {
const mode = document.querySelector(".app-header .mode");
if (mode) mode.before(keys);
shortcutsDlg.querySelector(".sc-close").addEventListener("click", () => shortcutsDlg.close());
- const tabFor = { 1: "tab-harbor", 2: "tab-switch", 3: "tab-engines", 4: "tab-about" };
+ const tabFor = { 1: "tab-harbor", 2: "tab-switch", 3: "tab-schedule", 4: "tab-engines", 5: "tab-about" };
for (const k of Object.keys(tabFor)) onKey(k, () => $(tabFor[k]).click());
onKey("b", () => setMode("basic", true));
onKey("a", () => setMode("advanced", true));
@@ -2097,6 +2104,22 @@ function claimLine(cl) {
function memoryWord(cl) {
return fmtGiB(cl.used || 0) + " GiB" + (cl.gtt_used ? " + " + fmtGiB(cl.gtt_used) + " GiB GTT" : "");
}
+// renderSchedules lists the engines held to hours, by the node's clock, as
+// berth status does: an engine shows here before its window opens.
+function renderSchedules(s) {
+ const list = el("div", "squatter-list");
+ const all = s.schedules || [];
+ const open = all.filter((sc) => sc.open);
+ schedulesMeta.textContent = all.length ? open.length + " open of " + all.length : "none";
+ if (all.length === 0) list.append(el("div", "empty-note", "no engine is held to hours: set window = \"23:00-07:00\" on an [[engine]]"));
+ for (const sc of all) {
+ const row = el("div", "sq sched" + (sc.open ? " sched-open" : ""));
+ const now = sc.open ? "open, closes in " + fmtAge(sc.change_s) : "closed, opens in " + fmtAge(sc.change_s);
+ row.append(el("div", "top", el("span", "name", sc.engine), el("span", null, sc.window)), el("div", "meta", now));
+ list.append(row);
+ }
+ schedulesRoot.replaceChildren(list);
+}
function renderClaims(s) {
const list = el("div", "squatter-list");
const all = s.claims || [];
@@ -2469,6 +2492,7 @@ function renderLight(s) {
paintTabCounts(entries.length, (s.engines || []).length);
renderBerths(s);
paintAlerts(s);
+ renderSchedules(s);
renderClaims(s);
renderStowaways(s);
renderNoTicket(s);
diff --git a/internal/harbor/web/index.html b/internal/harbor/web/index.html
index f70a8754..7d68f285 100644
--- a/internal/harbor/web/index.html
+++ b/internal/harbor/web/index.html
@@ -29,6 +29,7 @@
CARD
SWITCH
+ SCHEDULE
ENGINES & REQUESTS
ABOUT
@@ -168,6 +169,13 @@ THE MACHINE
+
+
+
SCHEDULES
+ engine hours
+
+
+
CLAIMS
@@ -202,6 +210,13 @@ CARDS
+
+
+
QUEUE
+ grant order: priority, then arrival
+
+
+
ENGINES
@@ -213,6 +228,36 @@
ENGINES
+
+
+
+
+
NOW
+ not read yet
+
+
+
+
+
+
7 DAYS
+
+ run priority down window live
+ BOOK
+
+
+
+
+
+
+
+
+
BOOKINGS
+ by start, node clock
+
+
+
+
+
Engines are what the engines report about themselves, not driver measurements. The ledger, leaderboard and request log are computed by the node from the requests it has seen.
@@ -350,7 +395,7 @@ GETTING AROUND
KEYBOARD
- 1 2 3 card, engines, about
+ 1 to 5 card, switch, schedule, engines, about
B A basic, advanced
P pause or resume the log
/ filter the log
@@ -502,7 +547,7 @@ ACTIONS
KEYBOARD SHORTCUTSclose
-
1 2 3 4 Card, switch, engines, about
+
1 2 3 4 5 Card, switch, schedule, engines, about
B A Basic, advanced mode
P Pause or resume the log
/ Filter the log
@@ -512,6 +557,45 @@
ACTIONS
+
+ BOOK THE CARD close
+
+
+
PAIR A PHONEclose
diff --git a/internal/harbor/web/schedule.js b/internal/harbor/web/schedule.js
new file mode 100644
index 00000000..982fa204
--- /dev/null
+++ b/internal/harbor/web/schedule.js
@@ -0,0 +1,388 @@
+// The SCHEDULE tab: the card's scheduler. NOW is what holds each card, the
+// waiters in grant order and the booking in force; 7 DAYS draws every
+// booking, each engine's window and the live holder on a 24 h track per day,
+// to the minute; BOOKINGS lists them. A paired device books, moves and
+// cancels through POST /v1/board/schedule, straight to berthd. API strings
+// only ever reach textContent.
+import { $, el } from "./util.js";
+import { showDialog, toast } from "./ui.js";
+
+const POLL_MS = 5000;
+const DAYS = 7;
+const HOUR = 3600e3;
+const SNAP_MIN = 15;
+const PICK_KEY = "berth.schedule.card";
+const KIND_WORD = { run: "RUN", priority: "PRIORITY", down: "DOWN" };
+
+const pad2 = (n) => String(n).padStart(2, "0");
+const hm = (d) => pad2(d.getHours()) + ":" + pad2(d.getMinutes());
+const dayWord = (d) => d.toLocaleDateString(undefined, { weekday: "short" }).toUpperCase() + " " + d.getDate();
+const when = (b) => {
+ const s = new Date(b.start), e = new Date(b.end);
+ const same = s.toDateString() === e.toDateString();
+ return dayWord(s) + " " + hm(s) + "-" + (same ? "" : dayWord(e) + " ") + hm(e);
+};
+function span(ms) {
+ const m = Math.max(0, Math.round(ms / 60e3));
+ const h = Math.floor(m / 60), r = m % 60;
+ if (h >= 48) return Math.round(h / 24) + "d";
+ return h > 0 ? h + "h" + (r ? pad2(r) + "m" : "") : r + "m";
+}
+// nowStep is now, floored to the start field's 5-minute step, so a booking
+// made "now" starts now and the field accepts it.
+const nowStep = () => Math.floor(Date.now() / 300e3) * 300e3;
+const localInput = (d) => d.getFullYear() + "-" + pad2(d.getMonth() + 1) + "-" + pad2(d.getDate()) + "T" + hm(d);
+const label = (b) => b.who + (b.engine && b.engine !== b.who ? " · " + b.engine : "") + (b.priority ? " · " + b.priority : "");
+
+// What a booking will do, in the words the operator reads before saving.
+function explain(b) {
+ const from = hm(new Date(b.start)), to = hm(new Date(b.end));
+ const eng = b.engine || "the engine";
+ switch (b.kind) {
+ case "run":
+ return eng + " goes first in line from " + from + " and runs at " + (b.priority || "foreground") +
+ "; until " + to + " nothing but a hold drains it. If it does not ask for the card, the card serves everyone else.";
+ case "priority":
+ return eng + " runs at " + (b.priority || "?") + " from " + from + " to " + to + ", then back to its own class. It claims no time on the card.";
+ case "down":
+ return "From " + from + " no new work starts and holders drain at their next boundary; a hold and a training stay. Work queued resumes at " + to + ".";
+ }
+ return "";
+}
+
+// Demo bookings, relative to now, so the board has something to show.
+function demoBookings() {
+ const at = (day, h, m = 0) => { const d = new Date(); d.setHours(0, 0, 0, 0); d.setDate(d.getDate() + day); d.setHours(h, m); return d; };
+ const out = [];
+ let n = 1;
+ const add = (kind, who, engine, priority, s, hours, note) =>
+ out.push({ id: "b" + n++, card: "gpu0", kind, who, engine, priority, start: s.toISOString(), end: new Date(s.getTime() + hours * HOUR).toISOString(), note });
+ for (let d = 0; d < DAYS; d++) {
+ add("run", "pi-pocket", "pocket-train", "batch", at(d, 23), 5, "omarchy nightly");
+ if (d % 7 !== 6) add("run", "operator", "comfyui", "", at(d, 9, 30), 8, "day work");
+ }
+ add("run", "training", "sft-4b", "foreground", at(1, 18), 3, "sft run 4");
+ add("priority", "operator", "llama-swap", "batch", at(1, 18), 3, "agents wait during training");
+ add("down", "operator", "", "", at(3, 4), 1, "driver update");
+ return out.sort((a, b) => a.start.localeCompare(b.start));
+}
+
+export function initSchedule({ demo }) {
+ const nowRoot = $("sb-now");
+ const nowMeta = $("sb-now-meta");
+ const boardRoot = $("sb-board");
+ const listRoot = $("sb-list");
+ const pickRoot = $("sb-card-pick");
+ const hint = $("sb-hint");
+ const dlg = $("sched-dlg");
+ const form = $("sched-form");
+ const explainEl = $("sched-explain");
+ const errEl = $("sched-error");
+ let st = null;
+ let paired = demo;
+ let timer = 0;
+ let pick = localStorage.getItem(PICK_KEY) || "";
+ let editing = null; // the booking the dialog moves, or null for a new one
+
+ async function read() {
+ try {
+ const res = await fetch(demo ? "demo.json" : "/v1/status", { headers: { accept: "application/json" } });
+ const j = await res.json();
+ if (!res.ok) throw new Error(j.message || res.statusText);
+ if (demo && !j.bookings) j.bookings = demoBookings();
+ st = j;
+ paint();
+ } catch (e) {
+ st = null;
+ nowRoot.replaceChildren(el("div", "empty-note error-note", "no status: " + e.message));
+ nowMeta.textContent = "not read";
+ }
+ }
+
+ function cards() {
+ const names = (st.gpus || []).filter((g) => g.scheduled).map((g) => g.card || "gpu0");
+ return names.length ? names : ["gpu0"];
+ }
+
+ function paint() {
+ const cs = cards();
+ if (!cs.includes(pick)) pick = cs[0];
+ pickRoot.replaceChildren();
+ if (cs.length > 1) {
+ for (const c of cs) {
+ const b = el("button", "btn sw-btn", c);
+ b.type = "button";
+ b.setAttribute("aria-pressed", String(c === pick));
+ b.addEventListener("click", () => { pick = c; localStorage.setItem(PICK_KEY, c); paint(); });
+ pickRoot.append(b);
+ }
+ }
+ paintNow(cs.length > 1);
+ paintBoard();
+ paintList(cs.length > 1);
+ $("sb-book").disabled = false;
+ hint.textContent = paired ? "click an empty hour to book it, or a booking to move or cancel it" : "read only: pair this device (SWITCH tab) to book";
+ }
+
+ function onCard(card) { return (card || "gpu0") === pick; }
+
+ function paintNow(many) {
+ const now = Date.now();
+ const rows = [];
+ const bs = (st.bookings || []);
+ for (const l of st.leases || []) {
+ if (!onCard(l.card)) continue;
+ const row = el("div", "sq sw-engine sw-up");
+ const right = el("span", "sw-state", el("span", "sw-chip", (l.priority || "").toUpperCase()), el("span", "sw-chip", (l.state || "").toUpperCase()));
+ if (l.booking) right.append(el("span", "sw-chip sb-chip-run", "BOOKED " + l.booking));
+ row.append(el("div", "top", el("span", "name", "HOLDS " + (l.group && l.group !== l.engine ? l.group + " (" + l.engine + ")" : l.engine)), right));
+ row.append(el("div", "meta", l.id + ", for " + span((l.age_seconds || 0) * 1000)));
+ rows.push(row);
+ }
+ if (!rows.length) rows.push(el("div", "empty-note", "nothing holds " + pick));
+ const waiting = (st.waiting || []).filter((l) => onCard(l.card));
+ waiting.forEach((l, i) => {
+ const row = el("div", "sq sw-engine sw-waiting");
+ const right = el("span", "sw-state", el("span", "sw-chip", (l.priority || "").toUpperCase()));
+ if (l.booking) right.append(el("span", "sw-chip sb-chip-run", "BOOKED " + l.booking));
+ row.append(el("div", "top", el("span", "name", (i + 1) + ". " + l.engine), right));
+ row.append(el("div", "meta", l.id + ", waiting " + span((l.age_seconds || 0) * 1000)));
+ rows.push(row);
+ });
+ const mine = bs.filter((b) => onCard(b.card));
+ const inForce = mine.filter((b) => new Date(b.start) <= now && now < new Date(b.end));
+ const next = mine.find((b) => new Date(b.start) > now);
+ for (const b of inForce) {
+ rows.push(el("div", "sb-line sb-line-" + b.kind, KIND_WORD[b.kind] + " NOW: " + label(b) + ", ends in " + span(new Date(b.end) - now) + (b.note ? " (" + b.note + ")" : "")));
+ }
+ if (next) rows.push(el("div", "sb-line", "NEXT: " + KIND_WORD[next.kind] + " " + label(next) + " in " + span(new Date(next.start) - now) + ", " + when(next)));
+ nowRoot.replaceChildren(...rows);
+ nowMeta.textContent = (many ? pick + ": " : "") + waiting.length + " waiting, " + mine.length + " booked";
+ }
+
+ function dayStart(offset) {
+ const d = new Date();
+ d.setHours(0, 0, 0, 0);
+ d.setDate(d.getDate() + offset);
+ return d;
+ }
+
+ // The part of [s, e) inside the day [d0, d1), as left and width percent.
+ function seg(s, e, d0, d1) {
+ const a = Math.max(s, d0), b = Math.min(e, d1);
+ if (b <= a) return null;
+ const len = d1 - d0;
+ return { left: ((a - d0) / len) * 100, width: ((b - a) / len) * 100, a, b };
+ }
+
+ function windowSegs(win, d0, d1) {
+ const m = /^(\d\d):(\d\d)-(\d\d):(\d\d)$/.exec(win || "");
+ if (!m) return [];
+ const at = (h, mm) => { const d = new Date(d0); d.setHours(+h, +mm, 0, 0); return d.getTime(); };
+ const s = at(m[1], m[2]), e = at(m[3], m[4]);
+ return s < e ? [[s, e]] : [[d0, e], [s, d1]];
+ }
+
+ function paintBoard() {
+ const now = Date.now();
+ const head = el("div", "sb-row sb-hours");
+ head.append(el("span", "sb-day", ""));
+ const scale = el("div", "sb-track sb-scale");
+ for (let h = 0; h < 24; h += 3) {
+ const t = el("span", "sb-tick", pad2(h));
+ t.style.left = (h / 24) * 100 + "%";
+ scale.append(t);
+ }
+ head.append(scale);
+ const rows = [head];
+ const bs = (st.bookings || []).filter((b) => onCard(b.card));
+ const live = (st.leases || []).filter((l) => onCard(l.card));
+ for (let i = 0; i < DAYS; i++) {
+ const d0 = dayStart(i).getTime(), d1 = dayStart(i + 1).getTime();
+ const row = el("div", "sb-row");
+ row.setAttribute("role", "row");
+ row.append(el("span", "sb-day" + (i === 0 ? " sb-today" : ""), i === 0 ? "TODAY" : dayWord(new Date(d0))));
+ const track = el("div", "sb-track");
+ track.setAttribute("role", "gridcell");
+ track.setAttribute("aria-label", dayWord(new Date(d0)));
+ if (i === 0) {
+ const past = el("div", "sb-past");
+ past.style.width = ((now - d0) / (d1 - d0)) * 100 + "%";
+ const line = el("div", "sb-nowline");
+ line.style.left = past.style.width;
+ line.title = "now " + hm(new Date(now));
+ track.append(past, line);
+ }
+ for (const s of st.schedules || []) {
+ for (const [a, b] of windowSegs(s.window, d0, d1)) {
+ const g = seg(a, b, d0, d1);
+ if (!g) continue;
+ const w = el("div", "sb-window");
+ w.style.left = g.left + "%";
+ w.style.width = g.width + "%";
+ w.title = "window " + s.window + ": " + s.engine;
+ track.append(w);
+ }
+ }
+ for (const l of live) {
+ const g = seg(now - (l.age_seconds || 0) * 1000, now, d0, d1);
+ if (!g) continue;
+ const bar = el("div", "sb-live");
+ bar.style.left = g.left + "%";
+ bar.style.width = Math.max(g.width, 0.3) + "%";
+ bar.title = "live: " + l.engine + " (" + l.priority + ") since " + hm(new Date(g.a));
+ track.append(bar);
+ }
+ const lanes = { run: 0, down: 0, priority: 1 };
+ for (const b of bs) {
+ const g = seg(new Date(b.start).getTime(), new Date(b.end).getTime(), d0, d1);
+ if (!g) continue;
+ const bar = el("button", "sb-bar sb-" + b.kind + (lanes[b.kind] ? " sb-lane2" : ""));
+ bar.type = "button";
+ bar.style.left = g.left + "%";
+ bar.style.width = g.width + "%";
+ const text = hm(new Date(g.a)) + " " + label(b);
+ bar.append(el("span", "sb-bar-text", text));
+ bar.title = KIND_WORD[b.kind] + " " + b.id + ": " + label(b) + "\n" + when(b) + " (" + span(new Date(b.end) - new Date(b.start)) + ")" + (b.note ? "\n" + b.note : "");
+ bar.setAttribute("aria-label", bar.title.replace(/\n/g, ", "));
+ bar.addEventListener("click", (e) => { e.stopPropagation(); openDialog(b); });
+ track.append(bar);
+ }
+ track.addEventListener("click", (e) => {
+ if (!paired) { toast("pair this device to book: berth pair on the machine", "info"); return; }
+ const r = track.getBoundingClientRect();
+ const t = d0 + ((e.clientX - r.left) / r.width) * (d1 - d0);
+ const snap = SNAP_MIN * 60e3;
+ openDialog(null, new Date(Math.max(Math.floor(t / snap) * snap, nowStep())));
+ });
+ row.append(track);
+ rows.push(row);
+ }
+ boardRoot.replaceChildren(...rows);
+ }
+
+ function paintList(many) {
+ const bs = (st.bookings || []).filter((b) => onCard(b.card));
+ if (!bs.length) { listRoot.replaceChildren(el("div", "empty-note", "nothing booked; every hour runs by priority, then arrival")); return; }
+ const now = Date.now();
+ listRoot.replaceChildren(...bs.map((b) => {
+ const row = el("div", "sq sw-engine sb-item sb-item-" + b.kind);
+ const right = el("span", "sw-state", el("span", "sw-chip", KIND_WORD[b.kind]));
+ if (new Date(b.start) <= now) right.append(el("span", "sw-chip sw-pinned", "IN FORCE"));
+ if (many) right.prepend(el("span", "sw-chip", b.card));
+ row.append(el("div", "top", el("span", "name", b.id + " " + label(b)), right));
+ row.append(el("div", "meta", when(b) + " (" + span(new Date(b.end) - new Date(b.start)) + ")" + (b.note ? ", " + b.note : "")));
+ if (paired) {
+ const edit = el("button", "sw-chip sb-edit", "EDIT");
+ edit.type = "button";
+ edit.setAttribute("aria-label", "move or cancel " + b.id);
+ edit.addEventListener("click", () => openDialog(b));
+ right.append(edit);
+ }
+ return row;
+ }));
+ }
+
+ function engineNames() {
+ const names = new Set();
+ for (const e of st.engines || []) if (e.name) names.add(e.name);
+ for (const f of st.footprints || []) if (f.engine) names.add(f.engine);
+ for (const l of [...(st.leases || []), ...(st.waiting || [])]) { names.add(l.engine); if (l.group) names.add(l.group); }
+ return [...names].sort();
+ }
+
+ function formBooking() {
+ const f = form.elements;
+ const start = new Date(f.start.value);
+ const b = {
+ kind: f.kind.value, who: f.who.value.trim(), engine: f.engine.value.trim(), priority: f.priority.value,
+ start: start.toISOString(), end: new Date(start.getTime() + Number(f.hours.value) * HOUR).toISOString(),
+ note: f.note.value.trim(), card: pick,
+ };
+ if (b.kind === "down") { b.engine = ""; b.priority = ""; }
+ if (editing) b.id = editing.id;
+ return b;
+ }
+
+ function syncForm() {
+ const kind = form.elements.kind.value;
+ form.querySelector(".sf-engine").hidden = kind === "down";
+ form.querySelector(".sf-prio").hidden = kind === "down";
+ form.querySelector(".sf-new").hidden = !!editing;
+ try { explainEl.textContent = explain(formBooking()); } catch (e) { explainEl.textContent = ""; }
+ }
+
+ function openDialog(b, at) {
+ if (!paired && !b) { toast("pair this device to book: berth pair on the machine", "info"); return; }
+ editing = b;
+ const f = form.elements;
+ $("sched-engines").replaceChildren(...engineNames().map((n) => { const o = el("option"); o.value = n; return o; }));
+ $("sched-dlg-title").textContent = b ? "BOOKING " + b.id : "BOOK " + pick.toUpperCase();
+ const start = b ? new Date(b.start) : at || new Date(nowStep());
+ f.kind.value = b ? b.kind : "run";
+ f.who.value = b ? b.who : "";
+ f.engine.value = b ? b.engine || "" : "";
+ f.priority.value = b ? b.priority || "" : "";
+ f.start.value = localInput(start);
+ f.hours.value = b ? String(Math.round(((new Date(b.end) - start) / HOUR) * 100) / 100) : "1";
+ f.days.value = "1";
+ f.earliest.checked = false;
+ f.note.value = b ? b.note || "" : "";
+ f.kind.disabled = !!b;
+ for (const x of form.querySelectorAll("input, select")) if (x !== f.kind) x.disabled = !paired;
+ $("sched-save").textContent = b ? "MOVE" : "BOOK";
+ $("sched-save").hidden = !paired;
+ $("sched-cancel").hidden = !b || !paired;
+ errEl.textContent = "";
+ syncForm();
+ showDialog(dlg);
+ }
+
+ async function send(action, booking, extra) {
+ if (demo) { toast("demo: " + action + " " + (booking.id || booking.who) + " is not sent", "info"); dlg.close(); return; }
+ errEl.textContent = "";
+ try {
+ const res = await fetch("/v1/board/schedule", {
+ method: "POST",
+ headers: { "X-Berth": "1", "Content-Type": "application/json", accept: "application/json" },
+ body: JSON.stringify({ action, booking, ...extra }),
+ });
+ const j = await res.json().catch(() => ({}));
+ if (!res.ok) {
+ if (res.status === 401) paired = false;
+ errEl.textContent = j.message || res.statusText;
+ return;
+ }
+ const made = j.bookings || [];
+ toast(action === "book" ? "booked " + made.map((x) => x.id).join(", ") : (action === "move" ? "moved " : "cancelled ") + booking.id, "ok");
+ dlg.close();
+ await read();
+ } catch (e) {
+ errEl.textContent = e.message;
+ }
+ }
+
+ form.addEventListener("input", syncForm);
+ form.addEventListener("submit", (e) => {
+ e.preventDefault();
+ const b = formBooking();
+ if (editing) send("move", b);
+ else send("book", b, { days: Number(form.elements.days.value) || 1, earliest: form.elements.earliest.checked });
+ });
+ $("sched-cancel").addEventListener("click", () => editing && send("cancel", { id: editing.id }));
+ dlg.querySelector(".sc-close").addEventListener("click", () => dlg.close());
+ $("sb-book").addEventListener("click", () => openDialog(null));
+ if (!demo) {
+ fetch("/v1/me", { headers: { accept: "application/json" } })
+ .then((r) => r.json()).then((j) => { paired = !!j.paired; if (st) paint(); }).catch(() => {});
+ }
+
+ return {
+ show() {
+ clearInterval(timer);
+ read();
+ timer = setInterval(() => { if (!document.hidden) read(); }, POLL_MS);
+ },
+ hide() { clearInterval(timer); timer = 0; },
+ };
+}
diff --git a/internal/harbor/web/switch.js b/internal/harbor/web/switch.js
index 8d597e5a..3ccb2429 100644
--- a/internal/harbor/web/switch.js
+++ b/internal/harbor/web/switch.js
@@ -15,10 +15,12 @@ export function initSwitch({ demo }) {
const cardsRoot = $("sw-cards");
const pickRoot = $("sw-picker");
const enginesRoot = $("sw-engines");
+ const queueRoot = $("sw-queue");
const noteRoot = $("sw-note");
const meta = $("sw-meta");
const pairDlg = $("pair-dlg");
let board = null;
+ let queue = null; // the waiters in grant order, from the node's status
let paired = demo;
let timer = 0;
let pick = localStorage.getItem(PICK_KEY) || "";
@@ -26,7 +28,18 @@ export function initSwitch({ demo }) {
const errors = new Map(); // engine -> the last action's error, until the next read
const open = new Set(); // engines whose model list is unfolded
+ async function readQueue() {
+ try {
+ const res = await fetch(demo ? "demo.json" : "/v1/status", { headers: { accept: "application/json" } });
+ const j = await res.json();
+ queue = res.ok ? (j.waiting || []) : null;
+ } catch (e) {
+ queue = null;
+ }
+ }
+
async function read() {
+ await readQueue();
try {
const res = await fetch(demo ? "demo-board.json" : "/v1/board", { headers: { accept: "application/json" } });
const j = await res.json();
@@ -66,6 +79,77 @@ export function initSwitch({ demo }) {
await read();
}
+ // queueAct bumps, re-ranks or drops one waiter. It goes to berthd through
+ // the harbor only (never the gateway), and only from a paired device.
+ async function queueAct(action, lease, priority) {
+ if (demo) { toast("demo: " + action + " " + lease + " is not sent", "info"); return; }
+ const key = "queue\u0000" + lease;
+ busy.add(key);
+ errors.delete(key);
+ paint();
+ try {
+ const res = await fetch("/v1/board/queue", {
+ method: "POST",
+ headers: { "X-Berth": "1", "Content-Type": "application/json", accept: "application/json" },
+ body: JSON.stringify(priority ? { lease, action, priority } : { lease, action }),
+ });
+ if (!res.ok) {
+ const j = await res.json().catch(() => ({}));
+ errors.set(key, j.message || res.statusText);
+ if (res.status === 401) paired = false;
+ }
+ } catch (e) {
+ errors.set(key, e.message);
+ }
+ busy.delete(key);
+ await read();
+ }
+
+ function paintQueue(many) {
+ if (queue === null) { queueRoot.replaceChildren(el("div", "empty-note", "queue not read")); return; }
+ if (queue.length === 0) { queueRoot.replaceChildren(el("div", "empty-note", "nobody is waiting")); return; }
+ queueRoot.replaceChildren(...queue.map((l, i) => queueRow(l, i, many)));
+ }
+
+ function queueRow(l, i, many) {
+ const key = "queue\u0000" + l.id;
+ const who = l.group && l.group !== l.engine ? l.group + " (" + l.engine + ")" : l.engine;
+ const right = el("span", "sw-state", el("span", "sw-chip", (l.priority || "interactive").toUpperCase()));
+ if (l.training) right.append(el("span", "sw-chip sw-pinned", "TRAINING"));
+ if (many && l.card) right.prepend(el("span", "sw-chip", l.card));
+ const row = el("div", "sq sw-engine sw-waiting");
+ row.append(el("div", "top", el("span", "name", (i + 1) + ". " + who), right));
+ row.append(el("div", "meta", l.id + ", waiting " + Math.round(l.age_seconds || 0) + " s"));
+ const err = errors.get(key);
+ if (err) row.append(el("div", "meta sw-error", err));
+ if (paired && !l.training && !l.pinned && l.priority !== "hold") {
+ const actions = el("div", "sw-actions");
+ const first = el("button", "btn sw-btn", busy.has(key) ? "…" : "FIRST");
+ first.type = "button";
+ first.disabled = busy.has(key) || i === 0;
+ first.setAttribute("aria-label", "move " + who + " to the front of its class");
+ first.addEventListener("click", () => queueAct("bump", l.id));
+ const prio = el("select", "sw-prio");
+ prio.setAttribute("aria-label", "priority of " + who);
+ for (const p of ["batch", "interactive", "foreground"]) {
+ const o = el("option", null, p);
+ o.value = p;
+ o.selected = (l.priority || "interactive") === p;
+ prio.append(o);
+ }
+ prio.disabled = busy.has(key);
+ prio.addEventListener("change", () => queueAct("prio", l.id, prio.value));
+ const drop = el("button", "btn sw-btn sw-stop", "DROP");
+ drop.type = "button";
+ drop.disabled = busy.has(key);
+ drop.setAttribute("aria-label", "drop " + who + " from the queue");
+ drop.addEventListener("click", () => queueAct("drop", l.id));
+ actions.append(first, prio, drop);
+ row.append(actions);
+ }
+ return row;
+ }
+
function pinnedHolders() {
const names = new Set();
for (const c of (board && board.cards) || []) for (const h of c.holders || []) if (h.pinned) names.add(h.name);
@@ -79,6 +163,7 @@ export function initSwitch({ demo }) {
if (pick && !cards.some((c) => c.card === pick)) pick = "";
meta.textContent = board.reachable ? (board.name || "this node") : "berthd is not answering; cards unknown";
cardsRoot.replaceChildren(...cards.map(cardItem));
+ paintQueue(cards.length > 1);
paintPicker(cards);
noteRoot.replaceChildren(paired ? pairButton() : el("div", "empty-note", "Read only. Run berth pair on the machine to switch from this device."));
const shown = pick ? engines.filter((e) => (e.cards || []).includes(pick)) : engines;
diff --git a/internal/node/node.go b/internal/node/node.go
index 67c13a5e..8d4d84b5 100644
--- a/internal/node/node.go
+++ b/internal/node/node.go
@@ -255,6 +255,30 @@ func (n *Node) Stop(engine string) ([]string, error) {
return slices.Compact(ids), nil
}
+// Queue applies the operator's change to the waiter id on every card it
+// waits on (a lease on several cards waits on each under one ID), and
+// returns the change. A card that has no such waiter is skipped; any other
+// refusal is the answer.
+func (n *Node) Queue(id string, action broker.QueueAction, prio string) (string, error) {
+ var detail string
+ var first error
+ for _, c := range n.cards {
+ d, err := c.B.Queue(id, action, prio)
+ if err != nil {
+ if !broker.IsNotFound(err) {
+ return "", err
+ }
+ first = cmpErr(first, err)
+ continue
+ }
+ detail = d
+ }
+ if detail == "" {
+ return "", first
+ }
+ return detail, nil
+}
+
// Batch starts or renews engine's batch on every card (see Broker.Batch).
func (n *Node) Batch(engine string, grace, max time.Duration) {
for _, c := range n.cards {
@@ -271,6 +295,28 @@ func (n *Node) EndBatch(engine string) bool {
return ended
}
+// Yield tells engine's runs on every card that its job yielded the card for
+// d, or took it back (d <= 0), and answers the leases it told.
+func (n *Node) Yield(engine string, d time.Duration) ([]string, error) {
+ var ids []string
+ var first error
+ for _, c := range n.cards {
+ got, err := c.B.Yield(engine, d)
+ switch {
+ case err != nil && !broker.IsNotFound(err):
+ return nil, err
+ case err != nil:
+ first = cmpErr(first, err)
+ continue
+ }
+ ids = append(ids, got...)
+ }
+ if len(ids) == 0 {
+ return nil, first
+ }
+ return ids, nil
+}
+
// Unpark evicts engine's parked leases ("" = every one) on every card.
func (n *Node) Unpark(engine string) []string {
ids := []string{}
@@ -529,3 +575,11 @@ func add(a, b proto.Counters) proto.Counters {
}
return a
}
+
+// SetBookings hands every card the schedule; each keeps the bookings that
+// name it.
+func (n *Node) SetBookings(bs []proto.Booking) {
+ for _, c := range n.cards {
+ c.B.SetBookings(slices.DeleteFunc(slices.Clone(bs), func(x proto.Booking) bool { return x.Card != c.Name }))
+ }
+}
diff --git a/internal/notify/berth.ico b/internal/notify/berth.ico
new file mode 100644
index 00000000..c31f2764
Binary files /dev/null and b/internal/notify/berth.ico differ
diff --git a/internal/notify/desktop_linux.go b/internal/notify/desktop_linux.go
index 3aa88afa..f64b9509 100644
--- a/internal/notify/desktop_linux.go
+++ b/internal/notify/desktop_linux.go
@@ -4,19 +4,22 @@
package notify
import (
+ "context"
"fmt"
"os/exec"
)
-// Desktop is the desktop's notification service: notify-send, which
-// speaks the freedesktop notification interface every Linux desktop and
-// notification daemon (mako, dunst, swaync, KDE, GNOME) serves.
-func Desktop() (Sender, error) {
+// Serve shows the notes follow sends through notify-send, which speaks
+// the freedesktop notification interface every Linux desktop and
+// notification daemon (mako, dunst, swaync, KDE, GNOME) serves. harbor is
+// unused here: a Linux notification has no click target.
+func Serve(_ context.Context, _ string, follow func(Sender)) error {
bin, err := exec.LookPath("notify-send")
if err != nil {
- return nil, fmt.Errorf("notify-send is not installed (libnotify): %w", err)
+ return fmt.Errorf("notify-send is not installed (libnotify): %w", err)
}
- return notifySend(bin), nil
+ follow(notifySend(bin))
+ return nil
}
type notifySend string
diff --git a/internal/notify/desktop_other.go b/internal/notify/desktop_other.go
index 9e85eb46..15eb8a3a 100644
--- a/internal/notify/desktop_other.go
+++ b/internal/notify/desktop_other.go
@@ -1,16 +1,17 @@
// Copyright 2026 Trevin Tindall (T-Crypt)
// SPDX-License-Identifier: Apache-2.0
-//go:build !linux
+//go:build !linux && !windows
package notify
import (
+ "context"
"errors"
"runtime"
)
-// Desktop has no notification service on this system yet.
-func Desktop() (Sender, error) {
- return nil, errors.New("desktop notifications are not supported on " + runtime.GOOS + " yet")
+// Serve has no notification service on this system yet.
+func Serve(context.Context, string, func(Sender)) error {
+ return errors.New("desktop notifications are not supported on " + runtime.GOOS + " yet")
}
diff --git a/internal/notify/notify.go b/internal/notify/notify.go
index 8e71ed04..7f501ad9 100644
--- a/internal/notify/notify.go
+++ b/internal/notify/notify.go
@@ -111,3 +111,20 @@ func (g *Gate) Let(n Note, now time.Time) bool {
g.last[key] = now
return true
}
+
+// Recent is the last few notes shown, newest first: the tray's menu.
+type Recent struct {
+ Max int
+ notes []Note
+}
+
+// Add puts n first and drops the oldest past Max.
+func (r *Recent) Add(n Note) {
+ r.notes = append([]Note{n}, r.notes...)
+ if len(r.notes) > r.Max {
+ r.notes = r.notes[:r.Max]
+ }
+}
+
+// Notes are the notes, newest first.
+func (r *Recent) Notes() []Note { return r.notes }
diff --git a/internal/notify/notify_test.go b/internal/notify/notify_test.go
index 2e62d91e..72204cbd 100644
--- a/internal/notify/notify_test.go
+++ b/internal/notify/notify_test.go
@@ -85,3 +85,14 @@ func TestNotesNameTheirCardOnAMultiCardMachine(t *testing.T) {
t.Fatalf("one-card note = %+v", n)
}
}
+
+func TestRecentKeepsTheNewest(t *testing.T) {
+ r := Recent{Max: 2}
+ for _, title := range []string{"a", "b", "c"} {
+ r.Add(Note{Title: title})
+ }
+ got := r.Notes()
+ if len(got) != 2 || got[0].Title != "c" || got[1].Title != "b" {
+ t.Fatalf("recent = %+v, want c then b", got)
+ }
+}
diff --git a/internal/notify/tray_windows.go b/internal/notify/tray_windows.go
new file mode 100644
index 00000000..97e1b8c2
--- /dev/null
+++ b/internal/notify/tray_windows.go
@@ -0,0 +1,364 @@
+// Copyright 2026 Trevin Tindall (T-Crypt)
+// SPDX-License-Identifier: Apache-2.0
+
+package notify
+
+// The Windows tray: one notification-area icon owned by berth notify in
+// the user's session. Its balloons are Windows' own toasts, its tooltip is
+// the latest event, and its menu opens the harbor, mutes the toasts, lists
+// the recent events and quits. Plain Win32 through x/sys/windows: no cgo
+// and no PowerShell.
+
+import (
+ "context"
+ _ "embed"
+ "errors"
+ "fmt"
+ "os"
+ "path/filepath"
+ "runtime"
+ "strings"
+ "sync"
+ "unsafe"
+
+ "golang.org/x/sys/windows"
+)
+
+//go:embed berth.ico
+var icon []byte
+
+var (
+ user32 = windows.NewLazySystemDLL("user32.dll")
+ shell32 = windows.NewLazySystemDLL("shell32.dll")
+
+ pRegisterClassEx = user32.NewProc("RegisterClassExW")
+ pCreateWindowEx = user32.NewProc("CreateWindowExW")
+ pDefWindowProc = user32.NewProc("DefWindowProcW")
+ pDestroyWindow = user32.NewProc("DestroyWindow")
+ pGetMessage = user32.NewProc("GetMessageW")
+ pTranslateMessage = user32.NewProc("TranslateMessage")
+ pDispatchMessage = user32.NewProc("DispatchMessageW")
+ pPostMessage = user32.NewProc("PostMessageW")
+ pPostQuitMessage = user32.NewProc("PostQuitMessage")
+ pLoadImage = user32.NewProc("LoadImageW")
+ pCreatePopupMenu = user32.NewProc("CreatePopupMenu")
+ pAppendMenu = user32.NewProc("AppendMenuW")
+ pDestroyMenu = user32.NewProc("DestroyMenu")
+ pTrackPopupMenu = user32.NewProc("TrackPopupMenu")
+ pGetCursorPos = user32.NewProc("GetCursorPos")
+ pSetForegroundWindow = user32.NewProc("SetForegroundWindow")
+ pRegisterWindowMessage = user32.NewProc("RegisterWindowMessageW")
+ pShellNotifyIcon = shell32.NewProc("Shell_NotifyIconW")
+ errTrayIcon = errors.New("the notification area refused the icon")
+ taskbarCreated uintptr // explorer restarted: add the icon again
+)
+
+const (
+ wmDestroy = 0x0002
+ wmClose = 0x0010
+ wmNull = 0x0000
+ wmContextMenu = 0x007B
+ wmLButtonUp = 0x0202
+ wmApp = 0x8000
+ wmTray = wmApp + 1
+
+ ninBalloonUserClick = 0x0405
+
+ nimAdd = 0
+ nimModify = 1
+ nimDelete = 2
+ nimSetVersion = 4
+
+ nifMessage = 0x01
+ nifIcon = 0x02
+ nifTip = 0x04
+ nifInfo = 0x10
+ nifShowTip = 0x80
+ niifInfo = 0x01
+ niifError = 0x03
+ niifNoSound = 0x10
+
+ mfString = 0x0000
+ mfGrayed = 0x0001
+ mfChecked = 0x0008
+ mfPopup = 0x0010
+ mfSeparator = 0x0800
+
+ tpmRightButton = 0x0002
+ tpmReturnCmd = 0x0100
+
+ imageIcon = 1
+ lrLoadFromFile = 0x0010
+ lrDefaultSize = 0x0040
+
+ menuOpen = 1
+ menuMute = 2
+ menuQuit = 3
+)
+
+type notifyIconData struct {
+ Size uint32
+ Wnd windows.HWND
+ ID uint32
+ Flags uint32
+ CallbackMessage uint32
+ Icon windows.Handle
+ Tip [128]uint16
+ State uint32
+ StateMask uint32
+ Info [256]uint16
+ Version uint32
+ InfoTitle [64]uint16
+ InfoFlags uint32
+ GUIDItem windows.GUID
+ BalloonIcon windows.Handle
+}
+
+type wndClassEx struct {
+ Size uint32
+ Style uint32
+ WndProc uintptr
+ ClsExtra int32
+ WndExtra int32
+ Instance windows.Handle
+ Icon windows.Handle
+ Cursor windows.Handle
+ Background windows.Handle
+ MenuName *uint16
+ ClassName *uint16
+ IconSm windows.Handle
+}
+
+type msg struct {
+ Wnd windows.HWND
+ Message uint32
+ WParam uintptr
+ LParam uintptr
+ Time uint32
+ Pt struct{ X, Y int32 }
+ Private uint32
+}
+
+// tray is the icon's state. The window procedure runs on the locked
+// thread; Send runs on the follow goroutine; mu covers what both touch.
+type tray struct {
+ harbor string
+ wnd windows.HWND
+ icon windows.Handle
+
+ mu sync.Mutex
+ muted bool
+ recent Recent
+}
+
+// Serve shows the notes follow sends from a tray icon until ctx ends or
+// the person picks Quit.
+func Serve(ctx context.Context, harbor string, follow func(Sender)) error {
+ runtime.LockOSThread()
+ defer runtime.UnlockOSThread()
+ t := &tray{harbor: harbor, recent: Recent{Max: 10}}
+ if err := t.create(); err != nil {
+ return err
+ }
+ go follow(t)
+ go func() {
+ <-ctx.Done()
+ _, _, _ = pPostMessage.Call(uintptr(t.wnd), wmClose, 0, 0)
+ }()
+ var m msg
+ for {
+ r, _, err := pGetMessage.Call(uintptr(unsafe.Pointer(&m)), 0, 0, 0)
+ switch int32(r) {
+ case -1:
+ return fmt.Errorf("tray message loop: %w", err)
+ case 0:
+ return nil
+ }
+ _, _, _ = pTranslateMessage.Call(uintptr(unsafe.Pointer(&m)))
+ _, _, _ = pDispatchMessage.Call(uintptr(unsafe.Pointer(&m)))
+ }
+}
+
+func (t *tray) create() error {
+ path := filepath.Join(os.TempDir(), "berth-notify.ico")
+ if err := os.WriteFile(path, icon, 0o644); err != nil {
+ return fmt.Errorf("tray icon: %w", err)
+ }
+ h, _, err := pLoadImage.Call(0, uintptr(unsafe.Pointer(utf16p(path))),
+ imageIcon, 0, 0, lrLoadFromFile|lrDefaultSize)
+ if h == 0 {
+ return fmt.Errorf("tray icon: %w", err)
+ }
+ t.icon = windows.Handle(h)
+
+ name := utf16p("BerthNotifyTray")
+ var inst windows.Handle
+ if err := windows.GetModuleHandleEx(0, nil, &inst); err != nil {
+ return fmt.Errorf("tray window: %w", err)
+ }
+ wc := wndClassEx{WndProc: windows.NewCallback(t.proc), Instance: inst, ClassName: name}
+ wc.Size = uint32(unsafe.Sizeof(wc))
+ if r, _, err := pRegisterClassEx.Call(uintptr(unsafe.Pointer(&wc))); r == 0 {
+ return fmt.Errorf("tray window class: %w", err)
+ }
+ wnd, _, err := pCreateWindowEx.Call(0, uintptr(unsafe.Pointer(name)), uintptr(unsafe.Pointer(name)),
+ 0, 0, 0, 0, 0, 0, 0, uintptr(inst), 0)
+ if wnd == 0 {
+ return fmt.Errorf("tray window: %w", err)
+ }
+ t.wnd = windows.HWND(wnd)
+ taskbarCreated, _, _ = pRegisterWindowMessage.Call(uintptr(unsafe.Pointer(utf16p("TaskbarCreated"))))
+ return t.add()
+}
+
+// add puts the icon in the notification area, with version 4 callbacks.
+func (t *tray) add() error {
+ d := t.data()
+ d.Flags = nifMessage | nifIcon | nifTip | nifShowTip
+ d.CallbackMessage = wmTray
+ d.Icon = t.icon
+ t.mu.Lock()
+ copyUTF16(d.Tip[:], tip(t.recent.Notes()))
+ t.mu.Unlock()
+ if r, _, _ := pShellNotifyIcon.Call(nimAdd, uintptr(unsafe.Pointer(d))); r == 0 {
+ return errTrayIcon
+ }
+ d.Version = 4
+ _, _, _ = pShellNotifyIcon.Call(nimSetVersion, uintptr(unsafe.Pointer(d)))
+ return nil
+}
+
+// tip is the tooltip for the newest of notes: the latest title, named as
+// Berth's once, or "Berth" before the first note.
+func tip(notes []Note) string {
+ switch {
+ case len(notes) == 0:
+ return "Berth"
+ case strings.HasPrefix(notes[0].Title, "Berth"):
+ return notes[0].Title
+ }
+ return "Berth: " + notes[0].Title
+}
+
+func (t *tray) data() *notifyIconData {
+ d := ¬ifyIconData{Wnd: t.wnd, ID: 1}
+ d.Size = uint32(unsafe.Sizeof(*d))
+ return d
+}
+
+// Send records n, shows it as the tooltip, and as a toast unless muted.
+func (t *tray) Send(n Note) error {
+ t.mu.Lock()
+ t.recent.Add(n)
+ muted := t.muted
+ t.mu.Unlock()
+ d := t.data()
+ d.Flags = nifTip | nifShowTip
+ copyUTF16(d.Tip[:], tip([]Note{n}))
+ if !muted {
+ d.Flags |= nifInfo
+ copyUTF16(d.InfoTitle[:], n.Title)
+ copyUTF16(d.Info[:], n.Body)
+ d.InfoFlags = niifInfo | niifNoSound
+ if n.Level == Error {
+ d.InfoFlags = niifError
+ }
+ }
+ if r, _, _ := pShellNotifyIcon.Call(nimModify, uintptr(unsafe.Pointer(d))); r == 0 {
+ return errTrayIcon
+ }
+ return nil
+}
+
+func (t *tray) proc(wnd, message, wparam, lparam uintptr) uintptr {
+ switch {
+ case message == wmTray:
+ switch lparam & 0xFFFF {
+ case wmContextMenu, wmLButtonUp:
+ t.menu()
+ case ninBalloonUserClick:
+ t.open()
+ }
+ return 0
+ case message == taskbarCreated && taskbarCreated != 0:
+ _ = t.add()
+ return 0
+ case message == wmClose:
+ _, _, _ = pDestroyWindow.Call(wnd)
+ return 0
+ case message == wmDestroy:
+ _, _, _ = pShellNotifyIcon.Call(nimDelete, uintptr(unsafe.Pointer(t.data())))
+ _, _, _ = pPostQuitMessage.Call(0)
+ return 0
+ }
+ r, _, _ := pDefWindowProc.Call(wnd, message, wparam, lparam)
+ return r
+}
+
+// menu builds the popup fresh each time, so the recent events are current.
+func (t *tray) menu() {
+ t.mu.Lock()
+ muted := t.muted
+ notes := append([]Note(nil), t.recent.Notes()...)
+ t.mu.Unlock()
+
+ m, _, _ := pCreatePopupMenu.Call()
+ defer func() { _, _, _ = pDestroyMenu.Call(m) }()
+ sub, _, _ := pCreatePopupMenu.Call()
+ if len(notes) == 0 {
+ appendMenu(sub, mfString|mfGrayed, 0, "Nothing yet")
+ }
+ for _, n := range notes {
+ appendMenu(sub, mfString|mfGrayed, 0, n.Title)
+ }
+ appendMenu(m, mfString, menuOpen, "Open harbor")
+ show := uintptr(mfString | mfChecked)
+ if muted {
+ show = mfString
+ }
+ appendMenu(m, show, menuMute, "Show notifications")
+ appendMenu(m, mfPopup, sub, "Recent events")
+ appendMenu(m, mfSeparator, 0, "")
+ appendMenu(m, mfString, menuQuit, "Quit")
+
+ var pt struct{ X, Y int32 }
+ _, _, _ = pGetCursorPos.Call(uintptr(unsafe.Pointer(&pt)))
+ _, _, _ = pSetForegroundWindow.Call(uintptr(t.wnd))
+ cmd, _, _ := pTrackPopupMenu.Call(m, tpmRightButton|tpmReturnCmd, uintptr(pt.X), uintptr(pt.Y), 0, uintptr(t.wnd), 0)
+ _, _, _ = pPostMessage.Call(uintptr(t.wnd), wmNull, 0, 0)
+ switch cmd {
+ case menuOpen:
+ t.open()
+ case menuMute:
+ t.mu.Lock()
+ t.muted = !t.muted
+ t.mu.Unlock()
+ case menuQuit:
+ _, _, _ = pPostMessage.Call(uintptr(t.wnd), wmClose, 0, 0)
+ }
+}
+
+func (t *tray) open() {
+ _ = windows.ShellExecute(0, utf16p("open"), utf16p(t.harbor), nil, nil, windows.SW_SHOWNORMAL)
+}
+
+func appendMenu(m, flags, id uintptr, text string) {
+ _, _, _ = pAppendMenu.Call(m, flags, id, uintptr(unsafe.Pointer(utf16p(text))))
+}
+
+// utf16p is s as a Win32 string; the texts here hold no NUL.
+func utf16p(s string) *uint16 {
+ p, _ := windows.UTF16PtrFromString(s)
+ return p
+}
+
+// copyUTF16 writes s into a fixed Win32 text field, cut to fit with its
+// terminating zero.
+func copyUTF16(dst []uint16, s string) {
+ u, _ := windows.UTF16FromString(s)
+ if len(u) > len(dst) {
+ u = u[:len(dst)]
+ u[len(u)-1] = 0
+ }
+ copy(dst, u)
+}
diff --git a/internal/notify/tray_windows_test.go b/internal/notify/tray_windows_test.go
new file mode 100644
index 00000000..c16207ee
--- /dev/null
+++ b/internal/notify/tray_windows_test.go
@@ -0,0 +1,26 @@
+// Copyright 2026 Trevin Tindall (T-Crypt)
+// SPDX-License-Identifier: Apache-2.0
+
+//go:build windows
+
+package notify
+
+import "testing"
+
+// #186: the tooltip read "Berth: Berth refused a request", and after
+// explorer restarted it went back to plain "Berth" while the recent events
+// were still there.
+func TestTrayTip(t *testing.T) {
+ for _, tc := range []struct {
+ notes []Note
+ want string
+ }{
+ {nil, "Berth"},
+ {[]Note{{Title: "Berth refused a request"}}, "Berth refused a request"},
+ {[]Note{{Title: "gpu1: GPU passed to a VM"}, {Title: "older"}}, "Berth: gpu1: GPU passed to a VM"},
+ } {
+ if got := tip(tc.notes); got != tc.want {
+ t.Errorf("tip(%v) = %q, want %q", tc.notes, got, tc.want)
+ }
+ }
+}
diff --git a/internal/proto/booking.go b/internal/proto/booking.go
new file mode 100644
index 00000000..31b11f31
--- /dev/null
+++ b/internal/proto/booking.go
@@ -0,0 +1,137 @@
+// Copyright 2026 Trevin Tindall (T-Crypt)
+// SPDX-License-Identifier: Apache-2.0
+
+package proto
+
+import (
+ "fmt"
+ "slices"
+ "time"
+)
+
+// BookingKind is what a booking does to its card while it holds.
+type BookingKind string
+
+const (
+ // BookRun is an engine's turn on the card: its waiter goes first, it
+ // runs at the booking's priority, and its holder keeps the card against
+ // everything but a hold until the booking ends.
+ BookRun BookingKind = "run"
+ // BookPriority moves an engine to another priority class for the
+ // booking's hours. It claims no time on the card.
+ BookPriority BookingKind = "priority"
+ // BookDown is downtime: the card takes no new work, and its holders
+ // drain when it starts. A hold and a training stay.
+ BookDown BookingKind = "down"
+)
+
+// Booking is one entry on a card's schedule, [Start, End) on the node's
+// clock.
+type Booking struct {
+ ID string `json:"id" yaml:"id"`
+ Card string `json:"card" yaml:"card"`
+ Kind BookingKind `json:"kind" yaml:"kind"`
+ Who string `json:"who" yaml:"who"`
+ Engine string `json:"engine,omitempty" yaml:"engine,omitempty"`
+ Priority string `json:"priority,omitempty" yaml:"priority,omitempty"`
+ Start time.Time `json:"start" yaml:"start"`
+ End time.Time `json:"end" yaml:"end"`
+ Note string `json:"note,omitempty" yaml:"note,omitempty"`
+}
+
+// Holds is whether t falls inside the booking.
+func (b Booking) Holds(t time.Time) bool { return !t.Before(b.Start) && t.Before(b.End) }
+
+// Claims is whether the booking takes the card's time, so two of them may
+// not overlap: a run or a downtime does, a priority change does not.
+func (b Booking) Claims() bool { return b.Kind == BookRun || b.Kind == BookDown }
+
+// Clashes is whether b and o may not share any hour on one card: two that
+// claim the card, or two priority changes for the same engine.
+func (b Booking) Clashes(o Booking) bool {
+ if b.Card != o.Card || !b.Start.Before(o.End) || !o.Start.Before(b.End) {
+ return false
+ }
+ if b.Claims() && o.Claims() {
+ return true
+ }
+ return b.Kind == BookPriority && o.Kind == BookPriority && b.Engine == o.Engine
+}
+
+// MaxBookingAhead is how far ahead a booking may start, and MaxBookingLen
+// how long one may run.
+const (
+ MaxBookingAhead = 14 * 24 * time.Hour
+ MaxBookingLen = 7 * 24 * time.Hour
+)
+
+var priorities = []string{"batch", "interactive", "foreground"}
+
+// Check is what a booking must be before it goes on a schedule at now: a
+// known kind, a name for who asked, an engine for a run or a priority
+// change, a priority class where it takes one, and hours that end in the
+// future, start inside the horizon and run no longer than a week.
+func (b Booking) Check(now time.Time) error {
+ switch b.Kind {
+ case BookRun, BookPriority, BookDown:
+ default:
+ return fmt.Errorf("kind %q: use run, priority or down", b.Kind)
+ }
+ switch {
+ case b.Who == "":
+ return fmt.Errorf("name who the booking is for")
+ case b.Kind != BookDown && b.Engine == "":
+ return fmt.Errorf("a %s booking names its engine", b.Kind)
+ case b.Kind == BookPriority && b.Priority == "":
+ return fmt.Errorf("a priority booking names its priority: batch, interactive or foreground")
+ case b.Kind == BookDown && b.Priority != "":
+ return fmt.Errorf("downtime takes no priority")
+ case b.Priority != "" && !slices.Contains(priorities, b.Priority):
+ return fmt.Errorf("priority %q: use batch, interactive or foreground", b.Priority)
+ case !b.End.After(b.Start):
+ return fmt.Errorf("the booking ends before it starts")
+ case !b.End.After(now):
+ return fmt.Errorf("the booking ends in the past")
+ case b.Start.Sub(now) > MaxBookingAhead:
+ return fmt.Errorf("a booking starts at most %s ahead", MaxBookingAhead)
+ case b.End.Sub(b.Start) > MaxBookingLen:
+ return fmt.Errorf("a booking runs at most %s", MaxBookingLen)
+ }
+ return nil
+}
+
+// ScheduleBoard is the scheduler's view of a node, as berth schedule and
+// GET /v1/berth/schedule give it: what holds a card, the waiters in grant
+// order, and every booking that has not ended.
+type ScheduleBoard struct {
+ Now time.Time `json:"now" yaml:"now"`
+ Running []Lease `json:"running" yaml:"running"`
+ Queued []Lease `json:"queued" yaml:"queued"`
+ Bookings []Booking `json:"bookings" yaml:"bookings"`
+}
+
+// BoardOf is st's scheduler view at now, with empty lists, never null.
+func BoardOf(st *Status, now time.Time) ScheduleBoard {
+ b := ScheduleBoard{Now: now, Running: st.Leases, Queued: st.Waiting, Bookings: st.Bookings}
+ if b.Running == nil {
+ b.Running = []Lease{}
+ }
+ if b.Queued == nil {
+ b.Queued = []Lease{}
+ }
+ if b.Bookings == nil {
+ b.Bookings = []Booking{}
+ }
+ return b
+}
+
+// BookingAt is the booking of kind k for engine that holds at t, if any.
+// An empty engine matches only downtime, which names none.
+func BookingAt(bs []Booking, k BookingKind, engine string, t time.Time) (Booking, bool) {
+ for _, b := range bs {
+ if b.Kind == k && b.Engine == engine && b.Holds(t) {
+ return b, true
+ }
+ }
+ return Booking{}, false
+}
diff --git a/internal/proto/proto.go b/internal/proto/proto.go
index 6cbd2b93..ef9cc04f 100644
--- a/internal/proto/proto.go
+++ b/internal/proto/proto.go
@@ -68,6 +68,18 @@ const (
// defaults). With End set, end Engine's batch now. The server answers
// with the same op, or OpRefused when End finds no batch.
OpBatch = "batch"
+ // OpQueue: client: the operator changes the waiter Lease: Action bump
+ // (front of its priority class), prio (to Priority: batch, interactive
+ // or foreground) or drop (refuse it). The server answers with the same
+ // op and Reason naming the change, or OpRefused.
+ OpQueue = "queue"
+ // OpYield: client: the running job of Engine (by name or group) let go
+ // of the card from inside for Yield seconds, or took it back (Yield 0),
+ // so a drain treats the job as at its boundary while it lasts (#299).
+ // Ending a yield resumes the engine if a drain parked it. The server
+ // answers with the same op and the Leases it told, or OpRefused. The
+ // server sends the same op to each lease's runner.
+ OpYield = "yield"
// OpPark: the runner holding a draining lease asks to park its engine
// in RAM; the daemon answers OpParkOK or OpRefused with the why line.
// OpParked: the runner's park call is done; the daemon watches for the
@@ -117,20 +129,41 @@ const (
OpPairCheck = "pair_check"
OpPairList = "pair_list"
OpPairRevoke = "pair_revoke"
+
+ // The card's schedule; berthd owns the file. OpBook puts Booking on
+ // the schedule, Days times on consecutive days (0 or 1 = once), and
+ // with Earliest moves it to the first free hours at or after its Start
+ // with the same length; it answers the same op with the Bookings made.
+ // OpUnbook takes Booking.ID off; OpRebook replaces the booking with
+ // Booking.ID by Booking. Both answer the same op with the Bookings
+ // changed. Any refusal is OpRefused with the reason.
+ OpBook = "book"
+ OpUnbook = "unbook"
+ OpRebook = "rebook"
)
// Msg is every message in both directions; unused fields are omitted.
type Msg struct {
- Op string `json:"op"`
- Engine string `json:"engine,omitempty"`
- Group string `json:"group,omitempty"` // the engine family the lease counts for in juggles and usage
- Priority string `json:"priority,omitempty"`
- Needs uint64 `json:"needs,omitempty"` // bytes; 0 = the whole card
+ Op string `json:"op"`
+ // Booking, Bookings, Days and Earliest belong to OpBook, OpUnbook and
+ // OpRebook.
+ Booking *Booking `json:"booking,omitempty"`
+ Bookings []Booking `json:"bookings,omitempty"`
+ Days int `json:"days,omitempty"`
+ Earliest bool `json:"earliest,omitempty"`
+ Engine string `json:"engine,omitempty"`
+ Group string `json:"group,omitempty"` // the engine family the lease counts for in juggles and usage
+ Priority string `json:"priority,omitempty"`
+ Needs uint64 `json:"needs,omitempty"` // bytes; 0 = the whole card
// NeedsEach is the size in bytes on each card Card names, in that
// order, when the cards differ. Needs is then the largest, so a berthd
// that predates NeedsEach reserves too much on a small card, never too
// little.
NeedsEach []uint64 `json:"needs_each,omitempty"`
+ // LearnSize asks berthd to size an OpAcquire or OpReattach with no
+ // Needs from the engine's measured peak. A berthd that predates it
+ // grants the whole card, as it would without.
+ LearnSize bool `json:"learn_size,omitempty"`
// Card is the card an OpAcquire or OpReattach is for: its name on the
// node (gpu1) or its PCI address, or a comma list of them for one lease
// on several cards. "" is the node's first card.
@@ -143,6 +176,7 @@ type Msg struct {
PID int `json:"pid,omitempty"`
PIDStart uint64 `json:"pid_start,omitempty"` // OpReattach: the engine's start time, so a reused PID is refused
Pinned bool `json:"pinned,omitempty"` // OpAcquire/OpReattach: never drained (holds, berth run --pinned)
+ Training bool `json:"training,omitempty"` // OpAcquire/OpReattach: berth train, locks the card
Parked bool `json:"parked,omitempty"` // OpReattach: the engine's weights are parked in RAM; its lease comes back parked
// Lend marks a run whose engine has lend calls (its entry says
// lend = "hook"): OpAcquire and OpReattach declare it, and the broker
@@ -158,10 +192,16 @@ type Msg struct {
// younger than it after the grant, no waiter drains it but a hold.
// OpAcquire and OpReattach; 0 = off.
MinTurn float64 `json:"min_turn,omitempty"`
+ // Held is how long, in seconds, an OpReattach's engine had held the card
+ // since its last grant or resume, so the new daemon goes on with that
+ // turn instead of starting a fresh one.
+ Held float64 `json:"held,omitempty"`
// Max and End are an OpBatch's cap, in seconds, and its end request;
// its idle grace travels in Grace.
Max float64 `json:"max,omitempty"`
End bool `json:"end,omitempty"`
+ // Yield is an OpYield's length in seconds; 0 ends the yield.
+ Yield float64 `json:"yield,omitempty"`
// ParkTTL is how long the lease may stay parked, in seconds, in place
// of the node's [park] ttl. OpAcquire and OpReattach; 0 = the node's.
ParkTTL float64 `json:"park_ttl,omitempty"`
@@ -180,6 +220,8 @@ type Msg struct {
// daemon with it, so the daemon still reads it; nothing sends it now.
Container string `json:"container,omitempty"`
Lease string `json:"lease,omitempty"`
+ // OpQueue: Action is bump, prio or drop; prio reads Priority.
+ Action string `json:"action,omitempty"`
// OpStop: Leases is the reply, the leases the daemon asked to drain.
// OpUnpark, OpParkMode: the leases evicted.
Leases []string `json:"leases,omitempty"`
@@ -205,7 +247,8 @@ type Msg struct {
Holders []Holder `json:"holders,omitempty"`
// Resume is set on an OpGranted that hands a parked lease the card
// back, so the runner knows to call the adapter's resume instead of
- // starting anything.
+ // starting anything. On the OpYield reply to an end, it says the end
+ // queued a parked engine, whose lease.resumed is still to come.
Resume bool `json:"resume,omitempty"`
// Lending (the OpLend / OpLent / OpLendFailed / OpReturn* ops). Needs
// carries the bytes to give on OpLend and what the engine gave on OpLent;
@@ -266,7 +309,10 @@ const (
EvStuck = "lease.stuck"
EvRefused = "lease.refused"
EvReattached = "lease.reattached"
- EvOrphaned = "lease.orphaned" // the attached process is gone and the broker released the lease itself
+ EvOrphaned = "lease.orphaned" // the attached process is gone and the broker released the lease itself
+ EvQueueOverride = "lease.queue_override" // the operator bumped, re-ranked or dropped a waiter (berth queue)
+ EvBooked = "schedule.booked" // a booking went on, came off or changed on the card's schedule
+ EvBookingPrio = "lease.booking_priority" // a booking moved a lease to another priority class, or its end moved it back
EvSquatter = "squatter.appeared"
EvSquatterGone = "squatter.gone"
EvProbeFailed = "probe.failed"
@@ -297,6 +343,9 @@ const (
// host driver. Each fires once per transition.
EvCardPassthrough = "card.passthrough"
EvCardReturned = "card.returned"
+ // EvNodeStarted: berthd started. Its detail is the version. The events
+ // before it are the last daemon's, read back from the event file.
+ EvNodeStarted = "node.started"
)
// Card states, the values Status.Passthrough.State takes. Normal is the
@@ -362,6 +411,12 @@ type Status struct {
Priority *PriorityStatus `json:"priority,omitempty"`
Claims []Claim `json:"claims,omitempty"`
Footprints []EngineFootprint `json:"footprints,omitempty"`
+ // Schedules are the engines held to hours, by engine name: each
+ // configured window, and a lease's own (berth run --window) in place of
+ // its engine's, whether or not anything runs now.
+ Schedules []Schedule `json:"schedules,omitempty"`
+ // Bookings are every card's schedule that has not ended, by start.
+ Bookings []Booking `json:"bookings,omitempty"`
// ParkedBytes is the RAM the parked engines hold: each one's Berth
// scope's anon plus shmem, or its VRAM footprint where Berth reads no
// scope. A reading, never a reservation.
@@ -395,6 +450,18 @@ type EngineFootprint struct {
Bytes uint64 `json:"estimated_vram_bytes"`
Runs int `json:"runs"`
At time.Time `json:"at"`
+ // HotPeakBytes is the most VRAM a run under this exact name held, the
+ // figure learn_size sizes a lease from.
+ HotPeakBytes uint64 `json:"hot_peak_vram_bytes,omitempty"`
+ // RAMPeakBytes is the most host RAM a run held while active.
+ RAMPeakBytes uint64 `json:"ram_peak_bytes,omitempty"`
+ // ParkedRAMBytes is the host RAM the engine last held parked.
+ ParkedRAMBytes uint64 `json:"parked_ram_bytes,omitempty"`
+ // Priority and MinTurnS are what the last run under this exact name
+ // asked for: its berth run line's --priority and --min-turn. Priority
+ // is "" until a run of the name is measured.
+ Priority string `json:"priority,omitempty"`
+ MinTurnS float64 `json:"min_turn_s,omitempty"`
}
// Claim is one claim rule and what it matches right now: work Berth does not
@@ -564,6 +631,9 @@ type Lease struct {
Contested uint64 `json:"contested,omitempty"`
Age float64 `json:"age_seconds"`
Pinned bool `json:"pinned,omitempty"` // never drained: a hold, a claim, berth run --pinned
+ // Training is a berth train lease: it locks its card, and every other
+ // request there is refused while it holds or waits.
+ Training bool `json:"training,omitempty"`
// ParkedSince is how long the engine has been parked, in seconds; 0
// when it is not parked.
ParkedSince float64 `json:"parked_since,omitempty"`
@@ -577,6 +647,21 @@ type Lease struct {
// expects the waiting engine to start, from the measured starts in
// its queue chain only; 0 = unknown (no book, or nothing measured).
EstimateS float64 `json:"estimate_start_s,omitempty"`
+ // Window is the hours the broker holds the lease to, "23:00-07:00";
+ // "" = none.
+ Window string `json:"window,omitempty"`
+ // Booking is the ID of the booking that sets Priority now; "" = the
+ // lease's own class.
+ Booking string `json:"booking,omitempty"`
+}
+
+// Holder names the lease for people: its engine, or "group (engine)" for a
+// model under a router (llama-swap (qwen)).
+func (l Lease) Holder() string {
+ if l.Group != "" && l.Group != l.Engine {
+ return l.Group + " (" + l.Engine + ")"
+ }
+ return l.Engine
}
type Squatter struct {
diff --git a/internal/proto/window.go b/internal/proto/window.go
index 47042244..a00cfaf4 100644
--- a/internal/proto/window.go
+++ b/internal/proto/window.go
@@ -89,6 +89,37 @@ func (w Window) Opens(t time.Time) time.Duration {
return time.Duration(left)*time.Minute - time.Duration(t.Second())*time.Second - time.Duration(t.Nanosecond())*time.Nanosecond
}
+// Closes is how long a lease inside w keeps its hours: 0 outside, and with
+// no window, which never closes.
+func (w Window) Closes(t time.Time) time.Duration {
+ if !w.Set() || !w.Contains(t) {
+ return 0
+ }
+ left := (w.End - minuteOf(t) + minutesPerDay) % minutesPerDay
+ return time.Duration(left)*time.Minute - time.Duration(t.Second())*time.Second - time.Duration(t.Nanosecond())*time.Nanosecond
+}
+
+// Schedule is one engine's hours as the node's clock reads them now.
+type Schedule struct {
+ Engine string `json:"engine"`
+ Window string `json:"window"`
+ Open bool `json:"open"`
+ // ChangeS is the seconds until the window closes when Open, else until
+ // it opens.
+ ChangeS float64 `json:"change_s"`
+}
+
+// ScheduleAt reads w for engine at t.
+func ScheduleAt(engine string, w Window, t time.Time) Schedule {
+ s := Schedule{Engine: engine, Window: w.String(), Open: w.Contains(t)}
+ if s.Open {
+ s.ChangeS = w.Closes(t).Seconds()
+ } else {
+ s.ChangeS = w.Opens(t).Seconds()
+ }
+ return s
+}
+
func minuteOf(t time.Time) int { return t.Hour()*60 + t.Minute() }
// String is the window as a config file writes it, "23:00-07:00". The zero
diff --git a/internal/proto/window_test.go b/internal/proto/window_test.go
index 22a889cf..5affd698 100644
--- a/internal/proto/window_test.go
+++ b/internal/proto/window_test.go
@@ -83,3 +83,39 @@ func TestWindowContainsAndOpens(t *testing.T) {
}
}
}
+
+func TestWindowCloses(t *testing.T) {
+ at := func(h, m int) time.Time { return time.Date(2026, 11, 2, h, m, 0, 0, zone) }
+ night := Window{Start: 23 * 60, End: 7 * 60}
+ work := Window{Start: 9*60 + 30, End: 17 * 60}
+ for _, tc := range []struct {
+ name string
+ w Window
+ at time.Time
+ closes time.Duration
+ }{
+ {"night at its start", night, at(23, 0), 8 * time.Hour},
+ {"night past midnight", night, at(0, 30), 6*time.Hour + 30*time.Minute},
+ {"night half a minute before its end", night, at(6, 59).Add(30 * time.Second), 30 * time.Second},
+ {"night at its end", night, at(7, 0), 0},
+ {"work at noon", work, at(12, 0), 5 * time.Hour},
+ {"work the evening after", work, at(18, 0), 0},
+ {"no window", Window{}, at(12, 0), 0},
+ } {
+ if got := tc.w.Closes(tc.at); got != tc.closes {
+ t.Errorf("%s: Closes = %v, want %v", tc.name, got, tc.closes)
+ }
+ }
+}
+
+func TestScheduleAt(t *testing.T) {
+ night := Window{Start: 23 * 60, End: 7 * 60}
+ open := ScheduleAt("agent", night, time.Date(2026, 11, 2, 1, 0, 0, 0, zone))
+ if want := (Schedule{Engine: "agent", Window: "23:00-07:00", Open: true, ChangeS: 6 * 3600}); open != want {
+ t.Errorf("inside: %+v, want %+v", open, want)
+ }
+ shut := ScheduleAt("agent", night, time.Date(2026, 11, 2, 20, 0, 0, 0, zone))
+ if want := (Schedule{Engine: "agent", Window: "23:00-07:00", Open: false, ChangeS: 3 * 3600}); shut != want {
+ t.Errorf("outside: %+v, want %+v", shut, want)
+ }
+}
diff --git a/internal/runner/card_test.go b/internal/runner/card_test.go
index c5adb202..b7ec0bb5 100644
--- a/internal/runner/card_test.go
+++ b/internal/runner/card_test.go
@@ -18,6 +18,7 @@ import (
"time"
"github.com/T-Crypt/berth/internal/broker"
+ "github.com/T-Crypt/berth/internal/config"
"github.com/T-Crypt/berth/internal/node"
"github.com/T-Crypt/berth/internal/procs"
"github.com/T-Crypt/berth/internal/proto"
@@ -230,6 +231,15 @@ func TestRunSizesEachCard(t *testing.T) {
}
}
+// learn_size on the entry reaches berthd on the acquire's re-attach too, so a
+// restarted daemon sizes the lease the same way.
+func TestReattachCarriesLearnSize(t *testing.T) {
+ o := OptionsFor(&config.Engine{Name: "llama", LearnSize: true}, Over{})
+ if m := reattachMsg(o, 100, 7, &containerRef{}, false, newLine(nil)); !m.LearnSize || m.Needs != 0 {
+ t.Errorf("learn_size %v, needs %d; want true and 0", m.LearnSize, m.Needs)
+ }
+}
+
// A re-attach carries the per-card sizes too, with the largest in Needs for
// a berthd that reads only that.
func TestReattachCarriesEachCardsSize(t *testing.T) {
diff --git a/internal/runner/comfy_session.go b/internal/runner/comfy_session.go
index fced5c05..a2f07f43 100644
--- a/internal/runner/comfy_session.go
+++ b/internal/runner/comfy_session.go
@@ -54,6 +54,10 @@ func (session *comfySession) wait(o Options, done <-chan struct{}, say func(stri
}
}
for {
+ if o.yielded != nil && o.yielded() {
+ say("ComfyUI yielded the card from inside its job; handing off")
+ return true
+ }
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
running, pending, err := comfy.Queue(httpauth.WithBearer(ctx, o.APIKey), o.DrainComfyUI)
cancel()
diff --git a/internal/runner/exclusive.go b/internal/runner/exclusive.go
new file mode 100644
index 00000000..85b8f990
--- /dev/null
+++ b/internal/runner/exclusive.go
@@ -0,0 +1,50 @@
+// Copyright 2026 Trevin Tindall (T-Crypt)
+// SPDX-License-Identifier: Apache-2.0
+
+package runner
+
+import (
+ "errors"
+ "fmt"
+ "runtime"
+ "strings"
+ "time"
+)
+
+// exclusiveTimeout bounds each nvidia-smi call the exclusive mode makes.
+const exclusiveTimeout = 30 * time.Second
+
+// exclusive puts the granted cards (their UUIDs, comma separated) in the
+// driver's exclusive-process compute mode, so the run's process is the only
+// one that can open a CUDA context there, and returns the func that puts
+// them back to the default mode. It refuses when another compute process is
+// already on the cards: in exclusive mode that process would lock the run
+// itself out. Setting the mode needs root, through `sudo -n`, so a missing
+// sudo rule refuses the run instead of starting it without the guard.
+func exclusive(o Options, uuids string) (func(func(string, ...any)), error) {
+ if runtime.GOOS != "linux" {
+ return nil, errors.New("exclusive compute mode is Linux only")
+ }
+ if uuids == "" {
+ return nil, errors.New("the granted card has no NVIDIA UUID")
+ }
+ out, err := o.CLI([]string{"nvidia-smi", "-i", uuids, "--query-compute-apps=pid,process_name", "--format=csv,noheader"}, exclusiveTimeout)
+ if err != nil {
+ return nil, fmt.Errorf("nvidia-smi could not list the compute processes: %v", err)
+ }
+ if apps := strings.TrimSpace(out); apps != "" {
+ return nil, fmt.Errorf("another compute process is on the card (%s); exclusive mode would lock the training out", strings.ReplaceAll(apps, "\n", "; "))
+ }
+ set := func(mode string) error {
+ _, err := o.CLI([]string{"sudo", "-n", "nvidia-smi", "-i", uuids, "-c", mode}, exclusiveTimeout)
+ return err
+ }
+ if err := set("EXCLUSIVE_PROCESS"); err != nil {
+ return nil, fmt.Errorf("could not set exclusive-process mode (it needs a sudo rule that runs nvidia-smi -c without a password): %v", err)
+ }
+ return func(say func(string, ...any)) {
+ if err := set("DEFAULT"); err != nil {
+ say("could not put %s back to the default compute mode: %v; run: sudo nvidia-smi -i %s -c DEFAULT", uuids, err, uuids)
+ }
+ }, nil
+}
diff --git a/internal/runner/exclusive_linux_test.go b/internal/runner/exclusive_linux_test.go
new file mode 100644
index 00000000..818420a3
--- /dev/null
+++ b/internal/runner/exclusive_linux_test.go
@@ -0,0 +1,84 @@
+// Copyright 2026 Trevin Tindall (T-Crypt)
+// SPDX-License-Identifier: Apache-2.0
+
+package runner
+
+import (
+ "errors"
+ "slices"
+ "strings"
+ "testing"
+ "time"
+)
+
+// fakeSMI answers the exclusive mode's nvidia-smi calls: apps is what the
+// compute-app query prints, sudoErr fails the mode change. It records every
+// argv.
+type fakeSMI struct {
+ apps string
+ sudoErr error
+ calls [][]string
+}
+
+func (f *fakeSMI) cli(argv []string, _ time.Duration) (string, error) {
+ f.calls = append(f.calls, argv)
+ if argv[0] == "sudo" {
+ return "", f.sudoErr
+ }
+ return f.apps, nil
+}
+
+// On an empty card the mode goes to exclusive-process before the run, and
+// back to the default when it ends, on the granted cards only.
+func TestExclusiveSetsAndRestoresTheMode(t *testing.T) {
+ f := &fakeSMI{}
+ restore, err := exclusive(Options{CLI: f.cli}, "GPU-a,GPU-b")
+ if err != nil {
+ t.Fatal(err)
+ }
+ set := []string{"sudo", "-n", "nvidia-smi", "-i", "GPU-a,GPU-b", "-c", "EXCLUSIVE_PROCESS"}
+ if len(f.calls) != 2 || !slices.Equal(f.calls[1], set) {
+ t.Fatalf("calls = %q, want the query then %q", f.calls, set)
+ }
+ restore(func(string, ...any) {})
+ back := []string{"sudo", "-n", "nvidia-smi", "-i", "GPU-a,GPU-b", "-c", "DEFAULT"}
+ if len(f.calls) != 3 || !slices.Equal(f.calls[2], back) {
+ t.Fatalf("calls = %q, want %q last", f.calls, back)
+ }
+}
+
+// A compute process already on the card would own its one context and lock
+// the training out, so the run is refused and the mode is left alone.
+func TestExclusiveRefusesWhenAProcessIsOnTheCard(t *testing.T) {
+ f := &fakeSMI{apps: "4242, python3\n"}
+ _, err := exclusive(Options{CLI: f.cli}, "GPU-a")
+ if err == nil || !strings.Contains(err.Error(), "4242, python3") {
+ t.Fatalf("err = %v, want a refusal naming the process", err)
+ }
+ for _, c := range f.calls {
+ if c[0] == "sudo" {
+ t.Fatalf("the mode was changed beside a running process: %q", c)
+ }
+ }
+}
+
+// Without a sudo rule the run is refused instead of starting without the
+// guard the operator asked for.
+func TestExclusiveRefusesWithoutSudo(t *testing.T) {
+ f := &fakeSMI{sudoErr: errors.New("a password is required")}
+ _, err := exclusive(Options{CLI: f.cli}, "GPU-a")
+ if err == nil || !strings.Contains(err.Error(), "sudo rule") {
+ t.Fatalf("err = %v, want it to name the sudo rule", err)
+ }
+}
+
+// A card with no NVIDIA UUID (an AMD card) has no exclusive mode to set.
+func TestExclusiveNeedsAnNVIDIACard(t *testing.T) {
+ f := &fakeSMI{}
+ if _, err := exclusive(Options{CLI: f.cli}, ""); err == nil {
+ t.Fatal("exclusive mode set on a card with no UUID")
+ }
+ if len(f.calls) != 0 {
+ t.Errorf("calls = %q, want none", f.calls)
+ }
+}
diff --git a/internal/runner/options.go b/internal/runner/options.go
index 1070c64c..23c5257a 100644
--- a/internal/runner/options.go
+++ b/internal/runner/options.go
@@ -24,6 +24,8 @@ type Over struct {
Front string
Group string
Pinned bool
+ Training bool
+ Exclusive bool
Switched bool
Wait time.Duration
Grace time.Duration
@@ -61,7 +63,7 @@ func OptionsFor(e *config.Engine, o Over) Options {
}
opts := Options{
Socket: o.Socket, Engine: e.Name, Card: e.GPU, Group: o.Group, Priority: prio,
- Needs: e.NeedsMiB.Bytes(), Pinned: o.Pinned, Switched: o.Switched, Wait: o.Wait,
+ Needs: e.NeedsMiB.Bytes(), Pinned: o.Pinned, Training: o.Training, Exclusive: o.Exclusive, Switched: o.Switched, Wait: o.Wait,
DrainPost: e.DrainPost, DrainSlots: e.DrainSlots, DrainComfyUI: e.DrainComfyUI,
DrainMax: e.DrainMax.Duration, MinTurn: e.MinTurn.Duration, ParkTTL: e.ParkTTL.Duration, Window: e.Window, Container: e.Container, Compose: e.Compose,
APIKeyEnv: e.APIKeyEnv, APIKeyFile: e.APIKeyFile,
@@ -115,7 +117,7 @@ func OptionsFor(e *config.Engine, o Over) Options {
// lend = "hook" gives the run its engine's lend calls. berthd asks for a
// lend only from a run whose entry has them: the protocol base is
// lend_url, and a kind with built-in calls (strata) uses its own URL.
- opts.WholeMachine = e.WholeMachine
+ opts.WholeMachine, opts.LearnSize = e.WholeMachine, e.LearnSize
if e.Lend == "hook" {
if adapter, ok := lend.For(e.Type, e.URL, e.LendURL); ok {
opts.Lend = &adapter
diff --git a/internal/runner/park.go b/internal/runner/park.go
index 56873233..430fcfd0 100644
--- a/internal/runner/park.go
+++ b/internal/runner/park.go
@@ -58,6 +58,65 @@ type leaseLine struct {
// the engine, so the watch never takes back a ticket the run is about to
// record.
endWatch context.CancelFunc
+ // yieldUntil is when a yield from inside a ComfyUI job ends: until then
+ // a drain treats the job as at its boundary (#299). A deadline and not a
+ // flag, so a node that dies mid-yield cannot leave it on.
+ yieldUntil time.Time
+ // turn is when the lease last got the card, at its grant or a resume. A
+ // re-attach tells the new daemon how long ago that was.
+ turn time.Time
+ // waking is open while a resume loads the engine back and closes when
+ // it ends; nil when none runs. A drain waits on it (#275).
+ waking chan struct{}
+}
+
+// awaitWake holds a drain until a resume in flight has ended, so the park
+// or stop never reaches an engine in the middle of its load.
+func (l *leaseLine) awaitWake(engine string, done <-chan struct{}, say func(string, ...any)) {
+ l.mu.Lock()
+ w := l.waking
+ l.mu.Unlock()
+ if w == nil {
+ return
+ }
+ say("waiting for %s to finish its resume before the drain", engine)
+ select {
+ case <-w:
+ case <-done:
+ }
+}
+
+// startTurn records that the lease got the card now.
+func (l *leaseLine) startTurn() {
+ l.mu.Lock()
+ defer l.mu.Unlock()
+ l.turn = time.Now()
+}
+
+// held is how long the lease has had the card in this turn, 0 before any.
+func (l *leaseLine) held() time.Duration {
+ l.mu.Lock()
+ defer l.mu.Unlock()
+ if l.turn.IsZero() {
+ return 0
+ }
+ return time.Since(l.turn)
+}
+
+// setYield starts a yield that lasts d, or ends it when d is not positive.
+func (l *leaseLine) setYield(d time.Duration) {
+ l.mu.Lock()
+ defer l.mu.Unlock()
+ l.yieldUntil = time.Time{}
+ if d > 0 {
+ l.yieldUntil = time.Now().Add(d)
+ }
+}
+
+func (l *leaseLine) yielding() bool {
+ l.mu.Lock()
+ defer l.mu.Unlock()
+ return time.Now().Before(l.yieldUntil)
}
func newLine(conn *proto.Conn) *leaseLine {
@@ -233,14 +292,26 @@ func (l *leaseLine) park(o Options, say func(string, ...any)) bool {
// resume drives the wake-up the runner hears as a grant with Resume set:
// the adapter's call, the readiness probe, the engine serving again. A
// wake that never answers stops the engine, which hands the card back:
-// a wedged engine must not keep the lease.
-func (l *leaseLine) resume(o Options, cmd *exec.Cmd, done <-chan struct{}, say func(string, ...any)) {
- if !l.isParked() {
- return
- }
+// a wedged engine must not keep the lease. beginWake starts it, on the
+// goroutine that reads berthd's messages, so a drain right behind the
+// grant already sees the wake; resume then runs it and ends it.
+func (l *leaseLine) beginWake() (chan struct{}, bool) {
l.mu.Lock()
- l.parked = false
- l.mu.Unlock()
+ defer l.mu.Unlock()
+ if !l.parked {
+ return nil, false
+ }
+ l.parked, l.waking = false, make(chan struct{})
+ return l.waking, true
+}
+
+func (l *leaseLine) resume(o Options, cmd *exec.Cmd, done <-chan struct{}, waking chan struct{}, say func(string, ...any)) {
+ defer func() {
+ l.mu.Lock()
+ l.waking = nil
+ l.mu.Unlock()
+ close(waking)
+ }()
start := time.Now()
adapter, hasWake := l.adapterFor(o)
if hasWake && adapter.Resume != nil {
diff --git a/internal/runner/park_test.go b/internal/runner/park_test.go
index a891bb15..c34c300b 100644
--- a/internal/runner/park_test.go
+++ b/internal/runner/park_test.go
@@ -147,6 +147,9 @@ type parkStub struct {
resumes int
wakeDelay time.Duration
failSleep bool
+ // waking is set while a /wake_up call runs; a /sleep that lands then
+ // answers 409, as an engine still loading does (#275).
+ waking bool
}
func (s *parkStub) count(what string) int {
@@ -171,7 +174,12 @@ func (s *parkStub) handler() http.Handler {
s.mu.Lock()
s.sleeps++
fail := s.failSleep
+ busy := s.waking
s.mu.Unlock()
+ if busy {
+ http.Error(w, "a load is running", http.StatusConflict)
+ return
+ }
if fail {
http.Error(w, "sleep mode is off", http.StatusInternalServerError)
return
@@ -182,8 +190,12 @@ func (s *parkStub) handler() http.Handler {
s.mu.Lock()
delay := s.wakeDelay
s.wakes++
+ s.waking = true
s.mu.Unlock()
time.Sleep(delay)
+ s.mu.Lock()
+ s.waking = false
+ s.mu.Unlock()
fmt.Fprint(w, `{}`)
})
mux.HandleFunc("/berth/park", func(w http.ResponseWriter, r *http.Request) {
@@ -667,3 +679,73 @@ func TestParkedEngineReattachesParked(t *testing.T) {
t.Fatalf("the parked engine was drained after the restart; log:\n%s", log.String())
}
}
+
+// A run whose berthd restarts re-attaches inside the turn it was in: the
+// new daemon counts the minimum turn from the old grant, so a deploy does
+// not give every holder a fresh turn.
+func TestReattachGoesOnWithTheTurn(t *testing.T) {
+ card := &parkCard{total: 24 << 30}
+ sock := filepath.Join(t.TempDir(), "berth.sock")
+ b, stop := parkDaemonAt(t, sock, card, 16<<30)
+ log := &syncBuf{}
+ exited := make(chan int, 1)
+ go func() {
+ exited <- Run(Options{Socket: sock, Engine: "vllm", Priority: "interactive", Needs: []uint64{8 << 30},
+ MinTurn: time.Hour, Grace: 200 * time.Millisecond, Argv: []string{"sleep", "30"}, Stderr: log})
+ }()
+ pid := attachAndMeasure(t, b, card)
+ stopAtEnd(t, pid, exited)
+
+ time.Sleep(2 * time.Second)
+ stop()
+ b2, _ := parkDaemonAt(t, sock, card, 16<<30)
+ waitFor(t, "the re-attach", func() bool { return strings.Contains(log.String(), "re-attached vllm") })
+ st := b2.Status().Leases
+ if len(st) != 1 {
+ t.Fatalf("leases after the restart = %+v", st)
+ }
+ if left := time.Duration(st[0].TurnS * float64(time.Second)); left > time.Hour-2*time.Second {
+ t.Fatalf("turn left after the restart = %s, want the 2s already held taken off the hour", left)
+ }
+}
+
+// #275: a drain that lands while a resume is still loading the engine waits
+// for the load, then parks. Before, the park call reached the engine
+// mid-load, the engine answered 409, and the runner stopped it.
+func TestDrainWaitsForAResumeInFlight(t *testing.T) {
+ card := &parkCard{total: 24 << 30}
+ sock, b := parkDaemon(t, card, 16<<30)
+ stub := &parkStub{wakeDelay: 1500 * time.Millisecond}
+ srv := httptest.NewServer(stub.handler())
+ defer srv.Close()
+ exited, log := runPark(t, sock, "vllm", srv.URL, false)
+ pid := attachAndMeasure(t, b, card)
+ stopAtEnd(t, pid, exited)
+
+ holder := acquireIn(b, broker.Request{Engine: "game", Priority: broker.Foreground})
+ waitFor(t, "the park call", func() bool { return stub.count("sleep") == 1 })
+ card.set(gpu.Proc{PID: pid, Name: "sleep", Used: 512 << 20})
+ waitFor(t, "parked", func() bool { return b.Status().Leases[0].State == "parked" })
+ (<-holder).Release()
+ if err := b.Resume("vllm", false); err != nil {
+ t.Fatal(err)
+ }
+ waitFor(t, "the wake call", func() bool { return stub.count("wake") == 1 })
+ card.set(gpu.Proc{PID: pid, Name: "sleep", Used: 8 << 30})
+
+ second := acquireIn(b, broker.Request{Engine: "game2", Priority: broker.Foreground})
+ waitFor(t, "the second park call", func() bool { return stub.count("sleep") == 2 })
+ card.set(gpu.Proc{PID: pid, Name: "sleep", Used: 512 << 20})
+ if hold := <-second; hold != nil {
+ hold.Release()
+ }
+ waitFor(t, "parked again", func() bool { return b.Status().Leases[0].State == "parked" })
+ select {
+ case code := <-exited:
+ t.Fatalf("the engine was stopped by a drain during its resume (exit %d); log:\n%s", code, log.String())
+ default:
+ }
+ if out := log.String(); strings.Contains(out, "409") {
+ t.Fatalf("the park reached the engine mid-load; log:\n%s", out)
+ }
+}
diff --git a/internal/runner/runner.go b/internal/runner/runner.go
index 7c175cab..0876b839 100644
--- a/internal/runner/runner.go
+++ b/internal/runner/runner.go
@@ -50,9 +50,17 @@ type Options struct {
Priority string
// Needs is the card memory in bytes, one value per card in Card's
// order; one value sizes every card, none takes the whole card.
- Needs []uint64
- Pinned bool // never drained: an app that must not be stopped (Blender, a capture)
- Switched bool // a person's switch (the wake page's Launch, /v1/berth/switch): wins a juggle
+ Needs []uint64
+ // LearnSize: with no Needs, berthd sizes the lease from the most this
+ // engine has been measured holding, not the whole card.
+ LearnSize bool
+ Pinned bool // never drained: an app that must not be stopped (Blender, a capture)
+ Training bool // berth train: a pinned hold that locks its cards; berthd refuses every other request there
+ // Exclusive: berth train --exclusive puts the granted cards in the
+ // driver's exclusive-process compute mode for the run's life, so no
+ // program outside Berth can open them either.
+ Exclusive bool
+ Switched bool // a person's switch (the wake page's Launch, /v1/berth/switch): wins a juggle
// MinTurn is a minimum turn: while the lease is younger than it after
// the grant, no request of equal or lower priority drains it, and
// neither does a higher one; only a hold (a game, a call, berth hold)
@@ -79,6 +87,9 @@ type Options struct {
// has nothing running or pending before any stop step, so a job is
// never cut off. There is no time limit; a job ends when it ends.
DrainComfyUI string
+ // yielded reports a yield from inside the running ComfyUI job, which a
+ // drain's boundary wait counts as the job's end. drain sets it.
+ yielded func() bool
// Session release is opt-in and uses the same drain ticket as daemon requests.
ExitComfyUIIdle bool
ExitComfyUIIdleAfter time.Duration
@@ -221,6 +232,12 @@ func Run(o Options) int {
say("berthd unreachable (%v); not starting %s", err, o.Engine)
return ExitRefused
}
+ // A training that lost berthd still owns the card: nothing starts
+ // beside it unprotected.
+ if t, ok := runs.Training(runs.Dir(o.Socket), procs.System{}.StartTime); ok {
+ say("berthd unreachable (%v) and %s is training on this node; not starting %s", err, t.Engine, o.Engine)
+ return ExitRefused
+ }
// The engine starts unleased and the line below dials on, so a
// daemon that comes up later (berthd starting beside this service
// at boot) re-attaches it the way it would a run it lost.
@@ -228,13 +245,25 @@ func Run(o Options) int {
conn, raw = nil, nil
}
+ if o.CLI == nil {
+ o.CLI = execCLI
+ }
var cardEnv string
if conn != nil {
defer raw.Close()
- var ok bool
- if cardEnv, ok = acquire(o, conn, say); !ok {
+ granted, ok := acquire(o, conn, say)
+ if !ok {
return ExitRefused
}
+ cardEnv = visibleDevices(o.Card, granted, os.LookupEnv, say)
+ if o.Exclusive {
+ restore, err := exclusive(o, granted.CardUUID)
+ if err != nil {
+ say("--exclusive: %v; not starting %s", err, o.Engine)
+ return ExitRefused
+ }
+ defer restore(say)
+ }
}
cmd := command(o, o.Output)
@@ -256,15 +285,15 @@ func Run(o Options) int {
done := make(chan struct{})
- if o.CLI == nil {
- o.CLI = execCLI
- }
var cid containerRef
o.container = &cid
// line is the lease's live conn and park state; the drain, the resume
// and the container resolver ride it across reattach loops.
line := newLine(conn)
+ if conn != nil {
+ line.startTurn()
+ }
go forwardSignals(cmd, done, func() bool { return o.Checkpoint && line.isParked() }, line.askStop)
// A container engine resolves its containers to their full IDs and names
@@ -356,7 +385,17 @@ func Run(o Options) int {
case m.Op == proto.OpReturn:
go line.returnStep(o, m.LendID, say)
case m.Op == proto.OpGranted && m.Resume:
- go line.resume(o, cmd, done, say)
+ line.startTurn()
+ if waking, ok := line.beginWake(); ok {
+ go line.resume(o, cmd, done, waking, say)
+ }
+ case m.Op == proto.OpYield:
+ line.setYield(time.Duration(m.Yield * float64(time.Second)))
+ if m.Yield > 0 {
+ say("%s yielded the card from inside its job for %s", o.Engine, time.Duration(m.Yield*float64(time.Second)))
+ } else {
+ say("%s ended its yield", o.Engine)
+ }
}
})
select {
@@ -404,30 +443,30 @@ func Run(o Options) int {
// acquire asks berthd for the card and waits for the grant. It returns the
// device entry the engine starts with, and false when there is no grant.
-func acquire(o Options, conn *proto.Conn, say func(string, ...any)) (string, bool) {
- if err := conn.Send(proto.Msg{Op: proto.OpAcquire, Card: o.Card, Engine: o.Engine, Group: o.Group, Priority: o.Priority, Needs: largest(o.Needs), NeedsEach: each(o.Needs), Wait: o.Wait.Seconds(), Pinned: o.Pinned, Switched: o.Switched, Lend: o.Lend != nil, WholeMachine: o.WholeMachine, MinTurn: o.MinTurn.Seconds(), ParkTTL: o.ParkTTL.Seconds(), Window: o.Window.String()}); err != nil {
+func acquire(o Options, conn *proto.Conn, say func(string, ...any)) (proto.Msg, bool) {
+ if err := conn.Send(proto.Msg{Op: proto.OpAcquire, Card: o.Card, Engine: o.Engine, Group: o.Group, Priority: o.Priority, Needs: largest(o.Needs), NeedsEach: each(o.Needs), LearnSize: o.LearnSize, Wait: o.Wait.Seconds(), Pinned: o.Pinned, Training: o.Training, Switched: o.Switched, Lend: o.Lend != nil, WholeMachine: o.WholeMachine, MinTurn: o.MinTurn.Seconds(), ParkTTL: o.ParkTTL.Seconds(), Window: o.Window.String()}); err != nil {
say("acquire: %v", err)
- return "", false
+ return proto.Msg{}, false
}
for {
m, err := conn.Recv()
if err != nil {
say("berthd went away while %s was waiting: %v", o.Engine, err)
- return "", false
+ return proto.Msg{}, false
}
switch m.Op {
case proto.OpWaiting:
say("waiting: %s", m.Reason)
case proto.OpRefused:
say("refused: %s", m.Reason)
- return "", false
+ return proto.Msg{}, false
case proto.OpGranted:
if m.Reason != "" {
say("granted %s, WARNING: %s", m.Lease, m.Reason)
} else {
say("granted %s", m.Lease)
}
- return visibleDevices(o.Card, m, os.LookupEnv, say), true
+ return m, true
}
}
}
@@ -491,8 +530,8 @@ func reattach(o Options, cmd *exec.Cmd, done <-chan struct{}, cid *containerRef,
// waiters behind memory that is already gone.
func reattachMsg(o Options, pid int, start uint64, cid *containerRef, parked bool, line *leaseLine) proto.Msg {
m := proto.Msg{Op: proto.OpReattach, Card: o.Card, Engine: o.Engine, Group: o.Group, Priority: o.Priority,
- Needs: largest(o.Needs), NeedsEach: each(o.Needs), Pinned: o.Pinned, Lend: o.Lend != nil, WholeMachine: o.WholeMachine,
- MinTurn: o.MinTurn.Seconds(), ParkTTL: o.ParkTTL.Seconds(), Window: o.Window.String(),
+ Needs: largest(o.Needs), NeedsEach: each(o.Needs), LearnSize: o.LearnSize, Pinned: o.Pinned, Training: o.Training, Lend: o.Lend != nil, WholeMachine: o.WholeMachine,
+ MinTurn: o.MinTurn.Seconds(), Held: line.held().Seconds(), ParkTTL: o.ParkTTL.Seconds(), Window: o.Window.String(),
PID: pid, PIDStart: start, Containers: cid.list(), Parked: parked}
if t, ok := line.openLend(); ok {
m.LendID, m.Lent = t.ID, t.MiB<<20
@@ -739,7 +778,7 @@ func record(o Options, cmd *exec.Cmd, reason string) func() {
pid, self := cmd.Process.Pid, os.Getpid()
remove, err := runs.Write(runs.Dir(o.Socket), runs.Record{
Engine: o.Engine, PID: pid, Start: procs.System{}.StartTime(pid), Started: time.Now(), Reason: reason,
- Wrapper: self, WrapperStart: procs.System{}.StartTime(self),
+ Wrapper: self, WrapperStart: procs.System{}.StartTime(self), Training: o.Training,
})
if err != nil {
fmt.Fprintf(o.Stderr, "berth: could not record the unprotected run: %v\n", err)
@@ -830,9 +869,11 @@ func drainWait(o Options) time.Duration {
func drain(o Options, cmd *exec.Cmd, line *leaseLine, done <-chan struct{}, reason string, say func(string, ...any)) {
say("draining %s: %s", o.Engine, reason)
+ line.awaitWake(o.Engine, done, say)
if o.DrainComfyUI != "" {
boundary := o
boundary.ExitComfyUIIdle = false
+ boundary.yielded = line.yielding
if !jobBoundary(boundary, done, say) {
return
}
diff --git a/internal/runner/yield_test.go b/internal/runner/yield_test.go
new file mode 100644
index 00000000..9ca83ea5
--- /dev/null
+++ b/internal/runner/yield_test.go
@@ -0,0 +1,169 @@
+// Copyright 2026 Trevin Tindall (T-Crypt)
+// SPDX-License-Identifier: Apache-2.0
+
+//go:build unix
+
+package runner
+
+import (
+ "fmt"
+ "net/http"
+ "net/http/httptest"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/T-Crypt/berth/internal/broker"
+ "github.com/T-Crypt/berth/internal/gpu"
+ "github.com/T-Crypt/berth/internal/park"
+ "github.com/T-Crypt/berth/internal/proto"
+ "github.com/T-Crypt/berth/internal/server"
+)
+
+// busyComfy is a ComfyUI whose one job never ends: the job is waiting on an
+// LLM node that waits on the card (#299).
+func busyComfy(t *testing.T) string {
+ t.Helper()
+ srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ switch r.URL.Path {
+ case "/queue":
+ fmt.Fprint(w, `{"queue_running":[["1","job"]],"queue_pending":[]}`)
+ default:
+ w.WriteHeader(http.StatusOK)
+ fmt.Fprint(w, `{}`)
+ }
+ }))
+ t.Cleanup(srv.Close)
+ return srv.URL
+}
+
+// #299: a ComfyUI job that waits on an LLM never reaches its job boundary,
+// so a drain for that LLM waited on the job and the job waited on the drain.
+// A yield from inside the job is a boundary for as long as it lasts.
+func TestYieldIsAJobBoundary(t *testing.T) {
+ for _, tc := range []struct {
+ name string
+ before time.Duration // a yield set before the drain starts
+ during time.Duration // a yield set while the drain waits
+ passes bool
+ }{
+ {name: "no yield", passes: false},
+ {name: "yield before the drain", before: time.Minute, passes: true},
+ {name: "yield while the drain waits", during: time.Minute, passes: true},
+ {name: "yield that expired", before: time.Nanosecond, passes: false},
+ } {
+ t.Run(tc.name, func(t *testing.T) {
+ url := busyComfy(t)
+ cmd, waited := startSleep(t)
+ line := newLine(nil)
+ if tc.before > 0 {
+ line.setYield(tc.before)
+ time.Sleep(time.Millisecond)
+ }
+ done := make(chan struct{})
+ var logs strings.Builder
+ go drain(Options{Engine: "comfyui", DrainComfyUI: url, Retry: 5 * time.Millisecond,
+ Grace: 50 * time.Millisecond}, cmd, line, done, "test", sayTo(&logs))
+ if tc.during > 0 {
+ time.Sleep(30 * time.Millisecond)
+ line.setYield(tc.during)
+ }
+ select {
+ case <-waited:
+ if !tc.passes {
+ t.Fatalf("the drain stopped ComfyUI mid-job with no yield")
+ }
+ case <-time.After(time.Second):
+ if tc.passes {
+ t.Fatalf("the drain still waits on the job after a yield")
+ }
+ close(done)
+ }
+ })
+ }
+}
+
+// sendYield is the call a ComfyUI node makes through berthd: a yield of d,
+// or its end when d is 0.
+func sendYield(t *testing.T, sock string, d time.Duration) proto.Msg {
+ t.Helper()
+ conn, raw, err := server.Dial(sock)
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer raw.Close()
+ if err := conn.Send(proto.Msg{Op: proto.OpYield, Engine: "comfyui", Yield: d.Seconds()}); err != nil {
+ t.Fatal(err)
+ }
+ m, err := conn.Recv()
+ if err != nil {
+ t.Fatal(err)
+ }
+ return m
+}
+
+// #299, the whole cycle with ComfyUI's park: the yield lets the drain park
+// ComfyUI mid-job, the LLM runs in the room the card shows free, and the end
+// of the yield gives the job its card back.
+func TestYieldParksAndItsEndResumes(t *testing.T) {
+ card := &parkCard{total: 24 << 30}
+ sock, b := parkDaemon(t, card, 16<<30)
+ stub := &comfyStub{}
+ stub.busy.Store(true)
+ srv := httptest.NewServer(stub.handler())
+ defer srv.Close()
+ adapter, ok := park.For("comfyui", "", "")
+ if !ok {
+ t.Fatal("the comfyui adapter is missing")
+ }
+ log := &syncBuf{}
+ t.Cleanup(func() {
+ if t.Failed() {
+ t.Logf("engine log:\n%s", log.String())
+ }
+ })
+ exited := make(chan int, 1)
+ go func() {
+ exited <- Run(Options{Socket: sock, Engine: "comfyui", Priority: "interactive", Needs: []uint64{8 << 30},
+ DrainComfyUI: srv.URL, Park: &adapter, ParkBase: srv.URL, ParkProbe: "/system_stats",
+ Retry: 10 * time.Millisecond, Grace: 200 * time.Millisecond, Argv: []string{"sleep", "30"}, Stderr: log})
+ }()
+ pid := attachAndMeasure(t, b, card)
+ stopAtEnd(t, pid, exited)
+
+ holder := acquireIn(b, broker.Request{Engine: "gemma", Priority: broker.Foreground})
+ waitFor(t, "the drain waiting on the job", func() bool {
+ return strings.Contains(log.String(), "waiting for ComfyUI to finish")
+ })
+
+ if m := sendYield(t, sock, time.Minute); m.Op != proto.OpYield {
+ t.Fatalf("yield reply = %+v", m)
+ }
+ waitFor(t, "the park unload", func() bool {
+ stub.mu.Lock()
+ defer stub.mu.Unlock()
+ return stub.frees >= 1
+ })
+ card.set(gpu.Proc{PID: pid, Name: "sleep", Used: 512 << 20})
+ waitFor(t, "parked", func() bool { return b.Status().Leases[0].State == "parked" })
+ hold := <-holder
+ if hold == nil {
+ t.Fatal("gemma was never granted beside the parked job")
+ }
+ hold.Release()
+
+ if m := sendYield(t, sock, 0); m.Op != proto.OpYield {
+ t.Fatalf("yield end reply = %+v", m)
+ }
+ waitFor(t, "the job's card back", func() bool {
+ st := b.Status()
+ return len(st.Leases) == 1 && st.Leases[0].Engine == "comfyui" && st.Leases[0].State == "active"
+ })
+ // The job runs again: its run must not still count it as at its boundary.
+ waitFor(t, "the run hearing the end", func() bool { return strings.Contains(log.String(), "comfyui ended its yield") })
+ select {
+ case code := <-exited:
+ t.Fatalf("ComfyUI exited (%d); a yield must park it, not stop it", code)
+ default:
+ }
+}
diff --git a/internal/runs/runs.go b/internal/runs/runs.go
index 644c3439..56127ad5 100644
--- a/internal/runs/runs.go
+++ b/internal/runs/runs.go
@@ -30,6 +30,10 @@ type Record struct {
// can re-attach the engine. 0 in a record from before v0.5.1.
Wrapper int `json:"wrapper_pid,omitempty"`
WrapperStart uint64 `json:"wrapper_start,omitempty"`
+ // Training marks a berth train run. While it lives without a lease, it
+ // still locks the node: berth run will not start an engine without
+ // berthd, and a new berthd refuses other requests until it re-attaches.
+ Training bool `json:"training,omitempty"`
}
// WrapperState is "alive" while the berth run that wrote r still runs,
@@ -83,6 +87,16 @@ func Write(dir string, r Record) (func(), error) {
return func() { os.Remove(p) }, nil
}
+// Training is a live training record in dir, and whether there is one.
+func Training(dir string, alive func(pid int) uint64) (Record, bool) {
+ for _, r := range List(dir, alive) {
+ if r.Training {
+ return r, true
+ }
+ }
+ return Record{}, false
+}
+
// List returns the records whose process is still the same process
// (alive reports its current start time, 0 if gone), and deletes the rest.
func List(dir string, alive func(pid int) uint64) []Record {
diff --git a/internal/runs/runs_test.go b/internal/runs/runs_test.go
new file mode 100644
index 00000000..f54b83d7
--- /dev/null
+++ b/internal/runs/runs_test.go
@@ -0,0 +1,34 @@
+// Copyright 2026 Trevin Tindall (T-Crypt)
+// SPDX-License-Identifier: Apache-2.0
+
+package runs
+
+import (
+ "testing"
+ "time"
+)
+
+// Training finds a live training record and passes over the rest: a plain
+// run, and a training whose process is gone.
+func TestTrainingIsALiveTrainingRecord(t *testing.T) {
+ dir := t.TempDir()
+ for _, r := range []Record{
+ {Engine: "chat", PID: 10, Start: 1, Started: time.Now(), Reason: ReasonLost},
+ {Engine: "train-old", PID: 11, Start: 1, Started: time.Now(), Reason: ReasonLost, Training: true},
+ {Engine: "train-a", PID: 12, Start: 5, Started: time.Now(), Reason: ReasonLost, Training: true},
+ } {
+ if _, err := Write(dir, r); err != nil {
+ t.Fatal(err)
+ }
+ }
+ alive := func(pid int) uint64 {
+ return map[int]uint64{10: 1, 12: 5}[pid]
+ }
+ r, ok := Training(dir, alive)
+ if !ok || r.Engine != "train-a" {
+ t.Fatalf("Training = %+v, %v; want train-a", r, ok)
+ }
+ if _, ok := Training(t.TempDir(), alive); ok {
+ t.Error("an empty directory has a training")
+ }
+}
diff --git a/internal/schedule/schedule.go b/internal/schedule/schedule.go
new file mode 100644
index 00000000..529d96e2
--- /dev/null
+++ b/internal/schedule/schedule.go
@@ -0,0 +1,184 @@
+// Copyright 2026 Trevin Tindall (T-Crypt)
+// SPDX-License-Identifier: Apache-2.0
+
+// Package schedule is the cards' booking file. berthd is the only process
+// that writes it, so the CLI, the gateway and the harbor reach it through
+// berthd's socket and no file lock is needed. A booking that has ended
+// leaves the file on the next write.
+package schedule
+
+import (
+ "encoding/json"
+ "errors"
+ "fmt"
+ "os"
+ "slices"
+ "strconv"
+ "sync"
+ "time"
+
+ "github.com/T-Crypt/berth/internal/fsx"
+ "github.com/T-Crypt/berth/internal/proto"
+)
+
+// MaxDays is how many consecutive days one request may book at once.
+const MaxDays = 14
+
+type file struct {
+ Next int `json:"next"`
+ Bookings []proto.Booking `json:"bookings"`
+}
+
+// Store is the booking file, held in memory once loaded.
+type Store struct {
+ path string
+ mu sync.Mutex
+ f file
+}
+
+// Open loads the file at path; a missing file is an empty schedule.
+func Open(path string) (*Store, error) {
+ s := &Store{path: path}
+ b, err := os.ReadFile(path)
+ switch {
+ case errors.Is(err, os.ErrNotExist):
+ return s, nil
+ case err != nil:
+ return nil, err
+ }
+ if err := json.Unmarshal(b, &s.f); err != nil {
+ return nil, fmt.Errorf("%s: %w", path, err)
+ }
+ return s, nil
+}
+
+// List is every booking that has not ended at now, by start.
+func (s *Store) List(now time.Time) []proto.Booking {
+ s.mu.Lock()
+ defer s.mu.Unlock()
+ return live(s.f.Bookings, now)
+}
+
+func live(bs []proto.Booking, now time.Time) []proto.Booking {
+ out := slices.DeleteFunc(slices.Clone(bs), func(b proto.Booking) bool { return !b.End.After(now) })
+ for i := range out {
+ out[i] = onClock(out[i], now)
+ }
+ slices.SortStableFunc(out, func(a, b proto.Booking) int { return a.Start.Compare(b.Start) })
+ return out
+}
+
+// Book puts b on the schedule on days consecutive days (0 = 1), all of
+// them or none. With earliest, each copy starts at the first hours at or
+// after its own start where it clashes with nothing, keeping its length.
+func (s *Store) Book(b proto.Booking, days int, earliest bool, now time.Time) ([]proto.Booking, error) {
+ if days <= 0 {
+ days = 1
+ }
+ if days > MaxDays {
+ return nil, fmt.Errorf("book at most %d days at once", MaxDays)
+ }
+ b = onClock(b, now)
+ s.mu.Lock()
+ defer s.mu.Unlock()
+ all := live(s.f.Bookings, now)
+ next := s.f.Next
+ var made []proto.Booking
+ for d := range days {
+ c := b
+ c.Start, c.End = b.Start.AddDate(0, 0, d), b.End.AddDate(0, 0, d)
+ if earliest {
+ c = firstFree(c, all)
+ }
+ next++
+ c.ID = "b" + strconv.Itoa(next)
+ if err := fits(c, all, now); err != nil {
+ return nil, err
+ }
+ all = append(all, c)
+ made = append(made, c)
+ }
+ return made, s.save(file{Next: next, Bookings: all})
+}
+
+// Unbook takes the booking id off the schedule.
+func (s *Store) Unbook(id string, now time.Time) (proto.Booking, error) {
+ s.mu.Lock()
+ defer s.mu.Unlock()
+ all := live(s.f.Bookings, now)
+ i := slices.IndexFunc(all, func(b proto.Booking) bool { return b.ID == id })
+ if i < 0 {
+ return proto.Booking{}, fmt.Errorf("no booking %s on the schedule", id)
+ }
+ gone := all[i]
+ return gone, s.save(file{Next: s.f.Next, Bookings: slices.Delete(all, i, i+1)})
+}
+
+// Rebook replaces the booking with b.ID by b: new hours, priority, engine
+// or note. It keeps its ID, and is checked as a new booking would be
+// against every other one.
+func (s *Store) Rebook(b proto.Booking, now time.Time) (proto.Booking, error) {
+ b = onClock(b, now)
+ s.mu.Lock()
+ defer s.mu.Unlock()
+ all := live(s.f.Bookings, now)
+ i := slices.IndexFunc(all, func(o proto.Booking) bool { return o.ID == b.ID })
+ if i < 0 {
+ return proto.Booking{}, fmt.Errorf("no booking %s on the schedule", b.ID)
+ }
+ rest := slices.Delete(slices.Clone(all), i, i+1)
+ if err := fits(b, rest, now); err != nil {
+ return proto.Booking{}, err
+ }
+ all[i] = b
+ return b, s.save(file{Next: s.f.Next, Bookings: all})
+}
+
+// fits is whether b may join others: a valid booking that clashes with none.
+func fits(b proto.Booking, others []proto.Booking, now time.Time) error {
+ if err := b.Check(now); err != nil {
+ return err
+ }
+ for _, o := range others {
+ if b.Clashes(o) {
+ return fmt.Errorf("%s %s-%s clashes with %s (%s for %s, %s-%s)", b.Kind, clock(b.Start), clock(b.End),
+ o.ID, o.Kind, o.Who, clock(o.Start), clock(o.End))
+ }
+ }
+ return nil
+}
+
+// firstFree moves b later, keeping its length, until it clashes with none
+// of others. Each clash it steps past ends later than the last, so it stops.
+func firstFree(b proto.Booking, others []proto.Booking) proto.Booking {
+ length := b.End.Sub(b.Start)
+ for {
+ i := slices.IndexFunc(others, b.Clashes)
+ if i < 0 {
+ return b
+ }
+ b.Start = others[i].End
+ b.End = b.Start.Add(length)
+ }
+}
+
+func clock(t time.Time) string { return t.Format("Mon 15:04") }
+
+// onClock is b with its hours read on now's clock, the node's: a browser
+// sends UTC, and the file, the status and every message read local time.
+func onClock(b proto.Booking, now time.Time) proto.Booking {
+ b.Start, b.End = b.Start.In(now.Location()), b.End.In(now.Location())
+ return b
+}
+
+func (s *Store) save(f file) error {
+ b, err := json.MarshalIndent(f, "", " ")
+ if err != nil {
+ return err
+ }
+ if err := fsx.WriteFile(s.path, b, 0o600); err != nil {
+ return err
+ }
+ s.f = f
+ return nil
+}
diff --git a/internal/schedule/schedule_test.go b/internal/schedule/schedule_test.go
new file mode 100644
index 00000000..2c2edf91
--- /dev/null
+++ b/internal/schedule/schedule_test.go
@@ -0,0 +1,179 @@
+// Copyright 2026 Trevin Tindall (T-Crypt)
+// SPDX-License-Identifier: Apache-2.0
+
+package schedule
+
+import (
+ "path/filepath"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/T-Crypt/berth/internal/proto"
+)
+
+// sat1200 is Saturday 2026-11-14 12:00 on a fixed clock.
+var sat1200 = time.Date(2026, 11, 14, 12, 0, 0, 0, time.UTC)
+
+func at(h, m int) time.Time { return time.Date(2026, 11, 14, h, m, 0, 0, time.UTC) }
+
+func run(who string, from, to time.Time) proto.Booking {
+ return proto.Booking{Card: "gpu0", Kind: proto.BookRun, Who: who, Engine: who, Start: from, End: to}
+}
+
+func open(t *testing.T) (*Store, string) {
+ t.Helper()
+ p := filepath.Join(t.TempDir(), "schedule.json")
+ s, err := Open(p)
+ if err != nil {
+ t.Fatal(err)
+ }
+ return s, p
+}
+
+// A booking survives a restart, keeps its ID, and a second booking that
+// claims the same hours on the card is refused with what it clashes with.
+func TestBookPersistsAndRefusesAClash(t *testing.T) {
+ s, p := open(t)
+ made, err := s.Book(run("pi-pocket", at(23, 0), at(23, 0).Add(5*time.Hour)), 1, false, sat1200)
+ if err != nil || len(made) != 1 || made[0].ID != "b1" {
+ t.Fatalf("book = %+v, %v; want one booking b1", made, err)
+ }
+ _, err = s.Book(run("train", at(22, 0), at(23, 30)), 1, false, sat1200)
+ if err == nil || !strings.Contains(err.Error(), "clashes with b1 (run for pi-pocket, Sat 23:00-Sun 04:00)") {
+ t.Fatalf("clash err = %v", err)
+ }
+ again, err := Open(p)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if got := again.List(sat1200); len(got) != 1 || got[0].ID != "b1" || got[0].Who != "pi-pocket" {
+ t.Fatalf("after reopen = %+v, want b1 for pi-pocket", got)
+ }
+}
+
+// A priority change claims no time: it books over a run, but not over
+// another priority change for the same engine.
+func TestPriorityBookingsOverlapRunsNotEachOther(t *testing.T) {
+ s, _ := open(t)
+ if _, err := s.Book(run("train", at(13, 0), at(16, 0)), 1, false, sat1200); err != nil {
+ t.Fatal(err)
+ }
+ p := proto.Booking{Card: "gpu0", Kind: proto.BookPriority, Who: "op", Engine: "dirk", Priority: "batch", Start: at(13, 0), End: at(16, 0)}
+ if _, err := s.Book(p, 1, false, sat1200); err != nil {
+ t.Fatalf("priority over a run: %v", err)
+ }
+ if _, err := s.Book(p, 1, false, sat1200); err == nil {
+ t.Fatal("a second priority change for dirk in the same hours was booked")
+ }
+}
+
+// Earliest moves a booking past every clash to the first free hours with
+// its length; days books the same hours on each day, all or none.
+func TestEarliestAndDays(t *testing.T) {
+ s, _ := open(t)
+ if _, err := s.Book(run("op", at(13, 0), at(18, 0)), 1, false, sat1200); err != nil {
+ t.Fatal(err)
+ }
+ if _, err := s.Book(run("pi", at(18, 0), at(19, 0)), 1, false, sat1200); err != nil {
+ t.Fatal(err)
+ }
+ got, err := s.Book(run("train", at(14, 0), at(17, 0)), 1, true, sat1200)
+ if err != nil || !got[0].Start.Equal(at(19, 0)) || !got[0].End.Equal(at(22, 0)) {
+ t.Fatalf("earliest = %+v, %v; want 19:00-22:00", got, err)
+ }
+ nightly, err := s.Book(run("pocket", at(23, 0), at(23, 0).Add(5*time.Hour)), 7, false, sat1200)
+ if err != nil || len(nightly) != 7 || !nightly[6].Start.Equal(at(23, 0).AddDate(0, 0, 6)) {
+ t.Fatalf("7 nights = %d, %v", len(nightly), err)
+ }
+ before := len(s.List(sat1200))
+ if _, err := s.Book(run("x", at(1, 0).AddDate(0, 0, 3), at(2, 0).AddDate(0, 0, 3)), 2, false, sat1200); err == nil {
+ t.Fatal("a repeat that clashes on one day was booked")
+ }
+ if after := len(s.List(sat1200)); after != before {
+ t.Fatalf("a refused repeat left %d bookings, want %d", after, before)
+ }
+}
+
+// Rebook moves a booking and keeps its ID; unbook takes it off; an ended
+// booking is no longer listed.
+func TestRebookUnbookAndEnd(t *testing.T) {
+ s, _ := open(t)
+ made, _ := s.Book(run("pi", at(13, 0), at(14, 0)), 1, false, sat1200)
+ moved := made[0]
+ moved.Start, moved.End, moved.Priority = at(15, 0), at(17, 0), "batch"
+ if _, err := s.Rebook(moved, sat1200); err != nil {
+ t.Fatal(err)
+ }
+ if got := s.List(sat1200); len(got) != 1 || got[0].ID != "b1" || !got[0].Start.Equal(at(15, 0)) || got[0].Priority != "batch" {
+ t.Fatalf("after rebook = %+v", got)
+ }
+ if got := s.List(at(17, 0)); len(got) != 0 {
+ t.Fatalf("an ended booking is still listed: %+v", got)
+ }
+ if _, err := s.Unbook("b1", sat1200); err != nil {
+ t.Fatal(err)
+ }
+ if _, err := s.Unbook("b1", sat1200); err == nil {
+ t.Fatal("unbooked twice")
+ }
+}
+
+// A browser sends UTC; the schedule keeps and names the node's own clock, so
+// a 23:00 booking at UTC-7 clashes "Sat 23:00", not "Sun 06:00".
+func TestBookingsReadTheNodesClock(t *testing.T) {
+ s, _ := open(t)
+ mst := time.FixedZone("MST", -7*3600)
+ now := sat1200.In(mst)
+ night := time.Date(2026, 11, 14, 23, 0, 0, 0, mst)
+ made, err := s.Book(run("pi", night.UTC(), night.Add(5*time.Hour).UTC()), 1, false, now)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if got := made[0].Start.Format("Mon 15:04 MST"); got != "Sat 23:00 MST" {
+ t.Errorf("start = %s, want Sat 23:00 MST", got)
+ }
+ _, err = s.Book(run("x", night.UTC(), night.Add(time.Hour).UTC()), 1, false, now)
+ if err == nil || !strings.Contains(err.Error(), "run Sat 23:00-Sun 00:00 clashes with b1 (run for pi, Sat 23:00-Sun 04:00)") {
+ t.Errorf("clash = %v", err)
+ }
+}
+
+func TestCheckRefusesBadBookings(t *testing.T) {
+ for _, tc := range []struct {
+ b proto.Booking
+ want string
+ }{
+ {proto.Booking{Kind: "nap", Who: "a", Start: at(13, 0), End: at(14, 0)}, "use run, priority or down"},
+ {proto.Booking{Kind: proto.BookRun, Who: "a", Start: at(13, 0), End: at(14, 0)}, "names its engine"},
+ {proto.Booking{Kind: proto.BookDown, Who: "a", Start: at(10, 0), End: at(11, 0)}, "ends in the past"},
+ {proto.Booking{Kind: proto.BookDown, Who: "a", Start: at(13, 0), End: at(13, 0).AddDate(0, 0, 8)}, "at most"},
+ {proto.Booking{Kind: proto.BookPriority, Who: "a", Engine: "e", Start: at(13, 0), End: at(14, 0)}, "names its priority"},
+ {proto.Booking{Kind: proto.BookRun, Who: "a", Engine: "e", Priority: "hold", Start: at(13, 0), End: at(14, 0)}, `priority "hold"`},
+ } {
+ if err := tc.b.Check(sat1200); err == nil || !strings.Contains(err.Error(), tc.want) {
+ t.Errorf("%+v: err = %v, want %q", tc.b, err, tc.want)
+ }
+ }
+}
+
+func TestParseAt(t *testing.T) {
+ for in, want := range map[string]time.Time{
+ "now": sat1200,
+ "22:00": at(22, 0),
+ "09:30": at(9, 30).AddDate(0, 0, 1),
+ "tue 22:00": at(22, 0).AddDate(0, 0, 3),
+ "sat 11:00": at(11, 0).AddDate(0, 0, 7),
+ "sat 13:00": at(13, 0),
+ "2026-11-15 06:00": at(6, 0).AddDate(0, 0, 1),
+ "2026-11-15T06:00": at(6, 0).AddDate(0, 0, 1),
+ } {
+ got, err := ParseAt(in, sat1200)
+ if err != nil || !got.Equal(want) {
+ t.Errorf("ParseAt(%q) = %v, %v; want %v", in, got, err, want)
+ }
+ }
+ if _, err := ParseAt("soon", sat1200); err == nil {
+ t.Error("ParseAt(soon) took it")
+ }
+}
diff --git a/internal/schedule/when.go b/internal/schedule/when.go
new file mode 100644
index 00000000..a2b2151f
--- /dev/null
+++ b/internal/schedule/when.go
@@ -0,0 +1,63 @@
+// Copyright 2026 Trevin Tindall (T-Crypt)
+// SPDX-License-Identifier: Apache-2.0
+
+package schedule
+
+import (
+ "fmt"
+ "strings"
+ "time"
+)
+
+// ParseAt reads a start time on the node's clock, as a person types it:
+// "now"; "22:00", the next time the clock reads it; "tue 22:00", the next
+// Tuesday at 22:00 (today if still ahead); "2026-11-14 22:00" or
+// "2026-11-14T22:00"; or RFC 3339.
+func ParseAt(s string, now time.Time) (time.Time, error) {
+ s = strings.TrimSpace(strings.ToLower(s))
+ loc := now.Location()
+ if s == "now" {
+ return now, nil
+ }
+ if t, err := time.Parse(time.RFC3339, strings.ToUpper(s)); err == nil {
+ return t.In(loc), nil
+ }
+ for _, layout := range []string{"2006-01-02 15:04", "2006-01-02t15:04"} {
+ if t, err := time.ParseInLocation(layout, s, loc); err == nil {
+ return t, nil
+ }
+ }
+ day, clock, hasDay := strings.Cut(s, " ")
+ if !hasDay {
+ clock = day
+ }
+ hm, err := time.ParseInLocation("15:04", clock, loc)
+ if err != nil {
+ return time.Time{}, fmt.Errorf("start %q: use now, 22:00, tue 22:00 or 2026-11-14 22:00", s)
+ }
+ t := time.Date(now.Year(), now.Month(), now.Day(), hm.Hour(), hm.Minute(), 0, 0, loc)
+ if !hasDay {
+ if t.Before(now) {
+ t = t.AddDate(0, 0, 1)
+ }
+ return t, nil
+ }
+ wd, ok := weekday(day)
+ if !ok {
+ return time.Time{}, fmt.Errorf("start %q: %q is not a day (mon, tue, ...)", s, day)
+ }
+ t = t.AddDate(0, 0, (int(wd)-int(t.Weekday())+7)%7)
+ if t.Before(now) {
+ t = t.AddDate(0, 0, 7)
+ }
+ return t, nil
+}
+
+func weekday(s string) (time.Weekday, bool) {
+ for d := time.Sunday; d <= time.Saturday; d++ {
+ if name := strings.ToLower(d.String()); len(s) >= 3 && strings.HasPrefix(name, s) {
+ return d, true
+ }
+ }
+ return 0, false
+}
diff --git a/internal/server/bookings.go b/internal/server/bookings.go
new file mode 100644
index 00000000..714fc44e
--- /dev/null
+++ b/internal/server/bookings.go
@@ -0,0 +1,71 @@
+// Copyright 2026 Trevin Tindall (T-Crypt)
+// SPDX-License-Identifier: Apache-2.0
+
+package server
+
+// OpBook, OpUnbook and OpRebook: the cards' schedule. berthd owns the file;
+// every change goes to the cards' brokers at once and onto the event
+// stream.
+
+import (
+ "fmt"
+ "log/slog"
+ "time"
+
+ "github.com/T-Crypt/berth/internal/node"
+ "github.com/T-Crypt/berth/internal/proto"
+ "github.com/T-Crypt/berth/internal/schedule"
+)
+
+func bookings(conn *proto.Conn, log *slog.Logger, req proto.Msg, n *node.Node, s *schedule.Store) {
+ refuse := func(why string) { reply(conn, log, proto.Msg{Op: proto.OpRefused, Reason: why}) }
+ if s == nil {
+ refuse("this berthd keeps no schedule (--schedule \"\")")
+ return
+ }
+ if req.Booking == nil {
+ refuse("the request carries no booking")
+ return
+ }
+ now := time.Now()
+ b := *req.Booking
+ var changed []proto.Booking
+ var err error
+ switch req.Op {
+ case proto.OpBook, proto.OpRebook:
+ c, cerr := n.Card(b.Card)
+ if cerr != nil {
+ refuse(cerr.Error())
+ return
+ }
+ b.Card = c.Name
+ if req.Op == proto.OpBook {
+ changed, err = s.Book(b, req.Days, req.Earliest, now)
+ } else {
+ var one proto.Booking
+ one, err = s.Rebook(b, now)
+ changed = []proto.Booking{one}
+ }
+ case proto.OpUnbook:
+ var one proto.Booking
+ one, err = s.Unbook(b.ID, now)
+ changed = []proto.Booking{one}
+ }
+ if err != nil {
+ refuse(err.Error())
+ return
+ }
+ n.SetBookings(s.List(now))
+ verb := map[string]string{proto.OpBook: "booked", proto.OpUnbook: "unbooked", proto.OpRebook: "rebooked"}[req.Op]
+ for _, x := range changed {
+ detail := fmt.Sprintf("%s %s %s for %s, %s-%s", verb, x.ID, x.Kind, x.Who, x.Start.Format("Mon Jan 2 15:04"), x.End.Format("15:04"))
+ if x.Engine != "" {
+ detail += ", " + x.Engine
+ }
+ if c, err := n.Card(x.Card); err == nil {
+ c.B.Note(proto.Event{Kind: proto.EvBooked, Engine: x.Engine, Detail: detail})
+ }
+ log.Info("schedule", "change", detail)
+ }
+ reply(conn, log, proto.Msg{Op: req.Op, Bookings: changed})
+}
diff --git a/internal/server/e2e_test.go b/internal/server/e2e_test.go
index e6e95f07..222c82d5 100644
--- a/internal/server/e2e_test.go
+++ b/internal/server/e2e_test.go
@@ -516,3 +516,36 @@ func TestComfyUIDrainWaitsForJobBoundary(t *testing.T) {
t.Fatalf("no handoff line:\n%s", out1.String())
}
}
+
+// The socket's status carries the configured hours of an engine with no
+// lease, so a client sees a schedule before its engine runs.
+func TestSocketStatusCarriesTheSchedules(t *testing.T) {
+ sock := filepath.Join(t.TempDir(), "berth.sock")
+ log := slog.New(slog.NewTextHandler(io.Discard, nil))
+ b := broker.New(broker.Config{Poll: 20 * time.Millisecond, DefaultWait: 5 * time.Second, JuggleWindow: -1, Log: log}, emptyCard{}, procs.System{})
+ ln, err := server.Listen(sock)
+ if err != nil {
+ t.Fatal(err)
+ }
+ ctx, cancel := context.WithCancel(context.Background())
+ t.Cleanup(cancel)
+ go b.Run(ctx)
+ windows := map[string]proto.Window{"nightly": {Start: 23 * 60, End: 7 * 60}, "chat": {}}
+ go server.Serve(ctx, ln, node.One(b), server.Extras{Windows: windows}, log)
+
+ conn, raw, err := server.Dial(sock)
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer raw.Close()
+ if err := conn.Send(proto.Msg{Op: proto.OpStatus}); err != nil {
+ t.Fatal(err)
+ }
+ m, err := conn.Recv()
+ if err != nil || m.Status == nil {
+ t.Fatalf("status: %v %+v", err, m)
+ }
+ if s := m.Status.Schedules; len(s) != 1 || s[0].Engine != "nightly" || s[0].Window != "23:00-07:00" || s[0].ChangeS <= 0 {
+ t.Fatalf("schedules = %+v, want nightly 23:00-07:00 alone", s)
+ }
+}
diff --git a/internal/server/queue.go b/internal/server/queue.go
new file mode 100644
index 00000000..aa24fa69
--- /dev/null
+++ b/internal/server/queue.go
@@ -0,0 +1,28 @@
+// Copyright 2026 Trevin Tindall (T-Crypt)
+// SPDX-License-Identifier: Apache-2.0
+
+package server
+
+// OpQueue. The operator bumps, re-ranks or drops one waiter; the reply names
+// the change, and the broker journals it as an event.
+
+import (
+ "log/slog"
+
+ "github.com/T-Crypt/berth/internal/broker"
+ "github.com/T-Crypt/berth/internal/node"
+ "github.com/T-Crypt/berth/internal/proto"
+)
+
+func queue(conn *proto.Conn, log *slog.Logger, req proto.Msg, n *node.Node) {
+ if req.Lease == "" {
+ reply(conn, log, proto.Msg{Op: proto.OpRefused, Reason: "lease ID is required"})
+ return
+ }
+ detail, err := n.Queue(req.Lease, broker.QueueAction(req.Action), req.Priority)
+ if err != nil {
+ reply(conn, log, proto.Msg{Op: proto.OpRefused, Reason: err.Error()})
+ return
+ }
+ reply(conn, log, proto.Msg{Op: proto.OpQueue, Lease: req.Lease, Reason: detail})
+}
diff --git a/internal/server/schedules.go b/internal/server/schedules.go
new file mode 100644
index 00000000..561209f8
--- /dev/null
+++ b/internal/server/schedules.go
@@ -0,0 +1,35 @@
+// Copyright 2026 Trevin Tindall (T-Crypt)
+// SPDX-License-Identifier: Apache-2.0
+
+package server
+
+import (
+ "slices"
+ "strings"
+ "time"
+
+ "github.com/T-Crypt/berth/internal/proto"
+)
+
+// schedules is every engine held to hours at now, one row per engine by
+// name: the configured windows, with a lease's own window in place of its
+// engine's, since that is the one the broker enforces.
+func schedules(configured map[string]proto.Window, st *proto.Status, now time.Time) []proto.Schedule {
+ hours := make(map[string]proto.Window, len(configured))
+ for name, w := range configured {
+ if w.Set() {
+ hours[name] = w
+ }
+ }
+ for _, l := range slices.Concat(st.Leases, st.Waiting) {
+ if w, err := proto.ParseWindow(l.Window); err == nil && w.Set() {
+ hours[l.Engine] = w
+ }
+ }
+ out := make([]proto.Schedule, 0, len(hours))
+ for name, w := range hours {
+ out = append(out, proto.ScheduleAt(name, w, now))
+ }
+ slices.SortFunc(out, func(a, b proto.Schedule) int { return strings.Compare(a.Engine, b.Engine) })
+ return out
+}
diff --git a/internal/server/schedules_test.go b/internal/server/schedules_test.go
new file mode 100644
index 00000000..801486e3
--- /dev/null
+++ b/internal/server/schedules_test.go
@@ -0,0 +1,35 @@
+// Copyright 2026 Trevin Tindall (T-Crypt)
+// SPDX-License-Identifier: Apache-2.0
+
+package server
+
+import (
+ "slices"
+ "testing"
+ "time"
+
+ "github.com/T-Crypt/berth/internal/proto"
+)
+
+// An engine with hours shows whether or not it runs, and a lease's own
+// window stands in for its engine's.
+func TestSchedulesJoinConfigAndLeases(t *testing.T) {
+ now := time.Date(2026, 11, 12, 20, 0, 0, 0, time.Local)
+ configured := map[string]proto.Window{
+ "nightly": {Start: 23 * 60, End: 7 * 60},
+ "chat": {},
+ "render": {Start: 9 * 60, End: 17 * 60},
+ }
+ st := &proto.Status{
+ Leases: []proto.Lease{{Engine: "render", Window: "18:00-22:00"}, {Engine: "chat"}},
+ Waiting: []proto.Lease{{Engine: "job", Window: "01:00-05:00"}},
+ }
+ want := []proto.Schedule{
+ {Engine: "job", Window: "01:00-05:00", Open: false, ChangeS: 5 * 3600},
+ {Engine: "nightly", Window: "23:00-07:00", Open: false, ChangeS: 3 * 3600},
+ {Engine: "render", Window: "18:00-22:00", Open: true, ChangeS: 2 * 3600},
+ }
+ if got := schedules(configured, st, now); !slices.Equal(got, want) {
+ t.Errorf("schedules =\n%+v\nwant\n%+v", got, want)
+ }
+}
diff --git a/internal/server/server.go b/internal/server/server.go
index 3ab49f70..c62bf720 100644
--- a/internal/server/server.go
+++ b/internal/server/server.go
@@ -24,6 +24,7 @@ import (
"github.com/T-Crypt/berth/internal/node"
"github.com/T-Crypt/berth/internal/pair"
"github.com/T-Crypt/berth/internal/proto"
+ "github.com/T-Crypt/berth/internal/schedule"
)
// Listen opens the socket, refusing if another berthd already answers on it
@@ -118,6 +119,11 @@ type Extras struct {
// Finished hears that a run's engine ended on purpose (exit 0, or a
// stop the run carried out), by the lease's engine name.
Finished func(engine string)
+ // Windows are the configured engines' hours, by engine name.
+ Windows map[string]proto.Window
+ // Schedule is the cards' booking file; nil turns OpBook and its
+ // siblings off.
+ Schedule *schedule.Store
}
// Serve answers clients on ln.
@@ -168,6 +174,10 @@ func handle(ctx context.Context, c net.Conn, n *node.Node, x Extras, log *slog.L
st := n.Status()
st.Engines, st.Host, st.Insights = x.Engines(), x.Host(), x.Insights()
st.Alerts = x.Alerts()
+ st.Schedules = schedules(x.Windows, &st, time.Now())
+ if x.Schedule != nil {
+ st.Bookings = x.Schedule.List(time.Now())
+ }
st.Counters.SendTimeouts = sendTimeouts.Load()
st.Counters.DroppedConsumers = droppedConsumers.Load()
reply(conn, log, proto.Msg{Op: proto.OpStatus, Status: &st})
@@ -190,6 +200,12 @@ func handle(ctx context.Context, c net.Conn, n *node.Node, x Extras, log *slog.L
cordon(conn, log, first, n)
case proto.OpBatch:
batch(conn, log, first, n)
+ case proto.OpQueue:
+ queue(conn, log, first, n)
+ case proto.OpBook, proto.OpUnbook, proto.OpRebook:
+ bookings(conn, log, first, n, x.Schedule)
+ case proto.OpYield:
+ yield(conn, log, first, n)
case proto.OpPairCode, proto.OpPairRedeem, proto.OpPairCheck, proto.OpPairList, proto.OpPairRevoke:
pairing(conn, log, first, x.Pair)
case proto.OpResume:
@@ -261,8 +277,10 @@ func hold(ctx context.Context, conn *proto.Conn, req proto.Msg, n *node.Node, fi
Group: req.Group,
Priority: prio,
Needs: req.Needs,
+ LearnSize: req.LearnSize,
Wait: time.Duration(req.Wait * float64(time.Second)),
Pinned: req.Pinned,
+ Training: req.Training,
Switched: req.Switched,
Parked: req.Parked,
Lend: req.Lend,
@@ -270,6 +288,7 @@ func hold(ctx context.Context, conn *proto.Conn, req proto.Msg, n *node.Node, fi
LendID: req.LendID,
LentBytes: req.Lent,
MinTurn: time.Duration(req.MinTurn * float64(time.Second)),
+ Held: time.Duration(req.Held * float64(time.Second)),
ParkTTL: time.Duration(req.ParkTTL * float64(time.Second)),
Window: win,
Containers: req.ContainerIDs(),
@@ -317,6 +336,10 @@ func hold(ctx context.Context, conn *proto.Conn, req proto.Msg, n *node.Node, fi
}
case why := <-lease.Waits():
u.send(proto.Msg{Op: proto.OpWaiting, Reason: why})
+ case d := <-lease.Yields():
+ if !u.send(proto.Msg{Op: proto.OpYield, Yield: d.Seconds()}) {
+ return
+ }
case <-lease.Resumed():
if !u.send(proto.Msg{Op: proto.OpGranted, Lease: lease.ID(), Resume: true}) {
return
@@ -395,7 +418,7 @@ func holdAll(ctx context.Context, u *consumer, req proto.Msg, r broker.Request,
for i, c := range cards {
bs[i], names[i], uuids[i] = c.B, c.Name, c.UUID
}
- r.Lend = false
+ r.Lend, r.LearnSize = false, false
var leases []*broker.Lease
var err error
if req.Op == proto.OpReattach {
diff --git a/internal/server/yield.go b/internal/server/yield.go
new file mode 100644
index 00000000..eb289987
--- /dev/null
+++ b/internal/server/yield.go
@@ -0,0 +1,30 @@
+// Copyright 2026 Trevin Tindall (T-Crypt)
+// SPDX-License-Identifier: Apache-2.0
+
+package server
+
+import (
+ "log/slog"
+
+ "github.com/T-Crypt/berth/internal/node"
+ "github.com/T-Crypt/berth/internal/proto"
+)
+
+// yield answers OpYield. Ending a yield also resumes the engine when a drain
+// parked it during the yield, so the job that yielded gets the card back by
+// the same queue a woken engine takes.
+func yield(conn *proto.Conn, log *slog.Logger, req proto.Msg, n *node.Node) {
+ if req.Engine == "" || req.Yield < 0 {
+ reply(conn, log, proto.Msg{Op: proto.OpRefused, Reason: "an engine name and a yield of 0 or more seconds are required"})
+ return
+ }
+ ids, err := n.Yield(req.Engine, seconds(req.Yield))
+ if err != nil {
+ reply(conn, log, proto.Msg{Op: proto.OpRefused, Reason: err.Error()})
+ return
+ }
+ // Resume says the end queued a parked engine: the caller waits for its
+ // lease.resumed before the job may load onto the card again.
+ resumed := req.Yield == 0 && n.Resume(req.Engine, false) == nil
+ reply(conn, log, proto.Msg{Op: proto.OpYield, Engine: req.Engine, Yield: req.Yield, Leases: ids, Resume: resumed})
+}
diff --git a/internal/server/yield_test.go b/internal/server/yield_test.go
new file mode 100644
index 00000000..65f16bc7
--- /dev/null
+++ b/internal/server/yield_test.go
@@ -0,0 +1,93 @@
+// Copyright 2026 Trevin Tindall (T-Crypt)
+// SPDX-License-Identifier: Apache-2.0
+
+//go:build unix
+
+package server_test
+
+import (
+ "fmt"
+ "net/http"
+ "net/http/httptest"
+ "strings"
+ "testing"
+ "time"
+
+ "github.com/T-Crypt/berth/internal/proto"
+ "github.com/T-Crypt/berth/internal/runner"
+ "github.com/T-Crypt/berth/internal/server"
+)
+
+// #299 end to end: a ComfyUI job waits on an LLM node, and the LLM waits for
+// the card ComfyUI holds. Its drain waits on the job, so neither moves. A
+// yield from inside the job lets the drain through, and the LLM gets the card.
+func TestYieldBreaksTheComfyUILLMDeadlock(t *testing.T) {
+ sock := start(t)
+ comfy := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ if r.URL.Path == "/queue" {
+ fmt.Fprint(w, `{"queue_running":[["1","job"]],"queue_pending":[]}`)
+ return
+ }
+ fmt.Fprint(w, `{}`)
+ }))
+ defer comfy.Close()
+
+ comfyui, comfyOut := launch(runner.Options{Socket: sock, Engine: "comfyui", Priority: "foreground",
+ DrainComfyUI: comfy.URL, Retry: 20 * time.Millisecond, Grace: 200 * time.Millisecond, Argv: []string{"sleep", "30"}})
+ waitFor(t, "comfyui grant", func() bool { return strings.Contains(comfyOut.String(), "granted") })
+
+ llm, llmOut := launch(runner.Options{Socket: sock, Engine: "gemma", Priority: "foreground", Argv: []string{"true"}})
+ waitFor(t, "the drain waiting on the job", func() bool {
+ return strings.Contains(comfyOut.String(), "waiting for ComfyUI to finish")
+ })
+ select {
+ case code := <-llm:
+ t.Fatalf("gemma ended (%d) while the job held the card; stderr:\n%s", code, llmOut.String())
+ case <-time.After(500 * time.Millisecond):
+ }
+
+ conn, raw, err := server.Dial(sock)
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer raw.Close()
+ if err := conn.Send(proto.Msg{Op: proto.OpYield, Engine: "comfyui", Yield: 60}); err != nil {
+ t.Fatal(err)
+ }
+ if m, err := conn.Recv(); err != nil || m.Op != proto.OpYield || len(m.Leases) != 1 {
+ t.Fatalf("yield reply = %+v, %v", m, err)
+ }
+
+ select {
+ case code := <-llm:
+ if code != 0 {
+ t.Fatalf("gemma exit = %d; stderr:\n%s", code, llmOut.String())
+ }
+ case <-time.After(5 * time.Second):
+ t.Fatalf("gemma never got the card after the yield; comfyui:\n%s", comfyOut.String())
+ }
+ if code := <-comfyui; code != 128+15 {
+ t.Fatalf("comfyui exit = %d, want 143 after its drain; stderr:\n%s", code, comfyOut.String())
+ }
+ if !strings.Contains(comfyOut.String(), "yielded the card from inside its job") {
+ t.Fatalf("the run never heard the yield:\n%s", comfyOut.String())
+ }
+}
+
+// A yield names a leased engine; one that has no lease is refused, so a
+// node pointed at the wrong name hears it at once.
+func TestYieldWithNoLeaseIsRefused(t *testing.T) {
+ sock := start(t)
+ conn, raw, err := server.Dial(sock)
+ if err != nil {
+ t.Fatal(err)
+ }
+ defer raw.Close()
+ if err := conn.Send(proto.Msg{Op: proto.OpYield, Engine: "comfyui", Yield: 60}); err != nil {
+ t.Fatal(err)
+ }
+ m, err := conn.Recv()
+ if err != nil || m.Op != proto.OpRefused || !strings.Contains(m.Reason, "no running engine named comfyui") {
+ t.Fatalf("reply = %+v, %v", m, err)
+ }
+}
diff --git a/internal/skill/SKILL.md b/internal/skill/SKILL.md
index 498ffd28..94feef94 100644
--- a/internal/skill/SKILL.md
+++ b/internal/skill/SKILL.md
@@ -18,7 +18,12 @@ berth status # the card, leases, waiters, stowaways, claims, the boar
berth status --format json | jq '.leases, .waiting, .squatters'
berth claims # games and apps Berth holds the card for, or names
berth engine check -v # what every engine reports right now
+berth contract ENGINE # one engine's VRAM, RAM, turn and switch times, each declared, measured or unknown
+berth fit MODEL --explain # where a request would run now (running, resume, free, drain, blocked) and why
+berth queue # waiters in grant order; bump/prio/drop are the operator's, never an agent's own call
+berth schedule # what holds the card, the queue, and every booking for the next 14 days
berth events -f # live: grants, drains, refusals, stowaways
+berth why ID|ENGINE # one lease's story: why it waited, was drained or refused, across restarts
```
If `berth status` says berthd is unreachable, Berth is not coordinating this card: say so and
@@ -32,6 +37,9 @@ do not assume any engine is protected.
gateway), or it can load onto a card that is already full and crash both.
- A **pinned** lease (a hold, a claim, `berth run --pinned`) is never drained. If a request waits
"held by ... pinned", the user's game or app has the card. Tell them; do not work around it.
+- A refusal that says a card "is locked for training", or that a run "is training on this node", means a
+ `berth train` run has it. Do not retry in a loop, start the engine another way, restart berthd, or stop the
+ training. Tell the user, or use another card or node.
- Do not edit berth.toml, systemd units or an engine's config without showing the change and its
undo first. Run `berth config check` after any berth.toml edit.
@@ -39,12 +47,14 @@ do not assume any engine is protected.
| Need | Command |
|---|---|
-| Run an engine under a lease | `berth run --as NAME [--priority batch\|interactive\|foreground] [--needs-mib N] -- CMD ARGS` |
+| Run an engine under a lease | `berth run --as NAME [--priority batch\|interactive\|foreground] [--needs-mib N \| --learn-size] -- CMD ARGS` |
| Run an engine on one card of several | `berth run --as NAME --gpu gpu1 -- CMD`, or `gpu = "gpu1"` in its engine entry (`berth status` names the cards; the engine gets `CUDA_VISIBLE_DEVICES` for that card unless already set) |
| Run an engine across several cards | `berth run --as NAME --gpu gpu0,gpu1 -- CMD`: one lease, granted on every card at once. It cannot park or lend yet |
| Run a configured engine without retyping its command | `berth run --as NAME` with no command runs the `start` of NAME's `[[engine]]` entry |
-| Run an app that must not be drained (Blender, a training job) | `berth run --as NAME --pinned -- CMD` |
+| Train or fine-tune a model (nothing automated may interrupt it) | `berth train --as NAME [--gpu gpu0,gpu1] -- CMD`: the whole card, no wait deadline, and every other request on its cards is refused until it exits. Another `berth train` waits in line |
+| Run an app that must not be drained but may share the card (Blender) | `berth run --as NAME --pinned -- CMD` |
| Give an engine a minimum turn before a request can drain it | `berth run --as NAME --min-turn 5m -- CMD`, or `min_turn = "5m"` in its engine entry (a hold still wins; `berth status` shows the turn left in the TURN column) |
+| Plan your own turn on the card (a nightly job, a long run) | `POST /v1/berth/schedule` on the gateway with `{"who":"ME","engine":"NAME","start":"23:00","for":"5h"}`, or `berth schedule book --who ME --engine NAME --at 23:00 --for 5h`. Read `berth schedule` first and pick free hours; a clash is a 409 naming the booking it hits. Book only your own runs: downtime and priority changes are the operator's |
| Free the card for the user (game, call) | `berth hold [--for 2h] [--reason game] [--gpu gpu1]` (ends on Ctrl-C; default the first card) |
| Call a model and let Berth start its engine | the gateway: `http://127.0.0.1:9799/v1`, bearer key from `berth key` |
| Wrap every llama-swap model at once | `berth wire llama-swap --config CFG` (plan), `--write` (apply, keeps a backup), `--write --update` (rewrite wire's flags on lines it wrote, keeping flags added by hand), `--wait 10m` (every model waits that long for the card) |
@@ -141,7 +151,8 @@ builtin = true # false turns the built-in pack off
1. `berth status`: is something else leased, waiting, or a stowaway?
2. A refusal reason names who holds the card and why the request could not fit
("needs 20480 MiB; at most 14336 MiB can be freed; claims hold 10240 MiB").
-3. `berth events --format json` after the failure shows the drain and grant sequence.
+3. `berth why ENGINE` after the failure tells that engine's last lease, wait by wait; `berth events --format json`
+ shows the whole drain and grant sequence.
4. Fix the cause (a smaller quant, `--needs-mib`, a priority, the user's hold ending), never by
killing the holder.
5. A lease stuck after its engine exited: `berth reclaim` lists what the lease recorded. Only with
diff --git a/internal/wire/wire.go b/internal/wire/wire.go
index a6284c06..7ea01d54 100644
--- a/internal/wire/wire.go
+++ b/internal/wire/wire.go
@@ -76,11 +76,22 @@ type Model struct {
Line string `json:"line,omitempty" yaml:"line,omitempty"` // the wrapper, for add and update
}
+// SwapGroup is a llama-swap group whose wired members llama-swap swaps
+// itself: loading one unloads the other, a voluntary release to Berth, so
+// min_turn, the ranking and the juggle guard never decide between them.
+// Group is "" for llama-swap's default group, which holds every model no
+// group names.
+type SwapGroup struct {
+ Group string `json:"group" yaml:"group"`
+ Members []string `json:"members" yaml:"members"`
+}
+
// Result is one run over one file.
type Result struct {
- Path string `json:"path" yaml:"path"`
- Models []Model `json:"models" yaml:"models"`
- Changed bool `json:"changed" yaml:"changed"` // any add or update
+ Path string `json:"path" yaml:"path"`
+ Models []Model `json:"models" yaml:"models"`
+ Changed bool `json:"changed" yaml:"changed"` // any add or update
+ Swapping []SwapGroup `json:"swapping,omitempty" yaml:"swapping,omitempty"`
content string // the edited file; NewContent reads it
}
@@ -130,7 +141,7 @@ func LlamaSwap(pathArg string, o Options) (*Result, error) {
// Every plan reads the original lines; the edits are applied afterwards,
// bottom-up, so one model's insert never moves another model's target.
r := &Result{Path: pathArg, Models: make([]Model, 0, len(names))}
- wrapped := map[string]string{} // model -> the fragment its cmd must carry
+ wrapped := map[string]bool{} // models whose cmd must run under berth run --as
var plans []editPlan
for i, name := range names {
end := blockEnd
@@ -143,7 +154,7 @@ func LlamaSwap(pathArg string, o Options) (*Result, error) {
}
switch m.Action {
case "wired", "add", "update":
- wrapped[name] = "run --as " + shellWord(name) + " "
+ wrapped[name] = true
}
r.Models = append(r.Models, m)
}
@@ -161,6 +172,7 @@ func LlamaSwap(pathArg string, o Options) (*Result, error) {
}
r.Changed = r.Count("add")+r.Count("update") > 0
r.content = joinLines(lines, trailingNL)
+ r.Swapping = swapping(text, names, wrapped)
// Verify against a parse, plan or not: the walker's output is the
// product, and a silent formatting surprise is the one failure worth
@@ -278,11 +290,11 @@ func planModel(lines []string, keyRow, end int, name string, o Options) (Model,
if !o.Update {
return Model{Name: name, Action: "wired", Reason: "already runs under berth run"}, nil
}
- kept, ok := handFlags(lines[wrapRow], o)
+ lead, kept, ok := handFlags(lines[wrapRow], o)
if !ok {
return Model{Name: name, Action: "wired", Reason: "the berth run line does not end at --; edit it by hand"}, nil
}
- desired := wrapperLine(name, o, wrapIndent, lineEnding(lines[wrapRow]), kept...)
+ desired := wrapperLine(name, o, wrapIndent, lineEnding(lines[wrapRow]), lead, kept...)
if lines[wrapRow] == desired {
return Model{Name: name, Action: "wired", Reason: "already runs under berth run with these flags"}, nil
}
@@ -305,16 +317,21 @@ func planModel(lines []string, keyRow, end int, name string, o Options) (Model,
if at < len(lines) {
ending = lineEnding(lines[at])
}
- line := wrapperLine(name, o, insertIndent, ending)
+ line := wrapperLine(name, o, insertIndent, ending, nil)
return Model{Name: name, Action: "add", Line: line}, &editPlan{at: at, line: line}
}
-// wrapperLine is the full line inserted into the block: berth run, the flags,
-// the operator's own flags (kept), and the -- that ends berth's own flag
-// list. The model's original cmd follows on the next line.
-func wrapperLine(name string, o Options, indent int, ending string, kept ...string) string {
+// wrapperLine is the full line inserted into the block: the operator's words
+// before berth (lead), berth run, the flags, the operator's own flags (kept),
+// and the -- that ends berth's own flag list. The model's original cmd
+// follows on the next line.
+func wrapperLine(name string, o Options, indent int, ending string, lead []string, kept ...string) string {
var b strings.Builder
b.WriteString(strings.Repeat(" ", indent))
+ for _, w := range lead {
+ b.WriteString(shellWord(w))
+ b.WriteString(" ")
+ }
b.WriteString(shellWord(o.Bin))
b.WriteString(" run --as ")
b.WriteString(shellWord(name))
@@ -346,18 +363,20 @@ func wrapperLine(name string, o Options, indent int, ending string, kept ...stri
return b.String()
}
-// handFlags is the words of a berth run line that wire did not write: every
-// word between run and the closing --, less the flags wire writes and their
-// values (each of them takes one). A --wait is wire's only when o.Wait is
-// set. ok is false when the line has no closing --.
-func handFlags(line string, o Options) (kept []string, ok bool) {
+// handFlags is the words of a berth run line that wire did not write: lead,
+// the words before the berth binary (a wrapper that runs berth run), and
+// kept, every word between run and the closing --, less the flags wire
+// writes and their values (each of them takes one). A --wait is wire's only
+// when o.Wait is set. ok is false when the line has no closing --.
+func handFlags(line string, o Options) (lead, kept []string, ok bool) {
owned := map[string]bool{"as": true, "group": true, "priority": true, "drain-post": true, "drain-slots": true, "wait": o.Wait > 0}
fields := shellFields(line)
run := slices.Index(fields, "run")
end := slices.Index(fields, "--")
- if run < 0 || end < run {
- return nil, false
+ if run < 1 || end < run {
+ return nil, nil, false
}
+ lead = fields[:run-1]
for i := run + 1; i < end; i++ {
name, _, eq := strings.Cut(strings.TrimLeft(fields[i], "-"), "=")
if strings.HasPrefix(fields[i], "-") && owned[name] {
@@ -368,7 +387,7 @@ func handFlags(line string, o Options) (kept []string, ok bool) {
}
kept = append(kept, fields[i])
}
- return kept, true
+ return lead, kept, true
}
// wrappedAs names the model a line wraps in `berth run --as NAME`, or "". The
@@ -376,8 +395,8 @@ func handFlags(line string, o Options) (kept []string, ok bool) {
// name all count.
func wrappedAs(line string) string {
fields := shellFields(line)
- for i := 1; i+2 < len(fields); i++ {
- if fields[i] != "run" || fields[i+1] != "--as" {
+ for i := 1; i < len(fields); i++ {
+ if fields[i] != "run" {
continue
}
base := fields[i-1]
@@ -387,7 +406,14 @@ func wrappedAs(line string) string {
if base != "berth" && !strings.EqualFold(base, "berth.exe") {
continue
}
- return fields[i+2]
+ for j := i + 1; j < len(fields) && fields[j] != "--"; j++ {
+ if fields[j] == "--as" && j+1 < len(fields) {
+ return fields[j+1]
+ }
+ if name, ok := strings.CutPrefix(fields[j], "--as="); ok {
+ return name
+ }
+ }
}
return ""
}
@@ -480,9 +506,70 @@ func roster(text string) ([]string, error) {
return nil, nil
}
+// swapping lists the groups where llama-swap swaps two or more wired models
+// itself, in roster order. Groups come from routing.router.settings.groups,
+// or the older top-level groups:; swap defaults to true, and a model no group
+// names sits in the default group, which swaps.
+func swapping(text string, names []string, wired map[string]bool) []SwapGroup {
+ var cfg struct {
+ Groups map[string]groupCfg `yaml:"groups"`
+ Routing struct {
+ Router struct {
+ Settings struct {
+ Groups map[string]groupCfg `yaml:"groups"`
+ } `yaml:"settings"`
+ } `yaml:"router"`
+ } `yaml:"routing"`
+ }
+ if yaml.Unmarshal([]byte(text), &cfg) != nil {
+ return nil
+ }
+ groups := cfg.Routing.Router.Settings.Groups
+ if groups == nil {
+ groups = cfg.Groups
+ }
+ groupOf := map[string]string{}
+ for g, c := range groups {
+ for _, m := range c.Members {
+ groupOf[m] = g
+ }
+ }
+ var out []SwapGroup
+ at := map[string]int{}
+ for _, name := range names {
+ if !wired[name] {
+ continue
+ }
+ g := groupOf[name]
+ if g != "" && groups[g].Swap != nil && !*groups[g].Swap {
+ continue
+ }
+ i, ok := at[g]
+ if !ok {
+ i = len(out)
+ at[g] = i
+ out = append(out, SwapGroup{Group: g})
+ }
+ out[i].Members = append(out[i].Members, name)
+ }
+ kept := out[:0]
+ for _, s := range out {
+ if len(s.Members) > 1 {
+ kept = append(kept, s)
+ }
+ }
+ return kept
+}
+
+// groupCfg is the part of a llama-swap group swapping reads.
+type groupCfg struct {
+ Swap *bool `yaml:"swap"`
+ Members []string `yaml:"members"`
+}
+
// verify re-parses the edited text and checks the roster order and every
// wrappable model's cmd against the parse, not against the line walker.
-func verify(text string, wantOrder []string, want map[string]string) error {
+func verify(text string, wantOrder []string, want map[string]bool) error {
names, err := roster(text)
if err != nil {
return fmt.Errorf("edited file does not parse: %w", err)
@@ -507,12 +594,11 @@ func verify(text string, wantOrder []string, want map[string]string) error {
models := doc.Content[i+1]
for j := 0; j+1 < len(models.Content); j += 2 {
name := models.Content[j].Value
- frag, ok := want[name]
- if !ok {
+ if !want[name] {
continue
}
- if !strings.Contains(modelCmd(models.Content[j+1]), frag) {
- return fmt.Errorf("edited cmd of %q does not carry %q", name, frag)
+ if wrappedAs(modelCmd(models.Content[j+1])) != name {
+ return fmt.Errorf("edited cmd of %q does not run under berth run --as %q", name, name)
}
}
}
diff --git a/internal/wire/wire_test.go b/internal/wire/wire_test.go
index 0677e907..13105661 100644
--- a/internal/wire/wire_test.go
+++ b/internal/wire/wire_test.go
@@ -4,6 +4,7 @@
package wire
import (
+ "fmt"
"os"
"path/filepath"
"runtime"
@@ -366,3 +367,127 @@ func TestWriteBacksUpAndReplaces(t *testing.T) {
t.Errorf("Write on a clean result = (%q, %v), want (\"\", nil)", b, err)
}
}
+
+// llama-swap swaps the members of a swap group itself, out of Berth's turns.
+// The result names each group holding two or more wired models that swap: the
+// default group when no group names them, a group without swap: false, the
+// older top-level groups: form too. Unwired models and swap: false groups are
+// left out.
+func TestLlamaSwapNamesTheGroupsItSwaps(t *testing.T) {
+ models := `models:
+ "a":
+ cmd: >
+ llama-server -m a.gguf
+ "b":
+ cmd: >
+ llama-server -m b.gguf
+ "c":
+ cmd: >
+ llama-server -m c.gguf
+ "d":
+ proxy: http://127.0.0.1:8000
+`
+ cases := []struct {
+ name, groups, want string
+ }{
+ {"no groups", "", `[{ [a b c]}]`},
+ {"routing group swaps", `routing:
+ router:
+ use: group
+ settings:
+ groups:
+ "chat":
+ members: ["a", "b", "d"]
+ "solo":
+ swap: true
+ members: ["c"]
+`, `[{chat [a b]}]`},
+ {"routing group swap off", `routing:
+ router:
+ use: group
+ settings:
+ groups:
+ "chat":
+ swap: false
+ exclusive: false
+ members: ["a", "b", "c"]
+`, `[]`},
+ {"top-level groups", `groups:
+ "chat":
+ swap: true
+ members: ["b", "c"]
+`, `[{chat [b c]}]`},
+ }
+ for _, tc := range cases {
+ t.Run(tc.name, func(t *testing.T) {
+ r, err := LlamaSwap(writeTemp(t, models+tc.groups), options())
+ if err != nil {
+ t.Fatal(err)
+ }
+ got := fmt.Sprint(r.Swapping)
+ if got != tc.want {
+ t.Fatalf("Swapping = %s, want %s", got, tc.want)
+ }
+ })
+ }
+}
+
+// A hand flag before --as still names the wrapped model: the line counts as
+// wired, and --update keeps the flag on one wrapper instead of wrapping the
+// model a second time.
+func TestLlamaSwapHandFlagBeforeAs(t *testing.T) {
+ in := `models:
+ "a":
+ cmd: >
+ /usr/bin/berth run --min-turn 5m --as a --drain-post http://127.0.0.1:9090/api/models/unload/a --
+ llama-server -m a.gguf
+`
+ r, err := LlamaSwap(writeTemp(t, in), options())
+ if err != nil {
+ t.Fatal(err)
+ }
+ if got := r.Models[0].Action; got != "wired" {
+ t.Fatalf("action = %q, want wired", got)
+ }
+ o := options()
+ o.Update = true
+ r, err = LlamaSwap(writeTemp(t, in), o)
+ if err != nil {
+ t.Fatal(err)
+ }
+ want := `models:
+ "a":
+ cmd: >
+ /usr/bin/berth run --as a --group llama-swap --priority foreground --min-turn 5m --drain-post http://127.0.0.1:9090/api/models/unload/a --drain-slots http://127.0.0.1:9090/upstream/a --
+ llama-server -m a.gguf
+`
+ if got := r.NewContent(); got != want {
+ t.Fatalf("updated:\n%s\nwant:\n%s", got, want)
+ }
+}
+
+// --update rewrites wire's flags from the berth binary on and keeps the words
+// before it, such as a wrapper that runs berth run.
+func TestLlamaSwapUpdateKeepsWordsBeforeBerth(t *testing.T) {
+ in := `models:
+ "a":
+ cmd: >
+ /usr/bin/clear-first FOO=1 /usr/bin/berth run --as a --drain-post http://127.0.0.1:9090/api/models/unload/a --
+ llama-server -m a.gguf
+`
+ o := options()
+ o.Update = true
+ r, err := LlamaSwap(writeTemp(t, in), o)
+ if err != nil {
+ t.Fatal(err)
+ }
+ want := `models:
+ "a":
+ cmd: >
+ /usr/bin/clear-first FOO=1 /usr/bin/berth run --as a --group llama-swap --priority foreground --drain-post http://127.0.0.1:9090/api/models/unload/a --drain-slots http://127.0.0.1:9090/upstream/a --
+ llama-server -m a.gguf
+`
+ if got := r.NewContent(); got != want {
+ t.Fatalf("updated:\n%s\nwant:\n%s", got, want)
+ }
+}