Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,10 @@ has a matching GitHub release whose body comes from its section below.
or a pinned lease. Each candidate names its reason, the tier its weights sit in, and the switch time the gateway
measured from that tier, or unknown. For a model, every engine that serves it is ranked the way the gateway picks
one. `--explain` lists every candidate. It is read only, and the gateway now picks through the same ranking.
- `berth queue` lists the waiters in the order berthd will grant them, with their lease IDs. `berth queue bump ID`
moves one to the front of its priority class, `berth queue prio ID batch|interactive|foreground` moves it to
another class, and `berth queue drop ID` refuses it now. Holds and trainings cannot be changed this way. Each
change is a `lease.queue_override` event.
- `berth wire llama-swap` names each llama-swap group where llama-swap swaps two or more wired models itself.
There llama-swap unloads the old model before Berth sees the request, so `min_turn`, the engine ranking and the
juggle guard never apply between them. The JSON carries the groups as `swapping`. The llama-swap section of
Expand Down
2 changes: 1 addition & 1 deletion docs/interfaces.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

- **CLI:** one command tree with one output contract. `berth status`, `berth events -f`,
`berth run`, `berth hold`, `berth stop`, `berth cordon|uncordon`, `berth notify`, `berth claims`, `berth pull`, `berth verify`, `berth models`, `berth gateway`,
`berth wire llama-swap`, `berth engine check`, `berth contract ENGINE`, `berth fit NAME`, `berth config check`, `berth config show`,
`berth wire llama-swap`, `berth engine check`, `berth contract ENGINE`, `berth fit NAME`, `berth queue`, `berth config check`, `berth config show`,
`berth export`, `berth key`, `berth skill install|show`, `berth update`, `berth version`,
and for the node itself `berth daemon`, `berth harbor`, `berth service install|uninstall|status`,
`berth reclaim`, `berth resume`, `berth prefetch`, `berth storage`, `berth civitai search|versions`
Expand Down
16 changes: 16 additions & 0 deletions docs/leasing.md
Original file line number Diff line number Diff line change
Expand Up @@ -163,6 +163,22 @@
- A hold cannot free memory held by something outside Berth: that memory is still reported,
as contested.

## Reorder the queue

Berth grants waiters by priority first, then in the order they arrived. Many devices can send work to one
card and nobody has to sort it out. When you want a different order, `berth queue` changes it:

```sh
berth queue # the waiters in grant order, with their lease IDs
berth queue bump llama-swap-41 # first in its priority class
berth queue prio comfyui-12 foreground
berth queue drop tagger-7 # refuse it now, as a timeout would
```

A bump puts the waiter ahead of everyone else in its priority class. To go ahead of a higher class, change its
priority with `prio`. Holds and trainings cannot be changed here: a hold is a pin, not a place in the queue, and
a training's place is fixed by the card's lock. Each change is a `lease.queue_override` event in `berth events`.

## Cordon a card

On a node with several cards, `berth cordon gpu1` stops new grants on that card while the others keep serving, for
Expand Down
11 changes: 9 additions & 2 deletions internal/broker/broker.go
Original file line number Diff line number Diff line change
Expand Up @@ -333,6 +333,9 @@ type lease struct {
// two leases granted in one quantum would otherwise tie, and which one
// is older, and so drained first, would be arbitrary.
arrivedN, grantedN, parkedN uint64
// bumpedN is set by berth queue bump: within its priority class the
// latest bump goes first, ahead of every waiter never bumped.
bumpedN uint64
// parkedAt is when the lease became parked; parkedCall is when the
// runner said its park call was done (the parked transition waits for
// a later card read). resumed marks a waiting lease that came from
Expand Down Expand Up @@ -1251,13 +1254,17 @@ func (b *Broker) grant(w *lease, now time.Time) {
b.log.Info("granted", "lease", w.id, "engine", w.engine, "priority", w.prio, "needs_mib", w.needs/gpu.MiB, "waited", now.Sub(w.arrived).Round(time.Millisecond))
}

// sortQueue orders the waiters: higher priority first, then by arrival on
// the node. b.mu must be held.
// sortQueue orders the waiters: higher priority first, then the operator's
// latest bump (berth queue bump), then by arrival on the node. b.mu must be
// held.
func (b *Broker) sortQueue() {
sort.SliceStable(b.queue, func(i, j int) bool {
if b.queue[i].prio != b.queue[j].prio {
return b.queue[i].prio > b.queue[j].prio
}
if b.queue[i].bumpedN != b.queue[j].bumpedN {
return b.queue[i].bumpedN > b.queue[j].bumpedN
}
return b.queue[i].arrivedN < b.queue[j].arrivedN
})
}
Expand Down
77 changes: 77 additions & 0 deletions internal/broker/override.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
// Copyright 2026 Trevin Tindall (T-Crypt)
// SPDX-License-Identifier: Apache-2.0

package broker

// The operator's hand on the queue. Berth orders waiters by priority, then
// arrival, and nobody has to manage it; berth queue is for the times the
// operator wants it otherwise. Every change is an event, so the log shows
// that a person moved the line, not the scheduler.

import (
"fmt"

"github.com/T-Crypt/berth/internal/proto"
)

// QueueAction is one operator change to a waiter.
type QueueAction string

const (
// QueueBump moves the waiter to the front of its priority class.
QueueBump QueueAction = "bump"
// QueuePrio moves the waiter to another priority class.
QueuePrio QueueAction = "prio"
// QueueDrop refuses the waiter, the way a timeout would.
QueueDrop QueueAction = "drop"
)

// dropReason is what a dropped waiter is told.
const dropReason = "dropped from the queue by the operator (berth queue drop)"

// Queue applies one operator change to the waiter with lease ID id. prio is
// the new class for QueuePrio: batch, interactive or foreground. A hold is a
// pin, not a queue position, and a training's place is fixed by the lock, so
// neither can be set here. The returned line says what changed.
func (b *Broker) Queue(id string, action QueueAction, prio string) (string, error) {
b.mu.Lock()
defer b.mu.Unlock()
var w *lease
for _, q := range b.queue {
if q.id == id {
w = q
}
}
if w == nil {
return "", notFound(fmt.Sprintf("no waiter %s on this card", id))
}
var detail string
switch action {
case QueueBump:
w.bumpedN = b.order()
b.sortQueue()
detail = "moved to the front of its class by the operator"
case QueuePrio:
if w.training {
return "", fmt.Errorf("%s is a training: its place is fixed by the card's lock", id)
}
p, err := ParsePriority(prio)
if err != nil || p == Hold {
return "", fmt.Errorf("priority %q: use batch, interactive or foreground", prio)
}
detail = fmt.Sprintf("priority %s -> %s by the operator", w.prio, p)
w.prio = p
b.sortQueue()
case QueueDrop:
select {
case w.lockCh <- dropReason:
default:
}
detail = "dropped by the operator"
default:
return "", fmt.Errorf("unknown queue action %q (bump, prio, drop)", action)
}
b.emit(proto.Event{Kind: proto.EvQueueOverride, Lease: w.id, Engine: w.engine, Detail: detail})
b.log.Info("queue override", "lease", w.id, "engine", w.engine, "change", detail)
return detail, nil
}
109 changes: 109 additions & 0 deletions internal/broker/override_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,109 @@
// Copyright 2026 Trevin Tindall (T-Crypt)
// SPDX-License-Identifier: Apache-2.0

package broker

import (
"context"
"errors"
"slices"
"strings"
"testing"
"time"

"github.com/T-Crypt/berth/internal/procs"
"github.com/T-Crypt/berth/internal/proto"
)

// queueOrder is the queue as the status reports it, engine names in order.
func queueOrder(b *Broker) []string {
var out []string
for _, l := range b.Status().Waiting {
out = append(out, l.Engine)
}
return out
}

func waiterID(t *testing.T, b *Broker, engine string) string {
t.Helper()
for _, l := range b.Status().Waiting {
if l.Engine == engine {
return l.ID
}
}
t.Fatalf("no waiter %s in %v", engine, queueOrder(b))
return ""
}

// The operator reorders the queue: a bump puts a waiter first in its class,
// a prio moves it to another class, a drop refuses it. Each is an event, and
// the next grant follows the new order.
func TestQueueOverride(t *testing.T) {
b, _, _ := setup(t, procs.Map{})
holder := must(t, acquire(b, Request{Engine: "holder", Priority: Foreground, Pinned: true}))
a := acquire(b, Request{Engine: "a", Priority: Interactive})
time.Sleep(10 * time.Millisecond)
bb := acquire(b, Request{Engine: "bb", Priority: Interactive})
time.Sleep(10 * time.Millisecond)
c := acquire(b, Request{Engine: "c", Priority: Batch})
time.Sleep(20 * time.Millisecond)
if got := queueOrder(b); !slices.Equal(got, []string{"a", "bb", "c"}) {
t.Fatalf("queue = %v, want priority then arrival", got)
}

if _, err := b.Queue(waiterID(t, b, "bb"), QueueBump, ""); err != nil {
t.Fatal(err)
}
if got := queueOrder(b); !slices.Equal(got, []string{"bb", "a", "c"}) {
t.Fatalf("after bump bb: %v", got)
}
detail, err := b.Queue(waiterID(t, b, "c"), QueuePrio, "foreground")
if err != nil || detail != "priority batch -> foreground by the operator" {
t.Fatalf("prio c: %q, %v", detail, err)
}
if got := queueOrder(b); !slices.Equal(got, []string{"c", "bb", "a"}) {
t.Fatalf("after prio c: %v", got)
}

if _, err := b.Queue(waiterID(t, b, "c"), QueuePrio, "hold"); err == nil {
t.Fatal("prio hold was accepted")
}
if _, err := b.Queue("nope-9", QueueBump, ""); !IsNotFound(err) {
t.Fatalf("unknown id: %v, want not found", err)
}

if _, err := b.Queue(waiterID(t, b, "a"), QueueDrop, ""); err != nil {
t.Fatal(err)
}
select {
case r := <-a:
if !errors.Is(r.err, ErrRefused) || !strings.Contains(r.err.Error(), "dropped from the queue by the operator") {
t.Fatalf("dropped a: %v", r.err)
}
case <-time.After(time.Second):
t.Fatal("dropped waiter still waiting")
}

var overrides []string
events, _, _ := b.EventLog().Events(0, false)
for _, e := range events {
if e.Kind == proto.EvQueueOverride {
overrides = append(overrides, e.Engine+": "+e.Detail)
}
}
want := []string{
"bb: moved to the front of its class by the operator",
"c: priority batch -> foreground by the operator",
"a: dropped by the operator",
}
if !slices.Equal(overrides, want) {
t.Fatalf("override events = %q, want %q", overrides, want)
}

holder.Release()
b.Tick(context.Background())
l := must(t, c)
pending(t, bb)
l.Release()
must(t, bb)
}
18 changes: 18 additions & 0 deletions internal/cli/commands.go
Original file line number Diff line number Diff line change
Expand Up @@ -398,6 +398,24 @@ Gated and private models read a token from $HF_TOKEN or $CIVITAI_API_KEY.`,
Formats: []Format{FormatHuman, FormatJSON, FormatYAML},
Run: runPull,
},
{
Name: "queue",
Summary: "the waiters in grant order; bump, re-rank or drop one",
Args: "[bump ID | prio ID batch|interactive|foreground | drop ID] [--socket P] [--format F]",
Help: `Without arguments, list every waiter in the order berthd will grant them:
priority first, then arrival. Nobody has to manage the queue; these are for
when you want it otherwise.

bump ID move the waiter to the front of its priority class
prio ID P move it to another class: batch, interactive or foreground
drop ID refuse it now, the way a timeout would

A hold is a pin, not a queue place, and a training's place is fixed by the
card's lock, so neither can be set here. Every change is journaled as a
lease.queue_override event (berth events).`,
Formats: []Format{FormatHuman, FormatJSON, FormatYAML},
Run: runQueue,
},
{
Name: "reclaim",
Summary: "what a stuck lease left on the card; --yes stops it",
Expand Down
102 changes: 102 additions & 0 deletions internal/cli/queue.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
// Copyright 2026 Trevin Tindall (T-Crypt)
// SPDX-License-Identifier: Apache-2.0

package cli

import (
"fmt"
"text/tabwriter"

"github.com/T-Crypt/berth/internal/proto"
"github.com/T-Crypt/berth/internal/server"
)

// queueView is the waiters in the order berthd will grant them.
type queueView struct {
waiting []proto.Lease
cards int
}

func (v queueView) Data() any { return v.waiting }

func (v queueView) Human(c *Context, ok bool) error {
if len(v.waiting) == 0 {
c.Printf("%s nobody is waiting\n", c.Style.OK(markOK))
return nil
}
tw := tabwriter.NewWriter(c.Out, 0, 4, 2, ' ', 0)
fmt.Fprint(tw, "POS\tID\tENGINE\tPRIORITY\t"+claimCardCol(v.cards, "CARD")+"WAITING\n")
for i, l := range v.waiting {
prio := l.Priority
if l.Training {
prio += ",training"
}
fmt.Fprintf(tw, "%d\t%s\t%s\t%s\t%s%s\n", i+1, l.ID, l.Holder(), prio, claimCardCol(v.cards, l.Card), age(l.Age))
}
return tw.Flush()
}

// queueChange is the reply to bump, prio and drop.
type queueChange struct {
Lease string `json:"lease" yaml:"lease"`
Change string `json:"change" yaml:"change"`
}

type queueChangeView struct{ c queueChange }

func (v queueChangeView) Data() any { return v.c }

func (v queueChangeView) Human(c *Context, ok bool) error {
c.Printf("%s %s: %s\n", c.Style.OK(markOK), v.c.Lease, v.c.Change)
return nil
}

func runQueue(c *Context) int {
fs := c.FlagSet()
sock := fs.String("socket", defaultSocket(), "berthd socket ($BERTH_SOCKET)")
if code, done := c.ParseInterspersed(fs); done {
return code
}
usage := "usage: berth queue | berth queue bump ID | berth queue prio ID batch|interactive|foreground | berth queue drop ID"
msg := proto.Msg{Op: proto.OpQueue}
switch args := c.Args; {
case len(args) == 0:
msg = proto.Msg{Op: proto.OpStatus}
case len(args) == 2 && (args[0] == "bump" || args[0] == "drop"):
msg.Action, msg.Lease = args[0], args[1]
case len(args) == 3 && args[0] == "prio":
msg.Action, msg.Lease, msg.Priority = args[0], args[1], args[2]
default:
return c.fail(Failf("berth queue: unexpected arguments %q", args).WithHint("%s", usage))
}

conn, raw, err := server.Dial(*sock)
if err != nil {
return c.fail(Systemf("berthd unreachable on %s: %v", *sock, err))
}
defer raw.Close()
if err := conn.Send(msg); err != nil {
return c.fail(Systemf("berthd on %s: %v", *sock, err))
}
m, err := conn.Recv()
if err != nil {
return c.fail(Systemf("no reply from berthd: %v", err))
}
switch {
case m.Op == proto.OpRefused:
return c.fail(Failf("%s", m.Reason).WithHint("berth queue lists the waiters and their IDs"))
case msg.Op == proto.OpStatus:
if m.Status == nil {
return c.fail(Systemf("no status from berthd: empty reply"))
}
w := m.Status.Waiting
if w == nil {
w = []proto.Lease{}
}
return c.Write(queueView{waiting: w, cards: scheduledCards(m.Status)}, nil)
case m.Op != proto.OpQueue:
return c.fail(Systemf("berthd answered %q: it may be older than berth queue", m.Op).
WithHint("restart berthd on this build"))
}
return c.Write(queueChangeView{queueChange{Lease: m.Lease, Change: m.Reason}}, nil)
}
Loading
Loading