Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,19 @@ has a matching GitHub release whose body comes from its section below.

### Added

- A schedule for each card, up to 14 days ahead. `berth schedule` shows what holds each card, the waiters in grant
order and every booking. `berth schedule book` books an engine's run (it goes first, runs at the booking's
priority, and keeps the card against everything but a hold), a priority change for set hours, or downtime (no
new work, holders drain at the start). `move` and `cancel` change a booking, and `--days` and `--earliest`
repeat one or find free hours. Hours nobody booked run by priority, then arrival, as before. berthd keeps the
bookings in `schedule.json`.
- The gateway's `GET`, `POST`, `PATCH` and `DELETE /v1/berth/schedule` let agents and devices read the scheduler
and book, move or cancel their own runs. Downtime and priority changes stay the operator's.
- The harbor's SCHEDULE tab: what runs and waits now, a 7-day board of 24-hour tracks with every booking to the
minute, each engine's window and the live holder, and a list of bookings. A paired device books by clicking an
empty hour and moves or cancels a booking by clicking it. Each booking form says what the booking will do before
you save it.

- `berth train --as NAME [--gpu CARDS] -- CMD` runs a training or fine-tune that nothing automated can interrupt.
It takes the whole of each card it names, waits for them with no deadline, and keeps them until the command
exits. While it holds a card or waits for it, berthd refuses every other request on that card at once: gateway
Expand Down
28 changes: 28 additions & 0 deletions docs/gateway.md
Original file line number Diff line number Diff line change
Expand Up @@ -309,6 +309,34 @@ on the node (`card`, such as `gpu1`), its base URL on this gateway (`base`, `/ca
`berth cordon` closed it, and the holders on that card alone. Each engine names the cards its entry runs on
(`cards`), so a board for several cards can put each engine under its card.

## Book the card from an agent

An agent, a timer or another device books its own run on the card's schedule (read
[Book the card](leasing.md#book-the-card)) with the node's key:

```sh
K="Authorization: Bearer $(berth key)"
curl -s http://127.0.0.1:9799/v1/berth/schedule -H "$K"
curl -s http://127.0.0.1:9799/v1/berth/schedule -H "$K" \
-d '{"who":"pi-pocket","engine":"pocket-train","start":"23:00","for":"5h","priority":"batch","days":7}'
curl -s -X PATCH http://127.0.0.1:9799/v1/berth/schedule/b4 -H "$K" -d '{"start":"01:00"}'
curl -s -X DELETE http://127.0.0.1:9799/v1/berth/schedule/b4 -H "$K"
```

- `GET` is the scheduler in one read: `now`, `running` (the leases that hold a card), `queued` (the waiters in
grant order) and `bookings` (every booking that has not ended, by start). A lease that a booking governs names it
in `booking`.
- `POST` books a run and answers `201` with `bookings`, the bookings it made. `start` takes what
`berth schedule --at` takes, and `for` is a length such as `5h` or `90m`. `days`, `earliest`, `card`, `priority`
and `note` are optional. A clash or an invalid booking is a `409` with the reason, such as
`run Sat 23:30-Sun 00:30 clashes with b1 (run for pi-pocket, Sat 23:00-Sun 04:00)`.
- `PATCH /v1/berth/schedule/ID` moves a run. It takes `start`, `for`, `priority` and `note`, keeps the rest and
keeps the ID. `DELETE` cancels a run. Both answer `200` with the booking.

The gateway books, moves and cancels runs only. A priority change or downtime acts on everyone else's work, so it
is the operator's: `berth schedule`, or a paired device in the harbor. `PATCH` or `DELETE` on one is a `403`. Any
holder of the key can change any run; per-device ownership comes with device keys.

## One URL per card

On a node with several cards, each card has its own base URL on the gateway: `/cards/CARD/v1`, where `CARD` is
Expand Down
45 changes: 45 additions & 0 deletions docs/leasing.md
Original file line number Diff line number Diff line change
Expand Up @@ -179,6 +179,51 @@ A bump puts the waiter ahead of everyone else in its priority class. To go ahead
priority with `prio`. Holds and trainings cannot be changed here: a hold is a pin, not a place in the queue, and
a training's place is fixed by the card's lock. Each change is a `lease.queue_override` event in `berth events`.

## Book the card

The queue decides who goes next. A booking decides whose hours they are, up to 14 days ahead. A booking has one of
three kinds:

- `run` is an engine's turn. While it lasts, the engine's waiter goes first in line and runs at the booking's
priority (foreground when the booking names none). It wins a tie against a holder that has no booking. Once it
holds the card, nothing but a hold drains it until the booking ends. A minimum turn (`min_turn`) still protects
the holder it takes over from.
- `priority` moves an engine to another class for set hours, such as llama-swap at batch while a training runs. It
claims no time on the card, so it may overlap a run.
- `down` is downtime. No new work starts, and holders drain at their usual boundary when it begins. A hold and a
training stay.

An hour nobody booked runs by priority, then arrival. A booked engine that does not ask for the card leaves it to
everyone else, so a booking never keeps the card idle.

```sh
berth schedule # what holds each card, the waiters in grant order, every booking
berth schedule book --who pi-pocket --engine pocket-train --at 23:00 --for 5h --days 7 --priority batch
berth schedule book --who training --engine sft --at "sun 18:00" --for 3h
berth schedule book --who operator --kind priority --engine llama-swap --priority batch --at "sun 18:00" --for 3h
berth schedule book --who operator --kind down --at "tue 04:00" --for 1h --note "driver update"
berth schedule move b4 --at 01:00 # new hours, same ID; --for, --priority and --note change those
berth schedule cancel b4
```

`--at` takes `now`, `22:00` (the next time the clock reads it), `tue 22:00`, `2026-11-14 22:00` or RFC 3339, on the
node's clock. `--days 7` books the same hours on 7 days in a row, all of them or none. `--earliest` moves the
booking to the first free hours at or after `--at`. `--engine` takes an engine or a group, such as `llama-swap`
for every model it serves.

Two runs or downtimes on one card may not overlap, and neither may two priority changes for the same engine. A
clash is refused with the booking it hits. A class set with `berth queue prio` wins over a booking, and a booking
never moves a hold or a training.

berthd keeps the bookings in `schedule.json` in its state directory (`berth daemon --schedule`). Each change is a
`schedule.booked` event, and each class a booking moves, or moves back at its end, is a `lease.booking_priority`
event. A lease that a booking governs shows the booking's ID in `berth status --format json` (`booking`).

The harbor's SCHEDULE tab shows the same scheduler. NOW lists what holds each card, the waiters and the booking in
force. 7 DAYS draws one 24-hour track per day with every booking to the minute, each engine's window and the live
holder. A paired device books by clicking an empty hour, and moves or cancels a booking by clicking it. Agents
book over the gateway: read [Book the card from an agent](gateway.md#book-the-card-from-an-agent).

## Cordon a card

On a node with several cards, `berth cordon gpu1` stops new grants on that card while the others keep serving, for
Expand Down
135 changes: 135 additions & 0 deletions internal/broker/booking.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,135 @@
// Copyright 2026 Trevin Tindall (T-Crypt)
// SPDX-License-Identifier: Apache-2.0

package broker

// The card's schedule. berthd's booking file says who has the card when,
// and the broker reads it on every tick. A run booking is an engine's turn:
// its waiter goes first, it runs at the booking's priority, and while it
// holds the card nothing but a hold drains it. A priority booking moves an
// engine to another class for its hours. Downtime takes no new work and
// drains its holders at the start. An hour nobody booked runs as it always
// did, by priority then arrival, so an unused booking leaves the card idle
// for nobody.

import (
"fmt"
"slices"
"time"

"github.com/T-Crypt/berth/internal/proto"
)

// SetBookings replaces the card's schedule with bs, and schedules the queue
// under it at once.
func (b *Broker) SetBookings(bs []proto.Booking) {
b.mu.Lock()
defer b.mu.Unlock()
b.bookings = slices.Clone(bs)
now := b.now()
b.closeForDowntime(now)
b.schedule(now)
}

// booked is the booking that sets l's priority at now: a run, then a
// priority change, for its engine or its group. A hold, a training and a
// class the operator set by hand are not moved.
func (b *Broker) booked(l *lease, now time.Time) (proto.Booking, bool) {
if l.basePrio == Hold || l.training || l.operatorPrio {
return proto.Booking{}, false
}
for _, k := range []proto.BookingKind{proto.BookRun, proto.BookPriority} {
for _, name := range []string{l.engine, l.group} {
if name == "" {
continue
}
if x, ok := proto.BookingAt(b.bookings, k, name, now); ok {
return x, true
}
}
}
return proto.Booking{}, false
}

// inRun is whether a run booking holds for l at now.
func (b *Broker) inRun(l *lease, now time.Time) bool {
x, ok := b.booked(l, now)
return ok && x.Kind == proto.BookRun
}

// bookedPrio is the class a booking gives: its own, and foreground for a
// run that names none.
func bookedPrio(x proto.Booking) Priority {
if x.Priority == "" {
return Foreground
}
p, err := ParsePriority(x.Priority)
if err != nil {
return Foreground
}
return p
}

// applyBookings sets every lease's class from the schedule at now: the
// booking's while one holds, its own when none does. Each move is an event.
// Called with b.mu held.
func (b *Broker) applyBookings(now time.Time) {
moved := false
for _, l := range slices.Concat(b.queue, mapValues(b.leases)) {
p, id := l.basePrio, ""
if x, ok := b.booked(l, now); ok {
p, id = bookedPrio(x), x.ID
}
if p == l.prio && id == l.booking {
continue
}
detail := fmt.Sprintf("priority %s -> %s: booking %s", l.prio, p, id)
if id == "" {
detail = fmt.Sprintf("priority %s -> %s: booking %s ended", l.prio, p, l.booking)
}
l.prio, l.booking, moved = p, id, true
b.emit(proto.Event{Kind: proto.EvBookingPrio, Lease: l.id, Engine: l.engine, Detail: detail})
b.log.Info("booking priority", "lease", l.id, "engine", l.engine, "change", detail)
}
if moved {
b.sortQueue()
}
}

// closeForDowntime drains the active holders, and evicts the parked ones,
// when downtime holds at now. A hold and a training keep the card. A lease
// already on its way out is not asked twice. Called with b.mu held.
func (b *Broker) closeForDowntime(now time.Time) {
d, ok := proto.BookingAt(b.bookings, proto.BookDown, "", now)
if !ok {
return
}
for _, l := range b.leases {
if l.prio == Hold || l.training {
continue
}
switch l.state {
case active:
b.drain(l, downWhy(d))
case parked:
b.evict(l, downWhy(d))
}
}
}

// downWhy is what a waiter and a drained holder read during downtime.
func downWhy(d proto.Booking) string {
why := fmt.Sprintf("the card is down until %s (booking %s by %s", d.End.Format("Mon 15:04"), d.ID, d.Who)
if d.Note != "" {
why += ": " + d.Note
}
return why + ")"
}

func mapValues[K comparable, V any](m map[K]V) []V {
out := make([]V, 0, len(m))
for _, v := range m {
out = append(out, v)
}
return out
}
Loading
Loading