Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,15 @@ has a matching GitHub release whose body comes from its section below.

### Added

- `berth why ID` tells one lease's story in plain words: when it asked for the card, each reason it waited, the
grant, each drain and who it was for, the release, and the berthd restarts it lived through. `berth why ENGINE`
reads that engine's newest lease. A lease berthd still holds ends with where it stands now, and one a restart
cut off names the lease its engine re-attached as.
- berthd keeps its event stream in `events.jsonl` in the state directory (`--events`, `""` for memory only). A
restart reads the last 1000 events back, goes on numbering after them, and goes on with the lease numbers, so
`berth events --since N` and `berth why` read across restarts and deploys. The file moves to `events.jsonl.1`
at 8 MiB. Each start is a `node.started` event that names the version.

- A schedule for each card, up to 14 days ahead. `berth schedule` shows what holds each card, the waiters in grant
order and every booking. `berth schedule book` books an engine's run (it goes first, runs at the booking's
priority, and keeps the card against everything but a hold), a priority change for set hours, or downtime (no
Expand Down
6 changes: 5 additions & 1 deletion docs/harbor.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,11 @@ for a phone.
first-seen time; optional desktop notification and webhook. Report-only by design.
- **Contested grants:** a grant made while stowaways hold memory is flagged, not hidden.
- **Event log:** every request, wait, drain, release, free, stuck lease, stowaway and
probe change, with gap-free sequence numbers; `berth events -f` follows it live.
probe change, with gap-free sequence numbers; `berth events -f` follows it live. berthd keeps the
stream in `~/.local/state/berth/events.jsonl` (`--events`), so the numbers and the last 1000 events carry
on across a restart. `berth why ID` tells one lease's part of it in plain words: why it waited, the
grant, each drain and who it was for, and the restarts it lived through. `berth why ENGINE` reads that
engine's newest lease.
- **The collisions board:** collisions avoided, VRAM handed off, drains, stuck releases
caught, stowaways caught, contested grants, refusals. Each count links back to a
recorded event. berthd keeps the board in `~/.local/state/berth/counters.json`
Expand Down
2 changes: 1 addition & 1 deletion docs/interfaces.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

- **CLI:** one command tree with one output contract. `berth status`, `berth events -f`,
`berth run`, `berth hold`, `berth stop`, `berth cordon|uncordon`, `berth notify`, `berth claims`, `berth pull`, `berth verify`, `berth models`, `berth gateway`,
`berth wire llama-swap`, `berth engine check`, `berth contract ENGINE`, `berth fit NAME`, `berth queue`, `berth config check`, `berth config show`,
`berth wire llama-swap`, `berth engine check`, `berth contract ENGINE`, `berth fit NAME`, `berth queue`, `berth why ID|ENGINE`, `berth config check`, `berth config show`,
`berth export`, `berth key`, `berth skill install|show`, `berth update`, `berth version`,
and for the node itself `berth daemon`, `berth harbor`, `berth service install|uninstall|status`,
`berth reclaim`, `berth resume`, `berth prefetch`, `berth storage`, `berth civitai search|versions`
Expand Down
16 changes: 16 additions & 0 deletions internal/cli/commands.go
Original file line number Diff line number Diff line change
Expand Up @@ -223,6 +223,22 @@ notify-send, so any desktop or notification daemon shows it. On Windows it is a
tray icon whose notifications are Windows' own toasts; a click opens the harbor.`,
Run: runNotify,
},
{
Name: "why",
Summary: "one lease's story: why it waited, was granted, drained or refused",
Args: "ID|ENGINE [--socket P] [--format F]",
Help: `Print every event berthd kept about one lease, oldest first, in plain words:
when it asked for the card, each reason it waited, the grant, each drain and
who it was for, the release, and the berthd restarts it lived through. ENGINE
picks that engine's newest lease. A lease that berthd still holds ends with
where it stands now; one a restart cut off names the lease its engine
re-attached as.

berthd keeps its events in events.jsonl in the state directory (--events), so
the story reads across restarts and deploys, back to the last 1000 events.`,
Formats: []Format{FormatHuman, FormatJSON, FormatYAML},
Run: runWhy,
},
{
Name: "fit",
Summary: "where a request for an engine or model would run now, and what the start costs",
Expand Down
12 changes: 12 additions & 0 deletions internal/cli/daemon.go
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ import (
"github.com/T-Crypt/berth/internal/claims"
"github.com/T-Crypt/berth/internal/config"
"github.com/T-Crypt/berth/internal/engines"
"github.com/T-Crypt/berth/internal/eventlog"
"github.com/T-Crypt/berth/internal/gpu"
"github.com/T-Crypt/berth/internal/history"
"github.com/T-Crypt/berth/internal/host"
Expand Down Expand Up @@ -54,6 +55,7 @@ func runDaemon(c *Context) int {
scheduleFile := fs.String("schedule", defaultStateFile("schedule.json"), "the cards' bookings (berth schedule): runs, priority changes, downtime; \"\" = no schedule")
usageFile := fs.String("usage", defaultStateFile("usage.json"), "card-hours per engine, the default juggle ranking; \"\" = memory only")
footprintFile := fs.String("footprint", defaultStateFile("footprint.json"), "VRAM measured per engine across restarts, the estimate behind EST. VRAM; \"\" = memory only")
eventsFile := fs.String("events", defaultStateFile("events.jsonl"), "the event stream across restarts, what berth events and berth why read; \"\" = memory only")
if code, done := c.Parse(fs); done {
return code
}
Expand Down Expand Up @@ -127,6 +129,16 @@ func runDaemon(c *Context) int {
allowList = splitList(*allow)
}
events, ids, seq, ledger := node.Shared()
if *eventsFile != "" {
if kept, err := eventlog.Open(*eventsFile); err != nil {
log.Error("event file; events are kept in memory only", "path", *eventsFile, "err", err)
} else {
events = kept
defer kept.Close()
ids.Store(kept.LastLease())
}
}
events.Emit(proto.Event{At: time.Now(), Kind: proto.EvNodeStarted, Detail: "berth " + c.Version})
procTable := procs.Shared(procs.List, *poll/2)
var cards []*node.Card
var snaps []gpu.Snapshot
Expand Down
206 changes: 206 additions & 0 deletions internal/cli/why.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,206 @@
// Copyright 2026 Trevin Tindall (T-Crypt)
// SPDX-License-Identifier: Apache-2.0

package cli

import (
"slices"
"time"

"github.com/T-Crypt/berth/internal/proto"
"github.com/T-Crypt/berth/internal/server"
)

// whyDoc is one lease's story: every event berthd kept about it, oldest
// first, with the restarts it lived through, and where it stands now.
type whyDoc struct {
Lease string `json:"lease" yaml:"lease"`
Engine string `json:"engine" yaml:"engine"`
Card string `json:"card,omitempty" yaml:"card,omitempty"`
Events []proto.Event `json:"events" yaml:"events"`
// Now is the lease as berthd holds it now, nil once it has ended.
Now *proto.Lease `json:"now,omitempty" yaml:"now,omitempty"`
// Next is the lease its engine re-attached as after a berthd restart
// cut this one off, "" when none did.
Next string `json:"next,omitempty" yaml:"next,omitempty"`
}

// whyVerb says what each kind of event did to the lease, in the words the
// story prints. A kind not listed prints as its name.
var whyVerb = map[string]string{
proto.EvRequested: "asked for the card",
proto.EvWaiting: "waited",
proto.EvGranted: "granted",
proto.EvReattached: "re-attached after a berthd restart",
proto.EvDraining: "asked to give the card back",
proto.EvReleased: "released the card",
proto.EvFreed: "its memory is free",
proto.EvRefused: "refused",
proto.EvOrphaned: "its process was gone, so berthd ended the lease",
proto.EvStuck: "its memory stayed on the card after the release",
proto.EvParked: "parked in RAM",
proto.EvResumed: "resumed from RAM",
proto.EvUnparked: "loaded back onto the card without a resume",
proto.EvParkRefused: "could not park",
proto.EvParkEvicted: "stopped while parked",
proto.EvJuggle: "a juggle for the card",
proto.EvQueueOverride: "the operator changed its place in line",
proto.EvBookingPrio: "a booking changed its priority",
proto.EvLending: "asked to lend part of the card",
proto.EvLent: "lent part of the card",
proto.EvReturned: "took its lent memory back",
proto.EvLendFailed: "the lend failed",
proto.EvNodeStarted: "berthd restarted",
}

// ends is the kinds after which a lease has no more to say.
var ends = []string{proto.EvFreed, proto.EvRefused, proto.EvOrphaned}

// whyOf finds the lease name means, a lease ID or else the engine's newest
// lease, and tells its story from evs. live is the leases berthd holds now.
func whyOf(evs []proto.Event, live []proto.Lease, name string) (whyDoc, bool) {
id := ""
for i := len(evs) - 1; i >= 0 && id == ""; i-- {
if evs[i].Lease == name {
id = name
}
}
for i := len(evs) - 1; i >= 0 && id == ""; i-- {
if evs[i].Engine == name && evs[i].Lease != "" {
id = evs[i].Lease
}
}
if id == "" {
return whyDoc{}, false
}
doc := whyDoc{Lease: id}
for _, l := range live {
if l.ID == id {
doc.Now = &l
}
}
first, last := -1, -1
for i, e := range evs {
if e.Lease == id {
if first < 0 {
first = i
}
last = i
doc.Engine, doc.Card = e.Engine, e.Card
}
}
ended := slices.Contains(ends, evs[last].Kind)
for i := first; i < len(evs); i++ {
e := evs[i]
switch {
case e.Lease == id:
doc.Events = append(doc.Events, e)
case e.Kind == proto.EvNodeStarted && (i < last || (!ended && doc.Now == nil)):
doc.Events = append(doc.Events, e)
if i > last {
doc.Next = reattachedAs(evs[i+1:], doc.Engine)
return doc, true
}
}
}
return doc, true
}

// reattachedAs is the lease engine re-attached as in evs, which start just
// after a berthd restart, "" when it did not before the next restart.
func reattachedAs(evs []proto.Event, engine string) string {
for _, e := range evs {
if e.Kind == proto.EvNodeStarted {
return ""
}
if e.Kind == proto.EvReattached && e.Engine == engine {
return e.Lease
}
}
return ""
}

type whyView struct{ doc whyDoc }

func (v whyView) Data() any { return v.doc }

func (v whyView) Human(c *Context, _ bool) error {
d := v.doc
head := d.Lease
if d.Engine != "" && d.Engine != d.Lease {
head += " " + d.Engine
}
if d.Card != "" {
head += " " + d.Card
}
c.Printf("%s\n", head)
for _, e := range d.Events {
verb, ok := whyVerb[e.Kind]
if !ok {
verb = e.Kind
}
line := verb
if e.Bytes != 0 {
line += ", " + server.MiB(e.Bytes)
}
if e.Detail != "" {
line += ": " + e.Detail
}
c.Printf(" %s %s\n", c.Style.Dim(e.At.Local().Format("Jan 02 15:04:05")), line)
}
switch {
case d.Now != nil:
c.Printf("now %s, %s", d.Now.State, d.Now.Priority)
if d.Now.Position > 0 {
c.Printf(", %d in line", d.Now.Position)
}
if d.Now.TurnS > 0 {
c.Printf(", %s of its minimum turn left", (time.Duration(d.Now.TurnS) * time.Second).Round(time.Second))
}
c.Printf("\n")
case d.Next != "":
c.Printf("its engine re-attached as %s: berth why %s\n", d.Next, d.Next)
}
return nil
}

func runWhy(c *Context) int {
fs := c.FlagSet()
sock := fs.String("socket", defaultSocket(), "berthd socket ($BERTH_SOCKET)")
if code, done := c.ParseInterspersed(fs); done {
return code
}
if len(c.Args) != 1 {
return c.fail(Systemf("berth why takes one lease ID or engine name").
WithHint("for example: berth why strata-24, or berth why comfyui"))
}
conn, raw, err := server.Dial(*sock)
if err != nil {
return c.fail(Systemf("berthd unreachable on %s: %v", *sock, err))
}
defer raw.Close()
if err := conn.Send(proto.Msg{Op: proto.OpEvents}); err != nil {
return c.fail(Systemf("berthd on %s: %v", *sock, err))
}
var evs []proto.Event
for {
m, err := conn.Recv()
if err != nil {
break
}
if m.Event != nil {
evs = append(evs, *m.Event)
}
}
st, _ := contractStatus(*sock)
var live []proto.Lease
if st != nil {
live = append(st.Leases, st.Waiting...)
}
doc, found := whyOf(evs, live, c.Args[0])
if !found {
return c.fail(Failf("no lease or engine named %q in berthd's last %d events", c.Args[0], len(evs)).
WithHint("berth status lists the leases now; berth events lists what berthd kept"))
}
return c.Write(whyView{doc: doc}, nil)
}
Loading
Loading