Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,14 @@ has a matching GitHub release whose body comes from its section below.

## [Unreleased]

### Added

- `berth schedule book --claim` books a run that claims its hours. While it lasts, the booked engine's waiter
drains any holder but a hold or a pinned lease, whatever the holder's priority. Before, a `batch` run lost its
hours to an idle `foreground` holder, and the workaround was a downtime booked just before it. The harbor's
booking form has the same choice. Only the operator books a claim; the gateway refuses to move or cancel one.
(#345)

### Fixed

- On Windows, `berth service install --notify` no longer opens a console window at logon. The Run key started
Expand Down
5 changes: 3 additions & 2 deletions docs/gateway.md
Original file line number Diff line number Diff line change
Expand Up @@ -333,8 +333,9 @@ curl -s -X DELETE http://127.0.0.1:9799/v1/berth/schedule/b4 -H "$K"
- `PATCH /v1/berth/schedule/ID` moves a run. It takes `start`, `for`, `priority` and `note`, keeps the rest and
keeps the ID. `DELETE` cancels a run. Both answer `200` with the booking.

The gateway books, moves and cancels runs only. A priority change or downtime acts on everyone else's work, so it
is the operator's: `berth schedule`, or a paired device in the harbor. `PATCH` or `DELETE` on one is a `403`. Any
The gateway books, moves and cancels runs only. A priority change, downtime or a run that claims its hours acts on
everyone else's work, so it is the operator's: `berth schedule`, or a paired device in the harbor. The gateway
books no claim, and `PATCH` or `DELETE` on one of these is a `403`. Any
holder of the key can change any run; per-device ownership comes with device keys.

## One URL per card
Expand Down
6 changes: 5 additions & 1 deletion docs/leasing.md
Original file line number Diff line number Diff line change
Expand Up @@ -187,7 +187,10 @@ three kinds:
- `run` is an engine's turn. While it lasts, the engine's waiter goes first in line and runs at the booking's
priority (foreground when the booking names none). It wins a tie against a holder that has no booking. Once it
holds the card, nothing but a hold drains it until the booking ends. A minimum turn (`min_turn`) still protects
the holder it takes over from.
the holder it takes over from. A `batch` run does not drain a `foreground` holder, so an engine left idle at a
higher class keeps the card through the booked hours. Book the run with `--claim` to take them back: while it
lasts, the booked engine's waiter drains any holder but a hold or a pinned lease, whatever its class. A training
stays too. Only the operator books a claim; the gateway does not.
- `priority` moves an engine to another class for set hours, such as llama-swap at batch while a training runs. It
claims no time on the card, so it may overlap a run.
- `down` is downtime. No new work starts, and holders drain at their usual boundary when it begins. A hold and a
Expand All @@ -200,6 +203,7 @@ everyone else, so a booking never keeps the card idle.
berth schedule # what holds each card, the waiters in grant order, every booking
berth schedule book --who pi-pocket --engine pocket-train --at 23:00 --for 5h --days 7 --priority batch
berth schedule book --who training --engine sft --at "sun 18:00" --for 3h
berth schedule book --who reviews --engine strata --priority batch --claim --at 00:00 --for 85m --days 7
berth schedule book --who operator --kind priority --engine llama-swap --priority batch --at "sun 18:00" --for 3h
berth schedule book --who operator --kind down --at "tue 04:00" --for 1h --note "driver update"
berth schedule move b4 --at 01:00 # new hours, same ID; --for, --priority and --note change those
Expand Down
11 changes: 10 additions & 1 deletion internal/broker/booking.go
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,8 @@ package broker
// The card's schedule. berthd's booking file says who has the card when,
// and the broker reads it on every tick. A run booking is an engine's turn:
// its waiter goes first, it runs at the booking's priority, and while it
// holds the card nothing but a hold drains it. A priority booking moves an
// holds the card nothing but a hold drains it. A run that claims its hours
// also drains a holder of a higher class for its waiter. A priority booking moves an
// engine to another class for its hours. Downtime takes no new work and
// drains its holders at the start. An hour nobody booked runs as it always
// did, by priority then arrival, so an unused booking leaves the card idle
Expand Down Expand Up @@ -57,6 +58,14 @@ func (b *Broker) inRun(l *lease, now time.Time) bool {
return ok && x.Kind == proto.BookRun
}

// claiming is whether a run booking that claims its hours holds for l at
// now: l's waiter may drain any holder but a hold or a pinned one, whatever
// the holder's class.
func (b *Broker) claiming(l *lease, now time.Time) bool {
x, ok := b.booked(l, now)
return ok && x.Kind == proto.BookRun && x.Claim
}

// bookedPrio is the class a booking gives: its own, and foreground for a
// run that names none.
func bookedPrio(x proto.Booking) Priority {
Expand Down
36 changes: 36 additions & 0 deletions internal/broker/booking_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,42 @@ func TestRunBookingKeepsItsHolderUntilItEnds(t *testing.T) {
}
}

// A batch run booking does not drain a foreground holder; one that claims
// its hours does, and still leaves a hold alone.
func TestClaimRunBookingDrainsAHigherHolder(t *testing.T) {
b, _ := startTurnBroker(t, procs.Map{})
b.SetBookings([]proto.Booking{book(b, "b1", proto.BookRun, "strata", "batch", 0, time.Hour)})
holder := must(t, acquire(b, Request{Engine: "strata-sc117", Priority: Foreground, Needs: 20 * GiB}))
w := acquireW(b, Request{Engine: "strata", Priority: Foreground, Needs: 20 * GiB})
pending(t, w.ch)
select {
case why := <-holder.Drains():
t.Fatalf("a batch run without a claim drained a foreground holder: %s", why)
default:
}

claim := book(b, "b1", proto.BookRun, "strata", "batch", 0, time.Hour)
claim.Claim = true
b.SetBookings([]proto.Booking{claim})
if got, want := drained(t, holder), "strata (batch) needs the card"; got != want {
t.Errorf("drain = %q, want %q", got, want)
}
holder.Release()
b.Tick(context.Background())
granted(t, b, "strata", w.ch)

b2, _ := startTurnBroker(t, procs.Map{})
b2.SetBookings([]proto.Booking{claim})
hold := must(t, acquire(b2, Request{Engine: "game", Priority: Hold, Needs: 20 * GiB}))
w2 := acquireW(b2, Request{Engine: "strata", Priority: Batch, Needs: 20 * GiB})
pending(t, w2.ch)
select {
case why := <-hold.Drains():
t.Fatalf("a claim drained a hold: %s", why)
default:
}
}

// A run booking puts its waiter ahead of an earlier foreground one.
func TestRunBookingGoesFirstInLine(t *testing.T) {
b, _ := startTurnBroker(t, procs.Map{})
Expand Down
2 changes: 1 addition & 1 deletion internal/broker/broker.go
Original file line number Diff line number Diff line change
Expand Up @@ -1579,7 +1579,7 @@ func (b *Broker) evaluate(w *lease, now time.Time) (bool, string, []*lease) {
// for it to become parked (or fall back to a drain); evicting
// it mid-park would throw the park away.
pending = append(pending, h)
case h.pinned || h.prio > w.prio, w.prio != Hold && b.inRun(h, now) && !b.inRun(w, now):
case h.pinned || h.prio > w.prio && (h.prio == Hold || !b.claiming(w, now)), w.prio != Hold && b.inRun(h, now) && !b.inRun(w, now):
blockers = append(blockers, h)
case w.prio != Hold && h.inTurn(now):
// A minimum turn: the holder keeps the card until its turn
Expand Down
9 changes: 7 additions & 2 deletions internal/cli/schedule.go
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,11 @@ func (v scheduleView) Human(c *Context, ok bool) error {
if x.Holds(v.b.Now) {
when += " (now)"
}
fmt.Fprintf(tw, "%s\t%s%s\t%s\t%s\t%s\t%s\t%s\n", x.ID, card(x.Card), x.Kind, x.Who, dash(x.Engine), dash(x.Priority), when, x.Note)
kind := string(x.Kind)
if x.Claim {
kind += " (claim)"
}
fmt.Fprintf(tw, "%s\t%s%s\t%s\t%s\t%s\t%s\t%s\n", x.ID, card(x.Card), kind, x.Who, dash(x.Engine), dash(x.Priority), when, x.Note)
}
return tw.Flush()
}
Expand Down Expand Up @@ -84,6 +88,7 @@ func runSchedule(c *Context) int {
earliest := fs.Bool("earliest", false, "book: take the first free hours at or after --at")
card := fs.String("card", "", "book: the card (default the first)")
note := fs.String("note", "", "book, move: a note the board shows")
claim := fs.Bool("claim", false, "book: a run that claims its hours drains a holder of any class but a hold for its engine")
if code, done := c.ParseInterspersed(fs); done {
return code
}
Expand All @@ -99,7 +104,7 @@ func runSchedule(c *Context) int {
return c.fail(Failf("berth schedule book needs --at and --for").WithHint("%s", usage))
}
b := proto.Booking{Card: *card, Kind: proto.BookingKind(*kind), Who: *who, Engine: *engine, Priority: *prio,
Start: start, End: start.Add(*forFlag), Note: *note}
Start: start, End: start.Add(*forFlag), Note: *note, Claim: *claim}
if err := b.Check(now); err != nil {
return c.fail(Failf("%v", err).WithHint("%s", usage))
}
Expand Down
17 changes: 17 additions & 0 deletions internal/cli/schedule_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ import (
"path/filepath"
"strings"
"testing"
"time"

"github.com/T-Crypt/berth/internal/proto"
"github.com/T-Crypt/berth/internal/schedule"
Expand Down Expand Up @@ -76,4 +77,20 @@ func TestScheduleAgainstADaemon(t *testing.T) {
if code != 0 || !strings.Contains(out, "nothing booked") || !strings.Contains(out, "RUNNING") {
t.Fatalf("board after cancel (exit %d): %s", code, out)
}

code, out, errOut = runCLI(t, "schedule", "book", "--who", "op", "--engine", "strata", "--priority", "batch", "--claim",
"--at", "now", "--for", "1h", "--socket", sock)
if code != 0 || !strings.Contains(out, "booked b2") {
t.Fatalf("book --claim (exit %d): %s %s", code, out, errOut)
}
if got := s.List(time.Now()); len(got) != 1 || !got[0].Claim {
t.Errorf("schedule after --claim = %+v, want one claiming run", got)
}
if code, out, _ = runCLI(t, "schedule", "--socket", sock); code != 0 || !strings.Contains(out, "run (claim)") {
t.Errorf("board (exit %d) does not show the claim: %s", code, out)
}
code, _, errOut = runCLI(t, "schedule", "book", "--who", "op", "--kind", "down", "--claim", "--at", "23:00", "--for", "1h", "--socket", sock)
if code != 1 || !strings.Contains(errOut, "only a run booking claims") {
t.Errorf("down --claim (exit %d): %s", code, errOut)
}
}
3 changes: 3 additions & 0 deletions internal/gateway/schedule.go
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,9 @@ func (g *Gateway) runBooking(w http.ResponseWriter, r *http.Request) (proto.Book
case st.Bookings[i].Kind != proto.BookRun:
apiError(w, http.StatusForbidden, fmt.Sprintf("%s is a %s booking: only the operator changes it (berth schedule, or the harbor)", id, st.Bookings[i].Kind))
return proto.Booking{}, false
case st.Bookings[i].Claim:
apiError(w, http.StatusForbidden, fmt.Sprintf("%s claims its hours: only the operator changes it (berth schedule, or the harbor)", id))
return proto.Booking{}, false
}
return st.Bookings[i], true
}
Expand Down
16 changes: 16 additions & 0 deletions internal/gateway/schedule_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,22 @@ func TestScheduleRoutesLeaveTheOperatorsBookings(t *testing.T) {
t.Errorf("%s downtime = %d %s, want 403", method, resp.StatusCode, body)
}
}

resp = call(t, "POST", srv.URL+"/v1/berth/schedule", `{"who":"agent","engine":"x","start":"`+now.Add(5*time.Hour).Format(time.RFC3339)+`","for":"30m","claim":true}`)
if made := decode[map[string][]proto.Booking](t, resp)["bookings"]; len(made) != 1 || made[0].Claim {
t.Fatalf("a claim in the body booked %+v (%d), want a run without one", made, resp.StatusCode)
}
claim, err := s.Book(proto.Booking{Card: "gpu0", Kind: proto.BookRun, Who: "operator", Engine: "x", Claim: true,
Start: now.Add(3 * time.Hour), End: now.Add(4 * time.Hour)}, 1, false, now)
if err != nil {
t.Fatal(err)
}
for _, method := range []string{"PATCH", "DELETE"} {
resp := call(t, method, srv.URL+"/v1/berth/schedule/"+claim[0].ID, `{"for":"3h"}`)
if body := readAll(resp); resp.StatusCode != http.StatusForbidden || !strings.Contains(body, "only the operator") {
t.Errorf("%s claim = %d %s, want 403", method, resp.StatusCode, body)
}
}
}

func readAll(resp *http.Response) string {
Expand Down
1 change: 1 addition & 0 deletions internal/harbor/web/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -586,6 +586,7 @@ <h3 class="sheet-sect" id="sheet-sec-act">ACTIONS</h3>
<label>DAYS IN A ROW <input name="days" type="number" min="1" max="14" value="1"></label>
<label class="sf-check"><input name="earliest" type="checkbox"> first free hours</label>
</div>
<label class="sf-check sf-claim"><input name="claim" type="checkbox"> claim the hours: drain any holder but a hold</label>
<label>NOTE <input name="note" maxlength="120" placeholder="optional"></label>
<p class="sf-explain" id="sched-explain"></p>
<p class="sf-error" id="sched-error" role="alert"></p>
Expand Down
9 changes: 7 additions & 2 deletions internal/harbor/web/schedule.js
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,9 @@ function explain(b) {
switch (b.kind) {
case "run":
return eng + " goes first in line from " + from + " and runs at " + (b.priority || "foreground") +
"; until " + to + " nothing but a hold drains it. If it does not ask for the card, the card serves everyone else.";
"; until " + to + " nothing but a hold drains it." +
(b.claim ? " When it asks, any holder but a hold drains for it, whatever its class." : "") +
" If it does not ask for the card, the card serves everyone else.";
case "priority":
return eng + " runs at " + (b.priority || "?") + " from " + from + " to " + to + ", then back to its own class. It claims no time on the card.";
case "down":
Expand Down Expand Up @@ -268,6 +270,7 @@ export function initSchedule({ demo }) {
listRoot.replaceChildren(...bs.map((b) => {
const row = el("div", "sq sw-engine sb-item sb-item-" + b.kind);
const right = el("span", "sw-state", el("span", "sw-chip", KIND_WORD[b.kind]));
if (b.claim) right.append(el("span", "sw-chip", "CLAIM"));
if (new Date(b.start) <= now) right.append(el("span", "sw-chip sw-pinned", "IN FORCE"));
if (many) right.prepend(el("span", "sw-chip", b.card));
row.append(el("div", "top", el("span", "name", b.id + " " + label(b)), right));
Expand Down Expand Up @@ -297,7 +300,7 @@ export function initSchedule({ demo }) {
const b = {
kind: f.kind.value, who: f.who.value.trim(), engine: f.engine.value.trim(), priority: f.priority.value,
start: start.toISOString(), end: new Date(start.getTime() + Number(f.hours.value) * HOUR).toISOString(),
note: f.note.value.trim(), card: pick,
note: f.note.value.trim(), card: pick, claim: f.kind.value === "run" && f.claim.checked,
};
if (b.kind === "down") { b.engine = ""; b.priority = ""; }
if (editing) b.id = editing.id;
Expand All @@ -308,6 +311,7 @@ export function initSchedule({ demo }) {
const kind = form.elements.kind.value;
form.querySelector(".sf-engine").hidden = kind === "down";
form.querySelector(".sf-prio").hidden = kind === "down";
form.querySelector(".sf-claim").hidden = kind !== "run";
form.querySelector(".sf-new").hidden = !!editing;
try { explainEl.textContent = explain(formBooking()); } catch (e) { explainEl.textContent = ""; }
}
Expand All @@ -327,6 +331,7 @@ export function initSchedule({ demo }) {
f.hours.value = b ? String(Math.round(((new Date(b.end) - start) / HOUR) * 100) / 100) : "1";
f.days.value = "1";
f.earliest.checked = false;
f.claim.checked = !!(b && b.claim);
f.note.value = b ? b.note || "" : "";
f.kind.disabled = !!b;
for (const x of form.querySelectorAll("input, select")) if (x !== f.kind) x.disabled = !paired;
Expand Down
5 changes: 5 additions & 0 deletions internal/proto/booking.go
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,9 @@ type Booking struct {
Start time.Time `json:"start" yaml:"start"`
End time.Time `json:"end" yaml:"end"`
Note string `json:"note,omitempty" yaml:"note,omitempty"`
// Claim lets a run's waiter drain any holder but a hold or a pinned
// one (a training, berth run --pinned), whatever the holder's class.
Claim bool `json:"claim,omitempty" yaml:"claim,omitempty"`
}

// Holds is whether t falls inside the booking.
Expand Down Expand Up @@ -86,6 +89,8 @@ func (b Booking) Check(now time.Time) error {
return fmt.Errorf("a priority booking names its priority: batch, interactive or foreground")
case b.Kind == BookDown && b.Priority != "":
return fmt.Errorf("downtime takes no priority")
case b.Claim && b.Kind != BookRun:
return fmt.Errorf("only a run booking claims its hours")
case b.Priority != "" && !slices.Contains(priorities, b.Priority):
return fmt.Errorf("priority %q: use batch, interactive or foreground", b.Priority)
case !b.End.After(b.Start):
Expand Down
1 change: 1 addition & 0 deletions internal/schedule/schedule_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -150,6 +150,7 @@ func TestCheckRefusesBadBookings(t *testing.T) {
{proto.Booking{Kind: proto.BookDown, Who: "a", Start: at(13, 0), End: at(13, 0).AddDate(0, 0, 8)}, "at most"},
{proto.Booking{Kind: proto.BookPriority, Who: "a", Engine: "e", Start: at(13, 0), End: at(14, 0)}, "names its priority"},
{proto.Booking{Kind: proto.BookRun, Who: "a", Engine: "e", Priority: "hold", Start: at(13, 0), End: at(14, 0)}, `priority "hold"`},
{proto.Booking{Kind: proto.BookPriority, Who: "a", Engine: "e", Priority: "batch", Claim: true, Start: at(13, 0), End: at(14, 0)}, "only a run booking claims"},
} {
if err := tc.b.Check(sat1200); err == nil || !strings.Contains(err.Error(), tc.want) {
t.Errorf("%+v: err = %v, want %q", tc.b, err, tc.want)
Expand Down
2 changes: 1 addition & 1 deletion internal/skill/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ do not assume any engine is protected.
| Train or fine-tune a model (nothing automated may interrupt it) | `berth train --as NAME [--gpu gpu0,gpu1] -- CMD`: the whole card, no wait deadline, and every other request on its cards is refused until it exits. Another `berth train` waits in line |
| Run an app that must not be drained but may share the card (Blender) | `berth run --as NAME --pinned -- CMD` |
| Give an engine a minimum turn before a request can drain it | `berth run --as NAME --min-turn 5m -- CMD`, or `min_turn = "5m"` in its engine entry (a hold still wins; `berth status` shows the turn left in the TURN column) |
| Plan your own turn on the card (a nightly job, a long run) | `POST /v1/berth/schedule` on the gateway with `{"who":"ME","engine":"NAME","start":"23:00","for":"5h"}`, or `berth schedule book --who ME --engine NAME --at 23:00 --for 5h`. Read `berth schedule` first and pick free hours; a clash is a 409 naming the booking it hits. Book only your own runs: downtime and priority changes are the operator's |
| Plan your own turn on the card (a nightly job, a long run) | `POST /v1/berth/schedule` on the gateway with `{"who":"ME","engine":"NAME","start":"23:00","for":"5h"}`, or `berth schedule book --who ME --engine NAME --at 23:00 --for 5h`. Read `berth schedule` first and pick free hours; a clash is a 409 naming the booking it hits. Book only your own runs: downtime, priority changes and `--claim` are the operator's |
| Free the card for the user (game, call) | `berth hold [--for 2h] [--reason game] [--gpu gpu1]` (ends on Ctrl-C; default the first card) |
| Call a model and let Berth start its engine | the gateway: `http://127.0.0.1:9799/v1`, bearer key from `berth key` |
| Wrap every llama-swap model at once | `berth wire llama-swap --config CFG` (plan), `--write` (apply, keeps a backup), `--write --update` (rewrite wire's flags on lines it wrote, keeping flags added by hand), `--wait 10m` (every model waits that long for the card) |
Expand Down
Loading