diff --git a/.claude/skills/verify-berthd/features/README.md b/.claude/skills/verify-berthd/features/README.md index 11a07f637..1976c64d5 100644 --- a/.claude/skills/verify-berthd/features/README.md +++ b/.claude/skills/verify-berthd/features/README.md @@ -15,8 +15,9 @@ The maintained list of what a person can do in the app, and how to prove each on | Sidebar places: Pinned, More, Hidden, counts on rows (Home's waiting, Review's, a plugin's) | [sidebar-nav.md](sidebar-nav.md) | `plugin-nav-badges` | | Back and forward: the window strip's arrows, ⌘[ and ⌘], the mouse's buttons; archiving or removing the worktree in front goes back, not Home; the dashboard keeps its scroll | [navigation-history.md](navigation-history.md) | `navigation-history`; `app/e2e/back-forward.spec.ts` asserts it | | Shortcuts in the keyboard's layout (⌘Å and ⌘¨ for Back and Forward on Norwegian) in tooltips, the Keyboard shortcuts sheet, Settings and ⌘K; the sheet's descriptions fit their lines | [navigation-history.md](navigation-history.md) (`history-layout`) | `keyboard-layout`, `keyboard-layout-us`; `app/e2e/back-forward.spec.ts` asserts it | -| Stopping agents from the Agent Dashboard: one, the picked ones, a column, Clean up; archiving their worktrees where it's safe | [dashboard-stop.md](dashboard-stop.md) | `dashboard-stop-archive`; `app/e2e/dashboard-stop-archive.spec.ts` asserts it | +| Stopping agents from the Agent Dashboard: one, the picked ones, a column, Clean up; archiving their worktrees where it's safe; the run in the background-jobs tray (progress, fold, Stop the rest, report, fade); a box with runs doing it itself, through the laptop sleeping and the app relaunching | [dashboard-stop.md](dashboard-stop.md) | `dashboard-stop-archive`; `dashboard-stop-live`, `dashboard-stop-lid`, `dashboard-stop-unanswered` and `dashboard-stop-interrupted-live` (a real box, `--agent`); `app/e2e/dashboard-stop-archive.spec.ts` asserts it | | Plugins: built-in and your own, what they add (screens, sidebar items, widgets, panels) | [plugins.md](plugins.md) | `plugin-nav-badges` (sidebar items); `plugin-attach-live` (`api.attach`, a real box, `--agent`) | +| Archiving and removing a worktree: Archive's dialog closes at once, Archiving… until the box is done (and in the tray), one at a time per box, the toasts; Remove worktree; a removal that failed says why, and one stopped halfway is finished by archiving it again | [worktree-archive.md](worktree-archive.md) | `worktree-archive`; `dashboard-stop-interrupted-live` (a real box, `--agent`); `app/e2e/worktree-remove.spec.ts` | | Home's widget grid | [home-widgets.md](home-widgets.md) | none yet; `app/e2e/home-widgets.spec.ts` asserts it | | Questions in chat: an agent's form of questions as a card, answered on a real box; kept with Retry and Copy when it can't be | [chat-questions.md](chat-questions.md) | `chat-questions-live` (a real box, `--agent`); `app/e2e/chat.spec.ts` on the mock | | Chat's foot and reply box: a live turn's spinner, timer, Stop and Queue; a turn that ended while its work runs on shows an hourglass and Send | [chat-turn-state.md](chat-turn-state.md) | `chat-turn-state`; `chat-turn-state-live` (a real box, `--agent`); `app/e2e/chat.spec.ts` asserts it | diff --git a/.claude/skills/verify-berthd/features/dashboard-stop.md b/.claude/skills/verify-berthd/features/dashboard-stop.md index aded969f5..f89ff05fc 100644 --- a/.claude/skills/verify-berthd/features/dashboard-stop.md +++ b/.claude/skills/verify-berthd/features/dashboard-stop.md @@ -1,13 +1,15 @@ # Stopping agents from the Agent Dashboard -Stopping ends an agent's session on its box (`app/src/views/dashboard/stop.tsx`). Its worktree and files stay unless the bulk dialog is told to archive them. The archive goes through `removeWorktreeOnBox(…, "archive")` (`lib/remove-worktree.ts`), the same path as the sidebar's Archive… and the Worktrees table's bulk archive. `lib/archive-after-stop.ts` decides which worktrees to archive and which to keep (unit tests next to it). The box refuses a worktree with uncommitted or untracked files before it stops anything or runs the archive script (`dropWorktree` in `internal/box/api.go`), in git's words, so every Archive path keeps dirty work up front. +Stopping ends an agent's session on its box (`app/src/views/dashboard/stop.tsx`). Its worktree and files stay unless the bulk dialog is told to archive them. The bulk dialog only asks: once confirmed it closes, and the run goes on as a job in the background-jobs tray (`components/jobs-tray.tsx`, store `lib/jobs.ts`, runner `lib/stop-job.ts`). A box with runs does its share itself, as one `stop` run (`internal/box/runs/templates/stop.json`, leaves `stop_session` and `archive_worktree` in `internal/box/runstop.go`, client `lib/stop-on-box.ts`): the job only follows it, so it goes on while the laptop sleeps or the app is closed, and a row fails only when the box says so. The job is kept (`berth.jobs.stopRuns`) before the start is sent, and a box that doesn't answer the start is asked again under the same idempotency key until it does (`startStopRun`), its rows `Waiting for BOX`; **Stop the rest** meanwhile asks the box for the run by its group: one it took is cancelled, and with none the rows read `Not stopped`. Only a box that answers neither reads `Failed: BOX didn't answer, so it may still stop it`. The box applies the same safety rules (never forced, branch kept, dirty / another agent / a failed stop keep it, the `session.stop` and `worktree.remove` before:-gates) and waits for the archive script, so a row reads `Archived` where the app's own loop said `Archiving`. A worktree already on its way out is joined, never archived twice (`dropBegin` in `dropWorktree`), and reads `Archived` once that archive is done. The box's stop runs archive one worktree at a time between them (`archiveTurn`), each looking for agents when its turn comes, and no session starts between that last look and the folder going (`removeAfter`). Boxes without runs, offline at the start, or from before stop runs (the start is refused) get the app's own loop below, for their rows only. The app's loop archives through `removeWorktreeOnBox(…, "archive")` (`lib/remove-worktree.ts`), the same path as the sidebar's Archive… and the Worktrees table's bulk archive, in the same per-box line (`inTurn`). `lib/archive-after-stop.ts` decides which worktrees to archive and which to keep. Unit tests sit next to each of these. The box refuses a worktree with uncommitted or untracked files before it stops its sessions and services or runs the archive script (`dropWorktree` in `internal/box/api.go`), in git's words, so every Archive path keeps dirty work up front. A bulk Stop still stops the agents it was asked to stop: what's kept for uncommitted changes is the worktree and its files, not the agents running. ## Sub-features - `stop-one`: a card's ⋯ → **Stop agent…** (asks first) and **Stop and remove worktree** (removes it without asking; git keeps a dirty one, and a toast offers Remove anyway). - `stop-picked`: **Select**, click cards, **Stop N…** (or ⌫). Also a column's ⋯ → **Stop all N in …**. The dialog lists them by box, each with a checkbox. A busy one says "working" or "needs you". - `cleanup`: **Clean up…** in the header, or a Done/Ready column's ⋯ → **Clean up idle agents…**. It ticks the ones idle longer than 1 hour / 4 hours / 1 day (remembered in `berth.dashboard.cleanupAfter`). A parked agent is listed but never ticked for you. -- `stop-archive`: in both bulk dialogs, **Archive their worktrees where it's safe**. It shows only when a ticked row has a worktree that isn't the main checkout, starts unticked, and is remembered (`berth.dashboard.stopArchives`). Ticked, the agents stop first. Then each of their worktrees is archived once: one at a time on each box, the boxes side by side. A row reads `Stopped · Archived; its branch is kept`, `Stopped · Archiving: its archive script runs, then it goes` (the location has an archive script), `Stopped · Kept: uncommitted changes` (git refused), `Stopped · Kept: another agent is still running there` (an agent session outside the batch is live in it or in a folder under it, by the box's own fresh listing; plain shells and services don't count), `Stopped · Kept: an agent there didn't stop`, `Stopped · Kept: couldn't check for other agents on its box` (its box didn't answer the listing), or `Stopped · Couldn't archive: …`. A kept or failed one holds the dialog open on "Stopped N agents · archived/archiving M worktrees". Otherwise it closes with a toast saying the same. +- `stop-archive`: in both bulk dialogs, **Archive their worktrees where it's safe**. It shows only when a ticked row has a worktree that isn't the main checkout, starts unticked, and is remembered (`berth.dashboard.stopArchives`). Ticked, the agents stop first. Then each of their worktrees is archived once: one at a time on each box, the boxes side by side. A row reads `Stopped · Archived; its branch is kept`, `Stopped · Archiving: its archive script runs, then it goes` (the location has an archive script), `Stopped · Kept: uncommitted changes` (git refused), `Stopped · Kept: another agent is still running there` (an agent session outside the batch is live in it or in a folder under it, by the box's own fresh listing; plain shells and services don't count), `Stopped · Kept: an agent there didn't stop`, `Stopped · Kept: couldn't check for other agents on its box` (its box didn't answer the listing), or `Stopped · Couldn't archive: …` in the box's words: git's own, or when git said nothing how it ended (`git worktree remove timed out after 10 min and was stopped; …`, `… was stopped (signal: killed) and said nothing`, `… failed with no output (exit status 1)`), never a bare `git worktree remove:`. +- `stop-tray`: confirming closes the dialog at once (under 1 s). The tray sits bottom right, above the status bar, the loops panel and a page's floating bar (the board's selection bar), with toasts above it. Its job reads `Stopping N agents · k/N` with every row grouped by box (`devl 3/3`), each `Waiting to stop`, `Stopping…`, `Stopped`, `Stopped · waiting to archive`, `Archiving…`, then its outcome; a stop that fails reads `Failed: …`. The chevron (**Fold**) folds the tray to one line with a spinner, remembered in `berth.jobs.folded`. **Stop the rest** sends nothing more, archives waiting their turn included: what was sent finishes, and the rest read `Not stopped` (or `Stopped · Not archived`). A worktree waiting to archive is marked Archiving… in the sidebar at once, so Archive… there leaves it be. In the app's own loop, one already on its way out reads `Stopped · Already being archived` and isn't asked for again; a box's run waits for that archive instead and reads `Archived`. The app's loop holds a box's line until the box says an archive script finished (`archiveEnded`), so a row reads `Archiving: …` and then `Archived; its branch is kept`. When its turn comes, the box's listing is read again, so an agent started there meanwhile keeps it (`Kept: another agent is still running there`). A clean run shows the old toast ("Stopped N agents · archived M worktrees") and a one-line summary ("17 stopped, 15 archived") that fades after about 10 s. Anything kept, failed or not sent leaves a report, the summary ("4 stopped, 1 archived, 1 archiving, 2 kept") over just those rows, until **Dismiss** (×). The job carries on if you leave the board. On a box with runs, **Stop the rest** cancels the box's run: the step in flight finishes there, the rest read `Not stopped` / `Stopped · Not archived`. +- `stop-reattach`: a job whose rows a box runs is kept in `berth.jobs.stopRuns` (its runs, their params and rows) from before its start is sent until it's dismissed, or for 24 hours at most: a run's journal goes after a while, so a job older than that isn't brought back. After a relaunch the tray brings it back, rows `Waiting for its box` until the box answers, then as the box's run is now or as it ended. A share whose start was never answered is asked for again under the same key: the box has it, or takes it then. A job you pressed **Stop the rest** on is kept as cancelled (`onCancel`) and comes back cancelled: an unanswered share is only looked up by its group, its run cancelled if the box took it, and never sent again. A laptop that sleeps mid-run just reads the run again when it wakes. The app's own loop (boxes without runs) still doesn't survive the app quitting. ## How to get to it (user POV) @@ -15,13 +17,20 @@ Stopping ends an agent's session on its box (`app/src/views/dashboard/stop.tsx`) ## Driving it with drive.mjs -- `node .claude/skills/verify-berthd/scripts/drive.mjs dashboard-stop-archive --port $PORT` picks four agents. It covers each outcome: search-perf archives through shop's archive script, ci-flake on gpu archives at once, checkout-fix is kept for uncommitted changes, and order-export is kept because two more agents run there. It ticks the box, stops them, checks each row, then checks the Worktrees table (two gone, two still there), and that **Clean up…** opens with the box ticked. +- `node .claude/skills/verify-berthd/scripts/drive.mjs dashboard-stop-archive --port $PORT` opens the mock with `noruns` (the app's own loop, for boxes without runs), `slowremove=4000` and `slowstop=1500`, and picks four agents. It covers each outcome: search-perf archives through shop's archive script, ci-flake on gpu archives after gpu's 4 s, checkout-fix is kept for uncommitted changes, and order-export is kept because two more agents run there. It ticks the box and stops them, and fails if the dialog stays open over 1 s. Then it shoots the tray mid-run (ci-flake `Archiving…`), folds it, opens the Worktrees view while the job runs, and checks the report rows and the table (two gone, two still there). It also checks that **Clean up…** opens with the box ticked, that a clean one-agent run fades on its own, and that **Stop the rest** leaves one of four `Not stopped`. +- On a real box, the tray end to end: `scripts/dev-box.sh up`, `eval "$(scripts/dev-box.sh env)"`, then `drive.mjs dashboard-stop-live --port $PORT --agent http://127.0.0.1:9378`. It makes its own worktrees of acme-shop (`verify-stop-a/b/c`, a Claude Code in each; `verify-stop-c` gets an uncommitted file; and `verify-solo`). It stops the three with archiving ticked, expects the report `3 stopped, 2 archived, 1 kept`, and reads `git worktree list` and `git branch` on the box: a and b gone, c still there, every branch kept. No request is refused: the box keeps verify-stop-c inside its run, and the scenario reads the box's stop runs back (`Stopping 3 agents: succeeded`). Then it archives `verify-solo` from the sidebar, waits for **Archived verify-solo** in the tray, checks git again, and finds that one a stop run on the box too (`Archive verify-solo: succeeded`). `dev-box.sh down` removes it all. It never touches a session it didn't start. - On a real box: `scripts/dev-box.sh up`, then `scripts/archive-dirty-live.sh` archives a dirty and a clean worktree of acme-shop under an archive script, through the agent's API. The dirty one is refused at once (400, "contains modified or untracked files"), and its file stays while the script doesn't run. The clean one answers 202, the script runs in it, and it goes. `dev-box.sh down` afterwards. -- `app/e2e/dashboard-stop-archive.spec.ts` asserts the same, plus unticked (nothing archived, the old copy and toast) and a main checkout alone (no box). +- The lid closing, on a real box: `scripts/dev-box.sh up`, `eval "$(scripts/dev-box.sh env)"`, then `drive.mjs dashboard-stop-lid --port $PORT --agent http://127.0.0.1:9378`. It makes seven throwaway worktrees of acme-shop with a Claude Code in each (`verify-lid-dirty` gets an uncommitted file, `verify-lid-shared` a second agent outside the batch), gives acme-shop a 6 s archive script, stops five with archiving ticked, and pauses the laptop agent (SIGSTOP) for 25 s while the box archives. Woken, the tray must read `5 stopped, 3 archived, 2 kept`, and git on the box agrees (branches kept, the outsider still running). Then it stops the other two and reloads the page mid-run: the job comes back from the box and ends `2 stopped, 2 archived`. +- The box going away as Stop is pressed, on a real box: `scripts/dev-box.sh up`, `eval "$(scripts/dev-box.sh env)"`, then `drive.mjs dashboard-stop-unanswered --port $PORT --agent http://127.0.0.1:9378` (with `VERIFY_BOX_DIR` if `up` had one). It makes two throwaway worktrees with a Claude Code in each, presses Stop with archiving ticked and kills the box's berthd (SIGKILL) at once, and starts it again 20 s later as `dev-box.sh up` does (its pid goes back in `berthd.pid`). Meanwhile no row may fail; they read `Waiting for devq` when the start went unanswered. Back, the job must end `2 stopped, 2 archived`, the box must have exactly one stop run for it (whether it took the first ask and resumed it, or took the ask sent again under the same key), and git agrees. +- A removal git doesn't finish, on a real box: `scripts/dev-box.sh up`, `eval "$(scripts/dev-box.sh env)"`, then `drive.mjs dashboard-stop-interrupted-live --port $PORT --agent http://127.0.0.1:9378` (with `VERIFY_BOX_DIR` if `up` had one). It restarts the box with a git on its PATH that, for `verify-slow` only, deletes the worktree's `.git` and `.gitignore` and then hangs, as a big folder's `git worktree remove` on a busy box did: acme-shop gets a committed `.gitignore` for `node_modules` first, and `verify-slow` a `node_modules`, so what's left holds files that only the lost `.gitignore` ignored, and with `BERTH_WORKTREE_REMOVE_TIMEOUT=5s`. It stops `verify-slow` and `verify-ok` with archiving ticked. The report must read `2 stopped, 1 archived, 1 failed`, and the failed row `Stopped · Couldn't archive: git worktree remove timed out after 5s and was stopped; the folder may be partly removed, and archiving it again finishes it`, never a bare `git worktree remove:`. git lists `verify-slow` as prunable with its files still there. Then **Archive…** from the sidebar finishes it: **Archived verify-slow**, the folder gone, git no longer lists it, both branches kept. At the end the box is restarted with its own git. `VERIFY_BERTHD=` runs it on another build for a before: the old box stops git after 60 s and the row ends at `git worktree remove:`. Make a fresh dev box for each pass (`down`, `up`), as it makes fixed-name worktrees. +- `app/e2e/dashboard-stop-archive.spec.ts` asserts the same on the mock, where devl and gpu play their stop runs out (`playStop` in `lib/mock-runs.ts`), plus `?noruns` (every box from before runs: the app's own loop, an archive script still running reads `archiving`), unticked (nothing archived, the old copy and toast), a main checkout alone (no box) and leaving the board mid-run. ## Gotchas - With the default prefs the sidebar is the rail layout, which shows one project at a time. Read the outcome in the Worktrees table, or open with `labs: false` prefs, as the e2e does, to use the sidebar's `worktree-row`s. - In the mock, checkout-fix's agent is in Needs you, not Done: it is the only dirty worktree in the fixtures. -- The mock's archive script takes 1.5s, so search-perf leaves the table a moment after the dialog says Archiving. +- The mock's archive script takes 1.5s, so search-perf leaves the table a moment after its row says Archiving. +- `?slowremove` slows only a worktree with no archive script (ci-flake); `?slowstop=MS` slows every stop's answer, which is what lets a test press **Stop the rest** with rows still unsent. `?noruns` takes `runs` out of every box's capabilities, for the app's own loop. +- The mock's stop run waits 1.6 s for shop's archive script, as a box does, so search-perf reads `Archived`, not `Archiving`, on devl. +- The tray is `getByRole("region", { name: "Background jobs" })`, a job `getByRole("group", { name: "Stopping N agents" })` (its first title, kept after it ends), its rows `listitem`s. - axe over the open dialog flags the board's cards behind it (`.opacity-70` meta text), not the dialog. diff --git a/.claude/skills/verify-berthd/features/worktree-archive.md b/.claude/skills/verify-berthd/features/worktree-archive.md new file mode 100644 index 000000000..a9e2cc5d5 --- /dev/null +++ b/.claude/skills/verify-berthd/features/worktree-archive.md @@ -0,0 +1,35 @@ +# Archiving and removing a worktree + +A worktree that is done goes one of two ways. **Archive…** asks first, then stops its sessions and services, runs the repo's archive script if it has one, and removes the folder; the branch stays, except a review worktree's own `review/` branch, which goes with it. **Remove worktree** asks nothing and removes it; the branch stays too, with the same exception. Both go through one queue (`app/src/lib/remove-worktree.ts`): the worktree is marked at once (`lib/removing.ts`), and each box takes one removal at a time, so archiving several in a row never waits on a dialog. On a box with runs, Archive… is a one-worktree `stop` run there (`archiveOnBox`, as the board's bulk Stop: `features/dashboard-stop.md`): its sessions stop, then it's archived where it's safe, in the box's own line (`archiveTurn`), and it finishes while the laptop sleeps or the app is closed. Elsewhere, and for Remove, the app sends it in the box's line (`inTurn`), which it holds until the box says the worktree went or the script failed (`archiveEnded`). + +## Sub-features + +- `archive-confirm`: the dialog names the worktree, its branch ("Branch … stays") and the archive script, if any. It closes as soon as Archive is pressed, before the box answers. +- `archive-mark`: from that moment the row is struck through and says **Archiving…** (`[data-leaving]`, `LeavingRow` in `components/sidebar/projects.tsx`); zen's switcher says the same, and the worktree's launcher says "Archiving NAME…". It can't be opened meanwhile. +- `archive-tray`: the archive is also a one-row job in the background-jobs tray (`queueRemoval(…, { kind: "archive" })`, `components/jobs-tray.tsx`): **Archiving NAME**, its row `Waiting for BOX` until its turn (or until the box answers), then `Archiving…` (and `Archiving: its archive script runs, then it goes` while the app waits on a script). Once the worktree is gone it reads **Archived NAME** and fades; kept, it stays as a report **Kept NAME** / `Kept: uncommitted changes` (or the box's other reason, such as `Kept: another agent is still running there`) until dismissed. Only Archive… from the sidebar, zen and the launcher uses the tray; Remove and the Worktrees table don't. +- `archive-done`: without a script the box answers once the folder is gone, then a toast **Archived NAME** / "Branch … is kept." With a script the box answers at once: a toast **Archiving NAME** / "Its archive script is running on the box…", and **Archived NAME** when `worktree.removed` arrives. On a box with runs it is one toast, **Archived NAME**, once the box's run has the worktree gone; kept there, **Couldn't archive NAME** says why. +- `archive-dirty`: uncommitted work makes git refuse. The worktree comes back as it was, with a warning toast **Couldn't archive NAME** / "NAME has uncommitted changes, so it was left as it is. Commit them first, or use Remove worktree to discard them." +- `archive-queue`: archives (and removals) on one box go one after another; each row keeps its mark until its turn is done, archive script included. Another box's don't wait. +- `remove`: Remove worktree, the dashboard's Stop and remove worktree, and Delete in the Worktrees table: no dialog, **Removing…**, and a dirty one gives **Kept NAME** with **Remove anyway**. +- `archive-interrupted`: git has 10 minutes to remove a worktree (`BERTH_WORKTREE_REMOVE_TIMEOUT` on the box sets another limit). One it doesn't finish in time is stopped, and the row says so (`git worktree remove timed out after … and was stopped; the folder may be partly removed, and archiving it again finishes it`). A removal stopped halfway leaves a folder with no `.git`, which git lists as prunable and the sidebar still shows. Archive… on it removes what's left and git's record of that one worktree (never `git worktree prune`, which would drop every half-removed one's), keeps the branch, and reads **Archived NAME**. It does so only when git, asked through the record's HEAD and index, sees nothing but files already deleted (any `.gitignore` the removal deleted is put back as committed first, so `node_modules` doesn't read as new); a changed or new file keeps it as `Kept: uncommitted changes`. So does one whose folder is gone altogether, which isn't listed. A folder that still has a `.git` is never deleted by hand. +- Bulk Archive in the Worktrees table (`views/worktrees/archive-dialog.tsx`) is separate: its dialog stays open on purpose, with each row's outcome (`use-bulk.ts`). On a box with runs, its eligible rows (and Stop sessions') are one `stop` run there (`startOnBoxes`, as the board's bulk Stop): the worktree's sessions stop first, then it's archived where it's safe, and a row the box keeps reads `Kept: …` (skipped). A row with uncommitted changes and every row of a box without runs go one after another from the app, as before; a row already on its way out is skipped (`already on its way out`) and left to that removal. + +## How to get to it (user POV) + +- Right-click a worktree in the sidebar, or its ⋯: **Archive…** or **Remove worktree**. Zen's worktree menu has the same items. +- A worktree with no tabs shows its launcher: **Archive this worktree…**. +- Worktrees (sidebar): select rows, then **Archive N…** or **Delete** in the bar at the bottom. + +## Driving it with drive.mjs + +- `node .claude/skills/verify-berthd/scripts/drive.mjs worktree-archive --port $PORT` opens the mock with `slowremove=4000` (a box takes 4 s to remove a worktree with no archive script, as a big `node_modules` does) and the list sidebar. It archives `gpu/ci-flake` and fails if the dialog stays open over 1 s, checks its row and its tray job say Archiving…, archives `gpu/judge-v2` straight after (both marked; judge-v2 waits its turn, `Waiting for gpu` in the tray), waits for **Archived ci-flake** with "Branch me/ci-flake is kept." at about 4 s and **Archived judge-v2** at about 8 s, then archives `devl/qa-deck` (script `pnpm db:drop`: **Archiving qa-deck**, then **Archived qa-deck**) and `devl/checkout-fix` (uncommitted: **Couldn't archive checkout-fix**, row still there, and a **Kept checkout-fix** report in the tray, then dismissed). Each step logs how long the dialog stayed open. +- Hooks: `[data-testid=worktree-row][data-worktree="BOX/NAME"]` for a row, `[data-leaving]` with the name's text for its leaving state, `getByRole("alertdialog")` for the dialog and its exact **Archive** button, `[data-slot=toast-viewport] > *` with the title's text for a toast. +- `app/e2e/worktree-remove.spec.ts` asserts the same on the mock: Remove (no dialog, Remove anyway), Archive (dialog gone in 1 s with `slowremove`, Archiving…, the toast) and Delete in the table. + +## Gotchas + +- `?slowremove=MS` slows only a worktree with no archive script; one with a script (devl's shop) answers at once and finishes 1.5 s later, as on a box. `checkout-fix` always refuses without force. +- The default sidebar is the rail, which shows one project at a time; the scenario sets `sidebar=list` to see devl's and gpu's worktrees together, as the e2e specs do. +- A worktree already marked is ignored by a second Archive or Remove until its mark goes. +- `dashboard-stop-interrupted-live` (see [dashboard-stop.md](dashboard-stop.md)) drives `archive-interrupted` on a real box, with a git that hangs for one worktree. +- The real box's slow part (deleting a big folder inside `git worktree remove`) isn't in the mock; a real box is needed to time it (`../scripts/dev-box.sh`). `dashboard-stop-live` (see [dashboard-stop.md](dashboard-stop.md)) archives `verify-solo` from the sidebar on one and checks git agrees. diff --git a/.claude/skills/verify-berthd/scripts/scenarios/dashboard-stop-archive.mjs b/.claude/skills/verify-berthd/scripts/scenarios/dashboard-stop-archive.mjs index 856aefe9e..680c0d46e 100644 --- a/.claude/skills/verify-berthd/scripts/scenarios/dashboard-stop-archive.mjs +++ b/.claude/skills/verify-berthd/scripts/scenarios/dashboard-stop-archive.mjs @@ -1,13 +1,21 @@ // The Agent Dashboard's bulk Stop, with "Archive their worktrees where it's -// safe" ticked (views/dashboard/stop.tsx): a clean worktree with an archive -// script archives in the background, one without archives at once, a dirty -// one is kept, and one with another agent still in it is kept. +// safe" ticked (views/dashboard/stop.tsx): the dialog closes at once and +// the run goes on in the background-jobs tray (components/jobs-tray.tsx). +// With ?noruns every box is one from before runs, so this is the app's own +// loop, the fallback (boxes with runs: dashboard-stop-lid on a real box, and +// the e2e). A clean worktree with an archive script archives in the +// background, one without archives (slowly: ?slowremove), a dirty one is +// kept, and one with another agent still in it is kept; the tray reports +// those two. Then a +// clean run fades, and Stop the rest leaves the unsent ones (?slowstop). // features/dashboard-stop.md. +export const params = { slowremove: "4000", slowstop: "1500", noruns: "1" }; + const picks = [ // devl / search-perf: shop has an archive script, so it runs first. "Speed up product search", - // gpu / ci-flake: no archive script, so it goes at once. + // gpu / ci-flake: no archive script, so it goes after gpu's 4 s. "Move acme billing to the new ledger", // devl / checkout-fix: uncommitted work, so git keeps it. "Fix checkout webhook retries", @@ -18,48 +26,116 @@ const picks = [ export async function run({ page, shot, step, expect }) { const dialog = page.getByRole("alertdialog"); const archiveBox = dialog.getByRole("checkbox", { name: /Archive their worktrees where it's safe/ }); - const row = (place) => dialog.getByRole("group", { name: "Agents to stop" }).getByRole("listitem").filter({ hasText: place }); + const tray = page.getByRole("region", { name: "Background jobs" }); + const job = (title) => tray.getByRole("group", { name: title }); + const jobRow = (title, label) => job(title).getByRole("listitem").filter({ hasText: label }); + const toast = (text) => page.locator("[data-slot=toast-viewport] > *", { hasText: text }); // The Worktrees table lists every box's; the sidebar shows one project. const listed = (name) => page.getByRole("checkbox", { name: `Select ${name}`, exact: true }); + const board = async () => { + await page.keyboard.press("Meta+j"); + await expect(page.getByRole("heading", { name: "Agent Dashboard" })).toBeVisible(); + }; + const stopPicked = async (titles) => { + // Still picking after the last run, or not yet. + const select = page.getByRole("button", { name: "Select", exact: true }); + if (await select.isVisible()) await select.click(); + for (const t of titles) await page.locator("article[role=button]", { hasText: t }).getByText(t).click(); + await page.getByRole("button", { name: `Stop ${titles.length}…` }).click(); + await expect(dialog).toBeVisible(); + }; + const confirmStop = async (n, label) => { + const t0 = Date.now(); + await dialog.getByRole("button", { name: `Stop ${n} agent${n === 1 ? "" : "s"}` }).click(); + const closed = dialog.waitFor({ state: "hidden", timeout: 15_000 }).then(() => Date.now() - t0); + await page.waitForTimeout(300); + await shot(label); + const ms = await closed; + step(`the dialog closed ${ms} ms after Stop was pressed`); + if (ms > 1000) throw new Error(`the dialog stayed open ${ms} ms, waiting on the boxes`); + }; step("open the Agent Dashboard, Select, and pick four agents"); - await page.keyboard.press("Meta+j"); - await expect(page.getByRole("heading", { name: "Agent Dashboard" })).toBeVisible(); - await page.getByRole("button", { name: "Select", exact: true }).click(); - for (const t of picks) await page.locator("article[role=button]", { hasText: t }).getByText(t).click(); - await page.getByRole("button", { name: `Stop ${picks.length}…` }).click(); + await board(); + await stopPicked(picks); await expect(archiveBox).not.toBeChecked(); await expect(dialog).toContainText("its worktree and files are kept"); - await shot("1-stop-dialog-unticked"); + await shot("01-stop-dialog-unticked"); step("tick 'Archive their worktrees where it's safe': the copy says so"); await archiveBox.click(); await expect(archiveBox).toBeChecked(); await expect(dialog).toContainText("then its worktree is archived where it's safe"); - await shot("2-stop-dialog-ticked"); + await shot("02-stop-dialog-ticked"); + + step("Stop 4 agents: the dialog goes at once, the tray takes over"); + await confirmStop(picks.length, "03-just-after-stop"); + const first = job("Stopping 4 agents"); + await expect(first).toContainText(/Stopping 4 agents · [0-3]\/4/); + + step("mid-run: the tray lists each by box, ci-flake Archiving… on gpu"); + await expect(jobRow("Stopping 4 agents", "ci-flake")).toContainText("Archiving…"); + await expect(first).toContainText("devl"); + await expect(first).toContainText("gpu"); + await shot("04-tray-mid-run"); - step("Stop 4 agents: kept work holds the dialog open, each row with its outcome"); - await dialog.getByRole("button", { name: `Stop ${picks.length} agents` }).click(); - await expect(dialog.getByRole("heading")).toHaveText("Stopped 4 agents · archiving 2 worktrees"); - await expect(row("search-perf")).toContainText("Stopped · Archiving: its archive script runs, then it goes"); - await expect(row("ci-flake")).toContainText("Stopped · Archived; its branch is kept"); - await expect(row("checkout-fix")).toContainText("Stopped · Kept: uncommitted changes"); - await expect(row("order-export")).toContainText("Stopped · Kept: another agent is still running there"); - await shot("3-stop-dialog-done"); + step("fold it: one line with the count and a spinner"); + await first.getByRole("button", { name: "Fold" }).click(); + const pill = tray.getByRole("button", { expanded: false }); + await expect(pill).toContainText(/Stopping 4 agents · [0-3]\/4/); + await shot("05-tray-folded"); - step("Done, then the Worktrees table: the archived ones are gone, the kept ones stay"); - await dialog.getByRole("button", { name: "Done" }).click(); + step("go elsewhere while it runs: the Worktrees view opens, the tray stays"); await page.getByTestId("nav-worktrees").click(); await expect(listed("checkout-fix")).toBeVisible(); + await expect(tray).toBeVisible(); + await shot("06-elsewhere-while-running"); + await pill.click(); + + step("it ends as a report of what was kept, each saying why"); + await expect(first).toContainText("4 stopped, 2 archived, 2 kept", { timeout: 15_000 }); + await expect(first.getByRole("listitem")).toHaveCount(2); + await expect(jobRow("Stopping 4 agents", "checkout-fix")).toContainText("Stopped · Kept: uncommitted changes"); + await expect(jobRow("Stopping 4 agents", "order-export")).toContainText("Stopped · Kept: another agent is still running there"); + await shot("07-tray-report"); + + step("in the Worktrees table the archived ones are gone, the kept ones stay"); + await expect(listed("checkout-fix")).toBeVisible(); await expect(listed("order-export")).toBeVisible(); await expect(listed("ci-flake")).toHaveCount(0, { timeout: 10_000 }); await expect(listed("search-perf")).toHaveCount(0, { timeout: 10_000 }); - await shot("4-worktrees-after"); + + step("dismiss the report"); + await first.getByRole("button", { name: "Dismiss" }).click(); + await expect(tray).toHaveCount(0); step("the choice is remembered: Clean up… opens with the box ticked"); - await page.keyboard.press("Meta+j"); + await board(); await page.getByRole("button", { name: "Clean up…" }).click(); await expect(archiveBox).toBeChecked(); await expect(dialog).toContainText("Their worktrees are archived where it's safe"); - await shot("5-cleanup-remembers"); + await shot("08-cleanup-remembers"); + await dialog.getByRole("button", { name: "Cancel" }).click(); + + step("a clean run (archive unticked): the toast, a one-line summary, then it fades"); + await stopPicked(["Tune the judge prompt"]); + await archiveBox.click(); + await expect(archiveBox).not.toBeChecked(); + await confirmStop(1, "09-clean-just-after-stop"); + await expect(toast("Stopped 1 agent")).toContainText("Their worktrees are kept.", { timeout: 10_000 }); + await expect(job("Stopping 1 agent")).toContainText("1 stopped"); + await shot("10-clean-summary"); + await expect(tray).toHaveCount(0, { timeout: 15_000 }); + step("the clean run's summary faded on its own"); + await shot("11-clean-faded"); + + step("Stop the rest: four picked, three in flight, the fourth never sent"); + await stopPicked(["Add tests for the export job", "Rerun the evals once the vendor replies", "Fix the cart badge after a refund", "Plan the checkout release"]); + await confirmStop(4, "12-four-just-after-stop"); + const rest = job("Stopping 4 agents"); + await expect(rest.getByRole("listitem").filter({ hasText: "Stopping…" })).toHaveCount(3); + await rest.getByRole("button", { name: "Stop the rest" }).click(); + await expect(rest).toContainText("3 stopped, 1 not stopped", { timeout: 10_000 }); + await expect(rest.getByRole("listitem").filter({ hasText: "Not stopped" })).toHaveCount(1); + await shot("13-stop-the-rest-report"); } diff --git a/.claude/skills/verify-berthd/scripts/scenarios/dashboard-stop-interrupted-live.mjs b/.claude/skills/verify-berthd/scripts/scenarios/dashboard-stop-interrupted-live.mjs new file mode 100644 index 000000000..2524bc6aa --- /dev/null +++ b/.claude/skills/verify-berthd/scripts/scenarios/dashboard-stop-interrupted-live.mjs @@ -0,0 +1,125 @@ +// dashboard-stop-interrupted-live: a removal git doesn't finish in time, +// on a real box (dev-box.sh's devq), as features/dashboard-stop.md +// describes. The box is restarted with a git on its PATH that, for +// verify-slow only, deletes the worktree's .git and .gitignore (so its +// node_modules reads as new) and then hangs, as a +// `git worktree remove` of a big folder on a busy box did, and with +// BERTH_WORKTREE_REMOVE_TIMEOUT=5s. A bulk Stop of verify-slow and +// verify-ok must report the cause in words (timed out), never a bare +// "git worktree remove:", and leave verify-slow half removed. Archive… +// from the sidebar then finishes it: the folder goes, git drops it, the +// branch stays. The box gets its own git back at the end. +// VERIFY_BERTHD= runs it on another build (a before: an old box +// stops git after 60 s and says nothing). +import { execFileSync } from "node:child_process"; +import { chmodSync, existsSync, mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { fileURLToPath } from "node:url"; + +export const live = true; +export const params = { sidebar: "list" }; +// The box is restarted twice on purpose: the agent answers 503 meanwhile. +// A stopped agent's card can still ask for its screen once (404); every +// failed request is logged as a step, so that one shows as such. +export const allowErrors = [/status of 503/, /status of 404/, /net::ERR_/]; + +const API = "/v1/boxes/devq/api"; +const SLOW = "verify-slow"; +const OK = "verify-ok"; +const devBox = fileURLToPath(new URL("../dev-box.sh", import.meta.url)); + +function restartBox(env) { + const args = ["restart", ...(process.env.VERIFY_BERTHD ? [process.env.VERIFY_BERTHD] : [])]; + return execFileSync(devBox, args, { encoding: "utf8", env: { ...process.env, ...env } }).trim(); +} + +export async function run({ page, shot, step, expect, call }) { + // Which request failed, since the console doesn't say. + page.on("response", (r) => r.status() >= 400 && step(`(${r.status()} ${r.request().method()} ${new URL(r.url()).pathname})`)); + const post = (path, body) => call(`${API}${path}`, { method: "POST", body: JSON.stringify(body) }); + const tray = page.getByRole("region", { name: "Background jobs" }); + const dialog = page.getByRole("alertdialog"); + const title = (n) => `Throwaway ${n}`; + + step(`restart the box with a git that deletes ${SLOW}'s .git and then hangs, and a 5 s removal budget`); + const bin = mkdtempSync(join(tmpdir(), "verify-git-")); + const real = execFileSync("sh", ["-c", "command -v git"], { encoding: "utf8" }).trim(); + writeFileSync(join(bin, "git"), `#!/bin/sh\ncase "$3 $4 $5" in\n "worktree remove "*/acme-shop-${SLOW}) rm -f "$5/.git" "$5/.gitignore"; exec sleep 900 ;;\nesac\nexec ${real} "$@"\n`); + chmodSync(join(bin, "git"), 0o755); + step(restartBox({ PATH: `${bin}:${process.env.PATH}`, BERTH_WORKTREE_REMOVE_TIMEOUT: "5s" })); + try { + await stopAndFinish({ page, shot, step, expect, call, post, tray, dialog, title }); + } finally { + step(`give the box its own git back: ${restartBox({})}`); + rmSync(bin, { recursive: true, force: true }); + } +} + +async function stopAndFinish({ page, shot, step, expect, call, post, tray, dialog, title }) { + // The restarted box answers again before anything is asked of it. + await expect.poll(async () => (await call(`${API}/locations`).catch(() => [])).length, { timeout: 30_000 }).toBeGreaterThan(0); + // A removal stopped halfway can delete .gitignore before what it ignores. + const acme = (await call(`${API}/locations`)).find((l) => l.name === "acme-shop").path; + if (!existsSync(join(acme, ".gitignore"))) { + writeFileSync(join(acme, ".gitignore"), "node_modules\n"); + execFileSync("git", ["-C", acme, "add", ".gitignore"]); + execFileSync("git", ["-C", acme, "-c", "user.name=acme", "-c", "user.email=dev@acme.test", "commit", "-qm", "ignore node_modules"]); + } + step(`make ${SLOW} and ${OK} in acme-shop, a Claude Code in each, and node_modules in ${SLOW}`); + const paths = {}; + for (const n of [SLOW, OK]) paths[n] = (await post("/locations/acme-shop/worktrees", { name: n })).path; + mkdirSync(join(paths[SLOW], "node_modules", "left-pad"), { recursive: true }); + writeFileSync(join(paths[SLOW], "node_modules", "left-pad", "index.js"), "module.exports = 1\n"); + for (const n of [SLOW, OK]) await post("/sessions", { name: `${n}-claude`, location: `acme-shop/${n}`, agent: "claude", title: title(n) }); + const repo = (await call(`${API}/locations`)).find((l) => l.name === "acme-shop").path; + const git = (...args) => execFileSync("git", ["-C", repo, ...args], { encoding: "utf8" }); + const worktrees = () => git("worktree", "list", "--porcelain"); + const branches = () => git("branch", "--list", "--format=%(refname:short)").split("\n"); + + step("stop both from the Agent Dashboard, archiving ticked"); + await page.keyboard.press("Meta+j"); + await expect(page.getByRole("heading", { name: "Agent Dashboard" })).toBeVisible(); + for (const n of [SLOW, OK]) await expect(page.locator("article[role=button]", { hasText: title(n) })).toBeVisible({ timeout: 30_000 }); + await page.getByRole("button", { name: "Select", exact: true }).click(); + for (const n of [SLOW, OK]) await page.locator("article[role=button]", { hasText: title(n) }).getByText(title(n)).click(); + await page.getByRole("button", { name: "Stop 2…" }).click(); + const archiveBox = dialog.getByRole("checkbox", { name: /Archive their worktrees where it's safe/ }); + if (!(await archiveBox.isChecked())) await archiveBox.click(); + await dialog.getByRole("button", { name: "Stop 2 agents" }).click(); + await expect(dialog).toBeHidden({ timeout: 1000 }); + const job = tray.getByRole("group", { name: "Stopping 2 agents" }); + // An old box gives up on git after 60 s. + await expect(job).toContainText("2 stopped, 1 archived, 1 failed", { timeout: 120_000 }); + const row = job.getByRole("listitem"); + await expect(row).toHaveCount(1); + const said = (await row.innerText()).replace(/\s+/g, " ").trim(); + step(`the failed row reads: ${said}`); + await shot("interrupted-1-tray-report"); + if (/git worktree remove:\s*$/.test(said)) throw new Error(`the row ends at "git worktree remove:", with no cause: ${said}`); + await expect(row).toContainText("Couldn't archive: git worktree remove timed out after 5s and was stopped"); + await expect(row).toContainText("archiving it again finishes it"); + + step(`git agrees: ${OK} is gone, ${SLOW} is half removed (prunable, its files still there)`); + const listed = worktrees(); + if (listed.includes(`worktree ${paths[OK]}\n`)) throw new Error(`${OK}'s worktree is still there`); + const entry = listed.split("\n\n").find((e) => e.startsWith(`worktree ${paths[SLOW]}\n`)); + if (!entry?.includes("prunable")) throw new Error(`git doesn't list ${SLOW} as prunable:\n${listed}`); + if (existsSync(join(paths[SLOW], ".git")) || existsSync(join(paths[SLOW], ".gitignore")) || !existsSync(join(paths[SLOW], "node_modules"))) throw new Error(`${SLOW} isn't half removed`); + step(`git worktree list:\n${listed.trim()}`); + await job.getByRole("button", { name: "Dismiss" }).click(); + + step(`Archive… ${SLOW} from the sidebar finishes it`); + await page.locator(`[data-testid=worktree-row][data-worktree="devq/${SLOW}"]`).click({ button: "right" }); + await page.getByRole("menuitem", { name: "Archive…" }).click(); + await dialog.getByRole("button", { name: "Archive", exact: true }).click(); + await expect(dialog).toBeHidden({ timeout: 1000 }); + const solo = tray.getByRole("group", { name: `Archiving ${SLOW}` }); + await expect(solo).toContainText(`Archived ${SLOW}`, { timeout: 30_000 }); + await shot("interrupted-2-finished"); + if (existsSync(paths[SLOW])) throw new Error(`${SLOW}'s folder is still there`); + if (worktrees().includes(paths[SLOW])) throw new Error(`git still lists ${SLOW}`); + for (const n of [SLOW, OK]) if (!branches().some((b) => b.endsWith(n))) throw new Error(`${n}'s branch is gone`); + await expect(page.locator(`[data-testid=worktree-row][data-worktree="devq/${SLOW}"]`)).toHaveCount(0); + step(`its folder is gone, git no longer lists it, and both branches stay:\n${worktrees().trim()}`); +} diff --git a/.claude/skills/verify-berthd/scripts/scenarios/dashboard-stop-lid.mjs b/.claude/skills/verify-berthd/scripts/scenarios/dashboard-stop-lid.mjs new file mode 100644 index 000000000..2d26bd5e5 --- /dev/null +++ b/.claude/skills/verify-berthd/scripts/scenarios/dashboard-stop-lid.mjs @@ -0,0 +1,147 @@ +// dashboard-stop-lid: the board's bulk Stop goes on while the laptop +// sleeps, on a real box (dev-box.sh's devq), as features/dashboard-stop.md +// describes. It makes its own throwaway worktrees of acme-shop with a +// Claude Code in each: three clean, one with an uncommitted file, and one +// shared with a second agent that isn't stopped. acme-shop gets a slow +// archive script, so the run is still archiving when the laptop agent is +// paused (SIGSTOP, the lid closing). The box carries on; once the agent is +// back the tray shows how it ended, and git on the box agrees. Then a +// second Stop, and the app is reloaded (a relaunch) mid-run: the tray +// brings the job back from the box. Last, the Worktrees table's Stop +// sessions and Archive, each a stop run on the box too. Nothing it didn't +// make is touched; dev-box.sh down removes the lot. +import { execFileSync } from "node:child_process"; +import { readFileSync, writeFileSync } from "node:fs"; +import { join } from "node:path"; + +export const live = true; +export const params = { sidebar: "list" }; +// While the agent is paused, the app's requests to it fail. +export const allowErrors = [/Failed to load resource/, /net::ERR_/]; + +const API = "/v1/boxes/devq/api"; +const LID = ["verify-lid-a", "verify-lid-b", "verify-lid-c", "verify-lid-dirty", "verify-lid-shared"]; +const RELAUNCH = ["verify-relaunch-a", "verify-relaunch-b"]; +const DIR = process.env.VERIFY_BOX_DIR ?? "/tmp/verify-berthd-box"; + +// The dev box's laptop agent: the one on 9378 with dev-box.sh's state. +function agentPid() { + const out = execFileSync("ss", ["-ltnpH", "sport = :9378"], { encoding: "utf8" }); + for (const m of out.matchAll(/pid=(\d+)/g)) { + const env = readFileSync(`/proc/${m[1]}/environ`, "utf8").split("\0"); + if (env.includes(`BERTH_HOME=${DIR}/lap-home`)) return Number(m[1]); + } + throw new Error("no dev-box agent on 9378"); +} + +export async function run({ page, shot, step, expect, call }) { + const post = (path, body) => call(`${API}${path}`, { method: "POST", body: JSON.stringify(body) }); + const tray = page.getByRole("region", { name: "Background jobs" }); + const dialog = page.getByRole("alertdialog"); + const title = (n) => `Throwaway ${n}`; + const card = (n) => page.locator("article[role=button]", { hasText: title(n) }); + const repo = (await call(`${API}/locations`)).find((l) => l.name === "acme-shop").path; + const git = (...args) => execFileSync("git", ["-C", repo, ...args], { encoding: "utf8" }); + const worktrees = () => git("worktree", "list", "--porcelain"); + const branches = () => git("branch", "--list", "--format=%(refname:short)").split("\n"); + const paths = {}; + + step("make the worktrees, a Claude Code in each, an uncommitted file in verify-lid-dirty and a second agent in verify-lid-shared"); + for (const n of [...LID, ...RELAUNCH]) paths[n] = (await post("/locations/acme-shop/worktrees", { name: n })).path; + for (const n of [...LID, ...RELAUNCH]) await post("/sessions", { name: `${n}-claude`, location: `acme-shop/${n}`, agent: "claude", title: title(n) }); + await post("/sessions", { name: "verify-lid-outsider", location: "acme-shop/verify-lid-shared", agent: "claude", title: "Throwaway outsider" }); + writeFileSync(join(paths["verify-lid-dirty"], "notes.txt"), "not committed\n"); + step("acme-shop's archive script takes 6 s, so the run is mid-archive when the lid closes"); + await call(`${API}/locations/acme-shop/scripts`, { method: "PUT", body: JSON.stringify({ archive: "sleep 6" }) }); + + const pick = async (names) => { + await page.keyboard.press("Meta+j"); + await expect(page.getByRole("heading", { name: "Agent Dashboard" })).toBeVisible(); + for (const n of names) await expect(card(n)).toBeVisible({ timeout: 30_000 }); + // Still selecting after the last Stop: no Select to press. + const select = page.getByRole("button", { name: "Select", exact: true }); + if (await select.isVisible()) await select.click(); + for (const n of names) await card(n).getByText(title(n)).click(); + await page.getByRole("button", { name: `Stop ${names.length}…` }).click(); + const archiveBox = dialog.getByRole("checkbox", { name: /Archive their worktrees where it's safe/ }); + if (!(await archiveBox.isChecked())) await archiveBox.click(); + await dialog.getByRole("button", { name: `Stop ${names.length} agents` }).click(); + await expect(dialog).toBeHidden({ timeout: 1000 }); + }; + + step("Stop the five with archiving ticked"); + await pick(LID); + const job = tray.getByRole("group", { name: "Stopping 5 agents" }); + await expect(job.getByRole("listitem").filter({ hasText: "Archiving…" })).toHaveCount(1, { timeout: 30_000 }); + await shot("lid-1-archiving"); + + step("close the lid: pause the laptop agent for 25 s while the box archives"); + const pid = agentPid(); + process.kill(pid, "SIGSTOP"); + const paused = Date.now(); + try { + while (Date.now() - paused < 25_000) await page.waitForTimeout(1000); + step(`while paused, git on the box: ${worktrees().split("\n").filter((l) => l.startsWith("worktree ") && l.includes("verify-lid")).length} verify-lid worktrees left`); + } finally { + process.kill(pid, "SIGCONT"); + } + step("open the lid: the tray catches up with the box"); + await expect(job).toContainText("5 stopped, 3 archived, 2 kept", { timeout: 60_000 }); + await expect(job.getByRole("listitem")).toHaveCount(2); + await expect(job.getByRole("listitem").filter({ hasText: "verify-lid-dirty" })).toContainText("Stopped · Kept: uncommitted changes"); + await expect(job.getByRole("listitem").filter({ hasText: "verify-lid-shared" })).toContainText("Stopped · Kept: another agent is still running there"); + await shot("lid-2-report-after-wake"); + + step("git on the box agrees: a, b and c archived, dirty and shared kept, every branch kept, the outsider still running"); + const after = worktrees(); + for (const n of ["verify-lid-a", "verify-lid-b", "verify-lid-c"]) if (after.includes(paths[n])) throw new Error(`${n}'s worktree is still there`); + for (const n of ["verify-lid-dirty", "verify-lid-shared"]) if (!after.includes(paths[n])) throw new Error(`${n} was archived`); + for (const n of LID) if (!branches().some((b) => b.endsWith(n))) throw new Error(`${n}'s branch is gone`); + const sessions = await call(`${API}/sessions`); + for (const n of LID) if (sessions.some((s) => s.name === `${n}-claude` && !s.exited)) throw new Error(`${n}-claude still runs`); + if (!sessions.some((s) => s.name === "verify-lid-outsider" && !s.exited)) throw new Error("the outsider was stopped"); + step(`git worktree list:\n${after.trim()}`); + const stops = (await call(`${API}/runs?template=stop&limit=5`)).filter((r) => r.title === "Stopping 5 agents"); + if (stops[0]?.status !== "succeeded" && stops[0]?.status !== "failed") throw new Error(`the box's run is ${stops[0]?.status}`); + step(`the box's run ${stops[0].id}: ${stops[0].status}`); + await job.getByRole("button", { name: "Dismiss" }).click(); + + step("Stop two more, and reload the app (a relaunch) while the box archives"); + await pick(RELAUNCH); + await expect(tray.getByRole("group", { name: "Stopping 2 agents" }).getByRole("listitem").filter({ hasText: "Archiving…" })).toHaveCount(1, { timeout: 30_000 }); + await page.reload(); + await expect(page.getByTestId("nav-home")).toBeVisible({ timeout: 30_000 }); + const back = tray.getByRole("group", { name: "Stopping 2 agents" }); + await expect(back).toBeVisible({ timeout: 30_000 }); + await shot("lid-3-reattached-running"); + await expect(back).toContainText("2 stopped, 2 archived", { timeout: 60_000 }); + await shot("lid-4-reattached-done"); + for (const n of RELAUNCH) if (worktrees().includes(paths[n])) throw new Error(`${n}'s worktree is still there`); + step("after the relaunch the tray showed the box's run, and it ended 2 stopped, 2 archived; git agrees"); + await call(`${API}/locations/acme-shop/scripts`, { method: "PUT", body: JSON.stringify({ archive: "" }) }); + + step("the Worktrees table: Stop sessions in verify-lid-shared, then Archive it, each as a stop run on the box"); + const ranOnBox = async (title) => { + let r; + await expect.poll(async () => (r = (await call(`${API}/runs?template=stop&limit=10`)).find((x) => x.title === title))?.status, { timeout: 30_000 }).toBe("succeeded"); + step(`the box's run "${title}": ${r.id} ${r.status}`); + }; + const row = page.getByRole("checkbox", { name: "Select verify-lid-shared" }); + await page.getByTestId("nav-worktrees").click(); + await row.locator("xpath=..").click(); + await page.getByRole("button", { name: "Stop sessions" }).click(); + await ranOnBox("Stop sessions in 1 worktree"); + if ((await call(`${API}/sessions`)).some((s) => s.name === "verify-lid-outsider" && !s.exited)) throw new Error("the outsider still runs"); + await shot("lid-5-table-stopped"); + // Dismissing the outcome keeps the row selected. + await page.getByRole("status").filter({ hasText: "Stop sessions:" }).getByRole("button", { name: "Dismiss" }).click(); + await expect(row).toBeChecked(); + await page.getByRole("button", { name: /^Archive/ }).click(); + await dialog.getByRole("button", { name: "Archive 1" }).click(); + await ranOnBox("Archive 1 worktree"); + await expect(dialog).toContainText("Archived 1 worktree", { timeout: 30_000 }); + await shot("lid-6-table-archived"); + if (worktrees().includes(paths["verify-lid-shared"])) throw new Error("verify-lid-shared is still there"); + if (!branches().some((b) => b.endsWith("verify-lid-shared"))) throw new Error("verify-lid-shared's branch is gone"); + step("git agrees: verify-lid-shared archived, its branch kept"); +} diff --git a/.claude/skills/verify-berthd/scripts/scenarios/dashboard-stop-live.mjs b/.claude/skills/verify-berthd/scripts/scenarios/dashboard-stop-live.mjs new file mode 100644 index 000000000..e24cc2c25 --- /dev/null +++ b/.claude/skills/verify-berthd/scripts/scenarios/dashboard-stop-live.mjs @@ -0,0 +1,99 @@ +// dashboard-stop-live: the board's bulk Stop and the sidebar's Archive… on +// a real box (dev-box.sh's devq), as features/dashboard-stop.md and +// features/worktree-archive.md describe. It makes its own throwaway +// worktrees of acme-shop, each with a Claude Code in it, one of them with +// an uncommitted file; stops the three with archiving ticked, and reads +// the result back from git on the box: two worktrees gone with their +// branches kept, the dirty one still there. Then it archives a fourth from +// the sidebar. The box does both as stop runs, which its runs list shows. +// Nothing it didn't make is touched; dev-box.sh down removes the lot. +import { execFileSync } from "node:child_process"; +import { writeFileSync } from "node:fs"; +import { join } from "node:path"; + +export const live = true; +export const params = { sidebar: "list" }; + +const API = "/v1/boxes/devq/api"; +const STOPPED = ["verify-stop-a", "verify-stop-b", "verify-stop-c"]; +const SOLO = "verify-solo"; + +export async function run({ page, shot, step, expect, call }) { + const post = (path, body) => call(`${API}${path}`, { method: "POST", body: JSON.stringify(body) }); + const tray = page.getByRole("region", { name: "Background jobs" }); + const dialog = page.getByRole("alertdialog"); + const title = (n) => `Throwaway ${n}`; + // The box's stop runs, newest first, by title. + const stopRuns = async () => (await call(`${API}/runs?template=stop&limit=20`)).map((r) => `${r.title}: ${r.status}`); + + step("make four worktrees of acme-shop, a Claude Code in three of them, and an uncommitted file in verify-stop-c"); + const paths = {}; + for (const n of [...STOPPED, SOLO]) paths[n] = (await post("/locations/acme-shop/worktrees", { name: n })).path; + for (const n of STOPPED) await post("/sessions", { name: `${n}-claude`, location: `acme-shop/${n}`, agent: "claude", title: title(n) }); + writeFileSync(join(paths["verify-stop-c"], "notes.txt"), "not committed\n"); + const repo = (await call(`${API}/locations`)).find((l) => l.name === "acme-shop").path; + const git = (...args) => execFileSync("git", ["-C", repo, ...args], { encoding: "utf8" }); + const worktrees = () => git("worktree", "list", "--porcelain"); + const branches = () => git("branch", "--list", "--format=%(refname:short)"); + for (const n of [...STOPPED, SOLO]) if (!worktrees().includes(paths[n])) throw new Error(`git doesn't list ${n}`); + step(`git on the box lists them:\n${worktrees().trim()}`); + + step("open the Agent Dashboard and pick the three throwaway agents"); + await page.keyboard.press("Meta+j"); + await expect(page.getByRole("heading", { name: "Agent Dashboard" })).toBeVisible(); + for (const n of STOPPED) await expect(page.locator("article[role=button]", { hasText: title(n) })).toBeVisible({ timeout: 30_000 }); + await page.getByRole("button", { name: "Select", exact: true }).click(); + for (const n of STOPPED) await page.locator("article[role=button]", { hasText: title(n) }).getByText(title(n)).click(); + await page.getByRole("button", { name: "Stop 3…" }).click(); + const archiveBox = dialog.getByRole("checkbox", { name: /Archive their worktrees where it's safe/ }); + if (!(await archiveBox.isChecked())) await archiveBox.click(); + await expect(archiveBox).toBeChecked(); + await shot("live-1-stop-dialog"); + + step("Stop 3 agents: the dialog goes at once, the tray runs it"); + const t0 = Date.now(); + await dialog.getByRole("button", { name: "Stop 3 agents" }).click(); + await expect(dialog).toBeHidden({ timeout: 1000 }); + step(`the dialog closed ${Date.now() - t0} ms after Stop was pressed`); + const job = tray.getByRole("group", { name: "Stopping 3 agents" }); + await expect(job).toBeVisible(); + await shot("live-2-tray-running"); + + step("the report: two archived, the dirty one kept"); + await expect(job).toContainText("3 stopped, 2 archived, 1 kept", { timeout: 60_000 }); + await expect(job.getByRole("listitem")).toHaveCount(1); + await expect(job.getByRole("listitem")).toContainText("Stopped · Kept: uncommitted changes"); + await shot("live-3-tray-report"); + + step("git on the box agrees: a and b's worktrees are gone, c's stays, every branch is kept"); + const after = worktrees(); + for (const n of ["verify-stop-a", "verify-stop-b"]) if (after.includes(paths[n])) throw new Error(`${n}'s worktree is still there`); + if (!after.includes(paths["verify-stop-c"])) throw new Error("the dirty worktree went"); + const kept = branches(); + for (const n of STOPPED) if (!kept.split("\n").some((b) => b.endsWith(n))) throw new Error(`${n}'s branch is gone`); + step(`git worktree list:\n${after.trim()}\ngit branch:\n${kept.trim()}`); + // The box keeps the dirty one inside its run (a skipped step), so no + // request was refused: the tray's row and git above say it was kept. + const runsNow = await stopRuns(); + if (!runsNow.includes("Stopping 3 agents: succeeded")) throw new Error(`the box has no finished stop run for it: ${runsNow.join(", ")}`); + step(`the box did it as a stop run: ${runsNow[0]}`); + const sessions = await call(`${API}/sessions`); + for (const n of STOPPED) if (sessions.some((s) => s.name === `${n}-claude` && !s.exited)) throw new Error(`${n}-claude still runs`); + step("the box lists none of the three agents"); + await job.getByRole("button", { name: "Dismiss" }).click(); + + step(`Archive… ${SOLO} from the sidebar: the dialog goes at once, a one-row job in the tray`); + await page.locator(`[data-testid=worktree-row][data-worktree="devq/${SOLO}"]`).click({ button: "right" }); + await page.getByRole("menuitem", { name: "Archive…" }).click(); + await dialog.getByRole("button", { name: "Archive", exact: true }).click(); + await expect(dialog).toBeHidden({ timeout: 1000 }); + const solo = tray.getByRole("group", { name: `Archiving ${SOLO}` }); + await expect(solo).toContainText(`Archived ${SOLO}`, { timeout: 30_000 }); + await shot("live-4-solo-archived"); + if (worktrees().includes(paths[SOLO])) throw new Error(`${SOLO}'s worktree is still there`); + if (!branches().split("\n").some((b) => b.endsWith(SOLO))) throw new Error(`${SOLO}'s branch is gone`); + step(`git agrees: ${SOLO}'s worktree is gone, its branch kept`); + const solos = await stopRuns(); + if (!solos.includes(`Archive ${SOLO}: succeeded`)) throw new Error(`the box has no finished stop run for ${SOLO}: ${solos.join(", ")}`); + step(`the box did it as a stop run too: ${solos[0]}`); +} diff --git a/.claude/skills/verify-berthd/scripts/scenarios/dashboard-stop-unanswered.mjs b/.claude/skills/verify-berthd/scripts/scenarios/dashboard-stop-unanswered.mjs new file mode 100644 index 000000000..32bdd8bb4 --- /dev/null +++ b/.claude/skills/verify-berthd/scripts/scenarios/dashboard-stop-unanswered.mjs @@ -0,0 +1,95 @@ +// dashboard-stop-unanswered: a bulk Stop whose box goes away as Stop is +// pressed, on a real box (dev-box.sh's devq), as features/dashboard-stop.md +// describes. It makes two throwaway worktrees of acme-shop with a Claude +// Code in each, kills the box's berthd (SIGKILL, a crash) as Stop is +// pressed, and starts it again 20 s later, as dev-box.sh does. Meanwhile the +// rows wait for the box rather than fail. Once it's back, the start is asked +// again under the same key, so the box ends with one stop run (whether or +// not it took the first ask), and the job ends as that run did; git on the +// box agrees. Nothing it didn't make is touched; dev-box.sh down removes the +// lot. +import { execFileSync, spawn } from "node:child_process"; +import { openSync, readFileSync, writeFileSync } from "node:fs"; +import { dirname, join } from "node:path"; +import { fileURLToPath } from "node:url"; + +export const live = true; +export const params = { sidebar: "list" }; +// While the box is paused, the app's requests to it fail. +export const allowErrors = [/Failed to load resource/, /net::ERR_/]; + +const API = "/v1/boxes/devq/api"; +// Unique per run, as a branch outlives its archived worktree. +const TAG = Date.now().toString(36).slice(-4); +const NAMES = [`verify-mute-${TAG}-a`, `verify-mute-${TAG}-b`]; +const DIR = process.env.VERIFY_BOX_DIR ?? "/tmp/verify-berthd-box"; +const ROOT = execFileSync("git", ["-C", dirname(fileURLToPath(import.meta.url)), "rev-parse", "--show-toplevel"], { encoding: "utf8" }).trim(); + +// startBox starts the dev box's berthd again, as dev-box.sh up does. +function startBox() { + const env = Object.fromEntries(Object.entries(process.env).filter(([k]) => !k.startsWith("BERTH_"))); + const log = openSync(join(DIR, "berthd.log"), "a"); + const p = spawn(join(ROOT, "bin/berthd"), ["serve", "--listen", "127.0.0.1:7544"], { env: { ...env, BERTH_HOME: `${DIR}/box-home`, BERTH_USER_DIR: `${DIR}/box-user`, TMUX_TMPDIR: `${DIR}/tmux` }, detached: true, stdio: ["ignore", log, log] }); + p.unref(); + writeFileSync(join(DIR, "berthd.pid"), `${p.pid}\n`); +} + +export async function run({ page, shot, step, expect, call }) { + const post = (path, body) => call(`${API}${path}`, { method: "POST", body: JSON.stringify(body) }); + const tray = page.getByRole("region", { name: "Background jobs" }); + const dialog = page.getByRole("alertdialog"); + const title = (n) => `Throwaway ${n}`; + const card = (n) => page.locator("article[role=button]", { hasText: title(n) }); + const repo = (await call(`${API}/locations`)).find((l) => l.name === "acme-shop").path; + const git = (...args) => execFileSync("git", ["-C", repo, ...args], { encoding: "utf8" }); + const paths = {}; + + step("make two worktrees of acme-shop, a Claude Code in each"); + for (const n of NAMES) paths[n] = (await post("/locations/acme-shop/worktrees", { name: n })).path; + for (const n of NAMES) await post("/sessions", { name: `${n}-claude`, location: `acme-shop/${n}`, agent: "claude", title: title(n) }); + await call(`${API}/locations/acme-shop/scripts`, { method: "PUT", body: JSON.stringify({ archive: "" }) }); + + step("pick them on the Agent Dashboard and tick archiving"); + await page.keyboard.press("Meta+j"); + await expect(page.getByRole("heading", { name: "Agent Dashboard" })).toBeVisible(); + for (const n of NAMES) await expect(card(n)).toBeVisible({ timeout: 30_000 }); + const select = page.getByRole("button", { name: "Select", exact: true }); + if (await select.isVisible()) await select.click(); + for (const n of NAMES) await card(n).getByText(title(n)).click(); + await page.getByRole("button", { name: `Stop ${NAMES.length}…` }).click(); + const archiveBox = dialog.getByRole("checkbox", { name: /Archive their worktrees where it's safe/ }); + if (!(await archiveBox.isChecked())) await archiveBox.click(); + + step("press Stop and kill the box's berthd at once: its answer, if any, is lost"); + const pid = Number(readFileSync(`${DIR}/berthd.pid`, "utf8").trim()); + const job = tray.getByRole("group", { name: "Stopping 2 agents" }); + const pressed = Date.now(); + await dialog.getByRole("button", { name: "Stop 2 agents" }).click(); + process.kill(pid, "SIGKILL"); + const down = Date.now(); + try { + await expect(dialog).toBeHidden({ timeout: 1000 }); + await page.waitForTimeout(8000); + await shot("mute-1-box-down"); + const rows = (await job.getByRole("listitem").allInnerTexts()).map((t) => t.replace(/\s+/g, " ")); + step(`with the box down the rows say: ${rows.join(" | ")}`); + while (Date.now() - down < 20_000) await page.waitForTimeout(1000); + const failed = await job.getByRole("listitem").filter({ hasText: "Failed" }).count(); + if (failed) throw new Error(`${failed} rows failed while the box was down`); + step(`after ${Math.round((Date.now() - down) / 1000)} s down, no row failed`); + } finally { + startBox(); + } + + step("the box is back: the job ends as its run did"); + await expect(job).toContainText("2 stopped, 2 archived", { timeout: 90_000 }); + await shot("mute-2-done"); + const runs = (await call(`${API}/runs?template=stop&limit=20`)).filter((r) => r.title === "Stopping 2 agents" && Date.parse(r.created) >= pressed - 2000); + if (runs.length !== 1) throw new Error(`the box has ${runs.length} runs for it, not one: ${runs.map((r) => r.id).join(", ")}`); + step(`one stop run on the box for it (group ${runs[0].group}): ${runs[0].id} ${runs[0].status}`); + const after = git("worktree", "list", "--porcelain"); + for (const n of NAMES) if (after.includes(paths[n])) throw new Error(`${n}'s worktree is still there`); + const branches = git("branch", "--list", "--format=%(refname:short)").split("\n"); + for (const n of NAMES) if (!branches.includes(n)) throw new Error(`${n}'s branch is gone`); + step("git on the box agrees: both worktrees gone, both branches kept"); +} diff --git a/.claude/skills/verify-berthd/scripts/scenarios/worktree-archive.mjs b/.claude/skills/verify-berthd/scripts/scenarios/worktree-archive.mjs new file mode 100644 index 000000000..bd9493cd5 --- /dev/null +++ b/.claude/skills/verify-berthd/scripts/scenarios/worktree-archive.mjs @@ -0,0 +1,95 @@ +// Archive… from a worktree's menu: the dialog closes as soon as Archive is +// pressed, the row says "Archiving…" until its box is done, it is a +// one-row job in the background-jobs tray, and a toast says how it went. +// The mock's devl and gpu have runs, so each archive is a stop run there, +// in the box's own line. The mock's box takes 4 s to remove a worktree with +// no archive script (?slowremove), as one with a big node_modules does. +// features/worktree-archive.md. + +export const params = { slowremove: "4000", sidebar: "list" }; + +const SLOW = 4000; + +export async function run({ page, shot, step, expect }) { + // A toast by its title: role dialog, unlike the tray's jobs (groups), so a + // second toast for the same archive fails strict mode. + const toast = (title) => page.getByRole("dialog", { name: title }); + const leaving = (name) => page.locator("[data-leaving]", { hasText: name }); + const job = (name) => page.getByRole("region", { name: "Background jobs" }).getByRole("group", { name: `Archiving ${name}` }); + const archive = async (w) => { + await page.locator(`[data-testid=worktree-row][data-worktree="${w}"]`).click({ button: "right" }); + await page.getByRole("menuitem", { name: "Archive…" }).click(); + const dialog = page.getByRole("alertdialog"); + await expect(dialog).toBeVisible(); + return dialog; + }; + // pressArchive confirms and says how long the dialog stayed open. The + // shot 300 ms in shows it spinning if it waits on the box. + const pressArchive = async (dialog, label) => { + const t0 = Date.now(); + await dialog.getByRole("button", { name: "Archive", exact: true }).click(); + const closed = dialog.waitFor({ state: "hidden", timeout: SLOW * 3 }).then(() => Date.now() - t0); + await page.waitForTimeout(300); + await shot(label); + const ms = await closed; + step(`the dialog closed ${ms} ms after Archive was pressed`); + if (ms > 1000) throw new Error(`the dialog stayed open ${ms} ms, waiting on the box`); + return t0; + }; + + step("gpu/ci-flake (no archive script): Archive… asks first"); + const dialog = await archive("gpu/ci-flake"); + await expect(dialog).toContainText("Archive ci-flake?"); + await expect(dialog).toContainText("Branch me/ci-flake stays"); + await shot("1-confirm"); + + step("press Archive: the dialog goes at once and the row says Archiving…"); + const t0 = await pressArchive(dialog, "2-just-after-archive"); + await expect(leaving("ci-flake")).toContainText("Archiving…"); + await expect(job("ci-flake").getByRole("listitem")).toContainText("Archiving…"); + + step("archive gpu/judge-v2 straight after: it waits its turn in gpu's line, marked too"); + await pressArchive(await archive("gpu/judge-v2"), "3-second-archive"); + await expect(leaving("ci-flake")).toContainText("Archiving…"); + await expect(leaving("judge-v2")).toContainText("Archiving…"); + step("the tray has both, each a run gpu has taken"); + await expect(job("judge-v2").getByRole("listitem")).toContainText("Archiving…"); + await shot("4-both-archiving"); + + step("when gpu answers, ci-flake goes and says its branch is kept"); + await expect(toast("Archived ci-flake")).toContainText("Branch me/ci-flake is kept.", { timeout: SLOW * 2 }); + step(`Archived ci-flake toast ${Date.now() - t0} ms after Archive`); + await expect(page.locator('[data-worktree="gpu/ci-flake"]')).toHaveCount(0); + await expect(leaving("ci-flake")).toHaveCount(0); + await expect(job("ci-flake")).toContainText("Archived ci-flake"); + await expect(leaving("judge-v2")).toContainText("Archiving…"); + await shot("5-first-archived"); + + step("judge-v2 goes after it, one at a time per box"); + await expect(toast("Archived judge-v2")).toContainText("Branch judge-v2 is kept.", { timeout: SLOW * 2 }); + step(`Archived judge-v2 toast ${Date.now() - t0} ms after the first Archive`); + await expect(page.locator('[data-worktree="gpu/judge-v2"]')).toHaveCount(0); + await shot("6-both-archived"); + + step("devl/qa-deck (archive script pnpm db:drop): the box's run waits for the script, and the tray with it"); + const scripted = await archive("devl/qa-deck"); + await expect(scripted).toContainText("pnpm db:drop"); + await pressArchive(scripted, "7-script-just-after-archive"); + await expect(job("qa-deck").getByRole("listitem")).toContainText("Archiving…"); + await shot("8-script-running"); + await expect(toast("Archived qa-deck")).toBeVisible({ timeout: 10_000 }); + await expect(job("qa-deck")).toContainText("Archived qa-deck"); + await expect(page.locator('[data-worktree="devl/qa-deck"]')).toHaveCount(0); + + step("devl/checkout-fix (uncommitted work): kept, and a warning says why"); + await pressArchive(await archive("devl/checkout-fix"), "9-dirty-just-after-archive"); + await expect(toast("Couldn't archive checkout-fix")).toContainText("checkout-fix has uncommitted changes, so it was left as it is. Commit them first, or use Remove worktree to discard them."); + await expect(page.locator('[data-testid=worktree-row][data-worktree="devl/checkout-fix"]')).toBeVisible(); + await expect(leaving("checkout-fix")).toHaveCount(0); + step("the tray keeps a report of it until it's dismissed"); + await expect(job("checkout-fix")).toContainText("Kept checkout-fix"); + await expect(job("checkout-fix").getByRole("listitem")).toContainText("Kept: uncommitted changes"); + await shot("10-dirty-kept"); + await job("checkout-fix").getByRole("button", { name: "Dismiss" }).click(); + await expect(job("checkout-fix")).toHaveCount(0); +} diff --git a/app/e2e/dashboard-stop-archive.spec.ts b/app/e2e/dashboard-stop-archive.spec.ts index 25358670b..2acc102e1 100644 --- a/app/e2e/dashboard-stop-archive.spec.ts +++ b/app/e2e/dashboard-stop-archive.spec.ts @@ -1,9 +1,14 @@ import { expect, type App, mockOnly, test } from "./fixtures"; -// The board's bulk Stop can archive the stopped agents' worktrees too, where -// it's safe. In the fixtures shop on devl has an archive script, so an -// archive runs it first; checkout-fix has uncommitted work, so git keeps it; -// order-export has three agents, so stopping one leaves the others there. +// The board's bulk Stop asks, then carries on in the background-jobs tray, +// its dialog gone at once. It can archive the stopped agents' worktrees +// too, where it's safe. Each box with runs does its share itself, as a stop +// run that waits for an archive script to finish; with ?noruns the boxes +// are older ones and the app does it all, as it used to. In the fixtures +// shop on devl has an archive script, so an archive runs it first; +// ci-flake on gpu has none, so it goes at once (slowly, with ?slowremove); +// checkout-fix has uncommitted work, so git keeps it; order-export has +// three agents, so stopping one leaves the others there. const prefs = { labs: false, labsChosen: true, version: 3 }; @@ -20,51 +25,118 @@ async function stopPicked(app: App, titles: string[]) { } const archiveBox = (app: App) => app.page.getByRole("checkbox", { name: /Archive their worktrees where it's safe/ }); -const row = (app: App, place: string) => app.page.getByRole("group", { name: "Agents to stop" }).getByRole("listitem").filter({ hasText: place }); +const tray = (app: App) => app.page.getByRole("region", { name: "Background jobs" }); +const job = (app: App, title: string) => tray(app).getByRole("group", { name: title }); +const jobRow = (app: App, title: string, label: string) => job(app, title).getByRole("listitem").filter({ hasText: label }); +const toast = (app: App, text: string) => app.page.locator("[data-slot=toast-viewport] > *", { hasText: text }); -test("ticked, Stop archives a clean worktree and keeps a dirty one, saying why", async ({ app }) => { +test("ticked, Stop closes at once; the tray shows the run, folds, and reports the kept worktree", async ({ app }) => { mockOnly("stops agents and archives worktrees"); - await app.open({ prefs }); - const dialog = await stopPicked(app, ["Speed up product search", "Fix checkout webhook retries"]); + // gpu takes 3 s to remove a worktree, as one with a big node_modules does. + await app.open({ prefs, params: { slowremove: "3000" } }); + const dialog = await stopPicked(app, ["Speed up product search", "Move acme billing to the new ledger", "Fix checkout webhook retries"]); await expect(archiveBox(app)).not.toBeChecked(); await archiveBox(app).click(); await expect(dialog).toContainText("then its worktree is archived where it's safe"); - await dialog.getByRole("button", { name: "Stop 2 agents" }).click(); + await dialog.getByRole("button", { name: "Stop 3 agents" }).click(); + await expect(dialog).toBeHidden({ timeout: 1000 }); + + // Mid-run: the title counts, and the slow one says it is archiving. + const running = job(app, "Stopping 3 agents"); + await expect(running).toContainText(/Stopping 3 agents · [0-2]\/3/); + await expect(jobRow(app, "Stopping 3 agents", "ci-flake")).toContainText("Archiving…"); + + // Folded, it is one line; the choice is remembered. + await running.getByRole("button", { name: "Fold" }).click(); + const pill = tray(app).getByRole("button", { expanded: false }); + await expect(pill).toContainText(/Stopping 3 agents · [0-2]\/3/); + await expect.poll(() => app.stored("berth.jobs.folded")).toBe(true); + await pill.click(); - // Kept work holds the dialog open, with each row's outcome. - await expect(dialog.getByRole("heading")).toHaveText("Stopped 2 agents · archiving 1 worktree"); - await expect(row(app, "search-perf")).toContainText("Stopped · Archiving: its archive script runs, then it goes"); - await expect(row(app, "checkout-fix")).toContainText("Stopped · Kept: uncommitted changes"); - await dialog.getByRole("button", { name: "Done" }).click(); + // Kept work makes a report that lists only it, until it's dismissed. The + // box waited for search-perf's archive script: archived, not archiving. + await expect(running).toContainText("3 stopped, 2 archived, 1 kept", { timeout: 10_000 }); + await expect(running.getByRole("listitem")).toHaveCount(1); + await expect(jobRow(app, "Stopping 3 agents", "checkout-fix")).toContainText("Stopped · Kept: uncommitted changes"); + await running.getByRole("button", { name: "Dismiss" }).click(); + await expect(tray(app)).toHaveCount(0); - // The clean one goes once its script has run; the dirty one stays. + // The clean ones go (search-perf once its script has run); the dirty one stays. + await expect(app.worktree("gpu/ci-flake")).toHaveCount(0, { timeout: 10_000 }); await expect(app.worktree("devl/search-perf")).toHaveCount(0, { timeout: 10_000 }); await expect(app.worktree("devl/checkout-fix")).toBeVisible(); }); -test("a worktree where another agent still runs is kept", async ({ app }) => { +test("on boxes without runs the app does it: an archive script says archiving until the box says it's gone", async ({ app }) => { + mockOnly("stops agents and archives worktrees"); + await app.open({ prefs, params: { noruns: "1", slowremove: "3000" } }); + const dialog = await stopPicked(app, ["Speed up product search", "Move acme billing to the new ledger", "Fix checkout webhook retries"]); + await archiveBox(app).click(); + await dialog.getByRole("button", { name: "Stop 3 agents" }).click(); + const running = job(app, "Stopping 3 agents"); + await expect(jobRow(app, "Stopping 3 agents", "ci-flake")).toContainText("Archiving…"); + // search-perf's archive script runs on: its row says so, and the job + // holds devl's line until the box says the worktree went. + await expect(jobRow(app, "Stopping 3 agents", "search-perf")).toContainText("Stopped · Archiving: its archive script runs, then it goes", { timeout: 10_000 }); + await expect(running).toContainText("3 stopped, 2 archived, 1 kept", { timeout: 10_000 }); + await expect(jobRow(app, "Stopping 3 agents", "checkout-fix")).toContainText("Stopped · Kept: uncommitted changes"); + await expect(app.worktree("gpu/ci-flake")).toHaveCount(0, { timeout: 10_000 }); + await expect(app.worktree("devl/search-perf")).toHaveCount(0, { timeout: 10_000 }); +}); + +test("a worktree where another agent still runs is kept, and the report says so", async ({ app }) => { mockOnly("stops agents and archives worktrees"); await app.open({ prefs }); const dialog = await stopPicked(app, ["Export orders as CSV from the admin"]); await archiveBox(app).click(); await dialog.getByRole("button", { name: "Stop 1 agent" }).click(); - await expect(dialog.getByRole("heading")).toHaveText("Stopped 1 agent"); - await expect(row(app, "order-export")).toContainText("Kept: another agent is still running there"); - await dialog.getByRole("button", { name: "Done" }).click(); + await expect(dialog).toBeHidden({ timeout: 1000 }); + await expect(job(app, "Stopping 1 agent")).toContainText("1 stopped, 1 kept"); + await expect(jobRow(app, "Stopping 1 agent", "order-export")).toContainText("Kept: another agent is still running there"); await expect(app.worktree("devl/order-export")).toBeVisible(); }); -test("unticked, Stop only stops: the worktree stays", async ({ app }) => { +test("unticked, a clean run only stops, says so in a toast, and fades from the tray", async ({ app }) => { mockOnly("stops agents"); await app.open({ prefs }); const dialog = await stopPicked(app, ["Speed up product search"]); await expect(dialog).toContainText("its worktree and files are kept"); await dialog.getByRole("button", { name: "Stop 1 agent" }).click(); await expect(dialog).toHaveCount(0); - await expect(app.page.locator("[data-slot=toast-viewport] > *", { hasText: "Stopped 1 agent" })).toContainText("Their worktrees are kept."); + await expect(toast(app, "Stopped 1 agent")).toContainText("Their worktrees are kept."); + await expect(job(app, "Stopping 1 agent")).toContainText("1 stopped"); + await expect(job(app, "Stopping 1 agent").getByRole("listitem")).toHaveCount(0); + await expect(tray(app)).toHaveCount(0, { timeout: 15_000 }); await expect(app.worktree("devl/search-perf")).toBeVisible(); }); +test("Stop the rest sends no more; the ones already asked stop, the rest say Not stopped", async ({ app }) => { + mockOnly("stops agents"); + // Each stop takes 2 s to answer, so devl's run has three in flight and + // two waiting. + await app.open({ prefs, params: { slowstop: "2000" } }); + const dialog = await stopPicked(app, ["Speed up product search", "Build the QA deck for the release", "Export orders as CSV from the admin", "Add tests for the export job", "Fix the cart badge after a refund"]); + await dialog.getByRole("button", { name: "Stop 5 agents" }).click(); + const running = job(app, "Stopping 5 agents"); + await expect(running.getByRole("listitem").filter({ hasText: "Stopping…" })).toHaveCount(3); + await running.getByRole("button", { name: "Stop the rest" }).click(); + await expect(running).toContainText("3 stopped, 2 not stopped", { timeout: 10_000 }); + await expect(running.getByRole("listitem").filter({ hasText: "Not stopped" })).toHaveCount(2); +}); + +test("the job carries on when you leave the board", async ({ app }) => { + mockOnly("stops agents and archives worktrees"); + await app.open({ prefs, params: { slowremove: "2000" } }); + const dialog = await stopPicked(app, ["Move acme billing to the new ledger"]); + await archiveBox(app).click(); + await dialog.getByRole("button", { name: "Stop 1 agent" }).click(); + await expect(jobRow(app, "Stopping 1 agent", "ci-flake")).toContainText("Archiving…"); + await app.page.getByTestId("nav-worktrees").click(); + await expect(app.page.getByRole("heading", { name: "Agent Dashboard" })).toHaveCount(0); + await expect(job(app, "Stopping 1 agent")).toContainText("1 stopped, 1 archived", { timeout: 10_000 }); + await expect(toast(app, "Stopped 1 agent · archived 1 worktree")).toContainText("Their branches are kept."); +}); + test("the archive choice is remembered, and not offered for the main checkout alone", async ({ app }) => { mockOnly("stops agents"); await app.open({ prefs }); @@ -86,3 +158,30 @@ test("the archive choice is remembered, and not offered for the main checkout al await expect(app.page.getByRole("alertdialog")).toContainText("its worktree and files are kept"); await expect(archiveBox(app)).toHaveCount(0); }); + +test("the Worktrees table's Stop sessions and Archive go through the box's stop run", async ({ app }) => { + mockOnly("stops sessions and archives worktrees"); + await app.open({ prefs }); + const { page } = app; + await page.getByTestId("nav-worktrees").click(); + const select = (name: string) => page.getByRole("checkbox", { name: `Select ${name}` }).locator("xpath=..").click(); + await select("order-export"); + await page.getByRole("button", { name: "Stop sessions" }).click(); + const bar = page.getByRole("status").filter({ hasText: "Stop sessions:" }); + await expect(bar).toContainText("Stop sessions: 1 done", { timeout: 10_000 }); + await bar.getByRole("button", { name: "Dismiss" }).click(); + await select("order-export"); + await select("search-perf"); + await select("checkout-fix"); + // devl's loop panel sits over the bar's right end: by keyboard, as a person can. + await page.getByRole("button", { name: "Archive…" }).press("Enter"); + const dialog = page.getByRole("alertdialog"); + await dialog.getByRole("button", { name: "Archive 2" }).click(); + // search-perf goes once shop's archive script ran; checkout-fix keeps its work. + await expect(dialog).toContainText("Archived 1 of 2", { timeout: 15_000 }); + await expect(dialog).toContainText("Has uncommitted changes"); + await page.keyboard.press("Escape"); + await expect(dialog).toHaveCount(0); + await expect(page.getByRole("checkbox", { name: "Select search-perf" })).toHaveCount(0, { timeout: 10_000 }); + await expect(page.getByRole("checkbox", { name: "Select checkout-fix" })).toBeVisible(); +}); diff --git a/app/e2e/worktree-remove.spec.ts b/app/e2e/worktree-remove.spec.ts index 3056b7cbc..27b457a1b 100644 --- a/app/e2e/worktree-remove.spec.ts +++ b/app/e2e/worktree-remove.spec.ts @@ -2,7 +2,8 @@ import { expect, mockOnly, test } from "./fixtures"; // Removing a worktree asks nothing: it shows "Removing…" and goes. One with // uncommitted work (checkout-fix, in the fixtures) is kept, and the toast -// that says so offers to remove it anyway. +// that says so offers to remove it anyway. Archive asks, then carries on +// the same way, its dialog gone at once. test("Remove worktree removes it without a dialog, and a dirty one only when told to", async ({ app }) => { mockOnly(); @@ -28,6 +29,41 @@ test("Remove worktree removes it without a dialog, and a dirty one only when tol await expect(dirty).toHaveCount(0, { timeout: 10_000 }); }); +test("Archive closes its dialog at once and the worktree goes when its box is done", async ({ app }) => { + mockOnly(); + // A box takes 4 s to remove a worktree, as one with a big node_modules does. + await app.open({ params: { slowremove: "4000" } }); + const page = app.page; + const toast = (text: string) => page.locator("[data-slot=toast-viewport] > *", { hasText: text }); + const dialog = page.getByRole("alertdialog"); + + const row = app.worktree("gpu/ci-flake"); + await row.click({ button: "right" }); + await page.getByRole("menuitem", { name: "Archive…" }).click(); + await dialog.getByRole("button", { name: "Archive", exact: true }).click(); + await expect(dialog).toBeHidden({ timeout: 1000 }); + await expect(page.locator("[data-leaving]", { hasText: "ci-flake" })).toContainText("Archiving…"); + // It is a one-row job in the tray, which sums it up when it's done. + const job = page.getByRole("region", { name: "Background jobs" }).getByRole("group", { name: "Archiving ci-flake" }); + await expect(job.getByRole("listitem")).toContainText("Archiving…"); + await expect(toast("Archived ci-flake")).toContainText("Branch me/ci-flake is kept.", { timeout: 10_000 }); + await expect(job).toContainText("Archived ci-flake"); + await expect(row).toHaveCount(0); + + // Uncommitted work: kept, and the toast says why. + const dirty = app.worktree("devl/checkout-fix"); + await dirty.click({ button: "right" }); + await page.getByRole("menuitem", { name: "Archive…" }).click(); + await dialog.getByRole("button", { name: "Archive", exact: true }).click(); + await expect(dialog).toBeHidden({ timeout: 1000 }); + await expect(toast("Couldn't archive checkout-fix")).toContainText("Commit them first, or use Remove worktree to discard them."); + await expect(dirty).toBeVisible(); + // The tray keeps a report of it until it's dismissed. + const kept = page.getByRole("region", { name: "Background jobs" }).getByRole("group", { name: "Archiving checkout-fix" }); + await expect(kept).toContainText("Kept checkout-fix"); + await expect(kept.getByRole("listitem")).toContainText("Kept: uncommitted changes"); +}); + test("Delete in the Worktrees table removes the selected ones without a dialog", async ({ app }) => { mockOnly(); await app.open(); diff --git a/app/src/App.tsx b/app/src/App.tsx index 3e98633ff..a84125990 100644 --- a/app/src/App.tsx +++ b/app/src/App.tsx @@ -11,6 +11,7 @@ import { TreeDockFrame } from "@/components/files/tree-dock"; import { WorktreePicker } from "@/components/workspace/worktree-picker"; import { ComposerDialog } from "@/components/conversation/composer-dialog"; import { NotificationCenter } from "@/components/notifications/notification-center"; +import { JobsTray } from "@/components/jobs-tray"; import { LoopsPanel } from "@/components/orchestrate/loops-panel"; import { PluginConsentDialog } from "@/components/plugin-consent-dialog"; import { PromptDialogs } from "@/components/prompts"; @@ -42,6 +43,8 @@ import { startOutdatedWatch } from "@/lib/outdated"; import { startBoxDefaultsSync } from "@/lib/box-defaults"; import { startSkillsNoticeWatch } from "@/lib/skills-notice"; import { startRunsWatch } from "@/lib/runs"; +import { reattachStopJobs } from "@/lib/stop-job"; +import { stopRuns } from "@/lib/stop-on-box"; import { watchStillness } from "@/lib/still"; import { watchKeyboardLayout } from "@/lib/keyboard-layout"; import { useStore } from "@/lib/store"; @@ -99,6 +102,9 @@ export default function App() { // panel, Automations and Review (lib/runs.ts). const connectedToAgent = useStore((s) => !!s.client); useEffect(() => (connectedToAgent ? startRunsWatch() : undefined), [connectedToAgent]); + // Stop jobs the boxes carried on with while the app was closed come back + // to the tray (lib/stop-job.ts). + useEffect(() => void (connectedToAgent && reattachStopJobs({ runs: stopRuns })), [connectedToAgent]); // Which boxes run an older berthd (lib/outdated.ts). useEffect(() => (connectedToAgent ? startOutdatedWatch() : undefined), [connectedToAgent]); // Agents started without the app (another agent, the CLI, a run) follow @@ -171,14 +177,15 @@ export default function App() { return ( {/* Toasts sit bottom-right, as in other desktop tools: above the status - bar (26px) and above the loop panel when there is one, which says - how tall it is in --berth-loops-h. Top-right covered the headers of + bar (26px) and above the loop panel and the background-jobs tray + when they're up, which say how tall they are in --berth-loops-h + and --berth-jobs-h. Top-right covered the headers of pages (Review's, say). A bar floating at a page's bottom (the Dashboard's selection, the Worktrees bulk bar) lifts them above it through --berth-bar-lift (hooks/lift-toasts.ts). With a dialog or sheet open they move to a corner clear of it; see components/ui/toast.tsx. The stack is as wide as the loops panel. */} - +
{fakeTrafficLights() && }
@@ -233,9 +240,11 @@ export default function App() { - {/* Loops float over the workspace, not over Settings' page. */} + {/* Loops and background jobs float over the workspace, not over + Settings' page. */}
+
diff --git a/app/src/components/jobs-tray.tsx b/app/src/components/jobs-tray.tsx new file mode 100644 index 000000000..0e84fbc73 --- /dev/null +++ b/app/src/components/jobs-tray.tsx @@ -0,0 +1,195 @@ +import { AlertTriangleIcon, CheckIcon, ChevronDownIcon, ChevronUpIcon, CircleStopIcon, MinusIcon, XIcon } from "lucide-react"; +import { memo, type ReactNode, useEffect, useLayoutEffect, useRef, useState } from "react"; + +import { Tip } from "@/components/tip"; +import { Button } from "@/components/ui/button"; +import { Spinner } from "@/components/ui/spinner"; +import { cancelJob, dismissJob, FADE_MS, type Job, type JobItem, leftOver, outcomeOf, progressOf, setFolded, settled, useJobs } from "@/lib/jobs"; +import { cn } from "@/lib/utils"; + +// The background-jobs tray (lib/jobs.ts): bottom right, above the status +// bar (26px) and the loops panel (--berth-loops-h), and above a bar floating +// at a page's bottom (--berth-bar-lift, hooks/lift-toasts.ts) so it never +// hides the bar's buttons. Toasts stack above it through --berth-jobs-h. Jobs stack newest on top. Folded, it is +// one line with the progress. With no jobs it is gone and reserves no room. +const STATUS_BAR = 26; +const GAP = 12; +const FADE = 700; + +export function JobsTray() { + const jobs = useJobs((s) => s.jobs); + const folded = useJobs((s) => s.folded); + const ref = useRef(null); + + useLayoutEffect(() => { + const root = document.documentElement; + const el = ref.current; + if (!el) { + root.style.setProperty("--berth-jobs-h", "0px"); + return; + } + // Hidden (Settings), it takes no room. + const sync = () => root.style.setProperty("--berth-jobs-h", `${el.offsetHeight ? el.offsetHeight + GAP : 0}px`); + sync(); + const ro = new ResizeObserver(sync); + ro.observe(el); + return () => { + ro.disconnect(); + root.style.setProperty("--berth-jobs-h", "0px"); + }; + }, [jobs.length, folded]); + + if (!jobs.length) return null; + const style = { bottom: `max(calc(var(--berth-status-h, ${STATUS_BAR}px) + ${GAP}px + var(--berth-loops-h, 0px)), var(--berth-bar-lift, 0px))`, right: `calc(${GAP}px + var(--berth-dock-w, 0px))` }; + if (folded) { + const outcomes = jobs.map(outcomeOf); + const all = jobs.flatMap((j) => j.items); + const label = jobs.length === 1 ? lineOf(jobs[0]) : `${jobs.length} jobs${outcomes.includes("running") ? ` · ${all.filter(settled).length}/${all.length}` : ""}`; + return ( +
+ +
+ ); + } + return ( +
+ {jobs.map((j) => ( + + ))} +
+ ); +} + +// lineOf is a job in one line: what it is doing and how far it got (a +// one-row job just says what it is doing), or, once done, what it did. +function lineOf(j: Job) { + if (j.summary !== undefined) return j.summary; + const p = progressOf(j); + return p.total > 1 ? `${j.title} · ${p.settled}/${p.total}` : j.title; +} + +function OutcomeIcon({ outcome }: { outcome: ReturnType }) { + if (outcome === "running") return ; + if (outcome === "report") return ; + return ; +} + +const JobCard = memo(function JobCard({ job: j }: { job: Job }) { + const outcome = outcomeOf(j); + const fading = useFading(j, outcome === "clean"); + // Running, every row; a report, only what didn't go as asked; clean, + // just the summary. + const rows = outcome === "running" ? j.items : outcome === "report" ? leftOver(j) : []; + const groups = [...new Set(rows.map((i) => i.group))].sort(); + const unsent = j.items.some((i) => i.state === "queued"); + return ( +
+
+ + + {lineOf(j)} + + + + + {outcome !== "running" && ( + + + + )} +
+ {rows.length > 0 && ( +
+ {groups.map((g) => { + const group = rows.filter((i) => i.group === g); + const all = j.items.filter((i) => i.group === g); + return ( +
+ {groups.length > 1 || (outcome === "running" && j.items.length > 1) ? ( +
+ {g} + {outcome === "running" && ( + + {all.filter(settled).length}/{all.length} + + )} +
+ ) : null} +
    + {group.map((i) => ( + + ))} +
+
+ ); + })} +
+ )} + {outcome === "running" && j.cancellable && unsent && ( +
+ +
+ )} +
+ ); +}); + +// useFading starts a clean job's fade just before the store lets it go. +function useFading(j: Job, clean: boolean) { + const [fading, setFading] = useState(false); + useEffect(() => { + if (!clean || j.finished === undefined) return; + const t = setTimeout(() => setFading(true), Math.max(0, j.finished + FADE_MS - FADE - Date.now())); + return () => clearTimeout(t); + }, [clean, j.finished]); + return fading; +} + +// How a row looks in each state: its icon, and the colour of its message. +const looks: Record = { + queued: { icon: , tone: "text-muted-foreground" }, + running: { icon: , tone: "text-muted-foreground" }, + done: { icon: , tone: "text-muted-foreground" }, + kept: { icon: , tone: "text-warning-foreground" }, + cancelled: { icon: , tone: "text-warning-foreground" }, + failed: { icon: , tone: "text-destructive-foreground" }, +}; + +const JobRow = memo(function JobRow({ item: i }: { item: JobItem }) { + const look = looks[i.state]; + return ( +
  • + {look.icon} + + + {i.label} + {i.aside && {i.aside}} + + {i.message && ( + {i.message} + )} + +
    • + ); +}); diff --git a/app/src/components/sidebar/actions.tsx b/app/src/components/sidebar/actions.tsx index 8f9c4af1f..54b0b45bb 100644 --- a/app/src/components/sidebar/actions.tsx +++ b/app/src/components/sidebar/actions.tsx @@ -23,10 +23,11 @@ import { portUrl } from "@/lib/browser-url"; import { agentOf, worktreeSessions } from "@/lib/derive"; import { ago, errorMessage } from "@/lib/format"; import { plainError } from "@/lib/errors"; +import { stopOnBox } from "@/lib/stop-on-box"; import { scheduleRefresh, useStore } from "@/lib/store"; import { cn } from "@/lib/utils"; import { isPinned, pinWorktree, unpinWorktree } from "@/lib/pinned-worktrees"; -import { queueRemoval, removeWorktreeOnBox } from "@/lib/remove-worktree"; +import { queueRemoval } from "@/lib/remove-worktree"; import { addGroup, isShown, refOf, selectWorktree, useWorkspaces, wsKey } from "@/lib/workspaces"; import { ToneItems } from "@/components/workspace/tab-group"; import { openWorktreePicker } from "@/components/workspace/worktree-picker"; @@ -404,7 +405,20 @@ export function worktreeActions(box: string, loc: Location, wt: Worktree): Actio confirm: "Stop sessions", destructive: true, run: async () => { - for (const s of sessions) await stopSession(box, s.name, true); + // A box with runs stops them itself (lib/stop-on-box.ts), so the + // laptop sleeping halfway leaves none running. + const read = await stopOnBox(box, `Stop everything in ${name}`, { sessions: sessions.map((s) => s.name), worktrees: [] }).catch((err: unknown) => { + toastManager.add({ title: "Could not stop the sessions", description: plainError(err), type: "error" }); + return null; + }); + if (read === null) return; + if (!read) { + for (const s of sessions) await stopSession(box, s.name, true); + return; + } + scheduleRefresh(box, ["sessions"]); + const failed = Object.values(read.rows).filter((r) => r.state === "failed"); + if (failed.length) toastManager.add({ title: `Could not stop ${failed.length === 1 ? "a session" : `${failed.length} sessions`}`, description: failed[0].message?.replace(/^Failed: /, ""), type: "error" }); }, }), { destructive: true }, @@ -422,7 +436,10 @@ export function worktreeActions(box: string, loc: Location, wt: Worktree): Actio // archiveWorktree asks, then archives a worktree: its sessions and services // stop, the repo's archive script runs (on the box, in the background), // then its folder goes. Its branch stays, and git refuses to lose -// uncommitted work, so nothing is lost; Remove is the way to discard. +// uncommitted work, so nothing is lost; Remove is the way to discard. The +// dialog closes as soon as it is confirmed; the row says "Archiving…" until +// the box is done, as a one-row job in the background-jobs tray, and a +// toast says how it went (lib/remove-worktree.ts). export function archiveWorktree(box: string, loc: Location, wt: Worktree) { const script = loc.scripts?.archive; confirm({ @@ -430,19 +447,7 @@ export function archiveWorktree(box: string, loc: Location, wt: Worktree) { description: `${script ? "The repo's archive script runs, then its" : "Its"} folder on ${box} goes and its sessions and services stop. ${wt.branch ? `Branch ${wt.branch} stays, so you can pick it up again.` : ""}`, detail: script ? {script} : undefined, confirm: "Archive", - run: async () => { - const client = useStore.getState().client; - if (!client) throw new Error("not connected"); - let res: { archive?: string }; - try { - res = await removeWorktreeOnBox(client, box, loc.name, wt, "archive"); - } catch (err) { - if (/modified|untracked|uncommitted|contains/i.test(errorMessage(err))) throw new Error(`${wt.name} has uncommitted changes, so it was left as it is. Commit them first, or use Remove worktree to discard them.`); - throw err; - } - if (res.archive) toastManager.add({ title: `Archiving ${wt.name}`, description: "Its archive script is running on the box; the worktree goes when it finishes.", type: "info" }); - else toastManager.add({ title: `Archived ${wt.name}`, description: wt.branch ? `Branch ${wt.branch} is kept.` : box, type: "success" }); - }, + run: () => queueRemoval(box, loc.name, wt, { kind: "archive" }), }); } diff --git a/app/src/hooks/lift-toasts.ts b/app/src/hooks/lift-toasts.ts index deea159b9..20cd750a7 100644 --- a/app/src/hooks/lift-toasts.ts +++ b/app/src/hooks/lift-toasts.ts @@ -3,7 +3,8 @@ // Put it on the bar as its ref: while the bar is up, --berth-bar-lift says // how far above the window's bottom edge the bar's top is (and a gap), and // the toast viewport (App.tsx) starts no lower than that. The loops panel -// does the same with --berth-loops-h. +// and the background-jobs tray do the same with --berth-loops-h and +// --berth-jobs-h. const GAP = 12; export function liftToasts(el: HTMLElement | null): (() => void) | undefined { diff --git a/app/src/lib/format.ts b/app/src/lib/format.ts index 8dd7b3231..d48ec73a6 100644 --- a/app/src/lib/format.ts +++ b/app/src/lib/format.ts @@ -22,3 +22,6 @@ export function ago(iso?: string): string { export function errorMessage(err: unknown): string { return err instanceof Error ? err.message : String(err); } + +// plural is "1 agent", "3 agents". +export const plural = (n: number, one: string, many = `${one}s`) => `${n} ${n === 1 ? one : many}`; diff --git a/app/src/lib/jobs.test.ts b/app/src/lib/jobs.test.ts new file mode 100644 index 000000000..ba56d1095 --- /dev/null +++ b/app/src/lib/jobs.test.ts @@ -0,0 +1,121 @@ +// node --experimental-strip-types --test src/lib/jobs.test.ts (pnpm test) +import assert from "node:assert/strict"; +import { mock, test } from "node:test"; + +const stored = new Map(); +globalThis.localStorage = { getItem: (k: string) => stored.get(k) ?? null, setItem: (k: string, v: string) => void stored.set(k, v) } as Storage; +// Folded when the app last quit. +stored.set("berth.jobs.folded", "true"); + +const { cancelJob, dismissJob, FADE_MS, finishJob, isCancelling, jobOf, leftOver, outcomeOf, progressOf, runPool, setFolded, setItem, startJob, useJobs } = await import("./jobs.ts"); + +const reset = () => useJobs.setState({ jobs: [] }); +const items = (...keys: string[]) => keys.map((key) => ({ key, group: "devl", label: key })); + +test("progress counts the rows that have settled, whichever way", () => { + reset(); + const id = startJob({ title: "Stopping 4 agents", items: items("a", "b", "c", "d") }); + assert.deepEqual(progressOf(jobOf(id)!), { settled: 0, total: 4 }); + setItem(id, "a", { state: "running" }); + setItem(id, "b", { state: "done" }); + setItem(id, "c", { state: "failed", message: "Failed: gone" }); + setItem(id, "d", { state: "cancelled" }); + assert.deepEqual(progressOf(jobOf(id)!), { settled: 3, total: 4 }); + assert.equal(outcomeOf(jobOf(id)!), "running"); +}); + +test("several jobs stack, newest on top", () => { + reset(); + const a = startJob({ title: "first", items: items("a") }); + const b = startJob({ title: "second", items: items("b") }); + assert.deepEqual( + useJobs.getState().jobs.map((j) => j.id), + [b, a], + ); +}); + +test("a clean job sums itself up and goes after a while; one with something kept stays as a report", () => { + reset(); + mock.timers.enable({ apis: ["setTimeout"] }); + try { + const clean = startJob({ title: "clean", items: items("a", "b") }); + setItem(clean, "a", { state: "done" }); + setItem(clean, "b", { state: "done" }); + finishJob(clean, "2 stopped"); + assert.equal(outcomeOf(jobOf(clean)!), "clean"); + + const kept = startJob({ title: "kept", items: items("c", "d") }); + setItem(kept, "c", { state: "done" }); + setItem(kept, "d", { state: "kept", message: "Stopped · Kept: uncommitted changes" }); + finishJob(kept, "2 stopped, 1 kept"); + assert.equal(outcomeOf(jobOf(kept)!), "report"); + assert.deepEqual( + leftOver(jobOf(kept)!).map((i) => i.key), + ["d"], + ); + + mock.timers.tick(FADE_MS - 1); + assert.ok(jobOf(clean)); + mock.timers.tick(1); + assert.equal(jobOf(clean), undefined); + assert.ok(jobOf(kept), "a report waits to be dismissed"); + dismissJob(kept); + assert.equal(jobOf(kept), undefined); + } finally { + mock.timers.reset(); + } +}); + +test("a cancelled job takes no more: only the items not yet sent are left out", async () => { + reset(); + const id = startJob({ title: "x", items: items("a", "b", "c", "d", "e"), cancellable: true }); + const sent: string[] = []; + let release!: () => void; + const hold = new Promise((r) => (release = r)); + const run = runPool( + ["a", "b", "c", "d", "e"], + 2, + async (k) => { + sent.push(k); + if (k === "b") { + cancelJob(id); + await hold; + } + }, + () => isCancelling(id), + ); + // a went and finished; b is in flight when the job is cancelled. + await new Promise((r) => setTimeout(r, 0)); + release(); + const unsent = await run; + assert.deepEqual(sent, ["a", "b"]); + assert.deepEqual(unsent.sort(), ["c", "d", "e"]); +}); + +test("the pool runs a few at a time", async () => { + let inFlight = 0; + let most = 0; + await runPool([1, 2, 3, 4, 5, 6, 7], 3, async () => { + most = Math.max(most, ++inFlight); + await new Promise((r) => setTimeout(r, 1)); + inFlight--; + }); + assert.equal(most, 3); +}); + +test("the pool runs every item it's given, undefined too", async () => { + const seen: (string | undefined)[] = []; + const unsent = await runPool(["a", undefined, "b"], 2, async (item) => void seen.push(item)); + assert.deepEqual(seen, ["a", undefined, "b"]); + assert.deepEqual(unsent, []); +}); + +test("folding the tray is remembered", () => { + assert.equal(useJobs.getState().folded, true, "as it was left"); + setFolded(false); + setFolded(true); + assert.equal(useJobs.getState().folded, true); + assert.equal(stored.get("berth.jobs.folded"), "true"); + setFolded(false); + assert.equal(stored.get("berth.jobs.folded"), "false"); +}); diff --git a/app/src/lib/jobs.ts b/app/src/lib/jobs.ts new file mode 100644 index 000000000..366851b29 --- /dev/null +++ b/app/src/lib/jobs.ts @@ -0,0 +1,144 @@ +import { create } from "zustand"; + +import { load, save } from "./storage.ts"; + +// Background jobs: work a dialog starts and then lets go of, such as the +// board's bulk Stop or Archive…, so you can get on with something else. The +// tray in the corner (components/jobs-tray.tsx) shows each one's rows as +// they go, folds to a line, and keeps a report when something was kept or +// failed. A clean job sums itself up and fades. The store is the app's, not +// a view's: a job carries on when you leave the page that started it. What +// the app does itself doesn't outlive it: rows not yet sent are lost when it +// quits. What a box does as a run does, and its job comes back on the next +// start (ref, lib/stop-job.ts). +// +// No imports from "@/": pnpm test runs this file in plain node. + +type ItemState = "queued" | "running" | "done" | "kept" | "failed" | "cancelled"; + +export interface JobItem { + key: string; + // Where it runs: a box. The tray groups rows by it. + group: string; + label: string; + // A second name, when the label alone is ambiguous. + aside?: string; + state: ItemState; + message?: string; +} + +export interface Job { + id: number; + // What it does, while it runs: "Stopping 17 agents". + title: string; + items: JobItem[]; + // Whether "Stop the rest" is offered. + cancellable: boolean; + cancelling?: boolean; + // Set when it ends: what it did, "17 stopped, 15 archived". + summary?: string; + finished?: number; + // What its starter keeps it by across a relaunch, until it's dismissed. + ref?: string; +} + +// A clean job's summary stays this long, then goes. +export const FADE_MS = 10_000; + +const FOLDED = "berth.jobs.folded"; + +export const useJobs = create<{ jobs: Job[]; folded: boolean }>()(() => ({ jobs: [], folded: load(FOLDED, false) })); + +let nextId = 0; + +// startJob puts a job on top of the tray and answers with its id. +export function startJob(job: { title: string; items: Omit[]; cancellable?: boolean; ref?: string }): number { + const id = ++nextId; + const items = job.items.map((i) => ({ ...i, state: "queued" as const })); + useJobs.setState((s) => ({ jobs: [{ id, title: job.title, items, cancellable: !!job.cancellable, ref: job.ref }, ...s.jobs] })); + return id; +} + +const patchJob = (id: number, f: (j: Job) => Job) => useJobs.setState((s) => ({ jobs: s.jobs.map((j) => (j.id === id ? f(j) : j)) })); + +export function setItem(id: number, key: string, patch: Partial>) { + patchJob(id, (j) => ({ ...j, items: j.items.map((i) => (i.key === key ? { ...i, ...patch } : i)) })); +} + +export const jobOf = (id: number) => useJobs.getState().jobs.find((j) => j.id === id); + +const cancelled: ((job: Job) => void)[] = []; + +// onCancel hears of each job asked to send nothing more. +export const onCancel = (fn: (job: Job) => void) => void cancelled.push(fn); + +// cancelJob asks a job to send nothing more. What it already sent finishes. +export function cancelJob(id: number) { + patchJob(id, (j) => ({ ...j, cancelling: true })); + const j = jobOf(id); + if (j) for (const fn of cancelled) fn(j); +} +export const isCancelling = (id: number) => !!jobOf(id)?.cancelling; + +// finishJob ends a job with its summary. A clean one goes after FADE_MS; one +// with a row kept, failed or not sent stays until it's dismissed. +export function finishJob(id: number, summary: string) { + patchJob(id, (j) => ({ ...j, summary, finished: Date.now() })); + const j = jobOf(id); + if (!j || outcomeOf(j) !== "clean") return; + // unref: node (pnpm test) needn't wait for it. + const t: unknown = setTimeout(() => dismissJob(id), FADE_MS); + (t as { unref?: () => void }).unref?.(); +} + +const dismissed: ((job: Job) => void)[] = []; + +// onDismiss hears of each job as it leaves the tray, dismissed or faded. +export const onDismiss = (fn: (job: Job) => void) => void dismissed.push(fn); + +export function dismissJob(id: number) { + const j = jobOf(id); + if (!j) return; + useJobs.setState((s) => ({ jobs: s.jobs.filter((x) => x.id !== id) })); + for (const fn of dismissed) fn(j); +} + +export function setFolded(folded: boolean) { + useJobs.setState({ folded }); + save(FOLDED, folded); +} + +const settledStates: ItemState[] = ["done", "kept", "failed", "cancelled"]; +export const settled = (i: JobItem) => settledStates.includes(i.state); + +export function progressOf(j: Job) { + return { settled: j.items.filter(settled).length, total: j.items.length }; +} + +// The rows a report lists: whatever didn't go as asked. +export const leftOver = (j: Job) => j.items.filter((i) => i.state === "kept" || i.state === "failed" || i.state === "cancelled"); + +export function outcomeOf(j: Job): "running" | "clean" | "report" { + if (j.finished === undefined) return "running"; + return leftOver(j).length ? "report" : "clean"; +} + +// runPool runs fn over items, a few at a time, in order. Once stop() says +// so it takes no more, and answers with the items it never sent. Any item +// counts, undefined too. +export async function runPool(items: T[], concurrency: number, fn: (item: T) => Promise, stop: () => boolean = () => false): Promise { + const queue = [...items]; + const unsent: T[] = []; + const worker = async () => { + while (queue.length) { + const item = queue.shift() as T; + if (stop()) { + unsent.push(item, ...queue.splice(0)); + return; + } + await fn(item); + } + }; + await Promise.all(Array.from({ length: Math.min(concurrency, items.length) }, worker)); + return unsent; +} diff --git a/app/src/lib/kept-reason.test.ts b/app/src/lib/kept-reason.test.ts index d2d005b08..81ac324e9 100644 --- a/app/src/lib/kept-reason.test.ts +++ b/app/src/lib/kept-reason.test.ts @@ -27,3 +27,14 @@ test("other failures aren't a reason to force", () => { assert.equal(keptReason("git worktree remove: fatal: '/home/acme/code/shop-fix' contains a .git file that is not a gitlink"), null); assert.equal(keptReason("git worktree remove: error: unable to unlink old 'src/modified.ts': Permission denied"), null); }); + +test("a removal that timed out, was stopped or failed silently is a failure, not a kept worktree", () => { + for (const msg of [ + "git worktree remove timed out after 10 min and was stopped; the folder may be partly removed, and archiving it again finishes it", + "git worktree remove was stopped (signal: killed) and said nothing", + "git worktree remove failed with no output (exit status 1)", + "git worktree unlock failed with no output (exit status 128)", + ]) { + assert.equal(keptReason(msg), null, msg); + } +}); diff --git a/app/src/lib/mock-runs.ts b/app/src/lib/mock-runs.ts index 778227da4..bbaec877f 100644 --- a/app/src/lib/mock-runs.ts +++ b/app/src/lib/mock-runs.ts @@ -1,4 +1,5 @@ -import { ApiError, type BerthEvent, type Run, type RunCandidate, type RunStep, type RunSummary } from "@/lib/api"; +import { ApiError, type BerthEvent, type Run, type RunCandidate, type RunStep, type RunSummary, type Session } from "@/lib/api"; +import { agentOf } from "@/lib/derive"; // Runs in the demo: the shop world on devl has a loop going, an attempts run // of three agents waiting for you to pick one in Review's Compare, one whose @@ -6,7 +7,8 @@ import { ApiError, type BerthEvent, type Run, type RunCandidate, type RunStep, t // Gitea repository that finished with its branch pushed and no pull // request, a flow's run, a review and a finished broadcast. Deciding, cancelling and starting // runs work, so the loops panel, Automations → Runs, Compare and the phone -// all have something to show. +// all have something to show. A bulk Stop's stop run plays out against the +// demo's own sessions and worktrees (playStop). type Emit = (e: Omit) => void; @@ -237,7 +239,117 @@ function world(): Run[] { const summary = (r: Run): RunSummary => ({ ...r, steps: r.steps.length, candidates: r.candidates?.length }); -export function runsCall(box: string, method: string, path: string, body: unknown, emit: Emit, delay: (v: T) => Promise): Promise | undefined { +// Act calls the mock box's own routes, as a run's steps do on a real one. +type Act = (method: string, path: string) => Promise; + +// Each box archives one worktree at a time across its stop runs, as a box +// does (archiveTurn in internal/box/api.go): archiveTurn answers with the +// release once it's this one's turn. +const archiveLines = new Map>(); +function archiveTurn(box: string): Promise<() => void> { + const before = archiveLines.get(box) ?? Promise.resolve(); + let release = () => {}; + const mine = new Promise((done) => (release = done)); + archiveLines.set( + box, + before.then(() => mine), + ); + return before.then(() => release); +} + +// playStop plays a stop run out as the box does (internal/box/runstop.go): +// the sessions stopped three at a time, then each worktree archived where +// it's safe, one at a time and in the box's line, through the mock's own +// routes. Cancelled, it finishes the step in flight and starts no more. +async function playStop(r: Run, box: string, act: Act, emit: Emit) { + const p = r.params as { sessions: string[]; worktrees: { location: string; name: string; path: string; sessions: string[] }[] }; + const stops: RunStep[] = []; + const archives: RunStep[] = []; + const show = () => + Object.assign(r, { + updated: new Date().toISOString(), + steps: [{ id: "stop", path: "0", kind: "map", status: "running", exit_code: 0, children: stops }, ...(archives.length ? [{ id: "archive", path: "1", kind: "map", status: "running", exit_code: 0, children: archives }] : [])], + }); + const put = (list: RunStep[], k: number, path: string, kind: string, status: string, more: Partial = {}) => { + const leaf: RunStep = { id: kind === "stop_session" ? "session" : "worktree", path, kind, status, exit_code: 0, ...more }; + list[k] = { id: `i${k}`, path: path.slice(0, -2), kind: "item", status, exit_code: 0, children: [leaf] }; + show(); + }; + const over = () => stopping.has(r.id); + const queue = p.sessions.map((s, k) => [s, k] as const); + const worker = async () => { + for (let next = queue.shift(); next && !over(); next = queue.shift()) { + const [s, k] = next; + put(stops, k, `0.i${k}.0`, "stop_session", "running"); + try { + await act("DELETE", `sessions/${encodeURIComponent(s)}`); + put(stops, k, `0.i${k}.0`, "stop_session", "succeeded", { output: "stopped" }); + } catch (err) { + put(stops, k, `0.i${k}.0`, "stop_session", "failed", { error: (err as Error).message, exit_code: 1 }); + } + } + }; + await Promise.all([worker(), worker(), worker()]); + for (const [j, w] of p.worktrees.entries()) { + if (over()) break; + const at = `1.i${j}.0`; + put(archives, j, at, "archive_worktree", "running"); + const release = await archiveTurn(box); + try { + if (over()) { + put(archives, j, at, "archive_worktree", "cancelled"); + break; + } + await archiveOne(j, at, w); + } finally { + release(); + } + } + async function archiveOne(j: number, at: string, w: (typeof p.worktrees)[number]) { + let live: Session[]; + try { + live = ((await act("GET", "sessions")) as Session[]).filter((s) => !s.exited && agentOf(s) && (s.dir === w.path || s.dir.startsWith(`${w.path}/`))); + } catch (err) { + put(archives, j, at, "archive_worktree", "failed", { error: (err as Error).message, exit_code: 1 }); + return; + } + if (live.length) { + put(archives, j, at, "archive_worktree", "skipped", { output: live.some((s) => p.sessions.includes(s.name)) ? "Kept: an agent there didn't stop" : "Kept: another agent is still running there" }); + return; + } + try { + const res = (await act("DELETE", `locations/${encodeURIComponent(w.location)}/worktrees/${encodeURIComponent(w.name)}`)) as { archive?: string }; + // The box waits for the archive script, then says it's gone. + if (res?.archive) await new Promise((ok) => setTimeout(ok, 1600)); + put(archives, j, at, "archive_worktree", "succeeded", { output: "Archived; its branch is kept" }); + } catch (err) { + const msg = (err as Error).message; + if (/modified or untracked/.test(msg)) put(archives, j, at, "archive_worktree", "skipped", { output: "Kept: uncommitted changes" }); + else if (/no worktree/.test(msg)) put(archives, j, at, "archive_worktree", "succeeded", { output: "already gone" }); + else put(archives, j, at, "archive_worktree", "failed", { error: msg, exit_code: 1 }); + } + } + const failed = stops.some((s) => s.status === "failed") || archives.some((s) => s.status === "failed"); + const status = over() ? "cancelled" : failed ? "failed" : "succeeded"; + // Its maps end too, as the box's do, so the run's timeline shows none + // still running. + const mapStatus = (items: RunStep[]) => (over() ? "cancelled" : items.some((c) => c.status === "failed") ? "failed" : "succeeded"); + Object.assign(r, { + status, + error: status === "succeeded" ? undefined : status === "cancelled" ? "cancelled" : "a step failed", + finished: new Date().toISOString(), + updated: new Date().toISOString(), + steps: r.steps.map((s) => ({ ...s, status: mapStatus(s.path === "0" ? stops : archives) })), + }); + emit({ type: "run.finished", box, data: { run: r.id, status } }); +} + +// The stop runs asked to stop: they end once the step in flight has. +const stopping = new Set(); +// The Idempotency-Key each run was started under, by run. +const idemKeys = new Map(); + +export function runsCall(box: string, method: string, path: string, body: unknown, emit: Emit, delay: (v: T) => Promise, act?: Act): Promise | undefined { if (box !== "devl" && box !== "gpu") { if (method === "GET" && path.startsWith("runs")) return delay(path.startsWith("runs?") ? [] : undefined); if (method === "GET" && path === "autofix") return delay([]); @@ -248,7 +360,8 @@ export function runsCall(box: string, method: string, path: string, body: unknow if (method === "GET" && path.startsWith("runs?")) { const q = new URLSearchParams(path.split("?")[1]); const t = q.get("template"); - return delay(all.filter((r) => !t || r.template === t).map(summary)); + const g = q.get("group"); + return delay(all.filter((r) => (!t || r.template === t) && (!g || r.group === g)).map(summary)); } if (method === "GET" && path === "runs/templates") return delay([]); let m = /^runs\/([^/?]+)$/.exec(path); @@ -264,6 +377,10 @@ export function runsCall(box: string, method: string, path: string, body: unknow m = /^runs\/([^/]+)\/cancel$/.exec(path); if (method === "POST" && m) { const r = all.find((x) => x.id === m![1]); + if (r?.template === "stop" && r.status === "running") { + stopping.add(r.id); + return delay({ cancelled: m[1] }); + } if (r) Object.assign(r, { status: "cancelled", finished: new Date().toISOString(), error: "cancelled", gate: undefined }); emit({ type: "run.finished", box, data: { run: m[1], status: "cancelled" } }); return delay({ cancelled: m[1] }); @@ -297,8 +414,12 @@ export function runsCall(box: string, method: string, path: string, body: unknow return delay({ decided: r.id, approve: d.approve }); } if (method === "POST" && path === "runs") { - const req = body as { template?: string; title?: string; params?: Record }; - const r: Run = { id: `r_mdemo${Math.random().toString(36).slice(2, 7)}`, template: req.template ?? "flow", title: req.title ?? `${req.template} run`, status: "running", created: new Date().toISOString(), updated: new Date().toISOString(), params: req.params, steps: [] }; + const req = body as { template?: string; title?: string; group?: string; params?: Record; idem_key?: string }; + // Asked again under the same key, the box answers with the run it has. + const had = req.idem_key ? all.find((x) => idemKeys.get(x.id) === req.idem_key) : undefined; + if (had) return delay(summary(had)); + const r: Run = { id: `r_mdemo${Math.random().toString(36).slice(2, 7)}`, template: req.template ?? "flow", title: req.title ?? `${req.template} run`, status: "running", created: new Date().toISOString(), updated: new Date().toISOString(), group: req.group, params: req.params, steps: [] }; + if (req.idem_key) idemKeys.set(r.id, req.idem_key); all.unshift(r); emit({ type: "run.created", box, data: { run: r.id, template: r.template, status: "running" } }); // A broadcast plays out: each prompt typed in, then, if asked, each @@ -326,6 +447,7 @@ export function runsCall(box: string, method: string, path: string, body: unknow wait ? 4200 : 1400, ); } + if (r.template === "stop" && act) void playStop(r, box, act, emit); return delay(summary(r)); } if (method === "GET" && path.startsWith("review?run=")) { diff --git a/app/src/lib/mock.ts b/app/src/lib/mock.ts index 729b3412a..44734ff05 100644 --- a/app/src/lib/mock.ts +++ b/app/src/lib/mock.ts @@ -100,6 +100,15 @@ const mockBuilds: Record = { devl: SHIPPED_BUILD, gpu: "304337b9 // ?mock=1&fresh=1 is a new account: no boxes yet, so onboarding shows. const fresh = new URLSearchParams(location.search).has("fresh"); +// ?mock=1&slowremove=MS: a box takes that long to remove a worktree with no +// archive script, as one with a big node_modules does, and answers once it +// is gone. +const slowRemove = Number(new URLSearchParams(location.search).get("slowremove")) || 0; +// ?mock=1&slowstop=MS: a box takes that long to answer a session's stop. +const slowStop = Number(new URLSearchParams(location.search).get("slowstop")) || undefined; +// ?noruns: every box is one from before runs, so the app does what a box's +// runs would (a bulk Stop's own loop, lib/stop-job.ts). +const noRuns = new URLSearchParams(location.search).has("noruns"); if (fresh) { status.boxes = []; status.forwards = []; @@ -370,7 +379,7 @@ if (!fresh && !__BERTH_DEMO__ && !new URLSearchParams(location.search).has("stil emit({ type: s.agent_state === "waiting" ? "agent.waiting" : "agent.started", box: "devl", origin: s.agent, data: { path: s.dir } }); }, 25_000); -const delay = (v: T) => new Promise((r) => setTimeout(() => r(structuredClone(v)), 120)); +const delay = (v: T, ms = 120) => new Promise((r) => setTimeout(() => r(structuredClone(v)), ms)); // mockOrchestration answers send, wait and exec: a prompted agent works for // a moment and finishes, and a worktree's check fails once, then passes, so @@ -695,7 +704,7 @@ function boxCall(box: string, method: string, path: string, body?: unknown): Pro if (team) return team; const flows = flowsCall(box, method, path, body, emit, delay); if (flows) return flows; - const runs = runsCall(box, method, path, body, emit, delay); + const runs = runsCall(box, method, path, body, emit, delay, (m, p) => boxCall(box, m, p)); if (runs) return runs; const browser = browserCall(box, method, path); if (browser) return delay(browser); @@ -787,7 +796,7 @@ function boxCall(box: string, method: string, path: string, body?: unknown): Pro home: HOME, // gpu runs an older berthd (mockBuilds): it can't keep worktree names, // and its agent browser has no tabs. - capabilities: ["diff", "turns", "queue", "ask", "answer", "journal", "runs", "exec.detach", "browser", "browser.devtools", "titles", "sample", "service.terminal", "session.home", "agents.install", "artifacts", "processes", ...(box === "gpu" ? [] : ["worktree.titles", "agents.paths", "browser.tabs"]), ...(structuredDemo ? ["driver.claude", "conversations", "sessions.past"] : [])], + capabilities: ["diff", "turns", "queue", "ask", "answer", "journal", ...(noRuns ? [] : ["runs"]), "exec.detach", "browser", "browser.devtools", "titles", "sample", "service.terminal", "session.home", "agents.install", "artifacts", "processes", ...(box === "gpu" ? [] : ["worktree.titles", "agents.paths", "browser.tabs"]), ...(structuredDemo ? ["driver.claude", "conversations", "sessions.past"] : [])], // Claude Code from npm under nvm, as the person's shell finds it; // Codex from Shipyard's own installer. agent_paths: [ @@ -885,6 +894,7 @@ function boxCall(box: string, method: string, path: string, body?: unknown): Pro }, 1500); return delay({ removing: wt, archive: l.scripts.archive }); } + if (slowRemove) return delay({ removed: wt }, slowRemove).finally(gone); setTimeout(gone, 50); return delay({ removed: wt }); } @@ -939,7 +949,7 @@ function boxCall(box: string, method: string, path: string, body?: unknown): Pro const name = decodeURIComponent(path.slice("sessions/".length)); sessions[box] = sessions[box].filter((s) => s.name !== name); setTimeout(() => emit({ type: "session.stopped", box, data: { name } }), 50); - return delay({ removed: name }); + return delay({ removed: name }, slowStop); } if (key === "POST locations") { const r = body as { name: string; path: string }; diff --git a/app/src/lib/orchestrate-core.ts b/app/src/lib/orchestrate-core.ts index 0ac44a482..e8d1f9632 100644 --- a/app/src/lib/orchestrate-core.ts +++ b/app/src/lib/orchestrate-core.ts @@ -334,11 +334,12 @@ export const runsApi = (call: BoxCaller) => ({ // idem_key is the Idempotency-Key, for callers that cannot set headers. call(box, "POST", "runs", idemKey ? { ...req, idem_key: idemKey } : req), get: (box: string, id: string) => call(box, "GET", `runs/${enc(id)}`), - list: async (box: string, o: { status?: string; template?: string; flow?: string; limit?: number } = {}) => { + list: async (box: string, o: { status?: string; template?: string; flow?: string; group?: string; limit?: number } = {}) => { const q = new URLSearchParams(); if (o.status) q.set("status", o.status); if (o.template) q.set("template", o.template); if (o.flow) q.set("flow", o.flow); + if (o.group) q.set("group", o.group); q.set("limit", String(o.limit ?? 50)); return (await call(box, "GET", `runs?${q}`)) ?? []; }, diff --git a/app/src/lib/remove-worktree.ts b/app/src/lib/remove-worktree.ts index b196e66e2..eb8abba79 100644 --- a/app/src/lib/remove-worktree.ts +++ b/app/src/lib/remove-worktree.ts @@ -1,14 +1,23 @@ import { toastError } from "@/components/error-note"; import { toastManager } from "@/components/ui/toast"; import type { Client } from "@/lib/api"; +import { plainError } from "@/lib/errors"; import { errorMessage } from "@/lib/format"; +import { finishJob, setItem, startJob } from "@/lib/jobs"; import { keptReason } from "@/lib/kept-reason"; import { unpinWorktree } from "@/lib/pinned-worktrees"; -import { clearRemoving, markRemoving, markScript, pruneRemovals, type RemovalKind, removalOf, removeResult, useRemovals } from "@/lib/removing"; +import { archivedMessage, clearRemoving, inTurn, KEPT_DIRTY, markRemoving, markScript, pruneRemovals, type RemovalKind, removalKey, removalOf, removeResult, unmarked, useRemovals } from "@/lib/removing"; +import { followStopRun } from "@/lib/stop-job"; +import { startStopOnBox, stopRuns } from "@/lib/stop-on-box"; import { scheduleRefresh, useStore } from "@/lib/store"; import { forgetWorktree } from "@/lib/workspaces"; import { worktreesApi } from "@/lib/worktrees"; +// How the removals sent with an archive script ended, by removalKey, from +// before they're sent: worktreeGone and archiveFailed say, as they drop the +// mark, for archiveEnded to read. +const waiting = new Map(); + // removeWorktreeOnBox asks the box to remove (or archive) a worktree and // follows it through. A plain removal is done when the answer comes; with an // archive script the box answers at once and runs it in the background, so @@ -25,15 +34,21 @@ export async function removeWorktreeOnBox( opts: { force?: boolean; branch?: boolean } = {}, ): Promise<{ archive?: string }> { markRemoving(box, wt.path, kind, wt.name); + // Noted before it's sent: a script that fails fast can be reported + // before the answer comes (archiveEnded). + const key = removalKey(box, wt.path); + waiting.set(key, undefined); let res: { archive?: string }; try { res = removeResult(await worktreesApi.remove(client, box, location, wt.name, opts)); } catch (err) { + waiting.delete(key); clearRemoving(box, wt.path); throw err; } if (res.archive) markScript(box, wt.path); else { + waiting.delete(key); clearRemoving(box, wt.path); forgetWorktree(box, wt.path); unpinWorktree(box, wt.path); @@ -42,55 +57,144 @@ export async function removeWorktreeOnBox( return res; } -// Removals asked for and not yet sent, per box. Each box takes one at a -// time, so a slow teardown never has others half done beside it. -const waiting = new Map(); - -// queueRemoval removes a worktree without asking first: it is marked -// "Removing…" at once and goes when its box gets to it. Its branch stays, -// and git still refuses to lose uncommitted work: then a toast offers to -// remove it anyway. -export function queueRemoval(box: string, location: string, wt: { name: string; path: string }, opts: { force?: boolean } = {}) { +// queueRemoval removes or archives a worktree in the background: it is +// marked "Removing…" or "Archiving…" at once, so whatever asked (archive's +// dialog) can close, and goes when its box gets to it, one at a time per +// box. Its branch stays, and git still refuses to lose uncommitted work: +// then a toast says so, and a removal's offers to remove it anyway. An +// archive is a one-row job in the background-jobs tray (lib/jobs.ts) too, +// until the worktree is gone or kept. On a box with runs an archive is the +// box's to do (archiveOnBox); elsewhere the app sends it in the box's line +// (inTurn), which it holds until the box says the worktree went. +export function queueRemoval(box: string, location: string, wt: Leaving, opts: { kind?: RemovalKind; force?: boolean } = {}) { + const kind = opts.kind ?? "remove"; if (removalOf(useRemovals.getState().byKey, box, wt.path)) return; - markRemoving(box, wt.path, "remove", wt.name); - const list = waiting.get(box); - if (list) { - list.push({ location, wt, force: opts.force }); - return; - } - waiting.set(box, [{ location, wt, force: opts.force }]); - void drain(box); + markRemoving(box, wt.path, kind, wt.name); + const job = kind === "archive" ? archiveJob(box, wt) : undefined; + if (job && !opts.force && stopRuns.onBox(box)) return void archiveOnBox(box, location, wt, job); + void inTurn(box, () => send(box, location, wt, kind, opts.force, job)); +} + +const KEY = "worktree"; + +function archiveJob(box: string, wt: Leaving) { + const id = startJob({ title: `Archiving ${wt.name}`, items: [{ key: KEY, group: box, label: wt.name, aside: wt.branch, message: `Waiting for ${box}` }] }); + const end = (item: Parameters[2], summary: string) => { + setItem(id, KEY, item); + finishJob(id, summary); + }; + return { + waiting: () => setItem(id, KEY, { message: `Waiting for ${box}` }), + sent: () => setItem(id, KEY, { state: "running", message: "Archiving…" }), + scripted: () => setItem(id, KEY, { state: "running", message: archivedMessage({ archive: "yes" }) }), + done: () => end({ state: "done", message: archivedMessage({}) }, `Archived ${wt.name}`), + kept: (why = KEPT_DIRTY) => end({ state: "kept", message: why }, `Kept ${wt.name}`), + failed: (why: string) => end({ state: "failed", message: `Couldn't archive: ${why}` }, `Couldn't archive ${wt.name}`), + }; +} + +type Leaving = { name: string; path: string; branch?: string }; + +// archiveEnded resolves once the box says how an archive whose script runs +// (removeWorktreeOnBox answered 202) ended: gone, failed, or undefined when +// its mark went some other way (the box stopped listing it, or it went +// stale). +export async function archiveEnded(box: string, path: string): Promise<"gone" | "failed" | undefined> { + const key = removalKey(box, path); + await unmarked(box, path); + const how = waiting.get(key); + waiting.delete(key); + return how; } -async function drain(box: string) { - const list = waiting.get(box)!; - while (list.length) { - const { location, wt, force } = list[0]; - try { - const client = useStore.getState().client; - if (!client) throw new Error("not connected"); - await removeWorktreeOnBox(client, box, location, wt, "remove", { force }); - } catch (err) { - clearRemoving(box, wt.path); - const why = force ? null : keptReason(errorMessage(err)); - if (why) { +async function send(box: string, location: string, wt: Leaving, kind: RemovalKind, force?: boolean, job?: ReturnType) { + try { + const client = useStore.getState().client; + if (!client) throw new Error("not connected"); + job?.sent(); + const res = await removeWorktreeOnBox(client, box, location, wt, kind, { force }); + if (kind === "archive" && res.archive) toastManager.add({ title: `Archiving ${wt.name}`, description: "Its archive script is running on the box; the worktree goes when it finishes.", type: "info" }); + else if (kind === "archive") toastManager.add({ title: `Archived ${wt.name}`, description: wt.branch ? `Branch ${wt.branch} is kept.` : box, type: "success" }); + if (!res.archive) return job?.done(); + // The script runs on the box: this box's turn, and the tray's row, last + // until it says the worktree went or archiving failed. + job?.scripted(); + if ((await archiveEnded(box, wt.path)) === "failed") job?.failed("its archive script failed, so it was left as it was"); + else job?.done(); + } catch (err) { + clearRemoving(box, wt.path); + const why = force ? null : keptReason(errorMessage(err)); + if (why) { + job?.kept(); + if (kind === "archive") { + toastManager.add({ + type: "warning", + title: `Couldn't archive ${wt.name}`, + description: `${wt.name} has uncommitted changes, so it was left as it is. Commit them first, or use Remove worktree to discard them.`, + }); + } else { toastManager.add({ type: "warning", title: `Kept ${wt.name}`, description: why, actionProps: { children: "Remove anyway", onClick: () => queueRemoval(box, location, wt, { force: true }) }, }); - } else toastError(err, { title: `Couldn't remove ${wt.name}`, box }); + } + } else { + job?.failed(plainError(err)); + toastError(err, { title: `Couldn't ${kind} ${wt.name}`, box }); } - list.shift(); } - waiting.delete(box); +} + +// archiveOnBox has a box with runs archive the worktree as a one-worktree +// stop run (lib/stop-job.ts): its sessions stop first, as archiving always +// ended them, then it's archived where it's safe, in the box's own line with +// every other archive there (archiveTurn in internal/box/api.go). The box +// finishes it while the laptop sleeps or the app is closed; the tray's row +// follows it until the worktree is gone or kept. A box that refuses the run +// (from before stop runs) gets the app's way. +async function archiveOnBox(box: string, location: string, wt: Leaving, job: ReturnType) { + const sessions = (useStore.getState().boxes[box]?.sessions ?? []).filter((s) => !s.exited && (s.dir === wt.path || s.dir.startsWith(`${wt.path}/`))).map((s) => s.name); + const started = await startStopOnBox(box, `Archive ${wt.name}`, { sessions, worktrees: [{ location, name: wt.name, path: wt.path, sessions }] }, { waiting: job.waiting }); + if (typeof started === "string") return void inTurn(box, () => send(box, location, wt, "archive", false, job)); + job.sent(); + const read = await followStopRun(stopRuns, box, started.id, { + onRead: (r) => { + if (r.worktrees[wt.path] === "archiving") markScript(box, wt.path); + }, + }); + scheduleRefresh(box, ["locations", "sessions", "services"]); + const outcome = read?.worktrees[wt.path]; + const said = read?.said[wt.path] ?? ""; + if (outcome === "archived" || outcome === "archiving") { + if (outcome === "archived") worktreeGone(box, wt.path); + job.done(); + toastManager.add({ title: `Archived ${wt.name}`, description: wt.branch ? `Branch ${wt.branch} is kept.` : box, type: "success" }); + return; + } + clearRemoving(box, wt.path); + if (outcome === "kept") { + job.kept(said); + toastManager.add({ + type: "warning", + title: `Couldn't archive ${wt.name}`, + description: said === KEPT_DIRTY ? `${wt.name} has uncommitted changes, so it was left as it is. Commit them first, or use Remove worktree to discard them.` : `${said.replace(/^Kept: /, "")}, so it was left as it is.`, + }); + return; + } + // No word on the worktree: the box lost the run, or an agent there + // didn't stop (its row says why). + const why = !read ? `${box} no longer has the run` : outcome ? said.replace(/^Couldn't archive: /, "") : (Object.values(read.rows).find((x) => x.state === "failed")?.message?.replace(/^Failed: /, "") ?? "it wasn't archived"); + job.failed(why); + toastManager.add({ type: "error", title: `Couldn't archive ${wt.name}`, description: why }); } // worktreeGone is the box saying a worktree went: its workspace (tabs, // panes) and its pin go too, and an archive the app was waiting on says it // is done. A failed archive keeps both (archiveFailed). export function worktreeGone(box: string, path: string) { + if (waiting.has(removalKey(box, path))) waiting.set(removalKey(box, path), "gone"); const was = clearRemoving(box, path); forgetWorktree(box, path); unpinWorktree(box, path); @@ -100,6 +204,7 @@ export function worktreeGone(box: string, path: string) { // archiveFailed puts the worktree back as it was; the failure itself is a // notification of its own (lib/events.ts), with the script's output. export function archiveFailed(box: string, path: string) { + if (waiting.has(removalKey(box, path))) waiting.set(removalKey(box, path), "failed"); clearRemoving(box, path); } diff --git a/app/src/lib/removing.test.ts b/app/src/lib/removing.test.ts index 662269e32..24d18499b 100644 --- a/app/src/lib/removing.test.ts +++ b/app/src/lib/removing.test.ts @@ -2,7 +2,7 @@ import assert from "node:assert/strict"; import { test } from "node:test"; -import { clearRemoving, markRemoving, markScript, pruneRemovals, removalLabel, removalOf, removeResult, STALE_MS, useRemovals } from "./removing.ts"; +import { clearRemoving, inTurn, markRemoving, markScript, pruneRemovals, removalLabel, removalOf, removeResult, STALE_MS, unmarked, useRemovals } from "./removing.ts"; const reset = () => useRemovals.setState({ byKey: {} }); const of = (box: string, path: string, now?: number) => removalOf(useRemovals.getState().byKey, box, path, now); @@ -50,3 +50,38 @@ test("a DELETE's answer is read from JSON or, from older boxes, text", () => { assert.deepEqual(removeResult("not json"), {}); assert.deepEqual(removeResult(undefined), {}); }); + +test("a box takes its removals one at a time; another box's don't wait", async () => { + const log: string[] = []; + let answer!: () => void; + const slow = new Promise((r) => (answer = r)); + void inTurn("devl", async () => { + log.push("a sent"); + await slow; + log.push("a done"); + }); + const b = inTurn("devl", async () => { + log.push("b sent"); + }); + await inTurn("gpu", async () => { + log.push("c sent"); + }); + assert.deepEqual(log, ["a sent", "c sent"]); + answer(); + await b; + assert.deepEqual(log, ["a sent", "c sent", "a done", "b sent"]); +}); + +test("unmarked waits until the box says the worktree went, and a stale mark lets go at once", async () => { + useRemovals.setState({ byKey: {} }); + markRemoving("devl", "/w/fix", "archive", "fix"); + let done = false; + const waiting = unmarked("devl", "/w/fix").then(() => (done = true)); + await new Promise((r) => setTimeout(r, 5)); + assert.equal(done, false); + clearRemoving("devl", "/w/fix"); + await waiting; + assert.equal(done, true); + markRemoving("devl", "/w/old", "archive", "old", Date.now() - STALE_MS - 1); + await unmarked("devl", "/w/old"); +}); diff --git a/app/src/lib/removing.ts b/app/src/lib/removing.ts index b3191f514..666ed81df 100644 --- a/app/src/lib/removing.ts +++ b/app/src/lib/removing.ts @@ -1,11 +1,16 @@ import { create } from "zustand"; -// Worktrees on their way out. Removing one is quick, but a repository with -// an archive script has the box run it first, in the background, and the -// worktree only goes when it succeeds: minutes, sometimes. Until the box -// says it is gone (worktree.removed) or that archiving failed, the app shows -// it as "Archiving…" or "Removing…" and keeps it from being opened, so -// nobody starts work in a folder that is about to go. +import { errorMessage } from "./format.ts"; +import { oneAtATime } from "./one-at-a-time.ts"; + +// Worktrees on their way out. The box answers a removal once the folder is +// gone, which for a big checkout (node_modules) takes seconds. A repository +// with an archive script has the box run it first, in the background, and +// the worktree only goes when it succeeds: minutes, sometimes. From the +// moment it is asked until the box says it is gone (worktree.removed) or +// that archiving failed, the app shows it as "Archiving…" or "Removing…" +// and keeps it from being opened, so nobody starts work in a folder that is +// about to go. // // No imports from "@/": pnpm test runs this file in plain node. @@ -58,9 +63,51 @@ export function removalOf(byKey: Record, box: string, path: str export const useRemoval = (box: string, path: string | undefined) => useRemovals((s) => (path ? removalOf(s.byKey, box, path) : undefined)); +// Each box's removals, in line. A box takes one at a time, so a slow +// teardown never has others half done beside it; another box's don't wait. +const lines = new Map>(); + +// inTurn runs job once the box's earlier removals are done, whether they +// worked or failed. +export function inTurn(box: string, job: () => Promise): Promise { + let line = lines.get(box); + if (!line) lines.set(box, (line = oneAtATime())); + return line(job); +} + +// unmarked resolves once a worktree's mark is dropped: its box said it went +// or that archiving failed, or stopped listing it. A mark gone stale lets +// go too. Holding a box's turn (inTurn) until then keeps its archive scripts +// from running side by side. +export function unmarked(box: string, path: string): Promise { + const key = removalKey(box, path); + return new Promise((resolve) => { + let timer: ReturnType | undefined; + const check = () => { + const r = useRemovals.getState().byKey[key]; + if (r && Date.now() - r.since < STALE_MS) return; + unsub(); + clearTimeout(timer); + resolve(); + }; + const unsub = useRemovals.subscribe(check); + timer = setTimeout(check, STALE_MS); + // unref: node (pnpm test) needn't wait for it. + (timer as { unref?: () => void }).unref?.(); + check(); + }); +} + // removalLabel is what a row says meanwhile. export const removalLabel = (r: Removal) => (r.kind === "archive" ? "Archiving…" : "Removing…"); +// keepsWork is git refusing, in its own words, to lose uncommitted work. +export const keepsWork = (err: unknown) => /modified|untracked|uncommitted|contains/i.test(errorMessage(err)); + +// What a row says once its box answered an archive, and when git kept it. +export const archivedMessage = (res: { archive?: string }) => (res.archive ? "Archiving: its archive script runs, then it goes" : "Archived; its branch is kept"); +export const KEPT_DIRTY = "Kept: uncommitted changes"; + // removeResult reads a DELETE's answer. Older boxes sent the 202 of // an archive as text/plain, which the client hands back as a string. export function removeResult(res: unknown): { archive?: string } { diff --git a/app/src/lib/stop-job.test.ts b/app/src/lib/stop-job.test.ts new file mode 100644 index 000000000..0f4c9e087 --- /dev/null +++ b/app/src/lib/stop-job.test.ts @@ -0,0 +1,525 @@ +// node --experimental-strip-types --test src/lib/stop-job.test.ts (pnpm test) +import assert from "node:assert/strict"; +import { beforeEach, test } from "node:test"; + +import { KEPT_OTHER_AGENT } from "./archive-after-stop.ts"; +import { cancelJob, dismissJob, jobOf, useJobs } from "./jobs.ts"; +import { inTurn, markRemoving, removalOf, useRemovals } from "./removing.ts"; +import { reattachStopJobs, readStopRun, runStopJob, type StopDeps, type StopItem, type StopRuns, type StopRunParams, type StopRunStep, stopSummary } from "./stop-job.ts"; + +const row = (session: string, wt?: string, box = "devl"): StopItem => ({ + key: `${box}/${session}`, + box, + session, + label: wt ?? "shop", + where: { location: "shop", worktree: wt ? { name: wt, path: `/w/${wt}` } : { name: "shop", path: "/w/shop", main: true } }, +}); + +function fakeBoxes(over: Partial = {}) { + const calls: string[] = []; + const deps: StopDeps = { + stop: async (box, s) => void calls.push(`stop ${box}/${s}`), + liveAgents: async () => [], + archive: async (box, _loc, wt) => { + calls.push(`archive ${box}:${wt.name}`); + return {}; + }, + errorText: (err) => (err instanceof Error ? err.message : String(err)), + ...over, + }; + return { deps, calls }; +} + +// The real archive clears its mark when the box answers; the fakes don't. +beforeEach(() => useRemovals.setState({ byKey: {} })); + +const itemsOf = (id: number) => Object.fromEntries(jobOf(id)!.items.map((i) => [i.key, `${i.state}: ${i.message}`])); + +test("a clean run stops them all, archives their worktrees, and sums it up", async () => { + useJobs.setState({ jobs: [] }); + const { deps, calls } = fakeBoxes(); + const res = await runStopJob([row("a", "fix"), row("b", "perf"), row("c")], true, deps); + assert.equal(res.clean, true); + assert.equal(jobOf(res.job)!.title, "Stopping 3 agents"); + assert.equal(jobOf(res.job)!.summary, "3 stopped, 2 archived"); + assert.deepEqual(itemsOf(res.job), { + "devl/a": "done: Stopped · Archived; its branch is kept", + "devl/b": "done: Stopped · Archived; its branch is kept", + "devl/c": "done: Stopped", + }); + assert.deepEqual(calls.filter((c) => c.startsWith("archive")), ["archive devl:fix", "archive devl:perf"]); +}); + +test("without archiving, only the agents stop", async () => { + const { deps, calls } = fakeBoxes(); + const res = await runStopJob([row("a", "fix")], false, deps); + assert.equal(res.clean, true); + assert.equal(jobOf(res.job)!.summary, "1 stopped"); + assert.deepEqual(calls, ["stop devl/a"]); +}); + +test("kept and failed rows make a report, each saying why", async () => { + const { deps } = fakeBoxes({ + stop: async (_box, s) => { + if (s === "d") throw new Error("box went away"); + }, + liveAgents: async () => [{ dir: "/w/busy", name: "someone-else" }], + archive: async (_box, _loc, wt) => { + if (wt.name === "dirty") throw new Error("contains uncommitted changes"); + if (wt.name === "broken") throw new Error("permission denied"); + return wt.name === "scripted" ? { archive: "pnpm archive" } : {}; + }, + }); + const res = await runStopJob([row("a", "dirty"), row("b", "busy"), row("c", "broken"), row("d", "gone"), row("e", "scripted")], true, deps); + assert.equal(res.clean, false); + assert.deepEqual(itemsOf(res.job), { + "devl/a": "kept: Stopped · Kept: uncommitted changes", + "devl/b": `kept: Stopped · Kept: ${KEPT_OTHER_AGENT}`, + "devl/c": "failed: Stopped · Couldn't archive: permission denied", + "devl/d": "failed: Failed: box went away", + "devl/e": "done: Stopped · Archiving: its archive script runs, then it goes", + }); + assert.equal(jobOf(res.job)!.summary, "4 stopped, 1 archiving, 2 kept, 2 failed"); +}); + +test("Stop the rest sends nothing more; what was sent finishes", async () => { + let release!: () => void; + const hold = new Promise((r) => (release = r)); + const sent: string[] = []; + const { deps } = fakeBoxes({ + stop: async (_box, s) => { + sent.push(s); + await hold; + }, + }); + const rows = ["a", "b", "c", "d", "e", "f"].map((s) => row(s, `wt-${s}`)); + const running = runStopJob(rows, true, deps); + await new Promise((r) => setTimeout(r, 0)); + const id = useJobs.getState().jobs[0].id; + assert.deepEqual(sent, ["a", "b", "c"], "three at a time"); + cancelJob(id); + release(); + const res = await running; + assert.deepEqual(sent, ["a", "b", "c"]); + const items = itemsOf(res.job); + for (const s of ["a", "b", "c"]) assert.equal(items[`devl/${s}`], "cancelled: Stopped · Not archived"); + for (const s of ["d", "e", "f"]) assert.equal(items[`devl/${s}`], "cancelled: Not stopped"); + assert.equal(jobOf(res.job)!.summary, "3 stopped, 3 not stopped, 3 not archived"); + assert.equal(res.clean, false); +}); + +test("the job carries on in the store with nothing holding it", async () => { + useJobs.setState({ jobs: [] }); + const { deps } = fakeBoxes(); + // As the dialog does: start it and let go, with no view to keep it alive. + void runStopJob([row("a", "fix")], true, deps); + const id = useJobs.getState().jobs[0].id; + for (let i = 0; i < 20 && !jobOf(id)!.finished; i++) await new Promise((r) => setTimeout(r, 0)); + assert.equal(jobOf(id)!.summary, "1 stopped, 1 archived"); +}); + +test("the summary names what was kept, failed and never sent", () => { + assert.equal(stopSummary(17, 17, 0, Array(15).fill("archived")), "17 stopped, 15 archived"); + assert.equal(stopSummary(5, 3, 1, ["archived", "kept", "failed"]), "3 stopped, 1 archived, 1 kept, 2 failed, 1 not stopped"); +}); + +test("an agent started in a worktree while its archive waited its turn keeps it", async () => { + let listings = 0; + const { deps, calls } = fakeBoxes({ + // The first look finds nobody; by its turn someone has started there. + liveAgents: async () => (++listings === 1 ? [] : [{ dir: "/w/fix/app", name: "newcomer" }]), + }); + const res = await runStopJob([row("a", "fix")], true, deps); + assert.equal(itemsOf(res.job)["devl/a"], `kept: Stopped · Kept: ${KEPT_OTHER_AGENT}`); + assert.deepEqual(calls.filter((c) => c.startsWith("archive")), []); + assert.equal(removalOf(useRemovals.getState().byKey, "devl", "/w/fix"), undefined, "its mark goes with it"); +}); + +test("a worktree already on its way out isn't asked for twice", async () => { + markRemoving("devl", "/w/fix", "archive", "fix"); + const { deps, calls } = fakeBoxes(); + const res = await runStopJob([row("a", "fix"), row("b", "perf")], true, deps); + assert.deepEqual(calls.filter((c) => c.startsWith("archive")), ["archive devl:perf"]); + assert.equal(itemsOf(res.job)["devl/a"], "done: Stopped · Already being archived"); + assert.equal(res.clean, true); +}); + +test("while archives wait their turn they can still be held back", async () => { + // Another removal on devl holds the line. + let release!: () => void; + const busy = inTurn("devl", () => new Promise((r) => (release = r))); + const { deps, calls } = fakeBoxes(); + const running = runStopJob([row("a", "fix"), row("b", "perf")], true, deps); + for (let i = 0; i < 20 && jobOf(useJobs.getState().jobs[0].id)!.items.some((x) => x.state !== "queued"); i++) await new Promise((r) => setTimeout(r, 0)); + const id = useJobs.getState().jobs[0].id; + assert.deepEqual(itemsOf(id), { "devl/a": "queued: Stopped · waiting to archive", "devl/b": "queued: Stopped · waiting to archive" }); + // Marked, so Archive… from the sidebar leaves them be meanwhile. + assert.ok(removalOf(useRemovals.getState().byKey, "devl", "/w/fix")); + cancelJob(id); + release(); + await busy; + const res = await running; + assert.deepEqual(calls.filter((c) => c.startsWith("archive")), []); + assert.equal(itemsOf(res.job)["devl/a"], "cancelled: Stopped · Not archived"); + assert.equal(removalOf(useRemovals.getState().byKey, "devl", "/w/fix"), undefined); +}); + +test("Stop the rest pressed while an archive looks for agents still holds it back", async () => { + let listings = 0; + const { deps, calls } = fakeBoxes({ + // The second look, in its turn, is when Stop the rest is pressed. + liveAgents: async () => { + if (++listings === 2) cancelJob(useJobs.getState().jobs[0].id); + return []; + }, + }); + const res = await runStopJob([row("a", "fix")], true, deps); + assert.deepEqual(calls.filter((c) => c.startsWith("archive")), []); + assert.equal(itemsOf(res.job)["devl/a"], "cancelled: Stopped · Not archived"); + assert.equal(removalOf(useRemovals.getState().byKey, "devl", "/w/fix"), undefined); +}); + +// A box that runs the stop itself (internal/box/runs/templates/stop.json), +// faked: each run is read back as the box would build its steps. held keeps +// it on its first three stops until it's cancelled or let go. +// startsLost: answers lost after the box took the run; startsDropped: +// asks that never reached it. forgets: the box no longer has the run. +// findFails: asking for a run by its group goes unanswered too. +function runsBox(o: { onBox?: (box: string) => boolean; refuses?: string[]; startsLost?: number; startsDropped?: number; stopFails?: Record; archive?: Record>; held?: boolean; awayFor?: number; forgets?: boolean; findFails?: boolean } = {}) { + const started: { box: string; title: string; group: string; params: StopRunParams; idemKey: string }[] = []; + const asked: string[] = []; + let lost = 0; + let dropped = 0; + const cancelled = new Set(); + let reads = 0; + const step = (path: string, kind: string, s: Partial): StopRunStep => ({ path, kind, status: "succeeded", ...s }); + const runs: StopRuns = { + onBox: (box) => o.onBox?.(box) ?? true, + start: async (box, req) => { + asked.push(req.idemKey); + if (o.refuses?.includes(box)) throw Object.assign(new Error('template stop: no template "stop"'), { status: 400 }); + if (dropped++ < (o.startsDropped ?? 0)) throw Object.assign(new Error(`${box} is offline`), { status: 503 }); + // The run is taken (once per key), but the answer never comes back. + if (!started.some((s) => s.idemKey === req.idemKey)) started.push({ box, ...req }); + if (lost++ < (o.startsLost ?? 0)) throw Object.assign(new Error(`${box} is offline`), { status: 503 }); + return { id: `r${started.findIndex((s) => s.idemKey === req.idemKey) + 1}` }; + }, + find: async (box, group) => { + if (o.findFails) throw Object.assign(new Error(`${box} is offline`), { status: 503 }); + const k = started.findIndex((s) => s.box === box && s.group === group); + return k < 0 ? undefined : { id: `r${k + 1}` }; + }, + get: async (_box, id) => { + // The laptop asleep, or the box out of reach: no answer. + if (++reads <= (o.awayFor ?? 0)) throw Object.assign(new Error("devl is offline"), { status: 503 }); + if (o.forgets) throw Object.assign(new Error("no run with that id"), { status: 404 }); + const { params } = started[Number(id.slice(1)) - 1]; + const held = o.held && !cancelled.has(id); + const sent = held || cancelled.has(id) ? params.sessions.slice(0, 3) : params.sessions; + const stops = sent.map((s, k) => step(`0.i${k}`, "item", { children: [step(`0.i${k}.0`, "stop_session", o.stopFails?.[s] ? { status: "failed", error: o.stopFails[s] } : held ? { status: "running" } : { output: "stopped" })] })); + const archives = held || cancelled.has(id) ? [] : params.worktrees.map((w, j) => step(`1.i${j}`, "item", { children: [step(`1.i${j}.0`, "archive_worktree", o.archive?.[w.name] ?? { output: "Archived; its branch is kept" })] })); + const status = held ? "running" : cancelled.has(id) ? "cancelled" : "succeeded"; + return { status, params: { ...params }, steps: [step("0", "map", { status, children: stops }), ...(archives.length ? [step("1", "map", { children: archives })] : [])] }; + }, + // The box takes a moment to cancel: reads meanwhile still say running. + cancel: async (_box, id) => { + await new Promise((resolve) => setTimeout(resolve, 10)); + cancelled.add(id); + }, + }; + return { runs, started, asked, cancelled, release: () => (o.held = false), reachable: () => Object.assign(o, { startsLost: 0, startsDropped: 0, findFails: false }) }; +} + +test("a box with runs does its share as one run; a box without gets the app's loop", async () => { + const { runs, started } = runsBox({ onBox: (box) => box === "devl" }); + const { deps, calls } = fakeBoxes({ runs, every: 1 }); + const res = await runStopJob([row("a", "fix"), row("b", "fix"), row("c"), row("d", "ci", "gpu")], true, deps); + assert.deepEqual( + started.map((s) => [s.box, s.params]), + [["devl", { sessions: ["a", "b", "c"], worktrees: [{ location: "shop", name: "fix", path: "/w/fix", sessions: ["a", "b"] }] }]], + ); + assert.equal(started[0].group, jobOf(res.job)!.ref); + assert.deepEqual(calls, ["stop gpu/d", "archive gpu:ci"]); + assert.deepEqual(itemsOf(res.job), { + "devl/a": "done: Stopped · Archived; its branch is kept", + "devl/b": "done: Stopped · Archived; its branch is kept", + "devl/c": "done: Stopped", + "gpu/d": "done: Stopped · Archived; its branch is kept", + }); + assert.equal(jobOf(res.job)!.summary, "4 stopped, 2 archived"); + assert.equal(res.clean, true); +}); + +test("an archive the box failed shows the box's own words: a timeout is a failure, not kept", async () => { + const timedOut = "git worktree remove timed out after 10 min and was stopped; the folder may be partly removed, and archiving it again finishes it"; + const silent = "git worktree remove failed with no output (exit status 1)"; + const { runs } = runsBox({ archive: { slow: { status: "failed", error: timedOut }, mute: { status: "failed", error: silent } } }); + const { deps } = fakeBoxes({ runs, every: 1 }); + const res = await runStopJob([row("a", "slow"), row("b", "mute")], true, deps); + assert.deepEqual(itemsOf(res.job), { + "devl/a": `failed: Stopped · Couldn't archive: ${timedOut}`, + "devl/b": `failed: Stopped · Couldn't archive: ${silent}`, + }); + assert.equal(jobOf(res.job)!.summary, "2 stopped, 2 failed"); +}); + +test("a box offline at the start, or from before stop runs, gets the app's loop for its rows", async () => { + const { runs, started } = runsBox({ onBox: (box) => box !== "gpu", refuses: ["devl"] }); + const { deps, calls } = fakeBoxes({ runs, every: 1 }); + const res = await runStopJob([row("a", "fix"), row("b", "ci", "gpu"), row("c", "eval", "mini")], true, deps); + assert.deepEqual( + started.map((s) => s.box), + ["mini"], + ); + assert.deepEqual(calls.sort(), ["archive devl:fix", "archive gpu:ci", "stop devl/a", "stop gpu/b"]); + assert.equal(jobOf(res.job)!.summary, "3 stopped, 3 archived"); +}); + +test("a start whose answer was lost is asked again under the same key: one run, followed, nothing done twice", async () => { + const { runs, started, asked } = runsBox({ startsLost: 2 }); + const { deps, calls } = fakeBoxes({ runs, every: 1 }); + const res = await runStopJob([row("a", "fix")], true, deps); + assert.equal(asked.length, 3); + assert.equal(new Set(asked).size, 1, "one idempotency key"); + assert.equal(started.length, 1); + assert.deepEqual(calls, [], "the app didn't do it as well"); + assert.equal(jobOf(res.job)!.summary, "1 stopped, 1 archived"); +}); + +test("a box that doesn't answer the start is asked until it does: its rows wait for it, and nothing fails", async () => { + const { runs, started, asked } = runsBox({ startsDropped: 6 }); + const { deps, calls } = fakeBoxes({ runs, every: 1 }); + const running = runStopJob([row("a", "fix")], true, deps); + for (let i = 0; i < 100 && asked.length < 2; i++) await new Promise((r) => setTimeout(r, 1)); + assert.equal(itemsOf(useJobs.getState().jobs[0].id)["devl/a"], "queued: Waiting for devl"); + const res = await running; + assert.equal(asked.length, 7); + assert.equal(started.length, 1); + assert.deepEqual(calls, [], "the app didn't do it as well"); + assert.equal(jobOf(res.job)!.summary, "1 stopped, 1 archived"); +}); + +test("Stop the rest while a start goes unanswered: a run the box took is cancelled there, and with none nothing is done", async () => { + // The box took it, but every answer was lost. + const took = runsBox({ startsLost: 1000, held: true }); + const a = fakeBoxes({ runs: took.runs, every: 1 }); + const first = runStopJob([row("a", "fix")], true, a.deps); + for (let i = 0; i < 100 && took.asked.length < 2; i++) await new Promise((r) => setTimeout(r, 1)); + cancelJob(useJobs.getState().jobs[0].id); + const r1 = await first; + assert.deepEqual([...took.cancelled], ["r1"], "its run is found by its group, and cancelled"); + assert.deepEqual(a.calls, []); + assert.equal(itemsOf(r1.job)["devl/a"], "cancelled: Stopped · Not archived"); + + // No ask ever reached the box. + const never = runsBox({ startsDropped: 1000 }); + const b = fakeBoxes({ runs: never.runs, every: 1 }); + const second = runStopJob([row("b", "perf")], true, b.deps); + for (let i = 0; i < 100 && never.asked.length < 2; i++) await new Promise((r) => setTimeout(r, 1)); + cancelJob(useJobs.getState().jobs[0].id); + const r2 = await second; + assert.equal(never.started.length, 0); + assert.deepEqual(b.calls, []); + assert.equal(itemsOf(r2.job)["devl/b"], "cancelled: Not stopped"); + assert.equal(jobOf(r2.job)!.summary, "0 stopped, 1 not stopped"); +}); + +test("a run its box no longer has fails its rows and lets go of their Archiving… marks", async () => { + const { runs } = runsBox({ forgets: true }); + const { deps } = fakeBoxes({ runs, every: 1 }); + const res = await runStopJob([row("a", "fix")], true, deps); + assert.equal(itemsOf(res.job)["devl/a"], "failed: Failed: devl no longer has this run"); + assert.equal(removalOf(useRemovals.getState().byKey, "devl", "/w/fix"), undefined); +}); + +test("an archive whose script runs holds its box's line until the box says the worktree went", async () => { + const order: string[] = []; + let gone: (how: "gone") => void = () => {}; + const { deps } = fakeBoxes({ + archive: async (_box, _loc, wt) => { + order.push(`archive ${wt.name}`); + return { archive: "pnpm archive" }; + }, + settled: (_box, path) => + new Promise((resolve) => { + order.push(`script ${path}`); + gone = resolve; + }), + }); + const running = runStopJob([row("a", "fix"), row("b", "perf")], true, deps); + for (let i = 0; i < 100 && order.length < 2; i++) await new Promise((r) => setTimeout(r, 1)); + await new Promise((r) => setTimeout(r, 20)); + assert.deepEqual(order, ["archive fix", "script /w/fix"], "perf waits for fix's script"); + gone("gone"); + for (let i = 0; i < 100 && order.length < 4; i++) await new Promise((r) => setTimeout(r, 1)); + gone("gone"); + const res = await running; + assert.deepEqual(order, ["archive fix", "script /w/fix", "archive perf", "script /w/perf"]); + assert.equal(jobOf(res.job)!.summary, "2 stopped, 2 archived"); +}); + +test("the run's steps become the rows, in the app's words, and a step not reached yet waits", () => { + const params: StopRunParams = { + sessions: ["a", "b", "c", "d", "e", "f"], + worktrees: [ + { location: "shop", name: "dirty", path: "/w/dirty", sessions: ["a"] }, + { location: "shop", name: "busy", path: "/w/busy", sessions: ["b"] }, + { location: "shop", name: "held", path: "/w/held", sessions: ["c", "d"] }, + { location: "shop", name: "scripted", path: "/w/scripted", sessions: ["e"] }, + { location: "shop", name: "next", path: "/w/next", sessions: ["f"] }, + ], + }; + const leaf = (path: string, status: string, more: Partial = {}): StopRunStep => ({ path: path.split(".").slice(0, 2).join("."), kind: "item", status, children: [{ path, kind: "leaf", status, ...more }] }); + const read = readStopRun({ + status: "running", + params: { ...params }, + steps: [ + { path: "0", kind: "map", status: "failed", children: ["a", "b", "c", "e", "f"].map((s) => leaf(`0.i${params.sessions.indexOf(s)}.0`, "succeeded")).concat(leaf("0.i3.0", "failed", { error: "a before:session.stop hook stopped it" })) }, + { + path: "1", + kind: "map", + status: "running", + children: [ + leaf("1.i0.0", "skipped", { output: "Kept: uncommitted changes" }), + leaf("1.i1.0", "skipped", { output: `Kept: ${KEPT_OTHER_AGENT}` }), + leaf("1.i2.0", "skipped", { output: "Kept: an agent there didn't stop" }), + leaf("1.i3.0", "running"), + ], + }, + ], + }); + assert.deepEqual(read.rows, { + a: { state: "kept", message: "Stopped · Kept: uncommitted changes" }, + b: { state: "kept", message: `Stopped · Kept: ${KEPT_OTHER_AGENT}` }, + c: { state: "kept", message: "Stopped · Kept: an agent there didn't stop" }, + d: { state: "failed", message: "Failed: a before:session.stop hook stopped it" }, + e: { state: "running", message: "Archiving…" }, + f: { state: "queued", message: "Stopped · waiting to archive" }, + }); + assert.deepEqual(read.worktrees, { "/w/dirty": "kept", "/w/busy": "kept", "/w/held": "kept" }); + assert.equal(read.over, false); +}); + +test("the laptop losing touch with the box fails nothing: the job waits for the box to say", async () => { + const { runs } = runsBox({ awayFor: 3, archive: { fix: { status: "succeeded", output: "Archiving: its archive script runs, then it goes" } } }); + const { deps } = fakeBoxes({ runs, every: 1 }); + const res = await runStopJob([row("a", "fix"), row("b")], true, deps); + assert.deepEqual(itemsOf(res.job), { "devl/a": "done: Stopped · Archiving: its archive script runs, then it goes", "devl/b": "done: Stopped" }); + assert.equal(jobOf(res.job)!.summary, "2 stopped, 1 archiving"); +}); + +test("Stop the rest cancels the box's run: what it began finishes there, the rest isn't done", async () => { + const { runs, cancelled } = runsBox({ held: true }); + const { deps, calls } = fakeBoxes({ runs, every: 1 }); + const rows = ["a", "b", "c", "d", "e", "f"].map((s) => row(s, `wt-${s}`)); + const running = runStopJob(rows, true, deps); + for (let i = 0; i < 50 && jobOf(useJobs.getState().jobs[0].id)!.items[0].state !== "running"; i++) await new Promise((r) => setTimeout(r, 1)); + cancelJob(useJobs.getState().jobs[0].id); + const res = await running; + assert.deepEqual([...cancelled], ["r1"]); + assert.deepEqual(calls, [], "the app sent nothing itself"); + const items = itemsOf(res.job); + for (const s of ["a", "b", "c"]) assert.equal(items[`devl/${s}`], "cancelled: Stopped · Not archived"); + for (const s of ["d", "e", "f"]) assert.equal(items[`devl/${s}`], "cancelled: Not stopped"); + assert.equal(jobOf(res.job)!.summary, "3 stopped, 3 not stopped, 3 not archived"); + for (const s of ["a", "d"]) assert.equal(removalOf(useRemovals.getState().byKey, "devl", `/w/wt-${s}`), undefined, "the marks go with them"); +}); + +test("after a relaunch the job comes back from the box's runs, with how they ended, until it's dismissed", async () => { + const kept = new Map(); + Object.assign(globalThis, { localStorage: { getItem: (k: string) => kept.get(k) ?? null, setItem: (k: string, v: string) => void kept.set(k, v) } }); + try { + useJobs.setState({ jobs: [] }); + const box = runsBox({ held: true }); + const { deps } = fakeBoxes({ runs: box.runs, every: 1 }); + const first = runStopJob([row("a", "fix"), row("b", "perf")], true, deps); + for (let i = 0; i < 50 && !kept.get("berth.jobs.stopRuns"); i++) await new Promise((r) => setTimeout(r, 1)); + // The app quits mid-run; the box carries on and finishes. + useJobs.setState({ jobs: [] }); + await first; + box.release(); + + await reattachStopJobs({ runs: box.runs, every: 1 }); + const [job] = useJobs.getState().jobs; + assert.equal(job.title, "Stopping 2 agents"); + assert.equal(job.summary, "2 stopped, 2 archived"); + assert.deepEqual(itemsOf(job.id), { "devl/a": "done: Stopped · Archived; its branch is kept", "devl/b": "done: Stopped · Archived; its branch is kept" }); + // Brought back once, however often the app reconnects. + await reattachStopJobs({ runs: box.runs, every: 1 }); + assert.equal(useJobs.getState().jobs.length, 1); + dismissJob(job.id); + assert.deepEqual(JSON.parse(kept.get("berth.jobs.stopRuns")!), []); + } finally { + Reflect.deleteProperty(globalThis, "localStorage"); + } +}); + +test("a job is kept before its start is sent, so one the app quit on before the box answered is asked for again after a relaunch", async () => { + const kept = new Map(); + Object.assign(globalThis, { localStorage: { getItem: (k: string) => kept.get(k) ?? null, setItem: (k: string, v: string) => void kept.set(k, v) } }); + try { + useJobs.setState({ jobs: [] }); + const box = runsBox({ startsDropped: 1000 }); + const { deps } = fakeBoxes({ runs: box.runs, every: 1 }); + const first = runStopJob([row("a", "fix")], true, deps); + for (let i = 0; i < 100 && box.asked.length < 2; i++) await new Promise((r) => setTimeout(r, 1)); + const [saved] = JSON.parse(kept.get("berth.jobs.stopRuns")!); + assert.equal(saved.runs[0].id, undefined, "kept with no run yet"); + assert.deepEqual(saved.runs[0].params.sessions, ["a"]); + // The app quits while the box is out of reach. + useJobs.setState({ jobs: [] }); + await first; + + // Back, the box answers: asked again under the same key, it takes it. + const back = runsBox(); + await reattachStopJobs({ runs: back.runs, every: 1 }); + assert.equal(back.started.length, 1); + assert.equal(back.started[0].idemKey, box.asked[0], "the same key"); + const [job] = useJobs.getState().jobs; + assert.equal(job.summary, "1 stopped, 1 archived"); + assert.equal(JSON.parse(kept.get("berth.jobs.stopRuns")!)[0].runs[0].id, "r1"); + dismissJob(job.id); + } finally { + Reflect.deleteProperty(globalThis, "localStorage"); + } +}); + +test("Stop the rest while the box is out of reach is kept: after a relaunch its run is only looked up and cancelled, never started", async () => { + const kept = new Map(); + Object.assign(globalThis, { localStorage: { getItem: (k: string) => kept.get(k) ?? null, setItem: (k: string, v: string) => void kept.set(k, v) } }); + try { + for (const tookIt of [true, false]) { + kept.clear(); + useJobs.setState({ jobs: [] }); + // The box took the first ask (its answer lost), or never got one; and + // it answers nothing, a lookup included, until the relaunch. + const box = runsBox(tookIt ? { startsLost: 1000, findFails: true, held: true } : { startsDropped: 1000, findFails: true }); + const { deps, calls } = fakeBoxes({ runs: box.runs, every: 1 }); + const first = runStopJob([row("a", "fix")], true, deps); + for (let i = 0; i < 100 && box.asked.length < 2; i++) await new Promise((r) => setTimeout(r, 1)); + cancelJob(useJobs.getState().jobs[0].id); + const r1 = await first; + assert.equal(itemsOf(r1.job)["devl/a"], "failed: Failed: devl didn't answer, so it may still stop it"); + assert.equal(JSON.parse(kept.get("berth.jobs.stopRuns")!)[0].cancelled, true, "the cancel is kept"); + const sent = box.asked.length; + + // Relaunched, the box back. + useJobs.setState({ jobs: [] }); + box.reachable(); + await reattachStopJobs({ runs: box.runs, every: 1 }); + assert.equal(box.asked.length, sent, "no start is sent again"); + assert.deepEqual(calls, []); + const [job] = useJobs.getState().jobs; + if (tookIt) { + assert.deepEqual([...box.cancelled], ["r1"], "the run it took is cancelled"); + assert.equal(itemsOf(job.id)["devl/a"], "cancelled: Stopped · Not archived"); + } else { + assert.equal(box.started.length, 0); + assert.equal(itemsOf(job.id)["devl/a"], "cancelled: Not stopped"); + } + dismissJob(job.id); + } + } finally { + Reflect.deleteProperty(globalThis, "localStorage"); + } +}); diff --git a/app/src/lib/stop-job.ts b/app/src/lib/stop-job.ts new file mode 100644 index 000000000..4ccbd5c0d --- /dev/null +++ b/app/src/lib/stop-job.ts @@ -0,0 +1,577 @@ +// The board's bulk Stop as a background job (lib/jobs.ts). A box with runs +// does its share itself, as one stop run (internal/box/runs/templates/ +// stop.json): it stops the agents a few at a time, then archives their +// worktrees where it's safe, one at a time, and keeps going while the +// laptop sleeps. The job only follows the run: a row fails when the box +// says so, never because the laptop lost touch. It is kept before the +// start is sent, so it comes back after a relaunch (reattachStopJobs), and +// a start whose answer never came is asked again or looked up by its group +// (startStopRun). A box without runs, or offline when +// it starts, gets the app's own loop for its rows: the agents stopped a few +// at a time, then their worktrees archived where it's safe +// (lib/archive-after-stop.ts), one at a time on each box in the same line +// as every other removal there (inTurn). "Stop the rest" sends nothing +// more: rows not yet stopped say "Not stopped", worktrees not yet archived +// are left as they are. +// +// No imports from "@/": pnpm test runs this file in plain node. The calls to +// the boxes come in as deps (lib/stop-on-box.ts, views/dashboard/stop.tsx). + +import { planArchive, type StopTarget, type WorktreeTarget } from "./archive-after-stop.ts"; +import { plural } from "./format.ts"; +import { cancelJob, finishJob, type JobItem, isCancelling, jobOf, onCancel, onDismiss, outcomeOf, runPool, setItem, startJob, useJobs } from "./jobs.ts"; +import { archivedMessage, clearRemoving, inTurn, KEPT_DIRTY, keepsWork, markRemoving, markScript, removalKey, removalOf, useRemovals } from "./removing.ts"; +import { load, save } from "./storage.ts"; + +export interface StopItem extends StopTarget { + session: string; + // What the tray calls it: its place, and its name when the place is shared. + label: string; + aside?: string; +} + +// A stop run as GET /v1/runs/{id} has it, as far as the job reads it. +export interface StopRunStep { + path: string; + kind: string; + status: string; + output?: string; + error?: string; + children?: StopRunStep[]; +} + +export interface StopRun { + status: string; + error?: string; + params?: Record; + steps: StopRunStep[]; +} + +// The stop template's parameters: the sessions, and the worktrees to archive +// once they stopped, each with the batch's sessions that worked there. +export interface StopRunParams { + sessions: string[]; + worktrees: { location: string; name: string; path: string; sessions: string[] }[]; +} + +// StopRuns reach the boxes' runs (lib/stop-on-box.ts). +export interface StopRuns { + // Whether the box does it itself: it has runs and is online. + onBox(box: string): boolean; + // idemKey makes asking again after an answer that never came safe. + start(box: string, req: { title: string; group: string; params: StopRunParams; idemKey: string }): Promise<{ id: string }>; + // The run started under group, when the box has one. + find(box: string, group: string): Promise<{ id: string } | undefined>; + get(box: string, id: string): Promise; + cancel(box: string, id: string): Promise; +} + +export interface StopDeps { + stop(box: string, session: string): Promise; + // The agent sessions still live on a box, fresh, or undefined when it + // doesn't answer. + liveAgents(box: string): Promise<{ dir: string; name: string }[] | undefined>; + archive(box: string, location: string, wt: { name: string; path: string }): Promise<{ archive?: string }>; + // How an archive whose script runs on the box ended, once it says: + // gone, failed, or undefined when it can't tell. The box's line waits for + // it, so its archive scripts never run side by side. + settled?(box: string, path: string): Promise<"gone" | "failed" | undefined>; + errorText(err: unknown): string; + // The boxes that do it themselves; without it the app does it all. + runs?: StopRuns; + // How often a run is read, in ms. + every?: number; +} + +export type WorktreeOutcome = "archived" | "archiving" | "kept" | "failed" | "cancelled"; + +export interface StopResult { + job: number; + // By worktree (removalKey). + worktrees: Record; + clean: boolean; +} + +// What one part of a job did, for its summary. +interface Tally { + stopped: number; + unsent: number; + // By worktree (removalKey). + worktrees: Record; +} + +// A few at a time, so a long list is quick without every request landing on +// a box at once. +const CONCURRENCY = 3; +// A run's map takes at most 64 items (internal/box/runs/exec.go): a box +// asked for more does it the app's way. +const MAX_ITEMS = 64; +const ARCHIVING = archivedMessage({ archive: "yes" }); + +const willArchive = (archive: boolean) => (r: StopItem) => archive && !!r.where && !r.where.worktree.main; + +export const fitsRun = (p: StopRunParams) => p.sessions.length <= MAX_ITEMS && p.worktrees.length <= MAX_ITEMS; + +export async function runStopJob(rows: StopItem[], archive: boolean, deps: StopDeps): Promise { + const title = `Stopping ${plural(rows.length, "agent")}`; + const ref = `stop-${Date.now().toString(36)}${Math.random().toString(36).slice(2, 6)}`; + const items = rows.map((r) => ({ key: r.key, group: r.box, label: r.label, aside: r.aside, message: "Waiting to stop" })); + const job = startJob({ title, items, cancellable: true, ref }); + + const byBox = new Map(); + for (const r of rows) byBox.set(r.box, [...(byBox.get(r.box) ?? []), r]); + // Each box with runs takes its share as one run; the rest is the app's. + const shares = [...byBox].map(([box, mine]) => ({ box, mine, params: stopParams(mine, archive) })); + const onBox = shares.filter((sh) => deps.runs?.onBox(sh.box) && fitsRun(sh.params)); + // Kept before anything is sent, so a relaunch before a box answers still + // finds its run, or asks for it again (reattachStopJobs). + if (onBox.length) { + const boxes = new Set(onBox.map((sh) => sh.box)); + remember({ ref, title, started: Date.now(), runs: onBox.map(({ box, params }) => ({ box, params })), items: items.filter((i) => boxes.has(i.group)) }); + } + const local = shares.filter((sh) => !onBox.includes(sh)).flatMap((sh) => sh.mine); + const tallies = await Promise.all([ + ...onBox.map(async ({ box, mine, params }) => { + const t = await onTheBox(job, { ref, title, box, params, keys: mine.map((r) => r.key) }, deps.runs!, deps.every); + // Refused (a box from before stop runs): nothing was sent, so the app + // does it. + return t ?? appStop(job, mine, archive, deps); + }), + local.length ? appStop(job, local, archive, deps) : undefined, + ]); + const worktrees: Record = Object.assign({}, ...tallies.map((t) => t?.worktrees ?? {})); + const sum = (k: "stopped" | "unsent") => tallies.reduce((n, t) => n + (t?.[k] ?? 0), 0); + finishJob(job, stopSummary(rows.length, sum("stopped"), sum("unsent"), Object.values(worktrees))); + const done = jobOf(job); + return { job, worktrees, clean: !!done && outcomeOf(done) === "clean" }; +} + +// onTheBox has box do one share of a job as a stop run and follows it, its +// rows (keys) waiting for the box until it answers. Undefined is the box +// refusing it: nothing was sent. +async function onTheBox(job: number, sh: { ref: string; title: string; box: string; params: StopRunParams; keys: string[]; asked?: boolean }, runs: StopRuns, every?: number): Promise { + const { ref, box, params, keys } = sh; + const started = await startStopRun(runs, box, { title: sh.title, group: ref, params }, { + every, + asked: sh.asked, + // A job that left the tray (the app quitting) asks no more; it stays + // kept for the next start. + cancelled: () => isCancelling(job) || !jobOf(job), + waiting: () => { + for (const k of keys) if (jobOf(job)?.items.find((i) => i.key === k)?.state === "queued") setItem(job, k, { message: `Waiting for ${box}` }); + }, + }); + const none: Tally = { stopped: 0, unsent: 0, worktrees: {} }; + if (!jobOf(job)) return none; + if (started === "refused") { + keptRun(ref, box, undefined); + return undefined; + } + if (started === "unsent") { + keptRun(ref, box, undefined); + for (const k of keys) setItem(job, k, { state: "cancelled", message: "Not stopped" }); + return { ...none, unsent: keys.length }; + } + if (started === "unknown") { + for (const k of keys) setItem(job, k, { state: "failed", message: `Failed: ${box} didn't answer, so it may still stop it` }); + return none; + } + keptRun(ref, box, started.id); + for (const w of params.worktrees) if (!removalOf(useRemovals.getState().byKey, box, w.path)) markRemoving(box, w.path, "archive", w.name); + return followJobRun(job, box, started.id, runs, every, params); +} + +// What became of asking a box to start a stop run: its id; refused (4xx: a +// box from before stop runs), so nothing was sent and the app does it; +// unsent, cancelled before the box had it; or unknown, cancelled while the +// box never answered (the share stays kept, cancelled, for the next start to +// look up again). +export type Started = { id: string } | "refused" | "unsent" | "unknown"; + +// Asked again no slower than this, while a box doesn't answer. +const MAX_RETRY_MS = 30_000; + +// startStopRun asks box to take its share as one stop run. A box that +// doesn't answer is asked again, under the same idempotency key, until it +// does: a run whose answer was lost is neither started twice nor given up +// on, and its rows never fail for the laptop losing touch. waiting hears +// each time it goes unanswered. Once cancelled says so it asks no more; a +// box asked before then is asked whether it has the run (its group), so a +// run it took is followed, and cancelled, rather than lost. asked says it +// may have been asked before (a share brought back after a relaunch), so a +// cancelled one is looked up and never sent. +export async function startStopRun(runs: StopRuns, box: string, req: { title: string; group: string; params: StopRunParams }, o: { every?: number; asked?: boolean; cancelled?(): boolean; waiting?(err: unknown): void } = {}): Promise { + const every = o.every ?? 1500; + for (let tries = 0; ; tries++) { + if (o.cancelled?.()) return tries || o.asked ? lookUp(runs, box, req.group, every) : "unsent"; + try { + return { id: (await runs.start(box, { ...req, idemKey: `${req.group}/${box}` })).id }; + } catch (err) { + const status = (err as { status?: number }).status ?? 0; + if (status >= 400 && status < 500) return "refused"; + o.waiting?.(err); + await sleep(Math.min(every * 2 * (tries + 1), MAX_RETRY_MS)); + } + } +} + +// lookUp asks box for the run started under group: a few times, as the +// user is waiting on Stop the rest. +async function lookUp(runs: StopRuns, box: string, group: string, every: number): Promise { + for (let tries = 1; ; tries++) { + try { + const run = await runs.find(box, group); + return run ? { id: run.id } : "unsent"; + } catch { + if (tries >= 3) return "unknown"; + await sleep(every * 2); + } + } +} + +// stopParams is one box's share as the stop template takes it: its +// sessions, and, when archiving, the worktrees they worked in, each once. +export function stopParams(rows: StopItem[], archive: boolean): StopRunParams { + const byPath = new Map(); + for (const r of rows.filter(willArchive(archive))) { + const wt = r.where!.worktree; + const w = byPath.get(wt.path) ?? { location: r.where!.location, name: wt.name, path: wt.path, sessions: [] }; + w.sessions.push(r.session); + byPath.set(wt.path, w); + } + return { sessions: rows.map((r) => r.session), worktrees: [...byPath.values()] }; +} + +export interface StopRead { + // By session: what its row says. + rows: Record>; + // By worktree path, once the box says how its archive went, and in what + // words: "Kept: uncommitted changes". + worktrees: Record; + said: Record; + stopped: number; + unsent: number; + over: boolean; +} + +const stepAt = (steps: StopRunStep[] | undefined, path: string): StopRunStep | undefined => { + for (const s of steps ?? []) { + if (s.path === path) return s; + if (path.startsWith(`${s.path}.`)) return stepAt(s.children, path); + } + return undefined; +}; + +const ended = (status: string) => status === "succeeded" || status === "failed" || status === "cancelled" || status === "interrupted"; + +// readStopRun reads a stop run's steps into its rows, in the words the +// app's own loop uses. Session k's stop is step 0.ik.0, worktree j's +// archive 1.ij.0. A step the run never got to is waiting while it goes, +// and, once it's over, not done (cancelled) or failed with the run. +export function readStopRun(run: StopRun): StopRead { + const sessions = ((run.params?.sessions ?? []) as (string | { session: string })[]).map((s) => (typeof s === "string" ? s : s.session)); + const worktrees = (run.params?.worktrees ?? []) as StopRunParams["worktrees"]; + const over = ended(run.status); + const cancelled = run.status === "cancelled"; + const why = (st?: StopRunStep) => st?.error || run.error || "the box didn't say why"; + const read: StopRead = { rows: {}, worktrees: {}, said: {}, stopped: 0, unsent: 0, over }; + const stopped = new Set(); + sessions.forEach((s, k) => { + const st = stepAt(run.steps, `0.i${k}.0`); + const status = st?.status ?? (!over ? "queued" : cancelled ? "cancelled" : "failed"); + if (status === "succeeded" || status === "skipped") { + stopped.add(s); + read.stopped++; + read.rows[s] = { state: "done", message: "Stopped" }; + } else if (status === "running" || status === "queued") { + read.rows[s] = status === "running" ? { state: "running", message: "Stopping…" } : { state: "queued", message: "Waiting to stop" }; + } else if (status === "cancelled") { + read.unsent++; + read.rows[s] = { state: "cancelled", message: "Not stopped" }; + } else { + read.rows[s] = { state: "failed", message: `Failed: ${why(st)}` }; + } + }); + worktrees.forEach((w, j) => { + const mine = w.sessions.filter((s) => stopped.has(s)); + // Every stop there failed: those rows say so, and nothing there changed. + if (w.sessions.length && !mine.length) return; + const st = stepAt(run.steps, `1.i${j}.0`); + const status = st?.status ?? (!over ? "queued" : cancelled ? "cancelled" : "failed"); + let outcome: WorktreeOutcome | undefined; + let row: Pick; + if (status === "queued") row = { state: "queued", message: "Stopped · waiting to archive" }; + else if (status === "running") row = { state: "running", message: "Archiving…" }; + else if (status === "succeeded") { + outcome = st?.output === ARCHIVING ? "archiving" : "archived"; + row = { state: "done", message: `Stopped · ${outcome === "archiving" ? ARCHIVING : archivedMessage({})}` }; + } else if (status === "skipped") { + outcome = "kept"; + row = { state: "kept", message: `Stopped · ${st?.output || "Kept"}` }; + } else if (status === "cancelled") { + outcome = "cancelled"; + row = { state: "cancelled", message: "Stopped · Not archived" }; + } else { + outcome = "failed"; + row = { state: "failed", message: `Stopped · Couldn't archive: ${why(st)}` }; + } + if (outcome) { + read.worktrees[w.path] = outcome; + read.said[w.path] = row.message!.replace(/^Stopped · /, ""); + } + for (const s of mine) read.rows[s] = row; + }); + return read; +} + +const sleep = (ms: number) => new Promise((res) => setTimeout(res, ms)); + +// followStopRun reads a box's stop run until it's over, telling onRead +// each time; onRead answering false stops following. A box that doesn't +// answer is asked again, so the laptop sleeping or losing touch fails +// nothing. Once cancelled() says so, the box is asked to stop the rest: the +// step in flight finishes there. Undefined is a run the box no longer has. +export async function followStopRun(runs: StopRuns, box: string, id: string, o: { every?: number; onRead(read: StopRead, run: StopRun): void | false; cancelled?(): boolean }): Promise { + const every = o.every ?? 1500; + let asked = false; + for (;;) { + if (!asked && o.cancelled?.()) { + asked = true; + runs.cancel(box, id).catch(() => (asked = false)); + } + let run: StopRun; + try { + run = await runs.get(box, id); + } catch (err) { + if ((err as { status?: number }).status === 404) return undefined; + await sleep(every * 2); + continue; + } + const read = readStopRun(run); + if (o.onRead(read, run) === false || read.over) return read; + await sleep(every); + } +} + +// followJobRun keeps a job's rows on box in step with its stop run. +async function followJobRun(job: number, box: string, id: string, runs: StopRuns, every?: number, params?: StopRunParams): Promise { + const read = await followStopRun(runs, box, id, { + every, + cancelled: () => isCancelling(job), + onRead: (r, run) => { + if (!jobOf(job)) return false; + for (const [s, item] of Object.entries(r.rows)) setItem(job, `${box}/${s}`, item); + for (const w of (run.params?.worktrees ?? []) as StopRunParams["worktrees"]) { + // Archived, its worktree.removed clears the mark; one the run left + // as it was, or never got to, is unmarked here. + const o = r.worktrees[w.path]; + if (o === "archiving") markScript(box, w.path); + else if (o ? o !== "archived" : r.over) clearRemoving(box, w.path); + } + }, + }); + if (!read) { + for (const i of jobOf(job)?.items ?? []) if (i.group === box && i.state !== "done") setItem(job, i.key, { state: "failed", message: `Failed: ${box} no longer has this run` }); + // Its worktrees were marked Archiving… for it: no word will come now. + for (const w of params?.worktrees ?? []) clearRemoving(box, w.path); + return { stopped: 0, unsent: 0, worktrees: {} }; + } + const worktrees: Record = {}; + for (const [path, o] of Object.entries(read.worktrees)) worktrees[removalKey(box, path)] = o; + return { stopped: read.stopped, unsent: read.unsent, worktrees }; +} + +// appStop is the app's own loop, for the rows of boxes that don't run it. +async function appStop(job: number, rows: StopItem[], archive: boolean, deps: StopDeps): Promise { + const stopped = new Set(); + const unsent = await runPool( + rows, + CONCURRENCY, + async (r) => { + setItem(job, r.key, { state: "running", message: "Stopping…" }); + try { + await deps.stop(r.box, r.session); + stopped.add(r.key); + setItem(job, r.key, willArchive(archive)(r) ? { state: "running", message: "Stopped" } : { state: "done", message: "Stopped" }); + } catch (err) { + setItem(job, r.key, { state: "failed", message: `Failed: ${deps.errorText(err)}` }); + } + }, + () => isCancelling(job), + ); + for (const r of unsent) setItem(job, r.key, { state: "cancelled", message: "Not stopped" }); + + const worktrees: Record = {}; + const candidates = rows.filter(willArchive(archive)); + if (candidates.length) { + // Only a worktree's stopped rows: one kept because a stop failed there + // also has rows that say Failed. + const setRows = (w: WorktreeTarget, item: Pick) => { + for (const k of w.rows) if (stopped.has(k)) setItem(job, k, item); + }; + const report = (w: WorktreeTarget, o: WorktreeOutcome, message: string) => { + worktrees[w.key] = o; + setRows(w, { state: o === "archived" || o === "archiving" ? "done" : o, message: `Stopped · ${message}` }); + }; + // planArchive over the box's own listing, fresh, so an agent started a + // moment ago counts. A box that doesn't answer keeps its worktrees. + const check = async (targets: StopItem[], box?: string) => { + const boxes = box ? [box] : [...new Set(targets.map((r) => r.box))]; + const listed = await Promise.all(boxes.map(async (b) => ({ box: b, agents: await deps.liveAgents(b).catch(() => undefined) }))); + const live = listed.flatMap(({ box: b, agents }) => (agents ?? []).map((a) => ({ box: b, ...a }))); + return planArchive(targets, stopped, live, new Set(listed.filter((l) => !l.agents).map((l) => l.box))); + }; + const plan = await check(candidates); + for (const k of plan.kept) report(k, "kept", `Kept: ${k.why}`); + const queued = plan.archive.filter((w) => { + // Already on its way out (Archive… from the sidebar): its box needn't + // hear it twice. + if (removalOf(useRemovals.getState().byKey, w.box, w.wt.path)) { + report(w, "archiving", "Already being archived"); + return false; + } + // Marked from now, so nothing else asks for it while it waits its turn. + markRemoving(w.box, w.wt.path, "archive", w.wt.name); + // Queued: not sent, so Stop the rest can still hold it back. + setRows(w, { state: "queued", message: "Stopped · waiting to archive" }); + return true; + }); + await Promise.all( + queued.map((w) => + inTurn(w.box, async () => { + const leave = (o: WorktreeOutcome, message: string) => { + clearRemoving(w.box, w.wt.path); + report(w, o, message); + }; + if (isCancelling(job)) return leave("cancelled", "Not archived"); + setRows(w, { state: "running", message: "Archiving…" }); + // Its turn may come minutes later: look again for an agent + // started there meanwhile. + const again = await check( + candidates.filter((r) => w.rows.includes(r.key)), + w.box, + ); + if (again.kept.length) return leave("kept", `Kept: ${again.kept[0].why}`); + // Stop the rest may have come while it looked. + if (isCancelling(job)) return leave("cancelled", "Not archived"); + try { + const res = await deps.archive(w.box, w.location, w.wt); + report(w, res.archive ? "archiving" : "archived", archivedMessage(res)); + if (res.archive && deps.settled) { + const how = await deps.settled(w.box, w.wt.path); + if (how === "gone") report(w, "archived", archivedMessage({})); + else if (how === "failed") report(w, "failed", "Couldn't archive: its archive script failed"); + } + } catch (err) { + if (keepsWork(err)) leave("kept", KEPT_DIRTY); + else leave("failed", `Couldn't archive: ${deps.errorText(err)}`); + } + }), + ), + ); + } + return { stopped: stopped.size, unsent: unsent.length, worktrees }; +} + +// The jobs whose runs the boxes carry on with, kept from before their start +// is sent until they're dismissed, so a relaunch brings them back. +const KEPT_JOBS = "berth.jobs.stopRuns"; +// A run's journal goes after a while; a job that old isn't brought back, +// dismissed or not. +const KEEP_MS = 24 * 60 * 60 * 1000; + +interface KeptJob { + ref: string; + title: string; + started: number; + // Stop the rest was pressed: brought back, it sends nothing, and a share + // whose start went unanswered is only looked up, to be cancelled. + cancelled?: boolean; + // Each box's share: id once the box answered the start; params to ask + // again with, under the same group and key, and for its worktrees. + runs: { box: string; id?: string; params?: StopRunParams }[]; + // The rows of those runs. + items: Omit[]; +} + +const keptJobs = () => load(KEPT_JOBS, []).filter((k) => Date.now() - k.started < KEEP_MS); +const remember = (k: KeptJob) => save(KEPT_JOBS, [...keptJobs().filter((x) => x.ref !== k.ref), k]); +// keptRun notes box's run for the job ref, or, without an id, that box has +// none (it refused, or nothing was sent): its share and rows aren't kept. +function keptRun(ref: string, box: string, id: string | undefined) { + save( + KEPT_JOBS, + keptJobs().flatMap((k) => { + if (k.ref !== ref) return [k]; + if (id) return [{ ...k, runs: k.runs.map((r) => (r.box === box ? { ...r, id } : r)) }]; + const runs = k.runs.filter((r) => r.box !== box); + return runs.length ? [{ ...k, runs, items: k.items.filter((i) => i.group !== box) }] : []; + }), + ); +} +onDismiss((j) => { + if (j.ref) save(KEPT_JOBS, keptJobs().filter((k) => k.ref !== j.ref)); +}); +// Kept at once, so a relaunch, even before the box answers, doesn't undo it. +onCancel((j) => { + if (j.ref) save(KEPT_JOBS, keptJobs().map((k) => (k.ref === j.ref ? { ...k, cancelled: true } : k))); +}); + +// reattachStopJobs brings back the stop jobs the boxes carried on with +// while the app was closed, each as its runs are now or as they ended. A +// row only says what its box says; until a box answers it waits for it. A +// share whose start was never answered is asked for again, under the same +// key: the box has it, or takes it now. +export async function reattachStopJobs(deps: { runs: StopRuns; every?: number }) { + const kept = keptJobs(); + save(KEPT_JOBS, kept); + await Promise.all( + kept + .filter((k) => !useJobs.getState().jobs.some((j) => j.ref === k.ref)) + .map(async (k) => { + const job = startJob({ title: k.title, items: k.items.map((i) => ({ ...i, message: "Waiting for its box" })), cancellable: true, ref: k.ref }); + if (k.cancelled) cancelJob(job); + const none: Tally = { stopped: 0, unsent: 0, worktrees: {} }; + const tallies = await Promise.all( + k.runs.map(async (r) => { + if (r.id) return followJobRun(job, r.box, r.id, deps.runs, deps.every, r.params); + if (!r.params) return none; + const keys = k.items.filter((i) => i.group === r.box).map((i) => i.key); + const t = await onTheBox(job, { ref: k.ref, title: k.title, box: r.box, params: r.params, keys, asked: true }, deps.runs, deps.every); + if (t) return t; + for (const key of keys) setItem(job, key, { state: "failed", message: `Failed: ${r.box} can't run it` }); + return none; + }), + ); + const worktrees = Object.values(Object.assign({}, ...tallies.map((t) => t.worktrees)) as Record); + if (jobOf(job)) + finishJob( + job, + stopSummary( + k.items.length, + tallies.reduce((n, t) => n + t.stopped, 0), + tallies.reduce((n, t) => n + t.unsent, 0), + worktrees, + ), + ); + }), + ); +} + +// stopSummary is what a finished job says: "17 stopped, 15 archived", and +// what was kept, failed or never sent. +export function stopSummary(total: number, stopped: number, unsent: number, worktrees: WorktreeOutcome[]) { + const count = (o: WorktreeOutcome) => worktrees.filter((w) => w === o).length; + const failed = total - stopped - unsent + count("failed"); + return [ + `${stopped} stopped`, + count("archived") && `${count("archived")} archived`, + count("archiving") && `${count("archiving")} archiving`, + count("kept") && `${count("kept")} kept`, + failed && `${failed} failed`, + unsent && `${unsent} not stopped`, + count("cancelled") && `${count("cancelled")} not archived`, + ] + .filter(Boolean) + .join(", "); +} diff --git a/app/src/lib/stop-on-box.ts b/app/src/lib/stop-on-box.ts new file mode 100644 index 000000000..53d0b259b --- /dev/null +++ b/app/src/lib/stop-on-box.ts @@ -0,0 +1,43 @@ +import { boxOffline } from "@/lib/queue"; +import { boxHasRuns, runs, scheduleRuns } from "@/lib/runs"; +import { fitsRun, followStopRun, type Started, type StopRead, type StopRun, type StopRunParams, type StopRuns, startStopRun } from "@/lib/stop-job"; + +// The boxes' stop runs (lib/stop-job.ts): what the board's bulk Stop, the +// Worktrees table's Stop sessions and Archive, the sidebar's Stop +// everything and Archive… ask a box with runs to do, so it carries on while +// the laptop sleeps. +export const stopRuns: StopRuns = { + onBox: (box) => boxHasRuns(box) && !boxOffline(box), + start: async (box, req) => { + const run = await runs.start(box, { template: "stop", title: req.title, group: req.group, params: { ...req.params } }, req.idemKey); + scheduleRuns(box, 0); + return run; + }, + find: async (box, group) => (await runs.list(box, { template: "stop", group, limit: 1 }))[0], + get: (box, id) => runs.get(box, id) as Promise, + cancel: (box, id) => runs.cancel(box, id), +}; + +// startStopOnBox asks box to stop sessions and archive worktrees as one +// run (startStopRun), or answers "refused" when the box can't run it (no +// runs, offline, or from before stop runs): the caller does it itself. A box +// that doesn't answer is asked again until cancelled says so. +export function startStopOnBox(box: string, title: string, params: StopRunParams, o: { cancelled?(): boolean; waiting?(): void } = {}): Promise { + if (!stopRuns.onBox(box) || !fitsRun(params)) return Promise.resolve("refused"); + return startStopRun(stopRuns, box, { title, group: `stop-${Date.now().toString(36)}${Math.random().toString(36).slice(2, 6)}`, params }, o); +} + +// How long stopOnBox, which a dialog waits on, asks a box that doesn't +// answer before looking the run up instead. +const PATIENCE_MS = 15_000; + +// stopOnBox is startStopOnBox, then the run followed to its end: its last +// read, or undefined when the box didn't take it, so the caller does it. It +// throws when the box never answered, which may have the run all the same. +export async function stopOnBox(box: string, title: string, params: StopRunParams, o: { onRead?(read: StopRead): void; cancelled?(): boolean } = {}): Promise { + const t0 = Date.now(); + const started = await startStopOnBox(box, title, params, { cancelled: () => !!o.cancelled?.() || Date.now() - t0 > PATIENCE_MS }); + if (started === "unknown") throw new Error(`${box} didn't answer, so it may still do it`); + if (typeof started === "string") return undefined; + return followStopRun(stopRuns, box, started.id, { onRead: (r) => o.onRead?.(r), cancelled: o.cancelled }); +} diff --git a/app/src/lib/team.ts b/app/src/lib/team.ts index b8737a4a5..5738f422a 100644 --- a/app/src/lib/team.ts +++ b/app/src/lib/team.ts @@ -549,10 +549,6 @@ export const SOURCE_LABEL: Record = { none: "no setup", }; -export function plural(n: number, one: string, many = `${one}s`) { - return `${n} ${n === 1 ? one : many}`; -} - // keysLine is the keys step in one line: "5 from 1Password · 1 to enter". export function keysLine(v: Pick, entered = 0, skipOP = false): string { const ops = skipOP ? (v.keys.onepassword ?? []).filter((k) => !k.set).length : 0; diff --git a/app/src/views/dashboard/stop.tsx b/app/src/views/dashboard/stop.tsx index 65508c4ce..ab9917b30 100644 --- a/app/src/views/dashboard/stop.tsx +++ b/app/src/views/dashboard/stop.tsx @@ -1,4 +1,4 @@ -import { AlertTriangleIcon, BrushCleaningIcon, CheckIcon, EllipsisIcon, ListChecksIcon, SquareIcon, Trash2Icon, XIcon } from "lucide-react"; +import { AlertTriangleIcon, BrushCleaningIcon, EllipsisIcon, ListChecksIcon, SquareIcon, Trash2Icon } from "lucide-react"; import { useRef, useState } from "react"; import { create } from "zustand"; @@ -12,18 +12,17 @@ import { AlertDialog, AlertDialogDescription, AlertDialogFooter, AlertDialogHead import { Button } from "@/components/ui/button"; import { Checkbox } from "@/components/ui/checkbox"; import { Menu, MenuItem, MenuPopup, MenuSeparator, MenuTrigger } from "@/components/ui/menu"; -import { Spinner } from "@/components/ui/spinner"; import { toastManager } from "@/components/ui/toast"; import type { SessionEntry } from "@/hooks/use-agent-counts"; import { boxApi } from "@/lib/api"; -import { planArchive, type WorktreeTarget } from "@/lib/archive-after-stop"; import { agentOf, type SessionState } from "@/lib/derive"; import { plainError } from "@/lib/errors"; -import { errorMessage } from "@/lib/format"; -import { removeWorktreeOnBox } from "@/lib/remove-worktree"; -import { removalKey } from "@/lib/removing"; +import { plural } from "@/lib/format"; +import { archiveEnded, removeWorktreeOnBox } from "@/lib/remove-worktree"; import { load, save } from "@/lib/storage"; import { scheduleRefresh, useStore } from "@/lib/store"; +import { runStopJob } from "@/lib/stop-job"; +import { stopRuns } from "@/lib/stop-on-box"; import { cn } from "@/lib/utils"; import { duration } from "@/views/dashboard/agent-card"; import { type AgentNames, describeAgent, startedAt } from "@/views/dashboard/names"; @@ -36,15 +35,10 @@ import { type AgentNames, describeAgent, startedAt } from "@/views/dashboard/nam export const entryKey = (e: SessionEntry) => `${e.box}/${e.session.name}`; const sinceOf = (e: SessionEntry) => new Date(e.session.state_since ?? e.session.created).getTime(); -const plural = (n: number, one: string, many = `${one}s`) => `${n} ${n === 1 ? one : many}`; // Busy agents: stopping one loses the turn it is in. const busy = (s: SessionState) => s === "running" || s === "waiting"; -// A few at a time, so a long list is quick without every request landing -// on a box at once. -const CONCURRENCY = 3; - // stopOne ends one session and drops it from the board straight away; the // box's next listing confirms it. async function stopOne(box: string, name: string) { @@ -58,91 +52,47 @@ async function stopOne(box: string, name: string) { scheduleRefresh(box, ["sessions"]); } -interface Progress { - state: "running" | "ok" | "failed"; - message?: string; -} - -// stopAgents stops several, CONCURRENCY at a time, reports each, and -// answers with the keys that stopped. -async function stopAgents(entries: SessionEntry[], onEach: (key: string, p: Progress) => void): Promise> { - const queue = [...entries]; - const stopped = new Set(); - const worker = async () => { - for (let e = queue.shift(); e; e = queue.shift()) { - const key = entryKey(e); - onEach(key, { state: "running" }); - try { - await stopOne(e.box, e.session.name); - stopped.add(key); - onEach(key, { state: "ok" }); - } catch (err) { - onEach(key, { state: "failed", message: plainError(err) }); - } - } - }; - await Promise.all(Array.from({ length: Math.min(CONCURRENCY, entries.length) }, worker)); - return stopped; -} - -// What happened to a stopped agent's worktree when the dialog archived. -interface WorktreeOutcome { - state: "running" | "archived" | "archiving" | "kept" | "failed"; - message: string; -} - -// keepsWork is git refusing, in its own words, to lose uncommitted work. -const keepsWork = (err: unknown) => /modified|untracked|uncommitted|contains/i.test(errorMessage(err)); - -const leftAsItWas = (o?: WorktreeOutcome) => o?.state === "kept" || o?.state === "failed"; - -// archiveStopped archives the stopped rows' worktrees the way the sidebar's -// Archive… does: the archive script runs, the branch is kept, and git -// refuses to lose uncommitted work. One at a time on each box, as removals -// go; the boxes side by side. It answers with each worktree's outcome. -async function archiveStopped(rows: Row[], stopped: Set, onEach: (wtKey: string, o: WorktreeOutcome) => void): Promise> { - const out: Record = {}; - const report = (key: string, o: WorktreeOutcome) => { - out[key] = o; - onEach(key, o); - }; - const { client } = useStore.getState(); - // The boxes' own listing, fresh, so an agent started a moment ago counts. - // A box that doesn't answer keeps its worktrees: nothing says they're free. - const candidates = [...new Set(rows.filter((r) => r.names.where && !r.names.where.worktree.main).map((r) => r.entry.box))]; - const listed = await Promise.all(candidates.map(async (box) => ({ box, sessions: client ? await boxApi.sessions(client, box).catch(() => undefined) : undefined }))); - const live = listed.flatMap(({ box, sessions }) => (sessions ?? []).filter((s) => !s.exited && agentOf(s)).map((s) => ({ box, dir: s.dir, name: s.name }))); - const plan = planArchive( - rows.map((r) => ({ key: r.key, box: r.entry.box, where: r.names.where && { location: r.names.where.location.name, worktree: r.names.where.worktree } })), - stopped, - live, - new Set(listed.filter((l) => !l.sessions).map((l) => l.box)), - ); - for (const k of plan.kept) report(k.key, { state: "kept", message: `Kept: ${k.why}` }); - const archiveOne = async (w: WorktreeTarget) => { - report(w.key, { state: "running", message: "Archiving…" }); - try { - if (!client) throw new Error("Not connected to the laptop agent"); - const res = await removeWorktreeOnBox(client, w.box, w.location, w.wt, "archive"); - report(w.key, res.archive ? { state: "archiving", message: "Archiving: its archive script runs, then it goes" } : { state: "archived", message: "Archived; its branch is kept" }); - } catch (err) { - report(w.key, keepsWork(err) ? { state: "kept", message: "Kept: uncommitted changes" } : { state: "failed", message: `Couldn't archive: ${plainError(err)}` }); - } - }; - const boxesOf = [...new Set(plan.archive.map((w) => w.box))]; - await Promise.all( - boxesOf.map(async (box) => { - for (const w of plan.archive.filter((x) => x.box === box)) await archiveOne(w); - }), +// stopInBackground stops the confirmed rows as a job in the tray +// (lib/stop-job.ts), archiving their worktrees where it's safe if asked: a +// box with runs does its share itself. A clean run ends with the toast it +// always had. +async function stopInBackground(rows: Row[], archive: boolean) { + const res = await runStopJob( + rows.map((r) => ({ + key: r.key, + box: r.entry.box, + session: r.entry.session.name, + label: r.names.place, + aside: r.names.crowded ? r.names.name : undefined, + where: r.names.where && { location: r.names.where.location.name, worktree: r.names.where.worktree }, + })), + archive, + { + stop: stopOne, + liveAgents: async (box) => { + const client = useStore.getState().client; + const sessions = client ? await boxApi.sessions(client, box) : undefined; + return sessions?.filter((s) => !s.exited && agentOf(s)).map((s) => ({ dir: s.dir, name: s.name })); + }, + archive: (box, location, wt) => { + const client = useStore.getState().client; + if (!client) throw new Error("Not connected to the laptop agent"); + return removeWorktreeOnBox(client, box, location, wt, "archive"); + }, + settled: archiveEnded, + errorText: plainError, + runs: stopRuns, + }, ); - return out; -} - -// archiveSummary is " · archived 2 worktrees" (or archiving, while a script -// runs), for the toast and the done title alike. -function archiveSummary(outcomes: WorktreeOutcome[]) { - const gone = outcomes.filter((o) => o.state === "archived" || o.state === "archiving"); - return gone.length ? ` · ${gone.some((o) => o.state === "archiving") ? "archiving" : "archived"} ${plural(gone.length, "worktree")}` : ""; + if (!res.clean) return; + const outcomes = Object.values(res.worktrees); + const gone = outcomes.length; + const boxes = [...new Set(rows.map((r) => r.entry.box))].map((b) => `${rows.filter((r) => r.entry.box === b).length} on ${b}`); + toastManager.add({ + title: `Stopped ${plural(rows.length, "agent")}${gone ? ` · ${outcomes.includes("archiving") ? "archiving" : "archived"} ${plural(gone, "worktree")}` : ""}`, + description: `${boxes.join(", ")}. ${gone ? "Their branches are kept." : "Their worktrees are kept."}`, + type: "success", + }); } function namesOf(e: SessionEntry): AgentNames { @@ -230,14 +180,14 @@ const AFTER: { value: After; label: string; ms: number }[] = [ { value: "1d", label: "1 day", ms: 86_400_000 }, ]; -// StopDialog is the host; the dashboard renders it once. +// StopDialog is the host; the dashboard renders it once. It only asks: +// once confirmed it closes, and the stopping goes on in the tray. export function StopDialog() { const req = useStopDialog((s) => s.req); - const [running, setRunning] = useState(false); - const close = () => !running && useStopDialog.setState({ req: undefined }); + const close = () => useStopDialog.setState({ req: undefined }); return ( !o && close()}> - {req && useStopDialog.setState({ req: undefined })} />} + {req && } ); } @@ -249,7 +199,7 @@ interface Row { idle: number; } -function Body({ req, onRunning, onClose }: { req: StopRequest; onRunning(r: boolean): void; onClose(): void }) { +function Body({ req, onClose }: { req: StopRequest; onClose(): void }) { const cleanup = req.kind === "cleanup"; // Names are taken when the dialog opens: they renumber as agents stop. const [rows] = useState(() => { @@ -261,20 +211,14 @@ function Body({ req, onRunning, onClose }: { req: StopRequest; onRunning(r: bool // time) is idle on purpose: it is listed, never ticked for you. const overFor = (a: After) => new Set(rows.filter((r) => !r.entry.session.parked && r.idle > AFTER.find((x) => x.value === a)!.ms).map((r) => r.key)); const [ticked, setTicked] = useState>(() => (cleanup ? overFor(after) : new Set(rows.map((r) => r.key)))); - const [progress, setProgress] = useState>({}); const [archive, setArchive] = useState(() => load("berth.dashboard.stopArchives", false)); - const [outcomes, setOutcomes] = useState>({}); - const [phase, setPhase] = useState<"confirm" | "running" | "done">("confirm"); const cancel = useRef(null); const chosen = rows.filter((r) => ticked.has(r.key)); const busyCount = chosen.filter((r) => busy(r.entry.state)).length; - const ok = rows.filter((r) => progress[r.key]?.state === "ok").length; - const failed = rows.filter((r) => progress[r.key]?.state === "failed"); // The box shows only when there is a worktree to archive. const canArchive = chosen.some((r) => r.names.where && !r.names.where.worktree.main); const archiving = archive && canArchive; - const outcomeOf = (r: Row) => (r.names.where ? outcomes[removalKey(r.entry.box, r.names.where.worktree.path)] : undefined); const boxes = [...new Set(rows.map((r) => r.entry.box))].sort(); const toggle = (key: string, on: boolean) => setTicked((s) => { @@ -284,44 +228,19 @@ function Body({ req, onRunning, onClose }: { req: StopRequest; onRunning(r: bool return n; }); - const run = async () => { - const targets = chosen.map((r) => r.entry); - setPhase("running"); - onRunning(true); - const stopped = await stopAgents(targets, (key, p) => setProgress((s) => ({ ...s, [key]: p }))); - const found = archiving ? Object.values(await archiveStopped(chosen, stopped, (key, o) => setOutcomes((s) => ({ ...s, [key]: o })))) : []; - onRunning(false); - if (stopped.size < targets.length || found.some(leftAsItWas)) { - setPhase("done"); - return; - } - const perBox = [...new Set(targets.map((t) => t.box))].map((b) => `${targets.filter((t) => t.box === b).length} on ${b}`); - toastManager.add({ - title: `Stopped ${plural(targets.length, "agent")}${archiveSummary(found)}`, - description: `${perBox.join(", ")}. ${found.length ? "Their branches are kept." : "Their worktrees are kept."}`, - type: "success", - }); + const run = () => { onClose(); + void stopInBackground(chosen, archiving); }; - const summary = archiveSummary(Object.values(outcomes)); - - const title = - phase === "done" - ? failed.length - ? `Stopped ${ok} of ${ok + failed.length}${summary}` - : `Stopped ${plural(ok, "agent")}${summary}` - : cleanup - ? "Clean up idle agents" - : req.column - ? `Stop ${chosen.length === rows.length ? "all " : ""}${plural(chosen.length, "agent")} in ${req.column}?` - : `Stop ${plural(chosen.length, "agent")}?`; - const description = - phase === "done" - ? `${failed.length ? "The rest were left running. " : ""}${Object.values(outcomes).some(leftAsItWas) ? "Some worktrees were kept as they are. " : ""}Each one says why below.` - : cleanup - ? `Stops agents in Done and Ready that have sat idle for a while. Working agents and ones waiting on you are left alone. ${archiving ? "Their worktrees are archived where it's safe; branches are kept." : "Worktrees and their files are kept."}` - : `Each one's session ends on its box; ${archiving ? "then its worktree is archived where it's safe, and its branch is kept." : "its worktree and files are kept."}${busyCount ? ` ${busyCount === 1 ? "One is" : `${busyCount} are`} still working or waiting on you, and ${busyCount === 1 ? "its turn ends" : "their turns end"} too.` : ""}`; + const title = cleanup + ? "Clean up idle agents" + : req.column + ? `Stop ${chosen.length === rows.length ? "all " : ""}${plural(chosen.length, "agent")} in ${req.column}?` + : `Stop ${plural(chosen.length, "agent")}?`; + const description = cleanup + ? `Stops agents in Done and Ready that have sat idle for a while. Working agents and ones waiting on you are left alone. ${archiving ? "Their worktrees are archived where it's safe; branches are kept." : "Worktrees and their files are kept."}` + : `Each one's session ends on its box; ${archiving ? "then its worktree is archived where it's safe, and its branch is kept." : "its worktree and files are kept."}${busyCount ? ` ${busyCount === 1 ? "One is" : `${busyCount} are`} still working or waiting on you, and ${busyCount === 1 ? "its turn ends" : "their turns end"} too.` : ""}`; return ( // The keyboard starts on Cancel: Enter right after ⌫ or a menu item must @@ -332,7 +251,7 @@ function Body({ req, onRunning, onClose }: { req: StopRequest; onRunning(r: bool {description} - {cleanup && phase === "confirm" && ( + {cleanup && (
    Idle longer than
      {group.map((r) => ( - toggle(r.key, on)} /> + toggle(r.key, on)} /> ))}
    @@ -370,13 +289,13 @@ function Body({ req, onRunning, onClose }: { req: StopRequest; onRunning(r: bool })}
    - {cleanup && phase === "confirm" && rows.length > 0 && ( + {cleanup && rows.length > 0 && (

    {chosen.length ? `${plural(chosen.length, "agent")} of ${rows.length} picked.` : `None idle that long.`} Tick or untick any of them.

    )} - {phase === "confirm" && canArchive && ( + {canArchive && (