Skip to content

feat(box): a bulk Stop runs on the box, so closing the laptop doesn't halt it - #192

Open
Stener1 wants to merge 23 commits into
cosscom:mainfrom
Stener1:bulk-stop-on-the-box
Open

Stener1 wants to merge 23 commits into
cosscom:mainfrom
Stener1:bulk-stop-on-the-box

Conversation

@Stener1

@Stener1 Stener1 commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #175 (archive in the background) and on the background-jobs tray branch (Stener1:background-jobs-tray, not opened upstream yet). Both are rebased onto current main. The diff includes their commits until they land: the tray is the six commits after #175's, and this PR's own work is the ten commits from feat(box): a bulk Stop runs on the box… on.

Summary

Closing the laptop mid bulk Stop used to halt it. The app ran the whole sequence, so in-flight and queued requests failed on wake, and agents ended up stopped but not archived. Now each box with runs does its share itself, as one durable stop run, and so does the sidebar's single Archive…:

runStopJob (app)                                   box (internal/box)
  split rows by box
  keep the job (berth.jobs.stopRuns) ── before anything is sent
  box has runs + online ──► POST /v1/runs {template: stop, group, idem_key}
     no answer ─► ask again, same key, until it answers (rows: Waiting for BOX)
                                                    map  stop_session ×N   (3 at a time, session.stop gate)
                                                    map  archive_worktree  (one at a time)
                                                           archiveTurn: one archive at a time on the box, across runs
                                                           live agent there?   → Kept: …
                                                           uncommitted work?   → Kept: uncommitted changes
                                                           worktree.remove gate
                                                           dropWorktreeIf(no force, branch kept)
                                                             archive script, then under lastLook:
                                                             re-check agents, remove (no session starts in between)
  follow GET /v1/runs/{id} ──► rows                (no answer = ask again; only the box fails a row)
  "Stop the rest" ──► cancel the run; a start still unanswered is looked up by group (GET /v1/runs?group=)
                      and cancelled if the box took it; the cancel is kept across a relaunch
  else: the app's own loop (older boxes, offline at start, a 4xx on start),
        each box's line held until the box says an archive script finished

Archive… (sidebar, zen, launcher)
  box has runs ──► one-worktree stop run: its sessions stop, then archive_worktree as above
  else         ──► the app's per-box line, held until worktree.removed / worktree.archive.failed
  • Box: the stop template and stop_session / archive_worktree leaves (runstop.go). dropWorktree gets an in-flight guard, so a second ask (a resumed run, a duplicate run, a second DELETE) joins the first instead of running the archive script again. The dirty check comes before that guard, so a second ask for a dirty worktree is refused, not answered 202. archiveTurn serializes archives across a box's stop runs. removeAfter / startUnseen close the check-then-remove race: no session starts between the last look for agents and the folder going. If berthd stops mid-archive, the step stays unfinished in the journal for the resumed run.
  • App: lib/stop-job.ts keeps the job before it sends, asks an unanswered box again under one key until it answers, and looks a run up by its group when Stop the rest comes first. A job comes back to the tray after a relaunch (for at most 24 h), cancelled if it was. The Worktrees table's Stop sessions and Archive, the sidebar's Stop everything in … and its Archive… use the same run (lib/stop-on-box.ts, archiveOnBox).
  • The mock plays stop runs out against its own fixtures (playStop), in each box's archive line. ?noruns gives the old path.
  • An archive that fails says why, and one stopped halfway can be finished (fix(box): an archive that fails says why…). git worktree remove ran under the 60 s limit that every git call has. On a busy box a big worktree took longer: git was killed mid-delete without a word, so the row read Couldn't archive: git worktree remove:, and the folder was left without its .git, which git then refused to remove again (validation failed … .git does not exist). Now:
    • removal gets 10 minutes (BERTH_WORKTREE_REMOVE_TIMEOUT sets another limit), and a failure with no output says how it ended: timed out, stopped by a signal, cancelled, or its exit status (gitFailed);
    • a worktree git lists as prunable is finished on archive (finishRemoval). What's left of its folder goes, then git's record of that one worktree, never git worktree prune, which would drop every half-removed worktree's record and orphan their folders. The branch is kept. It only does this when git, asked through the record's HEAD and index, sees nothing but files the removal already deleted, and no repository is inside the folder. Any .gitignore the removal deleted is put back as committed first, since otherwise node_modules reads as new files; otherwise it's Kept: uncommitted changes, and Remove anyway still forces. A worktree whose folder is gone is pruned the same way. Its archive script isn't run again;
    • whether a worktree still has its .git is checked each time the list is read, not taken from git's prunable line, so a cached listing (fix(box): listing projects runs git only when a repository changed #195) can't send such an archive back to git worktree remove;
    • a removal's last look for agents now holds only its own folder (lookGuard, which replaces the box-wide lastLook RWMutex), so a removal that takes minutes doesn't hold up session starts anywhere else on the box.
  • Docs: the stop template in guides/runs.mdx, the new step kinds, changelog bullets, and the verify skill's feature map and scenarios.

Evidence

All on the final head, with the verify-berthd skill.

A removal git doesn't finish in time (new dashboard-stop-interrupted-live, a real dev box, synthetic acme-shop). The box is restarted with a git on its PATH that, for verify-slow only, deletes the worktree's .git and .gitignore and then hangs (with a node_modules only that .gitignore ignored), as a big worktree's git worktree remove on a loaded box did. Then verify-slow and verify-ok are bulk-stopped with archiving ticked.

  • Before (VERIFY_BERTHD= this branch's previous head): the archive step failed after 1m0.283s, the 60 s git limit, with the error "git worktree remove: ". The tray row ended at the colon. git worktree list showed verify-slow as prunable with its files still there, and archiving it again failed for good: git worktree remove: fatal: validation failed, cannot remove working tree: '…/acme-shop-verify-slow/.git' does not exist.
  • After (BERTH_WORKTREE_REMOVE_TIMEOUT=5s):
- the failed row reads: verify-slow Stopped · Couldn't archive: git worktree remove timed out after 5s and was stopped; the folder may be partly removed, and archiving it again finishes it
- git agrees: verify-ok is gone, verify-slow is half removed (prunable, its files still there)
- Archive… verify-slow from the sidebar finishes it
- its folder is gone, git no longer lists it, and both branches stay
Before: a bare colon After: the cause After: Archive… finishes it

The scenario allows the console's 404 for GET sessions/<stopped agent>/screen: the board's card asks for a just-stopped agent's screen once. That happens without this change too, and each failed request is logged as a step.

A start the box never answers (new dashboard-stop-unanswered, a real dev box, synthetic acme-shop). berthd is killed (SIGKILL) as Stop is pressed and started again 20 s later.

  • Before: startStopRun gave up after three asks (about 6 s) and failed the rows, Failed: devq is offline, though the box might have the run. That was the old unit test a box that never answers the start fails its rows.
  • After: no row fails, the start is asked again once the box is back, and the box ends with one run for the job:
- with the box down the rows say: verify-mute-gp4k-a Waiting for devq | verify-mute-gp4k-b Waiting for devq
- after 20 s down, no row failed
- one stop run on the box for it (group stop-mvqsgss9yzpz): r_mvqshg2697ke succeeded
- git on the box agrees: both worktrees gone, both branches kept
Box down: rows wait Box back: done

Bulk Stop and sidebar Archive… on a real box (dashboard-stop-live): the report reads 3 stopped, 2 archived, 1 kept; git agrees; the box lists Stopping 3 agents: succeeded and Archive verify-solo: succeeded. No request is refused any more: the dirty worktree is kept inside the run.

Report Archive… as a stop run

The lid closing (dashboard-stop-lid): the laptop agent was paused for 25 s mid-archive, and on wake the tray read 5 stopped, 3 archived, 2 kept. A reload mid-run brought the job back, ending 2 stopped, 2 archived. The Worktrees table's Stop sessions and Archive each ran as a stop run.

After waking Reattached, done

One archive at a time per box (mock, worktree-archive): Archive… on gpu/ci-flake, then judge-v2 at once. Both are stop runs on gpu, and judge-v2 waits in gpu's line (Archived ci-flake at 5.5 s, Archived judge-v2 at 10.0 s). qa-deck's row stays Archiving… until its script is done, then shows one toast.

Both archiving Script running Dirty: kept

Tests:

  • Go, new:
    • TestStopRunsOnABoxArchiveOneAtATime: two runs' scripts log start end start end.
    • TestNoAgentStartsUnseenAsAWorktreeGoes: a session asked for during the last look waits, then gets no folder.
    • TestAWorktreeAskedAgainIsCheckedForWorkFirst: dirty while the first archive runs, so the second DELETE is refused.
    • The dedup test now waits for worktree.archive.started and quotes its path.
  • Go, new for the interrupted removal:
    • TestAWorktreeRemoveThatFailsWithNoOutputSaysWhy and TestAWorktreeRemoveThatTimesOutSaysSo, with a fake git on PATH; TestAStopRunSaysWhyARemovalFailed through a stop run.
    • TestAHalfRemovedWorktreeIsFinished: folder and record gone, branch kept, another half-removed worktree left alone; a live dirty worktree and the main checkout are still refused.
    • TestAHalfRemovedWorktreeWithWorkInItIsKept (a changed or an added file: kept in git's words; Remove anyway deletes it) and TestAHalfRemovedWorktreeHoldingARepositoryIsKept.
    • TestAWorktreeWhoseFolderIsGoneIsPruned (a worktree can be made at that path again) and TestAWorktreeGitStillHasIsNeverDeletedByHand.
    • TestAStopRunFinishesAHalfRemovedWorktree: half-removed → Archived, folder gone → already gone and pruned, one with work in it → Kept: uncommitted changes; the archive script never runs again.
    • TestAHalfRemovedWorktreeThatLostItsGitignore: .gitignore files deleted before node_modules → finished; with a really new file as well → kept.
    • TestAListingFromBeforeTheGitWentShowsItDropped: a listing read before the .git went still shows the worktree as dropped.
    • TestNoAgentStartsUnseenAsAWorktreeGoes now also starts a session in another worktree during the last look.
  • Unit, new: kept-reason (a timeout, a signal or a silent failure is a failure, not kept) and stop-job (the box's words reach the row unchanged).
  • Go, the full suite: go vet ./... && go test -race ./... fails only in tests that fail on main on this machine under load (TestTheCatalogIsCapped, TestReviewRoutesOverTheWire, TestDetach, TestServicesAreCounted); TestReviewRoutesOverTheWire passes 3/3 alone. On the interrupted-removal head it failed only TestDetach (it fails on main's tree too). Under load, earlier runs also failed TestForwardReachesTheBoxAndSurvivesAnAgentRestart, TestATeamSetupResumesAfterBerthdRestarts and TestSleepDurable; each passes 3/3 alone, on this branch and on its previous head.
  • Unit, new in stop-job.test.ts:
    • an unanswered start is asked until answered
    • Stop the rest while unanswered: a run the box took is cancelled, and with none nothing is done
    • kept before sending, then asked again after a relaunch
    • a cancel while out of reach is kept: after a relaunch it is only looked up and cancelled, never started (both with and without the box having taken it)
    • a 404'd run clears its Archiving… marks
    • an archive script holds its box's line
  • Unit, elsewhere: runPool with an undefined item, and unmarked. pnpm test: 636 pass.
  • E2E: dashboard-stop-archive.spec.ts and worktree-remove.spec.ts pass, 11/11.
  • These pass: typecheck:plugins, check:titles, check:type, check:grid, check:csp, check:themes, tsc -p e2e and pnpm build. cargo check was not run: GTK dev libs are missing on this machine, which is unrelated.
  • Codex review and adversarial review ran on the interrupted-removal change. They found three problems, all fixed in its commit: a remnant's files were deleted without checking them for work, a retry re-ran the archive script on a half-deleted folder, and another repository nested under an ignored folder could be deleted with it.
  • The 29 review threads from cubic are each answered with the commit that addresses them. Codex review and adversarial review ran on the fixes; their findings (a cancel lost across a relaunch, and an archive failure that beats the 202) are fixed in the last commit.

Merge Danger

Door: two-way

On the box it's additive: a new template, two step kinds, an in-flight guard, an archive line across stop runs, and a lock held only from a removal's last look for agents until the folder goes. Old clients are unaffected, and new clients fall back on boxes without runs. Reverting the PR restores the app's loop.

Blast Radius: bulk-stop and archive

  • Where it reaches: on boxes with runs, the board's bulk Stop / Clean up, the Worktrees table's Stop sessions and Archive, the sidebar's Stop everything and its single Archive… all go through the box.
  • What changes there:
    • A row reads Archived once the script is done, not Archiving while it runs.
    • Archive stops the worktree's sessions first, as archiving always ended them.
    • A worktree is kept if an agent starts there during the script.
    • A bulk Stop row waits for an unreachable box instead of failing.
    • A session start in a worktree being removed waits for that removal; starts elsewhere don't wait.
    • git worktree remove may take up to 10 minutes instead of 60 s, and a worktree git dropped the link to is finished by the box itself (its folder, then its record under .git/worktrees/), only when nothing in it is work. Branches are always kept.
  • On boxes without runs: the app holds a box's line until an archive script finishes, so queued archives there wait longer, but never overlap.

🤖 Generated with Claude Code

@vercel

vercel Bot commented Oct 27, 2026

Copy link
Copy Markdown

@Stener1 is attempting to deploy a commit to the seanbrydonio's projects Team on Vercel.

A member of the Team first needs to authorize it.

@Stener1

Stener1 commented Oct 27, 2026

Copy link
Copy Markdown
Contributor Author

Could a maintainer add run-ci? I can't label PRs here. The checks from AGENTS.md pass locally; details are in the PR body.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 41 files

Reply with feedback, questions, or to request a fix.

View guided diff | Re-trigger cubic

Comment thread .claude/skills/verify-berthd/features/dashboard-stop.md Outdated
Comment thread .claude/skills/verify-berthd/scripts/scenarios/dashboard-stop-live.mjs Outdated
Comment thread app/src/lib/remove-worktree.ts Outdated
Comment thread app/src/lib/removing.ts
Comment thread app/src/lib/mock-runs.ts Outdated
Comment thread docs/guides/runs.mdx Outdated
Comment thread .claude/skills/verify-berthd/features/worktree-archive.md Outdated
Comment thread .claude/skills/verify-berthd/features/worktree-archive.md Outdated
Comment thread app/src/App.tsx
Comment thread .claude/skills/verify-berthd/features/worktree-archive.md Outdated
@Stener1
Stener1 force-pushed the bulk-stop-on-the-box branch from 302c509 to 2575aac Compare October 27, 2026 14:40
@Stener1

Stener1 commented Oct 27, 2026

Copy link
Copy Markdown
Contributor Author

Rebased onto current main on top of the rebased #175 (and background-jobs-tray), head 2575aac3. All 29 cubic threads are addressed, each with a reply naming its commit; all were valid and fixed. The larger ones:

  • a stop run's start is asked again until the box answers, kept before it's sent, and looked up by group on Stop the rest or a relaunch (the P1)
  • the sidebar's Archive… is a one-worktree stop run on boxes with runs
  • the box archives one worktree at a time across its stop runs
  • no session can start between a removal's last look for agents and the folder going

Codex's review of the fixes found two more (a cancel lost across a relaunch, and an archive failure arriving before its 202); both are fixed in the last commit. The PR body has the new evidence, including dashboard-stop-unanswered, which kills berthd as Stop is pressed.

Stener1 and others added 17 commits October 28, 2026 21:53
…199)

Archive… still asks, but pressing Archive closes the dialog at once. The
worktree shows Archiving… and goes when its box gets to it, through the
same per-box queue Remove worktree uses, now carrying the kind. The
toasts are today's: Archived (branch kept), Archiving (its script is
running), and a warning for uncommitted work.

Bulk archive in the Worktrees table skips a worktree already on its way
out, so its box never gets a second request for it. The mock takes
?slowremove=MS to make a box slow to remove a worktree, and the verify
skill gains a worktree-archive scenario and feature file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Confirming Stop N agents? (and Clean up…) now closes the dialog at once.
The stopping, and the archiving when it was ticked, goes on in a tray in
the bottom-right corner, above the status bar and the loops panel: one
row per agent, grouped by box, with progress in the title ("Stopping 17
agents · 9/17"). It folds to one line (remembered), Stop the rest sends
nothing more, and a clean run sums itself up ("17 stopped, 15 archived"),
shows the toast it always did and fades. If something was kept or failed
it stays as a report of those rows until you dismiss it. Jobs live in an
app-level store, so they carry on when you leave the board.

The sidebar's Archive… shows up in the tray too, as a one-row job.
Concurrency and the archive safety rules are unchanged; bulk archives now
wait in the same per-box line as every other removal. The mock gains
?slowstop=MS to make stops slow.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The dashboard-stop-archive scenario now shows the dialog gone at once,
the tray mid-run, folded, while another view is open, its report, a
clean run fading and Stop the rest. worktree-archive checks the one-row
job. The feature files say how the tray behaves, and the changelog moves
the archive bullet back under Unreleased. The tray lifts above a page's
floating bar so it never hides the board's selection buttons, a long
summary wraps, and a one-row job drops its count.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dashboard-stop-live makes its own worktrees and agents on dev-box.sh's
box, stops three with archiving ticked, and checks the tray's report
against git worktree list and git branch there; then it archives a
fourth from the sidebar.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
keepsWork and the archive outcome messages live in lib/removing.ts, so
the stop job, the single archive and the Worktrees table share them, and
the stop job no longer takes keepsWork as a dep. Every archive through
queueRemoval is a tray job, so the tray flag goes. plural moves to
lib/format.ts. The tray uses Button for Fold and Dismiss, one icon per
outcome and per row state, and memoised cards and rows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…be held back

From the review: an archive waiting its turn on a box now reads the
box's agents again when its turn comes, so one started there meanwhile
keeps the worktree. Each worktree is marked as it is queued, so Archive…
from the sidebar can't ask for it a second time, and one already on its
way out is skipped. Rows waiting to archive count as unsent, so Stop the
rest stays offered until they go.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ds it back

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… halt it

The board's bulk Stop (and archiving where it's safe), the Worktrees
table's Stop sessions and Archive, and the sidebar's Stop everything are
now one stop run per box with runs: a new run template with two leaves,
stop_session and archive_worktree, that keep the app's safety rules
(never forced, branch kept, dirty / another agent / a failed stop keep
it, before:-gates honoured) and wait for the archive script. The app
follows each run; a row fails only when the box says so, and the job
comes back to the tray after a relaunch. dropWorktree joins a removal
already in flight rather than running the archive script twice. Boxes
without runs keep the app's own loop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… and keeps a worktree an agent started in meanwhile

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…it, and a lost one fails its rows in the table

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ent starts unseen as one goes

A stop run's archive step waits its turn on the box (archiveTurn) before
it looks the worktree up, so archive scripts from separate runs never run
side by side, and one archived meanwhile is found gone. The last look for
agents and the removal hold a lock that session starts take for reading
(removeAfter, startUnseen): an agent asked for in between waits, then
finds its folder gone. A session listing that fails now fails the step
instead of passing for "another agent". A worktree is checked for
uncommitted work before it counts as on its way out, so a second ask for
a dirty one is refused rather than answered 202. The dedup test waits for
worktree.archive.started and quotes its temp path.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ive… is the box's to finish

A bulk Stop is kept (berth.jobs.stopRuns) before its start is sent, with
each box's params. A box that doesn't answer is asked again under the
same idempotency key until it does, its rows "Waiting for BOX", instead
of failing them after three tries; after a relaunch an unanswered share
is asked for again. Stop the rest meanwhile looks the run up by its group
(GET runs?group=): one the box took is followed and cancelled, and with
none the rows read Not stopped. A run the box no longer has lets go of
its worktrees' Archiving… marks.

Archive… from the sidebar, zen and the launcher is a one-worktree stop
run on a box with runs: its sessions stop, it waits the box's own turn,
it finishes while the laptop sleeps, and the tray's row ends only when
the worktree is gone or kept. Elsewhere the app's line for the box (and
the board's own loop) holds until the box says the archive script
finished (archiveEnded), so scripts on one box never overlap and a row
no longer reads done while its script runs. The mock lists runs by
group, answers a repeated key with the run it has, ends a stop run's
maps with it, and fails an archive step whose session listing fails.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…al, the tray hidden takes no room

runPool walks its queue by length, so an undefined item runs too. team.ts
drops its copy of plural for format.ts's. The jobs tray, hidden under
Settings, sets --berth-jobs-h to 0 instead of its gap.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…the runs guide and the verify skill

The live scenario no longer expects the 400 a bulk Stop never sends: it
reads the box's stop runs back instead, for the bulk Stop and the
sidebar's Archive…. The feature map says how a start that goes
unanswered is handled, that jobs are kept 24 hours at most, that a run
joining an archive reads Archived, that a bulk Stop still stops agents
whose worktree is kept, the review/ branch exception, and the real
queueRemoval call.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…he archive scenarios expect the box to finish

The mock plays a box's archive line across its stop runs, as archiveTurn
does. worktree-archive expects Archive… on devl and gpu to be a stop run
there: judge-v2 waits in gpu's line, qa-deck's row stays Archiving… until
its script is done, and one toast says Archived. Toasts are found by role,
so a second one for the same archive fails. dashboard-stop-archive's report
reads 2 archived, since the app's loop now waits for the script.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…a real box

dashboard-stop-unanswered kills the dev box's berthd as Stop is pressed
and starts it again 20 s later: no row fails meanwhile, and the box ends
with one stop run for the job, followed to its end, with git agreeing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…at fails fast isn't read as archived

From Codex's review. Stop the rest pressed while a box answers nothing
is kept with the job (onCancel), so after a relaunch the job comes back
cancelled: a share whose start went unanswered is only looked up by its
group, its run cancelled if the box took it, and never sent again.
removeWorktreeOnBox notes how an archive ends from before it sends it, so
a worktree.archive.failed that beats the 202 still fails the tray's row.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Stener1
Stener1 force-pushed the bulk-stop-on-the-box branch from 2575aac to 38ab82b Compare October 28, 2026 22:05
@Stener1

Stener1 commented Oct 28, 2026

Copy link
Copy Markdown
Contributor Author

Rebased the stack (#175, background-jobs-tray, this PR) onto current main (71efa89a), head 38ab82b5. The 29 cubic threads are unchanged and stay resolved. The rebase only changed three things, each folded into the commit it belongs to:

  • queueRemoval's keep-work check now uses main's keptReason (from fix(box): worktrees with submodules can be removed when nothing would be lost #197), so only a real refusal over unsaved work offers Remove anyway, and its toast gives the box's own reason (submodule commits, couldn't check). The PR's archive toast is the same as before.
  • With that check, remove-worktree.ts no longer imports keepsWork, and errorMessage is imported again.
  • v0.3.31 renamed ## Unreleased, so the three changelog bullets had landed in the released section. They are under a new ## Unreleased again.

Re-run on the new head: tsc --noEmit; the unit tests for the changed libs (stop-job, jobs, removing, kept-reason, …), 81/81; go vet ./internal/box/...; go test -race on the box's stop-run, archive, worktree and submodule tests, and on internal/box/runs. It still needs run-ci.

Stener1 and others added 3 commits October 29, 2026 10:35
… be finished

git worktree remove ran under the 60 s limit every git call has. A big
worktree on a busy box took longer, git was killed mid-delete and said
nothing, so the row read "Couldn't archive: git worktree remove:" and the
folder was left without its .git, which git then refused to remove again.

- git worktree remove gets 10 minutes (BERTH_WORKTREE_REMOVE_TIMEOUT sets
  another), and a failure with no output says how it ended: timed out,
  stopped by a signal, cancelled, or its exit status.
- A worktree git lists as prunable is finished on archive: what's left of
  its folder, then git's record of that one worktree (never git worktree
  prune, which would drop every half-removed one's record), branch kept.
  Only when git, asked through the record's HEAD and index, sees nothing
  but files the removal already deleted, and no repository is inside.
  One whose folder is gone is pruned the same way. Its archive script
  isn't run again.
- A removal's last look for agents holds only its own folder, so a
  removal git takes minutes over doesn't hold up session starts elsewhere.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dashboard-stop-interrupted-live restarts the dev box with a git that
deletes one worktree's .git and then hangs, and a 5 s removal budget. A
bulk Stop's row must say it timed out, and Archive... must then finish the
half-removed worktree, keeping its branch. VERIFY_BERTHD runs it on an
older build for a before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Whether a worktree still has its .git is looked at each time the list is
read, rather than taken from git's "prunable" line, so a listing read
before the .git went (a cached one) doesn't show it as live and send its
archive back to git worktree remove, which refuses it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

8 issues found across 15 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="docs/changelog.mdx">

<violation number="1" location="docs/changelog.mdx:48">
P3: "as any archive does" overstates the branch behavior: a review worktree's `review/` branch is removed by every Archive. Name that exception here, as the runs guide does.</violation>
</file>

<file name="docs/concepts/projects-and-worktrees.mdx">

<violation number="1" location="docs/concepts/projects-and-worktrees.mdx:116">
P2: This overstates the recovery. Archiving a half-removed worktree only finishes it when it holds no unsaved work and no nested repository; otherwise it is refused unless forced. Say so here, and note that a review worktree's `review/` branch is removed.</violation>
</file>

<file name="internal/box/worktreeremove_test.go">

<violation number="1" location="internal/box/worktreeremove_test.go:35">
P3: Setup errors are discarded in `worktreeIn` and several tests here. If one of these writes or removes fails, the test reports a misleading refusal or a later assertion instead of the real setup failure. Check each one with `t.Fatal(err)`, as the rest of the file does.</violation>

<violation number="2" location="internal/box/worktreeremove_test.go:163">
P2: This check passes vacuously if `wt.Path` never appears in `parseWorktrees`'s output. Assert that the worktree was found in the listing, and that it reads as prunable.</violation>
</file>

<file name="internal/box/lastlook.go">

<violation number="1" location="internal/box/lastlook.go:23">
P3: The doc comment says the hold lasts until the folder is gone. The release actually runs whenever `removeAfter` returns, including on a failed removal while the folder still exists. Reword the comment to match.</violation>
</file>

<file name="internal/box/runstop.go">

<violation number="1" location="internal/box/runstop.go:98">
P2: (Based on your team's feedback about clearing pins on worktree removal.) 
This path removes the gone worktree's record without the bookkeeping `dropWorktreeIf` does on removal. No `worktree.removed` event is published, so the app's `worktreeGone` never unpins it or clears its removing state. The ports for that path are also never released. Publish the event and release the ports here, as `removed()` does.</violation>
</file>

<file name="internal/box/runstop_test.go">

<violation number="1" location="internal/box/runstop_test.go:381">
P3: The bystander session start is called synchronously while the removal is held. If sessions elsewhere ever block, this test hangs instead of failing. Run it in a goroutine and wait with a `select` timeout, as the `latecomer` check below already does.</violation>
</file>

<file name="internal/box/locations.go">

<violation number="1" location="internal/box/locations.go:794">
P2: When a prunable worktree’s record is missing or `git status` fails, this returns false; removal then stops its services before `RemoveWorktree` refuses to proceed. Propagate an explicit check error before teardown.</violation>
</file>

Reply with feedback, questions, or to request a fix.

View guided diff | Re-trigger cubic

to remove the folder (`BERTH_WORKTREE_REMOVE_TIMEOUT` in berthd's
environment sets another limit). A removal stopped halfway leaves a
folder git no longer counts as a worktree; archiving it again removes
what's left, and keeps its branch. Both scripts come from

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: This overstates the recovery. Archiving a half-removed worktree only finishes it when it holds no unsaved work and no nested repository; otherwise it is refused unless forced. Say so here, and note that a review worktree's review/ branch is removed.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At docs/concepts/projects-and-worktrees.mdx, line 116:

<comment>This overstates the recovery. Archiving a half-removed worktree only finishes it when it holds no unsaved work and no nested repository; otherwise it is refused unless forced. Say so here, and note that a review worktree's `review/` branch is removed.</comment>

<file context>
@@ -109,7 +109,11 @@ name moves to it.
+to remove the folder (`BERTH_WORKTREE_REMOVE_TIMEOUT` in berthd's
+environment sets another limit). A removal stopped halfway leaves a
+folder git no longer counts as a worktree; archiving it again removes
+what's left, and keeps its branch. Both scripts come from
 [`.berth/config.json`](/guides/project-config), a [kit](/guides/kits), or the
 box's own config for the location (`berth location scripts`). A
</file context>
Suggested change
what's left, and keeps its branch. Both scripts come from
what's left unless it holds unsaved work or a nested repository (then `--force`, or a fix by hand), and keeps its branch. Both scripts come from

Comment on lines +163 to +170
for _, w := range parseWorktrees(out, repo) {
if w.Path == wt.Path && !w.Prunable {
t.Fatalf("%s reads as a live worktree from an older listing", w.Name)
}
if w.Main && w.Prunable {
t.Fatal("the main checkout reads as dropped")
}
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: This check passes vacuously if wt.Path never appears in parseWorktrees's output. Assert that the worktree was found in the listing, and that it reads as prunable.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At internal/box/worktreeremove_test.go, line 163:

<comment>This check passes vacuously if `wt.Path` never appears in `parseWorktrees`'s output. Assert that the worktree was found in the listing, and that it reads as prunable.</comment>

<file context>
@@ -0,0 +1,267 @@
+	if err := os.Remove(filepath.Join(wt.Path, ".git")); err != nil {
+		t.Fatal(err)
+	}
+	for _, w := range parseWorktrees(out, repo) {
+		if w.Path == wt.Path && !w.Prunable {
+			t.Fatalf("%s reads as a live worktree from an older listing", w.Name)
</file context>
Suggested change
for _, w := range parseWorktrees(out, repo) {
if w.Path == wt.Path && !w.Prunable {
t.Fatalf("%s reads as a live worktree from an older listing", w.Name)
}
if w.Main && w.Prunable {
t.Fatal("the main checkout reads as dropped")
}
}
found := false
for _, w := range parseWorktrees(out, repo) {
if w.Main && w.Prunable {
t.Fatal("the main checkout reads as dropped")
}
if w.Path == wt.Path {
found = true
if !w.Prunable {
t.Fatalf("%s reads as a live worktree from an older listing", w.Name)
}
}
}
if !found {
t.Fatalf("%s is missing from the listing, so nothing was checked", wt.Name)
}

Comment thread internal/box/runstop.go
Comment on lines +98 to +100
if err := b.Locations.RemoveWorktree(ctx, location, name, false); err != nil {
return fail(err)
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2:

(Based on your team's feedback about clearing pins on worktree removal.)
This path removes the gone worktree's record without the bookkeeping dropWorktreeIf does on removal. No worktree.removed event is published, so the app's worktreeGone never unpins it or clears its removing state. The ports for that path are also never released. Publish the event and release the ports here, as removed() does.

View Feedback

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At internal/box/runstop.go, line 98:

<comment>

(Based on your team's feedback about clearing pins on worktree removal.) 
This path removes the gone worktree's record without the bookkeeping `dropWorktreeIf` does on removal. No `worktree.removed` event is published, so the app's `worktreeGone` never unpins it or clears its removing state. The ports for that path are also never released. Publish the event and release the ports here, as `removed()` does.</comment>

<file context>
@@ -91,6 +91,14 @@ func (h *runHost) archiveWorktree(ctx context.Context, x *runs.StepCtx) runs.Res
+		// halfway, then the rest deleted): what git kept goes too, so a
+		// worktree can be made there again.
+		if g, ok := goneWorktree(ctx, loc, name); ok && (path == "" || g.Path == path) {
+			if err := b.Locations.RemoveWorktree(ctx, location, name, false); err != nil {
+				return fail(err)
+			}
</file context>
Suggested change
if err := b.Locations.RemoveWorktree(ctx, location, name, false); err != nil {
return fail(err)
}
if err := b.Locations.RemoveWorktree(ctx, location, name, false); err != nil {
return fail(err)
}
b.Locations.Ports.Release(g.Path)
b.Events.Publish(events.Event{Type: "worktree.removed", Box: b.Name, Origin: origOf(x), Data: map[string]any{"location": loc.Name, "name": name, "path": g.Path}})

Comment thread internal/box/locations.go
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
defer cancel()
record, err := worktreeRecord(ctx, repo, dir)
return err == nil && leftoverWork(ctx, record, dir) == "contains modified or untracked files"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: When a prunable worktree’s record is missing or git status fails, this returns false; removal then stops its services before RemoveWorktree refuses to proceed. Propagate an explicit check error before teardown.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At internal/box/locations.go, line 794:

<comment>When a prunable worktree’s record is missing or `git status` fails, this returns false; removal then stops its services before `RemoveWorktree` refuses to proceed. Propagate an explicit check error before teardown.</comment>

<file context>
@@ -540,17 +729,70 @@ func unquoteGit(v string) string {
+		ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
+		defer cancel()
+		record, err := worktreeRecord(ctx, repo, dir)
+		return err == nil && leftoverWork(ctx, record, dir) == "contains modified or untracked files"
+	}
 	ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
</file context>

Comment thread docs/changelog.mdx
still fails, the row says how: timed out, stopped, or failed with no
output. Archiving a worktree that git has already let go of (its `.git`
is gone, or its whole folder) now removes what's left and keeps its
branch, as any archive does.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: "as any archive does" overstates the branch behavior: a review worktree's review/ branch is removed by every Archive. Name that exception here, as the runs guide does.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At docs/changelog.mdx, line 48:

<comment>"as any archive does" overstates the branch behavior: a review worktree's `review/` branch is removed by every Archive. Name that exception here, as the runs guide does.</comment>

<file context>
@@ -36,6 +36,16 @@ again.
+  still fails, the row says how: timed out, stopped, or failed with no
+  output. Archiving a worktree that git has already let go of (its `.git`
+  is gone, or its whole folder) now removes what's left and keeps its
+  branch, as any archive does.
 
 ## v0.3.31
</file context>
Suggested change
branch, as any archive does.
branch (a review worktree's `review/` branch goes, as with every archive).

t.Helper()
repo := gitRepo(t)
// What a removal stopped halfway leaves is mostly what git ignores.
os.WriteFile(filepath.Join(repo, ".git", "info", "exclude"), []byte("node_modules\n"), 0o644)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: Setup errors are discarded in worktreeIn and several tests here. If one of these writes or removes fails, the test reports a misleading refusal or a later assertion instead of the real setup failure. Check each one with t.Fatal(err), as the rest of the file does.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At internal/box/worktreeremove_test.go, line 35:

<comment>Setup errors are discarded in `worktreeIn` and several tests here. If one of these writes or removes fails, the test reports a misleading refusal or a later assertion instead of the real setup failure. Check each one with `t.Fatal(err)`, as the rest of the file does.</comment>

<file context>
@@ -0,0 +1,267 @@
+	t.Helper()
+	repo := gitRepo(t)
+	// What a removal stopped halfway leaves is mostly what git ignores.
+	os.WriteFile(filepath.Join(repo, ".git", "info", "exclude"), []byte("node_modules\n"), 0o644)
+	l := NewLocations(filepath.Join(t.TempDir(), "locations.json"))
+	if _, err := l.Add(context.Background(), "cal", repo); err != nil {
</file context>
Suggested change
os.WriteFile(filepath.Join(repo, ".git", "info", "exclude"), []byte("node_modules\n"), 0o644)
if err := os.WriteFile(filepath.Join(repo, ".git", "info", "exclude"), []byte("node_modules\n"), 0o644); err != nil {
t.Fatal(err)
}

Comment thread internal/box/lastlook.go
starting []string
}

// remove holds dir for a removal, once no session is starting in it or

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: The doc comment says the hold lasts until the folder is gone. The release actually runs whenever removeAfter returns, including on a failed removal while the folder still exists. Reword the comment to match.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At internal/box/lastlook.go, line 23:

<comment>The doc comment says the hold lasts until the folder is gone. The release actually runs whenever `removeAfter` returns, including on a failed removal while the folder still exists. Reword the comment to match.</comment>

<file context>
@@ -0,0 +1,71 @@
+	starting []string
+}
+
+// remove holds dir for a removal, once no session is starting in it or
+// under it and no other removal holds a folder that overlaps it.
+func (g *lookGuard) remove(dir string) (release func()) {
</file context>

<-looking
// A session anywhere else starts meanwhile: a removal git takes
// minutes over holds up only its own folder.
if st := f.do("POST", "/v1/sessions", SessionRequest{Name: "bystander", Location: "cal/elsewhere", Command: "claude"}, nil); st != 200 {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: The bystander session start is called synchronously while the removal is held. If sessions elsewhere ever block, this test hangs instead of failing. Run it in a goroutine and wait with a select timeout, as the latecomer check below already does.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At internal/box/runstop_test.go, line 381:

<comment>The bystander session start is called synchronously while the removal is held. If sessions elsewhere ever block, this test hangs instead of failing. Run it in a goroutine and wait with a `select` timeout, as the `latecomer` check below already does.</comment>

<file context>
@@ -306,17 +365,22 @@ func TestStopRunsOnABoxArchiveOneAtATime(t *testing.T) {
 	<-looking
+	// A session anywhere else starts meanwhile: a removal git takes
+	// minutes over holds up only its own folder.
+	if st := f.do("POST", "/v1/sessions", SessionRequest{Name: "bystander", Location: "cal/elsewhere", Command: "claude"}, nil); st != 200 {
+		t.Fatalf("a session in another worktree during the last look: %d", st)
+	}
</file context>
Suggested change
if st := f.do("POST", "/v1/sessions", SessionRequest{Name: "bystander", Location: "cal/elsewhere", Command: "claude"}, nil); st != 200 {
bystander := make(chan int, 1)
go func() {
bystander <- f.do("POST", "/v1/sessions", SessionRequest{Name: "bystander", Location: "cal/elsewhere", Command: "claude"}, nil)
}()
select {
case st := <-bystander:
if st != 200 {
t.Errorf("a session in another worktree during the last look: %d", st)
}
case <-time.After(5 * time.Second):
t.Error("a session in another worktree waited for the last look")
}

Stener1 and others added 2 commits October 29, 2026 10:56
…still be finished

A removal stopped halfway can have deleted .gitignore files before what
they ignore, so node_modules read as new files and every such worktree
was kept as having uncommitted changes. The deleted .gitignore files are
put back as committed, from the worktree's own index, and git is asked
again, so ignore rules apply as git worktree remove would see them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… real one did

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…it afterwards

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant