Repository navigation
fix(computer): bind hosts to their session and end every wait with its task - #288
Conversation
…s task Randy's post-merge review of #273 found that hosts trusted the caller's id, grants and waits outlived Stop, budgets raced, a model could clear clicks, snapshots crossed threads, and typed text reached the decision endpoint. - Hosts belong to the authenticated session that connected them; only its reconnect replaces one, in a single state change. - Each task's end races every reply, model call, tap and planner wait; waiters cannot register after it, answered-but-unused taps are revoked, and a grant is given only while the task still runs within its time. - A stopped task keeps its computer until the host confirms; a silent host is disconnected after the reply timeout. The desktop host always reports finished and refuses grants for a task it stopped. - Steps are reserved atomically, one at a time, and a per-task deadline ends the task whether or not the host speaks. - No model verdict grants a step: unresolved clicks always need a tap. - Agents read and stop only their own thread; typed text in approvals goes only to computer:approve sessions; steps and ended approvals are capped. - The decision endpoint gets an allowlisted, capped state with the typed text hidden, and a typed step must type exactly the task's text. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Thread transfer impact✅ Thread transfer remains within every enforced ceiling.
Other scenariosAll 64 scenario measurements match All 64 scenario measurements
Baseline: Scenario and decoded snapshot size10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.
Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed. |
|
Randy reviewed head 9c2bc7d. These pass their regression tests:
234 targeted tests pass, and the new tests fail on the base. Still needed:
On #279: taking #288 as it stands would bring the desktop Stop regression into #279. |
…, hide typed text everywhere Randy's second pass on #288: - Desktop Stop keeps the task in a stopping state until the helper's `finished` arrives, forwards it, and refuses grants meanwhile; the synthetic local acknowledgement is gone. - A grant's final check and its return have nothing awaited between them; the audit line is written after the handoff. - A control stays reserved until its reply or a disconnect, even past Stop. An unanswered request ends a standalone session by stopping its helper, so it can start again. - The typed text is hidden from the goal sent to the decision endpoint and from every snapshot field, short values and question keys included (keys are aliased and mapped back). - No accessibility role clears a click; radio buttons read as `radio`. - With no text, a task types only the exact address the helper's navigation parser reads from its goal. - Snapshot strings and whole snapshots have byte caps. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Randy's round 2 at 49eb182. These now pass:
306 targeted tests and
|
… late replies, trim tool text Randy's third pass on #288: - A goal that contains the task's text is refused. A field value the helper echoes is dropped when it holds the text; text of three characters or more is replaced elsewhere. Shorter text never alters a label, and screen text that holds it is left out whole. - A read-only approval is redacted when it is made and kept that way, so it stays redacted after its task is evicted. - A control's reply entry outlives its timed-out wait, and a late reply frees the host. - An unconfirmed stop checks the reservation and removes the host in one change, then audits. - Snapshot questions are bounded (4 questions, 64 KB per decision); when running tasks and pending approvals alone exceed the cap, running tasks lose their details oldest first. - Shorter computer tool descriptions, a thread-scoped Stop, and finite numbers in the act and rect schemas: the tool list is 1,741 bytes smaller. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ardening # Conflicts: # apps/server/src/provider/agentContextBudget.json
|
Randy reviewed head 161d78f (round 3). The head is now 8dfa439, which only adds a merge of main. Changes needed. Verified fixed:
Targeted Blocking:
Nits:
Context budget: Randy flagged that the budget passes only because main grew. Since then #279 has merged into main, and after the merge this PR's tool list is 1,741 B smaller than main's, so this item is closed. |
… it is typed, questions keep their moves Randy's fourth pass on #288: - A goal is the agent's words, with `{text}` references and quoted labels as separate parts. Nothing is matched against a label or against short text, so a task typing "OK" can click the "OK" button; only long text the agent wrote out anyway is replaced in its own words. - Short typed text leaves the screen alone until a `type` step is granted (tracked by the server); after that a region that holds it is left out. - A snapshot's questions place each question and its built-in moves before any control, by serialized size, so 200 long controls cannot push out `type_text`. The bounded view is computed once per decision. - Keys too long to show are aliased like keys that name the typed text, and answers decode through the same map. - Computer tool parameter descriptions are set on the encoded side, so `goal`, `hostId` and `taskId` reach the advertised schema. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ardening # Conflicts: # apps/server/src/computer/ComputerService.ts # apps/server/src/mcp/toolkits/computer/tools.ts # apps/server/src/provider/agentContextBudget.json
…ardening # Conflicts: # apps/server/src/provider/agentContextBudget.json
|
Randy reviewed head 592033a (round 4). Changes needed. Verified fixed:
The tool list is 76,170 B against main's 77,528 B. Targeted tests pass (269 passed, 3 skipped), and so does Blocking:
Nits:
|
…ardening # Conflicts: # apps/server/src/provider/agentContextBudget.json
Randy's fifth pass on #288: - Redaction is case-insensitive. Every `{text}` reads as the marker, quoted or not, and text of three or more characters is replaced everywhere in a goal. Shorter text may appear only in a quoted label of a goal that uses `{text}`; elsewhere the goal is refused. - An echoed value the helper clipped with "…" is stripped on the server, and the helper hides clipped and case-varied forms and scrubs the end surface. - decision.answer logs the aliased keys; task.end, approval.request, approval_skipped, grant, task.progress and act are redacted before they are recorded, the typed text kept only as a digest. - hostDecide reserves the step before building the state it sends, from the reserved task. - Reader approvals hide the text in the app and file path; snapshot choices are filled round-robin, and tab and app keys count as controls. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
This PR changes the test suite by +23 test cases (browser 0, unit +3, server +20). Keep tests that would fail for a real reason, and test logic below the UI where it can be. See Testing. |
|
Randy reviewed head 82f4844 (round 5). Changes needed. Verified fixed:
CI: 23 checks pass. Targeted tests: 340 passed, 3 skipped. The tool list is 75,686 B against main's 77,044 B. Blocking:
Nits:
|
Randy's post-merge review of #273 listed 7 blockers and 1 nit: hosts trusted a caller-supplied id, grants and waits outlived Stop, budgets raced, a model could clear clicks, snapshots crossed threads, and prepared text reached the decision endpoint. #279 merged into
main(a441afd), so this now targetsmain.Fixes
ws.ts): every host RPC carries the caller's authenticated session id. Calls from any other session are refused (no_host). Only that session's reconnect replaces its host, in one state change; another session naming the same device adds its own host.ComputerApprovals.cancelTask). The desktop host refuses a grant for a task it has stopped (the helper already checks its abort signal).endedsignal that every reply, decision-model call, tap and planner wait races against. Waiters cannot register once a task has ended, and an interrupted wait cleans up after itself. A stopped task keeps its host reserved until the host reportsfinished; a host that has not reported within the reply timeout (20 s) is disconnected. The desktop host now always forwardsfinished, since that is the shutdown confirmation.decider.judge,afterJudge, and the"judge"value ofComputerVerdict). A click the deterministic policy cannot show is safe always needs a tap. The decision endpoint still chooses next steps but never grants one.computer_stopare scoped to the caller's thread (status(threadId),stop({ threadId })).computer.subscribesends an approval's typed text only tocomputer:approvesessions. Steps per task are capped at 40 and finished approvals per snapshot at 10.typestep must type exactly the task's text, or, when the task has none, the address its goal names; otherwise the task ends blocked.TestClockis reported failed after 10 s, so the step goes to the planner.Tests: one regression test per blocker in
ComputerService.test.ts, plus a revoked-approval assertion inComputerApprovals.test.ts, policy golden rows (judge→tap), and theTestClockdecider test. With the base service, policy, approvals and decider swapped back in, all six new service tests, the approvals assertion and four golden rows fail. Targetedvp test run(136 tests at the time) andvp run check:changed origin/feat/computer-usepass.Round 3 (Randy's third review)
hideTyped): no more token-boundary replacement.startUserefuses a goal that contains the task'stext(as written for 3+ characters, as a word of its own for shorter). A field value the helper echoes (containing “…”) is dropped when it holds the text, at any length. Text of 3+ characters is still replaced elsewhere. Shorter text never alters a label, so a "No" button stays "No", and screen text that holds it (visible text, popup, title, URL, note, screen text) is left out whole. The goal is sent and shown as written, since it can no longer carry the text.ComputerApprovals.createtakes the reader binding, made when the approval is created, andlist({ revealTypedText })returns one or the other. Eviction of the source task no longer matters.endrun after.computer_useandcomputer_actwalkthroughs are shorter,status,controlandlookare tighter,computer_stopsays it is thread-scoped, and both say clicks wait for the user when approvals are on.ComputerAct.x/y/amountandComputerRectareSchema.Finite, which drops a 68 BInfinity/NaNunion per number field. The MCP tool list went from 68,635 to 66,894 B (−1,741 B). Against the 62,836 B main baseline in the failed run, the remaining increase is 4,058 B. After feat(computer): built-in computer use for agents #279 merged,mainmeasures 77,116 B and this branch measures 75,375 B, so the tool list is now 1,741 B smaller thanmainand no label is needed. Parameterdescriptionannotations on refined schemas (for examplegoal) never reachinputSchema, so they cost nothing and the model never sees them. That is worth a separate fix on feat(computer): built-in computer use for agents #279.statusbefore any competing start, and the standalone test now runs through the 10 s restart and a second, answered control.Round 3 tests: five new tests in
ComputerService.test.ts: goal refusal with echo and label handling plus long-text keys, reader approval after eviction, late control reply, atomic unconfirmed stop, and question bounds. With the head49eb1829bsources swapped back in, all five fail fast.ComputerApprovals.test.tsand the MCP handler fixture follow the new API and goal rule. Targeted suites pass (server computer, MCP computer and context budget 149; evals 1,004), and so doesvp run check:changed origin/feat/computer-use. Computer-use evals:--contextand--dry-runof all 12 cases pass. The paid run has not been done, because this machine has noANTHROPIC_API_KEYand the eval refuses the proxy token. Thecu-answers-offered-keyprior turn no longer puts its text in the goal.Round 4 (Randy's fourth review)
{text}references to the prepared text and quoted control labels as separate parts. Everyone but the helper reads{text}as[typed text]. Labels and short text are never matched, so a task typingOKcan still click the "OK" button, and punctuation no longer matters. Long text the agent wrote out anyway is still replaced in its own words. A goal that refers to{text}with no text is refused.computer_usesays to write{text}.typestep is granted, in the same state change as the grant. Before that, screen text is sent and shown whole (an invoice's number and balance survive prepared texta). After it, only regions that hold short text are left out. An echoed field value (containing “…”) is always dropped when it holds the text.c<n>control key) are placed first, then controls fill the rest in order. Sizes are measured as serialized JSON, so 200 long localized controls can no longer push outtype_text. The bounded view is computed once per decision rather than on every publish.key_<n>) like keys that name the typed text, before projecting, and answers decode through the same map.Schema.annotateEncoded. A handler test checks the advertisedgoal,hostIdandtaskIddescriptions. Other toolkits that annotate refined ids (for exampleThreadIdin issues and theme) have the same gap; that is not changed here.Round 4 tests: four new tests in
ComputerService.test.ts(goal reference with an "OK" label, screen kept until typed, built-in moves under the 200-control budget, long-key round trip) and the advertised-schema check inhandlers.test.ts. With the8dfa43997sources swapped back in, all five fail. Mergedmainagain (#302 chords, #303, #304 sheets/DevTools/choose_file) and folded their additions into the shortercomputer_look/computer_acttext. Targeted suites pass (server computer, MCP computer and budget 172; desktop computer 5; computer-helper 160, 3 skipped; evals 1,007), as doesvp run check:changed origin/main. Budget: 76,192 B against main's 77,550 B, so the tool list is 1,358 B smaller than main.Round 5 (Randy's fifth review)
hideTypedis case-insensitive (escapedgiregex). Every{text}reads as[typed text], quoted or not. Text of 3+ characters is replaced everywhere in a goal, quoted spans included. Shorter text may appear only in a quoted label of a goal that uses{text}. Anywhere else (type 42 into the PIN box,say a. to Dana, a quoted"OK"without{text}), the goal is refused with a hint to use{text}.record():decision.answer(logged under the aliased ids and keys),task.end(reason and surface),approval.request,approval_skippedandgrant(the reader binding, typed text only astextDigest),task.progressandact(how). The helper'scontroller.tsnow scrubs the endsurfacetoo.hostDeciderace: the step is reserved first, and the outbound state is built from the reserved task. The screen text it records is read inside the reservation.without()also hides the 60- and 90-character forms thatclipwrites, in any case. The server drops an echoed value whose…-cut prefix starts the typed text.readerBindingredactsbinding.appandaction.path.boundedQuestionsfills choices round-robin across questions, built-in moves first, andCONTROL_CHOICEis/^[cta]\d+$/.Round 5 tests: five new service tests (quoted secret plus quoted
{text}plus upper-cased and clipped echo; short text without{text}, including punctuation; every listed audit event; the skipped-approval audit; reader app and path; round-robin budget witht/akeys) and a helperwithout()test for a 72-character secret and a case echo. All six fail with the592033ad5sources swapped in. ThehostDeciderace has no deterministic test. On the old code the window was a scheduler preemption between two synchronous steps, which a test cannot force without a seam. The fix removes the window by construction. Mergedmainagain (#291, #292). Targeted suites pass (server 177, desktop 5, computer-helper 161 with 3 skipped), as doesvp run check:changed origin/main. Budget: 75,686 B against main's 77,044 B, so the tool list is 1,358 B smaller.For #279 (and the already-merged #283 work) to pick up
finishedafter a serverstop. Until they do, the computer stays reserved and the host is disconnected after 20 s. The desktop change is included here.ComputerService.statusandsubscribeare now functions. Fakes in evals and tests are updated here.ComputerVerdictno longer includes"judge".[typed text]in its place. Agents should not rely on reading back their own goal.attachLocalHostrequiresend. Any other in-process host in feat(computer): built-in computer use for agents #279 must pass one that closes its session.@t3tools/computer-helper/navigation. Packaging in feat(computer): built-in computer use for agents #279 must keep that subpath.{text}. Since round 5, a goal with short text (1–2 characters) must use{text}, and that text may appear elsewhere only in a quoted label.ComputerApprovals.create({ binding, readerBinding })andlist({ revealTypedText })replace the old signatures.Round 2 (Randy's second review)
ComputerHostProcess,DesktopComputerHost): Stop puts the task in a stopping state that lasts until the helper'sfinishedarrives. Thatfinishedis forwarded, and grants are refused meanwhile. The synthetic local acknowledgement is removed.ok: truehave nothing awaited between them. Thegrantaudit line is written after the handoff.attachLocalHosttakesend), so the session ends and the helper restarts.tab,rowandtextbox; every click without a lexicon match needs a tap. There is no trusted surface metadata yet to tell focus/select-only clicks apart. The helper mapsAXRadioButtonto a newradiorole.navigationTextmoved to@t3tools/computer-helper/navigation. With no text, atypestep must equal the address it parses from the goal exactly.Round 2 tests: six new tests in
ComputerService.test.ts(grant handoff before audit, control in flight after Stop, silent standalone helper, typed text in goal/keys/short values, exact-address typing, title cap), the desktop stopping test inComputerHostProcess.test.ts(forwardedfinished, late grant refused), the radio mapping inbackends.test.ts, and three policy golden rows (textbox/tab/row→ tap). With the head9c2bc7d696sources swapped back in, all eleven fail. Targeted suites pass (server computer + MCP 648, desktop computer 5, computer-helper 156), as doesvp run check:changed origin/feat/computer-use.Opened with Claude Opus 5.5 in Ada (Claude Code harness).
🤖 Generated with Claude Code