Skip to content

fix(computer): bind hosts to their session and end every wait with its task - #288

Merged
jayleaton merged 9 commits into
mainfrom
fix/computer-use-a2-hardening
Nov 21, 2026
Merged

jayleaton merged 9 commits into
mainfrom
fix/computer-use-a2-hardening

Conversation

@jayleaton

@jayleaton jayleaton commented Nov 18, 2026 •

Copy link
Copy Markdown
Owner

Randy's post-merge review of #273 listed 7 blockers and 1 nit: hosts trusted a caller-supplied id, grants and waits outlived Stop, budgets raced, a model could clear clicks, snapshots crossed threads, and prepared text reached the decision endpoint. #279 merged into main (a441afd), so this now targets main.

Fixes

  1. Host identity (ws.ts): every host RPC carries the caller's authenticated session id. Calls from any other session are refused (no_host). Only that session's reconnect replaces its host, in one state change; another session naming the same device adds its own host.
  2. Grant after Stop: a grant is checked against the current task state at the moment it is given. Ending a task now also revokes approved-but-unused taps (ComputerApprovals.cancelTask). The desktop host refuses a grant for a task it has stopped (the helper already checks its abort signal).
  3. Teardown: each task has an ended signal that every reply, decision-model call, tap and planner wait races against. Waiters cannot register once a task has ended, and an interrupted wait cleans up after itself. A stopped task keeps its host reserved until the host reports finished; a host that has not reported within the reply timeout (20 s) is disconnected. The desktop host now always forwards finished, since that is the shutdown confirmation.
  4. Budgets: the time and step checks and the counter update are one atomic reservation, and a task can have only one step in flight. A per-task timer enforces working and wall-clock time even when the host is silent. A grant past the deadline is refused.
  5. Model verdicts: the model judge is removed (decider.judge, afterJudge, and the "judge" value of ComputerVerdict). A click the deterministic policy cannot show is safe always needs a tap. The decision endpoint still chooses next steps but never grants one.
  6. Snapshots: MCP reads and computer_stop are scoped to the caller's thread (status(threadId), stop({ threadId })). computer.subscribe sends an approval's typed text only to computer:approve sessions. Steps per task are capped at 40 and finished approvals per snapshot at 10.
  7. Outbound text: the decision endpoint receives an allowlisted, capped state (goal, screen fields, steps), with the typed text hidden in that state and in the questions. A type step must type exactly the task's text, or, when the task has none, the address its goal names; otherwise the task ends blocked.
  8. Nit: a stalled provider under TestClock is reported failed after 10 s, so the step goes to the planner.

Tests: one regression test per blocker in ComputerService.test.ts, plus a revoked-approval assertion in ComputerApprovals.test.ts, policy golden rows (judge → tap), and the TestClock decider test. With the base service, policy, approvals and decider swapped back in, all six new service tests, the approvals assertion and four golden rows fail. Targeted vp test run (136 tests at the time) and vp run check:changed origin/feat/computer-use pass.

Round 3 (Randy's third review)

  1. Short typed text (hideTyped): no more token-boundary replacement. startUse refuses a goal that contains the task's text (as written for 3+ characters, as a word of its own for shorter). A field value the helper echoes (containing “…”) is dropped when it holds the text, at any length. Text of 3+ characters is still replaced elsewhere. Shorter text never alters a label, so a "No" button stays "No", and screen text that holds it (visible text, popup, title, URL, note, screen text) is left out whole. The goal is sent and shown as written, since it can no longer carry the text.
  2. Reader approvals: ComputerApprovals.create takes the reader binding, made when the approval is created, and list({ revealTypedText }) returns one or the other. Eviction of the source task no longer matters.
  3. Late control replies: a control's reply entry stays while its host is still reserved for it. A matching late reply frees the host instead of being rejected, so the stop timeout no longer disconnects a responsive helper.
  4. Unconfirmed stop: the reservation check and host removal happen in one state change; the audit line, the disconnect follow-up and end run after.
  5. Context budget: the computer_use and computer_act walkthroughs are shorter, status, control and look are tighter, computer_stop says it is thread-scoped, and both say clicks wait for the user when approvals are on. ComputerAct.x/y/amount and ComputerRect are Schema.Finite, which drops a 68 B Infinity/NaN union per number field. The MCP tool list went from 68,635 to 66,894 B (−1,741 B). Against the 62,836 B main baseline in the failed run, the remaining increase is 4,058 B. After feat(computer): built-in computer use for agents #279 merged, main measures 77,116 B and this branch measures 75,375 B, so the tool list is now 1,741 B smaller than main and no label is needed. Parameter description annotations on refined schemas (for example goal) never reach inputSchema, so they cost nothing and the model never sees them. That is worth a separate fix on feat(computer): built-in computer use for agents #279.
  6. Snapshot questions: at most 4 questions, keys at 64 B, instructions at 2,000 B, choice lines at 400 B, and choices kept in order while a decision fits 64 KB. When running tasks and pending approvals alone are over the 256 KB cap, running tasks lose steps, screen text and questions, oldest first. A snapshot still over the cap after that is sent as is, since dropping a running task or a pending approval would hide something the user must stop or answer.
  7. Tests: the in-flight and standalone tests assert the reservation from status before any competing start, and the standalone test now runs through the 10 s restart and a second, answered control.

Round 3 tests: five new tests in ComputerService.test.ts: goal refusal with echo and label handling plus long-text keys, reader approval after eviction, late control reply, atomic unconfirmed stop, and question bounds. With the head 49eb1829b sources swapped back in, all five fail fast. ComputerApprovals.test.ts and the MCP handler fixture follow the new API and goal rule. Targeted suites pass (server computer, MCP computer and context budget 149; evals 1,004), and so does vp run check:changed origin/feat/computer-use. Computer-use evals: --context and --dry-run of all 12 cases pass. The paid run has not been done, because this machine has no ANTHROPIC_API_KEY and the eval refuses the proxy token. The cu-answers-offered-key prior turn no longer puts its text in the goal.

Round 4 (Randy's fourth review)

  1. Structured goal: the goal refusal is gone. A goal is the agent's words, with {text} references to the prepared text and quoted control labels as separate parts. Everyone but the helper reads {text} as [typed text]. Labels and short text are never matched, so a task typing OK can still click the "OK" button, and punctuation no longer matters. Long text the agent wrote out anyway is still replaced in its own words. A goal that refers to {text} with no text is refused. computer_use says to write {text}.
  2. Screen context: the server marks a task once a type step is granted, in the same state change as the grant. Before that, screen text is sent and shown whole (an invoice's number and balance survive prepared text a). After it, only regions that hold short text are left out. An echoed field value (containing “…”) is always dropped when it holds the text.
  3. Question budget: each shown question and its built-in moves (any key that is not the helper's c<n> control key) are placed first, then controls fill the rest in order. Sizes are measured as serialized JSON, so 200 long localized controls can no longer push out type_text. The bounded view is computed once per decision rather than on every publish.
  4. Long keys (nit): keys longer than 64 B are aliased (key_<n>) like keys that name the typed text, before projecting, and answers decode through the same map.
  5. Advertised descriptions (nit): descriptions on refined or trimmed schemas were set on the decoded side, so JSON Schema (built from the encoded side) dropped them. The computer tools now use Schema.annotateEncoded. A handler test checks the advertised goal, hostId and taskId descriptions. Other toolkits that annotate refined ids (for example ThreadId in issues and theme) have the same gap; that is not changed here.

Round 4 tests: four new tests in ComputerService.test.ts (goal reference with an "OK" label, screen kept until typed, built-in moves under the 200-control budget, long-key round trip) and the advertised-schema check in handlers.test.ts. With the 8dfa43997 sources swapped back in, all five fail. Merged main again (#302 chords, #303, #304 sheets/DevTools/choose_file) and folded their additions into the shorter computer_look/computer_act text. Targeted suites pass (server computer, MCP computer and budget 172; desktop computer 5; computer-helper 160, 3 skipped; evals 1,007), as does vp run check:changed origin/main. Budget: 76,192 B against main's 77,550 B, so the tool list is 1,358 B smaller than main.

Round 5 (Randy's fifth review)

  1. Goal leaks: hideTyped is case-insensitive (escaped gi regex). Every {text} reads as [typed text], quoted or not. Text of 3+ characters is replaced everywhere in a goal, quoted spans included. Shorter text may appear only in a quoted label of a goal that uses {text}. Anywhere else (type 42 into the PIN box, say a. to Dana, a quoted "OK" without {text}), the goal is refused with a hint to use {text}.
  2. Audit leaks: these events are redacted before record(): decision.answer (logged under the aliased ids and keys), task.end (reason and surface), approval.request, approval_skipped and grant (the reader binding, typed text only as textDigest), task.progress and act (how). The helper's controller.ts now scrubs the end surface too.
  3. hostDecide race: the step is reserved first, and the outbound state is built from the reserved task. The screen text it records is read inside the reservation.
  4. Clipped secrets: the helper's without() also hides the 60- and 90-character forms that clip writes, in any case. The server drops an echoed value whose …-cut prefix starts the typed text.
  5. Nits: readerBinding redacts binding.app and action.path. boundedQuestions fills choices round-robin across questions, built-in moves first, and CONTROL_CHOICE is /^[cta]\d+$/.

Round 5 tests: five new service tests (quoted secret plus quoted {text} plus upper-cased and clipped echo; short text without {text}, including punctuation; every listed audit event; the skipped-approval audit; reader app and path; round-robin budget with t/a keys) and a helper without() test for a 72-character secret and a case echo. All six fail with the 592033ad5 sources swapped in. The hostDecide race has no deterministic test. On the old code the window was a scheduler preemption between two synchronous steps, which a test cannot force without a seam. The fix removes the window by construction. Merged main again (#291, #292). Targeted suites pass (server 177, desktop 5, computer-helper 161 with 3 skipped), as does vp run check:changed origin/main. Budget: 75,686 B against main's 77,044 B, so the tool list is 1,358 B smaller.

For #279 (and the already-merged #283 work) to pick up

  • Desktop and standalone hosts must report finished after a server stop. Until they do, the computer stays reserved and the host is disconnected after 20 s. The desktop change is included here.
  • More taps: with approvals on, every click the rules cannot clear now asks. This most affects feat(computer): agents use the default browser and saved sign-ins #283's saved sign-ins (autofill popup lines).
  • ComputerService.status and subscribe are now functions. Fakes in evals and tests are updated here.
  • ComputerVerdict no longer includes "judge".
  • Round 2: every click now needs a tap, including tabs, rows, text boxes and radio buttons. feat(computer): built-in computer use for agents #279's flows and evals that expect auto-granted clicks need updating, and auto-granting needs trusted surface metadata.
  • Round 2: the goal is hidden wherever it contains the typed text, so the MCP tools and clients see [typed text] in its place. Agents should not rely on reading back their own goal.
  • Round 2: attachLocalHost requires end. Any other in-process host in feat(computer): built-in computer use for agents #279 must pass one that closes its session.
  • Round 2: a control that never replies keeps the computer reserved for up to 40 s (reply timeout plus the stop timeout) before the host is disconnected.
  • Round 2: the server imports @t3tools/computer-helper/navigation. Packaging in feat(computer): built-in computer use for agents #279 must keep that subpath.
  • Round 4 replaces the round 3 goal refusal: goals refer to the prepared text as {text}. Since round 5, a goal with short text (1–2 characters) must use {text}, and that text may appear elsewhere only in a quoted label.
  • Round 3: ComputerApprovals.create({ binding, readerBinding }) and list({ revealTypedText }) replace the old signatures.
  • Round 3: with typed text of one or two characters, screen text that contains it is not sent to the decision endpoint, so the planner sees less on those tasks.
  • Round 3: the paid computer-use eval run, and acceptance of the remaining 4,058 B tool-list increase against the old main baseline.

Round 2 (Randy's second review)

  1. Desktop Stop (ComputerHostProcess, DesktopComputerHost): Stop puts the task in a stopping state that lasts until the helper's finished arrives. That finished is forwarded, and grants are refused meanwhile. The synthetic local acknowledgement is removed.
  2. Grant handoff: the final running/deadline check and ok: true have nothing awaited between them. The grant audit line is written after the handoff.
  3. Controls in flight: a control (or task) whose frame was sent keeps its host reserved until the host replies or disconnects, even past Stop. An unanswered request leads to the unconfirmed-stop disconnect. For a standalone server that now stops the helper (attachLocalHost takes end), so the session ends and the helper restarts.
  4. Prepared text: hidden from the goal sent to the decision endpoint, from every task field in every snapshot (goal, reason, surface, steps, screen text), and from read-only approval bindings (title, URL, goal, why, labels). Text of one or two characters is hidden where it stands alone. Question ids and choice keys that contain it are aliased, and answers are mapped back. User docs now say the goal goes to the configured decision endpoint.
  5. Roles: no role clears a click any more, including tab, row and textbox; every click without a lexicon match needs a tap. There is no trusted surface metadata yet to tell focus/select-only clicks apart. The helper maps AXRadioButton to a new radio role.
  6. URL fallback: navigationText moved to @t3tools/computer-helper/navigation. With no text, a type step must equal the address it parses from the goal exactly.
  7. Nit: binding fields are capped (400 B, URL 2,000 B), snapshot strings at 4,000 B (steps 400 B), and a whole snapshot at 256 KB, dropping the oldest ended tasks and then ended approvals.

Round 2 tests: six new tests in ComputerService.test.ts (grant handoff before audit, control in flight after Stop, silent standalone helper, typed text in goal/keys/short values, exact-address typing, title cap), the desktop stopping test in ComputerHostProcess.test.ts (forwarded finished, late grant refused), the radio mapping in backends.test.ts, and three policy golden rows (textbox/tab/row → tap). With the head 9c2bc7d696 sources swapped back in, all eleven fail. Targeted suites pass (server computer + MCP 648, desktop computer 5, computer-helper 156), as does vp run check:changed origin/feat/computer-use.

Opened with Claude Opus 5.5 in Ada (Claude Code harness).

🤖 Generated with Claude Code

…s task

Randy's post-merge review of #273 found that hosts trusted the caller's id,
grants and waits outlived Stop, budgets raced, a model could clear clicks,
snapshots crossed threads, and typed text reached the decision endpoint.

- Hosts belong to the authenticated session that connected them; only its
  reconnect replaces one, in a single state change.
- Each task's end races every reply, model call, tap and planner wait;
  waiters cannot register after it, answered-but-unused taps are revoked,
  and a grant is given only while the task still runs within its time.
- A stopped task keeps its computer until the host confirms; a silent host
  is disconnected after the reply timeout. The desktop host always reports
  finished and refuses grants for a task it stopped.
- Steps are reserved atomically, one at a time, and a per-task deadline
  ends the task whether or not the host speaks.
- No model verdict grants a step: unresolved clicks always need a tap.
- Agents read and stop only their own thread; typed text in approvals goes
  only to computer:approve sessions; steps and ended approvals are capped.
- The decision endpoint gets an allowlisted, capped state with the typed
  text hidden, and a typed step must type exactly the task's text.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added vouch:trusted Author is trusted by repo permissions or the VOUCHED list. size:XXL labels Nov 18, 2026
@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

Provider Metric Main baseline This PR Impact PR ceiling
Codex Total thread wire 5.0 KiB 5.0 KiB 0 B (0.0%) 6.8 KiB ✅
Codex Thread snapshot wire 3.8 KiB 3.8 KiB 0 B (0.0%) 4.9 KiB ✅
Codex Live turn WebSocket wire 1.2 KiB 1.2 KiB 0 B (0.0%) 2.0 KiB ✅
Codex Live turn WebSocket decoded 21.5 KiB 21.5 KiB 0 B (0.0%) 24.4 KiB ✅
Codex Live turn messages 1 1 0 (0.0%) 8 ✅
Claude Total thread wire 5.0 KiB 5.0 KiB 0 B (0.0%) 6.8 KiB ✅
Claude Thread snapshot wire 3.8 KiB 3.8 KiB 0 B (0.0%) 4.9 KiB ✅
Claude Live turn WebSocket wire 1.2 KiB 1.2 KiB 0 B (0.0%) 2.0 KiB ✅
Claude Live turn WebSocket decoded 21.9 KiB 21.9 KiB 0 B (0.0%) 24.4 KiB ✅
Claude Live turn messages 1 1 0 (0.0%) 8 ✅

Other scenarios

All 64 scenario measurements match main.

All 64 scenario measurements
Provider Scenario Metric Main baseline This PR Impact PR ceiling
Codex short turn resume WebSocket decoded 10.5 KiB 10.5 KiB 0 B (0.0%) 12.7 KiB ✅
Codex short turn resume WebSocket messages 2 2 0 (0.0%) 2 ✅
Codex long turn running, cold open HTTP decoded 281.7 KiB 281.7 KiB 0 B (0.0%) 302.7 KiB ✅
Codex long turn running, cold open rows 75 75 0 (0.0%) 75 ✅
Codex long turn running, history page HTTP decoded 92.4 KiB 92.4 KiB 0 B (0.0%) 107.4 KiB ✅
Codex long turn running, history page rows 75 75 0 (0.0%) 75 ✅
Codex long turn finished, cold open HTTP decoded 381.5 KiB 381.5 KiB 0 B (0.0%) 410.2 KiB ✅
Codex long turn finished, cold open rows 151 151 0 (0.0%) 151 ✅
Codex long turn finished, socket cold open WebSocket decoded 381.6 KiB 381.6 KiB 0 B (0.0%) 410.2 KiB ✅
Codex long turn finished, socket cold open WebSocket messages 2 2 0 (0.0%) 2 ✅
Codex long turn finished, socket cold open rows 151 151 0 (0.0%) 151 ✅
Codex long turn finished, work page HTTP decoded 186.2 KiB 186.2 KiB 0 B (0.0%) 200.2 KiB ✅
Codex long turn finished, work page rows 150 150 0 (0.0%) 150 ✅
Codex idle thread resume WebSocket decoded 65 B 65 B 0 B (0.0%) 200 B ✅
Codex idle thread resume WebSocket messages 1 1 0 (0.0%) 1 ✅
Codex fleet shell snapshot HTTP decoded 42.4 KiB 42.4 KiB 0 B (0.0%) 53.7 KiB ✅
Codex fleet shell snapshot rows 27 27 0 (0.0%) 27 ✅
Codex fleet shell subscribe WebSocket decoded 65 B 65 B 0 B (0.0%) 200 B ✅
Codex fleet shell subscribe WebSocket messages 1 1 0 (0.0%) 1 ✅
Codex fleet shell resume check HTTP decoded 18 B 18 B 0 B (0.0%) 100 B ✅
Codex fleet shell resume check rows 0 0 0 0 ✅
Codex stale shell reload HTTP decoded 42.4 KiB 42.4 KiB 0 B (0.0%) 53.7 KiB ✅
Codex stale shell reload rows 27 27 0 (0.0%) 27 ✅
Codex stale shell resume WebSocket decoded 140 B 140 B 0 B (0.0%) 400 B ✅
Codex stale shell resume WebSocket messages 2 2 0 (0.0%) 2 ✅
Codex stale todo list cold open HTTP decoded 72.4 KiB 72.4 KiB 0 B (0.0%) 83.0 KiB ✅
Codex stale todo list cold open rows 75 75 0 (0.0%) 75 ✅
Codex stale todo list cold open nodes 2 2 0 (0.0%) 4 ✅
Codex stale todo list cold open subagents 2 2 0 (0.0%) 4 ✅
Codex stale todo list cold open plans 1 1 0 (0.0%) 2 ✅
Codex stale todo list cold open providerTurns 1 1 0 (0.0%) 2 ✅
Codex stale todo list cold open checkpoints 1 1 0 (0.0%) 2 ✅
Claude short turn resume WebSocket decoded 10.6 KiB 10.6 KiB 0 B (0.0%) 12.7 KiB ✅
Claude short turn resume WebSocket messages 2 2 0 (0.0%) 2 ✅
Claude long turn running, cold open HTTP decoded 283.3 KiB 283.3 KiB 0 B (0.0%) 302.7 KiB ✅
Claude long turn running, cold open rows 75 75 0 (0.0%) 75 ✅
Claude long turn running, history page HTTP decoded 93.4 KiB 93.4 KiB 0 B (0.0%) 107.4 KiB ✅
Claude long turn running, history page rows 75 75 0 (0.0%) 75 ✅
Claude long turn finished, cold open HTTP decoded 384.2 KiB 384.2 KiB 0 B (0.0%) 410.2 KiB ✅
Claude long turn finished, cold open rows 151 151 0 (0.0%) 151 ✅
Claude long turn finished, socket cold open WebSocket decoded 384.3 KiB 384.3 KiB 0 B (0.0%) 410.2 KiB ✅
Claude long turn finished, socket cold open WebSocket messages 2 2 0 (0.0%) 2 ✅
Claude long turn finished, socket cold open rows 151 151 0 (0.0%) 151 ✅
Claude long turn finished, work page HTTP decoded 188.1 KiB 188.1 KiB 0 B (0.0%) 200.2 KiB ✅
Claude long turn finished, work page rows 150 150 0 (0.0%) 150 ✅
Claude idle thread resume WebSocket decoded 66 B 66 B 0 B (0.0%) 200 B ✅
Claude idle thread resume WebSocket messages 1 1 0 (0.0%) 1 ✅
Claude fleet shell snapshot HTTP decoded 43.0 KiB 43.0 KiB 0 B (0.0%) 53.7 KiB ✅
Claude fleet shell snapshot rows 27 27 0 (0.0%) 27 ✅
Claude fleet shell subscribe WebSocket decoded 66 B 66 B 0 B (0.0%) 200 B ✅
Claude fleet shell subscribe WebSocket messages 1 1 0 (0.0%) 1 ✅
Claude fleet shell resume check HTTP decoded 18 B 18 B 0 B (0.0%) 100 B ✅
Claude fleet shell resume check rows 0 0 0 0 ✅
Claude stale shell reload HTTP decoded 43.0 KiB 43.0 KiB 0 B (0.0%) 53.7 KiB ✅
Claude stale shell reload rows 27 27 0 (0.0%) 27 ✅
Claude stale shell resume WebSocket decoded 140 B 140 B 0 B (0.0%) 400 B ✅
Claude stale shell resume WebSocket messages 2 2 0 (0.0%) 2 ✅
Claude stale todo list cold open HTTP decoded 73.4 KiB 73.4 KiB 0 B (0.0%) 83.0 KiB ✅
Claude stale todo list cold open rows 75 75 0 (0.0%) 75 ✅
Claude stale todo list cold open nodes 2 2 0 (0.0%) 4 ✅
Claude stale todo list cold open subagents 2 2 0 (0.0%) 4 ✅
Claude stale todo list cold open plans 1 1 0 (0.0%) 2 ✅
Claude stale todo list cold open providerTurns 1 1 0 (0.0%) 2 ✅
Claude stale todo list cold open checkpoints 1 1 0 (0.0%) 2 ✅

Baseline: 48a0854 · PR result: 82f4844 · Source CI: success

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.

  • Codex decoded thread snapshot: 108.5 KiB
  • Claude decoded thread snapshot: 108.8 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.

@jayleaton

Copy link
Copy Markdown
Owner Author

Randy reviewed head 9c2bc7d. These pass their regression tests:

  • session ownership
  • concurrent step reservation
  • MCP Stop and status scoped to the calling thread
  • the 10 s TestClock fallback

234 targeted tests pass, and the new tests fail on the base. Still needed:

  1. Blocking: apps/desktop/src/computer/ComputerHostProcess.ts:593 and DesktopComputerHost.ts:226. Desktop Stop drops the shutdown acknowledgement. The parent clears the task before the child reports finished, so the server holds the host for 20 s and then disconnects it. Fix: keep the task in a stopping state until the child confirms, remove the synthetic local-Stop acknowledgement, and test both finished forwarding and late-grant rejection through the desktop path.
  2. Blocking: ComputerService.ts:1500. A grant can still return after Stop or the deadline, because the audit I/O is awaited between the state check and ok: true. Fix: serialize the final grant handoff with Stop and the deadline, with nothing awaited in between.
  3. Blocking: ComputerService.ts:742 and :1854. Stop frees a host that's running a fixed control before the native call returns. When a standalone helper goes silent, the timeout removes its registration, but the child process and its stream keep running and the restart never fires. Fix: keep controls reserved until they reply or disconnect, and have the timeout close the standalone session so it can restart.
  4. Blocking: ComputerService.ts:294 and :1864. Prepared text still reaches the provider and read-only clients when it appears in the goal. Fix: redact it from outbound goals and snapshot fields, including short values and question keys. Also document that standalone goal content is sent to the configured provider.
  5. Blocking: ComputerPolicy.ts:103 and packages/computer-helper/src/desktop.ts:287. Screen roles still approve clicks on their own. A model-picked click on an untrusted surface auto-grants on the tab, row or textbox roles, and the adapter maps generic radio buttons to tab. Fix: require a tap unless trusted surface metadata shows the action only focuses or selects.
  6. Blocking: ComputerService.ts:334. The URL-only typing fallback accepts any fragment of the goal. Fix: use the helper's navigation parser and require an exact match on the parsed address.
  7. Nit: ComputerService.ts:1285 and :1529. The snapshot row caps don't limit payload size. Fix: add per-field and total byte limits.

On #279: taking #288 as it stands would bring the desktop Stop regression into #279.

…, hide typed text everywhere

Randy's second pass on #288:

- Desktop Stop keeps the task in a stopping state until the helper's
  `finished` arrives, forwards it, and refuses grants meanwhile; the
  synthetic local acknowledgement is gone.
- A grant's final check and its return have nothing awaited between them;
  the audit line is written after the handoff.
- A control stays reserved until its reply or a disconnect, even past Stop.
  An unanswered request ends a standalone session by stopping its helper,
  so it can start again.
- The typed text is hidden from the goal sent to the decision endpoint and
  from every snapshot field, short values and question keys included
  (keys are aliased and mapped back).
- No accessibility role clears a click; radio buttons read as `radio`.
- With no text, a task types only the exact address the helper's
  navigation parser reads from its goal.
- Snapshot strings and whole snapshots have byte caps.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jayleaton

Copy link
Copy Markdown
Owner Author

Randy's round 2 at 49eb182. These now pass:

  • desktop Stop
  • grant/audit ordering
  • the role policy
  • the exact-address check
  • question and choice aliases, including a planner ID collision
  • the packaged navigation parser

306 targeted tests and check:changed pass. Still needed:

  1. Blocking: ComputerService.ts:304. Redacting short text both leaks and over-redacts. A prepared ok leaks when a field echoes draftok. A prepared No hides a real "No" button from the task. Fix: keep prepared input separate from goals and control labels, and leave out echoed field values that can't be safely told apart from the payload.
  2. Blocking: ComputerService.ts:704. A read-only approval shows on my way again once ten newer tasks evict the task that supplied its redaction. Fix: store a redacted approval projection when the approval is created, and keep it for the approval's lifetime.
  3. Blocking: ComputerService.ts:1029. A standalone control that replies just after the 20 s timeout leaves the host busy. Request cleanup threw away the reply entry, so the server rejects the confirmation and later disconnects a helper that was responding. Fix: keep the entry through teardown, and release the host when a matching late reply arrives.
  4. Blocking: ComputerService.ts:831. With the shutdown audit paused, the old task can acknowledge Stop and a new task can start. When the audit resumes, it disconnects the host that is running the new task. Fix: check the reservation and remove the host atomically, then audit.
  5. Blocking for the stack: mcp/toolkits/computer/tools.ts:108. The agent context budget is 5,799 bytes over main, and the tool descriptions alone are 3,569 bytes. Fix: shorten the computer_use and computer_act walkthroughs, drop duplicated parameter guidance, and say that Stop is thread-scoped and that clicks wait for the user when approvals are on. Then rerun the computer-use evals and get maintainer acceptance for whatever increase remains.
  6. Nit: ComputerService.ts:478. One active planner decision produced a 300,884-byte snapshot, over the 256,000-byte cap. Fix: bound question counts and strings, and define what happens when active rows alone exceed the cap.
  7. Nit: ComputerService.test.ts:1272. Fix: assert the reservation before the competing start, so the regression fails fast instead of hanging. Also extend the restart test through a second control. Randy confirmed that the 10 s restart works with a virtual clock.

… late replies, trim tool text

Randy's third pass on #288:

- A goal that contains the task's text is refused. A field value the helper
  echoes is dropped when it holds the text; text of three characters or
  more is replaced elsewhere. Shorter text never alters a label, and
  screen text that holds it is left out whole.
- A read-only approval is redacted when it is made and kept that way, so it
  stays redacted after its task is evicted.
- A control's reply entry outlives its timed-out wait, and a late reply
  frees the host.
- An unconfirmed stop checks the reservation and removes the host in one
  change, then audits.
- Snapshot questions are bounded (4 questions, 64 KB per decision); when
  running tasks and pending approvals alone exceed the cap, running tasks
  lose their details oldest first.
- Shorter computer tool descriptions, a thread-scoped Stop, and finite
  numbers in the act and rect schemas: the tool list is 1,741 bytes smaller.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jayleaton
jayleaton changed the base branch from feat/computer-use to main November 19, 2026 17:03
…ardening

# Conflicts:
#	apps/server/src/provider/agentContextBudget.json
@jayleaton

Copy link
Copy Markdown
Owner Author

Randy reviewed head 161d78f (round 3). The head is now 8dfa439, which only adds a merge of main. Changes needed.

Verified fixed:

  • approval eviction
  • the late standalone reply
  • the order of the shutdown audit
  • the fail-fast reservation and the helper restart
  • the original oversized snapshot case and the short echoed-field case

Targeted vp test run passed 310 tests (3 skipped), and check:changed passes.

Blocking:

  1. ComputerService.ts:1473: legitimate goals are refused. With prepared text OK, "click the existing OK button" fails before the task starts. Separately, short payloads that contain punctuation slip past the check, because it splits the goal into words before comparing. Fix: use a structured goal that carries the prepared-input references and the control labels separately.
  2. ComputerService.ts:354: short text wipes out unrelated screen context. With prepared text a, the decider got empty screen text for an invoice that showed its number and balance, before anything was typed. Fix: keep the structured screen context and suppress only the value of the edited field and any regions that can echo it.
  3. ComputerService.ts:269: the snapshot budget can squeeze out the action the planner needs. A localized form at the helper's 200-control limit used up the budget before type_text was included. Fix: reserve space for the built-in actions and for each question before adding control descriptions, and measure the serialized JSON.

Nits:

  • ComputerService.ts:272: answering with the displayed form of a key longer than 64 bytes returns {}. Allocate bounded, unique aliases before projecting the questions, and decode answers through the same map.
  • tools.ts:109 (belongs with feat(computer): built-in computer use for agents #279): parameter descriptions reach the advertised schema only for some fields. The goal, hostId and taskId descriptions are missing. Add a test that checks the advertised schema.

Context budget: Randy flagged that the budget passes only because main grew. Since then #279 has merged into main, and after the merge this PR's tool list is 1,741 B smaller than main's, so this item is closed.

jayleaton and others added 3 commits November 20, 2026 18:11
… it is typed, questions keep their moves

Randy's fourth pass on #288:

- A goal is the agent's words, with `{text}` references and quoted labels as
  separate parts. Nothing is matched against a label or against short text,
  so a task typing "OK" can click the "OK" button; only long text the agent
  wrote out anyway is replaced in its own words.
- Short typed text leaves the screen alone until a `type` step is granted
  (tracked by the server); after that a region that holds it is left out.
- A snapshot's questions place each question and its built-in moves before
  any control, by serialized size, so 200 long controls cannot push out
  `type_text`. The bounded view is computed once per decision.
- Keys too long to show are aliased like keys that name the typed text, and
  answers decode through the same map.
- Computer tool parameter descriptions are set on the encoded side, so
  `goal`, `hostId` and `taskId` reach the advertised schema.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ardening

# Conflicts:
#	apps/server/src/computer/ComputerService.ts
#	apps/server/src/mcp/toolkits/computer/tools.ts
#	apps/server/src/provider/agentContextBudget.json
…ardening

# Conflicts:
#	apps/server/src/provider/agentContextBudget.json
@jayleaton

Copy link
Copy Markdown
Owner Author

Randy reviewed head 592033a (round 4). Changes needed.

Verified fixed:

  • the "OK" goal refusal
  • reserving type_text and stuck in the snapshot budget
  • the aliases for keys over 64 B
  • the advertised parameter descriptions
  • the merge resolution

The tool list is 76,170 B against main's 77,528 B. Targeted tests pass (269 passed, 3 skipped), and so does check:changed.

Blocking:

  1. The prepared text still leaks through the goal and case-sensitive redaction (ComputerService.ts:360-398, :1517-1522). Each of these was reproduced live:

    • A quoted long secret (Type "sk-live-secret99") matches GOAL_PARTS and skips hideTyped.
    • A quoted "{text}" is left literal.
    • Short text written without {text} (type 42 into the PIN box) is accepted and shown.
    • hideTyped is case-sensitive while holds() is not, so an upper-cased echo leaks.

    Fix:

    • Make hideTyped case-insensitive (an escaped regex with gi), and always replace {text}, quoted or not.
    • Only exempt quoted spans when the goal already uses {text}.
    • When there is no {text} and the text is short, refuse if the goal quotes it or holds it as a standalone token.
  2. The prepared text leaks into audit.jsonl and computerAuditTail:

    • decision.answer (:1675-1685) logs the unaliased question id and choice.
    • task.end (:1030-1038) logs the raw surface and reason; the helper's controller.ts:283/409 never scrubs surface.
    • approval.request, approval_skipped and grant (:1317, :1866, :1827) log the raw binding and action (title, url, label, path).
    • task.progress and act (:1956, :1939) log the raw event.text and how.

    Fix: log the aliased keys, pass these fields through hideEchoed/hideTyped, and audit the redacted binding, keeping action.text only for textDigest.

  3. There is a race window in hostDecide (:1764-1801). outboundState is built before reserveStep. If hostAuthorize grants type and releases the step in between, the stale, unredacted screen goes to the decider. Fix: run reserveStep first, then build outbound from the reserved task.

  4. Prepared text over 60 characters leaks its first 60 characters (:351-366, controller.ts:162). describeControl clips the value to 60 characters plus "…" before without() runs, so the redaction no longer matches. Fix: run without() before clip, or omit the value for secrets. On the server, also strip a clipped echo whose prefix matches.

Nits:

  • :540-567: readerBinding doesn't redact action.path (from the merged choose_file) or binding.app.
  • :255-291:
    • next can use up the 64 KB cap before type_into gets any control. Fill the controls round-robin.
    • CONTROL_CHOICE should match /^[cta]\d+$/.

jayleaton and others added 2 commits November 21, 2026 06:04
…ardening

# Conflicts:
#	apps/server/src/provider/agentContextBudget.json
Randy's fifth pass on #288:

- Redaction is case-insensitive. Every `{text}` reads as the marker, quoted
  or not, and text of three or more characters is replaced everywhere in a
  goal. Shorter text may appear only in a quoted label of a goal that uses
  `{text}`; elsewhere the goal is refused.
- An echoed value the helper clipped with "…" is stripped on the server, and
  the helper hides clipped and case-varied forms and scrubs the end surface.
- decision.answer logs the aliased keys; task.end, approval.request,
  approval_skipped, grant, task.progress and act are redacted before they
  are recorded, the typed text kept only as a digest.
- hostDecide reserves the step before building the state it sends, from the
  reserved task.
- Reader approvals hide the text in the app and file path; snapshot choices
  are filled round-robin, and tab and app keys count as controls.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

This PR changes the test suite by +23 test cases (browser 0, unit +3, server +20).

Keep tests that would fail for a real reason, and test logic below the UI where it can be. See Testing.

@jayleaton
jayleaton marked this pull request as ready for review November 21, 2026 02:39
@jayleaton
jayleaton merged commit bba9c24 into main Nov 21, 2026
33 checks passed
@jayleaton
jayleaton deleted the fix/computer-use-a2-hardening branch November 21, 2026 02:39
@jayleaton

Copy link
Copy Markdown
Owner Author

Randy reviewed head 82f4844 (round 5). Changes needed.

Verified fixed:

  • the goal leaks for unclipped strings, and case-insensitive redaction
  • the redaction of all seven audit events
  • the hostDecide race, checked in the code and with a live concurrent test
  • a bare clipped echo of a secret over 60 characters
  • both round-4 nits

CI: 23 checks pass. Targeted tests: 340 passed, 3 skipped. The tool list is 75,686 B against main's 77,044 B.

Blocking:

  1. The server clips strings before redacting them (ComputerService.ts:1064-1065, :1103-1105, :1930-1932, :2013, :2019-2030).
    • finishTask and hostEvent apply ComputerAudit.clip (80 B) to reason, surface, how and text before hideTyped.
    • authorizeStep applies capBytes to app, title (400 B) and url before readerBinding.
    • Reproduced: an 88-character secret leaks 73–76 characters into task.progress, act, task.end and taskView. A 19-character secret placed after 68 bytes of prefix leaks hunter2-ride.
    • Fix: redact the full string first, then clip or cap.
  2. The helper clips before without() and never scrubs chooseSurface (controller.ts:161-190, :322, :469, :886, :907, :1003; server echoes() at :370-374).
    • An 80-character tab title with a secret reaches the decider.
    • "Bearer " + secret defeats both without() and echoes().
    • visibleText (1,500) and surface (60) clip first.
    • Fix: scrub before every clip, wrap chooseSurface questions in without(), and have the server also drop clipped echoes where any suffix of the cut matches a prefix of the typed text.
  3. The quoted-label exemption can be bypassed (:404-425).
    • {text} inside the quotes unlocks the exemption.
    • QUOTED allows long sentences, and it allows “…" across quote types.
    • standsAlone misses short text that starts or ends with punctuation or a space (a., 4, !1).
    • Fix:
      • Require {text} outside the quotes.
      • Limit quoted spans to short labels that don't cross quote types, e.g. /“[^“”"]{1,40}”|"[^“”"]{1,40}"/g.
      • Apply the \p{L}\p{N} boundaries only when the end character of the text is itself alphanumeric.
  4. readerBinding leaves ref, role, key, fieldRole, direction and what unredacted (:580-609). A scroll action carrying them is auto-granted and logged in grant. Fix: redact every optional string field except kind and text.

Nits:

  • :1573-1576: a host's refusal error string goes back through MCP unredacted. Pass it through hideTyped.
  • Add a regression test for the race: hostDecide returns busy while a type approval is in flight, then redacts after the grant.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XXL vouch:trusted Author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant