Repository navigation
fix(preview): keep browser calls within their budget and name html_preview refusals - #936
Merged
Merged
Conversation
…eview refusals - preview_open/navigate: the Accounts profile check waits at most 3 s for the page to load, then skips the check, logs why and returns the opened tab. - The account check before an action spends the action's own budget; the action gets what remains instead of a second full budget. - html_preview's failure result carries its closed code; html_preview and html_render note the specific refusal (source, path access, HtmlRender error) for analytics, and the tool descriptions state the source rules. - Analytics split timeouts into page_not_ready (host answered, external) and host_overrun (host never answered, scient), and attribute a thrown preview_evaluate script to the agent's code. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ol-list size The two source rules add 146 bytes to the advertised tool list (91330 -> 91476). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…owning evaluate failures After the Accounts check, the remaining budget is measured on a monotonic clock. With nothing left the call ends with PreviewAccountCheckTimeoutError and nothing is sent; otherwise the same whole budget goes in the broker envelope and in the input of every operation whose host reads input.timeoutMs, also when the agent left it to the default. An evaluate execution failure is no longer classed agent_script_error: the host gives a script that threw and a failed CDP call the same tag, so it stays unowned until a closed discriminator crosses the host boundary. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
Thread transfer impact✅ Thread transfer remains within every enforced ceiling.
Other scenariosAll 64 scenario measurements match All 64 scenario measurements
Baseline: Scenario and decoded snapshot size10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.
Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed. |
The remaining budget is checked before the Accounts status check as well as before the action, so nothing reaches the browser once the agent's time is spent. Found in re-check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ailures-20261115 # Conflicts: # apps/server/src/provider/agentContextBudget.scient.json # apps/server/src/telemetry/ScientMcpAnalytics.test.ts # apps/server/src/telemetry/ScientMcpAnalytics.ts
yaacovcorcos
enabled auto-merge
November 15, 2026 02:15
…ailures-20261115 # Conflicts: # apps/server/src/provider/agentContextBudget.scient.json # apps/server/src/telemetry/ScientMcpAnalytics.test.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What users get
preview_openno longer waits up to 15 s on a page that never finishes loading. After an https open (and afterpreview_navigate), Scient's Accounts check waits at most 3 s for the page'sloadevent. If the page is still loading, it skips the check, logs why, and returns the tab that opened. Before, a page with one stalled image, long-polling request or slow third-party script added the broker's whole 15 s to every open.timeoutMs, which the browser host reads (navigate, resize, click, hover, select, drag, upload, type, wait_for), also when the agent left it to the 15 s default. If the check used the whole budget, nothing is sent and the call fails withPreviewAccountCheckTimeoutError, so no action runs on the page after the agent was told it timed out. Before, the check had its own default budget (15 s, or 660 s on the preferred host), so a slow or stuck check could make a 5 spreview_wait_fortake 7 s, 15 s or more.code(invalid_request,capability_denied,orchestration_error) beside the message, and the tool descriptions state the source rules up front: pass exactly one ofhtmlorpath; only full-access threads may pass a path; the file must start with<!doctype html>or<html>.orchestrator_mcp_failure. They now report their code, plus afailureClassfor the specific refusal:html_source_invalid,html_path_denied,html_render_file,html_render_viewport_height,preview_browser_installing, and so on. html_render gets the same classes.timeoutis split:page_not_readywhen the browser host answered that the page never became ready (ownerexternal), andhost_overrunwhen the host never answered and the broker gave up (ownerscient).preview_evaluateexecution failures stay unclassified (ownerunknown). The host gives a script that threw on the page and a failed CDP call the samePreviewAutomationExecutionError, so analytics cannot yet tell the agent's code from a product fault; telling them apart needs a closed discriminator carried across the host/broker boundary (T3 mechanics, not in this PR).Root cause
accountWaitForLoadwith no budget of its own. It used the broker's 15 s default with no grace, so any page whoseloadwas late held the open for up to 15 s, after the tab had already opened. Its failures were already swallowed, so the cost was the time, not an error.statuswith notimeoutMs, so it used the broker default forstatus(15 s, 660 s on the preferred host) on top of the action's budget. Shortening only the broker's timeout is not enough: the hosts run most actions on the input's owntimeoutMs(or their 15 s default), so the input has to carry the same remainder.registerImageTool's failure path kept only_tag: OrchestratorMcpFailure, skipped the tool's failure note, and the html handler folded every HtmlRender tag into two codes. Analytics read the wrapper tag.noteToolFailuredid not tell a host-answered timeout (remoteTagset) from a broker give-up.The fix
Scient-owned code, plus small
SCIENT-FORKseams in T3 files:scient/accounts/agent/previewProfileFallback.ts: a 3 s load wait (PROFILE_CHECK_LOAD_WAIT_MS); every skip is logged with the operation and the error tag. Profile-grant enforcement inpreview_open(authorizePreviewProfile,recordProfileUse,filterPreviewProfiles) is unchanged and still fails the call.mcp/toolkits/preview/handlers.ts(existing Accounts seam): one budget (timeoutMs, else the broker default) starts on Effect's monotonic clock before the check. The pre-actionstatusgets what remains of it; after authorization, a remainder below 1 ms fails withPreviewAccountCheckTimeoutError(Scient-owned, declared in the preview tools' failure union) without dispatch, and otherwise the action gets the remainder in both the envelope and, for operations whose input declarestimeoutMs, the input. The broker exports that default asPREVIEW_AUTOMATION_DEFAULT_TIMEOUT_MS(seam).telemetry/ScientMcpAnalytics.ts: browser timeout classes and their owners;noteToolFailureClass; the failure note's code now takes precedence over a wrapper's tag.mcp/toolkits/html/handlers.tsandtools.ts(seams): note each refusal's class before it is folded; describe the source rules.mcp/McpHttpServer.tsimageToolFailure(seam): add the declared failure's closedcodeto the structured error and write the failure note.Not in this PR, by decision: T3's browser mechanics (deadlines across queue, select, evaluate, wait, snapshot pause/cleanup). We adopt upstream pingdotgg/t3code#32866 at alignment and remove our SessionControl seam then; the remaining gaps go to an upstream issue.
Tests
New, each failing on main for the reason given (negative controls run):
previewProfileFallback.test.ts"a page whose load never ends opens at once, left unchecked": on main the fallback returns after 15 000 ms of the test clock instead of ≤ 3 000 ms.handlers.scient.test.ts"the account check before an action spends the action's budget, not its own": the broker mock now runs on the input's budget like the real hosts. A 5 spreview_wait_forbehind a 2 s check is sent with 3 000 ms in both the envelope and the input and ends at 5 000 ms (main: 7 000 ms); with notimeoutMsit is sent with 13 000 ms in both; a check that takes the whole 5 s sends nothing and fails withPreviewAccountCheckTimeoutError; a stuck check ends at 5 000 ms (main: 15 000 ms). The previous head fails it (input 5 000 ≠ 3 000, dispatch after expiry).McpToolAnalytics.scient.test.ts"an HTML refusal reports its own code and reason…": through the productionlayerHtmlToolkitregistration, html_preview reportsinvalid_request/html_source_invalid,capability_denied/html_path_denied,invalid_request/html_render_fileto the agent and analytics, and html_renderhtml_render_file(main:orchestrator_mcp_failure, no class).ScientMcpAnalytics.test.ts"tells a page that never got ready from a host that never answered, and leaves evaluate failures unowned": a script exception (exceptionDetails) and a CDP rejection, both run through the realServerBrowserPage.evaluateandtoOperationError, reportexecutionwith ownerunknownand no class. The previous head fails it (agent_script_error, ownercode).Ran (macOS, Node 26.13.1, pnpm 12.14.0, live-app environment unset):
pnpm run typecheck(root): pass.vp fmt --check,vp lint --report-unused-disable-directives,knip:check(commit hook): pass.node scripts/scient-seam-check.mjs --base origin/main --head HEAD: all manifests pass.node scripts/verify-upstream-provenance.mjs: pass.src/mcp,src/telemetry,src/preview,src/scient/accountsandsrc/provider(3 395 passed, 2 failed: the PortScanner process-group and ClaudeSkills cases below, unrelated), root typecheck, fmt, lint, seam check and provenance: pass. A full-suite rerun was stopped at 30 minutes on a loaded machine; the failures it reached were only the unrelated ones below.apps/serversuite (first two commits): 16 992 passed, 12 failed in 7 files. One was mine (AgentContextBudget: the tool-list size grew) and is fixed in the second commit. The other 11 also fail on this machine with main's versions of the changed files, and none touches this change: ProviderSwitch prior-images handoffs (6), PortScanner process groups, ClaudeSkills (case-insensitive folder clash), ProjectSetupScriptRunner (expects bash's exit code 2; the shell here is zsh), the OMP native event budget, and the real LaTeX engine.preview_openon a local HTTPS page whose image never answers. The Accounts check gave up after 3.0 s and returned the tab (it used to wait up to 15 s). The whole open still took 24.5 s, because T3's own open waits forloadfor up to 15 s after tab setup; that is T3 browser mechanics, outside this PR, and goes into the upstream issue draft. Three html_preview refusals reached the local analytics outbox ashtml_source_invalid,html_render_fileandhtml_render_viewport_height(2–3 ms each).Agent context
The advertised tool list grows by 146 bytes (
mcpToolListJsonBytes91 330 → 91 476,agentContextBudget.scient.json): thehtml/pathparameter descriptions now say "Pass this or path, not both" and "Full-access threads only; others pass html". These are the two rules behind most fast html_preview refusals. The growth needs the owner'sagent-context-increasesign-off before merge; the label is theirs to add.macOS / Windows / Linux
nlink === 1check on synced folders) was not checked.html_path_deniedrefusal (the live thread was full-access), the pre-action budget, and the timeout classes; unit tests cover them. Only Claude was used live: Claude Code gives the model a tool result's text, not its structuredcode, so agents on Claude see the specific message rather than the code.Model and harness: Claude Opus 5.5 in Claude Code (Scient), implementation sub-agent.
🤖 Generated with Claude Code