Repository navigation
feat: v2.4.0 — Claude-in-Chrome parity: reliable targeting, shadow DOM, frozen-tab recovery, coordinates, GIF, multi-browser, shortcuts - #20
Merged
Conversation
…ewrite, shadow DOM
Refs were looked up by a data-mcp-ref attribute nothing writes any more, so
every ref action fell through to the smart-selector fallback, whose first
step returned the FIRST querySelector match — clicks "succeeded" on the
wrong element, and hover/select/drag/fill_form/upload/press_key could not
use snapshot refs at all.
- lib/page-dom.js: page runtime (install-once, execDom) with ref registry →
first VISIBLE selector match across open+closed shadow roots and
same-origin iframes → verified fallback (unique, or nth among exact
role/tag/name twins; ambiguous = REF_GONE instead of a guess). Composed
hit-testing for occlusion, flat-tree text, CSS-independent names.
- All tools use it: click, type, press_key, hover, select, drag, fill_form
(select also matches option labels), upload_file (CDP objectId, works in
shadow roots/frames), scroll.
- browser_find: tokenized scoring over names/labels/attributes, role words
("search input", "... button"), shadow DOM, wrapper/echo suppression,
sort before limit, optional role filter, hint when empty.
- browser_click_text: case-insensitive name/composed-text match (CSS
uppercase no longer breaks exact), shadow DOM, clicks the owning control
with a trusted CDP click.
- browser_wait: any visible match (not just the first), plus text and
urlIncludes conditions.
- browser_text: shadow-DOM content, mode:"article", offset paging, 100k cap.
- browser_snapshot: flat tree (closed shadow roots, slots, display:contents),
short refs, no isNew on the first snapshot, path-only same-origin hrefs,
landmarks named by labels only, filter/depth/ref/maxChars (default 20k).
- tests: page-dom resolver/find/click_text/wait/snapshot/text suites on a
small fake DOM.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Benchmark T1: nodejs.org/dist/index.json froze its tab; text, evaluate and navigate each timed out at 125 s because an in-flight executeScript can't be aborted and every later call queued behind it on the tab mutex. - safeExec races every page function against an 8 s budget; a timeout marks the tab wedged (TAB_WEDGED, with recovery advice). While wedged, calls pay one 1.5 s probe instead of queueing; a responding page clears the mark, so does a new navigation or closing the tab. - Overlay show/hide is best-effort (1.5 s cap, skipped on wedged tabs); CDP attach/viewport/focus-emulation probes are bounded (5 s); evaluate checks responsiveness before attaching. - navigate (and the new browser_tabs action "reload") bypass the queue for a wedged tab and replace it: a new tab at the same window/index/active state, the frozen one closed. Measured 0.6 s vs 47-57 s waiting on the page (Page.crash and tabs.discard don't help; discard even changes the tab id). Results carry replacedTabId + the new tabId; browser_batch follows it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…screenshots
Claude-in-Chrome style "computer" actions, all additive:
- browser_click x/y (a real click at a viewport point, reports what it hit),
clickCount 1-3 (triple click), modifiers (ctrl+click...).
- browser_hover x/y; browser_scroll x/y sends a real mouse-wheel event at
the point (inner panels, maps, virtual lists).
- browser_press_key: space-separated sequences ("ArrowDown ArrowDown Enter",
"ctrl+a Backspace") and repeat.
- browser_type without ref/selector types into the focused field (and says
so when nothing but the page body has focus).
- browser_screenshot region {x,y,width,height} zooms into a rectangle (scale
up to 4, default 2), and every CDP screenshot reports how image pixels map
to the x/y these tools take — no coordinate-frame guesswork.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ull URLs - New tool browser_resize_window: width/height and/or state (normal, maximized, minimized, fullscreen) for the window holding a tab; reports the resulting window and viewport size (Claude-in-Chrome resize_window). - browser_tabs create active:false opens a tab in the background (the user keeps their tab in front); focus window:true also raises the window; list fullUrls:true keeps long URLs. - Tool guidance updated for the new capabilities. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… network content.js patched console in the isolated world, so the page's console.log/warn/info/debug never reached browser_console (only uncaught errors did). console-main.js (a MAIN-world content script) patches the page's console and hands entries to content.js as JSON on a private DOM event. - browser_console: pattern (regex), level (one or several), limit (latest N); reports the unfiltered total. - browser_network: failed requests are recorded (webRequest.onErrorOccurred: DNS, blocked, aborted…), urlPattern (substring, Claude-in-Chrome style) next to the regex filter, failed:true for errors and HTTP 4xx/5xx. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ones browser_upload_file gains imageBase64 (+fileName/mimeType) and fromScreenshot (any tab, optional region) — the Claude-in-Chrome upload_image case. The File is built in the page from the bytes: into an <input type="file"> (files + input/change) or, for any other target, as a dragenter/dragover/drop with a DataTransfer, which is what upload drop zones listen for. Target by ref, selector, x/y, or the first file input. Local file paths keep going through CDP DOM.setFileInputFiles as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
New tool browser_gif (Claude-in-Chrome gif_creator): start / frame / stop / status / clear / export. While recording, the router captures a downscaled JPEG frame after every page-changing action (after the reply is sent, inside the tab mutex); export decodes the frames with OffscreenCanvas, marks click points with a red ring, and encodes a looping GIF89a with real pacing (0.4-2.5 s per frame). The MCP server writes the file (default ~/Downloads/browser-recording-<time>.gif) and returns its path, never the image bytes. lib/gif-encoder.js is dependency-free: fixed 256-colour palette (6x6x6 cube + 40 greys — UI screenshots are mostly greys and flat colours) and LZW; tests round-trip the LZW through a reference decoder, and Pillow reads the live output. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The bridge kept a single extension socket and closed it whenever another connected. Now every socket is its own connection (bridge-connections.ts): - Extensions announce a persistent browserId + label in helloAck (stored in chrome.storage.local); legacy ones count as "default". - A reconnect of the same browser replaces its old socket once the new one finished its handshake; other browsers stay connected. - Routing: a session's selected browser, else the most recently connected one — with one browser everything behaves exactly as before. - New tools, answered by the bridge itself: browser_list_browsers and browser_select_browser (browserId or label, "auto" = default). A session whose browser went away gets a clear error instead of the wrong browser; a released session forgets its choice. - A browser disconnecting fails only the calls sent to it; pings, pongs and handshake timers are per connection; control messages go to all. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…les}}
New tool browser_shortcuts — Browser Controller's counterpart of
Claude-in-Chrome's shortcuts: save a named list of steps (browser_batch
actions with {{variable}} placeholders), then run it in ONE call with vars
filled in; list / show / delete. A run is a browser_batch (stops at the
first failing step, default tab, output last by default); a whole-string
placeholder keeps the value's type. Missing vars are reported up front.
Stored locally in ~/.browser-controller/shortcuts.json (atomic write;
BC_SHORTCUTS_FILE overrides). Runs in the MCP process like browser_batch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ilities - browser_gif export travels in 600 KB parts (the daemon's WebSocket frames are capped at 1 MB); the MCP tool reassembles and writes the file. - Version 2.4.0 (package, manifest); README: key capabilities and the tool tables updated for shared ref resolution, shadow DOM, frozen-tab recovery, coordinate actions, GIF, shortcuts, multi-browser and the new params. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…returned Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…0 lines); lint Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Advanced Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
ⓘ Qodo reviews are paused because the subscription is no longer active. Ask your workspace admin to reactivate the subscription to resume reviews. Manage billing |
…pping, iframe visibility Review findings on #20: - [P1] A wedged tab's navigate skipped the queue and with it the lock check, so session B could replace and close a tab locked by session A. The router now checks ownership on that path, replaceFrozenTab refuses another session's tab too, and the owner's lock moves to the replacement tab instead of being dropped. - [P1] Screenshot coordinate mapping ignored the device pixel ratio (DPR 2: 1600x1200 image of an 800x600 viewport still said x = imageX). The mapping now comes from the real image size (PNG/JPEG header), and maxWidth caps the image by accounting for the ratio. - [P2] isVisible() did not look at enclosing iframes, so wait(visible) passed for a button inside an opacity:0 iframe. Visibility now requires every enclosing frame element to be visible. Each fix has a regression test that fails on the previous code. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2 tasks done
compnew2006
added a commit
that referenced
this pull request
Sep 30, 2026
release: v2.4.0 to main (PR #20 landed on feat/cic-parity)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Version 2.4.0. It closes the gaps a 10-test benchmark found against Claude in Chrome (CiC) and adds CiC's remaining features. No existing tool, param or default was removed. Stacked on #18 (
feat/cic-parity).Targeting correctness (root cause of the benchmark's wrong clicks)
data-mcp-refattribute that nothing writes any more. Every ref action therefore fell through to the fallback, which returned the firstquerySelectormatch.lib/page-dom.js, one resolver for every tool:browser_findrewritten: tokenized, role-aware ("search input", "… button"), searches shadow DOM, drops wrapper/echo matches, optional role filter.browser_click_text: case-insensitive accessible-name match (CSS uppercase is fine), searches shadow DOM, trusted click on the element that owns the text.browser_wait: succeeds on any visible match, plus newtextandurlIncludesconditions.browser_text: includes shadow DOM;mode:"article",offsetpaging.browser_snapshot: flat tree;filter/depth/ref/maxChars(default 20k); shorter refs; landmarks named only by explicit labels.Frozen tabs
TAB_WEDGED, and later calls pay one 1.5 s probe instead of queueing behind the frozen page.navigateand the newbrowser_tabs reloadreplace a frozen tab in place and returnreplacedTabId;browser_batchfollows it.Parity with Claude in Chrome (additive)
x/y,clickCount1–3, click modifiers.repeat; typing into the focused field.regionzoom, and every screenshot reports the pixel → x/y mapping.browser_resize_window, background tabs (active:false),focus window:true,list fullUrls.pattern/level/limitfilters.urlPatternandfailedfilters.browser_upload_fileacceptsimageBase64/fromScreenshot, into a file input or a drop zone.browser_gif: records the actions as a GIF with click markers; export is sent in parts to stay under the 1 MB WebSocket limit; the MCP side writes the file.bridge-connections.ts);browser_list_browsers/browser_select_browser. With a single browser, behaviour is unchanged.browser_shortcuts: saves action sequences with{{variables}}and replays them in one call.Measured on the benchmark failure cases (live)
find"search input" / "Disagree button"Test plan
npm test: 377 pass. New suites:page-dom(resolver / find / click_text / wait / snapshot / text on a fake DOM),wedged-tab,gif-encoder(LZW round-trip against a reference decoder; the live output also decodes with Pillow),bridge-multibrowser,window-tabs,tools/shortcuts,tools/gif, and new trusted-input cases.--agentsuite, which also fails on the base branch while a live daemon owns the pipe.🤖 Generated with Claude Code