Browser tools for lent sessions: relay pw-mcp (or CDP) through reverse attach, no screenshots needed

[chaodu-agent]

Problem

A coding CLI in an openab-pty session that has been lent a Mac (reverse attach, #7) can drive a browser only by osascript + screenshot + mouse. Reading a page title means cropping a screenshot at scale: 2 and OCR-ing with the model. Slow, brittle, and the machine has the DOM right there.

The obvious shortcut — the agent talks CDP to Brave on macmini:9222 — is not available by design: the sandbox has no egress (that is the property reverse attach exists to keep; measured in openab-pty#37). Anything browser-shaped must be served from the Mac over the socket the Mac already dialled.

What exists on macmini today

• @playwright/mcp 0.0.82 as a LaunchAgent on 127.0.0.1:8794 (poc/pw-mcp/), headed Chromium in the Aqua session, exposed to the tailnet on :8443. Rich tool set: browser_navigate, browser_snapshot (accessibility tree → text), browser_click, browser_type, browser_evaluate, tabs, …
• Brave running with --remote-debugging-port=9222 (user's own browser, own profile/logins).
• The reverse-attach socket carries plain MCP JSON-RPC and MCPServer already scopes tools per connection (ToolProfile).

Options

	A. Relay pw-mcp through instance-mcp	B. Native browser_* tools over CDP to Brave :9222	C. AppleScript only (today)
What the agent gets	Playwright MCP's full tool set, namespaced browser_*, appearing in tools/list under the sandbox profile	a small set we write: browser_open, browser_eval, browser_text(selector), browser_tabs	osascript → execute active tab javascript (needs Allow JavaScript from Apple Events once)
Browser	Playwright's Chromium, separate profile (no user logins unless set up)	the user's Brave, with their logins	any Chromium via AppleScript
Code	instance-mcp: an MCP client to 127.0.0.1:8794, merge its tools/list into the scoped server, forward tools/call with a prefix; per-profile allowlist (sandbox gets snapshot/navigate/click/type/evaluate; drop browser_install, file chooser, etc.)	CDP WebSocket client in Swift, target discovery via /json, Runtime.evaluate, Page.navigate	none
Risk	Playwright's tool surface is large; needs review of what sandbox may call. Two processes to keep alive	CDP gives full control of the user's logged-in browser — owner profile only, or per-site confirmation	JS-from-Apple-Events is a global Brave setting

Proposal

A first, because it is mostly plumbing and the tool set is already good: instance-mcp gains an "upstream MCP" concept — one or more loopback MCP servers whose tools are merged into the served list under a prefix, subject to the connection's ToolProfile. pw-mcp is the first upstream. The sandbox profile allows the read/navigate/interact subset; owner gets everything. Snapshot-based reading (browser_snapshot) answers "what is the first video's title" as text in one call.

B later if driving the user's own Brave (with logins) is wanted — gated to owner, or to a Connect approval tap per site.

C is documented as the zero-code path in the meantime.

Acceptance

• From a lent session (sandbox), tools/list shows browser_navigate, browser_snapshot, browser_click, browser_type, browser_evaluate and not browser_install/file-chooser tools.
• browser_navigate https://www.youtube.com/@AdmonGoldTV/videos then browser_snapshot returns the first video title as text; no screenshot involved.
• The browser window is visible on the Mac's desktop (Connect's Screens pane shows it).
• pw-mcp down → the browser_* tools are absent from tools/list and instance_status says so; nothing else breaks.
• Sandbox still has no exec*; the upstream's tools are filtered by the same profile mechanism.

Refs: openabdev/openab#1544, #1 (sandbox adapter), openab-pty#37.

[chaodu-agent]

Closed loop verified end to end — 2026-09-26

Option A shipped in #11 (0.6.0). The full loop ran today with a human in the seat: Connect on the laptop, oab-instance-mcp on macmini, the openab-pty pod on p1's k3s (pre-beta-kiro, tools plane on), and kiro-cli inside a lent session driving a real browser window on macmini through the reverse-attached socket.

flowchart TB
    subgraph human["👤 Human — OpenAB Connect (Mac) / Remote (iPhone)"]
        C1["attach: terminal into the session"]
        C2["Screens: live view of macmini's desktop"]
        C3["right-click session → Lend a Mac / Take back"]
    end

    subgraph tailnet["WireGuard tailnet"]
        TS((" "))
    end

    subgraph pod["p1 k3s pod — sandbox, no egress"]
        SC["tailscale sidecar<br/>inbound only"]
        subgraph rt["openab-pty runtime"]
            ADM[":8090 admin plane<br/>/pty/{s} · /tools/attach/{s}"]
            MUX["mux<br/>JSON-RPC id rewrite"]
            LB[":8091 loopback<br/>/mcp/{s}/&lt;key&gt;"]
        end
        subgraph sess["session &quot;mac&quot;"]
            SH["bash → kiro-cli chat<br/>MCP 'mac' = $OPENAB_TOOLS_MCP_URL"]
        end
    end

    subgraph mac["macmini — the hands"]
        IM["oab-instance-mcp 0.6.0<br/>MCP server · ToolProfile filter"]
        LOCAL["local tools<br/>sys_info screenshot mouse key osascript<br/><i>exec* hidden under sandbox</i>"]
        PW["upstream: @playwright/mcp :8794<br/>browser_* (16 of 32 under sandbox)"]
        BR["Chromium window on the desktop"]
    end

    C3 -->|"① POST /attach<br/>{runtime, session, profile, ttl, admin_credential}<br/>tailnet login + bearer"| IM
    IM -->|"② mint: POST /admin/sessions/mac/tools-attach<br/>pod keeps sha256 only"| ADM
    IM -->|"③ dials WS /tools/attach/mac<br/>Authorization: Bearer secret"| TS
    TS --> SC --> ADM
    ADM <--> MUX
    MUX <--> LB
    SH -->|"④ tools/call over plain HTTP loopback<br/>no token, no TLS, no proxy"| LB
    IM --> LOCAL
    IM --> PW --> BR
    C1 -->|"WSS /pty/mac"| TS
    C2 -->|"screenshot tool"| IM

    classDef sec fill:#fff4e6,stroke:#d9822b,color:#000
    classDef net fill:#e8f4fd,stroke:#3a87c8,color:#000
    classDef human fill:#e9f7ef,stroke:#2e8b57,color:#000
    class ADM,MUX,IM sec
    class SC,TS net
    class C1,C2,C3 human

Loading

Sequence of one "lend" and one tool call

sequenceDiagram
    autonumber
    participant H as Human (Connect)
    participant M as macmini<br/>oab-instance-mcp
    participant P as pod runtime<br/>openab-pty
    participant K as kiro-cli<br/>in session "mac"
    participant PW as playwright-mcp<br/>+ Chromium

    H->>M: POST /attach {runtime: ws://pod:8090, session: mac,<br/>profile: sandbox, ttl_secs: 3600, admin_credential}
    M->>P: POST /admin/sessions/mac/tools-attach (admin credential, used once)
    P-->>M: 201 {secret} — pod stores sha256 only · audit ToolsGrantMinted
    M->>P: WS /tools/attach/mac  Authorization: Bearer secret
    P-->>M: 101 · audit ToolsAttach · pod GET /admin/sessions → tools.mac.attached=true
    P->>M: initialize (runtime is the MCP client on this socket)
    M-->>P: serverInfo oab-instance-mcp 0.6.0
    M-->>H: 202 grant {state: attached} · Connect row shows the lent badge

    K->>P: POST $OPENAB_TOOLS_MCP_URL tools/list
    P->>M: tools/list (id rewritten)
    M->>PW: tools/list (Mcp-Session-Id held by the upstream client)
    PW-->>M: 32 browser_* tools
    M-->>P: local (no exec*) + 16 allowlisted browser_* + instance_status
    P-->>K: merged list

    K->>P: tools/call browser_navigate {url: youtube.com/@AsmonTV/videos}
    P->>M: forward
    M->>PW: browser_navigate
    PW->>PW: Chromium loads the page — visible in Connect's Screens pane
    PW-->>M: result (page title, status)
    M-->>P: relayed verbatim
    P-->>K: 200

    K->>P: tools/call browser_evaluate {querySelectorAll('a[href*="/watch"]')…}
    P->>M: forward
    M->>PW: browser_evaluate
    PW-->>K: {first: [{title: "My girlfriend exposed me", href: /watch?v=WUhC9aU8C28}, …]}

    H->>M: DELETE /attach/{id}
    M->>P: close 4010 TOOLS_REVOKED · audit ToolsDetach
    P-->>K: next tools/list → [instance_status] only

Loading

What the agent actually did (verbatim from kiro-cli chat in the pod)

Prompt: "browser_navigate 打開 https://www.youtube.com/@AsmonTV/videos 第一部影片標題是啥"

• browser_navigate → browser_find /video-title/i → two browser_evaluate calls (first with YouTube-specific selectors, then a generic a[href*="/watch"] scan) → answer "My girlfriend exposed me" (12:14, /watch?v=WUhC9aU8C28).
• Second prompt, "搜尋一下錫蘭的頻道最新影片": browser_navigate to a YouTube search, browser_evaluate to list channel links, picked @xilanceylan, navigated to /videos, browser_evaluate again → "24小時內下架" (0:33) with the note that the next one, "新深呼錫 - 我就懟！" (21:11), is likely the latest full video.

No screenshots were taken for either task; the agent chose the DOM path on its own once browser_* was in tools/list. The browser window was visible in Connect's Screens pane throughout.

Trust, as exercised

Party	Held	Never held
Human (Connect)	pod admin credential; macmini bearer token	the attach secret
macmini	one attach secret per grant (1 h, revocable); the only party that dials	the pod admin credential beyond the single mint call
kiro in the pod	one loopback URL with a per-session key	any tailnet address; any bearer; exec on the Mac; browser_run_code_unsafe (answered "unknown tool")

Gotchas met on the way (all recorded in READMEs / #37)

• @playwright/mcp --allowed-hosts compares Host verbatim: 127.0.0.1:8794 → 403, 127.0.0.1 → OK. The upstream client sends the bare host.
• Playwright answers as SSE (data: {…}) even for single responses; the client parses both.
• TCC is bound to the code-signing identity: an ad-hoc re-sign over SSH silently loses Screen Recording/Accessibility (Connect's Screens pane freezes). Sign with the team identity from the console; browser_* needs no TCC and kept working.
• Grants are in-memory: restarting oab-instance-mcp drops them and the human must lend again. Candidate follow-up: persist grants (secret in Keychain) and redial on start.
• @AdmonGoldTV does not exist on YouTube; the intended channel was @AsmonTV. A search-first step, which the agent did on its own for the second task, is the right habit.

Closing as done. Follow-ups worth their own issues: grant persistence across restarts; an exec/browser_run_code_unsafe approval hop through Connect for the sandbox profile; option B (the user's own Brave over CDP, owner-only).

[thepagent]