Repository navigation
feat: a Device tab shows a live iOS or Android simulator, and agents can drive it - #1010
Merged
Merged
Conversation
Device panel design (docs/design/devices.md), ported from T3 Code's Device panel without its cloud, pool and SSH hosts. Contracts: device schemas, the device channel over WS RPC, a WsDeviceRpcGroup, enableDevices and agentDeviceAccess settings, and the deviceTools session flag.
… device channel The Device tab reaches the device hub only through the app's WS RPC connection. hubTransport turns the device channel RPCs into a fetch that answers a streaming Response and a WebSocket-shaped socket, reassembles message parts, credits a chunk once its reader took it and never reopens a channel on its own. The stream client is T3 Code's, ported without 3D, Duo and WebRTC, and running on that transport: AVCC with WebCodecs, MJPEG painted to the canvas without it on iOS, and polled screenshots with live touch on Android without it. Also the screenshot capture and the pure state helpers (install line, platform status, new agent sessions).
Port T3 Code's device host to one local host per server. The pinned expo-device-hub and agent-device install on Threadlines' managed Node.js with install scripts on. The hub runs on loopback with --require-token; its token is read from the startup line, kept in memory and sent as a bearer on every request. hub.json lets the next server stop a leftover hub, checked by pid and command line. DeviceService keeps sessions in claims.json before reporting them open, reloads them on start, and a reactor ends them when a thread is archived, deleted or moved. Use devices off stops the hub and daemon; agent access off stops the daemon. The device channel carries the Device tab's requests and sockets over WS RPC with an allowlist (no serve-sim or serve-emu /api), 256 KiB chunks, a 2 MiB credit window and a slow cap. The threadlines_device MCP endpoint serves device_list, device_open (with a bound agent-device launcher), device_screenshot and device_close.
Sessions started with deviceTools (Use devices and Agents can use devices on, thread not fenced) get the threadlines_device server in Codex, Claude, ACP, Pi, Muse and OpenCode, plus a short instruction block everywhere but ACP. Nothing changes when the settings are off.
A loopback port whose listener is the device hub, serve-sim, serve-emu, the agent-device daemon or one of their children is refused, whoever started it, and left out of the local servers list. Found by command line and process tree; nothing is ever signalled this way.
Managed Node.js for the tools, hub state folder, how sessions end, the fence on device tools, the agent-opened signal, settings reactions, and what the live check found about serve-sim's ports.
The WS client and the environment API get a device group: the device calls, the state stream (followed again after a reconnect) and the device channel, which never resubscribes. A new optional capability, devices, lets clients hide the Device tab and its entry points from servers that lack it. client-runtime exports its device state helpers separately from the stream code, so the main bundle does not carry the stream.
The right panel gets a Device tab for project threads on computers with devices. On first use it explains what each step installs and installs nothing until Use devices is on. Then it lists the computer's simulators and emulators, opens one for the thread (starting it when off) and shows its screen in a flat frame sized to the device, with touch, keys and paste, a toolbar of hardware buttons, screenshot and power off, and a Tools drawer that only shows values the device read back. Only the device on screen streams: a hidden tab or another thread holds no channel. An agent opening a device brings the tab up on it for whoever is watching the thread.
Settings gets a Devices page with Use devices, Agents can use devices (user-only in the app guide) and the versions of the device hub and agent-device with Update. The app guide gets places for the page and its rows and a device-tab topic. The command palette gets Open device.
No device.toggle keybinding command (older clients decode the command list strictly), the devices capability, the toolbar layout, the device-tab topic id, how credits are counted, and the parts of T3's viewer not ported.
# Conflicts: # docs/design/devices.md
…ts exact bytes The web hides the Device tab unless the server reports the devices capability. An http channel counted base64 slices by estimate while the client credits exact decoded bytes, so the window crept shut over hours.
expo-device-hub's device list leaves out simulators with no usage timestamps. serve-sim's grid lists them; add the ones missing. Ported from T3 Code's fix (pingdotgg/t3code#28915).
simctl cannot report the foreground app, so the drawer asks the device's serve-sim helper every two seconds while it is open. The device header drops its truncated "started by Threadlines" note; the picker shows it.
The preview tunnel refuses this server's own device tool ports on every platform. Launcher pruning never follows a symlink. device_open never picks a physical phone by default. A refused channel open closes its hub socket. Switching agent access off refuses tools at once. A session the daemon did not confirm closing stays claimed; an open that finishes after its thread was put away records nothing. The state stream keeps only the newest snapshots for a stalled subscriber. The iOS screenshot fallback keeps its temp file in the state dir. Touch screens get a Keyboard button, and iOS types text as US-layout key presses.
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
…rame size Device tools now read the outside-work limit on every request, as the browser tools do, and refuse while an outside connection's work limits the session. The viewer stops buffering a video frame, still frame or socket message past 16 MiB and reconnects, so a compromised saved computer cannot exhaust the client's memory.
McpInvocationScope already carries outsideLimit; drop the extra type and build the test's caller in full.
…rlay, tabs, float, record, event log
The device listing now carries an optional model: an iOS simulator's device type from one cached simctl call, an emulator's AVD hardware profile, or a phone's ro.product.model. Frames will follow it instead of the name.
…accessibility tree Duo hinge and stance commands ride the admitted iOS input socket as tagged messages and wait for their reply. With panel sinks set, video comes from the fixed feed of the display in front and reopens on the other panel at a handoff; a refused panel feed falls back to the ordinary stream. Fold and accessibility reads go over the device channel.
Taps named from the accessibility tree, Home, rotate and Tools actions become a numbered step log; screenshots fill the draft's free slots (none for a model without images) and the stream canvas is recorded as a video under the 10 MiB attachment limit. Ported from T3 Code's open PR pingdotgg/t3code#38735.
Several devices open at once as chips at the top of the Device tab, with rename and close; only the shown one streams. A toolbar button floats the device over the chat, dragged by its title strip and resized from its edges, kept inside the window. Not offered in the phone layout.
…f ending the session The session's first-frame watchdog kept running after the video moved to a panel feed, so a silent feed stopped everything, input socket included.
…ep the plain frame, inspect labels stay in inspect A drag from the screen in front onto the dimmed display ends where it left. An emulator without a hardware profile keeps the plain frame unless it folds. A label tapped in inspect mode no longer sticks after leaving it, and hover shows the element under the pointer again.
avdmanager is Java. Android Studio ships its own Java but most people never set JAVA_HOME, so the hub could neither list stopped emulators nor start one, and starting failed as "Failed to allocate an emulator port". The server now finds JAVA_HOME, Android Studio's Java or an installed JDK and hands it to the hub; with none, Android stays available for running emulators and the setup screen and device list say what starting needs.
…s in color only while recording
…owns the model cache The host builds device operations each time the hub is ready, including for a hub already running. Going through a factory made once per host keeps the simulator and phone model caches for the host's whole life, and the test now lists through that factory the way LocalDeviceHost does.
serve-sim sends each touch to the display in front, so a release sent after a handoff landed on the other screen. The stream client now lifts a held touch on its own display before any Duo command goes out, and drops the rest of a gesture when the helper hands the input over on its own; the view no longer sends an end after a handoff.
The size rule for new attachments also dropped ones saved before it, and a write that overflowed storage made the next write leave out every saved attachment. Saving now lives in one function the composer and other callers share: a saved attachment keeps its saved data whatever its size, only new ones must fit, and after a failed write what storage still holds stays saved.
The hand-off saves the draft's attachments itself instead of waiting for the thread's composer, which is not mounted once the user moved on, and offers the video for download when it did not save. The step log follows the prompt without trimming it. The hand-off tests move to the browser suite, where local storage is real.
…o display Stop and both caps end capture before waiting on a running Tools action, which is still logged when it answers; the video also stops itself at its byte budget. The video copies whichever canvas drew the last frame, including a Duo display's own, and Duo's raw touches become steps.
The server published ready a moment before the first listing, so a fresh Device tab could show "0 devices" until a refresh. Ready now goes out with the listing, and the picker says "Starting device support…" or "Looking for devices…" instead of a count while there is no listing yet.
Reading files takes a while; if the draft's attachments changed meanwhile, the save writes nothing and answers stale, since the change that made the newer list saves it. Before, a slow save could overwrite the newer list and leave a freshly pasted file unsaved.
The hand-off saves again on the current list when the draft changed under it, then keeps any of its files the draft could not save, screenshots as well as the video, for download from the Device tab. The toast says the recording was not all saved instead of reporting it added.
github-merge-queue
Bot
removed this pull request from the merge queue due to a conflict with the base branch
Nov 20, 2026
badcuban
enabled auto-merge
November 20, 2026 19:01
# Conflicts: # apps/server/src/provider/agentContextBudget.json
This was referenced Nov 20, 2026
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Users building mobile apps had no way to see or touch an iOS Simulator or Android Emulator from Threadlines, and agents had no supported way to drive one while the user watches.
This adds a Device tab in the right panel, ported from T3 Code's Device panel (MIT, credited in NOTICE) and taken past it in places. Design:
docs/design/devices.md.What users get
What agents get
device_list,device_open,device_screenshot,device_closeon a newthreadlines_deviceMCP server, registered for Codex, Claude, ACP, OpenCode, Pi and Muse.device_openreturns a launcher bound to the thread's agent-device session, and the Device tab opens for anyone watching.How it works
expo-device-hub0.22.1 andagent-device0.21.36 at pinned versions into its state dir and runs them on Threadlines' managed Node.--require-token; only the server holds the token.JAVA_HOME, Android Studio's own, or an installed JDK) and says when there is none.DeviceSummary.modelcarries the simulator's device type, so frames never depend on a renamed simulator's name.Left out on purpose
Worth noting
ExecutionEnvironmentCapabilities.deviceshides everything for older servers.Checked
vp fmt,vp lintandvp run typecheckare clean.Not checked live
Follow-ups (already on main, not from this PR)