Skip to content

perf(ssh): one SSH connection per host, fewer round trips, no 15 s abort - #24

Merged
greaber merged 5 commits into
grant/mainfrom
grant/connect-faster
Nov 20, 2026
Merged

greaber merged 5 commits into
grant/mainfrom
grant/connect-faster

Conversation

@greaber

@greaber greaber commented Nov 20, 2026 •

Copy link
Copy Markdown
Owner

Connecting an SSH environment over a slow link (≈300 ms RTT with 1–2 s spikes) took 22–65 s after an app restart. Two causes, fixed in separate commits.

1. One SSH connection per remote host (perf(ssh), 1071871 and 3d415d7)

Every step opened its own ssh connection, and each costs several handshake round trips (3.5–15 s on that link): the remote launch/reuse, the tunnel's port forward, pairing, and the stop on disconnect.

Each tunnel is now one supervised OpenSSH control master, ssh -M -N with ControlMaster=yes, ControlPersist=no, ServerAliveInterval=15/CountMax=3. It is a foreground child in the tunnel's scope and replaces the old ssh -N -L process, so its exit means the connection is dead. Everything else for that target runs over it with ControlMaster=no, its ControlPath and ProxyCommand=false: the launch/reuse, the forward (ssh -O forward -L local:127.0.0.1:remote, so the forward lives in the master), pairing, the readiness log tail, and the stop on disconnect, which now runs before the master closes. ssh tries the master before it connects, so ProxyCommand=false makes a command fail when the master is dead instead of opening a second connection.

  • User ssh config (upstream [Bug]: SSH environment tunnel fails with ControlMaster auto / ControlPersist pingdotgg/t3code#3533): with ControlMaster auto/ControlPersist in the user's config, the old ssh -N -L handed its forward to an existing master and exited. Our explicit -o options win. Against a real sshd, with such a config and the user's own persistent master already running, the master stayed in the foreground, the forward listened in our master's process and closed with it, and the whole sequence opened 1 connection.
  • Socket: <stateDir>/ssh-control/<8 hex of target>-<8 random hex>, for example ~/.t3/userdata/ssh-control/… at about 57 bytes. ssh first binds path + ".XXXXXXXXXXXXXXXX", so some cases keep one connection per command, as before: a path over 86 bytes, one containing whitespace, % or ", Windows (whose OpenSSH has no ControlMaster), and a master whose stderr reports a control socket failure.
  • Make-before-break (fix(ssh): race a replacement tunnel instead of discarding a slow one #21): each tunnel entry gets a fresh socket. A replacement for a slow tunnel is a new master, raced against the old one, and the loser's master is stopped.
  • Cleanup: closing a tunnel (on disconnect, replacement, or quitting the app) SIGTERMs its master, which removes its forward and socket, and a finalizer removes the socket if the master had to be force-killed. Before it creates its first socket, the manager sends ssh -O exit to any socket that a crashed app left in the directory and removes it.
  • Auth prompts work as before: only the master authenticates, under the existing runWithSshAuth retry.

Connections per environment (processes that open their own SSH connection):

Case Before After
First connect (launch + forward + pairing) 3 1
Reconnect with saved credentials (launch + forward) 2 1
Connect, pair, then switch off (adds the stop) 4 1

The unit test connect, pair, and disconnect open $connections ssh connection(s) asserts 4 against 1, checks the explicit options on every spawned ssh, and covers the stale-socket sweep and the replacement's new master. The manual sshd run counted Accepted publickey lines: 3 for the old sequence, and 1 for master + launch + -O forward + pairing + stop.

Where the 16–24 s "launch/reuse" goes. J5 runs an external, service-managed server (0.1.9-grant.5, while the app is grant.7). Its load average was 16–24 on 160 cores. On that path the remote script starts the t3 binary three times, and I timed each start on J5 itself:

Step Time
sh -l 0.01 s
t3 --version (only proves the runtime is installed) 0.7–1.0 s
t3 __ssh-helper runtime-port 1.0 s
t3 __ssh-helper wait-ready (server answers immediately) 1.0 s

So about 3 s is remote work, and the rest of the 16–24 s is SSH connection setup (several RTTs at 300 ms with 1–2 s spikes) plus sending the ~30 KB launch script. The runner now has a prepare-only mode, so the launch no longer starts t3 --version. That saves about 1 s; the test an archive reconnect reuses… checks that no --version start happens. The other two starts are left as they are. With the master, the launch pays for the handshake once, and nothing after it pays again.

2. Don't abort a progressing SSH setup (fix(client-runtime), 0250200)

The 15 s budget is CONNECTION_ESTABLISHMENT_TIMEOUT in packages/client-runtime/src/connection/supervisor.ts: each attempt gets 15 s per route until the session synchronizes. For an SSH route it covered the whole desktop preparation (launch, tunnel, pairing, descriptor and session fetches), which on that link takes 30–45 s, so attempts were interrupted while still progressing, and a later attempt only succeeded by adopting the earlier attempt's still-running IPC work.

The driver now reports { stage: "preparing", route } per route, and the supervisor stops the setup clock while an SSH route is being prepared. Opening the WebSocket and synchronizing keep their usual budgets (15 s + 75 s).

Why stop the clock instead of raising it to ~45 s for SSH: desktop main already bounds each step of that preparation (ssh ConnectTimeout=10 and keepalives, launch/tunnel/readiness timeouts, DEFAULT_REMOTE_REQUEST_TIMEOUT_MS on each HTTP call, the password prompt's timeout), and it detects a dead tunnel itself. A second, fixed deadline on top can only abort work that is still progressing: a 45 s cap would still cut off a first install (an archive download) or a user typing a password. Detection of a dead tunnel is unchanged because it never depended on this budget.

3. Fewer sequential round trips (8a0b7b4, aee6f5a)

These are the sequential round trips on the critical path from "start connecting" to a sendable thread, for a reconnect with saved credentials after an app restart. I counted them from the code, not from a live trace; the existing ssh.* and relay.connection.attempt spans timestamp each step in a real trace. Each new TCP connection to the local forward also opens an SSH channel, which costs one RTT.

Step Before (RTT) After (RTT)
SSH connection: TCP, key exchange, auth ~4–5 ~4–5 (the master)
Remote launch/reuse: session, exec, output 2, plus three t3 starts (~3 s on J5) 2, plus two t3 starts (~2 s)
Second SSH connection for the tunnel ~4–5 0 (-O forward into the master)
Tunnel readiness probe (channel open + GET) 2 0: the launch already saw the server answer, and a forward in a live master listens at once
Descriptor read by the desktop (identity check before credentials) 2 2
Session state, checking the saved token's scopes 1 1
Descriptor read again by the renderer 2 0: the SSH gateway returns the descriptor the desktop just read
WebSocket ticket 2 2
WebSocket open (channel + upgrade) 2 2
Initial server config 1 1
Total before the shell and thread load ~22–24 RTT + ~3 s remote ~14–15 RTT + ~2 s remote

At about 300 ms RTT, that is roughly 2.5 s saved, besides the 3.5–15 s handshake of the second connection. A first connect also pairs; that is one more command over the master instead of a third connection.

The 15 s setup budget no longer cuts any of these off (part 2).

Left as they are, each for a reason:

  • Descriptor, then session state. These stay in order on purpose: the host proves which machine it is before the desktop sends the saved token.
  • WebSocket ticket. The protocol needs it, because a browser WebSocket cannot carry the bearer header.
  • Pairing on a first connect. Folding it into the launch command would save 2 RTT, but only on first connects, and it changes the remote script's output contract.
  • Initial RPCs after the socket opens (config, then the shell snapshot and thread subscription). Batching them would be a protocol change, so I left them for separate work.

Checks

Command SHA Result
vp test run src/tunnel.test.ts -t "control|ssh connection|archive reconnect" (packages/ssh; connection test fails on base with 4 against 1, --version assertion fails without the runner change) 3d415d7 pass
vp test run src/tunnel.test.ts (packages/ssh) 3d415d7 24/25; the known concurrent launches on one remote home times out when the file runs alone
vp test run src/connection/supervisor.test.ts -t "SSH route" (fails on base: attempt 4 instead of 1) 0250200 pass
vp test run src/connection/ (packages/client-runtime) 0250200 181 passed
vp test run src/ipc/methods/sshEnvironment.test.ts (apps/desktop) 1071871 pass
vp run typecheck (packages/ssh, packages/client-runtime, apps/desktop) 0250200 pass
vp run check:changed greaber/grant/main 0250200 pass
vp run check:changed greaber/grant/main 3d415d7 pass
vp test run src/tunnel.test.ts (packages/ssh; the connection test also asserts 0 HTTP probes over a master and 1 for a plain tunnel) 8a0b7b4 24/25, with the same known failure
vp test run src/authorization/layer.test.ts src/connection/resolver.test.ts (packages/client-runtime; new authorizes over SSH without reading the descriptor…) aee6f5a 34 passed
vp run typecheck (packages/client-runtime, apps/web) aee6f5a pass
vp run check:changed greaber/grant/main aee6f5a pass

Not covered: when the master cannot be created, setup falls back to one connection per command, which the original requirement asked for.

🤖 Generated with Claude Code

greaber and others added 2 commits November 20, 2026 10:29
On a slow link every new ssh connection costs seconds of handshake round
trips, and connecting an SSH environment opened one for the launch, one for
the port forward, and one for pairing.

Each tunnel now owns an OpenSSH control master with an app-owned socket under
the desktop state directory. The launch, the forward (`-O forward`), pairing,
and readiness diagnostics run over it, so a first connect opens one
connection instead of three and a reconnect one instead of two. The master
uses explicit options, so ssh config ControlMaster settings do not apply;
each tunnel gets a fresh socket name, so a replacement for a slow tunnel never
joins the master it may replace; closing the tunnel (including quitting the
app) stops the master and removes its socket. Windows, a socket path over the
macOS limit, and a master that cannot create its socket keep one connection
per command, as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…r 15 seconds

The supervisor gave each connection attempt 15 seconds per route before the
session synchronizes, and that budget included preparing an SSH route: the
desktop launching or reusing the remote server, starting the tunnel, pairing,
and authenticating over it. Over a slow link this takes longer, so attempts
were cut off while still making progress and only a later attempt, adopting
the earlier work, connected.

The driver now reports which route it is preparing, and the supervisor stops
the setup clock while an SSH route is being prepared. Desktop main already
bounds every SSH step and HTTP request in that preparation and detects a dead
link, so the extra deadline only aborted progress. Opening and synchronizing
the session keep their usual budgets.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. size:XL labels Nov 20, 2026
Disconnecting stopped the remote server over a new connection after closing
the tunnel, and a command whose master had died silently opened its own.

Disconnect now stops the server over the live master before closing it.
Commands over a master set ProxyCommand=false: ssh tries the master first,
so they still work, but a dead master fails them instead of opening a second
connection. Sockets a crashed app left behind get `ssh -O exit` before the
first new one is created. The remote launch also prepares the release
runtime without starting t3 (`--version` took about a second on a busy
host), which leaves two t3 starts in a launch that reuses a running server.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@greaber greaber changed the title perf(ssh): connect SSH environments faster over slow links perf(ssh): one SSH connection per remote host; don't abort slow SSH setup Nov 20, 2026
greaber and others added 2 commits November 20, 2026 11:21
…aster

The remote launch reports a port only once the server answered on the
host, and a forward added to a live control master listens at once. Probing
it through the forward again cost a channel open and a request, two round
trips, before the renderer made its own first request. Plain tunnels, whose
ssh may still be connecting, keep the probe.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…onnect

The desktop reads the descriptor through the SSH forward to check the
host's identity before it sends the saved token, and then the renderer read
it again before asking for a WebSocket ticket: a second channel open and
request on a slow link. The SSH gateway now returns the descriptor the
desktop read, and authorizing over SSH uses it when it names the expected
environment. Other routes, including plain HTTP addresses, still read it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@greaber greaber changed the title perf(ssh): one SSH connection per remote host; don't abort slow SSH setup perf(ssh): one SSH connection per host, fewer round trips, no 15 s abort Nov 20, 2026
@greaber
greaber merged commit fd5a87e into grant/main Nov 20, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XL vouch:trusted PR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant