Skip to content

Bound the client's upgrade with an optional timeout, and fail it when the connection ends first - #163

Open
tarag wants to merge 1 commit into
vapor:mainfrom
tarag:upgrade-timeout
Open

tarag wants to merge 1 commit into
vapor:mainfrom
tarag:upgrade-timeout

Conversation

@tarag

@tarag tarag commented Sep 24, 2026

Copy link
Copy Markdown

Background

We keep a WebSocket from a fleet of edge devices to a cloud service with WebSocketClient, and reconnect whenever it drops. A connect that never completes parks that loop for good: there is no error to back off from and no retry, until the process restarts. Today connect can do that in four ways:

  1. The server never answers the upgrade. Only the TCP connect is bounded (NIO's 10 s). Nothing bounds the TLS handshake or the HTTP upgrade that follow, so a proxy or load balancer holding the request is enough.
  2. The server closes cleanly before answering. An end of stream on ws, or a TLS close_notify on wss, raises no error. HTTPUpgradeRequestHandler has no channelInactive, so nothing fails the upgrade promise.
  3. The connection is lost between the 101 and onUpgrade. The future succeeds in the upgrade's completion handler, which NIO calls before the upgrader adds the WebSocket handlers. A caller that waits for onUpgrade to get its WebSocket then waits forever.
  4. A proxy refuses the CONNECT of a wss connection. The refusal closes the channel, but the connect has already succeeded and nothing fails the upgrade promise.

Changes

  • New WebSocketClient.Configuration.upgradeTimeout: TimeAmount?, nil by default (today's behaviour).
    • It bounds host resolution, the TCP connect, the TLS handshake, the proxy's CONNECT and the HTTP upgrade together.
    • When it runs out, connect fails with ChannelError.connectTimeout and the connection is closed, even one made after the timeout.
    • Once the server has accepted the upgrade, the timeout no longer applies.
  • A lost connection fails the upgrade. If the connection that was made closes before the upgrade completes, the upgrade fails with ChannelError.eof. This applies to that connection only: NIO closes the attempts a Happy Eyeballs connect raced against it, and those must not fail anything.
  • A refused CONNECT fails the connect with the proxy's error, for example NIOHTTP1ProxyConnectHandler.Error.proxyAuthenticationRequired.
  • The future completes once onUpgrade has run, not when the 101 arrives.
  • One event loop. The connection, the returned future and the timeout share one event loop. That orders them with one another, and nothing is completed across loops while an EventLoopGroup shuts down.
  • TLS setup errors are reported. A TLS handler that can't be created fails the connect with its own error, instead of closing the channel.
  • Docs. The docs of connect and onUpgrade now say when the future completes and how it fails.

No public API changes apart from the added configuration property (semver-minor).

Tests

Eight new tests in WebSocketKitTests run against a raw NIO server that misbehaves on purpose. Five of them fail on main today:

Test On main
testUpgradeTimeoutClosesAStalledHandshake fails: no timeout
testServerClosingBeforeTheUpgradeFailsTheConnect fails: connect never completes (the test gives up after 10 s)
testServerClosingTLSCleanlyBeforeTheUpgradeFailsTheConnect fails: connect never completes
testProxyRefusingTheTunnelFailsTheConnect fails: connect never completes
testConnectCompletesOnlyOnceOnUpgradeHasRun fails
testUpgradeTimeoutLeavesAnUpgradedConnectionAlone passes (guard)
testServerRefusingTheUpgradeFailsWithItsStatus passes (guard)
testConnectCompletesOnTheWebSocketsEventLoop passes (guard)

For the main column, the test file was compiled against main with an upgradeTimeout property that nothing reads.

  • The full suite passes on macOS (Swift 6.3.3; 33 tests, 1 skipped), with and without SWIFTNIO_STRICT=1. On Linux (swift:6.3.2-noble) every test passes when run on its own.
  • Under --sanitize=thread, TSan reports one Swift access race, in NIO's NIOWebSocketClientUpgrader.randomRequestKey(), during the existing testAlternateWebsocketConnectMethods. It is a false positive: the generator is a zero-sized local, and TSan sees every thread's copy at one static address. main reproduces it with nothing but a few threads calling randomRequestKey().
  • We have also run the patch under our own Vapor 4.122 service. Its handshake tests pass on macOS, and its integration suite of real edge ↔ cloud connections passes on Linux: 9 scenario phases, including sites failing and reconnecting.

🤖 Generated with Claude Code

`WebSocketClient.connect` could leave its caller waiting forever, with no
error, in four ways:

- A server that accepts the connection and never answers the upgrade request
  left the connect pending: nothing bounds the TLS handshake or the HTTP
  upgrade, only the TCP connect.
- A server that closes cleanly before answering (an end of stream on `ws`, a
  close_notify on `wss`) raised no error, so nothing failed the upgrade.
- The connect succeeded as soon as the 101 arrived, before `onUpgrade` ran; a
  connection lost in between left the caller waiting for an `onUpgrade` that
  never came.
- A proxy refusing the CONNECT of a `wss` connection closed the connection
  without failing the connect.

`Configuration.upgradeTimeout` (default `nil`, today's behaviour) bounds the
TCP connect, TLS handshake, proxy CONNECT and upgrade together: when it runs
out the connect fails with `ChannelError.connectTimeout` and the connection
is closed, even one made after the timeout. Once the server has accepted the
upgrade the timeout no longer applies. The connection that was made fails the
upgrade with `ChannelError.eof` if it ends before the upgrade completes (only
that one: the attempts a Happy Eyeballs connect races against it close too),
a refused CONNECT fails it with the proxy's error, and the upgrade completes
once `onUpgrade` has run. The connection, the future `connect` returns and its
timeout share one event loop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@tarag
tarag requested review from 0xTim and gwynne as code owners September 24, 2026 10:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant