fix: fail pending requests on disconnect, report a refused reconnect, and give the connection one state

[NickPadilla]

A client attached to a server that went away hung — with or without a request in flight — and never recovered. Seen in production; reproduced against a real cluster from structures.

Three defects

Pending requests were never failed on the transport's own reconnect. Only cleanup() failed them, and it ran on an explicit disconnect(), not on a socket close. The reply address a request waited on belonged to a server that could no longer answer, and nothing told the caller, so request() waited forever.

A refused reconnect deactivated the transport blind. A STOMP ERROR frame — a refused connect, or a reconnect presenting a session the server no longer had — was handled by calling rxStomp.deactivate() directly, skipping the manager's deactivation handler. Pending requests weren't failed, the connection went down without a word, and every later call threw a "call connect first" the caller had no way to anticipate. fatalErrors was meant to surface this, but its cleanup lived in a map operator nobody subscribed to.

A sticky connect deleted the caller's credentials. The library replaced the credentials in the caller's own connectHeaders object with the session id, so after a refused reconnect the caller could not connect() again with the ConnectionInfo it already held — which is precisely what it is expected to do, since the library holds no credentials.

Why the shape changed, not just the bugs

The first cut fixed those three by adding a second callback and more readable state to the manager. Review of that found five more defects, all the same shape: the manager and the EventBus shared the connection's lifecycle through two callbacks and five public fields that the EventBus read back mid-callback to reconstruct what had happened. fatalErrors emitted while the client was still set, so a handler that called connect() was refused; deactivate() re-entered during its own teardown and reported the same refusal twice; a deliberate disconnect() was indistinguishable from a loss; a failed stream's teardown queued a CANCEL that rx-stomp replayed after reconnect; three delete loops fought over one shared header map.

So the final commit gives the connection one state and one event stream instead.

The contract now

StompConnectionManager is inactive → active → closing → inactive, with a single private close(error?, force?) as the only way out of active. It marks itself closing before the socket watcher can mistake the close for a loss, returns to inactive before emitting, and rejects a still-pending activate() with the same error after that. Overlapping calls share the in-flight promise. It reports through events:

• lost — the socket of an established connection closed. Reconnection carries on underneath; anything in flight is failed now with ConnectionLostError, distinct from an application error so a caller can retry an idempotent call.
• closed { error? } — emitted exactly once per activation, with the manager already inactive, so a handler may call connect() right there. error is present when the server refused us (ConnectionRefusedError, carrying the server's own message and frame) or maxConnectionAttempts was reached; absent when disconnect() was called.

The EventBus is a policy layer on that stream: fatalErrors emits the closed error only after a successful connect() — before that, the rejected connect() promise carries it, as before. It never touches rxStomp; it uses manager.publish()/watch().

• CONNECT headers are built fresh per attempt — the session alone on a sticky reconnect, the caller's headers otherwise (copied if static, called again if a function). Nothing is mutated, so connectHeaders can say exactly what happens on reconnect in each mode.
• A server ERROR frame closes with force — nothing left to say, and the report no longer waits on the peer's close (the sticky restart test dropped from ~3 min to ~40 s).
• ConnectionRefusedError replaces guessing AuthenticationError from the message wording. connect() rejects with typed ContinuumErrors, not strings.
• requestStream marks a request finished when the connection fails it, so its teardown never sends a CANCEL into a dead client.
• No default request timeout — long-running requests are legitimate, and with pending requests failed on close the hang it would have guarded is gone. No re-authentication on the caller's behalf — a stored token may be stale; recovery is the caller's decision.

Tests

StickySession_GatewayRestart pins the contract against the real gateway: the in-flight request fails on close with ConnectionLostError; the refused reconnect is reported on fatalErrors as a ConnectionRefusedError exactly once, with the connection already down; and the recovery connect() is made from inside the fatalErrors handler, which is what pins the ordering. Depends on the delayed testMethodWithDelay in the companion continuum-framework PR (merged) to hold a request open deterministically.

ConnectionContract pins the second review's nine findings against a scripted in-process STOMP gateway (test/ScriptedGateway.ts), because a real gateway cannot be made to produce them on demand: a peer that never answers a DISCONNECT, a CONNECTED frame with no or malformed connected-info, a browser socket whose terminate() is synchronous, an instance with no memory of a session. Every case was committed failing (be4a423) before its fix (2de1cc9):

• disconnect(true) breaks a graceful close the server is not answering — an overlapping close() now contributes its force/reason instead of being folded into the first
• a stream failed by socket loss still cancels its server-side invocation once reconnected — the cancel is owed while the session lives, and goes out on the replacement connection
• a CONNECTED frame the client cannot use ends in a DISCONNECT, not a discarded socket, and the close is deferred out of rx-stomp's callback (browser terminate() is synchronous)
• credentials are cleared, in place, once CONNECTED arrives
• maxConnectionAttempts bounds each reconnect, not the life of the connection
• a rejecting connectHeaders() rejects connect() / reports on fatalErrors rather than hanging with the manager active
• malformed connected-info is treated as absent, not thrown from a subscriber
• errors carry their names
• a registered service is served again after fatalErrors → connect() — observe() now follows the connection across activations

A third pass found ten more, again pinned failing first (f7d9b18) then fixed (next commit), after teaching the scripted gateway three things the real one does: answer a reply-to-less SEND to nothing with an ERROR, double backslashes in CONNECTED values (vertx-stomp-lite HeaderCodec), and answer a DISCONNECT with an ERROR:

• the cancel for a stream failed by socket loss carries a reply-to, so the recovered connection is not ended by the gateway's ERROR
• a reconnect attempt abandoned by disconnect() cannot touch the connection that replaced it
• an ERROR during a requested disconnect() is not reported as fatal
• connected-info survives the gateway's CONNECTED escaping; JSON that is not an object is refused, not thrown
• an observe() made while connect() is pending is subscribed once; one made before connect() attaches when there is a connection; one result subscribed twice is one subscription on the wire
• connectTimeoutMs (new, default 10 s) bounds a handshake that never completes; a polite close is bounded at 5 s before the socket is discarded; an unusable host rejects connect() instead of hanging inside stompjs

A fourth pass found eight more, pinned failing first (eeba723) then fixed, after the scripted gateway learned to stall at the TCP level (not even a close frame answered) and to invoke a hosted service:

• abandoning a stalled attempt discards the socket (discardWebsocketOnCommFailure) instead of paying the socket library's ~30 s close timeout on top of connectTimeoutMs; the watcher is cleared on close; zero is not "off"; each attempt reports its own first error
• a hosted service's reply landing during a requested disconnect() is logged, not thrown from a handler nothing awaits
• requestStream re-checks the connection when subscribed, so a stream created before a close cannot set up the reply address of a dead connection for the next one to inherit
• disconnect() during a server-started close is still a requested close — not fatal, no reconnect mid-shutdown
• an unknown control value fails the request rather than the process
• a URL the WebSocket constructor refuses ends the attempt with the reason (socket factory) instead of hanging inside stompjs; a JSON-array connected-info is refused

A fifth pass, weighted at the double's fidelity, found four more (pinned failing first in e710827, fixed next) and one merge-blocker: the committed package.json still named @mindignited/continuum-client; it is now @kinotic/continuum-client, what npm publishes and every consumer pins.

• a hosted service whose return value cannot be serialised, or whose promise rejects with no value, answers with an error instead of ending the process
• a backslash in a credential is doubled on the way out — the gateway decodes CONNECT values though STOMP says not to, and dropped the frame silently
• connectTimeoutMs covers the caller's connectHeaders() too
• exhausting maxConnectionAttempts after a working connection is pinned as a fatalErrors emission
• heartbeatStrategy: Worker, so a throttled hidden tab keeps its heartbeats; ConnectionLostError.cause carries the close error so a caller can tell a request the server refused from a socket loss before retrying; docs for fatalErrors, disconnect() and connectTimeoutMs now state the full behaviour

Full suite against the published develop snapshot: 10/10 files, 71/71 tests.

Left deliberately, noted for continuum-framework: the gateway's HeaderCodec escapes CONNECT/CONNECTED values though STOMP 1.2 says those frames are unescaped (the client compensates in both directions; the two must change together); a JS-hosted service's reply to a caller that has already hung up is answered with an ERROR that ends the host's connection (EndpointConnectionHandler FIXME) — the client now at least reports that on fatalErrors instead of dying silently; a SEND whose destination is not a valid java.net.URI pauses the gateway connection without resuming it (DefaultStompServerHandler.send); frame-limit violations arrive as bare closes with no ERROR. Heartbeat timing (120000/30000) is unchanged from 2.14, which puts silent-partition detection at 4–6 minutes — a deliberate non-change in a fix release.

Version

3.0.0. connect() rejects with typed errors rather than strings, requests that used to wait on a lost connection now fail, and fatalErrors carries a different contract — consumers have to look. No IEventBus signature changed; ConnectionInfo gains the optional connectTimeoutMs.

Harness

Every test that starts a gateway does so through test/GatewayContainer.ts, with one rule: CONTINUUM_GATEWAY_IMAGE names the image, a supplied image is never pulled (a local build is not replaced by whatever CI last published mid-run), the default snapshot always is. The default is mindsignited/continuum-gateway-server:3.1.0-SNAPSHOT — the namespace the suite pointed at before does not exist on Docker Hub. The global teardown no longer stops a container it never started, and the test proxy targets org.kinotic, the package develop actually publishes.

🤖 Generated with Claude Code

https://claude.ai/code/session_01SAsEfzyJuEGKYmimnkuviH