Repository navigation
Daemon fast-send: sub-second chat delivery via the gateway relay socket - #85
Merged
Merged
Conversation
Time-to-first-token was dominated by transport, not the model: queue pickup + a fresh SSH handshake + remote CLI startup cost 10-15s before the gateway even saw the message. provisiond already holds an open loopback WebSocket to the gateway, so sends now ride it: - Redis outbox per server; provisiond (v0.5.0, capability chat-send-v1) long-polls GET /chat/outbox, fires chat.send over the relay socket within ~100ms, and acks via POST /chat/outbox/ack - DaemonChatSendService drops the send into the outbox and blocks briefly on the ack; ChatController attempts it inline in the web request (2s ack window) so the happy path skips the queue entirely, and SendAgentChatMessageJob retries it (4s window) before falling back to the unchanged SSH path — chat.send is idempotent by key, so a late daemon send after fallback is harmless - Scope: text-only messages; attachments keep the SSH staging path - Stale outbox entries (>30s, from requests that already fell back) are skipped at poll time Measured from a live browser against the E2E droplet: POST acked in ~600ms, first streamed delta at 7.0s total (was 17.8s) — the remainder is model TTFT, no longer transport.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to #84's streaming fix, attacking the other half of chat latency: time-to-first-token, which was ~10–15s of pure transport (queue pickup + SSH handshake + remote CLI startup) before the gateway saw the message.
provisiond already holds an authenticated loopback WebSocket to the gateway (the relay that streams deltas up). Sends now ride the same socket: the web request drops the message into a per-server Redis outbox and blocks ~600ms on the daemon's ack; the daemon long-polls the outbox, fires
chat.sendover its socket, and acks. The queued job retries the fast path with a longer window, then falls back to the unchanged SSH path —chat.sendis idempotent by key, so every failure mode degrades to today's behavior rather than losing a message.Measured (live browser vs E2E droplet)
Notes
chat-send-v1), 41 vitest tests; 7 new Pest tests; full suite 1147 green.