Repository navigation
fix(mapper): stop a slow control plane from piling up abandoned sessions - #66
Merged
Merged
Conversation
When map generation fell behind, a long poll waited forever in AddNode for its initial map. The client gave up after two minutes and opened a new poll, but the old session stayed registered: production had 82 connections for one node and 626 goroutines waiting, and the one worker still built a full map for every dead session before reaching a live one. A session now waits with its request context. When the client leaves, the connection is detached and the queued map is dropped unbuilt. After 60 seconds the client is answered 503 with a Retry-After of 30 to 89 seconds, which clients from capability version 147 (Tailscale 1.104) wait out instead of retrying at once. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
trancong12102
force-pushed
the
fix/batcher-abandoned-sessions
branch
from
October 14, 2026 09:13
82d2613 to
3baf989
Compare
Merged
trancong12102
added a commit
that referenced
this pull request
Oct 14, 2026
A release is ready. Read the notes below as an operator would, fix a wrong line by editing the commit's PR body with a BEGIN_COMMIT_OVERRIDE block, and merge to tag and publish. --- ## [0.58.2](v0.58.1...v0.58.2) (2026-10-14) ### Fixes * **capver:** track Tailscale 1.104 and refuse clients older than 1.84 ([#65](#65)) ([53a652f](53a652f)) * **mapper:** stop a slow control plane from piling up abandoned sessions ([#66](#66)) ([f4f8b87](f4f8b87)) * **state:** skip an app on every connector for machines at one connector's site ([#64](#64)) ([03bfc35](03bfc35)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Third PR after the 2026-10-14 incident (#62 fixed the slow route reduction that triggered it). This one removes the amplifier.
What happened. Once map generation fell behind, every long poll waited in
Batcher.AddNode→MapResponseFromChange. That wait had no context and no timeout. The client's watchdog cancelled after 120 s and it opened a new poll. The old session's connection was already registered, so it stayed counted. Production showed:active_connectionsper node in/debug/batcher;queueWorkand 176 inMapResponseFromChange, the oldest for 180 minutes;Change.
AddNodeContext(ctx, …)is whatserveLongPollnow uses;AddNodewraps it withcontext.Background(), so the 57 test callers are untouched.queueWorkand the wait for the result both honour the context.slopscale_mapper_changes_dropped_total{reason="abandoned"}.ErrBatcherBusy). The poll then answers 503 withRetry-After30–89 s. Clients at capability version 147+ (Tailscale 1.104) honour that and wait instead of retrying at once; older ones back off as on any failure. A client that already left gets nothing written and a debug log.Tests.
TestAddNodeContextAbandonedruns on an unstarted batcher, so the initial map can never be built, like a saturated worker. It checks that the call returns with the cause, the connection stops counting, and once a worker starts the abandoned map is never built (the mapper is nil, so building it would panic). 200 runs pass under-race.go test -short ./hscontrol/ ./hscontrol/{mapper,state,servertest}/...and golangci-lint pass.There is no automated test for the 503 path:
initialMapWaitis a 60 s constant. The switch inpoll.gois three lines.🤖 Generated with Claude Code