Skip to content

fix(mapper): stop a slow control plane from piling up abandoned sessions - #66

Merged
trancong12102 merged 1 commit into
mainfrom
fix/batcher-abandoned-sessions
Oct 14, 2026
Merged

trancong12102 merged 1 commit into
mainfrom
fix/batcher-abandoned-sessions

Conversation

@trancong12102

Copy link
Copy Markdown
Contributor

Third PR after the 2026-10-14 incident (#62 fixed the slow route reduction that triggered it). This one removes the amplifier.

What happened. Once map generation fell behind, every long poll waited in Batcher.AddNode → MapResponseFromChange. That wait had no context and no timeout. The client's watchdog cancelled after 120 s and it opened a new poll. The old session's connection was already registered, so it stayed counted. Production showed:

  • 60–85 active_connections per node in /debug/batcher;
  • 450 goroutines blocked in queueWork and 176 in MapResponseFromChange, the oldest for 180 minutes;
  • the single worker building full maps for dead sessions ahead of live ones.

Change.

  • AddNodeContext(ctx, …) is what serveLongPoll now uses; AddNode wraps it with context.Background(), so the 57 test callers are untouched. queueWork and the wait for the result both honour the context.
  • When the context ends, the connection is detached, the same as the existing error path.
  • Synchronous work carries the caller's context. A worker that reaches it after the caller gave up drops it unbuilt and counts it in slopscale_mapper_changes_dropped_total{reason="abandoned"}.
  • The initial map waits at most 60 s (ErrBatcherBusy). The poll then answers 503 with Retry-After 30–89 s. Clients at capability version 147+ (Tailscale 1.104) honour that and wait instead of retrying at once; older ones back off as on any failure. A client that already left gets nothing written and a debug log.

Tests.

  • TestAddNodeContextAbandoned runs on an unstarted batcher, so the initial map can never be built, like a saturated worker. It checks that the call returns with the cause, the connection stops counting, and once a worker starts the abandoned map is never built (the mapper is nil, so building it would panic). 200 runs pass under -race.
  • go test -short ./hscontrol/ ./hscontrol/{mapper,state,servertest}/... and golangci-lint pass.

There is no automated test for the 503 path: initialMapWait is a 60 s constant. The switch in poll.go is three lines.

🤖 Generated with Claude Code

When map generation fell behind, a long poll waited forever in AddNode
for its initial map. The client gave up after two minutes and opened a
new poll, but the old session stayed registered: production had 82
connections for one node and 626 goroutines waiting, and the one worker
still built a full map for every dead session before reaching a live one.

A session now waits with its request context. When the client leaves,
the connection is detached and the queued map is dropped unbuilt. After
60 seconds the client is answered 503 with a Retry-After of 30 to 89
seconds, which clients from capability version 147 (Tailscale 1.104)
wait out instead of retrying at once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@trancong12102
trancong12102 force-pushed the fix/batcher-abandoned-sessions branch from 82d2613 to 3baf989 Compare October 14, 2026 09:13
@trancong12102
trancong12102 merged commit f4f8b87 into main Oct 14, 2026
34 checks passed
@trancong12102
trancong12102 deleted the fix/batcher-abandoned-sessions branch October 14, 2026 09:43
trancong12102 added a commit that referenced this pull request Oct 14, 2026
A release is ready. Read the notes below as an operator would, fix a
wrong line by editing the commit's PR body with a BEGIN_COMMIT_OVERRIDE
block, and merge to tag and publish.
---


##
[0.58.2](v0.58.1...v0.58.2)
(2026-10-14)


### Fixes

* **capver:** track Tailscale 1.104 and refuse clients older than 1.84
([#65](#65))
([53a652f](53a652f))
* **mapper:** stop a slow control plane from piling up abandoned
sessions ([#66](#66))
([f4f8b87](f4f8b87))
* **state:** skip an app on every connector for machines at one
connector's site
([#64](#64))
([03bfc35](03bfc35))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant