Skip to content

driver/postgres: bound the startup by ConnectTimeout and end it with the caller's context (v1.6.0, Refs #1212) - #1216

Merged
FumingPower3925 merged 4 commits into
mainfrom
fix/postgres-startup-connect-timeout-1212
Nov 1, 2026
Merged

FumingPower3925 merged 4 commits into
mainfrom
fix/postgres-startup-connect-timeout-1212

Conversation

@FumingPower3925

@FumingPower3925 FumingPower3925 commented Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

driver/postgres: bound the startup by ConnectTimeout and end it with the caller's context. Refs #1212 (found by the review of #1209; same class as #1205).

The problem

Options.ConnectTimeout (the DSN's connect_timeout) is documented as bounding "dial + startup", but it only reached the net.Dialer. doStartup ran on the caller's context. The issue says a server that accepts TCP and never completes the startup holds Open/acquire "until the caller's context ends". Measured on main, it is worse than that (RED run below): the caller's own context did not end it either.

  • loop dials (dialConn on unix, dialConnNetConn): wait sends no CancelRequest (the startup has no BackendKeyData yet) and then waits up to 30 s for a reply that is not coming;
  • direct dial (dialDirectConn, which Pool uses standalone and on async engines): driveDirect checks the context only between reads, so a read blocked on the silent socket ignores it: never.

The change

  • doStartup is now bounded by two things, whichever comes first: the caller's context (a shorter deadline or a cancellation ends it at once) and ConnectTimeout. A startupBound fails the startup request and, in direct mode, sets a read deadline of "now" on the socket so the blocked read returns. The dial then closes the conn, as for any startup error. The timeout error is celeris-postgres: startup timed out (connect_timeout 200ms): context deadline exceeded (matches errors.Is(err, context.DeadlineExceeded)); the caller's own context error comes back as it is.
  • One budget for dial + startup (what the doc already said, and what libpq's connect_timeout means): each dial path records its start (pgConn.connectDeadline), the startup ends at that deadline. Redis (driver/redis: bound the connection handshake by DialTimeout (v1.6.0, Refs #1205) #1209) and memcached (driver/memcached: the binary-mode handshake is not bounded by the dial timeout #1213) use two separate budgets (DialTimeout each); postgres differs because its doc says "dial + startup". A slow connect leaves the startup less time.
  • The bound covers the whole startup exchange: the optional authentication exchange (cleartext/MD5/SCRAM round trips are the same request), ParameterStatus, BackendKeyData, up to ReadyForQuery. There is no SSL negotiation phase: grep -rn "SSLRequest\|80877103" internal/driver/postgres driver/postgres finds nothing; sslmode=require|verify-* is refused up front with ErrSSLNotSupported, prefer|allow downgrade to plaintext. So there is nothing to bound there.
  • A bound that fires at the instant the startup completes is handled (the startup is reported as ended, the conn is dropped); a bound that did not fire is disarmed, so it cannot touch the conn afterwards (TestStartupBoundDoesNotOutliveASuccessfulStartup1212 pings past the bound).
  • Docs: Options.ConnectTimeout now says what it bounds, the error, and that zero means no client-side timeout.

Unset ConnectTimeout (decision for the lead)

dsn.go documents zero as "no client-side timeout" and no default is applied anywhere (libpq/pgx behave the same). I kept that. The consequence, to be explicit: with context.Background() and no connect_timeout, a silent server still waits for ever. What changed for that case is that the caller's context now ends the startup at once (30 s / never before). Redis and memcached default to 5 s. If we want the same default here, it is a one-line change in the dial (connectDeadline), but it is a behaviour change for every postgres user, so I did not make it without a decision.

The decision the issue asked for (bound vs correct the docs)

Bound it. The doc was right about the intent, the code was wrong; correcting the doc would have left the silent-server hang as designed behaviour, and the caller's-context bug (30 s / never) needed the same mechanism anyway.

Pool acquisition path

Pool.dial only calls dialDirectConn or dialConn; both are covered, and TestPoolAcquireIsBoundedByConnectTimeout1212 goes through Open(...connect_timeout=1) + Ping(context.Background()) (standalone, so direct mode). The engine-mode pool dials through dialConn, which the eventloop subtests cover. After the failed dial the pool holds no conn (Stats().Open == 0).

Commits: f6e6583 the tests (RED on main), 09b3a12 the fix. startup_budget_1212_test.go is in the second commit because it references the new connectDeadline (it does not compile on main).

RED / GREEN

RED, f6e6583 alone (macOS arm64, go test ./driver/postgres -run 1212 -v; every failure is the watchdog message, not a test timeout):

--- FAIL: TestStartupIsBoundedByConnectTimeout1212          (silent|no-ready-for-query) x (direct|eventloop|netconn): 6/6 "the dial is still blocked after 2s: the startup is not bounded"
--- FAIL: TestStartupEndsWithTheCallersContext1212          (shorter-deadline|cancel-without-connect-timeout) x 3 paths: 6/6 still blocked after 2s
--- FAIL: TestPoolAcquireIsBoundedByConnectTimeout1212      "Ping is still blocked after 2s: connect_timeout=1 does not bound the startup"
--- PASS: TestStartupBoundDoesNotOutliveASuccessfulStartup1212   (guard, passes on main by design)

GREEN, 09b3a12: all pass. The bounded dials return after 200 ms (ConnectTimeout 200 ms), the caller-context ones after 200 ms, the pool one after 1.00 s; the budget test after 150 ms against a ConnectTimeout of 10 s (the dial "used" 9.85 s). Linux 2 CPU -race -count=20: the dials returned after 150-210 ms (bound 200 ms) and 1.00-1.01 s (pool).

Each test takes at most 1.0 s (the pool one); the other 1212 tests 0.2-0.3 s, parallel subtests.

Mutants (each applied alone to 09b3a12, go test -run 1212 -count=2)

mutant tests that fail
M1 no ConnectTimeout timer TestStartupIsBoundedByConnectTimeout, TestPoolAcquireIsBoundedByConnectTimeout, TestStartupHasWhatTheDialLeft...
M2 the caller's context does not end the startup TestStartupEndsWithTheCallersContext (all 6)
M3 direct mode: blocked read not woken TestStartupIsBoundedBy..., TestStartupEndsWithTheCallersContext, TestPoolAcquire..., TestStartupHasWhatTheDialLeft...
M6 timer counts from the startup, not the dial TestStartupHasWhatTheDialLeftOfConnectTimeout
M7 no done guard and timer not stopped (bound outlives a successful startup) TestStartupBoundDoesNotOutliveASuccessfulStartup (all 3 paths) + 1
M8a / M8b / M8c direct / unix / netconn dial does not set connectDeadline TestStartupIsBoundedByConnectTimeout (a: also the pool test)
M9 timeout error does not match context.DeadlineExceeded TestStartupIsBoundedBy..., TestPoolAcquire..., TestStartupHasWhatTheDialLeft...

An earlier version passed context.WithoutCancel(ctx) to wait so it could not race the bound. Mutant "pass the caller's context" survived every test, and the interleaving analysis shows why: whichever of the abort and driveDirect's own ctx.Err() branch runs first, the abort fails the request before it kicks the read, and the drain loop checks the request's flag before it reads. It was an equivalent mutant, so I removed the wrapper.

Checks

  • Whole driver/postgres package, -race -count=1, Linux docker golang:1.27.3 --cpus 2: ok (60 s). -race -run 1212 -count=20: Linux ok (61 s), macOS ok (39 s).
  • GOOS=windows go vet ./driver/postgres: clean. golangci-lint v2.15.0 GOOS=linux: nothing in the files of this PR (two errcheck findings in untouched test files are on main too).
  • Not run locally: the windows-latest and macos-latest Portable fallback jobs. The 1212 tests use dialConnNetConn over the portable loop, a 200 ms bound and a 2 s watchdog; loopback only, no reliance on clock resolution (the lower-bound checks are at half the bound).

Release note

connect_timeout (Options.ConnectTimeout) now bounds the dial and the startup together, as its doc says: a server that accepts the TCP connection and does not complete the startup (authentication, ParameterStatus, ReadyForQuery) in that time fails the dial with an error matching context.DeadlineExceeded, instead of holding Open/acquire. A server that is slow to authenticate (SCRAM with a high iteration count, a loaded server) now needs to answer within the whole budget, which before it did not: users who set a small connect_timeout may see dial errors where the startup used to take longer. The caller's context also ends a startup in progress at once (it took about 30 s on the event-loop paths and never in direct mode). With no connect_timeout the dial is still unbounded apart from the caller's context.

Review r1 follow-ups (commit 70248fb)

  • Budget counts from the start of a real dial (MINOR): a loopback connect takes no time, so the test slows the connect with a hook that runs right after the clock starts (beginDial, the one place all three dial paths start the clock). TestSlowConnectEatsTheConnectTimeoutBudget1212: connect 400 ms of a 500 ms ConnectTimeout, the dial ends after about 500 ms (not 900). Mutant "the clock starts after the connect" fails it on the three paths.
  • Bound armed before the first write (NIT): doStartup arms the bound before writeRaw; the abort sets the socket's read and write deadline, so a blocked write ends too. A write error still goes through failReq and finish.
  • DNS lookup (NIT, pre-existing): resolveAddr runs before the clock and without the context; documented in the ConnectTimeout doc rather than moved (it needs a context-aware resolver and a change of pgConn.addr).
  • Negative connect_timeout (NIT, pre-existing): rejected at DSN parse with connect_timeout = -1: must be >= 0 (0 or absent: no timeout) (key=value and URL forms; TestParseDSNRejectsANegativeConnectTimeout1212, mutant "no rejection" fails it).
  • No default for the unset case in this PR (separate decision).
  • Checks on 70248fb: -race -run '1212|ParseDSN' -count=20 Linux docker --cpus 2 ok (91 s), macOS ok (69 s); whole driver/postgres -race Linux ok (60 s); GOOS=windows go vet clean; golangci-lint (own cache) only the two pre-existing errcheck findings.

@FumingPower3925 FumingPower3925 added this to the v1.6.0 milestone Nov 1, 2026
@coderabbitai

coderabbitai Bot commented Nov 1, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Caution

Review failed

The pull request is closed.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

…get; arm the startup bound before the first write; reject a negative connect_timeout (Refs #1212)
@codecov

codecov Bot commented Nov 1, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 93.18182% with 3 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
driver/postgres/conn.go 92.10% 3 Missing ⚠️

📢 Thoughts on this report? Let us know!

@FumingPower3925
FumingPower3925 marked this pull request as ready for review November 1, 2026 19:51
@FumingPower3925
FumingPower3925 merged commit 3b4db8c into main Nov 1, 2026
56 checks passed
@FumingPower3925
FumingPower3925 deleted the fix/postgres-startup-connect-timeout-1212 branch November 1, 2026 19:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant