Stop building a Supabase client on every authenticated API request - #552
Merged
Merged
Conversation
ugig.net has been 502 since 2026-09-07 13:12 UTC. The container died with
`FATAL ERROR: Ineffective mark-compacts near heap limit` at 1046MB after
2h49m of uptime, restarted, and died the same way until Railway stopped
retrying. Redeploying restores it and buys about three hours, which is what
happened on 2026-09-01 and again on 2026-09-03.
authenticateApiKey() is the leak. It runs on every request that carries an
API key, and it built a fresh Supabase client per call:
const supabaseAdmin = createSupabaseAdmin<Database>(url, serviceKey);
Two things outlive the request that made them. Each createClient() allocates
a RealtimeClient holding WebSocket state, and passing no auth options leaves
`autoRefreshToken` at its default of true, which starts a token-refresh
interval that nothing ever clears. Neither is reachable once the response is
sent, and neither is released.
This is the shape #544 went looking for and the reason every other Supabase
client in the app is already careful: lib/supabase/server.ts and
lib/supabase/middleware.ts pass `disconnectRealtime: true`,
authenticateWithToken() calls realtime.disconnect(), and
lib/supabase/service.ts memoises a single service client and disconnects it.
This one call site did neither, on the hottest path in the app.
It now uses createServiceClient(). Five other routes built their own admin
client per request the same way and are switched over too: the two message
routes, the Stripe webhook, and directory/fetch-meta. api/callback/oauth and
api/auth/agentpass-login keep their local client, because they read
oauth_identities and other tables missing from the generated Database types
and the typed client does not compile against them; they call
realtime.disconnect() instead, which closes the same hole.
Why it surfaced now: the OVH range 51.254.0.0/15 accounted for 398 of the
~467 API requests in the last log window, all to authenticated endpoints. It
is a signed-in agent client, not a training crawler, so the crawl gateway
exempts it by design and every one of its requests minted a client. Nothing
is being blocked here; the fix is that authenticating no longer allocates.
Two regression tests, both verified against the old code first, where they
fail with 3 clients built for 3 authentications and 0 of 6 disconnected.
Full suite 2095 passing, tsc clean, lint unchanged at 0 errors, build green
and still listing the proxy.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013xPsu92cNeV4t5SEGFepL6
ThreatCrush Security Scan45 finding(s) HIGH/CRITICAL: 1 | MEDIUM: 8 | LOW: 36
Snippets are redacted; ThreatCrush never prints matched credential material. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The outage
ugig.net returned 502 from 2026-09-07 13:12 UTC until a manual redeploy today, about 27 hours. The container died with
FATAL ERROR: Ineffective mark-compacts near heap limitat 1046MB after 2h49m of uptime, restarted, and died the same way until Railway stopped retrying. The site is already back up — this PR is so it stays up.This is the third time: 2026-09-01 (#540 raised the cap 384 → 1024), 2026-09-03 (#544 closed an SSE channel leak), and now. #540 said an OOM at 1GB would be evidence of a real leak. This is that evidence, and it is a different leak.
The leak
authenticateApiKey()runs on every request carrying an API key, and it built a fresh Supabase client per call:Two things outlive the request that made them:
createClient()allocates aRealtimeClientholding WebSocket stateautoRefreshTokenat its default oftrue, which starts a token-refresh interval that nothing ever clearsNeither is reachable once the response is sent, and neither is released. That is the "per-request resource that outlived its request" shape, on the hottest path in the app.
Every other Supabase client already guards against exactly this:
lib/supabase/server.tsdisconnectRealtime: truelib/supabase/middleware.tsdisconnectRealtime: truelib/supabase/service.tsrealtime.disconnect()authenticateWithToken()realtime.disconnect()lib/auth/agentpass.tscreateServiceClient()lib/auth/api-key.tsThe change
authenticateApiKey()now uses the memoisedcreateServiceClient().Five other routes built their own admin client per request the same way and are switched over too:
conversations/[id]/messages,messages/send,stripe/webhook, anddirectory/fetch-meta.callback/oauthandauth/agentpass-loginkeep their local client — they readoauth_identitiesand other tables missing from the generatedDatabasetypes, so the typed client does not compile against them. They callrealtime.disconnect()instead, which closes the same hole. Regenerating the database types is the follow-up that would let those two use the singleton as well.Why it surfaced now
The OVH range
51.254.0.0/15accounted for 398 of the ~467 API requests in the last log window, all to authenticated endpoints (/api/conversations/*/messages,/api/applications/my,/api/profile). It is a signed-in agent client, not a training crawler, so the crawl gateway exempts it by design — and every one of its requests minted a client.Nothing is blocked here. The gateway's
denyCidrsis checked beforeexempt, so adding that range would 403 a signed-in agent account, which is the customer ugig is built for. The fix is that authenticating no longer allocates.Verification
tsc --noEmitcleaneslintunchanged at 0 errors, 40 pre-existing warningspnpm buildgreen, still listingƒ Proxy (Middleware)Separate issue found, not fixed here
src/proxy.tscaches throttled response bodies under a key of`${ip}:${path}`with no user identity in it, and three of the four throttled paths return per-user data (/api/notifications,/api/wallet/balance,/api/wallet/transactions). Two users behind one NAT or CGNAT can be served each other's wallet balance and notifications for up to 30s. Live since #546 put the proxy intosrc/on 2026-09-05. Worth its own PR.🤖 Generated with Claude Code
https://claude.ai/code/session_013xPsu92cNeV4t5SEGFepL6