Skip to content

LVT-382: Retry connections while sleeping Postgres wakes - #134

Merged
jackshim415 merged 2 commits into
mainfrom
codex/lvt-382-cold-start
Nov 20, 2026
Merged

jackshim415 merged 2 commits into
mainfrom
codex/lvt-382-cold-start

Conversation

@jackshim415

Copy link
Copy Markdown
Contributor

What changed

The isolated Railway sleep test returned HTTP 500 with ETIMEDOUT on the first database-backed request; the next request succeeded. Add two bounded retries for transient PostgreSQL connection acquisition failures while the service wakes.

Only obtaining a connection is retried. SQL statements, writes and transactions are never replayed; authentication and statement errors fail immediately. Each Prisma connection creates an owned pool which disconnects fully and can be recreated on a later query, preserving finite Worker execution.

Areas affected

  • Server
  • Worker and shared database client

Contracts and data

  • No API, report-schema, authentication, deep-link, season-metric, or database contract changed
  • Tenant and source-team visibility was reviewed
  • No credentials, production data, or private scouting notes are included

Verification

  • Shared database build passed.
  • Three retry behavior tests passed: transient recovery, non-retryable errors and bounded exhaustion.
  • Disposable local PostgreSQL tests passed for writes, relations, rollback, transactions, complete pool disposal and reconnect after disconnect.
  • Worker typecheck passed. Railway cold-query verification is running before production Cron activation.

Deployment

No migrations or new dependencies. Production Cron remains staged until the cold-query check passes. Only Postgres needs Serverless sleeping; Worker uses normal Cron execution. Related: LVT-382; PR #133 adds event-aware scheduled runs.

The Railway preview returned ETIMEDOUT on its first database-backed request after sleep. Retry only transient pool connection acquisition failures, never SQL execution, and retain process-owned pool disposal across reconnects. Bound retries and preserve authentication and statement failures.
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-19T17:16:53.618261Z 0418a2e PR opened
🔒 Security Review ✅ Completed 2026-11-19T17:19:17.956682Z 0418a2e PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Lovat AI review

Status: completed · Model: z-ai/glm-5.3-flash · Findings: 1

Reviewed commit: 7ee53d803def1f5d989a684a70f910a95fb0a4df

Critical: 0 · High: 1 · Medium: 0 · Low: 0

Overall assessment

The retry helper itself is well-bounded and preserves the original error, but the adapter integration is unsafe: every connect() call constructs a brand-new pg Pool (up to 10 connections each) that nothing ever disposes, and the pool created by the superclass is orphaned. Under Prisma, connect() is invoked per connection slot, so this can multiply Postgres connections instead of retrying them. The pool.connect monkey-patch also retries pool-saturation timeouts, which stacks duplicate waiters.

Findings

High (1)

  • packages/db/src/connection-retry.ts:84 — ConnectionRetryAdapter.connect() creates a new undisposed Pool on every connect call (PERFORMANCE · confidence 0.75)
    override connect() returns new PrismaPg(createRetryPool(this.poolConfig), { disposeExternalPool: true }).connect(). Prisma driver adapters call connect() once per connection slot (including per transaction), so each call spins up a fresh Pool with max:10. disposeExternalPool only takes effect if that inner adapter is disposed, but Prisma only ever disposes the outer ConnectionRetryAdapter, so none of the inner pools are ever closed. Additionally, super(poolConfig) creates a pool that is never used or disposed. The result is unbounded Postgres connection growth and leaked sockets, the opposite of the intended 'retry while Postgres wakes' behavior.

    Suggestion: Create the retrying pool once in the constructor (wrapping the single Pool with acquireConnection in pool.connect) and have connect() return connections from that same pool; do not instantiate a new PrismaPg per connect().

Review details

  • Review batches: 1
  • Diff characters reviewed: 7,362
  • Tokens: 4,278 prompt + 1,392 completion
  • Estimated inference cost: $0.0013
  • Findings below the 0.75 confidence threshold: 1

Limitations

  • The reviewer examined only changed lines and their diff context, not the whole repository.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0418a2eba8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread packages/db/src/connection-retry.ts Resolved

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lovat AI review: 1 new finding(s) at 7ee53d803def1f5d989a684a70f910a95fb0a4df.

The persistent Lovat AI review comment contains the latest complete result.

  • packages/db/src/connection-retry.ts:84 — ConnectionRetryAdapter.connect() creates a new undisposed Pool on every connect call (PERFORMANCE · confidence 0.75)

Comment thread packages/db/src/connection-retry.ts Resolved

@jackshim415 jackshim415 left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Final review: wrapped pg-pool timeouts are covered, retries occur only before a SQL connection is acquired, and statement/transaction execution is not replayed. PostgreSQL integration tests verify writes, rollback, transactions and zero owned connections after disconnect/reconnect. Required CI, Server, Worker and cross-app contracts passed. Three real Railway cold probes succeeded in 1,405 / 1,394 / 1,366 ms while recovering ETIMEDOUT and ECONNREFUSED. The pool-leak finding was checked against the installed adapter lifecycle and measured pg_stat_activity disposal test and is not applicable. Ready for the authorized Cron cutover.

@jackshim415
jackshim415 merged commit 84fb18c into main Nov 20, 2026
24 checks passed

This branch is being deployed

1 in progress deployment
lovat / lvt-382-verification — 7ee53d80 Deployed Nov 20, 2026 by railway-app[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant