Skip to content

models poll: a provider that hits D1 overload is failed and marked degraded instead of retried after the wait #635

Closed

Description

@tombeckenham

Problem

#625 / #627 stopped a single D1 overload from failing every provider polled in the next few seconds. Now only the provider that hit the overload fails. That provider still loses its poll, emits `ingest_failed`, and is written `degraded`, even though D1 recovers within the 10-second window that `pollAllProviders` already waits out (`D1_WINDOW_WAIT_MS`).

PostHog `ingest_failed` with `D1 DB is overloaded` since #627 was deployed (2026-11-10 UTC):

time provider
04:30:24 openai
05:11:12 worldlabs (+ degraded-status write failed)
05:11:12 zai-coding-plan (+ degraded-status write failed)
05:47:47 meta
07:00:20 openai

That's about one lost provider poll per hour. Each also flips `providers.status` to degraded until a later success, and the degraded write itself can fail on the same overload.

Ask

When `d1WindowExhausted(error)` is true for a provider's poll (`src/server/ingest/poll-models.ts`), wait `D1_WINDOW_WAIT_MS` and retry that provider once before recording it as failed. A second overload still fails loudly, as now: one retry only, no loop. Don't write `degraded` for a failure that the retry recovered.

The 05:11 pair came from the spec-sync shard firing (`sync.ts`), not the models poll. Check whether the shard loop needs the same wait-and-retry-once.

Acceptance

  • A unit test: a provider whose first poll throws a D1 overload error is polled again after the wait, and on success records no `ingest_failed` and no degraded status.
  • A second consecutive overload still records `ingest_failed`.
  • After deploy, PostHog shows no single-provider `D1 DB is overloaded` `ingest_failed` for 24h, unless two in a row hit the same provider.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions