You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
models poll: a provider that hits D1 overload is failed and marked degraded instead of retried after the wait #635
#625 / #627 stopped a single D1 overload from failing every provider polled in the next few seconds. Now only the provider that hit the overload fails. That provider still loses its poll, emits `ingest_failed`, and is written `degraded`, even though D1 recovers within the 10-second window that `pollAllProviders` already waits out (`D1_WINDOW_WAIT_MS`).
PostHog `ingest_failed` with `D1 DB is overloaded` since #627 was deployed (2026-11-10 UTC):
time
provider
04:30:24
openai
05:11:12
worldlabs (+ degraded-status write failed)
05:11:12
zai-coding-plan (+ degraded-status write failed)
05:47:47
meta
07:00:20
openai
That's about one lost provider poll per hour. Each also flips `providers.status` to degraded until a later success, and the degraded write itself can fail on the same overload.
Ask
When `d1WindowExhausted(error)` is true for a provider's poll (`src/server/ingest/poll-models.ts`), wait `D1_WINDOW_WAIT_MS` and retry that provider once before recording it as failed. A second overload still fails loudly, as now: one retry only, no loop. Don't write `degraded` for a failure that the retry recovered.
The 05:11 pair came from the spec-sync shard firing (`sync.ts`), not the models poll. Check whether the shard loop needs the same wait-and-retry-once.
Acceptance
A unit test: a provider whose first poll throws a D1 overload error is polled again after the wait, and on success records no `ingest_failed` and no degraded status.
A second consecutive overload still records `ingest_failed`.
After deploy, PostHog shows no single-provider `D1 DB is overloaded` `ingest_failed` for 24h, unless two in a row hit the same provider.
Activity
Retry a D1 overload once before failing that provider (#635)
Retry a D1 overload once before failing that provider (#635) (#636)