fix: retry a rate-limited turn instead of failing the caller

[EthanThatOneKid]

Data is up but not answering. POST /ask returns 502 whenever the agent's provider rate-limits the turn, and the provider is a free tier, so that happens on ordinary traffic.

Measured

Two consecutive live turns on 2026-09-26:

POST http://127.0.0.1:8788/ask  -> HTTP 502
  "code": 429, "status": "Too Many Requests", "retryable": true,
  "retryDelay": "45s", "error_type": "llm_error"

POST http://127.0.0.1:8788/ask  -> HTTP 502   (45s later, fresh run id)
  "retryDelay": "31s"

A turn against the same agent answered normally at 06:06:51, ~41s after the last 502, so the limit is per-minute capacity, not a broken key or a durable daily cap. The caller eats a hard failure on a minute the provider is busy.

Change

A rate-limited turn never reached the model, so nothing was written to the conversation or to memory, and the same question can be asked again. The service now waits and re-runs that turn in place, honoring the provider's own retryDelay when it names one and backing off otherwise, up to DATA_PROVIDER_RETRY_ATTEMPTS (default 2), with a cap of DATA_PROVIDER_RETRY_CAP_MS (default 60000) on any single wait.

The retry re-runs the same session, so a recovered turn continues the conversation the caller already had; it never starts a new one.

Three guards, so this cannot paper over a real error:

• Only explicitly retryable provider failures are retried (isRetryableProviderFailure). A missing conversation, an unconfigured provider, a timeout, and any other failure are untouched.
• The wait plus the retry must fit inside DATA_LETTA_TIMEOUT_MS; otherwise the retry is skipped and the original error is returned as-is.
• The budget check reserves at most a quarter of the turn timeout, so a shortened DATA_LETTA_TIMEOUT_MS still retries rather than refusing to.

What this does not fix

Data still has one provider and no second one, so a durable outage — an exhausted daily quota, or a provider that stays down — still fails the turn. That decision is wazootech/data#9 item 5.

Verification

npm run typecheck clean, npm test 34/34 (8 new in lib/provider-retry.test.ts).

End to end against a stub letta, all three paths:

Stub behaviour	Result	Child processes
429, then success	200 with the answer	2
429, always	502 with the provider's error	3 (initial + 2 retries)
Provider is not configured	502, no retry logged	1

[wazoocomputer]

This pull request modifies how Data handles rate-limited responses from its agent provider. Previously, a rate-limited turn would result in a 502 error for the caller. With this change, Data will now retry the turn, respecting the provider's retryDelay if specified, up to a configurable number of attempts and with a maximum wait time.

File Path	Change Type	Summary of Change
channels/http/README.md	modified	Updated documentation to reflect retry mechanism and new environment variables.
channels/http/index.ts	modified	Implemented retry logic for rate-limited provider responses and added associated environment variable parsing.
lib/provider-retry.test.ts	added	Added unit tests for the provider retry logic.
lib/provider-retry.ts	added	Implemented functions to detect retryable provider failures, extract retry delays, and plan retry intervals.
services/http-api.md	modified	Updated service documentation to include new environment variables and a change log entry for the retry feature.

Reviewers should start by examining lib/provider-retry.ts for the core retry logic.

[wazoocomputer]

This pull request modifies how Data handles rate-limited responses from its agent provider. Previously, a rate-limited turn would result in a 502 error for the caller. With this change, Data will now retry the turn, respecting the provider's retryDelay if specified, up to a configurable number of attempts and with a maximum wait time.

File Path	Change Type	Summary of Change
channels/http/README.md	modified	Updated documentation to reflect retry mechanism and new environment variables.
channels/http/index.ts	modified	Implemented retry logic for rate-limited provider responses and added associated environment variable parsing.
lib/provider-retry.test.ts	added	Added unit tests for the provider retry logic.
lib/provider-retry.ts	added	Implemented functions to detect retryable provider failures, extract retry delays, and plan retry intervals.
services/http-api.md	modified	Updated service documentation to include new environment variables and a change log entry for the retry feature.

Reviewers should start by examining lib/provider-retry.ts for the core retry logic.

[EthanThatOneKid]

Superseded by #22. Data now runs as a Zo persona reached directly by the Discord bridge, so the retry this adds has no caller: channels/http/ and its provider keys moved to archives/letta-brain/. Closing — the reasoning is carried forward in that directory's README.