[hallelx2]

The deployed server's answer-model client never retried. One Vertex 429 failed the whole /v1/answer request.

Measured

On vlb-engine I sent the pilot's 50 FinanceBench questions to /v1/answer, four at a time:

• 28 returned 500 "answer generation failed".
• Every one was a first-attempt refusal: the engine log shows synthesis failed: gemini: 429: vertex: Resource exhausted, after 67 to 422 ms.

Cause

cmd/server/main.go built reqtiming.WrapClient(limit.Client(...)(registry.Client())). The comment above it says it sits "outside the limiter and retries", but nothing retried. The Judge has retry.NewJudge; the generative client and the BYOK client from byokFactory had no retry.

Change

• The server's model client and every BYOK client go through retry.New:
  • 5 retries, with exponential backoff from 1 s, capped at 20 s;
  • Retry-After is honoured, clamped to the cap;
  • 400s and the other non-transient classes are not retried.
• The retry sits outside the limiter, so each attempt takes a slot.

Verification

• go test ./... passes on vlb-engine.
• New tests in cmd/server/retry_test.go:
  • two 429s and then an answer yields the answer after 3 calls;
  • a 400 is tried once;
  • the config is checked.

Closes HAL-3206

Summary by Sourcery

Add bounded retries for transient generative-model failures across server and BYOK clients.

Bug Fixes:

• Retry transient model-provider failures, including HTTP 429 and 5xx responses, so answer requests can recover instead of failing immediately.

Enhancements:

• Apply the same retry behavior to server model clients and BYOK clients while preserving non-transient errors such as 400 responses.
• Honor provider retry delays with bounded exponential backoff and ensure each retry passes through the concurrency limiter.

Tests:

• Add coverage for recovery after repeated 429 responses, non-retryable 400 responses, and retry configuration.

[sourcery-ai]

Sorry @hallelx2, you've used your own review budget of 250,000 diff characters for the last 7 days.

You can request another review in 3 days and 9 hours by commenting @sourcery-ai review. Upgrade to get a review now.

[sourcery-ai]

Reviewer's Guide

The server now retries transient 429 and 5xx failures for both its configured model client and BYOK clients, using bounded exponential backoff with Retry-After support while placing retries outside the limiter; focused tests cover recovery, non-retryable 400s, and configuration.

Sequence diagram for model request retries outside the limiter

sequenceDiagram
    participant Request as /v1/answer
    participant Retry as retry.New
    participant Limiter as limit.Client
    participant Model as Model provider

    Request->>Retry: Call model client
    Retry->>Limiter: Attempt 1
    Limiter->>Model: Generate answer
    Model-->>Limiter: 429 or 5xx
    Limiter-->>Retry: Transient failure
    Retry->>Retry: Retry-After or exponential backoff
    Retry->>Limiter: Attempt 2
    Limiter->>Model: Generate answer
    Model-->>Limiter: Answer
    Limiter-->>Retry: Success
    Retry-->>Request: Answer

Loading

File-Level Changes

Change	Details	Files
Add bounded retries for transient generative-model provider failures.	Wrap the registry model client with retry middleware while preserving request timing as the outermost layer. Configure up to five retries with one-second exponential backoff capped at 20 seconds, honoring and clamping Retry-After. Keep retries outside the concurrency limiter so each attempt consumes a limiter slot. Apply the same retry wrapper to clients created through the BYOK factory. Preserve nil clients without wrapping them.	cmd/server/main.go
Verify retry behavior and production retry configuration.	Test recovery after two 429 responses and verify the resulting call count. Test that a 400 response is returned without retrying. Validate retry bounds and nil-client behavior.	cmd/server/retry_test.go

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[coderabbitai]

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 33 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

• Configuration used: defaults
• Review profile: CHILL
• Plan: Advanced
• Run ID: f7381a92-c62c-4b2d-827d-09141aef892c

📥 Commits

Reviewing files that changed from the base of the PR and between ac02c29 and ec4e9f7.

📒 Files selected for processing: 2

• cmd/server/main.go
• cmd/server/retry_test.go

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}