[rita-aga]

Why

On 2026-10-23, TemperPaw S3 get calls were slow but still succeeding, and every Temper blob-transport monitor flapped all day. Some File $value reads failed outright.

Root cause

Every BlobStore::s3 built its own reqwest::Client, so each get paid for a new TCP and TLS handshake. That client had only a 10s connect timeout, no connection pool and no retry.

• Successful S3 get spans were idle for almost their whole duration: busy time was 0–8ms, and span p95 was about 1.9s. Semaphore wait stayed around 5ms p95, so callers weren't queueing.
• 43 gets were cancelled at about 7.9s while their socket was still idle. They never recorded an outcome, so the File response failed instead of returning the object.

Change

• All S3 calls now share one process-wide client: 2s connect timeout, 45s idle pool with up to 32 idle connections per host, and TCP and HTTP/2 keepalives.
• A buffered GET gets two tries inside a 7s budget, 3s each. It retries only on a timeout, a connect failure or a dropped body. HTTP statuses, including 404 and 5xx, are never retried.
• Puts and streaming reads use the same client and are not retried.
• The decision is recorded as ADR-0182.

Tests

cargo test -p temper-server --lib blob_store::: 18 passed. Two of them are new:

• a stalled get is abandoned and the retry returns the object within budget;
• a get that answers in time succeeds once and is not retried.

Note

The five Katagami read_app 502s between 6:34 and 6:49 PM ET are a separate failure. They come from fuel_exhausted in wasm.invoke.run during git-upload-pack, not from blob transport, and this PR doesn't fix them.

Confidence Score: 4/5

Fix the timeout for healthy, longer downloads before merging.

Findings

1.  Healthy downloads time out ▶
2.  Connection reuse is untested ▶

Fix with agent prompt

### Issue 1
crates/temper-server/src/blob_store/s3_transport.rs:156
`read_object_once` applies the three-second timeout to the entire download, not just a stalled connection. File `$value` reads use this buffered path. A healthy transfer of 20 MB at 5 MB/s is therefore cancelled on both attempts, failing a read that previously had a five-minute budget. Bound connection or idle waits separately, or preserve a longer budget for downloading the body.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

### Issue 2
crates/temper-server/src/blob_store/s3_transport.rs:65-67
Both new tests use `S3Http::for_test`, which creates a separate client and disables connection pooling. They can stay green if production stores start creating separate clients again, bringing back the repeated handshakes this PR fixes. Add a test that reads through two separately constructed `BlobStore::s3` instances at the same endpoint and checks that they reuse one TCP connection.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

Shares one S3 HTTP client across blob stores and adds a bounded retry for buffered GETs.

• S3 blob operations reuse one shared HTTP client.
• Buffered S3 reads retry a stalled request within a short budget.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Buffered blob GET] --> B[Shared S3 client]
  B --> C[Attempt: three seconds including body]
  C --> D{Result}
  D -->|Success or 404| E[Return result]
  D -->|Other HTTP status| F[Return error]
  D -->|Retryable transport error| G{Attempt remaining?}
  G -->|Yes, within budget| C
  G -->|No| F

Loading

_{Reviews (1) · Last reviewed commit: "fix(server): reuse the S3 client and ret..." · Reviewed by Greptile}

[greptile-apps]

Healthy downloads time out

read_object_once applies the three-second timeout to the entire download, not just a stalled connection. File $value reads use this buffered path. A healthy transfer of 20 MB at 5 MB/s is therefore cancelled on both attempts, failing a read that previously had a five-minute budget. Bound connection or idle waits separately, or preserve a longer budget for downloading the body.

Prompt To Fix With AI

This is a comment left during a code review.
Path: crates/temper-server/src/blob_store/s3_transport.rs
Line: 156

Comment:
**Healthy downloads time out**

`read_object_once` applies the three-second timeout to the entire download, not just a stalled connection. File `$value` reads use this buffered path. A healthy transfer of 20 MB at 5 MB/s is therefore cancelled on both attempts, failing a read that previously had a five-minute budget. Bound connection or idle waits separately, or preserve a longer budget for downloading the body.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[greptile-apps]

Connection reuse is untested

Both new tests use S3Http::for_test, which creates a separate client and disables connection pooling. They can stay green if production stores start creating separate clients again, bringing back the repeated handshakes this PR fixes. Add a test that reads through two separately constructed BlobStore::s3 instances at the same endpoint and checks that they reuse one TCP connection.

Prompt To Fix With AI

This is a comment left during a code review.
Path: crates/temper-server/src/blob_store/s3_transport.rs
Line: 65-67

Comment:
**Connection reuse is untested**

Both new tests use `S3Http::for_test`, which creates a separate client and disables connection pooling. They can stay green if production stores start creating separate clients again, bringing back the repeated handshakes this PR fixes. Add a test that reads through two separately constructed `BlobStore::s3` instances at the same endpoint and checks that they reuse one TCP connection.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.