AI-gateway calls from mainnet-alpha enclaves 502 at Cloudflare edge (asia-southeast1); same calls succeed from outside

[mmurrs]

AI-gateway calls from inside mainnet-alpha enclaves fail with 502 at the Cloudflare edge; same calls succeed from outside

Summary

Every /v1/chat/completions and /v1/models request from inside a mainnet-alpha TEE to https://ai-gateway.eigencloud.xyz returns HTTP 502 with a Cloudflare HTML error page. The identical requests from outside the enclave return clean JSON (401 for bad auth — i.e., they reach the API). The attestation JWT mints correctly in-enclave (auth is not the problem), all models fail identically, and the failure has persisted across freshly provisioned instances on 2026-07-09 and 2026-07-10 — so unlike the TPM issue (#169), this is not version skew.

New discriminating datapoint: the enclave runs in asia-southeast1-a (from its own attestation token), so its requests hit Cloudflare's Singapore edge; all my successful outside tests hit US edges (cf-ray …-EWR). A regional edge/WAF/origin-pool difference would explain the asymmetry exactly.

Ask: check Cloudflare edge logs / WAF events for requests to ai-gateway.eigencloud.xyz originating from the TEE egress IPs of project tee-compute-mainnet-prod (zone asia-southeast1-a), and whether the SIN edge has a different origin pool / security rule than US edges.

Evidence

Failing (from inside the enclave — app 0xF174BC083D3FDE2a9bEae3f34FC31791fb2ca5aE):

• Client: plain fetch from Node 22, POST https://ai-gateway.eigencloud.xyz/v1/chat/completions, bearer = KMS-attestation JWT (mints successfully; same KMS config passes attestation per Runtime attestation with extraData fails on mainnet-alpha: TPM nonce mismatch (KMS likely pre-eigenx-kms#17) #169 update).
• Result, every call, every model tried (anthropic/claude-sonnet-4.6, claude-haiku-4.5, claude-sonnet-4.5, openai/gpt-4o, gpt-4o-mini): HTTP 502, HTML body with Cloudflare fingerprint (<!--[if lt IE 7]> conditional-comment error page).
• GET /v1/models with the same JWT: same 502/HTML.
• Counters on the live app right now: 64/64 calls failed, zero successes ever, across 3 instance generations (7/9 ×2, 7/10 ×1).

Working (from outside, US location):

• Same endpoints, same request shape, deliberately bad bearers of 100 B → 64 KB: clean JSON 401 invalid token … every time — request traverses Cloudflare AND the Google LB and reaches the API.
• Fingerprint of the stack from response headers: server: cloudflare, via: 1.1 google, cf-ray: …-EWR.

Also true from inside the enclave (rules out general egress breakage):

• KMS calls succeed (env fetch at boot, attestation minting at runtime).
• Docker Hub pulls succeed at provisioning.

Hypotheses, ranked

1. Regional (SIN) edge difference — origin pool, WAF rule, or security level that differs from US edges. Discriminator: any successful request in CF logs from SIN to this hostname.
2. Cloudflare security rule matching the TEE egress (GCP ASN / IP reputation / bot rules) — though a challenge usually presents as 403, custom rules can 502.
3. Egress-proxy request mangling (SNI/Host/HTTP-version quirk on the TEE egress path) causing CF→origin failure only for enclave-originated connections.

Machine-readable references

error            : HTTP 502, HTML body (Cloudflare error page, "[if lt IE 7]" fingerprint)
endpoint         : https://ai-gateway.eigencloud.xyz/v1/chat/completions ; /v1/models
app_id           : 0xF174BC083D3FDE2a9bEae3f34FC31791fb2ca5aE (mainnet-alpha)
enclave_zone     : asia-southeast1-a  (project tee-compute-mainnet-prod, from attestation JWT submods.gce)
works_from       : outside enclave (US) — JSON 401s for any bearer, cf-ray EWR
fails_from       : inside enclave — all models, all calls, 3 instance generations (2026-07-09, 2026-07-10)
auth_ok          : KMS attestation JWT mints in-enclave (see #169 resolution comment)
egress_ok        : KMS + Docker Hub reachable from same enclave
related          : #169 (TPM nonce — resolved by rollout; this issue did NOT heal the same way)