Skip to content

fix: detect unresponsive runtime connections and expose transport failures - #64

Merged
sam2tom merged 2 commits into
betafrom
codex/fix-runtime-disconnect-beta
Oct 29, 2026
Merged

sam2tom merged 2 commits into
betafrom
codex/fix-runtime-disconnect-beta

Conversation

@sam2tom

@sam2tom sam2tom commented Oct 29, 2026 •

Copy link
Copy Markdown

Problem and behavior

A runtime connection can stop delivering events to Core while the daemon continues executing behind a half-open proxy connection. Core's underlying read/write errors were sent through an unset optional logging callback, leaving only read_ended in diagnostics.

Record structured transport failure classes and WebSocket close codes without peer-controlled text. Probe daemon WebSocket connections every 15 seconds and require inbound traffic or Pong within 60 seconds. Bound blocked inbound dispatch as well, so disconnect cleanup can stop the active Turn and retain ownership until Executor cleanup is confirmed. Healthy connections renew their deadline; this introduces no total Turn duration limit.

A Turn that fails only because its Runtime connection closed (event_stream_incomplete with connection_closed) is now eligible for automatic Turn retry under the existing OAC_TURN_AUTO_RETRIES policy, fingerprint runtime_disconnected: the same limit, backoff, cancellation check and idempotency key apply. Event window overflow and other subscription failures remain non-retryable. In production all five failed Turns in the last seven days were this class.

This PR targets beta, the deployed production baseline. It does not change ingress timeouts.

Validation

  • Transport, runtime gateway and execution tests passed with the race detector, including retry decisions for each event stream failure reason.
  • CLI, dispatch, wire conformance, execution and retry regression tests passed.
  • A real WebSocket fixture using the production 60-second deadline verified active Turn cancellation and confirmed owner cleanup before reconnect.
  • Healthy idle connections and blocked receive queues are covered.
  • Bilingual documentation checks and git diff whitespace checks passed.
  • Fresh independent reviews completed with no remaining implementation blockers.

Limits

This fixes diagnostic loss and bounds unresponsive WebSocket cleanup; it does not establish the initiating cause of the historical Core disconnect. Pong establishes WebSocket peer liveness, so a proxy that independently replies to probes can mask a backend failure. No live four-hour run or production rollout has been performed.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-29T13:20:47.362239Z ac5dd38 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@sam2tom
sam2tom merged commit 0d44677 into beta Oct 29, 2026
42 of 44 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant