Skip to content

fix: translate streaming responses to OpenAI format - #371

Merged
Priyanshu-u07 merged 1 commit into
mainfrom
fix/stream-response-transform
Oct 1, 2026
Merged

Priyanshu-u07 merged 1 commit into
mainfrom
fix/stream-response-transform

Conversation

@Priyanshu-u07

Copy link
Copy Markdown
Collaborator

call_upstream applied the adapter's transform_response; stream_upstream yielded raw chunks. A provider with its own SSE format answered a streaming request in that format, while the same request non-streaming came back translated.

Adds the streaming counterpart. transform_stream_event translates one provider event into zero or more OpenAI chunks, and finalize_stream emits anything the provider does not send itself. Both default to pass-through, so every provider already speaking OpenAI's format is untouched.

Four things the shape is deliberate about:

The hook returns a list because the mapping is not one to one. Anthropic's ping, content_block_start and content_block_stop have no OpenAI equivalent and are dropped, and Anthropic never sends the [DONE] sentinel an OpenAI client waits for, so finalize_stream supplies it.

Translator state lives in a per-stream dict rather than on the adapter. _ADAPTER_CACHE is module-level and shared by every concurrent request, so keeping a message id on self would have two simultaneous streams overwriting each other.

stream_is_openai_format gates the whole path. Without it, dropping the provider's own SSE field lines would also drop : keep-alive comments on ordinary OpenAI streams, which are what stop an idle connection timing out.

A translator that raises drops that event rather than passing it through: the client asked for OpenAI format, and handing it an untranslated event mid-stream is a chunk it cannot parse. Raising instead would discard a response the user has already half-read.

Cohere is not covered. Its /v1/chat endpoint does not appear to stream SSE, so the line framing here does not apply to it the way it does to Anthropic, and that could not be confirmed without a live key. Its hook stays the pass-through default, which is no worse than today. Tracked in #370 .

Closes #368

Signed-off-by: Priyanshu-u07 <connect.priyanshu8271@gmail.com>
@Priyanshu-u07
Priyanshu-u07 merged commit 9a75bdc into main Oct 1, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Streaming responses skip the provider response transform

1 participant