fix: translate streaming responses to OpenAI format - #371
Merged
Merged
Conversation
Signed-off-by: Priyanshu-u07 <connect.priyanshu8271@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
call_upstreamapplied the adapter'stransform_response;stream_upstreamyielded raw chunks. A provider with its own SSE format answered a streaming request in that format, while the same request non-streaming came back translated.Adds the streaming counterpart.
transform_stream_eventtranslates one provider event into zero or more OpenAI chunks, andfinalize_streamemits anything the provider does not send itself. Both default to pass-through, so every provider already speaking OpenAI's format is untouched.Four things the shape is deliberate about:
The hook returns a list because the mapping is not one to one. Anthropic's
ping,content_block_startandcontent_block_stophave no OpenAI equivalent and are dropped, and Anthropic never sends the[DONE]sentinel an OpenAI client waits for, sofinalize_streamsupplies it.Translator state lives in a per-stream dict rather than on the adapter.
_ADAPTER_CACHEis module-level and shared by every concurrent request, so keeping a message id onselfwould have two simultaneous streams overwriting each other.stream_is_openai_formatgates the whole path. Without it, dropping the provider's own SSE field lines would also drop: keep-alivecomments on ordinary OpenAI streams, which are what stop an idle connection timing out.A translator that raises drops that event rather than passing it through: the client asked for OpenAI format, and handing it an untranslated event mid-stream is a chunk it cannot parse. Raising instead would discard a response the user has already half-read.
Cohere is not covered. Its
/v1/chatendpoint does not appear to stream SSE, so the line framing here does not apply to it the way it does to Anthropic, and that could not be confirmed without a live key. Its hook stays the pass-through default, which is no worse than today. Tracked in #370 .Closes #368