Conversation
A Responses API stream cut short, for example by max_output_tokens, ends on response.incomplete instead of response.completed. The stream state only read usage and output from response.completed, so these generations were captured with no token counts and empty output even though the stop reason was set. Read them from every terminal event (completed, incomplete, failed), as the stop reason already does and as posthog-js does.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
💡 Motivation and Context
A Responses API stream that is cut short, for example by
max_output_tokens, ends onresponse.incompleteinstead ofresponse.completed._ResponsesStreamStatealready reads the stop reason from any terminal event (#919), butextract_openai_usage_from_chunkandextract_openai_content_from_chunkonly readresponse.completed.As a result, these generations are captured with
$ai_stop_reason: max_output_tokensbut with no$ai_input_tokens/$ai_output_tokensand an empty$ai_output_choices, even though the tokens were billed and the partial output was streamed. Cost for truncated runs is missing from LLM analytics.posthog-js's
OpenAIResponsesStreamAccumulatortakes usage and output from every terminal response. This PR does the same: usage and output are read fromresponse.completed,response.incompleteandresponse.failed. A stream ends with exactly one of these, so nothing is counted twice.💚 How did you test it?
test_streaming_responses_api_captures_usage_and_output_when_incompletestreams a realResponseIncompleteEvent(statusincomplete,incomplete_details.reason = max_output_tokens, usage 20/16, a partial output message) through the wrappedOpenAIclient.main(4a138e6) it fails:KeyError: '$ai_input_tokens'.pytest posthog4144 passed, 16 skipped.ruff format --check,ruff check, mypy throughmypy-baseline filterandpython -W error -c "import posthog"are clean.📝 Checklist
If releasing new changes
sampo addto generate a changeset file🤖 Agent context
Autonomy: Fully autonomous
mainbefore the fix.response.failedis included for parity with posthog-js and with the stop-reason handling. A failed response usually has no usage, and that case is already handled.Commits here are unsigned; this repo squash-merges, so GitHub signs the merged commit.