Review view: tool results as MCP content (frozen base 348231c) - #488
Open
rejojer wants to merge 3 commits into
Open
Review view: tool results as MCP content (frozen base 348231c)#488rejojer wants to merge 3 commits into
rejojer wants to merge 3 commits into
Conversation
…er it get_document_image returns an MCP image block, and the SDK flattened every tool result to text on its way to a framework, so an own-model chat received "[image/png content omitted: ~256 KB]" where the hosted lanes (HostedMCPTool, the http MCP config, the managed chat) received the page. The rule now: the SDK carries MCP types and renders nothing. Each framework gets the tool set the way it consumes MCP, and converts the content itself. - McpBridge.call_tool returns the content blocks untouched; the invoke contract behind _tool_specs is (content blocks, is_error) on the cloud and local paths alike (local tools send one text block). - openai-agents: an in-process MCPServer over the tool specs, and the FunctionTools are MCPUtil.to_function_tool over it. The framework's own MCP conversion builds the tools (schema verbatim, strict off) and renders results (text items, image data URLs); its failure pipeline answers malformed arguments. The hand-built FunctionTools are gone. - Anthropic tool runner: results validated as CallToolResult and rendered by the Anthropic SDK's mcp_content (text blocks, base64 image blocks), on the error channel too. - Claude Agent SDK: the content passes through, it is MCP already. - Plain functions (agent_tools): render_text, the former flattening, keeps the size stub for binary content; a string cannot carry an image. Wire consequences: tool results on the openai-agents lanes are the framework's structured items (a single text item for a text result) rather than a bare string, and the Anthropic tool_result content is a block list. ChatStream tool_result events carry that structured output; the woven display shows its text. mcp becomes a declared dependency: it already arrives with openai-agents, and both adapters use its types. Verified live on a 15-page paper, page 1, across chat() on the LiteLLM lane (gpt-5.6-luna), chat(protocol="responses"), and chat(protocol="messages", claude-sonnet-4-6): each answer described the red attribution text and the arXiv stamp rotated along the left margin. Claude-Session: https://claude.ai/code/session_01F8QMbFKngRT4TpBAKfuNVW
…ute names CI installs mcp 2.x, where CallToolResult exposes is_error and content blocks exposes mime_type as attributes, with the wire names (isError, mimeType) as aliases; mcp 1.x uses the wire names as attributes. The adapters only ever validate from and dump to the wire shape, so they run on both; the test now does the same. Claude-Session: https://claude.ai/code/session_01F8QMbFKngRT4TpBAKfuNVW
The woven [tool_result] line clipped the framework's image item to its
first 200 characters, which is the head of a data URL. The line now shows
the item as it is with the payload elided: {"type": "image", "image_url":
"data:image/png;base64,..."}. The item itself is unchanged.
Claude-Session: https://claude.ai/code/session_01F8QMbFKngRT4TpBAKfuNVW
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Review-only view of #487 (tool results reach the frameworks as MCP content). Base frozen at
348231c(review/base-348231c, never advances); never merge this PR.Later review rounds run here. Fixes push to
feat/multimodal-tool-resultsand reachmainthrough their own small PRs.https://claude.ai/code/session_01F8QMbFKngRT4TpBAKfuNVW