feat(server): add agent support with tool-call and reasoning parsing - #554
feat(server): add agent support with tool-call and reasoning parsing#554rubik-hua wants to merge 1 commit into
Conversation
1ae5426 to
0722a08
Compare
wooway777
left a comment
wooway777
left a comment
There was a problem hiding this comment.
在 Nvidia.40 的 /home/wuwei/InfiniLM 上检出本 PR 的 head 0722a081 后进行了验证。新增的 45 个解析器单测在隔离导入下通过，但以下问题均可在 PR 原始代码上稳定复现，建议修复后再合入。
[P1] 工具定义没有真正进入 chat template
inference_server.py:848/1039 将 chat_template_kwargs 传给 AsyncLLMEngine.add_chat_request()，但 llm.py:896-924 接收 **kwargs 后没有转发给 add_request()；后者调用 apply_chat_template() 时仍没有 tools。实测 add_chat_request(..., chat_template_kwargs={...}) 最终转发的参数中不存在 chat_template_kwargs。因此 OpenAI 和 Anthropic 请求中的工具定义只参与输出后处理，模型提示词本身看不到工具。
[P1] 流式解析器状态被并发请求共享
inference_server.py:521-532 将有状态解析器保存在 InferenceServer 实例上，_stream_chat() 又在每个新请求开始时清空这些实例（830-833）。并发请求 B 会重置请求 A 的 buffer、tool index 和 reasoning 状态。复现中，A 的前半段工具调用被 B 的 clear() 打断后，A 的后半段 "arguments":{"city":"Beijing"}} 被作为普通文本返回。解析器应按请求创建或保存在请求上下文中。
[P1] 默认安装下 Llama 流式工具参数会丢失
服务初始化 FunctionCallParser 时没有绑定 tools；流式增量虽然临时传入 tools，但 inference_server.py:957 调用的 parse_stream_end() 只检查构造时的 self.tools，因此直接返回空结果。项目也未声明 partial-json-parser 依赖，fallback 只能在 JSON 完整时解析。服务器实测结果为：
partial_json_available=false
first=[["get_weather", ""]]
end=[]
buffer_remaining=true
也就是只发出了工具名，参数仍留在 buffer。当前 single-chunk 测试只断言工具名，未校验参数。
[P1] 新增消息规范化破坏现有多模态请求
本 PR 将 _stream_chat() 和 _chat() 的原始 messages 改为 _normalize_messages(...)（inference_server.py:835/1030）。该方法把 content 数组压成纯文本并丢弃 image_url、video_url 等块，随后 resolve_multimodal_inputs() 已无法加载媒体。实测包含 text 和 image_url 的消息被转换为：
[{"role": "user", "content": "describe"}]图片信息完全丢失，这是对现有多模态接口的回归。
[P1] Anthropic SSE 内容块序列不合法
_anthropic_stream() 使用同一个 text_started 表示 text 和 thinking 块；reasoning 后的普通文本因此会把 text_delta 发到已声明为 thinking 的 index。进入工具调用时又先将 text_started=False，再在 inference_server.py:1280 计算 tool index，导致首个工具块复用 index 0。服务器复现的序列为：
starts=[[0, "thinking"], [0, "tool_use"]]
deltas=[[0, "thinking_delta"], [0, "text_delta"], [0, "input_json_delta"]]
建议维护明确的 active block type 和单调递增的 content block index。
[P2] deepseek-r1 / qwq 映射到错误的 reasoning 标签
reasoning_parser.py:195 将这些名称映射到只识别 <thinking>...</thinking> 的 ThinkTagDetector，但对应格式使用 <think>...</think>。实测 ReasoningParser("deepseek-r1") 将 <think>hidden reasoning</think>answer 整段作为 normal text 返回，reasoning 为空。应支持实际标签，或暂时移除这些 CLI 别名以避免静默错误。
验证说明
git diff --check通过。- PR 描述中的
python -m unittest test.agents.test_agents在服务器上报No module named 'test.agents'。 - 使用 discovery 直接导入完整包时，现有运行镜像与源码的 InfiniCore API 不匹配（缺少
infinicore.nn.RMSNorm）；绕过无关的模型初始化后，45 个新增解析器测试全部通过。 - 上述六项均通过执行 PR 中对应解析器/服务端方法的最小复现确认，并非仅由静态阅读推断。
| if self._tool_call_parser_instance: | ||
| self._tool_call_parser_instance.detector.clear() | ||
|
|
||
| messages = self._normalize_messages(data.get("messages", [])) |
There was a problem hiding this comment.
main分支中的有self._normalize_messages，但在inference_server.py文件中，没有被调用。应该是历史遗留函数。
这里感觉也不应该调用。
| current_text += token_output.token_text | ||
| delta_text = token_output.token_text | ||
|
|
||
| # --- Reasoning content parsing --- |
There was a problem hiding this comment.
这里一大段代码
输入是： token_output.token_text；
经过：reasoning_parser和tool_call_parser解析后，
返回值：reasoning，content，tool_calls 三者。
个人感觉可以直接封装到一个agents中的一个函数里，这一大段逻辑判断不要在这里展开。
| # Emit chunk if anything changed | ||
| if delta_reasoning or delta_content or delta_tool_calls: | ||
| chunk = json.dumps( | ||
| chunk_json( |
There was a problem hiding this comment.
上面构造了delta_tool_calls的信息，传递给了chunk_json函数。
这个delta_tool_calls应该是模型forward输出信息吧， 直接给chunk_json函数。 为什么是自己构造了一个delta_tool_calls 列表。
| }, | ||
| } | ||
| for call in end_calls | ||
| ] |
There was a problem hiding this comment.
这个信息，感觉不应该由server来构造吧。
https://docs.sglang.io/cookbook/autoregressive/Qwen/Qwen3 中的Handling Tool Call Results的例子中。
messages 信息中三类信息，都是外面提供传递给推理引擎的。
| @@ -537,17 +1068,63 @@ async def _chat(self, request_id: str, data: dict, http_request: Request): | |||
| break | |||
|
|
|||
| output_text = output_text.strip() | |||
There was a problem hiding this comment.
在vllm/entrypoints/openai/chat_completion/serving.py中定位到了代码。
if parser is not None:
reasoning, content, tool_calls = parser.parse(
output.text,
request,
enable_auto_tools=self.enable_auto_tools,
)
if not request.include_reasoning:
reasoning = None
else:
reasoning = None
content = output.text
tool_calls = []
对于_chat()函数，vllm是封装到parser对象的parse函数中。
在server.py中调用时，输入 output.text，输出 reasoning, content, tool_calls 即可。
rubik-hua
changed the title
7858643 to
33a1aab
Compare
There was a problem hiding this comment.
Re-review of head 33a1aab on .40. The six issues from the previous review now have fixes and their regression tests pass, but three blocking issues remain. [P1] python/infinilm/agents/detectors/qwen3_xml_detector.py:137,182,196 and glm4_chat_0414_detector.py:255 use the position of a function in the request tools list as OpenAI delta.tool_calls[].index. That index must be the ordinal of the call in this response. The first call to tools[1] is emitted with index 1, and two calls to tools[0] are both emitted with index 0, so clients merge or misplace calls. [P1] python/infinilm/agents/anthropic.py:38-40 accepts only text, tool_use, and tool_result blocks, while the same module emits a thinking block at line 271. Re-submitting a reasoning plus tool_use response with the subsequent tool_result therefore fails AnthropicMessagesRequest validation and the agent loop returns HTTP 400. [P2] python/infinilm/agents/reasoning_parser.py:67 treats output without a start tag as normal text even when stream_start_prefill is true. GLM-4.5 streaming explicitly assumes the template prefilled the opening marker, but the non-streaming path ignores that flag; reasoning plus the closing marker leaks into message.content. The same prefilled shape also fails for the DeepSeek/Qwen aliases. Validation: git diff --check passed; all 115 added agent tests passed after adding fastapi and uvicorn in an ephemeral container; independent repros produced Qwen indices 1 for the first call and 0,0 for repeated calls, rejected the emitted Anthropic thinking history, and returned prefilled reasoning as normal content.
Introduce a complete but minimal end-to-end agent layer into the InfiniLM
inference server, adding tool-call extraction and reasoning-content parsing
for streaming and non-streaming chat completions.
New package: infinilm.agents
-----------------------------
FunctionCallParser - Delegates to a family-specific detector.
Supports GLM-4, Llama-3.1/3.2 and Qwen3.
ReasoningParser - Splits reasoning/thinking blocks from
normal text. Supports GLM-4.5, DeepSeek-R1,
QwQ and Qwen3 thinking modes.
AgentStreamParser - Per-request stream parser turning model
output into reasoning / content /
OpenAI-format tool_calls deltas.
Glm4MoeDetector - GLM-4.5/4.6 xml-based tool-call format.
Glm4Chat0414Detector - GLM-4 / GLM-4-9B-0414 metadata-style format
(function-name line + JSON arguments line).
Llama32Detector - Llama-3.1/3.2 JSON tool-call format.
Qwen3XmlDetector - Qwen3 `<tool_call>` xml-wrapped JSON format.
Returns structural-tag metadata for future
XGrammar integration.
anthropic.py - Anthropic Messages API models,
request/response conversion and the
streaming SSE content-block converter.
message_adapter.py - Rewrites OpenAI tool history for the
GLM-4 metadata convention; explicitly
documents that Qwen3/Llama pass through
unchanged because their chat templates
natively understand standard OpenAI roles.
protocol.py - OpenAI response builders (chunk_json /
completion_json).
BaseReasoningFormatDetector - Generic streaming tag extractor, with
concrete detectors for `<thinking>` and the
` ` tag used by DeepSeek-R1 / QwQ / Qwen3.
Both tool-call detector families implement one-shot and incremental
(streaming) parsing so tool names and argument deltas can be emitted
token-by-token without buffering the entire response.
Server-side changes
-------------------
- OpenAI-compatible completions now return `tool_calls` and
`reasoning_content` in both streaming chunks and the final response.
- Tool definitions travel the full path into the prompt: OpenAI and
Anthropic requests pack `tools`/`tool_choice` into
`chat_template_kwargs`, which `AsyncLLMEngine.add_chat_request()` /
`add_request()` forward to `LLMEngine.apply_chat_template()`, so the
chat template itself sees the tools rather than only the output
post-processing.
- All agent logic lives in the agents package; inference_server.py
stays a thin HTTP layer:
* output parsing: `AgentStreamParser.process_delta()` for streams and
`parse_full_response()` for completions, one dedicated parser
instance per request so concurrent streams keep isolated buffers,
tool indices and reasoning state;
* Anthropic protocol (models, request/response conversion and the
SSE content-block state machine) in `agents/anthropic.py`, so the
`/v1/messages` route and the stream relay are a few lines each;
* OpenAI response builders (`chunk_json`/`completion_json`) moved
to `agents/protocol.py`.
- New Anthropic Messages API endpoint `POST /v1/messages` converts
requests to the internal OpenAI pipeline and re-structures the streamed
output into Anthropic SSE events (message_start,
content_block_start/delta/stop, message_delta, message_stop). Content
blocks are tracked by an explicit state machine (thinking / text /
tool_use) with monotonically increasing block indices.
- The only message adaptation is `agents.message_adapter.adapt_messages`,
scoped to the GLM-4 metadata convention; without it the GLM-4 chat
template silently drops OpenAI-style tool history and the tool loop
cannot close. Text-only content-part lists are joined into strings in
the base processor instead (agent clients commonly send several text
blocks per message); media parts remain the domain of the multimodal
processors.
- `partial-json-parser` is declared as a dependency so incomplete
tool-call arguments can be parsed incrementally mid-stream; if it is
unavailable at runtime, a `json.JSONDecoder` fallback flushes buffered
arguments once the JSON object is complete.
- Two new CLI flags are added to the inference server, each with a
`choices` list for automatic validation:
--tool-call-parser {glm,glm45,glm47,glm4,glm49b,glm4-9b-0414,
glm-4-9b-0414,llama3,llama31,llama32,
qwen3,qwen3-30b-a3b}
--reasoning-parser {glm45,glm-4,glm-4.5,glm-4.5-air,glm-4.5-flash,
think,thinking,deepseek,deepseek-r1,deepseek_r1,
qwq,qwq-32b,qwen3,qwen3-thinking}
Quick Start
-----------
Meta-Llama-3.1-8B-Instruct (tool calls + reasoning):
python -m infinilm.server.inference_server \
--model /path/to/Meta-Llama-3.1-8B-Instruct \
--tool-call-parser llama31 \
--reasoning-parser think
GLM-4-9B-0414 (tool calls only):
python -m infinilm.server.inference_server \
--model /path/to/GLM-4-9B-0414 \
--tool-call-parser glm4-9b-0414
DeepSeek-R1 / QwQ / Qwen3-Thinking (reasoning only):
python -m infinilm.server.inference_server \
--model /path/to/DeepSeek-R1 \
--reasoning-parser deepseek-r1
Qwen3-30B-A3B (tool calls + reasoning):
python -m infinilm.server.inference_server \
--model /path/to/Qwen3-30B-A3B \
--tool-call-parser qwen3 \
--reasoning-parser qwen3
Scope & Known Limitations
-------------------------
This is deliberately an MVP (Minimum Viable Product) launch to unblock
basic agent workflows while keeping the surface area small:
- Only two model families have been explicitly end-to-end validated:
Meta-Llama-3.1-8B-Instruct (--tool-call-parser llama31 --reasoning-parser think)
GLM-4-9B-0414 (--tool-call-parser glm4-9b-0414)
Other models sharing the same output formats (Llama-3.2, GLM-4.5-Air,
GLM-4.5-Flash, DeepSeek-R1, QwQ-32B, Qwen3, etc.) are expected to work
but have not yet been regression-tested.
- Constrained decoding / structural-tag support is intentionally
disabled (`get_structural_tag()` returns `None`). XGrammar or
equivalent integration will be wired in a follow-up iteration.
`Qwen3XmlDetector` already returns structural-tag metadata via
`structure_info()` so it is ready for that integration.
- Parallel tool calls stream correctly as separate tool_use blocks as
long as their argument deltas are not interleaved; true interleaved
parallel decoding is not supported yet.
- Auto-detection of the correct parser pair from `config.json` /
`generation_config.json` is not yet supported; `--tool-call-parser` and
`--reasoning-parser` must still be specified manually.
Tests
-----
CPU-only unit tests live under `test/agents/` and exercise the parsers,
detectors, chat-template tool forwarding, message normalization,
per-request parser isolation and the Anthropic SSE block sequencing:
python -m unittest discover -s test/agents
The tests run without a GPU and do not require the compiled engine
extension.
wooway777
left a comment
wooway777
left a comment
There was a problem hiding this comment.
Re-review of head 5db10f3 on .40: the three findings from the previous review remain unresolved. The tree diff from 33a1aab changes only agents/init.py, Qwen3 structure_info, and message_adapter; it does not change any failing path. [P1] qwen3_xml_detector.py:142,187,201 and glm4_chat_0414_detector.py:255 still derive OpenAI delta.tool_calls[].index from the function position in the request tools list. Fresh repro: a first call to tools[1] emits index 1, and two sequential calls to tools[0] emit 0,0, so this also fails without interleaved parallel decoding. [P1] anthropic.py:38-40 still excludes thinking from AnthropicContentBlock although line 271 emits it. Re-submitting that emitted reasoning plus tool_use content with a tool_result still raises seven AnthropicMessagesRequest validation errors, preventing the next agent turn. [P2] reasoning_parser.py:67 still classifies any output without the opening tag as normal text, ignoring stream_start_prefill. Fresh GLM-4.5 non-stream repro returns (None, reasoning plus the closing marker plus answer, []) instead of separating reasoning and content. Validation on 5db10f3: git diff --check passed and all 115 agent tests passed with fastapi, uvicorn, and partial-json-parser installed in an ephemeral container; the independent repros above still fail, which means the tests do not cover these cases.
3 participants
Introduce a minimal but end-to-end agent layer into the InfiniLM inference server, adding tool-call extraction and reasoning-content parsing for streaming and non-streaming chat completions.
New package: infinilm.agents
FunctionCallParser – delegates to a family-specific detector
ReasoningParser – splits reasoning blocks from normal text
Glm4MoeDetector – GLM-4/4.5 xml-based tool-call format
Llama32Detector – Llama-3.1/3.2 JSON tool-call format
BaseReasoningFormatDetector – generic streaming tag extractor
Both detectors implement one-shot and incremental (streaming) parsing so tool names and argument deltas can be emitted token-by-token without buffering the entire response.
Server-side changes
OpenAI-compatible completions now return
tool_callsandreasoning_contentin both streaming chunks and the final response.New Anthropic Messages API endpoint
POST /v1/messagesconverts requests to the internal OpenAI pipeline and re-structures the streamed output into Anthropic SSE events (message_start, content_block_start/delta/stop, message_delta, message_stop).LLMEngine.apply_chat_template() accepts an optional
tools=[]list and forwards it throughchat_template_kwargsso the processor can inject tool definitions into the prompt.Two new CLI flags are added to the inference server:
--tool-call-parser {glm,glm45,glm47,llama3,llama31,llama32}
--reasoning-parser {glm45,think,deepseek,deepseek-r1,qwq}
Quick Start
Meta-Llama-3.1-8B-Instruct (tool calls + reasoning):
GLM-4-9B-0414 (tool calls only):
Scope & Known Limitations
This is deliberately an MVP (Minimum Viable Product) launch to unblock basic agent workflows while keeping the surface area small:
Only two model families have been explicitly end-to-end validated:
Meta-Llama-3.1-8B-Instruct (--tool-call-parser llama31 --reasoning-parser think)
GLM-4-9B-0414 (--tool-call-parser glm)
Other models sharing the same output formats (Llama-3.2, GLM-4.5-Air, GLM-4.5-Flash, DeepSeek-R1, QwQ-32B, etc.) are expected to work but have not yet been regression-tested.
Constrained decoding / structural-tag support is intentionally disabled (
get_structural_tag()returnsNone). XGrammar or equivalent integration will be wired in a follow-up iteration.Streaming argument parsing falls back to
json.JSONDecoderwhenpartial_json_parseris unavailable; in that mode incomplete JSON mid-stream may raise until the object is fully formed.Tests
CPU-only unit tests live in
test/agents/test_agents.pyand exercise parsers, detectors, and the full server-side post-processing pipeline:Future Work
get_structural_tag()so the engine can enforce valid tool-call syntax at decode time rather than relying solely on post-hoc parsing.config.json/generation_config.jsonso most models work out-of-the-box without manual--tool-call-parserflags.<|im_start|>think,</think>variants).tool_resultblocks back to the model.agent验证，使用claude code，配置：
root@master-215:/workspace/InfiniLM# cat .claude/settings.json
{
"env": {
"ANTHROPIC_AUTH_TOKEN": "sk-dummy",
"ANTHROPIC_BASE_URL": "http://127.0.0.1:8000",
"CLAUDE_CODE_ATTRIBUTION_HEADER": "0",
"ENABLE_TOOL_SEARCH": "true",
"ANTHROPIC_MODEL": "GLM-4-9B-0414",
"ANTHROPIC_DEFAULT_HAIKU_MODEL": "GLM-4-9B-0414",
"ANTHROPIC_DEFAULT_SONNET_MODEL": "GLM-4-9B-0414",
"ANTHROPIC_DEFAULT_OPUS_MODEL": "GLM-4-9B-0414",
"CLAUDE_CODE_SUBAGENT_MODEL": "GLM-4-9B-0414"
},
"enabledPlugins": {
"clangd-lsp@claude-plugins-official": true
}
}
推理服务启动：
python python/infinilm/server/inference_server.py --device nvidia --model=/data/rubik/models/Meta-Llama-3.1-8B-Instruct --enable-paged-attn --tool-call-parser llama31 --reasoning-parser think
python python/infinilm/server/inference_server.py --device nvidia --model=/data/rubik/models/GLM-4-9B-0414 --enable-paged-attn --tool-call-parser glm
agent操作示例
不影响其它功能，用 python scripts/test_perf.py --verbose验证测试
