fix(server): hide qwen transport markers from semantic responses - #266
fix(server): hide qwen transport markers from semantic responses#266HaileyStorm wants to merge 1 commit into
Conversation
…arkers from semantic responses Upstream FlashML-org#266 at 90be981, merged onto deploy/chatdnp for the PR sweep. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0173pf9k9fSVtwbm3f898HDt
…gprobs for chat and legacy completions Upstream FlashML-org#224 at 855650d, merged onto deploy/chatdnp for the PR sweep. Conflicts: engine.py keeps FlashML-org#231's stats readout before the logprobs-aware return; openai_api.py keeps the vision `images` argument and FlashML-org#222's disconnect-watching drain with the logprobs entries added; generation.py keeps FlashML-org#266's marker filter and routes every content delta through FlashML-org#224's _content_delta so the logprobs entries ride the filtered text. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0173pf9k9fSVtwbm3f898HDt
|
Tried on 2 x RTX 6000 Ada (sm_89) serving Qwen3.8-Flash-Next (RadixArk NVFP4) at TP=2, offload backend, fp8 KV pool of 8 x 262,144 tokens, merged onto my deploy branch (main af71ba4 + #385/#386/#389/#392/#354 and ten other open PRs), tests run on the box, then put in production. The symptom is real on this deployment: asked to repeat One thing outside this PR's scope that the same probe showed: the tokenizer parses control tokens typed by a user in a message (a |
Summary
/generate, legacy completions, non-Qwen semantic output, and offline token IDs unchangedCloses #265.
Why this layer
The Qwen detokenizer must keep protocol tags visible long enough for reasoning and tool parsers to consume them. Filtering in the tokenizer would also alter raw-generation and offline compatibility surfaces.
generation.pyis the shared semantic waist used by OpenAI chat, Anthropic Messages, and Responses, so the cleanup happens once after parsing and before those adapters format their wires.The filter is request-local. It holds only an ambiguous marker prefix or the suffix of a quote after a complete hidden marker. Structured tool events encountered during that ambiguity are retained in the same sequence, preventing both marker leaks and content/tool reordering. An unmatched quote is sanitized at end-of-stream; a matching quote proves the marker spelling was deliberate literal content.
Validation
Original symptom: observed with a local Qwen3.8 Flash-Next derivative on an NVIDIA RTX 5090, then minimized to deterministic CPU-only server tests. This PR makes no performance claim and does not require model weights or GPU execution.
Ubuntu Linux, GPU visibility explicitly disabled:
Result:
198 passed, 1 warning in 2.89s. The warning is the existing FastAPI/Starlette TestClient deprecation warning.Also passed:
py_compilefor both changed Python filesgit diff --check origin/main...HEADCompatibility tests
The regression suite explicitly covers:
/generate, legacy completion output, and offline token IDs