Skip to content

fix(server): hide qwen transport markers from semantic responses - #266

Open
HaileyStorm wants to merge 1 commit into
FlashML-org:mainfrom
HaileyStorm:codex/qwen-semantic-marker-filter
Open

fix(server): hide qwen transport markers from semantic responses#266
HaileyStorm wants to merge 1 commit into
FlashML-org:mainfrom
HaileyStorm:codex/qwen-semantic-marker-filter

Conversation

@HaileyStorm

Copy link
Copy Markdown

Summary

  • suppress Qwen transport and internal multimodal markers at the protocol-neutral semantic generation boundary
  • run suppression only after reasoning and tool parsing, keeping parser tags, grounding markers, FIM markers, and tool arguments intact
  • preserve matching quoted/code literals while making marker matching chunk-safe and event-order-safe across streaming tool calls
  • leave raw /generate, legacy completions, non-Qwen semantic output, and offline token IDs unchanged

Closes #265.

Why this layer

The Qwen detokenizer must keep protocol tags visible long enough for reasoning and tool parsers to consume them. Filtering in the tokenizer would also alter raw-generation and offline compatibility surfaces. generation.py is the shared semantic waist used by OpenAI chat, Anthropic Messages, and Responses, so the cleanup happens once after parsing and before those adapters format their wires.

The filter is request-local. It holds only an ambiguous marker prefix or the suffix of a quote after a complete hidden marker. Structured tool events encountered during that ambiguity are retained in the same sequence, preventing both marker leaks and content/tool reordering. An unmatched quote is sanitized at end-of-stream; a matching quote proves the marker spelling was deliberate literal content.

Validation

Original symptom: observed with a local Qwen3.8 Flash-Next derivative on an NVIDIA RTX 5090, then minimized to deterministic CPU-only server tests. This PR makes no performance claim and does not require model weights or GPU execution.

Ubuntu Linux, GPU visibility explicitly disabled:

PYTHONDONTWRITEBYTECODE=1 PYTHONPATH=python CUDA_VISIBLE_DEVICES='' HIP_VISIBLE_DEVICES='' \
/home/hailey/AI/llm/src/freetoken-flash-next-9ef36513/.venv/bin/python -m pytest -q -p no:cacheprovider \
tests/server/test_qwen_semantic_special_tokens.py \
tests/server/test_streaming_model_matrix.py \
tests/server/test_generation_accounting.py \
tests/server/test_openai_api.py

Result: 198 passed, 1 warning in 2.89s. The warning is the existing FastAPI/Starlette TestClient deprecation warning.

Also passed:

  • py_compile for both changed Python files
  • git diff --check origin/main...HEAD
  • independent closure review, including 317,449 bounded text/chunk/event probes

Compatibility tests

The regression suite explicitly covers:

  • all hidden markers split at every character boundary
  • stream/buffer parity for reasoning, content, and structured tool calls
  • overlapping and disproved marker prefixes without event reordering
  • closed, escaped, typographic, and unterminated quotes
  • preserved grounding and FIM markers
  • non-Qwen semantic output
  • raw /generate, legacy completion output, and offline token IDs

gdevenyi added a commit to gdevenyi/FreeToken that referenced this pull request Sep 5, 2026
…arkers from semantic responses

Upstream FlashML-org#266 at 90be981, merged onto deploy/chatdnp for the PR sweep.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0173pf9k9fSVtwbm3f898HDt
gdevenyi added a commit to gdevenyi/FreeToken that referenced this pull request Sep 5, 2026
…gprobs for chat and legacy completions

Upstream FlashML-org#224 at 855650d, merged onto deploy/chatdnp for the PR sweep.
Conflicts: engine.py keeps FlashML-org#231's stats readout before the logprobs-aware return;
openai_api.py keeps the vision `images` argument and FlashML-org#222's disconnect-watching drain
with the logprobs entries added; generation.py keeps FlashML-org#266's marker filter and routes every
content delta through FlashML-org#224's _content_delta so the logprobs entries ride the filtered text.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0173pf9k9fSVtwbm3f898HDt
@gdevenyi

gdevenyi commented Sep 5, 2026

Copy link
Copy Markdown

Tried on 2 x RTX 6000 Ada (sm_89) serving Qwen3.8-Flash-Next (RadixArk NVFP4) at TP=2, offload backend, fp8 KV pool of 8 x 262,144 tokens, merged onto my deploy branch (main af71ba4 + #385/#386/#389/#392/#354 and ten other open PRs), tests run on the box, then put in production.

The symptom is real on this deployment: asked to repeat <|im_start|>user hello<|im_end|>, the chat content came back with the <|im_start|> marker in it; with this PR the content is user hello. Greedy outputs on three unrelated prompts are word-identical to the unfiltered build, and the 8-question probe is unchanged. Merged clean, in production.

One thing outside this PR's scope that the same probe showed: the tokenizer parses control tokens typed by a user in a message (a <|vision_pad|> in user text vanished from the prompt), so a typed <|im_end|> ends the turn early. Input side, not this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Qwen semantic responses can leak transport special-token markers

2 participants