glm5.2: preserve input marking in prompt tokenization (fix output stalls on stray message tags)

[mp4]

Allow input tags in input

Why this is needed

This change allows models to be able to modify the llama.cpp/beellama source code. Otherwise, when a model encounters certain message tags in the files it is reading, its output stalls: stray chat-template/special tokens present in file contents (e.g. when the model reads or edits source that contains such tags) get tokenized as real special tokens and break generation. With this fix, user-provided content is marked as input and tokenized with parse_special=false, so those tags are treated as plain text and generation no longer stalls.

What changed

• common/chat.h / common/chat.cpp: thread the jinja string parts (with is_input metadata) through chat template application, with BOS/EOS stripping applied to the parts consistently; new common_tokenize_parts() tokenizes each part with input-marking awareness (user content parse_special=false, template parts parse_special=true); common_chat_params now carries prompt_parts
• common/chat-auto-parser-generator.cpp: pass the parts through the auto-parser path
• tools/server/server-common.cpp: OAI-compat params parsing serializes prompt_parts (is_input + text) into the task params
• tools/server/server-context.cpp: handle_completions_impl consumes prompt_parts with input-marking-aware tokenization when present (hunk adapted to beellama's init_opt signatures)
• common/arg.cpp.patch: carried over as-is from the original commit (reference patch for --list-devices device-id display; not applied)

Verification

Compiles cleanly for ROCm (HIP: gfx900/gfx1030/gfx1100) + Vulkan in a single build: version: 0.4.5-dev (commit 9427726d9)

[Anbeeld]

Please retarget this PR from main to v0.4.7 and rebase or refresh the implementation against that branch.

I re-audited the resulting patch against v0.4.7. It applies without textual conflicts, but the following issues still need to be addressed before merge:

1. MTMD bypasses the new protection. In handle_completions_impl, the mctx != nullptr branch runs before the prompt_parts branch. process_mtmd_prompt() tokenizes the flattened prompt with parse_special=true, so OAI chat requests served by a multimodal-capable model do not receive the intended protection, even when no media is attached.

2. Assistant continuation content is marked as template text. The continuation paths construct a string containing reasoning_content and/or message content, then append the whole string as {false, generation_prompt}. This causes request-provided continuation text to be tokenized with parse_special=true. Please preserve provenance separately for template delimiters and continuation content in the automatic and specialized parser paths.

3. GPT-OSS tokenizes different text from prompt. The GPT-OSS handler replaces the final <|return|> with <|end|> in prompt, but does not make the same replacement in prompt_parts. Since the server tokenizes prompt_parts, the existing compatibility fix is bypassed.

4. Token-count endpoints do not use prompt_parts. handle_count_tokens still tokenizes the flattened prompt with parse_special=true. Its result can differ from actual completion tokenization, and it retains the same input-special-token problem. Please share the provenance-aware tokenization path between inference and token counting.

5. Independent part tokenization can change ordinary tokenization. common_tokenize_parts() tokenizes each Jinja part separately, which prevents normal tokenizer merges across template/input boundaries. Please preserve the existing token IDs for prompts that do not contain injected special-token text, and add a regression test demonstrating that behavior.

6. Remove common/arg.cpp.patch. It is unrelated to prompt tokenization, is unapplied and stale, and contains trailing whitespace that makes git diff --check fail.

Please also add focused regression tests covering:

• special-token text inside ordinary user/tool input;
• operation with an MTMD context;
• assistant continuation content;
• the GPT-OSS <|return|> replacement;
• consistency between completion and token-count paths;
• unchanged token IDs for a normal prompt without injected special tokens.

Once the PR targets v0.4.7 and these paths are covered, I can re-review it.