Repository navigation
fix: honor tokenizer turn-end EOS in batched serving - #230
Closed
gspark-etri wants to merge 2 commits into
Closed
gspark-etri wants to merge 2 commits into
gspark-etri wants to merge 2 commits into
Conversation
This was referenced Oct 23, 2026
gspark-etri
added a commit
that referenced
this pull request
Oct 30, 2026
…214, #229, #241) (#281) - llmd: HTTP handler writes via MSG_NOSIGNAL so a client disconnect raises BrokenPipeError instead of hitting a vendor SIGPIPE handler (#215). - serve_batched: add the tokenizer EOS to generation_config EOS ids (#230); --pd-prefill-chunk / --pd-prefill-no-prefix-cache for the Torch producer (#242). - test that serve_stage's handler actually writes through the no-signal writer. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Collaborator
Author
|
Landed on optimus-rm via #281. Closing; the branch is archived locally (experiment records and evidence included). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Background
KONI's generation_config declares EOS128001, while its tokenizer uses128009 (<|eot_id|>) as chat turn end. serve_batched used only the former, so a completed answer could continue into synthetic dialogue until max_tokens.
Changes
Normalize configured EOS IDs and include tokenizer EOS without duplicates in the common factory, before any backend's requests reach BatchedService. Explicit ignore_eos is unchanged. Add six factory/request regression cases and a real KONI HTTP negative-control/streaming/reuse checker with recorded evidence.
Verification
Base3ea0493; fix e5fa087, GPU/ATOM evidence f84823a. Factory regressions:3failed/3passed before,22related CPU tests pass after (factory, sampling, streaming, batched serving). New files Ruff-clean; diff check passes. Changed-file sensitive scan clean; full branch retains4 unchanged baseline findings.
Real A10080GB BF16 B1/slot cache/context2048 and ATOM B1/context9216/k1 existing artifact, KONI revision0172bf7e73977ae213613529f3d2d5400a33f678. Exact capital chat IDs committed. Production factory/owner/scheduler/HTTP server, one loaded model, max_tokens8:
Initial validation mistakenly read a final usage-only SSE event as a choice; corrected checker passed all five cases, then the final checked-in script was rerun successfully. Tests and exact reproduction command are in opt_prime/examples/pd/http_matrix/experiments/2026-10-23-serving-eos229.md. All owned HTTP threads/processes stopped.
Compatibility and limits
No weights, tokenizer files or SDK changes. Common factory behavior applies structurally to all backend branches; GPU and ATOM HTTP are verified on KONI. RNGD native HTTP and other-model hardware revalidation are not claimed. A tokenizer EOS now ends default generation even when absent from generation_config; ignore_eos remains available for fixed-length generation. This does not establish general model quality or heterogeneous PD performance.
Related Issue
Closes #229. Related #227/#201. Stacked on #216 (exp/211-atom-http).
Generated with Codex; maintainer review and merge required.