Repository navigation
serve: no SIGPIPE on disconnect, tokenizer EOS, producer cache flags (#214, #229, #241) - #281
Merged
Merged
Conversation
…214, #229, #241) - llmd: HTTP handler writes via MSG_NOSIGNAL so a client disconnect raises BrokenPipeError instead of hitting a vendor SIGPIPE handler (#215). - serve_batched: add the tokenizer EOS to generation_config EOS ids (#230); --pd-prefill-chunk / --pd-prefill-no-prefix-cache for the Torch producer (#242). - test that serve_stage's handler actually writes through the no-signal writer. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Background
Three serving fixes from a draft stack: a client disconnect during an HTTP response could deliver SIGPIPE to a vendor-installed handler (#214); chat checkpoints whose
generation_configkeeps a base-model EOS never stopped on the tokenizer's EOS (#229); the Torch PD producer had no flags for chunk size or prefix caching (#241).Changes
fix/214-http-sigpipe(fix: avoid native SIGPIPE handlers on HTTP disconnect #215):opt_prime/opt_prime/accel/serve/llmd.py(_NoSignalSocketWriter,sendall(..., MSG_NOSIGNAL), installed in theserve_stagehandler'ssetup()),opt_prime/tests/test_http_sigpipe.py.feat/241-pd-producer-cache(fix: honor tokenizer turn-end EOS in batched serving #230, feat: expose Torch PD producer cache and chunk settings #242):opt_prime/examples/pd/serve_batched.py(tokenizer EOS merged into the EOS ids;--pd-prefill-chunk,--pd-prefill-no-prefix-cache),opt_prime/tests/test_serving_eos_factory.py,opt_prime/tests/test_serving_prefill_cache.py. Diffed againstoptimus-rmfirst: the file change is additive only, so nothing onoptimus-rmis dropped.test_http_sigpipe.py:serve_stage's handler really writes through_NoSignalSocketWriter(GET /health through a live server).Verification
pytest -q tests/test_http_sigpipe.py tests/test_serving_eos_factory.py tests/test_serving_prefill_cache.py: 13 passed.setup()override removed, and the whole file fails to import againstoptimus-rm'sllmd.py.Compatibility and limits
MSG_NOSIGNALthe flag is 0 (previous behaviour); the signal test is skipped there.BaseHTTPRequestHandler's defaultwfilealready is.--pd-prefill-device(Torch producer); defaults match previous behaviour (chunk 2048, prefix cache on).Related Issue
Closes #214
Closes #229
Closes #241
Supersedes #215, #230, #242.
🤖 Generated with Claude Code