Repository navigation
fix: keep a PD stage accepting work after a refused stop (#174) - #183
Merged
Merged
Conversation
submit("stop") clears `accepting` before the owner checks quiescence. A
stop is refused (409) only on a non-quiescent stage, e.g. a poisoned one,
and afterwards /v1/admin/reset and a second stop were rejected with 503
"owner not accepting work" while the owner thread kept running. The owner
now restores `accepting` when the stop fails.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Background
OwnerExecutor.submit("stop")clearsacceptingbefore the owner thread checks quiescence. The owner refuses a stop (409 "owner not quiescent") when the stage is not idle/ready, e.g. poisoned, and after that/v1/admin/resetand a second stop were rejected with 503 "owner not accepting work" while the owner thread kept running. The stage could then neither recover nor shut down (#174: stop -> 409, reset -> 503, stop -> 503, thread still alive).Changes
opt_prime/opt_prime/accel/serve/llmd.py(OwnerExecutor.run, where the result is delivered): when a stop fails, the owner setsaccepting = Trueagain before failing the future, so reset and a retried stop are admitted. A successful stop is unchanged.opt_prime/tests/test_pd_recovery.py:test_refused_stop_still_allows_reset_and_stoppoisons a stage (fatal transfer error), checks stop -> 409,/v1/admin/resetover HTTP -> 200{"state": "idle"}, then a second stop succeeds and the owner thread exits.Verification
CPU plus one A100 (
CUDA_VISIBLE_DEVICES=1), Python 3.10, torch 2.10.0+cu128, transformers 5.1.0, base optimus-rm @253be20. The tests bind HTTP on port 0 only.TORCH_DEVICE_BACKEND_AUTOLOAD=0 CUDA_VISIBLE_DEVICES=1 PYTHONPATH=$PWD python3 -m pytest -q tests/test_pd_recovery.py -k refused_stop-> 1 failed: reset returned503(At index 0 diff: 503 != 200).pytest -q tests/test_pd_recovery.py tests/test_pd_llmd.py tests/test_pd_http_cancel.py tests/test_pd_metrics.py tests/test_pd_stream.py-> 25 passed.opt_prime/:TORCH_DEVICE_BACKEND_AUTOLOAD=0 CUDA_VISIBLE_DEVICES=1 PYTHONPATH=$PWD python3 -m pytest -q tests: before (@253be20, without the new test): 400 passed, 23 skipped, 135 subtests passed; after: 401 passed, 23 skipped, 135 subtests passed.stop -> 409 owner not quiescent,reset -> {'state': 'idle'},stop -> None, owner thread exits (was 409 / 503 / 503, thread alive).Compatibility and limits
_closeintest_pd_recovery.pystill forcesstarted/acceptingitself; it is left as it is.Related Issue
Closes #174
🤖 Generated with Claude Code