[gspark-etri]

Background

ProcessPrefillWorker's receiver thread blocked on _rep.get() and never checked the child. If the prefill process was OOM-killed or segfaulted, its futures never resolved, lanes stayed in prefilling, and later submits hung too. SubprocessPrefillWorker._receive already fails outstanding futures when its child exits.

Changes

opt_prime/opt_prime/accel/pd/pd_batching.py (ProcessPrefillWorker):

• _receive polls the reply queue with a 0.5 s timeout. When the queue is empty and the child is no longer alive, it fails every outstanding future with RuntimeError("prefill process exited (<exitcode>)"), sets _futures = None and returns. Replies already queued are drained first.
• submit raises RuntimeError("prefill process exited") once _futures is None, like a submit to a dead SubprocessPrefillWorker (BrokenPipeError).

opt_prime/tests/test_pd_batching_disagg.py::test_process_worker_child_death_fails_outstanding_prefills: the factory is functools.partial(time.sleep, 600), so the child never serves. The test submits a request, kills the child, and checks that f.exception(timeout=5) is a RuntimeError and that a later submit raises. The timeout keeps the suite from hanging on the unfixed code.

Verification

Python 3.10, torch 2.10.0+cu128, transformers 5.1.0, CPU plus one A100 (CUDA_VISIBLE_DEVICES=1) on the RNGD+A100 host. Base: optimus-rm @253be20.

• New test on the unfixed code: FAILED, concurrent.futures.TimeoutError after 5 s (future still pending).
• After the fix: tests/test_pd_batching_disagg.py gave 12 passed in 18 s. Futures fail within about 0.5 s of the child's death.
• Whole suite (From opt_prime/: TORCH_DEVICE_BACKEND_AUTOLOAD=0 CUDA_VISIBLE_DEVICES=1 PYTHONPATH=$PWD python3 -m pytest -q tests): 400 passed, 23 skipped before; 401 passed, 23 skipped after.

Compatibility and limits

• On a clean close() nothing changes. The receiver still exits on the None sentinel.
• The worker does not restart the child. After a death the worker stays failed, and the batcher fails each affected request through its normal error path. Readiness/health reporting that depends on the worker (mentioned in the issue) is not changed here.

Related Issue

Closes #168

🤖 Generated with Claude Code