Skip to content

test: exercise four-model live PD cancellation and worker faults - #245

Draft
gspark-etri wants to merge 1 commit into
feat/234-live-model-pdfrom
exp/244-live-pd-faults
Draft

gspark-etri wants to merge 1 commit into
feat/234-live-model-pdfrom
exp/244-live-pd-faults

Conversation

@gspark-etri

Copy link
Copy Markdown
Collaborator

Background

Post-first-token cancellation and CPU subprocess fault tests did not establish recovery while an actual RNGD prefill call was executing for an ATOM HTTP request.

Changes

Add an opt-in hardware fault mode to the live HTTP validator. A transparent test-only factory writes atomic native-call start/done markers; injection requires a matching unfinished Future. Exercise pending cancellation, owned worker termination with a queued request, later-request failure, slab unlink and explicit replacement at an idle scheduler boundary. Preserve failed-run evidence and report remote admission attempts separately from successful outputs.

Verification

  • Four actual RNGD->ATOM models: Qwen3-0.6B,Qwen3-4B,Gemma3-1B,KONI8B; medium1526–1643token fixtures,width2048,ATOM B1/context9216 (k3 except KONIk1). Existing artifacts; no device-wide reset or other workload touched.
  • All four pass pending cancellation499/reuse, two500responses at worker death plus one later500, slab unlink and replacement native-output equality.64/64normal output checks,88remote admission attempts,0local fallback.
  • Error-response/cleanup interval: Qwen0.6B0.4125s,Qwen4B1.3533s,Gemma0.5188s,KONI2.2284s. These times exclude replacement initialization.
  • Runtime snapshots: adapter04dab7c plus this harness, subprocess369c8ff (fix: prevent stranded prefill requests after subprocess failure #239), ATOMdf02223 (feat: support cache-off ATOM PD with one KV block #237), HTTP9e442cd (fix: avoid native SIGPIPE handlers on HTTP disconnect #215). SDKs/artifacts match the pinned feat: connect four-model RNGD producers to live ATOM HTTP PD #235 experiment setup. Exact commands, native-entry events and per-model JSON are in serving_matrix/experiments and evidence244.
  • Initial pre-device environment failure retained: container-global Torch conflicted with the ATOM extension. Process-local package-base selection restored existing Torch2.11.0+cpu/Transformers5.8.1 without modifying installations or HOME.
  • Entire serving_matrix CPU suite plus worker tests:50passed,5subtests. Observer tests preserve results/errors. Ruff, diff and scoped sensitive checks pass.

Compatibility and limits

Stacked on #235; optional fault mode only. Parent worker implementation is a trusted local candidate selected with --pd-module, not an SDK patch. This proves explicit healthy replacement, not automatic production restart or readiness behavior while a producer is terminally unavailable. No model-quality, performance-speedup or compute/transfer-overlap claim. Full broader goal remains active.

Related Issue

Closes #244; validates #238/#239 on hardware and extends #234.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant