Skip to content

fix: import dense subprocess KV into hybrid GPU cache groups - #259

Closed
gspark-etri wants to merge 2 commits into
feat/249-pd-stage-metricsfrom
fix/258-dense-hybrid-kv
Closed

gspark-etri wants to merge 2 commits into
feat/249-pd-stage-metricsfrom
fix/258-dense-hybrid-kv

Conversation

@gspark-etri

@gspark-etri gspark-etri commented Oct 23, 2026 •

Copy link
Copy Markdown
Collaborator

Background

The first live Gemma3-1B RNGD→GPU request crashed: the subprocess producer returned all26 model layers in one KV tensor, while the GPU receiver tried to copy it into its22-layer sliding-attention group. Existing grouped Torch producers did not exercise this contract.

Changes

PagedKV.write_span maps dense model-layer KV into receiver groups, preserves existing single-group/recurrent and grouped inputs, and validates group counts and shapes before copying. The subprocess wire format is unchanged. Added nonzero-offset layer-mapping and malformed-input regressions plus dense Gemma PD outputs compared against independent HF greedy beyond the sliding window.

Verification

Code32ce5d5, base273952a (#250); actual failed runtime3e16c33, BF16 Gemma3-1B, RNGD SDK2026.4, GPU Torch2.13.0+cu130, A10080GB, width8192,4slots/context9216/page64/cacheoff. Original failure retained; no completed hardware performance claim.

  • RED:3failures reproduced mapping, silent missing group and partial malformed-group write.
  • Focused:31passed,29.86s (including existing recurrent/hybrid/disaggregated tests).
  • Full CPU:505passed,34skipped,253subtests passed,151.69s; one existing flex-attention warning.
  • Commands/results in opt_prime/examples/pd/http_matrix/experiments/2026-10-24-dense-hybrid258.md; evidence258 contains original failure and test logs (trailing whitespace normalized).
  • Scoped sensitive scan and diff check clean. Corrected live Gemma32ce5d5 completed24timed batches;20/20nonstream token checks and60/60timed response texts equal unchanged nativeGPU references.84remote prefills,0local,0request errors. Inputs128/8192,outputs32/128,c1/4,3repeats. At8192/output128/c4 median55.019tok/s; no speedup comparison claimed. Raw original failure and corrected artifacts both retained.

Compatibility and limits

Dense spans require uniform head geometry; grouped tensors still support heterogeneous geometry. No default routing, cache policy or transport changes. The recorded hardware matrix now passes. This does not establish arbitrary-prompt quality or all numerical configurations; draft retained for maintainer review. Separate Qwen/KONI/Gemma numerical investigations remain open.

Related Issue

Refs #258, #201. Stacked on #250; maintainer reviews and merges.

gspark-etri added a commit that referenced this pull request Oct 30, 2026
)

PagedKV.write_span splits dense model-layer-order KV (subprocess producers)
into the receiver's per-geometry groups and validates all shapes before any
copy. From fix/258-dense-hybrid-kv (#259).

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@gspark-etri

Copy link
Copy Markdown
Collaborator Author

Landed on optimus-rm via #283. Closing; the branch is archived locally (experiment records and evidence included).

@gspark-etri gspark-etri closed this Nov 3, 2026
@gspark-etri
gspark-etri deleted the fix/258-dense-hybrid-kv branch November 3, 2026 17:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant