[gspark-etri]

Background

Full FT with --resident-layers N stopped after 3–13 steps on RNGD with torch.OutOfMemoryError: failed to allocate … bytes on DRAM_SHARED (#40). The failure was intermittent: 2 cards × 8 resident layers failed in 4 of 5 runs, and 1 card × 4 resident layers in 1 of 1. That blocked #34's resident step-time numbers.

Root cause (not out-of-memory). furiosa-torch 2026.2 keeps the device memory type (DRAM / DRAM_SHARED) in one flag for the whole process (furiosa.native_torch.memory._set_dram_shared_mode / _set_memory_type). Every compiled-graph call sets that flag to DRAM_SHARED for a moment. A device allocation from another thread at that moment lands in DRAM_SHARED. On a single-chip card that pool has 0 bytes (the capacity: 0 in the error), so the allocation fails. The layer workers upload bf16 weights while the main thread runs graphs. Resident runs overlap the two the most: the workers start uploading exactly when the main thread runs the resident AdamW graphs.

Evidence (1 card, experiments/data/dram-shared/):

• Uploads and graph outputs share one 47.5 GiB DRAM pool. Each fills to 47.4 GiB, and mixing them fills to the same total.
• A DRAM_SHARED mode set in one thread is visible in another. An 8 MB upload in that window reproduces the exact Resident device AdamW: intermittent RNGD out-of-memory (DRAM_SHARED) after 3-13 steps #40 error.
• A thread polling the flag during 300 x*2 graph calls saw DRAM_SHARED 1,978 times.
• Graph calls plus 4 uploading threads hit the OOM after 6 graph calls.

Changes

• npu_lora/backends.py: adds DEVICE_ALLOC_LOCK.
  • FuriosaBackend.compile returns Serialized(m): calls hold the lock, and other attributes pass through.
  • FuriosaBackend.to_device allocates with torch.empty(device=…) under the lock and runs copy_ outside it.
  • CPU and CUDA backends are unchanged.
• tests/test_cpu.py: test_serialized_graph_call_holds_the_device_alloc_lock.
• experiments/data/dram-shared/race_repro.py (for Furiosa) and pool_probe.py.
• Record experiments/2026-10-21-dram-shared-race.md.
• docs/compiler-workarounds.md gets a runtime section.
• docs/design.md: step-model rows updated, Resident device AdamW: intermittent RNGD out-of-memory (DRAM_SHARED) after 3-13 steps #40 note replaced.
• Pointer added in the 10-21 resident record; STATUS.

Verification

• CI cpu-tests green. Locally I ran the new test and the full-FT / two-device tests.

• Setup: RNGD, furiosa-torch 2026.2.0, torch 2.10.0+cpu, transformers 5.1.0, driver/firmware 2026.4.0. Llama-3.1-8B-Instruct, S=512, accum 4, lr 1e-5, nsmc_mix, seed 0, 25 steps.

• Before each run I checked that no process had /dev/rngd* open and that the cards were free in k8s. I used at most 2 cards on npu-node-1 and did not use npu-node-2.

• Upload speed is unchanged: 117 MB bf16 moves at 6.05 GB/s with t.to and 6.01 GB/s with empty + copy_, and the values match.

• race_repro.py:

  • without the lock: OOM after 6 graph calls;
  • with the lock: 120 s, 58,799 graph calls and 60,725 uploads, 0 errors;
  • with the [14336, 4096] resident AdamW graph: 120 s, 0 errors, and uploads read back bit-exact.

• Reliability. All 8 resident runs finished 25/25 steps with no OOM. Losses match the pre-fix resident runs bit for bit.

  configuration	runs	host load (96 cores)
  2 cards × 8 resident layers per card	3	350–500
  1 card × 4 resident layers	3	380–650
  2 cards × 8, timing run below	1	~50–80
  1 card × 4, timing run below	1	~100

• Full-FT offload is host-bound: measure resident device AdamW and update the placement model #34 timing (median of steps 6–25; host terms from the offload run at the same time):

  run	host load	step	tok/s	model	error
  2 cards × 8 resident	46–81	4.27 s	480	4.22 s	−1.1 %
  2 cards offload, run back to back with the row above	27–71	5.04 s	407	–	–
  1 card × 4 resident	27–111	9.72 s	211	10.82 s	+11.3 %
  1 card offload, run right before the row above	71–107	10.66 s	192	–	–

  2 cards × 8 resident layers is 15 % faster than 2-card offload (+18 % tok/s).

• Reproduce: commands in the record. The Job generator matches experiments/data/w7/jobgen.py, with labels changed.

Compatibility and limits

• Graph calls and device allocations are now serialized process-wide. The lock covers only the allocation and the graph submission, not the copy, so the measured overlap is unchanged: the 2-card offload took 5.04 s/step, against 5.09 s before.
• The runtime still has the bug. Any new code that allocates device tensors from a thread must use be.to_device or take DEVICE_ALLOC_LOCK. Reported to Furiosa via Resident device AdamW: intermittent RNGD out-of-memory (DRAM_SHARED) after 3-13 steps #40 with the repro.
• The 1-card resident timing was measured only at load ~100. The host did not get below load 60 within 1 h, so that row is contended. The uncontended 6–12-step value of 5.12 s, from before, stays in the docs.
• Not done: Resident device AdamW: intermittent RNGD out-of-memory (DRAM_SHARED) after 3-13 steps #40's "done when" asks for a 200-step 2-card resident run. That needs another 2-card slot.

Related Issue

Related to #40 (root cause fixed; the 200-step run remains), related to #34 (resident timing and the model check).

🤖 Generated with Claude Code