Fix resident-AdamW OOM on DRAM_SHARED: serialize device allocations with graph calls (#40)

.research/events.jsonl

@@ -9,5 +9,6 @@
 {"id": "1f544207d4084d928f2bad7d2d298be7", "ts": "2026-10-16T18:50:23+09:00", "task": "NPUTRAIN-19", "result": "8B full FT RNGD 1장 accum 4: 13.13 s/step(2048 tok) = 156 tok/s, accum 1(49.6 tok/s)의 3.1배. 전송 고정비 ~8.6 s는 그대로", "next": "A100 8B full FT 대조(e11 GPU 비면); 카드 2장 레이어 분할(NPUTRAIN-18)", "state": "active", "done": false, "blocker": "", "metrics": {"step_time_s": "13.13"}, "params": {"model": "Llama-3.1-8B-Instruct", "mode": "full", "seq_len": "512", "accum": "4", "device": "rngd", "node": "e11", "offload": "pipeline4"}, "artifacts": [{"path": "experiments/data/pipe8b4-rngd.jsonl", "size": 20585, "sha256": "0426259b65b231a8"}], "experiments": ["experiments/2026-10-16-full8b-offload-step.md"], "labels": ["benchmark", "fine-tuning", "rngd"], "review": false, "todos": [], "note": "", "agent": "claude", "host": "control-node", "branch": "main", "commit": "5de85aa-dirty"}
 {"id": "9f702fdb114e4b25b23508bb2bab2d14", "ts": "2026-10-21T15:37:39+09:00", "task": "#34 resident device AdamW + host-throughput step model", "result": "compile --adamw-only 150 s (was >2.5 h); 2 RNGD x 8 resident layers 4.57 s/step vs 1-card offload 5.22 s (+14% tok/s, steps 3-7, intermittent OOM #40); --update-workers 8 -8%; host-throughput model within 15% on 6 configs", "next": "fix resident OOM (#40), then 200-step run", "state": "active", "done": false, "blocker": "", "metrics": {"tok_s": "448"}, "params": {"seq_len": "512", "accum": "4", "resident_layers": "8", "cards": "2", "sdk": "furiosa-torch-2026.2.0"}, "artifacts": [], "experiments": ["experiments/2026-10-21-resident-adamw.md"], "labels": ["benchmark"], "review": true, "todos": [], "note": "", "agent": "claude", "host": "control-node", "branch": "perf/34-resident-adamw-compile", "commit": "8539644-dirty"}
 {"id": "f97382c821f344d08c88e8afc0c290ca", "ts": "2026-10-21T20:20:40+09:00", "task": "#33 Qwen2 gradient check on RNGD", "result": "npu-lora check passes (worst 3.1%, was 44%); 0 negative-loss steps in 400 (was 11); loss curve matches A100", "next": "Llama-3.1-8B check with the new head; Qwen3", "state": "active", "done": false, "blocker": "", "metrics": {"step_time_s": "2.74"}, "params": {}, "artifacts": [], "experiments": ["experiments/2026-10-21-qwen2-grad-precision.md"], "labels": ["bug", "experiment"], "review": true, "todos": [], "note": "", "agent": "claude", "host": "DESKTOP-R49IQ68", "branch": "fix/33-qwen-lora-grad", "commit": "a7e206a"}
+{"id": "aa4e47d3f9354d9ca18b563fd512c24c", "ts": "2026-10-21T21:10:59+09:00", "task": "#40 resident AdamW DRAM_SHARED OOM", "result": "cause: furiosa-torch 2026.2 process-global DRAM_SHARED flag raced by layer-worker uploads (not memory); fix: one lock for graph calls + device allocations; 8/8 resident runs 25/25 steps no OOM; 2 RNGD x 8 resident 4.27 s/step (480 tok/s) vs 2-card offload 5.04 s, model -1.1%", "next": "200-step 2-card resident run (#40 done-when); 1-card resident 4 timing at low host load; report the runtime flag to Furiosa", "state": "active", "done": false, "blocker": "", "metrics": {"tok_s": "479.6", "step_s": "4.27"}, "params": {"cards": "2", "resident_layers": "8", "host_load": "46-81", "sdk": "furiosa-torch-2026.2.0"}, "artifacts": [], "experiments": ["experiments/2026-10-21-dram-shared-race.md"], "labels": ["bug", "benchmark"], "review": true, "todos": [], "note": "", "agent": "claude", "host": "control-node", "branch": "fix/40-dram-shared-race", "commit": "0e58322-dirty"}
 {"id": "b958b783d2b041f2ba43e072eb284abc", "ts": "2026-10-21T21:57:00+09:00", "task": "#35 MoE hot experts batched", "result": "batched hot experts 13.0 s/step vs host-only 13.5 (alternating steps, Qwen3-30B-A3B, RNGD); per-expert main 42.4 vs 17.2", "next": "more rows per call (longer S), hot fraction 0.5, drop the recomputed forward", "state": "active", "done": false, "blocker": "", "metrics": {"step_time_s": "13.0"}, "params": {}, "artifacts": [], "experiments": ["experiments/2026-10-21-moe-batched-hot-experts.md"], "labels": ["experiment", "benchmark"], "review": false, "todos": [], "note": "", "agent": "claude", "host": "DESKTOP-R49IQ68", "branch": "perf/35-batch-hot-experts", "commit": "9d5e498"}
 {"id": "c37c232f20a045e0aa3315cc1c786106", "ts": "2026-10-21T22:27:58+09:00", "task": "#35 MoE hot experts batched", "result": "hot+keep_graph 5.19 s/step vs host-only+keep 8.00 (paired median -2.66 s, 6/6 rounds) and vs main host-only 9.21 s; Qwen3-30B-A3B S128 RNGD", "next": "", "state": "idle", "done": true, "blocker": "", "metrics": {"step_time_s": "5.19"}, "params": {}, "artifacts": [], "experiments": ["experiments/2026-10-21-moe-batched-hot-experts.md"], "labels": ["experiment", "benchmark"], "review": false, "todos": [], "note": "", "agent": "claude", "host": "DESKTOP-R49IQ68", "branch": "perf/35-hot-experts-faster", "commit": "1fee9f2-dirty"}

STATUS.md

@@ -1,12 +1,12 @@
 # STATUS — RNGD NPU 파인튜닝 학습
 
 <!-- research-hub:header:begin (maintained by .research/report.py; keep the "- **Key**: value" format) -->
-- **State**: idle
-- **Updated**: 2026-10-21 22:27+09:00
-- **Agent**: claude@DESKTOP-R49IQ68
-- **Task**: #35 MoE hot experts batched (done)
-- **Last result**: hot+keep_graph 5.19 s/step vs host-only+keep 8.00 (paired median -2.66 s, 6/6 rounds) and vs main host-only 9.21 s; Qwen3-30B-A3B S128 RNGD
-- **Next**: -
+- **State**: active
+- **Updated**: 2026-10-21 21:10+09:00
+- **Agent**: claude@control-node
+- **Task**: #40 resident AdamW DRAM_SHARED OOM
+- **Last result**: cause: furiosa-torch 2026.2 process-global DRAM_SHARED flag raced by layer-worker uploads (not memory); fix: one lock for graph calls + device allocations; 8/8 resident runs 25/25 steps no OOM; 2 RNGD x 8 resident 4.27 s/step (480 tok/s) vs 2-card offload 5.04 s, model -1.1%
+- **Next**: 200-step 2-card resident run (#40 done-when); 1-card resident 4 timing at low host load; report the runtime flag to Furiosa
 - **Blockers**: none
 <!-- research-hub:header:end -->
 
@@ -57,6 +57,15 @@
 - Experiments: experiments/2026-10-21-moe-batched-hot-experts.md
 - By: claude@DESKTOP-R49IQ68 · perf/35-batch-hot-experts@9d5e498
 
+### 2026-10-21 21:10+09:00 — #40 resident AdamW DRAM_SHARED OOM
+
+- Result: cause: furiosa-torch 2026.2 process-global DRAM_SHARED flag raced by layer-worker uploads (not memory); fix: one lock for graph calls + device allocations; 8/8 resident runs 25/25 steps no OOM; 2 RNGD x 8 resident 4.27 s/step (480 tok/s) vs 2-card offload 5.04 s, model -1.1%
+- Next: 200-step 2-card resident run (#40 done-when); 1-card resident 4 timing at low host load; report the runtime flag to Furiosa
+- Metrics: tok_s=479.6, step_s=4.27
+- Params: cards=2, resident_layers=8, host_load=46-81, sdk=furiosa-torch-2026.2.0
+- Experiments: experiments/2026-10-21-dram-shared-race.md
+- By: claude@control-node · fix/40-dram-shared-race@0e58322-dirty
+
 ### 2026-10-21 20:20+09:00 — #33 Qwen2 gradient check on RNGD
 
 - Result: npu-lora check passes (worst 3.1%, was 44%); 0 negative-loss steps in 400 (was 11); loss curve matches A100

docs/compiler-workarounds.md

@@ -45,3 +45,20 @@ apt-get install -y --no-install-recommends gcc-aarch64-linux-gnu libc6-dev-arm64
 
 `npu-lora job` 과 `lmd-top --plan train` 이 만드는 Job 은 이 설치를 먼저 한다. 컴파일러는 또
 `~/.cache/furiosa/compiler` 를 스스로 만들지 않으므로 백엔드가 미리 만든다.
+
+## 런타임: 여러 스레드에서 장치 메모리 할당 (#40)
+
+`torch.OutOfMemoryError: failed to allocate … bytes on DRAM_SHARED` 가 전체 파인튜닝 레이어 작업 스레드의 업로드에서
+간헐적으로 났다(`--resident-layers` 실행, 3–13 step 뒤). 원인은 메모리 부족이 아니다.
+
+- furiosa-torch 2026.2 는 장치 메모리 종류(DRAM / DRAM_SHARED)를 프로세스 전체에 하나뿐인 플래그로 둔다
+  (`furiosa.native_torch.memory._set_dram_shared_mode` / `_set_memory_type`). 한 스레드에서 켜면 다른 스레드도 켜진 값을 본다.
+- 컴파일된 그래프 호출은 매번 이 플래그를 잠깐 DRAM_SHARED 로 바꾼다(x·2 그래프 300번을 부르는 동안 다른 스레드가 플래그를 800만 번 읽어 1978번 DRAM_SHARED 를 봄).
+- 그 순간 다른 스레드가 장치 텐서를 만들면 DRAM_SHARED 에 잡히는데, 칩 1개 카드에서 그 영역은 0 바이트라 바로 실패한다.
+  보통 할당과 그래프 출력은 모두 DRAM(47.5 GiB 한 영역)에 잡힌다.
+- 메인 스레드가 그래프를 도는 동안 작업 스레드가 올리는 시간이 길수록 잘 겹친다. 상주 실행은 메인 스레드가 상주 AdamW
+  그래프를 돌리는 바로 그때 작업 스레드들이 업로드해서 몇 step 안에 걸렸고, 오프로드만 하는 실행은 덜 겹쳐서 운 좋게 피했다.
+
+대응: `backends.DEVICE_ALLOC_LOCK` — 그래프 호출(`Serialized`)과 장치 할당(`FuriosaBackend.to_device` 의 `torch.empty`)이
+같은 잠금을 잡는다. 복사(`copy_`)는 잠금 밖이라 업로드 속도는 같다(117 MB bf16: `t.to` 6.05 GB/s, `empty`+`copy_` 6.01 GB/s).
+재현: `experiments/data/dram-shared/race_repro.py`(잠금 없이 그래프 6번 만에 OOM, 잠금으로 120 s·그래프 58 799번 동안 없음).

docs/design.md

@@ -92,24 +92,29 @@ T_wait = max(0, N_off · (t_a + t_u) / W  −  T_res  −  N_res,first · t_fwd)
   forward waits for nearly all offloaded layers. Resident layers sit at the front of each card's range,
   so the next forward runs through them (`t_fwd` each) while the host jobs finish.
 
-Check on RNGD, Llama-3.1-8B, S=512, accum 4 (host terms from the one-card offload run of the same host;
-records: [resident AdamW](../experiments/2026-10-21-resident-adamw.md), [two cards](../experiments/2026-10-21-two-card-split.md)):
+Check on RNGD, Llama-3.1-8B, S=512, accum 4 (host terms from an offload run on the same host at the same load;
+records: [resident AdamW](../experiments/2026-10-21-resident-adamw.md), [two cards](../experiments/2026-10-21-two-card-split.md),
+[DRAM_SHARED race](../experiments/2026-10-21-dram-shared-race.md)):
 
 | host, configuration | predicted | measured | error |
 | --- | ---: | ---: | ---: |
 | 96 cores, 1 card, offload, W=8 (per-layer terms of the W=4 run) | 4.53 s | 4.82 s | −6 % |
-| 96 cores, 1 card, 4 resident layers (steps 6–12; stopped at 12, #40) | 5.10 s | 5.12 s | −0.4 % |
-| 96 cores, 2 cards, 8 resident layers per card (steps 3–7; the run stopped at 7, #40) | 4.33 s | 4.57 s | −5 % |
+| 96 cores, 1 card, 4 resident layers (steps 6–12, uncontended; that run stopped at 12, #40) | 5.10 s | 5.12 s | −0.4 % |
+| 96 cores at load ~100, 1 card, 4 resident layers (steps 6–25; terms of the 1-card offload run right before) | 10.82 s | 9.72 s | +11 % |
+| 96 cores, 2 cards, 8 resident layers per card (steps 6–25; terms of the 2-card offload run back to back) | 4.22 s | 4.27 s | −1 % |
 | 24 cores, 2 cards, offload | 8.33 s | 9.26 s | −10 % |
 
 The link-bandwidth model predicted the last row as 5.6 s (−40 %).
 
 What the model says: once half of the layers are resident, `T_wait` is a few hundred ms and `T_dev`
-(3.6 s on two cards) is the step. More resident layers only help until `T_wait` reaches zero; after that the
+(3.5 s on two cards) is the step: 2 cards × 8 resident layers run at 4.27 s/step, 15 % faster than 2-card offload (5.04 s). More resident layers only help until `T_wait` reaches zero; after that the
 device compute path, not the optimizer, is the next limit. A card holds about 9 resident 8B layers next to
 its share of bf16 weights and gradients, so all 32 layers resident needs 4 cards.
-Resident layers currently hit an intermittent device out-of-memory error after 3–13 steps
-([#40](https://github.com/ai-computing/npu-train/issues/40)).
+Resident runs used to stop with an intermittent `OutOfMemoryError ... on DRAM_SHARED` after 3–13 steps
+([#40](https://github.com/ai-computing/npu-train/issues/40)). The cause was not memory: furiosa-torch 2026.2 keeps the
+device memory type in one process-wide flag that every graph call briefly sets to DRAM_SHARED (0 bytes on a single-chip
+card), so a layer worker's upload at that moment failed. Graph calls and device allocations now share one lock
+(`backends.DEVICE_ALLOC_LOCK`; see [compiler workarounds](compiler-workarounds.md)).
 
 ## Job layout on Kubernetes