gspark-etri · Oct 21, 2026 · Oct 21, 2026
diff --git a/.research/events.jsonl b/.research/events.jsonl
@@ -7,3 +7,4 @@
 {"id": "5e9641ed983f4c599d8629bf643ee27b", "ts": "2026-10-16T15:52:32+09:00", "task": "NPUTRAIN-26", "result": "민감정보 제거·ai-rm.yaml·README ai-rm 절 추가, checker clean, push ad02402. examples/cluster_facts.py IP 3개(NAS·게이트웨이·Prometheus)를 MODEL_STORE_NAS_HOST/LLMD_GATEWAY_HOST/PROMETHEUS_HOST/PORT 환경변수(cluster.local.env)로, cli.py serve 예시 IP를 <vllm-host>로 교체. 생성 데이터는 이전과 동일 확인", "next": "lmd-top 내 vendored 사본도 같은 수정 필요(다른 agent). 정적 k8s 매니페스트·전용 이미지 없음", "state": "idle", "done": true, "blocker": "", "metrics": {}, "params": {}, "artifacts": [], "experiments": [], "labels": ["infra"], "review": false, "todos": ["deploy/k8s 정적 Job 매니페스트(kustomize) 추가"], "note": "", "agent": "claude", "host": "control-node", "branch": "main", "commit": "ad02402-dirty"}
 {"id": "46f3b37355d0471bb25b6f222771b396", "ts": "2026-10-16T17:02:12+09:00", "task": "NPUTRAIN-19", "result": "8B full FT가 RNGD 1장에서 돈다: S=512 accum1 10.33 s/step(49.6 tok/s). 계산 ~1.5 s, 나머지는 기울기 14 GB D2H(3.1 GB/s)+가중치 H2D 전송 대기 — PCIe 병목", "next": "accum 4 그래프 컴파일(npu-node-1, ~2h) 후 step 측정; 2카드 레이어 분할로 링크 2개 활용", "state": "active", "done": false, "blocker": "", "metrics": {"step_time_s": "10.33"}, "params": {"model": "Llama-3.1-8B-Instruct", "mode": "full", "seq_len": "512", "accum": "1", "device": "rngd", "node": "e11", "offload": "pipeline4"}, "artifacts": [{"path": "experiments/data/pipe8b-rngd.jsonl", "size": 17870, "sha256": "d3d4d7a91e5e7d48"}], "experiments": ["experiments/2026-10-16-full8b-offload-step.md"], "labels": ["benchmark", "fine-tuning", "rngd"], "review": false, "todos": [], "note": "", "agent": "claude", "host": "control-node", "branch": "main", "commit": "5de85aa-dirty"}
 {"id": "1f544207d4084d928f2bad7d2d298be7", "ts": "2026-10-16T18:50:23+09:00", "task": "NPUTRAIN-19", "result": "8B full FT RNGD 1장 accum 4: 13.13 s/step(2048 tok) = 156 tok/s, accum 1(49.6 tok/s)의 3.1배. 전송 고정비 ~8.6 s는 그대로", "next": "A100 8B full FT 대조(e11 GPU 비면); 카드 2장 레이어 분할(NPUTRAIN-18)", "state": "active", "done": false, "blocker": "", "metrics": {"step_time_s": "13.13"}, "params": {"model": "Llama-3.1-8B-Instruct", "mode": "full", "seq_len": "512", "accum": "4", "device": "rngd", "node": "e11", "offload": "pipeline4"}, "artifacts": [{"path": "experiments/data/pipe8b4-rngd.jsonl", "size": 20585, "sha256": "0426259b65b231a8"}], "experiments": ["experiments/2026-10-16-full8b-offload-step.md"], "labels": ["benchmark", "fine-tuning", "rngd"], "review": false, "todos": [], "note": "", "agent": "claude", "host": "control-node", "branch": "main", "commit": "5de85aa-dirty"}
+{"id": "9f702fdb114e4b25b23508bb2bab2d14", "ts": "2026-10-21T15:37:39+09:00", "task": "#34 resident device AdamW + host-throughput step model", "result": "compile --adamw-only 150 s (was >2.5 h); 2 RNGD x 8 resident layers 4.57 s/step vs 1-card offload 5.22 s (+14% tok/s, steps 3-7, intermittent OOM #40); --update-workers 8 -8%; host-throughput model within 15% on 6 configs", "next": "fix resident OOM (#40), then 200-step run", "state": "active", "done": false, "blocker": "", "metrics": {"tok_s": "448"}, "params": {"seq_len": "512", "accum": "4", "resident_layers": "8", "cards": "2", "sdk": "furiosa-torch-2026.2.0"}, "artifacts": [], "experiments": ["experiments/2026-10-21-resident-adamw.md"], "labels": ["benchmark"], "review": true, "todos": [], "note": "", "agent": "claude", "host": "control-node", "branch": "perf/34-resident-adamw-compile", "commit": "8539644-dirty"}
diff --git a/README.ko.md b/README.ko.md
@@ -137,6 +137,8 @@ npu-lora eval  --endpoint http://vllm:8000 --model my-adapter --data eval.jsonl
 첫 실행은 학습 그래프를 조각 단위로 컴파일하고(Llama-8B: 64코어 호스트에서 약 1.5시간) `~/.cache/npu-lora`에
 캐시한다. 이후 실행은 바로 시작한다. `npu-lora compile`은 장치를 열지 않고 캐시를 채우므로 아무 호스트에서나
 컴파일한 뒤 캐시를 복사할 수 있다.
+`npu-lora compile --adamw-only --full --model M`은 `--resident-layers N` 전체 파인튜닝이 쓰는 장치 AdamW 그래프만
+컴파일한다(Llama-8B: 그래프 4개, 150초). 가중치와 학습 그래프는 읽지 않는다.
 
 <details>
 <summary><b>전체 파인튜닝</b></summary>

diff --git a/README.md b/README.md
@@ -138,6 +138,8 @@ npu-lora eval  --endpoint http://vllm:8000 --model my-adapter --data eval.jsonl
 The first run compiles the training graphs part by part (Llama-8B: ~1.5 h on a 64-core host) and
 caches them under `~/.cache/npu-lora`; later runs start immediately. `npu-lora compile` fills the
 cache without opening a device, so you can compile on any host and copy the cache.
+`npu-lora compile --adamw-only --full --model M` compiles only the device AdamW graphs that full fine-tuning
+with `--resident-layers N` uses (Llama-8B: 4 graphs, 150 s), without loading weights or the training graphs.
 
 <details>
 <summary><b>Full fine-tuning</b></summary>

diff --git a/STATUS.md b/STATUS.md
@@ -1,12 +1,12 @@
 # STATUS — RNGD NPU 파인튜닝 학습
 
 <!-- research-hub:header:begin (maintained by .research/report.py; keep the "- **Key**: value" format) -->
-- **State**: idle
-- **Updated**: 2026-10-19 09:53+09:00
+- **State**: active
+- **Updated**: 2026-10-21 15:37+09:00
 - **Agent**: claude@control-node
-- **Task**: NPUTRAIN-32 (done)
-- **Last result**: PR #1 갱신(931a189, airm/v0.1): npu_lora_tokens_per_second를 airm_tok_s와 함께 프로토콜 0.2 전까지 재출력(README deprecated 표시), rm-core v0.1.1. 검증: conformance PASS 3/FAIL 0, test_airm 4/4, test_cpu 11/11, helm template OK, check_sensitive clean. airm.workloads conformance는 0.1.1에서도 KeyError(NPUTRAIN-31)
-- **Next**: PR 머지 후 lmd-top을 airm_tok_s 우선 읽기로 전환(NPUTRAIN-30), 프로토콜 0.2에서 예전 시리즈 제거
+- **Task**: #34 resident device AdamW + host-throughput step model
+- **Last result**: compile --adamw-only 150 s (was >2.5 h); 2 RNGD x 8 resident layers 4.57 s/step vs 1-card offload 5.22 s (+14% tok/s, steps 3-7, intermittent OOM #40); --update-workers 8 -8%; host-throughput model within 15% on 6 configs
+- **Next**: fix resident OOM (#40), then 200-step run
 - **Blockers**: none
 <!-- research-hub:header:end -->
 
@@ -42,6 +42,15 @@
 
 <!-- research-hub:log (report.py inserts new entries below this line, newest first) -->
 
+### 2026-10-21 15:37+09:00 — #34 resident device AdamW + host-throughput step model
+
+- Result: compile --adamw-only 150 s (was >2.5 h); 2 RNGD x 8 resident layers 4.57 s/step vs 1-card offload 5.22 s (+14% tok/s, steps 3-7, intermittent OOM #40); --update-workers 8 -8%; host-throughput model within 15% on 6 configs
+- Next: fix resident OOM (#40), then 200-step run
+- Metrics: tok_s=448
+- Params: seq_len=512, accum=4, resident_layers=8, cards=2, sdk=furiosa-torch-2026.2.0
+- Experiments: experiments/2026-10-21-resident-adamw.md
+- By: claude@control-node · perf/34-resident-adamw-compile@8539644-dirty
+
 ### 2026-10-19 09:53+09:00 — NPUTRAIN-32 ✅
 
 - Result: PR #1 갱신(931a189, airm/v0.1): npu_lora_tokens_per_second를 airm_tok_s와 함께 프로토콜 0.2 전까지 재출력(README deprecated 표시), rm-core v0.1.1. 검증: conformance PASS 3/FAIL 0, test_airm 4/4, test_cpu 11/11, helm template OK, check_sensitive clean. airm.workloads conformance는 0.1.1에서도 KeyError(NPUTRAIN-31)

diff --git a/docs/design.md b/docs/design.md
@@ -66,6 +66,51 @@ Llama-3.1-8B full fine-tuning, S=512, one card (tokens/s):
 
 The remaining ~3 s per step is shared between host AdamW over 7B parameters and the transfers.
 
+## Optimizer placement: step-time model
+
+With resident host buffers the offload path is bound by the host, not by the PCIe links: splitting the
+layers over two cards (two links) made the step slower on a 24-core host (6.36 → 9.26 s), where a model
+that charges transfer bytes / link bandwidth predicted 5.6 s
+([two-card record](../experiments/2026-10-21-two-card-split.md)). The model below charges host time instead.
+
+Layers are either *offloaded* (fp32 master and AdamW state on the host) or *resident* (`--resident-layers N`:
+the first N layers of each card keep fp32 master, m and v on the card and update with the `ResidentAdamW` graph).
+
+```
+T_step ≈ T_dev + T_res + T_wait
+T_res  = N_res · t_dev                      device AdamW, main thread (8B layer: 28–33 ms)
+T_wait = max(0, N_off · (t_a + t_u) / W  −  T_res  −  N_res,first · t_fwd)
+```
+
+- `T_dev`: forward + backward of the step on the device(s), including the hidden-state hops between cards.
+  Measure it once as step − `wait_weights_s` of an offload run on the same cards.
+- `t_a`, `t_u`: host AdamW and host→device upload time per offloaded layer, measured with W worker threads
+  (`--update-workers`, default 4) as `prev_adamw_s / 32` and `prev_upload_s / 32` of the offload run. They are
+  host throughput terms: on a contended or small host they grow, and adding cards does not shrink them.
+- Gradient downloads overlap the backward pass and are not on the critical path. The layer jobs start
+  when the global clip is known, and the pool runs them in backward order (layer 31 first), so the next
+  forward waits for nearly all offloaded layers. Resident layers sit at the front of each card's range,
+  so the next forward runs through them (`t_fwd` each) while the host jobs finish.
+
+Check on RNGD, Llama-3.1-8B, S=512, accum 4 (host terms from the one-card offload run of the same host;
+records: [resident AdamW](../experiments/2026-10-21-resident-adamw.md), [two cards](../experiments/2026-10-21-two-card-split.md)):
+
+| host, configuration | predicted | measured | error |
+| --- | ---: | ---: | ---: |
+| 96 cores, 1 card, offload, W=8 (per-layer terms of the W=4 run) | 4.53 s | 4.82 s | −6 % |
+| 96 cores, 1 card, 4 resident layers (steps 6–12; stopped at 12, #40) | 5.10 s | 5.12 s | −0.4 % |
+| 96 cores, 2 cards, 8 resident layers per card (steps 3–7; the run stopped at 7, #40) | 4.33 s | 4.57 s | −5 % |
+| 24 cores, 2 cards, offload | 8.33 s | 9.26 s | −10 % |
+
+The link-bandwidth model predicted the last row as 5.6 s (−40 %).
+
+What the model says: once half of the layers are resident, `T_wait` is a few hundred ms and `T_dev`
+(3.6 s on two cards) is the step. More resident layers only help until `T_wait` reaches zero; after that the
+device compute path, not the optimizer, is the next limit. A card holds about 9 resident 8B layers next to
+its share of bf16 weights and gradients, so all 32 layers resident needs 4 cards.
+Resident layers currently hit an intermittent device out-of-memory error after 3–13 steps
+([#40](https://github.com/ai-computing/npu-train/issues/40)).
+
 ## Job layout on Kubernetes
 
 furiosa-torch opens a card only from a privileged container, which then sees every RNGD on the node.