[gspark-etri]

Background

#42 fixed the cause of #40, a race on furiosa-torch's process-global DRAM_SHARED flag. That PR showed 25-step runs only. #40's done-when asks for a 200-step run of train --full --resident-layers 8 --backend furiosa:auto,furiosa:auto with no OOM. #34 still needed the 1 card × 4 resident timing on an uncontended host.

Changes

• experiments/2026-10-22-resident-200-steps.md: record of both runs, with losses, step times and the step-model check.
• experiments/data/resident-200/: step logs (r200.jsonl, t1c.jsonl), pre-submit checks and results (run200.log), and host load.
• docs/design.md: adds the low-load 1-card row to the step-model table and notes the 200-step result.
• STATUS.md and .research/events.jsonl via report.py.
• No code changes.

Verification

• Code: main b3cb03b, which includes Fix resident-AdamW OOM on DRAM_SHARED: serialize device allocations with graph calls (#40) #42 and Qwen2 on RNGD: pass npu-lora check (fp32 q/k + centred k), no negative loss (host target logit) #43.
• Hardware: RNGD on npu-node-1, furiosa-torch 2026.2.0, image digest sha256:37a85288…, driver/firmware 2026.4.0.
• Recipe: Llama-3.1-8B-Instruct, S=512, accum 4, lr 1e-5, nsmc_mix, seed 0.
• Jobs: one run-and-exit Job at a time, each with activeDeadlineSeconds and ttlSecondsAfterFinished.
• Pre-submit check before each Job: host load below 150, the needed cards free in k8s, and no process holding /dev/rngd. The checks logged load 2–21.

Run 1: 2 cards × 8 resident layers per card, 200 steps.

• 200/200 steps, no OOM, on the first attempt. Host load was 15–50.
• Step 6–200 median: 4.43 s/step, 463 tok/s. The steps 6–25 median was also 4.42 s, and the slowest step took 4.92 s, so step time did not drift.

steps	mean loss	median step
1–25	1.262	4.43 s
26–50	1.172	4.40 s
51–100	1.109	4.39 s
101–150	1.097	4.53 s
151–200	1.151	4.44 s

• Step 1 loss is 1.7202. Before Qwen2 on RNGD: pass npu-lora check (fp32 q/k + centred k), no negative loss (host target logit) #43 it was 1.7157, because Qwen2 on RNGD: pass npu-lora check (fp32 q/k + centred k), no negative loss (host target logit) #43 changed the head.

Run 2: 1 card, same Job, back to back, load 10–44. Median of steps 6–25:

run	step	tok/s	model	error
1-card offload	5.11 s	401	–	–
1 card × 4 resident	4.79 s	427	4.94 s	+3.0 %

The model's terms come from the offload run in the same Job.

Compatibility and limits

• No code changes.
• Head recompile on the card: Qwen2 on RNGD: pass npu-lora check (fp32 q/k + centred k), no negative loss (host target logit) #43 changed the head graphs, so the 200-step pod recompiled two head pieces on the device while holding 2 cards (94 s + 398 s). Next time main changes graphs, I'll run npu-lora compile in a CPU Job first.
• First submit failed: the node had lost the furiosa-llm:latest image, so the first submit hit ImagePullBackOff. I deleted that Job at once and re-pulled the image with a pod that holds no card.
• No same-time 2-card offload baseline: I used the 2-card slot only once, so this PR has no 2-card offload run at the same time. The 2-card model row (−4.4 %) uses the 10-21 T_dev from before Qwen2 on RNGD: pass npu-lora check (fp32 q/k + centred k), no negative loss (host target logit) #43 and is for reference only.
• Still open: reporting the runtime flag to Furiosa, with the repro from Fix resident-AdamW OOM on DRAM_SHARED: serialize device allocations with graph calls (#40) #42 (experiments/data/dram-shared/race_repro.py).

Related Issue

Closes #40. Related to #34.

🤖 Generated with Claude Code