Baselines on the ATOM set: OCI layers, zstd long-range, fixed blocks, Xet-style CDC vs onggi (#51)

README.ko.md

@@ -4,9 +4,10 @@
 
 [English](README.md) · 한국어
 
-> 상태: **P1.** 설계는 [docs/ko/design.md](docs/ko/design.md)에 있습니다. `onggi dedup-report`로 저장소가 얼마나
-> 줄어드는지 측정하고, `onggi push` / `pull`로 POSIX 마운트나 S3 서버에 결과물을 저장·복원합니다. 클러스터
-> 구성요소(노드 agent, CSI 드라이버, 컨트롤러)는 P2–P3에서 이어집니다.
+> 상태: **P2.** 설계는 [docs/ko/design.md](docs/ko/design.md)에 있습니다. `onggi push` / `pull`로 POSIX 마운트나 S3
+> 서버에 결과물을 저장·복원하며 GC, 노드 캐시, peer, 델타 적용을 지원합니다. CSI 드라이버([deploy/csi](deploy/csi/README.md))는
+> Kubernetes에서 결과물을 읽기 전용 볼륨으로 제공합니다. 컨트롤러(컴파일 팜, 프리페치)는 P3에서 이어집니다. 기존 방법과의
+> 비교는 [docs/evaluation.md](docs/evaluation.md)에 있습니다.
 
 NPU는 모델을 미리 컴파일하며 모델 × SDK × 장치 × 길이 구간 × 배치 × 옵션 조합마다 결과물이 하나씩 생깁니다. 연구실
 RNGD + ATOM 노드에서 Rebellions ATOM용 Llama-3.1-8B 빌드 묶음 하나가 **262 GB**를 차지합니다. 그래프마다 가중치가 통째로

README.md

@@ -4,9 +4,10 @@
 
 English · [한국어](README.ko.md)
 
-> Status: **P1.** The design is in [docs/design.md](docs/design.md). `onggi dedup-report` measures what a store would
-> deduplicate; `onggi push` / `pull` store and restore artifacts on a POSIX mount or an S3 server. The cluster
-> components (node agent, CSI driver, controller) follow in P2–P3.
+> Status: **P2.** The design is in [docs/design.md](docs/design.md). `onggi push` / `pull` store and restore artifacts
+> on a POSIX mount or an S3 server, with GC, a node cache, peers and delta apply; the CSI driver
+> ([deploy/csi](deploy/csi/README.md)) serves them as read-only volumes on Kubernetes. The controller (compile farm,
+> prefetch) follows in P3. Measurements against common methods: [docs/evaluation.md](docs/evaluation.md).
 
 NPUs compile models ahead of time, one artifact per model × SDK × device × sequence bucket × batch × options. On our
 RNGD + ATOM node, one Llama-3.1-8B build set for Rebellions ATOM takes **262 GB** because every graph embeds the

bench/baselines.sh

@@ -0,0 +1,64 @@
+#!/usr/bin/env bash
+# Baselines on one build set (#51): what storing the set and switching a node between two variants
+# cost with common methods and with onggi, on the same files.
+#
+#   bench/baselines.sh <onggi binary> <build dir> <work dir> <variant A file> <variant B file>
+#
+# The work dir needs room for two onggi stores of the set (about 2 × 50 GB for the ATOM
+# Llama-3.1-8B set). Needs zstd >= 1.4, tar, python3. Prints TSV rows: section, method, bytes,
+# seconds. Everything runs under nice; nothing outside the work dir is written.
+set -euo pipefail
+onggi=$1 src=$2 work=$3 va=$4 vb=$5
+mkdir -p "$work"
+row() { printf '%s\t%s\t%s\t%s\n' "$@"; }
+now() { date +%s.%N; }
+since() { python3 -c "import sys;print(round($(now)-$1,1))"; }
+json() { python3 -c "import json,sys;d=json.load(open(sys.argv[1]));print(eval(sys.argv[2]))" "$@"; }
+N="nice -n 19"
+
+raw=$(find "$src" -type f -printf '%s\n' | awk '{s+=$1} END {print s}')
+row storage raw "$raw" 0
+
+# OCI-style layers: one layer per distinct file (by sha256), each compressed with zstd -3
+t=$(now)
+oci=$(cd "$src" && find . -type f -print0 | sort -z | xargs -0 sha256sum | sort -u -k1,1 | cut -c67- |
+	while read -r f; do $N zstd -q -3 -T0 -c "$f" | wc -c; done | awk '{s+=$1} END {print s}')
+row storage oci-layers-zstd3 "$oci" "$(since "$t")"
+
+# the whole set as one tar, zstd with a 2 GiB long-range window
+t=$(now)
+lr=$(cd "$src" && tar cf - . | $N zstd -q -3 -T0 --long=31 -c | wc -c)
+row storage tar-zstd3-long31 "$lr" "$(since "$t")"
+
+# fixed blocks and FastCDC 64 KiB, dedup only
+for b in 64KiB 1MiB; do
+	t=$(now)
+	$N "$onggi" dedup-report --quiet --json --fixed-block "$b" "$src" > "$work/dr-$b.json"
+	row storage "fixed-$b-dedup" "$(json "$work/dr-$b.json" "d['fixed_block']['unique_bytes']")" "$(since "$t")"
+done
+row storage cdc64k-dedup "$(json "$work/dr-1MiB.json" "d['unique_bytes']")" -
+
+# Xet-style: CDC 64 KiB with per-chunk compression (onggi without sub-chunks), and onggi
+for m in xet:--sub=0 onggi:; do
+	name=${m%%:*} flag=${m#*:}
+	rm -rf "$work/$name"
+	t=$(now)
+	$N "$onggi" push --store "$work/$name" $flag --quiet --json --tmp "$work" "$src" > "$work/push-$name.json"
+	row storage "$name" "$(du -sb "$work/$name" | cut -f1)" "$(since "$t")"
+done
+
+# switching a node from variant A to B: what crosses the network
+row transfer oci-layer "$($N zstd -q -3 -T0 -c "$src/$vb" | wc -c)" -
+for m in xet:--sub=0 onggi:; do
+	name=${m%%:*} flag=${m#*:}
+	for v in a b; do
+		rm -rf "$work/v$v" && mkdir -p "$work/v$v"
+		f=$va; [ $v = b ] && f=$vb
+		ln "$src/$f" "$work/v$v/" 2>/dev/null || cp "$src/$f" "$work/v$v/"
+	done
+	rm -rf "$work/t-$name"
+	"$onggi" push --store "$work/t-$name" $flag --quiet --json --tmp "$work" "$work/va" > /dev/null
+	"$onggi" push --store "$work/t-$name" $flag --quiet --json --tmp "$work" "$work/vb" > "$work/t-$name.json"
+	row transfer "$name" "$(json "$work/t-$name.json" "d['stats']['pack_bytes'] if 'stats' in d else d['pack_bytes']")" -
+	rm -rf "$work/t-$name" "$work/va" "$work/vb"
+done

docs/evaluation.md

@@ -0,0 +1,43 @@
+# Evaluation
+
+Measured 2026-10-22 on the lab's RNGD + ATOM node with [`bench/baselines.sh`](../bench/baselines.sh) (#51), over one
+Rebellions ATOM Llama-3.1-8B build set: 18 files, 263.9 GB (decode and prefill graphs for several batch sizes and
+sequence buckets, each embedding the weights).
+
+## Storing the set
+
+| method | stored | vs raw | time |
+|---|---|---|---|
+| raw | 263.9 GB | 1.00× | — |
+| OCI-style layers: whole-file dedup, zstd -3 per file | 199.2 GB | 1.33× | 27.6 min |
+| one tar, zstd -3 `--long=31` (2 GiB window) | 199.1 GB | 1.33× | 28.3 min |
+| fixed-block dedup, 1 MiB | 116.4 GB | 2.27× | 26 s |
+| fixed-block dedup, 64 KiB | 113.4 GB | 2.33× | 27 s |
+| FastCDC 64 KiB dedup | 46.8 GB | 5.64× | — |
+| Xet-style: FastCDC 64 KiB + per-chunk compression (`onggi --sub 0`) | 29.5 GB | 8.95× | 1.0 min |
+| **onggi**: two-level chunking + per-chunk codecs | **17.0 GB** | **15.6×** | 1.3 min |
+
+Whole-file methods find nothing to share: every file differs. A long-range window of 2 GiB cannot reach the copy of
+the weights in the previous 15 GB file. Fixed blocks lose the weights as soon as a header shifts them. CDC finds the
+weights; sub-chunks also find the ones prefill graphs hold in a different order (#15). Sizes include each store's
+indexes and manifests.
+
+Xet itself needs the Hugging Face service; the Xet-style row uses the same idea (64 KiB CDC, per-chunk compression)
+inside onggi. The compression times run under `nice -n 19` on a shared node.
+
+## Switching a node from decode B2 to decode B8
+
+Bytes that cross the network when a node that holds B2 needs B8 (15.2 GB):
+
+| method | transferred |
+|---|---|
+| OCI layer (zstd -3) | 11.46 GB |
+| Xet-style CDC + per-chunk compression | **6.1 MB** |
+| onggi | 28.3 MB |
+
+The 218 MB that differ compress well. As 64 KiB chunks they shrink to 6.1 MB; onggi splits new chunks of ATOM files
+into ~4 KiB sub-chunks, which compress worse, so it sends 28.3 MB. #52 tracks compressing a chunk's new sub-chunks
+together.
+
+Applying the switch on the node (README, deploy/csi): a plain pull through the node cache takes 73.8 s; delta apply
+(#41) takes 5.7 s without and 8.9 s with the node cache, and a CSI pod is Running 12 s after creation.