diff --git a/README.ko.md b/README.ko.md index cbcef5a..37e692e 100644 --- a/README.ko.md +++ b/README.ko.md @@ -4,9 +4,10 @@ [English](README.md) · 한국어 -> 상태: **P1.** 설계는 [docs/ko/design.md](docs/ko/design.md)에 있습니다. `onggi dedup-report`로 저장소가 얼마나 -> 줄어드는지 측정하고, `onggi push` / `pull`로 POSIX 마운트나 S3 서버에 결과물을 저장·복원합니다. 클러스터 -> 구성요소(노드 agent, CSI 드라이버, 컨트롤러)는 P2–P3에서 이어집니다. +> 상태: **P2.** 설계는 [docs/ko/design.md](docs/ko/design.md)에 있습니다. `onggi push` / `pull`로 POSIX 마운트나 S3 +> 서버에 결과물을 저장·복원하며 GC, 노드 캐시, peer, 델타 적용을 지원합니다. CSI 드라이버([deploy/csi](deploy/csi/README.md))는 +> Kubernetes에서 결과물을 읽기 전용 볼륨으로 제공합니다. 컨트롤러(컴파일 팜, 프리페치)는 P3에서 이어집니다. 기존 방법과의 +> 비교는 [docs/evaluation.md](docs/evaluation.md)에 있습니다. NPU는 모델을 미리 컴파일하며 모델 × SDK × 장치 × 길이 구간 × 배치 × 옵션 조합마다 결과물이 하나씩 생깁니다. 연구실 RNGD + ATOM 노드에서 Rebellions ATOM용 Llama-3.1-8B 빌드 묶음 하나가 **262 GB**를 차지합니다. 그래프마다 가중치가 통째로 diff --git a/README.md b/README.md index f18d51d..266f550 100644 --- a/README.md +++ b/README.md @@ -4,9 +4,10 @@ English · [한국어](README.ko.md) -> Status: **P1.** The design is in [docs/design.md](docs/design.md). `onggi dedup-report` measures what a store would -> deduplicate; `onggi push` / `pull` store and restore artifacts on a POSIX mount or an S3 server. The cluster -> components (node agent, CSI driver, controller) follow in P2–P3. +> Status: **P2.** The design is in [docs/design.md](docs/design.md). `onggi push` / `pull` store and restore artifacts +> on a POSIX mount or an S3 server, with GC, a node cache, peers and delta apply; the CSI driver +> ([deploy/csi](deploy/csi/README.md)) serves them as read-only volumes on Kubernetes. The controller (compile farm, +> prefetch) follows in P3. Measurements against common methods: [docs/evaluation.md](docs/evaluation.md). NPUs compile models ahead of time, one artifact per model × SDK × device × sequence bucket × batch × options. On our RNGD + ATOM node, one Llama-3.1-8B build set for Rebellions ATOM takes **262 GB** because every graph embeds the diff --git a/bench/baselines.sh b/bench/baselines.sh new file mode 100644 index 0000000..10d91f6 --- /dev/null +++ b/bench/baselines.sh @@ -0,0 +1,64 @@ +#!/usr/bin/env bash +# Baselines on one build set (#51): what storing the set and switching a node between two variants +# cost with common methods and with onggi, on the same files. +# +# bench/baselines.sh +# +# The work dir needs room for two onggi stores of the set (about 2 × 50 GB for the ATOM +# Llama-3.1-8B set). Needs zstd >= 1.4, tar, python3. Prints TSV rows: section, method, bytes, +# seconds. Everything runs under nice; nothing outside the work dir is written. +set -euo pipefail +onggi=$1 src=$2 work=$3 va=$4 vb=$5 +mkdir -p "$work" +row() { printf '%s\t%s\t%s\t%s\n' "$@"; } +now() { date +%s.%N; } +since() { python3 -c "import sys;print(round($(now)-$1,1))"; } +json() { python3 -c "import json,sys;d=json.load(open(sys.argv[1]));print(eval(sys.argv[2]))" "$@"; } +N="nice -n 19" + +raw=$(find "$src" -type f -printf '%s\n' | awk '{s+=$1} END {print s}') +row storage raw "$raw" 0 + +# OCI-style layers: one layer per distinct file (by sha256), each compressed with zstd -3 +t=$(now) +oci=$(cd "$src" && find . -type f -print0 | sort -z | xargs -0 sha256sum | sort -u -k1,1 | cut -c67- | + while read -r f; do $N zstd -q -3 -T0 -c "$f" | wc -c; done | awk '{s+=$1} END {print s}') +row storage oci-layers-zstd3 "$oci" "$(since "$t")" + +# the whole set as one tar, zstd with a 2 GiB long-range window +t=$(now) +lr=$(cd "$src" && tar cf - . | $N zstd -q -3 -T0 --long=31 -c | wc -c) +row storage tar-zstd3-long31 "$lr" "$(since "$t")" + +# fixed blocks and FastCDC 64 KiB, dedup only +for b in 64KiB 1MiB; do + t=$(now) + $N "$onggi" dedup-report --quiet --json --fixed-block "$b" "$src" > "$work/dr-$b.json" + row storage "fixed-$b-dedup" "$(json "$work/dr-$b.json" "d['fixed_block']['unique_bytes']")" "$(since "$t")" +done +row storage cdc64k-dedup "$(json "$work/dr-1MiB.json" "d['unique_bytes']")" - + +# Xet-style: CDC 64 KiB with per-chunk compression (onggi without sub-chunks), and onggi +for m in xet:--sub=0 onggi:; do + name=${m%%:*} flag=${m#*:} + rm -rf "$work/$name" + t=$(now) + $N "$onggi" push --store "$work/$name" $flag --quiet --json --tmp "$work" "$src" > "$work/push-$name.json" + row storage "$name" "$(du -sb "$work/$name" | cut -f1)" "$(since "$t")" +done + +# switching a node from variant A to B: what crosses the network +row transfer oci-layer "$($N zstd -q -3 -T0 -c "$src/$vb" | wc -c)" - +for m in xet:--sub=0 onggi:; do + name=${m%%:*} flag=${m#*:} + for v in a b; do + rm -rf "$work/v$v" && mkdir -p "$work/v$v" + f=$va; [ $v = b ] && f=$vb + ln "$src/$f" "$work/v$v/" 2>/dev/null || cp "$src/$f" "$work/v$v/" + done + rm -rf "$work/t-$name" + "$onggi" push --store "$work/t-$name" $flag --quiet --json --tmp "$work" "$work/va" > /dev/null + "$onggi" push --store "$work/t-$name" $flag --quiet --json --tmp "$work" "$work/vb" > "$work/t-$name.json" + row transfer "$name" "$(json "$work/t-$name.json" "d['stats']['pack_bytes'] if 'stats' in d else d['pack_bytes']")" - + rm -rf "$work/t-$name" "$work/va" "$work/vb" +done diff --git a/docs/evaluation.md b/docs/evaluation.md new file mode 100644 index 0000000..1ffdcea --- /dev/null +++ b/docs/evaluation.md @@ -0,0 +1,43 @@ +# Evaluation + +Measured 2026-10-22 on the lab's RNGD + ATOM node with [`bench/baselines.sh`](../bench/baselines.sh) (#51), over one +Rebellions ATOM Llama-3.1-8B build set: 18 files, 263.9 GB (decode and prefill graphs for several batch sizes and +sequence buckets, each embedding the weights). + +## Storing the set + +| method | stored | vs raw | time | +|---|---|---|---| +| raw | 263.9 GB | 1.00× | — | +| OCI-style layers: whole-file dedup, zstd -3 per file | 199.2 GB | 1.33× | 27.6 min | +| one tar, zstd -3 `--long=31` (2 GiB window) | 199.1 GB | 1.33× | 28.3 min | +| fixed-block dedup, 1 MiB | 116.4 GB | 2.27× | 26 s | +| fixed-block dedup, 64 KiB | 113.4 GB | 2.33× | 27 s | +| FastCDC 64 KiB dedup | 46.8 GB | 5.64× | — | +| Xet-style: FastCDC 64 KiB + per-chunk compression (`onggi --sub 0`) | 29.5 GB | 8.95× | 1.0 min | +| **onggi**: two-level chunking + per-chunk codecs | **17.0 GB** | **15.6×** | 1.3 min | + +Whole-file methods find nothing to share: every file differs. A long-range window of 2 GiB cannot reach the copy of +the weights in the previous 15 GB file. Fixed blocks lose the weights as soon as a header shifts them. CDC finds the +weights; sub-chunks also find the ones prefill graphs hold in a different order (#15). Sizes include each store's +indexes and manifests. + +Xet itself needs the Hugging Face service; the Xet-style row uses the same idea (64 KiB CDC, per-chunk compression) +inside onggi. The compression times run under `nice -n 19` on a shared node. + +## Switching a node from decode B2 to decode B8 + +Bytes that cross the network when a node that holds B2 needs B8 (15.2 GB): + +| method | transferred | +|---|---| +| OCI layer (zstd -3) | 11.46 GB | +| Xet-style CDC + per-chunk compression | **6.1 MB** | +| onggi | 28.3 MB | + +The 218 MB that differ compress well. As 64 KiB chunks they shrink to 6.1 MB; onggi splits new chunks of ATOM files +into ~4 KiB sub-chunks, which compress worse, so it sends 28.3 MB. #52 tracks compressing a chunk's new sub-chunks +together. + +Applying the switch on the node (README, deploy/csi): a plain pull through the node cache takes 73.8 s; delta apply +(#41) takes 5.7 s without and 8.9 s with the node cache, and a CSI pod is Running 12 s after creation.