[lalalune]

What

A denser code layout for Phonon-2's five-value encoder weights. The GGUF shrinks from 330.6 MB to 208.9 MB (−37%), with bit-identical results.

	fv5-pair16-v1 (current)	fv5-b5x3t7-v1 (this PR, default)
bits per five-value weight	4.00	2.35 (K=4096) – 2.41 (K=1024)
phonon-2-fv5.gguf	330,632,256 B	208,932,352 B
deflate -9 (APK asset) / zstd -19	210.6 / 199.8 MB	171.6 / 170.0 MB
mapped page cache on the tablet (peak RssFile)	324 MB	205 MB
private copy (ELIZA_ASR_USE_MMAP=0, peak RssAnon)	404 MB	272–286 MB

Why the GGUF was bigger than the published 164 MB

• The 164 MB on Hugging Face is phonon-2.bps.tar.zst. It unpacks to model.fermion, 177.4 MB.

• In model.fermion, the 604M five-value weights take 159.4 MB (2.11 bits/weight):

  • sign trits packed five per byte (1.6 bits);
  • plus one hi/lo bit per non-zero weight (49% are non-zero).

  That bit stream has variable length, so a kernel cannot start decoding at an arbitrary block without prefix counts.

• fv5-pair16-v1 gave every weight a 4-bit code, so 302 MB of the 331 MB GGUF were codes. The rest is the int8 decoder tables (22.4 MB), F32 lo/hi (3.5 MB) and small F32 tensors.

• For reference: log2(5) = 2.32 bits, and the measured level entropy is 1.95 bits (levels {−hi, −lo, 0, +lo, +hi} = 9.1/15.3/51.3/15.3/9.1%).

Layout (pk-kernels.h, convert_phonon2_to_gguf.py pack_b5x3t7)

• Per 8-row block, take the K/2 16-byte panel vectors of the int8 GEMM three at a time. Lane j of a code vector = l(3t) + 5·l(3t+1) + 25·l(3t+2), which is below 125.

• Because codes fit in 7 bits, 8 code vectors go into 7 stored vectors: byte m = code m | (bit m of code 7) << 7.

• The decoder makes exactly the panel that pair16 makes, with the same 16-byte LUTs. Per code vector it runs:

  • x/5 = ((x+1)·51) >> 8: 2 UMULL + UZP2, exact on 0..124 (checked at bench start);
  • /25 by a 32-byte TBL;
  • two MLS;
  • 3 × (ORR + TBL).

  Code 7 costs a 7-step SRI gather.

• The AVX2 tier does the same with PMULLW/PACKUSWB/PSHUFB. The scalar tier is the reference.

• The loader accepts both formats (parakeet.weights.format). Tensors are named <w>.fv5t7, I8 [O/8, block bytes].

• The converter defaults to --layout b5x3t7, and --layout pair16 is still there. --verify re-expands either layout against the container bit for bit. Output is byte-reproducible (two runs: same sha256).

• pins.json pins the new GGUF (sha256 c0f6ad42…) and records the pair16 output of the same converter.

Alternatives measured with a prototype (same bench)

• 3 levels per byte, 2.67 bpw (230 MB): decodes at the same speed as the 7-bit packing on the A76 (666 vs 667 ns/block) and 15% faster on the M4. The 7-bit packing saves another 21 MB, so it was chosen.
• Fermion's variable-length layout: ruled out, because it needs per-block prefix counts of non-zeros.

Decode cost (tools/parakeet/tools/bench-fv5-packing.cpp, new)

Single thread, decode + the shipped tile per 8-row block, summed over the 24 layers' linears (ms per encoder pass). N = tokens (80 ms of audio each):

N	8	16	32	64	125	375
TB336FU Cortex-A76 (dotprod), pair16 → b5x3t7	101 → 121 (+20%)	176 → 191 (+8%)	324 → 330 (+2%)	596 → 613 (+3%)	1139 → 1142 (0%)	3661 → 3602 (−2%)
M4 Max (i8mm), pair16 → b5x3t7	25 → 33	45 → 52	85 → 91	167 → 171	321 → 326	957 → 960

• Decode alone, per 8-row block, K=1024:
  • A76: 386 → 663 ns;
  • M4: 89 → 167–174 ns;
  • Rosetta AVX2: 305 → 1055 ns (sanity check only, not x86 performance evidence).
• The extra cost is fixed per encoder pass, about 20 ms of single-thread time on the A76. So it shows only on very short windows: about 0.64 s of audio at N=8.

Equivalence evidence

• test-parakeet-kernels: new checks.
  • Independently repacked codes decode to the pair16 panel on every tier, for K = 48, 80, 256, 1024 and 4096 (so the short last group and the staging path are covered).
  • gemm_q8 is bit-identical for both layouts.
  • Passes on: Mac (scalar, dotprod, i8mm), the tablet (scalar, dotprod), and x86_64 under Rosetta (scalar, avx2).
• convert_phonon2_to_gguf.py --verify: OK for both layouts (264 five-value + 11 int records bit-identical, 723 records consumed, mel filterbank equal to librosa 0.11.0).
• parakeet-cli, 10 clips (5 say voices + 5 Paradee samples), batch and --stream:
  • On ARM, the transcripts are byte-identical between the two layouts, with each of i8mm, dotprod and scalar.
  • On x86 avx2 (Rosetta), the two layouts are also byte-identical to each other. WER is therefore unchanged by construction.
• eliza-inference-test-parakeet-asr on the TB336FU (NDK r29, arm64, Release): passes with both GGUFs, matching the exact expected transcript, mapped (5 runs each) and private (3 runs each).
  • Warm batch times overlap: 1.03–1.53 s for b5x3t7 vs 1.06–1.51 s for pair16, on 8.15 s of audio, with the tablet's load average around 17 from other work.

Not covered

• test-parakeet-parity was not run: it needs the reference fixtures from dump_parakeet_reference.py, which need torch. It is implied by the bit-identical panels and transcripts above.
• No APK was built with this GGUF, and no in-app run was done.
• Unrelated, pre-existing, not changed here: ARM and x86 builds give slightly different transcripts on 3 of 10 clips (casing, punctuation, one word), with either layout. That comes from fp32 paths, not from the weights.

Downstream

• A runtime built before this PR rejects the new GGUF with a clear error ("parakeet.weights.format is neither …"). So the GGUF and the runtime must move together.
• Builders that pin the GGUF's sha256 (eliza packages/app/platforms/android/paradee-voice/inputs.json, voice-model-versions.json, Grace voice.lock.json) need the new size and sha256.

AI-assisted change.

🤖 Generated with Claude Code