[lalalune]

Summary

This adds tools/parakeet, a native CPU runtime for FermionResearch/Phonon-2, and serves it through the existing eliza_inference_asr_* surface in libelizainference, including the streaming session (asr_stream_open/feed/partial/finish).

Phonon-2 is nvidia/parakeet-tdt-0.6b-v3 (FastConformer + TDT, English) with every encoder linear retrained to five values per row ({-hi,-lo,0,+lo,+hi}). It downloads as 164 MB. There is no ONNX, MLX, Core ML or Python at run time, and no new ABI symbols and no version bump. The base is now at v18, and the behaviour notes sit under that changelog entry.

Licence (flagged, not a blocker).

• The weights are CC-BY-4.0, derived from NVIDIA parakeet-tdt-0.6b-v3. Attribution and Fermion's NOTICE are required.
• The converter embeds the NOTICE in the GGUF (parakeet.notice, also available from pk_model_notice()).
• According to the NOTICE, the training audio includes CHiME-6 (CC-BY-SA-4.0) and SPGISpeech (Kensho terms).
• Fermion's Python helpers are Apache-2.0. None of Fermion's code is vendored: their fast CPU kernels ship only as closed binaries, and everything here is new.

What is in it

Converter. convert_phonon2_to_gguf.py reads the pinned phonon-2.bps.tar.zst and checks it against pins.json (HF revision 7e153e40…, sha256 of every input and archive member).

• The 264 five-value linears are kept exact as 4-bit level codes, stored in the order the kernel reads them (fv5-pair16-v1). The GGUF is mmapped and used in place.
• The int6 tables become exact int8 plus scales. BatchNorm is folded into the depthwise conv.
• --verify re-expands every record bit for bit against the container, checks that all 723 records are consumed, and checks that the mel filterbank matches librosa's. The output is 330 MB, deterministic for the same inputs.

Kernels. A per-8-row panel is decoded with one TBL per 16 weights, then multiplied with:

• SMMLA (i8mm), 8x8 tiles;
• SDOT (dotprod), on straight and half-swapped token pairs;
• a scalar reference, for x86_64 and old ARM.

Each NEON tier is its own translation unit with its own -march, entered after an HWCAP/HWCAP2 or sysctl check. All tiers are bit-identical.

Model. The encoder and decoder are plain C++ on a persistent worker pool. On Android the pool pins to the big cores; on Apple it uses QoS user-interactive.

• Activations entering a linear are quantised to int8 per token.
• The relative-position keys are cached per layer in fp16, computed with two-pass int8.
• TDT greedy decode follows NeMo's rules, with at most 10 symbols per frame.
• Word times come from the TDT token frames (80 ms).

Streaming. Each decode covers a pause-delimited window.

• The model needs this: the dense fp32 reference also drops or invents words when its input starts mid-word.
• Tokens are committed on two-decode agreement, on word boundaries, at least 800 ms behind the newest sample.
• A new segment starts in a ≥150 ms pause between tokens.
• Windows are capped at 12 s, and the decode interval grows with the window.

libelizainference (src/eliza-asr-parakeet.inc plus small #ifdef ELIZA_ENABLE_PARAKEET hooks).

• mmap_acquire("asr") loads a GGUF under <bundle>/asr/ whose general.architecture is parakeet-tdt. Otherwise the Qwen3-ASR path runs unchanged. ELIZA_ASR_BACKEND=qwen3|parakeet overrides the choice.
• asr_transcribe_timed returns real TDT word times.
• asr_stream_supported() is 1 while a context has the backend loaded.
• Stream token ids are not returned (*io_n_tokens = 0), because the vocabulary is not the text model's.
• asr_cancel is polled between encoder layers and decode steps.
• mmap_evict("asr") frees the model, and an open stream then fails cleanly.
• Android's no-mmap ASR policy (ELIZA_ASR_USE_MMAP) applies.
• pk_* symbols are not exported.

Review round (4efc306)

An adversarial review found these problems, now fixed:

• Transcript text in logs. PK_STREAM_DEBUG printed raw token pieces and tentative text. It now prints window geometry and token counts only. The FFI test no longer prints transcripts either.

• Feed/partial race. asr_stream_feed called from an audio thread raced partial/finish on the stream's buffer (use-after-free on vector growth). Feed now only resamples into a pending buffer under a per-stream mutex and never waits for a decode; partial/finish drain it under asr_mutex. Feed after finish fails. The new concurrent test (a feeder thread racing partial polls) passes under ThreadSanitizer with no reports.

• Windows grew through pauses. The last word before a pause waited for the next word and pinned the segment start. A 10 s utterance followed by 40 s of quiet re-decoded windows up to 41 s (about 1.5 s of compute each). Now:

  • the newest word commits once right_context passes with no further token;
  • with nothing pending and only quiet audio since the last decode, the decode is skipped and the window keeps 1.5 s of the pause;
  • a window past max_window with no token keeps its newest half.

  On the same clip that is 22 decodes and 2.2 s of compute instead of 32 and 13.5 s, and the stream no longer invents words in the pause. An intermediate fix that cut idle windows short produced "Okay. Yeah. Yeah." on 1.4 s noise windows, because the feature normalisation is per window. Skipping decodes on quiet audio avoids that.

• Loader. gguf_get_val_* aborts on a type mismatch, so value types are now checked first. An empty or negative TDT duration table is rejected (a negative duration looped forever), and so is blank_id != vocab.

• FFI. The model generation counter is now atomic, since contexts bump it under different mutexes. eliza_free_asr dereferenced ctx before its NULL check. Partials truncate on a UTF-8 boundary.

• Parity gate. An empty fixture set passed; it now fails. A pooled WER against the reference transcripts (PK_GATE_REF_WER, default 0.02) bounds how far the utterances that differ may drift, on top of the "90% identical" gate.

• Speed. The position-key cache now computes only the new rows when it grows. The result is bit-identical (rows are independent, and every transcript is unchanged), and a stream with a growing window no longer rebuilds all 24 layers every 64 frames.

  • On M4 the GEMM is at the hardware limit. Raw SMMLA peaks at about 5.4 G/s (≈172 GMAC/s) and SDOT at about 12 G/s (≈191 GMAC/s). The tile reaches 180–210 GMAC/s, and the full GEMM including decode reaches 180–186 GMAC/s. No kernel change can buy much on this host.
  • Phones with i8mm (Cortex-X/A7xx) may have more SMMLA headroom; measure that on a device.

Fresh numbers (this head, same subsets, Apple M4 Max as an ARM CPU proxy, not phone evidence; host load average 30–58)

1 thread (CPU time)	LibriSpeech 78 utt / 610 s	commands 20 / 30.5 s
RTFx	20.1–22.6x (base run interleaved: 21.7–22.6x)	16.5–20.4x
median first-1 s decode	52–60 ms	52–65 ms
WER (Open ASR normaliser)	4.31	4.0
identical to the dense fp32 reference	70/78	20/20
peak RSS	468–498 MB (330 MB mmapped GGUF)	355–358 MB

These differ from the table below (4.14 / 67/78). That table came from an older head; these come from this head and the previous one, run back to back.

At 4 threads the wall-clock RTFx was 21–72x on LibriSpeech and 26–66x on commands, with the median first-1 s decode at 17–31 ms. The spread comes from contention on the shared host.

Streaming, 1 thread, CPU.

• Commands: stream WER 4.0, equal to batch on 20/20, 0.16x real time.
• 16 long LibriSpeech utterances: stream WER 5.45 against batch 4.55, 0.24x real time.
• First partial decode, median: 26 ms.
• Identical to the previous head on these speech sets, so there is no regression.

Parity.

• 26/26 fixtures have identical text and token ids, and WER against the reference is 0.0000.
• Encoder relative L2 is at most 0.156; with PK_DEBUG_F32_ACTS, at most 0.0007.
• Mel max |diff| is 3.3e-4.

Android NDK r29 (android-28), build evidence only.

• Both arm64-v8a and x86_64 build cleanly, including libelizainference.so, parakeet_lib and the parakeet tests. No warnings come from the new code.
• The arm64 .so contains smmla and sdot and exports eliza_inference_asr_stream_*.
• No pk_* symbols are exported.
• Nothing was run on an emulator or a device.

Measurements (earlier head)

All figures come from an Apple M4 Max used as an ARM CPU proxy. This is not phone evidence. The host was shared and heavily loaded (load average 60–280 on 16 cores from other jobs). That makes wall-clock numbers unreliable, so single-thread speed uses process CPU time, minimum of 2–3 repeats.

The engines were run back to back on the same inputs: the 78-utterance LibriSpeech subset (610 s, test-clean + test-other, seed 1234) and 20 synthetic voice commands (30.5 s). WER uses the Open ASR Leaderboard normaliser.

1 thread	this PR	Fermion's closed CPU engine (fermion-research 0.2.11, i8mm)
RTFx, LibriSpeech (CPU time)	17.4x	15.1x
RTFx, commands (CPU time)	16.3x	13.5x
median CPU per command	95 ms	112 ms
first-window partial (decode of the first 1 s, median CPU)	69 ms	72 ms
WER LibriSpeech (clean / other)	4.14 (2.10 / 6.18)	4.14 (2.10 / 6.18)
WER commands	4.0	4.0
transcripts identical to the dense fp32 reference (reference_transformers.py, WER 4.26)	67/78 LS, 20/20 commands	67/78 LS, 20/20 commands
peak RSS	487 MB (330 MB of it is the mmapped GGUF)	–

Comparison with earlier evaluation runs on the same subset.

• whisper.cpp: tiny.en ran at 9.2x with 9.45% WER, and small.en at 1.2x with 5.25% WER.
• ggml parakeet.cpp proof of concept with exact five-value weights: 4.5x. This PR is about 3.9x faster.

Multi-thread. I have no meaningful numbers from this host. At 4 threads the load average was 85–215, so each thread got a fraction of a core.

• On LibriSpeech, wall-clock RTFx was 5.2x for this PR against 2.2x for Fermion.
• On commands, interleaved runs gave 4.5x / 3.8x for this PR against 2.2x / 3.5x for Fermion.

The pool now yields after a short spin, so a preempted worker is not starved. Scaling has to be measured on a phone's big cores.

Streaming (defaults, 1 thread, 16 long LibriSpeech utterances + 20 commands).

• Commands: the streamed transcript equals the batch transcript on 20/20.
• LibriSpeech: stream WER 5.00 against batch 4.55.
• Stream compute is about 0.33x real time (CPU) at one thread, about 5x the batch cost, because windows are re-decoded.
• The first partial is available after 0.4 s of audio.
• ELIZA_ASR_STREAM_{RIGHT,HOP,MAX_WINDOW}_MS trade cost against accuracy. For example, hop=800 gives about 0.25x real time with stream WER 5.45.

Silence. 3 s of digital silence and of -60 dB noise both transcribe to an empty string, with no VAD needed. Qwen3-ASR returns "The." on silence.

Kernel throughput (one P-core, CPU time).

• i8mm tile: 150–165 GMAC/s.
• Full GEMM including decode: 130–145 GMAC/s.
• dotprod: 126–138 GMAC/s.
• scalar: 5 GMAC/s.

Tests

• test-parakeet-kernels: dotprod and i8mm are bit-identical to scalar for the panel decode, every tile width, the matvec and the whole GEMM driver across thread counts. The GEMM matches an exact five-value product within 3e-4. Passing.
• test-parakeet-parity model.gguf fixtures/: the fixtures come from dump_parakeet_reference.py, which runs Fermion's own dense fp32 reference.
  • Shipped int8-activation path: transcripts and token ids are identical on 26/26 fixtures (6 LibriSpeech, 20 commands). Mel max |diff| is 3.3e-4. Encoder relative L2 is 0.07–0.16 and mean frame cosine at least 0.989. This is the int8-activation noise floor.
  • PK_DEBUG_F32_ACTS=1 (same graph and weights, fp32 activations), run on all 26 fixtures: encoder relative L2 is at most 0.0007 on LibriSpeech and at most 0.0105 on commands, mean frame cosine at least 0.99996, and 26/26 transcripts are identical. The remainder comes from the fp16 key cache and the int8 grid of the five-value levels. This confirms the graph matches the reference, so the int8 path's deviation is activation quantisation alone.
• eliza-inference-test-parakeet-asr runs through the real libelizainference: probe 0 → 1 → 0 across acquire and evict; batch equals timed equals expected; word count matches a whitespace split and word times are monotonic; a 48 kHz stream gives partials and a final transcript; asr_cancel from another thread returns ELIZA_ERR_CANCELLED; the evicted-stream error and re-acquire both work. Passing. It skips unless ELIZA_TEST_PARAKEET_BUNDLE/_PCM are set.
• eliza-inference-test-abi: passes, unchanged.
• Android NDK r28c (android-28) build evidence only. elizainference, parakeet_lib and the parakeet tests build for arm64-v8a and x86_64. The arm64 libelizainference.so contains the smmla and sdot paths and exports the asr_stream_* symbols. Nothing was run on an emulator or a device.

CI. Python type-check, flake8 Lint, model-naming and the wasm build pass.

• ubuntu-cpu, ggml-ci-*, ubuntu-24-webgpu, ubuntu-latest-rpc and windows-latest-hip fail. The base head 79b9437 fails the same jobs (plus more), so these failures are not from this change.
• I could not read the job logs here because of the GitHub API rate limit.
• Locally, everything new compiles warning-free with the CI warning set:
  • GCC 15 and Apple clang, both with -Wall -Wextra -Wpedantic -Wcast-qual -Wmissing-declarations -Wextra-semi;
  • the scalar paths with GCC;
  • the x86_64 paths with NDK clang;
  • the FFI translation unit with ELIZA_ENABLE_PARAKEET under GCC 15.

Not done

• Device evidence. RTFx at 1 thread and on the big cores, first partial, RSS and thermals on a Pixel, plus microphone acceptance. Host numbers do not stand in for these.

• eliza-side wiring. This needs:

  • a registry row in the mobile voice bundle (feat(voice): pinned mobile voice bundle (Paradee + Qwen3-ASR + Silero), AOSP/bionic TTS on Paradee, native WER gates eliza#36157 owns voice-model-versions.json);
  • publishing the converted GGUF to a pinned repo: phonon-2-fv5.gguf, 330 MB, sha256 dc552eb7…, byte-reproducible and pinned in pins.json. Its compressed size was not measured; the information content is that of the 164 MB release;
  • JNI streaming methods (feat(android): RegisterNatives JNI v17 with Paradee TTS, ASR cancel and the paradee-voice library eliza#36168 owns ElizaVoiceNative);
  • a submodule bump after this merges.

  The Qwen3-ASR Q8_0 pair it would replace maps about 1.0 GB.

• x86_64 speed. The scalar tier runs at about 0.6x real time, which is slow on x86_64 emulators. An AVX2 tier would fix that.

• Activation precision. Per-group activation scales would bring the int8 path closer to the reference, at about 10–15% speed. WER already equals Fermion's engine and beats the dense reference on this subset, so I left it out.

Coordination. #64 (Paradee), #63 (misaki) and #56 (ABI v18) have merged since this branch started. The current base is merged in, both sets of CMake hooks are kept, and the host FFI tests pass on the merged tree.

🤖 Generated with Claude Code