Repository navigation
Ternary decode on 4× Cortex-A78C: row-thread pool only 40–60 % busy, 1.8 tok/s (BitNet 2B4T) — dispatch-bound, not bandwidth-bound #1357
Description
Problem
During BitNet b1.58 2B4T decode on a linux-arm64 JVM with the native kernels loaded, the 4 worker cores are only 40–60 % busy, and decode is 1.8 tok/s. The implied weight traffic is about 1.4 GB/s against roughly 24 GB/s of measured DRAM read bandwidth, so decode is neither bandwidth-bound nor compute-saturated. That points at dispatch and row-thread overhead.
For comparison, on the same cores llama.cpp b9868 decodes the same model (TQ2_0, f16 tied head) at 17.3 tok/s.
Setup
- Arduino Ventuno Q (QCS8275),
docker --cpuset-cpus=0-3(4× Cortex-A78C up to 2.36 GHz), Temurin 21.0.12.1. - SKaiNET 0.61.0, native kernels cross-built for linux-arm64 (see skainet-backend-native-cpu: JVM artifact ships no linux-arm64 libskainet_kernels.so → silent ~1000× reference-kernel fallback on ARM Linux #1356).
skainet-cli -m ggml-model-i2_s.gguf -s 64 -k 0 "The capital of France is", with the two-stage lm_head enabled (BITNET_PLANES, top-200 rescoring).
Measurements
Per-core utilisation from mpstat, sampled every 8 s during decode:
t=49s decode cpu0=56% cpu1=43% cpu2=74% cpu3=61%
t=57s decode cpu0=46% cpu1=39% cpu2=43% cpu3=60%
t=65s decode cpu0=40% cpu1=40% cpu2=44% cpu3=58%
t=73s decode cpu0=42% cpu1=31% cpu2=62% cpu3=40%
The first SKaiNET frame on the main thread, across 12 jcmd Thread.print samples during decode:
| Frame | Samples |
|---|---|
NativeTernaryLmheadKernel.lmheadStage1 |
4 |
NativeTernaryF32GemvKernel.gemvPackedStorage |
4 |
BitNetTwoStageDecode.topK |
2 |
ResidualAdd.onForward |
1 |
JvmVectorKernels.binaryFloat |
1 |
Rebuilding the NeoGPU kernel and its adapter with -mcpu=cortex-a78c instead of the armv8-a pin made no difference (1.82 tok/s).
Questions / suggestions
- Row partitioning and barrier cost in
skainet_row_threads.cfor the many small 2560-wide GEMVs per token (q/k/v/o, gate/up/down × 30 layers). Is one job per GEMV too fine-grained? Could batching q/k/v and gate/up into one pool job help? - Do the FFM downcalls themselves add measurable overhead? 7 of the 10 deep samples were inside
DowncallStub. A per-token batched entry point might help. topKover the 128k-vocab stage-1 scores runs on the JVM main thread while the pool idles. Could it be parallelised or moved native?
Activity
michalharakal commentedon Oct 17, 2026
Two more data points from the same board (Ventuno Q, 4× Cortex-A78C, cpuset 0-3) and the same
ggml-model-i2_s.gguffile:-DBITNET_ARM_TL1=OFFAll three produce the same 32-token greedy continuation as
golden-greedy-2b4t.txt, so parity holds on ARM.The JVM/FFM-overhead hypothesis is refuted. Removing the JVM and FFM entirely (a K/N binary with the C archive linked in, kernels reported as
ternary_f32_gemv/cinteropandternary_planes_matmul/cinterop) makes decode 2× slower, not faster. The C kernels are the same in both builds. What differs is the Kotlin-side per-token work (norms, RoPE, attention, residuals, topK) and allocation.During K/N decode:
That points at per-token allocation. Next I'm profiling allocation for one decode step on the JVM (JFR allocation samples) to find the top allocation sites, and I'll post the results here.
K/N CLI used for this: a small
llm-apps:kbitnet-nativemodule (linuxX64 + linuxArm64 executables, about 60 lines). I can open a PR if useful.michalharakal commentedon Oct 17, 2026
Profile results, JVM decode only. JFR
settings=profilerecorded a 45 s decode window plus-Xlog:gc. Same board, cpuset 0-3, SKaiNET 0.61.0.Where the main thread spends time. JFR samples Java every 10 ms and native every 20 ms; this is weighted accordingly.
NativeTernaryF32GemvKernel.gemvPackedStorageNativeTernaryLmheadKernel.lmheadStage1BitNetTwoStageDecode.topK(full sorts over the 128k vocab)ScalarHeadAttentionKernel.decodeHeadTernaryCodec.decodeBitNetPlanesRowThe implied ternary GEMV rate is about 12 GMAC/s on 4× A78C, with the cores only 40–60 % busy. The native ternary kernels plus row-pool partitioning are the main lever. bitnet.cpp's i2_s path (int8 activations + dotprod) is about 4.8× faster end-to-end on the same file.
Allocation: about 100 MB per decoded token (~190 MB/s; G1 pauses only 4–10 ms, so cheap on the JVM, but this is what hurts K/N). 61 % of it comes from
FloatVector.fromArray0Template←JvmVectorKernels.binaryFloat←DefaultCpuOpsJvm.vectorFloatBinary.binaryFloattakes the op as aFunction2lambda, so C2 doesn't intrinsify the Vector API and runs the Java fallback, which copies arrays and boxesFloat128Vector. Specialising per op (VectorOperators.ADD/MULconstants) should remove most of it. The rest isArrays.copyOf(int[])andInteger.valueOffrom topK's boxed candidate list.Experiment: topK without full sorts. I replaced the sorts with an O(n·k) selection for the k-th lower bound and a primitive
IntArrayfor candidates, keeping the same candidate set (ties go to the lower row).BitNetTwoStageDecodeTestpasses 4/4 and the 128-token greedy output is byte-identical. Decode went from 1.857 to 1.886 tok/s (+1.6 %): the sorts look large in Java-only samples but are a small share of wall time. It's still worth having as a cleanup, and I can PR it to SKaiNET-transformers.Suggested order: (1) ternary GEMV / lm_head stage-1 efficiency and pool partitioning on A78C-class cores; (2) per-op
binaryFloatspecialisation; (3) the topK cleanup.perf(native-cpu): run the ternary f32 GEMV on the shared row pool
michalharakal commentedon Oct 17, 2026
Found the main cause of the 40–60 % core utilisation: the vendored NeoGPU kernel creates and joins 4 pthreads on every call (
dispatch()inhs_ml_ternary_neon.c), about 211 calls per decode step, bypassing the persistent row pool.I benchmarked the kernel alone at the 2B4T shapes on 4× A78C, with weights streamed from DRAM:
So stock kernels alone cap decode at 4.2 tok/s, and pooled kernels at 7.2 tok/s.
Fix: #1359. The adapter drives the vendored proj from
skainet_run_rowsin < 512-row slices, so each slice runs inline and the vendored file stays byte-identical. End to end: 1.90 → 2.26 tok/s (+19 %) with byte-identical greedy output. Also pooling lm_head (it needs an upstream NeoGPU entry point that doesn't spawn threads) gave 2.40 in my experiment.What's left:
binaryFloatVector API fix is the obvious first item there.tbl-based ternary kernel (T-MAC style), or an opt-in int8-activation mode, is the next kernel-side lever.Harness: a single C file that
#includes the vendored source and linksskainet_row_threads.c. Happy to add it underbenchmarks/if useful.