Repository navigation
ggml-cpu: no repack/GEMM path for PTQ1_0 on x86, CPU prompt processing runs at decode speed #324
Description
Prerequisites
- Latest code:
prism@ 6bfcd79. Numbers below are from 2459f68; no x86ggml-cpufile changed between the two. - Searched issues and PRs (PTQ1_0 CPU / prefill / repack / gemm): nothing open or closed for a PTQ1_0 CPU repack or GEMM. ggml-cpu: add x86 AVX-VNNI dot product for PTQ1_0 #181 (closed) and ggml-cpu : AVX-512 VNNI and AVX2 vec_dot for PTQ1_0 on x86 (4x token generation) #250 (open) are batch-one vec_dot.
Feature Description
On CPU, PTQ1_0 has only a row-by-row dot: nrows = 1, Q8_0 activations, and no entry in repack.cpp / arch/x86/repack.cpp. Prompt processing therefore runs one batch-one dot per token, and each packed block is decoded again for every token. Prefill runs at about decode speed.
i5-13450HX (AVX2 + AVX-VNNI, no AVX-512), WSL2, CPU-only Release build with GGML_NATIVE, 8 threads, llama-bench -p 512 -n 0 and -p 0 -n 32, -r 1 (single runs: direction only, not a benchmark claim). Profile: perf record -e cpu-clock (WSL2 has no hardware counters).
| model / build | pp512 t/s | tg32 t/s | top symbol in pp512 |
|---|---|---|---|
Ternary-Bonsai-2-27B PTQ1_0, prism (SSE dot from #248) |
1.95 | 1.59 | ggml_vec_dot_ptq1_0_q8_0 98.1% |
| same, + #250 AVX2/VNNI dot (local rebase) | 3.92 | 3.21 | ggml_vec_dot_ptq1_0_q8_0 96.6% |
same weights requantized to PQ2_0, prism |
5.32 | 5.15 | tinyBLAS_PQ2K_AVX::gemm<2,4> 95.0% |
Qwen3-0.6B Q8_0, prism (-r 3, reference) |
373 | 69.7 | - |
Rough throughput (2 x matmul params x pp t/s): PTQ1_0 + #250 about 200 GOPS, Q8_0 about 560 GOPS. So about 2.8x headroom on this CPU, with the caveat that the 0.6B model has a smaller working set.
test-quantize-perf --op vec_dot_q, single thread, cycles per 32 values: ptq1_0 (#250) ~3.95, q8_0 ~2.6-3.1. Trit unpack is about 35-40% of the PTQ1_0 dot. Removing only the repeated unpack caps the gain near 1.5x; the rest needs register tiling with weight and activation reuse, as in the Q8_0/Q4_0 GEMMs.
Motivation
- CPU-only users of the released PTQ1_0 Bonsai models get prefill at decode speed.
- Small verify batches (2-8 columns) for MTP/DFlash speculative decoding: multi-column PTQ1_0 paths exist for Metal (metal: process five PTQ1 rows in dense mat-vec #225, metal : PTQ1_0 multi-column mat-vec for 5-8 columns #277), CUDA (cuda: gate (SwiGLU) fused PTQ1_0 mat-vec for 2-4 columns #310, cuda: BATCH_INVARIANT: occupancy-independent FA KV split, PTQ1_0 mat-vec up to 8 columns #322) and Vulkan (vulkan: dedicated mul_mat_vec shader for PTQ1_0 (~3x on the dequant path) #252), not for CPU.
- Q1_0 (ggml-cpu: AVX2 / AVX-VNNI GEMV+GEMM kernels for Q1_0 (4x8 repack) + shared-scale Q8 activations #233) and PQ2_0 (ggml-cpu: AVX2/AVX-VNNI kernels for PQ2_0 + Q8_K activations + tiled GEMM (x86 without AVX-512) #206, ggml-cpu : PQ2_0 AVX-512 VNNI gemm on 8-row tiles (+30-40% prompt processing) #256) already have CPU repack/GEMM paths. PTQ1_0 does not.
Possible Implementation
x86 AVX2/AVX-VNNI first, in the pattern of #233: repack into interleaved blocks at load, GEMV + GEMM in arch/x86/repack.cpp, current vec_dot stays the fallback for other ISAs and shapes. #250's dot is the batch-one baseline, so this would come after #250 lands. Tests through existing test-backend-ops and test-quantize-fns, no new files under tests/.
Questions before I write code:
- Load-time repack (load time and memory cost on a 27B model) or an on-the-fly 2-4 column tile with no repack?
- Q8_0 or Q8_K activations?
- The PQ2_0
gemm<2,4>only reaches pp ~ tg on this CPU, so I would target the Q8_0 GEMM numbers, not PQ2_0. Does that match your view?
AI was used to help write the profiling scripts and to summarize the measurements; I ran and checked the numbers myself.
Activity
matheus-c-feitosa commentedon Oct 8, 2026
Two more data points, same
prismcommit6bfcd79, Ternary-Bonsai-2-27B (b072e1d),-t 2, single runs on shared machines (orders of magnitude):PTQ1_0:llama-perplexityover 2 × 512 tokens ran at ~0.5 t/s (1024 tokens in ~2048 s), the same as decode (0.44–0.51 t/s). So the missing prefill path shows up on ARM too, not only on x86.llama-completion:PTQ1_0prompt eval 0.31–0.34 t/s vs decode 0.31–0.33;PQ2_0on the same box 1.84–2.16 vs 0.86–0.87. (Not a long-prompt prefill measurement, but the same direction.)That fits
ggml_repack_get_optimal_repack_typeat6bfcd79(ggml/src/ggml-cpu/repack.cppL5226–L5456): there's noPTQ1_0entry for any CPU, andPQ2_0's repack is only selected on AVX-512+VNNI (L5448), so neither machine above used a repack for either type.