From c2312f9f516f5c5f4420cc1650a4dd6886240d10 Mon Sep 17 00:00:00 2001 From: bri-prism <288398250+bri-prism@users.noreply.github.com> Date: Tue, 13 Oct 2026 18:19:41 -0700 Subject: [PATCH] docs: Ternary Bonsai 2 on Intel GPUs with the SYCL backend Switches and their per-GPU defaults after #356/#358, B390 speeds, and a KLD check against the CPU backend. Credits the arc-b580 port and libxsmm/TernSYCL. --- docs/backend/SYCL-RELEASE.md | 2 ++ docs/backend/SYCL-bonsai.md | 64 ++++++++++++++++++++++++++++++++++++ 2 files changed, 66 insertions(+) create mode 100644 docs/backend/SYCL-bonsai.md diff --git a/docs/backend/SYCL-RELEASE.md b/docs/backend/SYCL-RELEASE.md index a27a94494595..d86f31835b01 100644 --- a/docs/backend/SYCL-RELEASE.md +++ b/docs/backend/SYCL-RELEASE.md @@ -27,4 +27,6 @@ The SYCL archive includes the oneAPI 2025.3.3 runtime DLLs used by the upstream Check that the Intel GPU appears before running inference. Use a model supported by the fork's SYCL backend, and verify output correctness as well as speed. Report the model filename, release tag, GPU, OS, driver and command when reporting a problem. +For the Ternary Bonsai 2 models (switches, expected speed, a KLD check), see [SYCL-bonsai.md](SYCL-bonsai.md). + For installation and backend options, see https://github.com/PrismML-Eng/llama.cpp/blob/prism/docs/backend/SYCL.md. diff --git a/docs/backend/SYCL-bonsai.md b/docs/backend/SYCL-bonsai.md new file mode 100644 index 000000000000..f4d664640b3a --- /dev/null +++ b/docs/backend/SYCL-bonsai.md @@ -0,0 +1,64 @@ +# Ternary Bonsai 2 on Intel GPUs (SYCL) + +How to run the PQ2_0 and PTQ1_0 Ternary Bonsai 2 models on Intel Arc and Core Ultra GPUs with the SYCL backend, which switches matter, and how to check the result. + +The Intel-specific paths come from Jesse Symons' [Torchit1/llama.cpp `arc-b580`](https://github.com/Torchit1/llama.cpp/tree/arc-b580) work (#356, #358). The int2 x int8 DPAS kernels are [libxsmm/TernSYCL](https://github.com/libxsmm/TernSYCL) by Evangelos Georganas, Alexander Heinecke and the libxsmm team, vendored under `ggml/src/ggml-sycl/ternsycl/` with its BSD-3-Clause license. + +## Get it + +- Prebuilt: the `win-sycl-x64` and `ubuntu-sycl-*` archives from a [Prism release](https://github.com/PrismML-Eng/llama.cpp/releases). See [SYCL-RELEASE.md](SYCL-RELEASE.md) for how to install them (on Windows the SYCL archive goes on top of the CPU archive). +- From source: follow [SYCL.md](SYCL.md); the Bonsai paths need no extra CMake options. + +Models: [prism-ml/Ternary-Bonsai-2-27B-gguf](https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf) (`PTQ1_0` is the smallest; `PQ2_0` is the same weights in a 2-bit layout). + +## Run + +```sh +llama-server -m Ternary-Bonsai-2-27B-PTQ1_0.gguf -ngl 99 -fa on -ctk q4_0 -ctv q4_0 -c 16384 +``` + +A q4_0 KV cache is what the XMX decode attention reads directly; with an f16 cache the regular attention path is used. + +## Switches + +All are environment variables. Defaults depend on the GPU: Xe2 is Arc B-series and Lunar Lake, Xe3 is Panther Lake and Wildcat Lake. + +| Variable | Default | Effect | +|---|---|---| +| `GGML_SYCL_PTQ1_T2` | off | `all` (or `ffn`) repacks PQ2_0/PTQ1_0 weights to 2-bit in place for the TernSYCL kernels. Xe2 and newer. Faster prompt processing; on Xe3 single-token decode is slower than the default path. | +| `GGML_SYCL_DISABLE_XMX` | `1` on Xe3, `0` elsewhere | `0` keeps PQ2_0/PTQ1_0 on the XMX weight layout, `1` uses the MMVQ reorder layout. Ignored for weights taken by `GGML_SYCL_PTQ1_T2`. | +| `GGML_SYCL_FA_DEC_DPAS` | `1` on Xe3, `0` elsewhere | XMX decode attention from a q4_0 KV cache for up to 256 query tokens (`N` sets another cap). Xe2 and newer. | +| `GGML_SYCL_QUANT_W8A8`, `GGML_SYCL_QUANT_S4_GEMM` | `1` | oneDNN int8 GEMM for large prompt batches. | +| `GGML_SYCL_ENABLE_FUSION` | `1` | Fused norm, rotation and gate kernels. Individual fusions also have their own `*_OFF` switches. | +| `SYCL_UR_USE_LEVEL_ZERO_V2` | `0` (set by the library) | Set `1` to use the newer Level Zero adapter. | +| `LLAMA_GDN_REPLAY` | on when one sequence and all recurrent layers are on SYCL | Speculative decoding rolls the delta-net state back by replay instead of snapshots. Needs `-np 1` on llama-server. | +| `LLAMA_MTP_KV_ONLY`, `LLAMA_SYNC_TRIM` | on with a SYCL device | MTP prompt pass for the draft head writes K/V only; fewer context synchronizations. | +| `GGML_SYCL_OP_TIMING` | `0` | `1` prints per-op GPU times. | + +## Expected speed + +Arc B390 (Xe3, Windows 11, driver 32.0.101.8860), Ternary Bonsai 2 27B PTQ1_0, `llama-bench -fa 1 -ctk q4_0 -ctv q4_0`, tokens/s: + +| | pp512 | tg128 | tg128 at 16K | +|---|---|---|---| +| defaults | 353 | 15.8 | 14.8 | +| `GGML_SYCL_PTQ1_T2=all` | 490 | 14.5 | 13.8 | + +Use `GGML_SYCL_PTQ1_T2=all` for long prompts and the defaults for chat-length prompts. Other Intel GPUs have not been measured with these defaults; please report numbers. + +## Check correctness + +Compare against the CPU backend with KL divergence on a short wikitext run: + +```sh +llama-perplexity -m Ternary-Bonsai-2-27B-PTQ1_0.gguf -f wiki.test.raw --chunks 16 -c 512 -b 256 -ub 256 \ + --device none --kl-divergence-base base.kld +llama-perplexity -m Ternary-Bonsai-2-27B-PTQ1_0.gguf -f wiki.test.raw --chunks 16 -c 512 -b 256 -ub 256 \ + -ngl 99 -fa on -ctk q4_0 -ctv q4_0 --kl-divergence-base base.kld --kl-divergence +``` + +Use `-b 256 -ub 256`: larger batches exceed the XMX decode attention's 256-token cap, so it would not be tested. On the B390 the defaults give mean KLD about 1.6e-3 and the same top token 98% of the time against a reference with the same q4_0 KV cache; the KV cache quantization adds to that against the CPU reference. + +Op-level checks: `test-backend-ops -b SYCL0 -o MUL_MAT`, `-o FLASH_ATTN_EXT`, `-o GATED_DELTA_NET`. + +When reporting a problem, include the release tag or commit, GPU, OS, driver, the command and any switches set.