[92es]

Increases -sm tensor prompt eval t/s by +19% t/s ( making -sm tensor as fast as -sm layer ) with zero PPL change

Overview

• increases -sm tensor` prompt-eval t/s: [+16.7% to +20.6%] (mean ≈ +19.0%) faster than stock, on every dense model tested
• tensor split prompt eval is now as fast as layer split prompt eval with this optimization patch
• PPL: no change in either split mode (tensor/tensor, layer/layer)

Additional information

• NOTE: PPL is inherently higher in tensor split mode than layer split in the base code, thus PPL has to be A/B compared with tensor/tensor or layer/layer, NOT tensor/layer. The same delta appears in both the mainline build and the patch build.

• NOTE: This a complementary patch with sycl: add set_tensor_2d for faster tensor-split loading #30358 authored by @Captain-Tripps . This patch picks up tensor split mode optimizations where his hands off (load => runtime). They do NOT depend on each other, but complement each other.

• NOTE: Related PR SYCL: reduce tensor allreduce sync with pinned host buffers #29604

• UPDATE: First pair of waits replaced with barriers to restore MTP and remove host.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES

Collab: google.ai and Qwen3.8-27B-UD-Q8_K_4XL
Code gen: Qwen3.8-27B-UD-Q8_K_4XL

Test environment

GPUs: Dual B70s, one B70 in Gen5 x16 PCIe slot and one in Gen4 x4 PCIe slot.
OS: Debian Trixie Backports, Kernel 7.9.2
Toolkit: oneAPI 2026.1.1

Test command

llama-perplexity -m ${LLM_NAME}.gguf -dev SYCL0,SYCL1 -f ggml-sycl.cpp -sm [tensor|layer] -ts 1,1 -ngl 99 --load-mode dio --chunks 12 -n 64 -b 1024 -ub 1024 -c 512 -v
Retested with single shot llama-cli to validate MTP consistency using --spec-draft-n-max 4, 5 and 6

Out of tree unit test showed Gen5x16 to Gen4x4 pathway increased from 5.3 GB/s ( mainline code ) to 7.1 GB/s ( +34% patch code ), almost maxing out the theoretical limit of 7.5.GB/s.

Test matrix

• stock layer vs stock tensor
• patch layer vs patch tensor
• stock layer vs patch layer
• stock tenser vs patch tenser

Tensor mode — patch gain over mainline

Model	stock → patched t/s	Δ t/s (%)
Qwen3.8-27B-Q8_0 (A)	1065.18 → 1277.07	+211.89 (+19.9%)
Qwen3.8-27B-Q8_0 (B)	1075.98 → 1277.58	+201.60 (+18.7%)
Qwen3.8-27B-UD-Q8_K_XL	1077.49 → 1287.47	+209.98 (+19.5%)
Qwen3.8-27B-UD-Q8_K_4XL	1100.91 → 1325.57	+224.66 (+20.4%)
Qwen3-4B-Q8_0	3984.03 → 4650.81	+666.78 (+16.7%)
Qwen3.8-27B-BF16	1169.55 → 1410.09	+240.54 (+20.6%)
Mistral-Small-24B-Q8_0	1530.72 → 1792.46	+261.74 (+17.1%)

Consistent tensor-mode gain of +16.7% … +20.6% (mean ≈ +19.0%) across every model and
quantization tested.

llama-perplexity PPL ( no change layer vs layer and tensor vs tensor )

Layer mode: stock vs patched PPL (no change)

Model	stock PPL	patched PPL	Δ patch−stock
Qwen3.8-27B-Q8_0 (A & B)	2.5204 ± 0.09071	2.5204 ± 0.09071	+0.0000
Qwen3.8-27B-UD-Q8_K_XL	2.5205 ± 0.09070	2.5205 ± 0.09070	+0.0000
Qwen3.8-27B-UD-Q8_K_4XL	2.5209 ± 0.09071	2.5209 ± 0.09071	+0.0000
Qwen3-4B-Q8_0	3.8538 ± 0.20216	3.8538 ± 0.20216	+0.0000
Qwen3.8-27B-BF16	2.5207 ± 0.09070	2.5207 ± 0.09070	+0.0000
Mistral-Small-24B-Q8_0	2.6652 ± 0.09276	2.6652 ± 0.09276	+0.0000

Tensor mode: stock vs patched PPL (no change)

Model	stock PPL	patched PPL	Δ patch−stock
Qwen3.8-27B-Q8_0 (A)	2.5214 ± 0.09080	2.5214 ± 0.09080	+0.0000
Qwen3.8-27B-Q8_0 (B)	2.5210 ± 0.09077	2.5213 ± 0.09079	+0.0003
Qwen3.8-27B-UD-Q8_K_XL	2.5209 ± 0.09075	2.5213 ± 0.09077	+0.0004
Qwen3.8-27B-UD-Q8_K_4XL	2.5213 ± 0.09073	2.5214 ± 0.09075	+0.0001
Qwen3-4B-Q8_0	3.8531 ± 0.20205	3.8531 ± 0.20205	+0.0000
Qwen3.8-27B-BF16	2.5214 ± 0.09078	2.5220 ± 0.09081	+0.0006
Mistral-Small-24B-Q8_0	2.6649 ± 0.09276	2.6648 ± 0.09274	−0.0001

UPDATE:

Thanks to @Captain-Tripps eval, he discovered there was a MTP degradation with the original patch that used wait().

Switching the first pair of wait() to ext_oneapi_submit_barrier() restored MTP consistency between mainline and patch.

ext_oneapi_submit_barrier() retained the patch's speed improvements and PPL equivalencies as previously documented, confirmed by an entire rerun of the regression test suite tabled above.

Mainline vs Patch wait vs Patch barrier

	CPU memcpy	Buffers	Mid-fn host block	Cross-queue sync	MTP impact
Baseline	2× serial	4	yes (D2H)	none needed (CPU did the swap)	stalls submission thread
Patch wait	none	2	yes (D2H) ← problem	host wait + resubmit	degraded (heartbeat stall + dead gap)
Patch barrier	none	2	no — final wait only, post-enqueue	device-side barrier, in-order queues	clean; pipeline stays full

[92es]

@Captain-Tripps

I did not find a comment section on your B70/MTP page so I am posting it here.

You had some good open questions to resolve.

I am making a hands-off scripted llama-cli script to regression test MTP. Here is the initial A/B run of a fixed 3,254 token prompt (C++ source code) that runs and exists without manual intervention to eliminate manual run-2-run variations.

My initial manual llama-cli MTP tests used much shorter prompts.

Preliminary results look promising:

Performance Comparison: Stock vs. Patched

A/B (hands off) scripted llama-cli Qwen3.6-27B-Q4_0-MTP.gguf on SYCL0, SYCL1

Both runs processed a 3,265 token prompt and 256 generation tokens.

Metric	Stock	Patch barrier	Change / Delta
Prompt Processing Speed	679.50 t/s	746.32 t/s	+9.83%
Prompt Processing Time	4805.02 ms	4374.81 ms	-8.95% (Faster)
Generation Speed	29.57 t/s	31.25 t/s	+5.68%
Generation Time	8622.43 ms	8159.23 ms	-5.37% (Faster)
Draft Acceptance Rate	76.13% (118/155)	80.00% (132/165)	+3.87%
Mean Draft Length	2.74 tokens	3.13 tokens	+14.23%
MTP Draft Gen Duration	871.44 ms	856.30 ms	-1.74% (shorter)
MTP Mean Acceptance	2.74	3.13	+14.2%

MTP

acc rate	positions
Stock	(0.882, 0.338, 0.235, 0.132, 0.074, 0.074)
Patch	(0.952, 0.419, 0.306, 0.210, 0.129, 0.113)

Notes

~10% improvement in prompt evaluation speed (Time-to-First-Token)
~5.7% lift in continuous generation speed.

[92es]

@Captain-Tripps

"... If you test the barrier approach where it does run, please check correctness with perplexity at ub 4 and ub 512 plus a long greedy decode against a stock build (a write-after-read race on reused staging buffers would show up there, not in a short MTP run), and post your OS, driver, and whether q0->get_context() == q1→get_context(). ..."

Requested Information:

The barrier patch runs with no crashes.

OS: Debian Trixie Backports, Kernel 7.2.9
oneAPI 2026.1.1 oneDNN

q0 and q1 get_context() are ALWAYS the same.

6145 tokens for all 4 runs with 60 second cool downs between runs.

Note: There are run-2-run +/- variance on both stock and patched binaries.

-dev SYCL0,SYCL1 -f ggml-sycl.cpp -sm tensor -ts 1,1 -ub 4
PPL 2.5196 +/- 0.09057 - Stock
PPL 2.5209 +/- 0.09065 - Patch barrier
Stock 74.91 t/s vs Patch 72.36 t/s

-dev SYCL0,SYCL1 -f ggml-sycl.cpp -sm tensor -ts 1,1 -ub 512
PLL 2.5217 +/- 0.09076 - Stock
PPL 2.5216 +/- 0.09075 - Patch barrier
Stock 1028.55 t/s vs Patch 1186.21 t/s

[92es]

@Captain-Tripps ,

I added "-seed 1234" to try and stabilize the +/- PPL variance in both the STOCK and PATCH runs, but the run-2-run variance still remains in both.

This run, the PATCH barrier's PPL is better than the STOCK with both -ub 4 and 512 where last night was better with just 512.

I think we are chasing random run-2-run noise levels ( and random seed mixes ? ).

Multiple interleaved hands off -seed 1234 -ub 4 runs:

-dev SYCL0,SYCL1 -f ggml-sycl.cpp -sm tensor -ts 1,1 -seed 1234 -ub 4

PPL 2.5203 +/- 0.09064 - Stock
PPL 2.5209 +/- 0.09064 - Stock
PPL 2.5210 +/- 0.09069 - Stock
PPL 2.5203 +/- 0.09062 - Stock

PPL 2.5196 +/- 0.09056 - Patch barrier
PPL 2.5189 +/- 0.09052 - Patch barrier
PPL 2.5202 +/- 0.09062 - Patch barrier
PPL 2.5194 +/- 0.09054 - Patch barrier

Tokens per second Stock 75.16 t/s vs Patch 72.24 t/s
Tokenization Stock 76.185 ms vs Patch 75.963 ms

-dev SYCL0,SYCL1 -f ggml-sycl.cpp -sm tensor -ts 1,1 -seed 1234 -ub 512

PPL 2.5218 +/- 0.09077 - Stock
PPL 2.5214 +/- 0.09074 - Patch barrier
Tokens per second Stock 1029.96 t/s vs Patch 1184.14 t/s
Tokenization Stock 75.431 ms vs Patch 76.777 ms

As for your Windows crash, make sure code is q0:d2h_1 and q1:d2h_0.

    q0->ext_oneapi_submit_barrier({e_d2h_1}); // q0 waits for q1's device-to-host copy
    q1->ext_oneapi_submit_barrier({e_d2h_0}); // q1 waits for q0's device-to-host copy

[Captain-Tripps]

Hi @92es -

Here is what we tested out:
q0->ext_oneapi_submit_barrier({e_d2h_1}) and q1->ext_oneapi_submit_barrier({e_d2h_0}), on 2x Arc Pro B70 running Windows 11 Pro, Level Zero driver 32.0.101.8976, with Qwen3.8-27B in Q8_0 and -sm tensor. This ran on my fork (r67c with #30956 ported by hand)

Here's what we found:
Unfortunately, the process aborts before producing any results, a fast-fail (0xc0000409) inside ze_intel_gpu64.dll. The stock exchange in the same build runs fine (pp512 971 t/s), so it’s the barrier that's the issue.

Why it fails here:
For windows, each device gets its' own SYCL context, so q0->get_context() != q1->get_context(), and a barrier on one queue can’t take an event from the other context. On Linux, the contexts are the same, which is why it works there.

What this likely means for this PR:
It would probably crash Windows multi-GPU users. A guard, using the barrier only when q0->get_context() == q1->get_context() and otherwise the current host-wait path, keeps the Linux speedup (+15% at ub 512, ub-4 perplexity identical) without breaking Windows.

What would fix it:
We believe that putting both cards in one SYCL context (like how Linux does it). That’s on our roadmap of next things to try, we haven't built it yet and aren't sure we'd see any additional speed up in PP or decode.

[92es]

Patch Update:

Updated the patch to react to SYCL_ENABLE_DEFAULT_CONTEXTS settings to switch between the original 4-buffer-wait version (separate SYCL contexts) and the new 2-buffer-barrier version (same SYCL context) using a function pointer to remove the runtime check out of the hot path as per @Captain-Tripps lead.

kl-divergence

Also ran llama-perplexity's --kl-divergence-base --kl-divergence to verify PPL and KLD consistency. -ub 4 was identical and -ub 512 oscillated back and forth in noise levels between baseline code and patch barrier (again, as per @Captain-Tripps lead).