[eduralph]

Overview

Since #29029 (0f8a414), Mistral Small 4 token generation on Vulkan dropped from ~41 t/s to ~3.4 t/s. Prompt processing is unaffected.

#29029 sets GGML_PREC_F32 on src1 of the Mistral 4 ffn_moe_down MUL_MAT_ID, and the Vulkan supports_op now returns false for any MUL_MAT_ID with that flag. That includes the mat-vec path used for token generation, so every layer's down projection goes to the CPU and the graph is split per layer. GGML_VK_PERF_LOGGER confirms the Q5_K/Q6_K MUL_MAT_ID ops are missing from the Vulkan timings.

The rejection is needed for the mat-vec path as it is: with n=1/4/8 added to the existing "src1 f16 overflow" tests, Vulkan returns NaN for Q4_K and Q5_K at amax=1e5/3e5. The cause is MMVQ (src1 quantized to q8_1). With GGML_VK_DISABLE_INTEGER_DOT_PRODUCT=1 all of these cases pass. Q6_K and Q8_0 pass either way.

This PR:

• skips MMVQ in ggml_vk_mul_mat_vec_id_q_f16 when the op has GGML_PREC_F32 set
• only rejects flagged MUL_MAT_ID in supports_op when it would not use the mat-vec path (batched cases still fall back as before)
• adds n=1/4/8 and Q5_K/Q6_K cases to the src1 f16 overflow tests

Additional information

Hardware: AMD Ryzen AI Max (Strix Halo), Radeon 8060S (gfx1151), RADV / Mesa 26.0.8, Linux 7.0, uma: 1, int dot: 1.
Model: unsloth/Mistral-Small-4-119B-2603-GGUF UD-Q4_K_XL (gate_up_exps Q4_K, down_exps Q5_K/Q6_K).

llama-bench -ngl 99 -fa 1 -p 512 -n 32:

build	pp512	tg32
fb92d8f (before #29029)	334	41.2
5cdadfc (master)	319	3.4
this PR (llama-server, same model)	-	39.5

Found with git bisect on a 4-layer cut of the same model (199 vs 28 t/s; 199 t/s with this PR).

test-backend-ops -o MUL_MAT_ID -b Vulkan0 and -o MUL_MAT -b Vulkan0 pass, including the new cases. Output of the full model with this PR is coherent at temp 0.

CUDA has the same unconditional rejection in supports_op and is probably affected in the same way. I can't test CUDA, so this PR only changes Vulkan.

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES. Claude Opus 5.5 (Claude Code) did the bisect, profiling and test runs, and wrote the patch and this description. I reviewed the change and the results.

[0cc4m]

this is generally valid, I've got a larger branch I'm working on for all precision changes. But it's okay to do this separately first.

[0cc4m]

@ggerganov @CISC https://github.com/ggml-org/llama.cpp/actions/runs/38710930783/job/116359711341?pr=30544 Do you know what caused this? I don't think it was this PR.

test-llama-context

/opt/actions-runner/_work/llama.cpp/llama.cpp/ggml/src/ggml-backend.cpp:1793: ggml_backend_sched_alloc_splits: unexpected graph reallocation (graph size = 139, nodes = 139, leafs = 46), debug_realloc = 1

[CISC]

@ggerganov @CISC https://github.com/ggml-org/llama.cpp/actions/runs/38710930783/job/116359711341?pr=30544 Do you know what caused this? I don't think it was this PR.

I think it is, as it happens on mistral4.

[jeffbolznv]

This change is very similar to #30105. My feedback on that one applies here as well.

[0cc4m]

You mean the feedback about adding a function to get the src1_prec value?

[jeffbolznv]

Yes, and Georgi's reply about explicitly handling all prec enums in a switch statement.

[eduralph]

Looks like I opened a can of worms. Fixing that one spot wasn't enough. On GPUs with fp16 support, the batch kernels hold their inputs as f16 in shared memory (the GPU's fast scratch memory) even when the input is F32. Values above 65504, the largest f16 number, turned into NaN, which is why the n ≥ 16 cases failed. Full-F32 versions of the same kernels (*_fp32) were already compiled, but only loaded on GPUs without fp16.

Changes (ggml-vulkan.cpp, ggml-vulkan-types.h):

• New key flag: the kernel lookup key gets an fp32 flag.
• Full-F32 kernels on every device: the batch MUL_MAT_ID kernels with F32 inputs are now registered with that flag on every device, coopmat2 included. Each tile size is kept only if it fits in shared memory at 4 bytes per value.
• Batch path: when src1 precision is F32 or BF16, it uses those kernels with F32 accumulation. It never quantizes to Q8_1 or takes Intel's f16 shortcut. Non-contiguous src1 is now copied to the kernel's input type; before, it was always copied to f16.
• supports_op: the answer no longer depends on batch size, which is what fixes the CI crash. F32-precision MUL_MAT_ID is rejected only if the device lacks the full-F32 kernels, and then it goes to the CPU at every batch size. BF16 weights fall in that case, since no BF16×F32 kernel exists.
• Cleanup: the Q8_1 checks in the mat-vec path now share a small helper.

Some caveats

• gpu-vulkan-nvidia-cm2: coopmat2 is the case I couldn't test locally. It now runs the plain kernels for this op instead of sending it to the CPU.
• gpu-vulkan-intel-linux and gpu-vulkan-apple: also not tested here.
• ubuntu-llvmpipe and gpu-vulkan-nvidia-cm: these should pass, since they're closest to what I tested.

[0cc4m]

#30656 should cover these cases in a much more performant way.

[eduralph]

@0cc4m - does that mean this should be closed instead of merged?

[0cc4m]

Yes, the primary issue has been solved by #30861. #30656 updates the Vulkan precision handling. GGML_PREC_F32 for activations stays unsupported, but also unused.

[eduralph]

Superceeded by #30861 and #30656 updates