Repository navigation
vulkan: fuse the sign multiply into the Hadamard FWHT - #349
Closed
elliotmasheal wants to merge 1 commit into
Closed
elliotmasheal wants to merge 1 commit into
elliotmasheal wants to merge 1 commit into
Conversation
MUL(signs) -> RESHAPE -> MUL_MAT(hadamard) now runs as one dispatch: the signed FWHT variant applies the sign vector while it loads the row, so the separate elementwise pass and the barrier that followed it are gone. Mirrors the Metal signed FWHT fusion. Ternary-Bonsai-2-27B on RX 6700S (RDNA2), -dev Vulkan1 -ngl 999 -p 512 -n 128 -r 3 -fa 1: PTQ1_0 tg128 18.16 -> 24.13 PQ2_0 tg128 16.88 -> 22.15 PPL 4.2521 +/- 0.29627 and the full test-backend-ops run are unchanged.
Author
bri-prism
added a commit
that referenced
this pull request
Oct 16, 2026
* vulkan: fuse the sign multiply into the Hadamard FWHT MUL(signs) -> RESHAPE -> MUL_MAT(hadamard) now runs as one dispatch: the signed FWHT variant applies the sign vector while it loads the row, so the separate elementwise pass and the barrier that followed it are gone. Mirrors the Metal signed FWHT fusion. Ternary-Bonsai-2-27B on RX 6700S (RDNA2), -dev Vulkan1 -ngl 999 -p 512 -n 128 -r 3 -fa 1: PTQ1_0 tg128 18.16 -> 24.13 PQ2_0 tg128 16.88 -> 22.15 PPL 4.2521 +/- 0.29627 and the full test-backend-ops run are unchanged. * vulkan: clear signed FWHT fusion when overlap disables fusion Reset the signed-FWHT dispatch flag along with the fused-node count when an overlapping buffer requires the unfused fallback. Otherwise MUL still dispatches the fused transform and the graph also executes the original MUL_MAT. Validated on Apple M5 Pro through MoltenVK with disjoint and partially overlapping buffers, block width 64, activation width 128, and 1/3/5 tokens. All three overlapping cases fail before this change; all six cases pass after it with maximum absolute error below 2.4e-7 against a scalar reference. --------- Co-authored-by: Elliot <elliotmasheal@outlook.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
The Vulkan backend was missing the signed Hadamard fusion that Metal already
has (
ggml_metal_op_can_fuse_fwht_signed). On the folded Bonsai graphs the signmultiply ran as its own elementwise pass over the activation, so every Hadamard
transform paid for an extra write, an extra read, and an extra barrier between
the two dispatches.
This folds
MUL(signs) -> RESHAPE -> MUL_MAT(hint=hadamard)into a singlesigned FWHT dispatch. The new shader variant reads the sign vector as a third
buffer and applies it while it loads the row, before the transform.
vulkan-shaders/fwht.comp:FWHT_SIGNEDvariant, third buffer for the signvector, broadcast over
signs->ne[0].vulkan-shaders-gen.cpp: registerfwht_signed_f32.ggml-vulkan.cpp:pipeline_fwht_signed_f32,ggml_vk_can_fuse_fwht_signed(subgraph
{MUL, RESHAPE, MUL_MAT}plus the hadamard hint, type, contiguityand shape checks), the dispatch, and the
GGML_OP_MULbranch.Additional information
Device: AMD Radeon RX 6700S (RDNA2, gfx1032), 8 GB, AMD proprietary driver,
Windows 11. Model: Ternary-Bonsai-2-27B (qwen35, Hadamard block 1024).
Command:
llama-bench -m <model> -dev Vulkan1 -ngl 999 -p 512 -n 128 -r 3 -fa 1Decode isolates this change: the prefill loader touches only prefill, so
tg128 18.16 -> 24.13on PTQ1_0 is the fusion alone (PQ2_016.88 -> 22.15).The prefill numbers are the combined run with the loader present; the fusion
also removes the per-transform MUL and barrier in prefill.
Correctness:
llama-perplexity -f README.md -c 512 -b 512 --chunks 6 -fa 1 -dev Vulkan1 -ngl 999gives 4.2521 +/- 0.29627 for both PTQ1_0 and PQ2_0, identical to the baseline.
test-backend-ops -b Vulkan1: 17788/17788 pass (the knownGATED_DELTA_NET rows_modecases are skipped as before).Limits and notes:
FWHT_SHMEMbranchfalls back to the unfused path, so correctness there is unchanged.
When it does not match, the graph falls back to the separate ops.
GGML_VK_PERF_LOGGER=1shows the MUL bucket at ~1.6 ms and the FWHT bucket at~4.8 ms per token, but the observed saving is ~13.9 ms/token. The win is not
just the elementwise pass: it is also the removed intermediate buffer traffic
and the barrier between the two dispatches.
prism. The numbers above were measured onthe parent revision 2459f68; the intervening commits do not touch the
Hadamard/FWHT path.
Requirements