Skip to content

vulkan: fuse the sign multiply into the Hadamard FWHT - #349

Closed
elliotmasheal wants to merge 1 commit into
PrismML-Eng:prismfrom
elliotmasheal:vulkan/signed-fwht-fusion
Closed

elliotmasheal wants to merge 1 commit into
PrismML-Eng:prismfrom
elliotmasheal:vulkan/signed-fwht-fusion

Conversation

@elliotmasheal

@elliotmasheal elliotmasheal commented Oct 11, 2026 •

Copy link
Copy Markdown

Overview

The Vulkan backend was missing the signed Hadamard fusion that Metal already
has (ggml_metal_op_can_fuse_fwht_signed). On the folded Bonsai graphs the sign
multiply ran as its own elementwise pass over the activation, so every Hadamard
transform paid for an extra write, an extra read, and an extra barrier between
the two dispatches.

This folds MUL(signs) -> RESHAPE -> MUL_MAT(hint=hadamard) into a single
signed FWHT dispatch. The new shader variant reads the sign vector as a third
buffer and applies it while it loads the row, before the transform.

  • vulkan-shaders/fwht.comp: FWHT_SIGNED variant, third buffer for the sign
    vector, broadcast over signs->ne[0].
  • vulkan-shaders-gen.cpp: register fwht_signed_f32.
  • ggml-vulkan.cpp: pipeline_fwht_signed_f32, ggml_vk_can_fuse_fwht_signed
    (subgraph {MUL, RESHAPE, MUL_MAT} plus the hadamard hint, type, contiguity
    and shape checks), the dispatch, and the GGML_OP_MUL branch.

Additional information

Device: AMD Radeon RX 6700S (RDNA2, gfx1032), 8 GB, AMD proprietary driver,
Windows 11. Model: Ternary-Bonsai-2-27B (qwen35, Hadamard block 1024).

Command: llama-bench -m <model> -dev Vulkan1 -ngl 999 -p 512 -n 128 -r 3 -fa 1

model step pp512 tg128
PTQ1_0 baseline 62.48 18.16
PTQ1_0 + PTQ1_0 prefill loader (separate change) 78.80 18.16
PTQ1_0 + this fusion 108.77 24.13
PQ2_0 + this fusion 115.95 22.15
PQ2_0-MTP + this fusion 116.22 22.01

Decode isolates this change: the prefill loader touches only prefill, so
tg128 18.16 -> 24.13 on PTQ1_0 is the fusion alone (PQ2_0 16.88 -> 22.15).
The prefill numbers are the combined run with the loader present; the fusion
also removes the per-transform MUL and barrier in prefill.

Correctness:

  • llama-perplexity -f README.md -c 512 -b 512 --chunks 6 -fa 1 -dev Vulkan1 -ngl 999
    gives 4.2521 +/- 0.29627 for both PTQ1_0 and PQ2_0, identical to the baseline.
  • test-backend-ops -b Vulkan1: 17788/17788 pass (the known
    GATED_DELTA_NET rows_mode cases are skipped as before).

Limits and notes:

  • The fusion needs the subgroup FWHT pipeline; the wide FWHT_SHMEM branch
    falls back to the unfused path, so correctness there is unchanged.
  • The fusion predicate is strict: the MUL output must feed only that RESHAPE.
    When it does not match, the graph falls back to the separate ops.
  • GGML_VK_PERF_LOGGER=1 shows the MUL bucket at ~1.6 ms and the FWHT bucket at
    ~4.8 ms per token, but the observed saving is ~13.9 ms/token. The win is not
    just the elementwise pass: it is also the removed intermediate buffer traffic
    and the barrier between the two dispatches.
  • Measured on one RDNA2 device only; other Vulkan devices need their own numbers.
  • This is rebased onto the current prism. The numbers above were measured on
    the parent revision 2459f68; the intervening commits do not touch the
    Hadamard/FWHT path.

Requirements

  • I have read and agree with the contributing guidelines
  • AI usage disclosure: YES - the implementation and the benchmark write-up were drafted with an AI coding assistant working from my measurements; I reviewed every line, built the branch, and ran the PPL and test-backend-ops gates on hardware.

MUL(signs) -> RESHAPE -> MUL_MAT(hadamard) now runs as one dispatch: the
signed FWHT variant applies the sign vector while it loads the row, so the
separate elementwise pass and the barrier that followed it are gone. Mirrors
the Metal signed FWHT fusion.

Ternary-Bonsai-2-27B on RX 6700S (RDNA2), -dev Vulkan1 -ngl 999 -p 512 -n 128 -r 3 -fa 1:
  PTQ1_0  tg128 18.16 -> 24.13
  PQ2_0   tg128 16.88 -> 22.15
PPL 4.2521 +/- 0.29627 and the full test-backend-ops run are unchanged.
@elliotmasheal

Copy link
Copy Markdown
Author

Folded into #366, which keeps this commit and adds the missing overlap-guard reset. Closing in favour of #366.

bri-prism added a commit that referenced this pull request Oct 16, 2026
* vulkan: fuse the sign multiply into the Hadamard FWHT

MUL(signs) -> RESHAPE -> MUL_MAT(hadamard) now runs as one dispatch: the
signed FWHT variant applies the sign vector while it loads the row, so the
separate elementwise pass and the barrier that followed it are gone. Mirrors
the Metal signed FWHT fusion.

Ternary-Bonsai-2-27B on RX 6700S (RDNA2), -dev Vulkan1 -ngl 999 -p 512 -n 128 -r 3 -fa 1:
  PTQ1_0  tg128 18.16 -> 24.13
  PQ2_0   tg128 16.88 -> 22.15
PPL 4.2521 +/- 0.29627 and the full test-backend-ops run are unchanged.

* vulkan: clear signed FWHT fusion when overlap disables fusion

Reset the signed-FWHT dispatch flag along with the fused-node count when an overlapping buffer requires the unfused fallback. Otherwise MUL still dispatches the fused transform and the graph also executes the original MUL_MAT.

Validated on Apple M5 Pro through MoltenVK with disjoint and partially overlapping buffers, block width 64, activation width 128, and 1/3/5 tokens. All three overlapping cases fail before this change; all six cases pass after it with maximum absolute error below 2.4e-7 against a scalar reference.

---------

Co-authored-by: Elliot <elliotmasheal@outlook.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant