Skip to content

cuda: GDN decode conv-chain fusion and alpha/beta mat-vec pair on by default (GGML_CUDA_GDN_FUSION=0 disables), bit-identical, +7.9% 9B decode on H200 - #398

Merged
bri-prism merged 2 commits into
prismfrom
cuda/gdn-fusion-default
Oct 25, 2026
Merged

bri-prism merged 2 commits into
prismfrom
cuda/gdn-fusion-default

Conversation

@bri-prism

@bri-prism bri-prism commented Oct 25, 2026 •

Copy link
Copy Markdown
Collaborator

What

This turns the exact GDN decode fusions from #383 on by default for the one-token path they cover:

  • the conv-state gather, concat, state write-back, ssm_conv+silu and q/k L2_NORM run as one ssm_conv_gdn_fused_f32 launch;
  • the ssm_alpha/ssm_beta mat-vecs run as one paired mul_mat_vec_f launch.

GGML_CUDA_GDN_FUSION=0 disables them, and any other value (or unset) leaves them on. The scope is unchanged: NVIDIA only, one token of one sequence. Only the default and the comments change. There is no kernel or matcher change.

Merging a default change is Brian's decision. This PR only puts the evidence in one place.

Why

The fusion is bit-exact and has been opt-in since #383, with these results:

Under MTP/DFlash it does not fire (verify steps are multi-token), so those modes are unaffected by this PR.

Evidence

From #383 (env 0 vs 1, H200):

model kernels/token, off → on tok/s off → on (tg128, depth 4096, two ABBA jobs)
A (smallest) 690 → 600 634.0 → 696.3 / 633.1 → 696.5
B - 394.1 → 422.5 / 394.3 → 429.5
C (9B) 941 → 821 277.2 → 293.2 / 275.1 → 294.3
D (27B) 1512 → 1272 141.0 → 151.5 / 141.7 → 149.6

#383 also reported:

  • bit-identical outputs from a standalone byte-for-byte check (6 shape sets);
  • test-backend-ops unchanged (GATED_DELTA_NET, SSM_CONV, SSM_CONV_BIAS_SILU, CONCAT, L2_NORM, GET_ROWS, CPY, CPY_SNAPSHOTS, SILU, MUL_MAT 1630/1630);
  • greedy tokens and top-5 logprobs identical on all four sizes, plus MTP K=2.

On this branch, at the default setting (no env var), H200, CUDA 12.8, sm_90 Release; base = prism tip:

  • test-backend-ops (CUDA0), default:

    op passed
    GDN_CONV_CHAIN 6/6 (also 6/6 with =0)
    MUL_MAT_VEC_PAIR 9/9 (also 9/9 with =0)
    GATED_DELTA_NET 74/74
    SSM_CONV 45/45
    CONCAT 177/177
    L2_NORM 20/20
    GET_ROWS 123/123
    CPY 252/252
    CPY_SNAPSHOTS 10/10
    MUL_MAT 1669/1669
  • The default really is on. nsys of 9B tg32:

    • default: 792 ssm_conv_gdn_fused_f32 launches (33 steps × 24 GDN layers), and no l2_norm_f32/concat/ssm_conv_f32;
    • =0: 792 each of l2_norm_f32, ssm_conv_f32 and concat_cont, and no fused kernel.
  • Greedy identity. llama-server, 5 prompts × 192 tokens, tokens and top-5 logprobs, default vs GGML_CUDA_GDN_FUSION=0 (same binary) and default vs the prism tip (old default, off). All identical:

    target default vs =0 default vs tip
    2B identical identical
    4B identical identical
    9B identical identical
    27B (format 1) identical identical
    27B (format 2) identical -
  • Speed. 9B, llama-bench tg128 -fa 1 r=5, default vs =0 in ABBA order, 3 rounds: 353.0 → 381.0 tok/s, ratios 1.079 / 1.079 / 1.081 (+7.9%).

Not covered

…ir on by default (GGML_CUDA_GDN_FUSION=0 disables)

The exact fusion from #383 (one launch for the conv-state gather, concat, state write-back, ssm_conv + silu and q/k
L2 norm of a one-token GDN step, and one paired launch for the ssm_alpha/ssm_beta mat-vecs) is now on by default for
the one-token path it covers. GGML_CUDA_GDN_FUSION=0 restores the unfused kernels. NVIDIA only, as before.
# Conflicts:
#	ggml/src/ggml-cuda/ggml-cuda.cu
#	tests/test-backend-ops.cpp
@bri-prism
bri-prism merged commit bf8c069 into prism Oct 25, 2026
6 of 7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant