Repository navigation
cuda: GDN decode conv-chain fusion and alpha/beta mat-vec pair on by default (GGML_CUDA_GDN_FUSION=0 disables), bit-identical, +7.9% 9B decode on H200 - #398
Merged
Conversation
…ir on by default (GGML_CUDA_GDN_FUSION=0 disables) The exact fusion from #383 (one launch for the conv-state gather, concat, state write-back, ssm_conv + silu and q/k L2 norm of a one-token GDN step, and one paired launch for the ssm_alpha/ssm_beta mat-vecs) is now on by default for the one-token path it covers. GGML_CUDA_GDN_FUSION=0 restores the unfused kernels. NVIDIA only, as before.
# Conflicts: # ggml/src/ggml-cuda/ggml-cuda.cu # tests/test-backend-ops.cpp
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
This turns the exact GDN decode fusions from #383 on by default for the one-token path they cover:
ssm_conv+siluand q/kL2_NORMrun as onessm_conv_gdn_fused_f32launch;ssm_alpha/ssm_betamat-vecs run as one pairedmul_mat_vec_flaunch.GGML_CUDA_GDN_FUSION=0disables them, and any other value (or unset) leaves them on. The scope is unchanged: NVIDIA only, one token of one sequence. Only the default and the comments change. There is no kernel or matcher change.Merging a default change is Brian's decision. This PR only puts the evidence in one place.
Why
The fusion is bit-exact and has been opt-in since #383, with these results:
Under MTP/DFlash it does not fire (verify steps are multi-token), so those modes are unaffected by this PR.
Evidence
From #383 (env 0 vs 1, H200):
#383 also reported:
On this branch, at the default setting (no env var), H200, CUDA 12.8, sm_90 Release; base =
prismtip:test-backend-ops (CUDA0), default:
=0)=0)The default really is on. nsys of 9B tg32:
ssm_conv_gdn_fused_f32launches (33 steps × 24 GDN layers), and nol2_norm_f32/concat/ssm_conv_f32;=0: 792 each ofl2_norm_f32,ssm_conv_f32andconcat_cont, and no fused kernel.Greedy identity. llama-server, 5 prompts × 192 tokens, tokens and top-5 logprobs, default vs
GGML_CUDA_GDN_FUSION=0(same binary) and default vs theprismtip (old default, off). All identical:=0Speed. 9B, llama-bench tg128
-fa 1r=5, default vs=0in ABBA order, 3 rounds: 353.0 → 381.0 tok/s, ratios 1.079 / 1.079 / 1.081 (+7.9%).Not covered