[bri-prism]

What

Extends the opt-in GDN fusions from #383 (GGML_CUDA_GDN_FUSION=1) from one-token decode to speculative verify widths: 1 to 16 tokens of one sequence, with or without rollback snapshots (n_rs_seq > 0, the #358 replay / MTP / DFlash path).

1. Conv chain, 5 kernels to 1 at any width. For an n_t-token batch, a GDN layer runs these kernels:

   • the conv-state GET_ROWS;
   • CONCAT (concat_dim0_transpose);
   • the 1 + n_rs_seq snapshot CPYs (one cpy_f32_batched launch since cuda: batch the rollback conv-state snapshot copies into one launch (+2.6% DFlash2 decode, exact) #379);
   • SSM_CONV+SILU;
   • L2_NORM of the q/k heads.

   ssm_conv_gdn_fused_f32 now takes n_t tokens and a list of up to 16 conv-state rows to write. Each row gets the d_conv-1 conv inputs at its own window offset, exactly as build_conv_state builds the snapshot CPYs. It writes the SiLU output and the L2-normalised q/k heads of every token.

2. Alpha/beta pair at 2-8 columns. The paired mul_mat_vec_f launch now runs at ncols_dst = n_t. This is the same template instance and block size that the cuda: thin F32 mat-muls take mmvf up to 8 columns on Ampere+ (+6% MTP decode on H200) #380 thin-F32 path uses for each product alone. Above 8 columns (where mul_mat_f takes over) the pair is skipped.

3. Narrower "nodes in between" check. With DFlash, the hidden-state taps change the buffer layout. In 6 of 24 GDN layers, the GDN state GET_ROWS output then lands on the dead qkv projection output, and cuda: opt-in exact fusion of the GDN decode conv chain and alpha/beta mat-vecs (GGML_CUDA_GDN_FUSION=1), +5-10% decode on H200 (stacked on #379) #383's check refused the fusion there. The check now protects only what moving the writes earlier requires:

   • no node in between may write or read the early SiLU/L2 outputs;
   • no node in between may write the cache, weights or ids, or read the cache.

   The projection output is read at the CONCAT position in both orders. The CONCAT/SSM_CONV buffers are never written by the fused launch.

Still opt-in under the same env var. If the separate default-on PR for the one-token path lands, it covers these widths too.

Why

Under speculation, GDN is about 7.5% of GPU time, and #383 did not fire there (verify batches are multi-token with snapshots). In a verify forward on H200 (9B target, nsys --cuda-graph-trace=node), the chain was 8 kernels per GDN layer.

How (exactness)

• Conv sum. It uses the same expression and order as ssm_conv_f32: sumf += x[t+j]*w[j] for j = 0..d_conv-1, then + 0.0f (the absent bias).
• L2 norm. Per token, it uses the same per-lane strided partial sums and warp_reduce_sum as l2_norm_f32<WARP_SIZE>, with one warp per token.
• Snapshot writes. Each thread first reads all of its own channel's state and token values into shared memory, then writes its channel's slice of every snapshot row. A snapshot row may be the row that was read, and the SiLU output may be the projection buffer element for element. Both are safe because they only alias within a thread.
• Matcher. It accepts the CPY run only if:
  • each copy is a view of the concat (column offset 0..n_t) into one whole cache row;
  • no row is written twice;
  • the CONCAT has exactly n_cpy + 1 users.
• Scope. NVIDIA only, single sequence, n_t ≤ 16, up to 16 snapshot rows, d_conv 3-5. The new device code uses only the helpers the cuda: opt-in exact fusion of the GDN decode conv chain and alpha/beta mat-vecs (GGML_CUDA_GDN_FUSION=1), +5-10% decode on H200 (stacked on #379) #383 kernel already used on HIP builds (warp_reduce_sum, ggml_cuda_pdl_*, __syncthreads).

Validation (H200, CUDA 12.8, sm_90 Release; base = prism tip)

Per-layer GDN chain kernels, one GDN layer of a verify forward (9B target; median kernel duration in µs over all verify forwards of a 3-prompt × 64-token run; env on, base = tip where #383 does not fire at these widths):

MTP K=3 verify, T = 4 tokens, 4 snapshot rows:

kernel	base µs	this PR µs
k_get_rows_float (conv state)	2.30	fused
concat_dim0_transpose_u32	2.53	fused
cpy_f32_batched (4 snapshot rows)	3.10	fused
ssm_conv_f32 (+silu)	3.10	fused
l2_norm_f32	3.26	fused
ssm_conv_gdn_fused_f32	-	3.68
mul_mat_vec_f<4> alpha / beta	5.02 + 5.95	7.46 (paired)
gated_delta_net_cuda	9.02	8.67
chain total	34.3 (8 kernels)	19.8 (3 kernels)

DFlash, draft 7 verify, T = 8 tokens, 8 snapshot rows:

kernel	base µs	this PR µs
k_get_rows_float	2.30	fused
concat_dim0_transpose_u32	2.69	fused
cpy_f32_batched (8 rows)	4.74	fused
ssm_conv_f32	2.56	fused
l2_norm_f32	3.26	fused
ssm_conv_gdn_fused_f32	-	4.50
mul_mat_vec_f<8> alpha / beta	8.26 + 10.02	11.84 (paired)
gated_delta_net_cuda	15.46	14.42
chain total	49.3 (8 kernels)	30.8 (3 kernels)

Per verify forward:

verify step	kernels/forward	GPU graph span (median)
MTP K=3, T=4	946 → 826	3857 → 3554 µs
DFlash 7, T=8	950 → 830	5470 → 5032 µs

The fused kernel runs in all 24 GDN layers in both cases.

Bit-exactness.

• A standalone ggml program builds the chain exactly as build_conv_state does (snapshot slots and window offsets included) plus an alpha/beta pair at n_t columns. It runs once with the env var at 0 and once at 1, and compares the L2 output, SiLU output, the whole conv cache and both products byte for byte.
• It covers n_t = 1..16 × n_rs_seq ∈ {0, 2, 3, 7, 15} × 5 shapes (d_conv 3/4/5, head 32-128, with the read row among the written rows and not).
• Result: 400/400 identical, and 128/128 again on the final commit. nsys shows the fused kernel and a single paired mul_mat_vec_f in those runs.

test-backend-ops (CUDA0), env 1 and env 0:

op	passed
GDN_CONV_CHAIN (6 → 34 cases: n_t 1-16, n_rs 0/2/3/7/15, short drafts with n_t < 1+n_rs)	34/34
MUL_MAT_VEC_PAIR (9 → 45 cases: 1-9 columns, F32/F16/BF16)	45/45
GATED_DELTA_NET	74/74
SSM_CONV	45/45
CONCAT	177/177
L2_NORM	20/20
GET_ROWS	123/123
CPY	252/252
CPY_SNAPSHOTS	10/10
MUL_MAT (env 1)	1669/1669

nsys shows the fused kernel in every GDN_CONV_CHAIN case.

Output identity. llama-server, greedy, 5 prompts plus 1 copy-heavy prompt, 192 tokens each. Tokens, top-5 logprobs and draft/accepted counts were compared env 1 vs env 0 (same binary) and env 1 vs the prism tip. All configurations below are identical:

target	plain	MTP K=2	MTP K=3	DFlash 7
9B (MTP head)	identical	identical	identical	identical
4B (MTP head)	identical	identical	identical	n/a (no drafter)
27B low-bit, community-grafted MTP head	-	identical	identical	-

#391 interaction.

• One-token decode (9B, tg32, env 1): add_rms_norm_fwht_f32 (both the FFN-input fusion and the layer-start look-ahead) and ssm_conv_gdn_fused_f32 all fire, with the same counts as the tip with env 1 (1320 + 825 fwht-fused launches, 792 conv-chain launches).
• At verify widths, the FFN-input add_rms_norm_fwht_f32 fires as on the tip. The layer-start look-ahead does not fire at these widths on the tip either.
• Greedy output is identical (above).

Speed. llama-server, 9B, same binary with the env var off/on in ABBA order (e0 e1 e1 e0) × 3 rounds per job. Prompts: 4 non-copy (essay/chat/explainer/story) and 2 copy-heavy (repeat a ~90-line Python module with a rename), 192 tokens each. The statistic is the per-prompt median tok/s over 6 runs per arm, then the geometric mean of the on/off ratios. Acceptance is identical on and off.

mode	prompts	job 1: off → on tok/s	job 1	job 2: off → on tok/s	job 2	accept rate
MTP K=3	non-copy	435.4 → 455.6	+4.6%	436.5 → 460.6	+5.5%	0.503 both
MTP K=3	copy	606.9 → 641.7	+5.8%	607.8 → 643.8	+6.0%	0.915 both
DFlash 7	non-copy	315.3 → 326.1	+3.3%	315.6 → 330.6	+4.8%	0.286 both
DFlash 7	copy	731.1 → 759.3	+3.9%	727.2 → 763.1	+4.9%	0.957 both

Per-prompt ratios were 1.014-1.069 in job 1 and 1.046-1.069 in job 2. Job 1 had one stalled request (115 tok/s against a usual ~420 tok/s, in an env-on run). Single stalls like that also hit env-off and tip runs elsewhere, and the per-prompt median absorbs them. Its plain per-round means were noncopy MTP −4.0% (one round 0.77 from that stall), copy MTP +6.5%, and DFlash +3.2% / +4.2%.

Not covered

• Multi-sequence verify (not matched; it runs unchanged).
• Verify batches wider than 16 tokens or with more than 16 snapshot rows (not matched).
• The alpha/beta pair above 8 columns.
• HIP/MUSA: gated off at runtime, and not compiled here (no ROCm toolchain on this machine).
• GPUs other than H200.
• DFlash on the 4B target (no 4B drafter).

Default-on for the one-token path is proposed separately in #398 (Brian decides on default changes).