Repository navigation
qwen27 CUDA: a pre-sm_90 decode matmul, ~52% faster serving on an RTX 3090 (exact; stacked on #3) - #4
Open
CanadaApollo6 wants to merge 6 commits into
Conversation
…m_90 (no clusters)
… captured unaligned variant The shared-round staging packed each part back to back, so _gdn_pre's WIN pointer landed off 16 bytes at some window sizes and no captured variant matched. Each part now starts on 16 bytes, and an aligned pointer or int can fall back to a variant captured without the alignment hint (never the reverse).
… host token table by env The server sized the scratch for 16 streams of 16 rows at any --parallel, and the attention partials in it grow with the window: on a 24 GB card no 32K stream fit beside it. The scratch now takes `streams` windows (the lane core's admission and the cost curve follow), so --parallel 1 needs one stream's rows. - TF_QWEN27_PROMPT_ROWS: a prompt chunk's rows (default 4096; 1024 frees ~1 GiB at a ~10% slower prompt). - TF_QWEN27_HOST_EMBED=1: the token table in mapped host memory (0.67 GiB; only prompt rows read it).
_tail's STREAM pointer followed W i32 rows directly, so it landed off 16 bytes when W % 4 != 0 (a lone stream's small windows) and matched no captured variant. Rows, streams, items and parents each start on 16 bytes.
…ghts sharing an input Before sm_90, decode matmuls take qmm_lane.cu: the weights as the mma's A operand, so 1-16 rows fill a tile, and each tile's K slices added in slice order, the bits of 0.6.6's qmm.cu on the same card. Weights over one input (GDN qkv/z/b/a, attention q/k/v, gate/up) share one segmented launch, each at its own K split, so every output equals its lone launch. Wider rounds take row tiles of 16-64, 128 rows a turn.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #3 (RTX 3090 support), which is stacked on ashhart#608. GitHub can't target a fork's branch, so this diff also shows #3's five commits. This PR's change is the last commit,
083532cf(5 files, +637 −18). Once #3 lands, the diff shrinks to that commit.On an RTX 3090 it makes served decode ~52% faster with identical tokens. Every row's logits are byte-equal to 0.6.6's.
What changes
qmm_lane.cu: a pre-sm_90 decode matmul. With Qwen3.8-27B on CUDA: RTX 3090 (sm_86) support, the scratch sized by --parallel #3, sm_86 runs split-K matmuls aslane_gemv, 16 rows a launch.qmm_laneputs the weights in the mma's A operand, so 1 to 16 rows fill a tile. Each tile's K slices are added in slice order, either in one block (2 or 4 slices side by side) or by the tile's last block. That is the order 0.6.6'sqmm.cuuses on the same card. Wider rounds take row tiles of 16, 32 or 64, at most 128 rows a launch.Ops.denseManyruns GDN's qkv/z/b/a, attention's q/k/v and the MLP's gate/up as one segmented launch for each K split they share. Each weight keeps its own split, so every output equals its lone launch.qmm_frag.cuh. sm_90 and newer, andTF_QWEN27_GEMV_ROWS, keep their current paths.Checks (RTX 3090, CUDA 13.4, driver 615.71, the same 0.6.6 glue set as #3)
tf-qwen27-cuda windowsagainst a freshcapture_windows.pycapture from 0.6.6 on the same card--widedoesn't fit 0.6.6 on 24 GB.run(serial)draftmulti --context 8192draftwithTF_QWEN27_HOST_EMBED=1 TF_QWEN27_PROMPT_ROWS=1024zig build testtools/zig/lean_check.pyServed decode, before (#3) and after, alternated
tensorfold-native serve --context 32768 --parallel 1with the DFlash2 drafter.tools/bench_openai.py --tokens 256 --reps 3 --temperatures 0, run as A, B, A, B on one machine:fibonacci-raw)gpu-chat-no-think)The token hashes of every reply are the same in all four runs.
Not checked: sm_80/sm_89 cards (I have neither). On sm_90 and newer the code path is unchanged.
Not in this PR
A rule that runs shared-input weights unsplit once they fill the card is worth another ~2-3% on the 3090. It changes K splits, and so the bits, so it needs a gate per CONTRIBUTING. I'll raise it with the 4-bit cache issue.
🤖 Generated with Claude Code