Qwen3.8-27B on CUDA: image input and --kv-cache fp8 (256K with vision on a 5090) by BobClawblaw · Pull Request #726 · ashhart/TensorFold

BobClawblaw · 2026-10-12T08:56:31Z

Image input for Qwen3.8-27B on CUDA, plus an fp8 KV cache (--kv-cache fp8) that reaches the full 262,144-token window with the image tower loaded, one stream on a 32 GB RTX 5090, and a host-memory prompt cache so a chat's next turn resumes its kept state.

Stacked on #608, #598 and #614 (merged in as the first commits) and merged with main as of 2984ea01 (1.0.11) in 4e15fae5 (1.0.9 in e3b72103, 1.0.10 in 72bf3994). The new work is six commits:

fa22819f Qwen3.5 vision tower on CUDA: qwen_vision.cu kernels, the Tower, and its check against 0.6.6
407cb284 Qwen image preprocessing on Linux: stb_image decode, EXIF, alpha, smart_resize, PIL bicubic, patches
ff7cc227 Native image input for Qwen3.8-27B on CUDA: tower rows spliced into the prompt pass, M-RoPE positions
5d8bf485 --kv-cache fp8, and attention partials sized for the admitted streams
06cc8115 A prompt cache, so a chat's next turn resumes its kept state
d49a906f No tooling that imports the Python engine in the tree

fp8 KV cache

Attention caches are stored as e4m3 rows with one fp32 scale per (position, key head) row, set to amax / 448. That is half the bytes of bf16.
tf_kv_quant quantizes rows on commit and at prompt time.
Prefill attention uses pattn8_kernel.
Decode uses the _shared8/_tail8 Triton kernels, captured outside the tree like the rest of the set. _tail8 quantizes the window's own rows exactly as their commit will. A key's bits therefore never depend on windows, drafts or batching.
bf16 stays the default and its bits are unchanged. Exactness vs 0.6.6: serial 4/4, windows 54/54, drafted 4/4, shared rounds 0 replies differ.
Attention partials are no longer part of the scratch arena. They are sized after admission for the streams actually admitted, which frees 1.4 GiB at 122,880 tokens and 2.9 GiB at 262,144.

Prompt cache

families/qwen27/cuda/reuse.zig saves a stream's state at a prompt mark into a pinned cuda.HostBuffer: each DeltaNet layer's recurrence and conv window, plus each attention layer's key and value rows up to the mark (and their fp8 scales). That is about 275 MiB at 4,096 tokens. Host memory is used because a long window's caches leave the device no room for copies.
It plugs into the shared prompt-cache store through a Snaps vtable in native/cuda.zig. The cache size is --prompt-cache-gib, 8 GiB by default, capped at half of MemAvailable.
While the cache is on, prompts are prefilled on a zero-anchored 4,096-row grid so a chat's marks line up between turns. On resume, the drafter's context is rebuilt from the new suffix only. Image prompts are neither kept nor resumed.
--prompt-cache-gib 0 and tf-qwen27-cuda keep the even chunk splits. Exactness against 0.6.6 is unchanged on the merged build: serial 4/4, windows 54/54, drafted 4/4, shared rounds 0 replies differ.
Results:
- A 104,688-token chat: first token in 40.4 s on turn 1, then 1.49 s on turns 2 and 3, each resuming 102,400 tokens.
- Qwen quality suite soak (3 chats x 20 turns beside a decoding stream): 60/60 equal to the same request alone, and 60/60 resumed. It took 118 s instead of 244 s.
- The rest of the suite scores the same as with the cache off.
- bf16 with the cache on, against a 0.6.6 server: 256-token greedy replies to prompts of 1.5K to 50K tokens, repeats, and a 3-turn chat are identical in 20/20. 16 of them resumed 4,096 to 49,152 kept tokens.

Tooling outside the tree

Following the 10 Oct rule, the two scripts this PR added that import the Python engine have been taken out of the tree: attention_kv8.py, which records the fp8 attention launches, and qwen_vision_fixtures.py, which records the vision fixtures. Their recordings stay fixture files, made once outside the tree. #608's capture*.py scripts and the recorded glue kernels are still here; #608's thread covers them.

Verification (RTX 5090, sm_120)

After the 1.0.10 and 1.0.11 merges (4e15fae5): the same checks below give the same results. The only conflicts were aot.zig (main's aot.zig: a launch with no exact captured variant takes one that assumes less of its arguments #794 already holds this PR's general pass, so main's version is taken), lane_host.zig (main's born lengths with this PR's image cache key), glm.zig and docs/api.md.
After the 1.0.9 merge (e3b72103):
- zig build test, test-cuda-host and test-server-cpu pass, and zig fmt --check is clean.
- bf16 against 0.6.6: serial 4/4, windows 54/54, drafted 4/4, shared rounds no reply differs.
- fp8 serial, drafted and shared-round replies are unchanged from before the merge.
- With the prompt cache on, the server matches 0.6.6 in 20/20, 12 of them resumed.
- Image chats, prompt-cache resumes and decode speed are unchanged.
- The merge adds takes_drafter to qwen27: 1.0.9's new --drafter refusal would otherwise refuse its DFlash2 drafter.
zig build test: 274/274 (after the merge with main).
fp8 vs bf16, teacher-forced with tf-qwen27-cuda kvcompare on prompts from 28 to 60K tokens: top-1 agreement 763/768, mean KL 0.0002–0.0013 nats/token.
Needle in a haystack at 255,055 tokens: found 4/4 at depths 0.10, 0.33, 0.50 and 0.90.
fp8 drafted decode equals fp8 serial decode. Concurrent replies equal their solo runs.
Image chats are correct under fp8.

Speed (fp8 vs bf16)

	fp8	bf16
Prefill, 100K tokens	38.7 s	36.6 s
Decode at 100K	206.6 tok/s	209.2 tok/s
Short prompts	about 5% slower (`_tail8` quantizes the window rows itself)	baseline
250K tokens	146 s prefill, 131 tok/s decode	does not fit

The lane host admits every queued request at once: a lone one still goes to the backend's driver, the rest to the core's addStreams (lanes/admit.zig). It runs the backend's optional prefill_many when there is more than one, draws each stream's first token, then asks for every stream's first drafts in one request, as multi.py's _fill_batch admits a batch. addStream is its one-stream case, with the same calls in the same order; a backend without prefill_many prefills each stream as before.

…ilies share it Source needed Nemotron's kernel handles only for the driver, the stream and whether the card is discrete; it now takes those as a Target (Target.of(ops) for Nemotron's own loader). cuda.zig exports the source module and the torch ops for the 27B's CUDA engine.

…y stream drafted A qwen3_5 checkpoint with the 27B's shapes opens zig/src/families/qwen27/cuda in tensorfold-native, beside the family's Metal engine. It reads the family's config.zig and refuses other shapes (cuda/shape.zig). The forward is the Python 0.6.6 engine's: its CUDA kernels (qmm_group, qmm_prefill at prompt tile 9, the GDN tree, replay and prompt chain, prefill attention) built from zig/kernels/cuda, and its Triton glue from a set captured on the chip (TENSORFOLD_CUDA_KERNELS). Verify rounds take every stream's window together, up to 16 streams of 16 rows, as multi_tree_forward lays them out, and commit only each stream's accepted path. DFlash2 (--drafter, else the pulled z-lab/Qwen3.8-27B-DFlash2) drafts every stream in one batch: each stream's kept taps absorbed together, every block in one forward, a best-first tree per stream, or a copied continuation where the context repeats. The lane core verifies the trees and trims them by each node's chance against a cost curve timed at load. Prompts that arrive together share one prefill pass (prefill_rows). Checked against captures of the Python engine (zig/tests/cuda/qwen27), on an RTX PRO 6000 and on one DGX Spark (GB10): serial replies equal token for token (a 4722-token prompt among them); 54 verify windows (chains, trees, shared rounds of 2 to 16 streams and up to 256 rows, every alignment of the packed inputs) equal row for row in their logits' bytes; drafted replies and each stream's rounds equal Python's; served replies of 1, 4 and 16 concurrent requests, greedy and sampled, equal the same request alone and its "draft": false decode.

…ore admission; GLM-5.3-Flash reads them on Metal Server (engine-independent): - `Info.vision` is an engine's offer: a `prepare` hook (an image's bytes to its tower's patches, grid, tokens and a content hash, decoded by the platform's own decoder), the image token, the wrapper's tokens, and a request's visual-token budget and image limit. `--vision`, `--vision-max-images` and `--vision-image-tokens` reach native engines. - User messages keep their image_url parts (data URLs) beside their text for the template. Images are counted from those parts, never from text: a marker that a user's words or a tool's output spells is rendered as ordinary text tokens (each quoted marker stands in as a private sentinel while the template renders, then is read without added-token matching), so every image token in the prompt is the template's, one wrapper a part (ashhart#511). A prompt that quotes no marker gets exactly the ids it got before. - Each image's placeholder is expanded to its tokens before the window, the chunks and the cache see the prompt; `Request.images` carries the prepared patches and positions, and `Request.cache_key` replaces each image's rows by ids derived from its content, so a kept state of one image never resumes another under the same placeholders. - 0.6.6's limits and refusals: 4 images, 10 MB each and 20 MB a request, 8192 px a side and 16 MP, a single frame, a shared visual-token budget, detail low at 256 tokens; PNG and JPEG (images.py's formats), WebP refused by name until a decoder exists; remote URLs refused. GLM-5.3-Flash on Metal: the image tower (24 pre-norm blocks with 2-D rotary attention, the 2x2 merge and the merger, its projections on gemm_nax), its rows written over the placeholders' embeddings in the prompt pass and one position on for the MTP head; preprocessing as 0.6.6's vision/images.py and mlx-vlm's Glm5NextImageProcessor (EXIF orientation, alpha over white, PIL's 8-bit bicubic resampler). tf-glm-vision compares a reference dump's stages. tools/zig/glm_vision_fixtures.py draws the fixed image set and dumps the Python frontend's tensors; tools/zig/vision_served.py checks a running server.

…0.6.6 did media_fetch.zig follows images_http.py: an https URL on port 443 without credentials, a fragment, a backslash or controls; every address the host resolves to public (Python's is_global less 192.0.0.0/24 and 168.63.129.16; IPv6 global unicast only, no 6to4 or Teredo); TLS verified for the host on the address that was checked; GET with identity encoding; up to 3 redirects, each checked again; the declared media type JPEG, PNG or WebP (WebP is then refused by its bytes, as with data URLs); the bytes within what the request has left of 10 MB an image and 20 MB in all; 10 s a download and 30 s for a request's downloads. std's connect has no timeout yet: a nonblocking connect and poll bound it, the socket's send and receive timeouts bound the rest. The HTTP exchange reads from any reader, so the response rules (redirects joined and rechecked, status, types, lengths, chunking, encodings, the deadline) are tested against canned responses; the URL and address rules are tested directly. Without --vision-urls an https part is refused with the flag's name, other schemes as 0.6.6 refused them; --vision-urls needs --vision. tools/zig/vision_urls_served.py checks a served model against public hosts.

Conflicts resolved toward main's admission (open.zig, chunked prompt passes); the PR's shared prefill batching (admit.zig, addStreams) is dropped, prefill_many stays as an unused optional vtable entry. Living Weights setup in native/cuda.zig is guarded by @hasDecl(F, "slide").

… URLs) onto qwen27-cuda-1.0.7

…its check against 0.6.6 Row cosine against 0.6.6's flash-attention tower on nine fixture images: mean 0.9997-0.99998, min 0.981-0.99998; 0.6.6's own math-vs-flash attention gives min 0.9895 on the same shapes image.

…rt_resize, PIL bicubic, patches PNG decode and patches equal 0.6.6's bit for bit on seven fixtures; JPEGs differ in 1-6% of bytes (stb's IDCT against libjpeg-turbo's).

…he prompt pass, M-RoPE positions - Media (vision.zig): the request's images encoded, each prompt row's feature row, and get_rope_index's three positions; rope_delta kept on the stream for decode and verify positions (the drafter's stay plain, as 0.6.6's do) - Forward.chunk: image rows take the tower's rows; a chunk holding them runs attn_prep with three position rows (qwen_vision.cu), others the captured kernel at their shifted 1-D positions - native/cuda.zig: --vision for families that serve it (Info.vision from the family's prepareImage and markers), the tower's call scratch kept out of the streams' room; other CUDA families refuse --vision - JPEGs through libturbojpeg when it loads (PIL's decoder): patches equal 0.6.6's on every fixture, PNG and JPEG - aot: a launch with no exact captured variant takes one that assumes less of its arguments; the 27B's attention plan segments start on 16-byte boundaries, as the Python engine's tensors do Checked on an RTX 5090 against 0.6.6: prompt token counts equal on five image requests, three replies word for word, text exactness unchanged (serial 4/4, windows 54/54, drafted 4/4, shared rounds 6/6), concurrent text replies equal solo beside an image request.

…itted streams --kv-cache fp8 keeps the 27B's attention caches as e4m3 rows with one fp32 scale a (position, key head) row (amax / 448), half the bytes of bf16: 256K tokens with the image tower in one stream on a 32 GB card. bf16 stays the default and its bits are unchanged (0.6.6 exactness: serial 4/4, windows 54/54, drafted 4/4, shared 0 replies differ). - tf_kv_quant (qwen_ops.cu) quantizes commits' and prompts' rows into the caches - pattn8_kernel: prefill attention staging e4m3 rows widened to bf16, a step ahead in registers (within 5% of bf16 at 100K) - _shared8/_tail8 (tests/cuda/qwen27/attention_kv8.py, captured like the other glue): the decode kernels over e4m3; _tail8 quantizes the window's own rows as their commit will, so a key's bits never depend on windows, drafts or batching (drafted equals serial, concurrent equals solo); attention.py's _merge folds the partials unchanged - attention partials leave the scratch arena: sized after admission for the streams admitted (Engine.fit, native fit hook) and charged to each stream, not for 16 streams at full context (1.4 GiB back at 122,880, 2.9 GiB at 262,144) - tf-qwen27-cuda kvcompare: fp8 against bf16 teacher-forced, top-1 agreement and KL

…tate The 27B serves the shared prompt cache (core/prompt_cache.zig) as Nemotron does, its copies in pinned host memory: a long window's caches leave the device no room for them. - reuse.zig: a kept state is a stream's DeltaNet recurrences and conv windows whole and its attention rows (and fp8 scales) through the mark, about 275 MiB at 4,096 tokens under fp8 - with a cache, prompts cut the zero-anchored 4,096-row grid (engine.Chunks), so a pass resumed or kept at a grid point computes what a fresh pass of the same prompt does; without one (--prompt-cache-gib 0, and tf-qwen27-cuda) the even splits stay, bit-exact against 0.6.6 - prefillFrom (engine, lone): a restored stream's pass from its grid point, stopping at the lane host's marks; a resumed drafting stream's context holds only the rows it read (drafts change no token) - image prompts neither resume nor keep (their rows' positions are the tower's) - native/cuda.zig: each family's snapshot vtable, and a host-memory budget (8 GiB or --prompt-cache-gib, at most half of MemAvailable) for a family whose copies live there - zig fmt over native/cuda.zig and qwen27/cuda/triton.zig (the fmt check) The qwen suite's soak check: 60/60 replies equal to the same request alone and 60/60 turns resumed (0/60 resumed before), in 117 s against 244 s; drafts, concurrent and behaviour unchanged.

Conflicts resolved toward both sides: - build/cuda.zig: nativeServer takes main's core module and the branch's qwen27 - engine_api, lanes/stream, lane_host: main's untaught and prompt_logprobs beside the branch's images and cache key; the lookup skips untaught and prompt_logprobs requests (main) and matches by cacheKey (branch) - native/cuda.zig: Nemotron's store as main makes it (keeps at any row, snap_ready's charge); the 27B's on its grid in host memory; prompt_logprobs and vision both in Info - glm_host: main's serves window with the branch's vision; growSnap and prepareImage both kept - metal.zig: the fp8 refusal before main's refusal(), the vision refusal before glm_moe_dsa - lane_host_reuse_test: the event switch handles prompt_logprobs zig build test 274/274; zig fmt clean; tf-qwen27-cuda against 0.6.6 unchanged (serial 4/4, windows 54/54, drafted 4/4, shared rounds 0 replies differ).

attention_kv8.py (which records the fp8 attention launches) and qwen_vision_fixtures.py (which records the vision fixtures) import tensorfold, so they move out of the tree; their recordings stay fixture files, made once outside it.

Conflicts resolved toward both sides: - build/cuda.zig: the CUDA test list keeps qwen27 beside main's tokenizer tests - lane_host: main's spans helper and its warm-pass lookahead rule - tokenizer: the branch's encodePlain beside main's memoized encodeSegment - cuda/kernels.zig: the branch's torch_norm_rotary beside main's norms and embed_rows - server/cli.zig: --cortheon beside --vision and --kv-cache - native/cuda.zig: main's shared lane binder (cuda_bound.zig); the startup line takes main's "passes of up to N rows" with the branch's fp8 note and host-memory cache; Nemotron's cache rules take main's .warm = .waits; warm turns stay off for the 27B, whose cache keeps states on its prompt grid Follow-ups the merge needs: - cuda_bound.zig forwards prefill_many, which the 27B's backend gives - qwen27's LoneRun takes main's resumed argument (the 27B has no lone driver) - qwen27 declares takes_drafter: main's new --drafter refusal would otherwise refuse its DFlash2 drafter Checked on an RTX 5090: zig build test 48/48; bf16 against 0.6.6 serial 4/4, windows 54/54, drafted 4/4, shared rounds no reply differs; fp8 serial, drafted and shared rounds unchanged; the server with the prompt cache on matches 0.6.6 in 20/20 (12 resumed); images, prompt-cache resumes and decode speed unchanged.

- aot.zig takes main's version: ashhart#794 already holds this branch's general pass, and also refuses a launch of 1 a runtime variant - lane_host.zig: main's born lengths and host spans, with this branch's cacheKey so an image prompt is still matched by its images' content - glm.zig: main's affine and wide modules beside vision and image

- docs/api.md: main's note that the Python 0.6.6 engine serves --vision, with the native server's image input for Qwen3.8-27B on CUDA and GLM-5.3-Flash on Metal; other native families refuse --vision

…emption Main's new lean check (94ab8f8) found 666 problems this branch adds: - 548 in zig/vendor/stb/stb_image.h, the third-party decoder kept as its authors wrote it. lean_check.py now skips zig/vendor/, as it skips generator outputs. - 118 in our own code. Each multi-line comment or docstring is now one line of at most 120 columns. Only comment and docstring text changes. - The vision fixture check moves from cli/qwen27_cuda.zig (635 lines) to cli/qwen27_vision_check.zig. The lean check against main now reports 0 new problems.

BobClawblaw · 2026-10-14T17:58:33Z

The zig check failed on the 1.0.11 merge because of the new lean check (94ab8f8a). The tests passed. The check counted 666 new problems, and bcd433c0 brings that to 0:

Our own code (118): each multi-line comment or docstring is now one line of at most 120 columns. Only comment and docstring text changed. tf-qwen27-cuda's vision fixture check moved to cli/qwen27_vision_check.zig, which brings qwen27_cuda.zig under 600 lines.
zig/vendor/stb/stb_image.h (548): this is the third-party image decoder, kept as its authors wrote it. Rewriting it to this style didn't seem right, so lean_check.py now skips zig/vendor/, the same way it skips generator outputs. This changes the rule itself, so please say if you'd rather handle vendored code another way, for example by keeping the header outside the tree.

Some detail was lost in shortening comments. The biggest cuts are in the file headers of the image and URL-fetch code and in the usage text of three Python scripts in tools/zig.

Checked on the RTX 5090 (sm_120) with the rebuilt binaries:

zig fmt --check, zig build test and test-golden pass (golden 309/319 equal, 0 unexpected differences).
bf16 against 0.6.6: serial 4/4, windows 54/54 with 0 rows differing, drafted 4/4, and shared rounds with 0 replies differing.
fp8 serial and drafted shas are the same as before this commit.
Vision fixtures: decode and patches equal 0.6.6 on all 9 images. Tower row cosines are 0.981 or more at worst, the same as before.
The image requests on a 256K fp8 server answer correctly.

I didn't rerun the prompt-cache or speed checks, since nothing outside comments changed in that code.

ashhart · 2026-10-14T18:16:12Z

Hello @BobClawblaw,

Thank you for the 27B CUDA image path, and for merging 1.0.11 into it.

JPEG is decided. The images use the same pinned libjpeg-turbo, loaded at runtime and refused by name when it is missing. image.zig, reuse.zig, and stb_image.h are the same bytes as the head reviewed before this merge. The 27B CUDA engine lands with #608. Nothing further is waiting on you for the pixels.

Thank you.

Main's lean check counted 47 new problems on this pull. Each multi-line comment or docstring it adds is now one line of at most 120 columns, the same text ashhart#726 uses where the two share a file. Only comment and docstring text changes.

Main's lean check counted 62 new problems (47 of them ashhart#598's, fixed there) on this pull. Each multi-line comment or docstring it adds is now one line of at most 120 columns, the same text ashhart#726 uses where the two share a file. Only comment and docstring text changes.

ashhart · 2026-10-15T08:56:47Z

Thank you, @BobClawblaw, for the CUDA vision tower, the M-RoPE integration and host memory prompt reuse, all with careful image and cache receipts.

We chose #836 as the one Qwen CUDA base, so we are closing this branch on the older engine. The vision and reuse pieces are exactly what #836 needs: these checkpoints read images, so the family must too, in our own kernels and with the shared picture decoder from #598.

Please bring them over as focused changes with their image, resume and concurrency checks. The fp8 cache can follow once the base's bf16 gates pass.

Posted by Ash's AI Assistant Synthena

…#726 Two image prompts with the same text render the same ids, so keyed by tokens alone one could resume the other's state. Request.cache_key holds the prompt with each image's placeholder rows as ids from its content; lookups, keeps and resumes take cacheKey() (the prompt when empty). The test is ashhart#726's.

…ry (ashhart#726's reuse) cuda_reuse.zig is the family's Snapshots target: a save copies a stream's recurrences and conv tails whole, each attention layer's key and value rows up to the mark, and its rope_delta, into pinned host memory; restore copies them back. The prompt pass restores a kept state and calls the lane host at the store's marks only where a cold pass cuts a chunk (every max_rows rows from 0, or a request's chunk start), so every resumed reply keeps a cold pass's bits. The native host takes any family's snap_* functions; a snap_host family's cache is --prompt-cache-gib of host memory (8 GiB unasked, at most half of MemAvailable), on planned rules over its prompt grid.

philip-pentatonic and others added 11 commits October 10, 2026 18:34

Merge PR ashhart#598 and ashhart#614 (native image input, HTTPS image…

9f99ca9

… URLs) onto qwen27-cuda-1.0.7

Qwen image preprocessing on Linux: stb_image decode, EXIF, alpha, sma…

407cb28

…rt_resize, PIL bicubic, patches PNG decode and patches equal 0.6.6's bit for bit on seven fixtures; JPEGs differ in 1-6% of bytes (stb's IDCT against libjpeg-turbo's).

BobClawblaw force-pushed the qwen27-cuda-1.0.7 branch from 4177b6e to 5d8bf48 Compare October 12, 2026 09:02

BobClawblaw added 3 commits October 12, 2026 12:42

BobClawblaw mentioned this pull request Oct 12, 2026

Qwen3.8-27B on CUDA: the qwen27 family's engine for NVIDIA GPUs, every stream drafted #608

Closed

8 tasks done

jregan-beasley-smc mentioned this pull request Oct 12, 2026

perf(qwen27 cuda): reduce repeated-prompt TTFT #747

Closed

This was referenced Oct 14, 2026

New family: MiniCPM-V 4.7 35B-A3B (Qwen3.6 MoE fine-tune + vision tower) #785

Open

Landing next #286

Open

BobClawblaw added 3 commits October 14, 2026 16:18

Merge main (1.0.11) into qwen27-cuda-1.0.7

4e15fae

- docs/api.md: main's note that the Python 0.6.6 engine serves --vision, with the native server's image input for Qwen3.8-27B on CUDA and GLM-5.3-Flash on Metal; other native families refuse --vision

ashhart mentioned this pull request Oct 15, 2026

Qwen3.6-35B-A3B on CUDA: routed experts and its MTP layer drafting chains (on #836) #856

Open

8 tasks done

ashhart closed this Oct 15, 2026

This was referenced Oct 15, 2026

Qwen3.5/3.8 on CUDA: image input on #836's engine (2B, 27B; on #836 and #853) #885

Open

Qwen3.5/3.8 on CUDA: kept prompt states in host memory on #836's engine (#726's reuse; on #885) #888

Open

Navigation Menu

Repository navigation

Qwen3.8-27B on CUDA: image input and --kv-cache fp8 (256K with vision on a 5090) - #726

Qwen3.8-27B on CUDA: image input and --kv-cache fp8 (256K with vision on a 5090)#726
BobClawblaw wants to merge 18 commits into
ashhart:mainfrom
BobClawblaw:qwen27-cuda-1.0.7

BobClawblaw commented Oct 12, 2026 •

edited

Loading

Uh oh!

BobClawblaw commented Oct 14, 2026

Uh oh!

ashhart commented Oct 14, 2026

Uh oh!

ashhart commented Oct 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

BobClawblaw commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

fp8 KV cache

Prompt cache

Tooling outside the tree

Verification (RTX 5090, sm_120)

Speed (fp8 vs bf16)

Uh oh!

BobClawblaw commented Oct 14, 2026

Uh oh!

ashhart commented Oct 14, 2026

Uh oh!

ashhart commented Oct 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

BobClawblaw commented Oct 12, 2026 •

edited

Loading