server/llama: honor per-request speculative.n_max, per-sequence snapshot budget, honest rollback refusal (multi-slot decode with MTP) by alenizy361 · Pull Request #208 · PrismML-Eng/llama.cpp

alenizy361 · 2026-09-19T07:32:40Z

Summary

Stacked on the in-place recurrent state PR (#207) — its 4 commits are included here; review the last 11 commits. Three server/speculation fixes found while running --parallel N with --spec-type draft-mtp:

Per-request speculative.n_max was dead code. The whole per-request speculative block in tools/server/server-schema.cpp was inside #if 0, and get_n_draft_max() consulted only the context-level setting, so every slot drafted every step. With several slots the per-slot token counts (1 + draft) differ, split_equal(sequential, n_rs_seq+1) fragments the ubatches and the server degrades to ~one sequence per forward. Now speculative.n_max is parsed and 0 means "no drafting for this request" (the slot simply does not set drafting). LLAMA_SERVER_SPEC_MULTISLOT=0 restores the previous policy (logged at init).
Per-sequence rollback-snapshot budget (llama_memory_seq_rs_snapshots). K (the number of state planes a ubatch writes) is now derived from the budgets of the sequences in the ubatch instead of being a context constant: an ubatch of opt-out slots writes one plane (K=1, one 96 MB plane per active cell instead of three), a ubatch containing a drafting slot keeps K = n_rs_seq + 1 so drafting slots stay bit-identical. The draft context itself no longer allocates snapshot planes.
Honest rollback refusal: llama_memory_seq_rm refuses a partial rollback beyond the planes the sequence's last ubatch actually wrote instead of installing a stale plane (the server never requests one; the rs-rollback examples and test-recurrent-state-rollback were adjusted to keep the rollback inside the verify ubatch). Invariant documented on llama_memory_seq_rm.

Measurements (live server, Bonsai-2 27B PQ2_0 + grafted MTP head, `--parallel 8`, 6 concurrent requests with `"speculative.n_max": 0`)

	before	after
6 concurrent opt-out requests, aggregate	6.3 tok/s (RSS 12.1 GB)	9.4–10.3 tok/s, no swap
anonymous memory per active slot	~490 MB	~220–370 MB
single MTP request (regression check)	7.3 tok/s	8.3 tok/s, md5-identical output

test-gdn-inplace: 360 cases, 0 failed (adds K=1 rollback-plane reads and a T=1 cross-K check).

Notes

Mixed ubatches (a drafting slot next to opt-out slots) are split exactly as before; slot-id gaps still split ubatches (sequential rule) — not addressed here.
Written with AI assistance (Claude Code) under my direction; adversarially reviewed before measurement; numbers from my machine.

…ate) New op form ggml_gated_delta_net_cache (op_params[2] = 1, src[6] = rows_in I32[n_seqs], src[9] = rows_out I64[min(T,K)*n_seqs]): the CPU kernel reads each sequence's state at cache row rows_in[seq], updates the live row rows_out[seq] in place (no copy when the rows coincide) and writes snapshot slots 1..min(T,K)-1 straight into the cache; the result carries the attention scores only. No get_rows gather, no dst state planes, no write-back cpy / set_rows per layer: 6 -> 2 state-sized moves per seq. Graph side: llama_memory_recurrent_context::rs_inplace_ok(n_seqs), a side-effect-free predicate (never calls s_copy, which consumes rs_idx): !is_full && n == n_seqs && cells[head+i].src0 == head+i. Stored in llm_graph_input_rs::rs_inplace and added to can_reuse_rs so a reused graph can never be in the wrong mode. build_rs_cache_view(inplace) asserts the extra relocation range is empty, which makes the documented rows-mode ordering hazard unreachable for the in-place consumer. Model side (qwen35): gdn_state_inplace_dev_ok (no GPU/IGPU device), the gate gdn_state_rows_env && inplace_dev_ok && inp->rs_inplace && fused_ok && cparams.n_rs_seq == 0 (Stage 7A: K == 1 only), the Metal rows+SET_ROWS mode kept for Metal-only machines; build_recurrent_attn gains the state_inplace branch, every legacy branch is untouched and serves the transition ubatches (join/leave/seq_cp/reorder) and the env switch. GGML_GDN_STATE_GATHER=1 still forces the legacy gathered path (A/B). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QrdPnP5KL449GsoWBZx52R

… + 1) Drop the cparams.n_rs_seq == 0 restriction of the qwen35 gate: with --spec-type draft-mtp the cache-writing op now updates slot 0 in place on the live row and writes slots 1..min(T, K)-1 straight into the snapshot planes of the same cell (rows_out = the existing s_write_rows mapping slot*mem_size + head + seq); a pending rollback reads plane rs_idx of the same cell (rows_in != rows_out -> copy-then-in-place in the kernel). No dst state planes, no SET_ROWS, no per-thread state scratch. Transition ubatches still take the gathered K-slot path. GGML_GDN_INPLACE_K1_ONLY=1 restores the previous commit's K == 1 gate so the two can be bisected on the same binary; GGML_GDN_STATE_GATHER=1 still forces the legacy gathered path for everything. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QrdPnP5KL449GsoWBZx52R

…paths tests/test-gdn-inplace.cpp (llama_build_and_test) runs, on identical synthetic inputs and in three separate ggml contexts, the gathered path (get_rows -> ggml_gated_delta_net(K) -> write-back cpy, exactly as build_recurrent_attn emits it for K == 1 and K > 1), the rows path (ggml_gated_delta_net_rows -> ggml_set_rows) and the new ggml_gated_delta_net_cache, then memcmp's the attention scores and the WHOLE cache tensor (touched rows identical, untouched rows untouched) and checks the in-place op actually modified the cache. Matrix: K in {1, 3}, T in {1, 3}, n_seqs in {1, 4, 8}, rows_in in plane 0 (identity) and in every rollback plane 1..K-1 of the same cell, raw gates on/off, H_k == H_v and H_k == H_v/2, 1 and 4 threads: 192 cases, 0 bytes differ. CPU only, no model. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QrdPnP5KL449GsoWBZx52R

The recurrent cache is mem_size*(1+n_rs_seq) rows per layer and was memset at construction (ggml_backend_buffer_clear), so every cell of every snapshot plane was resident from the start: --parallel 16 with n_rs_seq = 2 pinned ~4.7 GiB of the 15 GiB here while one active slot needs ~0.3 GiB, evicting the mmap'd weights that decode streams on every token. For the CPU buffer type on Linux the buffer is now one anonymous private mmap wrapped with ggml_backend_cpu_buffer_from_ptr (tensors placed with a ggml_tallocr exactly as ggml_backend_alloc_ctx_tensors_from_buft would), with no eager memset: the kernel hands out zero pages on first touch, so residency follows the cells actually written. clear(true) drops the pages with madvise(MADV_DONTNEED) instead of a memset; the destructor unmaps after freeing the buffers (from_ptr buffers do not own their memory). Non-host bufts, non-Linux hosts and LLAMA_RS_EAGER_ZERO=1 keep today's alloc + clear. Exactness argument (in the ctor comment): every row the graph reads is either the rs_z row zeroed in-graph before the read, a row written by an earlier step (src0 = src of a used cell), a snapshot plane written by the same sequence before a rollback reads it (rollback <= n_rs_seq, tail kept in one ubatch, slots 0..min(T,K)-1 written), or a row written by state_read_data; padding is never read. The startup line keeps its format ('RS buffer size = N MiB') and appends '(anonymous mmap, lazily zeroed)' when the lazy path is active. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QrdPnP5KL449GsoWBZx52R

…the request) The "speculative.n_max" request field was parsed inside an `#if 0` block and nothing in the server read task->params.speculative, so every generating slot drafted the global --spec-draft-n-max tokens whether or not the request asked for it. With draft-mtp the per-slot token counts then differed step by step (1 + d, d in 0..n_max), which split_equal(n_keep_tail = n_rs_seq + 1) cannot batch: N "opted-out" slots decoded as up to N ubatches per step instead of one. Move only the n_max field out of the `#if 0` (n_min/p_min/type/ngram_* stay disabled) and record the policy per slot at task launch: - spec_enabled = can_speculate() && n_max != 0 -> get_n_draft_max() returns 0 for an opted-out slot: dp.drafting stays false, no spec checkpoint update, handle_last_sampled_token adds exactly one token, post_decode samples it normally and the verify loop never sees the slot. N such slots with consecutive ids form one N x 1 ubatch (remaining == 0 satisfies the tail rule), the same call shape the no-spec server makes. - spec_n_max clamps the draft of a request that sends 0 < n_max; a request without the field carries the server-wide value, at which the MTP loop stops anyway, so its draft, batch, checkpoints and rollbacks are unchanged byte for byte. Mixed steps split as today: a drafting slot contributes 1 + d tokens and is emitted in its own equal-length ubatch (tail rule), the opted-out run forms a T = 1 ubatch. LLAMA_SERVER_SPEC_MULTISLOT=0 restores the previous policy (every slot drafts, the request field is ignored). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QrdPnP5KL449GsoWBZx52R

…pshots) The recurrent memory widens its R/S tensors to (1 + n_rs_seq) planes and every ubatch writes min(T, n_rs_seq + 1) of them for every sequence, whether or not that sequence will ever roll back. Add a per-seq budget rs_n_snap in [0, n_rs_seq], default n_rs_seq (every existing caller keeps today's behaviour), settable through the new public API and forwarded by the hybrid memory: - llama_memory_recurrent::seq_rm accepts a partial tail removal only up to the seq's budget; a budget-0 seq refuses exactly like a memory with n_rs_seq == 0, so a caller sees the FULL-type behaviour for that sequence. - llama_memory_recurrent_context::get_n_snap() = max budget over the seqs of the ubatch apply() just placed (n_rs_seq for the reserve context so the compute buffers fit the largest graph, 0 without snapshots). Side-effect free, so it is valid at graph build and again in can_reuse. Nothing resets the budget but the API (seq_rm / clear / state_read leave it), and no graph consumes it yet. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QrdPnP5KL449GsoWBZx52R

…gets The delta-net state write paths took K = cparams.n_rs_seq + 1 for every ubatch. Take K = 1 + llama_memory_recurrent_context::get_n_snap() instead, recorded once per graph in llm_graph_input_rs::rs_n_snap (build_rs_inp_impl, so both the recurrent and the hybrid callers get it) and made part of the graph reuse key: at T = 1 the s_write_rows length is identical for K = 1 and K = 3 while the op params (ggml_gated_delta_net_cache), the number of conv-state copies and the fused-gate decision differ. A ubatch that contains a drafting seq (budget n_rs_seq) gets exactly today's K on every path -- in-place, gathered, rows -- identical topology, identical s_write_rows contents, identical op params: bit-identical scores and cache rows. A ubatch of opted-out seqs (budget 0) gets K = 1: one conv-state copy, a single S plane on the in-place path (the kernel writes no snapshot rows for K == 1), the !keep single-row copy on the gathered path and n_written = 1 on the rows path -- also on the prompt ubatch and on the gathered transition ubatches, which is what bounds the per-slot residency to one plane. qwen35 keys fused_ok / the K1-only env / the Metal rows mode on K as well, so an opted-out ubatch builds node for node the graph the no-spec context builds (n_rs_seq == 0): the no-spec server becomes a usable md5 oracle for those requests. qwen35moe / qwen3next share build_recurrent_attn and keep today's K unless the new API is called. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QrdPnP5KL449GsoWBZx52R

At task launch, right after the drafting policy is fixed, set the slot's sequence budget on the target memory: llama_n_rs_seq planes for a drafting slot (today's K on every ubatch it takes part in), 0 for a slot whose request sent "speculative.n_max": 0. Such a slot decodes with K = 1 -- one conv-state copy, plane 0 only on the prompt ubatch, the generation ubatches and the gathered transition ubatches -- node for node the graph the no-spec server builds, so its residency is one S plane plus the prompt-end checkpoint that both servers keep. The refusal that a budget of 0 adds to llama_memory_seq_rm is unreachable from the server for these slots: every partial tail removal it issues comes from the verify loop (non-empty spec_draft only); the prompt-reuse path resets or restores a checkpoint and then removes from cell.pos + 1, the cache-reuse and context-shift ranges end at or before cell.pos, and rm_all never enters the partial branch -- the same calls the FULL-type no-spec server makes for every slot. A slot that inherits a pending rollback index from an earlier drafting request reads a fully written plane (covered by test-gdn-inplace). The MTP draft context on qwen35 holds a plain KV cache and needs no call. LLAMA_SERVER_SPEC_MULTISLOT=0 keeps every budget at n_rs_seq. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QrdPnP5KL449GsoWBZx52R

… check Two families on top of the existing 192 cases (whose inputs, order and output lines are unchanged), for the per-sequence snapshot budget: - K = 1 with rows_in in plane r in {1, 2} of an (r + 1)-plane cache: an opted-out request launched on a slot whose seq still carries a pending rollback index from a previous drafting request. The cache-writing op must read plane r and write plane 0 only, byte for byte like the gathered and rows paths (96 cases). - T = 1 cross-K on a 3-plane cache: the op with K = 1 and with K = 3 on identical inputs must leave byte-identical scores and cache -- the mixed T = 1 ubatch where a drafting seq with an empty draft shares the ubatch with opted-out seqs (K = 3) gives every seq exactly the K = 1 result (72 cases). The cache now allocates n_planes rows per cell (== K for the original matrix). 360 cases, 0 failed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QrdPnP5KL449GsoWBZx52R

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QrdPnP5KL449GsoWBZx52R

A partial seq_rm accepted any rollback in 1..n_rs_seq and pointed the next read at that snapshot plane, whether or not the ubatch that set cell.pos had written it: a T = 1 ubatch writes plane 0 only, so a rollback of 1 after it would read the state an older ubatch left in plane 1 (reachable today from llama_context::decode's failed-ubatch cleanup, which removes [pos_min, inf) of a T = 1 ubatch). Track rs_n_valid per seq -- min(T, K) - 1, recorded when the ubatch is applied (not in find_slot, which prepare() dry-runs), copied by seq_cp, 0 after rm_all / clear / state_read -- and refuse a rollback beyond it. Never refuses a rollback the server asks for: a drafting slot's verify ubatch has T = 1 + n_draft and K = n_rs_seq + 1, so rs_n_valid = min(n_draft, n_rs_seq) >= n_rollback (accepted >= 1, and n_rollback > n_rs_seq already takes the checkpoint path). The failed-ubatch cleanup now returns false (its result is ignored) instead of installing a stale read. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QrdPnP5KL449GsoWBZx52R

A micro-batch snapshots the state after each of its last min(T, K) - 1 tokens and never the state it started from (ops.cpp slot = n_tokens - 1 - t, in-place and gathered alike, upstream 1a07bfa included), so a partial removal of a sequence's tail is a rollback of n positions only while 1 <= n <= min(T - 1, n_snap) with T the token count of the sequence's last micro-batch: N single-token decodes followed by N positions back never had a snapshot to read and are refused since 317e4d0 instead of reading a plane an older micro-batch wrote. State it on the API. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QrdPnP5KL449GsoWBZx52R

…rent-state-rollback Two steps rolled back across earlier micro-batches, which the memory never snapshots (see llama_memory_seq_rm): the partial-flags checkpoint step replayed three positions one token at a time and then rolled the same three back -- on a6536c3 that passed only because plane 3, untouched by the T = 1 replays, still held the prompt ubatch's state for the identical tokens; since 317e4d0 it is refused (rs_n_valid == 0) and the test exited with 'partial rollback failed'. The multi-seq sub-test rolled back exactly the 3 tokens of a 3-token ubatch and read plane 3 of the 16-token prefill ubatch (the state after position 12, not 15) on every revision since the upstream merge. Partial step: extend both contexts with the same n_rollback + 1 tokens in ONE ubatch and roll back n_rollback, the largest rollback that ubatch supports; replay positions ext_pos + 1 .. ext_pos + 3 (still crossing DSV4's ratio-4 boundary). The 'full' replay logits are kept for the dirty-context check. Multi-seq: ctx_roll decodes [0, 15) then [15, 19) as one 4-token ubatch and rolls back 3 (plane 3 = state after 15); ctx_ref is restored from each seq's checkpoint at that point, which resolves the pending plane into a plain state, so both contexts replay from bit-identical bytes and the 1e-7 tolerance stays valid by construction. A forward-only reference cannot be compared bitwise: an honest rollback target sits inside a multi-token ubatch of ctx_roll and the CPU matmul kernels depend on the ubatch size (sgemm rejects small n). Not run here (model binaries are off limits in this session); established by construction from the kernel slot mapping and the memory's plane indexing. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QrdPnP5KL449GsoWBZx52R

…rollback stays inside it Both harnesses over-generated with single-token decodes and then rolled back rewind (+1) positions: with fixed snapshot planes those decodes write plane 0 only, so the rollback read a plane the prompt ubatch had written (a wrong state, silently) and is refused since 317e4d0 ('ROLLBACK REFUSED'). Decode the accepted token together with rewind draft tokens as ONE ubatch (the server's verify shape, logits for the accepted token's row only) and drop the drafts: with T = rewind + 1 a rollback of rewind positions is the largest that ubatch supports and lands on the state after the accepted token; needs n_rs_seq >= rewind (was rewind + 1). The multi-seq harness keeps the other sequences in lockstep with an ordinary single-token step at that iteration. Header comments state the invariant and that a mismatch right at a rollback step can be matmul rounding between the (R + 1)-token and single-token kernels. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QrdPnP5KL449GsoWBZx52R

The draft context inherited n_rs_seq = draft.n_max from common_context_params_to_llama, so common_context_can_seq_rm classified a recurrent drafter as RS and the server / speculative-simple removed the drafted tail with a partial seq_rm (server-context.cpp:2997, speculative-simple.cpp:209) instead of restoring the checkpoint. Every drafter decodes its draft one token per step (speculative.cpp draft loop), so the memory never held that snapshot: on a6536c3 the draft context continued from a stale plane (garbage drafts after the first round, silently); since 317e4d0 the refusal would GGML_ABORT at server-context.cpp:2998. With n_rs_seq = 0 a recurrent drafter is classified FULL and takes the existing checkpoint path (update_dft before drafting, load_dft before the now no-op seq_rm), the only correct one; a KV-cache drafter (transformer, the qwen35 / nemotron-h MTP contexts, dspark) is classified PART by the same probe it already ran and ignores n_rs_seq, and the target keeps its window. Also drops K planes of recurrent state per cell from the draft context. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QrdPnP5KL449GsoWBZx52R

bri-prism

Agent review: posted by the maintainer's coding agent at their request.

No additional confirmed finding in this source pass. A native ARM64 CPU build of this head succeeded, and test-gdn-inplace passed all 360 cases with zero failures, including the cross-budget cases.

This PR includes #207's implementation, so please make the dependency explicit and review/rebase it as a stack. The test above does not establish end-to-end server correctness: mixed drafting/non-drafting slots, prompt-cache reuse, checkpoint/replay, and rollback refusal still need model-backed integration coverage. Linux lazy allocation and GPU paths were not executed here.

Reviewed commit: 3ed7c4340d114c4e05c3aa5b7035b578e1bb7f8e.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

alenizy361 · 2026-09-20T08:42:30Z

Confirmed, and thank you for catching it — the dependency was real but implicit. Making it explicit here.

Stack structure. #208 is not independent of #207: c875ea4 (the #207 head you reviewed) is an ancestor of 3ed7c43, with 11 commits on top:

c875ea4  <- #207 head (in-place GDN state + lazily zeroed RS buffer)
  44e4b09  llama: per-sequence rollback snapshot budget (llama_memory_seq_rs_snapshots)
  7240566  gdn: derive the snapshot plane count K of a ubatch from the seqs' budgets
  fd9a0af  server: give an opted-out slot a rollback snapshot budget of 0
  e4be22f  tests: gdn in-place op with a K = 1 rollback read and a T = 1 cross-K check
  80164f7  server: document the per-request speculative.n_max
  b6d3cb7  memory-recurrent: only roll back onto planes the seq's last ubatch wrote
  0fd8b1d  llama: document the recurrent rollback invariant on llama_memory_seq_rm
  cbb4c55  tests: roll back only onto planes the last ubatch wrote
  2d2c3c3  examples: rs-rollback decodes the drafts as one verify ubatch
  3ed7c43  speculative: no rollback snapshot planes in the draft context  <- #208 head

So #207 should land first and #208 rebases onto whatever #207 becomes. I'd rather not squash them into one PR: #207 is a kernel/memory change that stands on its own and answers an open upstream request (ggml-org#28841), while #208 is server and scheduling behaviour on top. Reviewing them separately in that order seems cleaner, but if you'd prefer a single PR I'll collapse them.

On the coverage you flagged. You're right that test-gdn-inplace passing 360/360 says nothing about end-to-end server correctness. Mixed drafting/non-drafting slots, prompt-cache reuse, checkpoint/replay and rollback refusal are all still unproven, and Linux lazy allocation was inspected but not executed on your macOS host. That work is on my side — I have the Linux box and a real model — and I'll post the results on #207 rather than asking you to take the unit test as a proxy.

One related note: #206 just had its two review findings fixed in 10df298 (missing non-x86 arch-fallback.h alias, and a shape check that let 128-column tensors reach PQ2_0 with a zero-byte Q8_K activation row). That PR is independent of this stack.

Copilot

Copilot review overview

🟡 Changes recommended

Snapshot forwarding, failure-state accounting, backend gating, and mapping ownership contain unresolved correctness or reliability defects.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 2 · 2

Open (4)

Disable cache-mutating op for all non-CPU devices · New
Forward seq_rs_snapshots through hybrid ISWA memory · New
Use RAII to release mappings on partial construction failure · New
Commit valid-plane metadata only after successful graph execution · New

        // integrated GPUs (e.g. unified-memory CUDA devices) report IGPU, not GPU
        const bool is_gpu = ggml_backend_dev_type(ldev.dev) == GGML_BACKEND_DEVICE_TYPE_GPU ||
                            ggml_backend_dev_type(ldev.dev) == GGML_BACKEND_DEVICE_TYPE_IGPU;
+        if (is_gpu) {
+            // the cache-writing GDN op mutates one of its sources; only the CPU backend implements it
+            gdn_state_inplace_dev_ok = false;
+        }


+            // for an opted-out slot (K = 1: plane 0 only, the graph the no-spec server builds). Set
+            // before the slot's first ubatch. The MTP draft context holds a plain KV cache, so the
+            // call goes to ctx_tgt only.
+            llama_memory_seq_rs_snapshots(llama_get_memory(ctx_tgt), slot.id, slot.spec_enabled ? llama_n_rs_seq(ctx_tgt) : 0);


+            buf = rs_alloc_lazy_zero_host_buffer(ctx.get(), buft, &ptr, &size);
+            if (buf) {
+                mmaps.push_back({ buf, ptr, size });


+    // once the graph ran, planes 1..n_valid of every seq in the ubatch hold the states 1..n_valid
+    // tokens back (min(T, K) planes are written, K = 1 + get_n_snap()). Recorded here rather than
+    // in find_slot because prepare() dry-runs find_slot for every ubatch
+    if (mem->n_rs_seq > 0) {
+        const uint32_t n_valid = std::min<uint32_t>(ubatch.n_seq_tokens, 1 + get_n_snap()) - 1;
+
+        for (uint32_t s = 0; s < ubatch.n_seqs; ++s) {
+            const uint32_t i = s*ubatch.n_seq_tokens;
+            for (int32_t j = 0; j < ubatch.n_seq_id[i]; ++j) {
+                mem->set_rs_n_valid(ubatch.seq_id[i][j], n_valid);
+            }


bri-prism · 2026-10-02T19:03:55Z

Tested on an Intel box (Core Ultra, Arc B390 Xe3 iGPU), Windows 11, driver 32.0.101.8724, GCC 16.2 ucrt64, Release, static.

The head 3ed7c4340 conflicts with current prism in tools/server/server-context.cpp, so this was tested at its own merge-base 1a07bfa5f (47 commits behind prism) against that same base. It needs a rebase before it can land.

Correctness: test-gdn-inplace 360/360 including the new cross-K cases, every case attn diff 0 and cache diff 0. test-recurrent-state-rollback on a 2B GDN model (PTQ1_0) passes here, where the base fails with a dirty-context logits mismatch, noting that this PR also changes the test to keep the rollback inside the verify ubatch. Greedy parity vs base with 8 distinct prompts at temperature 0: 1/1, 4/4 and 8/8 identical on CPU, 4/4 on Vulkan.

Throughput, llama-server -c 1024*N -np N -fa on, aggregate t/s, base to PR: CPU 8.1 to 8.2 (1 slot), 10.9 to 11.1 (4), 12.0 to 12.1 (8); Vulkan 12.2 to 11.9 (4 slots, within the roughly 15% run-to-run noise on this driver). The absolute numbers are low because 1a07bfa5f predates later prism kernels; current prism runs the same model at 13 t/s CPU and 106 t/s Vulkan, so these only compare the PR to its own base.

Not run: 16 slots, a larger GDN model, or MTP speculation.

bri-prism · 2026-10-24T19:46:22Z

Same story as #207, which this is stacked on: GDN rollback on prism is now replay (#358), so the per-sequence snapshot budget (llama_memory_seq_rs_snapshots, K derived from the budgets) needs rethinking against replay before it can land. I'd suggest splitting off the server half. Honoring per-request "speculative.n_max" (with 0 opting the request out) is still missing on prism and is useful on its own. On rebase it needs to keep prism's new spec_depth_max cutoff alongside your !spec_enabled check in server-context.cpp. Also note that #320 adds the dp.n_max check to the MTP drafter, so coordinate on that one line.

alenizy361 and others added 15 commits September 19, 2026 10:28

server: document the per-request speculative.n_max

80164f7

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QrdPnP5KL449GsoWBZx52R

github-actions Bot added documentation Improvements or additions to documentation ggml testing server model labels Sep 19, 2026

alenizy361 mentioned this pull request Sep 19, 2026

Three CPU/AVX2 + recurrent-state PRs for Ternary Bonsai-2 27B (3.9 → 8.3 tok/s bit-exact on a laptop CPU) PrismML-Eng/Bonsai-demo#196

Open

bri-prism reviewed Sep 19, 2026

View reviewed changes

bri-prism mentioned this pull request Sep 19, 2026

llama: in-place recurrent (GDN) state update for multi-sequence decode + lazily zeroed RS buffer #207

Open

khosravipasha requested a balanced review from Copilot September 19, 2026 19:58

Copilot started reviewing on behalf of khosravipasha September 19, 2026 19:59

View session

Copilot AI reviewed Sep 19, 2026

khosravipasha requested a balanced review from Copilot September 21, 2026 00:32

Copilot started reviewing on behalf of khosravipasha September 21, 2026 00:32

View session

Copilot AI reviewed Sep 21, 2026

View reviewed changes

Navigation Menu

Repository navigation

server/llama: honor per-request speculative.n_max, per-sequence snapshot budget, honest rollback refusal (multi-slot decode with MTP) - #208

server/llama: honor per-request speculative.n_max, per-sequence snapshot budget, honest rollback refusal (multi-slot decode with MTP)#208
alenizy361 wants to merge 15 commits into
PrismML-Eng:prismfrom
alenizy361:server-spec-multislot

alenizy361 commented Sep 19, 2026

Uh oh!

bri-prism left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

alenizy361 commented Sep 20, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

bri-prism commented Oct 2, 2026

Uh oh!

bri-prism commented Oct 24, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

alenizy361 commented Sep 19, 2026

Summary

Measurements (live server, Bonsai-2 27B PQ2_0 + grafted MTP head, --parallel 8, 6 concurrent requests with "speculative.n_max": 0)

Notes

Uh oh!

bri-prism left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

alenizy361 commented Sep 20, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

bri-prism commented Oct 2, 2026

Uh oh!

bri-prism commented Oct 24, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Measurements (live server, Bonsai-2 27B PQ2_0 + grafted MTP head, `--parallel 8`, 6 concurrent requests with `"speculative.n_max": 0`)