Feature Request: adaptive-kv-streaming

[munchmunch77]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

Please implement adaptive-kv-streaming

Motivation

I can run higher quants for both model and KV and have a much higher context with minimal drop in performance.

Possible Implementation

Add a flag to enable or disable it.

[mgscreativa]

Really appreciated if merged! now I'm experimenting with kvarn5 and MTP on 200k context on my RTX3090 24GB and this looks promising.

[Midaychi]

What's the difference between this and just lowering your -c context?
I tried an adaption of this and the context beyond the 'arena' window seems to functionally not exist for the model.

[munchmunch77]

You can lower the context if you happy with a smaller context or running at lower KV quants

[Anbeeld]

I checked the initial repo and the article and I failed to find the most important thing there: KLD or some other measurements of how this affects output. If you are already a user of this fork, could you please benchmark that, or create an issue asking for that?

The thing is, if output quality does not match the full resident even roughly, then there's barely any point in the feature as a whole.

[Midaychi]

You can lower the context if you happy with a smaller context or running at lower KV quants

That's what I mean though. If you use this then sure the context number can be bigger but the model still can't actually read all the context at once it can only address it in arena-sized windows, and I DON'T mean dynamically as it generates.

Setting an 8 GB arena but setting max context vs constraining your context number till it only uses 8 GB is functionally identical except that the former's window slides through that big context (but the model can only ever see that window and that window doesn't move WHILE it's generating.).

Well that and it tends to show the model repeating gibberish at the cut off point s

[munchmunch77]

@Midaychi Took me a while to figure out what you meant and I think I understand now.

@Anbeeld
I did generate KLD and it was very similar, close to a rounding error comparing F16 cache to q4_0 on stock llama.cpp and then comparing q4_0 from the fork against F16 from stock.

q4_0 stock llama.cpp vs F16 stock llama.cpp
====== KL divergence statistics ======
Mean KLD: 0.003116 ± 0.000023
Maximum KLD: 0.488065
99.9% KLD: 0.089937
99.0% KLD: 0.024898
95.0% KLD: 0.010323
90.0% KLD: 0.006790
Median KLD: 0.001594
10.0% KLD: 0.000021
5.0% KLD: 0.000004
1.0% KLD: -0.000000
0.1% KLD: -0.000002
Minimum KLD: -0.000068

q4_0 phase arena fork vs F16 stock llama.cpp
====== KL divergence statistics ======
Mean KLD: 0.003028 ± 0.000055
Maximum KLD: 5.046766
99.9% KLD: 0.072822
99.0% KLD: 0.023757
95.0% KLD: 0.009928
90.0% KLD: 0.006562
Median KLD: 0.001540
10.0% KLD: 0.000020
5.0% KLD: 0.000003
1.0% KLD: -0.000000
0.1% KLD: -0.000003
Minimum KLD: -0.000039

The big outlier is the Maximum KLD which spiked to over 5.

My main issue though was my tests were extremely limited due to my limited hardware, and I was only able to test at around 100 000ctx due to system ram running out.

[Anbeeld]

@munchmunch77 can you please clarify what model and quant was used, what was the baseline (wikitext-2?), and have you actually offloaded q4_0 to RAM in the fork test, and how much of it?

[munchmunch77]

As explained, I am not well equipped to run these tests, and not even sure if I am doing it right. But the base model was Qwen3.8-27B-UD-Q3_K_XL at F16/F16 and q4_0 was compared to that on the same model for the fork and llama.cpp. I did set a value for kv-stream-stage-mib but it's unclear if it offloaded or not. Normally q4_0 does not fit but my 48GB RAM was maxed out during the tests, so I really don't have room to try anything else. You can disregard my tests for that reason, but I tried.

[Anbeeld]

The fork probably requires adding a custom argument that would force offload of KV cache into VRAM after a specific point, in order to be able to A/B test KLD at all. I'll try it when I have some free time.

[Anbeeld]

I had a slight misunderstanding of fork's mechanism, it can be tested as is with --kv-stream-stage-mib.

Result: it's clear now why there are no KLD benchmarks in the fork repo... it's quite lossy.

Same story every time. In this case, with horrific performance as well. You're literally better off dropping the KV cache to a lower quant.

KLD at -c 131072 / 128k, wikitext-2 test, Qwen3.5-4B-BF16

config	mean KLD	max KLD	99.9%	99.0%	same top p%	VRAM MiB	RAM MiB	s/pass
bf16 100%	-0.000000	0.000073	0.000051	0.000037	99.997	4096	0	62.3
bf16 50%	0.006173	19.76	0.960	0.0257	98.258	2048	4096	85.3
bf16 25%	0.012726	30.37	2.163	0.0571	97.169	1024	4096	104.7
q8_0 100%	0.003570	15.90	0.4223	0.009689	98.712	2176	0	61.3
q8_0 50%	0.008089	29.87	1.3326	0.033953	97.819	1093	2176	116.2
q8_0 25%	0.013393	30.74	2.4157	0.061644	97.029	549	2176	190.2
q5_0 100%	0.005457	24.22	0.6183	0.016506	98.043	1408	0	61.4
q5_0 50%	0.009251	29.80	1.4794	0.038484	97.356	707	1408	132.0
q5_0 25%	0.014260	30.82	2.5356	0.065611	96.703	355	1408	183.3
q4_0 100%	0.010044	25.94	1.222	0.0407	96.870	1152	0	62.4
q4_0 50%	0.014245	30.74	2.2191	0.057983	96.413	579	1152	78.4
q4_0 25%	0.018801	27.25	3.1603	0.086778	95.875	291	1152	88.1