fix(chat): free the conversation from the KV cache, and the cursor with the context

[andinux]

Two halves of one invariant: a conversation lives both in ai->chat and in the KV cache, and prev_len says how much of the rendered transcript is already in that cache. Neither llm_chat_free() nor llm_context_free() maintained the pairing, so each left the two sides disagreeing.

Found while answering "what can a user actually do when they hit Context size exceeded?" after #30.

llm_chat_free() didn't clear the cache

It reset prev_len to 0 and dropped the messages, but left the cache full. So the obvious recovery —

SELECT llm_chat_free();
SELECT llm_chat_create();

— failed with the very same Context size exceeded, the "new" chat having inherited the old one's tokens. Measured on 1.0.6: 254 tokens still resident after llm_chat_free().

llm_chat_create() and llm_chat_restore() both route through here and both wanted this already. Restore especially: it repopulates the history and zeroes prev_len, so it plainly meant "replay this transcript from scratch" and simply never cleared the cache to match.

llm_context_free() didn't reset the cursor

prev_len was left pointing at a context that no longer existed. Freeing and recreating therefore appeared to work while the model silently lost the conversation — the next turn sent only the newest message into an empty cache, and ai->chat.messages went on claiming turns the model had never re-read.

On 1.0.6: 93 tokens before the resize, 33 after the next turn. Post-fix: 93 → 0 → 117 — the transcript comes back.

prev_len belongs to the context, not the history, so freeing the context makes it zero. The history is deliberately kept — that's what separates freeing a context from freeing a chat, and it's what makes resizing a context mid-conversation work. No if (chat is active) guard is needed: with no chat, prev_len is already 0 and the reset is a no-op.

The three recovery paths, before and after

path	1.0.6	after
llm_chat_free() + llm_chat_create()	❌ Context size exceeded	✅ genuinely fresh chat
llm_context_free() + recreate	⚠️ works, model amnesia	✅ transcript replayed
save → prune → llm_chat_restore()	❌ needs a context rebuild too	✅ restore alone suffices

The third is the one users will want. Compaction falls out of the history being an ordinary SQL table:

SELECT llm_chat_save('before compaction');
DELETE FROM ai_chat_messages
 WHERE id NOT IN (SELECT id FROM ai_chat_messages ORDER BY id DESC LIMIT 4);
SELECT llm_chat_restore((SELECT uuid FROM ai_chat_history ORDER BY id DESC LIMIT 1));

22 messages pruned to 4 and replayed into a 50-token cache — and it no longer needs the context torn down and rebuilt around it.

Testing

One test per path. All three verified to fail against the 1.0.6 build, with the diagnostics quoted above:

chat_free_clears_kv_cache      cache still holds 254 tokens after llm_chat_free()
context_free_replays_chat      transcript was not replayed: 93 tokens before the
                               resize, only 33 after the next turn
chat_restore_compacts_context  fails outright: restore left the cache full

make test: 48/48 pass.

Version 1.0.6 → 1.0.7, patch.

---

Second round: review findings, all confirmed

An independent review raised three issues. All three were real; one is more severe than reported, and chasing the first turned up a fourth.

I need to retract something from the first version of this description, which claimed ai_free() left chat.vocab/chat.template dangling into a freed model. That is not a use-after-free: llm_chat_run re-fetches both from ai->model at the top of every turn (:1933-1934) and overwrites the stored copies before anything reads them. I could not reproduce a crash, and the code says why.

1. ai_free() bypassed the prev_len reset

The first commit taught llm_context_free() to reset it, but ai_free(free_llm=true) called llama_free(ai->ctx) directly — and both llm_model_free() and llm_model_load() route through it. The review's sharp detail: llm_context_create_with_options() only calls llm_context_free() when ai->ctx is non-NULL, so after a model free it is skipped entirely and prev_len survives against a brand-new empty cache.

Measured: 70 tokens of transcript before the reload, 28 after the next turn — the conversation silently gone.

Rather than add a third copy of the reset, each teardown now has one definition — ai_context_release() and ai_chat_release() — and every caller routes through it:

caller	calls
llm_context_free() (SQL)	ai_context_release()
llm_chat_free() (SQL)	ai_chat_release()
ai_free(free_llm) — model load/free	ai_context_release()
ai_free(free_ai) — connection close	ai_chat_release() + ai_context_release()

A context was destroyed in two places and a chat in two places, and only one of each maintained the invariant. That duplication was the actual defect.

A conversation deliberately survives a model swap — prev_len is reset with the context, so the next turn replays the transcript through the new model's template. That also sidesteps the byte-offset mismatch between two different templates, because the offset is zero.

2. Replaying a transcript submits it as one batch — and that aborts the process

The review expected -1: invalid input batch. It is not an error return. llama-context.cpp:1487 is:

GGML_ASSERT(n_tokens_all <= cparams.n_batch);

so exceeding n_batch kills the process:

llama-context.cpp:1487: GGML_ASSERT(n_tokens_all <= cparams.n_batch) failed
exit 134

Worth the correction — it changes the severity from a failed statement to a killed connection.

Only the context_size key ties n_batch to n_ctx, so llm_context_create_chat('n_ctx=4096') leaves n_batch at llama's default of 2048, and a single ~3000-token prompt reaches it with no replay involved — so this is pre-existing. But resetting prev_len turns a whole transcript into one batch, which widens it considerably. The chat path now chunks by n_batch the way llm_text_run() already did; only the final chunk's logits are sampled from.

3. llm_chat_free() left chat.batch dangling

Pointing into the token buffer it had just freed. No live dereference is reachable — every read is preceded by a tokenize that rewrites it — but it is the same invariant, so ai_chat_release() zeroes it.

4. Found while checking #1: closing a connection leaked the whole chat

llm_chat_free() was the only thing that ever released the history, buffers, prompt and token array — and it is a SQL function a caller may simply never invoke. Closing a connection leaked all of it: 29,792 bytes for a two-message chat. ai_free() now releases the chat when the connection goes away.

Every existing test called llm_chat_free() explicitly, which is exactly why nothing caught it.

Tests

Each verified against the previous build:

chat_survives_model_reload         "transcript was not replayed: 70 tokens
                                    before the reload, only 28 after"
chat_released_on_connection_close  "29792 byte(s) not released"
chat_prompt_larger_than_n_batch    exit 134 - the whole test binary aborts

make test: 51/51 pass.

🤖 Generated with Claude Code

https://claude.ai/code/session_012KRojTrn4Q4SaZQpqcwpAt

---

Third round: documentation

None of this behaviour was written down. The four entries involved read, in full:

entry	previous description
llm_model_free()	"Unloads the current model and frees associated memory."
llm_context_free()	"Frees the current inference context."
llm_chat_free()	"Ends the current chat session."
llm_model_load()	options only — nothing about what it releases

Since a conversation now deliberately outlives a model switch and a sampler deliberately does not, both are contract rather than incident, and shipping them undocumented is how they get reported as bugs later.

API.md gains a Lifecycle section up front:

you call	model	context / KV cache	sampler	chat history
llm_model_load() (again)	replaced	freed	freed	kept
llm_model_free()	freed	freed	freed	kept
llm_context_free()	kept	freed	kept	kept
llm_chat_free()	kept	cache cleared	kept	freed
closing the connection	freed	freed	freed	freed

plus the recipe for starting clean across a model switch, and the three ways out of a full context — free the chat, enlarge the context, or compact the history with SQL and restore it. The four entries above now say what they release and link to it.

Also corrects llm_chat_restore(), documented as returning NULL when it actually returns the number of messages restored — verified live: restore_returns=2 typeof=integer. Its entry now also records that it clears the KV cache, which is what lets the compaction recipe work without rebuilding the context (verified: llm_context_used() is 0 immediately after a restore).