Skip to content

fix: restore on an unloaded engine, and trust a finished stream - #1

Open
frodosik wants to merge 1 commit into
tfyl:mainfrom
frodosik:fix/unloaded-restore-and-sse-client-gone
Open

frodosik wants to merge 1 commit into
tfyl:mainfrom
frodosik:fix/unloaded-restore-and-sse-client-gone

Conversation

@frodosik

@frodosik frodosik commented Nov 16, 2026 •

Copy link
Copy Markdown

Summary

Three bug fixes for production failures seen with --lazy / idle_unload_s and SSE agent harnesses.

1. An unloaded engine made every session file look foreign (internal/strata/client.go, internal/store/store.go, internal/proxy/cache.go).
/metrics of an unloaded engine reports no context and no kv, so the fingerprint hash
contained the literal <nil>; learn() recorded that as a new fingerprint and
KeepFingerprint deleted every .session file on disk — the file that should restore the
conversation the engine just unloaded. Engine() now refuses an incomplete /metrics, and
a switch against an unloaded engine restores with the fingerprint it last learned, or the
newest file's, so Lookup can see the file without waking Strata:

prepare, on a switch
  en, err = Engine()
- if err: miss
+ if err and status says unloaded
+   fp = last fingerprint learned, else Store.AnyFingerprint()
+   if fp == "" -> miss engine_unloaded_cold (request passes through, no wakeup)
+ else -> miss engine_<kind>

The restore itself wakes the engine. Strata answers the refused slot call with
503 "not loaded", and a 1-token request loads the model:

restore(file)
  n = Restore(file)
+ if IsUnloaded(err)          # 503 "not loaded"
+   err = Wakeup()            # 1-token non-streaming /v1/chat/completions
+   if err: report the refused wakeup, not the 503
+   en, err = Engine()
+   if err: report it
+   if learn(en) != file.Fingerprint:  # the model changed while unloaded
+     log "woken into another engine"; failures{restore,fingerprint}; return false
+   Restore(file)             # retried once

Store.AnyFingerprint() is the fingerprint of the most recently used file on disk; two
files with the same mtime break the tie by name, so the answer does not depend on map
order. A refused wakeup, or a woken engine of another fingerprint, is logged and counted as
what it is, not as the initial 503.

2. A client that closes the socket after the stream finished counted as a cancellation
(internal/proxy/proxy.go). SSE harnesses (OpenAI SDKs, agent loops) read to [DONE] and
close; the proxy saw the request context cancelled, set client_gone, dropped live, and
the next turn was read fresh — the answer it had just served was never trusted, and the
idle save after a completed turn never ran. Every chunk written is scanned for a terminal
event, and the answer is remembered in a sticky flag:

statusWriter
  tail = "\n"                    # seeded once, so the first chunk starts a line

statusWriter.Write(chunk)
  if not done:
    scan tail + chunk for a terminal event
+   if found: done = true, tail = nil   # sticky, and no scanning or allocation after it
    tail = last 64 bytes of (tail + chunk)

defer of a generation request
- gone = ctx.Err() != nil
+ gone = ctx.Err() != nil && !sw.done

A marker is searched in the whole chunk, not in a fixed window at the end, because
response.completed is followed by the entire response object — kilobytes that push the
marker out of any tail. [DONE] is matched as \ndata: [DONE], and the newline is
seeded once at stream start: trimming the tail to 64 bytes can leave it starting right at
a quoted data: [DONE] inside JSON content, and the next chunk must not invent a line
start for it. Inside JSON strings newlines are escaped, so a model quoting
data: [DONE] in streamed content is not the end of the stream.

cache.finish keeps setting c.live = nil on a genuine mid-stream cancel, so a cut-off
generation is never trusted as live state.

3. A truncated .session file trapped the conversation (internal/proxy/cache.go). A
file whose name claims more tokens than the engine restores from it — a save cut short, an
interrupted rename — is Covered for every later turn, so every save is skipped as
on_disk within regrow-tokens and the conversation never reaches disk again. A restore
that returns fewer than half the tokens in the name forgets the file:

restore(file)
- Touch(file)
+ if n < file.Tokens/2: Forget(file, "truncated")   # the turn still serves
+ else: Touch(file)

stratatest gained what the tests need: an unloaded engine (/metrics without context
and kv, slot calls refused 503), a request that loads it back, a refused generation
request (GenStatus), SlotMessage, Finish, Content, TailDelay, terminal events in
all three dialects with the Responses response.completed carrying a multi-KB payload,
and a restore that reports its own token count instead of a fixed 20000. docs/design.md
gets the cases with their tests; CHANGELOG.md gets a ### Fixed section. Request bodies
are not touched.

Evidence

Production symptom, the turn after the engine unloaded (file names and timings shortened):

{"level":"INFO","msg":"evict","file":"dc-98764478cabe-…-20000.session","reason":"fingerprint"}
{"level":"INFO","msg":"request","path":"/v1/chat/completions","action":"miss","reason":"engine_http_503","client_gone":true}

After the fix, the same turn restores from disk (from TestColdEngineWakesAndRestores):

{"level":"INFO","msg":"waking engine for restore","tokens":20000}
{"level":"INFO","msg":"restore","tokens":20000,"mb":0,"ms":1}
{"level":"INFO","msg":"request","path":"/v1/chat/completions","action":"disk","reason":"restored","client_gone":false}

go vet ./... clean, golangci-lint (v2.12.2, repo config) 0 issues, go test -race -count=1 ./... all green:

--- PASS: TestEngineUnloaded               /metrics of an unloaded engine: no fingerprint
--- PASS: TestIsUnloaded                   503 "not loaded" vs 503 "refused" vs 507
--- PASS: TestWakeup                       one generation request, and a server that is down
--- PASS: TestAnyFingerprint               the newest file's fingerprint, empty store ""
--- PASS: TestAnyFingerprintTie            the same mtime breaks by name, 20 randomized loads
--- PASS: TestStatusWriterDone             tail cut right at a quoted data: [DONE]: done stays false,
                                           the real data: [DONE] makes it true
--- PASS: TestColdEngineWakesAndRestores   restore refused -> wakeup -> fingerprint check -> restore -> disk
--- PASS: TestColdEngineNoFilePassesThrough  no file -> pass through, no wakeup
--- PASS: TestUnloadedEngineKeepsFiles     the files of the loaded engine survive
--- PASS: TestWokenEngineWithNewFingerprint  fingerprint checked before the restore is retried
--- PASS: TestWakeupFailurePropagates      refused wakeup: reason http_502, not http_503
--- PASS: TestClientGoneAfterCompletion    [DONE], finish_reason tool_calls, message_stop, response.completed;
                                           the completed event is > 1 KB and the marker is buried,
                                           the state stays live and the idle save runs
--- PASS: TestClientCancel                 mid-stream cancel: client_gone=true, live=nil (unchanged)
--- PASS: TestTruncatedSessionFile         n < Tokens/2 -> forgotten, then saved afresh

TestClientGoneAfterCompletion keeps the cancellations honest: a cancel while the stream
runs, and a cancel at an answer that quotes data: [DONE] inside "content", are both
recorded client_gone=true and the next turn is read fresh.

Merge Danger

Door: two-way. One commit, no flag, no wire-format, storage-format, metric-name, or
config change; reverting it restores the previous behaviour.

Blast Radius: cache decisions only — which fingerprint a restore matches against,
when the engine is woken (one extra 1-token request per restore against an unloaded
engine), when client_gone is cleared, and when a file is forgotten. Prompt and answer
text, request bodies and the log fields are unchanged, except client_gone's value.

Checked against Strata v0.1.45.

@frodosik
frodosik force-pushed the fix/unloaded-restore-and-sse-client-gone branch from 6a88da0 to 7974e44 Compare November 16, 2026 18:57
An engine unloaded by --lazy / idle_unload_s reports no context and no KV type in
/metrics. Engine() read that as a fingerprint, so a request against an unloaded engine
evicted every session file. It now refuses an incomplete /metrics; a restore against an
unloaded engine uses the last fingerprint learned, or the newest file on disk, wakes the
engine with a 1-token request, checks that it is the engine the file was written for,
and retries the restore once. A refused wakeup, or a woken engine with another
fingerprint, is reported as what it is, not as the initial 503. A cold engine with no
files passes through untouched. Store.AnyFingerprint() breaks a tie between two files of
the same mtime by name, so the answer does not depend on map order.

A client that closes the connection right after the stream's terminal event got a
complete answer, not a cancellation. Every chunk written is scanned for the terminal
event and a sticky flag remembers it, so a Responses completion followed by kilobytes
counts; [DONE] is anchored to its line, and the rolling tail is seeded with a newline
once at stream start, so trimming it cannot invent a line start for a marker quoted
inside streamed content. A cancel while the stream runs is still not trusted as live
state.

A file that restores less than half the tokens its name claims is forgotten, so the
conversation is saved afresh instead of being skipped as already on disk.

Checked against Strata v0.1.45. New tests in internal/strata, internal/store and
internal/proxy, against stratatest, which now simulates an unloaded engine.
@frodosik
frodosik force-pushed the fix/unloaded-restore-and-sse-client-gone branch from 7974e44 to 130f8d0 Compare November 16, 2026 19:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant