Let the memvara provider select turns by role and fill a token budget

[sethigoldy]

Three switches on the memvara provider, every one off by default so the shipped provider renders byte-identically — pinned by a test that compares the default render against a fixture.

• MEMVARA_SEARCH_K — the k sent on every search (default 30).
• MEMVARA_ROLE_SELECT — off, user or route. user keeps user turns only. route keeps assistant turns only when the question asks what the assistant said — fourteen literal phrasings on word boundaries, case-insensitive, fitted on the 301 LongMemEval questions outside the 199-question sample and frozen — and user turns otherwise.
• MEMVARA_TOKEN_BUDGET — fills the turns block greedily in the order memvara returned, stopping at the first line that would push the whole block past the budget and always keeping the first, counted with the same o200k_base encoder the harness reports contextTokens with. Nothing is re-sorted.

Why: measured offline on the 199-question sample, gold turns render at a median of 165 tokens and sit at median rank 2, while the shipped 15-turn context costs 4,089; selecting on role is worth about eight times reranking at a 720-token budget, and the only question type a user-only rule loses is the one whose answer is something the assistant said — which the routing rule recovers. The judged arms these knobs exist for are memvara-routed720 and memvara-useronly720; findings are recorded in the memvara repository under docs/superpowers/plans/ dated 2026-09-03.

Two things a reviewer should look at rather than skim:

1. The token counter passes the special-token check off (encode(text, [], [])), matching the existing counter's fallback. One turn in the 500-question set carries a literal <|endoftext|>; without this a budget arm fails a question the control answers, and the gap would read as accuracy.
2. The two older truncation knobs (MEMVARA_HEAD_WHOLE, MEMVARA_TAIL_CHARS) now refuse values that are not non-negative integers instead of silently rendering the control on a typo.

The comment above the truncation knobs now carries both measurements: answer-string presence favours assistant turns; the dataset's own labels put 842 of 896 gold turns in user turns, 51 of the 54 assistant ones in single-session-assistant.

Every test was shown to fail under a mutation of the code it watches (18 mutations); the one gap that found — dropping the question from the renderer — got its own test. bun test src/providers/memvara: 60 pass. Prettier and tsc --noEmit clean on the changed files; six pre-existing files elsewhere still fail format:check and are untouched.

---

Review, and what it changed. A high-effort review filed twelve findings, nine confirmed. All are addressed in the second commit: one env parser for the six knobs with one rule for blank values; MEMVARA_TURNS_ONLY validated; both truncation knobs validated whenever either is set; the fill skips the encoder only when the block's UTF-8 byte length proves it fits (bytes, because a character can be several tokens); the recorded token count encodes with the special-token check off like the budget does; a block with a cut turn is no longer headed "verbatim"; selection that empties a turn list logs a warning; the k: 30 test is guarded against the arm's environment; the resolved settings are logged at init. The one finding not changed in code — the routing phrasings also match questions about the user's own history — is the benchmark-fitted behaviour the design records, and the docstring now says so.

The arms' renders are pinned byte-identical to the first commit by a fixture captured before any edit and re-derived independently from that commit; 4,560 differential renders agree. bun test src/providers/memvara: 93 pass.

Third commit — routing through a gateway that names models by provider. OPENAI_MODEL_PREFIX, unset by default, is prepended to the id sent for OpenAI-provider models only (openai/gpt-5.4), applied once in getModelConfig so both the answering client and the judge pick it up; the alias the run is started with and the checkpoint records stays gpt-5.4, so a run through such a gateway is comparable with one that went direct. Five tests; 98 pass overall.

Fifth commit — a second answer prompt. MEMVARA_ANSWER_PROMPT=v2 adds three reading bullets (later-dated value wins; list then count; ground advice in stated preferences) and one instruction (answer when the reasoning has the values), targeted at the thirteen judged misses where every gold excerpt was in the prompt. v1 is the default and byte-identical to before, pinned against the whole-prompt fixtures; the review checked 46,080 knob combinations against the previous file with zero mismatches. Selection and token counts do not change between the prompts. 108 tests.

Sixth commit: MEMVARA_CONTEXT_FILE

A judged arm can answer from pre-rendered context blocks: a JSONL file of {question, block} rows keyed by the question text with whitespace collapsed. When the knob is set, renderMemvaraContext returns the block verbatim and a question missing from the file aborts the run, so an arm never mixes overridden and provider-rendered contexts. The empty-context build the answer phase uses to measure contextTokens is left alone, so the recorded count is the block's size. Unset, rendering is byte for byte what it was; the pinned goldens still match. It exists so that a selection or rendering rule can be screened offline and judged before it is ported into the provider; the arms that used it are documented in the agent-memory repository's benchmark branch.

Seventh commit — MEMVARA_RANKED, and the offline scoring script for the parity run. One knob, off by default, adds ranked: true to the search request and nothing else — the request body is byte-unchanged with it unset, pinned by a test. A ranked hit's selected/span outcome now carries through onto the turn the provider returns, since the offline screen ahead of the judged run needs it to tell the kept set from the rest; a plain read or an older server leaves both fields off the turn entirely rather than null. memvaraProviderSettings refuses at startup a stack that pairs MEMVARA_RANKED=1 with MEMVARA_ROLE_SELECT or MEMVARA_CONTEXT_FILE, since either would hide the server's own order from what the parity run is measuring. src/scripts/score-ranked.ts is that offline screen: it reads a run checkpoint's search results, matches them against LongMemEval's own has_answer labels, and reports gold recall and the non-gold keep rate of the kept set in the shape the design's own extract.py prints. No judged or paid run has been made against this change yet — that is the next step, from the two branches this design also touches in the memvara core and cloud repositories. bun test: 135 pass.

Eighth commit — review of the seventh, fixing a scoring-population gap. A high-effort review of the ranked knob and the scoring script filed ten findings, eight confirmed and two plausible. All eight confirmed are fixed:

• score-ranked.ts counted every turn a ranked search returned as kept or missed, so MEMVARA_SEARCH_K set past the selector's own top_n folded turns the selector never evaluated (selected: null) into the same population as the ones it actually saw (true/false). That inflated the non-gold denominator and counted any gold turn ranked past top_n as missed even though the model never got the chance to keep it — the printed numbers were not comparable to extract.py's. selected: null is now skipped rather than scored.
• sawRanking used to go true on any selected !== undefined, so a run where every question was served unranked (selected: null throughout — no key, the switch, or a fallback) could still print a low recall that looked like a real result. It now requires a turn actually seen (true or false), so an all-null run reports nothing to score instead.
• Abstention questions (no has_answer turns) were excluded from the non-gold denominator entirely; extract.py scores every question's candidates regardless of whether it has gold turns, so they are folded back in.
• Gold matching now normalizes whitespace the way extract.py's own norm() does — a turn's text can pick up a line break between the dataset file and what memvara's search returns without the words changing.
• The MEMVARA_RANKED mutual-exclusion guard now also covers MEMVARA_TAIL_CHARS, which would otherwise truncate a turn by its position in the returned list regardless of whether the selector kept it — the one thing a ranked read promises never happens to a kept turn.
• The search response's selection object (outcome, candidates, kept) is now on the client type and carried into the search checkpoint as one extra context item, so a caller can tell an applied ranked call from a fallback or unconfigured one instead of inferring it from per-turn selected fields alone.
• Documented MEMVARA_RANKED, its exclusion rules, and the scoring script in src/providers/README.md, which previously named neither.

One finding is declined: matching a gold turn by its raw text alone can in principle collide with a non-gold turn sharing identical text in the same haystack. Fixing it needs a turn identity threaded from ingest through search results, which nothing in this pipeline carries today, and the dataset this scoring script runs against has zero such collisions across its 896 gold turns — recorded in the code rather than fixed.

Reviewed on the session model. bun test: 145 pass. tsc --noEmit and Prettier clean on every changed file.