From 70a63b606a779bc99a56666b4c19d630f9741044 Mon Sep 17 00:00:00 2001 From: Have-Purple <17238093+Have-Purple@users.noreply.github.com> Date: Sat, 14 Nov 2026 15:10:05 -0700 Subject: [PATCH] opencode: follow the session's model in the plugin's own calls With neither opencode_paid_model nor paid_model set, a side call ran 'opencode run' with no --model and took the CLI's configured default. That default can be a free model the session has already moved past; the free tier's 429 then looked like the plugin was broken while the session ran fine on a paid provider. The 2.x plugin now reports the model from session.step.started (provider/model#variant) with each hook; the session state keeps it, and the loop guard's judge and the micro-skills proposer use it where no paid_model is set. An explicit paid_model still wins, and the other hosts keep their own defaults (Claude's haiku): the fallback is opencode-only. The fold takes the model from the session state only: the session log keeps a bare model id, which 'opencode run --model' cannot resolve (it wants provider/model), so it is not a source. The usage read stays after ctx.host.settle() (Copilot writes it in the SessionEnd tail), and the model reaches the call through runner_for(role=..., session_model=...), as #185 landed. plugin/README.md and docs/PLUGIN.md say what a 2.x call runs on with no paid model configured. --- docs/PLUGIN.md | 8 ++- plugin/README.md | 11 +-- plugin/opencode/tokeneconomizer.ts | 13 ++++ tests/test_live_hooks.py | 102 +++++++++++++++++++++++++++ tests/test_opencode_plugin.py | 20 +++++- tokeneconomizer/plugin/controller.py | 5 ++ tokeneconomizer/plugin/fold.py | 13 +++- tokeneconomizer/plugin/hosts.py | 12 ++++ tokeneconomizer/plugin/judge.py | 7 +- tokeneconomizer/plugin/sidecall.py | 25 ++++--- 10 files changed, 196 insertions(+), 20 deletions(-) diff --git a/docs/PLUGIN.md b/docs/PLUGIN.md index 0ca122fc..a05cc749 100644 --- a/docs/PLUGIN.md +++ b/docs/PLUGIN.md @@ -95,7 +95,7 @@ event): no nudge is counted, and there is no one left to redirect. | Still stuck after the redirect | terminal abort, in autonomous runs | a one-line **notice** to the person, ending with how to stop the turn on that agent and in that mode (§3); the plugin never stops a turn, and `loopguard: stop` is read as `nudge` | Not every agent's hooks can stop a turn (Codex and Copilot ignore `continue: false`), and the person decides: the benchmark guard's own design note says "interactive callers should map it to a hand-off" | | Micro-skills' reading | a domain vocabulary per benchmark (Python/git, Node/web); the proposer's digest counts edited Python files, lists test invocations and flags Python import errors, and its prompt leads with the test command | **generic**, whatever the repository: every ecosystem's interpreters, package tools and test runners at once, builds and checks (`make`, `tsc`, `npm run build`, `cargo check`, ...) as their own intent, every edited file, and a language-neutral digest (failing commands with a normalized error line, paths relative to the repository) and prompt, with tests one topic among several. `vocabulary=python-git` or `node-web` selects an experiment's | A person's repository may be in any language, or several; the experiments keep the paper's reading (its prompt is pinned by a test) | | Micro-skills | on in the arms that measured it: the proposer every 5 tasks (on Codex, every 10, over the free layer) | **off by default**; `on` runs the free layer under the proposer, every `microskills_propose_every` sessions (3 by default: after the 2nd, 5th, 8th, ...), since a real session usually carries more work than one benchmark task | The proposer calls a model on the person's plan, and what it injects is derived from command output a repository controls: turning it on is the person's call. In the experiments the free layer alone formed almost nothing once the harness set each environment up properly, so `on` keeps the proposer | -| Optimizer model | the agent's own model, so overhead could not look cheaper than its arm | the cheapest capable model on the user's own agent CLI (`paid_model`; when empty, `haiku` on Claude Code, the agent's default model on Codex and Pi, and on Copilot a concrete `COPILOT_MODEL` or a BYOK provider's Haiku): the proposer when micro-skills is on, the loop guard's judge and redirect writer whenever the guard is on | Every call is still charged to the session ledger | +| Optimizer model | the agent's own model, so overhead could not look cheaper than its arm | the cheapest capable model on the user's own agent CLI (`paid_model`; when empty, `haiku` on Claude Code, the agent's default model on Codex and Pi, the session's own model on opencode 2.x, and on Copilot a concrete `COPILOT_MODEL` or a BYOK provider's Haiku): the proposer when micro-skills is on, the loop guard's judge and redirect writer whenever the guard is on | Every call is still charged to the session ledger | ## 3. Verified agent behaviour @@ -356,8 +356,10 @@ settings would be): only after `opencode service restart`. * Model side calls run `opencode run` with `--pure` (1.x) or `--standalone` (2.x, so the shared service's other plugins — this one included — cannot fire inside them) and `bash`, `edit`, - `webfetch`, `websearch` denied; a `read`, `task` or `*` denial stalls 1.19.5 at start-up. On - 2.x a reply that used no tool carries no usage, so its cost is estimated. + `webfetch`, `websearch` denied; a `read`, `task` or `*` denial stalls 1.19.5 at start-up. With + no `opencode_paid_model` or `paid_model`, a 2.x call runs on the model the session itself is + on, not the CLI's configured default. On 2.x a reply that used no tool carries no usage, so + its cost is estimated. **End to end, as a new user, 2026-10-14** — on each agent a fresh home followed `README.md` and this plugin's guide step by step (Claude Code 2.1.300, Codex 0.153.1, Copilot CLI 1.0.83, Pi diff --git a/plugin/README.md b/plugin/README.md index 0395d666..24600884 100644 --- a/plugin/README.md +++ b/plugin/README.md @@ -432,10 +432,13 @@ when its turn ends and a deleted session when it is deleted. Quitting the TUI gi chance to say so: that session is folded by the next session's start, once its export has been quiet for 10 minutes. -**Model calls.** The proposer and the loop guard's judge run through `opencode run` on your -default opencode model (`paid_model` names another): tools that change anything are denied, -the session is deleted afterwards, and no other plugin runs (`--pure` on 1.x; `--standalone` on -2.x). On 2.x a reply that used no tool carries no usage, so its cost is estimated. +**Model calls.** The proposer and the loop guard's judge run through `opencode run` on +`opencode_paid_model` (`paid_model` names another model for every agent). With neither set they +follow the model the session itself is running — the 2.x plugin reports it from +`session.step.started` — instead of the CLI's configured default, which can be a free model the +session has already moved past. Tools that change anything are denied, the session is deleted +afterwards, and no other plugin runs (`--pure` on 1.x; `--standalone` on 2.x). On 2.x a reply +that used no tool carries no usage, so its cost is estimated. **Verified live,** on opencode 1.19.5 and 2.2.0 (Python 3.12), with a scripted local model endpoint and, on 1.19.5, the free `opencode/big-pickle` model; the 2.x entry is also exercised diff --git a/plugin/opencode/tokeneconomizer.ts b/plugin/opencode/tokeneconomizer.ts index ac9ffa69..6ef0aa82 100644 --- a/plugin/opencode/tokeneconomizer.ts +++ b/plugin/opencode/tokeneconomizer.ts @@ -342,11 +342,15 @@ const setupV2 = async (ctx: any) => { const records = new Map(); // sessionID -> messages, in the export's shape const steps = new Map(); // assistantMessageID -> assistant message const names = new Map(); // tool call id -> tool name + // The model each session actually runs on, in the CLI's `provider/model#variant` form: the + // plugin's own calls follow it, rather than whatever the CLI's default happens to be. + const models = new Map(); const base = (sessionID: string, transcript = "") => ({ session_id: sessionID, transcript_path: transcript, cwd: directory, + ...(models.get(sessionID) ? { model: models.get(sessionID) } : {}), }); const notify = (text: unknown) => { if (typeof text === "string" && text) console.error(text); @@ -495,6 +499,15 @@ const setupV2 = async (ctx: any) => { if (t === "session.step.started") { const m = assistant(id, p.assistantMessageID); m.info.modelID = p.model?.id ?? p.model?.modelID ?? m.info.modelID; + // `session.step.started` names the model the turn runs on; the plugin's own calls use + // it when no `paid_model` is configured, so a paid provider is not looked past. + const provider = p.model?.providerID ?? p.model?.provider; + const modelID = p.model?.id ?? p.model?.modelID; + if (provider && modelID) + models.set( + id, + `${provider}/${modelID}${p.model?.variant ? `#${p.model.variant}` : ""}`, + ); } else if (t === "session.step.ended") { const m = assistant(id, p.assistantMessageID); m.info.cost = p.cost ?? 0; diff --git a/tests/test_live_hooks.py b/tests/test_live_hooks.py index 0513edaa..e39e1f21 100644 --- a/tests/test_live_hooks.py +++ b/tests/test_live_hooks.py @@ -1698,3 +1698,105 @@ def fail(tools, **kw): monkeypatch.setattr(toolcheck, "measure", fail) assert cli.main(["--host", "claude", "--cwd", repo, "toolprune", "measure", "Workflow"]) == 1 assert capsys.readouterr().out.strip() == "not measured: `claude -p` exited 1: boom" + + +# ---- the model a session runs on ----------------------------------------------------------- +def test_the_model_the_plugin_sends_is_kept_in_the_session_state(env, repo): + """`session.step.started` names the model; the 2.x plugin sends it with every hook, and the + session keeps it so a later judge or fold does not fall back to the CLI's default.""" + from tokeneconomizer.plugin import hosts + + assert hosts.OpenCode().session_model({"model": "opencode-go/m#high"}) == "opencode-go/m#high" + assert hosts.Host().session_model({"model": "opencode-go/m#high"}) == "" + base = {"session_id": "ses_model", "cwd": repo, "model": "opencode-go/m#high"} + call(env, dict(base, hook_event_name="UserPromptSubmit", prompt="hi", prompt_id="t1"), + host="opencode") + call(env, dict(base, hook_event_name="Stop", prompt_id="t1", stop_hook_active=False), + host="opencode") + s = store_for(env, repo, host="opencode").read("sessions/ses_model.json") + assert s["model"] == "opencode-go/m#high" + + +def test_a_side_call_follows_the_session_model_only_on_opencode(monkeypatch): + """No `paid_model`: opencode takes the session's own model. An explicit setting still wins, + and another host's default (Claude's haiku) is not replaced by a session model.""" + from tokeneconomizer.plugin import sidecall + from tokeneconomizer.plugin.config import LiveConfig + + seen = [] + monkeypatch.setattr(sidecall, "opencode_call", + lambda prompt, model="", **kw: (seen.append(model), ("", 0))[1]) + monkeypatch.setattr(sidecall, "claude_call", + lambda prompt, model="", **kw: (seen.append(model), ("", 0))[1]) + cfg = LiveConfig() + sidecall.runner_for("opencode", cfg, session_model="opencode-go/m#high")("x") + assert seen[-1] == "opencode-go/m#high" + cfg.opencode_paid_model = "explicit/model" + sidecall.runner_for("opencode", cfg, session_model="opencode-go/m#high")("x") + assert seen[-1] == "explicit/model" + sidecall.runner_for("claude", cfg, session_model="opencode-go/m#high")("x") + assert seen[-1] == "haiku" + + +def test_the_judge_follows_the_session_model(env, repo, monkeypatch): + """The loop guard's judge runs through the model the session is on.""" + from tokeneconomizer.plugin import judge, sidecall + + seen = {} + + def fake_runner_for(host, cfg, role=None, session_model=""): + seen[role] = session_model + return lambda prompt, _m="agent": ("", 0) + + monkeypatch.setattr(sidecall, "runner_for", fake_runner_for) + monkeypatch.setattr(sidecall, "paused", lambda *a, **k: None) + ctx = LiveContext.build(hosts.OpenCode(), repo, env) + ctx.store.write("sessions/ses_j.json", {"session_id": "ses_j", + "model": "opencode-go/j#high"}) + ctx.store.write("judge/ses_j.request.json", {"id": "job1", "digest": "d", "threshold": 0}) + judge.run_job(ctx, "ses_j", "job1") + # The judge follows the session; the nudge writer keeps the host's own default. + assert seen["judge"] == "opencode-go/j#high" + assert seen["nudge"] == "" + + +def test_a_fold_takes_the_model_from_the_session_state(env, repo, tmp_path, monkeypatch): + """The fold hands the proposer the model the plugin's hooks recorded for the session. The + log names another model, so a fallback to it would show: `opencode run --model` needs + `provider/model`, and the log keeps the bare id.""" + import tokeneconomizer.microskills as ms + from tokeneconomizer.plugin import fold, sidecall + + from fixtures import opencode_session_export + + seen = {} + + def fake_runner_for(host, cfg, role=None, session_model=""): + seen["model"] = session_model + return lambda prompt, _m="agent": ("", 0) + + monkeypatch.setattr(sidecall, "runner_for", fake_runner_for) + + class FakeStage: + def __init__(self, *a, **kw): + pass + + def after_task(self, *a, **kw): + return 0, "fake" + + def render(self): + return "" + + def injected_ids(self): + return [] + + monkeypatch.setattr(ms, "PrefixStage", FakeStage) + export = tmp_path / "ses_s.json" + export.write_text(opencode_session_export("ses_s"), encoding="utf-8") + ctx = LiveContext.build(hosts.OpenCode(), repo, env) + ctx.config.microskills = "envfacts+distiller" + ctx.store.write("sessions/ses_s.json", {"session_id": "ses_s", + "model": "opencode-go/s#high"}) + rec = fold.fold_session(ctx, "ses_s", transcript_path=str(export)) + assert rec is not None + assert seen["model"] == "opencode-go/s#high" diff --git a/tests/test_opencode_plugin.py b/tests/test_opencode_plugin.py index 2064e5e6..31fcd0dc 100644 --- a/tests/test_opencode_plugin.py +++ b/tests/test_opencode_plugin.py @@ -54,8 +54,12 @@ location: { directory: process.env.TE_WORK }, session: { hook: async (name: string, fn: any) => { hooks["session." + name] = fn; } }, tool: { hook: async (name: string, fn: any) => { hooks["tool." + name] = fn; } }, - event: { subscribe: async function* () {} }, + event: { subscribe: async function* () { + yield { type: "session.step.started", properties: { sessionID: "ses_t", assistantMessageID: "a1", model: { id: "te-model", providerID: "te", variant: "high" } } }; + } }, }); +// The event stream is consumed off the setup path; let it land before the hooks that carry it. +await new Promise((r) => setTimeout(r, 200)); await hooks["session.prompt"]({ sessionID: "ses_t", messageID: "m1", prompt: { text: "say hi" } }); await hooks["tool.execute.after"]({ sessionID: "ses_t", status: "completed", tool: "bash", id: "c1", input: { command: "echo hi" }, result: { content: "hi", metadata: { exit: 0 } } }); @@ -113,6 +117,20 @@ def test_the_plugins_hooks_open_no_console_window(tmp_path): assert all(r["visible"] for r in control if r["hook"]), control +def test_the_plugin_sends_the_model_the_session_runs_on(tmp_path): + """`session.step.started` names the model the turn runs on; the plugin passes it with the + hooks and the session state keeps it, so the judge and the fold need not fall back to the + CLI's configured default.""" + _run( + tmp_path, + os.path.join(REPO, "plugin", "opencode", "tokeneconomizer.ts"), + "model", + ) + (state,) = (tmp_path / "te-model").glob("repos/*/opencode/sessions/ses_t.json") + doc = json.loads(state.read_text(encoding="utf-8")) + assert doc.get("model") == "te/te-model#high", doc + + class _Chat(BaseHTTPRequestHandler): """An OpenAI-compatible chat endpoint that answers every request with "done".""" diff --git a/tokeneconomizer/plugin/controller.py b/tokeneconomizer/plugin/controller.py index cd338f70..1e4919fa 100644 --- a/tokeneconomizer/plugin/controller.py +++ b/tokeneconomizer/plugin/controller.py @@ -58,6 +58,11 @@ def _session(self): "cwd": self.host.cwd(self.p, self.ctx.env), "turns": [], "nudges": 0, "aborts": 0, "judge_tokens": 0.0, "writer_tokens": 0.0, **self.host.session_state(self.ctx.env)} + # The model the session runs on, when its payload names one: the plugin's own calls + # follow it (`sidecall`) instead of the CLI's default. + model = self.host.session_model(self.p) + if model: + s["model"] = model return s def _save(self, s): diff --git a/tokeneconomizer/plugin/fold.py b/tokeneconomizer/plugin/fold.py index efe98b79..d493e806 100644 --- a/tokeneconomizer/plugin/fold.py +++ b/tokeneconomizer/plugin/fold.py @@ -72,7 +72,8 @@ def _microskills_config(ctx, profile): max_tokens=ctx.config.microskills_max_tokens) -def _fold_microskills(ctx, adapter, run_dir, session_id, injected=None, imported=False): +def _fold_microskills(ctx, adapter, run_dir, session_id, injected=None, imported=False, + model=""): profile = ctx.config.microskills if profile == "off": return None @@ -83,7 +84,8 @@ def _fold_microskills(ctx, adapter, run_dir, session_id, injected=None, imported # layer learns from it, and the proposer waits for sessions the plugin sees end. runner = None if icfg.paid and not imported: - runner = (sidecall.runner_for(ctx.host.name, ctx.config, role="proposer") + runner = (sidecall.runner_for(ctx.host.name, ctx.config, role="proposer", + session_model=model) if not sidecall.paused(ctx.host.name, ctx.root) else sidecall.local_runner("proposer", ctx.config)) stage = PrefixStage(icfg, ctx.repo.label, ctx.mech_dir, False, runner=runner, @@ -540,9 +542,14 @@ def fold_locked(ctx, session_id, transcript_path=None, imported=False, span=None catalog = {} if imported else load_catalog(ctx.root, ctx.host.name) adapter = LiveAdapter(ctx.host, ctx.repo.root, vocabulary=ctx.config.vocabulary, catalog=catalog, protect=ctx.config.toolprune_protect) + # The model this session ran on, as its own hooks recorded it: the proposer follows + # it instead of the CLI's configured default. The session log keeps a bare model id + # (no provider), which `opencode run --model` cannot resolve, so it is not a source. + model = str(sess.get("model") or "") with ctx.store.locked("mechanisms"): inj = _fold_microskills(ctx, adapter, run_dir, session_id, - injected=sess.get("injected_ids"), imported=imported) + injected=sess.get("injected_ids"), imported=imported, + model=model) # A session that ran no prompt is no evidence about the tools. unt = None if prompts == 0 else _fold_toolprune( ctx, adapter, run_dir, session_id, transcript=transcript, diff --git a/tokeneconomizer/plugin/hosts.py b/tokeneconomizer/plugin/hosts.py index 5351a9e7..fe42c3d3 100644 --- a/tokeneconomizer/plugin/hosts.py +++ b/tokeneconomizer/plugin/hosts.py @@ -216,6 +216,13 @@ def session_state(self, env): """What a session's state keeps from the environment of its first hook.""" return {} + def session_model(self, p): + """The model the session runs on, in the agent CLI's own `--model` form, or "" when the + payload does not say. A host that reports it lets the plugin's own calls follow the + session instead of whatever model the CLI would pick by default -- which is how a + paid provider the person selected can be looked past.""" + return "" + def tool_usage(self, trace_path): return trace.tool_usage(trace_path) @@ -486,6 +493,11 @@ class OpenCode(Host): def turn_key(self, p): return str(p.get("prompt_id") or "") + def session_model(self, p): + """`session.step.started` names the model on `provider/model#variant`; the plugin sends + the session's own, so the plugin's calls do not fall back to the CLI default.""" + return str(p.get("model") or "") + def headless(self, env, transcript=None): return str(env.get("TE_OPENCODE_MODE", "")).strip() == "run" diff --git a/tokeneconomizer/plugin/judge.py b/tokeneconomizer/plugin/judge.py index 24b841d8..37e7b301 100644 --- a/tokeneconomizer/plugin/judge.py +++ b/tokeneconomizer/plugin/judge.py @@ -14,6 +14,7 @@ for the session: a verdict that lands after its turn -- or its session -- has ended was still paid for, and must not drop out of the account. """ +import os import time from . import sidecall @@ -68,10 +69,14 @@ def run_job(ctx, session_id, job_id): if req.get("id") != job_id: # superseded by a newer request return None paused = sidecall.paused(ctx.host.name, ctx.root) + # The model the session runs on, so the judge follows it rather than the CLI's default. + sess = ctx.store.read(os.path.join("sessions", f"{session_id}.json")) or {} + session_model = str(sess.get("model") or "") # Each call asks for its own runner: `local_roles` may put the judge on-device and leave # the redirect on the agent's CLI, or the other way round. A paused host's CLI is skipped, # but a local runner (which needs no login) still answers. - runner = sidecall.runner_for(ctx.host.name, ctx.config, role="judge") if not paused \ + runner = sidecall.runner_for(ctx.host.name, ctx.config, role="judge", + session_model=session_model) if not paused \ else sidecall.local_runner("judge", ctx.config) writer = sidecall.runner_for(ctx.host.name, ctx.config, role="nudge") if not paused \ else sidecall.local_runner("nudge", ctx.config) diff --git a/tokeneconomizer/plugin/sidecall.py b/tokeneconomizer/plugin/sidecall.py index 3589b309..2deaf7fc 100644 --- a/tokeneconomizer/plugin/sidecall.py +++ b/tokeneconomizer/plugin/sidecall.py @@ -29,9 +29,11 @@ extension registers needs that extension loaded (`pi_extensions`); `pi_cli` names the `pi` to run, where the first on PATH is a wrapper the calls should not go through. - * opencode: `opencode run --format json` in a throwaway directory, on the person's default - model, with `bash`, `edit`, `webfetch` and `websearch` denied, no other plugin (`--pure` on - 1.x, `--standalone` on 2.x) and its session deleted afterwards. + * opencode: `opencode run --format json` in a throwaway directory, on `opencode_paid_model` + (or `paid_model`); with neither, on the model the session itself is running -- the 2.x plugin + reports it from `session.step.started` -- rather than the CLI's configured default, which can + be a free model the person has already moved past. `bash`, `edit`, `webfetch` and `websearch` + denied, no other plugin (`--pure` on 1.x, `--standalone` on 2.x), session deleted afterwards. * Grok: `grok -p --output-format json` in a throwaway directory, on the person's default model, with only the read-only `read_file` tool allowed (`--tools read_file`), grok's MCP dispatchers removed (`--disallowed-tools`) and the shell, edit, write, web and subagent @@ -982,7 +984,7 @@ def call_name(host_name): return _CALLS.get(host_name, f"{host_name} -p") -def runner_for(host_name, config, role=None): +def runner_for(host_name, config, role=None, session_model=""): """The optimizer callable for this host. Each host's calls go through that host's own CLI, never another agent's. The caller decides whether a call is allowed: micro-skills' proposer when micro-skills is on, the loop guard's judge when the guard is. None for a host in @@ -990,8 +992,13 @@ def runner_for(host_name, config, role=None): `role` (`judge`, `nudge`, `proposer`) lets `local_model` run that call on-device instead (`local_runner`), falling back to the host's CLI when the local call fails; on a host in - `NO_SIDE_CALLS` the local model is the only runner.""" - call = _host_call(host_name, config) + `NO_SIDE_CALLS` the local model is the only runner. + + `session_model` is the model the session itself runs on, when the host reports it: on + opencode it stands in where no `paid_model` is configured, so a provider the person picked + in the session is not looked past in favour of the CLI's default. The other hosts name + their own default (Claude's `haiku`), which a session's model does not replace.""" + call = _host_call(host_name, config, session_model=session_model) host = None if call is None else _noted(host_name, call) if role is not None: local = local_runner(role, config, fallback=host) @@ -1000,7 +1007,7 @@ def runner_for(host_name, config, role=None): return host -def _host_call(host_name, config): +def _host_call(host_name, config, session_model=""): if host_name in NO_SIDE_CALLS: return None model = getattr(config, f"{host_name}_paid_model", "") or config.paid_model @@ -1012,7 +1019,9 @@ def _host_call(host_name, config): return lambda prompt, _m="agent": pi_call(prompt, model=model, cli=cli, extensions=extensions) if host_name == "opencode": - return lambda prompt, _m="agent": opencode_call(prompt, model=model) + # No configured model: follow the session's own, not the CLI's default (which can be a + # free model the person already moved past). + return lambda prompt, _m="agent": opencode_call(prompt, model=model or session_model) if host_name == "cursor": return lambda prompt, _m="agent": cursor_call(prompt, model=model) if host_name == "grok":