Skip to content

Commit 355b3ef

Browse files
committed
Add cli:zcode as an engine judge route through the production transport
IMPROVE_JUDGE=cli:zcode:GLM-5.3 runs judge calls through ZCodeTransport.complete — the same tool-less one-shot the kb gate's ModelGateway uses (empty scratch cwd, every native tool removed, containment audit, served-model assertion, ambient ZCODE_REASONING_LEVEL) — so the experiment-internal gate judge is the production instrument itself, not an API proxy of it. Subscription billing keeps usd=0; calls reserve/settle against the judge-call envelope like any CLI judge and record as model_call with the served model. The trusted kb-intake driver's Gateway picks this up unchanged (judge_spec_from now accepts the provider; unknown providers still refuse). Judge-tuning baseline test now pins recorded digests instead of git HEAD (which no longer carries the prompts).
1 parent 7aa922d commit 355b3ef

4 files changed

Lines changed: 216 additions & 34 deletions

File tree

‎doc/guide/self-evolution.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,11 @@ export TRACE_STORE_ROOT=/srv/copilot/traces
2727
export ECO_MODEL=YOUR_PINNED_GENERATOR_MODEL
2828
export PERFORMANCE_MODEL=YOUR_PINNED_INVESTIGATOR_MODEL
2929
export IMPROVE_JUDGE=api:YOUR_PINNED_JUDGE_MODEL
30+
# 判官也可走订阅 CLI:cli:<cursor|codex|claude|zcode>:<model>。cli:zcode 经生产
31+
# ZCodeTransport 的 tool-less 一次性调用执行(与 kb 质量门同一工具:空 scratch 目录、
32+
# 移除全部原生工具、容器审计、served-model 断言;推理档位取环境 ZCODE_REASONING_LEVEL),
33+
# 订阅计费只计入判官调用次数预算,USD 记 0。
34+
# export IMPROVE_JUDGE=cli:zcode:GLM-5.3
3035

3136
infermatrix-copilot improve workflows list
3237
infermatrix-copilot improve workflows check --workflow pr-review.agent.review_diff

‎src/infermatrix_copilot/improve/judges.py‎

Lines changed: 65 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -1,18 +1,25 @@
11
"""Judge calls, all under the engine's ledger (design §10).
22
33
The engine never starts a judge subprocess directly; every judge call goes
4-
through `run_judge`, which has two paths:
4+
through `run_judge`, which has three paths:
55
66
* **API judges** through `LLM.create` — captured as `model_call` records by
77
the choke point and priced against the dollar envelope (P3 governor);
88
* **CLI judges** (a subscription harness such as `cursor-agent`) through
99
`governed_subprocess` — one call reserved against the judge-call envelope,
1010
tool-less by construction (fresh empty workspace, `--mode ask`, any tool
1111
call in the stream fails the verdict), recorded as a `model_call` with
12-
``usd=0``.
13-
14-
A `Governor` (P3) gates both; without one the calls are unmetered but still
15-
recorded. `parse_verdict` extracts the first JSON object of a reply.
12+
``usd=0``;
13+
* **zcode judges** (``cli:zcode:<model>``) through the production
14+
`ZCodeTransport.complete` one-shot — the same tool-less instrument the
15+
knowledge gate's `ModelGateway` uses (empty scratch cwd, every native
16+
tool removed, containment audit, served-model assertion; the reasoning
17+
level is the ambient `ZCODE_REASONING_LEVEL`). Reserved and settled
18+
against the judge-call envelope like any CLI judge; subscription billing,
19+
so ``usd=0`` and only the call count is governed.
20+
21+
A `Governor` (P3) gates all of them; without one the calls are unmetered but
22+
still recorded. `parse_verdict` extracts the first JSON object of a reply.
1623
"""
1724

1825
from __future__ import annotations
@@ -38,7 +45,7 @@ class JudgeError(RuntimeError):
3845
class JudgeSpec:
3946
kind: str # "api" | "cli"
4047
model: str
41-
provider: str = "" # cli: cursor | codex | claude ; api: the LLM's provider
48+
provider: str = "" # cli: cursor | codex | claude | zcode ; api: the LLM's provider
4249
max_tokens: int = 2000
4350

4451

@@ -153,10 +160,38 @@ def governed_subprocess(argv: list[str], *, governor: Any = None, timeout: int =
153160
governor.settle_judge_call(token)
154161

155162

163+
def _run_zcode(spec: JudgeSpec, *, system: str, prompt: str, role: str,
164+
governor: Any) -> tuple[str, dict, str, str]:
165+
"""One governed zcode judge call through the production transport: the
166+
same tool-less one-shot the kb gate's ``ModelGateway`` uses (empty
167+
scratch cwd, every native tool removed, containment audit, served-model
168+
assertion), so an experiment's gate judge IS the production instrument,
169+
not an API proxy of it. Subscription billing: ``usd`` stays 0 and only
170+
the call count draws on the judge-call envelope."""
171+
from ..config import Settings
172+
from ..providers.registry import transport_for_id
173+
174+
token = governor.reserve_judge_call() if governor is not None else None
175+
try:
176+
transport = transport_for_id(Settings(), "zcode")
177+
reply = transport.complete(system=system,
178+
messages=[{"role": "user", "content": prompt}],
179+
model=spec.model, role=role)
180+
text = "".join(b.text for b in reply.blocks if b.type == "text")
181+
usage = dict(getattr(reply, "usage", None) or {})
182+
return text, usage, "" if text else "empty zcode judge reply", \
183+
str(getattr(reply, "model", "") or "")
184+
except Exception as exc: # noqa: BLE001 — the verdict records the reason
185+
return "", {}, f"zcode judge unavailable: {type(exc).__name__}: {exc}", ""
186+
finally:
187+
if governor is not None:
188+
governor.settle_judge_call(token)
189+
190+
156191
def run_judge(spec: JudgeSpec, *, system: str, prompt: str, llm: Any = None, governor: Any = None,
157192
role: str = "judge", runner: Callable[..., Any] | None = None) -> dict:
158193
"""Run one judge call and return its parsed JSON verdict (raises
159-
JudgeError on an unusable reply). Both paths are recorded as
194+
JudgeError on an unusable reply). All paths are recorded as
160195
``model_call`` records under the current trace context."""
161196
if spec.kind == "api":
162197
if llm is None:
@@ -167,25 +202,31 @@ def run_judge(spec: JudgeSpec, *, system: str, prompt: str, llm: Any = None, gov
167202
return parse_verdict(text)
168203
if spec.kind != "cli":
169204
raise JudgeError(f"unknown judge kind {spec.kind!r}")
170-
workspace = Path(tempfile.mkdtemp(prefix="judge-ws-"))
171-
workspace.chmod(0o700)
172205
started = time.monotonic()
173-
try:
174-
full = prompt if not system else f"{system}\n\n{prompt}"
175-
proc = governed_subprocess(_cli_argv(spec, full, workspace), governor=governor, cwd=str(workspace),
176-
runner=runner, input=full if spec.provider == "codex" else None)
177-
text, usage = _cli_result(spec, getattr(proc, "stdout", "") or "")
178-
error = "" if text else f"empty judge reply (rc={getattr(proc, 'returncode', '?')})"
179-
except JudgeError as exc:
180-
text, usage, error = "", {}, str(exc)
181-
finally:
182-
shutil.rmtree(workspace, ignore_errors=True)
206+
if spec.provider == "zcode":
207+
text, usage, error, served = _run_zcode(spec, system=system, prompt=prompt,
208+
role=role, governor=governor)
209+
else:
210+
workspace = Path(tempfile.mkdtemp(prefix="judge-ws-"))
211+
workspace.chmod(0o700)
212+
try:
213+
full = prompt if not system else f"{system}\n\n{prompt}"
214+
proc = governed_subprocess(_cli_argv(spec, full, workspace), governor=governor, cwd=str(workspace),
215+
runner=runner, input=full if spec.provider == "codex" else None)
216+
text, usage = _cli_result(spec, getattr(proc, "stdout", "") or "")
217+
error = "" if text else f"empty judge reply (rc={getattr(proc, 'returncode', '?')})"
218+
except JudgeError as exc:
219+
text, usage, error = "", {}, str(exc)
220+
finally:
221+
shutil.rmtree(workspace, ignore_errors=True)
222+
served = ""
183223
store = current_store()
184224
if store is not None:
185225
try:
186226
store.append("model_call", inputs={"system": system, "prompt": prompt},
187227
outputs={"reply": text} if text else {},
188-
model={"role": role, "provider": spec.provider, "model": spec.model, "served_model": ""},
228+
model={"role": role, "provider": spec.provider, "model": spec.model,
229+
"served_model": served},
189230
usage=usage, seconds=round(time.monotonic() - started, 3),
190231
result={"format": "prompt/1", "usd": 0.0, "subscription": True}, error=error)
191232
except Exception: # noqa: BLE001 - recording never changes a verdict
@@ -197,14 +238,16 @@ def run_judge(spec: JudgeSpec, *, system: str, prompt: str, llm: Any = None, gov
197238

198239
def judge_spec_from(settings: Any, override: str = "") -> JudgeSpec | None:
199240
"""The gold_match / paired judge from ``settings.improve_judge`` (or an
200-
explicit override): ``api:<model>`` or ``cli:<provider>:<model>``; None
201-
when unset."""
241+
explicit override): ``api:<model>`` or ``cli:<provider>:<model>`` where
242+
the provider is cursor | codex | claude | zcode; None when unset."""
202243
raw = str(override or getattr(settings, "improve_judge", "") or "")
203244
if not raw:
204245
return None
205246
parts = raw.split(":")
206247
if parts[0] == "api" and len(parts) == 2:
207248
return JudgeSpec("api", parts[1])
208249
if parts[0] == "cli" and len(parts) == 3:
250+
if parts[1] not in ("cursor", "codex", "claude", "zcode"):
251+
raise JudgeError(f"unknown CLI judge provider {parts[1]!r}")
209252
return JudgeSpec("cli", parts[2], provider=parts[1])
210253
raise JudgeError(f"improve_judge must be api:<model> or cli:<provider>:<model>, got {raw!r}")

‎test/test_improve_p2.py‎

Lines changed: 132 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -136,6 +136,138 @@ def create(self, **kw):
136136
run_judge(JudgeSpec("api", "m"), system="s", prompt="p")
137137

138138

139+
def test_zcode_judge_spec_parsing():
140+
from infermatrix_copilot.improve.judges import judge_spec_from
141+
142+
spec = judge_spec_from(SimpleNamespace(improve_judge="cli:zcode:GLM-5.3"))
143+
assert spec == JudgeSpec("cli", "GLM-5.3", provider="zcode")
144+
assert judge_spec_from(SimpleNamespace(improve_judge="")) is None
145+
with pytest.raises(JudgeError, match="unknown CLI judge provider"):
146+
judge_spec_from(SimpleNamespace(improve_judge="cli:cursor-agent:m"))
147+
with pytest.raises(JudgeError, match="improve_judge must be"):
148+
judge_spec_from(SimpleNamespace(improve_judge="zcode:GLM-5.3"))
149+
150+
151+
def test_zcode_judge_delegates_to_the_production_transport(tmp_path, monkeypatch):
152+
"""cli:zcode judging goes through ZCodeTransport.complete — the same
153+
tool-less one-shot the production kb gate uses — governed by the
154+
judge-call envelope and recorded like any CLI judge."""
155+
from infermatrix_copilot.improve.judges import _run_zcode
156+
from infermatrix_copilot.providers import registry
157+
158+
calls = []
159+
160+
class Gov:
161+
def reserve_judge_call(self):
162+
calls.append("reserve")
163+
return "2027-W03|judge|z"
164+
165+
def settle_judge_call(self, token=None):
166+
calls.append(("settle", token))
167+
168+
class Transport:
169+
def __init__(self, reply):
170+
self.reply = reply
171+
self.kwargs = None
172+
173+
def complete(self, **kw):
174+
self.kwargs = kw
175+
if isinstance(self.reply, Exception):
176+
raise self.reply
177+
return self.reply
178+
179+
ok = SimpleNamespace(blocks=[SimpleNamespace(type="text", text='{"status": "hit", "quote": "q"}')],
180+
usage={"input_tokens": 7, "output_tokens": 3}, model="GLM-5.3")
181+
transport = Transport(ok)
182+
monkeypatch.setattr(registry, "transport_for_id", lambda settings, pid: transport)
183+
184+
store = TraceStore(tmp_path / "t", environ={})
185+
with bind_store(store), trace_context(run_id="jz", unit_id="uz"):
186+
verdict = run_judge(JudgeSpec("cli", "GLM-5.3", provider="zcode"), system="s", prompt="p",
187+
governor=Gov())
188+
assert verdict["status"] == "hit"
189+
assert transport.kwargs == {"system": "s", "messages": [{"role": "user", "content": "p"}],
190+
"model": "GLM-5.3", "role": "judge"}
191+
assert calls == ["reserve", ("settle", "2027-W03|judge|z")]
192+
rec = store.query(kind="model_call", unit_id="uz")[0]
193+
assert rec["model"]["provider"] == "zcode" and rec["model"]["served_model"] == "GLM-5.3"
194+
assert rec["result"]["usd"] == 0.0 and rec["result"]["subscription"] is True
195+
assert rec["usage"]["input_tokens"] == 7 and not rec["error"]
196+
197+
down = Transport(RuntimeError("zcode CLI is not installed"))
198+
monkeypatch.setattr(registry, "transport_for_id", lambda settings, pid: down)
199+
with pytest.raises(JudgeError, match="zcode judge unavailable"):
200+
run_judge(JudgeSpec("cli", "GLM-5.3", provider="zcode"), system="s", prompt="p", governor=Gov())
201+
assert calls[-1] == ("settle", "2027-W03|judge|z") # a failed call still settles
202+
203+
empty = Transport(SimpleNamespace(blocks=[], usage={}, model="GLM-5.3"))
204+
monkeypatch.setattr(registry, "transport_for_id", lambda settings, pid: empty)
205+
with pytest.raises(JudgeError, match="empty zcode judge reply"):
206+
run_judge(JudgeSpec("cli", "GLM-5.3", provider="zcode"), system="s", prompt="p", governor=None)
207+
208+
209+
def test_zcode_run_helper_returns_the_parsed_triple(monkeypatch):
210+
"""The helper's contract: (text, usage, error, served) — the reply's
211+
text from text blocks only, non-text blocks ignored."""
212+
from infermatrix_copilot.improve.judges import _run_zcode
213+
from infermatrix_copilot.providers import registry
214+
215+
class Transport:
216+
def complete(self, **kw):
217+
return SimpleNamespace(blocks=[SimpleNamespace(type="reasoning", text="thinking"),
218+
SimpleNamespace(type="text", text='{"verdict": "consistent"}')],
219+
usage={"input_tokens": 1}, model="GLM-5.3")
220+
221+
monkeypatch.setattr(registry, "transport_for_id", lambda settings, pid: Transport())
222+
text, usage, error, served = _run_zcode(JudgeSpec("cli", "GLM-5.3", provider="zcode"),
223+
system="s", prompt="p", role="judge", governor=None)
224+
assert text == '{"verdict": "consistent"}' and not error and served == "GLM-5.3"
225+
assert usage == {"input_tokens": 1}
226+
227+
228+
def test_zcode_judge_serves_the_trusted_kb_gate(monkeypatch, tmp_path):
229+
"""End to end: the trusted driver's Gateway over a cli:zcode judge spec
230+
drives the production gate (judge_block + consistency) — the
231+
experiment-internal gate judge is the production instrument, with the
232+
ModelRole the gate records naming the zcode backend."""
233+
import json as _json
234+
235+
from infermatrix_copilot.improve.drivers import Gateway
236+
from infermatrix_copilot.improve.judges import judge_spec_from
237+
from infermatrix_copilot.kb_service import gate as kb_gate
238+
from infermatrix_copilot.kb_service.models import ModelRole
239+
from infermatrix_copilot.providers import registry
240+
from test_kb_intake_gate import PAGE, Op, _rule, _tree
241+
242+
class Transport:
243+
def complete(self, *, system, messages, model, role, **kw):
244+
prompt = messages[0]["content"]
245+
if '"directory"' in prompt: # the consistency call
246+
return SimpleNamespace(blocks=[SimpleNamespace(type="text", text='{"verdict": "consistent", "conflicts": []}')],
247+
usage={}, model=model)
248+
payload = _json.loads(prompt.split("<untrusted_data>\n", 1)[1].rsplit("\n</untrusted_data>", 1)[0])
249+
dims = {d: "yes" for d in payload["dimensions_to_answer"]}
250+
return SimpleNamespace(blocks=[SimpleNamespace(type="text", text=_json.dumps(
251+
{"dimensions": dims, "reasons": {}}))], usage={}, model=model)
252+
253+
monkeypatch.setattr(registry, "transport_for_id", lambda settings, pid: Transport())
254+
spec = judge_spec_from(SimpleNamespace(improve_judge="cli:zcode:GLM-5.3"))
255+
gateway = Gateway(None, spec, None)
256+
judge = ModelRole("judge", spec.provider, spec.model)
257+
258+
from infermatrix_copilot.kb_service.gate import changes_between
259+
from infermatrix_copilot.knowledge_service.ops import apply_operations
260+
261+
base = _tree()
262+
add = [Op("add", PAGE, "DEMO-2a", _rule("DEMO-2a", "PR #11"))]
263+
head = {**base, **apply_operations(base, add, release="v1", today="2026-09-28").files}
264+
decision = kb_gate.run_gate(base=base, head=head, changes=changes_between(base, head),
265+
external_texts={}, evidence=[{"source_reference": "PR #11"}],
266+
gateway=gateway, judge=judge, release="v1", repo_dir="repos/demo")
267+
assert decision.status == "pass"
268+
assert all(b.model == "GLM-5.3" for b in decision.blocks)
269+
270+
139271
def test_paired_statistics_labels_power_and_kappa():
140272
r = stats.paired({f"i{k}": [0.1, 0.12] for k in range(10)}, "recall")
141273
assert r.n_items == 10 and r.n_verdicts == 20 and abs(r.mean - 0.11) < 1e-9 and r.lo > 0

‎test/test_kb_intake_gate.py‎

Lines changed: 14 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -183,25 +183,27 @@ def test_gate_with_a_zcode_judge_uses_the_same_instrument():
183183
def test_judge_tuning_is_the_single_source_the_gate_uses():
184184
"""judge_tuning is the evolvable surface; gate.py must delegate to it and
185185
stay byte-identical to the pre-extraction baseline (evolution starts
186-
from the production behaviour, never from a silent rewording)."""
187-
import subprocess
186+
from the production behaviour, never from a silent rewording). The
187+
digests pin the shipped baseline — recorded when the module was
188+
extracted from gate.py (7aa922dd9^)."""
189+
import hashlib
188190

189191
from infermatrix_copilot.kb_service import gate, judge_tuning
190192

191193
assert gate.JUDGE_SYSTEM is judge_tuning.JUDGE_SYSTEM
192194
assert gate.CONSISTENCY_SYSTEM is judge_tuning.CONSISTENCY_SYSTEM
193195
assert gate.NEIGHBOUR_LIMIT is judge_tuning.NEIGHBOUR_LIMIT == 30
194196

195-
old = subprocess.run(["git", "show", "HEAD:src/infermatrix_copilot/kb_service/gate.py"],
196-
capture_output=True, text=True, check=True,
197-
cwd=Path(__file__).resolve().parents[1]).stdout
198-
import re
199-
200-
def extract(name):
201-
m = re.search(rf'^{name} = (""".*?""")$', old, re.M | re.S)
202-
return eval(m.group(1)) # noqa: S307 - test fixture over our own constant
203-
assert judge_tuning.JUDGE_SYSTEM == extract("JUDGE_SYSTEM")
204-
assert judge_tuning.CONSISTENCY_SYSTEM == extract("CONSISTENCY_SYSTEM")
197+
baseline = {
198+
"JUDGE_SYSTEM": "2d120ade58b9ac32c0af4b0ad8854205f80fbfcad4e89ff2204b47028ec0781d",
199+
"CONSISTENCY_SYSTEM": "db4fd2396f0400f9d45311f0c1460d181d83f8b64489c142c398bd1eca3f0b97",
200+
}
201+
for name, expected in baseline.items():
202+
actual = hashlib.sha256(getattr(judge_tuning, name).encode()).hexdigest()
203+
assert actual == expected, (
204+
f"{name} drifted from the production baseline the evolution "
205+
"engine starts from; rewording must arrive as an engine candidate, "
206+
"not as part of another change")
205207

206208

207209
def test_verdict_aggregation_keeps_the_fail_closed_ordering():

0 commit comments

Comments
 (0)