Repository navigation
Distillation P0: candidate contract + verify/accept ledger (okc distill) - #49
Merged
Merged
Conversation
…istill) Eval-task candidates live in .agent-harness/eval_candidates/<id>/ (task.json + fixture/ + provenance.json) with an append-only lifecycle ledger. okc distill --verify runs the deterministic anti-task checks: load via load_eval_task, verifier runnable, and the fail-before hard gate (the verifier must FAIL on the untouched fixture). --accept refuses unverified candidates and prints promote guidance only -- no code path writes tests/e2e/; a human commit is the sole promotion channel. EvalTaskSpec gains the optional source_sessions lineage field (empty for hand-written tasks) so held-out hygiene survives promotion. Candidates are hand-written in P0; the LLM distiller is P1.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
落地 docs/trajectory-distillation.md §9 的 P0——契约与手动通路。candidate 此阶段由人手写,先把生命周期和校验跑通;LLM distiller 是 P1。零 API、全确定性。
做了什么
.agent-harness/eval_candidates/<id>/(task.json+fixture/+provenance.json);EvalTaskSpec增加可选source_sessionslineage 字段(手写任务为空 → 永不被 lineage 排除),held-out 卫生从此能在 promote 后存活(§3,PR docs: trajectory-distillation design (weakness → executable held-out tasks) #48 的修正落进代码)。okc distill --verify <id>:确定性反任务校验(§5)——load_eval_task可加载、verifier 可执行、fail-before 硬门(verifier 在裸 fixture 上必须失败,空过的裁判什么都没测)。--accept / --reject / --list:append-only ledger(复用 harness_edits_apply 的 fold 手法),未过 verify 不可 accept(硬约束);accept 只打印 promote 指引——没有任何代码路径写tests/e2e/,人工 git commit 是唯一晋升通道(§6.1)。okc distill,复用eval.py的 verifier 执行路径(_run_verifier/_copy_fixture),不长第二套方言。验证
tests/test_distill.py(12):契约加载(含手写任务空 lineage)/ fail-before 拒空过 verifier / runnable 拒坏 argv / accept-before-verify 拒绝 / reject 任意时刻 + latest-decision-wins / ledger append-only / 重验失败降级 / CLI 全生命周期。界限(P0 刻意不做)
LLM distiller(P1)、lineage 过滤接进 holdout gate(P2)、metrics 半(P3)。