Repository navigation
docs: trajectory-distillation design (weakness → executable held-out tasks) - #48
Merged
Merged
Conversation
… tasks) New design doc: distill recorded trajectories + mined weaknesses into EvalTaskSpec candidates (fixture + prompt + argv verifier), deterministically verified (fail-before) and human-accepted into the held-out suite -- the judge's task set grows with observed weaknesses instead of staying hand-written. Cross-link weakness-mining-design.md: the evaluator surface row (was 'semi-closed') now points here; the effectiveness-signal residual is marked design-ready.
…op line number The lineage-exclusion filter reads source_sessions at holdout-suite assembly, but provenance.json stays in the gitignored candidate dir and does not survive promote. Carry source_sessions on task.json itself (optional EvalTaskSpec field, empty for hand-written tasks so they are never excluded). Also: correct the held-out gate entry points (okc propose --llm --validate-held-out / okc edits --revalidate), and drop a rot-prone mining.py line number.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
新设计文档 docs/trajectory-distillation.md:把已录制轨迹 + 挖出的弱点蒸馏成可执行的
EvalTaskSpec候选(fixture + prompt + argv verifier),经确定性 fail-before 校验与人工 accept 后进入 held-out suite——裁判的任务集随观察到的弱点生长,不再停在手写任务上。补上的正是 §5.6 那半环
weakness-mining-design.md §5.6 的
evaluatorsurface 原标"半闭环"。本设计把它变成与其他 surface 同构的 proposal-first 闭环:loop 产出裁判候选,人是唯一的 promote 通道(git commit),冻结面纪律原样保留。已交叉更新 §5.6 表格与 §9 的"仍未做"行(有效性判据:设计已就绪、实现未动)。设计要点
EvalTaskSpecJSON + fixture,load_eval_task/两个 gate 原样消费。.agent-harness/,suite 住 git 管控的tests/e2e/,okc distill无任何写 suite 的代码路径。纯文档;分阶段落地 P0–P3 已列,未动代码。