Repository navigation
Distillation P2: lineage exclusion in the held-out gate - #50
Merged
Merged
Conversation
make_eval_suite_runner drops tasks whose source_sessions intersect the edit's
lineage; if nothing independent remains it returns {} without running, and
run_holdout_gate reports HoldoutVerdict(conclusive=False). Empty exclusion set =
today's behavior (no exclusion).
…ed [distill P2/2] HarnessEdit gains source_sessions (populated from the playbook entry by the rule-based and LLM proposers, survives to_dict). build_held_out_validator now excludes that edit's sessions per-edit at suite assembly. _process_edit lands an inconclusive verdict (no lineage-independent task) as pending_unvalidated rather than pass/reject.
…d [distill P2/3] accept_edit persists source_sessions into the ledger; active_edits surfaces it; revalidation reconstructs the edit with lineage, excludes those sessions per-edit at held-out suite assembly, and treats an inconclusive verdict as skipped (leaving the stale edit untouched) instead of reactivate/retire.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
落地设计 §6.3 —— 修的是现有 holdout gate 的真实漏洞:今天一个 prompt edit 可以被"催生它的那个 session 蒸出来的任务"验证通过(拿作业题当期末考)。P0 已把
source_sessions放上EvalTaskSpec;本 PR 把排除端到端接通。纯确定性、零 API。三步
holdout_gate.py):make_eval_suite_runner(exclude_source_sessions=)在装配时丢掉 lineage 相交的任务;全排完 → runner 返回{}不跑 EvalRunner;run_holdout_gate此时返回HoldoutVerdict(conclusive=False)。HarnessEdit.source_sessions(rule/LLM proposer 从 playbook entry 填);build_held_out_validator按 edit 逐条排除;_process_edit把 inconclusive 落pending_unvalidated(不 pass 不 reject)。source_sessions经 accept ledger 存活到active_edits→ revalidate 重建 edit 时带上,逐条排除;inconclusive 落skipped(stale edit 原样不动)。守住
空 lineage = 不排除(向后兼容,手写任务/旧 edit 行为不变);现有 held-out 测试(fake validator,默认
conclusive=True)全绿;门仍在 loop 之外、不改任何 edit。验证
_process_editinconclusive → pending_unvalidated / rule-based edit 带 lineage / lineage 经 accept 存活 / revalidate inconclusive → skipped 且 edit 不动 / parse_edits 盖 lineage。界限
只剩 P3(成功轨迹 + metrics 半);P1(LLM distiller,花预算)按需再做。