Repository navigation
P3-metrics checklist + graded capability-probe eval suite - #51
Merged
Merged
Conversation
Three new stdlib-only tasks (arithmetic fix / recursive-merge / mutable-default bug) spanning easy->hard, plus a capability_probe suite grouping them with two existing tasks. Real-provider tasks for gauging OKC's coding ability and producing trajectories to mine/distill. tests/test_capability_probe.py is a deterministic (zero-API) quality gate: the suite loads and every task's verifier fails on the untouched fixture, so none is trivially passable.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
两件事,都为推进 P1/P3 铺路。
1. P3-metrics 可执行清单(纯文档)
docs/p3-metrics-gate-checklist.md—— 让两层门除 pass/fail 外也看效率回退(model_calls/tool_batches变多),补当前盲区(over_exploration 等只变慢不变错的机制)。关键边界讲清:held-in 抓 harness 机制侧效率(确定性零 API)、held-out 抓模型侧效率(花预算),两处各做一半。4 步 + 复用清单 + 一个设计决定(效率回退默认上报、阻塞 opt-in)。未动代码。2. 分级 capability-probe 题目集(真实任务 + 测试)
tier1_temperature_convert(改错公式)/tier2_config_merge(递归合并 + 不可变输入)/tier3_mutable_default(可变默认参数共享态的 subtle bug)。suites/capability_probe.json把三者 + 两个现有任务按难度排成一条探针,okc eval --suite ... --provider <p>一把量当前能力。tests/test_capability_probe.py:确定性零 API 质检门——suite 可加载,且每个任务 verifier 在裸 fixture 上必须失败(没有一个是白送的)。已验证三个新 fixture 全部 fail-before。验证
全量 564 passed(558 → 564)。真实 provider 跑分是 opt-in(不进 CI)。