fix(sleep): separate optimizer feedback from judge evidence - #270
Draft
diqierjia wants to merge 1 commit into
Draft
fix(sleep): separate optimizer feedback from judge evidence#270diqierjia wants to merge 1 commit into
diqierjia wants to merge 1 commit into
Conversation
diqierjia
force-pushed
the
codex/skillopt-feedback-separation
branch
from
September 4, 2026 08:36
d5f9869 to
b00456b
Compare
diqierjia
force-pushed
the
codex/skillopt-feedback-separation
branch
from
September 4, 2026 08:41
b00456b to
53e3fac
Compare
Author
|
@microsoft-github-policy-service agree |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Related to #154.
This draft addresses one concrete, reproducible failure mode: raw judge evidence can leak regexes and checker implementation details into SkillOpt-Sleep optimizer prompts. The optimizer may then learn the verifier syntax instead of the user-visible behavior.
Scope
ReplayResult.optimizer_feedbackchannel for learning-safe semantic feedback.fail_reasonandjudge_rationaleunchanged as raw audit/debug evidence.description; if it is absent, regex checks use a conservative non-pattern fallback.Data flow
Raw judge rationale remains available in replay results, evidence logs, and reports for auditability. Optimizer prompts receive only
optimizer_feedback(or a recomputed safe projection for legacy results). Rule scoring keeps the existing backward-compatiblescore_rule_judge()return shape; the new helper carries the additional semantic feedback.Backward compatibility
optimizer_feedbackis optional and defaults to an empty string. Older or manually deserializedReplayResultobjects without that attribute remain usable: rule tasks recompute feedback from the task judge, while non-rule legacy evidence falls back to a generic message rather than exposing raw rationale.Validation
WinError 1314(the process lacks the privilege required to create symlinks). No tests were changed to hide this environment limitation.python -m compileall -q skillopt skillopt_sleep plugins scripts tests: passed.git diff --check: passed.Non-goals
This is intentionally not a solution for the full intent-level mining or proxy-alignment problem described in #154. It is a narrow containment fix for judge-evidence leakage while preserving existing scoring and audit behavior.