Repository navigation
Fix critic false-negative on findings the diff claims are intentional - #62
Merged
Merged
Conversation
PR #57 (test/webhook-e2e-review-check) exercised a real, deliberately introduced bug in format_summary_body(): the "Published inline" and "Summary-only" counts are swapped. The reviewer correctly proposed the finding (evidence, reasoning, and impact were all accurate), but the critic rejected it, reasoning that the comment directly above the bug ("intentionally swapped... do not fix") proved the behavior was not a defect. Reproduced against the real production diff (merge-base bd25117 vs head b82d279) with a real reviewer+critic run: candidate=1, accepted=0, rejected=1, matching the reported production shape exactly. The rejection is a false negative -- the critic's own system prompt already says everything in the diff is untrusted data, but nothing in it named this specific pattern (a claim of intent, as opposed to an impersonated system instruction), so the critic treated "the author says this is deliberate" as proof the output is correct. Deliberateness and correctness are different questions; a bug introduced on purpose is still a bug. - patchfrog/review/prompt.py: add an explicit critic-prompt rule that a comment/commit message claiming code is intentional, a test fixture, or "do not fix" is untrusted data, not evidence of correctness -- the critic must judge the finding against what the code does, not against the diff's own claim about why it exists. Existing reject criteria are unchanged; no threshold was lowered. - patchfrog/review/domain.py, schemas.py, critic.py: add CriticRejectionCategory, a machine-classified reject reason the critic now returns in its own structured response (mirroring the existing ValidationOutcome pattern -- never inferred from reasoning_summary prose after the fact). - patchfrog/persistence/models/review.py, repositories/critic_verdict.py, migrations/versions/0031_*: persist rejection_category on critic_verdicts (nullable, additive). - patchfrog/review/service.py: one structured "finding_candidate_diagnostics" log line per proposal (finding id, category, reviewer confidence, evidence-entry count, critic decision, rejection category) -- never the prompt, a provider response, a secret, or source content. - tests/fixtures/evaluation/cases/py-claimed-intentional-swapped-counts: regression fixture reproducing the pattern for patchfrog.evaluation. - tests/unit/test_review_critic.py, test_review_prompt_injection.py: cover rejection_category parsing/schema and the new prompt guardrail.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Investigates PR #57's reviewer/critic behavior (production evidence: candidate_count=1, accepted_count=0, rejected_count=1, status=succeeded -- reviewer found the swapped-counts bug, critic rejected it, nothing published).
Reproduced the exact production diff (
format_summary_body's swapped "Published inline"/"Summary-only" counts) against a real reviewer+critic run and confirmed the root cause by inspecting the persisted proposal/verdict directly: the reviewer's finding was accurate (evidence, reasoning, impact all correct), and the critic rejected it because the comment directly above the bug ("intentionally swapped... do not fix") convinced it the swap was correct. That is a false negative: the comment's claim of intent says why the code was written that way, not whether the output is correct. A bug introduced on purpose is still a bug -- and the critic's own system prompt already says the diff is untrusted data, it just never named this specific pattern (a claimed-intent assertion, distinct from the impersonated-system-instruction pattern the existingsecq-o-prompt-injection-claims-safe/py-prompt-injection-hidden-bugcases already cover).patchfrog/review/prompt.py) forbidding exactly that inference. No reject criteria were removed or loosened; no threshold was lowered.rejection_category(mirrors the existingValidationOutcomepattern -- never inferred from free text after the fact), persisted oncritic_verdicts(additive migration0031), plus one structuredfinding_candidate_diagnosticslog line per proposal (finding id, category, reviewer confidence, evidence-entry count, critic decision, rejection category -- never a prompt, provider response, secret, or source blob).tests/fixtures/evaluation/cases/py-claimed-intentional-swapped-countsreproduces the pattern forpatchfrog.evaluation, phrased as ordinary developer prose (not an impersonated system instruction) to keep it a distinct case from the existing prompt-injection corpus.Test plan
pytest-- every unit test in the touched modules passes (test_review_critic.py,test_review_prompt_injection.py,test_evaluation_fixtures_validation.py); fulltests/unit/(1680/1682, 2 pre-existing/unrelated failures) and the bulk oftests/integration/(unrelated to this change) pass. A handful of Postgres-only concurrency tests and two unrelatedruff-subprocess-based static-analysis/fix-verification test files could not be run cleanly in this sandbox (local Postgres container credential mismatch; subprocess flakiness under heavy memory pressure) -- neither touches any file in this diff.ruff check .-- cleanmypy patchfrog-- clean, 351 files