Repository navigation
fix: verify numerical claims against computed evidence - #25
Merged
Merged
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
The external evaluation “duplicate” case currently generates claims from non-duplicated data, and _mean_claim can emit malformed empty- समूह text, which undermines the intended verification/evaluation semantics.
Get a fresh assessment by requesting another Copilot review.
Pull request overview
This PR tightens claim handling by only allowing descriptive claims that exactly match a canonical, recomputed mean-statement generated from the supplied data, preventing arbitrary numeric prose (or causal estimates) from being treated as “supported.”
Changes:
- Introduces a closed-form descriptive mean-claim verifier and requires exact text match for a descriptive claim to be allowed.
- Adds new integrity gating (invalid outcomes/null groups, multiple sources) plus adversarial regressions around invented numbers, swapped groups, counts, rounding, and extra prose.
- Updates CLI/Markdown/docs and expands the external evaluation suite (with an optional free-text extension that intentionally surfaces false blocks).
File summaries
| File | Description |
|---|---|
| tests/test_verdict_and_claims.py | Updates VerdictEngine unit test to pass verified descriptive claims. |
| tests/test_validity_adversarial.py | Aligns adversarial validity expectations with the new “no uncomputed claim support” rule. |
| tests/test_numeric_claims.py | Adds new regressions ensuring only exact recomputed mean statements are supported. |
| tests/test_external_evaluation.py | Updates expected counts for the expanded external evaluation and adds free-text visibility test. |
| tests/e2e/test_runner_edge_cases.py | Updates E2E expectation: assumptions don’t verify an uncomputed causal claim. |
| src/answerable/reports/markdown.py | Clarifies report semantics and updates “allowed claims” empty-state messaging. |
| src/answerable/evidence/verdict.py | Adds verified_descriptive_claims parameter and enforces exact-match support rules. |
| src/answerable/cli.py | Prints an explicit claim-verification boundary notice in CLI output. |
| src/answerable/application/assessment_runner.py | Computes canonical mean-claim, adds invalid-outcome blocker, fingerprints verifier revision, and records claim validation in observations. |
| scripts/verify_bike_claims.py | Adds a local reproduction/regression script for the Bike Sharing numeric-claim bug. |
| scripts/evaluate_external.py | Expands materialized external cases and adds optional free-text cases for measuring false blocks. |
| requirements/traceability.yaml | Links INV-001 to the new numeric-claim test module. |
| README.md | Documents the claim verification boundary and updates external evaluation counts/claims. |
| examples/campaign_retention/question.yaml | Updates example descriptive claim to the canonical mean-claim format. |
| docs/VALIDATION.md | Adds a numeric-claim correction note and distinguishes restricted vs free-text protocols. |
| docs/CLAIM_VERIFICATION.md | Documents the supported canonical mean-claim format and Windows retest steps. |
| docs/adr/0003-closed-numeric-claim-verification.md | Adds ADR describing the closed numeric-claim verification decision and compatibility notes. |
| benchmarks/external/README.md | Updates external benchmark documentation for the expanded restricted-format suite and optional free-text extension. |
Review details
- Files reviewed: 18/18 changed files
- Comments generated: 2
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+80
to
+84
| n0 = sum(r["group"] == 0 for r in selected) | ||
| n1 = sum(r["group"] == 1 for r in selected) | ||
| claim = ( | ||
| f"Among these recorded {source['unit']}s, the observed mean was {a:.6g} " | ||
| f"in January-June and {b:.6g} in July-December ({source['units']})." | ||
| 'Observed means of "value" by "group" among supplied records ' | ||
| f'(six decimals): "0"={a:.6f} (n={n0}); "1"={b:.6f} (n={n1}).' |
Comment on lines
+547
to
+552
| def _mean_claim(descriptive: _Descriptive, mapping: DataMapping) -> str: | ||
| """Closed, six-decimal grammar; never infer the meaning of arbitrary prose.""" | ||
| groups = "; ".join( | ||
| f"{json.dumps(group)}={rate:.6f} (n={count})" for group, count, rate in descriptive.by_group | ||
| ) | ||
| return ( |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A real UCI Bike Sharing test returned false means (99999 and 1) in
allowed_claims. Claim-class linting did not verify sentence contents, and declaring causal assumptions could admit uncomputed effect claims.Changes
Validation
Limits
This is a conservative safety fix, not arbitrary natural-language verification or a stable product claim. Historical benchmark artifacts are unchanged; zero false blocks in the restricted grammar must not be generalized to free prose.