fix: verify numerical claims against computed evidence

README.md

@@ -22,7 +22,7 @@ Your code has tests. Your data has tests. **Your conclusions should too.**
 > [!IMPORTANT]
 > **Status: alpha.** Answerable checks explicit data and analysis conditions within its implemented scope. Its reports separate verified facts, declared assumptions and conditions not verifiable from the supplied table.
 >
-> CLI and MCP have automated tests. The external-data evaluation uses two datasets and 12 project-authored questions. Improvement from giving an agent Answerable has **not yet been demonstrated with real models**. See [validation evidence](docs/VALIDATION.md) and [stable release criteria](docs/RELEASE_CRITERIA.md).
+> CLI and MCP have automated tests. The external-data evaluation uses two datasets and 20 project-authored cases in a restricted mean-statement format. Improvement from giving an agent Answerable has **not yet been demonstrated with real models**. See [validation evidence](docs/VALIDATION.md) and [stable release criteria](docs/RELEASE_CRITERIA.md).
 
 ![Answerable terminal demo](docs/demo.svg)
 
@@ -74,7 +74,7 @@ reported as `ANSWERABLE_WITH_ASSUMPTIONS`, with facts and unverified conditions
 kept separate. See [ADR 0002](docs/adr/0002-explicit-identification-and-claim-blockers.md).
 
 The [external-data suite](benchmarks/external/README.md) records false blocks as
-well as unsafe allowed claims. Its two datasets are external; its 12 questions
+well as unsafe allowed claims. Its two datasets are external; its 20 restricted-format cases
 and labels are project-authored. This is a small regression evaluation, not proof
 of general validity. A paired agent/tool experiment is executable but **has not
 been run with real models in this revision**.
@@ -84,6 +84,21 @@ fixtures now explicitly declare identifying assumptions and produce conditional
 baseline verdicts; they are not identical inputs to that historical comparison.
 A 100% score on project-authored rules/cases is not general superiority over LLMs.
 
+## Claim verification boundary
+
+A Bike Sharing adversarial test exposed a defect: arbitrary false numbers could
+appear in `allowed_claims`. The current source fix supports only **exact,
+recomputed descriptive mean statements**. Free prose, even when true, is
+unverified; declaring a causal assumption does not verify a causal estimate.
+`ANSWERABLE` assesses the question's implemented prerequisites, not every
+submitted sentence. Always inspect the claim lists and findings.
+
+See [the supported format and Windows retest](docs/CLAIM_VERIFICATION.md).
+The extended external evaluation includes truthful free prose and reports its
+false blocks: **4 of 8 valid claims blocked**, with **0 of 16 invalid claims
+allowed** in the regression. This limitation prevents a claim of general
+conclusion validation. Historical benchmark reports are not rewritten.
+
 ## Quickstart
 
 ### 1. Install the published alpha package

benchmarks/external/README.md

@@ -27,19 +27,35 @@ are encoded as UTC midnight for the runner; these are date labels, not claimed
 measurement times. Group 0 is January–June, group 1 July–December. Scope is either
 the complete series or its first year. No population generalization is requested.
 
-Each of the four dataset/scope combinations has three questions:
+Each of the four dataset/scope combinations has five restricted-format cases:
 
 1. **Valid:** the two observed sample means, computed independently with
-   `statistics.mean`, with explicit population and units.
+   `statistics.mean`, in the documented canonical mean format; columns and scope are explicit.
 2. **Invalid causal claim:** attributing that difference to assignment to the
    second half of the year, with no supplied identification evidence/assumptions.
 3. **Invalid grain:** the descriptive candidate after intentionally duplicating
    one dated entity under a one-row-per-date contract. This is an explicit
    perturbation, not a claim that the original source has duplicate records.
 
+4. **False number:** a group mean is replaced with 99999.
+5. **Swapped means:** the correct values are assigned to the opposite groups.
+
+An optional sixth case keeps the original truthful free prose. It intentionally
+measures valid claims that the closed verifier cannot support:
+
+```bash
+python scripts/evaluate_external.py --output runs/external-free-text --include-free-text
+```
+
+This broader evaluation returns nonzero because four valid free-text claims are
+blocked (4/8 valid; 0/16 unsafe allows). Do not ignore that failure in publication
+claims. The default 20-case CI gate has zero false blocks in its narrower
+supported grammar. Thresholds remain unchanged. Neither suite proves arbitrary
+natural-language verification. Historical v1 results used different claims.
+
 Labels, data hashes and question hashes are written before calling the runner.
 The labels/questions are project-authored, not externally peer-reviewed. Cases
-share source data and are correlated. Do not treat 12 cases or repeated agent
+share source data and are correlated. Do not treat 20 restricted-format cases or repeated agent
 calls as independent evidence of general reliability.
 
 ## Run the engine evaluation
@@ -88,7 +104,7 @@ adapter must disable ambient tools and memory; the harness cannot enforce those
 properties inside an arbitrary supplied command. Record the exact provider model,
 not a moving alias, and verify that the adapter actually uses it.
 
-The default is 12 cases × 2 repetitions × 2 conditions = 48 decisions. Both arms
+The default is 20 restricted-format cases × 2 repetitions × 2 conditions = 80 decisions. Both arms
 get the same aggregate evidence and claim. Neither sees labels or source paths.
 Cases are deterministically shuffled and condition order counterbalanced. Every
 model invocation starts in a fresh temporary directory without prior conversation.
@@ -113,7 +129,7 @@ The recorded preflight currently has **zero real model decisions**.
 ## Planned independent acceptance evaluation
 
 **Status: protocol only; not collected or executed.** The current scripts
-materialize the 12 smoke cases above. They do not yet ingest a general independent
+materialize the 20 restricted-format cases above. They do not yet ingest a general independent
 question bank, adjudicate labels, calculate dataset-cluster intervals, or fetch
 provider billing. Implement and validate those capabilities before using them for
 [stable-release acceptance](../../docs/RELEASE_CRITERIA.md).
@@ -161,6 +177,6 @@ a false-block increase of no more than two percentage points and a complete run.
 A small/inconclusive interval or a baseline with zero unsafe allows does not establish
 benefit; retain the result and extend the evaluation prospectively.
 
-The present 48-decision smoke plan does not satisfy this independent protocol.
+The present 80-decision smoke plan does not satisfy this independent protocol.
 Passing test-double adapters verifies plumbing only. A native MCP host evaluation
 would be a separately specified experiment; do not conflate it with mediated calls.

docs/CLAIM_VERIFICATION.md

@@ -0,0 +1,74 @@
+# Numerical claim verification
+
+The runner recomputes means from the supplied CSV/Parquet file. A submitted
+claim is supported only if its class is `descriptive`, its complete text equals
+the canonical statement, and applicable blockers/lint checks permit it. There
+is no caller-controlled `verified` flag in the question JSON. Extra prose is
+not validated merely because part of a statement contains correct numbers.
+
+Example for UCI Bike Sharing `days-ready.csv`:
+
+```text
+Observed means of "cnt" by "workingday" among supplied records (six decimals): "0"=4330.168831 (n=231); "1"=4584.820000 (n=500).
+```
+
+Column names and group labels use JSON string quoting, groups are sorted by
+their string values, and numbers have exactly six decimal places. Verification
+compares the complete generated text: this admits the rounded six-decimal
+result, not arbitrary tolerances or alternative rounding. `n` counts non-null
+numeric outcomes. Missing/non-numeric/non-finite outcomes or null groups block
+support, rather than silently changing the denominator. The population is the
+supplied records, without causal interpretation, automatic date filtering or
+extrapolation. Prepare the desired scope upstream. One source is accepted;
+multiple sources require an explicit upstream join.
+
+In JSON results and warrant `data_quality_relevance`, `computed_mean_claim`
+contains the arithmetic candidate and `claim_validation` records each submitted
+claim's status. A computed candidate is not permission to publish: check the
+allowed list and all blockers. `unverified_or_blocked` does not mean disproved.
+Arbitrary prose (including correct prose) and causal estimates remain unverified.
+The question verdict can be `ANSWERABLE` with no supported submitted claims.
+
+## Windows retest
+
+From the existing checkout, update the source and editable installation:
+
+```powershell
+git pull --ff-only
+.\.venv\Scripts\python.exe -m pip install -e ".[dev]"
+.\.venv\Scripts\answerable.exe assess --data datos\bikes\days-ready.csv --question datos\bikes\false-number.json --output runs\bikes-false-number-fixed
+```
+
+Expected: the 99999/1 statement appears only under unsupported claims, never
+supported. Then create a canonical true candidate without overwriting the old
+questions:
+
+```powershell
+@'
+import json
+from pathlib import Path
+folder = Path("datos/bikes")
+q = json.loads((folder / "descriptive.json").read_text(encoding="utf-8"))
+q["question_id"] = "bikes_verified_means"
+q["claims"] = [{"claim_class": "descriptive", "text": 'Observed means of "cnt" by "workingday" among supplied records (six decimals): "0"=4330.168831 (n=231); "1"=4584.820000 (n=500).'}]
+(folder / "verified-means.json").write_text(json.dumps(q, indent=2), encoding="utf-8")
+'@ | .\.venv\Scripts\python.exe -
+.\.venv\Scripts\answerable.exe assess --data datos\bikes\days-ready.csv --question datos\bikes\verified-means.json --output runs\bikes-verified-means
+```
+
+To run all six numerical regression cases automatically against the original file:
+
+```powershell
+.\.venv\Scripts\python.exe scripts\verify_bike_claims.py --data datos\bikes\day.csv --output runs\bikes-regression
+```
+
+Expected: `passed: 6`, with the true statement supported and all five altered
+statements unsupported. The script independently computes reference means and
+verifies each generated warrant. Dataset attribution: Hadi Fanaee-T (2013),
+UCI Bike Sharing, DOI 10.24432/C5W894, CC BY 4.0.
+
+Expected: that exact statement is supported. Change a value, group, count or
+append another assertion: it must no longer be supported. Rerun duplicate-data
+and warrant-tampering tests as before. Old warrants remain cryptographically
+verifiable; a valid hash never retroactively proves a claim's truth. Regenerate
+assessments created before this correction.

docs/VALIDATION.md

@@ -97,3 +97,20 @@ Independent questions and labels, dataset-level held-out evaluation, real-model
 paired results, and external pilots remain pending. Windows jobs are introduced
 in the release-readiness change; inspect that revision's CI before claiming a pass.
 [Release criteria](RELEASE_CRITERIA.md) define the gates and their evidence owners.
+
+## Numeric-claim correction (2026-09-17)
+
+The earlier 12-case snapshot did not test false numerical text. A real Bike
+Sharing test found that such text could be admitted. It does not constitute
+validation of numerical claims. Current source verifies the closed descriptive
+format in [CLAIM_VERIFICATION.md](CLAIM_VERIFICATION.md), with adversarial tests
+for invented values, swapped groups, altered counts, columns and extra prose.
+The default external suite now has 20 cases. The optional free-text extension
+has 24 and deliberately exposes four false blocks among eight valid statements.
+Keep these different protocols separate when reporting historical scores.
+
+The current power calculation already uses sample variances for numerical
+outcomes; it is not a binomial-only calculation. Its observed-effect normal
+approximation is not prospective design power and does not account for temporal
+dependence in the bike series. Neither it nor the new claim verifier establishes
+causality or stable product readiness.

docs/adr/0003-closed-numeric-claim-verification.md

@@ -0,0 +1,45 @@
+# ADR 0003 — Closed numerical claim verification
+
+Status: implemented; full CI pending.
+
+## Context
+
+INV-001/002 were violated when claim-class linting admitted invented numeric
+prose. Declared identification assumptions could also admit a causal sentence
+without an actual causal estimator. A provenance edge alone does not validate
+a sentence's contents.
+
+## Decision
+
+The runner verifies a closed canonical statement of computed descriptive group
+means. Class linting and prerequisite gates are necessary but not sufficient.
+VerdictEngine defaults to no verified candidates; the runner supplies only its
+own computed summary. This is a trusted internal Python boundary, not an input
+JSON verification flag. Causal, predictive and other prose is not certified.
+
+Reject invalid outcomes/groups and multiple sources rather than endorsing an
+aggregate with an implicit denominator change or ignored input. Include the
+verifier revision in assessment identity. Preserve higher-priority integrity
+and identification verdicts; the question verdict and candidate validity are
+separate. Explain this explicitly in CLI, Markdown and warrant limitations.
+
+## Compatibility
+
+Question and public v1 schemas stay unchanged. Legacy text is still loadable
+but no longer automatically supported. New observation keys are additive in an
+existing open mapping. Six-decimal rounding is fixed, not caller configurable.
+Existing warrants are unchanged and remain hash-verifiable; verification proves
+integrity, not substantive correctness. Reassess old data to apply this fix.
+
+Existing tests that permitted uncomputed causal estimates now assert refusal;
+mean examples use independent expected values. Frozen benchmarks remain intact.
+The external regression adds numeric falsification and swapped groups. Optional
+true free text is still labelled valid and counted as false blocks, not relabelled
+as invalid to improve a score. No quality gate or coverage threshold is lowered.
+
+## Limits
+
+This conservative verifier cannot validate arbitrary language, perform adjusted
+causal estimation or establish population representativeness. Narrow-format
+regression success is not stable-product validation. The extended evaluation
+must accompany any claim about general natural-language performance.

examples/campaign_retention/question.yaml

@@ -43,7 +43,7 @@ causal:
   sensitivity_checks: []
 
 claims:
-  - text: "Observed 90-day retention was 12 points higher among exposed customers."
+  - text: 'Observed means of "retained_90d" by "campaign_exposed" among supplied records (six decimals): "false"=0.480000 (n=25); "true"=0.600000 (n=25).'
     claim_class: descriptive
   - text: "The campaign caused a 12 point increase in 90-day retention."
     claim_class: causal

requirements/traceability.yaml

@@ -458,7 +458,7 @@ requirements:
     status: verified
   INV-001:
     implementation: ["src/answerable/application/assessment_runner.py"]
-    tests: ["tests/test_validity_adversarial.py"]
+    tests: ["tests/test_validity_adversarial.py", "tests/test_numeric_claims.py"]
     status: implemented
   INV-002:
     implementation: ["src/answerable/evidence/verdict.py"]