fix: enforce claim blockers and separate identifying assumptions

README.md

@@ -62,6 +62,26 @@ Verdict + blockers + allowed claims + forbidden claims
       Evidence Warrant
 ```
 
+## Validity review and external evaluation
+
+A targeted review corrected four issues: overlap no longer stands in for
+exchangeability, all target strata must have empirical support, blockers now
+apply to claim admissibility, and findings link to the checks that produced them.
+Identifying assumptions must be explicitly declared; conditional support is
+reported as `ANSWERABLE_WITH_ASSUMPTIONS`, with facts and unverified conditions
+kept separate. See [ADR 0002](docs/adr/0002-explicit-identification-and-claim-blockers.md).
+
+The [external-data suite](benchmarks/external/README.md) records false blocks as
+well as unsafe allowed claims. Its two datasets are external; its 12 questions
+and labels are project-authored. This is a small regression evaluation, not proof
+of general validity. A paired agent/tool experiment is executable but **has not
+been run with real models in this revision**.
+
+The historical EMT-v2 scores below remain historical frozen results. The live
+fixtures now explicitly declare identifying assumptions and produce conditional
+baseline verdicts; they are not identical inputs to that historical comparison.
+A 100% score on project-authored rules/cases is not general superiority over LLMs.
+
 ## Quickstart
 
 ### 1. Install the production package

benchmarks/external/README.md

@@ -0,0 +1,106 @@
+# External-data validation and paired agent experiment
+
+This suite is deliberately separate from EMT. It tests claim admissibility,
+including retention of valid descriptive claims, on data not generated by
+Answerable. It is a **smoke evaluation, not independent scientific validation**.
+
+## Data and labels
+
+`manifest.json` records source URLs, retrieval date, SHA-256 and normalization.
+Vendored CSV snapshots reproduce the run without a network request.
+
+- `flights`: 144 monthly observations from the seaborn example-data repository,
+  the international airline-passenger series also distributed as R AirPassengers.
+  Source: https://github.com/mwaskom/seaborn-data/blob/master/flights.csv
+- `temperatures`: 3,650 daily minimum-temperature observations from the public
+  time-series dataset collection maintained by Jason Brownlee.
+  Source: https://github.com/jbrownlee/Datasets/blob/master/daily-min-temperatures.csv
+
+Both CSVs contain aggregate numerical measurements rather than personal records.
+The source files are normalized to UTF-8/LF. Values are unchanged. Calendar dates
+are encoded as UTC midnight for the runner; these are date labels, not claimed
+measurement times. Group 0 is January–June, group 1 July–December. Scope is either
+the complete series or its first year. No population generalization is requested.
+
+Each of the four dataset/scope combinations has three questions:
+
+1. **Valid:** the two observed sample means, computed independently with
+   `statistics.mean`, with explicit population and units.
+2. **Invalid causal claim:** attributing that difference to assignment to the
+   second half of the year, with no supplied identification evidence/assumptions.
+3. **Invalid grain:** the descriptive candidate after intentionally duplicating
+   one dated entity under a one-row-per-date contract. This is an explicit
+   perturbation, not a claim that the original source has duplicate records.
+
+Labels, data hashes and question hashes are written before calling the runner.
+The labels/questions are project-authored, not externally peer-reviewed. Cases
+share source data and are correlated. Do not treat 12 cases or repeated agent
+calls as independent evidence of general reliability.
+
+## Run the engine evaluation
+
+```bash
+python scripts/evaluate_external.py --output runs/external-new
+```
+
+Metrics distinguish:
+
+- `false_blocks`: valid claims omitted from `allowed_claims`;
+- `unsafe_allows`: invalid claims included in `allowed_claims`;
+- `release_false_blocks`: valid claims denied by the combination of claim list
+  and primary verdict (the release gate may disagree with `allowed_claims`);
+- `errors`: failures recorded separately, never silently removed or counted as
+  safe decisions.
+
+The report includes every case, source/case hashes, independent and observed
+means, verdict, blockers, engine-source fingerprint and dependency versions.
+Recorded before/after reports are in `results/2026-09-14/`.
+
+## Run agent alone versus agent with Answerable
+
+```bash
+python scripts/compare_external_agents.py --output runs/agents-preflight
+```
+
+Without an available adapter and explicit model this writes `status: not_run`,
+null metrics and exits 2. It does not fabricate responses or recycle old EMT runs.
+
+For a real run, configure a **text-only** authenticated model command that reads
+a prompt from stdin and writes the model's plain-text answer to stdout. Pass its
+argv array with `--command-json` and pin its actual model with `--model`. For
+example, with a locally authenticated Claude CLI (replace MODEL_ID with your
+actual model identifier):
+
+```bash
+python scripts/compare_external_agents.py \
+  --model MODEL_ID \
+  --command-json '["claude","-p","--model","MODEL_ID","--output-format","text","--setting-sources","","--disallowedTools","*","--system-prompt","Follow the supplied analytical decision protocol exactly."]' \
+  --repetitions 2 --output runs/agents-real
+```
+
+Authentication/subscription access and the provider's usage charges apply. The
+adapter must disable ambient tools and memory; the harness cannot enforce those
+properties inside an arbitrary supplied command. Record the exact provider model,
+not a moving alias, and verify that the adapter actually uses it.
+
+The default is 12 cases × 2 repetitions × 2 conditions = 48 decisions. Both arms
+get the same aggregate evidence and claim. Neither sees labels or source paths.
+Cases are deterministically shuffled and condition order counterbalanced. Every
+model invocation starts in a fresh temporary directory without prior conversation.
+
+The tool arm may answer `CALL_ANSWERABLE`; the harness then executes the real
+`assess_answerability` handler, records its response, and asks the model for a
+final decision. There is at most one tool call and two model turns per decision.
+The model can decline to call the tool and can disagree; tool use is measured,
+not assumed. This is an intention-to-treat availability comparison on fixed
+summaries, not a native MCP/autonomous-shell benchmark. The second turn is part
+of the intervention and adds latency/model usage.
+
+Prompts, responses, adapter errors and actual tool calls are retained in
+`raw.jsonl`; the manifest pins data/question hashes, adapter argv, model, seed and
+repetitions. Metrics include false blocks, unsafe allows, tool calls, complete
+pairs, improvements, regressions and paired accuracy delta. Timeouts and malformed
+responses remain incomplete. Existing run reports cannot be overwritten.
+
+Unit-test adapters are test doubles only and are never published as agent results.
+The recorded preflight currently has **zero real model decisions**.

benchmarks/external/manifest.json

@@ -0,0 +1,23 @@
+{
+  "protocol": "external-smoke-v1",
+  "retrieved": "2026-09-14",
+  "normalization": "UTF-8 LF; dates encoded as UTC midnight for day/month labels, not measured instants; no row values changed",
+  "sources": [
+    {
+      "id": "flights",
+      "path": "sources/flights.csv",
+      "url": "https://raw.githubusercontent.com/mwaskom/seaborn-data/master/flights.csv",
+      "sha256": "237d834127d9c6355630d8f443a7a2377b5925923010009b59809ba0b67f4fac",
+      "unit": "month",
+      "units": "thousands of international airline passengers"
+    },
+    {
+      "id": "temperatures",
+      "path": "sources/temperatures.csv",
+      "url": "https://raw.githubusercontent.com/jbrownlee/Datasets/master/daily-min-temperatures.csv",
+      "sha256": "a31a6961536b0e48a1f8c886d91eb4de50e6747ea5cd5d2d45a8a5039068e406",
+      "unit": "day",
+      "units": "degrees Celsius, recorded minimum temperature"
+    }
+  ]
+}

benchmarks/external/results/2026-09-14/README.md

@@ -0,0 +1,50 @@
+# Recorded evaluation — 2026-09-14
+
+Both engine versions ran the same 12 cases and evaluator in this environment.
+The baseline is repository commit `59749d9060c43f0b833a85de1cc3cc3f7e39ddc9`.
+The corrected source is identified by `engine_sha256` in `after.json` and the
+commit containing this report. `cases_sha256` is identical in both reports.
+
+| Metric | Before | Corrected |
+| --- | ---: | ---: |
+| Valid exact descriptive claims | 4 | 4 |
+| Invalid claims | 8 | 8 |
+| Invalid claims in `allowed_claims` | 8 / 8 | 0 / 8 |
+| Valid claims omitted from `allowed_claims` | 0 / 4 | 0 / 4 |
+| Valid claims denied by the primary release gate | 2 / 4 | 0 / 4 |
+| Execution errors | 0 | 0 |
+
+The baseline's contradictory verdict/claim outputs are why both claim-level and
+release-level metrics are reported. The 8/8 count does not mean every downstream
+consumer would publish those claims: a consumer obeying the primary verdict
+might reject them despite their appearance in `allowed_claims`.
+
+This is a small, deliberately targeted regression suite. Datasets come from
+outside Answerable; questions and admissibility labels were authored for this
+review. Four duplicate cases are artificial perturbations. There is no claim
+that these observations estimate general accuracy or were independently judged.
+
+## Real agent comparison
+
+**Not run.** The environment had neither Claude/Codex CLI on PATH nor
+OpenAI/Anthropic/Gemini API keys. `agent-preflight.json` records zero decisions,
+null metrics and the missing adapter. Its manifest describes 48 planned decisions
+(12 cases × 2 repetitions × 2 arms); planned decisions are not observations.
+
+No agent/tool efficacy result is claimed. See the parent suite README for the
+command and adapter contract needed to run the paired experiment with a pinned,
+authenticated real model. Test doubles only exercise harness plumbing in tests.
+
+## Reproduce the before/after comparison
+
+From the corrected checkout, with dependencies installed:
+
+```bash
+python scripts/evaluate_external.py --output runs/external-after-reproduced
+git worktree add --detach ../answerable-before 59749d9060c43f0b833a85de1cc3cc3f7e39ddc9
+PYTHONPATH=../answerable-before/src python scripts/evaluate_external.py --output runs/external-before-reproduced
+```
+
+The baseline command intentionally exits 1 because unsafe allowed claims were
+found. The corrected command exits 0. Numerical observed means are also compared
+with independently computed means in the automated tests.