fix: enforce claim blockers and separate identifying assumptions

[Jairogelpi]

The assessment runner could treat overlap as exchangeability, accept support in only one stratum, and include invalid claims in allowed_claims despite integrity blockers. Finding provenance also pointed to the positivity check regardless of origin.

This PR fixes those four inconsistencies, separates verified facts, declared assumptions and unverifiable conditions, and adds adversarial regression coverage. Conditional identification now requires explicit named assumptions and yields ANSWERABLE_WITH_ASSUMPTIONS. Pure descriptive questions no longer require causal identification or inferential power. ADR 0002 explains compatibility and limits; this does not implement adjusted causal estimation or general semantic verification of arbitrary prose.

Validation:

• Six initial adversarial tests failed before implementation.
• make verify passed locally: 215 tests, 96.01% branch-aware coverage, strict mypy, Ruff, 21 schemas and 119 traceability entries.
• sdist/wheel built successfully; installed wheel and MCP server smoke check passed.
• Two external datasets, 12 project-authored cases: invalid claims in allowed_claims fall from 8/8 to 0/8; valid claims denied by the release gate fall from 2/4 to 0/4. All 4 valid claims remain allowed. Source snapshots, hashes, before/after results and reproduction instructions are included.

The paired agent-alone vs agent-with-tool harness is implemented and its actual handler invocation is tested. Real model comparison is NOT RUN: the environment has no authenticated model CLI or API keys. The preflight records zero decisions and null metrics. Test doubles are not published as model results. This draft remains explicit about that outstanding evaluation.

Frozen EMT releases and historical model outputs are unchanged. Live fixtures now declare identifying assumptions, so their conditional baseline verdicts are not identical inputs to the historical comparison. No quality threshold was lowered.

The remote tree exactly matches the locally tested commit d88db5d (tree dffc1cabc21d1a85189262ab908b67a05c27cbcf). Upload used the authorized GitHub connector because terminal git had no credentials.

Risks: stricter causal contracts may block previously accepted questions; discrete full-stratum support is conservative for continuous covariates. External questions/labels are project-authored, not an independent scientific validation. Rollback is reverting this commit; frozen warrants/results were not rewritten.

[Copilot]

🟡 Changes recommended

Unresolved critical and moderate review findings need correction before approval.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

This PR strengthens causal-identification gating, claim blocking, provenance tracking, and external regression coverage.

Changes:

• Adds explicit identification assumptions and conditional verdicts.
• Enforces claim-scoped blockers and improves evidence reporting.
• Adds adversarial tests, external evaluation artifacts, and agent/tool harness coverage.

File summaries

File	Summary
tests/test_validity_adversarial.py	Adds adversarial validity regression tests.
tests/test_mutation_benchmark.py	Updates mutation benchmark expectations.
tests/test_external_evaluation.py	Tests evaluation and harness behavior.
tests/e2e/test_runner_edge_cases.py	Covers conditional identification edge cases.
src/answerable/reports/markdown.py	Reports evidence status; nit (1): preserve percentage formatting for ratio metrics.
src/answerable/mutation_benchmark.py	Updates benchmark contracts and fixtures.
src/answerable/evidence/verdict.py	Critical (1): align claim blocking with precedence for scoped integrity blockers.
src/answerable/evidence/graph.py	Adds unverifiable-condition nodes.
src/answerable/evidence/claims.py	Enforces causal claim classes.
src/answerable/domain/serialization.py	Supports frozenset serialization.
src/answerable/causal/contract.py	Adds named identification assumptions.
src/answerable/application/spec_loader.py	Loads identification assumptions.
src/answerable/application/assessment_runner.py	Moderate (2): preserve candidate identity for duplicate claim text; moderate (1): derive ledger conditions from strategy and assumptions; moderate (1): correct prediction-leakage provenance.
scripts/evaluate_external.py	Moderate (1): exclude non-boolean decisions from rate denominators.
scripts/compare_external_agents.py	Moderate (1): resolve the contradictory tool-use prompt.
schemas/v1/causal-contract.schema.json	Defines the assumption field.
requirements/traceability.yaml	Records requirement coverage.
README.md	Documents validity review and evaluation limits.
docs/adr/0002-explicit-identification-and-claim-blockers.md	Records design decisions and limits.
benchmarks/external/sources/flights.csv	Adds external benchmark data.
benchmarks/external/results/2026-09-14/verification.txt	Records local verification.
benchmarks/external/results/2026-09-14/README.md	Documents recorded results.
benchmarks/external/results/2026-09-14/cases.json	Freezes evaluation cases.
benchmarks/external/results/2026-09-14/before.json	Records baseline results.
benchmarks/external/results/2026-09-14/agent-preflight.json	Records model preflight status.
benchmarks/external/results/2026-09-14/agent-preflight-manifest.json	Moderate (1): question hash conflicts with cases.json; regenerate or mark stale.
benchmarks/external/results/2026-09-14/after.json	Records corrected results.
benchmarks/external/README.md	Documents benchmark and harness usage.
benchmarks/external/manifest.json	Pins external source metadata and hashes.

Review details

Suppressed comments (6)

benchmarks/external/results/2026-09-14/agent-preflight-manifest.json:17

• The recorded preflight manifest hashes flights-full-observed's question as acdd508d..., while the frozen cases.json hashes the same case question as d2511422...; the data hash is identical. Since the harness imports the same materialize function used to create the case set, this leaves the reproducibility record internally inconsistent—regenerate this manifest from the current evaluator or mark it as stale.

      "question_sha256": "acdd508d8a59acb8d09c83713c18b092746a7512cceca2d57aa9a1f53d6087db",

scripts/compare_external_agents.py:67

• The shared prompt says Do not use external files, tools or memory, then the tool arm immediately instructs the model to call a tool. That contradictory protocol can suppress tool calls and bias the experiment that is meant to measure tool availability; prohibit external files and memory, but explicitly allow the offered tool in the tool arm.

        "Assess whether the exact claim is supported by the provided evidence. "
        "Descriptive sample statements do not imply population inference or causality. "
        "Do not use external files, tools or memory. Explain briefly, then end with exactly "
        "DECISION: ALLOW or DECISION: BLOCK on its own line.\n"

scripts/evaluate_external.py:144

• The rate denominators include records with allowed is None (for example, adapter errors), so an unparseable answer dilutes both rates instead of being excluded from the evaluated denominator. This can make a run with many errors look safer than it is; compute the rates over only valid boolean decisions and keep the error count separate.

        "false_block_rate": false_blocks / len(valid) if valid else None,
        "unsafe_allow_rate": unsafe / len(invalid) if invalid else None,

src/answerable/application/assessment_runner.py:290

• This ledger is hard-coded for every causal strategy. A randomized, IV, DiD, or discontinuity contract will still report absence of unmeasured confounding as the relevant unverified condition, and a causal contract with no declared assumptions will say that declared design assumptions require evidence. Derive these conditions from the selected strategy and declared assumptions, or use wording that does not assert assumptions were declared.

                "Absence of unmeasured confounding is not established by observed overlap.",
                "The declared design assumptions require evidence beyond this table.",

src/answerable/application/assessment_runner.py:187

• prediction_leakage is emitted by the prediction/feature-availability branch, but this provenance map points it to chk_temporal_maturity, whose registered check and rationale cover observation-window completion. The new graph will therefore still attribute leakage to the wrong computation; add a distinct leakage execution check (selected when those columns are mapped), or make the existing check explicitly cover both conditions.

    "invalid_event_time": "chk_temporal_maturity",
    "timezone_ambiguity": "chk_temporal_maturity",
    "immature_cohort": "chk_temporal_maturity",
    "right_censoring": "chk_temporal_maturity",
    "prediction_leakage": "chk_temporal_maturity",

src/answerable/reports/markdown.py:20

• The renderer now prints every outcome as a raw mean. That regresses ratio metrics such as retention: a value of 0.5 is shown as observed mean 0.5 instead of the existing 50.0%, making the human-facing warrant ambiguous. Branch on question['outcome']['value_type'] so ratios retain percentage formatting while continuous metrics use means.

        f"{item['group']}: observed mean {item['rate']:.6g}; "
        f"{item['entities']} {question['unit_of_analysis']}s"

• Files reviewed: 29/30 changed files
• Comments generated: 2
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.