fix: align alpha scope and add stable release validation gates

.github/workflows/ci.yml

@@ -15,13 +15,20 @@ concurrency:
 
 jobs:
   quality:
-    name: Quality / Python ${{ matrix.python-version }}
-    runs-on: ubuntu-latest
+    name: Quality / ${{ matrix.os }} / Python ${{ matrix.python-version }}
+    runs-on: ${{ matrix.os }}
     timeout-minutes: 15
     strategy:
       fail-fast: false
       matrix:
+        os: [ubuntu-latest]
         python-version: ["3.11", "3.12"]
+        include:
+          - os: windows-latest
+            python-version: "3.12"
+    defaults:
+      run:
+        shell: bash
     steps:
       - uses: actions/checkout@v4
         with:
@@ -59,6 +66,9 @@ jobs:
           || test $? -eq 2
           answerable --json warrant verify --warrant runs/campaign_retention/warrant.json
 
+      - name: Run external-data regression evaluation
+        run: python scripts/evaluate_external.py --output runs/external-ci
+
       - name: Run Epistemic Mutation Testing release gate
         if: matrix.python-version == '3.12'
         run: answerable benchmark mutations --output runs/epistemic-mutations
@@ -70,24 +80,26 @@ jobs:
         run: python scripts/validate_schemas.py
 
   package:
-    name: Package
-    runs-on: ubuntu-latest
-    timeout-minutes: 10
+    name: Package / ${{ matrix.os }} / Python ${{ matrix.python-version }}
+    runs-on: ${{ matrix.os }}
+    timeout-minutes: 15
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest]
+        python-version: ["3.11", "3.12"]
+        include:
+          - os: windows-latest
+            python-version: "3.12"
     steps:
       - uses: actions/checkout@v4
         with:
           persist-credentials: false
       - uses: actions/setup-python@v5
         with:
-          python-version: "3.12"
+          python-version: ${{ matrix.python-version }}
           cache: pip
       - run: python -m pip install --disable-pip-version-check build
       - run: python -m build
-      - name: Clean wheel install with MCP extra
-        run: |
-          python -m venv /tmp/answerable-wheel
-          WHEEL="$(ls dist/*.whl)"
-          /tmp/answerable-wheel/bin/python -m pip install "${WHEEL}[mcp]"
-          /tmp/answerable-wheel/bin/python -c "import answerable; print(answerable.__version__)"
-          /tmp/answerable-wheel/bin/python -c "from answerable.interfaces.mcp_stdio import build_server; build_server(); print('wheel MCP OK')"
-          /tmp/answerable-wheel/bin/answerable benchmark mutations --output /tmp/answerable-wheel-benchmark
+      - name: Clean wheel install, version agreement, CLI and MCP
+        run: python scripts/check_distribution.py

CHANGELOG.md

@@ -4,6 +4,21 @@ All notable changes follow Keep a Changelog and Semantic Versioning.
 
 ## [Unreleased]
 
+### Added
+- Explicit validation evidence and measurable stable-release criteria, including independent held-out questions, paired real-agent evaluation and external pilots.
+- Windows/Python 3.12 CI alongside Linux/Python 3.11 and 3.12; clean-wheel checks on both operating systems.
+- Regression checks for agreement between distribution, public Python and CLI versions.
+
+### Changed
+- README now identifies the package as alpha and distinguishes implemented checks from the complete target specification and historical benchmark results.
+- External evaluation protocol defines independent review, dataset-level holdout, error accounting and paired comparison criteria; these planned evaluations have not been completed.
+- Causal identification requires explicit named assumptions and reports conditional support separately from verified facts. Descriptive questions no longer require causal identification or inferential power (PR #22).
+
+### Fixed
+- Overlap no longer substitutes for exchangeability; support is required in every target stratum. Deterministic blockers govern claim lists and findings link to their producing checks (PR #22).
+- Public `answerable.__version__` agrees with package and CLI version `0.3.0` instead of reporting `0.2.0`.
+
+
 ## [0.3.0] - 2026-08-17
 
 ### Added

README.md

@@ -4,7 +4,7 @@
 
 ### Evidence before answers.
 
-**A deterministic validity layer for analytics and AI agents.**
+**Explicit data and analysis checks for analytics and AI agents.**
 
 Your code has tests. Your data has tests. **Your conclusions should too.**
 
@@ -20,13 +20,15 @@ Your code has tests. Your data has tests. **Your conclusions should too.**
 </div>
 
 > [!IMPORTANT]
-> **A correct number does not imply a justified conclusion.** Answerable checks whether the available evidence supports the claim an analyst or agent is about to make — and fails closed when it does not.
+> **Status: alpha.** Answerable checks explicit data and analysis conditions within its implemented scope. Its reports separate verified facts, declared assumptions and conditions not verifiable from the supplied table.
+>
+> CLI and MCP have automated tests. The external-data evaluation uses two datasets and 12 project-authored questions. Improvement from giving an agent Answerable has **not yet been demonstrated with real models**. See [validation evidence](docs/VALIDATION.md) and [stable release criteria](docs/RELEASE_CRITERIA.md).
 
 ![Answerable terminal demo](docs/demo.svg)
 
 ### AI agent + Answerable
 
-The agent explores and explains; Answerable deterministically checks whether the evidence supports the conclusion before it reaches a user.
+The agent explores and explains; Answerable checks the declared conditions before the agent publishes a conclusion. This is the intended integration workflow, not an experimentally demonstrated improvement in agent decisions.
 
 <img src="docs/answerable-agent-workflow.svg" alt="AI agent workflow with and without Answerable" width="100%">
 
@@ -84,7 +86,9 @@ A 100% score on project-authored rules/cases is not general superiority over LLM
 
 ## Quickstart
 
-### 1. Install the production package
+### 1. Install the published alpha package
+
+PyPI contains tagged releases; unreleased changes on `main` require a source installation.
 
 For CLI + MCP integration:
 
@@ -152,20 +156,19 @@ Most analytical tooling stops at one of three boundaries: whether the data is st
 
 > **Does this evidence justify this conclusion?**
 
-| Capability | Data tests | Statistical code | LLM-as-judge | Answerable |
-| --- | ---: | ---: | ---: | ---: |
-| Check schema/nulls/duplicates | Yes | Sometimes | Sometimes | **Yes** |
-| Check grain and joins | Sometimes | Manual | Sometimes | **Yes** |
-| Check temporal maturity | Rarely | Manual | Sometimes | **Yes** |
-| Check causal identification | No | Manual | Nondeterministic | **Yes** |
-| Check prediction-time leakage | Rarely | Manual | Nondeterministic | **Yes** |
-| Decide whether a claim must be retracted | No | No | Nondeterministic | **Yes** |
-| Produce machine-readable blockers | Sometimes | Custom | Variable | **Yes** |
-| Produce allowed/forbidden claims | No | No | Variable | **Yes** |
-| Produce a verifiable evidence artifact | No | No | No | **Evidence Warrant** |
-| Deterministic release gate | Yes | Possible | No | **Yes** |
-
-Answerable is not trying to replace data-quality frameworks, statistical libraries or LLMs. It is the **claim-validity layer between them and the statement that reaches a human or downstream agent**.
+| Implemented surface | What it checks or produces | Boundary |
+| --- | --- | --- |
+| File assessment | Declared entity grain, duplicates, timing and configured checks | Assessment currently evaluates the first source; this is not arbitrary multi-table validation. |
+| Descriptive comparisons | Observed group means and admissibility under the declared contract | No general numerical or semantic verification of arbitrary prose. |
+| Causal gate | Declared identifying assumptions, adjustment mapping and empirical binary-treatment support | Overlap is not exchangeability. Declared assumptions remain unverified; no adjusted causal effect estimation. |
+| Other analytical checks | Explicit metadata for timing leakage, metric changes, power and missingness in supported cases | Detector coverage is bounded; it does not certify arbitrary predictive or statistical analyses. |
+| Verdict and claim lists | Deterministic blockers and permitted candidate claim classes | Consumers must read the verdict, blockers and assumptions as well as the claim list. |
+| Evidence Warrant | Findings, provenance and integrity verification | Integrity verification does not establish scientific truth or independent certification. |
+| CLI, Python and MCP | Executable local interfaces | HTTP/web modules are contract surfaces, not a complete hosted product. |
+
+The complete target specification includes capabilities beyond the current executable
+assessment path. A domain module or passing unit test does not establish end-to-end
+support for every workflow described in that specification.
 
 ## Who it is for
 
@@ -212,7 +215,7 @@ Evidence Warrant
 
 ### Frozen benchmark
 
-The `emt-v2` benchmark is hash-addressed and immutable once published. The current release gate runs 112 paired evidence mutations across seven independent failure classes.
+The `emt-v2` benchmark is hash-addressed and immutable once published. The current release gate runs 112 paired evidence mutations across seven failure classes.
 
 ### Real external-agent runs
 
@@ -222,7 +225,7 @@ The repository includes full prompts, raw responses, decisions and scoring artif
 
 The engineering verification path includes branch-aware coverage of at least 95%, strict mypy, Ruff, schema validation, requirement traceability, clean package build/install, deterministic mutation testing and CodeQL.
 
-## Answerable vs real LLM agents
+## Historical comparison with real LLM agents
 
 On the frozen `emt-v2` case set, Answerable and external LLMs were asked to update a conclusion after the evidence changed. The repository contains the raw runs and scoring artifacts.
 
@@ -233,6 +236,8 @@ On the frozen `emt-v2` case set, Answerable and external LLMs were asked to upda
 | Overall oracle accuracy | **100%** | 79.0% | 60.7% |
 | Unsafe `KEEP` | **0%** | 0% | 0% |
 
+These are historical, project-authored oracle scores, not an independent estimate of general reliability. The paired agent-with-tool experiment remains pending.
+
 The interesting failure is not arithmetic. Both models handled familiar leakage failures well, but frequently softened several statistical and causal invalidations into `QUALIFY` instead of retracting the claim. The full methodology, raw responses, sample-size caveats and per-class results are preserved under [`benchmarks/epistemic_mutations/results/2026-08-17-emt-v2-claude-codex/`](benchmarks/epistemic_mutations/results/2026-08-17-emt-v2-claude-codex/).
 
 This benchmark measures the models **without access to Answerable**. Giving the model Answerable as a callable tool is a separate intervention; the point of the MCP integration is to move the validity decision outside the model's own judgment.

benchmarks/external/README.md

@@ -104,3 +104,58 @@ responses remain incomplete. Existing run reports cannot be overwritten.
 
 Unit-test adapters are test doubles only and are never published as agent results.
 The recorded preflight currently has **zero real model decisions**.
+
+## Planned independent acceptance evaluation
+
+**Status: protocol only; not collected or executed.** The current scripts
+materialize the 12 smoke cases above. They do not yet ingest a general independent
+question bank, adjudicate labels, calculate dataset-cluster intervals, or fetch
+provider billing. Implement and validate those capabilities before using them for
+[stable-release acceptance](../../docs/RELEASE_CRITERIA.md).
+
+1. Collect at least 100 questions across at least 10 externally sourced datasets.
+   Record source/license, retrieval date, transformations and content hashes. Include
+   valid, invalid, ambiguous and out-of-scope claims, not just mutations of failures.
+2. Split by whole dataset before engine evaluation: reserve at least five datasets
+   with at least 40 valid and 40 invalid questions in total. Keep development cases
+   separate; do not tune on reserved data. More questions than the minimum may be
+   needed to satisfy the split and coverage requirements.
+3. An independent domain reviewer, who did not implement the relevant checks, labels
+   each question before seeing engine output. Store claim, scope, expected decision,
+   factual basis, required assumptions, severity, reasoning and reviewer/date.
+   Record disagreements and adjudication; unresolved cases remain ambiguous.
+4. Include duplicates, missing values, temporal ambiguity, immature cohorts, partial
+   overlap, confounding, mismatched adjustments, metric changes, leakage and valid
+   descriptive questions. Include unsupported formats/designs explicitly. Do not
+   expect support beyond the advertised scope or secretly remove those cases.
+5. Freeze labels, split, hashes, engine revision, metrics and release thresholds.
+   Run every case and retain exceptions/timeouts. For valid/invalid labels report
+   both claim-list and verdict-gated decisions. Report ambiguous/out-of-scope cases
+   separately, with the engine's actual response and whether it states a limitation.
+6. Publish numerator/denominator for unsafe allows and false blocks, execution
+   errors, and per-dataset/per-class counts. Give uncertainty at the dataset level;
+   zero observed failures is not proof of zero risk. An errored case has no valid
+   decision and cannot be counted as a correct block.
+
+### Paired real-model extension
+
+Use exactly the frozen questions/evidence in both arms, a pinned model version,
+identical base prompts, fresh sessions, counterbalanced order and a declared budget.
+Keep the existing intention-to-treat policy: tool availability does not imply use.
+Record all prompts, responses, real tool calls, errors and latency. Capture provider
+usage and cost when the adapter exposes them; otherwise mark cost unavailable.
+Never infer actual billed cost from a guessed token count.
+
+Primary outcome: unsafe-allow rate alone minus unsafe-allow rate with the tool on
+invalid questions. Also report valid false-block differences, regressions,
+improvements, complete pairs and tool usage. Apply identical scoring in both arms.
+Use paired dataset-cluster resampling for a 95% interval; repeated calls and related
+questions stay in the same cluster. Freeze the resampling seed and method before
+running models. The release gate requires a positive safety improvement interval,
+a false-block increase of no more than two percentage points and a complete run.
+A small/inconclusive interval or a baseline with zero unsafe allows does not establish
+benefit; retain the result and extend the evaluation prospectively.
+
+The present 48-decision smoke plan does not satisfy this independent protocol.
+Passing test-double adapters verifies plumbing only. A native MCP host evaluation
+would be a separately specified experiment; do not conflate it with mediated calls.