Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
# Frozen benchmark releases are hash-addressed. Keep them byte-identical on
# every platform so checksums verify after a clone on Windows.
benchmarks/releases/** -text

# External source snapshots and recorded reports also have byte-level hashes.
# Do not rewrite their line endings during checkout, including on Windows.
benchmarks/external/** -text
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ All notable changes follow Keep a Changelog and Semantic Versioning.
- Causal identification requires explicit named assumptions and reports conditional support separately from verified facts. Descriptive questions no longer require causal identification or inferential power (PR #22).

### Fixed
- Preserve byte-identical external benchmark snapshots and reports on Windows checkouts; keep strict SHA-256 validation enabled and test `core.autocrlf=true` checkout reproducibility.
- Overlap no longer substitutes for exchangeability; support is required in every target stratum. Deterministic blockers govern claim lists and findings link to their producing checks (PR #22).
- Public `answerable.__version__` agrees with package and CLI version `0.3.0` instead of reporting `0.2.0`.

Expand Down
5 changes: 5 additions & 0 deletions benchmarks/external/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,11 @@ Answerable. It is a **smoke evaluation, not independent scientific validation**.

`manifest.json` records source URLs, retrieval date, SHA-256 and normalization.
Vendored CSV snapshots reproduce the run without a network request.
Git attributes preserve these snapshots and recorded reports byte for byte on
Windows as well as Linux. SHA-256 validation remains strict: modified sources
are rejected rather than silently normalized. If an older checkout already
converted the files to CRLF, use a fresh clone containing this fix; do not
regenerate the manifest hashes to accept the converted files.

- `flights`: 144 monthly observations from the seaborn example-data repository,
the international airline-passenger series also distributed as R AirPassengers.
Expand Down
15 changes: 14 additions & 1 deletion docs/VALIDATION.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Validation evidence

Status: alpha. Evidence reviewed on 2026-09-15. This document separates software
Status: alpha. Evidence reviewed on 2026-09-16. This document separates software
verification from evidence of analytical validity and integration benefit.
The complete target in PRODUCT_SPEC.md is not a statement that all workflows
are implemented or independently validated.
Expand Down Expand Up @@ -33,6 +33,19 @@ engine on frozen project oracles. They do not measure the effect of giving an
agent the tool. The new paired harness records **not_run**, zero actual decisions
and null metrics until an authenticated real-model adapter is provided.

## Windows portability follow-up

After PR #23 merged as `c432104cdcfdfcfd51655a232646d3e95db2999b`,
[CI](https://github.com/Jairogelpi/answerable_data/actions/runs/34963430790)
passed all three package jobs and both Linux quality jobs. Windows quality ended
with 213 passed and three failed tests: source hash mismatch in the external suite.
A successful package installation did not establish a passing Windows test suite.

A regression reproduces the mismatch with Git `core.autocrlf=true`. The follow-up
fix extends byte-preserving Git attributes to external snapshots and recorded
reports. Strict SHA-256 checking is retained. Its CI must pass on Windows before
the portability defect is considered resolved; the old failing run remains evidence.

## Current boundaries

- The runner currently assesses the first data source. Multi-source contracts or
Expand Down
6 changes: 3 additions & 3 deletions requirements/traceability.yaml
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
schema_version: "1.0"
requirements:
INV-006:
implementation: ["src/answerable/domain/serialization.py"]
tests: ["tests/test_serialization.py"]
status: verified
implementation: ["src/answerable/domain/serialization.py", ".gitattributes"]
tests: ["tests/test_serialization.py", "tests/test_benchmark_checkout.py"]
status: implemented
INV-011:
implementation: ["src/answerable/persistence/sqlite.py"]
tests: ["tests/test_sqlite_repository.py"]
Expand Down
38 changes: 38 additions & 0 deletions tests/test_benchmark_checkout.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
"""INV-006: external snapshots retain their recorded bytes on Windows checkouts."""

import hashlib
import json
import shutil
import subprocess
from pathlib import Path


def test_INV_006_external_snapshots_survive_autocrlf_checkout(tmp_path: Path) -> None:
root = Path(__file__).parents[1]
repository = tmp_path / "repository"
repository.mkdir()
subprocess.run(["git", "init", str(repository)], check=True, capture_output=True)
shutil.copyfile(root / ".gitattributes", repository / ".gitattributes")
sources = json.loads((root / "benchmarks/external/manifest.json").read_text())["sources"]
for source in sources:
relative = Path("benchmarks/external") / source["path"]
target = repository / relative
target.parent.mkdir(parents=True, exist_ok=True)
shutil.copyfile(root / relative, target)
subprocess.run(["git", "-c", "core.autocrlf=false", "add", "."], cwd=repository, check=True)
checkout = tmp_path / "checkout"
subprocess.run(
[
"git",
"-c",
"core.autocrlf=true",
"checkout-index",
"--all",
f"--prefix={checkout.as_posix()}/",
],
cwd=repository,
check=True,
)
for source in sources:
data = (checkout / "benchmarks/external" / source["path"]).read_bytes()
assert hashlib.sha256(data).hexdigest() == source["sha256"]
Comment on lines +17 to +38
Loading