Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
45 changes: 45 additions & 0 deletions docs/research/e2-wave1/E1/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
# E1 / B-04: evaluation stratum schema compatibility

Base: CONTRACT `bf5740c266ff495319442402e0fa7d6aa23ccafa`.
Applied after SITE E4 commit `c949b2b`.

The schema now requires a string `stratum` with exactly `harmful`,
`benign_eval`, `harmful_jbb`, or `benign_jbb`
(`schemas/cc.e2_observation_row.v1.json`). These are the four evaluation
populations frozen in SITE `experiments/e2/freeze/PREREG_SELECTION.md:25-49`;
calibration remains separate. The frozen preregistration and measurement
contract text are unchanged.

`pre.log` records the reproduced failure: **exit 1, 7 failed, 7 passed**.
The governing validator accepts three original reference rows, then the actual
SITE analyzer raises `KeyError: 'stratum'`. Adding a valid stratum is rejected
as an additional property by that original schema.

`post.log` records **exit 0, 36 passed**, including required/type/enum checks,
the actual SITE analyzer load/complete-case round trip, existing observation
contract tests, and the dry-run tests. The full SITE
collection → validation → analysis round trip remains E2's responsibility.

`audit.log` records **exit 0** for both reference datasets' real validators,
Ruff, and `git diff --check`. Its metadata-only comparison proves that all
existing fields in the 24 example rows and 66 dry-run rows equal HEAD after
removing the new `stratum: harmful` field. The dry-run emitter and existing test
row factory now supply that field. These are implementation fixtures, not E2
measurements.

Focused post-fix command, run from CONTRACT:

```sh
E2_SITE_ROOT=/Users/pranavbhave/Developer/GitHub/e2-wave1-site \
PYTHONPATH=/tmp/e2-wave1-20261024/test-dependencies:src:. \
PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 \
/tmp/cc-validation-20261023/bin/python -B -m pytest \
-o addopts= -p no:cacheprovider -q -rs \
tests/unit/research/test_e2_stratum.py \
tests/unit/research/test_e2_observation_contract.py \
tests/unit/research/test_e2_dryrun.py
```

The isolated test dependency directory supplies scikit-learn 1.9.1, joblib
1.6.0, threadpoolctl 3.7.0, cloudpickle 3.1.2, and narwhals 2.27.1; repository
dependency declarations were not changed.
117 changes: 117 additions & 0 deletions docs/research/e2-wave1/E1/audit.log
Original file line number Diff line number Diff line change
@@ -0,0 +1,117 @@
PASS metadata-only: examples/e2/observations.example.jsonl: 24 rows, stratum=harmful added, all existing fields equal HEAD
COMMAND: /tmp/cc-validation-20261023/bin/python -B scripts/validate_e2_observations.py examples/e2/observations.example.jsonl --json
{
"report": {
"rows": 24,
"guardrails": [
"gr.alpha",
"gr.beta",
"gr.gamma"
],
"items": 8,
"complete_case_items": 6,
"complete_case_fraction": 0.75,
"missingness_by_guardrail": {
"gr.alpha": {
"observed": 8
},
"gr.beta": {
"observed": 7,
"timeout": 1
},
"gr.gamma": {
"observed": 7,
"upstream_moderation": 1
}
},
"guardrail_versions": {
"gr.alpha": [
"1.4.2"
],
"gr.beta": [
"2026.07"
],
"gr.gamma": [
"unknown"
]
},
"shared_dependency_groups": {
"base:alpha-llm": [
"gr.alpha",
"gr.gamma"
]
},
"duplicate_item_content": {
"4d818b3323ae6c36decf05fc9794f7ef81726902d1c1e68102e6f2583776d0d3": [
"item-003",
"item-007"
]
},
"replicate_unstable_pairs": []
},
"violations": [],
"warnings": [
"gr.gamma: 1 row(s) with missingness_code 'upstream_moderation'; complete-case analysis may itself manufacture or suppress dependence.",
"gr.gamma: version is 'unknown'; this limits any replication claim.",
"declared shared dependency 'base:alpha-llm' spans gr.alpha, gr.gamma; observed dependence may be mechanism-induced rather than a property of composition.",
"duplicate item content 4d818b3323ae... under ids item-003, item-007.",
"complete-case population is 6/8 items (75.0%); the missingness sensitivity analysis is mandatory, not optional."
]
}

E2 conformance OK (warnings are disclosures, not passes)

EXIT_CODE=0
PASS metadata-only: experiments/e2_dryrun/results/observations.jsonl: 66 rows, stratum=harmful added, all existing fields equal HEAD
COMMAND: /tmp/cc-validation-20261023/bin/python -B scripts/validate_e2_observations.py experiments/e2_dryrun/results/observations.jsonl --json
{
"report": {
"rows": 66,
"guardrails": [
"gr.keyword",
"gr.regex",
"gr.semantic"
],
"items": 22,
"complete_case_items": 22,
"complete_case_fraction": 1.0,
"missingness_by_guardrail": {
"gr.keyword": {
"observed": 22
},
"gr.regex": {
"observed": 22
},
"gr.semantic": {
"observed": 22
}
},
"guardrail_versions": {
"gr.keyword": [
"keyword_blocker.v1"
],
"gr.regex": [
"regex_filter.v1"
],
"gr.semantic": [
"semantic_filter.tfidf.v1"
]
},
"shared_dependency_groups": {},
"duplicate_item_content": {},
"replicate_unstable_pairs": []
},
"violations": [],
"warnings": []
}

E2 conformance OK (warnings are disclosures, not passes)

EXIT_CODE=0
COMMAND: /tmp/cc-validation-20261023/bin/python -m ruff check scripts/validate_e2_observations.py tests/unit/research/test_e2_stratum.py tests/unit/research/test_e2_observation_contract.py
All checks passed!

EXIT_CODE=0
COMMAND: git diff --check

EXIT_CODE=0
5 changes: 5 additions & 0 deletions docs/research/e2-wave1/E1/post.log
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
COMMAND: E2_SITE_ROOT=/Users/pranavbhave/Developer/GitHub/e2-wave1-site PYTHONPATH=/tmp/e2-wave1-20261024/test-dependencies:src:. PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 /tmp/cc-validation-20261023/bin/python -B -m pytest -o addopts= -p no:cacheprovider -q -rs tests/unit/research/test_e2_stratum.py tests/unit/research/test_e2_observation_contract.py tests/unit/research/test_e2_dryrun.py
.................................... [100%]
36 passed in 21.01s

EXIT_CODE=0
135 changes: 135 additions & 0 deletions docs/research/e2-wave1/E1/pre.log
Original file line number Diff line number Diff line change
@@ -0,0 +1,135 @@
COMMAND: E2_SITE_ROOT=/Users/pranavbhave/Developer/GitHub/e2-wave1-site PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 /tmp/cc-validation-20261023/bin/python -B -m pytest -o addopts= -p no:cacheprovider -q tests/unit/research/test_e2_stratum.py
FFFFF.......FF [100%]
=================================== FAILURES ===================================
___________ test_stratum_is_required_and_has_the_frozen_string_enum ____________

def test_stratum_is_required_and_has_the_frozen_string_enum() -> None:
> assert "stratum" in SCHEMA["required"]
E AssertionError: assert 'stratum' in ['schema_version', 'study_id', 'run_id', 'item_id', 'item_source', 'item_source_version', ...]

tests/unit/research/test_e2_stratum.py:34: AssertionError
_________ test_frozen_evaluation_stratum_is_schema_valid[benign_eval] __________

stratum = 'benign_eval'

@pytest.mark.parametrize("stratum", sorted(STRATA))
def test_frozen_evaluation_stratum_is_schema_valid(stratum: str) -> None:
row = load_rows(REFERENCE)[0]
row["stratum"] = stratum
> assert schema_violations([row], SCHEMA) == []
E assert ['row 1: <row... unexpected)'] == []
E
E Left contains one more item: "row 1: <row>: Additional properties are not allowed ('stratum' was unexpected)"
E Use -v to get more diff

tests/unit/research/test_e2_stratum.py:44: AssertionError
__________ test_frozen_evaluation_stratum_is_schema_valid[benign_jbb] __________

stratum = 'benign_jbb'

@pytest.mark.parametrize("stratum", sorted(STRATA))
def test_frozen_evaluation_stratum_is_schema_valid(stratum: str) -> None:
row = load_rows(REFERENCE)[0]
row["stratum"] = stratum
> assert schema_violations([row], SCHEMA) == []
E assert ['row 1: <row... unexpected)'] == []
E
E Left contains one more item: "row 1: <row>: Additional properties are not allowed ('stratum' was unexpected)"
E Use -v to get more diff

tests/unit/research/test_e2_stratum.py:44: AssertionError
___________ test_frozen_evaluation_stratum_is_schema_valid[harmful] ____________

stratum = 'harmful'

@pytest.mark.parametrize("stratum", sorted(STRATA))
def test_frozen_evaluation_stratum_is_schema_valid(stratum: str) -> None:
row = load_rows(REFERENCE)[0]
row["stratum"] = stratum
> assert schema_violations([row], SCHEMA) == []
E assert ['row 1: <row... unexpected)'] == []
E
E Left contains one more item: "row 1: <row>: Additional properties are not allowed ('stratum' was unexpected)"
E Use -v to get more diff

tests/unit/research/test_e2_stratum.py:44: AssertionError
_________ test_frozen_evaluation_stratum_is_schema_valid[harmful_jbb] __________

stratum = 'harmful_jbb'

@pytest.mark.parametrize("stratum", sorted(STRATA))
def test_frozen_evaluation_stratum_is_schema_valid(stratum: str) -> None:
row = load_rows(REFERENCE)[0]
row["stratum"] = stratum
> assert schema_violations([row], SCHEMA) == []
E assert ['row 1: <row... unexpected)'] == []
E
E Left contains one more item: "row 1: <row>: Additional properties are not allowed ('stratum' was unexpected)"
E Use -v to get more diff

tests/unit/research/test_e2_stratum.py:44: AssertionError
__________________ test_missing_stratum_is_a_schema_violation __________________

def test_missing_stratum_is_a_schema_violation() -> None:
row = load_rows(REFERENCE)[0]
row.pop("stratum", None)
> assert any("stratum" in message for message in schema_violations([row], SCHEMA))
E assert False
E + where False = any(<generator object test_missing_stratum_is_a_schema_violation.<locals>.<genexpr> at 0x107272670>)

tests/unit/research/test_e2_stratum.py:57: AssertionError
________ test_three_reference_rows_round_trip_into_actual_site_analyzer ________

tmp_path = PosixPath('/private/var/folders/nj/2c1_c61j3hn7h5fgv1gk0q2m0000gn/T/pytest-of-pranavbhave/pytest-12/test_three_reference_rows_roun0')

def test_three_reference_rows_round_trip_into_actual_site_analyzer(tmp_path: Path) -> None:
site_root = os.environ.get("E2_SITE_ROOT")
if not site_root:
pytest.skip("set E2_SITE_ROOT for the cross-repository analyzer check")
run_dir = Path(site_root) / "experiments/e2/run"
sys.path.insert(0, str(run_dir))
spec = importlib.util.spec_from_file_location("e2_stratum_analyzer", run_dir / "analyze.py")
assert spec is not None and spec.loader is not None
analyzer = importlib.util.module_from_spec(spec)
spec.loader.exec_module(analyzer)

rows = load_rows(REFERENCE)[:3]
for row, guard in zip(rows, ("lg4", "lg3", "sg2b"), strict=True):
row["guardrail_id"] = guard
assert schema_violations(rows, SCHEMA) == []
assert design_violations(rows) == []
fixture = tmp_path / "reference-three.jsonl"
fixture.write_text("".join(json.dumps(row) + "\n" for row in rows))

> events, missing, synthetic = analyzer.load(fixture)
^^^^^^^^^^^^^^^^^^^^^^

tests/unit/research/test_e2_stratum.py:79:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

path = PosixPath('/private/var/folders/nj/2c1_c61j3hn7h5fgv1gk0q2m0000gn/T/pytest-of-pranavbhave/pytest-12/test_three_reference_rows_roun0/reference-three.jsonl')

def load(path: Path):
"""rows -> {stratum: {item_id: {guard: event_bit}}}, plus missing map."""
events: dict = {}
missing: dict = {}
synthetic = False
for line in path.read_text().splitlines():
r = json.loads(line)
synthetic = synthetic or bool(r.get("synthetic"))
> s = r["stratum"]
^^^^^^^^^^^^
E KeyError: 'stratum'

../e2-wave1-site/experiments/e2/run/analyze.py:70: KeyError
=========================== short test summary info ============================
FAILED tests/unit/research/test_e2_stratum.py::test_stratum_is_required_and_has_the_frozen_string_enum
FAILED tests/unit/research/test_e2_stratum.py::test_frozen_evaluation_stratum_is_schema_valid[benign_eval]
FAILED tests/unit/research/test_e2_stratum.py::test_frozen_evaluation_stratum_is_schema_valid[benign_jbb]
FAILED tests/unit/research/test_e2_stratum.py::test_frozen_evaluation_stratum_is_schema_valid[harmful]
FAILED tests/unit/research/test_e2_stratum.py::test_frozen_evaluation_stratum_is_schema_valid[harmful_jbb]
FAILED tests/unit/research/test_e2_stratum.py::test_missing_stratum_is_a_schema_violation
FAILED tests/unit/research/test_e2_stratum.py::test_three_reference_rows_round_trip_into_actual_site_analyzer
7 failed, 7 passed in 0.27s

EXIT_CODE=1
Loading
Loading