Add demo with committed reports, CI, and release README

.github/workflows/tests.yml

@@ -0,0 +1,20 @@
+name: tests
+
+on:
+  push:
+  pull_request:
+
+jobs:
+  pytest:
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: ["3.10", "3.11", "3.12"]
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+      - run: pip install -e ".[test]"
+      - run: pytest

README.md

@@ -1,5 +1,7 @@
 # constructcheck
 
+[![tests](https://github.com/divyashie/constructcheck/actions/workflows/tests.yml/badge.svg)](https://github.com/divyashie/constructcheck/actions/workflows/tests.yml)
+
 Flag construct mismatch in multimodal health ML datasets before publication.
 Companion to the JMIR preprint on construct validity in multimodal mental-health ML.
 
@@ -14,6 +16,30 @@ checks with any scikit-learn estimator and writes one Markdown report.
 | `out_of_population` | Does performance survive an unseen site/source? | Leave-one-group-out accuracy is > `max_drop` below within-population CV for any group |
 | `label_card` | Is it documented who labelled the data, how, with what, in whom, and when? | A required field is missing or empty |
 
+## Why
+
+This package grew out of a documented failure. A multimodal bipolar-disorder classifier reached
+82.6% macro-F1 and passed the usual validation checks, yet its modalities had zero subject overlap
+and shuffling the cross-modal pairing dropped performance by only 5.3%: the "fusion" paired
+strangers who happened to share a label. The full case is in the
+[JMIR preprint](PREPRINT_URL). These checks are the ones that would have caught it.
+
+## What it looks like
+
+[`examples/demo.py`](examples/demo.py) runs every check on both synthetic datasets. On
+`make_mismatched()` — two unrelated cohorts fused by matching rows on label — three of the four
+checks warn ([full report](examples/report_mismatched.md)):
+
+| Check | Status | Summary |
+|---|---|---|
+| pairing_test | WARN | Breaking the A-B pairing (within_class shuffle) changes accuracy by only +0.01 (threshold 0.10); the modalities may not be contributing jointly. |
+| subject_overlap | WARN | Possible construct mismatch: the two tables share no subject IDs. |
+| out_of_population | WARN | Accuracy drops by 0.40 when group 'site_2' is held out (threshold 0.10). |
+| label_card | PASS | Label card has all required provenance fields. |
+
+On `make_linked()`, where both modalities come from the same subjects, all four pass
+([full report](examples/report_linked.md)).
+
 ## Install
 
 ```bash
@@ -36,11 +62,17 @@ results = run_all(
     model=LogisticRegression(max_iter=1000),
     df_a=d["a"], df_b=d["b"], subject_col="subject_id",   # the separate modality tables
     group_col="source",
-    label_card_path="label_card.yaml",
+    label_card_path="label_card.yaml",  # copy the example card first; see below
 )
 write_report(results, "constructcheck_report.md")
 ```
 
+To try this before writing your own card, copy the example that ships with the package:
+
+```bash
+python -c "from importlib.resources import files; import shutil; shutil.copy(files('constructcheck') / 'label_card.example.yaml', 'label_card.yaml')"
+```
+
 Checks without the inputs they need are reported as `skip`. Each check is also callable on its own
 and returns a `CheckResult(name, status, summary, metrics, details)`.
 

examples/demo.py

@@ -0,0 +1,32 @@
+"""Run every check on the two synthetic datasets and write one report for each.
+
+    python examples/demo.py
+
+Writes examples/report_linked.md (should pass) and examples/report_mismatched.md (should warn).
+"""
+
+import os
+from importlib.resources import files
+from pathlib import Path
+
+from sklearn.linear_model import LogisticRegression
+
+from constructcheck import run_all, write_report
+from constructcheck.datasets import make_linked, make_mismatched
+
+OUT = Path(__file__).parent
+# The example card ships with the package; a relative path keeps the report free of local paths.
+LABEL_CARD = os.path.relpath(files("constructcheck") / "label_card.example.yaml")
+
+for name, make in [("linked", make_linked), ("mismatched", make_mismatched)]:
+    d = make()
+    results = run_all(
+        df=d["fused"],
+        modality_a_cols=d["a_cols"], modality_b_cols=d["b_cols"], label_col="label",
+        model=LogisticRegression(max_iter=1000),
+        df_a=d["a"], df_b=d["b"], subject_col="subject_id",
+        group_col="source",
+        label_card_path=LABEL_CARD,
+    )
+    path = write_report(results, OUT / f"report_{name}.md", title=f"constructcheck report: make_{name}()")
+    print(f"{path}: " + ", ".join(f"{r.name}={r.status}" for r in results))

examples/report_linked.md

@@ -0,0 +1,102 @@
+# constructcheck report: make_linked()
+
+| Check | Status | Summary |
+|---|---|---|
+| pairing_test | PASS | Breaking the A-B pairing (within_class shuffle) drops accuracy by 0.24; pairing carries signal. |
+| subject_overlap | PASS | 500 shared subjects (100% of all IDs) with consistent label definitions. |
+| out_of_population | PASS | Held-out-group accuracy stays within 0.10 of within-population accuracy for all 2 groups. |
+| label_card | PASS | Label card has all required provenance fields. |
+
+## pairing_test
+
+**Status: PASS**. Breaking the A-B pairing (within_class shuffle) drops accuracy by 0.24; pairing carries signal.
+
+| Metric | Value |
+|---|---|
+| paired_acc | 0.900 |
+| shuffled_acc | 0.658 |
+| drop | 0.242 |
+| modality_a_only_acc | 0.588 |
+| modality_b_only_acc | 0.590 |
+
+- **shuffle**: within_class
+- **cv**: 5
+- **min_drop**: 0.100
+- **n**: 500
+
+## subject_overlap
+
+**Status: PASS**. 500 shared subjects (100% of all IDs) with consistent label definitions.
+
+| Metric | Value |
+|---|---|
+| n_subjects_a | 500 |
+| n_subjects_b | 500 |
+| n_shared | 500 |
+| overlap_fraction | 1.000 |
+| label_tvd | 0.000 |
+| shared_label_agreement | 1.000 |
+
+- **label_values_a**: ['0', '1']
+- **label_values_b**: ['0', '1']
+- **label_distribution_a**: {'0': 0.526, '1': 0.474}
+- **label_distribution_b**: {'0': 0.526, '1': 0.474}
+
+## out_of_population
+
+**Status: PASS**. Held-out-group accuracy stays within 0.10 of within-population accuracy for all 2 groups.
+
+| Metric | Value |
+|---|---|
+| within_population_acc | 0.900 |
+| leave_one_group_out_acc | 0.888 |
+| drop | 0.012 |
+| worst_group | site_1 |
+| worst_group_drop | 0.020 |
+| n_groups | 2 |
+
+**per_group**
+
+|  | n | within_acc | held_out_acc | drop |
+|---|---|---|---|---|
+| site_2 | 252 | 0.905 | 0.901 | 0.004 |
+| site_1 | 248 | 0.895 | 0.875 | 0.020 |
+
+- **group_col**: source
+- **max_drop**: 0.100
+
+## label_card
+
+**Status: PASS**. Label card has all required provenance fields.
+
+| Metric | Value |
+|---|---|
+| n_required | 5 |
+| n_missing | 0 |
+
+Label card (`src/constructcheck/label_card.example.yaml`), verbatim:
+
+```yaml
+# constructcheck label card
+# One card per label column. Every field below is required; fill each in plainly.
+# If two modalities were labelled differently, write one card per modality.
+
+annotator: >-
+  Two board-certified psychiatrists, blind to the wearable data;
+  disagreements resolved by a third rater.
+method: >-
+  Structured clinical interview at enrolment; label = current mood episode
+  (euthymia / depression / mania) on the interview date.
+instrument: SCID-5-RV, with YMRS and MADRS severity scores
+population: >-
+  Adult outpatients (18-65) with a DSM-5 bipolar I/II diagnosis,
+  recruited from two university clinics.
+date_collected: 2024-03 to 2025-01
+
+# Optional, but useful for reviewers
+label_values:
+  0: euthymia
+  1: depression
+  2: mania
+notes: Label refers to the episode at interview, not to each sensor window.
+```

examples/report_mismatched.md

@@ -0,0 +1,102 @@
+# constructcheck report: make_mismatched()
+
+| Check | Status | Summary |
+|---|---|---|
+| pairing_test | WARN | Breaking the A-B pairing (within_class shuffle) changes accuracy by only +0.01 (threshold 0.10); the modalities may not be contributing jointly. |
+| subject_overlap | WARN | Possible construct mismatch: the two tables share no subject IDs. |
+| out_of_population | WARN | Accuracy drops by 0.40 when group 'site_2' is held out (threshold 0.10). |
+| label_card | PASS | Label card has all required provenance fields. |
+
+## pairing_test
+
+**Status: WARN**. Breaking the A-B pairing (within_class shuffle) changes accuracy by only +0.01 (threshold 0.10); the modalities may not be contributing jointly.
+
+| Metric | Value |
+|---|---|
+| paired_acc | 0.906 |
+| shuffled_acc | 0.898 |
+| drop | 0.008 |
+| modality_a_only_acc | 0.882 |
+| modality_b_only_acc | 0.672 |
+
+- **shuffle**: within_class
+- **cv**: 5
+- **min_drop**: 0.100
+- **n**: 500
+
+## subject_overlap
+
+**Status: WARN**. Possible construct mismatch: the two tables share no subject IDs.
+
+| Metric | Value |
+|---|---|
+| n_subjects_a | 500 |
+| n_subjects_b | 500 |
+| n_shared | 0 |
+| overlap_fraction | 0.000 |
+| label_tvd | 0.248 |
+| shared_label_agreement | None |
+
+- **label_values_a**: ['0', '1']
+- **label_values_b**: ['0', '1']
+- **label_distribution_a**: {'0': 0.524, '1': 0.476}
+- **label_distribution_b**: {'1': 0.724, '0': 0.276}
+
+## out_of_population
+
+**Status: WARN**. Accuracy drops by 0.40 when group 'site_2' is held out (threshold 0.10).
+
+| Metric | Value |
+|---|---|
+| within_population_acc | 0.906 |
+| leave_one_group_out_acc | 0.532 |
+| drop | 0.374 |
+| worst_group | site_2 |
+| worst_group_drop | 0.404 |
+| n_groups | 2 |
+
+**per_group**
+
+|  | n | within_acc | held_out_acc | drop |
+|---|---|---|---|---|
+| site_1 | 250 | 0.904 | 0.560 | 0.344 |
+| site_2 | 250 | 0.908 | 0.504 | 0.404 |
+
+- **group_col**: source
+- **max_drop**: 0.100
+
+## label_card
+
+**Status: PASS**. Label card has all required provenance fields.
+
+| Metric | Value |
+|---|---|
+| n_required | 5 |
+| n_missing | 0 |
+
+Label card (`src/constructcheck/label_card.example.yaml`), verbatim:
+
+```yaml
+# constructcheck label card
+# One card per label column. Every field below is required; fill each in plainly.
+# If two modalities were labelled differently, write one card per modality.
+
+annotator: >-
+  Two board-certified psychiatrists, blind to the wearable data;
+  disagreements resolved by a third rater.
+method: >-
+  Structured clinical interview at enrolment; label = current mood episode
+  (euthymia / depression / mania) on the interview date.
+instrument: SCID-5-RV, with YMRS and MADRS severity scores
+population: >-
+  Adult outpatients (18-65) with a DSM-5 bipolar I/II diagnosis,
+  recruited from two university clinics.
+date_collected: 2024-03 to 2025-01
+
+# Optional, but useful for reviewers
+label_values:
+  0: euthymia
+  1: depression
+  2: mania
+notes: Label refers to the episode at interview, not to each sensor window.
+```