diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml new file mode 100644 index 0000000..c03576b --- /dev/null +++ b/.github/workflows/tests.yml @@ -0,0 +1,20 @@ +name: tests + +on: + push: + pull_request: + +jobs: + pytest: + runs-on: ubuntu-latest + strategy: + fail-fast: false + matrix: + python-version: ["3.10", "3.11", "3.12"] + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + with: + python-version: ${{ matrix.python-version }} + - run: pip install -e ".[test]" + - run: pytest diff --git a/README.md b/README.md index 2c84e8e..3380f4b 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,7 @@ # constructcheck +[![tests](https://github.com/divyashie/constructcheck/actions/workflows/tests.yml/badge.svg)](https://github.com/divyashie/constructcheck/actions/workflows/tests.yml) + Flag construct mismatch in multimodal health ML datasets before publication. Companion to the JMIR preprint on construct validity in multimodal mental-health ML. @@ -14,6 +16,30 @@ checks with any scikit-learn estimator and writes one Markdown report. | `out_of_population` | Does performance survive an unseen site/source? | Leave-one-group-out accuracy is > `max_drop` below within-population CV for any group | | `label_card` | Is it documented who labelled the data, how, with what, in whom, and when? | A required field is missing or empty | +## Why + +This package grew out of a documented failure. A multimodal bipolar-disorder classifier reached +82.6% macro-F1 and passed the usual validation checks, yet its modalities had zero subject overlap +and shuffling the cross-modal pairing dropped performance by only 5.3%: the "fusion" paired +strangers who happened to share a label. The full case is in the +[JMIR preprint](PREPRINT_URL). These checks are the ones that would have caught it. + +## What it looks like + +[`examples/demo.py`](examples/demo.py) runs every check on both synthetic datasets. On +`make_mismatched()` — two unrelated cohorts fused by matching rows on label — three of the four +checks warn ([full report](examples/report_mismatched.md)): + +| Check | Status | Summary | +|---|---|---| +| pairing_test | WARN | Breaking the A-B pairing (within_class shuffle) changes accuracy by only +0.01 (threshold 0.10); the modalities may not be contributing jointly. | +| subject_overlap | WARN | Possible construct mismatch: the two tables share no subject IDs. | +| out_of_population | WARN | Accuracy drops by 0.40 when group 'site_2' is held out (threshold 0.10). | +| label_card | PASS | Label card has all required provenance fields. | + +On `make_linked()`, where both modalities come from the same subjects, all four pass +([full report](examples/report_linked.md)). + ## Install ```bash @@ -36,11 +62,17 @@ results = run_all( model=LogisticRegression(max_iter=1000), df_a=d["a"], df_b=d["b"], subject_col="subject_id", # the separate modality tables group_col="source", - label_card_path="label_card.yaml", + label_card_path="label_card.yaml", # copy the example card first; see below ) write_report(results, "constructcheck_report.md") ``` +To try this before writing your own card, copy the example that ships with the package: + +```bash +python -c "from importlib.resources import files; import shutil; shutil.copy(files('constructcheck') / 'label_card.example.yaml', 'label_card.yaml')" +``` + Checks without the inputs they need are reported as `skip`. Each check is also callable on its own and returns a `CheckResult(name, status, summary, metrics, details)`. diff --git a/examples/demo.py b/examples/demo.py new file mode 100644 index 0000000..8111e13 --- /dev/null +++ b/examples/demo.py @@ -0,0 +1,32 @@ +"""Run every check on the two synthetic datasets and write one report for each. + + python examples/demo.py + +Writes examples/report_linked.md (should pass) and examples/report_mismatched.md (should warn). +""" + +import os +from importlib.resources import files +from pathlib import Path + +from sklearn.linear_model import LogisticRegression + +from constructcheck import run_all, write_report +from constructcheck.datasets import make_linked, make_mismatched + +OUT = Path(__file__).parent +# The example card ships with the package; a relative path keeps the report free of local paths. +LABEL_CARD = os.path.relpath(files("constructcheck") / "label_card.example.yaml") + +for name, make in [("linked", make_linked), ("mismatched", make_mismatched)]: + d = make() + results = run_all( + df=d["fused"], + modality_a_cols=d["a_cols"], modality_b_cols=d["b_cols"], label_col="label", + model=LogisticRegression(max_iter=1000), + df_a=d["a"], df_b=d["b"], subject_col="subject_id", + group_col="source", + label_card_path=LABEL_CARD, + ) + path = write_report(results, OUT / f"report_{name}.md", title=f"constructcheck report: make_{name}()") + print(f"{path}: " + ", ".join(f"{r.name}={r.status}" for r in results)) diff --git a/examples/report_linked.md b/examples/report_linked.md new file mode 100644 index 0000000..63ba85f --- /dev/null +++ b/examples/report_linked.md @@ -0,0 +1,102 @@ +# constructcheck report: make_linked() + +| Check | Status | Summary | +|---|---|---| +| pairing_test | PASS | Breaking the A-B pairing (within_class shuffle) drops accuracy by 0.24; pairing carries signal. | +| subject_overlap | PASS | 500 shared subjects (100% of all IDs) with consistent label definitions. | +| out_of_population | PASS | Held-out-group accuracy stays within 0.10 of within-population accuracy for all 2 groups. | +| label_card | PASS | Label card has all required provenance fields. | + +## pairing_test + +**Status: PASS**. Breaking the A-B pairing (within_class shuffle) drops accuracy by 0.24; pairing carries signal. + +| Metric | Value | +|---|---| +| paired_acc | 0.900 | +| shuffled_acc | 0.658 | +| drop | 0.242 | +| modality_a_only_acc | 0.588 | +| modality_b_only_acc | 0.590 | + +- **shuffle**: within_class +- **cv**: 5 +- **min_drop**: 0.100 +- **n**: 500 + +## subject_overlap + +**Status: PASS**. 500 shared subjects (100% of all IDs) with consistent label definitions. + +| Metric | Value | +|---|---| +| n_subjects_a | 500 | +| n_subjects_b | 500 | +| n_shared | 500 | +| overlap_fraction | 1.000 | +| label_tvd | 0.000 | +| shared_label_agreement | 1.000 | + +- **label_values_a**: ['0', '1'] +- **label_values_b**: ['0', '1'] +- **label_distribution_a**: {'0': 0.526, '1': 0.474} +- **label_distribution_b**: {'0': 0.526, '1': 0.474} + +## out_of_population + +**Status: PASS**. Held-out-group accuracy stays within 0.10 of within-population accuracy for all 2 groups. + +| Metric | Value | +|---|---| +| within_population_acc | 0.900 | +| leave_one_group_out_acc | 0.888 | +| drop | 0.012 | +| worst_group | site_1 | +| worst_group_drop | 0.020 | +| n_groups | 2 | + +**per_group** + +| | n | within_acc | held_out_acc | drop | +|---|---|---|---|---| +| site_2 | 252 | 0.905 | 0.901 | 0.004 | +| site_1 | 248 | 0.895 | 0.875 | 0.020 | + +- **group_col**: source +- **max_drop**: 0.100 + +## label_card + +**Status: PASS**. Label card has all required provenance fields. + +| Metric | Value | +|---|---| +| n_required | 5 | +| n_missing | 0 | + +Label card (`src/constructcheck/label_card.example.yaml`), verbatim: + +```yaml +# constructcheck label card +# One card per label column. Every field below is required; fill each in plainly. +# If two modalities were labelled differently, write one card per modality. + +annotator: >- + Two board-certified psychiatrists, blind to the wearable data; + disagreements resolved by a third rater. +method: >- + Structured clinical interview at enrolment; label = current mood episode + (euthymia / depression / mania) on the interview date. +instrument: SCID-5-RV, with YMRS and MADRS severity scores +population: >- + Adult outpatients (18-65) with a DSM-5 bipolar I/II diagnosis, + recruited from two university clinics. +date_collected: 2024-03 to 2025-01 + +# Optional, but useful for reviewers +label_values: + 0: euthymia + 1: depression + 2: mania +notes: Label refers to the episode at interview, not to each sensor window. +``` diff --git a/examples/report_mismatched.md b/examples/report_mismatched.md new file mode 100644 index 0000000..b451101 --- /dev/null +++ b/examples/report_mismatched.md @@ -0,0 +1,102 @@ +# constructcheck report: make_mismatched() + +| Check | Status | Summary | +|---|---|---| +| pairing_test | WARN | Breaking the A-B pairing (within_class shuffle) changes accuracy by only +0.01 (threshold 0.10); the modalities may not be contributing jointly. | +| subject_overlap | WARN | Possible construct mismatch: the two tables share no subject IDs. | +| out_of_population | WARN | Accuracy drops by 0.40 when group 'site_2' is held out (threshold 0.10). | +| label_card | PASS | Label card has all required provenance fields. | + +## pairing_test + +**Status: WARN**. Breaking the A-B pairing (within_class shuffle) changes accuracy by only +0.01 (threshold 0.10); the modalities may not be contributing jointly. + +| Metric | Value | +|---|---| +| paired_acc | 0.906 | +| shuffled_acc | 0.898 | +| drop | 0.008 | +| modality_a_only_acc | 0.882 | +| modality_b_only_acc | 0.672 | + +- **shuffle**: within_class +- **cv**: 5 +- **min_drop**: 0.100 +- **n**: 500 + +## subject_overlap + +**Status: WARN**. Possible construct mismatch: the two tables share no subject IDs. + +| Metric | Value | +|---|---| +| n_subjects_a | 500 | +| n_subjects_b | 500 | +| n_shared | 0 | +| overlap_fraction | 0.000 | +| label_tvd | 0.248 | +| shared_label_agreement | None | + +- **label_values_a**: ['0', '1'] +- **label_values_b**: ['0', '1'] +- **label_distribution_a**: {'0': 0.524, '1': 0.476} +- **label_distribution_b**: {'1': 0.724, '0': 0.276} + +## out_of_population + +**Status: WARN**. Accuracy drops by 0.40 when group 'site_2' is held out (threshold 0.10). + +| Metric | Value | +|---|---| +| within_population_acc | 0.906 | +| leave_one_group_out_acc | 0.532 | +| drop | 0.374 | +| worst_group | site_2 | +| worst_group_drop | 0.404 | +| n_groups | 2 | + +**per_group** + +| | n | within_acc | held_out_acc | drop | +|---|---|---|---|---| +| site_1 | 250 | 0.904 | 0.560 | 0.344 | +| site_2 | 250 | 0.908 | 0.504 | 0.404 | + +- **group_col**: source +- **max_drop**: 0.100 + +## label_card + +**Status: PASS**. Label card has all required provenance fields. + +| Metric | Value | +|---|---| +| n_required | 5 | +| n_missing | 0 | + +Label card (`src/constructcheck/label_card.example.yaml`), verbatim: + +```yaml +# constructcheck label card +# One card per label column. Every field below is required; fill each in plainly. +# If two modalities were labelled differently, write one card per modality. + +annotator: >- + Two board-certified psychiatrists, blind to the wearable data; + disagreements resolved by a third rater. +method: >- + Structured clinical interview at enrolment; label = current mood episode + (euthymia / depression / mania) on the interview date. +instrument: SCID-5-RV, with YMRS and MADRS severity scores +population: >- + Adult outpatients (18-65) with a DSM-5 bipolar I/II diagnosis, + recruited from two university clinics. +date_collected: 2024-03 to 2025-01 + +# Optional, but useful for reviewers +label_values: + 0: euthymia + 1: depression + 2: mania +notes: Label refers to the episode at interview, not to each sensor window. +```