Skip to content

About

Checks AI-written clinical notes against the consultation transcript for omissions, changed facts and unsupported statements.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

scribecheck

Checks an AI-written clinical note against the consultation transcript, and reports omissions, changed facts and unsupported statements, each with the passage that supports the flag.

Status: pre-alpha. Rules mode (no model) runs; full mode arrives milestone by milestone; see docs/build-plan.md. Nothing here is ready for use yet.

Try it

Requires Python 3.11 or later. From a clone of this repository:

python -m pip install -e .
scribecheck demo

The demo runs offline in rules mode on three bundled synthetic consultations. Rules mode needs no model or API key; its findings are warnings and never fail a build. Check your own files with scribecheck check -t transcript.json -n note.md, or a folder of them with scribecheck batch cases/. scribecheck init writes a starter config file.

What it checks, and what it doesn't

  • It checks whether the note is faithful to the transcript, not whether the transcript is correct. A misheard drug name copied faithfully into the note will pass.
  • It is an evaluation tool for testing scribe systems, not a sign-off check on individual patients' notes in live care.
  • v1 supports English only.

Privacy

  • It runs on your machine. Rules mode makes no network calls; full mode talks only to the model endpoint you configure.
  • There is no telemetry, analytics or update checking.

Attribution

The omission-checking method follows Fox S, Markham L, Lail R, Karotsieris M, LLM Judges Verify Presence, Not Absence: Omission Blindness in AI Clinical Notes and What Recovers It, arXiv:2608.31016 (2026), which builds on MED-OMIT (Schumacher et al., 2023). Evaluation uses OmissionBench by Composo (data CC BY 4.0, harness MIT), which is built partly on PriMock57 and ACI-Bench. The prompts in src/scribecheck/prompts/route_one/ are copied unmodified from that harness (MIT, Copyright (c) 2026 Composo Limited; see the folder's LICENSE and README.md), and the severity rubric in scribecheck's own audit prompt is theirs.

Drug names come from RxNorm Current Prescribable Content (release of 5 October 2026). This product uses publicly available data from the U.S. National Library of Medicine (NLM), National Institutes of Health, Department of Health and Human Services; NLM is not responsible for the product and does not endorse or recommend this or any other product. Symptom terms come from the Symptom Ontology (SYMP, CC0 1.0).

Development

python -m venv .venv && source .venv/bin/activate   # Windows: .venv\Scripts\activate
python -m pip install -e ".[dev]"
python -m pytest -q
python -m ruff check . && python -m ruff format --check .
python -m mypy

Instructions for AI coding agents are in CLAUDE.md.

Licence

Apache-2.0. See LICENSE.

About

Checks AI-written clinical notes against the consultation transcript for omissions, changed facts and unsupported statements.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages