diff --git a/CHANGELOG.md b/CHANGELOG.md index 5d62a64..f3b9eb3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,65 @@ All notable changes to PaperTrace are documented here. The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); versions follow [SemVer](https://semver.org/). +## [Unreleased] + +### Added — `papertrace mcp`: an MCP server, for hosts that are not a shell + +The roadmap's *drive PaperTrace as a tool from any MCP-capable client*. +`papertrace mcp` serves nine tools over stdio, behind a new `[mcp]` extra +(`mcp>=2.2,<3`, the SDK's current major): `start_audit` and `audit_status` +run and follow the `papertrace run` pipeline; `audit_summary`, `list_claims`, +`get_claim`, `get_evidence`, `list_references`, `list_gaps` and `get_scout` +read any case folder, made over MCP or by the CLI. `get_evidence` returns the +red-box crops as images. + +The server is held to the reports' rules rather than given its own. It +computes no verdict. Every count and verdict carries the disclosures +`disclosures.py` decided for the four report formats — serialised the +viewer's way, with a parity test extending the four-format contract to this +fifth reader — and a limited audit says so first and last. A missing or +unreadable `results.json` is refused with its reason, never answered with +empty counts; `null` in the manifest's three-state fields stays `null`; no +local PDF path is served, and no evidence image from outside `/out/`. A +verdict read on its own — `get_claim`, the evidence caption — carries the +run-level caveats too. Each audit leaves `/mcp_audit.json`: a case whose +last MCP audit failed or never finished is refused rather than read, because +its files may mix two runs. The job refreshes the record every 15 s, so a +second host's server can tell a live audit from an abandoned one and will not +start another in the same folder. The tools' outputs and that record are +published in +`schemas/mcp_tools.schema.json` and `schemas/mcp_audit.schema.json`. + +An audit takes minutes and a host built on the TypeScript SDK gives a request +60 s, so `start_audit` returns at once and `audit_status` long-polls for at +most 50 s. The audit runs in-process — `ask.py` stays the only file in `src/` +that starts a process — on one thread at a time, because the pipeline's +console, `ask`'s model record and stdin are process-global. Inside it nobody +can be asked: a reference label whose readings disagree is withheld, as the +CLI does without a terminal, and settling one still takes a person at a +terminal. And each audit starts with `ask.forget_models()`: the CLI never +cleared the model record because it ran one command per process, and a +second audit whose judging made no calls would otherwise have named the +first one's model as its judge. Design, rejected alternatives and the +decision record: +`docs/superpowers/specs/2026-09-26-mcp-server-design.md`, +`docs/adr/0004-mcp-server.md`. + +The README's *From an MCP host* names the apps that can start it — Claude +Desktop, Claude Code, Cursor, VS Code, Windsurf, Gemini CLI, Codex CLI — and +the ones that cannot: ChatGPT and Claude in a browser reach MCP servers over +the internet only. Whichever app drives the tools, the audit is judged through +`claude -p`, so Claude Code must be installed and signed in. The server +installs from GitHub with `uv tool install`, no clone needed. Claude Code has +been seen connecting to that install; the other apps' configurations follow +their documentation and have not each been run. + +### Changed — `[full]` and `[dev]` carry the MCP SDK + +`[full]`, the quick start's everything-install, now includes the SDK, so +`papertrace mcp` works after it. `[dev]` — what CI installs — carries it so +the MCP tests run on every Python in the matrix instead of skipping. + ## [0.7.1] — 2026-09-19 (beta) Two fixes the 0.7.0 showcase run exposed, and the showcase itself. diff --git a/CLAUDE.md b/CLAUDE.md index a187136..e3d161a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -64,6 +64,8 @@ pytest tests/test_refs.py -q # one module pytest tests/test_refs.py::test_paywalled_is_honest -q # one test pytest -k coverage -q # by name pytest tests/test_report_viewer_js.py -q # the viewer's browser logic, under node (skips without it) +pytest tests/test_mcp_server.py -q # the MCP server, via the SDK's in-memory client + one real stdio run +papertrace mcp # serve over stdio for an MCP host ([mcp] extra); nothing on stdout ruff check src tests scripts evals uv build # sdist + wheel (python -m build is NOT installed) ``` @@ -101,7 +103,9 @@ and torch with it — but never *runs* it: the ~500 MB layout models download on first use, not on install, and every test pins `backend="pymupdf"` (`check_claims` makes `backend` a required argument so none can forget). `playwright` must still stay out of CI; it is the `png` extra, needed only for -`--png`, with `playwright install chromium` once. +`--png`, with `playwright install chromium` once. The MCP SDK is the opposite +case: the `[mcp]` extra, and in `[dev]` as well, so CI runs +`tests/test_mcp_server.py` instead of skipping it. ## Architecture @@ -262,6 +266,8 @@ ingest → extract → refs → scout → check → highlight → report global: `_LAST_MODEL` was overwritten by every call, so a run whose judging made zero calls — every cited source `not_retrieved`, nothing to judge — printed the reference-list model as the `Checker:` of verdicts it never saw. + The record assumes one command per process; `forget_models()` clears it for + a process that outlives one run, which the MCP server calls per audit. `_ask`'s signature is frozen at `(prompt, model=None)`; every offline test that patches the seam does so with a two-argument lambda, which is why the site travels out of band in a context manager instead of as a third @@ -326,7 +332,42 @@ ingest → extract → refs → scout → check → highlight → report DOM-free module (parsing `annotated.md`, anchoring quotes with the claim's `ctx_ids` block searched first, crop sets, filters, the claim map) that `tests/test_report_viewer_js.py` runs under node, and `viewer_app.js`, which - only draws. Disclosures are never re-derived in JS. + only draws. Disclosures are never re-derived in JS. The MCP server is a fifth + reader: it serialises the same `Disclosure` objects with `_disclosure_dict`, + generically, and `tests/test_mcp_server.py`'s parity loops assert every + token reaches it. +- **`mcp_server.py`** — `papertrace mcp`, an MCP server over stdio + (`mcp>=2.2,<3`, the `[mcp]` extra) with nine tools. Seven read a case folder + and compute nothing: every verdict and count travels with its disclosures, + the scope sentence first and again as the final `limited` key; a missing + `results.json` is a `ToolError`, never empty counts; evidence is served only + from inside `/out/`, and no `pdf_path` leaves. `start_audit` runs + `cli._run_pipeline` **in-process, on one daemon thread** — never a + subprocess, which `tests/test_ask.py` would refuse — and `audit_status` + long-polls for at most 50 s, inside the 60 s a TypeScript-SDK host allows a + request. A server outlives many audits, which the CLI never did, so the job + resets what a process used to reset by exiting: `cli.console` becomes the + job's log (under stdio, stdout is the protocol's wire), `sys.stdin` becomes + an empty stream (`_interactive()` is False, so a disputed label is withheld + with `chosen_by: "default"` and the person-gated escalation is unreachable — + do not offer MCP elicitation in its place: a host may be a model), and + `ask.forget_models()` runs first. Those three are process-global, hence + **one audit at a time**. The job writes `/mcp_audit.json` (`running`, + refreshed every `HEARTBEAT_SECONDS`, then the outcome), and read tools refuse + a case whose last MCP audit is running, failed, or never finished: a failed + rerun leaves one run's manifest beside another's verdicts. `running` is a + measurement — fresh is another process's live audit (refused, never started + over), unrefreshed for `STALE_AFTER_SECONDS` is `interrupted` — and never + say a cause the record's age does not show. The record on disk decides, not + a server's memory: a later `results.json`/`scout.json` supersedes a failure + or a run without the scout. A verdict read alone — `get_claim`, the evidence caption — carries + the run-level disclosures as well as its own. Outputs are published in + `schemas/mcp_tools.schema.json`, each definition's keys held equal to its + `TypedDict`'s by a test. **Any new process-global state in the pipeline + must be reset per audit here.** In SDK v2 only a `ToolError`'s message reaches the model — any + other exception is a bare "Error executing tool" — so every refusal is one. + Return types are `typing_extensions.TypedDict` with `structured_output=True`: + `typing.TypedDict` silently cost every tool its output schema below 3.12. The `case/` folder is the unit of work: one case per paper, guarded by `_guard_case`. It is gitignored by design — manuscripts stay local. diff --git a/README.md b/README.md index b18834f..2a01acb 100644 --- a/README.md +++ b/README.md @@ -62,7 +62,7 @@ pip install -e ".[full]" # layout-aware ingest — ~1.4 GB, export PAPERTRACE_EMAIL="you@example.org" # Unpaywall asks for a contact address ``` -Three ways in, all producing the same case folder and reports: +Four ways in, all producing the same case folder and reports: - **Guided** — run `papertrace` and answer the questions. It checks your setup first, then asks one thing at a time: the paper, where to keep the audit, @@ -79,6 +79,11 @@ Three ways in, all producing the same case folder and reports: interviews you (the paper, your PDFs, your journal's reviewer form as screenshots), retrieves and checks with evidence as it goes, drafts the findings, and ends by writing the viewer. +- **From an AI app** — Claude Desktop, Claude Code, Cursor, VS Code, Windsurf, + Gemini CLI or Codex CLI can run `papertrace mcp` as an MCP server: start an + audit, follow it, and read every verdict with its caveats and its evidence + crops. Not ChatGPT, nor Claude in a browser — neither can start a program on + your computer. See [From an MCP host](#from-an-mcp-host).

The guided audit in a terminal: a setup check, then the questions one at a time — the paper's path, the case folder, cited PDFs already to hand, the paper's own supplements, a DOI found on the first page, a contact email, whether to write the interactive viewer, whether to limit the audit — and the cost stated as a number of model calls before asking permission to start. @@ -94,6 +99,7 @@ Output lands in `/out/`: `report.md` always, `report_viewer.html` with |---|---| | `pip install -e .` | the standard install — layout-aware ingest of the paper **and** its cited sources (docling's ~500 MB layout models download on first run) | | `pip install -e ".[png]"` | plus PNG rendering of the report looks | +| `pip install -e ".[mcp]"` | plus `papertrace mcp`, the server for MCP hosts | | `pip install -e ".[dev]"` | the test and lint tooling, exactly what CI installs — run the suite as `python -m pytest` | | `pip install -e ".[dev,png]"` | everything | @@ -175,6 +181,9 @@ source that was read flat. verdict and evidence crop, and say how each supplement was attached. - Audit a slice on request — `--max-claims N`, `--max-sources N` — and end every report by stating, in numbers, what was left out. +- Serve the same audit to an MCP host (`papertrace mcp`): start one, follow + it, and read every verdict with the disclosures the reports carry and the + evidence crops as images. The server computes no verdict of its own. - Keep you responsible for interpretation: it prepares evidence and drafts; the conclusions are yours. @@ -202,6 +211,9 @@ source that was read flat. on a source's supplementary table is judged against its main text alone. - Guarantee an exhaustive literature search — the scout is search-based, and absence from its lists proves nothing. +- Settle a disputed reference label from an MCP host. Nobody is asked over + MCP, so verdicts on that label are withheld — the CLI's answer when no + terminal is attached — and `papertrace refs` in a terminal settles it. - Replace peer review or your research judgement. ## The viewer @@ -280,6 +292,135 @@ each records which model answered. Ingest, crops and reports give the same output for the same input. Retrieval and the scout query live services, so a re-run months later can find a different set of sources. +## From an MCP host + +`papertrace mcp` lets an AI app that speaks the +[Model Context Protocol](https://modelcontextprotocol.io) (MCP) drive +PaperTrace: start an audit, follow it, and read every verdict with its caveats +and its evidence crops. It runs on your own computer, beside your case +folders — the app starts it as a local program (MCP's *stdio* transport), and +nothing is served over the network. + +**Which apps.** Any app that can start a local MCP server: + +| App | Where PaperTrace is added | +|---|---| +| Claude Desktop | Settings → Developer → Edit Config: `claude_desktop_config.json` | +| Claude Code | `claude mcp add` — below | +| Cursor | `~/.cursor/mcp.json` | +| VS Code, GitHub Copilot in Agent mode | Command Palette → *MCP: Open User Configuration* | +| Windsurf | `~/.codeium/windsurf/mcp_config.json` | +| Gemini CLI | `~/.gemini/settings.json` | +| OpenAI Codex CLI | `codex mcp add`, or `~/.codex/config.toml` | + +**ChatGPT, and Claude in a web browser, cannot run it:** they reach MCP +servers over the internet only, and cannot start a program on your computer. +Use the Claude Desktop app, not claude.ai in a browser. + +Two things hold whichever app you use: + +- **The audit is judged by Claude, through Claude Code on the same computer.** + `start_audit` runs the pipeline `papertrace run` runs, and its model calls + are `claude -p` — so Claude Code must be installed and signed in, which needs + a paid Claude plan, even when the app driving the tools is Cursor, Gemini CLI + or Codex. That app's own model chooses the tools and relays what they return; + it does not judge the claims. Reading an audit that already exists needs no + Claude Code at all. +- **What has been tested** is the server against the MCP Python SDK's own + client, in memory and over a real stdio pipe, and Claude Code connecting to + an install made as below. The other configurations follow each app's + documentation; they have not each been run in the app itself. + +**Install.** [uv](https://docs.astral.sh/uv/) installs PaperTrace into an +environment of its own, and fetches a suitable Python if you have none — no +clone, no Git: + +```bash +uv tool install --python 3.12 "papertrace[mcp] @ https://github.com/defraction0/PaperTrace/archive/refs/heads/main.zip" +uv tool update-shell # puts `papertrace` on your PATH; open a new terminal after it +which papertrace # the full path the app needs — on Windows: where.exe papertrace +``` + +The download is large, most of it PyTorch for the layout-aware reader — 6 GB +installed on Linux, where PyTorch brings its GPU libraries. From a checkout, +`pip install -e ".[mcp]"` does the same. Give the app the **full path** +printed above: an app starts its servers from a working directory of its own, +with its own and often minimal `PATH`. + +**Claude Code**, for every project on this computer: + +```bash +claude mcp add --env PAPERTRACE_EMAIL=you@example.org --transport stdio --scope user \ + papertrace -- /full/path/to/papertrace mcp +``` + +Keep another option between `--env` and the name — Claude Code reads a name +straight after `--env` as one more `KEY=value`. `claude mcp list` then shows +whether it connected. + +**Claude Desktop, Cursor, Windsurf and Gemini CLI** share one shape: + +```json +{ + "mcpServers": { + "papertrace": { + "command": "/full/path/to/papertrace", + "args": ["mcp"], + "env": { + "PAPERTRACE_EMAIL": "you@example.org", + "PATH": "/Users/you/.local/bin:/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin" + } + } + } +} +``` + +`PATH` is there for `claude`. On a Mac an app opened from the Dock may not see +your terminal's `PATH`, and without `claude` `start_audit` refuses; the Claude +Code installer puts it in `~/.local/bin`, and `which claude` names the folder +if yours is elsewhere. Windows apps see the `PATH` a new terminal sees, so +there `PATH` can usually be left out — and the command is +`"C:\\Users\\you\\.local\\bin\\papertrace.exe"`, each `\` written twice, as +JSON requires. On Windows, some Claude Desktop installs read their +configuration from +`%LOCALAPPDATA%\Packages\Claude_…\LocalCache\Roaming\Claude\` rather than +the file *Edit Config* opens: if PaperTrace is missing under **+ → Connectors** +after a full restart, add the same entry there. + +**VS Code** takes the same command as a `"servers"` entry with +`"type": "stdio"`; **Codex CLI** as +`codex mcp add papertrace --env PAPERTRACE_EMAIL=you@example.org -- /full/path/to/papertrace mcp`. + +| Tool | What it does | +|---|---| +| `start_audit` | runs the `papertrace run` pipeline in the background and returns at once | +| `audit_status` | running, finished or failed, with the log tail; waits up to 50 s per call | +| `audit_summary` | verdict counts, references obtained, the judging model, coverage, and every run-level disclosure | +| `list_claims`, `get_claim` | each claim with the paper's own sentence, each source's verdict and rationale, and its caveats | +| `get_evidence` | the red-box page crops a verdict rests on, as images | +| `list_references` | the retrieval manifest and the numbering state — without local file paths | +| `list_gaps`, `get_scout` | uncited assertions, citation places no claim reached, unchecked claims; the scout's registers | + +The read tools work on any case folder, made over MCP or by the CLI, and +change nothing. Every verdict arrives with the caveats the reports print +beside it, and a limited audit says so first and last. An audit takes minutes, +longer than many apps wait for a single request, which is why it runs as a job +that `audit_status` follows. + +`start_audit` spends what `papertrace run` spends: `claude -p` calls on this +computer's Claude login, and the open-access services. The contact email comes +from `start_audit`'s `email`, from `PAPERTRACE_EMAIL`, or from the address +`papertrace` saved. A missing manuscript, a `claude` the server cannot find, no +email, or a case folder holding another paper is refused before anything is +spent. One audit runs at a time per server. A case folder is not read while +its audit runs, nor after one that failed or never finished — its files may +mix two runs — which the server knows from the `mcp_audit.json` each audit +leaves and keeps fresh while it runs; a second app's server will not start an +audit in a folder another is writing. Nobody is at an MCP call to answer a +question, so a reference label whose readings disagree is withheld, never +settled — that takes you, at a terminal. Nothing but the protocol reaches +stdout; what the pipeline prints comes back in `audit_status`'s log. + ## Tables and figures are evidence too A number in a table cell, or text drawn inside a vector figure, is in the PDF's @@ -367,8 +508,9 @@ paper.pdf ─────ingest──▶ clean.md + source_map.json (page Seven stages, each a subcommand, chained by `run`; every stage writes a file into the case folder and the next reads only that. The JSON contracts are versioned in [`schemas/`](schemas/). The two skills in -[`.claude/skills/`](.claude/skills/) drive the same tools interactively; the -audit craft lives in [`prompts/review_core.md`](prompts/review_core.md). +[`.claude/skills/`](.claude/skills/) drive the same tools interactively, and +`papertrace mcp` serves them to any MCP host; the audit craft lives in +[`prompts/review_core.md`](prompts/review_core.md). ## Ethics & scope @@ -453,7 +595,8 @@ box glyphs. - [ ] More scout backends (OpenAlex, Semantic Scholar) - [ ] Other LLM backends (local and API models alongside headless Claude Code) - [ ] Claude Desktop integration -- [ ] MCP server — drive PaperTrace as a tool from any MCP-capable client +- [x] MCP server — drive PaperTrace as a tool from any MCP-capable client: + `papertrace mcp` (unreleased) - [ ] DOCX ingest - [ ] Revision (R1) mode polish - [ ] Figure-vs-text consistency pass (batch) diff --git a/docs/adr/0004-mcp-server.md b/docs/adr/0004-mcp-server.md new file mode 100644 index 0000000..8582bce --- /dev/null +++ b/docs/adr/0004-mcp-server.md @@ -0,0 +1,58 @@ +# ADR 0004 — An MCP server, in-process, that can never ask + +- **Status:** accepted +- **Date:** 2026-09-26 +- **Related:** ADR [0003](0003-llm-reference-list.md), `src/papertrace/mcp_server.py`, + `src/papertrace/ask.py`, `docs/superpowers/specs/2026-09-26-mcp-server-design.md` + +## Context + +The roadmap asked for an MCP server: PaperTrace driven as a tool from any +MCP-capable host, not only from Claude Code through a shell. Two things make +that more than a wrapper here. + +The rules this codebase keeps were written for readers of a file and for a +process that lives for one command. An MCP server has a new reader — a model +reading JSON — and a new process shape, one that outlives many audits. +`ask._MODELS` was correct only because a process ran one command. + +And an audit takes minutes, while a host built on the TypeScript SDK times a +request out at 60 s by default. + +## Decision + +**`papertrace mcp` serves nine tools over stdio, behind an optional `[mcp]` +extra. It runs the pipeline in-process on one background thread at a time, and +the thread has nobody to ask.** + +1. **In-process, not a subprocess.** `ask.py` remains the only file in `src/` + that starts a process, as `tests/test_ask.py` asserts. The cost is that the + server must manage process-global state itself: the pipeline's console is + swapped for a capture sink, and `ask.forget_models()` clears the model record + at the start of every audit. +2. **One audit at a time.** The console, the model record and `sys.stdin` are + process-global. A second `start_audit` is refused, not queued, and read tools + refuse a case whose audit is still running, because the files on disk are + then another run's. +3. **Nobody to ask.** The job runs with an empty `sys.stdin`, so the interactive + escalation of a disputed reference label is unreachable and the label is + withheld with `numbering_chosen_by: "default"`. ADR 0003's gate is a person + shown the disagreement; an MCP client can be a model answering for itself, + so elicitation is not offered as a substitute. +4. **A job, not a blocking call.** `start_audit` returns at once; + `audit_status` long-polls for at most 50 s, under the 60 s default. +5. **The same disclosures, serialised the viewer's way.** Nothing is + re-derived. The parity suite asserts every token reaches the MCP output. + +## Consequences + +- A host that is not Claude Code can run and read an audit, and see the + evidence crops as images. +- A disputed label cannot be settled from an MCP host. That is deliberate; + `papertrace refs` in a terminal settles it. +- A running audit cannot be cancelled from the host. Stopping the server starts + no further stage or model call — a `claude -p` call already in flight runs to + its end — and the `mcp_audit.json` it leaves saying `running` keeps the half- + written case from being read as a result until an audit there finishes. +- Any future process-global state in the pipeline must be reset per audit, or + it will leak from one audit into the next. `CLAUDE.md` says so. diff --git a/docs/superpowers/plans/2026-09-26-mcp-server.md b/docs/superpowers/plans/2026-09-26-mcp-server.md new file mode 100644 index 0000000..eb8b16a --- /dev/null +++ b/docs/superpowers/plans/2026-09-26-mcp-server.md @@ -0,0 +1,176 @@ +# MCP Server — Implementation Plan + +**Goal:** Make PaperTrace usable from any MCP host — start an audit, follow it, +and read every finding with its caveats and its evidence images — without +weakening a single rule in `CLAUDE.md`. + +**Architecture:** One new module, `src/papertrace/mcp_server.py`, built on the +official SDK's `MCPServer` (`mcp>=2.2,<3`). Read tools load the case folder's +JSON through the existing dataclasses and serialise the existing disclosures. +The one driving tool runs the existing pipeline in-process on a background +thread. `papertrace mcp` serves it over stdio. + +**Spec:** `docs/superpowers/specs/2026-09-26-mcp-server-design.md` + +**Decision record:** `docs/adr/0004-mcp-server.md` + +## Global Constraints + +- **Tests stay offline.** No network, no model calls. Tools are exercised through + the SDK's in-memory `Client`; the pipeline is faked at `cli._run_pipeline`. +- **No `conftest.py`.** Every test module prepends `src/` to `sys.path` and + imports with `# noqa: E402`. +- **`ask.py` stays the only file in `src/` that starts a process** — + `tests/test_ask.py::test_only_ask_py_shells_out_to_a_model` must stay green. +- **No new case-folder field** (gate 2 untouched). **No ingest, highlight or + report *logic* changes** (gate 5 untouched — say so in the handover rather + than skip it silently). **`refs.py`, `check.py`, `scout.py` untouched** + (gate 4 untouched). +- **Red before green, pasted.** Each task: write the test, run it, see it fail + for the stated reason, implement, run it, see it pass. +- `ruff check src tests scripts evals` clean; never `ruff format`. + +## File Structure + +| File | Responsibility | Task | +|---|---|---| +| `src/papertrace/cli.py` *(modified)* | `run` split into `_run_pipeline` (keyword-only, plain defaults) and its Typer adapter; `papertrace mcp` command | 1, 5 | +| `src/papertrace/ask.py` *(modified, +1 function)* | `forget_models()` — a long-lived process starts each run with no recorded model | 2 | +| `src/papertrace/mcp_server.py` *(new)* | `build_server()`, the nine tools, the job registry, `serve()` | 3, 4 | +| `pyproject.toml` *(modified)* | `mcp` extra; `mcp` in `[dev]` and `[full]` | 5 | +| `tests/test_pipeline_split.py` *(modified)* | `run` delegates every parameter to `_run_pipeline` | 1 | +| `tests/test_ask.py` *(modified)* | `forget_models` | 2 | +| `tests/test_mcp_server.py` *(new)* | every tool, the parity loop, the job, stdio | 3, 4, 5 | +| `tests/test_packaging.py` *(modified)* | the extra exists and CI installs it | 5 | +| `README.md`, `CHANGELOG.md`, `CLAUDE.md` | docs | 6 | + +## Task 1 — split `run` into `_run_pipeline` + +`run` is the one stage function still called only through its Typer command, +and the MCP job would be its second caller. Calling a Typer command directly +hands every omitted option an `OptionInfo` sentinel — the flaw `cli.py` records +shipping twice — so the split is made *before* the new caller exists. + +- [ ] Red: `test_run_pipeline_rejects_a_positional_call` and + `test_run_command_delegates_to_the_pipeline_function` (exact kwargs dict, + in the style of the `refs` test beside them). Expect `AttributeError: no + _run_pipeline`. +- [ ] Green: move `run`'s body into `_run_pipeline(*, ...)`; `run` passes every + parameter by keyword. No behaviour change: the existing suite is the proof. + +## Task 2 — `ask.forget_models()` + +- [ ] Red: seed `_MODELS` for both sites, call `forget_models()`, expect + `model_for(...) is None` for both — `AttributeError` first. +- [ ] Green: one function, clearing the dict in place (tests monkeypatch the + attribute, so rebinding it would be invisible to a caller holding it). + +## Task 3 — the read tools + +Fixture: a case folder written with `RunResults.to_json`, +`RefManifest.to_json`, `ScoutResults.to_json`, and evidence PNGs drawn with +pymupdf. Each test calls through `Client(build_server())`. + +- [ ] Red, then green, one behaviour at a time: + - `list_tools` advertises nine tools; read tools carry + `read_only_hint=True, open_world_hint=False`; `start_audit` carries + `destructive_hint=True, open_world_hint=True`. + - `audit_summary` returns the counts and every run-level disclosure; a case + with no `results.json` is a `ToolError` naming the missing file; a malformed + one is a `ToolError`, never empty counts. + - A limited run: the scope disclosure is first **and** `limited` is the last + key of the summary and of `list_claims`. + - Parity: for each fixture in `test_disclosure_parity.py`'s shape, every run + token is in `audit_summary`'s text and every claim token in `get_claim`'s. + - `list_claims` filters on the headline verdict; the filter's enum is + `models.VERDICTS`, not a copy. + - `get_evidence` returns the headline judgement's primary and continuation + images plus a caption carrying the anchor disclosure; a named source picks + that judgement; a `not_retrieved` claim is a `ToolError` saying nothing was + read; an image path escaping `out/` is refused. + - `list_references` has no `pdf_path` anywhere; `labels_uncomparable: null` + stays `null` and `[]` stays `[]`; the status filter's enum is + `models.REF_STATUSES`. + - `list_gaps` lists uncited assertions, unreached occurrences and unchecked + claims; a coverage audit that never ran is `null`. + - `get_scout` carries the scout's own caveat; no `scout.json` is a + `ToolError`. + +## Task 4 — `start_audit` and `audit_status` + +- [ ] Red, then green: + - Preflight refusals are immediate `ToolError`s: missing manuscript, `claude` + not on `PATH`, no contact email, a case folder holding another paper. + - The job passes `_run_pipeline` every parameter explicitly. + - Inside the job: `cli._interactive()` is False, `input()` raises `EOFError`, + `ask.model_for(SITE_CHECK)` is None even when a previous audit left one, + and `cli.console` output lands in the job log, not on stdout. + - `audit_status` goes `running` → `finished`; a `typer.Exit(2)` from the + pipeline is `failed` with the console's reason; any other exception is + `failed` with its type and message. + - A second `start_audit` while one runs is refused, naming the running case. + - Read tools refuse the case whose audit is running. + - `wait_seconds` returns as soon as the job ends, and never waits past 50. + - A finished status carries no counts. + +## Task 5 — `papertrace mcp`, packaging, stdio + +- [ ] Red, then green: + - `[project.optional-dependencies]` has `mcp`, and `[dev]` and `[full]` carry + the same requirement (packaging tests). + - `papertrace mcp` with stdin closed exits 0 and writes **zero bytes** to + stdout (subprocess). + - A real stdio round trip — `Client(StdioServerParameters(...))` — lists the + nine tools and reads a fixture case. + - Without the `mcp` package, `papertrace mcp` exits 2 with the install line on + **stderr** and nothing on stdout. + +## Task 6 — docs + +- [ ] README: a "Use it from an MCP host" section (Claude Desktop, Claude Code, + Cursor/VS Code), the install matrix row, the does/does-not list, the + roadmap tick. Every statement checked against the code. +- [ ] CHANGELOG: an Unreleased entry. +- [ ] CLAUDE.md: the architecture bullet for `mcp_server.py`, and the + long-lived-process rule (`forget_models`, one job at a time). + +## Task 7 — the review round + +A code review of the branch (the `code-review` skill, high effort) returned ten +findings; nine held up against the code, each fixed test-first. The tenth — +that `_run_pipeline`'s OptionInfo guards are dead — did not: tests call +`cli.run()` directly and omit options, which hands the adapter's sentinels +straight through. + +- [x] A failed or interrupted audit's mixed case is refused, via the + `mcp_audit.json` record; `audit_status` reads an earlier server's record. +- [x] Run-level disclosures in `get_claim`, `list_claims` and the evidence + caption, beside the claim's own. +- [x] No fallback to the server's working directory for an unwritable paper + folder; the preflight's lines open the job's log. +- [x] A `scout.json` the latest audit did not write is refused; the caveat + carries the scan's error. +- [x] A 1.x SDK gets the install line; the command is never wrapped in two. +- [x] `schemas/mcp_tools.schema.json` and `schemas/mcp_audit.schema.json`, + with validation tests. +- [x] `disclosures.coverage_audited` and `cli.SCOUT_CAVEAT`: one rule and one + sentence each, not two copies. + +## Task 8 — the second review pass + +The review round's own diff, reviewed again: ten findings, all of which held +up, each fixed test-first. + +- [x] A `running` record is measured by a heartbeat: fresh is another + process's live audit (refused; never started over), stale is + `interrupted`. `audit_status` waits on another process's record too. +- [x] The disk record decides; later `results.json` / `scout.json` supersede + a failure or a run without the scout. +- [x] A named source's crop carries only its own anchor state. +- [x] `unreached_labels` is null unless the label audit read `clean.md`. +- [x] `manuscript` is the absolute path, from this server and from a record. +- [x] Recorded states never send the host to a log that was not kept. +- [x] The record is validated before anything is decided on it. +- [x] A 2.x SDK's import failure keeps its traceback. +- [x] The schema describes record-derived states, validated for each; the + prose timings are held to the constants. diff --git a/docs/superpowers/specs/2026-09-26-mcp-server-design.md b/docs/superpowers/specs/2026-09-26-mcp-server-design.md new file mode 100644 index 0000000..15a539d --- /dev/null +++ b/docs/superpowers/specs/2026-09-26-mcp-server-design.md @@ -0,0 +1,197 @@ +# An MCP server — PaperTrace as a tool for any MCP client + +Design, 2026-09-26. Status: implemented alongside this document on +`claude/gifted-ritchie-ga60nx`; open to review. + +## Why + +The roadmap has carried one line since 0.5.0: *MCP server — drive PaperTrace as +a tool from any MCP-capable client.* Today there are three ways in: the CLI, the +wizard, and the two Claude Code skills, which drive the CLI through a shell. A +host that is not Claude Code — Claude Desktop, Cursor, VS Code, anything that +speaks the Model Context Protocol — has no way to start an audit or read one +without a shell. MCP is the standard way to hand a local tool to those hosts. + +The risk is specific to this codebase. Every rule in `CLAUDE.md` was written for +readers of a **file** — a report, a manifest, a `results.json` — and for a +process that **lives for one command**. An MCP server is a new reader (a model) +and a new process shape (one that outlives many audits). Each rule has to be +re-established for both, or the server becomes the path by which a verdict +reaches a reader without its caveats. + +## What it is + +`papertrace mcp` serves over **stdio**, behind an optional extra +(`pip install -e ".[mcp]"`, `mcp>=2.2,<3` — the SDK's own versioning policy +promises breaking changes only in a major). Nine tools, in two groups: + +| Tool | Group | What it returns | +|---|---|---| +| `audit_summary` | read | counts, references, checker, coverage headline, **every run-level disclosure**, the scope statement last | +| `list_claims` | read | one row per claim (optionally one verdict), each with its claim-level caveats | +| `get_claim` | read | one claim in full: quote, per-source judgements, rationale, page, block, anchor phrases, every disclosure | +| `get_evidence` | read | the red-box crops for one claim (or one of its sources) **as images**, captioned with the anchor state | +| `list_references` | read | the retrieval manifest without local paths, with the numbering state as recorded | +| `list_gaps` | read | uncited assertions, citation places no claim reached, claims never checked | +| `get_scout` | read | the three scout registers, with the scout's own caveat | +| `start_audit` | drive | starts the full pipeline (`papertrace run`) as a background job and returns at once | +| `audit_status` | drive | the job's state and log tail; long-polls up to 50 s | + +Read tools are deterministic, offline and free: they read the case folder the +pipeline wrote. `start_audit` is the only tool that spends — model calls through +`claude -p`, and the open-access services `refs` and `scout` query. + +## The rules it inherits, and how each is kept + +1. **An unread source never receives a verdict.** The server computes no + verdict. It reads what the pipeline wrote, or runs the pipeline itself. + Nothing here derives, re-ranks or summarises a verdict. +2. **Disclosures travel with every count and every verdict.** The read tools + call `run_disclosures`, `claim_disclosures` and `judgement_disclosures` — the + same producers the four report formats use — and serialise them with the + viewer's own `report._disclosure_dict`. The MCP output is a fifth reader, and + the parity test is extended to it: every token reaches it. A host can read + one claim without ever reading the summary, so `get_claim`, `list_claims` + and the evidence caption carry the run-level disclosures too. +3. **A limited run never reads as a smaller paper.** The scope disclosure is + first among the disclosures, as in every report, and every result that states + counts or lists claims ends with a `limited` field carrying the same sentence + — the JSON a model reads has an end, and the scope is said there too. +4. **Absent is not zero.** A missing `results.json` is a `ToolError` naming what + is missing, never a result with empty counts. Three-state manifest fields + (`labels_uncomparable`, `numbering_corroborated`) pass through unchanged, so + `null` stays `null`. A coverage audit that never ran is `null`, not `""`. +5. **Failure is said, never guessed.** In SDK v2 only a `ToolError`'s message + reaches the model; any other exception becomes a message-less + `Error executing tool …`. So every refusal is a `ToolError` carrying the + reason — the same sentence the CLI would have printed — and a job that fails + records its reason in its state instead of surfacing as an anonymous error. +6. **The interactive gate on disputed labels cannot be reached.** The job runs + with nobody to ask: `sys.stdin` is an empty stream for its duration, so + `cli._interactive()` is False and any prompt reads end-of-file rather than + waiting. A disputed label is therefore withheld with + `numbering_chosen_by: "default"`, exactly as the CLI does without a + terminal. An MCP client cannot settle a dispute; a person at a terminal can, + with `papertrace refs`. +7. **`ask.py` stays the only file in `src/` that starts a process.** The server + runs the pipeline **in-process, on a thread** — never a `papertrace run` + child. `tests/test_ask.py` would go red otherwise, and the rule protects the + seam, not a module. +8. **The model a run pinned is the model it reports.** `ask._MODELS` was + correct because one process ran one command. A server runs audit after + audit, and a second audit whose judging made no calls would print the first + audit's judge as its `Checker:` — the bug `ask.py` was restructured to end. + `ask.forget_models()` clears the record at the start of every audit. +9. **Nothing reaches stdout but the protocol.** Under stdio, stdout *is* the + wire. The pipeline prints through `cli.console`; for a job that console is + swapped for a sink whose lines become the job's log. `papertrace mcp` prints + nothing before serving, and the SDK's stdio transport diverts fd 1 to stderr + while it serves, as a second line of defence. +10. **No path is served that the tool did not write.** References are listed + without `pdf_path`, as in the viewer. An evidence image is read only when it + resolves inside `/out/` and is a `.png` — a `results.json` edited to + point at `../../somewhere` is refused, not followed. +11. **A case is not a result until its audit has ended.** A rerun can rewrite + the manifest and die before `check`, leaving one run's references beside + another's verdicts. The job writes `/mcp_audit.json` — `running` + before the pipeline starts, the outcome when it ends — and read tools + refuse a case whose last MCP audit failed or never finished. The job + refreshes the record every 15 s, so `running` is measured rather than + assumed: fresh, the audit is live in some server process — perhaps another + host's own `papertrace mcp` — and the folder is neither read nor audited + again; unrefreshed for 60 s, the server stopped mid-run and the audit is + `interrupted`. The record on disk decides, not a server's memory: a failed + audit is superseded by a `results.json` written after it, and one run + without the scout by a `scout.json` written after it, so a case completed + since by other means reads again. + +## Long runs: a job, not a blocking call + +An audit takes minutes — the demo took 95 s with the layout models cached, and +the first docling run downloads about 500 MB. A client built on the TypeScript +SDK times a request out at **60 s** (`DEFAULT_REQUEST_TIMEOUT_MSEC = 60000` in +`@modelcontextprotocol/sdk` 1.30.1; resetting on progress is opt-in). A blocking +`run_audit` would fail there while the audit carried on unseen, still spending. + +So `start_audit` validates what it can synchronously — the manuscript exists, +`claude` is on this server's `PATH`, a contact email resolves, the case folder +is not another paper's — and then returns at once. `audit_status` reports +`running`, `finished` or `failed` with the log tail, and waits up to +`wait_seconds` (at most 50, under that 60 s) for a running job to end. + +**One audit at a time per server process.** The pipeline's console, `ask`'s +per-site model record and `sys.stdin` are process-global; two audits sharing +them would interleave logs and misattribute models. A second `start_audit` is +refused, naming the audit in progress. Across processes the record is the +guard: a fresh `running` record is another server's live audit, and +`start_audit` refuses that folder too. + +**Read tools refuse a case whose audit is running, failed or never finished.** +The files on disk then belong to an earlier run, are half-written, or mix the +two, and presenting them as a result would be the silent failure this codebase +exists to refuse. + +`audit_status` never returns counts. A count without its disclosures is exactly +what rule 2 forbids, so a finished job points at `audit_summary`. + +## What is rejected, and why + +- **A subprocess per audit.** The cleanest isolation — its own stdout, its own + globals, killable — and a second process-spawning file in `src/`, which + `tests/test_ask.py` exists to refuse. +- **A blocking `run_audit`.** Fails on 60 s clients and keeps spending after the + client has given up. +- **MCP elicitation to settle a disputed label.** The gate in + `_escalate_disputed` is a *person* shown the disagreement. An MCP client may be + a model answering on its own behalf, and `numbering_chosen_by: "user"` would + then record a choice nobody made. +- **One tool per stage** (`ingest`, `extract`, `refs`, `check`, `highlight`, + `report`). The `/review` skill already drives stages one by one in Claude + Code; for other hosts one audit plus inspection covers the use, and every + extra spending tool is another cost surface. Cherry-picking claims — `check` + on an array of ids, which `select_claims` already takes — is the natural next + tool, and is left for its own change. +- **MCP resources for `report.md` and `results.json`.** Tools are supported by + every host; resources are surfaced unevenly. Deferred. +- **MCP-level logging (`ctx.info`).** Deprecated by the 2026-07-28 protocol + revision (SEP-2577). The job log is returned by `audit_status` instead. +- **Streamable HTTP.** A case folder stays on the machine that made it, by + design. stdio only. +- **An output schema only in `tools/list`.** The SDK publishes one generated + from each tool's `TypedDict`, but `schemas/` is where this repository keeps + its contracts (gate 2), so `schemas/mcp_tools.schema.json` holds them too — + every tool output is validated against it, and each definition's keys are + held equal to its `TypedDict`'s, so the two cannot drift apart silently. + `schemas/mcp_audit.schema.json` covers the one file the server writes. + +## Deliberate omissions + +- `--png` is not exposed: it needs playwright and a browser, and the images the + server serves are the evidence crops, which need neither. +- A running audit cannot be cancelled. A thread cannot be stopped safely + mid-write; the CLI's Ctrl-C carries the same exposure. If the host stops the + server, no further stage and no further model call starts — but a `claude -p` + call already in flight runs to its end, bounded by `ask.py`'s timeout — and + the record left saying `running` marks the case incomplete. +- A job's log is not persisted; its outcome is, in `mcp_audit.json`, and an + earlier server's audit reports `log: null` rather than an empty log. +- Liveness across processes is a heartbeat window, not a lock. A server whose + refreshes stop for 60 s — a machine that slept — reads as stopped: its case + is refused rather than read, the safe side, but a second audit started in + that window is not refused. The CLI takes no part in the record at all. +- A CLI run that fails midway writes no record, so the server reads that case + as the reports would. + +## Testing + +- Every tool through the SDK's in-memory `Client(build_server())`: offline, no + model, fixtures built from the dataclasses, PNGs drawn with pymupdf. +- The job is exercised with `cli._run_pipeline` faked; the fake asserts what + the job guarantees from inside it (nobody to ask, no stale model, the console + captured). +- One real stdio round trip through a subprocess, and one test that + `papertrace mcp` writes nothing to stdout when its stdin closes. +- Parity: every run-level and claim-level disclosure token appears in the MCP + output for the same fixtures `test_disclosure_parity.py` uses. +- Packaging: `mcp` is in `[mcp]`, in `[full]`, and in `[dev]` — so CI runs these + tests rather than skipping them. diff --git a/pyproject.toml b/pyproject.toml index a987dde..0b2983d 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -39,14 +39,19 @@ dependencies = [ [project.optional-dependencies] png = ["playwright>=1.45"] +# `papertrace mcp`, for MCP hosts. The server is written against the SDK's 2.x +# API, and the SDK holds breaking changes for a major, hence the bound +mcp = ["mcp>=2.2,<3"] # kept as a no-op alias so `pip install papertrace[docling]` and the `[full]` # extra named all over the 0.4.x docs keep resolving instead of erroring docling = [] -full = ["playwright>=1.45"] +full = ["playwright>=1.45", "mcp>=2.2,<3"] dev = ["pytest>=8.0", "ruff>=0.5", "jsonschema>=4.0", # 3.10 has no tomllib, and without a parser the whole packaging # suite skipped on the oldest version requires-python allows - 'tomli>=2.0; python_version < "3.11"'] + 'tomli>=2.0; python_version < "3.11"', + # so CI runs tests/test_mcp_server.py instead of skipping it + "mcp>=2.2,<3"] [project.scripts] papertrace = "papertrace.cli:main" diff --git a/schemas/mcp_audit.schema.json b/schemas/mcp_audit.schema.json new file mode 100644 index 0000000..019a37c --- /dev/null +++ b/schemas/mcp_audit.schema.json @@ -0,0 +1,55 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "title": "mcp_audit", + "description": "/mcp_audit.json — what an MCP server (`papertrace mcp`) records about the last audit it started in this case folder, for itself and any other server process. The audit job writes it with state `running` before the pipeline touches anything, refreshes its modification time every 15 s while it runs, and rewrites it with the outcome when it ends. A `running` record is therefore measured: refreshed within 60 s, an audit is live in some server process, and no second one is started in the folder; staler, the server writing it stopped mid-run, and the folder is not read as a result. A `failed` record keeps the folder from being read until a results.json written after it shows a later run completed the judging. Absent means no MCP server has audited this case; the CLI never writes it.", + "type": "object", + "required": [ + "schema", + "state", + "manuscript", + "started", + "ended", + "error", + "scout" + ], + "properties": { + "schema": { + "type": "string", + "enum": [ + "mcp_audit/1" + ] + }, + "state": { + "type": "string", + "enum": [ + "running", + "finished", + "failed" + ], + "description": "`running` while the job runs — and after, when the server stopped first, which the record's unrefreshed modification time shows. `failed`: the pipeline stopped, and what is on disk may mix that run with an earlier one." + }, + "manuscript": { + "type": "string", + "description": "The audited PDF's absolute path — the same value audit_status gives." + }, + "started": { + "type": "string", + "description": "When the job started: UTC, ISO 8601, to the second." + }, + "ended": { + "type": [ + "string", + "null" + ], + "description": "When the job ended; null while it runs, and on an audit whose server stopped first." + }, + "error": { + "type": "string", + "description": "Why the audit failed — the last lines the pipeline printed, or the exception's type and message. Empty unless `state` is `failed`." + }, + "scout": { + "type": "boolean", + "description": "Whether this audit scanned the literature. false: any out/scout.json in the folder belongs to an earlier run, and is not served as this audit's." + } + } +} diff --git a/schemas/mcp_tools.schema.json b/schemas/mcp_tools.schema.json new file mode 100644 index 0000000..6b3ac33 --- /dev/null +++ b/schemas/mcp_tools.schema.json @@ -0,0 +1,1018 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "title": "mcp_tools", + "description": "What `papertrace mcp` returns: the structured content of each tool's result, one definition per tool under $defs (named after the tool), with the shapes they share. The SDK publishes each as the tool's outputSchema in tools/list; this file is the same contract where this repository keeps its contracts, and tests/test_mcp_server.py validates every tool's output against it. get_evidence returns content blocks — a caption and images — rather than structured content, and has no definition here.", + "$defs": { + "audit_summary": { + "description": "The result of one audit, from out/results.json and refs_manifest.json. Never returned for a case without results.json — the call fails with the reason instead — so no count here stands for 'never measured'.", + "properties": { + "case": { + "type": "string" + }, + "manuscript": { + "type": "string" + }, + "checker": { + "type": "string" + }, + "date": { + "type": "string" + }, + "converter": { + "type": "string" + }, + "references": { + "$ref": "#/$defs/RefCounts" + }, + "claims": { + "type": "integer" + }, + "counts": { + "additionalProperties": { + "type": "integer" + }, + "type": "object" + }, + "uncited_assertions": { + "type": "integer" + }, + "coverage": { + "type": [ + "string", + "null" + ], + "description": "The coverage headline; null when the coverage audit could not run — never an empty string or a zero ratio." + }, + "disclosures": { + "items": { + "$ref": "#/$defs/DisclosureOut" + }, + "type": "array", + "description": "Every run-level disclosure, in the order the reports print them — the scope first when the audit was limited." + }, + "reports": { + "items": { + "type": "string" + }, + "type": "array", + "description": "Absolute paths of the report files present in out/." + }, + "limited": { + "type": [ + "string", + "null" + ], + "description": "The scope sentence when the audit was limited on request, stated last as every report states it; null when the whole paper was audited." + } + }, + "required": [ + "case", + "manuscript", + "checker", + "date", + "converter", + "references", + "claims", + "counts", + "uncited_assertions", + "coverage", + "disclosures", + "reports", + "limited" + ], + "type": "object" + }, + "list_claims": { + "description": "One row per checked claim, each with its claim-level caveats, and the run-level disclosures once beside them.", + "properties": { + "case": { + "type": "string" + }, + "verdict_filter": { + "type": [ + "string", + "null" + ] + }, + "claims_total": { + "type": "integer" + }, + "claims": { + "items": { + "$ref": "#/$defs/ClaimRow" + }, + "type": "array" + }, + "run_disclosures": { + "items": { + "$ref": "#/$defs/DisclosureOut" + }, + "type": "array", + "description": "The caveats the reports print above every verdict, so a claim read without audit_summary still has them." + }, + "limited": { + "type": [ + "string", + "null" + ], + "description": "As audit_summary's: the scope sentence, or null for a whole-paper audit." + } + }, + "required": [ + "case", + "verdict_filter", + "claims_total", + "claims", + "run_disclosures", + "limited" + ], + "type": "object" + }, + "get_claim": { + "description": "One claim in full: the paper's sentence, each cited source's judgement, and every disclosure the claim and the run owe its reader.", + "properties": { + "case": { + "type": "string" + }, + "id": { + "type": "integer" + }, + "verdict": { + "type": "string" + }, + "label": { + "type": "string" + }, + "qualifier": { + "type": "string" + }, + "claim": { + "type": "string" + }, + "quote": { + "type": "string" + }, + "location": { + "type": "string" + }, + "refs": { + "items": { + "type": "string" + }, + "type": "array" + }, + "own_supplement": { + "type": "boolean" + }, + "rationale": { + "type": "string" + }, + "source": { + "type": [ + "string", + "null" + ] + }, + "page": { + "type": [ + "integer", + "null" + ] + }, + "block": { + "type": [ + "string", + "null" + ] + }, + "anchor_phrases": { + "items": { + "type": "string" + }, + "type": "array" + }, + "anchor_located": { + "type": [ + "boolean", + "null" + ] + }, + "evidence_images": { + "items": { + "type": "string" + }, + "type": "array", + "description": "Crop paths relative to /out/; get_evidence returns the images themselves." + }, + "judgements": { + "items": { + "$ref": "#/$defs/JudgementOut" + }, + "type": "array" + }, + "unjudged_refs": { + "items": { + "type": "string" + }, + "type": "array", + "description": "Co-cited labels whose source could not be obtained — nobody read them." + }, + "withheld_refs": { + "items": { + "type": "string" + }, + "type": "array", + "description": "Co-cited labels whose source was obtained and withheld, because the readings of the reference list disagree about which paper the label names." + }, + "disclosures": { + "items": { + "$ref": "#/$defs/DisclosureOut" + }, + "type": "array", + "description": "The claim-level disclosures, as every report format prints them beside the claim." + }, + "run_disclosures": { + "items": { + "$ref": "#/$defs/DisclosureOut" + }, + "type": "array", + "description": "The run-level disclosures: the caveats the reports print above every verdict." + } + }, + "required": [ + "case", + "id", + "verdict", + "label", + "qualifier", + "claim", + "quote", + "location", + "refs", + "own_supplement", + "rationale", + "source", + "page", + "block", + "anchor_phrases", + "anchor_located", + "evidence_images", + "judgements", + "unjudged_refs", + "withheld_refs", + "disclosures", + "run_disclosures" + ], + "type": "object" + }, + "list_references": { + "description": "The retrieval manifest and the state of the reference numbering. Local file paths are never included.", + "properties": { + "case": { + "type": "string" + }, + "manuscript": { + "type": "string" + }, + "total": { + "type": "integer" + }, + "available": { + "type": "integer" + }, + "by_status": { + "additionalProperties": { + "type": "integer" + }, + "type": "object" + }, + "numbering": { + "$ref": "#/$defs/NumberingOut" + }, + "limits": { + "additionalProperties": true, + "type": "object" + }, + "status_filter": { + "type": [ + "string", + "null" + ] + }, + "references": { + "items": { + "$ref": "#/$defs/ReferenceRow" + }, + "type": "array" + } + }, + "required": [ + "case", + "manuscript", + "total", + "available", + "by_status", + "numbering", + "limits", + "status_filter", + "references" + ], + "type": "object" + }, + "list_gaps": { + "description": "What the audit could not vouch for: uncited assertions, citation places no claim reached, claims never checked.", + "properties": { + "case": { + "type": "string" + }, + "uncited_assertions": { + "items": { + "$ref": "#/$defs/UncitedOut" + }, + "type": "array" + }, + "coverage": { + "type": [ + "string", + "null" + ], + "description": "The coverage headline; null when the coverage audit could not run." + }, + "unreached_labels": { + "items": { + "type": "string" + }, + "type": [ + "array", + "null" + ], + "description": "Labels no extracted claim reached; null when the coverage audit did not run — not 'none unreached'." + }, + "unreached_citations": { + "items": { + "$ref": "#/$defs/UnreachedOut" + }, + "type": [ + "array", + "null" + ], + "description": "Citation places no claim reached (uncovered) or reached ambiguously (uncertain); null when the audit did not run or recorded no occurrences." + }, + "unchecked_claims": { + "items": { + "$ref": "#/$defs/UncheckedOut" + }, + "type": "array" + }, + "disclosures": { + "items": { + "$ref": "#/$defs/DisclosureOut" + }, + "type": "array" + }, + "limited": { + "type": [ + "string", + "null" + ], + "description": "As audit_summary's: the scope sentence, or null for a whole-paper audit." + } + }, + "required": [ + "case", + "uncited_assertions", + "coverage", + "unreached_labels", + "unreached_citations", + "unchecked_claims", + "disclosures", + "limited" + ], + "type": "object" + }, + "get_scout": { + "description": "out/scout.json, with the scout's own caveat. Search-based: absence from these registers proves nothing.", + "properties": { + "case": { + "type": "string" + }, + "paper": { + "$ref": "#/$defs/ScoutPaperOut" + }, + "query": { + "type": "string" + }, + "date": { + "type": "string" + }, + "counts": { + "additionalProperties": { + "type": "integer" + }, + "type": "object" + }, + "newer": { + "items": { + "$ref": "#/$defs/ScoutHitOut" + }, + "type": "array" + }, + "overlooked": { + "items": { + "$ref": "#/$defs/ScoutHitOut" + }, + "type": "array" + }, + "same_year": { + "items": { + "$ref": "#/$defs/ScoutHitOut" + }, + "type": "array" + }, + "error": { + "type": "string" + }, + "caveat": { + "type": "string", + "description": "The scout's caveat, led by an identity warning and the scan's own error when either applies." + } + }, + "required": [ + "case", + "paper", + "query", + "date", + "counts", + "newer", + "overlooked", + "same_year", + "error", + "caveat" + ], + "type": "object" + }, + "audit_status": { + "description": "Where an audit an MCP server started stands. Never carries counts — they travel only with their disclosures, in audit_summary.", + "properties": { + "case": { + "type": "string" + }, + "state": { + "enum": [ + "running", + "finished", + "failed", + "interrupted", + "not_started" + ], + "type": "string", + "description": "The last audit an MCP server started in this case. running, finished, failed: this server's own job, with its log — or another server process's record of one, with log null. A record says running only while it is being refreshed (every 15 s); one not refreshed for 60 s is interrupted: the server writing it stopped mid-run. not_started: no MCP server has audited this case." + }, + "manuscript": { + "type": [ + "string", + "null" + ], + "description": "The audited PDF's absolute path; null when not_started." + }, + "started": { + "type": [ + "string", + "null" + ] + }, + "ended": { + "type": [ + "string", + "null" + ] + }, + "error": { + "type": "string" + }, + "log": { + "items": { + "type": "string" + }, + "type": [ + "array", + "null" + ], + "description": "This server's job's latest console lines; null when the audit is another server process's or was never started — an empty log would read as an audit that printed nothing." + }, + "log_lines_total": { + "type": [ + "integer", + "null" + ], + "description": "How many console lines this server's job printed; null when there is no log here." + }, + "next": { + "type": "string" + } + }, + "required": [ + "case", + "state", + "manuscript", + "started", + "ended", + "error", + "log", + "log_lines_total", + "next" + ], + "type": "object" + }, + "ClaimRow": { + "properties": { + "id": { + "type": "integer" + }, + "verdict": { + "type": "string" + }, + "label": { + "type": "string" + }, + "qualifier": { + "type": "string" + }, + "refs": { + "items": { + "type": "string" + }, + "type": "array" + }, + "location": { + "type": "string" + }, + "claim": { + "type": "string" + }, + "quote": { + "type": "string" + }, + "caveats": { + "items": { + "type": "string" + }, + "type": "array" + } + }, + "required": [ + "id", + "verdict", + "label", + "qualifier", + "refs", + "location", + "claim", + "quote", + "caveats" + ], + "type": "object" + }, + "DisclosureOut": { + "properties": { + "key": { + "type": "string" + }, + "level": { + "type": "string" + }, + "token": { + "type": "string", + "description": "The short literal every report format carries verbatim for this disclosure." + }, + "text": { + "type": "string" + }, + "short": { + "type": "string" + }, + "rows": { + "items": { + "type": "string" + }, + "type": "array" + } + }, + "required": [ + "key", + "level", + "token", + "text", + "short", + "rows" + ], + "type": "object" + }, + "JudgementOut": { + "properties": { + "source": { + "type": "string" + }, + "origin": { + "type": "string" + }, + "kind": { + "type": "string" + }, + "verdict": { + "type": "string" + }, + "label": { + "type": "string" + }, + "rationale": { + "type": "string" + }, + "page": { + "type": [ + "integer", + "null" + ] + }, + "block": { + "type": [ + "string", + "null" + ] + }, + "anchor_phrases": { + "items": { + "type": "string" + }, + "type": "array" + }, + "anchor_located": { + "type": [ + "boolean", + "null" + ] + }, + "evidence_images": { + "items": { + "type": "string" + }, + "type": "array" + }, + "disclosures": { + "items": { + "$ref": "#/$defs/DisclosureOut" + }, + "type": "array" + } + }, + "required": [ + "source", + "origin", + "kind", + "verdict", + "label", + "rationale", + "page", + "block", + "anchor_phrases", + "anchor_located", + "evidence_images", + "disclosures" + ], + "type": "object" + }, + "NumberingOut": { + "properties": { + "source": { + "type": "string" + }, + "verified": { + "type": "boolean", + "description": "numbering_verified: the chosen list accounts for exactly the cited labels — extent, not content." + }, + "note": { + "type": "string" + }, + "unverified_from": { + "type": [ + "integer", + "null" + ] + }, + "contested": { + "type": "boolean" + }, + "corroborated": { + "type": [ + "boolean", + "null" + ], + "description": "null: never computed. false: computed, and some cited label was not agreed by two readings." + }, + "corroborating_readings": { + "items": { + "type": "string" + }, + "type": "array" + }, + "labels_disputed": { + "items": { + "type": "string" + }, + "type": "array" + }, + "labels_uncomparable": { + "items": { + "type": "string" + }, + "type": [ + "array", + "null" + ], + "description": "The subset of labels_disputed where nothing comparable was said. null: never computed. []: computed, and every dispute was a contradiction." + }, + "labels_resolved": { + "items": { + "type": "string" + }, + "type": "array" + }, + "choice": { + "type": "string" + }, + "chosen_by": { + "type": "string", + "description": "'' when no escalation ran; 'default' when nobody was asked — always so for an audit started over MCP; 'user' for a person at a terminal." + } + }, + "required": [ + "source", + "verified", + "note", + "unverified_from", + "contested", + "corroborated", + "corroborating_readings", + "labels_disputed", + "labels_uncomparable", + "labels_resolved", + "choice", + "chosen_by" + ], + "type": "object" + }, + "RefCounts": { + "properties": { + "total": { + "type": "integer" + }, + "available": { + "type": "integer" + } + }, + "required": [ + "total", + "available" + ], + "type": "object" + }, + "ReferenceRow": { + "properties": { + "label": { + "type": "string" + }, + "status": { + "type": "string" + }, + "reason": { + "type": "string" + }, + "resolver": { + "type": [ + "string", + "null" + ] + }, + "doi": { + "type": [ + "string", + "null" + ] + }, + "year": { + "type": [ + "string", + "null" + ] + }, + "slug": { + "type": [ + "string", + "null" + ] + }, + "title_check": { + "type": [ + "string", + "null" + ] + }, + "skipped_by": { + "type": [ + "string", + "null" + ], + "description": "Why a skipped reference was skipped: 'claims' (no selected claim cites it) or 'sources' (the cap was reached); null for every other status." + }, + "boundary_ambiguous": { + "type": "boolean" + }, + "seen_in": { + "items": { + "type": "string" + }, + "type": "array" + }, + "supplements": { + "items": { + "$ref": "#/$defs/SupplementOut" + }, + "type": "array" + }, + "raw": { + "type": "string" + } + }, + "required": [ + "label", + "status", + "reason", + "resolver", + "doi", + "title", + "year", + "slug", + "title_check", + "skipped_by", + "boundary_ambiguous", + "seen_in", + "supplements", + "raw" + ], + "type": "object" + }, + "ScoutHitOut": { + "properties": { + "year": { + "type": [ + "integer", + "null" + ] + }, + "doi": { + "type": "string" + }, + "via": { + "type": "string" + }, + "journal": { + "type": "string" + }, + "authors": { + "type": "string" + } + }, + "required": [ + "title", + "year", + "doi", + "via", + "journal", + "authors" + ], + "type": "object" + }, + "ScoutPaperOut": { + "properties": { + "doi": { + "type": "string" + }, + "year": { + "type": [ + "integer", + "null" + ] + }, + "resolved_via": { + "type": "string" + }, + "identity": { + "type": "string" + } + }, + "required": [ + "title", + "doi", + "year", + "resolved_via", + "identity" + ], + "type": "object" + }, + "SupplementOut": { + "properties": { + "slug": { + "type": "string" + }, + "verified": { + "type": "boolean" + } + }, + "required": [ + "slug", + "verified" + ], + "type": "object" + }, + "UncheckedOut": { + "properties": { + "id": { + "type": "integer" + }, + "verdict": { + "type": "string" + }, + "label": { + "type": "string" + }, + "refs": { + "items": { + "type": "string" + }, + "type": "array" + }, + "location": { + "type": "string" + }, + "note": { + "type": "string" + } + }, + "required": [ + "id", + "verdict", + "label", + "refs", + "location", + "note" + ], + "type": "object" + }, + "UncitedOut": { + "properties": { + "id": { + "type": "integer" + }, + "claim": { + "type": "string" + }, + "quote": { + "type": "string" + }, + "location": { + "type": "string" + } + }, + "required": [ + "id", + "claim", + "quote", + "location" + ], + "type": "object" + }, + "UnreachedOut": { + "properties": { + "label": { + "type": "string" + }, + "status": { + "type": "string" + }, + "section": { + "type": "string" + }, + "sentence": { + "type": "string" + } + }, + "required": [ + "label", + "status", + "section", + "sentence" + ], + "type": "object" + } + } +} diff --git a/src/papertrace/ask.py b/src/papertrace/ask.py index 3ddc588..5b7f40a 100644 --- a/src/papertrace/ask.py +++ b/src/papertrace/ask.py @@ -71,6 +71,17 @@ def model_for(site: str) -> str | None: return _MODELS.get(site) +def forget_models() -> None: + """Start a run with no model recorded at any site. + + The CLI never needed this: one process ran one command, so the record began + empty. The MCP server runs audit after audit in one process, and a second + audit whose judging made no calls would otherwise report the first audit's + judge as its own — the misattribution the per-site record exists to end. + """ + _MODELS.clear() + + def claude_available() -> bool: return shutil.which("claude") is not None @@ -159,5 +170,6 @@ def _parse_json_object(text: str) -> dict: "SITE_REFS", "claude_available", "for_site", + "forget_models", "model_for", ] diff --git a/src/papertrace/cli.py b/src/papertrace/cli.py index cd93243..d8ec0e0 100644 --- a/src/papertrace/cli.py +++ b/src/papertrace/cli.py @@ -450,6 +450,42 @@ def init( console.print(f" [dim]hand this folder to every step: [cyan]-c {case}[/cyan][/dim]") +@app.command("mcp", rich_help_panel="Utilities") +def mcp_command() -> None: + """Serve PaperTrace to an MCP host over stdio — Claude Desktop, Cursor, VS Code, Claude Code.""" + # Nothing may reach stdout from here: under stdio it is the protocol's wire, + # and a banner or a warning printed before serving begins corrupts it. + try: + from .mcp_server import serve + except ImportError as e: + # Only the SDK earns the install line — absent (ModuleNotFoundError) or + # a 1.x that other tools installed, which has no `MCPServer` and fails + # with a plain ImportError. Any other import is a real fault and keeps + # its own traceback. + if (e.name or "").split(".")[0] != "mcp": + raise + from importlib.metadata import PackageNotFoundError, version + + try: + installed = version("mcp") + except PackageNotFoundError: + installed = None + # a 2.x SDK that still fails to import is a real fault: the install line + # would contradict itself and hide the traceback that names what broke + if installed is not None and installed.split(".")[0].isdigit() \ + and int(installed.split(".")[0]) >= 2: + raise + found = f" (found mcp {installed})" if installed else "" + # rich eats [mcp] as a style tag; escaping it is what prints the extra. + # soft_wrap: wrapped at the terminal's width, the command split in two + Console(stderr=True, soft_wrap=True).print( + f"[red]the MCP server needs version 2 of the MCP SDK{found}[/red] — " + r"pip install 'papertrace\[mcp]'" + ) + raise typer.Exit(2) from None + serve() + + def _ingest_pipeline( *, pdf: Path, @@ -1769,6 +1805,13 @@ def refs( claims=_first_n(max_claims), max_sources=_int(max_sources)) +# What the scout's candidates are, said once for the console and the MCP server. +SCOUT_CAVEAT = ( + "search-based — absence from these lists proves nothing; presence is a candidate " + "for your judgement, not an accusation." +) + + @app.command(rich_help_panel="Pipeline stages — `run` calls these in order") def scout( case: Path = typer.Option( @@ -1828,10 +1871,7 @@ def scout( console.print(f" [cyan]{h.year or '?'}[/cyan] {h.title[:76]}") if len(res.same_year) > 5: console.print(f" [dim]… {len(res.same_year) - 5} more in scout.json[/dim]") - console.print( - "[dim]search-based — absence from these lists proves nothing; presence is a" - " candidate for your judgement, not an accusation.[/dim]" - ) + console.print(f"[dim]{SCOUT_CAVEAT}[/dim]") def _check_pipeline( @@ -2177,62 +2217,29 @@ def report( _report_pipeline(case=case, png=png, formats=formats) -@app.command(rich_help_panel="Start here") -def run( - manuscript: Path = typer.Argument(..., exists=True), - case: Path = typer.Option( - None, "--case", "-c", - help="Case folder (default: a folder named after the paper, beside the paper)", - ), - provided: Path = typer.Option( - None, "--provided", - help="Folder of reference PDFs you already have; files match by name " - "-.pdf (e.g. pyrros-2023.pdf)", - ), - email: str = typer.Option(None, "--email", envvar=["PAPERTRACE_EMAIL", "MANUSCRIPTAGENT_EMAIL"]), - model: str = typer.Option(None, "--model"), - png: bool = typer.Option( - False, "--png/--no-png", - help="Also export PNG images of the report looks (one-time: playwright install chromium)", - ), - backend: str = typer.Option("auto", "--backend", help="auto | docling | pymupdf"), - with_scout: bool = typer.Option( - True, "--scout/--no-scout", - help="Also scan Europe PMC for newer + uncited literature", - ), - doi: str = typer.Option( - None, "--doi", - help="DOI of the paper itself — checks the reference numbering against the " - "publisher's deposited list, and pins the scout's literature search", - ), - formats: list[str] = typer.Option( - None, "--format", "-f", - help="Extra looks to render beside report.md: editor | terminal | viewer (repeatable)", - ), - supplement: list[Path] = typer.Option( - None, "--supplement", exists=True, - help="Supplementary material for THIS paper (repeatable). A cited work's " - "supplement needs no flag — drop it in the sources folder named after " - "the reference, e.g. pyrros-2023-supplement.pdf", - ), - llm_refs: bool = typer.Option( - True, "--llm-refs/--no-llm-refs", - help="Also have a model read the printed reference list as a second opinion " - "on the numbering (one extra model call — none under --parse-only, or on a " - "pymupdf backend, which leaves no second reading to compare)", - ), - max_claims: int = typer.Option( - None, "--max-claims", min=1, - help="Check only the first N extracted claims, in reading order; only the " - "references they cite are retrieved, and every report says so", - ), - max_sources: int = typer.Option( - None, "--max-sources", min=1, - help="Obtain and judge against at most N cited sources (PDFs), in bibliography " - "order; the rest are skipped, and every report says so", - ), +def _run_pipeline( + *, + manuscript: Path, + case: Path | None = None, + provided: Path | None = None, + email: str | None = None, + model: str | None = None, + png: bool = False, + backend: str = "auto", + with_scout: bool = True, + doi: str | None = None, + formats: list[str] | None = None, + supplement: list[Path] | None = None, + llm_refs: bool = True, + max_claims: int | None = None, + max_sources: int | None = None, ) -> None: - """Full pipeline: ingest → extract → refs → scout → check → highlight → report.""" + """`run`'s work: ingest → extract → refs → scout → check → highlight → report. + + Keyword-only with plain defaults, for the reason every stage function is: + `run` has a second caller — the MCP server's audit job — and a caller that + omitted an option from the Typer command would hand it an `OptionInfo`. + """ console.print(BANNER) email = _email(email) # fail fast — before the ingest models load, not after # the limits as the stages take them: an array of claim ids and a count. @@ -2305,6 +2312,68 @@ def run( ) +@app.command(rich_help_panel="Start here") +def run( + manuscript: Path = typer.Argument(..., exists=True), + case: Path = typer.Option( + None, "--case", "-c", + help="Case folder (default: a folder named after the paper, beside the paper)", + ), + provided: Path = typer.Option( + None, "--provided", + help="Folder of reference PDFs you already have; files match by name " + "-.pdf (e.g. pyrros-2023.pdf)", + ), + email: str = typer.Option(None, "--email", envvar=["PAPERTRACE_EMAIL", "MANUSCRIPTAGENT_EMAIL"]), + model: str = typer.Option(None, "--model"), + png: bool = typer.Option( + False, "--png/--no-png", + help="Also export PNG images of the report looks (one-time: playwright install chromium)", + ), + backend: str = typer.Option("auto", "--backend", help="auto | docling | pymupdf"), + with_scout: bool = typer.Option( + True, "--scout/--no-scout", + help="Also scan Europe PMC for newer + uncited literature", + ), + doi: str = typer.Option( + None, "--doi", + help="DOI of the paper itself — checks the reference numbering against the " + "publisher's deposited list, and pins the scout's literature search", + ), + formats: list[str] = typer.Option( + None, "--format", "-f", + help="Extra looks to render beside report.md: editor | terminal | viewer (repeatable)", + ), + supplement: list[Path] = typer.Option( + None, "--supplement", exists=True, + help="Supplementary material for THIS paper (repeatable). A cited work's " + "supplement needs no flag — drop it in the sources folder named after " + "the reference, e.g. pyrros-2023-supplement.pdf", + ), + llm_refs: bool = typer.Option( + True, "--llm-refs/--no-llm-refs", + help="Also have a model read the printed reference list as a second opinion " + "on the numbering (one extra model call — none under --parse-only, or on a " + "pymupdf backend, which leaves no second reading to compare)", + ), + max_claims: int = typer.Option( + None, "--max-claims", min=1, + help="Check only the first N extracted claims, in reading order; only the " + "references they cite are retrieved, and every report says so", + ), + max_sources: int = typer.Option( + None, "--max-sources", min=1, + help="Obtain and judge against at most N cited sources (PDFs), in bibliography " + "order; the rest are skipped, and every report says so", + ), +) -> None: + """Full pipeline: ingest → extract → refs → scout → check → highlight → report.""" + _run_pipeline(manuscript=manuscript, case=case, provided=provided, email=email, + model=model, png=png, backend=backend, with_scout=with_scout, doi=doi, + formats=formats, supplement=supplement, llm_refs=llm_refs, + max_claims=max_claims, max_sources=max_sources) + + def main() -> None: app() diff --git a/src/papertrace/disclosures.py b/src/papertrace/disclosures.py index c4a0c2e..34afefe 100644 --- a/src/papertrace/disclosures.py +++ b/src/papertrace/disclosures.py @@ -307,6 +307,22 @@ def _converter(name: str) -> Disclosure: ) +def coverage_audited(coverage: dict) -> bool: + """Did the coverage audit run at all? One rule, because two readers need it. + + The reports decide on it whether to print a ratio or `coverage not audited`, + and the MCP server whether its occurrence-level gap list is a measurement or + `null`. Were two copies to disagree, one reader would call "never measured" + what the other calls "measured, nothing missing". It answers for the audit + as a whole, not for each list: the label-level lists come from `clean.md` + and can be empty because nothing read them while the occurrences counted. + + Occurrences without labels means `clean.md` was missing while the source + map was not: the audit ran, so it must not read as "not audited". + """ + return bool(coverage.get("labels_in_text") or (coverage.get("occurrences") or {}).get("total")) + + def coverage_headline(coverage: dict) -> str: """The one sentence the coverage audit is allowed to be summarised as. @@ -1348,9 +1364,7 @@ def run_disclosures(results, manifest=None) -> list[Disclosure]: out.append(_supplement_identity(checked_sup, named_sup)) coverage = results.coverage or {} if coverage: - # occurrences without labels means clean.md was missing while the source - # map was not: the audit ran, so it must not read as "not audited" - if coverage.get("labels_in_text") or (coverage.get("occurrences") or {}).get("total"): + if coverage_audited(coverage): out.append(_coverage(coverage)) # only occurrence-level coverage owes the attribution caveat: the # label-level audit it replaces could not make that mistake diff --git a/src/papertrace/mcp_server.py b/src/papertrace/mcp_server.py new file mode 100644 index 0000000..8e3708d --- /dev/null +++ b/src/papertrace/mcp_server.py @@ -0,0 +1,1339 @@ +"""PaperTrace as an MCP server — audits started, followed and read from any MCP host. + +The server is a fifth reader of everything the four report formats carry, and +it is held to their rules: it computes no verdict, every count travels with the +disclosures `disclosures.py` decided for it, and every refusal says why. The +design and what it rejected: `docs/superpowers/specs/2026-09-26-mcp-server-design.md`. +""" + +# No `from __future__ import annotations`, unlike the rest of the package: the +# SDK builds each tool's input and output schema from its annotations when the +# tool is registered, and live types are what it reads without guessing. + +import datetime +import io +import json +import os +import sys +import threading +import time +from collections.abc import Callable, Iterator +from contextlib import contextmanager +from dataclasses import dataclass, field +from pathlib import Path +from typing import Annotated, Any, Literal + +import typer +from mcp.server import MCPServer +from mcp.server.mcpserver import Image +from mcp.server.mcpserver.exceptions import ToolError +from mcp.types import TextContent, ToolAnnotations +from pydantic import Field +from rich.console import Console + +# Not `typing.TypedDict`: below Python 3.12 pydantic cannot build a schema from +# it, and the SDK then drops the tool to unvalidated text without a word — every +# tool here published no output schema on 3.10 and 3.11 until a test asked. +# `structured_output=True` on each tool turns any repeat into an error at start. +from typing_extensions import TypedDict + +from . import __version__, cli +from .ask import forget_models +from .disclosures import ( + anchor_disclosure, + claim_disclosures, + coverage_audited, + coverage_headline, + ids_span, + judgement_disclosures, + run_disclosures, +) +from .models import ( + PIPELINE_STATES, + REF_STATUSES, + VERDICTS, + ClaimResult, + RefManifest, + RunResults, + ScoutResults, + SourceJudgement, +) + +# the viewer's serialisation of a disclosure, reused rather than restated: one +# shape for every reader that receives disclosures as data +from .report import FORMATS, _disclosure_dict + +INSTRUCTIONS = """\ +PaperTrace checks what a scientific paper claims against what its cited sources say. + +Workflow: start_audit, then audit_status until it is finished (wait_seconds up to 50), +then audit_summary. From there: list_claims, get_claim, get_evidence (the red-box crop +of the passage a verdict rests on), list_references, list_gaps, get_scout. A case folder +made by the `papertrace` CLI can be read the same way. + +Relaying results: +- An unread source never receives a verdict. `not_retrieved` means nobody read the + source: report it as a gap, never as support or contradiction. +- Relay the disclosures that come with a result. They say what the numbers do not. +- `limited` set means the audit was cut short on request: its counts describe a slice, + not the paper. +- Verdicts are a model's drafts for a person's judgement; the evidence images show the + passage each rests on. Scout hits are candidates, not accusations. +""" + +# read tools change nothing and consult nothing outside the case folder +_READ = ToolAnnotations(read_only_hint=True, open_world_hint=False) +# The one tool that spends says all of it, because a host decides on these +# whether to ask first: it rewrites the case folder, queries the network and +# calls a model, and no two runs of a model are the same run. +_SPEND = ToolAnnotations( + read_only_hint=False, destructive_hint=True, idempotent_hint=False, open_world_hint=True +) + +# The vocabularies `models.py` and `report.py` define, not copies of them. +# `Literal` over the tuple is the same type as the values spelled out. +Verdict = Literal[VERDICTS] +RefStatus = Literal[REF_STATUSES] +Format = Literal[FORMATS] +# No constant holds this one; `ingest.resolve_backend` refuses anything else, +# and a test holds the two together. +Backend = Literal["auto", "docling", "pymupdf"] + +# A TypeScript-SDK host gives a request 60 s; a status call waits for less. +MAX_WAIT_SECONDS = 50 + +CaseArg = Annotated[ + str, + Field(description=( + "The case folder of one audit: the folder holding refs_manifest.json and out/. " + "Use an absolute path — a relative one is resolved against this server's working " + "directory, which the host chose." + )), +] +ClaimId = Annotated[int, Field(description="A claim's number, as list_claims gives it.", ge=1)] + + +# -------------------------------------------------------------------------- +# what the tools return — each shape is also the published output schema +# -------------------------------------------------------------------------- + + +class DisclosureOut(TypedDict): + """One thing a result owes its reader, in the words every report format uses.""" + + key: str + level: str + token: str + text: str + short: str + rows: list[str] + + +class RefCounts(TypedDict): + total: int + available: int + + +class AuditSummary(TypedDict): + case: str + manuscript: str + checker: str + date: str + converter: str + references: RefCounts + claims: int + counts: dict[str, int] + uncited_assertions: int + coverage: str | None + disclosures: list[DisclosureOut] + reports: list[str] + limited: str | None + + +class ClaimRow(TypedDict): + id: int + verdict: str + label: str + qualifier: str + refs: list[str] + location: str + claim: str + quote: str + caveats: list[str] + + +class ClaimList(TypedDict): + case: str + verdict_filter: str | None + claims_total: int + claims: list[ClaimRow] + run_disclosures: list[DisclosureOut] + limited: str | None + + +class JudgementOut(TypedDict): + source: str + origin: str + kind: str + verdict: str + label: str + rationale: str + page: int | None + block: str | None + anchor_phrases: list[str] + anchor_located: bool | None + evidence_images: list[str] + disclosures: list[DisclosureOut] + + +class ClaimDetail(TypedDict): + case: str + id: int + verdict: str + label: str + qualifier: str + claim: str + quote: str + location: str + refs: list[str] + own_supplement: bool + rationale: str + source: str | None + page: int | None + block: str | None + anchor_phrases: list[str] + anchor_located: bool | None + evidence_images: list[str] + judgements: list[JudgementOut] + unjudged_refs: list[str] + withheld_refs: list[str] + disclosures: list[DisclosureOut] + run_disclosures: list[DisclosureOut] + + +class SupplementOut(TypedDict): + slug: str + verified: bool + + +class ReferenceRow(TypedDict): + label: str + status: str + reason: str + resolver: str | None + doi: str | None + title: str | None + year: str | None + slug: str | None + title_check: str | None + skipped_by: str | None + boundary_ambiguous: bool + seen_in: list[str] + supplements: list[SupplementOut] + raw: str + + +class NumberingOut(TypedDict): + source: str + verified: bool + note: str + unverified_from: int | None + contested: bool + corroborated: bool | None + corroborating_readings: list[str] + labels_disputed: list[str] + labels_uncomparable: list[str] | None + labels_resolved: list[str] + choice: str + chosen_by: str + + +class ReferenceList(TypedDict): + case: str + manuscript: str + total: int + available: int + by_status: dict[str, int] + numbering: NumberingOut + limits: dict[str, Any] + status_filter: str | None + references: list[ReferenceRow] + + +class UncitedOut(TypedDict): + id: int + claim: str + quote: str + location: str + + +class UnreachedOut(TypedDict): + label: str + status: str + section: str + sentence: str + + +class UncheckedOut(TypedDict): + id: int + verdict: str + label: str + refs: list[str] + location: str + note: str + + +class Gaps(TypedDict): + case: str + uncited_assertions: list[UncitedOut] + coverage: str | None + unreached_labels: list[str] | None + unreached_citations: list[UnreachedOut] | None + unchecked_claims: list[UncheckedOut] + disclosures: list[DisclosureOut] + limited: str | None + + +class ScoutPaperOut(TypedDict): + title: str + doi: str + year: int | None + resolved_via: str + identity: str + + +class ScoutHitOut(TypedDict): + title: str + year: int | None + doi: str + via: str + journal: str + authors: str + + +class ScoutOut(TypedDict): + case: str + paper: ScoutPaperOut + query: str + date: str + counts: dict[str, int] + newer: list[ScoutHitOut] + overlooked: list[ScoutHitOut] + same_year: list[ScoutHitOut] + error: str + caveat: str + + +class AuditStatus(TypedDict): + """Where an audit this server started stands. Never its counts: those travel + only with the disclosures that qualify them, in audit_summary.""" + + case: str + state: Literal["running", "finished", "failed", "interrupted", "not_started"] + manuscript: str | None + started: str | None + ended: str | None + error: str + log: list[str] | None + log_lines_total: int | None + next: str + + +# -------------------------------------------------------------------------- +# reading a case folder — every failure is a sentence, never an empty result +# -------------------------------------------------------------------------- + + +def _case_dir(case: str) -> Path: + return Path(case).expanduser().resolve() + + +def _read(path: Path, load, stage: str): + """One case file, loaded — or a refusal naming why it could not be.""" + try: + return load(path) + except (OSError, ValueError, KeyError, TypeError) as e: + raise ToolError( + f"{path} is unreadable ({type(e).__name__}: {str(e)[:200]}) — an audit may be " + f"writing it right now; if not, re-run `papertrace {stage}` for this case" + ) from None + + +def _results(case: Path) -> RunResults: + path = case / "out" / "results.json" + if not path.is_file(): + # absent is not zero: empty counts would read as an audit that found nothing + raise ToolError( + f"no audit results at {path}" + + ("" if case.is_dir() else f" — {case} does not exist") + + ". No claim has been checked in this case folder: start_audit runs the " + "whole pipeline, and `papertrace check -c ` runs the judging alone." + ) + return _read(path, RunResults.from_json, "check") + + +def _manifest(case: Path, *, required: bool) -> RefManifest | None: + path = case / "refs_manifest.json" + if not path.is_file(): + if required: + raise ToolError( + f"no retrieval manifest at {path} — the refs stage has not run in this " + "case folder, so no reference has been looked up there" + ) + return None + return _read(path, RefManifest.from_json, "refs") + + +def _disclosures(fired) -> list[DisclosureOut]: + return [_disclosure_dict(d) for d in fired] + + +def _limited(fired) -> str | None: + """The scope sentence, when the audit was limited on request — else None. + + Taken from the run disclosures rather than composed here, so the last thing + a result says is the sentence every report format ends with. + """ + return next((d.text for d in fired if d.key == "scope"), None) + + +def _find_claim(results: RunResults, claim_id: int) -> ClaimResult: + for c in results.claims: + if c.id == claim_id: + return c + ids = [c.id for c in results.claims] + raise ToolError( + f"no claim {claim_id} in this audit — its claims are {ids_span(ids) if ids else 'none'}" + + (" (the audit was limited on request; audit_summary's `limited` says what was left " + "out)" if results.scope else "") + ) + + +def _claim_row(c: ClaimResult, manifest: RefManifest | None) -> ClaimRow: + return { + "id": c.id, + "verdict": c.verdict, + "label": c.label, + "qualifier": c.headline_qualifier(), + "refs": list(c.refs), + "location": c.location, + "claim": c.claim, + "quote": c.quote, + # the terse line of each claim disclosure: every one carries its token + "caveats": [d.short for d in claim_disclosures(c, manifest)], + } + + +def _images_of(a) -> list[str]: + return [p for p in [a.evidence_image, *a.continuation_images] if p] + + +def _judgement_out(j: SourceJudgement) -> JudgementOut: + return { + "source": j.source_slug, + "origin": j.origin, + "kind": j.kind, + "verdict": j.verdict, + "label": j.label, + "rationale": j.note, + "page": j.source_page, + "block": j.source_block, + "anchor_phrases": list(j.anchor_phrases), + "anchor_located": j.anchor_located, + "evidence_images": _images_of(j), + "disclosures": _disclosures(judgement_disclosures(j)), + } + + +def _evidence_target(claim: ClaimResult, source: str | None): + """The judgement whose crop to show: the headline's, or the named source's.""" + if source is None: + return claim.deciding_judgement() or claim + for j in claim.judgements: + if j.source_slug == source: + return j + if not claim.judgements and claim.source_slug == source: + return claim + known = [j.source_slug for j in claim.judgements] or [s for s in [claim.source_slug] if s] + raise ToolError( + f"claim {claim.id} was not judged against {source!r} — " + f"its sources are {', '.join(known) if known else 'none'}" + ) + + +def _no_evidence(claim: ClaimResult, target) -> str: + """Why there is no picture — one reason per state, never a blank.""" + who = f"claim {claim.id}" + if target.verdict == "not_retrieved": + return ( + f"{who} has no evidence to show: its cited source was not retrieved, so nothing " + "was read — a recorded gap, not a finding" + ) + if target.verdict == "unchecked": + return ( + f"{who} has no evidence to show: no verdict was reached " + f"({target.note or 'the check did not complete'})" + ) + if target.verdict == "not_addressed": + return ( + f"{who}'s source was read and does not address the claim — that verdict rests " + "on no passage, so it carries no crop by design" + ) + return ( + f"{who} is {target.verdict}, but no evidence crop has been written for it — the " + "highlight stage writes the crops (`papertrace highlight -c `)" + ) + + +def _crop_paths(case: Path, claim: ClaimResult, rels: list[str]) -> list[Path]: + """The crops, resolved — refused if one points anywhere but `/out/`.""" + out = (case / "out").resolve() + paths = [] + for rel in rels: + p = (out / rel).resolve() + # `results.json` is a file anyone can edit: a path is followed only when + # it lands inside the folder the highlight stage writes, and is a PNG + if not p.is_relative_to(out) or p.suffix.lower() != ".png": + raise ToolError( + f"claim {claim.id}'s evidence path {rel!r} resolves outside {out} or is not " + "a PNG — refused rather than followed; this server serves only the crops " + "the highlight stage writes into the case folder" + ) + paths.append(p) + return paths + + +def _caption( + claim: ClaimResult, target, shown: list[Path], missing: list[str], manifest, run_fired +) -> str: + """What the crop is, and every caveat on the verdict it is shown under. + + The caveats travel with the picture because the picture is shown with its + verdict: a crop from a source whose label may name another paper, captioned + without saying so, would present the wrong paper as evidence. + """ + qualifier = claim.headline_qualifier() + lines = [f"claim {claim.id} — {claim.label}" + (f" ({qualifier})" if qualifier else "")] + if claim.quote: + lines.append(f"the paper's sentence: {claim.quote}") + origin = f"{target.origin} " if isinstance(target, SourceJudgement) else "" + lines.append( + f"{target.label} — {origin}{target.source_slug}" + + (f", page {target.source_page}" if target.source_page else "") + ) + if (d := anchor_disclosure(target)) is not None: + lines.append(d.text) + if target.anchor_phrases: + lines.append("anchor phrases: " + " · ".join(f"“{p}”" for p in target.anchor_phrases)) + if len(shown) > 1: + lines.append( + f"the passage crosses a column or page break — {len(shown)} images, in reading order" + ) + if missing: + lines.append(f"not in the case folder, so not shown: {', '.join(missing)}") + # every claim caveat but the claim-level anchor: that is the headline's, and + # the anchor line above already says the shown source's own + if caveats := [d.short for d in claim_disclosures(claim, manifest) if d.key != "anchor"]: + lines += ["caveats on this claim:", *(f"- {c}" for c in caveats)] + if run_fired: + lines += ["caveats on the whole audit:", *(f"- {d.short}" for d in run_fired)] + return "\n".join(lines) + + +def _scout_caveat(scout: ScoutResults) -> str: + """The scout's own caveat, led by what the report formats print beside it: + the identity warning, and the scan's own error when it did not complete.""" + identity = scout.paper_identity + if not scout.paper_title or identity == "confirmed": + lead = "" + elif identity == "mismatch": + lead = "The record found is NOT this paper, so these registers describe another one. " + elif identity == "unverified": + lead = "The paper's identity in Europe PMC is unverified — check the record is this paper. " + else: + lead = "The paper's identity was not recorded — check the record is this paper. " + if scout.error: + # the reports print it beside the caveat: an incomplete scan's empty + # register is absence of data, not a clean search + lead += f"The scan is incomplete: {scout.error}. " + return lead + cli.SCOUT_CAVEAT[0].upper() + cli.SCOUT_CAVEAT[1:] + + +# -------------------------------------------------------------------------- +# the audit job — the pipeline, in this process, with nobody to ask +# -------------------------------------------------------------------------- + + +def _now() -> str: + return datetime.datetime.now(datetime.timezone.utc).isoformat(timespec="seconds") + + +class _Lines(io.TextIOBase): + """A text stream kept as whole lines — the file the pipeline's console writes to.""" + + def __init__(self, lines: list[str]): + self._lines = lines + self._open = "" + self._lock = threading.Lock() + + def writable(self) -> bool: + return True + + def write(self, s: str) -> int: + with self._lock: + *whole, self._open = (self._open + s).split("\n") + self._lines.extend(whole) + return len(s) + + def close(self) -> None: + with self._lock: + if self._open: # a last line printed without its newline is still said + self._lines.append(self._open) + self._open = "" + super().close() + + +@contextmanager +def _console_into(lines: list[str]) -> Iterator[list[str]]: + """What the pipeline prints, as lines of text instead of output on stdout. + + Every stage prints through the module global `cli.console`, which writes to + stdout — under stdio, the protocol's own wire. Swapped for the length of one + call and handed back, so a job's log is exactly what the CLI would have shown. + """ + sink = _Lines(lines) + saved = cli.console + cli.console = Console(file=sink, force_terminal=False, no_color=True, soft_wrap=True) + try: + yield lines + finally: + cli.console = saved + sink.close() + + +@contextmanager +def _nobody_to_ask() -> Iterator[None]: + """Nobody is at an MCP call to answer a question, so nothing may wait for one. + + With stdin an empty stream, `cli._interactive()` is False and any prompt + reads end-of-file at once. A disputed reference label is then withheld with + `numbering_chosen_by: "default"` — the CLI's own answer when no terminal is + attached — and the gate on settling it, which is a person shown the + disagreement, is never reached by a host that may be a model. + """ + saved = sys.stdin + sys.stdin = io.StringIO() + try: + yield + finally: + sys.stdin = saved + + +def _last_said(lines: list[str], n: int = 4) -> str: + """The last lines a stage printed — where the CLI says why it stopped.""" + return " ".join(line.strip() for line in [x for x in lines if x.strip()][-n:]) + + +# What any server knows about the last audit an MCP server started in a case +# folder. The job writes it "running" before the pipeline touches anything, +# refreshes it every HEARTBEAT_SECONDS while it runs, and rewrites it with the +# outcome when it ends. So a "running" record is measured, not assumed: fresh, +# and an audit is live in some server process; stale, and the server writing it +# stopped mid-run. Its shape is `schemas/mcp_audit.schema.json`. +RECORD = "mcp_audit.json" +RECORD_SCHEMA = "mcp_audit/1" +_RECORD_STATES = ("running", "finished", "failed") +HEARTBEAT_SECONDS = 15.0 +# four missed refreshes: a laptop that slept longer reads its own audit as +# stopped, which refuses to read the case — the safe side +STALE_AFTER_SECONDS = 60.0 + + +@dataclass +class _Job: + """One audit this server process started.""" + + case: Path + manuscript: Path + started: str + scout: bool = True # False leaves an earlier run's scout.json in the folder + state: str = "running" # running | finished | failed + ended: str | None = None + error: str = "" + log: list[str] = field(default_factory=list) + done: threading.Event = field(default_factory=threading.Event) + + +def _write_record(job: _Job) -> None: + payload = { + "schema": RECORD_SCHEMA, + "state": job.state, + "manuscript": str(job.manuscript), # the same absolute path audit_status gives + "started": job.started, + "ended": job.ended, + "error": job.error, + "scout": job.scout, + } + (job.case / RECORD).write_text( + json.dumps(payload, indent=2, ensure_ascii=False) + "\n", encoding="utf-8" + ) + + +def _read_record(case: Path) -> dict | None: + """The last MCP audit's record, or None when no MCP server audited here — and + a refusal, never a guess, when there is one this server cannot vouch for.""" + path = case / RECORD + if not path.is_file(): + return None + try: + record = json.loads(path.read_text(encoding="utf-8")) + # every field a decision rests on, checked: "false" is not False, and a + # later schema's record may mean something this one does not + if not ( + isinstance(record, dict) + and record.get("schema") == RECORD_SCHEMA + and record.get("state") in _RECORD_STATES + and isinstance(record.get("scout"), bool) + and isinstance(record.get("started"), str) + ): + raise ValueError(f"not a {RECORD_SCHEMA} record this server can read") + except (OSError, ValueError) as e: + raise ToolError( + f"{path} is unreadable ({type(e).__name__}: {str(e)[:200]}), so whether the last " + "audit here finished cannot be told and nothing is read as its result — run " + "start_audit again" + ) from None + return record + + +def _record_age(case: Path) -> float: + """Seconds since the record was last written or refreshed.""" + return max(0.0, time.time() - (case / RECORD).stat().st_mtime) + + +def _written_after_record(case: Path, path: Path) -> bool: + """Was `path` written after the record's last update — by a later run?""" + return path.is_file() and path.stat().st_mtime > (case / RECORD).stat().st_mtime + + +def _heartbeat(job: _Job) -> None: + """Refresh the record while the job runs: freshness is what tells another + server this audit is alive rather than abandoned.""" + path = job.case / RECORD + while not job.done.wait(HEARTBEAT_SECONDS): + try: + os.utime(path) + except OSError: + return # the final write reports what the job could not keep up + + +class _Jobs: + """The audits this server process started — at most one running at a time. + + One, because what a run touches is process-global: `cli.console`, `ask`'s + per-site model record and `sys.stdin`. Two audits sharing them would + interleave their logs and name each other's models. Across processes the + record is the guard: a fresh "running" one is someone else's live audit. + """ + + def __init__(self): + self._lock = threading.Lock() # held across refuse, prepare and launch + self._running: _Job | None = None + self._by_case: dict[Path, _Job] = {} + + def last(self, case: Path) -> _Job | None: + return self._by_case.get(case) + + def refuse_live_elsewhere(self, case: Path) -> None: + """A second pipeline in a folder another process is writing is the + mixing `_guard_case` exists to prevent.""" + record = _read_record(case) + if record is not None and record["state"] == "running": + # both callers have ruled out this server's own job for this case, + # so a fresh record is another process's + age = _record_age(case) + if age < STALE_AFTER_SECONDS: + raise ToolError( + f"another server process is auditing {case} now (started " + f"{record['started']}; its record was refreshed {age:.0f} s ago). Follow " + "it from the host that started it, and start this one when it has ended." + ) + + def refuse_unfinished(self, case: Path) -> None: + """Nothing on disk is read as a result unless the last audit here ended well. + + The record on disk decides, not this server's memory of its own jobs: + a case completed since by other means must read again, here too. + """ + running = self._running + if running is not None and running.case == case: + raise ToolError( + f"an audit of {case} is running now (started {running.started}) — what is on " + "disk belongs to an earlier run or is half-written, so nothing is read from it " + "until the audit ends. Call audit_status." + ) + record = _read_record(case) + if record is None: + return # no MCP server audited here: read as the reports would + path = case / RECORD + if record["state"] == "running": + self.refuse_live_elsewhere(case) + raise ToolError( + f"an audit of {case} started {record['started']} did not finish: its record " + f"was last refreshed {_record_age(case):.0f} s ago, so the server running it " + "stopped mid-run. What is on disk may be half-written or mix two runs, and " + "none of it is read as a result. Run start_audit again — or, if the case has " + f"been completed since by other means, remove {path} first." + ) + if record["state"] == "failed" and not _written_after_record( + case, case / "out" / "results.json" + ): + raise ToolError( + f"the last audit of {case} (started {record['started']}) failed: " + f"{record.get('error') or 'no reason was recorded'}. What is on disk may mix " + "that run with an earlier one, so none of it is read as a result — fix the " + "cause and run start_audit again." + ) + + def start(self, prepare: Callable[[], tuple[Path, Path, dict, list[str]]]) -> _Job: + """Run `prepare` — every check that can refuse before anything is spent — + then the pipeline on its own thread, and return without waiting for it.""" + with self._lock: + if (running := self._running) is not None: + raise ToolError( + f"an audit is already running in this server: {running.case} (started " + f"{running.started}). One audit runs at a time — follow it with " + "audit_status, and start this one when it has ended." + ) + case, manuscript, kwargs, said = prepare() + job = _Job(case=case, manuscript=manuscript, started=_now(), + scout=kwargs["with_scout"], log=list(said)) + self._running = self._by_case[case] = job + # daemon: when the host stops the server, no further stage and no + # further model call starts. A `claude -p` call already in flight + # runs to its own end — `ask.py`'s timeout bounds it — and the record + # it stops refreshing is what tells a later server the case is + # incomplete. + threading.Thread( + target=self._run, args=(job, kwargs), name="papertrace-audit", daemon=True + ).start() + return job + + def _run(self, job: _Job, kwargs: dict) -> None: + state, error = "failed", "" + try: + with _console_into(job.log), _nobody_to_ask(): + # the folder and its .gitignore first, so the record never lands + # in a folder that is briefly untracked + cli._open_case(job.case) + _write_record(job) + threading.Thread(target=_heartbeat, args=(job,), name="papertrace-heartbeat", + daemon=True).start() + # the process outlives one audit: a model an earlier audit + # recorded must not be reported as this one's judge + forget_models() + cli._run_pipeline(**kwargs) + state = "finished" + except typer.Exit as e: + if e.exit_code == 0: + state = "finished" + else: + error = _last_said(job.log) or f"the pipeline stopped (exit {e.exit_code})" + except Exception as e: + # Broad on purpose, and recorded rather than swallowed: an exception + # leaving a tool reaches a model as a bare "Error executing tool", and + # one leaving this thread would leave the job "running" forever. The + # type and message are the finding, so the job keeps both. + error = f"{type(e).__name__}: {str(e)[:400]}" + finally: + if state == "failed" and not error: + error = "the audit was interrupted before it could finish" + job.state, job.error, job.ended = state, error, _now() + try: + _write_record(job) + except OSError as e: + # this server still knows the outcome; a later one reads the + # "running" left behind, unrefreshed, as a stopped audit — which + # refuses, the safe side + job.log.append(f"the audit record could not be updated: {e}") + with self._lock: + if self._running is job: + self._running = None + job.done.set() # last: whoever wakes on it sees the final state + + +_NEXT = { + "running": "call audit_status again, with wait_seconds up to 50, until the audit has ended", + "finished": "call audit_summary for the result — its counts travel with the caveats that " + "qualify them", + "failed": "read `error` and the log, fix what they name, and call start_audit again", + "not_started": "no MCP server has audited this case — audit_summary reads whatever a CLI " + "run left on disk", +} +# the same states read from a record another server process left — whose log +# is not here, so nothing may send the host to read one +_NEXT_RECORDED = { + "running": "another server process is running this audit — its log is there, not here; " + "call audit_status again, with wait_seconds up to 50, until it has ended", + "finished": "call audit_summary for the result — the audit was recorded by another server " + "process, whose log was not kept", + "failed": "read `error`, fix what it names, and call start_audit again — the audit was " + "recorded by another server process, whose log was not kept", + "interrupted": "the server running this audit stopped before it ended, and its log was not " + "kept — call start_audit to run it again", +} + + +def _recorded_state(case: Path, record: dict) -> str: + """A record's state as measured: `running` only while it is being refreshed.""" + if record["state"] != "running": + return record["state"] + return "running" if _record_age(case) < STALE_AFTER_SECONDS else "interrupted" + + +def _status(case: Path, job: _Job | None, log_lines: int, record: dict | None = None) -> AuditStatus: + """The job's state — this server's own, else the record another one left. + + `log` and `log_lines_total` are null, not empty, when no log exists here: an + empty log reads as an audit that printed nothing. + """ + if job is None and record is not None: + state = _recorded_state(case, record) + return { + "case": str(case), "state": state, "manuscript": record.get("manuscript"), + "started": record.get("started"), "ended": record.get("ended"), + "error": record.get("error") or "", "log": None, "log_lines_total": None, + "next": _NEXT_RECORDED[state], + } + if job is None: + return { + "case": str(case), "state": "not_started", "manuscript": None, "started": None, + "ended": None, "error": "", "log": None, "log_lines_total": None, + "next": _NEXT["not_started"], + } + log = list(job.log) + return { + "case": str(job.case), + "state": job.state, + "manuscript": str(job.manuscript), + "started": job.started, + "ended": job.ended, + "error": job.error, + "log": log[-log_lines:] if log_lines else [], + "log_lines_total": len(log), + "next": _NEXT[job.state], + } + + +# -------------------------------------------------------------------------- +# the server +# -------------------------------------------------------------------------- + + +def build_server() -> MCPServer: + """A fresh server with every tool registered. Nothing is served until `run()`.""" + server = MCPServer("papertrace", instructions=INSTRUCTIONS, version=__version__) + jobs = _Jobs() + + def _open(case: str) -> Path: + """A case folder to read — unless the last audit here has not ended well.""" + root = _case_dir(case) + jobs.refuse_unfinished(root) + return root + + @server.tool(title="Audit summary", annotations=_READ, structured_output=True) + def audit_summary(case: CaseArg) -> AuditSummary: + """The result of one audit: verdict counts, references obtained, the model that + judged, coverage, and every run-level disclosure — relay those with the counts. + `limited` is set when the audit was cut short on request.""" + root = _open(case) + results = _results(root) + manifest = _manifest(root, required=False) + fired = run_disclosures(results, manifest) + return { + "case": str(root), + "manuscript": results.manuscript, + "checker": results.checker, + "date": results.date, + "converter": results.converter, + "references": {"total": results.refs_total, "available": results.refs_available}, + "claims": len(results.claims), + "counts": results.counts(), + "uncited_assertions": len(results.uncited), + # null when the audit could not run — never "" or a zero ratio + "coverage": coverage_headline(results.coverage or {}) or None, + "disclosures": _disclosures(fired), + "reports": sorted(str(p) for p in (root / "out").glob("report*.*") if p.is_file()), + "limited": _limited(fired), + } + + @server.tool(title="List claims", annotations=_READ, structured_output=True) + def list_claims( + case: CaseArg, + verdict: Annotated[ + Verdict | None, + Field(description="Only claims whose headline verdict is this one."), + ] = None, + ) -> ClaimList: + """Every checked claim, one row each: its headline verdict, the paper's own + sentence, the labels it cites, and its caveats. A claim citing several sources is + headlined by the most adverse verdict; get_claim shows each source's.""" + root = _open(case) + results = _results(root) + manifest = _manifest(root, required=False) + rows = [_claim_row(c, manifest) for c in results.claims if verdict in (None, c.verdict)] + fired = run_disclosures(results, manifest) + return { + "case": str(root), + "verdict_filter": verdict, + "claims_total": len(results.claims), + "claims": rows, + # a host can list claims without ever reading the summary, so the + # caveats the reports print above every verdict come here too + "run_disclosures": _disclosures(fired), + "limited": _limited(fired), + } + + @server.tool(title="Claim detail", annotations=_READ, structured_output=True) + def get_claim(case: CaseArg, claim_id: ClaimId) -> ClaimDetail: + """One claim in full: the paper's sentence, each cited source's verdict with its + rationale, page, block and anchor phrases, and every disclosure the claim owes its + reader. get_evidence shows the passage itself.""" + root = _open(case) + results = _results(root) + manifest = _manifest(root, required=False) + c = _find_claim(results, claim_id) + return { + "case": str(root), + "id": c.id, + "verdict": c.verdict, + "label": c.label, + "qualifier": c.headline_qualifier(), + "claim": c.claim, + "quote": c.quote, + "location": c.location, + "refs": list(c.refs), + "own_supplement": c.own_supplement, + "rationale": c.note, + "source": c.source_slug, + "page": c.source_page, + "block": c.source_block, + "anchor_phrases": list(c.anchor_phrases), + "anchor_located": c.anchor_located, + "evidence_images": _images_of(c), + "judgements": [_judgement_out(j) for j in c.judgements], + "unjudged_refs": list(c.unjudged_refs), + "withheld_refs": list(c.withheld_refs), + "disclosures": _disclosures(claim_disclosures(c, manifest)), + # the caveats the reports print above every verdict: an unconfirmed + # source identity, a source read flat or cut short + "run_disclosures": _disclosures(run_disclosures(results, manifest)), + } + + @server.tool(title="Evidence crops", annotations=_READ, structured_output=False) + def get_evidence( + case: CaseArg, + claim_id: ClaimId, + source: Annotated[ + str | None, + Field(description="A source slug from get_claim's judgements; default: the source " + "behind the headline verdict."), + ] = None, + ) -> list[TextContent | Image]: + """The page crop a verdict rests on, as images, with the matched text boxed in red. + The boxes were drawn by code finding the model's anchor phrases in the PDF, not + placed by the model; the caption says whether the phrase was found.""" + root = _open(case) + results = _results(root) + manifest = _manifest(root, required=False) + claim = _find_claim(results, claim_id) + target = _evidence_target(claim, source) + rels = _images_of(target) + if not rels: + raise ToolError(_no_evidence(claim, target)) + paths = _crop_paths(root, claim, rels) + out = (root / "out").resolve() + missing = [str(p.relative_to(out)) for p in paths if not p.is_file()] + shown = [p for p in paths if p.is_file()] + if not shown: + raise ToolError( + f"claim {claim.id}'s evidence crop{'s' if len(missing) > 1 else ''} " + f"{', '.join(missing)} {'are' if len(missing) > 1 else 'is'} named in " + f"results.json but not in {out} — re-run `papertrace highlight -c `" + ) + return [ + TextContent(type="text", text=_caption( + claim, target, shown, missing, manifest, run_disclosures(results, manifest))), + *(Image(path=p) for p in shown), + ] + + @server.tool(title="References", annotations=_READ, structured_output=True) + def list_references( + case: CaseArg, + status: Annotated[ + RefStatus | None, + Field(description="Only references with this retrieval status."), + ] = None, + ) -> ReferenceList: + """The retrieval manifest: each cited reference, whether it was obtained and why + not, and the state of the reference numbering — the label is what joins a claim to + its source, so an unconfirmed or disputed numbering is a caveat on every verdict + that rests on it. Local file paths are not included.""" + root = _open(case) + m = _manifest(root, required=True) + rows: list[ReferenceRow] = [ + { + "label": e.num, + "status": e.status, + "reason": e.reason, + "resolver": e.resolver, + "doi": e.doi, + "title": e.title, + "year": e.year, + "slug": e.slug, + "title_check": e.title_check, + "skipped_by": e.skipped_by, + "boundary_ambiguous": e.boundary_ambiguous, + "seen_in": list(e.seen_in), + "supplements": [{"slug": s.slug, "verified": s.verified} for s in e.supplements], + "raw": e.raw, + } + for e in m.entries + if status in (None, e.status) + ] + return { + "case": str(root), + "manuscript": m.manuscript, + "total": len(m.entries), + "available": len(m.retrieved), + "by_status": {s: sum(1 for e in m.entries if e.status == s) for s in REF_STATUSES}, + # passed through as recorded: `null` is "never computed" and `[]` a + # measurement, and no reader may be handed one for the other + "numbering": { + "source": m.reference_source, + "verified": m.numbering_verified, + "note": m.numbering_note, + "unverified_from": m.unverified_from, + "contested": m.numbering_contested, + "corroborated": m.numbering_corroborated, + "corroborating_readings": list(m.corroborating_readings), + "labels_disputed": list(m.labels_disputed), + "labels_uncomparable": ( + None if m.labels_uncomparable is None else list(m.labels_uncomparable) + ), + "labels_resolved": list(m.labels_resolved), + "choice": m.numbering_choice, + "chosen_by": m.numbering_chosen_by, + }, + "limits": dict(m.limits), + "status_filter": status, + "references": rows, + } + + @server.tool(title="Gaps", annotations=_READ, structured_output=True) + def list_gaps(case: CaseArg) -> Gaps: + """What the audit could not vouch for: assertions carrying no citation, citation + places no extracted claim reached, and claims never checked (source not retrieved, + or no verdict reached). `null` means that audit did not run — not that it found + nothing.""" + root = _open(case) + results = _results(root) + manifest = _manifest(root, required=False) + coverage = results.coverage or {} + audited = coverage_audited(coverage) + occurrences = coverage.get("occurrences") + fired = run_disclosures(results, manifest) + return { + "case": str(root), + "uncited_assertions": [ + {"id": u.id, "claim": u.claim, "quote": u.quote, "location": u.location} + for u in results.uncited + ], + "coverage": coverage_headline(coverage) or None, + # the label lists come from clean.md: without it they are empty + # because nothing read them, while the occurrences may still count + "unreached_labels": ( + list(coverage.get("missing") or []) if coverage.get("labels_in_text") else None + ), + "unreached_citations": ( + [ + {"label": o["label"], "status": o["status"], + "section": o.get("section", ""), "sentence": o.get("sentence", "")} + for o in occurrences.get("items", []) + if o.get("status") in ("uncovered", "uncertain") + ] + if audited and occurrences else None + ), + "unchecked_claims": [ + {"id": c.id, "verdict": c.verdict, "label": c.label, "refs": list(c.refs), + "location": c.location, "note": c.note} + for c in results.claims + if c.verdict in PIPELINE_STATES + ], + "disclosures": _disclosures(fired), + "limited": _limited(fired), + } + + @server.tool(title="Literature scout", annotations=_READ, structured_output=True) + def get_scout(case: CaseArg) -> ScoutOut: + """Literature around the paper from Europe PMC: work published since, work that + existed before it and went uncited, and work from the same year. Search-based: + candidates for a person's judgement, and absence proves nothing.""" + root = _open(case) + path = root / "out" / "scout.json" + if not path.is_file(): + raise ToolError( + f"no literature scan at {path} — the audit ran without the scout, or the " + "scout stage has not run (`papertrace scout -c `)" + ) + record = _read_record(root) + if ( + record is not None and record["state"] == "finished" and record["scout"] is False + and not _written_after_record(root, path) + ): + raise ToolError( + f"the latest audit of {root} ran without the scout, so {path} belongs to an " + "earlier run — its registers may be anchored to another DOI or another " + "version of this paper, and are not served as this audit's" + ) + scout = _read(path, ScoutResults.from_json, "scout") + return {"case": str(root), **scout.to_dict(), "caveat": _scout_caveat(scout)} + + @server.tool(title="Start an audit", annotations=_SPEND, structured_output=True) + def start_audit( + manuscript: Annotated[str, Field(description="The paper to audit, a PDF — by absolute path.")], + case: Annotated[str | None, Field(description=( + "Case folder to write the audit into. Default: a folder named after the paper, " + "beside it; an earlier audit of the same paper there is amended."))] = None, + email: Annotated[str | None, Field(description=( + "Contact email Unpaywall requires. Default: PAPERTRACE_EMAIL, or the address " + "`papertrace` saved."))] = None, + model: Annotated[str | None, Field(description=( + "Model for `claude -p`, e.g. claude-opus-5. Pin one for a reproducible audit; " + "default: the account's default model."))] = None, + backend: Annotated[Backend, Field(description=( + "How PDFs are read: docling is layout-aware (tables stay tables), pymupdf is flat " + "text; auto prefers docling."))] = "auto", + scout: Annotated[bool, Field(description=( + "Also scan Europe PMC for literature published since, or uncited."))] = True, + doi: Annotated[str | None, Field(description=( + "DOI of the paper itself. Default: the one printed on page 1."))] = None, + formats: Annotated[list[Format] | None, Field(description=( + "Report looks to write beside report.md; `viewer` is the interactive page."))] = None, + provided: Annotated[str | None, Field(description=( + "Folder of cited PDFs already at hand; each is identified by its own DOI or " + "title."))] = None, + supplements: Annotated[list[str] | None, Field(description=( + "The paper's own supplementary PDFs, by absolute path."))] = None, + llm_refs: Annotated[bool, Field(description=( + "Also have a model read the printed reference list as a further check on its " + "numbering (one more model call)."))] = True, + max_claims: Annotated[int | None, Field(ge=1, description=( + "Check only the first N claims; only the references they cite are fetched. The " + "result says what was left out."))] = None, + max_sources: Annotated[int | None, Field(ge=1, description=( + "Obtain and judge against at most N cited sources. The result says what was left " + "out."))] = None, + ) -> AuditStatus: + """Start a full audit of one paper and return at once; follow it with audit_status. + + The pipeline `papertrace run` runs: read the PDF, extract its citation-backed + claims, fetch the cited sources through legal open-access routes, scout the + literature, judge each claim against its cited pages, crop the evidence and write + the reports. It SPENDS: `claude -p` model calls on this machine's logged-in Claude + account (one to extract, one to read the reference list, one per cited source + judged, each retried on failure), and queries to Crossref, Unpaywall, Europe PMC + and arXiv. It takes minutes. One audit runs at a time per server. + + A reference label whose readings disagree is withheld, never settled here: that + needs a person shown the disagreement, at a terminal (`papertrace refs`).""" + pdf = _case_dir(manuscript) + if not pdf.is_file(): + raise ToolError(f"no manuscript at {pdf} — pass the paper's PDF by absolute path") + own = [_case_dir(s) for s in supplements or []] + if absent := [str(s) for s in own if not s.is_file()]: + raise ToolError(f"no supplement at {', '.join(absent)}") + pdfs = _case_dir(provided) if provided else None + if pdfs is not None and not pdfs.is_dir(): + raise ToolError(f"`provided` must be a folder of PDFs, and {pdfs} is not one") + + def prepare() -> tuple[Path, Path, dict, list[str]]: + # everything that can refuse, refused before a model or a network is touched + from .check import claude_available + + if not claude_available(): + raise ToolError( + "the `claude` CLI is not on this server's PATH, and every claim is " + "judged with `claude -p` — install Claude Code and log in. If it is " + "installed, the host started this server with a PATH that does not " + "reach it: set PATH in the server's configuration." + ) + if case is None and not os.access(pdf.parent, os.W_OK): + # `papertrace run` falls back to its working directory, one the + # person chose by standing in it; this server's was chosen by the + # host, so the audit would land wherever that happens to be + raise ToolError( + f"the paper's folder {pdf.parent} is not writable, so the audit cannot sit " + "beside it — pass `case`, the folder to keep it in" + ) + said: list[str] = [] + try: + with _console_into(said): + root = (_case_dir(case) if case else cli.default_case(pdf)).resolve() + jobs.refuse_live_elsewhere(root) + address = cli._email(email) + cli._guard_case(root, pdf) # one case folder per paper + except typer.Exit: + raise ToolError( + "refused before anything was spent — " + _last_said(said, n=8) + ) from None + return root, pdf, { + "manuscript": pdf, "case": root, "provided": pdfs, "email": address, + "model": model, "png": False, "backend": backend, "with_scout": scout, + "doi": doi, "formats": list(formats or []), "supplement": own, + "llm_refs": llm_refs, "max_claims": max_claims, "max_sources": max_sources, + }, said # what the checks printed opens the job's log + + job = jobs.start(prepare) + return _status(job.case, job, log_lines=40) + + @server.tool(title="Audit status", annotations=_READ, structured_output=True) + def audit_status( + case: CaseArg, + wait_seconds: Annotated[int, Field(ge=0, le=MAX_WAIT_SECONDS, description=( + "Wait up to this many seconds for a running audit to end before answering " + "(at most 50, inside the 60 s a host allows a request)."))] = 0, + log_lines: Annotated[int, Field(ge=0, le=1000, description=( + "How many of the audit's latest console lines to include."))] = 40, + ) -> AuditStatus: + """Where an audit this server started stands — running, finished or failed — with + the last lines it printed, which carry its warnings as it goes. Poll with + wait_seconds rather than in a tight loop. A finished audit's result is + audit_summary; this never repeats its counts without their caveats.""" + root = _case_dir(case) + job = jobs.last(root) + if job is not None: + if wait_seconds: + job.done.wait(wait_seconds) + return _status(root, job, log_lines) + # another process's audit: no event to wait on, so watch its record + deadline = time.monotonic() + wait_seconds + record = _read_record(root) + while ( + record is not None and _recorded_state(root, record) == "running" + and time.monotonic() < deadline + ): + time.sleep(min(1.0, max(0.0, deadline - time.monotonic()))) + record = _read_record(root) + return _status(root, None, log_lines, record) + + return server + + +def serve() -> None: + """Serve over stdio until the host closes the connection. + + stdio only: a case folder stays on the machine that made it, and the host + that launched this process is the one client it answers. + """ + build_server().run("stdio") diff --git a/tests/test_ask.py b/tests/test_ask.py index 9a5b574..259ac09 100644 --- a/tests/test_ask.py +++ b/tests/test_ask.py @@ -140,6 +140,23 @@ def test_a_reply_that_names_no_model_does_not_erase_the_one_recorded(monkeypatch assert ask_mod.model_for(ask_mod.SITE_CHECK) == "claude-opus-5" +def test_forget_models_starts_a_run_with_no_recorded_model(monkeypatch): + """One process ran one command, so the record never needed clearing. The + MCP server runs audit after audit in one process: without this, a second + audit whose judging made no calls — every cited source unretrieved — would + print the first audit's judge as its `Checker:`, the misattribution the + per-site record exists to prevent.""" + monkeypatch.setattr( + ask_mod, "_MODELS", + {ask_mod.SITE_CHECK: "claude-opus-5", ask_mod.SITE_REFS: "claude-haiku-4-5"}, + ) + + ask_mod.forget_models() + + assert ask_mod.model_for(ask_mod.SITE_CHECK) is None + assert ask_mod.model_for(ask_mod.SITE_REFS) is None + + def test_for_site_restores_the_previous_site_on_the_way_out(monkeypatch): """Nesting must not leak. A leaked site is a misattributed model.""" monkeypatch.setattr(ask_mod, "_SITE", "outer") diff --git a/tests/test_mcp_server.py b/tests/test_mcp_server.py new file mode 100644 index 0000000..f3e4cdb --- /dev/null +++ b/tests/test_mcp_server.py @@ -0,0 +1,1535 @@ +"""The MCP server: PaperTrace as a tool for any MCP host. + +Every tool is called the way a host calls it — through the SDK's in-memory +`Client` — so what these tests read is what a model reads. Offline: case +folders are written with the dataclasses, evidence PNGs are drawn with +pymupdf, and the one tool that spends is exercised with the pipeline faked. + +The server is a fifth reader of every disclosure. The parity loops below are +`test_disclosure_parity.py`'s contract extended to it: a token a report format +must carry, the MCP output must carry too. +""" + +import json +import os +import sys +import types +from pathlib import Path + +import pytest + +sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "src")) + +# `[dev]` installs the SDK, so CI runs this module rather than skipping it; +# `tests/test_packaging.py` asserts that it stays there. A 1.x SDK — other tools +# install one — is skipped with the same reason rather than failing collection. +pytest.importorskip("mcp", reason="pip install -e '.[dev]' supplies the MCP SDK") + +from importlib.metadata import version as _installed # noqa: E402 + +if int(_installed("mcp").split(".")[0]) < 2: + pytest.skip("the server is written against the MCP SDK 2.x — pip install -e '.[dev]'", + allow_module_level=True) + +import anyio # noqa: E402 +import typer # noqa: E402 +from mcp import Client # noqa: E402 + +from papertrace import ask as ask_mod # noqa: E402 +from papertrace import check as check_mod # noqa: E402 +from papertrace import cli, mcp_server # noqa: E402 +from papertrace.disclosures import ( # noqa: E402 + ANCHOR_LOCATED_TOKEN, + ANCHOR_NOT_LOCATED_TOKEN, + SCOPE_TOKEN, + claim_disclosures, + judgement_disclosures, + run_disclosures, +) +from papertrace.models import ( # noqa: E402 + REF_STATUSES, + VERDICTS, + ClaimResult, + RefEntry, + RefManifest, + RunResults, + ScoutHit, + ScoutResults, + SourceJudgement, + UncitedClaim, +) + +READ_TOOLS = { + "audit_summary", "list_claims", "get_claim", "get_evidence", + "list_references", "list_gaps", "get_scout", +} +# one call to each read tool, with whatever arguments it needs beyond `case` +READ_CALLS = ( + ("audit_summary", {}), ("list_claims", {}), ("get_claim", {"claim_id": 1}), + ("get_evidence", {"claim_id": 1}), ("list_references", {}), ("list_gaps", {}), + ("get_scout", {}), +) +ROOT = Path(__file__).resolve().parent.parent + + +# -------------------------------------------------------------------------- +# calling the server the way a host does +# -------------------------------------------------------------------------- + + +def _session(server, fn): + async def go(): + async with Client(server, raise_exceptions=True) as client: + return await fn(client) + + return anyio.run(go) + + +def _call(tool: str, args: dict, server=None): + server = server or mcp_server.build_server() + return _session(server, lambda c: c.call_tool(tool, args)) + + +def _tools(server=None) -> dict: + server = server or mcp_server.build_server() + listing = _session(server, lambda c: c.list_tools()) + return {t.name: t for t in listing.tools} + + +def _text(result) -> str: + """Everything a model reads from one result, as one string.""" + return "\n".join(b.text for b in result.content if getattr(b, "type", "") == "text") + + +def _json(result) -> dict: + """The result as the model reads it — the JSON text, keys in the order sent.""" + assert not result.is_error, _text(result) + return json.loads(_text(result)) + + +# -------------------------------------------------------------------------- +# a case folder, written the way the pipeline writes one +# -------------------------------------------------------------------------- + + +def _png(path: Path) -> None: + try: + import pymupdf + except ImportError: # PyMuPDF < 1.24 + import fitz as pymupdf + path.parent.mkdir(parents=True, exist_ok=True) + doc = pymupdf.open() + page = doc.new_page(width=80, height=40) + page.insert_text((5, 20), path.stem[-12:]) + page.get_pixmap().save(str(path)) + doc.close() + + +def _split_claim() -> ClaimResult: + """Two cited sources that disagree: the headline is the contradiction.""" + c = ClaimResult( + id=1, + claim="the model reached an AUC of 0.94", + quote="The model reached an AUC of 0.94 [3, 4].", + location="Results §2", + refs=["3", "4"], + judgements=[ + SourceJudgement( + source_slug="alpha-2020", ref="3", verdict="supported", + note="Table 2 reports 0.94.", source_page=2, source_block="block_0012", + anchor_phrases=["AUC 0.94"], evidence_image="evidence/claim_01_alpha-2020_p2.png", + anchor_located=True, + ), + SourceJudgement( + source_slug="beta-2021", ref="4", verdict="contradicted", + note="The source reports 0.77, not 0.94.", source_page=5, + source_block="block_0040", anchor_phrases=["AUC of 0.77"], + evidence_image="evidence/claim_01_beta-2021_p5.png", + continuation_images=["evidence/claim_01_beta-2021_p5_cont2.png"], + anchor_located=False, + ), + ], + ) + c.apply_headline() + return c + + +def _unread_claim() -> ClaimResult: + return ClaimResult( + id=2, claim="attendance was 99%", quote="Attendance was 99% [7].", + location="Discussion", refs=["7"], verdict="not_retrieved", unjudged_refs=["7"], + ) + + +def _withheld_claim() -> ClaimResult: + return ClaimResult( + id=3, claim="the cohort was imaged twice", quote="The cohort was imaged twice [9].", + location="Methods", refs=["9"], verdict="unchecked", withheld_refs=["9"], + note="verdict withheld — the readings of the reference list disagree about [9]", + ) + + +def _paraphrase_claim(anchor_located=None) -> ClaimResult: + return ClaimResult( + id=4, claim="screening halves mortality", location="Introduction", refs=["3"], + verdict="supported", note="Stated in the abstract.", source_slug="alpha-2020", + source_page=1, source_block="block_0002", anchor_phrases=["halves mortality"], + evidence_image="evidence/claim_04_alpha-2020_p1.png", anchor_located=anchor_located, + ) + + +def _manifest() -> RefManifest: + return RefManifest( + manuscript="paper.pdf", + manuscript_sha256="0" * 64, + numbering_verified=False, + numbering_note="the parse and the deposit disagree from [5]", + unverified_from=5, + labels_disputed=["9"], + labels_uncomparable=[], + entries=[ + RefEntry(num="3", raw="Alpha A. A paper. 2020.", doi="10.1/alpha", status="retrieved", + resolver="unpaywall", slug="alpha-2020", title_check="verified", + pdf_path="/private/home/someone/sources_resolved/alpha-2020.pdf"), + RefEntry(num="4", raw="Beta B. Another. 2021.", status="provided", resolver="user", + slug="beta-2021", title_check="unverifiable", + pdf_path="/private/home/someone/my_pdfs/beta.pdf"), + RefEntry(num="7", raw="Gamma G. Paywalled. 2019.", doi="10.1/gamma", + status="paywalled", reason="no open-access copy found"), + RefEntry(num="8", raw="Delta D. Unasked. 2018.", status="skipped", + skipped_by="claims", reason="no selected claim cites it"), + RefEntry(num="9", raw="Epsilon E. Disputed. 2022.", status="retrieved", + resolver="crossref", slug="epsilon-2022", title_check="verified", + pdf_path="/private/home/someone/sources_resolved/epsilon-2022.pdf"), + ], + ) + + +OCCURRENCE_COVERAGE = { + "schema": "coverage/3", "unit": "occurrence", "source": "source_map", + "labels_in_text": ["3", "4", "7", "8", "9"], "covered": ["3", "4", "7", "9"], + "missing": ["8"], "labels_partially_covered": [], "labels_uncertain_only": [], + "occurrences": { + "total": 6, "covered": 4, "uncovered": 1, "uncertain": 1, + "items": [ + {"occ_id": "block_0020:40:0", "label": "8", "block": "block_0020", "page": 3, + "offset": 40, "group": "[8]", "section": "Methods", + "sentence": "Imaging followed the usual protocol [8].", + "covered_by": [], "status": "uncovered"}, + {"occ_id": "block_0031:12:0", "label": "4", "block": "block_0031", "page": 4, + "offset": 12, "group": "[4]", "section": "Results", + "sentence": "A second sentence citing the same source [4].", + "covered_by": [], "status": "uncertain"}, + ], + }, +} + + +def _results(claims=None, **kw) -> RunResults: + base = dict( + manuscript="paper.pdf", checker="claude -p · claude-opus-5", date="2026-09-26", + refs_total=5, refs_available=3, converter="pymupdf", + source_converters={"alpha-2020": "pymupdf", "beta-2021": "docling 2.118.1"}, + claims=claims if claims is not None else [ + _split_claim(), _unread_claim(), _withheld_claim(), _paraphrase_claim(), + ], + uncited=[UncitedClaim(id=1, claim="most clinics now screen", + quote="Most clinics now screen routinely.", location="Introduction")], + coverage=OCCURRENCE_COVERAGE, + truncated={"manuscript": {"chars": 90000, "limit": 60000}}, + ) + base.update(kw) + return RunResults(**base) + + +def _case(tmp_path, results=None, manifest="default", scout=None, images=True) -> Path: + case = tmp_path / "case" + (case / "out").mkdir(parents=True) + results = results if results is not None else _results() + results.to_json(case / "out" / "results.json") + manifest = _manifest() if manifest == "default" else manifest + if manifest is not None: + manifest.to_json(case / "refs_manifest.json") + if scout is not None: + scout.to_json(case / "out" / "scout.json") + if images: + for c in results.claims: + for a in [*c.judgements, c]: + for rel in [a.evidence_image, *a.continuation_images]: + if rel: + _png(case / "out" / rel) + (case / "out" / "report.md").write_text("# report\n") + return case + + +LIMITED_SCOPE = { + "claims": {"requested": [1, 2], "judged": [1, 2], "extracted": 4}, + "sources": {"cap": None, "selection": [1, 2], "skipped_for_claims": ["8"], + "skipped_by_cap": []}, +} + + +# -------------------------------------------------------------------------- +# what the server advertises +# -------------------------------------------------------------------------- + + +def test_every_read_tool_is_marked_read_only_and_closed_world(): + tools = _tools() + + assert READ_TOOLS <= set(tools) + for name in READ_TOOLS: + ann = tools[name].annotations + assert ann.read_only_hint is True, name + assert ann.open_world_hint is False, name + + +def _enum(schema): + """The first `enum` anywhere in a JSON schema fragment.""" + if isinstance(schema, dict): + if "enum" in schema: + return schema["enum"] + schema = list(schema.values()) + for part in schema if isinstance(schema, list) else []: + if (found := _enum(part)) is not None: + return found + return None + + +def test_the_filters_are_the_models_own_vocabularies(): + """A verdict or status spelled here a second time is a second vocabulary to + keep in step, and the one place it is defined is `models.py`.""" + tools = _tools() + + verdict = tools["list_claims"].input_schema["properties"]["verdict"] + status = tools["list_references"].input_schema["properties"]["status"] + assert _enum(verdict) == list(VERDICTS) + assert _enum(status) == list(REF_STATUSES) + for name in READ_TOOLS - {"get_evidence"}: + assert tools[name].output_schema, f"{name} publishes no output schema" + + +# -------------------------------------------------------------------------- +# audit_summary — every run-level disclosure reaches the model +# -------------------------------------------------------------------------- + + +@pytest.mark.parametrize("coverage", [ + OCCURRENCE_COVERAGE, + {"labels_in_text": ["1", "2"], "covered": ["1"], "missing": ["2"]}, + {"labels_in_text": [], "covered": [], "missing": []}, +]) +def test_audit_summary_carries_every_run_disclosure(tmp_path, coverage): + results = _results(coverage=coverage, scope=LIMITED_SCOPE) + case = _case(tmp_path, results=results) + + body = _text(_call("audit_summary", {"case": str(case)})) + + fired = run_disclosures(results, _manifest()) + assert {d.key for d in fired} >= { + "scope", "truncation", "converter", "source_fidelity", "source_identity", + "numbering", "labels_disputed", + } + for d in fired: + assert d.token in body, f"{d.key}: token {d.token!r} never reaches the MCP reader" + + +def test_audit_summary_states_the_counts_the_file_holds(tmp_path): + case = _case(tmp_path) + + summary = _json(_call("audit_summary", {"case": str(case)})) + + assert summary["counts"] == _results().counts() + assert summary["checker"] == "claude -p · claude-opus-5" + assert summary["references"] == {"total": 5, "available": 3} + assert summary["uncited_assertions"] == 1 + assert summary["case"] == str(case.resolve()) + assert str(case.resolve() / "out" / "report.md") in summary["reports"] + + +def test_a_limited_audit_is_stated_first_and_last(tmp_path): + """`CLAUDE.md`: the scope is stated with the caveats at the top AND at the + end of every format. The JSON a model reads has an end too.""" + case = _case(tmp_path, results=_results(scope=LIMITED_SCOPE)) + + summary = _json(_call("audit_summary", {"case": str(case)})) + listing = _json(_call("list_claims", {"case": str(case)})) + + assert summary["disclosures"][0]["key"] == "scope" + for result in (summary, listing): + assert list(result)[-1] == "limited" + assert SCOPE_TOKEN in result["limited"] + + +def test_a_whole_paper_audit_says_nothing_was_left_out(tmp_path): + case = _case(tmp_path) + + assert _json(_call("audit_summary", {"case": str(case)}))["limited"] is None + assert _json(_call("list_claims", {"case": str(case)}))["limited"] is None + + +def test_a_case_without_results_is_refused_rather_than_reported_empty(tmp_path): + """Absent is not zero. Empty counts for a case nobody checked would read as + an audit that found nothing.""" + case = tmp_path / "case" + case.mkdir() + + result = _call("audit_summary", {"case": str(case)}) + + assert result.is_error + assert "results.json" in _text(result) + assert "counts" not in _text(result) + + +def test_an_unreadable_results_file_is_refused_with_its_reason(tmp_path): + case = tmp_path / "case" + (case / "out").mkdir(parents=True) + (case / "out" / "results.json").write_text("{ half a file") + + result = _call("audit_summary", {"case": str(case)}) + + assert result.is_error + assert "unreadable" in _text(result) + + +def test_a_relative_case_path_is_resolved_and_said_back(tmp_path, monkeypatch): + """A host starts the server in a working directory of its own choosing, so a + relative path is echoed back resolved — the model can see where it looked.""" + _case(tmp_path) + monkeypatch.chdir(tmp_path) + + summary = _json(_call("audit_summary", {"case": "case"})) + + assert summary["case"] == str((tmp_path / "case").resolve()) + + +# -------------------------------------------------------------------------- +# list_claims and get_claim — every claim-level disclosure reaches the model +# -------------------------------------------------------------------------- + + +def test_list_claims_rows_carry_their_caveats(tmp_path): + case = _case(tmp_path) + + rows = _json(_call("list_claims", {"case": str(case)}))["claims"] + + assert [r["id"] for r in rows] == [1, 2, 3, 4] + assert [r["verdict"] for r in rows] == ["contradicted", "not_retrieved", "unchecked", + "supported"] + for c, row in zip(_results().claims, rows, strict=True): + for d in claim_disclosures(c, _manifest()): + assert any(d.token in line for line in row["caveats"]), (c.id, d.key) + + +def test_list_claims_filters_on_the_headline_verdict(tmp_path): + """Claim 1 has a supporting source, and its headline is the contradiction: + filtering on `supported` must not return it as if it were supported.""" + case = _case(tmp_path) + + listing = _json(_call("list_claims", {"case": str(case), "verdict": "supported"})) + + assert [r["id"] for r in listing["claims"]] == [4] + assert listing["claims_total"] == 4 + + +@pytest.mark.parametrize("claim", [ + _split_claim(), _unread_claim(), _withheld_claim(), + _paraphrase_claim(True), _paraphrase_claim(False), _paraphrase_claim(None), +], ids=["split", "unread", "withheld", "anchor-located", "anchor-not-located", "anchor-unknown"]) +def test_every_claim_disclosure_reaches_get_claim(tmp_path, claim): + case = _case(tmp_path, results=_results(claims=[claim])) + + body = _text(_call("get_claim", {"case": str(case), "claim_id": claim.id})) + + fired = claim_disclosures(claim, _manifest()) + fired += [d for j in claim.judgements for d in judgement_disclosures(j)] + assert fired, "a fixture that fires nothing proves nothing" + for d in fired: + assert d.token in body, f"claim {claim.id} {d.key}: {d.token!r} never reaches the model" + + +def test_get_claim_returns_each_source_judgement_with_its_rationale(tmp_path): + case = _case(tmp_path) + + detail = _json(_call("get_claim", {"case": str(case), "claim_id": 1})) + + assert detail["verdict"] == "contradicted" + assert detail["quote"] == "The model reached an AUC of 0.94 [3, 4]." + by_source = {j["source"]: j for j in detail["judgements"]} + assert by_source["beta-2021"]["rationale"] == "The source reports 0.77, not 0.94." + assert by_source["beta-2021"]["page"] == 5 + assert by_source["beta-2021"]["anchor_located"] is False + assert by_source["alpha-2020"]["verdict"] == "supported" + + +def test_get_claim_names_the_ids_that_exist_when_asked_for_one_that_does_not(tmp_path): + case = _case(tmp_path) + + result = _call("get_claim", {"case": str(case), "claim_id": 99}) + + assert result.is_error + assert "99" in _text(result) and "1–4" in _text(result) + + +# -------------------------------------------------------------------------- +# get_evidence — the crops themselves, never a picture the tool did not cut +# -------------------------------------------------------------------------- + + +def _images(result) -> list: + return [b for b in result.content if getattr(b, "type", "") == "image"] + + +def test_get_evidence_returns_the_headline_crops_with_their_anchor_caption(tmp_path): + """The headline is beta-2021's contradiction, whose passage crosses a break + and whose anchor was not boxed — two images, captioned as unboxed.""" + case = _case(tmp_path) + + result = _call("get_evidence", {"case": str(case), "claim_id": 1}) + + assert not result.is_error, _text(result) + images = _images(result) + assert len(images) == 2 + assert {i.mime_type for i in images} == {"image/png"} + caption = _text(result) + assert "beta-2021" in caption + for d in judgement_disclosures(_split_claim().judgements[1]): + assert d.token in caption + + +def test_get_evidence_for_a_named_source_shows_that_source(tmp_path): + case = _case(tmp_path) + + result = _call("get_evidence", {"case": str(case), "claim_id": 1, "source": "alpha-2020"}) + + assert len(_images(result)) == 1 + assert "alpha-2020" in _text(result) + + +def test_get_evidence_for_an_unread_source_says_nothing_was_read(tmp_path): + case = _case(tmp_path) + + result = _call("get_evidence", {"case": str(case), "claim_id": 2}) + + assert result.is_error + assert not _images(result) + assert "not retrieved" in _text(result) + + +def test_get_evidence_refuses_an_image_outside_the_case(tmp_path): + """`results.json` is a file anyone can edit. A crop path that resolves + outside `/out/` is refused, never followed.""" + elsewhere = tmp_path / "elsewhere.png" + _png(elsewhere) + claim = _paraphrase_claim(True) + claim.evidence_image = "../../elsewhere.png" + case = _case(tmp_path, results=_results(claims=[claim]), images=False) + + result = _call("get_evidence", {"case": str(case), "claim_id": 4}) + + assert result.is_error + assert not _images(result) + assert "outside" in _text(result) + + +def test_get_evidence_names_a_crop_that_was_never_written(tmp_path): + case = _case(tmp_path, images=False) + + result = _call("get_evidence", {"case": str(case), "claim_id": 4}) + + assert result.is_error + assert "claim_04_alpha-2020_p1.png" in _text(result) + + +# -------------------------------------------------------------------------- +# list_references — the manifest, as recorded +# -------------------------------------------------------------------------- + + +def test_list_references_carries_no_local_path(tmp_path): + """The viewer strips `pdf_path` for the same reason: where a PDF sits on the + auditor's disk is not a finding about the paper.""" + case = _case(tmp_path) + + body = _text(_call("list_references", {"case": str(case)})) + + assert "/private/home/someone" not in body + assert "pdf_path" not in body + + +def test_three_state_numbering_fields_keep_their_null(tmp_path): + """`[]` is a measurement and `null` is "never computed". Coercing one into + the other is the defect `CLAUDE.md` names twice.""" + measured = _manifest() + never = _manifest() + never.labels_uncomparable = None + never.numbering_corroborated = None + measured.numbering_corroborated = False + + for manifest, uncomparable, corroborated in ((measured, [], False), (never, None, None)): + case = _case(tmp_path / str(uncomparable), manifest=manifest) + numbering = _json(_call("list_references", {"case": str(case)}))["numbering"] + assert numbering["labels_uncomparable"] == uncomparable + assert numbering["corroborated"] is corroborated + assert numbering["verified"] is False + assert numbering["labels_disputed"] == ["9"] + + +def test_a_skipped_reference_says_why_it_was_skipped(tmp_path): + case = _case(tmp_path) + + listing = _json(_call("list_references", {"case": str(case), "status": "skipped"})) + + assert [r["label"] for r in listing["references"]] == ["8"] + assert listing["references"][0]["skipped_by"] == "claims" + assert listing["references"][0]["reason"] == "no selected claim cites it" + assert listing["by_status"]["skipped"] == 1 + assert listing["total"] == 5 + + +# -------------------------------------------------------------------------- +# list_gaps — the gap register is part of the result +# -------------------------------------------------------------------------- + + +def test_list_gaps_lists_every_register(tmp_path): + case = _case(tmp_path) + + gaps = _json(_call("list_gaps", {"case": str(case)})) + + assert [u["quote"] for u in gaps["uncited_assertions"]] == ["Most clinics now screen routinely."] + assert [(o["label"], o["status"]) for o in gaps["unreached_citations"]] == [ + ("8", "uncovered"), ("4", "uncertain"), + ] + assert gaps["unreached_labels"] == ["8"] + assert [c["id"] for c in gaps["unchecked_claims"]] == [2, 3] + assert "reached by an extracted claim" in gaps["coverage"] + + +def test_a_coverage_audit_that_never_ran_is_null_not_empty(tmp_path): + """No bracketed labels means the audit could not run — `null`, and the + caveat saying so — never an empty list that reads as nothing missed.""" + case = _case(tmp_path, results=_results( + coverage={"labels_in_text": [], "covered": [], "missing": []})) + + gaps = _json(_call("list_gaps", {"case": str(case)})) + + assert gaps["coverage"] is None + assert gaps["unreached_labels"] is None + assert gaps["unreached_citations"] is None + assert any(d["key"] == "coverage_caveat" for d in gaps["disclosures"]) + + +# -------------------------------------------------------------------------- +# get_scout — candidates, with the scout's own caveat +# -------------------------------------------------------------------------- + + +def test_get_scout_carries_its_caveat(tmp_path): + scout = ScoutResults( + paper_title="A paper", paper_year=2022, resolved_via="doi", paper_identity="unverified", + newer=[ScoutHit(title="A later paper", year=2024, doi="10.1/later", via="citing")], + ) + case = _case(tmp_path, scout=scout) + + result = _json(_call("get_scout", {"case": str(case)})) + + assert [h["title"] for h in result["newer"]] == ["A later paper"] + assert result["paper"]["identity"] == "unverified" + assert "absence from these lists proves nothing" in result["caveat"] + + +def test_get_scout_without_a_scan_is_refused(tmp_path): + case = _case(tmp_path) + + result = _call("get_scout", {"case": str(case)}) + + assert result.is_error + assert "scout.json" in _text(result) + + +# -------------------------------------------------------------------------- +# start_audit and audit_status — the one tool that spends, run as a job +# -------------------------------------------------------------------------- + + +def test_the_server_lists_nine_tools_and_marks_the_one_that_spends(): + tools = _tools() + + assert set(tools) == READ_TOOLS | {"start_audit", "audit_status"} + status = tools["audit_status"].annotations + assert status.read_only_hint is True and status.open_world_hint is False + # a host decides whether to ask before calling on these hints: the one tool + # that spends model calls, queries the network and rewrites a case folder + # must say all three + spend = tools["start_audit"].annotations + assert spend.read_only_hint is False + assert spend.destructive_hint is True + assert spend.open_world_hint is True + + +def test_every_backend_start_audit_offers_is_one_ingest_accepts(): + """The backend vocabulary has no constant to reuse, so the offer is checked + against the rule that enforces it: `resolve_backend` refuses anything else.""" + from papertrace.ingest import resolve_backend + + offered = _enum(_tools()["start_audit"].input_schema["properties"]["backend"]) + assert offered == ["auto", "docling", "pymupdf"] + for backend in offered: + resolve_backend(backend) + + +@pytest.fixture +def ready(monkeypatch, tmp_path): + """Everything start_audit checks before it spends, satisfied offline — and + never read from the machine: `_email` falls back to a SAVED CONFIG in the + developer's home, which is how a CLI test once passed locally and failed on + every CI python.""" + monkeypatch.setattr(check_mod, "claude_available", lambda: True) + monkeypatch.setenv("PAPERTRACE_EMAIL", "t@example.org") + monkeypatch.delenv("MANUSCRIPTAGENT_EMAIL", raising=False) + monkeypatch.setenv("PAPERTRACE_CONFIG", str(tmp_path / "no-saved-config.json")) + + +def _paper(tmp_path, name="paper.pdf") -> Path: + pdf = tmp_path / name + pdf.write_bytes(b"%PDF-1.4\n% never parsed: the pipeline is faked\n" + name.encode()) + return pdf + + +def _fake_pipeline(monkeypatch, body=None) -> list[dict]: + calls: list[dict] = [] + + def fake(**kw): + calls.append(kw) + if body is not None: + body(kw) + + monkeypatch.setattr(cli, "_run_pipeline", fake) + return calls + + +def _start(server, **args) -> dict: + return _json(_call("start_audit", {k: str(v) if isinstance(v, Path) else v + for k, v in args.items()}, server)) + + +def _await(server, case: str) -> dict: + return _json(_call("audit_status", {"case": case, "wait_seconds": 20}, server)) + + +def test_start_audit_refuses_a_manuscript_that_is_not_there(ready, monkeypatch, tmp_path): + calls = _fake_pipeline(monkeypatch) + + result = _call("start_audit", {"manuscript": str(tmp_path / "absent.pdf")}) + + assert result.is_error + assert "absent.pdf" in _text(result) + assert calls == [] + + +def test_start_audit_refuses_when_claude_is_not_on_its_path(ready, monkeypatch, tmp_path): + """A host starts the server with a PATH of its own choosing. Finding that + out after ingest has run would be finding it out late.""" + calls = _fake_pipeline(monkeypatch) + monkeypatch.setattr(check_mod, "claude_available", lambda: False) + + result = _call("start_audit", {"manuscript": str(_paper(tmp_path))}) + + assert result.is_error + assert "claude" in _text(result) and "PATH" in _text(result) + assert calls == [] + + +def test_start_audit_refuses_without_a_contact_email(ready, monkeypatch, tmp_path): + calls = _fake_pipeline(monkeypatch) + monkeypatch.delenv("PAPERTRACE_EMAIL") + + result = _call("start_audit", {"manuscript": str(_paper(tmp_path))}) + + assert result.is_error + assert "email" in _text(result).lower() + assert calls == [] + + +def test_start_audit_refuses_a_case_folder_holding_another_paper(ready, monkeypatch, tmp_path): + """One case folder per paper — `_guard_case`'s rule, applied before anything + is spent rather than after ingest and extraction already were.""" + calls = _fake_pipeline(monkeypatch) + case = tmp_path / "case" + case.mkdir() + RefManifest(manuscript="other.pdf", manuscript_sha256="f" * 64).to_json( + case / "refs_manifest.json") + + result = _call("start_audit", {"manuscript": str(_paper(tmp_path)), "case": str(case)}) + + assert result.is_error + assert "other.pdf" in _text(result) + assert calls == [] + + +def test_the_job_hands_the_pipeline_every_parameter(ready, monkeypatch, tmp_path): + import inspect + + params = set(inspect.signature(cli._run_pipeline).parameters) + calls = _fake_pipeline(monkeypatch) + pdf = _paper(tmp_path) + own = _paper(tmp_path, "paper-supplement.pdf") + provided = tmp_path / "my_pdfs" + provided.mkdir() + server = mcp_server.build_server() + + started = _start(server, manuscript=pdf, case=tmp_path / "case", model="claude-opus-5", + backend="pymupdf", scout=False, doi="10.1/x", formats=["viewer"], + provided=provided, supplements=[str(own)], llm_refs=False, + max_claims=3, max_sources=2) + assert _await(server, started["case"])["state"] == "finished" + + assert calls == [{ + "manuscript": pdf.resolve(), "case": (tmp_path / "case").resolve(), + "provided": provided.resolve(), "email": "t@example.org", "model": "claude-opus-5", + "png": False, "backend": "pymupdf", "with_scout": False, "doi": "10.1/x", + "formats": ["viewer"], "supplement": [own.resolve()], "llm_refs": False, + "max_claims": 3, "max_sources": 2, + }] + assert set(calls[0]) == params, "a pipeline parameter the job never decides on" + + +def test_the_job_runs_the_real_pipeline_body_and_keeps_what_it_prints( + ready, monkeypatch, tmp_path, capfd +): + """The other job tests replace `_run_pipeline` whole. Here only its seven + stages are stubbed, so the real body runs inside the job — email, case, + guard, the case folder's own .gitignore, the banner, the stage order and + the closing line — and anything it printed around `cli.console` would + reach the process's real stdout, which this test reads.""" + order = [] + for name in ("_ingest_pipeline", "_extract_pipeline", "_refs_pipeline", "scout", + "_check_pipeline", "highlight", "_report_pipeline"): + monkeypatch.setattr(cli, name, lambda _n=name, **kw: order.append(_n)) + monkeypatch.setattr(cli, "_detected_doi", lambda m: None) + server = mcp_server.build_server() + case = tmp_path / "case" + + status = _await(server, _start(server, manuscript=_paper(tmp_path), case=case, + formats=["viewer"])["case"]) + + assert status["state"] == "finished", status + assert order == ["_ingest_pipeline", "_extract_pipeline", "_refs_pipeline", "scout", + "_check_pipeline", "highlight", "_report_pipeline"] + assert (case / ".gitignore").read_text() == "*\n" + assert any("PaperTrace" in line for line in status["log"]), "the banner is in the log" + assert any("report_viewer.html" in line for line in status["log"]) + out, _ = capfd.readouterr() + assert out == "", "the real pipeline printed around the captured console" + +def test_start_audit_defaults_are_papertrace_runs(ready, monkeypatch, tmp_path): + """No case named: beside the paper, named after it, as `papertrace run` + does. Every other default is the CLI's own.""" + calls = _fake_pipeline(monkeypatch) + pdf = _paper(tmp_path) + server = mcp_server.build_server() + + started = _start(server, manuscript=pdf) + _await(server, started["case"]) + + assert started["case"] == str((tmp_path / "paper").resolve()) + assert calls[0] | {"manuscript": None, "case": None} == { + "manuscript": None, "case": None, "provided": None, "email": "t@example.org", + "model": None, "png": False, "backend": "auto", "with_scout": True, "doi": None, + "formats": [], "supplement": [], "llm_refs": True, "max_claims": None, + "max_sources": None, + } + + +class _Terminal: + """A stream that says it is a terminal — what `_interactive()` looks for.""" + + def __init__(self): + self.written = [] + + def isatty(self): + return True + + def write(self, s): + self.written.append(s) + return len(s) + + def flush(self): + pass + + def readline(self): + return "1\n" # a person answering the first menu, were one asked + + +def test_inside_the_job_nobody_can_be_asked_and_no_old_model_lingers( + ready, monkeypatch, tmp_path +): + """Three things the job guarantees, observed from inside it. + + A terminal on both streams and no `CI`: outside the job, `_interactive()` + would be True here, and a disputed label would be put to a person who is + not there. Inside it, nobody can be asked — a prompt reads end-of-file. + + A model left behind by an earlier audit in this process is gone, so a run + whose judging makes no calls cannot report that model as its judge. + + The pipeline's console lands in the job's log, never on stdout, which + under stdio is the protocol's own wire. + """ + terminal_in, terminal_out = _Terminal(), _Terminal() + monkeypatch.setattr(sys, "stdin", terminal_in) + monkeypatch.setattr(sys, "stdout", terminal_out) + monkeypatch.delenv("CI", raising=False) + assert cli._interactive(), "the fixture must look like a terminal, or this proves nothing" + monkeypatch.setattr(ask_mod, "_MODELS", {ask_mod.SITE_CHECK: "claude-haiku-4-5"}) + seen = {} + + def body(kw): + seen["interactive"] = cli._interactive() + try: + seen["answer"] = input() + except EOFError: + seen["answer"] = "end of file" + seen["model"] = ask_mod.model_for(ask_mod.SITE_CHECK) + cli.console.print("[green]✓[/green] said by the pipeline") + + _fake_pipeline(monkeypatch, body) + console = cli.console + server = mcp_server.build_server() + + status = _await(server, _start(server, manuscript=_paper(tmp_path))["case"]) + + assert status["state"] == "finished", status + assert seen == {"interactive": False, "answer": "end of file", "model": None} + assert "✓ said by the pipeline" in status["log"] + assert not any("said by the pipeline" in s for s in terminal_out.written) + assert cli.console is console, "the console must be handed back when the job ends" + + +def test_a_pipeline_that_stops_is_failed_with_the_reason_it_printed(ready, monkeypatch, tmp_path): + def body(kw): + cli.console.print("[red]refs_manifest.json not found[/red] — run `papertrace refs` first") + raise typer.Exit(1) + + _fake_pipeline(monkeypatch, body) + server = mcp_server.build_server() + + status = _await(server, _start(server, manuscript=_paper(tmp_path))["case"]) + + assert status["state"] == "failed" + assert "refs_manifest.json not found" in status["error"] + + +def test_an_unexpected_exception_is_failed_with_its_type_and_message( + ready, monkeypatch, tmp_path +): + """In SDK v2 an exception escaping a tool reaches the model as a bare + `Error executing tool`. A job records its reason instead of losing it.""" + def body(kw): + raise RuntimeError("the layout models could not be loaded") + + _fake_pipeline(monkeypatch, body) + server = mcp_server.build_server() + + status = _await(server, _start(server, manuscript=_paper(tmp_path))["case"]) + + assert status["state"] == "failed" + assert status["error"] == "RuntimeError: the layout models could not be loaded" + + +def test_a_second_audit_is_refused_while_one_runs(ready, monkeypatch, tmp_path): + """The console, the model record and stdin are process-global: two audits + sharing them would interleave logs and misattribute models.""" + import threading + + gate = threading.Event() + _fake_pipeline(monkeypatch, lambda kw: gate.wait(20)) + server = mcp_server.build_server() + first = _start(server, manuscript=_paper(tmp_path), case=tmp_path / "first") + + try: + refused = _call("start_audit", {"manuscript": str(_paper(tmp_path, "other.pdf")), + "case": str(tmp_path / "second")}, server) + assert refused.is_error + assert first["case"] in _text(refused) + finally: + gate.set() + assert _await(server, first["case"])["state"] == "finished" + + # and the slot is free again once it has ended + second = _start(server, manuscript=_paper(tmp_path, "other.pdf"), case=tmp_path / "second") + assert _await(server, second["case"])["state"] == "finished" + + +def test_read_tools_refuse_the_case_whose_audit_is_running(ready, monkeypatch, tmp_path): + """The results.json on disk then belongs to the previous run, or is half + written. Presenting it as this run's would be the silent failure.""" + import threading + + gate = threading.Event() + _fake_pipeline(monkeypatch, lambda kw: gate.wait(20)) + case = _case(tmp_path) + server = mcp_server.build_server() + RefManifest(manuscript="paper.pdf").to_json(case / "refs_manifest.json") + started = _start(server, manuscript=_paper(tmp_path), case=case) + + try: + for tool, args in READ_CALLS: + result = _call(tool, {"case": started["case"], **args}, server) + assert result.is_error, tool + assert "running" in _text(result), tool + finally: + gate.set() + _await(server, started["case"]) + assert not _call("audit_summary", {"case": started["case"]}, server).is_error + + +def test_audit_status_waits_for_the_end_and_no_longer(ready, monkeypatch, tmp_path): + import time + + _fake_pipeline(monkeypatch, lambda kw: time.sleep(0.3)) + server = mcp_server.build_server() + started = _start(server, manuscript=_paper(tmp_path)) + + t0 = time.monotonic() + status = _json(_call("audit_status", {"case": started["case"], "wait_seconds": 15}, server)) + + assert status["state"] == "finished" + assert time.monotonic() - t0 < 10 + # under the 60 s a TypeScript-SDK host allows a request, by schema + too_long = _call("audit_status", {"case": started["case"], "wait_seconds": 51}, server) + assert too_long.is_error + + +def test_a_finished_status_points_at_the_summary_and_carries_no_counts( + ready, monkeypatch, tmp_path +): + """A count without its disclosures is what every reader here is protected + from, so the status says where the result is rather than quoting it.""" + _fake_pipeline(monkeypatch) + server = mcp_server.build_server() + + status = _await(server, _start(server, manuscript=_paper(tmp_path))["case"]) + + assert "counts" not in status + assert "audit_summary" in status["next"] + + +def test_the_status_of_a_case_this_server_never_ran_says_so(tmp_path): + case = _case(tmp_path) + + status = _json(_call("audit_status", {"case": str(case)})) + + assert status["state"] == "not_started" + assert "MCP server" in status["next"] + + +# -------------------------------------------------------------------------- +# `papertrace mcp` — the command a host launches, over real stdio +# -------------------------------------------------------------------------- + +SRC = str(Path(__file__).resolve().parent.parent / "src") + + +def test_papertrace_mcp_writes_nothing_to_stdout_when_its_stdin_closes(tmp_path): + """Under stdio, stdout IS the protocol. A banner, a warning or a stray print + before serving begins lands on the wire and corrupts the first message, and + no in-memory test can see it — only a real process can.""" + import os + import subprocess + + proc = subprocess.run( + [sys.executable, "-m", "papertrace", "mcp"], + stdin=subprocess.DEVNULL, capture_output=True, timeout=120, + env={**os.environ, "PYTHONPATH": SRC, + "PAPERTRACE_CONFIG": str(tmp_path / "no-saved-config.json")}, + ) + + assert proc.returncode == 0, proc.stderr.decode()[-2000:] + assert proc.stdout == b"", proc.stdout[:500] + + +def test_a_host_lists_the_tools_and_reads_a_case_over_real_stdio(tmp_path): + """The whole path a host takes: launch the command, speak MCP over its + pipes, read an audit. The in-memory client skips exactly this.""" + from mcp import StdioServerParameters + + case = _case(tmp_path) + params = StdioServerParameters( + command=sys.executable, args=["-m", "papertrace", "mcp"], + env={"PYTHONPATH": SRC, "PAPERTRACE_CONFIG": str(tmp_path / "no-saved-config.json")}, + ) + + async def go(client): + names = {t.name for t in (await client.list_tools()).tools} + return names, await client.call_tool("audit_summary", {"case": str(case)}) + + names, summary = _session(params, go) + + assert names == READ_TOOLS | {"start_audit", "audit_status"} + assert _json(summary)["counts"] == _results().counts() + + +def test_without_the_sdk_papertrace_mcp_says_how_to_install_it(monkeypatch, capsys): + """On stderr, and nothing on stdout — a host reading the wire would take + even an error message for a malformed protocol message.""" + # every cached `mcp.*`, not only `mcp`: an import finds a cached submodule + # without consulting its parent, so masking the package alone hides nothing + for name in [n for n in sys.modules if n == "mcp" or n.startswith("mcp.")]: + monkeypatch.setitem(sys.modules, name, None) + monkeypatch.delitem(sys.modules, "papertrace.mcp_server", raising=False) + monkeypatch.setattr("importlib.metadata.version", _no_distribution) + + with pytest.raises(typer.Exit) as stop: + cli.mcp_command() + + out, err = capsys.readouterr() + assert stop.value.exit_code == 2 + assert out == "" + assert "pip install 'papertrace[mcp]'" in err + + +def _no_distribution(name): + from importlib.metadata import PackageNotFoundError + + raise PackageNotFoundError(name) + + +# -------------------------------------------------------------------------- +# review findings — each a way the server could present a case as a result +# it is not, or a verdict without what qualifies it +# -------------------------------------------------------------------------- + + +def _record(case: Path) -> dict: + return json.loads((case / "mcp_audit.json").read_text()) + + +def _left_record(case: Path, state: str, *, age: float = 0, **fields) -> Path: + """A record as another server process left it, last refreshed `age` s ago.""" + import time + + path = case / "mcp_audit.json" + path.write_text(json.dumps({ + "schema": "mcp_audit/1", "state": state, "manuscript": "/papers/paper.pdf", + "started": "2026-09-26T10:00:00+00:00", "ended": None, "error": "", "scout": True, + **fields, + })) + then = time.time() - age + os.utime(path, (then, then)) + return path + + +def _age_file(path: Path, seconds: float) -> None: + """Move a file's mtime by `seconds` — positive is later, negative earlier.""" + import time + + t = time.time() + seconds + os.utime(path, (t, t)) + + +def test_after_a_failed_audit_nothing_on_disk_is_served_as_its_result( + ready, monkeypatch, tmp_path +): + """A rerun can rewrite the manifest and die before `check`, leaving one + run's references beside another run's verdicts. The case is refused with + the failure, never served as a result.""" + def body(kw): + RefManifest(manuscript="paper.pdf").to_json(Path(kw["case"]) / "refs_manifest.json") + raise RuntimeError("claude -p failed: not logged in") + + case = _case(tmp_path) # an earlier, complete audit + RefManifest(manuscript="paper.pdf").to_json(case / "refs_manifest.json") + _fake_pipeline(monkeypatch, body) + server = mcp_server.build_server() + started = _start(server, manuscript=_paper(tmp_path), case=case) + assert _await(server, started["case"])["state"] == "failed" + + for tool, args in READ_CALLS: + result = _call(tool, {"case": started["case"], **args}, server) + assert result.is_error, tool + assert "failed" in _text(result) and "not logged in" in _text(result), tool + + +def test_an_audit_its_server_never_finished_is_refused_and_reported_interrupted(tmp_path): + """A host that stops the server stops the audit mid-run. Whatever it wrote + is on disk, and the only thing that says it is incomplete is the record the + job wrote when it started — read here by a server that never ran it.""" + case = _case(tmp_path) + _left_record(case, "running", age=120) + server = mcp_server.build_server() + + status = _json(_call("audit_status", {"case": str(case)}, server)) + summary = _call("audit_summary", {"case": str(case)}, server) + + assert status["state"] == "interrupted" + assert status["started"] == "2026-09-26T10:00:00+00:00" + assert summary.is_error and "did not finish" in _text(summary) + + +def test_the_record_outlives_the_server_that_wrote_it(ready, monkeypatch, tmp_path): + jsonschema = pytest.importorskip("jsonschema") + _fake_pipeline(monkeypatch) + first = mcp_server.build_server() + case = _await(first, _start(first, manuscript=_paper(tmp_path), scout=False)["case"])["case"] + + status = _json(_call("audit_status", {"case": case}, mcp_server.build_server())) + + assert status["state"] == "finished" + assert "not kept" in status["next"], "a log that did not survive must say so" + record = _record(Path(case)) + assert record["state"] == "finished" and record["scout"] is False + jsonschema.validate(record, json.loads((ROOT / "schemas" / "mcp_audit.schema.json").read_text())) + + +def test_a_claim_read_on_its_own_still_carries_the_runs_caveats(tmp_path): + """A host can call get_claim without ever calling audit_summary. The source + behind claim 1's headline is one whose identity nobody confirmed — the + reports say so above every verdict, so a verdict read alone says it too.""" + results = _results(scope=LIMITED_SCOPE) + case = _case(tmp_path, results=results) + + for tool, args in (("get_claim", {"claim_id": 1}), ("list_claims", {})): + body = _text(_call(tool, {"case": str(case), **args})) + for d in run_disclosures(results, _manifest()): + assert d.token in body, (tool, d.key) + + +def test_the_evidence_caption_carries_every_caveat_on_its_verdict(tmp_path): + """The crop is shown under its verdict, so the verdict's caveats come with + it — here [6] sits past where the numbering stopped being confirmed, and a + crop captioned without that would show the wrong paper as evidence.""" + claim = _paraphrase_claim(True) + claim.refs = ["6"] + results = _results(claims=[claim]) + case = _case(tmp_path, results=results) + + caption = _text(_call("get_evidence", {"case": str(case), "claim_id": 4})) + + fired = claim_disclosures(claim, _manifest()) + run_disclosures(results, _manifest()) + assert "claim_numbering" in {d.key for d in fired} + for d in fired: + assert d.token in caption, d.key + + +def test_start_audit_will_not_put_an_audit_wherever_the_host_started_it( + ready, monkeypatch, tmp_path +): + """`papertrace run` keeps the audit in the working directory when the + paper's folder is read-only — a directory the person chose by standing in + it. A host chose this server's, so start_audit asks for a case instead.""" + calls = _fake_pipeline(monkeypatch) + pdf = _paper(tmp_path) + access = os.access + monkeypatch.setattr(os, "access", lambda p, mode: Path(p) != pdf.parent and access(p, mode)) + + result = _call("start_audit", {"manuscript": str(pdf)}) + + assert result.is_error + assert str(pdf.parent) in _text(result) and "`case`" in _text(result) + assert calls == [] + + +def test_what_the_checks_said_before_the_audit_is_kept_in_its_log(ready, monkeypatch, tmp_path): + """A legacy case folder is accepted with a warning that its identity was + compared by name only. That warning is printed before the job starts, and + it belongs in the job's log, not a buffer nobody reads.""" + _fake_pipeline(monkeypatch) + case = tmp_path / "case" + case.mkdir() + RefManifest(manuscript="paper.pdf").to_json(case / "refs_manifest.json") + server = mcp_server.build_server() + + status = _await(server, _start(server, manuscript=_paper(tmp_path), case=case)["case"]) + + assert any("predates content hashing" in line for line in status["log"]) + + +def test_an_incomplete_scan_says_so_in_its_caveat(tmp_path): + scout = ScoutResults(paper_title="A paper", paper_identity="confirmed", + error="Europe PMC timed out") + case = _case(tmp_path, scout=scout) + + assert "Europe PMC timed out" in _json(_call("get_scout", {"case": str(case)}))["caveat"] + + +def test_a_scan_the_latest_audit_did_not_run_is_not_served_as_its_own( + ready, monkeypatch, tmp_path +): + """An audit run without the scout leaves an earlier run's scout.json on + disk, anchored to whatever paper and DOI that run used.""" + case = _case(tmp_path, scout=ScoutResults(paper_title="A paper", paper_identity="confirmed")) + RefManifest(manuscript="paper.pdf").to_json(case / "refs_manifest.json") + _fake_pipeline(monkeypatch) + server = mcp_server.build_server() + _await(server, _start(server, manuscript=_paper(tmp_path), case=case, scout=False)["case"]) + + result = _call("get_scout", {"case": str(case)}, server) + + assert result.is_error + assert "without the scout" in _text(result) + + +def test_an_sdk_too_old_for_the_server_gets_the_install_line_too(monkeypatch, capsys): + """A 1.x SDK — other tools install one — has no `MCPServer`, and fails with + an ImportError that is not a ModuleNotFoundError.""" + for name in [n for n in sys.modules if n == "mcp" or n.startswith("mcp.")]: + monkeypatch.delitem(sys.modules, name) + old = types.ModuleType("mcp") + old.__path__ = [] + monkeypatch.setitem(sys.modules, "mcp", old) + monkeypatch.setitem(sys.modules, "mcp.server", types.ModuleType("mcp.server")) + monkeypatch.delitem(sys.modules, "papertrace.mcp_server", raising=False) + monkeypatch.setattr("importlib.metadata.version", lambda name: "1.28.1") + + with pytest.raises(typer.Exit) as stop: + cli.mcp_command() + + out, err = capsys.readouterr() + assert stop.value.exit_code == 2 and out == "" + assert "pip install 'papertrace[mcp]'" in err + + +STRUCTURED = { + "audit_summary": "AuditSummary", "list_claims": "ClaimList", "get_claim": "ClaimDetail", + "list_references": "ReferenceList", "list_gaps": "Gaps", "get_scout": "ScoutOut", + "audit_status": "AuditStatus", +} + + +def test_every_tool_output_matches_its_schema_in_schemas(tmp_path): + """Gate 2: what the server publishes is a contract, and `schemas/` is where + this repository's contracts live. Each output validates against it, and + each schema names exactly the keys its tool returns — a field added to one + and not the other turns this red.""" + jsonschema = pytest.importorskip("jsonschema") + published = json.loads((ROOT / "schemas" / "mcp_tools.schema.json").read_text()) + case = _case(tmp_path, results=_results(scope=LIMITED_SCOPE), + scout=ScoutResults(paper_title="A paper", paper_identity="unverified")) + args = {"get_claim": {"claim_id": 1}} + + for tool, shape in STRUCTURED.items(): + out = _json(_call(tool, {"case": str(case), **args.get(tool, {})})) + jsonschema.validate(out, {**published, "$ref": f"#/$defs/{tool}"}) + declared = set(published["$defs"][tool]["properties"]) + assert declared == set(getattr(mcp_server, shape).__annotations__), tool + + +# -------------------------------------------------------------------------- +# second review pass — the record says only what it measured +# -------------------------------------------------------------------------- + + +def test_a_named_sources_crop_carries_only_its_own_anchor_state(tmp_path): + """Claim 1's headline source (beta-2021) was not boxed; alpha-2020 was. + alpha's crop captioned with beta's anchor state would put two + contradictory statements under one picture.""" + case = _case(tmp_path) + + caption = _text(_call("get_evidence", {"case": str(case), "claim_id": 1, + "source": "alpha-2020"})) + + assert ANCHOR_LOCATED_TOKEN in caption + assert ANCHOR_NOT_LOCATED_TOKEN not in caption + + +def test_an_audit_another_server_is_running_is_neither_read_nor_restarted( + ready, monkeypatch, tmp_path +): + """Two hosts can each launch their own server. A record refreshed moments + ago is a live audit in another process — not an interrupted one — and a + second pipeline in the same folder would be the mixing `_guard_case` + exists to prevent.""" + case = _case(tmp_path) + RefManifest(manuscript="paper.pdf").to_json(case / "refs_manifest.json") + _left_record(case, "running", age=1) + calls = _fake_pipeline(monkeypatch) + server = mcp_server.build_server() + + status = _json(_call("audit_status", {"case": str(case)}, server)) + summary = _call("audit_summary", {"case": str(case)}, server) + start = _call("start_audit", {"manuscript": str(_paper(tmp_path)), "case": str(case)}, server) + + assert status["state"] == "running" and status["log"] is None + assert summary.is_error and "another server process" in _text(summary) + assert start.is_error and "another server process" in _text(start) + assert calls == [] + + +def test_an_interrupted_audit_can_be_started_again(ready, monkeypatch, tmp_path): + case = _case(tmp_path) + RefManifest(manuscript="paper.pdf").to_json(case / "refs_manifest.json") + _left_record(case, "running", age=600) + _fake_pipeline(monkeypatch) + server = mcp_server.build_server() + + status = _await(server, _start(server, manuscript=_paper(tmp_path), case=case)["case"]) + + assert status["state"] == "finished" + assert _record(case)["state"] == "finished" + + +def test_the_job_keeps_its_record_fresh_while_it_runs(ready, monkeypatch, tmp_path): + """Freshness is the measurement that tells a live audit from an abandoned + one, so the job refreshes its record for as long as it runs.""" + import time + + monkeypatch.setattr(mcp_server, "HEARTBEAT_SECONDS", 0.05) + seen = {} + + def body(kw): + record = Path(kw["case"]) / "mcp_audit.json" + _age_file(record, -600) + time.sleep(0.5) + seen["age"] = time.time() - record.stat().st_mtime + + _fake_pipeline(monkeypatch, body) + server = mcp_server.build_server() + + _await(server, _start(server, manuscript=_paper(tmp_path))["case"]) + + assert seen["age"] < 5, seen + + +def test_a_label_audit_that_never_ran_is_null_even_when_occurrences_were_counted(tmp_path): + """Without clean.md the label-level lists are empty because nothing read + them, while the occurrence audit ran from the source map. `[]` would say + every cited label was reached.""" + occurrences_only = { + "labels_in_text": [], "covered": [], "missing": [], + "occurrences": {**OCCURRENCE_COVERAGE["occurrences"]}, + } + case = _case(tmp_path, results=_results(coverage=occurrences_only)) + + gaps = _json(_call("list_gaps", {"case": str(case)})) + + assert gaps["unreached_labels"] is None + assert [o["label"] for o in gaps["unreached_citations"]] == ["8", "4"] + + +def test_the_status_names_the_manuscript_the_same_way_before_and_after_a_restart( + ready, monkeypatch, tmp_path +): + """A published field has one meaning: the audited PDF's absolute path.""" + _fake_pipeline(monkeypatch) + pdf = _paper(tmp_path) + server = mcp_server.build_server() + running = _start(server, manuscript=pdf) + finished = _await(server, running["case"]) + + later = _json(_call("audit_status", {"case": running["case"]}, mcp_server.build_server())) + + assert running["manuscript"] == finished["manuscript"] == later["manuscript"] + assert later["manuscript"] == str(pdf.resolve()) + + +def test_a_later_run_that_finished_the_judging_supersedes_a_failed_audit( + ready, monkeypatch, tmp_path +): + """The failure was the last thing an MCP server saw — not the last thing + that happened. A results.json written after it is a later run's judging + of the manifest now on disk, so the case reads again, in this server too.""" + def body(kw): + raise RuntimeError("claude -p failed: not logged in") + + case = _case(tmp_path) + RefManifest(manuscript="paper.pdf").to_json(case / "refs_manifest.json") + _fake_pipeline(monkeypatch, body) + server = mcp_server.build_server() + _await(server, _start(server, manuscript=_paper(tmp_path), case=case)["case"]) + assert _call("audit_summary", {"case": str(case)}, server).is_error + + _results().to_json(case / "out" / "results.json") # e.g. `papertrace check`, later + _age_file(case / "out" / "results.json", +5) + + assert not _call("audit_summary", {"case": str(case)}, server).is_error + + +def test_a_scan_written_after_an_audit_without_the_scout_is_served(tmp_path): + case = _case(tmp_path, scout=ScoutResults(paper_title="A paper", paper_identity="confirmed")) + _left_record(case, "finished", age=60, ended="2026-09-26T10:05:00+00:00", scout=False) + _age_file(case / "out" / "scout.json", 0) # `papertrace scout`, after that audit + + result = _call("get_scout", {"case": str(case)}) + + assert not result.is_error, _text(result) + + +def test_a_broken_import_inside_a_current_sdk_keeps_its_traceback(monkeypatch): + """With a 2.x SDK installed, an import that still fails is a real fault. The + install line would contradict itself ("needs version 2 — found 2.x") and + hide the one thing that says what broke.""" + for name in [n for n in sys.modules if n == "mcp" or n.startswith("mcp.")]: + monkeypatch.delitem(sys.modules, name) + broken = types.ModuleType("mcp") + broken.__path__ = [] + monkeypatch.setitem(sys.modules, "mcp", broken) + monkeypatch.setitem(sys.modules, "mcp.server", types.ModuleType("mcp.server")) + monkeypatch.delitem(sys.modules, "papertrace.mcp_server", raising=False) + monkeypatch.setattr("importlib.metadata.version", lambda name: "2.3.0") + + with pytest.raises(ImportError, match="MCPServer"): + cli.mcp_command() + + +def test_a_recorded_failure_does_not_send_the_host_to_a_log_that_was_not_kept(tmp_path): + case = _case(tmp_path) + _left_record(case, "failed", age=60, ended="2026-09-26T10:05:00+00:00", + error="claude -p failed: not logged in") + + status = _json(_call("audit_status", {"case": str(case)})) + + assert status["state"] == "failed" and status["log"] is None + assert "the log" not in status["next"] + assert "not kept" in status["next"] + + +@pytest.mark.parametrize("field", [{"scout": "false"}, {"schema": "mcp_audit/2"}], + ids=["scout-not-a-bool", "a-later-schema"]) +def test_a_record_this_server_cannot_vouch_for_is_refused(tmp_path, field): + """A record read under the wrong rules is a guess. "false" is not False, + and a v2 record may mean something v1 does not.""" + case = _case(tmp_path) + _left_record(case, "finished", age=60, ended="2026-09-26T10:05:00+00:00", **field) + + result = _call("audit_summary", {"case": str(case)}) + + assert result.is_error and "unreadable" in _text(result) + + +def test_every_status_a_record_can_produce_matches_the_schema(tmp_path): + """The status schema describes what the server returns from its own job AND + from a record another server left — not only the shape one test happened + to build.""" + jsonschema = pytest.importorskip("jsonschema") + published = json.loads((ROOT / "schemas" / "mcp_tools.schema.json").read_text()) + cases = { + "not_started": {}, + "running": {"state": "running", "age": 1}, + "interrupted": {"state": "running", "age": 600}, + "finished": {"state": "finished", "age": 60, "ended": "2026-09-26T10:05:00+00:00"}, + "failed": {"state": "failed", "age": 60, "ended": "2026-09-26T10:05:00+00:00", + "error": "RuntimeError: boom"}, + } + + for expected, record in cases.items(): + case = _case(tmp_path / expected) + if record: + _left_record(case, record.pop("state"), **record) + status = _json(_call("audit_status", {"case": str(case)})) + assert status["state"] == expected + jsonschema.validate(status, {**published, "$ref": "#/$defs/audit_status"}) + + +def test_the_published_timings_are_the_ones_the_server_uses(): + """The schemas state the refresh and staleness windows in prose, because a + reader of a record has to know them — so a changed constant must not leave + the contract quoting the old one.""" + record = json.loads((ROOT / "schemas" / "mcp_audit.schema.json").read_text()) + tools = json.loads((ROOT / "schemas" / "mcp_tools.schema.json").read_text()) + state = tools["$defs"]["audit_status"]["properties"]["state"]["description"] + + for prose in (record["description"], state): + assert f"every {mcp_server.HEARTBEAT_SECONDS:g} s" in prose + assert f"{mcp_server.STALE_AFTER_SECONDS:g} s" in prose diff --git a/tests/test_packaging.py b/tests/test_packaging.py index 5fc2e4c..0d2b80f 100644 --- a/tests/test_packaging.py +++ b/tests/test_packaging.py @@ -313,11 +313,30 @@ def test_docling_is_a_base_dependency(): assert "docling" in names, "docling must be a base dependency, not an extra" +def test_the_mcp_extra_exists_and_ci_installs_it(): + """`papertrace mcp` needs the MCP SDK, so the SDK is an extra, not a base + dependency. `[dev]` — exactly what CI installs — carries the same + requirement: `tests/test_mcp_server.py` skips without the SDK, and a module + that skips in CI is a module nobody runs. This file's own history is the + 3.10 cell that silently skipped all 26 of its tests while CI was green. + + `<3` because the server is written against the SDK's 2.x API (`MCPServer`, + `Client(server)`), and the SDK's versioning policy holds breaking changes + for a major. `[full]` is the everything-install the quick start uses.""" + extras = PYPROJECT["project"]["optional-dependencies"] + sdk = [r for r in extras.get("mcp", []) if re.split(r"[<>=!~;\[ ]", r)[0] == "mcp"] + assert sdk, "no `mcp` extra carrying the MCP SDK" + assert "<3" in sdk[0], f"{sdk[0]!r} admits a new major of the SDK" + for name in ("dev", "full"): + assert sdk[0] in extras[name], f"[{name}] must carry {sdk[0]!r} as the [mcp] extra does" + + def test_the_docling_and_full_extras_still_resolve(): """`pip install 'papertrace[docling]'` and `[full]` appear throughout the 0.4.x docs and in other people's notes. Deleting the extras would turn those into an install error, so they stay as aliases — `[docling]` empty - because it is now redundant, `[full]` keeping only playwright.""" + because it is now redundant, `[full]` keeping playwright (and, since the MCP + server, the SDK — asserted above).""" extras = PYPROJECT["project"]["optional-dependencies"] assert "docling" in extras and extras["docling"] == [] full = [d.split(">")[0].strip() for d in extras["full"]] diff --git a/tests/test_pipeline_split.py b/tests/test_pipeline_split.py index b66b096..1b62c11 100644 --- a/tests/test_pipeline_split.py +++ b/tests/test_pipeline_split.py @@ -118,6 +118,44 @@ def test_run_hands_report_its_formats_rather_than_an_option_info(monkeypatch, tm assert seen["formats"] == ["md"], "run() must pass formats through explicitly" +def test_run_pipeline_rejects_a_positional_call(): + with pytest.raises(TypeError): + cli._run_pipeline(Path("whatever.pdf")) + + +def test_run_command_delegates_every_parameter_to_the_pipeline_function(monkeypatch, tmp_path): + """`run` gains a second caller — the MCP server's audit job — which is the + moment `cli.py`'s own comment names for splitting a stage: a caller that + omits an option hands a Typer command an `OptionInfo` for it, and a plain + keyword-only function has ordinary defaults instead. + + The signature comparison is what keeps the adapter whole: a parameter added + to one and not the other turns this red rather than reaching the pipeline as + its default without anyone deciding that.""" + import inspect + + # read before the patch below replaces it with a `**kw` lambda + pipeline_params = set(inspect.signature(cli._run_pipeline).parameters) + seen = {} + monkeypatch.setattr(cli, "_run_pipeline", lambda **kw: seen.update(kw)) + pdf = tmp_path / "p.pdf" + pdf.write_bytes(b"%PDF-1.4\n") + own = tmp_path / "p-supplement.pdf" + + cli.run(manuscript=pdf, case=tmp_path, provided=None, email="t@example.org", + model="claude-opus-5", png=False, backend="pymupdf", with_scout=False, + doi="10.1/x", formats=["viewer"], supplement=[own], llm_refs=False, + max_claims=3, max_sources=2) + + assert seen == { + "manuscript": pdf, "case": tmp_path, "provided": None, "email": "t@example.org", + "model": "claude-opus-5", "png": False, "backend": "pymupdf", "with_scout": False, + "doi": "10.1/x", "formats": ["viewer"], "supplement": [own], "llm_refs": False, + "max_claims": 3, "max_sources": 2, + } + assert set(inspect.signature(cli.run).parameters) == pipeline_params + + def test_check_pipeline_rejects_a_positional_call(): with pytest.raises(TypeError): cli._check_pipeline(Path("some-case"))