From fa0881c296ecfbd5b1212d1402550dae07026808 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 26 Sep 2026 12:16:25 +0000 Subject: [PATCH 01/13] docs: design, plan and ADR for an MCP server MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The roadmap's "drive PaperTrace as a tool from any MCP-capable client", designed before any code. Nine tools over stdio behind an optional extra: seven read the case folder with the same disclosures every report format carries, and one pair runs the existing pipeline as a background job. The spec walks each CLAUDE.md rule across to the two things that are new here — a model as the reader, and a process that outlives many audits — and records what was rejected: a subprocess per audit (a second spawning file in src/), a blocking run (a TypeScript-SDK host times out at 60 s), and elicitation to settle a disputed label (the gate is a person). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01J6JtZg47dHEKHDEfAJ5Kpf --- docs/adr/0004-mcp-server.md | 56 ++++++ .../plans/2026-09-26-mcp-server.md | 135 ++++++++++++++ .../specs/2026-09-26-mcp-server-design.md | 171 ++++++++++++++++++ 3 files changed, 362 insertions(+) create mode 100644 docs/adr/0004-mcp-server.md create mode 100644 docs/superpowers/plans/2026-09-26-mcp-server.md create mode 100644 docs/superpowers/specs/2026-09-26-mcp-server-design.md diff --git a/docs/adr/0004-mcp-server.md b/docs/adr/0004-mcp-server.md new file mode 100644 index 0000000..8bef922 --- /dev/null +++ b/docs/adr/0004-mcp-server.md @@ -0,0 +1,56 @@ +# ADR 0004 — An MCP server, in-process, that can never ask + +- **Status:** accepted +- **Date:** 2026-09-26 +- **Related:** ADR [0003](0003-llm-reference-list.md), `src/papertrace/mcp_server.py`, + `src/papertrace/ask.py`, `docs/superpowers/specs/2026-09-26-mcp-server-design.md` + +## Context + +The roadmap asked for an MCP server: PaperTrace driven as a tool from any +MCP-capable host, not only from Claude Code through a shell. Two things make +that more than a wrapper here. + +The rules this codebase keeps were written for readers of a file and for a +process that lives for one command. An MCP server has a new reader — a model +reading JSON — and a new process shape, one that outlives many audits. +`ask._MODELS` was correct only because a process ran one command. + +And an audit takes minutes, while a host built on the TypeScript SDK times a +request out at 60 s by default. + +## Decision + +**`papertrace mcp` serves nine tools over stdio, behind an optional `[mcp]` +extra. It runs the pipeline in-process on one background thread at a time, and +the thread has nobody to ask.** + +1. **In-process, not a subprocess.** `ask.py` remains the only file in `src/` + that starts a process, as `tests/test_ask.py` asserts. The cost is that the + server must manage process-global state itself: the pipeline's console is + swapped for a capture sink, and `ask.forget_models()` clears the model record + at the start of every audit. +2. **One audit at a time.** The console, the model record and `sys.stdin` are + process-global. A second `start_audit` is refused, not queued, and read tools + refuse a case whose audit is still running, because the files on disk are + then another run's. +3. **Nobody to ask.** The job runs with an empty `sys.stdin`, so the interactive + escalation of a disputed reference label is unreachable and the label is + withheld with `numbering_chosen_by: "default"`. ADR 0003's gate is a person + shown the disagreement; an MCP client can be a model answering for itself, + so elicitation is not offered as a substitute. +4. **A job, not a blocking call.** `start_audit` returns at once; + `audit_status` long-polls for at most 50 s, under the 60 s default. +5. **The same disclosures, serialised the viewer's way.** Nothing is + re-derived. The parity suite asserts every token reaches the MCP output. + +## Consequences + +- A host that is not Claude Code can run and read an audit, and see the + evidence crops as images. +- A disputed label cannot be settled from an MCP host. That is deliberate; + `papertrace refs` in a terminal settles it. +- A running audit cannot be cancelled from the host; stopping the server stops + it, and the case folder is re-runnable. +- Any future process-global state in the pipeline must be reset per audit, or + it will leak from one audit into the next. `CLAUDE.md` says so. diff --git a/docs/superpowers/plans/2026-09-26-mcp-server.md b/docs/superpowers/plans/2026-09-26-mcp-server.md new file mode 100644 index 0000000..b8fb41e --- /dev/null +++ b/docs/superpowers/plans/2026-09-26-mcp-server.md @@ -0,0 +1,135 @@ +# MCP Server — Implementation Plan + +**Goal:** Make PaperTrace usable from any MCP host — start an audit, follow it, +and read every finding with its caveats and its evidence images — without +weakening a single rule in `CLAUDE.md`. + +**Architecture:** One new module, `src/papertrace/mcp_server.py`, built on the +official SDK's `MCPServer` (`mcp>=2.2,<3`). Read tools load the case folder's +JSON through the existing dataclasses and serialise the existing disclosures. +The one driving tool runs the existing pipeline in-process on a background +thread. `papertrace mcp` serves it over stdio. + +**Spec:** `docs/superpowers/specs/2026-09-26-mcp-server-design.md` + +**Decision record:** `docs/adr/0004-mcp-server.md` + +## Global Constraints + +- **Tests stay offline.** No network, no model calls. Tools are exercised through + the SDK's in-memory `Client`; the pipeline is faked at `cli._run_pipeline`. +- **No `conftest.py`.** Every test module prepends `src/` to `sys.path` and + imports with `# noqa: E402`. +- **`ask.py` stays the only file in `src/` that starts a process** — + `tests/test_ask.py::test_only_ask_py_shells_out_to_a_model` must stay green. +- **No new case-folder field** (gate 2 untouched). **No ingest, highlight or + report *logic* changes** (gate 5 untouched — say so in the handover rather + than skip it silently). **`refs.py`, `check.py`, `scout.py` untouched** + (gate 4 untouched). +- **Red before green, pasted.** Each task: write the test, run it, see it fail + for the stated reason, implement, run it, see it pass. +- `ruff check src tests scripts evals` clean; never `ruff format`. + +## File Structure + +| File | Responsibility | Task | +|---|---|---| +| `src/papertrace/cli.py` *(modified)* | `run` split into `_run_pipeline` (keyword-only, plain defaults) and its Typer adapter; `papertrace mcp` command | 1, 5 | +| `src/papertrace/ask.py` *(modified, +1 function)* | `forget_models()` — a long-lived process starts each run with no recorded model | 2 | +| `src/papertrace/mcp_server.py` *(new)* | `build_server()`, the nine tools, the job registry, `serve()` | 3, 4 | +| `pyproject.toml` *(modified)* | `mcp` extra; `mcp` in `[dev]` and `[full]` | 5 | +| `tests/test_pipeline_split.py` *(modified)* | `run` delegates every parameter to `_run_pipeline` | 1 | +| `tests/test_ask.py` *(modified)* | `forget_models` | 2 | +| `tests/test_mcp_server.py` *(new)* | every tool, the parity loop, the job, stdio | 3, 4, 5 | +| `tests/test_packaging.py` *(modified)* | the extra exists and CI installs it | 5 | +| `README.md`, `CHANGELOG.md`, `CLAUDE.md` | docs | 6 | + +## Task 1 — split `run` into `_run_pipeline` + +`run` is the one stage function still called only through its Typer command, +and the MCP job would be its second caller. Calling a Typer command directly +hands every omitted option an `OptionInfo` sentinel — the flaw `cli.py` records +shipping twice — so the split is made *before* the new caller exists. + +- [ ] Red: `test_run_pipeline_rejects_a_positional_call` and + `test_run_command_delegates_to_the_pipeline_function` (exact kwargs dict, + in the style of the `refs` test beside them). Expect `AttributeError: no + _run_pipeline`. +- [ ] Green: move `run`'s body into `_run_pipeline(*, ...)`; `run` passes every + parameter by keyword. No behaviour change: the existing suite is the proof. + +## Task 2 — `ask.forget_models()` + +- [ ] Red: seed `_MODELS` for both sites, call `forget_models()`, expect + `model_for(...) is None` for both — `AttributeError` first. +- [ ] Green: one function, clearing the dict in place (tests monkeypatch the + attribute, so rebinding it would be invisible to a caller holding it). + +## Task 3 — the read tools + +Fixture: a case folder written with `RunResults.to_json`, +`RefManifest.to_json`, `ScoutResults.to_json`, and evidence PNGs drawn with +pymupdf. Each test calls through `Client(build_server())`. + +- [ ] Red, then green, one behaviour at a time: + - `list_tools` advertises nine tools; read tools carry + `read_only_hint=True, open_world_hint=False`; `start_audit` carries + `destructive_hint=True, open_world_hint=True`. + - `audit_summary` returns the counts and every run-level disclosure; a case + with no `results.json` is a `ToolError` naming the missing file; a malformed + one is a `ToolError`, never empty counts. + - A limited run: the scope disclosure is first **and** `limited` is the last + key of the summary and of `list_claims`. + - Parity: for each fixture in `test_disclosure_parity.py`'s shape, every run + token is in `audit_summary`'s text and every claim token in `get_claim`'s. + - `list_claims` filters on the headline verdict; the filter's enum is + `models.VERDICTS`, not a copy. + - `get_evidence` returns the headline judgement's primary and continuation + images plus a caption carrying the anchor disclosure; a named source picks + that judgement; a `not_retrieved` claim is a `ToolError` saying nothing was + read; an image path escaping `out/` is refused. + - `list_references` has no `pdf_path` anywhere; `labels_uncomparable: null` + stays `null` and `[]` stays `[]`; the status filter's enum is + `models.REF_STATUSES`. + - `list_gaps` lists uncited assertions, unreached occurrences and unchecked + claims; a coverage audit that never ran is `null`. + - `get_scout` carries the scout's own caveat; no `scout.json` is a + `ToolError`. + +## Task 4 — `start_audit` and `audit_status` + +- [ ] Red, then green: + - Preflight refusals are immediate `ToolError`s: missing manuscript, `claude` + not on `PATH`, no contact email, a case folder holding another paper. + - The job passes `_run_pipeline` every parameter explicitly. + - Inside the job: `cli._interactive()` is False, `input()` raises `EOFError`, + `ask.model_for(SITE_CHECK)` is None even when a previous audit left one, + and `cli.console` output lands in the job log, not on stdout. + - `audit_status` goes `running` → `finished`; a `typer.Exit(2)` from the + pipeline is `failed` with the console's reason; any other exception is + `failed` with its type and message. + - A second `start_audit` while one runs is refused, naming the running case. + - Read tools refuse the case whose audit is running. + - `wait_seconds` returns as soon as the job ends, and never waits past 50. + - A finished status carries no counts. + +## Task 5 — `papertrace mcp`, packaging, stdio + +- [ ] Red, then green: + - `[project.optional-dependencies]` has `mcp`, and `[dev]` and `[full]` carry + the same requirement (packaging tests). + - `papertrace mcp` with stdin closed exits 0 and writes **zero bytes** to + stdout (subprocess). + - A real stdio round trip — `Client(StdioServerParameters(...))` — lists the + nine tools and reads a fixture case. + - Without the `mcp` package, `papertrace mcp` exits 2 with the install line on + **stderr** and nothing on stdout. + +## Task 6 — docs + +- [ ] README: a "Use it from an MCP host" section (Claude Desktop, Claude Code, + Cursor/VS Code), the install matrix row, the does/does-not list, the + roadmap tick. Every statement checked against the code. +- [ ] CHANGELOG: an Unreleased entry. +- [ ] CLAUDE.md: the architecture bullet for `mcp_server.py`, and the + long-lived-process rule (`forget_models`, one job at a time). diff --git a/docs/superpowers/specs/2026-09-26-mcp-server-design.md b/docs/superpowers/specs/2026-09-26-mcp-server-design.md new file mode 100644 index 0000000..8d8d8f6 --- /dev/null +++ b/docs/superpowers/specs/2026-09-26-mcp-server-design.md @@ -0,0 +1,171 @@ +# An MCP server — PaperTrace as a tool for any MCP client + +Design, 2026-09-26. Status: implemented alongside this document on +`claude/gifted-ritchie-ga60nx`; open to review. + +## Why + +The roadmap has carried one line since 0.5.0: *MCP server — drive PaperTrace as +a tool from any MCP-capable client.* Today there are three ways in: the CLI, the +wizard, and the two Claude Code skills, which drive the CLI through a shell. A +host that is not Claude Code — Claude Desktop, Cursor, VS Code, anything that +speaks the Model Context Protocol — has no way to start an audit or read one +without a shell. MCP is the standard way to hand a local tool to those hosts. + +The risk is specific to this codebase. Every rule in `CLAUDE.md` was written for +readers of a **file** — a report, a manifest, a `results.json` — and for a +process that **lives for one command**. An MCP server is a new reader (a model) +and a new process shape (one that outlives many audits). Each rule has to be +re-established for both, or the server becomes the path by which a verdict +reaches a reader without its caveats. + +## What it is + +`papertrace mcp` serves over **stdio**, behind an optional extra +(`pip install -e ".[mcp]"`, `mcp>=2.2,<3` — the SDK's own versioning policy +promises breaking changes only in a major). Nine tools, in two groups: + +| Tool | Group | What it returns | +|---|---|---| +| `audit_summary` | read | counts, references, checker, coverage headline, **every run-level disclosure**, the scope statement last | +| `list_claims` | read | one row per claim (optionally one verdict), each with its claim-level caveats | +| `get_claim` | read | one claim in full: quote, per-source judgements, rationale, page, block, anchor phrases, every disclosure | +| `get_evidence` | read | the red-box crops for one claim (or one of its sources) **as images**, captioned with the anchor state | +| `list_references` | read | the retrieval manifest without local paths, with the numbering state as recorded | +| `list_gaps` | read | uncited assertions, citation places no claim reached, claims never checked | +| `get_scout` | read | the three scout registers, with the scout's own caveat | +| `start_audit` | drive | starts the full pipeline (`papertrace run`) as a background job and returns at once | +| `audit_status` | drive | the job's state and log tail; long-polls up to 50 s | + +Read tools are deterministic, offline and free: they read the case folder the +pipeline wrote. `start_audit` is the only tool that spends — model calls through +`claude -p`, and the open-access services `refs` and `scout` query. + +## The rules it inherits, and how each is kept + +1. **An unread source never receives a verdict.** The server computes no + verdict. It reads what the pipeline wrote, or runs the pipeline itself. + Nothing here derives, re-ranks or summarises a verdict. +2. **Disclosures travel with every count and every verdict.** The read tools + call `run_disclosures`, `claim_disclosures` and `judgement_disclosures` — the + same producers the four report formats use — and serialise them with the + viewer's own `report._disclosure_dict`. The MCP output is a fifth reader, and + the parity test is extended to it: every token reaches it. +3. **A limited run never reads as a smaller paper.** The scope disclosure is + first among the disclosures, as in every report, and every result that states + counts or lists claims ends with a `limited` field carrying the same sentence + — the JSON a model reads has an end, and the scope is said there too. +4. **Absent is not zero.** A missing `results.json` is a `ToolError` naming what + is missing, never a result with empty counts. Three-state manifest fields + (`labels_uncomparable`, `numbering_corroborated`) pass through unchanged, so + `null` stays `null`. A coverage audit that never ran is `null`, not `""`. +5. **Failure is said, never guessed.** In SDK v2 only a `ToolError`'s message + reaches the model; any other exception becomes a message-less + `Error executing tool …`. So every refusal is a `ToolError` carrying the + reason — the same sentence the CLI would have printed — and a job that fails + records its reason in its state instead of surfacing as an anonymous error. +6. **The interactive gate on disputed labels cannot be reached.** The job runs + with nobody to ask: `sys.stdin` is an empty stream for its duration, so + `cli._interactive()` is False and any prompt reads end-of-file rather than + waiting. A disputed label is therefore withheld with + `numbering_chosen_by: "default"`, exactly as the CLI does without a + terminal. An MCP client cannot settle a dispute; a person at a terminal can, + with `papertrace refs`. +7. **`ask.py` stays the only file in `src/` that starts a process.** The server + runs the pipeline **in-process, on a thread** — never a `papertrace run` + child. `tests/test_ask.py` would go red otherwise, and the rule protects the + seam, not a module. +8. **The model a run pinned is the model it reports.** `ask._MODELS` was + correct because one process ran one command. A server runs audit after + audit, and a second audit whose judging made no calls would print the first + audit's judge as its `Checker:` — the bug `ask.py` was restructured to end. + `ask.forget_models()` clears the record at the start of every audit. +9. **Nothing reaches stdout but the protocol.** Under stdio, stdout *is* the + wire. The pipeline prints through `cli.console`; for a job that console is + swapped for a sink whose lines become the job's log. `papertrace mcp` prints + nothing before serving, and the SDK's stdio transport diverts fd 1 to stderr + while it serves, as a second line of defence. +10. **No path is served that the tool did not write.** References are listed + without `pdf_path`, as in the viewer. An evidence image is read only when it + resolves inside `/out/` and is a `.png` — a `results.json` edited to + point at `../../somewhere` is refused, not followed. + +## Long runs: a job, not a blocking call + +An audit takes minutes — the demo took 95 s with the layout models cached, and +the first docling run downloads about 500 MB. A client built on the TypeScript +SDK times a request out at **60 s** (`DEFAULT_REQUEST_TIMEOUT_MSEC = 60000` in +`@modelcontextprotocol/sdk` 1.30.1; resetting on progress is opt-in). A blocking +`run_audit` would fail there while the audit carried on unseen, still spending. + +So `start_audit` validates what it can synchronously — the manuscript exists, +`claude` is on this server's `PATH`, a contact email resolves, the case folder +is not another paper's — and then returns at once. `audit_status` reports +`running`, `finished` or `failed` with the log tail, and waits up to +`wait_seconds` (at most 50, under that 60 s) for a running job to end. + +**One audit at a time per server process.** The pipeline's console, `ask`'s +per-site model record and `sys.stdin` are process-global; two audits sharing +them would interleave logs and misattribute models. A second `start_audit` is +refused, naming the audit in progress. + +**Read tools refuse a case whose audit is running.** The `results.json` on disk +then belongs to the previous run, or is half-written, and presenting it as this +run's would be the silent failure this codebase exists to refuse. + +`audit_status` never returns counts. A count without its disclosures is exactly +what rule 2 forbids, so a finished job points at `audit_summary`. + +## What is rejected, and why + +- **A subprocess per audit.** The cleanest isolation — its own stdout, its own + globals, killable — and a second process-spawning file in `src/`, which + `tests/test_ask.py` exists to refuse. +- **A blocking `run_audit`.** Fails on 60 s clients and keeps spending after the + client has given up. +- **MCP elicitation to settle a disputed label.** The gate in + `_escalate_disputed` is a *person* shown the disagreement. An MCP client may be + a model answering on its own behalf, and `numbering_chosen_by: "user"` would + then record a choice nobody made. +- **One tool per stage** (`ingest`, `extract`, `refs`, `check`, `highlight`, + `report`). The `/review` skill already drives stages one by one in Claude + Code; for other hosts one audit plus inspection covers the use, and every + extra spending tool is another cost surface. Cherry-picking claims — `check` + on an array of ids, which `select_claims` already takes — is the natural next + tool, and is left for its own change. +- **MCP resources for `report.md` and `results.json`.** Tools are supported by + every host; resources are surfaced unevenly. Deferred. +- **MCP-level logging (`ctx.info`).** Deprecated by the 2026-07-28 protocol + revision (SEP-2577). The job log is returned by `audit_status` instead. +- **Streamable HTTP.** A case folder stays on the machine that made it, by + design. stdio only. +- **A hand-written schema in `schemas/`.** The output schema is generated from + the tools' `TypedDict` return types and published by `tools/list`; a copy + would be a second contract to drift. No case-folder file gains a field, so + gate 2 is untouched. + +## Deliberate omissions + +- `--png` is not exposed: it needs playwright and a browser, and the images the + server serves are the evidence crops, which need neither. +- A running audit cannot be cancelled. A thread cannot be stopped safely + mid-write; the CLI's Ctrl-C carries the same exposure. If the host stops the + server, the job's daemon thread stops with it and the case folder is + re-runnable. +- Job state is not persisted across server restarts. `audit_status` says so + ("no audit of this case was started by this server process") rather than + guessing from the files on disk. + +## Testing + +- Every tool through the SDK's in-memory `Client(build_server())`: offline, no + model, fixtures built from the dataclasses, PNGs drawn with pymupdf. +- The job is exercised with `cli._run_pipeline` faked; the fake asserts what + the job guarantees from inside it (nobody to ask, no stale model, the console + captured). +- One real stdio round trip through a subprocess, and one test that + `papertrace mcp` writes nothing to stdout when its stdin closes. +- Parity: every run-level and claim-level disclosure token appears in the MCP + output for the same fixtures `test_disclosure_parity.py` uses. +- Packaging: `mcp` is in `[mcp]`, in `[full]`, and in `[dev]` — so CI runs these + tests rather than skipping them. From 94e0f17b82b652381a12e0e101e6025b607cfc37 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 26 Sep 2026 12:18:10 +0000 Subject: [PATCH 02/13] cli: split run into _run_pipeline before it gains a second caller MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `run` was the last stage still reachable only through its Typer command. The MCP server's audit job is about to call it, and a caller that omits an option from a Typer command receives an OptionInfo for it — the flaw cli.py records shipping twice. The body moves unchanged into a keyword-only function with plain defaults; `run` passes every parameter by keyword, and a test compares the two signatures so neither can gain a parameter the other does not. No behaviour change: the existing run tests fake the inner stages and still exercise the moved body. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01J6JtZg47dHEKHDEfAJ5Kpf --- src/papertrace/cli.py | 139 +++++++++++++++++++++-------------- tests/test_pipeline_split.py | 38 ++++++++++ 2 files changed, 122 insertions(+), 55 deletions(-) diff --git a/src/papertrace/cli.py b/src/papertrace/cli.py index cd93243..217440f 100644 --- a/src/papertrace/cli.py +++ b/src/papertrace/cli.py @@ -2177,62 +2177,29 @@ def report( _report_pipeline(case=case, png=png, formats=formats) -@app.command(rich_help_panel="Start here") -def run( - manuscript: Path = typer.Argument(..., exists=True), - case: Path = typer.Option( - None, "--case", "-c", - help="Case folder (default: a folder named after the paper, beside the paper)", - ), - provided: Path = typer.Option( - None, "--provided", - help="Folder of reference PDFs you already have; files match by name " - "-.pdf (e.g. pyrros-2023.pdf)", - ), - email: str = typer.Option(None, "--email", envvar=["PAPERTRACE_EMAIL", "MANUSCRIPTAGENT_EMAIL"]), - model: str = typer.Option(None, "--model"), - png: bool = typer.Option( - False, "--png/--no-png", - help="Also export PNG images of the report looks (one-time: playwright install chromium)", - ), - backend: str = typer.Option("auto", "--backend", help="auto | docling | pymupdf"), - with_scout: bool = typer.Option( - True, "--scout/--no-scout", - help="Also scan Europe PMC for newer + uncited literature", - ), - doi: str = typer.Option( - None, "--doi", - help="DOI of the paper itself — checks the reference numbering against the " - "publisher's deposited list, and pins the scout's literature search", - ), - formats: list[str] = typer.Option( - None, "--format", "-f", - help="Extra looks to render beside report.md: editor | terminal | viewer (repeatable)", - ), - supplement: list[Path] = typer.Option( - None, "--supplement", exists=True, - help="Supplementary material for THIS paper (repeatable). A cited work's " - "supplement needs no flag — drop it in the sources folder named after " - "the reference, e.g. pyrros-2023-supplement.pdf", - ), - llm_refs: bool = typer.Option( - True, "--llm-refs/--no-llm-refs", - help="Also have a model read the printed reference list as a second opinion " - "on the numbering (one extra model call — none under --parse-only, or on a " - "pymupdf backend, which leaves no second reading to compare)", - ), - max_claims: int = typer.Option( - None, "--max-claims", min=1, - help="Check only the first N extracted claims, in reading order; only the " - "references they cite are retrieved, and every report says so", - ), - max_sources: int = typer.Option( - None, "--max-sources", min=1, - help="Obtain and judge against at most N cited sources (PDFs), in bibliography " - "order; the rest are skipped, and every report says so", - ), +def _run_pipeline( + *, + manuscript: Path, + case: Path | None = None, + provided: Path | None = None, + email: str | None = None, + model: str | None = None, + png: bool = False, + backend: str = "auto", + with_scout: bool = True, + doi: str | None = None, + formats: list[str] | None = None, + supplement: list[Path] | None = None, + llm_refs: bool = True, + max_claims: int | None = None, + max_sources: int | None = None, ) -> None: - """Full pipeline: ingest → extract → refs → scout → check → highlight → report.""" + """`run`'s work: ingest → extract → refs → scout → check → highlight → report. + + Keyword-only with plain defaults, for the reason every stage function is: + `run` has a second caller — the MCP server's audit job — and a caller that + omitted an option from the Typer command would hand it an `OptionInfo`. + """ console.print(BANNER) email = _email(email) # fail fast — before the ingest models load, not after # the limits as the stages take them: an array of claim ids and a count. @@ -2305,6 +2272,68 @@ def run( ) +@app.command(rich_help_panel="Start here") +def run( + manuscript: Path = typer.Argument(..., exists=True), + case: Path = typer.Option( + None, "--case", "-c", + help="Case folder (default: a folder named after the paper, beside the paper)", + ), + provided: Path = typer.Option( + None, "--provided", + help="Folder of reference PDFs you already have; files match by name " + "-.pdf (e.g. pyrros-2023.pdf)", + ), + email: str = typer.Option(None, "--email", envvar=["PAPERTRACE_EMAIL", "MANUSCRIPTAGENT_EMAIL"]), + model: str = typer.Option(None, "--model"), + png: bool = typer.Option( + False, "--png/--no-png", + help="Also export PNG images of the report looks (one-time: playwright install chromium)", + ), + backend: str = typer.Option("auto", "--backend", help="auto | docling | pymupdf"), + with_scout: bool = typer.Option( + True, "--scout/--no-scout", + help="Also scan Europe PMC for newer + uncited literature", + ), + doi: str = typer.Option( + None, "--doi", + help="DOI of the paper itself — checks the reference numbering against the " + "publisher's deposited list, and pins the scout's literature search", + ), + formats: list[str] = typer.Option( + None, "--format", "-f", + help="Extra looks to render beside report.md: editor | terminal | viewer (repeatable)", + ), + supplement: list[Path] = typer.Option( + None, "--supplement", exists=True, + help="Supplementary material for THIS paper (repeatable). A cited work's " + "supplement needs no flag — drop it in the sources folder named after " + "the reference, e.g. pyrros-2023-supplement.pdf", + ), + llm_refs: bool = typer.Option( + True, "--llm-refs/--no-llm-refs", + help="Also have a model read the printed reference list as a second opinion " + "on the numbering (one extra model call — none under --parse-only, or on a " + "pymupdf backend, which leaves no second reading to compare)", + ), + max_claims: int = typer.Option( + None, "--max-claims", min=1, + help="Check only the first N extracted claims, in reading order; only the " + "references they cite are retrieved, and every report says so", + ), + max_sources: int = typer.Option( + None, "--max-sources", min=1, + help="Obtain and judge against at most N cited sources (PDFs), in bibliography " + "order; the rest are skipped, and every report says so", + ), +) -> None: + """Full pipeline: ingest → extract → refs → scout → check → highlight → report.""" + _run_pipeline(manuscript=manuscript, case=case, provided=provided, email=email, + model=model, png=png, backend=backend, with_scout=with_scout, doi=doi, + formats=formats, supplement=supplement, llm_refs=llm_refs, + max_claims=max_claims, max_sources=max_sources) + + def main() -> None: app() diff --git a/tests/test_pipeline_split.py b/tests/test_pipeline_split.py index b66b096..1b62c11 100644 --- a/tests/test_pipeline_split.py +++ b/tests/test_pipeline_split.py @@ -118,6 +118,44 @@ def test_run_hands_report_its_formats_rather_than_an_option_info(monkeypatch, tm assert seen["formats"] == ["md"], "run() must pass formats through explicitly" +def test_run_pipeline_rejects_a_positional_call(): + with pytest.raises(TypeError): + cli._run_pipeline(Path("whatever.pdf")) + + +def test_run_command_delegates_every_parameter_to_the_pipeline_function(monkeypatch, tmp_path): + """`run` gains a second caller — the MCP server's audit job — which is the + moment `cli.py`'s own comment names for splitting a stage: a caller that + omits an option hands a Typer command an `OptionInfo` for it, and a plain + keyword-only function has ordinary defaults instead. + + The signature comparison is what keeps the adapter whole: a parameter added + to one and not the other turns this red rather than reaching the pipeline as + its default without anyone deciding that.""" + import inspect + + # read before the patch below replaces it with a `**kw` lambda + pipeline_params = set(inspect.signature(cli._run_pipeline).parameters) + seen = {} + monkeypatch.setattr(cli, "_run_pipeline", lambda **kw: seen.update(kw)) + pdf = tmp_path / "p.pdf" + pdf.write_bytes(b"%PDF-1.4\n") + own = tmp_path / "p-supplement.pdf" + + cli.run(manuscript=pdf, case=tmp_path, provided=None, email="t@example.org", + model="claude-opus-5", png=False, backend="pymupdf", with_scout=False, + doi="10.1/x", formats=["viewer"], supplement=[own], llm_refs=False, + max_claims=3, max_sources=2) + + assert seen == { + "manuscript": pdf, "case": tmp_path, "provided": None, "email": "t@example.org", + "model": "claude-opus-5", "png": False, "backend": "pymupdf", "with_scout": False, + "doi": "10.1/x", "formats": ["viewer"], "supplement": [own], "llm_refs": False, + "max_claims": 3, "max_sources": 2, + } + assert set(inspect.signature(cli.run).parameters) == pipeline_params + + def test_check_pipeline_rejects_a_positional_call(): with pytest.raises(TypeError): cli._check_pipeline(Path("some-case")) From 3377fda667fa2a655d69c2bb5b12ec55a2e28cf4 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 26 Sep 2026 12:18:31 +0000 Subject: [PATCH 03/13] ask: forget_models, for a process that outlives one run MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The per-site model record was correct because one process ran one command, so it always began empty. A long-lived process — the MCP server — runs audit after audit, and a second audit whose judging made no calls would print the first audit's judge as its `Checker:`. `forget_models()` clears the record; the server calls it at the start of every audit. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01J6JtZg47dHEKHDEfAJ5Kpf --- src/papertrace/ask.py | 12 ++++++++++++ tests/test_ask.py | 17 +++++++++++++++++ 2 files changed, 29 insertions(+) diff --git a/src/papertrace/ask.py b/src/papertrace/ask.py index 3ddc588..5b7f40a 100644 --- a/src/papertrace/ask.py +++ b/src/papertrace/ask.py @@ -71,6 +71,17 @@ def model_for(site: str) -> str | None: return _MODELS.get(site) +def forget_models() -> None: + """Start a run with no model recorded at any site. + + The CLI never needed this: one process ran one command, so the record began + empty. The MCP server runs audit after audit in one process, and a second + audit whose judging made no calls would otherwise report the first audit's + judge as its own — the misattribution the per-site record exists to end. + """ + _MODELS.clear() + + def claude_available() -> bool: return shutil.which("claude") is not None @@ -159,5 +170,6 @@ def _parse_json_object(text: str) -> dict: "SITE_REFS", "claude_available", "for_site", + "forget_models", "model_for", ] diff --git a/tests/test_ask.py b/tests/test_ask.py index 9a5b574..259ac09 100644 --- a/tests/test_ask.py +++ b/tests/test_ask.py @@ -140,6 +140,23 @@ def test_a_reply_that_names_no_model_does_not_erase_the_one_recorded(monkeypatch assert ask_mod.model_for(ask_mod.SITE_CHECK) == "claude-opus-5" +def test_forget_models_starts_a_run_with_no_recorded_model(monkeypatch): + """One process ran one command, so the record never needed clearing. The + MCP server runs audit after audit in one process: without this, a second + audit whose judging made no calls — every cited source unretrieved — would + print the first audit's judge as its `Checker:`, the misattribution the + per-site record exists to prevent.""" + monkeypatch.setattr( + ask_mod, "_MODELS", + {ask_mod.SITE_CHECK: "claude-opus-5", ask_mod.SITE_REFS: "claude-haiku-4-5"}, + ) + + ask_mod.forget_models() + + assert ask_mod.model_for(ask_mod.SITE_CHECK) is None + assert ask_mod.model_for(ask_mod.SITE_REFS) is None + + def test_for_site_restores_the_previous_site_on_the_way_out(monkeypatch): """Nesting must not leak. A leaked site is a misattributed model.""" monkeypatch.setattr(ask_mod, "_SITE", "outer") From 978f8d3aa62b23d1bf88c18f80b735fd0b32d160 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 26 Sep 2026 12:25:52 +0000 Subject: [PATCH 04/13] mcp: read an audit from any MCP host, caveats and evidence included MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Seven read-only tools over a case folder — audit_summary, list_claims, get_claim, get_evidence, list_references, list_gaps, get_scout — built on the official SDK's MCPServer. They compute nothing: they load what the pipeline wrote and serialise the disclosures disclosures.py decided, with the viewer's own _disclosure_dict, so a model reading the result gets the same caveats every report format carries. The parity loops extend test_disclosure_parity.py's contract to this fifth reader. What the report formats promise, kept here: the scope sentence first among the disclosures and again as the last key; a missing or unreadable results.json refused with its reason rather than returned as empty counts; three-state manifest fields passed through with their null; no local PDF path; an evidence crop served only from inside /out/, so an edited results.json cannot point the server at another file. Found on the way: below Python 3.12 pydantic cannot build a schema from typing.TypedDict, and the SDK then drops the tool to unvalidated text without a word. Every tool here published no output schema on 3.10 and 3.11 until the test asked; typing_extensions.TypedDict fixes it and structured_output=True makes any repeat an error at registration. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01J6JtZg47dHEKHDEfAJ5Kpf --- src/papertrace/mcp_server.py | 753 +++++++++++++++++++++++++++++++++++ tests/test_mcp_server.py | 627 +++++++++++++++++++++++++++++ 2 files changed, 1380 insertions(+) create mode 100644 src/papertrace/mcp_server.py create mode 100644 tests/test_mcp_server.py diff --git a/src/papertrace/mcp_server.py b/src/papertrace/mcp_server.py new file mode 100644 index 0000000..45d3cb1 --- /dev/null +++ b/src/papertrace/mcp_server.py @@ -0,0 +1,753 @@ +"""PaperTrace as an MCP server — audits started, followed and read from any MCP host. + +The server is a fifth reader of everything the four report formats carry, and +it is held to their rules: it computes no verdict, every count travels with the +disclosures `disclosures.py` decided for it, and every refusal says why. The +design and what it rejected: `docs/superpowers/specs/2026-09-26-mcp-server-design.md`. +""" + +# No `from __future__ import annotations`, unlike the rest of the package: the +# SDK builds each tool's input and output schema from its annotations when the +# tool is registered, and live types are what it reads without guessing. + +from pathlib import Path +from typing import Annotated, Any, Literal + +from mcp.server import MCPServer +from mcp.server.mcpserver import Image +from mcp.server.mcpserver.exceptions import ToolError +from mcp.types import TextContent, ToolAnnotations +from pydantic import Field + +# Not `typing.TypedDict`: below Python 3.12 pydantic cannot build a schema from +# it, and the SDK then drops the tool to unvalidated text without a word — every +# tool here published no output schema on 3.10 and 3.11 until a test asked. +# `structured_output=True` on each tool turns any repeat into an error at start. +from typing_extensions import TypedDict + +from . import __version__ +from .disclosures import ( + anchor_disclosure, + claim_disclosures, + coverage_headline, + ids_span, + judgement_disclosures, + run_disclosures, +) +from .models import ( + PIPELINE_STATES, + REF_STATUSES, + VERDICTS, + ClaimResult, + RefManifest, + RunResults, + ScoutResults, + SourceJudgement, +) + +# the viewer's serialisation of a disclosure, reused rather than restated: one +# shape for every reader that receives disclosures as data +from .report import _disclosure_dict + +INSTRUCTIONS = """\ +PaperTrace checks what a scientific paper claims against what its cited sources say. + +Workflow: start_audit, then audit_status until it is finished (wait_seconds up to 50), +then audit_summary. From there: list_claims, get_claim, get_evidence (the red-box crop +of the passage a verdict rests on), list_references, list_gaps, get_scout. A case folder +made by the `papertrace` CLI can be read the same way. + +Relaying results: +- An unread source never receives a verdict. `not_retrieved` means nobody read the + source: report it as a gap, never as support or contradiction. +- Relay the disclosures that come with a result. They say what the numbers do not. +- `limited` set means the audit was cut short on request: its counts describe a slice, + not the paper. +- Verdicts are a model's drafts for a person's judgement; the evidence images show the + passage each rests on. Scout hits are candidates, not accusations. +""" + +# The CLI's sentence, verbatim: the scout's candidates are search results. +SCOUT_CAVEAT = ( + "search-based — absence from these lists proves nothing; presence is a candidate " + "for your judgement, not an accusation." +) + +# read tools change nothing and consult nothing outside the case folder +_READ = ToolAnnotations(read_only_hint=True, open_world_hint=False) + +# The vocabularies `models.py` defines, not copies of them. `Literal` over the +# tuple is the same type as the values spelled out. +Verdict = Literal[VERDICTS] +RefStatus = Literal[REF_STATUSES] + +CaseArg = Annotated[ + str, + Field(description=( + "The case folder of one audit: the folder holding refs_manifest.json and out/. " + "Use an absolute path — a relative one is resolved against this server's working " + "directory, which the host chose." + )), +] +ClaimId = Annotated[int, Field(description="A claim's number, as list_claims gives it.", ge=1)] + + +# -------------------------------------------------------------------------- +# what the tools return — each shape is also the published output schema +# -------------------------------------------------------------------------- + + +class DisclosureOut(TypedDict): + """One thing a result owes its reader, in the words every report format uses.""" + + key: str + level: str + token: str + text: str + short: str + rows: list[str] + + +class RefCounts(TypedDict): + total: int + available: int + + +class AuditSummary(TypedDict): + case: str + manuscript: str + checker: str + date: str + converter: str + references: RefCounts + claims: int + counts: dict[str, int] + uncited_assertions: int + coverage: str | None + disclosures: list[DisclosureOut] + reports: list[str] + limited: str | None + + +class ClaimRow(TypedDict): + id: int + verdict: str + label: str + qualifier: str + refs: list[str] + location: str + claim: str + quote: str + caveats: list[str] + + +class ClaimList(TypedDict): + case: str + verdict_filter: str | None + claims_total: int + claims: list[ClaimRow] + limited: str | None + + +class JudgementOut(TypedDict): + source: str + origin: str + kind: str + verdict: str + label: str + rationale: str + page: int | None + block: str | None + anchor_phrases: list[str] + anchor_located: bool | None + evidence_images: list[str] + disclosures: list[DisclosureOut] + + +class ClaimDetail(TypedDict): + case: str + id: int + verdict: str + label: str + qualifier: str + claim: str + quote: str + location: str + refs: list[str] + own_supplement: bool + rationale: str + source: str | None + page: int | None + block: str | None + anchor_phrases: list[str] + anchor_located: bool | None + evidence_images: list[str] + judgements: list[JudgementOut] + unjudged_refs: list[str] + withheld_refs: list[str] + disclosures: list[DisclosureOut] + + +class SupplementOut(TypedDict): + slug: str + verified: bool + + +class ReferenceRow(TypedDict): + label: str + status: str + reason: str + resolver: str | None + doi: str | None + title: str | None + year: str | None + slug: str | None + title_check: str | None + skipped_by: str | None + boundary_ambiguous: bool + seen_in: list[str] + supplements: list[SupplementOut] + raw: str + + +class NumberingOut(TypedDict): + source: str + verified: bool + note: str + unverified_from: int | None + contested: bool + corroborated: bool | None + corroborating_readings: list[str] + labels_disputed: list[str] + labels_uncomparable: list[str] | None + labels_resolved: list[str] + choice: str + chosen_by: str + + +class ReferenceList(TypedDict): + case: str + manuscript: str + total: int + available: int + by_status: dict[str, int] + numbering: NumberingOut + limits: dict[str, Any] + status_filter: str | None + references: list[ReferenceRow] + + +class UncitedOut(TypedDict): + id: int + claim: str + quote: str + location: str + + +class UnreachedOut(TypedDict): + label: str + status: str + section: str + sentence: str + + +class UncheckedOut(TypedDict): + id: int + verdict: str + label: str + refs: list[str] + location: str + note: str + + +class Gaps(TypedDict): + case: str + uncited_assertions: list[UncitedOut] + coverage: str | None + unreached_labels: list[str] | None + unreached_citations: list[UnreachedOut] | None + unchecked_claims: list[UncheckedOut] + disclosures: list[DisclosureOut] + limited: str | None + + +class ScoutPaperOut(TypedDict): + title: str + doi: str + year: int | None + resolved_via: str + identity: str + + +class ScoutHitOut(TypedDict): + title: str + year: int | None + doi: str + via: str + journal: str + authors: str + + +class ScoutOut(TypedDict): + case: str + paper: ScoutPaperOut + query: str + date: str + counts: dict[str, int] + newer: list[ScoutHitOut] + overlooked: list[ScoutHitOut] + same_year: list[ScoutHitOut] + error: str + caveat: str + + +# -------------------------------------------------------------------------- +# reading a case folder — every failure is a sentence, never an empty result +# -------------------------------------------------------------------------- + + +def _case_dir(case: str) -> Path: + return Path(case).expanduser().resolve() + + +def _read(path: Path, load, stage: str): + """One case file, loaded — or a refusal naming why it could not be.""" + try: + return load(path) + except (OSError, ValueError, KeyError, TypeError) as e: + raise ToolError( + f"{path} is unreadable ({type(e).__name__}: {str(e)[:200]}) — an audit may be " + f"writing it right now; if not, re-run `papertrace {stage}` for this case" + ) from None + + +def _results(case: Path) -> RunResults: + path = case / "out" / "results.json" + if not path.is_file(): + # absent is not zero: empty counts would read as an audit that found nothing + raise ToolError( + f"no audit results at {path}" + + ("" if case.is_dir() else f" — {case} does not exist") + + ". No claim has been checked in this case folder: start_audit runs the " + "whole pipeline, and `papertrace check -c ` runs the judging alone." + ) + return _read(path, RunResults.from_json, "check") + + +def _manifest(case: Path, *, required: bool) -> RefManifest | None: + path = case / "refs_manifest.json" + if not path.is_file(): + if required: + raise ToolError( + f"no retrieval manifest at {path} — the refs stage has not run in this " + "case folder, so no reference has been looked up there" + ) + return None + return _read(path, RefManifest.from_json, "refs") + + +def _disclosures(fired) -> list[DisclosureOut]: + return [_disclosure_dict(d) for d in fired] + + +def _limited(fired) -> str | None: + """The scope sentence, when the audit was limited on request — else None. + + Taken from the run disclosures rather than composed here, so the last thing + a result says is the sentence every report format ends with. + """ + return next((d.text for d in fired if d.key == "scope"), None) + + +def _find_claim(results: RunResults, claim_id: int) -> ClaimResult: + for c in results.claims: + if c.id == claim_id: + return c + ids = [c.id for c in results.claims] + raise ToolError( + f"no claim {claim_id} in this audit — its claims are {ids_span(ids) if ids else 'none'}" + + (" (the audit was limited on request; audit_summary's `limited` says what was left " + "out)" if results.scope else "") + ) + + +def _claim_row(c: ClaimResult, manifest: RefManifest | None) -> ClaimRow: + return { + "id": c.id, + "verdict": c.verdict, + "label": c.label, + "qualifier": c.headline_qualifier(), + "refs": list(c.refs), + "location": c.location, + "claim": c.claim, + "quote": c.quote, + # the terse line of each claim disclosure: every one carries its token + "caveats": [d.short for d in claim_disclosures(c, manifest)], + } + + +def _images_of(a) -> list[str]: + return [p for p in [a.evidence_image, *a.continuation_images] if p] + + +def _judgement_out(j: SourceJudgement) -> JudgementOut: + return { + "source": j.source_slug, + "origin": j.origin, + "kind": j.kind, + "verdict": j.verdict, + "label": j.label, + "rationale": j.note, + "page": j.source_page, + "block": j.source_block, + "anchor_phrases": list(j.anchor_phrases), + "anchor_located": j.anchor_located, + "evidence_images": _images_of(j), + "disclosures": _disclosures(judgement_disclosures(j)), + } + + +def _evidence_target(claim: ClaimResult, source: str | None): + """The judgement whose crop to show: the headline's, or the named source's.""" + if source is None: + return claim.deciding_judgement() or claim + for j in claim.judgements: + if j.source_slug == source: + return j + if not claim.judgements and claim.source_slug == source: + return claim + known = [j.source_slug for j in claim.judgements] or [s for s in [claim.source_slug] if s] + raise ToolError( + f"claim {claim.id} was not judged against {source!r} — " + f"its sources are {', '.join(known) if known else 'none'}" + ) + + +def _no_evidence(claim: ClaimResult, target) -> str: + """Why there is no picture — one reason per state, never a blank.""" + who = f"claim {claim.id}" + if target.verdict == "not_retrieved": + return ( + f"{who} has no evidence to show: its cited source was not retrieved, so nothing " + "was read — a recorded gap, not a finding" + ) + if target.verdict == "unchecked": + return ( + f"{who} has no evidence to show: no verdict was reached " + f"({target.note or 'the check did not complete'})" + ) + if target.verdict == "not_addressed": + return ( + f"{who}'s source was read and does not address the claim — that verdict rests " + "on no passage, so it carries no crop by design" + ) + return ( + f"{who} is {target.verdict}, but no evidence crop has been written for it — the " + "highlight stage writes the crops (`papertrace highlight -c `)" + ) + + +def _crop_paths(case: Path, claim: ClaimResult, rels: list[str]) -> list[Path]: + """The crops, resolved — refused if one points anywhere but `/out/`.""" + out = (case / "out").resolve() + paths = [] + for rel in rels: + p = (out / rel).resolve() + # `results.json` is a file anyone can edit: a path is followed only when + # it lands inside the folder the highlight stage writes, and is a PNG + if not p.is_relative_to(out) or p.suffix.lower() != ".png": + raise ToolError( + f"claim {claim.id}'s evidence path {rel!r} resolves outside {out} or is not " + "a PNG — refused rather than followed; this server serves only the crops " + "the highlight stage writes into the case folder" + ) + paths.append(p) + return paths + + +def _caption(claim: ClaimResult, target, shown: list[Path], missing: list[str]) -> str: + qualifier = claim.headline_qualifier() + lines = [f"claim {claim.id} — {claim.label}" + (f" ({qualifier})" if qualifier else "")] + if claim.quote: + lines.append(f"the paper's sentence: {claim.quote}") + origin = f"{target.origin} " if isinstance(target, SourceJudgement) else "" + lines.append( + f"{target.label} — {origin}{target.source_slug}" + + (f", page {target.source_page}" if target.source_page else "") + ) + if (d := anchor_disclosure(target)) is not None: + lines.append(d.text) + if target.anchor_phrases: + lines.append("anchor phrases: " + " · ".join(f"“{p}”" for p in target.anchor_phrases)) + if len(shown) > 1: + lines.append( + f"the passage crosses a column or page break — {len(shown)} images, in reading order" + ) + if missing: + lines.append(f"not in the case folder, so not shown: {', '.join(missing)}") + return "\n".join(lines) + + +def _coverage_audited(coverage: dict) -> bool: + """Did the coverage audit run? The rule `run_disclosures` applies.""" + return bool(coverage.get("labels_in_text") or (coverage.get("occurrences") or {}).get("total")) + + +def _scout_caveat(scout: ScoutResults) -> str: + """The scout's own caveat, led by the identity warning the CLI prints.""" + identity = scout.paper_identity + if not scout.paper_title or identity == "confirmed": + lead = "" + elif identity == "mismatch": + lead = "The record found is NOT this paper, so these registers describe another one. " + elif identity == "unverified": + lead = "The paper's identity in Europe PMC is unverified — check the record is this paper. " + else: + lead = "The paper's identity was not recorded — check the record is this paper. " + return lead + SCOUT_CAVEAT[0].upper() + SCOUT_CAVEAT[1:] + + +# -------------------------------------------------------------------------- +# the server +# -------------------------------------------------------------------------- + + +def build_server() -> MCPServer: + """A fresh server with every tool registered. Nothing is served until `run()`.""" + server = MCPServer("papertrace", instructions=INSTRUCTIONS, version=__version__) + + @server.tool(title="Audit summary", annotations=_READ, structured_output=True) + def audit_summary(case: CaseArg) -> AuditSummary: + """The result of one audit: verdict counts, references obtained, the model that + judged, coverage, and every run-level disclosure — relay those with the counts. + `limited` is set when the audit was cut short on request.""" + root = _case_dir(case) + results = _results(root) + manifest = _manifest(root, required=False) + fired = run_disclosures(results, manifest) + return { + "case": str(root), + "manuscript": results.manuscript, + "checker": results.checker, + "date": results.date, + "converter": results.converter, + "references": {"total": results.refs_total, "available": results.refs_available}, + "claims": len(results.claims), + "counts": results.counts(), + "uncited_assertions": len(results.uncited), + # null when the audit could not run — never "" or a zero ratio + "coverage": coverage_headline(results.coverage or {}) or None, + "disclosures": _disclosures(fired), + "reports": sorted(str(p) for p in (root / "out").glob("report*.*") if p.is_file()), + "limited": _limited(fired), + } + + @server.tool(title="List claims", annotations=_READ, structured_output=True) + def list_claims( + case: CaseArg, + verdict: Annotated[ + Verdict | None, + Field(description="Only claims whose headline verdict is this one."), + ] = None, + ) -> ClaimList: + """Every checked claim, one row each: its headline verdict, the paper's own + sentence, the labels it cites, and its caveats. A claim citing several sources is + headlined by the most adverse verdict; get_claim shows each source's.""" + root = _case_dir(case) + results = _results(root) + manifest = _manifest(root, required=False) + rows = [_claim_row(c, manifest) for c in results.claims if verdict in (None, c.verdict)] + return { + "case": str(root), + "verdict_filter": verdict, + "claims_total": len(results.claims), + "claims": rows, + "limited": _limited(run_disclosures(results, manifest)), + } + + @server.tool(title="Claim detail", annotations=_READ, structured_output=True) + def get_claim(case: CaseArg, claim_id: ClaimId) -> ClaimDetail: + """One claim in full: the paper's sentence, each cited source's verdict with its + rationale, page, block and anchor phrases, and every disclosure the claim owes its + reader. get_evidence shows the passage itself.""" + root = _case_dir(case) + results = _results(root) + manifest = _manifest(root, required=False) + c = _find_claim(results, claim_id) + return { + "case": str(root), + "id": c.id, + "verdict": c.verdict, + "label": c.label, + "qualifier": c.headline_qualifier(), + "claim": c.claim, + "quote": c.quote, + "location": c.location, + "refs": list(c.refs), + "own_supplement": c.own_supplement, + "rationale": c.note, + "source": c.source_slug, + "page": c.source_page, + "block": c.source_block, + "anchor_phrases": list(c.anchor_phrases), + "anchor_located": c.anchor_located, + "evidence_images": _images_of(c), + "judgements": [_judgement_out(j) for j in c.judgements], + "unjudged_refs": list(c.unjudged_refs), + "withheld_refs": list(c.withheld_refs), + "disclosures": _disclosures(claim_disclosures(c, manifest)), + } + + @server.tool(title="Evidence crops", annotations=_READ, structured_output=False) + def get_evidence( + case: CaseArg, + claim_id: ClaimId, + source: Annotated[ + str | None, + Field(description="A source slug from get_claim's judgements; default: the source " + "behind the headline verdict."), + ] = None, + ) -> list[TextContent | Image]: + """The page crop a verdict rests on, as images, with the matched text boxed in red. + The boxes were drawn by code finding the model's anchor phrases in the PDF, not + placed by the model; the caption says whether the phrase was found.""" + root = _case_dir(case) + claim = _find_claim(_results(root), claim_id) + target = _evidence_target(claim, source) + rels = _images_of(target) + if not rels: + raise ToolError(_no_evidence(claim, target)) + paths = _crop_paths(root, claim, rels) + out = (root / "out").resolve() + missing = [str(p.relative_to(out)) for p in paths if not p.is_file()] + shown = [p for p in paths if p.is_file()] + if not shown: + raise ToolError( + f"claim {claim.id}'s evidence crop{'s' if len(missing) > 1 else ''} " + f"{', '.join(missing)} {'are' if len(missing) > 1 else 'is'} named in " + f"results.json but not in {out} — re-run `papertrace highlight -c `" + ) + return [ + TextContent(type="text", text=_caption(claim, target, shown, missing)), + *(Image(path=p) for p in shown), + ] + + @server.tool(title="References", annotations=_READ, structured_output=True) + def list_references( + case: CaseArg, + status: Annotated[ + RefStatus | None, + Field(description="Only references with this retrieval status."), + ] = None, + ) -> ReferenceList: + """The retrieval manifest: each cited reference, whether it was obtained and why + not, and the state of the reference numbering — the label is what joins a claim to + its source, so an unconfirmed or disputed numbering is a caveat on every verdict + that rests on it. Local file paths are not included.""" + root = _case_dir(case) + m = _manifest(root, required=True) + rows: list[ReferenceRow] = [ + { + "label": e.num, + "status": e.status, + "reason": e.reason, + "resolver": e.resolver, + "doi": e.doi, + "title": e.title, + "year": e.year, + "slug": e.slug, + "title_check": e.title_check, + "skipped_by": e.skipped_by, + "boundary_ambiguous": e.boundary_ambiguous, + "seen_in": list(e.seen_in), + "supplements": [{"slug": s.slug, "verified": s.verified} for s in e.supplements], + "raw": e.raw, + } + for e in m.entries + if status in (None, e.status) + ] + return { + "case": str(root), + "manuscript": m.manuscript, + "total": len(m.entries), + "available": len(m.retrieved), + "by_status": {s: sum(1 for e in m.entries if e.status == s) for s in REF_STATUSES}, + # passed through as recorded: `null` is "never computed" and `[]` a + # measurement, and no reader may be handed one for the other + "numbering": { + "source": m.reference_source, + "verified": m.numbering_verified, + "note": m.numbering_note, + "unverified_from": m.unverified_from, + "contested": m.numbering_contested, + "corroborated": m.numbering_corroborated, + "corroborating_readings": list(m.corroborating_readings), + "labels_disputed": list(m.labels_disputed), + "labels_uncomparable": ( + None if m.labels_uncomparable is None else list(m.labels_uncomparable) + ), + "labels_resolved": list(m.labels_resolved), + "choice": m.numbering_choice, + "chosen_by": m.numbering_chosen_by, + }, + "limits": dict(m.limits), + "status_filter": status, + "references": rows, + } + + @server.tool(title="Gaps", annotations=_READ, structured_output=True) + def list_gaps(case: CaseArg) -> Gaps: + """What the audit could not vouch for: assertions carrying no citation, citation + places no extracted claim reached, and claims never checked (source not retrieved, + or no verdict reached). `null` means that audit did not run — not that it found + nothing.""" + root = _case_dir(case) + results = _results(root) + manifest = _manifest(root, required=False) + coverage = results.coverage or {} + audited = _coverage_audited(coverage) + occurrences = coverage.get("occurrences") + fired = run_disclosures(results, manifest) + return { + "case": str(root), + "uncited_assertions": [ + {"id": u.id, "claim": u.claim, "quote": u.quote, "location": u.location} + for u in results.uncited + ], + "coverage": coverage_headline(coverage) or None, + "unreached_labels": list(coverage.get("missing") or []) if audited else None, + "unreached_citations": ( + [ + {"label": o["label"], "status": o["status"], + "section": o.get("section", ""), "sentence": o.get("sentence", "")} + for o in occurrences.get("items", []) + if o.get("status") in ("uncovered", "uncertain") + ] + if audited and occurrences else None + ), + "unchecked_claims": [ + {"id": c.id, "verdict": c.verdict, "label": c.label, "refs": list(c.refs), + "location": c.location, "note": c.note} + for c in results.claims + if c.verdict in PIPELINE_STATES + ], + "disclosures": _disclosures(fired), + "limited": _limited(fired), + } + + @server.tool(title="Literature scout", annotations=_READ, structured_output=True) + def get_scout(case: CaseArg) -> ScoutOut: + """Literature around the paper from Europe PMC: work published since, work that + existed before it and went uncited, and work from the same year. Search-based: + candidates for a person's judgement, and absence proves nothing.""" + root = _case_dir(case) + path = root / "out" / "scout.json" + if not path.is_file(): + raise ToolError( + f"no literature scan at {path} — the audit ran without the scout, or the " + "scout stage has not run (`papertrace scout -c `)" + ) + scout = _read(path, ScoutResults.from_json, "scout") + return {"case": str(root), **scout.to_dict(), "caveat": _scout_caveat(scout)} + + return server diff --git a/tests/test_mcp_server.py b/tests/test_mcp_server.py new file mode 100644 index 0000000..49b1dd8 --- /dev/null +++ b/tests/test_mcp_server.py @@ -0,0 +1,627 @@ +"""The MCP server: PaperTrace as a tool for any MCP host. + +Every tool is called the way a host calls it — through the SDK's in-memory +`Client` — so what these tests read is what a model reads. Offline: case +folders are written with the dataclasses, evidence PNGs are drawn with +pymupdf, and the one tool that spends is exercised with the pipeline faked. + +The server is a fifth reader of every disclosure. The parity loops below are +`test_disclosure_parity.py`'s contract extended to it: a token a report format +must carry, the MCP output must carry too. +""" + +import json +import sys +from pathlib import Path + +import pytest + +sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "src")) + +# `[dev]` installs the SDK, so CI runs this module rather than skipping it; +# `tests/test_packaging.py` asserts that it stays there +pytest.importorskip("mcp", reason="pip install -e '.[dev]' supplies the MCP SDK") + +import anyio # noqa: E402 +from mcp import Client # noqa: E402 + +from papertrace import mcp_server # noqa: E402 +from papertrace.disclosures import ( # noqa: E402 + SCOPE_TOKEN, + claim_disclosures, + judgement_disclosures, + run_disclosures, +) +from papertrace.models import ( # noqa: E402 + REF_STATUSES, + VERDICTS, + ClaimResult, + RefEntry, + RefManifest, + RunResults, + ScoutHit, + ScoutResults, + SourceJudgement, + UncitedClaim, +) + +READ_TOOLS = { + "audit_summary", "list_claims", "get_claim", "get_evidence", + "list_references", "list_gaps", "get_scout", +} + + +# -------------------------------------------------------------------------- +# calling the server the way a host does +# -------------------------------------------------------------------------- + + +def _session(server, fn): + async def go(): + async with Client(server, raise_exceptions=True) as client: + return await fn(client) + + return anyio.run(go) + + +def _call(tool: str, args: dict, server=None): + server = server or mcp_server.build_server() + return _session(server, lambda c: c.call_tool(tool, args)) + + +def _tools(server=None) -> dict: + server = server or mcp_server.build_server() + listing = _session(server, lambda c: c.list_tools()) + return {t.name: t for t in listing.tools} + + +def _text(result) -> str: + """Everything a model reads from one result, as one string.""" + return "\n".join(b.text for b in result.content if getattr(b, "type", "") == "text") + + +def _json(result) -> dict: + """The result as the model reads it — the JSON text, keys in the order sent.""" + assert not result.is_error, _text(result) + return json.loads(_text(result)) + + +# -------------------------------------------------------------------------- +# a case folder, written the way the pipeline writes one +# -------------------------------------------------------------------------- + + +def _png(path: Path) -> None: + try: + import pymupdf + except ImportError: # PyMuPDF < 1.24 + import fitz as pymupdf + path.parent.mkdir(parents=True, exist_ok=True) + doc = pymupdf.open() + page = doc.new_page(width=80, height=40) + page.insert_text((5, 20), path.stem[-12:]) + page.get_pixmap().save(str(path)) + doc.close() + + +def _split_claim() -> ClaimResult: + """Two cited sources that disagree: the headline is the contradiction.""" + c = ClaimResult( + id=1, + claim="the model reached an AUC of 0.94", + quote="The model reached an AUC of 0.94 [3, 4].", + location="Results §2", + refs=["3", "4"], + judgements=[ + SourceJudgement( + source_slug="alpha-2020", ref="3", verdict="supported", + note="Table 2 reports 0.94.", source_page=2, source_block="block_0012", + anchor_phrases=["AUC 0.94"], evidence_image="evidence/claim_01_alpha-2020_p2.png", + anchor_located=True, + ), + SourceJudgement( + source_slug="beta-2021", ref="4", verdict="contradicted", + note="The source reports 0.77, not 0.94.", source_page=5, + source_block="block_0040", anchor_phrases=["AUC of 0.77"], + evidence_image="evidence/claim_01_beta-2021_p5.png", + continuation_images=["evidence/claim_01_beta-2021_p5_cont2.png"], + anchor_located=False, + ), + ], + ) + c.apply_headline() + return c + + +def _unread_claim() -> ClaimResult: + return ClaimResult( + id=2, claim="attendance was 99%", quote="Attendance was 99% [7].", + location="Discussion", refs=["7"], verdict="not_retrieved", unjudged_refs=["7"], + ) + + +def _withheld_claim() -> ClaimResult: + return ClaimResult( + id=3, claim="the cohort was imaged twice", quote="The cohort was imaged twice [9].", + location="Methods", refs=["9"], verdict="unchecked", withheld_refs=["9"], + note="verdict withheld — the readings of the reference list disagree about [9]", + ) + + +def _paraphrase_claim(anchor_located=None) -> ClaimResult: + return ClaimResult( + id=4, claim="screening halves mortality", location="Introduction", refs=["3"], + verdict="supported", note="Stated in the abstract.", source_slug="alpha-2020", + source_page=1, source_block="block_0002", anchor_phrases=["halves mortality"], + evidence_image="evidence/claim_04_alpha-2020_p1.png", anchor_located=anchor_located, + ) + + +def _manifest() -> RefManifest: + return RefManifest( + manuscript="paper.pdf", + manuscript_sha256="0" * 64, + numbering_verified=False, + numbering_note="the parse and the deposit disagree from [5]", + unverified_from=5, + labels_disputed=["9"], + labels_uncomparable=[], + entries=[ + RefEntry(num="3", raw="Alpha A. A paper. 2020.", doi="10.1/alpha", status="retrieved", + resolver="unpaywall", slug="alpha-2020", title_check="verified", + pdf_path="/private/home/someone/sources_resolved/alpha-2020.pdf"), + RefEntry(num="4", raw="Beta B. Another. 2021.", status="provided", resolver="user", + slug="beta-2021", title_check="unverifiable", + pdf_path="/private/home/someone/my_pdfs/beta.pdf"), + RefEntry(num="7", raw="Gamma G. Paywalled. 2019.", doi="10.1/gamma", + status="paywalled", reason="no open-access copy found"), + RefEntry(num="8", raw="Delta D. Unasked. 2018.", status="skipped", + skipped_by="claims", reason="no selected claim cites it"), + RefEntry(num="9", raw="Epsilon E. Disputed. 2022.", status="retrieved", + resolver="crossref", slug="epsilon-2022", title_check="verified", + pdf_path="/private/home/someone/sources_resolved/epsilon-2022.pdf"), + ], + ) + + +OCCURRENCE_COVERAGE = { + "schema": "coverage/3", "unit": "occurrence", "source": "source_map", + "labels_in_text": ["3", "4", "7", "8", "9"], "covered": ["3", "4", "7", "9"], + "missing": ["8"], "labels_partially_covered": [], "labels_uncertain_only": [], + "occurrences": { + "total": 6, "covered": 4, "uncovered": 1, "uncertain": 1, + "items": [ + {"occ_id": "block_0020:40:0", "label": "8", "block": "block_0020", "page": 3, + "offset": 40, "group": "[8]", "section": "Methods", + "sentence": "Imaging followed the usual protocol [8].", + "covered_by": [], "status": "uncovered"}, + {"occ_id": "block_0031:12:0", "label": "4", "block": "block_0031", "page": 4, + "offset": 12, "group": "[4]", "section": "Results", + "sentence": "A second sentence citing the same source [4].", + "covered_by": [], "status": "uncertain"}, + ], + }, +} + + +def _results(claims=None, **kw) -> RunResults: + base = dict( + manuscript="paper.pdf", checker="claude -p · claude-opus-5", date="2026-09-26", + refs_total=5, refs_available=3, converter="pymupdf", + source_converters={"alpha-2020": "pymupdf", "beta-2021": "docling 2.118.1"}, + claims=claims if claims is not None else [ + _split_claim(), _unread_claim(), _withheld_claim(), _paraphrase_claim(), + ], + uncited=[UncitedClaim(id=1, claim="most clinics now screen", + quote="Most clinics now screen routinely.", location="Introduction")], + coverage=OCCURRENCE_COVERAGE, + truncated={"manuscript": {"chars": 90000, "limit": 60000}}, + ) + base.update(kw) + return RunResults(**base) + + +def _case(tmp_path, results=None, manifest="default", scout=None, images=True) -> Path: + case = tmp_path / "case" + (case / "out").mkdir(parents=True) + results = results if results is not None else _results() + results.to_json(case / "out" / "results.json") + manifest = _manifest() if manifest == "default" else manifest + if manifest is not None: + manifest.to_json(case / "refs_manifest.json") + if scout is not None: + scout.to_json(case / "out" / "scout.json") + if images: + for c in results.claims: + for a in [*c.judgements, c]: + for rel in [a.evidence_image, *a.continuation_images]: + if rel: + _png(case / "out" / rel) + (case / "out" / "report.md").write_text("# report\n") + return case + + +LIMITED_SCOPE = { + "claims": {"requested": [1, 2], "judged": [1, 2], "extracted": 4}, + "sources": {"cap": None, "selection": [1, 2], "skipped_for_claims": ["8"], + "skipped_by_cap": []}, +} + + +# -------------------------------------------------------------------------- +# what the server advertises +# -------------------------------------------------------------------------- + + +def test_every_read_tool_is_marked_read_only_and_closed_world(): + tools = _tools() + + assert READ_TOOLS <= set(tools) + for name in READ_TOOLS: + ann = tools[name].annotations + assert ann.read_only_hint is True, name + assert ann.open_world_hint is False, name + + +def _enum(schema): + """The first `enum` anywhere in a JSON schema fragment.""" + if isinstance(schema, dict): + if "enum" in schema: + return schema["enum"] + schema = list(schema.values()) + for part in schema if isinstance(schema, list) else []: + if (found := _enum(part)) is not None: + return found + return None + + +def test_the_filters_are_the_models_own_vocabularies(): + """A verdict or status spelled here a second time is a second vocabulary to + keep in step, and the one place it is defined is `models.py`.""" + tools = _tools() + + verdict = tools["list_claims"].input_schema["properties"]["verdict"] + status = tools["list_references"].input_schema["properties"]["status"] + assert _enum(verdict) == list(VERDICTS) + assert _enum(status) == list(REF_STATUSES) + for name in READ_TOOLS - {"get_evidence"}: + assert tools[name].output_schema, f"{name} publishes no output schema" + + +# -------------------------------------------------------------------------- +# audit_summary — every run-level disclosure reaches the model +# -------------------------------------------------------------------------- + + +@pytest.mark.parametrize("coverage", [ + OCCURRENCE_COVERAGE, + {"labels_in_text": ["1", "2"], "covered": ["1"], "missing": ["2"]}, + {"labels_in_text": [], "covered": [], "missing": []}, +]) +def test_audit_summary_carries_every_run_disclosure(tmp_path, coverage): + results = _results(coverage=coverage, scope=LIMITED_SCOPE) + case = _case(tmp_path, results=results) + + body = _text(_call("audit_summary", {"case": str(case)})) + + fired = run_disclosures(results, _manifest()) + assert {d.key for d in fired} >= { + "scope", "truncation", "converter", "source_fidelity", "source_identity", + "numbering", "labels_disputed", + } + for d in fired: + assert d.token in body, f"{d.key}: token {d.token!r} never reaches the MCP reader" + + +def test_audit_summary_states_the_counts_the_file_holds(tmp_path): + case = _case(tmp_path) + + summary = _json(_call("audit_summary", {"case": str(case)})) + + assert summary["counts"] == _results().counts() + assert summary["checker"] == "claude -p · claude-opus-5" + assert summary["references"] == {"total": 5, "available": 3} + assert summary["uncited_assertions"] == 1 + assert summary["case"] == str(case.resolve()) + assert str(case.resolve() / "out" / "report.md") in summary["reports"] + + +def test_a_limited_audit_is_stated_first_and_last(tmp_path): + """`CLAUDE.md`: the scope is stated with the caveats at the top AND at the + end of every format. The JSON a model reads has an end too.""" + case = _case(tmp_path, results=_results(scope=LIMITED_SCOPE)) + + summary = _json(_call("audit_summary", {"case": str(case)})) + listing = _json(_call("list_claims", {"case": str(case)})) + + assert summary["disclosures"][0]["key"] == "scope" + for result in (summary, listing): + assert list(result)[-1] == "limited" + assert SCOPE_TOKEN in result["limited"] + + +def test_a_whole_paper_audit_says_nothing_was_left_out(tmp_path): + case = _case(tmp_path) + + assert _json(_call("audit_summary", {"case": str(case)}))["limited"] is None + assert _json(_call("list_claims", {"case": str(case)}))["limited"] is None + + +def test_a_case_without_results_is_refused_rather_than_reported_empty(tmp_path): + """Absent is not zero. Empty counts for a case nobody checked would read as + an audit that found nothing.""" + case = tmp_path / "case" + case.mkdir() + + result = _call("audit_summary", {"case": str(case)}) + + assert result.is_error + assert "results.json" in _text(result) + assert "counts" not in _text(result) + + +def test_an_unreadable_results_file_is_refused_with_its_reason(tmp_path): + case = tmp_path / "case" + (case / "out").mkdir(parents=True) + (case / "out" / "results.json").write_text("{ half a file") + + result = _call("audit_summary", {"case": str(case)}) + + assert result.is_error + assert "unreadable" in _text(result) + + +def test_a_relative_case_path_is_resolved_and_said_back(tmp_path, monkeypatch): + """A host starts the server in a working directory of its own choosing, so a + relative path is echoed back resolved — the model can see where it looked.""" + _case(tmp_path) + monkeypatch.chdir(tmp_path) + + summary = _json(_call("audit_summary", {"case": "case"})) + + assert summary["case"] == str((tmp_path / "case").resolve()) + + +# -------------------------------------------------------------------------- +# list_claims and get_claim — every claim-level disclosure reaches the model +# -------------------------------------------------------------------------- + + +def test_list_claims_rows_carry_their_caveats(tmp_path): + case = _case(tmp_path) + + rows = _json(_call("list_claims", {"case": str(case)}))["claims"] + + assert [r["id"] for r in rows] == [1, 2, 3, 4] + assert [r["verdict"] for r in rows] == ["contradicted", "not_retrieved", "unchecked", + "supported"] + for c, row in zip(_results().claims, rows, strict=True): + for d in claim_disclosures(c, _manifest()): + assert any(d.token in line for line in row["caveats"]), (c.id, d.key) + + +def test_list_claims_filters_on_the_headline_verdict(tmp_path): + """Claim 1 has a supporting source, and its headline is the contradiction: + filtering on `supported` must not return it as if it were supported.""" + case = _case(tmp_path) + + listing = _json(_call("list_claims", {"case": str(case), "verdict": "supported"})) + + assert [r["id"] for r in listing["claims"]] == [4] + assert listing["claims_total"] == 4 + + +@pytest.mark.parametrize("claim", [ + _split_claim(), _unread_claim(), _withheld_claim(), + _paraphrase_claim(True), _paraphrase_claim(False), _paraphrase_claim(None), +], ids=["split", "unread", "withheld", "anchor-located", "anchor-not-located", "anchor-unknown"]) +def test_every_claim_disclosure_reaches_get_claim(tmp_path, claim): + case = _case(tmp_path, results=_results(claims=[claim])) + + body = _text(_call("get_claim", {"case": str(case), "claim_id": claim.id})) + + fired = claim_disclosures(claim, _manifest()) + fired += [d for j in claim.judgements for d in judgement_disclosures(j)] + assert fired, "a fixture that fires nothing proves nothing" + for d in fired: + assert d.token in body, f"claim {claim.id} {d.key}: {d.token!r} never reaches the model" + + +def test_get_claim_returns_each_source_judgement_with_its_rationale(tmp_path): + case = _case(tmp_path) + + detail = _json(_call("get_claim", {"case": str(case), "claim_id": 1})) + + assert detail["verdict"] == "contradicted" + assert detail["quote"] == "The model reached an AUC of 0.94 [3, 4]." + by_source = {j["source"]: j for j in detail["judgements"]} + assert by_source["beta-2021"]["rationale"] == "The source reports 0.77, not 0.94." + assert by_source["beta-2021"]["page"] == 5 + assert by_source["beta-2021"]["anchor_located"] is False + assert by_source["alpha-2020"]["verdict"] == "supported" + + +def test_get_claim_names_the_ids_that_exist_when_asked_for_one_that_does_not(tmp_path): + case = _case(tmp_path) + + result = _call("get_claim", {"case": str(case), "claim_id": 99}) + + assert result.is_error + assert "99" in _text(result) and "1–4" in _text(result) + + +# -------------------------------------------------------------------------- +# get_evidence — the crops themselves, never a picture the tool did not cut +# -------------------------------------------------------------------------- + + +def _images(result) -> list: + return [b for b in result.content if getattr(b, "type", "") == "image"] + + +def test_get_evidence_returns_the_headline_crops_with_their_anchor_caption(tmp_path): + """The headline is beta-2021's contradiction, whose passage crosses a break + and whose anchor was not boxed — two images, captioned as unboxed.""" + case = _case(tmp_path) + + result = _call("get_evidence", {"case": str(case), "claim_id": 1}) + + assert not result.is_error, _text(result) + images = _images(result) + assert len(images) == 2 + assert {i.mime_type for i in images} == {"image/png"} + caption = _text(result) + assert "beta-2021" in caption + for d in judgement_disclosures(_split_claim().judgements[1]): + assert d.token in caption + + +def test_get_evidence_for_a_named_source_shows_that_source(tmp_path): + case = _case(tmp_path) + + result = _call("get_evidence", {"case": str(case), "claim_id": 1, "source": "alpha-2020"}) + + assert len(_images(result)) == 1 + assert "alpha-2020" in _text(result) + + +def test_get_evidence_for_an_unread_source_says_nothing_was_read(tmp_path): + case = _case(tmp_path) + + result = _call("get_evidence", {"case": str(case), "claim_id": 2}) + + assert result.is_error + assert not _images(result) + assert "not retrieved" in _text(result) + + +def test_get_evidence_refuses_an_image_outside_the_case(tmp_path): + """`results.json` is a file anyone can edit. A crop path that resolves + outside `/out/` is refused, never followed.""" + elsewhere = tmp_path / "elsewhere.png" + _png(elsewhere) + claim = _paraphrase_claim(True) + claim.evidence_image = "../../elsewhere.png" + case = _case(tmp_path, results=_results(claims=[claim]), images=False) + + result = _call("get_evidence", {"case": str(case), "claim_id": 4}) + + assert result.is_error + assert not _images(result) + assert "outside" in _text(result) + + +def test_get_evidence_names_a_crop_that_was_never_written(tmp_path): + case = _case(tmp_path, images=False) + + result = _call("get_evidence", {"case": str(case), "claim_id": 4}) + + assert result.is_error + assert "claim_04_alpha-2020_p1.png" in _text(result) + + +# -------------------------------------------------------------------------- +# list_references — the manifest, as recorded +# -------------------------------------------------------------------------- + + +def test_list_references_carries_no_local_path(tmp_path): + """The viewer strips `pdf_path` for the same reason: where a PDF sits on the + auditor's disk is not a finding about the paper.""" + case = _case(tmp_path) + + body = _text(_call("list_references", {"case": str(case)})) + + assert "/private/home/someone" not in body + assert "pdf_path" not in body + + +def test_three_state_numbering_fields_keep_their_null(tmp_path): + """`[]` is a measurement and `null` is "never computed". Coercing one into + the other is the defect `CLAUDE.md` names twice.""" + measured = _manifest() + never = _manifest() + never.labels_uncomparable = None + never.numbering_corroborated = None + measured.numbering_corroborated = False + + for manifest, uncomparable, corroborated in ((measured, [], False), (never, None, None)): + case = _case(tmp_path / str(uncomparable), manifest=manifest) + numbering = _json(_call("list_references", {"case": str(case)}))["numbering"] + assert numbering["labels_uncomparable"] == uncomparable + assert numbering["corroborated"] is corroborated + assert numbering["verified"] is False + assert numbering["labels_disputed"] == ["9"] + + +def test_a_skipped_reference_says_why_it_was_skipped(tmp_path): + case = _case(tmp_path) + + listing = _json(_call("list_references", {"case": str(case), "status": "skipped"})) + + assert [r["label"] for r in listing["references"]] == ["8"] + assert listing["references"][0]["skipped_by"] == "claims" + assert listing["references"][0]["reason"] == "no selected claim cites it" + assert listing["by_status"]["skipped"] == 1 + assert listing["total"] == 5 + + +# -------------------------------------------------------------------------- +# list_gaps — the gap register is part of the result +# -------------------------------------------------------------------------- + + +def test_list_gaps_lists_every_register(tmp_path): + case = _case(tmp_path) + + gaps = _json(_call("list_gaps", {"case": str(case)})) + + assert [u["quote"] for u in gaps["uncited_assertions"]] == ["Most clinics now screen routinely."] + assert [(o["label"], o["status"]) for o in gaps["unreached_citations"]] == [ + ("8", "uncovered"), ("4", "uncertain"), + ] + assert gaps["unreached_labels"] == ["8"] + assert [c["id"] for c in gaps["unchecked_claims"]] == [2, 3] + assert "reached by an extracted claim" in gaps["coverage"] + + +def test_a_coverage_audit_that_never_ran_is_null_not_empty(tmp_path): + """No bracketed labels means the audit could not run — `null`, and the + caveat saying so — never an empty list that reads as nothing missed.""" + case = _case(tmp_path, results=_results( + coverage={"labels_in_text": [], "covered": [], "missing": []})) + + gaps = _json(_call("list_gaps", {"case": str(case)})) + + assert gaps["coverage"] is None + assert gaps["unreached_labels"] is None + assert gaps["unreached_citations"] is None + assert any(d["key"] == "coverage_caveat" for d in gaps["disclosures"]) + + +# -------------------------------------------------------------------------- +# get_scout — candidates, with the scout's own caveat +# -------------------------------------------------------------------------- + + +def test_get_scout_carries_its_caveat(tmp_path): + scout = ScoutResults( + paper_title="A paper", paper_year=2022, resolved_via="doi", paper_identity="unverified", + newer=[ScoutHit(title="A later paper", year=2024, doi="10.1/later", via="citing")], + ) + case = _case(tmp_path, scout=scout) + + result = _json(_call("get_scout", {"case": str(case)})) + + assert [h["title"] for h in result["newer"]] == ["A later paper"] + assert result["paper"]["identity"] == "unverified" + assert "absence from these lists proves nothing" in result["caveat"] + + +def test_get_scout_without_a_scan_is_refused(tmp_path): + case = _case(tmp_path) + + result = _call("get_scout", {"case": str(case)}) + + assert result.is_error + assert "scout.json" in _text(result) From 330b2345e4f15eab8698f13b2be1d6145b246299 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 26 Sep 2026 12:31:46 +0000 Subject: [PATCH 05/13] =?UTF-8?q?mcp:=20start=20an=20audit=20and=20follow?= =?UTF-8?q?=20it=20=E2=80=94=20a=20job,=20with=20nobody=20to=20ask?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit start_audit runs `papertrace run`'s pipeline (via cli._run_pipeline) on a background thread and returns at once; audit_status reports running, finished or failed with the log tail, long-polling at most 50 s. An audit takes minutes and a TypeScript-SDK host times a request out at 60 s, so a blocking call would fail while the audit carried on spending unseen. What the job guarantees, each asserted from inside it by a test that fakes a terminal on both streams: - nobody can be asked: stdin is an empty stream for the job, so cli._interactive() is False and a prompt reads end-of-file. A disputed reference label is withheld with chosen_by "default", as the CLI does without a terminal; the gate on settling one is a person, and a host may be a model. - no model from an earlier audit lingers: ask.forget_models() first. - the pipeline's console becomes the job's log instead of reaching stdout, which under stdio is the protocol's wire. One audit at a time, because the console, the model record and stdin are process-global; a second start_audit is refused naming the running one, and read tools refuse a case whose audit is running, since what is on disk then belongs to an earlier run. Everything that can refuse — no manuscript, no `claude` on the server's PATH, no contact email, a case folder holding another paper — refuses before anything is spent. A failed job keeps the reason the pipeline printed, or the exception's type and message, which SDK v2 would otherwise reduce to a bare "Error executing tool". A finished status carries no counts: they travel only with their disclosures, in audit_summary. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01J6JtZg47dHEKHDEfAJ5Kpf --- src/papertrace/mcp_server.py | 371 +++++++++++++++++++++++++++++++++-- tests/test_mcp_server.py | 356 ++++++++++++++++++++++++++++++++- 2 files changed, 715 insertions(+), 12 deletions(-) diff --git a/src/papertrace/mcp_server.py b/src/papertrace/mcp_server.py index 45d3cb1..53899bb 100644 --- a/src/papertrace/mcp_server.py +++ b/src/papertrace/mcp_server.py @@ -10,14 +10,23 @@ # SDK builds each tool's input and output schema from its annotations when the # tool is registered, and live types are what it reads without guessing. +import datetime +import io +import sys +import threading +from collections.abc import Callable, Iterator +from contextlib import contextmanager +from dataclasses import dataclass, field from pathlib import Path from typing import Annotated, Any, Literal +import typer from mcp.server import MCPServer from mcp.server.mcpserver import Image from mcp.server.mcpserver.exceptions import ToolError from mcp.types import TextContent, ToolAnnotations from pydantic import Field +from rich.console import Console # Not `typing.TypedDict`: below Python 3.12 pydantic cannot build a schema from # it, and the SDK then drops the tool to unvalidated text without a word — every @@ -25,7 +34,8 @@ # `structured_output=True` on each tool turns any repeat into an error at start. from typing_extensions import TypedDict -from . import __version__ +from . import __version__, cli +from .ask import forget_models from .disclosures import ( anchor_disclosure, claim_disclosures, @@ -47,7 +57,7 @@ # the viewer's serialisation of a disclosure, reused rather than restated: one # shape for every reader that receives disclosures as data -from .report import _disclosure_dict +from .report import FORMATS, _disclosure_dict INSTRUCTIONS = """\ PaperTrace checks what a scientific paper claims against what its cited sources say. @@ -75,11 +85,24 @@ # read tools change nothing and consult nothing outside the case folder _READ = ToolAnnotations(read_only_hint=True, open_world_hint=False) +# The one tool that spends says all of it, because a host decides on these +# whether to ask first: it rewrites the case folder, queries the network and +# calls a model, and no two runs of a model are the same run. +_SPEND = ToolAnnotations( + read_only_hint=False, destructive_hint=True, idempotent_hint=False, open_world_hint=True +) -# The vocabularies `models.py` defines, not copies of them. `Literal` over the -# tuple is the same type as the values spelled out. +# The vocabularies `models.py` and `report.py` define, not copies of them. +# `Literal` over the tuple is the same type as the values spelled out. Verdict = Literal[VERDICTS] RefStatus = Literal[REF_STATUSES] +Format = Literal[FORMATS] +# No constant holds this one; `ingest.resolve_backend` refuses anything else, +# and a test holds the two together. +Backend = Literal["auto", "docling", "pymupdf"] + +# A TypeScript-SDK host gives a request 60 s; a status call waits for less. +MAX_WAIT_SECONDS = 50 CaseArg = Annotated[ str, @@ -301,6 +324,21 @@ class ScoutOut(TypedDict): caveat: str +class AuditStatus(TypedDict): + """Where an audit this server started stands. Never its counts: those travel + only with the disclosures that qualify them, in audit_summary.""" + + case: str + state: Literal["running", "finished", "failed", "not_started"] + manuscript: str | None + started: str | None + ended: str | None + error: str + log: list[str] + log_lines_total: int + next: str + + # -------------------------------------------------------------------------- # reading a case folder — every failure is a sentence, never an empty result # -------------------------------------------------------------------------- @@ -507,6 +545,202 @@ def _scout_caveat(scout: ScoutResults) -> str: return lead + SCOUT_CAVEAT[0].upper() + SCOUT_CAVEAT[1:] +# -------------------------------------------------------------------------- +# the audit job — the pipeline, in this process, with nobody to ask +# -------------------------------------------------------------------------- + + +def _now() -> str: + return datetime.datetime.now(datetime.timezone.utc).isoformat(timespec="seconds") + + +class _Lines(io.TextIOBase): + """A text stream kept as whole lines — the file the pipeline's console writes to.""" + + def __init__(self, lines: list[str]): + self._lines = lines + self._open = "" + self._lock = threading.Lock() + + def writable(self) -> bool: + return True + + def write(self, s: str) -> int: + with self._lock: + *whole, self._open = (self._open + s).split("\n") + self._lines.extend(whole) + return len(s) + + def close(self) -> None: + with self._lock: + if self._open: # a last line printed without its newline is still said + self._lines.append(self._open) + self._open = "" + super().close() + + +@contextmanager +def _console_into(lines: list[str]) -> Iterator[list[str]]: + """What the pipeline prints, as lines of text instead of output on stdout. + + Every stage prints through the module global `cli.console`, which writes to + stdout — under stdio, the protocol's own wire. Swapped for the length of one + call and handed back, so a job's log is exactly what the CLI would have shown. + """ + sink = _Lines(lines) + saved = cli.console + cli.console = Console(file=sink, force_terminal=False, no_color=True, soft_wrap=True) + try: + yield lines + finally: + cli.console = saved + sink.close() + + +@contextmanager +def _nobody_to_ask() -> Iterator[None]: + """Nobody is at an MCP call to answer a question, so nothing may wait for one. + + With stdin an empty stream, `cli._interactive()` is False and any prompt + reads end-of-file at once. A disputed reference label is then withheld with + `numbering_chosen_by: "default"` — the CLI's own answer when no terminal is + attached — and the gate on settling it, which is a person shown the + disagreement, is never reached by a host that may be a model. + """ + saved = sys.stdin + sys.stdin = io.StringIO() + try: + yield + finally: + sys.stdin = saved + + +def _last_said(lines: list[str], n: int = 4) -> str: + """The last lines a stage printed — where the CLI says why it stopped.""" + return " ".join(line.strip() for line in [x for x in lines if x.strip()][-n:]) + + +@dataclass +class _Job: + """One audit this server process started.""" + + case: Path + manuscript: Path + started: str + state: str = "running" # running | finished | failed + ended: str | None = None + error: str = "" + log: list[str] = field(default_factory=list) + done: threading.Event = field(default_factory=threading.Event) + + +class _Jobs: + """The audits this server process started — at most one running at a time. + + One, because what a run touches is process-global: `cli.console`, `ask`'s + per-site model record and `sys.stdin`. Two audits sharing them would + interleave their logs and name each other's models. + """ + + def __init__(self): + self._lock = threading.Lock() # held across refuse, prepare and launch + self._running: _Job | None = None + self._by_case: dict[Path, _Job] = {} + + def last(self, case: Path) -> _Job | None: + return self._by_case.get(case) + + def refuse_running(self, case: Path) -> None: + """A case whose audit is running has nothing on disk that belongs to it.""" + job = self._running + if job is not None and job.case == case: + raise ToolError( + f"an audit of {case} is running now (started {job.started}) — what is on " + "disk belongs to an earlier run or is half-written, so nothing is read from " + "it until the audit ends. Call audit_status." + ) + + def start(self, prepare: Callable[[], tuple[Path, Path, dict]]) -> _Job: + """Run `prepare` — every check that can refuse before anything is spent — + then the pipeline on its own thread, and return without waiting for it.""" + with self._lock: + if (running := self._running) is not None: + raise ToolError( + f"an audit is already running in this server: {running.case} (started " + f"{running.started}). One audit runs at a time — follow it with " + "audit_status, and start this one when it has ended." + ) + case, manuscript, kwargs = prepare() + job = _Job(case=case, manuscript=manuscript, started=_now()) + self._running = self._by_case[case] = job + # daemon: when the host stops the server, the audit stops with it + # rather than spending on for a client that has gone + threading.Thread( + target=self._run, args=(job, kwargs), name="papertrace-audit", daemon=True + ).start() + return job + + def _run(self, job: _Job, kwargs: dict) -> None: + state, error = "failed", "" + try: + with _console_into(job.log), _nobody_to_ask(): + # the process outlives one audit: a model an earlier audit + # recorded must not be reported as this one's judge + forget_models() + cli._run_pipeline(**kwargs) + state = "finished" + except typer.Exit as e: + if e.exit_code == 0: + state = "finished" + else: + error = _last_said(job.log) or f"the pipeline stopped (exit {e.exit_code})" + except Exception as e: + # Broad on purpose, and recorded rather than swallowed: an exception + # leaving a tool reaches a model as a bare "Error executing tool", and + # one leaving this thread would leave the job "running" forever. The + # type and message are the finding, so the job keeps both. + error = f"{type(e).__name__}: {str(e)[:400]}" + finally: + if state == "failed" and not error: + error = "the audit was interrupted before it could finish" + job.state, job.error, job.ended = state, error, _now() + with self._lock: + if self._running is job: + self._running = None + job.done.set() # last: whoever wakes on it sees the final state + + +_NEXT = { + "running": "call audit_status again, with wait_seconds up to 50, until the audit has ended", + "finished": "call audit_summary for the result — its counts travel with the caveats that " + "qualify them", + "failed": "read `error` and the log, fix what they name, and call start_audit again", + "not_started": "no audit of this case was started by this server process — audit_summary " + "reads whatever an earlier run left on disk", +} + + +def _status(case: Path, job: _Job | None, log_lines: int) -> AuditStatus: + if job is None: + return { + "case": str(case), "state": "not_started", "manuscript": None, "started": None, + "ended": None, "error": "", "log": [], "log_lines_total": 0, + "next": _NEXT["not_started"], + } + log = list(job.log) + return { + "case": str(job.case), + "state": job.state, + "manuscript": str(job.manuscript), + "started": job.started, + "ended": job.ended, + "error": job.error, + "log": log[-log_lines:] if log_lines else [], + "log_lines_total": len(log), + "next": _NEXT[job.state], + } + + # -------------------------------------------------------------------------- # the server # -------------------------------------------------------------------------- @@ -515,13 +749,20 @@ def _scout_caveat(scout: ScoutResults) -> str: def build_server() -> MCPServer: """A fresh server with every tool registered. Nothing is served until `run()`.""" server = MCPServer("papertrace", instructions=INSTRUCTIONS, version=__version__) + jobs = _Jobs() + + def _open(case: str) -> Path: + """A case folder to read — unless this server is writing it right now.""" + root = _case_dir(case) + jobs.refuse_running(root) + return root @server.tool(title="Audit summary", annotations=_READ, structured_output=True) def audit_summary(case: CaseArg) -> AuditSummary: """The result of one audit: verdict counts, references obtained, the model that judged, coverage, and every run-level disclosure — relay those with the counts. `limited` is set when the audit was cut short on request.""" - root = _case_dir(case) + root = _open(case) results = _results(root) manifest = _manifest(root, required=False) fired = run_disclosures(results, manifest) @@ -553,7 +794,7 @@ def list_claims( """Every checked claim, one row each: its headline verdict, the paper's own sentence, the labels it cites, and its caveats. A claim citing several sources is headlined by the most adverse verdict; get_claim shows each source's.""" - root = _case_dir(case) + root = _open(case) results = _results(root) manifest = _manifest(root, required=False) rows = [_claim_row(c, manifest) for c in results.claims if verdict in (None, c.verdict)] @@ -570,7 +811,7 @@ def get_claim(case: CaseArg, claim_id: ClaimId) -> ClaimDetail: """One claim in full: the paper's sentence, each cited source's verdict with its rationale, page, block and anchor phrases, and every disclosure the claim owes its reader. get_evidence shows the passage itself.""" - root = _case_dir(case) + root = _open(case) results = _results(root) manifest = _manifest(root, required=False) c = _find_claim(results, claim_id) @@ -611,7 +852,7 @@ def get_evidence( """The page crop a verdict rests on, as images, with the matched text boxed in red. The boxes were drawn by code finding the model's anchor phrases in the PDF, not placed by the model; the caption says whether the phrase was found.""" - root = _case_dir(case) + root = _open(case) claim = _find_claim(_results(root), claim_id) target = _evidence_target(claim, source) rels = _images_of(target) @@ -644,7 +885,7 @@ def list_references( not, and the state of the reference numbering — the label is what joins a claim to its source, so an unconfirmed or disputed numbering is a caveat on every verdict that rests on it. Local file paths are not included.""" - root = _case_dir(case) + root = _open(case) m = _manifest(root, required=True) rows: list[ReferenceRow] = [ { @@ -701,7 +942,7 @@ def list_gaps(case: CaseArg) -> Gaps: places no extracted claim reached, and claims never checked (source not retrieved, or no verdict reached). `null` means that audit did not run — not that it found nothing.""" - root = _case_dir(case) + root = _open(case) results = _results(root) manifest = _manifest(root, required=False) coverage = results.coverage or {} @@ -740,7 +981,7 @@ def get_scout(case: CaseArg) -> ScoutOut: """Literature around the paper from Europe PMC: work published since, work that existed before it and went uncited, and work from the same year. Search-based: candidates for a person's judgement, and absence proves nothing.""" - root = _case_dir(case) + root = _open(case) path = root / "out" / "scout.json" if not path.is_file(): raise ToolError( @@ -750,4 +991,112 @@ def get_scout(case: CaseArg) -> ScoutOut: scout = _read(path, ScoutResults.from_json, "scout") return {"case": str(root), **scout.to_dict(), "caveat": _scout_caveat(scout)} + @server.tool(title="Start an audit", annotations=_SPEND, structured_output=True) + def start_audit( + manuscript: Annotated[str, Field(description="The paper to audit, a PDF — by absolute path.")], + case: Annotated[str | None, Field(description=( + "Case folder to write the audit into. Default: a folder named after the paper, " + "beside it; an earlier audit of the same paper there is amended."))] = None, + email: Annotated[str | None, Field(description=( + "Contact email Unpaywall requires. Default: PAPERTRACE_EMAIL, or the address " + "`papertrace` saved."))] = None, + model: Annotated[str | None, Field(description=( + "Model for `claude -p`, e.g. claude-opus-5. Pin one for a reproducible audit; " + "default: the account's default model."))] = None, + backend: Annotated[Backend, Field(description=( + "How PDFs are read: docling is layout-aware (tables stay tables), pymupdf is flat " + "text; auto prefers docling."))] = "auto", + scout: Annotated[bool, Field(description=( + "Also scan Europe PMC for literature published since, or uncited."))] = True, + doi: Annotated[str | None, Field(description=( + "DOI of the paper itself. Default: the one printed on page 1."))] = None, + formats: Annotated[list[Format] | None, Field(description=( + "Report looks to write beside report.md; `viewer` is the interactive page."))] = None, + provided: Annotated[str | None, Field(description=( + "Folder of cited PDFs already at hand; each is identified by its own DOI or " + "title."))] = None, + supplements: Annotated[list[str] | None, Field(description=( + "The paper's own supplementary PDFs, by absolute path."))] = None, + llm_refs: Annotated[bool, Field(description=( + "Also have a model read the printed reference list as a further check on its " + "numbering (one more model call)."))] = True, + max_claims: Annotated[int | None, Field(ge=1, description=( + "Check only the first N claims; only the references they cite are fetched. The " + "result says what was left out."))] = None, + max_sources: Annotated[int | None, Field(ge=1, description=( + "Obtain and judge against at most N cited sources. The result says what was left " + "out."))] = None, + ) -> AuditStatus: + """Start a full audit of one paper and return at once; follow it with audit_status. + + The pipeline `papertrace run` runs: read the PDF, extract its citation-backed + claims, fetch the cited sources through legal open-access routes, scout the + literature, judge each claim against its cited pages, crop the evidence and write + the reports. It SPENDS: `claude -p` model calls on this machine's logged-in Claude + account (one to extract, one to read the reference list, one per cited source + judged, each retried on failure), and queries to Crossref, Unpaywall, Europe PMC + and arXiv. It takes minutes. One audit runs at a time per server. + + A reference label whose readings disagree is withheld, never settled here: that + needs a person shown the disagreement, at a terminal (`papertrace refs`).""" + pdf = _case_dir(manuscript) + if not pdf.is_file(): + raise ToolError(f"no manuscript at {pdf} — pass the paper's PDF by absolute path") + own = [_case_dir(s) for s in supplements or []] + if absent := [str(s) for s in own if not s.is_file()]: + raise ToolError(f"no supplement at {', '.join(absent)}") + pdfs = _case_dir(provided) if provided else None + if pdfs is not None and not pdfs.is_dir(): + raise ToolError(f"`provided` must be a folder of PDFs, and {pdfs} is not one") + + def prepare() -> tuple[Path, Path, dict]: + # everything that can refuse, refused before a model or a network is touched + from .check import claude_available + + if not claude_available(): + raise ToolError( + "the `claude` CLI is not on this server's PATH, and every claim is " + "judged with `claude -p` — install Claude Code and log in. If it is " + "installed, the host started this server with a PATH that does not " + "reach it: set PATH in the server's configuration." + ) + said: list[str] = [] + try: + with _console_into(said): + root = (_case_dir(case) if case else cli.default_case(pdf)).resolve() + address = cli._email(email) + cli._guard_case(root, pdf) # one case folder per paper + except typer.Exit: + raise ToolError( + "refused before anything was spent — " + _last_said(said, n=8) + ) from None + return root, pdf, { + "manuscript": pdf, "case": root, "provided": pdfs, "email": address, + "model": model, "png": False, "backend": backend, "with_scout": scout, + "doi": doi, "formats": list(formats or []), "supplement": own, + "llm_refs": llm_refs, "max_claims": max_claims, "max_sources": max_sources, + } + + job = jobs.start(prepare) + return _status(job.case, job, log_lines=40) + + @server.tool(title="Audit status", annotations=_READ, structured_output=True) + def audit_status( + case: CaseArg, + wait_seconds: Annotated[int, Field(ge=0, le=MAX_WAIT_SECONDS, description=( + "Wait up to this many seconds for a running audit to end before answering " + "(at most 50, inside the 60 s a host allows a request)."))] = 0, + log_lines: Annotated[int, Field(ge=0, le=1000, description=( + "How many of the audit's latest console lines to include."))] = 40, + ) -> AuditStatus: + """Where an audit this server started stands — running, finished or failed — with + the last lines it printed, which carry its warnings as it goes. Poll with + wait_seconds rather than in a tight loop. A finished audit's result is + audit_summary; this never repeats its counts without their caveats.""" + root = _case_dir(case) + job = jobs.last(root) + if job is not None and wait_seconds: + job.done.wait(wait_seconds) + return _status(root, job, log_lines) + return server diff --git a/tests/test_mcp_server.py b/tests/test_mcp_server.py index 49b1dd8..e808b29 100644 --- a/tests/test_mcp_server.py +++ b/tests/test_mcp_server.py @@ -23,9 +23,12 @@ pytest.importorskip("mcp", reason="pip install -e '.[dev]' supplies the MCP SDK") import anyio # noqa: E402 +import typer # noqa: E402 from mcp import Client # noqa: E402 -from papertrace import mcp_server # noqa: E402 +from papertrace import ask as ask_mod # noqa: E402 +from papertrace import check as check_mod # noqa: E402 +from papertrace import cli, mcp_server # noqa: E402 from papertrace.disclosures import ( # noqa: E402 SCOPE_TOKEN, claim_disclosures, @@ -625,3 +628,354 @@ def test_get_scout_without_a_scan_is_refused(tmp_path): assert result.is_error assert "scout.json" in _text(result) + + +# -------------------------------------------------------------------------- +# start_audit and audit_status — the one tool that spends, run as a job +# -------------------------------------------------------------------------- + + +def test_the_server_lists_nine_tools_and_marks_the_one_that_spends(): + tools = _tools() + + assert set(tools) == READ_TOOLS | {"start_audit", "audit_status"} + status = tools["audit_status"].annotations + assert status.read_only_hint is True and status.open_world_hint is False + # a host decides whether to ask before calling on these hints: the one tool + # that spends model calls, queries the network and rewrites a case folder + # must say all three + spend = tools["start_audit"].annotations + assert spend.read_only_hint is False + assert spend.destructive_hint is True + assert spend.open_world_hint is True + + +def test_every_backend_start_audit_offers_is_one_ingest_accepts(): + """The backend vocabulary has no constant to reuse, so the offer is checked + against the rule that enforces it: `resolve_backend` refuses anything else.""" + from papertrace.ingest import resolve_backend + + offered = _enum(_tools()["start_audit"].input_schema["properties"]["backend"]) + assert offered == ["auto", "docling", "pymupdf"] + for backend in offered: + resolve_backend(backend) + + +@pytest.fixture +def ready(monkeypatch, tmp_path): + """Everything start_audit checks before it spends, satisfied offline — and + never read from the machine: `_email` falls back to a SAVED CONFIG in the + developer's home, which is how a CLI test once passed locally and failed on + every CI python.""" + monkeypatch.setattr(check_mod, "claude_available", lambda: True) + monkeypatch.setenv("PAPERTRACE_EMAIL", "t@example.org") + monkeypatch.delenv("MANUSCRIPTAGENT_EMAIL", raising=False) + monkeypatch.setenv("PAPERTRACE_CONFIG", str(tmp_path / "no-saved-config.json")) + + +def _paper(tmp_path, name="paper.pdf") -> Path: + pdf = tmp_path / name + pdf.write_bytes(b"%PDF-1.4\n% never parsed: the pipeline is faked\n" + name.encode()) + return pdf + + +def _fake_pipeline(monkeypatch, body=None) -> list[dict]: + calls: list[dict] = [] + + def fake(**kw): + calls.append(kw) + if body is not None: + body(kw) + + monkeypatch.setattr(cli, "_run_pipeline", fake) + return calls + + +def _start(server, **args) -> dict: + return _json(_call("start_audit", {k: str(v) if isinstance(v, Path) else v + for k, v in args.items()}, server)) + + +def _await(server, case: str) -> dict: + return _json(_call("audit_status", {"case": case, "wait_seconds": 20}, server)) + + +def test_start_audit_refuses_a_manuscript_that_is_not_there(ready, monkeypatch, tmp_path): + calls = _fake_pipeline(monkeypatch) + + result = _call("start_audit", {"manuscript": str(tmp_path / "absent.pdf")}) + + assert result.is_error + assert "absent.pdf" in _text(result) + assert calls == [] + + +def test_start_audit_refuses_when_claude_is_not_on_its_path(ready, monkeypatch, tmp_path): + """A host starts the server with a PATH of its own choosing. Finding that + out after ingest has run would be finding it out late.""" + calls = _fake_pipeline(monkeypatch) + monkeypatch.setattr(check_mod, "claude_available", lambda: False) + + result = _call("start_audit", {"manuscript": str(_paper(tmp_path))}) + + assert result.is_error + assert "claude" in _text(result) and "PATH" in _text(result) + assert calls == [] + + +def test_start_audit_refuses_without_a_contact_email(ready, monkeypatch, tmp_path): + calls = _fake_pipeline(monkeypatch) + monkeypatch.delenv("PAPERTRACE_EMAIL") + + result = _call("start_audit", {"manuscript": str(_paper(tmp_path))}) + + assert result.is_error + assert "email" in _text(result).lower() + assert calls == [] + + +def test_start_audit_refuses_a_case_folder_holding_another_paper(ready, monkeypatch, tmp_path): + """One case folder per paper — `_guard_case`'s rule, applied before anything + is spent rather than after ingest and extraction already were.""" + calls = _fake_pipeline(monkeypatch) + case = tmp_path / "case" + case.mkdir() + RefManifest(manuscript="other.pdf", manuscript_sha256="f" * 64).to_json( + case / "refs_manifest.json") + + result = _call("start_audit", {"manuscript": str(_paper(tmp_path)), "case": str(case)}) + + assert result.is_error + assert "other.pdf" in _text(result) + assert calls == [] + + +def test_the_job_hands_the_pipeline_every_parameter(ready, monkeypatch, tmp_path): + import inspect + + params = set(inspect.signature(cli._run_pipeline).parameters) + calls = _fake_pipeline(monkeypatch) + pdf = _paper(tmp_path) + own = _paper(tmp_path, "paper-supplement.pdf") + provided = tmp_path / "my_pdfs" + provided.mkdir() + server = mcp_server.build_server() + + started = _start(server, manuscript=pdf, case=tmp_path / "case", model="claude-opus-5", + backend="pymupdf", scout=False, doi="10.1/x", formats=["viewer"], + provided=provided, supplements=[str(own)], llm_refs=False, + max_claims=3, max_sources=2) + assert _await(server, started["case"])["state"] == "finished" + + assert calls == [{ + "manuscript": pdf.resolve(), "case": (tmp_path / "case").resolve(), + "provided": provided.resolve(), "email": "t@example.org", "model": "claude-opus-5", + "png": False, "backend": "pymupdf", "with_scout": False, "doi": "10.1/x", + "formats": ["viewer"], "supplement": [own.resolve()], "llm_refs": False, + "max_claims": 3, "max_sources": 2, + }] + assert set(calls[0]) == params, "a pipeline parameter the job never decides on" + + +def test_start_audit_defaults_are_papertrace_runs(ready, monkeypatch, tmp_path): + """No case named: beside the paper, named after it, as `papertrace run` + does. Every other default is the CLI's own.""" + calls = _fake_pipeline(monkeypatch) + pdf = _paper(tmp_path) + server = mcp_server.build_server() + + started = _start(server, manuscript=pdf) + _await(server, started["case"]) + + assert started["case"] == str((tmp_path / "paper").resolve()) + assert calls[0] | {"manuscript": None, "case": None} == { + "manuscript": None, "case": None, "provided": None, "email": "t@example.org", + "model": None, "png": False, "backend": "auto", "with_scout": True, "doi": None, + "formats": [], "supplement": [], "llm_refs": True, "max_claims": None, + "max_sources": None, + } + + +class _Terminal: + """A stream that says it is a terminal — what `_interactive()` looks for.""" + + def __init__(self): + self.written = [] + + def isatty(self): + return True + + def write(self, s): + self.written.append(s) + return len(s) + + def flush(self): + pass + + def readline(self): + return "1\n" # a person answering the first menu, were one asked + + +def test_inside_the_job_nobody_can_be_asked_and_no_old_model_lingers( + ready, monkeypatch, tmp_path +): + """Three things the job guarantees, observed from inside it. + + A terminal on both streams and no `CI`: outside the job, `_interactive()` + would be True here, and a disputed label would be put to a person who is + not there. Inside it, nobody can be asked — a prompt reads end-of-file. + + A model left behind by an earlier audit in this process is gone, so a run + whose judging makes no calls cannot report that model as its judge. + + The pipeline's console lands in the job's log, never on stdout, which + under stdio is the protocol's own wire. + """ + terminal_in, terminal_out = _Terminal(), _Terminal() + monkeypatch.setattr(sys, "stdin", terminal_in) + monkeypatch.setattr(sys, "stdout", terminal_out) + monkeypatch.delenv("CI", raising=False) + assert cli._interactive(), "the fixture must look like a terminal, or this proves nothing" + monkeypatch.setattr(ask_mod, "_MODELS", {ask_mod.SITE_CHECK: "claude-haiku-4-5"}) + seen = {} + + def body(kw): + seen["interactive"] = cli._interactive() + try: + seen["answer"] = input() + except EOFError: + seen["answer"] = "end of file" + seen["model"] = ask_mod.model_for(ask_mod.SITE_CHECK) + cli.console.print("[green]✓[/green] said by the pipeline") + + _fake_pipeline(monkeypatch, body) + console = cli.console + server = mcp_server.build_server() + + status = _await(server, _start(server, manuscript=_paper(tmp_path))["case"]) + + assert status["state"] == "finished", status + assert seen == {"interactive": False, "answer": "end of file", "model": None} + assert "✓ said by the pipeline" in status["log"] + assert not any("said by the pipeline" in s for s in terminal_out.written) + assert cli.console is console, "the console must be handed back when the job ends" + + +def test_a_pipeline_that_stops_is_failed_with_the_reason_it_printed(ready, monkeypatch, tmp_path): + def body(kw): + cli.console.print("[red]refs_manifest.json not found[/red] — run `papertrace refs` first") + raise typer.Exit(1) + + _fake_pipeline(monkeypatch, body) + server = mcp_server.build_server() + + status = _await(server, _start(server, manuscript=_paper(tmp_path))["case"]) + + assert status["state"] == "failed" + assert "refs_manifest.json not found" in status["error"] + + +def test_an_unexpected_exception_is_failed_with_its_type_and_message( + ready, monkeypatch, tmp_path +): + """In SDK v2 an exception escaping a tool reaches the model as a bare + `Error executing tool`. A job records its reason instead of losing it.""" + def body(kw): + raise RuntimeError("the layout models could not be loaded") + + _fake_pipeline(monkeypatch, body) + server = mcp_server.build_server() + + status = _await(server, _start(server, manuscript=_paper(tmp_path))["case"]) + + assert status["state"] == "failed" + assert status["error"] == "RuntimeError: the layout models could not be loaded" + + +def test_a_second_audit_is_refused_while_one_runs(ready, monkeypatch, tmp_path): + """The console, the model record and stdin are process-global: two audits + sharing them would interleave logs and misattribute models.""" + import threading + + gate = threading.Event() + _fake_pipeline(monkeypatch, lambda kw: gate.wait(20)) + server = mcp_server.build_server() + first = _start(server, manuscript=_paper(tmp_path), case=tmp_path / "first") + + try: + refused = _call("start_audit", {"manuscript": str(_paper(tmp_path, "other.pdf")), + "case": str(tmp_path / "second")}, server) + assert refused.is_error + assert first["case"] in _text(refused) + finally: + gate.set() + assert _await(server, first["case"])["state"] == "finished" + + # and the slot is free again once it has ended + second = _start(server, manuscript=_paper(tmp_path, "other.pdf"), case=tmp_path / "second") + assert _await(server, second["case"])["state"] == "finished" + + +def test_read_tools_refuse_the_case_whose_audit_is_running(ready, monkeypatch, tmp_path): + """The results.json on disk then belongs to the previous run, or is half + written. Presenting it as this run's would be the silent failure.""" + import threading + + gate = threading.Event() + _fake_pipeline(monkeypatch, lambda kw: gate.wait(20)) + case = _case(tmp_path) + server = mcp_server.build_server() + RefManifest(manuscript="paper.pdf").to_json(case / "refs_manifest.json") + started = _start(server, manuscript=_paper(tmp_path), case=case) + + try: + for tool, args in (("audit_summary", {}), ("list_claims", {}), ("get_claim", {"claim_id": 1}), + ("get_evidence", {"claim_id": 1}), ("list_references", {}), + ("list_gaps", {}), ("get_scout", {})): + result = _call(tool, {"case": started["case"], **args}, server) + assert result.is_error, tool + assert "running" in _text(result), tool + finally: + gate.set() + _await(server, started["case"]) + assert not _call("audit_summary", {"case": started["case"]}, server).is_error + + +def test_audit_status_waits_for_the_end_and_no_longer(ready, monkeypatch, tmp_path): + import time + + _fake_pipeline(monkeypatch, lambda kw: time.sleep(0.3)) + server = mcp_server.build_server() + started = _start(server, manuscript=_paper(tmp_path)) + + t0 = time.monotonic() + status = _json(_call("audit_status", {"case": started["case"], "wait_seconds": 15}, server)) + + assert status["state"] == "finished" + assert time.monotonic() - t0 < 10 + # under the 60 s a TypeScript-SDK host allows a request, by schema + too_long = _call("audit_status", {"case": started["case"], "wait_seconds": 51}, server) + assert too_long.is_error + + +def test_a_finished_status_points_at_the_summary_and_carries_no_counts( + ready, monkeypatch, tmp_path +): + """A count without its disclosures is what every reader here is protected + from, so the status says where the result is rather than quoting it.""" + _fake_pipeline(monkeypatch) + server = mcp_server.build_server() + + status = _await(server, _start(server, manuscript=_paper(tmp_path))["case"]) + + assert "counts" not in status + assert "audit_summary" in status["next"] + + +def test_the_status_of_a_case_this_server_never_ran_says_so(tmp_path): + case = _case(tmp_path) + + status = _json(_call("audit_status", {"case": str(case)})) + + assert status["state"] == "not_started" + assert "this server" in status["next"] From fbe95c6a52e2e2a53b9665c3ac29346e55efe071 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 26 Sep 2026 12:34:38 +0000 Subject: [PATCH 06/13] cli: `papertrace mcp` serves the server over stdio, behind an [mcp] extra MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The command a host launches. It writes nothing to stdout — under stdio that is the protocol's wire, and a banner printed before serving begins corrupts the first message — and without the SDK it says how to install it on stderr and exits 2. Only the SDK's absence earns that line; any other missing module keeps its own traceback. `mcp>=2.2,<3` is an extra, and the same requirement is in [full] (the quick start's everything-install) and in [dev], which is exactly what CI installs: the MCP tests skip without the SDK, and a module that skips in CI is a module nobody runs. Two tests go through a real process, because the in-memory client skips exactly the part that can break here: one asserts zero bytes on stdout when stdin closes, the other launches the command with the SDK's stdio client, lists the nine tools and reads a case. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01J6JtZg47dHEKHDEfAJ5Kpf --- pyproject.toml | 9 +++-- src/papertrace/cli.py | 20 +++++++++++ src/papertrace/mcp_server.py | 9 +++++ tests/test_mcp_server.py | 64 ++++++++++++++++++++++++++++++++++++ tests/test_packaging.py | 21 +++++++++++- 5 files changed, 120 insertions(+), 3 deletions(-) diff --git a/pyproject.toml b/pyproject.toml index a987dde..0b2983d 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -39,14 +39,19 @@ dependencies = [ [project.optional-dependencies] png = ["playwright>=1.45"] +# `papertrace mcp`, for MCP hosts. The server is written against the SDK's 2.x +# API, and the SDK holds breaking changes for a major, hence the bound +mcp = ["mcp>=2.2,<3"] # kept as a no-op alias so `pip install papertrace[docling]` and the `[full]` # extra named all over the 0.4.x docs keep resolving instead of erroring docling = [] -full = ["playwright>=1.45"] +full = ["playwright>=1.45", "mcp>=2.2,<3"] dev = ["pytest>=8.0", "ruff>=0.5", "jsonschema>=4.0", # 3.10 has no tomllib, and without a parser the whole packaging # suite skipped on the oldest version requires-python allows - 'tomli>=2.0; python_version < "3.11"'] + 'tomli>=2.0; python_version < "3.11"', + # so CI runs tests/test_mcp_server.py instead of skipping it + "mcp>=2.2,<3"] [project.scripts] papertrace = "papertrace.cli:main" diff --git a/src/papertrace/cli.py b/src/papertrace/cli.py index 217440f..0b77147 100644 --- a/src/papertrace/cli.py +++ b/src/papertrace/cli.py @@ -450,6 +450,26 @@ def init( console.print(f" [dim]hand this folder to every step: [cyan]-c {case}[/cyan][/dim]") +@app.command("mcp", rich_help_panel="Utilities") +def mcp_command() -> None: + """Serve PaperTrace to an MCP host over stdio — Claude Desktop, Cursor, VS Code, Claude Code.""" + # Nothing may reach stdout from here: under stdio it is the protocol's wire, + # and a banner or a warning printed before serving begins corrupts it. + try: + from .mcp_server import serve + except ModuleNotFoundError as e: + # only the SDK's absence earns the install line; any other missing + # module is a real fault and says so with its own traceback + if (e.name or "").split(".")[0] != "mcp": + raise + # rich eats [mcp] as a style tag; escaping it is what prints the extra + Console(stderr=True).print( + r"[red]the MCP server needs the MCP SDK[/red] — pip install 'papertrace\[mcp]'" + ) + raise typer.Exit(2) from None + serve() + + def _ingest_pipeline( *, pdf: Path, diff --git a/src/papertrace/mcp_server.py b/src/papertrace/mcp_server.py index 53899bb..dc56fee 100644 --- a/src/papertrace/mcp_server.py +++ b/src/papertrace/mcp_server.py @@ -1100,3 +1100,12 @@ def audit_status( return _status(root, job, log_lines) return server + + +def serve() -> None: + """Serve over stdio until the host closes the connection. + + stdio only: a case folder stays on the machine that made it, and the host + that launched this process is the one client it answers. + """ + build_server().run("stdio") diff --git a/tests/test_mcp_server.py b/tests/test_mcp_server.py index e808b29..86c50ad 100644 --- a/tests/test_mcp_server.py +++ b/tests/test_mcp_server.py @@ -979,3 +979,67 @@ def test_the_status_of_a_case_this_server_never_ran_says_so(tmp_path): assert status["state"] == "not_started" assert "this server" in status["next"] + + +# -------------------------------------------------------------------------- +# `papertrace mcp` — the command a host launches, over real stdio +# -------------------------------------------------------------------------- + +SRC = str(Path(__file__).resolve().parent.parent / "src") + + +def test_papertrace_mcp_writes_nothing_to_stdout_when_its_stdin_closes(tmp_path): + """Under stdio, stdout IS the protocol. A banner, a warning or a stray print + before serving begins lands on the wire and corrupts the first message, and + no in-memory test can see it — only a real process can.""" + import os + import subprocess + + proc = subprocess.run( + [sys.executable, "-m", "papertrace", "mcp"], + stdin=subprocess.DEVNULL, capture_output=True, timeout=120, + env={**os.environ, "PYTHONPATH": SRC, + "PAPERTRACE_CONFIG": str(tmp_path / "no-saved-config.json")}, + ) + + assert proc.returncode == 0, proc.stderr.decode()[-2000:] + assert proc.stdout == b"", proc.stdout[:500] + + +def test_a_host_lists_the_tools_and_reads_a_case_over_real_stdio(tmp_path): + """The whole path a host takes: launch the command, speak MCP over its + pipes, read an audit. The in-memory client skips exactly this.""" + from mcp import StdioServerParameters + + case = _case(tmp_path) + params = StdioServerParameters( + command=sys.executable, args=["-m", "papertrace", "mcp"], + env={"PYTHONPATH": SRC, "PAPERTRACE_CONFIG": str(tmp_path / "no-saved-config.json")}, + ) + + async def go(client): + names = {t.name for t in (await client.list_tools()).tools} + return names, await client.call_tool("audit_summary", {"case": str(case)}) + + names, summary = _session(params, go) + + assert names == READ_TOOLS | {"start_audit", "audit_status"} + assert _json(summary)["counts"] == _results().counts() + + +def test_without_the_sdk_papertrace_mcp_says_how_to_install_it(monkeypatch, capsys): + """On stderr, and nothing on stdout — a host reading the wire would take + even an error message for a malformed protocol message.""" + # every cached `mcp.*`, not only `mcp`: an import finds a cached submodule + # without consulting its parent, so masking the package alone hides nothing + for name in [n for n in sys.modules if n == "mcp" or n.startswith("mcp.")]: + monkeypatch.setitem(sys.modules, name, None) + monkeypatch.delitem(sys.modules, "papertrace.mcp_server", raising=False) + + with pytest.raises(typer.Exit) as stop: + cli.mcp_command() + + out, err = capsys.readouterr() + assert stop.value.exit_code == 2 + assert out == "" + assert "pip install 'papertrace[mcp]'" in err diff --git a/tests/test_packaging.py b/tests/test_packaging.py index 5fc2e4c..0d2b80f 100644 --- a/tests/test_packaging.py +++ b/tests/test_packaging.py @@ -313,11 +313,30 @@ def test_docling_is_a_base_dependency(): assert "docling" in names, "docling must be a base dependency, not an extra" +def test_the_mcp_extra_exists_and_ci_installs_it(): + """`papertrace mcp` needs the MCP SDK, so the SDK is an extra, not a base + dependency. `[dev]` — exactly what CI installs — carries the same + requirement: `tests/test_mcp_server.py` skips without the SDK, and a module + that skips in CI is a module nobody runs. This file's own history is the + 3.10 cell that silently skipped all 26 of its tests while CI was green. + + `<3` because the server is written against the SDK's 2.x API (`MCPServer`, + `Client(server)`), and the SDK's versioning policy holds breaking changes + for a major. `[full]` is the everything-install the quick start uses.""" + extras = PYPROJECT["project"]["optional-dependencies"] + sdk = [r for r in extras.get("mcp", []) if re.split(r"[<>=!~;\[ ]", r)[0] == "mcp"] + assert sdk, "no `mcp` extra carrying the MCP SDK" + assert "<3" in sdk[0], f"{sdk[0]!r} admits a new major of the SDK" + for name in ("dev", "full"): + assert sdk[0] in extras[name], f"[{name}] must carry {sdk[0]!r} as the [mcp] extra does" + + def test_the_docling_and_full_extras_still_resolve(): """`pip install 'papertrace[docling]'` and `[full]` appear throughout the 0.4.x docs and in other people's notes. Deleting the extras would turn those into an install error, so they stay as aliases — `[docling]` empty - because it is now redundant, `[full]` keeping only playwright.""" + because it is now redundant, `[full]` keeping playwright (and, since the MCP + server, the SDK — asserted above).""" extras = PYPROJECT["project"]["optional-dependencies"] assert "docling" in extras and extras["docling"] == [] full = [d.split(">")[0].strip() for d in extras["full"]] From 7ee8d7d74ea98b506aa175d8bd8e29f7633c20f9 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 26 Sep 2026 12:35:48 +0000 Subject: [PATCH 07/13] tests: the audit job runs the real pipeline body, and nothing leaks to stdout MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The other job tests replace _run_pipeline whole. This one stubs only its seven stages, so the real body runs inside the job — email, case, guard, the case folder's own .gitignore, banner, stage order, closing line — and reads the process's real stdout for anything printed around the captured console. Integration coverage for the job already committed, not a new behaviour: it passed on first run, and fails when the console capture is removed (checked by mutation, as were the stdin swap and the model reset). Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01J6JtZg47dHEKHDEfAJ5Kpf --- tests/test_mcp_server.py | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/tests/test_mcp_server.py b/tests/test_mcp_server.py index 86c50ad..6f51ca6 100644 --- a/tests/test_mcp_server.py +++ b/tests/test_mcp_server.py @@ -777,6 +777,34 @@ def test_the_job_hands_the_pipeline_every_parameter(ready, monkeypatch, tmp_path assert set(calls[0]) == params, "a pipeline parameter the job never decides on" +def test_the_job_runs_the_real_pipeline_body_and_keeps_what_it_prints( + ready, monkeypatch, tmp_path, capfd +): + """The other job tests replace `_run_pipeline` whole. Here only its seven + stages are stubbed, so the real body runs inside the job — email, case, + guard, the case folder's own .gitignore, the banner, the stage order and + the closing line — and anything it printed around `cli.console` would + reach the process's real stdout, which this test reads.""" + order = [] + for name in ("_ingest_pipeline", "_extract_pipeline", "_refs_pipeline", "scout", + "_check_pipeline", "highlight", "_report_pipeline"): + monkeypatch.setattr(cli, name, lambda _n=name, **kw: order.append(_n)) + monkeypatch.setattr(cli, "_detected_doi", lambda m: None) + server = mcp_server.build_server() + case = tmp_path / "case" + + status = _await(server, _start(server, manuscript=_paper(tmp_path), case=case, + formats=["viewer"])["case"]) + + assert status["state"] == "finished", status + assert order == ["_ingest_pipeline", "_extract_pipeline", "_refs_pipeline", "scout", + "_check_pipeline", "highlight", "_report_pipeline"] + assert (case / ".gitignore").read_text() == "*\n" + assert any("PaperTrace" in line for line in status["log"]), "the banner is in the log" + assert any("report_viewer.html" in line for line in status["log"]) + out, _ = capfd.readouterr() + assert out == "", "the real pipeline printed around the captured console" + def test_start_audit_defaults_are_papertrace_runs(ready, monkeypatch, tmp_path): """No case named: beside the paper, named after it, as `papertrace run` does. Every other default is the CLI's own.""" From 9a070afdb1732195dd01e7368928326d3bc147f9 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 26 Sep 2026 12:39:08 +0000 Subject: [PATCH 08/13] docs: the MCP server in the README, CHANGELOG and CLAUDE.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit README: a fourth way in, a section for MCP hosts (the command by absolute path, Claude Code, Claude Desktop, Cursor and VS Code configuration, the nine tools, what start_audit spends and what it refuses before spending), the install-matrix row, one line on each side of the does/does-not list — it serves the audit; it cannot settle a disputed label — and the roadmap item ticked. "Claude Desktop integration" stays open: a one-click install may be what it means, and that is not what this is. CHANGELOG: an Unreleased entry. CLAUDE.md: the commands, why the SDK is in [dev], forget_models on the ask.py bullet, the server as the disclosures' fifth reader, and an mcp_server.py bullet recording what a future change must not undo — in-process, one audit at a time, nobody to ask, no elicitation in place of the person-gated escalation, and every new piece of process-global pipeline state reset per audit. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01J6JtZg47dHEKHDEfAJ5Kpf --- CHANGELOG.md | 42 ++++++++++++++++++++++++++++ CLAUDE.md | 35 +++++++++++++++++++++-- README.md | 79 +++++++++++++++++++++++++++++++++++++++++++++++++--- 3 files changed, 150 insertions(+), 6 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5d62a64..ad6f659 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,48 @@ All notable changes to PaperTrace are documented here. The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); versions follow [SemVer](https://semver.org/). +## [Unreleased] + +### Added — `papertrace mcp`: an MCP server, for hosts that are not a shell + +The roadmap's *drive PaperTrace as a tool from any MCP-capable client*. +`papertrace mcp` serves nine tools over stdio, behind a new `[mcp]` extra +(`mcp>=2.2,<3`, the SDK's current major): `start_audit` and `audit_status` +run and follow the `papertrace run` pipeline; `audit_summary`, `list_claims`, +`get_claim`, `get_evidence`, `list_references`, `list_gaps` and `get_scout` +read any case folder, made over MCP or by the CLI. `get_evidence` returns the +red-box crops as images. + +The server is held to the reports' rules rather than given its own. It +computes no verdict. Every count and verdict carries the disclosures +`disclosures.py` decided for the four report formats — serialised the +viewer's way, with a parity test extending the four-format contract to this +fifth reader — and a limited audit says so first and last. A missing or +unreadable `results.json` is refused with its reason, never answered with +empty counts; `null` in the manifest's three-state fields stays `null`; no +local PDF path is served, and no evidence image from outside `/out/`. + +An audit takes minutes and a host built on the TypeScript SDK gives a request +60 s, so `start_audit` returns at once and `audit_status` long-polls for at +most 50 s. The audit runs in-process — `ask.py` stays the only file in `src/` +that starts a process — on one thread at a time, because the pipeline's +console, `ask`'s model record and stdin are process-global. Inside it nobody +can be asked: a reference label whose readings disagree is withheld, as the +CLI does without a terminal, and settling one still takes a person at a +terminal. And each audit starts with `ask.forget_models()`: the CLI never +cleared the model record because it ran one command per process, and a +second audit whose judging made no calls would otherwise have named the +first one's model as its judge. Design, rejected alternatives and the +decision record: +`docs/superpowers/specs/2026-09-26-mcp-server-design.md`, +`docs/adr/0004-mcp-server.md`. + +### Changed — `[full]` and `[dev]` carry the MCP SDK + +`[full]`, the quick start's everything-install, now includes the SDK, so +`papertrace mcp` works after it. `[dev]` — what CI installs — carries it so +the MCP tests run on every Python in the matrix instead of skipping. + ## [0.7.1] — 2026-09-19 (beta) Two fixes the 0.7.0 showcase run exposed, and the showcase itself. diff --git a/CLAUDE.md b/CLAUDE.md index a187136..7be8f80 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -64,6 +64,8 @@ pytest tests/test_refs.py -q # one module pytest tests/test_refs.py::test_paywalled_is_honest -q # one test pytest -k coverage -q # by name pytest tests/test_report_viewer_js.py -q # the viewer's browser logic, under node (skips without it) +pytest tests/test_mcp_server.py -q # the MCP server, via the SDK's in-memory client + one real stdio run +papertrace mcp # serve over stdio for an MCP host ([mcp] extra); nothing on stdout ruff check src tests scripts evals uv build # sdist + wheel (python -m build is NOT installed) ``` @@ -101,7 +103,9 @@ and torch with it — but never *runs* it: the ~500 MB layout models download on first use, not on install, and every test pins `backend="pymupdf"` (`check_claims` makes `backend` a required argument so none can forget). `playwright` must still stay out of CI; it is the `png` extra, needed only for -`--png`, with `playwright install chromium` once. +`--png`, with `playwright install chromium` once. The MCP SDK is the opposite +case: the `[mcp]` extra, and in `[dev]` as well, so CI runs +`tests/test_mcp_server.py` instead of skipping it. ## Architecture @@ -262,6 +266,8 @@ ingest → extract → refs → scout → check → highlight → report global: `_LAST_MODEL` was overwritten by every call, so a run whose judging made zero calls — every cited source `not_retrieved`, nothing to judge — printed the reference-list model as the `Checker:` of verdicts it never saw. + The record assumes one command per process; `forget_models()` clears it for + a process that outlives one run, which the MCP server calls per audit. `_ask`'s signature is frozen at `(prompt, model=None)`; every offline test that patches the seam does so with a two-argument lambda, which is why the site travels out of band in a context manager instead of as a third @@ -326,7 +332,32 @@ ingest → extract → refs → scout → check → highlight → report DOM-free module (parsing `annotated.md`, anchoring quotes with the claim's `ctx_ids` block searched first, crop sets, filters, the claim map) that `tests/test_report_viewer_js.py` runs under node, and `viewer_app.js`, which - only draws. Disclosures are never re-derived in JS. + only draws. Disclosures are never re-derived in JS. The MCP server is a fifth + reader: it serialises the same `Disclosure` objects with `_disclosure_dict`, + generically, and `tests/test_mcp_server.py`'s parity loops assert every + token reaches it. +- **`mcp_server.py`** — `papertrace mcp`, an MCP server over stdio + (`mcp>=2.2,<3`, the `[mcp]` extra) with nine tools. Seven read a case folder + and compute nothing: every verdict and count travels with its disclosures, + the scope sentence first and again as the final `limited` key; a missing + `results.json` is a `ToolError`, never empty counts; evidence is served only + from inside `/out/`, and no `pdf_path` leaves. `start_audit` runs + `cli._run_pipeline` **in-process, on one daemon thread** — never a + subprocess, which `tests/test_ask.py` would refuse — and `audit_status` + long-polls for at most 50 s, inside the 60 s a TypeScript-SDK host allows a + request. A server outlives many audits, which the CLI never did, so the job + resets what a process used to reset by exiting: `cli.console` becomes the + job's log (under stdio, stdout is the protocol's wire), `sys.stdin` becomes + an empty stream (`_interactive()` is False, so a disputed label is withheld + with `chosen_by: "default"` and the person-gated escalation is unreachable — + do not offer MCP elicitation in its place: a host may be a model), and + `ask.forget_models()` runs first. Those three are process-global, hence + **one audit at a time**, and read tools refuse a case whose audit is still + running. **Any new process-global state in the pipeline must be reset per + audit here.** In SDK v2 only a `ToolError`'s message reaches the model — any + other exception is a bare "Error executing tool" — so every refusal is one. + Return types are `typing_extensions.TypedDict` with `structured_output=True`: + `typing.TypedDict` silently cost every tool its output schema below 3.12. The `case/` folder is the unit of work: one case per paper, guarded by `_guard_case`. It is gitignored by design — manuscripts stay local. diff --git a/README.md b/README.md index b18834f..775321b 100644 --- a/README.md +++ b/README.md @@ -62,7 +62,7 @@ pip install -e ".[full]" # layout-aware ingest — ~1.4 GB, export PAPERTRACE_EMAIL="you@example.org" # Unpaywall asks for a contact address ``` -Three ways in, all producing the same case folder and reports: +Four ways in, all producing the same case folder and reports: - **Guided** — run `papertrace` and answer the questions. It checks your setup first, then asks one thing at a time: the paper, where to keep the audit, @@ -79,6 +79,10 @@ Three ways in, all producing the same case folder and reports: interviews you (the paper, your PDFs, your journal's reviewer form as screenshots), retrieves and checks with evidence as it goes, drafts the findings, and ends by writing the viewer. +- **From an MCP host** — Claude Desktop, Cursor, VS Code or Claude Code, with + `papertrace mcp` as a server: start an audit, follow it, and read every + verdict with its caveats and its evidence crops. See + [From an MCP host](#from-an-mcp-host).

The guided audit in a terminal: a setup check, then the questions one at a time — the paper's path, the case folder, cited PDFs already to hand, the paper's own supplements, a DOI found on the first page, a contact email, whether to write the interactive viewer, whether to limit the audit — and the cost stated as a number of model calls before asking permission to start. @@ -94,6 +98,7 @@ Output lands in `/out/`: `report.md` always, `report_viewer.html` with |---|---| | `pip install -e .` | the standard install — layout-aware ingest of the paper **and** its cited sources (docling's ~500 MB layout models download on first run) | | `pip install -e ".[png]"` | plus PNG rendering of the report looks | +| `pip install -e ".[mcp]"` | plus `papertrace mcp`, the server for MCP hosts | | `pip install -e ".[dev]"` | the test and lint tooling, exactly what CI installs — run the suite as `python -m pytest` | | `pip install -e ".[dev,png]"` | everything | @@ -175,6 +180,9 @@ source that was read flat. verdict and evidence crop, and say how each supplement was attached. - Audit a slice on request — `--max-claims N`, `--max-sources N` — and end every report by stating, in numbers, what was left out. +- Serve the same audit to an MCP host (`papertrace mcp`): start one, follow + it, and read every verdict with the disclosures the reports carry and the + evidence crops as images. The server computes no verdict of its own. - Keep you responsible for interpretation: it prepares evidence and drafts; the conclusions are yours. @@ -202,6 +210,9 @@ source that was read flat. on a source's supplementary table is judged against its main text alone. - Guarantee an exhaustive literature search — the scout is search-based, and absence from its lists proves nothing. +- Settle a disputed reference label from an MCP host. Nobody is asked over + MCP, so verdicts on that label are withheld — the CLI's answer when no + terminal is attached — and `papertrace refs` in a terminal settles it. - Replace peer review or your research judgement. ## The viewer @@ -280,6 +291,64 @@ each records which model answered. Ingest, crops and reports give the same output for the same input. Retrieval and the scout query live services, so a re-run months later can find a different set of sources. +## From an MCP host + +`papertrace mcp` serves PaperTrace over stdio to any +[MCP](https://modelcontextprotocol.io) host. Install the extra, and give the +host the command by absolute path — a host starts its servers from its own +working directory, with its own and often minimal `PATH`: + +```bash +pip install -e ".[mcp]" +which papertrace # the path the host needs +claude mcp add papertrace -- /absolute/path/to/papertrace mcp # Claude Code +``` + +Claude Desktop reads it from `claude_desktop_config.json`, and Cursor from +`.cursor/mcp.json` in the same shape; VS Code takes the same command as a +`"servers"` entry with `"type": "stdio"` in `.vscode/mcp.json`: + +```json +{ + "mcpServers": { + "papertrace": { + "command": "/absolute/path/to/papertrace", + "args": ["mcp"], + "env": {"PAPERTRACE_EMAIL": "you@example.org"} + } + } +} +``` + +| Tool | What it does | +|---|---| +| `start_audit` | runs the `papertrace run` pipeline in the background and returns at once | +| `audit_status` | running, finished or failed, with the log tail; waits up to 50 s per call | +| `audit_summary` | verdict counts, references obtained, the judging model, coverage, and every run-level disclosure | +| `list_claims`, `get_claim` | each claim with the paper's own sentence, each source's verdict and rationale, and its caveats | +| `get_evidence` | the red-box page crops a verdict rests on, as images | +| `list_references` | the retrieval manifest and the numbering state — without local file paths | +| `list_gaps`, `get_scout` | uncited assertions, citation places no claim reached, unchecked claims; the scout's registers | + +The read tools work on any case folder, made over MCP or by the CLI, and +change nothing. Every verdict arrives with the caveats the reports print +beside it, and a limited audit says so first and last. An audit takes minutes, +longer than many hosts wait for a single request, which is why it runs as a +job that `audit_status` follows. + +`start_audit` spends what `papertrace run` spends: `claude -p` calls on this +machine's Claude login, and the open-access services. So `claude` has to be on +the server's `PATH` — if the host's is too minimal, add its directory in +`env` — and the contact email comes from `start_audit`'s `email`, from +`PAPERTRACE_EMAIL`, or from the address `papertrace` saved. A missing +manuscript, a `claude` the server cannot find, no email, or a case folder +holding another paper is refused before anything is spent. One audit runs at a +time per server, and a case folder the server is writing is not read until the +audit ends. Nobody is at an MCP call to answer a question, so a reference label +whose readings disagree is withheld, never settled — that takes you, at a +terminal. Nothing but the protocol reaches stdout; what the pipeline prints +comes back in `audit_status`'s log. + ## Tables and figures are evidence too A number in a table cell, or text drawn inside a vector figure, is in the PDF's @@ -367,8 +436,9 @@ paper.pdf ─────ingest──▶ clean.md + source_map.json (page Seven stages, each a subcommand, chained by `run`; every stage writes a file into the case folder and the next reads only that. The JSON contracts are versioned in [`schemas/`](schemas/). The two skills in -[`.claude/skills/`](.claude/skills/) drive the same tools interactively; the -audit craft lives in [`prompts/review_core.md`](prompts/review_core.md). +[`.claude/skills/`](.claude/skills/) drive the same tools interactively, and +`papertrace mcp` serves them to any MCP host; the audit craft lives in +[`prompts/review_core.md`](prompts/review_core.md). ## Ethics & scope @@ -453,7 +523,8 @@ box glyphs. - [ ] More scout backends (OpenAlex, Semantic Scholar) - [ ] Other LLM backends (local and API models alongside headless Claude Code) - [ ] Claude Desktop integration -- [ ] MCP server — drive PaperTrace as a tool from any MCP-capable client +- [x] MCP server — drive PaperTrace as a tool from any MCP-capable client: + `papertrace mcp` (unreleased) - [ ] DOCX ingest - [ ] Revision (R1) mode polish - [ ] Figure-vs-text consistency pass (batch) From cdd6339f155dbe44c16c10fa518d139984eed625 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 26 Sep 2026 12:56:56 +0000 Subject: [PATCH 09/13] =?UTF-8?q?mcp:=20review=20round=20=E2=80=94=20no=20?= =?UTF-8?q?mixed=20case=20read=20as=20a=20result,=20no=20verdict=20without?= =?UTF-8?q?=20its=20caveats?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fixes from a code review of this branch, each verified against the code and each with a test that failed first. - A failed or interrupted audit left one run's manifest beside another's verdicts, and the read tools served them. The job now writes /mcp_audit.json ("running" before the pipeline starts, the outcome when it ends); read tools refuse a case whose last MCP audit failed or never finished, and audit_status reports an earlier server's audit from the record — "interrupted" when it says "running". A log that was not kept is null, not an empty list. - get_claim and list_claims now carry the run-level disclosures, and the evidence caption carries the claim's and the run's: a host can read a verdict without ever calling audit_summary, and an unconfirmed source identity or a doubtful label belongs beside it. - With no case given and the paper's folder unwritable, start_audit refuses instead of falling back to the server's working directory, which the host chose; what the preflight printed opens the job's log. - get_scout refuses a scout.json the latest audit did not write (it ran without the scout), and its caveat carries the scan's own error. - `papertrace mcp` names the fix for a 1.x SDK too, which fails with a plain ImportError, and no longer wraps the install command in two; the test module skips on an SDK below 2 instead of failing collection. - Gate 2: schemas/mcp_tools.schema.json and schemas/mcp_audit.schema.json, with every tool output validated against the first, each definition's keys held equal to its TypedDict's, and the record validated against the second. - One rule, not two copies: disclosures.coverage_audited() decides whether coverage ran for the reports and the server alike, and cli.SCOUT_CAVEAT is the scout's sentence for both. Not taken: the finding that _run_pipeline's OptionInfo guards are dead. tests/test_pipeline_split.py calls cli.run() directly and omits options, which hands the adapter's sentinels straight through. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01J6JtZg47dHEKHDEfAJ5Kpf --- schemas/mcp_audit.schema.json | 55 ++ schemas/mcp_tools.schema.json | 1017 +++++++++++++++++++++++++++++++++ src/papertrace/cli.py | 34 +- src/papertrace/disclosures.py | 18 +- src/papertrace/mcp_server.py | 227 ++++++-- tests/test_mcp_server.py | 223 +++++++- 6 files changed, 1514 insertions(+), 60 deletions(-) create mode 100644 schemas/mcp_audit.schema.json create mode 100644 schemas/mcp_tools.schema.json diff --git a/schemas/mcp_audit.schema.json b/schemas/mcp_audit.schema.json new file mode 100644 index 0000000..7f1f97a --- /dev/null +++ b/schemas/mcp_audit.schema.json @@ -0,0 +1,55 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "title": "mcp_audit", + "description": "/mcp_audit.json — what an MCP server (`papertrace mcp`) knows about the last audit it started in this case folder. The audit job writes it with state `running` before the pipeline touches anything, and rewrites it when the job ends. A server stopped mid-audit therefore leaves `running` behind, and a later server reads that as an audit that never finished: it refuses to serve the folder as a result rather than present half-written files as one. Absent means no MCP server has audited this case; the CLI never writes it.", + "type": "object", + "required": [ + "schema", + "state", + "manuscript", + "started", + "ended", + "error", + "scout" + ], + "properties": { + "schema": { + "type": "string", + "enum": [ + "mcp_audit/1" + ] + }, + "state": { + "type": "string", + "enum": [ + "running", + "finished", + "failed" + ], + "description": "`running` while the job runs — and forever, when the server stopped before the job ended, which a later server reports as `interrupted`. `failed`: the pipeline stopped, and what is on disk may mix that run with an earlier one." + }, + "manuscript": { + "type": "string", + "description": "The audited PDF's file name, as refs_manifest.json keeps it." + }, + "started": { + "type": "string", + "description": "When the job started: UTC, ISO 8601, to the second." + }, + "ended": { + "type": [ + "string", + "null" + ], + "description": "When the job ended; null while it runs, and on an audit whose server stopped first." + }, + "error": { + "type": "string", + "description": "Why the audit failed — the last lines the pipeline printed, or the exception's type and message. Empty unless `state` is `failed`." + }, + "scout": { + "type": "boolean", + "description": "Whether this audit scanned the literature. false: any out/scout.json in the folder belongs to an earlier run, and is not served as this audit's." + } + } +} diff --git a/schemas/mcp_tools.schema.json b/schemas/mcp_tools.schema.json new file mode 100644 index 0000000..cf95afe --- /dev/null +++ b/schemas/mcp_tools.schema.json @@ -0,0 +1,1017 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "title": "mcp_tools", + "description": "What `papertrace mcp` returns: the structured content of each tool's result, one definition per tool under $defs (named after the tool), with the shapes they share. The SDK publishes each as the tool's outputSchema in tools/list; this file is the same contract where this repository keeps its contracts, and tests/test_mcp_server.py validates every tool's output against it. get_evidence returns content blocks — a caption and images — rather than structured content, and has no definition here.", + "$defs": { + "audit_summary": { + "description": "The result of one audit, from out/results.json and refs_manifest.json. Never returned for a case without results.json — the call fails with the reason instead — so no count here stands for 'never measured'.", + "properties": { + "case": { + "type": "string" + }, + "manuscript": { + "type": "string" + }, + "checker": { + "type": "string" + }, + "date": { + "type": "string" + }, + "converter": { + "type": "string" + }, + "references": { + "$ref": "#/$defs/RefCounts" + }, + "claims": { + "type": "integer" + }, + "counts": { + "additionalProperties": { + "type": "integer" + }, + "type": "object" + }, + "uncited_assertions": { + "type": "integer" + }, + "coverage": { + "type": [ + "string", + "null" + ], + "description": "The coverage headline; null when the coverage audit could not run — never an empty string or a zero ratio." + }, + "disclosures": { + "items": { + "$ref": "#/$defs/DisclosureOut" + }, + "type": "array", + "description": "Every run-level disclosure, in the order the reports print them — the scope first when the audit was limited." + }, + "reports": { + "items": { + "type": "string" + }, + "type": "array", + "description": "Absolute paths of the report files present in out/." + }, + "limited": { + "type": [ + "string", + "null" + ], + "description": "The scope sentence when the audit was limited on request, stated last as every report states it; null when the whole paper was audited." + } + }, + "required": [ + "case", + "manuscript", + "checker", + "date", + "converter", + "references", + "claims", + "counts", + "uncited_assertions", + "coverage", + "disclosures", + "reports", + "limited" + ], + "type": "object" + }, + "list_claims": { + "description": "One row per checked claim, each with its claim-level caveats, and the run-level disclosures once beside them.", + "properties": { + "case": { + "type": "string" + }, + "verdict_filter": { + "type": [ + "string", + "null" + ] + }, + "claims_total": { + "type": "integer" + }, + "claims": { + "items": { + "$ref": "#/$defs/ClaimRow" + }, + "type": "array" + }, + "run_disclosures": { + "items": { + "$ref": "#/$defs/DisclosureOut" + }, + "type": "array", + "description": "The caveats the reports print above every verdict, so a claim read without audit_summary still has them." + }, + "limited": { + "type": [ + "string", + "null" + ], + "description": "As audit_summary's: the scope sentence, or null for a whole-paper audit." + } + }, + "required": [ + "case", + "verdict_filter", + "claims_total", + "claims", + "run_disclosures", + "limited" + ], + "type": "object" + }, + "get_claim": { + "description": "One claim in full: the paper's sentence, each cited source's judgement, and every disclosure the claim and the run owe its reader.", + "properties": { + "case": { + "type": "string" + }, + "id": { + "type": "integer" + }, + "verdict": { + "type": "string" + }, + "label": { + "type": "string" + }, + "qualifier": { + "type": "string" + }, + "claim": { + "type": "string" + }, + "quote": { + "type": "string" + }, + "location": { + "type": "string" + }, + "refs": { + "items": { + "type": "string" + }, + "type": "array" + }, + "own_supplement": { + "type": "boolean" + }, + "rationale": { + "type": "string" + }, + "source": { + "type": [ + "string", + "null" + ] + }, + "page": { + "type": [ + "integer", + "null" + ] + }, + "block": { + "type": [ + "string", + "null" + ] + }, + "anchor_phrases": { + "items": { + "type": "string" + }, + "type": "array" + }, + "anchor_located": { + "type": [ + "boolean", + "null" + ] + }, + "evidence_images": { + "items": { + "type": "string" + }, + "type": "array", + "description": "Crop paths relative to /out/; get_evidence returns the images themselves." + }, + "judgements": { + "items": { + "$ref": "#/$defs/JudgementOut" + }, + "type": "array" + }, + "unjudged_refs": { + "items": { + "type": "string" + }, + "type": "array", + "description": "Co-cited labels whose source could not be obtained — nobody read them." + }, + "withheld_refs": { + "items": { + "type": "string" + }, + "type": "array", + "description": "Co-cited labels whose source was obtained and withheld, because the readings of the reference list disagree about which paper the label names." + }, + "disclosures": { + "items": { + "$ref": "#/$defs/DisclosureOut" + }, + "type": "array", + "description": "The claim-level disclosures, as every report format prints them beside the claim." + }, + "run_disclosures": { + "items": { + "$ref": "#/$defs/DisclosureOut" + }, + "type": "array", + "description": "The run-level disclosures: the caveats the reports print above every verdict." + } + }, + "required": [ + "case", + "id", + "verdict", + "label", + "qualifier", + "claim", + "quote", + "location", + "refs", + "own_supplement", + "rationale", + "source", + "page", + "block", + "anchor_phrases", + "anchor_located", + "evidence_images", + "judgements", + "unjudged_refs", + "withheld_refs", + "disclosures", + "run_disclosures" + ], + "type": "object" + }, + "list_references": { + "description": "The retrieval manifest and the state of the reference numbering. Local file paths are never included.", + "properties": { + "case": { + "type": "string" + }, + "manuscript": { + "type": "string" + }, + "total": { + "type": "integer" + }, + "available": { + "type": "integer" + }, + "by_status": { + "additionalProperties": { + "type": "integer" + }, + "type": "object" + }, + "numbering": { + "$ref": "#/$defs/NumberingOut" + }, + "limits": { + "additionalProperties": true, + "type": "object" + }, + "status_filter": { + "type": [ + "string", + "null" + ] + }, + "references": { + "items": { + "$ref": "#/$defs/ReferenceRow" + }, + "type": "array" + } + }, + "required": [ + "case", + "manuscript", + "total", + "available", + "by_status", + "numbering", + "limits", + "status_filter", + "references" + ], + "type": "object" + }, + "list_gaps": { + "description": "What the audit could not vouch for: uncited assertions, citation places no claim reached, claims never checked.", + "properties": { + "case": { + "type": "string" + }, + "uncited_assertions": { + "items": { + "$ref": "#/$defs/UncitedOut" + }, + "type": "array" + }, + "coverage": { + "type": [ + "string", + "null" + ], + "description": "The coverage headline; null when the coverage audit could not run." + }, + "unreached_labels": { + "items": { + "type": "string" + }, + "type": [ + "array", + "null" + ], + "description": "Labels no extracted claim reached; null when the coverage audit did not run — not 'none unreached'." + }, + "unreached_citations": { + "items": { + "$ref": "#/$defs/UnreachedOut" + }, + "type": [ + "array", + "null" + ], + "description": "Citation places no claim reached (uncovered) or reached ambiguously (uncertain); null when the audit did not run or recorded no occurrences." + }, + "unchecked_claims": { + "items": { + "$ref": "#/$defs/UncheckedOut" + }, + "type": "array" + }, + "disclosures": { + "items": { + "$ref": "#/$defs/DisclosureOut" + }, + "type": "array" + }, + "limited": { + "type": [ + "string", + "null" + ], + "description": "As audit_summary's: the scope sentence, or null for a whole-paper audit." + } + }, + "required": [ + "case", + "uncited_assertions", + "coverage", + "unreached_labels", + "unreached_citations", + "unchecked_claims", + "disclosures", + "limited" + ], + "type": "object" + }, + "get_scout": { + "description": "out/scout.json, with the scout's own caveat. Search-based: absence from these registers proves nothing.", + "properties": { + "case": { + "type": "string" + }, + "paper": { + "$ref": "#/$defs/ScoutPaperOut" + }, + "query": { + "type": "string" + }, + "date": { + "type": "string" + }, + "counts": { + "additionalProperties": { + "type": "integer" + }, + "type": "object" + }, + "newer": { + "items": { + "$ref": "#/$defs/ScoutHitOut" + }, + "type": "array" + }, + "overlooked": { + "items": { + "$ref": "#/$defs/ScoutHitOut" + }, + "type": "array" + }, + "same_year": { + "items": { + "$ref": "#/$defs/ScoutHitOut" + }, + "type": "array" + }, + "error": { + "type": "string" + }, + "caveat": { + "type": "string", + "description": "The scout's caveat, led by an identity warning and the scan's own error when either applies." + } + }, + "required": [ + "case", + "paper", + "query", + "date", + "counts", + "newer", + "overlooked", + "same_year", + "error", + "caveat" + ], + "type": "object" + }, + "audit_status": { + "description": "Where an audit an MCP server started stands. Never carries counts — they travel only with their disclosures, in audit_summary.", + "properties": { + "case": { + "type": "string" + }, + "state": { + "enum": [ + "running", + "finished", + "failed", + "interrupted", + "not_started" + ], + "type": "string", + "description": "running, finished, failed: this server's own job. interrupted: the record an earlier server left says running, so that audit never ended. not_started: no MCP server has audited this case." + }, + "manuscript": { + "type": [ + "string", + "null" + ] + }, + "started": { + "type": [ + "string", + "null" + ] + }, + "ended": { + "type": [ + "string", + "null" + ] + }, + "error": { + "type": "string" + }, + "log": { + "items": { + "type": "string" + }, + "type": [ + "array", + "null" + ], + "description": "The job's latest console lines; null when no log exists in this process — a recorded or never-started audit — because an empty log would read as an audit that printed nothing." + }, + "log_lines_total": { + "type": [ + "integer", + "null" + ], + "description": "How many console lines the job printed; null when no log exists in this process." + }, + "next": { + "type": "string" + } + }, + "required": [ + "case", + "state", + "manuscript", + "started", + "ended", + "error", + "log", + "log_lines_total", + "next" + ], + "type": "object" + }, + "ClaimRow": { + "properties": { + "id": { + "type": "integer" + }, + "verdict": { + "type": "string" + }, + "label": { + "type": "string" + }, + "qualifier": { + "type": "string" + }, + "refs": { + "items": { + "type": "string" + }, + "type": "array" + }, + "location": { + "type": "string" + }, + "claim": { + "type": "string" + }, + "quote": { + "type": "string" + }, + "caveats": { + "items": { + "type": "string" + }, + "type": "array" + } + }, + "required": [ + "id", + "verdict", + "label", + "qualifier", + "refs", + "location", + "claim", + "quote", + "caveats" + ], + "type": "object" + }, + "DisclosureOut": { + "properties": { + "key": { + "type": "string" + }, + "level": { + "type": "string" + }, + "token": { + "type": "string", + "description": "The short literal every report format carries verbatim for this disclosure." + }, + "text": { + "type": "string" + }, + "short": { + "type": "string" + }, + "rows": { + "items": { + "type": "string" + }, + "type": "array" + } + }, + "required": [ + "key", + "level", + "token", + "text", + "short", + "rows" + ], + "type": "object" + }, + "JudgementOut": { + "properties": { + "source": { + "type": "string" + }, + "origin": { + "type": "string" + }, + "kind": { + "type": "string" + }, + "verdict": { + "type": "string" + }, + "label": { + "type": "string" + }, + "rationale": { + "type": "string" + }, + "page": { + "type": [ + "integer", + "null" + ] + }, + "block": { + "type": [ + "string", + "null" + ] + }, + "anchor_phrases": { + "items": { + "type": "string" + }, + "type": "array" + }, + "anchor_located": { + "type": [ + "boolean", + "null" + ] + }, + "evidence_images": { + "items": { + "type": "string" + }, + "type": "array" + }, + "disclosures": { + "items": { + "$ref": "#/$defs/DisclosureOut" + }, + "type": "array" + } + }, + "required": [ + "source", + "origin", + "kind", + "verdict", + "label", + "rationale", + "page", + "block", + "anchor_phrases", + "anchor_located", + "evidence_images", + "disclosures" + ], + "type": "object" + }, + "NumberingOut": { + "properties": { + "source": { + "type": "string" + }, + "verified": { + "type": "boolean", + "description": "numbering_verified: the chosen list accounts for exactly the cited labels — extent, not content." + }, + "note": { + "type": "string" + }, + "unverified_from": { + "type": [ + "integer", + "null" + ] + }, + "contested": { + "type": "boolean" + }, + "corroborated": { + "type": [ + "boolean", + "null" + ], + "description": "null: never computed. false: computed, and some cited label was not agreed by two readings." + }, + "corroborating_readings": { + "items": { + "type": "string" + }, + "type": "array" + }, + "labels_disputed": { + "items": { + "type": "string" + }, + "type": "array" + }, + "labels_uncomparable": { + "items": { + "type": "string" + }, + "type": [ + "array", + "null" + ], + "description": "The subset of labels_disputed where nothing comparable was said. null: never computed. []: computed, and every dispute was a contradiction." + }, + "labels_resolved": { + "items": { + "type": "string" + }, + "type": "array" + }, + "choice": { + "type": "string" + }, + "chosen_by": { + "type": "string", + "description": "'' when no escalation ran; 'default' when nobody was asked — always so for an audit started over MCP; 'user' for a person at a terminal." + } + }, + "required": [ + "source", + "verified", + "note", + "unverified_from", + "contested", + "corroborated", + "corroborating_readings", + "labels_disputed", + "labels_uncomparable", + "labels_resolved", + "choice", + "chosen_by" + ], + "type": "object" + }, + "RefCounts": { + "properties": { + "total": { + "type": "integer" + }, + "available": { + "type": "integer" + } + }, + "required": [ + "total", + "available" + ], + "type": "object" + }, + "ReferenceRow": { + "properties": { + "label": { + "type": "string" + }, + "status": { + "type": "string" + }, + "reason": { + "type": "string" + }, + "resolver": { + "type": [ + "string", + "null" + ] + }, + "doi": { + "type": [ + "string", + "null" + ] + }, + "year": { + "type": [ + "string", + "null" + ] + }, + "slug": { + "type": [ + "string", + "null" + ] + }, + "title_check": { + "type": [ + "string", + "null" + ] + }, + "skipped_by": { + "type": [ + "string", + "null" + ], + "description": "Why a skipped reference was skipped: 'claims' (no selected claim cites it) or 'sources' (the cap was reached); null for every other status." + }, + "boundary_ambiguous": { + "type": "boolean" + }, + "seen_in": { + "items": { + "type": "string" + }, + "type": "array" + }, + "supplements": { + "items": { + "$ref": "#/$defs/SupplementOut" + }, + "type": "array" + }, + "raw": { + "type": "string" + } + }, + "required": [ + "label", + "status", + "reason", + "resolver", + "doi", + "title", + "year", + "slug", + "title_check", + "skipped_by", + "boundary_ambiguous", + "seen_in", + "supplements", + "raw" + ], + "type": "object" + }, + "ScoutHitOut": { + "properties": { + "year": { + "type": [ + "integer", + "null" + ] + }, + "doi": { + "type": "string" + }, + "via": { + "type": "string" + }, + "journal": { + "type": "string" + }, + "authors": { + "type": "string" + } + }, + "required": [ + "title", + "year", + "doi", + "via", + "journal", + "authors" + ], + "type": "object" + }, + "ScoutPaperOut": { + "properties": { + "doi": { + "type": "string" + }, + "year": { + "type": [ + "integer", + "null" + ] + }, + "resolved_via": { + "type": "string" + }, + "identity": { + "type": "string" + } + }, + "required": [ + "title", + "doi", + "year", + "resolved_via", + "identity" + ], + "type": "object" + }, + "SupplementOut": { + "properties": { + "slug": { + "type": "string" + }, + "verified": { + "type": "boolean" + } + }, + "required": [ + "slug", + "verified" + ], + "type": "object" + }, + "UncheckedOut": { + "properties": { + "id": { + "type": "integer" + }, + "verdict": { + "type": "string" + }, + "label": { + "type": "string" + }, + "refs": { + "items": { + "type": "string" + }, + "type": "array" + }, + "location": { + "type": "string" + }, + "note": { + "type": "string" + } + }, + "required": [ + "id", + "verdict", + "label", + "refs", + "location", + "note" + ], + "type": "object" + }, + "UncitedOut": { + "properties": { + "id": { + "type": "integer" + }, + "claim": { + "type": "string" + }, + "quote": { + "type": "string" + }, + "location": { + "type": "string" + } + }, + "required": [ + "id", + "claim", + "quote", + "location" + ], + "type": "object" + }, + "UnreachedOut": { + "properties": { + "label": { + "type": "string" + }, + "status": { + "type": "string" + }, + "section": { + "type": "string" + }, + "sentence": { + "type": "string" + } + }, + "required": [ + "label", + "status", + "section", + "sentence" + ], + "type": "object" + } + } +} diff --git a/src/papertrace/cli.py b/src/papertrace/cli.py index 0b77147..773d4df 100644 --- a/src/papertrace/cli.py +++ b/src/papertrace/cli.py @@ -457,14 +457,24 @@ def mcp_command() -> None: # and a banner or a warning printed before serving begins corrupts it. try: from .mcp_server import serve - except ModuleNotFoundError as e: - # only the SDK's absence earns the install line; any other missing - # module is a real fault and says so with its own traceback + except ImportError as e: + # Only the SDK earns the install line — absent (ModuleNotFoundError) or + # a 1.x that other tools installed, which has no `MCPServer` and fails + # with a plain ImportError. Any other import is a real fault and keeps + # its own traceback. if (e.name or "").split(".")[0] != "mcp": raise - # rich eats [mcp] as a style tag; escaping it is what prints the extra - Console(stderr=True).print( - r"[red]the MCP server needs the MCP SDK[/red] — pip install 'papertrace\[mcp]'" + from importlib.metadata import PackageNotFoundError, version + + try: + found = f" (found mcp {version('mcp')})" + except PackageNotFoundError: + found = "" + # rich eats [mcp] as a style tag; escaping it is what prints the extra. + # soft_wrap: wrapped at the terminal's width, the command split in two + Console(stderr=True, soft_wrap=True).print( + f"[red]the MCP server needs version 2 of the MCP SDK{found}[/red] — " + r"pip install 'papertrace\[mcp]'" ) raise typer.Exit(2) from None serve() @@ -1789,6 +1799,13 @@ def refs( claims=_first_n(max_claims), max_sources=_int(max_sources)) +# What the scout's candidates are, said once for the console and the MCP server. +SCOUT_CAVEAT = ( + "search-based — absence from these lists proves nothing; presence is a candidate " + "for your judgement, not an accusation." +) + + @app.command(rich_help_panel="Pipeline stages — `run` calls these in order") def scout( case: Path = typer.Option( @@ -1848,10 +1865,7 @@ def scout( console.print(f" [cyan]{h.year or '?'}[/cyan] {h.title[:76]}") if len(res.same_year) > 5: console.print(f" [dim]… {len(res.same_year) - 5} more in scout.json[/dim]") - console.print( - "[dim]search-based — absence from these lists proves nothing; presence is a" - " candidate for your judgement, not an accusation.[/dim]" - ) + console.print(f"[dim]{SCOUT_CAVEAT}[/dim]") def _check_pipeline( diff --git a/src/papertrace/disclosures.py b/src/papertrace/disclosures.py index c4a0c2e..7bf6ff1 100644 --- a/src/papertrace/disclosures.py +++ b/src/papertrace/disclosures.py @@ -307,6 +307,20 @@ def _converter(name: str) -> Disclosure: ) +def coverage_audited(coverage: dict) -> bool: + """Did the coverage audit run at all? One rule, because two readers need it. + + The reports decide on it whether to print a ratio or `coverage not audited`, + and the MCP server whether a gap list is a measurement or `null`. Were the + two copies to disagree, one reader would call "never measured" what the + other calls "measured, nothing missing". + + Occurrences without labels means `clean.md` was missing while the source + map was not: the audit ran, so it must not read as "not audited". + """ + return bool(coverage.get("labels_in_text") or (coverage.get("occurrences") or {}).get("total")) + + def coverage_headline(coverage: dict) -> str: """The one sentence the coverage audit is allowed to be summarised as. @@ -1348,9 +1362,7 @@ def run_disclosures(results, manifest=None) -> list[Disclosure]: out.append(_supplement_identity(checked_sup, named_sup)) coverage = results.coverage or {} if coverage: - # occurrences without labels means clean.md was missing while the source - # map was not: the audit ran, so it must not read as "not audited" - if coverage.get("labels_in_text") or (coverage.get("occurrences") or {}).get("total"): + if coverage_audited(coverage): out.append(_coverage(coverage)) # only occurrence-level coverage owes the attribution caveat: the # label-level audit it replaces could not make that mistake diff --git a/src/papertrace/mcp_server.py b/src/papertrace/mcp_server.py index dc56fee..36a9827 100644 --- a/src/papertrace/mcp_server.py +++ b/src/papertrace/mcp_server.py @@ -12,6 +12,8 @@ import datetime import io +import json +import os import sys import threading from collections.abc import Callable, Iterator @@ -39,6 +41,7 @@ from .disclosures import ( anchor_disclosure, claim_disclosures, + coverage_audited, coverage_headline, ids_span, judgement_disclosures, @@ -77,12 +80,6 @@ passage each rests on. Scout hits are candidates, not accusations. """ -# The CLI's sentence, verbatim: the scout's candidates are search results. -SCOUT_CAVEAT = ( - "search-based — absence from these lists proves nothing; presence is a candidate " - "for your judgement, not an accusation." -) - # read tools change nothing and consult nothing outside the case folder _READ = ToolAnnotations(read_only_hint=True, open_world_hint=False) # The one tool that spends says all of it, because a host decides on these @@ -169,6 +166,7 @@ class ClaimList(TypedDict): verdict_filter: str | None claims_total: int claims: list[ClaimRow] + run_disclosures: list[DisclosureOut] limited: str | None @@ -209,6 +207,7 @@ class ClaimDetail(TypedDict): unjudged_refs: list[str] withheld_refs: list[str] disclosures: list[DisclosureOut] + run_disclosures: list[DisclosureOut] class SupplementOut(TypedDict): @@ -329,13 +328,13 @@ class AuditStatus(TypedDict): only with the disclosures that qualify them, in audit_summary.""" case: str - state: Literal["running", "finished", "failed", "not_started"] + state: Literal["running", "finished", "failed", "interrupted", "not_started"] manuscript: str | None started: str | None ended: str | None error: str - log: list[str] - log_lines_total: int + log: list[str] | None + log_lines_total: int | None next: str @@ -503,7 +502,15 @@ def _crop_paths(case: Path, claim: ClaimResult, rels: list[str]) -> list[Path]: return paths -def _caption(claim: ClaimResult, target, shown: list[Path], missing: list[str]) -> str: +def _caption( + claim: ClaimResult, target, shown: list[Path], missing: list[str], manifest, run_fired +) -> str: + """What the crop is, and every caveat on the verdict it is shown under. + + The caveats travel with the picture because the picture is shown with its + verdict: a crop from a source whose label may name another paper, captioned + without saying so, would present the wrong paper as evidence. + """ qualifier = claim.headline_qualifier() lines = [f"claim {claim.id} — {claim.label}" + (f" ({qualifier})" if qualifier else "")] if claim.quote: @@ -523,16 +530,16 @@ def _caption(claim: ClaimResult, target, shown: list[Path], missing: list[str]) ) if missing: lines.append(f"not in the case folder, so not shown: {', '.join(missing)}") + if caveats := [d.short for d in claim_disclosures(claim, manifest)]: + lines += ["caveats on this claim:", *(f"- {c}" for c in caveats)] + if run_fired: + lines += ["caveats on the whole audit:", *(f"- {d.short}" for d in run_fired)] return "\n".join(lines) -def _coverage_audited(coverage: dict) -> bool: - """Did the coverage audit run? The rule `run_disclosures` applies.""" - return bool(coverage.get("labels_in_text") or (coverage.get("occurrences") or {}).get("total")) - - def _scout_caveat(scout: ScoutResults) -> str: - """The scout's own caveat, led by the identity warning the CLI prints.""" + """The scout's own caveat, led by what the report formats print beside it: + the identity warning, and the scan's own error when it did not complete.""" identity = scout.paper_identity if not scout.paper_title or identity == "confirmed": lead = "" @@ -542,7 +549,11 @@ def _scout_caveat(scout: ScoutResults) -> str: lead = "The paper's identity in Europe PMC is unverified — check the record is this paper. " else: lead = "The paper's identity was not recorded — check the record is this paper. " - return lead + SCOUT_CAVEAT[0].upper() + SCOUT_CAVEAT[1:] + if scout.error: + # the reports print it beside the caveat: an incomplete scan's empty + # register is absence of data, not a clean search + lead += f"The scan is incomplete: {scout.error}. " + return lead + cli.SCOUT_CAVEAT[0].upper() + cli.SCOUT_CAVEAT[1:] # -------------------------------------------------------------------------- @@ -620,6 +631,16 @@ def _last_said(lines: list[str], n: int = 4) -> str: return " ".join(line.strip() for line in [x for x in lines if x.strip()][-n:]) +# What a later server knows about the last audit an MCP server started in a case +# folder. The job writes it "running" before the pipeline touches anything and +# rewrites it when the job ends, so a server stopped mid-audit leaves "running" +# behind — the one thing on disk that says the files beside it are incomplete. +# Its shape is `schemas/mcp_audit.schema.json`. +RECORD = "mcp_audit.json" +RECORD_SCHEMA = "mcp_audit/1" +_RECORD_STATES = ("running", "finished", "failed") + + @dataclass class _Job: """One audit this server process started.""" @@ -627,6 +648,7 @@ class _Job: case: Path manuscript: Path started: str + scout: bool = True # False leaves an earlier run's scout.json in the folder state: str = "running" # running | finished | failed ended: str | None = None error: str = "" @@ -634,6 +656,40 @@ class _Job: done: threading.Event = field(default_factory=threading.Event) +def _write_record(job: _Job) -> None: + payload = { + "schema": RECORD_SCHEMA, + "state": job.state, + "manuscript": job.manuscript.name, # a name, as refs_manifest.json keeps it + "started": job.started, + "ended": job.ended, + "error": job.error, + "scout": job.scout, + } + (job.case / RECORD).write_text( + json.dumps(payload, indent=2, ensure_ascii=False) + "\n", encoding="utf-8" + ) + + +def _read_record(case: Path) -> dict | None: + """The last MCP audit's record, or None when no MCP server audited here — and + a refusal, never a guess, when there is one and it cannot be read.""" + path = case / RECORD + if not path.is_file(): + return None + try: + record = json.loads(path.read_text(encoding="utf-8")) + if not isinstance(record, dict) or record.get("state") not in _RECORD_STATES: + raise ValueError("no state this server knows") + except (OSError, ValueError) as e: + raise ToolError( + f"{path} is unreadable ({type(e).__name__}: {str(e)[:200]}), so whether the last " + "audit here finished cannot be told and nothing is read as its result — run " + "start_audit again" + ) from None + return record + + class _Jobs: """The audits this server process started — at most one running at a time. @@ -650,17 +706,49 @@ def __init__(self): def last(self, case: Path) -> _Job | None: return self._by_case.get(case) - def refuse_running(self, case: Path) -> None: - """A case whose audit is running has nothing on disk that belongs to it.""" - job = self._running - if job is not None and job.case == case: + def refuse_unfinished(self, case: Path) -> None: + """Nothing on disk is read as a result unless the last audit here ended. + + This server's own job decides it when there is one; otherwise the record + an earlier server left. No record means no MCP server audited here, and + the case is read as the CLI's reports would read it. + """ + if (job := self._by_case.get(case)) is not None: + state, started, error = job.state, job.started, job.error + elif (record := _read_record(case)) is not None: + state = "interrupted" if record["state"] == "running" else record["state"] + started, error = record.get("started"), record.get("error") or "" + else: + return + if state == "running": + raise ToolError( + f"an audit of {case} is running now (started {started}) — what is on disk " + "belongs to an earlier run or is half-written, so nothing is read from it " + "until the audit ends. Call audit_status." + ) + if state == "interrupted": + raise ToolError( + f"an audit of {case} started {started} did not finish: the server running it " + "stopped mid-run, so what is on disk may be half-written or mix two runs, and " + "none of it is read as a result. Run start_audit again — or, if the case has " + f"been completed since by other means, remove {case / RECORD} first." + ) + if state == "failed": raise ToolError( - f"an audit of {case} is running now (started {job.started}) — what is on " - "disk belongs to an earlier run or is half-written, so nothing is read from " - "it until the audit ends. Call audit_status." + f"the last audit of {case} (started {started}) failed: {error}. What is on " + "disk may mix that run with an earlier one, so none of it is read as a " + "result — fix the cause and run start_audit again." ) - def start(self, prepare: Callable[[], tuple[Path, Path, dict]]) -> _Job: + def scanned(self, case: Path) -> bool | None: + """Did the latest MCP audit here scan the literature? None: no MCP audit here, + or one whose record does not say.""" + if (job := self._by_case.get(case)) is not None: + return job.scout + record = _read_record(case) + return None if record is None else record.get("scout") + + def start(self, prepare: Callable[[], tuple[Path, Path, dict, list[str]]]) -> _Job: """Run `prepare` — every check that can refuse before anything is spent — then the pipeline on its own thread, and return without waiting for it.""" with self._lock: @@ -670,11 +758,15 @@ def start(self, prepare: Callable[[], tuple[Path, Path, dict]]) -> _Job: f"{running.started}). One audit runs at a time — follow it with " "audit_status, and start this one when it has ended." ) - case, manuscript, kwargs = prepare() - job = _Job(case=case, manuscript=manuscript, started=_now()) + case, manuscript, kwargs, said = prepare() + job = _Job(case=case, manuscript=manuscript, started=_now(), + scout=kwargs["with_scout"], log=list(said)) self._running = self._by_case[case] = job - # daemon: when the host stops the server, the audit stops with it - # rather than spending on for a client that has gone + # daemon: when the host stops the server, no further stage and no + # further model call starts. A `claude -p` call already in flight + # runs to its own end — `ask.py`'s timeout bounds it — and the record + # left saying "running" is what tells a later server the case is + # incomplete. threading.Thread( target=self._run, args=(job, kwargs), name="papertrace-audit", daemon=True ).start() @@ -684,6 +776,10 @@ def _run(self, job: _Job, kwargs: dict) -> None: state, error = "failed", "" try: with _console_into(job.log), _nobody_to_ask(): + # the folder and its .gitignore first, so the record never lands + # in a folder that is briefly untracked + cli._open_case(job.case) + _write_record(job) # the process outlives one audit: a model an earlier audit # recorded must not be reported as this one's judge forget_models() @@ -704,6 +800,13 @@ def _run(self, job: _Job, kwargs: dict) -> None: if state == "failed" and not error: error = "the audit was interrupted before it could finish" job.state, job.error, job.ended = state, error, _now() + try: + _write_record(job) + except OSError as e: + # this server still knows the outcome; a later one reads the + # "running" left behind as an interrupted audit, which refuses — + # the safe side + job.log.append(f"the audit record could not be updated: {e}") with self._lock: if self._running is job: self._running = None @@ -715,16 +818,32 @@ def _run(self, job: _Job, kwargs: dict) -> None: "finished": "call audit_summary for the result — its counts travel with the caveats that " "qualify them", "failed": "read `error` and the log, fix what they name, and call start_audit again", - "not_started": "no audit of this case was started by this server process — audit_summary " - "reads whatever an earlier run left on disk", + "interrupted": "the server running this audit stopped before it ended — call start_audit " + "to run it again", + "not_started": "no audit of this case was started by an MCP server — audit_summary reads " + "whatever a CLI run left on disk", } -def _status(case: Path, job: _Job | None, log_lines: int) -> AuditStatus: +def _status(case: Path, job: _Job | None, log_lines: int, record: dict | None = None) -> AuditStatus: + """The job's state — this server's own, else the record an earlier one left. + + `log` and `log_lines_total` are null, not empty, when no log exists here: an + empty log reads as an audit that printed nothing. + """ + if job is None and record is not None: + state = "interrupted" if record["state"] == "running" else record["state"] + return { + "case": str(case), "state": state, "manuscript": record.get("manuscript"), + "started": record.get("started"), "ended": record.get("ended"), + "error": record.get("error") or "", "log": None, "log_lines_total": None, + "next": f"{_NEXT[state]} — recorded by an earlier server process, whose log " + "was not kept", + } if job is None: return { "case": str(case), "state": "not_started", "manuscript": None, "started": None, - "ended": None, "error": "", "log": [], "log_lines_total": 0, + "ended": None, "error": "", "log": None, "log_lines_total": None, "next": _NEXT["not_started"], } log = list(job.log) @@ -752,9 +871,9 @@ def build_server() -> MCPServer: jobs = _Jobs() def _open(case: str) -> Path: - """A case folder to read — unless this server is writing it right now.""" + """A case folder to read — unless the last audit here has not ended well.""" root = _case_dir(case) - jobs.refuse_running(root) + jobs.refuse_unfinished(root) return root @server.tool(title="Audit summary", annotations=_READ, structured_output=True) @@ -798,12 +917,16 @@ def list_claims( results = _results(root) manifest = _manifest(root, required=False) rows = [_claim_row(c, manifest) for c in results.claims if verdict in (None, c.verdict)] + fired = run_disclosures(results, manifest) return { "case": str(root), "verdict_filter": verdict, "claims_total": len(results.claims), "claims": rows, - "limited": _limited(run_disclosures(results, manifest)), + # a host can list claims without ever reading the summary, so the + # caveats the reports print above every verdict come here too + "run_disclosures": _disclosures(fired), + "limited": _limited(fired), } @server.tool(title="Claim detail", annotations=_READ, structured_output=True) @@ -837,6 +960,9 @@ def get_claim(case: CaseArg, claim_id: ClaimId) -> ClaimDetail: "unjudged_refs": list(c.unjudged_refs), "withheld_refs": list(c.withheld_refs), "disclosures": _disclosures(claim_disclosures(c, manifest)), + # the caveats the reports print above every verdict: an unconfirmed + # source identity, a source read flat or cut short + "run_disclosures": _disclosures(run_disclosures(results, manifest)), } @server.tool(title="Evidence crops", annotations=_READ, structured_output=False) @@ -853,7 +979,9 @@ def get_evidence( The boxes were drawn by code finding the model's anchor phrases in the PDF, not placed by the model; the caption says whether the phrase was found.""" root = _open(case) - claim = _find_claim(_results(root), claim_id) + results = _results(root) + manifest = _manifest(root, required=False) + claim = _find_claim(results, claim_id) target = _evidence_target(claim, source) rels = _images_of(target) if not rels: @@ -869,7 +997,8 @@ def get_evidence( f"results.json but not in {out} — re-run `papertrace highlight -c `" ) return [ - TextContent(type="text", text=_caption(claim, target, shown, missing)), + TextContent(type="text", text=_caption( + claim, target, shown, missing, manifest, run_disclosures(results, manifest))), *(Image(path=p) for p in shown), ] @@ -946,7 +1075,7 @@ def list_gaps(case: CaseArg) -> Gaps: results = _results(root) manifest = _manifest(root, required=False) coverage = results.coverage or {} - audited = _coverage_audited(coverage) + audited = coverage_audited(coverage) occurrences = coverage.get("occurrences") fired = run_disclosures(results, manifest) return { @@ -988,6 +1117,12 @@ def get_scout(case: CaseArg) -> ScoutOut: f"no literature scan at {path} — the audit ran without the scout, or the " "scout stage has not run (`papertrace scout -c `)" ) + if jobs.scanned(root) is False: + raise ToolError( + f"the latest audit of {root} ran without the scout, so {path} belongs to an " + "earlier run — its registers may be anchored to another DOI or another " + "version of this paper, and are not served as this audit's" + ) scout = _read(path, ScoutResults.from_json, "scout") return {"case": str(root), **scout.to_dict(), "caveat": _scout_caveat(scout)} @@ -1049,7 +1184,7 @@ def start_audit( if pdfs is not None and not pdfs.is_dir(): raise ToolError(f"`provided` must be a folder of PDFs, and {pdfs} is not one") - def prepare() -> tuple[Path, Path, dict]: + def prepare() -> tuple[Path, Path, dict, list[str]]: # everything that can refuse, refused before a model or a network is touched from .check import claude_available @@ -1060,6 +1195,14 @@ def prepare() -> tuple[Path, Path, dict]: "installed, the host started this server with a PATH that does not " "reach it: set PATH in the server's configuration." ) + if case is None and not os.access(pdf.parent, os.W_OK): + # `papertrace run` falls back to its working directory, one the + # person chose by standing in it; this server's was chosen by the + # host, so the audit would land wherever that happens to be + raise ToolError( + f"the paper's folder {pdf.parent} is not writable, so the audit cannot sit " + "beside it — pass `case`, the folder to keep it in" + ) said: list[str] = [] try: with _console_into(said): @@ -1075,7 +1218,7 @@ def prepare() -> tuple[Path, Path, dict]: "model": model, "png": False, "backend": backend, "with_scout": scout, "doi": doi, "formats": list(formats or []), "supplement": own, "llm_refs": llm_refs, "max_claims": max_claims, "max_sources": max_sources, - } + }, said # what the checks printed opens the job's log job = jobs.start(prepare) return _status(job.case, job, log_lines=40) @@ -1097,7 +1240,7 @@ def audit_status( job = jobs.last(root) if job is not None and wait_seconds: job.done.wait(wait_seconds) - return _status(root, job, log_lines) + return _status(root, job, log_lines, None if job is not None else _read_record(root)) return server diff --git a/tests/test_mcp_server.py b/tests/test_mcp_server.py index 6f51ca6..eb23831 100644 --- a/tests/test_mcp_server.py +++ b/tests/test_mcp_server.py @@ -11,7 +11,9 @@ """ import json +import os import sys +import types from pathlib import Path import pytest @@ -19,9 +21,16 @@ sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "src")) # `[dev]` installs the SDK, so CI runs this module rather than skipping it; -# `tests/test_packaging.py` asserts that it stays there +# `tests/test_packaging.py` asserts that it stays there. A 1.x SDK — other tools +# install one — is skipped with the same reason rather than failing collection. pytest.importorskip("mcp", reason="pip install -e '.[dev]' supplies the MCP SDK") +from importlib.metadata import version as _installed # noqa: E402 + +if int(_installed("mcp").split(".")[0]) < 2: + pytest.skip("the server is written against the MCP SDK 2.x — pip install -e '.[dev]'", + allow_module_level=True) + import anyio # noqa: E402 import typer # noqa: E402 from mcp import Client # noqa: E402 @@ -52,6 +61,13 @@ "audit_summary", "list_claims", "get_claim", "get_evidence", "list_references", "list_gaps", "get_scout", } +# one call to each read tool, with whatever arguments it needs beyond `case` +READ_CALLS = ( + ("audit_summary", {}), ("list_claims", {}), ("get_claim", {"claim_id": 1}), + ("get_evidence", {"claim_id": 1}), ("list_references", {}), ("list_gaps", {}), + ("get_scout", {}), +) +ROOT = Path(__file__).resolve().parent.parent # -------------------------------------------------------------------------- @@ -957,9 +973,7 @@ def test_read_tools_refuse_the_case_whose_audit_is_running(ready, monkeypatch, t started = _start(server, manuscript=_paper(tmp_path), case=case) try: - for tool, args in (("audit_summary", {}), ("list_claims", {}), ("get_claim", {"claim_id": 1}), - ("get_evidence", {"claim_id": 1}), ("list_references", {}), - ("list_gaps", {}), ("get_scout", {})): + for tool, args in READ_CALLS: result = _call(tool, {"case": started["case"], **args}, server) assert result.is_error, tool assert "running" in _text(result), tool @@ -1006,7 +1020,7 @@ def test_the_status_of_a_case_this_server_never_ran_says_so(tmp_path): status = _json(_call("audit_status", {"case": str(case)})) assert status["state"] == "not_started" - assert "this server" in status["next"] + assert "MCP server" in status["next"] # -------------------------------------------------------------------------- @@ -1071,3 +1085,202 @@ def test_without_the_sdk_papertrace_mcp_says_how_to_install_it(monkeypatch, caps assert stop.value.exit_code == 2 assert out == "" assert "pip install 'papertrace[mcp]'" in err + + +# -------------------------------------------------------------------------- +# review findings — each a way the server could present a case as a result +# it is not, or a verdict without what qualifies it +# -------------------------------------------------------------------------- + + +def _record(case: Path) -> dict: + return json.loads((case / "mcp_audit.json").read_text()) + + +def test_after_a_failed_audit_nothing_on_disk_is_served_as_its_result( + ready, monkeypatch, tmp_path +): + """A rerun can rewrite the manifest and die before `check`, leaving one + run's references beside another run's verdicts. The case is refused with + the failure, never served as a result.""" + def body(kw): + RefManifest(manuscript="paper.pdf").to_json(Path(kw["case"]) / "refs_manifest.json") + raise RuntimeError("claude -p failed: not logged in") + + case = _case(tmp_path) # an earlier, complete audit + RefManifest(manuscript="paper.pdf").to_json(case / "refs_manifest.json") + _fake_pipeline(monkeypatch, body) + server = mcp_server.build_server() + started = _start(server, manuscript=_paper(tmp_path), case=case) + assert _await(server, started["case"])["state"] == "failed" + + for tool, args in READ_CALLS: + result = _call(tool, {"case": started["case"], **args}, server) + assert result.is_error, tool + assert "failed" in _text(result) and "not logged in" in _text(result), tool + + +def test_an_audit_its_server_never_finished_is_refused_and_reported_interrupted(tmp_path): + """A host that stops the server stops the audit mid-run. Whatever it wrote + is on disk, and the only thing that says it is incomplete is the record the + job wrote when it started — read here by a server that never ran it.""" + case = _case(tmp_path) + (case / "mcp_audit.json").write_text(json.dumps({ + "schema": "mcp_audit/1", "state": "running", "manuscript": "paper.pdf", + "started": "2026-09-26T10:00:00+00:00", "ended": None, "error": "", "scout": True, + })) + server = mcp_server.build_server() + + status = _json(_call("audit_status", {"case": str(case)}, server)) + summary = _call("audit_summary", {"case": str(case)}, server) + + assert status["state"] == "interrupted" + assert status["started"] == "2026-09-26T10:00:00+00:00" + assert summary.is_error and "did not finish" in _text(summary) + + +def test_the_record_outlives_the_server_that_wrote_it(ready, monkeypatch, tmp_path): + jsonschema = pytest.importorskip("jsonschema") + _fake_pipeline(monkeypatch) + first = mcp_server.build_server() + case = _await(first, _start(first, manuscript=_paper(tmp_path), scout=False)["case"])["case"] + + status = _json(_call("audit_status", {"case": case}, mcp_server.build_server())) + + assert status["state"] == "finished" + assert "not kept" in status["next"], "a log that did not survive must say so" + record = _record(Path(case)) + assert record["state"] == "finished" and record["scout"] is False + jsonschema.validate(record, json.loads((ROOT / "schemas" / "mcp_audit.schema.json").read_text())) + + +def test_a_claim_read_on_its_own_still_carries_the_runs_caveats(tmp_path): + """A host can call get_claim without ever calling audit_summary. The source + behind claim 1's headline is one whose identity nobody confirmed — the + reports say so above every verdict, so a verdict read alone says it too.""" + results = _results(scope=LIMITED_SCOPE) + case = _case(tmp_path, results=results) + + for tool, args in (("get_claim", {"claim_id": 1}), ("list_claims", {})): + body = _text(_call(tool, {"case": str(case), **args})) + for d in run_disclosures(results, _manifest()): + assert d.token in body, (tool, d.key) + + +def test_the_evidence_caption_carries_every_caveat_on_its_verdict(tmp_path): + """The crop is shown under its verdict, so the verdict's caveats come with + it — here [6] sits past where the numbering stopped being confirmed, and a + crop captioned without that would show the wrong paper as evidence.""" + claim = _paraphrase_claim(True) + claim.refs = ["6"] + results = _results(claims=[claim]) + case = _case(tmp_path, results=results) + + caption = _text(_call("get_evidence", {"case": str(case), "claim_id": 4})) + + fired = claim_disclosures(claim, _manifest()) + run_disclosures(results, _manifest()) + assert "claim_numbering" in {d.key for d in fired} + for d in fired: + assert d.token in caption, d.key + + +def test_start_audit_will_not_put_an_audit_wherever_the_host_started_it( + ready, monkeypatch, tmp_path +): + """`papertrace run` keeps the audit in the working directory when the + paper's folder is read-only — a directory the person chose by standing in + it. A host chose this server's, so start_audit asks for a case instead.""" + calls = _fake_pipeline(monkeypatch) + pdf = _paper(tmp_path) + access = os.access + monkeypatch.setattr(os, "access", lambda p, mode: Path(p) != pdf.parent and access(p, mode)) + + result = _call("start_audit", {"manuscript": str(pdf)}) + + assert result.is_error + assert str(pdf.parent) in _text(result) and "`case`" in _text(result) + assert calls == [] + + +def test_what_the_checks_said_before_the_audit_is_kept_in_its_log(ready, monkeypatch, tmp_path): + """A legacy case folder is accepted with a warning that its identity was + compared by name only. That warning is printed before the job starts, and + it belongs in the job's log, not a buffer nobody reads.""" + _fake_pipeline(monkeypatch) + case = tmp_path / "case" + case.mkdir() + RefManifest(manuscript="paper.pdf").to_json(case / "refs_manifest.json") + server = mcp_server.build_server() + + status = _await(server, _start(server, manuscript=_paper(tmp_path), case=case)["case"]) + + assert any("predates content hashing" in line for line in status["log"]) + + +def test_an_incomplete_scan_says_so_in_its_caveat(tmp_path): + scout = ScoutResults(paper_title="A paper", paper_identity="confirmed", + error="Europe PMC timed out") + case = _case(tmp_path, scout=scout) + + assert "Europe PMC timed out" in _json(_call("get_scout", {"case": str(case)}))["caveat"] + + +def test_a_scan_the_latest_audit_did_not_run_is_not_served_as_its_own( + ready, monkeypatch, tmp_path +): + """An audit run without the scout leaves an earlier run's scout.json on + disk, anchored to whatever paper and DOI that run used.""" + case = _case(tmp_path, scout=ScoutResults(paper_title="A paper", paper_identity="confirmed")) + RefManifest(manuscript="paper.pdf").to_json(case / "refs_manifest.json") + _fake_pipeline(monkeypatch) + server = mcp_server.build_server() + _await(server, _start(server, manuscript=_paper(tmp_path), case=case, scout=False)["case"]) + + result = _call("get_scout", {"case": str(case)}, server) + + assert result.is_error + assert "without the scout" in _text(result) + + +def test_an_sdk_too_old_for_the_server_gets_the_install_line_too(monkeypatch, capsys): + """A 1.x SDK — other tools install one — has no `MCPServer`, and fails with + an ImportError that is not a ModuleNotFoundError.""" + for name in [n for n in sys.modules if n == "mcp" or n.startswith("mcp.")]: + monkeypatch.delitem(sys.modules, name) + old = types.ModuleType("mcp") + old.__path__ = [] + monkeypatch.setitem(sys.modules, "mcp", old) + monkeypatch.setitem(sys.modules, "mcp.server", types.ModuleType("mcp.server")) + monkeypatch.delitem(sys.modules, "papertrace.mcp_server", raising=False) + + with pytest.raises(typer.Exit) as stop: + cli.mcp_command() + + out, err = capsys.readouterr() + assert stop.value.exit_code == 2 and out == "" + assert "pip install 'papertrace[mcp]'" in err + + +STRUCTURED = { + "audit_summary": "AuditSummary", "list_claims": "ClaimList", "get_claim": "ClaimDetail", + "list_references": "ReferenceList", "list_gaps": "Gaps", "get_scout": "ScoutOut", + "audit_status": "AuditStatus", +} + + +def test_every_tool_output_matches_its_schema_in_schemas(tmp_path): + """Gate 2: what the server publishes is a contract, and `schemas/` is where + this repository's contracts live. Each output validates against it, and + each schema names exactly the keys its tool returns — a field added to one + and not the other turns this red.""" + jsonschema = pytest.importorskip("jsonschema") + published = json.loads((ROOT / "schemas" / "mcp_tools.schema.json").read_text()) + case = _case(tmp_path, results=_results(scope=LIMITED_SCOPE), + scout=ScoutResults(paper_title="A paper", paper_identity="unverified")) + args = {"get_claim": {"claim_id": 1}} + + for tool, shape in STRUCTURED.items(): + out = _json(_call(tool, {"case": str(case), **args.get(tool, {})})) + jsonschema.validate(out, {**published, "$ref": f"#/$defs/{tool}"}) + declared = set(published["$defs"][tool]["properties"]) + assert declared == set(getattr(mcp_server, shape).__annotations__), tool From eecb731b32f51f230a19cf54a1fd9c75b8d0b07d Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 26 Sep 2026 12:58:43 +0000 Subject: [PATCH 10/13] docs: the review round in the spec, ADR, README, CHANGELOG and CLAUDE.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The spec gains its eleventh rule — a case is not a result until its audit has ended — and loses two claims the review showed wrong: that stopping the server stops the audit (a `claude -p` call in flight runs to its end), and that job state is not persisted (mcp_audit.json now carries the outcome). The rejected "schema only in tools/list" bullet now says what gate 2 got instead. Two limits are written down rather than left to be found: two server processes auditing one case are not detected, and a CLI run that fails midway leaves no record. README, CHANGELOG and CLAUDE.md say the same, and the plan records the review round, including the finding that was not taken and why. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01J6JtZg47dHEKHDEfAJ5Kpf --- CHANGELOG.md | 7 ++- CLAUDE.md | 12 +++-- README.md | 6 ++- docs/adr/0004-mcp-server.md | 6 ++- .../plans/2026-09-26-mcp-server.md | 22 ++++++++++ .../specs/2026-09-26-mcp-server-design.md | 44 +++++++++++++------ 6 files changed, 76 insertions(+), 21 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ad6f659..aae2cd4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -23,7 +23,12 @@ viewer's way, with a parity test extending the four-format contract to this fifth reader — and a limited audit says so first and last. A missing or unreadable `results.json` is refused with its reason, never answered with empty counts; `null` in the manifest's three-state fields stays `null`; no -local PDF path is served, and no evidence image from outside `/out/`. +local PDF path is served, and no evidence image from outside `/out/`. A +verdict read on its own — `get_claim`, the evidence caption — carries the +run-level caveats too. Each audit leaves `/mcp_audit.json`: a case whose +last MCP audit failed or never finished is refused rather than read, because +its files may mix two runs. The tools' outputs and that record are published in +`schemas/mcp_tools.schema.json` and `schemas/mcp_audit.schema.json`. An audit takes minutes and a host built on the TypeScript SDK gives a request 60 s, so `start_audit` returns at once and `audit_status` long-polls for at diff --git a/CLAUDE.md b/CLAUDE.md index 7be8f80..1e39bb5 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -352,9 +352,15 @@ ingest → extract → refs → scout → check → highlight → report with `chosen_by: "default"` and the person-gated escalation is unreachable — do not offer MCP elicitation in its place: a host may be a model), and `ask.forget_models()` runs first. Those three are process-global, hence - **one audit at a time**, and read tools refuse a case whose audit is still - running. **Any new process-global state in the pipeline must be reset per - audit here.** In SDK v2 only a `ToolError`'s message reaches the model — any + **one audit at a time**. The job writes `/mcp_audit.json` (`running`, + then the outcome), and read tools refuse a case whose last MCP audit is + running, failed, or never finished (a `running` record no live job owns is + `interrupted`): a failed rerun leaves one run's manifest beside another's + verdicts. A verdict read alone — `get_claim`, the evidence caption — carries + the run-level disclosures as well as its own. Outputs are published in + `schemas/mcp_tools.schema.json`, each definition's keys held equal to its + `TypedDict`'s by a test. **Any new process-global state in the pipeline + must be reset per audit here.** In SDK v2 only a `ToolError`'s message reaches the model — any other exception is a bare "Error executing tool" — so every refusal is one. Return types are `typing_extensions.TypedDict` with `structured_output=True`: `typing.TypedDict` silently cost every tool its output schema below 3.12. diff --git a/README.md b/README.md index 775321b..deab5df 100644 --- a/README.md +++ b/README.md @@ -343,8 +343,10 @@ the server's `PATH` — if the host's is too minimal, add its directory in `PAPERTRACE_EMAIL`, or from the address `papertrace` saved. A missing manuscript, a `claude` the server cannot find, no email, or a case folder holding another paper is refused before anything is spent. One audit runs at a -time per server, and a case folder the server is writing is not read until the -audit ends. Nobody is at an MCP call to answer a question, so a reference label +time per server. A case folder is not read while its audit runs, nor after one +that failed or never finished — its files may mix two runs — which the server +knows from the `mcp_audit.json` each audit leaves in the folder. Nobody is at an +MCP call to answer a question, so a reference label whose readings disagree is withheld, never settled — that takes you, at a terminal. Nothing but the protocol reaches stdout; what the pipeline prints comes back in `audit_status`'s log. diff --git a/docs/adr/0004-mcp-server.md b/docs/adr/0004-mcp-server.md index 8bef922..8582bce 100644 --- a/docs/adr/0004-mcp-server.md +++ b/docs/adr/0004-mcp-server.md @@ -50,7 +50,9 @@ the thread has nobody to ask.** evidence crops as images. - A disputed label cannot be settled from an MCP host. That is deliberate; `papertrace refs` in a terminal settles it. -- A running audit cannot be cancelled from the host; stopping the server stops - it, and the case folder is re-runnable. +- A running audit cannot be cancelled from the host. Stopping the server starts + no further stage or model call — a `claude -p` call already in flight runs to + its end — and the `mcp_audit.json` it leaves saying `running` keeps the half- + written case from being read as a result until an audit there finishes. - Any future process-global state in the pipeline must be reset per audit, or it will leak from one audit into the next. `CLAUDE.md` says so. diff --git a/docs/superpowers/plans/2026-09-26-mcp-server.md b/docs/superpowers/plans/2026-09-26-mcp-server.md index b8fb41e..b7b248f 100644 --- a/docs/superpowers/plans/2026-09-26-mcp-server.md +++ b/docs/superpowers/plans/2026-09-26-mcp-server.md @@ -133,3 +133,25 @@ pymupdf. Each test calls through `Client(build_server())`. - [ ] CHANGELOG: an Unreleased entry. - [ ] CLAUDE.md: the architecture bullet for `mcp_server.py`, and the long-lived-process rule (`forget_models`, one job at a time). + +## Task 7 — the review round + +A code review of the branch (the `code-review` skill, high effort) returned ten +findings; nine held up against the code, each fixed test-first. The tenth — +that `_run_pipeline`'s OptionInfo guards are dead — did not: tests call +`cli.run()` directly and omit options, which hands the adapter's sentinels +straight through. + +- [x] A failed or interrupted audit's mixed case is refused, via the + `mcp_audit.json` record; `audit_status` reads an earlier server's record. +- [x] Run-level disclosures in `get_claim`, `list_claims` and the evidence + caption, beside the claim's own. +- [x] No fallback to the server's working directory for an unwritable paper + folder; the preflight's lines open the job's log. +- [x] A `scout.json` the latest audit did not write is refused; the caveat + carries the scan's error. +- [x] A 1.x SDK gets the install line; the command is never wrapped in two. +- [x] `schemas/mcp_tools.schema.json` and `schemas/mcp_audit.schema.json`, + with validation tests. +- [x] `disclosures.coverage_audited` and `cli.SCOUT_CAVEAT`: one rule and one + sentence each, not two copies. diff --git a/docs/superpowers/specs/2026-09-26-mcp-server-design.md b/docs/superpowers/specs/2026-09-26-mcp-server-design.md index 8d8d8f6..df667ce 100644 --- a/docs/superpowers/specs/2026-09-26-mcp-server-design.md +++ b/docs/superpowers/specs/2026-09-26-mcp-server-design.md @@ -50,7 +50,9 @@ pipeline wrote. `start_audit` is the only tool that spends — model calls throu call `run_disclosures`, `claim_disclosures` and `judgement_disclosures` — the same producers the four report formats use — and serialise them with the viewer's own `report._disclosure_dict`. The MCP output is a fifth reader, and - the parity test is extended to it: every token reaches it. + the parity test is extended to it: every token reaches it. A host can read + one claim without ever reading the summary, so `get_claim`, `list_claims` + and the evidence caption carry the run-level disclosures too. 3. **A limited run never reads as a smaller paper.** The scope disclosure is first among the disclosures, as in every report, and every result that states counts or lists claims ends with a `limited` field carrying the same sentence @@ -89,6 +91,15 @@ pipeline wrote. `start_audit` is the only tool that spends — model calls throu without `pdf_path`, as in the viewer. An evidence image is read only when it resolves inside `/out/` and is a `.png` — a `results.json` edited to point at `../../somewhere` is refused, not followed. +11. **A case is not a result until its audit has ended.** A rerun can rewrite + the manifest and die before `check`, leaving one run's references beside + another's verdicts. The job writes `/mcp_audit.json` — `running` + before the pipeline starts, the outcome when it ends — and read tools + refuse a case whose last MCP audit failed or never finished. A server + stopped mid-audit leaves `running` behind, which a later server reports as + `interrupted`. An optional stage's file the latest audit did not write — a + `scout.json` from before a run without the scout — is refused as an + earlier run's. ## Long runs: a job, not a blocking call @@ -109,9 +120,10 @@ per-site model record and `sys.stdin` are process-global; two audits sharing them would interleave logs and misattribute models. A second `start_audit` is refused, naming the audit in progress. -**Read tools refuse a case whose audit is running.** The `results.json` on disk -then belongs to the previous run, or is half-written, and presenting it as this -run's would be the silent failure this codebase exists to refuse. +**Read tools refuse a case whose audit is running, failed or never finished.** +The files on disk then belong to an earlier run, are half-written, or mix the +two, and presenting them as a result would be the silent failure this codebase +exists to refuse. `audit_status` never returns counts. A count without its disclosures is exactly what rule 2 forbids, so a finished job points at `audit_summary`. @@ -139,10 +151,12 @@ what rule 2 forbids, so a finished job points at `audit_summary`. revision (SEP-2577). The job log is returned by `audit_status` instead. - **Streamable HTTP.** A case folder stays on the machine that made it, by design. stdio only. -- **A hand-written schema in `schemas/`.** The output schema is generated from - the tools' `TypedDict` return types and published by `tools/list`; a copy - would be a second contract to drift. No case-folder file gains a field, so - gate 2 is untouched. +- **An output schema only in `tools/list`.** The SDK publishes one generated + from each tool's `TypedDict`, but `schemas/` is where this repository keeps + its contracts (gate 2), so `schemas/mcp_tools.schema.json` holds them too — + every tool output is validated against it, and each definition's keys are + held equal to its `TypedDict`'s, so the two cannot drift apart silently. + `schemas/mcp_audit.schema.json` covers the one file the server writes. ## Deliberate omissions @@ -150,11 +164,15 @@ what rule 2 forbids, so a finished job points at `audit_summary`. server serves are the evidence crops, which need neither. - A running audit cannot be cancelled. A thread cannot be stopped safely mid-write; the CLI's Ctrl-C carries the same exposure. If the host stops the - server, the job's daemon thread stops with it and the case folder is - re-runnable. -- Job state is not persisted across server restarts. `audit_status` says so - ("no audit of this case was started by this server process") rather than - guessing from the files on disk. + server, no further stage and no further model call starts — but a `claude -p` + call already in flight runs to its end, bounded by `ask.py`'s timeout — and + the record left saying `running` marks the case incomplete. +- A job's log is not persisted; its outcome is, in `mcp_audit.json`, and an + earlier server's audit reports `log: null` rather than an empty log. +- Two server processes auditing one case at once are not detected: one audit + at a time is a per-process rule, and the CLI has no such guard either. +- A CLI run that fails midway writes no record, so the server reads that case + as the reports would. ## Testing From 4d6d528bbb7e1c91d0107302e7271203f53ed272 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 26 Sep 2026 13:08:18 +0000 Subject: [PATCH 11/13] =?UTF-8?q?mcp:=20second=20review=20pass=20=E2=80=94?= =?UTF-8?q?=20the=20record=20says=20only=20what=20it=20measured?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A second review, of the first round's fixes, found ten more; each held up against the code and each has a test that failed first. - A "running" record was reported "interrupted" whatever its age, which asserted a cause nobody measured: with two hosts each launching their own server, the other server may be alive, and "call start_audit" would put a second pipeline in its folder. The job now refreshes its record every 15 s; fresh means live in another process — read tools and start_audit refuse, audit_status reports "running" and waits on the record — and only a record unrefreshed for 60 s is "interrupted". - The disk record now decides, not this server's memory of its jobs: a case completed since by other means reads again. A failed audit is superseded by a results.json written after it, and an audit run without the scout by a scout.json written after it — the record's mtime is the measurement. - The evidence caption for a named source carried the headline source's anchor state beside its own; it now carries only the shown source's. - list_gaps reported unreached_labels [] for a label audit that never ran (no clean.md, occurrences counted); it is null unless labels were read. - audit_status's manuscript was a full path from this server and a file name from a record; it is the absolute path in both, and the record stores that. - A recorded failure's next step sent the host to read a log that was not kept; recorded states have their own wording. - The record is validated — schema version, state, scout as a bool — before anything is decided on it; "false" is not False. - `papertrace mcp` re-raises an import failure when a 2.x SDK is installed, instead of an install line contradicting itself over a hidden traceback. - The published schema describes record-derived states; a test validates every state a record can produce, and another holds the prose timings to the constants. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01J6JtZg47dHEKHDEfAJ5Kpf --- schemas/mcp_audit.schema.json | 6 +- schemas/mcp_tools.schema.json | 9 +- src/papertrace/cli.py | 10 +- src/papertrace/disclosures.py | 8 +- src/papertrace/mcp_server.py | 199 ++++++++++++++++++-------- tests/test_mcp_server.py | 257 +++++++++++++++++++++++++++++++++- 6 files changed, 416 insertions(+), 73 deletions(-) diff --git a/schemas/mcp_audit.schema.json b/schemas/mcp_audit.schema.json index 7f1f97a..019a37c 100644 --- a/schemas/mcp_audit.schema.json +++ b/schemas/mcp_audit.schema.json @@ -1,7 +1,7 @@ { "$schema": "https://json-schema.org/draft/2020-12/schema", "title": "mcp_audit", - "description": "/mcp_audit.json — what an MCP server (`papertrace mcp`) knows about the last audit it started in this case folder. The audit job writes it with state `running` before the pipeline touches anything, and rewrites it when the job ends. A server stopped mid-audit therefore leaves `running` behind, and a later server reads that as an audit that never finished: it refuses to serve the folder as a result rather than present half-written files as one. Absent means no MCP server has audited this case; the CLI never writes it.", + "description": "/mcp_audit.json — what an MCP server (`papertrace mcp`) records about the last audit it started in this case folder, for itself and any other server process. The audit job writes it with state `running` before the pipeline touches anything, refreshes its modification time every 15 s while it runs, and rewrites it with the outcome when it ends. A `running` record is therefore measured: refreshed within 60 s, an audit is live in some server process, and no second one is started in the folder; staler, the server writing it stopped mid-run, and the folder is not read as a result. A `failed` record keeps the folder from being read until a results.json written after it shows a later run completed the judging. Absent means no MCP server has audited this case; the CLI never writes it.", "type": "object", "required": [ "schema", @@ -26,11 +26,11 @@ "finished", "failed" ], - "description": "`running` while the job runs — and forever, when the server stopped before the job ended, which a later server reports as `interrupted`. `failed`: the pipeline stopped, and what is on disk may mix that run with an earlier one." + "description": "`running` while the job runs — and after, when the server stopped first, which the record's unrefreshed modification time shows. `failed`: the pipeline stopped, and what is on disk may mix that run with an earlier one." }, "manuscript": { "type": "string", - "description": "The audited PDF's file name, as refs_manifest.json keeps it." + "description": "The audited PDF's absolute path — the same value audit_status gives." }, "started": { "type": "string", diff --git a/schemas/mcp_tools.schema.json b/schemas/mcp_tools.schema.json index cf95afe..6b3ac33 100644 --- a/schemas/mcp_tools.schema.json +++ b/schemas/mcp_tools.schema.json @@ -466,13 +466,14 @@ "not_started" ], "type": "string", - "description": "running, finished, failed: this server's own job. interrupted: the record an earlier server left says running, so that audit never ended. not_started: no MCP server has audited this case." + "description": "The last audit an MCP server started in this case. running, finished, failed: this server's own job, with its log — or another server process's record of one, with log null. A record says running only while it is being refreshed (every 15 s); one not refreshed for 60 s is interrupted: the server writing it stopped mid-run. not_started: no MCP server has audited this case." }, "manuscript": { "type": [ "string", "null" - ] + ], + "description": "The audited PDF's absolute path; null when not_started." }, "started": { "type": [ @@ -497,14 +498,14 @@ "array", "null" ], - "description": "The job's latest console lines; null when no log exists in this process — a recorded or never-started audit — because an empty log would read as an audit that printed nothing." + "description": "This server's job's latest console lines; null when the audit is another server process's or was never started — an empty log would read as an audit that printed nothing." }, "log_lines_total": { "type": [ "integer", "null" ], - "description": "How many console lines the job printed; null when no log exists in this process." + "description": "How many console lines this server's job printed; null when there is no log here." }, "next": { "type": "string" diff --git a/src/papertrace/cli.py b/src/papertrace/cli.py index 773d4df..d8ec0e0 100644 --- a/src/papertrace/cli.py +++ b/src/papertrace/cli.py @@ -467,9 +467,15 @@ def mcp_command() -> None: from importlib.metadata import PackageNotFoundError, version try: - found = f" (found mcp {version('mcp')})" + installed = version("mcp") except PackageNotFoundError: - found = "" + installed = None + # a 2.x SDK that still fails to import is a real fault: the install line + # would contradict itself and hide the traceback that names what broke + if installed is not None and installed.split(".")[0].isdigit() \ + and int(installed.split(".")[0]) >= 2: + raise + found = f" (found mcp {installed})" if installed else "" # rich eats [mcp] as a style tag; escaping it is what prints the extra. # soft_wrap: wrapped at the terminal's width, the command split in two Console(stderr=True, soft_wrap=True).print( diff --git a/src/papertrace/disclosures.py b/src/papertrace/disclosures.py index 7bf6ff1..34afefe 100644 --- a/src/papertrace/disclosures.py +++ b/src/papertrace/disclosures.py @@ -311,9 +311,11 @@ def coverage_audited(coverage: dict) -> bool: """Did the coverage audit run at all? One rule, because two readers need it. The reports decide on it whether to print a ratio or `coverage not audited`, - and the MCP server whether a gap list is a measurement or `null`. Were the - two copies to disagree, one reader would call "never measured" what the - other calls "measured, nothing missing". + and the MCP server whether its occurrence-level gap list is a measurement or + `null`. Were two copies to disagree, one reader would call "never measured" + what the other calls "measured, nothing missing". It answers for the audit + as a whole, not for each list: the label-level lists come from `clean.md` + and can be empty because nothing read them while the occurrences counted. Occurrences without labels means `clean.md` was missing while the source map was not: the audit ran, so it must not read as "not audited". diff --git a/src/papertrace/mcp_server.py b/src/papertrace/mcp_server.py index 36a9827..8e3708d 100644 --- a/src/papertrace/mcp_server.py +++ b/src/papertrace/mcp_server.py @@ -16,6 +16,7 @@ import os import sys import threading +import time from collections.abc import Callable, Iterator from contextlib import contextmanager from dataclasses import dataclass, field @@ -530,7 +531,9 @@ def _caption( ) if missing: lines.append(f"not in the case folder, so not shown: {', '.join(missing)}") - if caveats := [d.short for d in claim_disclosures(claim, manifest)]: + # every claim caveat but the claim-level anchor: that is the headline's, and + # the anchor line above already says the shown source's own + if caveats := [d.short for d in claim_disclosures(claim, manifest) if d.key != "anchor"]: lines += ["caveats on this claim:", *(f"- {c}" for c in caveats)] if run_fired: lines += ["caveats on the whole audit:", *(f"- {d.short}" for d in run_fired)] @@ -631,14 +634,19 @@ def _last_said(lines: list[str], n: int = 4) -> str: return " ".join(line.strip() for line in [x for x in lines if x.strip()][-n:]) -# What a later server knows about the last audit an MCP server started in a case -# folder. The job writes it "running" before the pipeline touches anything and -# rewrites it when the job ends, so a server stopped mid-audit leaves "running" -# behind — the one thing on disk that says the files beside it are incomplete. -# Its shape is `schemas/mcp_audit.schema.json`. +# What any server knows about the last audit an MCP server started in a case +# folder. The job writes it "running" before the pipeline touches anything, +# refreshes it every HEARTBEAT_SECONDS while it runs, and rewrites it with the +# outcome when it ends. So a "running" record is measured, not assumed: fresh, +# and an audit is live in some server process; stale, and the server writing it +# stopped mid-run. Its shape is `schemas/mcp_audit.schema.json`. RECORD = "mcp_audit.json" RECORD_SCHEMA = "mcp_audit/1" _RECORD_STATES = ("running", "finished", "failed") +HEARTBEAT_SECONDS = 15.0 +# four missed refreshes: a laptop that slept longer reads its own audit as +# stopped, which refuses to read the case — the safe side +STALE_AFTER_SECONDS = 60.0 @dataclass @@ -660,7 +668,7 @@ def _write_record(job: _Job) -> None: payload = { "schema": RECORD_SCHEMA, "state": job.state, - "manuscript": job.manuscript.name, # a name, as refs_manifest.json keeps it + "manuscript": str(job.manuscript), # the same absolute path audit_status gives "started": job.started, "ended": job.ended, "error": job.error, @@ -673,14 +681,22 @@ def _write_record(job: _Job) -> None: def _read_record(case: Path) -> dict | None: """The last MCP audit's record, or None when no MCP server audited here — and - a refusal, never a guess, when there is one and it cannot be read.""" + a refusal, never a guess, when there is one this server cannot vouch for.""" path = case / RECORD if not path.is_file(): return None try: record = json.loads(path.read_text(encoding="utf-8")) - if not isinstance(record, dict) or record.get("state") not in _RECORD_STATES: - raise ValueError("no state this server knows") + # every field a decision rests on, checked: "false" is not False, and a + # later schema's record may mean something this one does not + if not ( + isinstance(record, dict) + and record.get("schema") == RECORD_SCHEMA + and record.get("state") in _RECORD_STATES + and isinstance(record.get("scout"), bool) + and isinstance(record.get("started"), str) + ): + raise ValueError(f"not a {RECORD_SCHEMA} record this server can read") except (OSError, ValueError) as e: raise ToolError( f"{path} is unreadable ({type(e).__name__}: {str(e)[:200]}), so whether the last " @@ -690,12 +706,34 @@ def _read_record(case: Path) -> dict | None: return record +def _record_age(case: Path) -> float: + """Seconds since the record was last written or refreshed.""" + return max(0.0, time.time() - (case / RECORD).stat().st_mtime) + + +def _written_after_record(case: Path, path: Path) -> bool: + """Was `path` written after the record's last update — by a later run?""" + return path.is_file() and path.stat().st_mtime > (case / RECORD).stat().st_mtime + + +def _heartbeat(job: _Job) -> None: + """Refresh the record while the job runs: freshness is what tells another + server this audit is alive rather than abandoned.""" + path = job.case / RECORD + while not job.done.wait(HEARTBEAT_SECONDS): + try: + os.utime(path) + except OSError: + return # the final write reports what the job could not keep up + + class _Jobs: """The audits this server process started — at most one running at a time. One, because what a run touches is process-global: `cli.console`, `ask`'s per-site model record and `sys.stdin`. Two audits sharing them would - interleave their logs and name each other's models. + interleave their logs and name each other's models. Across processes the + record is the guard: a fresh "running" one is someone else's live audit. """ def __init__(self): @@ -706,48 +744,57 @@ def __init__(self): def last(self, case: Path) -> _Job | None: return self._by_case.get(case) + def refuse_live_elsewhere(self, case: Path) -> None: + """A second pipeline in a folder another process is writing is the + mixing `_guard_case` exists to prevent.""" + record = _read_record(case) + if record is not None and record["state"] == "running": + # both callers have ruled out this server's own job for this case, + # so a fresh record is another process's + age = _record_age(case) + if age < STALE_AFTER_SECONDS: + raise ToolError( + f"another server process is auditing {case} now (started " + f"{record['started']}; its record was refreshed {age:.0f} s ago). Follow " + "it from the host that started it, and start this one when it has ended." + ) + def refuse_unfinished(self, case: Path) -> None: - """Nothing on disk is read as a result unless the last audit here ended. + """Nothing on disk is read as a result unless the last audit here ended well. - This server's own job decides it when there is one; otherwise the record - an earlier server left. No record means no MCP server audited here, and - the case is read as the CLI's reports would read it. + The record on disk decides, not this server's memory of its own jobs: + a case completed since by other means must read again, here too. """ - if (job := self._by_case.get(case)) is not None: - state, started, error = job.state, job.started, job.error - elif (record := _read_record(case)) is not None: - state = "interrupted" if record["state"] == "running" else record["state"] - started, error = record.get("started"), record.get("error") or "" - else: - return - if state == "running": + running = self._running + if running is not None and running.case == case: raise ToolError( - f"an audit of {case} is running now (started {started}) — what is on disk " - "belongs to an earlier run or is half-written, so nothing is read from it " + f"an audit of {case} is running now (started {running.started}) — what is on " + "disk belongs to an earlier run or is half-written, so nothing is read from it " "until the audit ends. Call audit_status." ) - if state == "interrupted": + record = _read_record(case) + if record is None: + return # no MCP server audited here: read as the reports would + path = case / RECORD + if record["state"] == "running": + self.refuse_live_elsewhere(case) raise ToolError( - f"an audit of {case} started {started} did not finish: the server running it " - "stopped mid-run, so what is on disk may be half-written or mix two runs, and " + f"an audit of {case} started {record['started']} did not finish: its record " + f"was last refreshed {_record_age(case):.0f} s ago, so the server running it " + "stopped mid-run. What is on disk may be half-written or mix two runs, and " "none of it is read as a result. Run start_audit again — or, if the case has " - f"been completed since by other means, remove {case / RECORD} first." + f"been completed since by other means, remove {path} first." ) - if state == "failed": + if record["state"] == "failed" and not _written_after_record( + case, case / "out" / "results.json" + ): raise ToolError( - f"the last audit of {case} (started {started}) failed: {error}. What is on " - "disk may mix that run with an earlier one, so none of it is read as a " - "result — fix the cause and run start_audit again." + f"the last audit of {case} (started {record['started']}) failed: " + f"{record.get('error') or 'no reason was recorded'}. What is on disk may mix " + "that run with an earlier one, so none of it is read as a result — fix the " + "cause and run start_audit again." ) - def scanned(self, case: Path) -> bool | None: - """Did the latest MCP audit here scan the literature? None: no MCP audit here, - or one whose record does not say.""" - if (job := self._by_case.get(case)) is not None: - return job.scout - record = _read_record(case) - return None if record is None else record.get("scout") - def start(self, prepare: Callable[[], tuple[Path, Path, dict, list[str]]]) -> _Job: """Run `prepare` — every check that can refuse before anything is spent — then the pipeline on its own thread, and return without waiting for it.""" @@ -765,7 +812,7 @@ def start(self, prepare: Callable[[], tuple[Path, Path, dict, list[str]]]) -> _J # daemon: when the host stops the server, no further stage and no # further model call starts. A `claude -p` call already in flight # runs to its own end — `ask.py`'s timeout bounds it — and the record - # left saying "running" is what tells a later server the case is + # it stops refreshing is what tells a later server the case is # incomplete. threading.Thread( target=self._run, args=(job, kwargs), name="papertrace-audit", daemon=True @@ -780,6 +827,8 @@ def _run(self, job: _Job, kwargs: dict) -> None: # in a folder that is briefly untracked cli._open_case(job.case) _write_record(job) + threading.Thread(target=_heartbeat, args=(job,), name="papertrace-heartbeat", + daemon=True).start() # the process outlives one audit: a model an earlier audit # recorded must not be reported as this one's judge forget_models() @@ -804,8 +853,8 @@ def _run(self, job: _Job, kwargs: dict) -> None: _write_record(job) except OSError as e: # this server still knows the outcome; a later one reads the - # "running" left behind as an interrupted audit, which refuses — - # the safe side + # "running" left behind, unrefreshed, as a stopped audit — which + # refuses, the safe side job.log.append(f"the audit record could not be updated: {e}") with self._lock: if self._running is job: @@ -818,27 +867,43 @@ def _run(self, job: _Job, kwargs: dict) -> None: "finished": "call audit_summary for the result — its counts travel with the caveats that " "qualify them", "failed": "read `error` and the log, fix what they name, and call start_audit again", - "interrupted": "the server running this audit stopped before it ended — call start_audit " - "to run it again", - "not_started": "no audit of this case was started by an MCP server — audit_summary reads " - "whatever a CLI run left on disk", + "not_started": "no MCP server has audited this case — audit_summary reads whatever a CLI " + "run left on disk", } +# the same states read from a record another server process left — whose log +# is not here, so nothing may send the host to read one +_NEXT_RECORDED = { + "running": "another server process is running this audit — its log is there, not here; " + "call audit_status again, with wait_seconds up to 50, until it has ended", + "finished": "call audit_summary for the result — the audit was recorded by another server " + "process, whose log was not kept", + "failed": "read `error`, fix what it names, and call start_audit again — the audit was " + "recorded by another server process, whose log was not kept", + "interrupted": "the server running this audit stopped before it ended, and its log was not " + "kept — call start_audit to run it again", +} + + +def _recorded_state(case: Path, record: dict) -> str: + """A record's state as measured: `running` only while it is being refreshed.""" + if record["state"] != "running": + return record["state"] + return "running" if _record_age(case) < STALE_AFTER_SECONDS else "interrupted" def _status(case: Path, job: _Job | None, log_lines: int, record: dict | None = None) -> AuditStatus: - """The job's state — this server's own, else the record an earlier one left. + """The job's state — this server's own, else the record another one left. `log` and `log_lines_total` are null, not empty, when no log exists here: an empty log reads as an audit that printed nothing. """ if job is None and record is not None: - state = "interrupted" if record["state"] == "running" else record["state"] + state = _recorded_state(case, record) return { "case": str(case), "state": state, "manuscript": record.get("manuscript"), "started": record.get("started"), "ended": record.get("ended"), "error": record.get("error") or "", "log": None, "log_lines_total": None, - "next": f"{_NEXT[state]} — recorded by an earlier server process, whose log " - "was not kept", + "next": _NEXT_RECORDED[state], } if job is None: return { @@ -1085,7 +1150,11 @@ def list_gaps(case: CaseArg) -> Gaps: for u in results.uncited ], "coverage": coverage_headline(coverage) or None, - "unreached_labels": list(coverage.get("missing") or []) if audited else None, + # the label lists come from clean.md: without it they are empty + # because nothing read them, while the occurrences may still count + "unreached_labels": ( + list(coverage.get("missing") or []) if coverage.get("labels_in_text") else None + ), "unreached_citations": ( [ {"label": o["label"], "status": o["status"], @@ -1117,7 +1186,11 @@ def get_scout(case: CaseArg) -> ScoutOut: f"no literature scan at {path} — the audit ran without the scout, or the " "scout stage has not run (`papertrace scout -c `)" ) - if jobs.scanned(root) is False: + record = _read_record(root) + if ( + record is not None and record["state"] == "finished" and record["scout"] is False + and not _written_after_record(root, path) + ): raise ToolError( f"the latest audit of {root} ran without the scout, so {path} belongs to an " "earlier run — its registers may be anchored to another DOI or another " @@ -1207,6 +1280,7 @@ def prepare() -> tuple[Path, Path, dict, list[str]]: try: with _console_into(said): root = (_case_dir(case) if case else cli.default_case(pdf)).resolve() + jobs.refuse_live_elsewhere(root) address = cli._email(email) cli._guard_case(root, pdf) # one case folder per paper except typer.Exit: @@ -1238,9 +1312,20 @@ def audit_status( audit_summary; this never repeats its counts without their caveats.""" root = _case_dir(case) job = jobs.last(root) - if job is not None and wait_seconds: - job.done.wait(wait_seconds) - return _status(root, job, log_lines, None if job is not None else _read_record(root)) + if job is not None: + if wait_seconds: + job.done.wait(wait_seconds) + return _status(root, job, log_lines) + # another process's audit: no event to wait on, so watch its record + deadline = time.monotonic() + wait_seconds + record = _read_record(root) + while ( + record is not None and _recorded_state(root, record) == "running" + and time.monotonic() < deadline + ): + time.sleep(min(1.0, max(0.0, deadline - time.monotonic()))) + record = _read_record(root) + return _status(root, None, log_lines, record) return server diff --git a/tests/test_mcp_server.py b/tests/test_mcp_server.py index eb23831..f3e4cdb 100644 --- a/tests/test_mcp_server.py +++ b/tests/test_mcp_server.py @@ -39,6 +39,8 @@ from papertrace import check as check_mod # noqa: E402 from papertrace import cli, mcp_server # noqa: E402 from papertrace.disclosures import ( # noqa: E402 + ANCHOR_LOCATED_TOKEN, + ANCHOR_NOT_LOCATED_TOKEN, SCOPE_TOKEN, claim_disclosures, judgement_disclosures, @@ -1077,6 +1079,7 @@ def test_without_the_sdk_papertrace_mcp_says_how_to_install_it(monkeypatch, caps for name in [n for n in sys.modules if n == "mcp" or n.startswith("mcp.")]: monkeypatch.setitem(sys.modules, name, None) monkeypatch.delitem(sys.modules, "papertrace.mcp_server", raising=False) + monkeypatch.setattr("importlib.metadata.version", _no_distribution) with pytest.raises(typer.Exit) as stop: cli.mcp_command() @@ -1087,6 +1090,12 @@ def test_without_the_sdk_papertrace_mcp_says_how_to_install_it(monkeypatch, caps assert "pip install 'papertrace[mcp]'" in err +def _no_distribution(name): + from importlib.metadata import PackageNotFoundError + + raise PackageNotFoundError(name) + + # -------------------------------------------------------------------------- # review findings — each a way the server could present a case as a result # it is not, or a verdict without what qualifies it @@ -1097,6 +1106,29 @@ def _record(case: Path) -> dict: return json.loads((case / "mcp_audit.json").read_text()) +def _left_record(case: Path, state: str, *, age: float = 0, **fields) -> Path: + """A record as another server process left it, last refreshed `age` s ago.""" + import time + + path = case / "mcp_audit.json" + path.write_text(json.dumps({ + "schema": "mcp_audit/1", "state": state, "manuscript": "/papers/paper.pdf", + "started": "2026-09-26T10:00:00+00:00", "ended": None, "error": "", "scout": True, + **fields, + })) + then = time.time() - age + os.utime(path, (then, then)) + return path + + +def _age_file(path: Path, seconds: float) -> None: + """Move a file's mtime by `seconds` — positive is later, negative earlier.""" + import time + + t = time.time() + seconds + os.utime(path, (t, t)) + + def test_after_a_failed_audit_nothing_on_disk_is_served_as_its_result( ready, monkeypatch, tmp_path ): @@ -1125,10 +1157,7 @@ def test_an_audit_its_server_never_finished_is_refused_and_reported_interrupted( is on disk, and the only thing that says it is incomplete is the record the job wrote when it started — read here by a server that never ran it.""" case = _case(tmp_path) - (case / "mcp_audit.json").write_text(json.dumps({ - "schema": "mcp_audit/1", "state": "running", "manuscript": "paper.pdf", - "started": "2026-09-26T10:00:00+00:00", "ended": None, "error": "", "scout": True, - })) + _left_record(case, "running", age=120) server = mcp_server.build_server() status = _json(_call("audit_status", {"case": str(case)}, server)) @@ -1252,6 +1281,7 @@ def test_an_sdk_too_old_for_the_server_gets_the_install_line_too(monkeypatch, ca monkeypatch.setitem(sys.modules, "mcp", old) monkeypatch.setitem(sys.modules, "mcp.server", types.ModuleType("mcp.server")) monkeypatch.delitem(sys.modules, "papertrace.mcp_server", raising=False) + monkeypatch.setattr("importlib.metadata.version", lambda name: "1.28.1") with pytest.raises(typer.Exit) as stop: cli.mcp_command() @@ -1284,3 +1314,222 @@ def test_every_tool_output_matches_its_schema_in_schemas(tmp_path): jsonschema.validate(out, {**published, "$ref": f"#/$defs/{tool}"}) declared = set(published["$defs"][tool]["properties"]) assert declared == set(getattr(mcp_server, shape).__annotations__), tool + + +# -------------------------------------------------------------------------- +# second review pass — the record says only what it measured +# -------------------------------------------------------------------------- + + +def test_a_named_sources_crop_carries_only_its_own_anchor_state(tmp_path): + """Claim 1's headline source (beta-2021) was not boxed; alpha-2020 was. + alpha's crop captioned with beta's anchor state would put two + contradictory statements under one picture.""" + case = _case(tmp_path) + + caption = _text(_call("get_evidence", {"case": str(case), "claim_id": 1, + "source": "alpha-2020"})) + + assert ANCHOR_LOCATED_TOKEN in caption + assert ANCHOR_NOT_LOCATED_TOKEN not in caption + + +def test_an_audit_another_server_is_running_is_neither_read_nor_restarted( + ready, monkeypatch, tmp_path +): + """Two hosts can each launch their own server. A record refreshed moments + ago is a live audit in another process — not an interrupted one — and a + second pipeline in the same folder would be the mixing `_guard_case` + exists to prevent.""" + case = _case(tmp_path) + RefManifest(manuscript="paper.pdf").to_json(case / "refs_manifest.json") + _left_record(case, "running", age=1) + calls = _fake_pipeline(monkeypatch) + server = mcp_server.build_server() + + status = _json(_call("audit_status", {"case": str(case)}, server)) + summary = _call("audit_summary", {"case": str(case)}, server) + start = _call("start_audit", {"manuscript": str(_paper(tmp_path)), "case": str(case)}, server) + + assert status["state"] == "running" and status["log"] is None + assert summary.is_error and "another server process" in _text(summary) + assert start.is_error and "another server process" in _text(start) + assert calls == [] + + +def test_an_interrupted_audit_can_be_started_again(ready, monkeypatch, tmp_path): + case = _case(tmp_path) + RefManifest(manuscript="paper.pdf").to_json(case / "refs_manifest.json") + _left_record(case, "running", age=600) + _fake_pipeline(monkeypatch) + server = mcp_server.build_server() + + status = _await(server, _start(server, manuscript=_paper(tmp_path), case=case)["case"]) + + assert status["state"] == "finished" + assert _record(case)["state"] == "finished" + + +def test_the_job_keeps_its_record_fresh_while_it_runs(ready, monkeypatch, tmp_path): + """Freshness is the measurement that tells a live audit from an abandoned + one, so the job refreshes its record for as long as it runs.""" + import time + + monkeypatch.setattr(mcp_server, "HEARTBEAT_SECONDS", 0.05) + seen = {} + + def body(kw): + record = Path(kw["case"]) / "mcp_audit.json" + _age_file(record, -600) + time.sleep(0.5) + seen["age"] = time.time() - record.stat().st_mtime + + _fake_pipeline(monkeypatch, body) + server = mcp_server.build_server() + + _await(server, _start(server, manuscript=_paper(tmp_path))["case"]) + + assert seen["age"] < 5, seen + + +def test_a_label_audit_that_never_ran_is_null_even_when_occurrences_were_counted(tmp_path): + """Without clean.md the label-level lists are empty because nothing read + them, while the occurrence audit ran from the source map. `[]` would say + every cited label was reached.""" + occurrences_only = { + "labels_in_text": [], "covered": [], "missing": [], + "occurrences": {**OCCURRENCE_COVERAGE["occurrences"]}, + } + case = _case(tmp_path, results=_results(coverage=occurrences_only)) + + gaps = _json(_call("list_gaps", {"case": str(case)})) + + assert gaps["unreached_labels"] is None + assert [o["label"] for o in gaps["unreached_citations"]] == ["8", "4"] + + +def test_the_status_names_the_manuscript_the_same_way_before_and_after_a_restart( + ready, monkeypatch, tmp_path +): + """A published field has one meaning: the audited PDF's absolute path.""" + _fake_pipeline(monkeypatch) + pdf = _paper(tmp_path) + server = mcp_server.build_server() + running = _start(server, manuscript=pdf) + finished = _await(server, running["case"]) + + later = _json(_call("audit_status", {"case": running["case"]}, mcp_server.build_server())) + + assert running["manuscript"] == finished["manuscript"] == later["manuscript"] + assert later["manuscript"] == str(pdf.resolve()) + + +def test_a_later_run_that_finished_the_judging_supersedes_a_failed_audit( + ready, monkeypatch, tmp_path +): + """The failure was the last thing an MCP server saw — not the last thing + that happened. A results.json written after it is a later run's judging + of the manifest now on disk, so the case reads again, in this server too.""" + def body(kw): + raise RuntimeError("claude -p failed: not logged in") + + case = _case(tmp_path) + RefManifest(manuscript="paper.pdf").to_json(case / "refs_manifest.json") + _fake_pipeline(monkeypatch, body) + server = mcp_server.build_server() + _await(server, _start(server, manuscript=_paper(tmp_path), case=case)["case"]) + assert _call("audit_summary", {"case": str(case)}, server).is_error + + _results().to_json(case / "out" / "results.json") # e.g. `papertrace check`, later + _age_file(case / "out" / "results.json", +5) + + assert not _call("audit_summary", {"case": str(case)}, server).is_error + + +def test_a_scan_written_after_an_audit_without_the_scout_is_served(tmp_path): + case = _case(tmp_path, scout=ScoutResults(paper_title="A paper", paper_identity="confirmed")) + _left_record(case, "finished", age=60, ended="2026-09-26T10:05:00+00:00", scout=False) + _age_file(case / "out" / "scout.json", 0) # `papertrace scout`, after that audit + + result = _call("get_scout", {"case": str(case)}) + + assert not result.is_error, _text(result) + + +def test_a_broken_import_inside_a_current_sdk_keeps_its_traceback(monkeypatch): + """With a 2.x SDK installed, an import that still fails is a real fault. The + install line would contradict itself ("needs version 2 — found 2.x") and + hide the one thing that says what broke.""" + for name in [n for n in sys.modules if n == "mcp" or n.startswith("mcp.")]: + monkeypatch.delitem(sys.modules, name) + broken = types.ModuleType("mcp") + broken.__path__ = [] + monkeypatch.setitem(sys.modules, "mcp", broken) + monkeypatch.setitem(sys.modules, "mcp.server", types.ModuleType("mcp.server")) + monkeypatch.delitem(sys.modules, "papertrace.mcp_server", raising=False) + monkeypatch.setattr("importlib.metadata.version", lambda name: "2.3.0") + + with pytest.raises(ImportError, match="MCPServer"): + cli.mcp_command() + + +def test_a_recorded_failure_does_not_send_the_host_to_a_log_that_was_not_kept(tmp_path): + case = _case(tmp_path) + _left_record(case, "failed", age=60, ended="2026-09-26T10:05:00+00:00", + error="claude -p failed: not logged in") + + status = _json(_call("audit_status", {"case": str(case)})) + + assert status["state"] == "failed" and status["log"] is None + assert "the log" not in status["next"] + assert "not kept" in status["next"] + + +@pytest.mark.parametrize("field", [{"scout": "false"}, {"schema": "mcp_audit/2"}], + ids=["scout-not-a-bool", "a-later-schema"]) +def test_a_record_this_server_cannot_vouch_for_is_refused(tmp_path, field): + """A record read under the wrong rules is a guess. "false" is not False, + and a v2 record may mean something v1 does not.""" + case = _case(tmp_path) + _left_record(case, "finished", age=60, ended="2026-09-26T10:05:00+00:00", **field) + + result = _call("audit_summary", {"case": str(case)}) + + assert result.is_error and "unreadable" in _text(result) + + +def test_every_status_a_record_can_produce_matches_the_schema(tmp_path): + """The status schema describes what the server returns from its own job AND + from a record another server left — not only the shape one test happened + to build.""" + jsonschema = pytest.importorskip("jsonschema") + published = json.loads((ROOT / "schemas" / "mcp_tools.schema.json").read_text()) + cases = { + "not_started": {}, + "running": {"state": "running", "age": 1}, + "interrupted": {"state": "running", "age": 600}, + "finished": {"state": "finished", "age": 60, "ended": "2026-09-26T10:05:00+00:00"}, + "failed": {"state": "failed", "age": 60, "ended": "2026-09-26T10:05:00+00:00", + "error": "RuntimeError: boom"}, + } + + for expected, record in cases.items(): + case = _case(tmp_path / expected) + if record: + _left_record(case, record.pop("state"), **record) + status = _json(_call("audit_status", {"case": str(case)})) + assert status["state"] == expected + jsonschema.validate(status, {**published, "$ref": "#/$defs/audit_status"}) + + +def test_the_published_timings_are_the_ones_the_server_uses(): + """The schemas state the refresh and staleness windows in prose, because a + reader of a record has to know them — so a changed constant must not leave + the contract quoting the old one.""" + record = json.loads((ROOT / "schemas" / "mcp_audit.schema.json").read_text()) + tools = json.loads((ROOT / "schemas" / "mcp_tools.schema.json").read_text()) + state = tools["$defs"]["audit_status"]["properties"]["state"]["description"] + + for prose in (record["description"], state): + assert f"every {mcp_server.HEARTBEAT_SECONDS:g} s" in prose + assert f"{mcp_server.STALE_AFTER_SECONDS:g} s" in prose From cdfa13e2206ba8f94cb0b70d049b02dd135fc180 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 26 Sep 2026 13:09:45 +0000 Subject: [PATCH 12/13] =?UTF-8?q?docs:=20the=20second=20review=20pass=20?= =?UTF-8?q?=E2=80=94=20a=20record=20measured=20by=20its=20heartbeat?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The spec's eleventh rule now says how "running" is measured and that the record on disk, not a server's memory, decides. Its omissions lose "two server processes auditing one case are not detected", which stopped being true, and gain the heartbeat's real limit instead: a machine that slept past the 60 s window reads as stopped — its case refused, the safe side — while a second audit started inside that window is not. README, CHANGELOG and CLAUDE.md say the same; the plan records the pass. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01J6JtZg47dHEKHDEfAJ5Kpf --- CHANGELOG.md | 5 +++- CLAUDE.md | 12 ++++++---- README.md | 3 ++- .../plans/2026-09-26-mcp-server.md | 19 +++++++++++++++ .../specs/2026-09-26-mcp-server-design.md | 24 ++++++++++++------- 5 files changed, 49 insertions(+), 14 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index aae2cd4..ca8f8ea 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -27,7 +27,10 @@ local PDF path is served, and no evidence image from outside `/out/`. A verdict read on its own — `get_claim`, the evidence caption — carries the run-level caveats too. Each audit leaves `/mcp_audit.json`: a case whose last MCP audit failed or never finished is refused rather than read, because -its files may mix two runs. The tools' outputs and that record are published in +its files may mix two runs. The job refreshes the record every 15 s, so a +second host's server can tell a live audit from an abandoned one and will not +start another in the same folder. The tools' outputs and that record are +published in `schemas/mcp_tools.schema.json` and `schemas/mcp_audit.schema.json`. An audit takes minutes and a host built on the TypeScript SDK gives a request diff --git a/CLAUDE.md b/CLAUDE.md index 1e39bb5..e3d161a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -353,10 +353,14 @@ ingest → extract → refs → scout → check → highlight → report do not offer MCP elicitation in its place: a host may be a model), and `ask.forget_models()` runs first. Those three are process-global, hence **one audit at a time**. The job writes `/mcp_audit.json` (`running`, - then the outcome), and read tools refuse a case whose last MCP audit is - running, failed, or never finished (a `running` record no live job owns is - `interrupted`): a failed rerun leaves one run's manifest beside another's - verdicts. A verdict read alone — `get_claim`, the evidence caption — carries + refreshed every `HEARTBEAT_SECONDS`, then the outcome), and read tools refuse + a case whose last MCP audit is running, failed, or never finished: a failed + rerun leaves one run's manifest beside another's verdicts. `running` is a + measurement — fresh is another process's live audit (refused, never started + over), unrefreshed for `STALE_AFTER_SECONDS` is `interrupted` — and never + say a cause the record's age does not show. The record on disk decides, not + a server's memory: a later `results.json`/`scout.json` supersedes a failure + or a run without the scout. A verdict read alone — `get_claim`, the evidence caption — carries the run-level disclosures as well as its own. Outputs are published in `schemas/mcp_tools.schema.json`, each definition's keys held equal to its `TypedDict`'s by a test. **Any new process-global state in the pipeline diff --git a/README.md b/README.md index deab5df..d8fcea6 100644 --- a/README.md +++ b/README.md @@ -345,7 +345,8 @@ manuscript, a `claude` the server cannot find, no email, or a case folder holding another paper is refused before anything is spent. One audit runs at a time per server. A case folder is not read while its audit runs, nor after one that failed or never finished — its files may mix two runs — which the server -knows from the `mcp_audit.json` each audit leaves in the folder. Nobody is at an +knows from the `mcp_audit.json` each audit leaves and keeps fresh while it runs; +a second host's server will not start an audit in a folder another is writing. Nobody is at an MCP call to answer a question, so a reference label whose readings disagree is withheld, never settled — that takes you, at a terminal. Nothing but the protocol reaches stdout; what the pipeline prints diff --git a/docs/superpowers/plans/2026-09-26-mcp-server.md b/docs/superpowers/plans/2026-09-26-mcp-server.md index b7b248f..eb8b16a 100644 --- a/docs/superpowers/plans/2026-09-26-mcp-server.md +++ b/docs/superpowers/plans/2026-09-26-mcp-server.md @@ -155,3 +155,22 @@ straight through. with validation tests. - [x] `disclosures.coverage_audited` and `cli.SCOUT_CAVEAT`: one rule and one sentence each, not two copies. + +## Task 8 — the second review pass + +The review round's own diff, reviewed again: ten findings, all of which held +up, each fixed test-first. + +- [x] A `running` record is measured by a heartbeat: fresh is another + process's live audit (refused; never started over), stale is + `interrupted`. `audit_status` waits on another process's record too. +- [x] The disk record decides; later `results.json` / `scout.json` supersede + a failure or a run without the scout. +- [x] A named source's crop carries only its own anchor state. +- [x] `unreached_labels` is null unless the label audit read `clean.md`. +- [x] `manuscript` is the absolute path, from this server and from a record. +- [x] Recorded states never send the host to a log that was not kept. +- [x] The record is validated before anything is decided on it. +- [x] A 2.x SDK's import failure keeps its traceback. +- [x] The schema describes record-derived states, validated for each; the + prose timings are held to the constants. diff --git a/docs/superpowers/specs/2026-09-26-mcp-server-design.md b/docs/superpowers/specs/2026-09-26-mcp-server-design.md index df667ce..15a539d 100644 --- a/docs/superpowers/specs/2026-09-26-mcp-server-design.md +++ b/docs/superpowers/specs/2026-09-26-mcp-server-design.md @@ -95,11 +95,15 @@ pipeline wrote. `start_audit` is the only tool that spends — model calls throu the manifest and die before `check`, leaving one run's references beside another's verdicts. The job writes `/mcp_audit.json` — `running` before the pipeline starts, the outcome when it ends — and read tools - refuse a case whose last MCP audit failed or never finished. A server - stopped mid-audit leaves `running` behind, which a later server reports as - `interrupted`. An optional stage's file the latest audit did not write — a - `scout.json` from before a run without the scout — is refused as an - earlier run's. + refuse a case whose last MCP audit failed or never finished. The job + refreshes the record every 15 s, so `running` is measured rather than + assumed: fresh, the audit is live in some server process — perhaps another + host's own `papertrace mcp` — and the folder is neither read nor audited + again; unrefreshed for 60 s, the server stopped mid-run and the audit is + `interrupted`. The record on disk decides, not a server's memory: a failed + audit is superseded by a `results.json` written after it, and one run + without the scout by a `scout.json` written after it, so a case completed + since by other means reads again. ## Long runs: a job, not a blocking call @@ -118,7 +122,9 @@ is not another paper's — and then returns at once. `audit_status` reports **One audit at a time per server process.** The pipeline's console, `ask`'s per-site model record and `sys.stdin` are process-global; two audits sharing them would interleave logs and misattribute models. A second `start_audit` is -refused, naming the audit in progress. +refused, naming the audit in progress. Across processes the record is the +guard: a fresh `running` record is another server's live audit, and +`start_audit` refuses that folder too. **Read tools refuse a case whose audit is running, failed or never finished.** The files on disk then belong to an earlier run, are half-written, or mix the @@ -169,8 +175,10 @@ what rule 2 forbids, so a finished job points at `audit_summary`. the record left saying `running` marks the case incomplete. - A job's log is not persisted; its outcome is, in `mcp_audit.json`, and an earlier server's audit reports `log: null` rather than an empty log. -- Two server processes auditing one case at once are not detected: one audit - at a time is a per-process rule, and the CLI has no such guard either. +- Liveness across processes is a heartbeat window, not a lock. A server whose + refreshes stop for 60 s — a machine that slept — reads as stopped: its case + is refused rather than read, the safe side, but a second audit started in + that window is not refused. The CLI takes no part in the record at all. - A CLI run that fails midway writes no record, so the server reads that case as the reports would. From 26b2d5409f4753f091cb33ac67ff9f4e52e023ff Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 26 Sep 2026 15:37:05 +0000 Subject: [PATCH 13/13] docs: which apps can run the MCP server, and a no-clone install MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The README named four hosts and installed from a clone. It now lists the apps that can start a local MCP server (Claude Desktop, Claude Code, Cursor, VS Code, Windsurf, Gemini CLI, Codex CLI) with where each takes its configuration, and says plainly that ChatGPT and Claude in a browser cannot: they reach MCP servers over the internet only. Whichever app drives the tools, every claim is judged through `claude -p`, so Claude Code has to be installed and signed in — said once, beside the list, rather than left for `start_audit`'s refusal to reveal. Install is `uv tool install` from the GitHub archive: no Git, and uv fetches a Python. Verified in a sandbox: the install, the executable launched by absolute path with a minimal environment (nine tools; start_audit refused while PATH did not reach `claude`, and refused on the email next once it did — nothing spent), and Claude Code's `claude mcp list` reporting it connected. That last is what "tested" now claims; the other apps' configurations follow their docs and have not each been run. Also: `claude mcp add` takes `--env` as variadic, so a name straight after it is read as another KEY=value (reproduced); PowerShell's `where` is Where-Object, so `where.exe`; a Windows path in JSON doubles each `\`. Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01J6JtZg47dHEKHDEfAJ5Kpf --- CHANGELOG.md | 9 ++++ README.md | 133 ++++++++++++++++++++++++++++++++++++++------------- 2 files changed, 110 insertions(+), 32 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ca8f8ea..f3b9eb3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -48,6 +48,15 @@ decision record: `docs/superpowers/specs/2026-09-26-mcp-server-design.md`, `docs/adr/0004-mcp-server.md`. +The README's *From an MCP host* names the apps that can start it — Claude +Desktop, Claude Code, Cursor, VS Code, Windsurf, Gemini CLI, Codex CLI — and +the ones that cannot: ChatGPT and Claude in a browser reach MCP servers over +the internet only. Whichever app drives the tools, the audit is judged through +`claude -p`, so Claude Code must be installed and signed in. The server +installs from GitHub with `uv tool install`, no clone needed. Claude Code has +been seen connecting to that install; the other apps' configurations follow +their documentation and have not each been run. + ### Changed — `[full]` and `[dev]` carry the MCP SDK `[full]`, the quick start's everything-install, now includes the SDK, so diff --git a/README.md b/README.md index d8fcea6..2a01acb 100644 --- a/README.md +++ b/README.md @@ -79,10 +79,11 @@ Four ways in, all producing the same case folder and reports: interviews you (the paper, your PDFs, your journal's reviewer form as screenshots), retrieves and checks with evidence as it goes, drafts the findings, and ends by writing the viewer. -- **From an MCP host** — Claude Desktop, Cursor, VS Code or Claude Code, with - `papertrace mcp` as a server: start an audit, follow it, and read every - verdict with its caveats and its evidence crops. See - [From an MCP host](#from-an-mcp-host). +- **From an AI app** — Claude Desktop, Claude Code, Cursor, VS Code, Windsurf, + Gemini CLI or Codex CLI can run `papertrace mcp` as an MCP server: start an + audit, follow it, and read every verdict with its caveats and its evidence + crops. Not ChatGPT, nor Claude in a browser — neither can start a program on + your computer. See [From an MCP host](#from-an-mcp-host).

The guided audit in a terminal: a setup check, then the questions one at a time — the paper's path, the case folder, cited PDFs already to hand, the paper's own supplements, a DOI found on the first page, a contact email, whether to write the interactive viewer, whether to limit the audit — and the cost stated as a number of model calls before asking permission to start. @@ -293,33 +294,103 @@ re-run months later can find a different set of sources. ## From an MCP host -`papertrace mcp` serves PaperTrace over stdio to any -[MCP](https://modelcontextprotocol.io) host. Install the extra, and give the -host the command by absolute path — a host starts its servers from its own -working directory, with its own and often minimal `PATH`: +`papertrace mcp` lets an AI app that speaks the +[Model Context Protocol](https://modelcontextprotocol.io) (MCP) drive +PaperTrace: start an audit, follow it, and read every verdict with its caveats +and its evidence crops. It runs on your own computer, beside your case +folders — the app starts it as a local program (MCP's *stdio* transport), and +nothing is served over the network. + +**Which apps.** Any app that can start a local MCP server: + +| App | Where PaperTrace is added | +|---|---| +| Claude Desktop | Settings → Developer → Edit Config: `claude_desktop_config.json` | +| Claude Code | `claude mcp add` — below | +| Cursor | `~/.cursor/mcp.json` | +| VS Code, GitHub Copilot in Agent mode | Command Palette → *MCP: Open User Configuration* | +| Windsurf | `~/.codeium/windsurf/mcp_config.json` | +| Gemini CLI | `~/.gemini/settings.json` | +| OpenAI Codex CLI | `codex mcp add`, or `~/.codex/config.toml` | + +**ChatGPT, and Claude in a web browser, cannot run it:** they reach MCP +servers over the internet only, and cannot start a program on your computer. +Use the Claude Desktop app, not claude.ai in a browser. + +Two things hold whichever app you use: + +- **The audit is judged by Claude, through Claude Code on the same computer.** + `start_audit` runs the pipeline `papertrace run` runs, and its model calls + are `claude -p` — so Claude Code must be installed and signed in, which needs + a paid Claude plan, even when the app driving the tools is Cursor, Gemini CLI + or Codex. That app's own model chooses the tools and relays what they return; + it does not judge the claims. Reading an audit that already exists needs no + Claude Code at all. +- **What has been tested** is the server against the MCP Python SDK's own + client, in memory and over a real stdio pipe, and Claude Code connecting to + an install made as below. The other configurations follow each app's + documentation; they have not each been run in the app itself. + +**Install.** [uv](https://docs.astral.sh/uv/) installs PaperTrace into an +environment of its own, and fetches a suitable Python if you have none — no +clone, no Git: + +```bash +uv tool install --python 3.12 "papertrace[mcp] @ https://github.com/defraction0/PaperTrace/archive/refs/heads/main.zip" +uv tool update-shell # puts `papertrace` on your PATH; open a new terminal after it +which papertrace # the full path the app needs — on Windows: where.exe papertrace +``` + +The download is large, most of it PyTorch for the layout-aware reader — 6 GB +installed on Linux, where PyTorch brings its GPU libraries. From a checkout, +`pip install -e ".[mcp]"` does the same. Give the app the **full path** +printed above: an app starts its servers from a working directory of its own, +with its own and often minimal `PATH`. + +**Claude Code**, for every project on this computer: ```bash -pip install -e ".[mcp]" -which papertrace # the path the host needs -claude mcp add papertrace -- /absolute/path/to/papertrace mcp # Claude Code +claude mcp add --env PAPERTRACE_EMAIL=you@example.org --transport stdio --scope user \ + papertrace -- /full/path/to/papertrace mcp ``` -Claude Desktop reads it from `claude_desktop_config.json`, and Cursor from -`.cursor/mcp.json` in the same shape; VS Code takes the same command as a -`"servers"` entry with `"type": "stdio"` in `.vscode/mcp.json`: +Keep another option between `--env` and the name — Claude Code reads a name +straight after `--env` as one more `KEY=value`. `claude mcp list` then shows +whether it connected. + +**Claude Desktop, Cursor, Windsurf and Gemini CLI** share one shape: ```json { "mcpServers": { "papertrace": { - "command": "/absolute/path/to/papertrace", + "command": "/full/path/to/papertrace", "args": ["mcp"], - "env": {"PAPERTRACE_EMAIL": "you@example.org"} + "env": { + "PAPERTRACE_EMAIL": "you@example.org", + "PATH": "/Users/you/.local/bin:/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin" + } } } } ``` +`PATH` is there for `claude`. On a Mac an app opened from the Dock may not see +your terminal's `PATH`, and without `claude` `start_audit` refuses; the Claude +Code installer puts it in `~/.local/bin`, and `which claude` names the folder +if yours is elsewhere. Windows apps see the `PATH` a new terminal sees, so +there `PATH` can usually be left out — and the command is +`"C:\\Users\\you\\.local\\bin\\papertrace.exe"`, each `\` written twice, as +JSON requires. On Windows, some Claude Desktop installs read their +configuration from +`%LOCALAPPDATA%\Packages\Claude_…\LocalCache\Roaming\Claude\` rather than +the file *Edit Config* opens: if PaperTrace is missing under **+ → Connectors** +after a full restart, add the same entry there. + +**VS Code** takes the same command as a `"servers"` entry with +`"type": "stdio"`; **Codex CLI** as +`codex mcp add papertrace --env PAPERTRACE_EMAIL=you@example.org -- /full/path/to/papertrace mcp`. + | Tool | What it does | |---|---| | `start_audit` | runs the `papertrace run` pipeline in the background and returns at once | @@ -333,24 +404,22 @@ Claude Desktop reads it from `claude_desktop_config.json`, and Cursor from The read tools work on any case folder, made over MCP or by the CLI, and change nothing. Every verdict arrives with the caveats the reports print beside it, and a limited audit says so first and last. An audit takes minutes, -longer than many hosts wait for a single request, which is why it runs as a -job that `audit_status` follows. +longer than many apps wait for a single request, which is why it runs as a job +that `audit_status` follows. `start_audit` spends what `papertrace run` spends: `claude -p` calls on this -machine's Claude login, and the open-access services. So `claude` has to be on -the server's `PATH` — if the host's is too minimal, add its directory in -`env` — and the contact email comes from `start_audit`'s `email`, from -`PAPERTRACE_EMAIL`, or from the address `papertrace` saved. A missing -manuscript, a `claude` the server cannot find, no email, or a case folder -holding another paper is refused before anything is spent. One audit runs at a -time per server. A case folder is not read while its audit runs, nor after one -that failed or never finished — its files may mix two runs — which the server -knows from the `mcp_audit.json` each audit leaves and keeps fresh while it runs; -a second host's server will not start an audit in a folder another is writing. Nobody is at an -MCP call to answer a question, so a reference label -whose readings disagree is withheld, never settled — that takes you, at a -terminal. Nothing but the protocol reaches stdout; what the pipeline prints -comes back in `audit_status`'s log. +computer's Claude login, and the open-access services. The contact email comes +from `start_audit`'s `email`, from `PAPERTRACE_EMAIL`, or from the address +`papertrace` saved. A missing manuscript, a `claude` the server cannot find, no +email, or a case folder holding another paper is refused before anything is +spent. One audit runs at a time per server. A case folder is not read while +its audit runs, nor after one that failed or never finished — its files may +mix two runs — which the server knows from the `mcp_audit.json` each audit +leaves and keeps fresh while it runs; a second app's server will not start an +audit in a folder another is writing. Nobody is at an MCP call to answer a +question, so a reference label whose readings disagree is withheld, never +settled — that takes you, at a terminal. Nothing but the protocol reaches +stdout; what the pipeline prints comes back in `audit_status`'s log. ## Tables and figures are evidence too