diff --git a/.claude/skills/diffle-review/SKILL.md b/.claude/skills/diffle-review/SKILL.md new file mode 100644 index 00000000..80e02109 --- /dev/null +++ b/.claude/skills/diffle-review/SKILL.md @@ -0,0 +1,72 @@ +--- +name: diffle-review +description: Review a diff through a running diffle: read what it shows, comment on lines, and push a report with findings that do not fit a line. Use when asked to review changes in diffle, annotate a diff, or leave review notes the reviewer sees beside the code. +--- + +# Review in diffle + +diffle is a local review server. While it runs for a repository, everything you do goes through its HTTP API; the reviewer sees comments inline in the diff and reports in the **Reports** view (`gr`), and can hand your comments to GitHub or copy them as a prompt. + +## Find the server + +```bash +cat "$(git rev-parse --absolute-git-dir)/diffle/server.json" # {"url":"http://127.0.0.1:4966/","pid":…} +``` + +No file, or no answer at the url: nobody is reviewing yet. Start one in the background and read the file again: + +```bash +diffle main...HEAD --no-open --keep-alive & # or `diffle working`, `diffle pr 27`, `diffle show ` +``` + +Every path below is relative to that url. Send JSON bodies with `content-type: application/json`. + +## Read the review + +| Request | Gives | +|---|---| +| `GET api/snapshot` | `changed` (path, status, additions, deletions, oldPath), `oldSha`, `newSha`, `review` | +| `GET api/patch` | every changed file's patch, one unified diff | +| `GET api/patch?path=` | one file's patch | +| `GET api/file?path=&rev=new` | a whole file on the new side (`rev=old` for the old one) | +| `GET api/threads?state=open` | comments already there; do not repeat one | + +Line numbers you quote are the new side's, as the file reads after the change, unless you comment on a removed line (`side: "old"`, old-side numbers). + +## Comment on lines + +`POST api/threads` takes one comment or an array. A comment lands under its last line; the reviewer sees it where the code is. + +```json +[ + { "path": "src/app.py", "startLine": 42, "endLine": 44, "body": "The refund path skips validation here." }, + { "path": "src/app.py", "startLine": 50, "body": "```suggestion\nreturn total.quantize(CENT)\n```" }, + { "path": "docs/guide.md", "body": "Whole-file note: this page still names the old command." } +] +``` + +- `startLine` alone comments one line; without it the comment is on the file as a whole. +- A ```` ```suggestion ```` fence proposes a replacement for the commented lines; it becomes a suggestion on GitHub. +- The reply is the created threads. A line the diff does not show is refused with a message; move the comment to a shown line or leave it on the file. +- Keep comments short and specific. One finding per comment; say what to change, not only what is wrong. + +## Push a report + +For findings that belong to no line, or a view of the whole change (a summary, a risk ranking, a dependency table, rendered output), push a report. It shows under **Reports** and goes stale, not away, when the diff moves on. + +```bash +diffle report put notes.md # the file's first heading is the title +diffle report put notes.md --anchor src/app.py:42 # the block points at a line; Enter opens it, c comments there +diffle report put review.json # a whole report: several blocks, rendered pages +``` + +Over HTTP: `PUT api/reports/` with `{ "title": "…", "blocks": [ { "kind": "markdown", "text": "…", "anchor": { "path": "src/app.py", "line": 42 } } ] }`. The id is lowercase letters, digits and dashes; putting it again replaces the report. Files a `pages` block names go up first with `PUT api/reports//files/` (raw image bytes). A 409 with `current` in the body means the diff moved on since you read it: read it again, then put again, naming `current` as `target` or leaving `target` out. + +`diffle report put --help` shows the full JSON. + +## Do this, in this order + +1. Find or start the server. Read the snapshot and the patches; read whole files where the patch is not enough. +2. Comment on the lines that need a change. Check `api/threads` first so you add to the review instead of repeating it. +3. Push one report named for what it is (`summary`, `risks`, `deps`) with what does not fit a line, each block anchored where it can be. +4. Say what you commented on and what the report holds; the reviewer takes it from there. Never submit to GitHub yourself: the reviewer does that from diffle. diff --git a/README.md b/README.md index 80d266ca..198aefce 100644 --- a/README.md +++ b/README.md @@ -202,6 +202,49 @@ returns to the graph as you left it. The graph needs the language servers, the same ones [Jump to definitions](#jump-to-definitions) uses. +## Add reports from plugins and agents + +Press `gr` for the **Reports** view: documents attached to the diff, beside it rather than in it. +They come from two places. + +A **plugin** is a program diffle runs whenever a changed file matches its globs, with copies of +both sides to read and the review as JSON on stdin. Two are built in and start on their own when +their programs are on `PATH`: `pixi` runs [`pixi-diff`](https://github.com/pavelzw/pixi-diff) and +[`pixi-diff-to-markdown`](https://github.com/pavelzw/pixi-diff-to-markdown) on a changed +`pixi.lock`, and `typst` runs a `diffle-typst` on changed `.typ` files to compare the rendered +pages. `diffle plugins` lists what a review would run and what is missing. Add your own, or +replace a built-in, in the config file (`diffle config` prints its path): + +```json +{ + "plugins": { + "lockfile": { "command": "my-lock-report $DIFFLE_OLD/$DIFFLE_FILE $DIFFLE_NEW/$DIFFLE_FILE", "when": ["*.lock"] }, + "typst": { "command": "" } + } +} +``` + +The command runs without a shell: a `|` pipes one program into the next, and `$DIFFLE_OLD`, +`$DIFFLE_NEW`, `$DIFFLE_OUT`, `$DIFFLE_ROOT` and `$DIFFLE_FILE` are substituted. A program that +writes `report.json` into `$DIFFLE_OUT` makes the whole report, rendered pages included; one that +only prints gets its stdout shown as markdown. `diffle report put --help` shows the JSON, and +`example/plugins/` holds two generators to start from: `diffle-pixi`, a shell script around +`pixi diff | pixi diff-to-markdown`, and `diffle-typst`, which compiles each side's pages. +Plugins come from your config alone, never from the reviewed tree, so a pull request cannot +bring its own. + +An **agent**, or you, pushes a report while diffle runs: + +```bash +diffle report put notes.md --anchor src/app.py:42 # one markdown block, pointing at a line +diffle report put review.json # a whole report, pages and all +``` + +Over HTTP it is `PUT api/reports/` with the same JSON; `/diffle/server.json` +names the url while diffle runs. Reports persist with the review and go stale, not away, when +the diff moves on. A block that points at a line opens the diff there with `Enter`, and `c` +comments on it, so a report's findings become ordinary comments, in the prompt and on GitHub. + ## Open from lazygit [lazygit](https://github.com/jesseduffield/lazygit) custom commands open diffle on what is diff --git a/crates/diffle/protocol.generated.json b/crates/diffle/protocol.generated.json index 6f49db93..24ad5da4 100644 --- a/crates/diffle/protocol.generated.json +++ b/crates/diffle/protocol.generated.json @@ -1681,6 +1681,34 @@ ], "description": "Lines of one side of a snapshot to link to on the forge origin points at." }, + "PluginConfig": { + "type": "object", + "properties": { + "command": { + "type": "string", + "description": "The program and its arguments, split as a shell would, without a shell: a `|` between words\npipes one program's stdout into the next, and `$DIFFLE_OLD`, `$DIFFLE_NEW`, `$DIFFLE_OUT`,\n`$DIFFLE_ROOT` and `$DIFFLE_FILE` are substituted in arguments. Empty turns a built-in plugin off." + }, + "when": { + "type": "array", + "items": { + "type": "string" + }, + "description": "Globs (picomatch syntax) of changed files that start it; a built-in plugin keeps its own when absent." + }, + "scope": { + "type": "string", + "enum": [ + "changed", + "tree" + ], + "description": "What the program gets copies of: the matched changed files, or every file the snapshot shows. Default changed." + } + }, + "required": [ + "command" + ], + "description": "A program that makes a report from a snapshot. It runs whenever a changed file matches `when`,\nwith copies of the files in DIFFLE_OLD and DIFFLE_NEW, a JSON description of the review on\nstdin, and DIFFLE_ROOT, DIFFLE_OUT in its environment. A `report.json` it writes into DIFFLE_OUT\nis the report, its files named relative to that directory; otherwise its stdout is one markdown\nblock. A command naming $DIFFLE_FILE runs once per matched file, with that path in the variable." + }, "PullRef": { "type": "object", "properties": { @@ -2014,6 +2042,400 @@ "body" ] }, + "Report": { + "type": "object", + "properties": { + "id": { + "type": "string", + "description": "Lowercase letters, digits and dashes; a generator's is its plugin name." + }, + "source": { + "$ref": "#/$defs/ReportSource" + }, + "title": { + "type": "string" + }, + "summary": { + "type": "string" + }, + "target": { + "$ref": "#/$defs/ReportTarget" + }, + "status": { + "$ref": "#/$defs/ReportStatus" + }, + "error": { + "type": [ + "string", + "null" + ], + "description": "The failure in one line: a program not on PATH, a nonzero exit, a report that did not validate." + }, + "stale": { + "type": "boolean", + "description": "The snapshot moved away from `target`: the blocks describe an older diff." + }, + "updatedAt": { + "type": "integer", + "minimum": -9007199254740991, + "maximum": 9007199254740991, + "description": "Milliseconds since the epoch." + }, + "blocks": { + "type": "array", + "items": { + "$ref": "#/$defs/ReportBlock" + } + } + }, + "required": [ + "id", + "source", + "title", + "summary", + "target", + "status", + "error", + "stale", + "updatedAt", + "blocks" + ] + }, + "ReportAnchor": { + "type": "object", + "properties": { + "path": { + "type": "string" + }, + "side": { + "$ref": "#/$defs/Side", + "description": "Default 'new'." + }, + "line": { + "type": "integer", + "minimum": -9007199254740991, + "maximum": 9007199254740991, + "description": "1-based." + }, + "endLine": { + "type": "integer", + "minimum": -9007199254740991, + "maximum": 9007199254740991, + "description": "Default `line`." + } + }, + "required": [ + "path", + "line" + ], + "description": "A place in the diff a report block points at: `c` on the block comments there. The new side\nunless `side` says old. A path the snapshot does not show is dropped when the report is served." + }, + "ReportBlock": { + "oneOf": [ + { + "type": "object", + "properties": { + "kind": { + "type": "string", + "const": "markdown" + }, + "text": { + "type": "string" + }, + "anchor": { + "$ref": "#/$defs/ReportAnchor" + } + }, + "required": [ + "kind", + "text" + ], + "description": "GitHub-flavored markdown, rendered as a comment body is: raw HTML stays literal and images are dropped." + }, + { + "type": "object", + "properties": { + "kind": { + "type": "string", + "const": "pages" + }, + "label": { + "type": "string" + }, + "pages": { + "type": "array", + "items": { + "$ref": "#/$defs/ReportPage" + } + } + }, + "required": [ + "kind", + "pages" + ] + } + ], + "description": "A report's blocks as the client gets them: every file resolved to a key." + }, + "ReportBlockInput": { + "oneOf": [ + { + "type": "object", + "properties": { + "kind": { + "type": "string", + "const": "markdown" + }, + "text": { + "type": "string" + }, + "anchor": { + "$ref": "#/$defs/ReportAnchor" + } + }, + "required": [ + "kind", + "text" + ], + "description": "GitHub-flavored markdown, rendered as a comment body is: raw HTML stays literal and images are dropped." + }, + { + "type": "object", + "properties": { + "kind": { + "type": "string", + "const": "pages" + }, + "label": { + "type": "string", + "description": "What the pages are, e.g. \"Rendered, removals struck through\"." + }, + "pages": { + "type": "array", + "items": { + "$ref": "#/$defs/ReportPageInput" + } + } + }, + "required": [ + "kind", + "pages" + ] + } + ], + "description": "A report's blocks as a generator or an agent writes them." + }, + "ReportFile": { + "type": "object", + "properties": { + "key": { + "type": "string" + }, + "type": { + "type": "string", + "description": "The media type its bytes were sniffed as; only raster images are ever served." + } + }, + "required": [ + "key", + "type" + ], + "description": "A file a report refers to, as the client fetches it: `GET /api/reports/{id}/file?key=`.\nThe key names fixed bytes, so what it serves may be cached for good." + }, + "ReportFileQuery": { + "type": "object", + "properties": { + "key": { + "type": "string" + } + }, + "required": [ + "key" + ] + }, + "ReportInput": { + "type": "object", + "properties": { + "title": { + "type": "string" + }, + "summary": { + "type": "string", + "description": "One line for the list; the first line of the first markdown block otherwise." + }, + "target": { + "$ref": "#/$defs/ReportTarget" + }, + "blocks": { + "type": "array", + "items": { + "$ref": "#/$defs/ReportBlockInput" + } + } + }, + "required": [ + "title", + "blocks" + ], + "description": "`PUT /api/reports/{id}`: a report as authored. Files it names must be uploaded first, with\n`PUT /api/reports/{id}/files/`. A `target` other than the current snapshot's is refused\nwith 409 and the current target, so an agent that fell behind can refetch; absent means current." + }, + "ReportLogResponse": { + "type": "object", + "properties": { + "lines": { + "type": "array", + "items": { + "type": "string" + } + } + }, + "required": [ + "lines" + ], + "description": "`GET /api/reports/{id}/log`: what the generator wrote to stderr, most recent run only." + }, + "ReportMarkdownBlock": { + "type": "object", + "properties": { + "kind": { + "type": "string", + "const": "markdown" + }, + "text": { + "type": "string" + }, + "anchor": { + "$ref": "#/$defs/ReportAnchor" + } + }, + "required": [ + "kind", + "text" + ], + "description": "GitHub-flavored markdown, rendered as a comment body is: raw HTML stays literal and images are dropped." + }, + "ReportPage": { + "type": "object", + "properties": { + "label": { + "type": "string" + }, + "old": { + "anyOf": [ + { + "$ref": "#/$defs/ReportFile" + }, + { + "type": "null" + } + ] + }, + "new": { + "anyOf": [ + { + "$ref": "#/$defs/ReportFile" + }, + { + "type": "null" + } + ] + }, + "anchor": { + "$ref": "#/$defs/ReportAnchor" + } + }, + "required": [ + "label", + "old", + "new" + ], + "description": "One rendered page or figure, as served: a side with a file is shown; both are compared." + }, + "ReportPageInput": { + "type": "object", + "properties": { + "label": { + "type": "string" + }, + "old": { + "type": [ + "string", + "null" + ] + }, + "new": { + "type": [ + "string", + "null" + ] + }, + "anchor": { + "$ref": "#/$defs/ReportAnchor" + } + }, + "required": [ + "label", + "old", + "new" + ], + "description": "One rendered page or figure, as authored: each side names a file next to the report, relative, or none." + }, + "ReportSource": { + "type": "string", + "enum": [ + "generator", + "pushed" + ], + "description": "A generator diffle ran for the snapshot, or something outside pushed it over the API." + }, + "ReportsResponse": { + "type": "object", + "properties": { + "current": { + "$ref": "#/$defs/ReportTarget" + }, + "reports": { + "type": "array", + "items": { + "$ref": "#/$defs/Report" + } + } + }, + "required": [ + "current", + "reports" + ], + "description": "`GET /api/reports`: every report, generated ones first, and the target a pushed one should name." + }, + "ReportStatus": { + "type": "string", + "enum": [ + "running", + "ready", + "failed" + ], + "description": "`running` keeps the previous blocks until the new ones land, as a snapshot refresh does." + }, + "ReportTarget": { + "type": "object", + "properties": { + "oldSha": { + "type": "string" + }, + "newSha": { + "type": "string" + }, + "digest": { + "type": "string" + } + }, + "required": [ + "oldSha", + "newSha", + "digest" + ], + "description": "The snapshot a report was made for. Two shas do not pin a worktree side, so `digest` hashes the\nchanged files' blobs too; `GET /api/reports` says the current one." + }, "ResolvedRequest": { "type": "object", "properties": { @@ -2309,6 +2731,18 @@ "type", "payload" ] + }, + { + "type": "object", + "properties": { + "type": { + "type": "string", + "const": "reports" + } + }, + "required": [ + "type" + ] } ] }, @@ -2525,6 +2959,16 @@ "type": "string" }, "description": "GitHub Enterprise hosts: the host in remote urls → the host serving the web UI and API, the\nsame one for most. github.com needs no entry. Read-only over HTTP; set by editing the\nconfig file." + }, + "plugins": { + "type": "object", + "propertyNames": { + "type": "string" + }, + "additionalProperties": { + "$ref": "#/$defs/PluginConfig" + }, + "description": "Report generators by name: a built-in plugin (`typst`, `pixi`) is replaced or turned off by\nits name, any other name adds one. Read-only over HTTP; set by editing the config file." } }, "required": [ @@ -2533,7 +2977,8 @@ "followRefs", "lineWrap", "lspCommands", - "githubHosts" + "githubHosts", + "plugins" ] }, "ViewedBulkRequest": { diff --git a/crates/diffle/src/cli/args.rs b/crates/diffle/src/cli/args.rs index a145ba2b..c28d217e 100644 --- a/crates/diffle/src/cli/args.rs +++ b/crates/diffle/src/cli/args.rs @@ -169,6 +169,13 @@ pub enum Command { Config, /// show the language server each language would get, and what is missing from PATH Lsp, + /// show the report generators a review would run, and what is missing from PATH + Plugins, + /// push a report into the diffle running for this repository, or remove one + Report { + #[command(subcommand)] + action: super::report::ReportAction, + }, /// print a shell completion script #[command(after_help = COMPLETION_HELP)] Completion { diff --git a/crates/diffle/src/cli/mod.rs b/crates/diffle/src/cli/mod.rs index d7704984..8430e76e 100644 --- a/crates/diffle/src/cli/mod.rs +++ b/crates/diffle/src/cli/mod.rs @@ -7,6 +7,8 @@ mod lsp; mod open; mod paint; mod parse; +mod plugins; +mod report; pub mod repository; mod serve; mod timing; @@ -47,6 +49,8 @@ async fn dispatch(cli: Cli) -> anyhow::Result<()> { let (req, later) = match cli.command { Some(Command::Config) => return config::run().await, Some(Command::Lsp) => return lsp::list().await, + Some(Command::Plugins) => return plugins::list().await, + Some(Command::Report { action }) => return report::run(action).await, Some(Command::Completion { shell }) => { completion::print(shell); return Ok(()); diff --git a/crates/diffle/src/cli/plugins.rs b/crates/diffle/src/cli/plugins.rs new file mode 100644 index 00000000..53b06f91 --- /dev/null +++ b/crates/diffle/src/cli/plugins.rs @@ -0,0 +1,45 @@ +//! The run's report generators: resolved from the catalog and the config, reported on stderr, and +//! listed by `diffle plugins`. +use super::paint::{dim, yellow}; +use crate::lsp::registry::default_lookup; +use crate::reports::catalog::{Plugin, Resolution, builtins, resolve}; +use crate::user_config::UserConfigStore; + +/// Which generators a run starts, from the user config and PATH. +pub fn resolved(config: &UserConfigStore) -> Resolution { + resolve(&config.get().plugins, &default_lookup()) +} + +/// The startup line: what will report on the diff, if anything. +pub fn report(plugins: &[Plugin]) { + if plugins.is_empty() { + return; + } + let names: Vec = + plugins.iter().map(|p| format!("{} {}", p.name, dim(&format!("({})", p.when.join(", "))))).collect(); + crate::status!("📄 {} {}", dim("reports"), names.join(", ")); +} + +/// `diffle plugins`: each generator a review would run and its command, then the built-in ones +/// whose programs are not on PATH, said the way the resolver decides it. +pub async fn list() -> anyhow::Result<()> { + let store = UserConfigStore::open(UserConfigStore::default_path()).await; + let config = store.get().plugins; + let resolution = resolve(&config, &default_lookup()); + for p in &resolution.plugins { + let from = if p.configured { dim(" (config)") } else { String::new() }; + println!("{} {} {}{from}", p.name, dim(&format!("[{}]", p.when.join(", "))), p.command); + } + for m in &resolution.missing { + println!("{} {} {}", m.name, dim(&format!("not on PATH (tried {})", m.tried.join(", "))), dim(&m.command)); + } + for b in builtins() { + if config.get(&b.name).is_some_and(|c| c.command.trim().is_empty()) { + println!("{} {}", b.name, dim("off in config")); + } + } + if resolution.plugins.is_empty() && resolution.missing.is_empty() { + crate::status!("{} no report generators", yellow("!")); + } + Ok(()) +} diff --git a/crates/diffle/src/cli/report.rs b/crates/diffle/src/cli/report.rs new file mode 100644 index 00000000..2c114f42 --- /dev/null +++ b/crates/diffle/src/cli/report.rs @@ -0,0 +1,262 @@ +//! `diffle report`: pushing a report into the diffle running for this repository, which announces +//! itself in `/diffle/server.json`. +use std::path::{Path, PathBuf}; +use std::time::Duration; + +use serde_json::{Value, json}; + +use super::paint::{cyan, dim}; +use crate::error::{Error, Result}; +use crate::git::GitRepo; +use crate::persist::write_file_atomic; +use crate::protocol::{ReportAnchor, ReportBlockInput, ReportInput}; + +/// What `diffle report` does. +#[derive(clap::Subcommand, Debug)] +pub enum ReportAction { + /// push a report: a markdown file becomes one block, a .json file is a whole report + #[command(after_help = PUT_HELP)] + Put { + /// the markdown or JSON file to push + #[arg(value_name = "file", value_hint = clap::ValueHint::FilePath)] + file: PathBuf, + /// the report's id (default: the file's name); putting an id again replaces it + #[arg(long, value_name = "id")] + id: Option, + /// the report's title (default: the file's first heading, else its name) + #[arg(long, value_name = "title")] + title: Option, + /// where a markdown report points: : or :- + #[arg(long, value_name = "path:line")] + anchor: Option, + /// run as if started in (any directory inside the reviewed repository) + #[arg(short = 'C', value_name = "path", value_hint = clap::ValueHint::DirPath)] + dir: Option, + }, + /// remove a pushed report + Delete { + /// the report's id + #[arg(value_name = "id")] + id: String, + /// run as if started in (any directory inside the reviewed repository) + #[arg(short = 'C', value_name = "path", value_hint = clap::ValueHint::DirPath)] + dir: Option, + }, +} + +const PUT_HELP: &str = "\ +A .json file is a report as the API takes it: + { \"title\": \"Dependency review\", + \"blocks\": [ + { \"kind\": \"markdown\", \"text\": \"...\", \"anchor\": { \"path\": \"pixi.lock\", \"line\": 1 } }, + { \"kind\": \"pages\", \"label\": \"Rendered\", + \"pages\": [{ \"label\": \"p. 1\", \"old\": \"old/1.png\", \"new\": \"new/1.png\" }] } ] } +The files a pages block names are read relative to the .json file and uploaded first; only +raster images are accepted. An anchor is dropped when the review does not show its path. + +Any other file is one markdown block; --anchor says what it is about. + +The report shows in the Reports view (gr) and goes stale when the diff moves on. Agents can +push one over HTTP instead: PUT api/reports/ with the same JSON; the url is in +/diffle/server.json while diffle runs."; + +/// How long the running diffle may take to answer. +const TIMEOUT: Duration = Duration::from_secs(10); + +fn announcement(git_dir: &Path) -> PathBuf { + git_dir.join("diffle").join("server.json") +} + +/// Records where this run listens, for `diffle report` and agents. A failure is not the run's. +pub async fn announce(git_dir: &Path, url: &str) { + let text = json!({ "url": url, "pid": std::process::id() }).to_string() + "\n"; + if let Err(e) = write_file_atomic(&announcement(git_dir), &text).await { + crate::status!("[diffle] announcing the server: {e:#}"); + } +} + +/// Withdraws this run's announcement, unless another run has since made its own. +pub async fn withdraw(git_dir: &Path) { + let file = announcement(git_dir); + if let Ok(text) = tokio::fs::read_to_string(&file).await + && serde_json::from_str::(&text).ok().and_then(|v| v["pid"].as_u64()) + == Some(u64::from(std::process::id())) + { + let _ = tokio::fs::remove_file(&file).await; + } +} + +pub async fn run(action: ReportAction) -> anyhow::Result<()> { + match action { + ReportAction::Put { file, id, title, anchor, dir } => put(&file, id, title, anchor, dir).await?, + ReportAction::Delete { id, dir } => { + crate::reports::input::check_id(&id)?; + let (url, client) = connect(dir).await?; + let res = client.delete(format!("{url}api/reports/{id}")).send().await.map_err(sent)?; + answered(res).await?; + crate::status!("{} report {id} removed", dim("✓")); + } + } + Ok(()) +} + +/// The running server's base url (with its trailing slash) and a client for it. +async fn connect(dir: Option) -> Result<(String, reqwest::Client)> { + let cwd = match dir { + Some(dir) => dir, + None => std::env::current_dir()?, + }; + let repo = GitRepo::open(&cwd).await?; + let not_running = + || Error::Conflict("no diffle is running for this repository; start one with `diffle working`".into()); + let text = tokio::fs::read_to_string(announcement(&repo.git_dir)).await.map_err(|_| not_running())?; + let url = serde_json::from_str::(&text) + .ok() + .and_then(|v| v["url"].as_str().map(str::to_owned)) + .ok_or_else(not_running)?; + let client = reqwest::Client::builder().timeout(TIMEOUT).build().map_err(anyhow::Error::from)?; + // The announcement outlives a crashed run: only an answer proves someone is listening. + if client.get(format!("{url}api/config")).send().await.is_err() { + return Err(not_running()); + } + Ok((url, client)) +} + +async fn put( + file: &Path, + id: Option, + title: Option, + anchor: Option, + dir: Option, +) -> Result<()> { + let text = tokio::fs::read_to_string(file) + .await + .map_err(|e| Error::Invalid(format!("reading {}: {e}", file.display())))?; + let stem = file.file_stem().map(|s| s.to_string_lossy().into_owned()).unwrap_or_default(); + let id = id.unwrap_or_else(|| id_of(&stem)); + crate::reports::input::check_id(&id)?; + let (mut report, base) = if file.extension().is_some_and(|e| e.eq_ignore_ascii_case("json")) { + let report: ReportInput = + serde_json::from_str(&text).map_err(|e| Error::Invalid(format!("{}: {e}", file.display())))?; + (report, file.parent().map(Path::to_path_buf)) + } else { + let anchor = anchor.as_deref().map(parse_anchor).transpose()?; + let heading = text.lines().map(str::trim).find(|l| !l.is_empty()).map(|l| l.trim_start_matches('#').trim()); + let title = heading.filter(|h| !h.is_empty()).map_or_else(|| stem.clone(), str::to_owned); + ( + ReportInput { + title, + summary: None, + target: None, + blocks: vec![ReportBlockInput::Markdown { text, anchor }], + }, + None, + ) + }; + if let Some(title) = title { + report.title = title; + } + let (url, client) = connect(dir).await?; + for name in named_files(&report) { + let path = base.clone().unwrap_or_default().join(&name); + let bytes = + tokio::fs::read(&path).await.map_err(|e| Error::Invalid(format!("reading {}: {e}", path.display())))?; + let res = client.put(format!("{url}api/reports/{id}/files/{name}")).body(bytes).send().await.map_err(sent)?; + answered(res).await?; + } + let res = client.put(format!("{url}api/reports/{id}")).json(&report).send().await.map_err(sent)?; + let created = res.status() == reqwest::StatusCode::CREATED; + let report = answered(res).await?; + crate::status!( + "{} report {} {} {}", + dim("✓"), + report["title"].as_str().unwrap_or(&id), + if created { "added" } else { "replaced" }, + dim(&format!("({id}, {url})")) + ); + crate::status!(" {}", cyan(&format!("{url}#reports"))); + Ok(()) +} + +/// A request that got no answer: the server went away mid-way, which the user can fix. +fn sent(e: reqwest::Error) -> Error { + Error::Conflict(format!("the running diffle did not answer: {e}")) +} + +/// The response's JSON, or its `error` as a user error. +async fn answered(res: reqwest::Response) -> Result { + let status = res.status(); + let text = res.text().await.map_err(anyhow::Error::from)?; + let value: Value = serde_json::from_str(&text).unwrap_or(Value::Null); + if status.is_success() { + return Ok(value); + } + let message = value["error"].as_str().map_or_else(|| format!("{status}: {text}"), str::to_owned); + Err(if status == reqwest::StatusCode::CONFLICT { Error::Conflict(message) } else { Error::Invalid(message) }) +} + +/// A file name as a report id: lowercased, anything else a dash, at most 64 long. +fn id_of(stem: &str) -> String { + let mut id = String::new(); + for c in stem.to_lowercase().chars() { + let keep = c.is_ascii_lowercase() || c.is_ascii_digit(); + if keep { + id.push(c); + } else if !id.ends_with('-') && !id.is_empty() { + id.push('-'); + } + } + let id = id.trim_end_matches('-'); + id.chars().take(64).collect::().trim_end_matches('-').to_owned() +} + +/// `path:line` or `path:start-end`. +fn parse_anchor(raw: &str) -> Result { + let invalid = || Error::Invalid(format!("--anchor: expected : or :-, got \"{raw}\"")); + let (path, lines) = raw.rsplit_once(':').ok_or_else(invalid)?; + let (start, end) = match lines.split_once('-') { + Some((a, b)) => (a, Some(b)), + None => (lines, None), + }; + let line = start.parse::().ok().filter(|n| *n > 0).ok_or_else(invalid)?; + let end_line = end.map(|e| e.parse::().ok().filter(|n| *n >= line).ok_or_else(invalid)).transpose()?; + if path.is_empty() { + return Err(invalid()); + } + Ok(ReportAnchor { path: path.to_owned(), side: None, line, end_line }) +} + +fn named_files(report: &ReportInput) -> Vec { + let mut names: Vec = report + .blocks + .iter() + .flat_map(|b| match b { + ReportBlockInput::Pages { pages, .. } => { + pages.iter().flat_map(|p| p.old.iter().chain(p.new.iter()).cloned()).collect() + } + ReportBlockInput::Markdown { .. } => Vec::new(), + }) + .collect(); + names.sort(); + names.dedup(); + names +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn derives_ids_and_parses_anchors() { + assert_eq!(id_of("Review Notes (v2)"), "review-notes-v2"); + assert_eq!(id_of("--"), ""); + assert_eq!(id_of(&"a".repeat(70)).len(), 64); + let a = parse_anchor("src/a.rs:12-14").unwrap(); + assert_eq!((a.path.as_str(), a.line, a.end_line), ("src/a.rs", 12, Some(14))); + let a = parse_anchor("C:/x/y.txt:3").unwrap(); + assert_eq!((a.path.as_str(), a.line, a.end_line), ("C:/x/y.txt", 3, None)); + for bad in ["a.rs", "a.rs:0", "a.rs:5-3", ":3", "a.rs:x"] { + assert!(parse_anchor(bad).is_err(), "{bad}"); + } + } +} diff --git a/crates/diffle/src/cli/serve.rs b/crates/diffle/src/cli/serve.rs index 4a18b66f..f7862dae 100644 --- a/crates/diffle/src/cli/serve.rs +++ b/crates/diffle/src/cli/serve.rs @@ -17,6 +17,7 @@ use crate::github::client::token_variables; use crate::github::hosts::Github; use crate::lsp::pool::LspPool; use crate::protocol::{EnterRequest, Live, Snapshot, ThreadQuery, ThreadState, comparison_label}; +use crate::reports::Reports; use crate::server::hub::Hub; use crate::server::{ApiDeps, DEFAULT_PORT, Server, ServerOptions}; use crate::session::{Session, SessionOptions}; @@ -56,6 +57,7 @@ pub async fn run(req: EnterRequest, opts: GlobalOpts) -> anyhow::Result<()> { overrides.extend(opts.lsp.iter().map(|o| (o.language, o.command.clone()))); super::lsp::start(&session, &hub, overrides, opts.lsp.iter().map(|o| o.language).collect()) }); + let reports = Reports::new(session.clone(), hub.clone(), &repo.git_dir, super::plugins::resolved(&config).plugins); let server: Arc>> = Arc::default(); let stop = Arc::new(Notify::new()); @@ -81,6 +83,7 @@ pub async fn run(req: EnterRequest, opts: GlobalOpts) -> anyhow::Result<()> { &server, &repo, lsp.clone(), + &reports, (&github, token_source.as_deref()), &mut timing, ); @@ -90,15 +93,18 @@ pub async fn run(req: EnterRequest, opts: GlobalOpts) -> anyhow::Result<()> { }; match started { Some(Err(e)) => { + reports.close(); dispose(&session, &server, &repo, lsp.as_deref()).await; Err(e) } Some(Ok(())) => { stop.notified().await; + reports.close(); shutdown(&session, &server, &repo, lsp.as_deref()).await; Ok(()) } None => { + reports.close(); shutdown(&session, &server, &repo, lsp.as_deref()).await; Ok(()) } @@ -115,6 +121,7 @@ async fn start( server: &Mutex>, repo: &GitRepo, lsp: Option>, + reports: &Arc, (github, token_source): (&Github, Option<&str>), timing: &mut Timing, ) -> anyhow::Result<()> { @@ -127,6 +134,7 @@ async fn start( extra_auto_viewed: opts.auto_viewed.clone(), hub: hub.clone(), lsp: lsp.clone(), + reports: reports.clone(), }; let server_opts = ServerOptions { port: opts.port.unwrap_or(DEFAULT_PORT), @@ -136,6 +144,7 @@ async fn start( }; let (bound, url) = Server::listen(deps, server_opts).await?; *server.lock().unwrap() = Some(bound); + super::report::announce(&repo.git_dir, &url).await; timing.mark("listen"); if !opts.no_open { let browser = if opts.host() == "0.0.0.0" || opts.host() == "::" { localhost(&url) } else { url.clone() }; @@ -162,6 +171,7 @@ async fn start( if let Some(lsp) = &lsp { super::lsp::report(lsp, &snap); } + super::plugins::report(reports.plugins()); timing.report(); Ok(()) } @@ -231,7 +241,7 @@ async fn dispose(session: &Arc, server: &Mutex>, repo: & }; let _ = tokio::time::timeout(CLOSE_BOUND, async { tokio::join!(sockets, servers) }).await; }; - tokio::join!(session.close(), close_server); + tokio::join!(session.close(), close_server, super::report::withdraw(&repo.git_dir)); if let Err(e) = repo.clean_review_refs().await { crate::status!("{} {e}", red("✖")); } diff --git a/crates/diffle/src/error.rs b/crates/diffle/src/error.rs index ba10263c..0c794b85 100644 --- a/crates/diffle/src/error.rs +++ b/crates/diffle/src/error.rs @@ -75,6 +75,9 @@ pub enum Error { /// A request body that is not valid JSON or not the shape the route takes. #[error("{0}")] Invalid(String), + /// A request that the current state refuses, such as replacing a generator's report: the API answers 409. + #[error("{0}")] + Conflict(String), #[error(transparent)] Other(Arc), } diff --git a/crates/diffle/src/lib.rs b/crates/diffle/src/lib.rs index a0ba973b..0a261e1d 100644 --- a/crates/diffle/src/lib.rs +++ b/crates/diffle/src/lib.rs @@ -23,6 +23,7 @@ pub mod lsp; pub mod mode; pub mod persist; pub mod protocol; +pub mod reports; pub mod server; pub mod session; pub mod snapshotter; diff --git a/crates/diffle/src/reports/catalog.rs b/crates/diffle/src/reports/catalog.rs new file mode 100644 index 00000000..de522657 --- /dev/null +++ b/crates/diffle/src/reports/catalog.rs @@ -0,0 +1,244 @@ +//! Every report generator diffle knows, and which to run: the built-in rows, replaced or turned off +//! by name in the user config, plus the config's own. +use regex::Regex; +use std::collections::HashMap; + +use crate::lsp::registry::Lookup; +use crate::lsp::which::argv0; +use crate::protocol::{ChangedFile, PluginConfig, PluginConfigScope}; + +/// Which files of a snapshot a generator gets copies of. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum Scope { + /// The changed files its globs matched, with a rename's old name on the old side. + Changed, + /// Every file the snapshot shows, so a document's imports resolve. + Tree, +} + +/// One generator to run: its report's id is `name`. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct Plugin { + pub name: String, + pub command: String, + pub when: Vec, + pub scope: Scope, + /// Where it came from, for `diffle plugins`. + pub configured: bool, +} + +/// A built-in generator whose programs are not on PATH. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct Missing { + pub name: String, + pub command: String, + /// The programs looked for, so the UI can name what to install. + pub tried: Vec, +} + +pub struct Resolution { + pub plugins: Vec, + pub missing: Vec, +} + +struct Builtin { + name: &'static str, + command: &'static str, + when: &'static [&'static str], + scope: Scope, +} + +/// The generators diffle starts on its own when their programs are on PATH. +const BUILTINS: &[Builtin] = &[ + Builtin { + name: "pixi", + command: "pixi-diff --before $DIFFLE_OLD/$DIFFLE_FILE --after $DIFFLE_NEW/$DIFFLE_FILE | pixi-diff-to-markdown", + when: &["pixi.lock"], + scope: Scope::Changed, + }, + Builtin { name: "typst", command: "diffle-typst", when: &["*.typ"], scope: Scope::Tree }, +]; + +/// The built-in rows, as `diffle plugins` lists them. +pub fn builtins() -> Vec { + BUILTINS + .iter() + .map(|b| plugin(b.name, b.command, b.when.iter().map(|w| (*w).to_owned()).collect(), b.scope, false)) + .collect() +} + +fn plugin(name: &str, command: &str, when: Vec, scope: Scope, configured: bool) -> Plugin { + Plugin { name: name.to_owned(), command: command.trim().to_owned(), when, scope, configured } +} + +/// The programs a command line runs: the first word of each `|`-separated stage. +pub fn programs_of(command: &str) -> Vec { + command.split('|').map(|stage| argv0(stage.trim())).filter(|p| !p.is_empty()).collect() +} + +/// Which generators to run. A config entry named like a built-in replaces it, and one with an +/// empty command turns it off; a built-in is never run while a program of its is off PATH, and is +/// listed as missing instead. Configured commands are not probed: the user named them, so a typo +/// surfaces as a failed report, not as silence. +pub fn resolve(config: &HashMap, lookup: &Lookup) -> Resolution { + let mut plugins = Vec::new(); + let mut missing = Vec::new(); + for b in BUILTINS { + let Some(entry) = config.get(b.name) else { + let tried: Vec = programs_of(b.command).into_iter().filter(|p| lookup(p).is_none()).collect(); + if tried.is_empty() { + plugins.push(plugin( + b.name, + b.command, + b.when.iter().map(|w| (*w).to_owned()).collect(), + b.scope, + false, + )); + } else { + missing.push(Missing { name: b.name.to_owned(), command: b.command.to_owned(), tried }); + } + continue; + }; + if entry.command.trim().is_empty() { + continue; + } + let when = + if entry.when.is_empty() { b.when.iter().map(|w| (*w).to_owned()).collect() } else { entry.when.clone() }; + plugins.push(plugin(b.name, &entry.command, when, scope_of(entry).unwrap_or(b.scope), true)); + } + let mut named: Vec<(&String, &PluginConfig)> = + config.iter().filter(|(name, _)| !BUILTINS.iter().any(|b| b.name == name.as_str())).collect(); + named.sort_by(|a, b| a.0.cmp(b.0)); + for (name, entry) in named { + if entry.command.trim().is_empty() || entry.when.is_empty() { + continue; + } + plugins.push(plugin(name, &entry.command, entry.when.clone(), scope_of(entry).unwrap_or(Scope::Changed), true)); + } + Resolution { plugins, missing } +} + +fn scope_of(entry: &PluginConfig) -> Option { + entry.scope.map(|s| match s { + PluginConfigScope::Changed => Scope::Changed, + PluginConfigScope::Tree => Scope::Tree, + }) +} + +/// The changed files `plugin` runs for: those a glob of its matches, by new path. +pub fn matched(plugin: &Plugin, changed: &[ChangedFile]) -> Vec { + let globs: Vec = plugin.when.iter().map(|g| Glob::new(g)).collect(); + changed.iter().filter(|f| globs.iter().any(|g| g.is_match(&f.path))).map(|f| f.path.clone()).collect() +} + +/// A picomatch-style glob: `*` and `?` stay within one path segment, `**` spans any number. A +/// pattern without a slash matches the file's name in any folder, as a gitignore line does. +pub struct Glob { + regex: Regex, + basename: bool, +} + +impl Glob { + pub fn new(pattern: &str) -> Self { + let pattern = pattern.trim().trim_start_matches("./"); + let basename = !pattern.contains('/'); + let mut re = String::from("^"); + let mut chars = pattern.chars().peekable(); + while let Some(c) = chars.next() { + match c { + '*' if chars.peek() == Some(&'*') => { + chars.next(); + // `**/` also matches no folder at all. + if chars.peek() == Some(&'/') { + chars.next(); + re.push_str("(?:.*/)?"); + } else { + re.push_str(".*"); + } + } + '*' => re.push_str("[^/]*"), + '?' => re.push_str("[^/]"), + other => re.push_str(®ex::escape(&other.to_string())), + } + } + re.push('$'); + Self { regex: Regex::new(&re).expect("escaped glob"), basename } + } + + pub fn is_match(&self, path: &str) -> bool { + let subject = if self.basename { path.rsplit('/').next().unwrap_or(path) } else { path }; + self.regex.is_match(subject) + } +} + +#[cfg(test)] +mod tests { + use super::*; + use std::path::PathBuf; + use std::sync::Arc; + + #[test] + fn globs_follow_picomatch_within_segments() { + assert!(Glob::new("pixi.lock").is_match("apps/web/pixi.lock")); + assert!(Glob::new("*.typ").is_match("thesis/chapters/intro.typ")); + assert!(!Glob::new("*.typ").is_match("thesis/intro.typst")); + assert!(Glob::new("docs/**/*.md").is_match("docs/a/b/c.md")); + assert!(Glob::new("docs/**/*.md").is_match("docs/c.md")); + assert!(!Glob::new("docs/*.md").is_match("docs/a/c.md")); + assert!(Glob::new("src/?.rs").is_match("src/a.rs")); + assert!(!Glob::new("src/?.rs").is_match("src/ab.rs")); + assert!(Glob::new("**/pixi.lock").is_match("pixi.lock")); + } + + fn lookup(present: &'static [&'static str]) -> Lookup { + Arc::new(move |p: &str| present.contains(&p).then(|| PathBuf::from(p))) + } + + #[test] + fn resolves_builtins_against_path_and_config() { + let r = resolve(&HashMap::new(), &lookup(&["pixi-diff", "pixi-diff-to-markdown"])); + assert_eq!(r.plugins.iter().map(|p| p.name.as_str()).collect::>(), ["pixi"]); + assert_eq!( + r.missing, + [Missing { name: "typst".into(), command: "diffle-typst".into(), tried: vec!["diffle-typst".into()] }] + ); + + let config = HashMap::from([ + ("pixi".to_owned(), PluginConfig { command: String::new(), when: vec![], scope: None }), + ("typst".to_owned(), PluginConfig { command: "my-typst --fast".into(), when: vec![], scope: None }), + ( + "notes".to_owned(), + PluginConfig { + command: "notes".into(), + when: vec!["*.md".into()], + scope: Some(PluginConfigScope::Tree), + }, + ), + ("nowhen".to_owned(), PluginConfig { command: "x".into(), when: vec![], scope: None }), + ]); + let r = resolve(&config, &lookup(&[])); + assert!(r.missing.is_empty()); + let names: Vec<(&str, &str, Scope, bool)> = + r.plugins.iter().map(|p| (p.name.as_str(), p.command.as_str(), p.scope, p.configured)).collect(); + assert_eq!(names, [("typst", "my-typst --fast", Scope::Tree, true), ("notes", "notes", Scope::Tree, true)]); + assert_eq!(r.plugins[0].when, ["*.typ"]); + } + + #[test] + fn matched_files_follow_the_globs() { + let plugin = plugin("p", "p", vec!["pixi.lock".into()], Scope::Changed, false); + let file = |path: &str| ChangedFile { + path: path.into(), + old_path: None, + status: crate::protocol::ChangeStatus::M, + additions: 0, + deletions: 0, + binary: false, + blob: String::new(), + old_blob: String::new(), + generated: false, + submodule: None, + }; + assert_eq!(matched(&plugin, &[file("a/pixi.lock"), file("pixi.toml")]), ["a/pixi.lock"]); + } +} diff --git a/crates/diffle/src/reports/exec.rs b/crates/diffle/src/reports/exec.rs new file mode 100644 index 00000000..0f113fac --- /dev/null +++ b/crates/diffle/src/reports/exec.rs @@ -0,0 +1,121 @@ +//! Running a generator's command line: the stages a `|` separates, the DIFFLE_* variables, and +//! the pipe from one program's stdout to the next, with no shell in between. +use std::path::PathBuf; +use std::process::Stdio; + +use tokio::io::AsyncWriteExt; + +use super::runner::{FILE_VAR, Failure, failed}; +use crate::error::Result; +use crate::lsp::which::find_on_path; + +/// Lines of stderr kept per run. +const LOG_LINES: usize = 200; + +pub(super) struct Vars { + pub(super) root: String, + pub(super) old: PathBuf, + pub(super) new: PathBuf, + pub(super) out: PathBuf, + pub(super) file: Option, +} + +impl Vars { + pub(super) fn pairs(&self) -> Vec<(&'static str, String)> { + let mut pairs = vec![ + ("DIFFLE_ROOT", self.root.clone()), + ("DIFFLE_OLD", self.old.to_string_lossy().into_owned()), + ("DIFFLE_NEW", self.new.to_string_lossy().into_owned()), + ("DIFFLE_OUT", self.out.to_string_lossy().into_owned()), + ]; + if let Some(file) = &self.file { + pairs.push((FILE_VAR, file.clone())); + } + pairs + } + + /// `word` with every `$NAME` and `${NAME}` of the variables above substituted. + pub(super) fn expand(&self, word: &str) -> String { + let mut out = word.to_owned(); + for (name, value) in self.pairs() { + out = out.replace(&format!("${{{name}}}"), &value).replace(&format!("${name}"), &value); + } + out + } +} + +/// The command line as stages, each a program and its arguments; a `|` word separates them. +pub(super) fn stages_of(command: &str) -> Result>> { + let words = shlex::split(command).ok_or_else(|| failed(format!("unbalanced quotes in \"{command}\"")))?; + let stages: Vec> = words.split(|w| w == "|").map(|s| s.to_vec()).collect(); + if stages.iter().any(|s| s.is_empty()) { + return Err(failed(format!("no program named in \"{command}\""))); + } + Ok(stages) +} + +pub(super) struct Output { + pub(super) stdout: Vec, + pub(super) log: Vec, +} + +/// Runs the stages in turn, each fed the previous one's stdout, the first `input`. +pub(super) async fn pipeline(stages: &[Vec], vars: &Vars, input: Vec) -> Result { + let mut data = input; + let mut log = Vec::new(); + for stage in stages { + let program = &stage[0]; + let args: Vec = stage[1..].iter().map(|w| vars.expand(w)).collect(); + let resolved = find_on_path(program).ok_or_else(|| failed(format!("{program}: not on PATH")))?; + let mut cmd = tokio::process::Command::new(resolved); + cmd.args(&args).envs(vars.pairs()).current_dir(&vars.out); + cmd.stdin(Stdio::piped()).stdout(Stdio::piped()).stderr(Stdio::piped()).kill_on_drop(true); + let mut child = cmd.spawn().map_err(|e| failed(format!("{program}: {e}")))?; + if let Some(mut stdin) = child.stdin.take() { + let feed = data; + // A program that never reads stdin closes it; that is not a failure. + tokio::spawn(async move { + let _ = stdin.write_all(&feed).await; + let _ = stdin.shutdown().await; + }); + } + let output = child.wait_with_output().await.map_err(|e| failed(format!("{program}: {e}")))?; + let stderr = String::from_utf8_lossy(&output.stderr); + let name = if stages.len() > 1 { format!("{program}: ") } else { String::new() }; + log.extend(stderr.lines().map(|l| format!("{name}{l}"))); + if log.len() > LOG_LINES { + log.drain(..log.len() - LOG_LINES); + } + if !output.status.success() { + let last = stderr.lines().rev().find(|l| !l.trim().is_empty()).unwrap_or_default(); + let how = output.status.code().map_or_else(|| "killed".to_owned(), |c| format!("exited {c}")); + let message = + format!("{program} {how}{}", if last.is_empty() { String::new() } else { format!(": {last}") }); + return Err(Failure { message, log }); + } + data = output.stdout; + } + Ok(Output { stdout: data, log }) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn splits_stages_and_expands_variables() { + let stages = stages_of("pixi-diff --before \"$DIFFLE_OLD/a b\" | md ${DIFFLE_FILE}").unwrap(); + assert_eq!(stages, [vec!["pixi-diff", "--before", "$DIFFLE_OLD/a b"], vec!["md", "${DIFFLE_FILE}"]]); + let vars = Vars { + root: "/r".into(), + old: "/o".into(), + new: "/n".into(), + out: "/out".into(), + file: Some("pixi.lock".into()), + }; + assert_eq!(vars.expand(&stages[0][2]), "/o/a b"); + assert_eq!(vars.expand(&stages[1][1]), "pixi.lock"); + assert!(stages_of("a | | b").is_err()); + assert!(stages_of("a \"b").is_err()); + } +} diff --git a/crates/diffle/src/reports/generated.rs b/crates/diffle/src/reports/generated.rs new file mode 100644 index 00000000..62fa2c30 --- /dev/null +++ b/crates/diffle/src/reports/generated.rs @@ -0,0 +1,129 @@ +//! Reports the generators make: started on every snapshot, superseded by the next, cached by input. +use std::collections::HashMap; +use std::sync::Arc; + +use super::catalog::{self, Plugin}; +use super::runner::{self, Failure, Produced}; +use super::store::Stored; +use super::{Reports, target_of}; +use crate::iterations::now_ms; +use crate::protocol::{ReportSource, ReportStatus, ReportTarget, ServerMessage, Snapshot}; + +impl Reports { + /// Follows a snapshot: loads its pushed reports and starts its generators. + pub(super) async fn sync(self: Arc, snap: Arc) { + if let Err(e) = self.load_pushed(&snap).await { + crate::status!("[diffle] loading reports: {e:#}"); + } + let target = target_of(&snap); + let mut changed = false; + { + let mut state = self.state.lock().unwrap(); + if snap.version < state.version { + return; + } + state.version = snap.version; + for plugin in &self.plugins { + let matched = catalog::matched(plugin, &snap.changed); + if matched.is_empty() { + changed |= state.generated.remove(&plugin.name).is_some(); + if let Some(run) = self.runs.lock().unwrap().remove(&plugin.name) { + run.abort(); + } + continue; + } + changed = true; + let previous = state.generated.get(&plugin.name); + let stored = Stored { + id: plugin.name.clone(), + source: ReportSource::Generator, + title: previous.map_or_else(|| plugin.name.clone(), |p| p.title.clone()), + summary: previous.map(|p| p.summary.clone()).unwrap_or_default(), + target: previous.map_or_else(|| target.clone(), |p| p.target.clone()), + status: ReportStatus::Running, + error: None, + updated_at: now_ms(), + blocks: previous.map(|p| p.blocks.clone()).unwrap_or_default(), + files: HashMap::new(), + }; + state.generated.insert(plugin.name.clone(), stored); + self.start(snap.clone(), target.clone(), plugin.clone(), matched); + } + } + if changed { + self.hub.broadcast(ServerMessage::Reports); + } + } + + /// Runs `plugin` for `snap` on its own task, in place of any run still going for it. + fn start(self: &Arc, snap: Arc, target: ReportTarget, plugin: Plugin, matched: Vec) { + let this = self.clone(); + let name = plugin.name.clone(); + let task = tokio::spawn(async move { + let key = format!("{}\0{}\0{}\0{}", plugin.name, plugin.command, plugin.when.join(","), target.digest); + let cached = this.cache.lock().unwrap().get(&key).cloned(); + let outcome = match cached { + Some(produced) => Ok(produced), + None => runner::run(&this.session, &this.store, &snap, &target, &plugin, &matched).await.map(Arc::new), + }; + if let Ok(produced) = &outcome { + this.cache.lock().unwrap().insert(key, produced.clone()); + } + this.finish(&snap, target, &plugin.name, outcome); + }); + if let Some(previous) = self.runs.lock().unwrap().insert(name, task.abort_handle()) { + previous.abort(); + } + } + + /// Commits a run's outcome, unless a newer snapshot has superseded it. + fn finish( + self: &Arc, + snap: &Snapshot, + target: ReportTarget, + name: &str, + outcome: Result, Failure>, + ) { + { + let mut state = self.state.lock().unwrap(); + if snap.version != state.version { + return; + } + let Some(stored) = state.generated.get_mut(name) else { return }; + match &outcome { + Ok(produced) => { + stored.title = produced.title.clone(); + stored.summary = produced.summary.clone(); + stored.blocks = produced.blocks.clone(); + stored.status = ReportStatus::Ready; + stored.error = None; + } + Err(e) => { + stored.status = ReportStatus::Failed; + stored.error = Some(e.message.clone()); + } + } + stored.target = target; + stored.updated_at = now_ms(); + let log = match &outcome { + Ok(produced) => produced.log.clone(), + Err(e) => e.log.clone(), + }; + state.logs.insert(name.to_owned(), log); + self.runs.lock().unwrap().remove(name); + } + if let Err(e) = &outcome { + crate::status!("[diffle] report {name}: {}", e.message); + } + let this = self.clone(); + tokio::spawn(async move { this.sweep().await }); + self.hub.broadcast(ServerMessage::Reports); + } + + /// Ends every run in flight. + pub fn close(&self) { + for (_, run) in self.runs.lock().unwrap().drain() { + run.abort(); + } + } +} diff --git a/crates/diffle/src/reports/input.rs b/crates/diffle/src/reports/input.rs new file mode 100644 index 00000000..5f8ee4cd --- /dev/null +++ b/crates/diffle/src/reports/input.rs @@ -0,0 +1,260 @@ +//! A report as authored, checked and resolved into one as served: every named file a key, every +//! anchor a place the snapshot shows, every limit kept. +use std::collections::HashMap; + +use crate::error::{Error, Result}; +use crate::protocol::{ + ReportAnchor, ReportBlock, ReportBlockInput, ReportFile, ReportInput, ReportPage, ReportPageInput, Side, +}; + +/// The most blocks one report holds; a reviewer reads a report, not a dump. +pub const MAX_BLOCKS: usize = 200; +/// The most pages one block holds. +pub const MAX_PAGES: usize = 500; +/// The longest markdown block, in bytes. +pub const MAX_MARKDOWN_BYTES: usize = 256 * 1024; +pub const MAX_TITLE_CHARS: usize = 120; +pub const MAX_SUMMARY_CHARS: usize = 200; +pub const MAX_LABEL_CHARS: usize = 80; + +/// A report checked and resolved; `notes` says what was dropped on the way. +#[derive(Debug)] +pub struct Validated { + pub title: String, + pub summary: String, + pub blocks: Vec, + pub notes: Vec, +} + +/// `id` as the API takes it: a lowercase word of letters, digits and dashes, at most 64 long. +pub fn check_id(id: &str) -> Result<()> { + let ok = !id.is_empty() + && id.len() <= 64 + && id.starts_with(|c: char| c.is_ascii_lowercase() || c.is_ascii_digit()) + && id.chars().all(|c| c.is_ascii_lowercase() || c.is_ascii_digit() || c == '-'); + if ok { Ok(()) } else { Err(invalid("id: lowercase letters, digits and dashes, at most 64")) } +} + +/// Whether `path` names a file next to a report: relative, with no `..` or empty segment, and no +/// backslash, so it names the same file on every platform. +pub fn safe_relative(path: &str) -> bool { + !path.is_empty() + && !path.starts_with('/') + && !path.contains('\\') + && !path.contains('\0') + && path.split('/').all(|seg| !seg.is_empty() && seg != "." && seg != "..") +} + +fn invalid(message: impl Into) -> Error { + Error::Invalid(message.into()) +} + +/// Checks `input` and resolves it against `files`, the uploaded files by authored path, and +/// `shown`, which says whether a path is in the snapshot on a side. A file the report names but +/// nobody uploaded fails; an anchor nothing shows is dropped with a note. +pub fn validate( + input: ReportInput, + files: &HashMap, + shown: impl Fn(&str, Side) -> bool, +) -> Result { + let title = input.title.trim().to_owned(); + if title.is_empty() { + return Err(invalid("title: required")); + } + if title.chars().count() > MAX_TITLE_CHARS { + return Err(invalid(format!("title: at most {MAX_TITLE_CHARS} characters"))); + } + if input.blocks.len() > MAX_BLOCKS { + return Err(invalid(format!("blocks: at most {MAX_BLOCKS}"))); + } + let mut notes = Vec::new(); + let mut blocks = Vec::with_capacity(input.blocks.len()); + for (i, block) in input.blocks.into_iter().enumerate() { + blocks.push(resolve_block(i, block, files, &shown, &mut notes)?); + } + let summary = match input.summary.map(|s| s.trim().to_owned()).filter(|s| !s.is_empty()) { + Some(s) if s.chars().count() > MAX_SUMMARY_CHARS => { + return Err(invalid(format!("summary: at most {MAX_SUMMARY_CHARS} characters"))); + } + Some(s) => s, + None => first_line(&blocks), + }; + Ok(Validated { title, summary, blocks, notes }) +} + +fn resolve_block( + i: usize, + block: ReportBlockInput, + files: &HashMap, + shown: &impl Fn(&str, Side) -> bool, + notes: &mut Vec, +) -> Result { + match block { + ReportBlockInput::Markdown { text, anchor } => { + if text.len() > MAX_MARKDOWN_BYTES { + return Err(invalid(format!("blocks[{i}].text: at most {MAX_MARKDOWN_BYTES} bytes"))); + } + let anchor = anchor.and_then(|a| placed(a, &format!("blocks[{i}]"), shown, notes)); + Ok(ReportBlock::Markdown { text, anchor }) + } + ReportBlockInput::Pages { label, pages } => { + if pages.len() > MAX_PAGES { + return Err(invalid(format!("blocks[{i}].pages: at most {MAX_PAGES}"))); + } + let label = label.map(|l| l.trim().to_owned()).filter(|l| !l.is_empty()); + if label.as_ref().is_some_and(|l| l.chars().count() > MAX_LABEL_CHARS) { + return Err(invalid(format!("blocks[{i}].label: at most {MAX_LABEL_CHARS} characters"))); + } + let pages = pages + .into_iter() + .enumerate() + .map(|(j, page)| resolve_page(&format!("blocks[{i}].pages[{j}]"), page, files, shown, notes)) + .collect::>>()?; + Ok(ReportBlock::Pages { label, pages }) + } + } +} + +fn resolve_page( + at: &str, + page: ReportPageInput, + files: &HashMap, + shown: &impl Fn(&str, Side) -> bool, + notes: &mut Vec, +) -> Result { + if page.label.chars().count() > MAX_LABEL_CHARS { + return Err(invalid(format!("{at}.label: at most {MAX_LABEL_CHARS} characters"))); + } + if page.old.is_none() && page.new.is_none() { + return Err(invalid(format!("{at}: a page needs an old or a new file"))); + } + let file = |side: &str, path: Option| -> Result> { + let Some(path) = path else { return Ok(None) }; + if !safe_relative(&path) { + return Err(invalid(format!("{at}.{side}: a relative path under the report's files, got \"{path}\""))); + } + files + .get(&path) + .cloned() + .map(Some) + .ok_or_else(|| invalid(format!("{at}.{side}: no file uploaded as \"{path}\""))) + }; + Ok(ReportPage { + label: page.label, + old: file("old", page.old)?, + new: file("new", page.new)?, + anchor: page.anchor.and_then(|a| placed(a, at, shown, notes)), + }) +} + +/// `anchor` when the snapshot shows its path on its side and its lines are in order, else a note. +fn placed( + anchor: ReportAnchor, + at: &str, + shown: &impl Fn(&str, Side) -> bool, + notes: &mut Vec, +) -> Option { + let side = anchor.side.unwrap_or(Side::New); + let end = anchor.end_line.unwrap_or(anchor.line); + let why = if anchor.line < 1 || end < anchor.line { + Some("lines must be 1-based and in order".to_owned()) + } else if !shown(&anchor.path, side) { + Some(format!("the review does not show {} on the {side} side", anchor.path)) + } else { + None + }; + match why { + Some(why) => { + notes.push(format!("{at}.anchor dropped: {why}")); + None + } + None => Some(ReportAnchor { end_line: anchor.end_line.filter(|e| *e != anchor.line), ..anchor }), + } +} + +/// The first non-blank line of the first markdown block, bare of heading marks, cut to the summary length. +fn first_line(blocks: &[ReportBlock]) -> String { + blocks + .iter() + .find_map(|b| match b { + ReportBlock::Markdown { text, .. } => text.lines().map(str::trim).find(|l| !l.is_empty()), + ReportBlock::Pages { .. } => None, + }) + .map(|l| l.trim_start_matches('#').trim().chars().take(MAX_SUMMARY_CHARS).collect()) + .unwrap_or_default() +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::protocol::ReportBlockInput as B; + + fn input(blocks: Vec) -> ReportInput { + ReportInput { title: " Notes ".into(), summary: None, target: None, blocks } + } + + fn anchor(path: &str, line: i64, end: Option) -> Option { + Some(ReportAnchor { path: path.into(), side: None, line, end_line: end }) + } + + #[test] + fn resolves_files_and_drops_unplaceable_anchors() { + let files = HashMap::from([("p/1.png".to_owned(), ReportFile { key: "k1".into(), type_: "image/png".into() })]); + let shown = |path: &str, _: Side| path == "a.txt"; + let v = validate( + input(vec![ + B::Markdown { text: "\n# Hello world\nmore".into(), anchor: anchor("a.txt", 3, Some(3)) }, + B::Markdown { text: "x".into(), anchor: anchor("gone.txt", 1, None) }, + B::Pages { + label: Some(" Rendered ".into()), + pages: vec![ReportPageInput { + label: "p. 1".into(), + old: None, + new: Some("p/1.png".into()), + anchor: anchor("a.txt", 0, None), + }], + }, + ]), + &files, + shown, + ) + .unwrap(); + assert_eq!((v.title.as_str(), v.summary.as_str()), ("Notes", "Hello world")); + assert_eq!(v.notes.len(), 2, "{:?}", v.notes); + let ReportBlock::Markdown { anchor: Some(a), .. } = &v.blocks[0] else { panic!("{:?}", v.blocks[0]) }; + assert_eq!((a.line, a.end_line), (3, None)); + let ReportBlock::Markdown { anchor: None, .. } = &v.blocks[1] else { panic!("{:?}", v.blocks[1]) }; + let ReportBlock::Pages { label, pages } = &v.blocks[2] else { panic!("{:?}", v.blocks[2]) }; + assert_eq!(label.as_deref(), Some("Rendered")); + assert_eq!(pages[0].new.as_ref().unwrap().key, "k1"); + assert!(pages[0].anchor.is_none()); + } + + #[test] + fn refuses_what_it_cannot_serve() { + let files = HashMap::new(); + let shown = |_: &str, _: Side| true; + let page = |path: &str| B::Pages { + label: None, + pages: vec![ReportPageInput { label: "p".into(), old: Some(path.into()), new: None, anchor: None }], + }; + let cases: Vec<(ReportInput, &str)> = vec![ + (ReportInput { title: " ".into(), summary: None, target: None, blocks: vec![] }, "title: required"), + (input(vec![page("../x.png")]), "relative path"), + (input(vec![page("x.png")]), "no file uploaded"), + (input(vec![B::Markdown { text: "x".repeat(MAX_MARKDOWN_BYTES + 1), anchor: None }]), "at most"), + (input(vec![B::Pages { label: None, pages: vec![] }; MAX_BLOCKS + 1]), "blocks: at most"), + ]; + for (report, needle) in cases { + let err = validate(report, &files, shown).unwrap_err().to_string(); + assert!(err.contains(needle), "{err}"); + } + for id in ["ok-1", "a"] { + check_id(id).unwrap(); + } + for id in ["", "A", "-a", "a_b", &"a".repeat(65)] { + assert!(check_id(id).is_err(), "{id}"); + } + assert!(safe_relative("a/b.png") && !safe_relative("/a") && !safe_relative("a/../b") && !safe_relative("a\\b")); + } +} diff --git a/crates/diffle/src/reports/mod.rs b/crates/diffle/src/reports/mod.rs new file mode 100644 index 00000000..ac3a4cc8 --- /dev/null +++ b/crates/diffle/src/reports/mod.rs @@ -0,0 +1,221 @@ +//! Reports: documents attached to a snapshot, made by the generators diffle runs for it or pushed +//! over the API by something outside, an agent say. One view in the client shows them all. +pub mod catalog; +mod exec; +mod generated; +pub mod input; +mod pushed; +pub mod runner; +mod sides; +pub mod store; + +use std::collections::{BTreeMap, HashMap, HashSet}; +use std::path::Path; +use std::sync::{Arc, Mutex, Weak}; + +use sha1::{Digest, Sha1}; +use tokio::task::AbortHandle; + +use crate::error::{Error, Result}; +use crate::protocol::{Report, ReportFile, ReportTarget, ReportsResponse, Snapshot}; +use crate::session::{Broadcaster, Session}; +use catalog::Plugin; +use runner::Produced; +use store::{ReportStore, Stored}; + +struct State { + /// The newest snapshot seen, which generated reports answer for. + version: i64, + /// The comment key `pushed` was loaded for. + key: Option, + generated: BTreeMap, + pushed: BTreeMap, + /// Files uploaded for a report not put yet, by id then authored path. + uploads: HashMap>, + /// What each generator wrote to stderr on its last run. + logs: HashMap>, +} + +/// Every report of the active review. Generated ones follow the snapshot: each new one starts +/// the generators whose globs match a changed file, a run still going for an older snapshot is +/// killed, and a result commits only while its snapshot is the newest. Pushed ones persist under +/// the review's comment key and go stale, never away, when the snapshot moves. +pub struct Reports { + session: Arc, + hub: Arc, + store: ReportStore, + plugins: Vec, + state: Mutex, + /// The run in flight per generator. + runs: Mutex>, + /// Results by generator and input, so stepping back to a snapshot seen before is instant. + cache: Mutex>>, +} + +/// The snapshot a report answers for: its endpoints, plus the changed files' blobs so a worktree +/// side that moved on is told apart. +pub fn target_of(snap: &Snapshot) -> ReportTarget { + let mut hasher = Sha1::new(); + hasher.update(snap.old_sha.as_bytes()); + hasher.update(b"\0"); + hasher.update(snap.new_sha.as_bytes()); + for f in &snap.changed { + for part in [&f.path, &f.status.to_string(), &f.blob, &f.old_blob] { + hasher.update(b"\0"); + hasher.update(part.as_bytes()); + } + } + ReportTarget { + old_sha: snap.old_sha.clone(), + new_sha: snap.new_sha.clone(), + digest: hex::encode(hasher.finalize()), + } +} + +pub fn same_target(a: &ReportTarget, b: &ReportTarget) -> bool { + a.old_sha == b.old_sha && a.new_sha == b.new_sha && a.digest == b.digest +} + +impl Reports { + /// Reports for `session`, kept under `git_dir`, with `plugins` run on every snapshot from now on. + pub fn new(session: Arc, hub: Arc, git_dir: &Path, plugins: Vec) -> Arc { + let plugins = plugins + .into_iter() + .filter(|p| { + let ok = input::check_id(&p.name).is_ok(); + if !ok { + crate::status!( + "[diffle] ignoring plugin \"{}\": a name is lowercase letters, digits and dashes", + p.name + ); + } + ok + }) + .collect(); + let this = Arc::new(Self { + session: session.clone(), + hub, + store: ReportStore::new(git_dir), + plugins, + state: Mutex::new(State { + version: 0, + key: None, + generated: BTreeMap::new(), + pushed: BTreeMap::new(), + uploads: HashMap::new(), + logs: HashMap::new(), + }), + runs: Mutex::new(HashMap::new()), + cache: Mutex::new(HashMap::new()), + }); + let weak: Weak = Arc::downgrade(&this); + session.on_snapshot(move |snap| { + if let Some(reports) = weak.upgrade() { + let snap = snap.clone(); + tokio::spawn(async move { reports.sync(snap).await }); + } + }); + this + } + + pub fn plugins(&self) -> &[Plugin] { + &self.plugins + } + + /// The current snapshot and its target; the pushed reports are loaded for its key first. + async fn current(&self) -> Result<(Arc, ReportTarget)> { + let snap = self.session.snapshotter()?.current().await?; + self.load_pushed(&snap).await?; + let target = target_of(&snap); + Ok((snap, target)) + } + + async fn load_pushed(&self, snap: &Snapshot) -> Result<()> { + if self.state.lock().unwrap().key.as_deref() == Some(&snap.comment_key) { + return Ok(()); + } + let pushed = self.store.load(&snap.comment_key).await?; + let mut state = self.state.lock().unwrap(); + state.key = Some(snap.comment_key.clone()); + state.pushed = pushed.into_iter().map(|r| (r.id.clone(), r)).collect(); + Ok(()) + } + + fn served(stored: &Stored, current: &ReportTarget) -> Report { + Report { + id: stored.id.clone(), + source: stored.source, + title: stored.title.clone(), + summary: stored.summary.clone(), + target: stored.target.clone(), + status: stored.status, + error: stored.error.clone(), + stale: !same_target(&stored.target, current), + updated_at: stored.updated_at, + blocks: stored.blocks.clone(), + } + } + + pub async fn list(&self) -> Result { + let (_, current) = self.current().await?; + let state = self.state.lock().unwrap(); + let reports = + state.generated.values().chain(state.pushed.values()).map(|r| Self::served(r, ¤t)).collect(); + Ok(ReportsResponse { current, reports }) + } + + pub async fn current_target(&self) -> Result { + Ok(self.current().await?.1) + } + + pub async fn get(&self, id: &str) -> Result { + let (_, current) = self.current().await?; + let state = self.state.lock().unwrap(); + let stored = + state.generated.get(id).or_else(|| state.pushed.get(id)).ok_or_else(|| Error::NotFound(id.into()))?; + Ok(Self::served(stored, ¤t)) + } + + /// What the report's maker wrote to stderr, or what validation dropped. + pub fn log(&self, id: &str) -> Result> { + let state = self.state.lock().unwrap(); + if !state.generated.contains_key(id) && !state.pushed.contains_key(id) { + return Err(Error::NotFound(id.into())); + } + Ok(state.logs.get(id).cloned().unwrap_or_default()) + } + + /// The bytes report `id` names under `key`, or none: a key the report does not name serves nothing. + pub async fn file(&self, id: &str, key: &str) -> Result, &'static str)>> { + let named = { + let state = self.state.lock().unwrap(); + let report = state.generated.get(id).or_else(|| state.pushed.get(id)); + report.is_some_and(|r| r.keys().any(|k| k == key)) + }; + if !named { + return Ok(None); + } + self.store.read_blob(key).await + } + + /// Drops blobs nothing names any more; a failure here costs disk, not correctness. + async fn sweep(&self) { + let keep: HashSet = { + let state = self.state.lock().unwrap(); + let kept = state.generated.values().chain(state.pushed.values()).flat_map(Stored::keys).map(str::to_owned); + kept.chain(state.uploads.values().flatten().map(|(_, f)| f.key.clone())).collect() + }; + if let Err(e) = self.store.sweep(&keep, runner::RUN_TIMEOUT).await { + crate::status!("[diffle] sweeping report files: {e:#}"); + } + } +} + +fn block_keys(block: &crate::protocol::ReportBlock) -> Vec<&str> { + match block { + crate::protocol::ReportBlock::Markdown { .. } => Vec::new(), + crate::protocol::ReportBlock::Pages { pages, .. } => { + pages.iter().flat_map(|p| [p.old.as_ref(), p.new.as_ref()]).flatten().map(|f| f.key.as_str()).collect() + } + } +} diff --git a/crates/diffle/src/reports/pushed.rs b/crates/diffle/src/reports/pushed.rs new file mode 100644 index 00000000..2d9d49a7 --- /dev/null +++ b/crates/diffle/src/reports/pushed.rs @@ -0,0 +1,84 @@ +//! Reports pushed over the API: files uploaded ahead, the put that resolves them, and removal. +use std::collections::HashSet; + +use super::store::Stored; +use super::{Reports, block_keys, input}; +use crate::error::{Error, Result}; +use crate::iterations::now_ms; +use crate::protocol::{Report, ReportFile, ReportInput, ReportSource, ReportStatus, ServerMessage, Side}; + +impl Reports { + /// Keeps `bytes` for the next put of report `id`, under the authored `path`. + pub async fn upload(&self, id: &str, path: &str, bytes: &[u8]) -> Result { + input::check_id(id)?; + if !input::safe_relative(path) { + return Err(Error::Invalid(format!("path: a relative path, got \"{path}\""))); + } + let file = self.store.put_blob(bytes).await?; + self.state.lock().unwrap().uploads.entry(id.to_owned()).or_default().insert(path.to_owned(), file.clone()); + Ok(file) + } + + /// Creates or replaces the pushed report `id` for the current snapshot. True when created. + pub async fn put(&self, id: &str, report: ReportInput) -> Result<(Report, bool)> { + input::check_id(id)?; + let (snap, current) = self.current().await?; + let (files, created) = { + let state = self.state.lock().unwrap(); + if state.generated.contains_key(id) { + return Err(Error::Conflict(format!("\"{id}\" is a generator's report; pick another id"))); + } + let mut files = state.pushed.get(id).map(|r| r.files.clone()).unwrap_or_default(); + // Uploads outlive a refused put: the caller fixes the report, not the files. + files.extend(state.uploads.get(id).cloned().unwrap_or_default()); + (files, !state.pushed.contains_key(id)) + }; + let session = self.session.clone(); + let shown = |path: &str, side: Side| -> bool { session.has_side(&snap, path, side) }; + let valid = input::validate(report, &files, shown)?; + let named: HashSet<&str> = valid.blocks.iter().flat_map(block_keys).collect(); + let files = files.into_iter().filter(|(_, f)| named.contains(f.key.as_str())).collect(); + let stored = Stored { + id: id.to_owned(), + source: ReportSource::Pushed, + title: valid.title, + summary: valid.summary, + target: current.clone(), + status: ReportStatus::Ready, + error: None, + updated_at: now_ms(), + blocks: valid.blocks, + files, + }; + let served = Self::served(&stored, ¤t); + let pushed: Vec = { + let mut state = self.state.lock().unwrap(); + state.uploads.remove(id); + state.logs.insert(id.to_owned(), valid.notes); + state.pushed.insert(id.to_owned(), stored); + state.pushed.values().cloned().collect() + }; + self.store.save(&snap.comment_key, &pushed).await?; + self.sweep().await; + self.hub.broadcast(ServerMessage::Reports); + Ok((served, created)) + } + + /// Removes the pushed report `id`; a generator's cannot go. + pub async fn delete(&self, id: &str) -> Result<()> { + let (snap, _) = self.current().await?; + let pushed: Vec = { + let mut state = self.state.lock().unwrap(); + if state.generated.contains_key(id) { + return Err(Error::Conflict(format!("\"{id}\" is a generator's report; turn the plugin off instead"))); + } + state.pushed.remove(id).ok_or_else(|| Error::NotFound(id.into()))?; + state.logs.remove(id); + state.pushed.values().cloned().collect() + }; + self.store.save(&snap.comment_key, &pushed).await?; + self.sweep().await; + self.hub.broadcast(ServerMessage::Reports); + Ok(()) + } +} diff --git a/crates/diffle/src/reports/runner.rs b/crates/diffle/src/reports/runner.rs new file mode 100644 index 00000000..54a65ae4 --- /dev/null +++ b/crates/diffle/src/reports/runner.rs @@ -0,0 +1,217 @@ +//! Running one generator for one snapshot: copies of the sides in a temporary folder, the program +//! started on them with the review on stdin, and what it wrote read back as a report. +use std::collections::HashMap; +use std::path::Path; +use std::sync::Arc; +use std::time::Duration; + +use serde_json::json; + +use super::catalog::{Plugin, Scope}; +use super::exec::{Vars, pipeline, stages_of}; +use super::input::{self, safe_relative}; +use super::sides::materialize; +use super::store::ReportStore; +use crate::error::{Error, Result}; +use crate::protocol::{ + ChangeStatus, ChangedFile, ReportAnchor, ReportBlock, ReportFile, ReportInput, ReportTarget, Side, Snapshot, +}; +use crate::session::Session; + +/// How long one generator may take for one snapshot. +pub const RUN_TIMEOUT: Duration = Duration::from_secs(600); +pub(super) const FILE_VAR: &str = "DIFFLE_FILE"; + +/// What a run made: a report's parts and the files it named, already stored. +pub struct Produced { + pub title: String, + pub summary: String, + pub blocks: Vec, + pub files: HashMap, + pub log: Vec, +} + +/// Why a run made no report, with what the program said on the way. +#[derive(Debug)] +pub struct Failure { + pub message: String, + pub log: Vec, +} + +impl From for Failure { + fn from(e: Error) -> Self { + Self { message: e.to_string(), log: Vec::new() } + } +} + +impl From for Failure { + fn from(e: std::io::Error) -> Self { + Self { message: e.to_string(), log: Vec::new() } + } +} + +pub(super) fn failed(message: impl Into) -> Error { + Error::Invalid(message.into()) +} + +/// Runs `plugin` for `snap`, whose changed files `matched` are the ones its globs chose. +pub async fn run( + session: &Arc, + store: &ReportStore, + snap: &Arc, + target: &ReportTarget, + plugin: &Plugin, + matched: &[String], +) -> Result { + let stages = stages_of(&plugin.command)?; + let dir = tempfile::Builder::new().prefix("diffle-report-").tempdir()?; + let (old, new, out) = (dir.path().join("old"), dir.path().join("new"), dir.path().join("out")); + materialize(session, snap, plugin.scope, matched, &old, &new).await?; + let per_file = + plugin.command.contains(&format!("${FILE_VAR}")) || plugin.command.contains(&format!("${{{FILE_VAR}}}")); + let runs: Vec> = if per_file { matched.iter().cloned().map(Some).collect() } else { vec![None] }; + let mut produced = Produced { + title: plugin.name.clone(), + summary: String::new(), + blocks: Vec::new(), + files: HashMap::new(), + log: Vec::new(), + }; + let changed: Vec<&ChangedFile> = snap.changed.iter().filter(|f| matched.contains(&f.path)).collect(); + for (n, file) in runs.iter().enumerate() { + let run_out = out.join(n.to_string()); + tokio::fs::create_dir_all(&run_out).await?; + let vars = Vars { + root: snap.root.clone(), + old: old.clone(), + new: new.clone(), + out: run_out.clone(), + file: file.clone(), + }; + let stdin = json!({ + "version": 1, + "review": snap.review, + "target": target, + "root": vars.root, + "old": vars.old, + "new": vars.new, + "out": vars.out, + "file": file, + "files": changed.iter().map(|f| json!({ "path": f.path, "oldPath": f.old_path, "status": f.status })).collect::>(), + "scope": if plugin.scope == Scope::Tree { "tree" } else { "changed" }, + }); + let output = + match tokio::time::timeout(RUN_TIMEOUT, pipeline(&stages, &vars, stdin.to_string().into_bytes())).await { + Ok(Ok(output)) => output, + Ok(Err(mut failure)) => { + produced.log.append(&mut failure.log); + return Err(Failure { message: failure.message, log: produced.log }); + } + Err(_) => { + return Err(Failure { + message: format!("timed out after {}s", RUN_TIMEOUT.as_secs()), + log: produced.log, + }); + } + }; + produced.log.extend(output.log); + let (anchor, heading) = match file.as_deref().and_then(|f| changed.iter().find(|c| c.path == f)) { + Some(f) => { + let side = if f.status == ChangeStatus::D { Side::Old } else { Side::New }; + ( + Some(ReportAnchor { path: f.path.clone(), side: Some(side), line: 1, end_line: None }), + Some(f.path.clone()), + ) + } + None => (None, None), + }; + if let Err(e) = collect(session, store, snap, &run_out, output.stdout, anchor, heading, &mut produced).await { + return Err(Failure { message: e.to_string(), log: produced.log }); + } + } + if produced.blocks.is_empty() { + return Err(Failure { + message: "the program wrote neither a report.json nor anything to stdout".into(), + log: produced.log, + }); + } + if produced.summary.is_empty() { + produced.summary = produced + .blocks + .iter() + .find_map(|b| match b { + ReportBlock::Markdown { text, .. } => text + .lines() + .map(str::trim) + .find(|l| !l.is_empty()) + .map(|l| l.trim_start_matches('#').trim().to_owned()), + ReportBlock::Pages { .. } => None, + }) + .unwrap_or_default(); + } + Ok(produced) +} + +/// Reads one run's output: `report.json` with its files, else stdout as one markdown block. +#[allow(clippy::too_many_arguments)] +async fn collect( + session: &Arc, + store: &ReportStore, + snap: &Arc, + out: &Path, + stdout: Vec, + anchor: Option, + heading: Option, + produced: &mut Produced, +) -> Result<()> { + let report_file = out.join("report.json"); + let Ok(text) = tokio::fs::read_to_string(&report_file).await else { + let text = String::from_utf8_lossy(&stdout).trim().to_owned(); + if !text.is_empty() { + let text = match heading { + Some(path) => format!("### {path}\n\n{text}"), + None => text, + }; + produced.blocks.push(ReportBlock::Markdown { text, anchor }); + } + return Ok(()); + }; + let report: ReportInput = serde_json::from_str(&text).map_err(|e| failed(format!("report.json: {e}")))?; + let mut files = HashMap::new(); + for path in named_files(&report) { + if !safe_relative(&path) { + return Err(failed(format!( + "report.json names \"{path}\": a relative path under the output folder is expected" + ))); + } + let bytes = + tokio::fs::read(out.join(&path)).await.map_err(|e| failed(format!("report.json names \"{path}\": {e}")))?; + let file = store.put_blob(&bytes).await.map_err(|e| failed(format!("\"{path}\": {e}")))?; + files.insert(path, file); + } + let shown = |path: &str, side: Side| -> bool { session.has_side(snap, path, side) }; + let valid = input::validate(report, &files, shown)?; + if produced.blocks.is_empty() { + produced.title = valid.title; + produced.summary = valid.summary; + } + produced.blocks.extend(valid.blocks); + produced.files.extend(files); + produced.log.extend(valid.notes); + Ok(()) +} + +fn named_files(report: &ReportInput) -> Vec { + use crate::protocol::ReportBlockInput; + let mut names = Vec::new(); + for block in &report.blocks { + if let ReportBlockInput::Pages { pages, .. } = block { + for page in pages { + names.extend(page.old.iter().chain(page.new.iter()).cloned()); + } + } + } + names.sort(); + names.dedup(); + names +} diff --git a/crates/diffle/src/reports/sides.rs b/crates/diffle/src/reports/sides.rs new file mode 100644 index 00000000..50384a45 --- /dev/null +++ b/crates/diffle/src/reports/sides.rs @@ -0,0 +1,71 @@ +//! The copies of the sides a generator reads: the matched changed files, or the whole snapshot, +//! each side as the review shows it. +use std::collections::HashMap; +use std::path::{Path, PathBuf}; +use std::sync::Arc; + +use futures::{StreamExt, TryStreamExt, stream}; + +use super::catalog::Scope; +use super::input::safe_relative; +use crate::error::{Error, Result}; +use crate::protocol::{ChangeStatus, ChangedFile, Side, Snapshot}; +use crate::session::Session; + +/// Concurrent side reads while copying. +const COPY_CONCURRENCY: usize = 8; + +/// Writes the files a generator gets into `old` and `new`: the matched changed files, or the +/// whole snapshot, each side as the review shows it. Ignored files are absent, as everywhere. +pub(super) async fn materialize( + session: &Arc, + snap: &Arc, + scope: Scope, + matched: &[String], + old: &Path, + new: &Path, +) -> Result<()> { + let by_path: HashMap<&str, &ChangedFile> = snap.changed.iter().map(|f| (f.path.as_str(), f)).collect(); + let new_paths: Vec = match scope { + Scope::Changed => matched + .iter() + .filter(|p| by_path.get(p.as_str()).is_none_or(|f| f.status != ChangeStatus::D)) + .cloned() + .collect(), + Scope::Tree => snap.tree.clone(), + }; + // Old side: every path read by its new name, written under its old one; deleted files too. + let mut old_paths: Vec<(String, String)> = match scope { + Scope::Changed => matched.to_vec(), + Scope::Tree => snap.tree.clone(), + } + .into_iter() + .filter(|p| by_path.get(p.as_str()).is_none_or(|f| f.status != ChangeStatus::A)) + .map(|p| { + let target = by_path.get(p.as_str()).and_then(|f| f.old_path.clone()).unwrap_or_else(|| p.clone()); + (p, target) + }) + .collect(); + if scope == Scope::Tree { + old_paths.extend( + snap.changed.iter().filter(|f| f.status == ChangeStatus::D).map(|f| (f.path.clone(), f.path.clone())), + ); + } + let copy = |side: Side, read_as: String, write_as: String, dir: PathBuf| async move { + if !safe_relative(&write_as) { + return Ok::<(), Error>(()); + } + let Some(bytes) = session.read_side(snap, &read_as, side).await? else { return Ok(()) }; + let file = dir.join(&write_as); + if let Some(parent) = file.parent() { + tokio::fs::create_dir_all(parent).await?; + } + tokio::fs::write(&file, bytes).await?; + Ok(()) + }; + let news = new_paths.into_iter().map(|p| copy(Side::New, p.clone(), p, new.to_path_buf())); + stream::iter(news).buffer_unordered(COPY_CONCURRENCY).try_collect::>().await?; + let olds = old_paths.into_iter().map(|(read_as, write_as)| copy(Side::Old, read_as, write_as, old.to_path_buf())); + stream::iter(olds).buffer_unordered(COPY_CONCURRENCY).try_collect::>().await?; + Ok(()) +} diff --git a/crates/diffle/src/reports/store.rs b/crates/diffle/src/reports/store.rs new file mode 100644 index 00000000..8574103d --- /dev/null +++ b/crates/diffle/src/reports/store.rs @@ -0,0 +1,170 @@ +//! What a report keeps on disk: pushed reports in `/diffle/reports.json`, under the +//! review's comment key, and every report's files as content-addressed blobs beside it. +use std::collections::{BTreeMap, HashMap, HashSet}; +use std::path::{Path, PathBuf}; +use std::time::Duration; + +use serde::{Deserialize, Serialize}; +use sha1::{Digest, Sha1}; + +use crate::error::{Error, Result}; +use crate::persist::{with_file_lock, write_file_atomic}; +use crate::protocol::{ReportBlock, ReportFile, ReportSource, ReportStatus, ReportTarget}; +use crate::server::image::image_type; + +/// The largest file a report may name. +pub const MAX_FILE_BYTES: usize = 25 * 1024 * 1024; + +/// A report as kept: what the client gets, minus `stale`, plus the files by their authored +/// path, so a report put again may name the same files without uploading them again. +#[derive(Clone, Debug, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +pub struct Stored { + pub id: String, + pub source: ReportSource, + pub title: String, + pub summary: String, + pub target: ReportTarget, + pub status: ReportStatus, + pub error: Option, + pub updated_at: i64, + pub blocks: Vec, + #[serde(default)] + pub files: HashMap, +} + +impl Stored { + /// Every blob key the report's pages name. + pub fn keys(&self) -> impl Iterator { + self.blocks.iter().flat_map(|b| match b { + ReportBlock::Markdown { .. } => Vec::new(), + ReportBlock::Pages { pages, .. } => { + pages.iter().flat_map(|p| [p.old.as_ref(), p.new.as_ref()]).flatten().map(|f| f.key.as_str()).collect() + } + }) + } +} + +#[derive(Serialize, Deserialize)] +struct StoreFile { + version: u8, + /// Pushed reports by comment key. + sets: BTreeMap>, +} + +/// Owns `reports.json` and the blob folder. Nothing is cached: every mutation is a read-modify-write +/// under the file lock, touching only its own key, so two servers on one repository keep each +/// other's reports. +pub struct ReportStore { + file: PathBuf, + blobs: PathBuf, +} + +impl ReportStore { + pub fn new(git_dir: &Path) -> Self { + let dir = git_dir.join("diffle"); + Self { file: dir.join("reports.json"), blobs: dir.join("reports") } + } + + /// The pushed reports under `key`, as on disk now. + pub async fn load(&self, key: &str) -> Result> { + Ok(read_store_file(&self.file).await?.sets.remove(key).unwrap_or_default()) + } + + /// Replaces the pushed reports under `key`. + pub async fn save(&self, key: &str, reports: &[Stored]) -> Result<()> { + with_file_lock(&self.file, || async { + let mut all = read_store_file(&self.file).await?; + if reports.is_empty() { + all.sets.remove(key); + } else { + all.sets.insert(key.to_owned(), reports.to_vec()); + } + write_file_atomic(&self.file, &serde_json::to_string_pretty(&all)?).await + }) + .await?; + Ok(()) + } + + /// Keeps `bytes` under their digest, when they are a raster image. The key is the digest, so + /// the same bytes land once however often they are put. + pub async fn put_blob(&self, bytes: &[u8]) -> Result { + let Some(type_) = image_type(bytes) else { + return Err(Error::Invalid("not a raster image (png, jpeg, gif, webp, avif, bmp)".into())); + }; + if bytes.len() > MAX_FILE_BYTES { + return Err(Error::Invalid(format!("at most {} bytes per file", MAX_FILE_BYTES))); + } + let key = hex::encode(Sha1::digest(bytes)); + let file = self.blobs.join(&key); + if tokio::fs::metadata(&file).await.is_err() { + tokio::fs::create_dir_all(&self.blobs).await?; + let tmp = self.blobs.join(format!("{key}.{}.tmp", uuid::Uuid::new_v4().simple())); + tokio::fs::write(&tmp, bytes).await?; + if let Err(e) = tokio::fs::rename(&tmp, &file).await { + let _ = tokio::fs::remove_file(&tmp).await; + return Err(e.into()); + } + } + Ok(ReportFile { key, type_: type_.to_owned() }) + } + + /// The bytes under `key` and their type, or none. + pub async fn read_blob(&self, key: &str) -> Result, &'static str)>> { + if key.len() != 40 || !key.bytes().all(|b| b.is_ascii_hexdigit()) { + return Ok(None); + } + match tokio::fs::read(self.blobs.join(key)).await { + Ok(bytes) => Ok(image_type(&bytes).map(|t| (bytes, t))), + Err(e) if e.kind() == std::io::ErrorKind::NotFound => Ok(None), + Err(e) => Err(e.into()), + } + } + + /// Removes every blob no pushed report on disk and nothing in `keep` names. A blob younger + /// than `grace` stays: a run still going may have written it and not named it yet. + pub async fn sweep(&self, keep: &HashSet, grace: Duration) -> Result<()> { + let mut named: HashSet = keep.clone(); + for report in read_store_file(&self.file).await?.sets.values().flatten() { + named.extend(report.keys().map(str::to_owned)); + } + let mut entries = match tokio::fs::read_dir(&self.blobs).await { + Ok(entries) => entries, + Err(e) if e.kind() == std::io::ErrorKind::NotFound => return Ok(()), + Err(e) => return Err(e.into()), + }; + while let Some(entry) = entries.next_entry().await? { + let name = entry.file_name().to_string_lossy().into_owned(); + let young = entry + .metadata() + .await + .and_then(|m| m.modified()) + .map(|at| at.elapsed().unwrap_or_default() < grace) + .unwrap_or(true); + // A `..tmp` is a put in flight; only a finished blob is ever swept. + let blob = name.len() == 40 && name.bytes().all(|b| b.is_ascii_hexdigit()); + if blob && !named.contains(&name) && !young { + let _ = tokio::fs::remove_file(entry.path()).await; + } + } + Ok(()) + } +} + +/// The file as stored, or an empty store when absent or of another format. Malformed JSON fails. +async fn read_store_file(file: &Path) -> anyhow::Result { + let text = match tokio::fs::read_to_string(file).await { + Ok(text) => text, + Err(e) if e.kind() == std::io::ErrorKind::NotFound => return Ok(empty()), + Err(e) => return Err(e.into()), + }; + let value: serde_json::Value = serde_json::from_str(&text)?; + if value.get("version").and_then(serde_json::Value::as_u64) != Some(1) || !value["sets"].is_object() { + return Ok(empty()); + } + Ok(serde_json::from_value(value)?) +} + +fn empty() -> StoreFile { + StoreFile { version: 1, sets: BTreeMap::new() } +} diff --git a/crates/diffle/src/server/mod.rs b/crates/diffle/src/server/mod.rs index 795b1e3d..3dd9f87d 100644 --- a/crates/diffle/src/server/mod.rs +++ b/crates/diffle/src/server/mod.rs @@ -32,6 +32,8 @@ pub const DEFAULT_PORT: u16 = 4966; const PROBE_PORTS: u16 = 100; /// JSON bodies are small (comments, viewed marks, config); anything bigger is not a client. const MAX_BODY_BYTES: usize = 1024 * 1024; +/// Reports carry rendered pages: their routes take a file's worth. +const MAX_REPORT_BYTES: usize = crate::reports::store::MAX_FILE_BYTES; pub struct ServerOptions { pub port: u16, @@ -49,6 +51,7 @@ pub struct ApiDeps { pub hub: Arc, /// The language servers, unless the run disabled them with `--no-lsp`. pub lsp: Option>, + pub reports: Arc, } struct App { @@ -145,6 +148,7 @@ async fn check_guard(State(app): State>, req: Request, next: Next) -> R /// Reads the body under the cap. An oversized upload is drained before the reply: closing /// mid-write gives clients EPIPE, not 413. async fn limit_body(req: Request, next: Next) -> Response { + let cap = if req.uri().path().starts_with("/api/reports/") { MAX_REPORT_BYTES } else { MAX_BODY_BYTES }; let (parts, mut body) = req.into_parts(); let mut kept = Vec::new(); let mut size = 0; @@ -152,12 +156,12 @@ async fn limit_body(req: Request, next: Next) -> Response { let Ok(frame) = frame else { return ApiError::bad_request("unreadable request body").into_response() }; if let Some(data) = frame.data_ref() { size += data.len(); - if size <= MAX_BODY_BYTES { + if size <= cap { kept.extend_from_slice(data); } } } - if size > MAX_BODY_BYTES { + if size > cap { return ApiError(StatusCode::PAYLOAD_TOO_LARGE, "request body too large".into()).into_response(); } next.run(Request::from_parts(parts, Body::from(Bytes::from(kept)))).await diff --git a/crates/diffle/src/server/routes/mod.rs b/crates/diffle/src/server/routes/mod.rs index 9cbb0207..d8c443b4 100644 --- a/crates/diffle/src/server/routes/mod.rs +++ b/crates/diffle/src/server/routes/mod.rs @@ -3,6 +3,7 @@ mod api; mod comments; mod files; mod lsp; +mod reports; use std::sync::Arc; @@ -16,7 +17,12 @@ use super::ApiDeps; use crate::error::Error; pub fn router() -> Router> { - Router::new().merge(api::routes()).merge(files::routes()).merge(comments::routes()).merge(lsp::routes()) + Router::new() + .merge(api::routes()) + .merge(files::routes()) + .merge(comments::routes()) + .merge(lsp::routes()) + .merge(reports::routes()) } /// An error as the API answers it: a status and `{ "error": message }`. @@ -40,7 +46,7 @@ impl From for ApiError { Error::NotFound(_) => Self::not_found("not found"), Error::Revspec(m) | Error::Invalid(m) | Error::Unquotable(m) => Self::bad_request(m), Error::Git(e) => Self::bad_request(e.message), - Error::LspUnavailable(m) | Error::LspUnsupported(m) => Self(StatusCode::CONFLICT, m), + Error::LspUnavailable(m) | Error::LspUnsupported(m) | Error::Conflict(m) => Self(StatusCode::CONFLICT, m), Error::Github(e) => Self(StatusCode::from_u16(e.status).unwrap_or(StatusCode::BAD_GATEWAY), e.message), Error::Other(e) => { crate::status!("[diffle] {e:#}"); diff --git a/crates/diffle/src/server/routes/reports.rs b/crates/diffle/src/server/routes/reports.rs new file mode 100644 index 00000000..560ceb30 --- /dev/null +++ b/crates/diffle/src/server/routes/reports.rs @@ -0,0 +1,74 @@ +//! Reports: listing, pushing, deleting, and the files and logs behind them. +use std::sync::Arc; + +use axum::Json; +use axum::Router; +use axum::body::Bytes; +use axum::extract::{Path, RawQuery, State}; +use axum::http::{StatusCode, header}; +use axum::response::IntoResponse; +use axum::routing::{get, put}; +use serde_json::json; + +use super::{ApiDeps, ApiError, ApiResult, body, query}; +use crate::protocol::{ReportFileQuery, ReportInput, ReportLogResponse}; +use crate::reports::same_target; + +pub fn routes() -> Router> { + Router::new() + .route("/api/reports", get(list)) + .route("/api/reports/{id}", get(one).put(put_report).delete(remove)) + .route("/api/reports/{id}/files/{*path}", put(upload)) + .route("/api/reports/{id}/file", get(file)) + .route("/api/reports/{id}/log", get(log)) +} + +async fn list(State(d): State>) -> ApiResult { + Ok(Json(d.reports.list().await?).into_response()) +} + +async fn one(State(d): State>, Path(id): Path) -> ApiResult { + Ok(Json(d.reports.get(&id).await?).into_response()) +} + +/// Creates (201) or replaces (200) a pushed report. A `target` that is not the current snapshot's +/// answers 409 with the current one, so the caller can refetch what it describes. +async fn put_report(State(d): State>, Path(id): Path, raw: Bytes) -> ApiResult { + let req: ReportInput = body(&raw)?; + if let Some(target) = &req.target { + let current = d.reports.current_target().await?; + if !same_target(target, ¤t) { + let error = "target: the snapshot moved on; fetch the review again and name the current target"; + return Ok((StatusCode::CONFLICT, Json(json!({ "error": error, "current": current }))).into_response()); + } + } + let (report, created) = d.reports.put(&id, req).await?; + let status = if created { StatusCode::CREATED } else { StatusCode::OK }; + Ok((status, Json(report)).into_response()) +} + +async fn remove(State(d): State>, Path(id): Path) -> ApiResult { + d.reports.delete(&id).await?; + Ok(StatusCode::NO_CONTENT.into_response()) +} + +/// A file the next put of report `id` may name as `path`: raw bytes, a raster image. +async fn upload(State(d): State>, Path((id, path)): Path<(String, String)>, raw: Bytes) -> ApiResult { + let file = d.reports.upload(&id, &path, &raw).await?; + Ok((StatusCode::CREATED, Json(file)).into_response()) +} + +async fn file(State(d): State>, Path(id): Path, RawQuery(raw): RawQuery) -> ApiResult { + let q: ReportFileQuery = query(raw.as_deref())?; + match d.reports.file(&id, &q.key).await? { + Some((bytes, kind)) => { + Ok(([(header::CONTENT_TYPE, kind), (header::CACHE_CONTROL, "private, max-age=31536000, immutable")], bytes) + .into_response()) + } + None => Err(ApiError::not_found("no such file in the report")), + } +} + +async fn log(State(d): State>, Path(id): Path) -> ApiResult { + Ok(Json(ReportLogResponse { lines: d.reports.log(&id)? }).into_response()) +} diff --git a/crates/diffle/src/user_config.rs b/crates/diffle/src/user_config.rs index 1145c987..07f948c6 100644 --- a/crates/diffle/src/user_config.rs +++ b/crates/diffle/src/user_config.rs @@ -6,7 +6,7 @@ use std::sync::Mutex; use serde_json::{Map, Value}; use crate::persist::{user_config_dir, write_file_atomic}; -use crate::protocol::{FollowRefs, LanguageId, UserConfig, language_ids}; +use crate::protocol::{FollowRefs, LanguageId, PluginConfig, UserConfig, language_ids}; /// Every field optional: what `set` merges in (routes' ConfigUpdate plus CLI-only lspCommands/githubHosts). #[derive(Default, Clone, Debug)] @@ -26,6 +26,7 @@ impl Default for UserConfig { line_wrap: true, lsp_commands: HashMap::new(), github_hosts: HashMap::new(), + plugins: HashMap::new(), } } } @@ -117,6 +118,7 @@ fn normalize(value: &Value) -> UserConfig { line_wrap, lsp_commands: lsp_commands(c.get("lspCommands")), github_hosts: github_hosts(c.get("githubHosts")), + plugins: plugins(c.get("plugins")), } } @@ -138,6 +140,22 @@ fn lsp_commands(value: Option<&Value>) -> HashMap { out } +/// Report generators by name. An entry that is not an object with a `command` string is dropped +/// with a warning: a malformed one would otherwise sit there looking effective. +fn plugins(value: Option<&Value>) -> HashMap { + let mut out = HashMap::new(); + for (name, entry) in value.and_then(Value::as_object).into_iter().flatten() { + let name = name.trim(); + match serde_json::from_value::(entry.clone()) { + Ok(plugin) if !name.is_empty() => { + out.insert(name.to_owned(), plugin); + } + _ => crate::status!("[diffle] ignoring config plugins.{name}: expected an object with a command"), + } + } + out +} + /// Lowercased host names; an entry without a web host is dropped, it would point nowhere. fn github_hosts(value: Option<&Value>) -> HashMap { let mut out = HashMap::new(); diff --git a/crates/diffle/tests/cli.rs b/crates/diffle/tests/cli.rs index fbc4242e..bf67f4bc 100644 --- a/crates/diffle/tests/cli.rs +++ b/crates/diffle/tests/cli.rs @@ -221,3 +221,43 @@ fn reports_the_review_and_hands_off_the_open_comments_on_stdout_when_stopped() { } assert_eq!(prompt, "app.py:2\n\n> y = 2\n\nrename y\n\n---\n"); } + +// Stopped with SIGTERM, which Windows has no way to send. +#[cfg(unix)] +#[test] +fn pushes_a_report_into_the_running_review_and_withdraws_the_announcement() { + let t = one_file_repo(); + t.write("app.py", "x = 1\ny = 2\n"); + let config = tempfile::tempdir().unwrap(); + let repo = t.path.to_str().unwrap(); + let notes = config.path().join("Review Notes.md"); + std::fs::write(¬es, "# Watch y\n\nIt is unused.\n").unwrap(); + + let out = run(config.path(), &["report", "put", notes.to_str().unwrap(), "-C", repo]); + assert_eq!(out.status.code(), Some(2), "{}", stderr(&out)); + assert!(stderr(&out).contains("no diffle is running"), "{}", stderr(&out)); + + let (mut child, url, mut lines) = serve(config.path(), &t.path); + let announced = std::fs::read_to_string(t.path.join(".git/diffle/server.json")).unwrap(); + assert!(announced.contains(&url), "{announced}"); + let out = run(config.path(), &["report", "put", notes.to_str().unwrap(), "--anchor", "app.py:2", "-C", repo]); + assert!(out.status.success(), "{}", stderr(&out)); + assert!(stderr(&out).contains("report Watch y added (review-notes, "), "{}", stderr(&out)); + let (status, body) = http(&url, "GET", "/api/reports/review-notes", ""); + assert!(status.contains("200"), "{status}"); + let report: serde_json::Value = serde_json::from_str(&body).unwrap(); + assert_eq!(report["blocks"][0]["anchor"], serde_json::json!({ "path": "app.py", "line": 2 })); + assert_eq!(report["summary"], "Watch y"); + let out = run(config.path(), &["report", "put", notes.to_str().unwrap(), "--id", "Nope", "-C", repo]); + assert_eq!(out.status.code(), Some(2), "{}", stderr(&out)); + let out = run(config.path(), &["report", "delete", "review-notes", "-C", repo]); + assert!(out.status.success(), "{}", stderr(&out)); + let (status, _) = http(&url, "GET", "/api/reports/review-notes", ""); + assert!(status.contains("404"), "{status}"); + + Command::new("kill").args(["-TERM", &child.id().to_string()]).status().unwrap(); + let mut rest = String::new(); + lines.read_to_string(&mut rest).unwrap(); + assert!(child.wait().unwrap().success(), "{rest}"); + assert!(!t.path.join(".git/diffle/server.json").exists()); +} diff --git a/crates/diffle/tests/common/server.rs b/crates/diffle/tests/common/server.rs index 541fbdac..dfdf3325 100644 --- a/crates/diffle/tests/common/server.rs +++ b/crates/diffle/tests/common/server.rs @@ -4,6 +4,8 @@ use std::sync::Arc; use diffle::github::hosts::Github; use diffle::lsp::pool::{LspPool, PoolOptions}; use diffle::protocol::{EnterRequest, FollowRefs, LanguageId, Side}; +use diffle::reports::Reports; +use diffle::reports::catalog::Plugin; use diffle::server::hub::Hub; use diffle::server::{ApiDeps, Server, ServerOptions}; use diffle::session::{Session, SessionOptions}; @@ -63,6 +65,7 @@ pub struct TestServer { pub base: String, server: Option, pub lsp: Option>, + pub reports: Arc, client: reqwest::Client, } @@ -97,20 +100,36 @@ impl TestServer { repo: TestRepo, req: EnterRequest, overrides: Option>, + ) -> Self { + Self::with_lsp_and_plugins(repo, req, overrides, Vec::new()).await + } + + /// A server over `repo` whose reports come from `plugins`, run as a run would. + pub async fn with_plugins(repo: TestRepo, req: EnterRequest, plugins: Vec) -> Self { + Self::with_lsp_and_plugins(repo, req, None, plugins).await + } + + async fn with_lsp_and_plugins( + repo: TestRepo, + req: EnterRequest, + overrides: Option>, + plugins: Vec, ) -> Self { let git = Arc::new(repo.open().await); let hub = Hub::new(); let config = Arc::new(UserConfigStore::open(repo.path.join("cfg").join("config.json")).await); let opts = SessionOptions { watch: false, github: Github::tokenless(), context: 3, follow_refs: FollowRefs::Off }; - let session = Session::new(git, hub.clone(), opts); + let session = Session::new(git.clone(), hub.clone(), opts); let lsp = overrides.map(|overrides| start_lsp(&session, overrides)); + let reports = Reports::new(session.clone(), hub.clone(), &git.git_dir, plugins); let deps = ApiDeps { session: session.clone(), config: config.clone(), extra_auto_viewed: Vec::new(), hub: hub.clone(), lsp: lsp.clone(), + reports: reports.clone(), }; let opts = ServerOptions { port: 0, @@ -120,7 +139,7 @@ impl TestServer { }; let (server, base) = Server::listen(deps, opts).await.unwrap(); session.start(session.resolve(req).await.unwrap()).await.unwrap(); - Self { repo, session, config, hub, base, server: Some(server), lsp, client: reqwest::Client::new() } + Self { repo, session, config, hub, base, server: Some(server), lsp, reports, client: reqwest::Client::new() } } /// `127.0.0.1:` @@ -145,6 +164,13 @@ impl TestServer { Res { status: res.status().as_u16(), body: res.text().await.unwrap() } } + /// A PUT of raw bytes, as a file upload: the status and the body text. + pub async fn client_put_bytes(&self, path: &str, bytes: Vec) -> (u16, String) { + let url = format!("{}{}", self.base.trim_end_matches('/'), path); + let res = self.client.put(url).body(bytes).send().await.unwrap(); + (res.status().as_u16(), res.text().await.unwrap()) + } + /// A GET whose response the caller reads itself: headers and bytes. pub async fn fetch(&self, path: &str) -> reqwest::Response { self.client.get(format!("{}{}", self.base.trim_end_matches('/'), path)).send().await.unwrap() @@ -158,7 +184,13 @@ impl TestServer { self.send(method, path, &[], Some(body.to_string())).await } - pub async fn close(mut self) { + pub async fn close(self) { + self.stop().await; + } + + /// Closes the server and hands back its repository, for a second server over the same one. + pub async fn stop(mut self) -> TestRepo { + self.reports.close(); if let Some(server) = self.server.take() { server.close().await; } @@ -166,5 +198,6 @@ impl TestServer { lsp.close().await; } self.session.close().await; + self.repo } } diff --git a/crates/diffle/tests/reports.rs b/crates/diffle/tests/reports.rs new file mode 100644 index 00000000..cbea9010 --- /dev/null +++ b/crates/diffle/tests/reports.rs @@ -0,0 +1,247 @@ +mod common; + +use std::path::PathBuf; +use std::time::{Duration, Instant}; + +use common::TestRepo; +use common::server::TestServer; +use diffle::protocol::EnterRequest; +use diffle::reports::catalog::{Plugin, Scope}; +use futures::StreamExt; +use serde_json::{Value, json}; +use tokio_tungstenite::tungstenite::client::IntoClientRequest; + +const PNG: [u8; 8] = [0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]; + +/// `a.txt` and `keep.md` committed; the worktree edits `a.txt` and adds `b.txt`. +fn repo() -> TestRepo { + let t = TestRepo::new(); + t.write("a.txt", "a\n"); + t.write("keep.md", "kept\n"); + t.commit("base"); + t.write("a.txt", "a\nb\n"); + t.write("b.txt", "bee\n"); + t +} + +async fn reports(s: &TestServer) -> Value { + let r = s.get("/api/reports").await; + assert_eq!(r.status, 200, "{}", r.body); + r.json() +} + +fn report<'a>(list: &'a Value, id: &str) -> Option<&'a Value> { + list["reports"].as_array().unwrap().iter().find(|r| r["id"] == id) +} + +/// The report `id` once its run ended, or none once it left the list. +async fn settled(s: &TestServer, id: &str) -> Option { + let deadline = Instant::now() + Duration::from_secs(20); + loop { + let list = reports(s).await; + match report(&list, id) { + Some(r) if r["status"] != "running" => return Some(r.clone()), + None => return None, + Some(_) => {} + } + assert!(Instant::now() < deadline, "report {id} never settled: {list}"); + tokio::time::sleep(Duration::from_millis(40)).await; + } +} + +#[tokio::test] +async fn pushes_lists_and_persists_a_report() { + let s = TestServer::over(repo(), EnterRequest::Working).await; + let mut ws = tokio_tungstenite::connect_async(s.ws_url().into_client_request().unwrap()).await.unwrap().0; + let body = json!({ + "title": " Notes ", + "blocks": [ + { "kind": "markdown", "text": "\n# Risky\n\nLook at b", "anchor": { "path": "a.txt", "line": 2 } }, + { "kind": "markdown", "text": "gone", "anchor": { "path": "nope.txt", "line": 1 } } + ] + }); + let r = s.json("PUT", "/api/reports/notes", body.clone()).await; + assert_eq!(r.status, 201, "{}", r.body); + let created = r.json(); + assert_eq!((created["title"].as_str(), created["summary"].as_str()), (Some("Notes"), Some("Risky"))); + assert_eq!((created["source"].as_str(), created["stale"].as_bool()), (Some("pushed"), Some(false))); + assert_eq!(created["blocks"][0]["anchor"], json!({ "path": "a.txt", "line": 2 })); + assert!(created["blocks"][1].get("anchor").is_none(), "{}", created["blocks"][1]); + let pushed = tokio::time::timeout(Duration::from_secs(5), ws.next()).await.unwrap().unwrap().unwrap(); + assert_eq!(pushed.into_text().unwrap().as_str(), r#"{"type":"reports"}"#); + + let list = reports(&s).await; + assert_eq!(list["current"]["newSha"], "worktree"); + assert_eq!(report(&list, "notes").unwrap()["target"], list["current"]); + let log = s.get("/api/reports/notes/log").await.json(); + assert!(log["lines"][0].as_str().unwrap().contains("blocks[1].anchor dropped"), "{log}"); + assert_eq!(s.json("PUT", "/api/reports/notes", body).await.status, 200); + + // The report outlives the server: the next one over the repository lists it, unstale while + // the worktree stands, stale once it moves. + let repo = s.stop().await; + let s = TestServer::over(repo, EnterRequest::Working).await; + let list = reports(&s).await; + assert_eq!(report(&list, "notes").unwrap()["stale"], false, "{list}"); + s.repo.write("a.txt", "a\nb\nc\n"); + s.session.refresh().await; + let list = reports(&s).await; + assert_eq!(report(&list, "notes").unwrap()["stale"], true, "{list}"); + assert_ne!(report(&list, "notes").unwrap()["target"]["digest"], list["current"]["digest"]); + + assert_eq!(s.send("DELETE", "/api/reports/notes", &[], None).await.status, 204); + assert_eq!(s.get("/api/reports/notes").await.status, 404); + assert_eq!(s.send("DELETE", "/api/reports/notes", &[], None).await.status, 404); + s.close().await; +} + +#[tokio::test] +async fn serves_uploaded_pages_by_key() { + let s = TestServer::over(repo(), EnterRequest::Working).await; + let png = [&PNG[..], b"one"].concat(); + let r = s.client_put_bytes("/api/reports/render/files/p/1.png", png.clone()).await; + assert_eq!(r.0, 201, "{}", r.1); + let file: Value = serde_json::from_str(&r.1).unwrap(); + let key = file["key"].as_str().unwrap().to_owned(); + assert_eq!(file["type"], "image/png"); + let bad = s.client_put_bytes("/api/reports/render/files/bad.bin", vec![0, 1, 2, 3]).await; + assert_eq!(bad.0, 400, "{}", bad.1); + let bad = s.client_put_bytes("/api/reports/render/files/../x.png", png.clone()).await; + assert!(bad.0 == 400 || bad.0 == 404, "{} {}", bad.0, bad.1); + + let pages = |file: &str| json!({ "title": "Rendered", "blocks": [{ "kind": "pages", "label": "Pages", "pages": [{ "label": "p. 1", "old": null, "new": file }] }] }); + let r = s.json("PUT", "/api/reports/render", pages("missing.png")).await; + assert_eq!(r.status, 400, "{}", r.body); + assert!(r.body.contains("no file uploaded"), "{}", r.body); + let r = s.json("PUT", "/api/reports/render", pages("p/1.png")).await; + assert_eq!(r.status, 201, "{}", r.body); + assert_eq!(r.json()["blocks"][0]["pages"][0]["new"]["key"], key); + + let res = s.fetch(&format!("/api/reports/render/file?key={key}")).await; + assert_eq!(res.status(), 200); + assert_eq!(res.headers()["content-type"], "image/png"); + assert!(res.headers()["cache-control"].to_str().unwrap().contains("immutable")); + assert_eq!(res.bytes().await.unwrap().to_vec(), png); + assert_eq!(s.fetch("/api/reports/render/file?key=0000000000000000000000000000000000000000").await.status(), 404); + assert_eq!(s.fetch(&format!("/api/reports/other/file?key={key}")).await.status(), 404); + s.close().await; +} + +#[tokio::test] +async fn refuses_bad_ids_and_stale_targets() { + let s = TestServer::over(repo(), EnterRequest::Working).await; + let notes = + |target: Value| json!({ "title": "Notes", "target": target, "blocks": [{ "kind": "markdown", "text": "hi" }] }); + let r = s.json("PUT", "/api/reports/Bad_Id", notes(Value::Null)).await; + assert_eq!(r.status, 400, "{}", r.body); + let r = s.json("PUT", "/api/reports/notes", notes(json!({ "oldSha": "x", "newSha": "y", "digest": "z" }))).await; + assert_eq!(r.status, 409, "{}", r.body); + let current = reports(&s).await["current"].clone(); + assert_eq!(r.json()["current"], current); + let r = s.json("PUT", "/api/reports/notes", notes(current)).await; + assert_eq!(r.status, 201, "{}", r.body); + let r = s.json("PUT", "/api/reports/notes", json!({ "title": "", "blocks": [] })).await; + assert_eq!(r.status, 400, "{}", r.body); + s.close().await; +} + +/// The generator tests spawn `node` with a POSIX path in the command line; a Windows path's +/// backslashes would need quoting the tests do not exercise. +#[cfg(unix)] +mod generators { + use super::*; + + fn fake(name: &str, args: &str, when: &[&str], scope: Scope) -> Plugin { + let script = PathBuf::from(env!("CARGO_MANIFEST_DIR")).join("../../test/reports/fake-generator.mjs"); + let script = dunce::canonicalize(script).unwrap(); + Plugin { + name: name.into(), + command: format!("node '{}' {args}", script.display()), + when: when.iter().map(|w| (*w).to_owned()).collect(), + scope, + configured: true, + } + } + + fn node() -> bool { + let found = diffle::lsp::which::find_on_path("node").is_some(); + if !found { + eprintln!("node is not on PATH; skipping the generator tests"); + } + found + } + + #[tokio::test] + async fn runs_generators_for_matching_files() { + if !node() { + return; + } + let plugins = vec![ + fake("fake", "json", &["*.txt"], Scope::Changed), + fake("each", "stdout $DIFFLE_FILE", &["*.txt"], Scope::Changed), + fake("tree", "json", &["b.txt"], Scope::Tree), + fake("broken", "fail", &["a.txt"], Scope::Changed), + fake("quiet", "json", &["*.rs"], Scope::Changed), + Plugin { + name: "missing".into(), + command: "no-such-program-xyz".into(), + when: vec!["a.txt".into()], + scope: Scope::Changed, + configured: true, + }, + ]; + let s = TestServer::with_plugins(repo(), EnterRequest::Working, plugins).await; + + let fake = settled(&s, "fake").await.unwrap(); + assert_eq!( + (fake["status"].as_str(), fake["title"].as_str(), fake["source"].as_str()), + (Some("ready"), Some("Fake report"), Some("generator")) + ); + let text = fake["blocks"][0]["text"].as_str().unwrap(); + assert!(text.contains("old: a.txt\nnew: a.txt, b.txt\nscope: changed\nfiles: a.txt, b.txt"), "{text}"); + assert!(text.contains("a.txt=a\nb"), "{text}"); + assert_eq!(fake["blocks"][0]["anchor"], json!({ "path": "a.txt", "line": 1 })); + let key = fake["blocks"][1]["pages"][0]["new"]["key"].as_str().unwrap(); + assert_eq!(s.fetch(&format!("/api/reports/fake/file?key={key}")).await.status(), 200); + assert_eq!(s.get("/api/reports/fake/log").await.json()["lines"], json!(["rendered 1 page"])); + + let each = settled(&s, "each").await.unwrap(); + let texts: Vec<&str> = each["blocks"].as_array().unwrap().iter().map(|b| b["text"].as_str().unwrap()).collect(); + assert_eq!( + texts, + ["### a.txt\n\na.txt: old=a.txt new=a.txt,b.txt", "### b.txt\n\nb.txt: old=a.txt new=a.txt,b.txt"] + ); + assert_eq!(each["blocks"][1]["anchor"], json!({ "path": "b.txt", "side": "new", "line": 1 })); + assert_eq!(each["summary"], "a.txt"); + + let tree = settled(&s, "tree").await.unwrap(); + let text = tree["blocks"][0]["text"].as_str().unwrap(); + assert!(text.contains("old: a.txt, keep.md\nnew: a.txt, b.txt, keep.md\nscope: tree\nfiles: b.txt"), "{text}"); + + let broken = settled(&s, "broken").await.unwrap(); + assert_eq!(broken["status"], "failed"); + assert_eq!(broken["error"], "node exited 3: boom: no such thing"); + assert_eq!(s.get("/api/reports/broken/log").await.json()["lines"], json!(["boom: no such thing"])); + let missing = settled(&s, "missing").await.unwrap(); + assert_eq!(missing["error"], "no-such-program-xyz: not on PATH"); + assert!(settled(&s, "quiet").await.is_none()); + + // A generator's id is its own. + let r = s.json("PUT", "/api/reports/fake", json!({ "title": "x", "blocks": [] })).await; + assert_eq!(r.status, 409, "{}", r.body); + assert_eq!(s.send("DELETE", "/api/reports/fake", &[], None).await.status, 409); + + // The worktree moves: the report follows it, then leaves with the last matching file. + s.repo.write("a.txt", "a\nb\nc\n"); + s.session.refresh().await; + let fake = settled(&s, "fake").await.unwrap(); + assert!(fake["blocks"][0]["text"].as_str().unwrap().contains("a.txt=a\nb\nc"), "{fake}"); + assert_eq!(fake["stale"], false); + s.repo.write("a.txt", "a\n"); + s.repo.rm("b.txt"); + s.session.refresh().await; + assert!(settled(&s, "fake").await.is_none()); + assert!(settled(&s, "each").await.is_none()); + s.close().await; + } +} diff --git a/crates/diffle/tests/server.rs b/crates/diffle/tests/server.rs index ccaba6e7..17276cdc 100644 --- a/crates/diffle/tests/server.rs +++ b/crates/diffle/tests/server.rs @@ -24,6 +24,7 @@ const BODY_ROUTES: &[(&str, &str)] = &[ ("POST", "/api/lsp/hover"), ("POST", "/api/lsp/references"), ("POST", "/api/lsp/occurrences"), + ("PUT", "/api/reports/notes"), ]; type Ws = tokio_tungstenite::WebSocketStream>; diff --git a/crates/diffle/tests/snapshots/cli__help.snap b/crates/diffle/tests/snapshots/cli__help.snap index 4089f65a..ab6b26c0 100644 --- a/crates/diffle/tests/snapshots/cli__help.snap +++ b/crates/diffle/tests/snapshots/cli__help.snap @@ -9,6 +9,8 @@ Usage: diffle [OPTIONS] [revs]... [COMMAND] Commands: config print the config file path and contents lsp show the language server each language would get, and what is missing from PATH + plugins show the report generators a review would run, and what is missing from PATH + report push a report into the diffle running for this repository, or remove one completion print a shell completion script Arguments: diff --git a/example/plugins/diffle-pixi b/example/plugins/diffle-pixi new file mode 100755 index 00000000..9170b2dd --- /dev/null +++ b/example/plugins/diffle-pixi @@ -0,0 +1,28 @@ +#!/bin/sh +# A diffle report generator for pixi.lock: the lock file's diff as pixi-diff-to-markdown writes it. +# +# diffle's built-in `pixi` plugin runs `pixi-diff | pixi-diff-to-markdown` straight from PATH when +# both are installed (`pixi global install pixi-diff pixi-diff-to-markdown`). This script is the +# same thing as a program of its own, for a repository that wants other flags, pixi's extension +# dispatch (`pixi diff`), or a report with a heading per lock file. Point the config at it: +# { "plugins": { "pixi": { "command": "/path/to/example/plugins/diffle-pixi $DIFFLE_FILE" } } } +# +# The contract: DIFFLE_OLD and DIFFLE_NEW hold copies of the sides, DIFFLE_FILE the matched path +# (the command names $DIFFLE_FILE, so diffle runs the script once per changed lock file), and the +# review arrives as JSON on stdin. Whatever is printed becomes one markdown block anchored at the +# file; a nonzero exit shows the last line of stderr as the report's error. +set -eu + +file="${1:-${DIFFLE_FILE:-pixi.lock}}" +before="$DIFFLE_OLD/$file" +after="$DIFFLE_NEW/$file" +# A lock file added or removed in this diff has one side only: nothing to diff, say so and succeed. +if [ ! -f "$before" ]; then + echo "\`$file\` is new in this diff: no earlier lock file to compare with." + exit 0 +fi +if [ ! -f "$after" ]; then + echo "\`$file\` is removed in this diff." + exit 0 +fi +pixi diff --before "$before" --after "$after" | pixi diff-to-markdown diff --git a/example/plugins/diffle-typst b/example/plugins/diffle-typst new file mode 100755 index 00000000..3375c3d5 --- /dev/null +++ b/example/plugins/diffle-typst @@ -0,0 +1,77 @@ +#!/usr/bin/env python3 +"""A reference diffle report generator: the rendered pages of changed Typst documents. + +diffle's built-in `typst` plugin runs a `diffle-typst` on PATH whenever a `.typ` file changed, with +copies of both sides of the review. This one compiles each changed document on each side with +`typst compile --format png` and reports the pages as pairs, so diffle shows them side by side, +swiped, or as a pixel difference, each page pointing at the document's first line. A document that +does not compile on its own (a chapter the main file includes) is skipped with a note on stderr. + +Install it on PATH, or point the config at it: + { "plugins": { "typst": { "command": "python3 /path/to/example/plugins/diffle-typst" } } } + +The contract this follows, for writing your own generator in any language: + - stdin: JSON with `review`, `target`, `root`, `old`, `new`, `out`, `file`, `files`, `scope` + - DIFFLE_OLD, DIFFLE_NEW: folders holding the sides (every file of the snapshot for `tree` scope) + - DIFFLE_OUT: where to write `report.json` and the files it names, relative to it + - a program that writes no report.json has its stdout shown as one markdown block +""" +import json +import os +import subprocess +import sys +from pathlib import Path + +PPI = "96" + + +def compile_pages(side_dir: Path, document: str, out: Path, tag: str) -> list[str]: + """The PNG pages of `document` under `side_dir`, as paths relative to `out`; empty when it does not compile.""" + source = side_dir / document + if not source.is_file(): + return [] + pattern = out / tag / f"{{p}}.png" + (out / tag).mkdir(parents=True, exist_ok=True) + run = subprocess.run( + ["typst", "compile", "--root", str(side_dir), "--format", "png", "--ppi", PPI, str(source), str(pattern)], + capture_output=True, + text=True, + ) + if run.returncode != 0: + sys.stderr.write(f"{document} ({tag}): {run.stderr.strip().splitlines()[-1] if run.stderr.strip() else 'did not compile'}\n") + return [] + pages = sorted((out / tag).glob("*.png"), key=lambda p: int(p.stem)) + return [str(p.relative_to(out)) for p in pages] + + +def main() -> int: + spec = json.load(sys.stdin) + old, new, out = Path(spec["old"]), Path(spec["new"]), Path(spec["out"]) + blocks = [] + for entry in spec["files"]: + document = entry["path"] + tag = document.replace("/", "_").removesuffix(".typ") + before = compile_pages(old, entry.get("oldPath") or document, out, f"{tag}-old") + after = compile_pages(new, document, out, f"{tag}-new") + if not before and not after: + continue + pages = [] + for n in range(max(len(before), len(after))): + pages.append( + { + "label": f"{document} · p. {n + 1}", + "old": before[n] if n < len(before) else None, + "new": after[n] if n < len(after) else None, + "anchor": {"path": document, "line": 1} if entry["status"] != "D" else None, + } + ) + blocks.append({"kind": "pages", "label": f"{document}: {len(before)} → {len(after)} pages", "pages": pages}) + if not blocks: + sys.stderr.write("no changed document compiled on its own\n") + return 1 + (out / "report.json").write_text(json.dumps({"title": "Rendered Typst", "blocks": blocks})) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/src/client/App.tsx b/src/client/App.tsx index 54fa25be..423f8b01 100644 --- a/src/client/App.tsx +++ b/src/client/App.tsx @@ -9,6 +9,7 @@ import { useStore } from './store.js'; import { FileTreePane } from './tree/FileTreePane.js'; import { TooltipHost } from './ui/TitleTooltips.js'; import './views/registry.js'; +import { useReports } from './reports/state.js'; import { gridColumns, listViews, panelOf, useViews } from './views/state.js'; export function App() { @@ -70,6 +71,7 @@ export function App() { const resync = () => { setLost(false); void boot().then(() => performance.mark('diffle:snapshot')); + void useReports.getState().refresh(); }; return connectWs( (msg) => { @@ -92,6 +94,9 @@ export function App() { case 'lsp': setLspStatus(msg.payload); break; + case 'reports': + void useReports.getState().refresh(); + break; default: break; } diff --git a/src/client/api.ts b/src/client/api.ts index 31ab338c..bf2ba8fc 100644 --- a/src/client/api.ts +++ b/src/client/api.ts @@ -29,6 +29,8 @@ import { PermalinkRequestSchema, PermalinkSchema, RefsResponseSchema, + ReportLogResponseSchema, + ReportsResponseSchema, SearchResponseSchema, ServerMessageSchema, SnapshotSchema, @@ -226,6 +228,13 @@ export const api = { /** Document symbols for a path, or workspace symbols matching a query. */ lspSymbols: (query: SymbolsQuery) => json(LspSymbolsResponseSchema, `api/lsp/symbols?${q(query)}`), + /** Every report of the review, and the target a pushed one should name. */ + reports: () => json(ReportsResponseSchema, 'api/reports'), + /** What a report's maker wrote to stderr, or what validation dropped. */ + reportLog: (id: string) => json(ReportLogResponseSchema, `api/reports/${encodeURIComponent(id)}/log`), + /** Where an `` loads a file a report names; `key` is the file's `ReportFile.key`. */ + reportFileUrl: (id: string, key: string) => + new URL(`api/reports/${encodeURIComponent(id)}/file?${q({ key })}`, location.href).href, config: () => json(UserConfigSchema, 'api/config'), /** lspCommands and githubHosts are CLI-only; the server ignores them. */ saveConfig: (config: ConfigUpdate) => diff --git a/src/client/keyboard/HelpOverlay.tsx b/src/client/keyboard/HelpOverlay.tsx index 7c988528..4fcab860 100644 --- a/src/client/keyboard/HelpOverlay.tsx +++ b/src/client/keyboard/HelpOverlay.tsx @@ -91,6 +91,7 @@ export const DIFF_HELP: HelpSection[][] = [ title: 'Change graph (language server)', rows: [ ['M', 'graph of changed symbols; ? there lists its keys'], + ['gr', 'reports from plugins and agents; ? there lists its keys'], ['right-click a folder', 'Show in graph: only the changes under it'], ], }, diff --git a/src/client/keyboard/useKeymap.ts b/src/client/keyboard/useKeymap.ts index af7a0565..76644c35 100644 --- a/src/client/keyboard/useKeymap.ts +++ b/src/client/keyboard/useKeymap.ts @@ -100,6 +100,7 @@ const KEYMAP: Record = { r: (s) => s.reloadIfMoved(), '?': (s) => s.setHelpOpen(!s.helpOpen), M: () => toggleView('graph'), + gr: () => toggleView('reports'), }; /** The page's text selection, else the focused word (`w`), as a search query: one line only, escaped when regex search is on. */ diff --git a/src/client/reports/Blocks.tsx b/src/client/reports/Blocks.tsx new file mode 100644 index 00000000..74f13b32 --- /dev/null +++ b/src/client/reports/Blocks.tsx @@ -0,0 +1,117 @@ +import { MessageSquarePlus } from 'lucide-react'; +import { twMerge } from 'tailwind-merge'; +import { api } from '@/client/api.js'; +import { FilePath } from '@/client/FilePath.js'; +import { Markdown } from '@/client/Markdown.js'; +import { Pictures } from '@/client/review/ImageDiff.js'; +import { Button } from '@/client/ui/Button.js'; +import type { ReportAnchor, ReportBlock, ReportPage } from '@/shared/protocol.js'; +import { useReports } from './state.js'; + +/** `path:line`, or `path:start-end`, as comments name their place. */ +export function anchorLabel(a: ReportAnchor): string { + const lines = a.endLine != null && a.endLine !== a.line ? `${a.line}-${a.endLine}` : `${a.line}`; + return `${a.path}:${lines}${a.side === 'old' ? ' (removed)' : ''}`; +} + +/** One block of the shown report. The cursor's block is outlined; its anchor is where `c` comments. */ +export function Block({ id, block, index }: { id: string; block: ReportBlock; index: number }) { + const atCursor = useReports((s) => s.cursor === index); + const setCursor = useReports((s) => s.setCursor); + return ( +
setCursor(index)} + className={twMerge( + 'rounded-md border border-border bg-card shadow-card [transition:border-color_120ms_ease]', + atCursor && 'border-accent', + )} + > + {block.kind === 'markdown' ? ( + <> + {block.anchor && } +
+ +
+ + ) : ( + <> + {block.label && ( +
+ {block.label} +
+ )} +
+ {block.pages.map((page, i) => ( + + ))} +
+ + )} +
+ ); +} + +/** Where a block points: a click opens the diff there, the button comments there. */ +function AnchorBar({ anchor }: { anchor: ReportAnchor }) { + const jump = useReports((s) => s.jump); + const comment = useReports((s) => s.comment); + return ( +
+ + +
+ ); +} + +function Page({ id, page }: { id: string; page: ReportPage }) { + const url = { + old: page.old ? api.reportFileUrl(id, page.old.key) : null, + new: page.new ? api.reportFileUrl(id, page.new.key) : null, + }; + const sides = page.old && page.new ? 'both' : page.old ? 'old' : 'new'; + return ( +
+
+ {page.label} + {page.anchor && } +
+ +
+ ); +} + +function PageAnchor({ anchor }: { anchor: ReportAnchor }) { + const jump = useReports((s) => s.jump); + return ( + + ); +} diff --git a/src/client/reports/ReportsPanel.tsx b/src/client/reports/ReportsPanel.tsx new file mode 100644 index 00000000..6cb3718c --- /dev/null +++ b/src/client/reports/ReportsPanel.tsx @@ -0,0 +1,58 @@ +import { AlertTriangle, Bot, FileText, LoaderCircle, Wrench } from 'lucide-react'; +import { twMerge } from 'tailwind-merge'; +import type { Report } from '@/shared/protocol.js'; +import { shownReport, useReports } from './state.js'; + +/** Every report, generated ones first: a click shows it in the main pane. */ +export function ReportsPanel() { + const reports = useReports((s) => s.reports); + const shown = useReports(shownReport); + const select = useReports((s) => s.select); + return ( + + ); +} + +function Row({ report: r, shown, select }: { report: Report; shown: boolean; select: (id: string) => void }) { + const Source = r.source === 'generator' ? Wrench : Bot; + return ( + + ); +} diff --git a/src/client/reports/ReportsView.tsx b/src/client/reports/ReportsView.tsx new file mode 100644 index 00000000..786770f6 --- /dev/null +++ b/src/client/reports/ReportsView.tsx @@ -0,0 +1,105 @@ +import { AlertTriangle, LoaderCircle, ScrollText } from 'lucide-react'; +import { relativeTime } from '@/client/model.js'; +import { Button, TOGGLE_ON } from '@/client/ui/Button.js'; +import type { Report } from '@/shared/protocol.js'; +import { Block } from './Blocks.js'; +import { shownReport, useReports } from './state.js'; + +/** The shown report in the diff's place: its header, its log when opened, and its blocks. */ +export function ReportsView() { + const report = useReports(shownReport); + const loaded = useReports((s) => s.loaded); + const error = useReports((s) => s.error); + return ( +
+ {report ? : } +
+ ); +} + +function Shown({ report }: { report: Report }) { + const log = useReports((s) => (s.log?.id === report.id ? s.log.lines : null)); + const toggleLog = useReports((s) => s.toggleLog); + return ( + <> +
+ {report.title} + + + {relativeTime(report.updatedAt)} + + +
+ {report.error && ( +

+ {report.error} +

+ )} + {log && ( + 
+          {log.length ? log.join('\n') : 'Nothing was logged.'}
+
+ )} +
+ {report.blocks.length === 0 && report.status === 'running' && ( +

+ Running +

+ )} + {report.blocks.map((block, i) => ( + + ))} +
+ + ); +} + +/** Running, failed, or stale, as one word with its mark; nothing while a report is simply ready. */ +function Status({ report }: { report: Report }) { + if (report.status === 'running') + return ( + + running + + ); + if (report.status === 'failed') return failed; + if (report.stale) + return ( + + stale + + ); + return null; +} + +function Empty({ loaded, error }: { loaded: boolean; error: string | null }) { + return ( +
+ {error ? ( +

{error}

+ ) : !loaded ? ( +

Loading reports…

+ ) : ( + <> +

No reports for this diff.

+

+ A plugin on PATH writes one when a changed file matches its globs; diffle plugins lists them. + Anything else can push one: PUT /api/reports/<id> with a title and markdown blocks, or{' '} + diffle report put <file> from a shell. +

+ + )} +
+ ); +} diff --git a/src/client/reports/help.ts b/src/client/reports/help.ts new file mode 100644 index 00000000..898d75c8 --- /dev/null +++ b/src/client/reports/help.ts @@ -0,0 +1,35 @@ +import type { HelpSection } from '@/client/views/state.js'; + +/** The reports' shortcuts: their own keys, then those they leave to the review. */ +export const REPORTS_HELP: HelpSection[][] = [ + [ + { + title: 'Reports', + rows: [ + ['j / k', 'next / previous block'], + ['J / K ] / [', 'next / previous report'], + ['Enter', 'open the block’s line in the diff'], + ['c', 'comment on the block’s line in the diff'], + ['click a report', 'show it; its log opens from the header'], + ], + }, + ], + [ + { + title: 'Review', + rows: [ + ['gr', 'back to the diff'], + ['M', 'the change graph'], + ['Ctrl+o / Ctrl+i', 'jump back / forward in history'], + ['⌘/Ctrl+b', 'toggle the file tree (Shift: reports list)'], + ['t', 'cycle theme'], + ['o', 'GitHub repository and pull request'], + ['m, then 1–6', 'open the mode picker and choose an entry'], + ], + }, + { + title: 'General', + rows: [['?', 'toggle this help']], + }, + ], +]; diff --git a/src/client/reports/keys.ts b/src/client/reports/keys.ts new file mode 100644 index 00000000..0c23e8ed --- /dev/null +++ b/src/client/reports/keys.ts @@ -0,0 +1,40 @@ +import { useStore } from '@/client/store.js'; +import { cursorAnchor, useReports } from './state.js'; + +/** Keys the reports leave to the review: help, theme, the menus, `M` to the graph, and `gr` back to the diff. */ +export const REPORTS_PASSES = new Set(['?', 't', 'm', 'M', 'o', 'g', 'r']); + +/** The reports view's own keys; true when `e` was one. The keymap calls it only while the view shows. */ +export function reportsKeys(e: KeyboardEvent): boolean { + const reports = useReports.getState(); + switch (e.key) { + case 'j': + case 'ArrowDown': + reports.moveCursor(1); + return true; + case 'k': + case 'ArrowUp': + reports.moveCursor(-1); + return true; + case 'J': + case ']': + reports.step(1); + return true; + case 'K': + case '[': + reports.step(-1); + return true; + case 'c': + case 'Enter': { + const anchor = cursorAnchor(reports); + if (!anchor) { + useStore.getState().flash('This block points at no line of the diff'); + return true; + } + void (e.key === 'c' ? reports.comment(anchor) : reports.jump(anchor)); + return true; + } + default: + return false; + } +} diff --git a/src/client/reports/state.ts b/src/client/reports/state.ts new file mode 100644 index 00000000..81aea8d9 --- /dev/null +++ b/src/client/reports/state.ts @@ -0,0 +1,147 @@ +import { create } from 'zustand'; +import { api } from '@/client/api.js'; +import { useStore, type ViewPlace } from '@/client/store.js'; +import { showView } from '@/client/views/state.js'; +import type { Report, ReportAnchor, ReportTarget } from '@/shared/protocol.js'; + +export interface ReportsState { + reports: Report[]; + /** The target a pushed report should name now; null before the first load. */ + current: ReportTarget | null; + /** The first load answered, so an empty list means no reports rather than none yet. */ + loaded: boolean; + error: string | null; + /** The report shown, by id; null shows the first. */ + selected: string | null; + /** The block the cursor is on in the shown report: `c` comments at its anchor. */ + cursor: number; + /** The log open under the shown report's header, by id. */ + log: { id: string; lines: string[] } | null; + refresh(): Promise; + select(id: string | null): void; + moveCursor(by: 1 | -1): void; + setCursor(index: number): void; + /** Shows the next (or previous) report in the list. */ + step(by: 1 | -1): void; + /** Leaves for the diff at `anchor`, and opens a comment there on the new side. */ + comment(anchor: ReportAnchor): Promise; + /** Leaves for the diff at `anchor`. */ + jump(anchor: ReportAnchor): Promise; + /** Opens the log of report `id`, or closes it when it is the one open. */ + toggleLog(id: string): Promise; +} + +let loads = 0; + +/** The reports view. It reads the review store and never the other way round. */ +export const useReports = create((set, get) => ({ + reports: [], + current: null, + loaded: false, + error: null, + selected: null, + cursor: 0, + log: null, + + async refresh() { + const generation = ++loads; + try { + const res = await api.reports(); + if (generation !== loads) return; + const shown = shownReport({ ...get(), reports: res.reports }); + set({ + reports: res.reports, + current: res.current, + loaded: true, + error: null, + cursor: Math.min(get().cursor, Math.max(0, (shown?.blocks.length ?? 1) - 1)), + }); + const log = get().log; + if (log && !res.reports.some((r) => r.id === log.id)) set({ log: null }); + } catch (e) { + if (generation === loads) set({ loaded: true, error: e instanceof Error ? e.message : String(e) }); + } + }, + + select(id) { + if (id === get().selected) return; + set({ selected: id, cursor: 0, log: null }); + }, + + moveCursor(by) { + const shown = shownReport(get()); + if (!shown || shown.blocks.length === 0) return; + const cursor = Math.max(0, Math.min(shown.blocks.length - 1, get().cursor + by)); + set({ cursor }); + document.querySelector(`[data-report-block="${cursor}"]`)?.scrollIntoView({ block: 'nearest' }); + }, + + setCursor(index) { + set({ cursor: index }); + }, + + step(by) { + const { reports } = get(); + if (reports.length === 0) return; + const shown = shownReport(get()); + const at = reports.findIndex((r) => r.id === shown?.id); + const next = reports[(at + by + reports.length) % reports.length]!; + get().select(next.id); + }, + + async comment(anchor) { + await get().jump(anchor); + const review = useStore.getState(); + // The cursor sits on the line once the jump landed on the new side; the old side takes no draft. + if (anchor.side === 'old') return review.flash('Press c on the removed line to comment there'); + if (review.selection) void review.openDraft(review.selection); + }, + + async jump(anchor) { + const review = useStore.getState(); + // Closed only once the jump is made: the jump list records the reports as the place left. + if (anchor.side === 'old') await review.openFile(anchor.path, anchor.endLine ?? anchor.line, 'old'); + else await review.goToLink(anchor.path, anchor.line); + showView('diff'); + }, + + async toggleLog(id) { + if (get().log?.id === id) return set({ log: null }); + try { + const { lines } = await api.reportLog(id); + set({ log: { id, lines } }); + } catch (e) { + useStore.getState().flash(`Loading the log failed: ${e instanceof Error ? e.message : String(e)}`); + } + }, +})); + +/** The report the view shows: the selected one, else the first. */ +export function shownReport(s: Pick): Report | null { + return s.reports.find((r) => r.id === s.selected) ?? s.reports[0] ?? null; +} + +/** The anchor the cursor's block points at, or null. A pages block anchors through its first page. */ +export function cursorAnchor(s: Pick): ReportAnchor | null { + const block = shownReport(s)?.blocks[s.cursor]; + if (!block) return null; + if (block.kind === 'markdown') return block.anchor ?? null; + return block.pages.find((p) => p.anchor)?.anchor ?? null; +} + +export function enterReports(): void { + const s = useReports.getState(); + if (!s.loaded) void s.refresh(); +} + +/** The reports as they stand, for the jump list: Ctrl+o after a jump into the diff restores them. */ +export function reportsPlace(): ViewPlace { + const { selected, cursor } = useReports.getState(); + return { + view: `reports:${selected ?? ''}:${cursor}`, + restore() { + useReports.setState({ selected, cursor }); + showView('reports'); + }, + }; +} diff --git a/src/client/reports/view.ts b/src/client/reports/view.ts new file mode 100644 index 00000000..a992ade5 --- /dev/null +++ b/src/client/reports/view.ts @@ -0,0 +1,24 @@ +import { FileText } from 'lucide-react'; +import type { ViewDef } from '@/client/views/state.js'; +import { REPORTS_HELP } from './help.js'; +import { REPORTS_PASSES, reportsKeys } from './keys.js'; +import { ReportsPanel } from './ReportsPanel.js'; +import { ReportsView } from './ReportsView.js'; +import { enterReports, reportsPlace } from './state.js'; + +/** Reports: what plugins made of the snapshot and what an agent pushed, beside the diff. */ +export const reportsView: ViewDef = { + id: 'reports', + label: 'Reports', + icon: FileText, + title: 'Reports from plugins and agents (gr)', + Main: ReportsView, + Side: ReportsPanel, + help: REPORTS_HELP, + panelLabel: 'reports list', + panelWidth: '18rem', + enter: enterReports, + place: reportsPlace, + keys: reportsKeys, + passes: REPORTS_PASSES, +}; diff --git a/src/client/review/ImageDiff.tsx b/src/client/review/ImageDiff.tsx index fa5d202d..d313fc4d 100644 --- a/src/client/review/ImageDiff.tsx +++ b/src/client/review/ImageDiff.tsx @@ -85,15 +85,19 @@ export function ImageDiff({ path, sides }: { path: string; sides: ImageSides }) }; return (
- {sides === 'both' ? ( - - ) : ( - - )} +
); } +/** + * The pictures of `path` by URL: one side alone, or both with the comparison modes. A report's rendered + * pages use it too, with the page's label as `path`. + */ +export function Pictures({ path, sides, url }: { path: string; sides: ImageSides; url: Record }) { + return sides === 'both' ? : ; +} + function Single({ path, side, url }: { path: string; side: Side; url: string | null }) { return
; } diff --git a/src/client/views/registry.ts b/src/client/views/registry.ts index f7a16e6d..1ba78c61 100644 --- a/src/client/views/registry.ts +++ b/src/client/views/registry.ts @@ -1,6 +1,7 @@ import { FileDiff } from 'lucide-react'; import { CommentPanel } from '@/client/comments/CommentPanel.js'; import { graphView } from '@/client/graph/view.js'; +import { reportsView } from '@/client/reports/view.js'; import { DiffStyleSwitch } from '@/client/header/DiffStyleSwitch.js'; import { DIFF_HELP } from '@/client/keyboard/HelpOverlay.js'; import { ReviewPane } from '@/client/review/ReviewPane.js'; @@ -21,7 +22,7 @@ const diffView: ViewDef = { panelWidth: '21.25rem', }; -defineViews([diffView, graphView]); +defineViews([diffView, graphView, reportsView]); useStore.getState().registerView({ here: () => activeView().place?.() ?? null, leave: () => showView('diff'), diff --git a/src/client/views/state.ts b/src/client/views/state.ts index f0345097..2ab4d4f8 100644 --- a/src/client/views/state.ts +++ b/src/client/views/state.ts @@ -3,7 +3,7 @@ import type { ComponentType } from 'react'; import { create } from 'zustand'; import { type LayoutState, type ReviewState, useStore, type ViewPlace } from '@/client/store.js'; -export type ViewId = 'diff' | 'graph'; +export type ViewId = 'diff' | 'graph' | 'reports'; type HelpRow = [keys: string, action: string]; export type HelpSection = { title: string; rows: HelpRow[] }; diff --git a/src/shared/protocol.ts b/src/shared/protocol.ts index 89771ede..bb66328b 100644 --- a/src/shared/protocol.ts +++ b/src/shared/protocol.ts @@ -599,6 +599,27 @@ export type SearchResponse = z.infer; export const FollowRefsSchema = z.enum(['on', 'auto', 'off']); export type FollowRefs = z.infer; +/** + * A program that makes a report from a snapshot. It runs whenever a changed file matches `when`, + * with copies of the files in DIFFLE_OLD and DIFFLE_NEW, a JSON description of the review on + * stdin, and DIFFLE_ROOT, DIFFLE_OUT in its environment. A `report.json` it writes into DIFFLE_OUT + * is the report, its files named relative to that directory; otherwise its stdout is one markdown + * block. A command naming $DIFFLE_FILE runs once per matched file, with that path in the variable. + */ +export const PluginConfigSchema = z.object({ + /** + * The program and its arguments, split as a shell would, without a shell: a `|` between words + * pipes one program's stdout into the next, and `$DIFFLE_OLD`, `$DIFFLE_NEW`, `$DIFFLE_OUT`, + * `$DIFFLE_ROOT` and `$DIFFLE_FILE` are substituted in arguments. Empty turns a built-in plugin off. + */ + command: z.string(), + /** Globs (picomatch syntax) of changed files that start it; a built-in plugin keeps its own when absent. */ + when: z.string().array().optional(), + /** What the program gets copies of: the matched changed files, or every file the snapshot shows. Default changed. */ + scope: z.enum(['changed', 'tree']).optional(), +}); +export type PluginConfig = z.infer; + export const UserConfigSchema = z.object({ /** Globs (picomatch syntax) for files that start viewed + collapsed. Empty by default. */ autoViewed: z.string().array(), @@ -619,6 +640,11 @@ export const UserConfigSchema = z.object({ * config file. */ githubHosts: z.record(z.string(), z.string()), + /** + * Report generators by name: a built-in plugin (`typst`, `pixi`) is replaced or turned off by + * its name, any other name adds one. Read-only over HTTP; set by editing the config file. + */ + plugins: z.record(z.string(), PluginConfigSchema), }); export type UserConfig = z.infer; @@ -629,6 +655,7 @@ export const DEFAULT_USER_CONFIG: UserConfig = { lineWrap: true, lspCommands: {}, githubHosts: {}, + plugins: {}, }; /** A point in a new-side file. LSP semantics: the column counts UTF-16 units, like a JS string index. */ @@ -891,6 +918,144 @@ export const GraphResponseSchema = z.object({ }); export type GraphResponse = z.infer; +/** + * A place in the diff a report block points at: `c` on the block comments there. The new side + * unless `side` says old. A path the snapshot does not show is dropped when the report is served. + */ +export const ReportAnchorSchema = z.object({ + path: z.string(), + /** Default 'new'. */ + side: SideSchema.optional(), + /** 1-based. */ + line: z.number().int(), + /** Default `line`. */ + endLine: z.number().int().optional(), +}); +export type ReportAnchor = z.infer; + +/** + * A file a report refers to, as the client fetches it: `GET /api/reports/{id}/file?key=`. + * The key names fixed bytes, so what it serves may be cached for good. + */ +export const ReportFileSchema = z.object({ + key: z.string(), + /** The media type its bytes were sniffed as; only raster images are ever served. */ + type: z.string(), +}); +export type ReportFile = z.infer; + +/** One rendered page or figure, as authored: each side names a file next to the report, relative, or none. */ +export const ReportPageInputSchema = z.object({ + label: z.string(), + old: z.string().nullable(), + new: z.string().nullable(), + anchor: ReportAnchorSchema.optional(), +}); +export type ReportPageInput = z.infer; + +/** One rendered page or figure, as served: a side with a file is shown; both are compared. */ +export const ReportPageSchema = z.object({ + label: z.string(), + old: ReportFileSchema.nullable(), + new: ReportFileSchema.nullable(), + anchor: ReportAnchorSchema.optional(), +}); +export type ReportPage = z.infer; + +/** GitHub-flavored markdown, rendered as a comment body is: raw HTML stays literal and images are dropped. */ +export const ReportMarkdownBlockSchema = z.object({ + kind: z.literal('markdown'), + text: z.string(), + anchor: ReportAnchorSchema.optional(), +}); + +/** A report's blocks as a generator or an agent writes them. */ +export const ReportBlockInputSchema = z.discriminatedUnion('kind', [ + ReportMarkdownBlockSchema, + z.object({ + kind: z.literal('pages'), + /** What the pages are, e.g. "Rendered, removals struck through". */ + label: z.string().optional(), + pages: ReportPageInputSchema.array(), + }), +]); +export type ReportBlockInput = z.infer; + +/** A report's blocks as the client gets them: every file resolved to a key. */ +export const ReportBlockSchema = z.discriminatedUnion('kind', [ + ReportMarkdownBlockSchema, + z.object({ + kind: z.literal('pages'), + label: z.string().optional(), + pages: ReportPageSchema.array(), + }), +]); +export type ReportBlock = z.infer; + +/** + * The snapshot a report was made for. Two shas do not pin a worktree side, so `digest` hashes the + * changed files' blobs too; `GET /api/reports` says the current one. + */ +export const ReportTargetSchema = z.object({ + oldSha: z.string(), + newSha: z.string(), + digest: z.string(), +}); +export type ReportTarget = z.infer; + +/** + * `PUT /api/reports/{id}`: a report as authored. Files it names must be uploaded first, with + * `PUT /api/reports/{id}/files/`. A `target` other than the current snapshot's is refused + * with 409 and the current target, so an agent that fell behind can refetch; absent means current. + */ +export const ReportInputSchema = z.object({ + title: z.string(), + /** One line for the list; the first line of the first markdown block otherwise. */ + summary: z.string().optional(), + target: ReportTargetSchema.optional(), + blocks: ReportBlockInputSchema.array(), +}); +export type ReportInput = z.infer; + +/** A generator diffle ran for the snapshot, or something outside pushed it over the API. */ +export const ReportSourceSchema = z.enum(['generator', 'pushed']); +export type ReportSource = z.infer; + +/** `running` keeps the previous blocks until the new ones land, as a snapshot refresh does. */ +export const ReportStatusSchema = z.enum(['running', 'ready', 'failed']); +export type ReportStatus = z.infer; + +export const ReportSchema = z.object({ + /** Lowercase letters, digits and dashes; a generator's is its plugin name. */ + id: z.string(), + source: ReportSourceSchema, + title: z.string(), + summary: z.string(), + target: ReportTargetSchema, + status: ReportStatusSchema, + /** The failure in one line: a program not on PATH, a nonzero exit, a report that did not validate. */ + error: z.string().nullable(), + /** The snapshot moved away from `target`: the blocks describe an older diff. */ + stale: z.boolean(), + /** Milliseconds since the epoch. */ + updatedAt: z.number().int(), + blocks: ReportBlockSchema.array(), +}); +export type Report = z.infer; + +/** `GET /api/reports`: every report, generated ones first, and the target a pushed one should name. */ +export const ReportsResponseSchema = z.object({ + current: ReportTargetSchema, + reports: ReportSchema.array(), +}); +export type ReportsResponse = z.infer; + +/** `GET /api/reports/{id}/log`: what the generator wrote to stderr, most recent run only. */ +export const ReportLogResponseSchema = z.object({ lines: z.string().array() }); +export type ReportLogResponse = z.infer; + +export const ReportFileQuerySchema = z.object({ key: z.string() }); + /** Derived per changed file: 'restale' means viewed at an older blob. */ export const ViewedStateSchema = z.enum(['unviewed', 'viewed', 'restale']); export type ViewedState = z.infer; @@ -918,6 +1083,10 @@ export const ServerMessageSchema = z.discriminatedUnion('type', [ type: z.literal('lsp'), payload: LspStatusSchema, }), + /** A report was added, replaced, removed, or changed status; `GET /api/reports` has the list. */ + z.object({ + type: z.literal('reports'), + }), ]); export type ServerMessage = z.infer; diff --git a/test/client/reports.test.ts b/test/client/reports.test.ts new file mode 100644 index 00000000..e288ff67 --- /dev/null +++ b/test/client/reports.test.ts @@ -0,0 +1,103 @@ +// @vitest-environment jsdom +import { describe, expect, it, vi } from 'vitest'; +import type { Report, ReportAnchor } from '@/shared/protocol.js'; + +const review = vi.hoisted(() => ({ + flash: vi.fn(), + goToLink: vi.fn(async () => {}), + openFile: vi.fn(async () => {}), + openDraft: vi.fn(async () => {}), + selection: null as { id: string } | null, +})); +vi.mock('@/client/store.js', () => ({ useStore: { getState: () => review } })); +const shown = vi.hoisted(() => ({ ids: [] as string[] })); +vi.mock('@/client/views/state.js', () => ({ showView: (id: string) => shown.ids.push(id) })); +const api = vi.hoisted(() => ({ reports: vi.fn(), reportLog: vi.fn() })); +vi.mock('@/client/api.js', () => ({ api })); + +const { cursorAnchor, shownReport, useReports } = await import('@/client/reports/state.js'); +const { reportsKeys } = await import('@/client/reports/keys.js'); + +const target = { oldSha: 'a', newSha: 'b', digest: 'c' }; +const anchor: ReportAnchor = { path: 'src/app.py', line: 3 }; +function report(id: string, blocks: Report['blocks']): Report { + return { + id, + source: 'pushed', + title: id, + summary: '', + target, + status: 'ready', + error: null, + stale: false, + updatedAt: 0, + blocks, + }; +} +const notes = report('notes', [ + { kind: 'markdown', text: 'plain' }, + { kind: 'markdown', text: 'placed', anchor }, + { + kind: 'pages', + pages: [{ label: 'p. 1', old: null, new: { key: 'k', type: 'image/png' }, anchor: { ...anchor, line: 9 } }], + }, +]); +const other = report('other', []); + +const key = (k: string) => new KeyboardEvent('keydown', { key: k }); + +describe('the reports view state', () => { + it('shows the selected report, else the first, and reads the cursor block’s anchor', () => { + expect(shownReport({ reports: [notes, other], selected: null })?.id).toBe('notes'); + expect(shownReport({ reports: [notes, other], selected: 'other' })?.id).toBe('other'); + expect(shownReport({ reports: [notes], selected: 'gone' })?.id).toBe('notes'); + expect(cursorAnchor({ reports: [notes], selected: null, cursor: 0 })).toBeNull(); + expect(cursorAnchor({ reports: [notes], selected: null, cursor: 1 })).toEqual(anchor); + expect(cursorAnchor({ reports: [notes], selected: null, cursor: 2 })?.line).toBe(9); + expect(cursorAnchor({ reports: [], selected: null, cursor: 0 })).toBeNull(); + }); + + it('refreshes from the API, keeping the cursor inside the shown report', async () => { + api.reports.mockResolvedValueOnce({ current: target, reports: [notes, other] }); + useReports.setState({ cursor: 7 }); + await useReports.getState().refresh(); + expect(useReports.getState().reports.map((r) => r.id)).toEqual(['notes', 'other']); + expect(useReports.getState().cursor).toBe(2); + expect(useReports.getState().loaded).toBe(true); + }); + + it('walks blocks and reports with the keys, and leaves for the diff at an anchor', async () => { + useReports.setState({ reports: [notes, other], selected: null, cursor: 0 }); + expect(reportsKeys(key('j'))).toBe(true); + expect(reportsKeys(key('j'))).toBe(true); + expect(reportsKeys(key('j'))).toBe(true); + expect(useReports.getState().cursor).toBe(2); + expect(reportsKeys(key('k'))).toBe(true); + expect(useReports.getState().cursor).toBe(1); + expect(reportsKeys(key('J'))).toBe(true); + expect(useReports.getState().selected).toBe('other'); + expect(useReports.getState().cursor).toBe(0); + expect(reportsKeys(key('['))).toBe(true); + expect(useReports.getState().selected).toBe('notes'); + expect(reportsKeys(key('x'))).toBe(false); + + // A block without an anchor says so; one with it opens the diff there, `c` with a comment. + useReports.setState({ cursor: 0 }); + expect(reportsKeys(key('Enter'))).toBe(true); + expect(review.flash).toHaveBeenCalledWith('This block points at no line of the diff'); + useReports.setState({ cursor: 1 }); + review.selection = { id: 'src/app.py' }; + expect(reportsKeys(key('c'))).toBe(true); + await vi.waitFor(() => expect(review.openDraft).toHaveBeenCalledWith({ id: 'src/app.py' })); + expect(review.goToLink).toHaveBeenCalledWith('src/app.py', 3); + expect(shown.ids).toEqual(['diff']); + }); + + it('jumps to a removed line without a draft', async () => { + review.openDraft.mockClear(); + await useReports.getState().comment({ path: 'src/app.py', side: 'old', line: 4, endLine: 5 }); + expect(review.openFile).toHaveBeenCalledWith('src/app.py', 5, 'old'); + expect(review.openDraft).not.toHaveBeenCalled(); + expect(review.flash).toHaveBeenCalledWith('Press c on the removed line to comment there'); + }); +}); diff --git a/test/client/store.test.ts b/test/client/store.test.ts index 6d9248d2..2700fae3 100644 --- a/test/client/store.test.ts +++ b/test/client/store.test.ts @@ -64,6 +64,7 @@ const api = { lineWrap: true, lspCommands: { python: 'pyrefly lsp' }, githubHosts: {}, + plugins: {}, })), lspStatus: vi.fn(async (): Promise => LSP_OFF), lspLog: vi.fn(async (): Promise => ({ entries: [] })), @@ -823,6 +824,7 @@ describe('client transitions', () => { lineWrap: true, lspCommands: {}, githubHosts: {}, + plugins: {}, }, lsp: LSP_OFF, }); @@ -835,6 +837,7 @@ describe('client transitions', () => { lineWrap: true, lspCommands: {}, githubHosts: {}, + plugins: {}, }); api.lspStatus.mockResolvedValueOnce(lspStatus('ready')); const boot = useStore.getState().boot(); @@ -1932,6 +1935,7 @@ describe('request ownership', () => { lineWrap: true, lspCommands: {}, githubHosts: {}, + plugins: {}, }); const load = deferred(); const save = deferred(); diff --git a/test/e2e/__snapshots__/reports.spec.ts/generator.png b/test/e2e/__snapshots__/reports.spec.ts/generator.png new file mode 100644 index 00000000..a7d1f506 Binary files /dev/null and b/test/e2e/__snapshots__/reports.spec.ts/generator.png differ diff --git a/test/e2e/__snapshots__/reports.spec.ts/list.aria.yml b/test/e2e/__snapshots__/reports.spec.ts/list.aria.yml new file mode 100644 index 00000000..f2e6e9d6 --- /dev/null +++ b/test/e2e/__snapshots__/reports.spec.ts/list.aria.yml @@ -0,0 +1,4 @@ +- complementary "Reports list": + - text: Reports 2 + - 'button /Fake report old: \.gitattributes, \.gitignore, README\.md, assets\/logo\.png, docs\/release notes\.md, docs\/Übersicht\.md, pyproject\.toml, requirements\.lock, scripts\/tally[\d,.]+[hmsp]+, tally\/__init__\.py, tally\/cli\.py, tally\/gener/' + - button "Review notes Watch the refund path" \ No newline at end of file diff --git a/test/e2e/__snapshots__/reports.spec.ts/pushed.aria.yml b/test/e2e/__snapshots__/reports.spec.ts/pushed.aria.yml new file mode 100644 index 00000000..69cc777b --- /dev/null +++ b/test/e2e/__snapshots__/reports.spec.ts/pushed.aria.yml @@ -0,0 +1,10 @@ +- region "Reports view": + - text: Review notes just now + - button "Log" + - article: + - button "tests/test_refunds.py:1" + - button "Comment" + - heading "Watch the refund path" [level=1] + - paragraph: The new test only covers the happy path. + - article: + - paragraph: A general remark, pointing nowhere. \ No newline at end of file diff --git a/test/e2e/__snapshots__/reports.spec.ts/pushed.png b/test/e2e/__snapshots__/reports.spec.ts/pushed.png new file mode 100644 index 00000000..1a0d5ace Binary files /dev/null and b/test/e2e/__snapshots__/reports.spec.ts/pushed.png differ diff --git a/test/e2e/fixtures.ts b/test/e2e/fixtures.ts index 14f0724c..6a04b9fd 100644 --- a/test/e2e/fixtures.ts +++ b/test/e2e/fixtures.ts @@ -20,6 +20,8 @@ export interface Options { args: string[]; /** Extra environment for the diffle process. */ env: NodeJS.ProcessEnv; + /** The user config the server starts with; empty by default. */ + config: object; } export interface Fixtures { @@ -37,6 +39,7 @@ export const test = base.extend({ revs: [[`${MAIN_BRANCH}...${FEATURE_BRANCH}`], { option: true }], args: [[], { option: true }], env: [{}, { option: true }], + config: [{}, { option: true }], // Copy the complete checkout, including staged/unstaged files; git clone would lose working-mode inputs. fixtureTemplate: [ @@ -65,8 +68,8 @@ export const test = base.extend({ } }, - diffle: async ({ repo, revs, args, env }, use) => { - const server = await startDiffle({ repo, revs, args, env }); + diffle: async ({ repo, revs, args, env, config }, use) => { + const server = await startDiffle({ repo, revs, args, env, config }); try { await use(server); } finally { diff --git a/test/e2e/reports.spec.ts b/test/e2e/reports.spec.ts new file mode 100644 index 00000000..05c33c61 --- /dev/null +++ b/test/e2e/reports.spec.ts @@ -0,0 +1,82 @@ +// The Reports view: what a plugin made of the snapshot and what was pushed over the API, beside the +// diff. The fake generator (`test/reports/fake-generator.mjs`) stands in for a plugin: it lists the +// files it was given and renders the fixture's logo as one page per side, so the pages block runs +// through the real runner, blob store, file route and image comparison without Typst. +import { join } from 'node:path'; +import { activePath } from './browser.js'; +import { expect, test } from './fixtures.js'; +import { REPO_ROOT } from './server.js'; + +const FAKE = join(REPO_ROOT, 'test/reports/fake-generator.mjs'); +const ANCHORED = 'tests/test_refunds.py'; + +/** Pushes a report as an agent would: the JSON straight at the API. */ +async function pushReport(url: string, id: string, report: object): Promise { + const response = await fetch(new URL(`api/reports/${id}`, url), { + method: 'PUT', + headers: { 'content-type': 'application/json' }, + body: JSON.stringify(report), + }); + if (!response.ok) throw new Error(`pushing ${id} failed: HTTP ${response.status} ${await response.text()}`); +} + +test.describe('reports', () => { + test.use({ + config: { + plugins: { + fake: { + command: `${JSON.stringify(process.execPath)} ${JSON.stringify(FAKE)} json`, + when: ['VERSION'], + scope: 'tree', + }, + }, + }, + }); + + test('gr shows the generator’s pages and a pushed report; Enter and c leave for the diff at a block’s line', async ({ + page, + diffle, + }) => { + await pushReport(diffle.url, 'notes', { + title: 'Review notes', + blocks: [ + { + kind: 'markdown', + text: '# Watch the refund path\n\nThe new test only covers the happy path.', + anchor: { path: ANCHORED, line: 1 }, + }, + { kind: 'markdown', text: 'A general remark, pointing nowhere.' }, + ], + }); + await page.keyboard.press('g'); + await page.keyboard.press('r'); + const view = page.getByRole('region', { name: 'Reports view' }); + const list = page.getByRole('complementary', { name: 'Reports list' }); + await expect(view).toBeVisible(); + + // The generator's report comes first and shows once its run ended: the files it saw, then the pages. + await expect(view.getByText(/^old: /)).toBeVisible({ timeout: 20000 }); + await expect(view.getByText('Loading image…')).toHaveCount(0); + await expect(view.getByText('Old · 128 × 96')).toBeVisible(); + await expect(view.getByText('New · 96 × 96')).toBeVisible(); + await expect(list).toMatchAriaSnapshot({ name: 'list.aria.yml' }); + await expect(page).toHaveScreenshot('generator.png'); + + // The pushed one lists beside it; its anchored block names its line, the other does not. + await list.getByRole('button', { name: /Review notes/ }).click(); + await expect(view.getByText('The new test only covers the happy path.')).toBeVisible(); + await expect(view).toMatchAriaSnapshot({ name: 'pushed.aria.yml' }); + await expect(page).toHaveScreenshot('pushed.png'); + + // Enter opens the diff at the block's line; gr brings the report back, and c comments on that line. + await page.keyboard.press('Enter'); + await expect(view).toBeHidden(); + await expect.poll(() => activePath(page)).toBe(ANCHORED); + await page.keyboard.press('g'); + await page.keyboard.press('r'); + await expect(view).toBeVisible(); + await page.keyboard.press('c'); + await expect(view).toBeHidden(); + await expect(page.getByPlaceholder('Leave a comment…')).toBeVisible(); + }); +}); diff --git a/test/e2e/server.ts b/test/e2e/server.ts index 7d3be57c..3ddb346f 100644 --- a/test/e2e/server.ts +++ b/test/e2e/server.ts @@ -1,7 +1,7 @@ import { execFileSync, spawn, type ChildProcess } from 'node:child_process'; import { createHash } from 'node:crypto'; import { existsSync, readFileSync, readdirSync, rmSync, statSync, writeFileSync } from 'node:fs'; -import { mkdtemp } from 'node:fs/promises'; +import { mkdir, mkdtemp, writeFile } from 'node:fs/promises'; import { tmpdir } from 'node:os'; import { rmTmp } from '../tmp.js'; import { dirname, join, relative, resolve } from 'node:path'; @@ -19,6 +19,8 @@ export interface DiffleOptions { /** Further CLI flags. */ args?: string[]; env?: NodeJS.ProcessEnv; + /** The user config this server starts with, as the file holds it; none by default. */ + config?: object; /** The diffle checkout to run, server and client; this one by default. See `baseCheckout`. */ root?: string; timeoutMs?: number; @@ -39,10 +41,15 @@ export async function startDiffle({ revs = [], args = [], env = {}, + config, root = REPO_ROOT, timeoutMs = 30000, }: DiffleOptions): Promise { const configDir = await mkdtemp(join(tmpdir(), 'diffle-e2e-config-')); + if (config) { + await mkdir(join(configDir, 'diffle'), { recursive: true }); + await writeFile(join(configDir, 'diffle', 'config.json'), JSON.stringify(config)); + } const [program, ...launcher] = diffleCommand(root); const proc = spawn( program!, diff --git a/test/reports/fake-generator.mjs b/test/reports/fake-generator.mjs new file mode 100644 index 00000000..f7b00707 --- /dev/null +++ b/test/reports/fake-generator.mjs @@ -0,0 +1,56 @@ +// Minimal report generator for the server tests. Reads the review from stdin and the copied sides +// from DIFFLE_OLD and DIFFLE_NEW. `json` (the default) writes report.json with what it saw and one +// PNG page into DIFFLE_OUT; `stdout` prints one markdown line for DIFFLE_FILE instead; `fail` exits 3. +import { existsSync, readdirSync, readFileSync, statSync, writeFileSync } from 'node:fs'; +import { join } from 'node:path'; + +const mode = process.argv[2] ?? 'json'; +const input = JSON.parse(readFileSync(0, 'utf8')); + +function walk(dir, prefix = '') { + let out = []; + for (const name of readdirSync(dir).sort()) { + const rel = prefix ? `${prefix}/${name}` : name; + if (statSync(join(dir, name)).isDirectory()) out = out.concat(walk(join(dir, name), rel)); + else out.push(rel); + } + return out; +} + +const old = walk(process.env.DIFFLE_OLD); +const next = walk(process.env.DIFFLE_NEW); +// The matched files' new text, for a test to check the copy; the rest of a tree would be noise. +const contents = input.files + .filter((f) => existsSync(join(process.env.DIFFLE_NEW, f.path))) + .map((f) => `${f.path}=${readFileSync(join(process.env.DIFFLE_NEW, f.path), 'utf8').trim()}`); + +if (mode === 'fail') { + process.stderr.write('boom: no such thing\n'); + process.exit(3); +} +if (mode === 'stdout') { + console.log(`${process.env.DIFFLE_FILE ?? 'all'}: old=${old.join(',')} new=${next.join(',')}`); + process.exit(0); +} +// A side's logo, when the repository has one (the e2e fixture does), else bytes that only sniff as PNG. +const stub = Buffer.concat([ + Buffer.from([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]), + Buffer.from(contents.join(';')), +]); +const logo = (dir) => (existsSync(join(dir, 'assets/logo.png')) ? readFileSync(join(dir, 'assets/logo.png')) : null); +writeFileSync(join(process.env.DIFFLE_OUT, 'page.png'), logo(process.env.DIFFLE_NEW) ?? stub); +const oldLogo = logo(process.env.DIFFLE_OLD); +if (oldLogo) writeFileSync(join(process.env.DIFFLE_OUT, 'old.png'), oldLogo); +const report = { + title: 'Fake report', + blocks: [ + { + kind: 'markdown', + text: `old: ${old.join(', ')}\nnew: ${next.join(', ')}\nscope: ${input.scope}\nfiles: ${input.files.map((f) => f.path).join(', ')}\ncontents: ${contents.join('; ')}`, + anchor: { path: input.files[0]?.path ?? 'none', line: 1 }, + }, + { kind: 'pages', label: 'Rendered', pages: [{ label: 'p. 1', old: oldLogo ? 'old.png' : null, new: 'page.png' }] }, + ], +}; +writeFileSync(join(process.env.DIFFLE_OUT, 'report.json'), JSON.stringify(report)); +process.stderr.write('rendered 1 page\n');