Repository navigation
feat: headless diffy-compare, wire format, bundled grammars and @seatedro/diffy - #51
Merged
Merged
Conversation
…hain The headless crates the compare chain is built from (diffy-diff, diffy-semantic, diffy-syntax, diffy-compare) are MIT OR Apache-2.0; the app stays GPL-3.0.
Project compare results into a layout-neutral, serde-serializable display format for JavaScript clients: hunks as context/removed/added/modified rows with UTF-16 change and syntax-class spans, collapsed gaps with an expand entry point, hunk headers from the semantic outline or the grammar's scope table, outline, moves, renames, and stable fallback kinds. FORMAT.md is the client-facing spec, checked against the types by a test.
A headless consumer (the Node binding) needs grammars with no runtime packs and no download step. `syntax-pack bundle` builds and checks each language as a host pack from its pinned, hashed sources (which adapts its queries), compiles the same sources into a static library per grammar for the target, links them together once to catch clashing symbols, and writes bundle.tsv, the queries, and THIRD_PARTY_NOTICES.md. diffy-syntax's `bundled` feature reads that directory from DIFFY_SYNTAX_BUNDLE in a dependency-free build.rs, links the libraries, and generates the table; without the variable it builds an empty bundle with a cargo warning. `StoreConfig::bundled()` adds the grammars after local packs and before downloads; they share tagging, queries, the ABI check and catalog reporting with packs, through `Grammar::compile` now taking the language function. `bundled::languages()` lists them. The app does not enable the feature.
diffy-compare now implements diff_file/diff_files: grammar lookup and rule compilation, injection embedding, the semantic diff, rows-to-document building, and the line diff (histogram + Git slider heuristics via gix-imara-diff, the same crate and version gix links). Documents are byte-for-byte what the app built before. New: per-file semantic budgets capped by a request deadline (files after it get line diffs with Fallback::RequestBudget) and cancellation between files (Error::Cancelled). The app keeps only the Git-facing parts in compare::semantic, uses FileInput in place of ChangedPath, reports progress and energy work through an Observer, and logs fallbacks from FileDiff. semantic-eval calls diffy-compare directly.
A napi-rs binding over diffy-compare for Node servers and Electron: diff() and expand() run each request on its own thread (never libuv's pool), settle a promise through a thread-safe function, and return the versioned display format as JSON. An AbortSignal cancels a request and a request budget falls back to line diffs. Grammars are compiled in from a syntax-pack bundle. The workspace gains a `node` profile (release with panic = "unwind") so a Rust panic rejects the call instead of aborting the host process. The app binary is unchanged. CI builds darwin-arm64, darwin-x64 (cross), linux-x64-gnu and linux-arm64-gnu (inside manylinux_2_28, glibc <= 2.28 checked) and win32-x64-msvc, tests each natively, and publishes on node-v* tags with npm provenance.
…quest budget wire::document highlighted every file serially after the diff, so an 876-file request with syntax took 10.9s and ran past budget_ms. Files are now projected on workers, and WireOptions gains a deadline and cancel flag past which files skip syntax spans and scope headers. The Node binding passes both; the gap test uses a file long enough to have gaps.
diff calls now wait in a process-wide FIFO queue: at most
max_concurrent_requests run at once (default: available cores / 4, at
least 1, since a request fans out to four workers). Waiting holds no
thread; a request's thread takes the next queued one when it finishes.
A request aborted while queued rejects at once and never starts, and
budget_ms keeps counting from the call. expand is not queued.
configure({ max_concurrent_requests }) sets the limit. diffJson resolves
to the document's JSON text, unparsed, for servers that forward it.
scripts/notices.mjs (npm run notices) lists the Rust crates diffy-node links on a target, with each one's license expression, repository, and the license files it ships, printing each distinct text once. The crate set comes from cargo tree, which resolves features for diffy-node alone; cargo metadata supplies package details. It fails, writing nothing, when a crate's license is not on a permissive allowlist. CI writes it into every platform package, and publishing checks it is there.
Outline items carried only a start line, so the wire projection named a hunk after the nearest changed item starting above it, and a change in a top-level statement below a function got that function's name. diffy_diff::OutlineItem now holds the item's line range on its side (`lines`, from the node diffy-semantic already measures), and the header is the innermost changed item containing the hunk's first changed line with text: items on the change's side first, then the other side, where a pure insertion or deletion must fall strictly inside the item. Leading blank lines are skipped so an added or removed item names itself rather than nothing. Without a containing item the scope table fallback is unchanged (still only with syntax), and otherwise there is no header. The format gains an optional `OutlineItem.end_line` (inclusive), so FORMAT_VERSION stays 1.
Add a `runtime` feature (tree-sitter, highlighting, injections, scopes, symbols, GrammarStore/StoreConfig) under `engine`, which now adds only local pack loading (libloading, ring, serde/serde_json manifests, thiserror). `bundled` depends on `runtime`, so a bundled-only build links no pack loader and has no `StoreConfig::local_packs`. Grammar compilation reports a crate-private GrammarError that converts into the same PackError variants, so pack errors are unchanged. diffy-compare needs only `runtime` (its tests take `engine` for `diffy_syntax::testing`); the app, diffy-ui and syntax-pack keep `engine`/`download` and their behaviour. @diffy/node drops libloading 0.8, ring, untrusted, getrandom and thiserror: the stripped linux-arm64 addon goes from 21,694,360 to 21,577,000 bytes (-115 KiB).
syntax-pack keeps its own unmodified Apache-2.0 text for crediting nvim-treesitter's queries.
Drops narration, history, measurements and restated code across the new crates, the binding, the workflow and the docs; keeps invariants, coordinates, contracts and the reasons behind surprising choices. No code changes.
seatedro
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Splits the code that turns a changed file's bytes into a diff out of the app, so a server (T3) can use it from Node.
What's in it
crates/diffy-compare— the headless compare chain, moved out ofsrc/core/compare/semantic.rs. It has no quark, UI or git dependencies. Input: status, old/new paths, old/new bytes. Output: a semantic diff where a grammar exists, otherwise a line diff plus the fallback reason. Each file gets a 400 ms budget; a request can also set a total deadline and a cancel flag. The app now calls this crate, with identical output: 4,306 file diffs matched byte for byte against the old code.diffy_compare::wire, behind theserdefeature) — versioned (v1) and documented incrates/diffy-compare/FORMAT.md. It covers rows with old and new line numbers, changed-word spans with intensity, moves, renames, outline items, hunk headers taken from the item that contains the change, optional syntax spans as class names, and gaps the client can expand. Span offsets are UTF-16 code units.@seatedro/diffy(bindings/node) — a napi-rs binding withdiff,diffJson,expand,languagesandconfigure.AbortSignalrejects with anAbortError;budget_mscaps the whole request.nodebuild profile usespanic = "unwind").bundledfeature ondiffy-syntax, plus asyntax-pack bundlecommand that statically compiles a fixed language set from the pinned, hash-checked sources. Bundled builds carry no pack-loading code (the runtime/engine feature split). The app still uses runtime packs.diffy-diff,diffy-semantic,diffy-syntax,diffy-compareand the binding are nowMIT OR Apache-2.0; the app stays GPL-3.0. Each package shipsTHIRD_PARTY_NOTICES.mdfor the grammars andRUST_NOTICES.mdfor the Rust crates compiled in.Testing
cargo test --workspace --exclude difftastic, with grammar packs required: 682 passed.cargo clippy --workspace --exclude difftastic --all-targets --all-features: no new warnings.JSON.parse(diffJsonavoids it).Not verified
node.ymlworkflow has never run, so the macOS, Windows, x64 Linux and manylinux builds are untested.cargo clippy --all-featurescan't build the vendoreddifftasticcrate here (itsdifftbinary fails to link), so it was excluded.Size
The full grammar bundle adds about 18.8 MB (stripped); Kotlin, C++ and Swift account for about 12 MB of that. The
.nodeis 21.6 MB.