WillBooster/tree-sitter

This is a fork of tree-sitter/tree-sitter. We are grateful to its authors and contributors. This is not an official release of that project.

This fork provides the Tree-sitter parsing library for WillBooster's products in two forms, and fixes the bugs they run into:

• the Rust crate willbooster-tree-sitter, used by code-gauge and ultra-uni, natively and compiled to Wasm;
• the npm package @willbooster/web-tree-sitter, which runs in browsers and in Cloudflare Workers.

The CLI and the parser generator under crates/ build the test fixtures and are not published as packages. The GitHub Releases from v1.0.7 on carry the CLI as tree-sitter-cli-<platform>.tar.gz for linux-x64, linux-arm64, macos-arm64, and macos-x64, so that the grammar repositories can generate their parsers with this generator and run tree-sitter fuzz on this runtime without building it. A workflow attaches the archives after a release is published, so the newest release lacks them for a while.

Development setup

Install mise and a C/C++ compiler. From a fresh checkout, run:

mise install
mise exec -- bun install --frozen-lockfile
mise exec -- bun run verify

Run mise install before activating the project environment: it installs the configured tools and trusts the repository configuration.

Keep Rust toolchain changes in rust-toolchain.toml; a rust tool entry in mise.toml overrides that file.

Parser generation and tuning

The CLI generates parsers with shared parse rows and lookahead bitsets, bitmap-ranked sparse lookup, compact metadata, and language ABI 0x57420007, the only ABI this fork's runtime accepts. The upper 16 bits (0x5742, "WB") identify the fork; the lower 16 bits count its ABI revisions. Generate every parser with this fork's CLI from the same release as the runtime, then rebuild native libraries and Wasm files. Parsers generated for other ABI versions are rejected, including those generated by upstream's CLI. Upstream's runtime rejects parsers generated here. Upgrade the runtime and grammar artifacts together.

Completed-token keyword lookup shares a compact ASCII table with the keyword lexer. Grammars with Unicode whitespace extras retain this lookup path when no extra overlaps the beginning of a word token. Otherwise, those grammars use branch-based keyword lexing to keep streaming and contiguous input consistent. Keywords containing non-ASCII characters use branch-based keyword lexing.

Keyword extraction applies to matching tokens regardless of their initial character, including digits and punctuation. If the declared word token includes punctuation that should instead separate raw operators or numbers, mark those lexical rules with token.nonkeyword.

Use token.nonkeyword(rule) when a token must match directly instead of being extracted as a keyword of the grammar's word token. For example, seq(token.nonkeyword('if'), $.identifier) can split ifname into if and name; an ordinary 'if' remains subject to whole-word recognition. This is a lexical choice for that token and does not change the grammar's other keywords. Wrap the result in alias to rename its node. Use token.nonkeyword(prec(2, 'if')) to assign lexical precedence, or token.nonkeyword(token.immediate(prec(2, 'if'))) to also prohibit preceding extras. Nested token wrappers preserve the ordinary token rules for lexical precedence and immediacy. JSON grammars use {"type":"NONKEYWORD_TOKEN","content":...}.

A reserved-word set must identify the same lexical declaration used in the rules. Ordinary and opted-out tokens with the same text are distinct. To reserve an opted-out token, define it once as a named rule, such as raw_if: () => token.nonkeyword('if'), and reference $.raw_if in both the set and the rule body. Matching inline token.nonkeyword('if') declarations also work; a plain 'if' set entry does not identify that opted-out token. Reservation preserves its direct-matching behavior.

To record parse-action and lexer-entry frequencies, generate an unprofiled parser first. From the grammar directory:

tree-sitter generate
tree-sitter profile --output profile.json --corpus test/corpus
tree-sitter generate --profile profile.json
tree-sitter build

A profile applies only to the exact unprofiled generated source it was recorded from. Regenerate and record again after changing the grammar, generation options, or generator version. Pass --disable-optimizations to both generation and recording when using that generation mode. Corpus examples marked as skipped, excluded on this platform, or restricted to other languages are omitted. Language tags match the selected grammar's names in tree-sitter.json, including aliases sharing its path; without a matching configuration, they match the grammar name. Unknown tags produce warnings. Successful recording reports the examined corpus examples, recorded corpus/file counts, and parse-action total. Recording warns when no parse actions were captured. Empty and malformed inputs remain accepted for EOF, lexical, and recovery training, but zero-action profiles provide no frequency guidance for parse-state ordering.

profile accepts additional source files and repeated --corpus paths. Corpus, source, and output paths resolve relative to the process working directory, even when --grammar-path selects a different directory. For example, from a repository root with a grammar in typescript/ and shared examples in test/corpus/:

tree-sitter generate typescript/grammar.js
tree-sitter profile --grammar-path typescript --output profile.json --corpus test/corpus
tree-sitter generate typescript/grammar.js --profile profile.json

Profiled generation also compresses sparse parse rows that training never visited, while retaining the lookup encoding of observed rows. Shared rows keep the encoding chosen for an observed state. A profile with no recorded parse actions does not enable this compression.

Compare fresh and incremental parsing on held-out valid and malformed inputs before choosing the profile-guided layout. The parser_study example measures both and reports tree hashes and error status:

cargo run --release -p tree-sitter-cli --features wasm --example parser_study -- native <language> <src-directory> <input>...
cargo run --release -p tree-sitter-cli --features wasm --example parser_study -- wasm <language> <wasm-file> <input>...

Add --profile <output.json> before native or wasm to record frequencies from these inputs. Recording requires an unprofiled parser. Native recording reads <src-directory>/parser.c; Wasm recording reads src/parser.c in the module's directory, which must contain the exact unprofiled source used to build that module.

build --optimization selects 2, 3, s, or z for native or Wasm builds. The defaults are native 2 and Wasm s. Compare build time, artifact size, and parsing time for the target compiler and runtime.

Tree-sitter is a parser generator tool and an incremental parsing library. It can build a concrete syntax tree for a source file and efficiently update the syntax tree as the source file is edited. Tree-sitter aims to be:

• General enough to parse any programming language
• Fast enough to parse on every keystroke in a text editor
• Robust enough to provide useful results even in the presence of syntax errors
• Dependency-free so that the runtime library (which is written in pure C) can be embedded in any application

External scanner columns

External scanners can call lexer->get_display_column(lexer, tab_width) to obtain the zero-based display column, including tabs inside previously lexed tokens. An initial BOM is an encoding marker of width zero. Every other Unicode scalar occupies one column; tabs advance to the next multiple of tab_width (zero means one), and LF, CR, and form feed reset the column. Columns saturate at UINT32_MAX. Only included source ranges contribute: columns accumulate across gaps on the same row and reset when the next included range starts on a different row. The call preserves the lexer position, lookahead, and token boundaries; get_column retains its existing scalar-column behavior.

Restricted rules with canonical supertype queries

This fork supports aliasing one declared supertype to another while keeping both hidden in the syntax tree. This lets a restricted grammar context retain queries such as (expression/identifier) without introducing a visible wrapper node. Generate the parser and run its queries with this fork's generator and runtime.

Declare both rules in supertypes, use the same named alias at every reference to the restricted rule, and leave the canonical supertype unaliased. Reference the canonical rule as a symbol in a reachable production, as $.expression in the general branch below does. A rule named only as an alias target is removed as unused, leaving an ordinary visible alias instead of a transparent supertype. The generated node schema combines the retained supertypes under the canonical name. Concrete-node aliases and inconsistent aliases still cannot reuse a canonical supertype's name.

supertypes: $ => [$.expression, $._restricted_expression],
rules: {
  source_file: $ => choice(
    seq('general', $.expression),
    seq('restricted', alias($._restricted_expression, $.expression)),
  ),
  expression: $ => choice($.identifier, $.number),
  _restricted_expression: $ => $.identifier,
  identifier: () => /[a-z]+/,
  number: () => /[0-9]+/,
},

Links

• Tree-sitter documentation (upstream)
• Rust binding
• Wasm binding