grammardoc: render the parse table as a manual

[zmaril]

Branches off main (post-#116), independent of the grammar stack.

Live example: Python Rulebook — all 173 productions.

Owning the grammars means owning their documentation, and the honest way to document a parser is to render the parse table rather than describe it.

python3 tools/grammardoc/emit.py crates/treebank-python /tmp/python.html
python3 tools/grammardoc/emit.py --check crates/treebank-python   # what CI runs

553 productions across the three grammars, ~70 ms each, stdlib only, no per-language code.

Why it's small

The input is src/grammar.json, not grammar.js. grammar.js is arbitrary JavaScript and reading it means running it. grammar.json is what tree-sitter generate normalises it into, and it is already an EBNF syntax tree over sixteen node kinds — SEQ, CHOICE, REPEAT, REPEAT1, SYMBOL, STRING, PATTERN, BLANK, FIELD, ALIAS, TOKEN, IMMEDIATE_TOKEN, RESERVED, and the four PREC forms.

So rendering is a fold over sixteen cases, not a parse. And because it reads the generated grammar, the page cannot drift from the parser: if a production is on the page, the parse table has it.

What it shows that a BNF listing can't

• Precedence in place — drawn around the production it applies to, as well as tabulated. EBNF cannot express precedence at all, which is why every language manual prints it separately.
• Fields — as captions inside the boxes, so the edge names a query can use sit next to the shape they attach to.
• Externals — the plum boxes are the external scanner, i.e. the part no diagram can explain because it is hand-written C.
• Vocabulary — productions are grouped under the supertype they answer, not listed alphabetically.

Decisions

Hidden rules stay visible. Tempting to inline _or_test and friends, but that chain is the precedence structure — it is what the MySQL manual shows as expr → boolean_primary → predicate → bit_expr → simple_expr. Inlining deletes the most informative part of the page.

The comma-list idiom is collapsed. seq(X, repeat(seq(',', X))) is recognised and drawn as one loop instead of five boxes. Without it roughly half the diagrams are unreadable.

The mono face is embedded, for a functional reason. SVG box widths are computed in the tool from a fixed character advance (DejaVu Sans Mono, 0.60205 em); a fallback in the browser would clip every label. ~1.3 MB per page, --no-fonts to drop it.

The CI check has teeth

to_rr and to_ebnf are total over grammar.json's node kinds and raise on one they do not know. Verified against a synthetic grammar carrying an invented node type:

$ emit.py --check <grammar with FUTURE_DSL_THING>
unhandled grammar node FUTURE_DSL_THING
exit 1

Without that, a missing case silently omits part of a production from the docs — which nobody would notice.

On the layout engine

railroad.py sizes a node in its constructor and draws it later, and the two must agree. Both bugs found while writing it were a choice sized with one formula and drawn with another, which reads as plausible source and as obvious nonsense in a picture. Offsets are now computed once in __init__ and read back by draw; preview.py rasterises a diagram offline so a layout change can be checked by looking at it.

Follow-ups, deliberately not here

• Rust and TypeScript pages generate identically; only Python is published above.
• Could become treebank docs <grammar> if the CLI should own it — it is Python today because that matches the existing tools/ and oracle scripts.
• Nothing publishes these anywhere yet; CI only checks that they render.

🤖 Generated with Claude Code

https://claude.ai/code/session_015ZQR1QTvqArX8vMxYKK3rv