Skip to content

Add the NDJSON parser of rulean-format - #25

Merged
tamnd merged 3 commits into
mainfrom
m1/format
Nov 21, 2026
Merged

tamnd merged 3 commits into
mainfrom
m1/format

Conversation

@tamnd

@tamnd tamnd commented Nov 21, 2026

Copy link
Copy Markdown
Owner

Refers to #2.

This PR adds the lean4export NDJSON 3.x parser of spec 07.1 to rulean-format. It is stacked on #24, because it writes into the term store.

What it does

  • json.rs is the JSON layer. It reads one object into a table of known keys, and it reads integers, booleans, strings and arrays. It checks every other value for well-formed JSON and then skips it. The skip keeps the nesting on a stack, so a deep mdata value uses no call stack.
  • ndjson.rs is the record parser. A name, level or expression record goes into the store, and its handle goes into the index table of its kind. A declaration record becomes a Decl.
  • decl.rs is the declaration stream of spec 04.7. It keeps every field of the input, also the fields that the checker computes again. The checker compares them and does not trust them (06.1).
  • error.rs has the error kinds. Each kind is a reject or a decline. An unknown version, an index above the table limit and a full store are declines.

Input rules of 07.1.2

  • An index must be defined before it is used, and it must not be defined two times. Gaps and any order are permitted. Name 0 and level 0 are implicit.
  • Duplicate keys are rejects, also when one key has an escape.
  • An integer with a sign, a fraction or an exponent is a reject. A value above 2^64 - 1 is a reject. A count field above 2^32 - 1 is a reject.
  • Each line must be valid UTF-8. A \u surrogate without its pair is a reject.
  • natVal has no leading zeros except for "0". The parser checks the bound of 05.8 from the string length before the conversion, and from the bit count after it.
  • mdata is stripped. The record index gets the handle of the inner expression.
  • An empty line, a truncated last line, a missing meta line and a second meta line are rejects. Major version 3 with minor version 0 or 1 is accepted.

Tests

  • tests/rules.rs has one test for each rule above.
  • tests/rules.rs also changes, deletes and adds bytes in a valid input 20,000 times. No run panics.
  • tests/arena.rs parses all Arena tests when RULEAN_ARENA or _tmp/arena/tests exists. Locally, all 225 tests parse, with 16,809 declarations. CI does not fetch the Arena in the test job yet. The next PR adds rulean check --parse-only on the Arena in CI.

Notes

  • The code is in the Aeneas subset: no ?, no closures and no early return in a loop. The first error of a line goes into the scanner state, and the rest of the line runs on dummy values. The parser writes into the store only when there is no error.
  • Performance is not tuned yet. In release mode the 50 MB of the Arena parse in about 1.8 s. The target of 07.1.4 is for M4.

@tamnd tamnd added this to the M1 Store and parser milestone Nov 21, 2026
@tamnd tamnd added area/format The NDJSON parser and the binary format type/feature New function labels Nov 21, 2026
Base automatically changed from m1/store to main November 21, 2026 15:38
The parser reads lean4export 3.x line by line into the term store and returns the declaration stream of spec 04.7. It checks the input rules of spec 07.1.2. The tests cover each rule, a run of changed inputs that must not panic, and all Arena tests when the Arena directory is present.
Aeneas does not translate a match on byte-string patterns, a match that gives string constants, or the loop state of skip_value. The key lookup now compares bytes in groups by the first byte. The message text is opaque in the extract. The loop body of skip_value is now in two functions.
@tamnd

tamnd commented Nov 21, 2026

Copy link
Copy Markdown
Owner Author

The extract job found three functions that Aeneas does not translate. I changed them in this PR:

  • keys::code used a match on byte-string patterns. It now selects a group by the first byte and then compares the bytes.
  • ErrorKind::text gives string constants. The text is for messages only, so the extract keeps it opaque with charon::opaque under --cfg rulean_extract.
  • Scanner::skip_value had a loop state that Aeneas could not find a fixed point for. The loop body is now in skip_open and skip_close.

The tests and the subset lint pass. I will merge when extract passes.

@tamnd
tamnd merged commit 5df264b into main Nov 21, 2026
17 checks passed
@tamnd
tamnd deleted the m1/format branch November 21, 2026 16:00
@tamnd tamnd mentioned this pull request Nov 21, 2026
6 of 16 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/format The NDJSON parser and the binary format type/feature New function

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant