Skip to content

Add the term store of rulean-store - #24

Merged
tamnd merged 1 commit into
mainfrom
m1/store
Nov 21, 2026
Merged

tamnd merged 1 commit into
mainfrom
m1/store

Conversation

@tamnd

@tamnd tamnd commented Nov 21, 2026

Copy link
Copy Markdown
Owner

This PR adds the term store of spec 04 to rulean-store. It is on top of #23, because the store keeps natural number literals as rulean_nat::Nat.

What the store has:

  • ExprId is a u32 with the tier in bit 31, the kind in bits 27 to 30 and the index in bits 0 to 26 (spec 04.2.1). The kind needs no memory load.
  • One node array for each of the 10 kinds. An application node is 8 bytes.
  • A side record of 8 bytes for each node: the hash, the loose bound variable range (24 bits, saturated) and the flags (free variable, level parameter, let, literal, projection). The store computes the record when it makes the node.
  • Table, the open-addressing table of 04.2.4. It maps a hash to u32 values, and the caller compares its own key on each value with the same hash. So the table needs no closure and no trait, and a lookup never trusts a hash alone. The table can carry epochs, so that clear is O(1) for the scratch tier.
  • Interning of strings, names, levels, level lists, natural number literals and expressions. NameId(0) is the anonymous name and LevelId(0) is zero, as in the export.
  • The hash is a MurmurHash3 step with no random seed. It is structural: a node hash mixes the hashes of its fields, not their handles. So the same term in two stores has the same hash. The hash of a literal covers all limbs.
  • Store::memory gives the bytes of each part, for the measure of 04.9.

The store does not normalize levels. That is the job of the checker in M2. The scratch tier comes with the checker too. This PR has only the frozen tier, but the handle already has the tier bit.

Capacity. When the array of one kind has 2^27 nodes, the store sets a flag and returns a handle to an old node. The terms are then wrong, so the driver must check Store::is_full and decline before it gives a verdict. The crate documentation says this.

Subset. The code stays in the subset of spec 10.6. The table method is put and not insert, because the subset lint reports .insert() on any receiver. rotate_left, wrapping_mul and is_empty are outside the modelled functions, so the code has its own forms of them.

This PR covers the two store items of the checklist. The twin and the R2 lemmas for the store are a separate item.

Refers to #2

@tamnd tamnd added this to the M1 Store and parser milestone Nov 21, 2026
@tamnd tamnd added area/store Handles, the term store, names and levels type/feature New function labels Nov 21, 2026
Base automatically changed from m1/nat to main November 21, 2026 15:35
@tamnd
tamnd merged commit 79276aa into main Nov 21, 2026
16 of 17 checks passed
@tamnd
tamnd deleted the m1/store branch November 21, 2026 15:38
@tamnd tamnd mentioned this pull request Nov 21, 2026
6 of 16 tasks
tamnd added a commit that referenced this pull request Nov 21, 2026
Refers to #2.

This PR adds the lean4export NDJSON 3.x parser of spec 07.1 to
`rulean-format`. It is stacked on #24, because it writes into the term
store.

## What it does

- `json.rs` is the JSON layer. It reads one object into a table of known
keys, and it reads integers, booleans, strings and arrays. It checks
every other value for well-formed JSON and then skips it. The skip keeps
the nesting on a stack, so a deep `mdata` value uses no call stack.
- `ndjson.rs` is the record parser. A name, level or expression record
goes into the store, and its handle goes into the index table of its
kind. A declaration record becomes a `Decl`.
- `decl.rs` is the declaration stream of spec 04.7. It keeps every field
of the input, also the fields that the checker computes again. The
checker compares them and does not trust them (06.1).
- `error.rs` has the error kinds. Each kind is a reject or a decline. An
unknown version, an index above the table limit and a full store are
declines.

## Input rules of 07.1.2

- An index must be defined before it is used, and it must not be defined
two times. Gaps and any order are permitted. Name 0 and level 0 are
implicit.
- Duplicate keys are rejects, also when one key has an escape.
- An integer with a sign, a fraction or an exponent is a reject. A value
above 2^64 - 1 is a reject. A count field above 2^32 - 1 is a reject.
- Each line must be valid UTF-8. A `\u` surrogate without its pair is a
reject.
- `natVal` has no leading zeros except for "0". The parser checks the
bound of 05.8 from the string length before the conversion, and from the
bit count after it.
- `mdata` is stripped. The record index gets the handle of the inner
expression.
- An empty line, a truncated last line, a missing meta line and a second
meta line are rejects. Major version 3 with minor version 0 or 1 is
accepted.

## Tests

- `tests/rules.rs` has one test for each rule above.
- `tests/rules.rs` also changes, deletes and adds bytes in a valid input
20,000 times. No run panics.
- `tests/arena.rs` parses all Arena tests when `RULEAN_ARENA` or
`_tmp/arena/tests` exists. Locally, all 225 tests parse, with 16,809
declarations. CI does not fetch the Arena in the test job yet. The next
PR adds `rulean check --parse-only` on the Arena in CI.

## Notes

- The code is in the Aeneas subset: no `?`, no closures and no early
return in a loop. The first error of a line goes into the scanner state,
and the rest of the line runs on dummy values. The parser writes into
the store only when there is no error.
- Performance is not tuned yet. In release mode the 50 MB of the Arena
parse in about 1.8 s. The target of 07.1.4 is for M4.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/store Handles, the term store, names and levels type/feature New function

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant