perf(tree): decode leaf records on demand so the catalog fits in the node cache

docs/architecture/overview.md

@@ -154,7 +154,7 @@ sequenceDiagram
 
 Notes:
 
-* Nodes are decoded once and then shared by every reader through `NodeCache`. It has 16 shards, each an open-addressing table keyed by page number plus a CLOCK (second-chance) ring, with a total byte budget of `node_cache_mb`, each node counted at an estimate of its heap size (`Node.heapBytes`). Lookups are lock-free; only admission takes a per-shard lock. Decoded nodes are immutable (`owner == null`), so sharing them needs no copying. See [pages.md](../storage/pages.md#caching-and-io-accounting).
+* Nodes are decoded once and then shared by every reader through `NodeCache`. Catalog and posting-list leaves are kept packed and decode each record the first time it is read. It has 16 shards, each an open-addressing table keyed by page number plus a CLOCK (second-chance) ring, with a total byte budget of `node_cache_mb`, each node counted at an estimate of its heap size (`Node.heapBytes`). Lookups are lock-free; only admission takes a per-shard lock. Decoded nodes are immutable (`owner == null`), so sharing them needs no copying. See [pages.md](../storage/pages.md#caching-and-io-accounting).
 * `IoTrace` is bound with `ScopedValue.where(...)` around a query. `recordRead` throws `HStoreException.limit` once page reads plus cache hits exceed the query's page budget (`query_page_budget`).
 * A tree changed since the last checkpoint starts with in-memory nodes (`Ref.Pending`). A read follows those directly and goes through `PagedNodeSource` only once it reaches a stored reference.
 * Historical reads use `TransactionManager.snapshotAt(generation, branch)`. It looks the generation up in the retained `history` map; see [catalog-and-generations.md](../storage/catalog-and-generations.md#history-retention-and-time-travel).
@@ -248,7 +248,7 @@ semantic/state                           53 B  persisted HNSW index state (datab
 
 | Path | Owner | Format |
 |---|---|---|
-| `FORMAT` | `StorageEngine.verifyFormat` | `Properties` with `format=7` (commits logged as key changes and tree pages written at checkpoints, page numbers resolved through a page directory, reference counts per page, small member trees inline in their edge record) and `page-size` (the maximum node size). Opening a directory written in an older format, or opening with a different page size, fails. |
+| `FORMAT` | `StorageEngine.verifyFormat` | `Properties` with `format=8` (leaf records framed by length, commits logged as key changes and tree pages written at checkpoints, page numbers resolved through a page directory, reference counts per page, small member trees inline in their edge record) and `page-size` (the maximum node size). Opening a directory written in an older format, or opening with a different page size, fails. |
 | `LOCK` | `StorageEngine.open` | An exclusive OS file lock. A second process gets `database ... is opened by another process`. |
 | `catalog/generations/%016x.cat` | `CatalogStore.publish` | `int magic 0x54414348`, `int crc32c(payload)`, `long length`, then the `CatalogImage` payload. Written to `*.tmp`, fsynced, atomically renamed, and the directory fsynced. The three newest files are kept. |
 | `catalog/manifest` | `CatalogStore` | The file name of the latest `.cat`, written durably the same way. |

docs/operations/docker.md

@@ -259,8 +259,8 @@ skipped because `FORMAT` exists. To verify a backup, run
 `docker run --rm -v restored:/var/lib/hstore/data hstore check /var/lib/hstore/data`.
 
 To upgrade, stop the container, back up the volume, and start the new image on the same volume. `FORMAT`
-records the on-disk format version (`format=7`, commits logged as key changes) and the page size. Both are checked at
+records the on-disk format version (`format=8`, leaf records framed by length) and the page size. Both are checked at
 open. A directory written in another format is refused with
-`database … uses storage format N; this build reads format 7 (commits logged as key changes); export and reload it`,
+`database … uses storage format N; this build reads format 8 (leaf records framed by length); export and reload it`,
 so the server never misreads it. Export the data with HQL against the old version and reload it into a fresh
 volume.

docs/operations/logging.md

@@ -115,7 +115,7 @@ and in Docker it is the last line of `docker logs`. The storage-integrity failur
 
 | Message | Meaning |
 |---|---|
-| `database <dir> uses storage format N; this build reads format 7 (commits logged as key changes); export and reload it` | `FORMAT` was written by an incompatible build. |
+| `database <dir> uses storage format N; this build reads format 8 (leaf records framed by length); export and reload it` | `FORMAT` was written by an incompatible build. |
 | `database uses N byte pages, options request M` | `page_size` differs from the value fixed at `init`. |
 | `write-ahead log segment <file> is damaged (<reason>) but later segments exist; replay would silently drop committed transactions` | `CORRUPT_LOG`: damage in a non-tail WAL segment. A torn record at the end of the *last* segment is normal after a crash. Replay simply ends there. Damage before later segments means storage corruption, so recovery refuses to guess. Restore from backup. |
 

docs/storage/maintenance.md

@@ -268,6 +268,7 @@ History and pins are also what hold dead pages back. A page waits until every ge
 * magic, format, page identity and CRC-32C (`PageHeader.verify`);
 * the schema id against the tree being read (`NodeCodec.decode`);
 * strictly ascending keys in leaves;
+* for a packed leaf, that the header's refers-to-pages flag matches its records once all of them are decoded. Loading a packed leaf skips that comparison, because it would decode every record;
 * separators that route each child;
 * uniform height (balance);
 * every stored child summary equal to its recomputed contents;

docs/storage/pages.md

@@ -76,6 +76,8 @@ zigzag  timeMax
 i64le   roleBits
 keys:   zigzag keys[0], then varint (keys[i] − keys[i−1]) for i = 1..slotCount−1
 LEAF:     schema.codec().encode(out, keys, values)          (value format owned by the schema)
+          or, for a codec with independentValues():
+          (varint length | value) × slotCount                  (catalog, posting lists, row codecs)
 INTERNAL: Ref × slotCount
           Ref := u8 0                                          empty tree
                | u8 1 | i64le pageId | varint units | Summary  stored tree
@@ -88,6 +90,8 @@ For an internal node, "keys" are the separators. `separators[0]` is the first ch
 
 **Reference encoding** (`tree/Ref.java`). Every reference starts with a tag byte. `0` is the empty tree and nothing follows. `1` is a stored tree: the page number, the image size in 64-byte units and the subtree's summary. `2` is an inline tree: the entry count `n` (1 to 32), then the leaf exactly as a leaf page payload encodes it, keys as a zigzag first key and varint gaps followed by `schema.codec().encode(...)`. There is no summary, because the reader recomputes it from the entries. An inline tree can only be decoded where the reader knows its schema, through `Ref.read(in, schema)`. `AtomCodec` passes the member schema of the edge's kind for `members` and `order-index` for `order`. The tree changes in a WAL `Changes` record are read with their schema as well, so a nested tree change can carry an inline tree. Everywhere else (branch children, catalog images, posting and property directories) `Ref.read(in)` rejects tag `2`, since only nested references in codecs that opt in with `inlinesSmallTrees()` are ever inline. An inline reference is never larger than `Ref.maxEncodedSize()` (108 bytes); see [persistent-tree.md](persistent-tree.md#inline-trees) for why that keeps pages from overflowing.
 
+**Framed values.** A codec whose values are written one after another, each without reference to the others, says so with `ValueCodec.independentValues()`: `AtomCodec` (the catalog), `PostingsCodec` (posting directories) and every `ValueCodec.rows` codec. In their leaves `NodeCodec` writes each value's length in front of it, so a reader can find one value without decoding the ones before it. The length is counted in `TreeSchema.entryBytes`, so the leaf budget still keeps every image within a page. Columnar codecs, such as the member and incidence codecs, keep the plain layout. When a framed leaf is read from a page, `NodeCodec.decode` walks the lengths to record where each value starts, rejects the image if a value runs past the payload or the values stop short of its end, and returns a packed leaf ([caching](#nodecache-treenodecachejava)). An inline tree of a framed codec uses the same layout and is decoded in full.
+
 An edge record with three plain members `{1000, 1001, 1005}` holds this as its `members` reference. It takes 7 bytes. A stored reference to the same tree would take 65 bytes, most of them summary, and point at an image of at least 128:
 
 ```text
@@ -346,10 +350,10 @@ and the reference journal.
 
 ### `NodeCache` (`tree/NodeCache.java`)
 
-The cache stores **decoded, frozen** `Node` objects, not bytes. A hit therefore costs no decoding and no allocation. The cache has 16 shards; a node goes to shard `mix(pageId) >>> 60`. Each shard has:
+The cache stores **decoded, frozen** `Node` objects, not bytes. A hit therefore costs no decoding and no allocation. The exception is a leaf of framed values, which stays **packed**: it holds the page image, the decoded keys and the offset of every value, and decodes a value the first time it is read (`Leaf.value`). The decoded value is kept in the leaf, published with a compare-and-set so another thread sees it whole, and two threads that decode the same value at once keep whichever got there first. A catalog lookup on a cold page therefore decodes the keys and one record instead of every record and every inline member list on the page. The cache has 16 shards; a node goes to shard `mix(pageId) >>> 60`. Each shard has:
 
 * an open-addressing table of `Slot(pageId, node, volatile referenced)`, probed linearly from `mix(pageId)`, so a lookup needs no boxed key. It is kept at most half full, counting removed slots, and is rebuilt into a new array when it would pass that;
-* a byte budget of `max(64 KiB, node_cache_mb / 16)` and a circular, doubly linked ring of slots with a CLOCK hand. Each slot weighs `Node.heapBytes()`, an estimate of the decoded node's heap size: 144 bytes for the node and its summary, its key and value or child arrays, then 96 bytes plus the encoded value per leaf entry, or 128 bytes per child reference in a branch. A leaf whose values are implied by its keys or member ids counts only its `long` arrays, and a catalog leaf adds the estimate of every inline tree it holds. Those constants come from measuring retained heap against cache contents at two dataset sizes;
+* a byte budget of `max(64 KiB, node_cache_mb / 16)` and a circular, doubly linked ring of slots with a CLOCK hand. Each slot weighs `Node.heapBytes()`, an estimate of the decoded node's heap size: 144 bytes for the node and its summary, its key and value or child arrays, then 96 bytes plus the encoded value per leaf entry, or 128 bytes per child reference in a branch. A leaf whose values are implied by its keys or member ids counts only its `long` arrays, and a catalog leaf adds the estimate of every inline tree it holds. A packed leaf starts at its image size plus its key and offset arrays. Each value it decodes adds 96 bytes plus the encoded value and any inline tree in it, and charges that to the leaf's slot (`Node.chargeTo`), which evicts under the shard lock until the shard is back within budget. A charge for a slot that was evicted, replaced or cleared is dropped. Those constants come from measuring retained heap against cache contents at two dataset sizes;
 * a `ReentrantLock` taken only for admission.
 
 **Reads are lock-free.** `get` probes the table, reading each slot with acquire ordering, plus setting `referenced = true` if it was clear. Writers change the table only under the shard lock, publishing each slot with release ordering, and a rebuild publishes the new array through a volatile field. A reader still probing the old array can find a node that was just evicted; that is harmless, because a page number's contents never change. The check avoids a redundant volatile write on hot entries.
@@ -390,7 +394,7 @@ Because images are never overwritten in place, a torn write can only damage an i
 
 | Version | Where | Meaning |
 |---|---|---|
-| `format=7` in `<data>/FORMAT` | `StorageEngine.verifyFormat` | Commits logged as key changes (`Changes` WAL records), tree pages written at checkpoints, no history in catalog images. Also tagged references with small member and order trees inline in their edge record, reference counts in `pages.ref`, page numbers resolved through `pages.dir`, images packed in 64-byte units, image sizes in every stored reference. Directories written by older builds (`format=1` fixed page slots, `format=2` references without sizes, `format=3` references by address, `format=4` no reference counts, `format=5` untagged references with every nested tree on its own page, `format=6` commits that logged their pages) fail to open: `uses storage format 6; this build reads format 7 (commits logged as key changes); export and reload it`. |
+| `format=8` in `<data>/FORMAT` | `StorageEngine.verifyFormat` | Leaf values of codecs with independent values framed by their length. Also commits logged as key changes (`Changes` WAL records), tree pages written at checkpoints, no history in catalog images, tagged references with small member and order trees inline in their edge record, reference counts in `pages.ref`, page numbers resolved through `pages.dir`, images packed in 64-byte units, image sizes in every stored reference. Directories written by older builds (`format=1` fixed page slots, `format=2` references without sizes, `format=3` references by address, `format=4` no reference counts, `format=5` untagged references with every nested tree on its own page, `format=6` commits that logged their pages, `format=7` unframed leaf values) fail to open: `uses storage format 7; this build reads format 8 (leaf records framed by length); export and reload it`. |
 | `page-size=N` in `<data>/FORMAT` | same | Fixed at creation. A mismatching `--page_size` is rejected. |
 | header byte 4 = `1` | `PageHeader.FORMAT` | Node image layout; unchanged by packing and by page numbers. |
 | `pages.dir` header | `PageDirectory` | `i32 magic 0x52494450`, `i32 format 1`. |

docs/storage/persistent-tree.md

@@ -204,7 +204,7 @@ An inline leaf is frozen (`owner == null`), so a writer copies it before changin
 
 **Equality.** `Ref.same` compares stored references by page id and pending ones by identity. A pending node stamped by a published epoch is also the same as the reference it was written as, and two such nodes are the same when they were written as the same reference. That matters to a transaction that began before a checkpoint: the checkpoint swaps the head generation's pending nodes for stored references, and without this rule every tree the transaction touched would look changed. Inline references have no identity worth comparing: two reads of the same catalog page decode two separate leaves. They are compared by content instead. Two inline references are the same when they belong to the same schema and have the same keys and equal values (`Leaf.sameEntries`). The same content test applies between a pending leaf and an inline reference, and between two pending leaves, when both are small enough that a checkpoint would store them inline (`Ref.fitsInline`). `Ref.Inline.equals` uses the content test, so `EdgeRecord.equals` compares inline member trees by content too. This is what the conflict checks in `Transaction` and `Op`, the shortcuts in `TreeAlgebra` and `TreeDiff`, and the read validation of serializable transactions rely on. A stored and an inline reference are never the same, even with equal entries. Every caller treats "not the same" as "look closer", so that only costs a slower path.
 
-**Heap.** The node cache weighs a catalog leaf with `Leaf.heapBytes()`, which adds the heap estimate of every inline leaf it holds, so inline trees count against `node_cache_mb` like any other node.
+**Heap.** The node cache weighs a catalog leaf with `Leaf.heapBytes()`, which adds the heap estimate of every inline leaf it holds, so inline trees count against `node_cache_mb` like any other node. A catalog leaf read from a page is packed: it decodes a record, and the inline trees in it, only when that record is read, and charges the cache as it does ([pages.md](pages.md#nodecache-treenodecachejava)).
 
 ## Reading
 

docs/transactions/wal-and-recovery.md

@@ -27,7 +27,7 @@ Related: [transactions](transactions.md), [change feed](../storage/change-feed.m
 
 ```text
 <db>/
-  FORMAT                         properties: format=7, page-size=<bytes>
+  FORMAT                         properties: format=8, page-size=<bytes>
   LOCK                           exclusive FileLock held while open
   catalog/manifest               name of the current catalog image file
   catalog/generations/%016x.cat  checkpointed catalog images (3 newest retained)

engine/src/main/java/io/hstore/engine/StorageEngine.java

@@ -44,7 +44,7 @@
 public final class StorageEngine implements AutoCloseable {
 
     private static final System.Logger LOG = System.getLogger("hstore.engine");
-    private static final String FORMAT_VERSION = "7";
+    private static final String FORMAT_VERSION = "8";
     private static final int CHANGED_COUNTS_LIMIT = 1 << 20;
 
     private static final int MAX_ATTEMPTS = 8;
@@ -135,7 +135,7 @@ private static void verifyFormat(Path directory, EngineOptions options) throws I
             }
             if (!format.getProperty("format", "1").equals(FORMAT_VERSION)) {
                 throw HStoreException.invalid("database " + directory + " uses storage format " + format.getProperty("format", "1")
-                        + "; this build reads format " + FORMAT_VERSION + " (commits logged as key changes); export and reload it");
+                        + "; this build reads format " + FORMAT_VERSION + " (leaf records framed by length); export and reload it");
             }
             int stored = Integer.parseInt(format.getProperty("page-size"));
             if (stored != options.pageSize()) {

engine/src/main/java/io/hstore/engine/catalog/AtomCodec.java

@@ -79,6 +79,11 @@ public boolean holdsRefs() {
         return true;
     }
 
+    @Override
+    public boolean independentValues() {
+        return true;
+    }
+
     @Override
     public boolean inlinesSmallTrees() {
         return true;

engine/src/main/java/io/hstore/engine/index/PostingsCodec.java

@@ -97,6 +97,11 @@ public boolean holdsRefs() {
         return true;
     }
 
+    @Override
+    public boolean independentValues() {
+        return true;
+    }
+
     @Override
     public boolean refersToPages(Postings<P> value) {
         return value instanceof Postings.Promoted<P>(Ref root) && root instanceof Ref.Stored;