docs: add the HStore paper and illustrate the README

README.md

@@ -13,6 +13,15 @@ query language (HQL), hypergraph relational operators, a semantic plane, multi-t
 
 It is written in Java 25 and ships as one GraalVM native executable, `hstore`.
 
+<p align="center">
+  <img src="docs/images/hypergraph.svg" width="100%"
+       alt="Three hyperedges from the clinical claims example: a claim whose members have roles, an ordered care pathway, and an investigation whose members are claims">
+</p>
+
+The three kinds of hyperedge HStore stores, taken from the example dataset in
+[`examples/clinical-claims.hql`](examples/clinical-claims.hql): a set hyperedge whose members have roles and weights,
+an ordered hyperedge, and a hyperedge whose members are other hyperedges.
+
 ## Features
 
 * **Native hypergraph topology.** Set and ordered hyperedges of any cardinality, including giant edges with
@@ -44,26 +53,90 @@ It is written in Java 25 and ships as one GraalVM native executable, `hstore`.
   spokes, hulls, ordered chains, time-travel slider), a schema browser, a live engine dashboard and access
   management.
 
+## How it works
+
+<p align="center">
+  <img src="docs/images/overview.svg" width="100%"
+       alt="Writers commit through a commit lock, which publishes new root vectors; a checkpoint writes changed nodes to segments; readers pin published generations">
+</p>
+
+Every structure is a copy-on-write B+tree over 64-bit keys, and the state of the database at one point in time is a
+*generation*: an immutable root vector with one tree per slot (the catalog, the reverse incidence index, type and
+canonical-key indexes, symbol tables, request ids). A commit logs the keys it changed and publishes a new generation
+whose changed nodes are still in memory (yellow). A background checkpoint later writes those nodes to append-only
+segment files, once for all the commits since the last checkpoint. Readers pin a generation and never take a lock.
+
+<p align="center">
+  <img src="docs/images/layout.svg" width="100%"
+       alt="The catalog is a tree of trees: each hyperedge record holds its member tree, inline when small; the reverse index maps each atom to the hyperedges that contain it">
+</p>
+
+The catalog is a tree of trees. A hyperedge's record holds the root of its membership tree, and holds the whole tree
+when it has up to 32 small members, which is the common case. The reverse index maps every atom to the set of
+hyperedges that contain it. Every child reference carries the subtree's entry count and a summary (key range, weight
+and time ranges, a role mask and a content fingerprint), so cardinality, rank, overlap and diff queries skip whole
+subtrees.
+
+<p align="center">
+  <img src="docs/images/wal.svg" width="100%"
+       alt="Recovery replays commits from the checkpoint LSN, stops at the first commit whose pages do not verify, and drops transactions that never committed">
+</p>
+
+Recovery reads the log from the checkpoint recorded in the newest catalog image and replays every commit whose
+`Commit` record is intact. It stops at the first commit whose referenced pages fail their checksum and drops
+transactions that never committed, so the recovered state is always a prefix of the commit order.
+
+<p align="center">
+  <img src="docs/images/compaction.svg" width="55%"
+       alt="Compaction copies live page images out of a mostly dead segment and repoints their directory entries; no tree changes">
+</p>
+
+Trees refer to pages by permanent page numbers, and a memory-mapped page directory maps each number to its current
+place in a segment. Compaction copies the live images out of a mostly dead segment and changes only their directory
+entries, so no tree, generation or branch is rewritten and commits keep running while it works.
+
+The [paper](docs/hstore-paper.pdf) and [docs/architecture/overview.md](docs/architecture/overview.md) cover each part
+in depth.
+
 ## Performance
 
-HStore against HyperGraphDB 1.4 on Berkeley DB JE, both with 64 MiB caches, at scale 1 (50,000 nodes, 100,000
-hyperedges) with `async` durability. Each figure is the median of 5 runs on one laptop:
+HStore against HyperGraphDB 1.4 on Berkeley DB JE, both with 64 MiB caches, on one Azure `Standard_D8ds_v5`
+machine (8 vCPUs, 32 GiB, local SSD). Scale 1 is 50,000 nodes and 100,000 hyperedges with `async` durability;
+each figure is the median of 5 runs per system:
 
 | Workload | HStore | HyperGraphDB | |
 |---|---:|---:|---:|
-| Incidence set of a node | 6.26 M/s | 1.49 M/s | 4.2× faster |
-| Hyperedges shared by two nodes | 610 k/s | 53.9 k/s | 11.3× faster |
-| Value updates, 1,000 per transaction | 69.9 k/s | 13.0 k/s | 5.4× faster |
-| Bytes written per single-update commit | 659 B | 6.9 KiB | 10.7× fewer |
-| Commit latency, p99 | 296 µs | 466 µs | 1.6× lower |
-| Disk after loading | 92 MiB | 346 MiB | 3.7× smaller |
-
-At scale 4 (200,000 nodes, 400,000 hyperedges), where the files of both stores outgrow their caches, the
-leads widen: incidence sets 11.4×, value updates 33×, and listing a hyperedge's members 3.2× faster.
-HyperGraphDB opens a store after a crash faster at scale 1, 76 ms against 128 ms. With `sync` durability,
-single commits in both are bound by the cost of a forced write. The
-[benchmark report](docs/benchmark-report.pdf) (PDF) covers every workload, configuration and caveat, and
-[docs/benchmarks.md](docs/benchmarks.md) explains how to rerun it.
+| Incidence set of a node | 3.10 M/s | 1.25 M/s | 2.5× faster |
+| Incidence sets from 8 threads | 14.1 M/s | 1.28 M/s | 11× faster |
+| Hyperedges shared by two nodes | 272 k/s | 49.8 k/s | 5.5× faster |
+| Value updates, 1,000 per transaction | 40.9 k/s | 7.26 k/s | 5.6× faster |
+| Bytes written per single-update commit | 659 B | 11.1 KiB | 17× fewer |
+| Single-update commit latency, p99 | 228 µs | 767 µs | 3.4× lower |
+| Reader latency during writes, p99.9 | 5.47 µs | 94.1 µs | 17× lower |
+| Disk after loading | 85.7 MiB | 400 MiB | 4.7× smaller |
+| Clean reopen | 78.5 ms | 978 ms | 12.5× faster |
+
+<p align="center">
+  <img src="docs/images/ratios-wide.svg" width="100%"
+       alt="HStore's advantage over HyperGraphDB for every workload in three configurations, on a log scale">
+</p>
+
+At scale 4 (200,000 nodes, 400,000 hyperedges), where the files of both stores outgrow their caches, the leads
+widen: hyperedge ingest 7.9×, value updates 12×, incidence sets read right after a reopen 82×, and 13.7× less disk.
+Opening a store whose ingest was killed with SIGKILL takes 162 ms against 449 ms, and no acknowledged commit was lost
+in any run of either system. Over a whole scale-4 run HStore writes 2.06 GiB to disk and HyperGraphDB 104 GiB, most of
+it while JE's cleaner reclaims space after the deletes.
+
+<p align="center">
+  <img src="docs/images/resources.svg" width="100%"
+       alt="Bytes written per operation, heap bytes allocated per read, and bytes on disk through the load, the deletes and reclamation">
+</p>
+
+Where HStore is not ahead: the median latency of a single commit under `async` durability is 143 µs against
+HyperGraphDB's 93.7 µs, although HStore's tail is much shorter. HStore keeps more heap after ingest (179 MiB against
+118 MiB at scale 1). At scale 4 its slowest single commit in a run is slower than HyperGraphDB's. The
+[paper](docs/hstore-paper.pdf) has every number, the latency percentiles, the before and after runs for three design
+choices, and the caveats; [docs/benchmarks.md](docs/benchmarks.md) explains how to rerun the comparison.
 
 ## Quick start with Docker
 
@@ -201,6 +274,7 @@ The full index is [docs/README.md](docs/README.md).
 | HStore Studio | [docs/operations/studio.md](docs/operations/studio.md) |
 | Benchmarks against HyperGraphDB | [docs/benchmarks.md](docs/benchmarks.md) |
 | Benchmark report (PDF) | [docs/benchmark-report.pdf](docs/benchmark-report.pdf) |
+| Paper: design and evaluation (PDF) | [docs/hstore-paper.pdf](docs/hstore-paper.pdf) |
 | Development | [docs/development.md](docs/development.md) |
 
 ## Contributing

docs/development.md

@@ -240,3 +240,21 @@ installs `org.hypergraphdb:hgdb` and `org.hypergraphdb:hgbdbje` version `1.4-994
 generator, `SCALE` × 50,000 nodes and × 100,000 hyperedges), writes `benchmarks/results/scale-N-durability/*.json`
 and prints a Markdown report whose *Results agree* column cross-checks every query checksum between the two
 engines. Methodology and published numbers: [benchmarks.md](benchmarks.md).
+
+## Paper
+
+`paper/` holds the HStore paper: the LaTeX sources (ACM `acmart`, `sigconf` layout), the R scripts that draw every
+figure and the result files the evaluation is built from. Nothing in the text is typed in by hand: `R/results.R` reads
+`paper/results/<configuration>/*.json` and `paper/ablation/<change>/*.json`, the same files `Comparison run` writes,
+and turns them into the plots, the tables and the `\M{...}` macros the text uses for every number.
+
+You need `tectonic` and R with `ggplot2`, `patchwork` and `jsonlite`. Then:
+
+```
+paper/build.sh            # figures, numbers and docs/hstore-paper.pdf
+paper/build.sh figures    # only the figures and paper/generated/*.tex
+```
+
+The first run compiles the paper once so that tectonic downloads the Libertine and Inconsolata fonts, and the R
+figures then use the same fonts as the text. To redo the evaluation with new numbers, replace the JSON files under
+`paper/results` and `paper/ablation` and build again.