[lucidprogrammer]

Adds the data rules of the new engine under v2/: what a table may declare and what every write must obey. Nothing that builds from crates/ changes; the root Cargo.toml and Cargo.lock are untouched. One commit on dev. The query language (joins, grouping, graph patterns, ranked search, unions, subqueries) is the next pull request; the store's on-disk layout is its own pull request after it.

What is in it

• A table can declare the rules its rows obey. Primary keys and UNIQUE rules on one column or several; REFERENCES and FOREIGN KEY onto a primary key or a UNIQUE rule, single or composite; IMMUTABLE on a table or a column; STATE MACHINE, DAG, PROPAGATE along references and along edges, PROPAGATE ... EXCLUDE VECTOR; MAINTAIN COUNT; CONTEXT_ID, SCOPE_LABEL and ACL REFERENCES; and a table with no primary key at all, whose rows are kept under a hidden key. A declaration prints back as it was written: Context Graph's 118 table declarations are all accepted and 93 of the 93 that were written out come back unchanged.
• Every write is checked against those rules, whoever makes it. The type of each value, NOT NULL, the key, every UNIQUE rule, references, the state machine, immutability and the graph's no-cycle rule are checked before anything changes; a statement that breaks one is refused and changes nothing. References and the cycle rule are judged when the transaction commits. A key is judged when the statement ends, so an UPDATE that swaps two keys or shifts a run of them succeeds.
• A refusal names the row. unique constraint violation: users (id) = (1); foreign key violation: orders (user_id) = (99) has no row in users; foreign key violation: users is still referenced by orders (user_id) = (1); invalid state transition of tasks.status: new -> shipped. A long text is cut at 64 bytes at a whole character and followed by its length. The value shown is always the one the writer gave, so a narrowed handle learns nothing about a row it cannot see.
• Tables can be changed and dropped. ALTER TABLE adds a column (earlier rows read its DEFAULT), drops one (RESTRICT, or CASCADE which reports the indexes it dropped), renames one, replaces RETAIN, HISTORY, SYNC, SYNC CONFLICT and MAINTAIN COUNT, and changes a vector column with SET MAX_PARTITIONS, SET SEARCH_MODE, SET AUTO_INDEX_AT and SET CONSOLIDATION (each with DEFAULT to put the default back). A column the table is declared by (its primary key, an IMMUTABLE column, a state machine's column, a MAINTAIN COUNT column, a vector partition key, the edge columns of a DAG, a column a rank policy reads) is neither dropped nor renamed, and the refusal names the rule. DROP TABLE refuses a table another table references; IF EXISTS makes a missing table or index a no-op.
• INSERT ... ON CONFLICT (columns) DO UPDATE, UPDATE, DELETE and SELECT over one table, with IN lists, COUNT, SUM, MIN, MAX, ORDER BY and LIMIT. An INSERT into a table with an index on a JSON path is accepted; the index is stored and printed and the query language fills and reads it.
• The shell prints how a command is used when it is used wrongly, and warns (warning: there is already a transaction in progress / there is no transaction in progress) when BEGIN comes inside a transaction or COMMIT or ROLLBACK outside one. set_memory_limit on a narrowed handle is refused with AdminRequired, as the statement is.
• The check. .github/workflows/check.yml states 21,000 shipped lines (it stated 15,500), 380 tests (it stated 260) and, new in this change, 38,000 test lines with a step that fails when they pass it.

What is refused, and what to write instead

Written	Answer
WITH (quantization = 'SQ4')	not part of this library: write 'SQ8' to store one byte a component, or leave the clause out to store each vector as given
IVF (...) or SEARCH (...), in a declaration or after ALTER COLUMN ... SET	not part of this library: the index sizes its own groups and measures how far a search scans; for the true nearest rows write USE VECTOR EXACT or SEARCH_MODE EXACT, and to move the size from which AUTO uses the index write AUTO_INDEX_AT
EXPIRES	EXPIRES is not supported in this version of the library (nothing expires yet; RETAIN is accepted and printed back)
CREATE UNIQUE INDEX	write UNIQUE on the column or in the table
ADD COLUMN ... NOT NULL with no DEFAULT on a table that has rows	refused: give it a DEFAULT
a column named twice in INSERT, SET or DO UPDATE SET	plan error: ... is named twice as a column ... is written in
setting a NOT NULL column to NULL, on UPDATE or DO UPDATE	ColumnNotNullable
a key, a UNIQUE rule or an index naming more than 32 columns	SchemaInvalid: name fewer
an index name beginning __pk_, __unique_, __fk_ or __graph_	ReservedIndexName: choose another name
DROP COLUMN of a column an index names	refused naming the index: drop the index first, or write CASCADE
INTEGER column referencing a UUID key	refused: give the column the key's type
text that is not JSON written to a JSON column	refused (the old engine stored it as a string)
DROP COLUMN of a state machine's column	SchemaInvalid naming the STATE MACHINE

JOIN, GROUP BY, HAVING, DISTINCT, LIKE, BETWEEN, OFFSET, UNION, WITH, subqueries, graph patterns and USE RANK are still refused by name, ... is not supported in this version of the library; they come with the query language.

What changed on purpose from the old engine

Seventeen rows of the Postgres-differences list (80 to 96) come from running 978 statements of the old engine's SQL scripts against both engines; three more (77 to 79) are the refused vector clauses above. Among them:

• An UPDATE that swaps two primary keys or shifts a run of them succeeds (row 96). The old engine refuses it as a unique violation. Postgres decides by the order it visits rows unless the key is DEFERRABLE. Here a key is judged when the statement ends, as the SQL standard says; two rows left under one key are still refused.
• ADD COLUMN ... DEFAULT fills earlier rows with the default (the old engine left them NULL); ADD COLUMN ... NOT NULL with no default on a table with rows is refused (the old engine wrote NULL into a NOT NULL column).
• DROP TABLE of a table that is not there answers table_not_found (the old engine answered "corrupt store"); a column named twice, NOT NULL set to NULL, an INTEGER column referencing a UUID key are refused (the old engine accepted them).
• A composite foreign key onto a composite primary key is accepted (the old engine refused it); RENAME COLUMN of a UNIQUE column a reference names renames it and keeps the reference.
• Two NULLs in a single-column UNIQUE column are two rows in ON CONFLICT (the old engine updated the first).
• PROPAGATE runs for every key type; the old engine ran it only for rows keyed by a UUID, so with INTEGER keys a declared rule silently never ran.
• Text written to a JSON column is parsed and kept as a value, and --json prints it as an object, not a string. A consumer that read the old string sees the change.

Measured at this commit

By v2/numbers.sh on this code: a release build on one machine with nothing else running; one query at a time on one thread, a read session at 1 GiB, 95th percentile of 100 questions. "dev before" is the same table taken on dev without this change. A figure is marked worse when it is worse than dev before by more than the spread the file prints for it, or by any amount where no spread is printed.

	dev before	This change
Ten nearest, 100,008 passages, one / several / all partitions, exact	1.41 / 3.84 / 10.88 ms	1.41 / 3.83 / 10.90 ms
the same, indexed	0.68 / 1.54 / 4.05 ms	0.70 (worse) / 1.57 / 4.12 ms (worse)
the same, recall	1.000 / 0.999 / 0.999	unchanged
One million generated vectors, exact / indexed	180.9 / 1.11 ms	180.4 / 1.12 ms (worse)
One million generated camera sightings, exact / indexed	175.4 / 30.35 ms	183.7 (worse) / 33.7 ms (worse)
A million rows summed (SELECT SUM)	236.6 ms	255.0 ms (worse)
A million rows loaded	10.5 s	10.7 s (worse)
Fresh clone to a green cargo test	3.2 min	4.0 min
First build of a program that embeds the library	8 s, 47 packages, 3.2 MiB	11 s, 47 packages, 4.1 MiB
100,008 passages, plain table, on disk (put in 298.6 MiB)	471.8 MiB	471.8 MiB
Shipped lines; tests	15,443; 253	20,747; 371 (339 listed and 32 SQL script files)
Test lines	25,234	37,974

The scan is 18.4 ms slower than dev before. Two costs of this change were removed (13 ms and 8 to 10 ms of it); about 20 ms are left, mostly the hand-off of each row in code shared with the query path, and the read and decode of the store alone is the same 148 ms as dev before. Fifteen keys of the numbers script are marked worse, the largest being the sightings indexed find (30.35 to 33.73 ms) and the fresh-clone build.

Store sizes

Context Graph's declared schema, loaded the same way into both engines, 500 rows to a commit (the old engine at its current commit on the same machine).

	Put in	Old engine	This change	Twice the data
100,008 passages, 13 indexes	307.2 MiB	2,855.9 MiB	632.8 MiB	614.4 MiB
62,447 commands (70,944 events, 138,084 anchors), 15 indexes	170.4 MiB	4,259.5 MiB	560.6 MiB	340.8 MiB

Both stores are above twice what was put in, and 4.5 and 7.6 times smaller than the old engine's for the same rows. The cause is measured and belongs to the store's layout, not to the data rules: the storage library leaves half-empty index pages when an index is written in runs, 62.1 MiB of the passages store. That is the layout pull request. The old engine's figure for the anchors was measured beside an earlier run of this engine (672.4 MiB); this run's 560.6 MiB follows a later cap on how much of a long text an index entry keeps.

Old engine against this one, statement by statement

978 statements of 20 SQL scripts, run against both engines: 778 answer the same, 18 are the new form (a vector search written the Postgres way), 44 differ on purpose and are written down, 88 follow from those, 28 differ only in wording, 22 are old-engine defects this change corrects, 0 are defects of this change, and 0 are unread.

Verification

• Every command of check.yml as written, on the commit: format, lint with warnings as errors, 339 of 339 tests, the documentation tests, the storage library's own 202.
• GitHub Actions on c955c0253: the run is linked in a comment below when it finishes.

Not in this change

• The query language. Joins, grouping, graph patterns, unions, subqueries, LIKE, the typed equality on a JSON field, USE RANK, the CREATE INDEX line of .explain, a statement that cannot overflow the stack and a SUM over a filter that passes only some rows are the next pull request.
• The store's layout. Full index pages, compression of rows and shared key prefixes in index pages, and the file compaction after a build.
• Expiry. RETAIN is accepted and printed back; no row expires yet.
• Sync, events, erase, limits and cursors, snapshots. Later pull requests.

Known limits of this version

• The numbers above that are marked worse are real: the scan and the sightings find cost a few per cent more than dev before.
• On a table that declares no primary key, the same row written twice is two rows, and a refusal about a row cannot name a key.
• A refusal of a NOT NULL column names the column, not the row.

[lucidprogrammer]

Hosted run: https://github.com/context-graph-ai/contextdb/actions/runs/43248584430 failed. lint and minimum-rust pass. Two tests fail, one on each runner: on macOS, reading_a_missing_store_is_refused_and_creates_nothing (a store path whose data file is a folder is reported as a corrupt store there and as an I/O error on Linux); on Linux, memory_stays_bounded_as_data_grows_vector reached its 60 s limit (it took 54 s on dev's last green run). A fix for each follows as one more commit on this branch.

[lucidprogrammer]

Hosted run on 4271914ff: https://github.com/context-graph-ai/contextdb/actions/runs/43260501343 passed (lint, minimum-rust, tests on Linux in 4m39s and on macOS in 4m53s). Two commits were added since the first run: a store whose data file is a folder is refused as an I/O error on every system, and eight tests do less work for the same proof; the million-row load now runs in this workflow's nightly schedule (and by hand) and not on a pull request.