Improve traversal and query performance, implement Return, close filter injection

[justintout]

Summary

Traversal cost now tracks the set of nodes a query reaches instead of graph size. On a 10k-node graph, a 6-hop traversal drops from 2.57s to 2.5ms. Return(), previously a no-op, now projects columns. Filter column names are no longer an SQL injection point. A new harness benchmarks the library against GraphQLite.

Before/after numbers come from 6 alternating runs of the old and new benchmark binaries (all p=0.002):

Benchmark	Before	After
10k nodes, 1 hop	4.57 ms	14 µs
10k nodes, 3 hops	96 ms	100 µs
10k nodes, 6 hops	2.57 s	2.5 ms
10k nodes, both directions, 3 hops	516 ms	487 µs
Dense 500 nodes, 10 hops	952 ms	9.8 ms
Tree, 1,093 nodes	97 ms	1.9 ms
Label + name match, 10k nodes	1.55 ms	4.6 µs
100-node chain, 1 hop	32 µs	8.8 µs
CreateNode	15.0 µs	11.6 µs
CreateEdge	14.6 µs	7.2 µs
CreateNode, on disk	378 µs	68 µs
Bulk insert, 1000 in a transaction	16.2 ms	6.5 ms

Changes

Migration 2 (schema)

• Without statistics, SQLite used idx_edges_type for every hop. Each expanded node then scanned all edges of that type.
• Edge indexes are replaced with covering (source_id, type, target_id) and (target_id, type, source_id) indexes. Prefix-redundant indexes are dropped.
• node_labels is rebuilt as a WITHOUT ROWID table.
• nodes(name) gets an index.
• Tested against an existing version-1 database with data.

Query compiler

• Each step compiles to a CTE of node IDs, and nodes is joined once for the final set. Before, traversals scanned nodes and ran DISTINCT over the JSON properties.
• CROSS JOIN fixes join order. Otherwise SQLite flattened chained steps into full edge scans.
• Both compiles to two indexed SELECTs instead of an OR join.
• A unary + on the label check makes a name equality drive the query.
• LIMIT/OFFSET are bound parameters, emitted only when set. Interpolated values grew the per-connection statement cache without bound. A bound LIMIT forces a re-prepare on every run, which is why it is omitted when unused.
• The final set is joined as a DISTINCT subquery instead of IN. IN builds a Bloom filter, which costs an mmap syscall per query under modernc.

Return()

• Selects only the named columns. Property paths are extracted in SQL, so rows skip decoding the whole JSON document.
• On a 6-hop traversal reaching ~700 nodes with 10-key properties:

Returned	Time	Allocations
Everything	6.3 ms	63k
name only	1.8 ms	2k
name + 2 properties	4.2 ms	16k

SQL injection

• Where and WhereRel wrote the field name into SQL unchecked. They now accept only id, name, created_at, updated_at, and properties.
• JSON paths in WhereJSON, WhereRelJSON, and Return are bound parameters. The measured cost is the same as literal paths.

Writes and connections

• Timestamps are generated in Go and bound into plain INSERT/UPDATE statements. That replaces the follow-up SELECT, and avoids RETURNING, which buffers rows in a temp table that spills to disk in large transactions.
• Savepoints are used only outside a transaction. Inside one, they made SQLite journal every touched page to a temp file.
• Together these cut a 100k-node / 500k-edge load in one Tx from 18.2 s to 5.3 s.
• The compiler builds SQL without fmt: simple matches ~12% faster, 1-hop traversals 10–18% faster.
• Connections open in WAL mode.

Benchmarks

• Added large-graph traversal, file-backed write, and Return benchmarks.
• Added a test that checks every traversal compilation path, in all directions, against walks computed in Go on a random graph with cycles.

Comparison with GraphQLite

bench/compare is a separate module, since it needs CGo. It loads the same graphs into three engines and fails if they disagree on any result size:

• ours: this library.
• our SQL on C SQLite: the SQL this library generates, run through C SQLite on the same database file.
• GraphQLite 0.8.0: variable-length Cypher queries.

Medians of 6 runs, Apple M3 Max:

Query	ours	our SQL on C SQLite	GraphQLite
10k random, 1 hop	11.0 µs	5.2 µs	9.4 µs
10k random, 3 hops	63 µs	34 µs	79 µs
10k random, 1..5 hops	458 µs	278 µs	778 µs
100k power-law, 1 hop	10.0 µs	4.4 µs	7.7 µs
100k power-law, 3 hops	50 µs	26 µs	66 µs
100k power-law, 1..5 hops	540 µs	332 µs	1.66 ms
Hub, in-edges, 1 hop (10k nodes)	7.9 ms	5.5 ms	26.8 ms
Hub, in-edges, 2 hops (45k nodes)	127 ms	83 ms	196 ms
Load 100k nodes / 500k edges	4.8 s	–	2.6 s

• Traversals: we're faster at 2+ hops and on the hub queries, by 1.1x to 3.4x. GraphQLite is ~20% faster on single hops.
• SQLite engine cost: our SQL on C SQLite beats GraphQLite on every query. That puts most of the remaining gap in the pure-Go SQLite engine rather than the schema or queries.
• Loading: GraphQLite's bulk path, which writes table rows directly, is ~1.9x faster than our per-row CreateNode/CreateEdge.

Behavior changes

• Breaking: Where and WhereRel return an error for columns other than id, name, created_at, updated_at, and properties.
• Return(): fields not returned are left zero. Properties is nil unless properties or a property path is returned.
• Fixed: WhereRel on an intermediate multi-hop step was silently ignored.
• Fixed: Offset without Limit produced invalid SQL.
• Count() on traversals ignores Limit/Offset, matching simple matches.
• On-disk databases get -wal and -shm files.
• A Tx method that returns an error may leave part of its writes in the transaction (for example, a node without its labels). Roll back after any error. This is documented on Tx.
• created_at/updated_at from the API come from the Go clock, in the same format as the schema default.

Test plan

• go vet ./... and go test ./... pass.
• New tests: WhereRel on an intermediate step, Both direction, offset-only queries, Return projection, hostile column names and JSON paths.
• EXPLAIN QUERY PLAN checked on a populated graph for single-hop, recursive, chained, filtered, and JSON-filter queries: no table scans, Bloom filters, or automatic indexes.
• A version-1 database migrates, then passes label, traversal, and cascade-delete checks.
• All three engines in bench/compare return identical result sizes for every query.

Follow-ups not included

• Bulk load: a batch-insert API to close the remaining ~1.9x load gap.
• Breadth-first traversal: 4.8x faster on dense graphs, but up to 7x slower on chains, and hop count doesn't predict which case applies. Planned as a separate, stacked PR.

🤖 Generated with Claude Code

https://claude.ai/code/session_01YB9owLYxU6NsNuVk2NVc7C