Skip to content

Add opt-in BreadthFirst traversal mode - #3

Merged
justintout merged 1 commit into
feature/perffrom
feature/bfs-traversal
Sep 14, 2026
Merged

justintout merged 1 commit into
feature/perffrom
feature/bfs-traversal

Conversation

@justintout

Copy link
Copy Markdown
Owner

Stacked on #2.

Summary

Adds Query.BreadthFirst(), which makes the most recent Related step expand each reached node once. Results are identical to the default mode.

g.Match("Person").
    Where("name", "=", "Alice").
    Related("KNOWS", 1, 6).
    BreadthFirst().
    Run(ctx)

Why opt-in

By default, a multi-hop step is a recursive CTE that deduplicates on (node, depth). A node reachable at several depths gets expanded once per depth. BreadthFirst compiles the step to one CTE per hop level, holding only nodes first reached at that level.

Each level costs a fixed ~20µs for SQLite's temporary tables. Which mode wins depends on how often the walk revisits nodes. That's a property of the data near the start node, and the query shape can't predict it. Choosing automatically would mean running a probe query on every deep traversal.

Medians of 6 runs, Apple M3 Max:

Query Default BreadthFirst
100-node chain, 10 hops 40 µs 243 µs
Tree, fanout 3, depth 6 1.81 ms 1.67 ms
Dense 500 nodes / 5 edges each, 5 hops 2.10 ms 1.44 ms
Dense 500 nodes / 5 edges each, 10 hops 9.51 ms 2.02 ms
10k random, 6 hops 1.87 ms 1.68 ms
10k random, both directions, 4 hops 2.28 ms 1.78 ms

Details

  • minHops: requires minHops == 1; otherwise Run returns an error. A larger minHops needs nodes reachable at an exact depth, and a visited set discards those.
  • maxHops == 1: compiles as a normal single hop.
  • Cycles: start nodes aren't excluded from later levels, so a cycle back to the start is found, matching the default mode.

Test plan

  • go vet ./... and go test ./... pass.
  • TestTraversalMatchesReference runs every minHops == 1 case in both modes, for all directions and 1–6 hops, against walks computed in Go on a random graph with cycles and self-loops.
  • Breaking the last breadth-first level on purpose makes that test fail.
  • TestWhereRelIntermediateStep covers BreadthFirst followed by WhereRel and another step.
  • BreadthFirst with minHops of 2 returns an error.

🤖 Generated with Claude Code

https://claude.ai/code/session_01YB9owLYxU6NsNuVk2NVc7C

A multi-hop step deduplicates on (node, depth), so a node reachable at
several depths is expanded at each. BreadthFirst compiles the step to
one CTE per hop level holding nodes first reached at that level, so each
node is expanded once. Results are identical.

Each level costs a fixed ~20us, so which mode wins depends on how much
the walk revisits nodes, which the query cannot predict. It is opt-in:
10 hops on a dense 500-node graph drops from 9.5ms to 2.0ms, and on a
100-node chain rises from 40us to 243us. It requires minHops of 1.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YB9owLYxU6NsNuVk2NVc7C
@justintout
justintout merged commit 1ba996e into feature/perf Sep 14, 2026
justintout added a commit that referenced this pull request Sep 14, 2026
…er injection (#2)

* Add large-graph traversal and file-backed write benchmarks

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YB9owLYxU6NsNuVk2NVc7C

* Rework traversal indexes and query compilation

Every traversal step used idx_edges_type, scanning all edges of a type
per expanded node, and the final SELECT scanned the nodes table. On a
10k-node graph a 6-hop query took 2.5s; it now takes 2.5ms.

Migration 2 replaces the edge indexes with covering (source_id, type,
target_id) and (target_id, type, source_id) indexes, rebuilds
node_labels as WITHOUT ROWID, and indexes nodes.name.

The compiler emits one CTE of node IDs per step, joins nodes once for
the final set, forces join order with CROSS JOIN, and splits Both into
two indexed SELECTs. LIMIT/OFFSET are bound, and emitted only when set,
so pagination does not grow the per-connection statement cache.

Fixes WhereRel on intermediate multi-hop steps being ignored and Offset
without Limit producing invalid SQL. Count now ignores Limit/Offset for
traversals, matching simple matches.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YB9owLYxU6NsNuVk2NVc7C

* Use RETURNING for node and edge inserts and updates

Drops the follow-up SELECT that read back IDs and timestamps.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YB9owLYxU6NsNuVk2NVc7C

* Open connections in WAL mode

File-backed autocommit CreateNode drops from ~380us to ~76us, and
readers no longer block on the writer.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YB9owLYxU6NsNuVk2NVc7C

* Update README for new query compilation and benchmarks

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YB9owLYxU6NsNuVk2NVc7C

* Implement Return projection and close filter SQL injection

Return was accepted but ignored. It now selects only the named columns
and extracts property paths in SQL, skipping the full JSON decode. On a
6-hop traversal reaching ~700 nodes with 10-key properties, returning
only name drops from 6.3ms to 1.8ms and 63k to 2k allocations.

Where and WhereRel wrote the field name into SQL unchecked; they now
accept only node columns. JSON paths in WhereJSON, WhereRelJSON, and
Return are bound parameters instead of interpolated literals, which
measures the same per execution.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YB9owLYxU6NsNuVk2NVc7C

* Add comparison benchmarks against GraphQLite

A separate module loads the same generated graphs into go-sqlite-graph,
GraphQLite (Cypher via its SQLite extension), and go-sqlite-graph's own
SQL on C SQLite, and fails if any engine disagrees on a result size.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YB9owLYxU6NsNuVk2NVc7C

* Document Return, filter columns, and comparison benchmarks

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YB9owLYxU6NsNuVk2NVc7C

* Remove RETURNING and in-transaction savepoints from writes

Both forced SQLite to write the statement sub-journal to a temp file on
large transactions: RETURNING buffers its rows in a temp table, and a
savepoint journals every page it touches. Loading 100k nodes and 500k
edges in one Tx drops from 18.2s to 5.3s.

Timestamps are now generated in Go and bound, in the schema default's
format. Savepoints are still used for multi-statement writes outside a
transaction; inside a Tx, a failed method may leave partial writes, and
the caller rolls back.

CreateNode -32%, CreateEdge -35%, BulkInsertTx/batch=1000 -62%.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YB9owLYxU6NsNuVk2NVc7C

* Build query SQL without fmt and test traversals against a reference

fmt allocated for each formatted value while compiling. Writing strings
directly makes simple matches ~12% faster and single-hop traversals
10-18% faster; heavy traversals are unchanged.

The new test compares every traversal compilation path, in all three
directions, against walks computed in Go on a random graph with cycles.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YB9owLYxU6NsNuVk2NVc7C

* Update README benchmarks for write and compile changes

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01YB9owLYxU6NsNuVk2NVc7C

* Add BreadthFirst traversal mode (#3)

A multi-hop step deduplicates on (node, depth), so a node reachable at
several depths is expanded at each. BreadthFirst compiles the step to
one CTE per hop level holding nodes first reached at that level, so each
node is expanded once. Results are identical.

Each level costs a fixed ~20us, so which mode wins depends on how much
the walk revisits nodes, which the query cannot predict. It is opt-in:
10 hops on a dense 500-node graph drops from 9.5ms to 2.0ms, and on a
100-node chain rises from 40us to 243us. It requires minHops of 1.


Claude-Session: https://claude.ai/code/session_01YB9owLYxU6NsNuVk2NVc7C

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant