Release 0.0.18: hub-node scale remediation - #44
Merged
Merged
Conversation
Allowlist and map the query-param HTTPS gate (plus WEBHOOK_HTTPS_REQUIRED) into ServerConfig so local HTTP tunnels can disable it via env. Co-authored-by: Cursor <cursoragent@cursor.com>
Phase 0 of the hub-node scale remediation. Seeds a Postgres hub with 1k/10k/100k edges (COPY) and records p50/p95 latency, DB round trips (db_op_counter) and on-disk sizes for connect(), save(), neighbour listings, the len(nodes()) count pattern and 32-way concurrent connect(). Opt-in via `-m bench` (`bench_slow` for 100k). The baseline confirms the expected shape on 0.0.17: connect()/save() grow with hub degree (save 11 ms -> 1.15 s from 1k to 100k), concurrent connects serialise on the hub row lock (11.7 s wall for 32 at 100k), and list-form nodes(limit=20) costs 1 + ceil(degree/500) + 1 round trips (201 at 100k) while the class form stays at 1. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Phase 1 of the hub-node scale remediation. Node rows no longer persist their incident edge ids in an `edges` array on backends whose edge collection is indexed on source/target (`edge_ids_mode="derive"`, new `Database` capability flag; JsonDB and DynamoDB keep "persist"). connect()/disconnect()/save() never rewrite or row-lock a node, so their cost no longer grows with degree and concurrent connects to one hub no longer serialise. edges(), connection_count(), cascade delete, expand_node/subgraph_bfs and Root rehydration read the edge collection; legacy arrays are ignored on read and dropped on the next save. - resolve_edge_ids_mode(): instance attribute > JVSPATIAL_NODE_EDGE_IDS env > adapter class default (wrappers unwrapped via `inner`) - `jvspatial migrate strip-node-edges` (dry run by default, --apply), backed by strip_node_edges() on PostgresDB (one PK keyset pass) and MongoDB (batched $unset) - expand_node pages from the edge collection by id; adds keyset `after` / `pagination.next_after` alongside the int cursor - Fix: QueryEngine.apply_update ignored $pull, leaving stale edge ids on Postgres rows in persist mode after disconnect/delete - Bench harness: warm pool and re-open connections before the burst; baseline re-measured, Phase 1 numbers in the bench doc Bench (100k hub, p50): connect 160 -> 1.9 ms, save 1161 -> 0.7 ms, 32x concurrent connect 10.3 s -> 19 ms wall, hub row 2.5 MB -> 136 B. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Phase 2 of the hub-node scale remediation. Node.nodes() normalises every
filter shape - classes (subclass-inclusive on the node side), names,
lists, {Name: criteria} dicts and property kwargs - to entity lists plus
record-path queries and sends them, limit included, to the backend's
find_connected_nodes in one round trip, direction="both" included.
- PostgresDB: shared SQL builder; join form (one edge type, one
direction: duplicate-free under the (source, target, entity) unique
index, LIMIT streams) or id IN (...) semi-join (dedups); filters that
do not translate raise NotImplementedError so nothing is dropped;
count_connected_nodes, find_connected_nodes_bulk (ROW_NUMBER per source)
- MongoDB ($lookup aggregation) and SQLite (json_extract join) implement
the same contract; table-alias support in both SQL translators
- Node.count_nodes(), Node.nodes_page() (keyset; cursor helpers moved to
core/pager.py and shared with GraphContext.find_page),
nodes_bulk(limit_per_source=); count_neighbors delegates to count_nodes
- Python fallback keeps every filter and filters in the database find
- Fix: list-form edge filters were ignored by nodes()
- Fix: ObservableDatabase exposed find_connected_nodes/traverse even when
the wrapped backend lacks them
Bench (100k hub): nodes(edge=[E], node=["Leaf"], limit=20) 4.9 s / 201
round trips -> 1.2 ms / 1; count 4.8 s (len(nodes())) -> 83 ms / 1.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ObservableDatabase (create_database(observe=True)) and CachingDatabase subclass Database, whose default bulk_save_detailed is a serial save() loop. Because neither wrapper defined the method, that default shadowed their __getattr__ forwarding: a wrapped Postgres COPY or Mongo bulk_write ran as one round trip per record (~0.6 ms/row on loopback). Both wrappers now forward it; the cache refreshes saved ids and drops failed ones. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Phase 3 of the hub-node scale remediation.
- PostgresDB.create_index indexes entity/id/tenant_id as the real
columns the translator compares, emits DESC NULLS LAST (the order
find(sort=...) uses), and rebuilds indexes defined by the old rules in
place (the edge (source, target, entity) unique index indexed
data->entity, which no query used)
- ensure_indexes scopes per-class (annotation-declared) indexes to the
class's entity: (entity, <fields>) by default, WHERE entity = ... with
partial_by_entity; the unscoped pre-0.0.18 index is dropped once
replaced. attribute()/compound_index() gain the scoping options
- gin_index="off" / JVSPATIAL_PG_GIN_INDEX=off skips the whole-document
GIN on new collections; find() warns once for $all/$elemMatch then
- $text: to_tsvector('simple', ...) @@ plainto_tsquery on Postgres,
GIN via @fulltext_index / attribute(fulltext=True); QueryEngine
evaluates the same semantics in memory; MongoDB strips $fields
- escape_regex() helper; find_edges_between(limit=)
- Bench: typed-find gate on a 1M-row shared node table
Bench: sorted, limited typed find at 1M rows, GIN off: p95 9.2 -> 3.0 ms,
one index scan and no sort (0.0.17 needed a BitmapAnd + sort).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Phase 5 of the hub-node scale remediation (verification of the tenancy and pooling paths). - traverse, find_one_and_update, find_one_and_delete and bulk_save_detailed took a raw pool connection without the app.tenant_id GUC, so under enable_rls they saw nothing (atomic ops returned None, traverse no hops) and COPY failed the policy's WITH CHECK before falling back to per-record saves. They now use the tenant-scoped connection (nested transactions become savepoints). - New RLS tests: two tenants sharing one hub id across find_connected_nodes / count_connected_nodes / _bulk, traverse, atomic ops and bulk_save_detailed (unprivileged role). - JVSPATIAL_POSTGRES_COMMAND_TIMEOUT for PostgresDB; postgres guide gains a pool-sizing rule of thumb and transaction-pooler notes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Derive adjacency from the edge table, push neighbour filters/limits/counts into SQL, entity-leading Postgres indexes, optional GIN, and $text search. Final pre-release bench at 51186f1 confirms gates still hold. Co-authored-by: Cursor <cursoragent@cursor.com>
Benchmark comparisonThreshold: ±25% (informational, does not block merge)
|
Rename bench/index fixture field track_id → group_id, neutralize consumer names in CHANGELOG, and use hub/Leaf in optimization examples so the remediation stays framework-generic. Co-authored-by: Cursor <cursoragent@cursor.com>
Benchmark comparisonThreshold: ±25% (informational, does not block merge)
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
edge_ids_mode="derive"on Postgres/MongoDB/SQLite): hubconnect()/save()no longer rewrite or row-lock node rows; cost is flat across degree. Migration:jvspatial migrate strip-node-edges --dsn … --apply(dry-run by default). Opt out withJVSPATIAL_NODE_EDGE_IDS=persist.nodes(edge=[E], node=[…], limit=N)is one round trip; newcount_nodes(),nodes_page(),nodes_bulk(limit_per_source=…).JVSPATIAL_PG_GIN_INDEX=off;$text→to_tsvector/plainto_tsquery;escape_regex().traverse;JVSPATIAL_POSTGRES_COMMAND_TIMEOUTand documented pooler sizing.docs/bench/2026-09-hub-node-baseline.md.Adopter notes (any jvspatial app)
After upgrade, apps that want the new defaults can set:
JVSPATIAL_NODE_EDGE_IDS=derive(already the Postgres/MongoDB/SQLite default)JVSPATIAL_PG_GIN_INDEX=off(recommended for large deployments whose queries are scalar/equality, not@>/$elemMatch)Then run
jvspatial migrate strip-node-edges --dsn "$JVSPATIAL_POSTGRES_DSN" --applyandVACUUM (ANALYZE)onnodewhen convenient.Test plan
pre-commit run --all-filestest_edge_ids_derive,test_node_query_pushdown,test_postgres_indexes_text,test_postgres_tenancy_graph)docs/bench/2026-09-hub-node-final.jsonl)0.0.18publish fromversion.pyorigin/devonto updatedmainNote: Local full
pytestalso surfaces pre-existingtests/storage/test_security.pyMIME detection failures that reproduce onmain(filetype/libmagic env drift) — not introduced here.