Skip to content

Subgraph mode: throughput - #1736

Closed
moose-code wants to merge 7 commits into
moose/subgraph-compat-fixesfrom
moose/subgraph-perf
Closed

moose-code wants to merge 7 commits into
moose/subgraph-compat-fixesfrom
moose/subgraph-perf

Conversation

@moose-code

@moose-code moose-code commented Oct 14, 2026 •

Copy link
Copy Markdown
Member

Stacked on #1732 (moose/subgraph-compat-fixes). Five perf commits, plus merges of #1732's head up to 787e304 (the only conflict was an import line). Measured on 9918a53.

What it changes

commit change
count the sync bridge's replay rounds ENVIO_SUBGRAPH_SYNC_STATS=1 prints SGSTATS {json} with runs, rounds and a rounds histogram for preload and for execute. It's off by default, and when off it costs one branch.
hash crypto.keccak256 natively Adds a napi subgraphKeccak256 (alloy) to replace viem's JS keccak and its hex round trip. A call drops from 6.4 µs to 1.1 µs. ENS hashes on every NewOwner, once per replay round, and keccak was 15% of its CPU.
let preload read past a miss; suspend without building an Error In preload, a miss now answers "absent" and the round keeps going, so independent reads land in the same batch. That's bounded at 32 reads per round, and an effect never runs on a speculated value. Execute still suspends on every miss. Separately, throw(Suspend) used to build an Error with a stack trace on every miss; it now throws one prebuilt value.
hold the mapping scope in a variable, not an AsyncLocalStorage Mapping rounds are synchronous. A module's top level keeps the scope until its import settles. On Node 22 this also removes the promise hooks that ran on every promise (about 4% of CPU).
convert rows and event parameters with less work per call The per-entity layout is cached, so we no longer build three new Set per load and per save. Bigint columns are no longer re-parsed. TypedMap.getEntry no longer allocates a closure. Each event parameter is decoded on first read instead of all at once.

Measurements

Paired A/B runs: the baseline (9918a53 plus the stats commit) and this branch index the same fixed window at the same time, so load from the shared box hits both equally. Envio 3.14 subgraph mode, Postgres, nice -n 10.

subgraph, window baseline this branch speedup
ENS, blocks 0→5,000,000, 267,661 events (handler-bound) 3,003 ev/s, 109.3 s CPU 3,827 ev/s, 87.9 s CPU x1.27 events/s, x1.24 CPU
Polymarket OI, 50.6M→61M from a snapshot, 4,033,124 events (solo, back to back) 26,057 ev/s, 161.2 s CPU 27,795 ev/s, 134.7 s CPU x1.07 events/s (this window is now HyperSync-bound), x1.20 CPU, x1.27 handler time

Per commit (ENS, cumulative): keccak x1.14 → + speculative preload x1.13 (within noise; rounds don't change on ENS) → + sync scope x1.22 → + conversions x1.25.

Correctness: after every pair I compared each entity table row for row (EXCEPT ALL both ways on the row's text form). Every comparison had 0 differing rows: ENS (27 tables, e.g. Domain 201,286 and NewOwner 226,531 rows) and Polymarket (4 tables).

Tests: on the merged head (787e304 merged in, addon rebuilt), vitest run test/Subgraph test/SyncContext passes 64 of 64 files and 185 of 185 tests. There are two new SyncContext cases: independent preload reads batch into one round, and an effect is only ever called with stored values. pnpm check-graph-ts is clean.

The full write-up (profiles, replay-round counts, what's left) is in spec/PERF.md in the hackathon workspace. The biggest remaining items:

  • performance.now() handler timing costs 11.5% of Polymarket CPU on a box with an HPET clocksource.
  • ENS execute-phase misses: preload can't see the batch's own writes, so Resolver.load costs a DB round trip on 6.5% of events.
  • ENSRainbow lookups go through undici fetch.

🤖 Generated with Claude Code

moose-code and others added 5 commits October 14, 2026 20:39
…SYNC_STATS

Prints one SGSTATS {json} line on stderr every 30 s and at exit: runs, rounds
and a rounds histogram per phase. A preload round that suspends is batching
working; an execute round that suspends is a database round trip the batch
waits on, one event at a time. Off by default, and then a single branch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ENS computes a node hash (keccak256 of parent node ++ label) on every NewOwner,
and the sync bridge reruns the mapping once per replay round, so the hash runs
3-5 times an event. viem's pure-JS keccak (plus the hex round trip back into a
ByteArray) took 6.4 us a call and 15% of all indexing CPU on ENS; the addon's
alloy keccak takes 1.1 us, napi crossing included.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…an Error

The sync bridge reruns a mapping from the top for each read that misses, so a
handler with three independent loads took four preload rounds, each a batched
database round trip and a rerun of everything before it (ENS: 2.8 rounds an
event). Preload now answers a miss as absent and runs on, so the reads after it
that don't depend on it join the same batch; the round is replayed once they
land. Preload results are discarded either way, so what is stored can't
change: execute still suspends on every miss. An effect never runs on a
speculated value (it could be an RPC call with a wrong argument), and at most
32 reads a round are speculated, for a mapping that loops on a read.

throw(Suspend) also built a fresh Error with a stack trace on every suspend;
one prebuilt value is thrown instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A mapping runs synchronously from the start of a round to its end, and every
graph-ts call that reads the scope happens inside that span. The one async
reader is a module's top level, which runs during its import; mappings are
imported one at a time, so the scope is simply held until the import settles.

On Node 22 a live AsyncLocalStorage installs promise hooks that run on every
promise the indexer creates — 4% of indexing CPU on Polymarket and ENS, in
code that has nothing to do with the mapping.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every load and save converts a row, and the schema doesn't change under a
running indexer:
- toEntity/toRow built three Sets from the schema on every call; the layout
  of each entity type, and each column's field and kind, are now worked out
  once.
- A bigint column read back through BigInt(String(value)); it is wrapped as is.
- TypedMap.getEntry allocated a closure per lookup.
- An event's parameters were all decoded (address checksums, hex to bytes,
  arrays of BigInts) the first time a handler read any one of them, through
  graph codegen's getters; each is now decoded the first time it is read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Oct 14, 2026

Copy link
Copy Markdown
Contributor

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
moose-code added a commit that referenced this pull request Oct 14, 2026
Host network slugs + evm-<id>, RPC sync source for non-HyperSync chains, AS-only npm libraries, graph-ts 0.38 exports. Its indexed-dynamic-param commit (f468ba6) duplicates fixer3's 6da34bf: kept arrivesAs on #1736's lazy params and dropped eventInputType. The resolve hook tries sg-install's fetched modules, then sg-names' AssemblyScript package entry (in the project, then in the fetched modules); calling_batch_size and FILE_DATA_SOURCE_ENTITY sit beside rpc_for_chain.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@DZakh

DZakh commented Oct 22, 2026

Copy link
Copy Markdown
Member

Merged

@DZakh DZakh closed this Oct 22, 2026

This branch was successfully deployed

1 active deployment
internal — 325e2ceb Deployed Oct 14, 2026 by moose-code via authorize #3721
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants