Repository navigation
Subgraph mode: throughput - #1736
Closed
moose-code wants to merge 7 commits into
Closed
moose-code wants to merge 7 commits into
moose-code wants to merge 7 commits into
Conversation
…SYNC_STATS
Prints one SGSTATS {json} line on stderr every 30 s and at exit: runs, rounds
and a rounds histogram per phase. A preload round that suspends is batching
working; an execute round that suspends is a database round trip the batch
waits on, one event at a time. Off by default, and then a single branch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ENS computes a node hash (keccak256 of parent node ++ label) on every NewOwner, and the sync bridge reruns the mapping once per replay round, so the hash runs 3-5 times an event. viem's pure-JS keccak (plus the hex round trip back into a ByteArray) took 6.4 us a call and 15% of all indexing CPU on ENS; the addon's alloy keccak takes 1.1 us, napi crossing included. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…an Error The sync bridge reruns a mapping from the top for each read that misses, so a handler with three independent loads took four preload rounds, each a batched database round trip and a rerun of everything before it (ENS: 2.8 rounds an event). Preload now answers a miss as absent and runs on, so the reads after it that don't depend on it join the same batch; the round is replayed once they land. Preload results are discarded either way, so what is stored can't change: execute still suspends on every miss. An effect never runs on a speculated value (it could be an RPC call with a wrong argument), and at most 32 reads a round are speculated, for a mapping that loops on a read. throw(Suspend) also built a fresh Error with a stack trace on every suspend; one prebuilt value is thrown instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A mapping runs synchronously from the start of a round to its end, and every graph-ts call that reads the scope happens inside that span. The one async reader is a module's top level, which runs during its import; mappings are imported one at a time, so the scope is simply held until the import settles. On Node 22 a live AsyncLocalStorage installs promise hooks that run on every promise the indexer creates — 4% of indexing CPU on Polymarket and ENS, in code that has nothing to do with the mapping. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every load and save converts a row, and the schema doesn't change under a running indexer: - toEntity/toRow built three Sets from the schema on every call; the layout of each entity type, and each column's field and kind, are now worked out once. - A bigint column read back through BigInt(String(value)); it is wrapped as is. - TypedMap.getEntry allocated a closure per lookup. - An event's parameters were all decoded (address checksums, hex to bytes, arrays of BigInts) the first time a handler read any one of them, through graph codegen's getters; each is now decoded the first time it is read. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Comment |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
moose-code
added a commit
that referenced
this pull request
Oct 14, 2026
Host network slugs + evm-<id>, RPC sync source for non-HyperSync chains, AS-only npm libraries, graph-ts 0.38 exports. Its indexed-dynamic-param commit (f468ba6) duplicates fixer3's 6da34bf: kept arrivesAs on #1736's lazy params and dropped eventInputType. The resolve hook tries sg-install's fetched modules, then sg-names' AssemblyScript package entry (in the project, then in the fetched modules); calling_batch_size and FILE_DATA_SOURCE_ENTITY sit beside rpc_for_chain. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Member
|
Merged |
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #1732 (
moose/subgraph-compat-fixes). Five perf commits, plus merges of #1732's head up to 787e304 (the only conflict was an import line). Measured on 9918a53.What it changes
ENVIO_SUBGRAPH_SYNC_STATS=1printsSGSTATS {json}with runs, rounds and a rounds histogram for preload and for execute. It's off by default, and when off it costs one branch.crypto.keccak256nativelysubgraphKeccak256(alloy) to replace viem's JS keccak and its hex round trip. A call drops from 6.4 µs to 1.1 µs. ENS hashes on every NewOwner, once per replay round, and keccak was 15% of its CPU.throw(Suspend)used to build anErrorwith a stack trace on every miss; it now throws one prebuilt value.new Setper load and per save. Bigint columns are no longer re-parsed.TypedMap.getEntryno longer allocates a closure. Each event parameter is decoded on first read instead of all at once.Measurements
Paired A/B runs: the baseline (9918a53 plus the stats commit) and this branch index the same fixed window at the same time, so load from the shared box hits both equally. Envio 3.14 subgraph mode, Postgres,
nice -n 10.Per commit (ENS, cumulative): keccak x1.14 → + speculative preload x1.13 (within noise; rounds don't change on ENS) → + sync scope x1.22 → + conversions x1.25.
Correctness: after every pair I compared each entity table row for row (
EXCEPT ALLboth ways on the row's text form). Every comparison had 0 differing rows: ENS (27 tables, e.g. Domain 201,286 and NewOwner 226,531 rows) and Polymarket (4 tables).Tests: on the merged head (787e304 merged in, addon rebuilt),
vitest run test/Subgraph test/SyncContextpasses 64 of 64 files and 185 of 185 tests. There are two newSyncContextcases: independent preload reads batch into one round, and an effect is only ever called with stored values.pnpm check-graph-tsis clean.The full write-up (profiles, replay-round counts, what's left) is in
spec/PERF.mdin the hackathon workspace. The biggest remaining items:performance.now()handler timing costs 11.5% of Polymarket CPU on a box with an HPET clocksource.Resolver.loadcosts a DB round trip on 6.5% of events.fetch.🤖 Generated with Claude Code