DDIR bench spike, parked: findings and open items - #860
Draft
frankmcsherry wants to merge 25 commits into
Draft
Conversation
…rofile `interactive/bench/bench.py` runs every workload as a session on the server binary (install, load, tick, tick R) across backends, worker counts and repetitions, and writes JSONL + markdown reports named by date and revision. Workloads: scc, reach, kcore, stable, tour, adt, ast, unnest on seeded random graphs at three scales, and the 33 AoC parts. `--profile` records one session under samply. Reports so far: the small-scale sweep, and a first reading of an scc/corgi profile with candidate hot spots (find_ranges seeking from scratch per needle; `CorgiChunk::advance` materializing and re-advancing repeated times). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QDsLC46QQW9aBrksaWad6T
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QDsLC46QQW9aBrksaWad6T
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QDsLC46QQW9aBrksaWad6T
…ngs so far unnest's 3.2s "load" was 91% stderr writes from its inspect tap, tour's 10s likewise. The harness now times stripped copies of the programs. Findings across the scc/ast/unnest profiles and the sweeps are in reports/2026-09-04-findings.md. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QDsLC46QQW9aBrksaWad6T
`CorgiChunk::advance` materialized a `Product<u64, PointStamp>` per row, joined it with the frontier (another allocation past one coordinate), sorted the owned times, and cloned the survivors. A batch holds few distinct times, so rows now carry an index into a per-batch table of distinct source times (kept as columnar refs, binary-searched) and their advanced forms; the owned time is built once per distinct time. One scratch buffer serves every group, where a fresh `Vec` per group had been an allocation per row. scc on corgi at medium scale: initial epoch 1.99→1.90 s, churn epoch 58.0→56.2 ms; kcore, reach and tour unchanged. Corgi gate passes. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QDsLC46QQW9aBrksaWad6T
…tput block The projection's operand shapes are static per join, as they are for the linear operators' Plan; the graph is now pinned at the first block's shapes and reused. Small effect on its own (one block per install on tiny inputs is also one compile), but it removes the per-block scratch lowering and type check from the join's hot loop. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QDsLC46QQW9aBrksaWad6T
`compile` asks `shape_of_term` for the shapes of an `if`'s branches (a `case`'s arms, a fold's parts, ...) before compiling them, and each ask is a full scratch lowering of that subterm plus a `corgi::shape_of` evaluation. So nested conditionals were re-lowered once per enclosing level — 3^depth in all. AoC day 7, five `if`s deep, spent half of every install there. The shape memo is keyed by term address, environment shapes and expected shape, and lives only for the span of the outermost `compile` / `shape_of_term` on the thread, so an address can never be met again with a different term behind it. Day 7 part 1's tick on corgi: 7.0 → 3.4 ms; the AoC suite and all tests pass. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QDsLC46QQW9aBrksaWad6T
cc.ddp propagates the least reachable node id over the edges read as undirected, staggered by magnitude as scc does; triangles.ddp counts a < b < c closures with two joins and no recursion. Both are in the corgi gate and in the benchmark harness, where the join-only program shows corgi's widest lead (2.4x per churn epoch at one worker). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QDsLC46QQW9aBrksaWad6T
…findings Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QDsLC46QQW9aBrksaWad6T
…n stamps Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QDsLC46QQW9aBrksaWad6T
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QDsLC46QQW9aBrksaWad6T
…ales) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QDsLC46QQW9aBrksaWad6T
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QDsLC46QQW9aBrksaWad6T
`collect_present` probed every chunk of the arrangement with the whole change set. A batch's chunks partition its key range, so each chunk now takes only the needles inside its own range (two partition_points on the sorted change set) and a chunk with none is skipped. Nothing changes at medium scale (one chunk per batch); at 1M nodes / 2M edges on corgi, one worker, scc's churn epoch goes 700 -> 657 ms and cc's 178 -> 160 ms. Corgi gate passes. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QDsLC46QQW9aBrksaWad6T
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QDsLC46QQW9aBrksaWad6T
…outs Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QDsLC46QQW9aBrksaWad6T
…es in the merge) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QDsLC46QQW9aBrksaWad6T
…mpare per row `merge_present` verifies that no id in a run covers two real keys before it trusts id order. For a non-leaf key that was a structural `compare_at` per adjacent pair, and every row of a group shares its id with its predecessor, so nearly every row of every retire paid one. One `compare_adjacent` pass over the key column answers the same question. ast_hier (a `(bucket, hash)` key) on corgi at medium, one worker: initial epoch 491 -> 422 ms, churn epoch 19.7 -> 17.0 ms; leaf-keyed programs are not on this path and are unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QDsLC46QQW9aBrksaWad6T
`CorgiChunk::merge` ordered its two-pointer walk with `compare_at` per row, a shape walk per comparison. When both chunks' keys and values are bare `u64` leaves — integer keys are their own identifiers, and integer values are the common case — or the value is unit, as under `distinct`, structural order is one or two integer compares read straight off the lanes. Other shapes keep `compare_at`. Medium scale on corgi, one worker, initial epoch / churn epoch: scc 1.90 s -> 1.72 s / 56 -> 54 ms, cc 587 -> 534 ms / 15.4 -> 14.7 ms, kcore 281 -> 253 ms / 2.3 -> 2.1 ms, reach 172 -> 168 ms / 10.3 -> 9.6 ms. Test suite passes. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QDsLC46QQW9aBrksaWad6T
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QDsLC46QQW9aBrksaWad6T
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QDsLC46QQW9aBrksaWad6T
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QDsLC46QQW9aBrksaWad6T
…-counted initial epochs Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QDsLC46QQW9aBrksaWad6T
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QDsLC46QQW9aBrksaWad6T
Disposition of everything the spike left behind: the six backend commits ready to land, the merge decision (row-wise fast path vs survey merge, and what the latter waits on), the open items with a measured shape, the things tried and recorded as no, and the loose ends elsewhere. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QDsLC46QQW9aBrksaWad6T
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A draft PR to hold the benchmark side spike that followed #858, so its findings and open items are public rather than private notes. Not for merging as is.
interactive/bench/reports/2026-09-06-open-items.md— the disposition of everything on the branch: six corgi-backend commits ready to land as a small PR, the merge decision (row-wise fast path vs the survey merge onsurvey-merge, and what the latter waits on in corgi), open items with a measured shape, things tried and recorded as no.interactive/bench/reports/2026-09-04-findings.md— twenty sections of findings (the reduce is the cost; corgi vs vec; four-worker scaling; TPC-H/LDBC; the merge, batched; sort breadth-first).interactive/bench/bench.py— the harness, driving the live server binary; the reports are its record.🤖 Generated with Claude Code
https://claude.ai/code/session_01QDsLC46QQW9aBrksaWad6T