DDIR: corgi pin to 13f3ab8, and the chunk merge on survey_groups - #861
Merged
Conversation
Two corgi PRs landed: the indexed sort (frankmcsherry/WIP#24), which arranges a chunk and orders the reduce's candidates without gathering at every level of the shape, and the rank-at-a-time survey (frankmcsherry/WIP#26), which reports the interleaving of two sorted columns as maximal exclusive ranges and equal classes as their ranges on both sides. The pin moves to master 13f3ab8. The chunk merge, which compared row by row with compare_at and whose own comment named survey_groups as the interface it waited for, is now the exclusive runs copied by range and each equal class merged once on its times; corgi walks the shape once per level per chunk. Medium scale, one worker, corgi backend, medians of three, de0f2ac -> here. Initial epoch: ast_hier 949 -> 686 ms, ast 562 -> 462, kcore 303 -> 265, scc 1.98 -> 1.78 s, stable 327 -> 295, tour 745 -> 704, reach 188 -> 177, unnest 248 -> 238, adt 44.0 -> 41.8. Churn epoch: stable 6.2 -> 5.9 ms, reach 10.1 -> 9.8, kcore 2.4 -> 2.3, tour 13.3 -> 13.1, scc 58.0 -> 57.8, the rest within a tenth. Of the initial-epoch gain, the sort alone was 2-10%, measured at the intermediate pin 5bee8df; the merge is the rest. Tests pass. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QDsLC46QQW9aBrksaWad6T
…s are 2^20 The chunker accumulated TARGET rows before sorting them, so the one constant set both how much arrives in one radix sort and how big the chunks the merge and advance emit are, which is what the fueled merger yields on. They pull different ways: a big bundle is cheaper to sort than to merge from pieces, while a big chunk makes the merger yield less often. INGEST is now the bundle, 2^24 rows, the whole of a large epoch's input at once; TARGET the chunk, 2^20. Swept at 1M nodes, one worker, medians of two, initial epoch / churn epoch: TARGET 2^18 (before): ast 6.98 s / 2.58 s, kcore 4.27 / 0.027, scc 22.99 / 0.711 TARGET 2^22 for both roles: ast 5.81 / 2.42, kcore 2.92 / 0.026, scc 22.73 / 0.692 INGEST 2^22, TARGET 2^18: ast 6.19 / 2.59, kcore 3.42 / 0.026, scc 22.16 / 0.705 INGEST 2^24, TARGET 2^20 (here): ast 5.46 / 2.57, kcore 3.05 / 0.026, scc 22.66 / 0.685 TARGET 2^24 for both roles: ast 5.07 / 2.69, kcore 2.85 / 0.026, scc 22.62 / 0.689 The ingest bundle alone is half to two thirds of the gain; the chunk size is the rest and the part that can hurt churn. At medium scale, where an epoch's input fits one bundle either way, 2^18 to 2^20 on the chunk takes ast 462 to 419 ms and kcore 265 to 236, churn flat. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QDsLC46QQW9aBrksaWad6T
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two corgi PRs landed this weekend: the indexed sort (WIP#24), which arranges a chunk and orders the reduce's candidates without gathering at every level of the shape, and the rank-at-a-time survey (WIP#26), which reports the interleaving of two sorted columns as maximal exclusive ranges and equal classes as their ranges on both sides. This moves the pin to master 13f3ab8 and takes the second one up.
The merge.
CorgiChunk::mergecompared row by row withcompare_at, and its own comment namedsurvey_groupsas the interface it was waiting for. It is now the exclusive runs copied by range (push_range,extend_from_slice) and each equal class merged once on its times, where consolidation happens; corgi walks the shape once per level per chunk. The group form is simpler than the pairwise one onsurvey-merge: no group-end scan, no done marks, nou64_lanes.Numbers. Medium scale, one worker, corgi backend, medians of three, de0f2ac → here:
Chunk sizing, second commit. The chunker accumulated
TARGETrows before sorting them, so one constant set both the ingest bundle and the chunks the merge and advance emit. They are two knobs now:INGESTat 2^24, the whole of a large epoch's input in one radix sort, andTARGETat 2^20. Swept at 1M nodes, one worker, medians of two, initial epoch / churn epoch: target 2^18 as before, ast 6.98 s / 2.58 s, kcore 4.27 / 0.027, scc 22.99 / 0.711; ingest 2^22 with target 2^18, ast 6.19 / 2.59, kcore 3.42 / 0.026, scc 22.16 / 0.705; ingest 2^24 with target 2^20, ast 5.46 / 2.57, kcore 3.05 / 0.026, scc 22.66 / 0.685; target 2^24 for both roles, ast 5.07 / 2.69, kcore 2.85 / 0.026. The bundle alone is half to two thirds of the gain with churn untouched; the chunk size is the rest and the part that costs churn past 2^22. At medium scale the chunk size alone takes ast 462 to 419 ms and kcore 265 to 236, churn flat.Of the initial-epoch gain from the first commit, the sort alone was 2 to 10%, measured at the intermediate pin 5bee8df; the merge is the rest. The numbers were taken with
bench-spike's harness copied in, since master-next does not carry it. Tests pass.🤖 Generated with Claude Code
https://claude.ai/code/session_01QDsLC46QQW9aBrksaWad6T