perf(evolution): ⚡ carry cross-rank queries as a width-adaptive record - #263
Open
diagonal-hamiltonian wants to merge 5 commits into
Open
perf(evolution): ⚡ carry cross-rank queries as a width-adaptive record#263diagonal-hamiltonian wants to merge 5 commits into
diagonal-hamiltonian wants to merge 5 commits into
Conversation
diagonal-hamiltonian
requested review from
fpietra,
ludmilaasb and
robertodr
as code owners
August 20, 2026 20:32
diagonal-hamiltonian
force-pushed
the
pr/query-wire-v3
branch
from
August 20, 2026 20:35
1a7bb1e to
407a6a2
Compare
|
Docs preview: https://pr-263.monoprop-docs.pages.dev |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #263 +/- ##
=======================================
Coverage 97.70% 97.70%
=======================================
Files 14 14
Lines 742 742
Branches 98 98
=======================================
Hits 725 725
Misses 12 12
Partials 5 5
Flags with carried forward coverage won't be shown. Click here to find out more. |
diagonal-hamiltonian
marked this pull request as draft
August 21, 2026 07:17
diagonal-hamiltonian
force-pushed
the
pr/query-wire-v3
branch
from
August 21, 2026 08:10
407a6a2 to
dce15d1
Compare
diagonal-hamiltonian
marked this pull request as ready for review
August 21, 2026 08:13
diagonal-hamiltonian
force-pushed
the
pr/query-wire-v3
branch
from
August 21, 2026 08:22
dce15d1 to
d05733a
Compare
github-actions
Bot
added
documentation
diagonal-hamiltonian
force-pushed
the
pr/query-wire-v3
branch
from
August 21, 2026 08:25
d05733a to
249aee8
Compare
This was referenced Aug 21, 2026
robertodr
force-pushed
the
pr/query-wire-v3
branch
from
August 24, 2026 13:46
8dcc0fd to
8579fa9
Compare
Member
|
you did indeed tell me to stack my stuff on top of this one |
diagonal-hamiltonian
added a commit
that referenced
this pull request
Aug 24, 2026
#267 landed on main while #263 was open, and its self_resolve_mark_bounded_by_combined_size fixture feeds the engine through detail::query_push -- the dense record #263 retires. The rebase onto that main left the call in place, so origin/pr/query-wire-v3 does not compile: query_push now lives in cpp/tests/dense_query_reference.h as test_ref::query_push, an oracle for the codec differential and not an engine entry point. Ported rather than renamed. Under the positions-staged self leg the fixture would have tripped resolve_self_queries' own assertion -- "a self-owned query was encoded instead of staged" -- because at my_rank == 0 the engine reads self_stage_ and requires queries_r[my_rank] empty. It now pushes each term's ascending positions onto self_stage_, which is what the scan does, and every assertion the case made about marking past combined_size is unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Five changes to the layer-build query path, squashed because the middle three share one data structure and the first is not measurable without them. The structural cutoff is decided from a (k, d) digest carried inline rather than from the wider comparison it replaced. Cross-rank queries then travel as a width-adaptive record, dropping the fixed stride the older CompactQuery design assumed. Queries whose owner is this rank resolve from positions instead of going through the cross-rank path at all. Two further changes ride with them and are named here because a reviewer cannot revert along a mechanism the message does not mention. Sink::kStride becomes incoming_layout()/querier_layout(): ContractSink's stride was the fused width while on_response_block was called with the plain buffer and took query_phase's plain-stride default, so the two agreed only by accident, and a wrong layout reads a neighbouring record's phase -- a coefficient sign flip, not a crash. And bulk_insert gains a group-prefetched hashed path, shared by the cross-rank and self inserts. Tests cover the paths that can disagree: a dense reference implementation the sparse path is differentially compared against, the digest's tie cases, the spill boundary where a row exceeds the inline width, and the prefetched insert. OperatorIndex gains a public overflow_size() so a test can assert the dense and positional insert paths spill identically -- the claim this change rests on, and otherwise unreachable from outside the class. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The self leg encoded a query record and decoded it straight back on the same rank: Scan pushed through QueryCodec for every destination including my_rank, and resolve_range_ inflated it again into self_pos_flat_ before calling find_batch_positions -- whose signature is already the flat-positions form the encoder had just destroyed. At one partition that round trip is 100% of queries against exchange_s = 0.0000. SelfQueryStage carries them as positions instead, and merge_partner_positions supplies the positions, k, the cutoff digest's d and the overlap in the one merge that replaces the walk. Four measured shapes were rejected on the way and are recorded at their sites so they are not tried again: a lambda in the merge (15,279,191 non-inlined calls), thread_local for the position buffer (__tls_get_addr, 54.7M), a byte loop for row_eq_positions (glibc's AVX2 memcmp wins at ~5 bytes), and walking the dense partner instead of merging (+229.1M, find_next is a serial dependence chain). Layout 1x1 against the query-wire record, 10 reps interleaved, 10/10 p=.00: build_graph[hubbard] 0.87x, propagate[hubbard] 0.90x, build_graph[pauli] 0.79x, propagate[pauli] 0.76x. energy/gradient 1.00x as the null control, peak RSS flat. Callgrind: 9,720.99M -> 9,265.72M instructions, floor 0.052%. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The record chose per-term between three closed forms -- raw lanes, gap coding and a raw bitmap -- behind a 2-bit mode field, and sized a stack Writer array from the worst case of the three. Replayed over the 106,368 captured query records, BITMAP was chosen 0 times and FIXED only on word-count ties, so the argmin was paying for two branches that never won. Gap coding alone, with the mode field's two bits and one of k's six spent on a narrower header: 11 bits against 14. gw = bit_width(max gap) <= kPosBits, so kPosBits + (k-1)*gw <= k*kPosBits and gap is never wider than raw lanes at any k. A raw mask does win once k*kPosBits > kBits, from k = 34 at 128 modes, where this record costs one word more -- above the k <= 32 that Pauli cutoff 16 implies and 11 above the widest k ever captured. That is the price of one code path, and sparse_record_documents_what_deleting_the_argmin_cost asserts it against the deleted encoder's own three formulas rather than leaving it to be rediscovered. Dropping the argmin also dropped the array it sized: gap coding emits bits monotonically, so the encoder streams into the output buffer through one accumulator and no longer zeroes 48 B per push. Bytes per record, predicted from the captured streams before the code existed and confirmed by it: hubbard c10 8.289 -> 8.038, pauli c12 15.519 -> 15.180, pauli c6 8.750 -> 8.438. Both formats decode the same positions on all 106,368 records (identical checksums). Callgrind over a standalone replay, where the binary is bit-exactly repeatable: push 345.1 -> 292.5 instructions per record (-15.2%), decode -3.0, every libc counter unchanged. The byte win is ~3% of a payload whose byte-proportional share of exchange is ~24%, so it is worth <= 0.2% of wall clock and no timing claim is made for it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
#267 landed on main while #263 was open, and its self_resolve_mark_bounded_by_combined_size fixture feeds the engine through detail::query_push -- the dense record #263 retires. The rebase onto that main left the call in place, so origin/pr/query-wire-v3 does not compile: query_push now lives in cpp/tests/dense_query_reference.h as test_ref::query_push, an oracle for the codec differential and not an engine entry point. Ported rather than renamed. Under the positions-staged self leg the fixture would have tripped resolve_self_queries' own assertion -- "a self-owned query was encoded instead of staged" -- because at my_rank == 0 the engine reads self_stage_ and requires queries_r[my_rank] empty. It now pushes each term's ascending positions onto self_stage_, which is what the scan does, and every assertion the case made about marking past combined_size is unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
prek pins clang-format v21.1.0 and three of the lines PR A and PR B wrote were not in its shape: two comment columns in PartnerMerge.h, a four-line parameter list in mpi_utils_tests.cpp that fits on one, and a BOOST_TEST message split a token earlier than needed. `git diff -w` reports PartnerMerge.h as empty and the other two as line joins with every token conserved, so nothing here can reach codegen -- it is the lint leg only, which was the one red left on the PR. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
diagonal-hamiltonian
force-pushed
the
pr/query-wire-v3
branch
from
August 26, 2026 21:51
94a33f5 to
9ff9d73
Compare
Collaborator
Author
|
Rebased onto current The commit that mattered is #270 ( Verified after the rebase: 270/270 #296 stacks GF(2)-linear rank routing on top of this and is now based on this branch. |
|
Panadestein
added a commit
that referenced
this pull request
Aug 27, 2026
## What this is Two commits that make a benchmark result **identify the geometry it was measured at**, plus the reader change that surfaces them. Nothing in the engine changes; both commits are Python, outside `cpp/`. - `benches/conftest.py` records `nodes`, `ranks_per_node`, `partitions_env` and `memhwm_max`. - `report.py` / `test_report.py` surface them. ## Why A scaling result is `(nodes, ranks_per_node, partitions)`, and until now a recorded run carried **none** of the three. `meta.ranks` is the MPI world size, which is the product of two of them and cannot be factored back. That matters more than it sounds, because the engine's flat world is `P = ranks_per_node × partitions × nodes` and `HybridComm::size()` returns `r_ * s_` — so `8 ranks × 16 partitions` on 32 nodes and `1 rank × 128 partitions` on 32 nodes are both `P = 4096` and are *not* the same measurement. They differ by 2.06x in wall time (table below). Two runs that differ 2x were previously indistinguishable in the recorded JSON. Three details worth review: - **`nodes` is collective.** It is an `allgather` of `socket.gethostname()`, so `pytest_configure` had to call it *before* the rank-0 early return — every rank must enter it or it hangs. Serial short-circuits to `(1, 1)` without touching MPI. - **`partitions_env` is the environment variable, not a resolved count**, and is labelled "Partitions (requested)" for that reason. No property exposes the count the engine actually chose. A bare `Partitions` column would have claimed otherwise. - **`memhwm_max` exists because `memhwm` is a sum over ranks.** The two differ by the rank count. The old heading said neither, so this renames the existing table to "summed across ranks" and gives the max its own. On a 1024-rank run the sum is 460 GiB and the max is 0.48 GiB; a reader who guessed wrong about which one they were looking at would be off by 1000x. `world` is deliberately *not* emitted: it is derivable from what is recorded, and a stored derived field is a second source of truth that can disagree. ## What it was used for A strong- and weak-scaling study of `propagate` on Hubbard at **1,569,152,761 terms** (cutoff 10, 60 sites, `lower_atol=2.6e-06`), on Deucalion x86 (2x EPYC 7742, 128 cores/node, SMT off). Layout `8 ranks x 16 partitions` per node, so the engine's flat world is `P = 128 x nodes`. Baseline is this branch's parent; the port arm is #263. 3 reps per cell, A/B interleaved inside one allocation with the arm order flipped per rep, ratios taken as the median of per-rep *paired* ratios. ### Strong scaling — the problem is fixed, only the hardware grows | N | P | propagate s | speedup | efficiency | ledger coverage | | ---: | ---: | ---: | ---: | ---: | ---: | | 1 | 128 | 281.0 | 1.00 | 100% | 91.0% | | 2 | 256 | 152.6 | 1.84 | 92% | 85.4% | | 4 | 512 | 103.2 | 2.72 | 68% | 76.5% | | 8 | 1024 | 82.2 | 3.42 | 43% | 65.2% | | 16 | 2048 | 69.4 | **4.05** | 25% | 49.9% | | 32 | 4096 | 88.5 | **3.18** | 10% | 40.4% | **Scaling reverses between 16 and 32 nodes.** Reproduced in two independent passes (a 1-rep pass gave 4.07 → 3.21, a 3-rep pass 4.05 → 3.18). Weak efficiency falls **100.0 / 81.6 / 56.2 / 38.5 / 27.1 / 16.0%** over N = 1…32, from main-arm walls of 18.7 / 21.7 / 32.4 / 49.0 / 69.8 / 116.6 s. It is normalised by the **measured** terms/node recorded in each cell's ledger (92.06–98.07M across the six rungs), not by an assumed-flat target — the rungs are close enough that the distinction is small, and far enough apart that assuming it would have been wrong. Instrument self-check, and the reason the `nodes` key earns its place: `strong/16` and `weak/16` are the *identical* cell (1.569G terms at N=16) run in two separate allocations, and they agree to **0.56%** (base) and **0.26%** (port). Before this commit the two were indistinguishable in the recorded JSON and could not have been cross-checked at all. ### Where the time goes — this is what `time -v` says once User and System are separated Same runs, main arm, median rep, summed over ranks: | N | ranks | user core-s | system core-s | system share | user vs N=1 | system vs N=1 | | ---: | ---: | ---: | ---: | ---: | ---: | ---: | | 1 | 8 | 29,008 | 6,262 | 17.8% | 1.00x | 1.00x | | 4 | 32 | 30,316 | 21,382 | 41.4% | 1.05x | 3.41x | | 8 | 64 | 38,402 | 45,287 | 54.1% | 1.32x | 7.23x | | 16 | 128 | 56,555 | 83,546 | 59.6% | 1.95x | 13.34x | | 32 | 256 | 117,014 | 241,152 | **67.3%** | 4.03x | **38.51x** | Total CPU consumed grows **10.14x** for a problem that does not change, against a 3.18x speedup — so ~90% of the CPU burned at N=32 does not exist at N=1. It is overwhelmingly *system* time, and system time is very nearly linear in MPI rank count (exponent log 38.51 / log 32 = 1.05). ### Layout at fixed flat world — six geometries, same P At N=32, `P = 4096` is reachable at six splits of 128 cores/node, with the MPI world size spanning 32x. This is the measurement `ranks_per_node` exists to make legible: | layout | partitions/rank | MPI ranks | propagate s | vs 8x16 | threads busy | peak RSS sum | | --- | ---: | ---: | ---: | ---: | ---: | ---: | | 1x128 | 128 | 32 | 212.8 | 2.24x | 95% | 173.3 GiB | | 2x64 | 64 | 64 | 122.5 | 1.29x | 70% | 177.6 GiB | | 4x32 | 32 | 128 | **94.4** | 0.99x | 78% | 190.6 GiB | | 8x16 | 16 | 256 | **95.0** | 1.00x | 58% | 218.7 GiB | | 16x8 | 8 | 512 | 127.3 | 1.34x | 66% | 278.6 GiB | | 32x4 | 4 | 1024 | 300.2 | 3.16x | 62% | 460.4 GiB | Three things worth having in the record: - **A U with its minimum at the current default.** `8x16` is within 0.2% of the best cell, so there is no free win in the geometry, and the 3.18x spread across the row is entirely invisible to a JSON that records only `ranks`. - **Thread occupancy is anti-correlated with speed.** `1x128` saturates its cores at 95% busy and is the *slowest* cell; `8x16` runs 58% busy and is the *fastest*. Busy is not productive. - **Peak RSS is linear in MPI ranks at ~0.29 GiB per process** (173.3 GiB at 32 ranks to 460.4 GiB at 1024, so +287 GiB over +992 ranks). This is what `memhwm_max` beside `memhwm` is for: the summed figure spans 173→460 GiB across the row while the per-rank max *falls* 5.55→0.48 GiB. A reader who mistook one for the other would be wrong by three orders of magnitude. ### #263, for reference Its paired ratio is ~0.8 at narrow widths and **0.96 at N=32**, and it tracks the paired CPU-seconds ratio to within +0.014/−0.025 at every rung — i.e. predominantly work removal, diluted as that work becomes a smaller share of a growing overhead. Per layout at N=32 it runs 0.95 / 0.85 / 0.80 / 0.91 / 0.97 / 0.73 for 1x128 / 2x64 / 4x32 / 8x16 / 16x8 / 32x4. **The last is not quotable and is shown only for completeness:** 32x4's three paired ratios are 0.975 / 0.727 / 0.541, and its main arm alone spans **1.93x** within the cell (300.2 / 332.1 / 579.2 s) where every other cell holds inside 6%. An earlier revision of this body printed 0.975 for it — that was rep 1 read while the cell was still filling, which is exactly the failure mode a 1024-rank cell invites. Any claim at this width needs far more than 3 reps. **Every ratio above is descriptive, not resolved.** Three paired reps floor the uncorrected two-sided sign test at p=0.25, so no 3-rep cell in this comment carries statistical significance; the ladder buys the shape. The layout comparison is additionally *unpaired* — six allocations on six node sets — so no sign test applies to it at all. #### Resolved at 10 reps, at the two rungs where the answer was in doubt | N | reps | port/main median | agree | sign p | system ratio | | ---: | ---: | ---: | ---: | ---: | ---: | | 16 | 10 | **0.691x** | **10/10** | **0.0020** | 0.633x | | 32 | 10 | 0.950x | 6/10 | 0.754 | 0.938x | Per-rep ratios, N=16: 0.671 / 0.677 / 0.680 / 0.689 / 0.689 / 0.693 / 0.715 / 0.723 / 0.755 / 0.858. N=32: 0.890 / 0.894 / 0.897 / 0.917 / 0.942 / 0.959 / 1.004 / 1.005 / 1.007 / 1.019. **The two distributions do not overlap at all** — every N=16 ratio is below every N=32 ratio. An exact two-sided Mann-Whitney (the appropriate test, because comparing two rungs is *unpaired*: separate allocations, separate node sets) gives **U = 0, p = 1.08e-5**, which is the floor for a 10-vs-10 design. So this is not "0.69 versus 0.95, probably different" — it is the strongest separation the experiment can express. **Conclusion: #263 buys a resolved ~1.45x at 16 nodes and is indistinguishable from no change at 32.** Quoting a single number for "#263's improvement" is therefore wrong; the benefit is scale-dependent and disappears at the flat world size where `propagate` itself reverses (P = 4096). Two independent ladders agree on this: the weak ladder's 32-node rung is likewise its weakest cell. **A calibration warning that applies to every 3-rep figure above, including mine.** The same comparison, same two binaries (installed `_core.so` md5 `edb00c44…` vs `7dbc7de1…`, neither rebuilt between runs — both `.so` files predate both jobs), byte-identical configuration, differing only in rep count, measured **0.857x at 3 reps and 0.691x at 10** on two different 16-node allocations. The 3-rep median landed at the very top of the 10-rep per-rep range. So a 3-rep ratio here can be ~24% away from the resolved value even with paired, order-flipped interleaving inside one allocation — far beyond the ~1.7% between-job drift seen elsewhere. Read the 3-rep tables above for shape only, never for magnitude. ## A large part of this wall was our own benchmark environment, not the engine Reported here because it changes how the walls above should be read, and because it is a trap any HPC harness for a multi-threaded engine can fall into. The harness that produced every number above exported `MALLOC_ARENA_MAX=$PARTITIONS` = 16. The engine runs **19** threads per rank (16 partition masters, the main thread, the OFI `async` progress thread, and one more Python thread), so late claimants share an arena and every `malloc`/`free` in the local-work region becomes a futex sleep. Unsetting it, one binary, 10 paired reps per rung with the arm order flipped every rep inside a single allocation: | N | unset / `=16` | reps agreeing | sign p | median wall | saving | peak RSS | | ---: | ---: | ---: | ---: | --- | ---: | ---: | | 16 | **0.7732x** | 10/10 | 0.0020 | 69.02 → 53.34 s | 15.67 s (22.7%) | +0.080 GiB/node | | 32 | **0.8304x** | 10/10 | 0.0020 | 94.52 → 76.32 s | 18.19 s (19.2%) | +0.107 GiB/node | **Do not subtract these savings from the walls in the tables above.** The `=16` arms here read **94.52 s** at N=32 where the strong-scaling table reports 88.5 s — a 6.8% gap — while a separate, independently-instrumented probe of the same configuration read 94.50 s, agreeing with these runs to **0.02%**. Two runs agreeing to two parts in ten thousand means 94.5 s is the figure for *this* provisioning and the 88.5 s cell was provisioned differently. So "88.5 − 18.19 = 70.3 s" is a configuration mix, and it looks entirely reasonable on the page. The paired ratios are safe precisely because each lives inside one allocation; only the cross-quoting of absolute seconds is not. `MALLOC_ARENA_MAX` is a **cap**, not an allocation: glibc creates an arena only when a thread finds the existing ones contended, at most one per thread, so unsetting it yields ~19 arenas rather than the `8 × ncores` = 1024 the documentation's formula suggests. That is why the memory cost is ~0.1 GiB/node against ~245 GiB of headroom, and why there is no trade to weigh here. **This is a wall win at every width, not a scaling fix, and the distinction is easy to miss.** Both walls fall substantially and the reversal gets *worse*: 1.370x → **1.431x**, because the relative saving is larger at the smaller rung. Removing a cost that is roughly rung-independent always shrinks the smaller wall proportionally more. A reader checking only that both numbers improved would conclude the opposite of what happened. Two consequences for the tables above, stated rather than quietly corrected: - **The A/B ratios stand.** Both arms of every comparison ran under the same contended allocator, and the comparisons are paired within one allocation. The #263 result (0.691x at N=16, 0.950x at N=32) is unaffected. - **The attribution *shares* are distorted**, because arena contention is charged to the barrier-wait and system-time terms. The `system share` column in particular — 17.8% → 67.3% across the ladder — is measuring the engine and this misconfiguration together, and no split between the two is offered here. One diagnostic worth recording, because it is what ruled out every topology explanation: **which** partition indices are penalised changes between runs of the identical configuration on the same nodes. A core, cache, NUMA, or IRQ-affinity story predicts a stable set. Only a lazy, racy arena assignment reproduces that. `voluntary_ctxt_switches` on an affected thread tracked the penalty exactly (557,537 vs 68) while minor faults stayed flat, so it is lock contention rather than page-fault churn. ## Reproducibility **The numbers above are not reproducible from this diff.** They were produced by a private harness (allocation management, A/B interleaving, `/usr/bin/time -v` collection, collation) that is not in this repo and is not proposed for it. This PR contains only the recording and reporting change that makes such a campaign *interpretable*. Every figure is quoted against the commit that measured it. --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: Ramón L. Panadés-Barrueta <rpana92@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Carry cross-rank queries as a width-adaptive record
pr/query-wire-v38dcc0fd, on PR 7 (#259)1f350ff. One commit of substance (249aee8) plus a merge of the updated PR 7; 19 files, +2574/−287 against that base.The layer-build query path carried one dense word per 64 modes per query, at a fixed stride. This replaces it with a per-term-minimal record and stops the two paths that never needed the wire from using it.
What changes
monoprop_DIGEST_CUTOFFknob.CompactQuerydesign assumed is gone.Sink::kStridebecomesincoming_layout()/querier_layout(), fixing a latent inconsistency:ContractSink::kStridewas the fused width whileon_response_blockgot the plain buffer and used the plain-stride default — correct only by accident. A wrong layout reads a neighbouring record's phase, which is a coefficient sign flip, not a crash.bulk_insertgains a group-prefetched hashed path, shared by the cross-rank and self inserts.They ship together because 2 and 3 share the
OperatorIndexposition store, and 1 is not measurable without them.Timings
Arms:
main48cadcbvs this branch, bothENABLE_PROFILE=OFF. Stacked, so this is PR 7 + PR 5 — PR 7's own timing was a null in both its campaigns (0 and 1 of 24, all inside 0.98–1.01x), and memory includes its 2 B/term.port/main, so every row below 1.00 is this branch faster. 10 interleaved reps per cell in one allocation, order flipped per (rep, cell); ratios divided per rep then median, never a ratio of medians.rungislower_atol(with Trotter steps where they differ) and separates cells that are different model sizes. No multiplicity correction — one branch against main is not a family — so everypis a raw two-sided sign test.Single thread (1 rank x 1 partition)
build_graph[hubbard]propagate[hubbard]build_graph[pauli]energy[pauli]gradient[pauli]propagate[pauli]Layouts A (1x128) and B (8x16)
build_graph[hubbard]propagate[hubbard]build_graph[pauli]energy[pauli]gradient[pauli]propagate[pauli]build_graph[hubbard]propagate[hubbard]build_graph[pauli]energy[pauli]gradient[pauli]propagate[pauli]build_graph[hubbard]propagate[hubbard]build_graph[pauli]energy[pauli]gradient[pauli]propagate[pauli]build_graph[hubbard]propagate[hubbard]build_graph[pauli]energy[pauli]gradient[pauli]propagate[pauli]Peak RSS (node sum)
hubbard-freshhubbard-freshpauli-freshpauli-graphhubbard-freshhubbard-freshpauli-freshpauli-graphhubbard-freshhubbard-freshpauli-freshpauli-graphhubbard-freshhubbard-freshpauli-freshpauli-graphhubbard-freshhubbard-freshpauli-freshpauli-graphWhat the grid shows, and where it does not
Nothing regresses at any production layout. Across P = 128 and P = 256, both models, all four operations, every resolved cell is a win (0.63–0.99x) and the rest are flat.
build_graphneeds more than one rank — and it is ranks, not partitions, since layouts A and B share the same flat world P. Both models are flat at 1 rank (1.005 7/10, 1.002 6/10) and win from 2 up, but the magnitude diverges: hubbard grows monotonically (1.22x / 1.42x / 1.59x at 2 / 8 / 16), pauli jumps once and saturates (0.938 / 0.949 / 0.947).propagatewins a flat 1.13–1.17x at every rank count from 1 to 16, all four cells resolved, so that share is not the exchange.energyandgradientare a null control inside the same cells. Neither touches the query wire; across the nine cells running them they land in 0.971–1.018, same binary, same allocation, same reps as the rows that moved.The scope limit: the record's fixed per-query cost needs something to amortise against, and at one partition there is nothing. Three of the four single-thread operations regress 1.21–1.54x, 10 of 10 reps each. This is not a narrow-system effect — pauli is the narrow model and it wins 0.92–0.95x once the problem is large (91.3M terms), so the axis is amortisation, not width. The one P = 1 gain,
build_graph[hubbard]at 1.11x, is the digest becoming visible where emit is ~67% of the layer instead of ~12%.The digest itself cannot be confirmed at 128 partitions, where it is ~0.8% of
build_graphwall — below its paired noise floor. Its independent evidence is retired instructions: 0.9128x aroundbuild_graph, 4 of 4 reps, −23.4 per emitted term — a different instrument on an earlier form of this branch, not this campaign.Bit-identity
Determinism holds only at fixed (ranks, partitions) — across partition counts the contract is tolerance-equality — so the geometry is part of the claim. 7 of 7 cells bit-identical at (R × S) in {1x1, 2x1, 2x8, 4x8} across build_graph, propagate and pauli, job
1825793, compared per rank, unmerged, byte for byte. Witnessed on the pre-re-cut form of these commits; it cannot be re-run on this base, because the gate needs each arm to supplybenches/_builders.pyfrom its own checkout and #227 moved those modules into a separate package.Gates and reproducibility
Re-gated on the merged tree (
c82155c, extension md56d6dea59…; the commits since carry PR 7's two review edits, a doc line and a code comment, neither reaching the binary,ENABLE_PROFILE=OFF):ctest -L unit252/252,-L serial251/251, and the Python MPI suite 592 passed at each of 1x1, 1x16, 2x8 and 8x16. On the pre-merge cut,strings -afound noLAYERPROF/COMMPROF— no profiling instrument linked — and did findmatched_scratch_bytes, confirming PR 7 is in the base.The merge of PR 7 was textually clean and would still not have compiled. It carries
main's #267, whose new test is written against the dense query wire this branch removes:detail::query_pushno longer exists, so the test now pushes throughQueryCodec<8>::push, and itsRecordingSinkgains thequerier_layout()that replacedSink::kStride—resolve_range_reads that unconditionally, and the assert on the query count reads it too, so underNDEBUGhalf of the breakage would have gone missing rather than failing loudly. #267's guard itself merged into the restructuredresolve_range_loop asfound[j] < combined_size: the same bound on the same index. Both are in the 252 above.Not reproducible from this diff. Deucalion, 2x AMD EPYC 7742 / 128 cores / SMT off / NPS4 / 242 GiB per node; a private harness that never ships; two prebuilt venvs, extension md5s
1250a27e(main) andb4e76812(this branch).