Conversation
…hanism" This reverts commit 71a21b6.
The relaxed wavefront keeps many bidegrees in flight at once, so at any instant it is likely that some job is inside a linear-algebra critical section (ParallelGuard). The scheduler re-spawned a bounced job immediately, which just re-checked is_in_parallel, found it still busy, and bounced again — spawning a whole rayon job per re-check and pegging every core on a retry storm that does no useful work. Instead the receiver checks the flag itself (a cheap atomic load) and parks a bidegree only when the section is genuinely busy. A job acquires and releases its guards many times and spends most of its time outside them, so the section frees far more often than jobs complete; parked bidegrees are therefore re-checked via a short recv_timeout while anything is parked, and re-spawned as soon as the section frees. Incoming messages are still handled the instant they arrive. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MCUtWj6P6suSZqvATCdg6d
is_in_parallel was a global count of active par_iter critical sections, so a step_resolution job bounced whenever *any* thread was in one. Under the relaxed wavefront many bidegrees are in flight, so that flag is almost always set and nearly every job bounced, producing the retry churn the parking mitigation only softened. The priority inversion the guard exists to prevent is narrower: a worker that initiated a par_iter blocks in the join and work-steals, and if it steals another (heavy, nested-parallel) resolution step, that step stalls the section the worker is blocked on. A stolen job runs on the stealer's own OS thread, so a thread-local depth counter reports exactly whether *this* worker is a blocked guard holder. Jobs picked up by a free worker read zero and run, letting independent bidegrees resolve concurrently instead of serializing behind any single critical section. The scheduler thread never holds a guard, so it can no longer read the flag to sense saturation; park bounced bidegrees and retry them on each completion or a short recv_timeout tick. Bounces are now rare (only a genuine steal-onto-a-blocked-holder), so the parking path barely engages. The classical scheduler shares the guard and benefits the same way, so its immediate-respawn no longer storms. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MCUtWj6P6suSZqvATCdg6d
Pins the invariant the previous commit relies on — a ParallelGuard held on one thread reads as absent on another — so a future change that reverts to a shared counter fails loudly instead of silently reintroducing the retry storm. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MCUtWj6P6suSZqvATCdg6d
…-graph-d69k0s' into hpc
…ead-local
The batched multiply serialized every launch behind one RESIDENT mutex held
across the whole marshal+upload+kernel+readback section, and additionally
pinned all work to CUDA stream 0. With the relaxed dependency graph exposing
~max_s-wide bidegree parallelism, that lock collapsed a ~12-core CPU wavefront
to ~2.6 busy cores and left the GPU idle 80% of the time — making NASSAU_GPU=1
a net 1.4x slowdown over CPU-only at stem 130 (193s vs 142s).
cubecl 0.10 does not need the lock: a per-device runner thread already
serializes all server access (concurrent client calls are memory-safe), and
memory pools are per-stream. So:
- RESIDENT becomes a thread_local RefCell: each rayon worker keeps its own
admissible cache and cs/mk device handles, created and consumed only on the
thread (and thus the default per-thread CUDA stream) that owns them, so no
handle ever crosses threads and no cross-stream event sync fires.
- The GPU_STREAM{value:0}.executes pin is removed; each worker launches on its
own default stream, so independent bidegrees marshal and execute
concurrently. memory_cleanup now trims only the calling worker's pool.
Stem 130 (S_2, s<=152, 16-core H200 box): 193s/2.6 cores (old mutex GPU) and
142s/10 cores (CPU-only) -> 44-49s/5.6 cores. Verified bit-identical to the
CPU path with NASSAU_GPU_VERIFY=1 at stem 80 (MIN_WORK=0, every build) and
stem 130 (default gate, all offloaded/chunked launches, concurrent workers).
Note: concurrency raises peak host memory (concurrent marshal buffers across
workers); a 16-worker VERIFY run at stem 130 exceeded a ~48GB cgroup, while
normal runs fit comfortably. Bound RAYON_NUM_THREADS if memory-constrained.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… resident master)
The mutex-removal commit let many workers run device sections concurrently, which
exposed three unbounded memory consumers at record stems (>100GB host AND device
by stem 150, measured):
1. Unbounded launch transients: the all-rows reuse build allocated its full output
in one shot, per in-flight worker. Fixed by splitting builds into row blocks
bounded by NASSAU_GPU_BLOCK_MB (default 512MB) of output AND GPU_PAIR_CHUNK
kernel threads — one launch per block, subsuming the former pair-chunk loop
(rows are independent, so blocks concatenate exactly).
2. Unbounded stream count: every worker thread got its own CUDA stream, and each
stream's pool retains freed slabs indefinitely. Fixed by NASSAU_GPU_CONCURRENCY
(default 8) permits that double as stream slots: at most 8 device sections run
at once, on 8 fixed streams. A permit must never be held across a rayon parallel
section (par_iter chunks execute on guard-free threads that can steal a bidegree
job which then parks on acquire — observed deadlock); it is acquired only for
the strictly sequential layout+device section. Do NOT raise to 16: measured
catastrophic (>30x) slowdown from cross-stream sync churn.
3. Per-thread resident duplication: the thread-local resident store copied the
admissible-matrix master (~8.5GB at stem 150, growing with degree) once per
worker, on host and device. Fixed by re-sharing it: host master behind an
RwLock (enumeration outside the write lock), one device mirror behind a small
mutex, handles shared across threads/slots (cubecl event-syncs cross-stream
reuse). Re-uploads are needs-based — only when a launch dereferences past the
uploaded prefix — since re-uploading on mere growth serialized multi-GB copies
on nearly every frontier launch (measured 1.5x wall regression).
Stem 150 (S_2, s<=152, 16-core H200 box), verified bit-identical to CPU at
stem 80 (every build, forced multi-block) and stem 130 (all offloaded launches):
wall cores host RSS device
before this commit 682s 3.3 137 GB 140 GB (full card)
after (32 workers) 721s 3.8 65 GB 37 GB
CPU-only reference 771s 10.3 4.4 GB —
Verdict: at record stems the GPU path now merely ties CPU-only while using far
more memory — the CPU path (no row-reuse matrix, per-signature builds) is both
frugal and wavefront-parallel. Recommend CPU-only for the stem-300 production
run; the GPU path remains correct, memory-bounded, and a real win at mid stems
(3x at stem 130).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replaces the count-based launch cap (NASSAU_GPU_CONCURRENCY=8 exclusive sections) with two decoupled controls: - NASSAU_GPU_MEM_BUDGET_MB (default 4096): admission weighted by a launch's output bytes, so dozens of small low-stem launches run concurrently again (the count cap throttled exactly the region that never had a memory problem) while the frontier stays bounded to ~budget/block-size in flight. - NASSAU_GPU_STREAMS (default 8): fixed CUDA stream slots, round-robin and SHARED (small launches serialize on a stream rather than demanding an exclusive one), so stream/pool count is bounded independently of concurrency. Master device uploads are now prefix-only with doubling: a launch ships max(need, 2*uploaded) entries, not the whole master, so frontier launches (which append new high-degree R each t) no longer re-ship gigabytes of untouched tail. Stem 130 improved 187s -> 150s; stem 150 memory 65/37 -> 68/30 GB, verified bit-identical (stem 80 all-builds, stem 130 all offloaded). But a slots x budget sweep is FLAT (8/4G=150s, 16/8G=170s, 32/16G=160s, 64/32G=201s): concurrency knobs are not the ceiling. The ceiling is Amdahl — the GPU accelerates only the Milnor multiply (~17% of frontier wall time; row_reduce/signature_matrix/readback dominate and are CPU/serial through cubecl's single runner thread), so the end-to-end GPU:CPU ratio is flat ~1.13x across the 130-150 heavy bands, not widening. Widening it would require offloading row_reduce (PR SpectralSequences#274's RREF). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…tiply path
The zero-signature image matrix (d_s applied to the zero-sig source basis,
column-masked to the zero-sig target) was the last per-bidegree Milnor multiply
still on the CPU — a serial per-row apply_to_basis_element_restricted, ~17% of
frontier wall time in the perf profile. But it is the *same* restricted multiply
as the QI-source `full_matrix` already built via restricted_partial_matrix_maybe_gpu,
just on d_s = differentials[b.s()] instead of d_{s-1}, and its target
mask/dimension are exactly the `target_mask`/`target_dim` already computed for the
bidegree (d_s and d_{s-1} share the target module modules[b.s()-1]). So route it
through the same GPU-offloaded, work-gated, already-verified path and apply the
column mask on CPU; drop the serial `signature_matrix` method. (Reinstates the
"signature_matrix offload" win from the original nassau_gpu branch, lost in the
SpectralSequences#272 relaxed-graph merge.) row_reduce stays on CPU — the signature-masked matrices
are very flat (~100 x 100000), a poor RREF target for the GPU.
Correctness: GPU Ext chart byte-identical to CPU-only through (100,152);
NASSAU_GPU_VERIFY passes at stem 130.
This shrinks the serial tail that Amdahl-capped the GPU:CPU ratio, so the
arithmetic-intensity advantage finally shows through and the gap WIDENS with stem
(S_2, s<=152, 16-core H200 box, w=32):
band GPU CPU ratio
130->140 159s 206s 1.30x
140->150 278s 423s 1.52x
cum 0->150 596s 771s 1.29x (was 723s, a near-tie)
Memory stays bounded by the same byte-budget/block machinery (this path reuses
multiply_batch_on_gpu). Next lever: the full-reuse-matrix readback.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… a host zero buffer Per-thread stack sampling at stem 145 showed the wavefront's serial stalls were a rayon worker pegged in __memcpy_ssse3 inside create_from_slice — host-side upload marshaling, NOT readback (cubecl 0.10 already does async D2H off pinned memory with the event wait on the worker thread, so the runner is free during the copy). The dominant offender: the batched multiply allocated + zeroed a host `vec![0u32; out_len]` (hundreds of MB at the frontier) and memcpy'd it up as the kernel's XOR accumulator, every launch/block. Allocate out_h uninitialized (client.empty) and zero it with a trivial on-device kernel (zero_u32), same stream as the multiply so it is ordered before it. Removes the host memset, the non-pinned host->device copy, and the transfer itself; on-device zeroing is memory-bound (microseconds on an H200). Verified: GPU Ext chart byte-identical to CPU through (100,152); NASSAU_GPU_VERIFY passes at stem 130. Bands (S_2, s<=152, 16-core H200 box, w=32), vs the prior signature-offload binary: 0->130 159 -> 141s (ties CPU-only 142; was a 0.89x loss) 0->140 318 -> 245s 130->140 marginal 104s vs CPU 206s = 1.98x (was 1.30x) peak RSS 46GB, GPU 28GB (both down). Next serial upload to check: term_pparts / the per-product record arrays. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The shared admissible master reaches ~3GB by stem 138 (cs 1.2GB + mk 1.9GB), and every launch took the RESIDENT_DEV mutex to read its device handles — with a growth-triggering launch doing a multi-GB create_from_slice re-upload *while holding that mutex*. Per-thread stack sampling showed the frontier collapsing to a single thread memcpy-ing gigabytes while every other bidegree blocked on the lock (upload byte-size instrumentation under NASSAU_GPU_DEBUG confirmed the master, not term data or the seqno table, as the giant upload). Make handle reads lock-free (RESIDENT_DEV: Mutex -> RwLock) and move the upload memcpy outside that lock, serialized only among uploaders by a separate RESIDENT_UPLOAD mutex with a re-check that coalesces a burst of growth-needing launches into one upload. A launch whose R's are already resident proceeds without ever blocking on someone else's upload. Verified GPU chart byte-identical to CPU through (100,152); NASSAU_GPU_VERIFY passes at stem 130. Removes the lock-held-across-copy stall, but occupancy only rose ~3.5->4.5 cores: the dominant limiter is upstream (thin GPU bidegrees + wavefront width), not this lock. Kept because it is correct and matters more at stem 300 where the master is larger. Also adds a per-buffer upload-size line to NASSAU_GPU_DEBUG. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…er-product alloc storm A frontier launch has ~1e5-1e6 products, and the marshal built term data as `Vec<(Vec<u16>, Vec<u32>)>` — two heap allocations per product (~1e6 tiny allocs per launch) — then extend-copied them into the flat upload buffers. Per-thread profiling of the GPU path showed this as a dominant chunk of the per-bidegree CPU "envelope" (~16% _int_malloc/_int_free plus the marshal copy) that wraps each (fast) kernel and, because the wavefront is only ~10-15 bidegrees wide, cannot be hidden — so the GPU sits idle between brief spikes. Precompute the term-count prefix sum (`term_off`), size the flat `term_pparts`/ `term_lens` once, and parallel-fill each product's disjoint slice in place (unsafe but sound: prefix-sum ranges never alias). The later layout loop just reads `term_off[pi]` for `prod_term_start` — no per-product allocation, no concat copy. Verified GPU chart byte-identical to CPU through (100,152). Same GPU results, far less allocation and marshal work per launch. (The remaining per-product `GpuProduct.term_indices: Vec<usize>` built in extract is the next alloc to flatten.) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…er-splitting The row-block splitter caps a launch at GPU_PAIR_CHUNK thread-pairs (the kernel indexes threads by u32 ABSOLUTE_POS, ceiling 2^32). It was set to 1<<30 (~1.07e9), ~4x below the real ceiling — so every billion-pair giant was chopped into ~4 launches, each a separate upload + kernel + BLOCKING readback round-trip, even though its output is only ~350 MB (well under gpu_block_bytes). Debug confirmed the giants pegged at 1.07e9 pairs; this, not the byte budget, was the binding split, which is why a NASSAU_GPU_BLOCK_MB sweep was flat. Raise it to 3.9e9 (leaves ~0.39e9 headroom under 2^32; the splitter always takes >=1 row and a lone row past 2^32 still trips the per-block u32::try_from assert; grid stays ~1.5e7 cubes, far under 2^31). Giants now run as a single launch (max total_pairs observed 3.90e9), collapsing 4 round-trips to 1. GPU 0->140 (w=100): ~245-288s -> 216s. Chart byte-identical to CPU through (100,152). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…dices The batched multiply re-gathered and re-uploaded every term's zero-padded p-part (`term_pparts`, width*2 bytes/term) on every launch — the dominant per-launch H2D transfer, plus a large parallel host gather. Make the basis itself resident on the device instead: build it once (grown incrementally as higher degrees appear, mirroring the admissible master) and upload only `term_gei[slot]`, the term's global basis-element index `global_base[s_degree] + ti` (4 bytes/term). The kernel reads the p-part from `basis_pparts[gei*width..]` with length `basis_lens[gei]`. At stem 140 the per-launch term transfer drops from ~width/2x larger to term_gei=95 MB, the basis is a one-time few-MB upload, and the per-term p-part gather is gone. Kernel change is minimal: params `term_pparts, term_lens` -> `basis_pparts, basis_lens, term_gei` (net +1 array arg), launch-arg order preserved 1:1 with the signature. `multiply_pair` is unchanged. An A/B toggle (`NASSAU_GPU_BASIS_PASSTHROUGH=1`) binds the per-launch term buffers as the "basis" with an identity index map, reproducing the old behaviour through the new kernel — so a single binary can isolate a kernel-signature bug from a resident host/upload bug. Both paths verified GPU==CPU per launch at stem 80 (MIN_WORK=0), and the resident path chart-matches CPU at (100,152). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
`NASSAU_GPU=1` could not use multiple CUDA streams: the shared resident admissible master + Milnor basis were re-`create_from_slice`d into a NEW device handle on every growth, and that handle churn broke cubecl's per-handle cross-stream synchronization, so a launch on one stream could read the master while another was mid-upload -> wrong multiply (`dx != 0`), "Memory page" panic, or hang. Fix: make the resident buffers STABLE and grow them IN PLACE — the read-only shared global (model-weights) pattern cubecl supports across streams. A small `copy_into_*` kernel writes the new tail (uploaded to scratch via `create_from_slice`) at the buffer's append offset; the handle changes only on a rare capacity doubling, which is barrier-protected (`RESIDENT_REALLOC`: device sections hold the read lock across the multiply, a realloc takes the write lock and quiesces them). Each worker gets a stable per-thread stream id (`thread_stream_id`); default `NASSAU_GPU_STREAMS = 8`. Key gotcha (a stem-150 `dx != 0`): cubecl's `ArrayArg` length is u32, so a buffer of exactly 2^32 elements truncates to length 0 and the copy writes nothing (buffer reads all zeros). The doubling `cap` jumped 2^31 -> 2^32 right at stem 150's `masks` size. Capacity is clamped to `RESIDENT_MAX_CAP = 2^32 - 1`; a single resident buffer cannot exceed that (a larger master needs splitting — the old create_from_slice path had the same limit). Copies are chunked under the u32 ABSOLUTE_POS thread limit. Validated: VERIFY (GPU==CPU per launch) at 8 streams, chart-match to CPU, 0 dx-crashes over many stem-150 reps at 1 and 8 streams. Perf note: multi-stream is correct but ~neutral vs single-stream at stem 140/150 (the per-device runner serializes kernel submission); it may help at higher stems with a wider heavy-bidegree wavefront. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
Instrument Matrix::row_reduce to emit a `fp::rr` tracing event for every p=2 reduction with min(rows,cols) >= 1024, recording rows/cols/min and whether the device RREF was taken (path="gpu") or it fell back to CPU M4RI (path="cpu"). The event inherits the active nassau span so each line carries its bidegree/signature context. Confirms on a stem-150 run that every reduction with min >= 8192 (up to 25091x30275, incl. the heavy zero-signature base solves) dispatches to the GPU with no fallbacks; everything below the 8192 FP_CUDA_RR_THRESHOLD stays on CPU. tracing is added as a gpu-gated optional dep. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
The intermittent stem-150 hang (flat sm=100%, ~1/6 of runs, always on the zero-signature base solve) was a deadlock in the fp-cuda cooperative row-reduce kernel `panel_factor_coop`. Its grid-wide spin barrier (launched via `cuLaunchCooperativeKernel`) requires ALL its CTAs co-resident, but the algebra Milnor multiply runs on a *separate* CUDA runtime (cubecl) whose kernels concurrently occupy SMs. When a cooperative row-reduce launched while cubecl multiply kernels were resident, its CTAs could not all co-reside; the missing ones never reached the barrier and the resident ones spun forever. Fix: a cross-runtime `fp::GPU_EXCLUSIVE` RwLock. The cooperative row-reduce takes the write lock (drains in-flight cubecl multiplies, then runs with the GPU to itself) around its launch+download; every cubecl multiply takes the read lock across its whole device section (launch through readback, so releasing means the kernel has actually completed). Readers run concurrently; a pending row-reduce briefly excludes them. No lock cycle (cubecl never takes the fp-cuda ctx lock) and no same-thread read->write (a solve's multiply releases before its row-reduce). Also adds a `gpu_row_reduce` tracing span around the GPU reduce -- the diagnostic that localized the wedge (an unclosed span names a stuck reduce, distinguishing an RREF hang from a multiply hang). Validated on H200: stem-150 x16 with 0 wedges (baseline ~1/6), wall time unchanged (261-359s), GPU-vs-CPU chart match preserved. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
The device RREF launched three cooperative kernels — panel_factor_coop, promote_coop, block_reduce_coop — synchronized by a hand-rolled grid-wide spin barrier. cuLaunchCooperativeKernel requires all the grid's CTAs to be co-resident, which only holds when this process owns the whole GPU. When another CUDA runtime shares the device (cubecl's Milnor multiply in the nassau GPU resolution), its kernels occupy SMs, the reduce's CTAs can't all co-reside, the missing ones never reach the barrier, and the resident ones spin forever — the intermittent stem-150 wedge (flat sm=100%). Add an FP_CUDA_RR_COOP switch (rr_coop()). Default off: the forward pass runs the single-CTA panel_factor one limb at a time, promotion uses the grid- strided promote_pivots, and back-substitution uses the single-CTA block_reduce_rref — none launched cooperatively, so the reduce composes with concurrent GPU work. Set FP_CUDA_RR_COOP=1 to opt into the cooperative path on a dedicated GPU (measured 2-3x faster at Nassau strides, up to ~10x on large dense half-rank matrices). Both paths validated bit-exact vs CPU row_reduce (row_reduce_demo) and vs the CPU BLAS3 oracle at 2^16/2^17 (reduce_pow2_half). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
The composable default from the previous commit fell back to single-CTA kernels
(panel_factor, block_reduce_rref) that use one SM of ~132 — 8-10x slower than the
cooperative path on large dense matrices, since the cooperative kernels' real win
is spreading each sequential bit-step across the whole grid.
Recover that all-SM parallelism WITHOUT a cooperative launch by replacing the
in-grid grid_sync with kernel-boundary (stream-ordered) synchronization, the way
cuSOLVER/cuBLAS build grid-wide multi-step algorithms:
- panel_factor: pf_find -> pf_swap -> pf_xor per bit-step (panel_factor_streamed)
- block_reduce: br_cond -> br_xor per pivot (block_reduce_elem streamed arm)
All per-step state stays on the device (pivot count, find-first result, pivword,
clear-conditions), so the host issues every launch without a readback and the
latency hides behind GPU work; only the final (pr, pivcols) is copied back. No
cuLaunchCooperativeKernel anywhere on the default path, so it can't deadlock
against a concurrent cubecl kernel. The streamed back-substitution also adopts the
cooperative path's wide-block TRSM (bp=1024 + X.U GEMM) since the GEMM composes.
Perf (H200 half-rank square, device-only) vs cooperative:
2^13 s128 0.080 vs 0.066 (1.21x) 2^15 s512 0.642 vs 0.577 (1.11x)
2^14 s256 0.195 vs 0.166 (1.17x) 2^16 s1024 2.27 vs 1.17 (1.95x)
2^17 s2048 8.63 vs 4.52 (1.91x)
Within ~1.1-1.2x of cooperative through stride 512 (the Nassau regime; the wedge
matrix was stride 215) and 3-6x faster than the single-CTA fallback at large
sizes. The residual past stride 1024 is where cooperative additionally fuses
promote/block-reduce; closable with CUDA graphs if those sizes ever matter.
Both paths validated bit-exact vs CPU row_reduce (row_reduce_demo) and the CPU
BLAS3 oracle at 2^16 (reduce_pow2_half).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
Merge pf_find and pf_swap into pf_find_swap using a threadfence "last-CTA finalize" (a grid-wide reduction, not a barrier, so no co-residency needed), cutting the streamed forward pass from 3 to 2 launches per bit-step and removing the serial 1-thread swap kernel. Cap the streamed grid at FP_CUDA_PF_CTAS (128) like the cooperative kernel. Add FP_CUDA_RR_TIMING to split forward/back timing. Diagnostic result: the streamed forward pass is work-bound (~cols^2), not launch-bound — merging, capping, and grid size leave it unchanged — so it is the per-step relaunch efficiency vs the persistent cooperative grid, ~2.7x at 2^17 but shrinking with size (3.0x at 2^16). Still bit-exact vs CPU row_reduce and the BLAS3 oracle at 2^16. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
pf_step does column cc-1's clear and column cc's find+swap in one launch, since the forward sweep alternates xor(col j) / find(col j+1) over the same below-row range — so each thread reads its row once and sees its own XOR before scanning. Halves the forward pass's launches (one per column instead of two). Marginal on its own (~5%), confirming the forward-pass gap vs cooperative is the per-step GPU relaunch cost of non-persistent kernels, not launch count — it resists launch reduction and only closes with scale (the O(cols) term vanishing against O(cols^2) work): ~2x total at 2^17, ~1.4x at 2^18. Bit-exact vs the CPU BLAS3 oracle at 2^16. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
The cross-runtime fp::GPU_EXCLUSIVE RwLock existed only to keep the cooperative fp-cuda row-reduce (cuLaunchCooperativeKernel grid barrier) from co-scheduling against a concurrent cubecl Milnor multiply, which could prevent CTA co-residency and deadlock the grid barrier (the stem-150 wedge). The default row-reduce is now the streamed kernel-boundary path (no cooperative launch anywhere), so it composes with concurrent GPU work by construction — the lock is unnecessary. Remove the RwLock, its fp re-export, and the read-side guard in the algebra Milnor-multiply device section. The gpu_row_reduce tracing span is kept (a useful wedge/hang diagnostic). FP_CUDA_RR_COOP=1 still selects the cooperative kernels for a dedicated GPU; do not combine that with the concurrent nassau multiply. Validated: stem-150 resolves in 242s with 30042 GPU row-reduce events and no wedge (full hunt in progress). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
The batched Milnor multiply silently corrupted results once the shared `masks` admissible-master crossed 2^32 u16 elements (~8.6 GB, around S_2 stem ~160-180): the buffer length and the per-R offsets were u32, so cubecl's default U32 `address_type` truncated them (`len as u32`), giving out-of-range reads and `dx != 0` (d^2 != 0) panics at e.g. (180,92). (160,82) stayed just under the ceiling and was clean. Fix uses cubecl 0.10's first-class 64-bit addressing rather than hand-rolled buffer splitting: - `multiply_batch_kernel` -> `#[cube(launch_unchecked, address_type = "u64")]`. Static u64 (not "dynamic"): dynamic picks u32 `usize` for small blocks and then narrows the u64 offset arrays on read (`usize::cast_from(u64)` under a u32 address type), corrupting results. `launch_unchecked` because checked mode emits `min(u64, u64)`, which NVRTC rejects as an ambiguous overload; every access is in-bounds by construction (uploaded `need_*` prefix + per-column `j` guards). - `RInfo.cs_off/mk_off` u32 -> u64; `r_cs_offset`/`r_mk_offset` bound as `Array<u64>`. - Resident grow copies (`copy_into_u16`/`_u32`) -> `#[cube(launch_unchecked, address_type = "dynamic")]` (scalar usize offsets adapt without narrowing; dynamic keeps small copies on u32). - Drop `RESIDENT_MAX_CAP` and its clamps (the u32 ceiling is gone); assert `out_len <= u32::MAX` loudly (the row-block splitter guarantees it). Validated on H200: (160,82) rc=0 clean, 517s vs 486s baseline (+6.4% for the static-u64 multiply, partly offset by unchecked dropping the bounds clamp); (180,92) ran 1500s with zero dx panics (previously a deterministic panic ~130-180s), i.e. correct well past the 2^32 masks boundary. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
…he resident host master `resolve_through_stem S_2 "" 180 92` was OOM-killed at the 500 GB cgroup limit (~280 GB anon + ~240 GB shmem). Measurement (env `NASSAU_MEM_REPORT`, plus a pure-CPU control at ~5-7 GB) showed the resolution's own data — differentials and module tables — is only ~1.5 GB; the ~500 GB was entirely GPU-runtime host memory. Two fixes cut it to ~90 GB at stem 180 (dx-clean): 1. Default `NASSAU_GPU_STREAMS` 8 -> 1. cubecl gives each CUDA stream its own page-locked (pinned) host pool that `memory_cleanup` never trims; ≥2 streams ballooned pinned host memory to 140-240 GB, single-stream holds it at ~4-6 GB. The payoff of multi-stream is ~nil here: cubecl's server is single-threaded, so extra streams buy no CPU concurrency, only GPU kernel overlap the big saturating multiplies barely use. Override for a dedicated large-RAM node. 2. Stop retaining the resident admissible master host-side. `RESIDENT_HOST` kept the full `col_sums`/`masks` (~54 GB at stem 180) forever, duplicating the device copy, even though after upload it is never read again (offsets come from `index`; growth uploads only the new tail; a capacity realloc copies the old *device* buffer). Now it keeps only the not-yet-uploaded tail (`*_pending`, ~sub-GB) plus a logical length, freeing each `R`'s data the moment it reaches the GPU — invariant `dev.uploaded == len - pending.len()`. Also: bound the pinned staging on resident growth to `STAGE_CHUNK` chunks, and a gated `NASSAU_MEM_REPORT` (differentials/modules/resident heap breakdown) for ongoing memory work. Validated on H200: stem 180 peak RSS ~90 GB (was 500 GB OOM), dx-clean through the >2^32 masks region and the heavy solves. Remaining growth for higher stems is real GPU working memory (concurrent dense output matrices) + GPU device memory, not retained host duplicates. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
…unded Even with the resident host-master freed, host RSS still grew ~linearly toward the cgroup limit at high stems — not a retained duplicate but the GPU path materializing whole dense output matrices. The CPU walks signatures one at a time (small working set); the GPU offload built the entire matrix at once and held its dense readback (`num_rows × num_limbs` u32, ~12 GB regions at stem 180) alongside the assembled matrix. Two caps bound it: 1. `reuse_full_matrix` now only builds the all-rows matrix when `rows × cols <= NASSAU_GPU_REUSE_MAX_WORK` (default 1e10). Above that it falls back to per-signature builds (each a bounded row subset, like the CPU), so the peak scales with the largest single signature, not the whole bidegree. 2. `get_partial_matrix_restricted` processes rows in batches sized so the dense readback stays under `NASSAU_GPU_MAX_READBACK_MB` (default 1024). Products are built in row order, so each batch is a contiguous slice with its `row` remapped batch-local; results XOR back into the global rows. The readback is freed between batches. Both are correctness-neutral (rows are independent): verified bit-for-bit vs the CPU under `NASSAU_GPU_VERIFY` with a 1 MB cap (many batches/build), 0 mismatches. Effect at stem 180 (streams=1 + resident de-dup + this): host anon goes from growing past 85 GB to a bounded ~18-24 GB, with no throughput loss — cross- bidegree rayon concurrency keeps the GPU fed, and the default cap only splits the few giant high-t builds into a handful of chunks. Peak RSS ~30 GB (was 500 GB OOM). This bounds the host working set so it scales toward higher stems; a D-deep async launch/collect pipeline (cubecl launches are async; only read_one blocks) can hide chunk latency if a much smaller cap is ever needed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
The resident admissible master (col_sums/masks) grows with degree and is the stem-300 device-memory wall: ~34 GB at stem 180, extrapolating past the H200's 143 GB before stem 300. This adds an opt-in eviction that bounds it. Policy (from the NASSAU_R_STATS probe): a *degree* threshold, not LRU. Low-degree R's are the stable hot core (reference span 0.99 of the run); high-degree R's are scattered-recurring (0.81) and also the biggest matrices, so a degree cap evicts the most bytes for the fewest references and is stem-independent (the kept set saturates). Env NASSAU_GPU_RESIDENT_MAX_DEGREE (default i32::MAX = keep all). Design — no kernel change. Every output row's products share one operation R (extract_restricted), so hot (deg<=cap) and cold rows are DISJOINT. multiply_batch_on_gpu partitions products, compacts each group to its own dense row range (so total readback stays num_rows, not 2x), runs the resident pass on hot and a Transient pass on cold, and scatters results back. Transient builds a per-block master with create_from_slice, freed with the launch, so the device copy never persists. Cold admissible data is cached host-side (COLD_HOST) so the expensive recompute is one-shot, like the resident path. Fast path (cap==MAX or all-hot) is byte-identical to before — zero regression by default. GpuProduct gains Clone for the partition. Validated bit-exact: NASSAU_GPU_VERIFY full S_2 stem-110 resolution at theta=10 (transient path heavily stressed) and theta=100: mismatches=0 dx=0 panics=0. Memory vs throughput (stem 180, ExclusivePages, streams=1): control (no eviction) 51 GB / 1349 s; theta=125 11 GB; theta=100 12 GB (master 34 -> ~1 GB), dx=0. But eviction costs 2-4x throughput even at a high theta (stem 150: control 282 s, theta=140 613 s) because the evicted high-degree matrices are the biggest and are re-uploaded every launch. So this is a fit-in-memory lever for stems where the master won't fit at all (a slow completion beats an OOM), tuned to the highest theta that fits; it is NOT a speedup. Next optimization to cut the re-upload: a bounded LRU device-side cold cache (upload once, reuse across a bidegree's launches). Also retained: the R-access probe (NASSAU_R_STATS/dump_r_stats) and the device/host [MEM] report (NASSAU_MEM_REPORT) used to characterize this — both env-gated no-ops. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
Foundational step toward generating admissible matrices (col_sums/masks) ON the GPU into transient scratch instead of storing/uploading the resident master -- the direction that eliminates both the stem-300 device-memory wall and the eviction re-upload cost (a given launch would enumerate its cold R's on-device, never uploading them). enumerate_admissible_ref reimplements AdmissibleMatrix::next using ONLY flag-guarded control flow -- no break, continue, or early return -- because that is the subset the cubecl DSL compiles cleanly (cf. multiply_pair, which tracks a `rejected` flag rather than breaking). `found` replaces the odometer's `return true`, `handled` replaces its `continue 'mid`. This validates the tricky restructuring on the CPU, where it is fast to debug, before the hard-to-debug #[cube] port -- which then becomes a mechanical transcription onto per-thread local Arrays (state is tiny: rows = |p_part|, cols <= 32). Test admissible_enum_ref_matches asserts bit-exact equivalence with admissible_matrices over every real R up to degree 60 (4155 R's, all match). Not yet ported to cubecl / not wired into the multiply -- next step. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
Transcribe enumerate_admissible_ref into a cubecl #[cube] kernel that generates each R's col_sums/masks for every admissible matrix directly into device scratch -- the on-GPU replacement for the resident/uploaded master. One thread per distinct R, all state in fixed-size per-thread local Arrays; flag-based control flow (while ... && !found / handled) since the DSL has no break/continue/return. Backend-agnostic host driver (generic over Runtime) so the identical kernel can run on CUDA and the cpu backend. New test admissible_enum_gpu_matches runs it on the H200 and asserts bit-exact output (values + per-R counts) vs the CPU reference: 1055 R's, 30385 matrices, all matching. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
Wire enumerate_admissible_kernel into the Transient (evicted) cold path of multiply_batch_block. A cold R's col_sums/masks are now GENERATED on the GPU into transient scratch (one enumeration launch, ordered before the multiply on the same stream) rather than built on the host and uploaded via create_from_slice. Only the small p-parts + per-R dimensions upload; the scratch is freed with the launch. This kills the per-launch H2D master re-upload that made eviction a 2-4x slowdown (bench 2026-07-27), the whole point of the in-kernel-enumeration direction. Also replace the COLD_HOST full-array cache with COLD_COUNT, a 12-byte-per-R (cs_len, mk_len, num_mats) shape cache. The evicted tail of the master now lives neither on the device nor the host -- only its sizes, needed up front to lay out the scratch offsets and the pair-count prefix sum. num_mats is counted once per distinct R (admissible_matrices, arrays dropped) and memoized. De-gate enumerate_admissible_kernel + ENUM_* caps + the MAX_XI_TAU import for production. Validated bit-exact: the isolation test (1055 R's / 30385 matrices) still passes, and NASSAU_GPU_VERIFY full S_2 stem-110 resolutions at theta=10 (nearly all R's cold -> enumeration path stressed) and theta=100 both report mismatches=0 dx=0 panics=0 rc=0. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
…refuted) Investigation of the eviction crash + a throughput bench that together show in-kernel enumeration cannot beat upload-based eviction: - enumerate_admissible_kernel -> u64 addressing (was default u32); did NOT fix the big-block CUDA_ERROR_LAUNCH_FAILED, so the fault is elsewhere in the Transient wiring, not the kernel. - admissible_enum_gpu_matches extended to degree 145, chunked per-degree: proves the kernel bit-exact vs the CPU reference (144903 R's, 185M matrices) across the full range the eviction path exercises -- so the kernel is correct. - bench_admissible_cpu_vs_gpu (ignored): CPU admissible_matrices 1.61s vs GPU kernel-only 2.02s (0.8x, SLOWER) for degrees 1-130. GPU enumeration is ~3x slower than just transferring the same arrays (0.68s readback) it replaces: the odometer is sequential per R with matrices-per-R spanning 1..millions, so the launch bottlenecks on its few longest threads at GPU scalar speed. The "trade GPU integer work for PCIe bandwidth" premise is refuted -- enumeration worsens the eviction re-upload cost instead of curing it. The enum wiring in multiply_batch_block's Transient path still faults at scale and should be reverted to upload-based eviction; kept here as validated, dormant code documenting the dead end. Default (theta=inf) path is unaffected. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
…ayon pool
Every profile in this file was taken either at theta=125 (a testbed handicap: 2.85x
slower than uncapped, and it inverts the bottleneck) or before the readback / `Arc`
term-list / `R`-memo fixes. This records one taken at theta uncapped, stem 150, after
those — i.e. the configuration we would actually ship, the 130 s run.
multiply_batch_grouped (+closure) 17.4% allocator 9.7%
memcpy + memset 13.8% add_masked 7.1%
Matrix::row_reduce 11.3% crossbeam_epoch (rayon) 5.8%
get_partial_matrix_restricted 9.3% resident_info 2.6%
The binary is 64.5% of cycles and `libcuda` does not reach the top 16. At correct theta
this is a CPU-bound workload that barely touches the GPU — the 1.56% occupancy and
52.67% spin-wait numbers describe the handicapped regime only. `resident_info` fell
12.93% -> 2.56%, which is the memo working as intended.
RULED OUT: shrinking the rayon pool. `crossbeam_epoch` at 5.8% is rayon's work-stealing
reclamation (confirmed by call graph — rayon_core worker threads, not our channel), and
the pool is 128 threads for a run using ~9 cores' worth of work, so idle stealers
churning epoch state looked like free money. Interleaved, 3 rounds:
RAYON_NUM_THREADS default(128) 137, 125, 130 mean 130.7 s
RAYON_NUM_THREADS 32 144, 144, 139 mean 142.3 s
RAYON_NUM_THREADS 16 140, 159, 129 mean 142.7 s
9% WORSE. The overhead rides on idle workers and costs no wall time, while the wide pool
is buying parallelism elsewhere. Same lesson as R-affine sharding and the block budget:
reducing work that is not on the critical path buys nothing here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
…ts on
`NASSAU_SPLIT_VERIFY` builds the reuse matrix over all rows and again over two row
ranges, and asserts they agree row by row.
This is the load-bearing claim for precomputing matrix rows AHEAD of a bidegree, which
is the remaining lever for capped-theta high stems: the rows coming from generators
that already exist can be computed early and concatenated with the rest. The argument
is that rows are generator-major and `FreeModule`'s basis tables are append-only
(`OnceVec`), so widening a restriction only appends and an early row block stays valid
verbatim — and a product `Sq(R)·gen` lands inside `gen`'s own block, so an old
generator can never write into a newly added column.
That is an argument from data-structure semantics. This checks it against real data:
S_2 stem 60 / max_s 40, theta uncapped 0 mismatches
S_2 stem 60 / max_s 40, theta=60 0 mismatches (transient path exercised)
So the decomposition holds in both regimes, including the capped one the design is for.
WHY THIS MATTERS, recorded because it is the one lever left. At capped theta the enum
kernel runs at 1.56% occupancy in launches of 3-104 blocks, and the reason is SUPPLY:
demand-driven work exposes only ~5-10k independent `R`s because the wavefront is ~5
bidegrees wide. Precomputing blocks ahead in `t` makes that a tuning knob instead of a
ceiling — the `R` set in flight becomes "however far ahead we choose to run" — which is
what turns 3-block launches into device-filling ones. Blocks are also independent, so
the same work can spill to the ~119 idle cores via the CPU path, which needs no master
at all.
What remains unbuilt is the scheduling layer: a cache keyed by bidegree, a background
filler that computes blocks for future `t` from already-known generators, and assembly
of (cached prefix ++ fresh suffix) in `step_resolution`. The primitive it needs already
exists — `restricted_partial_matrix_maybe_gpu` takes an arbitrary row list — and is now
verified.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
Builds a bidegree's full restricted differential matrix on background threads
BEFORE the bidegree runs, and hands it over when it starts.
The soundness argument was already in the code and is now used. `step_resolution_with_subalgebra`
deliberately ignores the target's degree-`t` generators, so its full matrix reads only data frozen
once `(s-1, t-1)` is committed -- strictly weaker than the condition for RUNNING `(s, t)`, which
also needs `(s, t-1)`. Everything between the two bounds is a bidegree whose matrix is fully
determined but which cannot run yet: the speculation window. `restricted_dims` makes the point
concrete -- the restriction bounds are `b.t()` and `b.t()-1`, properties of the bidegree, so no
subalgebra is needed and the matrix predates the choice of signature.
The window is widest where it pays. The bottleneck is the low-`s` rows, and for those the rows
below finished long ago, so lookahead is bounded by `NASSAU_SPECULATE_AHEAD`, not the wavefront.
Pieces:
- `speculate`: cache keyed by bidegree, byte-capped, plus a build queue ordered by ascending
`(t, s)`. The order matters -- a matrix that lands after its bidegree started is pure waste, so
builders must always take what the wavefront reaches soonest; FIFO would drain the deepest
speculation first.
- builders run OUTSIDE the rayon pool, in a `std::thread::scope`. Their whole point is to use
time the wavefront is not using; pool workers would take slots from the critical path.
- `enqueue_spec` in the scheduler emits the newly opened window on each commit.
- consumer takes from the cache, and on a shape mismatch warns and rebuilds rather than trusting
it.
Off by default (`NASSAU_SPECULATE=0`). Verified with `NASSAU_SPECULATE_VERIFY`, which rebuilds each
cached matrix at consumption time and asserts equality: stem 40 / max_s 30, 318 hits, 0 mismatches,
alongside `NASSAU_GPU_VERIFY`.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
… wait
Two bugs, found by measurement then by a hang.
DUPLICATION. The cache held only finished matrices, so a builder and the bidegree itself could
build the same matrix concurrently. A full build is most of a bidegree's cost, so that duplication
WAS the cost of speculation: stem 150 / theta=125 gave 442 built / 99 consumed / +18% wall at 2
threads, degrading monotonically to +44% at 8. `Slot::{InFlight,Ready,Taken}` gives every bidegree
exactly one owner: builders `claim` before building, and the consumer either takes the matrix,
waits for an in-flight build, or claims the slot so no builder starts work it is already doing.
DEADLOCK. Waiting unbounded then hung the run at stem 40, immediately. Builders call
`restricted_partial_matrix`, which is rayon-parallel, from OUTSIDE the pool: rayon injects the work
and blocks the builder until a pool worker runs it. A consumer is a pool worker, and one parked in
a condvar cannot steal -- so the wait closed a cycle. gdb showed it exactly: builders in
`in_worker_cold` -> `LockLatch::wait_and_reset`, workers parked in `take_or_claim`. The wait is now
bounded (`NASSAU_SPECULATE_WAIT_MS`, default 5s); on timeout the consumer claims the slot and
builds its own, and `publish` drops a matrix whose slot is no longer `InFlight` rather than parking
one nobody will collect.
Verified (`NASSAU_SPECULATE_VERIFY` + `NASSAU_GPU_VERIFY`, stem 40 / max_s 30): 0 mismatches,
built=411 hits=411 -- every published matrix consumed, wasted work 343 -> 9.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
Sampling a stem-200 run settles where the headroom is: CPU sits at 700-900% of a possible 12800% (~120 of 128 cores idle) while GPU 0 bursts to 100% and GPUs 1-3 idle. The run is GPU-bound with the CPU nearly free. That explains the stem-200 A/B: with duplication eliminated, speculation still measured only neutral (1745 s -> 1797 s, +3%). Speculative builds went to the GPU, so they queued behind the very critical path they were meant to relieve -- moving GPU work earlier cannot help when the GPU is the constraint. `NASSAU_SPECULATE_CPU` sends them to `restricted_partial_matrix` instead, where a hit costs the critical path nothing AND removes a launch from the queue it is waiting on. Also records the regime finding that governs this whole lever: at stem 150 speculation is +14% even with perfect dedup (built=2781, hits=2781), and at stem 200 it is +3% -- the penalty shrinks as the wavefront narrows, which is the direction that makes larger stems the place to judge it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
Two things were wrong with the warmer, both invisible because it runs off the critical path. DOUBLE ENUMERATION. `cold_count` runs `admissible_matrices` purely to read `num_mats`, discards the arrays, and then `resident_info` ran the same enumeration AGAIN for every `R` that passed the threshold. Every pinned `R` was enumerated twice. It now enumerates once, memoises the count, and appends the arrays already in hand. SERIAL. One thread warmed one `R` at a time while sampling a stem-200 run shows ~120 of 128 cores idle and the GPU as the constraint. The elements of a degree are independent -- warming an `R` is a pure function of its p-part, and the only shared state is the master's write lock, taken once per STORED `R` -- so the degree now splits across `NASSAU_GPU_PREFETCH_THREADS` (default 16). Plain `std::thread`, never rayon: this must not inject into the pool the wavefront uses, which is the deadlock the speculative builders hit. `resident_append` is factored out of `resident_info` for this, and is the same seam the batched GPU enumeration needs: one place assigns offsets, picks the shard and mutates the master, whether the enumeration came from a serial CPU call, many cores at once, or one big device launch. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
Saturates the enum kernel by giving it work that is not tied to a multiply.
A production enum launch carries only the R's of the ONE multiply that needs them -- Rs/launch mean
1293, waves/SM 0.013 -- and the GPU submission queue is ~1.5 deep, so there is essentially never a
peer launch to merge with. That is why the two obvious fixes measured nothing: widening the grid 6.5x
bought 3.4%, and two CUDA streams bought 527s vs 528s. There was nothing to overlap.
The warmer has no such constraint. Nothing downstream waits on it, so it can hand the kernel a whole
degree at once. `prefetch_degree_gpu` groups a degree's R's by owning device and runs the shards
concurrently -- which also puts the three idle GPUs to work (sampling shows GPU 0 near 100% and
GPUs 1-3 near zero).
Two passes, and no host enumeration anywhere:
- `enumerate_counts_gpu` runs with `emit = 0`, which compiles the stores out and leaves the
odometer alone, filling `out_counts`. So `num_mats` -- the number that decides whether an R is
worth keeping -- now comes from the device. Previously the threshold test alone cost a full CPU
enumeration of every R in the degree, kept or not. Counts are memoised into COLD_COUNT for every
R seen so the multiply path inherits them.
- `enumerate_batch_gpu` emits the arrays for the keepers, chunked to bound device scratch
(`NASSAU_GPU_ENUM_BATCH_MB`, default 2048); a degree's full output reaches many GB at high stems.
Off by default (`NASSAU_GPU_PREFETCH_GPU`). `NASSAU_GPU_ENUM_BATCH_VERIFY` checks every batch against
`admissible_matrices`.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
Parallelising it is 14% slower and warms nothing extra. Measured at stem 150 / theta=125, PIN=200, AHEAD=25, interleaved and counterbalanced: 1 thread 345 s / 336 s, 16 threads 396 s / 382 s -- and all four runs reached the SAME frontier (degree 176) with the SAME enumeration load on the critical path (Rs/launch 531). The premise was wrong, not the implementation. The warmer is limited by `prefetch_ahead`, not by its own throughput: it fills the window and sleeps. Extra threads cannot warm anything further ahead, so they only contend with the wavefront. The idle-core observation that motivated it is real but irrelevant here -- idle cores only help if the thing you parallelise is the bottleneck. The knob stays, with the numbers recorded next to it and a note that it is only worth raising together with `NASSAU_GPU_PREFETCH_AHEAD`, and only when `to_degree` shows the warmer falling behind. The single-enumeration fix from the previous commit is unaffected and stays. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
…-22%) `Rs/launch` sat at exactly 531 across every warmer configuration tried -- AHEAD=25 serial, AHEAD=100 serial (warmed to degree 186), AHEAD=100 with 16 threads (degree 211, 579k R's, twice the warming), and batched device enumeration. Warming further AHEAD never moved it, because the leftover R's were never ones the warmer failed to reach in time: they are the ones `pin_min_mats` deliberately skips as cheap. Warming further DOWN the cost distribution moves it 4.5x, and that is worth real time (stem 150, theta=125, interleaved, counterbalanced): pin 200 -> 418 s / 367 s, Rs/launch 531 pin 20 -> 294 s / 317 s, Rs/launch 117 pin 2 -> 304 s / 350 s, Rs/launch 4, a third fewer launches 20 beats 200 by 22% and beats running with no warmer at all (~384 s) by 20%. It turns over below that: 2 drives enumeration to essentially nothing and still loses, because warming every trivial R costs more than enumerating it. Capped theta only -- with theta uncapped everything is resident and the knob does nothing -- which is exactly the regime stems 250+ are forced into. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
…thing `NASSAU_GPU_ENUM_BATCH_MB` defaulted to 2048, which silently defeated the whole point. A pinned R (num_mats >= 200) is often megabytes of col_sums/masks, so a 2 GB emit budget fits only a few hundred of them. Measured consequence: the batched path ran at ~947 R's per launch -- BELOW the 531-mean multiply-driven launches it was supposed to dwarf -- and `Rs/launch max` came out IDENTICAL (13650) in the batched and un-batched arms, which is proof no batch was ever large. At ~30 blocks of the 3168 an H200 holds, waves/SM stayed at 0.005 exactly as before. So the earlier "batched enumeration is 18.5% slower" result does not test batching. It compares two small-launch configurations, one of which also pays device contention. Budget default is now 16384 MB. Batch sizes are now LOGGED rather than inferred: `batches= Rs/batch= max=` in the [batch-stats] line, from counters recorded at both launch sites. `Rs/launch` is dominated by the thousands of multiply-driven launches, which is what hid this. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
…wer" results Both results were implementation artefacts, not findings. The argument they appeared to contradict -- free cores mean a bidegree does strictly less work -- is sound; the code was not using free cores. SPECULATIVE CPU BUILDS RAN IN THE WAVEFRONT'S OWN POOL. `restricted_partial_matrix` is rayon-parallel, so each of 16 builders injected a large parallel job into the GLOBAL pool. Those cores are not free in that case: wavefront tasks queue behind speculative chunks and work-stealing spreads it. Measured at stem 150 / theta=125 / pin 20: 649 s against a 310 s baseline, a 2.1x regression, while the cache itself worked perfectly (53% hit rate, zero duplication, zero waste) -- exactly the signature of contention rather than a bad idea. Builds now `install` into a dedicated `MaybeThreadPool` (`NASSAU_SPECULATE_POOL`, default a quarter of the machine), which confines every nested `par_iter` to that pool. BATCHED ENUMERATION NEVER GOT BIG BATCHES. It swept ONE degree per launch, and a degree yields only a few thousand R's -- split four ways, ~740 per shard, about 23 blocks of the 3168 an H200 holds. That is the same starvation the multiply-driven launches suffer, which is why `Rs/launch max` came out identical (2965) in the batched and un-batched arms and waves/SM never moved. It now sweeps the whole lookahead window (`prefetch_degrees_gpu`) into one batch. Also: the batch-size fields were computed but never printed -- the format-string edit missed a `waves/SM=` prefix -- so batch sizes had to be inferred from `Rs/launch max`. Now printed directly as `batches= Rs/batch= max_batch=`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
`speculate` caches one whole matrix per bidegree, which caps how early a builder can start: `(s, t)`'s matrix is only determined once `(s - 1, t - 1)` is committed, giving one degree of slack per row and a ~30% hit rate. A block is the rows coming from generators of `modules[s - 1]` in a single degree -- the rows `Sq(R) * x` with `deg Sq(R) + deg x = t`. Two facts make a block available far earlier than the matrix containing it: * Its rows. A free module's basis at a fixed degree is generator-major and append-only, so the block's row range is frozen once `(s - 1, gen_deg)` is committed, not `(s - 1, t - 1)`. * Its columns. `d(x)` lands in the radical (minimality -- already what licenses the existing truncation in `apply_to_basis_element_restricted`), so the block is supported strictly below generator degree `gen_deg` and can be built that narrow, then zero-extended. So one commit opens blocks across a whole column of future bidegrees instead of a single matrix, which is what deepens the GPU submission queue. The consumer takes whatever landed and folds the rest into ONE coalesced build, so a miss costs only its own rows -- no waiting, and none of `speculate`'s deadlock hazard. Verified at stem 30/max_s 25 under NASSAU_SPECULATE_VERIFY + NASSAU_GPU_VERIFY: 0 mismatches, 93.0% of rows served from blocks. Two bugs the verifier caught, both fixed here: * The consumer re-derived each block's row range at consumption time and disagreed with the builder (start=23 for a block whose rows sat at 22), misplacing every row. Blocks now carry the offset they were built against; the consumer never re-derives it. * Blocks are validated for overlap and bounds before use rather than trusted, so a stale one degrades to a rebuild instead of corrupting the matrix. `NASSAU_BLOCK_DIAG` keeps the attribution probe that separated the row-range bug from the column bound (which measured clean: beyond_narrow=0 throughout). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
At stem 200 blocks LOST (1794s vs 1728s baseline) while holding 264GB against 116GB, with row_rate=16.1% and unused=150.3GB -- the cache ended pinned at its cap. The cause is a gap between producer and consumer. The consumer only assembles from blocks when the matrix is within `reuse_within_cap`; above it, it builds per signature and never calls `take_all`. But `speculate_block` did not check that cap (whole-matrix `speculate_build` does), so builders precomputed blocks for bidegrees that would never collect them: wasted CPU, and the blocks stayed in the cache until the run ended. Through `has_room()` that also starved the bidegrees that DO collect, which is the 16.1% coverage. Two fixes: * `speculate_block` bails on `!reuse_within_cap`. The dims are read before the bidegree's frontier is frozen, so they are a lower bound -- sound in one direction: already over the cap now means over it for good. * The consumer releases blocks for any bidegree taking the per-signature path, so whatever slips through the lower-bound test cannot leak. Counted as `released=` rather than inferred. The cap check reuses the single `block_ranges` pass instead of calling `restricted_dims`, which repeated the same `compute_basis` work per block and cost more than speculation gained (row coverage 93% -> 55% at stem 30). Hoisting `FreeModule::compute_basis` over the whole degree range would remove that cost entirely but is NOT valid: a module computed through max.t() up front can no longer have generators added back. Only the algebra is precomputed. Verified at stem 30/max_s 25 under NASSAU_SPECULATE_VERIFY + NASSAU_GPU_VERIFY: 0 mismatches, 84.1% of rows served from blocks. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
`row_rate` only counts bidegrees that take the full-matrix reuse path, so it cannot distinguish "speculation covered most of the work" from "speculation covered most of the little work it was allowed to see". `[reuse-split]` splits total multiply work (rows x cols) by whether the bidegree was within `reuse_within_cap`. The answer explains why block speculation won at stem 150 and did nothing at stem 200: stem 150: 100.0% reachable, 0 bidegrees over cap stem 200: 13.1% reachable, 941 bidegrees holding 86.9% of the work At stem 200 the 941 largest bidegrees each exceed the 1e10 cap, so they never take the full-matrix path -- no blocks, and no one-big-launch GPU amortization either; they fall back to per-signature builds. Blocks were competing over the remaining 13%, which is why 60% row coverage bought no wall time (1846s vs 1801s uncapped baseline). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
Signature-axis speculation is not viable: a signature mask needs the module basis at degree t, which needs progress[s-1] >= t-1 -- exactly the condition for the bidegree to be runnable. Zero lookahead, queue depth ~1, which is the starvation this design exists to escape. The generator axis is the only one with a window, since block `g` needs only progress >= g. So the question for the 87% of stem-200 work above `reuse_within_cap` is whether generator-blocks can serve it without materialising the full matrix. There is an asymmetry to exploit: blocks are stored NARROW (block `g` stops below generator degree `g` by minimality) while the full matrix pads every row to `next_dim`, so the block set is triangular where the matrix is rectangular. If a signature's rows were selected straight from the blocks, peak would be (block set + one signature) rather than (full matrix + one signature). `[reuse-split]` now reports that ratio. Near 0.5 the idea is worth building; near 1.0 the triangle is too shallow and relaxing the cap is the only lever. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
At stem 200, 941 bidegrees hold 86.9% of all multiply work and every one exceeds `reuse_within_cap`, so they never assemble a full matrix and plain blocks cannot reach them: block delivery needs every block resident at once, which is the memory bound the cap exists to hold. Signatures give a second axis. `signature_mask` selects, within a generator degree, the ops matching the packed signature from `ppart_table(t - gen_deg)` alone -- never the module's degree-`t` basis -- so a `(gen_deg, signature)` piece is determined exactly when the generators of `gen_deg` are, keeping the early window that makes speculation work at all. And because `signature_mask` walks generators in layout order, a piece is a CONTIGUOUS run of the signature's matrix, so assembly is row concatenation into a matrix preallocated at `next_dim`: matrices are row-major, and widening one afterwards would restride every row. The consumer takes one signature at a time and discards it, so peak stays at one signature however large the bidegree is. No extra multiplies: signatures partition the operations, so this is a strict refinement of the same rows. The builder now dispatches on the cap -- contiguous per-generator pieces below it, signature pieces above it -- and `release` no longer marks a bidegree done, which would have silenced signature delivery for exactly the bidegrees this targets. Also fixes a coverage bug: `iter_gen_offsets` yields one entry per GENERATOR, not per degree, and the old code took only the first entry per degree, so every other generator of a degree was silently left to the critical-path rebuild. Verified at stem 30 with `NASSAU_GPU_REUSE_MAX_WORK=1000` forcing 92.2% of work over the cap: 0 mismatches against fresh per-signature rebuilds, 82.0% of rows served from pieces. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
Covering every generator of a degree sounded like a free coverage win. It is
not: stem 150, theta=125, one binary, interleaved --
first generator only 173 s coverage 53.4%
merged over the degree 194 s coverage 53.3%
one piece per generator 202 s coverage 51.4%
Coverage is flat because a degree almost always carries one generator here, so
the extra generators are nearly empty. What changes is piece size and builder
load, and both bigger pieces and more pieces lose. Speculation is already at
its useful limit at stem 150; past it the builders just compete with the
wavefront.
The bisect also clears the signature scaffolding: with merging off, the new
binary matches the old one exactly (173 s), so none of the cost was the
(gen_deg, signature) path.
NASSAU_BLOCK_MERGE=1 restores full-degree coverage for regimes where generators
are denser.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
The first cut of signature speculation ran 5077 s at stem 200 against 3799 s for plain blocks, and 4530 s for no speculation at all. Two defects: * `signature_piece` rescanned every generator and its whole `ppart_table` once PER SIGNATURE, so a degree cost O(generators x table x signatures). Every operation has exactly one signature, and all signatures of a subalgebra share the same packed mask (it depends only on the profile), so one pass bucketing ops by `op.bits() & mask` does the whole degree for what a single signature used to cost. * It built 2 988 096 pieces averaging 32 rows each -- far too small to repay allocation, locking and bookkeeping. `NASSAU_SIG_MIN_ROWS` (default 512) drops them. Signature mass is very uneven (one signature is often ~96% of its bidegree), so a threshold discards most pieces while keeping most rows. Re-verified at stem 30 with `NASSAU_GPU_REUSE_MAX_WORK=1000` forcing 92% of work over the cap and no threshold: 0 mismatches, 78.3% of rows served. A huge `NASSAU_SIG_MIN_ROWS` disables the signature path exactly, so plain blocks and signature pieces can be A/B'd within one binary. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
With no pieces available, `assemble_signature` still allocated the result matrix, built a SECOND matrix for the missing rows, and copied row by row -- where the old code built once and returned it. That is pure overhead on every signature speculation did not reach, and it cost 7% at stem 200: the control arm (signature path disabled) ran 4082 s against the 3799 s the same configuration measured before this path existed. Return the direct build when nothing landed. Still counted as missed rows so `row_rate` stays honest. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
Stem 200, theta=125, counterbalanced within one binary (a huge
NASSAU_SIG_MIN_ROWS disables the path exactly, so both arms share a build):
signature path off 4170 s (cold, ran first) / 3826 s
min_rows=512 3786 s / 3775 s
Round 1 looked like a 9% win. It was not: the first stem-200 run after an idle
machine is reliably ~9% slow, and the control's round-2 number lands at 3826 s
against the treatment's 3781 s -- about 1%, i.e. neutral. `built`
(205 815 -> 207 904) and `row_rate` (28.1% -> 28.3%) say why: at 512 rows the
threshold makes the path inert.
So the affordable threshold is the inert one, and the range between "inert" and
the 5077 s of min_rows=1 is untested -- that measurement predates the
single-pass rewrite, which made low thresholds far cheaper. Sweep pending.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
The speculation pool defaulted to max_num_threads()/4, added as hygiene against
speculative par_iters stealing workers from the wavefront -- a fear that traced
back to a broken binary, not to measured contention. It capped speculation by
construction: ~20 of 128 cores in use, and block coverage plateaued near 52%
however the pieces were reshaped. Every granularity experiment (bigger blocks,
per-generator, per-signature) changed the SHAPE of speculative work while its
CAPACITY stayed pinned, which is why none of them moved the needle.
Stem 150, θ=125, control repeated last:
builders / pool wall row coverage
16 / 32 170 s 168 s 52.2% 50.6%
24 / 64 134 s 63.6%
32 / 96 119 s 70.8%
48 / 120 134 s 77.0%
So the contention is real but only past ~3/4 of the machine: at 120 threads
coverage still climbs while wall time regresses. Default is now 3/4.
That is -30% on top of the win speculation already had, and -63% against the
320 s no-speculation baseline at this bidegree.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
Blocks do what they were meant to at stem 200 -- bidegrees retire faster and the run goes 4465 s -> 3708 s (-17%). But row coverage sits at 33% against 71% at stem 150, so two thirds of the rows are still multiplied on the critical path, and total CPU is ~13 of 128 cores. With 96 speculation threads available that means the builders are IDLE rather than contended: they are bailing, and nothing recorded which bail-out they take. Two counters: * `bails(room/done/claimed/other)` on the [blocks] line. The leading suspect is `has_room()` against NASSAU_SPECULATE_MAX_GB=150 while peak RSS is already 196 GB, on matrices ~15x larger than stem 150 -- once the cache sits at its cap speculation simply stops, which no pool or granularity change can move. * `[wavefront] in-flight bidegrees mean/max`, to separate "too few bidegrees are eligible" from "eligible ones are not being run". Speculation shortens a bidegree but does not change admission -- `(s,t)` still needs `(s,t-1)` and `(s-1,t-1)` -- so the frontier stays a slope-1 staircase over `s`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
At stem 200, 718 171 queue items were popped and only 259 150 built -- 456 767 bailed because the bidegree was ALREADY COMPUTED. Two explanations were wrong: * Not the memory cap. Raising NASSAU_SPECULATE_MAX_GB 150 -> 400 left coverage at exactly 33.2% and reported bails(room)=0. The cap was never reached. * Not time lost discarding stale items. A `done` bail is a mutex and a bool; 456k of them across 3935 s is nothing. And the builders were not saturated either -- 1310% CPU of 128 cores with a 96-thread pool -- so they were neither too slow nor too busy. A block queued 20 internal degrees ahead should comfortably finish first, so something else is wrong. `[specqueue]` reports queue depth sampled at every successful pop, plus builder seconds split into idle-on-empty-queue versus building. Near-zero depth with large idle means the producer's gate (gen_deg <= min(progress[r-1], progress[r-2])) admits far less than the nominal 24-degree window suggests, and the lookahead is illusory. Large depth with large idle would instead mean builders block on something CPU accounting does not show. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
Two changes to speculation queue discipline, both from the stem-200 numbers: 718 171 items popped, 259 150 built (36%), 33% of rows covered. * Built fraction ~= coverage fraction means selection was size-NEUTRAL. Since only a fraction of the queue is ever built, taking the biggest first converts the same builder time into more coverage. So `rows` is now the secondary key. * It is only the SECONDARY key. Blocks have deadlines -- worthless once their bidegree runs, equally useful any time before -- and the queue never drains, so near blocks expire while far ones stay valid. Imminence-first is therefore earliest-deadline-first, the optimal policy for meeting deadlines. Ordering by size INSTEAD was tried and is wrong in principle; it lets builders work far ahead while imminent blocks expire. * `NASSAU_BLOCK_MIN_ROWS` (default 1024) drops tiny blocks entirely. A block that is not precomputed costs the consumer nothing extra -- its rows join the single coalesced build it was already making -- so a small block has near-zero value while still costing a queue slot, claim, lock, allocation and publish. With 718k items the tail was most of the bookkeeping and almost none of the benefit. Row count is `num_gens[g] x |ppart_table(t-g)|`, both O(1), so sizing every candidate at enqueue costs nothing. Stem 30 cannot discriminate the two orderings (46.8% vs 46.9% coverage) -- it is too fast for speculation to matter. Needs a stem-150 A/B. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
`[specqueue]` reported mean depth 133 589 (max 343 227) while `builder_idle` showed builders parked for 91% of their thread-time. Both were true and they describe different moments: depth was sampled only at a successful POP, and a pop only happens when there is work, so the statistic could only ever report "deep". The producer emits a whole strip per commit, floods the queue, and builders drain it over minutes; between bursts the queue is empty and nothing was observing it. A sampler thread now reads depth every 100 ms, giving a time-weighted mean and the fraction of samples with an EMPTY queue. Both figures are printed alongside the at-pop ones so the bias stays visible rather than being silently corrected. The arithmetic this has to explain: ~718k items produced, ~29 ms per build, 32 builders -> ~650 s of building spread over a 3685 s run, against 481 535 bails on already-computed bidegrees. That is consistent with bursty production the builders cannot drain before the wavefront passes, which is what makes queue ORDER (earliest-deadline-first, committed separately) the thing that matters. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
A mean hides the shape, and the shape is the question. Enqueues are triggered by COMMITS, and commits get sparser as bidegrees get larger, so the queue is expected to be front-loaded and to starve later -- exactly when the expensive bidegrees need it most. Averaging that away is how "mean depth 133 589" and "builders parked 91% of their thread-time" ended up in the same run. `[qtrace]` emits every 2 s: elapsed, depth, total ROWS queued (depth alone says nothing about whether the work is worth anything), tmin/tmax of the queued deadlines (near work or far), and how many builders are occupied at that instant. Together with the 100 ms time-weighted sampler this gives the full picture: whether the queue starves, when, and what is left in it when it does. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
The first sampler took the queue mutex and iterated the heap. At stem 200 that
emitted NOTHING: with ~343k items and 32 builders contending, a sampler never
wins the lock, and iterating under it would block every builder. Depth and
queued rows are now atomics maintained on push/pop, so sampling is two loads --
which also makes 60 Hz free, and resolution matters because the burst that
revealed the behaviour lasted under 2 s.
What it shows at stem 200, clean and uncontended:
t=61s depth=80283 rows=48.4M building=32 built=76400
t=182s depth=30987 building=32 built=139891
t=242s depth=0 building=7 built=176814
t=423s depth=0 building=0 built=196574
One burst saturates all 32 builders for four minutes, drains, and never refills;
`built` then creeps at ~5k/min. Enqueues are triggered by COMMITS, and once the
wavefront is grinding through a few large bidegrees the commits are sparse, so
production falls to a trickle builders consume instantly. Builders are starved
of eligible work for ~90% of the run -- which is the 6.5% busy / 33% coverage at
stem 200, against 70% busy / 70% coverage at stem 150.
Also retracted here: an earlier "speculation freezes after 6 s" reading came from
three runs racing on one machine after pkills silently failed. Clean stem-150
data shows `built` climbing steadily throughout.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
A block's lookahead is ~`t - gen_deg`, the OPERATION degree: it becomes eligible
when progress[s-1] reaches gen_deg, and its bidegree runs when progress[s-1]
reaches t-1. So blocks with gen_deg near t get almost no warning. If the ~33%
coverage ceiling at stem 200 is that rather than a tunable, misses must
concentrate in the LOW operation-degree buckets.
`NASSAU_OPDEG_HIST=1` buckets served vs rebuilt rows by operation degree.
Stem 150 confirms the mechanism -- coverage rises monotonically with lookahead:
op_deg 0-24 32.3%
op_deg 25-49 44.6%
op_deg 50-74 59.5%
op_deg 75-99 76.1%
op_deg 100-124 77.9%
op_deg 150-174 100.0%
The remaining question is whether stem 200's row mass has shifted into the
short-lookahead buckets, which would make the ceiling structural rather than
something capacity, memory, granularity or window size can move -- all four of
which have now been measured and rejected.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
`NASSAU_OPDEG_HIST=1` buckets served vs rebuilt rows by operation degree (a
block's lookahead). Coverage rises with lookahead at both stems, but the shape
is the same and the row mass has NOT shifted toward short lookahead:
op degree stem 150 stem 200
0-24 32.3% 28.6%
50-74 59.5% 49.9%
100-124 77.9% 75.0%
overall 68.7% 61.6%
The histogram runs only inside `assemble_full_restricted`, i.e. the UNDER-cap
path -- so blocks work about as well at stem 200 as at stem 150 where they
apply. The reported row_rate=25.9% is just 0.13 x 0.62 + 0.87 x 0: 87% of
stem-200 work sits in over-cap bidegrees that never assemble a full matrix and
get no coverage at all.
This corrects the previous commit's hypothesis, which guessed the ceiling was
structural lookahead. It is not; it is the reuse cap. That reopens
`(gen_deg, signature)` pieces, which exist precisely to reach over-cap work and
were called closed on a measurement using min_rows=512 -- a threshold that made
the path inert (built 205 815 -> 207 904). The range below it was never tested
after the single-pass rewrite made low thresholds ~46x cheaper.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
…e bug
"Signature pieces are neutral" was an artefact of NASSAU_SIG_MIN_ROWS=512, which
made the path inert (built moved 205 815 -> 207 904). That threshold was added
when pieces cost 5077 s and 2 988 096 of them -- BEFORE the single-pass rewrite
made a whole degree's pieces cost what one signature used to. Carrying it
forward after the fix, and then reading "inert" as "does not help", was the
error.
Stem 200, theta=125, control repeated last:
sig_min_rows wall built coverage builder idle
off 3836 s 246 974 32.5% 112 011 s
128 4044 s 312 795 36.1% 118 424 s
32 3329 s 1 278 972 43.7% 5 771 s
off (last) 3416 s 241 482 32.3% 98 419 s
At 32 rows the starvation that dominated every stem-200 measurement disappears:
builder idle 98 000 s -> 5 771 s, coverage 32% -> 44%, 5x the blocks built. This
is the design working as intended -- an over-cap bidegree never assembles a full
matrix, but it can still be served one signature at a time, so the reuse cap is
not a coverage ceiling after all.
Wall time is -2.5% against the trailing control, but the two control arms differ
by 12%, so no speedup is demonstrated yet; only that the mechanism engages.
Sweeping lower thresholds next.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Replaces the doubling-realloc-with-copy growth of every resident device buffer (admissible master
col_sums/masksand the resident basispparts/lens) with a single segmented, append-only, no-copy growth mechanism.Why
The stem-140+
dx != 0(d²≠0) failures were traced to cubecl silently handing back the wrong device buffer under memory pressure (ManagedMemoryDescriptor id mismatch), driven by the master's realloc-doubling transient: at a growth point the old (cap) and new (2×cap) buffers are both live plus an on-device copy between them — a ~2–3× spike that tips cubecl into its corruption regime. Verified it is not the RREF (checked to 2^18 square; failing matrices are only ~10^4 square) and not the multiply (CPU-RREF of GPU output is clean).How
SegBuf+seg_grow!replaceGrowBuf,resident_dev_handle!,basis_dev_handles!,stage_upload!,RESIDENT_REALLOC, andRESIDENT_INIT_CAP(all deleted — no two coexisting growth paths). Growth allocates only the new fixed-size segment(s) and stage-writes the tail; existing segments are never reallocated or copied, so device peak islive + one_segment.multiply_batch_kernelbinds each store asMASTER_MAX_SEG(=16) segmentArrays and gathers a thread's matrixcs/mk+ its term p-part into smallWORKING_CAPlocals viaseg_read_u16/seg_read_u32(correct at any offset — no layout padding), then calls the unchanged puremultiply_pairwith base 0.master_seg_elems()(envNASSAU_GPU_MASTER_SEG_ELEMS, default1<<31,< u32::MAX) => 64 GiB/buffer over 16 segments. Over-cap is a cleanassert!, not corruption.Validation (bit-exact on H200)
seg_read_matches_contiguous— the 16-arg segment-read primitive.multiply_batch_matches_referenceacrossNASSAU_GPU_MASTER_SEG_ELEMS768–4096 — multi-segment single-launch gather.multiply_batch_incremental_growthatseg_elems=8192— cross-launch append into the partially-filled last segment.algebraGPU suite green. (3 remaining suite failures —test_ppart_multiplier_3,basis_element_from_string_total_milnor,test_evaluate_3— are pre-existing odd-prime/CPU tests outside this diff.)Next
Fire the dx-clean stem-200 run and capture the flat
[MEM] master=growth curve, then decide host-paging vs 4-GPU sharding for stem 300.🤖 Generated with Claude Code
https://claude.ai/code/session_01SmhK9czwJj2dZeemDhXtBf