aggregate: the remaining assembly is latency (a ~20 s join-tree floor + a 27-step serial resolve chain), not work

[defenwycke]

⏰ Status, 2026-09-13 — two of the three levers have shipped; the measurement has not

Verified against main at 95c5e84, not from memory.

lever	state
1. resolve chain off the network	✅ shipped — HAZYNC_RESOLVE_LOCAL=1 (#270)
2. narrow top levels locally	⬜ not built
3. record per-worker RTT	✅ shipped — [rtt] peer=… kind=join rtt_ms=… bytes_out=… (#291)

⛔ The headline claim below is still unmeasured. This issue says run 4's 128.7 s of assembly was
geography, and says in as many words that "that part is inference, not measured (no RTTs were
recorded)". They are recorded now — timed on the server's clock at both ends, so
rtt_ms - compute_s*1000 gives transport and queueing without the two clocks having to agree,
which across rented boxes in several countries they do not.

Nobody has run it. The instrumentation exists; the number that would settle lever 1 and lever 3
needs the same receipts on one card type in two geographies, which needs cards. Until then this
issue's central claim remains inference, and lever 2 should not be built on the strength of it.

Follow-up to #235, which measured the aggregate's assembly on an N-sweep (A40s, same receipts and binary as milestone run 4). At fan-out, the part that no longer shrinks is latency, not work:

N	join tree	resolves (27, serial)	assembly
4	135.2 s	26.2 s	162.5 s
26	37.3 s	25.4 s	63.5 s

• Join tree ≈ 19.5 s + 463/N card-seconds. The wide levels scale with N. The narrow top levels (widths 37, 19, 10, 5, 3, 2, 1) are a critical path of ~1.1 s per level, each level waiting on the one below.
• Resolves: 27 steps, each consuming the previous conditional, median ~0.8 s each (A40), ~0.4 GPUs busy. They're flat in N by construction.
• Run 4 (4090s across 7 countries) measured 128.7 s of assembly where compute alone predicts ~17 s. Every join and resolve is a coordinator ↔ worker round trip carrying receipts, so geography is paid per step. That part is inference, not measured (no RTTs were recorded).

Levers (not built, not measured):

1. Run the resolve chain on the coordinator (or one co-located worker) instead of 27 network round trips. On a 4090 that's ~6 s of compute.
2. Do the narrow top levels locally: once a level is narrower than the fleet, stop distributing it.
3. Keep an aggregate's fleet in one region, and record per-worker RTT in the runner so the geography question gets answered.

The first measurement that would settle (1) and (3): the same receipts on one card type, in two geographies, with per-step timestamps (the #235 sweep's serve.sh wrapper does this).

[defenwycke]

The RTT-vs-compute measurement this issue has been waiting for — run 2026-09-17

This issue's status block said the central claim was inference:

Run 4 (4090s across 7 countries) measured 128.7 s of assembly where compute alone predicts ~17 s.
… That part is inference, not measured (no RTTs were recorded).

and that nothing could move until someone ran it:

Nobody has run it. The instrumentation exists; the number that would settle lever 1 and lever 3
needs the same receipts on one card type in two geographies, which needs cards.

It has now been run — not on the aggregate, but on a mode-6 board-block prove across three cards.
See docs/history/BENCH_MODE6_3xRTX4090_2026-09-17.md (pinned at 5bc4e48).

What was measured

Three RTX 4090s proving board block 74,928 (2,353 segments) via the seg-serve push transport, with
the #291 [rtt] instrumentation live. Both timestamps come from the coordinator's own clock
(main.rs:6062), so rtt_ms - compute_s*1000 is transport and queueing without the two machines'
clocks having to agree — the property this issue asked for.

job kind	n	compute p50	compute p90	compute max
segment	2,353	2.83 s	3.08 s	11.53 s
join	2,352	0.38 s	0.51 s	1.89 s

Coordinator dispatch-to-return for the same joins:

	n	p50	p90	max
join round trip	2,352	46.7 s	91.0 s	393.6 s

A join costs ~0.4 s of GPU and ~47 s of round trip — a ~120× gap that is entirely transport.

Per-worker, over 4,706 tasks and 3.53 GB on the wire:

pod	tasks	segment compute	join compute	wait
pod 1 (coordinator's own card, no network)	2,510	2,609 s	617 s	326 s
pod 2 (remote, SSH hub tunnel)	899	1,573 s	125 s	1,855 s
pod 3 (remote, SSH hub tunnel)	1,297	2,532 s	180 s	834 s

Pod 2 idled longer than it worked. Pod 1 — same card type, no network between it and the queue — did
2.8× pod 2's tasks.

What this settles

Lever 3's premise is no longer inference. On this codebase, with receipts on the wire per step, the
round trip dominates the compute by two orders of magnitude, and the pod with no network between it and
the queue outperformed identical hardware behind a tunnel by 2.8×. "Keep the fleet in one region" is
supported by measurement, not reasoning.

What this does NOT settle — please do not read it as more than it is

• ⛔ Different workload. This issue's tables are aggregate assembly on A40s (Aggregate: measure the N-curve past 2, and remove the ~20s non-scaling execute #235 sweep). This is a
  mode-6 range prove on 4090s. The transport principle transfers; the numbers do not substitute for an
  aggregate measurement. Lever 2 ("narrow the top levels locally") is better supported than it was, but
  still not on measured aggregate data.
• ⛔ Joins only. The coordinator emits [rtt] for join and lift, never for segment (2,352 and 1
  lines respectively). The 120× gap is demonstrated for joins alone; the segment-phase transport cost is
  still unmeasured.
• ⛔ Not a clean two-geography A/B. The variable was co-located vs SSH-tunnelled over a domestic
  uplink, not region per se — and the two remote pods differed from each other by more than 2× in wait
  (1,855 s vs 834 s), so tunnel quality is confounded with geography here.
• ⛔ No 1-card baseline (the reference run was stopped after execute to save pod time), and drivers
  differ across the three cards, so per-card timings are not strictly comparable.

Consequence already taken

#365 (no job deadline on the push transport) was built and merged as #370 off the back of this run: a
peer deadline of 8 × p90 RTT, floored at 600 s and capped at 1800 s, needing 16 samples before it
binds. Sized from this data — 600 s kills 0 healthy jobs here, 300 s would kill 59, 120 s would kill 111.