Skip to content

Commit aac0faa

Browse files
Flotapponnierflorent-sysclaude
authored
hotfix: head-lag Solana opens on First to report, 1 ms floor, 30 s presence ratio (cherry-picks) (#2393)
* bench spec: optional chart block (min_range, default_range, window_label) for daily-cadence benches (#2364) perp-daily-volume moves once per UTC day, so 1h/6h/24h drew one point at "now" and the ledger said "(24h)" for a closed-day figure. The chart now hides ranges below min_range, opens on default_range and the ledger's base window reads "last closed day". Claude-Session: https://claude.ai/code/session_01HJgbZCqjR4nvCfcJSzofbw Co-authored-by: Flotapponnier <florent@mobula.io> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> (cherry picked from commit f9e4360) * head-lag: Solana opens on First to report; leader lag floored at 1 ms; success ratio on the real 30 s scrape (#2392) - spec chart.default_panel_by_chain: a chain tab can open on a metric panel; Solana opens on First to report, Base keeps the head lag view. A ?view= in the URL still wins on first paint. - The site drops rows whose headline is exactly 0 (Singapore: Mobula won every race and vanished from the ranked chart); the race lag is now floored at 1 ms, below the tie window. - Presence ratio divided by 5,760 assumed a 15 s scrape; the store is scraped every 30 s, so every provider capped at 49.5 %. Now 2,880. Claude-Session: https://claude.ai/code/session_01HJgbZCqjR4nvCfcJSzofbw Co-authored-by: Flotapponnier <florent@mobula.io> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> (cherry picked from commit d1646ef) --------- Co-authored-by: Flotapponnier <florent@mobula.io> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent 6b3a189 commit aac0faa

9 files changed

Lines changed: 122 additions & 15 deletions

File tree

benchmarks/aggregator-head-lag.yml

Lines changed: 9 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -72,8 +72,8 @@ methodology:
7272
- "Regions: us-east, eu-west, sgp. Cross-region median reported in the headline."
7373
- "Reference on Base: the sequencer's flashblock preconfirmation stream, held by us in every region. A Base block becomes queryable about 0.36 s after the timestamp it carries, while flashblocks publish every 200 ms inside the 2 s interval, so that timestamp is not the moment a trade becomes knowable."
7474
- "Base emissions we cannot match to a flashblock are dropped rather than measured against a different ruler. Measuring against the block timestamp charged a provider reading preconfirmations with a negative lag, and described the chain's stamping convention rather than the provider's pipeline."
75-
- "Metric: gauge `head_lag_seconds`, sampled every 15 seconds. Aggregated over the 24-hour window using `quantile_over_time`."
76-
- "Success rate: presence ratio. Share of expected sampling slots where a value was actually emitted (5,760 expected per provider per day at the 15s cadence). 100% means the aggregator's feed was reachable for the full window."
75+
- "Metric: gauge `head_lag_seconds`, scraped every 30 seconds. Aggregated over the 24-hour window using `quantile_over_time`."
76+
- "Success rate: presence ratio. Share of expected sampling slots where a value was actually emitted (2,880 expected per provider per day at the 30 s scrape). 100% means the aggregator's feed was reachable for the full window."
7777
- "Cardinality: 4 aggregators × 4 chains × 3 regions = 48 active series."
7878
- "Probe placement: the three probes run in their named regions on Railway (europe-west4, us-east4, asia-southeast1). Until 2026-09-15 all three ran in asia-southeast1 under regional labels; region columns before that date are three Singapore vantage points."
7979

@@ -94,6 +94,10 @@ source: https://github.com/ChainBench/OpenChainBench/tree/main/harnesses/aggrega
9494
# reported first over the last 24 h, our node reference included as a
9595
# competitor. The avg by (aggregator) collapses chain x region series the
9696
# way the headline queries do; the chain filter narrows it per chain.
97+
chart:
98+
default_panel_by_chain:
99+
solana: first_share
100+
97101
metric_panels:
98102
- id: first_share
99103
label: First to report
@@ -188,7 +192,7 @@ providers:
188192
p90: avg by (aggregator) (quantile_over_time(0.90, head_lag_seconds{aggregator="mobula"}[24h])) * 1000
189193
p99: avg by (aggregator) (quantile_over_time(0.99, head_lag_seconds{aggregator="mobula"}[24h])) * 1000
190194
mean: avg by (aggregator) (avg_over_time(head_lag_seconds{aggregator="mobula"}[24h])) * 1000
191-
success: clamp_max(avg by (aggregator) (count_over_time(head_lag_seconds{aggregator="mobula"}[24h]) / 5760), 1)
195+
success: clamp_max(avg by (aggregator) (count_over_time(head_lag_seconds{aggregator="mobula"}[24h]) / 2880), 1)
192196
sample_size: sum(count_over_time(head_lag_seconds{aggregator="mobula"}[24h]))
193197
# Hybrid `unless` guard: drop the sample when the gauge saw no
194198
# fresh event in the last 15 min AND we have enough scrapes in
@@ -226,7 +230,7 @@ providers:
226230
p90: avg by (aggregator) (quantile_over_time(0.90, head_lag_seconds{aggregator="codex"}[24h])) * 1000
227231
p99: avg by (aggregator) (quantile_over_time(0.99, head_lag_seconds{aggregator="codex"}[24h])) * 1000
228232
mean: avg by (aggregator) (avg_over_time(head_lag_seconds{aggregator="codex"}[24h])) * 1000
229-
success: clamp_max(avg by (aggregator) (count_over_time(head_lag_seconds{aggregator="codex"}[24h]) / 5760), 1)
233+
success: clamp_max(avg by (aggregator) (count_over_time(head_lag_seconds{aggregator="codex"}[24h]) / 2880), 1)
230234
sample_size: sum(count_over_time(head_lag_seconds{aggregator="codex"}[24h]))
231235
series: avg_over_time(head_lag_seconds{aggregator="codex"}[1h]) * 1000 unless (changes(head_lag_seconds{aggregator="codex"}[15m]) == 0 and count_over_time(head_lag_seconds{aggregator="codex"}[15m]) > 5 and on(chain, region) sum by (chain, region) (changes(head_lag_seconds{}[15m])) > 5 and on(aggregator, region) avg_over_time(ws_connected{aggregator="codex", chain=""}[15m]) > 0.8)
232236
live_activity: sum(changes(head_lag_seconds{aggregator="codex"}[15m]))
@@ -250,7 +254,7 @@ providers:
250254
p90: avg by (aggregator) (quantile_over_time(0.90, head_lag_seconds{aggregator="geckoterminal"}[24h])) * 1000
251255
p99: avg by (aggregator) (quantile_over_time(0.99, head_lag_seconds{aggregator="geckoterminal"}[24h])) * 1000
252256
mean: avg by (aggregator) (avg_over_time(head_lag_seconds{aggregator="geckoterminal"}[24h])) * 1000
253-
success: clamp_max(avg by (aggregator) (count_over_time(head_lag_seconds{aggregator="geckoterminal"}[24h]) / 5760), 1)
257+
success: clamp_max(avg by (aggregator) (count_over_time(head_lag_seconds{aggregator="geckoterminal"}[24h]) / 2880), 1)
254258
sample_size: sum(count_over_time(head_lag_seconds{aggregator="geckoterminal"}[24h]))
255259
series: avg_over_time(head_lag_seconds{aggregator="geckoterminal"}[1h]) * 1000 unless (changes(head_lag_seconds{aggregator="geckoterminal"}[15m]) == 0 and count_over_time(head_lag_seconds{aggregator="geckoterminal"}[15m]) > 5 and on(chain, region) sum by (chain, region) (changes(head_lag_seconds{}[15m])) > 5)
256260
live_activity: sum(changes(head_lag_seconds{aggregator="geckoterminal"}[15m]))

harnesses/aggregator-head-lag/cmd/script/race.go

Lines changed: 12 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
package main
22

33
import (
4+
"math"
45
"sort"
56
"sync"
67
"time"
@@ -94,6 +95,15 @@ var race = &raceBook{
9495
participants: map[string]map[string]bool{},
9596
}
9697

98+
// raceLagSeconds floors the published lag at 1 ms. The first feed is 0 by
99+
// definition, but the site drops rows whose headline reads exactly zero
100+
// (a zero p50 means "no measurement" everywhere else), which hid the
101+
// leader on the Singapore tab where Mobula won every race. One
102+
// millisecond is below the tie window and reads as "first".
103+
func raceLagSeconds(d time.Duration) float64 {
104+
return math.Max(d.Seconds(), 0.001)
105+
}
106+
97107
func raceKey(chain, region, hash string) string {
98108
return chain + "|" + region + "|" + hash
99109
}
@@ -122,7 +132,7 @@ func (b *raceBook) observe(aggregator, chain, region, hash string, at time.Time,
122132
// Late arrival: it lost, and its lag is against the recorded first.
123133
b.note(chain, region, aggregator)
124134
if raceChains[chain] {
125-
RecordHeadLag(aggregator, chain, lagBlocks, at.Sub(e.t0).Seconds(), region, hash)
135+
RecordHeadLag(aggregator, chain, lagBlocks, raceLagSeconds(at.Sub(e.t0)), region, hash)
126136
}
127137
return
128138
}
@@ -196,7 +206,7 @@ func (b *raceBook) closeLocked(e *raceEntry, k string) {
196206
headLagFirst.WithLabelValues(o.aggregator, e.chain, e.region).Inc()
197207
}
198208
if o.aggregator != "reference" && raceChains[e.chain] {
199-
RecordHeadLag(o.aggregator, e.chain, o.lagBlocks, delta.Seconds(), e.region, hash)
209+
RecordHeadLag(o.aggregator, e.chain, o.lagBlocks, raceLagSeconds(delta), e.region, hash)
200210
}
201211
}
202212
if !refSeen {

src/components/benchmark-body.tsx

Lines changed: 22 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
"use client";
22

33
import { useSearchParams } from "next/navigation";
4-
import { useCallback, useEffect, useMemo, useState } from "react";
4+
import { useCallback, useEffect, useMemo, useRef, useState } from "react";
55
import type { Benchmark } from "@/types/benchmark";
66
import { liveResults } from "@/lib/provider-filters";
77
import { matchesChainSlug } from "@/lib/chain-aliases";
@@ -304,6 +304,27 @@ export function BenchmarkBody({
304304
const fallbackKind = kindOptions[0]?.value ?? null;
305305
const fallbackVenue = venueOptions[0]?.value ?? null;
306306
const effectiveChain = chainOptions.length > 0 ? (chain ?? fallbackChain) : null;
307+
308+
// Per-chain default view (spec chart.default_panel_by_chain). Applied
309+
// when the reader lands without ?view= and each time the chain tab
310+
// changes, so Solana on aggregator-head-lag opens on First to report
311+
// while Base keeps the head lag headline. A ?view= in the URL wins on
312+
// first paint; after that the chain tab drives it like a fresh visit.
313+
const defaultPanelByChain = Object.values(variants)[0]?.chart?.defaultPanelByChain;
314+
const urlViewPinned = useRef(Boolean(urlView));
315+
useEffect(() => {
316+
if (!defaultPanelByChain) return;
317+
if (urlViewPinned.current) {
318+
urlViewPinned.current = false;
319+
return;
320+
}
321+
const want = effectiveChain ? defaultPanelByChain[effectiveChain] ?? null : null;
322+
const seedPanels = Object.values(variants)[0]?.metricPanels ?? [];
323+
const valid = want && seedPanels.some((p) => p.id === want) ? want : null;
324+
// eslint-disable-next-line react-hooks/set-state-in-effect
325+
setActivePanelId(valid);
326+
// eslint-disable-next-line react-hooks/exhaustive-deps
327+
}, [effectiveChain]);
307328
const effectiveRegion = regionOptions.length > 0 ? (region ?? fallbackRegion) : null;
308329
const effectiveKind = kindOptions.length > 0 ? (kind ?? fallbackKind) : null;
309330
const effectiveVenue = venueOptions.length > 0 ? (venue ?? fallbackVenue) : null;

src/components/ledger-table.tsx

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -157,9 +157,12 @@ export function LedgerTable({
157157
}
158158
return colValue(r, col);
159159
};
160+
// A daily-cadence bench relabels the base window ("last closed day"
161+
// instead of "24h") via the spec's chart.window_label.
162+
const baseWindowLabel = benchmark.chart?.windowLabel ?? "24h";
160163
const colLabel = (col: LedgerColumn): string => {
161164
if (!hasWindows) return col.label;
162-
const w = windowKey !== "24h" && col.windows?.[windowKey] ? windowKey : "24h";
165+
const w = windowKey !== "24h" && col.windows?.[windowKey] ? windowKey : baseWindowLabel;
163166
return `${col.label} (${w})`;
164167
};
165168

@@ -361,6 +364,7 @@ export function LedgerTable({
361364
<button
362365
key={w}
363366
type="button"
367+
title={w === "24h" && baseWindowLabel !== "24h" ? baseWindowLabel : undefined}
364368
onClick={() => setWindowKey(w)}
365369
className={[
366370
"rounded px-2.5 py-1 text-[11px] font-sans tabular uppercase tracking-[0.1em] font-medium transition-colors",
@@ -369,7 +373,7 @@ export function LedgerTable({
369373
: "text-ink-muted hover:text-ink hover:bg-paper-soft",
370374
].join(" ")}
371375
>
372-
{w}
376+
{w === "24h" && baseWindowLabel !== "24h" ? baseWindowLabel : w}
373377
</button>
374378
))}
375379
</div>

src/components/time-series-chart/index.tsx

Lines changed: 15 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -119,7 +119,14 @@ export function TimeSeriesChart({
119119
longRangeDisabledTitle,
120120
onAvailableRangesChange,
121121
}: Props) {
122-
const [rangeLocal, setRangeLocal] = useState<Range>("24h");
122+
// Daily-cadence benches (spec `chart` block) hide sub-day ranges,
123+
// which would show one point at "now", and open on a longer window.
124+
const minRange: Range = benchmark.chart?.minRange ?? "1h";
125+
const rangeOrder: Range[] = ["1h", "6h", "24h", "7d", "30d", "90d", "180d", "1y", "all"];
126+
const rangeAllowed = (r: Range) => rangeOrder.indexOf(r) >= rangeOrder.indexOf(minRange);
127+
const [rangeLocal, setRangeLocal] = useState<Range>(
128+
benchmark.chart?.defaultRange ?? (rangeAllowed("24h") ? "24h" : minRange),
129+
);
123130
const range = rangeProp ?? rangeLocal;
124131
const setRange = (next: Range) => {
125132
if (onRangeChange) onRangeChange(next);
@@ -413,12 +420,14 @@ export function TimeSeriesChart({
413420
useEffect(() => {
414421
if (panelActive) return;
415422
/* eslint-disable react-hooks/set-state-in-effect */
416-
if (range === "7d" && lazySeries7d !== null && !seriesHasData(lazySeries7d)) setRange("24h");
417-
if (range === "30d" && lazySeries30d !== null && !seriesHasData(lazySeries30d)) setRange("24h");
423+
const base: Range = rangeAllowed("24h") ? "24h" : minRange;
424+
if (range === "7d" && lazySeries7d !== null && !seriesHasData(lazySeries7d) && base !== "7d") setRange(base);
425+
if (range === "30d" && lazySeries30d !== null && !seriesHasData(lazySeries30d) && base !== "30d")
426+
setRange(has7d && rangeAllowed("7d") ? "7d" : base);
418427
if (range === "90d" && lazySeries90d !== null && Object.keys(lazySeries90d).length === 0)
419-
setRange(has30d ? "30d" : "24h");
428+
setRange(has30d ? "30d" : base);
420429
if (range === "1y" && lazySeries1y !== null && Object.keys(lazySeries1y).length === 0)
421-
setRange(has90d ? "90d" : "24h");
430+
setRange(has90d ? "90d" : base);
422431
/* eslint-enable react-hooks/set-state-in-effect */
423432
// eslint-disable-next-line react-hooks/exhaustive-deps
424433
}, [range, panelActive, lazySeries7d, lazySeries30d, lazySeries90d, lazySeries1y]);
@@ -621,6 +630,7 @@ export function TimeSeriesChart({
621630
<div className="mb-4 flex flex-wrap items-center justify-between gap-3" data-chart-export-omit="true">
622631
<div className="flex flex-wrap items-center gap-1">
623632
{RANGES.map((r) => {
633+
if (!rangeAllowed(r)) return null;
624634
// Hide entirely once we have confirmed there's no data for this range.
625635
if (r === "7d" && lazySeries7d !== null && !has7d) return null;
626636
if (r === "30d" && lazySeries30d !== null && !has30d) return null;

src/lib/materialize/load.ts

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,24 @@ import {
3131
const SPECS_DIR =
3232
process.env.OCB_SPECS_DIR ?? path.join(process.cwd(), "benchmarks");
3333

34+
/** Camel-cased copy of the spec's optional `chart` block. */
35+
export function chartFromSpec(spec: {
36+
chart?: {
37+
min_range?: "24h" | "7d" | "30d";
38+
default_range?: "24h" | "7d" | "30d" | "90d" | "1y";
39+
window_label?: string;
40+
default_panel_by_chain?: Record<string, string>;
41+
};
42+
}): Benchmark["chart"] {
43+
if (!spec.chart) return undefined;
44+
return {
45+
minRange: spec.chart.min_range,
46+
defaultRange: spec.chart.default_range,
47+
windowLabel: spec.chart.window_label,
48+
defaultPanelByChain: spec.chart.default_panel_by_chain,
49+
};
50+
}
51+
3452
/** Side-effect hooks injected by the caller (site: KV snapshot write;
3553
* worker: store publish). Keeps this module free of persistence deps. */
3654
export type LoadHooks = {
@@ -139,6 +157,7 @@ export function buildEditorial(
139157
panelMainLabel: spec.panel_main_label,
140158
panelMainDescription: spec.panel_main_description,
141159
stackedShare: spec.stacked_share,
160+
chart: chartFromSpec(spec),
142161
unit: spec.unit,
143162
higherIsBetter: spec.higher_is_better,
144163
abstract: spec.abstract,

src/lib/spec-schema.ts

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -467,6 +467,33 @@ export const SpecSchema = z
467467
* where the relative contribution of each venue over time is the
468468
* primary signal. */
469469
stacked_share: z.boolean().optional(),
470+
/**
471+
* Chart cadence for benches whose gauges move once a day (closed
472+
* UTC-day series such as perp-daily-volume). Sub-day ranges (1h, 6h,
473+
* 24h) would render a single point at "now", so `min_range` hides
474+
* them and `default_range` opens the chart on a window that holds
475+
* data. `window_label` replaces the "24h" suffix on the ledger's
476+
* headline columns ("last closed day").
477+
*/
478+
chart: z
479+
.object({
480+
min_range: z.enum(["24h", "7d", "30d"]).optional(),
481+
default_range: z.enum(["24h", "7d", "30d", "90d", "1y"]).optional(),
482+
window_label: z.string().min(1).max(24).optional(),
483+
/** Metric panel to open on by chain (dimension value -> panel id).
484+
* aggregator-head-lag opens Solana on "first_share": the headline
485+
* there is the lag behind the first feed, so the leader reads 0 and
486+
* the share of trades each feed reported first is the view that
487+
* answers the question. Ignored when the URL carries ?view=. */
488+
default_panel_by_chain: z
489+
.record(
490+
z.string().regex(/^[A-Za-z0-9][A-Za-z0-9_-]{0,63}$/),
491+
z.string().regex(/^[a-z][a-z0-9_]*$/),
492+
)
493+
.optional(),
494+
})
495+
.strict()
496+
.optional(),
470497
metric_panels: z
471498
.array(
472499
z.object({

src/lib/spec.ts

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@ import type { Spec } from "@/lib/spec-schema";
2020
import { canonicalChainSlug } from "@/lib/chain-aliases";
2121
import { renderBenchmarkText } from "@/lib/bench-template";
2222
import {
23+
chartFromSpec,
2324
draftPlaceholderForSpec,
2425
filterSig,
2526
loadSpecsUncached,
@@ -137,6 +138,7 @@ export function overlayEditorial(stored: Benchmark, spec: Spec): Benchmark {
137138
// worker running a divergent branch (keyed-rpc-robinhood: main is
138139
// Singapore-only while dev/worker carries the region dims).
139140
dimensions: spec.dimensions,
141+
chart: chartFromSpec(spec) ?? stored.chart,
140142
// Same source-of-truth rule as dimensions: the live YAML decides
141143
// the aggregate pin, never the snapshot.
142144
aggregateFilters: spec.aggregate_filters,

src/types/benchmark.ts

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -335,6 +335,16 @@ export type Benchmark = {
335335
panelMainLabel?: string;
336336
/** Tooltip of the headline tab in the panel switcher. */
337337
panelMainDescription?: string;
338+
/** Chart cadence hints from the spec's `chart` block: hide sub-day
339+
* ranges, open on a longer window, relabel the "24h" ledger suffix.
340+
* Set for benches whose gauges move once per UTC day. */
341+
chart?: {
342+
minRange?: "24h" | "7d" | "30d";
343+
defaultRange?: "24h" | "7d" | "30d" | "90d" | "1y";
344+
windowLabel?: string;
345+
/** Chain dimension value -> metric panel id to open on. */
346+
defaultPanelByChain?: Record<string, string>;
347+
};
338348
/** When true the bench page renders a stacked bar chart (absolute + %
339349
* share over time) below the main chart. Set via `stacked_share: true`
340350
* in the bench YAML. Meaningful only for volume-share benches. */

0 commit comments

Comments
 (0)