fix: Summary quantiles collapsing for targeted quantiles with 2*epsilon >= 1-quantile

[olegkovalenko]

Fixes #2292. Alternative to #2316, addressing the issues raised in its review.

Problem

For targeted quantile configurations with 2*epsilon >= 1 - quantile — e.g. (0.9, 0.05) or (0.99, 0.005), both taken from real-world configurations — Summary reported values from far below the requested quantile, often the minimum of all observations, regardless of the input data.

The root causes all stem from the same property: below a target quantile, the CKMS error function f(r) = 2*epsilon*(n-r)/(1-q) is of order n-r when 2*epsilon >= 1-q. Three things break:

1. compress() destroys the sketch. A single sample may span all ranks from r to n, so compress() merges away the samples that hold the information needed to answer the quantile query. With {(0.9, 0.05), (0.99, 0.005)} the sample list collapsed to 3 samples no matter how many values were inserted.

2. insertBefore() assigns misleading deltas. Freshly inserted samples get delta = f(r) - 1, so below a target their possible-rank intervals are centered near rank n regardless of the sample's actual position — indistinguishable from genuine samples near the target.

3. get() stops too early. The scan stopped at the first sample with r + g + delta > desiredRank + f(desiredRank)/2 and returned the value of the sample before it; a single wide fresh sample (always present, since get() flushes the buffer right before scanning) tripped it far before the target rank.

Fix

1. Sample widths are additionally bounded by maxWidthNotCrossingTargets(r) at both places where widths are created — merging in compress() and delta assignment in insertBefore(), via a shared effectiveMaxWidth(r) — so every target quantile keeps enough resolution around its accuracy window [q*n - eps*n, q*n + eps*n]: below a window a sample may extend at most max(windowStart - r, 2*eps*n) — it can intrude into the window but never reach the window's end — and any sample overlapping a window has width at most the window's size 2*eps*n. So no single sample can span a whole window, and the center of a sample's possible-rank interval is within eps*n of any rank the sample covers inside the window. The bound is anchored at the window's start so it does not degenerate for targets with quantile + epsilon >= 1 (e.g. (0.99, 0.01), (0.95, 0.05)), where the window's end is rank n and an end-anchored bound would be no constraint at all. For configurations with 2*epsilon < 1-quantile the bound is larger than f() near the target, so behavior there is mostly unchanged.

2. get() returns the value of the sample whose possible rank interval [r+g, r+g+delta] is centered closest to the desired rank, which cannot be derailed by a single wide sample.

Relation to #2316 and its review

#2316 diagnoses the compress and get parts but bounds compress() differently (minimum of the error function over the merged interval) and leaves insert-time deltas unbounded. The review found a deterministic counterexample: single quantile (0.99, 0.005), values 1..10,000 shuffled with Random(2) — #2316 returns 9784, outside the allowed [9800, 10000].

This fix passes that case (returns 9940), and it is included as a regression test (testSingleTargetedQuantileSmallN).

On the selection rule in get(): the center of a sample's possible rank interval is its best point estimate of rank, and with the width bound applied at insert and merge time, samples near a target cannot be wide, so the estimate is tight exactly where it matters. This is backed by the verification below rather than a formal proof — happy to discuss if a proof-oriented selection is preferred.

Verification

Verified against exact percentiles on 3,720 test cases (31 quantiles across 13 configurations × 6 distributions — uniform, descending, heavy-tail from a production latency CDF, exponential, lognormal, gaussian — × 2 sizes × 10 seeds): worst rank error 1.75 * epsilon, no case above 2 * epsilon. Before the fix the worst rank error was 330 * epsilon. The evaluation harness (including instrumented variants of the query rule and the width bound that were compared before settling on this fix) is available at https://gist.github.com/olegkovalenko/83c58835a1357d3450e6538f89e2cda7.

Additional targeted sweeps with values 1..n shuffled (true rank = value), pass = rank error <= 2*epsilon, across n ∈ {1k, 10k, 100k} × 50 seeds:

Configuration	before	#2316	this PR
Review counterexample (0.99, 0.005), n=10k, seed 2	1.0 (min)	9784 ✗	9940 ✓
(0.99, 0.005) × 150	144 fail	2 fail	0 fail (worst 0.87ε)
(0.9, 0.05) + (0.99, 0.005) × 300	294 fail	5 fail	0 fail (worst 0.87ε)
(0.5, 0.05) + (0.9, 0.01) + (0.99, 0.001) × 450	0 fail	0 fail	0 fail (worst 1.17ε)
(0.9, 0.06) (strictly above boundary) × 150	150 fail	11 fail	0 fail (worst 1.30ε)
(0.99, 0.01) (window end = n) × 150	150 fail	90 fail	0 fail (worst 1.20ε)
(0.95, 0.05) (window end = n) × 150	150 fail	80 fail	0 fail (worst 1.00ε)

Memory impact of the width bound is a handful of extra samples on the affected configurations (e.g. (0.99, 0.005): 3–4 → 6–11 samples after 1M inserts); well-behaved configurations such as (0.5, 0.05)(0.9, 0.01)(0.99, 0.001) are unchanged (37–40 samples).

Tests

• testTargetedQuantilesDoNotCollapse — the original reproducer from Summary quantiles collapse to the minimum observation when 2·epsilon ≥ 1−quantile #2292
• testSingleTargetedQuantileDoesNotCollapse — single targeted quantile at the boundary
• testTargetedQuantilesWithMedian — collapse still occurred with a well-behaved quantile added
• testSingleTargetedQuantileSmallN — the deterministic small-n case from the fix: prevent Summary quantiles from collapsing to the minimum observation #2316 review
• testTargetedQuantileWindowReachingMaximum — the degenerate quantile + epsilon >= 1 family
• testTargetedQuantilesDescendingInput — descending input order, the worst case for these configurations

All assert the 2*epsilon rank bound via validateResults on values 1..n inserted in the respective order (value = true rank). All existing CKMSQuantilesTest cases pass unchanged (22 tests), as does the full prometheus-metrics-core suite (162 tests).