Measurement robustness - #108
Merged
Merged
Conversation
Four defects in the resampling path, three of them observable from the
public API.
Duplicate thresholds produced confidences outside [0,1]. The counting
loop iterated the caller's list rather than its distinct values, so a
threshold listed n times scored n hits per replicate: []float64{0.2,
0.2, 0.2} reported a confidence of 3. Thresholds are now deduplicated,
and the counters are indexed by position instead of keyed by float64,
which also lifts an overflow above 2^32 resamples.
CompareSamples sorted the caller's slice in place. Deduplicating
requires a copy anyway, so both go together.
NaN thresholds were answered with a silent zero. A NaN map key can be
written but never read back, and zero is an ordinary confidence value,
so an invalid input was indistinguishable from a real result.
CompareSamples now rejects NaN with an error naming the caller's index,
and BootstrapConfidence, which has no error channel, skips them.
Infinities are kept: they are degenerate but consistent, and are now
documented as such.
The unseeded path built a CPRNG per bootstrap sample, filling an 8 KiB
buffer from crypto/rand to consume about 400 bytes of it. Both paths now
draw every replicate from one generator created before the loop. For 101
measurements at 5000 resamples this takes the call from 44.97 ms and
87.0 MB to 15.18 ms and 8.6 MB.
Reseeding a DPRNG per replicate from consecutive seeds also left a
measurable trace, xorshift64 being linear over GF(2): a lag-1
correlation of 0.095 between the first index of consecutive replicates
against a noise band of 0.007, and a 2.26% repeat rate where 0.99% was
expected. It never reached the confidence estimates, which depend on a
median, but one stream is the sounder construction and costs nothing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
calcMinTimeSample probed the clock ten million times unconditionally, costing 764 ms on the first call in a process. The quantity it searches for has a hard floor, so it is reached almost at once and further probing cannot go below it: the minimum was already final after 1,000 probes and unchanged through all ten million. A smaller fixed count would be a guess about platforms this was not measured on, and Windows in particular reaches SampleTime through a LazyProc call rather than a vDSO. The search now stops after 50,000 consecutive probes without improvement and keeps the ten million as a ceiling, which adapts instead of assuming. First call: 764 ms to 4.15 ms, same result. If a rare minimum ever needed a longer run to appear, the value returned would be slightly too large, which sizes batches conservatively rather than wrongly. GetSampleTimePrecision is documented more precisely. It returns the smallest interval measurable with SampleTime, which is bounded by whichever is larger, the clock's tick or the cost of the two calls around it; which dominates is platform dependent. Also notes that SampleTime uses QueryPerformanceCounter on Windows, at a 100 ns tick, and is unaffected by the coarse 15.6 ms system clock. Two tests depended on execution order through the sync.Once guarding that computation, one needing it unfired and the other needing it fired. Both now arrange their own precondition. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CompareSamples takes two sets of measurements and says nothing about how to obtain them, which leaves the parts that systematically bias a comparison to each caller. An A/A experiment, the same code measured as both candidates, showed what that costs: measuring A before B in every repeat reported a mean confidence of 0.608 for "A is faster than B" where an unbiased harness must report 0.5. Collect owns the measurement loop and the things worth getting right in it. Order defaults to ABBA interleaving, which brought that same A/A experiment to 0.473; deciding the order happens before the clock is read and so costs nothing. Candidate carries optional Setup and Teardown that run outside the measured region, so per-batch preparation is not charged to the code under test. GCBetween and DisableGC place collection deterministically, together roughly halving the spread of an allocating candidate. Batches are func(n uint64) rather than a per-operation callback. The candidate owns its inner loop, so the compiler's optimization boundary sits where it would in production; the indirect call costs about 0.01 ns/op amortized and is paid identically by both candidates. CalibrateInnerLoops sizes batches so the clock contributes at most a requested share of relative error. Since that share is precision divided by batch duration, it depends only on how long a batch runs, so an expensive operation can calibrate to a batch of two while a cheap one needs thirteen thousand; measured batch durations stayed near 49 us across four orders of magnitude of operation cost. This is what makes differences below the clock's resolution recoverable: a per-operation difference of 1.89 ns was measured to within 0.07 percentage points against a 41 ns floor. Two limits are documented rather than papered over. Fixed per-operation overhead in the batch body attenuates the result, and subtracting an empty-loop baseline does not repair it because the compiler optimizes an empty loop differently. And ordering does not remove the noise floor: A/A runs of identical code still differed by 0.6% to 0.8%, so a result below roughly 1% is not resolved regardless of resample count. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Bootstrap resampling cannot say how much of an apparent difference a measurement setup invented. It quantifies how far the estimate would move if the same measurements were drawn again, which leaves a bias affecting all of them invisible, and reports a tight confidence around it. Measured on identical code, this package has seen apparent differences of 0.6% to 0.8% carried with high confidence. ValidateHarness runs a candidate against itself through Collect, repeatedly, under the options a real comparison would use. Every difference it finds is an artefact by construction, so the largest one is the floor below which that setup cannot tell a real difference from its own noise. Resolves reports whether a given result clears it. The calibration figure needed care. Confidence at threshold zero asks whether delta >= 0, so tied medians count as "A at least as fast", and timing measurements tie constantly: an unremarkable A/A run produced 102 samples holding 35 distinct values and tied in 17.8% of replicates. Left alone, the mean confidence sat at 0.551 where an unbiased setup must give 0.5, which would have had the API report bias on every quantized measurement it ever saw. Splitting ties removes it exactly, the offset matching half the tie rate to three decimals, and the mean then lands on 0.496 across even and odd repeat counts and both interleaving strategies. Ties are split through an identity available from the public API alone: with confAB = P(medA<medB) + P(tie) and confBA = P(medA>medB) + P(tie), (confAB + 1 - confBA)/2 gives P(medA<medB) + P(tie)/2 and confAB + confBA - 1 gives the tie rate, which is reported as well. A high tie rate says the measurement is too coarse for the question being asked. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
TestCPRNG_vs_DPRNG_Performance required DPRNG to lead CPRNG by at least 33.333% at 95% confidence. That is a claim about a particular machine rather than about the code, and this one disagrees: DPRNG leads by about 24% here, so the test failed for reporting the truth. It also measured CPRNG before DPRNG in every repeat, the ordering that an A/A experiment shows reports a mean confidence of 0.696 where 0.5 is correct. It now measures through Collect, which interleaves, and takes its threshold from ValidateHarness: the largest difference the setup demonstrably produces from identical code. Anything above that floor is a real claim, and the magnitude stays a property of the machine rather than something pinned in source. TestUInt32N_CompareToModulo compared two bootstrap confidences with != and failed when they differed. Both are fractions of 10,000 replicates, so one replicate landing differently makes them differ by 0.0001, which is what happened; it says nothing about either reduction. The claim under test is that neither beats the other by 7%, so both one-sided confidences must now simply be near zero. A genuine advantage would push one of them toward 1 and is still caught. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The TieRate field was added with a one-line note saying a longer batch or more repeats would buy resolution. Half of that was wrong, and it was never measured. Only the batch length matters. One measurement is an integer count of clock ticks divided by the batch size, so its granularity is precision/InnerLoops; raising InnerLoops makes each value finer and ties correspondingly rarer. Holding repeats at 51 and varying only the batch size took the tie rate from 86.3% at 1,000 operations to 0.0% at 400,000. Varying only the repeat count at a fixed batch size gave 2.2%, 2.3%, 2.8% and 1.8% for 21, 51, 101 and 201 repeats: no trend at all, because more repeats draw more values from the same coarse set. Expressed through the knob callers actually set, MaxQuantizationError, the default target of 0.001 leaves about 15% of replicates tied, and tightening it tenfold takes that to 0.6% at ten times the batch length. The default is sized for the accuracy of a difference's magnitude, where it does well, and not for the separate question of whether a difference exists at all; that distinction is now stated where the default is defined. CompareSamples gains the warning at the place a caller meets the problem: a threshold of 0.0 asks whether delta >= 0, which is "at least as fast" rather than "faster", and ties count towards it. Two claims of my own are corrected while here. The note about GetSampleTimePrecision costing 755 ms was left behind by the change that made that probe adaptive; it is about 4 ms now. And DefaultRepeats was justified as odd to keep the median from interpolating, which this package's Median never does: it returns an element either way, and the real reason to prefer odd is that an even count returns the upper of the two middle values. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The order calibration table in the Order documentation does not reproduce. It reported mean A/A confidences of 0.608 for Sequential, 0.473 for ABBA and 0.526 for Random, and presented Sequential as measurably biased on that basis. Re-measuring with 40 runs per order, and with the tie-split confidence that a later commit showed is needed for the figure to mean anything, gives 0.487, 0.475 and 0.517, each within one standard error of about 0.04 of the 0.5 an unbiased setup must produce. The original table rested on 25 runs, where the standard error is roughly 0.05, so it was never able to support the difference it was used to assert. Attempts to demonstrate the mechanism directly, by injecting a known drift across a run, did not produce a clean signal either, and taught something worth recording: a gradual trend contributes to an order effect only in proportion to how much the machine changes between two adjacent batches, which for a smooth drift across a whole run is a very small step. Interleaving stays the default, on the honest grounds rather than the invented ones: it removes a mechanism that is real in principle, its balance is exact rather than statistical, and it costs nothing because the order is chosen before the clock is read. The documentation now says that, points at ValidateHarness for finding out what a given machine actually does, and the test that carried the same claim in its name no longer implies an effect it does not check. Several numbers are corrected while auditing the rest. The attenuation example mixed figures from two different runs, quoting 34% alongside an overhead and workload measured in a run that gave 35%, and said the baseline correction recovered 2 of 16 missing points where it was 2 of 15. The tie example likewise mixed a 14.4% single run with a 17.8% average over 40. A sub-nanosecond recovery quoted as 0.06 percentage points was 0.07. Claims about Linux and Windows clock behaviour are now marked as coming from documentation rather than from measurements taken here, since neither platform was available. And the noise floor is given as the range actually observed rather than the narrower 0.6% to 0.8% that one early experiment happened to produce. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Bootstrap resampling treats the samples as an unordered bag. It asks how much the estimate would move if they were drawn again and discards the order they arrived in, so a machine that grew steadily slower during a run leaves no trace in its output. That is the situation in which a comparison misleads most, because the drift is charged to whichever candidate was measured later. DetectDrift correlates each sample with its position using Spearman's rank correlation, standardized as rho*sqrt(N-1), which is approximately standard normal when the samples are exchangeable. Ranks rather than values, because one preempted batch is an enormous outlier and would otherwise dominate; tied values take their average rank, which matters because quantized timings tie constantly. Significance and effect size are reported separately: a long run resolves a drift of a fraction of a percent as highly significant, which is worth knowing and may be far too small to matter. The false positive rate was measured rather than assumed. On synthetic series collapsed onto 3, 8 and 35 distinct values it fired at 4.9%, 3.9% and 3.9% against a nominal 5%, so ties do not break it. On real timing series with their order permuted, which preserves the values exactly while destroying any trend, it fired at 4.8% over 1200 permutations. Left in measurement order those same series tripped it in 20% of 60 runs, about three standard errors above what shuffling gave, and their mean lag-1 autocorrelation was +0.06, also about three standard errors above zero. Ordinary measurement series do carry order structure, and it is not an artefact of how their values are distributed. The test does not separate a slow trend from short-range correlation, and says so. ValidateHarness now reports DriftRate and MedianDriftShift alongside the noise floor, since whether the machine held still is part of what an A/A experiment is for. One correction to my own test while building this: the claim that a split-half shift equals half a linear trend is only a first-order approximation. Exactly, at 101 samples, it is 0.51*d/(1 + 0.25*d), which is 1.5% off at d = 0.02 and 9.3% off at d = 0.50. The test now stays inside the range where the approximation holds and records the exact relation. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The rate at which real measurement series trip the drift test was reported as 20% from 60 series. It is 13.0%, with a 95% interval of [10.3%, 15.7%], measured over 600. The original figure carried a standard error of 5.2 points and was never able to pin down a rate; it sat one and a half standard errors from the truth. Two things that went unchecked the first time are now settled. The permutations used for the null rate come in groups drawn from the same series and are not independent by construction, so a binomial interval over them could have been optimistic; the cluster-robust interval turns out to match it, the design effect being 0.95. And the null rate itself is 4.80% over 6000 permutations, a 95% interval of [4.27%, 5.33%] that covers the nominal 5%, where the earlier 1200 permutations gave a much looser bound. The lag-1 autocorrelation was quoted as +0.06 at about three standard errors from zero, using a standard deviation assumed from the null rather than measured. Measured, the spread is 18% wider than that assumption, and over 600 series the mean is +0.083 with a 95% interval of [+0.069, +0.096]. The conclusion is unchanged and considerably firmer: the gap between ordered and permuted series is 8.2 percentage points at z = 5.9. Only the numbers were wrong. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Resampling single observations assumes they are exchangeable. Real timing measurements are not: 600 series on this machine carried a mean lag-1 autocorrelation of +0.10. Correlated samples hold less information than the same number of independent ones, so a method assuming independence can end up more confident than the data warrant. How much that costs was measured before anything was built, on A/A simulations of an AR(1) process where identical inputs mean every reported difference is a false signal and 10% of runs should land outside a 90% band. At rho 0.08 the rate was 10.0%, at 0.2 it was 13.5%, at 0.4 21.7%, at 0.6 33.1%. So the effect is real but only above about 0.2, and the observed mean of +0.10 sits below that. Whether real series reach it needed the spread, not the mean. The spread observed across series was 0.155, but the lag-1 estimator itself has a standard deviation near 1/sqrt(n), here 0.14; removing that leaves a true spread between series of about 0.066. On that distribution roughly 6% of series are genuinely above 0.2 and almost none above 0.3, which works out to a fraction of a percentage point of aggregate inflation. On this machine blocks would buy nothing, and the documentation says so. They are provided because other environments are not this machine, and because they were verified to work: at rho 0.2 the moving block bootstrap restores the nominal rate, 12.8% to 10.9%. It is a partial remedy beyond that, 20.6% to 12.7% at rho 0.4 and 32.2% to 16.7% at 0.6, which is what fixed-length blocks can do; at that point the measurement is the problem rather than the statistics. Blocks longer than the dependence present are themselves mildly over-dispersed, so the automatic length of round(n^(1/3)) is worth preferring to a guess. BootstrapConfidence and BlockBootstrapConfidence now share an implementation, since a block length of one is single-observation resampling. That the shared path draws exactly the same values in the same order was checked against reference outputs captured before the change: nine seed and size combinations, identical to six decimals. ValidateHarness reports the autocorrelation it observed, so the decision about whether any of this is needed can be made from measurement rather than from assumption. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The example still drove the measurement by hand, with its own loop, its own GC calls and a hardcoded inner loop count, which is exactly the arrangement Collect was added to replace. It now uses the current API, and it does something more useful than demonstrate the happy path: it validates the harness against each candidate separately, and finds that the two are not equally well behaved. That was not staged. Median allocates a copy on every call, and the GC pressure that produces gives its measurements a lag-1 autocorrelation around +0.35, a noise floor near 8% and a trend in nine runs out of ten. QuickMedian, in the same harness, sits at +0.04 and under 1%. So the example compares against the worse of the two floors, and picks block resampling because the measured autocorrelation is past the point where ordinary resampling starts overstating confidence. Every piece of the API earns its place on a case that was not constructed for it. The README described a library that has since grown a measurement side, and its quickstart showed the hand-rolled loop. It now covers Collect, calibration, ValidateHarness, DetectDrift and the block bootstrap, and carries a section on the two things a confidence figure cannot tell you: that what is measured is the loop rather than the function, and that a result below the harness noise floor has resolved nothing however confident it looks. One claim is withdrawn, that the package yields confidence intervals; it reports probabilities that a threshold is met, which is not the same thing and is now said so explicitly. Also: gofmt on rtcompare_test.go, which had drifted, and DPRNG.UInt32N is renamed to Uint32N to match CPRNG and the Go convention that math/rand/v2 follows. The old spelling remains as a deprecated wrapper that delegates, so existing callers keep compiling, with a test pinning the two to each other. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two questions were missing an answer. CompareSamples takes thresholds and reports how confident one can be that each is met, which is right when a threshold is given and useless when none is. EstimateDifference reports how large the difference appears to be and how precisely that is known, so that the honest answer can be "somewhere between 2% and 19%". The interval is a percentile bootstrap, and its coverage was measured rather than assumed: 96 to 97% against a nominal 95%, over 2000 trials per cell across normal, lognormal and one-sidedly contaminated inputs at four sample sizes. It is conservative rather than optimistic, the misses splitting evenly at about 1.8% per side against a nominal 2.5%, and it narrows towards nominal only slowly. The cause is discreteness: a resampled median can only take values present in the sample, so its bootstrap distribution is coarser than its true one. I had written a plausible-looking coverage table into the documentation before measuring it, and every number in it was wrong, including the direction. The one-sided-noise argument for preferring a low quantile to the median does not survive measurement either. Simulated against a known difference with lognormal noise, the median wins on RMSE up to roughly 30% disturbed batches, because with contamination on fewer than half the samples the middle one is drawn from the clean part; a low quantile is merely noisier, and the minimum is worse still at every rate. Past 40% the median degrades sharply, but that regime announces itself: the A/A noise floor rises from 1.2% to 20.5% across the same range, so the measurement is visibly worthless before the estimator becomes the problem. No knob was added; the choice is now documented with what it rests on. Extracting the shared delta computation surfaced a defect in it. The epsilon guard, whose stated aim was to keep the result finite for a denominator near zero, could never do so: the test |b| < |b|*1e-12 is false for every non-zero b, so the branch never fired, and for b exactly zero it substituted a denormal that overflowed the division anyway. The guard is removed and the infinity reported plainly, which is both the honest answer and a useful signal, since a median measurement of zero means a batch fitted inside one clock tick. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Four items from a review of the branch. BlockBootstrapConfidence silently produced a confidence of exactly 0 or 1 for any block length at or above the input size, and for any negative one. A block as long as the input has a single start position, so every replicate reproduced the input, the resampled difference was constant, and the result read as certainty. Measured on A/A data where 0.5 is the answer, lengths of 101, 200 and -5 at n=101 all returned 0.0000. Block lengths are now clamped to half the input, which guarantees at least two blocks per replicate, and negative lengths take the same route as zero and select the automatic length. A block length of one still draws exactly what it drew before, so BootstrapConfidence is unchanged. HarnessValidation.NoiseFloor was the maximum of the observed A/A differences, which has no population value to converge on and grows with Runs. Measured over eight repetitions per row, it climbed from 0.208% at ten runs to 0.356% at eighty while the 90th percentile settled at 0.166%; across repetitions at eighty runs the maximum ranged over 0.168% to 0.834% and the quantile over 0.164% to 0.168%. Since Resolves compares against this number, validating more carefully raised the bar rather than informing it. NoiseFloor is now the NoiseFloorQuantile quantile, MaxObservedNoise keeps the old figure, and the documentation says plainly that the floor is no longer a bound: about one A/A run in ten exceeds it. DefaultValidationRuns goes from 10 to 40. FalseSignalRate and DriftRate are proportions over Runs observations, and at ten runs the standard error is 9.5 percentage points against a nominal 10%, so a calibrated setup would print 0.0% about a third of the time. Forty runs bring that to 4.7 points and cost 3.03 s against 0.76 s. The central claim of the package, that a per-operation difference far below one clock tick is recovered with the right magnitude, had no test. The two candidates in the new one share a loop body over n and 2n units, so the slower measured region is exactly twice the faster one including loop overhead and attenuation cannot shrink the true 0.5. It recovers 0.4998 while resolving a 0.99 ns difference with a clock that ticks every 41 ns. Documentation corrections: TieRate is the median across runs and not the mean, and the reason is now given; MedianDriftShift is an absolute value, so String no longer prints a sign it cannot carry; the cost note admits the two resampling passes per run; and the README quickstart validates both candidates, which is what the runnable example already argued for. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Using this package correctly took seven steps and three judgement calls: validate both candidates, take the worse noise floor, collect, test both series for drift, read the autocorrelation, decide whether it warrants block resampling, and only then ask for a confidence or an interval. Every one of those is easy to skip, and skipping any of them produces a confident number with nothing behind it. That is a poor bargain for someone who wants to know whether their optimisation worked. Compare performs all of it and makes the judgement calls from what it measured. Report.Resolved is the short answer, requiring the difference both to exclude zero and to clear the noise floor, and Report.Warnings is the fine print in plain sentences: validation skipped, difference inside the floor, interval spanning zero, a series that drifted, a measurement too coarse for the question, a harness that reports differences between identical code. The rest of the struct is the evidence, so nothing is hidden by the summary. One thing it fixes that the manual sequence got wrong. With InnerLoops left at zero, each ValidateHarness call and the Collect call calibrated separately, so the noise floor could describe a different batch size than the measurement it was meant to qualify. Compare sizes the batches once, up front, and holds that size fixed for everything that follows. How often does it cry wolf? Measured on identical candidates over 600 runs across three configurations, none resolved. The two conditions catch different things: the interval excluded zero in at most 0.5% of runs, while the difference cleared the noise floor in 4% to 16%, the latter being what a 90th-percentile floor implies. The test asserts this as a rate rather than as a single verdict, since one run is both flakier and weaker than the truth. EstimateDifference gains an internal form taking a block length, so that the interval Compare reports uses the same resampling scheme as its confidences. The exported signature is unchanged and still resamples single observations. The runnable example is rebuilt around the one call and then takes the same measurements apart by hand, which is also where EstimateDifference finally appears in runnable form. The README quickstart shrinks to one call with its output shown. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
TestCompareResolvesARealDifference failed on CI, reporting a point estimate of 56.06% where the construction guarantees 50%. The construction is not at fault: the two candidates share a loop body over n and 2n units, and measured locally at five batch sizes, with and without the atomic coverage instrumentation the CI uses, the ratio came out at 2.000 every time. The runner was simply noisy, and said so — it tied in 31% of replicates and reported an interval of [41.25%, 56.58%], which contains the truth. What was wrong was a fixed tolerance of 0.05 calibrated on an idle laptop. Two changes. The shared test options move from 21 repeats of 3000 inner loops to 51 of 20000, so that the batch is long enough to be worth measuring and the median has enough samples to survive a disturbed batch or two; that addresses the coarseness the tie rate was pointing at. And the magnitude band widens to 0.15, which still catches an inverted or mixed-up comparison while tolerating a shared runner. The tight check on the magnitude was never here anyway: it lives in TestCollectResolvesBelowTheClock, which calibrates the batch properly and judges three runs by the middle one. The false positive rate was re-measured at the new settings rather than assumed to carry over, since it is the property that matters most: over 200 runs of identical candidates none resolved, and the same held over another 200 under atomic coverage. The interval excluded zero in none of them while the difference cleared the noise floor in 15% to 20%, so the interval condition is carrying the gate and dropping it would be caught. Cost to CI is negligible: the whole compare set runs in 2.4 s of a 161 s instrumented suite. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The pieces are documented, but the sequence and the judgment calls between them were not written down anywhere a non-mathematician could follow: size the batches, validate both candidates, collect, check for drift, check autocorrelation, choose a resampling scheme, only then compute a confidence or an interval. Compare now performs that sequence, but its output still needs interpreting — a tie rate, a noise floor, a drift warning, a wide interval each call for a different response, and none of that was written down in one place before. HOWTO.md explains the question the library answers and why it's hard (a noisy machine, a coarse clock), gives the one-call version, walks through what each of Compare's steps does and why, explains attenuation and what "not resolved" does and does not mean, and closes with a troubleshooting table: symptom, what it means, what to do. Every number quoted in it (the 50%-measured-as-35% attenuation example, the 41ns/100ns clock figures, the 0.2 autocorrelation threshold, the quantization-error defaults) is pulled from the existing doc comments rather than restated from memory, and the internal anchor links were checked against GitHub's heading slugs. README links to it twice: once up front for a reader arriving cold, and once right after the Compare example for a reader looking at a warning they don't understand. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
… in HOWTO A reader asked a sharp question: doesn't interleaving the order already wash out drift, so why bother with a separate drift check and an autocorrelation threshold on top? The two protect against different failures — ABBA cancels a trend's effect on which candidate looks faster (a bias question), while autocorrelation is about how much independent information one candidate's own sequence of measurements actually contains (a variance question) — and interleaving the order does nothing about the second, which is why plain resampling stays overconfident under real dependence regardless of the order the batches were taken in. Added that distinction to the autocorrelation section, with the measured number that falsifies the "interleaving already handles it" reading: 21.7% false signals at a lag-1 correlation of 0.4 against a target of 10%, with ABBA already in effect. Also documents that the block-resampling decision is noisier under SkipValidation, where it comes from a single read of the comparison run itself rather than the median across ValidateHarness's several A/A experiments — one more thing SkipValidation trades away besides the noise floor. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
TestCalcMinTimeSample failed on GitHub Actions, reporting 60ns where the test required under 50 on Linux/amd64. GetSampleTimePrecision's own documentation already flagged that 50ns figure honestly: "This has not been measured here; the existing tests expect a value below 50 ns, which is consistent with it." It was an assumption carried into an assertion, not a fact about Linux/amd64 in general, and a shared, virtualized CI runner under coverage instrumentation was exactly the kind of environment where the underlying clock_gettime call plausibly costs more than on a quiet dedicated machine — the "call cost dominates the tick" case the same documentation already anticipated. The fix removes the tight per-OS/arch upper bounds this test and its sibling, TestGetSampleTimePrecisionSetsAndCaches, asserted for everything but Windows, replacing them with the generous, environment-agnostic ceiling the tests already had for "not catastrophically broken" (well under a millisecond). Windows keeps its exact 100ns expectation, because QueryPerformanceCounter's frequency comes from the hardware abstraction layer rather than from call overhead and is a hardware fact rather than a benchmark result — though this project's CI runs on Linux only, so that branch is untested here regardless. GetSampleTimePrecision's doc comment is updated to record the 60ns observation in place of the unmeasured 50ns guess. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This branch's earlier commits added seven new production files and eight new
test files at the repository root, each for a single, focused topic. That is
a reasonable way to write the code, but it took the root from 19 entries to
35, and GitHub renders every one of those as a row above the README on the
repository's front page — so the branch pushed the README most of a screen
further down for no reason a reader would find useful. A folder collapses to
one row regardless of what it holds, but splitting the package into an
internal one plus a façade would be a much larger and riskier change: several
of the new files share unexported helpers across file boundaries (blockSample
and lag1Autocorrelation each in three files, bootstrapConfidence in three
more), so a real package split would mean either exporting a batch of
internals or moving everything at once behind a hand-maintained forwarding
layer. Merging files within the same package needed neither.
Three merges, chosen for which files were already tightly coupled rather than
for an even split:
calibrate.go -> collect.go (CalibrateInnerLoops exists only to
size Collect's batches, and is
already called directly from it)
blockbootstrap.go -> rtcompare.go (BlockBootstrapConfidence and its
machinery are the generalization of
bootstrapConfidence, which already
lived here; blockLength=1 is the
plain bootstrap exactly)
drift.go+estimate.go -> diagnostics.go (the two things Compare consults
besides the main comparison itself)
Tests moved along the same seams, and thresholds_test.go — which tests
CompareSamples/BootstrapConfidence behaviour defined in rtcompare.go, not a
file of its own — joined rtcompare_test.go rather than staying separate.
compare.go and validate.go are left standalone: each is already a large,
single-topic file (Compare's orchestration, ValidateHarness's evidence-heavy
documentation), and merging either into something else would blur two of the
most-referenced concepts in the README and HOWTO without shortening the root
listing by more than one row.
Purely mechanical: import blocks were unioned, nothing was renamed, and the
exported and unexported symbol sets were diffed before and after to confirm
they are identical. Root entries: 19 on main, 35 before this commit, 28 after.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Found while verifying the file consolidation, not caused by it: full-suite
runs under the CI's coverage instrumentation intermittently failed
TestCompareResolvesARealDifference with "noise floor 0 is not a plausible
fraction for identical code" — about 1 run in 8. NoiseFloor is the 90th
percentile of the absolute A/A deltas from ValidationRuns experiments, and
fastCompare set that to 3. At three, the 90th percentile needs only the top
two of three deltas to land on exactly zero, which quantized timing does
routinely — that is not a defect in NoiseFloor, it is too small a sample for
a percentile to mean anything, the same complaint this project has raised
about its own sample sizes elsewhere.
Measured rather than guessed: 200 trials at each candidate count.
ValidationRuns NoiseFloor == 0
3 2.5%
5 0.0%
10 0.0%
15 0.0%
Set to 10, with margin above the smallest value that already measured zero
misses, since a shared CI runner has already shown worse tie rates than this
machine does locally. Cost stays small: validating against a fixed batch size
rather than calibrating, ten runs added about 4 seconds to the whole compare
test file.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Checked the linter config first, as asked: there is none (AGENTS.md already says so — default ruleset, no .golangci.yml), and that default is reasonable for this project. The finding itself is real but the underlying pattern is a known, unavoidable tension rather than a misconfiguration: sink is written to but never read, which is the whole point of it — it exists only so the compiler cannot prove the batch loop's result is unused and optimize it away. Reproduced the asymmetry in an isolated module to understand it rather than guess: the identical write-only pattern in this module's own test files (compareSink, collectSink, validateSink — all written via ^= or += and never read either) is not flagged, while cmd/rtcompare-example/main.go's sink is. The difference is package kind, not code shape: staticcheck's unused check (golangci-lint's `unused`) can see the whole closed program in a package main and correctly prove nothing downstream observes sink's value, whereas for a library package's test files it is deliberately more conservative. Any sink variable in a package main hits this; it is not specific to this file. The fix is the standard one for this exact situation: `_ = sink` after the last write, a read that is immediately discarded. It costs nothing at runtime and changes no output — verified by running the example before and after and diffing the two — and it is enough to tell the linter what the accumulation already told the compiler. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.