diff --git a/benchmark/README.md b/benchmark/README.md index 2f52a44f251a..852439f61816 100644 --- a/benchmark/README.md +++ b/benchmark/README.md @@ -10,6 +10,7 @@ directory, see [the guide on benchmarks](../doc/contributing/writing-and-running ## Table of Contents * [File tree structure](#file-tree-structure) +* [`node:bench` evaluation tools](#nodebench-evaluation-tools) * [Common API](#common-api) ## File tree structure @@ -44,15 +45,91 @@ directories. * `common.js`: see [Common API](#common-api). * `compare.js`: command line tool for comparing performance between different Node.js binaries. +* `compare-node-bench.js`: parallel comparison tool for explicit `node:bench` + files. It does not change or invoke `compare.js`. * `compare.R`: R script for statistically analyzing the output of `compare.js` * `run.js`: command line tool for running individual benchmark suite(s). * `scatter.js`: command line tool for comparing the performance between different parameters in benchmark configurations, for example to analyze the time complexity. +* `scatter-node-bench.js`: parallel scatter-data tool for an explicit + `node:bench` file. It does not change or invoke `scatter.js`. * `scatter.R`: R script for visualizing the output of `scatter.js` with scatter plots. +## `node:bench` evaluation tools + +The `compare-node-bench.js` and `scatter-node-bench.js` tools run explicit +`node:bench` files without changing the existing benchmark framework or its +tools. Each repeated observation for a benchmark identity is collected by a +separate process invocation with one measured sample. Benchmarks declared in +the same file still execute serially in that process and can share JIT, garbage +collector, heap, and cache state. This differs from legacy configuration-level +process isolation and must be considered when comparing the frameworks. + +Compare two binaries and analyze the compatible CSV using `compare.R`: + +```console +./node benchmark/compare-node-bench.js \ + --old ./node-main --new ./node-pr --runs 30 -- \ + benchmark/crypto/_create-hash.node-bench.js > compare-node-bench.csv +Rscript benchmark/compare.R < compare-node-bench.csv +``` + +Pass `--analyze` to run the same Welch analysis inline. `--max-regression N` +implies `--analyze` and makes the command fail only when the Holm-Bonferroni +adjusted p-value is below 0.05 and the full 95% confidence interval is worse +than `-N%`. Requiring both conditions prevents a noisy point estimate from +failing a regression gate. + +```console +./node benchmark/compare-node-bench.js \ + --old ./node-main --new ./node-pr --runs 30 \ + --max-regression 5 -- benchmark/crypto/_create-hash.node-bench.js +``` + +Collect parameter data for the parallel buffer benchmark and plot it using +`scatter.R`: + +```console +./node benchmark/scatter-node-bench.js --node ./node --runs 30 -- \ + benchmark/buffers/_buffer-compare-offset.node-bench.js \ + > scatter-node-bench.csv +Rscript benchmark/scatter.R --xaxis size --category method \ + --plot scatter-node-bench.png < scatter-node-bench.csv +``` + +Pass `--analyze` with an x-axis parameter to summarize the samples without R. +The output includes mean and median confidence intervals, skew warnings, an +optional bar chart, and Mann-Whitney U and Cliff's delta comparisons between +consecutive x-axis values. Use `--category` for a second grouping parameter and +`--no-chart` to omit the chart. + +Because configurations in one file share a process, inline analysis averages +aggregated configurations into one value per outer process. Consecutive +x-axis comparisons use alternating, disjoint process sets so the unpaired +Mann-Whitney test does not treat correlated values as independent samples. + +```console +./node benchmark/scatter-node-bench.js --runs 30 --analyze \ + --xaxis size --category method -- \ + benchmark/buffers/_buffer-compare-offset.node-bench.js +``` + +A file passed to `scatter-node-bench.js` must use one logical benchmark name. +Parameter values distinguish its configurations. The tool rejects unstable +identities and names or parameters that would merge unrelated CSV groups. + +The underscore-prefixed benchmark files are parallel ports used to compare the +measurement frameworks. Legacy discovery ignores them, so the original files +remain the source benchmarks for `run.js`, `compare.js`, and `scatter.js`. The +ports use the platform-specific original relative filename as their benchmark +name and preserve parameter column names to keep CSV grouping compatible. For +a direct framework comparison, collect the same number of runs from an +original benchmark with `scatter.js` and from its port with +`scatter-node-bench.js`, then compare their rate distributions. + ## Common API The common.js module is used by benchmarks for consistency across repeated diff --git a/benchmark/_node-bench-analysis.js b/benchmark/_node-bench-analysis.js new file mode 100644 index 000000000000..5fa3c34d3976 --- /dev/null +++ b/benchmark/_node-bench-analysis.js @@ -0,0 +1,736 @@ +'use strict'; + +const { createHistogram } = require('node:perf_hooks'); +const { inspect } = require('node:util'); + +function createRateHistogram(rates, scale, figures) { + const histogram = createHistogram({ figures }); + for (const rate of rates) { + const value = Math.max(1, Math.round(rate * scale)); + if (!Number.isSafeInteger(value)) { + throw new RangeError('Benchmark rate is too large for the histogram scale'); + } + histogram.record(value); + } + return histogram; +} + +function holmAdjust(pValues) { + const order = pValues + .map((p, index) => ({ index, p })) + .sort((a, b) => a.p - b.p); + const adjusted = new Array(order.length); + let running = 0; + for (let index = 0; index < order.length; index++) { + running = Math.max( + running, + Math.min(1, (order.length - index) * order[index].p), + ); + adjusted[order[index].index] = running; + } + return adjusted; +} + +function isRegressionFailure(row, maxRegression) { + return row.pAdjusted < 0.05 && + row.improvement + row.ci95 < -maxRegression; +} + +function analyzeCompare(samples, scale, maxRegression) { + const groups = new Map(); + for (const sample of samples) { + let group = groups.get(sample.identity); + if (group === undefined) { + const suffix = sample.configuration === '' ? + '' : ` ${sample.configuration}`; + group = { + name: `${sample.name}${suffix}`, + new: [], + old: [], + }; + groups.set(sample.identity, group); + } + group[sample.binary].push(sample.rate); + } + + const rows = []; + let skipped = 0; + for (const { name, old: oldRates, new: newRates } of groups.values()) { + if (oldRates.length < 2 || newRates.length < 2) { + skipped++; + continue; + } + + const oldHistogram = createRateHistogram(oldRates, scale, 3); + const newHistogram = createRateHistogram(newRates, scale, 3); + const oldMean = oldRates.reduce((sum, rate) => sum + rate, 0) / + oldRates.length; + const newMean = newRates.reduce((sum, rate) => sum + rate, 0) / + newRates.length; + const improvement = ((newMean - oldMean) / oldMean) * 100; + const w95 = oldHistogram.welchTest(newHistogram, { confidence: 0.95 }); + const w99 = oldHistogram.welchTest(newHistogram, { confidence: 0.99 }); + const w999 = oldHistogram.welchTest(newHistogram, { confidence: 0.999 }); + let stars = ''; + if (w95.pValue < 0.001) stars = '***'; + else if (w95.pValue < 0.01) stars = ' **'; + else if (w95.pValue < 0.05) stars = ' *'; + const ciPercent = (result) => { + const half = (result.confidenceInterval.upper - + result.confidenceInterval.lower) / 2; + return (half / (oldMean * scale)) * 100; + }; + rows.push({ + ci95: ciPercent(w95), + ci99: ciPercent(w99), + ci999: ciPercent(w999), + improvement, + name, + pValue: Number.isNaN(w95.pValue) ? 1 : w95.pValue, + stars, + }); + } + + const adjusted = holmAdjust(rows.map(({ pValue }) => pValue)); + let underpowered = 0; + for (let index = 0; index < rows.length; index++) { + const row = rows[index]; + row.pAdjusted = adjusted[index]; + row.inconclusive = maxRegression > 0 && + row.stars.trim() === '' && + row.ci95 > maxRegression; + if (row.inconclusive) underpowered++; + } + + const output = []; + const maxNameLength = rows.reduce( + (maximum, { name }) => Math.max(maximum, name.length), 0); + const pad = (value, length) => + value + ' '.repeat(Math.max(0, length - value.length)); + const padStart = (value, length) => + ' '.repeat(Math.max(0, length - value.length)) + value; + output.push(`${pad('', maxNameLength)} confidence` + + ' improvement accuracy (*) (**) (***)'); + for (const row of rows) { + const improvement = + `${row.improvement >= 0 ? '+' : ''}${row.improvement.toFixed(2)} %`; + output.push( + `${pad(row.name, maxNameLength)} ${pad(row.stars, 10)}` + + ` ${padStart(improvement, 11)}` + + ` ±${row.ci95.toFixed(2)}%` + + ` ±${row.ci99.toFixed(2)}%` + + ` ±${row.ci999.toFixed(2)}%` + + `${row.inconclusive ? ' (inconclusive)' : ''}`, + ); + } + + if (skipped > 0) { + output.push(''); + output.push( + `Note: ${skipped} configuration${skipped === 1 ? ' was' : 's were'}` + + ' skipped because Welch\'s t-test requires at least 2 samples per' + + ' binary. Use --runs 2 or higher.', + ); + } + printCompareChart(output, rows, maxNameLength); + + output.push(''); + output.push( + `Rates were scaled by ${scale}x into HdrHistogram (3 significant figures).`, + 'Use --scale to adjust precision if needed.', + '', + ); + const significant = rows.filter(({ pAdjusted }) => pAdjusted < 0.05).length; + output.push( + 'The confidence markers above are per-benchmark and uncorrected. ' + + `After Holm-Bonferroni correction across ${rows.length} comparison` + + `${rows.length === 1 ? '' : 's'}, ${significant} remain` + + `${significant === 1 ? 's' : ''} significant at 5%.`, + '--max-regression uses the corrected values.', + ); + + if (maxRegression > 0 && underpowered > 0) { + output.push(''); + output.push( + `Note: ${underpowered} of ${rows.length} comparison` + + `${rows.length === 1 ? '' : 's'} could not resolve an effect as small ` + + `as ${maxRegression}% and are marked (inconclusive). Raise --runs to ` + + 'narrow their confidence intervals.', + ); + } + + const failures = maxRegression > 0 ? + rows.filter((row) => isRegressionFailure(row, maxRegression)) : []; + if (failures.length > 0) { + output.push(''); + output.push( + `FAIL: ${failures.length} benchmark${failures.length === 1 ? '' : 's'}` + + ` regressed by more than ${maxRegression}% (the 95% interval excludes ` + + `the threshold and significance is family-wise corrected across ` + + `${rows.length} comparisons):`, + ); + for (const failure of failures) { + output.push( + ` ${failure.name} ${failure.improvement.toFixed(2)}% ` + + `(95% CI up to ${(failure.improvement + failure.ci95).toFixed(2)}%, ` + + `adjusted p=${failure.pAdjusted.toExponential(2)})`, + ); + } + } + + return { + failed: failures.length > 0, + output: `${output.join('\n')}\n`, + rows, + }; +} + +function printCompareChart(output, rows, maxNameLength) { + if (rows.length === 0) return; + const width = 40; + const halfWidth = width / 2; + let maximum = 0; + for (const row of rows) { + maximum = Math.max(maximum, Math.abs(row.improvement) + row.ci95); + } + if (maximum === 0) maximum = 1; + const left = `-${maximum.toFixed(1)}%`; + const right = `+${maximum.toFixed(1)}%`; + const centerLabel = '0%'; + const labelPadding = maxNameLength + 5; + output.push(''); + output.push( + ' '.repeat(labelPadding) + left + + ' '.repeat(Math.max( + 0, halfWidth - left.length - Math.floor(centerLabel.length / 2))) + + centerLabel + + ' '.repeat(Math.max( + 0, halfWidth - Math.ceil(centerLabel.length / 2) - right.length)) + + right, + ); + for (const row of rows) { + const center = halfWidth; + const result = center + (row.improvement / maximum) * halfWidth; + const lower = center + + ((row.improvement - row.ci95) / maximum) * halfWidth; + const upper = center + + ((row.improvement + row.ci95) / maximum) * halfWidth; + let bar = ''; + for (let index = 0; index < width; index++) { + const position = index + 0.5; + if (index === Math.floor(center)) { + bar += '|'; + } else if ((row.improvement >= 0 && + position > center && position <= result) || + (row.improvement < 0 && + position < center && position >= result)) { + bar += row.stars === '' ? '▓' : '█'; + } else if (position >= lower && position <= upper) { + bar += '░'; + } else { + bar += ' '; + } + } + const label = `${row.improvement >= 0 ? '+' : ''}` + + `${row.improvement.toFixed(2)}%`; + output.push( + `${row.name.padEnd(maxNameLength)} ${bar} ${label} ${row.stars.trim()}`, + ); + } +} + +function histogramScale(rates) { + let minimum = Infinity; + let maximum = 0; + for (const rate of rates) { + if (rate > 0 && rate < minimum) minimum = rate; + if (rate > maximum) maximum = rate; + } + if (!Number.isFinite(minimum) || maximum === 0) return 1; + let scale = 1; + while (minimum * scale < 1e6 && maximum * scale < 1e15) scale *= 10; + return scale; +} + +function validateScatterParameters(samples, xAxis, category) { + if (category !== undefined && category === xAxis) { + throw new Error('--xaxis and --category must name different parameters'); + } + for (const key of [xAxis, category]) { + if (key === undefined) continue; + if (samples.some(({ params }) => + !Object.hasOwn(params, key))) { + const available = [...new Set(samples.flatMap( + ({ params }) => Object.keys(params)))].sort(); + throw new Error( + `The variable '${key}' is not present in every configuration. ` + + `Available variables: ${available.join(', ')}`, + ); + } + } +} + +function analyzeScatter(samples, xAxis, category, showChart) { + validateScatterParameters(samples, xAxis, category); + + const parameterNames = [...new Set(samples.flatMap( + ({ params }) => Object.keys(params)))]; + const aggregated = parameterNames.filter((name) => { + if (name === xAxis || name === category) return false; + const first = samples[0].params[name]; + return samples.some(({ params }) => params[name] !== first); + }); + const groups = new Map(); + for (const sample of samples) { + const xValue = sample.params[xAxis]; + const categoryValue = category === undefined ? + undefined : sample.params[category]; + const key = valueKey([xValue, categoryValue]); + let group = groups.get(key); + if (group === undefined) { + group = { + categoryValue, + members: [], + observations: new Map(), + xValue, + }; + groups.set(key, group); + } + group.members.push(sample); + let rates = group.observations.get(sample.observation); + if (rates === undefined) { + rates = []; + group.observations.set(sample.observation, rates); + } + rates.push(sample.rate); + } + for (const group of groups.values()) { + group.processRates = [...group.observations].map(([observation, rates]) => ({ + observation, + rate: rates.reduce((sum, rate) => sum + rate, 0) / rates.length, + })); + group.rates = group.processRates.map(({ rate }) => rate); + } + + const scale = histogramScale(samples.map(({ rate }) => rate)); + const compareValues = (a, b) => { + if (typeof a === 'number' && typeof b === 'number') return a - b; + return String(a).localeCompare(String(b)); + }; + const rows = [...groups.values()] + .sort((a, b) => compareValues(a.xValue, b.xValue) || + compareValues(a.categoryValue, b.categoryValue)) + .map((group) => { + const histogram = createRateHistogram(group.rates, scale, 5); + const count = group.rates.length; + const mean = group.rates.reduce((sum, rate) => sum + rate, 0) / count; + const meanInterval = histogram.meanCI(); + const confidenceInterval = count > 1 ? + (meanInterval.upper - meanInterval.lower) / (2 * scale) : NaN; + const medianInterval = histogram.percentileCI(50); + const median = rawMedian(group.rates); + const skewed = count > 1 && + (Math.abs(histogram.skewness) > 1 || + Math.abs(median - mean) > confidenceInterval); + return { + ...group, + confidenceInterval, + count, + histogram, + mean, + median, + medianLower: medianInterval.lower / scale, + medianUpper: medianInterval.upper / scale, + skewed, + }; + }); + + const legend = assignLabels(rows, 'xValue', 'xLabel'); + if (category !== undefined) { + legend.push(...assignLabels(rows, 'categoryValue', 'categoryLabel')); + } + const output = []; + const contamination = new Map(); + for (const variable of aggregated) { + const share = varianceShare([...groups.values()], variable); + contamination.set(variable, share); + const percent = share === 1 ? + '100' : (share >= 0.995 ? '>99' : (share * 100).toFixed(0)); + const suffix = Number.isNaN(share) ? + '' : ` (explains ${percent}% of within-group variance)`; + output.push(`aggregating variable: ${variable}${suffix}`); + } + const dominant = aggregated.filter( + (variable) => contamination.get(variable) > 0.5); + if (dominant.length > 0) { + output.push(''); + wrapOutput( + output, + `${dominant.join(', ')} ${dominant.length === 1 ? 'explains' : 'explain'} ` + + 'most of the spread within each group. Pin the parameter or use it as ' + + '--category; increasing --runs will not remove this source of variance.', + ); + } + if (aggregated.length > 0) output.push(''); + printScatterTable(output, rows, xAxis, category); + if (showChart) printScatterChart(output, rows, xAxis, category); + printScatterComparisons(output, rows, xAxis, category); + if (legend.length > 0) { + output.push('', 'Abbreviated values:'); + for (const { full, label } of legend) { + output.push(` ${label}`, ` = ${full}`); + } + } + const singleSample = rows.filter(({ count }) => count < 2).length; + if (singleSample > 0) { + output.push(''); + wrapOutput( + output, + `Note: ${singleSample} group${singleSample === 1 ? ' has' : 's have'} ` + + 'only one sample, so no confidence interval could be estimated. Use ' + + '--runs 2 or higher.', + ); + } + if (rows.some(({ skewed }) => skewed)) { + output.push(''); + wrapOutput( + output, + '(!) marks groups where the median falls outside the mean confidence ' + + 'interval or the sample is strongly skewed. The median and its interval ' + + 'describe the typical run better for those groups.', + ); + } + return `${output.join('\n')}\n`; +} + +function rawMedian(rates) { + const sorted = [...rates].sort((a, b) => a - b); + const middle = Math.floor(sorted.length / 2); + return sorted.length % 2 === 0 ? + (sorted[middle - 1] + sorted[middle]) / 2 : sorted[middle]; +} + +function valueKey(value) { + return JSON.stringify(value, (_, item) => { + if (typeof item === 'bigint') return { bigint: String(item) }; + return item; + }); +} + +function varianceShare(groups, variable) { + let between = 0; + let total = 0; + for (const group of groups) { + if (group.members.length < 2) continue; + const mean = group.members.reduce( + (sum, sample) => sum + sample.rate, 0) / group.members.length; + const levels = new Map(); + for (const sample of group.members) { + const key = valueKey(sample.params[variable]); + let level = levels.get(key); + if (level === undefined) { + level = { count: 0, sum: 0 }; + levels.set(key, level); + } + level.count++; + level.sum += sample.rate; + } + for (const level of levels.values()) { + between += level.count * ((level.sum / level.count) - mean) ** 2; + } + for (const sample of group.members) total += (sample.rate - mean) ** 2; + } + return total === 0 ? NaN : Math.min(1, between / total); +} + +function effectSizeLabel(delta) { + const absolute = Math.abs(delta); + if (absolute < 0.147) return 'negligible'; + if (absolute < 0.33) return 'small'; + if (absolute < 0.474) return 'medium'; + return 'large'; +} + +const mannWhitneyFloors = new Map(); +function mannWhitneyFloor(firstCount, secondCount) { + const key = `${firstCount},${secondCount}`; + let floor = mannWhitneyFloors.get(key); + if (floor !== undefined) return floor; + const low = createHistogram({ figures: 5 }); + const high = createHistogram({ figures: 5 }); + for (let index = 0; index < firstCount; index++) low.record(10000 + index); + for (let index = 0; index < secondCount; index++) { + high.record(10000 + firstCount + index); + } + floor = high.mannWhitneyTest(low).pValue; + mannWhitneyFloors.set(key, floor); + return floor; +} + +function printScatterComparisons(output, rows, xAxis, category) { + const usable = rows.filter(({ count }) => count > 1); + if (usable.length < 2) return; + const series = new Map(); + for (const row of usable) { + const key = valueKey(row.categoryValue); + if (!series.has(key)) series.set(key, []); + series.get(key).push(row); + } + const sections = []; + let floor = 0; + for (const group of series.values()) { + if (group.length < 2) continue; + const entries = []; + for (let index = 1; index < group.length; index++) { + const previous = group[index - 1]; + const current = group[index]; + // Configurations in one file share a process. Split consecutive groups + // across disjoint outer-process sets so the unpaired test does not treat + // correlated observations as independent. + const parity = index % 2; + const previousRates = previous.processRates + .filter(({ observation }) => observation % 2 === parity) + .map(({ rate }) => rate); + const currentRates = current.processRates + .filter(({ observation }) => observation % 2 !== parity) + .map(({ rate }) => rate); + if (previousRates.length === 0 || currentRates.length === 0) continue; + const comparisonScale = histogramScale([ + ...previousRates, + ...currentRates, + ]); + const previousHistogram = + createRateHistogram(previousRates, comparisonScale, 5); + const currentHistogram = + createRateHistogram(currentRates, comparisonScale, 5); + const { pValue } = currentHistogram.mannWhitneyTest(previousHistogram); + const delta = currentHistogram.cliffsD(previousHistogram); + const previousMean = previousRates.reduce( + (sum, rate) => sum + rate, 0) / previousRates.length; + const currentMean = currentRates.reduce( + (sum, rate) => sum + rate, 0) / currentRates.length; + const change = ((currentMean - previousMean) / previousMean) * 100; + floor = Math.max( + floor, mannWhitneyFloor(previousRates.length, currentRates.length)); + let ratio = ''; + let exponent = ''; + if (typeof previous.xValue === 'number' && + typeof current.xValue === 'number' && + previous.xValue > 0 && current.xValue > 0 && + previous.xValue !== current.xValue && + previousMean > 0 && currentMean > 0) { + const xRatio = current.xValue / previous.xValue; + const value = Math.log(currentMean / previousMean) / Math.log(xRatio); + ratio = `${xRatio.toFixed(1)}x`; + exponent = `${value >= 0 ? '+' : ''}${value.toFixed(2)}`; + } + entries.push( + ` ${previous.xLabel} -> ${current.xLabel}` + + ` ${change >= 0 ? '+' : ''}${change.toFixed(2)}%` + + (exponent === '' ? '' : ` ${ratio} exponent=${exponent}`) + + ` p=${pValue < 1e-4 ? pValue.toExponential(1) : pValue.toFixed(4)}` + + ` delta=${delta >= 0 ? '+' : ''}${delta.toFixed(3)}` + + ` (${effectSizeLabel(delta)})`, + ); + } + if (entries.length > 0) { + sections.push({ + entries, + heading: category === undefined ? + undefined : `${category}=${group[0].categoryLabel}`, + }); + } + } + if (sections.length === 0) return; + output.push('', `Change between consecutive ${xAxis} values ` + + `(Mann-Whitney U on disjoint process sets, Cliff's delta):`); + for (const section of sections) { + output.push(''); + if (section.heading !== undefined) output.push(` ${section.heading}`); + output.push(...section.entries); + } + if (floor >= 0.05) { + output.push(''); + wrapOutput( + output, + `Warning: at this sample size the smallest p-value this test can ` + + `produce is ${floor.toFixed(4)}, so no comparison above can reach ` + + 'significance. Raise --runs.', + ); + } else if (floor >= 0.005) { + output.push(''); + wrapOutput( + output, + `Note: at this sample size the smallest p-value this test can produce ` + + `is ${floor.toFixed(4)}. Raise --runs to strengthen non-significant ` + + 'results.', + ); + } +} + +function formatRate(rate) { + return rate.toLocaleString('en-US', { + maximumFractionDigits: 1, + minimumFractionDigits: 1, + }); +} + +function displayWidth(value) { + return [...value].length; +} + +function pad(value, width, right) { + const padding = ' '.repeat(Math.max(0, width - displayWidth(value))); + return right ? padding + value : value + padding; +} + +function truncateMiddle(value, maximum = 24) { + const characters = [...value]; + if (characters.length <= maximum) return value; + const retained = maximum - 3; + const head = Math.ceil(retained / 2); + const tail = Math.floor(retained / 2); + return `${characters.slice(0, head).join('')}...` + + characters.slice(-tail).join(''); +} + +function assignLabels(rows, valueName, labelName) { + const assigned = new Map(); + const used = new Map(); + const legend = []; + for (const row of rows) { + const key = valueKey(row[valueName]); + let label = assigned.get(key); + if (label === undefined) { + const full = typeof row[valueName] === 'string' ? + inspect(row[valueName]) : String(row[valueName]); + const abbreviated = truncateMiddle(full); + const collisions = used.get(abbreviated) ?? 0; + used.set(abbreviated, collisions + 1); + label = collisions === 0 ? + abbreviated : `${abbreviated}~${collisions + 1}`; + assigned.set(key, label); + if (label !== full) legend.push({ full, label }); + } + row[labelName] = label; + } + return legend; +} + +function printScatterTable(output, rows, xAxis, category) { + const header = [xAxis]; + if (category !== undefined) header.push(category); + header.push( + 'samples', 'rate', 'confidence.interval', 'median', 'median.interval', ''); + const body = rows.map((row) => { + const values = [row.xLabel]; + if (category !== undefined) values.push(row.categoryLabel); + const medianInterval = row.count > 1 ? + `[${(((row.medianLower - row.median) / row.median) * 100).toFixed(2)}%, ` + + `+${(((row.medianUpper - row.median) / row.median) * 100).toFixed(2)}%]` : + 'NA'; + values.push( + String(row.count), + formatRate(row.mean), + Number.isNaN(row.confidenceInterval) ? + 'NA' : + `${formatRate(row.confidenceInterval)} ` + + `(±${((row.confidenceInterval / row.mean) * 100).toFixed(2)}%)`, + formatRate(row.median), + medianInterval, + row.skewed ? '(!)' : '', + ); + return values; + }); + const widths = header.map((value, index) => Math.max( + displayWidth(value), + ...body.map((values) => displayWidth(values[index])), + )); + const right = [typeof rows[0].xValue === 'number']; + if (category !== undefined) { + right.push(typeof rows[0].categoryValue === 'number'); + } + right.push(true, true, true, true, true, false); + const format = (values) => values.map( + (value, index) => pad(value, widths[index], right[index])).join(' ').trimEnd(); + output.push(format(header)); + for (const values of body) output.push(format(values)); +} + +function printScatterChart(output, rows, xAxis, category) { + if (rows.length === 0) return; + const width = 40; + let maximum = 0; + for (const row of rows) { + maximum = Math.max( + maximum, + row.mean + (Number.isNaN(row.confidenceInterval) ? + 0 : row.confidenceInterval), + ); + } + if (maximum === 0) return; + const labels = rows.map((row) => { + let label = `${xAxis}=${row.xLabel}`; + if (category !== undefined) label += ` ${category}=${row.categoryLabel}`; + return truncateMiddle(label, 44); + }); + const labelWidth = Math.max(...labels.map(displayWidth)); + const rateWidth = Math.max(...rows.map(({ mean }) => + displayWidth(formatRate(mean)))); + const axis = formatRate(maximum); + const indent = ' '.repeat(labelWidth + 2); + output.push( + '', + 'Rate in operations/second; longer is faster. │ marks the mean and the', + 'shaded band (░) is its 95% confidence interval.', + '', + `${indent}0${' '.repeat(Math.max(1, width - 1 - displayWidth(axis)))}${axis}`, + `${indent}+${'-'.repeat(width - 2)}+`, + ); + let previous; + for (let index = 0; index < rows.length; index++) { + const row = rows[index]; + if (previous !== undefined && previous !== row.xValue) output.push(''); + previous = row.xValue; + const interval = Number.isNaN(row.confidenceInterval) ? + 0 : row.confidenceInterval; + const end = (row.mean / maximum) * width; + const lower = ((row.mean - interval) / maximum) * width; + const upper = ((row.mean + interval) / maximum) * width; + const meanCell = Math.min(width - 1, Math.floor(end)); + let bar = ''; + for (let cell = 0; cell < width; cell++) { + const position = cell + 0.5; + if (cell === meanCell) bar += '│'; + else if (position >= lower && position <= upper) bar += '░'; + else if (position <= end) bar += '█'; + else bar += ' '; + } + output.push( + `${pad(labels[index], labelWidth, false)} ${bar} ` + + pad(formatRate(row.mean), rateWidth, true), + ); + } +} + +function wrapOutput(output, text, width = 76) { + let line = ''; + for (const word of text.split(/\s+/)) { + if (line === '') line = word; + else if (displayWidth(line) + displayWidth(word) + 1 <= width) { + line += ` ${word}`; + } else { + output.push(line); + line = word; + } + } + if (line !== '') output.push(line); +} + +module.exports = { + analyzeCompare, + analyzeScatter, + holmAdjust, + isRegressionFailure, + validateScatterParameters, +}; diff --git a/benchmark/_node-bench.js b/benchmark/_node-bench.js new file mode 100644 index 000000000000..6c8d1acffdfa --- /dev/null +++ b/benchmark/_node-bench.js @@ -0,0 +1,159 @@ +'use strict'; + +const { spawn } = require('node:child_process'); +const path = require('node:path'); +const { inspect } = require('node:util'); + +function parseInteger(value, defaultValue, name, minimum) { + if (value === undefined) return defaultValue; + if (!/^(?:0|[1-9]\d*)$/.test(value)) { + throw new TypeError(`${name} must be an integer`); + } + const number = Number(value); + if (!Number.isSafeInteger(number) || number < minimum) { + throw new RangeError(`${name} must be at least ${minimum}`); + } + return number; +} + +function parseNumber(value, defaultValue, name, minimum) { + if (value === undefined) return defaultValue; + if (value.trim() === '') throw new TypeError(`${name} must be a number`); + const number = Number(value); + if (!Number.isFinite(number)) throw new TypeError(`${name} must be a number`); + if (number < minimum) { + throw new RangeError(`${name} must be at least ${minimum}`); + } + return number; +} + +function csvEncode(value) { + if (typeof value === 'number' || typeof value === 'boolean') { + return String(value); + } + const string = String(value); + return `"${string.replace(/"/g, '""')}"`; +} + +function formatConfiguration(params) { + return Object.keys(params) + .map((key) => `${key}=${inspect(params[key])}`) + .join(' '); +} + +function durationToSeconds(duration) { + if (!/^\d+$/.test(duration)) { + throw new TypeError(`Invalid benchmark duration '${duration}'`); + } + const padded = duration.padStart(10, '0'); + return `${padded.slice(0, -9)}.${padded.slice(-9)}`; +} + +function runBenchmark(binary, file, options) { + const args = [ + ...options.nodeArgs, + '--no-warnings', + '--bench', + '--bench-reporter=json', + '--bench-samples=1', + `--bench-warmup=${options.warmup}`, + ]; + if (options.namePattern !== undefined) { + args.push(`--bench-name-pattern=${options.namePattern}`); + } + args.push('--', path.resolve(file)); + + return new Promise((resolve, reject) => { + const child = spawn(binary, args, { + env: process.env, + stdio: ['ignore', 'pipe', 'pipe'], + }); + child.stdout.setEncoding('utf8'); + child.stderr.setEncoding('utf8'); + + let stdout = ''; + let stderr = ''; + child.stdout.on('data', (data) => { stdout += data; }); + child.stderr.on('data', (data) => { stderr += data; }); + child.once('error', reject); + child.once('close', (code, signal) => { + let records; + try { + records = stdout.trim().split('\n') + .filter((line) => line.length > 0) + .map((line) => JSON.parse(line)); + } catch (error) { + reject(new Error( + `Could not parse benchmark output from '${binary}': ${error.message}`, + { cause: error }, + )); + return; + } + + const summary = records.findLast( + ({ type }) => type === 'bench:summary')?.data; + if (code !== 0 || signal !== null || summary?.success !== true) { + const diagnostics = records + .filter(({ type }) => type === 'bench:diagnostic') + .map(({ data }) => data.message) + .join('\n'); + const status = signal === null ? `exit code ${code}` : `signal ${signal}`; + const details = stderr || diagnostics; + reject(new Error( + `Benchmark '${file}' failed with ${status}` + + (details ? `:\n${details}` : ''), + )); + return; + } + + const samples = []; + for (const record of records) { + if (record.type !== 'bench:complete' || + record.data.skip !== undefined) { + continue; + } + if (record.data.error !== undefined) { + reject(new Error( + `Benchmark '${record.data.name}' failed: ` + + record.data.error.message, + )); + return; + } + if (record.data.samples.length !== 1) { + reject(new Error( + `Benchmark '${record.data.name}' did not produce exactly one sample`, + )); + return; + } + const sample = record.data.samples[0]; + if (!Number.isFinite(sample.rate)) { + reject(new Error( + `Benchmark '${record.data.name}' produced a non-finite rate`, + )); + return; + } + samples.push({ + configuration: formatConfiguration(record.data.params), + duration: durationToSeconds(sample.duration_ns), + identity: record.data.benchId, + logicalIdentity: JSON.stringify([ + record.data.file, + record.data.parentId, + record.data.name, + ]), + name: record.data.name, + params: record.data.params, + rate: sample.rate, + }); + } + resolve(samples); + }); + }); +} + +module.exports = { + csvEncode, + parseInteger, + parseNumber, + runBenchmark, +}; diff --git a/benchmark/buffers/_buffer-compare-offset.node-bench.js b/benchmark/buffers/_buffer-compare-offset.node-bench.js new file mode 100644 index 000000000000..b16c3a3da7db --- /dev/null +++ b/benchmark/buffers/_buffer-compare-offset.node-bench.js @@ -0,0 +1,37 @@ +'use strict'; + +const { bench } = require('node:bench'); +const path = require('node:path'); + +const methods = ['offset', 'slice']; +const sizes = [16, 512, 4096, 16386]; +const n = 1e6; +const name = path.join('buffers', 'buffer-compare-offset.js'); + +function compareUsingSlice(b0, b1, len, iterations) { + for (let i = 0; i < iterations; i++) + Buffer.compare(b0.slice(1, len), b1.slice(1, len)); +} + +function compareUsingOffset(b0, b1, len, iterations) { + for (let i = 0; i < iterations; i++) + b0.compare(b1, 1, len, 1, len); +} + +for (const method of methods) { + for (const size of sizes) { + const compare = method === 'slice' ? + compareUsingSlice : compareUsingOffset; + + bench(name, { + params: { method, n, size }, + }, (b) => { + b.start(); + compare(Buffer.alloc(size, 'a'), + Buffer.alloc(size, 'b'), + size >> 1, + n); + b.end(n); + }); + } +} diff --git a/benchmark/compare-node-bench.js b/benchmark/compare-node-bench.js new file mode 100644 index 000000000000..d2bc6071bed7 --- /dev/null +++ b/benchmark/compare-node-bench.js @@ -0,0 +1,116 @@ +'use strict'; + +const path = require('node:path'); +const CLI = require('./_cli.js'); +const { analyzeCompare } = require('./_node-bench-analysis.js'); +const { + csvEncode, + parseInteger, + parseNumber, + runBenchmark, +} = require('./_node-bench.js'); + +const cli = new CLI(`usage: ./node compare-node-bench.js [options] [--] ... + Run explicit node:bench files repeatedly with two Node.js binaries. Each + observation runs in a fresh process. Output is compatible with compare.R, + or --analyze can summarize it directly. + + --new binary new Node.js binary (required) + --old binary old Node.js binary (required) + --runs 30 observations per binary + --warmup 0 warmup samples before each observation + --name-pattern pattern only run matching benchmarks + --node-arg argument pass an argument to both binaries (repeatable) + --analyze analyze with Welch's t-test instead of writing CSV + --scale 1000 rate multiplier used for histogram precision + --max-regression N fail if a family-wise significant regression's + 95% confidence interval is entirely beyond N% + (implies --analyze) +`, { arrayArgs: ['node-arg'], boolArgs: ['analyze'] }); + +if (!cli.optional.new || !cli.optional.old || cli.items.length === 0) { + cli.abort(cli.usage); +} + +async function main() { + const runs = parseInteger(cli.optional.runs, 30, '--runs', 1); + const warmup = parseInteger(cli.optional.warmup, 0, '--warmup', 0); + const scale = parseInteger(cli.optional.scale, 1000, '--scale', 1); + const maxRegression = parseNumber( + cli.optional['max-regression'], 0, '--max-regression', 0); + const analyze = !!cli.optional.analyze || maxRegression > 0; + const options = { + namePattern: cli.optional['name-pattern'], + nodeArgs: cli.optional['node-arg'], + warmup, + }; + const binaries = [ + { label: 'old', path: cli.optional.old }, + { label: 'new', path: cli.optional.new }, + ]; + const rows = []; + const counts = new Map(); + const csvGroups = new Map(); + + for (const file of cli.items) { + const resolved = path.resolve(file); + for (let run = 0; run < runs; run++) { + const order = run % 2 === 0 ? binaries : [binaries[1], binaries[0]]; + for (const binary of order) { + const samples = await runBenchmark(binary.path, resolved, options); + for (const sample of samples) { + const identity = JSON.stringify([resolved, sample.identity]); + const csvGroup = JSON.stringify([ + sample.name, + sample.configuration, + ]); + const groupedIdentity = csvGroups.get(csvGroup); + if (groupedIdentity !== undefined && groupedIdentity !== identity) { + throw new Error( + `Distinct benchmarks would share the CSV group '${sample.name} ` + + `${sample.configuration}'`, + ); + } + csvGroups.set(csvGroup, identity); + let count = counts.get(identity); + if (count === undefined) { + count = { name: sample.name, new: 0, old: 0 }; + counts.set(identity, count); + } + count[binary.label]++; + rows.push({ binary: binary.label, ...sample }); + } + } + } + } + + if (rows.length === 0) { + throw new Error('No benchmark samples were produced'); + } + for (const count of counts.values()) { + if (count.old !== runs || count.new !== runs) { + throw new Error( + `Benchmark '${count.name}' was not reported by both binaries in every run`, + ); + } + } + + if (analyze) { + const result = analyzeCompare(rows, scale, maxRegression); + process.stdout.write(result.output); + if (result.failed) process.exitCode = 1; + return; + } + + const output = ['"binary","filename","configuration","rate","time"']; + for (const row of rows) { + output.push(`${csvEncode(row.binary)},${csvEncode(row.name)},` + + `${csvEncode(row.configuration)},${row.rate},${row.duration}`); + } + process.stdout.write(`${output.join('\n')}\n`); +} + +main().catch((error) => { + console.error(error.stack); + process.exitCode = 1; +}); diff --git a/benchmark/crypto/_create-hash.node-bench.js b/benchmark/crypto/_create-hash.node-bench.js new file mode 100644 index 000000000000..4481bc6c6527 --- /dev/null +++ b/benchmark/crypto/_create-hash.node-bench.js @@ -0,0 +1,19 @@ +'use strict'; + +const assert = require('node:assert'); +const { bench } = require('node:bench'); +const { createHash } = require('node:crypto'); +const path = require('node:path'); + +const n = 1e5; +const name = path.join('crypto', 'create-hash.js'); + +bench(name, { params: { n } }, (b) => { + const array = []; + for (let i = 0; i < n; ++i) array.push(null); + b.start(); + for (let i = 0; i < n; ++i) + array[i] = createHash('sha1'); + b.end(n); + assert.strictEqual(typeof array[n - 1], 'object'); +}); diff --git a/benchmark/scatter-node-bench.js b/benchmark/scatter-node-bench.js new file mode 100644 index 000000000000..4527ecc6b8b4 --- /dev/null +++ b/benchmark/scatter-node-bench.js @@ -0,0 +1,131 @@ +'use strict'; + +const CLI = require('./_cli.js'); +const { + analyzeScatter, + validateScatterParameters, +} = require('./_node-bench-analysis.js'); +const { + csvEncode, + parseInteger, + runBenchmark, +} = require('./_node-bench.js'); + +const cli = new CLI(`usage: ./node scatter-node-bench.js [options] [--] + Run an explicit node:bench file repeatedly and output each independent + observation with its benchmark parameters as CSV, or summarize the results + directly with --analyze. + + --node ./node Node.js binary + --runs 30 number of observations + --warmup 0 warmup samples before each observation + --name-pattern pattern only run matching benchmarks + --node-arg argument pass an argument to the binary (repeatable) + --analyze print a statistical summary instead of CSV + --xaxis parameter parameter to group by with --analyze (required) + --category parameter optional second grouping parameter + --no-chart omit the analysis bar chart +`, { + arrayArgs: ['node-arg'], + boolArgs: ['analyze', 'no-chart'], +}); + +if (cli.items.length !== 1) cli.abort(cli.usage); +if (cli.optional.analyze && cli.optional.xaxis === undefined) { + cli.abort('--analyze requires --xaxis '); +} + +async function main() { + const runs = parseInteger(cli.optional.runs, 30, '--runs', 1); + const warmup = parseInteger(cli.optional.warmup, 0, '--warmup', 0); + const options = { + namePattern: cli.optional['name-pattern'], + nodeArgs: cli.optional['node-arg'], + warmup, + }; + const binary = cli.optional.node || process.execPath; + const rows = []; + const paramNames = new Set(); + const csvGroups = new Map(); + let expectedIdentities; + let logicalIdentity; + + for (let run = 0; run < runs; run++) { + const samples = await runBenchmark(binary, cli.items[0], options); + if (run === 0 && cli.optional.analyze) { + validateScatterParameters( + samples, cli.optional.xaxis, cli.optional.category); + } + const identities = new Set(); + for (const sample of samples) { + if (logicalIdentity !== undefined && + logicalIdentity !== sample.logicalIdentity) { + throw new Error( + 'scatter-node-bench.js requires one logical benchmark name per file', + ); + } + logicalIdentity = sample.logicalIdentity; + if (identities.has(sample.identity)) { + throw new Error(`Benchmark '${sample.name}' was reported more than once`); + } + identities.add(sample.identity); + const csvGroup = JSON.stringify([sample.name, sample.params]); + const groupedIdentity = csvGroups.get(csvGroup); + if (groupedIdentity !== undefined && + groupedIdentity !== sample.identity) { + throw new Error( + `Distinct benchmarks would share the CSV group '${sample.name}'`, + ); + } + csvGroups.set(csvGroup, sample.identity); + rows.push({ ...sample, observation: run }); + for (const name of Object.keys(sample.params)) paramNames.add(name); + } + if (expectedIdentities === undefined) { + expectedIdentities = identities; + } else if (identities.size !== expectedIdentities.size || + ![...identities].every((id) => expectedIdentities.has(id))) { + throw new Error('The set of reported benchmarks changed between runs'); + } + } + if (rows.length === 0) { + throw new Error('No benchmark samples were produced'); + } + + const params = [...paramNames].sort(); + if (cli.optional.analyze) { + const output = analyzeScatter( + rows, + cli.optional.xaxis, + cli.optional.category, + !cli.optional['no-chart'], + ); + process.stdout.write(output); + return; + } + + for (const name of params) { + if (name === 'filename' || name === 'rate' || name === 'time') { + throw new Error(`Benchmark parameter '${name}' is reserved in scatter CSV`); + } + } + const header = ['filename', ...params, 'rate', 'time'] + .map(csvEncode) + .join(','); + const output = [header]; + for (const row of rows) { + const values = [ + csvEncode(row.name), + ...params.map((name) => csvEncode(row.params[name] ?? '')), + row.rate, + row.duration, + ]; + output.push(values.join(',')); + } + process.stdout.write(`${output.join('\n')}\n`); +} + +main().catch((error) => { + console.error(error.stack); + process.exitCode = 1; +}); diff --git a/doc/api/bench.md b/doc/api/bench.md new file mode 100644 index 000000000000..4acea0f98c9d --- /dev/null +++ b/doc/api/bench.md @@ -0,0 +1,456 @@ +# Benchmark runner + + + + + +> Stability: 1.0 - Early Development + + + +The `node:bench` module supports defining and running JavaScript benchmarks in +the current process. To access it: + +```mjs +import { bench, suite } from 'node:bench'; +``` + +```cjs +const { bench, suite } = require('node:bench'); +``` + +This module is only available under the `node:` scheme. + +## Example benchmark + +Save the following as `benchmark.mjs`: + +```mjs +import { bench, suite } from 'node:bench'; + +suite('URL', () => { + const input = 'https://example.com/a?b=c'; + + bench('construct', { + samples: 30, + params: { input: 'short' }, + }, (b) => { + const operations = 10_000; + + b.start(); + for (let i = 0; i < operations; i++) { + new URL(input); + } + b.end(operations); + }); +}); +``` + +Run the benchmark from the command line: + +```console +node --bench benchmark.mjs +``` + +Benchmarks are executed serially in declaration order. Declared benchmarks are +scheduled automatically. Call `run()` during the same turn as the declarations +to consume the event stream or configure filtering. +If an automatically scheduled run fails and `run()` was not called, the process +exit code is set to `1`. + +## Measurement model + +Each warmup and measured sample invokes the benchmark function once with a +fresh {BenchContext}. The function must call `context.start()` and +`context.end(operations)` exactly once. Setup before `start()` and cleanup after +`end()` are outside the measured region. Promise-returning functions are +awaited. + +An event loop turn occurs between sample invocations. The runner executes +benchmarks serially, but it does not provide process isolation. Other work in +the process, JIT compilation, garbage collection, CPU frequency changes, and +system load can all affect results. Keep raw samples when comparing results and +investigate noisy or skewed distributions rather than treating a confidence +interval as a pass/fail threshold. + +## Command-line runner + +The `--bench` flag runs one or more explicit benchmark files or glob patterns: + +```console +node --bench benchmark.mjs +node --bench --bench-reporter=json 'benchmarks/**/*.js' +``` + +Files are sorted and executed serially. The default +`--bench-isolation=process` mode runs each file in a separate child process and +emits one aggregate summary. Structured events are transferred to the parent +without JSON conversion, preserving BigInt durations, errors, and parameter +values. Child writes to stdout and stderr are emitted as diagnostic records so +they do not corrupt reporter output. + +`--bench-isolation=none` imports all files into the runner process. This mode +has lower startup overhead, but module, heap, and process state carry between +files, and user writes share stdout and stderr with reporters. + +Benchmark files passed to `--bench` should declare benchmarks but must not call +`run()`. The CLI supports `--bench-name-pattern`, `--bench-samples`, +`--bench-warmup`, `--bench-reporter`, and `--bench-reporter-destination`. See +the [command-line options documentation][] for details. + +## Benchmark reporters + +The built-in reporters are available from the scheme-only +`node:bench/reporters` module: + +```mjs +import { json, spec } from 'node:bench/reporters'; +``` + +```cjs +const { json, spec } = require('node:bench/reporters'); +``` + +Reporter values can be passed directly to `stream.compose()`: + +```mjs +import { bench, run } from 'node:bench'; +import { spec } from 'node:bench/reporters'; +import process from 'node:process'; + +bench('example', (b) => { + b.start(); + doWork(); + b.end(1); +}); + +run().compose(spec).pipe(process.stdout); +``` + +The `spec` reporter buffers results and outputs a concise table containing the +sample count, mean rate, 95% confidence interval for the mean, median rate, and +warnings. A coefficient of variation above 5% is reported as `noisy`, and an +absolute skewness above 1 is reported as `skewed`. The exact human-readable +format is subject to change. + +The `json` reporter emits every lifecycle record as newline-delimited JSON. +BigInt values, including `duration_ns`, are encoded as decimal strings. Errors +are represented using their `name`, `message`, `stack`, `code`, `cause`, and +`errors` properties. As required by JSON, non-finite numbers are encoded as +`null`. + +Custom reporters use the same composition contract. They can be transforms or +functions accepted by `stream.compose()`. The composed readable can be piped to +any writable destination: + +```mjs +import { run } from 'node:bench'; +import process from 'node:process'; + +async function* names(source) { + for await (const { type, data } of source) { + if (type === 'bench:complete') { + yield `${data.name}\n`; + } + } +} + +run().compose(names).pipe(process.stdout); +``` + +## `bench([name][, options], fn)` + + + +* `name` {string} The benchmark name. **Default:** The `name` property of `fn`, + or `''` when `fn` has no name. +* `options` {Object} + * `only` {boolean} When any benchmark or containing suite has `only` set, + benchmarks without `only` in their hierarchy are skipped. **Default:** + `false`. + * `params` {Object} String, finite number, or boolean metadata identifying + this benchmark configuration. Parameter keys are sorted when constructing + the stable benchmark identity. **Default:** An empty object. + * `samples` {number} The number of measured callback invocations. Must be a + positive 32-bit unsigned integer. **Default:** `30`. + * `signal` {AbortSignal} Allows aborting this benchmark. + * `skip` {boolean|string} If truthy, the benchmark is skipped. A string is + included in the result as the skip reason. **Default:** `false`. + * `tags` {string\[]} Labels associated with the benchmark. Tags are + lowercased, deduplicated, and inherited from containing suites by union. + **Default:** `[]`. + * `timeout` {number} The number of milliseconds after which the benchmark + fails. **Default:** `Infinity`. + * `warmup` {number} The number of unreported callback invocations before + measured samples. Must be a 32-bit unsigned integer. **Default:** `0`. +* `fn` {Function|AsyncFunction} The benchmark function. It receives a + {BenchContext}. +* Returns: {Promise} Fulfilled with the benchmark result after a top-level + benchmark finishes, or with `undefined` immediately when declared in a + suite. + +Warmup invocations use the same callback and timing contract as measured +samples, but their samples are discarded. An exception, rejection, timeout, +abort, missing timing call, or duplicate timing call stops the current +benchmark. Later benchmarks continue to run. + +A timeout or abort cannot interrupt synchronous JavaScript and does not forcibly +cancel asynchronous work that ignores `context.signal`. + +The stable `benchId` is based on the source file, hierarchical suite and +benchmark names, and canonicalized parameters. Declaring the same identity +more than once reports an error rather than merging the samples. + +### `bench.skip([name][, options], fn)` + + + +Shorthand for `bench(name, { ...options, skip: true }, fn)`. + +### `bench.only([name][, options], fn)` + + + +Shorthand for `bench(name, { ...options, only: true }, fn)`. + +## `suite([name][, options], fn)` + + + +* `name` {string} The suite name. **Default:** The `name` property of `fn`, or + `''` when `fn` has no name. +* `options` {Object} + * `only` {boolean} Selects all benchmarks nested in this suite. **Default:** + `false`. + * `skip` {boolean|string} Skips all benchmarks nested in this suite. + **Default:** `false`. + * `tags` {string\[]} Labels inherited by nested suites and benchmarks. + **Default:** `[]`. +* `fn` {Function|AsyncFunction} A function that declares nested suites, + benchmarks, and hooks. +* Returns: {Promise} Fulfilled when a top-level suite finishes, or with + `undefined` immediately when declared in another suite. + +Suite functions run while declarations are collected. Promise-returning suite +functions are awaited before benchmark execution begins. + +## `describe([name][, options], fn)` + + + +Alias for `suite()`. + +## `before(fn)` + + + +* `fn` {Function|AsyncFunction} The hook function. + +Registers a hook that runs once before the benchmarks in the current suite. + +## `after(fn)` + + + +* `fn` {Function|AsyncFunction} The hook function. + +Registers a hook that runs once after the benchmarks in the current suite. + +## `beforeEach(fn)` + + + +* `fn` {Function|AsyncFunction} The hook function. It receives an object with + the benchmark's `name`, `params`, and `signal`. + +Registers a hook that runs once before each complete logical benchmark in the +current suite. It does not run before every sample. Per-sample setup belongs in +the benchmark function before `context.start()`. + +## `afterEach(fn)` + + + +* `fn` {Function|AsyncFunction} The hook function. It receives an object with + the benchmark's `name`, `params`, and `signal`. + +Registers a hook that runs once after each complete logical benchmark in the +current suite. It does not run after every sample. Per-sample cleanup belongs +in the benchmark function after `context.end()`. + +## `run([options])` + + + +* `options` {Object} + * `namePattern` {string|RegExp} Only runs benchmarks whose full hierarchical + name matches the pattern. String values are interpreted as JavaScript + regular expressions. + * `samples` {number} Overrides the number of measured callback invocations + for every benchmark. Must be a positive 32-bit unsigned integer. + * `signal` {AbortSignal} Allows aborting in-progress benchmark execution. + * `warmup` {number} Overrides the number of unreported warmup callback + invocations for every benchmark. Must be a 32-bit unsigned integer. +* Returns: {BenchmarksStream} + +Returns the object-mode event stream for the in-process benchmark run. Call +`run()` during the same turn in which benchmarks are declared, before automatic +execution begins. Calling `run()` is optional when the returned stream is not +needed. + +```mjs +import { bench, run } from 'node:bench'; + +bench('example', { samples: 3 }, (b) => { + b.start(); + doWork(); + b.end(1); +}); + +for await (const { type, data } of run()) { + if (type === 'bench:complete' && data.error === undefined) { + console.log(data.name, data.summary.mean); + } +} +``` + +## Class: `BenchContext` + +An instance of `BenchContext` is passed to every benchmark invocation. A new +instance is created for every warmup and measured sample. + +### `context.name` + + + +* {string} + +The benchmark name. + +### `context.params` + + + +* {Object} + +The benchmark's canonicalized parameter metadata. + +### `context.signal` + + + +* {AbortSignal} + +An abort signal that is triggered when the benchmark is aborted, times out, or +finishes. + +### `context.start()` + + + +Starts the measured region using `process.hrtime.bigint()`. Calling `start()` +more than once is an error. + +### `context.end(operations)` + + + +* `operations` {number} The number of completed operations. Must be a positive + safe integer. + +Ends the measured region. The end timestamp is captured before `operations` is +validated. Calling `end()` before `start()`, calling it more than once, or +recording a zero-duration sample is an error. + +## Class: `BenchmarksStream` + +`BenchmarksStream` is an object-mode {stream.Readable}. Each lifecycle record is +both emitted as a named event and made available on the stream as +`{ type, data }`. + +The events are emitted in execution order: + +* `'bench:start'` +* `'bench:sample'` +* `'bench:complete'` +* `'bench:diagnostic'` +* `'bench:summary'` + +Every benchmark-scoped event contains `benchId` and `parentId`. +`'bench:complete'` data contains a [benchmark result][]. A failed result has an +additional `error` property and may contain samples recorded before the error. +A skipped result has an additional `skip` property and an empty `samples` +array. `'bench:diagnostic'` reports suite and hook errors. `'bench:summary'` +contains overall `success`, `counts`, `duration_ns`, and `file` properties. The +`file` is {string|null}; it is `null` when the summary aggregates multiple +files. + +## Sample result + +Each measured sample has the following properties: + +* `operations` {number} The positive operation count passed to + `context.end()`. +* `duration_ns` {bigint} The measured duration in nanoseconds. +* `rate` {number} Operations per second. + +## Benchmark result + +A completed benchmark result contains: + +* `benchId` {string} The stable benchmark identity. +* `parentId` {string|null} The stable containing suite identity. +* `name` {string} The benchmark name. +* `file` {string} The source file. +* `line` {number} The source line. +* `column` {number} The source column. +* `tags` {string\[]} The inherited canonical tags. +* `params` {Object} The canonical parameter metadata. +* `samples` {Object\[]} The exact measured samples. +* `summary` {Object} + * `mean` {number} The arithmetic mean of per-sample rates. + * `median` {number} The median per-sample rate. + * `min` {number} The minimum per-sample rate. + * `max` {number} The maximum per-sample rate. + * `stddev` {number} The population standard deviation of rates. + * `coefficientOfVariation` {number} `stddev / mean`. + * `confidenceInterval` {Object} The 95% Student's t confidence interval for + the mean rate, with `lower` and `upper` properties. + * `medianConfidenceInterval` {Object} The 95% nonparametric confidence + interval for the median rate, with `lower` and `upper` properties. + * `skewness` {number} The skewness of the scaled rate histogram. + +[benchmark result]: #benchmark-result +[command-line options documentation]: cli.md#--bench diff --git a/doc/api/cli.md b/doc/api/cli.md index 6ba509d25692..5ea579702ca4 100644 --- a/doc/api/cli.md +++ b/doc/api/cli.md @@ -450,6 +450,109 @@ Error: Access to this API has been restricted } ``` +### `--bench` + + + +> Stability: 1 - Experimental + +Starts the Node.js command-line benchmark runner. At least one explicit file or +glob pattern is required: + +```console +node --bench benchmark.mjs +node --bench 'benchmarks/**/*.js' +``` + +Quote glob patterns to prevent expansion by the shell. Matching files are +sorted and executed serially. By default, each file runs in a separate child +process. Benchmark files declare benchmarks using `node:bench`; they must not +call `run()` themselves. See the [benchmark runner][] documentation for more +details. + +This flag cannot be combined with `--test`, `--watch`, `--watch-path`, +`--check`, `--eval`, or `--interactive`. + +### `--bench-isolation=mode` + + + +> Stability: 1 - Experimental + +Configures benchmark file isolation. When `mode` is `'process'`, each matching +file runs in a separate child process. This is the default. Files are still run +serially so their measured work does not overlap. + +When `mode` is `'none'`, all matching files and benchmarks run serially in the +benchmark runner process. This reduces startup overhead but allows module, +heap, and process state to carry between files. User writes to stdout or stderr +also share destinations with benchmark reporters in this mode. + +### `--bench-name-pattern=pattern` + + + +> Stability: 1 - Experimental + +Only runs benchmarks whose full hierarchical name matches the JavaScript +regular expression `pattern`. Non-matching benchmarks are reported as skipped. + +### `--bench-reporter-destination=destination` + + + +> Stability: 1 - Experimental + +Specifies the destination for the corresponding benchmark reporter. The value +can be `stdout`, `stderr`, or a file path. A single reporter defaults to +`stdout` when no destination is specified. + +### `--bench-reporter=reporter` + + + +> Stability: 1 - Experimental + +Specifies a benchmark reporter. The built-in reporters are `spec` and `json`. +The `json` reporter emits newline-delimited JSON. A custom reporter can be +specified using a module specifier resolved from the current working directory. + +This option can be repeated. When multiple reporters are specified, each must +have a corresponding `--bench-reporter-destination`. The default reporter is +`spec`. + +### `--bench-samples=count` + + + +> Stability: 1 - Experimental + +Overrides the number of measured callback invocations for every selected +benchmark. `count` must be an integer between `1` and `4294967295`. + +### `--bench-warmup=count` + + + +> Stability: 1 - Experimental + +Overrides the number of unreported warmup callback invocations for every +selected benchmark. `count` must be an integer between `0` and `4294967295`. + ### `--build-sea=config` + +* `options` {Object} + * `confidence` {number} The confidence level for the interval, between + 0 and 1 (exclusive). **Default:** `0.95`. +* Returns: {Object} + * `mean` {number} The mean estimate, equivalent to `histogram.mean`. + * `lower` {number} The lower bound of the confidence interval. + * `upper` {number} The upper bound of the confidence interval. + +Returns a two-sided confidence interval for the mean using Student's +t-distribution and the sample standard error. A higher confidence level +produces a wider interval. This interval assumes that samples are independent +and approximately normally distributed, although the approximation is robust +for sufficiently large samples. + +The result reflects the histogram's configured precision and is calculated +from the values represented by its buckets. With fewer than two recorded +values, `lower` and `upper` are `NaN`. When all recorded values are equal, +`lower` and `upper` equal `mean`. + +```js +const { createHistogram } = require('node:perf_hooks'); + +const h = createHistogram(); +for (let i = 1; i <= 100; i++) h.record(i); + +const { mean, lower, upper } = h.meanCI(); +console.log(`mean=${mean}, 95% CI=[${lower}, ${upper}]`); +``` + ### `histogram.min`