perf: experiment with striped classic histogram accumulator

[zeitlinger]

Summary

This is an experimental alternate implementation for benchmarking against main and the synchronized candidate in #2252. It is not being presented as production-ready yet.

For classic-only histograms, this replaces the existing Buffer/LongAdder recording path with an epoch-striped, double-buffered per-thread accumulator. Each recording thread owns a cell, observations update plain thread-confined bucket/count/sum fields, and collection advances a global epoch before draining the inactive buffers. This avoids a global synchronized (DataPoint) monitor on the shared four-thread observe path while preserving coherent bucket/count/sum snapshots. getCount() and getSum() use the same snapshot mechanism.

Native-only and hybrid histograms keep their existing implementation. The public API is unchanged.

Local evidence

• mise run lint
• ./mvnw -pl prometheus-metrics-core -Dtest=HistogramTest test — 35 tests passed
• focused concurrent collect/lifecycle tests repeated 5 times
• ./mvnw -pl prometheus-metrics-core -am verify — reactor passed; core ran 157 tests
• git diff --check
• representative local JMH smoke run (JDK 25.0.3, 4 threads, 1 fork, 3x1s warmup, 5x1s measurement):
  • HistogramBenchmark.prometheusClassic: 40,944.515 ± 302.666 ops/s

The local JMH number is a smoke result, not a controlled cross-branch conclusion. The benchmark workflow is being requested so this branch and main are measured on the same runner.

Added concurrency coverage

• collection while eight shared observers record concurrently
• monotonic and coherent bucket/count/sum snapshots
• exact final getCount() / getSum() values
• retention of observations from short-lived recording threads
• clear() lifecycle behavior for old and newly-created data points

Remaining hardening before merge consideration

• Review/validate the epoch handoff under the Java Memory Model, ideally with dedicated jcstress coverage rather than relying only on repeated JUnit stress tests.
• Decide how to bound or reclaim retained per-thread cells for workloads with high thread churn; cells currently live for the data point lifetime so completed-thread observations cannot be lost.
• Measure scrape latency and the spin-wait behavior when a recording thread is delayed after entering an epoch.
• Measure the cost of getCount() / getSum(), which currently perform an epoch flip and cell scan to return exact accumulated values.
• Run the full CI benchmark matrix and test additional supported JDKs/architectures before treating this as mergeable.

Related: #2252

[github-actions]

⚠️ API changes detected — maintainer review required

This PR modifies the published API diff for the following module(s):

• prometheus-metrics-core

Please review the changes in docs/apidiffs/current_vs_latest/ carefully before approving.

[github-actions]

⚠️ API changes detected — maintainer review required

This PR modifies the published API diff for the following module(s):

• prometheus-metrics-core

Please review the changes in docs/apidiffs/current_vs_latest/ carefully before approving.

[github-actions]

Benchmark results

Benchmark run succeeded for e3e9831f89753bb4666704c0df3e427e7034805c.

• Workflow run: https://github.com/prometheus/client_java/actions/runs/32158203108
• Artifact: pr-benchmark-results-pr-2329-e3e9831f89753bb4666704c0df3e427e7034805c

Prometheus Java Client Benchmarks

Run Information

• Date: 2026-08-18T17:25:54Z
• Commit: e3e9831
• JDK: 25.0.3 (OpenJDK 64-Bit Server VM)
• Benchmark config: 3 fork(s), 3 warmup, 5 measurement, 4 threads
• Hardware: AMD EPYC 7763 64-Core Processor, 4 cores, 16 GB RAM
• OS: Linux 6.17.0-1022-azure

Comparison with base

• Head: e3e9831
• Base: c8e2c03
• Change: positive means the PR is faster than base.
• Note: Base and head are compared on the same runner within each topic.

Benchmark	PR	Base	Change	Result
CounterBenchmark.codahaleIncNoLabels	49.50K	49.26K	+0.5%	within noise
CounterBenchmark.openTelemetryAdd	13.01K	12.90K	+0.8%	within noise
CounterBenchmark.openTelemetryInc	14.76K	15.24K	-3.2%	within noise
CounterBenchmark.openTelemetryIncNoLabels	18.63K	18.50K	+0.7%	within noise
CounterBenchmark.prometheusAdd	51.32K	51.45K	-0.3%	within noise
CounterBenchmark.prometheusInc	65.83K	65.67K	+0.2%	within noise
CounterBenchmark.prometheusNoLabelsInc	57.10K	56.77K	+0.6%	within noise
CounterBenchmark.simpleclientAdd	6.32K	6.49K	-2.6%	within noise
CounterBenchmark.simpleclientInc	6.57K	6.55K	+0.3%	within noise
CounterBenchmark.simpleclientNoLabelsInc	6.34K	6.29K	+0.8%	within noise
HistogramBenchmark.openTelemetryClassic	626.45	626.14	+0.0%	within noise
HistogramBenchmark.openTelemetryExponential	493.53	523.41	-5.7%	within noise
HistogramBenchmark.prometheusClassic	9.50K	2.95K	+222.0%	faster
HistogramBenchmark.prometheusClassicPerThread	9.97K	7.80K	+27.9%	faster
HistogramBenchmark.prometheusClassicSingleThread	4.07K	3.27K	+24.3%	faster
HistogramBenchmark.prometheusNative	2.19K	2.26K	-3.1%	within noise
HistogramBenchmark.simpleclient	4.49K	4.47K	+0.6%	within noise
HistogramTextFormatBenchmark.openMetricsWriteToNull	27.42K	27.49K	-0.2%	within noise
HistogramTextFormatBenchmark.prometheusWriteToNull	27.21K	27.52K	-1.1%	within noise
TextFormatUtilBenchmark.openMetricsWriteToByteArray	538.52K	540.33K	-0.3%	within noise
TextFormatUtilBenchmark.openMetricsWriteToNull	549.63K	550.96K	-0.2%	within noise
TextFormatUtilBenchmark.prometheusWriteToByteArray	573.97K	577.01K	-0.5%	within noise
TextFormatUtilBenchmark.prometheusWriteToNull	582.94K	584.44K	-0.3%	within noise

• Benchmarks only in PR results: HistogramBenchmark.prometheusClassicCollectAfterThreadChurn, HistogramBenchmark.prometheusClassicGetCountAfterThreadChurn

Results for PR head

CounterBenchmark

Benchmark	Score	Error	Units	Within run
prometheusInc	65.83K	± 462.31	ops/s	fastest
prometheusNoLabelsInc	57.10K	± 79.91	ops/s	1.2x slower
prometheusAdd	51.32K	± 182.12	ops/s	1.3x slower
codahaleIncNoLabels	49.50K	± 2.26K	ops/s	1.3x slower
openTelemetryIncNoLabels	18.63K	± 95.22	ops/s	3.5x slower
openTelemetryInc	14.76K	± 315.48	ops/s	4.5x slower
openTelemetryAdd	13.01K	± 41.34	ops/s	5.1x slower
simpleclientInc	6.57K	± 29.45	ops/s	10x slower
simpleclientNoLabelsInc	6.34K	± 8.45	ops/s	10x slower
simpleclientAdd	6.32K	± 172.40	ops/s	10x slower

HistogramBenchmark

Benchmark	Score	Error	Units	Within run
prometheusClassicPerThread	9.97K	± 26.78	ops/s	fastest
prometheusClassic	9.50K	± 192.73	ops/s	1.0x slower
simpleclient	4.49K	± 46.71	ops/s	2.2x slower
prometheusClassicSingleThread	4.07K	± 30.31	ops/s	2.5x slower
prometheusClassicGetCountAfterThreadChurn	3.82K	± 415.44	ops/s	2.6x slower
prometheusClassicCollectAfterThreadChurn	3.19K	± 150.52	ops/s	3.1x slower
prometheusNative	2.19K	± 133.25	ops/s	4.6x slower
openTelemetryClassic	626.45	± 32.18	ops/s	16x slower
openTelemetryExponential	493.53	± 33.22	ops/s	20x slower

HistogramTextFormatBenchmark

Benchmark	Score	Error	Units	Within run
openMetricsWriteToNull	27.42K	± 162.29	ops/s	fastest
prometheusWriteToNull	27.21K	± 688.73	ops/s	1.0x slower

TextFormatUtilBenchmark

Benchmark	Score	Error	Units	Within run
prometheusWriteToNull	582.94K	± 7.18K	ops/s	fastest
prometheusWriteToByteArray	573.97K	± 4.89K	ops/s	1.0x slower
openMetricsWriteToNull	549.63K	± 3.80K	ops/s	1.1x slower
openMetricsWriteToByteArray	538.52K	± 2.32K	ops/s	1.1x slower

Raw Results

Benchmark                                            Mode  Cnt          Score        Error  Units
CounterBenchmark.codahaleIncNoLabels                thrpt   15      49497.938   ± 2264.375  ops/s
CounterBenchmark.openTelemetryAdd                   thrpt   15      13008.680     ± 41.336  ops/s
CounterBenchmark.openTelemetryInc                   thrpt   15      14763.480    ± 315.483  ops/s
CounterBenchmark.openTelemetryIncNoLabels           thrpt   15      18632.535     ± 95.217  ops/s
CounterBenchmark.prometheusAdd                      thrpt   15      51316.265    ± 182.120  ops/s
CounterBenchmark.prometheusInc                      thrpt   15      65825.899    ± 462.308  ops/s
CounterBenchmark.prometheusNoLabelsInc              thrpt   15      57097.040     ± 79.911  ops/s
CounterBenchmark.simpleclientAdd                    thrpt   15       6320.744    ± 172.398  ops/s
CounterBenchmark.simpleclientInc                    thrpt   15       6572.990     ± 29.447  ops/s
CounterBenchmark.simpleclientNoLabelsInc            thrpt   15       6336.473      ± 8.453  ops/s
HistogramBenchmark.openTelemetryClassic             thrpt   15        626.448     ± 32.182  ops/s
HistogramBenchmark.openTelemetryExponential         thrpt   15        493.525     ± 33.218  ops/s
HistogramBenchmark.prometheusClassic                thrpt   15       9503.102    ± 192.726  ops/s
HistogramBenchmark.prometheusClassicCollectAfterThreadChurn  thrpt   15       3186.346    ± 150.525  ops/s
HistogramBenchmark.prometheusClassicGetCountAfterThreadChurn  thrpt   15       3817.245    ± 415.436  ops/s
HistogramBenchmark.prometheusClassicPerThread       thrpt   15       9973.088     ± 26.784  ops/s
HistogramBenchmark.prometheusClassicSingleThread    thrpt   15       4065.719     ± 30.306  ops/s
HistogramBenchmark.prometheusNative                 thrpt   15       2189.963    ± 133.248  ops/s
HistogramBenchmark.simpleclient                     thrpt   15       4493.281     ± 46.706  ops/s
HistogramTextFormatBenchmark.openMetricsWriteToNull  thrpt   15      27418.581    ± 162.295  ops/s
HistogramTextFormatBenchmark.prometheusWriteToNull  thrpt   15      27211.025    ± 688.735  ops/s
TextFormatUtilBenchmark.openMetricsWriteToByteArray  thrpt   15     538520.665   ± 2319.899  ops/s
TextFormatUtilBenchmark.openMetricsWriteToNull      thrpt   15     549625.489   ± 3800.864  ops/s
TextFormatUtilBenchmark.prometheusWriteToByteArray  thrpt   15     573968.835   ± 4894.838  ops/s
TextFormatUtilBenchmark.prometheusWriteToNull       thrpt   15     582941.051   ± 7181.812  ops/s

Notes

• Score = Throughput in operations per second (higher is better)
• Error = 99.9% confidence interval
• Comparison with base uses JMH confidence intervals when available; overlapping intervals are marked "within noise".
• Within run compares benchmarks in the same result set, not against the base commit.

Benchmark Descriptions

Benchmark	Description
CounterBenchmark	Counter increment performance: Prometheus, OpenTelemetry, simpleclient, Codahale
HistogramBenchmark	Histogram observation performance (classic vs native/exponential)
TextFormatUtilBenchmark	Metric exposition format writing speed

[zeitlinger]

Evaluation (local, JDK 25.0.4, x86_64, 1 fork / 2 warmups / 5 measurements; indicative rather than production benchmark):

• Shared-observer JMH: perf: experiment with striped classic histogram accumulator #2329 44.4K ops/s vs perf: optimize classic-only histogram observe path #2252 1.63K ops/s; per-thread perf: experiment with striped classic histogram accumulator #2329 44.9K vs perf: optimize classic-only histogram observe path #2252 64.7K.
• A temporary churn probe created 1,000/5,000 retained cells from one-shot threads; getCount() grew from about 1 ms to 5 ms, and collection was about 13.8 ms at 5,000 cells.

Recommendation: use #2329 as the basis, not #2252, but do not merge yet. Please add jcstress/JMM coverage, bound or reclaim retained cells, and define bounded scrape progress/latency when a writer is delayed. Existing focused concurrency tests pass repeatedly, but they do not close those risks.

[zeitlinger]

Hardening update: pushed 03c3da30 (includes the additive merge of current main).

• Snapshot waits are bounded to a 1 ms total budget; a delayed writer's buffer is skipped safely and collected on a later epoch flip, so scrapes cannot spin indefinitely.
• Empty cells are safely deregistered/reclaimed after both buffers are collected and re-register when a thread-local is reused.
• Added JMM/concurrency coverage for stalled writers, eventual delayed-buffer collection, cell reclamation/reuse, and 8 concurrent writers (80,000 observations with coherent bucket/count/sum results).
• Added JMH probes for 1,000 short-lived recorder threads and the resulting epoch flip/cell scan. Local JDK 25.0.4, x86_64, 1 fork, 1×100 ms warmup, 2×100 ms measurement: collect 1,644 ops/s; getCount 1,941 ops/s. Thread setup is outside the timed method; these are indicative scrape/scan numbers, not a cross-branch comparison.

Focused accumulator/Histogram tests, repeated concurrency tests, benchmark compilation, and formatter/lint pass locally. Full CI benchmark matrix and cross-JDK/architecture validation remain appropriate before merge.

[zeitlinger]

Follow-up hardening pushed in ebe3d8e8:

• Removed the shared (and per-cell) spin/wait budget entirely. A snapshot now drains a cell immediately when its inactive buffer is not being written, and skips only an actively-writing same-parity cell. This makes total handoff latency proportional to the current cell/bucket scan, independent of any writer stall; delayed cells are intentionally stale until a later snapshot.
• Added a multi-cell regression (one stalled cell plus four healthy cells) proving healthy cells are drained, plus active-writer pause/resume coverage.
• Added explicit comments documenting stale collect(), getCount(), and getSum() semantics.

Local focused tests: 5 accumulator tests and 35 Histogram tests pass, with the accumulator suite repeated five times. JMH smoke (JDK 25.0.4, x86_64, 1 fork, 1×100 ms warmup, 2×100 ms measurement, 1,000 short-lived recorders; setup excluded): collect 1,218 ops/s and getCount 1,600 ops/s. These are indicative scan-cost measurements only.

The current native-tests CI failure is unrelated infrastructure: Maven Wrapper could not download Maven 3.9.16 from repo.maven.apache.org. No review threads are open.

[zeitlinger]

JCStress evidence added in 37792360:

• Added a dedicated prometheus-metrics-jcstress module using JCStress 0.16, with tests for epoch publication, stalled-writer parity and eventual collection, cell reclamation/re-registration, and concurrent observation count/sum coherence.
• Built the shaded harness and ran the four accumulator tests on JDK 25.0.4 with 2 CPUs, sanity mode, 1 normal + 1 stress fork, 5 iterations × 20 ms: 112/112 planned results passed; 0 forbidden/error results.
• Existing focused ClassicOnlyAccumulatorTest/HistogramTest, full Maven package, and Flint formatter/lint pass.

The stalled-writer test uses reflection only in the JCStress test module to model a pause at the internal epoch boundary; production code has no test hooks. Cross-JDK/architecture stress coverage remains for CI/maintainer validation.