Skip to content

fix(telemetry): do not export instruments carrying no data points - #216

Open
abnegate wants to merge 2 commits into
mainfrom
fix/telemetry-skip-empty-metrics
Open

fix(telemetry): do not export instruments carrying no data points#216
abnegate wants to merge 2 commits into
mainfrom
fix/telemetry-skip-empty-metrics

Conversation

@abnegate

@abnegate abnegate commented Sep 7, 2026

Copy link
Copy Markdown
Member

What breaks

An asynchronous instrument whose callback records nothing still reaches the exporter as a metric with an empty data point list. Serialized, it looks like this:

{"name":"never.observed","gauge":{}}

Prometheus answers such a payload with HTTP 500 for the entire OTLP request and runs app.Rollback() in a defer, so everything already appended is thrown away. One empty instrument therefore discards every healthy metric batched alongside it. Prometheus 3.5.0 logged the same condition as a warning and accepted the rest of the batch, which is why this only surfaced after the upgrade.

This is live in production

Three edge instruments partition their input across separate gauges, so the healthy steady state — nothing in arrears — guarantees one of them observes nothing. The resulting 500s ran flat at ~100-130 per 10 minutes across a 30-hour window in two production regions.

The collateral is what matters: edge_database_backup_schedule_age_seconds rides the same batch and observes unconditionally, and it was unobservable for roughly half of a four-hour window in both regions, taking the backup arrears alerting blind with it. A critical liveness alert fired for a service that was healthy the whole time.

The fix

SkipEmpty wraps the metric exporter and drops metrics whose data point list is empty before they are serialized. Populated metrics, and metrics whose data points are a lazy Traversable, pass through untouched — the check is === [], so nothing is consumed to decide.

It is applied where the reader is built, not in createExporter(). That matters: in createExporter() the filter would be something each exporter construction site had to opt into, and a subclass overriding createExporter() would silently lose it. Applied at the reader it covers whatever exporter arrives, and no caller wraps anything — consumers get it on composer update.

SkipEmpty takes MetricExporterInterface rather than an intersection type, because that is what initMeter() receives. ExportingReader probes the exporter for AggregationTemporalitySelectorInterface and PushMetricExporterInterface and changes behaviour when either is absent, so both are delegated only when the wrapped exporter provides them. This is load-bearing rather than defensive: ExportingReader::add() registers no metric source at all when the exporter has no temporality selector, so a decorator that dropped it would silently export nothing.

A decorator is the only interception point available — MetricExporter is final.

Verification

Tests drive the public path — createObservableGauge()observe()collect() → transport — rather than the decorator in isolation. Both behaviours were confirmed red before their respective fixes.

Without the filter:

1) testInstrumentThatRecordsNothingIsNotExported
Failed asserting that '...{"name":"never.observed","gauge":{}},{"name":"always.counted","sum":{...}}...'
does not contain "never.observed".

With the filter in createExporter() instead of at the reader:

1) testAnOverriddenExporterIsStillFiltered
Filtering must not depend on createExporter() remembering to apply it
Failed asserting that '...{"name":"never.observed","gauge":{}}...' does not contain "never.observed".

The leaked payload in that second case carries aggregationTemporality: 1 (DELTA), confirming the subclass's own exporter was genuinely in use rather than the test passing vacuously.

bin/monorepo test telemetry — 35 tests, 77 assertions, green.
bin/monorepo check telemetry — pint and rector clean. PHPStan reports 5 errors, all in Adapter/Test.php, which is byte-identical to main; this branch adds none.
bin/monorepo validate — all packages valid.

Upstream

This is a workaround. The SDK arguably should not emit an envelope for an instrument that recorded no measurements, filed as open-telemetry/opentelemetry-php#2052 with a standalone reproduction. The Prometheus side is prometheus/prometheus#19338, where the whole-batch rejection is a regression from the AppenderV2 switch; main is still affected there, so waiting it out is not an option.

🤖 Generated with Claude Code

An asynchronous instrument whose callback records nothing still reaches
the exporter as a metric with an empty data point list. Prometheus 3.13+
answers such a payload with HTTP 500 for the entire OTLP request and rolls
back everything already appended, so one empty instrument discards every
healthy metric batched alongside it.

This is live in production. Three edge instruments partition their input
across gauges, so the healthy steady state guarantees one of them observes
nothing, and the resulting 500s ran flat at ~100-130 per 10 minutes for
over 30 hours. Whole batches were lost with them: the edge database backup
scheduler stopped reporting entirely in two regions, taking the arrears
alerting blind with it.

The decorator keeps the AggregationTemporalitySelectorInterface delegation
deliberately. ExportingReader::add() registers no metric source at all when
the exporter does not implement it, so dropping it would silently export
nothing.

Upstream: prometheus/prometheus#19338

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…s built

Wrapping inside createExporter() made the filter something each exporter
construction site had to opt into, so a subclass overriding createExporter()
silently lost it. Wrapping where the reader is built applies it to whatever
exporter arrives instead, override or not, and no caller wraps anything.

SkipEmpty now takes MetricExporterInterface rather than the intersection
type, because that is what initMeter() receives. ExportingReader probes the
exporter for AggregationTemporalitySelectorInterface and PushMetricExporter-
Interface and changes behaviour when either is absent, so both are delegated
only when the wrapped exporter provides them; answering null temporality for
an exporter that selects none leaves ExportingReader::add() on its original
early return.

testAnOverriddenExporterIsStillFiltered pins the behaviour and was confirmed
red under the previous placement, with aggregationTemporality=1 in the leaked
payload proving the subclass exporter was genuinely in use.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant