fix(telemetry): do not export instruments carrying no data points - #216
Open
abnegate wants to merge 2 commits into
Open
fix(telemetry): do not export instruments carrying no data points#216abnegate wants to merge 2 commits into
abnegate wants to merge 2 commits into
Conversation
An asynchronous instrument whose callback records nothing still reaches the exporter as a metric with an empty data point list. Prometheus 3.13+ answers such a payload with HTTP 500 for the entire OTLP request and rolls back everything already appended, so one empty instrument discards every healthy metric batched alongside it. This is live in production. Three edge instruments partition their input across gauges, so the healthy steady state guarantees one of them observes nothing, and the resulting 500s ran flat at ~100-130 per 10 minutes for over 30 hours. Whole batches were lost with them: the edge database backup scheduler stopped reporting entirely in two regions, taking the arrears alerting blind with it. The decorator keeps the AggregationTemporalitySelectorInterface delegation deliberately. ExportingReader::add() registers no metric source at all when the exporter does not implement it, so dropping it would silently export nothing. Upstream: prometheus/prometheus#19338 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…s built Wrapping inside createExporter() made the filter something each exporter construction site had to opt into, so a subclass overriding createExporter() silently lost it. Wrapping where the reader is built applies it to whatever exporter arrives instead, override or not, and no caller wraps anything. SkipEmpty now takes MetricExporterInterface rather than the intersection type, because that is what initMeter() receives. ExportingReader probes the exporter for AggregationTemporalitySelectorInterface and PushMetricExporter- Interface and changes behaviour when either is absent, so both are delegated only when the wrapped exporter provides them; answering null temporality for an exporter that selects none leaves ExportingReader::add() on its original early return. testAnOverriddenExporterIsStillFiltered pins the behaviour and was confirmed red under the previous placement, with aggregationTemporality=1 in the leaked payload proving the subclass exporter was genuinely in use. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What breaks
An asynchronous instrument whose callback records nothing still reaches the exporter as a metric with an empty data point list. Serialized, it looks like this:
{"name":"never.observed","gauge":{}}Prometheus answers such a payload with HTTP 500 for the entire OTLP request and runs
app.Rollback()in a defer, so everything already appended is thrown away. One empty instrument therefore discards every healthy metric batched alongside it. Prometheus 3.5.0 logged the same condition as a warning and accepted the rest of the batch, which is why this only surfaced after the upgrade.This is live in production
Three edge instruments partition their input across separate gauges, so the healthy steady state — nothing in arrears — guarantees one of them observes nothing. The resulting 500s ran flat at ~100-130 per 10 minutes across a 30-hour window in two production regions.
The collateral is what matters:
edge_database_backup_schedule_age_secondsrides the same batch and observes unconditionally, and it was unobservable for roughly half of a four-hour window in both regions, taking the backup arrears alerting blind with it. A critical liveness alert fired for a service that was healthy the whole time.The fix
SkipEmptywraps the metric exporter and drops metrics whose data point list is empty before they are serialized. Populated metrics, and metrics whose data points are a lazyTraversable, pass through untouched — the check is=== [], so nothing is consumed to decide.It is applied where the reader is built, not in
createExporter(). That matters: increateExporter()the filter would be something each exporter construction site had to opt into, and a subclass overridingcreateExporter()would silently lose it. Applied at the reader it covers whatever exporter arrives, and no caller wraps anything — consumers get it oncomposer update.SkipEmptytakesMetricExporterInterfacerather than an intersection type, because that is whatinitMeter()receives.ExportingReaderprobes the exporter forAggregationTemporalitySelectorInterfaceandPushMetricExporterInterfaceand changes behaviour when either is absent, so both are delegated only when the wrapped exporter provides them. This is load-bearing rather than defensive:ExportingReader::add()registers no metric source at all when the exporter has no temporality selector, so a decorator that dropped it would silently export nothing.A decorator is the only interception point available —
MetricExporterisfinal.Verification
Tests drive the public path —
createObservableGauge()→observe()→collect()→ transport — rather than the decorator in isolation. Both behaviours were confirmed red before their respective fixes.Without the filter:
With the filter in
createExporter()instead of at the reader:The leaked payload in that second case carries
aggregationTemporality: 1(DELTA), confirming the subclass's own exporter was genuinely in use rather than the test passing vacuously.bin/monorepo test telemetry— 35 tests, 77 assertions, green.bin/monorepo check telemetry— pint and rector clean. PHPStan reports 5 errors, all inAdapter/Test.php, which is byte-identical tomain; this branch adds none.bin/monorepo validate— all packages valid.Upstream
This is a workaround. The SDK arguably should not emit an envelope for an instrument that recorded no measurements, filed as open-telemetry/opentelemetry-php#2052 with a standalone reproduction. The Prometheus side is prometheus/prometheus#19338, where the whole-batch rejection is a regression from the AppenderV2 switch;
mainis still affected there, so waiting it out is not an option.🤖 Generated with Claude Code