Skip to content

Write global tags once on the OTLP trace resource - #13065

Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 3 commits into
masterfrom
matthew.li/otlp-resource-global-tags
Oct 28, 2026
Merged

gh-worker-dd-mergequeue-cf854d[bot] merged 3 commits into
masterfrom
matthew.li/otlp-resource-global-tags

Conversation

@mhlidd

@mhlidd mhlidd commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

What Does This Do

Writes the trace's global tags (DD_TAGS, or their remote config tracing_tags override) once, as OTLP resource attributes, instead of also repeating them on every exported span.

  • Resource from the config snapshot. The OTLP trace resource is now built from the trace's ConfigSnapshot.globalTags (via the new DDSpanContext.getGlobalTags()) instead of static Config, so it follows remote config tag changes. Each collector caches the encoded resource and re-encodes only when the global-tags map changes.
  • Span attributes. The new OtlpSpanTags writes a span's direct tags (new TagMap.forEachLocal) plus the span defaults (Metadata.getSpanDefaultTags(): DD_TRACE_SPAN_TAGS, env, dsm/djm) the span still carries and doesn't set itself. Global tags inherited through the read-through parent are left to the resource. A span default the span removed, or that interception consumed into a span field (e.g. resource.name), is not written.
  • One resource per payload. OtlpPayloadDispatcher flushes the current payload before adding a trace whose global tags differ from the payload's (OtlpTraceCollector.sharesResource), so a remote config change mid-batch can't put traces under the wrong resource.
  • Fallback. A trace whose config snapshot can't be read (not a DDSpan) resolves to the startup global tags, so its resource keeps Config's global tags as a floor and a trace with different tags still gets its own resource.

Motivation

Global tags were exported twice: on the resource (built once from static Config) and on every span via the span's read-through parent. The static resource also went stale when remote config changed tracing_tags.

This supersedes #12813, rebuilt on #12937 / #12960 as suggested in review: no parent() / readsThroughToParent API and no value comparison against Config; the resource comes from the snapshot's global tags and spans write their direct tags plus span defaults.

Additional Notes

  • Known gaps (unchanged from master):
    • When the tracer tags need interception they're copied into each span, so global tags still also appear on the span (same value as the resource).
    • Agentless intake lets the resource attribute win over a span's own override of a global tag; Agent / DDOT let the span win.
  • Tag naming follows isTraceOtelSemanticsEnabled. Config.isTraceOtelSemanticsEnabled() (DD_TRACE_OTEL_SEMANTICS_ENABLED) decides which tag names the tracer exports over OTLP, for span attributes and now for the global tags on the resource alike: the OpenTelemetry name when enabled (e.g. db.system, http.request.method), the Datadog name otherwise (db.type, http.method). The resource's datadog.sdk.semantics attribute (otel / datadog) records the same choice. Global tags from the snapshot are a TagMap, which stores known tags under their Datadog name, so a key configured in the other spelling is normalized and two spellings of one tag collapse to one key. Metrics / logs resources keep Config's keys as configured.
  • Baggage (Export propagation tags in OTLP trace export #12985): baggage and propagation tags are still skipped when the span exports the same key. A baggage key matching a global tag is therefore skipped too, and that value is carried by the resource rather than the span.
  • Same-tags check is identity-based (config snapshots reuse the map while tags don't change). It can move to a TraceSegment stamp-then-equals check once TagMap gains equals / hashCode.
  • Interleaving after an RC change: traces created under the old and new snapshot can finish interleaved; each switch flushes a small payload until old traces drain. Bounded and only after a tag change.
  • Scope: traces only. OTLP metrics / logs resources still use the startup DD_TAGS (follow-up).
  • Tests: new OtlpSpanTagsTest (resource vs span attributes, span overrides, removed and consumed span defaults, sharesResource across tag maps) and TagMapReadThroughTest.forEachLocalSkipsParentEntries; existing OTLP trace/resource/dispatcher and TagMap* tests pass.

Contributor Checklist

🤖 Generated with Claude Code

@mhlidd mhlidd added type: feature Enhancements and improvements comp: core Tracer core tag: ai generated Largely based on code generated by an AI or LLM labels Oct 26, 2026
@mhlidd
mhlidd requested review from dougqh and mcculls October 26, 2026 20:50
@datadog-datadog-prod-us1

datadog-datadog-prod-us1 Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

🎯 Code Coverage (details)
• Patch Coverage: 90.48%
• Overall Coverage: 59.12% (-0.39%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: eb350ad | Docs | Give us feedback!

@dd-octo-sts

dd-octo-sts Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 14.78 s 14.71 s [-0.3%; +1.3%] (no difference)
startup:insecure-bank:tracing:Agent 13.55 s 13.69 s [-1.9%; -0.2%] (maybe better)
startup:petclinic:appsec:Agent 17.17 s 17.04 s [-0.1%; +1.6%] (no difference)
startup:petclinic:iast:Agent 16.98 s 17.13 s [-1.8%; +0.0%] (no difference)
startup:petclinic:profiling:Agent 16.88 s 16.97 s [-1.7%; +0.7%] (no difference)
startup:petclinic:sca:Agent 17.13 s 17.00 s [-0.1%; +1.7%] (no difference)
startup:petclinic:tracing:Agent 15.74 s 16.16 s [-6.9%; +1.6%] (no difference)

Commit: eb350ade · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

@dougqh dougqh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TagMap changes look good to me

Claude had a couple suggestions about other areas that you may want to address

@mhlidd
mhlidd marked this pull request as ready for review October 27, 2026 20:37
@mhlidd
mhlidd requested review from a team as code owners October 27, 2026 20:37
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-27T20:46:14.214111Z 5de3853 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5de38534f8

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@datadog-datadog-prod-us1 datadog-datadog-prod-us1 Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: FAIL

With OTel semantics enabled, DD_TAGS=db.system:postgresql loses its db.system attribute and is exported as db.type instead. Consumers relying on the configured OTLP attribute name will no longer find it.

Open Bits AI session

🤖 Bits Code Review · Commit 5de3853

@mcculls mcculls left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looking good - just the comments about canonical tags need checking over.

Also I agree with Doug that it would be nice to avoid prematurely flushing the payload if possible. That's best done as a follow-up PR though - basically you'd move that check into the JSON/Proto collectors and when the global tags differ you'd complete the current resourceSpans and start a new one.

@mcculls

mcculls commented Oct 28, 2026

Copy link
Copy Markdown
Contributor

Make sure to merge in latest origin/master to pick up the OTel semantic refactoring from @zacharycmontoya (it should merge cleanly)

Also could you coordinate with @anna-git since this overlaps slightly with #12985, basically agree which one will get merged first as the other will need to resolve any conflicts.

mhlidd and others added 3 commits October 28, 2026 13:02
Build the OTLP trace resource from each trace's config snapshot (DD_TAGS or
its remote config override) instead of static Config, and stop repeating
those global tags on every span. Spans keep their own tags plus the span
defaults they still carry. The payload is flushed when a trace's global
tags differ from the payload's, so each payload has one accurate resource.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A trace whose config snapshot can't be read (not a DDSpan) now resolves to
the startup global tags instead of null, so its payload's resource keeps
Config's global tags as a floor and a trace with different tags can't join
that payload and lose its own.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The trace's global tags come from its config snapshot as a TagMap, which
stores known tags under their Datadog name, so the resource emitted that
name even under OTel semantics (db.system came out as db.type). Name them
like span attributes instead: the OpenTelemetry name under OTel
semantics, else the Datadog name. Other exports keep Config's keys as
configured.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@mhlidd
mhlidd force-pushed the matthew.li/otlp-resource-global-tags branch from 4df682d to eb350ad Compare October 28, 2026 17:11
@mhlidd
mhlidd added this pull request to the merge queue Oct 28, 2026
@dd-octo-sts

dd-octo-sts Bot commented Oct 28, 2026

Copy link
Copy Markdown
Contributor

/merge

@gh-worker-devflow-routing-ef8351

gh-worker-devflow-routing-ef8351 Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

View all feedbacks in Devflow UI.

2026-10-28 18:17:50 UTC ℹ️ Start processing command /merge


2026-10-28 18:17:54 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in master is approximately 2h (p90).


2026-10-28 19:19:46 UTC ℹ️ MergeQueue: This merge request was merged

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 28, 2026
@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot merged commit d19c3c7 into master Oct 28, 2026
622 of 625 checks passed
@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot deleted the matthew.li/otlp-resource-global-tags branch October 28, 2026 19:19
@github-actions github-actions Bot added this to the 1.69.0 milestone Oct 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: core Tracer core tag: ai generated Largely based on code generated by an AI or LLM type: feature Enhancements and improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants