Skip to content

Keep traces with different global tags in one OTLP payload rather than eagerly flushing - #13112

Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 2 commits into
masterfrom
matthew.li/otlp-resource-spans-in-payload
Oct 29, 2026
Merged

gh-worker-dd-mergequeue-cf854d[bot] merged 2 commits into
masterfrom
matthew.li/otlp-resource-spans-in-payload

Conversation

@mhlidd

@mhlidd mhlidd commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

What Does This Do

When a remote config change updates the global tags (DD_TAGS), traces with the old and new tags are now exported in the same OTLP payload, each under its own resource carrying its tags. Previously, every switch between tag sets forced the current payload to be flushed.

This PR also slightly refactors work from #12985 to fit in better with the dedupe work from #13065.

Motivation

After a remote config tag change, traces started before the change (such as long-lived traces still sending partial flushes) keep their old global tags and arrive interleaved with new traces. Flushing on every switch could turn one batch into many small HTTP requests. OTLP already allows several ResourceSpans per request, so the tag change no longer needs to cut the payload.

Additional Notes

  • Follows Write global tags once on the OTLP trace resource #13065, which writes global tags once on the OTLP trace resource. This PR also merges OtlpSpanTags into OtlpSpanAttributes as a nested Writer (237ae23), a follow-up cleanup that didn't make it into Write global tags once on the OTLP trace resource #13065.
  • Both the JSON and protobuf collectors cache the current and previous resource, so alternating between the old and new tags swaps them instead of rebuilding each time.
  • Trade-off: if tag sets alternate trace by trace, each switch writes the resource attributes again. That is still cheaper than one request per switch.
  • Process tags are now written on the first span of each resource rather than once per payload.

Contributor Checklist

Jira ticket: N/A

🤖 Generated with Claude Code

@mhlidd mhlidd added type: feature Enhancements and improvements comp: core Tracer core tag: ai generated Largely based on code generated by an AI or LLM labels Oct 28, 2026
Base automatically changed from matthew.li/otlp-resource-global-tags to master October 28, 2026 19:19
mhlidd and others added 2 commits October 28, 2026 15:31
Both classes decide which span tags become OTLP attributes. Keep
OtlpSpanAttributes as the static helper for keys and baggage rules,
and nest the per-span tag writer as OtlpSpanAttributes.Writer.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Instead of flushing the payload whenever a trace's global tags differ
from the payload's, start a new resource within the same payload. Each
resource carries the global tags of its traces, and the collectors keep
the current and previous resource cached so traces interleaving old and
new tags after a remote config change don't rebuild them each time.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mhlidd
mhlidd force-pushed the matthew.li/otlp-resource-spans-in-payload branch from 97dfcd6 to 5142b32 Compare October 28, 2026 19:36
@mhlidd
mhlidd requested a review from mcculls October 28, 2026 19:38
@mhlidd
mhlidd marked this pull request as ready for review October 28, 2026 19:38
@mhlidd
mhlidd requested a review from a team as a code owner October 28, 2026 19:38
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-28T19:40:15.978892Z 5142b32 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@datadog-datadog-prod-us1-2

datadog-datadog-prod-us1-2 Bot commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

🎯 Code Coverage (details)
• Patch Coverage: 96.34%
• Overall Coverage: 60.75% (+1.24%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 5142b32 | Docs | Give us feedback!

@datadog-datadog-prod-us1-2 datadog-datadog-prod-us1-2 Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: PASS

More details

Resource transitions, framing, and trace-boundary metadata appear consistent in both OTLP encodings when tag sets alternate. Runtime behavior was not exercised.

Was this helpful? React 👍 or 👎

Open Bits AI session

🤖 Bits Code Review · Commit 5142b32

@dd-octo-sts

dd-octo-sts Bot commented Oct 28, 2026

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 14.79 s 14.70 s [-0.2%; +1.4%] (no difference)
startup:insecure-bank:tracing:Agent 13.62 s 13.68 s [-1.2%; +0.3%] (no difference)
startup:petclinic:appsec:Agent 17.88 s 17.65 s [+0.4%; +2.2%] (maybe worse)
startup:petclinic:iast:Agent 17.65 s 17.70 s [-1.2%; +0.6%] (no difference)
startup:petclinic:profiling:Agent 17.65 s 17.35 s [+0.4%; +3.0%] (maybe worse)
startup:petclinic:sca:Agent 17.87 s 17.84 s [-0.7%; +1.0%] (no difference)
startup:petclinic:tracing:Agent 16.68 s 16.80 s [-1.8%; +0.3%] (no difference)

Commit: 5142b32f · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

@mcculls
mcculls requested a review from dougqh October 29, 2026 11:14
@mhlidd mhlidd changed the title Keep traces with different global tags in one OTLP payload Keep traces with different global tags in one OTLP payload rather than eagerly flushing Oct 29, 2026
@mhlidd
mhlidd added this pull request to the merge queue Oct 29, 2026
@dd-octo-sts

dd-octo-sts Bot commented Oct 29, 2026

Copy link
Copy Markdown
Contributor

/merge

@gh-worker-devflow-routing-ef8351

gh-worker-devflow-routing-ef8351 Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

View all feedbacks in Devflow UI.

2026-10-29 13:27:11 UTC ℹ️ Start processing command /merge


2026-10-29 13:27:15 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in master is approximately 2h (p90).


2026-10-29 14:12:36 UTC ❌ MergeQueue: The build pipeline contains failing jobs for this merge request

Build pipeline has failing jobs for ba7a8cd:

⚠️ Do NOT retry failed jobs directly (why?).

What to do next?

  • Investigate the failures and when ready, re-add your pull request to the queue!
  • If your PR checks are green, try to rebase/merge. It might be because the CI run is a bit old.
  • Any question, go check the FAQ.
Details

Since those jobs are not marked as being allowed to fail, the pipeline will most likely fail.
Therefore, and to allow other builds to be processed, this merge request has been rejected and the pipeline got canceled.

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 29, 2026
@mhlidd
mhlidd added this pull request to the merge queue Oct 29, 2026
@dd-octo-sts

dd-octo-sts Bot commented Oct 29, 2026

Copy link
Copy Markdown
Contributor

/merge

@gh-worker-devflow-routing-ef8351

gh-worker-devflow-routing-ef8351 Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

View all feedbacks in Devflow UI.

2026-10-29 14:28:03 UTC ℹ️ Start processing command /merge


2026-10-29 14:28:07 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in master is approximately 2h (p90).


2026-10-29 15:38:58 UTC ℹ️ MergeQueue: This merge request was merged

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 29, 2026
@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot merged commit 98b579a into master Oct 29, 2026
637 checks passed
@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot deleted the matthew.li/otlp-resource-spans-in-payload branch October 29, 2026 15:38
@github-actions github-actions Bot added this to the 1.69.0 milestone Oct 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: core Tracer core tag: ai generated Largely based on code generated by an AI or LLM type: feature Enhancements and improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants