Skip to content

Skip Datadog tags a trace context merge never reads - #13034

Draft
dougqh wants to merge 3 commits into
dougqh/primitive-extracted-trace-idsfrom
dougqh/lazy-dd-member-merge
Draft

dougqh wants to merge 3 commits into
dougqh/primitive-extracted-trace-idsfrom
dougqh/lazy-dd-member-merge

Conversation

@dougqh

@dougqh dougqh commented Oct 23, 2026

Copy link
Copy Markdown
Contributor

What Does This Do

When the W3C trace context is merged into an already extracted context, its tracestate dd member is now parsed only as far as the merge needs.

  • The merge reads three things from the dd member:
    • the sampling priority (s);
    • the last parent id (p);
    • whether its tags are valid. A malformed tid drops the whole tracestate, and a malformed dm or ts drops the Datadog tags along with s and p.
  • Every other t.* value is never read, and only its syntax is checked, which the scan already does.
  • Until now, each of those values was still built as a TagValue (uncached for LLM Observability ids) and collected into the tagPairs list. t.ksr alone allocated that list on every same-version merge. The merge then threw all of it away.

The change:

  • In merge mode (Result.keepsDatadogTags() is false), the scan skips any tag whose key isn't validated, before building its value. Only dm, tid and ts are validated, and those are still built and checked exactly as before.
  • One key list: validateTagValue now checks only the keys hasValidatedValues names. A validation added without updating that list never runs anywhere, and its own tests fail, so merge mode can't quietly diverge.
  • A merge with no ot member no longer rescans the tracestate for one. The parse already looked.

Motivation

TraceSegment work (APMLP-2049), continuing the per-member lazy tracestate described in the design comments there.

Allocation (PropagationScenariosBenchmark, JDK 17, -t 4, 2 forks × 5 iterations, -prof gc, default sampler, base = #13033; measured before the rescan commit, which saves CPU rather than allocation):

scenario upstream #13033 B/op lazy dd B/op Δ B/op #13033 ops/µs lazy dd ops/µs
intermediate sameVersion 3384.0 ± 12.7 3336.0 ± 12.7 -48.0 1.179 ± 0.063 1.179 ± 0.038
intermediate withOtelMember 3514.5 ± 11.3 3445.4 ± 7.1 -69.1 1.056 ± 0.040 1.131 ± 0.033
leaf sameVersion 1776.0 ± 0.1 1747.4 ± 68.5 -28.6 2.027 ± 0.230 1.901 ± 0.061
leaf withOtelMember 1949.6 ± 36.8 1856.2 ± 12.2 -93.5 1.716 ± 0.021 1.724 ± 0.224

That's −29 to −94 B per extracted request.

Throughput isn't settled: this was a busy host at 2 forks, with error up to ±0.23 ops/µs. It will be covered by the stack's interleaved run.

Cost on the full-parse path: up to three extra TagKey.equals calls per validated tag. Those return immediately on identity for the cached keys the parser produces, so it's a few nanoseconds per extract.

Additional Notes

Output is unchanged:

  • CompoundExtractionGoldenTest (1274 curated cases) passes, and the 10150-case cartesian corpus is identical to the dump taken before Build one trace id per extracted compound context #13033.
  • Mutation check:
    • making merge mode skip the validated keys too fails 24 of 1274 cases;
    • making the full parse drop its tags fails 33 of 1274.

Tests: the full dd-trace-core suite passes locally, except for 2 Docker-dependent TracerConnectionReliabilityTest cases and the order-dependent PendingTraceBufferTest.testingTracerFlareDumpWithMultipleTraces.

Also included: a one-line fix to a stale CoreTracer comment about where request context data goes.

Stacked: master ← #13008 ← #13000 ← #13009 ← #13010 ← #13015 ← #13028 ← #13029 ← #13032 ← #13033 ← this PR.

🤖 Generated with Claude Code

@dougqh dougqh added comp: core Tracer core tag: performance Performance related changes tag: no release notes Changes to exclude from release notes type: refactoring comp: context propagation Trace context propagation tag: ai generated Largely based on code generated by an AI or LLM labels Oct 23, 2026
@datadog-datadog-prod-us1-2

datadog-datadog-prod-us1-2 Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 59.96% (+0.01%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: e519965 | Docs | Give us feedback!

@dd-octo-sts

dd-octo-sts Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 14.03 s 13.98 s [-0.4%; +1.1%] (no difference)
startup:insecure-bank:tracing:Agent 12.90 s 12.99 s [-1.6%; +0.2%] (no difference)
startup:petclinic:appsec:Agent 17.09 s 16.50 s [-0.8%; +7.9%] (no difference)
startup:petclinic:iast:Agent 16.94 s 16.60 s [-2.3%; +6.4%] (no difference)
startup:petclinic:profiling:Agent 16.84 s 16.86 s [-1.1%; +0.9%] (no difference)
startup:petclinic:sca:Agent 17.01 s 16.43 s [-0.9%; +7.9%] (no difference)
startup:petclinic:tracing:Agent 16.14 s 16.11 s [-0.9%; +1.2%] (no difference)

Commit: e519965b · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

dougqh and others added 3 commits October 23, 2026 18:08
Merging a trace context only reads the tracestate's dd member for the
sampling priority, the last parent id and whether its tags are valid.
Its other tags were still built as tag values, LLM Observability ids
uncached, and collected into a list that the merge then discarded. A
merge now skips any tag whose value isn't validated before building
it; the trace id, decision maker and trace source are still built and
validated as before.

validateTagValue now checks only the keys hasValidatedValues names, so
the two can't disagree.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: context propagation Trace context propagation comp: core Tracer core tag: ai generated Largely based on code generated by an AI or LLM tag: no release notes Changes to exclude from release notes tag: performance Performance related changes type: refactoring

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant