Skip to content

Introduce SpanStorage for eager tag post-processors (dense store - part 3) - #13001

Draft
dougqh wants to merge 1 commit into
dougqh/tagmap-slot-bucketsfrom
dougqh/eager-postprocessors-span-context
Draft

dougqh wants to merge 1 commit into
dougqh/tagmap-slot-bucketsfrom
dougqh/eager-postprocessors-span-context

Conversation

@dougqh

@dougqh dougqh commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

What Does This Do

Introduces SpanStorage, a span's tags as stored, and runs the eager tag post-processors against it instead of a TagMap.

  • SpanStorage has load(tag), loadString(tag), store(tag, value) and store(EntryReader). Stores go straight to storage, with no interception. A null value removes the tag, and a null entry is a no-op. Thread safety is left to each implementation.
  • Its own verbs. load/store are distinct from TagMap's get/set and the span's intercepted getTag/setTag. One type can offer the raw and the intercepted operations without a name clash, and a call site's verb says which one it means.
  • DDSpanContext implements SpanStorage. Its four methods are unsynchronized, like unsafeGetTag, because callers already hold the span's tag lock, as earlyProcessTags does.
  • TagMap implements it too, through its own get and set, so standalone maps and tests can be passed directly.
  • The eager processors switch to EagerTagsPostProcessor, which takes a SpanStorage. They are PeerServiceCalculator, InternalTagsAdder and HttpEndpointPostProcessor, chained by EagerPostProcessorChain. NamingSchema.ForPeerService and EndpointResolver take SpanStorage too.

This is the first slice: only what the eager processors need. Id variants, typed loads and primitive stores come as more of the pipeline moves over.

Motivation

The eager post-processors run on the application thread as a span finishes. Once a span's tags are no longer one TagMap (#13003), handing them a TagMap means building a view per span on the application thread. In earlier measurements that view cost +4–5% per trace. With SpanStorage, the span passes itself, and nothing is allocated. That matters because the processor chain loops over an array, so its call sites are megamorphic and any adapter object handed down it would escape.

Additional Notes

  • No behavior change, and no allocation change: the probe gives identical bytes at every step of the trace, 7,936 B per trace both before and after.
  • Tests: internal-api passes, as do the dd-trace-core post-processor, naming, endpoint, DDSpanContext and DDSpan suites. EndpointResolverTest now uses TagMap instead of HashMap.

Where this is going

This PR is one step in replacing the span's single TagMap with storage that knows each tag's kind. The goal is to cut what every span allocates on the application thread, without changing behavior.

The end state

  • A tag is classified once, at the boundary. A known tag becomes an id (from the tag registry, Resolve direction-scoped OpenTelemetry names at the OpenTelemetry boundary (otlp - tag registry - phase 3) #12731), and a custom tag stays a name. Instrumentation sets known tags by id (Route intercepted tags by id instead of by name (otlp - tag registry - phase 3) #12817), so the JIT can constant-fold the routing.
  • The span owns its tag stores directly. Known tags are kept by id and custom tags by name. TagMap becomes a thin view for code outside the span.
  • Known tags live in a dense store by default. Parallel arrays of serials and values replace an Entry object per tag, and each span's store is presized from a per-operation hint. With the flag on, a web-request-shaped trace allocates about 30% less on the application thread than on master.
  • Core reads and writes a span's stored tags through SpanStorage, not a TagMap. That covers post-processors now and serializers later, so no TagMap view gets built for them. SpanStorage has its own verbs, load and store, apart from TagMap's get/set and the span's intercepted getTag/setTag, so DDSpanContext can offer both. It starts with four methods and grows id, typed-load and primitive-store variants. Later, an intercepted counterpart (the setTag vocabulary) becomes the type setTagsFrom, TagContributor and TagExtractor take, with one lock per batch.

Later, not in this stack: cheaper read-through to parent spans, trace-level tags in a per-trace holder (TraceSegment), and primitive tag values without boxing.

The stack. This replaces #12918, split into pieces that can each be reviewed, and measured, on their own.

PR Step Allocation per 7-span trace (flag off unless noted)
#12999 A null tag value removes the tag in TagMap. This is the stack's only behavior change, kept separate on purpose. unchanged
#12992 Bucket known tags by slot in TagMap, and add TraceShapeBenchmark −96 B vs master
#13001 SpanStorage for the eager post-processors unchanged
#13002 Extract TagMap's bucket mechanics into HashSupport (a pure move) unchanged
#13006 Add the id and name tag stores, not yet used unchanged
#13007 Make TagMap a facade over the two stores unchanged
#13003 Have the span own its tag stores directly +104 B (+1.3%)
#13004 Dense id store behind trace.experimental.dense.tags.enabled (off by default) flag on: −30.6%
#12926 Turn the dense store on by default, add sizing hints and bulk prototype seeding flag on: about 3% less again

Every PR after #12999 is behavior-preserving.

The pieces' allocation figures come from a deterministic per-thread probe (ThreadMXBean allocated bytes) that builds the same 7-span trace as TraceShapeBenchmark. The #12992 and #12926 figures come from TraceShapeBenchmark itself.

🤖 Generated with Claude Code

@datadog-prod-us1-6

datadog-prod-us1-6 Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

🎯 Code Coverage (details)
• Patch Coverage: 68.29%
• Overall Coverage: 59.84% (+0.37%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 8e57e87 | Docs | Give us feedback!

@dd-octo-sts

dd-octo-sts Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 13.96 s 13.94 s [-0.7%; +0.9%] (no difference)
startup:insecure-bank:tracing:Agent 12.95 s 12.86 s [-0.2%; +1.7%] (no difference)
startup:petclinic:appsec:Agent 17.20 s 17.04 s [-0.1%; +1.9%] (no difference)
startup:petclinic:iast:Agent 17.02 s 16.61 s [-2.0%; +6.9%] (no difference)
startup:petclinic:profiling:Agent 16.79 s 16.77 s [-0.9%; +1.1%] (no difference)
startup:petclinic:sca:Agent 17.15 s 17.02 s [-0.1%; +1.7%] (no difference)
startup:petclinic:tracing:Agent 16.27 s 16.33 s [-1.5%; +0.7%] (no difference)

Commit: 8e57e875 · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

@dougqh dougqh changed the title Introduce SpanContents for eager tag post-processors Introduce SpanContents for eager tag post-processors (dense store - part 3) Oct 23, 2026
@bric3
bric3 added this pull request to stack #13019 October 23, 2026 16:02
Eager post-processors run on the application thread as a span finishes.
They now take a SpanStorage (a span's tags as stored) instead of a
TagMap, and DDSpanContext implements SpanStorage directly, so no TagMap
view needs to be built for them once span storage stops being a single
TagMap.

SpanStorage has its own verbs, load and store, apart from TagMap's
get/set and the span's intercepted getTag/setTag, so DDSpanContext can
offer the raw and the intercepted operations without a clash. Stores
bypass interception; thread safety is left to each implementation
(DDSpanContext's are unsynchronized, called under the tag lock). TagMap
implements it too, through its own get and set, for standalone maps.

This is the first slice: load, loadString and two stores.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dougqh
dougqh force-pushed the dougqh/eager-postprocessors-span-context branch from 195841c to 8e57e87 Compare October 23, 2026 16:57
@dougqh dougqh changed the title Introduce SpanContents for eager tag post-processors (dense store - part 3) Introduce SpanStorage for eager tag post-processors (dense store - part 3) Oct 23, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: core Tracer core tag: ai generated Largely based on code generated by an AI or LLM tag: no release notes Changes to exclude from release notes type: refactoring

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant