Skip to content

Emit OpenTelemetry semantic conventions for Akka HTTP, Play, and Spring WebClient - #12901

Open
zacharycmontoya wants to merge 24 commits into
masterfrom
zach.montoya/otel-http-semconv-followup
Open

zacharycmontoya wants to merge 24 commits into
masterfrom
zach.montoya/otel-http-semconv-followup

Conversation

@zacharycmontoya

@zacharycmontoya zacharycmontoya commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

What Does This Do

This PR completes the framework-specific HTTP semantic convention follow-ups from #12765 for Akka HTTP, Play, and Spring WebClient when DD_TRACE_OTEL_SEMANTICS_ENABLED=true.

Akka HTTP and Play apply matched routes to the root HTTP server span, allowing its name to follow the standard span name template {method} {http.route}. Spring WebClient backed by Netty produces one HTTP client span per request.

Implementation Details

Akka HTTP routes: When DD_TRACE_OTEL_SEMANTICS_ENABLED=true, we instrument Akka’s path matchers to reconstruct route templates for supported routing patterns, using matched literals and {param} placeholders for extracted values. The resulting template is applied to the existing local root HTTP server span through the shared HTTP resource decorator.

Play root-span routes: When DD_TRACE_OTEL_SEMANTICS_ENABLED=true, the matched Play route is applied to both the Play framework span and its local root HTTP server span. Notably, the root span receives the route before sampling occurs and the route continues to be dispatched to AppSec API security.

Spring WebClient and Netty: When DD_TRACE_OTEL_SEMANTICS_ENABLED=true, WebClient marks its span in the current context so Netty can track when the active scope is "suppressed" and it should not create an additional HTTP client span. The WebClient span context is still used for context propagation and for recording response/error information and span completion.

System Tests

This is being tested by this PR: DataDog/system-tests#8324. It enables the test cases in test_otel_http_semantics.py that were previously skipped for the akka-http and play weblog applications.

Motivation

When DD_TRACE_OTEL_SEMANTICS_ENABLED=true, the Java tracer should produce HTTP spans that align with the OpenTelemetry semantic conventions instead of emitting a mixture of Datadog and OpenTelemetry naming and attributes. The shared HTTP decorator changes in #12765 left several framework-specific gaps:

  • Akka HTTP and Play root spans did not include the http.route span attribute
  • Spring WebClient requests produced duplicate HTTP client spans.

Addressing these gaps makes the affected frameworks consistent with OpenTelemetry HTTP span naming and span-count expectations when OpenTelemetry semantics are enabled.

Additional Notes

These changes are opt-in only, and span information is unchanged for Datadog semantics.

Contributor Checklist

Jira ticket(s): APMAPI-2038, [APMAPI-2039]

@zacharycmontoya
zacharycmontoya force-pushed the otel-http-semconv branch 2 times, most recently from 9a7511d to 7654f82 Compare October 15, 2026 22:37
@datadog-prod-us1-4

This comment has been minimized.

@dd-octo-sts

dd-octo-sts Bot commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 13.98 s 13.95 s [-0.5%; +0.9%] (no difference)
startup:insecure-bank:tracing:Agent 12.90 s 12.88 s [-0.5%; +0.7%] (no difference)
startup:petclinic:appsec:Agent 16.63 s 16.99 s [-6.4%; +2.2%] (no difference)
startup:petclinic:iast:Agent 16.90 s 17.06 s [-1.7%; -0.1%] (maybe better)
startup:petclinic:profiling:Agent 16.69 s 16.84 s [-1.9%; +0.0%] (no difference)
startup:petclinic:sca:Agent 17.07 s 16.98 s [-0.3%; +1.3%] (no difference)
startup:petclinic:tracing:Agent 16.08 s 16.03 s [-0.6%; +1.2%] (no difference)

Commit: 89f58185 · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

@zacharycmontoya
zacharycmontoya force-pushed the otel-http-semconv branch 2 times, most recently from b5d0c00 to c8d62ca Compare October 16, 2026 00:40
@bric3
bric3 added this pull request to stack #12903 October 16, 2026 07:34
@zacharycmontoya
zacharycmontoya force-pushed the otel-http-semconv branch 2 times, most recently from b1d1812 to 32198d1 Compare October 19, 2026 19:26
@zacharycmontoya
zacharycmontoya removed this pull request from stack #12903 October 19, 2026 20:12
@zacharycmontoya
zacharycmontoya force-pushed the otel-http-semconv branch 2 times, most recently from 8a835c3 to fd65cad Compare October 20, 2026 23:41
Base automatically changed from otel-http-semconv to master October 23, 2026 22:30
@zacharycmontoya
zacharycmontoya force-pushed the zach.montoya/otel-http-semconv-followup branch from 5f15074 to b2a637e Compare October 23, 2026 22:34
Play already decorates its framework span with the matched route, but the
underlying HTTP server span also needs that route for OpenTelemetry naming.
Apply it to the local root before application code can trigger sampling,
using the shared HTTP resource decorator.

Keep the existing Play span decoration and API Security route dispatch.
Observe successful path matchers and combine their literal segments and
parameter placeholders into a route on the local root HTTP server span.
Reuse the shared HTTP resource decorator to set both http.route and the
method-and-route resource name without retaining parameter values.

Allow the required matcher classes to be transformed and include a forked
test that verifies stable routes across different parameter values.
Add a scoped context marker for HTTP client spans owned by higher-level
instrumentation. Enable suppression only with OpenTelemetry semantics and
only when the marker identifies the active request span.

When marked, Netty propagates that context without creating another HTTP
span. Response, error and close handlers retain the callback context and
leave decoration and completion to the instrumentation that owns the span.
Unmarked requests and Datadog semantics keep the existing Netty behavior.
Mark the WebClient span while subscribing so Netty reuses its context
instead of creating a duplicate HTTP client span. In OpenTelemetry mode,
finish the WebClient span before response or error callbacks, which may
wait for that span to complete.

Preserve the existing two spans and completion timing in Datadog mode.
Update the Spring 5 and Spring 6 client suites to check the mode-dependent
span counts and add forked OpenTelemetry coverage for both versions. Skip
HEAD response-body consumption only in OpenTelemetry mode in the Spring 5
test helper.
@zacharycmontoya
zacharycmontoya force-pushed the zach.montoya/otel-http-semconv-followup branch from b2a637e to 6b33693 Compare November 2, 2026 20:57
@zacharycmontoya zacharycmontoya added type: feature Enhancements and improvements inst: opentelemetry OpenTelemetry instrumentation labels Nov 3, 2026
@zacharycmontoya zacharycmontoya changed the title Followup changes to implement OpenTelemetry HTTP semantic conventions for additional frameworks Emit OpenTelemetry semantic conventions for Akka HTTP, Play, and Spring WebClient Nov 3, 2026
Wrap the request handler in a single try/finally so the AWS and
suppressed-span early returns no longer repeat the parent scope close,
and derive the client span context from the already-captured parent.
Keep the Uri.Path on each frame and render it only when the match
succeeds, since most path matcher attempts fail and never need it.
Nested matchers also reuse the root span checked by the outermost frame.
When the outer client span owns the request, run the callback under it
and return, leaving the existing client span handling unchanged instead
of threading the suppression flag through each branch.
Mirror the Akka HTTP path matcher instrumentation for Pekko HTTP so
successful matchers build a route on the local root HTTP server span,
and add a forked test that verifies stable routes across parameters.
Mirror the Play 2.6 change so the underlying HTTP server span also gets
the matched route for OpenTelemetry naming.
A path prefix that matched before the request was rejected left its
partial template on the span, so 404s under a valid prefix were reported
as real endpoints such as "GET /users/{param}".

Follow the OpenTelemetry Akka HTTP rule: the route only stands if the
last outermost path matcher tried also matched. Otherwise, when the
response is written, remove http.route and restore the method-only
resource that OpenTelemetry semantics give unrouted requests.
@zacharycmontoya
zacharycmontoya marked this pull request as ready for review November 3, 2026 19:07
@zacharycmontoya
zacharycmontoya requested review from a team as code owners November 3, 2026 19:07
@zacharycmontoya
zacharycmontoya requested review from claponcet, manuel-alvarez-alvarez and ygree and removed request for a team November 3, 2026 19:07
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-03T19:20:52.433424Z 5a4acd7 Draft marked ready
🔒 Security Review ✅ Completed 2026-11-03T19:22:17.221931Z 5a4acd7 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@zacharycmontoya
zacharycmontoya requested review from PerfectSlayer and removed request for a team November 3, 2026 19:07

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5a4acd77a5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Frame parent = CURRENT.get();
if (parent != null) {
// nested matchers share the root span already checked by the outermost frame
Frame frame = new Frame(path, parent.span, parent.state, parent);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Avoid allocating a frame for every failed path matcher

perf: With OpenTelemetry semantics enabled, this advice-reachable path allocates a Frame for every nested PathMatcher.apply before knowing whether it matched, and the outermost branch similarly allocates at line 44. Akka routes commonly search multiple alternatives that fail, so this adds short-lived allocations proportional to route-table search on every request; the mirrored Pekko implementation has the same behavior. Reuse a cleared ThreadLocal frame stack or pool, and verify the reduction with an allocation profile.

AGENTS.md reference: AGENTS.md:L84-L87

Useful? React with 👍 / 👎.

@datadog-prod-us1-4 datadog-prod-us1-4 Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: FAIL

With OpenTelemetry semantics enabled, WebClient URL-rewriting filters can bypass the Netty SSRF check. Akka/Pekko also lose matched route attributes for encoded paths when raw-resource mode is enabled.

Open Bits AI session

🤖 Bits Code Review · Commit 5a4acd7

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

inst: opentelemetry OpenTelemetry instrumentation type: feature Enhancements and improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants