Skip to content

Add Jetty 12.0 native WebSocket tracing and fix upgrade span completion - #12674

Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 25 commits into
masterfrom
ygree/jetty-native-websockets
Oct 20, 2026
Merged

gh-worker-dd-mergequeue-cf854d[bot] merged 25 commits into
masterfrom
ygree/jetty-native-websockets

Conversation

@ygree

@ygree ygree commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

What Does This Do

Adds tracing for Jetty 12.0.x's native WebSocket API:

  • Creates websocket.receive spans for client and server sessions, plus server-side websocket.close spans, covering listener and annotated endpoints, including handlers returning boxed Void.
  • Creates websocket.send spans for client and server sessions, using the active span as parent when present and creating root spans otherwise. Subsequent fragments preserve the first fragment's context.
  • Supports full and fragmented text and binary messages, plus streaming Reader and InputStream handlers, preserving handshake links, application context, and existing sampling behavior. Client receives link to the HTTP handshake span rather than its application root.
  • Handles overlapping messages and out-of-order callback completion, recording handler, send, and callback failures. Callbacks retain their message state; connection-level tracking retains only the unfinished fragment sequence.
  • On close, ends incomplete receive and send messages, waiting for active text handlers and outstanding callbacks before finishing their spans so failures reported after closure are retained.

Also fixes Jetty HTTP client handshake spans left unfinished by WebSocket upgrades. Activates the handshake span during upgrade so client sessions capture its context, then finishes it with the HTTP 101 response. Records upgrade failures that Jetty catches and reports through the session future.

Expands WebSocket Muzzle compatibility checks and latest-dependency tests to cover the 12.0.x release line, replacing the 12.0.17 upper bound. Rejects incompatible Jetty 12.1+ WebSocket instrumentation to prevent partial installation and missing close cleanup.

Motivation

Jetty native WebSocket endpoints need dedicated instrumentation to trace incoming messages, outbound sends, and server close handlers. Streaming handlers, fragmentation, and asynchronous callbacks require lifecycle tracking to avoid premature completion, mixed contexts, and lost errors while keeping per-message overhead low.

WebSocket upgrades bypass the HTTP client's normal response completion listeners, including when Jetty swallows an upgrade failure, requiring explicit handshake span completion and error reporting.

Additional Notes

Contributor Checklist

Jira ticket: FRAPMS-6125

image

@ygree ygree self-assigned this Sep 29, 2026
@ygree ygree added inst: jetty Jetty instrumentation inst: websocket WebSocket Instrumentation tag: ai generated Largely based on code generated by an AI or LLM labels Sep 29, 2026
@datadog-datadog-prod-us1

datadog-datadog-prod-us1 Bot commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 68.82% (+8.98%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 0b6a583 | Docs | Give us feedback!

@dd-octo-sts

dd-octo-sts Bot commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 14.77 s 14.64 s [+0.0%; +1.7%] (maybe worse)
startup:insecure-bank:tracing:Agent 13.58 s 13.70 s [-1.7%; -0.1%] (maybe better)
startup:petclinic:appsec:Agent 17.12 s 16.94 s [+0.3%; +1.8%] (maybe worse)
startup:petclinic:iast:Agent 17.04 s 17.06 s [-1.1%; +0.8%] (no difference)
startup:petclinic:profiling:Agent 16.61 s 16.85 s [-2.5%; -0.4%] (maybe better)
startup:petclinic:sca:Agent 16.94 s 16.91 s [-0.7%; +1.1%] (no difference)
startup:petclinic:tracing:Agent 16.07 s 16.26 s [-2.0%; -0.4%] (maybe better)

Commit: 0b6a5830 · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

@ygree
ygree force-pushed the ygree/jetty-native-websockets branch 2 times, most recently from a44a0c1 to 3b33f1a Compare September 29, 2026 03:17
@ygree
ygree force-pushed the ygree/jetty-native-websockets branch 8 times, most recently from 31a2644 to 5052a0c Compare October 3, 2026 00:14
@ygree

ygree commented Oct 3, 2026

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 3, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-07T17:23:15.901268Z a6e9a28 Manual request
🔒 Security Review ✅ Completed 2026-10-07T17:22:26.629859Z a6e9a28 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: 5052a0c550

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5052a0c550

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ygree
ygree force-pushed the ygree/jetty-native-websockets branch from 5052a0c to aac3649 Compare October 6, 2026 23:16
@ygree ygree changed the title Add Jetty 12 native WebSocket tracing and fix upgrade span completion Add Jetty 12.0 native WebSocket tracing and fix upgrade span completion Oct 7, 2026
@ygree
ygree force-pushed the ygree/jetty-native-websockets branch from bb5bd89 to 71a43bb Compare October 7, 2026 00:49
@ygree

ygree commented Oct 7, 2026

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: 71a43bb12a

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Breezy!

Reviewed commit: 71a43bb12a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@ygree
ygree marked this pull request as ready for review October 7, 2026 02:08
@ygree
ygree requested review from a team as code owners October 7, 2026 02:08
ygree added 2 commits October 7, 2026 11:39
Correct the EE9 directive name so Muzzle validates its instrumentation.
Add inverse assertions for EE8, EE9, and EE10 to verify rejection outside
the supported [12,12.1) range.
@ygree
ygree force-pushed the ygree/jetty-native-websockets branch from a6e9a28 to 449cb65 Compare October 7, 2026 18:39
@ygree ygree added the type: feature Enhancements and improvements label Oct 7, 2026
ygree added 4 commits October 7, 2026 19:01
Cover client receive spans, handshake links, and callback lifecycles.
Require an active parent when starting client send spans.

Leave 29 tests failing pending implementation.
Link client receive spans to the HTTP upgrade span. Require an active parent when starting client send spans, while preserving the span across subsequent fragments.
Let callbacks own pending messages and retain only the unfinished fragment sequence in connection state. Remove redundant error-handling locks.
Split mixed scenarios and remove redundant endpoint cases. Verify that rejected fragments preserve the unfinished message for close cleanup.
@tabgok
tabgok requested review from a team and ValentinZakharov and removed request for a team and vandonr October 13, 2026 15:50

@datadog-datadog-prod-us1 datadog-datadog-prod-us1 Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: FAIL

Closing a session while a text handler is running can truncate its receive span and lose a later handler error. Fragmented client messages that start without tracing can also acquire an unrelated parent on a later fragment.

Open Bits AI session

🤖 Bits Code Review · Commit a66ef77

ygree added 2 commits October 13, 2026 12:07
Defer receive span completion until active text handlers exit, preserving handler duration and errors when the connection closes concurrently.
Create a root send span when no parent is active and reuse it across fragments, preserving the handshake link and complete message counts.

@ValentinZakharov ValentinZakharov left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

ygree added 3 commits October 13, 2026 14:21
Move text receive state updates into synchronized methods so SpotBugs recognizes the locking, preserving span completion behavior.
@ygree
ygree force-pushed the ygree/jetty-native-websockets branch from def014f to 0b6a583 Compare October 20, 2026 17:15
@ygree

ygree commented Oct 20, 2026

Copy link
Copy Markdown
Contributor Author

/merge

@gh-worker-devflow-routing-ef8351

gh-worker-devflow-routing-ef8351 Bot commented Oct 20, 2026 •

Copy link
Copy Markdown

View all feedbacks in Devflow UI.

2026-10-20 18:35:33 UTC ℹ️ Start processing command /merge


2026-10-20 18:35:43 UTC ℹ️ MergeQueue: Pull request is not mergeable yet

It will be processed automatically as soon as GitHub reports it as mergeable. View in MergeQueue UI.

  • Run /code blockers to see what is blocking it.
  • Run /remove to cancel it.

2026-10-20 19:43:23 UTC ℹ️ MergeQueue: merge request added to the queue

The expected merge time in master is approximately 1h (p90).


2026-10-20 20:59:44 UTC ℹ️ MergeQueue: This merge request was merged

@ygree

ygree commented Oct 20, 2026

Copy link
Copy Markdown
Contributor Author

/code blockers

@gh-worker-devflow-routing-ef8351

gh-worker-devflow-routing-ef8351 Bot commented Oct 20, 2026 •

Copy link
Copy Markdown

View all feedbacks in Devflow UI.

2026-10-20 19:41:24 UTC ℹ️ Start processing command /code blockers


2026-10-20 19:41:25 UTC ℹ️ Devflow:

Checking merge blockers for #12674...


2026-10-20 19:41:30 UTC ℹ️ Devflow: /code blockers

Detected 1 merge blocker(s) to address:

🔴 Blocking

  • This pull request has unresolved conversations.
    1 conversation is not resolved in the changes view.
    Hint: Resolve every conversation before merging.

@ygree

ygree commented Oct 20, 2026

Copy link
Copy Markdown
Contributor Author

/merge

@gh-worker-devflow-routing-ef8351

gh-worker-devflow-routing-ef8351 Bot commented Oct 20, 2026 •

Copy link
Copy Markdown

View all feedbacks in Devflow UI.

2026-10-20 19:42:16 UTC ℹ️ Start processing command /merge


2026-10-20 19:42:18 UTC ❌ MergeQueue

PR already in the queue with status waiting

@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot merged commit b8a9912 into master Oct 20, 2026
611 checks passed
@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot deleted the ygree/jetty-native-websockets branch October 20, 2026 20:59
@github-actions github-actions Bot added this to the 1.68.0 milestone Oct 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

inst: jetty Jetty instrumentation inst: websocket WebSocket Instrumentation tag: ai generated Largely based on code generated by an AI or LLM type: feature Enhancements and improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants