[calvinbayer]

What Does This Do

Support test automation exclusions in ScalaTest. Wire test declaration metadata into Datadog policy selection and reporting, preserving assigned Test Management properties.

Motivation

Allow individual tests to prevent Datadog quarantine/disabling and exclude Datadog ATR, EFD and Attempt to Fix. Developer selection and framework retries retain their existing semantics.

Additional Notes

Stacked on codex/karate-automation-exclusions; review this PR against its base for the framework-specific change. Shared public API and policy are introduced by the first JUnit4 PR. No refactor commit was needed.

Validation: Module spotlessApply and instrumented ScalatestTest exclusion regression passed, including protected one-time failure, assignment tags, and developer-ignored test tags.

Snapshot dependency for inherited physical source paths: DataDog/test-environment#604.

test-environment-branch: codex/java-inherited-test-source-snapshots

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - sbt-scalatest

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	56.82	56.55	$\color{red}{\blacktriangle}$ +0.27	56.55	$\color{red}{\blacktriangle}$ +0.27	181/423
agentEvpProxy	56.92	n/a	n/a	n/a	n/a	-

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - nebula-release-plugin

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	37.21	37.15	$\color{red}{\blacktriangle}$ +0.06	37.15	$\color{red}{\blacktriangle}$ +0.06	92/225
agentless	36.30	37.15	$\color{green}{\blacktriangledown}$ -0.85	37.15	$\color{green}{\blacktriangledown}$ -0.85	92/198
agentlessCodeCoverage	44.75	46.30	$\color{green}{\blacktriangledown}$ -1.55	45.38	$\color{green}{\blacktriangledown}$ -0.63	92/197
agentlessLineCoverage	57.07	56.55	$\color{red}{\blacktriangle}$ +0.52	56.55	$\color{red}{\blacktriangle}$ +0.52	92/197

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[datadog-prod-us1-4]

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 59.92% (-0.00%)

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 627c8a5 | Docs | Give us feedback!}

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - pass4s

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	10.71	10.54	$\color{red}{\blacktriangle}$ +0.17	10.13	$\color{red}{\blacktriangle}$ +0.58	92/224
agentless	10.15	9.92	$\color{red}{\blacktriangle}$ +0.23	9.92	$\color{red}{\blacktriangle}$ +0.23	91/194
agentlessCodeCoverage	15.10	17.38	$\color{green}{\blacktriangledown}$ -2.28	17.73	$\color{green}{\blacktriangledown}$ -2.63	82/178

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - reactive-streams-jvm

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	22.65	22.54	$\color{red}{\blacktriangle}$ +0.11	22.54	$\color{red}{\blacktriangle}$ +0.11	122/257
agentless	19.87	19.59	$\color{red}{\blacktriangle}$ +0.28	19.59	$\color{red}{\blacktriangle}$ +0.28	117/220
agentlessCodeCoverage	21.89	20.39	$\color{red}{\blacktriangle}$ +1.50	20.39	$\color{red}{\blacktriangle}$ +1.50	116/219
agentlessLineCoverage	28.27	27.52	$\color{red}{\blacktriangle}$ +0.75	27.52	$\color{red}{\blacktriangle}$ +0.75	79/154

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - netflix-zuul

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	88.88	89.58	$\color{green}{\blacktriangledown}$ -0.70	89.58	$\color{green}{\blacktriangledown}$ -0.70	92/226
agentless	81.45	82.69	$\color{green}{\blacktriangledown}$ -1.24	82.69	$\color{green}{\blacktriangledown}$ -1.24	92/197
agentlessCodeCoverage	97.48	97.04	$\color{red}{\blacktriangle}$ +0.44	97.04	$\color{red}{\blacktriangle}$ +0.44	91/196
agentlessLineCoverage	112.92	113.88	$\color{green}{\blacktriangledown}$ -0.96	113.88	$\color{green}{\blacktriangledown}$ -0.96	91/196

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - heliboard

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	9.54	9.73	$\color{green}{\blacktriangledown}$ -0.19	9.92	$\color{green}{\blacktriangledown}$ -0.38	92/197

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - sonar-kotlin

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	12.88	13.67	$\color{green}{\blacktriangledown}$ -0.79	13.40	$\color{green}{\blacktriangledown}$ -0.52	104/239
agentless	13.19	12.37	$\color{red}{\blacktriangle}$ +0.82	12.37	$\color{red}{\blacktriangle}$ +0.82	104/209
agentlessCodeCoverage	15.09	15.72	$\color{green}{\blacktriangledown}$ -0.63	15.72	$\color{green}{\blacktriangledown}$ -0.63	104/209
agentlessLineCoverage	17.48	17.38	$\color{red}{\blacktriangle}$ +0.10	17.73	$\color{green}{\blacktriangledown}$ -0.25	102/207

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - jolokia

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	97.21	97.04	$\color{red}{\blacktriangle}$ +0.17	95.12	$\color{red}{\blacktriangle}$ +2.09	107/241
agentless	92.14	91.39	$\color{red}{\blacktriangle}$ +0.75	91.39	$\color{red}{\blacktriangle}$ +0.75	107/213
agentlessCodeCoverage	101.32	101.00	$\color{red}{\blacktriangle}$ +0.32	99.00	$\color{red}{\blacktriangle}$ +2.32	107/213
agentlessLineCoverage	103.81	103.04	$\color{red}{\blacktriangle}$ +0.77	101.00	$\color{red}{\blacktriangle}$ +2.81	116/228

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - okhttp

Job Status: 🔴 failed

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	18.39	18.82	$\color{green}{\blacktriangledown}$ -0.43	19.59	$\color{green}{\blacktriangledown}$ -1.20	107/240
agentless	17.42	18.45	$\color{green}{\blacktriangledown}$ -1.03	19.20	$\color{green}{\blacktriangledown}$ -1.78	106/215
agentlessCodeCoverage	21.19	21.65	$\color{green}{\blacktriangledown}$ -0.46	22.54	$\color{green}{\blacktriangledown}$ -1.35	108/219
agentlessLineCoverage	36.72	37.91	$\color{green}{\blacktriangledown}$ -1.19	38.67	$\color{green}{\blacktriangledown}$ -1.95	105/213

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[dd-octo-sts]

🟡 Java Benchmark SLOs — Performance SLO warning (near threshold)

Suite	Status
Startup	🟡 warning

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	14.70 s	14.61 s	[-0.3%; +1.5%] (no difference)
startup:insecure-bank:tracing:Agent	13.53 s	13.62 s	[-1.4%; +0.1%] (no difference)
startup:petclinic:appsec:Agent	17.76 s	17.53 s	[+0.4%; +2.2%] (maybe worse)
startup:petclinic:iast:Agent	17.46 s	17.57 s	[-1.4%; +0.1%] (no difference)
startup:petclinic:profiling:Agent	17.34 s	16.70 s	[-0.8%; +8.5%] (no difference)
startup:petclinic:sca:Agent	17.71 s	17.57 s	[-0.1%; +1.7%] (no difference)
startup:petclinic:tracing:Agent	16.56 s	16.74 s	[-2.3%; +0.2%] (no difference)

Commit: 627c8a57 · CI Pipeline · Benchmarking Platform UI

---

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - spring_boot

Job Status: 🔴 failed

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	10.44	16.36	$\color{green}{\blacktriangledown}$ -5.92	16.36	$\color{green}{\blacktriangledown}$ -5.92	104/237
agentless	10.02	9.92	$\color{red}{\blacktriangle}$ +0.10	9.92	$\color{red}{\blacktriangle}$ +0.10	104/209
agentlessCodeCoverage	14.32	13.94	$\color{red}{\blacktriangle}$ +0.38	13.67	$\color{red}{\blacktriangle}$ +0.65	104/211
agentlessLineCoverage	23.07	22.99	$\color{red}{\blacktriangle}$ +0.08	22.54	$\color{red}{\blacktriangle}$ +0.53	104/209

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-19T22:17:34.732623Z	1273e6b	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - sonar-java

Job Status: 🔴 failed

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	-7.19	14.51	$\color{green}{\blacktriangledown}$ -21.70	14.23	$\color{green}{\blacktriangledown}$ -21.42	104/237
agentless	-12.49	12.37	$\color{green}{\blacktriangledown}$ -24.86	12.87	$\color{green}{\blacktriangledown}$ -25.36	104/211
agentlessCodeCoverage	51.55	84.36	$\color{green}{\blacktriangledown}$ -32.81	81.05	$\color{green}{\blacktriangledown}$ -29.50	104/211
agentlessLineCoverage	112.34	120.92	$\color{green}{\blacktriangledown}$ -8.58	123.36	$\color{green}{\blacktriangledown}$ -11.02	104/212

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[datadog-prod-us1-4]

Running a ScalaTest test by name bypasses declaration-tag collection, allowing Datadog retries and failure suppression despite explicit exclusions.

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit 1273e6b}

[datadog-prod-us1-4]

The supplied diff breaks older TestNG test suites and reinstalls Datadog retries after a user transformer explicitly disables them.

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit 22ed8f1}

---

Findings that could not be posted inline

dd-java-agent/instrumentation/testng/testng-common/src/main/java/datadog/trace/instrumentation/testng/TestNGUtils.java:78

Preserve transformer overrides that disable retries

When a user annotation transformer explicitly selects DisabledRetryAnalyzer, hasFrameworkOverride is true, but the native-analyzer check falls through and installs Datadog's RetryAnalyzer. With ATR or EFD enabled on an unexcluded test, this re-enables retries against the developer's setting and can turn a failing test into a passing build. Preserve explicit transformer overrides even when they disable retries.

    if (RetryAnnotationTransformer.hasFrameworkOverride(annotation)) {
      return;
    }
    if (TestExclusions.noAutoRetry(source, Arrays.asList(method.getGroups()))) {

React: 👍 useful · 👎 wrong · 😕 unclear · 👀 right, but not acting on it

What each reaction means

• 👍 Useful: correct, and you're fixing it or will.
• 👎 Wrong: the finding is incorrect, for example the bug can't happen or the code was misread.
• 😕 Unclear: you can't tell what it means or why it matters.
• 👀 Right, not acting: correct, but intended, out of scope for this PR, a known risk, or too minor.

_{🤖 Bits Code Review · Open Bits AI session}

[calvinbayer]

Addressed the finding in this review with the inherited fix d308a2b.

The regression first reproduced the bug: an explicit DisabledRetryAnalyzer override still executed six times with TestNG 7.0 ATR, and five/three times with latest TestNG ATR/EFD. The transformer now preserves every explicit framework override before installing Datadog policy. The full configured suite passes 308 cases with 101 capability skips, plus Muzzle, forbidden-API and module checks. The supported scenarios execute exactly once and emit no retry metadata.

[calvinbayer]

@autotest Review

[datadog-prod-us1-4]

More details

ScalaTest declaration exclusions reach policy selection and reporting while preserving developer-ignored tests and assigned Test Management properties.

Was this helpful? React 👍 or 👎

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit ed05308}

[calvinbayer]

@autotest Review

[datadog-prod-us1-4]

More details

Source inspection supports that ScalaTest declaration tags reach Datadog policy selection and reporting while preserving native ignore handling. No reportable defects were identified.

Was this helpful? React 👍 or 👎

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit 825c4de}

[calvinbayer]

@autotest Review

[datadog-prod-us1-4]

More details

The completed static review found no reportable defect in ScalaTest exclusion-policy selection or reporting, including preservation of native ignore behavior. Confidence is limited to source inspection; tests were not run.

Was this helpful? React 👍 or 👎

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit 627c8a5}

[ygree]

Looks like a coverage gap to me. Should there be separate tests for noRetry and preventSuppression?

[calvinbayer]

Yes, the ScalaTest fixture only exercises the two tags together, so it does not independently verify each flag at the integration boundary. The shared policy has a noRetry-only test, and reporting is parameterized over the individual tags, but those do not replace separate ScalaTest cases. I found no demonstrated behavior bug in the independent flag paths; I am deferring the additional integration coverage for discussion under the current bug-fix-only scope.

[ygree]

IDM approved