[calvinbayer]

What Does This Do

Support test automation exclusions in Karate. Wire test declaration metadata into Datadog policy selection and reporting, preserving assigned Test Management properties.

Motivation

Allow individual tests to prevent Datadog quarantine/disabling and exclude Datadog ATR, EFD and Attempt to Fix. Developer selection and framework retries retain their existing semantics.

Additional Notes

Stacked on codex/cucumber-automation-exclusions; review this PR against its base for the framework-specific change. Shared public API and policy are introduced by the first JUnit4 PR.

Validation: Both Karate1 and Karate2 module spotlessApply and instrumented exclusion regressions passed on installed JDK21. Each protected scenario executes once with blocking failure, annotation tags and preserved assignment tags.

Snapshot dependency for inherited physical source paths: DataDog/test-environment#604.

test-environment-branch: codex/java-inherited-test-source-snapshots

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - sbt-scalatest

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	58.92	58.86	$\color{red}{\blacktriangle}$ +0.06	57.69	$\color{red}{\blacktriangle}$ +1.23	92/421
agentEvpProxy	59.38	n/a	n/a	n/a	n/a	-

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - nebula-release-plugin

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	36.76	38.67	$\color{green}{\blacktriangledown}$ -1.91	37.91	$\color{green}{\blacktriangledown}$ -1.15	48/212
agentless	37.09	37.91	$\color{green}{\blacktriangledown}$ -0.82	37.15	$\color{green}{\blacktriangledown}$ -0.06	48/213
agentlessCodeCoverage	45.31	46.30	$\color{green}{\blacktriangledown}$ -0.99	46.30	$\color{green}{\blacktriangledown}$ -0.99	48/212
agentlessLineCoverage	57.15	57.69	$\color{green}{\blacktriangledown}$ -0.54	56.55	$\color{red}{\blacktriangle}$ +0.60	48/212

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - heliboard

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	11.04	9.92	$\color{red}{\blacktriangle}$ +1.12	9.92	$\color{red}{\blacktriangle}$ +1.12	48/212

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - netflix-zuul

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	93.32	93.23	$\color{red}{\blacktriangle}$ +0.09	91.39	$\color{red}{\blacktriangle}$ +1.93	48/212
agentless	86.36	84.36	$\color{red}{\blacktriangle}$ +2.00	82.69	$\color{red}{\blacktriangle}$ +3.67	48/212
agentlessCodeCoverage	100.59	99.00	$\color{red}{\blacktriangle}$ +1.59	97.04	$\color{red}{\blacktriangle}$ +3.55	48/212
agentlessLineCoverage	116.85	116.18	$\color{red}{\blacktriangle}$ +0.67	113.88	$\color{red}{\blacktriangle}$ +2.97	48/213

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - pass4s

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	12.27	10.97	$\color{red}{\blacktriangle}$ +1.30	10.54	$\color{red}{\blacktriangle}$ +1.73	48/214
agentless	13.33	10.97	$\color{red}{\blacktriangle}$ +2.36	10.33	$\color{red}{\blacktriangle}$ +3.00	49/213
agentlessCodeCoverage	18.65	17.73	$\color{red}{\blacktriangle}$ +0.92	17.73	$\color{red}{\blacktriangle}$ +0.92	46/195

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - reactive-streams-jvm

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	23.08	22.54	$\color{red}{\blacktriangle}$ +0.54	22.54	$\color{red}{\blacktriangle}$ +0.54	50/247
agentless	20.78	19.59	$\color{red}{\blacktriangle}$ +1.19	19.59	$\color{red}{\blacktriangle}$ +1.19	47/237
agentlessCodeCoverage	20.43	20.39	$\color{red}{\blacktriangle}$ +0.04	20.39	$\color{red}{\blacktriangle}$ +0.04	46/237
agentlessLineCoverage	27.17	27.52	$\color{green}{\blacktriangledown}$ -0.35	27.52	$\color{green}{\blacktriangledown}$ -0.35	28/159

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - sonar-kotlin

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	14.06	13.94	$\color{red}{\blacktriangle}$ +0.12	13.67	$\color{red}{\blacktriangle}$ +0.39	48/224
agentless	13.12	12.37	$\color{red}{\blacktriangle}$ +0.75	12.37	$\color{red}{\blacktriangle}$ +0.75	48/224
agentlessCodeCoverage	15.51	16.04	$\color{green}{\blacktriangledown}$ -0.53	15.72	$\color{green}{\blacktriangledown}$ -0.21	48/224
agentlessLineCoverage	17.34	18.08	$\color{green}{\blacktriangledown}$ -0.74	17.73	$\color{green}{\blacktriangledown}$ -0.39	48/224

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - jolokia

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	97.98	97.04	$\color{red}{\blacktriangle}$ +0.94	97.04	$\color{red}{\blacktriangle}$ +0.94	52/232
agentless	93.94	93.23	$\color{red}{\blacktriangle}$ +0.71	91.39	$\color{red}{\blacktriangle}$ +2.55	50/229
agentlessCodeCoverage	102.13	101.00	$\color{red}{\blacktriangle}$ +1.13	101.00	$\color{red}{\blacktriangle}$ +1.13	51/230
agentlessLineCoverage	102.45	103.04	$\color{green}{\blacktriangledown}$ -0.59	103.04	$\color{green}{\blacktriangledown}$ -0.59	63/259

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[datadog-official]

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 59.77% (-0.01%)

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 004ac89 | Docs | Give us feedback!}

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - okhttp

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	17.46	19.20	$\color{green}{\blacktriangledown}$ -1.74	19.20	$\color{green}{\blacktriangledown}$ -1.74	50/229
agentless	16.99	18.82	$\color{green}{\blacktriangledown}$ -1.83	18.82	$\color{green}{\blacktriangledown}$ -1.83	50/231
agentlessCodeCoverage	21.09	21.65	$\color{green}{\blacktriangledown}$ -0.56	22.09	$\color{green}{\blacktriangledown}$ -1.00	50/235
agentlessLineCoverage	36.11	37.91	$\color{green}{\blacktriangledown}$ -1.80	37.91	$\color{green}{\blacktriangledown}$ -1.80	50/228

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - spring_boot

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	16.41	16.36	$\color{red}{\blacktriangle}$ +0.05	16.36	$\color{red}{\blacktriangle}$ +0.05	48/224
agentless	9.09	9.73	$\color{green}{\blacktriangledown}$ -0.64	9.92	$\color{green}{\blacktriangledown}$ -0.83	48/224
agentlessCodeCoverage	13.01	13.67	$\color{green}{\blacktriangledown}$ -0.66	13.67	$\color{green}{\blacktriangledown}$ -0.66	48/226
agentlessLineCoverage	22.87	22.54	$\color{red}{\blacktriangle}$ +0.33	22.54	$\color{red}{\blacktriangle}$ +0.33	48/224

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[dd-octo-sts]

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	14.02 s	13.96 s	[-0.3%; +1.1%] (no difference)
startup:insecure-bank:tracing:Agent	12.97 s	12.92 s	[-0.6%; +1.3%] (no difference)
startup:petclinic:appsec:Agent	17.14 s	17.03 s	[-0.3%; +1.6%] (no difference)
startup:petclinic:iast:Agent	16.98 s	17.04 s	[-1.1%; +0.4%] (no difference)
startup:petclinic:profiling:Agent	16.85 s	16.96 s	[-1.8%; +0.5%] (no difference)
startup:petclinic:sca:Agent	17.09 s	16.91 s	[+0.4%; +1.7%] (maybe worse)
startup:petclinic:tracing:Agent	16.15 s	16.21 s	[-1.5%; +0.7%] (no difference)

Commit: 004ac896 · CI Pipeline · Benchmarking Platform UI

---

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-19T22:17:07.831595Z	9fda4c1	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - sonar-java

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	35.77	14.51	$\color{red}{\blacktriangle}$ +21.26	14.23	$\color{red}{\blacktriangle}$ +21.54	48/224
agentless	10.58	11.65	$\color{green}{\blacktriangledown}$ -1.07	13.40	$\color{green}{\blacktriangledown}$ -2.82	48/224
agentlessCodeCoverage	84.91	87.80	$\color{green}{\blacktriangledown}$ -2.89	84.36	$\color{red}{\blacktriangle}$ +0.55	48/224
agentlessLineCoverage	140.64	120.92	$\color{red}{\blacktriangle}$ +19.72	123.36	$\color{red}{\blacktriangle}$ +17.28	48/224

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[datadog-official]

More details

Both Karate adapters now pass scenario tags into exclusion-aware skip decisions, consistent with existing retry policy selection and reporting. The completed static review identified no actionable regressions.

Was this helpful? React 👍 or 👎

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit 9fda4c1}

[calvinbayer]

@autotest Review

[datadog-official]

More details

Scenario exclusion tags reach skip-policy selection in both Karate versions, consistent with the existing execution-policy and reporting paths.

Was this helpful? React 👍 or 👎

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit 5186251}

[calvinbayer]

@autotest Review

[datadog-official]

More details

Both Karate integrations now pass effective scenario tags into skip-policy selection, consistent with the existing retry and reporting paths.

Was this helpful? React 👍 or 👎

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit 17a769a}

[calvinbayer]

@autotest Review

[datadog-official]

More details

Both Karate integrations supply skip selection with the same effective scenario tags already used by execution policy and reporting, preserving exclusion handling and assigned Test Management properties. The completed static review identified no actionable regression.

Was this helpful? React 👍 or 👎

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit 072b0d2}

[calvinbayer]

@autotest Review

[datadog-official]

More details

Both Karate integrations now pass scenario tags into exclusion-aware skip selection, consistent with their retry and reporting policies. The completed static review identified no reportable defects.

Was this helpful? React 👍 or 👎

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit 21b929b}

[calvinbayer]

@autotest Review

[datadog-official]

More details

Both Karate adapters consistently pass scenario tags into the existing exclusion-aware skip policy. Static review found no actionable defect in that wiring or the added exclusion regressions.

Was this helpful? React 👍 or 👎

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit f79b33e}

[calvinbayer]

Shepherd verification — Karate exclusions and packaged-agent smoke

Tested by: Shepherd Agent (autonomous QA for Datadog Test Optimization)
Revision tested: f79b33e9eaf77ae440c63ab4530200a54c4f2b96
Environment: Local macOS arm64, supported JDK 21, Mockdog.

Verification	Result
Native Karate 1.0 / 1.3.1 / 1.4.1 / latest 1.x, and Karate 2.0.9 including the retry-disabled forked case	182 passed, 10 existing capability skips; zero failures/errors
Annotation/exclusion regressions	All five actually executed and passed
Spotless and CodeNarc	Passed
Scoped technical-debt/performance review	No actionable finding
Packaged-agent Jolokia default smoke	All 35 assertions passed
Packaged and observed runtime version	All identify f79b33e9ea

Native tasks were forced to execute on JDK 21, with fresh XML results. An initial host-default JDK 25 run skipped the unsupported Karate 1.x containers; those skips were not counted as verification and the supported-JDK matrix was rerun. The native exclusion cases verify protected scenario execution/failure, exclusion tags and retained Test Management assignment tags.

The packaged smoke used the normal Shepherd Crook workflow with --debug and the exact local agent JAR. Mockdog recorded 1,769 events: 1,404 tests, 188 suites, 32 modules and one session, with compressed test-cycle data. All 17 embedded component version resources and all 66 observed runtime StatusLogger configurations identify the tested revision. Karate hook/listener packaged classes were checked against their module/build outputs. Agent SHA256: 51663809f4aeb08805fb101fd32e5abe22f6f39a9bc638b1880eb886db6aa709.

Scope: Native Karate instrumentation tests exercise the Karate behavior. Jolokia is a cross-framework Java/TestNG packaged-agent smoke; it is not a Karate workload. Data was verified through local Mockdog assertions, not the production backend or Datadog UI. No backend API request/response/parsing changes were found in this PR's eight-file diff.

Issues found: No additional Karate behavior bug was found. Dani's substantive shared/API recommendations are already inherited, and scenario categories are already reused. The remaining double blank lines in the two native test files are cosmetic and were left unchanged.

CI note: GitHub's pending dd-gitlab/test_inst_latest: [25, 1/6] status is stale. GitLab job 2186774974 completed successfully without retries on this exact commit; its parent pipeline is successful. No CI status was manually overridden.

Normal Crook cleanup removed the injected agent and stopped its owned Mockdog process. Port 5259 was verified free, playground state (including the pre-existing .java-version) was preserved, and the SDK source worktree remained clean.