Skip to content

Stop trusting compromised eMudhra roots in the exploration-tests image - #13026

Draft
eliottness wants to merge 1 commit into
masterfrom
eliottness/emudhra-ca-roots
Draft

eliottness wants to merge 1 commit into
masterfrom
eliottness/emudhra-ca-roots

Conversation

@eliottness

Copy link
Copy Markdown

What

Images used by this repository's CI still ship CA stores that trust emSign ECC Root CA - G3 and/or AAA Certificate Services, the roots behind the compromised eMudhra intermediates (incident IR-62343). The APPSEC-71273 ledger lists 17 impacted image references here. Images that CI never builds or pulls are left alone.

Changes

  • dd-java-agent/agent-debugger/exploration-tests/Dockerfile.exploration-tests
    • patched in place (no clean version to move to): re-run build-exploration-tests-image after merge; no clean published exploration-tests version identified: the tag still resolves to sha256:203f04aaab02f9bbb4a0ca9b092d317c1fb99d0e5c662ebd0da296fe79ba4a43 and wrapper scan finds both roots;

Verification

Every new image reference and every changed Dockerfile was pulled or built and scanned with the eMudhra scanner (checks every file in the image for the roots' public keys); all return empty hits. Of the 17 impacted references: 0 fixed here, 1 fixed at their in-repo image source (clean once the image job republishes), 16 follow-ups below.

  • dd-java-agent/agent-debugger/exploration-tests/Dockerfile.exploration-tests: full image build left to PR CI

Follow-ups (not in this PR)

  • Needs rebuild at source (13):
    • registry.ddbuild.io/images/mirror/amazon/aws-cli:2.4.29: DataDog/images mirror/amazon/aws-cli; upstream 2.38.5 still has seven affected paths and no 2.38.5 mirror tag exists.
    • 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/benchmarking-platform:java-dsm-kafka: DataDog/benchmarking-platform branch java-dsm-kafka; source checkout is outside this lane.
    • 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/benchmarking-platform:java-debugger: DataDog/benchmarking-platform branch java-debugger; source checkout is outside this lane.
    • registry.ddbuild.io/images/mirror/dd-apm-test-agent/ddapm-test-agent:v1.64.1: DataDog/dd-apm-test-agent and DataDog/images mirror; mirrored v1.68.0 and upstream v1.69.0 each retain six affected paths.
    • registry.ddbuild.io/images/mirror/datadog/agent:7.40.1: DataDog/datadog-agent and DataDog/images mirror; upstream 7.84.5 still embeds affected roots and its mirror tag is unavailable.
    • registry.ddbuild.io/images/mirror/sqlserver:latest: DataDog/images mirror/sqlserver; current latest still has OS and embedded SQL certificate hits.
    • registry.ddbuild.io/images/mirror/dd-trace-java-docker-build:ci-base: DataDog/images mirror/dd-trace-java-docker-build ci-base; patch OS, Python, bundled JDK, and datadog-ci stores.
    • registry.ddbuild.io/images/mirror/dd-trace-java-docker-build:ci-7: DataDog/images mirror/dd-trace-java-docker-build ci-7; patch OS, Python, bundled JDK, and datadog-ci stores.
    • registry.ddbuild.io/images/mirror/dd-trace-java-docker-build:ci-8: DataDog/images mirror/dd-trace-java-docker-build ci-8; patch OS, Python, bundled JDK, and datadog-ci stores.
    • registry.ddbuild.io/images/mirror/dd-trace-java-docker-build:ci-11: DataDog/images mirror/dd-trace-java-docker-build ci-11; patch OS, Python, bundled JDK, and datadog-ci stores.
    • registry.ddbuild.io/images/mirror/dd-trace-java-docker-build:ci-17: DataDog/images mirror/dd-trace-java-docker-build ci-17; patch OS, Python, bundled JDK, and datadog-ci stores.
    • registry.ddbuild.io/images/mirror/dd-trace-java-docker-build:ci-21: DataDog/images mirror/dd-trace-java-docker-build ci-21; patch OS, Python, bundled JDK, and datadog-ci stores.
    • ... 1 more in the APPSEC-71273 ledger
  • Current emulator test image still has 17 affected paths; upstream stable tag also has 12 vendored CA hits and cannot replace emulator tooling. (1):
    • gcr.io/google.com/cloudsdktool/google-cloud-cli:emulators: Current emulator test image still has 17 affected paths; upstream stable tag also has 12 vendored CA hits and cannot replace emulator tooling.
  • Current latest SQL Server test target embeds affected certificates in system.certificates.sfp; needs patched upstream SQL Server image. (1):
    • mcr.microsoft.com/mssql/server:latest: Current latest SQL Server test target embeds affected certificates in system.certificates.sfp; needs patched upstream SQL Server image.
  • pull (1):
    • 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/agent-key-management-tools/gpg:1: Error response from daemon: failed to resolve reference "486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/agent-key-management-tools/gpg:1": pull access denied, repository does not exist or may require authorization: auth

Jira: APPSEC-71277 (parent APPSEC-71273)

Upgrade ca-certificates in the debian:bookworm-slim base and delete emSign ECC Root CA - G3 / AAA Certificate Services
from every SDKMAN and Gradle-provisioned JDK cacerts in the exploration-tests image (IR-62343). The image is published
by the manual build-exploration-tests-image job; re-run it after merge.

Jira: APPSEC-71277
@datadog-datadog-prod-us1

datadog-datadog-prod-us1 Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 61.45% (+1.60%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 626140e | Docs | Give us feedback!

@dd-octo-sts

dd-octo-sts Bot commented Oct 23, 2026

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 13.99 s 13.95 s [-0.5%; +1.1%] (no difference)
startup:insecure-bank:tracing:Agent 12.95 s 13.02 s [-1.5%; +0.4%] (no difference)
startup:petclinic:appsec:Agent 17.06 s 16.96 s [-0.2%; +1.3%] (no difference)
startup:petclinic:iast:Agent 16.87 s 17.00 s [-1.6%; +0.1%] (no difference)
startup:petclinic:profiling:Agent 16.16 s 16.71 s [-7.5%; +0.9%] (no difference)
startup:petclinic:sca:Agent 16.57 s 16.92 s [-6.6%; +2.4%] (no difference)
startup:petclinic:tracing:Agent 16.25 s 16.24 s [-0.9%; +1.1%] (no difference)

Commit: 626140ec · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant