chore(runtime): refresh MicroVM identity

[litianningdatadog]

Stacked PRs:

• chore(runtime): add explicit identity refresh #19778 — runtime identity foundation (merged)
• chore(serverless): publish web request-start event #19898 — generic WSGI/ASGI/Django request-start event (merged)
  • This PR chore(runtime): refresh MicroVM identity #19939 — MicroVM /run listener and identity refresh
    • fix(tracer): isolate MicroVM identity refresh #19820 — tracer/native writer refresh
      • fix(telemetry): rebuild worker on identity refresh #19821 — telemetry worker refresh
        • chore(runtime): refresh runtime metric identity and collector state #19822 — runtime-metrics runtime-ID tags; registers the listener

Description

This PR adds the MicroVM side of the web.request.starting event that #19898 publishes. The method and path check lives in a MicroVM-specific listener instead of the integrations, as asked in r4146791727.

• maybe_refresh_identity(method, path) matches only POST /aws/lambda-microvms/runtime/v1/run and calls refresh_identity() once per process.

Nothing in this PR registers maybe_refresh_identity(). #19822 registers it with core.on() inside its MicroVM check, at the top of the stack, once the tracer, telemetry and runtime metrics can each rebuild on refresh. Merging this PR alone doesn't change runtime behavior.

WSGI and Django can both observe the same /run request, so the refresh has to happen exactly once. A forksafe.Event records that it's done and is checked first without a lock, so after the refresh the listener costs one flag check per request. A lock private to _runtime_id.py guards the refresh itself, so concurrent observers still rotate the ID once. No other module takes that lock.

/run reaches a process once, so there's no retry. If a refresh callback fails, the failure is logged and the remaining callbacks still run. Each consumer keeps its own state consistent with get_runtime_id().

Smaller changes:

• WEB_REQUEST_STARTING_EVENT moves to ddtrace.internal.constants, and WebFrameworkEvents.WEB_REQUEST_STARTING aliases it. chore(runtime): refresh runtime metric identity and collector state #19822 registers the listener from ddtrace/trace/__init__.py, which can't import ddtrace.contrib: its package __init__ loads trace_handlers, which imports ddtrace.trace back before tracer exists.
• MICROVM_RUN_HOOK_METHOD and MICROVM_RUN_HOOK_PATH live in ddtrace.internal.serverless.

Testing

tests/tracer/runtime/test_runtime_id.py:

• test_maybe_refresh_identity_matches_microvm_run_hook, test_maybe_refresh_identity_ignores_other_requests: only POST to the /run path refreshes.
• test_identity_refresh_hook_runs_before_root_span_creation: the refresh happens before the /run root span is created.
• test_maybe_refresh_identity_is_thread_safe: concurrent observers rotate the ID once.
• test_identity_refresh_callback_failure_is_logged_and_not_retried
• test_microvm_refresh_guard_resets_after_fork
• test_maybe_refresh_identity_noop_outside_microvm, test_identity_refresh_listener_noop_outside_microvm, test_identity_refresh_listener_noop_for_blank_microvm_env
• test_import_ddtrace_in_microvm_environment, test_import_ddtrace_outside_microvm_does_not_import_core

Integration tests:

• test_microvm_run_hook_refreshes_identity in both tests/contrib/wsgi/test_wsgi.py and tests/contrib/asgi/test_asgi.py: a /run request refreshes the identity end to end.
• tests/contrib/wsgi/test_wsgi.py::test_web_request_starting_isolates_listener_errors: a failing listener doesn't fail the request.

Risks

None outside a MicroVM: nothing in this PR registers the listener, #19822 registers it only inside a MicroVM, and maybe_refresh_identity() returns early. The import tests above check that the no-op path doesn't import ddtrace.internal.core. Inside a MicroVM nothing changes until #19822 registers the listener.

Additional Notes

An earlier revision shared a refresh lock with consumers (get_runtime_identity_refresh_lock()) and retried failed callbacks against the same identity. I dropped both after review:

• r4186658721: don't expose a global lock as cross-module API or take it on every span start and finish. Consumers now read a published identity snapshot instead.
• r4188215619: the ID is committed before the callbacks run, so retries couldn't stop mixed-identity data from getting out. Each consumer now owns its own consistency.
• r4188215636: callback order was nondeterministic and the tests forced it. Without a shared lock or retry queue, order doesn't matter.

Reference

• RFC

[datadog-datadog-prod-us1-2]

✅ All CI checks and tests passed.

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 174c28b | Docs | View more details | Give us feedback!}

[cit-pr-commenter-54b7da]

Dependency direction analysis

⚠️ Existing dependency direction violations

There are 163 dependency direction violations that already exist on the base branch and have not been changed by this PR.

Show existing violations (showing 5 of 163 highest severity)

ddtrace.internal.tracemethods -×-> ddtrace.trace  (internal-core -> product:tracing, score=121)
ddtrace.internal.test_visibility.api -×-> ddtrace.trace  (product:ci_visibility -> product:tracing, score=119)
ddtrace.llmobs._integrations.claude_agent_sdk -×-> ddtrace.trace  (product:llmobs -> product:tracing, score=119)
ddtrace.llmobs._integrations.llama_index -×-> ddtrace.trace  (product:llmobs -> product:tracing, score=119)
ddtrace.internal.ci_visibility.filters -×-> ddtrace.trace  (product:ci_visibility -> product:tracing, score=119)

To see all violations, download the layers-base.json and layers-pr.json artifacts from this CI job and run:

uv run --script scripts/import-analysis/layers.py compare layers-base.json layers-pr.json

[cit-pr-commenter-54b7da]

Circular import analysis

⚠️ Existing circular imports

There are 1 circular imports that already exist on the base branch and have not been changed by this PR.

ddtrace.errortracking._handled_exceptions.bytecode_injector -> ddtrace.errortracking._handled_exceptions.callbacks -> ddtrace.errortracking._handled_exceptions.collector -> ddtrace.errortracking._handled_exceptions.bytecode_reporting -> ddtrace.errortracking._handled_exceptions.bytecode_injector

[cit-pr-commenter-54b7da]

Codeowners resolved as

Resolved from the full PR diff against main using the target branch CODEOWNERS file.
CODEOWNERS team requests not listed below are not required by the current file set.

No remaining files require a CODEOWNERS review.

[pr-commenter]

Benchmarks

Benchmark execution time: 2026-10-10 04:17:36

Comparing candidate commit 174c28b in PR branch tianning.li/3-wsgi-asgi-integration-based-refresh with baseline commit 73de6c2 in branch main.

📊 Benchmarking dashboard

Found 0 performance improvements and 4 performance regressions! Performance is the same for 351 metrics, 9 unstable metrics, 4 known flaky benchmarks, 4 flaky benchmarks without significant changes.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

• 🟩 = significantly better candidate vs. baseline
• 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

scenario:httppropagationextract-tracecontext_headers

• 🟥 execution_time [+2.764µs; +2.992µs] or [+8.802%; +9.527%]

scenario:msgpackencoderscenario-simple_one_span

• 🟥 execution_time [+529.275ns; +592.509ns] or [+13.876%; +15.534%]

scenario:otelspan-start

• 🟥 execution_time [+2.753ms; +3.594ms] or [+10.837%; +14.148%]

scenario:recursivecomputation-shallow

• 🟥 execution_time [+60.851µs; +64.167µs] or [+8.784%; +9.263%]

Unstable benchmarks

These benchmarks have a confidence interval too wide to call a change; treat them as noise rather than signal.

scenario:coreapiscenario-context_with_data_listeners

• unstable execution_time [-727.330ns; +741.474ns] or [-6.980%; +7.116%]

scenario:coreapiscenario-core_dispatch_1_listener

• unstable execution_time [-38.733ns; +39.579ns] or [-5.827%; +5.954%]

scenario:coreapiscenario-core_dispatch_50_listeners

• unstable execution_time [-1946.069ns; +1891.510ns] or [-9.758%; +9.485%]

scenario:coreapiscenario-core_dispatch_exception_listeners

• unstable execution_time [-1921.872ns; +1767.208ns] or [-10.033%; +9.226%]

scenario:coreapiscenario-core_dispatch_listeners

• unstable execution_time [-368.592ns; +390.879ns] or [-8.662%; +9.186%]

scenario:coreapiscenario-core_dispatch_no_args_listeners

• unstable execution_time [-214.629ns; +242.006ns] or [-8.019%; +9.042%]

scenario:coreapiscenario-core_dispatch_with_results_1_listener

• unstable execution_time [-82.004ns; +102.597ns] or [-6.087%; +7.616%]

scenario:coreapiscenario-core_dispatch_with_results_50_listeners

• unstable execution_time [-3775.383ns; +5576.196ns] or [-7.853%; +11.599%]

scenario:coreapiscenario-core_dispatch_with_results_listeners

• unstable execution_time [-819.418ns; +1086.230ns] or [-8.048%; +10.669%]

Known flaky benchmarks

These benchmarks are marked as flaky and will not trigger a failure. Modify FLAKY_BENCHMARKS_REGEX to control which benchmarks are marked as flaky.

scenario:httppropagationinject-ids_only

• 🟥 execution_time [+2.845µs; +2.926µs] or [+21.458%; +22.065%]

scenario:span-start

• 🟥 execution_time [+1.504ms; +1.802ms] or [+13.687%; +16.396%]

scenario:telemetryaddmetric-1-count-metric-1-times

• 🟥 execution_time [+185.363ns; +221.801ns] or [+9.465%; +11.325%]

scenario:tracer-small

• 🟥 execution_time [+43.616µs; +45.434µs] or [+17.849%; +18.593%]

Known flaky benchmarks without significant changes:

• scenario:errortrackingflasksqli-baseline
• scenario:flasksimple-iast-get
• scenario:sethttpmeta-all-enabled
• scenario:telemetryaddmetric-record-100-metrics

[Copilot]

Pull request overview

This PR wires AWS Lambda MicroVM “/run” lifecycle detection into ddtrace startup so the runtime identity can be refreshed after import-time when a MicroVM reuses the same Python process for a new logical runtime.

Changes:

• Adds a MicroVM /run request listener (web.request.starting) that refreshes runtime identity once per process, with a forksafe/threadsafe guard.
• Centralizes MicroVM /run method/path constants and reuses them from the contrib web dispatch helper.
• Adds subprocess + integration test coverage to ensure refresh happens before root span creation and is thread-safe.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
ddtrace/internal/runtime/__init__.py	Adds MicroVM /run hook constants and request-event listener that refreshes identity once per process.
ddtrace/__init__.py	Registers the MicroVM identity-refresh hook during ddtrace import.
ddtrace/contrib/internal/web.py	Uses the shared MicroVM /run constants when dispatching web.request.starting.
ddtrace/contrib/_events/web_framework.py	Minor formatting-only change.
tests/tracer/runtime/test_runtime_id.py	Adds focused subprocess tests for hook registration behavior, matching logic, ordering vs. root span, thread-safety, and fork reset.
tests/contrib/asgi/test_asgi.py	Adds ASGI integration coverage that /run refreshes identity exactly once.
tests/contrib/wsgi/test_wsgi.py	Adds WSGI integration coverage that /run refreshes identity exactly once.
releasenotes/notes/aws-lambda-microvm-identity-refresh-3a672cd6bcbad16d.yaml	Adds a release note entry for the MicroVM identity refresh feature.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: cddc13b4a8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 09ba3d964b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1063bc02b4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 25e8bb8778

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[mabdinur]

Focused review of fork safety, transition stability, and request-path cost. I left the remaining findings inline.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6d2daa1e70

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[dubloom]

Don't see any blockers

[datadog-datadog-prod-us1-2]

More details

The MicroVM refresh guard prevents concurrent observers from rotating identity more than once and resets after fork. Listener registration remains deferred to the later stacked PR, so this change alone does not activate request-triggered refresh.

Was this helpful? React 👍 or 👎

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit ef4471a}

[emmettbutler]

lgtm

[litianningdatadog]

/merge

[gh-worker-devflow-routing-ef8351]

View all feedbacks in Devflow UI.

2026-10-13 01:05:14 UTC ℹ️ Start processing command /merge

---

2026-10-13 01:05:27 UTC ℹ️ MergeQueue: Pull request is not mergeable yet

It will be processed automatically as soon as GitHub reports it as mergeable. View in MergeQueue UI.

• Run /code blockers to see what is blocking it.
• Run /remove to cancel it.

---

2026-10-13 05:09:11 UTC ⚠️ MergeQueue: This merge request was unqueued

devflow unqueued this merge request: It did not become mergeable within the expected time

[litianningdatadog]

/code blockers

[gh-worker-devflow-routing-ef8351]

View all feedbacks in Devflow UI.

2026-10-13 13:51:56 UTC ℹ️ Start processing command /code blockers

---

2026-10-13 13:51:57 UTC ℹ️ Devflow:

Checking merge blockers for #19939...

---

2026-10-13 13:51:59 UTC ℹ️ Devflow: /code blockers

Detected 1 merge blocker(s) to address:

🔴 Blocking

• This pull request has unresolved conversations.
  1 conversation is not resolved in the changes view. Outdated conversations releasenotes/notes/aws-lambda-microvm-identity-refresh-3a672cd6bcbad16d.yaml, ddtrace/internal/_runtime_id.py, ddtrace/__init__.py, ddtrace/internal/_runtime_id.py, tests/tracer/runtime/test_runtime_id.py and 5 more are not resolved.
  Hint: Resolve every conversation before merging.

[litianningdatadog]

/code blockers

[gh-worker-devflow-routing-ef8351]

View all feedbacks in Devflow UI.

2026-10-13 13:55:00 UTC ℹ️ Start processing command /code blockers

---

2026-10-13 13:55:01 UTC ℹ️ Devflow:

Checking merge blockers for #19939...

---

2026-10-13 13:55:02 UTC ℹ️ Devflow: /code blockers

Detected 1 merge blocker(s) to address:

🔴 Blocking

• This pull request has unresolved conversations.
  Outdated conversations ddtrace/internal/_runtime_id.py, ddtrace/internal/_runtime_id.py and ddtrace/internal/_runtime_id.py are not resolved.
  Hint: Resolve every conversation before merging.

[litianningdatadog]

/code blocker

[litianningdatadog]

/merge

[gh-worker-devflow-routing-ef8351]

View all feedbacks in Devflow UI.

2026-10-13 13:59:56 UTC ℹ️ Start processing command /merge

---

2026-10-13 14:00:01 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in main is approximately 59m (p90).

---

2026-10-13 14:50:30 UTC ℹ️ MergeQueue: This merge request was merged