fix(runtime): reset runtime metric baselines on MicroVM identity refresh by litianningdatadog · Pull Request #20957 · DataDog/dd-trace-py

litianningdatadog · 2026-10-13T01:35:54Z

Stacked PRs:

chore(runtime): add explicit identity refresh #19778 — runtime identity foundation (merged)
chore(serverless): publish web request-start event #19898 — generic WSGI/ASGI/Django request-start event (merged)
- chore(runtime): refresh MicroVM identity #19939 — MicroVM /run listener and identity refresh
  - chore(tracer): rebuild the trace writer on MicroVM identity refresh #20952 — tracer: rebuild the trace writer (part 1 of fix(tracer): isolate MicroVM identity refresh #19820)
    - fix(telemetry): rebuild the worker on MicroVM identity refresh #20953 — telemetry: rebuild the worker (part 1 of fix(telemetry): rebuild worker on identity refresh #19821)
      - fix(aws_lambda): refresh the MicroVM identity when a web request starts #20954 — registers the /run listener; runtime-metrics runtime-ID tag (part 1 of chore(runtime): refresh runtime metric identity and collector state #19822)
        
        This PR fix(runtime): reset runtime metric baselines on MicroVM identity refresh #20957 — runtime metrics: reset collector baselines (part 2 of chore(runtime): refresh runtime metric identity and collector state #19822)
      - fix(telemetry): discard the old worker and exporter subscription on MicroVM refresh #20956 — telemetry: discard the old worker and exporter subscription (part 2 of fix(telemetry): rebuild worker on identity refresh #19821)
    - fix(tracer): drop trace state from before a MicroVM identity refresh #20955 — tracer: drop trace state from before the refresh (part 2 of fix(tracer): isolate MicroVM identity refresh #19820)

Part 1 PRs (#20952, #20953, #20954) can ship first. Part 2 PRs (#20955, #20956, #20957) can ship later.

Description

This PR is part 2 of #19822, on top of #20954. Its tip has exactly the code of #19822.

Review the last commit only. The first two commits repeat #20955's commit and #20956's last commit, so this branch reaches #19822's code without merge commits. They drop out when this branch is rebased after those PRs land.

#20954 makes the runtime-ID tag follow the refresh, but the collectors' first interval after /run still covers activity from before the snapshot. That sample is reported under the new runtime ID. The last commit:

Runtime metric collectors get reset(), which re-seeds their CPU time, context-switch, wall-clock and GC baselines. GCRuntimeMetricCollector.reset() re-seeds the collections baseline before clearing the pause window, so a collection between the two steps can show up as collections without a pause, but never as a pause without collections.
In a MicroVM with runtime-ID tagging enabled, RuntimeWorker registers for identity refreshes and resets its collectors. It records which runtime ID the baselines belong to (_collectors_runtime_id), updated only after a reset completes.
A flush() that sees a newer runtime ID resets and skips that interval. That covers a refresh callback that was missed (the worker was created mid-refresh) or failed. A reset that raises there is logged and retried on the next flush, instead of stopping the worker thread.
The reset and the flush share _identity_refresh_lock, so a reset never lands in the middle of a send. fix(aws_lambda): refresh the MicroVM identity when a web request starts #20954's flush() moves under that lock.
A fork hook resets again in the child, and stop() unregisters both hooks.

Testing

tests/runtime/test_runtime_metrics_api.py:

test_runtime_metrics_microvm_identity_refresh_resets_collectors, test_runtime_metrics_microvm_refresh_reset_waits_for_in_progress_flush, test_runtime_metrics_microvm_flush_skips_when_id_rotates_during_tag_collection
test_runtime_metrics_microvm_flush_resets_after_failed_refresh_reset, test_runtime_metrics_microvm_flush_resets_after_refresh_during_construction: a missed or failed reset is caught by the next flush, which sends nothing.
test_runtime_metrics_microvm_forked_child_flush_does_not_reset, test_runtime_metrics_microvm_fork_resets_collectors_for_child_id, test_runtime_metrics_microvm_failed_fork_reset_keeps_mismatch
test_runtime_metrics_non_microvm_flush_ignores_runtime_id_change, test_runtime_metrics_non_microvm_identity_refresh_does_not_reset_collectors, test_runtime_metrics_microvm_without_runtime_id_does_not_register_refresh, test_runtime_metrics_microvm_stop_unregisters_refresh_hook
test_runtime_metrics_non_microvm_flush_keeps_platform_tags_cached: also checks that there's no refresh lock outside a MicroVM.

tests/tracer/runtime/test_metric_collectors.py: the reset() tests for NativeProcessMetricCollector and GCRuntimeMetricCollector, including a collection during GCRuntimeMetricCollector.reset() and reset() on a disabled collector.

tests/tracer/runtime/test_runtime_id.py: test_import_ddtrace_in_microvm_wires_run_hook_to_tracer goes back to #19822's check of the tracer's published identity.

Local runs (Python 3.13):

runtime (venv 16cc321): all tests added here pass (49 passed, 7 failed). The 7 failures are the same older tests that fail on chore(runtime): refresh runtime metric identity and collector state #19822 and on chore(runtime): refresh MicroVM identity #19939's branch (see fix(aws_lambda): refresh the MicroVM identity when a web request starts #20954). The runtime suite is skip: true in tests/suitespec.yml.
tracer (venv 1ef5a52): 226 passed, 10 skipped, 1 xpassed. telemetry (venv 7dec5d4): 148 passed. appsec::sca (venv 1a7f51a): 127 passed.
At the repeated commits: tracer 225 passed after the first, and tracer 226 and telemetry 148 passed after the second.
scripts/lint style and scripts/lint typing on the changed files: passed.

Risks

Limited to MicroVMs with runtime-ID tagging. Elsewhere nothing is registered and flush() takes its existing path.

A flush waits for an in-flight reset, and a reset waits for an in-flight flush.
When a refresh reset is missed or fails, the next flush resets and sends nothing, so runtime metrics have a one-interval gap. That includes a skipped final flush if it happens at shutdown.

Reference

RFC

🤖 Generated with Claude Code

cit-pr-commenter-54b7da · 2026-10-13T01:37:20Z

Dependency direction analysis

⚠️ Existing dependency direction violations

There are 163 dependency direction violations that already exist on the base branch and have not been changed by this PR.

Show existing violations (showing 5 of 163 highest severity)

ddtrace.internal.tracemethods -×-> ddtrace.trace  (internal-core -> product:tracing, score=121)
ddtrace.llmobs._integrations.langgraph -×-> ddtrace.trace  (product:llmobs -> product:tracing, score=119)
ddtrace.llmobs._integrations.bedrock_agents -×-> ddtrace.trace  (product:llmobs -> product:tracing, score=119)
ddtrace.appsec._listeners -×-> ddtrace.trace  (product:appsec -> product:tracing, score=119)
ddtrace.llmobs._integrations.vllm -×-> ddtrace.trace  (product:llmobs -> product:tracing, score=119)

To see all violations, download the layers-base.json and layers-pr.json artifacts from this CI job and run:
uv run --script scripts/import-analysis/layers.py compare layers-base.json layers-pr.json

cit-pr-commenter-54b7da · 2026-10-13T01:37:43Z

Codeowners resolved as

Resolved from the full PR diff against tianning.li/3-5-2part-1-run-hook-and-runtime-id-tag using the target branch CODEOWNERS file.
CODEOWNERS team requests not listed below are not required by the current file set.

ddtrace/_trace/context.py                                               @DataDog/apm-sdk-capabilities
ddtrace/_trace/processor/__init__.py                                    @DataDog/apm-sdk-capabilities
ddtrace/_trace/span.py                                                  @DataDog/apm-sdk-capabilities @DataDog/apm-core-python
ddtrace/_trace/tracer.py                                                @DataDog/apm-sdk-capabilities
ddtrace/internal/runtime/collector.py                                   @DataDog/apm-sdk-capabilities
ddtrace/internal/runtime/metric_collectors.py                           @DataDog/apm-sdk-capabilities
ddtrace/internal/runtime/runtime_metrics.py                             @DataDog/apm-sdk-capabilities
ddtrace/internal/telemetry/writer.py                                    @DataDog/apm-python
ddtrace/internal/writer/writer.py                                       @DataDog/apm-core-python
releasenotes/notes/fix-aws-lambda-microvm-stale-traces-a26d7bf374c1b2c2.yaml  @DataDog/apm-python
releasenotes/notes/fix-runtime-metrics-microvm-identity-refresh-57f069db02623f9d.yaml  @DataDog/apm-python
tests/runtime/test_runtime_metrics_api.py                               @DataDog/apm-sdk-capabilities
tests/telemetry/test_writer.py                                          @DataDog/apm-core-python @DataDog/apm-python
tests/tracer/runtime/test_metric_collectors.py                          @DataDog/apm-sdk-capabilities
tests/tracer/runtime/test_runtime_id.py                                 @DataDog/apm-sdk-capabilities
tests/tracer/test_writer.py                                             @DataDog/apm-sdk-capabilities

cit-pr-commenter-54b7da · 2026-10-13T01:38:48Z

Circular import analysis

⚠️ Existing circular imports

There are 1 circular imports that already exist on the base branch and have not been changed by this PR.

ddtrace.errortracking._handled_exceptions.bytecode_injector -> ddtrace.errortracking._handled_exceptions.callbacks -> ddtrace.errortracking._handled_exceptions.collector -> ddtrace.errortracking._handled_exceptions.bytecode_reporting -> ddtrace.errortracking._handled_exceptions.bytecode_injector

datadog-datadog-us1-prod · 2026-10-13T01:38:54Z

✅ All CI checks and tests passed.

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 1fa33c7 | Docs | View more details | Give us feedback!}

pr-commenter · 2026-10-13T02:07:36Z

Benchmarks

Benchmark execution time: 2026-10-14 21:00:12

Comparing candidate commit 1fa33c7 in PR branch tianning.li/3-5-2part-2-reset-runtime-metric-baselines with baseline commit 4f08498 in branch tianning.li/3-5-2part-1-run-hook-and-runtime-id-tag.

📊 Benchmarking dashboard

Found 0 performance improvements and 4 performance regressions! Performance is the same for 359 metrics, 9 unstable metrics, 4 known flaky benchmarks, 4 flaky benchmarks without significant changes.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

🟩 = significantly better candidate vs. baseline
🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

scenario:httppropagationextract-full_t_id_datadog_headers

🟥 execution_time [+1.942µs; +2.074µs] or [+13.419%; +14.337%]

scenario:msgpackencoderscenario-simple_one_span

🟥 execution_time [+432.293ns; +474.414ns] or [+11.148%; +12.234%]

scenario:otelspan-start

🟥 execution_time [+2.818ms; +3.687ms] or [+10.909%; +14.269%]

scenario:recursivecomputation-shallow

🟥 execution_time [+66.919µs; +70.674µs] or [+9.600%; +10.139%]

Unstable benchmarks

These benchmarks have a confidence interval too wide to call a change; treat them as noise rather than signal.

scenario:coreapiscenario-context_with_data_listeners

unstable execution_time [-786.008ns; +682.508ns] or [-7.530%; +6.539%]

scenario:coreapiscenario-core_dispatch_1_listener

unstable execution_time [-41.070ns; +37.199ns] or [-6.178%; +5.596%]

scenario:coreapiscenario-core_dispatch_50_listeners

unstable execution_time [-2006.118ns; +1834.037ns] or [-10.039%; +9.178%]

scenario:coreapiscenario-core_dispatch_exception_listeners

unstable execution_time [-1988.377ns; +1622.073ns] or [-10.444%; +8.520%]

scenario:coreapiscenario-core_dispatch_listeners

unstable execution_time [-365.816ns; +391.965ns] or [-8.611%; +9.227%]

scenario:coreapiscenario-core_dispatch_no_args_listeners

unstable execution_time [-216.081ns; +240.518ns] or [-8.084%; +8.998%]

scenario:coreapiscenario-core_dispatch_with_results_1_listener

unstable execution_time [-124.568ns; +60.865ns] or [-8.997%; +4.396%]

scenario:coreapiscenario-core_dispatch_with_results_50_listeners

unstable execution_time [-4033.861ns; +5264.871ns] or [-8.363%; +10.915%]

scenario:coreapiscenario-core_dispatch_with_results_listeners

unstable execution_time [-844.472ns; +1043.576ns] or [-8.313%; +10.274%]

Known flaky benchmarks

These benchmarks are marked as flaky and will not trigger a failure. Modify FLAKY_BENCHMARKS_REGEX to control which benchmarks are marked as flaky.

scenario:httppropagationinject-ids_only

🟥 execution_time [+2.836µs; +2.927µs] or [+21.307%; +21.992%]

scenario:span-start

🟥 execution_time [+1.538ms; +1.819ms] or [+13.872%; +16.413%]

scenario:telemetryaddmetric-1-count-metric-1-times

🟥 execution_time [+178.070ns; +219.301ns] or [+8.991%; +11.073%]

scenario:tracer-small

🟥 execution_time [+48.741µs; +50.118µs] or [+20.118%; +20.687%]

Known flaky benchmarks without significant changes:

scenario:errortrackingflasksqli-baseline
scenario:flasksimple-iast-get
scenario:sethttpmeta-all-enabled
scenario:telemetryaddmetric-record-100-metrics

Part 1 stops the old trace writer the normal way. The old writer still flushes traces and client stats buffered before the refresh under the snapshot's runtime ID, /run waits for that final flush, and spans that were open across /run join the new runtime. This change replaces that with a discard: - Writers get drop_buffered_traces(). In MicroVMs it closes the write gate and drops the native exporter without shutting it down. A discarded writer skips its final flush, never starts another send, and its stop() does not wait for a send stuck in retries. - The span aggregator publishes a (generation, runtime ID) identity. A refresh bumps the generation, drops the aggregator's trace buffer and discards the writer under the aggregator lock. Other writer swaps take the same lock. - Spans and retained Context objects from an older generation are rejected before they reach the new writer. The context that was active at /run is detached. - A sync-mode write is buffered under the aggregator lock and sent after the lock is released, so a send stuck in retries cannot block span starts or /run. This replaces part 1's _stop_accepting_writes() with drop_buffered_traces(). It also switches part 1's refresh from _recreate(reset_buffer=False, flush_writer=False) to _recreate(reset_buffer=True, drop_buffered_traces=True). Repeat of the commit on tianning.li/3-3-2part-2-discard-stale-traces, on top of #19821 and #19822 part 1. It drops out once that branch lands and this one is rebased. Combining it with #19821 part 1 needs one extra change: _downgrade() reads the old exporter before calling _create_exporter(), which in MicroVMs publishes its result. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…icroVM refresh - _discard_worker() uses the native worker's non-flushing drop() when it has one. It falls back to stop(), which flushes once under the previous runtime ID. The bundled TelemetryWorker does not expose drop() yet, so the fallback is what runs today. - The trace exporter unsubscribes from telemetry worker changes when #19820's refresh drops it (_drop_exporter() and the dropped path of on_shutdown()). Tests return to #19821's drop() versions, and the end-to-end test now checks that the tracer refresh with telemetry does not wait for a stuck send. Repeat of the last commit on tianning.li/3-4-2part-2-discard-stale-telemetry, on top of #19822 part 1. It drops out once that branch lands and this one is rebased. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Part 1 makes the runtime-id tag follow the refresh, but the collectors' first interval after /run still covers activity from before the snapshot. That sample is reported under the new runtime ID. - Runtime metric collectors get reset(), which re-seeds their CPU time, context-switch, wall-clock and GC-pause baselines. - In MicroVMs that tag runtime metrics with runtime-id, RuntimeWorker registers for identity refreshes and resets its collectors. It records which runtime ID the baselines belong to. A flush that sees a newer runtime ID resets and skips that interval, which covers a refresh callback that was missed or failed. The reset and the flush share a lock. A fork hook resets again in the child. Part 1's flush() moves under that lock. The wiring test returns to checking the tracer's published identity. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

litianningdatadog added the aws-microvm Work related to AWS MicroVM onboarding label Oct 13, 2026

litianningdatadog force-pushed the tianning.li/3-5-2part-2-reset-runtime-metric-baselines branch from e9cacbf to e97ac10 Compare October 13, 2026 19:10

litianningdatadog force-pushed the tianning.li/3-5-2part-1-run-hook-and-runtime-id-tag branch 2 times, most recently from 18c7d5e to bd81cc2 Compare October 13, 2026 19:25

litianningdatadog force-pushed the tianning.li/3-5-2part-2-reset-runtime-metric-baselines branch from e97ac10 to 8f5c7f3 Compare October 13, 2026 19:25

litianningdatadog force-pushed the tianning.li/3-5-2part-1-run-hook-and-runtime-id-tag branch from bd81cc2 to 6560882 Compare October 14, 2026 17:35

litianningdatadog force-pushed the tianning.li/3-5-2part-2-reset-runtime-metric-baselines branch 3 times, most recently from bb78835 to 5af6b8e Compare October 14, 2026 19:56

litianningdatadog force-pushed the tianning.li/3-5-2part-1-run-hook-and-runtime-id-tag branch from 6560882 to 3c5a337 Compare October 14, 2026 19:56

litianningdatadog and others added 3 commits October 14, 2026 16:24

litianningdatadog force-pushed the tianning.li/3-5-2part-1-run-hook-and-runtime-id-tag branch from 3c5a337 to 4f08498 Compare October 14, 2026 20:27

litianningdatadog force-pushed the tianning.li/3-5-2part-2-reset-runtime-metric-baselines branch from 5af6b8e to 1fa33c7 Compare October 14, 2026 20:27

Conversation

litianningdatadog commented Oct 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Testing

Risks

Reference

Uh oh!

cit-pr-commenter-54b7da Bot commented Oct 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Dependency direction analysis

⚠️ Existing dependency direction violations

Uh oh!

cit-pr-commenter-54b7da Bot commented Oct 13, 2026

Codeowners resolved as

Uh oh!

cit-pr-commenter-54b7da Bot commented Oct 13, 2026

Circular import analysis

⚠️ Existing circular imports

Uh oh!

datadog-datadog-us1-prod Bot commented Oct 13, 2026 • edited by datadog-datadog-prod-us1-2 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pr-commenter Bot commented Oct 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Explanation

More details about the CI and significant changes

scenario:httppropagationextract-full_t_id_datadog_headers

scenario:msgpackencoderscenario-simple_one_span

scenario:otelspan-start

scenario:recursivecomputation-shallow

Unstable benchmarks

scenario:coreapiscenario-context_with_data_listeners

scenario:coreapiscenario-core_dispatch_1_listener

scenario:coreapiscenario-core_dispatch_50_listeners

scenario:coreapiscenario-core_dispatch_exception_listeners

scenario:coreapiscenario-core_dispatch_listeners

scenario:coreapiscenario-core_dispatch_no_args_listeners

scenario:coreapiscenario-core_dispatch_with_results_1_listener

scenario:coreapiscenario-core_dispatch_with_results_50_listeners

scenario:coreapiscenario-core_dispatch_with_results_listeners

Known flaky benchmarks

scenario:httppropagationinject-ids_only

scenario:span-start

scenario:telemetryaddmetric-1-count-metric-1-times

scenario:tracer-small

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

litianningdatadog commented Oct 13, 2026 •

edited

Loading

cit-pr-commenter-54b7da Bot commented Oct 13, 2026 •

edited

Loading

datadog-datadog-us1-prod Bot commented Oct 13, 2026 •

edited by datadog-datadog-prod-us1-2 Bot

Loading

pr-commenter Bot commented Oct 13, 2026 •

edited

Loading