Repository navigation
chore(runtime): refresh runtime metric identity and collector state - #19822
litianningdatadog wants to merge 1 commit into
Conversation
litianningdatadog
added
changelog/no-changelog
Circular import analysis
|
Dependency direction analysis
|
Codeowners resolved asResolved from the full PR diff against |
litianningdatadog
changed the title
|
✅ All CI checks and tests passed. 🎉 All green!🧪 All tests passed 🔗 Commit SHA: e5e703f | Docs | View more details | Give us feedback! |
BenchmarksBenchmark execution time: 2026-10-15 22:32:52 Comparing candidate commit e5e703f in PR branch Found 3 performance improvements and 4 performance regressions! Performance is the same for 354 metrics, 12 unstable metrics, 3 known flaky benchmarks, 5 flaky benchmarks without significant changes.
|
d59e112 to
16a5332
Compare
d6a1546 to
335f6ac
Compare
16a5332 to
8dd7e8e
Compare
335f6ac to
2e625cf
Compare
cde3045 to
a0e3c42
Compare
2e625cf to
7d36a71
Compare
There was a problem hiding this comment.
Pull request overview
This PR ensures runtime-metrics platform tags that include runtime identity (notably runtime-id) are refreshed when the runtime identity changes (e.g., after a MicroVM /run-triggered identity refresh), so runtime metrics emitted afterward carry the new identity.
Changes:
- Register a runtime-id change listener in
RuntimeWorkerto rebuild cached platform tags when identity changes. - Refactor platform-tag construction into a helper and add an identity-refresh callback method.
- Add a subprocess test asserting
runtime-id:tags update afterruntime.refresh_identity().
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
ddtrace/internal/runtime/runtime_metrics.py |
Builds platform tags via a helper and hooks runtime-id changes to refresh cached _platform_tags. |
tests/runtime/test_runtime_metrics_api.py |
Adds a subprocess test covering platform-tag refresh behavior on identity refresh. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
75336e0 to
4fed6d7
Compare
30e6c88 to
e484bf7
Compare
4fed6d7 to
c037394
Compare
e484bf7 to
fb28096
Compare
c037394 to
e7d5175
Compare
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e7d51759d3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
![datadog-prod-us1-4[bot]](https://avatars.githubusercontent.com/u/365230?s=60&v=4){kind=small}
There was a problem hiding this comment.
The two most critical issues are: (1) a retry-idempotency gap in the identity-refresh coordinator — when any subscriber fails, already-completed callbacks (tracer writer recreation and runtime metrics reset) re-execute for the same runtime ID, dropping buffered traces and discarding legitimate metric deltas; and (2) a race between reset() and flush() in the runtime metrics collector that can emit negative or cross-identity deltas when an identity refresh overlaps a periodic flush.
🤖 Bits Code Review · Commit 3d4a374 · @DataDog review to ask questions
emmettbutler
left a comment
emmettbutler
left a comment
There was a problem hiding this comment.
Deferring review since the base branch is not main
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fd2f295d2c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| runtime_id = get_runtime_id() | ||
| self._platform_tags = self._collect_platform_tags() | ||
| runtime_metrics = list(self._runtime_metrics) | ||
| if runtime_id == get_runtime_id(): |
There was a problem hiding this comment.
When the MicroVM hook rotates the runtime ID but an identity-refresh callback raises before this worker's callback runs—or this worker's own collector reset fails partway—the coordinator leaves callbacks pending for a later request while the new ID is already visible. A periodic flush between those attempts reaches this block with the old collector baselines, recollects the new runtime-id, and sends a mixed pre/post-refresh interval under that new ID. The before/after comparison cannot detect this because both reads occur while holding the refresh lock; track the ID associated with the collector baselines and reset before collection when it differs.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 051c02d. RuntimeWorker now records the runtime ID its collector baselines belong to (_collectors_runtime_id), updated only after reset() returns. flush() compares it with the current ID under the refresh lock and resets before collecting on a mismatch, so a failed or still-pending callback can't send pre-refresh baselines under the new runtime-id.
Covered by test_runtime_metrics_microvm_flush_resets_after_failed_refresh_reset.
There was a problem hiding this comment.
Follow-up in 7e091b1: on a mismatch, flush() now resets and sends nothing for that interval rather than collecting right after the reset (a just-reset interval is microseconds long). If that reset raises, it's logged and retried on the next flush instead of stopping the worker's periodic thread. A retried callback for an ID a flush already reset for is a no-op (see the thread on _on_identity_refresh).
| with self._identity_refresh_lock: | ||
| self._runtime_metrics.reset() |
There was a problem hiding this comment.
On every successful MicroVM refresh, the runtime ID is rotated before the callbacks are invoked, and those callbacks come from an unordered set. If the tracer or telemetry rebuild runs before this callback, its CPU, context switches, and GC activity occur under the new identity but are discarded here; if this callback happens first, the same activity is retained. Runtime metrics therefore vary with callback iteration order and can omit initialization work from the new runtime, so the collector reset needs a defined position immediately at the identity boundary rather than an unordered subscriber callback.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Deferring this one to a follow-up. What changes with callback order is whether the CPU and GC activity of the subscribers that run before this one (tracer/writer and telemetry worker rebuilds) lands in the first post-refresh interval or is discarded. That work happens during the transition itself and doesn't clearly belong to either identity.
A fixed position needs ordering support in the refresh coordinator in _runtime_id.py (#19939), e.g. an early callback tier that runs right after rotation. Calling runtime metrics directly from there isn't an option: runtime_metrics already imports from _runtime_id, so it would create an import cycle. I'd rather not widen this PR with a coordinator change.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e16f6204f3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| monitor.reset() | ||
| self._reset_state() |
There was a problem hiding this comment.
When a collection starts after monitor.reset() but before gc.get_stats() is sampled by _reset_state()—which can itself occur while allocating the returned stats/list—the monitor records that pause while the new collections baseline already includes the collection. The next flush therefore reports a positive GC pause with zero corresponding collection delta. Reset the pause window and reseed the collection baseline under one synchronization boundary so activity during the identity transition is either retained or discarded consistently.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Reordered rather than made atomic (051c02d): reset() now reseeds the collections baseline before clearing the pause window, the same order _on_modules_load() uses at enable time.
A collection that lands between the two steps (including one triggered by an allocation inside gc.get_stats()) now shows up as a collections delta with its pause discarded. The reverse, a pause with no collections delta, can no longer happen. Worst case, one collection's pause is missing from the first interval after a refresh.
True atomicity needs both values from one source, i.e. counting per-generation collections in GCPauseMonitor._on_gc instead of reading gc.get_stats(). That changes where gc.collections.* comes from and needs a fallback when gc callbacks aren't supported, so I'd rather keep it out of this PR. Holding the monitor lock across both steps doesn't help: _on_gc gives up when it can't take the lock, so a collection during the reset still ends up as a collection without a pause.
Covered by test_reset_collection_mid_reset_is_not_a_pause_without_collections.
| self._identity_refresh_lock = get_runtime_identity_refresh_lock() if self._identity_refresh_enabled else None | ||
| if self._identity_refresh_enabled: | ||
| on_runtime_identity_refresh(self._on_identity_refresh) |
There was a problem hiding this comment.
When the public RuntimeMetrics.enable() is called concurrently with a MicroVM /run refresh, the collectors are constructed and seed their CPU/GC baselines before the callback is registered here, without holding the refresh lock. If identity rotation lands in that window, the new worker misses the transition entirely; its first flush recollects the new runtime-id tag but reports deltas whose baselines came from the previous identity. Construct and register the worker state under the shared refresh lock, or verify the identity after registration and reseed when it changed.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 051c02d with the verify option: _collectors_runtime_id is read before RuntimeMetrics() seeds the baselines, so a refresh that lands before the callback is registered shows up as a mismatch, and the first flush() reseeds before anything is sent. Nothing is sent before that first flush, so checking there is equivalent to checking right after registration.
Covered by test_runtime_metrics_microvm_flush_resets_after_refresh_during_construction.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 051c02d2a1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| with self._identity_refresh_lock: | ||
| self._reset_collectors(runtime_id) |
There was a problem hiding this comment.
When this callback's collector reset fails transiently, the MicroVM coordinator keeps the callback pending. If a periodic flush subsequently repairs the state through the new runtime-ID mismatch path and advances _collectors_runtime_id, the next /run retry still invokes this unconditional reset, discarding valid CPU and GC activity accumulated since the successful repair. Check whether runtime_id already equals _collectors_runtime_id before resetting so pending callback retries are idempotent.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 7e091b1: _on_identity_refresh now returns early when runtime_id == self._collectors_runtime_id, so a retried callback after a flush already reset for that ID is a no-op and the post-repair interval is kept.
Covered by test_runtime_metrics_microvm_flush_resets_after_failed_refresh_reset.
quinna-h
left a comment
quinna-h
left a comment
There was a problem hiding this comment.
I'm also deferring review until the PR's base branch is main
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f6bc6fd247
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| # A fork rotates the runtime ID without an identity refresh, and the collectors already | ||
| # reseed in their own fork hooks, so the child's baselines belong to its new ID. Runs | ||
| # after _runtime_id's fork hook (registered at import), so this reads the child's ID. | ||
| self._collectors_runtime_id = get_runtime_id() |
There was a problem hiding this comment.
In a runtime-ID-enabled MicroVM child, if either collector’s earlier fork hook raises (for example, gc.get_stats() or native.process_metrics() fails), forksafe.run_hooks() logs the exception and continues to this hook. This assignment then marks the inherited baselines as belonging to the child ID, so flush() sees no mismatch and skips its recovery reset, potentially emitting parent-derived CPU/GC deltas under the child runtime ID. Preserve the mismatch unless the collector resets are known to have succeeded. ddtrace/internal/README.mdL263-L270
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 1a6ac93. _on_fork no longer assumes the collectors' own fork hooks succeeded. It calls _reset_collectors() itself, so the ID only moves to the child's once a reset has returned. If that reset raises too, the mismatch stays and the child's first flush resets instead.
Covered by test_runtime_metrics_microvm_fork_resets_collectors_for_child_id and test_runtime_metrics_microvm_failed_fork_reset_keeps_mismatch.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: bfba2dbfa1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if get_runtime_id() != runtime_id: | ||
| # The ID rotated while the tags were collected. Don't pair new-ID tags with old | ||
| # baselines; the next flush resets. | ||
| return | ||
| self._flush_unlocked(self._runtime_metrics) |
There was a problem hiding this comment.
When /run refresh begins after this second ID check but before _flush_unlocked() finishes iterating the collectors, _refresh_runtime_id() immediately publishes the new ID and only then invokes this worker's callback, which blocks on _identity_refresh_lock. The flush can therefore collect post-rotation CPU/GC activity and send it with the cached old runtime-id before allowing the reset. The fresh evidence is that the newly added check still precedes both collector iteration and sending, while identity rotation itself does not acquire this lock; synchronize the rotation boundary or collect and revalidate the complete batch before sending.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Not changing this one. The race is real: _refresh_runtime_id() doesn't take _identity_refresh_lock, so the ID can rotate while _flush_unlocked() iterates the collectors. But that batch covers the whole interval since the last flush, and only the last few microseconds of it can come after the rotation. The old runtime-id is the right tag for almost all of it.
Closing the gap fully means one of two things. The coordinator could take this lock, but runtime_metrics already imports from _runtime_id, so that would create an import cycle. Or the flush could collect, recheck the ID, and drop the batch on a mismatch, which throws away a full interval of correctly tagged data to avoid mislabeling a few microseconds. Neither seems worth it.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 86bb4e3246
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| from ddtrace.internal._runtime_id import on_runtime_identity_refresh | ||
|
|
||
| on_runtime_identity_refresh(tracer._refresh_runtime_identity) | ||
| core.on(WEB_REQUEST_STARTING_EVENT, maybe_refresh_identity) |
There was a problem hiding this comment.
When the first MicroVM /run request is handled through ASGI and carries distributed-tracing headers, this listener fires from asgi/middleware.py:279, after activate_distributed_headers() has already activated the incoming context at lines 268–271. The resulting tracer refresh then clears that active Context in _detach_stale_context() (tracer.py:472–475), so the request span created immediately afterward loses its upstream parent. Move the ASGI request-start dispatch ahead of distributed-header activation, matching the existing WSGI ordering and the tracer refresh invariant.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
The request span didn't actually lose its parent. _start_span runs activate_distributed_headers again from distributed_headers, and because the refresh had just detached the context, it activated the extracted parent a second time.
The ordering was still wrong, though. The headers were activated, detached, and activated again, so http.activate_distributed_headers and distributed_context.activated fired twice. Fixed in #19939 (174c28b): the ASGI dispatch now runs before header extraction and activation, the same as WSGI. #19820 adds an end-to-end test that sends distributed headers on /run and checks the parent is kept and the context is activated once.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 300990a307
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| # would otherwise mix pre- and post-refresh activity into one sample. Only register this | ||
| # for MicroVMs tagging runtime-id: everyone else keeps a single identity for their | ||
| # lifetime and pays nothing extra. | ||
| self._identity_refresh_enabled = in_aws_lambda_microvm() and config._runtime_metrics_runtime_id_enabled |
There was a problem hiding this comment.
In an AWS Lambda MicroVM with runtime metrics enabled but the runtime-ID tag left at its default disabled value, this condition prevents both callback registration and the ID-mismatch recovery path. Each restored instance therefore retains the snapshot's CPU, context-switch, and GC baselines, so its first interval includes and can duplicate pre-snapshot activity—the behavior this change is intended to prevent. Collector resets should follow the MicroVM identity transition regardless of whether that identity is emitted as a metric tag.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Out of scope here. This stack exists to make sure the runtime ID is refreshed on the MicroVM /run transition. The collector reset is gated on runtime-ID tagging because that's the only mode where pre-refresh baselines could go out under a runtime-id they don't belong to. With tagging off there's no runtime-id on the metrics, so nothing gets attributed to the wrong identity. Baselines carried over from the snapshot in that mode are a general snapshot/restore issue that predates this change, and would need their own fix.
| def reset(self) -> None: | ||
| for collector in self._collectors: | ||
| collector.reset() |
There was a problem hiding this comment.
If a later collector fails during this loop—for example, the native process-metrics read fails after the GC collector reset succeeds—the worker deliberately leaves its runtime ID mismatched and retries the entire loop on every flush. Each retry clears the already-repaired GC collector again, discarding valid post-refresh collections and pause data until the native collector recovers; track reset completion per collector or otherwise avoid rerunning successful resets.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Leaving this as is. While the ID is mismatched, flush() sends nothing, so the GC data a retry discards was never going to be reported anyway. And when the retry finally succeeds, both collectors reseed at the same moment, so their first post-refresh intervals line up. Tracking resets per collector would keep the GC data from the retry window, but GC's first interval would then be longer than the native collector's.
A native collector that keeps failing isn't new either. ValueCollector.collect() doesn't catch exceptions, so a persistent native.process_metrics() failure already breaks flush() without this change.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d81c45c3c7
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| # assume they succeeded: if this raises too, the mismatch stays and the child's first flush | ||
| # retries. Runs after _runtime_id's fork hook (registered at import), so this reads the | ||
| # child's ID. | ||
| self._reset_collectors(get_runtime_id()) |
There was a problem hiding this comment.
When runtime-ID metrics are enabled in a MicroVM process that forks, this after-child hook calls NativeProcessMetricCollector.reset(), whose native.process_metrics() implementation reads /proc/self/stat, /proc/self/status, and /proc/self/statm. The internal fork-safety guide explicitly prohibits I/O in after-child hooks because it can delay child readiness; leave the collector ID mismatched and perform this reseed during the first post-fork flush instead.
AGENTS.md reference: AGENTS.md:L139-L140
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Leaving this as-is. The I/O isn't new. On main, NativeProcessMetricCollector already registers _reset_state as a fork hook, and it reads the same /proc/self/* files through native.process_metrics(). _on_fork repeats that reset as a backstop, because forksafe logs and skips a hook that raises (earlier thread).
These are procfs reads, not network I/O, so they don't hold up the child the way a flush would. Moving the reset to the first flush would cost one interval of metrics in every forked child.
Replied with the help from Claude Code
| if self._identity_refresh_enabled: | ||
| on_runtime_identity_refresh(self._on_identity_refresh) | ||
| forksafe.register(self._on_fork) |
There was a problem hiding this comment.
If PeriodicThread.start() raises, such as from thread-resource exhaustion, RuntimeWorker.enable() exits before assigning _instance and never calls stop(), but these registrations retain the unstarted worker indefinitely. A later identity refresh or fork will invoke that dead worker, and each retry of enable() accumulates another callback and fork hook; register only after successful startup or unwind both registrations on the failure path.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Not changing this here. RuntimeMetrics() already registers the collectors' fork hooks in __init__, and the GC collector acquires the shared pause monitor there too. So a failed start() leaks all of that on main today, and these two registrations follow the same pattern. It only happens when the process can't start a thread.
If startup cleanup is worth doing, it should cover the collectors as well, in its own PR.
Replied with the help from Claude Code
When runtime-ID tagging is enabled, recollect platform tags during the periodic runtime-metrics flush instead of keeping the pre-refresh list, so an AWS Lambda MicroVM identity refresh is reflected on the next flush. Platform tags stay cached when runtime-ID tagging is disabled. A MicroVM /run transition also rotates the runtime ID mid-process without a fork, which left the CPU time, context-switch, wall-clock, and GC pause collectors mid-interval: their next sample would mix pre- and post-refresh activity, or (for a fresh process) reports the whole process lifetime as one delta. Reset those collectors' baselines on the existing runtime identity refresh callback, registered only for MicroVM processes with runtime-ID tagging enabled. Non-MicroVM processes and MicroVMs without runtime-ID tagging register nothing and are unaffected. This does not add another identity-refresh callback beyond the existing one, and does not change trace, writer, or telemetry-worker lifecycle behavior. This is the top of the identity refresh stack, so it also registers the /run listener next to the tracer refresh callback in ddtrace/trace/__init__.py. RuntimeWorker uses its own lock to keep collector resets atomic with flush(), and skips a flush whose ID rotated during tag collection. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Stacked PRs:
Description
After a MicroVM identity refresh, runtime metrics must use the new runtime identity and
must not mix a pre-refresh measurement interval with post-refresh process activity. This
PR covers both halves of that:
Platform tags. When runtime-ID tagging is enabled, runtime metrics recollect platform
tags during the periodic runtime-metrics flush instead of keeping the tag list built before
/run, so a MicroVM identity refresh is reflected on the next flush. When runtime-IDtagging is disabled, platform tags stay cached (no behavior change).
Collector state. A MicroVM
/runtransition rotates the runtime ID mid-process withouta fork. Left alone, the CPU time, context-switch, wall-clock, and GC pause/collections
collectors would report a delta spanning both the old and new identity on their next sample.
RuntimeWorkernow resets those collectors' baselines through the existingon_runtime_identity_refreshcallback, registered only when the process is a MicroVM andruntime-ID tagging is enabled; it's unregistered on
stop(). Non-MicroVM processes andMicroVMs without runtime-ID tagging register nothing and pay no extra cost.
The worker also records the runtime ID its collector baselines belong to, updated only after a
reset completes. If that ID differs from the current one,
flush()resets and sends nothing forthat interval, which covers a refresh callback that was missed (the worker was created mid-refresh)
or failed and is still pending. A reset that raises there is logged and retried on the next flush
instead of stopping the worker thread, and a retried refresh callback for an ID a flush already
reset for is a no-op. A fork hook moves that ID to the child's new runtime ID, since the collectors
already reseed on fork, so a forked child's first flush doesn't reset them again.
GCRuntimeMetricCollector.reset()reseeds the collections baseline before clearing the pausewindow, matching enable time: a collection between the two steps can show up as collections
without a pause, never as a pause without collections.
Refresh synchronization. The worker shares the fork-safe MicroVM refresh lock from #19939.
It holds that lock across collector reset and the full runtime-metrics flush, including tag
collection, sampling, and sending, so identity rotation cannot overlap an in-flight export.
The lock is not used by non-MicroVM or runtime-ID-disabled paths.
This does not add a second identity-refresh callback/guard, and does not touch trace,
writer, or telemetry-worker lifecycle code.
Reference
Testing
Added focused subprocess/unit coverage for:
RuntimeMetrics.reset())RuntimeWorker.stop()unregistering the refresh callback and fork hookflush()resetting (and sending nothing) after a failed refresh reset, and after a refresh thatlands between collector seeding and callback registration
resetting again once a flush has
GCRuntimeMetricCollector.reset()never surfacing as a pause withoutcollections
GCRuntimeMetricCollector/NativeProcessMetricCollector.reset()discarding pre-refreshpause/collections and CPU-time/context-switch/wall-clock state
The previous test that forced refreshes to interleave inside a flush was replaced because the
shared lock makes that interleaving impossible; the new regression test verifies the intended
serialization boundary directly.
Validation:
scripts/run-tests --venv 817352e -- -p no:randomly tests/runtime/test_runtime_metrics_api.py(Python 3.14): all tests added or changed here pass (21 passed, 2 xfailed). The 9 failures predate this PR:RuntimeMetrics.enable()no longer acceptsflush_interval(removed in 4.0), theDD_RUNTIME_METRICS_INTERVALassertions, a dogstatsdConnection refusedon stderr with no local agent, andtest_runtime_metrics_experimental_runtime_tag(fails in the test container; the runtime-id tag is present when checked outside it). Theruntimesuite isskip: trueintests/suitespec.yml, so CI does not run it.scripts/run-tests -s --venv 817352e -- -p no:randomly -k "microvm or refresh_identity or non_microvm" tests/runtime/test_runtime_metrics_api.py(11 passed)scripts/lint fmt -- ddtrace/internal/runtime/metric_collectors.py ddtrace/internal/runtime/runtime_metrics.py tests/runtime/test_runtime_metrics_api.py tests/tracer/runtime/test_metric_collectors.pyscripts/lint typing -- ddtrace/internal/runtime/metric_collectors.py ddtrace/internal/runtime/runtime_metrics.pyRisks
Limited to runtime-metrics platform-tag collection and the two metric collectors' interval
state. The shared lock adds synchronization only for MicroVMs with runtime-ID tagging; the
runtime-ID-disabled and non-MicroVM paths are unchanged. A flush now waits for an in-flight
MicroVM identity transition, and a transition waits for an in-flight metrics flush, preventing
partial identity attribution. When a refresh reset was missed or failed, the next flush resets
and sends nothing, so runtime metrics have a one-interval gap there (including a skipped final
flush if that happens at shutdown).