fix(telemetry): rebuild the worker on MicroVM identity refresh

[litianningdatadog]

Stacked PRs:

• chore(runtime): add explicit identity refresh #19778 — runtime identity foundation (merged)
• chore(serverless): publish web request-start event #19898 — generic WSGI/ASGI/Django request-start event (merged)
  • chore(runtime): refresh MicroVM identity #19939 — MicroVM /run listener and identity refresh
    • chore(tracer): rebuild the trace writer on MicroVM identity refresh #20952 — tracer: rebuild the trace writer (part 1 of fix(tracer): isolate MicroVM identity refresh #19820)
      • This PR fix(telemetry): rebuild the worker on MicroVM identity refresh #20953 — telemetry: rebuild the worker (part 1 of fix(telemetry): rebuild worker on identity refresh #19821)
        • fix(aws_lambda): refresh the MicroVM identity when a web request starts #20954 — registers the /run listener; runtime-metrics runtime-ID tag (part 1 of chore(runtime): refresh runtime metric identity and collector state #19822)
          • fix(runtime): reset runtime metric baselines on MicroVM identity refresh #20957 — runtime metrics: reset collector baselines (part 2 of chore(runtime): refresh runtime metric identity and collector state #19822)
        • chore(telemetry): discard the old worker and exporter subscription on MicroVM refresh #20956 — telemetry: discard the old worker and exporter subscription (part 2 of fix(telemetry): rebuild worker on identity refresh #19821)
      • fix(tracer): drop trace state from before a MicroVM identity refresh #20955 — tracer: drop trace state from before the refresh (part 2 of fix(tracer): isolate MicroVM identity refresh #19820)

Part 1 PRs (#20952, #20953, #20954) can ship first. Part 2 PRs (#20955, #20956, #20957) can ship later.

Description

This PR and #20956 split #19821. This PR rebuilds the telemetry worker for the new runtime ID after a MicroVM identity refresh. #20956 adds the worker access lock, the changes to how the trace exporter gets and follows the worker, and the discard path that builds on #19820's.

The native telemetry worker copies the runtime and session IDs when it's built, so after refresh_identity() it keeps reporting the snapshot's IDs until it's replaced.

• In a MicroVM, TelemetryWriter registers _refresh_runtime_identity(), and disable() unregisters it through the new remove_runtime_identity_refresh().

• On refresh, under the enable lock, the writer:

  • stops the old worker (_discard_worker());
  • re-schedules dependency reports (DependencyTracker.refresh());
  • rebuilds the worker with enable();
  • calls app_started() again if the old worker had started.

  The refresh re-enters enable(), and so does bootstrap configuration reporting, so the enable lock is an RLock. A failed stop leaves the old worker in place and is raised, and so is a failed rebuild. The writer records the runtime ID each worker was built with, and in a MicroVM periodic() rebuilds a worker still bound to an older one, so the next heartbeat retries a refresh that failed on /run. If the rebuild itself failed, the next telemetry call builds the worker.

• A rebuilt worker starts with empty native state, so in a MicroVM the writer replays its last-known state into every worker enable() builds:

  • the latest integration and product-activation state;
  • accepted configuration events, in order, with fresh sequence IDs (bounded at 5000, like libdatadog's store). The bootstrap configurations are recorded once, ahead of the first replay, so every worker gets them, and before any update made while no worker existed;
  • every dependency, re-reported even with SCA disabled.

  Outside a MicroVM the writer keeps none of this state and replays nothing, so worker rebuilds there (agentless switch, test token, payload directory) behave as before.

The refresh replaces the worker on the /run thread while other threads may be using it. This PR doesn't lock the data paths (#20956 does). Instead:

• Data paths read self._worker once, so a refresh between the check and the native call can't hand them None. A call that reaches the stopped worker is dropped by the native worker.
• enable() starts the worker (in forked children) under the enable lock, so a refresh can't replace the worker between building and starting it. Outside a MicroVM it also reports the bootstrap configurations there; in a MicroVM the replay carries them.
• _register_metric_context() doesn't cache a context registered on a worker that was replaced in the meantime. enable() clears the context cache and publishes the worker under the metric lock, so that check and the cache write can't straddle the swap.
• The replay iterates over snapshots of the state it replays, and runs before the worker is published. A caller that reads the new worker sends after the replay, so older replayed state can't overwrite its update.
• In a MicroVM, integration, product and configuration updates record their replay state and send under the enable lock, so a refresh can't replay an update between the two and have it sent again, with a newer sequence ID, after later updates. An update that finds no worker is delivered by the replay into the worker its enable() builds, not sent again.
• enable() reads the runtime ID once and builds the worker with it, so the worker's runtime and session IDs match the ID periodic() compares.
• In a MicroVM, dependency reporting takes the enable lock, so a refresh can't land between reading the worker and consuming the tracker's report, which would send the full report it schedules to the stopped worker.
• In a MicroVM, app_started() takes the enable lock, so a refresh can't land between start() and setting started, which would leave started set for a replacement that was never started.
• Endpoint replay iterates over a snapshot of the endpoint collection, so a route registered on another thread during the replay can't abort it.
• MetricRecorder._rebind() no longer clears the recorder's context. add() reads the worker and the context separately, so a rebind between the two reads (a refresh on another thread) could hand the old worker None and raise TypeError from span start or finish. The no-op and pending workers ignore the stale context.
• In a MicroVM, the telemetry log handler only takes ERROR records. While holding its handler lock, it forwards ERROR records to add_log(), which can wait for the enable lock. The refresh can log DEBUG while holding the enable lock, so a DEBUG record must never wait for the handler lock.
• In a MicroVM, a Rust panic (pyo3_runtime.PanicException, which derives from BaseException) is contained like an exception: in the refresh, which keeps the stale worker for the periodic retry; in that retry; in enable()'s worker build, which can run on an app thread after a refresh whose rebuild panicked; and in app_started()'s start(), which periodic() retries, because an exception that escapes periodic() ends the heartbeat thread. Other BaseExceptions still propagate, and outside a MicroVM nothing changes. If replaying state into a new worker fails, enable() stops that unpublished worker, which releases its native task, and returns False.
• In a MicroVM, disable() takes the enable lock, so it waits for a refresh that is building a worker and stops the worker that refresh publishes, instead of returning while _worker is None.

How a refresh runs, and what another thread sees meanwhile:

sequenceDiagram
    participant Run as /run thread
    participant W as TelemetryWriter
    participant Old as old worker
    participant New as new worker
    participant App as another thread

    Run->>W: _refresh_runtime_identity(new_id)
    Note over W: holds the enable lock until the end
    W->>Old: stop()  (flushes once under the old runtime ID)
    W->>W: _worker = None, unbind recorders, notify(None)
    W->>W: DependencyTracker.refresh()
    W->>New: _build_worker()  (new runtime and session IDs)
    W->>New: replay configurations, integrations, products
    W->>W: publish _worker = New (under the metric lock), bind recorders, notify(New)
    W->>New: start() via app_started(), if the old worker had started
    Note over W: releases the enable lock

    App->>W: product_activated() / add_*()
    alt read _worker before the discard
        App->>Old: call reaches the stopped worker and is dropped (known gap, closed in part 2)
    else _worker is None mid-refresh
        W-->>App: enable() waits on the enable lock, then sends to New after the replay
    else after the publish
        App->>New: sends normally
    end

Loading

Nothing registers maybe_refresh_identity() until #20954, so merging this PR alone doesn't change runtime behavior.

Known gaps, closed by #20956's worker access lock and exporter changes:

• Telemetry recorded while the refresh runs can reach the stopped worker and be dropped.
• An exporter built on another thread during a refresh (for example by _downgrade()) can get the old worker and subscribe after both worker-change notifications have fired. Its trace_api.* metrics then go to the stopped worker and are dropped, not reported under the old runtime ID. On /run the tracer and telemetry refreshes run one after the other on the same thread, so this needs an exporter rebuild on another thread at the same moment.

NoOpTelemetryWriter gets a no-op _refresh_runtime_identity().

Testing

tests/telemetry/test_writer.py:

• test_microvm_identity_refresh_rebuilds_worker_with_new_runtime_id: the rebuilt worker has the new runtime and session IDs and the replayed integration, product and configuration state.
• test_microvm_identity_refresh_emits_new_runtime_id: end to end against the test agent.
• test_microvm_identity_refresh_stops_legacy_worker_without_drop, test_microvm_identity_refresh_stop_failure_propagates: the old worker is stopped, and a failed stop keeps it and is raised.
• test_microvm_identity_refresh_retries_failed_rebuild, test_microvm_identity_refresh_failed_replacement_start_recovers_on_periodic, test_microvm_periodic_retries_failed_discard, test_periodic_skips_stale_worker_check_outside_microvm
• test_microvm_identity_refresh_reports_bootstrap_configurations_once, test_configuration_replay_store_is_microvm_only, test_integration_and_product_replay_store_is_microvm_only
• test_microvm_identity_refresh_replay_does_not_overwrite_concurrent_update: a product change made on another thread during the replay is the last state the new worker gets. It fails if the worker is published before the replay.
• test_microvm_error_handler_skips_records_below_error: in a MicroVM a DEBUG record never reaches the handler; outside one the handler is unchanged.
• test_microvm_identity_refresh_contains_native_panic_and_keeps_stale_worker: a Rust panic from the old worker's discard doesn't escape the refresh and keeps the stale worker for the retry; KeyboardInterrupt still propagates.
• test_microvm_periodic_contains_native_panic_from_stale_worker_rebuild: a Rust panic while the heartbeat retries the stale worker doesn't escape periodic(), and the next heartbeat rebuilds it.
• test_microvm_disable_waits_for_in_flight_rebuild_and_stops_replacement: disable() during a refresh rebuild waits for it and stops the worker the refresh publishes.
• test_enable_contains_native_panic_from_worker_build_only_in_microvm: a Rust panic building the worker is contained in a MicroVM and still raises outside one.
• test_microvm_product_update_racing_refresh_reaches_new_worker: a refresh that runs between an update's worker read and its replay record doesn't lose the update.
• test_dependency_report_holds_enable_lock_only_in_microvm: dependency collection holds the enable lock in a MicroVM and not outside one.
• test_start_retry_contains_native_panic_only_in_microvm: a panicking start() retried by periodic() doesn't escape it in a MicroVM, and still raises outside one.
• test_microvm_replay_failure_stops_unpublished_worker: a replacement whose replay fails with an exception or a Rust panic is stopped, and the refresh reports the failure.
• test_microvm_update_that_builds_the_worker_is_sent_once (4 cases): an update that builds the worker reaches it once, through the replay.
• test_microvm_configuration_update_racing_rebuild_is_sent_once: a configuration update made while a refresh builds the replacement reaches it once.
• test_microvm_bootstrap_configurations_precede_update_that_retries_failed_build: when an update retries a failed first build, the bootstrap configurations are still reported, and before the update.
• test_enable_builds_worker_with_the_runtime_id_it_records: the worker's runtime and session IDs equal the recorded ID even if the ID rotates during the build.
• test_bootstrap_reporting_reenters_enable_after_worker_is_cleared: outside a MicroVM, bootstrap reporting re-enters enable() instead of deadlocking when an agentless switch clears the worker.
• test_microvm_refresh_during_root_start_still_starts_replacement: a refresh that lands while app_started() is starting the old worker still starts the replacement.
• test_app_started_holds_enable_lock_only_in_microvm: app_started() holds the enable lock in a MicroVM and not outside one.
• test_endpoint_replay_survives_route_registered_during_replay: a route registered during the endpoint replay doesn't abort it.
• test_metric_recorder_rebind_keeps_context_paired_with_worker: a rebind between add()'s two reads doesn't hand the old worker None.
• test_metric_context_registered_on_replaced_worker_is_not_cached: a context registered on a worker that was replaced in the meantime isn't cached for the new one.
• test_non_microvm_writer_does_not_register_identity_refresh

tests/telemetry/test_dependency.py: test_refresh_preserves_and_rereports_dependency_metadata, test_refresh_rereports_dependencies_when_sca_disabled.

Local runs:

• telemetry (venv 70966a9, Python 3.14) on tests/telemetry/test_writer.py tests/telemetry/test_dependency.py tests/appsec/sca/test_telemetry.py: 216 passed.
• scripts/lint style on ddtrace/internal/telemetry/ and tests/telemetry/test_writer.py, and scripts/lint typing on ddtrace/internal/telemetry/writer.py: passed.

Risks

Low outside a MicroVM. The refresh callback is never registered there, and nothing is kept or replayed for worker rebuilds. What does apply everywhere:

• enable() now starts the worker (forked children) and reports the bootstrap configurations under the enable lock, and publishes the worker under the metric lock. Both happen once per worker build.
• The enable lock is an RLock, because bootstrap reporting calls enable() under it and an agentless switch can clear the worker in between. It's only taken while no worker exists.
• The worker is built with the runtime ID enable() reads, instead of reading it again for each field.
• The data paths read self._worker into a local instead of reading it twice.
• MetricRecorder._rebind() keeps the old context. Rebinds are rare, and the stale context only reaches workers that ignore it.
• DependencyTracker.collect_report() checks one more flag per heartbeat.

Inside a MicroVM:

• The old worker is stopped with stop(), which flushes its queued telemetry and app-closing under the previous runtime ID. fix(telemetry): rebuild worker on identity refresh #19821 does the same today, because the bundled TelemetryWorker has no non-flushing drop().
• TelemetryWorker.stop() blocks until the worker has drained, with no timeout in this code, so /run also waits for that send.
• The known gaps above.

Additional Notes

#20956's tip has exactly the code of #19821. #20956 holds:

• the worker access lock and the _without_lock twins, which replace this PR's single worker reads, metric-context cache check, metric-lock publish and replay snapshots. With the lock comes add_log() formatting before the lock;
• the stale-worker check in _get_shared_worker();
• _subscribe_worker_changes(callback, expected_worker, late_callback), publish-before-subscribe in NativeWriter._create_exporter(), and the matching _downgrade() change;
• parking a worker change that arrives during a stuck send;
• unsubscribing the exporter on teardown and on fix(tracer): isolate MicroVM identity refresh #19820's drop paths;
• the native drop() branch in _discard_worker().

This PR doesn't touch ddtrace/internal/writer/writer.py.

Reference

• RFC

🤖 Generated with Claude Code

[cit-pr-commenter-54b7da]

Codeowners resolved as

Resolved from the full PR diff against main using the target branch CODEOWNERS file.
CODEOWNERS team requests not listed below are not required by the current file set.

ddtrace/internal/_runtime_id.py                                         @DataDog/apm-core-python
ddtrace/internal/runtime/__init__.py                                    @DataDog/apm-sdk-capabilities
ddtrace/internal/telemetry/dependency.py                                @DataDog/apm-python
ddtrace/internal/telemetry/dependency_tracker.py                        @DataDog/apm-python
ddtrace/internal/telemetry/metrics.py                                   @DataDog/apm-python
ddtrace/internal/telemetry/noop_writer.py                               @DataDog/apm-python
ddtrace/internal/telemetry/writer.py                                    @DataDog/apm-python
releasenotes/notes/fix-telemetry-worker-identity-refresh-19fe1775be48b696.yaml  @DataDog/apm-python
tests/appsec/sca/test_telemetry.py                                      @DataDog/asm-python
tests/telemetry/test_dependency.py                                      @DataDog/apm-core-python @DataDog/apm-python
tests/telemetry/test_writer.py                                          @DataDog/apm-core-python @DataDog/apm-python

[cit-pr-commenter-54b7da]

Dependency direction analysis

⚠️ Existing dependency direction violations

There are 167 dependency direction violations that already exist on the base branch and have not been changed by this PR.

Show existing violations (showing 5 of 167 highest severity)

ddtrace.internal.tracemethods -×-> ddtrace.trace  (internal-core -> product:tracing, score=121)
ddtrace.internal.ci_visibility.api._base -×-> ddtrace.trace  (product:ci_visibility -> product:tracing, score=119)
ddtrace.llmobs._integrations.bedrock -×-> ddtrace.trace  (product:llmobs -> product:tracing, score=119)
ddtrace.llmobs._integrations.google_adk -×-> ddtrace.trace  (product:llmobs -> product:tracing, score=119)
ddtrace.llmobs._integrations.base -×-> ddtrace.trace  (product:llmobs -> product:tracing, score=119)

To see all violations, download the layers-base.json and layers-pr.json artifacts from this CI job and run:

uv run --script scripts/import-analysis/layers.py compare layers-base.json layers-pr.json

[cit-pr-commenter-54b7da]

Circular import analysis

⚠️ Existing circular imports

There are 1 circular imports that already exist on the base branch and have not been changed by this PR.

ddtrace.errortracking._handled_exceptions.bytecode_injector -> ddtrace.errortracking._handled_exceptions.callbacks -> ddtrace.errortracking._handled_exceptions.collector -> ddtrace.errortracking._handled_exceptions.bytecode_reporting -> ddtrace.errortracking._handled_exceptions.bytecode_injector

[datadog-prod-us1-5]

 

❌ Errors

Your PR has failed checks. Please review the issues below and take necessary action before merging.

🚦 2 Pipeline jobs failed

DataDog/apm-reliability/dd-trace-py | debugging/debugger — 🔄 Retry may pass, looks flaky

View more details · View in GitLab

DataDog/apm-reliability/dd-trace-py | build linux serverless: [arm64, cp310-cp310, v133965516-8f84bce-musllinux_1_2_aarch64, 1]

View more details · View in GitLab

ℹ️ Info

No other issues found (see more)

🧪 All tests passed
❄️ No new flaky tests detected

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: da7829a | Docs | View more details | Give us feedback!}

[pr-commenter]

Benchmarks

Benchmark execution time: 2026-10-15 22:07:57

Comparing candidate commit da7829a in PR branch tianning.li/3-4-2part-1-telemetry-identity-refresh with baseline commit 8b7678b in branch main.

📊 Benchmarking dashboard

Found 0 performance improvements and 3 performance regressions! Performance is the same for 348 metrics, 9 unstable metrics, 4 known flaky benchmarks, 4 flaky benchmarks without significant changes.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

• 🟩 = significantly better candidate vs. baseline
• 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

scenario:httppropagationextract-b3_single_headers

• 🟥 execution_time [+1.137µs; +1.212µs] or [+16.605%; +17.704%]

scenario:otelspan-start

• 🟥 execution_time [+2.531ms; +3.725ms] or [+9.743%; +14.338%]

scenario:recursivecomputation-shallow

• 🟥 execution_time [+58.228µs; +61.006µs] or [+8.268%; +8.662%]

Unstable benchmarks

These benchmarks have a confidence interval too wide to call a change; treat them as noise rather than signal.

scenario:coreapiscenario-context_with_data_listeners

• unstable execution_time [-745.259ns; +726.383ns] or [-7.000%; +6.823%]

scenario:coreapiscenario-core_dispatch_1_listener

• unstable execution_time [-59.742ns; +19.195ns] or [-8.545%; +2.746%]

scenario:coreapiscenario-core_dispatch_50_listeners

• unstable execution_time [-2114.514ns; +1712.673ns] or [-10.546%; +8.542%]

scenario:coreapiscenario-core_dispatch_exception_listeners

• unstable execution_time [-2189.401ns; +1801.944ns] or [-11.288%; +9.290%]

scenario:coreapiscenario-core_dispatch_listeners

• unstable execution_time [-466.025ns; +295.121ns] or [-10.725%; +6.792%]

scenario:coreapiscenario-core_dispatch_no_args_listeners

• unstable execution_time [-273.612ns; +182.677ns] or [-10.069%; +6.722%]

scenario:coreapiscenario-core_dispatch_with_results_1_listener

• unstable execution_time [-120.725ns; +75.237ns] or [-8.402%; +5.236%]

scenario:coreapiscenario-core_dispatch_with_results_50_listeners

• unstable execution_time [-5844.763ns; +3896.001ns] or [-11.504%; +7.668%]

scenario:coreapiscenario-core_dispatch_with_results_listeners

• unstable execution_time [-1116.264ns; +840.043ns] or [-10.526%; +7.921%]

Known flaky benchmarks

These benchmarks are marked as flaky and will not trigger a failure. Modify FLAKY_BENCHMARKS_REGEX to control which benchmarks are marked as flaky.

scenario:httppropagationinject-ids_only

• 🟥 execution_time [+2.219µs; +2.337µs] or [+16.219%; +17.085%]

scenario:span-start

• 🟥 execution_time [+0.915ms; +1.195ms] or [+7.891%; +10.308%]

scenario:telemetryaddmetric-1-count-metric-1-times

• 🟥 execution_time [+158.040ns; +193.597ns] or [+7.930%; +9.714%]

scenario:tracer-small

• 🟥 execution_time [+45.235µs; +46.753µs] or [+18.259%; +18.872%]

Known flaky benchmarks without significant changes:

• scenario:errortrackingflasksqli-baseline
• scenario:flasksimple-iast-get
• scenario:sethttpmeta-all-enabled
• scenario:telemetryaddmetric-record-100-metrics

[Copilot]

🟡 Changes recommended

Unresolved concurrency issues can deadlock logging, lose recorder metrics, and overwrite newer telemetry state during replay.

3 open findings

• Telemetry and worker locks can deadlock MicroVM logging · New
• Worker replay can overwrite concurrent state updates · New
• Metric recording can target a closing worker during refresh · New

What changed in this PR

Rebuilds telemetry workers after MicroVM identity refresh, extending the runtime-identity work to telemetry and shared trace-exporter metrics.

Changes:

• Recreates workers and replays configuration, integration, product, and dependency state.
• Synchronizes MicroVM worker access and updates exporter subscriptions.
• Adds refresh, retry, replay, and concurrency coverage.

File	Description
tests/​tracer/​test_writer.py	Tests exporter updates and concurrent sends.
tests/​telemetry/​test_writer.py	Tests worker refresh, recovery, replay, and locking.
tests/​telemetry/​test_dependency.py	Tests dependency re-reporting after refresh.
tests/​appsec/​sca/​test_telemetry.py	Updates the writer fixture for locking.
releasenotes/​notes/​fix-telemetry-worker-identity-refresh-19fe1775be48b696.yaml	Documents the telemetry identity fix.
ddtrace/​internal/​writer/​writer.py	Tracks worker replacements and manages subscriptions.
ddtrace/​internal/​telemetry/​writer.py	Implements refresh, state replay, and synchronization.
ddtrace/​internal/​telemetry/​noop_writer.py	Adds compatible no-op lifecycle methods.
ddtrace/​internal/​telemetry/​dependency.py	Resets dependency reporting status.
ddtrace/​internal/​telemetry/​dependency_tracker.py	Schedules full dependency re-reporting.
ddtrace/​internal/​runtime/​__init__.py	Exports callback removal.
ddtrace/​internal/​_runtime_id.py	Implements refresh callback removal.

🧠 Review effort: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[litianningdatadog]

The rebind half is fixed: _rebind() no longer clears the context, and _PendingWorker sets the context before the worker, so add() can't pair a worker with None. Covered by test_metric_recorder_rebind_keeps_context_paired_with_worker.

Points recorded while the old worker's stop() drains can still reach that worker. Making MetricRecorder.add() wait for the refresh would add synchronization to the span hot path for every process, not just MicroVMs, so recorders stay lock-free. The loss is limited to the refresh window, and the PR description lists the stop() flush under the previous runtime ID as a known risk.

[litianningdatadog]

the rebind half is fixed; the rest is explained as a bounded, disclosed loss.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-15T21:51:23.572477Z	da7829a	New commits
🔒 Security Review	✅ Completed 2026-10-15T21:48:33.213788Z	da7829a	New commits

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[datadog-prod-us1-5]

More details

Static review supports the refresh failure recovery and state replay ordering. The documented refresh concurrency gaps remain deferred to #20956.

Was this helpful? React 👍 or 👎

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit c00bdc9}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c00bdc9175

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Contain native panics during identity refresh

In a MicroVM, this native stop() call runs synchronously from the /run identity-refresh callback. A Rust panic exposed by PyO3 is a pyo3_runtime.PanicException derived directly from BaseException, while _notify_runtime_identity_refresh_callbacks() catches only Exception; consequently, a panic while stopping the restore-sensitive worker escapes into and aborts the request. The tracer's equivalent refresh path explicitly recognizes and contains this panic type, and telemetry should do the same while preserving the stale worker for the periodic retry and still allowing unrelated BaseException types to propagate.

Useful? React with 👍 / 👎.

[litianningdatadog]

Valid. Fixed in f28baa3. In a MicroVM, a Rust panic (pyo3_runtime.PanicException) is now contained like an exception in _refresh_runtime_identity(), which keeps the stale worker in place for the periodic retry, and in that retry (_rebuild_stale_worker_without_lock()). Other BaseExceptions still propagate. The same applies to enable()'s worker build, which can run on an app thread after a refresh whose rebuild panicked. The refresh logs the panic at WARNING, below the MicroVM log handler's ERROR level, so logging it can't wait on the enable lock it holds. Covered by test_microvm_identity_refresh_contains_native_panic_and_keeps_stale_worker, test_microvm_periodic_contains_native_panic_from_stale_worker_rebuild and test_enable_contains_native_panic_from_worker_build_only_in_microvm.

[chatgpt-codex-connector]

Recheck disablement before publishing the replacement

If disable() or app_shutdown() runs while a MicroVM refresh is inside _build_worker(), it can set _enabled to false, observe _worker as None, unregister the callback, and return; this path then publishes the newly built worker without checking _enabled again. The disabled writer is left owning a spawned native worker after its collector and shutdown path have already been torn down. Serialize disable() with this lock or recheck _enabled immediately before publishing and dispose of the unpublished worker.

Useful? React with 👍 / 👎.

[litianningdatadog]

Valid. Fixed in f28baa3. In a MicroVM, disable() now takes the enable lock, so it waits for a refresh that is building a worker and then stops the worker that refresh published. Outside a MicroVM, disable() is unchanged and lock-free. Covered by test_microvm_disable_waits_for_in_flight_rebuild_and_stops_replacement.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5bb98191c9

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Serialize replayable updates with the worker swap

When an integration/configuration/product update calls enable() just before a MicroVM refresh acquires _enable_lock, it can retain the old worker and then pause before recording its replay state. The refresh can stop that worker, snapshot the old state, and publish the replacement; when the update resumes, it stores the new state but sends the event only to the stopped worker, so the replacement never receives it. The update path must either participate in the refresh lock or recheck the worker generation before sending/replaying.

Useful? React with 👍 / 👎.

[litianningdatadog]

Valid. Fixed in 6140e16. Integration, product and configuration updates now record their replay state before calling enable() and reading the worker. A concurrent refresh then either replays the update, or has already published the worker the update reads and sends to. While a refresh is between discard and publish, _worker is None, so enable() waits on the enable lock. The worst case is a duplicate event, which is harmless for these. Covered by test_microvm_product_update_racing_refresh_reaches_new_worker.

[chatgpt-codex-connector]

Keep dependency refresh state paired with its worker

If the dependency collector reads the old worker here immediately before identity refresh, it can resume after _dependency_tracker.refresh(), consume the forced full report, mark every dependency sent, and enqueue that report on the already-stopped worker. The refresh thread's subsequent report against the replacement then sees nothing, leaving the new worker permanently without the dependency snapshot. Serialize collection with the refresh or verify the worker generation before consuming tracker state.

Useful? React with 👍 / 👎.

[litianningdatadog]

Valid. Fixed in 6140e16. In a MicroVM, _report_dependencies() now takes the enable lock, which the refresh holds throughout, so a refresh can't land between reading the worker and consuming the tracker's report. Outside a MicroVM it stays lock-free. Covered by test_dependency_report_holds_enable_lock_only_in_microvm.

[chatgpt-codex-connector]

Prevent replay from overflowing the native mailbox

When a MicroVM has accumulated enough replay state to fill the native worker's mailbox, this tight loop can enqueue up to 5,000 configurations before integrations and products are replayed. The binding implements add_configuration() with try_send_msg() and explicitly drops data when the bounded channel is full (src/native/telemetry.rs lines 69-75 and 319-340), yet every call still returns successfully. The refresh therefore publishes and starts a worker with silently incomplete state; replay needs a capacity-safe or acknowledged path rather than the best-effort live-update API.

Useful? React with 👍 / 👎.

[litianningdatadog]

The native add_* calls are best-effort by design (try_send_msg(), drop_on_err), and the mailbox holds 5000 messages. A MicroVM process with ddtrace.auto records about 260 replay messages at startup: 250 configuration events, 3 integrations and 7 products. That's about 5% of the capacity, and the replay store's 5000 bound matches libdatadog's own. A capacity-safe, acknowledged replay would need a native API change, so it's out of scope here.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f28baa3021

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Contain panics during the periodic start retry

If the replacement worker's start() raises a PyO3 PanicException here, _refresh_runtime_identity() contains the first panic and leaves started false as intended, but the promised retry occurs through periodic() calling app_started(), whose handler catches only Exception. A repeated native panic therefore escapes the periodic target and terminates its thread instead of preserving future retries; recognize is_panic_exception() on that retry path while still allowing unrelated BaseException types to propagate.

Useful? React with 👍 / 👎.

[litianningdatadog]

Valid. Fixed in 6140e16. In a MicroVM, app_started() now contains a Rust panic from start() like an exception, so the retry through periodic() can't end the heartbeat thread. Other BaseExceptions still propagate, and outside a MicroVM it's unchanged. Covered by test_start_retry_contains_native_panic_only_in_microvm.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6140e16025

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Do not resend a configuration replayed by enable

When an identity refresh has cleared _worker, a concurrent configuration update appends its value here and then blocks in enable(). If _replay_worker_state() snapshots that append, it sends the value to the replacement worker; after enable() returns, add_configuration() sends the same value again with another sequence ID. The current append-before-enable ordering fixes the earlier lost-update race but creates a duplicate configuration change; add_configurations() has the same problem. Serialize replay with the send or record whether the triggering update was already replayed.

Useful? React with 👍 / 👎.

[litianningdatadog]

Valid. Fixed in db97dfc. In a MicroVM, integration, product and configuration updates now record their replay state and send it while holding the enable lock, so a refresh can't replay an update between the two. An update that finds no worker is delivered by the replay into the worker its enable() builds, and isn't sent again. That also closes the worse case, where the late send carried a newer sequence ID than an update made after it. Covered by test_microvm_update_that_builds_the_worker_is_sent_once and test_microvm_configuration_update_racing_rebuild_is_sent_once.

[chatgpt-codex-connector]

Track bootstrap configuration replay explicitly

If the initial MicroVM worker build fails and a later add_configuration() call triggers the successful retry, that call has already made _configurations nonempty before entering enable(). This condition therefore skips get_python_config_vars() even though none of the bootstrap configurations were ever stored or sent, leaving the eventual app-started payload permanently incomplete. Use an explicit bootstrap-recorded flag rather than treating any saved configuration as proof that bootstrap reporting happened.

Useful? React with 👍 / 👎.

[litianningdatadog]

Valid. Fixed in db97dfc. In a MicroVM, the first worker build now records the bootstrap configurations at the front of the replay store, behind an explicit _bootstrap_configurations_recorded flag, and the replay sends them. An update that retries a failed first build therefore still gets them reported, and ahead of itself. Covered by test_microvm_bootstrap_configurations_precede_update_that_retries_failed_build.

[datadog-prod-us1-5]

A lazy telemetry rebuild can deadlock with an agentless switch outside MicroVMs because bootstrap reporting re-enters a non-reentrant enable lock after the switch clears the worker. This can indefinitely block the telemetry caller and LLMObs.enable().

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit 6140e16}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3d91ccf245

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Retry the dependency snapshot after a send failure

When an identity refresh is replaying the full dependency set and any add_dependency() call raises, collect_report() has already marked every dependency as sent and cleared _report_all. The refresh callback failure is then isolated, and a later periodic() starts the replacement without retrying the failed or remaining dependencies, leaving that worker with an incomplete dependency snapshot. Keep the refresh report pending until every native call succeeds, or restore the tracker's refresh state when a send fails.

Useful? React with 👍 / 👎.

[litianningdatadog]

The native add_dependency() doesn't raise when a send fails. It goes through drop_on_err(), which logs and drops the datum when the channel is full or the worker has stopped, and returns normally (src/native/telemetry.rs). Only a fork-runtime error or a Rust panic could raise there. Marking dependencies as sent before the native call is also how every report already works on main, since a call that returns can still be dropped natively. Keeping the refresh report pending until delivery is confirmed would need an acknowledged native path, so I'm leaving this as is.

[chatgpt-codex-connector]

Retry endpoint replay when the refresh aborts

During this refresh, enable() publishes the replacement before _report_endpoints() finishes. If that replay raises—for example, request-time route registration mutates the live endpoint_collection.endpoints set while it is being iterated—the callback failure is isolated but the current-runtime worker remains installed. Subsequent enable() and periodic() calls therefore never replay the remaining pre-refresh endpoints, leaving the replacement's endpoint snapshot permanently incomplete; replay from a stable snapshot and retain a pending/retry state until it succeeds.

Useful? React with 👍 / 👎.

[litianningdatadog]

Fixed in da7829a. The realistic failure here was iterating the live endpoint_collection.endpoints set while another thread registers a route, which raises RuntimeError: Set changed size during iteration. _report_endpoints() now replays from a list(...) snapshot, so that can't happen. I didn't add a pending/retry state: once the set can't change, add_endpoint() can only raise for reasons a retry would hit again. A full channel or a stopped worker is dropped by drop_on_err() without raising. What's left is argument conversion on the same endpoint object, or a post-fork runtime error that every native call on that worker would hit. Covered by test_endpoint_replay_survives_route_registered_during_replay.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: db97dfc85f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Serialize root worker start with identity refresh

When a root app_started() retry overlaps the /run refresh, the old worker's start() can return successfully and the thread can pause before setting self.started; the refresh then observes started == False, stops that worker, publishes a replacement without starting it, and the old thread resumes and marks the replacement as started. Subsequent heartbeats skip app_started(), so the replacement never emits lifecycle events or starts its heartbeat schedule. Hold _enable_lock across the root start and flag update, or verify that the worker generation is unchanged before setting started.

Useful? React with 👍 / 👎.

[litianningdatadog]

Valid. Fixed in da7829a. In a MicroVM, app_started() now holds the enable lock across start() and setting started, through an _app_started_without_lock() twin, so a refresh can't land in between. The refresh already calls app_started() under that lock, which is reentrant. Outside a MicroVM it's unchanged and lock-free. Covered by test_microvm_refresh_during_root_start_still_starts_replacement and test_app_started_holds_enable_lock_only_in_microvm.

[chatgpt-codex-connector]

Make worker subscription atomic with refresh

When a concurrent Tracer.configure() rebuild reaches _get_shared_worker() while /run is swapping workers, it can capture the old worker and pause; the refresh then sends both the None and replacement notifications, after which the new trace exporter installs the stopped handle and subscribes too late to receive either notification. That exporter permanently drops its trace_api.* health metrics because there may be no further worker change. Return-and-subscribe to the current worker atomically with the refresh, or recheck the generation after registering the subscriber.

Useful? React with 👍 / 👎.

[litianningdatadog]

This is one of the known gaps, #20956 covers that.

[datadog-prod-us1-5]

More details

No actionable regression was identified beyond the telemetry-drop and exporter-subscription races explicitly deferred to part 2. Missing native extensions prevented runtime validation.

Was this helpful? React 👍 or 👎

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit db97dfc}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: da7829aacb

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Retain unpublished workers when cleanup fails

When replay fails because ensure_runtime_after_fork() raises or panics, this cleanup calls stop(), which enters the same guard and can fail before consuming the registration handle. The new catch then continues to return False and drops the only Python reference; src/native/telemetry.rs documents that dropping this handle leaks the worker until runtime shutdown, so repeated rebuild attempts can accumulate registered workers. This cleanup-failure branch is fresh evidence beyond the earlier replay-cleanup finding; retain the worker for a later stop attempt or provide a disposal path that consumes its registration without the failing guard.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Keep post-build setup serialized with disable

When a MicroVM data path rebuilds after a failed refresh, disable() can acquire _enable_lock immediately after this block releases it, stop the newly published worker, clear the endpoint callback, and set _deps_collector to None; the rebuilding thread then resumes at the dedented setup and re-registers the callback and starts a new periodic collector on the permanently disabled writer. Keep endpoint subscription and collector creation inside the disable serialization, or recheck _enabled before performing that setup.

Useful? React with 👍 / 👎.