Repository navigation
fix(tracer): drop trace state from before a MicroVM identity refresh - #20955
Draft
litianningdatadog wants to merge 1 commit into
Draft
litianningdatadog wants to merge 1 commit into
litianningdatadog wants to merge 1 commit into
Conversation
This was referenced Oct 13, 2026
Codeowners resolved asResolved from the full PR diff against |
Circular import analysis
|
Dependency direction analysis
|
BenchmarksBenchmark execution time: 2026-10-15 21:39:21 Comparing candidate commit fdf64c5 in PR branch Found 0 performance improvements and 3 performance regressions! Performance is the same for 366 metrics, 10 unstable metrics, 3 known flaky benchmarks, 5 flaky benchmarks without significant changes.
|
litianningdatadog
force-pushed
the
tianning.li/3-3-2part-1-tracer-identity-refresh
branch
from
October 13, 2026 19:10
db329bc to
adebde7
Compare
litianningdatadog
force-pushed
the
tianning.li/3-3-2part-2-discard-stale-traces
branch
from
October 13, 2026 19:10
f915d89 to
883ac9f
Compare
This comment has been minimized.
This comment has been minimized.
litianningdatadog
force-pushed
the
tianning.li/3-3-2part-1-tracer-identity-refresh
branch
from
October 13, 2026 19:25
adebde7 to
c5961b2
Compare
litianningdatadog
force-pushed
the
tianning.li/3-3-2part-2-discard-stale-traces
branch
from
October 13, 2026 19:25
883ac9f to
78840aa
Compare
litianningdatadog
force-pushed
the
tianning.li/3-3-2part-1-tracer-identity-refresh
branch
3 times, most recently
from
October 14, 2026 17:35
fc8abd4 to
fc73ae2
Compare
litianningdatadog
force-pushed
the
tianning.li/3-3-2part-2-discard-stale-traces
branch
2 times, most recently
from
October 14, 2026 20:27
3f03564 to
436ee86
Compare
litianningdatadog
force-pushed
the
tianning.li/3-3-2part-1-tracer-identity-refresh
branch
from
October 14, 2026 20:27
fc73ae2 to
3d14424
Compare
litianningdatadog
force-pushed
the
tianning.li/3-3-2part-2-discard-stale-traces
branch
from
October 15, 2026 15:10
436ee86 to
d11fb3b
Compare
litianningdatadog
force-pushed
the
tianning.li/3-3-2part-1-tracer-identity-refresh
branch
from
October 15, 2026 15:10
3d14424 to
b3d0d84
Compare
litianningdatadog
force-pushed
the
tianning.li/3-3-2part-2-discard-stale-traces
branch
from
October 15, 2026 15:58
d11fb3b to
82aa5f4
Compare
litianningdatadog
force-pushed
the
tianning.li/3-3-2part-1-tracer-identity-refresh
branch
from
October 15, 2026 15:58
b3d0d84 to
4467f33
Compare
litianningdatadog
force-pushed
the
tianning.li/3-3-2part-2-discard-stale-traces
branch
from
October 15, 2026 18:55
82aa5f4 to
8c40c3d
Compare
litianningdatadog
force-pushed
the
tianning.li/3-3-2part-1-tracer-identity-refresh
branch
from
October 15, 2026 18:55
4467f33 to
1b08ad8
Compare
gh-worker-dd-mergequeue-cf854d Bot
pushed a commit
that referenced
this pull request
Oct 15, 2026
…20952) Stacked PRs: - #19778 — runtime identity foundation (merged) - #19898 — generic WSGI/ASGI/Django request-start event (merged) - #19939 — MicroVM `/run` listener and identity refresh (merged) - This PR #20952 — tracer: rebuild the trace writer (part 1 of #19820) - #20953 — telemetry: rebuild the worker (part 1 of #19821) - #20954 — registers the `/run` listener; runtime-metrics runtime-ID tag (part 1 of #19822) - #20957 — runtime metrics: reset collector baselines (part 2 of #19822) - #20956 — telemetry: discard the old worker and exporter subscription (part 2 of #19821) - #20955 — tracer: drop trace state from before the refresh (part 2 of #19820) Part 1 PRs (#20952, #20953, #20954) can ship first. Part 2 PRs (#20955, #20956, #20957) can ship later. **Release coupling:** #20952 is intended to ship together with #20954. This PR registers the tracer as an identity-refresh subscriber; #20954 registers `maybe_refresh_identity()` on `WEB_REQUEST_STARTING_EVENT`, which is what makes the `/run` request trigger the refresh in production. Do not release #20952 without #20954. ## Description This PR and #20955 split #19820, so the runtime-ID fix can ship before the discard logic. This PR makes the tracer report the new runtime ID after a MicroVM identity refresh. #20955 drops trace state from before the refresh. The native trace exporter copies the runtime ID when it's built (`set_runtime_id(get_runtime_id())` in `_build_base_exporter_builder()`), so after `refresh_identity()` the tracer keeps reporting the snapshot's runtime ID until its writer is rebuilt. The root span's `runtime-id` tag already reads `get_runtime_id()` for each span and needs no change. - `ddtrace/trace/__init__.py` registers `tracer._refresh_runtime_identity` with `on_runtime_identity_refresh()` inside a MicroVM check. - `Tracer._refresh_runtime_identity()` rebuilds the writer the way the post-fork path does (`_recreate(reset_buffer=False, flush_writer=False)`), then stores the tracer metadata again, in a `finally`, so a failed rebuild still records the new runtime ID. A failed rebuild is retried on the next span start. A Rust panic from the rebuild is logged instead of re-raised, because the refresh dispatcher only contains `Exception` and the panic would otherwise abort the `/run` request. The span-start retry contains its own failures in a MicroVM: an `Exception` or Rust panic from the rebuild is logged (rate-limited) and the retry stays pending, so it doesn't fail the app's span start. `KeyboardInterrupt` and `SystemExit` still propagate, and outside a MicroVM a failed post-fork rebuild still raises, as on `main`. - The refresh and `shutdown()` serialize on `_post_fork_lock`. `shutdown()` sets `_shutting_down` under that lock, so it waits for a refresh in flight and stops the writer that refresh installs, and a refresh that starts after shutdown builds nothing. The lock is now reentrant, because the exit-signal handler can run `shutdown()` on the thread that is refreshing. If that handler returns, the refresh sees `_shutting_down` after `_recreate()` and stops the writer it just installed, shutting down the exporter of a `NativeWriter` that never started. - In a MicroVM, `shutdown(timeout)` waits at most `timeout` (clamped at 0) for `_post_fork_lock`, then sets `_shutting_down` either way. Without the bound, an exit signal could deadlock: a refresh on another thread holds `_post_fork_lock` while waiting for the old writer's `_writer_lock`, which the thread running the signal handler holds. A refresh still running after the wait stops the writer it installs, through the same `_shutting_down` check. `shutdown()` without a timeout still waits as before. The span-start retry of a failed refresh (`_ensure_post_fork_writer()`) does the same in a MicroVM: it builds nothing once `_shutting_down` is set, and stops a writer it installs after that wait. Outside a MicroVM the post-fork rebuild is unchanged. The rebuild stops the old writer the normal way, so the old writer still flushes what it buffered. A span finished on another thread after that stop would restart the old writer through the lazy start in `_write_with_client()`, leaving a writer that nothing stops. So the refresh first closes the old writer to new spans, using #19820's MicroVM-only write gate: `_writer_lock` and `_accepting_writes` on `HTTPWriter` and `NativeWriter`, checked in `write()`. `TraceWriter._stop_accepting_writes()` closes the gate without dropping anything. #20955 replaces it with `drop_buffered_traces()`. How a refresh runs, and what a span finished on another thread sees meanwhile: ```mermaid sequenceDiagram participant Run as /run thread participant T as Tracer participant Old as old writer participant New as new writer participant App as span-finishing thread Run->>T: _refresh_runtime_identity(new_id) Note over T: holds _post_fork_lock (reentrant) until the end opt shutdown already started T-->>Run: return, build nothing end T->>Old: _stop_accepting_writes() (under the old writer's _writer_lock) T->>Old: _recreate(reset_buffer=False, flush_writer=False) stops it, flushing what it buffered T->>New: built by recreate() with the new runtime ID T->>T: _store_metadata() in a finally, so the new runtime ID is recorded even if the rebuild failed opt shutdown() ran during the rebuild T->>New: stop the writer it just installed end Note over T: releases _post_fork_lock App->>T: span finishes, aggregator writes the trace alt before the old writer is closed T->>Old: written, and flushed when the old writer stops else after closing, before the swap T->>Old: write() returns without writing, so the closed writer is not restarted else after the swap T->>New: written under the new runtime ID end ``` Nothing registers `maybe_refresh_identity()` until #20954, so merging this PR alone doesn't change runtime behavior. Smaller changes: - `NativeWriter.write()` takes the write gate through the sync-mode flush, as `HTTPWriter.write()` does. Otherwise the refresh could close the writer between the buffering and the flush, and `recreate()` would shut down the exporter that flush then sends through. #19820 keeps the flush outside the gate and instead rechecks `_discarded_by_refresh` before each send; those rechecks belong to #20955. ## Testing `tests/tracer/runtime/test_runtime_id.py`: - `test_tracer_microvm_identity_refresh_recreates_exporter_without_fork_side_effects`: the refresh rebuilds the writer with the post-fork arguments and stores the metadata, without fork side effects. - `test_tracer_microvm_identity_refresh_closes_and_flushes_old_writer`: the old writer sends its buffer once and shuts its exporter down. A later write neither buffers nor restarts it. - `test_tracer_microvm_identity_refresh_after_shutdown_does_not_recreate_writer` - `test_tracer_microvm_identity_refresh_keeps_pending_after_recreate_failure`: a failed rebuild keeps the pending post-fork rebuild and still stores the metadata. - `test_tracer_microvm_identity_refresh_schedules_rebuild_after_recreate_failure`: a failed rebuild is left pending for the next span. `RuntimeError` and `KeyboardInterrupt` propagate; a Rust panic doesn't. - `test_tracer_identity_refresh_contains_native_panic_from_writer_rebuild`: a panic from the rebuild doesn't escape `refresh_identity()`, and other refresh subscribers still run. - `test_tracer_microvm_identity_refresh_retries_failed_writer_recreation_on_next_span`: after a failed rebuild, the next span rebuilds the writer, and the stored metadata carries the refreshed runtime ID. - `test_tracer_shutdown_waits_for_in_flight_identity_refresh`: shutdown during a refresh stops the writer the refresh installs. - `test_tracer_identity_refresh_during_shutdown_does_not_recreate_writer`: a refresh that starts while shutdown runs builds no writer. - `test_tracer_shutdown_on_refreshing_thread_stops_replacement_writer`: shutdown run on the refreshing thread (the exit-signal case) doesn't deadlock, and the replacement writer is stopped, or has its exporter shut down if it never started. - `test_tracer_signal_shutdown_does_not_deadlock_with_refresh_waiting_on_writer_lock`: shutdown run by a thread that holds the writer lock a refresh is waiting on (the cross-thread exit-signal case) returns after its timeout, and the refresh then stops the writer it installed. - `test_tracer_microvm_shutdown_accepts_negative_timeout` - `test_tracer_timed_out_shutdown_stops_writer_from_retried_refresh_rebuild`: a `shutdown(timeout=0)` that finishes while the span-start retry of a failed refresh is rebuilding still gets the rebuilt writer stopped. It fails without the fix. - `test_tracer_post_fork_rebuild_ignores_shutdown_flag_outside_microvm`: outside a MicroVM the post-fork rebuild ignores `_shutting_down`, as before. - `test_tracer_microvm_failed_rebuild_retry_is_contained_in_start_span`: in a MicroVM a span-start retry that fails again with a `RuntimeError` or Rust panic doesn't raise and stays pending, `KeyboardInterrupt` still propagates, and a later span start recovers. It fails without the fix. - `test_tracer_post_fork_rebuild_failure_still_raises_outside_microvm`: outside a MicroVM a failed post-fork rebuild still raises from the span start, as before. - `test_identity_refresh_hook_notifies_global_tracer`, `test_deferred_global_tracer_registers_identity_refresh_callback`, `test_deferred_global_tracer_does_not_register_identity_refresh_callback_outside_microvm`: the global tracer registers only in a MicroVM, including with deferred initialization. `tests/tracer/test_writer.py`: - `test_native_writer_closed_microvm_writer_rejects_writes_without_restarting`, `test_http_writer_closed_microvm_writer_rejects_writes`: a closed writer drops later spans and isn't restarted by them. Spans buffered before the close stay for the final flush. - `test_native_writer_microvm_close_waits_for_in_flight_sync_flush`: closing a sync-mode writer waits for a flush that is sending, and nothing is flushed through the closed writer. - `test_native_writer_stop_accepting_writes_is_noop_outside_microvm`, `test_log_writer_stop_accepting_writes_is_noop` `tests/tracer/test_tracer.py`: - `test_tracer_shutdown_waits_for_in_flight_post_fork_writer_recreation`: shutdown waits for a post-fork writer rebuild on another thread and stops the rebuilt writer. - `test_tracer_shutdown_timeout_does_not_bound_post_fork_wait_outside_microvm`: outside a MicroVM, a shutdown timeout doesn't cut that wait short. Each new concurrency test fails without the change it covers. Local runs (Python 3.14): - `scripts/run-tests --venv 128b106 -- tests/tracer/test_tracer.py tests/tracer/test_writer.py tests/tracer/runtime/test_runtime_id.py`: 316 passed, 13 skipped, 1 xpassed. - After rebasing onto `main` `2ad3da71cb`: `scripts/run-tests -r --venv 128b106 -- tests/tracer/runtime/test_runtime_id.py tests/tracer/test_tracer.py tests/tracer/test_writer.py -k "test_runtime_id or test_writer or shutdown or post_fork"`: 209 passed, 10 skipped, 1 xpassed. - With the shutdown-retry and retry-containment fixes (Python 3.13): `scripts/run-tests --venv 1ef5a52 -- tests/tracer/test_writer.py tests/tracer/runtime/test_runtime_id.py tests/tracer/test_tracer.py`: 320 passed, 13 skipped, 1 xpassed. - `scripts/lint fmt` and `scripts/lint typing` on the changed files: passed. Not done yet: measuring `/run` in a MicroVM with the agent stopped (see Risks). ## Risks Outside a MicroVM the callback isn't registered and `_writer_lock` is `None`, so `write()` takes the lock-free path. The one change there: `shutdown()` now takes `_post_fork_lock` once, so a shutdown that races a post-fork writer rebuild on another thread waits for it and then stops the rebuilt writer, instead of leaving it running. The shutdown timeout doesn't bound that wait outside a MicroVM. Inside a MicroVM, once #20954 registers the listener and until #20955 lands: - `/run` waits for the old writer's final flush, because `stop()` joins the writer thread. That's one send plus `exporter.shutdown()`, which can take up to 3 s. With a healthy agent that's milliseconds. With an unreachable agent it's the send's retries plus up to 3 s. - Traces and client stats buffered before `/run` are sent under the snapshot's runtime ID, once by each restored instance. - Spans that other threads finish during that window are dropped. - Spans open across `/run` keep the old root `runtime-id` tag and go to the new writer. - A writer rebuild running at the same time as the refresh (`Tracer.configure()`, LLMObs) can still install a writer built for the old runtime ID. In supported usage these rebuilds run at startup, before the snapshot. Remote config, whose AppSec activation calls `tracer.configure()` at runtime, isn't supported in MicroVMs. #20955 serializes writer swaps with the refresh. - If `_recreate()` raises, the closed old writer stays installed and drops spans until a span start retries the rebuild successfully. While the rebuild keeps failing, every span start retries it, and the warning is rate-limited to one a minute. - In the exit-signal deadlock case, `shutdown(timeout)` can take up to about 2 × `timeout`: the bounded wait for an in-flight refresh, then the normal flush. If it gives up waiting while the span-start retry of a failed refresh (`_ensure_post_fork_writer()`) holds the lock, that retry stops the writer it installs. - A Rust panic from the native metadata write still reaches `/run`, as it does from `__init__` and after a fork. The write runs after the rebuild, so it can't skip it. Unchanged from `main`, and not specific to the refresh: - An exit signal that arrives while a writer rebuild is inside the old writer's `stop()` deadlocks on that writer's `_service_lock`. `Tracer.configure()` and the post-fork rebuild have the same exposure. - Outside a MicroVM, a post-fork rebuild interrupted by a same-thread shutdown whose signal handler returns can still install a writer after shutdown. ## Additional Notes Each part 2 tip has the same tree as its original PR: #20955 matches #19820, #20956 matches #19821, and #20957 matches #19822. So the fixes made on this PR and on the originals are all in part 2. Left out on purpose: - A cap on the `/run` wait. The telemetry worker's `stop()` (#20953) also blocks `/run`, and this code sets no timeout on it, so capping the trace writer alone wouldn't bound `/run`. - #19820's span-aggregator writer-swap lock. Without the discard, it would be held across the old writer's final flush and block every span start and finish. - #19820's detach of the context active at `/run`. ## Reference - [RFC](https://docs.google.com/document/d/17lde4Zak2YRBuDvf32KFanXj9TdpOv24GYaEY8swA1w/edit?tab=t.0#heading=h.wwu37decnk8j) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: tianning.li <tianning.li@datadoghq.com>
Base automatically changed from
tianning.li/3-3-2part-1-tracer-identity-refresh
to
main
October 15, 2026 20:54
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Failure cleanup and concurrent writer recreation can still preserve stale parentage or publish mismatched runtime identities.
2 open findings
What changed in this PR
Builds on the MicroVM writer refresh to discard pre-refresh trace state rather than flush it.
Changes:
- Rejects stale spans and contexts using identity generations.
- Discards writer buffers and defers synchronous sends outside aggregation locks.
- Adds refresh, recovery, and concurrency regression tests.
| File | Description |
|---|---|
| tests/tracer/test_writer.py | Tests writer discard and send races. |
| tests/tracer/runtime/test_runtime_id.py | Tests identity isolation and refresh lifecycle. |
| releasenotes/notes/fix-aws-lambda-microvm-stale-traces-a26d7bf374c1b2c2.yaml | Documents stale-trace prevention. |
| ddtrace/internal/writer/writer.py | Adds non-flushing discard behavior. |
| ddtrace/_trace/tracer.py | Rejects stale spans and detaches inherited contexts. |
| ddtrace/_trace/span.py | Defines the identity-generation key. |
| ddtrace/_trace/processor/__init__.py | Coordinates generations, writer swaps, and deferred sends. |
| ddtrace/_trace/context.py | Tracks context generations through weak references. |
🧠 Review effort: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+732
to
+734
| if self._runtime_identity is not None: | ||
| # A writer rebuilt after a fork is built for the child's runtime ID. | ||
| self._runtime_identity = (self._runtime_identity[0], get_runtime_id()) |
Contributor
Author
There was a problem hiding this comment.
Thanks, the race exists, but I'm leaving the code as is. When /run's rotation lands inside a non-refresh writer rebuild, the exporter takes whichever ID is current at the moment it's built (set_runtime_id(get_runtime_id())). So reading the ID before the rebuild instead of after only moves the mismatch to the other interleaving; it doesn't remove it. The window is also short: the refresh waits on the same aggregator lock and runs right after. It drops the old writer's buffered traces, advances the generation so spans started in the window are rejected, and rebuilds the writer for the new ID. Only a sync-mode send inside that window could go out mismatched. An exact fix would need every writer to report the ID it was built with, which isn't worth it for that window.
Comment on lines
+491
to
+497
| finally: | ||
| self._store_metadata() | ||
| self._post_fork_writer_pending = False | ||
| if self._shutting_down: | ||
| self._stop_replacement_writer() | ||
| return | ||
| self._detach_stale_context(stale_generation) |
Contributor
Author
There was a problem hiding this comment.
Valid. Fixed in 66e2a3d. When the writer rebuild fails, _refresh_runtime_identity() now detaches the inherited context in its failure path too, before re-raising or returning, since the generation may already have advanced and the span-start retry doesn't detach. An inherited active span no longer makes new spans stale after the retry rebuilds the writer, and an inherited remote Context is stamped with the old generation and deactivated, so it can't carry snapshot parentage into the new runtime. Covered by test_tracer_microvm_failed_refresh_still_detaches_inherited_contexts, which checks both context types and recovery through the span-start retry.
litianningdatadog
force-pushed
the
tianning.li/3-3-2part-2-discard-stale-traces
branch
from
October 15, 2026 21:16
8c40c3d to
fdf64c5
Compare
Part 1 stops the old trace writer the normal way. The old writer still flushes traces and client stats buffered before the refresh under the snapshot's runtime ID, /run waits for that final flush, and spans that were open across /run join the new runtime. This change replaces that with a discard: - Writers get drop_buffered_traces(). In MicroVMs it closes the write gate and drops the native exporter without shutting it down. A discarded writer skips its final flush, never starts another send, and its stop() does not wait for a send stuck in retries. - The span aggregator publishes a (generation, runtime ID) identity. A refresh bumps the generation, drops the aggregator's trace buffer and discards the writer under the aggregator lock. Other writer swaps take the same lock. - Spans and retained Context objects from an older generation are rejected before they reach the new writer. The context that was active at /run is detached, also when the writer rebuild fails. - A sync-mode write is buffered under the aggregator lock and sent after the lock is released, so a send stuck in retries cannot block span starts or /run. This replaces part 1's _stop_accepting_writes() with drop_buffered_traces(). It also switches part 1's refresh from _recreate(reset_buffer=False, flush_writer=False) to _recreate(reset_buffer=True, drop_buffered_traces=True). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
litianningdatadog
force-pushed
the
tianning.li/3-3-2part-2-discard-stale-traces
branch
from
October 15, 2026 22:07
fdf64c5 to
66e2a3d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked PRs:
/runlistener and identity refresh/runlistener; runtime-metrics runtime-ID tag (part 1 of chore(runtime): refresh runtime metric identity and collector state #19822)Part 1 PRs (#20952, #20953, #20954) can ship first. Part 2 PRs (#20955, #20956, #20957) can ship later.
Description
This PR is part 2 of #19820, on top of #20952. Its tip has exactly the code of #19820.
#20952 stops the old trace writer the normal way: it flushes buffered traces and client stats under the snapshot's runtime ID,
/runwaits for that flush, and spans open across/runjoin the new runtime. This PR replaces that with a discard:drop_buffered_traces()on the writers. In a MicroVM it closes the write gate and releases the native exporter withTraceExporter.drop()instead of the flushingshutdown(). A discarded writer:stop()for a send stuck in retries;_downgrade().A send already in flight drops the exporter once it releases the exporter lock.
The span aggregator publishes a
(generation, runtime ID)identity and, in a MicroVM, uses aforksafe.RLock. The refresh bumps the generation, resets the aggregator's trace buffer and discards the writer under that lock.reset()andconfigure_agentless_writer()take the same lock, so a concurrent writer swap can't install a writer built for the previous runtime ID.Spans carry their identity generation, and old-generation spans are rejected:
trace.span_finishlisteners and span processors run;Retained
Contextobjects keep their generation, so a child of a pre-refresh context is rejected too. A root rejected instart_span()records its context's generation, sochild_of=span.contextis rejected as well._detach_stale_context()deactivates the context active in the thread handling/run. The/runhook fires before distributed-header extraction, so that context belongs to the snapshot. A refresh whose writer rebuild fails detaches it too, because the generation may already have advanced and the span-start retry doesn't detach.In sync mode, a span finish buffers the trace under the writer and aggregator locks.
Tracer._on_span_finishsends it after releasing the aggregator lock, so a send stuck in retries can't block span starts or/run.What this replaces from #20952:
_refresh_runtime_identity()switches to_recreate(reset_buffer=True, drop_buffered_traces=True).TraceWriter._stop_accepting_writes()becomesdrop_buffered_traces()inSpanAggregator._reset_writer().Smaller changes:
_build_base_exporter_builder()moves.set_git_commit_sha(commit_sha)after.set_runtime_id(...), as fix(tracer): isolate MicroVM identity refresh #19820 does. Behaviour doesn't change.Testing
tests/tracer/runtime/test_runtime_id.py:test_tracer_microvm_identity_refresh_drops_direct_and_indirect_trace_buffers,test_tracer_microvm_identity_refresh_drops_inflight_old_trace: buffered and in-flight pre-refresh traces never reach the new writer.test_tracer_microvm_identity_refresh_rejects_stale_completion_for_reused_trace_id,test_tracer_microvm_identity_refresh_rejects_stale_span_before_finish_callbacks,test_tracer_microvm_identity_refresh_records_generation_of_span_rejected_at_start,test_tracer_microvm_identity_refresh_records_generation_of_root_rejected_by_start_span,test_tracer_microvm_identity_refresh_rejects_stale_context_before_start_callbacks,test_tracer_microvm_identity_refresh_detaches_inherited_contexts: old-generation spans and contexts are rejected at each step.test_asgi_run_hook_keeps_distributed_parent_after_tracer_refresh: after the detach, the/runrequest span keeps its upstream parent.test_tracer_microvm_failed_refresh_still_detaches_inherited_contexts: a refresh whose writer rebuild fails still detaches an inherited active span or remote context, and spans recover through the span-start retry.test_tracer_microvm_send_stuck_in_retries_does_not_block_spans_or_refresh,test_tracer_microvm_sync_send_stuck_in_retries_does_not_block_spans_or_refresh: with a send stuck in retries, an unrelated span andrefresh_identity()each finish in under 1 s.test_tracer_microvm_identity_refresh_serializes_writer_recreation_with_configure,test_root_span_runtime_id_matches_writer_identity_before_tracer_refresh,test_span_aggregator_lock_resets_after_fork,test_span_aggregator_outside_microvm_keeps_plain_lock_and_no_identitytests/tracer/test_writer.py:test_native_writer_refresh_drops_exporter_without_shutdown,test_native_writer_drop_exporter_drops_once,test_native_writer_discarded_stop_does_not_join_and_on_shutdown_drops_exporter,test_native_writer_stop_skips_join_only_for_discarded_microvm_writer,test_native_writer_del_drops_exporter_when_refresh_lost_lock_to_non_sendertest_native_writer_drop_buffered_traces_does_not_wait_for_send,test_native_writer_write_does_not_wait_for_send,test_native_writer_send_payload_rechecks_gate_under_exporter_lock,test_native_writer_recreate_after_sync_drop_does_not_wait_for_active_send,test_native_writer_flush_skips_send_after_microvm_drop_startstest_native_writer_send_finishing_after_refresh_skips_response_callback,test_native_writer_downgrade_skipped_after_microvm_drop,test_native_writer_downgrade_rechecks_gate_when_replacing_exporter,test_native_writer_downgrade_builds_exporter_outside_exporter_locktest_native_writer_refresh_does_not_touch_inherited_encoder,test_native_writer_refresh_does_not_touch_encoder_while_write_finishes,test_native_writer_drop_buffered_traces_clears_encoder_outside_microvm, and theHTTPWriterand CI Visibility writer versions: the MicroVM path never touches an encoder whose lock may be held by a thread lost in the snapshot.Local runs (Python 3.13):
scripts/run-tests --venv ed437ab -- tests/tracer/test_writer.py tests/tracer/runtime/test_runtime_id.py: 229 passed, 10 skipped, 1 xpassed.scripts/lint styleandscripts/lint typingon the changed files: passed.Risks
Pre-refresh trace data is intentionally discarded during a MicroVM identity refresh. Outside a MicroVM the identity, locks and generation checks are disabled, and non-MicroVM behavior, including post-fork writer recreation, is unchanged.
Inside a MicroVM:
TraceExporter.drop()doesn't stop the workers the replaced exporter registered on the shared native runtime. Its client-side stats worker can still send pre-refresh stat buckets under the snapshot's runtime ID. Closing that needs a libdatadog exporter shutdown that discards without flushing.Contextthat survived the snapshot without being activated, used as a parent, or copied into a context that was used as a parent, are treated as current.Tracer.configure()) holds the aggregator lock while the old writer flushes.send()when the refresh starts still runs its retries on its own thread. Nothing else waits on it.Additional Notes
After this merges, rebase #20956 and #20957: their first commit repeats this PR's commit and drops out.
Reference
🤖 Generated with Claude Code