Repository navigation
chore(runtime): add explicit identity refresh - #19778
Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 1 commit intoSep 28, 2026
Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 1 commit into
gh-worker-dd-mergequeue-cf854d[bot] merged 1 commit into
Conversation
Circular import analysis
|
Codeowners resolved asResolved from the full PR diff against No remaining files require a CODEOWNERS review. |
Dependency direction analysis
|
Contributor
|
✅ All CI checks and tests passed. 🎉 All green!🧪 All tests passed 🔗 Commit SHA: a444344 | Docs | View more details | Give us feedback! |
BenchmarksBenchmark execution time: 2026-09-25 22:05:56 Comparing candidate commit a444344 in PR branch Found 0 performance improvements and 6 performance regressions! Performance is the same for 350 metrics, 9 unstable metrics, 4 known flaky benchmarks, 4 flaky benchmarks without significant changes.
|
Contributor
There was a problem hiding this comment.
Pull request overview
Adds an explicit, non-fork “process identity refresh” pathway for runtimes that can resume from snapshots without an OS-level fork, along with a weakly-referenced subscriber mechanism so long-lived components can rebuild identity-bound state without leaking instances.
Changes:
- Introduces
ddtrace.internal.runtime.refresh_identity()to rotate the runtime id without recording fork lineage and to notify registered subscribers. - Refactors
on_runtime_id_change()to store subscribers via weak references and isolates subscriber exceptions during notification. - Expands runtime-id test coverage for refresh semantics, lineage invariants, subscriber notification, exception isolation, and weakref cleanup.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
ddtrace/internal/runtime/__init__.py |
Adds refresh_identity(), weakref-based subscriber registry, and subscriber notification isolation; keeps fork behavior on the forksafe hook without subscriber notification. |
tests/tracer/runtime/test_runtime_id.py |
Adds subprocess-based tests for refresh behavior and subscriber semantics; updates imports for runtime module usage. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
litianningdatadog
force-pushed
the
tianning.li/1-runtime-identity-refresh
branch
from
August 20, 2026 10:06
aac2ec3 to
2fa8088
Compare
litianningdatadog
added
the
changelog/no-changelog
litianningdatadog
changed the title
litianningdatadog
changed the title
litianningdatadog
added
changelog/no-changelog
litianningdatadog
marked this pull request as ready for review
August 20, 2026 15:20
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2fa8088e77
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
brettlangdon
left a comment
brettlangdon
left a comment
Member
There was a problem hiding this comment.
This feels very over-engineered, I don't feel like it is clear why we are moving everything to be weakrefs? it adds a ton of machinery that isn't clear the value we are getting from it.
I am also surprised that removing notifications from _set_runtime_id doesn't break a bunch of things, or how it wouldn't break a bunch of things.
litianningdatadog
force-pushed
the
tianning.li/1-runtime-identity-refresh
branch
from
August 23, 2026 20:33
d4a3faa to
1cbc151
Compare
This was referenced Aug 23, 2026
litianningdatadog
requested review from
brettlangdon and
juanjux
and removed request for
a team
September 21, 2026 20:11
brettlangdon
approved these changes
Sep 23, 2026
litianningdatadog
force-pushed
the
tianning.li/1-runtime-identity-refresh
branch
5 times, most recently
from
September 23, 2026 16:47
a9aa117 to
1b30f8f
Compare
Contributor
Author
|
/merge |
|
View all feedbacks in Devflow UI.
It will be processed automatically as soon as GitHub reports it as mergeable. View in MergeQueue UI.
devflow unqueued this merge request: It did not become mergeable within the expected time |
rachelyangdog
approved these changes
Sep 25, 2026
Contributor
Author
|
/merge |
|
View all feedbacks in Devflow UI.
The expected merge time in
This PR is rejected because it was updated |
litianningdatadog
force-pushed
the
tianning.li/1-runtime-identity-refresh
branch
from
September 25, 2026 15:50
976f653 to
b06e5c0
Compare
litianningdatadog
added this pull request to stack #20582
September 25, 2026 15:50
Signed-off-by: Tianning Li <tianning.li@datadoghq.com>
litianningdatadog
force-pushed
the
tianning.li/1-runtime-identity-refresh
branch
from
September 25, 2026 21:37
b06e5c0 to
a444344
Compare
juanjux
approved these changes
Sep 28, 2026
gh-worker-dd-mergequeue-cf854d
Bot
deleted the
tianning.li/1-runtime-identity-refresh
branch
September 28, 2026 15:37
gh-worker-dd-mergequeue-cf854d Bot
pushed a commit
that referenced
this pull request
Oct 8, 2026
Stacked PRs: - #19778 — runtime identity foundation - This PR #19898 — generic WSGI/ASGI/Django request-start event - #19939 — MicroVM request-start listener and identity refresh - #19820 — tracer/native writer exporter refresh - #19821 — telemetry worker refresh - #19822 — runtime-metrics runtime-ID tags ## Description This PR provides the framework side of the MicroVM identity-refresh flow. WSGI, ASGI, and Django publish `web.request.starting` before they create a root span, passing the request method and normalized path. The publisher is deliberately product-neutral: it does not import MicroVM helpers, recognize the MicroVM `/run` endpoint, or decide whether to refresh a runtime ID. PR #19939 registers the MicroVM-only listener and owns that policy. When no product listens for the event, the publisher returns after a single listener check, before reading the request or normalizing the path. Two helpers in `trace_utils` own the publishing: - `dispatch_wsgi_web_request_starting(environ)` is used by `DDWSGIMiddleware` and Django's `get_response` hook. WSGI and Django share an environ marker so nested layers publish one event. The Django hook also covers Django 3.0 ASGI requests, which reach the sync `get_response` because `get_asgi_application` is only wrapped on Django 3.1+. - `dispatch_asgi_web_request_starting(scope)` publishes only from the outer ASGI application. It accepts scopes whether or not `path` already includes `root_path`, so mounted or server-prefixed apps do not produce a duplicated prefix. Scope: the event is published for the generic WSGI/ASGI middleware and Django. Frameworks whose automatic instrumentation creates the root span without `DDWSGIMiddleware` (Falcon, Bottle, Pyramid, CherryPy, Molten) are out of scope for this PR and will follow separately. This implements the requested split in [review comment r4146791727](#19898 (comment)). ## Reference - [RFC](https://docs.google.com/document/d/17lde4Zak2YRBuDvf32KFanXj9TdpOv24GYaEY8swA1w/edit?tab=t.0#heading=h.wwu37decnk8j) ## Testing - `scripts/lint fmt` and `scripts/lint typing` on the changed files - focused WSGI request-start tests (py3.13): 7 passed - focused ASGI request-start tests (py3.13): 48 passed - focused Django request-start tests: 3 passed on Django 3.0 (py3.9) and on Django 5.1 (py3.13), including a real `ASGIRequest` reaching `get_response` ## Risks There is no public API change. The only production listener in this stack is registered by PR #19939 in MicroVM processes, so non-MicroVM requests do not dispatch this event. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: tianning.li <tianning.li@datadoghq.com>
This was referenced Oct 13, 2026
gh-worker-dd-mergequeue-cf854d Bot
pushed a commit
that referenced
this pull request
Oct 13, 2026
Stacked PRs: - #19778 — runtime identity foundation (merged) - #19898 — generic WSGI/ASGI/Django request-start event (merged) - This PR #19939 — MicroVM `/run` listener and identity refresh - #19820 — tracer/native writer refresh - #19821 — telemetry worker refresh - #19822 — runtime-metrics runtime-ID tags; registers the listener ## Description This PR adds the MicroVM side of the `web.request.starting` event that #19898 publishes. The method and path check lives in a MicroVM-specific listener instead of the integrations, as asked in [r4146791727](#19898 (comment)). - `maybe_refresh_identity(method, path)` matches only `POST /aws/lambda-microvms/runtime/v1/run` and calls `refresh_identity()` once per process. Nothing in this PR registers `maybe_refresh_identity()`. #19822 registers it with `core.on()` inside its MicroVM check, at the top of the stack, once the tracer, telemetry and runtime metrics can each rebuild on refresh. Merging this PR alone doesn't change runtime behavior. WSGI and Django can both observe the same `/run` request, so the refresh has to happen exactly once. A `forksafe.Event` records that it's done and is checked first without a lock, so after the refresh the listener costs one flag check per request. A lock private to `_runtime_id.py` guards the refresh itself, so concurrent observers still rotate the ID once. No other module takes that lock. `/run` reaches a process once, so there's no retry. If a refresh callback fails, the failure is logged and the remaining callbacks still run. Each consumer keeps its own state consistent with `get_runtime_id()`. Smaller changes: - `WEB_REQUEST_STARTING_EVENT` moves to `ddtrace.internal.constants`, and `WebFrameworkEvents.WEB_REQUEST_STARTING` aliases it. #19822 registers the listener from `ddtrace/trace/__init__.py`, which can't import `ddtrace.contrib`: its package `__init__` loads `trace_handlers`, which imports `ddtrace.trace` back before `tracer` exists. - `MICROVM_RUN_HOOK_METHOD` and `MICROVM_RUN_HOOK_PATH` live in `ddtrace.internal.serverless`. ## Testing `tests/tracer/runtime/test_runtime_id.py`: - `test_maybe_refresh_identity_matches_microvm_run_hook`, `test_maybe_refresh_identity_ignores_other_requests`: only `POST` to the `/run` path refreshes. - `test_identity_refresh_hook_runs_before_root_span_creation`: the refresh happens before the `/run` root span is created. - `test_maybe_refresh_identity_is_thread_safe`: concurrent observers rotate the ID once. - `test_identity_refresh_callback_failure_is_logged_and_not_retried` - `test_microvm_refresh_guard_resets_after_fork` - `test_maybe_refresh_identity_noop_outside_microvm`, `test_identity_refresh_listener_noop_outside_microvm`, `test_identity_refresh_listener_noop_for_blank_microvm_env` - `test_import_ddtrace_in_microvm_environment`, `test_import_ddtrace_outside_microvm_does_not_import_core` Integration tests: - `test_microvm_run_hook_refreshes_identity` in both `tests/contrib/wsgi/test_wsgi.py` and `tests/contrib/asgi/test_asgi.py`: a `/run` request refreshes the identity end to end. - `tests/contrib/wsgi/test_wsgi.py::test_web_request_starting_isolates_listener_errors`: a failing listener doesn't fail the request. ## Risks None outside a MicroVM: nothing in this PR registers the listener, #19822 registers it only inside a MicroVM, and `maybe_refresh_identity()` returns early. The import tests above check that the no-op path doesn't import `ddtrace.internal.core`. Inside a MicroVM nothing changes until #19822 registers the listener. ## Additional Notes An earlier revision shared a refresh lock with consumers (`get_runtime_identity_refresh_lock()`) and retried failed callbacks against the same identity. I dropped both after review: - [r4186658721](#19939 (comment)): don't expose a global lock as cross-module API or take it on every span start and finish. Consumers now read a published identity snapshot instead. - [r4188215619](#19939 (comment)): the ID is committed before the callbacks run, so retries couldn't stop mixed-identity data from getting out. Each consumer now owns its own consistency. - [r4188215636](#19939 (comment)): callback order was nondeterministic and the tests forced it. Without a shared lock or retry queue, order doesn't matter. ## Reference - [RFC](https://docs.google.com/document/d/17lde4Zak2YRBuDvf32KFanXj9TdpOv24GYaEY8swA1w/edit?tab=t.0#heading=h.wwu37decnk8j) Co-authored-by: tianning.li <tianning.li@datadoghq.com>
gh-worker-dd-mergequeue-cf854d Bot
pushed a commit
that referenced
this pull request
Oct 15, 2026
…20952) Stacked PRs: - #19778 — runtime identity foundation (merged) - #19898 — generic WSGI/ASGI/Django request-start event (merged) - #19939 — MicroVM `/run` listener and identity refresh (merged) - This PR #20952 — tracer: rebuild the trace writer (part 1 of #19820) - #20953 — telemetry: rebuild the worker (part 1 of #19821) - #20954 — registers the `/run` listener; runtime-metrics runtime-ID tag (part 1 of #19822) - #20957 — runtime metrics: reset collector baselines (part 2 of #19822) - #20956 — telemetry: discard the old worker and exporter subscription (part 2 of #19821) - #20955 — tracer: drop trace state from before the refresh (part 2 of #19820) Part 1 PRs (#20952, #20953, #20954) can ship first. Part 2 PRs (#20955, #20956, #20957) can ship later. **Release coupling:** #20952 is intended to ship together with #20954. This PR registers the tracer as an identity-refresh subscriber; #20954 registers `maybe_refresh_identity()` on `WEB_REQUEST_STARTING_EVENT`, which is what makes the `/run` request trigger the refresh in production. Do not release #20952 without #20954. ## Description This PR and #20955 split #19820, so the runtime-ID fix can ship before the discard logic. This PR makes the tracer report the new runtime ID after a MicroVM identity refresh. #20955 drops trace state from before the refresh. The native trace exporter copies the runtime ID when it's built (`set_runtime_id(get_runtime_id())` in `_build_base_exporter_builder()`), so after `refresh_identity()` the tracer keeps reporting the snapshot's runtime ID until its writer is rebuilt. The root span's `runtime-id` tag already reads `get_runtime_id()` for each span and needs no change. - `ddtrace/trace/__init__.py` registers `tracer._refresh_runtime_identity` with `on_runtime_identity_refresh()` inside a MicroVM check. - `Tracer._refresh_runtime_identity()` rebuilds the writer the way the post-fork path does (`_recreate(reset_buffer=False, flush_writer=False)`), then stores the tracer metadata again, in a `finally`, so a failed rebuild still records the new runtime ID. A failed rebuild is retried on the next span start. A Rust panic from the rebuild is logged instead of re-raised, because the refresh dispatcher only contains `Exception` and the panic would otherwise abort the `/run` request. The span-start retry contains its own failures in a MicroVM: an `Exception` or Rust panic from the rebuild is logged (rate-limited) and the retry stays pending, so it doesn't fail the app's span start. `KeyboardInterrupt` and `SystemExit` still propagate, and outside a MicroVM a failed post-fork rebuild still raises, as on `main`. - The refresh and `shutdown()` serialize on `_post_fork_lock`. `shutdown()` sets `_shutting_down` under that lock, so it waits for a refresh in flight and stops the writer that refresh installs, and a refresh that starts after shutdown builds nothing. The lock is now reentrant, because the exit-signal handler can run `shutdown()` on the thread that is refreshing. If that handler returns, the refresh sees `_shutting_down` after `_recreate()` and stops the writer it just installed, shutting down the exporter of a `NativeWriter` that never started. - In a MicroVM, `shutdown(timeout)` waits at most `timeout` (clamped at 0) for `_post_fork_lock`, then sets `_shutting_down` either way. Without the bound, an exit signal could deadlock: a refresh on another thread holds `_post_fork_lock` while waiting for the old writer's `_writer_lock`, which the thread running the signal handler holds. A refresh still running after the wait stops the writer it installs, through the same `_shutting_down` check. `shutdown()` without a timeout still waits as before. The span-start retry of a failed refresh (`_ensure_post_fork_writer()`) does the same in a MicroVM: it builds nothing once `_shutting_down` is set, and stops a writer it installs after that wait. Outside a MicroVM the post-fork rebuild is unchanged. The rebuild stops the old writer the normal way, so the old writer still flushes what it buffered. A span finished on another thread after that stop would restart the old writer through the lazy start in `_write_with_client()`, leaving a writer that nothing stops. So the refresh first closes the old writer to new spans, using #19820's MicroVM-only write gate: `_writer_lock` and `_accepting_writes` on `HTTPWriter` and `NativeWriter`, checked in `write()`. `TraceWriter._stop_accepting_writes()` closes the gate without dropping anything. #20955 replaces it with `drop_buffered_traces()`. How a refresh runs, and what a span finished on another thread sees meanwhile: ```mermaid sequenceDiagram participant Run as /run thread participant T as Tracer participant Old as old writer participant New as new writer participant App as span-finishing thread Run->>T: _refresh_runtime_identity(new_id) Note over T: holds _post_fork_lock (reentrant) until the end opt shutdown already started T-->>Run: return, build nothing end T->>Old: _stop_accepting_writes() (under the old writer's _writer_lock) T->>Old: _recreate(reset_buffer=False, flush_writer=False) stops it, flushing what it buffered T->>New: built by recreate() with the new runtime ID T->>T: _store_metadata() in a finally, so the new runtime ID is recorded even if the rebuild failed opt shutdown() ran during the rebuild T->>New: stop the writer it just installed end Note over T: releases _post_fork_lock App->>T: span finishes, aggregator writes the trace alt before the old writer is closed T->>Old: written, and flushed when the old writer stops else after closing, before the swap T->>Old: write() returns without writing, so the closed writer is not restarted else after the swap T->>New: written under the new runtime ID end ``` Nothing registers `maybe_refresh_identity()` until #20954, so merging this PR alone doesn't change runtime behavior. Smaller changes: - `NativeWriter.write()` takes the write gate through the sync-mode flush, as `HTTPWriter.write()` does. Otherwise the refresh could close the writer between the buffering and the flush, and `recreate()` would shut down the exporter that flush then sends through. #19820 keeps the flush outside the gate and instead rechecks `_discarded_by_refresh` before each send; those rechecks belong to #20955. ## Testing `tests/tracer/runtime/test_runtime_id.py`: - `test_tracer_microvm_identity_refresh_recreates_exporter_without_fork_side_effects`: the refresh rebuilds the writer with the post-fork arguments and stores the metadata, without fork side effects. - `test_tracer_microvm_identity_refresh_closes_and_flushes_old_writer`: the old writer sends its buffer once and shuts its exporter down. A later write neither buffers nor restarts it. - `test_tracer_microvm_identity_refresh_after_shutdown_does_not_recreate_writer` - `test_tracer_microvm_identity_refresh_keeps_pending_after_recreate_failure`: a failed rebuild keeps the pending post-fork rebuild and still stores the metadata. - `test_tracer_microvm_identity_refresh_schedules_rebuild_after_recreate_failure`: a failed rebuild is left pending for the next span. `RuntimeError` and `KeyboardInterrupt` propagate; a Rust panic doesn't. - `test_tracer_identity_refresh_contains_native_panic_from_writer_rebuild`: a panic from the rebuild doesn't escape `refresh_identity()`, and other refresh subscribers still run. - `test_tracer_microvm_identity_refresh_retries_failed_writer_recreation_on_next_span`: after a failed rebuild, the next span rebuilds the writer, and the stored metadata carries the refreshed runtime ID. - `test_tracer_shutdown_waits_for_in_flight_identity_refresh`: shutdown during a refresh stops the writer the refresh installs. - `test_tracer_identity_refresh_during_shutdown_does_not_recreate_writer`: a refresh that starts while shutdown runs builds no writer. - `test_tracer_shutdown_on_refreshing_thread_stops_replacement_writer`: shutdown run on the refreshing thread (the exit-signal case) doesn't deadlock, and the replacement writer is stopped, or has its exporter shut down if it never started. - `test_tracer_signal_shutdown_does_not_deadlock_with_refresh_waiting_on_writer_lock`: shutdown run by a thread that holds the writer lock a refresh is waiting on (the cross-thread exit-signal case) returns after its timeout, and the refresh then stops the writer it installed. - `test_tracer_microvm_shutdown_accepts_negative_timeout` - `test_tracer_timed_out_shutdown_stops_writer_from_retried_refresh_rebuild`: a `shutdown(timeout=0)` that finishes while the span-start retry of a failed refresh is rebuilding still gets the rebuilt writer stopped. It fails without the fix. - `test_tracer_post_fork_rebuild_ignores_shutdown_flag_outside_microvm`: outside a MicroVM the post-fork rebuild ignores `_shutting_down`, as before. - `test_tracer_microvm_failed_rebuild_retry_is_contained_in_start_span`: in a MicroVM a span-start retry that fails again with a `RuntimeError` or Rust panic doesn't raise and stays pending, `KeyboardInterrupt` still propagates, and a later span start recovers. It fails without the fix. - `test_tracer_post_fork_rebuild_failure_still_raises_outside_microvm`: outside a MicroVM a failed post-fork rebuild still raises from the span start, as before. - `test_identity_refresh_hook_notifies_global_tracer`, `test_deferred_global_tracer_registers_identity_refresh_callback`, `test_deferred_global_tracer_does_not_register_identity_refresh_callback_outside_microvm`: the global tracer registers only in a MicroVM, including with deferred initialization. `tests/tracer/test_writer.py`: - `test_native_writer_closed_microvm_writer_rejects_writes_without_restarting`, `test_http_writer_closed_microvm_writer_rejects_writes`: a closed writer drops later spans and isn't restarted by them. Spans buffered before the close stay for the final flush. - `test_native_writer_microvm_close_waits_for_in_flight_sync_flush`: closing a sync-mode writer waits for a flush that is sending, and nothing is flushed through the closed writer. - `test_native_writer_stop_accepting_writes_is_noop_outside_microvm`, `test_log_writer_stop_accepting_writes_is_noop` `tests/tracer/test_tracer.py`: - `test_tracer_shutdown_waits_for_in_flight_post_fork_writer_recreation`: shutdown waits for a post-fork writer rebuild on another thread and stops the rebuilt writer. - `test_tracer_shutdown_timeout_does_not_bound_post_fork_wait_outside_microvm`: outside a MicroVM, a shutdown timeout doesn't cut that wait short. Each new concurrency test fails without the change it covers. Local runs (Python 3.14): - `scripts/run-tests --venv 128b106 -- tests/tracer/test_tracer.py tests/tracer/test_writer.py tests/tracer/runtime/test_runtime_id.py`: 316 passed, 13 skipped, 1 xpassed. - After rebasing onto `main` `2ad3da71cb`: `scripts/run-tests -r --venv 128b106 -- tests/tracer/runtime/test_runtime_id.py tests/tracer/test_tracer.py tests/tracer/test_writer.py -k "test_runtime_id or test_writer or shutdown or post_fork"`: 209 passed, 10 skipped, 1 xpassed. - With the shutdown-retry and retry-containment fixes (Python 3.13): `scripts/run-tests --venv 1ef5a52 -- tests/tracer/test_writer.py tests/tracer/runtime/test_runtime_id.py tests/tracer/test_tracer.py`: 320 passed, 13 skipped, 1 xpassed. - `scripts/lint fmt` and `scripts/lint typing` on the changed files: passed. Not done yet: measuring `/run` in a MicroVM with the agent stopped (see Risks). ## Risks Outside a MicroVM the callback isn't registered and `_writer_lock` is `None`, so `write()` takes the lock-free path. The one change there: `shutdown()` now takes `_post_fork_lock` once, so a shutdown that races a post-fork writer rebuild on another thread waits for it and then stops the rebuilt writer, instead of leaving it running. The shutdown timeout doesn't bound that wait outside a MicroVM. Inside a MicroVM, once #20954 registers the listener and until #20955 lands: - `/run` waits for the old writer's final flush, because `stop()` joins the writer thread. That's one send plus `exporter.shutdown()`, which can take up to 3 s. With a healthy agent that's milliseconds. With an unreachable agent it's the send's retries plus up to 3 s. - Traces and client stats buffered before `/run` are sent under the snapshot's runtime ID, once by each restored instance. - Spans that other threads finish during that window are dropped. - Spans open across `/run` keep the old root `runtime-id` tag and go to the new writer. - A writer rebuild running at the same time as the refresh (`Tracer.configure()`, LLMObs) can still install a writer built for the old runtime ID. In supported usage these rebuilds run at startup, before the snapshot. Remote config, whose AppSec activation calls `tracer.configure()` at runtime, isn't supported in MicroVMs. #20955 serializes writer swaps with the refresh. - If `_recreate()` raises, the closed old writer stays installed and drops spans until a span start retries the rebuild successfully. While the rebuild keeps failing, every span start retries it, and the warning is rate-limited to one a minute. - In the exit-signal deadlock case, `shutdown(timeout)` can take up to about 2 × `timeout`: the bounded wait for an in-flight refresh, then the normal flush. If it gives up waiting while the span-start retry of a failed refresh (`_ensure_post_fork_writer()`) holds the lock, that retry stops the writer it installs. - A Rust panic from the native metadata write still reaches `/run`, as it does from `__init__` and after a fork. The write runs after the rebuild, so it can't skip it. Unchanged from `main`, and not specific to the refresh: - An exit signal that arrives while a writer rebuild is inside the old writer's `stop()` deadlocks on that writer's `_service_lock`. `Tracer.configure()` and the post-fork rebuild have the same exposure. - Outside a MicroVM, a post-fork rebuild interrupted by a same-thread shutdown whose signal handler returns can still install a writer after shutdown. ## Additional Notes Each part 2 tip has the same tree as its original PR: #20955 matches #19820, #20956 matches #19821, and #20957 matches #19822. So the fixes made on this PR and on the originals are all in part 2. Left out on purpose: - A cap on the `/run` wait. The telemetry worker's `stop()` (#20953) also blocks `/run`, and this code sets no timeout on it, so capping the trace writer alone wouldn't bound `/run`. - #19820's span-aggregator writer-swap lock. Without the discard, it would be held across the old writer's final flush and block every span start and finish. - #19820's detach of the context active at `/run`. ## Reference - [RFC](https://docs.google.com/document/d/17lde4Zak2YRBuDvf32KFanXj9TdpOv24GYaEY8swA1w/edit?tab=t.0#heading=h.wwu37decnk8j) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: tianning.li <tianning.li@datadoghq.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked PRs:
Description
Adds
ddtrace.internal.runtime.refresh_identity()for runtimes that need a fresh runtime ID without going through an OS fork.This introduces
on_runtime_identity_refresh(), a dedicated callback registry for consumers that should react only to explicit identity refreshes. Existingon_runtime_id_change()behavior remains available for fork-related and general runtime-ID change handling.refresh_identity()rotates the runtime ID without recording parent or ancestor lineage because a resumed MicroVM run is not a real fork. Existing fork behavior and lineage tracking remain unchanged.Refresh callbacks are isolated by default so one failed consumer does not block others. Callers coordinating a refresh transaction can opt into error propagation with
raise_on_error=Trueand retry the same identity refresh when a rebuild fails.Reference
Testing
Added coverage for:
refresh_identity()raise_on_error=TrueValidation:
scripts/run-tests -s --venv 190fcc7 -- -q tests/tracer/runtime/test_runtime_id.py— 21 passedscripts/lint checks— passedRisks
Low. The new path is opt-in and is not wired to traffic in this PR. Existing fork behavior is preserved.