Repository navigation
chore(tracer): rebuild the trace writer on MicroVM identity refresh - #20952
litianningdatadog wants to merge 1 commit into
Conversation
litianningdatadog
added
changelog/no-changelog
Circular import analysis
|
Codeowners resolved asResolved from the full PR diff against |
Dependency direction analysis
|
|
✅ All CI checks and tests passed. 🎉 All green!🧪 All tests passed 🔗 Commit SHA: 3d14424 | Docs | View more details | Give us feedback! |
BenchmarksBenchmark execution time: 2026-10-14 21:04:49 Comparing candidate commit 3d14424 in PR branch Found 0 performance improvements and 4 performance regressions! Performance is the same for 357 metrics, 9 unstable metrics, 3 known flaky benchmarks, 5 flaky benchmarks without significant changes.
|
There was a problem hiding this comment.
🟡 Changes recommended
Concurrent synchronous writes or shutdown can leave exporter operations or replacement writers active after teardown.
2 open findings
What changed in this PR
Rebuilds the trace writer after MicroVM identity refreshes so exports use the new runtime ID.
Changes:
- Registers the global tracer for MicroVM identity refresh callbacks.
- Recreates writer state and prevents writes to replaced writers.
- Adds coverage for refresh, shutdown, and writer-gating behavior.
| File | Description |
|---|---|
ddtrace/trace/__init__.py |
Registers the MicroVM refresh callback. |
ddtrace/_trace/tracer.py |
Recreates the writer after identity refresh. |
ddtrace/internal/writer/writer.py |
Adds write gates and refactors native writes. |
tests/tracer/runtime/test_runtime_id.py |
Tests tracer refresh behavior. |
tests/tracer/test_writer.py |
Tests closed-writer handling. |
🧠 Review effort: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
c5961b2 to
f6d4c74
Compare
litianningdatadog
changed the title
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f6d4c74bd6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| except BaseException: | ||
| # /run is not repeated, so a failed rebuild would leave the closed writer installed. | ||
| # Retry it on the next span start, like a post-fork rebuild. | ||
| self._post_fork_writer_pending = True | ||
| raise |
There was a problem hiding this comment.
When writer reconstruction raises a PyO3 panic—the new tests explicitly model this BaseException path—this handler schedules a retry but then re-raises. _notify_runtime_identity_refresh_callbacks() catches only Exception, and core event dispatch also propagates BaseException, so an identity refresh can escape into and abort the triggering /run request before refresh completion is recorded or other subscribers run. Suppress the known native panic at the lifecycle callback boundary after scheduling the retry.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in fc8abd4. The rebuild still marks itself pending first, but a Rust panic (is_panic_exception) is now logged instead of re-raised, so it no longer gets past the refresh dispatcher into /run. KeyboardInterrupt, SystemExit and ordinary exceptions still propagate, and the dispatcher logs the ordinary ones. test_tracer_identity_refresh_contains_native_panic_from_writer_rebuild drives this through refresh_identity().
One gap stays as is: a panic from the native metadata write can still reach /run. __init__ and the fork path have the same exposure. The write now runs after the rebuild, so it can't skip it.
| except BaseException: | ||
| # /run is not repeated, so a failed rebuild would leave the closed writer installed. | ||
| # Retry it on the next span start, like a post-fork rebuild. | ||
| self._post_fork_writer_pending = True | ||
| raise | ||
| self._post_fork_writer_pending = False |
There was a problem hiding this comment.
If the initial refresh rebuild raises an ordinary exception, the callback framework logs and suppresses it and this flag makes the next start_span() retry through _ensure_post_fork_writer(). That retry recreates the writer and clears the flag but never calls _store_metadata(), so the exporter recovers with the new runtime ID while the persisted tracer metadata remains associated with the previous identity for the rest of the process. Preserve that this pending rebuild came from an identity refresh and update metadata when the retry succeeds.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in fc8abd4. _store_metadata() now runs in a finally after the rebuild, so the new runtime ID is stored whether the rebuild succeeds or fails, and the retry in _ensure_post_fork_writer() doesn't need to know where the pending rebuild came from. The two failure-path tests now expect one metadata write. test_tracer_microvm_identity_refresh_retries_failed_writer_recreation_on_next_span checks that the stored runtime ID is the refreshed one.
![datadog-prod-us1-6[bot]](https://avatars.githubusercontent.com/u/365230?s=60&v=4){kind=small}
There was a problem hiding this comment.
An exit signal can deadlock shutdown when a MicroVM refresh waits for a writer lock held by the interrupted thread. Successful recovery after a failed writer rebuild also leaves stored tracer metadata on the old runtime ID.
🤖 Bits Code Review · Commit f6d4c74
| # Do not recreate an inherited writer only to shut it down. The span aggregator | ||
| # discards its buffered traces during shutdown. | ||
| self._post_fork_writer_pending = False | ||
| with self._post_fork_lock: |
There was a problem hiding this comment.
In a MicroVM, refresh on another thread can hold _post_fork_lock while waiting for _writer_lock held by the main thread inside write(). If SIGTERM interrupts that write, its handler invokes shutdown() and blocks here on _post_fork_lock. The interrupted write cannot release its lock until the handler returns, so refresh and shutdown deadlock permanently. A synchronous flush makes this window especially likely. Reentrancy does not resolve this cross-thread cycle; the refresh and signal-shutdown locking need coordinated changes.
React: 👍 useful · 👎 wrong · 😕 unclear · 👀 right, but not acting on it
What each reaction means
- 👍 Useful: correct, and you're fixing it or will.
- 👎 Wrong: the finding is incorrect, for example the bug can't happen or the code was misread.
- 😕 Unclear: you can't tell what it means or why it matters.
- 👀 Right, not acting: correct, but intended, out of scope for this PR, a known risk, or too minor.
🤖 Bits Code Review · Open Bits AI session
There was a problem hiding this comment.
Fixed in fc8abd4. The reentrant lock only covered the handler running on the refreshing thread. In a MicroVM, shutdown(timeout) now waits at most timeout (clamped at 0) for _post_fork_lock, then sets _shutting_down either way. A refresh still running after that sees the flag once its rebuild returns and stops the writer it installed, the same handoff the same-thread case already used. shutdown() with no timeout still waits as before, and outside MicroVMs the wait isn't bounded at all.
test_tracer_signal_shutdown_does_not_deadlock_with_refresh_waiting_on_writer_lock reproduces the cycle with threads: one holds the writer lock and calls shutdown(), while the refresh holds _post_fork_lock and waits on that writer lock. It fails without the fix.
The cost is that in this case a MicroVM shutdown can take up to about 2 × timeout: the lock wait, then the flush. That's noted in the PR description.
| except BaseException: | ||
| # /run is not repeated, so a failed rebuild would leave the closed writer installed. | ||
| # Retry it on the next span start, like a post-fork rebuild. | ||
| self._post_fork_writer_pending = True |
There was a problem hiding this comment.
If writer recreation fails after the runtime ID rotates, this schedules recovery through _ensure_post_fork_writer() and skips _store_metadata(). When the next span successfully retries, that method rebuilds the writer and clears the pending flag without updating metadata. Stored tracer metadata therefore retains the snapshot's runtime ID while the replacement exporter and new spans use the refreshed ID. Update metadata on successful deferred recovery and extend the recovery test to verify it.
React: 👍 useful · 👎 wrong · 😕 unclear · 👀 right, but not acting on it
What each reaction means
- 👍 Useful: correct, and you're fixing it or will.
- 👎 Wrong: the finding is incorrect, for example the bug can't happen or the code was misread.
- 😕 Unclear: you can't tell what it means or why it matters.
- 👀 Right, not acting: correct, but intended, out of scope for this PR, a known risk, or too minor.
🤖 Bits Code Review · Open Bits AI session
There was a problem hiding this comment.
Same fix as the Codex thread on this file, in fc8abd4: metadata is stored in a finally after the rebuild, so a failed rebuild still records the new runtime ID. As suggested, the recovery test (test_tracer_microvm_identity_refresh_retries_failed_writer_recreation_on_next_span) now checks that the stored metadata carries the refreshed runtime ID after the retry.
f6d4c74 to
fc8abd4
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fc8abd48f0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| # A refresh still in flight after the wait stops the writer it installs once it sees the flag. | ||
| # Lock.acquire() rejects negative timeouts other than -1, which shutdown() used to accept. | ||
| wait = max(timeout, 0) if timeout is not None and in_aws_lambda_microvm() else -1 | ||
| locked = self._post_fork_lock.acquire(timeout=wait) |
There was a problem hiding this comment.
When an identity-refresh rebuild has failed, the next start_span() retries via _ensure_post_fork_writer() under this same lock. If shutdown(timeout=...) times out here while that retry is rebuilding, shutdown proceeds and stops the old writer, but _ensure_post_fork_writer() can subsequently install a replacement without checking _shutting_down or stopping it; the already-entered start_span() can also continue after shutdown. Coordinate this timeout path with deferred rebuilds so any replacement created after shutdown is disposed.
AGENTS.md reference: AGENTS.md:L11-L11
Useful? React with 👍 / 👎.
| # waiting for a writer lock held by the thread an exit-signal handler interrupted to run this. | ||
| # A refresh still in flight after the wait stops the writer it installs once it sees the flag. | ||
| # Lock.acquire() rejects negative timeouts other than -1, which shutdown() used to accept. | ||
| wait = max(timeout, 0) if timeout is not None and in_aws_lambda_microvm() else -1 |
There was a problem hiding this comment.
When documented-default shutdown(timeout=None) is called from a signal handler that interrupted a MicroVM writer while it holds _writer_lock, another thread may hold _post_fork_lock while waiting for that writer lock during identity refresh. Selecting -1 here makes the handler wait forever for _post_fork_lock, while the refresh cannot progress until the handler returns and releases _writer_lock; the bounded-wait protection added for this lock inversion therefore needs to cover the None case as well.
AGENTS.md reference: AGENTS.md:L11-L11
Useful? React with 👍 / 👎.
| if writer_lock is None: | ||
| self._write_unlocked(spans) | ||
| return | ||
| with writer_lock: |
There was a problem hiding this comment.
When another thread calls the public Tracer.flush() during an identity refresh, these new lifecycle locks do not protect flush_queue(): the flush can remove a payload from the encoder, pause before acquiring _exporter_lock, and then resume after refresh has shut down that exporter. The send then targets the dead exporter and the already-encoded payload is lost (or a native failure escapes), so explicit flushes need to participate in the same close/replacement synchronization as write().
AGENTS.md reference: AGENTS.md:L11-L11
Useful? React with 👍 / 👎.
fc8abd4 to
fc73ae2
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fc73ae210e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| return | ||
| # The old writer still flushes what it holds when stopped, but a span written to it | ||
| # after that would restart it with no owner, so close it to new spans first. | ||
| self._span_aggregator.writer._stop_accepting_writes() |
There was a problem hiding this comment.
When another thread finishes a trace during an identity refresh, the aggregator still references the old writer after this call and until _recreate() finishes stopping/flushing it, constructing the exporter, and publishing the replacement. Because the newly added write gate returns immediately once _accepting_writes is false, every trace completed in that interval is silently discarded rather than sent through either writer; a writer swap should block or reroute these concurrent writes until the replacement is installed.
Useful? React with 👍 / 👎.
| try: | ||
| self._recreate(reset_buffer=False, flush_writer=False) |
There was a problem hiding this comment.
If Tracer.configure() starts recreating a writer before the /run identity refresh and pauses after constructing it, that operation holds no _post_fork_lock. The refresh can therefore install a writer carrying the new runtime ID here, after which the earlier configuration operation overwrites it with an exporter that captured the snapshot runtime ID during construction; the tracer then keeps reporting that stale ID indefinitely. All writer construction/publication paths need to share the refresh's swap synchronization or validate the runtime ID before publishing.
Useful? React with 👍 / 👎.
emmettbutler
left a comment
emmettbutler
left a comment
There was a problem hiding this comment.
Logic and code organization look appropriate
| if self._shutting_down: | ||
| # Nothing would stop a writer built now, so its native exporter workers would leak. | ||
| return | ||
| # The old writer still flushes what it holds when stopped, but a span written to it |
There was a problem hiding this comment.
I think the explanatory comments here actually make the code harder to read because of how verbose they are. I'd prefer a single docstring that explains the overall approach in one or two sentences.
There was a problem hiding this comment.
Let me revise this (AI slop) :-)
The native trace exporter copies the runtime ID when it is built, so after a MicroVM identity refresh the tracer kept reporting the snapshot's runtime ID. In MicroVMs the tracer now registers for identity refreshes. On refresh it rebuilds the writer the same way the post-fork path does, then stores the tracer metadata again, whether or not the rebuild succeeded. A failed rebuild is retried on the next span start, and a Rust panic from the rebuild is logged instead of aborting the /run request. Before stopping the old writer, the refresh closes it to new spans using the MicroVM-only write gate (_writer_lock, _accepting_writes). The old writer still flushes what it buffered. A span written after that stop no longer restarts it as an orphaned writer. Outside MicroVMs the writer has no lock and nothing changes. The refresh and shutdown() now serialize on _post_fork_lock, so a writer built by a refresh is always the one shutdown stops, and a refresh that starts after shutdown builds nothing. The lock is reentrant because the exit-signal handler can run shutdown() on the refreshing thread; if that handler returns, the refresh stops the writer it just installed. In a MicroVM, shutdown(timeout) waits at most timeout for the lock, because a refresh on another thread can hold it while waiting for a writer lock held by the thread the signal interrupted. A refresh still running after that wait stops the writer it installs. A MicroVM sync-mode write now holds the write gate through its flush, so the refresh cannot shut down the exporter while that flush is sending. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
fc73ae2 to
3d14424
Compare
Stacked PRs:
/runlistener and identity refresh (merged)/runlistener; runtime-metrics runtime-ID tag (part 1 of chore(runtime): refresh runtime metric identity and collector state #19822)Part 1 PRs (#20952, #20953, #20954) can ship first. Part 2 PRs (#20955, #20956, #20957) can ship later.
Description
This PR and #20955 split #19820, so the runtime-ID fix can ship before the discard logic. This PR makes the tracer report the new runtime ID after a MicroVM identity refresh. #20955 drops trace state from before the refresh.
The native trace exporter copies the runtime ID when it's built (
set_runtime_id(get_runtime_id())in_build_base_exporter_builder()), so afterrefresh_identity()the tracer keeps reporting the snapshot's runtime ID until its writer is rebuilt. The root span'sruntime-idtag already readsget_runtime_id()for each span and needs no change.ddtrace/trace/__init__.pyregisterstracer._refresh_runtime_identitywithon_runtime_identity_refresh()inside a MicroVM check.Tracer._refresh_runtime_identity()rebuilds the writer the way the post-fork path does (_recreate(reset_buffer=False, flush_writer=False)), then stores the tracer metadata again, in afinally, so a failed rebuild still records the new runtime ID. A failed rebuild is retried on the next span start. A Rust panic from the rebuild is logged instead of re-raised, because the refresh dispatcher only containsExceptionand the panic would otherwise abort the/runrequest.shutdown()serialize on_post_fork_lock.shutdown()sets_shutting_downunder that lock, so it waits for a refresh in flight and stops the writer that refresh installs, and a refresh that starts after shutdown builds nothing. The lock is now reentrant, because the exit-signal handler can runshutdown()on the thread that is refreshing. If that handler returns, the refresh sees_shutting_downafter_recreate()and stops the writer it just installed, shutting down the exporter of aNativeWriterthat never started.shutdown(timeout)waits at mosttimeout(clamped at 0) for_post_fork_lock, then sets_shutting_downeither way. Without the bound, an exit signal could deadlock: a refresh on another thread holds_post_fork_lockwhile waiting for the old writer's_writer_lock, which the thread running the signal handler holds. A refresh still running after the wait stops the writer it installs, through the same_shutting_downcheck.shutdown()without a timeout still waits as before.The rebuild stops the old writer the normal way, so the old writer still flushes what it buffered. A span finished on another thread after that stop would restart the old writer through the lazy start in
_write_with_client(), leaving a writer that nothing stops. So the refresh first closes the old writer to new spans, using #19820's MicroVM-only write gate:_writer_lockand_accepting_writesonHTTPWriterandNativeWriter, checked inwrite().TraceWriter._stop_accepting_writes()closes the gate without dropping anything. #20955 replaces it withdrop_buffered_traces().Nothing registers
maybe_refresh_identity()until #20954, so merging this PR alone doesn't change runtime behavior.Smaller changes:
NativeWriter.write()takes the write gate through the sync-mode flush, asHTTPWriter.write()does. Otherwise the refresh could close the writer between the buffering and the flush, andrecreate()would shut down the exporter that flush then sends through. fix(tracer): isolate MicroVM identity refresh #19820 keeps the flush outside the gate and instead rechecks_discarded_by_refreshbefore each send; those rechecks belong to fix(tracer): drop trace state from before a MicroVM identity refresh #20955.Testing
tests/tracer/runtime/test_runtime_id.py:test_tracer_microvm_identity_refresh_recreates_exporter_without_fork_side_effects: the refresh rebuilds the writer with the post-fork arguments and stores the metadata, without fork side effects.test_tracer_microvm_identity_refresh_closes_and_flushes_old_writer: the old writer sends its buffer once and shuts its exporter down. A later write neither buffers nor restarts it.test_tracer_microvm_identity_refresh_after_shutdown_does_not_recreate_writertest_tracer_microvm_identity_refresh_keeps_pending_after_recreate_failure: a failed rebuild keeps the pending post-fork rebuild and still stores the metadata.test_tracer_microvm_identity_refresh_schedules_rebuild_after_recreate_failure: a failed rebuild is left pending for the next span.RuntimeErrorandKeyboardInterruptpropagate; a Rust panic doesn't.test_tracer_identity_refresh_contains_native_panic_from_writer_rebuild: a panic from the rebuild doesn't escaperefresh_identity(), and other refresh subscribers still run.test_tracer_microvm_identity_refresh_retries_failed_writer_recreation_on_next_span: after a failed rebuild, the next span rebuilds the writer, and the stored metadata carries the refreshed runtime ID.test_tracer_shutdown_waits_for_in_flight_identity_refresh: shutdown during a refresh stops the writer the refresh installs.test_tracer_identity_refresh_during_shutdown_does_not_recreate_writer: a refresh that starts while shutdown runs builds no writer.test_tracer_shutdown_on_refreshing_thread_stops_replacement_writer: shutdown run on the refreshing thread (the exit-signal case) doesn't deadlock, and the replacement writer is stopped, or has its exporter shut down if it never started.test_tracer_signal_shutdown_does_not_deadlock_with_refresh_waiting_on_writer_lock: shutdown run by a thread that holds the writer lock a refresh is waiting on (the cross-thread exit-signal case) returns after its timeout, and the refresh then stops the writer it installed.test_tracer_microvm_shutdown_accepts_negative_timeouttest_identity_refresh_hook_notifies_global_tracer,test_deferred_global_tracer_registers_identity_refresh_callback,test_deferred_global_tracer_does_not_register_identity_refresh_callback_outside_microvm: the global tracer registers only in a MicroVM, including with deferred initialization.tests/tracer/test_writer.py:test_native_writer_closed_microvm_writer_rejects_writes_without_restarting,test_http_writer_closed_microvm_writer_rejects_writes: a closed writer drops later spans and isn't restarted by them. Spans buffered before the close stay for the final flush.test_native_writer_microvm_close_waits_for_in_flight_sync_flush: closing a sync-mode writer waits for a flush that is sending, and nothing is flushed through the closed writer.test_native_writer_stop_accepting_writes_is_noop_outside_microvm,test_log_writer_stop_accepting_writes_is_nooptests/tracer/test_tracer.py:test_tracer_shutdown_waits_for_in_flight_post_fork_writer_recreation: shutdown waits for a post-fork writer rebuild on another thread and stops the rebuilt writer.test_tracer_shutdown_timeout_does_not_bound_post_fork_wait_outside_microvm: outside a MicroVM, a shutdown timeout doesn't cut that wait short.Each new concurrency test fails without the change it covers.
Local runs (Python 3.14):
scripts/run-tests --venv 128b106 -- tests/tracer/test_tracer.py tests/tracer/test_writer.py tests/tracer/runtime/test_runtime_id.py: 316 passed, 13 skipped, 1 xpassed.main2ad3da71cb:scripts/run-tests -r --venv 128b106 -- tests/tracer/runtime/test_runtime_id.py tests/tracer/test_tracer.py tests/tracer/test_writer.py -k "test_runtime_id or test_writer or shutdown or post_fork": 209 passed, 10 skipped, 1 xpassed.scripts/lint fmtandscripts/lint typingon the changed files: passed.Not done yet: measuring
/runin a MicroVM with the agent stopped (see Risks).Risks
Outside a MicroVM the callback isn't registered and
_writer_lockisNone, sowrite()takes the lock-free path. The one change there:shutdown()now takes_post_fork_lockonce, so a shutdown that races a post-fork writer rebuild on another thread waits for it and then stops the rebuilt writer, instead of leaving it running. The shutdown timeout doesn't bound that wait outside a MicroVM.Inside a MicroVM, once #20954 registers the listener and until #20955 lands:
/runwaits for the old writer's final flush, becausestop()joins the writer thread. That's one send plusexporter.shutdown(), which can take up to 3 s. With a healthy agent that's milliseconds. With an unreachable agent it's the send's retries plus up to 3 s./runare sent under the snapshot's runtime ID, once by each restored instance./runkeep the old rootruntime-idtag and go to the new writer.Tracer.configure(), LLMObs) can still install a writer built for the old runtime ID. In supported usage these rebuilds run at startup, before the snapshot. Remote config, whose AppSec activation callstracer.configure()at runtime, isn't supported in MicroVMs. fix(tracer): drop trace state from before a MicroVM identity refresh #20955 serializes writer swaps with the refresh._recreate()raises, the closed old writer stays installed and drops spans until the next span start retries the rebuild.shutdown(timeout)can take up to about 2 ×timeout: the bounded wait for an in-flight refresh, then the normal flush. If it gives up waiting while a deferred rebuild retry (_ensure_post_fork_writer()) holds the lock, that retry doesn't check_shutting_down, so the writer it installs isn't stopped./run, as it does from__init__and after a fork. The write runs after the rebuild, so it can't skip it.Unchanged from
main, and not specific to the refresh:stop()deadlocks on that writer's_service_lock.Tracer.configure()and the post-fork rebuild have the same exposure.Additional Notes
Each part 2 tip has the same tree as its original PR: #20955 matches #19820, #20956 matches #19821, and #20957 matches #19822. So the fixes made on this PR and on the originals are all in part 2.
Left out on purpose:
/runwait. The telemetry worker'sstop()(fix(telemetry): rebuild the worker on MicroVM identity refresh #20953) also blocks/run, and this code sets no timeout on it, so capping the trace writer alone wouldn't bound/run./run.Reference
🤖 Generated with Claude Code