Export Workers user spans over OTLP by danlapid · Pull Request #7686 · cloudflare/workerd

danlapid · 2026-10-10T16:45:45Z

Workers user spans reach a tracing backend today only through Streaming Tail Workers: the Workers Observability tail worker rebuilds them in JavaScript from tail events. Spans built that way are cut off when a Durable Object hands off work, carry coarse I/O-clock timestamps, and are sampled only after they already exist.

This gives the runtime what it needs to export user spans itself, as OTLP, the way other Cloudflare services already do:

an OTLP encoder for spans, written in Rust, behind the existing span submitter interface;
a tracer hook that lets the embedder open each invocation's root span, so the invocation and all of its user spans form one exported trace;
precise timestamps for user spans, which no longer reach user code directly.

workerd can also export on its own: a new tracing.otlp setting sends each invocation's spans to an OTLP/HTTP collector such as a local Jaeger. That makes Workers traces visible in local development without a tail worker.

Nothing changes unless an embedder installs the hook or the setting is configured.

github-actions · 2026-10-16T14:11:41Z

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

wperron · 2026-10-16T14:30:20Z

I have read the CLA Document and I hereby sign the CLA

github-actions · 2026-10-27T14:45:29Z

The generated output of @cloudflare/workers-types matches the snapshot in types/generated-snapshot 🎉

clanky-the-clanker

Superseded by the review of 91d0101e.

This review, of 4f06a7ee

🛑 Significant concerns — 1 BLOCKING: UserSpanObserver::getTime() now uses the precise system clock for every user span, including the existing streaming-tail path. Tail workers, which are user code, now receive precise timestamps, and their spans mix a precise start time with a coarse I/O-clock end time. There are also 4 WARNINGs, among them an ungated change to the fetch spans tail workers receive and a Durable Object id tag that is never set.

Findings: 1 blocking · 4 warnings · 1 info · 1 suggestion (4 are inline comments on the diff)

_{The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.}

🔧 Fix with your agent

Paste the following into your coding agent to investigate and fix these issues.

Fix the following code review issues found in pull request #7686.

Before making changes, review each issue and present a brief summary table:
- For each finding, state whether you agree, disagree, or need clarification
- If you disagree (e.g., the fix would require disproportionate changes
  for minimal ROI, or the finding is factually incorrect), explain why
- If you need to clarify something before deciding, ask those questions
- Then share your plan for which issues to tackle and in what order

## BLOCKING (1)

### [Security] `UserSpanObserver::getTime()` now uses `systemPreciseCalendarClock()` for every user span, including the existing streaming-tail path. Tail workers (user code) receive precise start and exception timestamps, which gets around the coarse I/O clock. Tail spans also combine a precise start with an I/O-clock end set by `adjustSpanTime()`, so durations can be zero or negative. The change is not gated.
- **File:** src/workerd/io/tracer.c++:838

## WARNING (4)

### [Code Quality] The internal-binding fetch filter drops `fetch` spans for D1, Images, AI and Vectorize hosts from every streaming tail worker, not only when OTLP export is enabled. Because it matches by host only, a user's own fetches to those hosts lose their spans as well.
- **File:** src/workerd/api/http.c++:1922

### [Code Quality] `OtlpServiceInfo.durableObjectId` is never set, so exported Durable Object spans never carry the `cloudflare.durable_object.id` tag that the header documents.
- **File:** src/workerd/server/factory/worker-factory.c++:1128

### [Code Quality] `exportSpans()` starts a new POST task for every batch, with no limit on how many run at once and no timeout. When the collector is slow or down, memory grows without limit and an ERROR is logged for every batch.
- **File:** src/workerd/server/factory/worker-factory-otlp.c++:27

### [Documentation] `docs/async-trace.md` still says user spans use the Spectre-safe clock, but `UserSpanObserver::getTime()` now uses the precise system clock.
- **File:** docs/async-trace.md:231

## INFO (1)

### [Documentation] The new `tracing.otlp` setting is described only in the `workerd.capnp` schema comments. A short guide, such as how to point a local workerd at Jaeger, would help users find it.

## SUGGESTION (1)

### [Code Quality] Rust unit tests are written as inline `mod tests` blocks in `buffer.rs` and `rules.rs`. The repository convention puts them in `buffer-test.rs` and `rules-test.rs` beside the module.
- **File:** src/rust/otlp/buffer.rs:309

📋 Code Quality: ⚠️ 3 warnings · 1 suggestion

An ungated change to tail-visible fetch spans, a Durable Object id tag that is never set, unbounded in-flight OTLP exports, and inline Rust tests.

Recommendation: 📝 Minor Comments

[WARNING] The internal-binding fetch filter drops fetch spans for D1, Images, AI and Vectorize hosts from every streaming tail worker, not only when OTLP export is enabled. Because it matches by host only, a user's own fetches to those hosts lose their spans as well. (src/workerd/api/http.c++:1922)
[WARNING] OtlpServiceInfo.durableObjectId is never set, so exported Durable Object spans never carry the cloudflare.durable_object.id tag that the header documents. (src/workerd/server/factory/worker-factory.c++:1128)
[WARNING] exportSpans() starts a new POST task for every batch, with no limit on how many run at once and no timeout. When the collector is slow or down, memory grows without limit and an ERROR is logged for every batch. (src/workerd/server/factory/worker-factory-otlp.c++:27)
[SUGGESTION] Rust unit tests are written as inline mod tests blocks in buffer.rs and rules.rs. The repository convention puts them in buffer-test.rs and rules-test.rs beside the module. (src/rust/otlp/buffer.rs:309)

View full review

Internal-binding fetch filter applies to everyone (src/workerd/api/http.c++:1616-1629, 1922)

fetchImplNoOutputLockAttempt now passes OmitUserSpan(isInternalBindingFetch(...)) for every worker, whether or not OTLP export is enabled. Every streaming tail worker stops receiving fetch spans for these hosts. The updated D1 and Images instrumentation tests confirm this. Matching is by host only, so a user's own fetch() to js.images.cloudflare.com or to http://d1/ also loses its span. Commit 837539b explains why the filter exists. Still, this is a visible change to tail output with no gate, and it contradicts the description's "nothing changes" claim. Consider gating it on OTLP export being active, or recording the compatibility decision explicitly.

Durable Object id is never exported (src/workerd/server/factory/worker-factory.c++:1128-1131)

The OtlpServiceInfo initializer sets only .serviceName and .entrypoint. The durableObjectId computed at lines 1086-1096 is moved into the Trace, so makeOtlpIdentityTags() never emits cloudflare.durable_object.id, although the doc comment in otlp-span-submitter.h promises it on every span.

Unbounded in-flight exports (src/workerd/server/factory/worker-factory-otlp.c++:25-49)

Each batch adds a TaskSet task that creates a new HTTP client and POSTs up to about 1 MiB. Nothing caps concurrency, and there is no timeout or drop policy. When the collector is slow or hung, tasks and their request bodies pile up for as long as the server runs. taskFailed also logs an ERROR for every failed batch. Consider capping the number of in-flight exports, dropping batches beyond the cap with spansDropped(), adding a timeout, and rate-limiting the log.

Inline Rust tests (src/rust/otlp/buffer.rs:309, src/rust/otlp/rules.rs:84)

The repository convention puts Rust unit tests in <module>-test.rs, never in an inline mod tests. wd_rust_crate already splits *-test.rs files into the test target.

📝 Documentation: ⚠️ 1 warning · 1 info

The timing description in docs/async-trace.md is now out of date, and tracing.otlp is not documented.

Recommendation: 📝 Minor Comments

[WARNING] docs/async-trace.md still says user spans use the Spectre-safe clock, but UserSpanObserver::getTime() now uses the precise system clock. (docs/async-trace.md:231)
[INFO] The new tracing.otlp setting is described only in the workerd.capnp schema comments. A short guide, such as how to point a local workerd at Jaeger, would help users find it.

View full review

docs/async-trace.md:231-234 says user-facing spans use the Spectre-safe clock. With this change, UserSpanObserver::getTime() uses the precise clock. Once the clock question in the security section is settled, update this text to match.
The new tracing.otlp setting has no guide in docs/. The schema comments in workerd.capnp are its only documentation.

🔒 Security: 🛑 1 blocking

The precise clock change also applies to the existing streaming-tail path, so tail workers receive precise timestamps.

Recommendation: 🔴 Serious Concerns

[BLOCKING] UserSpanObserver::getTime() now uses systemPreciseCalendarClock() for every user span, including the existing streaming-tail path. Tail workers (user code) receive precise start and exception timestamps, which gets around the coarse I/O clock. Tail spans also combine a precise start with an I/O-clock end set by adjustSpanTime(), so durations can be zero or negative. The change is not gated. (src/workerd/io/tracer.c++:838)

View full review

Precise timestamps reach tail workers, and tail spans mix two clocks (src/workerd/io/tracer.c++:837-840)

UserSpanObserver::getTime() used to return IoContext::current().now(). It now returns kj::systemPreciseCalendarClock().now(). This observer is not specific to OTLP. worker-factory.c++ installs it with a SequentialSpanSubmitter for every streaming tail worker, and the production embedder's tail path uses it too.

SpanBuilder takes its start time from observer->getTime() (trace.c++:1965). WorkerTracer::addSpanOpen then forwards that start time unchanged to the tail stream (tracer.c++:213-234). Exception timestamps (trace.c++:2089) are handled the same way.
A tail worker is user code. A worker can bracket any piece of its own code with enterSpan() and read nanosecond start times in its tail worker. That gets around the coarse I/O clock that exists to deny Workers a high-resolution timer. It also contradicts the description's statement that precise timestamps "no longer reach user code directly".
WorkerTracer::addSpanClose still calls adjustSpanTime(), which overwrites endTime with context.now() on the I/O clock (tracer.c++:534-560). Tail spans therefore have a precise start and a coarse end. The I/O clock does not advance while JavaScript runs, so the end time can be earlier than the start time. Existing tail consumers would then see zero or negative durations.
This change is not gated. It takes effect whether or not the OTLP hook or tracing.otlp is configured, which also contradicts "Nothing changes unless an embedder installs the hook or the setting is configured".

Fix: use the precise clock only for observers created by the OTLP export path, for example with a flag or a dedicated observer, and keep IoContext::now() on the tail-stream path.

Apart from this, the untrusted inputs into the Rust encoder are handled safely: UTF-8 is converted lossily, the open-span and batch buffers have size limits, and query strings are redacted by rebuilding url.full.

✅ All other checks passed

📖 AGENTS.md: ✅ LGTM

The addition to src/rust/AGENTS.md is consistent with the rest of that file.

⚡ Performance: ✅ LGTM

The issue of unbounded in-flight OTLP exports is covered in the code quality section. src/rust/otlp/buffer.rs limits open spans to MAX_OPEN_SPANS and limits batches by span count and by bytes.

_{Reviewed commit: 4f06a7ee}

clanky-the-clanker

Superseded by the review of 855b0965.

This review, of 91d0101e

🛑 Significant concerns — 1 BLOCKING: UserSpanObserver::getTime() now uses the precise system clock for every user span, including the existing streaming-tail path. Tail workers, which are user code, now receive precise timestamps, and their spans mix a precise start time with a coarse I/O-clock end time. There are also 4 WARNINGs, among them an ungated change to the fetch spans tail workers receive and a Durable Object id tag that is never set.

Findings: 1 blocking · 4 warnings · 1 info · 1 suggestion

_{The decision is advisory: this review is a comment and never approves, blocks or requests changes on the pull request.}

🔧 Fix with your agent

Paste the following into your coding agent to investigate and fix these issues.

Fix the following code review issues found in pull request #7686.

Before making changes, review each issue and present a brief summary table:
- For each finding, state whether you agree, disagree, or need clarification
- If you disagree (e.g., the fix would require disproportionate changes
  for minimal ROI, or the finding is factually incorrect), explain why
- If you need to clarify something before deciding, ask those questions
- Then share your plan for which issues to tackle and in what order

## BLOCKING (1)

### [Security] `UserSpanObserver::getTime()` now uses `systemPreciseCalendarClock()` for every user span, including the existing streaming-tail path. Tail workers (user code) receive precise start and exception timestamps, which gets around the coarse I/O clock. Tail spans also combine a precise start with an I/O-clock end set by `adjustSpanTime()`, so durations can be zero or negative. The change is not gated.
- **File:** src/workerd/io/tracer.c++:838

## WARNING (4)

### [Code Quality] The internal-binding fetch filter drops `fetch` spans for D1, Images, AI and Vectorize hosts from every streaming tail worker, not only when OTLP export is enabled. Because it matches by host only, a user's own fetches to those hosts lose their spans as well.
- **File:** src/workerd/api/http.c++:1922

### [Code Quality] `OtlpServiceInfo.durableObjectId` is never set, so exported Durable Object spans never carry the `cloudflare.durable_object.id` tag that the header documents.
- **File:** src/workerd/server/factory/worker-factory.c++:1128

### [Code Quality] `exportSpans()` starts a new POST task for every batch, with no limit on how many run at once and no timeout. When the collector is slow or down, memory grows without limit and an ERROR is logged for every batch.
- **File:** src/workerd/server/factory/worker-factory-otlp.c++:27

### [Documentation] `docs/async-trace.md` still says user spans use the Spectre-safe clock, but `UserSpanObserver::getTime()` now uses the precise system clock.
- **File:** docs/async-trace.md:231

## INFO (1)

### [Documentation] The new `tracing.otlp` setting is described only in the `workerd.capnp` schema comments. A short guide, such as how to point a local workerd at Jaeger, would help users find it.

## SUGGESTION (1)

### [Code Quality] Rust unit tests are written as inline `mod tests` blocks in `buffer.rs` and `rules.rs`. The repository convention puts them in `buffer-test.rs` and `rules-test.rs` beside the module.
- **File:** src/rust/otlp/buffer.rs:309

📋 Code Quality: ⚠️ 3 warnings · 1 suggestion

An ungated change to tail-visible fetch spans, a Durable Object id tag that is never set, unbounded in-flight OTLP exports, and inline Rust tests.

Recommendation: 📝 Minor Comments

[WARNING] The internal-binding fetch filter drops fetch spans for D1, Images, AI and Vectorize hosts from every streaming tail worker, not only when OTLP export is enabled. Because it matches by host only, a user's own fetches to those hosts lose their spans as well. (src/workerd/api/http.c++:1922)
[WARNING] OtlpServiceInfo.durableObjectId is never set, so exported Durable Object spans never carry the cloudflare.durable_object.id tag that the header documents. (src/workerd/server/factory/worker-factory.c++:1128)
[WARNING] exportSpans() starts a new POST task for every batch, with no limit on how many run at once and no timeout. When the collector is slow or down, memory grows without limit and an ERROR is logged for every batch. (src/workerd/server/factory/worker-factory-otlp.c++:27)
[SUGGESTION] Rust unit tests are written as inline mod tests blocks in buffer.rs and rules.rs. The repository convention puts them in buffer-test.rs and rules-test.rs beside the module. (src/rust/otlp/buffer.rs:309)