Skip to content

feat(telemetry): add the transport seam and LiveKit Cloud destinations - #1480

Open
pblazej wants to merge 3 commits into
mainfrom
blaze/telemetry-stack/2-transport
Open

pblazej wants to merge 3 commits into
mainfrom
blaze/telemetry-stack/2-transport

Conversation

@pblazej

@pblazej pblazej commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Summary

How batches leave the device and where they go. The core classifies every answer itself; the backend contract, row by row with its tests, is in #1484.

Changes

  • TelemetryTransport: moves encoded batches and returns status, headers and body
  • NetTransport over livekit-net (feature net)
  • Destinations: the ingest URL derived from a Room's server URL (LiveKit Cloud only, parsed with WHATWG rules)
  • Credentials: the token's unverified grant and expiry, keyed by project and session
  • LK_TELEMETRY_ENDPOINT points uploads at a local collector
  • SPEC.md: Destination and credentials
  • A server URL that is not on LiveKit Cloud gets a routing key of its own (scheme, host and port), so it never revives or silences a Cloud project's token
  • A Cloud host must be a DNS host name, and every routing key is a valid file name on every platform, because it ends each cached batch's id
  • The core alone reads the Cloud "disabled" answer, matched on the whole body; ExportError has no Disabled variant
  • The native livekit-net client no longer follows a redirect that changes the scheme, so Authorization never reaches a plaintext URL
  • In the crate's first stage: a record carries each attribute key once (the last value wins) and one otel.event.name, and report counts saturate instead of wrapping
Verification

At 3e68d917, from a clean checkout (CI's test workflow also runs on this PR because it targets main; there is no clippy job in CI):

  • cargo fmt --check
  • cargo clippy -p livekit-telemetry --all-targets -- -D warnings, also with --all-features and as a cargo hack --feature-powerset --depth 2 run
  • cargo test -p livekit-telemetry: 56 unit, 0 doc
  • cargo test -p livekit-net --features native, and its cargo hack --feature-powerset --depth 2 clippy run
  • RUSTDOCFLAGS="-D warnings" cargo doc -p livekit-telemetry --no-deps reports only the unresolved link to Telemetry::stats in stats.rs, which main (6473ec17) has too; feat(telemetry): add the pipeline, sessions, spans and exporter #1483 adds the item

@devin-ai-integration devin-ai-integration Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Newer findings are available below. Devin Review posted a newer report on this PR, in addition to the findings presented here.

Devin Review found 5 potential issues.

2 flags not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)

Devin Review

[features]
# Default HTTP transport over the pluggable `livekit-net` client (native backend, or one
# the host registered with `livekit_net::set_http_client`).
net = ["dep:livekit-net"]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Default network transport cannot start

With only net enabled, NetTransport::from_registry() returns None unless the host registers a client. livekit-net has no native backend enabled by default, leaving these builds unable to export telemetry.

Learn more

The net feature activates the dependency but not its native backend. In a build that selects only this crate's net feature, http_client returns None unless an application called set_http_client, because native is disabled by default. NetTransport::from_registry therefore cannot construct the transport promised by this feature.

Example: An application enables livekit-telemetry/net on a native target without registering an HTTP client. from_registry() returns None instead of a built-in client, so no default exporter can send its batches.

Recommended fix: Arrange for the native livekit-net backend and an appropriate TLS backend to be enabled for native default-transport builds, while retaining an explicit registered-client path for host-provided builds. Validate the standalone net feature without another dependency enabling livekit-net/native.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

Comment thread livekit-telemetry/src/destination.rs Outdated
Comment thread livekit-telemetry/src/destination.rs
Comment thread livekit-telemetry/src/transport.rs
Comment thread livekit-telemetry/src/destination.rs
Base automatically changed from blaze/telemetry-stack/1-data-model to main October 20, 2026 13:35
@pblazej
pblazej force-pushed the blaze/telemetry-stack/2-transport branch from e8facef to 2812b2d Compare October 20, 2026 13:35
`TelemetryTransport` moves encoded batches off the device (`NetTransport`
over `livekit-net` behind the `net` feature) and returns status, headers
and body; the core classifies every answer (partial success, 4xx, 413,
401/403, 404, 429/503 `Retry-After` and `RetryInfo`, 5xx, no answer).
Destinations turn a Room's server URL and token into the ingest URL
(LiveKit Cloud hosts only, parsed, never string-matched), read the token's
unverified grant and expiry, and key credentials by project and session.
`LK_TELEMETRY_ENDPOINT` points every upload at a local collector.
A non-Cloud server URL now gets a routing key of its own (scheme, host
and port), so a plaintext or userinfo URL on a Cloud host can no longer
revive that project with its token or silence it. `alive` takes the
session, so a Room without a server waits for its first project instead
of being judged as process data. The test override swaps only a whole
`logs` path segment. The project table drops entries that no live
session or cached batch needs, except a project known to receive nothing
(not Cloud, no grant, 404, disabled). A Cloud host must be a DNS
hostname, and every other key escapes its host and port, so a key is a
valid file name on every platform: it ends each cached batch's id. The
`Server` and `set` docs describe both kinds of key. A Room's first
project is recorded even when that project receives nothing, so what the
Room captured before its first server is dropped with it instead of
moving to a later project.

The Cloud "disabled" phrase is matched on the whole error body, and only
the logged reason is cut, to a named limit. `ExportError` loses
`Disabled`, because a transport must not interpret the answer.
`NetTransport::new` is crate-private, the public exports are listed, and
`ExportResponse` and `Token` get `From` conversions. Doc comments now
name the real classifier, the real body types and the real redirect
behavior, and every public field has a doc line. The README names the
transport and is now the crate doc, and a changeset records the stage.
The `Retry-After` date parser checks the day against the length of the
month, leap years included, and the retry verdict names 500 with
`RetryInfo` as the only retryable 500. The native `livekit-net` HTTP
client keeps its limit of 10 redirects but no longer follows one that
changes the scheme, so `Authorization` never reaches a plaintext URL on
the same host and port; a `livekit-net` changeset records it.

In the code the crate first shipped with, the queue-full warning is
logged after the queue lock, a pipeline attribute can no longer add a
second `otel.event.name`, a record ships each attribute key once (of a
key the caller repeats, the last value wins; the session's keys still
replace the caller's), and report counts saturate at `i64::MAX` instead
of wrapping. Finished spans wait in a `VecDeque`, `reserved` is
`is_reserved`, the severity text is `Severity::as_str`, and
`Spans::dropped` and `SpanKind` are crate-private. Every exported field
and variant has a doc line, the limits and the size of a number are
named constants, the duplicated wire values and the test hooks say why
they exist, and the docs and SPEC state the real log floor, trace id,
failure count and `queue_full` meaning.
@pblazej
pblazej force-pushed the blaze/telemetry-stack/2-transport branch from 2812b2d to 3e68d91 Compare October 23, 2026 06:09
@pblazej
pblazej requested a review from jhugman as a code owner October 23, 2026 06:09

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 1 new potential issue.

1 flag not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)

Devin Review

let mut safe = String::with_capacity(text.len());
for byte in text.bytes() {
match byte {
b'a'..=b'z' | b'0'..=b'9' | b'.' | b'-' | b'_' => safe.push(byte as char),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Absolute hostnames break Windows cache filenames

For ws://localhost., file_safe leaves the routing key ending in a dot. Cached batch filenames end with that key, so Windows cannot store those batches.

Learn more

A trailing dot is valid in an absolute DNS hostname. Server::parse preserves it in a non-Cloud routing key, and the key ends the cached batch filename. Windows does not permit file names ending with a period, despite permitting periods elsewhere.

Example: set("ws://localhost.", token, "room") returns ws+localhost.. A batch named with that suffix cannot be written to the Windows cache.

Recommended fix: Escape or otherwise encode a terminal period in routing keys while keeping their server identity unambiguous. Test terminal periods as well as reserved characters in the filename check.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

internal to tag changes that don't require changelog documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant