Repository navigation
Conversation
Pre-encoded H.265 could only be ingested as Annex-B, and a single-layer pre-encoded H.265 track could not be published at all. - livekit-capture: `LengthPrefixedAccessUnitParser` (was the test-only AVC parser) handles H.264 (avc3) and H.265 (hev1) length-prefixed streams, with an opt-in single-slice flush that completes an access unit as soon as its slice arrives instead of a frame later. - livekit-capture: `access_unit_from_h265_hvcc` mirrors `access_unit_from_h264_avc`; the GStreamer source accepts `hev1` caps. - livekit-capture: `frame_metadata_from_sei` reads the LKTS packet-trailer SEI (the Go SDK `ReaderTrackWithPacketTrailer` convention) for the pump's frame metadata callback. - livekit: announce the codec of single-layer video tracks in AddTrackRequest. Without it the SFU answered with its default (VP8), so a pre-encoded H.265 track negotiated VP8 and the pass-through dropped every access unit. The Go SDK always sends it. - local_video: `encoded_publisher` publishes a length-prefixed stream received over TCP. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
libwebrtc paces packets at 2.5x its bandwidth estimate, which starts at
1 Mbps for camera tracks. A pre-encoded stream arrives at the bitrate its
upstream encoder chose and cannot slow down, so for the first ~9 s frames
queued in the pacer: up to ~420 ms (Rust) / ~570 ms (C++) at p99 for a
1080p30 ~5 Mbps H.265 stream, and 16-50 ms after each keyframe in some
runs once the estimate settled.
- webrtc-sys: read field trials from LIVEKIT_WEBRTC_FIELD_TRIALS
("Name/Value/" format), taking precedence over the SDK's own. With
"WebRTC-Video-Pacing/factor:10/" startup p99 drops to ~1.5-2.2 ms.
- livekit: exempt pre-encoded tracks from the 1 Mbps cap on
x-google-start-bitrate, like screen share (halves the startup penalty
on its own).
- local_video: `encoded_publisher --stats` logs the pacer delay per
packet (totalPacketSendDelay / packetsSent), target bitrate and
bandwidth estimate.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- livekit-capture: the length-prefixed parser copied each access unit twice and rescanned it byte by byte for start codes to classify the frame (~205 us per 1080p access unit). It now converts in place during one copy and classifies from the NAL ranges it already has (~2.5 us). find_start_code jumps between 0x01 bytes instead of matching at every offset, which speeds up every Annex-B path; an exhaustive test pins it to the old behaviour. Steady-state publish latency p50 1.21 -> 0.81 ms. - livekit: publish pre-encoded video without simulcast. FFI clients such as the C++ SDK leave simulcast on by default; the track then announced 360p/180p layers whose senders forwarded copies of the full stream, requested keyframes on almost every frame, and competed with the real layer in the pacer. - local_video: `encoded_publisher` counts keyframe requests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
There was a problem hiding this comment.
Devin Review found 5 potential issues.
2 flags not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)
| if let Some(split_at) = self.split_index()? { | ||
| return self.take_access_unit(split_at); | ||
| } | ||
| if at_eof && !self.nal_ranges.is_empty() { |
There was a problem hiding this comment.
🟡 Trailing AUD becomes a bogus video frame
When an encoded stream ends with a trailing AUD, flush emits it as a delta frame without a picture. The pump forwards that frame to subscribers.
(Refers to this code)
Learn more
A length-prefixed NAL stream can end just after an access unit delimiter (AUD), which starts the next access unit but contains no slice. The new public parser treats any remaining complete NAL as a final access unit. EncodedVideoPump::run forwards it once it has seen the initial keyframe. This produces an extra video frame without picture data even when all input pictures were valid.
Example: With single-slice flush, push VPS, SPS, PPS, IDR, AUD, drain the IDR, then call flush() at EOF. The parser emits the AUD as a delta frame; the expected result is no remaining picture.
Recommended fix: Before flushing a pending unit, require a VCL NAL in its ranges and discard a non-VCL-only tail. Cover a trailing AUD with and without single-slice flush.
Was this helpful? React with 👍 or 👎 to provide feedback.
| impl TcpLengthPrefixedSource { | ||
| fn stream(&mut self) -> Result<&mut TcpStream, SourceError> { | ||
| if self.stream.is_none() { | ||
| let stream = TcpStream::connect(&self.addr).map_err(SourceError::new)?; |
There was a problem hiding this comment.
🟡 Publisher hangs during encoder connection
If the encoder address blackholes connections, TcpStream::connect blocks without checking the stop handle. Ctrl-C then waits for the pump thread until TCP times out.
Learn more
The pump opens the TCP socket on a dedicated blocking thread. A connect to a reachable address succeeds or fails promptly, but a blackholed address can keep the OS connect call blocked for much longer. main stops the pump on Ctrl-C and then awaits its thread; the stop flag cannot interrupt a blocking connect.
Example: Set --src to a routed address that silently drops SYN packets, then press Ctrl-C. The process waits for the OS connection timeout rather than exiting promptly.
Recommended fix: Resolve and connect with a short TcpStream::connect_timeout per attempt, checking PumpStop between attempts, or use a cancellable nonblocking connection process.
Was this helpful? React with 👍 or 👎 to provide feedback.
| if args.stats { | ||
| tokio::spawn(log_sender_stats(track)); | ||
| } |
There was a problem hiding this comment.
🟡 Stats task outlives the published track
With --stats, tokio::spawn leaves an endless task holding the track after the pump stops. Neither cancellation nor joining precedes room shutdown.
Learn more
The stats loop in log_sender_stats has no exit condition. main discards its join handle and closes the room after the pump exits, while the task still holds a clone of the local track. The root AGENTS.md requires each spawned task to have a named owner and deterministic cancellation and join.
Example: Start the publisher with --stats, let the encoder stream end, and observe that the stats loop still exists when room.close() begins; only the Tokio runtime's subsequent shutdown stops it.
Recommended fix: Keep the stats task's JoinHandle in main, signal it to stop when the pump ends, and await its completion before closing the room.
Was this helpful? React with 👍 or 👎 to provide feedback.
| info!("Connected to encoder at {}", self.addr); | ||
| self.stream = Some(stream); | ||
| } | ||
| Ok(self.stream.as_mut().unwrap()) |
There was a problem hiding this comment.
🟡 Socket setup violates the no-unwrap rule
The new stream method uses unwrap outside tests, despite the repository's explicit rule to avoid it. An invariant message is required if unwrapping cannot be avoided.
| Ok(self.stream.as_mut().unwrap()) | |
| Ok(self.stream.as_mut().expect("TCP stream initialized above")) |
Was this helpful? React with 👍 or 👎 to provide feedback.
| } | ||
|
|
||
| fn now_us() -> i64 { | ||
| SystemTime::now().duration_since(UNIX_EPOCH).unwrap().as_micros() as i64 |
There was a problem hiding this comment.
🟡 Pre-epoch clock panics the publisher
If the system clock precedes 1970, now_us panics while timestamping a frame. Its production unwrap also violates the repository's no-unwrap rule.
Learn more
SystemTime::duration_since(UNIX_EPOCH) returns an error for dates before 1970. next_access_unit stamps every emitted frame with now_us, so a misconfigured clock stops publishing through a pump-thread panic. The root AGENTS.md requires avoiding unwrap outside tests; the existing wall_clock_now_us shows a fallible clock-read pattern.
Example: If the host clock reads 1969-12-31, the first completed frame triggers a panic instead of being timestamped safely or returning an error.
Recommended fix: Handle duration_since without unwrap and define a safe nonzero fallback, as wall_clock_now_us does.
Was this helpful? React with 👍 or 👎 to provide feedback.
|
The main motivation here is to enable the H265 pre-encoded ingest path with length prefixed NAL units more or less trying to mirror Livekit Go SDK pre-encoded publish path Along the way, benchmarking the changes against a Go publisher client, I found that there was differences in the pacer logic in libwebrtc and some smaller gains in the publish path that could be made. I've addressed those here in a separate commits so they can easily be excluded if risky This PR can probably be broken down but I chose to put them all in one as they go hand in hand for testing/verification. Happy to break this down into a smaller piece or whatever is preferable |
Publishing pre-encoded H.265 from hardware encoder bridges, which can deliver 4-byte length-prefixed NAL units (
hev1). One commit per stage, each with its own changeset:1. Publish length-prefixed pre-encoded H.265
livekit-capture:LengthPrefixedAccessUnitParser(previously test-only, H.264 only) handles H.264 and H.265, with an opt-in single-slice flush that completes a picture as soon as its slice arrives instead of a frame later. Addsaccess_unit_from_h265_hvcc,hev1caps in the GStreamer source, andframe_metadata_from_seifor the LKTS packet-trailer SEI (the Go SDK'sReaderTrackWithPacketTrailerconvention).livekit: single-layer video tracks announce their codec insimulcast_codecs, as the Go SDK does.local_video: newencoded_publisherexample that publishes a length-prefixed stream received over TCP.2. Let pre-encoded tracks leave the pacer promptly
A pre-encoded stream arrives at its encoder's bitrate, but the starting estimate is capped at 1 Mbps, so packets queued in the pacer for ~8 s after publishing (~510 ms per packet at the peak for 1080p30 at 4 Mbps).
webrtc-sys: read field trials fromLIVEKIT_WEBRTC_FIELD_TRIALS. WithWebRTC-Video-Pacing/factor:10/the startup delay drops to ~4 ms, then ~0.livekit: pre-encoded tracks are exempt from the starting-estimate cap, like screen share (peak ~510 ms → ~210–440 ms on its own).encoded_publisher --statslogs codec, frames sent and pacer delay from standard WebRTC stats.3. Trim the publish path
find_start_codealso skips between0x01bytes, which speeds up every Annex-B path (an exhaustive test pins it to the old behaviour).Testing
cargo test -p livekit-capture --features source-gstreamer(53 pass),cargo test -p livekit --lib,cargo test -p webrtc-sys --lib.gst-launch-1.0 videotestsrc ! nvcudah265enc ! h265parse config-interval=-1 ! video/x-h265,stream-format=hev1,alignment=au ! tcpserversinkintoencoded_publisher --single-slice-flush --statsagainstlivekit-server --dev. Cross-checked end to end with a packet capture and a subscriber: steady-state publish latency p50 0.84 ms (Go SDK: 0.49 ms).Notes
🤖 Generated with Claude Code