Skip to content

Add LengthPrefixedAccessUnitParser for length-prefixed H.264/H.265 - #301

Draft
haffi96 wants to merge 1 commit into
livekit:mainfrom
haffi96:feat/h265-pre-encoded-length-prefixed
Draft

haffi96 wants to merge 1 commit into
livekit:mainfrom
haffi96:feat/h265-pre-encoded-length-prefixed

Conversation

@haffi96

@haffi96 haffi96 commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

EncodedVideoSource takes Annex-B access units, but hardware encoder bridges can commonly deliver 4-byte length-prefixed NAL units (avc3 / hev1). This adds a parser that turns such a stream, in any chunking (e.g. straight from a TCP socket), into ready-to-capture EncodedVideoSource::Frames.

  • Converts to Annex-B in place during the single copy out of its buffer (~2 µs per 1080p access unit).
  • Sets is_keyframe with the same rules as livekit-capture (H.265 needs VPS/SPS/PPS with the IDR).
  • single_slice_flush completes an access unit as soon as its slice arrives instead of when the next one begins, which otherwise holds every picture for a frame interval.
  • frame_metadata_from_sei fills Frame::metadata from an LKTS packet-trailer SEI, the convention used by the LiveKit Go SDK.
LengthPrefixedAccessUnitParser::Options options;
options.single_slice_flush = true;
LengthPrefixedAccessUnitParser parser(VideoCodec::H265, options);
parser.push(buf, n);
while (auto frame = parser.next()) {
  source->captureFrame(*frame);
}

This is a C++ port of the LengthPrefixedAccessUnitParser added to livekit-capture in livekit/rust-sdks#1579, not a wrapper over FFI, so it works with the current Rust SDK and needs no new FFI messages; the cost is keeping the two parsers in sync. If a single implementation is preferred, a Rust-side source that takes raw length-prefixed bytes over FFI would be the follow-up. That PR also publishes pre-encoded video as a single layer, which matters for C++ because TrackPublishOptions::simulcast defaults to on.

Testing

  • 10 new unit tests (LengthPrefixedAccessUnitParser*), including chunked-vs-one-shot equivalence and emulation-prevention handling in the SEI.
  • Manual: published a 1080p30 hev1 stream from GStreamer over TCP; steady-state publish latency p50 0.83 ms.
  • Also verified with a GStreamer camera pipeline that sends length-prefixed H.265 with SEI timestamp metadata over TCP, published through this parser (single_slice_flush, frame_metadata_from_sei) to livekit-server --dev and LiveKit Cloud; every frame carried its SEI metadata and the subscriber received 720p30 with the user timestamp and frame ID intact.
  • Unrelated: AudioProcessingModuleTest.AGCAttenuatesLoudSpeech fails locally both with and without this change.

Created with Claude Opus 5.5 in Claude Code.

@haffi96
haffi96 requested review from a team and xianshijing-lk as code owners October 27, 2026 09:15

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 5 potential issues.

1 flag not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)

Devin Review

Comment thread src/length_prefixed_access_unit_parser.cpp Outdated
Comment thread src/length_prefixed_access_unit_parser.cpp
Comment thread src/length_prefixed_access_unit_parser.cpp
Comment thread include/livekit/length_prefixed_access_unit_parser.h
Comment thread src/length_prefixed_access_unit_parser.cpp
@haffi96
haffi96 marked this pull request as draft October 27, 2026 09:20
@haffi96

haffi96 commented Oct 27, 2026

Copy link
Copy Markdown
Contributor Author

The main motivation here is to enable the H265 pre-encoded ingest path with length prefixed NAL units more or less trying to mirror Livekit Go SDK pre-encoded publish path

See rust PR livekit/rust-sdks#1579

EncodedVideoSource takes Annex-B access units, but hardware encoder
bridges commonly deliver 4-byte length-prefixed NAL units (avc3/hev1).
The parser accepts any chunking of such a stream and returns
EncodedVideoSource::Frame values: Annex-B (converted in place during the
single copy), keyframe flag, and optional packet-trailer metadata from
an LKTS SEI (the LiveKit Go SDK convention).

single_slice_flush completes an access unit as soon as its slice
arrives instead of when the next one begins, which otherwise holds every
picture for a frame interval.

Mirrors livekit-capture's LengthPrefixedAccessUnitParser.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@haffi96
haffi96 force-pushed the feat/h265-pre-encoded-length-prefixed branch from b3e92e8 to 4382a5b Compare October 30, 2026 09:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant