Skip to content

Resolve received media by mid, not didAddTrack - #1188

Open
xianshijing-lk wants to merge 1 commit into
mainfrom
sxian/mid-based-received-media-resolution
Open

xianshijing-lk wants to merge 1 commit into
mainfrom
sxian/mid-based-received-media-resolution

Conversation

@xianshijing-lk

Copy link
Copy Markdown
Contributor

Extracted from #1112, which bundled this with preallocated receive sections. This is the correctness half only — no preallocation, no behaviour change outside single peer connection mode.

Problem

In single PC mode received media is assigned to the publisher's recvonly m-sections, and the SDK identifies it from libwebrtc's didAddTrack. That callback fires when a transceiver is created. It does not fire when the server binds a track to a section that was already negotiated, changing only the a=msid — which is what the SFU does to a section the client offered before there was media for it, via MediaSectionsRequirement.

When that happens the track is never attached. There is no error and no retry: the subscription simply stays pending.

This is the same defect webrtc-sdk/webrtc#321 patches in libwebrtc, and the reason livekit-client skips up-front recvonly sections on React Native (livekit/client-sdk-js#1984) and rust-sdks synthesizes the event itself (livekit/rust-sdks#1151).

Fix

Browsers do not hit this because blink diffs receiver streams on every setRemoteDescription (RTCPeerConnection::DidModifyTransceivers). This is that diff, at the SDK layer:

  • SignalClient passes the answer's mid_to_track_id through didReceiveAnswer.
  • Transport.receivedMedia(applying:) merges it into a per-connection mapping, then scans _pc.transceivers and reports bound sections the room has not been given yet, deduped by (mid, trackSid).
  • Room.attachReceivedMedia attaches them, calling undispatch when the publication has not arrived yet so a later scan retries.

Keyed by (mid, trackSid) rather than receiver streams because the ObjC RTCRtpReceiver does not expose streams.

The trigger is the scan, not a change in the mapping. That is what makes it self-healing — a section negotiated before it carries a track, or named before its publication arrives, is picked up by a later scan with no bookkeeping at the point of failure.

room_moved is handled explicitly: it rebuilds every publication on the same peer connection, so the dispatch record is dropped or a carried-over section would still read as dispatched and never re-attach.

Relationship to #1112 and webrtc-sdk#321

Testing

  • xcodebuild build -scheme LiveKit -destination 'platform=macOS' — succeeds, no new warnings.
  • swiftlint Sources Tests — clean.
  • LiveKitCoreTests, 588 tests — one failure, droppingTheBinderDetaches, a known pre-existing flake in VideoView code this PR does not touch (passes in isolation in 0.008s).
  • Benchmarked against staging: single PC late-joiner at 1 and 5 publishers, 34 samples, zero failed iterations; dual PC unchanged.

Not covered: a room move in single PC mode is reasoned about but not exercised by a test. Adding one needs a harness that can drive room_moved.

🤖 Generated with Claude Code

In single PC mode received media lands on the publisher's `recvonly`
sections. libwebrtc fires `didAddTrack` only when a transceiver is first
created, never when the server binds a track to a section that was already
negotiated — which is what it does to a section the client offered before
there was media for it. That subscription then stays pending with no error.

Browsers do not hit this because blink diffs receiver streams on every
`setRemoteDescription` (`RTCPeerConnection::DidModifyTransceivers`);
rust-sdks reimplements that diff in `process_remote_track_addition`. This is
the same diff, driven from the server's `mid` → track SID mapping and keyed
by `(mid, trackSid)` because the ObjC `RTCRtpReceiver` does not expose
`streams`.

Scanning the transceivers rather than diffing the mapping is what makes it
self-healing: a section negotiated before it carries a track, or named
before its publication arrives, is picked up by a later scan.

A room move is handled explicitly — it rebuilds every publication on the
same peer connection, so the dispatch record has to be dropped or a section
the server carries over unchanged never re-attaches.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 3 potential issues.

1 flag not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)

Devin Review

Comment on lines +376 to +377
if publisher.singlePCMode {
await attachReceivedMedia(on: publisher, midToTrackSid: midToTrackSid)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Newly negotiated tracks attach twice

When a new receive section has a stream ID, attachReceivedMedia attaches its track alongside the existing didAddTrack callback. Both call addSubscribedMediaTrack, replacing the publication's track and emitting duplicate subscription events.

Learn more

Single-PC peer connections still forward native receiver callbacks through didAddTrack. When a receive section first gains a transceiver and a real stream ID, that callback queues an attachment. The answer path also reports that same section, and addSubscribedMediaTrack always creates a new track and sends subscription events. The two paths therefore replace each other's track and announce the same subscription twice.

Example: A publisher joins with an audio track. The subscriber negotiates a new audio section; its receiver callback queues a subscribe, while the answer scan attaches the same SID. The queued callback attaches it again, leaving two subscribe events for one publication.

Recommended fix: In single-PC mode, route received media exclusively through the MID scan, or share one atomic deduplication gate across the callback and scan before either calls addSubscribedMediaTrack. Keep the native callback path for dual-PC mode.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

///
/// - Returns: Bound sections not yet dispatched, for the caller to attach.
func receivedMedia(applying mapping: [String: Track.Sid]) -> [ReceivedMedia] {
_midToTrackSid.merge(mapping) { _, new in new }

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Resubscriptions stay detached on reused sections

After set(subscribed: false) clears a track, receivedMedia retains its dispatch record across an unbound answer. The old MID mapping and receiver track keep that section present, so resubscribing the same SID cannot reattach it.

Learn more

The mapping is cumulative, but an answer can leave a previously used MID unbound. set(subscribed:) proactively clears the publication when unsubscribing because the transceiver is reused. This scan counts any non-nil receiver track as still bound by the cached MID, even if the new answer no longer binds that SID; _dispatchedSections therefore keeps its prior entry. If the same SID is subscribed again on that section, the scan sees it as already dispatched and the native callback does not fire for an existing transceiver.

Example: Audio SID TR_A arrives on MID 2. Unsubscribe clears its publication; the next answer has no 2 binding, while the receiver object remains. Resubscribe TR_A on MID 2: the cached (2, TR_A) entry prevents attachment.

Recommended fix: Derive current bindings from the negotiated answer's actual media sections, including which MIDs have no a=msid, and remove dispatch records when a binding disappears. Preserve mappings only where the protocol guarantees they remain valid.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

Comment on lines +403 to +405
log("No publication yet for \(media.trackSid), deferring attach", .debug)
await publisher.undispatch(mid: media.mid, trackSid: media.trackSid)
continue

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Late publications leave subscriptions pending

If an answer precedes its publication update, attachReceivedMedia defers the track and clears its dispatch record. didUpdateParticipants never rescans, so the publication remains unsubscribed until another answer arrives.

Learn more

A received answer supplies a MID-to-SID binding, but the matching publication is created from a separate participant update. When the answer arrives first, this branch undoes the dispatch and waits for another scan. didUpdateParticipants creates the publication but never calls receivedMedia; only subsequent answers do. Thus the track remains absent if no later negotiation occurs.

Example: An answer binds MID 3 to TR_B, followed by a participant update announcing TR_B. The first scan defers TR_B; the update creates its publication, but no other answer occurs, so it never subscribes.

Recommended fix: Retry pending MID attachments when participant updates add publications, using the last applied answer's mapping and current transceivers. Ensure retry dispatch remains deduplicated against normal answer scans.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant