Repository navigation
macOS: convert VideoToolbox frames instead of dropping blocked output - #8
Merged
Merged
Conversation
…utput On macOS VLC decodes with VideoToolbox, so icop received opaque CVPX frames. With no CVPX backend it fell back to image conversion: blocked frames were dropped (black screen instead of blur/warning), the debug overlay could not render, and synchronous analysis held decoder buffers until VideoToolbox stalled (pic_holder_wait timed out). Decline CVPX input in Open so VLC's chain filter inserts the cvpx converter and reopens icop with I420, which runs the normal CPU path. Only decline when the cvpx module exists; otherwise keep the opaque fallback so icop is never dropped from the chain (fail-closed). Verified with VLC 3.0.23 on Apple Silicon: blur and overlay render, no decoder stalls or dropped frames. The macOS smoke test now fails if blocked frames would be dropped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every ONNX session was pinned to one intra-op thread, which assumes one CPU worker per core. GPU providers (CoreML on macOS) and hardware backends run a single worker, so marqo took ~120 ms per inference on one thread. With the automatic stride (every 3rd frame) a 720p30 video needed more than a second of inference per second and fell steadily behind: 380 late frames, median 3.5 s, in 30 s. The plugin now exports NSFW_ONNX_INTRA_OP_THREADS = cores / workers (1-4) unless the user set it, and icop_core applies it to the session. Same file afterwards: 5 late frames, median 37 ms, no decoder stalls. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Owner
Author
|
Added 446b611: several ONNX threads when icop runs a single detector worker. Every ONNX session was pinned to
marqo ORT benchmark: 1 thread 122 ms → 4 threads 37 ms. Moving marqo onto CoreML (fixed shape + MLProgram, 591/604 nodes) gave no speedup over the multi-threaded CPU, so that isn't included. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #7. Retarget to
mainafter #7 merges.Problem
VLC on macOS decodes with VideoToolbox, so icop received opaque
CVPXframes. There is no CVPX backend, so icop used the opaque fallback:pic_holder_wait timed out, many "picture is too late").Fix
nsfw_backend_opennow declines CVPX input. VLC'schainfilter then inserts itscvpxconverter and reopens icop withI420, which runs the normal CPU path: async worker, block styles and overlay. This uses VLC's existing fallback (filter_chain_AppendInner→chain→BuildFilterChain) and doesn't depend on any private VLC structs.It is fail-closed. icop only declines when
module_exists("cvpx")is true. Without that converter VLC would drop icop from the chain and play unfiltered, so in that case the old opaque fallback is kept.The macOS smoke test now fails if the log says blocked frames would be dropped.
Verification (VLC 3.0.23, Apple Silicon, 1080p clip, hardware decoding on)
macos_vlc_smoke_test.shpasses on the installed build and logsdeclining VideoToolbox chroma CVPN→video backend=cpu input=I420.🤖 Generated with Claude Code