Skip to content

macOS: convert VideoToolbox frames instead of dropping blocked output - #8

Merged
asayed18 merged 2 commits into
mainfrom
claude/macos-cvpx-backend
Oct 1, 2026
Merged

asayed18 merged 2 commits into
mainfrom
claude/macos-cvpx-backend

Conversation

@asayed18

Copy link
Copy Markdown
Owner

Stacked on #7. Retarget to main after #7 merges.

Problem

VLC on macOS decodes with VideoToolbox, so icop received opaque CVPX frames. There is no CVPX backend, so icop used the opaque fallback:

  • Blocked frames were dropped, so the user saw a black screen instead of blur or warning.
  • The debug overlay never rendered.
  • Analysis ran synchronously on the video thread and held decoder buffers, so VideoToolbox stalled (pic_holder_wait timed out, many "picture is too late").

Fix

nsfw_backend_open now declines CVPX input. VLC's chain filter then inserts its cvpx converter and reopens icop with I420, which runs the normal CPU path: async worker, block styles and overlay. This uses VLC's existing fallback (filter_chain_AppendInner → chain → BuildFilterChain) and doesn't depend on any private VLC structs.

It is fail-closed. icop only declines when module_exists("cvpx") is true. Without that converter VLC would drop icop from the chain and play unfiltered, so in that case the old opaque fallback is kept.

The macOS smoke test now fails if the log says blocked frames would be dropped.

Verification (VLC 3.0.23, Apple Silicon, 1080p clip, hardware decoding on)

Run Decoder stalls Late frames Blocked output
Before (30 s) 2–138 89–138 dropped (black)
After (~95 s) 0 32 blur drawn, overlay visible
  • macos_vlc_smoke_test.sh passes on the installed build and logs declining VideoToolbox chroma CVPN → video backend=cpu input=I420.
  • Not addressed here: the software path logs VLC's "Unsupported timestamp modifications done by chain_interactive" warning. This was already the case before this change, and it comes from icop holding frames before output.

🤖 Generated with Claude Code

asayed18 and others added 2 commits October 1, 2026 01:51
…utput

On macOS VLC decodes with VideoToolbox, so icop received opaque CVPX
frames. With no CVPX backend it fell back to image conversion: blocked
frames were dropped (black screen instead of blur/warning), the debug
overlay could not render, and synchronous analysis held decoder buffers
until VideoToolbox stalled (pic_holder_wait timed out).

Decline CVPX input in Open so VLC's chain filter inserts the cvpx
converter and reopens icop with I420, which runs the normal CPU path.
Only decline when the cvpx module exists; otherwise keep the opaque
fallback so icop is never dropped from the chain (fail-closed).

Verified with VLC 3.0.23 on Apple Silicon: blur and overlay render, no
decoder stalls or dropped frames. The macOS smoke test now fails if
blocked frames would be dropped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every ONNX session was pinned to one intra-op thread, which assumes one
CPU worker per core. GPU providers (CoreML on macOS) and hardware backends
run a single worker, so marqo took ~120 ms per inference on one thread.
With the automatic stride (every 3rd frame) a 720p30 video needed more
than a second of inference per second and fell steadily behind: 380 late
frames, median 3.5 s, in 30 s.

The plugin now exports NSFW_ONNX_INTRA_OP_THREADS = cores / workers (1-4)
unless the user set it, and icop_core applies it to the session. Same file
afterwards: 5 late frames, median 37 ms, no decoder stalls.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@asayed18

asayed18 commented Oct 1, 2026

Copy link
Copy Markdown
Owner Author

Added 446b611: several ONNX threads when icop runs a single detector worker.

Every ONNX session was pinned to SetIntraOpNumThreads(1). CoreML on macOS runs a single worker, so marqo needed ~120 ms per inference. With the automatic stride (every 3rd frame) a 720p30 video needed more than 1 s of inference per second of video and fell steadily behind. The plugin now exports NSFW_ONNX_INTRA_OP_THREADS = cores / workers (1–4) unless the user set it.

30 s run, VLC 3.0.23, Apple Silicon (8P+2E) Late frames Median late Max late
720p30, before 380 3542 ms 8740 ms
720p30, after 5 37 ms 45 ms
1080p, after 1 22 ms 22 ms

marqo ORT benchmark: 1 thread 122 ms → 4 threads 37 ms. Moving marqo onto CoreML (fixed shape + MLProgram, 591/604 nodes) gave no speedup over the multi-threaded CPU, so that isn't included.

Base automatically changed from claude/fix-macos-apple-silicon to main October 1, 2026 00:43
@asayed18
asayed18 merged commit 0e96b1f into main Oct 1, 2026
5 checks passed
@asayed18
asayed18 deleted the claude/macos-cvpx-backend branch October 1, 2026 00:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant