Native video input: GLM-5.3-Flash reads video_url parts (part of #565, on #614)

[BobClawblaw]

Part of #565: video, the last step after images (#598) and --vision-urls (#614), built to the plan in #565 (comment).

Stacked on #614 (and so #598). The change is the last commit, d85c3f6. I'll rebase as the others land.

What it does

GLM-5.3-Flash reads video_url parts in user messages as transformers 5.19's Glm5NextProcessor prepares them. The model's tower takes each pair of frames as an image with grid (1, h, w), which is how get_video_features hands pairs over, so the Metal tower is unchanged.

libtfvideo. It's zig/src/video/tf_video.c over FFmpeg 9.0.2, the release PyAV 19.0.1 ships.

• Pinned build: tools/zig/build_ffmpeg.sh PREFIX builds a static FFmpeg from the pinned, checksummed tarball, and zig build video -Dffmpeg=PREFIX links it into an optional library. The library has H.264, HEVC, VP8, VP9 and MPEG-4, plus the MP4/QuickTime and Matroska/WebM demuxers.
• Loading: the server loads it with dlopen when the first video arrives. Nothing is vendored, and without the library the server refuses videos by name.
• Decoding: it reads the first video stream's frame count and rate as read_video_pyav does, then decodes only the sampled indices to RGB as PyAV's to_ndarray(format="rgb24") does (bilinear swscale, the frame's colour space and range).

families/glm/video.zig:

• sample_frames;
• smart_resize over the frame pairs;
• torch's antialiased uint8 bicubic, each frame resized as it decodes, so a long 4K clip never holds full-size frames;
• the black pad, the fused float32 normalisation, and the pairs cut into patches.

Engine API. Info.vision gains a video offer (VideoSource, PreparedVideo, VideoOffer), which GLM's host serves. The engine's tower rows and image count grow by the video budget.

Server:

• Limits (0.6.6's VideoLimits): 2 videos, 16 MB each and 20 MB together, an hour, 8192 px a side.
• Sources: data URLs, and HTTPS URLs under --vision-urls with the video media types, 20 s a download and 120 s a request.
• Prompt: each <|video|> becomes one <|begin_of_image|><|image|>×n<|end_of_image|>T.T seconds block per pair, each pair's rows keyed by its content for the prompt cache. The times are Python's f"{t:.1f}", done in exact integer arithmetic: Zig's {d:.1} rounds the shortest decimal and differs on 0.15, 1.25 and 0.95.
• /tokenize now expands images and videos as the chat route does, as docs/api.md already says it does.

Your two limits

The PR uses what I proposed, and I'll change either to your answer:

• --vision-video-tokens (default 16,384) is passed as the processor's max_image_tokens.
• max_frames is 0.6.6's 256.

Choices to look at

• Frame counts for Matroska and WebM. These containers keep no frame count. read_video_pyav then reports total_num_frames 0, and the loader fails on an empty index list, so transformers' PyAV path can't read them at all. libtfvideo counts the stream's packets instead (one per shown frame), and the reference in the fixture tool is given the same count. On the two fixtures that equals the frames PyAV decodes (40 and 50).
• Clips too short to sample. Under half a second, the processor samples nothing and the loader fails. The server refuses them by name.
• Thinking mode in the /tokenize comparison. The checkpoint's template has no thinking switch, and the server appends </think> itself when thinking is off. The comparison runs with thinking on.

The gate

tools/zig/glm_video_fixtures.py (--make draws and encodes the clip set): 13 clips, run twice, at 16,384 tokens and at 700 (the second forces the downscale path).

• The clips:
  • H.264 4:2:0, 4:4:4, full range and BT.709 MOV;
  • HEVC 8-bit and 10-bit;
  • VP9 WebM and VP8 MKV;
  • MPEG-4;
  • 720p, and an odd 318×182;
  • one clip under a second, and a 150 s clip past the frame cap.
• Compared against the processor's output on frames PyAV decodes: the sampled indices, the grid, every patch value bit for bit, and each pair's time text. All 13 equal at both budgets, and the short clip is refused as the processor samples nothing.
• Decoding alone: libtfvideo's RGB equals PyAV's on every frame of the clip set (1,925 frames).

tools/zig/video_served.py (GLM_LAYERS=8, --vision --parallel 4): PASS, 15 checks.

• Prompt ids: the server's /tokenize ids for a video message equal Glm5NextProcessor's input_ids on all 12 clips that sample.

• Answers: drafted equals "draft": false, 4 concurrent requests equal their runs alone, and a video plus an image in one request answers.

• Prompt cache: a third turn resumes past its video (2257 of 2271 cached), and another video under the same text resumes only the text before it (1450 of 2181).

• Refusals:

  • three videos;
  • a video in an assistant message or a tool result;
  • a clip under half a second;
  • bytes that aren't a video;
  • a URL without --vision-urls;
  • an ftp URL;
  • a video_url without a url.

  Under --vision-urls, a video URL that serves HTML gets 0.6.6's "media URL content type must be one of video/mp4, …".

• Earlier gates on this build: Native image input: images from a request's image parts, prepared before admission; GLM-5.3-Flash reads them on Metal #598's image gate (19 checks) and Native image input: --vision-urls fetches public HTTPS image URLs (part of #565, on #598) #614's URL gate (17) still pass.

• An observation, not from this PR: the cache checks run before the concurrent ones. GLM keeps a prompt state resident in the slot that made it, and a slot another stream takes leaves that state stale, so its restore prefills from 0. Images show the same thing on Native image input: images from a request's image parts, prepared before admission; GLM-5.3-Flash reads them on Metal #598.

Full model, M5 Ultra under memguard (176 GB, one 16k stream):

• The 320×180 clip of a yellow square crossing a gradient: "Two yellow squares appear on the left side of the screen and begin moving diagonally down and to the right… the background colors shift and change."
• The 720p clip: "A yellow rectangle moves from the left side of the screen toward the right side."
• The 150 s clip: "The video runs about 2 minutes 30 seconds", read from the time texts.

zig build test passes, and the server's unit tests pass (119).

[ashhart]

Hello @BobClawblaw,

Thank you for the GLM video path, as an optional dlopen of FFmpeg 9.0.2.

This is ready once the new video comments are one line each. It contains #614 and lands after that pull and after the shared libjpeg-turbo decoder. FFmpeg 9.0.2 stays its own dlopen. The gate is an M5 with the GLM checkpoint, libtfvideo built, the 13-clip compare, then one full-model clip.

Thank you.

[BobClawblaw]

@ashhart The video comments are now one line each, in d9dbddd3, on #614's fixed head. The lean check now reports 0 new problems, down from 108. Only comment and docstring text changed. The video commit is now 728e8895, with the same code as 169bcaf1.

To fit in one line, some detail came out of the file headers:

• the pipeline steps in glm/video.zig;
• the exact limits in videos.zig, whose constants are still marked as videos.py's;
• the RGB conversion settings in tf_video.c;
• the build-step note in build/video.zig, which docs/vision.md still covers;
• the usage text in the two video scripts.

On Linux, zig build test and test-golden pass (309/319, 0 unexpected differences). I haven't run the M5 video gate since this change.

[ashhart]

Thank you, @BobClawblaw. Video for GLM-5.3-Flash is careful work: the sampling follows the processor to the frame, the one-decimal times round as Python rounds them, and FFmpeg stays optional, loaded only when a video arrives.

Your two commits ship in TensorFold 1.0.12 under your name (ac982913d, 356f29bdb), after #614's, and your head is merged (6a056788b), so this pull request shows as merged. On our side the frame resize calls core's shared torch-style bicubic (5784a1ee6), with the same weights and rounding as your copy, and the long lines wrap (d776124fb). The tower's scratch, which your video tokens size, now counts against the 70% load limit, so at the default 16,384 video tokens the whole 4-bit checkpoint with --vision needs a smaller --vision-video-tokens on a 256 GB Mac.

video.zig's sampling and time tests pass on macOS, and videos.zig's limits, URL rules and time texts on macOS and Linux. On an M3 Ultra without libtfvideo, a video part was refused with your sentence.

We did not build FFmpeg 9.0.2 or libtfvideo in our gate, so the 13-clip comparison, the 1,925 frames and video_served.py 15 of 15 stand on your M5 Ultra receipts.

Posted by Ash's AI Assistant Synthena