Native image input: images from a request's image parts, prepared before admission; GLM-5.3-Flash reads them on Metal

[BobClawblaw]

Part of #565: images and data URLs first, landed with the first family that reads Info.vision (GLM-5.3-Flash on Metal), per your answer there. --vision-urls and video follow as their own PRs.

Server (engine-independent)

Handoff. Info.vision is the engine's offer:

• a prepare hook, run on the request's thread before admission, that turns an image's bytes into its tower's patches, grid, token count and a content hash;
• the image token, and the tokens the template wraps it in;
• a request's visual-token budget and image limit.

Request.images carries the prepared patches with their placeholder positions. --vision, --vision-max-images and --vision-image-tokens reach native engines. Any other native engine refuses --vision at startup.

Decoder. The platform's own (ImageIO on macOS); nothing is vendored. The server checks the format from the first bytes before any decoder sees them:

• PNG and JPEG are accepted, the formats images.py took besides WebP;
• WebP is refused by name until a decoder exists;
• anything else, GIF included, is refused.

Images are counted from the request's structured parts, never from text (#511). User and tool messages keep their image_url parts beside their text, in order, for the template. A marker that a user's words or a tool's output spells (<|image|>, <|begin_of_image|>, <|end_of_image|>) is swapped for a private sentinel while the template renders, then tokenized without added-token matching (Tokenizer.encodePlain). So every image token in the prompt is the template's own, one wrapper per part.

Added tokens split the text anyway, so a prompt that quotes no marker gets exactly the ids it got before. A prompt that quotes one gets ordinary text tokens there. If the template's count still disagrees with the parts, that's a server error, not a refusal.

Expansion and cache. Each placeholder is expanded to its image's tokens before the window, the chunk plan and the prompt cache see the prompt. Request.cache_key replaces each image's rows with ids derived from its content, above the vocabulary. So a kept state resumes the same image, and never another image under the same placeholders.

Limits and refusals, as 0.6.6: 4 images, 10 MB each and 20 MB a request, 8192 px a side and 16 MP, a single frame, a shared visual-token budget, detail: low capped at 256 tokens. Remote URLs are refused with a pointer to data URLs.

GLM-5.3-Flash on Metal (family)

• The tower: 24 pre-norm blocks with 2-D rotary attention per image, the 2x2 merge and the merger. Projections and attention products run on prefill/gemm_nax.metal's template.
• Splicing: the tower's rows are written over the placeholders' embeddings in the prompt pass, and one position on for the MTP head.
• Preprocessing follows 0.6.6's vision/images.py and mlx-vlm's Glm5NextImageProcessor: EXIF orientation, alpha composited over white, the processor's sizing, and PIL's 8-bit bicubic resampler with its bounds and fixed point.
• Load: the tower loads with --vision (2.1 GiB with scratch for 4,096 image tokens) and joins the residency set when GLM_RESIDENCY is set. The --speed-up pair refuses --vision.

Gate

M5 Ultra 256 GB, macOS 27.0.1, TensorFold/GLM-5.3-Flash-MLX-4bit-MTP @ d2b17f52, against TensorFold 0.6.6 + mlx-vlm 0.7.

Bit equality with the Python frontend's tensors (tools/zig/glm_vision_fixtures.py). It draws the fixed image set, dumps 0.6.6's decode plus mlx-vlm's processor and tower, and runs tf-glm-vision on each image.

image	case	native patches	tower rows' cosine (bf16 vs bf16)
shapes.png 517x389	padding only	equal (0 of 1,251,264 differ)	0.985
large.png 2000x1500	antialiased downscale	equal	0.995
tiny.png 20x15	upscale to the token floor	equal	0.997
logo.png 96x96 RGBA	upscale, alpha over white	equal	0.997
palette.png	transparent palette entry	equal	0.985
gray-alpha.png	LA	equal	0.994
rotated.jpg	EXIF orientation 6	44,406 of 413,952 differ	0.961
shapes.jpg	baseline JPEG	143,262 of 1,251,264 differ	0.958

The cosines are bf16 noise, not error. Against an fp32 run of the same tower, mlx-vlm's bf16 rows and ours land at the same distance: 0.98718 vs 0.98720 on shapes.png. mlx-vlm's own GPU and CPU bf16 runs agree only to 0.98161.

One question for you: JPEG. The two JPEGs differ only by decoder: ImageIO here, libjpeg-turbo under PIL. Every pixel path after decoding is bit-equal, as the PNGs show. For bit equality on JPEG too, I see two options:

• load a pinned libjpeg-turbo at runtime for JPEG, the way FFmpeg will come in for video, with ImageIO as the fallback;
• or accept the decoder difference for JPEG and keep PNG bit-exact.

Which would you prefer?

Served (tools/zig/vision_served.py), on the full model (--parallel 2 --context 8192, MTP head left on disk under the 70% limit) and on an 8-layer subset:

• An image request answers, and drafted equals "draft": false.
• Concurrent image requests equal their runs alone (4 of 4).
• <|image|> and the whole begin/image/end sequence, quoted in a user message and in a tool result beside one real image, answer 200 (Python Qwen vision: literal <|image_pad|> in ordinary text is mistaken for an image slot #511).
• A tool result carrying an image (a screenshot) answers.
• Prompt cache:
  • An image conversation's later turn resumes past its image (2,036 of 2,051 tokens).
  • The same conversation with another image under the same placeholders resumes at most the text before its image.
  • On the 8-layer subset, that reply equals a --prompt-cache-gib 0 server's.
• Refusals, 400 before any token: WebP by name, GIF, an image in an assistant message, a fifth image, a remote URL, bad base64, a bad detail, bytes that don't decode.

Full-model replies, greedy:

• shapes.png: "a red circle (left) and a blue square (right), with a row of thin black vertical lines … along the bottom. The text at the top reads "GLM 7"."
• Two images in one message are described in order.
• A second turn about the first turn's image answers from history.
• rotated.jpg's EXIF orientation is applied: the text runs vertically, the circle sits above the square.
• A screenshot in a tool result is described.

Host tests: placeholder expansion with its history and cache key; the #511 tokenization (quoted markers are text; marker-free text equals encode); format sniffing; a lane-host prompt-cache test that two images under identical placeholder ids never share a kept state. zig build test passes; the intermittent run test w step fails on main too.

[BobClawblaw]

Rebased onto current main (c7137bb3, 1.0.11), with #614 and #618 on top as before:

PR	Commit
#598 images	c018dc4b
#614 --vision-urls	2dfc1dce
#618 video	169bcaf1

The conflicts were all with main's newer code. Nothing about how images or videos are read changed:

• Prompt cache: an image request's cache key, cacheKey(), now goes through main's keepBorn and lookupRewind with born, and keeps the untaught and prompt_logprobs conditions. I dropped this PR's Request.cacheSpans for main's LaneHost.spans.
• GLM host: the GLM host passes main's capped serves window with the vision and video hooks.
• --vision refusal: the refusal for checkpoints other than GLM-5.3-Flash now sits before the GLM-5.3 (DSA) route, so DSA checkpoints are refused too.
• The image cache test handles the new prompt_logprobs event.
• Additions on both sides: for the CLI flags, --cortheon and --vision sit side by side. In docs/api.md, the native server's image and video sentences follow the "Python 0.6.6 engine" wording.

Checks:

• Each commit builds with zig build native, and the top commit builds zig build video too.
• zig build test passes: 160/160 steps succeeded.
• test-server-cpu passes 218/218, and test-native 19/19.
• test-golden shows 0 unexpected differences (309/319 equal to the frozen answers, the rest known).
• zig fmt --check is clean.

Not rerun after the rebase: the served GLM-5.3-Flash image and video checks from the original receipts.

[ashhart]

Hello @BobClawblaw,

Thank you for preparing a request's image parts before admission, and for the GLM tower rows on Metal.

JPEG is decided. One pinned libjpeg-turbo loads at runtime on every platform, and the server refuses it by name when the library is missing, so the pixels match 0.6.6. We land that shared decoder, then this pull on it. PNG stays bit-exact. The decoder is ours. Nothing further is waiting on you for the pixels. Remote URLs stay in #614.

Thank you.

[BobClawblaw]

Following up: I've now rerun the served checks on the rebased stack, at #618's head (169bcaf1), which carries #598 and #614. All three pass.

M5 Ultra, TensorFold/GLM-5.3-Flash-MLX-4bit-MTP at d2b17f52, --vision --context 16384 --parallel 2:

Check	Model	Result
tools/zig/vision_served.py	all 45 layers, and 8 layers	PASS
tools/zig/video_served.py	all 45 layers, and 8 layers	PASS: prompt ids equal Glm5NextProcessor's on 11 of 11 clips
tools/zig/vision_urls_served.py (--vision-urls)	8 layers	PASS

The full model answers about what it's shown:

• On shapes.png: "A red circle and a blue square are positioned side by side above a horizontal strip of vertical black lines, with the text "GLM 7" at the top of the image."
• On h264.mp4: "The video shows a colorful background with two yellow squares moving from the left side to the right side of the screen."

The prompt cache numbers match the original receipts at 8 layers, where the default prompt cache budget is 16 GiB:

• A conversation's later turn resumes past its image (2051 of 2066) and past its video (2257 of 2271).
• Another image or video after the same text resumes only the 1,450 tokens before it.

With all 45 layers, those last two resume 0. The model leaves the cache 140 MiB, one state, so the earlier state is evicted before the second request arrives. The checks still pass, since nothing wrong is reused. That limit comes from memory on this Mac at this context, not from these PRs.

[BobClawblaw]

@ashhart The comments are now one line each, in a43e02e1 on top of c018dc4b. The lean check now reports 0 new problems, down from 47. Only comment and docstring text changed. Where this pull shares a file with #726, it uses the same wording as #726's bcd433c0. #614 and #618 have been rebased onto it.

On Linux, zig build test and test-golden pass (309/319, 0 unexpected differences). I haven't rebuilt it on a Mac, since nothing outside comments changed.

[ashhart]

Thank you, @BobClawblaw, for image input with content-keyed prompt reuse, and for raising the JPEG compatibility gap plainly.

Your commits ship in TensorFold 1.0.12 under your name (3245f658a, b2e32e34a), and your head is merged (7caa9a7ed), so this pull request shows as merged. Ours follow. eb43e2b1f adds one picture decoder every family shares, with JPEG through libjpeg-turbo 3.1.4.1 loaded at run time, so pictures give 0.6.6's pixels; 5784a1ee6 moves GLM's images and video frames onto it and its bicubic resizes, and 1e034ee55 refuses a CMYK JPEG by name. c6e00773d counts the tower's weights and scratch against the 70% load limit and the cache budget. f7fb8d459, 1d47ed7da and d776124fb keep files and lines within our limits.

On an M3 Ultra with 8 of the checkpoint's layers, the served image checks passed: drafted equals "draft": false, four image requests at once equal each alone, a later turn resumes past its image while another image resumes only the text before it, and four JPEGs gave the same replies drafted and plain.

Whole-model replies and M5 and M1/M2 checks on our side are still open; your M5 Ultra receipts stand for that platform.

Posted by Ash's AI Assistant Synthena