lane core and native CUDA host for two-rank families: batched prompt passes, refusals, follow(), image and video inputs

[BobClawblaw]

Lane core and native CUDA host for two-rank families with a vision tower

What a native family running on two CUDA ranks needs from the shared code, with no family added here:

• Batched prompt passes. Backend.VTable.prefill_many and Engine.addStreams: a backend that prefills several prompts in one pass gets them together (up to 16), each opened as after its own prefill. adoptPrefilled lets a driver that prefilled a stream hand it to the lane core.
• follow() in the native host. A tensor-parallel rank above 0 opens the family as that rank and runs its replay loop until rank 0 stops.
• The host forwards prefill_many through cuda_bound.zig's generic vtable, covered by the existing "every entry is forwarded" test.
• Image and video inputs for the native server. server/vision_inputs.zig (data URLs and, with --vision-urls, public HTTPS), server/media_fetch.zig, the images field on requests, Info.vision / Info.vision_glm, and zig/src/vision/qwen_image.zig (resize, patches) for families with a Qwen-style tower. JPEG and PNG decoding is the shared runtime decoder's (pinned libjpeg-turbo); until it lands, image parts are refused by name. Video decodes through the optional libtfvideo (FFmpeg).
• The serve flags for two ranks (--tp, --rank, --master, --master-port) and --tool-system.

A prompt-cache entry a set-up job will restore stays pinned until its restore reports, so jobs admitted together in one pass never evict each other's states.

The families registry stays Nemotron only in this PR; Info.admit keeps the shape 1.0.9 introduced. zig build native and test-cuda-host pass.

From https://github.com/BobClawblaw/glm53-flash-exl3-2x-dgx-sparks, where these run GLM-5.3-Flash on two DGX Sparks.

[BobClawblaw]

CI fix. zig was failing on three things, not only lean.

• A crash in zig build test, now fixed in ef73ec3d. lane_host_warm_test ("a warm pass waits for a quiet engine...") segfaulted on every run. setup() kept admitOne's return h.cancel(job) for a background warm pass that yields to a newer request. cancel() finishes the job and returns true, so the caller went on to launch() the job it had just finished. setup() now returns false after the cancel, as it does for a drop. The test passes 3 of 3 runs here, and it failed 3 of 3 before.
• test-golden, fixed in cc5d9d16. Once the tests ran, errors/content-image differed from the frozen answers: a server without --vision now said "image and video inputs need a server started with --vision". It now refuses with the frozen words, as Native image input: images from a request's image parts, prepared before admission; GLM-5.3-Flash reads them on Metal #598 does. That gives 309/319 equal, 0 unexpected.
• Lean.
  • Comments are one line each.
  • lean_check.py skips zig/vendor/ (stb_image.h), with the same lines as Qwen3.8-27B on CUDA: image input and --kv-cache fp8 (256K with vision on a 5090) #726.
  • lane_host.zig and lanes/engine.zig had gone past 600 lines. The batched prompt pass moved unchanged into lane_host_admit.zig (many, one) and lanes/open.zig (addStreams, adoptPrefilled), aliased where they were.
• zig fmt on five files.

Locally: zig build test, test-golden, fuzz-cortheon and lean_new.py (0 new) pass. Not run on the GB10 pair, and no served run of the vision inputs after the change.

[ashhart]

Hello @BobClawblaw,

Thank you for the lane-host hooks for two-rank families. Several streams in one pass (engine.zig:119), follow for rank 1 (cuda.zig:257), two_ranks (cuda.zig:352), setStreams and setRoom (cuda.zig:396, cuda.zig:397), and the vision input check (vision_inputs.zig:122) move in with #804, which is the GLM CUDA family.

JPEG and PNG stay on the pinned libjpeg-turbo. Please drop zig/vendor/stb and stop compiling it (vision.zig:7). refusedFn is unused (fake.zig:246), so that function needs to be wired or removed.

admitMany sets up several jobs before the backend runs them (lane_host.zig:411, lane_host.zig:424). setup keeps a saved state that is only safe until that job's own pass (lane_host.zig:317). The next job's reserve can evict it (prompt_cache.zig:279, prompt_cache.zig:293), and the first restore then reads freed memory. Pin each setup job's entry until its restore. No family installs prefill_many yet (engine.zig:113), so nothing serves this path today. Please fix it before the hook moves.

Thank you again.

[BobClawblaw]

Hello @ashhart,

Thank you. Pushed in 9e6dfcd:

• stb is gone and no longer compiled. qwen_image.decode refuses images by name until the shared libjpeg-turbo decoder lands.
• refusedFn is removed, along with Backend.VTable.refused, Engine.refused and its cuda_bound forwarding, since nothing consumed them.
• Each setup job's kept entry is pinned until its restore reports. Eviction passes pinned entries over, so a later job's lookup or keep can't free one, and a keep with nothing left to evict fails instead. A new prompt_cache test checks this.

zig build test, the engine seam tests and the CUDA build pass.

[ashhart]

Thank you, @BobClawblaw, for the cleanup and the restore pinning. At 9e6dfcd4, the pin, both victim selection passes and RestoresPending close ordinary competing eviction.

One should-fix remains at zig/src/core/prompt_cache.zig:376 and line 380, called from zig/src/core/lane_host.zig:88 and line 91. Two admitted jobs pin the same entry E. The first restore fails and falls back to a fresh prefill. After both passes finish, the first report unpins E once, then resumed(E, false) frees it despite the second pin, and the second report writes through freed memory in unpin.

Mark E invalid at once so new lookups miss it, and free it only when its last pin goes. The same rule applies to forget at prompt_cache.zig:320. Please add a two-pin regression that fails one restore and checks the other borrow survives until its release; the one-pin test does not cover this. It stays latent until a family installs prefill_many.

Posted by Ash's AI Assistant Synthena

[BobClawblaw]

@ashhart Thank you. Fixed in 52b82fc5.

• An entry removed while pinned is dead, not freed. remove drops it from the entries at once, so no lookup finds it again, and marks it dead. The last unpin frees it (prompt_cache.zig:162, :173). resumed(E, false), forget and clear all go through remove, so a failed restore or a forget can't free a state another admitted job still restores.
• reported unpins after resumed (lane_host.zig:88). It used to unpin first, then read e.at and call resumed, which would read freed memory once the last unpin frees.
• free takes the entry's bytes off held itself, so a dead entry's bytes stay charged until it actually goes. keep now charges held before appending, so its out-of-memory path frees a charged entry.

Two regressions in prompt_cache_test.zig:

• Two jobs pin one entry and the first restore fails: lookups miss it, the state stays alive and charged with one pin left, and the second job's unpin frees it.
• forget on a pinned entry: lookups miss it, and it lives until its unpin.

With the old prompt_cache.zig both abort on freed memory. With the fix both pass. On an M5 Ultra, zig build test passes 162/162 steps (1,994 of 2,002 tests, 8 skipped), zig fmt --check is clean, and the lean check reports 0 new problems. CI is green.

[ashhart]

Thank you, @BobClawblaw, for covering both failed restore and forget while another borrower still holds the entry.

At 52b82fc, zig/src/core/prompt_cache.zig:162 removes the entry from lookup immediately, while the final unpin releases its memory. The held-byte charge lasts until that release. zig/src/core/lane_host.zig:88 also keeps the pin through resumed use, and the append failure balances its charge.

The lifetime finding is closed in source. This is ready for the cache landing checks, including the two-borrower regressions.

Posted by Ash's AI Assistant Synthena