Repository navigation
Vision on the MoE side: qwen3_5_moe (Qwen3.6-35B-A3B) gated out of --vision — the dense path ports over in ~70 lines #652
Description
Vision on the MoE side: qwen3_5_moe (Qwen3.6-35B-A3B) is gated out of --vision
TensorFold 0.6.x serves vision on the dense families (Qwen3.5/3.8, GLM-5.3-Flash, Flash Next). The MoE side — qwen3_5_moe, i.e. Qwen3.6-35B-A3B — has no path: the vision gates reject the family, and past the gates the MoE engine has none of the dense side's vision machinery (no tower attach, no vision= through generate/scheduler/serial paths, no admit-time encode in the multi).
The tower itself is the same stock Qwen3.5 ViT the dense models carry, and every hard prerequisite (mrope positions, the CUDA rotary/attention path, the KV layout) already works — the MoE decode reuses the dense prefill_chunk. What's missing is gates + wiring, not architecture.
The dense-vs-MoE feature diff (what the port fills)
| site | dense (ships) | qwen3_5_moe (this port) |
|---|---|---|
vision/config.py |
family gate (dense tuple) | + qwen3_5_moe |
vision/qwen_cuda.py |
tower gate | + qwen3_5_moe |
vision/qwen_mlx.py, vision/qwen_processing.py |
model-type gates | + qwen3_5_moe |
families/qwen3_5_moe/cuda/engine.py |
no vision attr; generate() has no vision=; scheduler submit drops it |
attach_vision() (the QwenCudaVision tower on the engine's device), encode-once before prefill, vision= through the scheduler submit, cache/stops stand down on vision requests |
families/qwen3_5_moe/cuda/decode.py |
prefill/extend/prefill_chunk take no vision |
vision threaded end-to-end — both serial paths, MTP drafts included |
families/qwen3_5_moe/cuda/multi.py |
no admit-time encode | the dense multi's contract mirrored: admit-encode, per-stream vision, cache-hit skip |
families/qwen3_5_moe/__init__.py |
factory ignores --vision |
attaches post-construct |
Verified end-to-end (0.6.0 wheel, single DGX Spark GB1, Qwen3.6-35B-A3B MLX-4bit + MTP drafts)
- 4-quadrant spatial read (the mrope stress test): exact — Red/Green/Yellow/Blue clockwise, on the serial path and
--parallel 4 - text-only regression: unchanged; repeats bit-deterministic
- true concurrency: 2 vision + 1 text lanes in flight, all correct, isolated, ~0.8 s wall
- encoder admission ≈ 0.9 GiB on top of the signed pool
Total diff: 8 files, +46/−22 — the dense vision path, mirrored for the MoE. Happy to send the anchor-based patch kit we used to apply/audit it against an installed tree (TF_ROOT override) if that's useful for review.
Draft PR to python-0.6 follows.
Environment: TensorFold 0.6.0 (pip), TensorFold/Qwen3.6-35B-A3B-MLX-4bit-MTP, CUDA on a single DGX Spark (GB10, 128 GB unified), --parallel 4, MTP drafts on.
Activity
ashhart commentedon Oct 14, 2026
Hello @hennas-waifson,
Thank you for the four-quadrant read on Qwen3.6-35B-A3B, and for showing text-only stayed bit-deterministic with drafts on.
That run is the Python 0.6.0 engine. The Python line is frozen at v0.6.6. A draft PR to python-0.6 stays on that tag.
The native binary has no qwen3_5_moe family, so pull and info refuse the checkpoint (docs/recipes/qwen3.6-moe.md:9). CUDA on main serves Nemotron only (zig/src/native/cuda.zig:13). Wiring vision onto a family the native engine does not have yet is a maintainer decision. What would help, once that family exists, is the same spatial read with drafts on, plus a text-only check that drafted output matches a run with drafts off.
Thank you again.
ashhart commentedon Oct 15, 2026
Thank you, @hennas-waifson, for the four quadrant spatial check and the run with two image requests and one text request together.
We'll coordinate one native image integration across #856 and #885, using the Qwen CUDA base in #836 and shared tower preparation. Please bring those spatial and mixed request fixtures into the native integration PR. We'll review the image wiring there, including image identity in the prompt cache and rotary positions, then check drafted against plain replies, resumed against fresh prompts, and concurrent against solo requests. This takes your checks forward with the shared Zig engine.
Posted by Ash's AI Assistant Synthena
ashhart commentedon Oct 17, 2026
Thank you, @hennas-waifson, for the four-quadrant fixture and the two-image, one-text concurrency check.
The native CUDA MoE image path is still queued. We are keeping one integration: the Qwen base in #836, MoE in #856 and its follow-up #972, and the owned image path in #885. The Python patch remains a reference, not a change to the frozen engine.
Please bring those fixtures to the native integration PR so the spatial result, image cache identity and rotary positions can be checked alongside drafted/plain, resumed/fresh and concurrent/solo replies. This stays open until that native path is qualified.
Posted by Ash's AI Assistant Synthena
ashhart commentedon Oct 18, 2026
Thank you, @hennas-waifson, for the four-quadrant fixture and two-image/one-text concurrency check. 1.1.0 carries the Qwen CUDA MoE image path through #836, #856/#972 and #885's owned tower. It keeps native picture preparation, M-RoPE positions, media cache identity and shared lanes. Once the final MoE image, drafted/plain, resumed/fresh and concurrent/solo gates pass and it ships, this closes the native vision request. The earlier Python patch remains reference evidence.
GB10 MoE image fixtures reported 26 passes/six failures, including warm-turn/JPEG limits; the full quadrant/two-image parity matrix is not separately recorded.
Posted by Ash's AI Assistant Synthena