ggml/meta: tensor split support for Hadamard-folded qwen35 (Bonsai-2)

[Spacecer2]

What

Makes -sm tensor usable for qwen35/Bonsai-2 instead of aborting during graph allocation, by giving the meta backend the split-state awareness and all-gather the Hadamard path needs.

The bug

Ternary-Bonsai-2-27B (arch qwen35, hybrid attention with Gated Delta Net) aborts under tensor split:

ggml/src/ggml-backend-meta.cpp:1086: GGML_ASSERT(split_state.ne[j] % div == 0) failed
[META-SPLIT-FAIL] tensor=final_output-0 (reshaped) op=RESHAPE
                  ne=[128,16,3,2] axis=2 div=6144 ne_j=9216 nb=2

Note the model loads and starts running before aborting, so it looks like it will work.

Root cause

Two things are worth separating, because the second is why this isn't a one-liner.

QWEN35 is not in the llm_arch_supports_sm_tensor denylist, and its segmented split already exists: get_split_segments in src/llama-model.cpp carries bespoke logic for QWEN3NEXT/QWEN35/QWEN35MOE (ssm_out → {key_dim, head_ratio}). So tensor support was intended.

The failure is that this split is never propagated onto the activation final_output. It reaches the Hadamard reshape in llm_graph_context::build_lora_mm (src/llama-graph.cpp) as a plain axis 0, n_seg=1, nr=1. The fold reshapes the activation feature dim 6144 = 128*16*3 into three axes, and the single-axis meta representation cannot express that — the ratio check then aborts. The FIXME in ggml-backend-meta.cpp about preserving/erasing n_segments inconsistently sits right on top of this.

Making it correct is architectural: there are no all-gather / reduce-scatter primitives in the meta backend (only ggml_backend_comm_allreduce_tensor), and the Hadamard path has no split awareness at all. Correct TP for a Hadamard-folded weight needs the activation replicated (the rotation scrambles the feature axis) with the weight split on its output dim — which requires an all-gather to materialise the replicated activation.

What the patch does

7 files, +631/−46:

• GGML_HINT_HADAMARD_TILING — marks the reshape that tiles the activation feature axis into [hd, nk, rep]; its input must be replicated under tensor parallelism.
• GGML_HINT_META_GATHER — marks a node that must materialise a full (replicated) copy of a split input, i.e. an all-gather across the TP devices.
• ggml-backend-meta.cpp (~440 lines) — treats a replicated operand as compatible with a split one (each backend already holds the full copy it needs), and adds the gather path the backend previously lacked.
• src/llama-graph.cpp / src/llama-model.cpp / src/llama-context.cpp / src/llama-impl.h / src/llama.cpp — tag the activation nodes and thread the split state through.

Verification

Measured locally on a 2-device setup, q8_0 KV, -fa on, -ngl 99:

llama-cli -m Ternary-Bonsai-2-27B-PQ2_0.gguf \
    -ngl 99 -fa on -sm tensor -ctk q8_0 -ctv q8_0 -c 512 -n 8 -p "Hi"

• before: aborts with the assertion above
• after: runs to completion at ~84.9 t/s prompt / ~13.9 t/s generation, with the new gather path exercised throughout the graph

Caveats — please read before merging

1. Debug tracing is still in this patch. fprintf(stderr, …) / fflush(stderr) printing [META-*], [DBG*], [META-SPLIT-FAIL], [SPLITCFG] are unconditional (~25 prints, fires thousands of times per run). They are left in so the behaviour is exactly reproducible as measured. I will strip them on request — say the word and I'll push a cleaned revision rather than have you ask for it.
2. Based on 9a9394a, 63 commits behind prism — not rebased, so a conflict is possible.
3. Not included: the immediate safety fix of adding LLM_ARCH_QWEN35 / QWEN35MOE / QWEN3NEXT to llm_arch_supports_sm_tensor so tensor mode fails cleanly rather than aborting. That is orthogonal and probably still worth taking regardless of this patch.

Minimal repro

git clone --depth 1 --branch prism-b10709-9a9394a https://github.com/PrismML-Eng/llama.cpp
hf download prism-ml/Ternary-Bonsai-2-27B-gguf Ternary-Bonsai-2-27B-PQ2_0.gguf --local-dir .
./build/bin/llama-cli -m Ternary-Bonsai-2-27B-PQ2_0.gguf \
    -ngl 99 -fa on -sm tensor -ctk q8_0 -ctv q8_0 -c 512 -n 8 -p "Hi"

[bri-prism]

Thanks for digging into this, the diagnosis is right: -sm tensor aborts on qwen35 at the meta backend split check because the Hadamard reshape gets a split state it can't express.

We had a smaller fix for the same assert in a private branch (the split rules from upstream ggml-org#29600 plus a guard), and I've opened it as #363: 3 files, +50/-4, no new ggml.h hints and no gather path. On 2x H200 it runs Bonsai 2 27B PQ2_0 and Q2_0 under tensor split with mean KLD around 2.5e-4 vs single GPU, and it's bit identical to base on Qwen3-4B and Qwen3.5-4B in both single and tensor mode.

Since that one is much smaller, we'd rather land it and close this one. If you can run #363 on your 2 device setup and confirm it works there, that would be really helpful.