[Feature]Request: Add FP8 (e4m3) and fix Q6_K/Q8_0 Vulkan and Rocm10.0 support for MiniMax-H3

[farawayso]

Feature Summary

Q6_K/Q8_0 crash on Vulkan and Rocm10.0

Detailed Description

I'm running MiniMax-H3 video generation on AMD 7900 XTX (24GB VRAM) with stable-diffusion.cpp using Vulkan backend. Current setup works well with Q4_K_M quantization, but I'd like to use higher-quality or more efficient quantization formats.

Alternatives you considered

or q6.0 q8.0 gguf

Additional context

I'm running MiniMax-H3 video generation on AMD 7900 XTX (24GB VRAM) with stable-diffusion.cpp using Vulkan backend. Current setup works well with Q4_K_M quantization, but I'd like to use higher-quality or more efficient quantization formats.

Problem 1: Q6_K/Q8_0 crash on Vulkan

When selecting Q6_K or Q8_0 quantization, sd.cpp crashes with:

ggml/src/ggml.c:3301: GGML_ASSERT(!ggml_is_transposed(a)) failed

This happens in the Vulkan matmul kernel path. Q4_K_M works fine, suggesting the dequant+matmul pipeline for these quant types isn't fully implemented for Vulkan.

Test command:

sd-cli.exe -M vid_gen --diffusion-model minimax_h3_fl2va_pruned-Q6_K.gguf --llm qwen3vl_32b_minimax_h3-Q4_K_M.gguf --vae minimax_h3_video_vae_fp16.safetensors -p "test" -W 64 -H 64 -o out.png -v

Expected: Model loads and runs (same as Q4_K_M)
Actual: Assertion failure during graph execution

Problem 2: No FP8 (e4m3) support

The official MiniMax-H3 models include FP8 e4m3 checkpoints:

• minimax_h3_fl2va_pruned_fp8_e4m3fn.safetensors
• minimax_h3_ref2va_pruned_fp8_e4m3fn.safetensors

FP8 would be ideal because:

• ~2x memory savings vs BF16
• Near-lossless quality compared to BF16
• Native hardware support on modern GPUs

GPU Details

GPU: AMD Radeon RX 7900 XTX (gfx1100, RDNA3)
Vulkan: 1.3.274
Driver: AMD proprietary (25.30.1)
VRAM: 24GB

Note: RDNA3 has cooperative matrix support but lacks VK_EXT_shader_float8 (that's RDNA4/gfx1200+). The portable e4m3 dequant-to-f16 path from ggml PR #1528 would work here.

Suggested Implementation

1. Fix Q6_K/Q8_0 Vulkan matmul: Add proper transposed-tensor handling or route to CPU fallback
2. Add FP8 e4m3 support: The ggml PR fix: avoid Vulkan f16 repeat in LTX audio VAE #1528 already implements this - verify sd.cpp properly wires it up for diffusion models

Reference

• ggml PR fix: avoid Vulkan f16 repeat in LTX audio VAE #1528: vulkan: add e4m3 (fp8) matmul for AMD RDNA4
• MiniMax-H3 FP8 discussion: Running MiniMax H3 (FL2VA) on 4x L40S with SGLang: dynamic FP8, ~2x speedup vs 2-GPU BF16, 14GB/card sgl-project/sglang#34079

---

Would be happy to help test any fixes!