HyperQuant: A Rate-Distortion-Optimal Quantization Pipeline for Large Language and Diffusion Models #9

Is your feature request related to a problem? Please describe.

diffusers' quantizer backends are exclusively thin adapters over third-party libraries (autoround, torchao, quanto, bitsandbytes, gguf, modelopt) that only load pre-quantized checkpoints — none implements a quantization algorithm natively, and there is no custom-kernel surface. HyperQuant: A Rate-Distortion-Optimal Quantization Pipeline for Large Language and Diffusion Models is a new paper proposing a post-training weight + KV-cache pipeline (per-tile Randomized Hadamard Transform → optimal-lattice quantization onto E8/D4/A2/Z → lossless bit-stripping + near-entropy-optimal Rice coding of lattice indices → KV-cache bias correction) that reports ~3.9x linear-weight compression and ~3.79x KV compression at near-lossless quality at 4 bps on an H100, with no per-frame artifacts on a 19B LTX-2 DiT. But HyperQuant ships no library, no reference kernels, and no pre-quantized checkpoints, so there is nothing to wrap or load today — a diffusers backend would have to reimplement the algorithm from scratch, breaking the established integration model.

Describe the solution you'd like.

A team decision and a scoped first slice. Prototype Randomized Hadamard Transform + lattice (E8/D4) weight quantization for a single DiT (e.g. Ideogram 4) in a standalone notebook and measure reconstruction error and generation fidelity (e.g. CLIP-score / FID on a small prompt set) at 4 bps versus the existing AutoRound and torchao backends — explicitly excluding the inference-speed and memory-compression claims, which require the paper's custom Tensor-Core MMA kernels that diffusers does not host. The result tells the team whether a native quality-only slice is worth merging before kernels exist.

Describe alternatives you've considered.

• Wait and adapt. Wait for a third-party HyperQuant library to mature, then add an adapter backend — the diffusers-consistent path, mirroring AutoRound/torchao.
• Own native + kernels. Commit to a native RHT+lattice implementation plus custom 8/4-bit Tensor-Core MMA kernels (fp8-e4m3, int8, nvfp4, mxfp4; the paper notes int8 beats fp8 on the post-RHT lattice output). This would be the first native quantization algorithm and the first to require bespoke CUDA kernels in diffusers.
• Quality-only proof first. Land a weight-only RHT+lattice prototype that measures reconstruction/generation quality only (no speed/memory), deferring the kernel and packed Rice-coded checkpoint format. Note diffusers currently loads safetensors to standard tensors and has no path to load a packed lattice-index format.

Additional context.

What blocks a clean implementation

• No adapter target. Every diffusers quantizer backend is a thin wrapper over an external library and only loads pre-quantized checkpoints (AutoRound's quantizer explicitly states on-the-fly calibration is unsupported). HyperQuant has nothing to wrap.
• Value needs infrastructure the repo lacks. The measured gains (4-bit inference speedup, ~3.9x memory compression) require (a) the paper's custom Tensor-Core MMA dequantization kernels and (b) a packed, Rice-coded lattice-index checkpoint format. diffusers has neither custom CUDA kernels nor a loader for such a packed format; a PyTorch-only RHT+lattice prototype can measure quality but cannot deliver the speed/memory the headline comparison implies.
• Freestanding-native scope decision. A from-scratch HyperQuant backend would be a meaningful scope/architecture expansion (first native quantizer, first bespoke kernels) the maintainers should sign off on rather than land via a drive-by PR.

Questions to unblock

• Integration model: (a) wait for a third-party HyperQuant library to mature and add an adapter, or (b) own a native RHT+lattice implementation plus custom MMA kernels?
• Scope the measurable slice: is a quality-only proof a worthwhile first PR, or is the speed/memory story a prerequisite for merging?
• Kernel ownership: who maintains the custom int8/fp8 lattice-MMA paths and on which hardware (the suggested AWS Neuron path vs. the paper's H100 CUDA)?
• Checkpoint format: appetite to add a packed Rice-coded loader path, or store HyperQuant outputs as dequantized standard tensors (forfeiting the compression benefit)?

Why this is an Issue instead of a PR

Pre-flight routed this to an Issue before the coding agent ran. The scoped "wire it into the quantizers" slice has no clean drop-in: HyperQuant is a brand-new paper with no released library or checkpoints to wrap, and its headline value (4-bit inference speedup, ~3.9x compression) depends on custom Tensor-Core MMA kernels plus a packed Rice-coded checkpoint format that diffusers has no path for. Routed to an Issue for the team to decide whether to wait for a third-party library to adapt, or to take on native algorithm + kernel ownership.

License & code availability

🟡 No code repository surfaced — no LICENSE could be fetched to evaluate. Worth confirming the paper has an open release before investing in adoption.

• Code / model: no repository or model URL surfaced in the paper, recommendation envelope, or arxiv abstract page.
• License: (none detected) (class: no-code-link, compat: 0.30).

Drafted by Outrider — paper: arXiv:2606.23406.