[OMNIML-6227] Convert GGML checkpoints to INT8 compressed-tensors for vLLM

.github/workflows/example_tests.yml

@@ -70,6 +70,7 @@ jobs:
               - "!**/*.{md,rst,png,ipynb}"
             vllm:
               - "{examples,tests/examples}/vllm_serve/**"
+              - examples/hf_ptq/convert_ggml_to_int8.py # tests/examples/vllm_serve serves its output
               - "!**/*.{md,rst,png,ipynb}"
             trtllm:
               - "{examples,tests/examples}/{gpt-oss,hf_ptq,llm_eval}/**"

CHANGELOG.rst

@@ -11,6 +11,7 @@ Changelog
 - On Linux x86-64 with Python 3.11 and later, the ONNX extra now selects ONNX Runtime 1.30.x and CUDA 13 packages by default. GPU calibration requires a compatible CUDA 13 runtime and driver; Python 3.10 and other platforms retain their existing dependency selections.
 - Add imatrix calibration (``algorithm: {method: imatrix}``) for the GGML formats with a weighted search: each weight is encoded once, its search weighted by the square root of its inputs' per-column ``E[x^2]`` (llama.cpp's imatrix statistic), and HF fused-MoE experts get one statistic per expert. A weight that already holds final codes is left alone, so ``[{method: gptq}, {method: imatrix}]`` runs GPTQ on the dense linears and imatrix on the experts GPTQ skips; the built-in IQ1 and IQ2 recipes now do this, and new ``<format>_experts_only`` recipes apply imatrix to the routed MoE experts alone.
 - Add a checkpoint-specific NVIDIA Nemotron 3.5 Super VL AutoQuantize recipe with a 4.9-effective-bit NVFP4-MSE/FP8/BF16 search and fixed NVFP4 MTP experts. Select it under ``models/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16/auto_quantize/``.
+- Add ``examples/hf_ptq/convert_ggml_to_int8.py`` to expand a checkpoint exported with a GGML recipe (IQ formats, ``q8_0``) into an INT8 weight-only compressed-tensors checkpoint. Stock vLLM serves it with its integer kernels at about half the BF16 weight memory, reproducing the GGML weights up to rounding their scales to the model dtype.
 
 *Speculative Decoding*
 

examples/hf_ptq/README.md

@@ -707,6 +707,20 @@ llm_fp8 = LLM(model="<the exported model path>", quantization="modelopt")
 print(llm_fp8.generate(["What's the age of the earth? "]))
 ```
 
+#### vLLM with GGML recipes (IQ, Q8_0)
+
+vLLM cannot load the GGML checkpoints ModelOpt exports, which are Hugging Face checkpoints holding packed GGML blocks. To serve one on vLLM's integer kernels, expand it into an INT8 weight-only compressed-tensors checkpoint with [convert_ggml_to_int8.py](./convert_ggml_to_int8.py):
+
+```bash
+python hf_ptq.py --pyt_ckpt_path <huggingface_model_card> --recipe general/ptq/iq1_s --export_path <ggml_ckpt_path>
+python convert_ggml_to_int8.py <ggml_ckpt_path> <int8_ckpt_path>
+vllm serve <int8_ckpt_path>
+```
+
+Each weight keeps its GGML integer level, and the step of its 16- or 32-weight sub-block becomes the INT8 group scale. The INT8 checkpoint therefore holds the same weights up to rounding the scales to the model dtype, at about half the BF16 weight memory. Its config names each converted layer explicitly, so every other layer loads unquantized. The GGML checkpoint is left unchanged.
+
+vLLM serves group-32 layers (IQ1_S, IQ2_XXS, Q8_0) with Marlin and group-16 layers (IQ1_M, IQ2_XS, IQ2_S) with Humming. On Blackwell GPUs, Marlin is much slower at large batch sizes, so set `VLLM_DISABLED_KERNELS=MarlinLinearKernel` to use Humming for group 32 as well.
+
 #### SGLang
 
 ```python

examples/hf_ptq/convert_ggml_to_int8.py

@@ -0,0 +1,255 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Expand a GGML checkpoint exported by hf_ptq.py into an INT8 checkpoint that vLLM serves.
+
+python hf_ptq.py --pyt_ckpt_path <model> --recipe general/ptq/iq1_s --export_path <ggml_dir>
+python convert_ggml_to_int8.py <ggml_dir> <int8_dir>
+vllm serve <int8_dir>
+"""
+
+import argparse
+import json
+import os
+import shutil
+from pathlib import Path
+
+import torch
+from safetensors import safe_open
+from safetensors.torch import save_file
+
+from modelopt.torch.quantization.ggml import GGML_FORMAT_REGISTRY, GGMLFormat
+from modelopt.torch.utils.plugins.hf_checkpoint_utils import (
+    copy_non_model_files,
+    copy_off_index_safetensors,
+    indexed_weight_map,
+)
+
+_INDEX = "model.safetensors.index.json"
+
+
+def convert_ggml_export_to_int8(
+    ggml_dir: str | os.PathLike,
+    output_dir: str | os.PathLike,
+    max_shard_size: int = 10 * 2**30,
+) -> None:
+    """Write a GGML export as an INT8 weight-only compressed-tensors checkpoint, which vLLM serves.
+
+    Every GGML weight is an integer level of at most 8 bits times the step of its 16- or
+    32-weight sub-block, so storing those levels with the steps as group scales reproduces the
+    GGML weights up to rounding the steps to the model dtype. Unquantized tensors and the other
+    files are copied as they are; the GGML export is left unchanged.
+
+    Args:
+        ggml_dir: A Hugging Face export whose quantized layers are all GGML formats, as written
+            by hf_ptq.py with a GGML recipe.
+        output_dir: Where to write the INT8 checkpoint. Must be empty or not exist yet.
+        max_shard_size: Target bytes per output safetensors shard; a larger tensor gets a shard
+            of its own. Conversion holds one output shard in memory at a time.
+    """
+    ggml_dir, output_dir = Path(ggml_dir), Path(output_dir)
+    config = json.loads((ggml_dir / "config.json").read_text())
+    quantization = json.loads((ggml_dir / "hf_quant_config.json").read_text())["quantization"]
+    scale_dtype = getattr(torch, config.get("dtype") or config["torch_dtype"])
+    weight_map = indexed_weight_map(ggml_dir)
+    layer_formats = _ggml_layer_formats(ggml_dir, weight_map, quantization)
+    if output_dir.exists() and any(output_dir.iterdir()):
+        raise ValueError(f"{output_dir} is not empty.")
+    output_dir.mkdir(parents=True, exist_ok=True)
+
+    try:
+        shards = sorted(set(weight_map.values()))
+        new_weight_map = _write_int8_shards(
+            ggml_dir, shards, output_dir, layer_formats, scale_dtype, max_shard_size
+        )
+        total_size = sum((output_dir / s).stat().st_size for s in set(new_weight_map.values()))
+        index = {"metadata": {"total_size": total_size}, "weight_map": new_weight_map}
+        (output_dir / _INDEX).write_text(json.dumps(index, indent=2))
+        config["quantization_config"] = _compressed_tensors_config(layer_formats)
+        (output_dir / "config.json").write_text(json.dumps(config, indent=4))
+        # Both skip files already written, and copy_non_model_files skips hf_quant_config.json,
+        # which describes the GGML payloads this replaced.
+        copy_off_index_safetensors(ggml_dir, output_dir)
+        copy_non_model_files(ggml_dir, output_dir)
+    except BaseException:
+        # Empty the directory again, so a rerun is not refused over a partial checkpoint.
+        for path in output_dir.iterdir():
+            if path.is_dir():
+                shutil.rmtree(path)
+            else:
+                path.unlink()
+        raise
+
+
+def _ggml_layer_formats(
+    ggml_dir: Path, weight_map: dict[str, str], quantization: dict
+) -> dict[str, GGMLFormat]:
+    """The GGML format of every layer holding a packed payload, refusing anything else."""
+    if quantization.get("kv_cache_quant_algo") is not None:
+        raise ValueError("INT8 conversion of a GGML export does not carry KV cache quantization.")
+    mixed = quantization.get("quant_algo") == "MIXED_PRECISION"
+    layer_configs = quantization.get("quantized_layers", {}) if mixed else {}
+    algos = (
+        {cfg.get("quant_algo") for cfg in layer_configs.values()}
+        if mixed
+        else {quantization.get("quant_algo")}
+    )
+    unsupported = sorted(str(a) for a in algos if str(a).lower() not in GGML_FORMAT_REGISTRY)
+    if unsupported:
+        raise ValueError(f"Only GGML layers convert to INT8, but the export has {unsupported}.")
+
+    layer_formats = {}
+    for shard in sorted(set(weight_map.values())):
+        with safe_open(ggml_dir / shard, framework="pt") as f:
+            # GGML payloads are the only uint8 weights an export writes.
+            payloads = {
+                name: f.get_slice(name).get_shape()
+                for name in f.keys()  # noqa: SIM118 -- a safe_open handle is not a dict
+                if name.endswith(".weight") and f.get_slice(name).get_dtype() == "U8"
+            }
+        for name, shape in payloads.items():
+            layer = name.removesuffix(".weight")
+            # A 2-D weight's payload is [rows, blocks, block bytes]; weight_shape and the int32
+            # packing assume that layout.
+            if len(shape) != 3:
+                raise ValueError(f"{layer} is not a 2-D weight; only 2-D weights convert to INT8.")
+            if mixed and layer not in layer_configs:
+                raise ValueError(f"{layer} holds a uint8 weight but is not in quantized_layers.")
+            algo = layer_configs[layer]["quant_algo"] if mixed else quantization["quant_algo"]
+            layer_formats[layer] = GGML_FORMAT_REGISTRY[algo.lower()]
+    if not layer_formats:
+        raise ValueError(f"{ggml_dir} holds no GGML weights to convert.")
+    return layer_formats
+
+
+def _write_int8_shards(
+    ggml_dir: Path,
+    shards: list[str],
+    output_dir: Path,
+    layer_formats: dict[str, GGMLFormat],
+    scale_dtype: torch.dtype,
+    max_shard_size: int,
+) -> dict[str, str]:
+    """Rewrite every shard's tensors into size-bounded output shards; return the weight map."""
+    device = "cuda" if torch.cuda.is_available() else "cpu"
+    pending: dict[str, torch.Tensor] = {}
+    written: list[list[str]] = []
+
+    def flush():
+        save_file(pending, output_dir / f"part-{len(written):05d}.safetensors", {"format": "pt"})
+        written.append(list(pending))
+        pending.clear()
+
+    for shard in shards:
+        with safe_open(ggml_dir / shard, framework="pt") as f:
+            for name in f.keys():  # noqa: SIM118 -- a safe_open handle is not a dict
+                layer = name.removesuffix(".weight")
+                ggml_format = layer_formats.get(layer) if name.endswith(".weight") else None
+                if ggml_format is None:
+                    tensors = {name: f.get_tensor(name)}
+                else:
+                    tensors = _int8_tensors(
+                        layer, f.get_tensor(name), ggml_format, scale_dtype, device
+                    )
+                if pending and _nbytes(pending) + _nbytes(tensors) > max_shard_size:
+                    flush()
+                pending.update(tensors)
+    flush()
+
+    weight_map = {}
+    for i, names in enumerate(written):
+        shard_name = f"model-{i + 1:05d}-of-{len(written):05d}.safetensors"
+        os.replace(output_dir / f"part-{i:05d}.safetensors", output_dir / shard_name)
+        weight_map.update(dict.fromkeys(names, shard_name))
+    return dict(sorted(weight_map.items()))
+
+
+def _int8_tensors(
+    layer: str,
+    payload: torch.Tensor,
+    ggml_format: GGMLFormat,
+    scale_dtype: torch.dtype,
+    device: str,
+) -> dict[str, torch.Tensor]:
+    """A layer's GGML payload as compressed-tensors ``weight_packed``, ``weight_scale`` and shape."""
+    shape = torch.tensor((*payload.shape[:-2], payload.shape[-2] * ggml_format.block_size))
+    levels, steps = ggml_format.decode_int8(payload.to(device), shape)
+    return {
+        # compressed-tensors packs 8-bit values offset by 128, four little-endian per int32.
+        f"{layer}.weight_packed": (levels.view(torch.uint8) ^ 0x80).view(torch.int32).cpu(),
+        f"{layer}.weight_scale": steps.to(scale_dtype).cpu(),
+        f"{layer}.weight_shape": shape,
+    }
+
+
+def _nbytes(tensors: dict[str, torch.Tensor]) -> int:
+    return sum(t.numel() * t.element_size() for t in tensors.values())
+
+
+def _compressed_tensors_config(layer_formats: dict[str, GGMLFormat]) -> dict:
+    """One config group per INT8 group size, targeting its layers by name.
+
+    vLLM maps explicit layer names onto its own modules, but not ``re:`` patterns.
+    """
+    layers_by_group_size: dict[int, list[str]] = {}
+    for layer, ggml_format in sorted(layer_formats.items()):
+        layers_by_group_size.setdefault(ggml_format.int8_group_size, []).append(layer)
+    config_groups = {
+        f"group_{i}": {
+            "targets": layers,
+            "weights": {
+                "num_bits": 8,
+                "type": "int",
+                "symmetric": True,
+                "strategy": "group",
+                "group_size": group_size,
+                "dynamic": False,
+            },
+            "input_activations": None,
+            "output_activations": None,
+            "format": "pack-quantized",
+        }
+        for i, (group_size, layers) in enumerate(sorted(layers_by_group_size.items()))
+    }
+    return {
+        "quant_method": "compressed-tensors",
+        "format": "pack-quantized",
+        "quantization_status": "compressed",
+        "config_groups": config_groups,
+        "ignore": [],
+    }
+
+
+def main():
+    parser = argparse.ArgumentParser(
+        description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter
+    )
+    parser.add_argument("ggml_dir", help="Checkpoint exported with a GGML recipe (IQ, q8_0).")
+    parser.add_argument("output_dir", help="Where to write the INT8 checkpoint; must be empty.")
+    parser.add_argument(
+        "--max_shard_size_gb",
+        type=float,
+        default=10,
+        help="Target size of each output shard in GiB; a larger tensor gets a shard of its own.",
+    )
+    args = parser.parse_args()
+    convert_ggml_export_to_int8(
+        args.ggml_dir, args.output_dir, max_shard_size=int(args.max_shard_size_gb * 2**30)
+    )
+    print(f"INT8 checkpoint written to {args.output_dir}")
+
+
+if __name__ == "__main__":
+    main()

modelopt/torch/quantization/ggml/common.py

@@ -247,6 +247,11 @@ class GGMLFormat:
     # its search temporaries, decoding runs every forward and is bounded by kernel launches.
     block_chunk_size: int
     decode_chunk_size: int
+    # Every decoded weight is an int8 level times a float32 step shared by ``int8_group_size``
+    # weights. ``unpack_int8`` returns both for a chunk of blocks, and the torch decoder
+    # multiplies them out, so the integer view and the decoding cannot disagree.
+    int8_group_size: int
+    unpack_int8: Callable[[torch.Tensor], tuple[torch.Tensor, torch.Tensor]]
     # Whether ``quantize`` takes a per-input-column ``importance`` that weights its search.
     weighted: bool = False
 
@@ -255,6 +260,36 @@ def effective_bits(self) -> float:
         """Packed storage cost per weight."""
         return self.block_bytes * 8 / self.block_size
 
+    def decode_int8(
+        self, packed_weights: torch.Tensor, weight_shape: torch.Tensor
+    ) -> tuple[torch.Tensor, torch.Tensor]:
+        """Decode a payload into int8 levels and the float32 step of each ``int8_group_size``.
+
+        ``levels * steps`` (each step repeated over its group) equals ``dequantize`` in float32
+        exactly, so a weight-only integer kernel with these group scales reproduces the format.
+
+        Returns:
+            ``levels`` shaped like the weight and ``steps`` shaped
+            ``[*weight_shape[:-1], weight_shape[-1] // int8_group_size]``.
+        """
+        shape = validate_packed_weights(
+            packed_weights,
+            weight_shape,
+            block_bytes=self.block_bytes,
+            block_size=self.block_size,
+            format_name=self.name.upper(),
+        )
+        blocks = packed_weights.contiguous().reshape(-1, self.block_bytes)
+        groups = self.block_size // self.int8_group_size
+        levels = torch.empty((len(blocks), self.block_size), dtype=torch.int8, device=blocks.device)
+        steps = torch.empty((len(blocks), groups), dtype=torch.float32, device=blocks.device)
+        for start in range(0, len(blocks), self.decode_chunk_size):
+            stop = start + self.decode_chunk_size
+            chunk_levels, chunk_steps = self.unpack_int8(blocks[start:stop])
+            levels[start:stop] = chunk_levels
+            steps[start:stop] = chunk_steps
+        return levels.reshape(shape), steps.reshape(*shape[:-1], -1)
+
     def fake_quant(
         self,
         inputs: torch.Tensor,
@@ -354,6 +389,26 @@ def decoding(storage_dtype):
         return None
 
 
+def dequantize_blocks(
+    blocks: torch.Tensor,
+    unpack_int8: Callable[[torch.Tensor], tuple[torch.Tensor, torch.Tensor]],
+    *,
+    block_size: int,
+    group_size: int,
+    dtype: torch.dtype,
+    block_chunk_size: int,
+) -> torch.Tensor:
+    """Decode ``[blocks, block_bytes]`` payload rows a chunk at a time as ``levels * steps``."""
+    decoded = torch.empty((len(blocks), block_size), dtype=dtype, device=blocks.device)
+    for start in range(0, len(blocks), block_chunk_size):
+        stop = start + block_chunk_size
+        levels, steps = unpack_int8(blocks[start:stop])
+        decoded[start:stop] = (
+            levels.reshape(len(levels), -1, group_size) * steps.unsqueeze(-1)
+        ).reshape(-1, block_size)
+    return decoded
+
+
 def narrow_to_float32(blocks: torch.Tensor) -> torch.Tensor:
     """Narrow ``blocks`` to float32 the way the CUDA ``load_float`` helper does.