From e31b3ac8257affe8dfd8e51e4fe5cddf7726bec6 Mon Sep 17 00:00:00 2001 From: Chenjie Luo Date: Sun, 25 Oct 2026 06:51:32 +0000 Subject: [PATCH 1/2] Document installing flash-linear-attention for linear-attention models Transformers runs the linear-attention layers of Qwen3-Next, Qwen3.5, Qwen3.8, GLM-5.x Flash, Kimi Linear and OLMo Hybrid through its PyTorch reference kernels unless the flash-linear-attention kernels are importable. Those kernels are in fla-core, which installing flash-linear-attention with --no-deps leaves out. Note the install in the hf_ptq README and the ptq skill. Co-Authored-By: Claude Opus 5.5 (1M context) Signed-off-by: Chenjie Luo --- examples/hf_ptq/README.md | 2 ++ plugins/modelopt/skills/ptq/SKILL.md | 1 + 2 files changed, 3 insertions(+) diff --git a/examples/hf_ptq/README.md b/examples/hf_ptq/README.md index 422729277e1..b79aa8f4914 100755 --- a/examples/hf_ptq/README.md +++ b/examples/hf_ptq/README.md @@ -175,6 +175,8 @@ scripts/huggingface_example.sh --model $HF_PATH --quant --tp [1|2|4|8] > *FP8 calibration over a large model with limited GPU memory is not recommended but possible with the [accelerate](https://huggingface.co/docs/accelerate/en/usage_guides/big_modeling) package. Please tune the device_map setting in [`example_utils.py`](./example_utils.py) if needed for model loading and the calibration process can be slow.* +> *Models with linear-attention layers (Qwen3-Next, Qwen3.5, Qwen3.8, GLM-5.x Flash, Kimi Linear, OLMo Hybrid) run those layers through Transformers' slow PyTorch reference kernels unless the [flash-linear-attention](https://github.com/fla-org/flash-linear-attention) kernels are installed. Install them without changing your torch and Transformers versions with `pip install --no-deps flash-linear-attention fla-core einops`; the kernels live in `fla-core`, so both packages are needed. On GLM-5.3-Flash over 4x GB300 they more than halved layerwise GPTQ calibration, mostly because the fused kernels need far less memory per sample, so a larger calibration batch fits.* + > *Huggingface models trained with `modelopt.torch.speculative` can be used as regular Huggingface models in PTQ. Note: there is a known issue with Huggingface models loaded across multiple GPUs for inference (i.e., "Expected all tensors to be on the same device, but found at least two devices..."). When encountered this error in PTQ of speculative decoding models, try reducing the number of GPUs used.* > *Calibration by default uses left padding_side for the Huggingface tokenizer as it usually leads to lower accuracy loss. The exported tokenizer files restores the default padding_side.* diff --git a/plugins/modelopt/skills/ptq/SKILL.md b/plugins/modelopt/skills/ptq/SKILL.md index 7b3521b9f36..291a6690915 100644 --- a/plugins/modelopt/skills/ptq/SKILL.md +++ b/plugins/modelopt/skills/ptq/SKILL.md @@ -191,6 +191,7 @@ Report the gate result before moving on. Follow the canonical report format and ## Common Pitfalls - **Model-specific dependencies**: Models with `trust_remote_code` may import packages not in the container (e.g., `mamba-ssm` for hybrid Mamba models). See Step 2.5. Use `EXTRA_PIP_DEPS` env var with the launcher, or install manually before running `hf_ptq.py` +- **Linear-attention models** (Qwen3-Next/3.5/3.8, GLM-5.x Flash, Kimi Linear, OLMo Hybrid) fall back to slow, memory-hungry PyTorch kernels without `pip install --no-deps flash-linear-attention fla-core einops` (kernels are in `fla-core`); install it before calibrating - **Transformers version**: New models may need a newer version of transformers than what's installed. Check `config.json` for `transformers_version`. In containers, beware of `PIP_CONSTRAINT` blocking upgrades — see `references/slurm-setup-ptq.md` for workarounds - **Gated datasets**: Some calibration datasets require HF authentication. Set `HF_TOKEN` in the job environment. Use `--dataset cnn_dailymail` only as the constrained-environment fallback described in Step 4, not as the preferred calibration set - **NFS root_squash + Docker**: See the common skill's `slurm-setup.md` section 5 From 43e746f9fea2b468acdc992b3cc827616b4cca7d Mon Sep 17 00:00:00 2001 From: Chenjie Luo Date: Sun, 25 Oct 2026 06:58:24 +0000 Subject: [PATCH 2/2] Recommend fla-core alone and note it is GPU-only Transformers binds the kernels from fla-core alone, which depends only on einops, while flash-linear-attention pins Transformers. With the kernels importable, Transformers also sends CPU tensors to them, which fails. Co-Authored-By: Claude Opus 5.5 (1M context) Signed-off-by: Chenjie Luo --- examples/hf_ptq/README.md | 2 +- plugins/modelopt/skills/ptq/SKILL.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/examples/hf_ptq/README.md b/examples/hf_ptq/README.md index b79aa8f4914..40fe0452151 100755 --- a/examples/hf_ptq/README.md +++ b/examples/hf_ptq/README.md @@ -175,7 +175,7 @@ scripts/huggingface_example.sh --model $HF_PATH --quant --tp [1|2|4|8] > *FP8 calibration over a large model with limited GPU memory is not recommended but possible with the [accelerate](https://huggingface.co/docs/accelerate/en/usage_guides/big_modeling) package. Please tune the device_map setting in [`example_utils.py`](./example_utils.py) if needed for model loading and the calibration process can be slow.* -> *Models with linear-attention layers (Qwen3-Next, Qwen3.5, Qwen3.8, GLM-5.x Flash, Kimi Linear, OLMo Hybrid) run those layers through Transformers' slow PyTorch reference kernels unless the [flash-linear-attention](https://github.com/fla-org/flash-linear-attention) kernels are installed. Install them without changing your torch and Transformers versions with `pip install --no-deps flash-linear-attention fla-core einops`; the kernels live in `fla-core`, so both packages are needed. On GLM-5.3-Flash over 4x GB300 they more than halved layerwise GPTQ calibration, mostly because the fused kernels need far less memory per sample, so a larger calibration batch fits.* +> *Models with linear-attention layers (Qwen3-Next, Qwen3.5, Qwen3.8, GLM-5.x Flash, Kimi Linear, OLMo Hybrid) run those layers through Transformers' slow PyTorch reference kernels unless the [flash-linear-attention](https://github.com/fla-org/flash-linear-attention) kernels are installed: `pip install fla-core`. Install it only where these models run on GPU, since Transformers then uses the kernels for CPU tensors too, and they fail there. On GLM-5.3-Flash over 4x GB300 they more than halved layerwise GPTQ calibration, mostly because the fused kernels need far less memory per sample, so a larger calibration batch fits.* > *Huggingface models trained with `modelopt.torch.speculative` can be used as regular Huggingface models in PTQ. Note: there is a known issue with Huggingface models loaded across multiple GPUs for inference (i.e., "Expected all tensors to be on the same device, but found at least two devices..."). When encountered this error in PTQ of speculative decoding models, try reducing the number of GPUs used.* diff --git a/plugins/modelopt/skills/ptq/SKILL.md b/plugins/modelopt/skills/ptq/SKILL.md index 291a6690915..96c82815afe 100644 --- a/plugins/modelopt/skills/ptq/SKILL.md +++ b/plugins/modelopt/skills/ptq/SKILL.md @@ -191,7 +191,7 @@ Report the gate result before moving on. Follow the canonical report format and ## Common Pitfalls - **Model-specific dependencies**: Models with `trust_remote_code` may import packages not in the container (e.g., `mamba-ssm` for hybrid Mamba models). See Step 2.5. Use `EXTRA_PIP_DEPS` env var with the launcher, or install manually before running `hf_ptq.py` -- **Linear-attention models** (Qwen3-Next/3.5/3.8, GLM-5.x Flash, Kimi Linear, OLMo Hybrid) fall back to slow, memory-hungry PyTorch kernels without `pip install --no-deps flash-linear-attention fla-core einops` (kernels are in `fla-core`); install it before calibrating +- **Linear-attention models** (Qwen3-Next/3.5/3.8, GLM-5.x Flash, Kimi Linear, OLMo Hybrid) fall back to slow, memory-hungry PyTorch kernels without `pip install fla-core`; install it before calibrating on GPU (it breaks CPU runs of these models) - **Transformers version**: New models may need a newer version of transformers than what's installed. Check `config.json` for `transformers_version`. In containers, beware of `PIP_CONSTRAINT` blocking upgrades — see `references/slurm-setup-ptq.md` for workarounds - **Gated datasets**: Some calibration datasets require HF authentication. Set `HF_TOKEN` in the job environment. Use `--dataset cnn_dailymail` only as the constrained-environment fallback described in Step 4, not as the preferred calibration set - **NFS root_squash + Docker**: See the common skill's `slurm-setup.md` section 5