feat(gguf): implement Q8_0 dequantization - #368
Open
Cyber-Marty wants to merge 1 commit into
Open
Conversation
dequant.py declared Q8_0 in BLOCK_SHAPE, GGML_NAME and __all__ but had no dequant_q8_0, so dequantize() raised NotImplementedError on the format that the most widely distributed Gemma-4 GGUFs (unsloth's UD-*_XL dynamic quants) use on attention projections, the dense FFN and token_embd -- 237 of 658 tensors in the 26B-A4B UD-Q6_K_XL file. dequant_q8_0 follows the ggml block_q8_0 layout: fp16 scale d + 32 int8 quants, w = d*q with no offset. Tests round-trip through a reference quantizer mirroring quantize_row_q8_0 (d = max|w|/127, q = round(w/d)), bounding the error at half a quantization step, plus a hand-computed exact-values case and a dequantize() dispatch check. All pure torch / CPU. Fixes FlashML-org#358
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
dequant.pydeclared Q8_0 inBLOCK_SHAPE,GGML_NAMEand__all__, but there was nodequant_q8_0, sodequantize()raisedNotImplementedErroron the format that the mostwidely distributed Gemma-4 GGUFs use: unsloth's
UD-*_XLdynamic quants place Q8_0 on theattention projections, the dense FFN and
token_embd— 237 of 658 tensors ingemma-4-26B-A4B-it-UD-Q6_K_XL.gguf(see #358).Fix
dequant_q8_0follows the ggmlblock_q8_0layout: per 32-elem block, fp16 scaledq,w = d*qwith no offset (unlike Q4_0's(q-8)*d— ggml'squantize_row_q8_0storesq = round(w/d),d = max|w|/127).Registered in
_DEQUANTand__all__; module docstring updated to list Q8_0. Noexisting code path changes — Q4_0 / Q6_K / F32 / F16 / BF16 untouched.
Tests
New
tests/models/test_dequant_q8_0.py(4 cases, pure torch / CPU):quantize_row_q8_0) →dequant_q8_0→ reconstruction error bounded by half a quantization step;d=0.5, quants incl. -128/127/0) decodes exactly;dequantize(raw, GGML_Q8_0)routes instead of raising;BLOCK_SHAPE[Q8_0] == (32, 34)androw_bytesmath unchanged.tests/models/test_gemma4_gguf_config.py+test_dequant_q8_0.py+test_cpu_moe_q4_0.py:7 passed, 5 skipped (CUDA-only cases, by design).
Fixes #358