[Qubitium]

Two independent EXL3 defects found while producing a 2.5 bpw checkpoint.

1. perf(exl3) — tile scratch is 2 GiB per stream

EXL3 allocates its tile scratch in _allocate_temp_buffers as

max_batch_size = 256
temp_costs = torch.empty((max_batch_size, 2, 65536 >> K), dtype=torch.half, ...)
temp_edges = torch.empty((max_batch_size, 256, 65536 >> K), dtype=torch.short, ...)

temp_edges is 2 GiB at K=2, and the buffer is keyed per CUDA stream while the quantizer runs one stream per worker thread (get_quant_stream). A wide run therefore holds several GiB of pure scratch per device.

Two concrete consequences observed:

• Quantizing a 2-layer, 256-hidden Ripple fixture needed ~6 GiB of transient memory before doing any useful work, so it could not run alongside another job on the same 48 GiB card.
• On a 65B model the scratch competes directly with the layer being quantized, which is what makes the low-bit path fail to start on constrained hosts.

The batch of 256 is chosen for launch occupancy (a 56-tile grid fills less than a quarter of the machine), so it should not simply be lowered for everyone.

Change

• GPTQMODEL_EXL3_TILE_BATCH sets the slot count. Default is 256 — behaviour is unchanged unless the variable is set. Values are clamped to [1, 256] and malformed values fall back to the default.
• The workspace registry key now includes the batch, so a process that changes the setting does not reuse a wrongly sized buffer.
• When a tile set exceeds the slots, _quantize_tiles_with_workspace splits the launch. Tiles carry their own trellis and each writes all 256 of its entries, so there is no cross-tile state and batching is exactly equivalent to one launch.

Verification

Bit-identical output on the Ripple fixture at 2.5 bpw EXL3:

tile batch 8 vs 32	0 of 81 tensors mismatched (torch.equal, shapes and dtypes identical)
saved config	bits=2.5, method=exl3, format=exl3, codebook=mcg
reload	forwards, logits (4, 64, 256) float16

Tests (tests/test_exl3_workspace.py, 4 passed / 1 skipped, ruff clean):

• default is 256 and is unaffected by an unset variable;
• 8/1/0/-5/4096/junk clamp or fall back as documented;
• 5 tiles at 2 slots produce launches of 2 + 2 + 1, and a set that fits is not split;
• existing per-stream isolation still holds (buffer keyed by device, K, stream and batch).

2. fix(exl3) — bf16 checkpoints are unloadable

LinearEXL3 asserts the unpacked sign vectors are torch.half:

if suh is not None: assert suh.dtype == torch.half, "suh is wrong datatype"
if svh is not None: assert svh.dtype == torch.half, "svh is wrong datatype"

The Hadamard kernels are typed for fp16, and the checkpoint stores them that way. But nn.Module._apply casts every floating-point buffer to the model dtype, so loading with dtype=bfloat16 produces bf16 sign vectors and the first forward dies with AssertionError: suh is wrong datatype.

The quantizer writes bf16 checkpoints without complaint, so the defect only appears on reload — when the saved artifact is supposed to be trustworthy. Reproduced on a 2-layer Ripple fixture:

reload dtype	before	after
float16	forwards	forwards
bfloat16 (the checkpoint default)	AssertionError: suh is wrong datatype	forwards and generates

Change

ExllamaV3Linear restores the kernel dtypes after every cast, in _apply and again in _ensure_inner/post_init, so the invariant holds however the module is materialized. Packed su/sv are int16 and are untouched by the cast.

Verification

On the fixture checkpoint:

• bf16 reload forwards and generates;
• repeated forwards are bit-identical;
• fp16-vs-bf16 logits differ by 0.018 (bf16 rounding on the non-quantized tensors only).

Tests: tests/test_exl3_buffer_dtype.py (4 passed, hardware-free) covers bf16 and fp32 casts, int16 preservation, and sign-value preservation. tests/test_exllamav3_meta_import.py, test_exl3_rng_isolation.py, test_exl3_workspace.py still pass (7 passed / 4 skipped). Ruff clean.

Notes

Both changes are prerequisites for running EXL3 on a 65B with a per-device memory budget. Neither alters the quantizer's numerics: the tile batching is bit-identical, and the dtype fix only affects buffer dtypes on load.