Fix float16_to_float(vint4) decoding every negative half as NaN on F16C builds - #668
Fix float16_to_float(vint4) decoding every negative half as NaN on F16C builds#668lenamonj wants to merge 1 commit into
Conversation
…6C builds The F16C path narrowed the four lanes with _mm_packs_epi32, which saturates signed, so any half code with the sign bit set (0x8000 to 0xFFFF, a lane value above 0x7FFF) became 0x7FFF and decoded as NaN. Pack with _mm_packus_epi32 instead: the lanes hold zero-extended 16-bit codes, so the unsigned pack keeps every one of them. The unit test now covers negative halves, which the AVX2 unit binary fails without this change.
|
Is this behavior actually reachable? Do you have any cases that trigger astcenc to actually try to convert a negative value into fp16? The only case where I think its possible gets clamped to zero before reaching the fp16 path. |
|
Yes, through the input side rather than the encoder's float-to-fp16 path: Reproducer: a 16x16 R16G16B16A16_FLOAT DDS whose texels cycle -1.0, 0.5, -0.5, 1.0, compressed with import struct
hdr = struct.pack('<7I', 124, 0x1007, 16, 16, 0, 0, 0) + b'\x00' * 44
hdr += struct.pack('<8I', 32, 0x4, 0x30315844, 0, 0, 0, 0, 0) + struct.pack('<5I', 0x1000, 0, 0, 0, 0)
dx10 = struct.pack('<5I', 10, 3, 0, 1, 0)
pattern = [0xBC00, 0x3800, 0xB800, 0x3C00]
body = b''.join(struct.pack('<H', pattern[(x + y + c) % 4]) for y in range(16) for x in range(16) for c in range(4))
open('f16neg.dds', 'wb').write(b'DDS ' + hdr + dx10 + body) |
On an F16C build,
float16_to_float(vint4)narrows the four lanes with_mm_packs_epi32, which saturates signed. A lane holding a half code with the sign bit set is a value above 0x7FFF, so it saturates to 0x7FFF and_mm_cvtph_psdecodes it as NaN. Every negative half decodes as NaN on AVX2 builds, while the SSE4.1, SSE2 and no-intrinsics backends decode it correctly.The lanes hold zero-extended 16-bit codes, so
_mm_packus_epi32keeps all of them.Test:
SuiteVfloat4.float16_to_floatnow includes negative halves.test-unit-avx2fails on main (all four lanes NaN) and passes with the change; the other unit binaries pass both ways. The x64 CI recipe was run locally: release and debug builds, ctest, functional and image tests for all four encoders, example build.