Skip to content

Fix float16_to_float(vint4) decoding every negative half as NaN on F16C builds - #668

Open
lenamonj wants to merge 1 commit into
ARM-software:mainfrom
lenamonj:f16c-negative-half
Open

Fix float16_to_float(vint4) decoding every negative half as NaN on F16C builds#668
lenamonj wants to merge 1 commit into
ARM-software:mainfrom
lenamonj:f16c-negative-half

Conversation

@lenamonj

@lenamonj lenamonj commented Sep 8, 2026

Copy link
Copy Markdown

On an F16C build, float16_to_float(vint4) narrows the four lanes with _mm_packs_epi32, which saturates signed. A lane holding a half code with the sign bit set is a value above 0x7FFF, so it saturates to 0x7FFF and _mm_cvtph_ps decodes it as NaN. Every negative half decodes as NaN on AVX2 builds, while the SSE4.1, SSE2 and no-intrinsics backends decode it correctly.

The lanes hold zero-extended 16-bit codes, so _mm_packus_epi32 keeps all of them.

Test: SuiteVfloat4.float16_to_float now includes negative halves. test-unit-avx2 fails on main (all four lanes NaN) and passes with the change; the other unit binaries pass both ways. The x64 CI recipe was run locally: release and debug builds, ctest, functional and image tests for all four encoders, example build.

…6C builds

The F16C path narrowed the four lanes with _mm_packs_epi32, which saturates
signed, so any half code with the sign bit set (0x8000 to 0xFFFF, a lane
value above 0x7FFF) became 0x7FFF and decoded as NaN. Pack with
_mm_packus_epi32 instead: the lanes hold zero-extended 16-bit codes, so the
unsigned pack keeps every one of them. The unit test now covers negative
halves, which the AVX2 unit binary fails without this change.
@solidpixel

solidpixel commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Is this behavior actually reachable? Do you have any cases that trigger astcenc to actually try to convert a negative value into fp16?

The only case where I think its possible gets clamped to zero before reaching the fp16 path.

@lenamonj

lenamonj commented Sep 8, 2026

Copy link
Copy Markdown
Author

Yes, through the input side rather than the encoder's float-to-fp16 path: load_texel_f16 in astcenc_image.cpp decodes half-float input texels with float16_to_float(vint4), so a half-float image with negative components reaches it as raw codes with the sign bit set. With no preprocessing the later clamp hides the difference, which matches what you describe. -pp-normalize and -pp-premultiply use the decoded values before that clamp, so a half-float normal map with negative components compresses differently on AVX2.

Reproducer: a 16x16 R16G16B16A16_FLOAT DDS whose texels cycle -1.0, 0.5, -0.5, 1.0, compressed with -cH in.dds out.astc 4x4 -medium -pp-normalize. On main, astcenc-avx2 gives 9e434c47..., astcenc-sse4.1 and astcenc-none give 11f96a63...; with this change all three give 11f96a63.... Same with -pp-premultiply (619b74c7... versus 0a8afc84...); identical everywhere with no preprocessing.

import struct
hdr = struct.pack('<7I', 124, 0x1007, 16, 16, 0, 0, 0) + b'\x00' * 44
hdr += struct.pack('<8I', 32, 0x4, 0x30315844, 0, 0, 0, 0, 0) + struct.pack('<5I', 0x1000, 0, 0, 0, 0)
dx10 = struct.pack('<5I', 10, 3, 0, 1, 0)
pattern = [0xBC00, 0x3800, 0xB800, 0x3C00]
body = b''.join(struct.pack('<H', pattern[(x + y + c) % 4]) for y in range(16) for x in range(16) for c in range(4))
open('f16neg.dds', 'wb').write(b'DDS ' + hdr + dx10 + body)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants