Systematic integer truncation hardening

[krishna28238-arch]

Systematic integer truncation hardening
Summary

This PR eliminates the remaining cases where a value derived from untrusted data can be truncated by a narrowing integer conversion before or during validation.

Motivating pattern (the class this PR removes): a count of 260 stored as uint8_t truncates to 4; a "count <= limit" check on the truncated value passes while a loop iterates 260 times — validation bypass by truncation/wrap-around.
The audit

A systematic sweep of the whole codebase:

every explicit (uint8_t)/(uint16_t)/(int) narrowing cast in src/, include/, apps/;
every uint8_t/uint16_t struct field that can receive file-derived counts or sizes, traced to its data source;
complete GCC builds with -Wconversion -Wsign-conversion (with and without AVIF_ENABLE_EXPERIMENTAL_MINI/EXTENDED_PIXI, apps and examples included): all 167 warnings triaged;
parser-by-parser review of every count/size read from file data (iloc, iinf, iref, ipma, pixi, a1lx, lsel, sato, grid payload, tmap, a1op, track path …);
allocation-site audit for overflow-before-multiply.

Most sites are already guarded (see "Notable sites verified safe" below). Three instances of the pattern survived and are fixed here.
Fixes

1. Sample Transform input item count was uint8_t (validation on a wrapped value)

avifDecoderData::sampleTransformNumInputImageItems (src/read.c) counted 'dimg' inputs of a 'sato' item into a uint8_t. A crafted file with 259 input items wrapped the counter to 3, bypassing both the "at most 32 input items" format check and the implementation limit check. Release builds then rejected the file with a misleading error (DECODE_SAMPLE_TRANSFORM_FAILED "not a supported image type" or INTERNAL_ERROR) instead of the intended clean failure; debug builds abort on the later invariant.

The field is now uint32_t so the checks validate the true count.
The internal avifImageApplyExpression() / avifImageApplyOperations() numInputImageItems parameters are widened from uint8_t to uint32_t so the pattern cannot re-enter through those interfaces.
New regression test AvifDecodeTest.SampleTransformTooManyInputItems crafts the 259-input file from weld_sato_12B_8B_q0.avif (by patching the iref/dimg box) and asserts the clean AVIF_RESULT_BMFF_PARSE_FAILED with the "too many input items … got 259" diagnostic. Verified to fail on the vulnerable code.

2. OBU size computed through (int) cast of size_t

av1SequenceHeaderParse() (src/obu.c) computed the size of an OBU without a size field as (int)obus.size - 1 - obu_extension_flag. For payloads larger than INT_MAX this is an implementation-defined conversion with signed overflow on the way (UB; UBSan abort). The computation is now done in unsigned 64-bit arithmetic with an explicit bound check before the narrowing store.
3. a1lx layer sizes silently truncated to 32 bits

The 'a1lx' writer (src/write.c) stored each layer's payload size_t size into a uint32_t without a check, so a layer larger than 4 GiB would be written with a wrapped layer size, signaling wrong layer boundaries to decoders. It now refuses to encode such layers (AVIF_ASSERT_OR_RETURN(size <= UINT32_MAX)).
Changes withdrawn after review

An earlier revision of this PR also enabled -Wconversion for C sources on GCC and added the explicit casts needed to keep that build warning-free. Per review feedback, all of those changes are withdrawn in 20f4a67:

the casts on masked byte extraction (data[i] = (x >> n) & 0xff) were unnecessary because the expression is always in the range 0..0xff and fits the uint8_t destination;
the redundant float casts were unnecessary because when the first operand of an arithmetic expression is float, the other operand is implicitly converted to float (e.g. (float)f.n / f.d, or rgbSumLinear / ((size_t)width * height) where rgbSumLinear is already float);
the -Wconversion flag and the libyuv -Wno-conversion scoping in CMakeLists.txt go away with them, restoring -Wall -Wextra.

Only the three behavioral fixes above and the regression test remain.
Notable sites verified safe (no change needed)

All 16-bit item ID / item_count / entry_count writes in avifEncoderFinish are selected by the largeItemIDs invariant (lastItemID > UINT16_MAX || items.count > UINT16_MAX, PR #3357) or asserted.
Grid payload: 32-bit fields chosen when width/height > 65535; rows/columns validated ≤ 256 (and ≤ 65535 cells).
iloc/iref/ipma/pixi/a1op/lsel parse loops: wire-width-bounded counts with offset/size overflow checks.
Image/RGB/gain-map allocations: explicit UINT32_MAX / channelSize, PTRDIFF_MAX / rowBytes, SIZE_MAX / sizeof guards.
libyuv int parameters: dimension-limit checks plus the conservative 16384-pixel source cap for ScalePlane.

Verification

Full build with the upstream flags (GCC, library + apps + examples, with and without experimental features): 0 warnings.
Full ctest: 51/51 pass (including the 65538-item-ID grid test and the new regression test).
A/B check: reverting only the uint32_t field widening makes the new test fail with the old misleading error; the fix restores the clean rejection.
clang-format and cmake-format: clean.

[wantehchang]

Krishna: Thank you for your interest in libavif. I will need to decline most of the changes in this pull request.

For example, the casts added to this file are not necessary because the expressions in question are in the range of 0..0xff and fit in the uint8_t type. In this case we can choose to omit the explicit uint8_t casts for brevity.

[krishna28238-arch]

tell me the changes and i will change those

[wantehchang]

Here is another example of omitting unnecessary explicit casts. If the first operand is float, then the second operand is implicitly converted to float. So one explicit cast to float suffices. We take advantage of this rule to minimize the number of explicit casts and avoid visual clutter in the source code.

[wantehchang]

Similarly, since the rgbSumLinear variable is of the float type, we omit the explicit float cast on the denominator deliberately.

[krishna28238-arch]

Thanks for the review! Commit 3258cc1 addresses the comments:

 avifEncoderDataCreateItem() now returns an avifResult and outputs the new item through an out-parameter, so the "no item ID left" check is an AVIF_ASSERT_OR_RETURN() invariant (the comment is kept as suggested).
 largeItemIDs in avifEncoderFinish() now also depends on encoder->data->items.count, so the 'iloc' item_count and 'iinf' entry_count fields are no longer silently truncated. The same was done in avifEncoderWriteTrackMetaBox() for metadataItemCount, as suggested.
 Commit-like sentences were removed from comments.

While auditing every 16-bit field written by the encoder, I found one more silent truncation: the 'iref' reference_count field is 16-bit in both versions, so a 256x256 grid (65536 cells) cannot be represented. avifEncoderAddImageGrid() now rejects it with AVIF_RESULT_INVALID_IMAGE_GRID (new test GridApiTest.CellCountExceeding16BitReferenceCount). Happy to drop that change if you prefer to keep this PR minimal.

[krishna28238-arch]

No, it is not recent. The very first commit of libavif (444f051, Jan 2019)
already had a findItemID() linear scan over the item array in src/read.c.
It became avifMetaFindItem() in 2020 (9f2b87b, "Move contents of meta boxes
into an avifMeta structure") and was renamed avifMetaFindOrCreateItem()
in #1757 (Nov 2023, "Do not store item pointers until all items are
created"), which also added a few call sites that previously stored item
pointers. But the iinf/iloc/iref entries have performed a linear
lookup per entry (i.e. O(N^2) parsing for N items) since the beginnin