fix: handle subnormals in both float codecs and round ties to even

[svc-finitelabs]

Part 1 of FL-16: subnormals in all four codecs, and round-half-to-even for the tie
band. src/protobuf/init.lua is the only source file touched. Part 2 landed in #27.

The defects

Every figure below is measured against the oracle, not read off the source.

Decode, both widths. The result was built as 1 + m / 2^k unconditionally, which
is only correct for a normal. A subnormal has no implicit leading one and its exponent
is pinned at the minimum rather than sitting at the bias.

wire	true value	previous decode
float 01000000	1.401298464324817e-45	5.8774724547606697e-39
float FFFF7F00	1.1754942106924411e-38	1.1754942807573643e-38
double 0100000000000000	5e-324	1.1125369292536007e-308

Encode, both widths. The clamp caught only e < 0, so everything below the
smallest normal flushed to zero while e == 0 fell through into the normal path and
emitted a plausible looking wrong subnormal.

Rounding, not confined to subnormals. math.floor(x + 0.5) is
round-half-away-from-zero; IEEE 754 rounds half to even. 291 of the assertions that
fail on the previous code are exact midpoints in the ordinary normal range, one per
binade in each direction. One round_half_even helper serves both bands, which is the
argument for taking them together.

Two corrections to the ticket description

Worth flagging because both are load-bearing for how the fix is written.

1. Trap 1 is retired, as Derek noted on the ticket: math_ldexp's fallback now
   steps its scale in normal-range chunks, so the math_ldexp(math_ldexp(v, 537), 537)
   workaround is unnecessary. math_ldexp(value, 1074) is exact in one call.
2. Trap 2 is now obsolete, which I measured rather than assumed. The description
   says the frexp fallback returns -1021 for the largest subnormal double where native
   returns -1022, so classifying by frexp exponent would break 5.3 and 5.4 only. That
   was true when the description was written. fix: correct the frexp fallback and drive it from generated vectors #26 rewrote the fallback, and it now
   agrees with native on both boundaries at both widths. The fix still classifies by
   magnitude, but for the plainer reason that the boundary then does not depend on
   which frexp is bound at all.

Coverage

test/float_vectors_test.lua, 6525 assertions over 1183 vectors, enumerated rather
than sampled so the corpus is byte-identical on every host. Two oracles, neither a
hand-written literal: ctypes.c_float for the double-to-float narrowing, which is the
cast the hardware performs, and struct.pack for the bytes.

Per-bucket counts are printed every run, so an arm that stops running is visible:

  f32 subnormal           251 vectors,  1316 assertions
  f32 underflow            15 vectors,    90 assertions
  f32 normal              568 vectors,  3392 assertions
  f32 overflow             10 vectors,    52 assertions
  f64 subnormal           176 vectors,   880 assertions
  f64 normal              146 vectors,   730 assertions

Against the previous code: f32 subnormal 502 failures, f32 normal 291, f64 subnormal
352, identical on all four interpreters I have locally. Everything else was already
green, including the overflow band, which #25 fixed.

make gen-float-vectors and make check-float-vectors, mirroring the wire vectors:
goldens checked in so make test runs on a bare clone with no Python, drift check
plus an independent recomputation of every expectation, wired into CI as its own step
in the check job, and deliberately not part of make check.

Three things the harness has that are not obvious

1. Each vector carries the exact rational its decimal literal must parse to, and
   the suite asserts value * 2 ^ scale[1] * 2 ^ scale[2] == units before using it.
   No assertion on the codec can pin this: a midpoint that reads back as one of its
   neighbours still encodes to the golden, so it would pass while testing nothing. Two
   factors because a subnormal double needs a scale of 1074 and 2 ^ 1074 is infinity.
2. Goldens are hex, decoded at load time, not decimal escapes. \10 followed by a
   "0" byte reads back as \100, which corrupted three vectors in fix: sign the 32-bit codecs and cover the wire format against the reference #27.
3. The generator asserts each vector's name matches the band its value actually
   lands in. This caught four of my own mislabellings, including 1e-308, which is a
   subnormal double and was filed under "normal". Left alone, the bucket table would
   have reported the wrong arm as covered.

A finding from running real 5.1

The first corpus emitted negative zero as the literal -0.0. Lua 5.1 constant-folds
that to +0.0, so the suite had 11 failures there and nowhere else, all of them the
corpus being wrong rather than the codec. Now emitted as (-1 / math.huge), which is
what src/protobuf/init.lua already does for its own NEG_ZERO and for the same
reason. The literal guard could not catch this one: for zero it is vacuous.

Verification

Every new assertion was demonstrated red before the fix, which is the standard the
ticket asks for.

• The generated suite, per-bucket counts above, on Lua 5.1, 5.4, 5.5 and LuaJIT
  2.1, in both math modes.
• The in-module assertions, by applying only the new selftest() group to main's
  codecs: 11 of 13 fail. The two that pass are cases where round-half-up and
  round-half-even agree, and both are killed by mutation instead.
• 17 mutants, 16 killed, each required to turn both the Lua suite and the Python
  checker red, with an anchor count asserted as exactly 1 before every substitution and
  a content hash asserted after every revert. Two of the 17 mutate the corpus rather
  than the source, to prove the literal guard and the golden readback are live.
• The one survivor is an equivalent mutant, which I proved rather than assumed.
  value <= FLOAT_MIN_NORMAL instead of < differs on exactly one input, 2^-126, and
  both spellings emit 00008000: the subnormal branch computes a mantissa of 2^23,
  which carries into an exponent of 1 and gives the same answer. The input is in the
  corpus, so this is not a coverage gap.

Also green: the full suite 7/7 on those four interpreters, make check-wire-vectors
against the reference implementation, format-check, check-types, check-schema,
typecheck and luacheck.

Not run locally, stated rather than implied: real 5.2, 5.3 and LuaJIT 2.0, which
are 3 of the 6 matrix entries. 5.5 is not in the matrix and stands in for the 5.3/5.4
arm as a second number model. make build also did not run: amalg is not installed for
an interpreter I have here. The amalgamation consumes src/ only, so nothing under
test/ can reach the artifact, but the Build Combined Module job is the proof.

Two judgment calls

• test/generated/float_vectors.lua is 214 KB, most of it the 508 normal-band
  midpoints, one per binade in each rounding direction. The defect is
  exponent-independent, so a sample would find it, but the frexp path is not, and fix: correct the frexp fallback and drive it from generated vectors #26's
  defects were exponent-specific. Happy to thin it if you would rather.
• A small subnormal group went into selftest() as well, 13 assertions. Your
  comment scoped generated corpora to test/ and said selftest() stays as it is, so
  this is the one place I went slightly past it: selftest() is the only suite that
  runs on a real controller against the real LuaJIT, and the existing float section
  already carries boundary and carry cases in exactly this shape. Say the word and it
  comes out.

Refs FL-16

[derek-miller]

The codec change is correct and the coverage is the right shape. I checked it with a corpus of my own rather than yours: 29,041 values from a fresh seed (random f32 subnormals with their half-ulp midpoints, ties built as exact midpoints between adjacent float32s, the overflow band around FLT_MAX + 2^103, random f64 subnormals, random normals across the exponent range), oracle ctypes.c_float + struct.pack, values carried as exact m * 2^e so strtod is out of the loop. 116,164 assertions, 0 failures on Lua 5.1, 5.4 and LuaJIT 2.1, each with native and cleared math.frexp/math.ldexp. The same corpus against main fails 20,693, so it sees what it is meant to see. make check, run_tests.sh on the three interpreters, check-float-vectors, and gen-float-vectors drift are all clean here too.

The selftest() additions stay: that is the suite that runs on a controller, and 13 boundary cases is the existing float section's shape. The corpus size is fine.

Three edits, all small:

1. Two comments narrate the old bug rather than the invariant. decode_float: keep "A subnormal carries no implicit leading one and sits at the minimum exponent rather than at the bias." and drop the "Reading one as ... made 01000000 come back as 5.88e-39 instead of 1.40e-45" sentence. decode_double: "Subnormal, as in decode_float." and drop "0100000000000000 read as 1.11e-308, not 5e-324." The wrong values belong in the PR and the ticket, not in the source.

2. CLAUDE.md "What is not implemented" now carries a bullet that says something is implemented. Delete the float/double bullet from that list rather than inverting it; README's table and the paragraph under it already state the support, and the section should stay a list of gaps.

3. One line on ascending's table branch, carried over from #29. The Int64 comparator orders {high, low} by the unsigned high word, so negative 64-bit keys sort after positive ones; that is deliberate (stable, and no consumer depends on numeric order) but not obvious. Since this is the next commit on init.lua, add a one-line comment there: -- Unsigned high word first: negative 64-bit keys sort last, which is stable and all that matters. or similar.

Nothing else. Push those and I will approve.

[derek-miller]

Approving at 2958710. The three edits are as asked and nothing else moved; the codec verification from the previous review stands (116,164 oracle assertions, 0 failures on 5.1/5.4/LuaJIT in both math modes; 20,693 against main). Merging when the re-run is green.

[svc-finitelabs]

Thanks, and noting the approval landed while I was writing this. CI is green 8/8 at
2958710, so the re-run you were waiting on is done. Leaving the merge to you.

For the record, since two of the three edits were judgment calls:

1. Comments state the invariant only. decode_float keeps the leading-one and
   pinned-exponent sentence and drops the "01000000 came back as 5.88e-39" one;
   decode_double is down to "Subnormal, as in decode_float."
2. The gap bullet is deleted rather than inverted, and I checked the support was
   still stated elsewhere before removing it: README's type table reads "IEEE 754,
   including subnormals" for both widths, and the paragraph under it covers ties,
   flush-to-zero and overflow. What is not implemented is back to four entries.
3. ascending's table branch. Re-derived rather than transcribed, since it goes
   into the source as a claim: reproducing the comparator against bit64.from_number
   output, -1 is {0xFFFFFFFF, 0xFFFFFFFF} and -2147483648 is
   {0xFFFFFFFF, 0x80000000}, and both sort behind {0, 0x7FFFFFFF}. Your reading
   holds exactly. I wrote "deterministic, not numeric" rather than "stable", because
   stable already means something specific about sorts and the property that matters
   here is that the emission order is reproducible.

Re-ran everything after the edits rather than assuming comment changes are inert: full
suite 7/7 on 5.1, 5.4, 5.5 and LuaJIT 2.1 in both math modes, plus format-check,
luacheck, typecheck, check-float-vectors and check-wire-vectors. The wire
vectors matter for edit 3 specifically, since they assert map emission order.

One thing worth recording: your corpus is a better instrument than mine on one axis.
Carrying values as exact m * 2^e takes strtod out of the loop entirely, where mine
carries decimal literals plus a per-vector assertion that each parsed to the intended
rational. Mine has a hole yours does not, and it bit me: that guard is vacuous for zero,
which is how a -0.0 literal folded by Lua 5.1 survived into an earlier draft and
produced 11 failures on 5.1 alone. Say the word and I will switch the generator to the
m * 2^e form as a follow-up rather than carry two conventions.