Skip to content

fix: convert to the nearest f64 - #45

Open
Nick-Lafky wants to merge 1 commit into
loyd:masterfrom
Nick-Lafky:f64-nearest
Open

Nick-Lafky wants to merge 1 commit into
loyd:masterfrom
Nick-Lafky:f64-nearest

Conversation

@Nick-Lafky

@Nick-Lafky Nick-Lafky commented Oct 12, 2026 •

Copy link
Copy Markdown

f64::from(fp) currently splits the number into integral and fractional parts, converts each to f64, and adds them. That's three roundings, and the result is often one ULP away from the nearest double:

let x = FixedPoint::<i64, U9>::from_str_exact("1.0353")?;
assert_eq!(f64::from(x), 1.0353); // fails: 1.0352999999999999

For 4-decimal values in [1, 2) about 17% come out wrong this way.

This change makes the conversion return the nearest f64:

  • If the mantissa is at most 2^53 and the precision is at most 22, both operands are exact doubles, so a single division is correctly rounded (Clinger's fast path). This covers every i16/i32 layout and nearly all practical i64 ones.
  • Otherwise the number is written as <mantissa>e<exponent> into a stack buffer and parsed with str::parse::<f64>. The standard library's parser is the only stable entry point to a correctly rounded decimal-to-binary conversion (its dec2flt internals aren't public), and it's the same approach java.math.BigDecimal#doubleValue used until JDK 21.

Why it's strictly better: for every input the new result is the nearest double, and the old result was either the same or farther away. There's no input where the old code was closer.

Where it's worse: the fallback is slower than the division (it formats and parses a short string) and runs for i128 values with mantissas above 2^53 or any precision above 22. The fast path is unchanged in cost. And since results can move by one ULP for existing users, this is a behaviour change even though it's a fix. You may want to call it out in the release notes beyond the changelog line.

The fallback doesn't go through Display, because Stringify overflows for MIN at precision 0 ((inner / COEF).abs()) and for negative 39-digit i128 values (fractional + COEF).

Tests: values the old sum gets wrong on both the i64 and i128 layouts, both sides of the 2^53 mantissa bound, a precision above 22, and MIN/MAX for each layout.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant