Skip to content

Instantiate the generic functions compared inside of another while its bounds are checked - #444

Open
Rigidity wants to merge 12 commits into
mainfrom
rigidity/inference-restructure-13d
Open

Rigidity wants to merge 12 commits into
mainfrom
rigidity/inference-restructure-13d

Conversation

@Rigidity

@Rigidity Rigidity commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

#362's step 13d. A generic function compared inside another function comparison can now still be instantiated while that comparison checks its bounds.

Closes #363
Part of #362

The problem (#363)

fn map<A, B>(function: fn(a: A) -> B, value: A) -> B { function(value) }
fn call<B>(function: fn() -> B) -> B { function() }
fn pairs<T>() -> (T, T) { ... }
let result: (Int, Int) = map(call, pairs);

This failed with "Cannot assign fn<B>(function: fn() -> B) -> B to fn(a: fn<T>() -> (T, T)) -> (Int, Int)". Comparing call with that type gives its B' the bounds B' ≥ (T', T'), from fn<T>() -> (T, T) against fn() -> B', and B' ≤ (Int, Int). But T', the instance of pairs's generic, was rigid once its own function comparison had finished, so (T', T') ≤ (Int, Int) failed, although T' = Int works.

The change (compare_structure's function arm, check_bounds)

  • Reopening: while a function comparison checks its own instances' bounds against each other, the instances of the generic functions compared inside it can still be instantiated (Instance::open). They keep their bounds after their own comparison finishes.
  • Closed instances only: an inner instance is reopened only if none of its bounds refer to a binder (is_closed). One that refers to other binders stays rigid, which can only reject more. Without this, summaries grew without bound on two corpus programs (seeds 834 and 877 timed out or gave up).
  • Once for all bounds: each reopened instance is instantiated once for all of the outer bounds. Afterwards, every instance whose bounds grew is checked again with every instance rigid, so its own bounds still have to hold. For example, T' ≥ Int from a parameter can't also be Bytes32. Enclosing comparisons' instances stay rigid.
  • Occurs check: an instance isn't bounded by a type that refers to it (refers_to). Otherwise T' ≤ T' or T' ≤ T' | nil, from an inner function compared with itself, conflicted with its other bounds once it was rigid again (Twin, Opt in the snapshot tests).
  • Summaries: a memo hit or a revisit used to bring the inner instances back as rigid binders, so memo on and off disagreed.
    • A summary now carries every bound of each closed inner instance that its key's bounds refer to. That includes bounds a function comparison inside it added while checking its own bounds. An instance with no bounds gets ≤ Any.
    • apply_summary makes those binders instances again.
    • canonicalize keeps a third copy of a component that bounds a local.
  • Checking without inference: holds_without_inference clears the reopened flags while it runs, so a union member that fails there leaves no bound on a reopened instance.

Breaking changes

None. The change only accepts more programs. The fuzzer and corpus runs below show no program going from accepted to rejected and no change in output.

Limitations

  • One level deep. Instances that only a reopened instance's bounds refer to stay rigid. deeper_instances_stay_rigid pins one that's still rejected.
  • Closed instances only (above).
  • Same type in both elements of a pair. At the type level, (G, G) against two function types that G is each assignable to is still Invalid. The compiler accepts the Rue equivalent. This is documented on test_bound_added_inside_reused.
  • (T, Int) ≤ T with T = Any is rejected, as on main.

Results

Head cf137f1f was checked against main 09000478. The full four-configuration run was on dc74328e against main 6aea07a5, before the rebase and the last two fixes. After them, one range per generator was run again and matched.

Fuzzer Variants Changed Newly inferred Regressions Unsound
Generator 1, seeds 0..4000 11,638 3 2 (seeds 2333, 2633) 0 0
Generator 1, seeds 4000..8000 (dc74328e) 11,641 1 1 (seed 5721) 0 0
Generator 2, seeds 0..4000 14,785 2 1 (seed 82) 0 0
Generator 2, seeds 4000..8000 (dc74328e) 14,725 2 1 (seed 5221) 0 0
  • Changes that aren't newly inferred:
    • Seeds 3634 (generator 1) and 1686 (generator 2) are still rejected, with a different diagnostic.
    • Seed 6274 (generator 2) has a different fingerprint.
  • Newly compiling programs:
  • Corpus (nested-corpus, seeds 0..2500): identical to main, 626 compiled and 1,874 rejected, with none too complex, timed out or crashed.
  • Ignored memo differentials: all 3 pass, in 308 s.

Review rounds

  • Round 1, 3 reviewers: 1 major. Memo on and off disagreed, because summaries brought inner instances back as rigid locals. Fixed by exporting their bounds, limited to one level after exporting them recursively made summaries about 7× slower.
  • Round 2, 3 reviewers: 2 majors.
    • Self-bounding (Twin): fixed with the occurs check.
    • A gap in the joint-instantiation tests: tests added.
  • Round 3, 3 reviewers: 1 major. Opt (T' ≤ T' | nil): fixed with the occurs check. The fuzzer and corpus runs then found the corpus regressions (seeds 834 and 877), fixed by reopening closed instances only. That broke Opt again, because local choices weren't opened while instances were reopened. Fixed through the open counter.
  • Check of the closed-instances rule:
    • Blocker: summaries kept only the bounds an instance had when its own comparison finished, so a memo hit accepted an ill-typed program. Fixed in c63ebc23, pinned by test_bound_added_inside_reused and bound_kept_when_reused.
    • Minor: open instances were instantiable inside holds_without_inference. Fixed in c63ebc23.
  • Verification of that fix: the "same bounds with memo on and off" argument held against every probe tried, including the large differentials. The second fix was incomplete when the uninferred comparison reopened an instance of its own. Fixed in cf137f1f by clearing the flags, pinned by the nested case of reopened_instances_stay_rigid_without_inference.

Provisional decisions, for you

  • One level deep (above).
  • All pairs of a grown instance are checked again, including those that already passed. That's redundant, but simple.
  • An inner instance with no bounds gets ≤ Any in a summary, a bound that always holds, rather than a per-local flag on Summary.
  • Three copies of a component that bounds a local in canonicalize, against two for the others. No test shows a result that needs the third, so it's kept on the argument alone.

Note

Medium Risk

This changes when a generic function's instance can be instantiated in rue-types' comparisons, and what memo summaries carry, so a mistake would accept ill-typed programs. It only accepts more programs. Instances that refer to other binders stay rigid, the second check keeps each reopened instance's bounds, and the memo on/off differentials, the fuzzer (0 unsound) and the corpus all agree.

  • compare.rs: reopening in the function arm, check_bounds, is_closed, the occurs check, summaries exporting and re-creating inner instances, and holds_without_inference.
  • summary.rs: the copy count for components that bound a local in canonicalize.
  • Tests: type-level cases (joint instantiation, lower bounds, memo on/off, without inference), and two snapshot files (Infer a generic through another generic's generic function candidate #363's program and variants, plus cases that must stay rejected).

@Rigidity Rigidity changed the title rigidity/inference restructure 13d Instantiate the generic functions compared inside of another while its bounds are checked Nov 19, 2026
@Rigidity
Rigidity force-pushed the rigidity/inference-restructure-13d branch from 30264e8 to 5ab0f78 Compare November 19, 2026 07:00
…s bounds are checked

A bound of an instance can refer to the instance of a generic function compared inside of the same
comparison, such as B ≥ (T, T) from fn<T>() -> (T, T) compared with fn() -> B. That instance was
rigid by then, so B ≤ (Int, Int) failed, although T can be Int (#363). Each such instance can now
still be instantiated while the outer bounds are checked, once for all of them, and its own bounds
are checked again with what that adds to them.
- Check the instances' bounds in place rather than cloning them
- Take the inner instances' index right after binding
- Update the docs of Instance and instantiable_from
A comparison reused from the memo, or a revisit, applied its summary with the instances of the
generic functions compared inside of it as rigid binders, so an enclosing comparison couldn't
instantiate them while checking its bounds, and the memo changed the result. A summary now carries
the bounds of each such instance that the bounds of its key's instances refer to, and applying it
makes those binders instances again.

- Test it with the memo on and off, and with a struct whose two fields reuse one comparison
- Test that an inner instance is instantiated once, along with its own bounds
- Keep three copies of a summary component that bounds a local binder, since checking an inner
  instance's bounds against each other can need bounds from two other copies
- Build the summary in two passes, and the applied local binders up front
- Drop a needless clone, and update the docs that still said local binders are rigid
…h itself

An inner instance can be instantiated while an enclosing comparison checks bounds that refer to it
on both sides, such as (T, T) against itself. Bounding it by itself conflicted with its other bounds
once it was rigid, so a program main accepts was rejected.
Replaces the arm for a binder compared with itself with an occurs check on the instance arms, which
also covers T ≤ T | nil while an enclosing comparison checks bounds on both sides.
Exporting every inner instance's own bounds in summaries made them grow without bound on nested
programs (corpus seeds 834 and 877 went from rejected in under a second to giving up), since
those bounds refer to more local binders, which merge copies into ever larger components. An inner
instance is now reopened only if its own bounds refer to no binder, which is intrinsic to it, so a
memo hit, a revisit and computing it again agree, and its exported bounds are closed. The others
stay rigid, as on main.

- Reopened instances are marked open rather than lowering instantiable_from, and choices are
  recorded while any is open, so a union member that holds without bounds still holds the others
- Test the occurs check at type level
@Rigidity
Rigidity force-pushed the rigidity/inference-restructure-13d branch from dc74328 to c63ebc2 Compare November 19, 2026 08:43

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Infer a generic through another generic's generic function candidate

1 participant