Skip to content

Rule sort refactor - #1989

Draft
mmatera wants to merge 6 commits into
masterfrom
rule_sort_refactor
Draft

mmatera wants to merge 6 commits into
masterfrom
rule_sort_refactor

Conversation

@mmatera

@mmatera mmatera commented Oct 12, 2026

Copy link
Copy Markdown
Contributor

This PR implements an alternative way to sort rules, which seems to be closer to the one in WMA. The rule order is not a total order but a partial order, where one rule comes before other rule if the first is more specific than the second. Two rules like F[x_Integer] and F[x_Real] are not comparable, so they are added to a set of rules (let's say, downvalues) in the order they are introduced.

This would solve #1978


rules = {
"CubeRoot[n_?NumberQ]": "If[n > 0, Power[n, Divide[1, 3]], Times[-1, Power[Times[-1, n], Divide[1, 3]]]]",
"CubeRoot[n_Integer|n_Real|n_Rational]": "If[n > 0, Power[n, Divide[1, 3]], Times[-1, Power[Times[-1, n], Divide[1, 3]]]]",

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This rule must change, because otherwise would come before CubeRoot[n_Complex], just because WL rules come before FunctionRules.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a particular example or test that demonstrates why this is right?

Comment thread mathics/core/builtin.py
)
)

for pattern, function in self.get_functions(

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now FunctionApplyRules come after RewriteRules.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In what way do we see the effects of this change?


def analyze_rule(rule) -> Optional[RuleShape]:
"""Return the RuleShape of `rule`, or None if it is not understood."""
cached: Optional[RuleShape] | int = getattr(rule, "_specificity_shape", 0)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the idea is right, this function would play the same role that now play the attribute pattern_precedence. By now it is implemented from "outside" to be able to compare behaviors.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also would like to see if these "shapes" can be useful to build a "hash" to classify rules candidates to match against different expressions. This would be the final goal of the "dispatch" refactor.

@rocky

rocky commented Oct 12, 2026

Copy link
Copy Markdown
Member

I am glad to see this long-standing problem or black hole being understood. Thanks for putting us on that path.

I will be looking at the code in more detail later.

One thing that bothers me, because I thought or hoped we went over it, is the terminology. At minimum, it is misleading and may hide various algorithmic choices.

(A reason for defining concepts precisely is that if one is looking for pre-existing solutions and technology, as one should try to do, it is easier to find a fit if more precise and commonplace nomenclature is used.)

"Sorting" is something that is done on an entire set of elements. A sort key is used in a sort as a comparison function.

Here, there is no requirement to "sort" anything.

The requirement is that there be an ordering between patterns which determines which pattern to try first. The ordering here is a total ordering because for any pair of arbitrary patterns there is always one "right" answer.

The terms "order" and "precedence" are, I think, a better fit for what we have here: whenever there is a possibility of several pattern matches, one of these takes precedence over the others, or precedes the other possibilities. In the (hopefully rare) situation we have exhausted all possibilities, then yes, we will have done the work to sort the patterns.

But it is an algorithmic choice whether we want to sort elements eagerly or do this lazily.

Yes, ordering that is based solely on properties of a pattern and that does not consider the time or order at which the pattern was defined is a partial order. But once the time at which a pattern was defined is included, we have a total ordering.

It is also an algorithmic choice whether we want to break up the total ordering into components, some of which are partial orderings.

@mmatera

mmatera commented Oct 13, 2026

Copy link
Copy Markdown
Contributor Author

I am glad to see this long-standing problem or black hole being understood. Thanks for putting us on that path.

I was working on this for a while, and this is just a basic prototype to discuss the general idea. What maybe is the most valueble thing here are the tests that I added and that I will add later.

I will be looking at the code in more detail later.

One thing that bothers me, because I thought or hoped we went over it, is the terminology. At minimum, it is misleading and may hide various algorithmic choices.

(A reason for defining concepts precisely is that if one is looking for pre-existing solutions and technology, as one should try to do, it is easier to find a fit if more precise and commonplace nomenclature is used.)

I agree with this. However, since I found some problems fitting this behaviour on the concepts I have in mind, I found it better to make my findings explicit, and then try to narrow them.

"Sorting" is something that is done on an entire set of elements. A sort key is used in a sort as a comparison function.

Here, there is no requirement to "sort" anything.

This difference is something that confuses me a little bit: in Spanish, we have a single verb for both words "ordenar", and I do not understand the nuance between "sort" and "order".

The requirement is that there be an ordering between patterns which determines which pattern to try first. The ordering here is a total ordering because for any pair of arbitrary patterns there is always one "right" answer.

From mathematics, a total order is something that decides which of two elements comes "first", let's say, real numbers have a total order because if you take two different real numbers x and y, x<y or y<x. In matrix analysis, the order between two matrices is a "partial order", defined by the condition X-Y>0 if X-Y has positive eigenvalues. So it could happen that X>Y, X<Y or the order is indecidible. In both cases, the order (or not) is an intrinsic property of the elements, not something external like the history of how these elements are presented.

The terms "order" and "precedence" are, I think, a better fit for what we have here: whenever there is a possibility of several pattern matches, one of these takes precedence over the others, or precedes the other possibilities. In the (hopefully rare) situation we have exhausted all possibilities, then yes, we will have done the work to sort the patterns.

Precedence is a closer concept: a b^c^d is interpreted as a*(b^(c^d)) because ot the operator precedence, and a grouping rule, which defines the "associativity". However, I am not sure if it is the same that the concept of "rule precedence": in this other case, the precedence order depends both on the partial order of the rules (r<s if r is "more specific than" s) and the time at which both rules are introduced in the definition. So, I was doing some research on how to reproduce what WL does using different algorithms (like bisect left, bisect right and the one I put here) and the one that matches better was this one.

But it is an algorithmic choice whether we want to sort elements eagerly or do this lazily.

Yes, ordering that is based solely on properties of a pattern and that does not consider the time or order at which the pattern was defined is a partial order. But once the time at which a pattern was defined is included, we have a total ordering.

Indeed, we always have that trick: add a timestamp. But I am not sure if this is the most sensible way to get it.

It is also an algorithmic choice whether we want to break up the total ordering into components, some of which are partial orderings.

So, in resume, yes, it should be better implementations, and probably there is an specific name for this particular way to assign precedence to rule transformations. But at some point, I would like to find something that works and can be described in clear terms. What this does is fix one issue, and avoid breaking almost anything of the existing tests (up to small adjustments on some particular rules)
The next step would be to make the _shape parameter behave closer to the pattern_precedence attribute, which is defined on each class, and see what is still wrong with the order in which the many, many rules produced in Rubi are stored.

@rocky

rocky commented Oct 13, 2026

Copy link
Copy Markdown
Member

I am glad to see this long-standing problem or black hole being understood. Thanks for putting us on that path.

I was working on this for a while, and this is just a basic prototype to discuss the general idea. What maybe is the most valueble thing here are the tests that I added and that I will add later.

Absolutely! Very useful and thanks!

...

"Sorting" is something that is done on an entire set of elements. A sort key is used in a sort as a comparison function.
Here, there is no requirement to "sort" anything.

This difference is something that confuses me a little bit: in Spanish, we have a single verb for both words "ordenar", and I do not understand the nuance between "sort" and "order".

I asked Gemini what Donald Knuth's Volume 3 book "Sorting and Searching" is called in Spanish. Let me know if what it reports makes sense:

me:
What is the title of Donald Knuth's book Sorting and Searching called in Spanish?

gemini:

The title of Donald Knuth's book Sorting and Searching (Volume 3 of The Art of Computer Programming) in Spanish is:

Clasificación y búsqueda

... While the word "ordenamiento" is often used in Latin America for "sorting" (e.g., Ordenamiento y búsqueda), the official published Spanish edition by Editorial Reverté uses Clasificación y búsqueda.

me

The requirement is that there be an ordering between patterns which determines which pattern to try first. The ordering here is a total ordering because for any pair of arbitrary patterns there is always one "right" answer.

mmatera

From mathematics, a total order is something that decides which of two elements comes "first", let's say, real numbers have a total order because if you take two different real numbers x and y, x<y or y<x. In matrix analysis, the order between two matrices is a "partial order", defined by the condition X-Y>0 if X-Y has positive eigenvalues. So it could happen that X>Y, X<Y or the order is indecidible. In both cases, the order (or not) is an intrinsic property of the elements, not something external like the history of how these elements are presented.

In OO languages, there is fluidity between what is stored in the object and what is outside of it. A pattern can have a time as part of the parameter. "Time:" is intended in the Lamport Timestamp sense (which was derived from the relativity time event sense, or a simulation time counter). It does not have to be, and probably would not be, a calendar date and time.

The terms "order" and "precedence" are, I think, a better fit for what we have here: whenever there is a possibility of several pattern matches, one of these takes precedence over the others, or precedes the other possibilities. In the (hopefully rare) situation we have exhausted all possibilities, then yes, we will have done the work to sort the patterns.

Precedence is a closer concept: a b^c^d is interpreted as a*(b^(c^d)) because ot the operator precedence, and a grouping rule, which defines the "associativity". However, I am not sure if it is the same that the concept of "rule precedence":

I see it as a direct extension of pattern operator precedence, which is definitely a thing. When I write in Python:

m = r"(?P<tag1>a)|(?P<tag2>.)|(?P<tag3>[a-z])")
print(m.group("tag1")
print(m.group("tag2")
print(m.group("tag3")
yy = r"(?P<tag3>[a-z])|(?P<tag1>a)|(?P<tag2>.)"
m2 = re.match(yy, "a")
print(m.group("tag3")

This feels very much the same thing that is happening in WMA.

A search for the word Precedence used this way turns up from 22.3.5.2 Operator Precedence from Oracle's SQL documentation describing pattern matching.

in this other case, the precedence order depends both on the partial order of the rules (r<s if r is "more specific than" s) and the time at which both rules are introduced in the definition. So, I was doing some research on how to reproduce what WL does using different algorithms (like bisect left, bisect right and the one I put here) and the one that matches better was this one.

Please give me some time to double check this.

But it is an algorithmic choice whether we want to sort elements eagerly or do this lazily.
Yes, ordering that is based solely on properties of a pattern and that does not consider the time or order at which the pattern was defined is a partial order. But once the time at which a pattern was defined is included, we have a total ordering.

Indeed, we always have that trick: add a timestamp. But I am not sure if this is the most sensible way to get it.

Again, to be clear, a timestamp in the Lamport sense. And I agree; I don't know if this is the best way to implement it. It could be done as the position that a pattern appears in a list.

But while that's an implementation choice which we may or may not use, the concept still is that there is a total order and the ordering function depends on the pattern as well as its timestamp, however it is implemented.

It is also an algorithmic choice whether we want to break up the total ordering into components, some of which are partial orderings.

So, in resume, yes, it should be better implementations,

I wasn't suggesting anything about the implementation. The first step is to understand the concepts clearly and cleanly and in a way that it is likely to be found elsewhere or generally used and understood.

and probably there is an specific name for this particular way to assign precedence to rule transformations. But at some point, I would like to find something that works and can be described in clear terms. What this does is fix one issue, and avoid breaking almost anything of the existing tests (up to small adjustments on some particular rules) The next step would be to make the _shape parameter behave closer to the pattern_precedence attribute, which is defined on each class, and see what is still wrong with the order in which the many, many rules produced in Rubi are stored.

I appreciate that it solves a specific problem while not breaking other things. I just want to make sure we don't get into the situation where we are trying to solve a Rubik's Cube by looking at one piece at a time when the problem is better thought of in terms of the subgroups of the cube.

@rocky

rocky commented Oct 13, 2026

Copy link
Copy Markdown
Member

I have been thinking more about the "sort" vs. "order" usage. In English, outside of computer and mathematical use, they mean about the same thing.

However, in mathematics, ordering with the "ing" refers to an existential property, while in computer science, "sort" is a materialization into a list of the existential property.

# use insort_left to guarantee that if equal rules exist, newer rules will
# get higher precedence by being inserted before them. see DownValues[].
bisect.insort_left(values, rule, key=lambda x: x.pattern_precedence)
# Mathematica orders rules with a partial order: a rule goes before

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here we seem to do eager sorting (as before) instead of, say, lazy-on-demand order improvement.

That said, we probably could still use bisect.insort by storing the information needed to make the sort key a total ordering.

@@ -0,0 +1,323 @@
"""

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This module seems a bit complex, and I wonder if there is a way to make it seem more natural or simpler.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The module has two parts, the one that computes the "RuleShape" and the part that compares two RuleShapes to decide if one is more general than the other one, or if the order is indeterminate. The first part should be similar to the "pattern_precedence", and should be delegated to the Rule class an Pattern subclasses. This module is just a quick and dirty way to test that this behavior improves the current behavior in master.

@rocky

rocky commented Oct 28, 2026 •

Copy link
Copy Markdown
Member

@mmatera: I have been trying (unsuccessfully) to get this working. I have been doing this in rule_sort_refactor-update.

It looks to me from recent CProfile data in https://github.com/orgs/Mathics3/discussions/2065 this area may be one of the reasons Mathics3 is so slow.

A question: suppose I have rules:

F[1] = 1
F[2] = 4
F[3] = 9
...
F[10] = 100

And now, in my Mathics3 program, I refer to F[10]. Since all of these rules contain a literal pattern, "F" subscripted by a literal value, does this PR search linearly from 1 to 10 (order n for n rules), or do an $O(1)$ dictionary/hash lookup for the literal pattern F[10], or something else (e.g., binary search would be $O(log(n))$?

@mmatera

mmatera commented Oct 28, 2026

Copy link
Copy Markdown
Contributor Author

@mmatera: I have been trying (unsuccessfully) to get this working. I have been doing this in rule_sort_refactor-update.

It looks to me from recent CProfile data in https://github.com/orgs/Mathics3/discussions/2065 this area may be one of the reasons Mathics3 is so slow.

A question: suppose I have rules:

F[1] = 1
F[2] = 4
F[3] = 9
...
F[10] = 100

And now, in my Mathics3 program, I refer to F[10]. Since all of these rules contain a literal pattern, "F" subscripted by a literal value, does this PR search linearly from 1 to 10 (order n for n rules), or do an O ( 1 ) dictionary/hash lookup for the literal pattern F[10], or something else (e.g., binary search would be O ( l o g ( n ) ) ?

I guess this does not work because of a merge issue. In any case, maybe the most valuable thing here are the new tests. I have some work in progress for a better implementation that would also support the log(n) lookup (this does not do that).

@rocky

rocky commented Oct 28, 2026 •

Copy link
Copy Markdown
Member

I guess this does not work because of a merge issue. In any case, maybe the most valuable thing here are the new tests. I have some work in progress for a better implementation that would also support the log(n) lookup (this does not do that).

First and foremost, thanks for being such a good sport about the breakage and having to rework this.

I have some thoughts now that I understand what's happening better. However, some of this has to be a little hand-wavy.

The whole thrust of pattern-matching technology is to avoid linear searches over every pattern.
So others have put a lot of time, effort, and thought into this.

The problem we or any Mathematica implementation has, though, is trying to map existing work into WMA's situation.

While the power of extended regular expressions and WMA's patterns may be theoretically equivalent, figuring out how to use and map existing well-studied and proven technology to the WMA situation, I think, will require a bit of thought, discussion, and benchmarking.

Finally, I'd like to add that using hash tables or dictionaries for literal patterns like F[1], F[2], ... is an O(1) operation, which is better than $O(log n)$. As an engineering thing, then, it might be good to split these out just as, if I have it correct, conditional patterns and "non-literal" patterns are split off.

BTW, to me, this again emphasizes the stupidity in thinking that rewriting this in C or using Cython is somehow going to fix such fundamental algorithmic flaws.

@rocky

rocky commented Oct 28, 2026 •

Copy link
Copy Markdown
Member

the most valuable thing here are the new tests.

I totally agree. The new tests are much needed and invaluable.

I had, however, to ask Claude Code to give me simple examples.

While the two examples are not 100% the same and in different frameworks, showing the difference I hope may be instructive.

Instead of:

        (
            "ClearAll[f, g, h, k]; Do[f[i] = i^2, {i, 20}]; f[x_] := lin[x];"
            " {f[3], f[20], f[21], f[x]}",
            "{9, 400, lin[21], lin[x]}",
            "Stored values and a pattern rule",
        ),
        ("f[3] = new; {f[3], f[4]}", "{new, 16}", "A stored value replaced"),
        ("f[5] =.; {f[5], f[6]}", "{lin[5], 36}", "A stored value removed"),
        (
            "f[7] := cond /; True; {f[7], f[8]}",
            "{cond, 64}",
            "A conditional rule for a stored value comes first",
        ),
        (
            "f[9] := never /; False; f[9]",
            "81",
            "A failing condition falls back to the stored value",
        ),
        (
            "DownValues[f] = {HoldPattern[f[x_]] :> first[x], HoldPattern[f[1]] :> one};"
            " {f[1], f[2]}",
            "{first[1], first[2]}",
            "DownValues set in an order of their own",
        ),

Consider instead:

# Simple overwrite of the same literal
ClearAll[f];
f[3] = 9;
f[3] = new;
f[3]

# Overwrite while another literal rule exists
ClearAll[f];
f[1] = a;
f[2] = b;
f[1] = c;
DownValues[f]

# Overwrite after Unset
ClearAll[f];
f[1] = a;
f[2] = b;
f[1] =.;
f[1] = c;
f[1] = d;
DownValues[f]

Notice that each test tests one thing, and one thing only.
Instead of setting a grab bag of variables that will be used in future tests, variables are cleared before use.

I am guessing that in @tombouctou comments on running tests, and some were expected to fail due to the way he was running the test, stuff like this was getting in the way of running tests in parallel and independently.

@mmatera

mmatera commented Oct 28, 2026

Copy link
Copy Markdown
Contributor Author

Here we have two different problems that could be solved at a time.

One is the ordering of the rules: as the order in which available rules are applied to an expression in the evaluation process is different between WMA and Mathics3, the results are different, and that causes some evaluations to crash.

The other is a performance problem: we try to apply a lot of rules that we could avoid. In the example of F[1]=1; F[2]=2;... F[10000]=10000; we pass through all the rules before getting the right one. The trick that WMA uses to avoid this linear cost is to use hash tables, in a way that expressions with certain symbols and a certain n-arity look first for rules associated with these symbols and n-arities. Moreover, patterns that are literal expressions (like the one in the example) can be directly dispatched using a dictionary.

Regarding the first problem, the problem is that the ordering criteria is not intrinsic to the rule, but involves the time in which the rule is added to a definition. Intrinsic properties of the rules do not provide a total order but a partial order. Still, this partial order has precedence over the timestamp ordering.

The other problem requires classifying rules, and for that, we can use the information collected to build the sort key used for the partial order. This is what I am trying to code now.

rocky pushed a commit that referenced this pull request Oct 28, 2026
Since pytest 9.2.0 came out (2026-10-27), the "Run performance
benchmarks" job fails on every pull request, including #1989, #2032,
#2069 and #2070. All 73 benchmarks error at setup:

```
AttributeError: 'Function' object has no attribute '_nodeid'. Did you mean: 'nodeid'?
```

pytest-benchmark 5.3.0, its latest release, uses a private attribute
that pytest 9.2 removed: ionelmc/pytest-benchmark#321, still open. The
last passing run (master, 2026-10-27 10:20) had pytest 9.1.1; the
failing ones have 9.2.0. pytest-benchmark is 5.3.0 in both.

This keeps pytest below 9.2 in the benchmark workflow, with a comment
pointing at the upstream issue, so the pin can go once pytest-benchmark
is fixed. `pyproject.toml` does not pin pytest, so the later `pip
install -e .[dev,full]` keeps 9.1.x.

Checked with a one-test benchmark: pytest 9.2.0 + pytest-benchmark 5.3.0
gives the same error, and pytest 9.1.1 + pytest-benchmark 5.3.0 passes.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@tombouctou

Copy link
Copy Markdown
Collaborator

@mmatera, since you're coding the classification now, I won't start a parallel version. Here is what I can take on, if it helps:

  1. WMA 15 ground truth for the tests. I have WMA 15 locally, so I can run your CompareOrder from Wrong order for matching in Rule #1978 over any list of left-hand sides and give you the whole pairwise table: more specific, less specific, or incomparable (definition order). Some cases I checked while writing the developer's guide page (Add a page on the order of rules and the canonical order Mathics3-development-guide#51), where Mathics3 currently differs:

    Definitions Input WMA 15 Mathics3
    f[x_?EvenQ] := even; f[x_?PrimeQ] := prime f[2] even prime
    f[x_ /; x > 1] := a; f[x_ /; x > 0] := b f[2] a b
    f[x_, y_Integer] := a; f[x_Integer, y_] := b f[1, 2] a b
    f[x_Real] := r; f[x_Integer] := i DownValues[f] x_Real first x_Integer first
    f[x_, x_] := same; f[x_, y_] := diff DownValues[f] f[x_, x_] first f[x_, y_] first
    f[x_, y_:1] := o; f[x_, y_] := t DownValues[f] f[x_, y_] first f[x_, y_:1] first
  2. The two indexes that rely on the current order. insert_rule (insert_rule: find the rule to replace through an index, not by a full scan #2001) and get_rules_for (Find stored values such as f[1] = a through an index, not by trying every rule #2013) assume a rule list sorted by pattern_precedence. find_rule bisects on it, and LiteralRuleIndex places a literal rule by its key. With a partial order plus definition time, that assumption no longer holds. Once your classification is in, I can adapt both, or fold the literal dispatch into your per-symbol/arity tables, whichever you prefer.

  3. Full runs. I can run your branch through the full pytest + doctest suite and the Rubi/Combinatorica cases on our runner, next to master, and report what changes.

Tell me which of these you want, or none, and I'll keep out of your way otherwise.

rocky added a commit that referenced this pull request Oct 28, 2026
Following the suggestion in #2067 ("the same thing for matching rule and
element order precedence"). This adds a new tutorial section to the
Manual, "Order of Rules and of Elements", right after "Functions and
Patterns".

- **Which Rule Is Tried First**: the more specific rule is used. A value
such as `f[1]` comes before `f[x_]`, `x_Integer` before `x_`, `_` before
`__` before `___`, a conditional rule before the same rule without a
condition, and `{x_}` before `x_List` before `x_`. A rule with the same
left-hand side replaces the old one.
- **Canonical Order**: what `Sort`, `Order` and `Orderless` use. Numbers
come first, then strings, then symbols, then other expressions. Strings
and symbols are ordered without regard to case, with lower case first.
Monomials are ordered as the terms of a polynomial, and general
expressions by head, then by length, then element by element.

Every example is a doctest, and every result was checked against WMA 15.
I left out the cases where Mathics3 and WMA differ:

- Rules that cannot be compared, such as `f[x_?EvenQ]` and
`f[x_?PrimeQ]`: WMA tries them in the order of definition. That is
#1978, and the draft #1989 addresses it.
- `Optional` and repeated pattern names.
- The place of sums in the canonical order.

These are listed, with WMA 15 values, in the companion developer's guide
page, Mathics3/Mathics3-development-guide#51.

`docpipeline.py -c "Language Tutorial"`: all pass.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: R. Bernstein <rocky@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants