Repository navigation
Canonical order: sums, symbols from other contexts and coefficients, as in WMA - #2073
Merged
Merged
Conversation
…as in WMA Fixes most of #2071. Three changes to element_order, each as in WMA 15: - A sum whose largest term is a monomial sorts with the monomials of its largest variable, after all of them: 1 + x comes after x^3 and before y. Its key is that variable to an infinite power; Monomial now handles equal infinite exponents. Two such sums are compared term by term from the largest term, the shorter first. - Symbols are ordered by their names without the context, with the full name as a tie-break (wma_symbol_sort_key): Pi + x, not x + Pi. - Monomials that differ only in a numeric coefficient are ordered by it: -x < x < 2 x. On 3003 pairs of expressions, Order now agrees with WMA 15 on 767 more pairs and on 40 fewer. Those 40 are products with factors that are not symbols, such as Pi g[x] or Sqrt[Pi] Erf[1/x]: element_order skips such factors, which the context of System`Pi used to hide. One test (Integrate[Exp[-1/x^2], x]) is marked xfail for that, and one doctest in Hypergeometric1F1 now shows that order. Doctests and tests whose expected output was the old order are updated; the new outputs were checked against WMA 15. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tombouctou
added a commit
to tombouctou/mathics-core
that referenced
this pull request
Oct 28, 2026
… of a monomial, as in WMA On top of Mathics3#2073. As in WMA 15, every factor of a product counts as a variable of the monomial, not only the symbols: - an expression that is not a symbol, such as f[x] or Erf[1/x], is a variable (GeneralVariable) after all the symbols, ordered as a general expression: a f[x] comes after f[x] and before g[x], and b Cos[x] before a Sin[x]. A general expression on its own is a monomial in itself. - a power with an exponent that is not a number comes after the numeric powers of its base: E^x after E^2 and before F. - a sum, alone or as a factor, sorts right after the largest variable of its largest term: (3 x + 3 x^2) y comes before y^3, and x (a + x) before (a + x)^2. This replaces the infinite exponent of Mathics3#2073. On 3003 pairs of expressions, Order now differs from WMA 15 on 20 pairs, down from 425 with Mathics3#2073 and 1152 on master, and no pair that either got right goes wrong. The 40 pairs that Mathics3#2073 made worse are fixed, and the xfail it added to test_calculus.py is removed. Speed: element_order is kept on the expression with the head and elements it was built from, Sort and Orderless sort with it as a key, and Monomial has a fast path for one variable and a cheap ==. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Collaborator
Author
|
The 40 regressions listed above are fixed in #2074, stacked on this branch: factors such as |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tombouctou
added a commit
that referenced
this pull request
Oct 28, 2026
… of a monomial, as in WMA On top of #2073. As in WMA 15, every factor of a product counts as a variable of the monomial, not only the symbols: - an expression that is not a symbol, such as f[x] or Erf[1/x], is a variable (GeneralVariable) after all the symbols, ordered as a general expression: a f[x] comes after f[x] and before g[x], and b Cos[x] before a Sin[x]. A general expression on its own is a monomial in itself. - a power with an exponent that is not a number comes after the numeric powers of its base: E^x after E^2 and before F. - a sum, alone or as a factor, sorts right after the largest variable of its largest term: (3 x + 3 x^2) y comes before y^3, and x (a + x) before (a + x)^2. This replaces the infinite exponent of #2073. On 3003 pairs of expressions, Order now differs from WMA 15 on 20 pairs, down from 425 with #2073 and 1152 on master, and no pair that either got right goes wrong. The 40 pairs that #2073 made worse are fixed, and the xfail it added to test_calculus.py is removed. Speed: element_order is kept on the expression with the head and elements it was built from, Sort and Orderless sort with it as a key, and Monomial has a fast path for one variable and a cheap ==. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rocky
reviewed
Oct 28, 2026
| BASIC_NUMERIC_EXPRESSION_ELT_ORDER, | ||
| Monomial, | ||
| wma_str_sort_key, | ||
| wma_symbol_sort_key, |
Member
There was a problem hiding this comment.
Yay! The whole str/symbol thing was a mistake (and it is also buried deep inside definitions) that I'd love to see corrected. This is a small step, but one in the right direction.
… of a monomial, as in WMA (#2074) **Stacked on #2073**: the base is its branch. Once #2073 is merged, this one moves to master. It removes the 40 regressions that #2073 lists, and with them the `xfail` that #2073 added. Part of #2071. In WMA, every factor of a product is a variable of the monomial, not only the symbols. Mathics3 skipped factors such as `f[x]` or `Erf[1/x]` when it built the key of a product. This PR counts them, as WMA 15 does: - **An expression that is not a symbol is a variable** (`GeneralVariable` in `keycomparable.py`). These variables come after the symbols and are ordered among themselves as general expressions (head, length, elements). A general expression on its own is a monomial in itself. ``` Sort[{x f[x], Pi g[x], y f[x], a f[x], f[x], z, g[x], f[x]^2, 1 + f[x]}] {z, f[x], a f[x], x f[x], y f[x], f[x]^2, 1 + f[x], g[x], Pi g[x]} Sort[{a Sin[x], b Cos[x]}] {b Cos[x], a Sin[x]} Integrate[Exp[-1/x^2], x] E^(-1/x^2) x + Sqrt[Pi] Erf[1/x] ``` - **A power with an exponent that is not a number** comes after the numeric powers of its base: `Sort[{E^x, E, E^2, F, x}]` is `{E, E^2, E^x, F, x}`, and `x E^x` is `E^x x`. - **A sum**, alone or as a factor, sorts right after the largest variable of its largest term. Its key extends that variable's key with the terms. This replaces the infinite exponent from #2073, so that a sum *as a factor* goes there too: ``` Collect[3 x y + 2 Sin[x z] (x + 2 y^2 + x) + (x + y)^3, y] x^3 + (3 x + 3 x^2) y + y^3 + 4 x Sin[x z] + y^2 (3 x + 4 Sin[x z]) Expand[(x + a)^2 + (y + a)^2 + (x + y) (x + a), y] a^2 + x (a + x) + (a + x)^2 + 2 a y + (a + x) y + y^2 ``` Both are now exactly what WMA gives. ## How much closer to WMA `Order` on the same 3003 pairs as in #2073, compared with WMA 15: | | pairs that differ from WMA | |---|---| | master | 1152 | | #2073 | 425 | | this PR | **20** | No pair that master or #2073 gets right goes wrong. The 20 that remain are cases the WMA rule for which is not clear yet (see #2071): - a sum whose largest term has a negative coefficient, or with a negative constant (`1 - x`, `-1 + x`, `x - y`); - `"s"` against `Sqrt[2]`. Also still as before: a power with a numeric base and a symbolic exponent (`2^x`), and numeric expressions such as `-Sqrt[3]` against `Sqrt[3]`. ## Doctests and tests - **47 doctests** had the old order as expected output. I ran every new output back through WMA 15 (parsed held, evaluated by WMA, compared as InputForm). In 35 of them WMA keeps exactly the Mathics3 order; only 5 of the old expectations had it. In the other 12, WMA simplifies differently, e.g. `Sqrt[2/Pi]`, `1/E^x` or a third argument of `Root`, and the order of the terms is the same. Two differ in order and belong to the known cases above (`2^Re[z]`, `-Sqrt[3]`). - **pytest**: expectations updated in `test_basic.py`, `test_algebra.py`, `test_calculus.py`, `test_diffeqns.py` and `test_linalg.py`. I checked each new output against WMA 15; the `test_linalg` eigenvalues match WMA up to parentheses. `test_canonical_order` gets 7 more cases; 6 of them fail on #2073. - **Full run on kimmy: doctests 4622 passed, 0 failed.** pytest with `-n 4 --dist loadfile`: 20 failed, in `test_tensor.py`, `test_series.py`, `test_list.py`, `test_testing_expressions.py` and `test_latex.py`. Each of these files passes on its own; the `Max[x]` failure, for example, is an `x = 3` left by another file in the same worker. ## Speed With general expressions now monomials, comparisons got deeper. To keep this from slowing things down: - `element_order` is kept on the expression, together with the head and elements it was built from, so a changed expression rebuilds it; - `Sort` and `Orderless` sort with `element_order` as a key, which builds each key once instead of at every comparison; - `Monomial` has a fast path for a single variable and an `==` that compares the exponents. A tuple comparison calls `==` before `<`, and before this both ran the full comparison. On kimmy, best of 7, master vs this PR: | | master | this PR | |---|---|---| | `Sort` of 3000 `f[n, g[k]]` | 125 ms | 216 ms | | `Table[f[Mod[7 k, 101], g[k]], {k, 3000}]` | 2017 ms | 2022 ms | | `Plus @@ Table[Sin[k x], {k, 1500}]` | 1528 ms | 1471 ms | | `Total[Table[a[k] x^k, {k, 1500}]]` | 382 ms | 378 ms | | `Expand[(1 + a + b + c + x + y)^6]` | 229 ms | 171 ms | | `Expand[(a + b + c + d)^8]` | 99 ms | 77 ms | | `Times @@ Table[f[k]^k, {k, 1000}]` | 749 ms | 762 ms | | `Sort` of 3000 `x^i y^j` | 4156 ms | 3866 ms | The one slowdown left is a bare `Sort` of general expressions, 1.7×. Sums and products of such expressions are as fast as before or faster. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Member
|
LGTM |
rocky
added a commit
that referenced
this pull request
Nov 4, 2026
…#2104) Stored values such as `f[1] = a` are meant to be found through an index (`LiteralRuleIndex`, #2013). The index keeps the sort keys (`pattern_precedence`) of the rules in a set. Since #2073 those keys can contain a `Monomial`. `Monomial` defines `__eq__`, so Python dropped its default `__hash__`. As a result: - Building the index raises `TypeError`. - The index takes that to mean "this rule list can't be indexed". - Every lookup goes through all the rules again, as before #2013. No test caught it, because #2013's tests did not come along with it. **Change:** `Monomial.__hash__` hashes the (variable, exponent) pairs that `__eq__` compares, so equal monomials hash equally (`1` and `1.0` included). **Measurements** - 100 lookups among 400 stored values: 5050 `SameQ` calls on master, now 200. - Combinatorica's `SetGraphOptions`, rewritten to look options up in such a table (Mathics3-Package-Combinatorica, `set-graph-options-index`), becomes linear in the number of edges with this fix. **Tests:** `test/core/test_literal_rule_index.py`. Both tests fail on master. - Monomials hash the way they compare. - The index is built for 400 stored values plus a pattern rule, and lookups through it cost a few comparisons each; results are checked, including the fallback to the pattern rule. The full pytest run and the doctests pass. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: R. Bernstein <rocky@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes most of #2071. The canonical order (
Sort,Order,Orderlessheads) now follows WMA 15 in three more ways.1. Sums. A sum whose largest term is a monomial sorts with the monomials of its largest variable, after all of them. Its key is that variable to an infinite power (
Expression._sum_element_order). Two such sums are compared term by term from the largest term down, and if one runs out first, it comes first.Monomialnow handles two equal infinite exponents; before,inf - infgavenan.2. Symbols from other contexts. Symbols are ordered by their names without the context, and the full name only breaks ties (
wma_symbol_sort_key). Before,Globalxcame beforeSystemPi:3. Coefficients. Monomials that differ only in a numeric coefficient are ordered by it:
Sort[{x, -x, 2 x}]is{-x, x, 2 x}.How much closer to WMA
I took 78 expressions (numbers, strings, symbols including
PiandE, monomials with coefficients, sums, products with general factors, general expressions) and computedOrderfor all 3003 pairs in WMA 15 and in Mathics3:767 pairs are fixed. 40 get worse, all of one kind: products with a factor that is not a symbol, such as
Pi g[x]orSqrt[Pi] Erf[1/x].element_orderskips such factors in a product. Before this PR, the context inSystemPihappened to hide that. WMA treatsf[x],Erf[1/x]andE^xas variables of the monomial. That is the next step, in a separate PR. Because of it,Integrate[Exp[-1/x^2], x]intest_calculus.pyis markedxfail, and oneHypergeometric1F1` doctest now shows the other order of its two terms.Still left as before (listed in #2071):
1 - x, or a sum with a negative constant: WMA puts these before their largest variable;f[x]^2and1 + f[x].Tests
test_canonical_orderintest/builtin/test_testing_expressions.py: 16 cases with WMA 15 values; 13 of them fail on master.Sqrt[2/Pi]inBesselJ[1/2, x]), the order of the factors still matches WMA.-n 4 --dist loadfile: 15 failed. They are intest_tensor.py,test_eol.py,test_latex.pyandtest_diffeqns.py, and those files pass on their own (test_diffeqns.pyfails on master in the same run too). These are the order-dependent failures that master has as well.The developer's guide page on the canonical order (Mathics3/Mathics3-development-guide#51) lists the place of sums as a difference from WMA; I'll update it there once this lands.
🤖 Generated with Claude Code