Skip to content

Canonical order: sums, symbols from other contexts and coefficients, as in WMA - #2073

Merged
rocky merged 5 commits into
masterfrom
canonical-order-of-sums
Oct 29, 2026
Merged

rocky merged 5 commits into
masterfrom
canonical-order-of-sums

Conversation

@tombouctou

Copy link
Copy Markdown
Collaborator

Fixes most of #2071. The canonical order (Sort, Order, Orderless heads) now follows WMA 15 in three more ways.

1. Sums. A sum whose largest term is a monomial sorts with the monomials of its largest variable, after all of them. Its key is that variable to an infinite power (Expression._sum_element_order). Two such sums are compared term by term from the largest term down, and if one runs out first, it comes first.

Sort[{a, a + b, b, c, x}]                        {a, b, a + b, c, x}
Sort[{x, 1 + x, y, x^2, 1 + x^2, x y}]           {x, x^2, 1 + x, 1 + x^2, y, x y}
Sort[{f[x], 1 + x, g[x], a + b + c, h[a, b, c]}] {a + b + c, 1 + x, f[x], g[x], h[a, b, c]}
Collect[a x + b y + c x + d y, y]                a x + c x + (b + d) y

Monomial now handles two equal infinite exponents; before, inf - inf gave nan.

2. Symbols from other contexts. Symbols are ordered by their names without the context, and the full name only breaks ties (wma_symbol_sort_key). Before, Globalxcame beforeSystemPi:

x + Pi                   Pi + x        (was x + Pi)
Sort[{x, Pi, E, a, z}]   {a, E, Pi, x, z}

3. Coefficients. Monomials that differ only in a numeric coefficient are ordered by it: Sort[{x, -x, 2 x}] is {-x, x, 2 x}.

How much closer to WMA

I took 78 expressions (numbers, strings, symbols including Pi and E, monomials with coefficients, sums, products with general factors, general expressions) and computed Order for all 3003 pairs in WMA 15 and in Mathics3:

pairs that differ from WMA
master 1152
this PR 425

767 pairs are fixed. 40 get worse, all of one kind: products with a factor that is not a symbol, such as Pi g[x] or Sqrt[Pi] Erf[1/x]. element_order skips such factors in a product. Before this PR, the context in SystemPihappened to hide that. WMA treatsf[x], Erf[1/x]andE^xas variables of the monomial. That is the next step, in a separate PR. Because of it,Integrate[Exp[-1/x^2], x]intest_calculus.pyis markedxfail, and one Hypergeometric1F1` doctest now shows the other order of its two terms.

Still left as before (listed in #2071):

  • a sum whose largest term has a negative coefficient, such as 1 - x, or a sum with a negative constant: WMA puts these before their largest variable;
  • powers and sums of general expressions, such as f[x]^2 and 1 + f[x].

Tests

  • New test_canonical_order in test/builtin/test_testing_expressions.py: 16 cases with WMA 15 values; 13 of them fail on master.
  • Doctests and tests whose expected output was the old order are updated (15 doctests, 8 test cases). I checked each new output against WMA 15. Where WMA simplifies differently (e.g. Sqrt[2/Pi] in BesselJ[1/2, x]), the order of the factors still matches WMA.
  • Full run on kimmy: doctests 4622 passed, 0 failed. pytest with -n 4 --dist loadfile: 15 failed. They are in test_tensor.py, test_eol.py, test_latex.py and test_diffeqns.py, and those files pass on their own (test_diffeqns.py fails on master in the same run too). These are the order-dependent failures that master has as well.

The developer's guide page on the canonical order (Mathics3/Mathics3-development-guide#51) lists the place of sums as a difference from WMA; I'll update it there once this lands.

🤖 Generated with Claude Code

tombouctou and others added 2 commits October 28, 2026 20:49
…as in WMA

Fixes most of #2071. Three changes to element_order, each as in WMA 15:

- A sum whose largest term is a monomial sorts with the monomials of its
  largest variable, after all of them: 1 + x comes after x^3 and before
  y. Its key is that variable to an infinite power; Monomial now handles
  equal infinite exponents. Two such sums are compared term by term from
  the largest term, the shorter first.
- Symbols are ordered by their names without the context, with the full
  name as a tie-break (wma_symbol_sort_key): Pi + x, not x + Pi.
- Monomials that differ only in a numeric coefficient are ordered by it:
  -x < x < 2 x.

On 3003 pairs of expressions, Order now agrees with WMA 15 on 767 more
pairs and on 40 fewer. Those 40 are products with factors that are not
symbols, such as Pi g[x] or Sqrt[Pi] Erf[1/x]: element_order skips such
factors, which the context of System`Pi used to hide. One test
(Integrate[Exp[-1/x^2], x]) is marked xfail for that, and one doctest
in Hypergeometric1F1 now shows that order.

Doctests and tests whose expected output was the old order are updated;
the new outputs were checked against WMA 15.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tombouctou added a commit to tombouctou/mathics-core that referenced this pull request Oct 28, 2026
… of a monomial, as in WMA

On top of Mathics3#2073. As in WMA 15, every factor of a product counts as a
variable of the monomial, not only the symbols:

- an expression that is not a symbol, such as f[x] or Erf[1/x], is a
  variable (GeneralVariable) after all the symbols, ordered as a general
  expression: a f[x] comes after f[x] and before g[x], and b Cos[x]
  before a Sin[x]. A general expression on its own is a monomial in
  itself.
- a power with an exponent that is not a number comes after the
  numeric powers of its base: E^x after E^2 and before F.
- a sum, alone or as a factor, sorts right after the largest variable of
  its largest term: (3 x + 3 x^2) y comes before y^3, and x (a + x)
  before (a + x)^2. This replaces the infinite exponent of Mathics3#2073.

On 3003 pairs of expressions, Order now differs from WMA 15 on 20 pairs,
down from 425 with Mathics3#2073 and 1152 on master, and no pair that either
got right goes wrong. The 40 pairs that Mathics3#2073 made worse are fixed, and
the xfail it added to test_calculus.py is removed.

Speed: element_order is kept on the expression with the head and
elements it was built from, Sort and Orderless sort with it as a key,
and Monomial has a fast path for one variable and a cheap ==.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@tombouctou

Copy link
Copy Markdown
Collaborator Author

The 40 regressions listed above are fixed in #2074, stacked on this branch: factors such as f[x], Erf[1/x] and E^x become variables of the monomial, as in WMA. With both, Order differs from WMA 15 on 20 of the 3003 pairs (1152 on master), and the xfail added here is gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tombouctou added a commit that referenced this pull request Oct 28, 2026
… of a monomial, as in WMA

On top of #2073. As in WMA 15, every factor of a product counts as a
variable of the monomial, not only the symbols:

- an expression that is not a symbol, such as f[x] or Erf[1/x], is a
  variable (GeneralVariable) after all the symbols, ordered as a general
  expression: a f[x] comes after f[x] and before g[x], and b Cos[x]
  before a Sin[x]. A general expression on its own is a monomial in
  itself.
- a power with an exponent that is not a number comes after the
  numeric powers of its base: E^x after E^2 and before F.
- a sum, alone or as a factor, sorts right after the largest variable of
  its largest term: (3 x + 3 x^2) y comes before y^3, and x (a + x)
  before (a + x)^2. This replaces the infinite exponent of #2073.

On 3003 pairs of expressions, Order now differs from WMA 15 on 20 pairs,
down from 425 with #2073 and 1152 on master, and no pair that either
got right goes wrong. The 40 pairs that #2073 made worse are fixed, and
the xfail it added to test_calculus.py is removed.

Speed: element_order is kept on the expression with the head and
elements it was built from, Sort and Orderless sort with it as a key,
and Monomial has a fast path for one variable and a cheap ==.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment thread mathics/core/symbols.py
BASIC_NUMERIC_EXPRESSION_ELT_ORDER,
Monomial,
wma_str_sort_key,
wma_symbol_sort_key,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yay! The whole str/symbol thing was a mistake (and it is also buried deep inside definitions) that I'd love to see corrected. This is a small step, but one in the right direction.

… of a monomial, as in WMA (#2074)

**Stacked on #2073**: the base is its branch. Once #2073 is merged, this
one moves to master. It removes the 40 regressions that #2073 lists, and
with them the `xfail` that #2073 added. Part of #2071.

In WMA, every factor of a product is a variable of the monomial, not
only the symbols. Mathics3 skipped factors such as `f[x]` or `Erf[1/x]`
when it built the key of a product. This PR counts them, as WMA 15 does:

- **An expression that is not a symbol is a variable**
(`GeneralVariable` in `keycomparable.py`). These variables come after
the symbols and are ordered among themselves as general expressions
(head, length, elements). A general expression on its own is a monomial
in itself.
  ```
Sort[{x f[x], Pi g[x], y f[x], a f[x], f[x], z, g[x], f[x]^2, 1 + f[x]}]
     {z, f[x], a f[x], x f[x], y f[x], f[x]^2, 1 + f[x], g[x], Pi g[x]}
  Sort[{a Sin[x], b Cos[x]}]                 {b Cos[x], a Sin[x]}
Integrate[Exp[-1/x^2], x] E^(-1/x^2) x + Sqrt[Pi] Erf[1/x]
  ```
- **A power with an exponent that is not a number** comes after the
numeric powers of its base: `Sort[{E^x, E, E^2, F, x}]` is `{E, E^2,
E^x, F, x}`, and `x E^x` is `E^x x`.
- **A sum**, alone or as a factor, sorts right after the largest
variable of its largest term. Its key extends that variable's key with
the terms. This replaces the infinite exponent from #2073, so that a sum
*as a factor* goes there too:
  ```
  Collect[3 x y + 2 Sin[x z] (x + 2 y^2 + x) + (x + y)^3, y]
     x^3 + (3 x + 3 x^2) y + y^3 + 4 x Sin[x z] + y^2 (3 x + 4 Sin[x z])
  Expand[(x + a)^2 + (y + a)^2 + (x + y) (x + a), y]
     a^2 + x (a + x) + (a + x)^2 + 2 a y + (a + x) y + y^2
  ```
  Both are now exactly what WMA gives.

## How much closer to WMA

`Order` on the same 3003 pairs as in #2073, compared with WMA 15:

| | pairs that differ from WMA |
|---|---|
| master | 1152 |
| #2073 | 425 |
| this PR | **20** |

No pair that master or #2073 gets right goes wrong. The 20 that remain
are cases the WMA rule for which is not clear yet (see #2071):
- a sum whose largest term has a negative coefficient, or with a
negative constant (`1 - x`, `-1 + x`, `x - y`);
- `"s"` against `Sqrt[2]`.

Also still as before: a power with a numeric base and a symbolic
exponent (`2^x`), and numeric expressions such as `-Sqrt[3]` against
`Sqrt[3]`.

## Doctests and tests

- **47 doctests** had the old order as expected output. I ran every new
output back through WMA 15 (parsed held, evaluated by WMA, compared as
InputForm). In 35 of them WMA keeps exactly the Mathics3 order; only 5
of the old expectations had it. In the other 12, WMA simplifies
differently, e.g. `Sqrt[2/Pi]`, `1/E^x` or a third argument of `Root`,
and the order of the terms is the same. Two differ in order and belong
to the known cases above (`2^Re[z]`, `-Sqrt[3]`).
- **pytest**: expectations updated in `test_basic.py`,
`test_algebra.py`, `test_calculus.py`, `test_diffeqns.py` and
`test_linalg.py`. I checked each new output against WMA 15; the
`test_linalg` eigenvalues match WMA up to parentheses.
`test_canonical_order` gets 7 more cases; 6 of them fail on #2073.
- **Full run on kimmy: doctests 4622 passed, 0 failed.** pytest with `-n
4 --dist loadfile`: 20 failed, in `test_tensor.py`, `test_series.py`,
`test_list.py`, `test_testing_expressions.py` and `test_latex.py`. Each
of these files passes on its own; the `Max[x]` failure, for example, is
an `x = 3` left by another file in the same worker.

## Speed

With general expressions now monomials, comparisons got deeper. To keep
this from slowing things down:

- `element_order` is kept on the expression, together with the head and
elements it was built from, so a changed expression rebuilds it;
- `Sort` and `Orderless` sort with `element_order` as a key, which
builds each key once instead of at every comparison;
- `Monomial` has a fast path for a single variable and an `==` that
compares the exponents. A tuple comparison calls `==` before `<`, and
before this both ran the full comparison.

On kimmy, best of 7, master vs this PR:

| | master | this PR |
|---|---|---|
| `Sort` of 3000 `f[n, g[k]]` | 125 ms | 216 ms |
| `Table[f[Mod[7 k, 101], g[k]], {k, 3000}]` | 2017 ms | 2022 ms |
| `Plus @@ Table[Sin[k x], {k, 1500}]` | 1528 ms | 1471 ms |
| `Total[Table[a[k] x^k, {k, 1500}]]` | 382 ms | 378 ms |
| `Expand[(1 + a + b + c + x + y)^6]` | 229 ms | 171 ms |
| `Expand[(a + b + c + d)^8]` | 99 ms | 77 ms |
| `Times @@ Table[f[k]^k, {k, 1000}]` | 749 ms | 762 ms |
| `Sort` of 3000 `x^i y^j` | 4156 ms | 3866 ms |

The one slowdown left is a bare `Sort` of general expressions, 1.7×.
Sums and products of such expressions are as fast as before or faster.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@rocky
rocky requested a review from mmatera October 29, 2026 12:12
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@rocky

rocky commented Oct 29, 2026

Copy link
Copy Markdown
Member

LGTM

@rocky
rocky merged commit dca341d into master Oct 29, 2026
15 checks passed
@rocky
rocky deleted the canonical-order-of-sums branch October 29, 2026 21:05
rocky added a commit that referenced this pull request Nov 4, 2026
…#2104)

Stored values such as `f[1] = a` are meant to be found through an index
(`LiteralRuleIndex`, #2013). The index keeps the sort keys
(`pattern_precedence`) of the rules in a set.

Since #2073 those keys can contain a `Monomial`. `Monomial` defines
`__eq__`, so Python dropped its default `__hash__`. As a result:
- Building the index raises `TypeError`.
- The index takes that to mean "this rule list can't be indexed".
- Every lookup goes through all the rules again, as before #2013.

No test caught it, because #2013's tests did not come along with it.

**Change:** `Monomial.__hash__` hashes the (variable, exponent) pairs
that `__eq__` compares, so equal monomials hash equally (`1` and `1.0`
included).

**Measurements**
- 100 lookups among 400 stored values: 5050 `SameQ` calls on master, now
200.
- Combinatorica's `SetGraphOptions`, rewritten to look options up in
such a table (Mathics3-Package-Combinatorica,
`set-graph-options-index`), becomes linear in the number of edges with
this fix.

**Tests:** `test/core/test_literal_rule_index.py`. Both tests fail on
master.
- Monomials hash the way they compare.
- The index is built for 400 stored values plus a pattern rule, and
lookups through it cost a few comparisons each; results are checked,
including the fallback to the pattern rule.

The full pytest run and the doctests pass.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: R. Bernstein <rocky@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants