Faster Mathics3 exits by rocky · Pull Request #2061 · Mathics3/mathics-core

rocky · 2026-10-20T12:53:44Z

We have massive garbage collection issues on exit, possibly because there are massive Generation 2 (oldest GC generation) long-lived objects that need to be freed.

This is particularly noticeable when exiting Rubi.

As a workaround, freeze garbage collection before final exit so that PyGC_Collect does not have to walk Generation 2 objects during _Py_Finalize.

Also, pytest-benchmark should be added to the "dev" set of packages for those who run pytest.

tombouctou

Thanks for tracking down the slow exit. The diagnosis (a huge Generation 2 graph swept at finalization) looks right, but os._exit() skips all interpreter cleanup, and some of what it skips is user-visible. Inline notes below. In a follow-up comment I'll try gc.freeze() as an alternative that keeps the normal exit path.

tombouctou · 2026-10-20T13:23:04Z

+            # Extract numeric exit code (default to 0 if None)
+            code = e.code if isinstance(e.code, int) else 0
+
+            os._exit(code)


os._exit() skips atexit handlers. repl.py registers user_write_history_file via atexit.register (lines 98/103), so a session ended with Quit[] never writes ~/.config/Mathics3/history. Before this change the re-raised SystemExit ran atexit and saved it.

Great observation!

tombouctou · 2026-10-20T13:23:04Z

+            #    (gc_list_size) to find unreachable sub-graphs.
+
+            # Flush standard streams so terminal output isn't lost
+            sys.stdout.flush()


~~Only stdout/stderr are flushed. Buffered user streams aren't, e.g. s = OpenWrite["out.txt"]; WriteString[s, "data"]; Quit[] can leave out.txt empty or truncated.~~

Correction: this didn't reproduce. With Rubi loaded, OpenWrite + WriteString + Quit[3] wrote all 4 bytes under this PR, the same as on master (see #2061 (comment)). Please disregard this point.

tombouctou · 2026-10-20T13:23:04Z

+            sys.stderr.flush()
+
+            # Extract numeric exit code (default to 0 if None)
+            code = e.code if isinstance(e.code, int) else 0


Non-int codes map to 0, which turns failures into success. sys.exit("fatal: ...") normally prints the message and exits 1; here it exits 0 silently. Also, a code that doesn't fit a C int (Quit[10^20]) makes os._exit raise OverflowError inside the handler, giving a traceback instead of an exit.

tombouctou · 2026-10-20T13:23:04Z

-        except SystemExit:
-            # raise to pass the error code on, e.g. Quit[1]
-            raise
+        except SystemExit as e:


interactive_eval_loop is also the nested loop for Dialog[] (mathics/eval/debug_tuning/dialog.py:51). A hard exit there kills the process without unwinding any outer finally/with blocks (stream __exit__, temp-file deletion, prompt restore), and an embedding host can no longer catch SystemExit.

It also only covers Quit[]/Exit[]: Ctrl-D, mathics -e ... and mathics script.m still pay the full sweep. A single fix at the top level (__main__.main) would cover all of them.

tombouctou · 2026-10-20T13:23:04Z

-            raise
+        except SystemExit as e:
+            # As of Oct. 2026 we have huge Python garbage collection problems.
+            # To mitigate we use os._exit() to Py_Finalize() and avoid


Nit: os._exit() skips Py_Finalize(); the comment reads as if it runs it. Also "There is a Bottleneck is" and "why we great massive" look garbled, and the commented-out # raise could go.

tombouctou · 2026-10-20T13:23:04Z

+    Disables cyclic GC right before pytest exits, preventing PyGC_Collect from
+    walking Generation 2 during _Py_Finalize.
+    """
+    gc.disable()


gc.disable() doesn't stop the shutdown collection: _PyGC_CollectNoFail (called from finalize_modules) ignores the enabled flag. On a synthetic 3M-cyclic-object graph (Python 3.14), shutdown took 0.73s with no change, 0.49s with gc.disable(), 0.04s with gc.freeze() and 0.01s with os._exit. So gc.freeze() gets most of the gain here.

Thanks for the investigation and information. I'll switch to gc.freeze() then

tombouctou · 2026-10-20T13:31:57Z

Follow-up: I tried gc.freeze() in place of os._exit(), measured with Rubi loaded.

Change (on top of master, mathics/__main__.py):

import gc
...
def main() -> int:
    ...
    exit_rc = 0

    # Registered first, so it runs last among the atexit handlers.
    # Freezing moves every tracked object into the permanent generation,
    # so the collection in Py_Finalize() does not walk the large
    # Generation 2 graph left by packages such as Rubi. Unlike os._exit(),
    # the other atexit handlers, stream flushes and exit codes are kept.
    atexit.register(gc.freeze)

Setup: Python 3.12.13 on Linux. master 13a2ca6, this PR 959234f, and master with the change above. Each run does Get["Rubi.wl"] (2819 Int rules, ~2 min load), prints a marker, then exits either with Quit[3] or by EOF (Ctrl-D). "exit" is the time from the marker to process exit. Two runs each.

variant	`Quit[3]` exit	EOF / Ctrl-D exit	exit code after `Quit[3]`
master	10.85 s, 10.88 s	10.84 s, 10.74 s	3
this PR (`os._exit`)	0.14 s, 0.13 s	10.85 s, 10.54 s	3
`atexit.register(gc.freeze)`	0.15 s, 0.15 s	0.14 s, 0.15 s	3

So freezing gets the same ~10.7 s saving as os._exit(). It also covers the Ctrl-D path, which this PR doesn't touch, and it keeps the normal shutdown: atexit (readline history), SystemExit for Dialog[]/embedders, and non-int exit codes. Since it is an atexit hook, it should also apply to mathics -e ... and mathics FILE, though I only measured Quit[] and EOF.

One correction to my review: the OpenWrite/WriteString + Quit[] case wrote all 4 bytes in every variant, including this PR. So the buffered-stream loss I flagged on line 408 didn't reproduce for that case. The history/atexit, Dialog[] and exit-code points still stand.

For test/conftest.py, the same idea would be gc.freeze() in pytest_sessionfinish instead of gc.disable().

An alternative to #2061 for the slow exit after loading a large package such as Rubi. At exit, `Py_Finalize()` runs a full cyclic garbage collection that walks the huge Generation 2 graph Rubi leaves behind. This registers `gc.freeze` as the first atexit handler, so it runs last. Every tracked object moves into the permanent generation, and the final collection has almost nothing to walk. Unlike `os._exit()`, the normal shutdown stays intact. The other atexit handlers still run (readline history, `--show-statistics`, trace dumps), `SystemExit` still propagates (`Dialog[]`, embedding hosts), and exit codes are unchanged. Because the hook is on `main()` rather than on the `Quit[]` branch of the REPL loop, it also applies on Ctrl-D. ## Measurements Python 3.12.13 on Linux, `Get["Rubi.wl"]` (2819 `Int` rules), then exit. Time is from the last output to process exit, two runs each: | variant | `Quit[3]` | EOF / Ctrl-D | |---|---|---| | master | 10.85 s, 10.88 s | 10.84 s, 10.74 s | | #2061 (`os._exit`) | 0.14 s, 0.13 s | 10.85 s, 10.54 s | | this PR | 0.15 s, 0.15 s | 0.14 s, 0.15 s | Exit code after `Quit[3]` is 3 in all variants. Details are in #2061 (comment). `test/test_main.py` and `test/test_returncode.py` pass locally. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

Use os._exit() to bypass garbage collection via "raise SystemExit"

We have massive garbage collection issues on exit, possibly because there is massive Generation 2 (oldest GC generation) long-lived object it needs to free. As a workaround, disable garbage collection before final exit.

gc.freeze (not gc.disable) prevents PyGC_Collect from walking Generation 2 during _Py_Finalize.

tombouctou

LGTM. Thanks for switching to gc.freeze(). With the os._exit part now covered by #2062, what's left is the pytest-side freeze plus pytest-benchmark for test/timings. Nit: the PR description still says "disable garbage collection", so it could be updated to match.

mmatera · 2026-10-20T19:29:56Z

LGTM. Also , I wonder if we could get something by calling gc.collect() after each Evaluation.evaluate() call.

rocky · 2026-10-20T20:10:27Z

LGTM. Also , I wonder if we could get something by calling gc.collect() after each Evaluation.evaluate() call.

I will leave that for others.

My main annoyance was having to hit Ctrl-D after exiting Rubi or a full pytest. (And the warning every time I ran pytest)

rocky requested review from mmatera and tombouctou October 20, 2026 12:55

tombouctou reviewed Oct 20, 2026

View reviewed changes

tombouctou mentioned this pull request Oct 20, 2026

Freeze GC objects at exit to skip the Generation 2 sweep #2062

Merged

rocky added 3 commits October 20, 2026 13:12

Workaround a massive Genreation 2 GC sweep at exit

8425b94

Use os._exit() to bypass garbage collection via "raise SystemExit"

Speed up exiting Mathics3

57bd86a

We have massive garbage collection issues on exit, possibly because there is massive Generation 2 (oldest GC generation) long-lived object it needs to free. As a workaround, disable garbage collection before final exit.

Remove overlap with PR 2062

97e4bfe

rocky force-pushed the faster-exits branch from 959234f to 97e4bfe Compare October 20, 2026 17:14

gc.disable -> gc.freeze

0a08ffc

gc.freeze (not gc.disable) prevents PyGC_Collect from walking Generation 2 during _Py_Finalize.

tombouctou approved these changes Oct 20, 2026

View reviewed changes

Comment changes

f09f675

rocky merged commit dcf6053 into master Oct 20, 2026
15 checks passed

rocky deleted the faster-exits branch October 20, 2026 20:10

Conversation

rocky commented Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tombouctou left a comment

Choose a reason for hiding this comment

Uh oh!

tombouctou Oct 20, 2026

Choose a reason for hiding this comment

Uh oh!

rocky Oct 20, 2026

Choose a reason for hiding this comment

Uh oh!

tombouctou Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tombouctou Oct 20, 2026

Choose a reason for hiding this comment

Uh oh!

tombouctou Oct 20, 2026

Choose a reason for hiding this comment

Uh oh!

tombouctou Oct 20, 2026

Choose a reason for hiding this comment

Uh oh!

tombouctou Oct 20, 2026

Choose a reason for hiding this comment

Uh oh!

rocky Oct 20, 2026

Choose a reason for hiding this comment

Uh oh!

tombouctou commented Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tombouctou left a comment

Choose a reason for hiding this comment

Uh oh!

mmatera commented Oct 20, 2026

Uh oh!

rocky commented Oct 20, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

rocky commented Oct 20, 2026 •

edited

Loading

tombouctou Oct 20, 2026 •

edited

Loading

tombouctou commented Oct 20, 2026 •

edited

Loading