Skip to content

Delete the compiled code of vanished checkouts, and scan for outdated compiled code at most once a day - #908

Merged
perlinm merged 3 commits into
mainfrom
fix/compiled-cache-cleanup
Nov 16, 2026
Merged

perlinm merged 3 commits into
mainfrom
fix/compiled-cache-cleanup

Conversation

@perlinm

@perlinm perlinm commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator

Motivation

qLDPC caches numba-compiled code in qldpc-<hash> directories, each with a marker that names the package directory it was compiled from. On every import, _delete_outdated_cache_directories read the marker of every directory unused for more than 7 days, and deleted those of the same package directory. Directories of deleted checkouts, and directories without a marker, never matched, so they were never deleted, yet their markers were read on every import. On one maintainer's machine this left 6,522 directories (9.9 GB), and each import read about 4,200 markers, costing about 0.45 s.

Changes

  • Aged qldpc-<hash> directories are now also deleted if their marker is missing or undecodable, or is not the absolute path of an existing directory. Any other error at a directory keeps it without ending the scan. The marked path is checked with os.stat, since Path.is_dir reports a permission error as a missing directory on Python 3.14.
  • Each package directory scans at most once a day, as recorded by a hidden stamp in the base directory that is named by a hash of the package path, so neither a source change nor another checkout's scan starts or skips a scan. Scans delete stamps more than a day old.
  • An undecodable marker, or a package path that UTF-8 cannot encode, no longer makes import qldpc fail.

The docstring of _delete_outdated_cache_directories states the rule. The tests are in src/qldpc/_compilation_test.py, a protected file, and come from a patch that a maintainer approved. They pass on Python 3.13 and 3.14.

Validation

Import time with a synthetic cache of 6,000 directories aged 30 days, over 10 interleaved rounds: with markers naming deleted checkouts, the median import fell from 1.78 s on main to 1.10 s (faster in 10 of 10 rounds). With markers naming one existing checkout, so that only the throttle helps, it fell from 1.88 s to 1.24 s. After the cleanup, the branch imports as fast as main with an empty cache. An import that scans pays about 5 s once to delete 6,000 directories, and about 2.5 s for a daily scan of 6,000 kept ones.

Notes

A checkout on a drive that is not mounted, or on another machine that shares the cache, loses its compiled code after 7 days unused, and compiles it again when next used.

perlinm and others added 3 commits November 16, 2026 15:19
… compiled code at most once a day

Each import of qLDPC read the marker of every directory of compiled code unused for more than
7 days, but deleted only those of its own package directory.  Directories of deleted checkouts,
or without a marker, were never deleted, and their markers were read on every import.

_delete_outdated_cache_directories now also deletes aged qldpc-<hash> directories whose marker
is missing or undecodable, or is not the absolute path of an existing directory, checked with
os.stat since Path.is_dir reports a permission error as a missing directory on Python 3.14.  Any
other error at one directory keeps it without ending the scan.  Each package directory scans at
most once a day, as recorded by a hidden stamp in the base directory named by a hash of its
path, and scans delete stamps more than a day old.  The marking and the scan also ignore a
ValueError, such as from a package path that UTF-8 cannot encode.

The tests of these behaviors change src/qldpc/_compilation_test.py, a protected file, and come as
a separate patch for a maintainer to approve.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Drop the relative and file markers from the deletion test, fold the check that package
directories scan separately into the throttle test, and check one directory whose marked path
cannot be checked rather than an error partway through the scan.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@perlinm
perlinm merged commit 3013577 into main Nov 16, 2026
8 checks passed
@perlinm
perlinm deleted the fix/compiled-cache-cleanup branch November 16, 2026 20:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant