Repository navigation
Delete the compiled code of vanished checkouts, and scan for outdated compiled code at most once a day - #908
Merged
Merged
Conversation
… compiled code at most once a day Each import of qLDPC read the marker of every directory of compiled code unused for more than 7 days, but deleted only those of its own package directory. Directories of deleted checkouts, or without a marker, were never deleted, and their markers were read on every import. _delete_outdated_cache_directories now also deletes aged qldpc-<hash> directories whose marker is missing or undecodable, or is not the absolute path of an existing directory, checked with os.stat since Path.is_dir reports a permission error as a missing directory on Python 3.14. Any other error at one directory keeps it without ending the scan. Each package directory scans at most once a day, as recorded by a hidden stamp in the base directory named by a hash of its path, and scans delete stamps more than a day old. The marking and the scan also ignore a ValueError, such as from a package path that UTF-8 cannot encode. The tests of these behaviors change src/qldpc/_compilation_test.py, a protected file, and come as a separate patch for a maintainer to approve. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Drop the relative and file markers from the deletion test, fold the check that package directories scan separately into the throttle test, and check one directory whose marked path cannot be checked rather than an error partway through the scan. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
qLDPC caches numba-compiled code in
qldpc-<hash>directories, each with a marker that names the package directory it was compiled from. On every import,_delete_outdated_cache_directoriesread the marker of every directory unused for more than 7 days, and deleted those of the same package directory. Directories of deleted checkouts, and directories without a marker, never matched, so they were never deleted, yet their markers were read on every import. On one maintainer's machine this left 6,522 directories (9.9 GB), and each import read about 4,200 markers, costing about 0.45 s.Changes
qldpc-<hash>directories are now also deleted if their marker is missing or undecodable, or is not the absolute path of an existing directory. Any other error at a directory keeps it without ending the scan. The marked path is checked withos.stat, sincePath.is_dirreports a permission error as a missing directory on Python 3.14.import qldpcfail.The docstring of
_delete_outdated_cache_directoriesstates the rule. The tests are insrc/qldpc/_compilation_test.py, a protected file, and come from a patch that a maintainer approved. They pass on Python 3.13 and 3.14.Validation
Import time with a synthetic cache of 6,000 directories aged 30 days, over 10 interleaved rounds: with markers naming deleted checkouts, the median import fell from 1.78 s on main to 1.10 s (faster in 10 of 10 rounds). With markers naming one existing checkout, so that only the throttle helps, it fell from 1.88 s to 1.24 s. After the cleanup, the branch imports as fast as main with an empty cache. An import that scans pays about 5 s once to delete 6,000 directories, and about 2.5 s for a daily scan of 6,000 kept ones.
Notes
A checkout on a drive that is not mounted, or on another machine that shares the cache, loses its compiled code after 7 days unused, and compiles it again when next used.