Fix PDF.close() closing the wrong Page instances (memory leak, #1339) - #1390
Open
timothyohare wants to merge 1 commit into
Open
timothyohare wants to merge 1 commit into
timothyohare wants to merge 1 commit into
Conversation
close() called flush_cache() -- which clears the cached `_pages` list -- before iterating `self.pages` to close each page. With `_pages` gone, that iteration re-triggers the `pages` property's parsing branch, producing a fresh, never-otherwise-referenced set of Page objects, and closes *those* instead of the ones the caller actually used (e.g. via `for page in pdf.pages: page.extract_text()`). The real pages' per-page caches (`_objects`, `_layout`, and the `get_textmap` lru_cache) are therefore never released by close(), so a long-lived process that opens many PDFs sequentially sees memory grow roughly unbounded rather than being freed after each `with pdfplumber.open(...)` block exits. Three independent reports of this same symptom: jsvine#1339, jsvine#1229, jsvine#1189. Fix: close the pages before flushing the cache that they're read from, so close() operates on the actual pages instead of a discarded reparse. Added a regression test (test_issue_1339) that fails against the old ordering and passes with the fix -- confirmed both directions locally.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #1339 (and the same underlying issue as #1229 and #1189):
PDF.close()doesn't actually release the pages it's supposed to.
close()currently does:flush_cache()clearsPDF.cached_properties, which includes_pages. Soby the time
for page in self.pagesruns,_pagesno longer exists, and thepagesproperty's fallback branch re-parses the document from scratch(
PDFPage.create_pages(self.doc)), builds a brand-new list ofPageobjects, and closes those. The pages the caller actually used --
e.g. via
for page in pdf.pages: page.extract_text()-- are never closed atall.
Since those real pages are never closed, their per-page caches (
_objects,_layout, and theget_textmaplru_cache result) are never released. In along-lived process that opens many PDFs sequentially, this shows up as
memory that appears to grow without bound rather than being freed when each
with pdfplumber.open(...)block exits -- reported independently by threedifferent users (#1339, #1229, #1189), including one (#1339) who traced it
to this exact line pair in
close().Fix
Swap the order: close the pages (while
self.pagesstill returns the onesactually used) before flushing the cache that
self.pagesdepends on.This is a 3-line reorder, no behavior change other than closing the correct
objects.
Test plan
test_issue_1339intests/test_issues.py: opens a PDF, forceseach page's
_objectscache to populate viaextract_text(), callspdf.close(), then asserts the original page objects (the ones thecaller held references to) had
_objectscleared andget_textmap.cache_info().currsize == 0.developordering(
assert not hasattr(page, "_objects")→AssertionError: assert not True)and passes with the fix.
python -m pytest tests/-- 174 passed (175 with the new test).black --check,flake8clean on changed files. (Pre-existing note:isort --check pdfplumber/pdf.pyreports an unrelated, pre-existingdiff on this file's
typingimport line that's present ondevelopbefore this change too -- not touched by this PR.)
Found this while root-causing an OOM in a personal project's daily cron job
(pdfplumber 0.11.10) that parses a few hundred ASX regulatory PDFs
sequentially in one process -- happy to share more detail on that
reproduction if useful.