Skip to content

Fix PDF.close() closing the wrong Page instances (memory leak, #1339) - #1390

Open
timothyohare wants to merge 1 commit into
jsvine:developfrom
timothyohare:fix/1339-close-order
Open

timothyohare wants to merge 1 commit into
jsvine:developfrom
timothyohare:fix/1339-close-order

Conversation

@timothyohare

Copy link
Copy Markdown

Summary

Fixes #1339 (and the same underlying issue as #1229 and #1189): PDF.close()
doesn't actually release the pages it's supposed to.

close() currently does:

def close(self) -> None:
    self.flush_cache()

    for page in self.pages:
        page.close()
    ...

flush_cache() clears PDF.cached_properties, which includes _pages. So
by the time for page in self.pages runs, _pages no longer exists, and the
pages property's fallback branch re-parses the document from scratch
(PDFPage.create_pages(self.doc)), builds a brand-new list of Page
objects, and closes those. The pages the caller actually used --
e.g. via for page in pdf.pages: page.extract_text() -- are never closed at
all.

Since those real pages are never closed, their per-page caches (_objects,
_layout, and the get_textmap lru_cache result) are never released. In a
long-lived process that opens many PDFs sequentially, this shows up as
memory that appears to grow without bound rather than being freed when each
with pdfplumber.open(...) block exits -- reported independently by three
different users (#1339, #1229, #1189), including one (#1339) who traced it
to this exact line pair in close().

Fix

Swap the order: close the pages (while self.pages still returns the ones
actually used) before flushing the cache that self.pages depends on.

def close(self) -> None:
    for page in self.pages:
        page.close()

    self.flush_cache()
    ...

This is a 3-line reorder, no behavior change other than closing the correct
objects.

Test plan

  • Added test_issue_1339 in tests/test_issues.py: opens a PDF, forces
    each page's _objects cache to populate via extract_text(), calls
    pdf.close(), then asserts the original page objects (the ones the
    caller held references to) had _objects cleared and
    get_textmap.cache_info().currsize == 0.
  • Verified the test fails on the current develop ordering
    (assert not hasattr(page, "_objects")AssertionError: assert not True)
    and passes with the fix.
  • Full suite: python -m pytest tests/ -- 174 passed (175 with the new test).
  • black --check, flake8 clean on changed files. (Pre-existing note:
    isort --check pdfplumber/pdf.py reports an unrelated, pre-existing
    diff on this file's typing import line that's present on develop
    before this change too -- not touched by this PR.)

Found this while root-causing an OOM in a personal project's daily cron job
(pdfplumber 0.11.10) that parses a few hundred ASX regulatory PDFs
sequentially in one process -- happy to share more detail on that
reproduction if useful.

close() called flush_cache() -- which clears the cached `_pages` list --
before iterating `self.pages` to close each page. With `_pages` gone, that
iteration re-triggers the `pages` property's parsing branch, producing a
fresh, never-otherwise-referenced set of Page objects, and closes *those*
instead of the ones the caller actually used (e.g. via
`for page in pdf.pages: page.extract_text()`).

The real pages' per-page caches (`_objects`, `_layout`, and the
`get_textmap` lru_cache) are therefore never released by close(), so a
long-lived process that opens many PDFs sequentially sees memory grow
roughly unbounded rather than being freed after each `with pdfplumber.open(...)`
block exits. Three independent reports of this same symptom: jsvine#1339, jsvine#1229,
jsvine#1189.

Fix: close the pages before flushing the cache that they're read from, so
close() operates on the actual pages instead of a discarded reparse.

Added a regression test (test_issue_1339) that fails against the old
ordering and passes with the fix -- confirmed both directions locally.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants