Repository navigation
3.7.17.x: gh-151497 tarfile extended-header pre-allocation - #120
Merged
Merged
Conversation
…file extended headers (pythonGH-151498) (pythonGH-151978) tarfile reads a member's extended header (a GNU long name/link or a pax header) with a single read sized by the header's size field: buf = tarfile.fileobj.read(self._block(self.size)) The size is taken from the archive and is not validated, so a ~512-byte crafted file can claim several gigabytes (or, via base-256 encoding, far more) and make read() pre-allocate that much memory -- on open/iterate, before any extraction filter runs. Read the extended-header data in bounded chunks instead, so an oversized or truncated header can no longer force a huge allocation. The bytes returned for valid archives are unchanged. (cherry picked from commit da99711) Co-authored-by: Shardul Deshpande <iamsharduld@users.noreply.github.com> (cherry picked from commit b83961a) 3.7.17.x: _safe_read lands after 3.7's deprecated tarfile.filemode(); the test_tarfile conflict was context only (test_extractfile_attrs, pythongh-74468, isn't on this line), so only the ReadSizeRecorder and ExtendedHeaderMemoryTest classes are added.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
manoelc-dev
self-requested a review
October 29, 2026 22:37
ezequielp-activestate
approved these changes
Oct 29, 2026
ezequielp-activestate
left a comment
ezequielp-activestate
left a comment
There was a problem hiding this comment.
Approving.
- The pythongh-151497 port matches upstream 3.13 (
b83961ade8) line for line, apart from one blank line. Upstream has no follow-ups to it. - I built this branch on Linux and ran
test_tarfile:- With this branch's
tarfile.py, everything passes. - With the 3.7.17.x
tarfile.py, it fails.
- With this branch's
- The What's New entry reads well.
manoelc-dev
removed their request for review
October 29, 2026 22:44
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
pythongh-151497 for 3.7.17.8, the follow-up listed under "Not included" in #117. 3.7.17.8 isn't tagged yet, so this goes into that release; the commit adds it to the 3.7.17.8 What's New.
What's in it
tarfileread it with oneread(size), so a 512-byte file could maketarfile.open()allocate gigabytes before any extraction filter runs. The header is now read in 1 MiB chunks. Cherry-picked from upstream 3.13 (b83961a); upstream only backported it to 3.13 and 3.14. The conflicts were context only:_safe_readlands after 3.7's deprecatedtarfile.filemode(), andtest_extractfile_attrs(object returned by tarfile.extractfile has an incorrect value of name attribute python/cpython#74468) isn't on this line.The
pyexpatns.hfollow-up from #117 is already in 3.7.17.8.Testing
tarfileraisesMemoryErrortrying to allocate the claimed size; with the fix it raisesReadErrorwithout the allocation. The three newExtendedHeaderMemoryTestcases and the whole oftest_tarfilepass.