Skip to content

Handle corrupted gzip files with unexpected EOF #41668

Open
#160777
Open
Handle corrupted gzip files with unexpected EOF#41668
#160777
Assignees
serhiy-storchaka
Labels
stdlibStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or error

Description

@calvin
calvin
mannequin
opened on Mar 8, 2005
BPO 1159051
Nosy @loewis, @birkenfeld, @doko42, @larryhastings, @devdanzin, @benjaminp, @serhiy-storchaka
Files
  • t.gz: test gzip file with missing/corrupted CRC checksum
  • t.py: test script
  • gzip_eof.diff: patch
  • test_gzip_error.py
  • gzip_eof-3.4.patch: Patch for 3.4
  • gzip_eof-3.4_2.patch
  • lzma_deferred_error.patch: Defer the error if it is possible to return some data
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/serhiy-storchaka'
closed_at = None
created_at = <Date 2005-03-08.13:57:41.000>
labels = ['type-bug', 'library']
title = 'Handle corrupted gzip files with unexpected EOF'
updated_at = <Date 2013-05-12.10:47:24.205>
user = 'https://bugs.python.org/calvin'

    bugs.python.org fields:

    activity = <Date 2013-05-12.10:47:24.205>
actor = 'python-dev'
assignee = 'serhiy.storchaka'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2005-03-08.13:57:41.000>
creator = 'calvin'
dependencies = []
files = ['6528', '6529', '6530', '8610', '28727', '28795', '28809']
hgrepos = []
issue_num = 1159051
keywords = ['patch']
message_count = 20.0
messages = ['47892', '47893', '56754', '82185', '143719', '179969', '180265', '180288', '180289', '180399', '180513', '181237', '184259', '184345', '184688', '185555', '188313', '188946', '189002', '189010']
nosy_count = 11.0
nosy_names = ['loewis', 'georg.brandl', 'doko', 'calvin', 'larry', 'ajaksu2', 'nadeem.vawda', 'benjamin.peterson', 'python-dev', 'yeeeev', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue1159051'
versions = ['Python 3.4']

    Linked PRs

    Activity

    1. serhiy-storchaka
      added
      type-featureA feature request or enhancement
      on Jan 24, 2013

    2. nadeemvawda commented on Feb 3, 2013

      @nadeemvawda
      nadeemvawdamannequin
      on Feb 3, 2013
      Mannequin

      I think the new behavior should be controlled by a constructor flag, maybe
      named "defer_errors". I don't like the idea of adding the flag to read(),
      since that makes us diverge from the standard file interface. Making a
      distinction between size<0 and size=None seems confusing and error-prone,
      not to mention that we (again) would have read() work differently from most
      other file classes.

      I'd prefer it if the new behavior is not enabled by default for size>=0,
      even if this wouldn't break well-behaved code. Having a flag that only
      controls the size<0 case is inelegant, and I don't think we should change
      the default behavior unless there is a clear benefit to doing so.

    3. doko42 commented on Mar 15, 2013

      @doko42
      doko42
      on Mar 15, 2013
      Member

      this change breaks a test case in the bzr testsuite; will try to get to it next week. See https://launchpad.net/bugs/1116079

    4. doko42
      added
      release-blocker
      on Mar 15, 2013

    5. serhiy-storchaka commented on Mar 16, 2013

      @serhiy-storchaka
      serhiy-storchaka
      on Mar 16, 2013
      Member

      tuned_gzip does dangerous things, it overloads private methods of GzipFile.

      From Bazaar 2.3 Release Notes:

      • Stop using bzrlib.tuned_gzip.GzipFile. It is incompatible with
        python-2.7 and was only used for Knit format repositories, which haven't
        been recommended since 2007. The file itself will be removed in the next
        release. (John Arbash Meinel)

      Current version is 2.6b2. bzrlib.tuned_gzip.GzipFile should be removed two releases ago.

    6. doko42
      removed
      release-blocker
      on Mar 19, 2013

    7. serhiy-storchaka commented on Mar 19, 2013

      @serhiy-storchaka
      serhiy-storchaka
      on Mar 19, 2013
      Member

      I will be offline some time. Feel free to revert these changes in 2.7-3.3 if it
      is necessary.

    8. doko42 commented on Mar 30, 2013

      @doko42
      doko42
      on Mar 30, 2013
      Member

      another test case failure with this patch:
      https://launchpad.net/ubuntu/+archive/test-rebuild-20130329/+build/4416983

      reproducible with feedparser 5.1.3 from pypi, on x86 (but not x86_64).

      ERROR: test_gzip_struct_error (main.TestCompression)
      ----------------------------------------------------------------------

      Traceback (most recent call last):
  File "./feedparsertest.py", line 433, in test_gzip_struct_error
    f = feedparser.parse('http://localhost:8097/tests/compression/gzip-struct-error.gz')
  File "/build/buildd/feedparser-5.1.2/feedparser/feedparser.py", line 3836, in parse
    data = gzip.GzipFile(fileobj=_StringIO(data)).read()
  File "/usr/lib/python2.7/gzip.py", line 253, in read
    while self._read(readsize):
  File "/usr/lib/python2.7/gzip.py", line 323, in _read
    self._read_eof()
  File "/usr/lib/python2.7/gzip.py", line 340, in _read_eof
    crc32, isize = struct.unpack("<II", self._read_exact(8))
  File "/usr/lib/python2.7/gzip.py", line 189, in _read_exact
    raise EOFError("Compressed file ended before the "
EOFError: Compressed file ended before the end-of-stream marker was reached

      Ran 4237 tests in 5.190s

      FAILED (errors=1)
      ----------------------------------------
      Exception happened during processing of request from ('127.0.0.1', 43939)

    9. serhiy-storchaka commented on May 3, 2013

      @serhiy-storchaka
      serhiy-storchaka
      on May 3, 2013
      Member

      In both cases broken applications use the undocumented implementation details. But such changes are extremely strong for bugfix release and they should not be done without a special need. I propose to revert these changes in 2.7, 3.2 and 3.3 (possibly leaving in the default branch). Unfortunately, I was not online right before the release of the latest bugfix release and failed to do this. Fortunately, it is now possible to fix this in regression fix releases.

    10. serhiy-storchaka
      added
      release-blocker
      type-bugAn unexpected behavior, bug, or error
      and removed
      type-featureA feature request or enhancement
      on May 3, 2013

    11. python-dev commented on May 11, 2013

      @python-dev
      python-devmannequin
      on May 11, 2013
      Mannequin

      New changeset abc780332b60 by Benjamin Peterson in branch '2.7':
      backout 214d8909513d for regressions (bpo-1159051)
      http://hg.python.org/cpython/rev/abc780332b60

    12. python-dev commented on May 12, 2013

      @python-dev
      python-devmannequin
      on May 12, 2013
      Mannequin

      New changeset 854ba6f414a8 by Georg Brandl in branch '3.2':
      Issue bpo-1159051: Back out a fix for handling corrupted gzip files that
      http://hg.python.org/cpython/rev/854ba6f414a8

    13. birkenfeld
      removed
      release-blocker
      on May 12, 2013

    14. python-dev commented on May 12, 2013

      @python-dev
      python-devmannequin
      on May 12, 2013
      Mannequin

      New changeset 9c2831fe84e9 by Georg Brandl in branch '3.3':
      Back out patch for bpo-1159051, which caused backwards compatibility problems.
      http://hg.python.org/cpython/rev/9c2831fe84e9

      New changeset 5400e8fbc1de by Georg Brandl in branch 'default':
      null-merge reversion of bpo-1159051 patch from 3.3
      http://hg.python.org/cpython/rev/5400e8fbc1de

    15. ezio-melotti
      transferred this issue fromon Apr 9, 2022
    16. serhiy-storchaka
      added this to Compression issues 🗜on May 21, 2022
    17. harisawan-bit
      added 2 commits that reference this issue on Oct 28, 2026

      pythongh-41668: Retrigger CI for macOS runner

      2053884

      pythongh-41668: Retrigger CI

      dc08ea5
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    Labels

    stdlibStandard Library Python modules in the Lib/ directorytype-bugAn unexpected behavior, bug, or error

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions