Skip to content

io.TextIOWrapper.tell() can read past its snapshot buffer after seek() past a carriage return #160811

Open

Description

@tonghuaroot

Bug report

io.TextIOWrapper.tell() can compute a skip larger than its snapshot buffer after seek() to a position that follows a carriage return. with universal newlines the decoder keeps a pending \r, so the seek cookie has chars_to_skip > 0; seek() replays the input chunk into a new snapshot but leaves the bytes-per-char ratio from the previous read(). tell() then multiplies that stale ratio by chars_to_skip. on a debug build the assertion in _io_TextIOWrapper_tell_impl fires:

import io

f = io.TextIOWrapper(io.BytesIO(b"one\rtwo\rthree\r"), encoding="utf-8")
f.read(4)
pos = f.tell()
f.read(1)
f.seek(pos)
f.read(1)
print(f.tell())
Assertion failed: (skip_bytes <= PyBytes_GET_SIZE(next_input)), function _io_TextIOWrapper_tell_impl, file textio.c, line 2987.

release builds do not assert. the oversized length is handed to the decoder, which reads past the bytes object, so tell() can fail on valid input. on 3.13.13 and 3.14.7:

import io

data = ('ab\r' + '\u20ac' * 15).encode()
f = io.TextIOWrapper(io.BytesIO(data), encoding='utf-8')
f.read(3)
pos = f.tell()
f.read(1)
f.seek(pos)
f.read(1)
f.tell()
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x82 in position 4: invalid start byte

_pyio.TextIOWrapper fails the same assertion, and under python -O it returns a position one character too far: after the seek it reads 'o\nthree\n' where 'wo\nthree\n' is expected.

setting the ratio from the replayed chunk in seek(), as _read_chunk() does, fixes both implementations.

CPython versions tested on:

3.13, 3.14, 3.16

Operating systems tested on:

macOS

Linked PRs

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions