Repository navigation
Conversation
With newline='', the mode the csv documentation recommends, readline() and iteration searched for '\r' or '\n' one character at a time. So did the newline decoder, both when updating the newlines attribute with newline='' and when translating '\r\n' and '\r' to '\n' with the default newline=None, which it does for every chunk that contains '\r'. For 1-byte strings, these loops now first skip ahead one size_t word at a time. A cheap test skips words with no byte below 0x0E, and an exact test finds '\r' and '\n' in the rest, so tabs do not stop the search. On little-endian machines, ctz() gives the position, as in find_first_nonascii() in unicodeobject.c. The translation copies whole words as it goes. Each 1-byte loop is the old loop with the skip in front of it, so line endings are handled as before. If either of the first two characters is '\r' or below, the old loop runs directly, since it is fastest for short lines. 2- and 4-byte strings run the same code as before, as does _PyIO_find_line_ending(), which StringIO also uses.
|
All commit authors signed the Contributor License Agreement. |
|
@ptweezy Please keep both the pr description and comments in the code concise. Also see https://devguide.python.org/getting-started/ai-tools/ |
eendebakpt
added
the
AI-needs-explanation
|
Thank you for your interest in contributing to Python. This is a community-driven project, and we want it to be a welcoming place to learn and collaborate. Contributors are responsible for reviewing and understanding what they submit, checking its claims, and being able to explain changes in their own words. This helps reviewers focus on the contribution and leaves more time to support other contributors. This contribution appears to have been authored with AI assistance. Please explain your contribution in your own words and describe how you checked its correctness. Please read our guidelines for using AI tools. |
|
In my own words: The prior code went one byte at a time looking for line endings. For any 1-byte string (ASCII and Latin-1 both), this patch instead puts a check in front that chews through 8 bytes (at least on a 64-bit build) per iteration using a bit-manipulation test. If none of the bytes are below 0x0E, skip the whole word. If one is, a second exact check for \n and \r runs on that word: when neither is present (like a tab) the loop skips the word and keeps going a word at a time. When one is, little-endian returns its exact position, and big-endian returns the start of the word and lets the byte loop scan those 8 bytes to find the newline. The byte loop only takes over outright when the first or second character is \r or below, which is the short-line shortcut. readline() with newline='' ends up roughly 1.3x–1.8x faster and the newline decoder got the same word-at-a-time skip. Wider strings still enter the old loop. How I checked correctness:
|
For 1-byte strings (no character above U+00FF), three loops in
Modules/_io/textio.cthat look for\rand\none character at a time now first skip ahead asize_tword at a time:readline()and iteration withnewline='', the mode thecsvdocs recommend.newlinesup to date withnewline=''. For files with only\r\nendings, ascsv.writerwrites, it runs on every chunk.\r\nand\rto\nwith the defaultnewline=None.2- and 4-byte strings,
_PyIO_find_line_ending()andStringIOrun the same code as before. The new code is in separate functions because putting the skip inside the existing loops made the 2-byte path up to 9% slower.Benchmarks. Speedup over
mainreading a 24 MB file of 300,000 ASCII CSV rows (about 80 characters each) on an Intel Core i5-12400T. Each number is the median of 10 rounds of best-of-7.newline='',\nendingsnewline='',\r\nendingsnewline=None,\r\nendingscsv.reader,\r\nendingsnewline='\n'Without PGO (gcc and clang -O3, MSVC Release),
newline=''is 1.3x to 1.5x faster on these files. Files of 1-character lines gain little (0.97x to 1.11x). With clang -O3,newline=Noneandnewline='\n'on\nfiles, which run unchanged code, were 2% to 4% slower. I can share the full results and the benchmark script.Testing. The new
test_newlines_input_long_linesreads lines of many lengths with each kind of line ending, with and without a NUL, tab, form feed, 0x0E oréin them, usingnewline=''andnewline=None, for both the C and Python implementations.test_ioandtest_csvpass on gcc, clang, debug, free-threaded, ASan, UBSan, and MSVC x64 and Win32 builds. I also compared the new functions with the old loops on generated inputs, compiled for 32-bit x86 and big-endian targets as well, and compared Ciowith_pyioon edge cases such as\r\nsplit across chunks.