Repository navigation
Conversation
rows_from_file() sniffed the CSV dialect with no delimiters= list, so
csv.Sniffer considered every character. A single-column file then either
had no consistent candidate (csv.Error: Could not determine delimiter) or
was split on a letter that repeats on each line: a column of URLs came out
as {"ur": "https://examp", "": "e.com/a"}.
Format detection now sniffs for a comma, a semicolon, a tab or a pipe,
and reads a file with none of them as plain CSV.
Closes simonw#907
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #907
When
format=is not given,rows_from_file()sniffs the CSV dialect. It now passesdelimiters=",;\t|"tocsv.Sniffer.sniff(). If none of those four characters is found, the file is read withcsv.excelas a single column instead of raisingcsv.Erroror splitting on a letter. Semicolon and tab files are detected as before. The new(b"id;name\n1;Cleo", Format.CSV)case intest_rows_from_file_detect_formatpins that.Before:
After:
Tests:
test_rows_from_file_detect_format_single_columncovers the two files above.test_memory_csv_single_columnchecks the same thing throughsqlite-utils memory.All three fail on
main. The full test suite passes (1501), and black, flake8, mypy, pyright and cog are clean.tyreports only its existing redundant-cast warning indb.py.insert --sniffis unchanged, since there the user explicitly asks for detection.