Skip to content

Read a single-column CSV file as one column when detecting the format - #908

Open
L4XB wants to merge 1 commit into
simonw:mainfrom
L4XB:single-column-csv-detection
Open

L4XB wants to merge 1 commit into
simonw:mainfrom
L4XB:single-column-csv-detection

Conversation

@L4XB

@L4XB L4XB commented Oct 17, 2026 •

Copy link
Copy Markdown

Closes #907

When format= is not given, rows_from_file() sniffs the CSV dialect. It now passes delimiters=",;\t|" to csv.Sniffer.sniff(). If none of those four characters is found, the file is read with csv.excel as a single column instead of raising csv.Error or splitting on a letter. Semicolon and tab files are detected as before. The new (b"id;name\n1;Cleo", Format.CSV) case in test_rows_from_file_detect_format pins that.

Before:

$ sqlite-utils memory names.csv "select * from names"
_csv.Error: Could not determine delimiter
$ sqlite-utils memory urls.csv "select * from urls"
[{"ur": "https://examp", "": "e.com/a"},
 {"ur": "https://examp", "": "e.org/b"}]

After:

$ sqlite-utils memory names.csv "select * from names"
[{"name": "Ann"},
 {"name": "Bob"}]
$ sqlite-utils memory urls.csv "select * from urls"
[{"url": "https://example.com/a"},
 {"url": "https://example.org/b"}]

Tests:

  • test_rows_from_file_detect_format_single_column covers the two files above.
  • test_memory_csv_single_column checks the same thing through sqlite-utils memory.

All three fail on main. The full test suite passes (1501), and black, flake8, mypy, pyright and cog are clean. ty reports only its existing redundant-cast warning in db.py.

insert --sniff is unchanged, since there the user explicitly asks for detection.

rows_from_file() sniffed the CSV dialect with no delimiters= list, so
csv.Sniffer considered every character. A single-column file then either
had no consistent candidate (csv.Error: Could not determine delimiter) or
was split on a letter that repeats on each line: a column of URLs came out
as {"ur": "https://examp", "": "e.com/a"}.

Format detection now sniffs for a comma, a semicolon, a tab or a pipe,
and reads a file with none of them as plain CSV.

Closes simonw#907
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

rows_from_file() splits a single-column CSV file on a letter, or raises csv.Error

1 participant