Repository navigation
CSV dialect sniffer gets wrong dialect (excel) when tab delimited csv file has been truncated #110128
Closed as not planned
Description
Bug report
Bug description:
CSV dialect sniffer gets wrong dialect (excel) when tab delimited csv file has long header fields.
This issue is lifted up here as it was initially raised in the jc cli github repo. Here:
CPython versions tested on:
3.10
Operating systems tested on:
Linux
Activity
sunmy2019 commentedon Sep 29, 2023
You'd better provide a repro code that only utilizes Python stdlib.
You raised a problem with
csv.Sniffer.sniff. You think one of its sniff results is inaccurate.But, it's just a guess. And guess cannot be perfect.
What new algo will you raise and how will it affect the current use case?
g-yziquel commentedon Sep 29, 2023
Sure. Any more accurate sniffer is arguably not to be provided out of the box by the stdlib.
It remains that one of its sniffs is inaccurate, and I could reproduce it with the stdlib. (If that would be genuinely welcome).
I would however be very wary of making changes to the sniffing algo without a series of tests on a corpus of such CSV files.
As of today, I have no clue as to how this should be done. But I do believe the CSV files I gave should, in an ideal world, be part of such a testing corpus.
Feel free to close the issue. I raised it out of personal ethos. If it's just noise to you, please do close it.
ws-garcia commentedon Jan 5, 2025
Hi @g-yziquel, the research on the subject is on its way! Please take a look to the Table Uniformity Method (TUM), also implemented in Python using the built-in
csvmodule.serhiy-storchaka commentedon Jul 21, 2026
The problem is not the length of the header fields, but the truncation of the sample:
jcsniffs a fixed-size prefix, and the old code counted the incomplete last row as a real one, so the tab was no longer consistent andsniff()raisedcsv.Error.DummyData.Right.csvis 1024 bytes and fits in the sample entirely,DummyData.Wrong.csvis 1046 bytes and does not; the "right" file fails too if the sample is cut at 512 bytes.Fixed in 3.16 by the
csv.Snifferrewrite (gh-83273): the last row is not counted if the sample can be cut off in the middle of it. The maintenance branches are still affected, and a targeted fix for them is welcome.[-]CSV dialect sniffer gets wrong dialect (excel) when tab delimited csv file has long header fields[/-][+]CSV dialect sniffer gets wrong dialect (excel) when tab delimited csv file has been truncated[/+][3.15] pythongh-110128: Ignore a truncated final row in csv.Sniffer
encukou commentedon Oct 13, 2026
@serhiy-storchaka: Myself, I'd leave long-standing behaviour alone in bugfix releases, rather than change it with fixes that aren't in main.
@hugovk, what's your opinion?
hugovk commentedon Oct 29, 2026
Agreed, let's keep the existing behaviour for 3.15 and 3.14.
The larger #153694 rewrite fixes it for 3.16 (and we decided not to backport that #153694 (comment)).
Thanks anyway!