Skip to content

Speed up documents with many lists - #772

Open
deadstrobe5 wants to merge 2 commits into
trentm:masterfrom
deadstrobe5:perf/do-lists
Open

deadstrobe5 wants to merge 2 commits into
trentm:masterfrom
deadstrobe5:perf/do-lists

Conversation

@deadstrobe5

Copy link
Copy Markdown

_do_lists() takes quadratic time on documents with many lists. After every list it converts, it searches the rest of the text once for the next unordered list and once for the next ordered list, and it splices the converted list back into the full text. A document that only uses one list style rescans to the end for the other style every time, and every splice copies the whole document.

This change finds the next list marker with one small pattern, tries the unordered and ordered patterns at that position, and collects the converted lists in a list of pieces that is joined once at the end. A converted list always ends with a newline (a blank line at the top level), so right after a list the pattern is tried without the blank-line prefix. That keeps an unordered list directly followed by an ordered one (#16) working as before. The output is unchanged.

Input Before After Speedup
rust-lang/rust RELEASES.md, first 200 KB 3676 ms 1200 ms 3.1x
openai-python api.md (153 KB) 1224 ms 374 ms 3.3x
awesome-python README.md (83 KB) 1319 ms 420 ms 3.1x
1500 lists separated by paragraphs 4340 ms 303 ms 14x
2 KB README excerpt 4.19 ms 4.30 ms within noise

Median of 5 alternating rounds in one process (21 for the small input) with the old module loaded next to the new one, CPython 3.14, Linux. python test.py -- -knownfailure passes, and the output matches master on 86,000 random inputs (lists, indentation, code spans, inline HTML, links, backslash escapes, safe_mode and common extras). The new ReDoS case many_lists takes 0.6 s here and 7.2 s on master. The new tm-cases/many_lists case covers lists separated by paragraphs, a horizontal rule, a numbered line inside a paragraph, and an unordered list directly followed by an ordered one. Its expected HTML is what master produces.

@nicholasserra nicholasserra added the Tech debt Technical debt label Nov 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Tech debt Technical debt

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants