Repository navigation
[2.7.18.16] Fix O(n^2) canonical ordering in unicodedata.normalize() (CVE-2026-3276) - #107
Conversation
…ze() (CVE-2026-3276) Backport of python/cpython pythongh-149079 (991224b; 3.10: db744c0) to 2.7.18.x. nfd_nfkd(), which every normalization form goes through, put combining characters into canonical order with an in-place insertion sort. A long run of combining marks in alternating canonical combining class order made it quadratic. On 2.7.18.15, NFC of u"a" + u"\u0300\u0327" * n takes 0.22 s for n=20000, 1.6 s for n=50000 and 19.9 s for n=200000. Each run of non-starters is now sorted on its own: insertion sort for runs shorter than 20, otherwise a stable counting sort on the combining class through a reusable buffer, and runs already in order are skipped. The same input now takes 0.001 s, 0.004 s and 0.015 s. 2.7.18.x: the sort works on the Py_UNICODE result buffer instead of PyUnicode_READ/WRITE. On narrow (UCS-2) builds it therefore sees UTF-16 code units, as the old sort did; surrogates have combining class 0, so they end runs as before. C89 declarations. The buffer grows through PyMem_Realloc rather than upstream's PyMem_Resize, which in 2.7 overwrites its pointer argument and would leak the old buffer on failure. Tests: upstream's test_long_combining_mark_run and test_combining_mark_run_fast_paths, plus test_combining_mark_run_is_linear (a 100000-mark run through the counting sort). On 20000 random strings for each of three seeds, mostly combining marks with runs of 1-200, NFC/NFD/NFKC/NFKD from both unicodedata and ucd_3_2_0 hash identically to 2.7.18.15. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ezequielp-activestate
left a comment
There was a problem hiding this comment.
No blockers. Approving.
The algorithm is upstream's (991224b, pythongh-149079 / pythonGH-149080), including the later "Optimize sorting" change (9074876) that the 3.10–3.12 backports carry: runs of non-starters are sorted separately, already-ordered runs are skipped, runs under 20 use insertion sort, and longer runs use a stable counting sort on the combining class. The one later upstream change to this code, c38d75c ("Preserve the normalization buffer on allocation failure"), is already covered by using PyMem_Realloc instead of PyMem_Resize.
Checked on a narrow (UCS-2) build, which is the configuration the description lists as not built. This was a debug build with AddressSanitizer, Linux x86_64:
-
Same output as the base. I generated 3 × 6,000 random strings: starters (Latin, Hangul syllables and jamo, kana, CJK, U+FDFA, singletons) followed by combining-mark runs of length 0–200. The marks cover Latin, Hebrew, Thai, Devanagari, kana, U+1DC0 and U+20D0 marks, plus astral marks (U+1D165, U+1D167, U+1D16D, U+1D17B, U+110B9, U+1E8D0, U+10A0F), which appear as surrogate pairs on this build. NFC, NFD, NFKC and NFKD from both
unicodedataanducd_3_2_0hash identically on2.7.18.16and this PR. -
Linear. NFC of
u"a" + u"̧̀" * n:n 2.7.18.16this PR 5,000 0.211 s 0.003 s 20,000 3.30 s 0.012 s -
No memory errors. ASAN reports nothing in the sort, the buffer growth or the differential run.
-
Tests:
test_unicodedataruns 26 OK (base 23).test_normalizationwith-uurlfetch, which runs the full NormalizationTest.txt, is OK on both.
Not assessed: Windows, and a UCS-4 build (the description covers UCS-4).
Backport of upstream pythongh-149079 (CVE-2026-3276): canonical ordering in
unicodedata.normalize()was quadratic. Part of 2.7.18.16 (Linear PROD-1588).Problem
nfd_nfkd(), which every normalization form goes through, put combining characters into canonical order with an in-place insertion sort. A long run of combining marks in alternating combining-class order makes it O(n²). NFC ofu"a" + u"\u0300\u0327" * ntakes:Fix (same as upstream)
Each run of non-starters is sorted on its own:
2.7 adaptation:
Py_UNICODEbuffer instead ofPyUnicode_READ/WRITE, and uses C89 declarations.PyMem_Realloc. Upstream'sPyMem_Resizeoverwrites its pointer argument in 2.7, so it would leak the old buffer on allocation failure.Testing (Linux x86_64, UCS-4 build)
test_long_combining_mark_runandtest_combining_mark_run_fast_pathspass. So does the newtest_combining_mark_run_is_linear, which checks a 100,000-mark run through the counting sort. test_unicodedata, test_unicode and test_normalization pass.unicodedataanducd_3_2_0, hash identically on both builds.