Skip to content

[2.7.18.16] Fix O(n^2) canonical ordering in unicodedata.normalize() (CVE-2026-3276) - #107

Merged
icanhasmath merged 1 commit into
2.7.18.16from
2.7.18.16-unicodedata-normalize
Oct 28, 2026
Merged

icanhasmath merged 1 commit into
2.7.18.16from
2.7.18.16-unicodedata-normalize

Conversation

@icanhasmath

Copy link
Copy Markdown

Backport of upstream pythongh-149079 (CVE-2026-3276): canonical ordering in unicodedata.normalize() was quadratic. Part of 2.7.18.16 (Linear PROD-1588).

Problem

nfd_nfkd(), which every normalization form goes through, put combining characters into canonical order with an in-place insertion sort. A long run of combining marks in alternating combining-class order makes it O(n²). NFC of u"a" + u"\u0300\u0327" * n takes:

n 2.7.18.15 this PR
20,000 0.22 s 0.001 s
50,000 1.6 s 0.004 s
200,000 19.9 s 0.015 s

Fix (same as upstream)

Each run of non-starters is sorted on its own:

  • Runs that are already in order are skipped.
  • Runs shorter than 20 use insertion sort.
  • Longer runs use a stable counting sort on the combining class, through a reusable buffer.

2.7 adaptation:

  • The sort works on the Py_UNICODE buffer instead of PyUnicode_READ/WRITE, and uses C89 declarations.
  • On narrow (UCS-2, i.e. Windows) builds it sees UTF-16 code units, just as the old sort did. Surrogates have combining class 0, so they still end a run.
  • The buffer grows with PyMem_Realloc. Upstream's PyMem_Resize overwrites its pointer argument in 2.7, so it would leak the old buffer on allocation failure.

Testing (Linux x86_64, UCS-4 build)

  • Upstream's test_long_combining_mark_run and test_combining_mark_run_fast_paths pass. So does the new test_combining_mark_run_is_linear, which checks a 100,000-mark run through the counting sort. test_unicodedata, test_unicode and test_normalization pass.
  • Differential check against 2.7.18.15: I generated 20,000 random strings for each of three seeds. They're mostly combining marks (U+0300–U+036F plus Hebrew, Thai, Devanagari, kana and CJK marks), mixed with starters, Hangul and U+FDFA, with run lengths from 1 to 200. All four forms, from both unicodedata and ucd_3_2_0, hash identically on both builds.
  • Windows x64 (narrow build): not built here.

…ze() (CVE-2026-3276)

Backport of python/cpython pythongh-149079 (991224b; 3.10: db744c0) to
2.7.18.x.

nfd_nfkd(), which every normalization form goes through, put combining
characters into canonical order with an in-place insertion sort.  A long
run of combining marks in alternating canonical combining class order
made it quadratic.  On 2.7.18.15, NFC of u"a" + u"\u0300\u0327" * n
takes 0.22 s for n=20000, 1.6 s for n=50000 and 19.9 s for n=200000.

Each run of non-starters is now sorted on its own: insertion sort for
runs shorter than 20, otherwise a stable counting sort on the combining
class through a reusable buffer, and runs already in order are skipped.
The same input now takes 0.001 s, 0.004 s and 0.015 s.

2.7.18.x: the sort works on the Py_UNICODE result buffer instead of
PyUnicode_READ/WRITE.  On narrow (UCS-2) builds it therefore sees UTF-16
code units, as the old sort did; surrogates have combining class 0, so
they end runs as before.  C89 declarations.  The buffer grows through
PyMem_Realloc rather than upstream's PyMem_Resize, which in 2.7 overwrites
its pointer argument and would leak the old buffer on failure.

Tests: upstream's test_long_combining_mark_run and
test_combining_mark_run_fast_paths, plus test_combining_mark_run_is_linear
(a 100000-mark run through the counting sort).  On 20000 random strings
for each of three seeds, mostly combining marks with runs of 1-200,
NFC/NFD/NFKC/NFKD from both unicodedata and ucd_3_2_0 hash identically
to 2.7.18.15.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@ezequielp-activestate ezequielp-activestate left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No blockers. Approving.

The algorithm is upstream's (991224b, pythongh-149079 / pythonGH-149080), including the later "Optimize sorting" change (9074876) that the 3.10–3.12 backports carry: runs of non-starters are sorted separately, already-ordered runs are skipped, runs under 20 use insertion sort, and longer runs use a stable counting sort on the combining class. The one later upstream change to this code, c38d75c ("Preserve the normalization buffer on allocation failure"), is already covered by using PyMem_Realloc instead of PyMem_Resize.

Checked on a narrow (UCS-2) build, which is the configuration the description lists as not built. This was a debug build with AddressSanitizer, Linux x86_64:

  • Same output as the base. I generated 3 × 6,000 random strings: starters (Latin, Hangul syllables and jamo, kana, CJK, U+FDFA, singletons) followed by combining-mark runs of length 0–200. The marks cover Latin, Hebrew, Thai, Devanagari, kana, U+1DC0 and U+20D0 marks, plus astral marks (U+1D165, U+1D167, U+1D16D, U+1D17B, U+110B9, U+1E8D0, U+10A0F), which appear as surrogate pairs on this build. NFC, NFD, NFKC and NFKD from both unicodedata and ucd_3_2_0 hash identically on 2.7.18.16 and this PR.

  • Linear. NFC of u"a" + u"̧̀" * n:

    n 2.7.18.16 this PR
    5,000 0.211 s 0.003 s
    20,000 3.30 s 0.012 s
  • No memory errors. ASAN reports nothing in the sort, the buffer growth or the differential run.

  • Tests: test_unicodedata runs 26 OK (base 23). test_normalization with -uurlfetch, which runs the full NormalizationTest.txt, is OK on both.

Not assessed: Windows, and a UCS-4 build (the description covers UCS-4).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants