Refresh bundled Unicode confusables data to UTS #39 v17.0.0 (2025-07-22)

[codeboxbot]

Summary

Refreshes the bundled Unicode confusables data from the 2022-08-26 copy (UTS #39 15.0.0) to the current approved 17.0.0 release dated 2025-07-22, regenerates every derived artefact, and adds regression tests that prove the refreshed data is used and that generation is byte-for-byte reproducible.

Addresses #18 (the bundled data — and the README's Unicode reference — were out of date). Not marked as auto-closing so a maintainer can decide.

Behaviour and the public API are unchanged. Multi-codepoint confusable mappings are still skipped exactly as before (that is deliberately left to a separate change for #4).

Data provenance

• Source: https://www.unicode.org/Public/security/latest/confusables.txt
• UTS #39 version 17.0.0, header date 2025-07-22
• sha256 of the original official file: 091c7f82fc39ef208faf8f94d29c244de99254675e09de163160c810d13ef22a

Note on the bundled file: the automated contribution path used here cannot transmit the full 745 KB official file, so the committed generator/source_data/confusables.txt keeps the official header and every data row's two codepoint columns (source ; target) plus the mapping type, but drops the trailing per-line # ( x → y ) NAME descriptive comments. The generator only reads the source/target columns, so the generated artefacts are byte-for-byte identical whether the bundled file is this de-commented copy or the pristine official file — a maintainer can drop the original file straight in and every test still passes. Provenance (URL + original sha256) is recorded both here and in a comment block at the top of the bundled file.

Generated-file changes (all regenerated by generator/main.py)

• raw_data/char_codes.txt, raw_data/chars.txt — equivalence classes: 1851 → 1915.
• javascript/src/homoglyph.js, node/index.js — regenerated char maps for the 63 search characters.
• javascript/tests/js/tests/DataTests.js, node/test/spec/DataTests.js — regenerated data tests.

Generator changes (small, behaviour-preserving)

• Deterministic output: the per-character homoglyph lists in the generated JS/JS-tests are now sorted by codepoint. Previously they followed Python set-iteration order, so homoglyph.js / index.js / DataTests.js were not reproducible across runs (they varied with PYTHONHASHSEED). Ordering does not affect behaviour — lookups are membership tests.
• Node test spec is now generated. node/test/spec/DataTests.js was a hand-maintained copy that had drifted out of sync with node/index.js; on master the node suite fails 32 specs because of it. main.py now generates it (via a require() prelude) alongside the browser copy, so npm test exercises the shipped data and stays in sync.

Tests / verification

• New generator/tests/test_generator.py (stdlib unittest, run: python3 -m unittest discover -s generator/tests):
  • source provenance — bundled header is UTS #39 17.0.0 / 2025-07-22;
  • reproducibility — two independent generator runs (different PYTHONHASHSEED) produce byte-identical output;
  • committed artefacts match a fresh regeneration of the bundled source;
  • refreshed data is used — U+0192 (ƒ, newly confusable with f in 17.0.0) appears grouped with f.
  • Result: 4 passed.
• Node suite (cd node && npm test): 2213 specs, 0 failures (was 1564 specs / 32 failures on master).
• Java data-test example strings were checked against the refreshed data via the identical Node algorithm and all still match; the Java toolchain was not available in the build environment, so gradle test was not run here.

Out of scope (called out, not touched)

The README Java quick-start references Maven version 1.2.1 while build.gradle/package.json are at 1.3.0. That pre-existing version-metadata discrepancy is unrelated to the data refresh and is left unchanged.

🤖 Generated with Claude Code