Skip to content

Add multi-codepoint confusable support to homoglyph search - #22

Open
codeboxbot wants to merge 1 commit into
codebox:masterfrom
codeboxbot:marvin/multi-codepoint-confusables
Open

Add multi-codepoint confusable support to homoglyph search#22
codeboxbot wants to merge 1 commit into
codebox:masterfrom
codeboxbot:marvin/multi-codepoint-confusables

Conversation

@codeboxbot

Copy link
Copy Markdown

Summary

Adds support for multi-codepoint confusables — cases where a single glyph in the text stands in for a sequence of characters in the target word. For example the ligature (U+FB03) resembles ffi, so office is now found when searching for office; the letter m resembles rn, so comer is found when searching for corner.

Previously the generator's data-file parser called chr(int(code, 16)) on the target field, which raised on any multi-codepoint target (e.g. "0066 0066 0069") and silently dropped it, so all ~2000 multi-codepoint confusables in the UTS #39 data were discarded. This PR captures them via a separate mapping and threads them through every consumer.

This is separate from PR #21 (the UTS #39 v17 data refresh) and does not touch it.

Semantics

When matching a target word against the text, a single text symbol matches either:

  • one word character (existing behaviour: exact match or single-codepoint homoglyph), or
  • a run of word characters, when that symbol is a multi-codepoint confusable whose prototype sequence equals that run (case-insensitively).

Alternatives are explored with backtracking, so a multi-codepoint substitution composes with ordinary single-codepoint substitution (e.g. 0fficeoffice, where 0o and ffi). Single-codepoint behaviour and the public search(...) APIs are unchanged; the matched substring may now be shorter than the target word.

Changes

Generator

  • data_file_parser.py: detects multi-codepoint targets (is_multi/get_multi) and routes them to a new manager instead of dropping them.
  • multi_char_manager.py (new): collects glyph → prototype-sequence mappings.
  • output_multi_char_codes.py (new) + templates/multi_char_codes.txt (new): emit raw_data/multi_char_codes.txt.
  • output_js.py: emits a second MULTI_CHAR_MAP into the JS artifacts and now sorts the single-codepoint glyph lists so regeneration is deterministic.
  • main.py: wires the new manager and output in.

Runtime

  • raw_data/multi_char_codes.txt (new): <glyph hex>;<space-separated prototype hex> per line.
  • javascript/src/homoglyph.js, node/index.js: regenerated — new backtracking matcher + embedded MULTI_CHAR_MAP. buildSearchFunction takes an optional second multiCharMap argument (defaults to {}, so existing callers are unaffected).
  • Java Homoglyph.java: new backtracking matcher; new Homoglyph(List<Set<Integer>>, Map<Integer,List<int[]>>) constructor (old constructor preserved). HomoglyphBuilder.java: loads the optional bundled multi_char_codes.txt; new build(Reader, Reader) overload. build.gradle: bundles the new resource.

Tests

  • generator/tests/test_multi_codepoint.py (new): parser/manager/output regressions using real UTS #39 lines — 13 tests.
  • node/test/spec/MultiCharTests.js + javascript/tests/js/tests/MultiCharTests.js (new): end-to-end JS regressions — 15 specs.
  • src/test/java/net/codebox/homoglyph/HomoglyphMultiCodepointTest.java (new): end-to-end Java regressions — 12 tests.

Test results (all run locally)

  • Python generator suite: 13/13 pass.
  • Java (HomoglyphLogicTest + HomoglyphDataTest + HomoglyphMultiCodepointTest, Temurin JDK 17, JUnit 4.11): 24/24 pass (12 pre-existing + 12 new).
  • Node (npm test): 15 new multi-codepoint specs pass. Total 1579 specs, 32 failures — the 32 failures are pre-existing on upstream master (the hand-maintained node/test/spec/DataTests.js has drifted from the bundled data; unrelated to this change).
  • raw_data/char_codes.txt and raw_data/chars.txt are byte-identical to master (single-codepoint data unchanged).

Caveats

  • Scope of the raw multi data: only confusables whose prototype is composed of ordinary word characters ([A-Za-z0-9.-]) are emitted (124 mappings), matching how the JS single-codepoint map is already filtered. Combining-mark and other non-word prototype sequences are excluded because they can never occur in a searchable target word. (Java therefore supports multi-codepoint detection for word-character prototypes only; single-codepoint detection is unchanged and still uses the full data.)
  • Determinism: the single-codepoint JS glyph lists are now emitted in sorted order so python generator/main.py is reproducible; this accounts for the reordering in the homoglyph.js/node/index.js diffs (membership is unchanged).
  • Performance: matching is now recursive with backtracking. Word lengths are tiny and the multi-map is only consulted when a text symbol is itself a multi-codepoint confusable, so the impact is negligible; worst case is bounded by target-word length.
  • The pre-existing node/test/spec/DataTests.js drift is left untouched (out of scope).

@codeboxbot
codeboxbot force-pushed the marvin/multi-codepoint-confusables branch 2 times, most recently from fad437e to d633810 Compare August 27, 2026 02:29
@codeboxbot
codeboxbot force-pushed the marvin/multi-codepoint-confusables branch from d633810 to 9a24b3e Compare August 27, 2026 02:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant