Add multi-codepoint confusable support to homoglyph search - #22
Open
codeboxbot wants to merge 1 commit into
Open
Add multi-codepoint confusable support to homoglyph search#22codeboxbot wants to merge 1 commit into
codeboxbot wants to merge 1 commit into
Conversation
codeboxbot
force-pushed
the
marvin/multi-codepoint-confusables
branch
2 times, most recently
from
August 27, 2026 02:29
fad437e to
d633810
Compare
codeboxbot
force-pushed
the
marvin/multi-codepoint-confusables
branch
from
August 27, 2026 02:42
d633810 to
9a24b3e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds support for multi-codepoint confusables — cases where a single glyph in the text stands in for a sequence of characters in the target word. For example the ligature
ffi(U+FB03) resemblesffi, soofficeis now found when searching foroffice; the lettermresemblesrn, socomeris found when searching forcorner.Previously the generator's data-file parser called
chr(int(code, 16))on the target field, which raised on any multi-codepoint target (e.g."0066 0066 0069") and silently dropped it, so all ~2000 multi-codepoint confusables in the UTS #39 data were discarded. This PR captures them via a separate mapping and threads them through every consumer.This is separate from PR #21 (the UTS #39 v17 data refresh) and does not touch it.
Semantics
When matching a target word against the text, a single text symbol matches either:
Alternatives are explored with backtracking, so a multi-codepoint substitution composes with ordinary single-codepoint substitution (e.g.
0ffice→office, where0→oandffi→ffi). Single-codepoint behaviour and the publicsearch(...)APIs are unchanged; the matched substring may now be shorter than the target word.Changes
Generator
data_file_parser.py: detects multi-codepoint targets (is_multi/get_multi) and routes them to a new manager instead of dropping them.multi_char_manager.py(new): collects glyph → prototype-sequence mappings.output_multi_char_codes.py(new) +templates/multi_char_codes.txt(new): emitraw_data/multi_char_codes.txt.output_js.py: emits a secondMULTI_CHAR_MAPinto the JS artifacts and now sorts the single-codepoint glyph lists so regeneration is deterministic.main.py: wires the new manager and output in.Runtime
raw_data/multi_char_codes.txt(new):<glyph hex>;<space-separated prototype hex>per line.javascript/src/homoglyph.js,node/index.js: regenerated — new backtracking matcher + embeddedMULTI_CHAR_MAP.buildSearchFunctiontakes an optional secondmultiCharMapargument (defaults to{}, so existing callers are unaffected).Homoglyph.java: new backtracking matcher; newHomoglyph(List<Set<Integer>>, Map<Integer,List<int[]>>)constructor (old constructor preserved).HomoglyphBuilder.java: loads the optional bundledmulti_char_codes.txt; newbuild(Reader, Reader)overload.build.gradle: bundles the new resource.Tests
generator/tests/test_multi_codepoint.py(new): parser/manager/output regressions using real UTS #39 lines — 13 tests.node/test/spec/MultiCharTests.js+javascript/tests/js/tests/MultiCharTests.js(new): end-to-end JS regressions — 15 specs.src/test/java/net/codebox/homoglyph/HomoglyphMultiCodepointTest.java(new): end-to-end Java regressions — 12 tests.Test results (all run locally)
13/13pass.HomoglyphLogicTest+HomoglyphDataTest+HomoglyphMultiCodepointTest, Temurin JDK 17, JUnit 4.11):24/24pass (12 pre-existing + 12 new).npm test): 15 new multi-codepoint specs pass. Total1579 specs, 32 failures— the 32 failures are pre-existing on upstream master (the hand-maintainednode/test/spec/DataTests.jshas drifted from the bundled data; unrelated to this change).raw_data/char_codes.txtandraw_data/chars.txtare byte-identical to master (single-codepoint data unchanged).Caveats
[A-Za-z0-9.-]) are emitted (124 mappings), matching how the JS single-codepoint map is already filtered. Combining-mark and other non-word prototype sequences are excluded because they can never occur in a searchable target word. (Java therefore supports multi-codepoint detection for word-character prototypes only; single-codepoint detection is unchanged and still uses the full data.)python generator/main.pyis reproducible; this accounts for the reordering in thehomoglyph.js/node/index.jsdiffs (membership is unchanged).node/test/spec/DataTests.jsdrift is left untouched (out of scope).