Fix katakana lookup: normalize halfwidth forms to fullwidth

[abdulazizalmalki-gh]

Problem

Halfwidth katakana (U+FF61-FF9F, common in manga and legacy UI text) never matches fullwidth headwords: hovering over ﾃﾚﾋﾞ finds nothing for the dictionary entry テレビ.

Fix

When the hovered text contains halfwidth katakana, its NFKC-normalized fullwidth form is added as an additional candidate chain, so the usual prefix decomposition runs on both spellings:

• ﾃﾚﾋﾞ → ﾃﾚﾋﾞ / ﾃﾚﾋ / ﾃﾚ / ﾃ and テレビ / テレ / テ
• ｶ + ﾞ combining sequences fold to voiced characters (ｶﾞｽ → ガス)

No other behavior changes: fullwidth katakana runs are still cut into all prefixes, exactly as before (hovering ソ in ソフトウェアエンジニア still yields ソフトウェア…, ソフトウェア, ソフト, ソ), and hiragana/kanji okurigana handling is untouched. The エンジニア-part lookup and the エンジニアー long-vowel spelling also keep their original chains.

(Prior revision removed per @wtetsu's review — dropped the mid-run collapse guard to restore the show-all-plausible-candidates design.)

Tests

Added to the existing "Test Japanese words" suite: halfwidth→fullwidth headword lookup with both prefix chains, voiced-fold case, and a regression assertion that fullwidth katakana prefix cutting still works. npm test: 274/274 pass.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.75%. Comparing base (1fe59fd) to head (52f6cb5).

Additional details and impacted files

@@           Coverage Diff           @@
##           master     #106   +/-   ##
=======================================
  Coverage   98.74%   98.75%           
=======================================
  Files          51       51           
  Lines        2155     2170   +15     
  Branches      415      421    +6     
=======================================
+ Hits         2128     2143   +15     
  Misses         27       27           

Flag	Coverage Δ
unittests	98.75% <100.00%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[wtetsu]

Thank you for the proposal.
I agree that supporting halfwidth katakana would be useful.

However, I'm afraid this change doesn't work as intended.

For example, with the current behavior, hovering over "ソ" in "ソフトウェアエンジニア" should show all of the following candidates at once (depending on the installed dictionaries):

• ソフトウェアエンジニア
• ソフトウェア
• ソフト
• ソ

Hovering over the "エンジニア" part should show the following (again, depending on the installed dictionaries):

• エンジニア
• エン
• エ

This is the intended behavior, and this PR breaks it.

Furthermore, with a spelling like "ソフトウェアエンジニアー", nothing would be shown at all unless the dictionary contains that exact spelling.

A core design principle of Mouse Dictionary is to show all plausible candidates at once, and this change goes against that principle, so I can't merge it as is.

[abdulazizalmalki-gh]

@wtetsu You're right — the mid-run collapse guard broke the show-all-candidates design, and I verified it against your exact examples before reworking: with the guard, hovering ソ in ソフトウェアエンジニア yielded only the full string (no ソフトウェア/ソフト/ソ), ノーベル賞 lost ノー… i.e. every katakana prefix cut was suppressed, and ソフトウェアエンジニアー fell back to nothing beyond the raw string.

I've dropped that part entirely. The PR now does only the halfwidth normalization you agreed would be useful: for halfwidth input the NFKC form is added as a second candidate chain, so all the usual prefixes are generated for both spellings and fullwidth katakana behavior is byte-for-byte unchanged.

Confirmed against your examples on the new code (via createLookupWordsJa on the hovered run):

• ソフトウェアエンジニア → …ソフトウェアエンジニア/…/ソフトウェア/ソフト/ソフ/ソ ✔
• エンジニア → エンジニア/エンジニ/エンジ/エン/エ ✔
• ソフトウェアエンジニアー → still contains ソフトウェアエンジニア ✔
• ノーベル賞 → ノーベル賞/ノーベル/ノーベ/ノー/ノ ✔ (no more ノー suppression)

Rebased onto current master; 274/274 tests pass.

[wtetsu]

If the goal of this PR is to handle halfwidth katakana, applying normalize("NFKC") to the whole string probably causes unintended conversions in the non-katakana parts as well.

NFKC may well be useful for some characters other than halfwidth katakana. But the current code applies NFKC to the entire string as soon as it contains even a single halfwidth katakana character, which seems a bit odd.

To keep this PR focused on its goal, would it be possible to convert only the halfwidth katakana?

[abdulazizalmalki-gh]

Good point — that was sloppy. Reworked in 15a0ddc: the conversion now runs per character inside the halfwidth-katakana range only, so nothing else in the string gets a compatibility fold (① stays ① — whole-string NFKC would have rewritten it to "1").

The one thing I couldn't drop is the fold of ｶ + ﾞ -> ガ. Converting U+FF9E/FF9F alone yields the combining marks ゙/ ゚, and the precomposed fullwidth letters (ガ, ﾀ->ダ etc.) don't exist as single conversions of the halfwidth pairs, so a canonical-composition pass is needed after the scoped replace. I used NFC: it composes canonically (exactly what the voiced marks need) and, unlike NFKC, performs no compatibility mappings, so it can't touch anything outside katakana. A test now pins the scoping: "①ﾃﾚﾋﾞ" produces "①テレビ/①テレ/…" chains and asserts "1テレビ" is never generated.

[wtetsu]

The intent of this code isn't very clear, and it seems inefficient. Could you rewrite it?

For example, I don't see any reason to define two nearly identical regexes and use one for test() and the other for replace(). Isn't replace() alone enough? If there's no halfwidth katakana in the string, it simply does nothing.

Also, this code calls the conversion function once per character, which looks inefficient. I think this could also cause dakuten-related processing to produce incorrect results.

Wouldn't a regex like /[\uFF61-\uFF9F]+/g solve this?

"ﾃﾚﾋﾞ".replace(/[\uFF61-\uFF9F]+/g, (run) => run.normalize("NFKC")) // => "テレビ"

[abdulazizalmalki-gh]

Done in 4fe574f — the run-based regex is exactly right and simpler than what I had. One regex, one NFKC call per maximal run, and the dakuten concern you raised is handled by it directly: NFKC over the run composes ｶ + ﾞ -> ガ in the same pass, whereas my per-character version produced a decomposed カ+combining mark until a second NFC pass fixed it (and would have left ﾞ following non-kanakana input decomposed).

The test()/replace() duplication is gone too: a string without halfwidth katakana comes back unchanged from replace(), so the existing converted !== str check is the only guard needed.

Behavior verified unchanged on top of this: ﾃﾚﾋﾞ/ｶﾞｽ chains as before, ①ﾃﾚﾋﾞ keeps ① intact, あﾞ outside a katakana run stays out of scope. 274/274 tests, biome/tsc clean.

[wtetsu]

Thanks!