[lalalune]

Adds tools/misaki-g2p: a C++17 port of misaki 0.9.4's American English G2P. It is set up the way paradee/tts.py uses it, because Paradee-8M was trained only on misaki phonemes:
en.G2P(trf=False, british=False, fallback=espeak.EspeakFallback(british=False), unk=''), plus paradee's _chunks.

It has no ggml dependency and builds standalone or through add_subdirectory. The root and tools/ CMake hooks belong to the paradee-native change, so this PR does not touch them.

What's in it

• convert_misaki_lexicon.py converts the pinned us_gold.json/us_silver.json (hexgrad/misaki@fba1236, the same bytes as the 0.9.4 wheel) into misaki-en-us.lex. Before converting, it checks the sha256 of both inputs.
  • It applies grow_dictionary.
  • The output is sorted and mmap-able, and keeps the per-tag variants, including null values.
  • The output is deterministic: 12,947,790 B, sha256 fbf09090…3925, pinned in pins.json.
• The library ports:
  • spaCy 3.8's English tokenizer: prefix, suffix and infix rules, URL_MATCH, the exception list and special-case matching.
  • misaki's subtokenizer.
  • A rule-based tagger in place of en_core_web_sm.
  • Lexicon lookup, stemming and stress.
  • Number words, written independently because num2words is LGPL. They match num2words on 22,201 of 22,201 checks.
  • NNP spelling, merge_tokens, resolve_tokens, and the markdown [word](/phonemes/) features.
  • The eSpeak NG fallback, called the way phonemizer calls it (ties, punctuation preserve/restore), then misaki's E2M table. That table is cross-checked against paradee's web/misaki.js.
  • paradee's 510-symbol chunking.
• C API: misaki_g2p_open, misaki_g2p_phonemize (returns '\n'-separated chunks; fails on any symbol outside the Paradee vocab), misaki_g2p_fallback_kind (1 = eSpeak+E2M, 0 = spell-out), misaki_g2p_free, misaki_g2p_close, and misaki_g2p_espeak_linked.
• C++ interface (eliza_misaki::misaki_load_lexicon, misaki_get_fallback_kind, misaki_phonemize, misaki_phonemize_chunks) for the ABI v17 bridge already merged in libelizainference ABI v17: Paradee + misaki TTS entries, ASR cancel/supported, TTS cancel and evict for Kokoro/Paradee #59. The declarations match tools/omnivoice/tests/paradee-contract/misaki-g2p.h.

Agreement gate (CTest misaki_g2p_agreement)

The reference is pinned Python misaki 0.9.4 with spaCy 3.8.16, en_core_web_sm 3.8.0, phonemizer-fork 3.3.2 and espeakng-loader 0.2.4 (libespeak-ng 1.52.0). Its output is committed as tests/fixtures/reference.txt with a .sha256 file that the gate verifies.

corpus	words	sentences
Gate: paradee samples (5) + WikiText held-out first 200 + eliza qualify sentences.txt (25)	3751/3752, 99.97% (gate 99.5%)	226/230, 98.26% (gate 98%)
Assistant-style sentences (70, also gated)	619/619	70/70
WikiText 200..1700 (not gated)	99.93%	96.1%
WikiText 9000..11000 (not gated, never tuned on)	99.89%	95.8%

The gate's report lists every mismatch. The four remaining gate failures all come from spaCy's statistical tags, which the rules can't reproduce:

• two straight-quote directions in WikiText's " spaced " quotes;
• a dash spaCy tagged .;
• "wear and tear" tagged as a verb.

The gate passes with one sentence of margin (226 against the 225.4 the threshold needs). The not-gated rows show where the rules stop generalising.

Unit tests cover the tokenizer, subtokenizer, stress, E2M and numbers against expectation tables generated from the reference implementations. They also cover the C and C++ APIs, chunking, vocabulary validation, and bounded behaviour on hostile input.

The library builds cleanly with -Wall -Wextra -Wpedantic -Wshadow -Wmissing-declarations -Werror on macOS, and for arm64-v8a with NDK r28 (-Werror, no eSpeak).

Licence flags (flagged, not blocking)

• eSpeak NG 1.52.0 is GPL-3.0-or-later. It is linked only when MISAKI_G2P_ESPEAK=ON and the library is found. Without it, unknown words are spelled out with the lexicon's letter pronunciations, fallback_kind is 0, and those words differ from what Paradee was trained on. See LICENSES/eSpeak-NG-NOTICE.txt.
• WikiText sentences in tests/corpus are CC-BY-SA-3.0 test data.
• num2words (LGPL) is used only by the reference script.
• spaCy's tables are MIT (LICENSES/spaCy-MIT.txt).
• misaki itself is Apache-2.0 (LICENSES/misaki-*).

Notes for the sibling changes

• Integration needs add_subdirectory(tools/misaki-g2p) in front of omnivoice, plus the espeak-ng target or library and the en data from kokoro-voice/inputs.json for Android. libespeak-ng keeps its data path in a 160-byte buffer, so the on-device espeak-ng-data path must stay under about 135 bytes.
• Chunking differs from paradee. eliza_paradee_chunk packs several sentences into one forward pass. paradee runs one pass per _chunks item, which is what misaki_phonemize_chunks / misaki_g2p_phonemize return. Synthesizing each chunk separately would match the reference inference.
• eSpeak is process-global. If kokoro_lib also initialises eSpeak in the same process, the two will re-initialise it with different data paths. Kokoro is being retired from the mobile path.

AI assistance: Claude wrote this change and its tests. The reference outputs come from the pinned Python misaki.

🤖 Generated with Claude Code