Repository navigation
tokenizer: index added tokens by first char and length - #1443
Merged
Merged
Conversation
AddedTokenMatcher compared every added token against every input position. Gemma's GGUF declares ~6,400 added tokens, so one 70-character chat turn cost ~450k region compares. A token can only match at a position if it starts with that character, and at most one token per length can match, so the matcher now keeps the distinct lengths per first character and does one map lookup per length. Same results (longest first, single_word falls back to shorter); 2.0 ms -> 36 us per encode on the JVM with a Gemma-3 270M vocab, ids identical. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
michalharakal
added a commit
that referenced
this pull request
Oct 24, 2026
…nd on AVX2 CHANGELOG [0.62.2] - 2026-10-24: the schedule-driven split of the dense FP32 x·Wᵀ product (#1441), the runtime-dispatched AVX2/FMA lane for the same kernel on x86-64 (#1442), the CoroutineSchedule.forRange empty-range fix, the added-token index in the text front-end (#1443), and the dependabot bumps. The #1441 entry, dropped from Unreleased by a develop merge, is restored. VERSION_NAME, CITATION, Antora version, README bom coordinate, What's New and contributors, the Android tutorial coordinates; regenerated operator and kernel-support matrices (version stamp).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
AddedTokenMatcher(behindSpecialTokenSplitterand the inline matchers) compared every added token against every input position. Gemma's GGUF declares ~6,400 added tokens, so one 70-character chat turn cost about 450kregionMatchesperencode.A token can only match at a position if it starts with that position's character, and at most one token of each length can match there. The matcher now keeps the distinct token lengths per first character and does one map lookup per length, longest first. The semantics are unchanged: the longest match wins, and a rejected
single_wordmatch falls back to the next shorter token.Measured (JVM, Gemma-3 270M fine-tune vocab, 262k pieces, 6,442 added tokens, five chat-turn prompts): 2.0 ms → 36 µs per encode, ids byte-identical for all five prompts. On a 32-bit ARM Android device the old encode took 36–166 ms per prompt.
Tests:
skainet-frontend-textjvmTest 67/67 andskainet-io-corejvmTest 143/143. A newSpecialTokenSplitterTestcase covers thesingle_wordfallback to a shorter token with the same first character, and a candidate longer than the remaining text. It passes both with and without the change.🤖 Generated with Claude Code