Repository navigation

tokenizer: index added tokens by first char and length - #1443

Merged

michalharakal merged 2 commits into

perf/sentencepiece-encode

Oct 24, 2026

Merged

tokenizer: index added tokens by first char and length#1443
michalharakal merged 2 commits into
developfrom
perf/sentencepiece-encode

Commits on Oct 24, 2026
- tokenizer: index added tokens by first char and length
  
  michalharakal
  and
  claude
  committed
  d4f827b
- Merge branch 'develop' into perf/sentencepiece-encode
  michalharakal
  authored
  cf0b487