Repository navigation

Add SentencePiece/LLaMA tokenizer path (#464) - #466

Merged

michalharakal merged 4 commits into

feature/464-tokenizer-path

Apr 13, 2026

Merged

Add SentencePiece/LLaMA tokenizer path (#464)#466
michalharakal merged 4 commits into
developfrom
feature/464-tokenizer-path

Commits on Apr 12, 2026
- Add SentencePieceTokenizer core (#464 step 1)
  
  michalharakal
  and
  claude
  committed
  1d8ad07
- Wire SentencePieceTokenizer into TokenizerFactory (#464 step 2)
  
  michalharakal
  and
  claude
  committed
  9e061cd
Commits on Apr 13, 2026
- Add TinyLlama end-to-end fixture test for SentencePiece (#464 step 3)
  
  michalharakal
  and
  claude
  committed
  eb8f350
- Document #464 SentencePiece path in CHANGELOG (#464 step 4)
  
  michalharakal
  and
  claude
  committed
  9a8fdc9