#

byte-pair

Here are 2 public repositories matching this topic...

VenkatRamaraju / polyglot

a multilingual tokenizer written from scratch in go

tokenizer lllm byte-pair

Updated Sep 15, 2025
Go

99degree / gpu_assisted_bpe

Use my old buddy iGPU HD620 to do token matching.

gpu vulkan tokens tokenization utf-32 byte-pair-encoding byte-pair tokenizerbpe utf8-utf32

Updated Apr 1, 2026
C++

Add this topic to your repo

To associate your repository with the byte-pair topic, visit your repo's landing page and select "manage topics."