GPT-style language model with Byte Pair Encoding tokenizer, built from scratch in PyTorch.
-
Updated
May 5, 2026 - Python
GPT-style language model with Byte Pair Encoding tokenizer, built from scratch in PyTorch.
Java library implementing Byte-Pair Encoding Tokenization
A lightweight, from-scratch implementation of Byte Pair Encoding (BPE) tokenization in Python.
This project implements a tokenizer based on the Byte Pair Encoding (BPE) algorithm, with additional custom tokenizers, including one similar to the GPT-4 tokenizer.
[Rust] Unofficial implementation of "SuperBPE: Space Travel for Language Models" in Rust
From-scratch Marathi BPE tokenizer with Flask API and web interface for encoding/decoding Marathi text using Byte Pair Encoding algorithm.
A python package to build a corpus vocabulary using the byte pair methodology and also a tokenizer to tokenize input texts based on the built vocab.
A web app to compare pre-built or self-built tokenizers
MayaTok is a Byte Pair Encoding based Tokenizer.
Strings Tokenization with Byte Pair Encoding (BPE).
A pure Python implementation of Byte Pair Encoding (BPE) tokenizer. Train on any text, encode/decode with saved models, and explore BPE tokenization fundamentals.
A byte-level BPE tokenizer built from scratch in Python with Unicode support, vocabulary learning, encode/decode, serialization, tests, and compression evaluation.
This repository is reimplementation of Transformer model which was introduced in 2017 NeurIPS paper "Attention is all you need"
self made byte-pair-encoding tokenizer
Natural Language Processing course assignments @ University of Tehran
A 27.8M-parameter GPT language model trained from scratch on TinyStories, featuring custom BPE tokenization, KV-cache inference, speculative decoding, and Intel Arc GPU benchmarking.
Byte Pair Encoding (BPE) in the Zig programming language (0.13.0)
implementation of BPE algorithm and training of the tokens generated
To associate your repository with the bytepairencoding topic, visit your repo's landing page and select "manage topics."