Skip to content
art-test-stackPublic

Repository files navigation

stars-shield license-shield deepwiki-shield hf-page-shield linkedin-shield


      ___           ___           ___           ___       ___           ___     
     /\  \         /\  \         /\  \         /\__\     /\  \         /\  \    
    /::\  \       /::\  \        \:\  \       /:/  /    /::\  \       /::\  \   
   /:/\:\  \     /:/\:\  \        \:\  \     /:/  /    /:/\:\  \     /:/\:\  \  
  /:/  \:\  \   /::\~\:\  \       /::\  \   /:/  /    /::\~\:\  \   /::\~\:\__\ 
 /:/__/_\:\__\ /:/\:\ \:\__\     /:/\:\__\ /:/__/    /:/\:\ \:\__\ /:/\:\ \:|__|
 \:\  /\ \/__/ \/__\:\/:/  /    /:/  \/__/ \:\  \    \/__\:\/:/  / \:\~\:\/:/  /
  \:\ \:\__\        \::/  /    /:/  /       \:\  \        \::/  /   \:\ \::/  / 
   \:\/:/  /         \/__/     \/__/         \:\  \       /:/  /     \:\/:/  /  
    \::/  /                                   \:\__\     /:/  /       \::/__/   
     \/__/                                     \/__/     \/__/         ~~       

gpt-lab* is a light-weight library for monitoring small LLM trainings, supporting inference, for small-scale ablation studies. It also includes an interface to chat with the model, and with models from 🤗 API, locally or remotely.
Explore the docs »

Request Feature »

*This name is quite pompous and vague, I admit it. Any suggestions for a better one are welcomed!

Table of Contents

About The Project

d12 model training on 4 A100 GPUs

422M-parameter model (depth 12) training loss on 4 A100 GPUs.

Purpose

This project is primarily educational*. It implements transformer-based language models from scratch to expose and understand their core mechanisms.

While modern LLMs can generate strong implementations, true understanding comes from building. This repository follows that philosophy: learning through construction and internalization, permitting elaboration. That said, building alone does not guarantee understanding.

"What I cannot create, I do not understand." - Richard Feynman 🐐

This is not a production-ready library. It is a lightweight, transparent playground for training small models, running experiments, ablation studies, and exploring architectural ideas.

Components are often adapted from existing work and properly credited. The goal is not to reinvent the wheel, but to understand it well enough to modify and improve it. At least, that is the intention.

gpt-lab supports distributed training on at most a single GPU node. It is not optimized for large-scale training, but it is designed to be modular and extensible.

A simple but important question would be: Why this repo exists (vs nanoGPT / HF Trainer). Here are some of the motivations:

  • modular training instrumentation for ablations
  • pluggable optimizer + architecture factory
  • distributed streaming dataloaders designed for throughput balance
  • tokenizer experimentation pipeline
  • built-in evaluation / inference interface

*For the non-initiated, there is of course better free-online resources available. Find some at the references section.

Built With

Torch <<3 🐐 (sorry JAX-ers)
huggingface-shield (datasets, transformers, tokenizer, hub)
wandb-shield (training monitoring)
tiktoken-shield (very fast tokenizer encoder)
gradio-shield (web interface -- not really actively developed; may have some bugs and issues)
uv-shield (dependency management and CLI)

Get Started

Setup requirements

This project has been developed and tested with Python 3.12. gpt-lab uses uv to manage dependencies.

  • Clone the repo

    git clone git@github.com:art-test-stack/gpt-lab.git
  • Install dependencies for CUDA device:

    uv sync --extra gpu

    or install dependencies for CPU/MPS device:

    uv sync --extra cpu
  • Install dependencies for development (optional, but recommended if you want to contribute):

    uv sync --group=dev
  • To use the library in jupyter notebooks:

    uv sync --group=notebook

Note

Make sure to adjust the CUDA version in uv.toml if needed. This extra is only available for Linux systems with compatible NVIDIA GPUs. It permits using flash_attention for faster attention computation. Default mode uses kernels implementation, making the installation easier.

Usage

There is many layers in the library, and many components that can be used and customized. The main ones are the following:

I recommend to check out the corresponding deepwiki for more detailed documentation and explanations on the different components of the library. The sketchs generated explain well the interaction between the different modules.

Scripts

The installed gpt-lab command groups tokenizer training, base training/resume, data preparation, cache management, and chat. Start with gpt-lab --help (or uv run gpt-lab --help). See the CLI docstring for commands, defaults, distributed launch, checkpoint resume, and extension points. Existing scripts/ entry points remain compatibility wrappers. Standalone dataloader benchmarks and tokenizer scaling live in experiments/.

For a lightweight recap of runs, curves, evaluation results, and configurations, see the results dashboard guide. The viewer in spaces/results/ can be deployed as a static Hugging Face Space.

Data

TLDR; The dataloader is easily built with the following code snippet. It employs the following strategies:

  • Streaming (not mmap, not in-memory)
  • Packed (not padded batching)
  • Distributed (not replicated dataset)
  • Lazy (not pre-tokenized)
from gpt_lab.data.loader import build_dataloader, DistDataLoader

data_loader: DistDataLoader = build_dataloader(
    name="climbix-base",
    tokenizer=tokenizer,
    column="text",
    split="train",
    seq_len=model.config.max_context,
    batch_size=32,
    base_url="karpathy/climbmix-400b-shuffle", # for starting point
    max_shards=6542 # last shard id for the given dataset (if not provided, it will be computed by probing the server, which can take a while)
) 

Notice that if you use split val, you will get a simple function that returns a dataloader; which permits to ensure the same validation set across different training runs, steps, etc.

from gpt_lab.data.loader import build_dataloader

val_loader_fn = build_dataloader(
    name="climbix-base",
    tokenizer=tokenizer,
    column="text",
    split="val", # same config except here
    seq_len=model.config.max_context,
    batch_size=32,
    base_url="karpathy/climbmix-400b-shuffle", 
    max_shards=6542 
) 

val_loader = val_loader_fn() # called as a function to get the dataloader instance in trainer

We can do whatever we want with maths, modelization, the implementation in PyTorch, etc, but the core component of any Machine Learning system is still its data.

The data processing has roughly three folds. One for train a tokenizer (see Tokenization), one for model training and one for model evaluation. We focus here on model training/evaluation.

The data pipeline basically works with for any dataset available on internet that can be reshaped into contiguous Parquet shards. Any good starting point, are karpathy/climbmix-400b-shuffle or HuggingFaceFW/fineweb-edu (but needs to be re-sharded! see (scripts/reshard_dataset.py)).

Warning

The data loader inside the library assumes a specific structure for the dataset: it needs to be split into shard files with names in the format shard_{:05d}.parquet, where the ids are contiguous integers. If shard names do not follow this format under base_url, they would be simply ignored by the downloader.

sequence packing Sequence packing strategy. From The Smol Training Playbook.

The build_dataloader function the data loader, which is accessible from gpt_lab.data.DistDataLoader. It employs a distributed streaming data pipeline over Parquet shards with on-the-fly tokenization and greedy document packing into fixed-length sequences. It creates cpu and gpu buffers to pre-load the data in contiguous memory, stream the data from local shards downloaded from the given dataset, and feed the model with the data with a packing strategy, to maximize the throughput of the training loop by avoiding the use of padding tokens. It also supports distributed training setups, and can be used with DDP or other distributed training frameworks.

The critical point regarding model training, is that we must make sure to have a good balance between loader time and model forward/backward time to avoid bottlenecks from the data loading process. Given that constraint, the implementated data loader is satisfying.

Automatic configuration

The library provides a minimalistic automatic configuration system that computes optimal model architecture, tokenizer settings, and training hyperparameters using scaling laws.

Subclass AutoGPTConfig to override individual scaling rules; see its class docstring for the methods and an example.

Pass architecture_config="configs/my-architecture.yaml" to AutoGPTConfig to load architecture defaults from a flat YAML mapping, using its field names (for example, act_func: swiglu and d_head: 128). Explicit constructor fields override the file, including False and None. The resolved architecture is used for the vocabulary probe, target model, and reference model before training scaling is computed. Training and tokenizer settings do not belong in this file. Saved configurations contain the resolved values and do not require the YAML file when loaded again.

The CLI accepts the same file with --architecture-config <path>. Omitted architecture options use the file, while explicit options override it. The generated model log and board configuration include the resolved depth and effective aspect_ratio (d_model / n_layers), after rounding width to whole attention heads. The effective ratio can differ from the requested sizing ratio. --disable-value-embeds removes value embedding tables and gates; value_embs must then be absent from --tie-embeddings (for example, use 'embs;head'). --freeze-res-connections makes the residual mixing parameters non-trainable and excludes them from the optimizer. Both flags accept an optional true or false; a bare flag means true. Explicit false reverses a preset setting. Set --res-h 1 --res-x0 0 to preserve the hidden stream without blending the initial embeddings at each layer. These values can also be set as res_h and res_x0 in YAML; omitted values retain the existing initialization schedules. The switches and residual values are saved with the model configuration. Existing checkpoints default to enabled value embeddings and trainable residuals.

Use gpt-lab train --architecture-config configs/llama3.yaml --tokenizer-model gpt2 for the Llama-3-style preset with the GPT-2 tokenizer. Its aspect_ratio: 96 is an experimental sizing choice. With d_head: 128, the default depth of 12 gives width 1152 and nine query/KV heads; depth 16 gives width 1536 and twelve query/KV heads. YAML n_kv_heads: null selects MHA at both target and reference depths. Explicit --n-kv-heads overrides select GQA and must divide both target and reference query head counts. Width scaling keeps its existing rules. RMSNorm stays parameter-free; this preset retains the current input/Q/K normalization, logit softcap, and 4x FFN width, so it does not reproduce the original Llama 3 equations exactly. With 50,257 vocabulary entries, padded to 50,304 embedding rows, it contains 370,704,408 parameters including 24 frozen residual scalars at depth 12 (counted on meta).

The width/depth ratio is a sizing choice, not a constant across the Llama family:

Published model Hidden width Layers Width / layers
Llama 3 8B 4096 32 128
Llama 3 70B 8192 80 102.4
Llama 3.1 405B 16384 126 130.03

Dimensions come from Meta's report, Table 3.

Changing the aspect ratio and model components does not recalibrate the existing scaling laws. Automatic training settings for this preset are experimental: the 12-layer reference model now has this architecture too, while the reference batch size remains 524,288 tokens. The data/parameter target (default 11), batch-size exponent (0.383), batch learning-rate multiplier, weight-decay correction, and embedding/head learning-rate width correction (sqrt(768 / d_model)) have not been validated for this variant. Parameter and FLOPs counts are recomputed from the resolved model; that accounting does not establish optimal training settings. The vocabulary-sizing rule is also unchanged and remains unverified here. --tokenizer-model gpt2 bypasses that estimate and uses the loaded tokenizer's vocabulary. Explicit batch size and --num-steps can bypass automatic batch and horizon sizing; optimizer profiles still receive the existing batch/width/decay multipliers. No training calibration or hardware throughput measurement is implied by this preset.

At depth 16, this preset has 758,513,696 total parameters, including 32 frozen residual scalars, and 681,246,720 scaling parameters (blocks plus output head). A 15-token/parameter target at 524,288 tokens per update gives a 19,490-update horizon. At sequence length 2048 with four GPUs and device batch size 16, this requires four accumulation microsteps per update.

A 1,300-step warmup is ceil(681246720 / 524288), following the one-parameter-token warmup heuristic from Porian et al.. That heuristic was studied with AdamW and is uncalibrated for this Muon profile or its peak learning rates. A ten-update diagnostic reaches only 10/1300 of peak LR and cannot establish stability at peak LR.

The system is based on the AutoConfig class (available in gpt_lab.model.auto.py) and is used by gpt-lab train; scripts/train_base.py delegates to the same command.

It automatically determines:

  • model depth scaling
  • width / aspect ratio expansion
  • vocabulary size (scaling-law driven or tokenizer-based)
  • batch size and gradient accumulation
  • training horizon (steps / FLOPs / data ratio)
from gpt_lab.model.auto import AutoConfig

# Automatic full training configuration
cfg = AutoConfig(
    basename="ic1-125M",
    depth=12,
    aspect_ratio=16,
)
meta_config = cfg.generate_gpt_config(device="cuda")

Note

Setting vocab_size = -1 enables automatic scaling-law vocabulary selection.
Setting training_time, n_steps, target_flops, or target_param_data_ratio controls training horizon priority.
The system automatically builds a reference model (12-layer baseline) to normalize scaling-law computations.

--num-steps sets the complete scheduling horizon. To stop execution earlier without shortening LR, momentum, or weight-decay schedules, use --num-steps 2317 --stop-after-step 945. The cap counts completed optimizer steps since training began, including steps before a resume. It defaults to unset; resume retains a saved cap unless explicitly overridden. A capped run uses normal final checkpointing. See the PR35 comparison for the pinned Modal adapter and verification results.

Training raw loss and its smoothed value use the mean across ranks (equal token counts per rank). The default momentum schedule keeps each optimizer group's configured momentum fixed, independently of LR warmup and warmdown; an explicit Trainer(muon_momentum_schedule=...) still overrides the multiplier. This corrects the previous LR-shaped momentum multiplier and changes resumed training trajectories under the default schedule. Checkpoint formats are unchanged. Old checkpoints' loss EMA retains its previous rank-local history until it decays.

Training FLOPs and MFU use the PaLM convention adapted in nanochat: 6 * (block parameters + output-head parameters) plus 12 * d_model * effective_seq per attention layer, where effective_seq is the context length for full attention or min(left_window, context_length) for sliding attention. This rectangular estimate does not discount causal masks or short prefixes. It excludes input/value embedding lookups, elementwise and optimizer work, casts, and recomputation; it does not measure executed kernel FLOPs. The output head is counted even when tied. MFU is estimated FLOPs/token times global tokens/second divided by the aggregate GPU peak FLOPs/second.

Next sections detail the different generated components.

Tokenization

The tokenization system lives in gpt_lab/tokenizer/ and is organized as follows:

tokenizer/
├── auto.py            # tokenizer resolution and scaling-law selection
├── base.py            # base Tokenizer class and utilities for internal logic
├── corpus.py          # TokenizerCorpus class and training data generation logic
├── hf.py              # HuggingFace backend wrapper and conversion
├── serialization.py   # msgpack save/load, validation, fingerprinting
├── tokenizer.py       # core Tokenizer class and public API
└── truncation.py      # vocabulary truncation helpers

The library supports BPE tokenization only (SentencePiece is a TODO). Training uses the HuggingFace backend; encoding and decoding use tiktoken.

Tokenizers are serialized with msgpack (binary, rank-sorted, SHA-256 fingerprinted) rather than pickle. Legacy vocab.pkl files are still readable with a deprecation warning.

Using a pretrained tokenizer

from gpt_lab.tokenizer import Tokenizer

# Load from tiktoken (default)
tokenizer = Tokenizer.from_pretrained("cl100k_base")

# Load from a specific source
tokenizer = Tokenizer.from_pretrained("cl100k_base", source="tiktoken")

Loading a truncated tokenizer

Truncated tokenizers are identified by name suffix and handled automatically. If a cached version exists on disk it is loaded directly; otherwise the base tokenizer is loaded, truncated, saved, and returned.

# Automatically builds cl100k_base truncated to 32k mergeable ranks
# and caches it to disk for future loads
tokenizer = Tokenizer.from_pretrained("cl100k_base_truncated_32000")

Truncation always preserves all 256 byte-level tokens and reassigns ranks to be contiguous from 0.

Training a tokenizer

from gpt_lab.tokenizer import Tokenizer
from gpt_lab.tokenizer.corpus import TokenizerCorpus
from gpt_lab.utils.schemas import TokenizerConfig, TokenizerTrainerConfig

# uses default corpus settings (mixture of HuggingFaceFW/fineweb-edu, HuggingFaceFW/fineweb-2, HuggingFaceTB/finemath and codeparrot/codeparrot-clean)
corpus = TokenizerCorpus.from_sources(random_seed=42)
trainer_cfg = TokenizerTrainerConfig(
    source="huggingface", # training backend (e.g., "huggingface", "tiktoken", "bpe", "fbpe", "rbpe", "dummy")
    to_save=True, # pattern for pre-tokenization (e.g., "gpt2", "cl100k-base", etc., or regex pattern for custom pre-tokenization)
)
cfg = TokenizerConfig(
    name="my_tokenizer",
    vocab_size=32_000,
    pat_str="gpt2", 
    trainer=trainer_cfg,# whether to save the trained tokenizer to disk
)
tokenizer = Tokenizer.train_from_iterator(iterator=corpus.iterator(), config=trainer_cfg)

Loading a locally saved tokenizer

tokenizer = Tokenizer.from_disk("my_tokenizer")

Expects the directory structure:

<TOKENIZERS_FOLDER>/my_tokenizer/
├── tokenizer_config.json       # includes SHA-256 fingerprint
├── mergeable_ranks.msgpack
└── token_bytes.safetensors

Automatic tokenizer resolution

When using AutoConfig, the tokenizer is resolved automatically based on scaling laws. You can also call the resolver directly:

from gpt_lab.tokenizer.auto import resolve_tokenizer

tokenizer = resolve_tokenizer("cl100k_base", vocab_size=32_000)

Which tokenizer to use?

The training script is at scripts/train_tokenizer.py. For encoding during training and inference, tiktoken is always used regardless of which backend trained the tokenizer.

Note

HuggingFace BpeTrainer does not expose a random seed, so trained tokenizers are not guaranteed to be bit-for-bit reproducible across runs. Pretrained and truncated tokenizers are fully deterministic.

Using a pre-trained tokenizer

from gpt_lab.tokenizer import Tokenizer
from gpt_lab.utils.schemas import TokenizerConfig

tokenizer = Tokenizer.from_pretrained("cl100k-base", source="tiktoken")

Which tokenizer implementation to choose?

The tokenizer training script is located in scripts/train_tokenizer.py. It allows you to train a BPE tokenizer on a custom corpus, using different implementations (tiktoken, HuggingFace, or custom BPE implementations). You can also choose to write the corpus from sources (e.g., Wikipedia, OpenWebText) or load an existing corpus.

Training time benchmarks for different implementations and configurations. All the tokenizers were trained on corpus generated from gpt_lab.tokenizer.corpus.TokenizerCorpus() with default settings, tuned with variable vocab_size.

Scaling laws for tokenizer training

The tokenizer scaling experiment compares ByteLevel BPE tokenizers across corpus sizes, vocabulary sizes, and split patterns to explore their impact on tokenization quality and efficiency.

To experiment yourself tokenizer scaling, you can run the following command from the root directory of the repo:

uv run --extra cpu python -m experiments.tokenizer_scaling \
    --seed 42 \
    --num-procs 16 \
    --vocab-sizes 20000,50000,100000 \
    --pat-strs gpt2,cl100k_base \
    --write-corpus \
    --corpus-sizes-gb 10,50,100,500,1000,5000,10000 

More details are in the experiment guide or --help:

uv run --extra cpu python -m experiments.tokenizer_scaling --help

Model architecture

The library provides a modular implementation of a GPT-style DenseTransformer, where architectural components (attention, feedforward blocks, normalization, and positional encoding) are fully decoupled and configurable. The core model is defined in gpt_lab.model.gpt, while reusable layer primitives are implemented in gpt_lab.model.layers. Model behavior is controlled via a Pydantic configuration class (TransformerConfig) defined in gpt_lab.utils.schemas (but also accessible under gpt_lab.model.gpt), enabling structured extension of architectural variants and hyperparameters.

Each decoder block applies the attention and FFN residual additions. SwiGLU and squared-ReLU FFNs return only their transformed branch. Older checkpoints trained with the additional inner FFN skip still load with the same parameter keys, but their outputs and resumed training trajectories change under this architecture.

Here a simple example to produce a Llama-like architecture:

from gpt_lab.model.gpt import TransformerConfig, DenseTransformer

cfg = TransformerConfig(
    vocab_size=32_000,
    max_context=2048,
    d_model=512,
    n_heads=8,
    n_kv_heads=8,
    n_layers=6,
    d_ffn=2048,
    # Select "sdpa", "fa2", "fa3", or a Hub "owner/kernel" name.
    attn_impl="sdpa",
    use_packed_attn_kernel=False,  # Opt in only when useful for your workload.
    act_func="swiglu",
    normalization="rms",
)
model = DenseTransformer(cfg)

attn_impl="fa3" loads kernels-community/flash-attn3 version 2; "fa2" loads kernels-community/flash-attn2 version 4. Explicit requests fail if the selected kernel cannot load; they do not switch to SDPA.

The single packed QKV projection is split into Q, K, and V head views. Q and K each pass through RoPE, then the configured per-head normalization, in PyTorch; torch.compile can fuse this preprocessing. The gated value residual is preserved. Attention uses separate Q/K/V inputs by default. To opt into packed attention during training, set use_packed_attn_kernel=True or pass --use-packed-attn-kernel with --attn-impl fa3 (or another compatible kernel). FA3 then concatenates the results into [B, T, Hq + 2*Hkv, D], including GQA. FA2 and custom kernels can use their five-dimensional packed layout for MHA. Unsupported layouts or kernels without a packed entrypoint keep separate inputs; SDPA and inference also keep their existing paths. The flag is saved in the model config and restored on resume; configs without the field default to separate inputs. CausalSelfAttention loads its configured backend once through load_flash_attn. The helper selects separate SDPA, FA2, FA3 or Hub execution functions and returns a SimpleNamespace, including a fixed KV-cache function. Model forward calls never select or reload a backend. The existing module-level functions remain thin compatibility entrypoints for direct callers, retaining their arguments; require_flash_attn2/require_flash_attn3, availability warnings and the public flash_attn namespace remain available. Hardware initialization happens on first load, after rank-local device setup. The wrappers retain their dtype handling; the layer adds no extra V cast. Direct KV-cache calls keep their backend=None default and existing automatic selection behavior.

A name containing /, such as attn_impl="kernels-community/flash-attn3" or --attn-impl owner/kernel, selects that Hub kernel's main revision. This explicitly permits loading code from the named repository. The kernel must export a FlashAttention-compatible flash_attn_func; an optional flash_attn_qkvpacked_func enables opt-in packed dispatch, and flash_attn_with_kvcache is required for KV-cache decoding. Dropout and ALiBi are forwarded only when the selected entrypoint supports them; requesting an unsupported option raises an error. Attention weights and arbitrary masks require SDPA. Bare custom names follow main, so their code may change between downloads; the built-in fa3 alias selects version 2 instead.

The kernel loader uses kernels==0.18.3 and PyTorch 2.11.0+cu128 for the GPU extra. The optional fp8 extra pins torchao==0.17.0. Models saved with an older PyTorch/kernel combination retain their architecture, but matching their previous numerical results requires the original software environment.

During training, dropout masks the attention output and FFN branch before their residual additions, preserving the skip path. Evaluation disables dropout.

Optimization

The optimizer system uses a registry-based, strategy-pattern architecture that decouples optimizer logic from execution mode (single-GPU vs. distributed). Optimizers are built and configured by calling DenseTransformer.build_optimizer in the Trainer class (available in gpt_lab.train.trainer) using the optimizer configuration from configs/optim.yaml, which can specify mixed optimizer groups.

Optimizer Architecture

The system is organized around three key components:

  • Registry (gpt_lab.optim.registry): Defines optimizer specifications with required hyperparameters and validates YAML configs early.
  • Strategy Interface (gpt_lab.optim.strategy): Abstract OptimizerStrategy base class with methods for local_step, dist_reduce, and dist_compute (3-phase async pattern).
  • Concrete Strategies (gpt_lab.optim.strategies/): Per-optimizer implementations (AdamW, Muon, etc.). No cross-strategy duplication.
  • Factory (gpt_lab.optim.factory): Single entry point that automatically selects single-GPU or distributed backend and delegates to the appropriate strategies.

Configuration Example

opt: muon retains the existing variant: Polar Express orthogonalization, shape-dependent factored normalization, and cautious weight decay. opt: normuon follows the official NorMuon reference update: BF16 fixed-coefficient Newton–Schulz iterations, row-wise second-moment normalization, and standard decoupled weight decay. Both use the existing whole-matrix optimizer sharding and require same-shaped 2D parameters per group. The configuration key beta is NorMuon's beta2; momentum is its first-moment coefficient. The existing Muon momentum schedule also applies to NorMuon. NorMuon uses a compiled update kernel; fusion may change intermediate BF16 rounding relative to the eager reference implementation.

Use --optim-config-path configs/normuon.yaml for a NorMuon override profile. For Muon or NorMuon, an optimizer YAML override can select the QKV update granularity.

transformer:
  muon_qkv: projection

Fresh runs can also pass --muon-qkv joint|projection|head to gpt-lab train auto; an explicit CLI value overrides the loaded optimizer YAML profile.

  • joint (default) updates the entire QKV matrix together.
  • projection updates Q, K, and V independently, including unequal row counts in GQA.
  • head updates each Q, K, and V head independently.

All modes retain one fused QKV linear projection. Attention dispatch and the packed-attention opt-in are independent of this setting. Split modes reuse the compiled optimizer step on batches of equal-sized blocks, with normalization and learning-rate scaling based on each logical matrix shape. Resume with the same split layout to restore momentum and variance state. The default joint mode retains the existing optimizer checkpoint layout; incompatible split layouts are rejected on load.

Splitting uses writable views of the existing stacked optimizer buffers. MHA projection and head modes each use one batched step; unequal-width GQA projections use two (Q, then K/V). Muon head splitting can increase factored variance storage: wide heads keep one value per input column per head, while the joint tall matrix keeps one per output row. NorMuon keeps row-wise variance in every mode. On CPU, noncontiguous Muon blocks use the existing eager kernel to avoid a PyTorch 2.11 Inductor fusion failure; contiguous CPU batches and CUDA keep the compiled path. Throughput and peak memory for these modes have not been benchmarked.

Explicit preserved Muon/NorMuon/spectral-only ANVIL experiment profiles are in optimizer comparison. Its hyperparameters are starting values, not calibrated replacements for the current Muon profile. Existing defaults and Muon checkpoints retain their behavior. Choose NorMuon for a fresh run; changing a saved group's optimizer name is not a checkpoint conversion because normalization state layouts can differ.

The isolated communication experiment compares the original transport, fixed buckets, a packing pipeline, and backward hooks. It preserves NCCL checks and measured negative results; baseline communication remains recommended.

default:
  opt: "adamw"
  eps: 1e-10
  weight_decay: 1e-3
embeddings: 
  opt: "adamw"
  lr: .3
  betas: [.8, .995]
  eps: 1e-10
  weight_decay: 1e-3
transformer:
  opt: "muon"
  lr: 2e-2
  momentum: .95
  ns_steps: 5
  beta: .9
  weight_decay: .28
...

Usage

from gpt_lab.optim import OptimizerFactory

model = ...
optim_cfg = ... # dict of optimizer hyperparameters per group
param_groups = [
    {"params": model.embeddings.parameters(), **optim_cfg["embeddings"]},
    {"params": model.blocks.parameters(), **optim_cfg["transformer"]},
    ...
]
optimizer = OptimizerFactory(param_groups)

The factory automatically:

  • Validates all required keys for each optimizer at construction (fails fast on config errors)
  • Selects local or distributed optimizer execution
  • Manages a reusable scalar cache for torch.compile stability
  • Routes parameter groups to the correct strategy at each step

Adding a New Optimizer

To add a new optimizer (e.g., Aurora), you only need to edit one place:

  1. Create strategy file src/gpt_lab/optim/strategies/aurora.py:

    from gpt_lab.optim.strategy import OptimizerStrategy
    
    class AuroraStrategy(OptimizerStrategy):
        def local_step(self, group: dict) -> None:
            # Single-GPU step logic
            pass
        
        def dist_reduce(self, group: dict, world_size: int) -> dict:
            # Phase 1: launch async reductions, return info dict
            pass
        
        def dist_compute(self, group: dict, info: dict, gather_list: list, rank: int, world_size: int) -> None:
            # Phases 2-3: wait, compute, launch gathers
            pass
  2. Export it from src/gpt_lab/optim/strategies/__init__.py:

    from gpt_lab.optim.strategies.aurora import AuroraStrategy
    __all__ = [..., "AuroraStrategy"]
  3. Register in src/gpt_lab/optim/factory.py (in the "Register built-in optimizers" section):

    register_optimizer(OptimizerSpec(
        name="aurora",
        required_keys=frozenset({"opt", "lr", "momentum", "weight_decay"}),
        strategy_class=AuroraStrategy,
    ))
  4. Use in YAML:

    optimizer:
      - opt: aurora
        lr: 1e-4
        momentum: 0.9
        weight_decay: 0.0

Current Implementations

  • AdamW (strategies/adamw.py): Standard AdamW with ZeRO-2 style sharding for large parameters.
  • Muon (strategies/muon.py): MomentUm Orthogonalized by Newton-Schulz with efficient stacked parameter updates.

Both support single-GPU and multi-GPU distributed training with async communication patterns.

Pre training

Use gpt-lab train to train from scratch on prepared local Parquet shards, or gpt-lab train resume to restore a saved run. Training automatically configures the model and training parameters from depth and scaling laws; --train-tokenizer trains a new tokenizer. The existing scripts/train_base.py entry point remains a compatibility wrapper.

Explicit batch, vocabulary, and horizon options override automatic values. Optimizer base values come from YAML and are scaled once; use --optim-config-path to override them through a file. Resume restores the saved architecture and rejects architecture options.

The same workflow accepts Python keyword arguments:

from gpt_lab.utils.schemas import OptimizerConfig
from gpt_lab.workflows.train_base import train_base

optim_config = OptimizerConfig.from_file("configs/optim.yaml")
train_base(depth=12, total_batch_size=524288, optim_config=optim_config)
# Single-device training
uv run gpt-lab train --model-name ic1 --run-name baseline --depth 12

# Eight GPUs on one node
uv run torchrun --standalone --nproc-per-node=8 --no-python \
  gpt-lab train --model-name ic1 --run-name baseline --depth 12

Distributed utilities return a frozen ComputeContext through the init_dist_groups() and get_comp_info() functions. Pass it as comp_info and use attributes such as .rank, .world_size, .device, and .compute_dtype. For CPU runs, .device_type and .device_name both return "cpu"; .gpu_peak_flops remains unknown (float("inf")). Distributed runs use a one-dimensional PyTorch DeviceMesh named "dp", exposed as .dp_mesh; single-device runs have no mesh or process group. CPU runs use Gloo and CUDA runs use NCCL. Distributed AdamW and Muon wait on async Work handles; Gloo gradients use SUM followed by averaging, while NCCL retains AVG. Trainer configs exclude this runtime object when saved; loading a checkpoint attaches the current launch context, including for older dictionary metadata. Sharded optimizer resumes still require the original world size. Checkpoint APIs use optimizer_state_layout="full" or "sharded", independently of local or distributed optimizer execution. The current distributed optimizer shards its state, so its checkpoints use "sharded"; local runs use "full". Readers accept legacy mode="ddp" / "shard" metadata. One-worker sharded checkpoints contain full optimizer state and can resume locally: loaders read the rank-0 files, while subsequent local checkpoints use "full".

--no-python tells torchrun to execute the installed gpt-lab command directly. With an activated environment, omit uv run. See the CLI docstring for module-mode launches and checkpoint selection.

Explicit CUDA capture (experimental)

--cuda-graph-mode forward_backward opts into fixed-shape forward/backward capture around compiled kernels. disabled preserves existing compiled execution. The captured path disables compiler-managed CUDA graphs to avoid nested capture. It requires CUDA, FP32 or BF16, the default compute_loss, and a fixed pair of input/label tensors. FP16/scaler and FP8 capture are currently rejected.

One graph replay consumes one microbatch. Inputs and gradients use persistent storage; gradient accumulation, dataloader work, finite-loss checks, scheduler, optimizer and communication remain outside capture. Gradients are cleared in place after the update. Warmup restores parameters, model buffers, gradients and CPU/CUDA RNG state. Python forward side effects and changing parameter usage are unsupported. Evaluation uses the uncaptured evaluation path; resuming rebuilds the graph over restored state. Omit the mode to retain a saved mode on resume, or explicitly pass disabled when resuming without capture.

train/capture_startup_seconds reports compilation/warmup/capture separately from step timing. Small-model real-CUDA equivalence checks passed on eight A100 SXM4 GPUs. See the experiment README for measured results and limits. Optimizer-update and communication capture are pending.

The standalone scripts/modal_cuda_graphs.py launcher follows the existing local runs/modal_trainbase.py. It requires a clean branch whose HEAD matches its live origin tip, Modal 1.6.2 authentication, the gpt-lab secret (HF_TOKEN, WANDB_API_KEY), and existing gpt-lab-cache and gpt-lab-runs volumes. It clones the exact selected SHA; it never uploads local source or starts a GPU for --dry-run.

Every GPU job first runs the small real-CUDA equivalence check in scripts/benchmark/cuda_graph_equivalence.py. It uses a two-layer production transformer, the explicitly selected optimizer profile, and six updates with changing production LR/momentum/decay schedules. Both paths use compiled kernels with compiler-managed graphs disabled. It compares accumulated loss, pre-update accumulated gradients, parameters, and all optimizer state after each update. It also checks in-place gradient clearing, fixed gradient addresses, warmup restoration of model/buffers/optimizer/RNG, and replicated parameters across ranks. Distributed checks use the actual NCCL reduce/scatter/gather backend outside capture. Checks collect every rank's result before allowing timing. A skipped or failed check cannot authorize a run.

The small check uses batch 2 and sequence 16, with the requested run's accumulation count. If the timed run uses one microstep, the gate checks both one and two microsteps from matching seeded states. Floating-point comparisons use rtol/atol 1e-3/1e-6 for BF16 and 1e-5/1e-6 for FP32; Python counters and state keys must match exactly. It saves observed maximum absolute errors and all failures. These tolerances are explicit acceptance criteria; the experiment README records the completed user-launched CUDA checks. No optimizer or communication capture is added.

Run these commands from the exp/cuda-graphs worktree after pushing. They allocate paid GPUs only when the user executes them. First run the small check:

uv run --no-project --python 3.12 --with modal==1.6.2 python scripts/modal_cuda_graphs.py \
  --gpus 1 --mode verify --require-sxm --run-name cg-check-1-v1 \
  -- --optim-config-path configs/normuon.yaml --board dummy

The optional matched eight-GPU comparison verifies distributed equivalence again on its own allocation before either timed run. The eight-GPU preflight enforces microbatch 16 or 32, sequence 2048, global batch 524288, horizon 2317, warmup 300 and cap 150. Microbatch 16 uses two accumulation microsteps; microbatch 32 uses one. Both modes retain BF16, seed 42, depth 12, width 768, fused attention, and the same explicit optimizer profile. Evaluation, generation and all model/optimizer checkpoint saves are disabled.

uv run --no-project --python 3.12 --with modal==1.6.2 python scripts/modal_cuda_graphs.py \
  --gpus 8 --mode paired --require-sxm --run-name cg-8-matched-v1 \
  -- --optim-config-path configs/normuon.yaml --board dummy --n-acc-steps 2

For device batch 32, use --device-batch-size 32 --n-acc-steps 1 after the -- separator in both the paired timing and separate profile commands. Use distinct run names such as cg-8-b32-matched-v1 and cg-8-b32-profile-v1. The global batch and scheduler remain unchanged.

Paired runs execute sequentially within one GPU worker, keeping the allocation and topology fixed. Each mode reconstructs initial state in a fresh process from the same seed and restarts the deterministic dataloader at its initial position. Before capture starts, the second process must match the first process's model, optimizer, RNG, first-input hashes and initial data position. Final data positions and execution settings must also match before a paired comparison is accepted. This is a verified fresh-process reset, not a continuation from the first run. Each timed process has a fresh compiler cache. Compiler build threads are fixed to one per rank to bound concurrent compilation on eight ranks.

Collect the separate short profile with the same model/data settings:

uv run --no-project --python 3.12 --with modal==1.6.2 python scripts/modal_cuda_graphs.py \
  --gpus 8 --mode paired --profile --require-sxm --run-name cg-8-profile-v1 \
  -- --optim-config-path configs/normuon.yaml --board dummy --n-acc-steps 2

Profile jobs preserve the 2317/300 scheduler and use a separate 14-update execution cap. PyTorch CPU/CUDA profiling waits ten updates, warms up one, then records three. Open trace_rank_N.json in Perfetto or a Chrome trace viewer to inspect host gaps, CUDA launches and kernel activities. kernels_rank_N.txt contains the top 40 entries by self device time. Regions label batch preparation, accumulated updates and optimizer/communication. Profiling overhead is excluded from all throughput comparisons by using separate jobs and profile-only summaries.

The runs volume stores /runs/<run-name>/comparison.json, equivalence.log, per-rank equivalence JSON, gpus.csv (all GPU names/UUIDs/bus IDs/memory/driver), and topology.txt (nvidia-smi topo -m). The status of each hardware query is recorded in hardware_metadata.json and the root comparison, with stdout/stderr saved separately. If an NVML query fails, execution continues with explicitly partial hardware metadata and attempts cuda_connectivity.json for GPU names/memory and CUDA peer-access capability. This fallback does not identify physical NVLink/PCIe links or their bandwidth. GPU validation and the all-rank equivalence gate remain mandatory before timing. Execution-mode directories disabled/ and forward_backward/ contain launch settings, logs, per-rank metrics and summary.json. Timed summaries report global tokens/s over updates 11–150, per-rank startup/steady peak allocated and reserved memory, trainer elapsed time including capture setup, and complete subprocess elapsed time including model initialization, compilation, data setup and shutdown. The root comparison also reports equivalence time and total worker elapsed time. Modal image build and provisioning remain outside worker timers and appear in Modal logs.

Use new run names for retries. --require-sxm rejects allocations other than NVIDIA A100-SXM4-80GB before checking/training; it cannot force Modal placement. --mode disabled and --mode forward_backward remain available for individual runs, and --gpus accepts 1, 4 or 8. --mode verify needs no cached dataset. --detach submits one worker in the background. These diagnostics do not measure convergence quality; timing and profile claims await real user-launched results.

Customizing the trainer

Subclass Trainer and override train_step for a complete optimizer step, or prepare_batch, compute_loss, and optimizer_step for smaller changes. update_metrics updates per-step state; log_metrics handles reporting. Override evaluate_bpb, evaluate_core, and evaluate_samples independently; the main loop schedules evaluations. Use self.compiled_model for training forwards. See the class docstring for contracts and DistillationTrainer for an example.

Checkpointing

The framework provides a simple checkpointing system that is integrated with the training loop. The checkpointing system is implemented in the CheckpointManager class (available in gpt_lab.model.checkpoint) and allows you to save and load model checkpoints during training. Models should be saved in the following directory structure:

<CACHE_DIR>/
└── models/
    └── <model_name>/                               # e.g., "ic1", "gpt2-small", "llama2"
        └── <run_name>/
            ├── meta.pskl                           # immutable (model, tokenizer, git)
            └── <source>/                           # base / sft / rl
                ├── training_config.safetensors     # per-phase config
                ├── checkpoint_state.safetensors    # best bpb/core steps and values
                ├── checkpoint_step_000000/
                │   ├── model.safetensors
                │   ├── optim_rank0.safetensors     # optimizer state (otherwise optim.safetensors)
                │   ├── optim_rank1.safetensors
                │   ├── ...                         # more optimizer shards if needed
                │   ├── trainer_state_rank0.json
                │   ├── rng_state_rank0.safetensors
                │   └── metrics.pkl
                ├── checkpoint_step_000100/
                │   └── ...
                └── ...

Model, optimizer, RNG, scaler, and trainer configuration files use safetensors. Nested state is described by JSON metadata inside each file; model weights keep their original state-dict keys. Tokenizer tensor caches and diagnostic batches also use safetensors. On first load, legacy .pt, .pth, and checkpoint .pkl files are converted to safetensors beside the originals, then loaded through the new format. Conversion requires a writable directory; originals are retained, and existing safetensors files take precedence.

Board

Vizualize the training progress in the board of your choice (Tensorboard, Weights & Biases, or Trackio). You can also log to a dummy board that does not log anything, for faster training without logging overhead.

Chat with the model

Use a saved local checkpoint in the console:

uv run gpt-lab chat console --model-name ic1 --run-name baseline

The existing experimental web interface is available through:

uv run gpt-lab chat app

See the CLI docstring for UI dependency requirements, remote-completion configuration, and the existing Gradio compatibility limitation.

Development Notes

Some components are intentionally incomplete. Contributors (including automated tools) are encouraged to explore TODOs and propose improvements via pull requests.

References

Nice repositories to check out for inspiration and reference

  1. karpathy/nanoGPT by Andrej Karpathy for pretraining code .
  2. karpathy/nanochat by Andrej Karpathy for full training (base, sft, grpo) and inference pipeline.
  3. KellerJordan/modded-nanogpt by Jordan Keller for speedrunner implementation and optimization techniques.

Some nice blogs and web-articles

  1. Huggingface or nanotron playbooks. All of them are very good. It takes days to read them all, and more to diggest, but they are worth it.
  2. frontier model training methodologies by Alex Wa (DJ Dumpling). Quite compact compared with the HuggingFace playbooks, but still very informative and insightful (it is quite a condensed version of it).
  3. Making Deep Learning Go Brrrr From First Principles by Horace He (PyTorch). Very nice intro on basics of GPU computation for deep learning.
  4. Tokenizers by Karparthy for a very nice overview of tokenization for LLMs.

Some bibliography

Note

All of the literature ressources below all participated in some way to the development of the library. I have probably forgotten some, and I apologize for that. If you think some important papers are missing please feel free to add one (or suggest one) via pull request. Some papers are not directly cited in the code, I will try to add some as much as possible in the future.

Title Authors Journal Year DOI
How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models nan nan nan
Tokenization Is More Than Compression nan nan nan
Practical Efficiency of Muon for Pretraining AI et al. arXiv 2025 2505.02222
The Potential of Second-Order Optimization for LLMs: A Study with Full Gauss-Newton Abreu et al. arXiv 2025 2510.09378
Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-training Bergsma et al. arXiv 2025 2505.13738
Knowledge distillation: A good teacher is patient and consistent Beyer et al. arXiv 2021 2106.05237
Language Models are Few-Shot Learners Brown et al. arXiv 2020 10.48550/arXiv.2005.14165
PaLM: Scaling Language Modeling with Pathways Chowdhery et al. arXiv 2022 10.48550/arXiv.2204.02311
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness Dao et al. NeurIPS 2022 2205.14135
FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning Dao arXiv 2023 10.48550/arXiv.2307.08691
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning DeepSeek-AI et al. Nature volume 645, pages 633-638 (2025) 2025 10.1038/s41586-025-09422-z
QLoRA: Efficient Finetuning of Quantized LLMs Dettmers et al. arXiv 2023 10.48550/arXiv.2305.14314
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Devlin et al. arXiv 2018 10.48550/arXiv.1810.04805
Fewer Truncations Improve Language Modeling Ding et al. arXiv 2024 2404.10830
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity Fedus et al. ICML 2021 2101.03961
How to Train Long-Context Language Models (Effectively) Gao et al. arXiv 2024 10.48550/arXiv.2410.02660
Accelerating Newton-Schulz Iteration for Orthogonalization via Chebyshev-type Polynomials Grishina et al. arXiv 2025 2506.10935
Shampoo: Preconditioned Stochastic Tensor Optimization Gupta et al. arXiv 2018 10.48550/arXiv.1802.09568
Training Compute-Optimal Large Language Models Hoffmann et al. arXiv 2022 10.48550/arXiv.2203.15556
LoRA: Low-Rank Adaptation of Large Language Models Hu et al. ICLR 2021 2106.09685
Block-Recurrent Transformers Hutchins et al. arXiv 2022 2203.07852
Mistral 7B Jiang et al. arXiv 2023 10.48550/arXiv.2310.06825
Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention Katharopoulos et al. arXiv 2020 10.48550/arXiv.2006.16236
KellerJordan/Muon KellerJordan GitHub 2024 github:kellerjordan/muon
KellerJordan/modded-nanogpt KellerJordan GitHub 2024 github:kellerjordan/modded-nanogpt
KIMI K2: OPEN AGENTIC INTELLIGENCE Kimi Team arXiv 2025 10.48550/arXiv.2507.20534
Attention Residuals Kimi Team arXiv 2026 10.48550/arXiv.2603.15031
Decoding-time Realignment of Language Models Liu et al. arXiv 2024 2402.02992
Muon is Scalable for LLM Training Liu et al. 2025 arXiv 2502.16982
StarCoder 2 and The Stack v2: The Next Generation Lozhkov et al. arXiv 2024 10.48550/arXiv.2402.19173
YaRN: Efficient Context Window Extension of Large Language Models Peng et al. arXiv 2023 10.48550/arXiv.2309.00071
Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free Qiu et al. arXiv 2025 2505.06708
Language models are unsupervised multitask learners Radford et al. OpenAI 2019 unsupervised-multitask
SQUAD: 100,000+ Questions for Machine Comprehension of Text Rajpurkar et al. arXiv 2016 10.48550/arXiv.1606.05250
Observational Scaling Laws and the Predictability of Language Model Performance Ruan et al. arXiv 2024 10.48550/arXiv.2405.10938
Observational Scaling Laws and the Predictability of Language Model Performance Ruan et al. arXiv 2024 2405.10938
SlimPajama-DC: Understanding Data Combinations for LLM Training Shen et al. arXiv 2023 2309.10818
How to Train Your Energy-Based Models Song et al. arXiv 2021 10.48550/arXiv.2101.03288
Building Bridges between Regression, Clustering, and Classification Stewart et al. arXiv 2025 2502.02996
RoFormer: Enhanced Transformer with Rotary Position Embedding Su et al. arXiv 2021 10.48550/arXiv.2104.09864
Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies Tao et al. arXiv 2024 2407.13623
Efficient Transformers: A Survey Tay et al. LLMs BasicsarXiv 2020 10.48550/arXiv.2009.06732
Attention is all you need Vaswani et al. arXiv 2017 10.48550/arXiv.1706.03762
ResidualTransformer: Residual Low-Rank Learning with Weight-Sharing for Transformer Layers Wang and Li arXiv 2023 2310.02489
Fantastic Pretraining Optimizers and Where to Find Them Wen et al. arXiv 2025 2509.02046
Unified Training of Universal Time Series Forecasting Transformers Woo et al. arXiv 2024 2402.02592
Effective Long-Context Scaling of Foundation Models Xiong et al. arXiv 2023 10.48550/arXiv.2309.16039
HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughly Yen et al. arXiv 2024 10.48550/arXiv.2410.02694
Recursive Language Models Zhang et al. arXiv 2025 2512.24601
dLLM: Simple Diffusion Language Modeling Zhou et al. arXiv 2026 10.48550/arXiv.2602.22661

Bibliography made with art-test-stack/MyBible.

Some video ressources

For the laziest (😛), there are also a lot of Youtube videos that explain well the different components of the library, and how to implement them. Here are some of them that I found useful:

  1. Andrej Karpathy's YouTube channel for his unmatched expertise in the field, and his ability to explain complex concepts in a simple and intuitive way. His videos on Transformers and LLMs are particularly useful for understanding the architecture and training of these models.
  2. Stanfords CME295 course for the very nice lecture on Transformers and LLMs by Afshine and Shervine Amidi. They currently releasing lectures of CME296, which is on diffusion & LVMs.

Extra

  1. Banner made with: hacker-tools/ascii-banner
  2. For some extra documentation: Ask DeepWiki

TODOs

Here a non-exhaustive list of features that I aim to implement. Stars correspond to the priority level. Contributions are welcome!

  • Tokenization ⭐️
    • BPE implementation in Python
    • Rust implementation
  • Architecture ⭐️⭐️⭐️
    • Alibi
    • MoE
    • Mixture of Depths
  • Optimization ⭐️⭐️
    • Shampoo optimizer
    • LION optimizer
    • MARS optimizer
  • Precision ⭐️⭐️
    • model and optimizer quantization
  • Training ⭐️⭐️⭐️
    • fine-tuning / intruction tuning
    • grpo
  • Cross-lib features ⭐️⭐️
    • HuggingFace integration (model loading, tokenizers, etc.)
    • vLLM, DeepSpeed, Megatron-LM, etc. integration

License

GPT-Lab's original code is distributed under the MIT License. Third-party components remain under their original licenses; see THIRD_PARTY_NOTICES.md for details.

Contact

Arthur Testard - arthur.testard.pro@gmail.com

Project Link: art-test-stack/gpt-lab

Citation

If you use this work in your research, please consider citing the following:

@misc{gptlab2026,
  author={Testard, Arthur},
  title={gpt-lab: A light-weight library for fast-ablation studies on GPT-like LMs},
  year={2026},
  publisher={GitHub},
  url={https://github.com/art-test-stack/gpt-lab}
}

(back to top)

Used by

Contributors

Languages