Skip to content

Home

Jump to bottom
JamePeng edited this page Aug 24, 2026 · 11 revisions

llama-cpp-python Wiki

logo

Welcome to the llama-cpp-python wiki :)

This wiki provides source-aligned documentation for the public APIs, core classes, feature workflows, examples, and maintainer tooling in llama-cpp-python. The latest code in llama_cpp/ and the corresponding vendored llama.cpp APIs remain the source of truth.

Quick Navigation

Getting Started

Page Description
Installation Build and source-installation guide covering Python setup, CMake options, native backends, hardware acceleration, rebuilds, and verification.

Core API

Page Description
Llama Main high-level interface for GGUF loading, text and chat completion, tokenization, embeddings, sampling, speculative decoding, caching, and lifecycle management.

Modules

Page Description
Llama Cache Cache interfaces and implementations for reusing model state across repeated prompts.
Llama Embedding Dedicated embedding APIs, configuration, output formats, and batching behavior.
Llama Grammar Grammar parsing and constrained-generation utilities.
Llama Speculative Decoding Stateful MTP, DFlash, DSpark, and n-gram engines; configuration, lifecycle, rollback, statistics, and benchmarks.
Logger Python and native logging configuration, callbacks, levels, filtering, and output routing.

Feature Guides

Page Description
Embeddings and Reranking End-to-end sentence embeddings, token-level vectors, normalization, streaming batches, similarity output, and cross-encoder reranking.

Development

Page Description
Git Commit Generation Agent Maintainer workflow for producing clear, structured, and source-aware Git commit messages.

Wiki Maintenance

Page Description
Wiki Schema Documentation structure, page templates, source requirements, and maintenance rules.
Contributing to the Wiki Contribution workflow for creating and updating documentation.

Recommended Reading Order

For general model loading and generation:

  1. Installation
  2. Llama
  3. Llama Cache
  4. Llama Grammar
  5. Logger

For embeddings and reranking:

  1. Llama Embedding
  2. Embeddings and Reranking

For speculative decoding:

  1. Llama
  2. Llama Speculative Decoding

For documentation contributors:

  1. Wiki Schema
  2. Contributing to the Wiki
  3. Git Commit Generation Agent

Documentation Status

Completed pages currently linked from this index:

  • install.md
  • core/Llama.md
  • modules/LlamaCache.md
  • modules/LlamaEmbedding.md
  • modules/LlamaGrammar.md
  • modules/LlamaSpeculative.md
  • modules/Logger.md
  • features/embeddings-rerank.md
  • development/git-commit-generation-agent.md
  • SCHEMA.md
  • contributing-to-wiki.md

The repository also contains empty placeholder files for planned documentation. They are intentionally not linked as usable pages until content has been added and checked against the implementation.

Planned areas

  • Basic and chat-completion examples
  • Speculative-decoding examples
  • Vision and audio examples
  • Caching, grammar, multi-model, and tool-call feature guides
  • Low-level llama.cpp and MTMD ctypes bindings
  • Common and MCP type references
  • Troubleshooting and backend diagnostics

Documentation Principles

  • Treat source code as the source of truth.
  • Keep parameters, defaults, version availability, and behavior aligned with the latest implementation.
  • Prefer complete, runnable examples without local machine-specific paths.
  • Clearly mark deprecated APIs, preview features, and current limitations.
  • Distinguish stable public interfaces from private implementation helpers.
  • Do not link empty placeholder pages as finished documentation.
  • Keep pages concise, practical, and easy to navigate.

Project Links

Clone this wiki locally