Our take
- Activity
- 72
- Community
- 62
- Issues
- 35
- Pull requests
- 66
+39 stars in the last 30 days
- Releases more often than 93% of the projects we track
- More contributors than 78% of the projects we track
- Slower to close issues than 98% of the projects we track
Project history
Daily GitHub snapshots since Oct 7, 2026. History covers stars and forks; health signals are not tracked historically.
Stars
10,690Forks
1,490Change since Nov 18, 2026: +2 stars·0 forks
Overview
llama-cpp-python connects Python applications to llama.cpp, enabling local inference with compatible models. It offers both a managed Python interface and direct access to llama.cpp’s C API, so projects can choose convenience or lower-level control.
It is aimed at developers building local language-model features or serving models behind an OpenAI-compatible API. Unlike a hosted model service, it runs the model on the user’s machine, with performance and setup depending on the selected model, hardware, and build configuration.
Key Features
- Generate text and chat completions with the
LlamaPython API. - Access the underlying llama.cpp C API through Python
ctypesbindings. - Run an OpenAI-compatible web server for existing API clients.
- Configure CPU and hardware acceleration backends, including CUDA, Metal, HIP, and Vulkan.
- Support structured JSON output, JSON Schema, and tool or function calling.
- Work with supported multimodal models, including image input.
- Create embeddings and use speculative decoding.
- Load compatible GGUF models from local files or the Hugging Face Hub.
Use Cases
- Python developers can add local text or chat generation to an application without sending prompts to a hosted inference API.
- Teams can expose a local GGUF model through an OpenAI-compatible endpoint for clients that already use that API style.
- Researchers and engineers can experiment with model settings, embeddings, or low-level llama.cpp functions from Python.
- Developers can build image-and-text workflows with supported multimodal models and chat handlers.
Getting Started
Install the package with pip:
pip install llama-cpp-python
The default installation builds llama.cpp from source. See the README for model examples, hardware-specific build options, and server setup.
Alternatives
- textgen: TextGen offers a user-facing web interface and model management, while llama-cpp-python provides Python bindings and an OpenAI-compatible server for llama.cpp.
- LocalAI: LocalAI is a multi-backend server for language, vision, audio, and image models, while llama-cpp-python focuses on llama.cpp inference from Python and its compatible server.
| Project | Language | License | Stars | Status |
|---|---|---|---|---|
| llama-cpp-python | Python | MIT | 10.6k | Not checked yet |
| textgen | Python | AGPL-3.0 | 47.7k | Active |
| LocalAI | Go | MIT | 49.4k | Active |
Considerations
- Installation requires Python 3.8 or later and a C compiler. Building from source may require configuring CMake and the desired acceleration backend.
- Inference is local, so available memory and hardware constrain which models and context sizes are practical. GPU acceleration requires a compatible build and device.
- The Python API depends on llama.cpp, so changes to its C API may require corresponding binding updates.
- The repository is actively developed, and the README describes installation and supported options that can vary by platform and backend.
Found this useful?
Share it with someone who would like llama-cpp-python.