{"name":"llama-cpp-python: Python Bindings for llama.cpp","description":"llama-cpp-python provides robust Python bindings for the popular llama.cpp library, enabling efficient local inference with large language models. It offers a high-level API compatible with OpenAI's API, facilitating easy integration into existing applications. The project also includes a powerful web server for local deployment and supports various hardware acceleration backends.","github":"https://github.com/abetlen/llama-cpp-python","url":"https://osrepos.com/repo/abetlen-llama-cpp-python","source":"osrepos.com","sourceDescription":"This repository profile is provided by osrepos.com, an open source repository discovery platform.","repositoryProfile":"https://osrepos.com/repo/abetlen-llama-cpp-python","generatedFor":"open source discovery and AI-assisted research","markdown":"https://osrepos.com/repo/abetlen-llama-cpp-python.md","json":"https://osrepos.com/repo/abetlen-llama-cpp-python.json","topics":["Python","AI","Machine Learning","LLM","llama.cpp","OpenAI API","Local Inference","NLP"],"keywords":["Python","AI","Machine Learning","LLM","llama.cpp","OpenAI API","Local Inference","NLP"],"stars":null,"summary":"llama-cpp-python provides robust Python bindings for the popular llama.cpp library, enabling efficient local inference with large language models. It offers a high-level API compatible with OpenAI's API, facilitating easy integration into existing applications. The project also includes a powerful web server for local deployment and supports various hardware acceleration backends.","content":"### Introduction\n`llama-cpp-python` is a crucial project that brings the power of `llama.cpp` to the Python ecosystem. It offers simple yet comprehensive Python bindings, allowing developers to interact with large language models (LLMs) locally. This package is designed to provide both low-level access to the C API via `ctypes` and a high-level Python API for common tasks like text completion, chat completion, and embeddings. With support for OpenAI-like API, LangChain, and LlamaIndex compatibility, `llama-cpp-python` makes local LLM deployment and experimentation accessible to a broader audience.\n\n### Installation\nGetting started with `llama-cpp-python` is straightforward. The primary method involves installing directly via `pip`, which also builds `llama.cpp` from source to optimize for your system.\n\nbash\npip install llama-cpp-python\n\n\nFor basic CPU support, pre-built wheels are also available:\n\nbash\npip install llama-cpp-python \\\n  --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cpu\n\n\nTo leverage hardware acceleration like CUDA, Metal (MPS), or OpenBLAS, you can set `CMAKE_ARGS` environment variables during installation. For example, with CUDA:\n\nbash\nCMAKE_ARGS=\"-DGGML_CUDA=on\" pip install llama-cpp-python\n\n\nDetailed instructions for various backends and pre-built CUDA/Metal wheels can be found in the official documentation.\n\n### Examples\n`llama-cpp-python` offers a high-level API designed for ease of use, mimicking the OpenAI API for familiar workflows.\n\n**Text Completion:**\npython\nfrom llama_cpp import Llama\n\nllm = Llama(\n      model_path=\"./models/7B/llama-model.gguf\",\n      # n_gpu_layers=-1, # Uncomment to use GPU acceleration\n)\noutput = llm(\n      \"Q: Name the planets in the solar system? A: \",\n      max_tokens=32,\n      stop=[\"Q:\", \"\\n\"],\n      echo=True\n)\nprint(output)\n\n\n**Chat Completion:**\nThe API supports various chat formats, making it easy to interact with models designed for conversational AI.\n\npython\nfrom llama_cpp import Llama\nllm = Llama(\n      model_path=\"path/to/llama-2/llama-model.gguf\",\n      chat_format=\"llama-2\"\n)\nllm.create_chat_completion(\n      messages = [\n          {\"role\": \"system\", \"content\": \"You are an assistant who perfectly describes images.\"},\n          {\n              \"role\": \"user\",\n              \"content\": \"Describe this image in detail please.\"\n          }\n      ]\n)\n\n\n**Multi-modal Models (e.g., LLaVA):**\nThe library also supports multi-modal models, allowing for image and text input.\n\npython\nfrom llama_cpp import Llama\nfrom llama_cpp.llama_chat_format import Llava15ChatHandler\nchat_handler = Llava15ChatHandler(clip_model_path=\"path/to/llava/mmproj.bin\")\nllm = Llama(\n  model_path=\"./path/to/llava/llama-model.gguf\",\n  chat_handler=chat_handler,\n  n_ctx=2048,\n)\nllm.create_chat_completion(\n    messages = [\n        {\"role\": \"system\", \"content\": \"You are an assistant who perfectly describes images.\"},\n        {\n            \"role\": \"user\",\n            \"content\": [\n                {\"type\" : \"text\", \"text\": \"What's in this image?\"},\n                {\"type\": \"image_url\", \"image_url\": {\"url\": \"https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg\" } }\n            ]\n        }\n    ]\n)\n\n\n### Why Use It\n`llama-cpp-python` stands out for several reasons:\n*   **Local Inference**: Run powerful LLMs directly on your machine, ensuring data privacy and reducing reliance on cloud APIs.\n*   **OpenAI API Compatibility**: Seamlessly integrate with existing applications built for the OpenAI API, minimizing code changes.\n*   **Hardware Acceleration**: Supports various backends like CUDA, Metal, OpenBLAS, and ROCm, optimizing performance on different hardware.\n*   **Rich Feature Set**: Beyond basic completion, it offers chat completion, function calling, multi-modal support, JSON mode, speculative decoding, and embeddings.\n*   **Web Server**: Includes an OpenAI-compatible web server for easy local deployment and access from any client.\n*   **Active Development**: The project is actively maintained and welcomes contributions, ensuring continuous improvement and new features.\n\n### Links\n*   **GitHub Repository**: [https://github.com/abetlen/llama-cpp-python](https://github.com/abetlen/llama-cpp-python){:target=\"_blank\"}\n*   **Official Documentation**: [https://llama-cpp-python.readthedocs.io/en/latest](https://llama-cpp-python.readthedocs.io/en/latest){:target=\"_blank\"}","metrics":{"detailViews":8,"githubClicks":5},"dates":{"published":null,"modified":"2025-11-11T20:00:56.000Z"}}