ggml: A Low-Level Tensor Library for Machine Learning

Introduction

ggml is a powerful tensor library specifically engineered for machine learning applications. It stands out for its low-level, cross-platform implementation, making it highly versatile and efficient. The library is under active development, with significant contributions also happening within related projects such as llama.cpp and whisper.cpp.

Key features of ggml include integer quantization support, broad hardware compatibility, and built-in automatic differentiation. It also incorporates ADAM and L-BFGS optimizers, all without relying on any third-party dependencies, ensuring a lean and performant codebase. A notable design principle is its commitment to zero memory allocations during runtime, which contributes to its exceptional efficiency.

Installation

To get started with ggml, clone the repository and follow the build steps:

git clone https://github.com/ggml-org/ggml
cd ggml

# install python dependencies in a virtual environment
python3.10 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

# build the examples
mkdir build && cd build
cmake ..
cmake --build . --config Release -j 8

Examples

Here's an example of how to run GPT-2 inference using ggml:

# run the GPT-2 small 117M model
../examples/gpt-2/download-ggml-model.sh 117M
./bin/gpt-2-backend -m models/gpt-2-117M/ggml-model.bin -p "This is an example"

For more detailed examples and usage scenarios, explore the examples folder within the repository.

Why Use ggml?

ggml offers several compelling advantages for machine learning developers:

• Low-level and Cross-platform: Provides fine-grained control and runs efficiently across various operating systems and architectures.
• Integer Quantization Support: Enables efficient model deployment on resource-constrained devices by reducing model size and computational requirements.
• Broad Hardware Support: Designed to work across a wide range of hardware, including specialized accelerators.
• Automatic Differentiation: Simplifies the implementation of complex machine learning models and training algorithms.
• Optimizers Included: Comes with ADAM and L-BFGS optimizers built-in, ready for use in training processes.
• No Third-Party Dependencies: Reduces complexity, potential conflicts, and improves portability.
• Zero Memory Allocations During Runtime: Ensures highly efficient memory usage, crucial for performance-critical applications.

Links

• GitHub Repository: https://github.com/ggml-org/ggml
• Roadmap: https://github.com/users/ggerganov/projects/7
• Manifesto: https://github.com/ggerganov/llama.cpp/discussions/205
• Introduction to ggml (Hugging Face Blog): https://huggingface.co/blog/introduction-to-ggml
• The GGUF file format: https://github.com/ggerganov/ggml/blob/master/docs/gguf.md