Repository History
2 repositories tagged with quantization
LLM Compressor: Optimize LLMs for Deployment with vLLM
LLM Compressor is a Transformers-compatible Python library designed to apply various compression algorithms to Large Language Models (LLMs). It enables optimized deployment, especially with vLLM, by offering a comprehensive set of quantization techniques for weights, activations, and KV Cache. This tool seamlessly integrates with Hugging Face models, making LLM optimization accessible and efficient.
ggml: A Low-Level Tensor Library for Machine Learning
ggml is an innovative tensor library designed for machine learning, emphasizing low-level, cross-platform implementation. It offers features like integer quantization, automatic differentiation, and broad hardware support, all while maintaining zero third-party dependencies and efficient memory usage. This project is actively developed and forms the backbone for other popular projects like llama.cpp and whisper.cpp.