llm-compressor: Compress Models for vLLM Inference

Overview

LLM Compressor is a Python library for reducing and transforming large language models before deployment with vLLM. It provides post-training compression methods that can reduce model storage and inference memory needs while producing checkpoints in the compressed-tensors format.

It is most useful when you control the model preparation stage and want to choose a quantization method or precision for a target deployment. It integrates with Hugging Face models and includes workflows for distributed processing and disk offloading, but it is a compression toolkit rather than an inference server.

Key Features

• Quantizes weights and activations using formats including INT4, INT8, FP8, NVFP4, and MXFP4.
• Provides algorithms such as GPTQ, AWQ, SmoothQuant, AutoRound, and simple post-training quantization.
• Supports attention and KV-cache quantization, in addition to model weight and activation compression.
• Saves checkpoints in compressed-tensors format for use with vLLM.
• Integrates with Hugging Face models and repositories.
• Offers distributed data parallel and disk-offloading options for large-model compression.
• Includes examples for MoE, vision-language, and audio-language models.

Use Cases

• Inference engineers can prepare quantized checkpoints when full-precision model memory requirements are too high for their serving hardware.
• Model platform teams can compare supported quantization schemes and formats for a vLLM deployment target.
• Teams working with large models can use distributed processing or disk offloading when model compression exceeds a single-device workflow.
• Practitioners deploying MoE or multimodal models can start from architecture-specific examples rather than building a compression flow from scratch.

Getting Started

Install the package:

pip install llmcompressor

For quantization examples, vLLM inference setup, and detailed configuration, see the repository README and documentation.

Alternatives

• AIMET: AIMET offers broader quantization and compression tools for deep learning models, while llm-compressor focuses on preparing Hugging Face models for vLLM.

Considerations

• Compression requires loading and processing model weights, so hardware and memory needs depend on the model, algorithm, and workflow. The project documents distributed and disk-offloading options, but these do not remove the need to plan resources.
• Quantization formats and algorithms have different accuracy and hardware trade-offs. Validate the resulting checkpoint for your workload and target vLLM environment.
• Some capabilities are explicitly experimental, including the README's SpinQuant attention quantization example.
• LLM Compressor prepares model checkpoints; serving them requires a compatible inference runtime such as vLLM.