optimum: Optimize Model Training and Inference on Target Hardware

Overview

Optimum is a Python library that connects Hugging Face model workflows with optimization and acceleration tools for different hardware. It helps developers export, run, and train models using backends such as OpenVINO, ONNX Runtime, ExecuTorch, and accelerator-specific integrations.

It is most useful when a standard Transformers workflow needs to target a particular device or inference runtime. The core package is a starting point, while many backends require separate extras or companion projects.

Key Features

• Integrates optimization workflows with Transformers, Diffusers, TIMM, and Sentence Transformers.
• Supports model export and optimized inference across several runtimes and hardware ecosystems.
• Provides integrations for Intel OpenVINO, Intel Gaudi, AWS Inferentia and Trainium, AMD hardware, NVIDIA TensorRT-LLM, and other providers.
• Offers training integrations for supported accelerators, including Gaudi and AWS Trainium.
• Includes command-line and programmatic workflows for supported export and optimization tasks.
• Connects users to separate projects for some capabilities, including Optimum ONNX, Optimum Quanto, and Optimum ExecuTorch.

Use Cases

• A machine-learning engineer can export a Transformers model for an inference runtime supported by their deployment hardware.
• An application team can evaluate OpenVINO or another supported backend when deploying models on a specific accelerator ecosystem.
• A model trainer can use the Gaudi or Trainium integrations when training on supported hardware.
• An edge developer can use the Optimum ExecuTorch project to export Transformers models for PyTorch-based on-device inference.

Getting Started

Install the base package:

python -m pip install optimum

Accelerator integrations may need additional dependencies. See the README and documentation for backend-specific setup and workflows.

Alternatives

• litgpt: LitGPT focuses on configurable LLM training, fine-tuning, evaluation, and serving, while Optimum focuses on hardware-specific optimization across model libraries.

Considerations

• The base install does not by itself provide every accelerator integration. Select the relevant extra and follow its hardware and dependency requirements.
• ONNX integration has moved to the separate optimum-onnx repository, so use that project's installation guidance for ONNX workflows.
• Some paths depend on specific hardware or external runtimes, and their setup differs by provider.
• The repository has 302 open issues, so check current documentation and issue status when assessing a particular integration.