LLMBox: Train and Evaluate Large Language Models

Overview

LLMBox brings large language model training and evaluation into one Python library. It is intended to reduce the need to assemble separate tools for fine-tuning models, preparing training data, running inference, and comparing results across benchmarks.

The project is a fit for researchers and developers working with supported Hugging Face or API-based models. Its training workflows include several fine-tuning approaches, while its utilization tools cover different evaluation methods, in-context learning, and chain-of-thought evaluation. See the repository README and documentation for setup and supported models and datasets.

Key Features

• Training workflows for supervised fine-tuning, pre-training, PPO, and DPO.
• LoRA and QLoRA support for parameter-efficient fine-tuning.
• Dataset mixing and Self-Instruct or Evol-Instruct data construction options.
• Tokenizer vocabulary merging for extending vocabularies.
• Evaluation across 59+ datasets and benchmarks, including MMLU, GSM8K, and HellaSwag.
• Evaluation methods for generation, likelihood-based ranking, and option probabilities.
• In-context learning and chain-of-thought strategies for supported datasets.
• Inference acceleration options include prefix caching and vLLM; BitsAndBytes and GPTQ quantization are also supported.

Use Cases

• Researchers can run benchmark evaluations across supported datasets and compare evaluation methods.
• Model developers can fine-tune supported LLMs with supervised, preference-based, or continued pre-training workflows.
• Teams preparing instruction data can combine datasets or apply Self-Instruct and Evol-Instruct processing.
• Developers testing local or API-based models can use a shared evaluation interface rather than build each benchmark workflow from scratch.

Getting Started

Clone the repository and install its dependencies:

git clone https://github.com/RUCAIBox/LLMBox.git && cd LLMBox
pip install -r requirements.txt

For an OpenAI-compatible evaluation-only setup, the README points to requirements-openai.txt. Consult the README and documentation for model-specific setup and usage.

Alternatives

• torchtune: torchtune focuses on editable PyTorch recipes for LLM post-training, while LLMBox combines training with broader model and benchmark evaluation.
• DataDreamer: DataDreamer emphasizes reproducible workflows and synthetic data generation alongside training and alignment; LLMBox centers on configurable training and evaluation.
• lighteval: Lighteval focuses on evaluating models across tasks and inference backends, rather than providing LLMBox’s combined training and evaluation pipeline.

Considerations

Training local models can require substantial compute and model-specific dependencies. The README documents integrations such as DeepSpeed, Flash Attention, vLLM, and quantization, which may require additional setup. Model and dataset support is broad but not universal, and evaluation methods and strategies vary by dataset. Check the project documentation for compatibility and configuration details before planning a workflow.