huggingface

lighteval: Evaluate Language Models Across Backends

Lighteval is a Python toolkit for running LLM evaluations across local models and remote inference backends. It combines a broad task catalog with custom metrics and detailed sample-level results for teams comparing or debugging model performance.

View on GitHub Compare
Stars
2.5k
Forks
566
Stars, last 30 days
+25

Our take

Health score

55/100

How it is scored
Activity
50
Community
90
Issues
32
Pull requests
46

+25 stars in the last 30 days

  • More contributors than 78% of the projects we track
  • Slower to close issues than 93% of the projects we track

Project history

Daily GitHub snapshots since Oct 7, 2026. History covers stars and forks; health signals are not tracked historically.

Selected snapshot

Stars

2,586

Forks

613

Change since Nov 14, 2026: 0 stars·-1 forks

Overview

Lighteval helps researchers and developers evaluate language models using a shared toolkit instead of building separate evaluation workflows for each model-serving setup. It supports models already loaded in memory as well as models accessed through supported inference backends.

The project is useful when teams need to compare model behavior across established tasks, inspect individual responses to debug results, or extend evaluations with custom tasks and metrics. Its task catalog covers areas including knowledge, math, coding, multilingual evaluation, and language understanding.

Key Features

  • Runs evaluations through multiple backends, including Accelerate, Nanotron, vLLM, SGLang, and inference endpoints.
  • Supports models already loaded in memory through a Python API.
  • Includes more than 1,000 evaluation tasks, spanning multiple domains and languages.
  • Provides sample-by-sample results for examining model outputs and diagnosing failures.
  • Lets users add custom evaluation tasks and metrics.
  • Offers command-line entry points for remote services and local or distributed model evaluation.

Use Cases

  • Model developers can compare candidate models on established benchmarks before choosing one for further work.
  • Evaluation teams can run the same task suite against models served by different supported backends.
  • Researchers can inspect individual answers to understand unexpected scores or model errors.
  • Practitioners working in multilingual settings can select tasks for languages and regions represented in the catalog.
  • Teams with specialized evaluation needs can define custom tasks or metrics rather than relying only on built-in options.

Getting Started

Install from PyPI:

pip install lighteval

See the repository README and documentation for backend setup and evaluation examples.

Alternatives

  • Verifiers: Verifiers evaluates models through configurable environments and reward functions, with a focus on capability testing and reinforcement-learning workflows.
  • Ragas: Ragas evaluates LLM applications with application-focused metrics and test generation, rather than emphasizing broad model benchmarks.
  • Weave by Weights & Biases: Weave centers on tracing, debugging, and evaluating LLM application inputs and outputs, rather than running a broad catalog of model benchmarks.
ProjectLanguageLicenseStarsStatus
lightevalPythonMIT2.5kNot checked yet
verifiersPythonMIT4.7kActive
ragasPythonApache-2.015.9kQuiet
weavePythonApache-2.01.1kActive

Considerations

  • The README says Windows is currently untested and unsupported; Mac and Linux are expected to work.
  • Installation extras are available, and the appropriate dependencies vary with the backend and use case.
  • GPU, distributed, and endpoint evaluation require setup for the selected backend or service.
  • The repository lists 429 open issues, so check current issue discussions and documentation when assessing a particular workflow.

Found this useful?

Share it with someone who would like lighteval.

Comparisons