llmgym: Build and Benchmark LLM Applications That Learn

Overview

LLMGym is a Python interface for running LLM agents in interactive environments and comparing how they perform while learning from feedback. It addresses the difficulty of evaluating agents consistently as the range of benchmarks grows, by making multiple task environments available through a common interface.

Its environments span instruction following, question answering, customer service, and terminal tasks. It is most useful when building or researching agents that take actions over multiple steps, rather than testing a model with isolated prompts.

Key Features

• A shared environment API for resetting tasks, submitting actions, and observing episode completion.
• BabyAI grid-world tasks adapted for text-based instruction following.
• Multi-hop question answering with iterative search and note-taking.
• Customer-service scenarios through the Tau Bench adapter.
• Containerized task support through Harbor and Docker-based Terminal Bench environments.
• Additional tasks for named entity recognition and Twenty-One Questions.
• Examples covering concurrent episodes and supervised fine-tuning.

Use Cases

• LLM researchers can compare agent behavior across different task types using a consistent interaction pattern.
• Agent developers can prototype multi-step tool use, such as search, note-taking, or terminal operations.
• Teams exploring customer-service agents can run tasks in airline and retail domains through the Tau Bench environment.
• Educators and students can use smaller interactive tasks, such as BabyAI or Twenty-One Questions, to study agent behavior.

Getting Started

The README specifies Python >=3.12 and <3.14, plus uv. Clone the repository and install its dependencies:

git clone https://github.com/tensorzero/llmgym.git
cd llmgym
uv venv
source .venv/bin/activate
uv sync

See the README for API key setup, notebooks, and further installation details.

Alternatives

• Verifiers: Verifiers builds custom dataset-and-reward environments for LLM training and evaluation, while LLMGym focuses on interactive tasks for applications learning from feedback.

Considerations

• The repository README warns that the project is under active development and may introduce breaking changes.
• The repository metadata marks it as archived, so verify its current maintenance status before adopting it.
• The documented setup requires Python 3.12 or 3.13 and uv.
• Using the OpenAI agent example requires an OPENAI_API_KEY; some environments also depend on external projects or containerized tasks.