# LLMGym: A Unified Environment for LLM Agent Development and Benchmarking

This repository profile is provided by osrepos.com, an open source repository discovery platform.

Source: osrepos.com
Repository profile: https://osrepos.com/repo/tensorzero-llmgym
Generated for open source discovery and AI-assisted research.

LLMGym is a unified environment interface designed for developing and benchmarking LLM applications that learn from feedback. It provides a suite of seamlessly swappable environments, making fair and comprehensive comparisons easier for researchers and developers. This project aims to be the "gym" for LLM agents, offering an intuitive interface for various tasks.

GitHub: https://github.com/tensorzero/llmgym
OSRepos URL: https://osrepos.com/repo/tensorzero-llmgym

## Summary

LLMGym is a unified environment interface designed for developing and benchmarking LLM applications that learn from feedback. It provides a suite of seamlessly swappable environments, making fair and comprehensive comparisons easier for researchers and developers. This project aims to be the "gym" for LLM agents, offering an intuitive interface for various tasks.

## Topics

- Python
- LLM
- AI Agents
- Benchmarking
- Machine Learning
- Reinforcement Learning
- Environment Interface

## Repository Information

Last analyzed by OSRepos: Tue May 12 2026 13:29:35 GMT+0100 (Western European Summer Time)
Detail views: 3
GitHub clicks: 13

## Safety Notice

OSRepos shares public repositories for knowledge and discovery only. Review source code, dependencies, licenses, and security implications before running or installing anything.

## Content

## Introduction

LLMGym is a unified environment interface designed for developing and benchmarking LLM applications that learn from feedback. Drawing inspiration from the popular [Gymnasium](https://gymnasium.farama.org/) library, LLMGym aims to be the "gym" for LLM agents. As the landscape of LLM benchmarks rapidly expands, LLMGym provides an intuitive interface for a suite of environments that can be seamlessly swapped out for research and development purposes, facilitating fair and comprehensive comparisons.

**Important Note:** This repository is still under active development. Expect breaking changes.

LLMGym includes a diverse set of environments:

*   **[BabyAI](https://github.com/tensorzero/llmgym/tree/main/llmgym/envs/babyai/)**: Text-based versions of BabyAI grid world environments for instruction following.
*   **[Harbor](https://github.com/tensorzero/llmgym/tree/main/llmgym/envs/harbor/)**: An adapter for Harbor tasks, allowing you to run any containerized task as an LLMGym environment.
*   **[Multi-Hop](https://github.com/tensorzero/llmgym/tree/main/llmgym/envs/multi_hop/)**: Multi-hop question answering with iterative search and note-taking.
*   **[NER](https://github.com/tensorzero/llmgym/tree/main/llmgym/envs/ner/)**: Named Entity Recognition tasks.
*   **[Tau Bench](https://github.com/tensorzero/llmgym/tree/main/llmgym/envs/tau_bench/)**: Customer service environments for airline and retail domains.
*   **[Terminal Bench](https://github.com/tensorzero/llmgym/tree/main/llmgym/envs/terminal_bench/)**: Docker-based terminal environments for solving programming and system administration tasks.
*   **[Twenty-One Questions](https://github.com/tensorzero/llmgym/tree/main/llmgym/envs/twenty_one_questions/)**: The classic guessing game where agents ask yes/no questions to identify a secret.

## Installation

To get started with LLMGym, follow these installation steps:

### Prerequisites

*   Install Python >=3.12, <3.14.
*   Install [uv](https://docs.astral.sh/uv/getting-started/installation/).

### Setup LLMGym

bash
git clone git@github.com:tensorzero/llmgym.git
cd llmgym
uv venv
source .venv/bin/activate
uv sync


### Verify the Installation

bash
python -c "import llmgym; print(llmgym.__version__)"


### Setting Environment Variables

To set the `OPENAI_API_KEY` environment variable, run the following command:

bash
export OPENAI_API_KEY="your_openai_api_key"


It is recommended to use [direnv](https://direnv.net/) and create a local `.envrc` file to manage environment variables. For example, your `.envrc` file might look like this:

bash
export OPENAI_API_KEY="your_openai_api_key"


Then, run `direnv allow` to load the environment variables.

## Examples

Here's a quickstart example demonstrating how to use LLMGym:

python
import llmgym
from llmgym.logs import get_logger
from llmgym.agents import OpenAIAgent

env  = llmgym.make("21_questions_v0")

agent = llmgym.agents.OpenAIAgent(
    model_name="gpt-4o-mini",
    function_configs=env.functions,
    tool_configs=env.tools,
)
# Get default horizon
max_steps = env.horizon

# Reset the environment
reset_data = await env.reset()
obs = reset_data.observation

# Run the episode
for _step in range(max_steps):
    # Get action from agent
    action = await agent.act(obs)

    # Step the environment
    step_data = await env.step(action)
    obs = step_data.observation

    # Check if the episode is done
    done = step_data.terminated or step_data.truncated
    if done:
        break
await env.close()


You can find more examples and tutorials in the project's notebooks:

*   [Quickstart Notebook](https://github.com/tensorzero/llmgym/blob/main/examples/quickstart.ipynb)
*   [Tutorial Notebook](https://github.com/tensorzero/llmgym/blob/main/examples/tutorial.ipynb)
*   [Tau Bench Example](https://github.com/tensorzero/llmgym/blob/main/examples/tau_bench.ipynb)
*   [21 Questions Example](https://github.com/tensorzero/llmgym/blob/main/examples/21_questions.ipynb)
*   [Supervised Finetuning Example](https://github.com/tensorzero/llmgym/blob/main/examples/supervised_fine_tuning.ipynb)

## Why Use LLMGym?

LLMGym offers a powerful and flexible solution for anyone working with LLM agents that learn from feedback. Its key advantages include:

*   **Unified Interface**: Provides a consistent API for interacting with various LLM environments, simplifying development.
*   **Benchmarking**: Designed to make fair and comprehensive comparisons across different LLM applications and agents easier.
*   **Diverse Environments**: Comes with a rich set of pre-built environments, covering tasks from instruction following to programming and customer service.
*   **Accelerated Development**: Streamlines the process of building, testing, and iterating on LLM agents.
*   **Research Ready**: Offers a robust platform for academic and industrial research into LLM agent behavior and learning.

## Links

*   **GitHub Repository**: [tensorzero/llmgym](https://github.com/tensorzero/llmgym)
*   **Gymnasium (Inspiration)**: [https://gymnasium.farama.org/](https://gymnasium.farama.org/)
*   **uv Installation**: [https://docs.astral.sh/uv/getting-started/installation/](https://docs.astral.sh/uv/getting-started/installation/)
*   **direnv**: [https://direnv.net/](https://direnv.net/)