LLMGym: A Unified Environment for LLM Agent Development and Benchmarking

Introduction

LLMGym is a unified environment interface designed for developing and benchmarking LLM applications that learn from feedback. Drawing inspiration from the popular Gymnasium library, LLMGym aims to be the "gym" for LLM agents. As the landscape of LLM benchmarks rapidly expands, LLMGym provides an intuitive interface for a suite of environments that can be seamlessly swapped out for research and development purposes, facilitating fair and comprehensive comparisons.

Important Note: This repository is still under active development. Expect breaking changes.

LLMGym includes a diverse set of environments:

• BabyAI: Text-based versions of BabyAI grid world environments for instruction following.
• Harbor: An adapter for Harbor tasks, allowing you to run any containerized task as an LLMGym environment.
• Multi-Hop: Multi-hop question answering with iterative search and note-taking.
• NER: Named Entity Recognition tasks.
• Tau Bench: Customer service environments for airline and retail domains.
• Terminal Bench: Docker-based terminal environments for solving programming and system administration tasks.
• Twenty-One Questions: The classic guessing game where agents ask yes/no questions to identify a secret.

Installation

To get started with LLMGym, follow these installation steps:

Prerequisites

• Install Python >=3.12, <3.14.
• Install uv.

Setup LLMGym

git clone git@github.com:tensorzero/llmgym.git
cd llmgym
uv venv
source .venv/bin/activate
uv sync

Verify the Installation

python -c "import llmgym; print(llmgym.__version__)"

Setting Environment Variables

To set the OPENAI_API_KEY environment variable, run the following command:

export OPENAI_API_KEY="your_openai_api_key"

It is recommended to use direnv and create a local .envrc file to manage environment variables. For example, your .envrc file might look like this:

export OPENAI_API_KEY="your_openai_api_key"

Then, run direnv allow to load the environment variables.

Examples

Here's a quickstart example demonstrating how to use LLMGym:

import llmgym
from llmgym.logs import get_logger
from llmgym.agents import OpenAIAgent

env  = llmgym.make("21_questions_v0")

agent = llmgym.agents.OpenAIAgent(
    model_name="gpt-4o-mini",
    function_configs=env.functions,
    tool_configs=env.tools,
)
# Get default horizon
max_steps = env.horizon

# Reset the environment
reset_data = await env.reset()
obs = reset_data.observation

# Run the episode
for _step in range(max_steps):
    # Get action from agent
    action = await agent.act(obs)

    # Step the environment
    step_data = await env.step(action)
    obs = step_data.observation

    # Check if the episode is done
    done = step_data.terminated or step_data.truncated
    if done:
        break
await env.close()

You can find more examples and tutorials in the project's notebooks:

• Quickstart Notebook
• Tutorial Notebook
• Tau Bench Example
• 21 Questions Example
• Supervised Finetuning Example

Why Use LLMGym?

LLMGym offers a powerful and flexible solution for anyone working with LLM agents that learn from feedback. Its key advantages include:

• Unified Interface: Provides a consistent API for interacting with various LLM environments, simplifying development.
• Benchmarking: Designed to make fair and comprehensive comparisons across different LLM applications and agents easier.
• Diverse Environments: Comes with a rich set of pre-built environments, covering tasks from instruction following to programming and customer service.
• Accelerated Development: Streamlines the process of building, testing, and iterating on LLM agents.
• Research Ready: Offers a robust platform for academic and industrial research into LLM agent behavior and learning.

Links

• GitHub Repository: tensorzero/llmgym
• Gymnasium (Inspiration): https://gymnasium.farama.org/
• uv Installation: https://docs.astral.sh/uv/getting-started/installation/
• direnv: https://direnv.net/