{"name":"LLMGym: A Unified Environment for LLM Agent Development and Benchmarking","description":"LLMGym is a unified environment interface designed for developing and benchmarking LLM applications that learn from feedback. It provides a suite of seamlessly swappable environments, making fair and comprehensive comparisons easier for researchers and developers. This project aims to be the \"gym\" for LLM agents, offering an intuitive interface for various tasks.","github":"https://github.com/tensorzero/llmgym","url":"https://osrepos.com/repo/tensorzero-llmgym","source":"osrepos.com","sourceDescription":"This repository profile is provided by osrepos.com, an open source repository discovery platform.","repositoryProfile":"https://osrepos.com/repo/tensorzero-llmgym","generatedFor":"open source discovery and AI-assisted research","markdown":"https://osrepos.com/repo/tensorzero-llmgym.md","json":"https://osrepos.com/repo/tensorzero-llmgym.json","topics":["Python","LLM","AI Agents","Benchmarking","Machine Learning","Reinforcement Learning","Environment Interface"],"keywords":["Python","LLM","AI Agents","Benchmarking","Machine Learning","Reinforcement Learning","Environment Interface"],"stars":null,"summary":"LLMGym is a unified environment interface designed for developing and benchmarking LLM applications that learn from feedback. It provides a suite of seamlessly swappable environments, making fair and comprehensive comparisons easier for researchers and developers. This project aims to be the \"gym\" for LLM agents, offering an intuitive interface for various tasks.","content":"## Introduction\n\nLLMGym is a unified environment interface designed for developing and benchmarking LLM applications that learn from feedback. Drawing inspiration from the popular [Gymnasium](https://gymnasium.farama.org/) library, LLMGym aims to be the \"gym\" for LLM agents. As the landscape of LLM benchmarks rapidly expands, LLMGym provides an intuitive interface for a suite of environments that can be seamlessly swapped out for research and development purposes, facilitating fair and comprehensive comparisons.\n\n**Important Note:** This repository is still under active development. Expect breaking changes.\n\nLLMGym includes a diverse set of environments:\n\n*   **[BabyAI](https://github.com/tensorzero/llmgym/tree/main/llmgym/envs/babyai/)**: Text-based versions of BabyAI grid world environments for instruction following.\n*   **[Harbor](https://github.com/tensorzero/llmgym/tree/main/llmgym/envs/harbor/)**: An adapter for Harbor tasks, allowing you to run any containerized task as an LLMGym environment.\n*   **[Multi-Hop](https://github.com/tensorzero/llmgym/tree/main/llmgym/envs/multi_hop/)**: Multi-hop question answering with iterative search and note-taking.\n*   **[NER](https://github.com/tensorzero/llmgym/tree/main/llmgym/envs/ner/)**: Named Entity Recognition tasks.\n*   **[Tau Bench](https://github.com/tensorzero/llmgym/tree/main/llmgym/envs/tau_bench/)**: Customer service environments for airline and retail domains.\n*   **[Terminal Bench](https://github.com/tensorzero/llmgym/tree/main/llmgym/envs/terminal_bench/)**: Docker-based terminal environments for solving programming and system administration tasks.\n*   **[Twenty-One Questions](https://github.com/tensorzero/llmgym/tree/main/llmgym/envs/twenty_one_questions/)**: The classic guessing game where agents ask yes/no questions to identify a secret.\n\n## Installation\n\nTo get started with LLMGym, follow these installation steps:\n\n### Prerequisites\n\n*   Install Python >=3.12, <3.14.\n*   Install [uv](https://docs.astral.sh/uv/getting-started/installation/).\n\n### Setup LLMGym\n\nbash\ngit clone git@github.com:tensorzero/llmgym.git\ncd llmgym\nuv venv\nsource .venv/bin/activate\nuv sync\n\n\n### Verify the Installation\n\nbash\npython -c \"import llmgym; print(llmgym.__version__)\"\n\n\n### Setting Environment Variables\n\nTo set the `OPENAI_API_KEY` environment variable, run the following command:\n\nbash\nexport OPENAI_API_KEY=\"your_openai_api_key\"\n\n\nIt is recommended to use [direnv](https://direnv.net/) and create a local `.envrc` file to manage environment variables. For example, your `.envrc` file might look like this:\n\nbash\nexport OPENAI_API_KEY=\"your_openai_api_key\"\n\n\nThen, run `direnv allow` to load the environment variables.\n\n## Examples\n\nHere's a quickstart example demonstrating how to use LLMGym:\n\npython\nimport llmgym\nfrom llmgym.logs import get_logger\nfrom llmgym.agents import OpenAIAgent\n\nenv  = llmgym.make(\"21_questions_v0\")\n\nagent = llmgym.agents.OpenAIAgent(\n    model_name=\"gpt-4o-mini\",\n    function_configs=env.functions,\n    tool_configs=env.tools,\n)\n# Get default horizon\nmax_steps = env.horizon\n\n# Reset the environment\nreset_data = await env.reset()\nobs = reset_data.observation\n\n# Run the episode\nfor _step in range(max_steps):\n    # Get action from agent\n    action = await agent.act(obs)\n\n    # Step the environment\n    step_data = await env.step(action)\n    obs = step_data.observation\n\n    # Check if the episode is done\n    done = step_data.terminated or step_data.truncated\n    if done:\n        break\nawait env.close()\n\n\nYou can find more examples and tutorials in the project's notebooks:\n\n*   [Quickstart Notebook](https://github.com/tensorzero/llmgym/blob/main/examples/quickstart.ipynb)\n*   [Tutorial Notebook](https://github.com/tensorzero/llmgym/blob/main/examples/tutorial.ipynb)\n*   [Tau Bench Example](https://github.com/tensorzero/llmgym/blob/main/examples/tau_bench.ipynb)\n*   [21 Questions Example](https://github.com/tensorzero/llmgym/blob/main/examples/21_questions.ipynb)\n*   [Supervised Finetuning Example](https://github.com/tensorzero/llmgym/blob/main/examples/supervised_fine_tuning.ipynb)\n\n## Why Use LLMGym?\n\nLLMGym offers a powerful and flexible solution for anyone working with LLM agents that learn from feedback. Its key advantages include:\n\n*   **Unified Interface**: Provides a consistent API for interacting with various LLM environments, simplifying development.\n*   **Benchmarking**: Designed to make fair and comprehensive comparisons across different LLM applications and agents easier.\n*   **Diverse Environments**: Comes with a rich set of pre-built environments, covering tasks from instruction following to programming and customer service.\n*   **Accelerated Development**: Streamlines the process of building, testing, and iterating on LLM agents.\n*   **Research Ready**: Offers a robust platform for academic and industrial research into LLM agent behavior and learning.\n\n## Links\n\n*   **GitHub Repository**: [tensorzero/llmgym](https://github.com/tensorzero/llmgym)\n*   **Gymnasium (Inspiration)**: [https://gymnasium.farama.org/](https://gymnasium.farama.org/)\n*   **uv Installation**: [https://docs.astral.sh/uv/getting-started/installation/](https://docs.astral.sh/uv/getting-started/installation/)\n*   **direnv**: [https://direnv.net/](https://direnv.net/)","metrics":{"detailViews":3,"githubClicks":13},"dates":{"published":null,"modified":"2026-05-12T12:29:35.000Z"}}