autoresearch: Let AI Agents Run LLM Training Experiments

Overview

autoresearch is a compact research loop for experimenting with language-model training. An AI agent changes the training code, runs a fixed five-minute experiment, and compares validation bits per byte to decide whether a change is worth keeping.

It is aimed at people who want to delegate iterative training experiments rather than edit code by hand. The setup is intentionally narrow: one NVIDIA GPU, one training file for the agent to modify, and a human-maintained instruction file that guides its work.

Key Features

• Lets an AI coding agent modify the model, optimizer, training loop, and hyperparameters in train.py.
• Uses a fixed five-minute training budget to make experiments comparable on the same hardware.
• Evaluates experiments with validation bits per byte, where lower is better.
• Keeps data preparation and runtime utilities separate from the agent-edited training code.
• Provides program.md as a lightweight place to define the agent's research instructions.
• Includes a manual training run so users can check their setup before starting agent-driven experiments.

Use Cases

• A researcher with a single NVIDIA GPU can delegate overnight exploration of model and training changes.
• A developer can test whether an AI coding agent can make measurable improvements under a consistent time budget.
• An educator or learner can inspect a small training setup and observe how model and hyperparameter changes affect validation results.
• Users with smaller or non-NVIDIA hardware can use the project as a starting point for a fork, adapting data and model settings to their device.

Getting Started

The documented setup uses Python 3.10+, uv, and a single NVIDIA GPU. After installing dependencies and preparing the data, run a manual experiment:

uv sync
uv run prepare.py
uv run train.py

See the README for agent setup and configuration details.

Considerations

• The project currently requires a single NVIDIA GPU; the README notes it was tested on an H100. Other hardware is not supported by the main setup.
• Each experiment has a fixed five-minute training budget, but results are not directly comparable across different hardware platforms.
• Agent-driven changes can affect training behavior. Review the agent's instructions and changes, and take care when granting an agent repository permissions.
• The README describes the default program.md as a bare-bones baseline, so users should expect to refine the research instructions for their goals.