AutoResearch: AI/ML Research Agents from Idea to Paper-Ready Evidence

Introduction

AutoResearch, an open-source project by EvoMap, is an advanced agent workflow designed to streamline AI and machine learning research. It guides the entire research process, from initial idea generation and experiment planning to implementation, execution, result analysis, and independent evaluation. The goal is to produce a comprehensive evidence package ready for paper writing, significantly reducing manual effort and enhancing research quality.

This system addresses common challenges in AI/ML research by grounding problem discovery in real signals, incorporating domain knowledge, and validating claims through cross-model review, source records, experiment logs, critic reports, and blind reviews. It aims to minimize unsupported generation, unclear provenance, inflated self-evaluation, and overinterpretation of results, while preserving all necessary evidence and state for researcher review.

Why Use It and Key Benefits

AutoResearch offers several core capabilities that make it a powerful tool for researchers:

Cross-domain Idea Generation: Discover new problems by combining recent external signals (papers, communities, open-source trends) with your local domain knowledge and constraints.
Independent Multi-Model Review: Utilize at least three distinct models during idea generation to ensure independent development and cross-review of candidate ideas, enhancing objectivity.
Stateful Experiment Execution: Plans, code, queues, logs, and conclusions are persistently stored, allowing long-running experiments to resume seamlessly after interruptions.
Pilot Before Scaling: Test the feasibility of experiments at a lower cost before committing to full-scale execution, enabling early stopping or revision.
Traceable Evidence and Sources: All research artifacts, including Forge sources, knowledge directions, experiment results, critic reports, and blind reviews, are meticulously recorded for full transparency.
Support for Negative Results: The system preserves evidence and allows for early termination when a hypothesis fails, preventing the forced narrative of success in every experiment.

Installation

To get started with AutoResearch, follow these steps to set up your environment:

Clone the Repository and Check the Environment:
Prepare a Linux or SSH machine with Git, Python 3.10+, and python3-venv installed.
```
git clone https://github.com/EvoMap/AutoResearch.git
cd AutoResearch
bash scripts/bringup.sh
```
bringup.sh sets up the Python environment, installs dependencies, runs baseline tests, and checks model configuration without contacting model services.
Configure Model Services:
Create local configuration files for API keys and provider settings.
```
test -f .env || cp .env.example .env
test -f config/providers.local.json || \
  cp config/providers.example.json config/providers.local.json
```
Edit .env for API URLs/keys and config/providers.local.json for endpoints and model assignments. AutoResearch supports various combinations of Gemini, GPT, or Claude models.
Test the APIs:
Verify your model configurations by sending a small number of real requests.
```
set -a
. ./.env
set +a
.venv/bin/python scripts/preflight.py --live
```
An exit code 0 indicates that normal roles have usable models and multi-model stages meet independence requirements.

Examples

AutoResearch supports two main workflows: Idea Generation and Idea Execution.

Idea Generation: Cross-Domain Discovery

This workflow helps you discover and plan new research directions.

Run the Pipeline:
Execute the default entrypoint to collect signals, filter, intersect with local knowledge, generate and review ideas, and plan experiments.
```
.venv/bin/python idea_generation.py
```
Main outputs are found in data/candidates/, data/verified/, data/idea_forge/, and logs/.

Select and Export an Idea:
List executable plans from a forge file and then export a selected plan for execution.

.venv/bin/python src/idea_provenance.py list \
  --forge-file data/idea_forge/forge_YYYYMMDD_HHMM.json

.venv/bin/python src/idea_provenance.py export \
  --forge-file data/idea_forge/forge_YYYYMMDD_HHMM.json \
  --result-index 1 \
  --plan-index 1 \
  --output data/ideas/my_experiment.txt

Idea Execution

If you already have a research idea, you can directly proceed with execution.

Prepare the Execution Environment:
Install Bun 1.3+, Node.js, Conda (or another Python environment manager), and the Ralph Loop plugin for Claude Code.
```
cd ar-runtime
bun install --frozen-lockfile
cd ..
```

Generate the Execution Configuration:
Create machine-local settings and project the unified provider configuration.

test -f ar-runtime/.claude/settings.local.json || \
  cp ar-runtime/.claude/settings.local.example.json \
     ar-runtime/.claude/settings.local.json

set -a
. ./.env
set +a
.venv/bin/python scripts/render_env.py
.venv/bin/python scripts/preflight.py --live --tools

Start the Coordinator:
Run the Claude Code CLI in an isolated environment and start the coordinator with your idea.

cd ar-runtime
claude --dangerously-skip-permissions
# Inside Claude Code:
# /ar-coordinator ../data/ideas/my_experiment.txt ../data/projects/my_experiment

For non-interactive runs, use the supervisor script:

cd ar-runtime
scripts/ar-supervisor.sh \
  ../data/ideas/my_experiment.txt \
  ../data/projects/my_experiment

Inspect Project Results:
Generate a dashboard to visualize the progress and results of your project.
```
.venv/bin/python src/generate_project_dashboard.py my_experiment
```