Paper2Code: Generate Code Repositories from ML Papers

Overview

Paper2Code introduces PaperCoder, a multi-agent system that converts scientific machine-learning papers into code repositories. Specialized agents handle planning, analysis, and code generation, addressing the gap between a paper's description of a method and an implementation that can be inspected or evaluated.

The repository includes the PaperCoder workflow, benchmark datasets, and scripts for evaluating generated repositories. It is aimed at research and experimentation rather than serving as a general-purpose, ready-to-deploy code generator.

Key Features

• Generates repository implementations from paper inputs through a planning, analysis, and code-generation pipeline.
• Supports papers supplied as structured JSON converted from PDF or as LaTeX source.
• Offers an OpenAI API path and a vLLM path for open-source models.
• Includes the Paper2Code benchmark dataset, also available on Hugging Face.
• Evaluates generated repositories in reference-based and reference-free modes.
• Produces intermediate planning, analysis, and coding artifacts alongside the final repository.

Use Cases

• ML researchers can create a starting implementation from a method paper when no author-released code is available.
• Reproducibility teams can compare a generated implementation with an official repository using reference-based evaluation.
• Developers investigating paper-to-code workflows can run the example or adapt the scripts to test other papers.
• Benchmark researchers can use the included dataset and evaluation workflow to study generated implementation quality.

Getting Started

For the example using the OpenAI API, install the client, set your API key, and run the provided script:

pip install openai
export OPENAI_API_KEY="<OPENAI_API_KEY>"
cd scripts
bash run.sh

The example runs on the Attention Is All You Need paper. See the README for vLLM setup, LaTeX inputs, PDF conversion, and evaluation instructions.

Considerations

• Running the OpenAI-backed example requires an API key and incurs usage costs. The README estimates $0.50–$0.70 for the example with o3-mini.
• The vLLM option requires installing and configuring vLLM; the default model is DeepSeek-Coder-V2-Lite-Instruct.
• PDF input requires an additional conversion workflow using s2orc-doc2json and its PDF processing service. LaTeX source can skip that step.
• Generated repositories should be treated as outputs to inspect and evaluate, not assumed to be faithful implementations. The project provides evaluation scripts, but their model-based evaluation uses OpenAI credentials.
• The README describes a research system and example workflow. Adapting it to other papers requires changing paths and environment variables.