evalplus: Rigorously Evaluate LLM-Generated Code

Overview

EvalPlus is a Python framework for assessing code generated by large language models. It addresses a weakness of standard code-generation benchmarks: a small set of tests may accept solutions that fail on less obvious inputs. HumanEval+ and MBPP+ expand the tests for their respective benchmarks, while EvalPerf evaluates code efficiency as well as correctness.

Use it when you need repeatable benchmark evaluation across local models or supported API backends, rather than relying on model-generated examples or the original benchmark tests alone. The project provides command-line evaluation and generation workflows, datasets, and leaderboard results.

Key Features

• Expanded correctness tests for HumanEval and MBPP through HumanEval+ and MBPP+.
• EvalPerf benchmark for measuring the efficiency of generated code.
• Command-line workflows for generating samples and evaluating them.
• Supports local inference backends including Hugging Face Transformers and vLLM.
• Supports hosted and compatible API backends, including OpenAI, Anthropic, Google, Amazon Bedrock, and Ollama.
• Offers Docker-based execution for running generated code in a container.
• Publishes leaderboards and research papers for its evaluation benchmarks.

Use Cases

• Model researchers can compare code-generation systems on more demanding correctness tests than the original HumanEval or MBPP tests.
• LLM developers can check whether generated solutions are fragile on additional inputs before using benchmark results to guide model improvements.
• Inference teams can evaluate local or API-served models through supported backends using a consistent command-line workflow.
• Performance-focused teams can use EvalPerf when they need to assess generated code efficiency alongside correctness.

Getting Started

Install the package with a vLLM extra and run an evaluation:

pip install --upgrade "evalplus[vllm]"
evalplus.evaluate --model "ise-uiuc/Magicoder-S-DS-6.7B" --dataset humaneval --backend vllm --greedy

See the repository README for backend setup, dataset options, Docker execution, and EvalPerf instructions.

Alternatives

• Promptfoo: Promptfoo evaluates and red-teams prompts and LLM applications, while EvalPlus benchmarks generated code with expanded HumanEval and MBPP correctness tests.

Considerations

• Running local models can require substantial compute and model-specific setup. The README also documents API backends for hosted inference.
• Generated programs are executed during correctness evaluation. Docker execution is provided as an isolation option, but users should review the execution guidance before evaluating untrusted code.
• EvalPerf is limited to *nix systems and requires enabling performance-counter access; its example setup uses elevated privileges.
• Results depend on the selected dataset, backend, prompting mode, and evaluation settings. Compare runs with consistent configurations when interpreting model differences.