EvalPlus

Spec sheet

Purpose

EvalPlus is a framework for rigorously evaluating large language models on code related tasks.Read 11 Oct 2026 · github.com

Security

The README describes Docker-based code execution as safe code execution.Read 5 Oct 2026 · github.com

License

The repository provides the software under the Apache License 2.0, with files under evalplus/eval additionally complying with the MIT License.Read 11 Oct 2026 · github.com

Benchmarks

The framework supports HumanEval, MBPP, and EvalPerf evaluation.Read 11 Oct 2026 · github.com

Docker

The README documents running code evaluation in Docker.Read 5 Oct 2026 · github.com

Efficiency evaluation

EvalPerf evaluates the efficiency of LLM-generated code using performance-exercising tasks and test inputs.Read 11 Oct 2026 · github.com

EvalPerf leaderboard

The EvalPlus site provides a leaderboard for EvalPerf code efficiency results.Read 5 Oct 2026 · evalplus.github.io

Execution

The README provides Docker-based examples for running code evaluation in a container.Read 11 Oct 2026 · github.com

HumanEval+

HumanEval+ expands the original HumanEval tests by 80 times.Read 11 Oct 2026 · evalplus.github.io

Installation

EvalPlus can be installed from its GitHub repository or as a stable release using pip.Read 11 Oct 2026 · github.com

Intended audience

The project is intended for evaluating LLM performance on code-related tasks.Read 5 Oct 2026 · evalplus.github.io

Leaderboard

The project links leaderboards for EvalPlus and EvalPerf benchmarks.Read 11 Oct 2026 · evalplus.github.io

MBPP+

MBPP+ expands the original MBPP tests by 35 times.Read 11 Oct 2026 · evalplus.github.io

Model backends

Documented backends include Hugging Face, vLLM, OpenAI-compatible servers, OpenAI, Anthropic, Google Gemini, Amazon Bedrock, Ollama, and Intel Gaudi.Read 11 Oct 2026 · github.com

Operating system limit

The README labels EvalPerf as *nix only.Read 11 Oct 2026 · github.com

Platform limit

The README labels EvalPerf code efficiency evaluation as *nix only.Read 5 Oct 2026 · github.com

RepoQA

The EvalPlus site describes RepoQA as work on evaluators for long-context code understanding.Read 11 Oct 2026 · evalplus.github.io

Use by model teams

The README says EvalPlus datasets have been used by teams including Meta Llama, Allen AI TÜLU, Qwen, DeepSeek, Snowflake, StarCoder2, Magicoder, and WizardCoder.Read 11 Oct 2026 · github.com

Where Specifiction places it

• LLM Evaluation Tools#20 of 297.3