Skip to content
SPECIFICTION
Get listed

SWE-bench

swebench.com

WebWindowsmacOSLinuxAPISelf-hosted
Visit swebench.com
Spec score 7.0 #25 of 29 in LLM Evaluation Tools

What it is

SWE-bench evaluates language models on software issues collected from GitHub. A model generates a patch, which the evaluation harness applies in a Docker environment before running the repository tests to check whether the issue is resolved. The original benchmark contains 2,294 task instances from 12 Python repositories. Other listed datasets include SWE-bench Lite, Verified, Multimodal and Multilingual. Verified contains 500 instances engineers checked as solvable; Multilingual has 300 instances across nine languages and 42 repositories. Multimodal includes 100 development instances with screenshots and interface elements, while test evaluation uses the SWE-bench API. Its CLI can submit predictions, retrieve reports, manage evaluation runs and track performance in the cloud. Evaluations can run locally or in the cloud using Modal and sb-cli. Installation requires Python 3.9 or later and Docker. Guides cover macOS, Linux and Windows, with WSL 2 required on Windows. The Docker guide recommends at least 120 GB of free disk space and 16 GB of RAM; caching all Docker images can require about 2,000 GB. SWE-bench is free.

Spec sheet

Purpose
SWE-bench evaluates language models on real-world GitHub software issues by asking them to generate patches that resolve the issues.Read 9 Oct 2026 · swebench.com
Integrations
The project documents using Hugging Face datasets to load SWE-bench data and Modal for cloud evaluations.Read 9 Oct 2026 · swebench.com
Support
The contact page directs general inquiries to the SWE-bench team by email.Read 9 Oct 2026 · swebench.com
Founded
2023Read 28 Sep 2026 · swebench.com
Benchmark variants
The site lists SWE-bench Lite, Verified, Multimodal, and Multilingual datasets alongside the original benchmark.Read 8 Oct 2026 · swebench.com
CLI
SWE-bench CLI can submit model predictions for evaluation, retrieve reports, manage evaluation runs, and track performance in the cloud.Read 8 Oct 2026 · swebench.com
Cloud evaluation
Evaluations can run in the cloud using Modal, or be submitted through the sb-cli tool.Read 9 Oct 2026 · swebench.com
Cloud evaluations
The documentation describes cloud-based evaluations using Modal and the sb-cli tool.Read 8 Oct 2026 · swebench.com
Dataset variants
The project lists SWE-bench, SWE-bench Lite, SWE-bench Verified, and SWE-bench Multimodal as available datasets.Read 9 Oct 2026 · swebench.com
Evaluation limit
Docker is required for consistent evaluation environments, according to the FAQ.Read 8 Oct 2026 · swebench.com
Evaluation limits
The guide notes that caching all Docker images can require about 2,000 GB of storage.Read 9 Oct 2026 · swebench.com
Evaluation method
Evaluation applies generated patches to repositories and runs tests in Docker containers to check whether issues are resolved.Read 9 Oct 2026 · swebench.com
Installation
The project requires Python 3.9 or later and Docker for the evaluation environment.Read 9 Oct 2026 · swebench.com
Multilingual dataset
SWE-bench Multilingual contains 300 tasks from 42 repositories across nine programming languages.Read 9 Oct 2026 · swebench.com
Multimodal dataset
SWE-bench Multimodal contains 480 instances with issues described using visual elements.Read 9 Oct 2026 · swebench.com
Operating systems
The Docker setup guide provides installation instructions for macOS, Linux, and Windows, with WSL 2 required on Windows.Read 9 Oct 2026 · swebench.com
Original dataset
The original benchmark contains 2,294 task instances drawn from pull requests and issues in 12 Python repositories.Read 9 Oct 2026 · swebench.com
Resource requirements
The Docker guide recommends at least 120 GB of free disk space and 16 GB or more of RAM.Read 9 Oct 2026 · swebench.com
Verified dataset
SWE-bench Verified is a human-filtered subset containing 500 instances.Read 9 Oct 2026 · swebench.com

Strong points and limits

Strong points

  • Original benchmark has 2,294 tasks from 12 repositories
  • Verified dataset includes 500 engineer-checked instances
  • CLI manages submissions, reports and evaluation runs
  • Cloud evaluations use Modal and sb-cli

Limits

  • Docker is required for consistent evaluation environments
  • Windows requires WSL 2
  • Guide recommends 120 GB free disk space and 16 GB RAM
  • Caching all Docker images can require about 2,000 GB

Who it suits

SWE-bench suits people evaluating language models on real software issues, particularly those able to set up Docker-based runs. Its datasets and CLI also support cloud evaluation workflows.

Verdict

SWE-bench provides several datasets and a test-based way to evaluate generated patches. Its Docker requirement and substantial storage guidance are important setup considerations.

Where Specifiction places it

Alternatives to SWE-benchAll alternatives →