AgentBench vs SWE-bench
AgentBench
github.comSWE-bench
swebench.comSide by side
| Figure | AgentBench | SWE-bench |
|---|---|---|
| Price from | — | — |
| Free plan | — | — |
| Free trial | — | — |
| Open source | — | — |
| Web | — | Yes |
| Windows | — | Yes |
| macOS | — | Yes |
| Linux | Yes | Yes |
| API | Yes | Yes |
| Self-hosted | Yes | Yes |
| Plans published | 0 | 0 |
| Spec score · LLM Evaluation Tools | 7.8 (#10) | 7.0 (#25) |
| Founded | 2023 | 2023 |
| Purpose | AgentBench is a benchmark for evaluating the reasoning and decision-making abilities of LLMs acting as agents. | SWE-bench evaluates language models on real-world GitHub software issues by asking them to generate patches that resolve the issues. |
| Support | The project invites users to join its Slack for questions and collaboration, and lists a Google Group for questions about AgentBench FC. | The contact page directs general inquiries to the SWE-bench team by email. |
In their own words
SWE-bench
SWE-bench evaluates language models on software issues collected from GitHub. A model generates a patch, which the evaluation harness applies in a Docker environment before running the repository tests to check whether the issue is resolved. The original benchmark contains 2,294 task instances from 12 Python repositories. Other listed datasets include SWE-bench Lite, Verified, Multimodal and Multilingual. Verified contains 500 instances engineers checked as solvable; Multilingual has 300 instances across nine languages and 42 repositories. Multimodal includes 100 development instances with screenshots and interface elements, while test evaluation uses the SWE-bench API. Its CLI can submit predictions, retrieve reports, manage evaluation runs and track performance in the cloud. Evaluations can run locally or in the cloud using Modal and sb-cli. Installation requires Python 3.9 or later and Docker. Guides cover macOS, Linux and Windows, with WSL 2 required on Windows. The Docker guide recommends at least 120 GB of free disk space and 16 GB of RAM; caching all Docker images can require about 2,000 GB. SWE-bench is free.