Skip to content
SPECIFICTION
Get listed

Galileo vs SWE-bench

Galileo

galileo.ai

SWE-bench

swebench.com

Side by side

FigureGalileoSWE-bench
Price from$100/mo—
Free plan——
Free trial——
Open source——
WebYesYes
Windows—Yes
macOS—Yes
Linux—Yes
APIYesYes
Self-hostedYesYes
Plans published30
Spec score · LLM Evaluation Tools7.8 (#11)7.0 (#25)
Founded20212023
IntegrationsDocumented integrations include CrewAI, Google ADK, LangChain and LangGraph, OpenAI, Microsoft Agent Framework, Strands Agents, and Vercel AI SDK.The project documents using Hugging Face datasets to load SWE-bench data and Modal for cloud evaluations.
PurposeGalileo is an observability, evaluation, and guardrail platform for GenAI and agentic applications.SWE-bench evaluates language models on real-world GitHub software issues by asking them to generate patches that resolve the issues.
SupportThe Pro plan includes dedicated Slack support, and Enterprise includes 24/7 support via Slack, email, or phone.The contact page directs general inquiries to the SWE-bench team by email.

Plans

Galileo

  • Pro$100 a month
  • EnterprisePriced on request
  • FreePrice not published

Galileo pricing →

SWE-bench

No plans published.

SWE-bench pricing →

In their own words

Galileo

Galileo is an observability, evaluation and guardrail platform for generative AI and agentic applications. It provides ready-made evaluations for RAG, agents, safety and security, as well as custom evaluators. Observability tools analyze agent behavior to surface patterns and failure modes, while Galileo Experiments let teams evaluate datasets and prompts systematically and compare results. Evaluation scores can guide agent actions, tool access and escalation paths. Developers can use Python and TypeScript SDKs or the API, with integrations including CrewAI, Google ADK, LangChain and LangGraph, OpenAI, Microsoft Agent Framework, Strands Agents and Vercel AI SDK. Deployment options include SaaS, virtual private cloud and on-premises. The Free plan includes 5,000 traces per month, unlimited users and unlimited custom evaluations. Pro costs $100 a month, billed yearly, and includes 50,000 traces per month; pricing scales with trace volume. SSO supports OIDC and SAML only on a customer's own Galileo cluster, not the free version or hosted app.galileo.ai service.

Visit galileo.ai

SWE-bench

SWE-bench evaluates language models on software issues collected from GitHub. A model generates a patch, which the evaluation harness applies in a Docker environment before running the repository tests to check whether the issue is resolved. The original benchmark contains 2,294 task instances from 12 Python repositories. Other listed datasets include SWE-bench Lite, Verified, Multimodal and Multilingual. Verified contains 500 instances engineers checked as solvable; Multilingual has 300 instances across nine languages and 42 repositories. Multimodal includes 100 development instances with screenshots and interface elements, while test evaluation uses the SWE-bench API. Its CLI can submit predictions, retrieve reports, manage evaluation runs and track performance in the cloud. Evaluations can run locally or in the cloud using Modal and sb-cli. Installation requires Python 3.9 or later and Docker. Guides cover macOS, Linux and Windows, with WSL 2 required on Windows. The Docker guide recommends at least 120 GB of free disk space and 16 GB of RAM; caching all Docker images can require about 2,000 GB. SWE-bench is free.

Visit swebench.com