Skip to content
SPECIFICTION
Get listed

DeepEval

deepeval.com

From $200/mo Free plan Open source WebWindowsmacOSLinuxAPISelf-hosted
Visit deepeval.com
Spec score 9.4 #1 of 30 in AI LLM Evaluation Tools

What it is

DeepEval is a free, open-source framework for building evaluation pipelines to test AI systems. Its Pytest-native evaluations run in CI/CD or as Python scripts, and the framework lists more than 50 research-backed metrics, including checks for hallucination, faithfulness, answer relevancy, summarization, toxicity and bias. It evaluates text, images and audio, including conversational and voice interactions, using methods such as G-Eval, DAG, QAG and JevEval. DeepEval can generate synthetic goldens from a knowledge base and model conversations across user personas. It also traces agent steps so they can be graded and inspected in a terminal or test runner. Integrations cover agent frameworks including LangChain, Pydantic AI, OpenAI Agents, LangGraph, LlamaIndex and CrewAI, alongside a range of model providers. By default, it sends basic telemetry to PostHog, excluding personally identifiable information and stored results; users can opt out with an environment variable. The maker describes DeepEval OS as for pre-production testing, with results in local files and an engineer-owned test runner. Enterprise capabilities are offered through Confident AI Evals, with self-hosted or maker-cloud deployment options.

Spec sheet

What it does
DeepEval is an open-source framework for testing and evaluating LLM applications locally or in CI/CD.Read 15 Oct 2026 · deepeval.com
Purpose
DeepEval is an open-source LLM evaluation framework for building evaluation pipelines to test AI systems.Read 28 Sep 2026 · deepeval.com
Integrations
Listed integrations include LangChain, Pydantic AI, OpenAI Agents, LangGraph, AWS AgentCore, Strands, Google ADK, LlamaIndex, and CrewAI.Read 28 Sep 2026 · deepeval.com
License
The repository and package identify DeepEval as Apache Software License 2.0.Read 15 Oct 2026 · github.com
Headquarters
San Francisco, California, United StatesRead 23 Sep 2026 · deepeval.com
API and SDKs
Confident AI exposes APIs, a TypeScript SDK and OpenTelemetry support alongside DeepEval's Python framework.Read 15 Oct 2026 · deepeval.com
CLI
The `deepeval test run` command is built on Pytest, while `evaluate()` provides a Python-programmatic interface.Read 15 Oct 2026 · deepeval.com
Cloud data
The maker says data sent to Confident AI is stored in databases in its private AWS cloud, except for organizations on the VIP plan.Read 28 Sep 2026 · deepeval.com
Company relationship
Confident AI created and maintains DeepEval and describes it as one of its open-source testing packages.Read 15 Oct 2026 · linkedin.com
Company size
LinkedIn lists Confident AI as a privately held software company with 11–50 employees and San Francisco headquarters.Read 15 Oct 2026 · linkedin.com
Component evaluation
Teams can evaluate individual spans such as tools, planners, retrievers and generators.Read 15 Oct 2026 · deepeval.com
Customization
Metrics, prompts, judge models and evaluation templates can be customized.Read 15 Oct 2026 · deepeval.com
Documentation
The documentation contains installation, quickstarts, metric references, integrations, tracing, datasets, CLI, environment-variable and troubleshooting sections.Read 15 Oct 2026 · deepeval.com
Enterprise deployment
The enterprise offering is available on Confident AI Evals and can be self-hosted on a customer's infrastructure or run in the maker's cloud.Read 28 Sep 2026 · deepeval.com
Enterprise security
The hosted enterprise offering states SOC 2 Type II, GDPR compliance, SSO, RBAC, audit logs and custom retention.Read 15 Oct 2026 · deepeval.com
Evaluation methods
Its evaluation techniques include G-Eval, DAG, QAG, and JevEval.Read 28 Sep 2026 · deepeval.com
Founders
Y Combinator identifies Jeffrey Ip and Kritin Vongthongsri as Confident AI's founders.Read 15 Oct 2026 · ycombinator.com
Framework integrations
Official integrations include LangChain, LlamaIndex, CrewAI, OpenAI Agents, LangGraph, PydanticAI, Google ADK, AgentCore, Strands and OpenTelemetry.Read 15 Oct 2026 · deepeval.com
Install package
PyPI provides a platform-independent Python wheel and source distribution for DeepEval.Read 15 Oct 2026 · pypi.org
Limit: evaluator cost
AI Tools Atlas notes that LLM-judge metrics require provider API calls, so cost rises with dataset size and metric count.Read 15 Oct 2026 · aitoolsatlas.ai
Limit: reported defect
A GitHub issue reports the introductory tutorial failing with a `ModuleNotFoundError` when running `deepeval test run`.Read 15 Oct 2026 · github.com
Limit: runtime
AI Tools Atlas reports that large or multi-turn evaluations can be computationally slow and expensive.Read 15 Oct 2026 · aitoolsatlas.ai
Limit: team features
AI Tools Atlas says collaboration, dataset management, monitoring and dashboards require the hosted Confident AI service.Read 15 Oct 2026 · aitoolsatlas.ai
Local operation
DeepEval can run entirely locally without Confident AI or an account.Read 15 Oct 2026 · deepeval.com
Local telemetry
By default, DeepEval sends basic telemetry to PostHog, excludes personally identifiable information and stored results, and supports opting out with DEEPEVAL_TELEMETRY_OPT_OUT=1.Read 28 Sep 2026 · deepeval.com
Metrics
The project advertises more than 50 metrics spanning quality, agents, RAG, conversations, safety, voice and multimodal use cases.Read 15 Oct 2026 · deepeval.com
Modalities
The framework supports evaluation of text, images, and audio, including conversational and voice evaluations.Read 28 Sep 2026 · deepeval.com
Model integrations
Supported judge-model providers include OpenAI, Azure OpenAI, Ollama, OpenRouter, Anthropic, Bedrock, Gemini, DeepSeek, Vertex AI, Grok, Moonshot, Portkey, vLLM, LM Studio and LiteLLM.Read 15 Oct 2026 · deepeval.com
Model providers
Evaluation model integrations include OpenAI, Azure OpenAI, Ollama, OpenRouter, Anthropic, Amazon Bedrock, Gemini, DeepSeek, Vertex AI, Grok, Moonshot, Portkey, vLLM, LM Studio, and LiteLLM.Read 28 Sep 2026 · deepeval.com
Notable 2026 change
A 2026 release added Strands OpenTelemetry tracing and span collection.Read 15 Oct 2026 · deepeval.com
Notable limit
The maker describes DeepEval OS as limited to pre-production testing, with results in local files and an engineer-owned test runner.Read 28 Sep 2026 · deepeval.com
Praise: breadth and CI/CD
AI Tools Atlas highlights the broad metric set and suitability for Pytest-based CI/CD quality gates.Read 15 Oct 2026 · aitoolsatlas.ai
Praise: integrations
A G2 reviewer says the Confident AI platform connected smoothly to an existing developer stack through the DeepEval ecosystem.Read 15 Oct 2026 · g2.com
Praise: usability and speed
G2 reviewers describe the hosted platform as intuitive, fast on complex evaluations and easy to onboard.Read 15 Oct 2026 · g2.com
Privacy telemetry
Default telemetry sends PostHog event names, metric names, notebook status, storage type, anonymous ID and public IP; an environment variable disables it.Read 15 Oct 2026 · deepeval.com
Python requirement
The PyPI package declares Python 3.9 or newer and below Python 4.0.Read 15 Oct 2026 · pypi.org
Rating on AI Tools Atlas
4.4 out of 5; the page does not state a review countRead 15 Oct 2026 · aitoolsatlas.ai
Rating on G2
4.6 out of 5 from 5 reviewsRead 15 Oct 2026 · g2.com
Release activity
The public changelog records frequent releases, feature additions, bug fixes and breaking changes by year.Read 15 Oct 2026 · deepeval.com
Self-hosting
Confident AI's enterprise service can be deployed on the customer's own infrastructure or used in the vendor cloud.Read 15 Oct 2026 · deepeval.com
Storage
CLI settings support local JSON or SQLite result backends.Read 15 Oct 2026 · deepeval.com
Support and collaboration
The enterprise page invites prospective customers to book a demo and describes shared workspaces, no-code evaluation workflows, and annotation queues.Read 28 Sep 2026 · deepeval.com
Synthetic data
DeepEval includes synthetic dataset generation for difficult-to-collect edge cases.Read 15 Oct 2026 · deepeval.com
Testing
It provides Pytest-native evaluations that run in CI/CD or as Python scripts.Read 28 Sep 2026 · deepeval.com
Tracing
DeepEval traces agent steps so they can be graded and inspected in the terminal and test runner.Read 28 Sep 2026 · deepeval.com
Trajectory evaluation
Agent trajectories can be scored across plans, model calls, tools and handoffs.Read 15 Oct 2026 · deepeval.com
Unit testing
It provides Pytest-style assertions and pass/fail test runs for LLM outputs.Read 15 Oct 2026 · deepeval.com

Plans and prices

DeepEval Free Open-source LLM evaluation framework · Apache 2.0 licensed · local and CI/CD test runner Read 28 Sep 2026 · deepeval.com
Free Free Forever $0 2 user seats · 1 project · 5 test runs per week · 1 GB-month trace spans Read 15 Oct 2026 · confident-ai.com
Starter $200 a month $200 / month per organization Unlimited seats · 5 projects · 5 GB-month trace spans · $1 per additional GB-month Read 15 Oct 2026 · confident-ai.com
Team $2,000 a month $2,000 / month per organization Unlimited seats · unlimited projects · 75 GB-month trace spans · $1 per additional GB-month Read 15 Oct 2026 · confident-ai.com
Enterprise Priced on request Custom pricing Unlimited seats · unlimited projects · unlimited GB-month trace spans · unlimited online-evaluation metric runs Read 15 Oct 2026 · confident-ai.com

Every DeepEval plan and price →

Strong points and limits

Strong points

  • Free and Apache 2.0 licensed.
  • More than 50 listed evaluation metrics.
  • Evaluates text, images and audio.
  • Pytest-native evaluations run in CI/CD.

Limits

  • DeepEval OS is limited to pre-production testing.
  • Results in DeepEval OS are kept in local files.
  • The test runner is engineer-owned.
  • Basic telemetry is sent by default.

Who it suits

DeepEval suits developers building evaluation pipelines for AI systems, particularly those who want tests to run in CI/CD or Python scripts. Teams needing enterprise collaboration and security features can consider the separate Confident AI Evals offering.

DeepEval in depth

Read the full write-up

DeepEval is a free, open-source evaluation framework for teams building tests into Python-based AI development. It suits engineers who want broad checks for models and agents in local runs and CI/CD; its main trade-off is that DeepEval OS is for pre-production testing, not a shared production evaluation workspace.

Overview

Pytest-native evaluations and Python scripts let teams make AI checks part of their development and regression-testing routines. Results stay in local files and the test runner is engineer-owned, which keeps the workflow close to code but leaves collaboration and production workflows to another layer. DeepEval OS is explicitly limited to pre-production testing.

Key features

More than 50 research-backed metrics cover hallucination, faithfulness, answer relevancy, summarization, toxicity, and bias. That breadth can serve teams evaluating different failure modes without building every check from scratch. Support for text, images, and audio, including conversational and voice evaluations, makes the framework relevant beyond text-only applications. G-Eval, DAG, QAG, and JevEval offer multiple evaluation techniques.

Teams can generate synthetic goldens from a knowledge base and simulate conversations across user personas, useful when assembling evaluation cases. DeepEval traces agent steps for grading and inspection in the terminal and test runner. Integrations include LangChain, Pydantic AI, OpenAI Agents, LangGraph, AWS AgentCore, Strands, Google ADK, LlamaIndex, and CrewAI; evaluation model integrations range from OpenAI and Anthropic to Ollama, Gemini, Amazon Bedrock, vLLM, and LiteLLM.

The framework supports tool-call checks, trace ingestion, safety evaluations, regression runs, custom metrics, LLM-as-a-judge, human review workflows, prompt versioning, and CI/CD integration. Basic telemetry is sent to PostHog by default, excluding personally identifiable information and stored results; teams can opt out with DEEPEVAL_TELEMETRY_OPT_OUT=1. Data sent to Confident AI is stored in databases in its private AWS cloud, except for VIP organizations.

Pricing

DeepEval: free. The plan is an Apache 2.0-licensed open-source framework with a local and CI/CD test runner. It is the natural starting point for engineering teams that can own evaluation tests and do not need centralized collaboration. The free plan's practical limit is its pre-production, local-file workflow.

Enterprise capabilities are offered through Confident AI Evals with custom pricing. The enterprise offering can be self-hosted on a customer's infrastructure or run in the maker's cloud. It adds shared workspaces, no-code evaluation workflows, annotation queues, SSO, role-based access control, granular permissions, audit logs, SOC 2 Type II, GDPR compliance, and custom data retention. Prospective customers can book a demo. This is the route for organizations needing collaborative workflows or enterprise controls rather than a larger self-managed test suite.

Platforms

DeepEval supports Linux, macOS, Windows, and self-hosted deployment, aligning with teams that run evaluations in their own development and CI/CD environments.

Who it's for

Choose DeepEval if engineers need an open-source Python testing framework with broad metric coverage, multimodal evaluation, agent tracing, and CI/CD support. It is less suitable as a standalone choice for teams that need production monitoring, shared evaluation workspaces, or no-code workflows; those needs point toward the Confident AI Evals enterprise offering.

Pros and cons

Pros: A free, Apache 2.0-licensed framework combines 50+ metrics, multiple evaluation methods, synthetic test-case generation, and support for text, image, and audio checks.

Pros: Pytest-native tests, tracing, and CI/CD integration put regression checks in an engineer-owned workflow.

Cons: DeepEval OS is limited to pre-production testing, with results in local files and an engineer-owned runner; teams needing centralized collaboration must use another offering.

Cons: Basic telemetry goes to PostHog by default, although it excludes personally identifiable information and stored results and can be disabled with an environment variable.

Alternatives

Promptfoo is worth considering for teams seeking a free community plan with 10k red-team probes per month, broad evaluation features, and local or self-hosted use. Maxim AI offers a free Developer plan with up to 3 seats, one workspace, 10k logs per month, three-day retention, and email support, as well as a free trial.

Whisper is a free, locally installable, MIT-licensed code-and-model option for readers seeking speech recognition rather than a general evaluation framework. Giskard has a free open-source library with local deployment and basic LLM vulnerability scanning and RAG evaluation.

Braintrust may fit teams wanting a hosted-style evaluation workflow with a free Starter tier that includes 1 GB processed data, 10,000 scores, 14-day retention, and unlimited users, projects, and datasets. Galileo offers a Pro plan at $100 a month, billed yearly, with 50,000 traces per month, standard RBAC, advanced analytics, and dedicated Slack support.

garak is a free open-source option for readers who want a tool focused on probing LLM vulnerabilities. Parea AI has a free plan capped at two team members, 3k logs per month with one-month retention, and 10 deployed prompts, making it an option for teams whose needs fit those limits.

Browse LLM Evaluation Tools, AI Agent Evaluation Tools, and AI LLM Evaluation Tools for more options.

Verdict

DeepEval is a strong choice for engineering teams that want a free, extensible way to turn broad AI evaluations into local and CI/CD tests. Its multimodal coverage and agent-focused checks are compelling, but teams that need shared, production-oriented workflows should look to Confident AI Evals or another platform.

Verdict

DeepEval offers a free framework for evaluating AI systems across modalities, with metrics, tracing and CI/CD support. Its DeepEval OS offering is oriented to pre-production tests; enterprise capabilities are described through Confident AI Evals.

Where Specifiction places it

Alternatives to DeepEvalAll alternatives →