Pydantic Evals

Spec sheet

Purpose

Pydantic Evals tests and evaluates AI systems, from simple LLM calls to complex multi-agent applications.Read 30 Sep 2026 · pydantic.dev

Security

Pydantic’s Logfire security page states that it has SOC 2 Type 2 audited controls, GDPR compliance, and HIPAA support under a signed Business Associate Agreement.Read 29 Sep 2026 · pydantic.dev

Maker

Pydantic describes itself as a remote-first company.Read 30 Sep 2026 · pydantic.dev

Agent behavior checks

Built-in evaluator examples cover tool correctness, trajectory matching, argument correctness, maximum tool calls, and maximum model requests.Read 29 Sep 2026 · pydantic.dev

Code-first workflow

Evaluation components are defined in Python, and experiment reports can be printed, serialized, or viewed in Pydantic Logfire.Read 30 Sep 2026 · pydantic.dev

Cost consideration

LLM-as-a-judge evaluations can take seconds, cost money, and produce non-deterministic results; deterministic evaluators are described as having no cost.Read 29 Sep 2026 · pydantic.dev

Costs and tradeoffs

LLM judges can be slower, cost money, be non-deterministic, and have biases.Read 30 Sep 2026 · pydantic.dev

Datasets and cases

Datasets group test cases, which can include inputs, expected outputs, metadata, and case-specific evaluators.Read 30 Sep 2026 · pydantic.dev

Dependencies

Pydantic Evals does not depend on pydantic-ai, and Logfire is optional.Read 29 Sep 2026 · pydantic.dev

Evaluator types

Evaluators include deterministic checks, LLM judges, span-based and agentic evaluators, and custom evaluators.Read 30 Sep 2026 · pydantic.dev

Evaluators

It includes built-in deterministic evaluators and supports custom evaluators, LLM judges, and evaluators that return assertions, scores, or labels.Read 29 Sep 2026 · pydantic.dev

Installation

The package installs with `pip install pydantic-evals` or `uv add pydantic-evals`.Read 30 Sep 2026 · pydantic.dev

Logfire integration

Logfire is an optional dependency for using OpenTelemetry traces in evaluations or sending evaluation results to Logfire.Read 30 Sep 2026 · pydantic.dev

Online evaluation

Evaluators can run in the background on every production or staging call, or on a sampled subset of traffic.Read 30 Sep 2026 · pydantic.dev

Production evaluation

Online evaluation can attach evaluators to production or staging traffic so every call or a sampled subset is graded in the background.Read 29 Sep 2026 · pydantic.dev

Related service security

Pydantic says Logfire is SOC 2 Type 2 audited, GDPR aligned, and can support HIPAA workloads under a signed Business Associate Agreement.Read 30 Sep 2026 · pydantic.dev

Third-party integrations

The documentation shows adapters for Ragas and DeepEval, which are optional dependencies and are not installed with Pydantic Evals.Read 30 Sep 2026 · pydantic.dev

What it evaluates

It can grade an agent’s final outputs and tool-call trajectory against datasets or sampled live production traffic.Read 30 Sep 2026 · pydantic.dev

Where Specifiction places it

• AI LLM Evaluation Tools#29 of 306.6