Skip to content
SPECIFICTION
Get listed

OpenAI Evals

evals.openai.com

WebAPISelf-hosted
Visit evals.openai.com
Spec score 7.4 #24 of 30 in AI LLM Evaluation Tools

Spec sheet

Purpose
Evals is a framework for evaluating large language models and LLM systems, with an open-source benchmark registry.Read 1 Oct 2026 · github.com
Integrations
The repository supports logging eval results to Snowflake and says evals can be run and created using Weights & Biases.Read 1 Oct 2026 · github.com
Alternative
OpenAI recommends trying Datasets for a more iterative environment to experiment while building an eval.Read 30 Sep 2026 · developers.openai.com
API access
Evals can be created and managed programmatically through the Evals API or configured in the dashboard.Read 1 Oct 2026 · developers.openai.com
Asynchronous scale
Evals run asynchronously, support larger data volumes, and let users monitor performance across versions.Read 1 Oct 2026 · developers.openai.com
Custom endpoint security
Custom model endpoints must use HTTPS, and OpenAI says it encrypts API keys supplied for them.Read 30 Sep 2026 · developers.openai.com
Dashboard
Evals can be configured and run directly in the OpenAI Dashboard.Read 1 Oct 2026 · github.com
Dashboard and API
You can configure evals in the OpenAI dashboard or programmatically with the Evals API.Read 30 Sep 2026 · developers.openai.com
Data handling
Calls to external models pass data to third parties and are subject to different terms and weaker safety guarantees than calls to OpenAI models.Read 30 Sep 2026 · developers.openai.com
Data residency
The /v1/evals endpoint supports United States and Europe (EEA plus Switzerland) data residency, with service-level support listed.Read 1 Oct 2026 · developers.openai.com
Deprecation
OpenAI says the Evals platform becomes read-only for existing users on October 31, 2026, and the dashboard and API are scheduled to shut down on November 30, 2026.Read 1 Oct 2026 · developers.openai.com
Evaluation criteria
An eval uses a data-source schema and testing criteria consisting of graders that determine whether model output is correct.Read 1 Oct 2026 · developers.openai.com
External model limits
Tool calls are currently not supported when running evals with external models.Read 30 Sep 2026 · developers.openai.com
External models
The OpenAI Platform can run evals against third-party models and custom model endpoints.Read 1 Oct 2026 · developers.openai.com
Graders
Supported grader types include string check, text similarity, score model, label model, and Python code execution graders.Read 1 Oct 2026 · developers.openai.com
Local installation
The open-source package can be installed locally with pip, and the repository requires Python 3.9 or newer.Read 1 Oct 2026 · github.com
Model support
Evaluations can use OpenAI models, third-party models, and custom model endpoints.Read 30 Sep 2026 · developers.openai.com
Private evaluations
Users can build private evals from their own data without exposing that data publicly.Read 1 Oct 2026 · github.com
Security compliance
OpenAI states that its API business services have undergone an independent SOC 2 Type 2 examination and that it maintains ISO/IEC 27001:2022 and ISO/IEC 27701:2019 certifications for supporting systems.Read 1 Oct 2026 · openai.com
Test data
An eval run uses a test dataset representing the kind of data you expect your prompt to handle.Read 30 Sep 2026 · developers.openai.com
Third-party eligibility
Using third-party models requires an OpenAI organization at usage tier 1 or higher and an admin to enable the feature and accept its usage disclaimer.Read 30 Sep 2026 · developers.openai.com
Third-party providers
The listed third-party model providers are Google, Anthropic hosted on AWS Bedrock, Together, and Fireworks.Read 30 Sep 2026 · developers.openai.com
Third-party spend limits
OpenAI currently covers third-party inference costs up to monthly limits of $5, $25, $50, $100, and $200 for usage tiers 1 through 5, respectively.Read 30 Sep 2026 · developers.openai.com
Usage cost
Running evals requires an OpenAI API key and incurs the API costs associated with model usage.Read 1 Oct 2026 · github.com

Where Specifiction places it

Alternatives to OpenAI EvalsAll alternatives →