OpenAI Evals
evals.openai.com
Spec sheet
- Purpose
- Evals is a framework for evaluating large language models and LLM systems, with an open-source benchmark registry.Read 1 Oct 2026 · github.com
- Integrations
- The repository supports logging eval results to Snowflake and says evals can be run and created using Weights & Biases.Read 1 Oct 2026 · github.com
- Alternative
- OpenAI recommends trying Datasets for a more iterative environment to experiment while building an eval.Read 30 Sep 2026 · developers.openai.com
- API access
- Evals can be created and managed programmatically through the Evals API or configured in the dashboard.Read 1 Oct 2026 · developers.openai.com
- Asynchronous scale
- Evals run asynchronously, support larger data volumes, and let users monitor performance across versions.Read 1 Oct 2026 · developers.openai.com
- Custom endpoint security
- Custom model endpoints must use HTTPS, and OpenAI says it encrypts API keys supplied for them.Read 30 Sep 2026 · developers.openai.com
- Dashboard
- Evals can be configured and run directly in the OpenAI Dashboard.Read 1 Oct 2026 · github.com
- Dashboard and API
- You can configure evals in the OpenAI dashboard or programmatically with the Evals API.Read 30 Sep 2026 · developers.openai.com
- Data handling
- Calls to external models pass data to third parties and are subject to different terms and weaker safety guarantees than calls to OpenAI models.Read 30 Sep 2026 · developers.openai.com
- Data residency
- The /v1/evals endpoint supports United States and Europe (EEA plus Switzerland) data residency, with service-level support listed.Read 1 Oct 2026 · developers.openai.com
- Deprecation
- OpenAI says the Evals platform becomes read-only for existing users on October 31, 2026, and the dashboard and API are scheduled to shut down on November 30, 2026.Read 1 Oct 2026 · developers.openai.com
- Evaluation criteria
- An eval uses a data-source schema and testing criteria consisting of graders that determine whether model output is correct.Read 1 Oct 2026 · developers.openai.com
- External model limits
- Tool calls are currently not supported when running evals with external models.Read 30 Sep 2026 · developers.openai.com
- External models
- The OpenAI Platform can run evals against third-party models and custom model endpoints.Read 1 Oct 2026 · developers.openai.com
- Graders
- Supported grader types include string check, text similarity, score model, label model, and Python code execution graders.Read 1 Oct 2026 · developers.openai.com
- Local installation
- The open-source package can be installed locally with pip, and the repository requires Python 3.9 or newer.Read 1 Oct 2026 · github.com
- Model support
- Evaluations can use OpenAI models, third-party models, and custom model endpoints.Read 30 Sep 2026 · developers.openai.com
- Private evaluations
- Users can build private evals from their own data without exposing that data publicly.Read 1 Oct 2026 · github.com
- Security compliance
- OpenAI states that its API business services have undergone an independent SOC 2 Type 2 examination and that it maintains ISO/IEC 27001:2022 and ISO/IEC 27701:2019 certifications for supporting systems.Read 1 Oct 2026 · openai.com
- Test data
- An eval run uses a test dataset representing the kind of data you expect your prompt to handle.Read 30 Sep 2026 · developers.openai.com
- Third-party eligibility
- Using third-party models requires an OpenAI organization at usage tier 1 or higher and an admin to enable the feature and accept its usage disclaimer.Read 30 Sep 2026 · developers.openai.com
- Third-party providers
- The listed third-party model providers are Google, Anthropic hosted on AWS Bedrock, Together, and Fireworks.Read 30 Sep 2026 · developers.openai.com
- Third-party spend limits
- OpenAI currently covers third-party inference costs up to monthly limits of $5, $25, $50, $100, and $200 for usage tiers 1 through 5, respectively.Read 30 Sep 2026 · developers.openai.com
- Usage cost
- Running evals requires an OpenAI API key and incurs the API costs associated with model usage.Read 1 Oct 2026 · github.com
Where Specifiction places it
- AI LLM Evaluation Tools#24 of 307.4