Skip to content
SPECIFICTION
Get listed

HELM

crfm.stanford.edu

Free plan WebAPISelf-hosted
Visit crfm.stanford.edu
Spec score 8.3 #7 of 29 in LLM Evaluation Tools

What it is

HELM is an open-source Python framework for evaluating foundation models, including large language and multimodal systems. It supplies standardized datasets and benchmarks such as MMLU-Pro, GPQA, IFEval, and WildBench, with a unified interface for models from providers including OpenAI, Anthropic, and Google. Evaluations can examine accuracy, efficiency, bias, and toxicity, and can include custom metrics, LLM-as-a-judge checks, safety evaluations, and human review workflows. Users can configure private checkpoints or model servers, run inference locally, or send requests to an API. A web interface displays individual prompts and responses, while leaderboards compare models and benchmarks across areas such as medicine, finance, multilinguality, and world knowledge. HELM can reproduce published evaluation results and documents commands for running benchmarks, summarizing results, and starting a local server. It is free, Apache-2.0 licensed, and entered maintenance mode on June 1, 2026.

Spec sheet

Purpose
HELM is an open-source Python framework for holistic, reproducible, and transparent evaluation of foundation models, including language and multimodal models.Read 8 Oct 2026 · crfm-helm.readthedocs.io
Support
Users are directed to open GitHub issues for questions, which maintainers address when bandwidth is available.Read 8 Oct 2026 · crfm-helm.readthedocs.io
License
The repository identifies HELM as open source and links to an Apache-2.0 license.Read 8 Oct 2026 · github.com
Founded
2022Read 28 Sep 2026 · crfm.stanford.edu
Benchmarks
It provides datasets and benchmarks in a standardized format, including MMLU-Pro, GPQA, IFEval, and WildBench.Read 8 Oct 2026 · crfm-helm.readthedocs.io
Compatibility limit
Scenarios and models may break as external APIs change or providers deprecate models, and the maintainers recommend testing them for intended use.Read 8 Oct 2026 · crfm-helm.readthedocs.io
Custom models
Users can configure private models, including a checkpoint on local disk or a model server on a private network.Read 8 Oct 2026 · github.com
Evaluation coverage
The project says it maintains leaderboards across domains such as medicine and finance, and aspects such as multilinguality and world knowledge.Read 8 Oct 2026 · github.com
Evaluation metrics
Its metrics measure aspects beyond accuracy, including efficiency, bias and toxicity.Read 8 Oct 2026 · github.com
Inference options
Model deployments can run local inference or send requests to an API.Read 8 Oct 2026 · github.com
Inspection tools
HELM includes a web UI for inspecting individual prompts and responses and a web leaderboard for comparing results.Read 8 Oct 2026 · github.com
Inspection UI
A web UI lets users inspect individual prompts and responses.Read 8 Oct 2026 · crfm-helm.readthedocs.io
Install and run
The project documents installation from PyPI with `pip install crfm-helm` and commands to run benchmarks, summarize results, and start a local web server.Read 8 Oct 2026 · crfm-helm.readthedocs.io
Installation
The project instructs users to install HELM from PyPI with `pip install crfm-helm`.Read 8 Oct 2026 · github.com
Known limitation
The project describes HELM as intentionally incomplete and invites the community to identify gaps and contribute scenarios, metrics and models.Read 8 Oct 2026 · crfm.stanford.edu
Leaderboards
A web leaderboard compares results across models and benchmarks, with official leaderboards including Capabilities, Safety, and VHELM.Read 8 Oct 2026 · crfm-helm.readthedocs.io
Maintenance
HELM entered maintenance mode on June 1, 2026; no new features or leaderboard evaluations will be added.Read 8 Oct 2026 · crfm-helm.readthedocs.io
Maintenance status
The repository says HELM entered maintenance mode on June 1, 2026.Read 8 Oct 2026 · github.com
Metrics
Its metrics cover aspects beyond accuracy, including efficiency, bias, and toxicity.Read 8 Oct 2026 · crfm-helm.readthedocs.io
Model providers
Models from providers including OpenAI, Anthropic, and Google are accessible through a unified interface.Read 8 Oct 2026 · crfm-helm.readthedocs.io
Reproducibility
The framework can be used to reproduce published model evaluation results.Read 8 Oct 2026 · crfm-helm.readthedocs.io

Plans and prices

HELM Free Open-source Python framework; no paid plans listed Read 8 Oct 2026 · github.com

Every HELM plan and price →

Strong points and limits

Strong points

  • Standardized datasets and benchmarks
  • Unified interface for multiple providers
  • Metrics beyond accuracy
  • Private model configuration
  • Web inspection and leaderboards

Limits

  • Maintenance mode since June 1, 2026
  • No new leaderboard evaluations planned
  • External API changes may break scenarios
  • Intentionally incomplete coverage

Who it suits

HELM suits teams and researchers needing self-hosted, reproducible model evaluation across capabilities and safety. It also fits users who need private models, custom metrics, or human review workflows.

Verdict

HELM combines standardized benchmarks, broad evaluation metrics, private-model support, and inspection tools in a free open-source framework. Its maintenance status and possible provider breakage matter when planning ongoing use.

Where Specifiction places it

Alternatives to HELMAll alternatives →