Skip to content
SPECIFICTION
Get listed

LM Evaluation Harness

github.com

Free plan LinuxAPISelf-hosted
Visit github.com
Spec score 7.5 #17 of 29 in LLM Evaluation Tools

What it is

LM Evaluation Harness is a free, open-source framework for evaluating generative language models across many tasks. It lists more than 60 standard academic benchmarks, with hundreds of subtasks and variants. Supported model options include Hugging Face Transformers, GPT-NeoX, Megatron-DeepSpeed, and vLLM. The harness also connects to hosted OpenAI and Anthropic APIs and to local servers compatible with OpenAI APIs. Users can define prompts and metrics, configure tasks with YAML, and extend model backends, filters, metrics, and aggregations through external packages or local modules. Publicly available prompts are intended to support reproducibility and comparison between papers. The base package contains the core framework; model backends are separate optional installs. It runs self-hosted on Linux and API environments, with documented Windows ML and ONNX Runtime accelerator options. MIT licensing applies. APIs without logits or log probabilities are limited to generate-until tasks.

Spec sheet

Purpose
LM Evaluation Harness is a unified framework for evaluating generative language models across evaluation tasks.Read 4 Oct 2026 · github.com
Support
The project directs users to open a GitHub issue or join the EleutherAI Discord for support.Read 4 Oct 2026 · github.com
API integrations
The harness supports hosted APIs including OpenAI and Anthropic, plus local servers compatible with OpenAI APIs.Read 4 Oct 2026 · github.com
Benchmarks
The project lists over 60 standard academic LLM benchmarks with hundreds of subtasks and variants.Read 4 Oct 2026 · github.com
Custom evaluation
Users can define custom prompts and evaluation metrics, and can configure tasks with YAML.Read 4 Oct 2026 · github.com
Installation
The base package provides the core framework while model backends are installed separately as optional extras.Read 4 Oct 2026 · github.com
Model support
Supported model options include Hugging Face Transformers, GPT-NeoX, Megatron-DeepSpeed, and vLLM.Read 4 Oct 2026 · github.com
Notable limitation
For model APIs that do not provide logits or log probabilities, the documentation limits use to generate-until tasks.Read 4 Oct 2026 · github.com
Plugins
Model backends, filters, metrics, and aggregations can be registered from external packages through entry points or loaded from a local module.Read 4 Oct 2026 · github.com
Reproducibility
The project says its use of publicly available prompts supports reproducibility and comparability between papers.Read 4 Oct 2026 · github.com
Security and license
The repository provides the software under the MIT License, which states that it is provided “AS IS” without warranty.Read 4 Oct 2026 · github.com
Supported platforms
Documented runtime options include Windows ML and ONNX Runtime providers for CPU, CUDA, DirectML, NPU, ROCm, and other accelerators.Read 4 Oct 2026 · github.com
Users
The project says the harness serves as the backend for Hugging Face’s Open LLM Leaderboard and is used by research papers and organizations.Read 4 Oct 2026 · github.com

Plans and prices

LM Evaluation Harness Free MIT-licensed software · model backends and task dependencies may require separate optional installs Read 4 Oct 2026 · github.com

Every LM Evaluation Harness plan and price →

Strong points and limits

Strong points

  • Over 60 benchmarks with hundreds of subtasks
  • Supports major model backends and hosted APIs
  • Custom prompts, metrics, and YAML task configuration
  • Extensible through packages or local modules
  • MIT-licensed and self-hosted

Limits

  • Model backends require separate optional installs
  • Some APIs only support generate-until tasks
  • No free trial; software is self-hosted

Who it suits

It suits researchers, organizations, and developers comparing language models across standard or custom evaluations. It is relevant to teams needing self-hosted runs, API connections, or configurable metrics.

Verdict

LM Evaluation Harness combines broad benchmark coverage with configurable tasks and model connections. Check backend installation needs and API log-probability limits before adopting it.

Where Specifiction places it

Alternatives to LM Evaluation HarnessAll alternatives →