Skip to content
SPECIFICTION
Get listed

LiveBench

livebench.ai

WebWindowsmacOSLinuxSelf-hosted
Visit livebench.ai
Spec score 7.2 #22 of 29 in LLM Evaluation Tools

What it is

LiveBench is an open-source benchmark for evaluating large language models, designed to limit test-set contamination and use objective scoring. Its current scope is 23 tasks across Reasoning, Coding, Agentic Coding, Mathematics, Data Analysis, Language, and Instruction Following, with a refresh every six months. The leaderboard presents overall and category scores, and model rows expand to show subtasks. Insight views cover quality versus cost, ranked cost, and category profiles; cost per successful task is calculated as (Σ cost ÷ Σ questions ÷ score) × 100 for the selected scope. Questions have verifiable ground-truth answers and can be scored automatically without an LLM judge. The public project provides benchmark code and links to data on Hugging Face, as well as a Python command-line evaluation pipeline. It supports OpenAI-compatible endpoints and implemented inference for Anthropic, Cohere, Mistral, Together, and Google models. Local inference is unmaintained. Agentic coding evaluations require Docker, and task images may take up to 150 GB to store.

Spec sheet

Purpose
LiveBench is an LLM benchmark designed to limit test set contamination and use objective evaluation.Read 9 Oct 2026 · github.com
Product
LiveBench is described as a challenging, contamination-free LLM benchmark.Read 4 Oct 2026 · livebench.ai
Support
The repository invites model evaluation requests by GitHub issue or email at [email protected].Read 9 Oct 2026 · github.com
Agentic coding requirements
Evaluating agentic coding tasks requires Docker, and the README says task-specific images may require up to 150 GB of storage.Read 9 Oct 2026 · github.com
Benchmark scope
The current site describes 23 objective tasks across 7 categories, refreshed every six months.Read 4 Oct 2026 · livebench.ai
Categories
The website lists Reasoning, Coding, Agentic Coding, Mathematics, Data Analysis, Language, and Instruction Following.Read 9 Oct 2026 · livebench.ai
Contamination controls
The repository README says questions draw on recently released datasets, papers, news articles, and IMDb movie synopses to limit potential contamination.Read 9 Oct 2026 · github.com
Cost metric
The site defines cost per successful task as (Σ cost ÷ Σ questions ÷ score) × 100 over the selected scope.Read 4 Oct 2026 · livebench.ai
Current benchmark
The website describes 23 objective tasks across 7 categories, refreshed every six months.Read 9 Oct 2026 · livebench.ai
Insights
The website provides quality-versus-cost, cost-ranked, and category-profile views.Read 9 Oct 2026 · livebench.ai
Leaderboard
The leaderboard lets visitors sort overall and category scores, inspect subtask scores, and compare cost per successful task.Read 9 Oct 2026 · livebench.ai
Local inference
The README describes local model inference as unmaintained and recommends serving models through an OpenAI-compatible API using vLLM.Read 9 Oct 2026 · github.com
Local models
The README says local model inference is unmaintained and recommends serving models through an OpenAI-compatible API using vLLM.Read 4 Oct 2026 · github.com
Model compatibility
The README says OpenAI-compatible API endpoints can be used and lists implemented inference support for Anthropic, Cohere, Mistral, Together, and Google models.Read 4 Oct 2026 · github.com
Model endpoints
The repository documents custom OpenAI-compatible API endpoints and inference support for Anthropic, Cohere, Mistral, Together, and Google models.Read 9 Oct 2026 · github.com
Open source
The project publishes its code in a public GitHub repository and links to benchmark data on Hugging Face.Read 4 Oct 2026 · github.com
Run evaluations
The repository provides a Python script that can generate model answers, score them, and show results; it supports selecting a model and benchmark subset.Read 9 Oct 2026 · github.com
Scoring
The repository README says questions have verifiable objective ground-truth answers and are scored automatically without an LLM judge.Read 9 Oct 2026 · github.com

Strong points and limits

Strong points

  • 23 tasks across seven named categories
  • Overall, category, and subtask leaderboard scores
  • Automatic scoring without an LLM judge
  • Public code and Hugging Face data links

Limits

  • Local model inference is unmaintained
  • Agentic coding requires Docker
  • Task images may take up to 150 GB

Who it suits

LiveBench suits people comparing language models on objective tasks or running their own evaluations through supported API endpoints. Those evaluating agentic coding should account for the Docker requirement and image storage needs.

Verdict

LiveBench pairs a multi-category leaderboard with tools and documentation for running evaluations. Its objective scoring and contamination controls are central to its design, while local inference and agentic-coding setup have stated constraints.

Where Specifiction places it

Alternatives to LiveBenchAll alternatives →