Skip to content
SPECIFICTION
Get listed

OpenCompass

opencompass.org.cn

WindowsLinuxAPISelf-hosted

Spec sheet

Purpose
OpenCompass is a platform and toolkit for evaluating large language models and large vision-language models.Read 8 Oct 2026 · github.com
Intended users
The project describes CompassHub and CompassKit for researchers and practitioners evaluating language and vision-language models.Read 4 Oct 2026 · github.com
License
The GitHub repository links an Apache-2.0 license.Read 4 Oct 2026 · github.com
API support
The installation guide documents API evaluation and lists dependencies for OpenAI, Anthropic, Qwen, and ByteDance Volcano Engine APIs.Read 4 Oct 2026 · opencompass.readthedocs.io
Benchmarks
The project describes support for more than 100 datasets covering areas such as knowledge, reasoning, coding, science, language, long context, and safety.Read 4 Oct 2026 · github.com
Community support
The README links to Discord and WeChat and directs users to GitHub for reporting issues.Read 8 Oct 2026 · github.com
Compatibility limit
The README says APPS, TACO, and LiveCodeBench code-generation evaluations are unavailable without pyext, and recommends Python 3.10 for those datasets.Read 8 Oct 2026 · github.com
Dataset limitation
The basic installation supports most fundamental datasets, while some datasets require additional dependencies or manual setup.Read 4 Oct 2026 · opencompass.readthedocs.io
Datasets
OpenCompass publishes recommended configurations for supported evaluation datasets.Read 8 Oct 2026 · doc.opencompass.org.cn
Distributed evaluation
The project says it supports task division and distributed evaluation.Read 8 Oct 2026 · github.com
Evaluation dimensions
Its documentation describes evaluations across language, knowledge, understanding, reasoning, safety, and specialized capabilities.Read 8 Oct 2026 · doc.opencompass.org.cn
Evaluation methods
It supports objective and subjective evaluation, including model-assisted judging and human-feedback-based judging.Read 8 Oct 2026 · doc.opencompass.org.cn
Extensibility
The project says users can add models and datasets and customize task division strategies and cluster management support.Read 4 Oct 2026 · github.com
Inference backends
The README gives LMDeploy and vLLM as accelerated inference backend options.Read 8 Oct 2026 · github.com
Installation
The project documents installation with pip and from source, and its README recommends using conda to manage the Python environment.Read 8 Oct 2026 · github.com
Model integrations
Documented model integrations include OpenAI, Gemini, Claude, Qwen, and other API models, as well as open-source models.Read 4 Oct 2026 · github.com
Model support
The README describes support for Hugging Face models, API models, and custom models.Read 8 Oct 2026 · github.com
Reporting
The quick start says users can enable Lark reporting for status updates in Lark clients.Read 4 Oct 2026 · opencompass.readthedocs.io
Requirements
The installation guide states that OpenCompass requires PyTorch 1.13 or later.Read 4 Oct 2026 · opencompass.readthedocs.io
Safety evaluation
Its evaluation dimensions include fairness, legality, harmlessness, ethical considerations, and privacy protection.Read 8 Oct 2026 · doc.opencompass.org.cn
Scale
The project describes distributed evaluation and a scheme with 70+ datasets and about 400,000 questions.Read 4 Oct 2026 · github.com
Windows compatibility
A release note says OpenCC was made optional to restore Windows installation compatibility.Read 8 Oct 2026 · github.com
Workflow and outputs
The quick start describes configuring, running inference and evaluation, and saving results as CSV and TXT files.Read 4 Oct 2026 · opencompass.readthedocs.io

Where Specifiction places it

Alternatives to OpenCompassAll alternatives →