Skip to content
SPECIFICTION
Get listed

DecodingTrust

decodingtrust.github.io

APISelf-hosted
Visit decodingtrust.github.io
Spec score 7.5 #16 of 29 in LLM Evaluation Tools

Spec sheet

Purpose
DecodingTrust is a research project for assessing trustworthiness in GPT models and helping researchers and practitioners understand their capabilities, limitations, and deployment risks.Read 11 Oct 2026 · decodingtrust.github.io
Intended users
The project describes its resources as helping researchers and practitioners assess language model capabilities, limitations, and risks.Read 11 Oct 2026 · decodingtrust.github.io
Support
The project invites questions at [email protected].Read 11 Oct 2026 · decodingtrust.github.io
License
The dataset and project are distributed under the CC BY-SA 4.0 license.Read 4 Oct 2026 · decodingtrust.github.io
Assessment areas
The benchmark covers toxicity, stereotype and bias, adversarial robustness, out-of-distribution robustness, privacy, adversarial demonstrations, machine ethics, and fairness.Read 11 Oct 2026 · decodingtrust.github.io
Compute requirements
The installation instructions note that the pip installation includes PyTorch and caution users to check CUDA compatibility with the PyTorch pip wheel.Read 11 Oct 2026 · github.com
Container option
The repository provides Docker instructions and a Docker image for running DecodingTrust.Read 11 Oct 2026 · github.com
Content warning
The project warns that its data contains model outputs that may be considered offensive.Read 11 Oct 2026 · decodingtrust.github.io
Dataset
The dataset is distributed under the CC BY-SA 4.0 license.Read 11 Oct 2026 · decodingtrust.github.io
Deployment options
The project documents Docker and Singularity or Apptainer container usage, including for HPC environments.Read 4 Oct 2026 · github.com
Evaluation areas
The benchmark covers toxicity, stereotype and bias, adversarial robustness, out-of-distribution robustness, privacy, adversarial demonstrations, machine ethics, and fairness.Read 4 Oct 2026 · decodingtrust.github.io
Evaluation scripts
The project provides evaluation scripts in subdirectories for its trustworthiness facets.Read 11 Oct 2026 · decodingtrust.github.io
Installation
The repository recommends cloning the project and installing it in editable mode with pip; it also describes standalone package installation.Read 11 Oct 2026 · github.com
Model coverage limit
The repository says its benchmark mainly focuses on GPT-3.5-turbo-0301 and GPT-4-0314 for consistent conclusions and results.Read 4 Oct 2026 · github.com
Models
The project says its evaluations mainly focus on GPT-3.5 and GPT-4, and it also supports causal LLMs hosted on Hugging Face or locally.Read 4 Oct 2026 · github.com
Models evaluated
The project says its evaluation mainly focuses on GPT-3.5 and GPT-4.Read 11 Oct 2026 · decodingtrust.github.io
Ongoing contributions
The team invites users to submit model generations and corresponding scores through GitHub pull requests and says the dataset is expected to evolve.Read 11 Oct 2026 · decodingtrust.github.io
Platform support
The repository says it supports the ppc64le architecture on IBM Power-9 platforms.Read 11 Oct 2026 · github.com
Reproducibility
The benchmark uses timestamped GPT-3.5 and GPT-4 model versions to support consistent results and reproducibility.Read 4 Oct 2026 · github.com
Resources
The project provides a dataset and evaluation scripts organized by trustworthiness area.Read 4 Oct 2026 · decodingtrust.github.io
Supported architecture
The repository says it supports the ppc64le architecture on IBM Power-9 platforms.Read 4 Oct 2026 · github.com

Where Specifiction places it

Alternatives to DecodingTrustAll alternatives →