Skip to content
SPECIFICTION
Get listed

HarmBench

github.com

LinuxAPISelf-hosted
Visit github.com
Spec score 7.2 #21 of 29 in LLM Evaluation Tools

Spec sheet

Purpose
HarmBench is an open-source framework for evaluating automated red teaming methods and LLM attacks and defenses.Read 8 Oct 2026 · github.com
Support
The repository documentation invites users to submit a pull request for support for another cluster manager.Read 8 Oct 2026 · github.com
License
The public GitHub repository lists an MIT license.Read 8 Oct 2026 · github.com
Classifiers
The project provides three classifier models for standard, contextual, and multimodal behaviors, including a validation classifier.Read 8 Oct 2026 · github.com
Compute requirements
The documentation says evaluation requires enough GPUs for the target model, attack, and classifier, and its default GPU settings were chosen for 80GB A100s.Read 8 Oct 2026 · github.com
Custom methods
Users can add red teaming methods by creating a subfolder in the baselines directory and implementing the RedTeamingMethod class.Read 8 Oct 2026 · github.com
Custom models
Users can add Hugging Face Transformers models through the model configuration file, though AutoDAN, PAIR, and TAP require manual experiment configuration for new models.Read 8 Oct 2026 · github.com
Customization
Users can add Hugging Face Transformers models through the model configuration and add red-teaming methods in the baselines directory.Read 8 Oct 2026 · github.com
Evaluation uses
The framework supports evaluating red teaming methods against LLMs and evaluating LLMs against red teaming methods.Read 8 Oct 2026 · github.com
Evaluation workflow
Its pipeline generates red-team test cases, prompts target models to generate completions, and classifies those completions to calculate attack success rate.Read 8 Oct 2026 · github.com
Execution
The pipeline can run sequentially on the current machine, in parallel across GPUs on one machine with Ray, or with SLURM across machines.Read 8 Oct 2026 · github.com
Installation
The README instructs users to clone the repository, install requirements with pip, and download the en_core_web_sm spaCy model.Read 8 Oct 2026 · github.com
Model support
HarmBench supports Transformers-compatible LLMs, numerous closed-source APIs, and several multimodal models out of the box.Read 8 Oct 2026 · github.com
Pipeline
Its evaluation pipeline generates test cases, generates model completions, and evaluates completions, with an optional test-case merging step.Read 8 Oct 2026 · github.com
Pipeline execution
The pipeline can run locally, in parallel across GPUs on one machine with Ray, or through SLURM jobs.Read 8 Oct 2026 · github.com
Security and compliance
The repository README and linked HarmBench website page provide no security or compliance claims; the website page opened here only says JavaScript must be enabled.Read 8 Oct 2026 · harmbench.org
Usage caveat
The documentation says its text and multimodal classifier models are test classifiers intended only for evaluation.Read 8 Oct 2026 · github.com
Who it serves
The project describes uses for evaluating red-teaming methods against LLMs and evaluating LLMs against red-teaming methods.Read 8 Oct 2026 · github.com

Where Specifiction places it

Alternatives to HarmBenchAll alternatives →