Skip to content
SPECIFICTION
Get listed

CyberSecEval

meta-llama.github.io

Free plan Open source WebmacOSLinuxAPISelf-hosted
Visit meta-llama.github.io
Spec score 8.9 #1 of 31 in LLM Security Tools

What it is

CyberSecEval 4 is a benchmark suite for assessing cybersecurity vulnerabilities and defensive capabilities in large language models. Its tests measure model responses to cyberattack requests and whether borderline benign requests are incorrectly refused. Secure-code checks look for insecure suggestions in both instruction and autocomplete settings. Prompt-injection tests cover textual, multilingual textual, and visual attacks; a judge LLM evaluates whether injected instructions succeeded. The code-interpreter benchmark evaluates compliance with malicious prompts and sorts judged responses into three categories: extremely malicious, potentially malicious, or non-malicious. AutoPatchBench measures patch-generation agents on fuzzed C and C++ vulnerabilities, while CyberSOCEval, developed with CrowdStrike, covers malware analysis and threat-intelligence reasoning. The guide lists API support for OpenAI, Anyscale, and Together, and explains adding custom providers for self-hosted models. Setup uses a Python virtual environment, installed requirements, and a Python command-line module. The suite is free under the MIT license for research and commercial use. AutoPatch needs substantial resources: the page recommends at least 80 CPUs and 8 TB of storage.

Spec sheet

What it does
CyberSecEval is Meta’s benchmark suite for measuring cybersecurity risks and defensive capabilities of large language models.Read 15 Oct 2026 · meta-llama.github.io
Purpose
CyberSecEval 4 is a benchmark suite for assessing cybersecurity vulnerabilities and defensive capabilities in large language models.Read 8 Oct 2026 · meta-llama.github.io
Intended users
The project describes itself as tools and evaluations intended to help the community build responsibly with open generative AI models.Read 8 Oct 2026 · github.com
License
Cybersecurity evaluations and benchmarks in Purple Llama are licensed under MIT, while included Llama safeguard models use separate Llama Community licenses.Read 15 Oct 2026 · github.com
Automated evaluation
The original research describes automated test-case generation and an evaluation pipeline for LLM cybersecurity safety.Read 15 Oct 2026 · arxiv.org
AutoPatch requirements
AutoPatch requires Podman and substantial compute and storage; the page recommends at least 80 CPUs and 8 TB storage and says Apple Silicon MacBooks are not currently supported.Read 8 Oct 2026 · meta-llama.github.io
AutoPatchBench
AutoPatchBench measures AI patch-generation agents on C/C++ vulnerabilities found by fuzzing.Read 8 Oct 2026 · meta-llama.github.io
Code interpreter
The code interpreter benchmark assesses whether models comply with malicious prompts and categorizes judged responses as extremely malicious, potentially malicious, or non-malicious.Read 8 Oct 2026 · meta-llama.github.io
Company founded
Meta’s SEC filing says the company was incorporated in Delaware in July 2004.Read 15 Oct 2026 · sec.gov
Company headquarters
Meta lists its principal executive offices at 1 Meta Way, Menlo Park, California 94025.Read 15 Oct 2026 · sec.gov
Company legal name
Meta’s SEC filing identifies the maker as Meta Platforms, Inc., a Delaware corporation.Read 15 Oct 2026 · sec.gov
Company size
LinkedIn lists Meta as having more than 10,001 employees and headquarters in Menlo Park, California.Read 15 Oct 2026 · linkedin.com
Current version
CyberSecEval 4 adds AutoPatchBench for measuring an LLM agent’s ability to automatically patch vulnerabilities in native code.Read 15 Oct 2026 · meta-llama.github.io
CyberSecEval 2 additions
CyberSecEval 2 added prompt-injection, code-interpreter-abuse and vulnerability-exploitation evaluations.Read 15 Oct 2026 · arxiv.org
CyberSecEval 3 coverage
CyberSecEval 3 evaluates eight risks across risks to third parties and risks to application developers or end users.Read 15 Oct 2026 · arxiv.org
CyberSecEval 3 features
CyberSecEval 3 added automated social engineering, scaled manual offensive operations and autonomous offensive operations tests.Read 15 Oct 2026 · arxiv.org
Documentation depth
The repository documents benchmark commands, provider formats, self-hosted integration, datasets, judging stages and developer tests.Read 15 Oct 2026 · github.com
Extensibility
Developers can add providers by implementing an LLM class, registering it in the factory and supplying query and chat methods.Read 15 Oct 2026 · github.com
False-refusal metric
CyberSecEval 2 introduced False Refusal Rate to quantify the tradeoff between rejecting unsafe requests and answering benign ones.Read 15 Oct 2026 · arxiv.org
Independent assessment
The peer-reviewed critique describes CyberSecEval as useful in a young field but identifies features that limit its utility.Read 15 Oct 2026 · arxiv.org
Independent cost observation
An Inspect Evals report estimated full one-epoch CyberSecEval 4 runs at roughly $29 to $213 per model depending on the model used.Read 15 Oct 2026 · github.com
Independent implementation limit
Inspect Evals intentionally omits the autonomous-uplift and autopatching prototypes from its public CyberSecEval 4 task surface.Read 15 Oct 2026 · github.com
Installation
The documented setup creates a Python virtual environment and installs dependencies from requirements.txt.Read 15 Oct 2026 · github.com
Limit: narrow ruleset
The critique reports that Meta’s 89 Semgrep rules cover eight languages, versus 2,116 rules and 28 languages in the comparison repository.Read 15 Oct 2026 · kairos.fm
Limit: prompt construction
The critique found 23.5% of Instruct prompts problematic because they tested refusal behavior rather than genuine insecure-code propensity.Read 15 Oct 2026 · kairos.fm
Limit: static analysis
An independent critique says CyberSecEval’s insecure-code detector relies on 189 rules and cannot understand code context, creating false positives.Read 15 Oct 2026 · kairos.fm
Limit: superficial cues
Removing comments and identifiers from Autocomplete examples increased secure-code pass rates by 12.2 to 22.2 percentage points in the independent re-run.Read 15 Oct 2026 · kairos.fm
MITRE alignment
Meta says the benchmarks use industry guidance including CWE and MITRE ATT&CK.Read 15 Oct 2026 · github.com
MITRE and refusal tests
Its MITRE tests assess compliance with cyberattack requests, while False Refusal Rate tests measure incorrect refusals of borderline benign queries.Read 8 Oct 2026 · meta-llama.github.io
Model providers
The getting-started guide lists API support for OpenAI, Anyscale, and Together, and describes how to add custom provider support for self-hosted models.Read 8 Oct 2026 · meta-llama.github.io
Notable news: launch
Axios reported on December 7, 2023 that Meta released CyberSec Eval as part of Purple Llama to help developers assess insecure-code and cyberattack-assistance risks.Read 15 Oct 2026 · axios.com
Operating-system dependencies
Threat-intelligence report processing documents Poppler installation through apt on Ubuntu or Debian, pacman on Arch and Homebrew on macOS.Read 15 Oct 2026 · github.com
Original benchmark scope
The first CyberSecEval measured insecure-code generation and compliance with requests to assist cyberattacks.Read 15 Oct 2026 · arxiv.org
Praise: shared yardstick
An engineer’s review says CyberSecEval is valuable because a shared measurement yardstick is preferable to vendor-specific marketing numbers.Read 15 Oct 2026 · safeguard.sh
Prompt attacks
The related Prompt Guard model targets jailbreaks and indirect prompt injections contained in third-party data.Read 15 Oct 2026 · meta-llama.github.io
Prompt injection
Prompt injection benchmarks cover textual, multilingual textual, and visual attacks, with a judge LLM evaluating whether injected instructions succeeded.Read 8 Oct 2026 · meta-llama.github.io
Prompt-injection result
The CyberSecEval 2 paper reports successful prompt-injection rates of 26% to 41% for the tested models.Read 15 Oct 2026 · arxiv.org
Reproducibility
Inspect Evals fetches PurpleLlama datasets from a pinned commit and verifies archived report PDFs with SHA256 checksums.Read 15 Oct 2026 · github.com
Runtime requirement
The CyberSecEval 4 repository requires Python 3.10.Read 15 Oct 2026 · github.com
Secure code
Secure code tests measure insecure code suggestions in instruction and autocomplete settings, using an insecure code detector to evaluate responses.Read 8 Oct 2026 · meta-llama.github.io
Security metrics
The project provides metrics for cybersecurity risk, insecure-code suggestion frequency and assistance to cyberattackers.Read 15 Oct 2026 · github.com
Self-hosting
Self-hosted models are supported by extending llm.py with query, chat and system-prompt chat implementations.Read 15 Oct 2026 · github.com
Setup
The guide instructs users to create a Python virtual environment, install the listed requirements, and run benchmarks through a Python command-line module.Read 8 Oct 2026 · meta-llama.github.io
SOC benchmarks
CyberSOCEval, developed with CrowdStrike, includes malware analysis and threat intelligence reasoning benchmarks for defensive capabilities.Read 8 Oct 2026 · meta-llama.github.io
Supported provider APIs
The command-line runner accepts OpenAI, Google, Anthropic, Llama and Together model specifications, including custom OpenAI-compatible base URLs.Read 15 Oct 2026 · github.com

Plans and prices

CyberSecEval 4 Free Open source benchmark suite Read 8 Oct 2026 · github.com

Every CyberSecEval plan and price →

Strong points and limits

Strong points

  • Free under the MIT license
  • Tests cover textual, multilingual, and visual injection
  • Includes malware-analysis and threat-intelligence benchmarks
  • Supports API, Linux, and self-hosted use

Limits

  • AutoPatch recommends at least 80 CPUs and 8 TB storage
  • Apple Silicon MacBooks are not currently supported

Who it suits

It suits researchers and teams evaluating cybersecurity behavior in generative AI models, including those building with open models. AutoPatch users should have the recommended compute and storage available.

Verdict

CyberSecEval covers a broad range of security behaviors, from insecure code suggestions to injection and malicious prompts. Its AutoPatch resource requirements and Apple Silicon limitation matter for teams planning that benchmark.

Where Specifiction places it

Alternatives to CyberSecEvalAll alternatives →