Skip to content
SPECIFICTION
Get listed

CyberSecEval vs PII Firewall

CyberSecEval

meta-llama.github.io

PII Firewall

pii-firewall.com

Side by side

FigureCyberSecEvalPII Firewall
Price fromFreeFree
Free planYesYes
Free trial——
Open sourceYes—
WebYes—
macOSYes—
LinuxYesYes
APIYes—
Self-hostedYesYes
Plans published11
Spec score · AI Guardrail Software9.0 (#1)7.2 (#18)
Intended usersThe project describes itself as tools and evaluations intended to help the community build responsibly with open generative AI models.The maker highlights healthcare, finance, and legal use cases, including clinical NLP, fintech support automation, and legal document review.
LicenseCybersecurity evaluations and benchmarks in Purple Llama are licensed under MIT, while included Llama safeguard models use separate Llama Community licenses.The project is open source under the Apache 2.0 license.
PurposeCyberSecEval 4 is a benchmark suite for assessing cybersecurity vulnerabilities and defensive capabilities in large language models.Intercepts and anonymizes PII before it reaches an LLM, then restores original values in the response.

Plans

CyberSecEval

  • CyberSecEval 4Free

CyberSecEval pricing →

PII Firewall

  • Open sourceFree

PII Firewall pricing →

In their own words

CyberSecEval

CyberSecEval 4 is a benchmark suite for assessing cybersecurity vulnerabilities and defensive capabilities in large language models. Its tests measure model responses to cyberattack requests and whether borderline benign requests are incorrectly refused. Secure-code checks look for insecure suggestions in both instruction and autocomplete settings. Prompt-injection tests cover textual, multilingual textual, and visual attacks; a judge LLM evaluates whether injected instructions succeeded. The code-interpreter benchmark evaluates compliance with malicious prompts and sorts judged responses into three categories: extremely malicious, potentially malicious, or non-malicious. AutoPatchBench measures patch-generation agents on fuzzed C and C++ vulnerabilities, while CyberSOCEval, developed with CrowdStrike, covers malware analysis and threat-intelligence reasoning. The guide lists API support for OpenAI, Anyscale, and Together, and explains adding custom providers for self-hosted models. Setup uses a Python virtual environment, installed requirements, and a Python command-line module. The suite is free under the MIT license for research and commercial use. AutoPatch needs substantial resources: the page recommends at least 80 CPUs and 8 TB of storage.

Visit meta-llama.github.io

PII Firewall

PII Firewall is an open-source, self-hosted tool that intercepts personally identifiable information before it reaches an LLM, then restores original values in the response. The maker highlights healthcare, finance, and legal workflows, including clinical language processing, fintech support automation, and legal document review. Built-in profiles cover healthcare, finance, legal, and generic use, with custom profiles also available.

Detection options include regex, Presidio, GLiNER, Transformers, OpenAI Privacy Filter, hybrid setups, and custom backends. Users can choose to keep, pseudonymize, redact, generalize, mask, or hash detected information. The product says it supports 55+ languages with automatic language detection and locale-specific patterns. Integration examples include OpenAI, Anthropic, LangChain, FastAPI, streaming transport, Ollama, LiteLLM, and arbitrary HTTP LLM endpoints. Tenant-scoped token mappings and a forget operation support deleting vault mappings for a thread or case. Each call produces a trace with detected entity types, confidence scores, and replacements. The project uses the Apache 2.0 license and provides a Python package; its regex-only base install does not require ML dependencies. The maker says hash is for analytics workflows and is not used in built-in presets.

Visit pii-firewall.com