CyberSecEval vs PII Firewall
CyberSecEval
meta-llama.github.ioPII Firewall
pii-firewall.comSide by side
| Figure | CyberSecEval | PII Firewall |
|---|---|---|
| Price from | Free | Free |
| Free plan | Yes | Yes |
| Free trial | — | — |
| Open source | Yes | — |
| Web | Yes | — |
| macOS | Yes | — |
| Linux | Yes | Yes |
| API | Yes | — |
| Self-hosted | Yes | Yes |
| Plans published | 1 | 1 |
| Spec score · AI Guardrail Software | 9.0 (#1) | 7.2 (#18) |
| Intended users | The project describes itself as tools and evaluations intended to help the community build responsibly with open generative AI models. | The maker highlights healthcare, finance, and legal use cases, including clinical NLP, fintech support automation, and legal document review. |
| License | Cybersecurity evaluations and benchmarks in Purple Llama are licensed under MIT, while included Llama safeguard models use separate Llama Community licenses. | The project is open source under the Apache 2.0 license. |
| Purpose | CyberSecEval 4 is a benchmark suite for assessing cybersecurity vulnerabilities and defensive capabilities in large language models. | Intercepts and anonymizes PII before it reaches an LLM, then restores original values in the response. |
Plans
In their own words
CyberSecEval
CyberSecEval 4 is a benchmark suite for assessing cybersecurity vulnerabilities and defensive capabilities in large language models. Its tests measure model responses to cyberattack requests and whether borderline benign requests are incorrectly refused. Secure-code checks look for insecure suggestions in both instruction and autocomplete settings. Prompt-injection tests cover textual, multilingual textual, and visual attacks; a judge LLM evaluates whether injected instructions succeeded. The code-interpreter benchmark evaluates compliance with malicious prompts and sorts judged responses into three categories: extremely malicious, potentially malicious, or non-malicious. AutoPatchBench measures patch-generation agents on fuzzed C and C++ vulnerabilities, while CyberSOCEval, developed with CrowdStrike, covers malware analysis and threat-intelligence reasoning. The guide lists API support for OpenAI, Anyscale, and Together, and explains adding custom providers for self-hosted models. Setup uses a Python virtual environment, installed requirements, and a Python command-line module. The suite is free under the MIT license for research and commercial use. AutoPatch needs substantial resources: the page recommends at least 80 CPUs and 8 TB of storage.
PII Firewall
PII Firewall is an open-source, self-hosted tool that intercepts personally identifiable information before it reaches an LLM, then restores original values in the response. The maker highlights healthcare, finance, and legal workflows, including clinical language processing, fintech support automation, and legal document review. Built-in profiles cover healthcare, finance, legal, and generic use, with custom profiles also available.
Detection options include regex, Presidio, GLiNER, Transformers, OpenAI Privacy Filter, hybrid setups, and custom backends. Users can choose to keep, pseudonymize, redact, generalize, mask, or hash detected information. The product says it supports 55+ languages with automatic language detection and locale-specific patterns. Integration examples include OpenAI, Anthropic, LangChain, FastAPI, streaming transport, Ollama, LiteLLM, and arbitrary HTTP LLM endpoints. Tenant-scoped token mappings and a forget operation support deleting vault mappings for a thread or case. Each call produces a trace with detected entity types, confidence scores, and replacements. The project uses the Apache 2.0 license and provides a Python package; its regex-only base install does not require ML dependencies. The maker says hash is for analytics workflows and is not used in built-in presets.