CyberSecEval vs Inferwall

CyberSecEval 4 is a benchmark suite for assessing cybersecurity vulnerabilities and defensive capabilities in large language models. Its tests measure model responses to cyberattack requests and whether borderline benign requests are incorrectly refused. Secure-code checks look for insecure suggestions in both instruction and autocomplete settings. Prompt-injection tests cover textual, multilingual textual, and visual attacks; a judge LLM evaluates whether injected instructions succeeded. The code-interpreter benchmark evaluates compliance with malicious prompts and sorts judged responses into three categories: extremely malicious, potentially malicious, or non-malicious. AutoPatchBench measures patch-generation agents on fuzzed C and C++ vulnerabilities, while CyberSOCEval, developed with CrowdStrike, covers malware analysis and threat-intelligence reasoning. The guide lists API support for OpenAI, Anyscale, and Together, and explains adding custom providers for self-hosted models. Setup uses a Python virtual environment, installed requirements, and a Python command-line module. The suite is free under the MIT license for research and commercial use. AutoPatch needs substantial resources: the page recommends at least 80 CPUs and 8 TB of storage.

Inferwall is a self-hosted AI firewall that scans inputs and outputs in LLM applications for threats including prompt injection, jailbreaks, data leakage, and unsafe content. It uses heuristic, machine learning, semantic, and configurable LLM judge detection layers. Deployment options are a Python SDK, standalone HTTP server, or transparent reverse proxy. Listed wrappers include OpenAI, Anthropic, LangChain, and FastAPI, with other stack integrations such as Ollama, Hugging Face, LlamaIndex, Pydantic, vLLM, and Mistral. Users can set allow, flag, or block thresholds by route and modify signatures and version policies under ~/.inferwall/. The Lite profile is about 8 MB; Standard adds about 730 MB of models, and Full adds the LLM-judge layer and requires LLM credentials. Inferwall says its Rust heuristic engine runs in under 0.3 ms at p99. The engine uses Apache 2.0; community signatures use CC BY-SA 4.0. The project cautions that detection is imperfect and positions the firewall as one part of defense in depth.