Basilisk vs CyberSecEval

Basilisk is an open-source framework for authorized security testing, vulnerability research, and academic study of AI and LLM systems. It includes 32 attack modules across eight categories mapped to the OWASP LLM Top 10. Its Smart Prompt Evolution engine combines 15 mutation operators, five crossover strategies, and multi-signal fitness scoring. Multi-turn modules examine prompt cultivation, authority escalation, and sycophancy exploitation, with guardrail drift monitoring. Basilisk connects to providers including OpenAI, Anthropic, Google, xAI Grok, Groq, Azure OpenAI, GitHub Models, and Ollama, plus custom HTTP or WebSocket endpoints. Reports can be exported as HTML, SARIF 2.1.0, JSON, Markdown, or PDF. GitHub Actions and GitLab CI integrations use SARIF output and can fail pipelines on critical findings. Install options include desktop packages for Windows, macOS, and Linux, pip, Docker, and source. The CLI requires Python 3.11 or higher and an API key for the provider being tested. Prompt evolution, payload mutation, and report generation run locally. It is free under AGPL-3.0. Users must have explicit written authorization before testing systems they do not own.

CyberSecEval 4 is a benchmark suite for assessing cybersecurity vulnerabilities and defensive capabilities in large language models. Its tests measure model responses to cyberattack requests and whether borderline benign requests are incorrectly refused. Secure-code checks look for insecure suggestions in both instruction and autocomplete settings. Prompt-injection tests cover textual, multilingual textual, and visual attacks; a judge LLM evaluates whether injected instructions succeeded. The code-interpreter benchmark evaluates compliance with malicious prompts and sorts judged responses into three categories: extremely malicious, potentially malicious, or non-malicious. AutoPatchBench measures patch-generation agents on fuzzed C and C++ vulnerabilities, while CyberSOCEval, developed with CrowdStrike, covers malware analysis and threat-intelligence reasoning. The guide lists API support for OpenAI, Anyscale, and Together, and explains adding custom providers for self-hosted models. Setup uses a Python virtual environment, installed requirements, and a Python command-line module. The suite is free under the MIT license for research and commercial use. AutoPatch needs substantial resources: the page recommends at least 80 CPUs and 8 TB of storage.