Basilisk vs PyRIT

Basilisk is an open-source framework for authorized security testing, vulnerability research, and academic study of AI and LLM systems. It includes 32 attack modules across eight categories mapped to the OWASP LLM Top 10. Its Smart Prompt Evolution engine combines 15 mutation operators, five crossover strategies, and multi-signal fitness scoring. Multi-turn modules examine prompt cultivation, authority escalation, and sycophancy exploitation, with guardrail drift monitoring. Basilisk connects to providers including OpenAI, Anthropic, Google, xAI Grok, Groq, Azure OpenAI, GitHub Models, and Ollama, plus custom HTTP or WebSocket endpoints. Reports can be exported as HTML, SARIF 2.1.0, JSON, Markdown, or PDF. GitHub Actions and GitLab CI integrations use SARIF output and can fail pipelines on critical findings. Install options include desktop packages for Windows, macOS, and Linux, pip, Docker, and source. The CLI requires Python 3.11 or higher and an API key for the provider being tested. Prompt evolution, payload mutation, and report generation run locally. It is free under AGPL-3.0. Users must have explicit written authorization before testing systems they do not own.

PyRIT is a free, open-source framework for automated and human-led red teaming of generative AI systems, aimed at assessing their security and safety. It supports single-turn and multi-turn attack strategies, including Crescendo, TAP, and Skeleton Key. Scenarios combine strategies and datasets for repeatable assessments of content harms, psychosocial risks, data leakage, and other objectives. Assessments can run through a command-line scanner, an interactive shell, the CoPyRIT graphical interface, or custom workflows. Documented targets include OpenAI, Azure, Anthropic, Google, Hugging Face, custom HTTP endpoints and WebSockets, and web apps tested with Playwright. Its modular components include targets, converters, scorers, memory, datasets, attacks, and scenarios. Converters can encode, obfuscate, translate, or semantically alter prompts, and transform among text, images, audio, video, and files. Scorers can return Boolean or normalized 0.0–1.0 scores using LLMs, Azure AI Content Safety, or custom logic. Built-in memory tracks conversations, scores, and attack results using SQLite or Azure SQL. PyRIT is self-hosted and requires a Python environment and configured AI endpoints.