AIRTA Red Team vs PyRIT

AIRTA Red Team is a self-hosted tool for authorized adversarial security assessments of LLMs and LLM-driven applications. It builds test suites from security playbooks, discovers target components and authentication, runs attacks, assesses outcomes, and exports reports. Listed playbooks include OWASP LLM, OWASP Agent, MITRE ATLAS, jailbreak, system prompt exfiltration, and custom scenarios. Strategies include zero-shot, multi-shot, few-shot, iterative, multimodal, jailbreak, and chain-of-thought testing. Targets can be reached through a Playwright browser runner or direct HTTP API transports, including file uploads and tool calls. It can run locally in Docker, behind a firewall, in CI, or against external endpoints for authorized black-box assessments. Runs record prompts, uploads, responses, and tool calls in attack_log.json; pipeline_report.json summarizes categories and severity. Results can be exported to AIRTA Systems for review and regression tracking. The free plan has no license fee or subscription, but users must supply an LLM API key. Assessments focus on observable runtime behavior, not source-code static analysis. The repository says the tool is deprecated and no longer maintained.

PyRIT is a free, open-source framework for automated and human-led red teaming of generative AI systems, aimed at assessing their security and safety. It supports single-turn and multi-turn attack strategies, including Crescendo, TAP, and Skeleton Key. Scenarios combine strategies and datasets for repeatable assessments of content harms, psychosocial risks, data leakage, and other objectives. Assessments can run through a command-line scanner, an interactive shell, the CoPyRIT graphical interface, or custom workflows. Documented targets include OpenAI, Azure, Anthropic, Google, Hugging Face, custom HTTP endpoints and WebSockets, and web apps tested with Playwright. Its modular components include targets, converters, scorers, memory, datasets, attacks, and scenarios. Converters can encode, obfuscate, translate, or semantically alter prompts, and transform among text, images, audio, video, and files. Scorers can return Boolean or normalized 0.0–1.0 scores using LLMs, Azure AI Content Safety, or custom logic. Built-in memory tracks conversations, scores, and attack results using SQLite or Azure SQL. PyRIT is self-hosted and requires a Python environment and configured AI endpoints.