AIRTA Red Team vs Crucible

AIRTA Red Team is a self-hosted tool for authorized adversarial security assessments of LLMs and LLM-driven applications. It builds test suites from security playbooks, discovers target components and authentication, runs attacks, assesses outcomes, and exports reports. Listed playbooks include OWASP LLM, OWASP Agent, MITRE ATLAS, jailbreak, system prompt exfiltration, and custom scenarios. Strategies include zero-shot, multi-shot, few-shot, iterative, multimodal, jailbreak, and chain-of-thought testing. Targets can be reached through a Playwright browser runner or direct HTTP API transports, including file uploads and tool calls. It can run locally in Docker, behind a firewall, in CI, or against external endpoints for authorized black-box assessments. Runs record prompts, uploads, responses, and tool calls in attack_log.json; pipeline_report.json summarizes categories and severity. Results can be exported to AIRTA Systems for review and regression tracking. The free plan has no license fee or subscription, but users must supply an LLM API key. Assessments focus on observable runtime behavior, not source-code static analysis. The repository says the tool is deprecated and no longer maintained.

Crucible is a self-hosted security scanner for testing AI agents before production. It checks attack behavior such as prompt injection, goal hijacking, jailbreaks, unsafe output, and tool misuse. Instead of reviewing generated text alone, it evaluates tool calls, side effects, and outcomes. The site lists 11 modules and 153 attack vectors, covering eight of ten OWASP Agentic risks, with two in development. Crucible runs 90 attack payloads in under 60 seconds and produces an A–F report with severity, OWASP mapping, and reproduction steps. It supports LangChain, CrewAI, AutoGen, raw MCP endpoints, and agents with HTTP or MCP interfaces. The pip-installed CLI supports Python 3.10+. Local scanning needs no signup and has no telemetry by default. Community is free; Team and Enterprise add monitoring, dashboards, support, SSO, audit logs, deployment options, and custom modules.