Skip to content
SPECIFICTION
Get listed

garak vs Inspect AI

garak

garak.ai

Inspect AI

inspect.aisi.org.uk

Side by side

FiguregarakInspect AI
Price fromFreeFree
Free planYesYes
Free trial——
Open source——
WindowsYes—
macOSYes—
LinuxYes—
API—Yes
Self-hostedYes—
Plans published11
Spec score · LLM Evaluation Tools7.8 (#12)7.4 (#19)
Purposegarak is an LLM vulnerability scanner that probes language models and systems using them for security weaknesses.Inspect is an open-source framework for evaluating large language models across coding, agentic tasks, reasoning, knowledge, behavior, and multimodal understanding.

Plans

garak

  • FreeFree

garak pricing →

Inspect AI

  • Inspect AIFree

Inspect AI pricing →

In their own words

garak

garak is a command-line scanner for probing language models and systems that use them for security weaknesses. Its checks cover issues such as hallucination, data leakage, prompt injection, misinformation, toxicity generation, and jailbreaks. Probes try to elicit failures, while detectors inspect responses for patterns such as unsafe content or a failure to refuse. A standard scan can run without supervision and produce a report, with scan settings available for customization. Integrations include OpenAI, Hugging Face, Cohere, Replicate, AWS Bedrock, LiteLLM, REST-accessible models, and custom Python integrations. Hugging Face models can run locally or be accessed through an inference API or private endpoint. Output includes screen results, a detailed report log, a hit log, and a debug log. garak assesses text output only. It is developed in Linux and OSX and can be installed with pip or from source; listed platforms include Linux, macOS, Windows, and self-hosted use. It is free, with a free plan and no free trial. Listed capabilities include custom metrics, LLM-as-a-judge, and safety evaluations.

Visit garak.ai

Inspect AI

Inspect AI is a free, open-source, self-hosted framework for evaluating large language models. It covers coding, agentic tasks, reasoning, knowledge, behavior, and multimodal understanding. Evaluations combine datasets, solvers, and scorers, with more than 200 pre-built evaluations available. Inspect View provides web-based monitoring and visualization, while a VS Code extension supports log browsing, task execution, debugging, and option configuration. The framework has built-in support for more than 20 model providers and local inference through Hugging Face, vLLM, and SGLang. Documented provider groups include OpenAI, Anthropic, Google, AWS Bedrock, Azure AI, Ollama, and others. Standard tools cover web search, code execution, text editing, computer use, and web browsing, with MCP support. Agents from OpenAI Agents SDK, Pydantic AI, LangChain, Claude Code, Codex CLI, and Gemini CLI can be integrated. Untrusted model code can run in Docker, Kubernetes, Modal, Proxmox, Vagrant, and other systems through an extension API.

Visit inspect.aisi.org.uk