Skip to content
SPECIFICTION
Get listed

Inspect AI

inspect.aisi.org.uk

Free plan APIBrowser extension
Visit inspect.aisi.org.uk
Spec score 7.4 #19 of 29 in LLM Evaluation Tools

What it is

Inspect AI is a free, open-source, self-hosted framework for evaluating large language models. It covers coding, agentic tasks, reasoning, knowledge, behavior, and multimodal understanding. Evaluations combine datasets, solvers, and scorers, with more than 200 pre-built evaluations available. Inspect View provides web-based monitoring and visualization, while a VS Code extension supports log browsing, task execution, debugging, and option configuration. The framework has built-in support for more than 20 model providers and local inference through Hugging Face, vLLM, and SGLang. Documented provider groups include OpenAI, Anthropic, Google, AWS Bedrock, Azure AI, Ollama, and others. Standard tools cover web search, code execution, text editing, computer use, and web browsing, with MCP support. Agents from OpenAI Agents SDK, Pydantic AI, LangChain, Claude Code, Codex CLI, and Gemini CLI can be integrated. Untrusted model code can run in Docker, Kubernetes, Modal, Proxmox, Vagrant, and other systems through an extension API.

Spec sheet

Purpose
Inspect is an open-source framework for evaluating large language models across coding, agentic tasks, reasoning, knowledge, behavior, and multimodal understanding.Read 4 Oct 2026 · inspect.aisi.org.uk
Founded
2024Read 28 Sep 2026 · inspect.aisi.org.uk
Agent integrations
Inspect can integrate agents built with OpenAI Agents SDK, Pydantic AI, and LangChain, as well as CLI agents such as Claude Code, Codex CLI, and Gemini CLI.Read 4 Oct 2026 · inspect.aisi.org.uk
Editor integration
The Inspect VS Code extension provides tools for browsing evaluation logs, running and debugging tasks, and configuring options.Read 4 Oct 2026 · inspect.aisi.org.uk
Evaluation results
Inspect View is a web-based tool for monitoring and visualizing evaluations.Read 4 Oct 2026 · inspect.aisi.org.uk
Installation requirement
The getting-started instructions install Inspect from PyPI with pip install inspect-ai and say evaluation development and runs also require access to a model.Read 4 Oct 2026 · inspect.aisi.org.uk
Model integrations
Inspect has built-in support for more than 20 model providers and local inference with Hugging Face, vLLM, and SGLang.Read 4 Oct 2026 · inspect.aisi.org.uk
Provider examples
Documented provider groups include OpenAI, Anthropic, Google, AWS Bedrock, Azure AI, Hugging Face, vLLM, Ollama, and SGLang.Read 4 Oct 2026 · inspect.aisi.org.uk
Reusable components
Evaluations combine datasets, solvers, and scorers, and can use more than 200 pre-built evaluations.Read 4 Oct 2026 · inspect.aisi.org.uk
Sandboxing
The framework supports running untrusted model code in Docker, Kubernetes, Modal, Proxmox, Vagrant, and other systems through an extension API.Read 4 Oct 2026 · inspect.aisi.org.uk
Tools and MCP
Inspect includes standard tools for web search, code execution, text editing, computer use, and web browsing, and supports MCP tools.Read 4 Oct 2026 · inspect.aisi.org.uk

Plans and prices

Inspect AI Free Open-source framework for large language model evaluations Read 4 Oct 2026 · inspect.aisi.org.uk

Every Inspect AI plan and price →

Strong points and limits

Strong points

  • Open-source and free
  • Includes over 200 pre-built evaluations
  • Supports more than 20 model providers
  • VS Code extension aids debugging
  • Provides multiple sandboxing options

Limits

  • Self-hosted deployment is required
  • Evaluation runs require model access
  • Installation uses Python's pip
  • Sandbox systems require configuration

Who it suits

Inspect AI suits developers and teams building or comparing language-model evaluations, agents, and safety checks. It is especially relevant to users comfortable with self-hosted tooling and model-provider access.

Verdict

Inspect AI provides a broad, extensible evaluation framework with visualization, editor tooling, provider integrations, and sandboxing. Its free status is clear, but users must operate it themselves and supply access to a model.

Where Specifiction places it

Alternatives to Inspect AIAll alternatives →