Skip to content
SPECIFICTION
Get listed

AgentClash

agentclash.dev

From $49/mo Free plan WebAPISelf-hosted
Visit agentclash.dev
Spec score 8.5 #10 of 28 in AI Agent Evaluation Tools

What it is

AgentClash is an open-source platform for evaluating AI agents on real tasks. It runs each agent in a fresh Firecracker microVM with isolated files and network, then removes the sandbox after the run. Multi-turn evaluations score tool choices, cost, latency, recovery, and final outcomes. YAML challenge packs define tools, policies, scoring rules, and starting state; available tools include file operations, data queries, HTTP, shell, and test runners. Scoring can combine deterministic, mathematical, behavioral, and language-model judges with configurable weights and consensus. Failed runs can become permanent regression tests. CI/CD checks run through GitHub Actions, webhooks, or the CLI and can fail builds when correctness, cost, latency, or evidence regresses. Adapters cover OpenAI, Anthropic, Gemini, xAI, Mistral, and OpenRouter. It offers API, web, and self-hosted use. Pricing is freemium: Free, or paid from $39/mo (annual).

Spec sheet

Purpose
AgentClash is an open-source AI-agent evaluation platform that runs agents on real tasks, scores outcomes, replays steps, and turns failures into regression tests.Read 1 Oct 2026 · agentclash.dev
Integrations
CI/CD integrations can run regression tests from GitHub Actions, a webhook, or the CLI and fail builds when correctness, cost, latency, or required evidence regresses.Read 1 Oct 2026 · agentclash.dev
Security
API keys, database credentials, and OAuth tokens are stored in a scoped secret vault and injected at tool-call time without appearing in prompts, traces, or replays.Read 1 Oct 2026 · agentclash.dev
Agent evaluation
It evaluates multi-turn agents that take actions in a real sandbox and scores tool choices, cost, latency, recovery, and the final result.Read 1 Oct 2026 · agentclash.dev
Documentation
The public documentation covers the CLI, local stack, Fleet eval sets, datasets, regression gates, multi-turn human takeover, security stress harnesses, and runtime components.Read 1 Oct 2026 · agentclash.dev
Knowledge sources
Knowledge sources include PDFs, wikis, Notion, codebases, and custom APIs, with provenance attached to retrieved facts.Read 1 Oct 2026 · agentclash.dev
Open source and hosting
AgentClash is MIT licensed, can be self-hosted as a full stack, or used against the hosted backend; its CLI installs from npm as the agentclash package.Read 1 Oct 2026 · agentclash.dev
Providers
First-class adapters support OpenAI, Anthropic, Gemini, xAI, Mistral, and OpenRouter, with more than 300 models available through OpenRouter.Read 1 Oct 2026 · agentclash.dev
Regression loop
When a model fails a challenge, AgentClash freezes the failing trace into a permanent test that future evaluations replay.Read 1 Oct 2026 · agentclash.dev
Sandboxing
Each agent runs in a fresh Firecracker microVM with an isolated filesystem and network, and the sandbox is torn down after the run.Read 1 Oct 2026 · agentclash.dev
Scoring
Runs combine deterministic, mathematical, behavioural, and LLM-based judges with configurable consensus aggregation and weights.Read 1 Oct 2026 · agentclash.dev
Tools
Agents can use file I/O, data queries, HTTP, shell, and test runners, with declarative YAML challenge packs defining tools, policy, scoring, and starting state.Read 1 Oct 2026 · agentclash.dev
Workloads
The product is positioned for coding, research, SRE, multi-step operations, codebase question answering, and support workloads.Read 1 Oct 2026 · agentclash.dev

Plans and prices

Free Free 1 workspace · 25 eval runs / month · up to 4 models per run · 7-day replay retention · BYO LLM API key · BYO E2B sandbox token · community support Read 1 Oct 2026 · agentclash.dev
Pro $49 a month Billed monthly; annual billing $39 / month ($468 / yr) 500 eval runs / workspace / month · up to 8 models per run · 30-day replay retention · hosted sandbox with included credit · private challenge packs · CI integration · 3 concurrent eval runs · email support < 1 business day Read 1 Oct 2026 · agentclash.dev
Team $100 a month Billed monthly; annual billing $80 / month ($960 / yr) 2,000 eval runs / workspace / month · up to 12 models per run · 90-day replay retention · 10 concurrent eval runs · multiple workspaces · workspace-level audit log · Slack notifications · priority email support < 4 business hours Read 1 Oct 2026 · agentclash.dev
Enterprise Priced on request Custom SSO / SAML · org-wide audit logs · unlimited replay retention · 99.9% uptime SLA · dedicated support channel · custom MSA / billing terms Read 1 Oct 2026 · agentclash.dev

Every AgentClash plan and price →

Strong points and limits

Strong points

  • Fresh isolated Firecracker microVM for every run
  • Scores cost, latency, recovery, and final results
  • Supports multiple model providers and OpenRouter
  • Failed traces become replayable regression tests
  • CI integrations can fail regressing builds

Limits

  • Free plan allows 25 evaluation runs monthly
  • Free plan requires your LLM API key
  • Free plan requires your E2B sandbox token

Who it suits

AgentClash suits teams evaluating multi-step agents for coding, research, SRE, operations, support, or codebase question answering. It also fits organizations needing self-hosting, regression gates, and CI integration.

Verdict

AgentClash combines sandboxed multi-turn evaluation with configurable scoring and regression testing. Its free tier provides a defined starting point, while hosted and self-hosted options cover different deployment needs.

Where Specifiction places it

Alternatives to AgentClashAll alternatives →