VRUNAI

Spec sheet

What it does

VRUNAI evaluates LLM agents across execution path, tool calls, and final outcomes.Read 7 Oct 2026 · vrunai.com

Intended users

The site presents VRUNAI for people evaluating LLM agents and comparing their behavior across providers.Read 7 Oct 2026 · vrunai.com

Privacy

The CLI runs on the user's machine and the site says its keys stay in the terminal; web app keys are stored in browser localStorage and sent directly to configured provider APIs.Read 7 Oct 2026 · vrunai.com

Support

The site links to GitHub for bug reports and feature requests.Read 7 Oct 2026 · vrunai.com

License

VRUNAI is open source under the AGPL-3.0 license.Read 7 Oct 2026 · vrunai.com

Agent definitions

Users define tools, mock data, conditional flows, and test scenarios in YAML without writing code.Read 7 Oct 2026 · vrunai.com

Backend and telemetry

The site states that VRUNAI has no accounts, backend, or telemetry and is fully client-side.Read 7 Oct 2026 · vrunai.com

Consistency

It can run each scenario multiple times and score how consistently the agent follows the same path.Read 7 Oct 2026 · vrunai.com

Cost tracking

It calculates cost per scenario and provider from token usage and includes model pricing and context window statistics.Read 7 Oct 2026 · vrunai.com

Interfaces

VRUNAI offers a terminal interface and a React web app, which can run locally or be tried in a hosted browser version.Read 7 Oct 2026 · vrunai.com

Local model support

The FAQ says it supports local inference through Ollama and custom API endpoints, including OpenAI-compatible APIs.Read 7 Oct 2026 · vrunai.com

Notable cost limit

VRUNAI itself has no usage limits, but users pay for LLM API calls to the providers they configure.Read 7 Oct 2026 · vrunai.com

Providers

The site lists OpenAI, Anthropic, Google, xAI, DeepSeek, Mistral, Ollama, and custom providers.Read 7 Oct 2026 · vrunai.com

Plans and prices

Every VRUNAI plan and price →

Where Specifiction places it

• AI Agent Evaluation Tools#27 of 287.0