Skip to content
SPECIFICTION
Get listed

OpenAgent Eval

openagenthq.github.io

WindowsmacOSLinuxAPISelf-hosted
Visit openagenthq.github.io
Spec score 7.3 #25 of 28 in AI Agent Evaluation Tools

Spec sheet

Purpose
OpenAgent Eval is an open-source, local-first framework for evaluating RAG systems and AI agents.Read 8 Oct 2026 · openagenthq.github.io
Integrations
Documented LLM providers include OpenAI, Anthropic, Gemini, Groq, OpenRouter, Ollama, and a mock provider.Read 8 Oct 2026 · openagenthq.github.io
Support
The project directs users to GitHub Issues for bugs and GitHub Discussions for questions and ideas.Read 8 Oct 2026 · github.com
License
The project is licensed under Apache License 2.0.Read 7 Oct 2026 · github.com
Data handling
The project says it runs on the user's machine without dashboards or accounts and that data never leaves the laptop.Read 8 Oct 2026 · openagenthq.github.io
Embedders
Sentence Transformers is the listed built-in embedder, and custom embedders can be added through provider base classes.Read 7 Oct 2026 · openagenthq.github.io
Extensibility
A plugin architecture supports custom metrics, providers, and report generators.Read 7 Oct 2026 · openagenthq.github.io
Framework support
The framework describes itself as compatible with LangChain, LlamaIndex, and custom RAG pipelines.Read 8 Oct 2026 · openagenthq.github.io
LLM integrations
Documented LLM providers include OpenAI, Anthropic, Gemini, Groq, OpenRouter, Ollama, and a mock provider.Read 7 Oct 2026 · openagenthq.github.io
Local operation
The project says it runs on the user's machine without telemetry or required network calls.Read 7 Oct 2026 · openagenthq.github.io
Metrics
Built-in metrics cover retrieval, generation, latency, and token cost.Read 8 Oct 2026 · openagenthq.github.io
Notable dependency limit
Some retrievers and embedders require extra dependencies, with an `[all]` package extra offered for the full set.Read 8 Oct 2026 · openagenthq.github.io
Notable limitation
Some retrievers and embedders require extra dependencies, such as chromadb, sentence-transformers, faiss-cpu, or qdrant-client.Read 7 Oct 2026 · openagenthq.github.io
Offline use
Built-in mock providers can run a complete evaluation locally without network calls or credentials.Read 8 Oct 2026 · github.com
Reports
Reports can be output in terminal, Markdown, HTML, or JSON formats, with built-in failure analysis.Read 8 Oct 2026 · openagenthq.github.io
Requirements
The installation guide lists Python 3.11 or later as a requirement.Read 8 Oct 2026 · openagenthq.github.io
Retriever integrations
Documented retrievers include Chroma, Qdrant, Pinecone, Weaviate, FAISS, PGVector, Elasticsearch, BM25, Memory, HTTP, and Mock.Read 7 Oct 2026 · openagenthq.github.io
Retrievers
Documented retriever providers include Chroma, Qdrant, Pinecone, Weaviate, FAISS, PGVector, Elasticsearch, BM25, Memory, HTTP, and Mock.Read 8 Oct 2026 · openagenthq.github.io
Usage
It provides the `oaeval` command-line interface and a typed Python SDK for embedding evaluations in test suites.Read 8 Oct 2026 · openagenthq.github.io
Ways to use
It provides the `oaeval` command-line interface and a Python SDK for embedding evaluations in test suites.Read 7 Oct 2026 · openagenthq.github.io

Where Specifiction places it

Alternatives to OpenAgent EvalAll alternatives →