Skip to content
SPECIFICTION
Get listed

Rhesis AI

rhesis.ai

Free plan WebLinuxAPISelf-hosted
Visit rhesis.ai
Spec score 7.5 #23 of 30 in AI LLM Evaluation Tools

What it is

Rhesis AI is an open-source collaboration layer for developing and evaluating AI agents. Domain experts, engineers, and product managers can work in a shared space to review test cases and agent versions, annotate them, and record expectations. Teams can generate test sets from application requirements or constraints, including single-turn prompts and multi-turn conversations, then run them against an application endpoint and evaluate responses using configured metrics. Results are saved in test runs. Rhesis Tracing uses OpenTelemetry to capture language-model calls, tool invocations, and retrieval operations in test or production mode. Evaluation options include offline evaluation, online trace metrics, LLM-as-a-judge, custom metrics, and adversarial red-teaming. Model providers connect through LiteLLM, including cloud and local options such as OpenAI, Anthropic, Google Gemini, Mistral, Ollama, and vLLM. Teams can use DeepEval and Ragas metrics or import Garak probes. A REST API supports test sets, runs, results, and trace inspection for integrations and CI/CD. Rhesis offers managed, Docker-local, and self-hosted use; it says it is developed in Germany and hosted in the EU. Both listed free editions cost $0.

Spec sheet

Product
Rhesis is an open-source collaboration layer that brings domain-expert knowledge into AI agent development.Read 30 Sep 2026 · rhesis.ai
Intended users
Rhesis describes itself as built for experts, engineers, and product managers working on AI agents.Read 30 Sep 2026 · rhesis.ai
Support
The getting-started page points users to in-app support, documentation, a support email, GitHub, and Discord.Read 30 Sep 2026 · docs.rhesis.ai
Headquarters
Potsdam, GermanyRead 28 Sep 2026 · rhesis.ai
API and CI/CD
The REST API supports managing test sets, triggering test runs, fetching results, and inspecting traces for custom integrations and CI/CD pipelines.Read 30 Sep 2026 · docs.rhesis.ai
Deployment and data location
Rhesis can run as a managed service, locally with Docker, or self-hosted; the site says it is developed in Germany and hosted in the EU.Read 30 Sep 2026 · docs.rhesis.ai
Evaluation integrations
Rhesis supports DeepEval and Ragas metrics and can import Garak probes as test sets for adversarial scanning.Read 30 Sep 2026 · docs.rhesis.ai
Evaluations
Rhesis runs test sets against an application endpoint, evaluates responses using configured metrics, and records results in test runs.Read 30 Sep 2026 · docs.rhesis.ai
Model integrations
Rhesis supports cloud and local model providers through LiteLLM, including OpenAI, Anthropic, Google Gemini, Mistral, Cohere, Groq, Together AI, Ollama, and vLLM.Read 30 Sep 2026 · docs.rhesis.ai
Security testing
Rhesis integrates Garak to probe applications for issues such as prompt injection, data leakage, and harmful content generation.Read 30 Sep 2026 · docs.rhesis.ai
Team workflow
Domain experts, engineers, and product managers can share a workspace for reviews, annotations, and expectations attached to test cases and agent versions.Read 30 Sep 2026 · rhesis.ai
Test generation
Rhesis can generate test sets from application requirements or constraints, including single-turn prompts and multi-turn conversations.Read 30 Sep 2026 · docs.rhesis.ai
Tool integrations
The integrations page lists Notion, GitHub, Jira, GitLab, Asana, Azure DevOps, Linear, and Shortcut as tool connections.Read 30 Sep 2026 · docs.rhesis.ai
Tracing
Rhesis Tracing uses OpenTelemetry to capture LLM calls, tool invocations, and retrieval operations in test or production mode.Read 30 Sep 2026 · docs.rhesis.ai

Plans and prices

Rhesis Cloud Free Free Start free on Rhesis Cloud with a monthly allowance; exact allowance not stated on the opened pricing page. Read 30 Sep 2026 · rhesis.ai
Self-hosted open source edition Free Nothing metered. Read 30 Sep 2026 · rhesis.ai

Every Rhesis AI plan and price →

Strong points and limits

Strong points

  • Generates single-turn and multi-turn test sets
  • Captures model, tool, and retrieval traces
  • Supports cloud and local model providers
  • Free cloud and self-hosted editions are listed

Limits

  • Managed and self-hosted use require deployment setup
  • Hosted in the EU, according to the site

Who it suits

Rhesis suits experts, engineers, and product managers building or evaluating AI agents. It also fits teams that need API-based test workflows or managed and self-hosted deployment options.

Verdict

Rhesis combines shared review workflows with testing, tracing, and security evaluation options.

Where Specifiction places it

Alternatives to Rhesis AIAll alternatives →