Skip to content
SPECIFICTION
Get listed

RAGChecker

github.com

Free plan Self-hosted
Visit github.com
Spec score 7.1 #24 of 29 in LLM Evaluation Tools

Spec sheet

Purpose
RAGChecker is an automatic evaluation framework for assessing and diagnosing retrieval-augmented generation systems.Read 4 Oct 2026 · github.com
License
The project is licensed under Apache-2.0.Read 4 Oct 2026 · github.com
Benchmark
The repository says its benchmark was released and links a guide for using it.Read 4 Oct 2026 · github.com
Fine-grained evaluation
Its fine-grained evaluation uses claim-level entailment operations.Read 4 Oct 2026 · github.com
Generator metrics
Documented generator metrics include context utilization, noise sensitivity, hallucination, self-knowledge, and faithfulness.Read 4 Oct 2026 · github.com
Input requirement
For each query, the only required annotation is the ground-truth answer; generated responses and retrieved context are also accepted in the input format.Read 4 Oct 2026 · github.com
Installation
The maker documents installation with pip install ragchecker and requires downloading the spaCy en_core_web_sm model in its quick start.Read 4 Oct 2026 · github.com
Integration
The README documents an integration with LlamaIndex for evaluating RAG applications built with it.Read 4 Oct 2026 · github.com
Model setup
The example pipeline uses configurable extractor and checker models and documents an AWS Bedrock Llama 3 70B example.Read 4 Oct 2026 · github.com
Pipeline metrics
It provides overall metrics for the RAG pipeline and diagnostic metrics for retrieval and generation components.Read 4 Oct 2026 · github.com
Retriever metrics
Documented retriever metrics include claim recall and context precision.Read 4 Oct 2026 · github.com
Security reporting
The contribution guide asks users to report potential security issues to AWS/Amazon Security through its vulnerability reporting page rather than public GitHub issues.Read 4 Oct 2026 · github.com
Support and contributions
The project directs bug reports and feature requests to its GitHub issue tracker and welcomes pull requests.Read 4 Oct 2026 · github.com
Usage
The framework can be run through a CLI or Python API, with selectable metric groups or all metrics.Read 4 Oct 2026 · github.com

Plans and prices

RAGChecker Free Apache-2.0 licensed open-source framework · installable with pip Read 4 Oct 2026 · github.com

Every RAGChecker plan and price →

Where Specifiction places it

Alternatives to RAGCheckerAll alternatives →