amazon-science/RAGChecker

Health v2 · maintenance only

Adoption not part of the score

1126 stars · 92 forks observed · 2026-10-07

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

RAGChecker is an automatic evaluation framework for diagnosing Retrieval-Augmented Generation (RAG) systems. It provides holistic and diagnostic metrics for retrieval and generation components, using claim-level entailment for fine-grained analysis, plus a benchmark dataset.

Use cases

• evaluate my rag pipeline quality
• diagnose whether retrieval or generation is failing in my rag system
• benchmark rag systems across domains
• measure faithfulness and relevance of rag answers
• compare rag checker metrics with human judgments
• run claim-level entailment evaluation on rag outputs

When to choose

• you need fine-grained, component-level diagnosis of a RAG pipeline
• you want automated metrics correlated with human judgments
• you need a benchmark dataset for RAG evaluation research

When to avoid

• you need to build or serve a RAG system rather than evaluate one
• you need a lightweight evaluation without LLM-based entailment costs
• you need non-English evaluation support

Facets

framework · maturity active

rag benchmarking testing nlp llm-inference cli large-language-models machine-learning developer-tools python cli cross-platform rag-evaluation diagnostic-metrics claim-level-entailment evaluation-framework retrieval-evaluation generator-evaluation retrieval-augmented-generation natural-language-processing

Member repositories

For agents

markdown · JSON · MCP: product_card(name="amazon-science/RAGChecker")