rlancemartin/auto-evaluator

Health v2 · maintenance only

Flags: no_releases no_license

Adoption not part of the score

1101 stars · 92 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

A lightweight Streamlit-based evaluation tool for LLM question-answering chains built with LangChain. It auto-generates question-answer pairs from user documents, runs a configurable QA chain against them, and uses an LLM to score responses across different chain configurations.

Use cases

• evaluate llm question-answering chains
• auto-generate eval questions from documents
• compare rag chain configurations
• score qa responses with an llm grader
• benchmark retrieval chunking and embedding settings
• build an evaluation dashboard for langchain qa apps

When to choose

• you use LangChain QA/RAG chains and want automated evaluation
• you lack a labeled eval set and want LLM-generated question-answer pairs
• you want to compare chunking, embedding, and retriever configurations side by side

When to avoid

• you need a general-purpose LLM eval framework beyond QA chains
• you require a maintained library with a stable API - the project appears inactive since 2023
• you cannot share documents with OpenAI or Anthropic APIs
• you need deterministic, human-annotated evaluation metrics

Facets

application · maturity maintenance

machine-learning llm-inference rag testing benchmarking large-language-models machine-learning developer-tools python self-hosted langchain llm-evaluation question-answering streamlit openai qa-chains auto-evaluation retrieval-augmented-generation web-server

Member repositories

For agents

markdown · JSON · MCP: product_card(name="rlancemartin/auto-evaluator")