langchain-ai/openevals

Health v2 · maintenance only

Adoption not part of the score

1178 stars · 119 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

OpenEvals is a library of readymade evaluators for testing LLM applications, including LLM-as-judge evaluators and reference-based/reference-free checks. It is available in both Python and TypeScript and is intended as a starting point for building custom application-specific evals.

Use cases

• evaluate llm app outputs for correctness
• run llm-as-judge evaluations
• test rag pipeline answer quality
• check if model responses are concise
• grade llm outputs against reference answers
• build custom evaluators for ai agents

When to choose

• you need off-the-shelf evaluators for LLM app testing
• you want LLM-as-judge scoring with minimal setup
• you use LangChain/LangSmith and want compatible evals

When to avoid

• you need agent-trajectory-specific evaluation (use agentevals instead)
• you need deterministic unit tests without an LLM judge
• you need a full eval orchestration platform rather than evaluator primitives

Facets

library · maturity active

testing llm-inference prompt-engineering benchmarking large-language-models machine-learning developer-tools testing python cross-platform llm-evaluation llm-as-judge evals rag-evaluation langchain nodejs

Member repositories

For agents

markdown · JSON · MCP: product_card(name="langchain-ai/openevals")