braintrustdata/autoevals

Health v2 · maintenance only

Adoption not part of the score

1009 stars · 78 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

Autoevals is a Python and TypeScript library for automatically evaluating AI model outputs using LLM-as-a-judge, heuristic, and statistical methods. It offers prebuilt evaluators like Factuality plus support for custom model-graded prompts.

Use cases

• evaluate llm outputs automatically
• score ai model responses for factuality
• run llm-as-a-judge evaluations
• compute bleu or levenshtein scores for generated text
• build custom model-graded eval prompts
• test chatbot answer quality

When to choose

• you need ready-made evaluators for subjective LLM tasks like factuality or safety
• you want a lightweight library usable from Python or TypeScript
• you want to tweak or debug evaluation prompts on individual examples

When to avoid

• you need a full experiment-tracking or observability platform rather than just evaluators
• you cannot call an LLM API, since model-graded evals require one
• you only need deterministic string metrics with no LLM judging

Facets

library · maturity active

machine-learning llm-inference testing benchmarking nlp large-language-models machine-learning developer-tools artificial-intelligence python cross-platform llm-evaluation llm-as-a-judge evals ai-quality braintrust nodejs

Member repositories

For agents

markdown · JSON · MCP: product_card(name="braintrustdata/autoevals")