mattpocock/evalite

Health v2 · maintenance only

Adoption not part of the score

1669 stars · 102 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

Evalite is a TypeScript-native eval runner for testing LLM-powered applications, built on Vitest with a local dev server UI for exploring outputs, traces, and scores. It supports any LLM provider, scoring libraries like autoevals, and CI-friendly static HTML export with score thresholds.

Use cases

• evaluate llm app outputs in typescript
• run evals on my ai-powered app
• test chatbot responses against expected answers
• score llm outputs with levenshtein or llm-based scorers
• view llm eval traces in a local ui
• fail ci builds when eval scores drop below threshold
• compare outputs across different llm providers

When to choose

• you build LLM-powered apps in TypeScript and want a Vitest-familiar eval workflow
• you want a local-first eval UI without vendor lock-in
• you need evals that run in CI with score thresholds

When to avoid

• you need Python-based eval frameworks like promptfoo alternatives in Python
• you need large-scale distributed eval pipelines or hosted eval dashboards
• your project is not in the Node/TypeScript ecosystem

Facets

library · maturity active

testing e2e-testing benchmarking llm-inference developer-tools large-language-models machine-learning developer-tools testing artificial-intelligence cli llm-evals vitest eval-runner local-first ai-testing scoring nodejs typescript web-server

Member repositories

For agents

markdown · JSON · MCP: product_card(name="mattpocock/evalite")