# confident-ai/deepeval

The LLM Evaluation Framework

Repository: https://github.com/confident-ai/deepeval
Canonical: https://ross.abutalabs.com/products/deepeval
Homepage: https://deepeval.com
Language: Python
License: Apache-2.0
License Family: permissive
Topics: evaluation-metrics, evaluation-framework, llm-evaluation, llm-evaluation-framework, llm-evaluation-metrics, python
Last push: 2026-08-26T07:53:19+00:00

## Health v2 (maintenance only)
Score: 95/100 (v2, computed 2026-09-03T02:20:16.233290+00:00)
- activity 99, release rhythm 99, longevity 79
- inputs: {"age_days": 1119, "days_push": 7, "days_rel": 9, "gap_med": 11.5, "n_releases_24m": 27}
- flags: none
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 17882, forks 1858 (observed 2026-08-28T04:11:20.504047+00:00)

## What it is
DeepEval is an open-source Python framework for evaluating LLM applications with pytest-style unit tests and 50+ research-backed metrics including LLM-as-a-judge, RAG, agent, conversational, safety, and multimodal metrics. It supports end-to-end, trajectory-based, and component-level evals, synthetic dataset generation, and integrations with frameworks like LangChain, LlamaIndex, CrewAI, and OpenAI Agents.

## Use cases
- evaluate rag pipeline faithfulness and hallucination
- unit test llm outputs in ci/cd with pytest
- evaluate ai agent trajectories and tool use
- score chatbot multi-turn conversation quality
- generate synthetic test datasets for llm edge cases
- run llm-as-a-judge metrics with custom criteria
- test llm safety for toxicity and bias
- trace and evaluate langchain or llamaindex apps

## When to choose
- you need pytest-native LLM evaluation that runs in CI/CD
- you want ready-made research-backed metrics for RAG, agents, or chatbots
- you need to evaluate agent execution traces across popular orchestration frameworks
- you want local-first evaluation with optional cloud dashboards

## When to avoid
- you need a general ML model evaluation library for classical ML tasks
- you want a hosted-only evaluation service without running code locally
- your project is not Python-based

## Facets
- artifact type: framework
- maturity: active
- function: testing, machine-learning, llm-inference, rag, agent-framework, benchmarking, data-generation, cli
- domain: large-language-models, machine-learning, chatbots, developer-tools, testing
- platform: python, cli, cross-platform
- tags: llm-evaluation, llm-as-a-judge, pytest, evaluation-metrics, synthetic-data, ci-cd, hallucination-detection, multimodal-evaluation, trajectory-evaluation, ai-agents, retrieval-augmented-generation

## Member repositories
- confident-ai/deepeval (main) score 95

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:11:20.504047+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-29T17:02:17.830814+00:00, confidence not recorded.
  - readme: https://github.com/confident-ai/deepeval (fetched 2026-08-28T04:11:20.504047+00:00, sha d023b67ac363)
  - homepage: https://deepeval.com (fetched 2026-08-29T08:00:33.974835+00:00, sha dd1c68efed67)
  - site_page: https://deepeval.com/docs/introduction (fetched 2026-08-29T08:00:33.977984+00:00, sha bd70c6cc717c)
  - site_page: https://deepeval.com/integrations (fetched 2026-08-29T08:00:33.980263+00:00, sha cd3537a63c2e)
  - site_page: https://deepeval.com/changelog (fetched 2026-08-29T08:00:33.982072+00:00, sha ae121dd15282)
  - site_page: https://deepeval.com/integrations/frameworks/langchain (fetched 2026-08-29T08:00:33.984128+00:00, sha b1df154f8041)
  - site_page: https://deepeval.com/integrations/frameworks/llamaindex (fetched 2026-08-29T08:00:33.986484+00:00, sha 9c533558eb44)
  - site_page: https://deepeval.com/integrations/frameworks/crewai (fetched 2026-08-29T08:00:33.989047+00:00, sha 77064da92efd)
  - site_page: https://deepeval.com/integrations/frameworks/openai-agents (fetched 2026-08-29T08:00:33.992163+00:00, sha d8a0ef65d6a6)
  - site_page: https://deepeval.com/integrations/frameworks/langgraph (fetched 2026-08-29T08:00:33.995092+00:00, sha 8ee0acb42e1f)
- Data as of 2026-08-30T08:39:29.467469+00:00.