confident-ai/deepeval

Health v2 · maintenance only

Adoption not part of the score

17882 stars · 1858 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

DeepEval is an open-source Python framework for evaluating LLM applications with pytest-style unit tests and 50+ research-backed metrics including LLM-as-a-judge, RAG, agent, conversational, safety, and multimodal metrics. It supports end-to-end, trajectory-based, and component-level evals, synthetic dataset generation, and integrations with frameworks like LangChain, LlamaIndex, CrewAI, and OpenAI Agents.

Use cases

• evaluate rag pipeline faithfulness and hallucination
• unit test llm outputs in ci/cd with pytest
• evaluate ai agent trajectories and tool use
• score chatbot multi-turn conversation quality
• generate synthetic test datasets for llm edge cases
• run llm-as-a-judge metrics with custom criteria
• test llm safety for toxicity and bias
• trace and evaluate langchain or llamaindex apps

When to choose

• you need pytest-native LLM evaluation that runs in CI/CD
• you want ready-made research-backed metrics for RAG, agents, or chatbots
• you need to evaluate agent execution traces across popular orchestration frameworks
• you want local-first evaluation with optional cloud dashboards

When to avoid

• you need a general ML model evaluation library for classical ML tasks
• you want a hosted-only evaluation service without running code locally
• your project is not Python-based

Facets

framework · maturity active

testing machine-learning llm-inference rag agent-framework benchmarking data-generation cli large-language-models machine-learning chatbots developer-tools testing python cli cross-platform llm-evaluation llm-as-a-judge pytest evaluation-metrics synthetic-data ci-cd hallucination-detection multimodal-evaluation trajectory-evaluation ai-agents retrieval-augmented-generation

Member repositories

For agents

markdown · JSON · MCP: product_card(name="confident-ai/deepeval")