# JudgmentLabs/judgeval

The Continuous-Improvement Stack for Agents. Our environment data and evals power agent improvement and monitoring.

Repository: https://github.com/JudgmentLabs/judgeval
Canonical: https://ross.abutalabs.com/products/judgeval
Homepage: https://judgmentlabs.ai/
Language: Python
License: Apache-2.0
License Family: permissive
Topics: langchain, langgraph, llama-index, llm, llm-evaluation, llm-observability, open-source, openai, prompt-engineering, agent, agentic-ai, agents, grpo, reinforcement-learning, rl
Last push: 2026-08-24T00:18:45+00:00

## Health v2 (maintenance only)
Score: 84/100 (v2, computed 2026-09-03T02:39:23.370411+00:00)
- activity 99, release rhythm 84, longevity 48
- inputs: {"age_days": 677, "days_push": 10, "days_rel": 30, "gap_med": 1.5, "n_releases_24m": 109}
- flags: none
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 1058, forks 96 (observed 2026-08-28T04:03:25.088647+00:00)

## What it is
Judgeval is an open-source Python SDK for LLM agent improvement, providing OpenTelemetry-based tracing and prompt-based agent-judge evaluation. It detects production failures, scores live traffic, and validates fixes against real traces before shipping.

## Use cases
- trace llm agent calls and token usage
- evaluate agent behavior with custom llm judges
- monitor production agent traffic for regressions
- replay historical traces to validate agent fixes
- alert on agent failure patterns in slack
- instrument openai or langgraph agents with auto-instrumentation

## When to choose
- you ship llm agents to production and need failure detection and root-cause triage
- you want opentelemetry-compatible tracing with minimal code changes
- you need structured, repeatable evaluation of agent behaviors over time

## When to avoid
- you only need simple offline unit tests without tracing or monitoring
- you require a fully self-hosted solution with no external API key or service dependency
- your stack is not python-based

## Facets
- artifact type: library
- maturity: active
- function: monitoring, tracing, testing, llm-inference, agent-framework, prompt-engineering, alerting, analytics
- domain: artificial-intelligence, large-language-models, developer-tools, monitoring, machine-learning
- platform: python, cross-platform
- tags: llm-evaluation, llm-observability, agent-judges, opentelemetry, evals, agent-monitoring, langgraph, langchain, llama-index, ai-agents

## Member repositories
- JudgmentLabs/judgeval (main) score 84

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:03:25.088647+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-30T06:57:32.162204+00:00, confidence not recorded.
  - readme: https://github.com/JudgmentLabs/judgeval (fetched 2026-08-28T04:03:25.088647+00:00, sha d57c75e9035a)
  - homepage: https://judgmentlabs.ai/ (fetched 2026-08-29T12:59:27.710007+00:00, sha 9eae22a0514b)
  - registry_pypi: https://pypi.org/pypi/judgeval/json (fetched 2026-08-29T12:59:27.713621+00:00, sha 275347e1572e)
- Data as of 2026-08-30T08:39:29.467469+00:00.