JudgmentLabs/judgeval

Health v2 · maintenance only

Adoption not part of the score

1068 stars · 103 forks observed · 2026-10-19

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

Judgeval is an open-source Python SDK for LLM agent improvement, providing OpenTelemetry-based tracing and prompt-based agent-judge evaluation. It detects production failures, scores live traffic, and validates fixes against real traces before shipping.

Use cases

• trace llm agent calls and token usage
• evaluate agent behavior with custom llm judges
• monitor production agent traffic for regressions
• replay historical traces to validate agent fixes
• alert on agent failure patterns in slack
• instrument openai or langgraph agents with auto-instrumentation

When to choose

• you ship llm agents to production and need failure detection and root-cause triage
• you want opentelemetry-compatible tracing with minimal code changes
• you need structured, repeatable evaluation of agent behaviors over time

When to avoid

• you only need simple offline unit tests without tracing or monitoring
• you require a fully self-hosted solution with no external API key or service dependency
• your stack is not python-based

Facets

library · maturity active

monitoring tracing testing llm-inference agent-framework prompt-engineering alerting analytics artificial-intelligence large-language-models developer-tools monitoring machine-learning python cross-platform llm-evaluation llm-observability agent-judges opentelemetry evals agent-monitoring langgraph langchain llama-index ai-agents

Member repositories

For agents

markdown · JSON · MCP: product_card(name="JudgmentLabs/judgeval")