# prometheus-eval/prometheus-eval

Evaluate your LLM's response with Prometheus and GPT4 💯

Repository: https://github.com/prometheus-eval/prometheus-eval
Canonical: https://ross.abutalabs.com/products/prometheus-eval
Language: Python
License: Apache-2.0
License Family: permissive
Topics: evaluation, litellm, llm, llmops, python, vllm, gpt4, llm-as-a-judge, llm-as-evaluator
Last push: 2025-04-25T03:58:37+00:00

## Health v2 (maintenance only)
Score: 23/100 (v2, computed 2026-09-02T17:46:02.011165+00:00)
- activity 18, release rhythm 8, longevity 61
- inputs: {"age_days": 867, "days_push": 495, "days_rel": null, "gap_med": null, "n_releases_24m": 0}
- flags: none
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 1107, forks 68 (observed 2026-08-28T04:03:36.634502+00:00)

## What it is
Prometheus-Eval is a Python library for evaluating LLM generation outputs using the Prometheus family of open evaluator models and GPT-4 as LLM judges. It provides evaluation pipelines, datasets like BiGGen-Bench, and trained judge models for absolute grading and pairwise comparison tasks.

## Use cases
- evaluate llm responses with an open-source judge model
- score model outputs with llm-as-a-judge
- run pairwise comparisons between two llm answers
- benchmark my fine-tuned model against gpt-4 evaluation
- grade llm outputs against custom evaluation criteria
- build an automated evaluation pipeline for my chatbot
- use prometheus 2 with vllm for fast evaluation

## When to choose
- you need reproducible, open-source LLM evaluation instead of closed APIs
- you want to run an LLM judge locally with vLLM
- you need absolute grading or pairwise ranking of model outputs
- you are doing LLM research on meta-evaluation or reward models

## When to avoid
- you only need simple unit tests for code, not LLM output quality assessment
- you cannot host GPU inference and only want a hosted eval service
- you need evaluation of non-text modalities like images or audio

## Facets
- artifact type: library
- maturity: active
- function: machine-learning, llm-inference, benchmarking, data-science
- domain: large-language-models, machine-learning, developer-tools
- platform: python
- tags: llm-as-a-judge, evaluation, litellm, vllm, llmops, prometheus, gpt4, meta-evaluation, natural-language-processing

## Member repositories
- prometheus-eval/prometheus-eval (main) score 23

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:03:36.634502+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-30T06:44:03.039000+00:00, confidence not recorded.
  - readme: https://github.com/prometheus-eval/prometheus-eval (fetched 2026-08-28T04:03:36.634502+00:00, sha 7a2c3b33eeb8)
- Data as of 2026-08-30T08:39:29.467469+00:00.