prometheus-eval/prometheus-eval

Health v2 · maintenance only

Adoption not part of the score

1107 stars · 68 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

Prometheus-Eval is a Python library for evaluating LLM generation outputs using the Prometheus family of open evaluator models and GPT-4 as LLM judges. It provides evaluation pipelines, datasets like BiGGen-Bench, and trained judge models for absolute grading and pairwise comparison tasks.

Use cases

• evaluate llm responses with an open-source judge model
• score model outputs with llm-as-a-judge
• run pairwise comparisons between two llm answers
• benchmark my fine-tuned model against gpt-4 evaluation
• grade llm outputs against custom evaluation criteria
• build an automated evaluation pipeline for my chatbot
• use prometheus 2 with vllm for fast evaluation

When to choose

• you need reproducible, open-source LLM evaluation instead of closed APIs
• you want to run an LLM judge locally with vLLM
• you need absolute grading or pairwise ranking of model outputs
• you are doing LLM research on meta-evaluation or reward models

When to avoid

• you only need simple unit tests for code, not LLM output quality assessment
• you cannot host GPU inference and only want a hosted eval service
• you need evaluation of non-text modalities like images or audio

Facets

library · maturity active

machine-learning llm-inference benchmarking data-science large-language-models machine-learning developer-tools python llm-as-a-judge evaluation litellm vllm llmops prometheus gpt4 meta-evaluation natural-language-processing

Member repositories

For agents

markdown · JSON · MCP: product_card(name="prometheus-eval/prometheus-eval")