Maluuba/nlg-eval

Health v2 · maintenance only

Flags: no_license

Adoption not part of the score

1391 stars · 226 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

A Python library and CLI tool that computes unsupervised automated metrics (BLEU, METEOR, ROUGE, CIDEr, SPICE, SkipThought, and embedding-based similarities) for evaluating Natural Language Generation output. It takes hypothesis and reference files as input and outputs metric scores, commonly used for dialogue and captioning evaluation.

Use cases

• evaluate generated dialogue responses against references
• compute BLEU and ROUGE scores for machine translation output
• score image captions with CIDEr and SPICE
• compare NLG model outputs with embedding-based similarity metrics
• benchmark text generation models with standard NLG metrics
• evaluate task-oriented dialogue response generation

When to choose

• you need multiple standard NLG metrics from a single tool
• evaluating dialogue systems or captioning models in Python
• you want reproducible metric scores for research papers

When to avoid

• you need human evaluation or learned metrics like BERTScore
• you cannot install Java 1.8+ (required for METEOR/SPICE)
• you need actively maintained tooling with modern Python support
• evaluating LLM outputs where GPT-based judges are preferred

Facets

library · maturity maintenance

nlp machine-learning benchmarking cli machine-learning data-science python cli cross-platform nlg-evaluation bleu meteor rouge cider spice skip-thoughts text-generation-metrics dialogue-evaluation natural-language-processing

Member repositories

For agents

markdown · JSON · MCP: product_card(name="Maluuba/nlg-eval")