# Maluuba/nlg-eval

Evaluation code for various unsupervised automated metrics for Natural Language Generation.

Repository: https://github.com/Maluuba/nlg-eval
Canonical: https://ross.abutalabs.com/products/nlg-eval
Homepage: http://arxiv.org/abs/1706.09799
Language: Python
License: NOASSERTION
License Family: other
Topics: natural-language-generation, natural-language-processing, nlg, nlp, evaluation, bleu, bleu-score, meteor, cider, rouge, rouge-l, task-oriented-dialogue, machine-translation, dialog, dialogue, skip-thought-vectors, skip-thoughts
Last push: 2024-08-20T16:26:23+00:00

## Health v2 (maintenance only)
Score: 23/100 (v2, computed 2026-09-03T02:20:16.233290+00:00)
- activity 0, release rhythm 8, longevity 100
- inputs: {"age_days": 3354, "days_push": 743, "days_rel": null, "gap_med": null, "n_releases_24m": 0}
- flags: no_license
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 1391, forks 226 (observed 2026-08-28T04:04:35.940318+00:00)

## What it is
A Python library and CLI tool that computes unsupervised automated metrics (BLEU, METEOR, ROUGE, CIDEr, SPICE, SkipThought, and embedding-based similarities) for evaluating Natural Language Generation output. It takes hypothesis and reference files as input and outputs metric scores, commonly used for dialogue and captioning evaluation.

## Use cases
- evaluate generated dialogue responses against references
- compute BLEU and ROUGE scores for machine translation output
- score image captions with CIDEr and SPICE
- compare NLG model outputs with embedding-based similarity metrics
- benchmark text generation models with standard NLG metrics
- evaluate task-oriented dialogue response generation

## When to choose
- you need multiple standard NLG metrics from a single tool
- evaluating dialogue systems or captioning models in Python
- you want reproducible metric scores for research papers

## When to avoid
- you need human evaluation or learned metrics like BERTScore
- you cannot install Java 1.8+ (required for METEOR/SPICE)
- you need actively maintained tooling with modern Python support
- evaluating LLM outputs where GPT-based judges are preferred

## Facets
- artifact type: library
- maturity: maintenance
- function: nlp, machine-learning, benchmarking, cli
- domain: machine-learning, data-science
- platform: python, cli, cross-platform
- tags: nlg-evaluation, bleu, meteor, rouge, cider, spice, skip-thoughts, text-generation-metrics, dialogue-evaluation, natural-language-processing

## Member repositories
- Maluuba/nlg-eval (main) score 23

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:04:35.940318+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-30T04:39:35.121200+00:00, confidence not recorded.
  - readme: https://github.com/Maluuba/nlg-eval (fetched 2026-08-28T04:04:35.940318+00:00, sha 927eb7b36463)
  - homepage: http://arxiv.org/abs/1706.09799 (fetched 2026-08-29T11:54:40.830899+00:00, sha 4aaab4c1334b)
  - site_page: https://info.arxiv.org/about/donate.html (fetched 2026-08-29T11:54:40.839984+00:00, sha cca9c3a11c56)
  - site_page: https://info.arxiv.org/about/ourmembers.html (fetched 2026-08-29T11:54:40.843323+00:00, sha 47cbc55ff1de)
  - site_page: https://info.arxiv.org/about (fetched 2026-08-29T11:54:40.845030+00:00, sha a1f16f915a9a)
  - site_page: https://info.arxiv.org/labs/index.html (fetched 2026-08-29T11:54:40.841757+00:00, sha b14a8d05a0ec)
- Data as of 2026-08-30T08:39:29.467469+00:00.