# marcotcr/checklist

Beyond Accuracy: Behavioral Testing of NLP models with CheckList

Repository: https://github.com/marcotcr/checklist
Canonical: https://ross.abutalabs.com/products/checklist
Language: Jupyter Notebook
License: MIT
License Family: permissive
Last push: 2024-01-09T01:46:07+00:00

## Health v2 (maintenance only)
Score: 32/100 (v2, computed 2026-09-03T02:20:16.233290+00:00)
- activity 0, release rhythm 35, longevity 100
- inputs: {"age_days": 2368, "days_push": 968, "days_rel": null, "gap_med": null, "n_releases_24m": 0}
- flags: no_releases
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 2050, forks 211 (observed 2026-08-28T04:06:09.126370+00:00)

## What it is
CheckList is a Python library for behavioral testing of NLP models, implementing the ACL 2020 paper 'Beyond Accuracy: Behavioral Testing of NLP models with CheckList'. It lets users generate test suites (invariance, directional, and minimum-functionality tests) for tasks like sentiment analysis, QA, and paraphrase detection, including support for Hugging Face transformer pipelines.

## Use cases
- test nlp models beyond accuracy metrics
- generate behavioral test suites for sentiment classifiers
- test huggingface transformer pipelines
- create invariance and directional tests for qa models
- find weaknesses in nlp models with perturbation tests

## When to choose
- you want systematic behavioral testing of an NLP model rather than just benchmark accuracy
- you need to generate perturbation-based test data for classification or QA models
- you work in Jupyter and want interactive test visualizations

## When to avoid
- you need testing for non-NLP machine learning models
- you want a maintained tool with active development - the project is in maintenance mode
- you rely on visualizations in JupyterLab or Colab, which are unsupported

## Facets
- artifact type: library
- maturity: maintenance
- function: testing, nlp, fuzzing, data-generation
- domain: machine-learning, testing
- platform: python, cross-platform
- tags: behavioral-testing, nlp-model-evaluation, acl-2020, jupyter-notebooks, huggingface, natural-language-processing

## Member repositories
- marcotcr/checklist (main) score 32

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:06:09.126370+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-30T02:57:36.247411+00:00, confidence not recorded.
  - readme: https://github.com/marcotcr/checklist (fetched 2026-08-28T04:06:09.126370+00:00, sha 1659f5869924)
  - registry_pypi: https://pypi.org/pypi/checklist/json (fetched 2026-08-29T10:38:03.193160+00:00, sha 73ed2b051784)
- Data as of 2026-08-30T08:39:29.467469+00:00.