marcotcr/checklist

Health v2 · maintenance only

Flags: no_releases

Adoption not part of the score

2050 stars · 211 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

CheckList is a Python library for behavioral testing of NLP models, implementing the ACL 2020 paper 'Beyond Accuracy: Behavioral Testing of NLP models with CheckList'. It lets users generate test suites (invariance, directional, and minimum-functionality tests) for tasks like sentiment analysis, QA, and paraphrase detection, including support for Hugging Face transformer pipelines.

Use cases

• test nlp models beyond accuracy metrics
• generate behavioral test suites for sentiment classifiers
• test huggingface transformer pipelines
• create invariance and directional tests for qa models
• find weaknesses in nlp models with perturbation tests

When to choose

• you want systematic behavioral testing of an NLP model rather than just benchmark accuracy
• you need to generate perturbation-based test data for classification or QA models
• you work in Jupyter and want interactive test visualizations

When to avoid

• you need testing for non-NLP machine learning models
• you want a maintained tool with active development - the project is in maintenance mode
• you rely on visualizations in JupyterLab or Colab, which are unsupported

Facets

library · maturity maintenance

testing nlp fuzzing data-generation machine-learning testing python cross-platform behavioral-testing nlp-model-evaluation acl-2020 jupyter-notebooks huggingface natural-language-processing

Member repositories

For agents

markdown · JSON · MCP: product_card(name="marcotcr/checklist")