awslabs/deequ

Health v2 · maintenance only

Adoption not part of the score

3643 stars · 584 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

Deequ is a Scala library built on Apache Spark for defining 'unit tests for data' that measure data quality in large datasets. It computes data quality metrics and verifies constraints on tabular data before it feeds downstream systems or machine learning pipelines.

Use cases

• validate data quality in large spark dataframes
• define unit tests for datasets
• detect data errors before loading into ml pipelines
• compute data quality metrics on big data
• check constraints like completeness and uniqueness on tables
• profile large datasets for quality issues

When to choose

• you already run Apache Spark and need scalable data quality checks
• you want declarative constraint verification on large tabular datasets
• you need to catch data errors before feeding consuming systems or ML models

When to avoid

• your data is small and fits in pandas or a single machine
• you don't use Spark and can't justify its overhead
• you need streaming or non-tabular (e.g., unstructured text) data validation

Facets

library · maturity active

testing data-science etl monitoring data-science big-data analytics jvm python data-quality apache-spark unit-tests-for-data data-validation pydeequ data-engineering spark

Member repositories

For agents

markdown · JSON · MCP: product_card(name="awslabs/deequ")