# awslabs/deequ

Deequ is a library built on top of Apache Spark for defining "unit tests for data", which measure data quality in large datasets.

Repository: https://github.com/awslabs/deequ
Canonical: https://ross.abutalabs.com/products/deequ
Language: Scala
License: Apache-2.0
License Family: permissive
Topics: dataquality, spark, unit-testing, scala
Last push: 2026-07-21T17:55:47+00:00

## Health v2 (maintenance only)
Score: 93/100 (v2, computed 2026-09-03T02:20:16.233290+00:00)
- activity 93, release rhythm 89, longevity 100
- inputs: {"age_days": 2948, "days_push": 43, "days_rel": 76, "gap_med": 5.0, "n_releases_24m": 13}
- flags: none
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 3643, forks 584 (observed 2026-08-28T04:08:12.883792+00:00)

## What it is
Deequ is a Scala library built on Apache Spark for defining 'unit tests for data' that measure data quality in large datasets. It computes data quality metrics and verifies constraints on tabular data before it feeds downstream systems or machine learning pipelines.

## Use cases
- validate data quality in large spark dataframes
- define unit tests for datasets
- detect data errors before loading into ml pipelines
- compute data quality metrics on big data
- check constraints like completeness and uniqueness on tables
- profile large datasets for quality issues

## When to choose
- you already run Apache Spark and need scalable data quality checks
- you want declarative constraint verification on large tabular datasets
- you need to catch data errors before feeding consuming systems or ML models

## When to avoid
- your data is small and fits in pandas or a single machine
- you don't use Spark and can't justify its overhead
- you need streaming or non-tabular (e.g., unstructured text) data validation

## Facets
- artifact type: library
- maturity: active
- function: testing, data-science, etl, monitoring
- domain: data-science, big-data, analytics
- platform: jvm, python
- tags: data-quality, apache-spark, unit-tests-for-data, data-validation, pydeequ, data-engineering, spark

## Member repositories
- awslabs/deequ (main) score 93

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:08:12.883792+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-29T18:31:46.404538+00:00, confidence not recorded.
  - readme: https://github.com/awslabs/deequ (fetched 2026-08-28T04:08:12.883792+00:00, sha ceddc246f629)
- Data as of 2026-08-30T08:39:29.467469+00:00.