# capitalone/DataProfiler

What's in your data? Extract schema, statistics and entities from datasets

Repository: https://github.com/capitalone/DataProfiler
Canonical: https://ross.abutalabs.com/products/dataprofiler
Homepage: https://capitalone.github.io/DataProfiler
Language: Python
License: Apache-2.0
License Family: permissive
Topics: python, privacy, pii, npi, nlp, data-science, gdpr, data-analysis, data-labels, avro, dataprofiling, sensitive-data, security, pandas, csv, tabular-data, dataset, network-data, graph-data, machine-learning
Last push: 2026-08-26T15:47:52+00:00

## Health v2 (maintenance only)
Score: 79/100 (v2, computed 2026-09-03T02:20:16.233290+00:00)
- activity 99, release rhythm 41, longevity 100
- inputs: {"age_days": 2123, "days_push": 7, "days_rel": 399, "gap_med": 30.0, "n_releases_24m": 5}
- flags: none
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 1578, forks 187 (observed 2026-08-28T04:05:06.208163+00:00)

## What it is
DataProfiler is a Python library that loads CSV, AVRO, Parquet, JSON, text, or URL data into a pandas-compatible DataFrame and profiles it with a single command. It extracts schema, statistics, and entities, including sensitive data (PII/NPI) detection via a pre-trained deep learning model.

## Use cases
- profile a csv dataset to get schema and statistics
- detect PII and sensitive data in datasets for GDPR compliance
- automatically load csv, avro, parquet, or  files into a pandas dataframe
- identify entities like names, emails, and phone numbers in tabular or text data
- generate data quality reports for downstream monitoring
- profile graph or network data
- extend entity recognition with custom labels or regex pipelines

## When to choose
- you need automated data profiling with statistics and schema extraction
- you must scan datasets for PII/NPI to meet privacy or compliance requirements
- you want a pre-trained sensitive-data detection model without building your own NLP pipeline
- you work with heterogeneous file formats and want a unified pandas-compatible loader

## When to avoid
- you need a lightweight tool and cannot install TensorFlow or heavy ML dependencies
- you only need simple descriptive statistics already covered by pandas describe()
- you need real-time streaming profiling rather than batch analysis

## Facets
- artifact type: library
- maturity: active
- function: data-science, nlp, machine-learning, privacy, security, analytics
- domain: data-science, privacy, security, developer-tools
- platform: python, cross-platform
- tags: data-profiling, pii-detection, sensitive-data, schema-extraction, pandas, csv, avro, parquet, gdpr, entity-recognition, natural-language-processing

## Member repositories
- capitalone/DataProfiler (main) score 79

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:05:06.208163+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-30T03:56:54.761174+00:00, confidence not recorded.
  - readme: https://github.com/capitalone/DataProfiler (fetched 2026-08-28T04:05:06.208163+00:00, sha 77966b7ad765)
  - homepage: https://capitalone.github.io/DataProfiler (fetched 2026-08-29T11:27:15.588633+00:00, sha 901a5548453e)
- Data as of 2026-08-30T08:39:29.467469+00:00.