dedupeio/dedupe

Health v2 · maintenance only

Flags: no_releases

Adoption not part of the score

4507 stars · 576 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

Dedupe is a Python library that uses machine learning to perform fuzzy matching, deduplication, and entity resolution on structured data. It learns from human-labeled training examples to automatically derive optimal matching weights and blocking rules, scaling to large databases.

Use cases

• remove duplicate entries from a spreadsheet of names and addresses
• link a customer list to an order history list without shared unique IDs
• identify which campaign contribution records were made by the same person despite name variations
• deduplicate messy CSV files
• perform entity resolution across large structured datasets
• train a custom fuzzy matching model from human-labeled examples

When to choose

• you need to deduplicate or link structured records with fuzzy, inconsistent text fields
• you have (or can collect) some human-labeled training pairs and want the model to learn matching rules automatically
• you want a well-established, MIT-licensed Python library that runs on a laptop without special infrastructure

When to avoid

• you need exact matching on clean keys or unique identifiers - simple database joins suffice
• you need real-time streaming deduplication rather than batch processing
• you want a turnkey GUI or hosted service rather than writing Python code (consider Dedupe.io instead)

Facets

library · maturity stable

machine-learning nlp data-science etl data-science machine-learning developer-tools python cross-platform fuzzy-matching record-linkage entity-resolution deduplication active-learning clustering data-engineering

Member repositories

For agents

markdown · JSON · MCP: product_card(name="dedupeio/dedupe")