moj-analytical-services/splink

Health v2 · maintenance only

Adoption not part of the score

2364 stars · 259 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

Splink is a Python library for fast, scalable probabilistic record linkage and entity resolution, based on the Fellegi-Sunter model with unsupervised EM training. It deduplicates and links records lacking unique identifiers, running on DuckDB locally or Spark/Athena for 100M+ records.

Use cases

• deduplicate customer records without a unique ID
• link records across two datasets probabilistically
• entity resolution on millions of rows
• fuzzy match names and dates of birth
• train a record linkage model without labelled data
• scale data linkage to Spark or Athena

When to choose

• you need to deduplicate or link tabular data lacking unique identifiers
• you want unsupervised probabilistic matching with no training labels
• you need to scale linkage from a laptop to hundreds of millions of records
• you want interactive diagnostics to understand match quality

When to avoid

• your data has a reliable unique identifier - just join directly
• your columns are highly correlated or few in number
• you need deep-learning or embedding-based entity resolution
• your data is unstructured text rather than structured columns

Facets

library · maturity active

data-science etl search-engine data-science analytics big-data python cli cloud record-linkage entity-resolution deduplication probabilistic-matching fellegi-sunter duckdb spark fuzzy-matching em-algorithm data-engineering docker

Member repositories

For agents

markdown · JSON · MCP: product_card(name="moj-analytical-services/splink")