# ekzhu/datasketch

MinHash, LSH, LSH Forest, Weighted MinHash, HyperLogLog, HyperLogLog++, LSH Ensemble and HNSW

Repository: https://github.com/ekzhu/datasketch
Canonical: https://ross.abutalabs.com/products/datasketch
Homepage: https://ekzhu.github.io/datasketch
Language: Python
License: MIT
License Family: permissive
Topics: python, lsh-forest, jaccard-similarity, hyperloglog, lsh, minhash, weighted-quantiles, top-k, search, data-sketches, data-summary, lsh-ensemble, locality-sensitive-hashing, hnsw
Last push: 2026-08-09T07:37:02+00:00

## Health v2 (maintenance only)
Score: 91/100 (v2, computed 2026-09-03T02:20:16.233290+00:00)
- activity 96, release rhythm 80, longevity 100
- inputs: {"age_days": 4185, "days_push": 24, "days_rel": 59, "gap_med": 64.5, "n_releases_24m": 5}
- flags: none
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 2959, forks 318 (observed 2026-08-28T04:07:32.292707+00:00)

## What it is
datasketch is a Python library of probabilistic data structures (MinHash, Weighted MinHash, HyperLogLog, HyperLogLog++) and indexes (MinHash LSH, LSHBloom, LSH Forest, LSH Ensemble, HNSW) for processing and searching very large datasets with little loss of accuracy. It supports scalable storage layers via Redis and Cassandra and optional GPU acceleration.

## Use cases
- find near-duplicate documents at scale
- estimate Jaccard similarity between large sets
- estimate cardinality of huge streams with HyperLogLog
- build a top-k similarity search index with LSH Forest
- deduplicate records or emails in a large dataset
- query sets by containment with LSH Ensemble
- nearest neighbor search with HNSW

## When to choose
- you need fast approximate set similarity or cardinality estimation on big data
- you want sub-linear query time for Jaccard threshold or top-k queries
- you need scalable sketch indexes backed by Redis or Cassandra

## When to avoid
- you need exact similarity or exact distinct counts
- your datasets are small enough for brute-force computation
- you need general-purpose vector embeddings search beyond what HNSW here provides

## Facets
- artifact type: library
- maturity: stable
- function: search-engine, data-science, benchmarking
- domain: data-science, big-data, machine-learning
- platform: python
- tags: minhash, locality-sensitive-hashing, hyperloglog, lsh-forest, lsh-ensemble, hnsw, probabilistic-data-structures, jaccard-similarity, cardinality-estimation, deduplication, algorithms, search

## Member repositories
- ekzhu/datasketch (main) score 91

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:07:32.292707+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-30T07:32:10.742943+00:00, confidence not recorded.
  - readme: https://github.com/ekzhu/datasketch (fetched 2026-08-28T04:07:32.292707+00:00, sha a34ca531d33a)
  - homepage: https://ekzhu.github.io/datasketch (fetched 2026-08-29T09:47:31.765481+00:00, sha b3d8791fa28e)
  - site_page: https://ekzhu.com/datasketch/documentation.html (fetched 2026-08-29T09:47:31.775294+00:00, sha 52b67c163b9c)
  - registry_pypi: https://pypi.org/pypi/datasketch/json (fetched 2026-08-29T09:47:31.778919+00:00, sha 39104584c2e4)
- Data as of 2026-08-30T08:39:29.467469+00:00.