ekzhu/datasketch

Health v2 · maintenance only

Adoption not part of the score

2959 stars · 318 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

datasketch is a Python library of probabilistic data structures (MinHash, Weighted MinHash, HyperLogLog, HyperLogLog++) and indexes (MinHash LSH, LSHBloom, LSH Forest, LSH Ensemble, HNSW) for processing and searching very large datasets with little loss of accuracy. It supports scalable storage layers via Redis and Cassandra and optional GPU acceleration.

Use cases

• find near-duplicate documents at scale
• estimate Jaccard similarity between large sets
• estimate cardinality of huge streams with HyperLogLog
• build a top-k similarity search index with LSH Forest
• deduplicate records or emails in a large dataset
• query sets by containment with LSH Ensemble
• nearest neighbor search with HNSW

When to choose

• you need fast approximate set similarity or cardinality estimation on big data
• you want sub-linear query time for Jaccard threshold or top-k queries
• you need scalable sketch indexes backed by Redis or Cassandra

When to avoid

• you need exact similarity or exact distinct counts
• your datasets are small enough for brute-force computation
• you need general-purpose vector embeddings search beyond what HNSW here provides

Facets

library · maturity stable

search-engine data-science benchmarking data-science big-data machine-learning python minhash locality-sensitive-hashing hyperloglog lsh-forest lsh-ensemble hnsw probabilistic-data-structures jaccard-similarity cardinality-estimation deduplication algorithms search

Member repositories

For agents

markdown · JSON · MCP: product_card(name="ekzhu/datasketch")