# visual-layer/fastdup

fastdup is a powerful, free tool designed to rapidly generate valuable insights from image and video datasets. It helps enhance the quality of both images and labels, while significantly reducing data operation costs, all with unmatched scalability.

Repository: https://github.com/visual-layer/fastdup
Canonical: https://ross.abutalabs.com/products/fastdup
Homepage: https://docs.visual-layer.com/fastdup_docs_old/First%20Steps/getting-started
Language: Python
License: NOASSERTION
License Family: other
Topics: data-curation, dataset, deep-learning, image-duplicate-detection, machine-learning, novelty-detection, object-detection, outlier-detection, python, visual-search, data-augmentation, image-classification, image, image-classfication, image-processing, visualization-tools, image-analysis, visualization, image-similarity
Last push: 2026-08-23T12:48:46+00:00

## Health v2 (maintenance only)
Score: 67/100 (v2, computed 2026-09-02T17:46:02.011165+00:00)
- activity 99, release rhythm 8, longevity 100
- inputs: {"age_days": 1575, "days_push": 10, "days_rel": null, "gap_med": null, "n_releases_24m": 0}
- flags: no_license
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 1904, forks 93 (observed 2026-08-28T04:05:51.830203+00:00)

## What it is
fastdup is a free Python tool for rapidly analyzing image and video datasets to surface duplicates, outliers, broken, dark, bright, blurry, and mislabeled images, plus visual clusters. It scales to large datasets on CPU and helps improve dataset and label quality while cutting data operation costs.

## Use cases
- find duplicate images in a dataset
- detect outliers and broken images before training
- find mislabeled images in an image classification dataset
- cluster visually similar images for exploration
- clean and curate a large image or video dataset
- run visual similarity search over a photo collection
- reduce dataset storage by removing redundant frames

## When to choose
- you need to audit and clean large image/video datasets quickly on CPU
- you want duplicate, outlier, and quality issue detection in a few lines of Python
- you are preparing computer vision training data and want better labels

## When to avoid
- you need a permissively licensed tool for commercial redistribution (CC BY-NC-ND)
- you need a full GUI dataset management platform rather than a Python library
- you work with non-visual data like text or tabular datasets

## Facets
- artifact type: library
- maturity: active
- function: image-processing, machine-learning, data-science, search-engine
- domain: computer-vision, machine-learning, data-science, image-processing
- platform: python, windows
- tags: data-curation, duplicate-detection, outlier-detection, image-similarity, dataset-quality, visual-search, computer-vision, linux, macos

## Member repositories
- visual-layer/fastdup (main) score 67

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:05:51.830203+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-30T03:11:52.248040+00:00, confidence not recorded.
  - readme: https://github.com/visual-layer/fastdup (fetched 2026-08-28T04:05:51.830203+00:00, sha c980d865c9e1)
  - homepage: https://docs.visual-layer.com/fastdup_docs_old/First%20Steps/getting-started (fetched 2026-08-29T10:50:59.367734+00:00, sha db11b9b852e1)
  - site_page: https://docs.visual-layer.com/docs/introduction/key-concepts (fetched 2026-08-29T10:50:59.383921+00:00, sha 862049e86bf0)
  - site_page: https://docs.visual-layer.com/docs/release-notes/release-notes (fetched 2026-08-29T10:50:59.386111+00:00, sha 6cd4a095e916)
  - site_page: https://docs.visual-layer.com (fetched 2026-08-29T10:50:59.376993+00:00, sha c9a467b55d35)
  - site_page: https://docs.visual-layer.com/docs/introduction/introduction (fetched 2026-08-29T10:50:59.378735+00:00, sha 508c9eb2ced4)
  - site_page: https://docs.visual-layer.com/docs/self-hosting/System_requirements (fetched 2026-08-29T10:50:59.380393+00:00, sha f68dbf3f0fb1)
  - site_page: https://docs.visual-layer.com/docs/introduction/prerequisites (fetched 2026-08-29T10:50:59.382025+00:00, sha 6109cb6bb318)
  - site_page: https://docs.visual-layer.com/docs/quick-start/tutorial-create-dataset (fetched 2026-08-29T10:50:59.388594+00:00, sha 427b383ff7d4)
  - site_page: https://docs.visual-layer.com/docs/quick-start/understanding-clusters (fetched 2026-08-29T10:50:59.390407+00:00, sha 54a4bae91ade)
- Data as of 2026-08-30T08:39:29.467469+00:00.