shubhamwagh / fastdup

fastdup is a powerful free tool designed to rapidly extract valuable insights from your image & video datasets. Assisting you to increase your dataset images & labels quality and reduce your data operations costs at an unparalleled scale.

Manage, Clean & Curate Visual Data - Fast and at Scale

An unsupervised and free tool for image and video dataset analysis.
Explore the docs »
Features · Report Bug · Read Blog · Quickstart · Enterprise Edition · About us

🔥 We've released fastdup V1.0! View the release notes here.

What's Included

fastdup lets you identify -

Additional features -

Why fastdup?

• Quality: Find and remove anomalies and outliers from your dataset, including duplicates and similar images and videos at a large scale.
• Cost: Reduce data operation costs by intelligently sampling high-quality or novel datasets before labeling and assessing labeled data quality.
• Scale: fastdup's C++ graph engine is highly efficient and can handle up to 400M images on a single CPU machine.

Setting up

Prerequisites

Supported Python versions:

Supported operating systems:

Installation

Option 1 - Install fastdup via PyPI:

# upgrade pip to its latest version
pip install -U pip

# install fastdup
pip install fastdup
    
# Alternatively, use explicit python version (XX)
python3.XX -m pip install fastdup 

Option 2 - Install fastdup via an Ubuntu 20.04 Docker image on DockerHub:

docker pull karpadoni/fastdup-ubuntu-20.04

Detailed installation instructions and common errors here.

Getting Started

Run fastdup with only 3 lines of code.

Visualize the result.

Here are the 8 lines of code you'll need in most cases.

import fastdup

fd = fastdup.create(work_dir, images_dir)
fd.run(nearest_neighbors_k=5, cc_threshold=0.96)

fd.vis.duplicates_gallery()    # create a visual gallery of found duplicates
fd.vis.outliers_gallery()      # create a visual gallery of anomalies
fd.vis.component_gallery()     # create a visualization of connected components
fd.vis.stats_gallery()         # create a visualization of images statistics (for example blur)
fd.vis.similarity_gallery()    # create a gallery of similar images

View the API docs here.

Learn from Examples

	Quick Dataset Analysis: In this example, we show how you can quickly analyze a dataset for potential issues. Using a few lines of code you can identify duplicates, outliers, dark/bright/blurry images and cluster similar images. If you're new, start here.
	Cleaning Image Dataset: In this tutorial, we use fastdup to clean and analyze a food-101 dataset. The cleaning process includes identifying and removing duplicates, broken images, outliers, as well as the darkest, brightest, and blurriest images.
	Analysing Labeled Image Classification Dataset: In this tutorial, we analyze the Imagenette dataset, a 10-class, 13k image subset of ImageNet. We show how to use fastdup to analyze the dataset for similarity and outlier images.
	Preparing an object dataset for training: In this tutorial we will load and analyze the mini-coco dataset which is labeled with bounding boxes and classes. Using fastdup, we discover duplicates, outliers, and possible mislabeled bounding boxes.

Getting Help

Get help from the fastdup team or community members via the following channels -

• Slack.
• GitHub issues.
• Discussion forum.

Community Contributions

The following are community-contributed blog posts about fastdup -

• Master Data Integrity to Clean Your Computer Vision Datasets
• fastdup: A Powerful Tool to Manage, Clean & Curate Visual Data at Scale on Your CPU - For Free.
• Clean Up Your Digital Life: Simplify Your Photo Organization and Say Goodbye to Photo Clutter
• The weighty significance of data cleanliness — or as I like to call it, “cleanliness is next to model-ness” — cannot be overstated.

What our users think about fastdup

License

fastdup is licensed under Creative Commons 4.0 license. See LICENSE.

For any queries, reach us at [e-mail]

Disclaimer

Usage Tracking

We have added an experimental crash report collection, using sentry.io. It does not collect user data other than anonymized IP address data, and it only logs fastdup library's own actions. We do NOT collect folder names, user names, image names, image content only aggregate performance statistics like total number of images, average runtime per image, total free memory, total free disk space, number of cores, etc. Collecting fastdup crashes will help us improve stability.

The code for the data collection is found here. On MAC we use Google crashpad.

It is always possible to opt out of the experimental crash report collection via either of the following two options:

• Define an environment variable called SENTRY_OPT_OUT
• or run() with turi_param='run_sentry=0'

About Visual-Layer

fastdup is founded by the authors of XGBoost, Apache TVM & Turi Create - Danny Bickson, Carlos Guestrin and Amir Alush.

Learn more about Visual Layer here.