There are 4 repositories under data-deduplication topic.
Self-contained C# library for data deduplication using Sqlite
Fast and efficient content-defined chunking for data deduplication. Java implementation of FastCDC as library.
Optimal distributed data deduplication and supervised learning pipeline using Apache Spark
General deduping engine for JDBC sources with output to JDBC/csv targets
A JAVA project that splits data using hashing techniques and removes duplicate blocks to save cloud storage. This project also uses the CloudSim framework for cloud storage simulation.
Fellow is a package for creating people that can be unified by their shared values via a singleton list on the class
Practical backups. The Unix toolkit way.
This is a server client architecture based data deduplication implementation
A calculator for storage and transmission of deduplicated data presentation in charts and tables
This repository contains SQL scripts and documentation for cleaning and standardizing data in the NashvilleHousing table within the sqlproject2 database. The project aims to prepare the dataset for analysis by addressing inconsistencies, filling missing values, standardizing formats, and removing duplicates.