data-deduplication

There are 4 repositories under data-deduplication topic.

• dpc / rdedup

  Data deduplication engine, supporting optional compression and public key encryption.

  data-deduplicationbackupencryptiondeduplication
  Language:Rust 820

• sail-sg / sailcraft

  Data Toolkit for Sailor Language Models

  data-cleaningdata-deduplication
  Language:Python 65

• jchristn / WatsonDedupe

  Self-contained C# library for data deduplication using Sqlite

  compresscompressiondeduplicationstoragededupedata-deduplicationnugetsqlite-databaseduplicate-datachunk-keychunk-datachunk
  Language:C# 35

• Zabuzard / FastCDC4J

  Fast and efficient content-defined chunking for data deduplication. Java implementation of FastCDC as library.

  fastcdccdccontent-defined-chunkingchunkingdata-deduplicationlibraryjava
  Language:Java 19

• david-siqi-liu / sparklyclean

  Optimal distributed data deduplication and supervised learning pipeline using Apache Spark

  sparkdata-engineeringdata-cleaningdata-deduplicationdeduplicationdata-sciencehadoopdistributed-systems
  Language:Scala 10

• bmiller1009 / deduper

  General deduping engine for JDBC sources with output to JDBC/csv targets

  deduplicationdata-deduplicationdeduplicate-datadeduplicate
  Language:Kotlin 5

• shubham-thakare / data-deduplication

  A JAVA project that splits data using hashing techniques and removes duplicate blocks to save cloud storage. This project also uses the CloudSim framework for cloud storage simulation.

  cloud-storagecloudsimcloudsim-frameworkdata-deduplicationjava
  Language:Java 3

• Anveshika06 / VIT-VTAS-TY-2022

  data-deduplicationhashing-algorithm
  Language:Python 1

• bevry / fellow

  Fellow is a package for creating people that can be unified by their shared values via a singleton list on the class

  client-sidedata-deduplicationmodelnodejs
  Language:TypeScript 1
• PolyDeDupe

  gagan3012 / PolyDeDupe

  PolyDeDupe: Multi-Lingual Data Deduplication

  data-deduplicationmultilingualnlp
  Language:Python 1

• baraverkstad / mixtape

  Practical backups. The Unix toolkit way.

  backupdata-deduplicationbashshelllinuxcommand-lineunix
  Language:Shell 0

• imehar / data-deduplication

  This is a server client architecture based data deduplication implementation

  data-deduplicationhashingserver-client
  Language:C++ 0

• Jim-JMCD / Data_storage_network_deduplication_calculator

  A calculator for storage and transmission of deduplicated data presentation in charts and tables

  data-deduplicationdeduplicationdeduplication-calculatornetwork-deduplication-calculatorstorage-deduplication-calculator
  0