dedup

There are 1 repository under dedup topic.

• Zygo / bees

  Best-Effort Extent-Same, a btrfs dedupe agent

  btrfsdedupdedupe
  Language:C++ 602

• markusressel / py-image-dedup

  CLI utility to find near duplicate images and remove all but the best copy.

  pythonpython3python-3image-analysisduplicate-detectionduplicate-imagesdeduplicationdedupfind-duplicatesimage-comparisonhacktoberfest
  Language:Python 150

• adlibre / adlibre-backup

  High performance rsync backup utilising BTRFS / ZFS filesystem features

  backupbackup-serverbtrfsdeduprsyncsnapshotzfs
  Language:Shell 36

• chucheng92 / HadoopDedup

  :watermelon:基于Hadoop和HBase的大规模海量数据去重

  big-datadedupcdcfspmapreduce
  Language:Java 27

• lkarlslund / stringdedup

  String deduplication package for Go

  dedupdeduplicationgolangstringxxhash
  Language:Go 15

• ParaGroup / p3arsec

  Parallel Patterns Implementation of PARSEC Benchmark Applications

  parallel-patternsbenchmarkbenchmarkingparsec-benchmarkblackscholesdedupferretvipsfreqminecannealfluidanimatestreamclusterfacesimbodytrackraytraceswaptionsskeleton-applicationskeletonskeleton-programming
  Language:C++ 15

• EastTower16 / LLMDataDistill

  distill large scale web page text

  datacleaningdatasetsdedupdeep-neural-networksflaxjaxllmlshlsh-implementation
  Language:C++ 12

• veqryn / slog-dedup

  Golang structured logging (slog) deduplication and sorting for use with json logging

  dedupdeduplicationgolanggolang-libraryjsonloggingslogstructured-logging
  Language:Go 10

• glehmann / hld

  Hard Link Deduplicator

  deduplicationhardlinksreflinksrustdedup
  Language:Rust 5

• helloall1900 / vhash

  A C++ reimplementation of Near Duplicate Video Detection - Get a 64-bit comparable hash-value for any video (Video Hash).

  dedupduplicatehashimagevideodeduplicate
  Language:C++ 3

• uicoolcn / UiCoolVisualWebSpider

  📄【优爱酷可视化网站网页数据采集系统】 采用先进的可视化采集技术，智能识别网页元素类型，如：图片、文字、链接、HTML 、文件等，支持运行Javascript脚本、应用正则表达式、自动滚屏、自动翻页、打开弹出窗口并采集数据，支持数据自动去重、仿人工间歇暂停防IP阻塞、自动保存等采集设置；支持浏览器Cookie和缓存等浏览器设置；支持代理轮换科学上网采集；支持“类别/关键字”；支持图像重命名等； 更可支持多线程采集等高级采集选项设置，vip版还可支持定时计划采集。

  visual-web-spidervisual-website-minerrunjavascriptauto-scrool-webchromium-webbrowser-corehigh-speedmulti-threadimages-extratorwords-text-minerlinks-spiderauto-save-to-excelexport-to-databaseregexregex-patternproxy-swithscheduled-data-minerdedupdeduplicationtable-list-grid-autotech
  3

• eminence / deduprs

  Hardlink deduplication tool for Linux

  dedupdeduplicationhard-linkrust
  Language:Rust 2

• go-utils / dedupe

  Easy Deduplication

  gogolangdedupdeduplication
  Language:Go 2

• rongrimes / zipfile-dedup

  Project to take two similar zipfiles, and to dedupe files that have the same tiemstamp in the older file.

  zipzipfile7zdeduplicationdeduplicatededupededuplicatordedupingdeduplicating-backupsdeduppython3python-3utility
  Language:Python 2

• carlinhosfranco / BenSP-Suite

  BenSP is a suite of parameterizable benchmarks for stream parallelism which is used to evaluate stream processing characteristics.

  dedupparsecparsec-benchmarkparametricstream-processingbenchmarkferretstreamparallel-processing
  Language:C 1

• harshasrisri / dedup

  Remove local files that are duplicates of files in another path

  rustdedupduplicate-filesduplicate-removaldeduplication
  Language:Rust 1

• hekmon / deduper

  Analyse 2 paths to found identical files and hard link them to save space

  deduplicationhardlinkdeduplicatededuplicatorfilesystemhardlinkinghardlinkslinuxsavingspaceduplicate-filesscanscannertoolutilitydedup
  Language:Go 1

• horgh / dupefile

  Detect and optionally delete duplicate files in a directory tree

  dedupfilefile-managementmanagement
  Language:Go 1

• jamjamjon / ilytix

  A CLI tool for images analysis: checking image integrity, images deduplication, image retrieval.

  dedupimage-deduplicationrust
  Language:Rust 1

• JumperBot / whitespace-sifter

  Sift duplicate whitespaces away!

  dedupdeduplicatorrustrust-craterust-langrust-languagerust-librarysiftwhitespace-removalstringstring-formattingstring-manipulationtext
  Language:Rust 1

• prebuilder / rdfind.py

  A python wrapper to rdfind

  dedupdeduplicationsymlink
  Language:Python 1

• xyb / chunkdup

  Find (partial content) duplicate files.

  content-defined-chunkingdeduplicationfastcdcrolling-hashdedup
  Language:Python 1

• dim-geo / analyze-dedup

  python script to analyze dedup usage in btrfs

  deduplicationdedupdedupepython3btrfsbtrfs-subvolumebtrfs-filesystem
  Language:Python 0

• xyb / chunksum

  Print FastCDC rolling hash chunks and checksums.

  checksumdedupfastcdcrolling-hashchunkingcontent-defined-chunkingdeduplication
  Language:Python 0

• yugn / yadupe

  Yet another tool to find and remove duplicate files.

  dedupdedupededuplicateduplicate-filesduplicatefilefinderduplicatesremove-duplicate-filesremove-duplicatessearch-duplicates
  Language:Python 0