duplicate-detection

There are 9 repositories under duplicate-detection topic.

• nomic-ai / nomic

  Interact, analyze and structure massive text, image, embedding, audio and video datasets

  pythonclusteringduplicate-detectionembeddingstexttopic-modelingunstructured-data
  Language:Python 1015
• duplicut

  nil0x42 / duplicut

  Remove duplicates from MASSIVE wordlist, without sorting it (for dictionary-based password cracking)

  hashcatpasswordhashescrackingcwordlistwordlistsuniquniquededuperemove-duplicateswordlist-generatorduplicate-detectionpassword-crackingdictionary
  Language:C 797
• depp

  CryogenicPlanet / depp

  ⚡ Check your npm modules for unused and duplicate dependencies fast

  dependencyunusedduplicate-detectionnpmmodulesmonorepo
  Language:Go 265
• videohash

  akamhy / videohash

  Near Duplicate Video Detection (Perceptual Video Hashing) - Get a 64-bit comparable hash-value for any video.

  near-duplicate-video-clip-detectionnear-duplicate-videoduplicate-videosfind-similar-videos-by-contentduplicate-detectionvideopythonffmpegvideo-deduplicationduplicate-video-finderndvdndvrvideo-similarity-searchvisual-claim
  Language:Python 259
• deduplicator

  sreedevk / deduplicator

  Filter, Sort & Delete Duplicate Files Recursively

  deduplicationduplicate-detectionduplicate-filesduplicatefilefinderfilesystemrust
  Language:Rust 256
• zoplicate

  ChenglongMa / zoplicate

  A plugin that does one thing only: Detect and manage duplicate items in Zotero.

  duplicate-detectionzoterozotero-addonzotero-pluginzotero7zotero6
  Language:TypeScript 198

• JorenSix / Panako

  The Panako acoustic fingerprinting system.

  music-information-retrievalacoustic-fingerprintingduplicate-detectionaudio-processing
  Language:Java 175
• SIFTImageSimilarity

  adumrewal / SIFTImageSimilarity

  Interactive code for image similarity using SIFT algorithm

  image-similarityopencvimage-processingobject-detectionsiftsift-algorithmsift-descriptorssift-featurespythonimage-analysisclosenessresemblancecomparisonfeature-extractionfeature-matchingfeature-mappingduplicate-detectionduplicate-imagescomputer-visionhomography
  Language:Python 164

• markusressel / py-image-dedup

  CLI utility to find near duplicate images and remove all but the best copy.

  dedupdeduplicationduplicate-detectionduplicate-imagesfind-duplicateshacktoberfestimage-analysisimage-comparisonpythonpython-3python3
  Language:Python 150

• umbertogriffo / fast-near-duplicate-image-search

  Fast Near-Duplicate Image Search and Delete using pHash, t-SNE and KDTree.

  computer-visionduplicate-detectionduplicates-removedhashingimage-deduplicationimage-processingkd-treekdtreenear-duplicatenearest-neighbor-searchnearest-neighborsperceptual-hashingphasht-sne
  Language:Python 140

• kristiankoskimaki / vidupe

  Vidupe is a program that can find duplicate and similar video files. V1.211 released on 2019-09-18, Windows exe here:

  videosduplicatesduplicate-detectionduplicate-videos
  Language:C++ 136

• unmade / audiomatch

  Find similar audio files easily

  duplicate-detectionaudio-fingerprintingchromaprintsound-analysispythoncythoncommand-line
  Language:Python 136

• itwillwork / ostap

  CLI tool that fast checks if your bundle contains multiple versions of the same package, only by looking in package.json.

  cli-appfrontendduplicate-detectionwebpackbundle
  Language:JavaScript 115

• logpai / bughub

  A collection of free-text bug reports for duplicate issue identification

  bug-reportsdatasetsduplicate-detectionnlp
  112

• Chinmay26 / Near-Duplicate-Video-Detection

  Detecting near-duplicate videos by aggregating features from intermediate CNN layers

  duplicate-detectionalexnetvgg16cnn-layersduplicate-videos
  Language:Jupyter Notebook 95

• cloud-py-api / mediadc

  Nextcloud Media Duplicate Collector application

  nextcloudnextcloud-appsmediaduplicatesduplicate-detectioncollectormediadcopen-sourcesingle-page-appphpvuenextcloud-vue-apppythonpython3
  Language:PHP 93

• dlidstrom / Duplo

  Duplicates finder for various source code formats.

  c-plus-pluscode-qualityduplicate-detection
  Language:C++ 86
• removedupes

  eyalroz / removedupes

  Remove Duplicate Messages

  thunderbirdmozillamail-clientemailduplicatesduplicate-detectionmail-foldersemail-parsingproductivitycleanupcleanerthunderbird-extensionthunderbird-addon
  Language:JavaScript 79
• cbird

  scrubbbbs / cbird

  Command-line program for managing a media collection, with focus on Content-Based Image Retrieval (Computer Vision) methods for finding duplicates.

  command-line-interfacecontent-based-image-retrievalcomputer-visionduplicate-detectionduplicate-filesduplicatessimilarity-searchopencvqt6ffmpeg
  Language:C++ 77

• deplicate / deplicate

  Advanced Duplicate File Finder for Python

  duplicate-filesduplicatefilefinderduplicate-detectionduplicatesduplicateduplicates-removedscanningfinderpypipythonpurge-duplicate-filesmulti-filteringunixwindowsmacosxdeplicateduplication-finder
  Language:Python 76
• pyJedAI

  AI-team-UoA / pyJedAI

  An open-source library that leverages Python’s data science ecosystem to build powerful end-to-end Entity Resolution workflows.

  deduplicationentity-matchingentity-resolutionpythonlink-discoverydata-matchingfuzzy-matchingmachine-learningdata-disambigationduplicate-detection
  Language:Python 62
• dude

  PJDude / dude

  Duplicates Detector is a cross-platform GUI utility for finding duplicate files, allowing you to delete or link them to save space. Duplicate files are displayed and processed on two synchronized panels for efficient and convenient operation.

  duplicate-detectionduplicate-filesduplicatesduplicates-removalpythontkinterdeduplicationsha1utilityutility-applicationpython3guigui-applicationthreadseasyeasy-to-useeasyuiduplicatecli
  Language:Python 56

• src-d / gemini

  Advanced similarity and duplicate source code at scale.

  duplicate-detectionsparkduplicateshashsource-code-analysis
  Language:Scala 54

• src-d / apollo

  Advanced similarity and duplicate source code proof of concept for our research efforts.

  duplicatessimilaritysource-codeduplicate-detectionsimilarity-searchpython
  Language:Python 52

• marius-sucan / Quick-Picto-Viewer

  A uniquely crafted image viewer and editor with options to organize files, and maintain large lists of image files for slideshows, dupes detection or other purposes.

  image-viewerslideshow-makerslideshowimage-organizerimageeditorimageprocessingimage-ediimageimage-manipulationimage-processingpaintpaint-applicationdupes-finderduplicate-detectionfileorganizerfiles-managementorganizer
  Language:AutoHotkey 51

• jRimbault / yadf

  Yet Another Dupes Finder

  duplicate-detectiondeduplicationfdupesdupes-finderfile-deduplicationdedupe
  Language:Rust 50

• rdgd / twly

  Wanna get DRY? Static analysis tool for detecting repeat code.

  static-analysisdryduplicate-detectionjavascriptjavascript-toolsnodejs
  Language:JavaScript 47

• fffaraz / QtHashSum

  File Checksum Integrity Verifier & Duplicate File Finder written in C++ Qt

  qthashc-plus-plussha1sha3md5cryptographic-hashsha1summd5sumchecksumduplicate-detectionchecksum-integrity-verifierduplicate-fileshashingfilesystem-indexerduplicatefilefinderintegrity
  Language:C++ 40

• sed-inf-u-szeged / OpenStaticAnalyzer

  OpenStaticAnalyzer is a source code analyzer tool, which can perform deep static analysis of the source code of complex systems.

  static-analysisstatic-code-analysismetricsclone-detectionduplicate-detectioncoding-standardscoding-stylecoding-rulescoding-issues
  Language:C++ 38
• snowman

  HPI-Information-Systems / snowman

  Welcome to Snowman App – a Data Matching Benchmark Platform.

  matchingentity-resolutionkpisdata-matchingduplicate-detectionbenchmarksnowmandata-stewards
  Language:TypeScript 37

• appzcoder / phpcloc

  :rocket: Cloc & duplicate code checker tool

  cloccode-duplicationconsoleduplicate-detectionphp
  Language:PHP 35

• mattpodolak / duplicate-img-detection

  A basic duplicate image detection service using perceptual image hash functions and nearest neighbor search, implemented using faiss, fastapi, and imagehash

  pythonimage-hashingnearest-neighbor-searchfaissduplicate-detection
  Language:Python 30
• demystify

  exponential-decay / demystify

  Engine for analysis of Siegfried export files and DROID CSV. The tool has three purposes, break the export into its components and store them within a SQLite database; create additional columns to augment the output where useful; and query the SQLite database, outputting results in a readable form useful for analysis by researchers and archivists within digital preservation departments in memory institutions. The tool will find duplicates, unidentified files, blacklisted objects, character encoding issues, and more.

  archivescode4libcollection-profilingdigipresdigital-preservationduplicate-detectionformat-analysispronom
  Language:HTML 23

• akcarsten / Duplicate-Finder

  This Python packages identifies duplicate files in a folder of interest.

  duplicate-detectionpython
  Language:Python 22

• Valires / er-evaluation

  An End-to-End Evaluation Framework for Entity Resolution Systems

  data-sciencededuplicationdisambiguationduplicate-detectionentity-resolutionevaluationfuzzy-matchingmatchingml-evaluationml-testingrecord-linkagestatisticsinventor-name-disambiguationauthor-name-disambiguation
  Language:Python 22

• github-userx / Awesome-Duplication-Finders

  Apps to find duplicate files including same/similar images & videos (with computer vision/AI)

  deduplicationduplicate-detectionduplicate-filesduplicate-imagesduplicate-video-finderduplicate-videosduplicatefilefinderduplicatesduplicationduplication-finder
  21