dedupe

There are 8 repositories under dedupe topic.

restic / restic
Fast, secure, efficient backup program
backup dedupe deduplication go restic secure-by-default
Language:Go 23703
borgbackup / borg
Deduplicating archiver with compression and authenticated encryption.
python python-3 compression dedupe ssh deduplication backup borgbackup encryption cython c
Language:Python 10507
dedupeio / dedupe
:id: A python library for accurate and scalable fuzzy matching, record deduplication and entity-resolution.
clustering datamade de-duplicating dedupe dedupe-library entity-resolution python python-library record-linkage
Language:Python 3967
scinos / yarn-deduplicate
Deduplication tool for yarn.lock files
dedupe duplicated-packages duplicates lock-file yarn yarn-lock
Language:TypeScript 1373
recordlinkage
J535D165 / recordlinkage
A powerful and modular toolkit for record linkage and duplicate detection in Python
record-linkage entity-resolution dedupe string-distance machine-learning privacy python python-library data-matching deduplication similarity utrecht-university
Language:Python 905
zingg
zinggAI / zingg
Scalable identity resolution, entity resolution, data mastering and deduplication using ML
fuzzymatch fuzzy-matching deduplication dedupe masterdata dataengineering data-transformation analytics-engineering entity-resolution identity-resolution data-transformations data-science spark ml etl dataquality identity modern-data-stack analytics datalake
Language:Java 875
duplicut
nil0x42 / duplicut
Remove duplicates from MASSIVE wordlist, without sorting it (for dictionary-based password cracking)
hashcat password hashes cracking c wordlist wordlists uniq unique dedupe remove-duplicates wordlist-generator duplicate-detection password-cracking dictionary
Language:C 781
blakeembrey / free-style
Make CSS easier and more maintainable by using JavaScript
css css-in-js css-string dedupe hash javascript js minification typescript
Language:TypeScript 698
Zygo / bees
Best-Effort Extent-Same, a btrfs dedupe agent
btrfs dedup dedupe
Language:C++ 585
dedupeio / csvdedupe
:id: Command line tool for deduplicating CSV files
dedupe cli record-linkage entity-resolution csv-files
Language:Python 398
dedupeio / dedupe-examples
:id: Examples for using the dedupe library
dedupe record-linkage entity-resolution python
Language:Python 393
knjcode / imgdupes
Identifying and removing near-duplicate images using perceptual hashing.
dedupe deduplicate image perceptual-hashes perceptual-hashing
Language:Python 334
kornelski / dupe-krill
A fast file deduplicator
dedupe dupes file-deduplication hardlinks macos rust-library
Language:Rust 183
Lakshmipathi / dduper
Fast block-level out-of-band BTRFS deduplication tool.
btrfs btrfs-progs btrfs-tools deduplication dedupe
Language:Python 162
mail-deduplicate
kdeldycke / mail-deduplicate
📧 CLI to deduplicate mails from mail boxes.
babyl cleanup cli dedupe deduplication email mail mailbox maildir mbox mh mmdf python
Language:Python 156
jason89521 / daxus
Daxus is a server state management library for React that provides full control over data, leading to a better user experience.
react data hook cache revalidate dedupe server-state-management user-experience
Language:TypeScript 90
jRimbault / yadf
Yet Another Dupes Finder
duplicate-detection deduplication fdupes dupes-finder file-deduplication dedupe
Language:Rust 48
Senzing / awesome
Curated list of awesome software and resources for Senzing, The First Real-Time AI for Entity Resolution.
senzing entity-resolution entity resolution record-linkage entity-matching resource awesome data-matching de-duplicating dedupe deduplication fuzzy-matching fuzzymatch identity identity-resolution disambiguation entities entity-linking senzing-community
Language:Python 46
dssg / pgdedupe
A simple command line interface to the datamade/dedupe library.
deduplication dedupe python data-cleaning record-linkage postgresql database
Language:Jupyter Notebook 42
jchristn / WatsonDedupe
Self-contained C# library for data deduplication using Sqlite
compress compression deduplication storage dedupe data-deduplication nuget sqlite-database duplicate-data chunk-key chunk-data chunk
Language:C# 35
mighty-justice / django-super-deduper
Utilities for de-duping Django model instances
dedupe django python
Language:Python 30
zirkonit / duplex
Duplicate code finder for Elixir
elixir escript duplex dedupe static-analysis static-code-analysis lint
Language:Elixir 21
kevinpollet / pocket-deduper
Remove duplicates from your Pocket list.
go golang pocket dedupe duplicates tool cli
Language:Go 18
dedupeio / dedupe-variable-address
Address Variable Type for dedupe
dedupe dedupe-variable
Language:Python 8
dedupeio / dedupe-variable-name
name variable type for dedupe
dedupe dedupe-variable
Language:Python 7
samhirtarif / helper-methods-js
A repo that contains helper methods for common and not-so-common use cases
async dedupe deduplication deepcopy indexesof isasync
Language:TypeScript 6
dedupeio / dedupe-variable-employer
dedupe dedupe-variable
Language:Python 5
mterron / swuniq
A command-line tool for deduplicating entries in a file or stream with constant memory usage
uniq filter dedupe deduping deduplication deduplicate cli sliding-window
Language:C 5
blocksync-fast
nethappen / blocksync-fast
Fast block device sync with digest, designed to improve block-based backups.
fast backup backup-tool block-based block-device block-level block-sync btrfs checksums cloning-tool compression copy-on-write dedupe deduplication delta-file digest disk-image linux ssh synchronization
Language:C 5
amoghe / dedup
Deduplicator
compression compressor dedupe deduplication
Language:Go 4
dedupeio / dedupe-variable-fuzzycategory
Dedupe Variable for Fuzzy Categories
dedupe dedupe-variable
Language:Python 4
lilydjwg / android-dedupefs
A filesystem for reading Android dedupe backup
android backup dedupe fuse
Language:C 4
OSPG / godedupe
Improved and modern fdupes alternative
dedupe deduplication modern
Language:Go 4
shashank-mugiwara / dedupeknn
Fast Scalable Dedupe - Fuzzy Matching With Opensearch + nmslib + Rapidfuzz
address-matching dedupe fuzzymatching nmslib opensearch rapidfuzz
Language:Python 4
dedupeio / parseratorvariable
Base class for dedupe variables for parsed fields
dedupe dedupe-variable
Language:Python 3
chansooligans / oagdedupe
a Python library for scalable entity resolution, using active learning to learn blocking configurations, generate comparison pairs, then clasify matches
dedupe machine-learning entity-resolution record-linkage
Language:Python 2