dedupe

There are 9 repositories under dedupe topic.

restic / restic
Fast, secure, efficient backup program
go restic backup deduplication dedupe secure-by-default
Language:Go 24064
borgbackup / borg
Deduplicating archiver with compression and authenticated encryption.
python python-3 compression dedupe ssh deduplication backup borgbackup encryption cython c
Language:Python 10631
dedupeio / dedupe
:id: A python library for accurate and scalable fuzzy matching, record deduplication and entity-resolution.
dedupe record-linkage python python-library entity-resolution dedupe-library de-duplicating datamade clustering
Language:Python 3992
scinos / yarn-deduplicate
Deduplication tool for yarn.lock files
yarn duplicates dedupe duplicated-packages yarn-lock lock-file
Language:TypeScript 1375
recordlinkage
J535D165 / recordlinkage
A powerful and modular toolkit for record linkage and duplicate detection in Python
record-linkage entity-resolution dedupe string-distance machine-learning privacy python python-library data-matching deduplication similarity utrecht-university
Language:Python 916
zingg
zinggAI / zingg
Scalable identity resolution, entity resolution, data mastering and deduplication using ML
fuzzymatch fuzzy-matching deduplication dedupe masterdata dataengineering data-transformation analytics-engineering entity-resolution identity-resolution data-transformations data-science spark ml etl dataquality identity modern-data-stack analytics datalake
Language:Java 898
duplicut
nil0x42 / duplicut
Remove duplicates from MASSIVE wordlist, without sorting it (for dictionary-based password cracking)
hashcat password hashes cracking c wordlist wordlists uniq unique dedupe remove-duplicates wordlist-generator duplicate-detection password-cracking dictionary
Language:C 799
blakeembrey / free-style
Make CSS easier and more maintainable by using JavaScript
css js dedupe hash typescript css-in-js css-string javascript minification
Language:TypeScript 700
Zygo / bees
Best-Effort Extent-Same, a btrfs dedupe agent
btrfs dedup dedupe
Language:C++ 600
dedupeio / csvdedupe
:id: Command line tool for deduplicating CSV files
dedupe cli record-linkage entity-resolution csv-files
Language:Python 404
dedupeio / dedupe-examples
:id: Examples for using the dedupe library
dedupe record-linkage entity-resolution python
Language:Python 394
knjcode / imgdupes
Identifying and removing near-duplicate images using perceptual hashing.
image dedupe perceptual-hashing perceptual-hashes deduplicate
Language:Python 336
kornelski / dupe-krill
A fast file deduplicator
dedupe file-deduplication rust-library hardlinks dupes macos
Language:Rust 183
Lakshmipathi / dduper
Fast block-level out-of-band BTRFS deduplication tool.
btrfs btrfs-progs btrfs-tools deduplication dedupe
Language:Python 163
mail-deduplicate
kdeldycke / mail-deduplicate
📧 CLI to deduplicate mails from mail boxes.
python mail dedupe cli maildir mailbox mbox deduplication cleanup babyl mh mmdf email
Language:Python 160
jason89521 / daxus
Daxus is a server state management library for React that provides full control over data, leading to a better user experience.
react data hook cache revalidate dedupe server-state-management user-experience
Language:TypeScript 90
jRimbault / yadf
Yet Another Dupes Finder
dedupe deduplication dupes-finder duplicate-detection fdupes file-deduplication
Language:Rust 50
Senzing / awesome
Curated list of awesome software and resources for Senzing, The First Real-Time AI for Entity Resolution.
awesome data-matching de-duplicating dedupe deduplication disambiguation entities entity entity-linking entity-matching entity-resolution fuzzy-matching fuzzymatch identity identity-resolution record-linkage resolution resource senzing senzing-community
Language:Python 48
dssg / pgdedupe
A simple command line interface to the datamade/dedupe library.
deduplication dedupe python data-cleaning record-linkage postgresql database
Language:Jupyter Notebook 42
jchristn / WatsonDedupe
Self-contained C# library for data deduplication using Sqlite
compress compression deduplication storage dedupe data-deduplication nuget sqlite-database duplicate-data chunk-key chunk-data chunk
Language:C# 35
mighty-justice / django-super-deduper
Utilities for de-duping Django model instances
dedupe django python
Language:Python 30
zirkonit / duplex
Duplicate code finder for Elixir
dedupe duplex elixir escript lint static-analysis static-code-analysis
Language:Elixir 22
kevinpollet / pocket-deduper
Remove duplicates from your Pocket list.
go golang pocket dedupe duplicates tool cli
Language:Go 18
dedupeio / dedupe-variable-address
Address Variable Type for dedupe
dedupe dedupe-variable
Language:Python 8
dedupeio / dedupe-variable-name
name variable type for dedupe
dedupe dedupe-variable
Language:Python 7
blocksync-fast
nethappen / blocksync-fast
Fast block device sync with digest, designed to improve block-based backups.
fast backup backup-tool block-based block-device block-level block-sync btrfs checksums cloning-tool compression copy-on-write dedupe deduplication delta-file digest disk-image linux ssh synchronization
Language:C 6
samhirtarif / helper-methods-js
A repo that contains helper methods for common and not-so-common use cases
async dedupe deduplication deepcopy indexesof isasync
Language:TypeScript 6
dedupeio / dedupe-variable-employer
dedupe dedupe-variable
Language:Python 5
mterron / swuniq
A command-line tool for deduplicating entries in a file or stream with constant memory usage
uniq filter dedupe deduping deduplication deduplicate cli sliding-window
Language:C 5
amoghe / dedup
Deduplicator
deduplication dedupe compression compressor
Language:Go 4
dedupeio / dedupe-variable-fuzzycategory
Dedupe Variable for Fuzzy Categories
dedupe dedupe-variable
Language:Python 4
lilydjwg / android-dedupefs
A filesystem for reading Android dedupe backup
android backup fuse dedupe
Language:C 4
OSPG / godedupe
Improved and modern fdupes alternative
deduplication dedupe modern
Language:Go 4
shashank-mugiwara / dedupeknn
Fast Scalable Dedupe - Fuzzy Matching With Opensearch + nmslib + Rapidfuzz
address-matching dedupe fuzzymatching nmslib opensearch rapidfuzz
Language:Python 4
dedupeio / parseratorvariable
Base class for dedupe variables for parsed fields
dedupe dedupe-variable
Language:Python 3
chansooligans / oagdedupe
a Python library for scalable entity resolution, using active learning to learn blocking configurations, generate comparison pairs, then clasify matches
dedupe machine-learning entity-resolution record-linkage
Language:Python 2