record-linkage

There are 23 repositories under record-linkage topic.

dedupeio / dedupe
:id: A python library for accurate and scalable fuzzy matching, record deduplication and entity-resolution.
clustering datamade de-duplicating dedupe dedupe-library entity-resolution python python-library record-linkage
Language:Python 3971
openvenues / libpostal
A C library for parsing/normalizing street addresses around the world. Powered by statistical NLP and open geo data.
address-parser machine-learning nlp address international c deduplication record-linkage deduping natural-language-processing
Language:C 3943
moj-analytical-services / splink
Fast, accurate and scalable probabilistic data linkage with support for multiple SQL backends
record-linkage spark em-algorithm deduplication deduplicate-data entity-resolution data-matching fuzzy-matching data-science duckdb uk-gov-data-science
Language:Python 1079
recordlinkage
J535D165 / recordlinkage
A powerful and modular toolkit for record linkage and duplicate detection in Python
record-linkage entity-resolution dedupe string-distance machine-learning privacy python python-library data-matching deduplication similarity utrecht-university
Language:Python 906
Yomguithereal / talisman
Straightforward fuzzy matching, information retrieval and NLP building blocks for JavaScript.
natural-language-processing machine-learning fuzzy-matching clustering record-linkage information-retrieval deduplication
Language:JavaScript 695
dedupeio / csvdedupe
:id: Command line tool for deduplicating CSV files
dedupe cli record-linkage entity-resolution csv-files
Language:Python 398
dedupeio / dedupe-examples
:id: Examples for using the dedupe library
dedupe record-linkage entity-resolution python
Language:Python 394
J535D165 / data-matching-software
A list of free data matching and record linkage software.
data-matching record-linkage entity-resolution deduplication fuzzy-matching software awesome awesome-list machine-learning open-source
344
maxharlow / csvmatch
🔎 Finds fuzzy matches between CSV files
data-matching entity-resolution fuzzy-matching record-linkage csv
Language:Python 175
vintasoftware / entity-embed
PyTorch library for transforming entities like companies, products, etc. into vectors to support scalable Record Linkage / Entity Resolution using Approximate Nearest Neighbors.
entity-resolution record-linkage representation-learning embeddings approximate-nearest-neighbors deduplication entity-matching data-matching deep-learning python pytorch
Language:Jupyter Notebook 139
zouzias / spark-lucenerdd
Spark RDD with Lucene's query and entity linkage capabilities
spark lucene rdd spatial-search record-linkage deduplication entity-linking linkage hacktoberfest
Language:Scala 127
dice-group / LIMES
Link Discovery Framework for Metric Spaces.
machine-learning artificial-intelligence optimization linked-data semantic-web record-linkage scalability rdf
Language:JavaScript 126
ropeladder / record-linkage-resources
Resources for tackling record linkage / deduplication / data matching problems
record-linkage deduplication data-matching entity-resolution python java javascript
104
usc-isi-i2 / rltk
Record Linkage ToolKit (Find and link entities)
linkage similarity similarity-metric string-similarity record-linkage entity-resolution deduplication
Language:Python 103
Wikidata / soweego
Link Wikidata items to large catalogs
wikimedia wikidata knowledge-graph record-linkage data-matching entity-resolution identifiers entity-linking
Language:Python 95
fritshermans / deduplipy
Python package for deduplication/entity resolution using active learning
deduplication entity-resolution fuzzy-matching record-linkage
Language:Python 72
linktransformer
dell-research-harvard / linktransformer
A convenient way to link, deduplicate, aggregate and cluster data(frames) in Python using deep learning
deep-learning entity-matching entity-resolution huggingface-transformers nlp python record-linkage sentence-transformers transformers
Language:Python 69
data61 / anonlink
Python implementation of anonymous linkage using cryptographic linkage keys
privacy-enhancing-technologies privacy-preserving record-linkage entity-resolution cryptography
Language:Python 59
cleanzr / dblink
Distributed Bayesian Entity Resolution in Apache Spark
entity-resolution apache-spark bayesian-inference record-linkage distributed-machine-learning mcmc
Language:Scala 54
data61 / clkhash
CLK hash: hash pii for entity matching
record-linkage privacy-enhancing-technologies hashing entity-matching privacy-preserving-record-linkage
Language:Python 47
Senzing / awesome
Curated list of awesome software and resources for Senzing, The First Real-Time AI for Entity Resolution.
senzing entity-resolution entity resolution record-linkage entity-matching resource awesome data-matching de-duplicating dedupe deduplication fuzzy-matching fuzzymatch identity identity-resolution disambiguation entities entity-linking senzing-community
Language:Python 46
dssg / pgdedupe
A simple command line interface to the datamade/dedupe library.
deduplication dedupe python data-cleaning record-linkage postgresql database
Language:Jupyter Notebook 42
J535D165 / recordlinkage-annotator
A browser user interface for manual labeling of record pairs.
record-linkage entity-resolution data-matching deduplication machine-learning annotation-tool labeling-tool
Language:JavaScript 40
entrepreneur-interet-general / Merge-Machine
Merge Dirty Data with Clean Reference Tables
record-linkage python entity-resolution elasticsearch entrepreneur-interet-general
Language:Python 35
iesl / stance
Learned string similarity for entity names using optimal transport.
string-similarity optimal-transport stance aliases entity-resolution string-distance string-matching record-linkage
Language:Python 33
k3jph / phonics-in-r
Phonetic Spelling Algorithms in R
phonetic-spelling-algorithms soundex phonics nysiis metaphone text-processing linguistics record-linkage bsd-2-license
Language:R 28
ing-bank / spark-matcher
Record matching and entity resolution at scale in Spark
deduplication entity-resolution record-linkage spark
Language:Python 27
data61 / anonlink-entity-service
Privacy Preserving Record Linkage Service
privacy-preserving-record-linkage record-linkage privacy-enhancing-technologies
Language:Python 25
vaneseltine / nominally
A maximum-strength name parser for record linkage.
data-matching data-science deduplication entity-resolution human-name parser parsing record-linkage
Language:Python 25
J535D165 / FEBRL-fork-v0.4.2
Fork of the Freely Extensible Biomedical Record Linkage program
record-linkage entity-resolution deduplication matching python-library
Language:Python 23
Valires / er-evaluation
An End-to-End Evaluation Framework for Entity Resolution Systems
data-science deduplication disambiguation duplicate-detection entity-resolution evaluation fuzzy-matching matching ml-evaluation ml-testing record-linkage statistics inventor-name-disambiguation author-name-disambiguation
Language:Python 21
data61 / blocklib
Python implementations of record linkage blocking techniques.
privacy-preserving-record-linkage privacy-enhancing-technologies record-linkage
Language:Python 19
gpoulter / pydedupe
(Archived) A Python library for record linkage and deduplication.
deduplication record-linkage
Language:Python 19
ngmarchant / comparator
Similarity and distance measures for clustering and record linkage applications in R
r-package distance-measures similarity-measures string-similarity record-linkage distance-metrics entity-resolution clustering
Language:R 15
zouzias / spark-lucenerdd-examples
Examples of spark-lucenerdd
docker entity-linking examples lucene lucene-query record-linkage spark
Language:Scala 15
cleanzr / fasthash
Performs unique entity estimation corresponding to Chen, Shrivastava, Steorts (2018).
entity-estimation entity-resolution record-linkage hashing blocking
Language:Python 14