data-cleaning

There are 45 repositories under data-cleaning topic.

cleanlab / cleanlab
Cleanlab's open-source library is the standard data-centric AI package for data quality and machine learning with messy, real-world data and labels.
active-learning annotation data-centric-ai data-cleaning data-curation data-labeling data-profiling data-quality data-science data-validation dataops dataquality datasets exploratory-data-analysis labeling llms noisy-labels out-of-distribution-detection outlier-detection weak-supervision
Language:Python 10893
fiftyone
voxel51 / fiftyone
Refine high-quality datasets and visual AI models
machine-learning artificial-intelligence deep-learning computer-vision developer-tools data-science python active-learning data-centric-ai data-cleaning data-curation data-quality image-classification object-detection unstructured-data vector-search visualization
Language:Python 9868
miller
johnkerl / miller
Miller is like awk, sed, cut, join, and sort for name-indexed data such as CSV, TSV, and tabular JSON
data-processing data-cleaning csv csv-format streaming-data streaming-algorithms tsv json json-data data-reduction data-regression statistics statistical-analysis devops devops-tools tabular-data command-line command-line-tools unix-toolkit miller
Language:Go 9456
unionai-oss / pandera
A light-weight, flexible, and expressive statistical data testing library
pandas validation schema dataframes testing pandas-validation pandas-dataframe data-validation data-cleaning data-check testing-tools assertions data-assertions data-verification dataframe-schema hypothesis-testing pandas-validator data-processing
Language:Python 4014
justmarkham / pandas-videos
Jupyter notebook and datasets from the pandas video series
data-analysis data-cleaning data-science jupyter-notebook pandas python tutorial
Language:Jupyter Notebook 2241
justmarkham / DAT8
General Assembly's 2015 Data Science course in Washington, DC
data-science machine-learning scikit-learn data-analysis pandas jupyter-notebook python course linear-regression logistic-regression model-evaluation naive-bayes natural-language-processing decision-trees ensemble-learning clustering regular-expressions web-scraping data-visualization data-cleaning
Language:Jupyter Notebook 1616
hi-primus / optimus
:truck: Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark
spark pyspark data-wrangling bigdata big-data-cleaning data-science data-cleansing data-cleaner data-transformation machine-learning data-profiling data-extraction data-exploration data-analysis data-preparation cudf dask dask-cudf data-cleaning
Language:Python 1517
skrub
skrub-data / skrub
Machine learning with dataframes
machine-learning data-science data-cleaning data data-preparation data-preprocessing data-analysis dirty-data data-wrangling dataframe dataframes
Language:Python 1457
sfirke / janitor
simple tools for data cleaning in R
data-analysis data-cleaning data-science dirty-data excel pivot-tables r spss tabulations tidyverse
Language:R 1407
data-forge / data-forge-ts
The JavaScript data transformation and analysis toolkit inspired by Pandas and LINQ.
csv data data-analysis data-cleaning data-cleansing data-forge data-management data-manipulation data-munging data-visualization data-wrangling javascript json linq nodejs pandas visualization
Language:TypeScript 1375
ECNU-ICALK / EduChat
An open-source educational chat model from ICALK, East China Normal University. 开源中英教育对话大模型。(通用基座模型，GPU部署，数据清理) 致敬: LLaMA, MOSS, BELLE, Ziya, vLLM
belle chinese-nlp data-cleaning education llama llm moss open-models
Language:Jupyter Notebook 844
klib
akanz1 / klib
Easy to use Python library of customized functions for cleaning and analyzing data.
data-science data-analysis klib data-visualization python feature-selection data-cleaning data-preprocessing
Language:Python 523
schema-inspector / schema-inspector
Schema-Inspector is a simple JavaScript object sanitization and validation module.
javascript validation sanitization data-cleaning
Language:JavaScript 504
encord-team / encord-active
The toolkit to test, validate, and evaluate your models and surface, curate, and prioritize the most valuable data for labeling.
computer-vision data data-science data-validation deep-learning machine-learning ml mlops python active-learning annotations data-centric data-cleaning data-quality label-errors model-quality noisy-labels label-quality object-detection
Language:Python 453
data-cleaning / validate
Professional data validation for the R environment
data-cleaning r validation
Language:R 425
desbordante-core
Desbordante / desbordante-core
Desbordante is a high-performance data profiler that is capable of discovering many different patterns in data using various algorithms. It also allows to run data cleaning scenarios using these algorithms. Desbordante has a console version and an easy-to-use web application.
data-analytics data-cleaning data-cleansing data-engineering data-exploration data-mining data-profiling data-science data-wrangling data-preprocessing feature-selection feature-engineering feature-extraction spreadsheets tabular-data anomaly-detection data-mining-algorithms exploratory-data-analysis knowledge-discovery correlations
Language:C++ 419
jim-schwoebel / voicebook
🗣️ A book and repo to get you started programming voice computing applications in Python (10 chapters and 200+ scripts).
voice voice-assistant voice-recognition voice-recording transcription featurization data data-cleaning visualization generation voice-activity-detection voice-control server security encryption-decryption python3 machine-learning wake-word-detection voice-computing
Language:Python 386
msamogh / nonechucks
Deal with bad samples in your dataset dynamically, use Transforms as Filters, and more!
pytorch data-processing data-preprocessing data-pipeline data-cleaning preprocessing machine-learning torch
Language:Python 379
DataWithBaraa / sql-data-warehouse-project
A comprehensive guide to building a modern data warehouse with SQL Server, including ETL processes, data modeling, and analytics.
data-analysis data-analytics data-cleaning data-engineering data-lakehouse data-science data-warehouse data-warehousing datalake datascience datawarehouse datawarehousing etl etl-job etl-pipeline medallion-architecture sql sql-query sql-server sqlserver
Language:TSQL 319
rasgointelligence / feature-engineering-tutorials
Data Science Feature Engineering and Selection Tutorials
notebook tutorials tutorial python pandas data-science machine-learning scikit-learn feature-engineering feature-selection features xgboost pandas-profiling sweetviz pyrasgo jupyter exploratory-data-analysis data-cleaning
Language:Jupyter Notebook 286
ajaymache / data-analysis-using-python
Exploratory data analysis 📊using python 🐍of used car 🚘 database taken from ⓚ𝖆𝖌𝖌𝖑𝖊
data-science data-analysis data-visualization data-cleaning data-cleansing data-wrangling data-science-python data-analytics data-analysis-python eda exploratory-data-analysis kaggle-competition kaggle-dataset kaggle-used-cars-dataset
Language:Jupyter Notebook 227
probcomp / PClean
A domain-specific probabilistic programming language for scalable Bayesian data cleaning
probabilistic-programming probabilistic-graphical-models data-cleaning data-cleansing bayesian-inference
Language:Julia 221
genomoncology / FuzzTypes
Pydantic extension for annotating autocorrecting fields.
data-cleaning fuzzy-string-matching named-entity-linking pydantic
Language:Python 219
CambioML / uniflow-llm-based-pdf-extraction-text-cleaning-data-clustering
LLM-based text extraction from unstructured data like PDFs, Words and HTMLs. Transform and cluster the text into your desired format. Less information loss, more interpretation, and faster R&D!
data-cleaning generative-ai llm
Language:Python 208
charlesdedampierre / BunkaTopics
🗺️ Data Cleaning and Textual Data Visualization 🗺️
natural-language-processing nlp topic-modeling cartography llms machine-learning summarization data-cleaning explainability fine-tuning
Language:Python 187
BdR76 / CSVLint
CSV Lint plug-in for Notepad++ for syntax highlighting, csv validation, automatic column and datatype detecting, fixed width datasets, change datetime format, decimal separator, sort data, count unique values, convert to xml, json, sql etc. A plugin for data cleaning and working with messy data files.
csv notepad-plus-plus plugin fixed-width datasets metadata sorting sql syntax-highlighting tabular-data validate validation csv-to-sql quality data-quality validator mysql postgresql data-cleaning data-wrangling
Language:C# 182
jim-schwoebel / allie
🤖 An automated machine learning framework for audio, text, image, video, or .CSV files (50+ featurizers and 15+ model trainers). Python 3.6 required.
machine-learning deep-learning machine-learning-library machine-learning-api automl tpot data-augmentation data-cleaning datasets machine-learning-models ludwig voice-computing model-compression model-deployment data-visualization data-cleaning-pipeline data-transformation autokeras autopytorch
Language:Python 147
ekstroem / dataMaid
An R package for data screening
data-screening reproducible-research data-cleaning
Language:HTML 143
hi-primus / bumblebee
🚕 A spreadsheet-like data preparation web app that works over Optimus (Pandas, Dask, cuDF, Dask-cuDF, Spark and Vaex)
data-profiling data-cleaning bumblebee gui data-preparation python dask optimus gpu cudf dask-cudf prepare-data datasets
Language:Vue 141
Skytrax-Data-Warehouse
iam-mhaseeb / Skytrax-Data-Warehouse
A full data warehouse infrastructure with ETL pipelines running inside docker on Apache Airflow for data orchestration, AWS Redshift for cloud data warehouse and Metabase to serve the needs of data visualizations such as analytical dashboards.
python python3 database data-visualization data-analysis data-warehouse data-warehousing redshift airflow docker sql data-orchestration data-analytics s3 s3-bucket metabase data-engineering data-cleaning data-processing
Language:Python 138
KulikDM / pythresh
Outlier Detection Thresholding
contamination-detection data-cleaning filtering-algorithm outlier-detection outlier-selection thresholding
Language:Jupyter Notebook 128
aai-institute / pyDVL
pyDVL is a library of stable implementations of algorithms for data valuation and influence function computation
data-valuation shapley-value machine-learning transferlab least-core influence-functions game-theory data-quality robust-machine-learning data-centric-ai banzhaf-index data-cleaning data-pruning
Language:Python 124
xShaimaa / Data-Analysis-Projects
Practices on data analysis including: cleaning, visualization and EDA on different datasets using Python, SQL, Power BI, etc.
data-analysis data-visualization data-cleaning pandas matplotlib seaborn jupyterlab eda kaggle udacity-data-analyst-nanodegree dashboard datacamp powerbi powerquery udacity sql
Language:Jupyter Notebook 107
Iqrar99 / data-analytics-portfolio
Portfolio of data science and data analyst projects completed by me for academic, self learning, and hobby purposes.
data-analysis data-analyst data-analytics data-cleaning data-mining data-science data-visualisation data-visualization data-visualization-project deep-learning machine-learning portfolio python python-3
Language:Jupyter Notebook 105
ChrisMuir / refinr
Cluster and merge similar string values: an R implementation of Open Refine clustering algorithms
openrefine fuzzy-matching ngram approximate-string-matching data-cleaning data-clustering clustering cran r rstats
Language:C++ 104
opendataval / opendataval
OpenDataVal: a Unified Benchmark for Data Valuation in Python (NeurIPS 2023)
data-valuation machine-learning python research statistics data-centric-ai data-cleaning game-theory
Language:Python 96