data-quality

There are 53 repositories under data-quality topic.

Made-With-ML
GokuMohandas / Made-With-ML
Learn how to design, develop, deploy and iterate on production-grade ML applications.
data-engineering data-quality data-science deep-learning distributed-ml distributed-training llms machine-learning mlops natural-language-processing python pytorch ray
Language:Jupyter Notebook 37165
applied-ml
eugeneyan / applied-ml
📚 Papers & tech blogs by companies sharing their work on data science & machine learning in production.
applied-data-science applied-machine-learning computer-vision data-discovery data-engineering data-quality data-science deep-learning machine-learning natural-language-processing production recsys reinforcement-learning search
27195
ydataai / ydata-profiling
1 Line of code data quality profiling & exploratory data analysis for Pandas and Spark DataFrames.
pandas-profiling pandas-dataframe statistics jupyter-notebook exploration data-science python pandas machine-learning deep-learning exploratory-data-analysis eda data-quality html-report data-exploration data-analysis jupyter big-data-analytics data-profiling hacktoberfest
Language:Python 12423
great-expectations / great_expectations
Always know what to expect from your data.
pipeline-tests dataquality datacleaning datacleaner data-science data-profiling pipeline pipeline-testing cleandata dataunittest data-unit-tests eda exploratory-data-analysis exploratory-analysis exploratorydataanalysis data-quality data-engineering pipeline-debt data-profilers mlops
Language:Python 9863
cleanlab / cleanlab
The standard data-centric AI package for data quality and machine learning with messy, real-world data and labels.
weak-supervision data-cleaning data-quality data-science noisy-labels data-centric-ai out-of-distribution-detection outlier-detection active-learning data-labeling data-profiling data-validation labeling data-curation annotation dataops dataquality llms datasets exploratory-data-analysis
Language:Python 9446
kestra
kestra-io / kestra
Orchestration and automation platform to execute millions of scheduled and event-driven workflows declaratively in code and from the UI
workflow workflow-engine orchestration scheduler data-pipeline elt etl data data-engineering data-orchestration data-orchestrator low-code data-integration reverse-etl data-quality pipeline hacktoberfest
Language:Java 9341
fiftyone
voxel51 / fiftyone
The open-source tool for building high-quality datasets and computer vision models
active-learning artificial-intelligence computer-vision data-centric-ai data-cleaning data-curation data-quality data-science deep-learning developer-tools image-classification machine-learning object-detection python unstructured-data vector-search visualization
Language:Python 8520
feast-dev / feast
The Open Source Feature Store for Machine Learning
big-data data-engineering data-quality data-science feature-store features machine-learning ml mlops python
Language:Python 5517
OpenMetadata
open-metadata / OpenMetadata
OpenMetadata is a unified metadata platform for data discovery, data observability, and data governance powered by a central metadata repository, in-depth column level lineage, and seamless team collaboration.
data-catalog data-collaboration data-contracts data-discovery data-governance data-lineage data-observability data-profiling data-quality data-quality-checks data-science data-validation datacatalog datadiscovery dataengineering dataquality dbt metadata metadata-management snowflake
Language:TypeScript 5274
evidentlyai / evidently
Evidently is an open-source ML and LLM observability framework. Evaluate, test, and monitor any AI-powered system or data pipeline. From tabular data to Gen AI. 100+ metrics.
data-drift data-quality data-science data-validation generative-ai hacktoberfest html-report jupyter-notebook llm llmops machine-learning mlops model-monitoring pandas-dataframe
Language:Jupyter Notebook 5193
lakeFS
treeverse / lakeFS
lakeFS - Data version control for your data lake | Git for data
apache-spark apache-sparksql aws-s3 azure-blob-storage azure-storage data-engineering data-lake data-quality data-version-control data-versioning datalake datalakes git-for-data go golang google-cloud-storage hadoop-filesystem lakefs object-storage
Language:Go 4375
datafold / data-diff
Compare tables within or across databases
data data-diffing data-engineering data-quality data-quality-monitoring data-science database databricks-sql dataengineering dataquality dbt mysql oracle-database postgres postgresql python rdbms snowflake sql trino
Language:Python 2935
mlops-course
GokuMohandas / mlops-course
Learn how to design, develop, deploy and iterate on production-grade ML applications.
data-engineering data-quality data-science deep-learning distributed-ml llms machine-learning mlops natural-language-processing python pytorch ray
Language:Jupyter Notebook 2915
whylabs / whylogs
An open-source data logging library for machine learning models and data pipelines. 📚 Provides visibility into data quality & model performance over time. 🛡️ Supports privacy-preserving data collection, ensuring safety & robustness. 📈
ai-pipelines approximate-statistics statistical-properties data-quality calculate-statistics python logging mlops dataops ml-pipelines data-pipeline dataset machine-learning data-science analytics constraints data-constraints model-performance
Language:Jupyter Notebook 2635
feathr-ai / feathr
Feathr – A scalable, unified data and AI engineering platform for enterprise
apache-spark artificial-intelligence azure data-engineering data-quality data-science feature-engineering feature-governance feature-management feature-marketplace feature-metadata feature-platform feature-store machine-learning mlops
Language:Scala 1978
soda-core
sodadata / soda-core
:zap: Data quality testing for the modern data stack (SQL, Spark, and Pandas) https://www.soda.io
data-contracts data-engineering data-governance data-monitoring data-observability data-profiling data-quality data-quality-checks data-quality-monitoring data-quality-testing data-reliability data-testing data-unit-tests data-validation dataquality datatesting dbt pipeline-testing python snowflake
Language:Python 1873
featureform
featureform / featureform
The Virtual Feature Store. Turn your existing data infrastructure into a feature store.
data-quality data-science embeddings embeddings-similarity feature-engineering feature-store hacktoberfest machine-learning ml mlops python vector-database
Language:Jupyter Notebook 1800
re-data / re-data
re_data - fix data issues before your users & CEO would discover them 😊
data-analysis data-monitoring data-observability data-quality data-quality-checks data-quality-monitoring data-reliability data-testing dataquality dbt dbt-packages open-source-tooling
Language:HTML 1543
odd-platform
opendatadiscovery / odd-platform
First open-source data discovery and observability platform. We make a life for data practitioners easy so you can focus on your business.
oss data-platform metadata metadata-management data-pipelines data-engineering observability data-catalog datacatalog data-discovery data-lineage bigdata alerting lineage data-profiling data-exploration data-governance data-quality data-science data-observability
Language:Java 1204
data-centric-AI
daochenzha / data-centric-AI
A curated, but incomplete, list of data-centric AI resources.
artificial-intelligence data-centric-ai ai machine-learning data-curation data-centric data-centric-machine-learning data-science data-quality data-engineering
1033
cleanlab / cleanvision
Automatically find issues in image datasets and practice data-centric computer vision.
computer-vision data-centric-ai data-exploration data-quality data-validation deep-learning exploratory-data-analysis image-analysis image-classification image-generation image-quality image-segmentation data-profiling data-science
Language:Python 1009
pointblank
rstudio / pointblank
Data quality assessment and metadata reporting for data frames and database tables
data-validation database-tables data-dictionaries easy-to-understand data-frames reporting-tool data-profiler data-management schema-validation data-verification data-checker data-assertions yaml-configuration data-inference testing-tools data-quality
Language:R 867
Failed-ML
kennethleungty / Failed-ML
Compilation of high-profile real-world examples of failed machine learning projects
ai artificial-intelligence data-science deep-learning machine-learning ml failed-ml computer-vision data-engineering data-quality natural-language-processing production recsys failed-machine-learning failed-data-science regression forecasting classification fml
709
WeBankFinTech / Qualitis
Qualitis is a one-stop data quality management platform that supports quality verification, notification, and management for various datasource. It is used to solve various data quality problems caused by data processing. https://github.com/WeBankFinTech/Qualitis
quality quality-check quality-improvement data-quality linkis dss datashperestudio workflow data-quality-model compare
Language:Java 695
awesome-data-catalogs
opendatadiscovery / awesome-data-catalogs
📙 Awesome Data Catalogs and Observability Platforms.
data-catalog data-discovery metadata dataops awesome observability data-engineering data-quality big-data opensource open-source ml awesome-list oss opendata datadiscovery metadata-management datacatalog
683
polyaxon / traceml
Engine for ML/Data tracking, visualization, explainability, drift detection, and dashboards for Polyaxon.
pandas pandas-summary dataframes data-science spark dask plotly statistics matplotlib data-profiling data-visualization data-exploration dataops mlops data-quality data-quality-checks explainable-ai pytorch tensorflow tracking
Language:Python 501
NVIDIA / NeMo-Curator
Scalable data pre processing and curation toolkit for LLMs
data data-curation data-prep data-preparation data-processing data-processing-pipelines data-quality datacuration datarecipes deduplication fast-data-processing fine-tuning large-language-models large-scale-data-processing llm llm-data-quality llmapps python semantic-deduplication
Language:Jupyter Notebook 482
piperider
InfuseAI / piperider
Code review for data in dbt
data-pipeline data-profiling data-quality data-science data-exploration eda exploratory-data-analysis data-testing python data-observability data-profiler data-reliability data-visualization dbt dbt-metrics code-review reporting pull-requests continuous-integration
Language:Python 480
encord-team / encord-active
The toolkit to test, validate, and evaluate your models and surface, curate, and prioritize the most valuable data for labeling.
computer-vision data data-science data-validation deep-learning machine-learning ml mlops python active-learning annotations data-centric data-cleaning data-quality label-errors model-quality noisy-labels label-quality object-detection
Language:Python 433
open-data-contract-standard
bitol-io / open-data-contract-standard
Home of the Open Data Contract Standard (ODCS).
data data-contract data-contracts data-engineering data-mesh data-quality standard
Language:Shell 321
data-drift / data-drift
Metrics Observability & Troubleshooting
context data-monitoring data-observability data-quality dbt metrics semantic-layer dbt-metrics analytics data-diffing data-version-control drill-down reconciliation bigquery data-governance data-lineage data-reliability dbt-packages redshift snowflake
Language:HTML 317
Data-Centric-AI-Community / awesome-data-centric-ai
Open-Source Software, Tutorials, and Research on Data-Centric AI 🤖
ai artificial-intelligence awesome awesome-list data data-labelling data-quality data-science data-visualization machine-learning open-source pandas python python-library research synthetic-data tools tutorials
Language:Jupyter Notebook 315
alibaba / feathub
FeatHub - A stream-batch unified feature store for real-time machine learning
apache-flink data-engineering data-science feature-engineering feature-store machine-learning streaming data data-quality mlops
Language:Python 313
ubisoft / mobydq
:whale: Tool to automate data quality checks on data pipelines
big-data data-pipeline data-quality data-quality-checks data-quality-monitoring data-warehouse
Language:Vue 246
adidas / lakehouse-engine
The Lakehouse Engine is a configuration driven Spark framework, written in Python, serving as a scalable and distributed engine for several lakehouse algorithms, data flows and utilities for Data Products.
big-data configuration-driven data-engineering data-quality databricks delta-lake framework great-expectations lakehouse spark
Language:Python 217
frederick0329 / TracIn
Implementation of Estimating Training Data Influence by Tracing Gradient Descent (NeurIPS 2020)
data-quality influence
Language:Jupyter Notebook 215