data-quality

There are 50 repositories under data-quality topic.

Made-With-ML
GokuMohandas / Made-With-ML
Learn how to design, develop, deploy and iterate on production-grade ML applications.
machine-learning deep-learning pytorch natural-language-processing data-science python mlops data-engineering data-quality distributed-ml llms ray distributed-training
Language:Jupyter Notebook 35617
applied-ml
eugeneyan / applied-ml
📚 Papers & tech blogs by companies sharing their work on data science & machine learning in production.
applied-machine-learning production applied-data-science machine-learning data-science reinforcement-learning data-engineering recsys search deep-learning data-quality data-discovery computer-vision natural-language-processing
25863
ydataai / ydata-profiling
1 Line of code data quality profiling & exploratory data analysis for Pandas and Spark DataFrames.
pandas-profiling pandas-dataframe statistics jupyter-notebook exploration data-science python pandas machine-learning deep-learning exploratory-data-analysis eda data-quality html-report data-exploration data-analysis jupyter big-data-analytics data-profiling hacktoberfest
Language:Python 12020
great-expectations / great_expectations
Always know what to expect from your data.
pipeline-tests dataquality datacleaning datacleaner data-science data-profiling pipeline pipeline-testing cleandata dataunittest data-unit-tests eda exploratory-data-analysis exploratory-analysis exploratorydataanalysis data-quality data-engineering pipeline-debt data-profilers mlops
Language:Python 9440
cleanlab / cleanlab
The standard data-centric AI package for data quality and machine learning with messy, real-world data and labels.
weak-supervision data-cleaning data-quality data-science noisy-labels data-centric-ai out-of-distribution-detection outlier-detection active-learning data-labeling data-profiling data-validation data-analysis labeling data-curation annotation dataops dataquality llms datasets
Language:Python 8606
fiftyone
voxel51 / fiftyone
The open-source tool for building high-quality datasets and computer vision models
machine-learning artificial-intelligence deep-learning computer-vision developer-tools data-science python active-learning data-centric-ai data-cleaning data-curation data-quality image-classification object-detection unstructured-data vector-search visualization
Language:Python 6654
kestra
kestra-io / kestra
Infinitely scalable, event-driven, language-agnostic orchestration and scheduling platform to manage millions of workflows declaratively in code.
workflow workflow-engine orchestration scheduler data-pipeline elt etl data data-engineering data-orchestration data-orchestrator low-code data-integration reverse-etl data-quality pipeline
Language:Java 6288
feast-dev / feast
Feature Store for Machine Learning
machine-learning features ml big-data feature-store python mlops data-engineering data-science data-quality
Language:Python 5251
OpenMetadata
open-metadata / OpenMetadata
Open Standard for Metadata. A Single place to Discover, Collaborate and Get your data right.
metadata datadiscovery data-science dataquality data-profiling metadata-management dataengineering data-catalog data-observability dbt data-discovery datacatalog data-contracts data-governance data-lineage data-validation snowflake data-quality data-quality-checks data-collaboration
Language:TypeScript 4100
lakeFS
treeverse / lakeFS
lakeFS - Data version control for your data lake | Git for data
data-engineering data-versioning go object-storage data-lake aws-s3 data-quality azure-blob-storage google-cloud-storage golang git-for-data lakefs apache-spark apache-sparksql hadoop-filesystem datalake datalakes data-version-control azure-storage
Language:Go 4058
datafold / data-diff
Compare tables within or across databases
database mysql postgresql snowflake rdbms trino data-engineering data-quality data-science data-quality-monitoring databricks-sql dataengineering dataquality oracle-database postgres sql dbt python data data-diffing
Language:Python 2832
mlops-course
GokuMohandas / mlops-course
Learn how to design, develop, deploy and iterate on production-grade ML applications.
machine-learning deep-learning pytorch mlops data-engineering data-quality data-science distributed-ml llms natural-language-processing python ray
Language:Jupyter Notebook 2738
whylabs / whylogs
An open-source data logging library for machine learning models and data pipelines. 📚 Provides visibility into data quality & model performance over time. 🛡️ Supports privacy-preserving data collection, ensuring safety & robustness. 📈
ai-pipelines approximate-statistics statistical-properties data-quality calculate-statistics python logging mlops dataops ml-pipelines data-pipeline dataset machine-learning data-science analytics constraints data-constraints model-performance
Language:Jupyter Notebook 2540
feathr-ai / feathr
Feathr – A scalable, unified data and AI engineering platform for enterprise
feature-engineering feature-store artificial-intelligence mlops data-engineering data-quality machine-learning apache-spark azure data-science feature-management feature-marketplace feature-governance feature-metadata feature-platform
Language:Scala 1927
soda-core
sodadata / soda-core
:zap: Data quality testing for the modern data stack (SQL, Spark, and Pandas) https://www.soda.io
python data-engineering data-governance data-monitoring data-observability data-profiling data-quality data-quality-checks data-quality-monitoring data-reliability data-testing data-unit-tests data-validation dataquality datatesting dbt pipeline-testing snowflake data-quality-testing data-contracts
Language:Python 1746
featureform
featureform / featureform
The Virtual Feature Store. Turn your existing data infrastructure into a feature store.
machine-learning data-science vector-database embeddings-similarity embeddings hacktoberfest feature-store mlops data-quality feature-engineering ml python
Language:Jupyter Notebook 1679
re-data / re-data
re_data - fix data issues before your users & CEO would discover them 😊
data-monitoring data-analysis data-quality data-quality-monitoring open-source-tooling data-observability dataquality data-testing data-quality-checks dbt dbt-packages data-reliability
Language:HTML 1521
odd-platform
opendatadiscovery / odd-platform
First open-source data discovery and observability platform. We make a life for data practitioners easy so you can focus on your business.
oss data-platform metadata metadata-management data-pipelines data-engineering observability data-catalog datacatalog data-discovery data-lineage bigdata alerting lineage data-profiling data-exploration data-governance data-quality data-science data-observability
Language:Java 1108
data-centric-AI
daochenzha / data-centric-AI
A curated, but incomplete, list of data-centric AI resources.
artificial-intelligence data-centric-ai ai machine-learning data-curation data-centric data-centric-machine-learning data-science data-quality data-engineering
970
cleanlab / cleanvision
Automatically find issues in image datasets and practice data-centric computer vision.
computer-vision data-centric-ai data-exploration data-quality data-validation deep-learning exploratory-data-analysis image-analysis image-classification image-generation image-quality image-segmentation data-profiling data-science
Language:Python 919
pointblank
rstudio / pointblank
Data quality assessment and metadata reporting for data frames and database tables
data-validation database-tables data-dictionaries easy-to-understand data-frames reporting-tool data-profiler data-management schema-validation data-verification data-checker data-assertions yaml-configuration data-inference testing-tools data-quality
Language:R 824
WeBankFinTech / Qualitis
Qualitis is a one-stop data quality management platform that supports quality verification, notification, and management for various datasource. It is used to solve various data quality problems caused by data processing. https://github.com/WeBankFinTech/Qualitis
quality quality-check quality-improvement data-quality linkis dss datashperestudio workflow data-quality-model compare
Language:Java 657
Failed-ML
kennethleungty / Failed-ML
Compilation of high-profile real-world examples of failed machine learning projects
ai artificial-intelligence data-science deep-learning machine-learning ml failed-ml computer-vision data-engineering data-quality natural-language-processing production recsys failed-machine-learning failed-data-science regression forecasting classification fml
615
awesome-data-catalogs
opendatadiscovery / awesome-data-catalogs
📙 Awesome Data Catalogs and Observability Platforms.
data-catalog data-discovery metadata dataops awesome observability data-engineering data-quality big-data opensource open-source ml awesome-list oss opendata datadiscovery metadata-management datacatalog
577
polyaxon / traceml
Engine for ML/Data tracking, visualization, explainability, drift detection, and dashboards for Polyaxon.
pandas pandas-summary dataframes data-science spark dask plotly statistics matplotlib data-profiling data-visualization data-exploration dataops mlops data-quality data-quality-checks explainable-ai pytorch tensorflow tracking
Language:Python 493
piperider
InfuseAI / piperider
Code review for data in dbt
data-pipeline data-profiling data-quality data-science data-exploration eda exploratory-data-analysis data-testing python data-observability data-profiler data-reliability data-visualization dbt dbt-metrics code-review reporting pull-requests continuous-integration
Language:Python 466
encord-team / encord-active
The toolkit to test, validate, and evaluate your models and surface, curate, and prioritize the most valuable data for labeling.
computer-vision data data-science data-validation deep-learning machine-learning ml mlops python active-learning annotations data-centric data-cleaning data-quality label-errors model-quality noisy-labels label-quality object-detection
Language:Python 420
Data-Centric-AI-Community / awesome-data-centric-ai
Open-Source Software, Tutorials, and Research on Data-Centric AI 🤖
ai artificial-intelligence awesome awesome-list data data-labelling data-quality data-science data-visualization machine-learning open-source pandas python python-library research synthetic-data tools tutorials
Language:Jupyter Notebook 300
data-drift / data-drift
Metrics Observability & Troubleshooting
context data-monitoring data-observability data-quality dbt metrics semantic-layer dbt-metrics analytics data-diffing data-version-control drill-down reconciliation bigquery data-governance data-lineage data-reliability dbt-packages redshift snowflake
Language:HTML 296
alibaba / feathub
FeatHub - A stream-batch unified feature store for real-time machine learning
apache-flink data-engineering data-science feature-engineering feature-store machine-learning streaming data data-quality mlops
Language:Python 293
ubisoft / mobydq
:whale: Tool to automate data quality checks on data pipelines
data-quality data-pipeline data-warehouse big-data data-quality-checks data-quality-monitoring
Language:Vue 239
frederick0329 / TracIn
Implementation of Estimating Training Data Influence by Tracing Gradient Descent (NeurIPS 2020)
data-quality influence
Language:Jupyter Notebook 204
open-data-contract-standard
bitol-io / open-data-contract-standard
Home of the Open Data Contract Standard (ODCS).
data data-contract data-contracts data-engineering data-mesh data-quality
196
adidas / lakehouse-engine
The Lakehouse Engine is a configuration driven Spark framework, written in Python, serving as a scalable and distributed engine for several lakehouse algorithms, data flows and utilities for Data Products.
big-data configuration-driven data-engineering data-quality databricks delta-lake framework great-expectations lakehouse spark
Language:Python 183
Hyhyhyhyhyhyh / Django-Data-quality-system
数据治理、数据质量检核/监控平台（Django+jQuery+MySQL）
data-quality data-quality-checks data-quality-monitor data-quality-monitoring
Language:Python 178
whylabs / whylogs-java
Profile and monitor your ML data pipeline end-to-end
ai-pipelines aiops apache-spark approximate-statistics calculate-statistics data-quality dataset java mlops spark statistical-properties statistics whylogs
Language:Java 176