data-profiling
There are 21 repositories under data-profiling topic.
ydataai / ydata-profiling
1 Line of code data quality profiling & exploratory data analysis for Pandas and Spark DataFrames.
Language:Python 13249cleanlab / cleanlab
Cleanlab's open-source library is the standard data-centric AI package for data quality and machine learning with messy, real-world data and labels.
Language:Python 11082great-expectations / great_expectations
Always know what to expect from your data.
Language:Python 10913- OpenMetadata
open-metadata / OpenMetadata
OpenMetadata is a unified metadata platform for data discovery, data observability, and data governance powered by a central metadata repository, in-depth column level lineage, and seamless team collaboration.
Language:TypeScript 7946 fbdesignpro / sweetviz
Visualize and compare datasets, target values and associations, with one line of code.
Language:Python 3056- soda-core
sodadata / soda-core
:zap: Data quality testing for the modern data stack (SQL, Spark, and Pandas) https://www.soda.io
Language:Python 2226 hi-primus / optimus
:truck: Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark
Language:Python 1524- odd-platform
opendatadiscovery / odd-platform
First open-source data discovery and observability platform. We make a life for data practitioners easy so you can focus on your business.
Language:Java 1365 Automatically find issues in image datasets and practice data-centric computer vision.
Language:Python 1131datavane / datavines
Know your data better！Datavines is Next-gen Data Observability Platform, support metadata manage and data quality.
Language:Java 687polyaxon / traceml
Engine for ML/Data tracking, visualization, explainability, drift detection, and dashboards for Polyaxon.
Language:Python 520- popmonLanguage:Python 509
- piperiderLanguage:Python 492
- Language:Python 451
- desbordante-core
Desbordante / desbordante-core
Desbordante is a high-performance data profiler that is capable of discovering many different patterns in data using various algorithms. It also allows to run data cleaning scenarios using these algorithms. Desbordante has a console version and an easy-to-use web application.
data-analyticsdata-cleaningdata-cleansingdata-engineeringdata-explorationdata-miningdata-profilingdata-sciencedata-wranglingdata-preprocessingfeature-selectionfeature-engineeringfeature-extractionspreadsheetstabular-dataanomaly-detectiondata-mining-algorithmsexploratory-data-analysisknowledge-discoverycorrelationsLanguage:C++ 429 - dqx
databrickslabs / dqx
Databricks framework to validate Data Quality of pySpark DataFrames
Language:Python 334 - dqo
dqops / dqo
Data Quality and Observability platform for the whole data lifecycle, from profiling new data sources to full automation with Data Observability. Configure data quality checks from the UI or in YAML files, let DQOps run the data quality checks daily to detect data quality issues.
Language:Java 171 🚕 A spreadsheet-like data preparation web app that works over Optimus (Pandas, Dask, cuDF, Dask-cuDF, Spark and Vaex)
Language:Vue 141- data-observability-installer
DataKitchen / data-observability-installer
Installer for DataKitchen's Open Source Data Observability Products. Data breaks. Servers break. Your toolchain breaks. Ensure your team is the first to know and the first to solve with visibility across and down your data estate. Save time with simple, fast data quality test generation and execution. Trust your data, tools, and systems end to end.
Language:Python 129 SJTU-DMTai / awesome-ml-data-quality-papers
Papers about training data quality management for ML models.
- swiple
Swiple / swiple
Swiple enables you to easily observe, understand, validate and improve the quality of your data
Language:Python 84 psebenick / data-profiling
a set of scripts to pull meta data and data profiling metrics from relational database systems
Language:Python 77apicrafter / metacrafter
Metadata and data identification tool and Python library. Identifies PII, common identifiers, language specific identifiers. Fully customizable and flexible rules
Language:Python 45Open-source metadata collector based on ODD Specification
Language:Python 44VIDA-NYU / auctus
Dataset search engine, discovering data from a variety of sources, profiling it, and allowing advanced queries on the index
Language:Python 44Client interface to Cleanlab Studio
Language:Python 32tsegall / fta
Metadata/data identification Java library. Identifies Semantic Type information (e.g. Gender, Age, Color, Country,...). Extensive country/language support. Extensible via user-defined plugins. Comprehensive Profiling support.
Language:Java 31ismaildawoodjee / GreatEx
A project for exploring how Great Expectations can be used to ensure data quality and validate batches within a data pipeline defined in Airflow.
Language:Python 23- Language:Python 22
- Language:Python 18
baligoyem / dataqtor
🔍Your Data Quality Detector / Gain insight into your data and get it ready for use before you start working with it 💡📊🛠💎
Language:Python 16Open Standard for Metadata. A Single place to Discover, Collaborate and Get your data right.
Language:TypeScript 15ahmadassaf / roomba
A Node.js tool to examine the correctness of Open Data Metadata and build custom dataset profiles
Language:JavaScript 12- Language:Python 12
LieseB-1746743 / data-cleaning
Data cleaning tool.
Language:JavaScript 9SebastianSchmidl / distod
DISTOD algorithm: Distributed discovery of bidirectional order dependencies
Language:Scala 9
