Data Quality Profiler and Rules Engine
Provides the following:
- Data Profilers for large volume data profiling in Spark
- Assertion rule definitions and checking
- Reference data loading and joining
- Excel and CSV reference data parsing
- JSON output enriched with data quality markers/profilers
- Metrics and summary dataframe output
- Dimensional tagging of profiler outputs (additional identifiers)
- JSON flattener
- JSON and CSV loader, extensible to other formats
- Custom key pre-processor and custom parquet row reader functionality
- Comprehensive built-in assertion rules modules, extensible
- Built-in set of field-level profile masks
- Compound assertion rule definition (i.e. a set of sub-rules must all pass)
- Human-readable Data Quality and Assertion Rule Compliance report output
Repository Layout
- Data Quality Profiler and Rules Engine code: data-quality-profiler
- Examples and Usage: examples
Licence
Licensed under the MIT License. See LICENSE