atgoud / mobydq

:whale: Tool to automate data quality checks on data pipelines

MobyDQ

MobyDQ is a tool for data engineering teams to automate data quality checks on their data pipeline, capture data quality issues and trigger alerts in case of anomaly, regardless of the data sources they use.

This tool has been inspired by an internal project developed at Ubisoft Entertainment in order to measure and improve the data quality of its Enterprise Data Platform. However, this open source version has been reworked to improve its design, simplify it and remove technical dependencies with commercial software.

Getting Started

Skip the bla bla and run your data quality indicators by following the Getting Started page. The complete documentation is also available on Github Pages: https://mobydq.github.io.

Run Tests

You can run all tests locally using the following commands:

$ cd mobydq
$ # Backend
$ test/run-tests.sh
$ # Frontend
$ app/run-container.sh npm run test

Run Linter

Depending on the used editor, eslint and pylint can be integrated. You can run all linters locally using the following commands:

$ cd mobydq
$ # Backend
$ test/run-linter.sh
$ # Frontend
$ app/run-container.sh npm run lint

Dependencies

Docker Images

The containers run by docker-compose have dependencies with the following Docker images:

• postgres (tag: 11.0-alpine)
• graphile/postgraphile (tag: latest)
• python (tag: 3.6.6-alpine3.8)
• python (tag: 3.6.6-slim-stretch)
• nginx (tag: latest)

Python Packages

• docker (3.5.0)
• flask (1.0.2)
• flask_cors (3.0.6)
• flask_restplus (0.11.0)
• graphql_py (0.7.1)
• jinja2 (2.10.0)
• jwt (0.5.4)
• numpy (1.14.0)
• pandas (0.23.0)
• pyodbc (4.0.23)
• requests (2.20.0)
• requests_oauthlib(1.0.0)