giuseppelt/grai-core

Documentation • Website • Slack Community • Want to Chat?

Introduction

Data lineage made simple. Grai makes it easy to understand and test how your data relates across databases, warehouses, APIs and dashboards.

Pre-built connectors. Automatically synchronize lineage from across the stack so your metadata is never out of date.
Centralized data tests. Write data validation tests which run whenever upstream data sources change.
Integrated with GitHub. Run data validation tasks as part of your CI/CD process to test changes everywhere your data is used.
Your data, your cloud. Grai is fully open sourced and self hosted. You maintain full control over your data and hosting environment.

How it works

Automatically build column level lineage spanning your warehouse and production services with connectors for dbt, Snowflake, Fivetran and more (see below).
Get alerts in your CI/CD workflows whenever changes to a production system will impact your warehouse or dbt projects with GitHub Actions.
Self host the project or run it in the Grai Cloud for free.

Connectors

We currently support pre-built integrations for a variety of sources including

	integration	install
	Snowflake	`pip install grai-source-snowflake`
	BigQuery	`pip install grai-source-bigquery`
	Redshift	`pip install grai-source-redshift`
	Postgres	`pip install grai-source-postgres`
	MySQL	`pip install grai-source-mysql`
	SQL Server	`pip install grai-source-mssql`
	dbt	`pip install grai-source-dbt`
	Fivetran	`pip install grai-source-fivetran`
	csv, parquet, feather	`pip install grai-source-flat-file`
	Metabase (alpha)	`pip install grai-source-metabase`

Quickstart

You can find a full quickstart guide in the documentation which covers deploying your own instance of Grai and getting set up with your first connector in python.

Running Locally

You can always find pre-built images of the backend server at ghcr.io/grai-io/grai-core/grai-server:latest and the frontend at ghcr.io/grai-io/grai-core/grai-frontend:latest, however, if you prefer to build from source, you can do so with docker compose.

git clone https://github.com/grai-io/grai-core
cp examples/deployment/docker-compose/docker-compose.yml
docker compose up

The backend server will be available at http://localhost:8000/ and the frontend is now here http://localhost:3000/.

Default login credentials:

username: null@grai.io
password: super_secret

After logging in and connecting a data source you'll be greeted with a lineage graph looking something like this

For more information about using the web application check out the getting started guide.

Other Deployment Mechanisms

You can find example configurations for docker compose and Kubernetes in the examples folder.

Helm

We also publish a set of Helm charts which are available if you prefer.

helm repo add grai https://charts.grai.io
helm install grai grai/grai

Other Features

CLI Library

Programmatically interact with your data lineage from the command line using the grai-cli. Although we strongly encourage using something like pipx, for installing the cli it's otherwise as simple as

pip install grai-cli

After installation you can invoke the CLI with the command grai. The first time you run the CLI you should be greeted with installation instructions, but you can always rerun them with the command grai config init.

Python Client Library

The client library provides programmatic access to the data lineage server. You can use the client library for any programmatic functionality you want to add on top of the core Grai installation.

pip install grai-client

To work with the client, learn more here.

Repo Activity

Community

Email us: founders@grai.io

Join us on Slack:

Check us out at www.grai.io

giuseppelt / grai-core