rmendesl / datahub

A Generalized Metadata Search & Discovery Tool

DataHub: A Generalized Metadata Search & Discovery Tool

✨Feb 2020 Update:

• Our blog post on open sourcing DataHub is out!
• DataHub v0.3.0 is released!

Introduction

DataHub is LinkedIn's generalized metadata search & discovery tool. To learn more about DataHub, check out our LinkedIn blog post and Strata presentation. You should also visit DataHub Architecture to get a better understanding of how DataHub is implemented and DataHub Onboarding Guide to understand how to extend DataHub for your own use case.

This repository contains the complete source code for both DataHub's frontend & backend. You can also read about how we sync the changes between our the internal fork and GitHub.

Quickstart

1. Install docker and docker-compose. Make sure to configure Docker to allocate enough hardware resources for Docker engine. Tested & confirmed config: 4 CPUs, 8GB RAM, 2GB Swap area.
2. Open Docker either from the command line or the Desktop app and ensure it is up and running.
3. Clone this repo and cd into the root directory for the cloned repository.
4. Run below command to download and run all Docker containers in your local:

   cd docker/quickstart && docker-compose pull && docker-compose up --build
   

   This step takes long time and it might be hard to figure out when DataHub is fully up. You can refer to this guide to verify if DataHub is up and running.
5. At this point, you should be able to start DataHub by opening http://localhost:9001 in your browser. You can sign in using datahub as both username and password. However, there is no data just yet.
6. To ingest provided sample data to DataHub, switch to a new terminal, cd into the cloned datahub repo, and run below command:

   docker build -t ingestion -f docker/ingestion/Dockerfile . && cd docker/ingestion && docker-compose up
   

   After running this, you should be able to see sample data in DataHub.

Refer to debugging guide if you have issues in any of the above steps.

Quicklinks

• DataHub Architecture
• DataHub Onboarding Guide
• Docker Images
• Frontend App
• Web Client App
• Generalized Metadata Service
• Metadata Consumer Jobs
• Metadata Ingestion

Releases

See Releases page for more details.

Contributing

We welcome contributions from the community. Please refer to the guidelines for more details. We also have a contrib directory for incubation.

Roadmap

Check out DataHub's roadmap.