DataHub: A Generalized Metadata Search & Discovery Tool
📣Next DataHub town hall meeting on May 27th, 9am-10am PDT (convert to your local time)
- Monthly project update: Apr 2021 Edition.
- Unleashing Excellent DataOps with LinkedIn DataHub: DataOps Unleashed Talk.
- Latest blog post DataHub: Popular Metadata Architectures Explained @ LinkedIn Engineering Blog.
- We've released v0.7.1. You can find release notes here
- We're on Slack now! Ask questions and keep up with the latest announcements.
DataHub is LinkedIn's generalized metadata search & discovery tool. Read about the architectures of different metadata systems and why DataHub excels here. Also read our LinkedIn Engineering blog post, check out our Strata presentation and watch our Crunch Conference Talk. You should also visit DataHub Architecture to get a better understanding of how DataHub is implemented and DataHub Onboarding Guide to understand how to extend DataHub for your own use cases.
Please follow the DataHub Quickstart Guide to get a copy of DataHub up & running locally using Docker. As the guide assumes some basic knowledge of Docker, we'd recommend you to go through the "Hello World" example of A Docker Tutorial for Beginners if Docker is completely foreign to you.
Demo and Screenshots
There's a hosted demo environment where you can play around with DataHub before installing.
Source Code and Repositories
- linkedin/datahub: This repository contains the complete source code for both DataHub's frontend & backend services. We currently follow a hybrid open source model for development in this repository. See this blog post for details on how we do it.
- linkedin/datahub-gma: This repository contains the source code for DataHub's metadata infrastructure libraries (Generalized Metadata Architecture, or GMA). We follow an open-source-first model for development in this repository.
We have documentation available at https://datahubproject.io/docs/.
Frequently Asked Questions about DataHub can be found here.
Features & Roadmap
Here are the companies that have officially adopted DataHub. Please feel free to add yours to the list if we missed it.
- Expedia Group
- Saxo Bank
- Shanghai HuaRui Bank
Here is a list of companies that are currently building POC or seriously evaluating DataHub.
- Bizzy Group
- City of New York, DoITT
- Kindred Group
- Morgan Stanley
- Orange Telecom
- REEF Technology
- Sysco AS
- University of Phoenix
Select Articles & Talks
- DataHub: A Generalized Metadata Search & Discovery Tool
- DataHub: Popular Metadata Architectures Explained
- Open sourcing DataHub: LinkedIn’s metadata search and discovery platform
- Driving DataOps Culture with LinkedIn DataHub @ DataOps Unleashed 2021
- DataHub: Powering LinkedIn's Metadata @ Budapest Data Forum 2020
- Taming the Data Beast Using DataHub @ Data Engineering Melbourne Meetup November 2020
- Metadata Management And Integration At LinkedIn With DataHub @ Data Engineering Podcast
- The evolution of metadata: LinkedIn’s story @ Strata Data Conference 2019
- Journey of metadata at LinkedIn @ Crunch Data Conference 2019
- DataHub Journey with Expedia Group
- Saxo Bank's Data Workbench
- Data Discoverability at SpotHero
- Data Catalogue — Knowing your data
- LinkedIn DataHub Application Architecture Quick Understanding
- A Dive Into Metadata Hubs
- 25 Hot New Data Tools and What They DON’T Do
- Emerging Architectures for Modern Data Infrastructure
See the full list here.