Awesome Data Contracts
Data contracts is an amazing initiative to bring data producers and data consumers together. This is a curated list of awesome articles, tools, resources that will help you understand concepts and start your data contracts journey.
Articles and Blogs
- Improving data quality with data contracts (GoCardless) What are data contracts and GoCardless implementation of data contracts
- Interfaces and Breaking Stuff by Tristan Handy (CEO, DBT) Need of data contracts as data interfaces and analogy with software APIs
- Implementing data contracts at GoCardless Data contract specification examples from GoCardless systems
- 7 key learnings from implementing data contracts (Bar Moses and Andrew Jones) Sample data contract example and learnings from implementations
- A PM's thoughts on data contracts A product manager’s view on data contracts and challenges they can solve
- Engineer’s guide to data contracts part I - entities covers implementing data contracts for entities
- Engineer’s guide to data contracts part II - application events cover enforcing and monitoring data contracts using transactional outbox pattern
- Data Contracts: The Mesh Glue Scope of data contract and how it can be a layer with data mesh architectures
- Yet another post on data contracts - part I Thoughts from David J. on what is data contracts and what it should contain
- Yet another post on data contracts - part II What kind of data should be covered by data contracts and some highlights of tooling that’s available
- Yet another post on data contracts - part III Deeper dive on current tooling that can be used for data contracts implementation
- Are we setting data contracts in the right place? Considerations and analogies from software architecture world while defining data contracts
- Implementing data contracts Specifications and real world architecture example of data contract implementation
- The best data contract is the pull request Datafold’s view on how the best data contract is pull request
- An open letter to data ninjas - yes, you need to implement data contracts View on implementations and specifications of data contracts
- Data person: attorney at law Interesting view on data contracts which turn data engineers into data lawyers by Stephen Bailey
- Demystifying event streams - New event streams model at Merit and event specifications being treated as data contracts - named OMG contract
- Fine let’s talk about data contracts - Different view on viability and usability of data contracts
- Brain, kidney, cancer research and data contracts (Schemata project) - Example of data contract explanation / discussion in simple terms for people outside of data space
- The rise of data contracts - (Q/A style article on what are data contracts and why we need them)
- Data contracts from zero to hero - a pragmatic approach to data contracts
- Data contracts: ensure robustness in your data mesh architecture - Data contracts as part of metadata driven ingestion network
- Implementing data contracts - Implementing basic schema based data contract with Kafka schema registry
- Data contracts - ensure robustness in your data mesh architecture - Interesting view on using data sharing agreements in data mesh architecutre and how data contracts validation should be done as data products sidecar
- Data contracts wrapped 2022 - Data contracts creation, storage, enforcement and application
- McDonald's event driven architecture and data journey - Use of data contracts in McDonald's event driven architecture
- Implementing data contracts in data warehouse - Data contracts validation in data warehouse and downstream systems and not just limited to source systems.
Videos
- Inventing data contracts and building data for a FinTech unicorn with Andrew Jones of GoCardless
- Data contracts and domain ownership
- How data contracts can kill innovation
- Getting jiggy with jsonschema: The power of contracts for building data systems
- Data Contracts: Accountable Data Quality w/ Chad Sanderson - UDEM October 2022
- Data Contract Battle Royale w/ Chad Sanderson vs Ethan Aaron
- Driving ML data quality with data contracts
Podcasts
- Why You'll Need Data Contracts (w/ Chad Sanderson + Prukalpa)
- The Analytics Engineering Podcast (11/18/2022)
- Data Contracts & Domain Ownership w/ Ananth Packkildurai
- Monday Morning Data Chat (5/24/2022)
Tools
- Avo.app - Avo is a data governance platform that error-proofs your analytics events and speeds up implementation.
- Reflekt - Define tracking plans as code for analytics
- Iteratively - Bring your data teams, product managers and engineers together to define, instrument, verify and collaborate on analytics tracking
- Buz - Buz collects, validates, and delivers schematized data to where it needs to bee
- Benthos - Benthos is a high performance and resilient stream processor, able to connect various sources and sinks in a range of brokering patterns and perform hydration, enrichments, transformations and filters on payloads.
- Schemata - Schemata is a schema modeling framework for decentralized domain-driven ownership of data. Schemata combines a set of standard metadata definitions for each schema & data field and a scoring algorithm to provide a feedback loop on how efficient the data modeling of your data warehouse is.
Related
- Establish formal data contracts between consumers and producers Older article from 2018 that explored the concept of data contracts (Older generation article)
- Data Mesh — A Data Movement and Processing Platform @ Netflix - Schema enforcement in Netflix’s data mesh platform
Contributing
Your contributions are most welcome! Please go ahead and create a pull request. We'll review each and every one. We will keep some pull requests open if we not sure whether the PRs are awesome or relevant, you could vote for them by adding 👍 to them. Pull requests will be merged when their votes reach 5.