There are 16 repositories under data-ingestion topic.
ingestr is a CLI tool to copy data between any databases with a single command seamlessly.
Apache Paimon(incubating) is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations.
A Python library that enables ML teams to share, load, and transform data in a collaborative, flexible, and efficient way :chestnut:
Orbital automates integration between data sources (APIs, Databases, Queues and Functions). BFF's, API Composition and ETL pipelines that adapt as your specs change.
The Data Engineering Book - หนังสือวิศวกรรมข้อมูล ของคนไทย เพื่อคนไทย
Apache Spark examples exclusively in Java
Squirrel dataset hub
Enables custom tracing of Java applications in Dynatrace
Sample code for the AWS Big Data Blog Post Building a scalable streaming data processor with Amazon Kinesis Data Streams on AWS Fargate
Download and warehouse historical trading data
Enables custom tracing of Python applications in Dynatrace
The Data Integration Library project provides a library of generic components based on a multi-stage architecture for data ingress and egress.
Describes technical concepts of Dynatrace OneAgent SDK
Enables custom tracing of native applications in Dynatrace
Enables custom tracing of Node.js applications in Dynatrace
Enables custom tracing of .NET applications in Dynatrace
Airbyte clone written in Go and Vue.js. Works with Airbyte connectors.
OpenKit .NET Reference Implementation
Product scraping from Walmart Canada website, with further cleaning and integration of data from a different store.
Dynatrace agent for PaaS environments
End-to-end data engineering processes for the NIGERIA Health Facility Registry (HFR). The project leveraged Selenium, Pandas, PySpark, PostgreSQL and Airflow
Shift is a high performance better alternative to Airbyte, Singer, Meltano
This script automates the Dynatrace Agent installation for Azure WebApps
A Broadway producer for Redis lists
A Demo Project Implementing CSV Ingestion Pipelines using the Broadway Framework (Elixir)
Make your development databases gobble up known data