esakik / data-engineering-essentials

Samples related to data engineering, e.g. spark, embulk, airflow, etc.

Data Platform Index

Samples and documents of data platform.

Data Format

• Protocol Buffers
• Apache Avro

Data Collector

• Fluentd (Streaming)

• Embulk (Batch)

Data Pipeline

• Apache Hadoop
• Apache Spark
• Apache Beam

Workflow Engine

• Apache Airflow
• Digdag

Public Cloud

GCP

• Google Cloud Dataflow

AWS

• Amazon EMR