Data Platform Index

Samples and documents of data platform.

Data Format

• Protocol Buffers
• Apache Avro

Data Collector

• Fluentd (Streaming)

• Embulk (Batch)

Data Pipeline

• Apache Hadoop
• Apache Spark
• Apache Beam

Workflow Engine

• Apache Airflow
• Digdag

Public Cloud

GCP

• Google Cloud Dataflow

AWS

• Amazon EMR