kanterov / scio

A Scala API for Apache Beam and Google Cloud Dataflow

Scio

Ecclesiastical Latin IPA: /ˈʃi.o/, [ˈʃiː.o], [ˈʃi.i̯o]

Verb: I can, know, understand, have knowledge.

Scio is a Scala API for Apache Beam and Google Cloud Dataflow inspired by Apache Spark and Scalding. See the current API documentation for more information.

Scio 0.3.0 and future versions depend on Apache Beam (org.apache.beam) while earlier versions depend on Google Cloud Dataflow SDK (com.google.cloud.dataflow). See this page for a list of breaking changes.

Features

• Scala API close to that of Spark and Scalding core APIs
• Unified batch and streaming programming model^{1, 2}
• Fully managed service²
• Integration with Google Cloud products: Cloud Storage, BigQuery, Pub/Sub, Datastore, Bigtable²
• HDFS source/sink
• Interactive mode with Scio REPL
• Type safe BigQuery
• Integration with Algebird and Breeze
• Pipeline orchestration with Scala Futures
• Distributed cache

¹ provided by Apache Beam

² provided by Google Cloud Dataflow

Quick Start

The ubiquitous word count example can be run directly with SBT in local mode, using README.md as input.

sbt "scio-examples/run-main com.spotify.scio.examples.WordCount --input=README.md --output=wc"
cat wc/part-00000-of-00001.txt

Documentation

• Scio Wiki - wiki page
• ScalaDocs - current API documentation
• Big Data Rosetta Code - comparison of code snippets in Scio, Scalding and Spark

Artifacts

Scio includes the following artifacts:

• scio-core: core library
• scio-test: test utilities, add to your project as a "test" dependency
• scio-bigquery: Add-on for BigQuery, included in scio-core but can also be used standalone
• scio-bigtable: Add-on for Bigtable
• scio-extra: Extra utilities for working with collections, Breeze, etc.
• scio-hdfs: Add-on for HDFS

License

Copyright 2016 Spotify AB.

Licensed under the Apache License, Version 2.0: http://www.apache.org/licenses/LICENSE-2.0