aparra / snowplow-docker

Docker images for Snowplow, Iglu and associated projects

Snowplow Docker

Introduction

This repository contains the Docker images for the following Snowplow components:

• Scala Stream Collector
• Stream Enrich
• Elasticsearch Loader
• S3 Loader
• Iglu Server

They are published in the snowplow-docker-registry.bintray.io docker registry.

Pulling

You can pull the images from the registry directly:

# NSQ Scala Stream Collector image, there are others available for Kinesis, Kafka and Google PubSub
docker pull snowplow-docker-registry.bintray.io/snowplow/scala-stream-collector-nsq:0.13.0

# NSQ Stream Enrich image, there are others available for Kinesis, Kafka and Google PubSub
docker pull snowplow-docker-registry.bintray.io/snowplow/stream-enrich-nsq:0.18.0

# Elasticsearch Loader image
docker pull snowplow-docker-registry.bintray.io/snowplow/elasticsearch-loader:0.10.1

# S3 Loader image
docker pull snowplow-docker-registry.bintray.io/snowplow/s3-loader:0.6.0

# Iglu Server image
docker pull snowplow-docker-registry.bintray.io/snowplow/iglu-server:0.3.0

Building

Alternatively, you can build them yourself:

# All images are based on the base images
docker pull snowplow-docker-registry.bintray.io/snowplow/base-alpine:0.2.0
docker pull snowplow-docker-registry.bintray.io/snowplow/base-debian:0.1.0

# NSQ Scala Stream Collector image, there are others available for Kinesis, Kafka and Google PubSub
docker build -t snowplow/scala-stream-collector-nsq:0.13.0 scala-stream-collector/0.13.0/nsq

# NSQ Stream Enrich image, there are others available for Kinesis, Kafka and Google PubSub
docker build -t snowplow/stream-enrich-nsq:0.18.0 stream-enrich/0.18.0/nsq

# Elasticsearch Loader image
docker build -t snowplow/elasticsearch-loader:0.10.1 elasticsearch-loader/0.10.1

# S3 Loader image
docker build -t snowplow/s3-loader:0.6.0 s3-loader/0.6.0

# Iglu Server image
docker build -t snowplow/iglu-server:0.3.0 iglu-server/0.3.0

Running

Create your own config file filling it according to your setup, you can find examples at the following locations:

• Scala Stream Collector configuration
• Stream Enrich configuration
• Elasticsearch Loader configuration
• S3 Loader configuration
• Iglu Server configuration

Next, you can run a container for each component by mounting your configuration directory:

# NSQ Scala Stream Collector container, there are others available for Kinesis, Kafka and Google PubSub
docker run \
  -v $PWD/scala-stream-collector-config:/snowplow/config \
  snowplow/scala-stream-collector-nsq:0.13.0 \ # if you have built the image
  # snowplow-docker-registry.bintray.io/snowplow/scala-stream-collector-nsq:0.13.0 if you have pulled the image
  --config /snowplow/config/config.hocon

# NSQ Stream Enrich container, there are others available for Kinesis, Kafka and Google PubSub
docker run \
  -v $PWD/stream-enrich-config:/snowplow/config \
  snowplow/stream-enrich-nsq:0.18.0 \ # if you have built the image
  # snowplow-docker-registry.bintray.io/snowplow/stream-enrich-nsq:0.18.0 if you have pulled the image
  --config /snowplow/config/config.hocon \
  --resolver file:/snowplow/config/resolver.json \
  --enrichments file:/snowplow/config/enrichments/ \
  --force-cached-files-download

# Elasticsearch Loader
docker run \
  -v $PWD/elasticsearch-loader-config:/snowplow/config \
  snowplow/elasticsearch-loader:0.10.1 \ # if you have built the image
  # snowplow-docker-registry.bintray.io/snowplow/elasticsearch-loader:0.10.1 if you have pulled the image
  --config /snowplow/config/config.hocon

# S3 Loader
docker run \
  -v $PWD/s3-loader-config:/snowplow/config \
  snowplow/s3-loader:0.6.0 \ # if you have built the image
  # snowplow-docker-registry.bintray.io/snowplow/s3-loader:0.6.0 if you have pulled the image
  --config /snowplow/config/config.hocon

# Iglu Server
docker run \
  -v ${PWD}/iglu-server-config:/snowplow/config \
  snowplow/iglu-server:0.3.0 \ # if you have built the image
  # snowplow-docker-registry.bintray.io/snowplow/iglu-server:0.3.0 if you have pulled the image
  --config /snowplow/config/application.conf

You can find more information in the readme for each image:

• Scala Stream Collector readme
• Stream Enrich readme
• Elasticsearch Loader readme
• S3 Loader readme
• Iglu Server readme

There is a Docker Compose example in the example folder. Iglu Server also has a Docker Compose example in a separate example folder.

Find out more

Technical Docs	Setup Guide	Roadmap & Contributing
Technical Docs	Setup Guide	coming soon

Copyright and license

Copyright 2017-2018 Snowplow Analytics Ltd.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this software except in compliance with the License.

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.