peteatkinson / snowplow

Cloud-native web, mobile and event analytics, running on AWS and GCP

Snowplow

Snowplow is an enterprise-strength marketing and product analytics platform. It does three things:

1. Identifies your users, and tracks the way they engage with your website or application
2. Stores your users' behavioural data in a scalable "event data warehouse" you control: Amazon Redshift, Google BigQuery, Snowflake or Elasticsearch
3. Lets you leverage the biggest range of tools to analyze that data, including big data tools (e.g. Spark) via EMR or more traditional tools e.g. Looker, Mode, Superset, Re:dash to analyze that behavioural data

To find out more, please check out Snowplow website and the docs website.

If you wish to get everything setup and managed for you, you can take a look at our commercial offer, Snowplow Insights.

Snowplow technology 101

The repository structure follows the conceptual architecture of Snowplow, which consists of six loosely-coupled sub-systems connected by five standardized data protocols/formats:

To briefly explain these six sub-systems:

• Trackers fire Snowplow events. Currently we have 12 trackers, covering web, mobile, desktop, server and IoT
• Collector receives Snowplow events from trackers. Currently we have one official collector implementation with different sinks: Apache Kafka, Amazon Kinesis, NSQ
• Enrich cleans up the raw Snowplow events, enriches them and puts them into storage. Currently we have several implementations, built for different environments (GCP, AWS, Apache Kafka) and one core library
• Storage is where the Snowplow events live. Currently we store the Snowplow events in a flatfile structure on S3, and in the Redshift, Postgres, Snowflake and BigQuery databases
• Data modeling is where event-level data is joined with other data sets and aggregated into smaller data sets, and business logic is applied. This produces a clean set of tables which make it easier to perform analysis on the data. We have data models for Redshift and Looker
• Analytics are performed on the Snowplow events or on the aggregate tables.

For more information on the current Snowplow architecture, please see the Technical architecture.

About this repository

This repository used to be an umbrella repository for all loosely-coupled Snowplow components. However, since June 2020 all components have been extracted into their dedicated repositories (more info here) and this repository serves as an entry point for OSS users and historical artifact.

Components that have been extracted to their own repo are still here as git submodules.

Please use directly the repo of a component to report issues and create PRs:

Trackers

• Android
• Command line
• .NET
• Go
• Java
• Javascript
• Node.js
• Objective-C
• PHP
• Python
• React Native
• Ruby
• Scala
• Unity

Collector

Enrich

Loaders

• BigQuery (streaming)
• Redshift (batch)
• Snowflake (batch)
• Google Cloud Storage (streaming)
• Amazon S3 (streaming)
• Postgres (streaming)
• Elasticsearch (streaming)

Testing

• Mini
• Micro

Parsing enriched event

• Analytics SDK Scala
• Analytics SDK Python
• Analytics SDK .NET
• Analytics SDK Javascript

Bad rows

Need help?

We want to make it super-easy for Snowplow users and contributors to talk to us and connect with each other, to share ideas, solve problems and help make Snowplow awesome. Here are the main channels we're running currently, we'd love to hear from you on one of them:

Discourse

This is for all Snowplow users: engineers setting up Snowplow, data modelers structuring the data and data consumers building insights. You can find guides, recipes, questions and answers from Snowplow users including the Snowplow team.

We welcome all questions and contributions!

Twitter

@SnowplowData for official news or @SnowplowLabs for engineering-heavy conversations.

GitHub

If you spot a bug, then please raise an issue in the GitHub repo of this component. Likewise if you have developed a cool new feature or an improvement, please open a pull request, we'll be glad to integrate it in the codebase!

If you want to brainstorm a potential new feature, then Discourse is probably a better place to start.

Email

[e-mail]

If you want to talk directly to us (e.g. about a commercially sensitive issue), email is the easiest way.

Copyright and license

Snowplow is copyright 2012-2020 Snowplow Analytics Ltd.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this software except in compliance with the License.

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.