kangkot / onyx

Distributed, masterless, high performance, fault tolerant data processing

Onyx

What is it?

• a masterless, cloud scale, fault tolerant, high performance distributed computation system
• batch and stream hybrid processing model
• exposes an information model for the description and construction of distributed workflows
• Competes against Storm, Cascading, Cascalog, Spark, Map/Reduce, Sqoop, etc
• written in pure Clojure

What would I use this for?

• Realtime event stream processing
• CQRS
• Continuous computation
• Extract, transform, load
• Data transformation à la map-reduce
• Data ingestion and storage medium transfer
• Data cleaning

Installation

Available on Clojars:

[org.onyxplatform/onyx "0.10.0-beta17"]

Plugins and Libraries

Unsupported plugins

Some plugins are currently unsupported in onyx 0.10.x. These are:

• onyx-sql
• onyx-durable-queue
• onyx-elasticsearch
• onyx-http
• onyx-kafka-0.8

Plugin Use

To use the supported plugins, please use version coordinates such as [org.onyxplatform/onyx-amazon-sqs "0.10.0.0-alpha2"], and read the READMEs on the 0.10.x branches linked above.

Build Status

Component	release	unstable	compatibility
onyx core			-
onyx-kafka
onyx-kafka-0.8
onyx-datomic
onyx-redis
onyx-sql
onyx-bookkeeper
onyx-durable-queue
onyx-elasticsearch
onyx-amazon-sqs
onyx-amazon-s3
onyx-http
learn-onyx		-
onyx-examples
onyx-dashboard
onyx-metrics
onyx-peer-http-query

• release: stable, released content
• unstable: unreleased content
• compatibility: edge, unstable, unreleased content depending on core master

Companies Running Onyx in Production

                  

                  

                  

                  

Quick Start Guide

Feeling impatient? Hit the ground running ASAP with the onyx-starter repo and walkthrough. You can also boot into preloaded a Leiningen application template.

User Guide 0.10.0-beta17

• User Guide Table of Contents
• API docs
• Cheat Sheet

Developer's Guide 0.10.0-beta17

• Branch Policy
• Release Checklist
• Deployment Process

API Docs 0.10.0-beta17

Code level API documentation can be found here.

Official plugin listing

Official plugins are vetted by Michael Drogalis. Ensure in your project that plugin versions directly correspond to the same Onyx version (e.g. onyx-kafka version 0.10.0-beta17.0-SNAPSHOT goes with onyx version 0.10.0-beta17). Fixes to plugins can be applied using a 4th versioning identifier (e.g. 0.10.0-beta17.1-SNAPSHOT).

• onyx-core-async
• onyx-kafka
• onyx-kafka-0.8
• onyx-datomic
• onyx-redis
• onyx-sql
• onyx-bookkeeper
• onyx-seq
• onyx-durable-queue
• onyx-elasticsearch
• onyx-http
• onyx-amazon-sqs
• onyx-amazon-s3

Generate plugin templates through Leiningen with onyx-plugin.

3rd Party plugin listing

Unofficial plugins have not been vetted.

• onyx-rethink

Offical Dashboard and Metrics

You can run a dashboard to monitor Onyx cluster activity, found here. Further, you can collect metrics and send them to the dashboard, or anywhere, by using the onyx-metrics plugin.

Need help?

Check out the Onyx Google Group.

Want the logo?

Feel free to use it anywhere. You can find a few different versions here.

Running the tests

A simple lein test will run the full suite for Onyx core.

Contributor list

• Michael Drogalis
• Lucas Bradstreet
• Owen Jones
• Bruce Durling
• Malcolm Sparks
• Bryce Blanton
• David Rupp
• sbennett33
• Tyler van Hensbergen
• David Leatherman
• Daniel Compton
• Jeff Rose
• Ole Krüger
• Juho Teperi
• Nicolas Ha
• Andrew Meredith
• Bridget Hillyer
• Ivan Mushketyk
• Jochen Rau
• Tienson Qin
• Roman Volosovskyi
• Vijay Kiran
• Paul Kehrer
• Scott Bennett
• Nathan Todd.stone
• Mariusz Jachimowicz
• Jason Bell

Acknowledgements

Some code has been incorporated from the following projects:

• [Riemann] (https://github.com/aphyr/riemann)
• [zookeeper-clj] (https://github.com/liebke/zookeeper-clj)

License

Copyright © 2015 Michael Drogalis

Distributed under the Eclipse Public License, the same as Clojure.

Profiler

YourKit supports open source projects with its full-featured Java Profiler. YourKit, LLC is the creator of YourKit Java Profiler and YourKit .NET Profiler, innovative and intelligent tools for profiling Java and .NET applications.