dhruvbansal / bohr

Makes observations of your systems

Bohr

Bohr is a scientist who continuously observes your system and submits his results to journals, forever, till he dies. Silly guy.

The bohr program is a single-executable file that can be run on any host to collect system (CPU, memory, disk, filesystem, network, &c.) and service (database, webserver, end-user application &c.) data and forward it to downstream data stores and/or other services.

Jump to:

• Installation
• Usage
• Configuration
• Available Observers
• Available Journals
• Customizing Bohr
• Contributing to Bohr

Tools similar to Bohr include:

• metrics
• telegraf
• collectd
• riemann-tools

Design

The fundamental concept in Bohr is an observation: a measurement of some aspect of the environment. Measurements always have a host and a time. They are typically (but not necessarily) quantitative in nature and can be decorated with various metadata.

Bohr does not store, downsample, plot/graph, index, alert on, or analyze any of its observations. It merely publishes them to downstream services (called journals) which you configure.

Running Bohr processes on different hosts do not need to communicate with each other in any way.

Design Goals

Reliability

The collection of metrics is the first step in a long-chain of systems and code supporting business-critical monitoring and alerting. If it fails, everything downstream of it fails, so Bohr tries to keep observing, no matter what.

Transparency

It’s always easy to ask Bohr what observations it makes based on the current configuration.

Modularity

Bohr lets you include and exclude observations and journals with fine-grained control.

Extensibility

Bohr can be extended via a Clojure DSL to make new, custom observations of your systems or submit to new downstream journals.

Efficiency

Bohr consumes minimal resources as it runs.

Portability

Bohr runs on any system which can run Java.

Adaptability

Bohr adapt the observations it makes appropriately for the environment it’s running within (e.g. - OS X vs. Linux).

Simplicity

Bohr is packaged as a single, self-contained, executable Java jar and auto-discovers common settings.

Why Clojure?

Bohr was written in Clojure for two major reasons:

1. the JVM
   • allows porting across different runtimes (Windows, OS X, Linux, &c.)
   • allows simple packaging as a self-executable JAR file
   • connects to JMX directly, making it possible to observe (the plethora of) Java-based services out there
2. LISPs
   • have eval allowing Bohr to extend itself dynamically at runtime without recompilation
   • have permissive syntax & macros for Bohr’s DSL

Concepts

Bohr has three central concepts:

1. Observations are data.
2. Journals are code which publish data to downstream services.
3. Observers are code which make observations and submit them to journals.

Using Bohr effectively requires:

1. Choosing journals which publish to your environment’s data collection and/or processing services (see available journals below).
2. For the relevant aspects of your host and its services:

   a) Choosing bundled observers if available (see available observers below). b) Writing your own observers in Bohr’s DSL (see customizing Bohr below).

3. Deploying Bohr with an appropriate configuration file defining & configuring these journals & observers (see deployment below).

More details on each of these concepts follows.

Observations

An observation is simply a measurement of some aspect of a host, a service it runs, or anything else you can think of.

Each observation has a

name

A string, typically dot-separated (e.g. - mem.util.free). (required)

timestamp

When the observation was made. (required) [Default: current time]

value

Any string, numeric, or boolean value (or nil). [Default: =nil=]

units

A string representing the units of value. [Default: =nil=]

description

A string decribing the observation. [Default: =nil=]

attributes

A map of key-value pairs to associate with the observation. [Default: ={}=]

Observers

An observer makes observations and submits them to journals.

Each observer has a

name

A string describing that observer’s subject (e.g. memory). (required)

period

An integer time in seconds between consecutive observations made by this observer. A nil value implies an observer which only runs once, at Bohr startup. [Default: =nil=]

Journals

A journal accepts submitted observations from observers and publishes them to external services.

Each journal has a

name

A string describing this journal’s publishing target (e.g. riemann). (required)

Installation

For simplicity, Bohr is distributed as a single file which you can download and run:

$ curl https://github.com/dhruvbansal/bohr/releases/download/v0.1.0/bohr > bohr
$ chmod +x bohr
$ ./bohr --help

Production Deployments

While running, Bohr does not daemonize itself or manage any of its output. You should configure services such as upstart and logrotate. Example configuration files for such services are provided in the examples directory.

An Ansible role for installing Bohr is also available, see the bohr-ansible-role repository.

Usage

Bohr’s user interface is the command line program bohr (see Installation above if you don’t have bohr installed). You can get help from Bohr via the following command:

$ bohr --help

Bohr has three modes of operation:

Summary

In summary mode, Bohr will make all his observations (see Configuration below) and print out his ready-to-publish values in a human-readable table. No observations will actually be submitted to journals to be published downstream.

Bohr will default to summary mode when invoked:

$ bohr ...

Summary mode works with configuration files and passing additional observers and journals on the command-line:

$ bohr my_observer.clj my_journal.clj --config /etc/bohr/bohr.yml --config /etc/bohr/conf.d

This makes summary mode useful to gain transparency into what Bohr is observing.

Once

The --once (or -o) option

$ bohr --once ...
$ bohr -o ...

makes Bohr skip printing a summary table and instead submit all observations to journals ONCE. This is useful when testing or debugging because Bohr will simply “ping” downstream services with a single wave of observations instead of continuously sending them (see the periodic mode below).

If Bohr doesn’t detect any journals at runtime, it will default to the console journal which prints observations to STDOUT.

As in summary mode, configuration files and additional observers and journals passed in on the command line can be used:

$ bohr my_observer.clj my_journal.clj --config /etc/bohr/bohr.yml --config /etc/bohr/conf.d --once

Periodic

The --periodic (or -p) option

$ bohr --periodic ...
$ bohr -p ...

makes Bohr run continuously, making and submitting observations to journals based on their pre-defined periods. This is the mode Bohr should run in production.

If Bohr doesn’t detect any journals at runtime, it will default to the console journal which prints observations to STDOUT.

As in summary mode, configuration files and additional observers and journals passed in on the command line can be used:

$ bohr my_observer.clj my_journal.clj --config /etc/bohr/bohr.yml --config /etc/bohr/conf.d --periodic

Configuration

Bohr accepts several configuration options via the command-line but more complex configuration for specific observers or journals is best provided via a YAML configuration file.

Tell Bohr to read a YAML configuration file (or a directory of such files) by passing the --config option to the bohr command:

$ bohr --config=/etc/bohr/bohr.yml --config=/etc/bohr/conf.d ...

Bohr’s configuration files can be used to:

• limit which bundled observers Bohr will run
• provide locations from which to load external observer & journal definitions
• declare which journals Bohr will submit observations to, and provide configuration options for them (hosts, ports, credentials, &c.)
• filter observers and observations
• provide observer-specific configuration options (processes, files, directories to watch, &c.)

A complete example configuration file lists available configuration options.

Bohr will merge configuration files in the order it reads them, with later files taking precedence. This merge is done intelligently, allowing for configurations such as the following example using the ps observer, which can be configured to watch the processes defined in the processes.tracked list:

# in app1.yml
---
processes.tracked:
  - name: app1
    user: app1_user
    cmd:  'java.*com.example.app1.*start'

# in app2.yml
---
processes.tracked:
  - name: app2
    user: root
    cmd:  'python.*app2'

Now invoking Bohr as follows:

$ bohr --config app1.yml --config app2.yml ...

would cause the ps observer to watch both applications as the processes.tracked lists are concatenated during the merge. This intelligent merging also works for hashes.

The end-result is that it is easy to shard Bohr’s configuration among many different (per-application, perhaps) files in a directory (say /etc/bohr.conf.d) and have Bohr intelligently observe all of them. This is especially useful from a DevOps perspective.

Observers

Bohr knows how to observe the following aspects of your systems:

TBD – just run `bohr` for now and look at its summary…

Journals

In addition to publishing observations to STDOUT via the console journal, Bohr knows how to publish to the following services:

• Riemann
• Many more planned…

Customizing

Bohr can be extended in two ways:

• by writing new observers
• by writing new journals to which observers submit observations

Clojure scripts implementing new observers and journals are automatically evaluated in a namespace containing several helper functions and macros – essentially a lightweight DSL.

These scripts can be passed to Bohr directly on the command line:

$ bohr my_observer.clj my_journal.clj ...

or included via a configuration file:

---
load:
  - /var/lib/bohr/*.clj

Writing Observers

Observers are data structures with the following fields:

name

The name of the observer (e.g. - mem)

period

The number of seconds an observer will wait between observations.

prefix

A prefix to add to the name of any observations made by the observer.

suffix

A suffix to add to the name of any observations made by the observer.

units

Default units for any observations made by the observer.

attributes

Default key-value pairs for any observations made by the observer.

Observers also contain a function which should implement making the observation and submitting it journals.

Observer Functions

The following functions are available from within observers

Observers are created using Bohr’s observe DSL macro.

Writing Journals

A journal is a Clojure function which takes the following arguments defining an observation:

name

The name of the observation (e.g. - mem.util.free)

value

The value of the observation (e.g. - 37)

options

A map with the following keys:

units

The units of the value (e.g. - “B”, “s”, ‘%”, &c.)

desc

A description of the observation (e.g. - “Percentage of memory free”)

attributes

Arbitrary key-value data about the observation

and publishes this to some downstream data store or service. A function is defined to be a journal via the define-journal! function. The example below defines a simple version of the built-in console journal:

;; in simple_console_journal.clj 
(define-journal!
  "simple_console"
  (fn [name value options]
    (println name value)))

See the built-in journals for more detailed examples.

Contributing

To contribute, follow the instructions below on how to develop on Bohr and then create a pull request!

Developer Installation & Commands

Bohr uses the Leiningen build tool. Once you have the lein command installed, checkout a copy of the Bohr source:

$ git clone https://github.com/dhruvbansal/bohr
$ cd bohr

Leiningen has many useful commands, but the following are especially so:

lein run

This is used for running the bohr command from the current source (not the compiled classes). Make sure to include the double-hyphen (--) to separate Leiningen options from options passed to bohr:

$ lein run -- --help
$ lein run -- --config bohr.yml --periodic ...

lein repl

Starts a Clojure REPL in the =’bohr.core= namespace.

  $ lein repl
...
bohr.core=> (println (observer-count))
0
nil
bohr.core=>

lein bin

Compiles and packages Bohr into a single executable JAR file:

$ lein bin
...
$  ./target/bohr --help

This task is used when preparing a new Bohr release.

Integration testing

It’s important to perform integration tests with the final executable JAR as there are several kinds of bugs (mostly related to resolving resource files) which crop up due to different behavior in development (lein run -- ...) and “production” (lein bin followed by target/bohr ...).

lein test

Runs Bohr’s test-suite.

$ lein test

See the testing section below for more details.

Repository Layout

The core code for Bohr lives in the src/bohr directory just like in any other Clojure project. This core defines the central concepts of Bohr (observers, journals, configuration, logging, &c.) but not any particular implementations of observers or journals.

Available observer and journal implementations are instead defined in the resources directory. These files will not be compiled when Bohr’s core is compiled but they will be contained within the self-executable JAR file Bohr is distributed as. They are loaded from this JAR file at runtime, if required.

Development on Bohr consists then of two distinct kinds of activity:

• working on Bohr core itself in src/bohr
• implementing particular observers and/or journals in resources

Testing

TBD.

Legal Information

Copyright © 2016 Dhruv Bansal

Distributed under the Apache Public License version 2.0.