OpenCompany Storage Service

Background

A lack of transparency results in distrust and a deep sense of insecurity.

-- Dalai Lama

Teams struggle to keep everyone on the same page. People are hyper-connected in the moment with chat and email, but it gets noisy as teams grow, and people miss key information. Everyone needs clear and consistent leadership, and the solution is surprisingly simple and effective - great leadership updates that build transparency and alignment.

With that in mind we designed Carrot, a software-as-a-service application powered by the open source OpenCompany platform and a source-available web UI.

With Carrot, important company updates, announcements, stories, and strategic plans create focused conversations that keep everyone aligned without interruptions. When information is shared transparently, it inspires trust, new ideas and new levels of stakeholder engagement. Carrot makes it easy for leaders to engage with employees, investors, and customers, creating alignment for everyone.

Transparency expectations are changing. Organizations need to change as well if they are going to attract and retain savvy teams, investors and customers. Just as open source changed the way we build software, transparency changes how we build successful companies with information that is open, interactive, and always accessible. Carrot turns transparency into a competitive advantage.

To get started, head to: Carrot

Overview

The OpenCompany Storage Service handles data access and data management of open company content and data. It supports other OpenCompany services, such as the Web application and the Digest Service, as well as open API access.

Local Setup

Prospective users of Carrot should get started by going to Carrot.io. The following local setup is for developers wanting to work on the OpenCompany Storage Service.

Most of the dependencies are internal, meaning Leiningen will handle getting them for you. There are a few exceptions:

Java - a Java 8+ JRE is needed to run Clojure
Leiningen 2.9.1+ - Clojure's build and dependency management tool
RethinkDB v2.3.6+ - a multi-modal (document, key/value, relational) open source NoSQL database

Java

Your system may already have Java 8+ installed. You can verify this with:

java -version

If you do not have Java 8+ download it and follow the installation instructions.

An option we recommend is OpenJDK. There are instructions for Linux and Homebrew can be used to install OpenJDK on a Mac with:

brew update && brew cask install adoptopenjdk8

Leiningen

Leiningen is easy to install:

Download the latest lein script from the stable branch.
Place it somewhere that's on your $PATH (env | grep PATH). /usr/local/bin is a good choice if it is on your PATH.
Set it to be executable. chmod 755 /usr/local/bin/lein
Run it: lein This will finish the installation.

Then let Leiningen install the rest of the dependencies:

git clone https://github.com/open-company/open-company-storage.git
cd open-company-storage
lein deps

RethinkDB

RethinkDB is easy to install with official and community supported packages for most operating systems.

RethinkDB for Mac OS X via Brew

Assuming you are running Mac OS X and are a Homebrew user, use brew to install RethinkDB:

brew update && brew install rethinkdb

If you already have RethinkDB installed via brew, check the version:

rethinkdb -v

If it's older, then upgrade it with:

brew update && brew upgrade rethinkdb && brew services restart rethinkdb

Follow the instructions provided by brew to run RethinkDB every time at login:

ln -sfv /usr/local/opt/rethinkdb/*.plist ~/Library/LaunchAgents

And to run RethinkDB now:

launchctl load ~/Library/LaunchAgents/homebrew.mxcl.rethinkdb.plist

Verify you can access the RethinkDB admin console:

open http://localhost:8080/

After installing with brew:

Your RethinkDB binary will be at /usr/local/bin/rethinkdb
Your RethinkDB data directory will be at /usr/local/var/rethinkdb
Your RethinkDB log will be at /usr/local/var/log/rethinkdb/rethinkdb.log
Your RethinkDB launchd file will be at ~/Library/LaunchAgents/homebrew.mxcl.rethinkdb.plist

RethinkDB for Mac OS X (Binary Package)

If you don't use brew, there is a binary installer package available for Mac OS X from the Mac download page.

After downloading the disk image, mounting it (double click) and running the rethinkdb.pkg installer, you need to manually create the data directory:

sudo mkdir -p /Library/RethinkDB
sudo chown <your-own-user-id> /Library/RethinkDB
mkdir /Library/RethinkDB/data

And you will need to manually create the launchd config file to run RethinkDB every time at login. From within this repo run:

cp ./opt/com.rethinkdb.server.plist ~/Library/LaunchAgents/com.rethinkdb.server.plist

And to run RethinkDB now:

launchctl load ~/Library/LaunchAgents/com.rethinkdb.server.plist

Verify you can access the RethinkDB admin console:

open http://localhost:8080/

After installing with the binary package:

Your RethinkDB binary will be at /usr/local/bin/rethinkdb
Your RethinkDB data directory will be at /Library/RethinkDB/data
Your RethinkDB log will be at /var/log/rethinkdb.log
Your RethinkDB launchd file will be at ~/Library/LaunchAgents/com.rethinkdb.server.plist

RethinkDB for Linux

If you run Linux on your development environment (good for you, hardcore!) you can get a package for you distribution or compile from source. Details are on the installation page.

RethinkDB for Windows

RethinkDB isn't supported on Windows directly. If you are stuck on Windows, you can run Linux in a virtualized environment to host RethinkDB.

Configuration

An AWS SQS queue is used to pass messages to the OpenCompany Storage service from the OpenCompany Authentication service. Setup SQS Queues and key/secret access to the queue using the AWS Web Console or API.

You will also need to subscribe the aws-sqs-auth-queue SQS queue to the Authentication service SNS topic:

Go to the AWS SQS Console and select the aws-sqs-auth-queue SQS queue configured above. From the 'Queue Actions' dropdown, select 'Subscribe Queue to SNS Topic'. Select the SNS topic you've configured your Authentication service instance to publish to, and click the 'Subscribe' button.

Make sure you update the section in project.clj that looks like this to contain your specific configuration:

:dev [:qa {
  :env ^:replace {
    :db-name "open_company_storage_dev"
    :liberator-trace "false" ; liberator debug data in HTTP response headers
    :hot-reload "true" ; reload code when changed on the file system
    :open-company-auth-passphrase "this_is_a_dev_secret" ; JWT secret
    :aws-access-key-id "CHANGE-ME"
    :aws-secret-access-key "CHANGE-ME"
    :aws-sqs-bot-queue "CHANGE-ME" ; SQS queue to pass on requests to the Slack Bot
    :aws-sqs-email-queue "CHANGE-ME" ; SQS queue to pass on requests to the Email service
    :aws-sqs-auth-queue "CHANGE-ME" ; SQS queue to read notifications from the Auth service
    :aws-sqs-storage-queue "CHANGE-ME" ; SQS queue to read requests from the Bot service
    :aws-sns-storage-topic-arn "" ; SNS topic to publish notifications (optional)
    :log-level "debug"
  }

You can also override these settings with environmental variables in the form of OPEN_COMPANY_AUTH_PASSPHRASE and AWS_ACCESS_KEY_ID, etc. Use environmental variables to provide production secrets when running in production.

Shared Secrets

A secret, open-company-auth-passphrase, is shared between the OpenCompany services for creating and validating JSON Web Tokens.

HTTP Behavior

There are two options controlling HTTP behavior:

liberator-trace will include detailed debug info in HTTP response headers that correspond to each step in the Liberator decision graph for that request.

The hot-reload configuration controls if each request checks for updated code to load, useful for development, but not for production.

Both of these settings take the string true or false.

AWS

AWS SQS queues are used to pass messages from the Storage Service to other OpenCompany services. Setup an SQS Queue and key/secret access to the queue using the AWS Web Console or API and update the corresponding aws- configuration properties with the key, secret and queue names.

An optional AWS SNS pub/sub topic is used to push notifications of content changes to interested listeners. If you want to take advantage of this capability, configure the aws-sns-storage-topic-arn with the ARN (Amazon Resource Name) of the SNS topic you setup in AWS.

Technical Design

The Storage service is composed of these main responsibilities:

CRUD of orgs, boards, and entries
Access control to orgs, boards, and entries
Notifying the Bot service of new orgs via SQS
Notifying the Bot service and Email service of share requests via SQS
Notifying the Bot service and Email service of new invites via SQS
Notifying the Email service of password reset and email validation requests via SQS
Publishing org, board and entry change notifications to interested subscribers via SNS

The Storage service provides a HATEOAS REST API:

The Storage service shares a RethinkDB database instance with the Interaction service.

Usage

Prospective users of Carrot should get started by going to Carrot.io. The following usage is for developers wanting to work on the OpenCompany Storage Service.

You can verify all is well with your RethinkDB instance and get familiar with RethinkDB ReQL query language by using the Data Explorer:

open http://localhost:8080/

REPL

Next, you can try some things with Clojure by running the REPL from within this project:

lein migrate-db
lein repl

Server

Start a production instance:

lein start!

Or start a development instance:

lein start

API Requests

You'll need a JWToken to use the REST API via cURL as an authenticated user. The token is passed in the Authorization header with each request. You can either extract your own token from the headers of requests from your web browser, or you can generate a token using the Authentication service.

You can see a list of common API requests and responses by loading the oc-storage.paw file from this repository into the Paw API Development tool.

Import sample data

To import company sample data from an edn file run:

lein run -m oc.storage.util.import -- ./opt/samples/18f.edn

use -d to erase the company while importing like this:

lein run -m oc.storage.util.import -- -d ./opt/samples/green-labs.edn

To add all the company sample data in a directory (each file with a .edn extension), run:

lein run -m oc.storage.util.import -- ./opt/samples/

use -d to erase companies while importing like this:

lein run -m oc.storage.util.import -- -d ./opt/samples/

To add sample data on a production environment, specify the production database name:

DB_NAME="open_company_storage" lein run -m oc.storage.util.import -- -d ./opt/samples/18F.edn

DB_NAME="open_company_storage" lein run -m oc.storage.util.import -- -d ./opt/samples/

Generate sample data

To generate sample data into an existing org, run:

lein run -m oc.storage.util.generate -- <org-slug> <config-file> <start-date> <end-data>

e.g.

lein run -m oc.storage.util.generate -- 18f ./opt/generate.edn 2017-01-01 2017-06-30

See the sample generation config file ./opt/generate.edn for how the sample data generation can be customized.

Force initial indexing or re-indexing

You can initiate an initial indexing of all the content stored in the storage service, or a re-indexing of the content from the commandline. Ensure you have AWS_SQS_SEARCH_INDEX_QUEUE set to the search service SQS queue you are using:

export AWS_SQS_SEARCH_INDEX_QUEUE=change-me

Then run:

lein full-index

Or you can do it from the REPL. Ensure you have the aws-sqs-search-index-queue configuration set to the search service SQS queue you are using:

config/aws-sqs-search-index-queue

then, from the REPL, run:

(send-data-to-search-index)

Testing

Tests are run in continuous integration of the master and mainline branches on Travis CI:

To run the tests locally:

lein test!

Participation

Please note that this project is released with a Contributor Code of Conduct. By participating in this project you agree to abide by its terms.

License

Distributed under the GNU Affero General Public License Version 3.

This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details.

open-company / open-company-storage