exafox / qu

qu is an in-progress data platform created by the CFPB to serve their public data sets. This is a public domain work of the US Government. This is the stable version; day-to-day development is at https://github.com/cndreisbach/qu.

qu

qu is an in-progress data platform created by the CFPB to serve their public data sets.

The goals of this platform are to:

• Import data in our Google-Dataset-inspired format
• Query data using our Socrata-Open-Data-API-inspired API
• Export data in JSON or CSV format

Getting Started

Prerequisites

In order to work on qu, you need the following languages and tools installed:

• Java
• Node.js
• Leiningen
• Grunt
• Bower
• MongoDB

Setup

Once you have the prerequisites installed and the code downloaded and expanded into a directory (which we will call "qu"), run the following commands:

cd qu
lein deps
npm install -g grunt-cli bower
npm install && bower install
grunt

If editing the JavaScript or CSS, run the following to watch the JS and CSS and make sure your changes are compiled:

grunt watch

You can run grunt to compile the files once.

To start a Clojure REPL to work with the software, run:

lein repl

In order to run the API as a web server, run:

lein ring server

Go to http://localhost:3000 and you should see the app running.

Before starting the API, you will want to start MongoDB and load some data into it. Currently, qu only supports connecting to a local MongoDB connection.

Loading data

Make sure you have MongoDB started. To load some sample data, run lein repl and enter the following:

(require 'cfpb.qu.loader)
(in-ns 'cfpb.qu.loader)
(mongo/connect!)
(load-dataset "county_taxes")
(load-dataset "census") ; Takes quite a while to run; can skip.
(mongo/disconnect!)

Testing

We use Midje to test this project, so to execute the tests, run:

lein midje

If you want the tests to automatically run whenever you change the code, eliminating the JVM startup time and generally being great, run:

lein midge :autotest