What's In My Food?

An example full stack web application that explores New York's public data for restaurant inspection results combined with business data retrieved from Yelp.

Getting Started with Yelp.

If you want to include Yelp data in the ingest you will first need to sign up for a Yelp developer account. After that, follow the instructions to obtain a token. You will need to have this token in your environment as the variable YELP_API_TOKEN no matter which of the three methods you choose from below. Note that this is an optional step. If you choose not to ingest the Yelp data, you can still explore the NY Restaurant Grade data.

Getting Started With Docker/Gradle

Install a Java 8 JDK
Install Docker and Docker Compose
Run CI=true ./gradlew build dockerBuildImage to build and tag the docker images
Run docker-compose up -d to start the application and it's dependencies
Visit http://localhost:9000 in your browser

Getting Started - Java Only

Install a Java 8 JDK
Run a Postgres server and create a database
Run CI=true ./gradlew build
Unzip the archive in the ./wimf-ingest/build/distributions directory and invoke the wimf-ingest script. Use wimf-ingest --help for guidance on providing the correct database connection parameters.
Unzip the archive in the ./wimf-services/build/distributions directory and invoke the wimf-services script. Use wimf-services --help for guidance on providing the correct database connection parameters.
Run ./gradlew wimf-web:run. This will start the UI development server and open a new tab in your default browser with the correct URL. It will take several seconds for the assets to compile before you see the UI.

Getting Started - Docker Only

Install Docker and Docker Compose
Run ./test_docker.sh && docker-compose up -d
Visit http://localhost:9000 in your browser

Now What

It should take less than a minute to ingest all 200,000+ inspection reports. However the Yelp data takes a lot longer. You don't need to wait for it though, you can begin exploring the data once the web server is running. The UI is a series of charts that show the breakdown of the NY Health Department Restaurant grades across different categories such as location of the Restaurant and the type of Cuisine served. If you chose to populate the Yelp data you can also view the relationship between grade and Yelp's price category. You can also manipulate the filters in the side bar and the date range slider to view different slices of the data. Finally, there is a list of violations that occurred at each inspection in the dataset. Enjoy!

Motivation

I've been working primarily in Go for some time now and the last time I was seriously involved with a JVM based project, I was working in Groovy/Grails. With this project I am attempting to relearn "the basics". As such I've made a conscious choice to avoid large frameworks that are aimed at rapid development in favor of a focused set of technologies with the goal of understanding how things are pieced together.

What's In My Food uses the following core technologies:

React for the UI

React is a declarative, component based Javascript view library. You can think of a React component as a function that takes data and turns it into another form of data (the DOM) - the same input always produces the same output. This makes components written in React easy to understand and test. In addition to React, What's In My Food features CSS written entirely in Javascript using Glamor. There are many CSS in JS solutions out there, but Glamor is one of the few that feels "right". It's real CSS but without the selector madness and you get to treat styles as values which is important in a component based UI.

I used Facebook's Create React App CLI to get up and running. It uses Webpack under the hood to intelligently bundle javascript and other assets. This was one of my few concessions to the need to "move fast". Create React App is focused entirely on working out of the box so it doesn't offer any hooks to the underlying Webpack configuration. I have lots of experience with tinkering with Webpack so I was fully prepared to "eject" from CRA and manage things myself (Webpack is notoriously hard to get started with) - but it turned out I never had to.

Other notables include Flow for static type checking, Jest for unit testing, Prettier for code formatting and ESLint for linting. Most of this works out of the box with Create React App.

Postgres and JDBI for data

Perhaps a bit of an anachronism in the NoSQL age and maybe even a bit counter intuitive - after all, if relational, why not ORM? However, What's In My Food is entirely about data exploration, and SQL is a good way to talk to data. JDBI on the surface is a straightforward way to use SQL in your application. In practice though it took a little longer than I expected to figure things out. The documentation is sparse and I had to rely on some trial and error/reading of source code to get things working. If I had to do it again with the benefit of hindsight, I might have been swayed by the relative ease of something like Elasticsearch.

Jersey for REST

I chose Jersey because it's well documented, mature and has a very straightforward API. It also makes very few assumptions about the rest of your stack, and can be deployed in a number of ways so it's good for the sort of mix and match approach that I took here. It was also easy to test my Jersey resources.

JavaRx

To be honest, this was a bit of a luxury choice. I've been wanting to dive into FRP style programming for awhile and I thought I had a somewhat decent use case for it here. And it did seem to work out well for the ingest side of things. I just barely scratched the surface with JavaRx and observables but I found it productive to think of processing event streams in the same way that I would a generic collection (map, reduce, etc). As a bonus it turns out that testing these streams is very straightforward.

Other bits and pieces

Hibernate Validator for checking user inputs and keeping nasty stuff out of the data layer.
Jackson for going to/from JSON
React Virtualized for efficiently rendering large lists
Recharts for charts
HK2 for a little bit of dependency injection

What did I Learn?

I learned that this is a relatively laborious way to build an application. But that was somewhat expected and intentional. It is also something that I'm sure gets easier with practice. The payoff was that I did find it edifying to deal directly with the details that in the past were hidden behind many layers of Grail/Spring/Hibernate. I feel like I have an incrementally better understanding of how to build multi tiered systems in Java.

That being said however there are toolkits out there that potentially represent better tradeoffs in terms of the rapid development of something like Grails and the flexibility and greater control you get with this approach. When programming in Go, I frequently use Gokit and for Java, Dropwizard looks promising.

Prior art

Many thanks to Jonathan Gilday for showing me how to microservice in Java and helping to review this repo.

rszewczyk / wimf