cjsaylor / datamnom

Generic data ingestion for Elasticsearch to be visualized by Kibana.

Datamnom

tl;dr: Generic data ingestion for Elasticsearch to be visualized by Kibana.

Originally created for a local Hack for Change event, this project aims to get quick visualizations via Kibana that look like this:

Dashboard above produced in Kibana from an import of Florida Vendor data from 2014 that was open sourced by FL state government.

Why this method, why not make an API

Elasticsearch serves as both a generic search tool, but it also functions as an API through Elasticsearch's rest interface while simultaneously supplying an aggregation and visualization framework through Kibana (among other tools).

Requirements

• NodeJS 10+

Install

• docker-compose up -d
• npm install

Usage

1. docker-compose up -d (if you don't already have elasticsearch running locally)
2. Create a configuration file (see examples).
3. Run node bin/datamnom.js --config=/path/to/config/from/step/1 --files=/glob/path/to/files
4. Open http://localhost:5601 to see your data imported