Irigo-ETL
Simple ETL exercise.
It loads open datas exposed by the french town of Angers.
Datasets used
Components
The project a simple architecture :
- Nginx as a server,
- a websocket server built with Socket.io,
- Redis as in-memory datasource for the websocket,
- TimscaleDB a time-series database built on top of PostgreSQL,
- PipelineDB an alternatif to TimescaleDB,
- Tile38 a Redis-based service for managing geofencing.
Requirements
- Python 3.6.x
- Docker & Docker-compose
Use it
cd irigo-etl/
docker-compose up -d
python -m venv irigo
source bin/activate
pip install -r requirements.txt
python src/main.py
python src/realtime.pyOpen the websocket page to see the updates
You can also view the logs from the websocket-server :
docker-compose logs -f websocket-serverGenerate Bonobo's graphs
bonobo inspect --graph src/main.py | dot -o main.png -T png
bonobo inspect --graph src/realtime.py | dot -o realtime.png -T png
