kavisek / demo-merge-pipeline

a demo pipeline to merge data from a pandas dataframe to a database table

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

demo-merge-pipeline

This is a demo project to show how to merge data in postgres. The project is a simple python script that merges data from two tables. The script is scheduled to run every minute.

make start

Overview

  • local postgres database
  • flyway manages the migrations
  • a simple crontab scheduler
  • configParser stores the configuration details

Local Database Credentials

host: localhost
port: 5432
username: postgres
password: docker

References

About

a demo pipeline to merge data from a pandas dataframe to a database table

License:MIT License


Languages

Language:Python 72.2%Language:Shell 11.3%Language:Makefile 10.9%Language:Dockerfile 5.6%