kloeckner-i / genstage_importer

Example of data import pipeline using Flow library

genstage_importer

Demo project showing a concept of CSV import with GenStage/Flow data pipeline in Elixir. This is an illustration for a blogpost in Kloeckner's dev blog. CSV files to import are products.csv and orders.csv.

Setup

Requirements:

• docker
• docker-compose

$ docker-compose up -d --build

Create a database:

$ docker-compose exec app mix ecto.setup

To run the sync:

$ docker-compose exec app iex -S mix
iex(1) > GenstageImporter.Pipeline.Product.import()

Generate test data

Requirements:

• ruby

Edit script files/gen_csv.rb:

PRODUCTS_NUMBER = 1_000 # set number of products in CSV file here

Then run it:

$ ./files/gen_csv.rb

It will replace existing CSV files in ./files folder with new autogenerated data.