This project is a project to do a simple ETL. This project include these tech:
- Python programming language
- Docker
- PostgreSQL
- data dir: contains all the data that is already downloaded from kaggel Global Energy Statistics - Kaggle
- etl dir: contains all python scripts that is used to do ETL
This is the step of ETL that I used:
- Extract
Extract stage is used to extract data from data sources. Because in this database we only have one format of data, therefore I only use pandas library to extract data from csv into data frame and return that data frame.
- Transform
In this transform stage, I used python to transform data to what I need, what I did is:
- I remove the unnecessary columns. In this case I only used the column [Continent, Country, and the last column of the file].
- Remove missing values from the dataset using .dropna() function from pandas
- Because the data type of the last column is float and has many number after comma, then I transform to round up the value therefore the value only has 2 number after comma
- After transform the values of the last column I create new data frame for transformed column and replace the old column with the new transformed column
- Load
Load stage is used to load the data that has been transformed to the data warehouse or database. In this case I used PostgreSQL with docker image that I will explain more later in docker section. I used psycopg2 to help me connect python with PostgreSQL. Not just that, I also used argparse to help me create command arguments in python therefore, I can input csv file, database name, host name, username, password, and port of PostgreSQL flexibly.
- Clone this repository
- Create volume name etl_energy_consumption with this command
docker volume create etl_energy_consumption
- PostgreSQL using this path to save their data in docker /var/lib/postgresql/data. Therefore if you want to use volume then the path of your volume will be like this
[your volume name]:/var/lib/postgresql/data
. So the volume will be like thisetl_energy_consumption:/var/lib/postgresql
- Pull postgres image from docker using this command line
docker pull postgres
- create network for docker by using this command line
docker network create energy_consumption_network
- Pull pgadmin in docker using this command line
docker pull dpage/pgadmin4
- Run docker compose using this command line
docker-compose up -d
NOTE:
If you want to connect docker with network you cannot used localhost, but you have to see the IP of the connection using:
docker network inspect [network name]
Then you will see the IP of the connection
or you can just use name of the container to connect each container
This is the image of your pgadmin configuration
- install pgcli to see your PostgreSQL from command line
pip install pgcli
- Run this to access PostgreSQL from command line
pgcli -h localhost -p 5432 -u postgres -d energy_consumption
- Build image for our etl script by running this command
docker build -t python-etl .
- Go to etl dir and run
docker run -it --network=energy_consumption_network python-etl -f [file-path] -db energy_consumption -hs localhost -u postgres -pass 123456 -p 5432
- Then ETL process will completed 👍👨🏻💻