This is the read me file for how the project should be run.

All files & packages needed are already in the github repo shared

first deploy the s3 cloudformation stack:(you can deply via console or cli)

sfl-s3-cloudformation.yaml.
This stack would create an s3 bucket in the specified region. The create a folder called raw-input where your csv file would be put.

The next step is deploy vpc cloudformation stack.

sfl-vpc-cloudformation.yaml
This stack would deploy the vpc and the postgres RDS

Create a secret manager to manage all database secrets

name it secret

Now use AWS serverless application model(sam) to deploy the lambda cloudformation stack

template.yaml
All modules have been installed in the python folder called packages
Activate a virtual environment
Cd to the etl-scripts folder and start your servelss deployment
use the following command (note you should specify a different s3 bucket to store sam) 'sam build' 'sam deploy --stack-name lambda-etl2 --s3-bucket forex-bucket-samsideen --s3-prefix sam-content --capabilities CAPABILITY_IAM CAPABILITY_AUTO_EXPAND CAPABILITY_NAMED_IAM'

Once rds has been created, connect to the DB and create a table with the schema of our csv. Like below

CREATE TABLE cleandata (

id int NOT NULL, first_name int NOT NULL, last_name varchar NOT NULL, email varchar DEFAULT NULL, gender varchar NOT NULL, ip_address varchar NOT NULL, gender_category varchar NOT NULL, full_name varchar NOT NULL );

Create a Trigger Rule

Once lambda has been created successfully create a trigger rule from the console. Navigate to the function created (ETL_trigger) on the console, collaspe function overview section then click add trigger and follow the instruction

choose s3
choose the s3 bucket name created
select 'put' as your event type
let the prefixes be 'raw-input'
let suffixes be '.csv'
then click add

To test test the lambda, drop DATA.csv into the s3bucket under 'raw-input' folder then go to the database table created to see the file content ingested

tblake84 / ETL-dataingestion-samsideen