ETL pipeline to process, enrich and load the data collected from user searches on SpaN portal.
In the directory etl_project/etl_dir/:
- ETL_fun.py contains the functionality for the ETL pipeline.
- etl_main.py contains the main function, that initializes the sparkSession.
- preprocess_data.py initializes a sparkSession to preprocess the log files.
In the project directory, two shell scripts exist, that activate the conda enviroment and manage the directories.
In the directory report/:
- the file slides.pdf is a small report on my approach.
- tools: pyspark, pandas
- python version: 3.8.5