Amazing Prime loves the dataset and wants to keep it updated on a daily basis. Britta needs assistance to create an automated pipeline that takes in new data, performs the appropriate transformations, and loads the data into existing tables. this project will refactor the code from this module to create one function that takes in the three files—Wikipedia data, Kaggle metadata, and the MovieLens rating data—and performs the ETL process by adding the data to a PostgreSQL database.
In this Deliverable we are going to do ETL process, code refactoring and write a function that reads the 3 data files and creates the data frames. The code for this deliverable part is :
The output of the 3 data frames are :
In this deliverable we are going to do ETL process and code refactoring. We are going to extract and transform the wikipedia data so merge can be done with Kaggle metadata. Also, extracting the IMDB IDs using a regular expression string and dropping duplicates we are going to use the try-expect block to catch the errors.
the dataframes that are created in this deliverable are:
In this deliverable we are going to use the pandas, ETL process and code refactoring. we will extract and transform the Kaggle metadata and MovieLens rating data then convert the transformed data into separate dataframes. It will merge the Kaggle metadata DataFrame with the Wikipedia movies DataFrame to create the movies_df DataFrame. Finally, merging the MovieLens rating data DataFrame with the movies_df DataFrame to create the movies_with_ratings_df.
The dataframes associated with this task are:
In this deliverable we are going to use the ETL process, Pandas, code refactoring and PostgreSQL to add the movies_df dataframe and MovieLens rating CSV data to the SQL database.
Note: The movies_metadata.csv in the Resources folder is in compressed mode please download if you like to view it.