For this lab, we will be using the same dataset we used in the previous labs. We recommend using the same notebook since you will be reusing the same variables you previous created and used in labs.
So far we have worked on EDA. This lab will focus on data cleaning and wrangling from everything we noticed before.
- We will start with removing outliers. So far, we have discussed different methods to remove outliers. Use the one you feel more comfortable with, define a function for that. Use the function to remove the outliers and apply it to the dataframe.
- Create a copy of the dataframe for the data wrangling.
- Normalize the continuous variables. You can use any one method you want.
- Encode the categorical variables
- The time variable can be useful. Try to transform its data into a useful one. Hint: Day week and month as integers might be useful.
- Since the model will only accept numerical data, check and make sure that every column is numerical, if some are not, change it using encoding.
Hint for Categorical Variables
- You should deal with the categorical variables as shown below (for ordinal encoding, dummy code has been provided as well):
# One hot to state
# Ordinal to coverage
# Ordinal to employmentstatus
# Ordinal to location code
# One hot to marital status
# One hot to policy type
# One hot to policy
# One hot to renew offercustomer_df
# One hot to sales channel
# One hot vehicle class
# Ordinal vehicle size
data["coverage"] = data["coverage"].map({"Basic" : 0, "Extended" : 1, "Premium" : 2})
# given that column "coverage" in the dataframe "data" has three categories:
# "basic", "extended", and "premium" and values are to be represented in the same order.