- Handling Missing Values
- Scaling and Normalization
- Parsing Dates
- Character Encodings
- Inconsistent Data Entry
The procces of cleaning data:
- read cvs file
- show the head() of the data
- use isnull() pandas' function to see the number of null rows in the dataset
- count the total of null value using sum() functuin
- find the shape of this dataset (rows & colms)
- find the product of the data; meaning rows*columns,
which you can do uding numpy function product; data_cells = np.product(mydata.shape) - find the percentage of these missing data; percetage = ((total_of_missing_values)/(data_cells))*100
- use your data intution to know why these values were missing; not recorded or don't exixit.
- if the missing vlues are 80% and above we will delete the row or the column
- we can fill the null values with mean, median, mode or 0 or any other values
Steps to prase the dates:
- find the data types
- display the column
- check the length of the date
- manually modify the dates if they are entired in inncorrect way using .loc
- cast the data type from object to the specfic format such as format="%m/%d/%Y"
- Visualze the data to double check. Use the seaborn library as: sns.distplot(column, kde=False, bins=31)
- Access certian day or month as follows: day_of_month_earthquakes = earthquakes['date_parsed'].dt.month or day_of_month_earthquakes = earthquakes['date_parsed'].dt.day
we have to first check the type of the column as follows:
mydataframe['mycolumn'].dtype
dtype is a short cut for data type.
then show a portion of the data to check:
mydataframe[300:400]
we then see the length of these indeses:
date_lengths = earthquakes.Date.str.len()
date_lengths.value_counts()