The purpose of this project is to gain some experience working with the tools you will use throughout the course. You will learn:
- The basics of data loading and preparation
- Classification using the k-NN algorithm
- Classification using a Support Vector Machine (SVM)
- Calculation of metrics to assess the performance of your model
To complete the project you will need to download the Cifar10 dataset and devise an appropriate training set split. You may acquire the dataset using the method of your choice. Note that some sources e.g. sklearn have reduced the samples to only 1797 (this could negatively affect your results). If you use the full dataset, you are free to sample from the dataset to reduce the overal training samples. The overall accuracy is less important than your observations and comparisons. For example classifying the full 10,000 test images might take approximately 15 minutes using k-NN. In this section:
- Download the dataset as described above
- Divide the raw data into appropriate training and test sets for both the images and corresponding labels for use with the subsequent parts of the project.
- Visualize five samples of each class by plotting a grid using the matplotlib library.
For this section you will implement a simple kNN Classifier on the Cifar10 dataset. To do this you will need to perform the following steps:
- Create a k-NN class. Your class must contain a method that returns predictions for your test set.
- Provide a suitable distance metric that you will use to calculate the nearest neightbours. You may choose the distance metric you believe is most suitable.
- Calculate the k nearest neighbours and make predictions.
- Choose the a value for k that results in the highest accuracy on your test set. Show how you found this value.
When your classifier is working:
- Use sklearn to calculate accuracy and plot a confusion matrix using your predictions.
- Provide a brief discussion of your results
When you compute the distance metric you can acheive better performance if you vectorize the computation instead of using for loops to iterate through the values.
For this section you will perform binary classification using logistic regression. Just as in Part 1. you will use the Cifar10 dataset, however to obtain a result for each class using logistic regression you will need to use a One-vs-Rest (OvR) approach to acheive multi-class classification.
Using LogisticRegression() in sklearn, write a function to execute the OvR strategy for the Cifar10 classes. Do not use the built-in OneVsResClassifier() method. You will need to follow these basic steps:
- Train a binary classifier for each class, where the target class is a "positive" results and the combination of the remaining classes are "negative". For Cifar10 you will need 10 models.
- For each test sample compute the probabilities for each model
- Select the argmax of the probabilities to obtain the predicted class
Collect your predictions from the test set and compute the accuracy score and plot a confusion matrix.
In Part 3. you will use Scikit-learn to perform classification, again on the Cifar10 dataset. You can use the built in SVM library for classification. As with logistic regression, SVM is designed for binary classification. However, in this case Scikit-learn will handle the OvR models behind the scenes.
Your task is to compare different modes of the SVM and determine the best performer.
Create an SVM baseline using the LinearSVC() function. Make sure to use the primal solution and use "ovr" for multiclass Calculate the accuracy score for comparison.
Next you will explore the effect of the cost parameter on the accuracy.
- Run the classification with a range of C values For example: [0.0001, 0.001, 0.01, 0.1, 1, 10, 100, 1000 ]
- Plot the results as an accuracy vs. C-parameter curve on a logarithmic scale.
Add a regularization term.
- Rerun the above experiment, but this time use L1 regularization.
- Again, plot the results as an accuracy vs. C-parameter curve on a logarithmic scale.
For the final experiment you will use the SVC() function to run the classifer with a kernel.
- Use a radial basis function when training a new model
- Find the optimal combination of values for the cost and gamma parameters. Use the following values in your loop:
for cost in [0.01, 0.1, 1, 10, 100]:
for gamma in [0.01, 0.1, 1, 10, 100]: - Again, plot the results as an accuracy vs. C-parameter curve on a logarithmic scale.
Choose the model with the highest accuracy and plot the confusion matrix. In your discussion explain the results of your experiments and the reason for increased performance from the baseline (if any). Comment on the effect of the cost-paramenter and the L1 penalty on accuracy as well as any overfitting you observed. Discuss the confusion matrix of the model accuracy and provide some reasons for high-values found off the main diagonal.
Instead of using raw pixel values compute an alternate feature representation for your dataset and re-run train the model. Compare the accuracy of the model using the new feature representation with the model trained with pixel values.