Please read the ML_classification_methods_performance_report.pdf for the algorithms, cross validation scores and more details.
The MNIST dataset is a benchmark in the field of statistics, machine learning and computer vision, consisting of 70,000 grayscale images of handwritten digits, each of size 28x28 pixels, with each image labeled from 0 to 9. For our exercises, we have 10,000 images’ data as the training dataset, and 5,000 images’ data as the testing dataset out of the original 70,000 images.
The task is to develop a model to accurately classify the handwritten digits. The motivation behind this project is to explore and evaluate the accuracy of different statistical models in addressing the MNIST classification problem. Building on the results of a naive probabilistic model, which achieved a misclassification rate of about 22%, this project aims to push the boundaries of accuracy by employing more sophisticated models and techniques. The specific goals I’ve set for the project are:
(a) Multinomial Logistic Regression (with Lasso, Ridge, and Elastic Net)
(b) Decision Trees and Random Forests
(c) Multi-class Support Vector Machine (SVM)
(d) Multi-class Linear Discriminant Analysis (LDA)
(e) Deep Neural Network (DNN and CNN)
| Method | Accuracy | Misclassification Rate |
|---|---|---|
| Multinomial Logistic Regression | 0.899 | 11% |
| Decision Trees | 0.8047 | 20% |
| Random Forests | 0.925 | 7.5% |
| Support Vector Machine | 0.937 | 6.3% |
| Linear Discriminant Analysis | 0.845 | 15.5% |
| Deep Neural Network | 0.9287 | 7.1% |
| Convolutional Neural Network | 0.9784 | 2.2% |
Not surprisingly, the Convolutional Neural Network (CNN) outperforms all other models with a misclassification rate of only 2.2%. This superior performance can be attributed to CNN’s ability to capture spatial hierarchies and local dependencies in the image data, making it highly effective for image classification tasks like MNIST. The Deep Neural Network (DNN) also shows strong performance with a misclassification rate of 7.1%. The Support Vector Machine (SVM) achieves an accuracy of 0.937 and a misclassification rate of 6.3%. SVMs are powerful classifiers that perform well on high-dimensional data, but they lack the spatial feature extraction capabilities of CNNs. Random Forests achieve a misclassification rate of 7.5%. As an ensemble method, Random Forests combine the predictions of multiple decision trees, leading to better generalization than individual Decision Trees, which have a misclassification rate of 20%. Multinomial Logistic Regression and Linear Discriminant Analysis (LDA) achieve misclassification rates of 11% and 15.5%, respectively. While these linear models are simpler and faster to train, they are less capable of capturing the complex patterns in image data compared to non-linear models like CNNs and SVMs.
Given the results:
• Use CNNs for the best performance in image classification tasks like MNIST due to
their ability to capture spatial hierarchies and local dependencies.
• Consider DNNs if computational resources are limited, as they still provide strong
performance without the same level of complexity as CNNs.
• Use SVMs and Random Forests if a balance between accuracy and interpretability
is needed, and when computational efficiency is a priority.
• Use Decision Trees, Logistic Regression or LDA only for simpler, less complex
classification tasks or when quick, interpretable models are required.
Overall, CNNs are recommended for the highest accuracy in MNIST classification, but
the choice of model should also consider the specific requirements and constraints of the task
at hand.