Authors: Saikat Roy, Albert Gubaidullin
Repository of the CudaVision Lab at University of Bonn (SS19) implemented (mostly) on PyTorch, Python3 and Jupyter notebooks. The project begins from the basics of neural networks and continues to deeper models. The following projects are contained in the respective folders:
Involves using softmax regression with manual gradient calculation for classifying the MNIST dataset. Training and test set accuracies after a simple 5 iteration run was 0.8931 and 0.8866 respectively.
Involves training simple multilayer neural networks using vanilla SGD on PyTorch with k-fold monte-carlo cross validation for hyperparameter (learning rate and batch size) search. Classification was done on the CIFAR-10 dataset. A confusion matrix after a simple 50 iteration run on a 3072-128-128-10 architecture is given below, with a training and test set accuracy of 0.6647 and 0.5117 respectively.
The project involved applying SGD, Adam, RMSprop, Adagrad and Adadelta to the CIFAR-10 dataset. A 3072-128-128-10 architecture with 0.2 Dropout between hidden units was used for all algorithms.
Also tested were different non-linearities for the hidden units.
In each case, the results were kind of counterintuitive as SGD and sigmoid performed the best. However, it might be possible that the convergence rates simply might be different. Additionally, with the network being this shallow, the benefits of the non-linearities used typically in 'Deep' networks might simply not reflect on a network of this scale. And if run long enough, SGD generally converges to a better minima than other optimizers like Adam.
The project introduces one to convolutional neural networks and applied a CNN on the CIFAR-10 dataset.