Objective: Study the impact of loss function and activation function on a sample Convolutional neural network (CNN)
A 3-layer CNN with three activation functions was designed and used to classify the images in the well-known CIFAR-10 set, and the outcome was compared with the expected results for CIFAR-10.
Activation Function:
Out of the three common methods, namely Sigmoid, ReLU and ELU, the latter was used in this project because the first two methods have the following downsides: 1- Sigmiod has high fluctuation and vanishing gradainet, 2- ReLU is linear for all positive values, but zero for negative ones, 3- ELU has a small slopeSGD for negative values, which addresses the downsides of the previous funcitons.
Optimization Methods:
SGD, Adam, Adagrad and RMSProp were used for optimization, out of which SGD had the lowest accuracy.
Results:
As seen in the results, the Dropout layer prevents the network from overfitting.