In this project, I look at various gradient descent optimization methods such as Adam, Adagrad, RMSProp and examine their impact on training loss and training time. I also implement cyclical learning rate policy.
Resources and Citation: 1.An Overview of gradient descent optimization algorithms https://ruder.io/optimizing-gradient-descent/
-
Leslie N. Smith Cyclical Learning Rates for Training Neural Networks. Available at https://arxiv.org/abs/1506.01186.
-
Cyclical learning rate policy in Keras: https://www.pyimagesearch.com/2019/08/05/keras- learning-rate-finder/.