Exp4: Weight Decay
Micky774 opened this issue · comments
Test out various weight decay values and their effects on rate of convergence and loss stability
We will test the L2 weight decay values: {1e-7, 1e-6, 1e-5, 1e-4, 1e-3} over 30 epochs.
Script written and ready to be run.
Updated experiment to set batch-size to 160 over 2 GPU's. Currently running.
Concluded that 1e-3 should be used as it has a higher variance, but it has lower minima. (Meekail agrees.)