Exp4: Weight Decay
Micky774 opened this issue · comments
Meekail Zain commented
Test out various weight decay values and their effects on rate of convergence and loss stability
Bella Humphrey commented
We will test the L2 weight decay values: {1e-7, 1e-6, 1e-5, 1e-4, 1e-3} over 30 epochs.
Bella Humphrey commented
Script written and ready to be run.
Meekail Zain commented
Updated experiment to set batch-size to 160 over 2 GPU's. Currently running.
Bella Humphrey commented
Concluded that 1e-3 should be used as it has a higher variance, but it has lower minima. (Meekail agrees.)