Exp14: NVP_4 vs NVP
idhumphrey opened this issue · comments
Run NVP_4 for 90 epochs with a lsdim of 500.
Script has been written and ready to run.
Running
Experiment terminated after ~60 epochs. Loss failed to reduce past 140 consistently while NVP succeeds at a significantly reduced run time. Considerations for improvements on NVP_4 are welcome. It's worth noting that we may want to revisit this experiment once learning-rate/batch-size scheduling is up and running. I'm inclined to believe we may get different results upon either increasing the learning rate, or decreasing the batch size at the start and scheduling them. Another approach is to implement #39.
Running again with increased learning rate (1e-4 vs 1e-5) just for comparison's sake
Significant improvement has been observed in NVP_4 when utilizing learning rate of 1e-4. For future considerations we ought to test 1e-3, as well as learning rate scheduling strategies.