Discussion: Beta scheduling
Micky774 opened this issue · comments
The overfit test got me thinking: what if we scheduled beta? If initially beta started off at 0, or at least low like .1, then throughout training leveled out at 1. I'm thinking that this may encourage the network to first learn meaningful patterns and features in the encoder, and then ramp up the task to also having a cohesive latent space. Thoughts?
I think that before we implement this idea, it would be helpful to run experiment 10 to explore what initial beta we would want and then in that case, we could compare the results of having a scheduled beta to the best result from exp10.