Categorical Reparameterization with Gumbel-Softmax
The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
Bayesian Deep Learning with torch distributions
Jang's approach : prior distribution as categorical distribution
python main.py --sampling=TDModel --kld=eric
Maddison's approach : prior distribution as Concrete Distribution
python main.py --sampling=TDModel --kld=madisson
Maddison's treatment : prior distribution as ExpConcrete Distribution
python main.py --sampling=ExpTDModel --kld=madisson
Maddison's way is sensitive and hard to optimize, but Jang's trick is easy to optimize.
Furthermore, Maddison's treatment for numerical issue has no effect in discrete-VAE.
args.min_temp mainly determines the degree of discreteness.