Reproducibility is not guaranteed
lg2840 opened this issue · comments
Hi,
I've read your paper and the idea is impressive.
I successfully run your sample code, but each time I run the same input I got different results. Though the seed is set fixed, it seems that there is still some randomness in the sample procedure. This problem appears on both CPU and CUDA environments.
Could you explain which step may cause this? If possible, how can we fix this so that same results can be reproduced?
Thanks!
Kris
Hi,
A proper approach is running the sampling algorithm multiple times and report the mean and variance of the results.
In our latest study that will be presented in ICML 2022, we have reported the performance with error bars:
https://arxiv.org/abs/2205.07249
https://github.com/pengxingang/Pocket2Mol