Can DDP mode cause the rehearsal exemplars different on GPUs?
zihuanqiu opened this issue · comments
I wonder if runs in DDP mode, will each GPU computes and stores their own exemplars?
Thus, the total number of saved exemplars may excess the buffer size.
Thanks!
For example, GPU:0 may stored exemplars 1 2 3 4, and GPU:1 may stored exemplars 1 2 3 5. If this happens, and it actually stores 5 samples for next task, even though they were not on the same GPU.
That is a good remark. I'm on it.
Hi @zihuanqiu @arthurdouillard
Previously, I have also checked on this point.
The exemplars are identical for icarl rehearsal because all GPUs use the same model and procedure to produce the exemplars.
For selection with randomness, you need to set the the random seed to ensure this.
@zihuanqiu you were right.
@GengDavid you could have been right, but DyTox was actually using data augmentation when extracting features, thus examplars were not the same at all among GPUs.
I've uploaded an erratum here.
Sorry for the inconvenience, and thanks you for your help.