BaguaSys / bagua

Bagua Speeds up PyTorch

https://tutorials-8ro.pages.dev/

support gradient accumulation

wangraying opened this issue 3 years ago · comments

Rui Wang commented 3 years ago

refer to: https://pytorch.org/docs/stable/_modules/torch/nn/parallel/distributed.html#DistributedDataParallel.no_sync

Jianbin Chang commented 3 years ago

it is now supported by #312