Multi-GPU version runs much slower than on single card

[T.C.Sheep]

I found this problem at last weekend. The GPU volatile utilizing is only 11% on each card (I am using 4 titan xp). However when I use single card, the usage would be 45% (not that optimal, maybe >= 80% would be better). I guess it has something to do with the EMA thing. I will try to debug it. Just an info.

Best,
Shu

[T.C.Sheep]

I have tested several parameters.

Single GPU:

batch size 32:
epoch[1/30] 1417.54 examples/sec, 0.59 min/epoch. Rest runtime is 0.28 hour

batch_size 64:
epoch[1/30] 2640.38 examples/sec, 0.32 min/epoch. Rest runtime is 0.15 hour

4 GPUs:
batch size 128 (32 on each):
epoch[1/30] 845.78 examples/sec, 0.99 min/epoch. Rest runtime is 0.48 hour

batch size 256 (64 on each):
epoch[1/30] 1402.12 examples/sec, 0.60 min/epoch. Rest runtime is 0.29 hour

As shows, by running the generator and discriminator on 4 GPU will even make it worse.
I have no idea whether we can put EMA as DataParallel on multiple GPU as well. A simple:
ema = torch.nn.DataParallel(ema)
will not work. Since PyTorch will try to devide the input for ema, which are the current parameters of network, as 4 different batches into different 4GPUs to update the moving parameters. However parameters are not databatches, I am not sure whether we can find the correct part of the parameter for ema to update distributedly.

Best,
Shu

[T.C.Sheep]

Without Ema, the things does not speed up significantly:

4 GPUs with EMA:
batch size 128 (32 on each):
epoch[1/30] 845.78 examples/sec, 0.99 min/epoch. Rest runtime is 0.48 hour

4 GPUs without EMA:
batch size 128 (32 on each):
epoch[1/30] 926.86 examples/sec, 0.90 min/epoch. Rest runtime is 0.43 hour

EMA might not be the problem.

[T.C.Sheep]

Oops, as the depth goes deeper, the utilization of GPUs increases, and I think it should be much faster than on single card.

Best,
Shu

[Animesh Karnewar]

@deartonym,

Well, that is indeed too much to process for me 😆 (it was night when you opened the issues and I was sleeping). Still, thanks for doing all the analysis. You cannot wrap ema inside torch.nn.Dataparalllel because, it is not a torch.nn.Module. It is a mechanism for applying small exponential moving average decay over the parameters, where we do not need to calculate gradients (That would really slow things down). About utilisation, well, you need to adjust batch-size for every depth. You can use the --start_depth argument to check utilisation directly at a higher depth. Hope this helps.

Best regards,
@akanimax

[T.C.Sheep]

@deartonym,

Well, that is indeed too much to process for me laughing (it was night when you opened the issues and I was sleeping). Still, thanks for doing all the analysis. You cannot wrap ema inside torch.nn.Dataparalllel because, it is not a torch.nn.Module. It is a mechanism for applying small exponential moving average decay over the parameters, where we do not need to calculate gradients (That would really slow things down). About utilisation, well, you need to adjust batch-size for every depth. You can use the --start_depth argument to check utilisation directly at a higher depth. Hope this helps.

Best regards,
@akanimax

@akanimax No worries! I just put what I think on this! You always response really fast, and many thanks for your concern!