Does the test accuracy need to be synchronized in distributed.py?

[Yifan Jiang]

If directly output the test accuracy, will the code automatically synchronize the accuracy between each GPUs?

[ZHANG Zhi]

Nope, if u really need it, u can use .share_memory() to share a Tensor's memory.
All in all, most distributed lib only help u to handle the synchronization of data, parameters, and gradient.

[Yifan Jiang]

Could you give a specific example?
Thanks!

[ZHANG Zhi]

hm, m afraid that's not right.
Here are two ways to communicate between torch.multiprocessing:

1. if u dont care the running results, u can use share_memory_ like this, which is more faster:

import time
import random

import torch
import torch.distributed as dist
import torch.multiprocessing as mp


def evaluate(rank):
    torch.cuda.manual_seed(rank)
    local_acc = torch.randn(1)[0].cuda(rank)

    print("local_acc:", local_acc)

    return local_acc

def main_worker(gpu, ngpus_per_node, args):
    dist.init_process_group(backend='nccl', init_method='tcp://127.0.0.1:23456', world_size=4, rank=gpu)

    local_acc = evaluate(gpu)

    global_acc, global_count = args['global_acc'], args['global_count']

    global_acc += local_acc.cpu()
    global_count += 1

    print("global_acc:", global_acc / global_count)

if __name__ == '__main__':
    global_acc = torch.tensor(.0)
    global_count = torch.tensor(.0)
    
    global_acc.share_memory_()
    global_count.share_memory_()

    args = {
        'global_acc': global_acc,
        'global_count': global_count
    }
    
    mp.spawn(main_worker, nprocs=4, args=(4, args))

But if you really need to synchronize the accuracy, I suggest this kind of implement or something else using all_reduce:

import time
import random

import torch
import torch.distributed as dist
import torch.multiprocessing as mp


def evaluate(rank):
    torch.cuda.manual_seed(rank)
    local_acc = torch.randn(1)[0].cuda(rank)

    print("local_acc:", local_acc)

    return local_acc

def main_worker(gpu, ngpus_per_node, args):
    dist.init_process_group(backend='nccl', init_method='tcp://127.0.0.1:23456', world_size=4, rank=gpu)

    local_acc = evaluate(gpu)

    dist.all_reduce(local_acc, op=dist.reduce_op.SUM)
    global_acc = local_acc / ngpus_per_node

    print("global:", global_acc)

if __name__ == '__main__':
    args = {}
    mp.spawn(main_worker, nprocs=4, args=(4, args))

[ZHANG Zhi]

m not sure if u understand, if not u can directly use this code:

acc1, acc5 = accuracy(output, target, topk=(1, 5))
...
dist.all_reduce(acc1, op=dist.reduce_op.SUM)
...
top1.update(acc1[0] / 4 , images.size(0))

Btw, I don't think we really need to calculate the average accuracy during training, which is the waste of time.

[Yifan Jiang]

Thanks!
Should the code synchronize the loss between each gpus before loss.backward()? Or the backward function will synchronize automatically?

[ZHANG Zhi]

No. That's DistributedDataParallel's job.
Wrap your model with DistributedDataParallel and just call backward() as usual. During the backwards pass, gradients from each node are averaged (same as your saying "synchronize the loss") and parameters are synchronized automatically.
Check it here: https://github.com/pytorch/pytorch/blob/46539eee0363e25ce5eb408c85cefd808cd6f878/torch/nn/parallel/distributed.py#L378-L382

[Yifan Jiang]

thanks