Problem with training part.
NathanJHLee opened this issue · comments
Hi, I am Nathan and i am facing some problem with training part.
My env
Centos7.5
#PIP
pytorch-wpe 0.0.1
rotary-embedding-torch 0.5.3
torch 1.12.1+cu113 //To use cuda, I did reinstall torch and torchaudio.
torch-complex 0.4.3
torchaudio 0.12.1+cu113
torchvision 0.13.1+cu113
#rpm
libcudnn8-devel-8.2.0.53-1.cuda11.3.x86_64
libcudnn8-8.2.0.53-1.cuda11.3.x86_64
libnccl-devel-2.9.9-1+cuda11.3.x86_64
libnccl-2.9.9-1+cuda11.3.x86_64
To run a script , I follow 'egs/voxceleb/sv-ecapa/run.sh'
I set 4 gpus. (When i set single gpu, It's not working too)
But I got error as below.
Stage3: Training the speaker model...
WARNING:torch.distributed.run:
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
2024-02-15 14:31:58,001 - INFO: Use GPU: 3 for training.
2024-02-15 14:31:58,003 - INFO: Use GPU: 2 for training.
2024-02-15 14:31:58,009 - INFO: Use GPU: 1 for training.
2024-02-15 14:31:58,011 - INFO: Use GPU: 0 for training.
Traceback (most recent call last):
File "speakerlab/bin/train.py", line 176, in
main()
File "speakerlab/bin/train.py", line 60, in main
model = torch.nn.parallel.DistributedDataParallel(model)
File "/home/asr/miniconda3/envs/3D-Speaker/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 646, in init
Traceback (most recent call last):
File "speakerlab/bin/train.py", line 176, in
_verify_param_shape_across_processes(self.process_group, parameters)
File "/home/asr/miniconda3/envs/3D-Speaker/lib/python3.8/site-packages/torch/distributed/utils.py", line 89, in _verify_param_shape_across_processes
return dist._verify_params_across_processes(process_group, tensors, logger)
RuntimeError: NCCL error in: ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1191, unhandled system error, NCCL version 2.10.3
ncclSystemError: System call (e.g. socket, malloc) or external library call failed or device error. It can be also caused by unexpected exit of a remote peer, you can check NCCL warnings for failure reason and see if there is connection closure by a peer.
main()
File "speakerlab/bin/train.py", line 60, in main
model = torch.nn.parallel.DistributedDataParallel(model)
File "/home/asr/miniconda3/envs/3D-Speaker/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 646, in init
_verify_param_shape_across_processes(self.process_group, parameters)
File "/home/asr/miniconda3/envs/3D-Speaker/lib/python3.8/site-packages/torch/distributed/utils.py", line 89, in _verify_param_shape_across_processes
return dist._verify_params_across_processes(process_group, tensors, logger)
RuntimeError: NCCL error in: ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1191, unhandled system error, NCCL version 2.10.3
ncclSystemError: System call (e.g. socket, malloc) or external library call failed or device error. It can be also caused by unexpected exit of a remote peer, you can check NCCL warnings for failure reason and see if there is connection closure by a peer.
Traceback (most recent call last):
File "speakerlab/bin/train.py", line 176, in
main()
File "speakerlab/bin/train.py", line 60, in main
model = torch.nn.parallel.DistributedDataParallel(model)
File "/home/asr/miniconda3/envs/3D-Speaker/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 646, in init
_verify_param_shape_across_processes(self.process_group, parameters)
File "/home/asr/miniconda3/envs/3D-Speaker/lib/python3.8/site-packages/torch/distributed/utils.py", line 89, in _verify_param_shape_across_processes
return dist._verify_params_across_processes(process_group, tensors, logger)
RuntimeError: NCCL error in: ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1191, unhandled system error, NCCL version 2.10.3
ncclSystemError: System call (e.g. socket, malloc) or external library call failed or device error. It can be also caused by unexpected exit of a remote peer, you can check NCCL warnings for failure reason and see if there is connection closure by a peer.
Traceback (most recent call last):
File "speakerlab/bin/train.py", line 176, in
main()
File "speakerlab/bin/train.py", line 60, in main
model = torch.nn.parallel.DistributedDataParallel(model)
File "/home/asr/miniconda3/envs/3D-Speaker/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 646, in init
_verify_param_shape_across_processes(self.process_group, parameters)
File "/home/asr/miniconda3/envs/3D-Speaker/lib/python3.8/site-packages/torch/distributed/utils.py", line 89, in _verify_param_shape_across_processes
return dist._verify_params_across_processes(process_group, tensors, logger)
RuntimeError: NCCL error in: ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1191, unhandled system error, NCCL version 2.10.3
ncclSystemError: System call (e.g. socket, malloc) or external library call failed or device error. It can be also caused by unexpected exit of a remote peer, you can check NCCL warnings for failure reason and see if there is connection closure by a peer.
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 121550 closing signal SIGTERM
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 121547) of binary: /home/asr/miniconda3/envs/3D-Speaker/bin/python
Traceback (most recent call last):
File "/home/asr/miniconda3/envs/3D-Speaker/bin/torchrun", line 8, in
sys.exit(main())
File "/home/asr/miniconda3/envs/3D-Speaker/lib/python3.8/site-packages/torch/distributed/elastic/multiprocessing/errors/init.py", line 345, in wrapper
return f(*args, **kwargs)
File "/home/asr/miniconda3/envs/3D-Speaker/lib/python3.8/site-packages/torch/distributed/run.py", line 761, in main
run(args)
File "/home/asr/miniconda3/envs/3D-Speaker/lib/python3.8/site-packages/torch/distributed/run.py", line 752, in run
elastic_launch(
File "/home/asr/miniconda3/envs/3D-Speaker/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 131, in call
return launch_agent(self._config, self._entrypoint, list(args))
File "/home/asr/miniconda3/envs/3D-Speaker/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 245, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
speakerlab/bin/train.py FAILED
Failures:
[1]:
time : 2024-02-15_14:32:03
host : e7bcf3a85e2c
rank : 1 (local_rank: 1)
exitcode : 1 (pid: 121548)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
[2]:
time : 2024-02-15_14:32:03
host : e7bcf3a85e2c
rank : 2 (local_rank: 2)
exitcode : 1 (pid: 121549)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
Root Cause (first observed failure):
[0]:
time : 2024-02-15_14:32:03
host : e7bcf3a85e2c
rank : 0 (local_rank: 0)
exitcode : 1 (pid: 121547)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
I successfully re-cloned the repository and executed the run.sh script without encountering any errors. The versions of PyTorch and CUDA installed on my system are 1.12.0 and 10.2, respectively.
You must attempt the following steps:
- Verify the execution permissions for the Python script to ensure it is executable.
- The speakerlab within the run.sh directory is a symbolic link. Consider copying the file directory that 3D-Speaker/speakerlab points to and replace the symbolic link with the actual directory.
Hi, I have one more question.
I believe i solve problem with nccl.
But I got another error.
I encounter error 'model = torch.nn.parallel.DistributedDataParallel(model)' in train.py
It's mismatch tensor size.
I think '192' is embedding_size according to ecapa_tdnn.yaml.
please check my error log.
Thank you.
Error log is here when i try to use single gpu.
Stage3: Training the speaker model...
2024-02-22 18:15:58,831 - INFO: Use GPU: 1 for training.
d5acf849f4d8:167887:167887 [1] NCCL INFO Bootstrap : Using eth0:172.17.0.2<0>
d5acf849f4d8:167887:167887 [1] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation