NVMLError(ret) pynvml.nvml.NVMLError_InvalidArgument: Invalid Argument

Question

NVMLError(ret) pynvml.nvml.NVMLError_InvalidArgument: Invalid Argument

GabrielZZZ opened this issue 2 years ago · comments

Yifan Zhang Gabriel commented 2 years ago

Hi,

I am using Ubuntu 20.04 with Nvidia RTX3090. When I followed the instructions to train the model, it always gives me this error:
File "/opt/conda/lib/python3.8/site-packages/pynvml/nvml.py", line 366, in check_return raise NVMLError(ret) pynvml.nvml.NVMLError_InvalidArgument: Invalid Argument
Does anyone know any possible solutions? That would be very helpful.

Gienapp · Answer 1 · Mon Nov 07 2022 19:43:59 GMT+0800 (China Standard Time)

Hi, I'm having the same problem. Did you come up with a solution?

Yifan Zhang Gabriel · Answer 2 · Tue Nov 08 2022 07:36:18 GMT+0800 (China Standard Time)

I am afraid not. It appears the forum is not very active.

Gienapp · Answer 3 · Tue Nov 08 2022 17:02:18 GMT+0800 (China Standard Time)

For me the solution was to change the number nproc_per_node in the training command from 8 to the number of GPUs my server has.