Why i got Nan when i train the ncsnpp?

Question

Why i got Nan when i train the ncsnpp?

manmushanhe opened this issue a year ago · comments

haoyu lu commented a year ago

I change the nfft to 510, and backbone to ncsnpp, but i got nan in the training. Could you help me ?

zoupengpeng128 · Answer 1 · Mon Nov 06 2023 17:12:24 GMT+0800 (China Standard Time)

I change the nfft to 510, and backbone to ncsnpp, but i got nan in the training. Could you help me ?

I have the same problem. Have you solved it?

haoyu lu · Answer 2 · Wed Nov 08 2023 16:08:59 GMT+0800 (China Standard Time)

I change the nfft to 510, and backbone to ncsnpp, but i got nan in the training. Could you help me ?

I have the same problem. Have you solved it?

No, I didn't train successfully

Julius Richter · Answer 3 · Sun Feb 04 2024 22:48:29 GMT+0800 (China Standard Time)

We have never encountered a problem like this before. Maybe it has something to do with the internal precision of matrix multiplications in PyTorch.

In any case, we have just updated the code to PyTorch 2.X and Lightning 2.X to ensure support for newer GPUs. Try running the code again after you have created a virtual environment with the new requirements.