Validation results show nan all the time

Question

Validation results show nan all the time

FairyFali opened this issue a year ago · comments

I encounter a strange result during validating. the result is

Starting train epoch...
Epoch X: Val NLL nan -- Val Atom type KL nan -- Val Edge type KL: nan
Val loss: nan Best val loss: 100000000.0000

the NLL is always nan, why?

Clement Vignac · Answer 1 · Wed Jun 21 2023 23:27:39 GMT+0800 (China Standard Time)

Hello, this is not normal. What command are you running? Is it on a custom dataset?

Fali Wang · Answer 2 · Thu Jun 22 2023 00:56:43 GMT+0800 (China Standard Time)

Hello, this is not normal. What command are you running? Is it on a custom dataset?

I run the command mentioned in the readme file. It is on qm9 with discrete noise. Specifically, it is python3 main.py.

Clement Vignac · Answer 3 · Mon Jun 26 2023 20:49:56 GMT+0800 (China Standard Time)

I'm not sure where the exact issue came from (probably a different behavior of mask_distributions with recent python versions), but it's now fixed. You can use the latest commit.

Fali Wang · Answer 4 · Sat Jul 08 2023 15:01:48 GMT+0800 (China Standard Time)

I'm not sure where the exact issue came from (probably a different behavior of mask_distributions with recent python versions), but it's now fixed. You can use the latest commit.

I think I figure it out. because you are using .log or torch.log in functions like kl_prior, and compute_Lt but not consider the zero to log(), so it will appear nan.