RuntimeError: CUDA error: throwing an instance of 'c10::Error'

[Omran Alamayreh]

Hi all,

I'm getting this error while training vid2vid;

RuntimeError: CUDA error: the launch timed out and was terminated
terminate called after throwing an instance of 'c10::Error' 

The error pops up when the script starts the validation phase; here is the entire log;
log.txt

I'm training vid2vid on the Kitti dataset using a server with 8 X 12GB Quadro M6000 GPUs.
I'm also attaching the config file;
ampO1.txt

Any suggestions?

Thanks :)

[o98k]

I have met this error before, maybe resource is not enough. Try to reduce dataset.