Error while training on custom dataset
peach6 opened this issue · comments
Hi,
Great work. I am trying to train the model on my custom dataset but I am stuck on this error.
File "train_PBAFN_stage1_fs.py", line 106, in
concat = torch.cat([preserve_mask.cuda(),densepose,pose.cuda()],1)
RuntimeError: CUDA error: device-side assert triggered
terminate called without an active exception
hi, how did you manage to get the npy files for densepose ?
@tengolagan If you can get the jpg, .npy is simply another form, you can use np.asarray(img)
ref: https://www.geeksforgeeks.org/how-to-convert-images-to-numpy-array/
@peach6 I think it's something to do with the image size not sure though. I'm able to train on the VITON_traindata dataset which has images of size 192,256. I'm trying to do the same for 768,1024 size images, I've changed the hardcoded values (wherever 256 * 192 was present) in train_PFAFN_stage1_fs.py, data/base_dataset.py, data/aligned_dataset.py but I'm still getting this error.
@SenHe Can we have some support for different image sizes? 192,256 is too low a resolution.
@ds-nat-j @peach6 Hi, are you able to resolve this error while training on a custom dataset? If yes, could you please tell me what changes have you made? I was trying to train for 1024x768 resolution. Please let me know! @SenHe could you please resolve this issue?
Nope, I gave up on this implementation (and also on the vton space for a while) but maybe try GP-VTON
They release a pre-trained model recently that accepts higher dimension, I also think they follow a PF-AFN type approach like this paper. Might be worth a shot.
@ds-nat-j @peach6 Hi, are you able to resolve this error while training on a custom dataset? If yes, could you please tell me what changes have you made? I was trying to train for 1024x768 resolution. Please let me know! @SenHe could you please resolve this issue?
Nope, I gave up on this implementation (and also on the vton space for a while) but maybe try GP-VTON They release a pre-trained model recently that accepts higher dimension, I also think they follow a PF-AFN type approach like this paper. Might be worth a shot.
GP-VTON hasn't yet released the full inference code.
Could you please tell me how can I train the Flow-Style model for training on images of size 384x512 or higher dim?
If you have modified code for this or a similar higher resolution please share it.
I'm really stuck at this point. And want to finish this project ASAP!
A little help from your end would be great!
@dnyandeepc16 Hi, have you manage to solve the error while training on larger size(like 384x512) dataset?
Thank you