help with training

[Yuxin Chen]

Thanks for the awesome code! I am training my own model right now and have a few questions:

• currently I am using 100k (out of around 1.8m) images from CelebAMask-HQ, ffhq and vggface to train the model. did you use the full set to train your model?
• I didn't see large improvement for most losses anymore (160k steps trained, 4gpus x 12images/batch); is this normal? should I just continue training for more steps?
  
  
  
  
  
• I also checked the validation results, and the reconstruction is not good.
  
  
  
• I noticed shuffle for the training dataloader is not set to True, did you use the same setting?

Thanks!

[Changho Choi 최창호]

Hi! You did very fast training!

1. Yes, I used full-set dataset. I don't know about IJB-C dataset. The distribution of dataset can influence to your model.

2. In the paper, they trained for 500K steps. I trained for over 500K. In my eye, your losses are getting down for attribute_loss but unstable for Rec and ID loss. In my case, the two losses are more stable and lower at the same steps.

3. shuffle option in training dataloader should be True. It is clearly my mistake while publishing.

[Yuxin Chen]

Hi! You did very fast training!

1. Yes, I used full-set dataset. I don't know about IJB-C dataset. The distribution of dataset can influence to your model.
2. In the paper, they trained for 500K steps. I trained for over 500K. In my eye, your losses are getting down for attribute_loss but unstable for Rec and ID loss. In my case, the two losses are more stable and lower at the same steps.
3. shuffle option in training dataloader should be True. It is clearly my mistake while publishing.

Thanks for your reply.

1. I just corrected the description, I am using the same datasets (CelebAMask-HQ, ffhq and vggface) as well.

• So in your case, each step has 64 images; and let's say there are 1.5m images in those three datasets, so you trained for around 4 epochs (= 64 * 500000 / 1500000 / 5 ) in total?
• In my case, each step only has 48 images, so maybe that's why the two losses are higher at the same steps.
• I found the Rec loss is going much lower in the third epoch, and the results are much better than before. I will continue my current training and see what's going on.

3. Thanks for the clarification, I also changed to True during my training.

[Changho Choi 최창호]

• I trained with 32 batch size, it is the same as the paper. (Two V100 32G GPUs, 16 batch size for each)

• Training GAN is very unstable. If your loss is going down, I think it works well.

[QiulinW]

Hi! You did very fast training!

1. Yes, I used full-set dataset. I don't know about IJB-C dataset. The distribution of dataset can influence to your model.
2. In the paper, they trained for 500K steps. I trained for over 500K. In my eye, your losses are getting down for attribute_loss but unstable for Rec and ID loss. In my case, the two losses are more stable and lower at the same steps.
3. shuffle option in training dataloader should be True. It is clearly my mistake while publishing.

Thanks for your reply.

1. I just corrected the description, I am using the same datasets (CelebAMask-HQ, ffhq and vggface) as well.

• So in your case, each step has 64 images; and let's say there are 1.5m images in those three datasets, so you trained for around 4 epochs (= 64 * 500000 / 1500000 / 5 ) in total?
• In my case, each step only has 48 images, so maybe that's why the two losses are higher at the same steps.
• I found the Rec loss is going much lower in the third epoch, and the results are much better than before. I will continue my current training and see what's going on.

1. Thanks for the clarification, I also changed to True during my training.

Hi, did you change the coefficients of different loss terms? I found my training unstable with the coeffs provided by the author...

[Hanieh Khosravi]

• I trained with 32 batch size, it is the same as the paper. (Two V100 32G GPUs, 16 batch size for each)
• Training GAN is very unstable. If your loss is going down, I think it works well.

it means that you have used 'dp' instead of 'ddp'?
since in 'ddp' mode the whole batch is not devided between GPUs.

[payne4handsome]

@y-x-c Hi, have you got the satisfying result? I trained just with FFHQ and CelebA-HQ datasets about 90 thousand images. The result is bad just like below.

[princessmittens]

By 4 epoch's you mean 26...

I am about two weeks into training at about the halfway mark. I noticed that some of the results on the colab show some image artifacts. Is that present in all final results? Did you manage to fix that with more training?

[Hanieh Khosravi]

By 4 epoch's you mean 26...

I am about two weeks into training at about the halfway mark. I noticed that some of the results on the colab show some image artifacts. Is that present in all final results? Did you manage to fix that with more training?

@princessmittens I am also working on this paper and I have some questions about this implementation. May I have your email address?

[princessmittens]

@usingcolor I am near the end of training and these are my results. I have trained on 8 gpu's with 32 gig ram and a 21 batch size/ per gpu. The results have been pretty bad so far. I tried my best to recreate the exact parameters with all 3 datasets (~1.3 million images after processing) and have trained for about 2-3 weeks. With my current batch size and according to the results, I'm at the 79% mark in reference to 500k.

@y-x-c -Have you been able to recreate better results? Is it worth continuing?

This has cost a lot of money/time Any input would be great.

@hanikh Not sure how much I can help you considering my results but my email is <>

Src:

Target:


Results:

[Tam Nguyen]

Hello, anyone here got good result?

[princessmittens]

No-I have talked to @hanikh. I don't think anyone has been able to recreate the results as of yet.

[Antonio Gonzalez]

I have better results than this, princessmittens, can you leave your e-mail and I will reach out to you?

No-I have talked to @hanikh. I don't think anyone has been able to recreate the results as of yet.

[Tam Nguyen]

@cwalt2014 Can you please share the source code or pretrained weights? I will appreciate that. My email: [e-mail]
Thanks.

[Delia Jiamin]

@cwalt2014 Dear friend！Can you please share the source code? Thanks a million🙏🙏. My email: [e-mail]
Thanks！🙏🙏🙏

[princessmittens]

@cwalt2014 my email is [e-mail]

[Lucifer_Toledo]

@cwalt2014 my email is [e-mail]

[Poloangelo]

@cwalt2014 I would also love to know what changes you would suggest to get better results 🙏 my mail is [e-mail]

[Seanseattle]

@cwalt2014 Could you please share the source code or pretrained weights? Thank you. My Email: [e-mail]

[chinasilva]

@cwalt2014 Thank you very much. My Email:[e-mail]

[lefsiva]

@cwalt2014 Thank you. My email is: [e-mail]

[liuyijiao]

@cwalt2014 Thank you very much. My email is: [e-mail]

[tryink]

I have better results than this, princessmittens, can you leave your e-mail and I will reach out to you?

No-I have talked to @hanikh. I don't think anyone has been able to recreate the results as of yet.

Hi, @cwalt2014, could you please send me some of your results? I wonder what the possible results looks like. My email is [e-mail]. Any reply will be appreciated.

[suzie26]

@cwalt2014 Can you share your code or pretrained weights?? Thank you soooo much!!
My email is: [e-mail]

[Daichi Zhang]

@cwalt2014 Could you share your code or pretrained weights?? Thank you very much!! My email is: [e-mail]

[Changho Choi 최창호]

@Daisy-Zhang @suzie26 @tyrink @akafen @lefsiva @chinasilva @Seanseattle @Poloangelo @ZhiluDing @princessmittens @DeliaJIAMIN @tamnguyenvan @cwalt2014 Check out HifiFace, our implementation of a more recent face-swapping model with the pre-trained model.

[chuer-yu]

@cwalt2014 hello, could you please share your pretrained weights? Thank you so much! My email is: [e-mail]

[ywon0925]

@cwalt2014 could you please share your pre-trained weights? I would really appreciate it! My email is: [e-mail]

[niuyuanc]

@antonsanchez Could you please share the pretrained weights? Thank you so much 🙏🙏🙏
My email: [e-mail]
Thanks🙏🙏🙏

[galmizush]

@cwalt2014 could you please share your pre-trained weights? I would really appreciate it! My email is: [e-mail]

[JaeHyun Park]

@cwalt2014 could you please share your pre-trained weights? My email is: [e-mail]
Thank you so much