the face toonify good ,but the body area works bad

[Owen]

I tested my images by vtoonify_t_arcane ckpt with below code :

python style_transfer.py --content ./data/038648.jpg \
       --scale_image --backbone toonify \
       --ckpt ./checkpoint/vtoonify_t_arcane/vtoonify.pt \
       --padding 600 600 600 600

I found it sometimes work bad in body area , the background style is padding to the body area :

[Shuai Yang]

Thank you for your interest in our work.

The problem is because StyleGAN is trained on face images. Our method uses StyleGAN as generator, and is also trained on face images.
So our method is mainly for portait toonification not for full body toonification.
Although can handle non-face regions, cannot handle them perfectly.
We have also dicussed it in the limitation Section in our paper.

[Owen]

thanks for you fast reply ! I see . Is there any methods to fix it ? training a half-length portrait by StyleGAN work?

[Shuai Yang]

That is one possible solution.

You can also try vtoonify_d_arcane model. It might preserve the background better.
And you can manually modify the mask m_E, which is used to pass the details of the input to the output.
You can set large values to m_E in the background region. I suppose this might be a solution?

VToonify/model/vtoonify.py

Lines 258 to 260 in db57c27

 out, m_E = self.fusion_out[fusion_index](out, f_E, d_s) 
 skip = self.fusion_skip[fusion_index](torch.cat([skip, f_E*m_E], dim=1)) 
 m_Es += [m_E] 

[Owen]

OK ,I have a try ! thanks !