How to feed latent.npy to fine-tuned StyleGAN 3 model?

[kimyanna]

Hi, first of all thank you for putting this repo together, you are demonstrating so much skill!

I have a noob question: When I invert an image with inference_iterative.py and get latents.npy, how can I then feed latents.npy back into my fine-tuned StyleGAN 3 model? Would I have to somehow modify gen_images.py of the main SG3 repo and feed the vector into that?

This is how I'm getting latents.npy but I'm not sure how to then feed the vector into my fine-tuned model:

python /content/stylegan3-editing/inversion/scripts/inference_iterative.py \
  --output_path /content/inference \
  --checkpoint_path /content/stylegan3-editing/restyle_pSp_ffhq.pt \
  --data_path /content/data \
  --test_batch_size 4 \
  --test_workers 4 \
  --n_iters_per_batch 3

Thank you!

[yuval-alaluf]

If you have loaded the latents that were saved in the npy file, you can generate the corresponding images using something like:

# load latents 
latents = np.load("latents.py", allow_pickle=True).item() 
# get one of the latent codes and get the last code that was saved
latent = latents["image.png"][-1]
# generate the image
images = self.generator.synthesis(latent, noise_mode='const')
# convert the PIL
images = [tensor2im(image) for image in images]

I hope something like that works, but I hope the flow is clear.
(self.generator can be defined as: generator = SG3Generator(checkpoint_path="generator_path").decoder)

[kimyanna]

Thank you very much, I think I'm getting closer with the following code:

from models.stylegan3.model import SG3Generator
from utils.common import tensor2im

generator = SG3Generator(checkpoint_path="/content/stylegan3-t-ffhqu-1024x1024.pkl").decoder

# load latents 
latents = np.load("/content/inference/latents.npy", allow_pickle=True).item() 
# get one of the latent codes and get the last code that was saved
latent = latents["will2.jpg"][-1]
# generate the image
images = generator.synthesis(latent, noise_mode='const')
# convert the PIL
images = [tensor2im(image) for image in images]

But it throws the following assertion error:

Loading StyleGAN3 generator from path: /content/stylegan3-t-ffhqu-1024x1024.pkl
Done!
---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-41-7de6c774a1d9> in <module>
     10 latent = latents["will2.jpg"][-1]
     11 # generate the image
---> 12 images = generator.synthesis(latent, noise_mode='const')
     13 # convert the PIL
     14 images = [tensor2im(image) for image in images]

1 frames
<string> in forward(self, ws, **layer_kwargs)

/content/stylegan3-editing/torch_utils/misc.py in assert_shape(tensor, ref_shape)
     84 def assert_shape(tensor, ref_shape):
     85     if tensor.ndim != len(ref_shape):
---> 86         raise AssertionError(f'Wrong number of dimensions: got {tensor.ndim}, expected {len(ref_shape)}')
     87     for idx, (size, ref_size) in enumerate(zip(tensor.shape, ref_shape)):
     88         if ref_size is None:

AssertionError: Wrong number of dimensions: got 2, expected 3

Any tip on what might be causing this? Thank you!

[Laura Álvarez]

As far as I know before synthesis you have to add this one

w= generator.mapping(latent, None, truncation_psi=truncation_psi)

images = generator.synthesis(w, noise_mode='const')

[yuval-alaluf]

But it throws the following assertion error:

You might just need to do

images = generator.synthesis(latent.unsqueeze(0), noise_mode='const')

I believe latent should be of dimension 1,18,512 or something like that.