Is rotated generated images a normal part of the training progression.
Tearumapart opened this issue · comments
I have been training a gans with the stylegans3-r configuration for about 7kimgs. I am using your fork and have taken advice from some of the other issues on the main repo that you have commented on in the past.
I have been using these arguments: --gpus=8 --batch=32 --gamma=32 --aug=ada --augpipe=bg --target=0.8 --initstrength=[last training round augment score] --snap=10 --img-snap=10 --mirror=1 --metrics=none --resume-kimg=[foobar k imgs]
At a little over 5kimgs the generated images began to be rotated 90 degrees right, then after some time they transitioned to being rotated 90 degrees the opposite direction. They have currently been rotated upside down.
Is this a normal part of the training progression in which the generated images will return to the proper orientation, or is this some type of mode collapse or augmentation leak?
In case its useful the images were non square so white bars have been added to each side to make them square.
The objects in the images themselves are not perfectly symmetrical especial since there is various high and low perspectives.
The images themselves look ok but definitely not converged.
As stylegans3 training is significantly slower than 2 is I have put the training on hiatus until I know if I can continue training this model or if I should start over from scratch.
Sorry for the late reply, but indeed StyleGAN3's config-r is super slow and doesn't really bring much to the table. At best, I use config-t, but I've found myself mostly using StyleGAN2's config-f. The augmentations might've leaked, but I haven't really encountered them myself. I can look into it, assuming you still have this problem, if not then I might just close this issue.