lyc0929 / OOTDiffusion-train

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

getting blurred result for trained model

rohitpaul23 opened this issue · comments

Have finetuned the model using smaller test images for 1k epochs. The loss graph is not converging and when inferencing on final and intermediate saved models, it gives a similar result to that of the base-trained model

So to be sure, I have trained the model on base SD1.5, I used 10 images and trained it for 500 epochs and it seems it's not learning as the result I am getting from the trained model is blurry
have

what could be the problem here
prob

When debugged, it seems the model is not training. I was trying to train the model in fp16 with deepspeed. It run ok with fp32