Noisy image generations
machineminded opened this issue · comments
Hello, I am getting noisy generations and the likeness is not there. Here is my code:
import torch
import os
import sys
sys.path.append('../photomaker')
from diffusers import EulerDiscreteScheduler, DPMSolverMultistepScheduler, AutoencoderKL
from photomaker import PhotoMakerStableDiffusionXLPipeline
from huggingface_hub import hf_hub_download
import modules.default_pipeline as pipeline
base_model_path = pipeline.model_base.filename
photomaker_ckpt = hf_hub_download(repo_id="TencentARC/PhotoMaker", filename="photomaker-v1.bin", repo_type="model")
def generate_photomaker(prompt, input_id_images, negative_prompt, steps, seed, width, height, guidance_scale):
print(f"Using base model: {base_model_path} for PhotoMaker")
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
pipe = PhotoMakerStableDiffusionXLPipeline.from_single_file(
base_model_path,
torch_dtype=torch.bfloat16,
use_safetensors=True,
variant="fp16"
).to(device)
pipe.load_photomaker_adapter(
os.path.dirname(photomaker_ckpt),
subfolder="",
weight_name=os.path.basename(photomaker_ckpt),
trigger_word="img"
)
pipe.id_encoder.to(device)
pipe.scheduler = DPMSolverMultistepScheduler.from_config(pipe.scheduler.config)
pipe.fuse_lora()
generator = torch.Generator(device=device).manual_seed(seed)
images = pipe(
prompt=prompt,
input_id_images=input_id_images,
negative_prompt=negative_prompt,
num_images_per_prompt=1,
num_inference_steps=steps,
width=width,
height=height,
start_merge_step=10,
generator=generator,
guidance_scale=guidance_scale
).images
return images
Any idea what might be going on?
I found that you are using dpmsolver. Could you increase the number of steps and try again?
Maybe you can also choose to use the quality
option
Hello. I did more experimenting today, and it appears to be due to my simple prompt and lack of a negative prompt. Apply a prompt from the style seems to resolve it. I will post my findings in the morning. Thank you.
Cool! Thanks again for your efforts!
It's really cool!
It seems like the euler scheduler generates better images than DPM++. I even tried adding euler_at_final
but the generations with DPM++ still are noisy. Even with 60+ steps.
DPMSolverMultistepScheduler(algorithm_type="sde-dpmsolver++", euler_at_final=True)
DPMSolverMultistepScheduler(algorithm_type="sde-dpmsolver++", euler_at_final=True);
DPM_hash($sove)=>solution;
@machineminded Nice works! Could you please elaborate a bit more on how you combining photomaker with inswapper_128 for these nice photos?
@machineminded Nice works! Could you please elaborate a bit more on how you combining photomaker with inswapper_128 for these nice photos?
@CHELSEA234 I am using my fork of Fooocus with Inswapper and Photomaker! Soon instantID will be integrated as well.
To elaborate, I use PhotoMaker to create a likeness "substrate" and then inswapper lays the face on top.