Diffusers pipeline error

[Eliseu Silva]

When i try replicate infer_style example in diffusers pipeline didnt work:
i have this error in sdp attention:

Exception has occurred: TypeError       (note: full exception trace is shown but execution is paused at: _run_module_as_main)
unsupported operand type(s) for *: 'dict' and 'Tensor'
  File "F:\Projetos\InstantStyle\venv\Lib\site-packages\diffusers\models\attention_processor.py", line 2417, in __call__
    hidden_states = hidden_states + scale * current_ip_hidden_states
  File "F:\Projetos\InstantStyle\venv\Lib\site-packages\diffusers\models\attention_processor.py", line 522, in forward
    return self.processor(
  File "F:\Projetos\InstantStyle\venv\Lib\site-packages\torch\nn\modules\module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "F:\Projetos\InstantStyle\venv\Lib\site-packages\torch\nn\modules\module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "F:\Projetos\InstantStyle\venv\Lib\site-packages\diffusers\models\attention.py", line 366, in forward
    attn_output = self.attn2(
  File "F:\Projetos\InstantStyle\venv\Lib\site-packages\torch\nn\modules\module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "F:\Projetos\InstantStyle\venv\Lib\site-packages\torch\nn\modules\module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "F:\Projetos\InstantStyle\venv\Lib\site-packages\diffusers\models\transformers\transformer_2d.py", line 397, in forward
    hidden_states = block(
  File "F:\Projetos\InstantStyle\venv\Lib\site-packages\torch\nn\modules\module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "F:\Projetos\InstantStyle\venv\Lib\site-packages\torch\nn\modules\module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "F:\Projetos\InstantStyle\venv\Lib\site-packages\diffusers\models\unets\unet_2d_blocks.py", line 1279, in forward
    hidden_states = attn(
  File "F:\Projetos\InstantStyle\venv\Lib\site-packages\torch\nn\modules\module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "F:\Projetos\InstantStyle\venv\Lib\site-packages\torch\nn\modules\module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "F:\Projetos\InstantStyle\venv\Lib\site-packages\diffusers\models\unets\unet_2d_condition.py", line 1216, in forward
    sample, res_samples = downsample_block(
  File "F:\Projetos\InstantStyle\venv\Lib\site-packages\torch\nn\modules\module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "F:\Projetos\InstantStyle\venv\Lib\site-packages\torch\nn\modules\module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "F:\Projetos\InstantStyle\venv\Lib\site-packages\diffusers\pipelines\stable_diffusion_xl\pipeline_stable_diffusion_xl.py", line 1174, in __call__
    noise_pred = self.unet(
  File "F:\Projetos\InstantStyle\venv\Lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "F:\Projetos\InstantStyle\infer_style_diff.py", line 53, in <module>
    images = pipe(
  File "C:\Users\pelol\.pyenv\pyenv-win\versions\3.10.11\Lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Users\pelol\.pyenv\pyenv-win\versions\3.10.11\Lib\runpy.py", line 196, in _run_module_as_main (Current frame)
    return _run_code(code, main_globals, None,
TypeError: unsupported operand type(s) for *: 'dict' and 'Tensor'

My code:

import torch
from diffusers import StableDiffusionXLPipeline
from PIL import Image
from transformers import CLIPVisionModelWithProjection
from ip_adapter import IPAdapterXL
from ip_adapter.utils import get_generator

base_model_path = "stabilityai/stable-diffusion-xl-base-1.0"
device = "cuda"
seed = 42

image_encoder = CLIPVisionModelWithProjection.from_pretrained(
    "h94/IP-Adapter", subfolder="sdxl_models/image_encoder", torch_dtype=torch.float16
).to("cuda")

# load SDXL pipeline
pipe = StableDiffusionXLPipeline.from_pretrained(
    base_model_path,
    image_encoder=image_encoder,
    torch_dtype=torch.float16,
    add_watermarker=False,
).to(device)

# load ip-adapter
pipe.load_ip_adapter("h94/IP-Adapter",
    subfolder="sdxl_models",
    weight_name="ip-adapter_sdxl.bin" ,
    image_encoder_folder="image_encoder",   
)

pipe.enable_vae_tiling() 

# configure ip-adapter scales.
scale = {
    "down": {"block_2": [0.0, 1.0]},
    "up": {"block_0": [0.0, 1.0, 0.0]},
}
pipe.set_ip_adapter_scale(scale)

# load ip-adapter
# target_blocks=["block"] for original IP-Adapter
# target_blocks=["up_blocks.0.attentions.1"] for style blocks only
# target_blocks = ["up_blocks.0.attentions.1", "down_blocks.2.attentions.1"] # for style+layout blocks
#ip_model = IPAdapterXL(pipe, image_encoder_path, ip_ckpt, device, target_blocks=["up_blocks.0.attentions.1"])
generator = get_generator(seed, device)
image = "./assets/0.jpg"
image = Image.open(image)
image.resize((512, 512))

# generate image
images = pipe(
    prompt="a cat, masterpiece, best quality, high quality",
    negative_prompt= "text, watermark, lowres, low quality, worst quality, deformed, glitch, low contrast, noisy, saturation, blurry",
    ip_adapter_image=image,    
    guidance_scale=5,    
    num_inference_steps=30,
    num_images_per_prompt=1,    
    generator=generator
).images[0]


images[0].save("result_1_diff.png")

[Eliseu Silva]

i found the problem was diffuser version, we need clone source and install from source 0.28.0.dev0.

Since I'm here, I would like to ask if it is possible to run multiple ip adapters with masks like example provided using diffusers, in ?the IPAdapterXL class ?

[Mohammed Ali]

Facing the same issue

[Dana Cohen-Bar]

@elismasilva
I can't find the diffusers version you mentioned (0.28.0.dev0) , can you please link it?

[Eliseu Silva]

@elismasilva I can't find the diffusers version you mentioned (0.28.0.dev0) , can you please link it?

just clone main branch and install from source.