photomaker_demo.ipynb throws error on Apple Silicon

[Zheng Qu]

It would be great if Apple Silicon is supported.

photomaker_demo.ipynb is throwing error on Apple Silicon. See the post below for the error message.

[Zhen Li]

Does this PR #17 solve your problem?

[Zheng Qu]

@Paper99 Thanks for the quick response. I tried the notebook https://github.com/TencentARC/PhotoMaker/blob/main/photomaker_demo.ipynb
and changed the device to mps. That gave me the following error:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[3], line 26
     24 pipe.scheduler = DDIMScheduler.from_config(pipe.scheduler.config)
     25 # pipe.set_adapters(["photomaker"], adapter_weights=[1.0])
---> 26 pipe.fuse_lora()

File ~/.miniconda3/envs/photomaker/lib/python3.10/site-packages/diffusers/loaders/lora.py:1050, in LoraLoaderMixin.fuse_lora(self, fuse_unet, fuse_text_encoder, lora_scale, safe_fusing, adapter_names)
   1048 if fuse_unet:
   1049     unet = getattr(self, self.unet_name) if not hasattr(self, "unet") else self.unet
-> 1050     unet.fuse_lora(lora_scale, safe_fusing=safe_fusing, adapter_names=adapter_names)
   1052 if USE_PEFT_BACKEND:
   1053     from peft.tuners.tuners_utils import BaseTunerLayer

File ~/.miniconda3/envs/photomaker/lib/python3.10/site-packages/diffusers/loaders/unet.py:512, in UNet2DConditionLoadersMixin.fuse_lora(self, lora_scale, safe_fusing, adapter_names)
    510 self.lora_scale = lora_scale
    511 self._safe_fusing = safe_fusing
--> 512 self.apply(partial(self._fuse_lora_apply, adapter_names=adapter_names))

File ~/.miniconda3/envs/photomaker/lib/python3.10/site-packages/torch/nn/modules/module.py:884, in Module.apply(self, fn)
    849 r"""Applies ``fn`` recursively to every submodule (as returned by ``.children()``)
    850 as well as self. Typical use includes initializing the parameters of a model
    851 (see also :ref:`nn-init-doc`).
   (...)
    881 
    882 """
    883 for module in self.children():
--> 884     module.apply(fn)
    885 fn(self)
    886 return self

File ~/.miniconda3/envs/photomaker/lib/python3.10/site-packages/torch/nn/modules/module.py:884, in Module.apply(self, fn)
    849 r"""Applies ``fn`` recursively to every submodule (as returned by ``.children()``)
    850 as well as self. Typical use includes initializing the parameters of a model
    851 (see also :ref:`nn-init-doc`).
   (...)
    881 
    882 """
    883 for module in self.children():
--> 884     module.apply(fn)
    885 fn(self)
    886 return self

    [... skipping similar frames: Module.apply at line 884 (5 times)]

File ~/.miniconda3/envs/photomaker/lib/python3.10/site-packages/torch/nn/modules/module.py:884, in Module.apply(self, fn)
    849 r"""Applies ``fn`` recursively to every submodule (as returned by ``.children()``)
    850 as well as self. Typical use includes initializing the parameters of a model
    851 (see also :ref:`nn-init-doc`).
   (...)
    881 
    882 """
    883 for module in self.children():
--> 884     module.apply(fn)
    885 fn(self)
    886 return self

File ~/.miniconda3/envs/photomaker/lib/python3.10/site-packages/torch/nn/modules/module.py:885, in Module.apply(self, fn)
    883 for module in self.children():
    884     module.apply(fn)
--> 885 fn(self)
    886 return self

File ~/.miniconda3/envs/photomaker/lib/python3.10/site-packages/diffusers/loaders/unet.py:545, in UNet2DConditionLoadersMixin._fuse_lora_apply(self, module, adapter_names)
    539 elif "adapter_names" not in supported_merge_kwargs and adapter_names is not None:
    540     raise ValueError(
    541         "The `adapter_names` argument is not supported with your PEFT version. Please upgrade"
    542         " to the latest version of PEFT. `pip install -U peft`"
    543     )
--> 545 module.merge(**merge_kwargs)

File ~/.miniconda3/envs/photomaker/lib/python3.10/site-packages/peft/tuners/lora/layer.py:305, in Linear.merge(self, safe_merge, adapter_names)
    303     base_layer.weight.data = orig_weights
    304 else:
--> 305     base_layer.weight.data += self.get_delta_weight(active_adapter)
    306 self.merged_adapters.append(active_adapter)

File ~/.miniconda3/envs/photomaker/lib/python3.10/site-packages/peft/tuners/lora/layer.py:343, in Linear.get_delta_weight(self, adapter)
    340     weight_A = weight_A.float()
    341     weight_B = weight_B.float()
--> 343 output_tensor = transpose(weight_B @ weight_A, self.fan_in_fan_out) * self.scaling[adapter]
    345 if cast_to_fp32:
    346     output_tensor = output_tensor.to(dtype=dtype)

RuntimeError: MPS device does not support mm for non-float inputs

[Zhen Li]

Can you successfully run our gradient demo?

[Zheng Qu]

Can you successfully run our gradient demo?

Yes. your gradient demo works great! I will modify the title of the issue.

[pstadnyk]

Following. I am also getting a PEFT error "ValueError: PEFT backend is required for 'set_adapters()"

I'm running this on a RunPod and haven't been able to find a fix.