How to convert and use ckpt sd models with prompt free diffusion

[Vineet]

Hi,
I am trying to use different base diffusion model in place of the provided ones:- RealisticVision-2.0, Deliberate-v2.0 etc. I am using the ckpt file and getting key error while loading the state_dict.

File "/home/user/.pyenv/versions/3.10.11/lib/python3.10/site-packages/gradio/routes.py", line 422, in run_predict
    output = await app.get_blocks().process_api(
  File "/home/user/.pyenv/versions/3.10.11/lib/python3.10/site-packages/gradio/blocks.py", line 1323, in process_api
    result = await self.call_function(
  File "/home/user/.pyenv/versions/3.10.11/lib/python3.10/site-packages/gradio/blocks.py", line 1051, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "/home/user/.pyenv/versions/3.10.11/lib/python3.10/site-packages/anyio/to_thread.py", line 33, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/home/user/.pyenv/versions/3.10.11/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
    return await future
  File "/home/user/.pyenv/versions/3.10.11/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 807, in run
    result = context.run(func, *args)
  File "/home/user/app/app.py", line 260, in action_inference
    self.action_load_diffuser(tag_diffuser)
  File "/home/user/app/app.py", line 226, in action_load_diffuser
    self.load_diffuser(pretrained)
  File "/home/user/app/app.py", line 196, in load_diffuser
    self.net.load_state_dict(sd, strict=True)
  File "/home/user/.pyenv/versions/3.10.11/lib/python3.10/site-packages/torch/nn/modules/module.py", line 2041, in load_state_dict
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for PromptFreeDiffusion_with_control:
	Missing key(s) in state_dict: "diffuser.image.time_embed.0.weight", "diffuser.image.time_embed.0.bias", "diffuser.image.time_embed.2.weight", "diffuser.image.time_embed.2.bias", "diffuser.image.data_blocks.0.0.weight", "diffuser.image.data_blocks.0.0.bias", "diffuser.image.data_blocks.1.0.in_layers.0.weight", "diffuser.image.data_blocks.1.0.in_layers.0.bias", "diffuser.image.data_blocks.1.0.in_layers.2.weight", "diffuser.image.data_blocks.1.0.in_layers.2.bias", "diffuser.image.data_blocks.1.0.emb_layers.1.weight", "diffuser.image.data_blocks.1.0.emb_layers.1.bias", "diffuser.image.data_blocks.1.0.out_layers.0.weight", "diffuser.image.data_blocks.1.0.out_layers.0.bias", "diffuser.image.data_blocks.1.0.out_layers.3.weight", "diffuser.image.data_blocks.1.0.out_layers.3.bias", "diffuser.image.data_blocks.2.0.in_layers.0.weight", "diffuser.image.data_blocks.2.0.in_layers.0.bias", "diffuser.image.data_blocks.2.0.in_layers.2.weight", "diffuser.image.data_blocks.2.0.in_layers.2.bias", "diffuser.image.data_blocks.2.0.emb_layers.1.weight", "diffuser.image.data_blocks.2.0.emb_layers.1.bias", "diffuser.image.data_blocks.2.0.out_layers.0.weight", "diffuser.image.data_blocks.2.0.out_layers.0.bias", "diffuser.image.data_blocks.2.0.out_layers.3.weight", "diffuser.image.data_blocks.2.0.out_layers.3.bias",  .......................................

It seems diffusers/ other structure is expected.
Can you share how to convert basic ckpt models to use with prompt-free diffusion?

[Xingqian Xu]

Please see tools/model_conversion.py