unable to use ControlNet extensions when tensorRTs are in use

[randomname1212]

with the following Settings --> Show all Pages --> SD Unet set to "automatic". everything runs fine when generating images I get a a nice boost to the iterations per second. However, when i try to use controlnet extensions for example open pose i get the error

"RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument mat1 in method wrapper_CUDA_addmm)"

Can this error be fixed by converting the .pth files found in "D:\sd.webuiTensor\webui\extensions\sd-webui-controlnet\models"
i.e .pth to .ONNX to .TRT

The work around is to set the SD Unet to "none" each time you want to use the control net extensions then set SD Unet back to "automatic" when i'm not using ControlNet extensions.

Thank you so much developers and community for your hard work on these projects, I really appreciate it.

Full error below:

*** Error completing request
*** Arguments: ('task(aoav9synepbw0lt)', 'a man filming the moon landing,', '', [], 30, 0, False, False, 1, 1, 7, -1.0, -1.0, 0, 0, 0, False, 512, 512, False, 0.7, 2, 'Latent', 0, 0, 0, 0, '', '', [], <gradio.routes.Request object at 0x000002B01B1B1C00>, 0, <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at 0x000002B01BC7F7F0>, <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at 0x000002B01BC7F9D0>, <scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at 0x000002B01B1B05B0>, False, False, 0, None, [], 0, False, [], [], False, 0, 1, False, False, 0, None, [], -2, False, [], False, 0, None, None, False, False, 'Denoised', 5.0, 0.0, 0.0, 'Standard operation', 'mp4', 'h264', 2.0, 0.0, 0.0, False, 0.0, True, True, False, False, False, False, False, 'positive', 'comma', 0, False, False, '', 1, '', [], 0, '', [], 0, '', [], True, False, False, False, 0, '', '', 0, None, None, False, None, None, False, None, None, False, 50) {}
Traceback (most recent call last):
File "D:\sd.webuiTensor\webui\modules\call_queue.py", line 58, in f
res = list(func(*args, **kwargs))
File "D:\sd.webuiTensor\webui\modules\call_queue.py", line 37, in f
res = func(*args, **kwargs)
File "D:\sd.webuiTensor\webui\modules\txt2img.py", line 59, in txt2img
processed = processing.process_images(p)
File "D:\sd.webuiTensor\webui\extensions\sd-webui-prompt-history\lib_history\image_process_hijacker.py", line 21, in process_images
res = original_function(p)
File "D:\sd.webuiTensor\webui\modules\processing.py", line 624, in process_images
res = process_images_inner(p)
File "D:\sd.webuiTensor\webui\extensions\sd-webui-controlnet\scripts\batch_hijack.py", line 42, in processing_process_images_hijack
return getattr(processing, '__controlnet_original_process_images_inner')(p, *args, **kwargs)
File "D:\sd.webuiTensor\webui\modules\processing.py", line 743, in process_images_inner
samples_ddim = p.sample(conditioning=p.c, unconditional_conditioning=p.uc, seeds=p.seeds, subseeds=p.subseeds, subseed_strength=p.subseed_strength, prompts=p.prompts)
File "D:\sd.webuiTensor\webui\extensions\sd-webui-controlnet\scripts\hook.py", line 350, in process_sample
return process.sample_before_CN_hack(*args, **kwargs)
File "D:\sd.webuiTensor\webui\modules\processing.py", line 996, in sample
samples = self.sampler.sample(self, x, conditioning, unconditional_conditioning, image_conditioning=self.txt2img_image_conditioning(x))
File "D:\sd.webuiTensor\webui\modules\sd_samplers_kdiffusion.py", line 439, in sample
samples = self.launch_sampling(steps, lambda: self.func(self.model_wrap_cfg, x, extra_args={
File "D:\sd.webuiTensor\webui\modules\sd_samplers_kdiffusion.py", line 278, in launch_sampling
return func()
File "D:\sd.webuiTensor\webui\modules\sd_samplers_kdiffusion.py", line 439, in
samples = self.launch_sampling(steps, lambda: self.func(self.model_wrap_cfg, x, extra_args={
File "D:\sd.webuiTensor\webui\venv\lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context return func(*args, **kwargs)
File "D:\sd.webuiTensor\webui\repositories\k-diffusion\k_diffusion\sampling.py", line 145, in sample_euler_ancestral
denoised = model(x, sigmas[i] * s_in, **extra_args)
File "D:\sd.webuiTensor\webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "D:\sd.webuiTensor\webui\modules\sd_samplers_kdiffusion.py", line 158, in forward
x_out = self.inner_model(x_in, sigma_in, cond=make_condition_dict([cond_in], image_cond_in))
File "D:\sd.webuiTensor\webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "D:\sd.webuiTensor\webui\repositories\k-diffusion\k_diffusion\external.py", line 112, in forward
eps = self.get_eps(input * c_in, self.sigma_to_t(sigma), **kwargs)
File "D:\sd.webuiTensor\webui\repositories\k-diffusion\k_diffusion\external.py", line 138, in get_eps
return self.inner_model.apply_model(*args, **kwargs)
File "D:\sd.webuiTensor\webui\modules\sd_hijack_utils.py", line 17, in
setattr(resolved_obj, func_path[-1], lambda *args, **kwargs: self(*args, **kwargs))
File "D:\sd.webuiTensor\webui\modules\sd_hijack_utils.py", line 28, in call
return self.__orig_func(*args, **kwargs)
File "D:\sd.webuiTensor\webui\repositories\stable-diffusion-stability-ai\ldm\models\diffusion\ddpm.py", line 858, in apply_model
x_recon = self.model(x_noisy, t, **cond)
File "D:\sd.webuiTensor\webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "D:\sd.webuiTensor\webui\repositories\stable-diffusion-stability-ai\ldm\models\diffusion\ddpm.py", line 1335, in forward
out = self.diffusion_model(x, t, context=cc)
File "D:\sd.webuiTensor\webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1538, in _call_impl
result = forward_call(*args, **kwargs)
File "D:\sd.webuiTensor\webui\extensions\sd-webui-controlnet\scripts\hook.py", line 661, in forward_webui
return forward(*args, **kwargs)
File "D:\sd.webuiTensor\webui\extensions\sd-webui-controlnet\scripts\hook.py", line 577, in forward
emb = self.time_embed(t_emb)
File "D:\sd.webuiTensor\webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "D:\sd.webuiTensor\webui\venv\lib\site-packages\torch\nn\modules\container.py", line 217, in forward
input = module(input)
File "D:\sd.webuiTensor\webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "D:\sd.webuiTensor\webui\extensions-builtin\Lora\lora.py", line 400, in lora_Linear_forward
return torch.nn.Linear_forward_before_lora(self, input)
File "D:\sd.webuiTensor\webui\venv\lib\site-packages\torch\nn\modules\linear.py", line 114, in forward
return F.linear(input, self.weight, self.bias)
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument mat1 in method wrapper_CUDA_addmm)

side note: I have successfully converted the modules below to TensorRTs and seen a significant boost in it/s from around 12 to 15 it/s to around 25 to 33 it/s
I think my settings for the conversion was min & max size of 512x512, maximum batch of 6, Maximum prompt token count 525

chilloutmix_NiPrunedFp32Fix.safetensors [fc2511737a] @Index:0,
deliberate_v2.safetensors [9aba26abdf] @Index:1,
dreamlike-photoreal-2.0.ckpt [fc52756a74] @Index:2,
dreamshaper_331BakedVae.safetensors [9e9fa0d822] @Index:3,
f222.ckpt [9e2c6ceff3] @Index:4,
HassanBlend1.4_Safe.safetensors [b08fdba169] @Index:5,
lyriel_v16.safetensors [ec6f68ea63] @Index:6,
mdjrny-v4.ckpt [5d5ad06cc2] @Index:7,
realisticVisionV20_v20NoVAE.safetensors [c0d1994c73] @Index:8,
v1-5-pruned-emaonly.safetensors [6ce0161689] @Index:9,
v2-1_512-ema-pruned.ckpt [88ecb78256] @Index:10

[78Alpha]

As stated in the README, TensorRT does not work with controlnet. CKPT and SAFETENSOR can be altered on the fly without issue. TensorRT isn't as graceful. What you gain in speed you lose in utility. Maybe it could be done, but you would have to speak with the creator of the controlnet models.