Instruct pix2pix RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.HalfTensor) should be the same
SuperMaximus1984 opened this issue · comments
When I run InstructPix2Pix_cuda:0 I receive an error
RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.HalfTensor) should be the same
Whenver I run it InstructPix2Pix_cpu everything works well, but slow.
I have torch 2 with CUDA, GroundingDINO installed with GPU.
How do I fix the RuntimeError and run InstructPix2Pix_cuda:0?
same here
same problem.
Entering new AgentExecutor chain...
Action: Segment the Image
Action Input: image/c4ca40ac.png
Observation: image/02c3_segment-image_c4ca40ac_c4ca40ac.png
Thought:Action: Remove Something From The Photo
Action Input: image/02c3_segment-image_c4ca40ac_c4ca40ac.png, small brown dogimage_path=image/02c3_segment-image_c4ca40ac_c4ca40ac.png, to_be_replaced_txt= small brown dog
100%
50/50 [00:10<00:00, 4.86it/s]
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/gradio/routes.py", line 437, in run_predict
output = await app.get_blocks().process_api(
File "/usr/local/lib/python3.10/dist-packages/gradio/blocks.py", line 1352, in process_api
result = await self.call_function(
File "/usr/local/lib/python3.10/dist-packages/gradio/blocks.py", line 1077, in call_function
prediction = await anyio.to_thread.run_sync(
File "/usr/local/lib/python3.10/dist-packages/anyio/to_thread.py", line 33, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "/usr/local/lib/python3.10/dist-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
return await future
File "/usr/local/lib/python3.10/dist-packages/anyio/_backends/_asyncio.py", line 807, in run
result = context.run(func, *args)
File "/content/TaskMatrix/visual_chatgpt.py", line 1520, in run_text
res = self.agent({"input": text.strip()})
File "/usr/local/lib/python3.10/dist-packages/langchain/chains/base.py", line 168, in call
raise e
File "/usr/local/lib/python3.10/dist-packages/langchain/chains/base.py", line 165, in call
outputs = self._call(inputs)
File "/usr/local/lib/python3.10/dist-packages/langchain/agents/agent.py", line 503, in _call
next_step_output = self._take_next_step(
File "/usr/local/lib/python3.10/dist-packages/langchain/agents/agent.py", line 420, in _take_next_step
observation = tool.run(
File "/usr/local/lib/python3.10/dist-packages/langchain/tools/base.py", line 71, in run
raise e
File "/usr/local/lib/python3.10/dist-packages/langchain/tools/base.py", line 68, in run
observation = self._run(tool_input)
File "/usr/local/lib/python3.10/dist-packages/langchain/agents/tools.py", line 17, in _run
return self.func(tool_input)
File "/content/TaskMatrix/visual_chatgpt.py", line 1384, in inference_remove
return self.inference_replace_sam(f"{image_path},{to_be_removed_txt},background")
File "/content/TaskMatrix/visual_chatgpt.py", line 1408, in inference_replace_sam
updated_image = self.inpaint(prompt=replace_with_txt, image=image_pil,
File "/content/TaskMatrix/visual_chatgpt.py", line 1180, in call
update_image = self.inpaint(prompt=prompt, image=image.resize((width, height)),
File "/usr/local/lib/python3.10/dist-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_inpaint.py", line 1056, in call
image, has_nsfw_concept = self.run_safety_checker(image, device, prompt_embeds.dtype)
File "/usr/local/lib/python3.10/dist-packages/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_inpaint.py", line 536, in run_safety_checker
image, has_nsfw_concept = self.safety_checker(
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/diffusers/pipelines/stable_diffusion/safety_checker.py", line 51, in forward
pooled_output = self.vision_model(clip_input)[1] # pooled_output
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/transformers/models/clip/modeling_clip.py", line 940, in forward
return self.vision_model(
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/transformers/models/clip/modeling_clip.py", line 865, in forward
hidden_states = self.embeddings(pixel_values)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(input, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/transformers/models/clip/modeling_clip.py", line 195, in forward
patch_embeds = self.patch_embedding(pixel_values) # shape = [, width, grid, grid]
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/conv.py", line 463, in forward
return self._conv_forward(input, self.weight, self.bias)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/conv.py", line 459, in _conv_forward
return F.conv2d(input, weight, bias, self.stride,
RuntimeError: Input type (torch.cuda.HalfTensor) and weight type (torch.cuda.FloatTensor) should be the same
Use search and replace tools to replace all torch.float16 to torch.float32!
Use search and replace tools to replace all torch.float16 to torch.float32!
In this file?
pipeline_stable_diffusion_inpaint.py
Setting
self.torch_dtype = torch.float32 instead
self.torch_dtype = torch.float16 if 'cuda' in device else torch.float32
in all model class initialization @ visual_chatgpt.py worked on RTX 3090 CUDA 11.7!