Missing weights for depth estimation.
JiuTongBro opened this issue · comments
Hi, thanks for your impressive work firstly.
However, when I try to add an instance to the scene, the code threw an error:
FileNotFoundError: [Errno 2] No such file or directory: '.cache/dpt/omnidata_dpt_depth_v2.ckpt'
I suppose it is because the weights for depth estimation is missed. I tried to download it manually and put it under the utils/cache/dpt/ folder, but it still doesn't work. Do you know how to solve it?
This is the full logs about the error, it happened after the Wonder3D reconstruction.
Traceback (most recent call last):
File "/home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/lib/python3.8/site-packages/mediapy/__init__.py", line 1739, in write_video
writer.add_image(image)
File "/home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/lib/python3.8/site-packages/mediapy/__init__.py", line 1648, in add_image
if self._proc.stdin.write(data) != len(data):
BrokenPipeError: [Errno 32] Broken pipe
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "train_from_mesh.py", line 261, in <module>
cams, images, camera_extent = render_multiview_images_from_mesh(args.mesh,
File "/home/zhonghongliang/diffusion_pro/gsedit_pro/GaussianEditor/threestudio/utils/render.py", line 150, in render_multiview_images_from_mesh
write_video(save_path, frames, fps=2)
File "/home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/lib/python3.8/site-packages/mediapy/__init__.py", line 1739, in write_video
writer.add_image(image)
File "/home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/lib/python3.8/site-packages/mediapy/__init__.py", line 1610, in __exit__
self.close()
File "/home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/lib/python3.8/site-packages/mediapy/__init__.py", line 1661, in close
raise RuntimeError(f"Error writing '{self.path}': {s}")
RuntimeError: Error writing '/home/zhonghongliang/diffusion_pro/gsedit_pro/GaussianEditor/tmp_add/mesh_rendering.mp4': /home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/bin/ffmpeg: error while loading shared libraries: libopenh264.so.5: cannot open shared object file: No such file or directory
/home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/lib/python3.8/site-packages/timm/models/_factory.py:117: UserWarning: Mapping deprecated model name vit_base_resnet50_384 to current vit_base_r50_s16_384.orig_in21k_ft_in1k.
model = create_fn(
Traceback (most recent call last):
File "/home/zhonghongliang/diffusion_pro/gsedit_pro/GaussianEditor/extern/viser/src/viser/infra/_infra.py", line 449, in wrapped
inner()
File "/home/zhonghongliang/diffusion_pro/gsedit_pro/GaussianEditor/extern/viser/src/viser/infra/_infra.py", line 257, in <lambda>
lambda: self._handle_incoming_message(client_id, message)
File "/home/zhonghongliang/diffusion_pro/gsedit_pro/GaussianEditor/extern/viser/src/viser/infra/_infra.py", line 94, in _handle_incoming_message
cb(client_id, message)
File "/home/zhonghongliang/diffusion_pro/gsedit_pro/GaussianEditor/extern/viser/src/viser/_gui_api.py", line 157, in _handle_gui_updates
cb(GuiEvent(client, client_id, handle))
File "webui.py", line 511, in _
self.add(self.camera)
File "/home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "webui.py", line 1371, in add
depth_estimator = DPT(get_device(), mode="depth")
File "/home/zhonghongliang/diffusion_pro/gsedit_pro/GaussianEditor/threestudio/utils/dpt.py", line 1023, in __init__
checkpoint = torch.load(path, map_location="cpu")
File "/home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/lib/python3.8/site-packages/torch/serialization.py", line 986, in load
with _open_file_like(f, 'rb') as opened_file:
File "/home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/lib/python3.8/site-packages/torch/serialization.py", line 435, in _open_file_like
return _open_file(name_or_buffer, mode)
File "/home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/lib/python3.8/site-packages/torch/serialization.py", line 416, in __init__
super().__init__(open(name, mode))
FileNotFoundError: [Errno 2] No such file or directory: '.cache/dpt/omnidata_dpt_depth_v2.ckpt'
Thanks for your interest.
It supposed to be automatically downloaded from online.
I'm not sure why the download failed in your case.
Hi, JiuTongBro
You can try downloading the checkpoint file directly using the following commands:
If necessary:
mkdir -p .cache/dpt
pip install gdownDownload pre-trained models
gdown '1Jrh-bRnJEjyMCS7f-WsaFlccfPjJPPHI&confirm=t' -O ./cache/dptHi, JiuTongBro
You can try downloading the checkpoint file directly using the following commands: If necessary:
mkdir -p .cache/dpt pip install gdownDownload pre-trained models
gdown '1Jrh-bRnJEjyMCS7f-WsaFlccfPjJPPHI&confirm=t' -O ./cache/dpt
Thanks, it works, but then the pipeline broke again when reconstructing Gaussian from the mesh. Here is the full log:
Traceback (most recent call last):
File "/home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/lib/python3.8/site-packages/mediapy/__init__.py", line 1739, in write_video
writer.add_image(image)
File "/home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/lib/python3.8/site-packages/mediapy/__init__.py", line 1648, in add_image
if self._proc.stdin.write(data) != len(data):
BrokenPipeError: [Errno 32] Broken pipe
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "train_from_mesh.py", line 261, in <module>
cams, images, camera_extent = render_multiview_images_from_mesh(args.mesh,
File "/home/zhonghongliang/diffusion_pro/gsedit_pro/GaussianEditor/threestudio/utils/render.py", line 150, in render_multiview_images_from_mesh
write_video(save_path, frames, fps=2)
File "/home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/lib/python3.8/site-packages/mediapy/__init__.py", line 1739, in write_video
writer.add_image(image)
File "/home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/lib/python3.8/site-packages/mediapy/__init__.py", line 1610, in __exit__
self.close()
File "/home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/lib/python3.8/site-packages/mediapy/__init__.py", line 1661, in close
raise RuntimeError(f"Error writing '{self.path}': {s}")
RuntimeError: Error writing '/home/zhonghongliang/diffusion_pro/gsedit_pro/GaussianEditor/tmp_add/mesh_rendering.mp4': /home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/bin/ffmpeg: error while loading shared libraries: libopenh264.so.5: cannot open shared object file: No such file or directory
/home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/lib/python3.8/site-packages/timm/models/_factory.py:117: UserWarning: Mapping deprecated model name vit_base_resnet50_384 to current vit_base_r50_s16_384.orig_in21k_ft_in1k.
model = create_fn(
/home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/lib/python3.8/site-packages/torchvision/transforms/functional.py:1603: UserWarning: The default value of the antialias parameter of all the resizing transforms (Resize(), RandomResizedCrop(), etc.) will change from None to True in v0.17, in order to be consistent across the PIL and Tensor backends. To suppress this warning, directly pass antialias=True (recommended, future default), antialias=None (current default, which means False for Tensors and True for PIL), or antialias=False (only works on Tensors - PIL will still use antialiasing). This also applies if you are using the inference transforms from the models weights: update the call to weights.transforms(antialias=True).
warnings.warn(
depth valid percent: tensor(0.9923, device='cuda:0')
Traceback (most recent call last):
File "/home/zhonghongliang/diffusion_pro/gsedit_pro/GaussianEditor/extern/viser/src/viser/infra/_infra.py", line 449, in wrapped
inner()
File "/home/zhonghongliang/diffusion_pro/gsedit_pro/GaussianEditor/extern/viser/src/viser/infra/_infra.py", line 257, in <lambda>
lambda: self._handle_incoming_message(client_id, message)
File "/home/zhonghongliang/diffusion_pro/gsedit_pro/GaussianEditor/extern/viser/src/viser/infra/_infra.py", line 94, in _handle_incoming_message
cb(client_id, message)
File "/home/zhonghongliang/diffusion_pro/gsedit_pro/GaussianEditor/extern/viser/src/viser/_gui_api.py", line 157, in _handle_gui_updates
cb(GuiEvent(client, client_id, handle))
File "webui.py", line 511, in _
self.add(self.camera)
File "/home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "webui.py", line 1436, in add
new_object_gaussian.load_ply(gs_path)
File "/home/zhonghongliang/diffusion_pro/gsedit_pro/GaussianEditor/gaussiansplatting/scene/vanilla_gaussian_model.py", line 293, in load_ply
plydata = PlyData.read(path)
File "/home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/lib/python3.8/site-packages/plyfile.py", line 158, in read
(must_close, stream) = _open_stream(stream, 'read')
File "/home/zhonghongliang/diffusion_pro/gsedit_pro/gsed_env/lib/python3.8/site-packages/plyfile.py", line 1345, in _open_stream
return (True, open(stream, read_or_write[0] + 'b'))
FileNotFoundError: [Errno 2] No such file or directory: '/home/zhonghongliang/diffusion_pro/gsedit_pro/GaussianEditor/tmp_add/inpaint_gs.obj'
I checked the tmp_add folder, there are four items: multiview_pred_images (a folder), inpaint_mesh.obj, inpainted.png, and removed_bg.png. I suppose it is caused by a failure in train_from_mesh.py, which convert the 3D mesh reconstructed by Wonder3D to Gaussians. Could you please check the code and fix this potential bug? Thanks.
This error is related to ffmpeg (used inside mediapy), you can try reinstall ffmepg.