Result camera poses

[AI葵]

In the notebook there is only depth visualization, and in result folder structure I don't see anything related to the camera poses.

How do you output the estimated poses and visualize them like in the README?
Thanks.

[Xuejian Rong]

The camera poses are recorded in the video.dat file, which is in a custom binary format. We plan to clean up and release the format conversion (most likely to the COLMAP format) and visualization code then.

[Huihui1002]

Looking forward to the visualisation code!

[XiaoqiangZhou]

@ionvision Thanks for planning to release the format conversion (especially the COLMAP format). Looking forward to such implementation!

[ZhoutongZhang]

FWIW, you can export the camera poses by dumping the depth frame properties in python. What works for me is adding the following code after this line

        depth_stream_id = 2
        pose_export_path = os.path.join(self.base_dir, 'camera_params')
        os.makedirs(pose_export_path, exist_ok=True)
        n_frames = self.depth_video.numFrames()
        for n in range(n_frames):
            print(f'saving cam params of frame {n}')
            depth_frame = self.depth_video.depthStream(depth_stream_id).frame(n)
            world2cam = depth_frame.extrinsics.worldToCamera()
            vF =depth_frame.intrinsics.vFov
            hF =depth_frame.intrinsics.hFov
            np.savez(os.path.join(pose_export_path, f'frame_{n:06d}.npz'), vF=vF, hF=hF, world2cam=world2cam)

[txytju]

The camera poses are recorded in the video.dat file, which is in a custom binary format. We plan to clean up and release the format conversion (most likely to the COLMAP format) and visualization code then.

@ionvision will you release code recently?

[ZhoutongZhang]

@txytju Here's a script I use to extract camera poses from .dat files:

import os.path as osp
import sys
import numpy as np
from os import makedirs
sys.path.append(osp.abspath(__file__))
sys.path.append(osp.join(osp.dirname(__file__), "lib/build"))
from lib_python import (
    DepthVideo
)
track_path = "" # path to dat file
depth_video = DepthVideo()
depth_video.load(track_path)
n_frames = depth_video.numFrames()
for n in range(n_frames):
    depth_frame = depth_video.depthStream(2).frame(n)
    world2cam = depth_frame.extrinsics.worldToCamera()
    vF =depth_frame.intrinsics.vFov
    hF =depth_frame.intrinsics.hFov
    makedirs(osp.join(track_path, 'camera_poses'), exist_ok=True)
    np.savez(os.path.join(track_path, 'camera_poses', f'frame_{n:06d}.npz'), vF=vF, hF=hF, world2cam=world2cam)

[txytju]

@txytju Here's a script I use to extract camera poses from .dat files:

import os.path as osp
import sys
import numpy as np
from os import makedirs
sys.path.append(osp.abspath(__file__))
sys.path.append(osp.join(osp.dirname(__file__), "lib/build"))
from lib_python import (
    DepthVideo
)
track_path = "" # path to dat file
depth_video = DepthVideo()
depth_video.load(track_path)
n_frames = depth_video.numFrames()
for n in range(n_frames):
    depth_frame = depth_video.depthStream(2).frame(n)
    world2cam = depth_frame.extrinsics.worldToCamera()
    vF =depth_frame.intrinsics.vFov
    hF =depth_frame.intrinsics.hFov
    makedirs(osp.join(track_path, 'camera_poses'), exist_ok=True)
    np.savez(os.path.join(track_path, 'camera_poses', f'frame_{n:06d}.npz'), vF=vF, hF=hF, world2cam=world2cam)

@ztzhang Thanks for your reply and example! It helps a lot!

[txytju]

@txytju Here's a script I use to extract camera poses from .dat files:

import os.path as osp
import sys
import numpy as np
from os import makedirs
sys.path.append(osp.abspath(__file__))
sys.path.append(osp.join(osp.dirname(__file__), "lib/build"))
from lib_python import (
    DepthVideo
)
track_path = "" # path to dat file
depth_video = DepthVideo()
depth_video.load(track_path)
n_frames = depth_video.numFrames()
for n in range(n_frames):
    depth_frame = depth_video.depthStream(2).frame(n)
    world2cam = depth_frame.extrinsics.worldToCamera()
    vF =depth_frame.intrinsics.vFov
    hF =depth_frame.intrinsics.hFov
    makedirs(osp.join(track_path, 'camera_poses'), exist_ok=True)
    np.savez(os.path.join(track_path, 'camera_poses', f'frame_{n:06d}.npz'), vF=vF, hF=hF, world2cam=world2cam)

Have you met error when running depth_video.load(track_path) ? I thinkload() and save() method of class DepthVideo don't match.

[Phong Nguyen Ha]

I am having similar problem with @txytju, I have this error when load the .dat file

root@f2f96bd31e3c:/cvd2# python read_pose.py --track_path ./family_run_output/
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1010 12:44:02.790782   490 DepthVideo.cpp:124] Loading depth video './family_run_output/'...
Traceback (most recent call last):
  File "read_pose.py", line 21, in <module>
    depth_video.load(track_path)
ValueError: basic_string::_M_create

[txytju]

I am having similar problem with @txytju, I have this error when load the .dat file

root@f2f96bd31e3c:/cvd2# python read_pose.py --track_path ./family_run_output/
WARNING: Logging before InitGoogleLogging() is written to STDERR
I1010 12:44:02.790782   490 DepthVideo.cpp:124] Loading depth video './family_run_output/'...
Traceback (most recent call last):
  File "read_pose.py", line 21, in <module>
    depth_video.load(track_path)
ValueError: basic_string::_M_create

There are a few mismatch in load() and save(), I tried to fixed but I could not correct them all.
P.S. I have just done reading your paper "Free-Viewpoint RGB-D Human Performance Capture and Rendering". What a small world!

[jayaramreddy10]

@ztzhang , I am also facing same issue, any suggestions to resolve this?