How to get camera matrix in Mujoco
shinshiner opened this issue · comments
Hi,
I wanted to convert the depth image to 3D point cloud in mujoco. I was trying to find camera intrinsic matrix for the camera body that I attached to the robot, but I couldn't find any information regarding this. Is there any way to attach this information ?
Maybe there is no camera matrix in Mujoco ? If so, how to implement such conversion?
I haven't found camera matrix in mujoco, however, I could construct it.
I assumed that the principal point is not shifted and pixels are of square form. You need to know frame size (width, height) and focus distance (in pixels), which can be computed as following:
fovy = sim.model.cam_fovy[cam_id]
f = 0.5 * height / math.tan(fovy * math.pi / 360)
Then you get camera matrix as np.array(((f, 0, width / 2), (0, f, height / 2), (0, 0, 1)))
It worked for me.
For more info about camera matrix look here, for more detailed understanding you may have a look on this book
It works perfectly! Thank you very much @InverseNova : )
@InverseNova thanks for the help here.
I am stuck at a similar situation - I wish to model a camera in mujoco and wish to specify the internal parameters. I can only find the way to specify fovy , however I wish to specify it in detail (fx, fy,cx,cy,s). Is this possible?
@beingkartik According to the MuJoCo API documentation, the horizontal field of view (fovx) is calculated based on the horizontal field of view and the windows size (width, height). If you use, a square image they be equal.
Hi @shinshiner and @InverseNova
I'm also working on a 3d reconstrction project in MuJoCo. But I had a problem about how to extract the real depth value from the depth image rendered by sim.render(). So basically, the depth image we get from the simulation is normalized to [0,1]. Could you guys let me know how to convert it back to the real depth value?
I also submitted an issue #520 with description of my problem in detail.
Thanks in advance!
Anwser to the depth conversion problem.
I tried using this camera matrix for projecting 3d coordinates into 2d pixel coordinates. But I found the horizontal pixel dimension to be flipped. The problem seems to be with the camera matrix, where the first element is -f, not f. Indeed, if you check deepmind's camera matrix, they also negate the first f in the diagonal.
So the camera matrix should be:
K = np.array(((-f, 0, width / 2), (0, f, height / 2), (0, 0, 1)))