Stretch Camera Extrinsics

[Matthew Trang]

Are the camera extrinsics for the Stretch with respect to the body written anywhere? Also, is the translation and rotation given as observations done with respect to the body frame, or the camera frame?

[Chris Paxton]

camera_pose is in world frame in the observations object; at runtime we're publishing the camera frame from the robot, as per its kinematic model

It's not anywhere in particular, but if you look in the URDF, it's going to be the camera_rgb_optical_frame (presuming you're familiar with ROS/URDF files?)

For what it's worth we're extending the deadline by over a week to account for our low number of evaluations so far

[Matthew Trang]

Thanks, that helps. Specifically for the obs.gps and obs.compass, are those with respect to the robot frame as well then?

I'm somewhat familiar with URDF files, but just by looking at the hab_stretch.urdf file, I see the following reference for camera_color_optical_frame (camera_rgb_optical_frame doesn't seem to exist)

 <!-- camera color joints and links -->
  <joint name="camera_color_joint" type="fixed">
    <origin rpy="0 0 0" xyz="0 0.015 0"/>
    <parent link="camera_link"/>
    <child link="camera_color_frame"/>
  </joint>
  <link name="camera_color_frame"/>
  <joint name="camera_color_optical_joint" type="fixed">
    <origin rpy="-1.5707963267948966 0 -1.5707963267948966" xyz="0 0 0"/>
    <parent link="camera_color_frame"/>
    <child link="camera_color_optical_frame"/>
  </joint>
  <link name="camera_color_optical_frame"/>

However, this doesn't really make sense to me, as the camera rpy doesn't reflect the physical tilt and there's no translation here. How does this work?

Also, extending the deadline would help a lot, I haven't been able to actually submit, but with some more time I hope to.

[Chris Paxton]

Yes it's extended to 10/20, we will send an announcement soon. And you're right, it's camera_color_optical_frame
The URDF file just captures one link in a kinematic chain. We actually provide functionality to compute pose given a robot configuration here: https://github.com/facebookresearch/home-robot/blob/main/src/home_robot/home_robot/motion/stretch.py#L588

Another tool you may want to look at is urdfpy, which lets you parse and visualize an urdf: https://github.com/mmatl/urdfpy

The camera on Stretch is at the end of a kinematic chain (a bunch of links, rigid building blocks of a robot) which can actually be in different poses. In particular the relevant ones here are:
base_x, base_y, and base_theta
head pan and tilt

[Matthew Trang]

Just FYI, it seems like the stretch.py tool is broken due to circular imports of Robot right now

 python src/home_robot/home_robot/motion/stretch.py 
pybullet build time: May 20 2022 19:45:31
Traceback (most recent call last):
  File "/home/ma32004/home-robot/src/home_robot/home_robot/motion/stretch.py", line 12, in <module>
    import home_robot.utils.bullet as hrb
  File "/home/ma32004/home-robot/src/home_robot/home_robot/utils/bullet.py", line 15, in <module>
    from home_robot.motion.ik_solver_base import IKSolverBase
  File "/home/ma32004/home-robot/src/home_robot/home_robot/motion/__init__.py", line 5, in <module>
    from .base import Planner, PlanResult
  File "/home/ma32004/home-robot/src/home_robot/home_robot/motion/base.py", line 8, in <module>
    from home_robot.motion.robot import Robot
  File "/home/ma32004/home-robot/src/home_robot/home_robot/motion/robot.py", line 77, in <module>
    class Robot(abc.ABC):
  File "/home/ma32004/home-robot/src/home_robot/home_robot/motion/robot.py", line 88, in Robot
    def get_backend(self) -> hrb.PbClient:
AttributeError: partially initialized module 'home_robot.utils.bullet' has no attribute 'PbClient' (most likely due to a circular import)

I was able to get it to run just by commenting out the circular import from base.py and motion/init.py, but I'm sure there's a better fix.

[Chris Paxton]

Thanks for finding this, will fix it asap. Glad it does not block you.

[Chris Paxton]

This will be fixed in #435