[ROS2] Face Detection Accuracy Check
amalnanavati opened this issue · comments
In MoveToMouth (as of #42 ), the robot arm often gets closer than 5cm to the detected mouth, and also oftentimes is in front of the nose rather than the mouth. The face detection img seems to be correct. Further, it does move 5cm away from the mouth of the head mesh in the MoveIt planning scene (it determines the planning scene head location based on the stomion detected from face detection). This makes me think that the stomion location detected in FaceDetection may be wrong (although it is possible that maybe the issue lies in the MoveToMouth behaviors).
This issue is focused on checking, in a very detailed manner, how well the detected mouth location aligns with the real mouth location. One way to do this can be by using depth_image_proc to visualize the depth image as a pointcloud in RVIZ, and seeing how the actual mouth center aligns with where the robot moved the mouth center in the MoveIt planning scene.
(Note that if you use depth image proc, you have to comment out lines 47-49 and 72-74 for the launchfile to work)
Incidentally, using depth_image_proc to check the alignment between the real head and the detected head is a great stepping stone into addressing the first todo of #65 , where we check whether the depth image of the wheelchair aligns with the object in the planning scene.
Note that if there is a camera calibration issue, that could lso be cauing this behavior.
Also, if the aligned depth image has the noisy translations we were seeing before, and the Face Detection node happens to calculate depth on that frame, that may also have these issues.
Notes re. Face Detection while testing #72 :
- We have fixed the aligned camera issue, so that is no longer a possible cause for a bug.
- If the fork is overlapping the mouth, the detected mouth depth might be the fork. This is bad for a couple of reasons:
- The robot won't move to the user's mouth.
- Because we move the head collision object to the location detected by face detection, we move the head collision object to overlap with the fork, which means that no subsequent plans will succeed.
- To address the above issue, I could potentially change the "check face function" to not only require that the face is <= 1.5m away from the camera, but also >= 0.4m from the camera. An alternate approach is to get the mouth depth using other features of the face, like eyes and nose. What do you think @taylorkf ?
- Sometimes, with the mannequin head, face detection does wrongly detect the nose as the mouth. I'm not sure if that is only when the fork overlaps the mouth or at other times as well. I'm also not sure if this is only with the mannequin head. We should look more into this.
I think it might be a camera calibration issue, but it is worth looking more into this.
@taylorkf Two updates:
- Ethan and I did some testing and think it is very likely that the "moving to nose" issue is due to camera calibration, because the whole nano enclosure rotates a bit as the robot moves. We will have Haya modify the enclosure to address that.
- The issue of the robot detecting the fork depth as the mouth depth is still an issue. Ethan mentioned that in past years, using the depth of the stomion point was unreliable for them, and they instead did the following:
- Get the pixel at the average position of the two eyes (e.g., the bridge of the nose).
- Take a 9x9 px square around that point and average the depth of those points.
As discussed in #102
Throwing in here that we can use pyrealsense2 to do intrinsics calculations.
Closed with #130