Question: Effectiveness of Learnt Grasping Strategy

[Peter David Fagan]

I am in the process of reimplementing contact_graspnet so I can use it in my own research. One item that crossed my mind is the fact that while a stable grasp may be achievable with the given method, the grasp may be suboptimal for subsequent task completion.

For instance suppose I wish to place an object on a cluttered shelf, how I grasped the object will become quite important for solving this task.

Do you have thoughts and/or opinions on approaches that tackle the above problem? Maybe there is a loss term that can be introduced to help ensure the chosen grasp provides sufficient manoeuvrability.

Or maybe how grasps are being evaluated could be altered?

[danialdunson]

I am not the author of the nvidia neural net so ask the nvidia repo. I cannot recommend anything in good conscious. I suspect the OG author will tell to generate new contact points if your gripper is significantly different than the stock panda gripper he uses to provide contact points. The simulator used to generate successful contact points could be altered to test to successful gripper contact point that are more resilient to maneuverability. IMO I hope you found this helpful Get Outlook for iOS<https://aka.ms/o0ukef>

…

________________________________ From: Peter David Fagan ***@***.***> Sent: Wednesday, May 10, 2023 11:22:57 AM To: NVlabs/contact_graspnet ***@***.***> Cc: Subscribed ***@***.***> Subject: [NVlabs/contact_graspnet] Question: Effectiveness of Grasping Strategy (Issue #33) [External Email] I am in the process of reimplementing contact_graspnet so I can use it in my own research. One item that crossed my mind is the fact that while a stable grasp may be achievable with the given method, the grasp itself may not account for subsequent task completion. For instance suppose I wish to place an object on a cluttered shelf, how I grasped the object will become quite important for solving this task. Do you have thoughts and/or opinions on approach that resolve the above problem? Maybe there is a loss term that can be introduced to help ensure the chosen grasp provides a lot of manoeuvrability. — Reply to this email directly, view it on GitHub<#33>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AQ3IEZZKIYJ2NUZTWJKKMYTXFOXFDANCNFSM6AAAAAAX44QBEU>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[Peter David Fagan]

Thanks for sharing your opinion @danialdunson.

[Martin Sundermeyer]

Hi @peterdavidfagan,

Concerning your idea of retraining: Yes, the Contact-GraspNet framework could be extended to predict a per-grasp-point suitability for different tasks. There are some works on task-specific grasping in literature, e.g. https://arxiv.org/pdf/2011.06431.pdf, and I also saw a paper that I can't disclose here that applied something similar to Contact-GraspNet. However, this way is kind of limited to few tasks and also has no information about the environment, e.g. where the cluttered shelf is located wrt. the robot which can have an impact on suitable grasps.

Of course there are easier ways to solve the specific shelve task without re-training, e.g. by simply filtering out top-grasps. If this is insufficient you can infer your planner with different Contact-GraspNet predicitions and see if the robot can execute the task without collision.

If this is still insufficient, the next step is to determine whether your grasped object won't collide and will be stable after placing assuming the same z-orientation. Full tasks are complex, I recommend this recent paper which uses Contact-GraspNet for re-arranging objects in shelves:
https://arxiv.org/pdf/2304.09302.pdf

[Peter David Fagan]

Thanks @MartinSmeyer for the detailed response to my question.