jc-bao / roller_grasper_tacto

Roller Grapser v4 environment based on Tacto and Pybullet.

Roller Grasper v4 Tacto Environment

Based on PyBullet and Tacto.

Environment

Reconstruction Environment

System overview

[new depth image, old point clouds, old orientation, esitimated delta orientation] -process pcds-> [new point clouds, new orientation]

Details:

[image data] -pcd_from_depth(depth_image)-> [point cloud (in camera frame)] -pcd_cam2world(pcd, camera_pos)-> [point cloud (in world frame)] -merge_pcds(old_pcds, new_pcd, old_orientation, delta_orientation)-> [point clouds, new_orintation]

Demos

• Single sensor case

• Multi-sensor with ICP refinement

• Pose estimation with ICP+GO under noisy position esitimatioon

• Reconstuct object from narrow observation angle

Physical Environment

Roller Toy Environment	Roller Env (Random explore)	Render gelsight environment

Rolling Action	Pitching Action	Wrist Action
act['roll_l_vel'][0] = 1; act['roll_r_vel'][0] = 1	act['pitch_l_vel'][0] = 1; act['pitch_r_vel'][0] = 1	act['wrist_vel'] = 1

Handcrafted Policy

Rolling Action	Pitching Action	Wrist Action	Compositional Action

Usage

# install a FORTRAN compiler for opto
sudo apt-get install gfortran

# install opto
git clone https://github.com/robertocalandra/opto.git
cd opto
pip install -r requirements.txt
python setup.py install
pip install scipyplot deepdish tacto

TODO List

L1

• Replace Sensor with Roller (2022.07.21)
• Handcrafted policy for reorientation
  • Single rotation handcrafted policy (2022.7.22)
  • attach sensor to fixed joint (2022.7.23)
  • Compositonal handcrafted policy
• Reconstruct the object from depth image
  • With wide camera range (2022.7.27)
    • With true object position
    • With position esitimate from roller angle
  • With small camera range (2022.7.27)
  • With moving camera (2022.7.27)
• Reconstruct of different shapes (e.g. EDGA dataset) (2022.7.28)
• Wrapper up reconstruction function
  • wrap up functions (2022.7.31)
  • test functions and data wrap up (2022.8.1)
  • solve the problem of phi close to each other. (caused by not clean the debug code timely) (2022.8.27)
  • solve the problem of be optimistic of unexplored area. (2022.8.27) (reason: the section far away from the center, has large phi, thus its variance is constrained. solution: make the initial guess more pessimistic)
• Efficient way to detect close loop
• Using ICP to matching points

L2

• Gym Wrapper (2022.07.21)
• Roller Control Suite
  • Add roller model to simulation (2022.7.22)
  • Add sensor to simulation (2022.7.22)
• off screen rendering (2022.7.27)

L3

• Add blender to simulation
• Reconstruct the object from tactile sensor

Hardware Deployment

1. get getsight depth image
   1. Requirement: from 0 -> max_depth
2. generate esistimated angle
3. run the test code in test set

Possible Gaps

• The Gel is rounded but our sensor get a plane
• The Gel has less depth

Constrain with depth and size	Constrain with Gel's shape

Details

• roller size: R=0.02m H=0.05m
• this package use scipy, in which quaternion is [x,y,z,w], in pybullet, quaternion is also [x,y,z,w]. but in mujoco, quaternion is [w,x,y,z]