Based on PyBullet and Tacto.
[new depth image, old point clouds, old orientation, esitimated delta orientation] -process pcds
-> [new point clouds, new orientation]
Details:
[image data] -pcd_from_depth(depth_image)
-> [point cloud (in camera frame)] -pcd_cam2world(pcd, camera_pos)
-> [point cloud (in world frame)] -merge_pcds(old_pcds, new_pcd, old_orientation, delta_orientation)
-> [point clouds, new_orintation]
- Single sensor case
- Multi-sensor with ICP refinement
- Pose estimation with ICP+GO under noisy position esitimatioon
- Reconstuct object from narrow observation angle
Roller Toy Environment | Roller Env (Random explore) | Render gelsight environment |
---|---|---|
Rolling Action | Pitching Action | Wrist Action |
---|---|---|
act['roll_l_vel'][0] = 1; act['roll_r_vel'][0] = 1 |
act['pitch_l_vel'][0] = 1; act['pitch_r_vel'][0] = 1 |
act['wrist_vel'] = 1 |
Rolling Action | Pitching Action | Wrist Action | Compositional Action |
---|---|---|---|
![]() |
# install a FORTRAN compiler for opto
sudo apt-get install gfortran
# install opto
git clone https://github.com/robertocalandra/opto.git
cd opto
pip install -r requirements.txt
python setup.py install
pip install scipyplot deepdish tacto
- Replace Sensor with Roller (2022.07.21)
- Handcrafted policy for reorientation
- Single rotation handcrafted policy (2022.7.22)
- attach sensor to fixed joint (2022.7.23)
- Compositonal handcrafted policy
- Reconstruct the object from depth image
- With wide camera range (2022.7.27)
- With true object position
- With position esitimate from roller angle
- With small camera range (2022.7.27)
- With moving camera (2022.7.27)
- With wide camera range (2022.7.27)
- Reconstruct of different shapes (e.g. EDGA dataset) (2022.7.28)
- Wrapper up reconstruction function
- wrap up functions (2022.7.31)
- test functions and data wrap up (2022.8.1)
- solve the problem of phi close to each other. (caused by not clean the debug code timely) (2022.8.27)
- solve the problem of be optimistic of unexplored area. (2022.8.27) (reason: the section far away from the center, has large phi, thus its variance is constrained. solution: make the initial guess more pessimistic)
- Efficient way to detect close loop
- Using ICP to matching points
- Gym Wrapper (2022.07.21)
- Roller Control Suite
- Add roller model to simulation (2022.7.22)
- Add sensor to simulation (2022.7.22)
- off screen rendering (2022.7.27)
- Add blender to simulation
- Reconstruct the object from tactile sensor
- get getsight depth image
- Requirement: from 0 -> max_depth
- generate esistimated angle
- run the test code in test set
Possible Gaps
- The Gel is rounded but our sensor get a plane
- The Gel has less depth
Constrain with depth and size | Constrain with Gel's shape |
---|---|
- roller size: R=0.02m H=0.05m
- this package use scipy, in which quaternion is [x,y,z,w], in pybullet, quaternion is also [x,y,z,w]. but in mujoco, quaternion is [w,x,y,z]