K-Scale Open Source Robotics

K-Scale Manipulation Suite

Gymnasium+MuJoCo Environments


KManipSoloArm environment has one 7dof arm with a 1dof gripper. KManipSoloArmVision has a gripper cam, a head cam, and an overhead cam.	KManipDualArm environment has two 7dof arms with 1dof grippers. KManipDualArmVision has 2 gripper cams, a head cam, and an overhead cam.	KManipTorso environment has a 2dof head, two 6dof arms with 1dof grippers. KManipTorsoVision has 2 gripper cams, a head cam, and an overhead cam.

Setup

clone and install dependencies

git clone https://github.com/kscalelabs/gym-kmanip.git && cd gym-kmanip
conda create -y -n gym-kmanip python=3.10 && conda activate gym-kmanip
pip install -e .

run tests to verify installation

pip install pytest
pytest

for installation on the edge, just install on bare metal

sudo apt-get install libhdf5-dev
git clone https://github.com/kscalelabs/gym-kmanip.git && cd gym-kmanip
pip install -e .

Getting Started

visualize one of the mujoco environments using the viewer

python gym_kmanip/examples/1_view_env.py

alternatively mujoco provides a nice standalone visualizer

download standalone mujoco

tar -xzf ~/Downloads/mujoco-3.1.5-linux-x86_64.tar.gz -C /path/to/mujoco-3.1.5
/path/to/mujoco-3.1.5/bin/simulate gym_kmanip/assets/_env_solo_arm.xml

Recording Data

data can be recorded to a rerun .rrd file for visualization

python gym_kmanip/examples/record_to_rrd.py
rerun gym_kmanip/data/foo/episode_1.rrd

data can be recorded to a h5py .hdf5 file for training models

python gym_kmanip/examples/record_to_hdf5.py

you can also record data to a video file for sharing on social media

python gym_kmanip/examples/record_to_mp4.py

Teleop

teleop can be used to control the robot and optionally record datasets

🤗 K-Scale HuggingFace Datasets

you will need additional dependencies

pip install vuer==0.0.30

start the server on the robot computer

python gym_kmanip/examples/4_record_data_teleop.py

start ngrok on the robot computer.

ngrok http 8012

open the browser app on the vr headset and go to the ngrok url

Real Robot

the real robot works just like any other environment

you will need additional dependencies

pip install opencv-python==4.9.0.80

Help Wanted

✅ solo arm w/ vision

✅ dual arm w/ vision

✅ torso w/ vision

✅ inverse kinematics using mujoco

⬜️ tune and improve ik

⬜️ recording dataset via teleop

⬜️ training policy from dataset

⬜️ evaluating policy on robot

Dependencies

Gymnasium is used for environment
MuJoCo is used for physics simulation
PyTorch is used for model training
Rerun is used for visualization
H5Py is used for logging datasets
HuggingFace is used for dataset & model storage
Vuer teleop only is used for visualization
ngrok teleop only is used for networking

helpful links and repos

Citation

@misc{teleop-2024,
  title={gym-kmanip},
  author={Hugo Ponte},
  year={2024},
  url={https://github.com/kscalelabs/gym-kmanip}
}

hu-po / gym-kmanip