cog-isa / rozumarm-vima

Code for the "Fine-tuning Multimodal Transformer Models for Generating Actions in Virtual and Real Environments" paper

How to install:

1. Clone to the project root or install the following repos: cv-utils, cam-utils, cv-repo, ultralytics, and arm-utils.
2. Install this package and dependencies from requirements.txt.
3. Add to PYTHONPATH the paths to rozumarm_vima_cv, utils, and camera_utils directories that you cloned.
4. Download all missing VIMA checkpoints from https://github.com/vimalabs/VIMA

How to run:

• to start (cube detector -> sim -> oracle -> arm) pipeline, run scripts/run_aruco2sim_loop.py
• to start (cube detector -> sim -> ML model -> arm) pipeline, run scripts/run_model_loop.py
• to start (cam image -> ML model -> arm) pipeline, set USE_OBS_FROM_SIM=False in scripts/run_model_loop.py and run it

Links to datasets:

• Sweep-Plan
• Sweep-Seg