Course: Master Degree in Computer Science
Department: DISIM - Dipartimento di Ingegneria e Scienze dell'Informazione e Matematica, Università degli studi dell'Aquila
Supervisor: Dr. Giovanni De Gasperis
Author: Rahul Pankajakshan
This thesis aims to achieve a seamless integration of the Large Language Model (LLM) with a behaviour tree control system, specifically designed for a robotic system operating within the CoppeliaSim environment. The primary goal is to assess the system's efficiency in handling various levels of command complexity when executing pick and place tasks.
A demo of the system's working can be found here: here
This repository is organized as follows:
-
evaluation_set: Contains different command sets for level 0, level 1, level 2, and robustness testing.
-
prompts: Contains all system prompts used in the experiment.
-
scene: Contains the CoppeliaSim scene file with the ABB IRB 4600-40 robot model and its environment.
-
src: Contains various source code files categorised as follows:
- actuator: Functions and API calls to handle the inverse-kinematics and robot's movement.
- sensor: Function and API calls to deal with camera and proximity sensor.
- detector: API calls and helper functions for image processing.
- behaviour: Behaviours and its definition. The control system of the robot.
- processor: Functions to derive execution logic based on user input and perception data.
- state: State management for the robot and it's environment
Install all Python (>=3.10) requirements:
pip imnstall -r requirements.txt
Clone the CoppeliaSim remote API:
git clone https://github.com/CoppeliaRobotics/zmqRemoteApi
based on the remote API CoppeliaSim forum post
For MacOS users who are facing connectivity issues with zmqRemoteApi,
set export COPPELIASIM_ROOT_DIR=/Applications/coppeliaSim.app/Contents/Resources environment variable.
based on the CoppeliaSim forum post