[SoRo Project] Human touch-gesture recognition using MiRo Social Robot
Project Contributers:
- Prajval Kumar Murali (prajval10@gmail.com)
- Nicolò Bastianell (nicolobastianelli@gmail.com)
- Luca Macchiusi (macchiusiluca@gmail.com)
Objective:
Detecting and classifying touch patterns with MiRo. Doing deeplearning using TensorFlow (keras).
Accomplishments
Collected (recorded from MiRo tactile sensors and saved to a text file) 6 types of gesture data from 9 persons. Each person repeating the gesture 10 times
Gestures:
- Caress body top-bottom
- Caress body bottom-top
- Pat body
- Fixed body
- Pat Head
- Fixed Head
Actions
- Go straight forward
- Go straight backward
- Go in circle
- Stop motion
- Move head left to right
- Stop head and reorient to center
For all gesture recognitions we have very good accuracy (94%) and for some high precision and for some low precision.
Gestures with low precision:
- Pat Body (high false positive rate: we predict pat-body when it is actually not pat-body)
Gestures with High precision (because the sensors are decopled):
- Pat Head
- Fixed Head
Modules or Nodes in the system
Offline part: 2 files (collecting and storing data, training the RNN) Online part: 2 ROS-nodes (data input and machinelearning, miro actions)
Limitations of the system
Offline part:
Improve the precision and recall for all the gestures
Optimize the number of Hidden Neurons
Online part:
After classification(recognizing) of gesture, we must have a voting system. As there is a sliding window, there are jumps in classified outputs, hence we must take the average (do the voting system)
Analyse the temporal performace (latency), ie, the time taken for action to happen after doing the gesture.
How to run:
Configure MiRo with your workstation
To configure MiRo with your workstation follow the setup guide on official website or https://github.com/EmaroLab/MIRO
To Run our application
Offline part (Collecting data and training RNN)
#Collecting data after connecting to MiRo ROS node
python miro_touch_sub.py robot=rob01 name_file="name_of_gesture"
#Training the network
source ~/tensorflow/bin/activate
python DataSet.py #remember to change the path to dataset
python testset.py #to evaluate your model
Online part (Classifying gestures in real-time)
#Data input and miro model node
source ~/tensorflow/bin/activate
python Data_input.py #remember to change the path to model :my_model_new.h5
#miro actions
python miro_action.py #to move miro according to the gestures
Project Video
Can be found here: https://www.youtube.com/watch?v=N7AkDnkxILg