Gesture Controlled Virtual Mouse makes human computer interaction simple by making use of Hand Gestures and Voice Commands. The computer requires almost no direct contact. All i/o operations can be virtually controlled by using static and dynamic hand gestures along with a voice assistant. This project makes use of the state-of-art Machine Learning and Computer Vision algorithms to recognize hand gestures and voice commands, which works smoothly without any additional hardware requirements. It leverages models such as CNN implemented by MediaPipe running on top of pybind11. It consists of two modules: One which works direct on hands by making use of MediaPipe Hand detection, and other which makes use of Gloves of any uniform color. Currently it works on Windows platform.
Video Demonstration: link
Note: Use Python version: 3.8.5
Features
click on dropdown to know more
Gesture Recognition:
Move Cursor
![Move Cursor](https://github.com/xenon-19/Gesture_Controller/raw/e20edfb1f368ffa600d96bd91031942ec97cb2ab/demo_media/move%20mouse.gif?raw=true)
Cursor is assigned to the midpoint of index and middle fingertips. This gesture moves the cursor to the desired location. Speed of the cursor movement is proportional to the speed of hand.
Scrolling
![Scrolling](https://github.com/xenon-19/Gesture_Controller/raw/9be82cfc75aa4c04fff0e12dd4de853f9d83a101/demo_media/Scrolling.gif?raw=true)
Dynamic Gestures for horizontal and vertical scroll. The speed of scroll is proportional to the distance moved by pinch gesture from start point. Vertical and Horizontal scrolls are controlled by vertical and horizontal pinch movements respectively.
Drag and Drop
![Drag and Drop](https://github.com/xenon-19/Gesture_Controller/raw/9be82cfc75aa4c04fff0e12dd4de853f9d83a101/demo_media/drag%20and%20drop.gif?raw=true)
Gesture for drag and drop functionality. Can be used to move/tranfer files from one directory to other.
Volume Control
![Volume Control](https://github.com/xenon-19/Gesture_Controller/raw/9be82cfc75aa4c04fff0e12dd4de853f9d83a101/demo_media/Volume%20control.gif?raw=true)
Dynamic Gestures for Volume control. The rate of increase/decrease of volume is proportional to the distance moved by pinch gesture from start point.
Brightness Control
![Brightness Control](https://github.com/xenon-19/Gesture_Controller/raw/9be82cfc75aa4c04fff0e12dd4de853f9d83a101/demo_media/Brigntness%20Control.gif?raw=true)
Dynamic Gestures for Brightness control. The rate of increase/decrease of brightness is proportional to the distance moved by pinch gesture from start point.
Voice Assistant ( Proton ):
Launch / Stop Gesture Recognition
![launch stop gesture recognition](https://github.com/xenon-19/Gesture_Controller/raw/4041eedc2f75fa2923902000b606a05a677629e8/demo_media/voice%20commands/proton%20launch%20stop%20gest.png?raw=true)
-
Proton Launch Gesture Recognition
Turns on webcam for hand gesture recognition. -
Proton Stop Gesture Recognition
Turns off webcam and stops gesture recognition. (Termination of Gesture controller can also be done via pressingEnter
key in webcam window)
Google Search
![proton search github](https://github.com/xenon-19/Gesture_Controller/raw/4041eedc2f75fa2923902000b606a05a677629e8/demo_media/voice%20commands/proton%20search.png?raw=true)
-
Proton search {text_you_wish_to_search}
Opens a new tab on Chrome Browser if it is running, else opens a new window. Searches the given text on Google.
Find a Location on Google Maps
![proton find location](https://github.com/xenon-19/Gesture_Controller/raw/4041eedc2f75fa2923902000b606a05a677629e8/demo_media/voice%20commands/proton%20find%20location.png?raw=true)
-
Proton Find a Location
Will ask the user for the location to be searched. -
{Location_you_wish_to_find}
Will find the required location on Google Maps in a new Chrome tab.
File Navigation
![proton list files](https://github.com/xenon-19/Gesture_Controller/raw/4041eedc2f75fa2923902000b606a05a677629e8/demo_media/voice%20commands/proton%20list%20files.png?raw=true)
![proton open](https://github.com/xenon-19/Gesture_Controller/raw/4041eedc2f75fa2923902000b606a05a677629e8/demo_media/voice%20commands/proton%20open.png?raw=true)
![proton go back](https://github.com/xenon-19/Gesture_Controller/raw/4041eedc2f75fa2923902000b606a05a677629e8/demo_media/voice%20commands/proton%20go%20back.png?raw=true)
-
Proton list files
/Proton list
Will list the files and respective file_numbers in your Current Directory (by default C:) -
Proton open {file_number}
Opens the file / directory corresponding to specified file_number. -
Proton go back
/Proton back
Changes the Current Directory to Parent Directory and lists the files.
Current Date and Time
![proton date / time](https://github.com/xenon-19/Gesture_Controller/raw/d49c868acc41ac6c89489bfd80e5e5015a8cb571/demo_media/voice%20commands/proton%20date%20time.png?raw=true)
-
Proton what is today's date
/Proton date
Proton what is the time
/Proton time
Returns the current date and time.
Copy and Paste
![proton copy](https://github.com/xenon-19/Gesture_Controller/raw/4041eedc2f75fa2923902000b606a05a677629e8/demo_media/voice%20commands/proton%20copy.png?raw=true)
![proton paste](https://github.com/xenon-19/Gesture_Controller/raw/4041eedc2f75fa2923902000b606a05a677629e8/demo_media/voice%20commands/proton%20paste.png?raw=true)
-
Proton Copy
Copies the selected text to clipboard.
-
Proton Paste
Pastes the copied text.
Sleep / Wake up Proton
![proton sleep / wake up](https://github.com/xenon-19/Gesture_Controller/raw/4041eedc2f75fa2923902000b606a05a677629e8/demo_media/voice%20commands/proton%20bye%20wake%20up.png?raw=true)
-
Sleep
Proton bye
Pauses voice command execution till the assistant is woken up. -
Wake up
Proton wake up
Resumes voice command execution.
Getting Started
Pre-requisites
Python: (3.6 - 3.8.5)
Anaconda Distribution: To download click here.
Procedure
git clone https://github.com/xenon-19/Gesture-Controlled-Virtual-Mouse.git
For detailed information about cloning visit here.
Step 1:
conda create --name gest python=3.8.5
Step 2:
conda activate gest
Step 3:
pip install -r requirements.txt
Step 4:
conda install PyAudio
conda install pywin32
Step 5:
cd to the GitHub Repo till src folder
Command may look like: cd C:\Users\.....\Gesture-Controlled-Virtual-Mouse\src
Step 6:
For running Voice Assistant:
python Proton.py
( You can enable Gesture Recognition by using the command "Proton Launch Gesture Recognition" )
Or to run only Gesture Recognition without the voice assisstant:
Uncomment last 2 lines of Code in the file Gesture_Controller.py
python Gesture_Controller.py
Collaborators
Viral Doshi | GitHub | |||
Nishiket Bidawat | Github | |||
Ankit Sharma | GitHub | |||
Parth Sakariya | Github |