The “Gaze controlled keyboard” is a project where we will control the keyboard through our eyes using Python with Opencv, completely from scratch.The goal of such app is to write without using the hands.
To implement the technology for a better tomorrow is the motivation for this project. I came across Opencv and read about it through various blogs on it. Thus the zeal to work on opencv sparked within me and the idea to create the virtual keyboard which could be controlled by our gaze without using the hands.Such applications are really important for people affected by quadriplegia who completely lost the control of their limbs.
This project is built in 2 main parts.
- Eye detection: detection of the eyes, their movement and most important their blinking.
- Virtual keyboard: a keyboard on the screen where we’re going to select the letters by just using our eyes.
We are taking the frames in real time from the webcam, so to detect the eyes we will use face landmarks detection approach. We can find 68 specific landmarks of the face. To each point there is a specific index assigned. We will use the following landmark points to detect the eye.
- Left eye points: (36, 37, 38, 39, 40, 41)
- Right eye points: (42, 43, 44, 45, 46, 47)
when we detected the eye, we also detected two lines: an horizontal line and a vertical line crossing the eye.
This is how the lines look like when the eye is open.
This when the eye is closed.
We will take horizontal line as the point of reference, and from this we calculate the ratio in comparison with the vertical line. If this ratio is more than 5.7 than the eyes are blinking otherwise they are open.
The idea is to devide the keyboard in two parts. If we look on the left side only the left part of the keyboard will be activated, while if we look on the right side only the letter on the right part of the keyboard will light up.
Hence for that we need to detect the gaze of our eyes wether they are looking in left or right differection.
What at a first glance is really clear, by looking at the image above, it’s that the sclera (white part of the eye) fills the right part of the eye when the eye is looking at the left, the opposite happens when it’s looking to the right and when it’s looking to the center the white is well balanced between left and right.
The idea is to split the eye in two parts and to find out in which of the two parts there is more sclera visible.
If the sclera is more visible on the right part, so the eye is looking at the left (our left) like in this case.Technically to detect the sclera we convert the eye into grayscale, we find a treshold and we count the white pixels.
We divide the white pixels of the left part and those of the right part and we get the gaze ratio. If the gaze ratio is smaller than 1 when looking to the right side and greater than 1.7 when the eyes are looking to the left side.
The idea is to display the Keys on the screen and light them up one at time. Once the key we want to press is lighted up, we simply would need to close our eyes and the key will be pressed.
Using numpy and cv2 we will create a simple keyboard
This is a left side keyboard.
Basically we are lighting up a letter for 10 frames, and after that we light up the next one, so that when we reach the letter we want to press, we simply close our eyes and it gets pressed.
Final result
Thus able to type hello through my eye motion.
- Pyscource blogs by Sergio Canu
