Real Time Driver State Detection
Real Time webcam based driver attention state detection using Python, OpenCV and Dlib.
Note: This work is partially based on this paper for the scores and methods used.
How It Works?
This script searches for the driver face, then use the dlib library to predict 68 facial keypoints. The enumeration and location of all the face keypoints/landmarks can be seen here.
With those keypoints, the following scores are computed:
- EAR: Eye Aspect Ratio, it's the normalized average eyes aperture, and it's used to see how much the eyes are opened or closed
- Gaze Score: L2 Norm (Euclidean distance) between the center of the eye and the pupil, it's used to see if the driver is looking away or not
- Head Pose: Roll, Pitch and Yaw of the head of the driver. The angles are used to see if the driver is not looking straight ahead or doesn't have a straight head pose (is probably unconscious)
- PERCLOS: PERcentage of CLOSure eye time, used to see how much time the eyes are closed in a minute. A threshold of 0.2 is used in this case (20% of a minute) and the EAR score is used to estimate when the eyes are closed.
The driver states can be classified as:
- Normal: no messages are printed
- Tired: when the PERCLOS score is > 0.2, a warning message is printed on screen
- Asleep: when the eyes are closed (EAR < closure_threshold) for a certain amount of time, a warning message is printed on screen
- Looking Away: when the gaze score is higher than a certain threshold for a certain amount of time, a warning message is printed on screen
- Distracted: when the head pose score is higher than a certain threshold for a certain amount of time, a warning message is printed on screen
Demo
demo.mp4
The Scores Explained
EAR
Eye Aspect Ratio is a normalized score that is useful to understand the rate of aperture of the eyes.
Using the dlib keypoints for each eye (six for each), the eye lenght and width are estimated and using this data the EAR score is computed as explained in the image below:
NOTE: the average of the two eyes EAR score is computed
Gaze Score Estimation
The gaze score gives information about how much the driver is looking away without turning his head.
To understand this, the distance between the eye center and the position of the pupil is computed. The result is then normalized by the eye width that can be different depending on the driver physionomy and distance from the camera.
The below image explains graphically how the Gaze Score for a single eye is computed:
NOTE: the average of the two eyes Gaze Score is computed
Eye processing for gaze score:
Updated version of the processing for computing the gaze score:
For the first version, an adaptive thresholding was used to aid the Hough transform for detecting the iris position. In the updated version, only a Hough transform is used and the ROI of the eyes has been reduced in size.
Demo
Gaze.Score.demo.mp4
The white line is the Euclidean (L2) distance between the black dot (center of the ROI of the eyes) and the white dot (estimated center of the iris/pupil).
Head Pose Estimation
For the head pose estimation, a standard 3d head model in world coordinates was considered, in combination of the respective dlib keypoints in the image plane. In this way, using the solvePnP function of OpenCV, estimating the rotation and translation vector of the head in respect to the camera is possible. Then the 3 Euler angles are computed. The partial snippets of code used for this task can be found in this article
Installation
This projects runs on Python 3.9 with the following libraries:
- numpy
- OpenCV (opencv-python)
- Dlib and cmake
The Dlib predictor for face keypoints is already included in the "predictor" folder
IMPORTANT: Dlib installation
Dlib is a library that needs a C/C++ compiler installed and also the Cmake library. Please follow this guide to install dlib propely on your machine.
If you have already all the prerequisites in your machine to install dlib and cmake you can use the requirements.txt file provided in the repository or you can execute the following pip commands on terminal:
pip install numpy
pip install opencv-python
pip install cmake
pip install dlib --verbose