This repository provides a python wrapper to the open-source pupillometry system PupilEXT (Link). The binding module is created using Pybind11. PyPupilEXT allows to measure the pupil diameter from images that were captured by stereo- and mono camera systems. The pupil diameter can be detected using one of the state-of-the-art open-source algorithms, Starburst [1], Swirski2D [2], ExCuSe [3], ElSe [4], PuRe [5], and PuReST [6].
The PyPupilEXT package is authored by Moritz Lode and Babak Zandi from the Technical University of Darmstadt.
More information about our open-source pupillometry project can be found here: https://github.com/openPupil/Open-PupilEXT
Caution: This project and its repository is under development and currently in an experimental stage.
Contact: Babak Zandi, Mail: zandi@lichttechnik.tu-darmstadt.de
Internal ToDo list
\1) Automate the build process using GitHub Actions to provide universal wheel files. Comment: GitHub Actions currently does not work as it appears an error when using vcpkg. Need to be checked later.
\2) Integrate a test procedure with several examples, including stereo-matching.
\3) Upload wheel files for macOS and Windows 10 to pip.
\4) Specify which python versions work with this package. Currently, only python 3.7 is tested.
Coming soon. Currently, you need to build your own wheel files to use PyPupilEXT (see next steps)
Step 1: Clone this repository with the included submodule.
git clone --recurse-submodules https://github.com/openPupil/PyPupilEXTThe --recurse-submodules option is important, as vcpkg is a submodule. Without this option, the 3rdparty folder will not contain vcpkg packet manager.
Step 2: Create a new python 3.7. environment on your machine.
conda create -n pypupilenv python=3.9
conda activate pypupilenv
conda install numpy
conda install matplotlib
conda install pandas
python -m pip install --upgrade pip
python -m pip install opencv-pythonStep 3: Create a wheel file from the source code to pip install the library.
In PyPupilEXT, the pupil detection algorithms are included as C++ files. Therefore, it is necessary to load different C++ libraries to build the pupil detection algorithms. This process is fully automated with vcpkg. You only need to open the PyPupilEXT folder and call a one-liner. Note that the build process will take a while because the C++ libs need to be downloaded and compiled.
cd PyPupilEXT
conda activate pypupilenv
python setup.py sdist bdist_wheelThen, in the folder PyPupilEXT/dist, there should be a new *.whl file, which is on a mac machine, for example PyPupilEXT-0.0.1-cp37-cp37m-macosx_10_15_x86_64.whl. This file can be used to install the python package. For this, locate the dist folder and pip install it. If you experience any platform compability error, then you need to rename the file PyPupilEXT-0.0.1-cp37-cp37m-macosx_10_15_x86_64.whlto PyPupilEXT-0.0.1-cp37-none-any.whl
cd dist
pip install PyPupilEXT-0.0.1-cp37-cp37m-macosx_10_15_x86_64.whlNow you can use the PyPupilEXT package in Python and proceed with the examples provided section two.
If the build process fails, it may be due to the setup.py file. In such a case, it could be useful to compile the C++ files manually. Firstly, you need to find the path to the C++ NumPy header, which is necessary during the compilation. For this, type the following in your shell
conda activate pypupilenv
python
import numpy as np
print(np.get_include())
exit()Copy the printed path, which is in my case /Users/papillonmac/miniconda3/envs/pypupilENV/lib/python3.7/site-packages/numpy/core/include. This path needs to be included in the PyPupilEXT/CMakeLists.txt file. In the CMakeLists.txt file, you need to find the following line:
set(Python_NumPy_INCLUDE_DIR "/Users/papillonmac/miniconda3/envs/TestENV/lib/python3.9/site-packages/numpy/core/include")Update this path with your own path to the C++ NumPy header. Now you are ready to build the C++ files. For this, open your terminal, locate the PyPupilEXT folder and make sure you are in the right conda env as specified previously.
cd PyPupilEXT
cd build
cmake .. -G "Unix Makefiles" -DCMAKE_BUILD_TYPE=Release -DVCPKG_TARGET_TRIPLET=x64-osx -DCMAKE_TOOLCHAIN_FILE=3rdparty/vcpkg/scripts/buildsystems/vcpkg.cmake
cmake --build . --config ReleaseThis process will again take a while as the libs need to be downloaded and compiled. However, if you are on a Windows machine, use the following code instead:
cd PyPupilEXT
cd build
cmake .. -DCMAKE_BUILD_TYPE=Release -DVCPKG_TARGET_TRIPLET=x64-windows-static-md -DCMAKE_TOOLCHAIN_FILE=3rdparty/vcpkg/scripts/buildsystems/vcpkg.cmake
cmake --build . --config ReleaseThen, go to the PyPupilEXT/build folder and copy the _pypupil.cpython-37m-darwin.so file (it may be labeled different on your system) into the PyPupilEXT/pypupilext folder. Next, you can install the package into your python env using the following command
python -m pip install . -vPyPupilEXT contains the pupil detection algorithms Starburst [1], Swirski2D [2], ExCuSe [3], ElSe [4], PuRe [5], and PuReST [6]. Each algorithm is implemented using the PupilDetectionMethod interface, exposing the function run and runWithConfidence for pupil detection on an image. The method runWithConfidence additionally applies an outline confidence measure in the range of [0, 1] on the pupil detection, accessible by the field pupil.outline_confidence.
The algorithms can be instantiated by creating objects of the classes ElSe, ExCuSe, PuRe, PuReST, Starburst, and Swirski2D. Further image undistortion and stereo triangulation procedures are available in which a camera calibration from the PupilEXT software platform is loaded and used to calculate undistorted images, or in the stereo camera case, a stereo triangulation of the physical pupil size.
Example 1: Loading an image file using OpenCV and applying a pupil detection algorithm to it:
import pypupilext as pp
import cv2
algorithm = pp.PuRe()
# Images are read in grayscale, as the pupil detection algorithms usually operate on grayscale images
image = cv2.imread('tests/1.bmp', cv2.IMREAD_GRAYSCALE)
# Images can be undistorted by loading a calibration file from the PupilEXT software either creating a SingleCalibration or StereoCalibration calibration object
calibration = pp.SingleCalibration('single_calibration.xml')
# Undistort the image using the calibration object
image = calibration.undistortImage(image)
# Result of the pupil detection is a Pupil object
# pupil = algorithm.run(image)
# Run the algorithm with an outline confidence measure, which is independend of the used algorithm. A value of 1 indicates a perfect ellipse fit around the pupil's contour fit.
pupil = algorithm.runWithConfidence(image)
print(pupil.diameter())
print(pupil.outline_confidence)
# Some algorithms also deliver their own confidence measure
# print(pupil.confidence)Example 2: Run a pupil detection algorithm on an eye image and fit an ellipse around the pupil contour.
For this example, you can use the provided test image in this repository in tests/1.bmp. Note that if you use your own images, you need to adjust the parameters of the pupil detection algorithm to match your image resolution. For this, adjust the pure.maxPupilDiameterMM and pure.pure.minPupilDiameterMM appropriately. The pupil.outline_confidence value can be used as an estimate of how well the pupil fit worked. A value of 1 indicates a perfect ellipse fit around the pupil's contour. It is also possible to adjust the parameters of pupil detection algorithm in the GUI of PupilEXT (Link: https://github.com/openPupil/Open-PupilEXT) and then transfer the values into your python script.
import pypupilext as pp
import cv2
import pandas as pd
import time
import matplotlib.pyplot as plt
def ResizeWithAspectRatio(image, width=None, height=None, inter=cv2.INTER_AREA):
dim = None
(h, w) = image.shape[:2]
if width is None and height is None:
return image
if width is None:
r = height / float(h)
dim = (int(w * r), height)
else:
r = width / float(w)
dim = (width, int(h * r))
return cv2.resize(image, dim, interpolation=inter)
img = cv2.imread("tests/1.bmp", cv2.IMREAD_GRAYSCALE)
pupilClass = pp.Pupil()
assert pupilClass.confidence == -1
pure = pp.PuRe()
pure.maxPupilDiameterMM = 7
im_reized = img
pupil = pure.runWithConfidence(im_reized)
data = pd.DataFrame([{'Outline Conf': pupil.outline_confidence, 'PupilDiameter': pupil.diameter()}])
print(data)
img = cv2.cvtColor(img, cv2.COLOR_GRAY2BGR)
img_plot = cv2.ellipse(img,
(int(pupil.center[0]), int(pupil.center[1])),
(int(pupil.minorAxis()/2), int(pupil.majorAxis()/2)), pupil.angle,
0, 360, (0, 0, 255), 1)
resize = ResizeWithAspectRatio(img_plot, width=800)
fig = plt.figure(figsize=(20, 8))
ax1 = plt.subplot(1, 2, 1)
im1 = ax1.imshow(cv2.cvtColor(resize, cv2.COLOR_BGR2RGB))
fig.tight_layout()
plt.show()
# If you want to show the image using an opencv window instead of matplotlib
#cv2.imshow("window", resize)
#cv2.waitKey(0)
#cv2.destroyAllWindows()
#cv2.waitKey(1)Class representing a pupil detection algorithm. Each of the available algorithms, implements the following functions.
Applies pupil detection on a given image and returns a pupil result.
Parameters:
| Name | Description |
|---|---|
| image | Numpy.array Array representing an image from i.e. cv2.imread. |
Pupil object representing the pupil detection
Applies pupil detection on a given image and returns a pupil result. The pupil result contains an additional outline_confidence value, representing the goodness of the detection in the range [0, 1].
Parameters:
| Name | Description |
|---|---|
| image | Numpy.array Array representing an image from i.e. cv2.imread. |
Pupil object representing the pupil detection
Class representing a pupil detection result. The pupil contains all ellipse parameter with which the pupil contour ellipse can be reconstructed.
| Name | Description |
|---|---|
| center | int Diameter of the ellipse defined as its major axis. |
| undistortedDiameter | int Diameter of the ellipse defined as its major axis. |
| physicalDiameter | int Diameter of the ellipse defined as its major axis. |
| eyelid | int Diameter of the ellipse defined as its major axis. |
| size | int Diameter of the ellipse defined as its major axis. |
| angle | int Diameter of the ellipse defined as its major axis. |
| confidence | int Diameter of the ellipse defined as its major axis. |
| outline_confidence | int Diameter of the ellipse defined as its major axis. |
| Name | Returns, Description |
|---|---|
| rectPoints() | list Returns list of point tuples, representing the corner points of the ellipse rectangle (bounding box). |
| shift(point) | None Shifts the pupil center according to the given point. |
| valid(confidence_threshold) | bool Checks wherever the pupil is valid based on a given confidence threshold, and size of the ellipse > 0. |
| resize(x, y) | int Scale the pupil ellipse by the given factors. |
| width() | int Width of the ellipse defined as the width of the ellipse rectangle. |
| height() | int Height of the ellipse defined as the width of the ellipse rectangle. |
| diameter() | int Diameter of the ellipse defined as its major axis. |
| majorAxis() | int Large value of the two fields width and height. |
| majorAxis() | int Smaller value of the two fields width and height. |
| circumference() | float Circumference of the ellipse. |
| clear() | None Resets the pupil ellipse parameter to -1 (invalid). A reset pupil always returns false on valid(). |
Class representing a calibration for a single camera. Images undistorted by this object must be recorded by the same camera for which the calibration was loaded.
Loads a given calibration file for a single camera. The calibration file should be in OpenCV's file storage format and be created using the PupilEXT software for correct formats.
Parameters:
| Name | Description |
|---|---|
| calibration_file | Str Path to the calibration file for a single camera. |
Applies image undistortion on a given image and returns the new undistorted image. If no valid calibration is loaded, the unchanged image is returned.
Parameters:
| Name | Description |
|---|---|
| image | Numpy.array Array representing an image from i.e. cv2.imread. |
Image undistorted.
Applies point undistortion on a given detected pupil and returns the undistorted pupil size. If no valid calibration is loaded, the unchanged pupil size is returned.
This method is faster than complete image distortion as only contour points of the pupil are undistorted.
Parameters:
| Name | Description |
|---|---|
| pupil | Pupil Array representing an image from i.e. cv2.imread. |
Pupil size undistorted, float
Class representing a calibration for a stereo camera system. Images undistorted by this object must be recorded by the same camera system for which the calibration was loaded.
Loads a given calibration file for a single camera. The calibration file should be in OpenCV's file storage format and be created using the PupilEXT software for correct formats.
Parameters:
| Name | Description |
|---|---|
| calibration_file | Str Path to the calibration file for a single camera. |
Applies image undistortion on a given set of images and returns the new undistorted images. If no valid calibration is loaded, the unchanged images are returned.
Image describes the image from the main camera, image_secondary from the secondary camera in the stereo system.
Parameters:
| Name | Description |
|---|---|
| image | Numpy.array Array representing an image from i.e. cv2.imread. |
| image_secondary | Numpy.array Array representing an image from i.e. cv2.imread. |
Images undistorted, tuple
Applies point undistortion on a given detected pupils and returns the undistorted pupil sizes. If no valid calibration is loaded, the unchanged pupil sizes are returned.
This method is faster than complete image distortion as only contour points of the pupil are undistorted.
Parameters:
| Name | Description |
|---|---|
| pupil | Pupil Array representing an image from i.e. cv2.imread. |
| pupil_secondary | Pupil Array representing an image from i.e. cv2.imread. |
Pupil sizes undistorted, tuple of floats
Based on the pupil detection on a set of stereo images capturing the pupil at the same time, the function triangulates the two pupil contours and returns a physical pupil size in mm. (Or whatever metric was used in the calibration)
If no valid calibration was loaded, or one of the two pupil detections were invalid, the value -1 is returned.
Parameters:
| Name | Description |
|---|---|
| pupil | Pupil Array representing an image from i.e. cv2.imread. |
| pupil_secondary | Pupil Array representing an image from i.e. cv2.imread. |
Physical pupil size, float
Please consider to cite our work if you find this repository useful for your research:
B. Zandi, M. Lode, A. Herzog, G. Sakas, and T. Q. Khanh, “PupilEXT: Flexible Open-Source Platform for High-Resolution Pupillometry in Vision Research,” Front. Neurosci., vol. 15, Jun. 2021, doi: 10.3389/fnins.2021.676220.
@Article{10.3389/fnins.2021.676220,
AUTHOR = {Zandi, Babak and Lode, Moritz and Herzog, Alexander and Sakas, Georgios and Khanh, Tran Quoc},
TITLE = {PupilEXT: Flexible Open-Source Platform for High-Resolution Pupillometry in Vision Research},
JOURNAL = {Frontiers in Neuroscience},
VOLUME={15},
PAGES={603},
YEAR={2021},
URL={https://www.frontiersin.org/article/10.3389/fnins.2021.676220},
DOI={10.3389/fnins.2021.676220},
ISSN={1662-453X}}[1] Dongheng Li and Derrick J. Parkhurst. Starburst: A robust algorithm for video-based eye tracking. IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), September 2005.
[2] Lech Swirski, Andreas Bulling, and Neil Dodgson. Robust real-time pupil tracking in highly off-axis images. In Proceedings - 2012 ACM Symposium on Eye Tracking Research and Applications (ETRA), pages 173–176, 2012.
[3] Wolfgang Fuhl, Thomas Kübler, Katrin Sippel, Wolfgang Rosenstiel, and Enkelejda Kasneci. Excuse: robust pupil detection in real-world scenarios. In International Conference on Computer Analysis of Images and Patterns, pages 39–51. Springer, 2015.
[4] Wolfgang Fuhl, Thiago C. Santini, Thomas Kübler, and Enkelejda Kasneci. ElSe: Ellipse selection for robust pupil detection in real-world environments. In Proceedings - 2016 ACM Symposium on Eye Tracking Research and Applications (ETRA), volume 14, pages 123–130, 2016.
[5] Thiago Santini, Wolfgang Fuhl, and Enkelejda Kasneci. PuRe: Robust pupil detection for real-time pervasive eye tracking. Computer Vision and Image Understanding, 170:40–50, 2018.
[6] Thiago Santini, Wolfgang Fuhl, and Enkelejda Kasneci. PuReST: Robust pupil tracking for real-time pervasive eye tracking. In Proceedings - 2018 ACM Symposium on Eye Tracking Research and Applications (ETRA). ACM, 2018.
[7] Thiago Santini, Wolfgang Fuhl, David Geisler and Enkelejda Kasneci. EyeRecToo: Open-source Software for Real-time Pervasive Head-mounted Eye Tracking. VISIGRAPP 2017.
PyPupilEXT integrates several open source libraries. This document provides a list of the used libraries. The respective licenses of the libraries are provided as *.txt file in in the subfolder 3rdparty/PupilEXT_Third_Party_Licenses.
EyeRecToo is an open-source eye tracking software for head-mounted eye tracker and integrates the most advanced state-of-the-art open-source pupil detection algorithms. We used the implementation of the EyeRecToo’s pupil class and the integrated detection methods for PupilEXT. (License: Copyright (c) 2018, Thiago Santini / University of Tübingen). License: For academic and non-commercial use only (Link License | Project Page).
PuRe Thiago Santini, Wolfgang Fuhl, Enkelejda Kasneci, PuRe: Robust pupil detection for real-time pervasive eye tracking. Computer Vision and Image Understanding. 2018, ISSN 1077-3142. https://doi.org/10.1016/j.cviu.2018.02.002. Part of the EyeRecToo software. Copyright (c) 2018, Thiago Santini, University of Tübingen. License: For non-commercial purposes only (Link).
PuReST Thiago Santini, Wolfgang Fuhl, Enkelejda Kasneci. PuReST: Robust pupil tracking for real-time pervasive eye tracking. Symposium on Eye Tracking Research and Applications (ETRA). 2018. https://doi.org/10.1145/3204493.3204578. Part of the EyeRecToo software. Copyright (c) 2018, Thiago Santini, University of Tübingen. License: For non-commercial purposes (Link).
ElSe Wolfgang Fuhl, Thiago Santini, Thomas Kübler, Enkelejda Kasneci. ElSe: Ellipse Selection for Robust Pupil Detection in Real-World Environments. ETRA 2016 : Eye Tracking Research and Application. 2016. Part of the EyeRecToo software. Copyright (c) 2018, Thiago Santini, University of Tübingen. License: For non-comercial use only (Link).
ExCuSe Wolfgang Fuhl, Thomas Kübler, Katrin Simpel, Wolfgang Rosenstiel, Enkelejda Kasneci. ExCuSe: Robust Pupil Detection in Real-World Scenarios. CAIP 2015 : Computer Analysis of Images and Patterns. 2015. Part of the EyeRecToo software. Copyright (c) 2018, Thiago Santini, University of Tübingen. License: For non-comercial use only (Link).
Starburst Dongheng Li, Winfield, D., Parkhurst, D. J. Starburst: A hybrid algorithm for video-based eye tracking combining feature-based and model-based approaches. in 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05) - Workshops vol. 3 79–79 (IEEE, 2005). https://doi.org/10.1109/CVPR.2005.531. Based on the cvEyeTracker Version 1.2.5 implementation. License: GNU General Public License (Link)
Swirski2D Lech Swirski, Andreas Bulling, Neil Dodgson. Robust real-time pupil tracking in highly off-axis images. ETRA 2012: Proceedings of the Symposium on Eye Tracking Research and Applications. 2012. https://doi.org/10.1145/2168556.2168585. License: MIT License, Copyright (c) 2014 Lech Swirski (Link)
Swirski2D Lech Swirski, Neil Dodgson. A fully-automatic, temporal approach to single camera, glint-free 3D eye model fitting. Proceedings of ECEM 2013. 2013. License: MIT License, Copyright (c) 2014 Lech Swirski (Link)
QT is an open-source widget toolkit for creating graphical user interfaces as well as cross-platform applications that run on various software and hardware platforms such as Linux, Windows, macOS, Android or embedded systems. (License: GPL 3.0)
QCustomPlot is a Qt C++ widget for plotting and data visualization. It has no further dependencies and is well documented. (License: GPL 3.0)
OpenCV is a highly optimized computer vision library with focus on real-time applications. In this repository it is used for image manipulation and plotting of ellipse pupil detections. (License: Apache 2 / BSD)
Glog is a library for logging. (License)
Boost is a set of various C++ libraries for processing tasks. (License)
Ceres-Solver is a optimisation library. (License)
Eigen is a library for linear algebra. (License)
Spii is a library for optimisation. (License)
Tbb is for parallel programming. (License)
Breeze Icons is a set of icons. (License)
Gflags is a library for comandline processing. (License)
pybind11 is a lightweight header-only library that exposes C++ types in Python and vice versa, mainly to create Python bindings of existing C++ code. In this repository it is used to create Python bindings for the C++ pupil detection algorithm implementation. (License)
We thank the German Research Foundation (DFG) by funding the research (grant number: 450636577).
This project was made possible by the outstanding previous published open-source projects in the field of pupil detection and eye-tracking. Therefore, we would like to thank the authors of the ground-breaking algorithms PuRe, PuReST, ElSe, ExCuSe, Starburst and Swirski, who made their methods available to the public. Namely, we have to thank Wolfgang Fuhl, Thiago Santini, Thomas Kübler, Enkelejda Kasneci, Katrin Sippel, Wolfgang Rosenstiel, Dongheng Li, D. Winfield, D. Parkhurst, Lech Swirski, Andreas Bulling and Neil Dodgson for their open-source contributions which are part of PyPupilEXT. Additionally, we would like to thank the outstanding developers of the software EyeRecToo, whose open-source eye-tracking software inspired us for this work. We used the implementation of the EyeRecToo’s pupil class and the integrated detection methods for PyPupilEXT.
The software PyPupilEXT is licensed under GNU General Public License v.3.0., Copyright (c) 2021 Technical University of Darmstadt. The pupil detection functionalities of PyPupilEXT are for academic and non-commercial use only. Please note that third-party libraries used in PyPupilEXT may be distributed under other open-source licenses. Please read the above section about the open source projects inside PyPupilEXT.
This program is distributed in the hope that it will be useful, but without any warranty, without even the implied warranty of fitness for a particular purpose.