oguzhanmeteozturk / drishti

Real time eye tracking for embedded and mobile devices.

drishti

Real time eye tracking for embedded and mobile devices in C++11.

Overview

Goal: SDK size <= 1 MB and combined resources (object detection + regression models) <= 4 MB.

• Hunter package management and CMake build system by Ruslan Baratov, as well as much of the cross platform Qt work: "Organized Freedom!" :)
• A C++ and OpenGL ES 2.0 implementation of Fast Feature Pyramids for Object Detection (see Piotr's Matlab Toolbox) for face and eye detection; UPDATE: The acf code has been moved to a separate repository with a hunter package HERE
• Iris ellipse fitting via Cascaded Pose Regression (Piotr Dollar, et al) + XGBoost regression (Tianqi Chen, et al)
• Face landmarks and eye contours provided by "One Millisecond Face Alignment with an Ensemble of Regression Trees" (Kazemi, et al) using a modified implementation from Dlib (Davis King) (normalized pixel differences, line indexed features, PCA size reductions)
• OpenGL ES friendly GPGPU shader processing and efficient iOS + Android texture handling using a modified version of ogles_gpgpu (Markus Kondrad) with a number of shader implementations taken directly from GPUImage (Brad Larson)

iPhone @ 30 FPS (VIDEO)

Drishti Right Eye Annotation Scheme

FEATURE	SPECIFICATION
eyelids	2D points 0-15
crease	2D points 16-24
iris center	2D point 25
outer limbus	limbus intersection with ray from outer corner to iris center
inner limbus	limbus intersection with ray from inner corner to iris center
iris ellipse	2D center, minor axis, major axis, angle (radians)
pupil ellipse	2D center, minor axis, major axis, angle (radians)

• the left eye is obtained by Y axis mirroring
• total (27*2)+(2*5) = 64 parameters
• the eye crease is useful for pose indexing, but better guidelines are needed
• the 2D limbus points are slightly redundant (given the ellipse iris model) but the intersection points are stable with respect to squinting and provide an efficient anchor for posed indexed features (accurate point-to-ellipse distances are non-trivial and are fairly computationally intensive)
• currently 2D only (gaze angle ground truth would be beneficial)

Quick Start (i.e., How do I make this library work?)

Drishti is a CMake based project that uses the Hunter package manager to download and build project dependencies from source as needed. Hunter contains detailed documentation, but a few high level notes and documentation links are provided here to help orient first time users. In practice, some working knowledge of CMake may also be required. Hunter itself is written in CMake, and is installed as part of the build process from a single HunterGate() macro at the top of the root CMakeLists.txt file (typically cmake/Hunter/HunterGate.cmake) (you don't have to build or install it). Each CMake dependency's find_package(FOO) call that is paired with a hunter_add_package(FOO CONFIG REQUIRED) will be managed by Hunter. In most cases, the only system requirement for building a Hunter project is a recent CMake with CURL support and a working compiler correpsonding to the operative toolchain. Hunter will maintain all dependencies in a versioned local cache by default (typically ${HOME}/.hunter) where they can be reused in subsequent builds and shared between different projects. They can also be stored in a server side binary cache -- select toolchains will be backed by a server side binary cache (https://github.com/elucideye/hunter-cache) and will produce faster first time builds (use them if you can!).

The Travis (Linux/OSX/iOS/Android) and Appveyor (Windows) CI scripts in the project's root directory can serve as a reference for basic setup when building from source. To support cross platform builds and testing, the CI scripts make use of Polly: a set of common CMake toolchains paired with a simple polly.py CMake build script. Polly is used here for convenience to generate CMake command line invocations -- it is not required for building Hunter projects.

To reproduce the CI builds on a local host, the following setup is recommended:

• Install compiler: http://cgold.readthedocs.io/en/latest/first-step.html
• Install CMake (and add to PATH)
• Install Python (for Polly)
• Clone Polly and add <polly>/bin to PATH

Note: Polly is not a build requirement, CMake can always be used directly, but it is used here for convenience.

The bin/hunter_env.{sh,cmd} scripts (used in the CI builds) can be used as a fast shortcut to install these tools for you. You may want to add the PATH variables permanently to your .bashrc file (or equivalent) for future sessions.

Linux/OSX/Android/iOS	Windows
source bin/hunter_env.sh	bin\hunter_env.cmd

After the environment is configured, you can build for any supported Polly toolchain (see polly.py --help) with a command like this:

polly.py --toolchain ${TOOLCHAIN} --config ${CONFIG} --fwd HUNTER_CONFIGURATION_TYPES=${CONFIG} --install --verbose

Applications

Please see the README for the drishti-hci console application to see an example of a full eye tracking pipeline with the GPGPU optimizations.

Integration

Drishti is also available as a hunter package. If you would like to integrate drishti in your project, please see the hunter drishti package documentation.

Steps:

Add cmake/HunterGate.cmake and a minimal cmake/Hunter/config.cmake to your project:

mkdir -p cmake/Hunter
wget https://raw.githubusercontent.com/hunter-packages/gate/master/cmake/HunterGate.cmake -O cmake/HunterGate.cmake
wget https://raw.githubusercontent.com/ruslo/hunter/master/examples/drishti/config.cmake -O cmake/Hunter/config.cmake

Add HunterGate(URL <url> SHA1 <sha1>) to the top of your CMakeLists.txt (You can find updated release information here).

include("cmake/HunterGate.cmake")
HunterGate(
    URL "https://github.com/ruslo/hunter/archive/v0.19.140.tar.gz"
    SHA1 "f2c30348c05d0d424976648ce3560044e007496c"
    LOCAL # use cmake/Hunter/config.cmake
)

Finally, add the drishti package to your CMakeLists.txt and link it to your target:

hunter_add_package(drishti)
find_package(drishti CONFIG REQUIRED)
target_link_libraries(your_app_or_lib drishti::drishti)

You can customize the drishti package (and dependencies) by specifying a VERSION and/or CMAKE_ARGS (options) list for each package in cmake/Hunter/config.cmake.

Please see https://github.com/elucideye/drishti_hunter_test for a minimal working example using the drishti hunter package.

Toolchains

The configurations listed below have all been tested. In general, most C++11 toolchains should work with minimal effort. A CI comment indicates that the configuration is part of the Travis or Appveyor CI tests, so all Hunter packages will be available in the server side binary cache.

Linux (Ubunty Trusty 14.04):

• TOOLCHAIN=gcc-5-pic-hid-sections-lto CONFIG=Release # CI
• TOOLCHAIN=libcxx CONFIG=Release # w/ clang 3.8

OSX:

• TOOLCHAIN=osx-10-11-hid-sections-lto CONFIG=Release # CI
• TOOLCHAIN=osx-10-12-sanitize-address-hid-sections CONFIG=Release # CI
• TOOLCHAIN=xcode-hid-sections CONFIG=Release # generic

iOS:

• TOOLCHAIN=ios-nocodesign-10-1-arm64-dep-9-0-device-libcxx-hid-sections-lto CONFIG=MinSizeRel # CI
• TOOLCHAIN=ios-10-1-arm64-dep-8-0-hid-sections CONFIG=Release

Android (from OSX):

• TOOLCHAIN=android-ndk-r10e-api-19-armeabi-v7a-neon-hid-sections CONFIG=MinSizeRel # CI
• TOOLCHAIN=android-ndk-r10e-api-19-armeabi-v7a-neon-hid-sections-lto CONFIG=MinSizeRel

Windows:

• TOOLCHAIN=vs-14-2015-sdk-8-1 CONFIG=Release # CI
• TOOLCHAIN=vs-14-2015-sdk-8-1 CONFIG=Debug # CI
• TOOLCHAIN=vs-14-2015-win64-sdk-8-1 CONFIG=Release # CI
• TOOLCHAIN=vs-14-2015-win64-sdk-8-1 CONFIG=Debug # CI

The polly out of source build trees are located in _builds/${TOOLCHAIN}, the final build products (the stuff you want) are installed in _install/${TOOLCHAIN}, and the build logs are dumped in _logs/${TOOLCHAIN}. The iOS frameworks are installed in _frameworks/${TOOLCHAIN}.