tianxingyzxq / leap

LEAP Estimates Animal Pose

LEAP

Full movie: YouTube

This repository contains code for LEAP (LEAP Estimates Animal Pose), a framework for animal body part position estimation via deep learning.

Preprint: Pereira et al., bioRxiv (2018)

We are still working on documentation and preparing parts of the code. See the Features section below for an overview and status of each component.

We recommend starting with the Tutorial: Training Leap From Scratch.

Features

• Tracking and alignment code
• Cluster sampling GUI
• Skeleton creation GUI (create_skeleton)
• GUI for labeling new dataset (label_joints)
• Network training through the labeling GUI
• MATLAB (predict_box.m) and Python (leap.training) interfaces for predicting on new data
• GUI for predicting on new data
• Training data + labels for main fly dataset used in analyses
• Trained network for predicting on main fly dataset
• Analysis/figure generation code
• Documentation
• Examples of usage

Installation

MATLAB dependencies

GUIs and analyses are implemented in MATLAB, but is not required for using the neural network functionality implemented in Python.

We use MATLAB R2018a with the following toolboxes: Parallel Computing Toolbox, Statistics and Machine Learning Toolbox, Computer Vision Toolbox, Image Processing Toolbox, Signal Processing Toolbox.

All MATLAB external toolboxes are included in the leap/toolbox subfolder. Just add the leap subdirectory to the MATLAB Search Path to access all functionality:

addpath(genpath('leap'))

Python dependencies

All neural network and GPU functionality is implemented in Python. The library was designed to be easy to use by providing commandline interfaces, but it can also be used programatically if the MATLAB GUIs are not required.

For the Python environment, we recommend Anaconda 5.1.0 with Python 3.6.4.

The versions below were used during development of LEAP but other versions will also likely work.

Libraries required are easily installable via the pip package manager:

pip install -Iv numpy==1.14.1
pip install -Iv h5py==2.7.1
pip install -Iv clize==4.0.3

You will also need OpenCV 3 with Python bindings. We recommend using skvark's excellent precompiled packages:

pip install -Iv opencv-python==3.4.0.12

For GPU support, you'll want to first install the CUDA drivers with CuDNN and then install these packages:

pip install -Iv tensorflow-gpu==1.6.0
pip install -Iv keras==2.1.4

See the TensorFlow installation guide for more info.

Usage

Refer to the Tutorial: Training Leap From Scratch.

Preprocessing

GUI Workflow

1. Cluster sampling: Call cluster_sample from MATLAB commandline to launch GUI.
2. Create skeleton: Call create_skeleton from MATLAB commandline to launch GUI.
3. Label data and train: Call label_joints from MATLAB commandline to launch GUI.
4. Batch estimation: Coming soon.

Programmatic API

See leap/training.py and leap/predict_box.py for more info.

Contact and more information

Reach out to us via email: Talmo Pereira ([e-mail])