CMGreenspon / NCams

Toolbox for using multiple cameras from intrinsic calculations to reconstructing kinematics

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

NCams

NCams is a toolbox to use multiple cameras to track and reconstruct the kinematics of primate limbs. NCams leverages state-of-the-art machine learning approaches for image tracking (DeepLabCut) and musculoskeletal modeling (OpenSIM) and includes integration and processing software specifically developed for primate limb tracking.

NCams is installed as a Python module with several submodules that include camera calibration, estimation of relative camera positions, triangulation of the marker information from multiple cameras. The module runs on Windows or *nix.

Features

  1. Intrinsic camera calibration - any number of cameras can be quickly calibrated with support for checkerboard or charucoboards.

  1. Extrinsic calibration/camera pose estimation - multiple methods for calculating the camera extrinsics are available (one-shot, stereo-sequential, and common-point).

  1. Multiple triangulation/3D reconstruction methods including processing/filtering.
  2. Inverse kinematics...

Getting Started

This project's code is available on GitHub. The example raw data and raw images are available online.

Prerequisites

Software:

If you have problems with installations, check out our installation tips.

There are no hardware prerequisites for the core functionality of the toolbox. If using DeepLabCut or the spinnaker_tools, however, then an NVIDIA videocard that supports CUDA or FLIR cameras are required respectively.

Installation

  1. Download the repository or clone it using git: git clone https://github.com/CMGreenspon/NCams.git.
  2. Open Terminal, Command Line, or the desired Anaconda environment (e.g. the one with DeepLabCut installed) in the project folder.
  3. Run python setup.py install.

Example calibration, pose estimation, and triangulation

Intrinsic and Extrinsic Calibration contains example code for setting up creating an NCams_Config, creating a compatible Charucoboard, calibrating camera intrinsics individually or in bulk, and using either one-shot or stereo-sequential extrinsic calibration. Example data for calibration may be found at Box.

Triangulation and Plotting Example walks through triangulating markers based on multiple CSVs (generated by DLC, though conversions can be made), smoothing & filtering in both 2D and 3D, interactive 3D plots of markers over time, and exporting videos with or without skeletal frames.

Labeling and 3D marker reconstruction

The analysis goes over marking images with DeepLabCut, training a network, and triangulation of the marker data.

Analysis of multiple sessions follows the analysis example, but is specifically designed to handle multiple sessions of recordings from the same cameras.

Tips have suggestions on NCams/DeepLabCut use that can be useful.

Inverse kinematics

The guide describes the necessary tools and steps to obtain joint angles of a skeletal model that follow the measured markers using OpenSim (SimTK). We use a [publically available skeletal model](??? not yet) of the human right arm and hand that we modified and markered to fit our purposes.

Dr. Greenspon eating a marshmallow, stills

Structure of the repository

  • ncams -- imported module
  • Examples -- examples and guides on using NCams
  • documentation -- various documentation on NCams
  • opensim_models -- OpenSim skeletal models that we use for inverse kinematics
  • dlc_markers -- images with location of markers relative to bone segments and a DLC config with marker names and skeleton
  • images -- demonstrational images and gifs

Authors

Keywords

multi-camera calibration, triangulation, inverse kinematics, 3d reconstruction

About

Toolbox for using multiple cameras from intrinsic calculations to reconstructing kinematics

License:MIT License


Languages

Language:Python 100.0%