canwe / mirror

(EMBC 2020) Camera-based Hand Tracking using a Mirror-based Multi-view Setup

Camera-based Hand Tracking using a Mirror-based Multi-view Setup

This repo contains the sample code for the paper: Camera-based Hand Tracking using a Mirror-based Multi-view Setup.

Paper | Project

If you find our work useful, please consider citing

@inproceedings{mirror:2020,
  title     = {Camera-based Hand Tracking using a Mirror-based Multi-view Setup},
  author    = {Guan Ming, Lim and Prayook, Jatesiktat and Christopher Wee Keong, Kuah and Wei Tech, Ang},
  booktitle = {42st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC)},
  year      = {2020}
}

Usage

If the camera intrinsics and extrinsics have been calibrated and saved, can proceed to step 4) directly

1. Calibrate camera intrinsics from video images of chessboard pattern placed at different pose

python 00_calib_intrin_video.py

2. Prepare data for calibrating camera extrinsics. Note: This program requires manual input to guide the process of whether to flip the projected axis (for mirror image) and number the camera views (in clockwise order starting from the actual camera view)

python 01_prep_extrin_video.py

3. Calibrate camera extrinsics. Note: This program uses a bundle-adjustment like method and the optimization process uses lmfit

python 02_calib_extrin_video.py 

4. Perform model fitting loop. Note: This program assumes the bounding boxes (ROI) of the hand have been determined in the first frame and then perform the subsequent image processing pipelines (2D hand keypoint detection -> 3D model fitting) automatically. This offline processing will save images of 2D hand keypoint and projected 3D hand skeleton, as well as generate a video.mp4 file to view the qualitative result.

python 07_model_fitting_video.py --file ball

Qualitative Results

1. Hand interacting with a ball

2. Hand interacting with a cup

3. Hand interacting with a cube (length of 5 cm)

4. Hand interacting with a bigger cube (length of 7.5 cm)