This repository implements a training pipeline for Anchor to Joint A2J paper to train a hand posture estimation model based on depth maps. We will use NYU Hands dataset to train our model.
The original authors did a great job publishing their code in the following repository Link
The main reason for a new pipeline is to modify the number of joints, adding visualization, and some preprocessing steps to be able to run real time inference with Azure Kinect camera.
NOTE Please refer to the following repository for running realtime inference on a Jetson platform after training the model Link
Pleae Download the NYU Hand Pose Dataset and unzip the folder.
Please modify the following in pipeline/constants.py
- DATASET_PATH.
DATASET_PATH="FULL/PATH/TO/DATASET"- (OPTIONAL) SAVE_PATH. this must be the full path to the checkpoint file. by default it will save all the checkpoint files for PATH/TO/PROJECT/check_point.
- Model naming convention: A2J_BACKBONE_NUMJOINTS_CAMERAPOSE.pth
- setup max epoch in pipeline/constants.py*
SAVE_PATH="FULL/PATH/TO/SAVE/DIRECTORY"-
(OPTIONAL) BACKBONE_NAME_SWITCHER. This will modify which network is used as the backbone. Set the backbone you want to use to True
-
(OPTIONAL) CAMERA_VIEW. NYU dataset has been collected with 3 different camera view points. A2J Trained their model on the front view, we will do the same. You can also set this value to ALL in order to use all the different views.
-
(OPTIONAL) NUM_JOINTS. You can set this to either 16 or 36. By default it is set to 16 joints.
Having the dataset ready We need to train the model.
in pipeline dircorty run training by
python3 train # To start training a model from scratch
python3 train --resume True # To continue an already trained model
python3 train --resume_from_model "FULL/PATH/TO/MODEL" # To continue an already trained modelrun training by
python3 visualize.py # To run inference on a validation image
