Multi-class Dense Object Nets

Code for our paper "Multi-step Pick-and-Place Tasks Using Object-centric Dense Correspondences" (IROS 2019)

Training Environment

OS: Ubuntu 18.04 with Python Version=3.6
Device and Packages: using single GTX-1080Ti / RTX-2080Ti with CUDA 10.1 and

torch==1.1.0
opencv-python==4.10.0.*
torchvision==0.2.2*

Install NVIDIA/apex package for FP16 optimization.

git clone https://github.com/NVIDIA/apex
git checkout 8818ba9
python setup.py install --cuda_ext --cpp_ext

Please make sure the correct installation of pytorch(1.1.0) and apex, otherwise the memory usage may excess 11GB.

Data Download

Pretrained Feature-pyramid networks on PascalVOC 2007

Download fpn50.weight
The weight file was extracted from the pretrained FPNSSD512 model released from TorchCV: a PyTorch vision library mimics ChainerCV

Pascal VOC 2012 Images for background randomization

Download the VOC2012 Data( VOCtrainval_11-May-2012.tar)

Our Dataset

Download DataMCD-net.zip

Modifications for the configuration files

Modify the following lines in the configuration file you are going to run, e.g., configs/mcdon.yaml

FPN_pretrained: 'YOUR_PATH_FPN50/fpn50.weight'
texture_base: 'YOUR_PATH_VOC2012/VOCdevkit/VOC2012'
progress_path: 'YOUR_PATH_CHECKPOINT/progress' 
result_path: 'YOUR_PATH_CHECKPOINT/result'
base_dir: 'YOUR_PATH_DATASET/DataMCD-net'

YOUR_PATH_FPN50: the path for finding the pretrained weight file fpn50.weight
YOUR_PATH_VOC2012: the path where you extract the downloaded VOCtrainval_11-May-2012.tar
YOUR_PATH_CHECKPOINT: any desired path for the checkpoints during training
YOUR_PATH_DATASET: the path where you extract our dataset file

Run the training

Train the DON Baseline on 7 classes of objects

python trainer.py configs/baseline.yaml
Train our MCDON (MHSCT-12D M=5,N=2) on 7 classes of objects

python trainer.py configs/mcdon.yaml

Run the visual evaluation(on novel objects) after training

python descriptor_evaluation configs/mcdon.yaml

Pass -tps N to manually specify the restoring N epoch, otherwise we use the tps value in the configuration file
In the evaluation window,
- Adjust the proper threshold on the bar
- Click on either side of the color images to find the matching region in the other side
- Press s to sample new pairs for comparison, q to exit

Pretrained Weights

Pretrained weight file of our MCDON using the setting configs/mcdon.yaml
- Download progress50.tpdata
- Place the weight file in the path
```
YOUR_PATH_CHECKPOINT/progress/mcdon/progress50.tpdata
```
- Run the visual evaluation
```
python descriptor_evaluation.py configs/mcdon.yaml -tps 50
```

cychai1995 / mcdons