BardOfCodes / seg_unravel

Code Related to "Per-Pixel Feedback for improving Semantic Segmentation", My dissertation submission.

HeatMap Horse	HeatMap Person

Fixation Flow	Fixation Flow

Seg-Unravel

This code contains implementation for Segmentation-unravel, something I attempted in my dissertation thesis, "Per-Pixel Feedback for improving Semantic Segmentation" Arxiv Link.

Segmentation-Unravel is an extension of CNN-Fixations for networks performing semantic segmentation. The motive for this work was

• to see if global context played a role in segmentation, and

• identifying if a few parts of object played a important role in segmentation. (For example a dog's head for dog's segmentation.)

Segmentation-Unravel backtracks the highest contributors of the segmentation output from the layer below. This unravelling process is continued till the image level to find 'fixation'-like points which might have been crucial for the segmentation.

This repository contains code for Seg-Unravel on the following Segmentation Networks (in caffe):

1. fcn-alexnet from "Fully Convolutional Networks for Semantic Segmentation". (Arxiv link.)

2. DeepLab VGG16 Large-FOV from "Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs". (Arxiv Link)

3. DeepLab Resnet 101 Multi-Scale from "DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs". (Arxiv Link)

Instructions

First, run the setup.sh file, to

• download the caffemodel and prototxt files for the segmentation models
• download the caffe versions for each of these networks
• adapt the prototxt files for this project.

source setup.sh

Kindly install caffe according to your setup(with/without CUDA, for instance). As there are way too many variables while installing caffe, it has not been automated.

Now, we can run the demo.py file with various options to run Seg-Unravel on any selected network:

python demo.py --network <network> --shift_type <shift-type> --image <image-path>

This will save temp.npy with fixations at various layers, temp_<class-name>.png as image-level fixation map for the detected classes, and temp.gif as the fixation backtrack through layers animation for biggest segmented class(except background) in the input image.

Note This code is a post-apocalypse recovery of code written during the project, about an year ago. It is not the cleanest code, nor the most lucid. However, it gets the job done, without errors.

Note In context of segmentation, these 'fixations' might not be very useful. However, if anyone wants to use it, they may!

TODO Correct GIF Function for Resnet_MSc.

Acknowledgement

I would like to thank:

1. My Dissertation Guides, Mopuri Konda Reddy and Ravi Kiran Sarvadevabhatla.

2. Utsav Garg for the CNN-fixation code.

3. My Advisor for this project Professor R. Venkatesh Babu.

4. Members of Video Analytics Lab for general advice.