CRF-RNN for Semantic Image Segmentation - Keras/Tensorflow version

Live demo: http://crfasrnn.torr.vision
Caffe version: http://github.com/torrvision/crfasrnn

This repository contains the codes implementing the following papers:

Keras/Tensorflow code for the "CRF-RNN" semantic image segmentation method, published in the ICCV 2015 paper Conditional Random Fields as Recurrent Neural Networks. This paper was initially described in an arXiv tech report. The online demo of this project won the Best Demo Prize at ICCV 2015. Original Caffe-based code of this project can be found here. Results produced with this Keras/Tensorflow code are almost identical to that with the Caffe-based version.

If you use this code/model for your research, please cite the following paper:

@inproceedings{crfasrnn_ICCV2015,
    author = {Shuai Zheng and Sadeep Jayasumana and Bernardino Romera-Paredes and Vibhav Vineet and
    Zhizhong Su and Dalong Du and Chang Huang and Philip H. S. Torr},
    title  = {Conditional Random Fields as Recurrent Neural Networks},
    booktitle = {International Conference on Computer Vision (ICCV)},
    year   = {2015}
}

The above code is modified to incorporate papers published in ICIP 2018, Superpixel-enhanced Pairwise Conditional Random Fields for Semantic Segmentation. The modified code is capable of incorporating superpixel-cues in the CRFs to further increase the accuracy simply by feeding in one or multiple filtered images in the input.

If you choose to add superpixels by using this code/model for your research, please cite the following paper:

@INPROCEEDINGS{8451218,
author={L. Sulimowicz and I. Ahmad and A. Aved},
booktitle={2018 25th IEEE International Conference on Image Processing (ICIP)},
title={Superpixel-Enhanced Pairwise Conditional Random Field for Semantic Segmentation},
year={2018},
volume={},
number={},
pages={271-275},
month={Oct},}

Installation Guide

Step 1: Clone the repository

$ git clone https://github.com/sadeepj/crfasrnn_keras.git

The root directory of the clone will be referred to as crfasrnn_keras hereafter.

Step 2: Install dependencies

Note: If you are using a Python virtualenv, make sure it is activated before running each command in this guide.

Use the requirements.txt file (or requirements_gpu.txt, if you have a GPU device) in this repository to install all the dependencies via pip:

$ cd crfasrnn_keras
$ pip install -r requirements.txt  # If you have a GPU device, use requirements_gpu.txt instead

As you can notice from the contents of requirements.txt, we only depend on tensorflow, keras, and h5py. Additionally, Pillow is required for running the demo. After installing the dependencies, run the following commands to make sure they are properly installed:

$ python
>>> import tensorflow
>>> import keras

You should not see any errors while importing tensorflow and keras above.

Step 3: Build CRF-RNN custom op C++ code

Run make inside the crfasrnn_keras/src/cpp directory:

$ cd crfasrnn_keras/src/cpp
$ make

Note that the python command in the console should refer to the Python interpreter associated with your Tensorflow installation before running the make command above.

You will get a new file named high_dim_filter.so from this build. If it fails, refer to the official Tensorflow guide for building a custom op for help.

Note: This make script works on Linux and macOS, but not on Windows OS. If you are on Windows, please check this issue and the comments therein for build instructions. The official Tensorflow guide for building a custom op does not yet include build instructions for Windows.

Step 4: Download the pre-trained model weights

Download the model weights from here or here and place it in the crfasrnn_keras directory with the file name crfrnn_keras_model.h5.

Step 5: Run the demo

$ cd crfasrnn_keras
$ python run_demo.py

If all goes well, you will see the segmentation results in a file named "labels.png".

About incorporating unsupervised segmentation cues

Any extra set of pre-processed segmented image needs to be firstly uploaded:

seg_data, seg_h, seg_w = util.get_preprocessed_image(segment_file)

Together with img_data, we feed data into the system as a list [img_data, seg_data]. If we want to feed in more than one set of segmented images, simply put them in the list.

probs = model.predict([img_data, seg_data], verbose=False)[0, :, :, :]

The following images show the difference w/o CRF.

No CRF With Pairwise CRF With superpixel

Notes

Current implementation of the CrfRnnLayer only supports batch_size == 1
An experimental GPU version of the CrfRnnLayer that has been tested on CUDA 9 and Tensorflow 1.7 only, is available under the gpu_support branch. This code was contributed by thwjoy.

liyin2015 / superpixel_crfasrnn