JCSA-RM RGBD Image Segmentation and Analysis Method [1,2,3]

Repository for the source code of MATLAB implementation of the "RGB-D image Segmentation using the Joint Color-Spatial-Axial clustering and Region Merging (JCSA-RM)" method.

The JCSA-RM method is an RGB-D (joint color+depth) image segmentation method. This repo provides demos with/without a GUI with MATLAB code to perform the following tasks:
a. load RGB-D image data from a mat file (contains RGB, Depth and Image Normals in an structure) and display them.
b. Generate segmented image and display it.

How to use demo (tested in Matlab2017b):

Run the MATLAB file name: RGBD_Seg_JCSA_RM.m for the GUI version and demo_NO_GUI.m otherwise
Load data/samples files name: rgbd_info_1.mat, rgbd_info_2.mat, rgbd_info_1_better_normals.mat and rgbd_info_2_better_normals.mat.

Select _better_normals in order to experiment with unambiguas surface normals.
Choose different methods for testing, among: (a) JCSA, (b) JCSD, (c) JCSA-RM and (d) JCSD-RM.

Application:

This segmentation method has been used to segment/analyze RGB-D images captured by the Microsoft Kinect camera. For details and other possible applications please see the references.

Results to compare:

JCSD_RM_Results.zip file contains the results of applying JCSA-RM [1,2,3] method on the NYU depth database (NYUD2) [4]. Each result file consists of segmentation - labels of pixels and final scores – VoI, BDE, PRI and GTRC for the 1449 NYUD2 [4] images in half scaled (down) image.

Code running issues:

It runs on Matlab2017b. If you encounter error with - computeTraceTerm then go to the directory called 'rgbd' and compile mex file as:
mex computeTraceTerm.cpp.

Extensions and scopes:

You can extend the JCSA method for clustering heterogeneous data. However, for now the method is limited to the 3 Dimensional data with the directional distributions [5,6]. You can also extend it to work with higher dimensional data by extending [5] or [6].
You can use the RM method independently to perform segmentation. It requires the clustering labels, the color image and image normals as input.

References:

[1] Abul Hasnat, Olivier Alata and Alain Trmeau, Joint Color-Spatial-Directional clustering and Region Merging (JCSD-RM) for unsupervised RGB-D image segmentation, In IEEE Trans. on Pattern Analysis and Machine Intelligence (TPAMI), Vol 38, Issue 11, pages 2255 - 2268, 2015. pdf download

[2] M Hasnat, O Alata, A Trémeau, Joint Color-Spatial-Directional clustering and Region Merging (JCSD-RM) for unsupervised RGB-D image segmentation, in arXiv preprint arXiv:1509.01788, 2015. pdf download

[3] M. A. Hasnat, O. Alata, and A. Tremeau, “Unsupervised RGB-D image segmentation using joint clustering and region merging,” in British Machine Vision Conference (BMVC). BMVA Press, 2014. pdf download

[4] N. Silberman, D. Hoiem, P. Kohli, and R. Fergus, “Indoor segmentation and support inference from RGBD images,” in ECCV 2012. Springer, 2012.

[5] Abul Hasnat, Olivier Alata and Alain Trmeau, Model-Based Hierarchical Clustering with Bregman Divergence and Fisher Mixture Model: Application to Depth Image Analysis, In Statistics and Computing (STCO), Vol 26, Issue 4, pp 861880, 2015. pdf download

[6] Abul Hasnat, Olivier Alata and Alain Tremeau, Unsupervised Clustering of Depth Images using Watson Mixture Model, In Int. Conference on Pattern Recognition (ICPR) 2014, Stockholm, Sweden. pdf download

Dylanliu612 / JCSA_RM_Image_Seg