Superpixel segmentation consists of partitioning images into regions composed of similar and connected pixels. Its methods have been widely used in many computer vision applications since it allows for reducing the workload, removing redundant information, and preserving regions with meaningful features. Due to the rapid progress in this area, the literature fails to catch up on more recent works among the compared ones and to categorize the methods according to all existing strategies. This repository allows to evaluate 20 strategies based on nine criteria: connectivity, compactness, delineation, control over the number of superpixels, color homogeneity, robustness, running time, stability, and visual quality. In [1] we provide a comprehensive review with new taxonomy for superpixel segmentation and revisit the recent and popular literature according to our taxonomy. Our experiments show the trends of each approach in pixel clustering and discuss individual trade-offs.
We evaluated 23 superpixel methods (and a grid segmentation baseline) and provided the code for each in this repository in methods. Be aware that we have adapted these codes to match our system setup, and the benchmark inputs and outputs. Some of these methods require a saliency map (ODISF and SICLE) or a contour prior map (SCALP) to compute superpixels. In others folder, we provide the code of U-2-Net [2] and Structured Edge Detection Toolbox [3,4,5] to compute a saliency map and a contour prior map, repectively. The following Table presents the reference paper and code of each superpixel method.
| Method | Ref. | Code link |
|---|---|---|
| AINET | Paper | AINET |
| CRS | Paper | CRS |
| DAL-HERS | Paper | DAL-HERS |
| DISF | Paper | DISF |
| DRW | Paper | DRW |
| ERGC | Paper | ERGC |
| ERS | Paper | ERS |
| ETPS | Paper | ETPS |
| GMMSP | Paper | GMMSP |
| IBIS | Paper | IBIS |
| ISF | Paper | ISF |
| LNSNet | Paper | LNSNet |
| LSC | Paper | LSC |
| ODISF | Paper | OISF |
| RSS | Paper | RSS |
| SCALP | Paper | SCALP |
| SEEDS | Paper,Paper | SEEDS |
| SH | Paper | SH |
| SICLE | Paper,Paper | SICLE |
| SIN | Paper | SIN |
| SLIC | Paper | SLIC |
| SNIC | Paper | SNIC |
| SSFCN | Paper | SSFCN |
We selected five datasets which impose different challenges for superpixel segmentation: Birds [6]; Insects [6]; Sky [7]; and ECSSD [8]; NYUV2 [9]. Birds has 150 natural images of birds with thin and elongated objects’ parts. Similarly, Insects has 130 images of invertebrates with less texture on background regions. Sky has 60 images for sky segmentation with large homogeneous regions with subtle luminosity variations. The Extended Complex Scene Saliency Dataset (ECSSD) is composed of 1000 images with objects and backgrounds whose textures are complex. Finally, the NYUV2 dataset is composed of 1449 video sequences from several indoor scenes recorded by Microsoft Kinect.
We assess connectivity, compactness, delineation, color homogeneity, robustness, running time, stability, control over the number of superpixels, and visual quality in superpixel segmentation methods. This repository contais our evaluation code (evaluation) with five superpixel evaluation measures: Similarity between Image and Reconstruction from Superpixels (SIRS) [1], Boundary Recall (BR) [10], Explained Variation (EV) [11], Undersegmentation Error (UE) [12], and Compactness (CO) [13]. In addition, we provide code to assess running time, control over the number of superpixels, connectivity, and robustness.
Please make sure your system have the following packages:
- OpenMP
- OpenCV 4.5
- CMake
- Python 3
- Pipenv
- Cuda (only for deep-based strategies)
- Boost library:
sudo apt install libboost-all-dev
This project was developed under Ubuntu operational system; therefore, it is NOT GUARANTEED to work properly in other systems (e.g. Windows and macOS).
Cuda and the python pakages torch, torchvision, and torchaudio are required for the deep-based strategies. In ubuntu systems, the following instruction may help:
pipenv install- Run
nvidia-smiand copy the cuda version - Run
pipenv shell - Find your version in https://pytorch.org/get-started/previous-versions
- Run the following command with your version. Example for cuda 10.1:
pip install ==1.8.1+cu101 torchvision==0.9.1+cu101 torchaudio==0.8.1 -f https://download.pytorch.org/whl/cu101/torch_stable.html - Test whether the installed version foud the cuda driver:
python3 -c "import torch; print(torch.cuda.is_available())"
- To compile all files:
bash make.sh- Every method that needs an executable file contains a Makefile in its folder.
make.shjust call each one and call the Makefile in evaluation folder.
- Every method that needs an executable file contains a Makefile in its folder.
- Scripts has bash and python scripts to run saliency/contour maps (others), segmentation (Segmentation), and evaluation (Evaluation).
- To run segmentation, execute
bash run_segmentation.sh(run_segmentation.sh).- The folder Methods has scripts for each method.
- One may specify the methods, datasets, directories and parameters in
run_segmentation.sh. - Similarly, one can generate segmentation for images with average blur or salt and peper noise.
- To generate segmentation for images with average blur or salt and peper noise:
- Run
python3 add_noise.py(add_noise.py) to generate noised images - Then, generate the segmentations by running
run_segmentation_robustness.sh(run_segmentation_robustness.sh). - One may specify the methods, datasets, directories and parameters in
run_segmentation_robustness.sh.
- Run
- To evaluate:
- Run
bash run_eval.sh(run_eval.sh) to assess BR, UE, SIRS, EV, CO, superpiixel connectivity, and the control over the superpixel number. - Run
bash run_eval_robustness.sh(run_eval.sh) to assess BR, UE, SIRS, EV, CO, superpiixel connectivity, and the control over the superpixel number in the noised images. - Run
python3 eval_stability.py(eval_stability.py) to assess the stability of superpixel evaluation.
- Run
This code was implemented and evaluated in computers with the following specifications:
-
All methods:
- Operational System: Ubuntu 22.04
- CPU: Intel® Core™ i5-7200U CPU @ 2.50GHz × 4
- Memory: 24GB RAM
- GPU: NVIDIA GeForce 920MX with 2GB RAM
-
Only AINET, DAL-HERS, SIN, and SSFCN:
- Operational System: Ubuntu 16.04
- CPU: Intel® Core™ i7-8700 @ 3.20GHz x 12
- Memory: 31GB RAM
- GPU: Nvidia GeForce GTX 1080 with 8GB RAM.
The authors thank the Conselho Nacional de Desenvolvimento Científico e Tecnológico – CNPq – (Universal 407242/2021-0, PQ 303808/2018-7, 310075/2019-0), the Fundação de Amparo a Pesquisa do Estado de Minas Gerais – FAPEMIG – (PPM-00006-18), the Fundação de Amparo a Pesquisa do Estado de São Paulo – FAPESP – (2014/12236-1) and the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (COFECUB 88887.191730/2018-00 and Finance code 001) for the financial support.
If you have any questions or faced an unexpected behavior (e.g. bugs) or sugestions, please feel free to contact through the following email address: isabela.borlido@sga.pucminas.br
If this work was useful for your research, please cite our paper:
@article{10.1145/3652509,
author = {Barcelos, Isabela Borlido and Bel\'{e}m, Felipe de Castro and Jo\~{a}o, Leonardo de Melo and Jr., Zenilton K. G. do Patroc\'{\i}nio and Falc\~{a}o, Alexandre Xavier and Guimar\~{a}es, Silvio Jamil Ferzoli},
title = {A comprehensive review and new taxonomy on superpixel segmentation},
year = {2024},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
issn = {0360-0300},
url = {https://doi.org/10.1145/3652509},
doi = {10.1145/3652509},
note = {Just Accepted},
journal = {ACM Comput. Surv.},
month = {mar}
}
[1] Isabela B Barcelos, Felipe De C Belém, Leonardo De M João, Alexandre X Falcão, and Guimarães Silvio JF. 2022. Improving color homogeneity measure in superpixel segmentation assessment. In 2022 35th SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI), Vol. 1. 79–84. https://doi.org/10.1109/SIBGRAPI55357.2022.9991772.
[2] QIN, Xuebin et al. U2-Net: Going deeper with nested U-structure for salient object detection. Pattern recognition, v. 106, p. 107404, 2020.
[3] DOLLÁR, Piotr; ZITNICK, C. Lawrence. Structured forests for fast edge detection. In: Proceedings of the IEEE international conference on computer vision. 2013. p. 1841-1848.
[4] DOLLÁR, Piotr; ZITNICK, C. Lawrence. Fast edge detection using structured forests. IEEE transactions on pattern analysis and machine intelligence, v. 37, n. 8, p. 1558-1570, 2014.
[5] ZITNICK, C. Lawrence; DOLLÁR, Piotr. Edge boxes: Locating object proposals from edges. In: Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13. Springer International Publishing, 2014. p. 391-405.
[6] Lucy A. C. Mansilla and Paulo A. V. Miranda. 2016. Oriented Image Foresting Transform Segmentation: Connectivity Constraints with Adjustable Width. In 2016 29th SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI). 289–296. https://doi.org/10.1109/SIBGRAPI.2016.047
[7] Eduardo Barreto Alexandre, Ananda Shankar Chowdhury, Alexandre Xavier Falcao, and Paulo A. Vechiatto Miranda. 2015. IFT-SLIC: A General Framework for Superpixel Generation Based on Simple Linear Iterative Clustering and Image Foresting Transform. In 2015 28th SIBGRAPI Conference on Graphics, Patterns and Images. 337–344. https://doi.org/10.1109/SIBGRAPI.2015.20
[8] Jianping Shi, Qiong Yan, Li Xu, and Jiaya Jia. 2015. Hierarchical image saliency detection on extended CSSD. IEEE transactions on pattern analysis and machine intelligence 38, 4 (2015), 717–729.
[9] SILBERMAN, Nathan et al. Indoor segmentation and support inference from rgbd images. In: Computer Vision–ECCV 2012: 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part V 12. Springer Berlin Heidelberg, 2012. p. 746-760.
[10] David R Martin, Charless C Fowlkes, and Jitendra Malik. 2004. Learning to detect natural image boundaries using local brightness, color, and texture cues. IEEE transactions on pattern analysis and machine intelligence 26, 5 (2004), 530–549. https://doi.org/10.1109/TPAMI.2004.1273918
[11] Alastair P Moore, Simon JD Prince, Jonathan Warrell, Umar Mohammed, and Graham Jones. 2008. Superpixel lattices. In 2008 IEEE conference on computer vision and pattern recognition. IEEE, 1–8.
[12] Peer Neubert and Peter Protzel. 2012. Superpixel benchmark and comparison. In Proc. Forum Bildverarbeitung, Vol. 6. KIT Scientific Publishing, 1–12.
[13] Alexander Schick, Mika Fischer, and Rainer Stiefelhagen. 2012. Measuring and evaluating the compactness of superpixels. In Proceedings of the 21st international conference on pattern recognition (ICPR2012). IEEE, 930–934.