On the Texture Bias for Few-Shot CNN Segmentation
This repository contains the code for deep auto-encoder-decoder network for few-shot semantic segmentation with state of the art results on FSS 1000 class dataset and Pascal 5i. This method embeds different ferequncy information in the CNN representation to overcome with the texture bias and applies bidirectional convolutional LSTM layers to perform non-liner parametric k shot setting for few-shot semantic segmentation. If this code helps with your research please consider citing the following paper:
R. Azad, Abdur. R. Fayjie, Claude Kauffman, Ismail Ben Ayed, Marco Pedersoli and Jose Dolz "On the Texture Bias for Few-Shot CNN Segmentation", arXiv preprint arXiv, 2020, download link.
Please consider starring us, if you found it useful. Thanks
Updates
- March 7, 2020: Implementation code is available now.
Prerequisties and Run
This code has been implemented in python language using Keras libarary with tensorflow backend and tested in ubuntu OS, though should be compatible with related environment. following Environement and Library needed to run the code:
- Python 3
- Keras version 2.2.0
- tensorflow backend version 1.13.1
Run Demo
The implementation code is availabel in Source Code folder.
1- Download the FSS1000 dataset from this link and extract the dataset.
2- Run Train_DOGLSTM.py for training Scale Space Encoder model using k-shot episodic training. The model will be train for 50 epochs and for each epoch it will itterate 1000 episodes to train the model. The model will saves validation performance history and the best weights for the valiation set. It also will report the MIOU performance on the test set. The model by default will use VGG backbone with combining Block 3,4 and 5 but other combination can be call in creatign the model. It is also possible to use any backbone like Resnet, Inception and etc....
3- Run Train_weak.py for training Scale Space Encoder model using k-shot episodic training and evaluatign on the weak annotation test set. This code will use weaklly annotated bouning box as a label for the support set on test time.
Notice: parser_utils.py can be used for hyper parameter setting and defining data set address, k-shot and n-way.
Quick Overview
Structure of the Proposed Scale Space encoder for reducing texture bias effect
Visual representation of 21 classes from 1000-class dataset with their masks and generated bounding box Download link
Results
For evaluating the performance of the proposed method, Two challenging few-shot semantic segmentaion data sets have been considered. In bellow, results of the proposed approach illustrated.
Task 1: FSS-1000: A 1000-Class Dataset for Few-Shot Segmentation
In order to compare the proposed method with state of the art appraoches on few-shot semantic segmentation, we reported our result using mean Intersection over Unition (mIoU) metric on both 1-shot and 5-shot settings.
Table 1: Results of 1-way 1-shot segmentation on the FSS-1000 data set employing the mIoU metric.
| Methods | Year | mIoU |
|---|---|---|
| Shaban et. all OSLSM | 2017 | 70.29% |
| Rakelly et. all co-FCN | 2018 | 71.94% |
| Wei et. all FSS-1000 | 2019 | 73.47% |
| Hendryx et. all FOMAML | 2020 | 75.19% |
| Azad et. all Proposed Method (Baseline) | 2020 | 74.19% |
| Azad et. all Proposed Method (Baseline + DoG) | 2020 | 78.71% |
| Azad et. all Proposed Method (Baseline + DoG + BConvLSTM) | 2020 | 80.83% |
Table 2: Results of 1-way 5-shot segmentation on the FSS-1000 data set employing the mIoU metric.
| Methods | Year | mIoU |
|---|---|---|
| Shaban et. all OSLSM | 2017 | 73.02% |
| Rakelly et. all co-FCN | 2018 | 74.27% |
| Wei et. all FSS-1000 | 2019 | 80.12% |
| Hendryx et. all FOMAML+ regularization | 2020 | 80.60% |
| Hendryx et. all FOMAML+ regularization | 2020 | 82.19% |
| Azad et. all Proposed Method (Baseline + DoG + BConvLSTM) non-parametric fusion | 2020 | 81.65% |
| Azad et. all Proposed Method (Baseline + DoG + BConvLSTM) parametric fusion | 2020 | 83.36% |
Visual Segmentation result on FSS-1000
Sample of 1-shot segmentation result on the FSS-1000 dataset
Task 2: Pascal 5i Dataset for Few-Shot Segmentation
In order to compare the proposed method with state of the art appraoches on few-shot semantic segmentation, we reported our result using mean Intersection over Unition (mIoU) metric on both 1-shot and 5-shot settings.
Table 1: Results of 1-way 1-shot & 5-shot segmentation on the Pascal 5i data set employing the mIoU metric.
| Methods | Fold 1 | Fold 2 | Fold 3 | Fold 4 | Mean | Fold 1 | Fold 2 | Fold 3 | Fold 4 | Mean | 1 to 5 shot Improvement |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Wei et. all FSS-1000 | - | - | - | - | - | 37.4 | 60.9 | 46.6 | 42.2 | 56.8 | - |
| Shaban et. all OSLSM | 33.6 | 55.3 | 40.9 | 33.5 | 40.8 | 35.9 | 58.1 | 42.7 | 39.1 | 43.9 | 3.1 |
| Rakelly et. all co-FCN | 36.7 | 50.6 | 44.9 | 32.4 | 41.1 | 37.5 | 50.0 | 44.1 | 33.9 | 41.4 | 0.3 |
| Rakelly et. all SG-One | 40.2 | 58.4 | 48.4 | 38.4 | 46.3 | 41.9 | 58.6 | 48.6 | 39.4 | 47.1 | 0.8 |
| Rakelly et. all AMP | 41.9 | 50.2 | 46.7 | 34.7 | 43.4 | 41.8 | 55.5 | 50.3 | 39.9 | 46.9 | 3.5 |
| Rakelly et. all PANet | 42.3 | 58.0 | 51.1 | 41.2 | 48.1 | 51.8 | 64.6 | 59.8 | 46.5 | 55.7 | 7.6 |
| Rakelly et. all Feat Weight | 51.3 | 64.5 | 56.7 | 52.2 | 56.2 | 54.9 | 67.4 | 62.2 | 55.3 | 59.9 | 3.7 |
| Rakelly et. all Meta-Seg | 42.2 | 59.6 | 48.1 | 44.4 | 48.6 | 43.1 | 62.5 | 49.9 | 45.3 | 50.2 | 1.6 |
| Rakelly et. all MDL | 39.7 | 58.3 | 46.7 | 36.3 | 45.3 | 40.6 | 58.5 | 47.7 | 36.6 | 45.9 | 0.6 |
| Rakelly et. all OSAdv | 46.9 | 59.2 | 49.3 | 43.4 | 49.7 | 47.2 | 58.8 | 48.8 | 47.4 | 50.6 | 0.9 |
| Rakelly et. all AMCG | - | - | - | - | 61.2 | - | - | - | - | 62.2 | 1.0 |
| Rakelly et. all CANet | 52.5 | 65.9 | 51.3 | 51.9 | 55.4 | 55.5 | 67.8 | 51.9 | 53.2 | 57.1 | 1.7 |
| Rakelly et. all LTM | 52.8 | 69.6 | 53.2 | 52.3 | 57.0 | 57.9 | 69.9 | 56.9 | 57.5 | 60.6 | 3.6 |
| Rakelly et. all PGNet | 56.0 | 66.9 | 50.6 | 50.4 | 56.0 | 57.7 | 68.7 | 52.9 | 54.6 | 58.5 | 2.5 |
| Azad et. all Proposed Method | 56.2 | 66.0 | 56.1 | 53.8 | 58.0 | 57.5 | 70.6 | 56.6 | 57.7 | 60.6 | 2.6 |
Visual Segmentation result on Pascal 5i
Sample of 1-shot segmentation result on the Pascal 5i dataset
Extra results
Visual representation of proposed method performing segmentation on 1000 class dataset.
Visual demonstration of the proposed method limitation in segmenting the object of interest
Visual representation of proposed method performing segmentation on 1000 class dataset with weak annotation (bounding box)
Query
All implementation done by Reza Azad. For any query please contact us for more information.
rezazad68@gmail.com