Semantic-Segmentation

A list of all papers and resoureces on Semantic Segmentation.

Dataset importance

SemanticSegmentation_DL

Some implementation of semantic segmantation for DL model

Dataset

Resources

Survey papers

A 2017 Guide to Semantic Segmentation with Deep Learning by Qure AI [Blog about different sem. segm. methods]
A Review on Deep Learning Techniques Applied to Semantic Segmentation [Survey paper with a special focus on datasets and the highest performing methods]
Computer Vision for Autonomous Vehicles: Problems, Datasets and State-of-the-Art [Survey paper about all aspects of autonomous vehicles, including sem. segm.] [Webpage with a summary of all relevant publications]

Online demos

2D Semantic Segmentation

Papers:

Stacked Deconvolutional Network for Semantic Segmentation-2017 [Paper]
Deeplab v3: Rethinking Atrous Convolution for Semantic Image Segmentation-2017(DeeplabV3) [Paper]
Learning Object Interactions and Descriptions for Semantic Image Segmentation-2017 [Paper]
Pixel Deconvolutional Networks-2017 [Code-Tensorflow] [Paper]
Dilated Residual Networks-2017 [Paper]
A Review on Deep Learning Techniques Applied to Semantic Segmentation-2017 [Paper]
BiSeg: Simultaneous Instance Segmentation and Semantic Segmentation with Fully Convolutional Networks [Paper]
ICNet for Real-Time Semantic Segmentation on High-Resolution Images-2017 [Project] [Code] [Paper] [Video]
Not All Pixels Are Equal: Difficulty-Aware Semantic Segmentation via Deep Layer Cascade-2017 [Paper]
Loss Max-Pooling for Semantic Image Segmentation-2017 [Paper]
Annotating Object Instances with a Polygon-RNN-2017 [Project] [Paper]
Feature Forwarding: Exploiting Encoder Representations for Efficient Semantic Segmentation-2017 [Project] [Code-Torch7]
Reformulating Level Sets as Deep Recurrent Neural Network Approach to Semantic Segmentation-2017 [Paper]
Adversarial Examples for Semantic Image Segmentation-2017 [Paper]
Large Kernel Matters - Improve Semantic Segmentation by Global Convolutional Network-2017 [Paper]
Label Refinement Network for Coarse-to-Fine Semantic Segmentation-2017 [Paper]
PixelNet: Representation of the pixels, by the pixels, and for the pixels-2017 [Project] [Code-Caffe] [Paper]
LabelBank: Revisiting Global Perspectives for Semantic Segmentation-2017 [Paper]
Progressively Diffused Networks for Semantic Image Segmentation-2017 [Paper]
Understanding Convolution for Semantic Segmentation-2017 [Model-Mxnet] [Paper] [Code]
Predicting Deeper into the Future of Semantic Segmentation-2017 [Paper]
Pyramid Scene Parsing Network-2017 [Project] [Code-Caffe] [Paper] [Slides]
FCNs in the Wild: Pixel-level Adversarial and Constraint-based Adaptation-2016 [Paper]
FusionNet: A deep fully residual convolutional neural network for image segmentation in connectomics-2016 [Code-PyTorch] [Paper]
RefineNet: Multi-Path Refinement Networks for High-Resolution Semantic Segmentation-2016 [Code-MatConvNet] [Paper]
The One Hundred Layers Tiramisu: Fully Convolutional DenseNets for Semantic Segmentation [Code-Theano] [Code-Keras1] [Code-Keras2] [Paper]
Full-Resolution Residual Networks for Semantic Segmentation in Street Scenes [Code-Theano] [Paper]
PixelNet: Towards a General Pixel-level Architecture-2016 [Paper]
Recalling Holistic Information for Semantic Segmentation-2016 [Paper]
Semantic Segmentation using Adversarial Networks-2016 [Paper] [Code-Chainer]
Region-based semantic segmentation with end-to-end training-2016 [Paper]
Exploring Context with Deep Structured models for Semantic Segmentation-2016 [Paper]
Better Image Segmentation by Exploiting Dense Semantic Predictions-2016 [Paper]
Boundary-aware Instance Segmentation-2016 [Paper]
Improving Fully Convolution Network for Semantic Segmentation-2016 [Paper]
Deep Structured Features for Semantic Segmentation-2016 [Paper]
v2: DeepLab:Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs-2016** [Project] [Code-Caffe] [Code-Tensorflow] [Code-PyTorch] [Paper]
DeepLab v1: Semantic Image Segmentation With Deep Convolutional Nets and Fully Connected CRFs-2014** [Code-Caffe1] [Code-Caffe2] [Paper]
Deep Learning Markov Random Field for Semantic Segmentation-2016 [Project] [Paper]
Convolutional Random Walk Networks for Semantic Image Segmentation-2016 [Paper]
ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation-2016 [Code-Caffe1][Code-Caffe2] [Paper] [Blog]
High-performance Semantic Segmentation Using Very Deep Fully Convolutional Networks-2016 [Paper]
ScribbleSup: Scribble-Supervised Convolutional Networks for Semantic Segmentation-2016 [Paper]
Object Boundary Guided Semantic Segmentation-2016 [Code-Caffe] [Paper]
Segmentation from Natural Language Expressions-2016 [Project] [Code-Tensorflow] [Code-Caffe] [Paper]
Seed, Expand and Constrain: Three Principles for Weakly-Supervised Image Segmentation-2016 [Code-Caffe] [Paper]
Global Deconvolutional Networks for Semantic Segmentation-2016 [Paper]
Learning Transferrable Knowledge for Semantic Segmentation with Deep Convolutional Neural Network-2015 [Project] [Code-Caffe] [Paper]
Learning Dense Convolutional Embeddings for Semantic Segmentation-2015 [Paper]
ParseNet: Looking Wider to See Better-2015 [Code-Caffe] [Model-Caffe] [Paper]
Decoupled Deep Neural Network for Semi-supervised Semantic Segmentation-2015 [Project] [Code-Caffe] [Paper]
SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation-2015 [Project] [Code-Caffe] [Paper] [Tutorial1] [Tutorial2]
SegNet: A Deep Convolutional Encoder-Decoder Architecture for Robust Semantic Pixel-Wise Labelling-2015 [Code-Caffe] [Code-Chainer] [Paper]
Semantic Image Segmentation with Task-Specific Edge Detection Using CNNs and a Discriminatively Trained Domain Transform-2015 [Paper]
Semantic Segmentation with Boundary Neural Fields-2015 [Code] [Paper]
Semantic Image Segmentation via Deep Parsing Network-2015 [Project] [Paper1] [Paper2] [Slides]
What’s the Point: Semantic Segmentation with Point Supervision-2015 [Project] [Code-Caffe] [Model-Caffe] [Paper]
U-Net: Convolutional Networks for Biomedical Image Segmentation-2015 [Project] [Code+Data] [Code-Keras] [Code-Tensorflow] [Paper] [Notes]
Learning Deconvolution Network for Semantic Segmentation(DeconvNet)-2015 [Project] [Code-Caffe] [Paper] [Slides]
Multi-scale Context Aggregation by Dilated Convolutions-2015 [Project] [Code-Caffe] [Code-Keras] [Paper] [Notes]
ReSeg: A Recurrent Neural Network-based Model for Semantic Segmentation-2015 [Code-Theano] [Paper]
BoxSup: Exploiting Bounding Boxes to Supervise Convolutional Networks for Semantic Segmentation-2015 [Paper]
Feedforward semantic segmentation with zoom-out features-2015 [Code] [Paper] [Video]
Conditional Random Fields as Recurrent Neural Networks-2015 [Project] [Code-Caffe1] [Code-Caffe2] [Demo] [Paper1] [Paper2]
Efficient Piecewise Training of Deep Structured Models for Semantic Segmentation-2015 [Paper]
Fully Convolutional Networks for Semantic Segmentation-2015 [Code-Caffe] [Model-Caffe] [Code-Tensorflow1] [Code-Tensorflow2] [Code-Chainer] [Code-PyTorch] [Paper1] [Paper2] [Slides1] [Slides2]
Deep Joint Task Learning for Generic Object Extraction-2014 [Project] [Code-Caffe] [Dataset] [Paper]
Highly Efficient Forward and Backward Propagation of Convolutional Neural Networks for Pixelwise Classification-2014 [Code-Caffe] [Paper]
Wider or deeper: Revisiting the resnet model for visual recognition [Paper]
Describing the Scene as a Whole: Joint Object Detection, Scene Classification and Semantic Segmentation[Paper]
Analyzing Semantic Segmentation Using Hybrid Human-Machine CRFs[Paper]
Convolutional Patch Networks with Spatial Prior for Road Detection and Urban Scene Understanding[Paper]
Deep Deconvolutional Networks for Scene Parsing[Paper]
FusionSeg: Learning to combine motion and appearance for fully automatic segmention of generic objects in videos[Paper][Poject]
Deep Dual Learning for Semantic Image Segmentation [Paper]
From image-level to pixel level labeling with convolutional networks [Paper]
Scene Segmentation with DAG-Recurrent Neural Networks [Paper]
Learning to Segment Every Thing [Paper]
Weakly-Supervised Dual Clustering for Image Semantic Segmentation [Paper]
Panoptic Segmentation [Paper]
The Devil is in the Decoder [Paper]
Attention to Scale: Scale-aware Semantic Image Segmentation [Paper][Project]
Convolutional Oriented Boundaries: From Image Segmentation to High-Level Tasks [Paper] [Project]
Scale-Aware Alignment of Hierarchical Image Segmentation [Paper] [Project]
ICCV-2017 Semi Supervised Semantic Segmentation Using Generative Adversarial Network[Paper]

3D Semantic Segmentation

Papers

PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation [Paper]
PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space (2017) [Paper]

3.Learning 3D Mesh Segmentation and Labeling (2010) [Paper]

4.Unsupervised Co-Segmentation of a Set of Shapes via Descriptor-Space Spectral Clustering (2011) [Paper]

5.Single-View Reconstruction via Joint Analysis of Image and Shape Collections (2015) [Paper]

6.3D Shape Segmentation with Projective Convolutional Networks (2017) [Paper]

7.Learning Hierarchical Shape Segmentation and Labeling from Online Repositories (2017) [Paper]

8.3D Graph Neural Networks for RGBD Semantic Segmentation (2017) [Paper]

9.3DCNN-DQN-RNN: A Deep Reinforcement Learning Framework for Semantic Parsing of Large-scale 3D Point Clouds (2017) [Paper]

Scene Understanding

Papers

Dataset & Resources

SUNRGB-D 3D Object Detection Challenge [Link] 19 object categories for predicting a 3D bounding box in real world dimension Training set: 10,355 RGB-D scene images, Testing set: 2860 RGB-D images
SceneNN (2016) [Link] 100+ indoor scene meshes with per-vertex and per-pixel annotation.
ScanNet (2017) [Link] An RGB-D video dataset containing 2.5 million views in more than 1500 scans, annotated with 3D camera poses, surface reconstructions, and instance-level semantic segmentations.
Matterport3D: Learning from RGB-D Data in Indoor Environments (2017) [Link]
10,800 panoramic views (in both RGB and depth) from 194,400 RGB-D images of 90 building-scale scenes of private rooms. Instance-level semantic segmentations are provided for region (living room, kitchen) and object (sofa, TV) categories.
SUNCG: A Large 3D Model Repository for Indoor Scenes (2017) [Link]
The dataset contains over 45K different scenes with manually created realistic room and furniture layouts. All of the scenes are semantically annotated at the object level.
MINOS: Multimodal Indoor Simulator (2017) [Link] MINOS is a simulator designed to support the development of multisensory models for goal-directed navigation in complex indoor environments. MINOS leverages large datasets of complex 3D environments and supports flexible configuration of multimodal sensor suites. MINOS supports SUNCG and Matterport3D scenes.
Facebook House3D: A Rich and Realistic 3D Environment (2017) [Link]
House3D is a virtual 3D environment which consists of 45K indoor scenes equipped with a diverse set of scene types, layouts and objects sourced from the SUNCG dataset. All 3D objects are fully annotated with category labels. Agents in the environment have access to observations of multiple modalities, including RGB images, depth, segmentation masks and top-down 2D map views.
HoME: a Household Multimodal Environment (2017) [Link]
HoME integrates over 45,000 diverse 3D house layouts based on the SUNCG dataset, a scale which may facilitate learning, generalization, and transfer. HoME is an open-source, OpenAI Gym-compatible platform extensible to tasks in reinforcement learning, language grounding, sound-based navigation, robotics, multi-agent learning.
AI2-THOR: Photorealistic Interactive Environments for AI Agents [Link]
AI2-THOR is a photo-realistic interactable framework for AI agents. There are a total 120 scenes in version 1.0 of the THOR environment covering four different room categories: kitchens, living rooms, bedrooms, and bathrooms. Each room has a number of actionable objects.

Medical Image Semantic Segmentation

Papers

Semantic Image Segmentation with Deep Learning [Paper]
Automatic Liver and Tumor Segmentation of CT and MRI Volumes Using Cascaded Fully Convolutional Neural Networks [Paper]
DeepNAT: Deep Convolutional Neural Network for Segmenting Neuroanatomy [Paper]
CNN-based Segmentation of Medical Imaging Data [Paper]
Deep Retinal Image Understanding (http://www.vision.ee.ethz.ch/~cvlsegmentation/driu/data/paper/DRIU_MICCAI2016.pdf)

Codes

Weakly-Supervised-Segmentation

Di Lin, Jifeng Dai, Jiaya Jia, Kaiming He, and Jian Sun.ScribbleSup: Scribble-Supervised Convolutional Networks for Semantic Segmentation. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016
Pathak, Deepak, Philipp Krahenbuhl, and Trevor Darrell. Constrained convolutional neural networks for weakly supervised segmentation. Proceedings of the IEEE International Conference on Computer Vision. 2015.
Papandreou, George, et al. Weakly-and semi-supervised learning of a DCNN for semantic image segmentation. arXiv preprint arXiv:1502.02734 (2015).
Xu, Jia, Alexander G. Schwing, and Raquel Urtasun. Learning to segment under various forms of weak supervision. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2015.

Video Semantic Segmentation

Feature Space Optimization for Semantic Video Segmentation[Paper][Slides]
The Basics of Video Object Segmentation [Blog]
ICCV2017----SegFlow_Joint Learning for Video Object Segmentation and Optical Flow
OSVOS:One-Shot Video Object Segmentation
Surveillance Video Parsing with Single Frame Supervision
The 2017 DAVIS Challenge on Video Object Segmentation
Video Propagation Networks

Result of DAVIS-Challenge 2017 2016----A Benchmark Dataset and Evaluation Methodology for Video Object Segmentation
2016----Clockwork Convnets for Video Semantic Segmentation
2016----MaskTrack ----Learning Video Object Segmentation from Static Images
2017----DAVIS-Challenge-1st----Video Object Segmentation with Re-identification
2017----DAVIS-Challenge-2nd----Lucid Data Dreaming for Multiple Object Tracking
2017----DAVIS-Challenge-3rd----Instance Re-Identification Flow for Video Object Segmentation
2017----DAVIS-Challenge-4th----Multiple-Instance Video Segmentation with Sequence-Specific Object Proposals
2017----DAVIS-Challenge-5th Online Adaptation of Convolutional Neural Networks for the 2017 DAVIS Challenge on Video Object Segmentation
2017----DAVIS-Challenge-6th ----Learning to Segment Instances in Videos with Spatial Propagation Network
2017----DAVIS-Challenge-7th----Some Promising Ideas about Multi-instance Video Segmentation
2017----DAVIS-Challenge-8th----One-Shot Video Object Segmentation with Iterative Online Fine-Tuning
2017----DAVIS-Challenge-9th----Video Object Segmentation using Tracked Object Proposals

Road Segmentation

Papers:

MultiNet: Real-time Joint Semantic Reasoning for Autonomous Driving [Paper]
self-driving-car-road-segmentation [Link]
Efficient Deep Models for Monocular Road Segmentation[Paper]
Semantic Road Segmentation via Multi-scale Ensembles of Learned Features [Paper]
Distantly Supervised Road Segmentation [Paper]
Deep Fully Convolutional Networks with Random Data Augmentation for Enhanced Generalization in Road Detection [Paper]
Real-time category-based and general obstacle detection for autonomous driving [Paper]
Road Scene Segmentation from a Single Image [Paper]
FoveaNet: Perspective-aware Urban Scene Parsing [Paper]

Codes

Transferable Semantic Segmentation

Weakly Supervised Object Localization Using Things and Stuff Transfer [Paper]
Semi and Weakly Supervised Semantic Segmentation Using Generative Adversarial Network [Paper]

Real-Time Semantic Segmentation

LinkNet: Exploiting Encoder Representations for Efficient Semantic Segmentation [Paper]
ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation-2016 [Code-Caffe1][Code-Caffe2] [Paper] [Blog]
Efficient Deep Models for Monocular Road Segmentation[Paper]
Real-Time Coarse-to-fine Topologically Preserving Segmentation[Paper]

Part Semantic Segmentation

Look into Person: Self-supervised Structure-sensitive Learning and A New Benchmark for Human Parsing-2017 [Project] [Code-Caffe] [Paper]
Deep Learning for Human Part Discovery in Images-2016 [Code-Chainer] [Paper]
A CNN Cascade for Landmark Guided Semantic Part Segmentation-2016 [Project] [Paper]
Deep Learning for Semantic Part Segmentation With High-level Guidance-2015 [Paper]
Neural Activation Constellations-Unsupervised Part Model Discovery with Convolutional Networks-2015 [Paper]
Human Parsing with Contextualized Convolutional Neural Network-2015 [Paper]
Part detector discovery in deep convolutional neural networks-2014 [Code] [Paper]
Hypercolumns for object segmentation and fine-grained localization [Paper]

Clothes Parsing

Looking at Outfit to Parse Clothing-2017 [Paper]
Semantic Object Parsing with Local-Global Long Short-Term Memory-2015 [Paper]
A High Performance CRF Model for Clothes Parsing-2014 [Project] [Code] [Dataset] [Paper]
Clothing co-parsing by joint image segmentation and labeling-2013 [Project] [Dataset] [Paper]
Parsing clothing in fashion photographs-2012 [Project] [Paper]

Instance Segmentation

Pixelwise Instance Segmentation with a Dynamically Instantiated Network-2017 [Paper]
Semantic Instance Segmentation via Deep Metric Learning-2017 [Paper]
Mask R-CNN-2017 [Code-Tensorflow] [Paper]
Pose2Instance: Harnessing Keypoints for Person Instance Segmentation-2017 [Paper]
Pixelwise Instance Segmentation with a Dynamically Instantiated Network-2017 [Paper]
Fully Convolutional Instance-aware Semantic Segmentation-2016 [Code] [Paper]
Instance-aware Semantic Segmentation via Multi-task Network Cascades-2015 [Code] [Paper]
Recurrent Instance Segmentation-2015 [Project] [Code-Torch7] [Paper] [Poster] [Video]
Annotating Object Instances with a Polygon-RNN [Paper]
MaskLab: Instance Segmentation by Refining Object Detection with Semantic and Direction Features [Paper]
BlitzNet: A Real-Time Deep Network for Scene Understanding [Paper]
FCIS [Paper]Code
MNC [Paper]Code
DeepMask [Paper]Code
SharpMask [Paper]Code
RIS [Paper]Code
FastMask [Paper]Code

Segment Object Candidates

FastMask: Segment Object Multi-scale Candidates in One Shot-2016 [Code-Caffe] [Paper]
Learning to Refine Object Segments-2016 [Code-Torch] [Paper]
Learning to Segment Object Candidates-2015 [Code-Torch] [Code-Theano-Keras] [Paper]