Computational Reconstruction from RGB to Hyperspectral Imaging: A Survey
A list of papers and resources for spectral reconstruction from images. This page will continue to be updated and will upload our latest research results.
Contents
Introduction
This page mainly describes the overview of spectral reconstruction from RGB images. This work conducts a thorough investigation of more than 25 state-of-the-art spectral reconstruction methods with respect to available database.
Overview
The overall taxonomy of the spectral reconstruction methods.
Datasets
Dataset | Amount | Resolution | Spectral channels | Spectrum/(nm) | Featured scenes |
---|---|---|---|---|---|
CAVE | 32 | 31 | 400-700 | skin, hair, food and drink | |
ICVL | 203 | 31 | 400-700 | urban, rural, indoor and plant | |
BGU-HS | 286 | 31 | 400-700 | urban, rural, indoor and plant | |
ARAD-HS | 510 | 31 | 400-700 | statue, vehicle and paint |
Algorithms
Prior-based methods
Method | Category | Priors |
---|---|---|
Sparse Coding | Dictionary Learning | sparsity |
SR A+ | Dictionary Learning | sparsity, local euclidean linearity |
Multiple Non-negative Sparse Dictionaries | Dictionary Learning | spatial structure similarity, spectral correlation |
Local Linear Embedding Sparse Dictionary | Dictionary Learning | color and texture, local linearity |
Spatially Constrained Dictionary Learning | Dictionary Learning | spatial context |
SR Manifold Mapping | Manifold Learning | low-dimensional manifold |
SR Gaussian Process | Gaussian Process | spectral physics, spatial structure similarity |
Data-driven methods
Linear CNN
This kind of network is stacked convolutional layers, and the design has only one path and does not include multiple branches. (a) HSCNN (b) SR2D/3DNet (c) Residual HSRCNN
U-Net
The U-Net model is composed of an encoder and a decoder. (a) SRUNet (b) SRMSCNN (c) SRMXRUNet (d) SRBFWU-Net with supervised learning and unsupervised learning.
GAN
The generative adversarial network (GAN) model is composed of a generator and a discriminator. (a) SRCGAN takes Conditional GAN as the main framework. (b) SAGAN includes SAP-UNet withoutboundary supervision and SAP-WNet with boundary supervision.
Dense Network
The core idea of dense network is to densely con-nect all front and back layers to achieve higher flexibil-ity and richer feature representation, which can reducethe vanishing of gradients and ensure the stability ofthe network. (a) SRTiramisuNet (b) HSCNN+
Residual Network
Compared with the linear CNN, the residual network can avoid the vanishing of the gradient by further deepening the network. (a) SREfficientNet (b) SREfficientNet+
Attention Network
The attention-based model allows this flexibility and takesinto account that not all features are important for SR.(a) SRAWAN (b) SRHRNet (c) SRRPAN
Multi-branch Network
In contrast to single-stream CNN, the goal of multi-branch network is to obtain a diverse set of features on multiple context scales.This design can alsoachieve multi-path signal flow, leading to better infor-mation exchange during training. (a) SRLWRDNet (b) SRPFMNet
References
Papers - Prior-based methods
- Arad, Boaz, and Ohad Ben-Shahar. "Sparse Recovery of Hyperspectral Signal from Natural RGB Images." In ECCV, 2016.[Paper]
- Aeschbacher, Jonas, Jiqing Wu, and Radu Timofte. "In Defense of Shallow Learned Spectral Reconstruction from RGB Images." In ICCVW, 2017.[Paper][code]
- Fu, Ying, et al. "Spectral Reflectance Recovery from A Single RGB Image." IEEE Transactions on Computational Imaging, 2018.[paper]
- Li, Yuqi, Chong Wang, and Jieyu Zhao. "Locally Linear Embedded Sparse Coding for Spectral Reconstruction from RGB Images." IEEE Signal Processing Letters, 2017.[paper]
- Geng, Yunhao, et al. "Spatial Constrained Hyperspectral Reconstruction from RGB Inputs Using Dictionary Representation." IGARSS ,2019.[paper]
- Jia, Yan, et al. "From RGB to Spectrum for Natural Scenes via Manifold-based Mapping." In ICCV, 2017.[paper]
- Akhtar, Naveed, and Ajmal Mian. "Hyperspectral Recovery from RGB Images Using Gaussian Processes." IEEE transactions on pattern analysis and machine intelligence, 2018.[paper]
Papers - Data-driven methods
- Xiong, Zhiwei, et al. "HSCNN: CNN-based Hyperspectral Image Recovery from Spectrally Undersampled Projections." In ICCVW, 2017.[paper]
- Koundinya, Sriharsha, et al. "2D-3D CNN based Architectures for Spectral Reconstruction from RGB Images." In ICCVW, 2018.[paper]
- Han, Xian-Hua, Boxin Shi, and Yinqiang Zheng. "Residual HSRCNN: Residual Hyperspectral Reconstruction CNN from an RGB Image." In ICPR, 2018.[paper]
- Stiebel, Tarek, et al. "Reconstructing Spectral Images from RGB Images Using a Convolutional Neural Network."In ICCVW, 2018.[paper]
- Yan, Yiqi, et al. "Accurate Spectral Super-resolution from Single RGB Image Using Multi-scale CNN." In PRCV, 2018.[paper][code]
- Banerjee, Atmadeep, and Akash Palrecha. "Mxr-u-nets for Real Time Hyperspectral Reconstruction." arXiv, 2020.[paper][code]
- Fubara, Biebele Joslyn, Mohamed Sedky, and David Dyke. "RGB to Spectral Reconstruction via Learned Basis Functions and Weights." In CVPRW, 2020.[paper]
- Alvarez-Gila, Aitor, Joost Van De Weijer, and Estibaliz Garrote. "Adversarial Networks for Spatial Context-aware Spectral Image Reconstruction from RGB." In CVPRW, 2018.[paper]
- Liu, Pengfei, and Huaici Zhao. "Adversarial Networks for Scale Feature-Attention Spectral Image Reconstruction from a Single RGB." Sensors, 2020.[paper]
- Galliani, Silvano, et al. "Learned Spectral Super-resolution." arXiv ,2017.[paper]
- Shi, Zhan, et al. "HSCNN+: Advanced CNN-based Hyperspectral Recovery from RGB Images." IN CVPRW, 2018.[paper][code]
- Can, Yigit Baran, and Radu Timofte. "An Efficient CNN for Spectral Reconstruction from RGB Images." arXiv, 2018.[paper]
- Kaya, Berk, Yigit Baran Can, and Radu Timofte. "Towards Spectral Estimation from a Single RGB Image in the Wild." In ICCVW, 2019.[paper]
- Li, Jiaojiao, et al. "Adaptive Weighted Attention Network with Camera Spectral Sensitivity Prior for Spectral Reconstruction from RGB Images." In CVPRW, 2020.[paper][code]
- Zhao, Yuzhi, et al. "Hierarchical Regression Network for Spectral Reconstruction from RGB Images." In CVPRW, 2020.[paper][code]
- Peng, Hao, Xiaomei Chen, and Jie Zhao. "Residual Pixel Attention Network for Spectral Reconstruction from RGB Images." In CVPRW, 2020.[paper]
- Nathan, D. Sabari, et al. "Light Weight Residual Dense Attention Net for Spectral Reconstruction from RGB Images." arXiv, 2020.[paper]
- Zhang, Lei, et al. "Pixel-aware Deep Function-mixture Network for Spectral Super-resolution." In AAAI, 2020.[paper]